Inductive Bias
The assumptions a model makes before seeing any data — without them learning is impossible, and there's a theorem.
Reviewed July 14, 2026Stable
When not to use it
- (It's unavoidable. The question is which one.)*
- A weak bias with little data. A ViT on a million images loses to a CNN. That's the trade, quantified.
- A strong bias that's wrong. Rotation-invariance on digits turns 6 into 9.
- Assuming No Free Lunch means all algorithms are equal. It means your advantage comes from your assumptions matching reality.
- Trying to eliminate bias. You can make it general. You can't make it zero.
Reach for something else instead
- (Ways to get the bias from somewhere else.)*
- More data — buys you the right to a weaker bias.
- Transfer learning — inherit a bias someone else paid to learn.
- Data augmentation — state your invariances in data rather than architecture.
- Architecture choice — the most direct lever, and it's a bet on the domain.
Read more on the blog
- Why neural networks generalize when theory says they can'tA network with more parameters than training examples can memorize random labels perfectly. The same network, on real data, generalizes. Classical learning theory has no account of why, and the reason it fails is more precise than "it was wrong."
- Regularization: L1, L2, dropout and what unites themL2 regularization is not analogous to a Gaussian prior. It is exactly a Gaussian prior, and the identity is provable. Six techniques taught separately turn out to be one idea in different clothes.
- Neurosymbolic AI: what the symbolic half gets rightOn a structured manipulation task in early 2026, a neurosymbolic system scored 95% against 34% for a fine-tuned vision-language model, using less energy. Symbolic AI lost the last argument on cost, not on merit.
Further reading
- Mitchell (1980), The Need for Biases in Learning Generalizations — a bias-free learner cannot generalise. The formal statement.
- Wolpert & Macready (1997), No Free Lunch Theorems for Optimization — averaged over all problems, all algorithms tie. Read what it actually claims.
- Battaglia et al. (2018), Relational inductive biases, deep learning, and graph networks — the clearest map of which architecture assumes what.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating bias as a flaw. Without it, learning is impossible — that's Mitchell's theorem.
- Reading No Free Lunch as "nothing matters." It says your assumptions are the source of all your performance.
- Using a low-bias architecture on a small dataset, then blaming the architecture.
- Not noticing the biases you didn't choose — your optimiser and your augmentations have opinions.
At a glance
FieldMachine Learning
What it isthe assumptions that let you pick one hypothesis from infinitely many
The theorema bias-free learner cannot generalise at all (Mitchell, 1980)
The tradestrong bias, less data, wrong when it misfits
The demonstrationViT beats CNN at 300M images, loses at 1M
DifficultyIntermediate
Often compared with
Strong vs. weak inductive bias — one learns from little data and breaks when its assumption is wrong; the other needs enormous data and finds a structure better than the one you'd have specified.
Where this sits
4 concepts come first. Understanding it opens up 1 more.
4Levelsteps in
4Needs firstconcepts
1Opens up0% of 310
1Areastays here
Learn these firstGeneralization
Computed from the prerequisite graph, not assigned. How this works