Inductive Bias
The assumptions a model makes before seeing any data — without them learning is impossible, and there's a theorem.
When not to use it
- (It's unavoidable. The question is which one.)*
- A weak bias with little data. A ViT on a million images loses to a CNN. That's the trade, quantified.
- A strong bias that's wrong. Rotation-invariance on digits turns 6 into 9.
- Assuming No Free Lunch means all algorithms are equal. It means your advantage comes from your assumptions matching reality.
- Trying to eliminate bias. You can make it general. You can't make it zero.
Reach for something else instead
- (Ways to get the bias from somewhere else.)*
- More data — buys you the right to a weaker bias.
- Transfer learning — inherit a bias someone else paid to learn.
- Data augmentation — state your invariances in data rather than architecture.
- Architecture choice — the most direct lever, and it's a bet on the domain.
Sources & further reading
- Mitchell (1980), The Need for Biases in Learning Generalizations — a bias-free learner cannot generalise. The formal statement.
- Wolpert & Macready (1997), No Free Lunch Theorems for Optimization — averaged over all problems, all algorithms tie. Read what it actually claims.
- Battaglia et al. (2018), Relational inductive biases, deep learning, and graph networks — the clearest map of which architecture assumes what.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating bias as a flaw. Without it, learning is impossible — that's Mitchell's theorem.
- Reading No Free Lunch as "nothing matters." It says your assumptions are the source of all your performance.
- Using a low-bias architecture on a small dataset, then blaming the architecture.
- Not noticing the biases you didn't choose — your optimiser and your augmentations have opinions.
At a glance
FieldMachine Learning
What it isthe assumptions that let you pick one hypothesis from infinitely many
The theorema bias-free learner cannot generalise at all (Mitchell, 1980)
The tradestrong bias, less data, wrong when it misfits
The demonstrationViT beats CNN at 300M images, loses at 1M
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Strong vs. weak inductive bias — one learns from little data and breaks when its assumption is wrong; the other needs enormous data and finds a structure better than the one you'd have specified.