Bias-Variance Tradeoff
The two ways a model can be wrong, and the classical claim that fixing one worsens the other — which modern deep learning appears to violate.
When not to use it
- As a law about deep networks. Overparameterised models violate the classical prediction, and double descent shows the curve isn't a U.
- As something you can measure. You can't observe your model's bias on real data. It's a framework for reasoning, not a metric.
- To justify a simpler model on principle. "Avoiding variance" is not a reason if the flexible model demonstrably generalises. Check, don't theorise.
Reach for something else instead
- (Ways to think about the same question.)*
- Learning curves — plot train and test error against dataset size. Answers "will more data help" directly and empirically.
- Cross-validation — measures generalisation without needing to decompose why.
- Double descent framing — for modern overparameterised models, a better mental picture.
Sources & further reading
- Geman, Bienenstock & Doursat (1992), Neural Networks and the Bias/Variance Dilemma — the classical statement.
- Belkin et al. (2019), Reconciling Modern Machine Learning Practice and the Classical Bias-Variance Trade-off — double descent; the U-curve is only half the story.
- Zhang et al. (2017), Understanding Deep Learning Requires Rethinking Generalization — networks can memorise random labels and still generalise on real data. The puzzle, stated cleanly.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Collecting more data to fix high bias. It won't help; the model can't represent the answer regardless of how many examples it sees.
- Adding regularisation to a model that's underfitting. You're making the actual problem worse.
- Treating the U-curve as universal. Past the interpolation threshold, it descends again.
- Assuming more parameters means more overfitting. Zhang et al. and double descent both say otherwise, and nobody fully knows why.
- Chasing error below the irreducible noise floor. That budget is gone; you're fitting randomness.
At a glance
FieldMachine Learning
Biaswrong the same way every time
Variancewrong differently every time
Classical claimyou must trade them
Modern wrinkledouble descent
Bagging fixesvariance
Boosting fixesbias
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Bias vs. variance — too simple to be right vs. too flexible to be stable. The classical trade-off is real for classical models and incomplete for modern ones.