Bias-Variance Tradeoff
The two ways a model can be wrong, and the classical claim that fixing one worsens the other — which modern deep learning appears to violate.
Reviewed July 11, 2026Stable
When not to use it
- As a law about deep networks. Overparameterised models violate the classical prediction, and double descent shows the curve isn't a U.
- As something you can measure. You can't observe your model's bias on real data. It's a framework for reasoning, not a metric.
- To justify a simpler model on principle. "Avoiding variance" is not a reason if the flexible model demonstrably generalises. Check, don't theorise.
Reach for something else instead
- (Ways to think about the same question.)*
- Learning curves — plot train and test error against dataset size. Answers "will more data help" directly and empirically.
- Cross-validation — measures generalisation without needing to decompose why.
- Double descent framing — for modern overparameterised models, a better mental picture.
Read more on the blog
- Why neural networks generalize when theory says they can'tA network with more parameters than training examples can memorize random labels perfectly. The same network, on real data, generalizes. Classical learning theory has no account of why, and the reason it fails is more precise than "it was wrong."
- Regularization: L1, L2, dropout and what unites themL2 regularization is not analogous to a Gaussian prior. It is exactly a Gaussian prior, and the identity is provable. Six techniques taught separately turn out to be one idea in different clothes.
- Why does deep learning work? The generalization mysteryThe most important technology of the decade rests on a foundation we do not fully understand. By the textbook, deep networks are so oversized they should memorize their training data and fail on everything else. Instead they generalize beautifully, and nobody can fully explain why. This is the generalization mystery, and it is one of the deepest open problems in AI.
More data cannot fix a model that cannot represent the answer.
Linear model0.00high bias — watch it not move
Flexible model0.00high variance — data helps
14 points
Both models are fitted live by least squares on the same generated sample. Drag the data slider: the flexible model steadily improves, because its errors come from having too little data to pin down a shape it can express. The straight line does not improve at all — it is already fitting the best line there is, and the best line is wrong. Its error is not uncertainty; it is the shape of the model. No quantity of data changes the shape. This is the difference between high variance, which data fixes, and high bias, which data cannot touch.
Further reading
- Geman, Bienenstock & Doursat (1992), Neural Networks and the Bias/Variance Dilemma — the classical statement.
- Belkin et al. (2019), Reconciling modern machine-learning practice and the classical bias–variance trade-off — double descent; the U-curve is only half the story.
- Zhang et al. (2017), Understanding Deep Learning Requires Rethinking Generalization — networks can memorise random labels and still generalise on real data. The puzzle, stated cleanly.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Collecting more data to fix high bias. It won't help; the model can't represent the answer regardless of how many examples it sees.
- Adding regularisation to a model that's underfitting. You're making the actual problem worse.
- Treating the U-curve as universal. Past the interpolation threshold, it descends again.
- Assuming more parameters means more overfitting. Zhang et al. and double descent both say otherwise, and nobody fully knows why.
- Chasing error below the irreducible noise floor. That budget is gone; you're fitting randomness.
At a glance
FieldMachine Learning
Biaswrong the same way every time
Variancewrong differently every time
Classical claimyou must trade them
Modern wrinkledouble descent
Bagging fixesvariance
Boosting fixesbias
DifficultyIntermediate
Often compared with
Bias vs. variance — too simple to be right vs. too flexible to be stable. The classical trade-off is real for classical models and incomplete for modern ones.
Where this sits
5 concepts come first. Understanding it opens up 1 more.
5Levelsteps in
5Needs firstconcepts
1Opens up0% of 310
1Areastays here
Learn these firstOverfitting
Computed from the prerequisite graph, not assigned. How this works