Overfitting
When a model memorizes its training data instead of learning the general pattern — so it looks great in training but fails on new data.
When not to use it
- As the explanation for every disappointing model. Poor test performance is just as often bad features, leaked data, a mismatched test set, or a task the model can't do.
- As a reason to always simplify. Underfitting is the opposite failure and gets diagnosed far less often, because a simple model failing looks like an honest attempt.
Reach for something else instead
- Regularisation, dropout, early stopping — the standard tools, and they work.
- More or better data beats every clever fix. Diversity in the training set does more than any hyperparameter.
- Cross-validation so you find out on your own machine rather than in production.
This entry is part of a longer guide: What is machine learning?
Read more on the blog
- Fixing the seed does not make it reproducibleTraining the same network fifty times with an identical seed produced almost as much variance as fifty different seeds. Roughly 80% of the spread came from GPU arithmetic, not from randomness anyone controls.
- Why neural networks generalize when theory says they can'tA network with more parameters than training examples can memorize random labels perfectly. The same network, on real data, generalizes. Classical learning theory has no account of why, and the reason it fails is more precise than "it was wrong."
- Why machine learning does not do error barsModels differing only by random seed showed 0.057% variance in accuracy and 28.9% in certified robustness. The variance is not uniform, and knowing where it concentrates matters more than demanding error bars everywhere.
- Regularization: L1, L2, dropout and what unites themL2 regularization is not analogous to a Gaussian prior. It is exactly a Gaussian prior, and the identity is provable. Six techniques taught separately turn out to be one idea in different clothes.
Training loss goes down by definition. That is what training does.
Both losses come from a real least-squares fit computed in the page — a polynomial of the degree you choose, solved against the same 14 training points every time. Drag right and watch the two numbers separate: training error falls toward zero because a high-degree polynomial can pass through anything, while test error climbs because passing through the noise is not the same as learning the signal. A model that fits its training data perfectly has told you nothing except that it had enough parameters to memorise it.
Further reading
- Srivastava et al. (2014), Dropout: A Simple Way to Prevent Neural Networks from Overfitting.
- Zhang et al. (2017), Understanding deep learning requires rethinking generalization — networks can memorise pure noise, which broke the textbook story.
- Belkin et al. (2019), Reconciling modern machine-learning practice and the classical bias–variance trade-off — double descent, and why the classic U-shaped curve isn't the whole picture.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Tuning against the test set. Do it enough times and you've overfitted to your own evaluation while believing you're measuring generalisation.
- Watching only training loss. It goes down by definition; that's what training does.
- Assuming a big gap between train and test always means overfitting. It can also mean your test set is drawn from a different world than your training set.
At a glance
Where this sits
4 concepts come first. Understanding it opens up 4 more.
Computed from the prerequisite graph, not assigned. How this works