Overfitting
When a model memorizes its training data instead of learning the general pattern — so it looks great in training but fails on new data.
When not to use it
- As the explanation for every disappointing model. Poor test performance is just as often bad features, leaked data, a mismatched test set, or a task the model can't do.
- As a reason to always simplify. Underfitting is the opposite failure and gets diagnosed far less often, because a simple model failing looks like an honest attempt.
Reach for something else instead
- Regularisation, dropout, early stopping — the standard tools, and they work.
- More or better data beats every clever fix. Diversity in the training set does more than any hyperparameter.
- Cross-validation so you find out on your own machine rather than in production.
Sources & further reading
- Srivastava et al. (2014), Dropout: A Simple Way to Prevent Neural Networks from Overfitting.
- Zhang et al. (2016), Understanding deep learning requires rethinking generalization — networks can memorise pure noise, which broke the textbook story.
- Belkin et al. (2019), Reconciling modern machine learning practice and the bias-variance trade-off — double descent, and why the classic U-shaped curve isn't the whole picture.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Tuning against the test set. Do it enough times and you've overfitted to your own evaluation while believing you're measuring generalisation.
- Watching only training loss. It goes down by definition; that's what training does.
- Assuming a big gap between train and test always means overfitting. It can also mean your test set is drawn from a different world than your training set.
At a glance
FieldMachine Learning
Core ideamemorizing instead of generalizing
Detected bytrain vs. validation gap
Remediesmore data, regularization, early stopping
DifficultyBeginner → Intermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4