Regularization
Anything that stops a model fitting the training data too well — a collection of tricks, held together by a story that modern deep learning broke.
When not to use it
- When you're underfitting. Adding regularization to a model that can't fit the training data makes the actual problem worse.
- Instead of more data. More data is the better version and people reach for the dial first.
- Dropout in modern transformers, reflexively. Largely absent from current architectures for a reason.
- L2 under Adam, expecting weight decay. They're not equivalent under adaptive optimisers. Use AdamW.
Reach for something else instead
- More data — the honest answer.
- Data augmentation — more data, synthesised.
- Early stopping — free, and underused.
- A smaller model — sometimes the right call, though double descent complicates the reflex.
Regularising an underfitting model is the opposite of the fix.
Two ridge regressions over the same data, sharing one penalty slider. For the flexible model the familiar story holds: a little penalty helps, and there is a best value. For the straight line there is no best value — every increase makes it worse, monotonically, because regularisation shrinks coefficients toward zero and a model that is already too rigid does not need more rigidity. The two curves have different shapes, not different positions on the same shape. Which is why “the model is bad, add regularisation” is a coin flip: it is the correct move for one failure and the exact wrong move for the other.
Further reading
- Zhang et al. (2017), Understanding Deep Learning Requires Rethinking Generalization — networks memorise random labels with regularization on. The paper that broke the story.
- Srivastava et al. (2014), Dropout: A Simple Way to Prevent Neural Networks from Overfitting — the technique that defined an era.
- Belkin et al. (2019), Reconciling modern machine-learning practice and the classical bias–variance trade-off — double descent; more capacity, better generalisation, past the threshold.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Regularising an underfitting model, which is the opposite of the fix.
- Believing the capacity-constraint story. Zhang et al. showed networks memorise random labels with regularization enabled.
- Using L2 with Adam and thinking you have weight decay. You have a distorted version of it.
- Stacking every technique at once, so you can't tell which is doing anything.
- Assuming more parameters means more overfitting. Double descent says otherwise, and nobody fully knows why.
At a glance
Often compared with
Where this sits
5 concepts come first. Understanding it opens up 1 more.
Computed from the prerequisite graph, not assigned. How this works