Regularization
Anything that stops a model fitting the training data too well — a collection of tricks, held together by a story that modern deep learning broke.
When not to use it
- When you're underfitting. Adding regularization to a model that can't fit the training data makes the actual problem worse.
- Instead of more data. More data is the better version and people reach for the dial first.
- Dropout in modern transformers, reflexively. Largely absent from current architectures for a reason.
- L2 under Adam, expecting weight decay. They're not equivalent under adaptive optimisers. Use AdamW.
Reach for something else instead
- More data — the honest answer.
- Data augmentation — more data, synthesised.
- Early stopping — free, and underused.
- A smaller model — sometimes the right call, though double descent complicates the reflex.
Sources & further reading
- Zhang et al. (2017), Understanding Deep Learning Requires Rethinking Generalization — networks memorise random labels with regularization on. The paper that broke the story.
- Srivastava et al. (2014), Dropout: A Simple Way to Prevent Neural Networks from Overfitting — the technique that defined an era.
- Belkin et al. (2019), Reconciling Modern Machine Learning Practice and the Classical Bias-Variance Trade-off — double descent; more capacity, better generalisation, past the threshold.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Regularising an underfitting model, which is the opposite of the fix.
- Believing the capacity-constraint story. Zhang et al. showed networks memorise random labels with regularization enabled.
- Using L2 with Adam and thinking you have weight decay. You have a distorted version of it.
- Stacking every technique at once, so you can't tell which is doing anything.
- Assuming more parameters means more overfitting. Double descent says otherwise, and nobody fully knows why.
At a glance
FieldMachine Learning
Classical storyconstrain capacity, prevent memorisation
Status of that storybroken by Zhang et al. (2017)
Best versionmore data
Cheapestearly stopping
Under Adamuse AdamW, not L2
Honest stateempirical folklore
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Regularization vs. more data — one constrains the model, the other removes the reason to constrain it. Only the second works for the reason the textbook gives.