Grokking
A model that memorises, plateaus at chance on unseen data for a very long time, then abruptly generalises — and the sudden part turns out not to be sudden.
Reviewed July 16, 2026Stable
When not to use it
- As a training strategy. It's demonstrated on small algorithmic tasks under specific conditions; "train much longer" is not general advice.
- As evidence of insight or understanding. The mechanistic account describes a circuit forming gradually, which is the opposite of a sudden realisation.
- To explain a plateau in your own training. Most plateaus are a learning rate, a dead layer, or a saddle point — check those first, all three are more likely.
Reach for something else instead
- Standard early stopping is still correct for essentially all production training.
- Learning-rate schedules address the plateaus you'll actually encounter.
- Mechanistic interpretability is the honest tool if you want to know what's forming inside — which is exactly what resolved grokking.
Sources & further reading
- Power et al. (2022), Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — the paper that documented and named it.
- Nanda et al. (2023), Progress Measures for Grokking via Mechanistic Interpretability — reverse-engineered the learned algorithm and showed the circuit forms gradually across the plateau.
- Zhang et al. (2017), Understanding Deep Learning Requires Rethinking Generalization — the memorisation backdrop grokking plays out against.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Describing it as sudden understanding. The internal circuit forms gradually; only the test-accuracy metric is discontinuous — the same shape as emergence.
- Expecting it on real tasks. The conditions are narrow: small algorithmic data, weight decay on, and enormous patience.
- Concluding your plateau is a grokking plateau. It's almost certainly a decayed learning rate or a saddle point, and those have fixes.
At a glance
FieldDeep Learning
Shapememorise, long plateau, abrupt generalisation
Requiressmall algorithmic data + weight decay
Resolved bymechanistic interpretability, 2023
DifficultyAdvanced