Learning Rate
How big a step to take when the model updates — the single most important number in training, and the one most people leave at the default.
When not to use it
- (You always have one. The question is when the default betrays you.)*
- A constant learning rate on a transformer. No warmup means divergence, often in the first hundred steps.
- A pretraining learning rate for fine-tuning. 1e-3 on a pretrained model destroys what it knew.
- The same LR after changing batch size. They're coupled — double the batch, double the rate.
- Someone else's config, unexamined. It was tuned for their model, their data, their batch size.
Reach for something else instead
- LR range test — ten minutes, and it just tells you.
- Learning-rate-free optimisers — adapt the step automatically; genuinely promising.
- μP — tune on a small model, transfer to the large one. What frontier labs do.
- Cosine with warmup — the default that works when you don't want to think.
One number. Same model, same steps, three different outcomes.
Gradient descent on an elongated bowl, 60 steps from the same start, computed live. There is a hard limit here and it is not a matter of taste: on a quadratic with curvature k, any rate above 2/k diverges — the step overshoots further than it came, every time, forever. Below it you converge; far below it you converge too slowly to matter. That is why this is the hyperparameter that decides whether the run works at all, and why a diverging transformer is usually a learning-rate problem being blamed on the architecture.
The full account
The threshold the figure computes
Classical optimisation theory gives a clean answer for how large a step you may take. On a quadratic with curvature k — formally, where the largest eigenvalue of the Hessian is k — gradient descent converges if and only if the step size η stays below 2/k. Above it, each step overshoots by more than it corrects, the error amplifies, and the iterate diverges. The figure above computes this exactly: at k=14 the training blows up at precisely η=2/14=0.143, not near it.
This is not a heuristic. It is a theorem about quadratics, it is tight, and it is the reason every practitioner's mental model of the learning rate is a cliff you must stay behind.
What networks actually do
Cohen, Kaur, Li, Kolter and Talwalkar measured the sharpness — the top Hessian eigenvalue — along real training runs in 2021, across architectures, data modalities and loss functions, and found neural networks do something the theory does not permit.
Training has two phases. In the first, which they named progressive sharpening, the sharpness rises steadily. The optimiser is not seeking flat regions; it is climbing into sharper ones, all on its own, while the loss falls monotonically. It keeps climbing until it reaches 2/η — the exact divergence threshold, the number in your figure.
Then it stops climbing and stays there. This is the edge of stability. The sharpness hovers at the boundary, frequently crossing above it. The loss stops falling monotonically and starts oscillating. And it keeps decreasing on average anyway. Training continues, successfully, in a regime classical theory says is divergent — indefinitely for mean-squared loss; for cross-entropy the sharpness eventually declines again later in training.
So the cliff is real, the arithmetic is exact, and gradient descent walks along its edge on purpose for the entire run.
| Classical theory | What Cohen et al. measured | |
|---|---|---|
| Stable when | sharpness < 2/η, strictly | sharpness sits at 2/η, often above |
| Sharpness during training | a property of the problem | rises to meet the threshold — progressive sharpening |
| Above the threshold | divergence | loss oscillates, decreases on average |
| Your learning rate | must stay under the cliff | selects which cliff the network climbs to |
The inversion worth carrying
Read the second column again, because it turns the standard mental model inside out. You do not choose η to stay below the sharpness. The sharpness rises to meet whatever η you chose. The learning rate is not a safety margin against a fixed landscape — it is a selector for which region of the landscape training ends up in. A small η permits and produces a sharp solution; a large η forces a flat one, because anything sharper would be unstable at that step size.
That connects the learning rate directly to the flat-minima literature and to generalisation, and it explains something practitioners have known and been unable to justify: large initial learning rates often generalise better, at the cost of slower early loss reduction. You are not tolerating a worse optimiser. You are constraining which minima are reachable at all.
Why it doesn't just explode
Damian, Nichani and Lee supplied the mechanism in 2022 and named it self-stabilization. When the sharpness pushes above 2/η, the iterate starts oscillating along the sharpest direction. Those oscillations grow — and as they grow, a third-order term in the loss expansion kicks in that reduces the sharpness, pushing the iterate back toward flatter ground. The instability is its own correction. Gradient descent has a negative feedback loop nobody designed, which is why the boundary is an attractor rather than a cliff edge.
This has been extended to adaptive methods: Cohen and colleagues showed in 2022 that Adam does the same thing, with the threshold adjusted for its preconditioner. It is not an artefact of vanilla SGD.
The caveat that undermines the whole thing
Here is the part that a tidy version of this page would leave out. Cohen's result is about full-batch gradient descent, and nobody trains that way.
Andreyev and Beneventano made the point directly: existing edge-of-stability analyses explicitly do not apply to mini-batch SGD, because no curvature quantity like λ_max is known to constrain SGD the same way. They argue mini-batch training operates in a different regime — what saturates at 2/η is not the Hessian's top eigenvalue but a quantity they call batch sharpness, the expected directional curvature of mini-batch Hessians along their own gradients. And λ_max, which is generally smaller, gets suppressed — which happens to explain the long-standing observation that smaller batches and larger steps find flatter minima.
They also note something that should make you cautious about diagnosing this in your own runs: for constant-step-size SGD, loss oscillations are essentially ubiquitous, and therefore not a signal that you are at an instability boundary. The most visible symptom of the phenomenon is not diagnostic of it.
So the honest 2026 position: the most striking result about the learning rate concerns an algorithm nobody uses, the mini-batch version is a different phenomenon that was only characterised recently, and the practical advice — warm up, use a schedule, tune it, and expect large rates to generalise better — long predates any of the theory and did not come from it.
What to do on Monday
Tune the learning rate first and by a wide margin. It remains the single most consequential hyperparameter, and the figure shows why: the failure is a cliff, not a slope, and everything else is a rounding error next to being on the wrong side of it.
Warm up. Starting at your target rate from a random initialisation is the one regime where the sharpness has not yet organised itself, and it is where divergence actually happens in practice.
Don't read oscillation as a bug to eliminate. It may be the edge of stability doing its job — or, per Andreyev and Beneventano, it may be nothing at all, since constant-step SGD oscillates regardless.
And if you are scaling a model up, look at μP (Yang et al., 2022) before you re-tune anything. It reparameterises the network so the optimal learning rate is approximately invariant to width — tune on a small proxy model, transfer to the large one. That changes the economics of a large run more than any insight on this page.
Further reading
- Smith (2017), Cyclical Learning Rates for Training Neural Networks — the LR range test; ten minutes that beats guessing.
- Loshchilov & Hutter (2017), SGDR: Stochastic Gradient Descent with Warm Restarts — cosine schedules, now the default.
- Yang et al. (2022), Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer — μP; tune small, transfer to large.
- Cohen, Kaur, Li, Kolter & Talwalkar (2021), Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability — ICLR; sharpness climbs to exactly 2/η and stays there, and training works anyway.
- Damian, Nichani & Lee (2022), Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability — why it doesn't diverge: a third-order term steers the iterate back.
- Andreyev & Beneventano (2024), Edge of Stochastic Stability — the caveat that matters: the full-batch result doesn't transfer to mini-batch SGD, which is what everyone runs.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Leaving it at the default and tuning everything else. It's the one that matters most.
- No warmup on a transformer. It will diverge and you'll blame the architecture.
- Fine-tuning at pretraining rates. The model gets worse at everything, confusingly.
- Changing batch size without changing the learning rate, then concluding large batches don't work.
- Assuming theory can tell you the right value. Seventy years of optimisation theory, and it's still a plot.
At a glance
Often compared with
Where this sits
10 concepts come first. Understanding it opens up 1 more.
Computed from the prerequisite graph, not assigned. How this works