Gradient Descent
Walking downhill on the error surface, one small step at a time — how a model's weights actually get updated.
When not to use it
- On non-differentiable objectives. No slope, no descent. You need a different family of methods entirely.
- On small convex problems with closed-form solutions. If linear regression has an exact answer, take the exact answer.
- When the function is expensive and the parameters are few. Bayesian optimisation is better suited to that shape of problem.
Reach for something else instead
- Second-order methods — use curvature, converge in fewer steps, historically too expensive at scale but currently being revisited.
- Evolutionary strategies for black-box objectives where you can't compute a gradient at all.
- Closed-form solutions when they exist. They're exact and instant, and it's worth checking before reaching for an optimiser.
Read more on the blog
- How neural networks work: the idea under all of itUnder every transformer, every image generator, every language model, sits one idea: the neural network. Here's how it actually works, neurons, weights, layers, and the simple trick by which it learns from its own mistakes, explained so it finally makes sense.
- Why does deep learning work? The generalization mysteryThe most important technology of the decade rests on a foundation we do not fully understand. By the textbook, deep networks are so oversized they should memorize their training data and fail on everything else. Instead they generalize beautifully, and nobody can fully explain why. This is the generalization mystery, and it is one of the deepest open problems in AI.
Gradient descent: roll downhill to the lowest error.
Picture the model's error as a landscape — high where the model is wrong, low where it's right. Gradient descent computes the slope at the current point and takes a step downhill. The step size is the learning rate: too big and it overshoots, too small and it crawls. Repeat until it settles in a valley — a set of weights with low error.
Worked example
One concrete step. Suppose we're minimising a simple error curve f(w) = w², currently sitting at w = 4, with learning rate 0.1.
at w=4: gradient = 2×4 = 8
new w = w − (learning rate × gradient)
new w = 4 − (0.1 × 8) = 4 − 0.8 = 3.2
The error at w=4 was 16; at w=3.2 it's 10.24 — we moved downhill. Take another step:
new w = 3.2 − (0.1 × 6.4) = 2.56
Each step is smaller than the last, because the slope flattens as we near the bottom (w=0, the minimum). That's gradient descent in miniature: read the slope, step against it, repeat. Too large a learning rate — say 1.0 — and the step would overshoot (4 − 8 = −4) and diverge instead of settling.
Further reading
- Kingma & Ba (2014), Adam: A Method for Stochastic Optimization — the default optimiser, and why adaptive rates work.
- Smith (2015), Cyclical Learning Rates for Training Neural Networks — where the learning rate finder comes from.
- Wilson et al. (2017), The Marginal Value of Adaptive Gradient Methods in Machine Learning — the counter-argument: SGD can generalise better. Worth reading alongside Adam.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Tuning everything except the learning rate. It matters more than architecture choices people agonise over.
- Reading a plateau as convergence. It may be a saddle point, or a decayed rate, or a dead layer.
- Copying a learning rate from a paper with a different batch size. They scale together, and the number alone means nothing.
At a glance
Often compared with
Where this sits
6 concepts come first. Understanding it opens up 10 more.
Computed from the prerequisite graph, not assigned. How this works