Gradient Descent
Walking downhill on the error surface, one small step at a time — how a model's weights actually get updated.
When not to use it
- On non-differentiable objectives. No slope, no descent. You need a different family of methods entirely.
- On small convex problems with closed-form solutions. If linear regression has an exact answer, take the exact answer.
- When the function is expensive and the parameters are few. Bayesian optimisation is better suited to that shape of problem.
Reach for something else instead
- Second-order methods — use curvature, converge in fewer steps, historically too expensive at scale but currently being revisited.
- Evolutionary strategies for black-box objectives where you can't compute a gradient at all.
- Closed-form solutions when they exist. They're exact and instant, and it's worth checking before reaching for an optimiser.
Sources & further reading
- Kingma & Ba (2014), Adam: A Method for Stochastic Optimization — the default optimiser, and why adaptive rates work.
- Smith (2015), Cyclical Learning Rates for Training Neural Networks — where the learning rate finder comes from.
- Wilson et al. (2017), The Marginal Value of Adaptive Gradient Methods in Machine Learning — the counter-argument: SGD can generalise better. Worth reading alongside Adam.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Tuning everything except the learning rate. It matters more than architecture choices people agonise over.
- Reading a plateau as convergence. It may be a saddle point, or a decayed rate, or a dead layer.
- Copying a learning rate from a paper with a different batch size. They scale together, and the number alone means nothing.
At a glance
FieldDeep Learning
Core ideastep downhill on the error surface
Key diallearning rate
Default optimiserAdam
Real obstaclesaddle points, not local minima
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Gradient descent vs. backpropagation — taking the step vs. computing which way is downhill.