Backpropagation
The algorithm that works out which weights caused a mistake and by how much — the reason neural networks can learn at all.
When not to use it
- Writing it yourself, outside of learning. Every framework does this correctly and faster than you will. Hand-rolled gradients are a source of subtle bugs, not insight.
- On non-differentiable objectives. If your loss has hard jumps or discrete decisions, there's no gradient to propagate. That's a different family of methods.
- As an explanation of how brains learn. It's an engineering algorithm, not a model of biology, and the resemblance is mostly metaphorical.
Reach for something else instead
- Evolutionary methods for non-differentiable or black-box objectives — far less efficient, but they don't need gradients.
- Gradient-free optimisation when the parameter count is small and the function is expensive or opaque.
- Forward-mode differentiation in the rare case where you have few inputs and many outputs. For neural nets it's the wrong direction and that's why nobody uses it.
Sources & further reading
- Rumelhart, Hinton & Williams (1986), Learning representations by back-propagating errors — the paper that made neural networks trainable.
- He et al. (2015), Deep Residual Learning for Image Recognition — residual connections, and the clearest practical answer to vanishing gradients.
- Baydin et al. (2015), Automatic Differentiation in Machine Learning: a Survey — what your framework is actually doing.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Forgetting to zero gradients between steps, so they accumulate. Training still runs. It just isn't doing what you think.
- Blaming the model for a vanishing gradient. Deep stacks without residuals or normalisation will starve their early layers no matter how good the architecture is elsewhere.
- Assuming a NaN loss means bad data. It's often exploding gradients, and gradient clipping fixes it in one line.
At a glance
FieldDeep Learning
Core ideaassign blame backwards
Mechanismchain rule on a graph
Cost~2× the forward pass, plus stored activations
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Backpropagation vs. gradient descent — working out which direction to move vs. actually taking the step.