Backpropagation
The algorithm that works out which weights caused a mistake and by how much — the reason neural networks can learn at all.
Reviewed July 10, 2026Stable
When not to use it
- Writing it yourself, outside of learning. Every framework does this correctly and faster than you will. Hand-rolled gradients are a source of subtle bugs, not insight.
- On non-differentiable objectives. If your loss has hard jumps or discrete decisions, there's no gradient to propagate. That's a different family of methods.
- As an explanation of how brains learn. It's an engineering algorithm, not a model of biology, and the resemblance is mostly metaphorical.
Reach for something else instead
- Evolutionary methods for non-differentiable or black-box objectives — far less efficient, but they don't need gradients.
- Gradient-free optimisation when the parameter count is small and the function is expensive or opaque.
- Forward-mode differentiation in the rare case where you have few inputs and many outputs. For neural nets it's the wrong direction and that's why nobody uses it.
Read more on the blog
- Who invented deep learning, and why it took so longBackpropagation was invented at least four times before it stuck. The ideas behind deep learning were mostly in place by 1990. What was missing was not insight.
- How neural networks work: the idea under all of itUnder every transformer, every image generator, every language model, sits one idea: the neural network. Here's how it actually works, neurons, weights, layers, and the simple trick by which it learns from its own mistakes, explained so it finally makes sense.
- What is deep learning? The complete guideAlmost every AI system that impresses today, from chatbots to image generators to voice assistants, runs on deep learning. Its core idea is a single powerful shift: instead of humans hand-crafting the features a model uses, deep networks learn their own layered representations of data, from simple edges to whole objects. This guide explains what deep learning is, how it works, its architectures, and its limits.
Backpropagation: the error flows backward, adjusting every weight.
A forward pass produces a prediction; comparing it to the truth gives an error. Backpropagation sends that error backward through the network, using the chain rule to compute how much each weight contributed — then each weight is nudged to reduce the error. Repeat over many examples and the network learns. The green arrow is the forward pass; the amber is the error propagating back.
Further reading
- Rumelhart, Hinton & Williams (1986), Learning representations by back-propagating errors — the paper that made neural networks trainable.
- He et al. (2016), Deep Residual Learning for Image Recognition — residual connections, and the clearest practical answer to vanishing gradients.
- Baydin et al. (2015), Automatic Differentiation in Machine Learning: a Survey — what your framework is actually doing.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Forgetting to zero gradients between steps, so they accumulate. Training still runs. It just isn't doing what you think.
- Blaming the model for a vanishing gradient. Deep stacks without residuals or normalisation will starve their early layers no matter how good the architecture is elsewhere.
- Assuming a NaN loss means bad data. It's often exploding gradients, and gradient clipping fixes it in one line.
At a glance
FieldDeep Learning
Core ideaassign blame backwards
Mechanismchain rule on a graph
Cost~2× the forward pass, plus stored activations
DifficultyIntermediate
Often compared with
Backpropagation vs. gradient descent — working out which direction to move vs. actually taking the step.
Where this sits
7 concepts come first. Understanding it opens up 6 more.
6Levelsteps in
7Needs firstconcepts
6Opens up2% of 310
1Areastays here
Computed from the prerequisite graph, not assigned. How this works