Vanishing Gradient
The signal dying on its way back through a deep network — the problem that kept deep learning impossible for twenty years.
When not to use it
- (It's a failure mode, not a technique. The equivalent is when to suspect it.)*
- When early layers barely move. Log gradient norms per layer; the answer will be visible.
- In RNNs over long sequences. This is the original case and it's what LSTMs were built for.
- Whenever you see sigmoid or tanh in hidden layers. Derivative caps at 0.25. It's arithmetic.
- In a deep network without residuals. There's no reason to build one in 2026.
Reach for something else instead
- (Fixes, not substitutes.)*
- Residual connections — the structural answer. Gradient gets an unattenuated path.
- ReLU-family activations — gradient of 1, no attenuation.
- He / Xavier initialisation — start with the variance preserved.
- Gradient clipping — for the exploding version. Cap the norm.
The diagnosis is one plot away.
Backpropagation multiplies one factor per layer, so the gradient reaching layer 1 is that factor raised to the depth. A sigmoid's derivative peaks at exactly 0.25 — that is its ceiling, not its average — so even in the best case the signal is quartered at every step down. The y-axis is logarithmic because nothing else can hold the range. Residual connections do not remove this; they add a path whose local derivative is 1, so the product has something to multiply by that isn't small. That is why the problem is managed rather than solved, and why long recurrent sequences still hit it.
The full account
The thesis nobody read
In June 1991, a student at the Technical University of Munich submitted his diploma thesis. It was titled Untersuchungen zu dynamischen neuronalen Netzen — Investigations into Dynamic Neural Networks — his advisor was Jürgen Schmidhuber, and it contained the first formal analysis of why deep and recurrent networks fail to learn long-range dependencies. Sepp Hochreiter had proved that the error signal, propagated backward through a network, shrinks exponentially with the number of layers or time steps it traverses. Not slowly. Exponentially.
The thesis was written in German. It was not widely circulated. The field largely did not read it, and kept building shallow networks.
Bengio, Simard and Frasconi derived the same result independently in English in 1994, three years later, and that is the paper most people cite. Hochreiter published the English version of his own analysis in 1998 — seven years after proving it. The single most important negative result in the history of neural networks sat in a language most of its audience couldn't read, while the field wondered why deep networks wouldn't train.
The arithmetic, which is brutal and simple
The figure above computes it exactly, and the number to hold onto is 0.25.
The sigmoid activation was the standard choice for years. Its derivative has a maximum value of 0.25, achieved at exactly one point; everywhere else it is smaller. Backpropagation multiplies these derivatives together, once per layer, by the chain rule. So the gradient reaching layer 1 of a deep sigmoid network is scaled by a product of numbers each at most a quarter.
At depth 32, in the best possible case where every single derivative sits at its maximum, that product is 0.25^15 — and the figure returns 2.2 × 10⁻¹⁹. That is the optimistic number. The real one is worse, because the derivative is only 0.25 at one point on the curve and the network does not arrange to sit there.
A gradient of 10⁻¹⁹ is not a small update. It is no update. The early layers of the network are frozen at their random initialisation, and the network is effectively a shallow model with a large amount of untrained noise attached to the front. This is why the first two decades of neural network research produced shallow networks: not stubbornness, arithmetic.
Toggle the figure to a residual connection and the same depth-32 network holds a gradient of 1.0.
Why the fixes work
Each of the interventions that unlocked deep learning attacks a different term in that product, which is why they compose rather than compete.
ReLU replaces the sigmoid. Its derivative is 1 for positive inputs — not 0.25, exactly 1 — so the chain-rule product stops shrinking on every active path. The cost is dead units on the negative side, which is a real problem and a much smaller one.
Careful initialisation — Glorot and Bengio in 2010 — sets the initial weight variance so activations and gradients keep roughly constant scale across layers. Their contribution was showing that the standard initialisation of the time was itself causing the saturation, which meant part of the problem was self-inflicted.
Batch normalisation keeps activations in the region where derivatives are large rather than out on the flat tails, which is where sigmoid derivatives approach zero. Note that this is a different claim from the one BatchNorm's own paper made, and it survives while that one didn't.
Residual connections — He et al., 2015 — do something categorically different, and it is the one that mattered most. Instead of improving the multiplication, they add a path that skips it: the gradient can flow through an identity connection whose derivative is exactly 1, all the way from the loss to the earliest layer, without passing through a single activation. You are not fighting the exponential decay; you are routing around it. The figure shows the result — depth 32, gradient 1.0, unchanged.
| Fix | Attacks | Gradient at depth 32 |
|---|---|---|
| Sigmoid (the default until ~2011) | — | 2.2 × 10⁻¹⁹, best case |
| ReLU | derivative magnitude — 1 instead of 0.25 | survives, with dead units |
| Glorot init | initial scale, so you don't start saturated | helps; doesn't remove the product |
| BatchNorm | keeps activations off the flat tails | helps; doesn't remove the product |
| Residual connection | the product itself — an identity path | 1.0 |
The irony at the centre
Hochreiter proved the problem in 1991. Six years later, with the same advisor, he published the architecture built specifically to defeat it: LSTM (Hochreiter & Schmidhuber, 1997), whose constant error carousel is a path along which the gradient neither decays nor explodes — an identity connection through time, eighteen years before ResNet applied the same idea through depth.
LSTM then went on to power speech recognition, translation and text prediction for two decades. So the person who demonstrated that recurrent networks could not learn long-term dependencies also built the recurrent network that could, and the field's response to his proof was to ignore it in German for three years and then rediscover it in English.
What's left of it
The vanishing gradient is not solved so much as routed around. Every modern architecture has residual connections, and they are not an optimisation — they are the reason depth is possible at all. Remove them from a transformer and it will not train. The exponential decay is still sitting there in the mathematics, exactly as Hochreiter described it, and every deep network you use is built to avoid ever multiplying through it.
The failure mode you will actually meet is the mirror image: exploding gradients, where the product runs the other way, and the fix is a clip. That one announces itself with NaNs. Vanishing gradients don't announce anything — the loss just plateaus, the early layers stay at their initialisation, and the network quietly underperforms while reporting no error at all.
Further reading
- Hochreiter (1991), Untersuchungen zu dynamischen neuronalen Netzen — the thesis that identified the problem, years before anyone could act on it.
- Glorot & Bengio (2010), Understanding the Difficulty of Training Deep Feedforward Neural Networks — Xavier initialisation, and a clear diagnosis.
- He et al. (2016), Deep Residual Learning for Image Recognition — residual connections; the structural fix that made real depth possible.
- Bengio, Simard & Frasconi (1994), Learning Long-Term Dependencies with Gradient Descent Is Difficult — the independent English derivation, three years after the thesis nobody could read.
- Hochreiter & Schmidhuber (1997), Long Short-Term Memory — the architecture built specifically to defeat the problem the first author had proved.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating it as historical. It's managed, not removed, and RNNs over long sequences still hit it.
- Not logging per-layer gradient norms. The diagnosis is one plot away.
- Confusing it with exploding gradients. Exploding crashes loudly; vanishing looks like mediocre training.
- Building a deep network without residuals and blaming the depth.
At a glance
Often compared with
Where this sits
9 concepts come first. Understanding it opens up 4 more.
Computed from the prerequisite graph, not assigned. How this works