LSTM
An RNN with gates that decide what to remember and what to forget — the fix that made sequence learning work, and it held for twenty years.
When not to use it
- For anything a transformer handles. Still sequential. The gates fixed the gradient, not the parallelism.
- On very long sequences. Better than a vanilla RNN, still not attention connecting positions directly.
- When a GRU would do. Fewer parameters, faster, usually equivalent. Check before assuming you need three gates.
- On large-scale language. That contest is over.
Reach for something else instead
- GRU — two gates, no cell state, usually as good.
- Transformer — parallel and direct. What replaced it.
- State-space models — recurrence with modern mathematics, linear cost.
- Temporal convolutions — parallel, fixed receptive field, often enough.
Sources & further reading
- Hochreiter & Schmidhuber (1997), Long Short-Term Memory — the paper; designed from a diagnosis rather than found by search.
- Gers, Schmidhuber & Cummins (1999), Learning to Forget: Continual Prediction with LSTM — the forget gate, and why discarding matters as much as retaining.
- Chung et al. (2014), Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling — GRU vs. LSTM; simpler is usually equivalent.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming LSTMs failed. They worked for twenty years and lost on parallelism, not quality.
- Reaching for an LSTM when a GRU is simpler and comparable.
- Missing that the
+in the cell update is the whole idea. Additive updates don't decay; multiplicative ones do. - Thinking gating is historical. It's in residuals, in GLU variants, in Mamba's selection.
At a glance
FieldDeep Learning
The fixan additive memory line, gated
Three gatesforget, input, output
The key mechanismconstant error carousel; gradient flows through a sum, not a product
Why it lostsequential, not parallelisable
What survivedgating, everywhere
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
LSTM vs. GRU — three gates and a separate cell state, versus two gates and none. The simpler one is usually just as good.