LSTM
An RNN with gates that decide what to remember and what to forget — the fix that made sequence learning work, and it held for twenty years.
When not to use it
- For anything a transformer handles. Still sequential. The gates fixed the gradient, not the parallelism.
- On very long sequences. Better than a vanilla RNN, still not attention connecting positions directly.
- When a GRU would do. Fewer parameters, faster, usually equivalent. Check before assuming you need three gates.
- On large-scale language. That contest is over.
Reach for something else instead
- GRU — two gates, no cell state, usually as good.
- Transformer — parallel and direct. What replaced it.
- State-space models — recurrence with modern mathematics, linear cost.
- Temporal convolutions — parallel, fixed receptive field, often enough.
Read more on the blog
- How transformers work: the architecture that ate AIOne 2017 paper replaced the entire previous approach to sequence modeling and made modern AI possible. Here's what a transformer actually is, why the attention mechanism was such a breakthrough, why parallelism is the real secret, and where, in 2026, the architecture is finally being challenged.
- What is natural language processing (NLP)?Every search, translation, voice assistant, and chatbot runs on natural language processing, the field of getting computers to work with human language. Its defining modern story is a quiet revolution: the dozens of separate, task-specific methods that made up NLP for decades collapsed into a single general approach, the large language model.
An LSTM adds gates that decide what to keep, forget, and output.
The LSTM's answer to the vanishing-gradient problem is a protected cell state — a memory conveyor running straight through — regulated by three gates. The forget gate drops irrelevant memory, the input gate writes new information, and the output gate decides what to emit. Because the cell state flows with minimal interference, gradients survive over long sequences where a plain RNN's would vanish.
Further reading
- Hochreiter & Schmidhuber (1997), Long Short-Term Memory — the paper; designed from a diagnosis rather than found by search.
- Gers, Schmidhuber & Cummins (1999), Learning to Forget: Continual Prediction with LSTM — the forget gate, and why discarding matters as much as retaining.
- Chung et al. (2014), Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling — GRU vs. LSTM; simpler is usually equivalent.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming LSTMs failed. They worked for twenty years and lost on parallelism, not quality.
- Reaching for an LSTM when a GRU is simpler and comparable.
- Missing that the
+in the cell update is the whole idea. Additive updates don't decay; multiplicative ones do. - Thinking gating is historical. It's in residuals, in GLU variants, in Mamba's selection.
At a glance
Often compared with
Where this sits
11 concepts come first. Understanding it opens up 1 more.
Computed from the prerequisite graph, not assigned. How this works