State Space Model
Recurrence rebuilt with control theory — constant memory, linear cost, and the most credible challenger the transformer has.
When not to use it
- When exact recall from context matters. Fixed state means compression, and compression is lossy. This is where transformers win.
- When ecosystem maturity matters. Fewer kernels, fewer tools, fewer people who have debugged it.
- As a settled replacement. The quality gap is small and it hasn't closed.
- On short sequences. The linear-cost advantage needs length to pay for itself.
Reach for something else instead
- Transformer — unbounded state, exact recall, quadratic cost, the entire ecosystem.
- Hybrid (SSM + a few attention layers) — currently the best of it, and what most serious attempts converge on.
- Linear attention — a different route to linear cost, with its own quality trade.
- Sliding-window attention — cap the window, get linear cost, lose the far past explicitly.
Sources & further reading
- Gu, Goel & Ré (2021), Efficiently Modeling Long Sequences with Structured State Spaces — S4; where HiPPO initialisation makes long-range memory work.
- Gu & Dao (2023), Mamba: Linear-Time Sequence Modeling with Selective State Spaces — selectivity, and the hardware-aware scan that made it practical.
- Jelassi et al. (2024), Repeat After Me: Transformers are Better than State Space Models at Copying — the recall gap, characterised precisely.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading "linear scaling" as strictly better. Fixed state means forgetting, and what it forgets may matter.
- Expecting a pure SSM to match a transformer on in-context recall. It structurally can't; that's the trade.
- Assuming linear cost wins automatically. Ecosystem advantage is real, and slightly-better doesn't displace entrenched.
- Treating this as settled either way. It's the first live architectural contest in eight years.
At a glance
FieldDeep Learning
Origincontrol theory, 1960s
Key theoryHiPPO initialisation, so memory provably survives
Trainingparallel via scan
Inferencerecurrent, constant state, no KV cache
The tradefixed state means compression means forgetting
Loses atexact recall from context
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
State space model vs. transformer — bounded state that must forget, versus unbounded state that keeps everything at quadratic cost. The best current systems mix them.