Transformer
The neural-network architecture behind almost every modern AI model — built around attention, which lets it weigh every word against every other, all at once.
When not to use it
- For small, local, structured problems. If only a fixed neighbourhood matters, a convolution is cheaper and often better — most image tasks never needed a transformer.
- On very long inputs with a tight budget. Attention cost grows with the square of length, so a 100k-token input isn't ten times a 10k one, it's roughly a hundred times.
- On tabular data. Gradient-boosted trees still win on spreadsheet-shaped problems, train in seconds, and can explain themselves.
Reach for something else instead
- Convolutional networks where the signal is local and position-invariant — still the efficient choice for many vision and audio tasks.
- State-space models (Mamba and kin) scale linearly with sequence length and are competitive on long sequences, though not yet across the board at frontier quality.
- Sparse or linear attention variants when you genuinely need the length and can't pay the quadratic bill — you trade a little quality for a lot of context.
Sources & further reading
- Vaswani et al. (2017), Attention Is All You Need — the original, and still readable.
- Alammar (2018), The Illustrated Transformer — the explanation most practitioners actually learned from.
- Gu & Dao (2023), Mamba: Linear-Time Sequence Modeling with Selective State Spaces — the most credible challenger to attention's quadratic cost.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming more context is free. Doubling the prompt roughly quadruples attention cost, and can lower answer quality by burying the relevant passage.
- Reading attention weights as an explanation of the model's reasoning. They show where it looked, which is not the same as why it answered — and the field is genuinely divided on this.
- Confusing the transformer with attention. Attention is one mechanism inside the block; the feed-forward layers, residuals, and normalisation are doing real work too.
At a glance
FieldDeep Learning
Introduced2017 — "Attention Is All You Need"
Core ideaSelf-attention
PowersGPT, Claude, Gemini, and most modern AI
Key limitationCost grows with the square of input length
DifficultyIntermediate → Advanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Transformer vs. CNN — attention over the whole sequence vs. local convolutional filters.