Attention
The mechanism that lets an AI decide which other words matter when interpreting each word — the core idea behind transformers.
When not to use it
- Very long sequences on a tight budget. Standard attention costs grow with the square of length, so a 100k-token input isn't ten times a 10k one — it's about a hundred times. At some point the answer is retrieval, not a bigger window.
- As an explanation of model reasoning. Attention weights show where the model looked, which is tempting to read as why it answered. Research is genuinely divided on whether that inference holds.
- Small, local, structured problems. If a fixed neighbourhood is all that matters, a convolution or a plain feed-forward net is cheaper and often better.
Reach for something else instead
- Convolutions win when the signal is local and translation-invariant — most image tasks, plenty of audio.
- State-space models (Mamba and kin) scale linearly with sequence length and are competitive on long sequences, though not yet at frontier-model quality across the board.
- Sparse and linear attention variants trade a little quality for a lot of length. Useful when you genuinely need the context and can't afford the quadratic bill.
Read more on the blog
- Where the attention mechanism actually came fromAttention was a fix for a specific engineering failure in 2014, three years before the paper that made it famous. It was not designed as a theory of cognition, and the name was applied afterwards by analogy.
- How AI generates video: from noise to motionText-to-video went from a novelty to convincing minute-long clips with synchronised audio in about two years. The technology behind it extends image generation into time, and the hardest part is not making a frame look good but making a thousand frames hang together. Here is how it works.
- How a sentence becomes an answer: an LLM end to endMost explanations of large language models cover one piece, attention, or tokens, or sampling, in isolation. This follows a single sentence all the way through the machine, from the moment you hit enter to the words that come back, so the pieces finally connect.
- How LLM inference works: why it's bound by memory, not computeBuying a faster GPU often does not make an LLM generate text any faster, and the reason is one of the more counterintuitive facts in AI systems. Generating tokens is limited by memory bandwidth, not compute. Here is how inference actually works: the two phases, the KV cache that dominates it, and why long context costs what it does.
Doubling the prompt quadruples the attention cost.
Attention compares every token to every other token, so the score matrix has n² entries. That single fact is the whole cost story: the grey line is what people budget for — twice the tokens, twice the cost — and the green line is what they get. Everything here is n² against 2n, computed from the slider, with no model-specific assumptions. This is also why a longer prompt is not free even when it fits: you pay quadratically for context the model may not use, and the flat parts of the transformer — the feed-forward blocks — are the reason real models don't scale quite this badly at short lengths, and cannot save you at long ones.
The full account
Attention is older than the Transformer
The 2017 paper is called Attention Is All You Need, and the title has quietly convinced a generation that attention arrived with it. It did not. Bahdanau, Cho and Bengio introduced attention in 2014, inside a recurrent encoder-decoder, to fix a specific and unglamorous problem.
Machine translation at the time compressed an entire source sentence into one fixed-length vector, and the decoder generated from that. This works acceptably for short sentences and degrades badly for long ones, for the obvious reason: everything has to fit through the same narrow pipe regardless of how much there is. Bahdanau's fix was to let the decoder look back at all the encoder states and weight them — soft alignment, learned end to end, so the model decides for itself which source words matter for the word it is currently producing.
The Transformer's actual contribution was to notice that if attention can do the alignment, the recurrence around it is doing less work than assumed — and removing it makes the whole thing parallelisable, which is the real reason the architecture won. The title is a claim about what you can delete. It has been widely read as a claim about what was invented.
The debate with the titles
Attention weights are the most-looked-at numbers in deep learning. They are the right shape for a heat map, they are produced for free, and they appear to say which words the model cared about. Thousands of papers print them as evidence for claims about model behaviour.
In 2019, Jain and Wallace asked whether that inference holds, and titled the paper Attention is not Explanation. Their argument had two parts. First, attention weights correlate only weakly with gradient-based measures of feature importance — the two methods disagree about which tokens mattered. Second, and more damaging, they constructed adversarial attention distributions: alternative weights, very different from the learned ones, that yield the same prediction. If a completely different set of weights produces the identical output, the learned weights cannot be the explanation for that output.
Wiegreffe and Pinter replied the same year with Attention is not not Explanation, which is a title doing real work. Their objection was that the argument smuggles in an unstated definition. If explanation means the unique reason for the output, the adversarial construction refutes it — but that definition is too strong for any component of any model. If explanation means plausible information about what the model attended to, the test is too easy: they showed that constructing adversarial weights that survive proper training, rather than being fitted after the fact, is much harder than Jain and Wallace's procedure made it look. Freely optimising weights to preserve a prediction is not the same as showing an alternative model could have been trained to use them.
Serrano and Smith arrived from a third direction with erasure experiments — remove what attention says is important and see whether the prediction changes — and reached a middle answer: attention weights are partially informative and less reliable than their use in the literature implies.
| Position | Claim | Test |
|---|---|---|
| The common practice | Attention heat maps show what the model used | none |
| Jain & Wallace (2019) | Not explanation — adversarial weights give the same output | swap weights post-hoc, compare predictions |
| Wiegreffe & Pinter (2019) | Depends what you mean; the adversarial test is too permissive | require alternatives to survive training |
| Serrano & Smith (2019) | Partially informative, less than assumed | erase attended components, measure the change |
Where that leaves the heat map
The productive reading is that nobody won, and the disagreement located the actual problem: explanation was never defined. The field printed attention maps for years without stating what claim the map was supposed to support, and the argument only became tractable when two papers were forced to say out loud what they meant.
The practical position that survives all three papers is narrow and worth holding. An attention map tells you what the model attended to at that layer. It does not tell you why the output was what it was, because the value pathway, the residual stream, and every subsequent layer all intervene between the weight and the answer. Treating the map as a causal account is the specific move that all three papers, from three directions, found unsupported.
This matters more now than it did in 2019, because the same inference has been quietly transplanted. Reasoning traces are the new attention maps: legible, produced for free, apparently a window into the process, and load-bearing in exactly the places where being wrong is expensive. The literature that established you cannot read attention weights as explanation was published, cited, and then largely not applied to the next artefact with the same shape.
The cost that survives all of it
Whatever attention explains, it costs n². Every token attends to every token, and doubling the sequence quadruples the work — the figure above computes it: 1k to 128k tokens is a 16,384× increase.
The persistent misconception is that FlashAttention solved this. It did not. FlashAttention makes the same computation dramatically faster by moving less memory, and it is exact — bit-for-bit identical output. The FLOPs are untouched. What became linear is memory, not compute, and if your costs scale badly with length then the kernel has already been applied and the quadratic is still yours to deal with.
Further reading
- Bahdanau, Cho & Bengio (2015), Neural Machine Translation by Jointly Learning to Align and Translate — attention before transformers, and still the clearest motivation for it.
- Vaswani et al. (2017), Attention Is All You Need — the paper that dropped recurrence entirely.
- Jain & Wallace (2019), Attention is not Explanation — the counter-argument to reading attention weights as reasoning, and the reply, Attention is not not Explanation (Wiegreffe & Pinter, 2019), which is worth reading alongside it.
- Bahdanau, Cho & Bengio (2015), Neural Machine Translation by Jointly Learning to Align and Translate — attention, three years before the Transformer, invented to fix a bottleneck rather than to replace recurrence.
- Jain & Wallace (2019), Attention is not Explanation — attention weights correlate poorly with gradient-based importance, and adversarial weights produce identical predictions.
- Wiegreffe & Pinter (2019), Attention is not not Explanation — the titled rebuttal; the claim depends on what you meant by explanation, and the adversarial test is too easy.
- Serrano & Smith (2019), Is Attention Interpretable? — erasure experiments; attention weights only partly identify the components that matter.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading attention maps as interpretability. They're suggestive, not evidence.
- Assuming more context is free. Doubling the prompt roughly quadruples attention cost and can lower answer quality by burying the relevant part.
- Confusing attention with the transformer. Attention is one mechanism inside the architecture; the block also has feed-forward layers, residuals, and normalisation doing real work.
At a glance
Where this sits
9 concepts come first. Understanding it opens up 75 more.
Computed from the prerequisite graph, not assigned. How this works