Where the attention mechanism actually came from

The 2017 paper that made attention famous is called Attention Is All You Need. The title is a claim about what you can remove, not about what to add.

By 2017 attention had been in use for three years. What the paper proposed was deleting the recurrent network that had always accompanied it, and showing that the remainder still worked. The title says so plainly and it is almost universally read backwards.

The mechanism itself arrived in 2014, in a paper about machine translation, as a fix for a specific engineering failure. It was not proposed as a theory of how thinking works. It was proposed because sentences longer than about thirty words were being translated badly, and the reason was known.

The bottleneck

The state of the art in 2014 was the RNN encoder-decoder. An encoder network read the source sentence one word at a time, updating an internal state. When it reached the end, that state, a single fixed-length vector, was handed to a decoder, which generated the translation from it.

Everything the source sentence contained had to fit in that vector. A four-word sentence and a forty-word sentence got the same number of dimensions.

The consequence was measured and unambiguous: translation quality degraded sharply as sentences got longer. The encoder had to discard information to compress, and what it discarded was disproportionately from the beginning of the sentence, since each update overwrote a little more of what came before.

That is a clean engineering problem with a visible cause. It is also wasteful in the other direction, since short sentences got a representation sized for long ones.

The fix, stated plainly

Bahdanau, Cho and Bengio proposed something almost obvious once the problem is stated: stop discarding the intermediate states.

The encoder already produced a hidden state at every word. The original design threw all of them away except the last. The new design kept them all and let the decoder, at each output step, compute a weighted combination of them, with the weights depending on what it was currently trying to produce.

Their own description of the benefit is worth reading as engineering rather than philosophy: it frees the model from encoding a whole source sentence into a fixed-length vector, and lets it focus only on information relevant to generating the next target word.

That is a bottleneck being removed. There is no claim about cognition in it.

The model was called RNNSearch, because the decoder searches the encoder states. The mechanism acquired its familiar name because the authors likened the behaviour to the human notion of attention, and the analogy stuck harder than the description.

Two things the standard account gets wrong

It was not the first attention mechanism. Work published the same year applied a similar idea to learning alignments between different modalities, connecting image regions to actions in a control problem. Bahdanau and colleagues applied it to translation, where it mattered enormously, and the priority claim usually attached to them is not quite right.

And the name came after the mechanism. The design was arrived at by asking what to do about a fixed-length vector. The cognitive framing was applied to something already built, which is the normal direction for such things and the opposite of how it is usually taught. Nobody set out to give a network attention; someone set out to stop throwing away encoder states.

This matters beyond pedantry, because the cognitive framing carries implications the mechanism does not support. A weighted average over positions is a weighted average over positions. That it resembles something we call attention is an observation about the metaphor, not a property of the computation.

Why the alignment picture was so persuasive

One aspect of the 2014 paper did more for its reception than the numbers, and it is worth understanding.

The attention weights could be plotted as a matrix: source words along one axis, generated words along the other, brightness showing how much each source word contributed to each output word. The resulting pictures showed a bright diagonal where languages agree on word order, and clean off-diagonal excursions exactly where they disagree.

For anyone who had worked on statistical machine translation, this was immediately legible. Word alignment had been a central problem in that field for decades, with dedicated models built to estimate it. Here it fell out of a network trained only to translate, and nobody had asked for it.

That is a genuine result and it also created a durable expectation: that attention weights show what the model is using. They show what the weighted average weighted. Whether that constitutes an explanation has been argued about ever since.

The three years in between

Attention did not sit still between 2014 and 2017, and the intermediate work explains what the transformer actually changed.

Luong and colleagues, 2015, simplified the scoring. Bahdanau computed alignment scores with a small feed-forward network; Luong showed a plain dot product between encoder and decoder states worked comparably and was much cheaper. They also used the current decoder state rather than the previous one. Dot-product scoring is what the transformer inherited.

The remaining constraint was recurrence. Attention had removed the information bottleneck and the sequential dependency was untouched: each decoder step still needed the previous step's state, so the computation could not be parallelised across positions. Training time scaled with sentence length in a way no hardware improvement addressed.

That is the problem the 2017 paper solved, and it is why the title is about removal. Take out the recurrence, keep the attention, add positional encodings so word order survives, and the whole sequence can be processed at once. The transformer's contribution was parallelism, and attention was the component that made removing recurrence survivable.

The same move, four times

Once you see the shape of the 2014 fix, it recurs. Each of these identifies a hard constraint, removes it, and keeps whatever still works. None of them is a new idea about intelligence.

1997, long short-term memory. The constraint was vanishing gradients over long sequences, so error signals died before reaching early steps. The fix added gates that let information pass unchanged. What it kept: recurrence.

2014, attention. The constraint was the fixed-length context vector. The fix kept every encoder state and combined them by relevance. What it kept: recurrence, still.

2017, the transformer. The constraint was recurrence itself, which forced sequential computation and made training scale badly with length. The fix removed it and added positional encodings so word order survived. What it kept: attention.

2024 onward, state-space and linear-attention models. The constraint is quadratic attention cost, which makes long contexts expensive. The fixes trade exact all-pairs comparison for something cheaper. What they keep is under active negotiation, which is why this round is not settled.

The pattern is worth naming because it predicts where to look. Architectural progress in this field has come almost entirely from identifying the currently binding constraint and paying to remove it, and the price is always some capability that turned out to be less load-bearing than assumed.

It also explains why each step looks obvious afterwards and was not obvious before. The hard part was never the fix. It was establishing which constraint was actually binding, which requires the previous fix to have been in use long enough for its successor to become visible.

What this history is actually good for

Not trivia. Three things follow that are useful when reading current claims.

Mechanisms get named by analogy, and the analogy then does unearned work. Attention, memory, reasoning, hallucination, understanding. Each names something that resembles a human capacity and each was chosen for resemblance rather than derived from one. Reading the original motivation is usually deflationary and usually clarifying.

Architectures are shaped by constraints that may no longer bind. The fixed-length vector was a real limit in 2014. Quadratic attention cost is a real limit now, and the current work on state-space models and linear attention variants is the same kind of move: identify the binding constraint, remove it, keep what still works.

And the celebrated paper is often not the originating one. The 2017 paper is cited orders of magnitude more than the 2014 one, and it built on a mechanism it did not invent, to solve a problem the 2014 authors were not addressing. Both are real contributions. Only one is generally remembered.

What is unresolved

Whether attention weights explain anything. A substantial literature argues both sides. One position holds that alternative weight distributions can produce identical predictions, so the weights cannot be the explanation. The reply is that this proves attention is not the only possible explanation rather than that it is not one. The dispute is unsettled and it bears directly on every saliency visualisation anyone shows you.

Whether the resemblance to human attention is more than nominal. Some work argues the correspondence is closer than the deflationary reading suggests. Whether that reflects convergence on a good solution or a metaphor being read into the data is not established.

Whether quadratic cost is fundamental. Attention compares every position to every other, which is where the cost comes from and also where the capability comes from. Linear-attention variants and state-space models trade some of the second for the first, and whether anything recovers the full capability at lower cost is the open architectural question.

And whether removing recurrence lost something. Recurrent models have an unbounded implicit state; transformers have a bounded window. That was a good trade for the tasks of 2017. Whether it remains so for tasks requiring very long-range dependence is being actively revisited.

The counter-argument

Priority disputes are mostly uninteresting. The 2017 paper is celebrated because it produced the architecture everything now uses, and it is reasonable for the field to remember the paper that changed practice over the one that proposed a component. Insisting on the 2014 attribution can be pedantry dressed as correction.

The cognitive framing may have earned its keep. Calling it attention made it intelligible, memorable and teachable, and the mechanism spread faster because of the name. A purely technical description would have been more accurate and less useful, and framing is part of how ideas propagate.

And the deflationary reading can go too far. Saying it is "just a weighted average" is true and unilluminating, in the way that saying a brain is just electrochemistry is true. The interesting question is what the weighting learns to do, and dismissing the mechanism as simple sidesteps that.

The short version

Attention Is All You Need is a claim about what can be removed. By 2017 attention had been in use for three years, and the paper proposed deleting the recurrent network that had always accompanied it. The title says this plainly and is almost universally read backwards.

The mechanism arrived in 2014 as a fix for a measured engineering failure. The RNN encoder-decoder compressed an entire source sentence into a single fixed-length vector, so translation quality degraded sharply with sentence length, with information from the beginning of the sentence disproportionately lost. The fix was to stop discarding the encoder's intermediate states and let the decoder compute a weighted combination of all of them at each output step.

The model was called RNNSearch. The name "attention" was applied afterwards, by analogy to a human capacity, to a mechanism designed to remove a bottleneck. It was also not the first such mechanism: comparable work that year learned alignments between image regions and control actions.

The alignment visualisations did more for its reception than the accuracy numbers. Plotting the weights produced a bright diagonal where languages agree on word order and clean excursions where they do not, which was immediately legible to a field that had spent decades building dedicated word-alignment models. It fell out of a network trained only to translate.

Between 2014 and 2017, dot-product scoring replaced the small feed-forward scorer, and the remaining constraint was recurrence: each step still needed the previous one, so nothing parallelised. The transformer's contribution was parallelism, and attention was the component that made removing recurrence survivable.

Common questions

Who invented the attention mechanism? Bahdanau, Cho and Bengio introduced it for machine translation in a 2014 paper, "Neural Machine Translation by Jointly Learning to Align and Translate," which is the version that became influential. It was not the first such mechanism: comparable work published the same year applied the idea to learning alignments between image regions and actions in a control problem. The 2017 transformer paper did not invent attention and is frequently credited with doing so.

What problem did the attention mechanism solve? The information bottleneck in RNN encoder-decoder models. Those systems compressed an entire source sentence into a single fixed-length vector, so all sentences got the same representational budget regardless of length, and translation quality degraded sharply on longer inputs as the encoder discarded information to fit. Attention removed the bottleneck by keeping every encoder hidden state and letting the decoder combine them by relevance at each output step.

Why is it called attention? Because the authors likened the behaviour to the human notion of attention, after building it. The model in the original paper was called RNNSearch, describing what the decoder does: it searches the encoder states. The cognitive name was applied to an existing mechanism rather than the mechanism being derived from a theory of cognition, which is the reverse of how it is usually taught.

What does "Attention Is All You Need" actually claim? That the recurrent network can be removed. Attention had been standard since 2014, always paired with an RNN, and the 2017 contribution was showing the RNN was unnecessary: keep attention, add positional encodings to preserve word order, and the whole sequence can be processed in parallel rather than step by step. The real gain was parallelism during training, which recurrence had made impossible.

What is the difference between Bahdanau and Luong attention? Two things. Bahdanau computes alignment scores with a small feed-forward network, while Luong uses a plain dot product between encoder and decoder states, which is cheaper and works comparably. And Bahdanau uses the decoder's previous hidden state to compute the alignment, while Luong uses the current one. Dot-product scoring is what the transformer inherited.

What was the encoder-decoder bottleneck? The constraint that everything in a source sequence had to fit into one fixed-size vector passed from encoder to decoder. Each encoder step overwrote part of what came before, so early information was disproportionately lost, and performance fell as sequences lengthened. It was also wasteful in the other direction, since short inputs received a representation sized for long ones.

Do attention weights explain what a model is doing? Contested. They show which positions the weighted average weighted, which is a fact about the computation. Whether that constitutes an explanation is disputed, with one position arguing that alternative weight distributions can produce identical predictions so the weights cannot be the explanation, and the reply that this shows attention is not the only possible explanation rather than that it is not one. The dispute bears on every saliency visualisation.

Why did the alignment pictures matter so much? Because they made an old problem look solved by accident. Plotting attention weights as a source-by-target matrix produced a bright diagonal where two languages share word order and clean off-diagonal excursions exactly where they reorder. Word alignment had been a central problem in statistical machine translation for decades with dedicated models built to estimate it, and here it emerged from a network trained only to translate.