Cross-Attention
Attention where the queries come from one sequence and the keys and values from another — the mechanism that lets a decoder read an encoder, and the one that makes multimodal models possible.
Reviewed July 16, 2026Stable
When not to use it
- In decoder-only architectures, where there is no separate sequence to attend to.
- Where concatenation is sufficient and simpler, which for short auxiliary context it usually is.
Reach for something else instead
- Concatenation into a single sequence, letting self-attention do the work.
- Adapter layers that project one modality into another's space without attention.
- Late fusion, combining separate model outputs rather than their internals.
Sources & further reading
- Vaswani et al. (2017), Attention Is All You Need — defines encoder-decoder attention alongside self-attention.
- Alayrac et al. (2022), Flamingo — gated cross-attention for vision-language models.
- Borgeaud et al. (2022), Improving Language Models by Retrieving from Trillions of Tokens — cross-attention over retrieved chunks rather than prompt concatenation.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Confusing it with self-attention in architecture diagrams; the giveaway is two inputs rather than one.
- Assuming the cost scales like self-attention. It is rectangular, and the keys and values are cacheable across decoding steps.
- Expecting decoder-only models to have it. They do not.
At a glance
FieldDeep Learning
Queries fromsequence A
Keys and values fromsequence B
Attention matrixrectangular
Used inencoder-decoder and multimodal
DifficultyTechnical
Where this sits
A destination. 13 concepts lead here, and nothing in the corpus depends on it.
8Levelsteps in
13Needs firstconcepts
0Opens upnothing further
1Areastays here
Computed from the prerequisite graph, not assigned. How this works