Self-Attention
The specific form of attention where every element of a sequence attends to every other element in the same sequence — the operation at the heart of the transformer.
Reviewed July 16, 2026Contested
When not to use it
- On very long sequences under a tight budget, where the quadratic cost dominates and retrieval or a sub-quadratic architecture fits better.
- As an explanation of a model's reasoning; attention patterns show where the model looked, not why it answered.
- For small, local, structured problems where a convolution or feed-forward layer is cheaper and sufficient.
Reach for something else instead
- Convolutions for local, translation-invariant structure such as many vision and audio tasks.
- State-space models (Mamba and kin) that scale linearly with sequence length on long inputs.
- Sparse and linear attention variants that trade some quality for far longer contexts.
Sources & further reading
- Vaswani et al. (2017), Attention Is All You Need — introduced the transformer and made self-attention the central operation.
- Dao et al. (2022), FlashAttention — exact self-attention made memory-efficient, not cheaper in FLOPs.
- Tay et al. (2022), Efficient Transformers: A Survey — the landscape of attempts to beat self-attention's quadratic cost.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Confusing self-attention with attention in general; self-attention is the case where a sequence attends to itself, and it is one mechanism inside the larger transformer block.
- Reading attention maps as faithful explanations of the model's reasoning.
- Forgetting that self-attention has no built-in sense of order, so position must be added separately.
At a glance
FieldDeep Learning
Core ideaevery element attends to every other in the same sequence
CostO(n²) in length
Variantsfull vs causal, multi-head
DifficultyIntermediate → Advanced