FlashAttention
An attention implementation that computes exactly the same answer far faster by moving less memory — and it does not make attention subquadratic.
Reviewed July 16, 2026Stable
When not to use it
- As a fix for quadratic cost. The FLOPs are unchanged; if length is your cost problem, you need a different architecture.
- As a reason to skip retrieval. Long context being possible is not long context being good or cheap — RULER and lost-in-the-middle both still apply.
- As something to implement yourself. It's default in every major framework, and hand-rolling GPU kernels is not your comparative advantage.
Reach for something else instead
- Approximate attention — sparse, linear, low-rank variants — actually reduces FLOPs, at a cost in exactness. The comparison people skip is that FlashAttention is exact and often faster anyway.
- State-space models attack the quadratic architecturally rather than in the kernel.
- Shorter context remains the cheapest optimisation available and the one nobody wants to hear.
Sources & further reading
- Dao et al. (2022), FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — the tiling, kernel fusion and online-softmax result; exact, not approximate.
- Dao (2023), FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — better work partitioning across the GPU.
- Vaswani et al. (2017), Attention Is All You Need — the operation being optimised, and the quadratic that survives the optimisation.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Believing it's an approximation. It's exact — bit-for-bit identical output — which is precisely why it was adopted everywhere without a quality debate.
- Believing it makes attention subquadratic. It doesn't. Memory becomes linear; compute stays n².
- Optimising FLOPs when memory traffic is the bottleneck. This is the mistake FlashAttention exists to correct, and it's still the default assumption in a lot of efficiency work.
At a glance
FieldDeep Learning
Outputexact, bit-for-bit identical
Fixesmemory traffic, not FLOPs
Memoryquadratic → linear in sequence length
DifficultyAdvanced