KV Cache
The memory that stops a model re-reading its own conversation every token — the reason generation is fast, and the reason serving is expensive.
When not to use it
- (You always want it. The question is what you give up to fit it.)*
- Cache quantization, when quality is critical. It's the cheapest concurrency win and it does cost something.
- Token eviction, when the discarded context might matter. Every eviction policy is a bet about the future.
- Huge contexts at high batch size. The cache scales with both. Something has to give and it's usually your margin.
Reach for something else instead
- Grouped-Query Attention — nearly free cache reduction; standard in current models.
- PagedAttention / vLLM-style serving — recovers the memory that fragmentation wasted.
- State-space models — constant-size state instead of a growing cache. Solves it architecturally, at a small quality cost.
- Prompt caching — reuse the cache for a fixed prefix across requests. Real money, underused.
Sources & further reading
- Shazeer (2019), Fast Transformer Decoding: One Write-Head is All You Need — Multi-Query Attention; shrinking the cache by sharing keys and values.
- Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention — vLLM; the OS-paging insight that changed serving economics.
- Pope et al. (2022), Efficiently Scaling Transformer Inference — the arithmetic of why decode is memory-bound.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating context length as a feature rather than a cost. Cache scales linearly with it, per user.
- Assuming a faster GPU speeds up generation. Decode is memory-bandwidth-bound; the compute is idle.
- Ignoring prompt caching with a large fixed system prompt. That's prefill you're paying for repeatedly.
- Benchmarking prefill and calling it throughput. They're separate constraints and a system can be good at one only.
At a glance
FieldLanguage & LLMs
What it cacheskeys and values, not queries
Scales withcontext length × batch
Prefillcompute-bound
Decodememory-bound
Standard mitigationGrouped-Query Attention
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Prefill vs. decode — parallel and compute-bound vs. sequential and memory-bound. Two different bottlenecks that get averaged into one misleading number.