KV Cache
The memory that stops a model re-reading its own conversation every token — the reason generation is fast, and the reason serving is expensive.
When not to use it
- (You always want it. The question is what you give up to fit it.)*
- Cache quantization, when quality is critical. It's the cheapest concurrency win and it does cost something.
- Token eviction, when the discarded context might matter. Every eviction policy is a bet about the future.
- Huge contexts at high batch size. The cache scales with both. Something has to give and it's usually your margin.
Reach for something else instead
- Grouped-Query Attention — nearly free cache reduction; standard in current models.
- PagedAttention / vLLM-style serving — recovers the memory that fragmentation wasted.
- State-space models — constant-size state instead of a growing cache. Solves it architecturally, at a small quality cost.
- Prompt caching — reuse the cache for a fixed prefix across requests. Real money, underused.
Read more on the blog
- How LLM inference works: why it's bound by memory, not computeBuying a faster GPU often does not make an LLM generate text any faster, and the reason is one of the more counterintuitive facts in AI systems. Generating tokens is limited by memory bandwidth, not compute. Here is how inference actually works: the two phases, the KV cache that dominates it, and why long context costs what it does.
- How quantization shrinks AI models without breaking themA 70-billion-parameter model needs about 140 GB of memory at full precision. Your laptop has 16. Quantization is how the model fits anyway, by storing each weight in far fewer bits, and the surprising part is that you can throw away most of that precision and the model barely notices. Here is why, and where it finally breaks.
- Speculative decoding: faster LLM generation, same outputThere is a way to make a large language model generate text two to four times faster while producing output that is mathematically identical to the slow way. It sounds impossible, but it works, and it is now standard in production serving. The trick is to let a small model guess ahead and have the big model check the guesses in parallel.
- What is a context window? Why bigger isn't betterEvery model launch brags about a bigger context window, now measured in millions of tokens. The part the marketing leaves out is that models do not use long context well. A model's advertised window and the window it can actually reason over are very different numbers, and the gap explains a lot of real-world AI failures.
The cache scales with context times users. The weights don't.
2 × 32 layers × 8 KV heads × 128 dim × 2 bytes = 128 KiB / token — Llama-style 7B with GQA, fp16; stated so you can check it
Exact arithmetic for a stated architecture — a Llama-style 7B: 32 layers, 8 KV heads (grouped-query attention), head dimension 128, fp16. Keys plus values cost 2 × 32 × 8 × 128 × 2 bytes = 128 KiB per token, and that buys nothing shareable: every concurrent user carries their own. The model's weights are 14 GB once, however many users you serve. This is why context length is priced as a cost and not a feature, and why serving stacks obsess over evicting, paging and quantising the cache — at long context it is bigger than the model.
Further reading
- Shazeer (2019), Fast Transformer Decoding: One Write-Head is All You Need — Multi-Query Attention; shrinking the cache by sharing keys and values.
- Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention — vLLM; the OS-paging insight that changed serving economics.
- Pope et al. (2022), Efficiently Scaling Transformer Inference — the arithmetic of why decode is memory-bound.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating context length as a feature rather than a cost. Cache scales linearly with it, per user.
- Assuming a faster GPU speeds up generation. Decode is memory-bandwidth-bound; the compute is idle.
- Ignoring prompt caching with a large fixed system prompt. That's prefill you're paying for repeatedly.
- Benchmarking prefill and calling it throughput. They're separate constraints and a system can be good at one only.
At a glance
Often compared with
Where this sits
24 concepts come first. Understanding it opens up 1 more.
Computed from the prerequisite graph, not assigned. How this works