Prompt Caching
Reusing the computation for a prompt prefix you've sent before — the largest cost saving available to most applications, and most of them don't use it.
When not to use it
- With short prompts. Below the provider's minimum it doesn't cache at all.
- With anything variable at the top. A timestamp in your system prompt disables it for the whole application, silently.
- With sparse traffic. A five-minute TTL and a request every ten minutes never hits.
- Where a cache write costs a premium and you won't hit. You've made it more expensive.
Reach for something else instead
- Reordering your prompt — this is the fix, not an alternative. Stable first, variable last.
- A shorter prompt — if it won't cache, it should be small.
- Fine-tuning — bake the instructions into the weights instead of sending them.
- Batching — different lever, also worth pulling.
Sources & further reading
- Anthropic, Prompt caching documentation — the primary source for the constraints; read the provider's, not the coverage.
- Gim et al. (2023), Prompt Cache: Modular Attention Reuse for Low-Latency Inference — the research attempt at escaping the prefix constraint.
- Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention — vLLM; the KV-cache management this rests on.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Putting the user's question first. Nothing after it caches, ever.
- A timestamp or session ID near the top. Silently disables everything, and the bill never explains itself.
- Not checking cache-hit metrics. Every provider reports them; almost nobody looks.
- Keeping prompts short out of habit. If the prefix is cached, long is nearly free.
At a glance
FieldLanguage & LLMs
What it cachesthe KV state of a prompt prefix
Savingup to ~90% on the cached portion
The one rulestable content first, variable content last
Why prefix-onlyevery token's state depends on all prior tokens; that's causal attention
Typical TTL~5 minutes
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Prompt caching vs. shortening your prompt — one makes a long prefix nearly free, the other throws it away. If your prompt is stable, the first is strictly better.