Batching
Processing many requests together to use hardware that's mostly idle — the largest cost lever in inference, and it costs you latency.
Reviewed July 13, 2026Stable
When not to use it
- When latency is the product. Someone waits for the batch. That's the trade and it's unavoidable.
- Static batching for generation. The longest sequence blocks the batch. Use continuous batching.
- At low traffic. You can't batch what hasn't arrived. This is why self-hosting rarely saves money.
- With long contexts and a large batch. KV cache memory is your real limit, and it grows with context.
Reach for something else instead
- Continuous batching — the version that works for generation.
- Offline batch inference — if answers can wait, this is a large discount.
- Speculative decoding — different attack on the same idle compute.
- Quantization — fetch fewer bytes rather than doing more with them.
Further reading
- Yu et al. (2022), Orca: A Distributed Serving System for Transformer-Based Generative Models — continuous batching at the iteration level.
- Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention — the KV cache was mostly waste; paging fixed it.
- Williams, Waterman & Patterson (2009), Roofline: An Insightful Visual Performance Model — arithmetic intensity; why any of this works.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating batching as free. It's free in compute and costs latency, always.
- Using static batching for generation and watching utilisation collapse.
- Assuming compute limits your batch size. KV cache memory does.
- Not taking the batch discount for work that could wait overnight.
At a glance
FieldTools & Ecosystem
Why it worksdecoding is memory-bound; fetched weights are used once
The tradethroughput for latency, always
The fix for generationcontinuous batching, at the iteration level
What limits batch sizeKV cache memory, not compute
The consequenceinference has enormous economies of scale
DifficultyIntermediate
Often compared with
Static vs. continuous batching — one waits for the whole batch to finish and wastes the GPU; the other keeps it full every step. The difference is multiples.
Where this sits
A destination. 7 concepts lead here, and nothing in the corpus depends on it.
4Levelsteps in
7Needs firstconcepts
0Opens upnothing further
1Areastays here
Computed from the prerequisite graph, not assigned. How this works