Batching
Processing many requests together to use hardware that's mostly idle — the largest cost lever in inference, and it costs you latency.
When not to use it
- When latency is the product. Someone waits for the batch. That's the trade and it's unavoidable.
- Static batching for generation. The longest sequence blocks the batch. Use continuous batching.
- At low traffic. You can't batch what hasn't arrived. This is why self-hosting rarely saves money.
- With long contexts and a large batch. KV cache memory is your real limit, and it grows with context.
Reach for something else instead
- Continuous batching — the version that works for generation.
- Offline batch inference — if answers can wait, this is a large discount.
- Speculative decoding — different attack on the same idle compute.
- Quantization — fetch fewer bytes rather than doing more with them.
Sources & further reading
- Yu et al. (2022), Orca: A Distributed Serving System for Transformer-Based Generative Models — continuous batching at the iteration level.
- Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention — the KV cache was mostly waste; paging fixed it.
- Williams, Waterman & Patterson (2009), Roofline: An Insightful Visual Performance Model — arithmetic intensity; why any of this works.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating batching as free. It's free in compute and costs latency, always.
- Using static batching for generation and watching utilisation collapse.
- Assuming compute limits your batch size. KV cache memory does.
- Not taking the batch discount for work that could wait overnight.
At a glance
FieldTools & Ecosystem
Why it worksdecoding is memory-bound; fetched weights are used once
The tradethroughput for latency, always
The fix for generationcontinuous batching, at the iteration level
What limits batch sizeKV cache memory, not compute
The consequenceinference has enormous economies of scale
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Static vs. continuous batching — one waits for the whole batch to finish and wastes the GPU; the other keeps it full every step. The difference is multiples.