Model Serving
Getting a trained model to answer requests reliably — where the model is the easy part and the queue is the hard one.
When not to use it
- (Serving decisions that are usually wrong.)*
- Real-time, when batch would do. A lot of "real-time" requirements dissolve under questioning, and batch is enormously cheaper.
- Self-hosting for efficiency. vLLM and TGI are excellent. Self-host for residency, volume or an unserved model — not for speed.
- Optimising total latency for a streaming UI. TTFT is what users feel; TPOT above reading speed is invisible.
- Naive batching. One long generation blocks the batch. Continuous batching is multiples better.
Reach for something else instead
- Managed inference APIs — someone else's serving problem, and they're good at it.
- Batch/offline + cache — if answers can wait, this is the cheapest thing available.
- vLLM / TGI — if you must self-host, don't write your own.
- A smaller model — the serving problem you don't have.
Sources & further reading
- Yu et al. (2022), Orca: A Distributed Serving System for Transformer-Based Generative Models — continuous batching; the biggest single throughput win.
- Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention — vLLM; the KV cache was mostly waste.
- Crankshaw et al. (2017), Clipper: A Low-Latency Online Prediction Serving System — the general problem, pre-LLM, and it's the same problem.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating the model as the hard part. It's a function call. The queue is the problem.
- Measuring total latency instead of TTFT. Users feel first token, not completion.
- Assuming batching is free. It's free in compute and costs latency — that's the trade.
- Self-hosting to save money at low volume. You won't.
At a glance
FieldTools & Ecosystem
The tradelatency vs. throughput, always
The LLM metricsTTFT (what users feel) and TPOT (invisible above reading speed)
The key factdecoding is memory-bandwidth-bound; the GPU is idle
The biggest wincontinuous batching; multiples, not percentages
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Latency vs. throughput — batching buys one with the other, and every serving decision is a point on that curve. Know which one you're selling.