Training vs Inference
Building the model versus using it — two completely different activities with different costs, hardware, and constraints.
Reviewed July 11, 2026Stable
When not to use it
- As a reason to train your own model. Most teams should never train; the fine-tune-or-prompt decision is the real one.
- Ignoring inference cost during model selection. The demo runs once; production runs forever, and the arithmetic changes the answer.
- Assuming training hardware requirements tell you deployment requirements. Inference fits in far less, which is often the whole plan.
Reach for something else instead
- Fine-tuning — training, but small enough to be practical.
- Prompting — no training at all, and it solves more than people expect.
- A hosted API — someone else's training, someone else's inference optimisation, and you do the arithmetic on volume.
Read more on the blog
- How to read a model releaseEvery few weeks a lab announces a new frontier model and every headline says the same thing. Here's how to work out what actually changed, what the benchmark numbers mean, and which parts of the announcement are marketing.
- How LLM inference works: why it's bound by memory, not computeBuying a faster GPU often does not make an LLM generate text any faster, and the reason is one of the more counterintuitive facts in AI systems. Generating tokens is limited by memory bandwidth, not compute. Here is how inference actually works: the two phases, the KV cache that dominates it, and why long context costs what it does.
- How quantization shrinks AI models without breaking themA 70-billion-parameter model needs about 140 GB of memory at full precision. Your laptop has 16. Quantization is how the model fits anyway, by storing each weight in far fewer bits, and the surprising part is that you can throw away most of that precision and the model barely notices. Here is why, and where it finally breaks.
- Speculative decoding: faster LLM generation, same outputThere is a way to make a large language model generate text two to four times faster while producing output that is mathematically identical to the slow way. It sounds impossible, but it works, and it is now standard in production serving. The trick is to let a small model guess ahead and have the big model check the guesses in parallel.
Further reading
- Kaplan et al. (2020), Scaling Laws for Neural Language Models — the training-compute side, and the framing that dominated for years.
- Pope et al. (2022), Efficiently Scaling Transformer Inference — what actually costs money at serving time.
- Snell et al. (2024), Scaling LLM Test-Time Compute Optimally — the argument that inference-time compute can substitute for training-time compute.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Budgeting for training and discovering inference is the real bill.
- Optimising the model for training speed when latency is what users feel.
- Forgetting the KV cache grows with context length, so long conversations get progressively more expensive to serve.
At a glance
FieldFoundations
Trainingbuild once, memory-hungry, expensive
Inferencerun constantly, latency-bound, where the money goes
Gaptraining needs several times the memory
DifficultyBeginner → Intermediate
Often compared with
Training vs. inference — writing the book once vs. every reader reading it.
Where this sits
3 concepts come first. Understanding it opens up 7 more.
2Levelsteps in
3Needs firstconcepts
7Opens up2% of 310
4Areascrosses fields
Learn these firstMachine Learning
Computed from the prerequisite graph, not assigned. How this works