Training vs Inference
Building the model versus using it — two completely different activities with different costs, hardware, and constraints.
When not to use it
- As a reason to train your own model. Most teams should never train; the fine-tune-or-prompt decision is the real one.
- Ignoring inference cost during model selection. The demo runs once; production runs forever, and the arithmetic changes the answer.
- Assuming training hardware requirements tell you deployment requirements. Inference fits in far less, which is often the whole plan.
Reach for something else instead
- Fine-tuning — training, but small enough to be practical.
- Prompting — no training at all, and it solves more than people expect.
- A hosted API — someone else's training, someone else's inference optimisation, and you do the arithmetic on volume.
Sources & further reading
- Kaplan et al. (2020), Scaling Laws for Neural Language Models — the training-compute side, and the framing that dominated for years.
- Pope et al. (2022), Efficiently Scaling Transformer Inference — what actually costs money at serving time.
- Snell et al. (2024), Scaling LLM Test-Time Compute Optimally — the argument that inference-time compute can substitute for training-time compute.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Budgeting for training and discovering inference is the real bill.
- Optimising the model for training speed when latency is what users feel.
- Forgetting the KV cache grows with context length, so long conversations get progressively more expensive to serve.
At a glance
FieldFoundations
Trainingbuild once, memory-hungry, expensive
Inferencerun constantly, latency-bound, where the money goes
Gaptraining needs several times the memory
DifficultyBeginner → Intermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Training vs. inference — writing the book once vs. every reader reading it.