Scaling Laws
The finding that model performance improves predictably with size, data and compute — the empirical result that justified spending billions, and it isn't a law.
When not to use it
- To predict capabilities. They predict loss. The map from loss to "can it do the job" is not part of the theory.
- Outside the observed range. They're empirical fits. Extrapolation is a bet, and it's a large one.
- On your fine-tuning run. These describe pretraining at scale. Your 5,000-example fine-tune is governed by other things entirely.
- As justification on their own. "Scaling will fix it" is a prediction about loss, and your problem probably isn't loss.
Reach for something else instead
- (Other ways to reason about what improves a model.)*
- Data quality work — often beats scale at fixed cost, and is less fashionable for that reason.
- Post-training — RLHF and instruction tuning changed usefulness far more than the loss curve suggests.
- Retrieval — adding knowledge without adding parameters.
- Better architectures — the thing scaling laws made everyone stop looking for.
Sources & further reading
- Kaplan et al. (2020), Scaling Laws for Neural Language Models — the paper that made compute a strategy.
- Hoffmann et al. (2022), Training Compute-Optimal Large Language Models — Chinchilla; the correction that made models smaller and better.
- Schaeffer, Miranda & Koyejo (2023), Are Emergent Abilities of Large Language Models a Mirage? — the argument that emergence is substantially a metric artefact.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Saying "scaling laws" as though they're laws. They're a fitted empirical regularity over an observed range.
- Confusing loss with capability. Smooth loss does not imply smooth usefulness, in either direction.
- Ignoring the irreducible floor. The curve asymptotes to the entropy of language and never crosses it.
- Quoting pre-Chinchilla folklore about parameters mattering most. That was corrected in 2022.
- Treating emergence as established. The measurement-artefact argument is serious and unresolved.
At a glance
FieldFoundations
What's predictableloss
What isn'tcapability
Chinchilla rulescale data and parameters together, ~20 tokens/param
Exponentssmall; needs orders of magnitude
Statusempirical regularity, not law
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Scaling laws vs. emergence — one says everything is smooth and predictable; the other says capabilities appear abruptly. They can't both be fully right, and the disagreement may be about the metric rather than the model.