Scaling Laws
The finding that model performance improves predictably with size, data and compute — the empirical result that justified spending billions, and it isn't a law.
When not to use it
- To predict capabilities. They predict loss. The map from loss to "can it do the job" is not part of the theory.
- Outside the observed range. They're empirical fits. Extrapolation is a bet, and it's a large one.
- On your fine-tuning run. These describe pretraining at scale. Your 5,000-example fine-tune is governed by other things entirely.
- As justification on their own. "Scaling will fix it" is a prediction about loss, and your problem probably isn't loss.
Reach for something else instead
- (Other ways to reason about what improves a model.)*
- Data quality work — often beats scale at fixed cost, and is less fashionable for that reason.
- Post-training — RLHF and instruction tuning changed usefulness far more than the loss curve suggests.
- Retrieval — adding knowledge without adding parameters.
- Better architectures — the thing scaling laws made everyone stop looking for.
Read more on the blog
- AI scaling laws: why bigger wins, and whether it's endingOne empirical discovery explains most of the last six years of AI and the hundreds of billions spent on it: model performance improves in a smooth, predictable way as you add size, data, and compute. Here is what scaling laws are, why they reshaped the field, the Chinchilla correction, and the live 2026 question of whether pure scaling is running out.
- Superintelligence: the empirical record is zeroSixty years after the intelligence explosion was described, no system has demonstrated sustained open-ended self-improvement. The public forecasts come from five people with the same financial interest.
- How AI models are trained: from raw text to a system that helpsA language model isn't programmed, it's grown, in stages, from a firehose of text into a system that answers helpfully. The full modern pipeline: pretraining, supervised fine-tuning, preference alignment, and the reasoning training that defines 2026, what each stage does, why none can be skipped, and how the recipe changed.
- What is synthetic data? Training AI on AI-made dataFaced with running out of human text to train on, AI labs increasingly train models on data the models generate themselves. This is synthetic data, and it comes with a famous warning called model collapse. The resolution of that tension, that verification and curation are what separate collapse from improvement, is the whole story and one of the most important ideas in how modern AI is built.
The curve asymptotes to a floor, and never crosses it.
L(C) = 1.70 + 8.4 · C−0.09 — illustrative constants, the form is the one these are fitted with
A power law with an irreducible term — the form these are actually fitted with — plotted from parameters stated in the page. On log-log axes across the observed range it is a convincing straight line, which is where the word law comes from. Extend the axis and the line bends, because it was never a line: it is approaching a floor set by the entropy of the data itself. No quantity of compute buys prediction of something inherently unpredictable, and the straightness that makes the law look inevitable is an artefact of only ever having plotted the part that hasn't bent yet.
The full account
Two numbers, and a two-year argument about why they differ
Kaplan et al. (2020) and Hoffmann et al. (2022) asked an identical question — given a compute budget, how large should the model be? — and returned answers that are not close:
Kaplan: N_optimal ∝ C^0.73 · Chinchilla: N_optimal ∝ C^0.50
That gap is not academic. At Kaplan's exponent, most new compute should buy parameters; at Chinchilla's, parameters and data scale together. The industry followed Kaplan for two years and built models that were, by Chinchilla's account, badly undertrained for their size — Gopher's 280B parameters should have been roughly 70B trained on four times the data. Chinchilla demonstrated this by training a 70B model that beat a 280B one. Billions of dollars of compute were allocated on the wrong exponent.
Hoffmann's paper offered an explanation for the discrepancy: Kaplan used a cosine learning-rate schedule whose cycle length didn't match each run's training horizon, so his shorter runs were evaluated mid-decay and looked artificially bad, biasing the fit toward parameters. It is a plausible mechanism, it was widely repeated, and — this is the part worth the page — it appears to be wrong.
The correction to the correction
In 2024 two independent groups set out to explain the gap properly.
Porian and colleagues reproduced Kaplan's study across more than 900 training runs on two datasets and isolated three causes, in decreasing order of significance. First, FLOP counting: Kaplan measured non-embedding compute, excluding the last layer, which at his model scale is a large fraction of the total — the accounting choice alone shifts the exponent. Second, warmup duration: Kaplan's warmup was too long for his smaller models, handicapping exactly the runs that anchor the low end of the fit. Third, scale-dependent optimizer tuning: batch size and AdamW's β₂ need to be re-tuned as scale changes, and holding them fixed biases the result.
Correct those three and Kaplan's setup reproduces Chinchilla's exponent. And their paper states the negative result directly: counter to a hypothesis of Hoffmann et al., careful learning rate decay is not essential for the validity of their scaling law. The explanation the field had accepted for two years was not the mechanism.
Pearce and Song reached a compatible conclusion independently and concurrently, attributing most of the gap to Kaplan counting non-embedding rather than total parameters, combined with the small scale of his models — under 1B, where the embedding fraction is largest. Two groups, different methods, same answer, neither of them the received one.
And Chinchilla has its own problem
Besiroglu and colleagues went at it from the other side. Hoffmann's paper used three estimation methods; the third, a parametric fit, gave results slightly off from the other two, which nobody had explained. So they extracted the raw (N, D, L) data points from Hoffmann's own Figure 4 and re-ran the fit — and found the reported estimates for method 3 don't fit the data the paper itself published, with implausibly tight confidence intervals.
So the position in 2026 is: the paper that corrected the field had an error in one of its three estimators, and the explanation that paper gave for why its predecessor was wrong has itself been refuted. The conclusion — scale data and parameters together — survives all of this, corroborated by independent reproduction. Everything told about why did not.
| Kaplan (2020) | Chinchilla (2022) | What 2024 found | |
|---|---|---|---|
| N ∝ C^ | 0.73 | 0.50 | 0.50 — Kaplan's setup reproduces it once corrected |
| Cause of the gap | — | "Kaplan's LR schedule" | FLOP counting, warmup, optimizer tuning. Not LR decay. |
| Model scale | up to ~1B | up to 16B | Kaplan's small scale magnified the embedding-count error |
| Its own errors | non-embedding FLOP count | method-3 fit doesn't match its own Figure 4 | both papers have problems; the conclusion survives |
What this should do to your priors
The useful lesson is not about exponents. It is that the most consequential empirical result in modern AI was mis-specified by an accounting choice, corrected by a paper with its own estimation error, and the field's explanation for the correction was wrong for two years — during which the explanation was repeated in surveys, lectures and this page.
That is not a scandal. Nobody behaved badly. It is what happens when a result is expensive to reproduce: Kaplan's study cost enough that nobody re-ran it for four years, and when two groups finally did, they found the answer in the FLOP accounting rather than anywhere anyone had looked. Scaling laws are the load-bearing empirical claim underneath the industry's capital allocation, and the number of independent replications you can count on one hand.
The figure above is deliberately built from the arithmetic rather than from anyone's reported coefficients, for exactly this reason. Extrapolate it and watch the curve bend toward its floor — the irreducible loss term Kaplan's fit omitted entirely, and which is a large part of why his exponent and Chinchilla's could not agree.
Further reading
- Kaplan et al. (2020), Scaling Laws for Neural Language Models — the paper that made compute a strategy.
- Hoffmann et al. (2022), Training Compute-Optimal Large Language Models — Chinchilla; the correction that made models smaller and better.
- Schaeffer, Miranda & Koyejo (2023), Are Emergent Abilities of Large Language Models a Mirage? — the argument that emergence is substantially a metric artefact.
- Porian, Wortsman, Jitsev, Schmidt & Carmon (2024), Resolving Discrepancies in Compute-Optimal Scaling of Language Models — over 900 runs; the cause was FLOP counting, warmup and optimizer tuning, and — counter to Hoffmann's own hypothesis — not learning-rate decay.
- Pearce & Song (2024), Reconciling Kaplan and Chinchilla Scaling Laws — concurrent and independent; attributes most of the gap to counting non-embedding rather than total parameters at small scale.
- Besiroglu et al. (2024), Chinchilla Scaling: A Replication Attempt — re-extracts Hoffmann's own Figure 4 data and finds their third estimator doesn't fit it.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Saying "scaling laws" as though they're laws. They're a fitted empirical regularity over an observed range.
- Confusing loss with capability. Smooth loss does not imply smooth usefulness, in either direction.
- Ignoring the irreducible floor. The curve asymptotes to the entropy of language and never crosses it.
- Quoting pre-Chinchilla folklore about parameters mattering most. That was corrected in 2022.
- Treating emergence as established. The measurement-artefact argument is serious and unresolved.
At a glance
Often compared with
Where this sits
7 concepts come first. Understanding it opens up 3 more.
Computed from the prerequisite graph, not assigned. How this works