Perplexity
How surprised a model is by text — the number that drives all of pretraining, and correlates poorly with whether the model is any good.
Reviewed July 11, 2026Stable
When not to use it
- To compare models with different tokenizers. Different units. The comparison is meaningless and it's made constantly.
- To predict usefulness. Post-training improves usefulness and often worsens perplexity. They can move in opposite directions.
- On public benchmark text. Contamination lowers it dramatically for the wrong reason.
- On instruction-tuned models, as a quality measure. They were optimised away from it deliberately.
Reach for something else instead
- Task benchmarks — measure what you want, with all the contamination caveats.
- Human evaluation — expensive, noisy, and closer to the actual question.
- Your own eval set — thirty real examples; still the most useful thing available.
- Perplexity on your domain text — the one legitimate use: detecting whether the model knows your area.
Further reading
- Jelinek et al. (1977), Perplexity — a measure of the difficulty of speech recognition tasks — where the measure comes from.
- Shannon (1951), Prediction and Entropy of Printed English — the floor; how predictable language actually is.
- Ouyang et al. (2022), Training language models to follow instructions with human feedback — the source of the term "alignment tax". Note the paper largely answers it: mixing pretraining gradients back in (PPO-ptx) removes most of the regression.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Comparing across tokenizers. The single most common error with this metric.
- Comparing across context lengths. More context lowers perplexity for free.
- Assuming lower perplexity means a better assistant. RLHF makes it worse and the model better.
- Reporting it on public data without considering contamination.
- Treating it as a metric rather than the training objective. It isn't measuring the model from outside — it's what the model was built to minimise.
At a glance
FieldLanguage & LLMs
Definitionexp(cross-entropy), the training objective in readable units
Comparable across tokenizersno
Floorthe entropy of language
Correlates with usefulnessweakly, sometimes inversely
Legitimate usedetecting domain mismatch
DifficultyIntermediate
Often compared with
Perplexity vs. benchmarks — one is cheap, dense and misaligned with what you want; the other is expensive, gameable and closer to it. The field trains on the first and markets on the second.
Where this sits
24 concepts come first. Understanding it opens up 1 more.
11Levelsteps in
24Needs firstconcepts
1Opens up0% of 310
2Areascrosses fields
Computed from the prerequisite graph, not assigned. How this works