Perplexity
How surprised a model is by text — the number that drives all of pretraining, and correlates poorly with whether the model is any good.
When not to use it
- To compare models with different tokenizers. Different units. The comparison is meaningless and it's made constantly.
- To predict usefulness. Post-training improves usefulness and often worsens perplexity. They can move in opposite directions.
- On public benchmark text. Contamination lowers it dramatically for the wrong reason.
- On instruction-tuned models, as a quality measure. They were optimised away from it deliberately.
Reach for something else instead
- Task benchmarks — measure what you want, with all the contamination caveats.
- Human evaluation — expensive, noisy, and closer to the actual question.
- Your own eval set — thirty real examples; still the most useful thing available.
- Perplexity on your domain text — the one legitimate use: detecting whether the model knows your area.
Sources & further reading
- Jelinek et al. (1977), Perplexity — a measure of the difficulty of speech recognition tasks — where the measure comes from.
- Shannon (1951), Prediction and Entropy of Printed English — the floor; how predictable language actually is.
- Ouyang et al. (2022), Training language models to follow instructions with human feedback — the alignment tax; usefulness up, perplexity worse.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Comparing across tokenizers. The single most common error with this metric.
- Comparing across context lengths. More context lowers perplexity for free.
- Assuming lower perplexity means a better assistant. RLHF makes it worse and the model better.
- Reporting it on public data without considering contamination.
- Treating it as a metric rather than the training objective. It isn't measuring the model from outside — it's what the model was built to minimise.
At a glance
FieldLanguage & LLMs
Definitionexp(cross-entropy), the training objective in readable units
Comparable across tokenizersno
Floorthe entropy of language
Correlates with usefulnessweakly, sometimes inversely
Legitimate usedetecting domain mismatch
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Perplexity vs. benchmarks — one is cheap, dense and misaligned with what you want; the other is expensive, gameable and closer to it. The field trains on the first and markets on the second.