Sampling
Choosing the next token from the model's probability distribution — where always picking the most likely word produces worse text, which is not what anyone expected.
When not to use it
- Greedy (T=0) for creative text. It produces bland, repetitive output and falls into loops. That's the whole finding.
- High temperature for anything factual. You're sampling from the tail, which is where the model is least calibrated.
- Beam search for open-ended generation. It optimises likelihood, which is what makes text bland.
- Both temperature and top-p at once. Pick one, or you can't reason about what you set.
Reach for something else instead
- Top-p (nucleus) — adaptive to the distribution's shape. The default that works.
- Constrained decoding — when the output must satisfy a grammar. Syntax guaranteed, truth not.
- Best-of-n with a scorer — sample several, pick with something external. Often better than tuning.
- Beam search — for translation, where there is a right answer.
Sources & further reading
- Holtzman et al. (2019), The Curious Case of Neural Text Degeneration — nucleus sampling, and the finding that human text is not high-probability text.
- Fan, Lewis & Dauphin (2018), Hierarchical Neural Story Generation — top-k sampling.
- Hewitt, Manning & Liang (2022), Truncation Sampling as Language Model Desmoothing — a principled account of why truncation works at all.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming the most likely token is the best token. It isn't, and that's the counterintuitive core of this.
- Tuning temperature and top-p together, then not knowing what changed.
- Reaching for repetition penalties, which suppress the terms you wanted along with the ones you didn't.
- Believing temperature 0 is deterministic. Batching and floating-point non-associativity say otherwise.
At a glance
FieldLanguage & LLMs
What it doesturns a probability distribution into one token
The surprisegreedy decoding produces worse text
Whyhuman text is not high-probability text
The knob that matterstop-p, 0.9–0.95
Nevertune temperature and top-p together
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Greedy vs. nucleus sampling — one picks the model's best guess every time and reads like a machine; the other truncates the unreliable tail and keeps the variability that makes text read as written.