Sampling
Choosing the next token from the model's probability distribution — where always picking the most likely word produces worse text, which is not what anyone expected.
Reviewed July 12, 2026Stable
When not to use it
- Greedy (T=0) for creative text. It produces bland, repetitive output and falls into loops. That's the whole finding.
- High temperature for anything factual. You're sampling from the tail, which is where the model is least calibrated.
- Beam search for open-ended generation. It optimises likelihood, which is what makes text bland.
- Both temperature and top-p at once. Pick one, or you can't reason about what you set.
Reach for something else instead
- Top-p (nucleus) — adaptive to the distribution's shape. The default that works.
- Constrained decoding — when the output must satisfy a grammar. Syntax guaranteed, truth not.
- Best-of-n with a scorer — sample several, pick with something external. Often better than tuning.
- Beam search — for translation, where there is a right answer.
Read more on the blog
- Why AI gives different answers: sampling and temperatureAsk a chatbot the same question twice and you often get two different answers. That is not a glitch. A language model does not choose the next word; it rolls weighted dice over thousands of options, and a setting called temperature controls how loaded those dice are. Here is how the last step of generation actually works.
- How a sentence becomes an answer: an LLM end to endMost explanations of large language models cover one piece, attention, or tokens, or sampling, in isolation. This follows a single sentence all the way through the machine, from the moment you hit enter to the words that come back, so the pieces finally connect.
- What is generative AI? The complete guideGenerative AI is the technology behind the chatbots, image makers, and video tools of the last few years, and it is usually explained as if each were a separate trick. Underneath, they share one idea: learn the probability distribution of some kind of data, then sample from it to make new examples. This guide explains what generative AI is, how it works, the model families, and where it really falls short.
- Speculative decoding: faster LLM generation, same outputThere is a way to make a large language model generate text two to four times faster while producing output that is mathematically identical to the slow way. It sounds impossible, but it works, and it is now standard in production serving. The trick is to let a small model guess ahead and have the big model check the guesses in parallel.
Further reading
- Holtzman et al. (2019), The Curious Case of Neural Text Degeneration — nucleus sampling, and the finding that human text is not high-probability text.
- Fan, Lewis & Dauphin (2018), Hierarchical Neural Story Generation — top-k sampling.
- Hewitt, Manning & Liang (2022), Truncation Sampling as Language Model Desmoothing — a principled account of why truncation works at all.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming the most likely token is the best token. It isn't, and that's the counterintuitive core of this.
- Tuning temperature and top-p together, then not knowing what changed.
- Reaching for repetition penalties, which suppress the terms you wanted along with the ones you didn't.
- Believing temperature 0 is deterministic. Batching and floating-point non-associativity say otherwise.
At a glance
FieldLanguage & LLMs
What it doesturns a probability distribution into one token
The surprisegreedy decoding produces worse text
Whyhuman text is not high-probability text
The knob that matterstop-p, 0.9–0.95
Nevertune temperature and top-p together
DifficultyBeginner
Often compared with
Greedy vs. nucleus sampling — one picks the model's best guess every time and reads like a machine; the other truncates the unreliable tail and keeps the variability that makes text read as written.
Where this sits
24 concepts come first. Understanding it opens up 5 more.
11Levelsteps in
24Needs firstconcepts
5Opens up2% of 310
2Areascrosses fields
Learn these firstLarge Language Model
Computed from the prerequisite graph, not assigned. How this works