Temperature
A single setting that controls how random or predictable an AI's output is — low for focused, high for creative.
When not to use it
- As a quality dial. Lower temperature doesn't make a model more correct — it makes it more predictable. A confidently wrong answer at temperature 0 is still wrong, just reliably so.
- To fix hallucination. Determinism isn't accuracy; the model still generates from the same flawed distribution, just less adventurously.
- Without pinning it. An unset temperature means your outputs change between runs and you'll debug ghosts.
Reach for something else instead
- Top-p / nucleus sampling gives finer control over the tail than temperature alone, and the two interact.
- Structured output — schemas or constrained decoding — when what you actually wanted was reliable format, not low randomness.
- Better prompts or examples when what you actually wanted was better content.
Read more on the blog
- Why AI gives different answers: sampling and temperatureAsk a chatbot the same question twice and you often get two different answers. That is not a glitch. A language model does not choose the next word; it rolls weighted dice over thousands of options, and a setting called temperature controls how loaded those dice are. Here is how the last step of generation actually works.
- Knowledge distillation: how small models learn from big onesThe small, fast AI models you run on a laptop or serve cheaply to millions of users were often not just shrunk from big models. They were taught by them. Knowledge distillation trains a compact student model to mimic a large teacher, and the surprising part is that the student learns more from the teacher's uncertainty than from the raw right answers.
- How a sentence becomes an answer: an LLM end to endMost explanations of large language models cover one piece, attention, or tokens, or sampling, in isolation. This follows a single sentence all the way through the machine, from the moment you hit enter to the words that come back, so the pieces finally connect.
- How to reduce AI hallucinations: what actually worksYou cannot instruct a model into being truthful, because the process that invents a fact is the same one that recalls a real one. Every technique that measurably reduces hallucination works by adding something outside the model. Here is what works, in order of impact, and what only appears to.
High temperature buys variety, not imagination.
pi = softmax(logiti / T) — logits fixed at 6.2 (Paris) down to −2.4 (banana); only T moves
The bars are an exact softmax over a fixed set of logits — the model's preferences never change here, only the scaling applied before sampling. Turn it up and the distribution flattens, but look at what gains probability: the tail. Temperature has no way to invent a better continuation, because there isn't one hiding in the distribution; all it can do is make the worse ones more likely. That is why cranking it up for creativity buys incoherence. The illustrative logits are stated in the page; the arithmetic is the real softmax used everywhere.
Worked example
Temperature reshapes a model's probabilities before it samples the next token. Say the raw scores (logits) for three candidate words are 2.0, 1.0, 0.5. Softmax turns them into probabilities — but temperature divides the logits first.
probs ≈ 0.59, 0.22, 0.13 ← top word usually wins
Temperature = 0.5 (logits ÷ 0.5, i.e. ×2):
probs ≈ 0.79, 0.11, 0.04 ← sharper, safer, repetitive
Temperature = 2.0 (logits ÷ 2):
probs ≈ 0.44, 0.27, 0.21 ← flatter, riskier, creative
Low temperature sharpens the distribution toward the top choice (deterministic, good for facts); high temperature flattens it so unlikely words get a real chance (diverse, good for brainstorming). At temperature 0 the model always picks the single most likely token. Same model, same logits — the only thing that changed is how boldly it gambles.
The full account
The paradox at the centre of it
A language model is trained on one objective: assign high probability to real human text. It gets very good at this. Now ask it for its highest-probability text — the single most likely continuation, found by greedy decoding or beam search, which is the output the model itself considers best.
You get garbage. Bland, generic, and eventually stuck in a loop, repeating the same clause until you stop it. Holtzman and colleagues demonstrated this in 2020 in a paper whose framing is exactly right: it is curious. The model was trained to maximise the likelihood of human text. When you ask it to maximise the likelihood of its output, it produces text that is unmistakably not human. The training objective and the decoding objective are the same equation, and they point in different directions.
Nobody expected this. The natural assumption — the one everybody held before someone checked — is that a better model assigns higher probability to more human-like text, so searching harder for high-probability text should get you closer to human. It does the opposite. Beam search, which searches harder than greedy, degenerates faster.
Why human text isn't likely text
The resolution is in the data, and once seen it is difficult to unsee. Human writing is not a sequence of maximally predictable choices. Real text has a characteristic level of surprise: writers routinely pick words that were not the most probable, because the most probable word is the boring one, and language that is entirely predictable conveys no information. Holtzman's analysis showed human text sitting in a band of per-token probability that is nowhere near the maximum, and — this is the part that lands — visibly fluctuating, where beam-search text is a flat line at high probability.
So the model's probability distribution is a good model of human text taken as a distribution. The argmax of that distribution is not a sample from it, and there is no reason it should look like one. Asking for the most likely text is asking for the mode, and the mode of a high-dimensional distribution over language is a degenerate object that no draw from that distribution resembles.
This is why temperature exists, and why the correct value is almost never 0 for anything you want a person to read. Temperature 0 is the mode. The mode is repetitive.
What temperature actually does, and what it doesn't
Temperature divides the logits before the softmax. Below 1 it sharpens the distribution — the already-likely get likelier; above 1 it flattens — the tail rises.
The figure above computes what that means, and the result is narrower than the folklore. Ask a factual question and the correct answer wins at temperature 0.10 and still wins at 2.20. The ranking barely moves, because temperature is a monotonic transformation: it cannot make the second-place token beat the first. What changes is the tail — the probability of some token far down the list rising from effectively zero to a few percent. Multiply a few percent across hundreds of tokens and you get one surprising word per paragraph, which is precisely the effect people describe as creativity.
So "high temperature makes the model creative" is a description of a real phenomenon and a wrong account of the mechanism. Temperature doesn't add ideas. It raises the odds that an unlikely token gets drawn, and unlikely tokens are where both the interesting choices and the errors live. You cannot get one without the other, because they are the same operation.
| What it changes | What it can't do | |
|---|---|---|
| Temperature | The sharpness of the whole distribution | Reorder the tokens — the argmax at T=0.1 is the argmax at T=2 |
| Top-k | Truncates to k candidates, then samples | Adapt to how peaked the distribution is at this step |
| Top-p (nucleus) | Truncates to the smallest set holding p of the mass | Help when the model's tail is wrong rather than long |
| Temperature 0 | Nothing — you get the mode | Produce text a human would write, per Holtzman |
Nucleus sampling, and why it beat the alternatives
Holtzman's proposed fix is a truncation rule with one good idea in it. Top-k sampling — Fan et al., two years earlier — keeps the k most likely tokens and samples among them. The problem is that k is fixed while the distribution is not: at a step where the model is genuinely certain, k=40 drags in 39 tokens that shouldn't be there; at a step where a hundred continuations are all reasonable, k=40 arbitrarily cuts sixty of them.
Nucleus sampling makes the cut adaptive. Keep the smallest set of tokens whose cumulative probability exceeds p, and sample within it. When the model is confident, that set collapses to a handful; when it is genuinely uncertain, the set widens. Holtzman observed the nucleus ranging from one to a thousand candidates depending on the step. The rule tracks the model's own certainty rather than imposing a constant, which is why top-p became the default everywhere and top-k mostly did not.
It took four more years to explain
Nucleus sampling worked, and the account of why was an intuition: models put too much probability on tokens that should have near-zero probability, so truncating the tail helps. That is a description of the symptom.
Finlayson and colleagues supplied a mechanism in 2024, and the title says the field noticed the gap: Closing the Curious Case of Neural Text Degeneration. Their argument traces the bad tail to the softmax bottleneck — the output layer projects from a hidden dimension smaller than the vocabulary, so the set of distributions the model can represent is constrained, and it cannot assign true zero to tokens that deserve it. The errors have a structure, which means the threshold can be derived rather than tuned.
That is the shape worth noticing. A technique was found empirically in 2020, adopted universally, and understood in 2024. For four years every deployed system used a decoding rule whose justification was "it produces better text," which is not nothing but is also not a reason.
What to actually do
If there is a right answer, temperature 0 and stop reading. Extraction, classification, structured output, code that must parse — you want the mode, the mode is fine, and the reproducibility is worth more than the variety.
If a person will read it, do not use temperature 0, because Holtzman established that the mode is where degeneration lives. Use top-p around 0.9–0.95 with temperature near 1, which is the default in most APIs for reasons that are now well-founded rather than customary.
And do not reach for temperature to fix a hallucination. The tail is not where wrong facts come from — a confidently wrong model puts the wrong answer at the top of the distribution, and sharpening the distribution makes it more confidently wrong, not less. That is a retrieval problem wearing a sampling costume, and no decoding parameter has ever fixed one.
Further reading
- Holtzman et al. (2020), The Curious Case of Neural Text Degeneration — where nucleus (top-p) sampling comes from, and why pure likelihood produces bad text.
- Guo et al. (2017), On Calibration of Modern Neural Networks — model confidence is not probability.
- Fan, Lewis & Dauphin (2018), Hierarchical Neural Story Generation — top-k sampling, the other lever people reach for.
- Finlayson, May, Durrett & Ren (2024), Closing the Curious Case of Neural Text Degeneration — ICLR; traces the degenerate tail to the softmax bottleneck, and derives a threshold from it rather than guessing.
- Welleck et al. (2020), Neural Text Generation with Unlikelihood Training — the other repair: change the training objective instead of the decoding rule.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Setting temperature to 0 and expecting perfect reproducibility. Batching, hardware, and floating-point nondeterminism can still shift output.
- Cranking it up for "creativity" and getting incoherence. High temperature buys variety, not imagination.
- Adjusting temperature and top-p simultaneously, then not knowing which one changed anything.
At a glance
Where this sits
A destination. 25 concepts lead here, and nothing in the corpus depends on it.
Computed from the prerequisite graph, not assigned. How this works