Why AI gives different answers: sampling and temperature

Ask a chatbot the same question twice, in fresh conversations, and you will often get two different answers. Reword nothing, change nothing, and the wording still shifts. People find this unsettling, because software is supposed to be deterministic: same input, same output. A calculator does not return a different answer to 2+2 depending on its mood. So why does a language model?

The answer is a single step at the very end of how a model generates text, and it is the one step most explanations skip. A language model does not actually pick the next word. At each step it produces a probability distribution over its entire vocabulary, and then a separate procedure, called sampling, chooses one token from that distribution, often with a deliberate dose of randomness. This piece explains that final step: what the model really outputs, how the chosen word gets selected from thousands of candidates, what the temperature setting actually does to the odds, and why all of this is a control you can turn toward reliable-and-repetitive or creative-and-varied depending on what you need. It is the missing piece in the pipeline from prompt to answer, and it explains a surprising amount of everyday AI behaviour.

What the model actually outputs: a distribution, not a word

The first thing to correct is the intuition that a model outputs words. It does not. At each step, the model computes a raw score for every single token in its vocabulary, which can be a hundred thousand or more, and those scores are turned into probabilities that sum to one. The output of a generation step is therefore not "Paris" but a full ranked list: Paris with, say, 90 percent probability, then Lyon at 3 percent, Marseille at 2 percent, and a long tail of thousands of other tokens with tiny shares. The model's real answer to "the capital of France is" is this entire distribution of confidence, not a single choice.

Something then has to collapse that distribution into one actual token to put on the screen, and that something is the sampling step. This is the point where determinism is won or lost, and where all the behaviour people find puzzling comes from. The model's job is to produce good probabilities; the sampler's job is to choose from them, and how it chooses is a decision separate from the model itself.

Two ways to choose: greedy or sampled

There are two broad ways to turn the distribution into a token, and they sit at opposite ends of a spectrum.

The simplest is greedy selection: always pick the single highest-probability token. For "the capital of France is," greedy always picks "Paris," every time, deterministically. This is predictable and reliable, which is what you want for a factual answer, but as a way of writing it is dull and repetitive, because always taking the most likely next word produces flat, generic text that often falls into loops. Real language has variety; humans do not always choose the most obvious next word, and neither should a model that is trying to write naturally.

The alternative is sampling: choose a token at random, weighted by its probability. A token with 90 percent probability is picked about 90 percent of the time, one with 3 percent about 3 percent of the time, and so on. The most likely token usually still wins, but not always, and the occasional less-likely choice is what gives the text variety, surprise, and a natural feel. Almost every modern chatbot uses sampling rather than greedy selection, and this is the direct answer to the opening puzzle: you get different answers to the same prompt because the model is rolling weighted dice over its vocabulary each time, not looking up a fixed reply. The variation is not a bug; it is the mechanism that makes the output feel written rather than retrieved.

Temperature: loading the dice

If sampling is rolling weighted dice, temperature is the knob that decides how loaded those dice are, and it is the single most important generation setting to understand. Temperature reshapes the probability distribution before a token is drawn from it.

The effect is intuitive even without the math. A low temperature sharpens the distribution: the high-probability tokens get even more of the probability and the long tail shrinks toward zero, so the top choice dominates and the output becomes more focused, predictable, and repetitive. A high temperature flattens the distribution: probability is spread out so that lower-ranked tokens become viable contenders, making the output more varied, surprising, and creative. At temperature zero, the distribution is sharpened all the way to a single point, which is just greedy selection: always the top token, fully deterministic.

The "capital of France" example makes it concrete. At a low temperature, "Paris" has almost all the probability and is chosen essentially every time. At a moderate temperature, "Paris" still wins most of the time but other tokens have a real share. Crank the temperature high and "Paris" still leads, but its lead shrinks dramatically while "Lyon," "Marseille," and others start to compete seriously, so over many runs you would see genuine variety, and eventually, if you push far enough, occasional nonsense, because the thousands of tokens that should have near-zero probability have now been handed enough to be picked. That last point is the danger of very high temperature: it does not just add creativity, it eventually admits incoherence.

Trimming the tail: top-p, top-k, and min-p

High temperature raises a problem: it can give absurd, near-zero-probability tokens (word fragments, off-topic junk) enough of a share to occasionally get selected, which is how a coherent sentence suddenly lurches into gibberish. A second family of controls exists to prevent this by cutting off the tail of the distribution, so unlikely nonsense can never be drawn.

Top-k keeps only the k most probable tokens as candidates and discards the rest: pick the top forty, say, and sample only among those. Top-p, also called nucleus sampling, is smarter: instead of a fixed count, it keeps the smallest set of top tokens whose probabilities add up to some threshold like 0.9, which means the candidate pool automatically grows when the model is uncertain and shrinks when it is confident. A newer variant, min-p, sets the cutoff relative to how likely the top token is, adapting to the model's own confidence and holding up better across temperature changes, which is why many local-model users now favour a simple temperature-plus-min-p setup.

The clean way to hold these together: temperature controls the shape of the distribution, how sharp or flat it is, while top-p, top-k, and min-p control the size of the candidate pool, how many tokens are even eligible. In practice they are combined: temperature sets the sharpness, then a tail-cutter removes whatever low-probability tokens survive, giving you creativity without the risk of the model reaching for genuine garbage.

The real point: a dial between reliable and creative

Step back and the whole apparatus is one adjustable trade-off between coherence and creativity, and knowing which end you want is what separates good use of these models from bad. Turn the settings down, low temperature, tight tail, toward greedy, and you get focused, consistent, repeatable output: the right choice for factual answers, data extraction, code generation, classification, and anything where you want the same input to give the same result. Turn them up and you get varied, surprising, imaginative output: the right choice for brainstorming, story writing, and dialogue that should not feel canned. A common starting temperature for general use sits in the middle, around 0.7, low enough to stay coherent and high enough to avoid monotony, with lower values for factual and coding work and higher for creative writing.

This dial also explains behaviour beyond mere variety. Higher temperature does not only increase creativity; it increases the chance of the model sampling a lower-probability token that happens to be wrong, which is one reason turning the temperature up tends to increase hallucination on factual tasks. It is why evaluation harnesses usually run at temperature zero: you want the model's capability measured, not the noise of the sampler, so flaky eval scores are often a temperature problem rather than a model problem. It also connects to reasoning models, which benefit from some sampling variety to explore different lines of thought, and to speculative decoding, whose whole guarantee is that it reproduces exactly this sampling distribution while running faster. One honest caveat worth knowing: temperature zero gets you close to deterministic but not perfectly so on most production systems, because the underlying parallel hardware can introduce tiny numerical variations that occasionally flip a near-tie between two top tokens.

The short version

A language model does not output a word; at each step it outputs a probability distribution over its whole vocabulary. Sampling is the final step that picks one token from that distribution. Greedy selection always takes the highest-probability token, which is deterministic but repetitive; sampling picks randomly in proportion to probability, which gives varied, natural text and is why the same prompt yields different answers each time. Temperature reshapes the distribution before sampling: low temperature sharpens it toward the top token (focused, predictable), high temperature flattens it (creative, but eventually incoherent), and zero is greedy. Top-p, top-k, and min-p trim the low-probability tail so high temperature cannot reach for nonsense. Together they form one dial from reliable-and-repeatable to creative-and-varied.

The idea to hold onto is that a model produces confidence, not a choice, and a separate sampling step turns that confidence into an actual word, with temperature deciding how much randomness to allow, which is why AI output is variable by design and adjustable from deterministic to wildly creative. The variation that feels like a flaw is the same mechanism that lets these models write rather than merely recite, and once you can see the dial, you can set it for the job instead of accepting whatever the default happens to be.

Common questions

Why does AI give different answers to the same question? Because a language model does not deterministically pick the next word; it samples one from a probability distribution over its whole vocabulary, weighted by each token's likelihood. Most chatbots use this sampling with some randomness rather than always choosing the single most probable token, so each run rolls the weighted dice differently and produces slightly different wording. The variation is intentional: it is what makes the output feel natural and written rather than a fixed, retrieved reply. Setting the temperature to zero makes the output nearly deterministic.

What is temperature in an LLM? Temperature is a setting that controls the randomness of a model's output by reshaping its probability distribution before a token is chosen. Low temperature sharpens the distribution so the most probable tokens dominate, giving focused, predictable, repetitive output. High temperature flattens it so lower-probability tokens become viable, giving varied and creative but eventually less coherent output. Temperature zero collapses the distribution to always pick the single highest-probability token (greedy decoding), producing the most deterministic results. A common general-purpose value is around 0.7.

What does temperature 0 do? Temperature zero turns off the randomness and makes the model always pick the single most probable next token, which is called greedy decoding. This gives the most predictable, repeatable output and is the right choice for factual answers, classification, data extraction, and evaluation, where you want the same input to yield the same result. Note that on most production APIs, temperature zero gets very close to deterministic but not perfectly, because the parallel hardware can introduce tiny numerical variations that occasionally change a near-tied choice.

What is the difference between temperature, top-p, and top-k? Temperature controls the shape of the probability distribution, how sharp (focused) or flat (varied) it is. Top-k and top-p control the size of the candidate pool, how many tokens are eligible to be picked. Top-k keeps a fixed number of the most probable tokens; top-p (nucleus sampling) keeps the smallest set whose probabilities sum to a threshold, adapting to the model's confidence. They are used together: temperature sets the randomness, and top-p or top-k trims off the unlikely tail so high temperature cannot select nonsense tokens.

What is a good temperature setting? It depends on the task. For factual answers, code generation, data extraction, and anything needing consistency, use a low temperature (0 to about 0.3). For general-purpose conversation, around 0.7 is a common balance between coherence and variety. For creative writing and brainstorming, higher values (roughly 0.9 to 1.2) add useful diversity. Going much above that, especially without a top-p or top-k cutoff, risks incoherent output because near-zero-probability tokens start getting enough share to be selected. Adjust one parameter at a time and test.

Does higher temperature cause more hallucination? It can. Higher temperature increases the chance of sampling a lower-probability token, and on factual tasks a lower-probability token is more likely to be wrong, so raising the temperature tends to increase hallucination and reduce factual reliability. This is why factual, extraction, and coding tasks are usually run at low temperature, and why creative settings, which benefit from variety, are less suitable when accuracy matters. Temperature does not create hallucination on its own, but it amplifies the model's tendency to occasionally pick a wrong-but-plausible token.

What is greedy decoding versus sampling? Greedy decoding always selects the single most probable next token, producing deterministic and consistent but often repetitive, generic text. Sampling instead chooses a token at random weighted by its probability, so the most likely token usually wins but not always, producing varied and natural-sounding output. Almost all conversational LLMs use sampling because greedy output reads as flat and can loop. Greedy (equivalently temperature zero) is preferred for tasks needing reliability and repeatability, while sampling with a moderate temperature suits open-ended and creative generation.