What an AI confidence score actually means

Ask a large model how confident it is and you will usually get a number between 80 and 100.

Studies of preference-tuned models find exactly that: verbalized confidence clusters in the top fifth of the range, and expected calibration error reaches 0.30 or higher on knowledge-intensive tasks. The stated confidence overshoots reality by thirty percentage points.

That is the well-known problem. The less-known problem is worse, and it survives fixing the first one.

A confidence score answers a question about frequencies. What anyone reading it wants is a question about this instance. Those are different questions belonging to different theories of probability, and no amount of calibration converts one into the other. A perfectly calibrated model that says 70% is telling you about a reference class it has placed you in. It is not telling you whether this particular answer is right.

Two accounts of what a probability is

The dispute is three centuries old and it resolves the confusion entirely once stated.

Frequentist. A probability is a long-run frequency. Saying an event has probability 0.7 means that in a long sequence of similar trials, it occurs about 70% of the time. On this account a probability attaches to a procedure, not to a single event, and asking "what is the probability that this specific answer is correct" is close to meaningless, because this answer is either correct or it is not.

Bayesian. A probability is a degree of belief given available information. Saying 0.7 means you would accept a bet at those odds. On this account a probability attaches to a proposition, so "the probability that this answer is correct" is perfectly sensible and requires a prior belief before evidence.

The distinction is not philosophical decoration. It determines what a number licenses you to conclude.

The classic illustration is the confidence interval, which almost everyone misreads. A 95% frequentist confidence interval does not mean there is a 95% chance the true value lies inside it. It means the procedure that generated it produces intervals containing the true value 95% of the time. The Bayesian equivalent, a credible interval, does mean what people think, and requires a prior to obtain.

Which one is a model's confidence score?

Frequentist, in every practical case, and this is where the trouble starts.

Calibration is a frequentist property. A model is calibrated when, among all the predictions it assigns confidence 0.7, roughly 70% turn out correct. Expected calibration error measures the average gap between stated confidence and observed accuracy across such buckets. That is a statement about buckets.

So a calibrated model is making a promise about aggregates. Take a thousand predictions where it said 70% and about seven hundred will be right. Which seven hundred, it has not said, and cannot.

The question a user is actually asking, standing in front of a single output, is whether to trust this one. That is a Bayesian question about a proposition, and the frequentist score cannot answer it. This is not a limitation of current models. It is a limitation of the kind of number being offered.

There is a second problem underneath. A softmax output is not a probability in either sense by default. It is a normalised score that happens to sum to one, and treating it as a probability is an assumption, not a reading. Calibration is the process of making that assumption approximately true, and it is a post-hoc correction rather than a property the training produced.

The gap in one worked example

Abstract talk about reference classes is easy to nod at. Here is the same point as a number.

A model reviews insurance claims and flags likely fraud. It is perfectly calibrated: among everything it scores at 0.70, exactly 70% are fraudulent. This is the best case, not a straw man.

A claim arrives with a score of 0.70. Should you investigate it?

The score has told you that you are in a bucket where 70% are fraud. It has not told you which. Investigating this claim, you will be right 70% of the time in the same sense that a weather forecast is right: over many claims, not on this one.

Now change one thing. Your investigators cost money, and you can afford to check the top 5% of claims by score. Suddenly the score is exactly what you need, because ranking is a frequentist question and you were asking a frequentist question all along. Which 5% is best to check is answered well by a calibrated score.

Change it again. A regulator asks why this specific claim was investigated and this one was not. "The model said 0.70" is not an answer to that question, because the score is a property of a bucket and the regulator is asking about a person. Nothing in the calibration provides what is being requested.

The pattern: a calibrated score answers volume questions and cannot answer instance questions. Route the queue, set the cutoff, size the team, forecast the workload, all fine. Justify a single decision, explain one outcome, decide whether to trust this answer, all outside what the number contains.

Almost every complaint about confidence scores being unhelpful turns out, on inspection, to be a volume tool being asked an instance question.

Two kinds of uncertainty, one number

The distinction that matters most operationally, and a single confidence score destroys it.

Aleatoric uncertainty is in the world. The question is truly ambiguous, the evidence truly underdetermines the answer, and more data would not help. A coin flip has irreducible uncertainty no matter how much you study coins.

Epistemic uncertainty is in the model. It has not seen enough of this kind of input, the case falls outside its training distribution, and more data would help.

These demand opposite responses. High aleatoric uncertainty means the answer is unavailable and the correct action is to say so and stop. High epistemic uncertainty means the answer exists and this system is not the one to produce it, and the correct action is to escalate or retrieve.

A single number cannot distinguish them, and a system reporting 0.6 could be in either state. Ensembles and Monte Carlo dropout are approximations of Bayesian methods that attempt to separate the two by measuring disagreement, on the reasoning that disagreement indicates the model does not know rather than that the answer is undetermined. They are approximations and they are better than nothing.

What the recent evidence actually shows

Four findings, and the fourth is the one worth reorganising around.

Overconfidence is universal and does not improve much with scale. Among models above 70 billion parameters, expected calibration error sits around 0.1, meaning stated confidence deviates from true accuracy by about ten percentage points on average. And a finding that should be more widely known: most of the calibration improvement that comes with scale comes from higher accuracy rather than reduced overconfidence. The models get better; they do not get more honest about being wrong.

Preference training is a mechanism, not a side effect. Reward models favour confident-sounding responses, so training against them selects for confidence independent of correctness. One line of work frames the result as reproducing a human bias: overconfident in areas of real ignorance, better calibrated where pretraining coverage was strong.

Verbalized confidence saturates. Asking a model for a number tends to produce 0.9 or 1.0, which makes the output useless for ranking or thresholding even when it is directionally right. Granularity has to be engineered in.

And the finding that matters most: models do not act on their own stated uncertainty. Work in early 2026 found that models can verbalize uncertainty reasonably well in isolation and then fail to use it to guide behaviour. A system will say it is not entirely sure and then take an irreversible action as though it were certain. The confidence estimate and the decision policy are not connected.

That last one reframes the whole problem. Improving calibration does not help if nothing downstream consumes the number, and in most deployed systems nothing does.

What to do with a confidence score

Given all that, the practical rules are narrower than the metric suggests.

Use it for ranking, not for deciding. Even a poorly calibrated score frequently orders cases usefully. Sorting a review queue by confidence works when the absolute numbers are meaningless.

Set thresholds empirically against your own data. A cutoff of 0.9 means whatever it means for this model on this distribution, which is not what it meant for another model or the same model last quarter. Calibration degrades under distribution shift, so the threshold has to be revalidated when the input changes.

Connect it to something. A confidence number that no code branches on is decoration. If low confidence does not trigger abstention, escalation or a cheaper fallback, measuring it changes nothing.

Measure disagreement instead of asking. Sampling a model several times and measuring variation in its answers is generally more informative than asking it how sure it is, because it observes behaviour rather than eliciting a self-report.

And treat the two uncertainties differently. If you can distinguish ambiguity from ignorance, do, because they need different handling. If you cannot, at least know that you cannot, rather than reading one number as though it meant one thing.

What is unresolved

Whether models can be truly well calibrated. Every intervention so far is post-hoc correction or training against a scoring rule, and both improve the metric without clearly improving the underlying epistemic state. Whether a language model has something worth calling a confidence that calibration is revealing, or whether calibration is fitting a number to observed accuracy, is not settled.

How to separate aleatoric from epistemic at scale. Ensembles work and cost several times inference. Cheap approximations exist and are unreliable. No method is both, which is the same trade seen across evaluation.

Whether a single scalar is the right interface at all. Recent work argues that confidence at the level of a whole response is insufficient, and that claim-level or span-level uncertainty is what downstream systems need. That is a harder problem and it may be the correct one.

And what users should be shown. One study found that medium expressed uncertainty produced higher trust and better task performance than either high or low, which is an awkward result: the most useful thing to display may not be the most accurate thing.

The counter-argument

Calibration is still worth having. The argument that a frequentist score cannot answer an instance-level question is correct and does not make the score useless. Aggregate guarantees are the basis of most engineering, and a system whose stated confidence tracks its accuracy is better than one whose does not, whatever philosophy says.

The Bayesian framing has its own problem. A degree of belief requires a prior, and for a language model nobody can say what the prior is or should be. Criticising frequentist calibration for not answering the instance question invites the reply that the Bayesian alternative answers it only by assuming something unverifiable.

Overconfidence may be partly rational. A model trained to be helpful, deployed to answer, and penalised for hedging is responding to its incentives. The miscalibration is a product of what was optimised, and calling it a flaw of the model rather than of the objective misplaces the responsibility.

And people are badly calibrated too. Expert forecasters, doctors and engineers are all systematically overconfident, and the comparison class for a deployed system is usually a person rather than an oracle. Holding models to a standard humans do not meet is not obviously the right bar.

The short version

Preference-tuned models cluster their stated confidence between 80 and 100, with expected calibration error reaching 0.30 on knowledge-intensive tasks, meaning confidence overshoots accuracy by thirty points. Fixing that does not fix the deeper issue.

A confidence score is a frequentist quantity and readers want a Bayesian answer. Frequentist probability is a long-run frequency attaching to a procedure; Bayesian probability is a degree of belief attaching to a proposition. Calibration means that among predictions assigned 0.7, about 70% are correct, which is a promise about buckets. Which members of the bucket are the correct ones is not stated and cannot be. A user looking at one output and asking whether to trust it is asking the other kind of question.

A softmax output is also not a probability in either sense by default. It is a normalised score, and calibration is a post-hoc correction making that assumption approximately true rather than a property training produced.

One number also destroys the distinction that matters most. Aleatoric uncertainty is in the world and means the answer is unavailable, so the correct action is to stop. Epistemic uncertainty is in the model and means the answer exists elsewhere, so the correct action is to escalate. A system reporting 0.6 could be in either state.

And the finding that reframes everything: models do not act on their own stated uncertainty. Recent work found systems can verbalize uncertainty reasonably in isolation and then take irreversible actions as though certain, because the confidence estimate and the decision policy are not connected. Improving calibration changes nothing if nothing downstream consumes the number, and in most deployments nothing does.

Common questions

What does an AI confidence score mean? Usually a frequentist claim about aggregates: among all predictions the model assigns that confidence, roughly that fraction should be correct. It is a statement about a bucket of cases rather than about the specific answer in front of you. A perfectly calibrated model saying 70% is telling you about the reference class it has placed your query in, and it has not told you whether this particular answer is one of the correct ones.

Are LLM confidence scores reliable? Not in absolute terms. Preference-tuned models cluster their stated confidence between 80 and 100, and expected calibration error can reach 0.30 on knowledge-intensive tasks, meaning stated confidence overshoots true accuracy by around thirty percentage points. Even large models sit around 0.1 error. They are frequently useful for ranking cases relative to each other and unreliable as absolute probabilities.

What is model calibration? The property that stated confidence matches observed accuracy: among predictions assigned confidence p, about a fraction p are correct. It is measured by expected calibration error, which averages the gap between confidence and accuracy across buckets. Calibration is a frequentist property, achieved post-hoc through methods like temperature scaling, and it is a correction applied to scores rather than something training naturally produces.

What is the difference between aleatoric and epistemic uncertainty? Aleatoric uncertainty is in the world: the question is truly ambiguous and more data would not help, as with a coin flip. Epistemic uncertainty is in the model: it has not seen enough of this kind of input, and more data would help. They require opposite responses, since the first means the answer is unavailable and the second means the answer exists but this system is not the one to produce it. A single confidence number cannot distinguish them.

Why are language models overconfident? Partly by construction. Reward models used in preference training favour confident-sounding responses, so training selects for confidence independent of correctness. One characterisation is that this reproduces a human bias: overconfidence in areas of real ignorance, with better calibration where pretraining coverage was strong. Notably, most calibration improvement that comes with scale comes from higher accuracy rather than reduced overconfidence.

Can I just ask a model how confident it is? You can, and the answer tends to saturate at 0.9 or 1.0, which makes it useless for ranking or thresholding even when directionally correct. A more informative approach is sampling the model several times and measuring how much its answers vary, since that observes behaviour rather than eliciting a self-report. Disagreement across samples is evidence the model does not know.

What is the difference between Bayesian and frequentist probability? Frequentist probability is a long-run frequency and attaches to a procedure, so a 95% confidence interval means the procedure produces intervals containing the true value 95% of the time, not that this interval has a 95% chance of containing it. Bayesian probability is a degree of belief and attaches to a proposition, so it can express how likely a specific claim is, at the cost of requiring a prior belief before evidence.

How should I use a confidence score in production? For ranking rather than deciding, since even poorly calibrated scores often order cases usefully. Set thresholds empirically against your own data and revalidate them when the input distribution changes, because calibration degrades under shift. And connect the number to something: if low confidence does not trigger abstention, escalation or a cheaper fallback, then measuring it changes nothing, which is the state most deployed systems are in.