How to check an AI claim before you believe it

A healthcare AI vendor advertised a hallucination rate below 0.001%. The Texas Attorney General investigated, because the number had no evidence behind it and no security or procurement team had asked for any.

That is the extreme case, and the ordinary one is more common and less actionable. In one survey, 72% of enterprise buyers said AI capability influenced their purchase, and 58% said the platform failed to deliver on its AI promises within the first year.

The gap is not usually dishonesty. It is that a number was produced under conditions nobody specified, read by someone who assumed conditions that were never met, and acted on before anyone asked what it was measured against.

Marketing claims are not evidence, and neither are benchmark scores, published papers or demo performance. Each is a measurement taken under conditions, and the conditions determine whether the number transfers to you. Six questions establish whether they do, and none requires technical expertise to ask.

The six questions

They work on a vendor pitch, a research paper, a blog post or an internal result. Each has an answer that either exists or does not, and the absence is itself informative.

1. What was it compared against?

A number alone carries no information. Improved accuracy by 40% may describe moving from 50% to 70% on a binary task, which is a large relative gain and possibly still poor absolute performance. Best in class is meaningless without the class.

Ask what the comparison was, when it was made, and whether the competitor was configured with the same care as the product. That last part is the one people never ask and it is where most comparisons fail: tuning your own system carefully and a rival's casually is nearly universal and almost never disclosed.

2. How many times was it measured?

These systems are stochastic. A single run reports one draw from a distribution, and the spread varies enormously by setting: negligible for standard benchmarks and large enough to swamp a claimed improvement for derived metrics, shifted distributions or anything involving an environment.

Ask for the number of runs and the spread. If the answer is one run, the claim is a hypothesis. If the improvement is smaller than the variance you would expect in that setting, it is a hypothesis regardless of how many decimal places it carries.

3. Could anyone else get the same result?

Not whether you will reproduce it, but whether it could be. Is the evaluation set available or described. Are the conditions stated. Was the model version pinned.

There is a subtlety worth knowing: fixing a random seed does not deliver reproducibility, because parallel floating-point arithmetic sums identical values in a different order depending on scheduling. Roughly 80% of run-to-run variance comes from sources seeding does not control. A vendor who tells you their results are exactly reproducible either has done unusual engineering or has not checked.

4. What does the documentation say it cannot do?

Go to the limitations section first and weigh its length and specificity against the capability claims. A long, specific limitations section on a modestly-claimed system is a good sign. A short, generic one on an ambitiously-claimed system is the tell.

Generic caveats that apply to any model, that it may reflect training biases or should not be the sole basis for decisions, cost nothing to write and reveal nothing. A useful limitations section names the specific inputs on which this specific system fails.

5. Who scored it, and did they share failure modes with what they scored?

Increasingly the evaluator is another model. That scales in a way human annotation does not, and it introduces a specific problem: judge and subject frequently share training data and architecture, so the judge tends to err where the subject errs. A judge from the same model family as the generator will also score its relatives generously.

Ask which model judged, whether it came from a different family than the system under test, and whether its scores were ever calibrated against human labels on data like yours. If the answer to the last one is no, the score correlates with nothing anyone can name.

6. Does it hold on your own inputs, repeatedly?

The only question that fully answers itself. Take twenty inputs from your actual traffic, including the malformed and the ambiguous. Run each several times. Score against what you actually need rather than the vendor's rubric.

This is the check that supersedes the other five, and the reason to ask them first is that they are free and this one costs a week.

What different claims owe you

The right evidence depends on the kind of claim, and applying the wrong standard produces unfair criticism as often as it catches a problem.

An accuracy claim owes a named benchmark, a stated baseline, the number of runs and the evaluation conditions. Without those it is a number, not a result.

A capability claim, such as handling your document type, owes a demonstration on unselected inputs, not a prepared example. Ask them to run your material live rather than showing you a case they chose.

A safety claim owes an independent evaluation, not a self-assessment. The distinguishing question: who conducted it, and what did they find that was negative?

A compliance claim owes documentation an auditor would accept, which is a different artifact from documentation a developer finds useful. Marketing describing a system as responsible or aligned is not governance evidence.

A cost claim owes the unit. Cost per token is a supplier metric; cost per completed task, including failed runs and human review, is what you will pay.

Red flags, ordered by how fast they should stop you

Disqualifying.

A precise figure with no methodology. The sub-0.001% hallucination rate is the type case. A number that specific implies a measurement, and a measurement implies conditions somebody can state.

Refusing a time-boxed proof of concept on your data. The strongest single signal in procurement, because it costs the vendor only effort. Willingness to be measured is what capability looks like from outside.

Evaluation on data used during development. Not subtle, and more common than it should be, usually through a held-out set consulted during tuning.

Serious.

Fully autonomous claims with no described guardrails, review flow or stopping mechanism. Autonomy without a described failure path means nobody has thought about failure.

A limitations section of three sentences. Not proof of a problem, and reliable evidence that nobody looked hard.

Benchmark scores with no version, date or conditions. Providers update models under stable names, so an undated score describes a system that may no longer exist.

Worth noting.

Metrics introduced by the vendor that claim improvement on themselves. Possibly legitimate, and it requires an argument that the metric measures something independent of the product.

Comparisons against a competitor's default configuration. Nearly always favourable and nearly never stated.

The proof of concept that predicts something

Most evaluations end in a pilot, and most pilots are designed to succeed, which makes them uninformative. Four changes fix that.

Sample the inputs, do not choose them. A random draw from real traffic including the messy cases. Selecting clean examples is how a pilot measures the wrong distribution.

Have someone who did not build it drive. Ideally someone representing the eventual user, given no coaching. The gap between their success rate and the vendor's is a direct measurement of how much the demo depended on the operator.

Write the success criteria first, with a number and a baseline, agreed by someone who is not selling it. Criteria written afterwards will be written to describe whatever happened.

Count the interventions. How many times did someone rephrase, restart or nudge. That number is the honest headline result and it is never reported.

The published guidance converges on ninety days for enterprise due diligence, and the honest framing of that is uncomfortable: ninety days is fast for evaluation and slow for a sponsor who wanted to launch last quarter. Compressing it moves the failure modes past the contract signature, where the exit cost is high.

The same six, turned on your own work

The uncomfortable part of this list is that it applies inwards, and internal claims get less scrutiny than vendor ones despite carrying more weight in decisions.

An internal result has a seller too. The team that built something wants it approved, and the same selection pressures that shape a vendor demo shape an internal pilot: cleaner inputs, a knowledgeable operator, criteria written after the fact. A pilot run by people who want the answer to be yes is not neutral evidence, and calling it internal does not make it so.

Run the questions before the review, not after. What did we compare against, and did we tune the alternative as carefully. How many runs is this. Could someone else on the team reproduce it from what we wrote down. What did we find that it cannot do, and is that written anywhere. Who scored it. And does it hold on inputs nobody selected.

The one that catches the most: what is the strongest case against this? If nobody on the team can state it, the evaluation was not adversarial and the confidence is unearned.

There is an asymmetry worth naming. A vendor overselling you costs a purchase. An internal team overselling itself costs a deployment, a quarter, and the credibility of the next honest result from the same team. The scrutiny should run at least as hard inwards as outwards, and it almost never does.

What is unresolved

How to check claims about systems nobody can reproduce. The six questions assume evidence could in principle be examined. For frontier systems trained at costs beyond nearly every institution, on undisclosed data, no external party can verify anything, and what evidentiary standard replaces reproduction is not established. This is the largest open question in the field's methodology and it is being deferred.

Whether documentation requirements produce transparency or paperwork. Regulation now mandates much of this, with real penalties. Compliance audits already find documentation that satisfies the template without carrying the information the template was for. Whether enforcement evolves to evaluate substance is not yet clear.

Whether the burden is correctly placed. Every check here asks the buyer to do work the seller could have done. Fields with mature evidence standards moved that burden through regulation or professional norms, and AI has neither yet.

What a small buyer can realistically do. Ninety days of due diligence assumes a procurement function. Most organisations buying AI tools do not have one, and the checks that scale down to a two-person team are not well established.

The counter-argument

This is a lot of friction for a software purchase. Nobody applies six-question scrutiny to a project management tool, and treating every AI purchase as a research evaluation would stop organisations adopting anything. The scrutiny should scale with the stakes, and for a low-risk internal tool most of this is overhead.

Vendors cannot always answer. A startup may not have run five seeds or built a calibration set, and holding them to a standard the field itself does not meet selects for large incumbents with compliance teams rather than for better products.

Some claims are unfalsifiable in practice and still true. A system may work well without its maker being able to demonstrate why in the terms above, and demanding evidence that does not exist can reject something that would have worked.

And the buyer's own evaluation is not neutral either. A pilot run by a team that wants the purchase approved has the same selection problems as a vendor demo. The checks apply to your own evidence as much as to theirs, which is the part most readers will skip.

The short version

A healthcare AI vendor advertised a hallucination rate below 0.001% with no evidence behind it, and a state attorney general investigated. The ordinary version is less dramatic: 72% of enterprise buyers report AI capability influencing a purchase and 58% report the platform failing to deliver within a year. The cause is usually a number produced under unstated conditions and read by someone assuming conditions that were never met.

Six questions establish whether a claim transfers. What was it compared against, since a number alone carries no information and an unstated baseline can turn 50-to-70 on a binary task into a 40% improvement. How many times was it measured, since a single run is one draw and an improvement smaller than the setting's variance is a hypothesis. Could anyone else get the same result, noting that fixing a seed does not deliver reproducibility because roughly 80% of run-to-run variance comes from sources seeding does not control. What does the documentation say it cannot do, where a short generic limitations section is the reliable tell. Who scored it, and did the judge share failure modes with the thing it judged. And does it hold on your own inputs, run repeatedly.

Different claims owe different evidence. Accuracy owes a benchmark, baseline, run count and conditions. Capability owes a demonstration on unselected inputs. Safety owes an independent evaluation and the negative findings from it. Compliance owes documentation an auditor would accept. Cost owes the unit, since cost per token is a supplier metric and cost per completed task is what you pay.

The strongest single signal in the whole process is willingness to be measured. A vendor who declines a time-boxed proof of concept on your own data has answered the question, and it costs them nothing but effort. Everything else on this list is a way of finding out cheaply what that one refusal tells you immediately.

Common questions

How do I verify an AI vendor's accuracy claim? Ask four things: what it was compared against, how many runs produced the figure, what the spread across those runs was, and under what conditions it was measured. A number without a baseline carries no information, since "improved accuracy by 40%" can describe moving from 50% to 70% on a binary task. Then ask whether the competing system was tuned with the same care, which is where most comparisons quietly fail and is almost never disclosed.

What questions should I ask an AI vendor? Six. What was this compared against. How many times was it measured and what was the spread. Could an independent party reproduce it. What does your documentation say the system cannot do. Who or what scored these results, and was that scorer from the same model family as the system. And will you run a time-boxed proof of concept on our own unselected data. The last one supersedes the rest and is the one that costs a week rather than a conversation.

What are the red flags in AI vendor claims? Disqualifying: a precise figure with no methodology behind it, refusal of a time-boxed proof of concept on your data, and evaluation on material used during development. Serious: fully autonomous claims with no described guardrails or stopping mechanism, a limitations section of three sentences, and benchmark scores without a version, date or stated conditions. Worth noting: vendor-invented metrics that the vendor's product improves on, and comparisons against a competitor's default configuration.

Why do AI demos not predict production performance? Because a demo is the endpoint of a search for something that works, which makes it evidence about the search rather than the system. Inputs get cleaned, edge cases deferred, integration sidestepped and scope narrowed until something succeeds. The operator also knows the system's shape and steers around failure modes unconsciously. Published figures put proofs of concept that never reach production at around 88%.

Is a published benchmark score reliable? It describes the benchmark, under the conditions used, at the time it was run. Measured gaps between benchmark performance and real deployment run around 37%. Scores saturate, so differences between leading models at the top of a benchmark stop being statistically meaningful while continuing to be quoted. And providers update models under stable names, so an undated score may describe a system that no longer exists. Use benchmarks to narrow a shortlist, not to predict your outcome.

How do I run a proof of concept that actually predicts production? Sample inputs randomly from real traffic rather than choosing them, including the malformed and ambiguous. Have someone who did not build it drive, with no coaching, and treat the gap between their success rate and the builder's as a measurement. Write the success criteria first with a number and a baseline, agreed by someone not selling it. And count interventions, meaning every time someone rephrased, restarted or nudged, because that count is the honest headline result and is never reported.

What evidence should an AI vendor provide? It depends on the claim. An accuracy claim owes a named benchmark, stated baseline, run count and conditions. A capability claim owes a demonstration on inputs you supply rather than a prepared example. A safety claim owes an independent evaluation and specifically the negative findings from it. A compliance claim owes documentation an auditor would accept, which is different from documentation a developer finds useful. A cost claim owes the unit, since cost per completed task including failures and human review is what you actually pay.

What if a vendor cannot answer these questions? Distinguish cannot from will not. A small team may not have run five seeds or built a calibration set, and holding them to a standard the field itself does not meet mostly selects for incumbents with compliance functions. But refusal of a time-boxed evaluation on your own data is different, because it costs only effort. Willingness to be measured is what capability looks like from the outside, and its absence is the single strongest signal available.