Testing a system that answers differently every time

Around 40% of organisations deploying LLM applications report significant quality regressions within the first ninety days of production.

The cause is rarely the model. It is that the team brought a testing discipline built on an assumption that no longer holds: that the same input produces the same output, so a test can assert equality and mean something.

Drop that assumption and every familiar tool breaks in a specific way. Exact-match assertions reject valid answers phrased differently and occasionally accept wrong ones that happen to match. A test that passes today fails tomorrow with no code change. And the natural response, setting temperature to zero, does not restore determinism, because identical inputs can still produce different outputs when floating-point arithmetic on parallel hardware sums the same values in a different order.

The fix requires two changes at once, and teams almost always make only one. Assertions have to move from exact to property-based, and results have to move from binary to statistical. A property-based assertion evaluated on a single run is still a coin flip, and a statistical threshold over exact matching is still measuring phrasing.

The two axes

Worth separating, because they fail differently and are fixed differently.

Exact to property-based. Instead of asserting the output equals a string, assert it has the properties that make it correct. The formulation that works is an expected output profile rather than an expected output: the required intent, the facts that must appear, the content that must not appear, and the format it must take.

This is not a weaker test. In several respects it is stronger, because it states what actually matters. An exact-match test on a paragraph asserts several hundred things, of which perhaps four matter, and it fails on the other several hundred without distinguishing them.

Binary to statistical. Instead of pass or fail on one run, run the case several times and assert a rate. Three runs is the common floor and more is better where variance is high. The assertion becomes "passes at least four of five times" rather than "passes".

Making only the first change gives you a test that checks the right thing unreliably. Making only the second gives you a reliable measurement of the wrong thing. Both together is the working configuration.

Not everything should be statistical

The most useful structural move is deciding what is allowed to be non-deterministic, because far less of the system needs to be than teams assume.

Deterministic, and should be tested conventionally: whether the right tool was called, whether the output parses as valid JSON, whether required fields are present, whether a value falls in a permitted range, whether a permission check ran, whether the retrieval query was constructed correctly.

Non-deterministic, and needs statistical treatment: whether the generated prose is faithful to the retrieved context, whether the tone is right, whether the answer is helpful, whether a summary preserved the load-bearing facts.

The split has a practical consequence: deterministic checks are cheap, fast and can run on every commit, while statistical evaluation is slow and expensive and belongs on merges or nightly. A team running the whole suite statistically has made its feedback loop slow enough that people stop running it, which is worse than not having it.

The routing layer, the parsing layer and the permission layer are all deterministic. Only the generation is not, and generation is usually a minority of the pipeline.

The golden set

The evaluation set is the artifact everything else depends on, and there is reasonable convergence on its shape.

Start at 25 to 50 cases and grow. A production set typically lands between 100 and 300 diverse, mutually exclusive cases. Bigger is not better past that point: the marginal case adds cost on every run and rarely adds information.

Every case needs its criteria attached, not just its input. Required intent, mandatory facts, forbidden content, format. The criteria are the test; the input is only how you get there.

Fill it from three sources. The important use cases, which everyone does. Known failure modes from past incidents, which converts every production bug into permanent coverage and is the highest-value habit here. And edge cases found through production monitoring, which is where the cases nobody imagined come from.

Freeze it. A set that keeps changing cannot support comparison over time, which was the point of having one. Version it, and when you add cases, note that scores before and after the change are not comparable.

The tension is real and worth naming: a frozen set drifts away from live traffic, and a refreshed set loses its baseline. The workable answer is a frozen core plus a rotating supplement, scored separately.

A worked suite for a support assistant

The abstract split between deterministic and statistical is easier to apply against a concrete system, so here is one, with every check assigned.

The system takes a customer question, retrieves policy documents, and drafts a reply.

Deterministic, every commit, milliseconds:

• The retrieval query was constructed from the question, not from the raw input
• Exactly the top five passages were passed forward, not twenty
• The draft parses as valid JSON with body, confidence and citations present
• Every citation identifier resolves to a document that exists
• No passage from a restricted collection appears in the context
• Total prompt length is under the configured ceiling

Six checks, no model calls beyond the pipeline itself, and they catch the assembly and permission failures that account for a large share of production incidents.

Statistical, on merge, five runs per case:

• The reply is faithful to the retrieved passages, at 95% or better
• Every policy figure mentioned appears in a cited passage, at 98% or better
• The reply does not promise anything outside policy, hard gate at 100%
• Tone matches the brand rubric, at 90% or better

Four checks against a hundred-case golden set, five runs each: two thousand judgements, which is the real cost and the reason this does not run on every commit.

The part teams skip: the numbers above are not thresholds you set once. They are the current production baseline, measured. The gate is that a change may not drop any of them by more than a point, which requires having measured them first. A team that sets 95% aspirationally without knowing the current figure has written a wish rather than a gate.

What a threshold should actually be

Setting a pass mark is where most suites go wrong, and the errors are predictable.

Not an absolute score. "Must exceed 0.85" is meaningless without knowing what 0.85 means for this rubric on this data, and it will be either trivially met or impossible.

A regression tolerance against the current production version. This is the formulation that works: no metric may fall more than X below what is currently deployed. It is comparative, so it needs no absolute calibration, and it answers the question a release actually poses, which is whether this change makes things worse.

Per dimension, blocking independently. An aggregate lets a strong score in one dimension purchase a collapse in another. Faithfulness, format compliance and safety should each be able to block a release on their own.

With safety as a hard gate rather than a threshold. Some things do not get a tolerance.

Where the regressions actually come from

The ninety-day figure needs explaining, because the obvious causes are not the common ones.

Provider model updates. A routine update from a model provider, under a stable name, can silently degrade a behaviour your product depends on. You did not change anything. This is the single most common cause of a regression nobody can attribute, and it is why pinning model versions and re-running the suite on provider changes matters more than most testing advice.

Prompt edits with unmeasured side effects. A change made to improve one behaviour degrades another that nobody tested. Without a suite this is invisible; with one it is a blocked merge.

Retrieval corpus growth. Adding documents changes what competes for retrieval. A system that worked at ten thousand chunks can degrade at a hundred thousand with no code change at all.

Drift in what users ask. The input distribution moves and the golden set does not, so the suite keeps passing while the live experience deteriorates.

Notice that three of the four involve no change to your code. A testing discipline that only runs on commits will miss most of them, which is the argument for running the suite on a schedule as well as on changes.

How many runs, and what that buys

The advice to run each case several times is universal and never quantified, which makes it easy to under-do. The arithmetic is simple enough to carry.

If a case truly passes 80% of the time, a single run reports pass 80% of the time and fail 20%. So a suite of one run per case, on a system with several 80% cases, produces a different result on every execution and looks broken.

Three runs gives you 0, 1, 2 or 3 passes. On an 80% case you see three passes about half the time and two passes about 38%. Enough to notice the case is unstable, not enough to estimate its rate.

Five runs narrows it usefully and is where most practical guidance lands. A case at 80% shows four or five passes about 74% of the time. You can set a gate at "four of five" and have it hold.

Ten runs distinguishes 80% from 90%, which three or five cannot. Worth it for a small number of critical cases and not for the whole suite.

The practical shape that follows: run the whole set three times, and a designated critical subset ten times. Uniform run counts either under-measure the cases that matter or overspend on the ones that do not.

One consequence worth stating plainly. A gate of "must pass all five runs" on a case that truly passes 95% of the time will fail about 23% of the time through no regression at all. Gates set at 100% on statistical cases produce exactly the flaky-suite problem teams then blame on the model.

The flakiness trap

One failure mode deserves separate treatment because the natural response makes it worse.

A case that passes sometimes and fails sometimes looks like a flaky test, and the reflex from conventional testing is to stabilise or remove it. Here it is usually a finding. A case that passes four times in five is telling you the system handles it 80% of the time, which is information about the system rather than noise in the test.

Deleting it removes the evidence. Loosening the criteria until it always passes removes it more quietly. The correct response is to record the rate and decide whether 80% is acceptable for that case, which is a product question rather than a testing one.

The cases with the lowest and least stable pass rates are the most informative in the whole suite, and they are the ones most likely to be quietly retired for being annoying.

What is unresolved

Whether semantic similarity is the right property check. Comparing an output to a reference by embedding proximity is the common approach and inherits everything embeddings do badly, including that opposites embed close together. A response saying the reverse of the reference can score highly.

How to score multi-step outcomes. Scoring a final answer is tractable. Scoring a forty-step run where an early error was recovered and a later one was not is not, and no accepted methodology exists. Current practice scores the endpoint, which cannot distinguish reliability from luck.

Where the judge sits in all this. Using a model to score outputs makes the evaluation itself non-deterministic and inherits the judge's own biases. Findings on rubric stability are not reassuring: ordering, score identifiers and whether a reference is included all affect scores. The measuring instrument needs its own measurement.

What confidence a small set can support. A hundred cases run five times each gives five hundred observations, which sounds substantial and is spread across many distinct behaviours. The statistical power for any individual behaviour is low, and nobody has established what set size supports what claim.

The counter-argument

Most of this is standard practice from other stochastic domains. Statistical acceptance criteria, property-based testing and repeated trials are established in performance engineering, hardware validation and clinical work. Framing it as a new discipline overstates the novelty, and teams from those backgrounds find it familiar.

Cost is a real constraint, not an excuse. Running a 200-case suite five times on every merge is a thousand model calls per merge, and at frontier pricing on a busy repository that is a meaningful line item. The advice to run more is advice to spend more, and a smaller suite run properly beats a larger one run once.

Determinism is available for parts of the system. Temperature zero plus a fixed seed gets you close for many operations, and the residual hardware non-determinism affects outputs less often than the framing here implies. Some teams are fine with near-deterministic testing and do not need the statistical apparatus.

And a suite can be theatre. A pipeline producing green checkmarks on cases nobody chose carefully, against criteria nobody validated, provides confidence rather than information. The existence of an evaluation suite is not evidence that it measures anything.

The short version

Around 40% of organisations deploying LLM applications hit significant quality regressions within ninety days, and the cause is usually a testing discipline built on an assumption that no longer holds: that the same input yields the same output. Exact-match assertions reject valid answers phrased differently and occasionally accept wrong ones, and setting temperature to zero does not restore determinism, since parallel floating-point arithmetic can sum identical values in a different order.

Two changes are required together and teams typically make one. Assertions move from exact matching to an expected output profile stating required intent, mandatory facts, forbidden content and format. Results move from binary to statistical, asserting a pass rate across several runs rather than a pass on one. A property-based assertion on a single run is still a coin flip; a statistical threshold over exact matching still measures phrasing.

The highest-value structural move is deciding what is allowed to be non-deterministic, since less of the system is than teams assume. Tool selection, output parsing, field presence, value ranges and permission checks are all deterministic and belong in fast tests on every commit. Only generation quality needs statistical treatment, and it belongs on merges or nightly, because a suite slow enough to skip is worse than none.

Thresholds should be regression tolerances against the current production version rather than absolute scores, set per dimension so each can block independently, with safety as a hard gate. And three of the four common regression causes involve no change to your code: provider model updates under a stable name, retrieval corpus growth, and drift in what users ask.

The trap worth naming: a case that passes four times in five looks like a flaky test and is usually a finding. It is telling you the system handles that input 80% of the time, which is information about the system rather than noise in the suite. Deleting it removes the evidence, and loosening the criteria until it passes removes it more quietly. The least stable cases are the most informative ones you have.

Common questions

How do you regression test a non-deterministic system? Two changes together. Replace exact-match assertions with an expected output profile specifying required intent, facts that must appear, content that must not, and the required format. Then replace binary pass or fail with a rate across multiple runs, typically at least three, asserting something like four passes in five. Making only one change leaves you either checking the right thing unreliably or reliably checking the wrong thing.

Does setting temperature to zero make testing deterministic? No, and this is a common surprise. Identical inputs can still produce different outputs at temperature zero, because tensor operations split across parallel hardware and floating-point addition is not associative, so the summation order changes the result in the last decimal places. Temperature zero substantially reduces variation without eliminating it, and results are not guaranteed identical across different hardware.

What should go in a golden set? Start with 25 to 50 cases and grow toward 100 to 300 diverse, mutually exclusive cases. Draw from three sources: your most important use cases, known failure modes from past incidents so every production bug becomes permanent coverage, and edge cases discovered through production monitoring. Each case carries its criteria rather than just its input, since the criteria are the actual test.

How large should an evaluation set be? A production set typically lands between 100 and 300 cases, which balances coverage against the cost of running it repeatedly. Bigger is not automatically better, since every additional case is paid for on every run and the marginal case rarely adds information. A smaller set run several times per case is generally more informative than a larger set run once.

What should the pass threshold be? A regression tolerance against the current production version rather than an absolute score. Absolute thresholds are meaningless without knowing what a given score means for your rubric on your data, and they end up either trivially met or impossible. Set them per dimension so faithfulness, format compliance and safety can each block a release independently, and treat safety as a hard gate rather than a tolerance.

Should every test be statistical? No, and deciding otherwise is the most valuable structural choice available. Tool selection, JSON validity, required field presence, value ranges, permission checks and query construction are all deterministic and belong in fast conventional tests running on every commit. Only generation quality needs repeated runs and statistical thresholds, and that belongs on merges or nightly, because a suite slow enough that people skip it is worse than no suite.

Why do LLM applications regress without any code change? Because three of the four common causes are external. Providers update models under stable names, which can silently degrade a behaviour your product depends on. Retrieval corpora grow, changing what competes for retrieval, so a system that worked at ten thousand chunks degrades at a hundred thousand. And the distribution of what users ask drifts away from the frozen evaluation set, so the suite keeps passing while the live experience deteriorates.

What do I do about a test that passes intermittently? Treat it as a measurement rather than a flake. A case passing four times in five is telling you the system handles that input 80% of the time, which is a fact about the system. Record the rate and decide whether 80% is acceptable for that case, which is a product question. Deleting the case removes the evidence, and loosening the criteria until it always passes removes it less visibly. The least stable cases are usually the most informative in the suite.