Your agent works in the demo because the demo is the easy case

The demo worked. Everyone in the room saw it work. Three months later the rollout has stalled, accuracy is worse than it was, and nobody can explain what changed, because from the team's perspective nothing did.

The numbers on this are not marginal. IDC finds that 88% of AI proofs of concept never reach production, with roughly four of every thirty-three launched projects surviving the journey. S&P Global found 42% of companies abandoned most of their AI initiatives in 2025, up from 17% the year before, and that the average organisation scrapped 46% of its proofs of concept. MIT's 2025 study of enterprise generative AI put it most starkly: 95% of pilots delivered no measurable impact on profit and loss.

The usual explanation is that production is harder than a demo. That is true and it is not the interesting part, because it suggests the difference is one of degree. It is not.

A demo is not a small production system. It is the survivor of a search process designed to find something that works, which makes it evidence about the search rather than about the system. Everything that made the demo succeed was selected for, usually without anyone noticing they were selecting.

The demo is a survivor

Start with how a demo actually gets built, rather than how it gets described afterwards.

You try an approach. It fails on some inputs. You adjust the prompt, or swap the model, or change the retrieval, or pick different example documents. It fails less. You repeat this until the thing works on the cases you are testing with. Then you show that.

What you are showing is the endpoint of a search. Every intermediate failure was information, and the process discarded it. The artifact that survives is the one configuration, out of the many you tried, that happened to work on the specific inputs you happened to use.

This is survivorship bias, and it is built into the object rather than into the interpretation of it. The demo cannot be unbiased, because the process that produced it was a filter.

The consequence is specific and it is not intuitive: a demo's success rate is not an estimate of the system's success rate. It is an estimate of the success rate on the inputs used during development, which is a set the system has effectively been fitted to. In machine learning this has a name, and everyone who builds demos knows what overfitting is. Almost nobody applies the concept to the demo itself.

What the selection removed

The filtering happens in four places, and each removes a different part of the problem.

The inputs got cleaned. Teams building a proof of concept pick the cleanest available data. Not dishonestly, usually not consciously; the messy records slow you down and the point of the exercise is speed. But the messy records are not noise around a signal. In most enterprise systems they are a substantial fraction of the actual distribution, and they are the fraction where the value is, because the clean cases were already handled by whatever existed before.

The edge cases were deferred. Every proof of concept has a moment where someone says "we'll handle that later." What follows that phrase is almost always the long tail, and the long tail is not a rounding error on the work. It frequently is the work. The clean centre of the distribution was tractable with existing tools; the reason anyone wanted an agent was the tail.

Integration was sidestepped. The demo ran against a copy of the data, or an export, or a curated subset in a scratch environment. Production needs the real system, with its access model, its rate limits, its inconsistencies and its downtime. This is the most consistently underestimated part of the transition and the point where the largest share of pilots stop.

Scope was narrowed until it worked. If the original question was broad and the demo answers a narrow version of it, the narrowing was a finding and it was recorded as a success.

The watcher effect

There is a person in the room during the demo who wants it to work.

That person notices when the output is subtly wrong and rephrases the question. They know which examples to use. They recognise a failure mode from previous runs and steer around it before it manifests. None of this is dishonest, and most of it is unconscious: a practised operator of a system develops an intuition for how to hold it.

In production there is no such person. There is a user who does not know the system's shape, phrases things in ways the developers never anticipated, and does not recognise a wrong answer as wrong.

The gap between an expert operator and a naive user is large for any tool. For a system that accepts natural language and responds plausibly regardless of whether it is correct, the gap is much larger than usual, because the naive user has no feedback signal telling them they are holding it wrong.

This is worth stating precisely because it is the hardest part of the demo problem to correct for. You can widen the input distribution deliberately. You cannot easily un-learn how to use your own system.

Static data, moving world

A proof of concept validates against a snapshot. Production runs against a distribution that moves.

Two things change under it. The inputs shift: what users ask, how they phrase it, which documents exist, what the business is doing this quarter. And the relationship between inputs and correct outputs shifts, which is the harder of the two because it is invisible in the inputs.

A pilot measured over two weeks cannot observe either. It has no baseline to compare against and no elapsed time in which to drift. The result is that the demo's accuracy figure is not merely optimistic; it is a measurement of a system property that the production system does not have, because the production system exists in time and the demo did not.

Concurrency, state and everything else that only exists at scale

A demo is one request at a time, in a fresh session, with an empty state.

Production has concurrent requests sharing infrastructure, so latency distributions widen and rate limits bind. It has accumulated state, so an agent may act on something recorded weeks ago that is no longer true. It has partial failures, where step four succeeded and step five did not, leaving a system in a condition the demo never produced because the demo never failed halfway. And it has retries, which for any operation with side effects raises a question the demo never had to answer.

None of these are performance problems in the usual sense. They are new categories of failure that do not exist in the demo environment, which means the demo cannot provide evidence about them in either direction.

Length is the multiplier

The reliability arithmetic that governs agent workflows applies with particular force here, because demos are short and production tasks are not.

A demo shows three or four steps. A real workflow runs to fifteen or twenty. At 95% per-step reliability, four steps complete about 81% of the time and twenty steps complete about 36%. The demo was not lucky and the production system is not broken. They are the same system measured at different lengths, and the difference between those two numbers is entirely a function of how many steps you watched.

This means demo length is a variable worth controlling deliberately. A demo of the full workflow, end to end, on unselected inputs, is a much stronger piece of evidence than a demo of the interesting middle section, and it is considerably less impressive to watch, which is why it is rarely what gets shown.

The same mechanism, outside AI

It helps to notice that none of this is specific to agents, because the general version has been studied and the AI version is a special case with the dial turned up.

Drug trials have the same structure. A compound is tested on a selected population under controlled conditions, and effect sizes routinely shrink when it reaches a general population with comorbidities and imperfect adherence. The response was to make the trial harder rather than the reporting more optimistic: randomisation, pre-registration, intention-to-treat analysis, and phase structure that deliberately widens the population before approval.

Software performance testing has it too. A benchmark run on a quiet machine with a warm cache and a synthetic workload predicts production latency poorly, and the profession's answer was load testing against realistic traffic shapes rather than better benchmarks.

Two things make the AI case worse than either.

The output is plausible when wrong. A drug that does not work produces no effect. A load test that fails produces a timeout. An agent that fails produces a confident, well-formed answer, which means the demo can be failing while appearing to succeed, and neither the operator nor the audience has a signal.

The system was fitted to the demo inputs. A drug is not adjusted between patients based on whether the last one responded. A prompt is adjusted between runs based on exactly that, which makes the demo inputs part of the development set rather than a test of it. This is the part with no analogue elsewhere, and it is the reason the overfitting framing is precise rather than a metaphor.

The useful conclusion from the comparison is that other fields solved their version of this by changing what counts as evidence rather than by improving the technology. That is the available move here too, and it is cheaper than waiting for models to improve.

What a demo can legitimately tell you

Being fair to the format, because the argument so far could be read as saying demos are worthless and they are not.

That the capability exists at all. If the system cannot do the task on clean inputs with an expert operator, it certainly cannot do it on messy inputs without one. A failed demo is strong evidence and it is cheap.

That the integration is conceivable. Even against a copy of the data, connecting the pieces reveals whether the pieces can be connected.

Where the hard parts are. A demo built honestly surfaces which steps needed the most iteration, and those steps are where production will fail. This information is usually discarded because it is not part of the presentation.

What the interface should be. Watching someone use the thing, even in a controlled setting, tells you a great deal about what they expected it to do.

A failed proof of concept is also not wasted work, and the framing that treats it as such produces bad incentives. It prevents you from scaling something that would not have worked, at a cost measured in weeks rather than the several million a failed production deployment runs to. The problem is not that demos fail. It is that successful demos are read as evidence they are not.

Running a demo that predicts something

If the goal is a demo whose result generalises, the changes are unglamorous and mostly involve making it harder.

Sample the inputs, do not choose them. Take a random draw from real traffic, including the malformed and the ambiguous. If you cannot get real traffic yet, the demo is measuring something else and you should say so.

Feed it the mess deliberately. Adversarial testing during the pilot rather than after it: incomplete records, contradictory documents, questions outside scope, inputs in the wrong format. If the system cannot handle imperfection at small scale it will not survive at large scale, and finding that out in week two is much cheaper than finding out in month nine.

Have someone else drive. Not the person who built it. Ideally someone who represents the eventual user, given no coaching, with their session recorded. The difference between their success rate and the builder's is a direct measurement of the watcher effect.

Run the whole workflow. Not the interesting section. End to end, including the steps everyone assumes are trivial, because the assumption that a step is trivial is itself untested.

Write the success metric first. Before the demo, with a number and a baseline, agreed by someone who is not building it. A metric written afterwards will be written to describe whatever happened.

Count the interventions. How many times did someone rephrase, restart, or nudge? That number is the honest headline result, and it is never reported.

The evidence on whether this discipline pays is limited and pointed in the right direction: organisations that define success metrics upfront and invest substantially in data preparation report materially higher success rates than those that do not. Treat that as encouraging rather than established, since it comes from self-reported industry survey work rather than controlled comparison.

What is unresolved

Whether better models close the gap or move it. If model capability improves enough, some share of demo-to-production failures becomes a timing problem rather than a methodological one. Against that, the selection effects described here are properties of how demos are built rather than of how models perform, and a more capable model demoed the same way produces a more impressive demo with the same bias. The honest position is that capability improvement helps and does not address the mechanism.

Whether the numbers describe failure or normal attrition. An 88% proof-of-concept mortality rate is alarming if those projects had real use cases and were killed by preventable methodology, and unremarkable if most were exploratory by design. The published figures do not distinguish these, and the distinction changes what conclusion to draw entirely. Anyone quoting the number as evidence that AI does not work is over-reading it.

How much of the failure is organisational rather than technical. One survey attributes 77% of AI project failures to organisational rather than technical causes, which is consistent with the broader deployment literature and rests on self-reported categorisation by people with an interest in where blame lands. The direction is probably right and the precision is not.

The counter-argument

Several things weigh against the framing above.

Demos are supposed to be optimistic. The purpose of a proof of concept is to establish whether something is possible, not to estimate its production performance. Criticising a demo for being unrepresentative is criticising a hammer for being a poor screwdriver, and the failure is in the reading rather than in the artifact.

Adversarial testing has a cost. Building a pilot against the full messy distribution takes considerably longer, and a substantial share of pilots should be killed quickly rather than engineered carefully. There is a real argument that fast, unrepresentative demos are the correct filter precisely because they are cheap, and that the discipline described above should apply only after a demo has passed.

The industry numbers are soft. Figures ranging from 80% to 95% failure come from surveys with different definitions of failure, different populations and commercial sponsors. They agree in direction, which is meaningful. Their precision should not be relied upon, and the variance between them is itself informative about the measurement.

And the base rate for new technology is bad. Most projects in most technology waves fail, and pointing at a high failure rate in a category attracting this much capital is not obviously a finding about the category. It might just be a finding about experimentation.

The short version

A demo is not a small production system. It is the endpoint of a search process in which you tried configurations until one worked on the inputs you were using, which makes it the survivor of a filter rather than a sample of behaviour. Its success rate estimates performance on the development inputs, which the system has effectively been fitted to, and calling that overfitting is exact rather than metaphorical.

Four things get selected out during that process. Inputs get cleaned, because messy records slow development down. Edge cases get deferred, and the long tail is frequently where the value was. Integration gets sidestepped by running against a copy, and real system access is where most pilots stop. Scope narrows until something works, and the narrowing gets recorded as success rather than as a finding.

A person who wants it to work is also operating it, rephrasing around failure modes they have learned to anticipate. Production has a naive user with no feedback signal telling them they are holding it wrong. Production also has concurrency, accumulated state, partial failures and retries with side effects, none of which exist in the demo environment, so the demo provides no evidence about them in either direction. And demos are short: at 95% per-step reliability, four steps complete 81% of the time and twenty complete 36%, so demo length alone accounts for much of the apparent regression.

The published figures are consistent in direction and soft in precision: 88% of proofs of concept never reaching production, 42% of companies abandoning most initiatives, 95% of enterprise pilots showing no measurable financial impact.

The correction is to make the demo harder rather than more impressive: sample inputs rather than choosing them, feed it the mess deliberately, have someone who did not build it drive, run the whole workflow rather than the interesting section, write the success metric before you start, and report the intervention count, which is the honest headline result and is never included.

Common questions

Why do AI demos work when production does not? Because a demo is the survivor of a search process. You adjusted the approach until something worked on the inputs you were testing with, so the artifact is the configuration that happened to succeed on that specific set. Its measured success rate estimates performance on the development inputs rather than on the real distribution, which is overfitting in the exact rather than metaphorical sense. Everything that made the demo work was selected for, usually without anyone noticing.

What percentage of AI proofs of concept reach production? IDC finds 88% never do, with roughly four of thirty-three surviving. S&P Global reports 42% of companies abandoning most AI initiatives in 2025, up from 17% the previous year, and an average of 46% of proofs of concept scrapped. MIT's 2025 enterprise study found 95% of generative AI pilots delivered no measurable profit-and-loss impact. The figures use different definitions and populations, so treat their agreement in direction as meaningful and their precision as approximate.

What gets left out of a demo? Four things, systematically. Clean data is chosen because messy records slow development. Edge cases are deferred, and the long tail is frequently where the value was, since the clean centre was already handled by existing tools. Integration is sidestepped by running against a copy or an export rather than the real system with its access model and rate limits. And scope narrows until something works, with the narrowing recorded as success rather than as a finding.

What is the watcher effect in AI demos? The person operating a demo wants it to succeed and has learned the system's shape. They pick good examples, rephrase when output is subtly wrong, and steer around failure modes before those manifest. Little of this is conscious. Production has a naive user who phrases things unexpectedly and, because the system responds plausibly whether or not it is correct, receives no signal that they are using it badly. The gap between expert operator and naive user is larger here than for most tools.

Why does workflow length matter so much? Because per-step reliability compounds. At 95% per step, a four-step demo completes about 81% of the time while a twenty-step production workflow completes about 36%. That is the same system measured at different lengths, not a regression. Demos typically show three or four steps and real workflows run to fifteen or twenty, so length alone accounts for a large share of the apparent drop.

How do I run a proof of concept that predicts production performance? Sample inputs randomly from real traffic rather than choosing them. Deliberately feed in incomplete, contradictory and out-of-scope material during the pilot rather than after. Have someone who did not build it drive, with no coaching, and measure the difference from the builder's success rate. Run the entire workflow rather than the interesting section. Write the success metric with a number and a baseline before starting, agreed by someone not building it. And count interventions, which is the honest headline result and is almost never reported.

Is a failed proof of concept wasted work? No, and the framing that treats it as such produces bad incentives. A failed pilot prevents you from scaling something that would not have worked, at a cost measured in weeks against production failures running to several million. It also surfaces data, access and governance gaps that needed addressing regardless. The problem is not that demos fail. It is that successful demos are read as evidence they are not.

What can a demo legitimately prove? That the capability exists at all, which a failure establishes cheaply and conclusively. That the integration is conceivable, since connecting the pieces even against a copy reveals whether they connect. Where the hard parts are, since the steps requiring the most iteration are where production will fail, though this information is usually discarded because it is not part of the presentation. And what the interface should be, since watching someone use it reveals what they expected it to do.