What to measure before you deploy an agent

Enterprise deployments report agents achieving around 60% success on a single run of a task. Run the same task eight times and the proportion that succeed on every attempt drops to about 25%.

Both numbers describe the same agent. The first is what gets reported. The second is what a user experiences when they need the thing to work.

Task completion is the metric every team reports and it is wrong in three separate ways: it is usually measured on what the agent claims rather than what happened, it is measured once rather than repeatedly, and it is aggregated with other scores in a way that hides the dimension most likely to end you. Fixing all three costs a day and changes what you learn.

Wrong one: measuring the claim instead of the end state

An agent completes a booking task. The final message says the flight is booked. Scored on that message, the run passes.

The flight was not booked.

This is not hypothetical, and it is common enough that benchmarks have been built specifically to avoid it: rather than reading the agent's summary, they check the database. Did a record appear. Is its state what was requested. The agent's account of its own work is not evidence about its work, and treating it as such passes a system that confidently narrates actions it did not take.

The generalisation is straightforward and almost nobody applies it. Success must be verified against the state of the world the agent was supposed to change, not against the agent's description of it. If your agent files tickets, query the ticket system. If it updates records, read the records. If it sends messages, check the outbox.

Where the task has no verifiable end state, which is true of drafting, summarising and advising, you have a different measurement problem and should say so rather than substituting a self-report.

Wrong two: measuring once

A single-run success rate is a point estimate of a non-deterministic system, and agents are strongly non-deterministic: the same input produces different execution paths, different tool calls and different outcomes.

The gap between 60% on one run and 25% across eight is the entire subject of reliability, and reporting only the first is how a team convinces itself a system is ready.

Two derived measures are worth more than the headline figure.

Variance across runs. Standard deviation of success on repeated trials of identical tasks. A system at 70% with low variance is predictable and can be designed around. A system at 70% with high variance is two different systems, one that works and one that does not, and the user cannot tell which they will get.

Consistency at the depth the user needs. If a user will retry once, measure success across two runs. If the workflow is unattended and must work first time, measure exactly that. Reporting single-run success for an unattended workflow measures a condition that does not occur.

The uncomfortable implication: to measure this you must run each task multiple times, which multiplies evaluation cost. Most teams do not, and the reason they do not is that it is expensive rather than that it is unnecessary.

Why consistency and accuracy come apart

The distinction between a 60% single-run rate and a 25% eight-run rate is worth working through, because the arithmetic reveals something the headline hides.

If failures were independent, an agent succeeding 60% of the time would succeed on all eight runs about 1.7% of the time. The observed figure is around 25%, which is far higher. That tells you the failures are not independent: some tasks the agent handles reliably every time, and others it fails on almost every time.

That is a much more useful finding than a single average, and it changes what you should do.

An agent with independent failures is a reliability problem. Every task is a coin flip and the fix is making the model or the workflow better across the board.

An agent with clustered failures is a scoping problem. There is a subset of tasks it handles well and a subset it does not, and the fix is identifying the boundary and routing the second subset elsewhere. That is far cheaper than improving the system, and it is invisible in an aggregate score.

The diagnostic is simple. Run each task in your evaluation set five times and sort by how many runs succeeded. If you get a bimodal distribution, mostly fives and mostly zeros, you have a scoping problem and should be looking at what distinguishes the two groups. If you get a spread clustered around the middle, you have a genuine reliability problem and no amount of routing will fix it.

Most teams never see this because they run each task once and average.

Wrong three: aggregating

A team scores an agent across dimensions and gates on the average. The agent passes at 0.80 overall: 0.95 on task completion, 0.90 on instruction adherence, 0.50 on error recovery.

It fails on its first API timeout.

Aggregate thresholds let a strong score in one dimension purchase a fatal weakness in another, and error recovery is the dimension most often traded away because it is the hardest to test and the least visible in a demo. The fix is unglamorous: per-dimension thresholds, each blocking independently. An agent that recovers badly does not ship regardless of how well it performs when nothing goes wrong.

The dimensions worth gating separately are a short list, and the exact set matters less than that they are separate.

Three layers, answering different questions

Evaluation splits into layers that are frequently conflated, and each answers a question the others cannot.

Final answer. Score the last output against an expected result. This is what every benchmark measures and what most teams build first. It is necessary and it is the least informative layer, because a run can produce the right answer through a path you would never accept.

Trajectory. Score the sequence: which tools were called, in what order, with what arguments, and how failures were handled. This is where the signal for improvement lives, because an agent that reaches the right answer in nineteen steps when three would do has a problem the final answer cannot show. Step and loop counts are invisible in a total: a three-step turn and a twenty-step turn both return one answer, and you only see the difference if you attribute cost and time per step.

Per-turn in production. Score individual turns on live traffic. This is the only layer that observes what real users actually do, and it is where the failures your test set never imagined appear.

A held-out benchmark covers the first layer. The production signal comes from the third. Teams that build only the first have a regression guard and no visibility, which is a common and expensive place to be.

The short list that predicts production behaviour

More metrics is not better; a set nobody maintains is worse than four that are watched. These are the ones that have repeatedly turned out to matter.

Verified task completion. Against the end state, per the first section.

Consistency at the required depth. Success across the number of runs your workflow actually gets.

Step efficiency. Actual steps against the minimum viable path for that task class. Rising step counts are the earliest signal of planning degradation and they show up before success rates move.

Error recovery. Deliberately fail a tool and observe. Does the agent retry sensibly, escalate, or loop. This is the dimension that separates systems that survive contact with real infrastructure from those that do not, and it cannot be measured without injecting failures.

Cost per completed task, and its variance. The average hides the tail, and the tail is where the budget goes.

Intervention rate. How often a human steps in. Rising intervention is degradation that no automated metric will show you, and falling intervention may mean improvement or may mean the reviewers stopped looking.

Blast radius. Not a score but a bounded fact: what is the worst thing this agent can do with the credentials it holds. If nobody has written this down, it has not been assessed.

Abstention rate. How often the agent declines. A rate of zero means it never declines, which means it answers everything including what it cannot know.

The five dimensions, and what each one is actually asking

Frameworks converge on a similar set, and the useful thing is not the taxonomy but the question each dimension answers, because a dimension you cannot phrase as a question is one you will not measure.

Accuracy: does it produce the right outcome? Verified against the world, not the transcript. The dimension everyone measures and the only one most teams measure.

Efficiency: does it get there sensibly? Steps against the minimum, tokens against the task, latency against what a user will wait. An agent taking nineteen steps to do three steps of work is failing even when it succeeds, and the failure compounds with cost.

Resilience: what happens when something breaks? Tool timeouts, malformed responses, rate limits, partial failures. Untestable without deliberate injection, which is why it goes unmeasured and why it is the most common production failure.

Safety: what can it do that it should not? Two halves that get conflated. Whether it refuses harmful requests, which is model behaviour, and whether it can be manipulated into acting through its tools, which is architecture. Benchmarks exist for both, and the second matters more for anything with credentials.

Experience: is the output usable by the person receiving it? Not accuracy. A correct answer in the wrong format, at the wrong length, or arriving after the user gave up is a failed interaction that every accuracy metric records as a success.

The reason to separate them is the gating argument from earlier. An agent can be excellent on accuracy and unacceptable on resilience, and the aggregate will not tell you. Each dimension needs its own threshold and its own veto.

The reason to phrase them as questions is that it exposes which ones you have no answer for. Most teams reading that list can answer the first and have never asked the third or the fifth.

What benchmarks cannot tell you

Public agent benchmarks are useful for comparing models and poor for predicting your deployment, and it is worth being specific about why.

The gap is measured. Research examining the difference between benchmark performance and real-world deployment has found gaps around 37%. That is not noise; it is a systematic difference in conditions.

Saturation destroys discrimination. Once frontier models cluster above 88 to 90% on a benchmark, differences between them stop being statistically meaningful while continuing to be reported as if they were. Several of the most-cited benchmarks have reached this point.

The benchmarks themselves have problems. A University of California group examining eight prominent agent benchmarks, including several that drive published progress claims, found substantial issues with how they are constructed. When the instruments have problems the scores inherit them, and a leaderboard position is a weaker claim than it appears.

And the task distribution is not yours. A benchmark measures a fixed distribution under controlled conditions. Your inputs are messier, your tools flakier, your users stranger. The published number describes the benchmark.

Use them to narrow a shortlist. Do not use them to predict your outcome.

What actually causes production failure

Worth stating because it redirects where measurement effort should go. Analyses of production failures attribute roughly 60% to data quality, context, and governance rather than to model limitations.

That means most of what breaks is upstream or around the model rather than in it. The measurement implication is direct: instrumenting model output while leaving retrieval quality, context assembly and permission scope unmeasured watches the smallest of the three contributors.

It also explains a pattern from the deployment surveys, where 78% of enterprises report agent pilots and fewer than 15% reach production scale. If the dominant failures were model capability, better models would have closed that gap by now. They have not, because the gap is elsewhere.

Before you deploy: the shortest useful checklist

Can you verify success without asking the agent? If not, you cannot measure completion.

Have you run each test task at least five times? If not, you have a point estimate of a stochastic system.

Do your gates block per dimension? If they aggregate, one weak dimension is currently purchasable.

Have you injected tool failures? If not, error recovery is untested and it is the most common production failure.

Can you attribute cost and steps per turn? If not, loops are invisible until the invoice.

Is there a held-out set the production pipeline never touches? If approved outputs feed retraining and also feed evaluation, your measurements are contaminated.

Do you know the blast radius? If nobody has written down the worst outcome, it has not been assessed.

Seven questions. A team that can answer all seven is in a small minority, and the answers take a day to establish rather than a quarter.

What is unresolved

How to score multi-step outcomes properly. Scoring the final state is tractable. Scoring a forty-step run where step nineteen was wrong but recovered and step thirty-one was subtly wrong and not recovered is not, and no accepted methodology exists. Current practice scores the endpoint, which cannot distinguish a system that got there reliably from one that got there by luck.

Whether trajectory matching is the right idea at all. Comparing an execution path against an expected sequence assumes there is a correct path. For open-ended tasks there are many acceptable paths, and penalising divergence penalises legitimate variation. Any-order and partial matching are attempts to soften this and none is principled.

Whether model-based scoring can be trusted at scale. Using a model to judge agent behaviour scales in a way human review does not, and its errors correlate with the errors of the system under test, since both share training data and failure modes. That correlation is worst exactly where independence would matter.

What consistency target is appropriate. Nobody has established what success-across-N-runs figure is required for a given class of workflow. Teams pick a threshold and defend it afterwards, which is not a standard.

The counter-argument

Measurement has a cost and it competes with shipping. Running every task five times multiplies evaluation spend, injecting failures takes engineering, and per-dimension gates block releases that an aggregate would pass. For a low-stakes internal tool this discipline is overhead, and the correct answer for many deployments is to measure less and watch production closely.

The numbers quoted here are soft. Figures for single-run versus multi-run success, the benchmark-to-deployment gap, and the attribution of failures to data rather than models come from industry surveys and vendor research with varying methodology. They agree in direction, which is meaningful. Their precision should not be relied on, and several come from companies selling evaluation tooling.

And benchmarks are not worthless. The critique above is aimed at using them as deployment predictors. As instruments for comparing models on a fixed task, tracking field progress, and providing a shared reference point, they do work that nothing else does, and a field without them would be worse at knowing anything.

The short version

Agents reported at 60% success on a single run drop to around 25% success across eight runs of the same task. Both figures describe the same system; the first is what gets reported and the second is what an unattended workflow experiences.

Task completion, the standard metric, is wrong in three ways. It is usually scored on the agent's final message rather than the state of the world, which passes a system that says it booked the flight without booking it, so success must be verified against the end state and not the description. It is measured once, when the systems are strongly non-deterministic, so variance across repeated runs and consistency at the depth the workflow actually gets matter more than the headline. And it is aggregated, so an agent scoring 0.95 on completion and 0.50 on error recovery clears a 0.80 gate and then fails at its first timeout, which is why thresholds must block per dimension.

Evaluation has three layers answering different questions: final answer, which every benchmark measures and which is least informative; trajectory, where the improvement signal lives because a twenty-step and a three-step run return the same single answer; and per-turn on production traffic, which is the only layer that sees what users actually do.

Public benchmarks are for narrowing a shortlist rather than predicting outcomes. Measured gaps between benchmark and deployment run around 37%, saturation above 88 to 90% makes leaderboard differences statistically meaningless, and an examination of eight prominent agent benchmarks found substantial construction problems.

The redirection worth acting on: roughly 60% of production failures are attributed to data quality, context and governance rather than model limitations, which means instrumenting model output while leaving retrieval, context assembly and permission scope unmeasured watches the smallest of the three contributors.

Common questions

What should I measure before deploying an AI agent? Verified task completion against the end state rather than the agent's claim. Consistency across the number of runs the workflow actually gets. Step efficiency against the minimum viable path. Error recovery under injected tool failures. Cost per completed task and its variance. Intervention rate. Blast radius, meaning the worst thing the agent can do with its credentials. And abstention rate, since a rate of zero means it never declines anything.

Why is task completion the wrong metric? Three reasons. It is usually scored on the agent's final message, which passes a system that claims to have booked a flight it did not book, so completion must be verified against the state of the system the agent was meant to change. It is measured once on a non-deterministic system, hiding the difference between 60% on one run and 25% across eight. And it is typically aggregated with other scores, letting a strong completion figure purchase a fatal weakness elsewhere.

How many times should I run each evaluation task? At least five, and ideally the number of attempts your workflow actually gets. If a user will retry once, measure success across two runs. If the process is unattended and must work first time, measure exactly that. Reporting single-run success for an unattended workflow measures a condition that never occurs. The cost is that evaluation spend multiplies, which is why most teams skip it.

What is trajectory evaluation? Scoring the sequence of an agent's execution rather than its final output: which tools were called, in what order, with what arguments, and how failures were handled. It matters because an agent reaching a correct answer in nineteen steps when three would do has a problem invisible in the answer, and because step and loop counts do not appear in a total unless cost and time are attributed per step.

Why should evaluation gates not use an aggregate score? Because aggregation lets one dimension subsidise another. An agent scoring 0.95 on task completion, 0.90 on instruction adherence and 0.50 on error recovery passes a 0.80 aggregate gate and then fails on its first API timeout. Error recovery is the dimension most often traded away, since it is hardest to test and invisible in a demo. Per-dimension thresholds, each blocking independently, prevent the trade.

Can I rely on public agent benchmarks? For narrowing a shortlist, yes. For predicting your deployment, no. Measured gaps between benchmark performance and real-world outcomes run around 37%. Saturation above 88 to 90% accuracy makes differences between leading models statistically meaningless while they continue to be reported. And an academic examination of eight prominent agent benchmarks found substantial problems in how they are constructed, which the scores inherit.

How do I test error recovery? By causing failures deliberately. Make a tool return an error, a timeout, or malformed output, and observe whether the agent retries sensibly, escalates, or loops. This cannot be measured passively, because a test environment where nothing fails produces no evidence about failure handling. It is the most common production failure and the least commonly tested dimension.

What causes most agent production failures? Not model capability. Analyses attribute roughly 60% to data quality, context and governance issues. This is consistent with the deployment pattern where 78% of enterprises report pilots and fewer than 15% reach production scale: if capability were the constraint, improving models would have closed that gap. The measurement implication is that retrieval quality, context assembly and permission scope deserve at least as much instrumentation as model output.