61% once, 25% eight times running

TL;DR. τ-bench introduced pass^k, the probability an agent succeeds on all k independent attempts at the same task. On retail agent tasks, GPT-4o scored 61% on a single attempt and 25% across eight. That gap is the finding. Pass@k, the metric everyone quotes, counts a task as solved if any one of k attempts works, which describes a developer retrying and not a customer. Real users do not get to retry until it works. Reported pass^4 scores commonly run 15 to 25 points below pass^1, and one analysis puts a 90% benchmark score at roughly 70% production reliability. And the composition is worse than the average. A model succeeding half the time with a consistent strategy and one succeeding half the time by working perfectly then failing catastrophically produce the same score and are different products.

---

Status: established, and the primary sources are open. Yao and colleagues, τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv:2406.12045, which introduced pass^k. Subsequent work extends it to duration and to divergence analysis. Benchmark leaderboard figures are scaffold-dependent and are labelled where used.

---

The two metrics

pass@k counts a task as solved if at least one of k attempts succeeds. It is the standard in coding benchmarks and it is an optimistic measure.

pass^k counts a task as solved only if all k attempts succeed. It was introduced by τ-bench and it is a strict measure of consistency.

The difference is who is allowed to retry.

Pass@k describes a developer with a loop. Run it five times, take the one that worked, ship that. For code generation with a test suite, that is a reasonable description of the workflow and the metric fits it.

Pass^k describes a customer. One attempt, live, on their account, with real consequences. They do not see the four runs that failed, and if the run they got was one of those, the product failed.

Almost every agent benchmark reports the first and almost none reports the second.

The number

On τ-bench retail tasks, GPT-4o achieved 61% pass^1 and 25% pass^8.

Succeeding six times out of ten on one try, and succeeding eight times running on a quarter of tasks.

Both figures describe the same model on the same tasks in the same evaluation. The first is the one that appears in comparisons.

Related work reports the same shape at smaller k: pass^4 scores commonly running 15 to 25 points below pass^1, and one analysis translating a 90% benchmark result into roughly 70% reliability in production, where the same task is attempted by different sessions.

Even top models score below 50% success on these tasks and fall under 25% at pass^8, which is a summary worth holding against any claim that agent reliability is a solved problem in customer-facing deployment.

Why the gap is bigger than intuition suggests

Multi-step tasks multiply.

An agent succeeding on 95% of individual steps completes a twenty-step task 36% of the time, if the steps are independent. At 90% per step it is 12%. Nothing is broken in that model. The arithmetic is simply unforgiving, and per-step accuracy is what benchmarks and demonstrations show.

Steps are not independent in practice, which cuts both ways: a good agent recovers from an early error, and a bad one compounds it. Recovery behaviour is the thing that matters and almost nothing measures it.

One analysis traced divergence to specific decision points, finding that runs of the same task separate early, around the second step, and that path length predicts inconsistency. An agent that takes a longer route is more likely to disagree with itself, which is a usable diagnostic and is not in any leaderboard.

What the score conceals

This is the part that should change how a benchmark result is read.

Two models both score 50% on a two-hour task. The first has a consistent strategy that works half the time. The second succeeds completely on some runs and fails catastrophically on others.

Identical scores. Completely different products.

Pass@1 cannot distinguish them and neither can any mean. What separates them is variance, which requires repeated runs to measure and which most evaluations do not perform because repeated runs cost k times as much.

The METR long-horizon work is the clearest case. It measures the task duration at which a model's pass@1 drops to 50%, and found that horizon doubling every seven months from 2019 to 2024, which is a striking capability result. It does not analyse variance across repeated runs, so it cannot tell you which of those two models it measured.

That is not a criticism of the work. It is a statement about what a single-attempt metric can support, and the answer is capability rather than dependability.

Three other things benchmarks leave out

Cost is absent from the primary scoring of every major agent benchmark surveyed. Zero out of fifteen integrate it.

A score of 88% achieved with $50 of inference per task is treated as identical to one achieved with $0.50. For a procurement decision those are not comparable numbers, and teams end up composing the cost layer themselves by dividing pass rate by dollars per task.

Partial credit is absent from thirteen of fifteen, which use binary success. An agent that completes 90% of a workflow and stops cleanly scores identically to one that completes nothing, and in production those outcomes are worth very different amounts because one leaves a human finishing a task and the other leaves them starting it.

And graceful failure is unscored everywhere. An agent that recognises it is out of depth and hands off is strictly better than one that proceeds confidently and corrupts state, and no primary rubric distinguishes them.

Scaffolding, and which number you are reading

Agent benchmark scores are heavily scaffold-dependent. Model, tool access, retry budget and evaluator version all materially change the reported figure.

One useful framing separates three numbers. The model number tells you about the language model. The scaffolded number tells you about a vendor's product around it. The system number tells you about an integrator's whole stack.

These are routinely compared as though they were the same measurement.

WebArena illustrates it. Scores moved from a 14.41% baseline to 61.7% by early 2025, and the analysis attributes the gain to modular planner-executor-memory architectures rather than to a single model breakthrough. The models improved and the scaffolding improved more.

Which means a leaderboard position is a statement about an assembled system, and the part of it you can buy is often not the part that produced the score.

Where the benchmarks still say the gap is wide

Worth stating, because the reliability point can read as though capability were settled.

OSWorld tests real computer use across 369 cross-application tasks and opened with a 60-point gap between human and machine performance.

ARC-AGI-1 is saturated above 90%. ARC-AGI-2 has a leader at 77.1% as of February 2026. ARC-AGI-3 launched in March 2026 and all frontier systems score below 1%.

Those are capability measurements, they are single-attempt, and the reliability discount applies to them too.

The honest summary is that capability is improving quickly on tasks with clear structure, that reliability lags capability by a large and mostly unreported margin, and that a benchmark result tells you what a system can do rather than what it will do.

What the numbers look like across k

The decay is worth tabulating, because a single figure at k=1 tells you nothing about its shape.

Take a task an agent solves 80% of the time on a single attempt, and assume attempts are independent, which is the optimistic case.

Nothing in that table is a failure of the agent. It is 80% repeated, and 80% is a good single-attempt score.

The table also explains a pattern most teams encounter and few name. A pilot goes well, because a pilot is a small number of attempts watched closely by people who want it to work. Production goes badly, because production is the right-hand column.

And it explains why measured pass^8 of 25% against pass^1 of 61% is not anomalous. Independent attempts at 61% would give 2.1% at k=8. The observed 25% is far better than independence predicts, which means the agent's successes and failures are correlated across runs: some tasks it reliably solves and some it reliably does not.

That correlation is good news and it is the more useful finding. An agent with correlated outcomes has a subset of tasks it handles dependably, which can be identified and routed to it, while independence would mean no subset is safe.

Which points at the deployment strategy the numbers actually support: measure pass^k per task type, route the reliable ones, escalate the rest, and stop treating an aggregate score as a property of the agent rather than of the task mix.

The measurement enterprises could do and do not

Every figure in this article comes from a research benchmark. None comes from a deployment.

That is remarkable given who is running agents. Reported figures put more than 80% of the Fortune 500 with agents in production, and every one of those deployments logs its own outcomes.

Computing pass^k in production requires only running the same task twice, which any organisation with a staging environment can do, and comparing outcomes. Nobody publishes it.

The reasons are ordinary rather than sinister. A poor internal reliability figure is commercially awkward, no regulation requires disclosure, and the number would be read as a statement about the vendor when it is substantially a statement about the integration.

But the consequence is the same one this corpus has found in every territory. The party with the data has no obligation to publish, so the public evidence base consists of benchmarks built by researchers on tasks they chose, and the gap between that and deployment is the thing everyone most wants to know.

One specific disclosure would change the field more than any new benchmark: an enterprise publishing pass^k by task category for a production agent over a quarter, with the task mix described. It costs a rerun and an afternoon, and it would be the first datapoint of its kind.

Three things this establishes

The reported metric answers the developer's question, not the user's. Pass@k assumes retries. A customer gets one attempt, and the metric describing that outcome exists, is published in the same paper, and is almost never quoted.

Variance is the missing measurement and it is expensive to obtain. Distinguishing a consistent middling agent from a bimodal one requires k runs and therefore k times the cost, which is why almost nothing reports it. The reason is economic rather than methodological, which means it will not fix itself.

And a leaderboard number is a property of a system, not of a model. Scaffolding moved WebArena further than model improvements did over the same period. Buying the model does not buy the score.

What it does not establish

That agents do not work. τ-bench measures a hard class of task, and the same period shows large capability gains on structured problems.

That pass^k is the right production metric. It is stricter than most deployments require, since many workflows do tolerate a retry, and the appropriate k depends on the application.

That benchmark authors are at fault. τ-bench published pass^k itself. The gap is between what was measured and what gets quoted.

And nothing about any specific vendor's product. Every figure here is a research benchmark, not a deployment measurement.

What this does to the pricing question

The previous article established that customer service agents are now billed per resolution, at rates from $0.50 to $2.00, with the definition of a resolution set by the vendor.

Put the two findings together and a question appears that neither raises alone: who pays for the attempts that fail?

Under per-conversation billing the answer is the buyer, explicitly. Agentforce's model charges for a 24-hour conversation window whether or not anything resolved, so a 61% success rate means 39% of the bill buys nothing.

Under per-outcome billing the answer is meant to be the vendor, which is the whole appeal. A failed attempt is unbilled, so reliability is the vendor's problem.

Except that an assumed resolution converts a specific class of failure back into a billable event. A customer who received an unhelpful answer and gave up is silent for 24 hours, which is the same signal as success, and the pass^k literature says that customer exists in substantial numbers.

So the two mechanisms interact in a way neither article can quantify. The rate at which agents fail is measured on benchmarks. The rate at which failures produce silence rather than escalation is measured by nobody. The product of those two numbers is the fraction of an outcome-priced invoice that is buying failure, and it is unknown.

The useful move is not indignation. It is that the pass^k question and the assumed-resolution question are the same question asked from two sides, and an organisation that can measure one can measure the other. Rerun a sample of tasks, count how often the outcome differs, and compare that to the share of billed resolutions with no confirming signal.

Both numbers are computable from logs an enterprise already holds, and neither is published by anyone.

What is unresolved

What pass^k looks like on current frontier models. The widely cited 61% to 25% figure is from GPT-4o, independent reproductions for newer models are partial, and one reliability dashboard reports an airline figure without a matching retail one.

Whether recovery behaviour can be scored. An agent that errs and corrects is the thing production needs, and no primary rubric measures it.

How variance behaves with task duration. One line of work extends pass^k across duration buckets and the results are early.

And what production reliability actually is. Every number in this article is a benchmark. No enterprise publishes its agents' pass^k, and they all have the logs to compute it.

The counter-argument

Pass^k may be too strict to be the headline. Requiring eight consecutive successes is a bar most software does not meet and most workflows do not need. A support agent that fails one attempt in five and escalates cleanly is a usable product, and scoring it at 25% describes a failure mode nobody experiences.

Benchmark tasks are adversarially selected. τ-bench is built around policy compliance and stateful tools precisely because those are hard, so its numbers describe the difficult end of the distribution rather than typical deployment work.

The compounding arithmetic assumes independence and steps are not independent. Real agents observe results, notice errors and adjust. Presenting 0.95 to the twentieth power as the expected outcome overstates the problem, and this article does exactly that before qualifying it.

And the cost objection cuts against the corpus's own preference. Demanding that benchmarks report pass^k, cost per task, partial credit and graceful failure multiplies evaluation expense at a time when evaluation is already the bottleneck. A cheaper imperfect benchmark that many teams run may produce more total information than an expensive rigorous one that few do.

The short version

τ-bench introduced pass^k, the probability an agent succeeds on all k attempts. On retail agent tasks GPT-4o scored 61% pass^1 and 25% pass^8, and both numbers describe the same model in the same evaluation. Only one of them circulates.

The difference is who retries. Pass@k describes a developer running the task five times and keeping the good one. Pass^k describes a customer, who gets one attempt and does not see the failures.

The arithmetic is unforgiving. 95% per-step accuracy over twenty independent steps completes 36% of the time; at 90% it is 12%. Steps are not independent, which is why recovery behaviour matters and why nothing measures it.

And the mean conceals the composition. Two models scoring 50% can be a consistent middling performer and a bimodal one that either sails through or destroys state. Identical scores, different products, and no single-attempt metric can separate them.

Three further gaps. Zero of fifteen major agent benchmarks include cost in primary scoring, so 88% at $50 a task ranks with 88% at $0.50. Thirteen of fifteen use binary success, so finishing 90% of a workflow scores as zero. And graceful failure is unscored everywhere, though an agent that hands off cleanly is strictly better than one that proceeds and corrupts state.

Finally, the score belongs to a system. WebArena moved from 14.41% to 61.7% largely through planner-executor-memory scaffolding rather than model improvement. Buying the model does not buy the number.

Common questions

What is the difference between pass@k and pass^k? pass@k counts a task as solved if at least one of k attempts succeeds, which is an optimistic measure standard in coding benchmarks. pass^k, introduced by τ-bench, counts a task as solved only if all k independent attempts succeed. The difference is who is permitted to retry: pass@k describes a developer running something five times and keeping the successful run, while pass^k describes a customer who gets one attempt and never sees the failures.

What is the headline reliability number? On τ-bench retail agent tasks, GPT-4o achieved 61% on a single attempt and 25% across eight consecutive attempts. Both figures come from the same evaluation of the same model, and the first is the one that appears in comparisons. Related analyses report pass^4 running 15 to 25 points below pass^1, and one puts a 90% benchmark score at roughly 70% reliability in production.

Why do multi-step tasks fail more than expected? Because errors multiply. An agent succeeding on 95% of individual steps completes a twenty-step task 36% of the time if steps are independent; at 90% per step it is 12%. In practice steps are not independent, which cuts both ways: a capable agent recovers from an early mistake and a weaker one compounds it. Recovery behaviour is what determines the outcome and almost nothing measures it. One analysis traced divergence to early decision points, around the second step, and found path length predicts inconsistency.

What does a mean score conceal? The composition of the failures. Two models both scoring 50% on a long task can be completely different products: one with a consistent strategy that works half the time, and one that succeeds fully on some runs and fails catastrophically on others. No single-attempt metric distinguishes them, and separating them requires repeated runs, which cost k times as much. That expense is why variance is rarely reported, which makes it an economic problem rather than a methodological one.

What else do agent benchmarks leave out? Three things. Cost is absent from the primary scoring of every major agent benchmark surveyed, so 88% achieved at $50 per task ranks identically with 88% at $0.50. Thirteen of fifteen use binary success, so an agent finishing 90% of a workflow and stopping cleanly scores the same as one finishing nothing. And graceful failure is unscored everywhere, although an agent that recognises it is out of depth and hands off is strictly better than one that proceeds confidently and corrupts state.

Why does scaffolding matter? Because a leaderboard score is a property of an assembled system rather than of a model. Model choice, tool access, retry budget and evaluator version all materially change the number. WebArena scores moved from a 14.41% baseline to 61.7% by early 2025, and that gain is attributed largely to modular planner-executor-memory architectures rather than to a single model breakthrough. It is worth separating the model number, the vendor's scaffolded number and an integrator's system number, since they are routinely compared as though identical.

Is capability improving? Yes, quickly, on tasks with clear structure. METR measured the task duration at which a model's single-attempt success drops to 50% and found that horizon doubling roughly every seven months from 2019 to 2024. At the same time OSWorld opened with a 60-point human-machine gap across 369 cross-application tasks, and ARC-AGI-3, launched in March 2026, has all frontier systems below 1%. Capability is rising and reliability lags it by a large and mostly unreported margin.

Is pass^k too strict? Arguably, and this is the strongest objection. Requiring eight consecutive successes is a bar most software does not meet and most workflows do not need, and a support agent that fails one attempt in five and escalates cleanly is a usable product that pass^8 scores harshly. The appropriate k depends on the application. What the metric does establish is that a single-attempt score systematically overstates dependability, and the size of the overstatement is only visible if somebody runs the task more than once.