AI in software: 19% slower, and they felt 20% faster

TL;DR. Software engineering is the best-measured domain in this series, because the people building AI are also its subjects. A randomised controlled trial of 16 experienced developers across 246 real tasks in repositories they had maintained for an average of five years found they took 19% longer with AI tools. They had predicted 24% faster beforehand and reported 20% faster afterwards. The researchers expected a speedup and said so in the paper. Three findings sit on top of each other: a 39-point gap between felt and measured productivity, an arithmetic ceiling because writing code is only a quarter of the work, and organisational gains that stay near 10% at 90%-plus adoption. And the same group's follow-up found a speedup and published that it was unreliable.

---

In July 2025 a research organisation published the first randomised controlled trial of AI coding assistants. Sixteen experienced open-source developers, 246 real tasks, in repositories they had maintained for an average of five years. Randomly assigned, task by task, to work with AI or without.

They took 19% longer with it.

The researchers had expected the opposite and said so in the paper. They were broadly expecting positive speedup.

The number that matters more is what the developers believed. Beforehand they predicted AI would make them 24% faster. Afterwards, having completed the tasks and experienced the slowdown, they reported it had made them 20% faster.

A 39-point gap between what happened and what the people it happened to reported. Not a survey of people who had not tried it. Developers who had just done the work, on their own code, estimating their own performance in the wrong direction.

Why this domain is the strongest evidence in the series

Every other article here has an evidence problem. Medicine clears devices on resemblance. Education cannot blind its trials. Support is measured by parties who all want the same answer.

Software has three things none of the others do.

The subjects can read the study. Developers are technically capable of evaluating the methodology, and they did, at length and publicly. A weak study in this field gets taken apart by its own audience within days.

The work is instrumented by default. Commits, pull requests, review times, incident rates and deployment frequency are recorded as a byproduct of the tools, not because anyone set out to measure AI. That makes organisational telemetry available at a scale no other domain has: one analysis covers 22,000 developers across more than 4,000 teams.

And the people building the AI are the same people. The strongest claims about coding assistants come from firms whose engineers use them daily, which means the claims are checkable against the behaviour of the people making them.

So when this domain produces a negative result, it is worth more than a positive result from anywhere else in the series.

The three findings, stacked

They are usually reported separately and they compound.

One: the perception gap

The 39-point spread above. Developers cannot accurately estimate their own productivity with these tools, and the error is systematically optimistic.

The proposed mechanisms are worth naming because they are not about self-deception. Waiting is less unpleasant than thinking. Time spent reading a generated suggestion feels lighter than time spent working out the answer, even when it is longer. Fluency reads as progress. Code appearing on screen feels like advancement whether or not it is correct. And the failures are invisible in retrospect. A suggestion accepted, debugged for twenty minutes and then reverted does not feel like twenty minutes lost; it feels like part of the work.

This is the same finding as the education one, where students reported roughly twice the improvement they demonstrated. Self-reported benefit is a measure of experience, not of output, and the two diverge in a consistent direction.

Two: the arithmetic ceiling

The constraint nobody mentions in a vendor deck.

Writing code is roughly 25 to 35% of the software development lifecycle. The rest is requirements, design, review, testing, deployment, incident response, coordination and the meetings that attach to all of it.

Even a 100% speedup on the coding portion yields at most 15 to 25% improvement overall. That is Amdahl's Law applied to a working day, and it bounds the outcome regardless of how good the tool becomes.

Which means the ceiling on any coding assistant is set by the fraction of the job that is coding, and no model improvement moves it.

Three: the gains do not survive the organisation

The most consistently reported result across independent sources.

Teams with high adoption merge substantially more pull requests, one analysis putting the increase at 98%. Review time rose 91% in the same data, and DORA delivery metrics were unchanged across more than 10,000 developers.

That is a queue moving, not a system improving. More work arrives at the review stage, review becomes the bottleneck, and throughput at the end of the pipeline stays flat.

Six independent research efforts converge on roughly 10% organisational productivity gains at around 90% monthly adoption with a quarter or more of production code AI-generated. That is a real gain and it is an order of magnitude below what the adoption rate implies people expect.

The quality signal, which points the other way

Three measurements, from separate sources, all in the same direction.

Delivery stability declines as adoption rises. The DORA research programme reports AI adoption correlating with higher throughput and lower stability: more changes shipping, each slightly more likely to cause an incident. One figure puts pull requests per developer up 20% and incidents per pull request up 23.5%.

Code churn nearly doubled. Code rewritten or deleted within two weeks of being committed rose from 3.1% to 5.7% between 2020 and 2024 as AI assistance scaled. Churn is not automatically bad, and a doubling of code that does not survive a fortnight is a signal about what is being produced.

And trust is falling while usage rises. Adoption above 84% alongside 46% reporting distrust of the output is an unusual combination, and it suggests developers have calibrated to something the productivity claims have not.

The follow-up, and why it is the best thing in this literature

In February 2026 the same group published an update, and it is the most creditable act in this article.

The later results showed some evidence of speedup. They published that the estimate was unreliable.

The reason was selection. Developers were reluctant to participate if they might be assigned to work without AI, and some avoided submitting the tasks they most wanted AI for. So the follow-up systematically under-sampled exactly the people and problems where AI helps most, which biases the result in a direction nobody can quantify.

They changed the study design rather than publishing the more flattering number.

That is what evaluation integrity looks like, and it is rarer than any finding in this series. A group that found a result matching the industry narrative, identified a reason to distrust it, and said so has established more credibility than the original 19% figure did.

It also means the honest summary is narrower than either camp wants. The trial is strong evidence about one setting in early 2025. It is weaker evidence about tools now, and weaker still about what they become. Anyone quoting 19% as a current fact is overreading it, and anyone quoting the later speedup without the selection caveat is doing the same thing in the other direction.

What the spread actually looks like

Published results range from 26% faster to 19% slower, which sounds like chaos and is mostly a task-mix effect.

One large study of 4,500 developers found 46% time savings on routine tasks and under 10% on complex work. That single split explains most of the variance in the literature. A team writing boilerplate, tests and glue code will measure large gains. A team doing novel design in an unfamiliar system will measure little or none.

So the question "does AI make developers faster" has no answer. The answerable questions are which tasks, in which codebase, with what review capacity, measured how. A study reporting a large gain and a study reporting a loss can both be correct and usually are.

How to measure this in your own organisation

Six things, and the first is the one almost nobody does.

Do not use self-report. The 39-point gap is the finding. Surveys measure how the work felt, which is worth knowing and is not productivity.

Measure at the end of the pipeline, not the start. Pull requests opened is an input. Change lead time, deployment frequency, change failure rate and time to restore are outputs, and the evidence says the first moves while the last four do not.

Watch review capacity. If merged volume rises 98% and review time rises 91%, you have moved the constraint rather than removed it. Review is the bottleneck in most organisations adopting these tools and it is rarely resourced for the new volume.

Track churn. Code deleted or rewritten within two weeks is the cheapest available proxy for whether output is holding up.

Split by task type before drawing conclusions. Routine and complex work show a fourfold difference in measured savings. An aggregate across both tells you about your task mix, not about the tool.

And establish the baseline before you deploy. Almost every organisation adopting these tools has no pre-adoption measurement, which makes any subsequent number uninterpretable. That is the same failure as the domains with no evaluation at all, arriving in the domain best equipped to avoid it.

What is unresolved

Whether the perception gap closes with experience. Developers in the trial had used the tools before and still misestimated. Whether calibration improves over years, or whether the mechanisms producing the error are structural, is untested.

Whether the organisational ceiling is a transition. The Solow paradox in the 1980s took roughly a decade to resolve, and it resolved in ways nobody predicted. Ten percent at 90% adoption could be an early reading of a curve that steepens, or it could be the arithmetic ceiling doing what arithmetic does.

What happens to skill formation. Work has begun examining whether AI assistance impairs conceptual understanding, code reading and debugging. If the tools substitute for the practice by which those develop, the cost lands on people who have not yet acquired them and arrives years later.

And whether agentic coding changes the analysis. Systems that execute multi-step tasks autonomously are a different proposition from inline completion, and essentially none of the evidence above applies to them. The measurement will have to be redone.

The counter-argument

The trial is 16 developers. That is a small sample, on open-source work, in mature repositories the participants knew intimately, using tools from early 2025. It is the best-designed study available and it is one setting. Treating it as the verdict on a technology used by millions is more weight than the design supports, and the authors say so themselves.

Deep familiarity is the hardest case for AI. A developer who has maintained a codebase for five years has most of it loaded in their head. The gap between their unaided speed and their assisted speed is naturally smallest, and possibly negative, precisely because they need the least help. Most professional work happens in less familiar code.

Speed was never the whole claim. Developers report that these tools make work more pleasant, lower the friction of starting, help with unfamiliar APIs and remove tedium. Those are real benefits that a stopwatch does not capture, and a technology can be worth using while making you marginally slower.

And a 10% organisational gain is not a failure. It is a large return for a tool costing a few tens of dollars per developer per month. The finding is that it is 10% rather than the transformation the discourse implies, and reading "only 10%" as disappointment says more about the expectation than the result.

The short version

A randomised controlled trial put 16 experienced developers through 246 real tasks in repositories they had maintained for an average of five years. They took 19% longer with AI tools. They had predicted 24% faster and reported 20% faster afterwards, a 39-point gap between measured and felt productivity, from people who had just done the work on their own code. The researchers expected a speedup and published the opposite.

Three findings stack. The perception gap, which mirrors the education result where students reported roughly twice the improvement they demonstrated. An arithmetic ceiling, since writing code is 25 to 35% of the lifecycle, so even a 100% coding speedup yields at most 15 to 25% overall and no model improvement moves that bound. And organisational gains near 10% at 90%-plus adoption, with six independent efforts converging there.

The mechanism for the third is visible in the data: merged pull requests up 98%, review time up 91%, delivery metrics unchanged across 10,000 developers. That is a queue moving, not a system improving.

Quality signals point the same way. Delivery stability falls as adoption rises, incidents per pull request up 23.5%. Code churn, meaning code rewritten or deleted within two weeks, nearly doubled from 3.1% to 5.7%. Adoption above 84% coexists with 46% distrust.

And the best thing in this literature is the follow-up. The same group later found evidence of a speedup and published that the estimate was unreliable, because developers reluctant to work without AI self-selected out and tasks people most wanted AI for went unsubmitted. They changed the design rather than shipping the flattering number. That is worth more than either result.

Which makes the honest summary narrower than either side wants. The spread across studies runs from 26% faster to 19% slower, and one study of 4,500 developers explains most of it: 46% savings on routine tasks, under 10% on complex ones. There is no answer to whether AI makes developers faster. There are only answers about which tasks, in which codebase, with what review capacity, measured how.

Common questions

Does AI actually make developers faster? It depends on the task, and the strongest evidence is uncomfortable. A randomised controlled trial of 16 experienced developers on 246 real tasks in their own repositories found they took 19% longer with AI tools. A separate study of 4,500 developers found 46% time savings on routine tasks and under 10% on complex work, which explains most of the spread in the literature. There is no general answer, only answers about specific task types.

What was the METR study and why does it matter? The first randomised controlled trial of AI coding assistants, published July 2025. Sixteen experienced open-source developers were randomly assigned, task by task, to work with or without AI across 246 real tasks in repositories they had maintained for about five years. They took 19% longer with AI. It matters because the researchers expected a speedup, said so in the paper, and published the opposite, and because randomised assignment removes the selection effects that make survey evidence unreliable.

Why do developers think AI makes them faster when it does not? Three mechanisms, none of which is self-deception. Waiting for and reading a suggestion feels lighter than working the answer out yourself, even when it takes longer. Code appearing on screen reads as progress whether or not it is correct. And a suggestion accepted, debugged and then reverted does not register as lost time, it registers as part of the work. The same pattern appears in education, where students report roughly twice the improvement they demonstrate.

Why don't individual productivity gains show up at company level? Because writing code is only 25 to 35% of the development lifecycle, so even doubling coding speed yields at most 15 to 25% overall. And because faster code production moves the bottleneck rather than removing it: one analysis found merged pull requests up 98% with review time up 91% and delivery metrics unchanged across 10,000 developers. Six independent research efforts converge on roughly 10% organisational gains at 90%-plus adoption.

Does AI-generated code have more bugs? The evidence points that way without being conclusive. The DORA research programme reports AI adoption correlating with higher throughput and lower delivery stability, with one figure showing incidents per pull request up 23.5%. Code churn, meaning code rewritten or deleted within two weeks of being committed, nearly doubled from 3.1% to 5.7% between 2020 and 2024 as AI assistance scaled.

Did the METR follow-up study reverse the finding? Partly, and the handling is the important part. A later run found evidence of a speedup, and the researchers published that the central estimate was unreliable because of selection effects: developers were reluctant to participate if they might have to work without AI, and some avoided submitting the tasks they most wanted AI for, so the study under-sampled exactly where AI helps most. They changed the design rather than publishing the more flattering number.

How should a company measure whether AI coding tools are working? Not by asking. The 39-point gap between measured and reported productivity is the central finding. Measure at the end of the pipeline rather than the start, since pull requests opened is an input while change lead time, deployment frequency, change failure rate and time to restore are outputs. Watch review capacity, since that is where the new volume lands. Track code churn. Split results by task type. And establish a baseline before deploying, which almost nobody does.

Is AI going to replace software engineers? Nothing in the measured evidence supports that reading. At roughly 90% adoption with a quarter or more of production code AI-generated, organisational productivity has moved about 10%, delivery stability has declined slightly, and review has become the binding constraint. The observable change is in what the work consists of, with more time reviewing and less writing, rather than in how many people are needed to do it.