A/B Testing
Showing two versions to two random groups and measuring — the only method that tells you whether your model actually helped anyone.
Reviewed July 12, 2026Stable
When not to use it
- When you can't randomise. Pricing, legal constraints, one-off launches. Quasi-experimental methods exist and are weaker.
- With network effects, naively. If treatment affects control, randomisation is broken.
- For long-term effects. It measures weeks. Retention and trust take months and get confounded.
- When you lack the traffic. An underpowered test returns a null you'll misread as "it doesn't work."
Reach for something else instead
- Offline evaluation — a filter, not a decision. Necessary and insufficient.
- Interleaving — for ranking, mix both systems' results and see what's clicked. Far more sensitive per user.
- Quasi-experiments — difference-in-differences, regression discontinuity. When randomisation isn't available.
- Shadow deployment — run the new model without acting on it, compare. Safe, and it measures agreement rather than outcome.
Read more on the blog
- AI in software: 19% slower, and they felt 20% fasterA randomised trial put experienced developers 19% behind on their own repositories while they reported being 20% ahead. The researchers had expected a speedup and published the opposite.
- How to read an AI paperA meaningful fraction of state-of-the-art results, in the highest-prestige venues, could not be reproduced from the published artifact. The failures were structural and they are visible from the outside if you know where to look.
- Why machine learning does not do error barsModels differing only by random seed showed 0.057% variance in accuracy and 28.9% in certified robustness. The variance is not uniform, and knowing where it concentrates matters more than demanding error bars everywhere.
- The model was right. Acting on it would have killed people.A pneumonia model learned that asthma lowers your risk of dying. It was correct about the data and dangerously wrong as guidance, and almost nothing in machine learning is built to tell the difference.
Further reading
- Kohavi, Longbotham et al. (2009), Controlled Experiments on the Web: Survey and Practical Guide — the standard reference; most ideas fail.
- Kohavi et al. (2012), Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained — surprising results usually have mundane causes. Read before believing your finding.
- Johari et al. (2017), Peeking at A/B Tests: Why It Matters, and What to Do About It — the cost of checking daily, and the sequential fix.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Peeking daily and stopping at significance. That's p-hacking with a dashboard.
- Randomising by session rather than user, so people see both versions.
- Choosing the metric after seeing the data. Something always moved.
- Running underpowered and reading the null as evidence of no effect.
- Ignoring guardrails. Most wins are a metric improving at something else's expense.
At a glance
FieldApplied AI
Why it worksrandomisation makes the groups identical except for the treatment
The sobering numberroughly a third of tested ideas help, a third do nothing, a third hurt
Offline metricsa filter, not a decision
The killer mistakepeeking and stopping
The limitit can't see months, and can't choose your metric
DifficultyBeginner
Often compared with
A/B testing vs. offline evaluation — one measures what happened to real users, the other measures a proxy on data the old system generated. Only the first is a decision.
Where this sits
A destination. 3 concepts lead here, and nothing in the corpus depends on it.
3Levelsteps in
3Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstTrain/Test Split
Computed from the prerequisite graph, not assigned. How this works