A/B Testing
Showing two versions to two random groups and measuring — the only method that tells you whether your model actually helped anyone.
When not to use it
- When you can't randomise. Pricing, legal constraints, one-off launches. Quasi-experimental methods exist and are weaker.
- With network effects, naively. If treatment affects control, randomisation is broken.
- For long-term effects. It measures weeks. Retention and trust take months and get confounded.
- When you lack the traffic. An underpowered test returns a null you'll misread as "it doesn't work."
Reach for something else instead
- Offline evaluation — a filter, not a decision. Necessary and insufficient.
- Interleaving — for ranking, mix both systems' results and see what's clicked. Far more sensitive per user.
- Quasi-experiments — difference-in-differences, regression discontinuity. When randomisation isn't available.
- Shadow deployment — run the new model without acting on it, compare. Safe, and it measures agreement rather than outcome.
Sources & further reading
- Kohavi, Longbotham et al. (2009), Controlled Experiments on the Web: Survey and Practical Guide — the standard reference; most ideas fail.
- Kohavi et al. (2012), Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained — surprising results usually have mundane causes. Read before believing your finding.
- Johari et al. (2017), Peeking at A/B Tests: Why It Matters, and What to Do About It — the cost of checking daily, and the sequential fix.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Peeking daily and stopping at significance. That's p-hacking with a dashboard.
- Randomising by session rather than user, so people see both versions.
- Choosing the metric after seeing the data. Something always moved.
- Running underpowered and reading the null as evidence of no effect.
- Ignoring guardrails. Most wins are a metric improving at something else's expense.
At a glance
FieldApplied AI
Why it worksrandomisation makes the groups identical except for the treatment
The sobering numberroughly a third of tested ideas help, a third do nothing, a third hurt
Offline metricsa filter, not a decision
The killer mistakepeeking and stopping
The limitit can't see months, and can't choose your metric
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
A/B testing vs. offline evaluation — one measures what happened to real users, the other measures a proxy on data the old system generated. Only the first is a decision.