Bias & Fairness
The problem of AI systems producing unfair or discriminatory outcomes — usually by absorbing biases present in their training data.
When not to use it
- As a metric you can max out. Fairness definitions conflict mathematically — you cannot satisfy them all at once, and choosing between them is a value judgement, not an optimisation.
- As a post-hoc audit only. Bias enters through the problem framing and the data collection, long before the model exists. Auditing at the end finds it too late to fix cheaply.
- As a technical fix for a policy problem. Sometimes the right answer is not to build the system.
Reach for something else instead
- Better data collection — representative sampling addresses more bias than any debiasing algorithm applied afterwards.
- Not automating the decision. For high-stakes, contested judgements, a documented human process may be both fairer and more defensible.
- Simpler, interpretable models where you can see and argue about what's driving the outcome.
Read more on the blog
- Amazon's hiring AI: the case with no primary sourceThe most-cited AI bias case in the world rests on one news investigation, five anonymous sources, no published numbers, and an operator that disputes the central harm claim.
- COMPAS: both sides of the dispute were correctA newspaper said a risk score was biased. The vendor said it was fair. Two research teams then proved independently that both claims were true and cannot both be fixed.
- AI bias and fairness: why 'fair' has no single answerAI now helps decide who gets a loan, an interview, bail, or medical priority, and the fear is that it does so unfairly. The instinct is to remove the bias and make the model fair. But a mathematical result makes that impossible in a precise way: several reasonable definitions of fairness cannot all hold at once, so fairness is not a bug to fix but a choice among competing values.
- Williams v Detroit: the match was not the failureThe first publicly reported wrongful arrest from a face recognition match. The settlement's remedy is procedural, and it names a failure that has nothing to do with model accuracy.
The full account
The argument that couldn't be settled
In May 2016, ProPublica published an investigation of COMPAS, a risk-assessment tool used in American courts to score a defendant's likelihood of reoffending. Their finding was stark: among defendants who did not go on to reoffend, Black defendants were roughly twice as likely as white defendants to have been labelled high risk. A false positive — being marked dangerous when you are not — was falling disproportionately on Black defendants.
Northpointe, the company behind COMPAS, responded with a defence that was equally specific and, crucially, equally verifiable: the tool is calibrated. Among defendants given a score of 7, about the same fraction reoffend regardless of race. A 7 means the same thing for everyone. By that standard, the tool is scrupulously fair.
The natural reaction is that one side must be wrong, or hiding something. Neither was. Both claims were true of the same tool at the same time, and the reason is not political. It is arithmetic.
The impossibility result
Within months, two papers arrived independently at the same theorem. Kleinberg, Mullainathan and Raghavan formalised three conditions any risk score might plausibly be asked to satisfy:
Calibration — a score of 7 means the same probability of reoffending for every group. Balance for the negative class — people who won't reoffend get the same average score regardless of group. Balance for the positive class — people who will reoffend get the same average score regardless of group.
Each is a reasonable thing to want. Each corresponds to something a person would call fairness without hesitation. The theorem says: you cannot have all three, except in two situations that never occur in practice — either the predictor is perfect, or the base rates are identical across groups.
Chouldechova derived essentially the same result from a different direction, working directly on COMPAS, and stated it in the terms the argument was actually being conducted in: if base rates differ between groups, a calibrated classifier must have different false positive and false negative rates across those groups. Not "might". Must. It falls out of the confusion matrix algebra — calibration, base rate, and error rates are bound together, and fixing two determines the third.
So the COMPAS argument had this structure: ProPublica measured error-rate balance and found it violated. Northpointe measured calibration and found it satisfied. Both measurements were correct. Because recidivism base rates differed between the groups — themselves a product of policing, poverty, and history that no classifier chose — satisfying one criterion mathematically forced violating the other. There was no version of COMPAS that passed both tests, and no better engineering that could have produced one.
| Criterion | What it demands | Who invoked it |
|---|---|---|
| Calibration | A score means the same thing across groups | Northpointe — and COMPAS satisfied it |
| Balanced FPR | Equal false-alarm rates across groups | ProPublica — and COMPAS violated it |
| Balanced FNR | Equal miss rates across groups | Also violated, in the opposite direction |
| All three at once | — | Impossible when base rates differ |
What follows from it
The first consequence is that "is this model fair?" is not a well-posed question, and treating it as one guarantees an argument that cannot resolve. The well-posed question is which fairness criterion, and the answer is not discoverable from the data. It is a choice about which error you would rather inflict on whom, and that is a normative decision that belongs to whoever bears the consequences — not to the person tuning the threshold.
The second consequence is unwelcome for a large and growing industry. Fairness cannot be fixed by a library. Toolkits will compute a dozen fairness metrics for your model, and the theorem guarantees you cannot satisfy them simultaneously. A tool that reports twelve metrics and flags the ones you fail is reporting, with a straight face, that the mathematics is intact.
The third is the one most often skipped: the impossibility bites because base rates differ, and base rates differ because of the world, not the model. A classifier trained on arrest data inherits which neighbourhoods were policed. The impossibility theorem is therefore not an excuse — it does not say "unfairness is inevitable, carry on". It says the unfairness cannot be resolved inside the model, which relocates the question to whether the system should exist, which is exactly where the people deploying it least want it to be.
What honest practice looks like
State the criterion before you build. Write down which error is worse and for whom, and get someone who is not an engineer to agree with it — because you are making that choice whether or not you write it down, and the default is whatever your loss function happened to encode.
Report the base rates. Almost every published fairness claim omits them, and without them a reader cannot tell whether a criterion was satisfied by good work or by the groups being similar.
Treat a tool that passes all metrics with suspicion, not relief. Either the base rates are equal, the classifier is perfect, or someone is measuring something other than what they claim.
And notice that "remove the protected attribute" — still the most common first instinct — does nothing. The attribute is reconstructible from postcode, name, purchase history, and a hundred other correlates, so blinding the model removes your ability to measure the disparity while leaving the disparity in place. It is the one intervention that makes the problem invisible without making it smaller.
Further reading
- Buolamwini & Gebru (2018), Gender Shades — error rates on commercial systems broken down by skin tone and gender. The paper that made this concrete.
- Kleinberg, Mullainathan & Raghavan (2016), Inherent Trade-Offs in the Fair Determination of Risk Scores — the proof that fairness definitions conflict mathematically.
- Mitchell et al. (2019), Model Cards for Model Reporting — the documentation practice, if you want somewhere to start.
- Kleinberg, Mullainathan & Raghavan (2016), Inherent Trade-Offs in the Fair Determination of Risk Scores — the impossibility result: three natural fairness conditions cannot hold together except in degenerate cases.
- Chouldechova (2017), Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments — the same impossibility, derived independently, applied directly to COMPAS.
- Hardt, Price & Srebro (2016), Equality of Opportunity in Supervised Learning — equalised odds, and how to post-process a classifier to satisfy it.
- Angwin, Larson, Mattu & Kirchner (2016), Machine Bias — the ProPublica investigation that started the argument.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Removing the protected attribute and declaring the model fair. Proxies remain — postcode carries race, first name carries gender.
- Reporting one fairness metric without stating which definition it encodes and what it trades away.
- Testing on aggregate accuracy, which can look excellent while the model fails badly for a subgroup that's small in the data and large in reality.
At a glance
Where this sits
3 concepts come first. Understanding it opens up 3 more.
Computed from the prerequisite graph, not assigned. How this works