Bias & Fairness
The problem of AI systems producing unfair or discriminatory outcomes — usually by absorbing biases present in their training data.
When not to use it
- As a metric you can max out. Fairness definitions conflict mathematically — you cannot satisfy them all at once, and choosing between them is a value judgement, not an optimisation.
- As a post-hoc audit only. Bias enters through the problem framing and the data collection, long before the model exists. Auditing at the end finds it too late to fix cheaply.
- As a technical fix for a policy problem. Sometimes the right answer is not to build the system.
Reach for something else instead
- Better data collection — representative sampling addresses more bias than any debiasing algorithm applied afterwards.
- Not automating the decision. For high-stakes, contested judgements, a documented human process may be both fairer and more defensible.
- Simpler, interpretable models where you can see and argue about what's driving the outcome.
Sources & further reading
- Buolamwini & Gebru (2018), Gender Shades — error rates on commercial systems broken down by skin tone and gender. The paper that made this concrete.
- Kleinberg, Mullainathan & Raghavan (2016), Inherent Trade-Offs in the Fair Determination of Risk Scores — the proof that fairness definitions conflict mathematically.
- Mitchell et al. (2019), Model Cards for Model Reporting — the documentation practice, if you want somewhere to start.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Removing the protected attribute and declaring the model fair. Proxies remain — postcode carries race, first name carries gender.
- Reporting one fairness metric without stating which definition it encodes and what it trades away.
- Testing on aggregate accuracy, which can look excellent while the model fails badly for a subgroup that's small in the data and large in reality.
At a glance
FieldSafety & Ethics
Core ideaunfair outcomes from biased data
Key trapremoving a sensitive attribute doesn't remove bias
Realityfairness metrics conflict
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4