AI bias and fairness: why 'fair' has no single answer

Artificial intelligence increasingly helps decide who gets a loan, who is shortlisted for a job, who is granted bail, and who is flagged for extra medical attention. As these systems spread into consequential decisions, a natural fear follows: that they make those decisions unfairly, producing systematically worse outcomes for some groups of people than others. This fear is well founded, and documented cases of algorithmic bias are real. It sits alongside alignment and safety as one of the central concerns about deploying AI responsibly. The instinctive response is equally natural: find the bias, remove it, and make the model fair.

That response runs into a wall that most discussions of AI fairness never mention, and it is the single most important thing to understand about the topic. There is a mathematical result proving that several reasonable, widely-accepted definitions of fairness cannot all be satisfied at the same time. It is not that we have not yet built a clever enough algorithm; it is that the definitions themselves conflict, so satisfying one means violating another. This means fairness is not a technical bug waiting for a fix. It is an unavoidable choice among competing values, and pretending otherwise is how well-meaning efforts go wrong. This piece explains what algorithmic bias actually is, where it comes from, why it is so hard to remove, the impossibility result at the heart of the problem, and what can honestly be done about it.

What algorithmic bias is

Algorithmic bias is the tendency of an AI system to produce systematically different, and often worse, outcomes for different groups of people, typically along lines like race, gender, or age. A hiring model that ranks qualified women lower than equally qualified men, a facial recognition system that misidentifies darker-skinned faces more often, a risk score that flags one group as high-risk more frequently: these are algorithmic bias.

The important thing to be clear about is what this bias is not. It is usually not the model being prejudiced in the way a person can be, harbouring animosity or intent. A model has no beliefs or feelings. Algorithmic bias is statistical: the system has learned patterns that lead to unequal outcomes, whether or not anyone intended that result. This makes it both less and more troubling than human prejudice. Less, because there is no malice; more, because it can operate invisibly, at enormous scale, with a veneer of mathematical objectivity that makes it easy to trust and hard to challenge. A biased human decision affects the people that one person sees. A biased model can affect millions, consistently, while looking neutral.

Where it comes from: the mirror, not the malice

To fix bias you have to know where it enters, and it almost never enters as prejudice deliberately coded in. It enters mainly through the data, because a machine learning model learns the patterns present in the data it is trained on, and that data reflects the world, including the world's inequalities. A model is a mirror held up to its training data, and if the data carries bias, the mirror shows it back, sometimes amplified.

This happens in a few distinct ways worth separating. The first is unrepresentative data: if a facial recognition system is trained mostly on light-skinned faces, it learns those faces well and performs worse on the faces it saw less of, not by design but by the arithmetic of what it was shown. The second is historical bias baked into the data: if a hiring model is trained on a company's past hiring decisions, and those decisions favoured one group, the model learns to reproduce that favouritism, treating a historically biased pattern as the target to imitate. The model is faithfully learning what the data taught it; the unfairness was in the data. The third, and subtlest, comes from design choices: what outcome you decide to predict, and what measurable stand-in you use for it. A well-known case involved a healthcare system that used past medical spending as a proxy for medical need, which seemed reasonable but encoded a bias, because less money had historically been spent on some patients for reasons unrelated to how sick they were, so the model underestimated their need. No one intended discrimination; a plausible modelling choice quietly encoded it.

Seeing bias as a mirror reframes the whole problem. The model did not invent unfairness; it reflected and scaled the unfairness in its data and design. That is why bias is so persistent, and why it cannot be scrubbed out with a simple switch: it is woven into the information the system learned from and the choices its builders made.

The hard question: what does "fair" even mean?

Here the problem deepens in a way that surprises most people. Suppose you have found bias and you want to correct it. You quickly discover that you first have to decide what "fair" means, and there is more than one reasonable answer. Consider a model that assigns each person a score and makes a yes-or-no decision, like approving a loan. Three different, intuitive definitions of fairness immediately present themselves.

One is demographic parity: the model should approve the same proportion of people in each group. If it approves loans for forty percent of one group, it should approve forty percent of another. This captures a notion of equal representation in outcomes.

A second is equal error rates, sometimes called equalized odds: the model should make mistakes at the same rate for each group. It should not wrongly reject qualified applicants from one group more often than another, nor wrongly approve unqualified ones more often. This captures a notion of procedural justice, that the system's errors should not fall harder on one group.

A third is calibration: a given score should mean the same thing regardless of group. If the model assigns a score that it says corresponds to a seventy percent chance of repaying a loan, then among everyone who gets that score, seventy percent should actually repay, whichever group they belong to. This captures a notion of accuracy, that the score is equally meaningful for everyone.

Each of these is defensible. Each corresponds to a real value people care about. And here is the problem: in general, you cannot have all three.

The impossibility theorem

This is the result that changes how the whole subject should be understood. Researchers proved, independently and around the same time, that when the underlying base rates differ between groups, meaning the actual outcome occurs at different frequencies in each group, it is mathematically impossible for a model to be calibrated and have equal false positive rates and have equal false negative rates across those groups, unless the model is a perfect predictor, which never happens in practice. The three cannot all hold at once. Satisfying calibration forces the error rates to differ; equalizing the error rates breaks calibration.

The key point is that this is not a limitation of any particular algorithm, dataset, or company. It is a property of the definitions themselves, a direct consequence of how these quantities relate through the mathematics of probability once the base rates differ. You cannot engineer your way around it, because it is not an engineering problem. It applies to every binary decision system operating across groups whose outcomes occur at different rates, which describes most real applications, from lending to medicine to criminal justice. Wherever the base rates differ, the choice is forced: you can have equal error rates or you can have calibration, but not both, and you must decide which.

COMPAS: when both sides were right

The most famous illustration of this makes it concrete. A tool called COMPAS was used in United States courts to estimate the risk that a defendant would reoffend. In 2016, an investigation found that the tool produced higher false positive rates for Black defendants than white defendants, meaning it wrongly labelled Black defendants as high-risk more often. By the standard of equal error rates, this was clear unfairness.

The company that built the tool responded that it was, in fact, fair: it was calibrated, meaning a given risk score corresponded to the same actual reoffending rate regardless of race. By the standard of calibration, the tool was fair. Both claims were true. The tool did have unequal error rates, and it was calibrated. What looked like a factual dispute about whether the system was biased was actually a disagreement about which definition of fairness to use, and the impossibility theorem explains why they could not both be satisfied: because reoffending base rates differed between the groups, no tool could have equal error rates and be calibrated at once. Neither side was lying or confused. They had chosen different, incompatible, defensible notions of fairness, and the mathematics guaranteed they would clash.

The real lesson: fairness is a choice among values

This is why the framing of fairness as a technical problem to be solved is misleading, and why the honest account is different. Because the definitions conflict, choosing a fairness criterion is choosing which value to prioritize, and each choice has a constituency. Calibration serves the value of accuracy, of scores meaning what they say. Equal error rates serve the value of procedural justice, of the system not making its mistakes disproportionately at one group's expense. Demographic parity serves the value of equal representation in outcomes. These are real, different commitments about what a just decision looks like, and reasonable people weigh them differently depending on the stakes and the context.

So "make the AI fair" is not a well-defined instruction. It resolves into "decide which conception of fairness matters most here, accept that you are trading off the others, and be accountable for that choice." This is a social, ethical, and often legal question, not one a model can answer for you. The mathematics does not tell you which fairness to pick; it only proves that you cannot dodge the choice by picking all of them. Fairness in AI turns out to be a place where a hard technical result forces a value judgment into the open rather than settling it.

What can honestly be done

None of this means bias is unaddressable or that effort is pointless; it means the goal has to be stated honestly. Several things do help. The most fundamental is improving the data: collecting more representative training data closes gaps like the facial-recognition disparity, and scrutinising whether historical data encodes past discrimination lets you avoid blindly reproducing it. Careful attention to design choices, especially the proxies a model optimises, can catch problems like the healthcare-spending case before they cause harm. There are technical mitigation methods that adjust data, training, or outputs to reduce disparities on a chosen fairness metric, and there are tools and audits for measuring how a model performs across groups, along with legal rules of thumb such as the four-fifths guideline for detecting disparate impact. Regulation is also arriving: frameworks like the EU AI Act require high-risk systems to implement bias detection and correction using representative data, pushing fairness work from optional to mandatory.

But all of this operates within the constraint the impossibility theorem sets. You can reduce bias, measure it, and choose which fairness criterion to uphold, and you can do so transparently and accountably. What you cannot do is satisfy every notion of fairness simultaneously or hand the value judgment to an algorithm. The mature approach treats fairness as an ongoing, context-specific negotiation, informed by the mathematics, guided by human judgment about which values matter most in a given domain, and honest about the tradeoffs being made. (Generative models raise a related version of the same issue: because they learn the distribution of their training data, they can reproduce and amplify its biases in what they generate, from skewed representation in images to stereotyped associations in text, another face of the mirror. Being able to inspect why a model behaves as it does, through interpretability, may eventually help detect such biases from the inside rather than only in the outputs.)

The short version

Algorithmic bias is an AI system producing systematically different, often worse, outcomes for different groups, and it arises not from the model's malice but from its data and design: unrepresentative data, historical discrimination baked into training examples, and biased choices about what to predict and what proxies to use. The model is a mirror that reflects and can amplify the unfairness in what it learned from. Correcting bias runs into a deeper problem: there are several reasonable definitions of fairness, including demographic parity, equal error rates, and calibration, and a mathematical impossibility theorem proves that when base rates differ between groups, these cannot all hold at once. The COMPAS controversy was exactly this, with both sides correct under different definitions. So fairness is not a technical property to optimise but a choice among competing values, each serving a different notion of justice. Bias can be reduced and measured, but the tradeoff cannot be escaped.

The idea to hold onto is that making an AI fair is not an engineering task with one right answer but a value choice the mathematics forces into the open, because reasonable definitions of fairness are provably incompatible when groups differ, so the honest goal is not a perfectly fair model but a transparent, accountable decision about which fairness to prioritise and which tradeoffs to accept. Bias is a mirror of our data and choices, and fairness is a negotiation we cannot hand to a formula.

Common questions

What is algorithmic bias? Algorithmic bias is the tendency of an AI system to produce systematically different, and often worse, outcomes for different groups of people, typically along lines such as race, gender, or age. It usually is not the model being prejudiced in a human sense, since a model has no beliefs or intent; it is statistical, meaning the system learned patterns that lead to unequal outcomes. This can be more troubling than individual human bias because it operates invisibly, at massive scale, with an appearance of mathematical objectivity that makes it easy to trust and hard to challenge.

Where does AI bias come from? Mainly from data and design, not deliberate prejudice. A model learns the patterns in its training data, and that data reflects the world's inequalities. Bias enters through unrepresentative data, such as a facial recognition system trained mostly on light-skinned faces performing worse on darker ones; through historical bias, such as a hiring model learning from past biased decisions and reproducing them; and through design choices, such as picking a proxy like medical spending to represent medical need, which can quietly encode disparity. The model is a mirror that reflects, and can amplify, the bias in its data and the choices of its builders.

What is the impossibility theorem of fairness? It is a proven mathematical result that several reasonable definitions of fairness cannot all be satisfied at once. Specifically, when the base rates of an outcome differ between groups, no model can simultaneously be calibrated, have equal false positive rates, and have equal false negative rates across those groups, unless it is a perfect predictor, which never happens in practice. This is not a flaw in any particular algorithm; it follows from the mathematics of probability itself. It means that improving one fairness metric necessarily worsens another, so fairness requires choosing which definition to prioritise rather than satisfying them all.

What are the main definitions of AI fairness? Three common ones illustrate the conflict. Demographic parity requires the model to produce positive outcomes at equal rates across groups, capturing equal representation. Equal error rates, or equalized odds, require the model to make false positive and false negative errors at equal rates across groups, capturing procedural justice. Calibration requires that a given score means the same probability of the outcome regardless of group, capturing accuracy. Each is intuitive and defensible, and each serves a different value, but the impossibility theorem shows they generally cannot all hold together when groups have different base rates.

Was the COMPAS algorithm biased? It depends on which definition of fairness you use, which is exactly the point. An investigation found COMPAS, a criminal risk tool, produced higher false positive rates for Black defendants, making it unfair by the standard of equal error rates. The company countered that it was calibrated, meaning a given score corresponded to the same actual reoffending rate across races, making it fair by that standard. Both claims were true. Because reoffending base rates differed between groups, the impossibility theorem guaranteed the tool could not have both equal error rates and calibration. The dispute was about which fairness to value, not about the facts.

Can AI bias be completely eliminated? No, not in the sense of a perfectly fair model satisfying every definition, because the impossibility theorem shows the main fairness criteria conflict when groups differ. Bias can, however, be meaningfully reduced and managed: with more representative data, scrutiny of historical bias and design proxies, technical mitigation methods, fairness audits, and regulation. What cannot be done is satisfy all notions of fairness at once or hand the underlying value judgment to an algorithm. The honest goal is a transparent, accountable choice about which fairness to prioritise in a given context, not the elimination of every tradeoff.

How should organisations handle AI fairness? By treating it as an ongoing, context-specific value decision rather than a technical box to check. That means measuring how a model performs across groups, improving data representativeness, examining design choices and proxies, and explicitly choosing which definition of fairness matters most for the specific application, since the mathematics prevents satisfying them all. It also means being transparent about the tradeoffs accepted and remaining accountable for them, complying with regulations that increasingly mandate bias detection and correction, and keeping human judgment in the loop for consequential decisions rather than deferring to a score that only looks objective.