Inter-annotator Agreement
How often your human labellers agree with each other — the real ceiling on your model, and the number most projects never compute.
When not to use it
- Raw agreement, on skewed data. Two annotators both guessing the majority class agree 95% of the time and know nothing.
- Kappa alone, on skewed data. The kappa paradox: low kappa despite high raw agreement, because chance agreement is already at ceiling.
- As a single ground truth, on genuinely subjective tasks. Majority voting finds the majority's reading and erases minority interpretations.
- Never — which is what most projects do. It's a day of work and it tells you your ceiling.
Reach for something else instead
- Krippendorff's alpha — handles missing data, multiple annotators, ordinal scales. Most flexible, least used.
- Keeping the label distribution — don't collapse disagreement; model it.
- Expert adjudication — for high-stakes labels, a third annotator resolves conflicts.
- Better guidelines — most of the improvement comes from the second pass, after you read the disagreements.
Sources & further reading
- Cohen (1960), A Coefficient of Agreement for Nominal Scales — kappa; chance-corrected agreement.
- Artstein & Poesio (2008), Inter-Coder Agreement for Computational Linguistics — the careful practical treatment, including the paradoxes.
- Aroyo & Welty (2015), Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation — disagreement is signal, not noise. The reframing.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Never measuring it, so you never learn your project's ceiling.
- Reporting raw agreement on imbalanced data as if it means something.
- Reading low kappa as bad annotators when it's the kappa paradox on skewed prevalence.
- Blaming the model for what's annotation inconsistency. The bottleneck was upstream.
- Majority-voting subjective tasks and treating the result as truth.
At a glance
FieldMachine Learning
What it ishow often your labellers agree with each other
Why it mattersit's the hard ceiling on model accuracy
Default metricCohen's kappa
Typical subjective task0.6–0.8
The reframingdisagreement is signal, not noise
How often computedrarely
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Inter-annotator agreement vs. model accuracy — one is the ceiling, the other is the score. Reporting accuracy above the agreement rate tells you about your labels.