Data Labeling
Humans deciding what each example is — the least visible and most determinative work in supervised learning, done by people the field rarely names.
When not to use it
- Before measuring agreement. Below 0.6 kappa, more labels don't help — the task isn't defined.
- At scale, before piloting twice. Your guidelines are wrong. They're always wrong on the first pass.
- With majority voting on subjective tasks. It manufactures confidence on exactly the ambiguous cases.
- Randomly, when active learning exists. You're paying for labels the model already knows.
Reach for something else instead
- Active learning — label what the model is unsure about. Often a fraction of the cost.
- Programmatic / weak supervision — noisy rules combined statistically.
- Pretrained models — the label you needed may already be in someone's model.
- Keeping the distribution — don't collapse disagreement; train on it.
Sources & further reading
- Gray & Suri (2019), Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass — the labour that the automation conceals.
- Aroyo & Welty (2015), Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation — disagreement is signal; majority voting destroys it.
- Ratner et al. (2017), Snorkel: Rapid Training Data Creation with Weak Supervision — labels from noisy rules, no annotators.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Writing guidelines and not labelling 50 yourself first. You'd have found the ambiguity in an hour.
- Scaling before piloting. The disagreements are where the guideline is broken.
- Treating disagreement as noise. On subjective tasks it's the signal.
- Assuming noisy labels produce obviously bad models. Networks memorise noise — you get confident, patterned errors.
At a glance
FieldMachine Learning
What it really isyour definition of the task, written by whoever wrote the guideline
The gatemeasure agreement before scaling; below 0.6 kappa, stop
Where quality comes fromreading disagreements, not more labels
The reframingdisagreement is signal, not noise
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Random labelling vs. active learning — one pays for examples the model already knows; the other buys only the ones it's unsure about, often at a fraction of the cost.