Image Segmentation
Labelling every pixel rather than drawing a box — what you need when the exact shape matters.
When not to use it
- When a box is enough. Detection is cheaper to label, train, and run. Don't buy boundaries you won't use.
- When you can't afford the annotation. Pixel-accurate labels are five to ten times the cost of boxes, and the estimate is usually optimistic.
- On thin or ambiguous structures without checking annotator agreement first. If two experts disagree, your ceiling is that disagreement.
Reach for something else instead
- Object detection when location is enough and shape isn't.
- Classification when you only need to know whether something's present.
- SAM or similar zero-shot models when the objects are visually distinct — you may not need a labelled dataset at all.
Sources & further reading
- Ronneberger, Fischer & Brox (2015), U-Net: Convolutional Networks for Biomedical Image Segmentation — still the default for scientific work a decade on.
- Kirillov et al. (2023), Segment Anything — promptable zero-shot segmentation, and the paper that made much annotation optional.
- Isensee et al. (2020), nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation — the argument that configuration beat architecture.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reporting pixel accuracy. On an image that's 98% background, predicting background everywhere scores 98%. Use IoU or Dice.
- Building semantic segmentation and then needing to count objects, which requires instance segmentation and a rebuild.
- Ignoring inter-annotator variation, then chasing a metric ceiling that's actually label noise.
At a glance
FieldComputer Vision
Core ideaa label for every pixel
Typessemantic, instance, panoptic
Default architectureU-Net
MetricIoU or Dice, never accuracy
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Segmentation vs. detection — the exact outline vs. a box that's good enough.