Image Segmentation
Labelling every pixel rather than drawing a box — what you need when the exact shape matters.
Reviewed July 11, 2026Stable
When not to use it
- When a box is enough. Detection is cheaper to label, train, and run. Don't buy boundaries you won't use.
- When you can't afford the annotation. Pixel-accurate labels are five to ten times the cost of boxes, and the estimate is usually optimistic.
- On thin or ambiguous structures without checking annotator agreement first. If two experts disagree, your ceiling is that disagreement.
Reach for something else instead
- Object detection when location is enough and shape isn't.
- Classification when you only need to know whether something's present.
- SAM or similar zero-shot models when the objects are visually distinct — you may not need a labelled dataset at all.
Segmentation labels every pixel, not just a box.
Detection gives a rough box; segmentation is precise to the pixel — each pixel is assigned to an object or region, tracing exact shapes. That precision is what medical imaging, self-driving perception, and photo-editing tools need, and it's far more demanding than drawing a rectangle.
Further reading
- Ronneberger, Fischer & Brox (2015), U-Net: Convolutional Networks for Biomedical Image Segmentation — still the default for scientific work a decade on.
- Kirillov et al. (2023), Segment Anything — promptable zero-shot segmentation, and the paper that made much annotation optional.
- Isensee et al. (2020), nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation — the argument that configuration beat architecture.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reporting pixel accuracy. On an image that's 98% background, predicting background everywhere scores 98%. Use IoU or Dice.
- Building semantic segmentation and then needing to count objects, which requires instance segmentation and a rebuild.
- Ignoring inter-annotator variation, then chasing a metric ceiling that's actually label noise.
At a glance
FieldComputer Vision
Core ideaa label for every pixel
Typessemantic, instance, panoptic
Default architectureU-Net
MetricIoU or Dice, never accuracy
DifficultyAdvanced
Often compared with
Segmentation vs. detection — the exact outline vs. a box that's good enough.
Where this sits
A destination. 11 concepts lead here, and nothing in the corpus depends on it.
8Levelsteps in
11Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstObject Detection
Computed from the prerequisite graph, not assigned. How this works