What is image segmentation?
Image segmentation is the computer vision task of labeling every pixel in an image. Where object detection draws a rectangle around a car, segmentation traces the car’s exact outline and classifies the road, the curb, and the sky around it. The output is a mask: an image the same size as the input in which each pixel carries a class or an object ID.
Segmentation is the right tool when shape, area, or boundary matters. A bounding box around a rust patch says the rust exists; a mask says how many square centimeters it covers and whether it touches the weld.
Types of image segmentation
- Semantic segmentation assigns a class to each pixel without separating individual objects, so all pallets in the image share one label.
- Instance segmentation separates individual objects, so each pallet gets its own mask and can be counted and measured on its own.
- Panoptic segmentation combines the two, labeling countable things as instances and background regions such as floor and wall as classes.
Models are trained on images where annotators have traced each region by hand, which is slow and expensive work. Foundation models for segmentation now produce a usable mask from a click or a box, and annotation teams use them to cut labeling time sharply.
Image segmentation in practice
A wound care clinic photographs chronic wounds at each visit with a tablet. A segmentation model outlines the wound bed and the surrounding redness, and the software converts the mask to an area using a calibration marker placed next to the wound. Clinicians see a healing curve over weeks instead of relying on a ruler and an estimate. The measurement is stored in the electronic health record with the image, and any case where the model’s mask is uncertain is flagged for manual tracing.
Property insurers apply the same technique to aerial roof images, measuring the area of hail damage per roof section to speed up estimates.
What to watch for
- Pixel-level labeling costs several times more per image than drawing boxes, so confirm that the use case needs outlines before committing to it.
- Agree on the boundary definition with domain experts first, because two annotators will trace a wound edge differently unless the rule is written down.
- Measure quality with intersection over union (IoU) on your own images rather than accepting a benchmark figure.
- Converting pixels to real-world units needs a known scale in the image, such as a reference marker or a fixed camera distance.