Image Segmentation

From boxes to pixels

A bounding box tells you roughly where an object is, but the box is always a rectangle — it includes background pixels around the object's actual irregular outline. Image segmentation goes further: it classifies every single pixel in the image, producing a mask that traces the exact shape of each object or region. Instead of "there's a car somewhere in this rectangle," segmentation tells you precisely which pixels belong to the car and which belong to the road behind it.

This is a fundamentally different output structure from classification or detection. Rather than a handful of numbers (class scores, or box coordinates), the output is itself an image — the same height and width as the input, where every pixel carries a predicted label.

Semantic segmentation vs. instance segmentation

Semantic segmentation assigns every pixel a class label (road, sky, car, person) but does not distinguish between separate instances of the same class — two different cars both just get labeled "car," merged into one blob if they're adjacent. Instance segmentation goes a step further, separating individual objects of the same class: car #1 and car #2 get distinct masks, even though both are "cars." Panoptic segmentation combines both: every pixel gets a class label, and object classes additionally get separated into distinct instances, giving a single complete scene description.

Segmentation types compared

The U-Net architecture

U-Net, originally designed for biomedical image segmentation, is one of the most influential segmentation architectures and a good model for understanding the general approach. It has a symmetric "U" shape: an encoder (contracting path) progressively downsamples the image through convolution and pooling layers, building up increasingly abstract features while losing spatial resolution — exactly like a standard CNN classifier. A decoder (expanding path) then progressively upsamples back to the original resolution, turning those abstract features back into a full-resolution pixel map.

The key architectural trick is skip connections: at each resolution level, the decoder receives not just the upsampled features from the layer below, but also the original encoder features at that same resolution, concatenated directly in. This matters because downsampling destroys fine spatial detail (exact edges, small structures) that pooling throws away — skip connections let that detail flow directly to the decoder instead of forcing the network to somehow reconstruct it from a heavily compressed representation.

Why segmentation matters in practice

In medical imaging, segmentation is used to precisely outline tumors, organs, or lesions in MRI and CT scans — a task where pixel-level precision directly affects diagnosis and treatment planning, and where U-Net's original biomedical origins make it a natural fit. In autonomous driving, semantic segmentation of road scenes (drivable road, lane markings, pedestrians, other vehicles) feeds directly into path planning, since knowing the exact pixel boundary of the road surface matters far more than a coarse bounding box. Satellite imagery analysis, photo editing (background removal), and agricultural crop monitoring are other domains where pixel-precise understanding, not just coarse localization, is the actual requirement.

Seeing the encoder-decoder structure

U-Net's encoder-decoder structure is still a layered neural network trained end-to-end with backpropagation, just like the general network below — the distinguishing feature is simply the skip connections bridging matching resolution levels. Use the visualization to build intuition for how a deep stack of layers transforms information from input to output.

What's next

Segmentation and detection both operate on natural photographs and video. OCR & Document AI applies similar vision techniques to a different kind of image: scanned text and documents. Continue to OCR & Document AI to see how vision and language models combine to read and understand document layouts.