Computer Vision: Object Detection

Detection is harder than classification

Image classification answers one question: what is the single dominant thing in this image? Object detection is a much harder, more useful problem: given an image that may contain zero, one, or dozens of objects of different classes, find *where* each one is and *what* it is. The output isn't a single label — it's a list of bounding boxes, each with a class label and a confidence score.

This matters enormously in practice. A self-driving car doesn't need to know "this scene contains a pedestrian" in the abstract — it needs to know exactly where that pedestrian is, in pixel coordinates, updated every frame, so it can act. Detection is what turns a CNN's pattern-recognition ability into something that can be used for navigation, surveillance, retail analytics, and robotics.

Representing a detection: the bounding box

A bounding box is typically represented as four numbers: the (x, y) coordinates of its center (or top-left corner), plus its width and height. A detection result is then a set of (box, class, confidence) triples. The model has to solve two coupled subproblems simultaneously: a regression problem (predicting the four continuous box coordinates precisely) and a classification problem (deciding what's inside that box) — which is part of why detection architectures are more involved than plain classifiers.

Two-stage detectors: the R-CNN family

The original R-CNN (2014) worked in two explicit stages: first, propose a few thousand candidate regions in the image that might contain an object (using a separate, non-learned algorithm), then run a CNN classifier on each cropped region independently. This was accurate but painfully slow — running a full CNN forward pass thousands of times per image.

Fast R-CNN and Faster R-CNN progressively fixed this by sharing computation: run the CNN once over the whole image to get a feature map, then generate region proposals *from that shared feature map* instead of from raw pixels, and classify each proposed region using features sliced out of the same map. Faster R-CNN introduced a learned Region Proposal Network that generates candidate boxes directly, replacing the old non-learned proposal algorithm entirely — making the whole pipeline trainable end-to-end.

Anchor boxes: giving the model a starting guess

Rather than asking a network to predict four box coordinates from nothing, most detectors use anchor boxes: a fixed set of reference boxes of different sizes and aspect ratios, tiled densely across the image at every spatial location in the feature map. The network's job becomes much easier and more stable — for each anchor, predict whether an object is present, which class it is, and a small adjustment (offset) to the anchor's coordinates to better fit the real object, rather than inventing a box from scratch. Anchors of multiple sizes let the same detector handle both a tiny distant object and a large nearby one using the same underlying feature map.

One-stage detectors: YOLO

YOLO ("You Only Look Once") dispensed with the separate proposal stage entirely. A single CNN pass over the image directly predicts, for every cell in a grid overlaid on the image, a set of bounding boxes, class probabilities, and confidence scores — all in one forward pass. This made YOLO dramatically faster than two-stage detectors, fast enough for real-time video, at some initial cost to accuracy on small or densely packed objects. Successive YOLO versions narrowed that accuracy gap substantially while keeping the single-pass speed advantage, making the YOLO family the default choice for most real-time detection applications today.

Two-stage vs. one-stage detectors

What's next

Object detection localizes objects with boxes, which is coarse — it doesn't tell you the exact outline of an object. Image Segmentation goes further, classifying every individual pixel. Continue to Image Segmentation to see how semantic and instance segmentation refine detection down to pixel-level precision.