CNN Building Blocks: Convolution & Pooling Layers

Why fully connected layers are the wrong tool for images

A fully connected layer treats every pixel as an independent feature, connecting each one to every neuron in the next layer with its own weight. For a modest 224x224 RGB image, that's over 150,000 input values — a single fully connected hidden layer of even 1,000 units would need over 150 million weights, and the network would have to relearn "edge detector" from scratch at every possible pixel location independently.

Convolutional layers fix this by exploiting two properties of images that fully connected layers ignore: locality (nearby pixels are related; a pixel in the corner has little direct relationship to one in the center) and translation invariance (a cat's ear looks the same whether it appears top-left or bottom-right of the image, so the same detector should work everywhere). A convolutional filter is small, shared across the entire image, and specifically exploits both properties.

How a convolutional filter slides and detects

A convolutional filter (also called a kernel) is a small grid of learnable weights — commonly 3x3 or 5x5 — that slides across the input image one position at a time. At each position, it computes an elementwise multiplication between the filter weights and the patch of pixels it's currently covering, then sums the results into a single number. That number becomes one entry in the output, called a feature map.

Because the exact same filter weights are reused at every position (this is the "shared weights" idea), the filter learns to detect one specific pattern — an edge, a corner, a particular texture — and finds it wherever it occurs in the image, using dramatically fewer parameters than a fully connected layer would need for the same job. A convolutional layer typically applies many filters in parallel, each learning to detect a different pattern, producing a stack of feature maps.

Stride and padding: controlling output size

Two parameters control exactly how the filter moves and what happens at the image's edges:

Stride is how many pixels the filter moves between each application. A stride of 1 moves it one pixel at a time, producing a densely sampled output. A stride of 2 skips every other position, halving the output's height and width roughly — useful for downsampling without a separate pooling layer.

Padding adds extra pixels (usually zeros) around the border of the input before convolving. Without padding ("valid" convolution), the output shrinks slightly every layer because the filter can't center itself on edge pixels. With padding ("same" convolution, sized so the output matches the input's spatial dimensions), the output stays the same size, which matters when you want to stack many convolutional layers without the feature map shrinking to nothing.

The general formula for output size along one dimension, given input size $n$, filter size $f$, padding $p$, and stride $s$, is:

$$\text{output size} = \left\lfloor \frac{n + 2p - f}{s} \right\rfloor + 1$$

Pooling: downsampling for efficiency and robustness

Pooling layers reduce the spatial size of feature maps by summarizing small neighborhoods into a single value, with no learnable weights involved. This serves two purposes: it reduces the computational cost of subsequent layers (fewer values to process), and it adds a degree of translation invariance — a feature that shifts by a pixel or two still produces roughly the same pooled output.

Max pooling takes the maximum value within each small window (commonly 2x2), keeping only the strongest activation in each region — the intuition being that if a feature (like an edge) is present anywhere in that small window, that's what matters, not its exact pixel position.

Average pooling takes the mean value within each window instead, producing a smoother, less "spiky" summary. Max pooling has historically been more popular in classification CNNs because it preserves the strongest signal, while average pooling is common in contexts where you want to retain more information about the overall magnitude of activation across a region (and is used, for instance, in global average pooling right before a classification head).

Convolution vs. pooling vs. fully connected layers

Visualizing feature detection

See how stacking convolution and pooling operations progressively transforms raw pixel values into higher-level feature representations.

What's next

Convolution and pooling are the atomic building blocks, but the way they're arranged — how many layers, how channels grow, where shortcuts are added — is what separates one CNN architecture from another. The next lesson covers the famous architectures that defined computer vision's progress: VGG, ResNet, and the ideas that followed.