Famous CNN Architectures: VGG, ResNet & Beyond

Why architecture history still matters

It's tempting to treat architecture choice as a solved problem — just use whatever pretrained model tops a leaderboard. But every major CNN architecture solved a specific, concrete problem that the previous generation ran into. Understanding that progression explains *why* modern vision models are built the way they are, and the lessons — particularly around skip connections — now show up well beyond computer vision, in transformers and nearly every deep architecture trained today.

VGG: depth through brute-force simplicity

VGG (2014) made one deceptively simple architectural bet: instead of experimenting with different filter sizes and clever layer arrangements, use only small 3x3 convolutional filters, stacked repeatedly, with periodic max pooling to downsample, and just make the network deep — 16 or 19 layers, which was considered very deep at the time.

The insight behind using only 3x3 filters: stacking two 3x3 convolutional layers gives an effective receptive field of 5x5 (the second layer's filter sees outputs that already summarized a 3x3 neighborhood each), but with fewer parameters and more nonlinearities than a single 5x5 filter would need. VGG's uniform, repetitive structure made it easy to understand and extend, and it became a widely used backbone for transfer learning for years, despite being parameter-heavy (VGG-16 has about 138 million parameters, most of them in its fully connected layers at the end).

The wall VGG and its peers hit: vanishing gradients in very deep nets

VGG proved that depth helps — up to a point. Researchers tried simply stacking more layers (going from 19 layers toward 30+), expecting monotonically better results. Instead, training error actually got *worse* for sufficiently deep plain networks, not just test error. This wasn't overfitting; the very deep network was failing to even fit the training data well, which pointed to an optimization problem, not a capacity problem.

The cause is the same vanishing gradient problem that plagues deep sigmoid networks, showing up here through a different mechanism: as gradients backpropagate through dozens of stacked convolutional + activation layers, repeated multiplication by small derivative values and weight matrices causes the gradient signal reaching early layers to shrink toward zero, or occasionally to explode. Either way, early layers stop receiving a useful learning signal, and adding more layers made things worse rather than better.

ResNet: skip connections as the fix

ResNet (2015) introduced a deceptively small structural change that solved the degradation problem directly: skip connections (also called residual connections). Instead of a block computing a completely new output from scratch, $y = F(x)$, a residual block computes $y = F(x) + x$ — the block's transformation is added to its own input, which is passed through unchanged via a shortcut path.

This has a profound effect on gradients. During backpropagation, the gradient flowing out of a residual block is $\frac{\partial y}{\partial x} = \frac{\partial F(x)}{\partial x} + 1$. That "+1" term means there is always a direct, undiminished path for the gradient to flow backward through, regardless of how small or poorly-conditioned $\frac{\partial F(x)}{\partial x}$ becomes. Gradients can no longer vanish to zero simply from depth — there's always at least the identity path carrying signal straight through. This let ResNet train networks with over 150 layers, something that was simply infeasible with plain stacked convolutions.

Why residual learning is easier, not just deeper

There's a second, subtler benefit beyond gradient flow: residual blocks reframe what each layer needs to learn. Instead of learning an entire transformation from scratch, $F(x) = y$, a residual block only needs to learn the *difference* (the "residual") between its input and desired output, $F(x) = y - x$. If the ideal transformation for a given block is close to the identity function (common in very deep networks, where later blocks make only small refinements), learning a near-zero residual is far easier for gradient descent than learning an exact identity mapping from a generic set of weights — which plain convolutional layers would otherwise have to approximate exactly, a surprisingly hard optimization target in practice.

VGG vs. ResNet

The legacy: skip connections are everywhere now

ResNet's residual connection turned out to be far bigger than one architecture. The same idea — add a shortcut path so gradients have a direct route backward — is now a near-universal default in deep learning. Every transformer block wraps its self-attention and feed-forward sub-layers in residual connections for exactly the same reason ResNet needed them: without that shortcut, stacking dozens of transformer blocks (GPT-3 stacks 96) would suffer the identical vanishing gradient degradation that plain CNNs hit a decade earlier. Later CNN architectures like DenseNet pushed the idea further (connecting each layer to *every* previous layer, not just the immediately preceding one), and EfficientNet and vision transformers built on top of these lessons about depth, connectivity, and efficient parameter use.

What's next

With CNN fundamentals and their architectural evolution covered, you've now seen how the same core challenge — getting gradients to flow through deep stacks of layers — shows up and gets solved across very different architectures, from plain MLPs to CNNs to transformers. From here, it's worth revisiting the transformer lesson with this lens: notice how its residual connections and layer normalization are doing conceptually the same job that ResNet's skip connections do.