Residual Connections and Skip Connections

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

The problem deep networks run into

Stacking more layers should, in theory, only help — a deeper network can always learn to copy a shallower one by setting extra layers to the identity function. In practice, plain deep networks get *harder* to train as depth increases, not easier. Accuracy on both training and test data can degrade past a certain depth, a phenomenon documented clearly in the 2015 ResNet paper (He et al.). This isn't classical overfitting — it's an optimization failure. Gradients computed via backpropagation have to pass through many layers of matrix multiplications and nonlinearities, and they shrink or distort at each one.

Residual connections: learning a difference, not a transformation

A residual (or skip) connection adds the input of a block directly to its output: instead of a layer learning $y = F(x)$, it learns $y = F(x) + x$. The block $F$ only has to learn the residual — the correction needed on top of the identity mapping — rather than the full transformation. If the optimal function for that block really is close to identity, $F(x)$ only needs to push toward zero, which is a much easier target for gradient descent to find than reconstructing an exact identity through a stack of nonlinear layers.

residual_block.py

python

import torch
import torch.nn as nn

class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn1 = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn2 = nn.BatchNorm2d(channels)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        identity = x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = out + identity          # the skip connection
        return self.relu(out)

Why this fixes the gradient flow problem

During backpropagation, the gradient of a residual block with respect to its input is $\frac{\partial y}{\partial x} = \frac{\partial F(x)}{\partial x} + 1$. That '+1' is the key: no matter how small or distorted the gradient through $F$ becomes, the identity term guarantees a direct, undiminished path for the gradient back to earlier layers. Across a network with dozens or hundreds of residual blocks, this creates a highway that gradients can travel largely undamaged, instead of a chain of multiplications that vanish toward zero. This is the same structural idea that motivates highway networks and, later, the residual-style connections inside transformer blocks (around both the attention and feed-forward sub-layers).

Identity mapping as a free fallback

A residual block can always fall back to an identity function by driving $F(x) \to 0$ — something plain feedforward stacks cannot do cheaply. This means adding more residual blocks can never make a network's representational capacity *worse* in principle, only potentially redundant. That property is what let ResNets scale to 50, 101, and 152 layers (and later work past 1000) when prior architectures plateaued or degraded well before that.

Shortcut types: identity vs. projection

The skip path isn't always a literal copy. When a block changes the number of channels or spatial resolution (common at stage transitions in ResNet), the identity shortcut is replaced with a projection shortcut — typically a 1x1 convolution with stride — so the dimensions of $F(x)$ and the shortcut match for the addition. The original ResNet paper tested both pure identity shortcuts (zero-padded to match dimensions) and learned projection shortcuts, finding projections perform marginally better but add parameters and compute at every transition, so many implementations only use them where dimensions actually change.

Connection type	Extra parameters	Where used
Identity shortcut	None	Blocks where input/output shape match
Projection (1x1 conv) shortcut	Small	Blocks that change channels/stride
Dense connections (DenseNet)	Moderate-high	Concatenate all prior layer outputs
Highway network gating	Moderate	Learned scalar gate per connection

Skip connections are not a universal fix

Residual connections address vanishing gradients and optimization difficulty, but they don't automatically fix exploding gradients, poor initialization, or badly chosen learning rates. They're also typically paired with normalization layers (BatchNorm, LayerNorm) — removing those while keeping only the skip path can reintroduce instability, especially in very deep or very wide networks.

What's Next?

Residual connections are one piece of the toolkit that makes very deep networks trainable; normalization layers and careful weight initialization are the others, and they tend to interact — changing one often changes how much the others matter. Seeing how these pieces combine in a full modern architecture, like a transformer block, makes the dependency between them concrete.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →