Backpropagation Explained

The problem backpropagation solves

A neural network learns by adjusting its weights to reduce error. To know how to adjust a weight, you need to know how sensitive the final error is to that specific weight — mathematically, the partial derivative of the loss with respect to that weight. The challenge: a deep network might have millions of weights spread across dozens of layers, and the loss is computed only at the very end, after the input has passed through every layer.

Backpropagation is the algorithm that computes every one of those derivatives efficiently, in a single backward sweep through the network, by repeatedly applying one tool from calculus: the chain rule.

Forward pass vs. backward pass

Training a network on one example happens in two passes:

Forward pass: the input flows through the network layer by layer. Each layer computes a weighted sum of its inputs, applies an activation function, and passes the result to the next layer. The final layer produces a prediction, which is compared to the true label to compute a loss (a single number measuring how wrong the prediction was).

Backward pass: starting from the loss, the algorithm works backward through the network, computing how much each layer's output contributed to the error, then how much each weight contributed to that layer's output. By the time it reaches the first layer, every weight in the network has a gradient — a number telling you which direction to nudge it to reduce the loss.

The chain rule, the engine underneath everything

The chain rule says that if $y$ depends on $x$ through an intermediate variable $u$ — that is, $y = f(u)$ and $u = g(x)$ — then $\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$.

A neural network is just a long chain of nested functions: input → layer 1 → layer 2 → ... → loss. Backpropagation applies the chain rule repeatedly, layer by layer, walking backward from the loss to the input. At each layer, it only needs two things: the gradient flowing in from the layer after it, and the local derivative of that layer's own computation. Multiply them together and pass the result further back. This is why it's efficient — each layer does a small, local calculation, and the chain rule stitches all the local calculations into the correct global derivative.

Worked numeric example: a tiny two-layer network

Let's compute real numbers. Take a network with one input $x$, one hidden unit, and one output, using the sigmoid activation $\sigma(z) = \frac{1}{1+e^{-z}}$, trained with squared error loss.

Setup: $x = 2$, hidden weight $w_1 = 0.5$, hidden bias $b_1 = 0$, output weight $w_2 = 1.0$, output bias $b_2 = 0$. True label $t = 1$.

Forward pass:
$z_1 = w_1 x + b_1 = 0.5 \times 2 = 1.0$
$a_1 = \sigma(z_1) = \sigma(1.0) \approx 0.7311$
$z_2 = w_2 a_1 + b_2 = 1.0 \times 0.7311 = 0.7311$
$\hat{y} = \sigma(z_2) = \sigma(0.7311) \approx 0.6750$
$L = \frac{1}{2}(\hat{y} - t)^2 = \frac{1}{2}(0.6750 - 1)^2 \approx 0.0528$

Reading the result

Notice the gradient for $w_1$ (≈0.0059) is much smaller than for $w_2$ (≈0.0400). This is the chain rule compounding: $w_1$'s gradient had to pass through two sigmoid derivatives (at both layers) instead of one, and each sigmoid derivative is at most 0.25. This compounding effect, repeated across dozens of layers, is exactly what causes the vanishing gradient problem in deep networks — gradients shrink multiplicatively as they travel backward through many saturating activation functions. It's a major reason modern networks favor ReLU-family activations (which don't saturate for positive inputs) and use residual connections (which give gradients a shortcut path).

Try it yourself — watch gradients drive weight updates downhill

What's next

Backpropagation computes gradients for ordinary feedforward and convolutional networks the same way — but recurrent networks have to backpropagate through time, which introduces its own version of the vanishing gradient problem. See rnn-lstm for how that's handled, or revisit training-ml for how gradients feed into the broader training loop.