Every training step a neural network takes boils down to one question: "if I nudge this weight up a tiny bit, does the loss go up or down, and by how much?" That question has a precise mathematical answer, and it's called the derivative. Training a model is, almost literally, computing millions of derivatives and walking downhill.

The derivative: instantaneous rate of change

The derivative of a function $f(x)$ at a point tells you the slope of the function there — how fast $f$ changes as $x$ changes, formally $f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}$.

For $f(x) = x^2$, the derivative is $f'(x) = 2x$. At $x = 3$, the slope is $6$ — meaning a small increase in $x$ near 3 increases $f(x)$ about 6 times as fast. Check it numerically: $f(3) = 9$, $f(3.01) = 9.0601$, and $(9.0601 - 9)/0.01 = 6.01$ — matching the formula almost exactly.

python

Partial derivatives: slope along one axis at a time

ML loss functions depend on thousands or millions of weights at once, so we need a derivative with respect to each one, holding the others fixed. That's a partial derivative, written $\frac{\partial f}{\partial w_i}$.

For $f(w_1, w_2) = w_1^2 w_2 + 3w_2$, the partial derivative with respect to $w_1$ treats $w_2$ as a constant: $\frac{\partial f}{\partial w_1} = 2w_1 w_2$. The partial with respect to $w_2$ treats $w_1$ as a constant: $\frac{\partial f}{\partial w_2} = w_1^2 + 3$. At $(w_1, w_2) = (2, 1)$: $\frac{\partial f}{\partial w_1} = 2(2)(1) = 4$ and $\frac{\partial f}{\partial w_2} = 2^2 + 3 = 7$.

Note

Stack all the partial derivatives of a function into one vector, and you get the gradient: $\nabla f = [\frac{\partial f}{\partial w_1}, \frac{\partial f}{\partial w_2}, \dots]$. For the example above, $\nabla f = [4, 7]$. The gradient points in the direction of steepest increase of $f$ — which is exactly why gradient descent moves in the opposite direction, $-\nabla f$, to decrease the loss.

The chain rule: the engine behind backpropagation

Neural networks are functions composed inside other functions — a layer's output feeds into the next layer, which feeds into the loss. The chain rule tells you how to differentiate a composition: if $y = f(u)$ and $u = g(x)$, then $\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$.

Concretely, let $u = g(x) = 2x$ and $y = f(u) = u^2$, so $y = (2x)^2 = 4x^2$. Direct differentiation: $\frac{dy}{dx} = 8x$. Via the chain rule: $\frac{dy}{du} = 2u$ and $\frac{du}{dx} = 2$, so $\frac{dy}{dx} = 2u \cdot 2 = 4u = 4(2x) = 8x$. Same answer, computed by multiplying local derivatives along the chain — exactly how backpropagation computes gradients through a 100-layer network without ever writing out the full composed formula.

python

Note

Backpropagation multiplies many local derivatives together across layers (the chain rule applied repeatedly). If each local derivative is a number smaller than 1 — common with sigmoid activations, whose derivative maxes out at 0.25 — the product shrinks exponentially with depth. After 20 layers, a chain of 0.25s multiplies down to a number smaller than $10^{-12}$, and earlier layers effectively stop learning. This is the vanishing gradient problem, and it's a direct, mechanical consequence of the chain rule, not a bug.

Calculus concepts and their ML role

Concept	Formula	ML role
Derivative	$f'(x) = \lim_{h\to0} \frac{f(x+h)-f(x)}{h}$	Direction and steepness of the loss curve for one parameter
Partial derivative	$\partial f / \partial w_i$	Sensitivity of loss to one weight among millions
Gradient	Vector of all partial derivatives	Direction gradient descent steps against
Chain rule	$\frac{dy}{dx} = \frac{dy}{du}\cdot\frac{du}{dx}$	Backpropagation through stacked layers

What's next

Derivatives tell you which way is downhill; the next lesson — Optimization Algorithms: SGD, Adam & RMSprop Compared — covers exactly how models use that information to actually take steps, and why different optimizers take very different kinds of steps from the same gradient.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →