Optimization Algorithms: SGD, Adam & RMSprop Compared

Same gradient, different steps

Every optimizer starts from the same raw material: the gradient, computed by backpropagation. What separates SGD, Adam, RMSprop, and their relatives is purely how they turn that gradient into a parameter update — how big a step to take, and in what adjusted direction. These are implementation choices with large practical consequences for how fast and how reliably a model trains.

Vanilla gradient descent and SGD

Plain gradient descent updates every parameter by the rule $\theta \leftarrow \theta - \eta \nabla L(\theta)$, where $\eta$ is the learning rate. Stochastic gradient descent (SGD) is the same rule, but the gradient $\nabla L$ is estimated from a small random mini-batch instead of the whole dataset — far cheaper per step, at the cost of a noisier gradient estimate.

Say $\theta = 5.0$, $\eta = 0.1$, and the gradient at this point is $\nabla L = 4.0$. The update is $\theta \leftarrow 5.0 - 0.1(4.0) = 4.6$. Simple, but if the loss surface has steep ravines, plain SGD oscillates across the ravine walls instead of moving efficiently along the ravine floor.

Momentum: remembering previous directions

SGD with momentum keeps a running average of past gradients, $v \leftarrow \beta v + (1-\beta)\nabla L$, and updates using $v$ instead of the raw gradient: $\theta \leftarrow \theta - \eta v$. This smooths out noisy, oscillating gradients and lets the optimizer build speed in directions where gradients consistently point the same way — like a ball rolling downhill and accumulating velocity.

With $\beta = 0.9$, $v_0 = 0$, and gradients $\nabla L = 4.0$ then $\nabla L = 3.6$ on two consecutive steps: $v_1 = 0.9(0) + 0.1(4.0) = 0.4$, then $v_2 = 0.9(0.4) + 0.1(3.6) = 0.36 + 0.36 = 0.72$. The velocity builds up across steps that keep pointing the same direction.

RMSprop: adapting the step size per parameter

RMSprop tracks a running average of the squared gradient for each parameter, $s \leftarrow \beta s + (1-\beta)(\nabla L)^2$, then divides the learning rate by $\sqrt{s} + \epsilon$: $\theta \leftarrow \theta - \frac{\eta}{\sqrt{s}+\epsilon}\nabla L$.

The effect: parameters with consistently large gradients get their effective step size shrunk, and parameters with small, infrequent gradients get a relatively larger step. This matters enormously in deep networks where different layers (and different weights within a layer) can have wildly different gradient magnitudes — a single global learning rate would be either too big for some weights or too small for others.

Adam: momentum and RMSprop combined

Adam (Adaptive Moment Estimation) tracks both a momentum-style running average of the gradient ($m$, the "first moment") and an RMSprop-style running average of the squared gradient ($s$, the "second moment"), then bias-corrects both before using them: $\theta \leftarrow \theta - \eta \frac{\hat{m}}{\sqrt{\hat{s}}+\epsilon}$.

This gives Adam two complementary benefits at once: momentum smooths the direction of travel, and the per-parameter adaptive scaling (from the squared-gradient term) keeps step sizes sensible across parameters with very different gradient magnitudes. In practice this combination converges fast and tends to need less learning-rate tuning than plain SGD, which is why it became the default choice for training most deep networks, including transformers.

SGD vs. RMSprop vs. Adam

What's next

Optimizers determine how a model searches a loss landscape, but the shape of that landscape matters just as much. The next lesson — Convex vs. Non-Convex Optimization — explains why neural network loss surfaces are riddled with local minima and saddle points, and why that turns out to be less catastrophic than it sounds.