The Adam Optimizer Explained

Why plain gradient descent isn't enough

Vanilla gradient descent updates every parameter by the same learning rate, in the direction of the negative gradient: θ = θ − η∇L. This works, but it has two chronic problems in deep networks. First, different parameters often need very different effective step sizes — some dimensions of the loss landscape are steep, others nearly flat, and one fixed learning rate can't serve both well. Second, gradient descent on noisy mini-batches zigzags, wasting steps oscillating across a ravine instead of moving steadily toward the minimum.

Adam (Adaptive Moment Estimation), introduced in 2014, fixes both problems at once by combining two separate ideas that had each been developed to solve one of them: momentum, and adaptive per-parameter learning rates. It has been the default optimizer for training most transformer-based models ever since.

Ingredient one: momentum

Momentum keeps a running, exponentially decaying average of past gradients instead of using only the current gradient. This smooths out the noise from mini-batch sampling and lets the optimizer build speed in directions where gradients consistently point the same way, while canceling out oscillation in directions where gradients keep flipping sign. It's the same intuition as a ball rolling downhill — it accelerates along a consistent slope and doesn't instantly change direction every time the local gradient wobbles.

Ingredient two: adaptive learning rates

RMSProp, a predecessor Adam builds on, keeps a separate running average of the *squared* gradients for each parameter, and divides the update by its square root. This means parameters with historically large gradients get their effective learning rate shrunk, while parameters with historically small gradients get a relative boost. The practical effect: every parameter gets its own adaptive step size, instead of one global learning rate trying to serve every parameter in the network at once.

The actual update equations

At each training step t, given gradient g_t for a parameter:

1. First moment (momentum): m_t = β₁·m_{t-1} + (1 − β₁)·g_t
2. Second moment (adaptive scale): v_t = β₂·v_{t-1} + (1 − β₂)·g_t²
3. Bias correction (both start at zero, which biases early estimates toward zero): m̂_t = m_t / (1 − β₁ᵗ), v̂_t = v_t / (1 − β₂ᵗ)
4. Parameter update: θ_t = θ_{t-1} − η · m̂_t / (√v̂_t + ε)

Default hyperparameters (from the original paper, and still the near-universal defaults): β₁ = 0.9, β₂ = 0.999, ε = 1e-8. The bias correction step matters most early in training, when m and v are still close to their zero initialization and would otherwise systematically underestimate the true moving averages.

See the optimization trajectory directly

This gradient descent visualization shows how an optimizer's path across a loss surface changes with different update rules. Plain gradient descent zigzags across steep, narrow valleys; Adam's combination of momentum (smoothing the direction) and per-parameter scaling (adapting the step size to each dimension's typical gradient magnitude) produces a noticeably more direct path toward the minimum.

Adam vs. its predecessors

What's next

Adam adapts the learning rate per parameter automatically, but the base learning rate η you supply still matters enormously, and a fixed value is rarely optimal across an entire training run. The next lesson, Learning Rate Schedules, covers how warmup, cosine annealing, and step decay shape that base rate over time to get better and more stable results.