Early in training, weights are randomly initialized and far from any good solution — large steps make fast progress across the broad loss landscape. Late in training, the model is near a good solution, and large steps start to overshoot and oscillate rather than refine. A single fixed learning rate forces a compromise: too large for the end of training, or too small for the beginning.

A learning rate schedule changes the learning rate as a function of training step or epoch, instead of holding it fixed. Nearly every state-of-the-art training recipe — from ResNets to GPT — uses one, and the specific shape of the schedule is itself a meaningful design choice, not an afterthought.

Warmup: don't take big steps on a random network

Learning rate warmup starts training with a small learning rate and ramps it up linearly (or sometimes with another curve) over the first several hundred to few thousand steps, before switching to the main schedule. The reasoning: with randomly initialized weights, early gradients can be large and poorly informative about the right direction, and Adam's adaptive second-moment estimate is especially unreliable before it has accumulated enough history. A big learning rate applied to that noisy, uninformed signal can destabilize training or even cause divergence.

Warmup is close to mandatory when training transformers, and became standard practice following the original 'Attention Is All You Need' training recipe, which paired linear warmup with a subsequent decay.

python

Step decay: simple, predictable drops

Step decay multiplies the learning rate by a fixed factor (commonly 0.1 or 0.5) every N epochs — for example, dropping from 0.1 to 0.01 at epoch 30, then to 0.001 at epoch 60. It's the oldest and simplest schedule, and it's easy to reason about and debug, but the sudden drops are somewhat arbitrary: performance often jumps right after each drop (visible as a step in the training loss curve), which is a sign the schedule, not just the model, is driving the dynamics. Choosing exactly when to drop and by how much requires either manual tuning or prior experience with a similar architecture.

Cosine annealing: smooth, no magic drop points to tune

Cosine annealing decreases the learning rate smoothly following a cosine curve, from the base rate down to near zero (or a small minimum) over the course of training: lr(t) = lr_min + 0.5(lr_max − lr_min)(1 + cos(πt/T)). It starts the decay slowly, accelerates through the middle of training, and flattens out again near the end — avoiding both step decay's sudden jumps and the need to hand-pick when drops happen.

A common variant, cosine annealing with warm restarts, periodically resets the learning rate back up to its maximum and runs the cosine decay again, on the idea that occasional large steps can help the optimizer escape a sharp local minimum and find a flatter, better-generalizing one.

python

Visualizing why the step size needs to change over time

This gradient descent visualization shows convergence behavior under a fixed step size. Early in optimization, a larger step covers ground quickly across a broad loss surface; as the trajectory approaches the minimum, that same large step starts to overshoot and oscillate. Learning rate schedules exist precisely to supply a large effective step early and a small one late, within a single run, instead of locking in one compromise value for the whole trajectory.

📉 Gradient Descent Optimization

Interactive

Learning Rate (η)0.10

Steps taken:0

Current Weight (w):0.6000

Loss Value:6.2600

Gradient (dL/dw):-4.8000

Learning rate schedules compared

Schedule	Shape	Tuning effort	Typical use case
Constant	Flat	Minimal — just pick one value	Quick prototyping, simple convex problems
Step decay	Sharp drops at fixed intervals	Medium — must choose drop epochs and factor	CNN image classification, classical recipes
Linear decay	Straight-line decrease to zero	Low	Simple fine-tuning runs, short training budgets
Cosine annealing	Smooth S-shaped decrease	Low — mainly just total step count	Transformer pretraining, most modern deep learning
Cosine with warm restarts	Repeated cosine cycles	Medium — cycle length and restart count	Long training runs where escaping sharp minima helps
Warmup (paired with any of the above)	Ramp up, then follow main schedule	Low — mainly warmup step count	Nearly all transformer training

Note

A cosine schedule tuned for 100,000 steps will have an entirely different shape — and different optimal warmup length — if you change the total number of training steps or significantly change the batch size. Schedules are typically defined relative to total training steps for this reason, and copying a schedule's raw hyperparameters between runs with different step counts or batch sizes without rescaling is a common source of training instability.

What's next

Learning rate schedules control the size of each update, but another failure mode can wreck training regardless of schedule: individual gradients that spike to enormous values and blow up the weights in a single step. The next lesson, Gradient Clipping, covers how to cap that damage directly.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →