Batch Normalization & Layer Normalization
Why deep networks are hard to train
As a neural network trains, every layer's weights are updated simultaneously. This creates a subtle problem: the distribution of inputs each layer receives from the layer before it keeps shifting as those earlier layers' weights change. A layer that was working well with inputs in one numerical range suddenly has to cope with inputs in a different range after the previous layer updates. This phenomenon was named internal covariate shift by the authors of the original Batch Normalization paper (2015), and it was proposed as the reason very deep networks were slow and finicky to train: each layer is perpetually chasing a moving target, forcing the use of small learning rates and careful initialization just to avoid divergence.
Batch Normalization: normalize across the batch
Batch Normalization (BatchNorm) addresses this directly: for each feature (each neuron's output), it normalizes activations to have zero mean and unit variance, computed *across the examples in the current mini-batch*. After this normalization, it applies a learned scale and shift (two extra parameters per feature), so the network can still represent any distribution it needs — it just controls that distribution explicitly rather than leaving it to drift.
The practical effects were dramatic: networks could be trained with much higher learning rates, converged faster, and were noticeably less sensitive to initialization. BatchNorm also has an incidental regularizing effect, since the mini-batch statistics introduce a small amount of noise into training.
Note
Layer Normalization: normalize across features instead
Layer Normalization (LayerNorm, 2016) normalizes differently: instead of computing statistics across the batch for each feature, it computes the mean and variance across all the features *within a single example*, independently for each example in the batch. This makes LayerNorm's behavior completely independent of batch size and batch composition — even a batch size of 1 behaves identically to a batch size of 1,000, since each example's normalization never depends on any other example.
This property makes LayerNorm the natural choice for sequence models and transformers, where you often process variable-length sequences, sometimes one token or one example at a time, and where normalizing 'across the batch' for a given sequence position is awkward or ill-defined. Every major transformer architecture — BERT, GPT, and their successors — uses LayerNorm (or close variants like RMSNorm) rather than BatchNorm.
BatchNorm vs LayerNorm
| Property | Batch Normalization | Layer Normalization |
|---|---|---|
| Normalizes across | The batch dimension, per feature | The feature dimension, per example |
| Dependent on batch size | Yes -- degrades at small batch sizes | No -- identical for any batch size |
| Needs running stats at inference | Yes (running mean/variance) | No -- computed fresh per example, same as training |
| Typical domain | Convolutional networks, image classification | Transformers, RNNs, sequence models |
| Behavior on variable-length sequences | Awkward -- padding skews batch statistics | Natural -- each position normalized independently |
Note
What's next
Normalization techniques like BatchNorm and LayerNorm are part of the broader engineering toolkit that made very deep networks trainable at all -- alongside residual connections, careful initialization, and the attention-based architectures covered earlier in this track. With the core deep learning and RL toolkit now covered, the remaining frontier is applying these building blocks to the frontier model architectures and training regimes used in practice today.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.