Weight Initialization Strategies

Before training even starts, initialization has already decided a lot

A neural network's weights need some starting values before gradient descent can begin adjusting them. It's tempting to think the starting point barely matters — training will sort it out. In practice, it matters enormously: with the wrong initialization, signals passing through a deep network can shrink toward zero or grow without bound as they pass through each layer, purely as a consequence of how the weights were set — before any data has even been seen, and before a single useful gradient step can happen.

This is why weight initialization is treated as a first-class design decision, with specific schemes derived mathematically to keep signal variance roughly stable layer to layer, rather than left as an afterthought like 'just use small random numbers.'

Why naive initialization breaks deep networks

Suppose every weight is initialized from a standard normal distribution (mean 0, variance 1), and a layer computes a weighted sum over its inputs. The variance of that sum grows with the number of inputs — each layer's output variance scales up roughly in proportion to the number of units feeding into it. Stack enough layers like this, and the activations' variance can explode exponentially with depth, often with outputs eventually overflowing to inf or saturating activation functions into flat zero-gradient regions.

Go the other direction and initialize weights too small, and the opposite happens: each layer shrinks the signal's variance, and by the time it reaches a deep layer, activations (and later, gradients flowing backward) have collapsed toward zero — the classic vanishing gradient problem. Either failure mode means the network can't learn: gradients are either meaninglessly huge or too small to produce any real update.

Xavier / Glorot initialization: for sigmoid and tanh

Xavier initialization (also called Glorot initialization, after its 2010 paper) sets each weight's variance to 1/n_in (or, in the commonly used symmetric form, 2/(n_in + n_out)), where n_in and n_out are the number of input and output units for that layer. This keeps the variance of activations roughly stable across layers specifically when paired with activation functions that are roughly linear near zero, like sigmoid and tanh.

Weights are typically drawn either from a uniform distribution scaled to this variance, or from a normal distribution with standard deviation √(2/(n_in + n_out)).

He initialization: for ReLU

ReLU zeroes out roughly half its inputs (every negative value), which effectively halves the variance passed forward compared to a linear activation. Xavier initialization doesn't account for this, and networks using ReLU with Xavier init tend to see activation variance quietly shrink layer by layer anyway.

He initialization (2015, named after Kaiming He) compensates directly: it sets weight variance to 2/n_in — exactly double the Xavier formula's input-only term — to offset the variance ReLU removes. This became the standard choice as ReLU and its variants (Leaky ReLU, GELU) replaced sigmoid/tanh as the default activation in most modern architectures.

Initialization strategies compared

What's next

Weight initialization sets the starting conditions training optimizes from, learning rate schedules and gradient clipping govern the stability of each update, and mixed precision and the Adam optimizer shape the mechanics of the update itself — together, this set of lessons covers the practical levers that determine whether a deep network trains successfully at all, independent of the architecture or dataset. From here, the natural next step is applying these levers in combination on a real architecture, returning to topics like hyperparameter tuning with this fuller picture of what's actually being tuned.