Neural Network Hyperparameters Explained

Hyperparameters vs. parameters

A neural network's parameters — weights and biases — are learned automatically by gradient descent during training. Its hyperparameters are everything you set before training begins that controls how that learning happens: how big a step to take, how many examples to look at per step, how much to randomly disable, and how to initialize everything before learning starts.

Get the hyperparameters wrong and the exact same architecture can fail to train at all, train painfully slowly, or overfit badly — even though the model's theoretical capacity never changed. Understanding what each one actually does mechanically (not just "tune it with a sweep") makes debugging a stalled or diverging training run far faster.

Learning rate: the most important knob

The learning rate $\eta$ scales how large a step gradient descent takes at each update: $w \leftarrow w - \eta \nabla L(w)$. It is widely considered the single most important hyperparameter to get right.

Too high, and updates overshoot the minimum of the loss landscape — the loss can oscillate wildly or diverge to infinity/NaN. Too low, and training crawls: the model technically moves toward a good solution but takes so many steps that it looks stalled, or gets permanently stuck in a shallow local plateau it could have escaped with a bigger step. The textbook diagnostic: plot the loss curve. A spiky, exploding loss means the learning rate is too high; a flat, barely-moving loss (that isn't already near its best achievable value) means it's too low.

Batch size: a tradeoff between gradient quality and speed

The batch size determines how many training examples are used to compute each gradient estimate before updating weights. A larger batch gives a gradient estimate that's closer to the true gradient over the full dataset (less noisy), and modern hardware can process larger batches more efficiently in parallel. A smaller batch gives a noisier gradient estimate, but that noise can actually help — it acts as a mild regularizer, nudging the optimizer out of sharp, narrow minima that tend to generalize poorly, toward flatter minima that tend to generalize better.

In practice, very large batches often need a correspondingly larger learning rate to train in a similar number of steps (since each step covers more data, you can afford to move further), and very small batches need a smaller learning rate to avoid instability from the noisier gradient. Batch size and learning rate are not independent knobs — changing one usually means revisiting the other.

Dropout: regularization by random deletion

Dropout randomly deactivates a fraction $p$ of neurons' outputs during each training step (commonly $p = 0.2$ to $0.5$), forcing the network to not rely too heavily on any single neuron or narrow combination of neurons. Because a different random subset is dropped on every forward pass, the network effectively trains a huge ensemble of overlapping sub-networks that share weights, and that ensembling effect is a strong regularizer against overfitting.

At test time, dropout is turned off entirely — all neurons are active — but outputs are typically rescaled (or, in the "inverted dropout" convention used by most frameworks, scaling is baked into training instead) so that the expected magnitude of activations matches what the network saw during training. Forgetting to disable dropout at inference time is a common and confusing bug: it makes evaluation outputs nondeterministic and the model perform worse than it actually is.

Weight initialization: setting the starting point

Before training starts, weights must be initialized to some value — and the choice matters more than it might seem. Initializing all weights to zero is catastrophic: every neuron in a layer receives identical gradients and stays identical forever, so the network can never break symmetry and effectively collapses to a single neuron per layer.

Random initialization with the wrong scale causes its own problems. Too large, and activations and gradients can explode as they compound through layers (each layer's output feeds the next with ever-growing magnitude). Too small, and activations shrink toward zero layer by layer — a form of vanishing signal even before you've picked an activation function. Xavier/Glorot initialization scales initial weights based on the number of inputs and outputs of a layer to keep activation variance roughly stable across layers, and is well-suited to sigmoid/tanh activations. He initialization uses a similar idea but accounts for the fact that ReLU zeroes out half its inputs on average, making it the standard choice for ReLU-family networks.

Hyperparameter cheat sheet

What's next

With activation functions and training hyperparameters covered, the next lesson zooms out to architecture: the Multilayer Perceptron, how stacking perceptron-like units into layers gives a network the ability to approximate far more complex functions than any single unit could alone.