Activation Functions Deep Dive: ReLU, ELU, Sigmoid, Softmax & Swish
Why the activation function matters so much
Without a nonlinear activation function, a neural network — no matter how many layers it has — collapses mathematically into a single linear transformation. Stacking linear layers just produces another linear layer; you'd gain nothing from depth. The activation function is what gives a network the ability to approximate curved, complex decision boundaries and functions.
But the choice of activation function isn't just "linear vs. nonlinear." Different nonlinearities have dramatically different training dynamics — some cause gradients to vanish in deep networks, some cause neurons to permanently die, and some trade a bit of computational cost for noticeably better convergence. Picking the right one is one of the most consequential and most overlooked architecture decisions.
Sigmoid: the classical choice, and its fatal flaw
The sigmoid function squashes any real input into the range $(0, 1)$: $\sigma(z) = \frac{1}{1 + e^{-z}}$. It was the default activation in early neural networks because its output has a clean probabilistic interpretation and it's smooth and differentiable everywhere.
Its fatal flaw is the vanishing gradient problem. The derivative of sigmoid is $\sigma'(z) = \sigma(z)(1 - \sigma(z))$, which has a maximum value of just 0.25 at $z=0$ and approaches zero as $|z|$ grows large — the function saturates (flattens) for large positive or negative inputs. During backpropagation, gradients are multiplied layer by layer going backward. Multiply several numbers each at most 0.25 together across many layers, and the gradient reaching early layers shrinks toward zero exponentially. Those early layers then barely update — they stop learning, especially in deep networks.
Note
ReLU: simple, fast, and mostly non-saturating
The Rectified Linear Unit is defined simply as $\text{ReLU}(z) = \max(0, z)$. Its derivative is 1 for any positive input and 0 for any negative input. Because the gradient for active (positive) neurons is a constant 1 rather than a shrinking fraction, gradients pass through many layers without multiplicatively vanishing — this single property is largely responsible for making genuinely deep networks trainable.
ReLU is also computationally trivial (just a max operation, no exponentials), which matters enormously at the scale of billions of activations per forward pass. Its weakness is the dying ReLU problem: if a neuron's weighted input is consistently negative across the training set (often due to a large negative bias update, or a too-high learning rate pushing weights into a bad region), its gradient is permanently 0, and it stops updating forever — it's "dead" and contributes nothing to the network from then on.
Leaky ReLU and ELU: fixing dead neurons
Leaky ReLU patches the dying neuron problem by allowing a small, non-zero gradient when the input is negative: $\text{LeakyReLU}(z) = z$ if $z > 0$, else $\alpha z$ for a small constant like $\alpha = 0.01$. This keeps a trickle of gradient flowing even for negative inputs, so neurons can recover instead of dying permanently.
ELU (Exponential Linear Unit) goes further: for negative inputs it uses a smooth exponential curve instead of a straight line, $\text{ELU}(z) = z$ if $z > 0$, else $\alpha(e^z - 1)$. This makes the function smooth everywhere (including at $z=0$, where Leaky ReLU still has a sharp kink) and pushes mean activations closer to zero, which tends to speed up convergence. The tradeoff is a small computational cost from the exponential.
Swish and the self-gated activations
Swish, $\text{Swish}(z) = z \cdot \sigma(z)$, multiplies the input by its own sigmoid — a form of self-gating where the function smoothly decides how much of $z$ to pass through. Unlike ReLU, Swish is smooth everywhere and slightly non-monotonic (it dips slightly below zero for small negative inputs before flattening), which empirically helps optimization in some deep architectures. It was discovered partly through automated search over candidate activation functions and has since been adopted in several modern architectures (it's closely related to GELU, used in transformers like BERT and GPT). The cost is extra compute per activation compared to plain ReLU, which matters when you're running billions of activations.
Softmax: not a hidden-layer activation
Softmax is categorically different from the others — it's not used in hidden layers at all. It converts a vector of raw scores (logits) into a probability distribution over classes: $\text{softmax}(z)_i = \dfrac{e^{z_i}}{\sum_j e^{z_j}}$. Every output is positive and the full vector sums to 1, making it the standard choice for the output layer of a multi-class classifier. It's a generalization of sigmoid to more than two classes — in fact, softmax over two classes reduces mathematically to sigmoid applied to the logit difference.
Activation function comparison
| Function | Range | Vanishing gradient risk | Common use |
|---|---|---|---|
| Sigmoid | (0, 1) | High — saturates at both ends | Binary output layer, gates in LSTMs |
| Softmax | (0, 1), sums to 1 | N/A — output layer only | Multi-class output layer |
| ReLU | [0, ∞) | Low, but neurons can die | Default for hidden layers, CNNs |
| Leaky ReLU | (-∞, ∞) | Low, mitigates dying neurons | Hidden layers where ReLU underperforms |
| ELU | (-α, ∞) | Low, smooth negative region | Hidden layers needing faster convergence |
| Swish / GELU | (~-0.28, ∞) | Low, smooth and self-gated | Transformers, modern deep architectures |
Note
What's next
Activation functions are one knob among many that determine how well a network trains. The next lesson, Neural Network Hyperparameters Explained, covers the other major knobs — learning rate, batch size, dropout, and weight initialization — and how each one actually changes the training dynamics you've just seen gradients move through.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.