The Multilayer Perceptron (MLP)

From one unit to a network

A single perceptron draws one straight decision line (or, in higher dimensions, one flat hyperplane) through the input space. That's its entire expressive power — it can separate classes that happen to be linearly separable, and nothing more. The Multilayer Perceptron (MLP) overcomes this by organizing many perceptron-like units into successive layers, where the output of one layer becomes the input of the next.

An MLP has three kinds of layers: an input layer that simply holds the raw features, one or more hidden layers where the actual computation and representation-building happens, and an output layer that produces the final prediction. Every unit in one layer typically connects to every unit in the next layer — this is why MLPs are also called "fully connected" or "dense" networks.

What a hidden layer actually buys you

Each hidden unit computes the same basic operation as a perceptron: a weighted sum of its inputs, plus a bias, passed through a nonlinear activation function. The critical difference from a single perceptron is that the hidden layer produces several of these transformed values simultaneously, and the output layer then combines *those* transformed values — not the raw inputs — into its final decision.

This matters because each hidden unit can learn to detect a different pattern or "carve" a different region of input space, and the output layer combines these regions by voting on or weighting them together. A single hidden layer with enough units can combine straight-line decision boundaries from multiple hidden units into a genuinely curved, complex boundary — something no single perceptron could ever produce alone.

Solving XOR: the problem a single perceptron couldn't

The XOR function is the textbook example of what a single perceptron cannot do: its four input/output pairs — (0,0)→0, (0,1)→1, (1,0)→1, (1,1)→0 — cannot be separated by any single straight line. No matter how you set a single perceptron's weights and bias, at least one point ends up misclassified.

An MLP with just two hidden units solves this easily. Each hidden unit can learn its own straight-line boundary (e.g., one roughly implementing OR, another roughly implementing NAND), and the output layer combines those two intermediate results with an AND-like operation. The combination of two linear boundaries, passed through hidden-layer nonlinearities, produces a boundary that isn't a single straight line at all — it's effectively two lines working together, which is exactly what XOR requires.

The universal approximation intuition

The universal approximation theorem states that a feedforward network with a single hidden layer containing a finite but sufficiently large number of units, using a reasonable nonlinear activation function, can approximate any continuous function on a bounded input region to arbitrary precision. In plain terms: a wide enough one-hidden-layer MLP can, in theory, represent essentially any smooth function you care about.

The important caveat is that this is an existence proof, not a practical recipe. It says such a network exists, not that gradient descent will find it, and not that it will be *efficient*. In practice, making a network deeper (more layers, each moderately wide) rather than wider (one layer, enormously wide) tends to represent complex functions far more efficiently and is dramatically easier to train — which is precisely why modern deep learning favors depth over single giant hidden layers.

How an MLP actually learns: backpropagation in one paragraph

Training an MLP means adjusting every weight in every layer to reduce a loss function measuring prediction error. Backpropagation computes how much each weight contributed to that error by applying the chain rule backward through the network, layer by layer, starting from the output. Gradients computed at the output layer get propagated backward to the last hidden layer, which uses them to compute its own gradients, which get propagated further back, and so on until every weight in the network has a gradient telling it which direction to move to reduce the loss. Gradient descent then nudges every weight slightly in that direction, and the process repeats across many training examples until the loss stops improving meaningfully.

Single perceptron vs. multilayer perceptron

Visualizing a multilayer network

Explore how signals flow forward through an input layer, hidden layer, and output layer, and how each connection's weight shapes the final prediction.

What's next

Fully connected MLPs treat every input feature as independent, which works fine for tabular data but scales poorly for images, where spatial structure (nearby pixels relate to each other) matters enormously. The next lesson on CNN building blocks introduces convolution and pooling — the mechanisms that let networks exploit that spatial structure directly instead of connecting every pixel to every neuron.