Neural Networks

From one neuron to many

A single perceptron can only draw a straight line through data — it famously cannot solve the XOR problem. The breakthrough insight was simple: stack perceptrons in layers, and suddenly the network can learn curved, complex decision boundaries that no single neuron could ever represent.

A neural network is just a directed graph of neurons. Inputs flow in one direction — forward — through one or more hidden layers, and produce an output. Each connection between neurons has a weight. Each neuron has a bias. The network learns by adjusting every single one of these values until its outputs match the training data.

Anatomy of a neural network

Every network has three kinds of layers:

Input layer — not really neurons, just placeholders. One node per feature in your data. If you're classifying 28×28 pixel images, you have 784 input nodes.

Hidden layers — where the actual computation happens. Each hidden neuron receives every output from the previous layer, multiplies by its weights, adds a bias, and passes the result through an activation function. You can have one hidden layer or dozens (the latter is what "deep" in deep learning refers to).

Output layer — the final prediction. One output node for regression (predict a price), one per class for multi-class classification (is this a cat, dog, or bird?).

How information flows forward

The forward pass is the network making a prediction. For each neuron in a hidden layer, the computation is identical to a perceptron:

1. Multiply each incoming value by its weight
2. Sum everything up and add the bias
3. Pass the result through an activation function

The output of that activation becomes the input to every neuron in the next layer. This cascades all the way to the output layer, where the final numbers are your prediction.

How the network learns: backpropagation

After a forward pass, the network has made a prediction. We compare it to the true answer using a loss function — a number that measures how wrong the prediction was. The goal of training is to make that number as small as possible.

Backpropagation is the algorithm that figures out how to adjust every weight and bias to reduce the loss. It works by calculating the gradient of the loss with respect to each parameter, starting from the output layer and moving backwards through the network — hence the name.

Once we have the gradients, we update each weight using gradient descent:

w = w - learning_rate × gradient

Do this for thousands or millions of examples, and the network gradually gets better and better at the task.

Build a network — add layers and run a forward pass

Depth vs. width

A wider network has more neurons per layer. A deeper network has more layers. Both increase capacity, but depth matters more in practice.

Deeper networks can learn hierarchical representations more efficiently — you can represent complex functions with far fewer total parameters in a deep network than in a shallow one. This is why modern AI systems are deep: GPT-4 has 96 transformer layers, not one enormous flat layer.

The tradeoff: deeper networks are harder to train. Gradients tend to shrink as they travel backwards through many layers — the vanishing gradient problem — making early layers learn very slowly. Techniques like residual connections (used in ResNets and transformers) solve this by giving gradients a shortcut path back to early layers.

Overfitting: when a network memorises instead of learns

A network with enough neurons can memorise your entire training set — achieving 100% accuracy on training data while failing completely on new examples. This is overfitting.

The fixes: dropout (randomly disable neurons during training, forcing the network to develop redundant representations), regularisation (penalise large weights), and simply collecting more data. You also always hold out a validation set that the network never trains on, so you can detect overfitting early.

What's next

Neural networks are the engine behind everything in modern AI — CNNs for images, RNNs for sequences, and transformers for language all share this same forward-pass / backprop core. The next lesson covers how training actually works in practice: loss functions, gradient descent variants, learning rates, and what an epoch is.