How Training Works
The feedback loop that makes learning possible
A model doesn't learn by being told the right answer. It learns by making predictions, measuring how wrong those predictions are, and adjusting weights to be slightly less wrong next time.
This cycle — predict, measure error, adjust, repeat — is the entire engine of machine learning. Everything else (architectures, optimizers, regularization techniques) is just refinement of this core loop.
The three things you need to understand: loss functions, gradient descent, and learning rate.
Loss functions: putting a number on "how wrong"
A loss function converts the gap between prediction and reality into a single number. The higher the number, the worse the model is doing. Training is just the process of making this number smaller.
Common loss functions:
Mean Squared Error (MSE) — for regression problems.
Loss = average of (prediction − actual)² across all examples.
Squaring penalizes big mistakes more than small ones.
Cross-Entropy Loss — for classification problems.
Penalizes confident wrong predictions harshly.
If the model says "99% sure this is a cat" and it's a dog, the loss is enormous.
If it says "51% sure this is a cat" and it's wrong, the loss is much smaller.
The goal of training is always the same: find the weights that minimize the loss function across your entire training dataset.
Why validation loss matters more than training loss
Gradient descent: rolling downhill
Imagine the loss function as a landscape of hills and valleys. Your model starts at a random point on that landscape. Gradient descent is how it finds the valley.
The gradient is the slope of the loss function at your current position — it tells you which direction is "uphill." To reduce loss, you step in the opposite direction (downhill).
The update rule:new_weight = old_weight − (learning_rate × gradient)
You repeat this for every parameter in the model, thousands of times.
Three variants:
- Batch gradient descent: compute gradient over entire dataset before updating. Stable but slow.
- Stochastic gradient descent (SGD): update after every single example. Fast but noisy.
- Mini-batch gradient descent: update after a small batch (32–256 examples). The standard. Gets the benefits of both.
Gradient Descent Optimization
Watch how different learning rates affect gradient descent optimization in real time using the simulator below.
📉 Gradient Descent Optimization
InteractiveLearning rate: the most important hyperparameter
The learning rate controls how big each step downhill is.
Too high: you overshoot the valley, bounce around, never converge. Loss oscillates or diverges entirely.
Too low: you crawl toward the minimum. Training takes forever.
Just right: steady descent, convergence in reasonable time.
Finding the right learning rate is more art than science. Common approaches:
- Start with 0.001 and see what happens
- Use a learning rate scheduler (decay LR over time)
- Use adaptive optimizers like Adam, which adjust LR per parameter automatically
Adam is the default choice for most modern neural networks. It maintains a moving average of gradients and adapts the learning rate for each weight individually.
What actually happens in an epoch
One epoch = one complete pass through your entire training dataset.
The process per epoch:
1. Shuffle the dataset
2. Split into mini-batches
3. For each batch: forward pass → compute loss → backward pass (compute gradients) → update weights
4. Track average loss across the epoch
You typically train for tens to hundreds of epochs. Early epochs show dramatic improvement. Later epochs show diminishing returns. Stop when validation loss stops improving (or starts getting worse — that's overfitting).
Overfitting vs. underfitting
Underfitting: the model hasn't learned enough. Loss is high on both training and validation data. The model is too simple, or training was too short.
Overfitting: the model has learned the training data too well — including its noise. Loss is low on training data but high on validation data. The model can't generalize.
Signs of overfitting: training loss keeps decreasing while validation loss starts increasing.
Common fixes:
- More data (most effective)
- Dropout: randomly zero out neurons during training, forces redundancy
- L2 regularization: adds a penalty to the loss for large weight values
- Early stopping: stop training when validation loss stops improving
- Data augmentation: artificially expand dataset with transformations
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.