Linear Regression

What linear regression actually does

Linear regression learns a function that maps input features to a numeric output. Given historical house data — size, number of rooms, location — it learns to predict price. Given patient vitals, it predicts a blood pressure reading. The relationship it learns is always a weighted sum of the inputs.

For a single input x, the hypothesis is: h(x) = w₁x + w₀

For multiple inputs (multivariate), it becomes: h(x) = w₀ + w₁x₁ + w₂x₂ + ... + wₙxₙ

Every w is a weight — a number the model learns. w₀ is the intercept (also called bias) — the baseline prediction when all features are zero. Training is the process of finding the weight values that make the predictions as accurate as possible.

The loss function: measuring how wrong you are

To train the model, we need a way to measure its error. Linear regression uses squared error loss — for each training example, compute the difference between the prediction and the true value, then square it:

Loss = (y_predicted - y_true)²

Squaring does two things: it makes all errors positive (so over-predictions and under-predictions don't cancel out), and it penalises large errors more heavily than small ones. The total loss over all training examples is the Mean Squared Error (MSE).

The goal of training is to find weights w that minimise MSE.

Finding the optimal weights

There are two ways to find the best weights:

Closed-form solution (analytical): Calculus gives us the exact formula: w* = (XᵀX)⁻¹ Xᵀy — where X is the matrix of training inputs and y is the vector of true outputs. This is exact and fast for small datasets, but computing (XᵀX)⁻¹ becomes very expensive for large datasets or many features.

Gradient descent (iterative): Start with random weights, compute the loss, compute which direction would reduce the loss (the gradient), and take a small step in that direction. Repeat until the loss stops improving. This is slower per step but scales to millions of examples.

Multivariate linear regression

Real problems have many features. A house price model might use size, rooms, age, distance to transport, and neighbourhood rating all at once. Multivariate linear regression handles this naturally — each feature gets its own weight.

With multiple features, the update rule for each weight wᵢ becomes:

wᵢ ← wᵢ + α × Σ(yⱼ - h(xⱼ)) × xⱼᵢ

where α is the learning rate and the sum is over all training examples j. In practice, this is computed as a matrix multiplication, which GPUs handle extremely efficiently.

Overfitting and regularisation

When to use linear regression

Linear regression is your first tool to try when the target is a continuous number. It is fast to train, easy to interpret (each weight tells you the marginal effect of that feature), and provides a strong baseline. If a more complex model can't beat it significantly, the data may genuinely be close to linear.

Use it for: price prediction, demand forecasting, scoring models, any task where you need an explainable model and a linear relationship is plausible.

What's next

Linear regression predicts continuous values. Logistic regression adapts the same framework to predict probabilities — turning it into a classifier. The next lesson covers how the sigmoid function and cross-entropy loss transform regression into the building block of most classification pipelines.