Simple linear regression predicts a target from a single input: price from square footage, say. The real world rarely cooperates with that constraint. House price depends on square footage, bedroom count, location, age, and a dozen other things simultaneously. Multivariate regression (more precisely, multiple linear regression) extends the same idea to handle several predictors at once:

y = b0 + b1*x1 + b2*x2 + ... + bp*xp + e

Each xi is a predictor, each bi is a coefficient the model learns, and e is irreducible error. The fitting procedure — minimizing the sum of squared residuals — is the same ordinary least squares machinery as before, just generalized to a matrix equation: b = (X^T X)^-1 X^T y.

Reading coefficients correctly

The single most common mistake with multivariate regression is reading a coefficient as if it operated in isolation. It does not. b1 is the expected change in y for a one-unit increase in x1, holding every other predictor in the model constant.

This matters enormously. A predictor's simple correlation with y can differ sharply from its coefficient in a multivariate model, because the coefficient has already 'controlled for' the other variables. A classic example: in a dataset of ice cream sales, temperature alone may show a strong positive relationship with drowning deaths. Add temperature as a control and the apparent 'ice cream causes drowning' relationship vanishes — both were driven by summer. Multivariate regression lets you isolate the marginal effect of each variable instead of being fooled by confounds.

Note

A coefficient's sign can even flip when you add or remove other predictors — a phenomenon sometimes called Simpson's paradox. Always interpret a coefficient relative to the specific set of other variables in the model, never as a universal truth about that variable's effect on the target.

python

R-squared always goes up — adjusted R-squared doesn't

R-squared measures the fraction of variance in y explained by the model. The problem: adding any predictor, even pure noise, can never decrease R-squared — at worst it adds zero explanatory power, and in finite samples it almost always adds a little by chance. That makes raw R-squared useless for comparing models with different numbers of predictors.

Adjusted R-squared fixes this by penalizing additional predictors:

Adjusted R^2 = 1 - (1 - R^2) * (n - 1) / (n - p - 1)

where n is sample size and p is the number of predictors. Adding a genuinely useful predictor raises adjusted R-squared; adding a useless one lowers it. Use adjusted R-squared, not R-squared, whenever you're comparing models of different sizes.

Multicollinearity: when predictors talk to each other

Multiple regression assumes the predictors are at least somewhat independent of each other. When two or more predictors are highly correlated — square footage and number of rooms, say — the model struggles to assign credit between them. This is multicollinearity.

Symptoms: coefficients with implausible magnitudes or signs, coefficients that swing wildly when you add or remove a single observation, and standard errors that are large even though the model fits the data well overall. The underlying issue is that (X^T X) becomes close to singular — nearly non-invertible — so the solution for b becomes numerically unstable.

The standard diagnostic is the Variance Inflation Factor (VIF), computed by regressing each predictor on all the others. A VIF above 5 to 10 signals a multicollinearity problem worth addressing.

python

Note

Common fixes: drop one of the correlated predictors, combine them into a single composite feature (e.g. total room area instead of sqft and room count separately), use dimensionality reduction like PCA before regressing, or switch to a regularized method like Ridge regression, which handles correlated predictors far more gracefully than ordinary least squares.

Simple vs. multiple regression vs. regularized regression

Method	Predictors	Handles correlated features?	Typical use
Simple linear regression	1	N/A	Single clear driver, exploratory analysis
Multiple (multivariate) regression	Many	Poorly — multicollinearity distorts coefficients	Controlled inference, explaining effects
Ridge / Lasso regression	Many	Well — regularization stabilizes coefficients	Prediction with many correlated features

Checking the assumptions

A multivariate regression's coefficients and p-values are only trustworthy if the standard linear regression assumptions roughly hold: linearity (the relationship is additive and linear in the parameters), homoscedasticity (residual variance is roughly constant across predicted values), independence of errors, and approximately normal residuals for valid inference. Plot residuals against fitted values and against each predictor before trusting the output — a model can have a high R-squared and still violate these assumptions badly enough to make its p-values meaningless.

What's next

Multivariate regression still assumes the relationship between predictors and target is linear in the parameters. When the real relationship is curved, you need either Polynomial Regression — which stays linear in form but adds curved terms — or genuinely Non-Linear Least Squares Fitting, covered next.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →