Multicollinearity occurs when two or more predictor variables in a regression model are highly correlated with each other. It doesn't necessarily hurt a model's raw predictive accuracy — but it devastates the model's interpretability and the stability of its coefficients, which is often the entire point of using a linear model in the first place.

Linear regression estimates each coefficient as the effect of one predictor holding all others constant. If two predictors move together almost perfectly — say, square footage and number of rooms in a housing dataset — the model has no way to tell which one is driving the price. The math can still produce a best-fit line, but the individual coefficients become wildly unstable: tiny changes in the data can flip their signs or swing their magnitudes.

A concrete intuition

Imagine predicting salary from years of experience and age. These two are naturally highly correlated — people with more experience tend to be older. If you fit a regression with both, the model has to somehow split credit between them even though they carry almost the same signal. You might get a coefficient on experience that's large and positive and a coefficient on age that's negative and seemingly nonsensical, purely because the optimizer found one particular way to split overlapping information that fits the training data, not because age genuinely has a negative effect on salary.

Add one more data point and refit — the split between the two coefficients can change dramatically even though the model's overall predictions barely move. That instability is the signature of multicollinearity.

Note

A regression under multicollinearity can still have unbiased coefficient estimates on average — the problem is variance, not bias. The standard errors on the affected coefficients balloon, which means your confidence intervals widen and your p-values become unreliable. You might conclude a genuinely important predictor is "not statistically significant" purely because it shares variance with a correlated partner, not because it lacks real explanatory power.

The Variance Inflation Factor

The Variance Inflation Factor (VIF) quantifies how much a predictor's coefficient variance is inflated due to its correlation with other predictors. To compute it for predictor $x_j$:

1. Regress $x_j$ on all the other predictors in the model (not on the target $y$).
2. Take the $R^2$ of that regression — call it $R_j^2$. It measures how well the other predictors explain $x_j$.
3. Compute $\text{VIF}_j = \dfrac{1}{1 - R_j^2}$.

If $x_j$ is completely uncorrelated with the other predictors, $R_j^2 = 0$ and $\text{VIF}_j = 1$ — no inflation. As $R_j^2$ approaches 1 (meaning $x_j$ is almost perfectly predictable from the other features), VIF explodes toward infinity. A VIF of 5 means the variance of that coefficient's estimate is 5 times larger than it would be if the predictor were uncorrelated with everything else.

python

Reading VIF values

VIF range	Interpretation	Typical action
1	No correlation with other predictors	No action needed
1 – 5	Moderate correlation	Usually fine, keep monitoring
5 – 10	High correlation, meaningful variance inflation	Investigate; consider removing or combining
> 10	Severe multicollinearity	Address before trusting coefficients

Fixing multicollinearity

Several standard remedies, roughly in order of how much they change your model:

Drop one of the correlated variables — the simplest fix when two features are measuring almost the same thing (e.g., keep square footage, drop number of rooms).

Combine correlated features — merge them into a single composite via domain knowledge (a "size score") or dimensionality reduction like PCA, which creates orthogonal (uncorrelated) components by construction.

Regularization (Ridge regression) — adding an L2 penalty shrinks coefficients and is mathematically well-suited to handling correlated predictors, since it stabilizes the solution without requiring you to drop any variable outright.

Collect more data — multicollinearity is partly a sample-size issue; more varied data can sometimes break apart correlations that happen to hold in a small sample.

The right fix depends on whether you care about prediction (where moderate multicollinearity often doesn't matter much) or inference (where it can completely undermine the conclusions you draw from coefficients).

Note

If your only goal is predictive accuracy and you're not interpreting individual coefficients, multicollinearity is often harmless — the model will still make good predictions on data that resembles the training distribution, because it doesn't need to definitively assign credit between correlated features. It becomes a serious problem specifically when you want to explain *why* the model predicts what it does, or when the correlated relationship might not hold in future data (e.g., if your test set has a different mix of experience and age than training).

What's next

Multicollinearity is a specific symptom of a broader theme: a model's internal validity (can you trust its coefficients) is a different question from its external validity (does it generalize). The next lesson on cross-validation strategies addresses the generalization half of that equation — how to design evaluation schemes that actually measure whether your model works on new data.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →