Ordinary least squares picks the coefficients that minimize the sum of squared residuals, with no other constraint. When you have many predictors, especially correlated ones, this freedom becomes a liability: the model can assign huge, offsetting coefficients to correlated features (one large positive, one large negative) that fit the training data perfectly but generalize terribly. It can also latch onto noise in features that have no real relationship to the target, simply because doing so reduces training error by a tiny amount.

Regularization adds a penalty for coefficient size directly into the loss function, trading a little bit of training fit for a model that generalizes better. Instead of minimizing only the sum of squared residuals, you minimize residuals plus a penalty term.

Ridge regression: L2 penalty

Ridge regression adds a penalty proportional to the sum of squared coefficients:

Loss = sum((y - y_hat)^2) + alpha * sum(bi^2)

The alpha (sometimes called lambda) controls the strength of the penalty. As alpha increases, coefficients shrink toward zero — but, crucially, they rarely reach exactly zero. Ridge keeps every feature in the model, just with dampened influence. This makes Ridge particularly effective against multicollinearity: when two predictors are highly correlated, Ridge tends to split the coefficient roughly evenly between them rather than letting OLS assign an unstable, arbitrarily large coefficient to one and a compensating negative one to the other.

Lasso regression: L1 penalty

Lasso (Least Absolute Shrinkage and Selection Operator) uses the sum of absolute coefficient values instead of squared values:

Loss = sum((y - y_hat)^2) + alpha * sum(|bi|)

This small change in the penalty's geometry has a big practical consequence: Lasso drives some coefficients exactly to zero, effectively performing automatic feature selection. The intuition is geometric — the L1 penalty's constraint region is a diamond with sharp corners on the axes, and the optimal solution tends to land on one of those corners, where some coefficients are exactly zero. The L2 penalty's constraint region is a smooth circle with no corners, so solutions rarely land exactly on an axis.

Note

Picture the loss surface's contours overlaid on the penalty's constraint boundary. The L1 (diamond) boundary has corners exactly on the axes — a disproportionate share of the boundary's area is concentrated at those corners, so the optimal point is likely to land there, zeroing out one or more coefficients. The L2 (circular) boundary is smooth everywhere, so the optimum typically lands at some interior point on the circle where every coefficient is small but non-zero.

python

Note

Both Ridge and Lasso penalize coefficients based on their raw magnitude. If features are on different scales, the penalty will unfairly shrink coefficients for features measured in large units (e.g. income in dollars) much more than features measured in small units (e.g. a 0-1 ratio), regardless of their actual importance. Always standardize features to zero mean and unit variance before fitting either model.

Elastic Net: getting both behaviors

Elastic Net combines both penalties with a mixing parameter:

Loss = sum((y - y_hat)^2) + alpha * (r * sum(|bi|) + (1-r) * sum(bi^2))

where r (often called l1_ratio) blends between pure Lasso (r=1) and pure Ridge (r=0). Elastic Net is useful when you have many correlated features and want some feature selection (Lasso's strength) without Lasso's tendency to arbitrarily pick just one feature from a correlated group and zero out the rest. In practice, when predictors are highly correlated, Elastic Net often outperforms pure Lasso because it can keep or drop groups of correlated features together rather than making an unstable arbitrary choice.

Ridge vs. Lasso vs. Elastic Net

Property	Ridge (L2)	Lasso (L1)	Elastic Net
Coefficients can reach exactly zero?	No	Yes	Yes
Performs feature selection?	No	Yes	Yes
Behavior with correlated features	Shrinks correlated groups together	Arbitrarily picks one, zeroes the rest	Keeps or drops correlated groups together
Best when	Many small/moderate effects, all relevant	Few truly relevant features among many	Many correlated, partially relevant features

Choosing alpha

The regularization strength alpha is a hyperparameter, not something the model learns — it must be chosen via cross-validation. sklearn provides RidgeCV and LassoCV, which automatically search a range of alpha values and select the one that minimizes cross-validated error. As a rule of thumb: alpha too small and you're back to overfitting OLS; alpha too large and the model underfits, shrinking every coefficient toward zero regardless of actual signal.

What's next

Ridge and Lasso solve overfitting in linear models through regularization. Polynomial Regression tackles a different problem: how to let a fundamentally linear modeling technique capture curved relationships — and why that flexibility reintroduces the overfitting risk that regularization is often used to control.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →