Regularization: L1 vs L2 Explained
The shared goal: penalize complexity
Both L1 and L2 regularization modify a model's loss function by adding a penalty term based on the magnitude of its weights, discouraging the optimizer from fitting training data using unnecessarily large coefficients — large weights tend to make a model more sensitive to small input changes and more prone to overfitting noise. The two differ in exactly how they measure 'magnitude,' and that difference in geometry produces very different practical behavior.
L2 regularization (ridge): smooth shrinkage
L2 regularization adds a penalty proportional to the sum of squared weights: $L_{\text{total}} = L_{\text{data}} + \lambda \sum_i w_i^2$. Its gradient with respect to a weight is $2\lambda w_i$ — proportional to the weight's current value, meaning the penalty's pull shrinks as the weight approaches zero and never quite pushes it there in finite time. This produces smooth, proportional shrinkage of all weights without driving any of them to exactly zero, which is why L2-regularized models typically keep every feature in play, just with smaller, more conservative coefficients.
L1 regularization (lasso): a sparsity-inducing penalty
L1 regularization adds a penalty proportional to the sum of absolute weight values: $L_{\text{total}} = L_{\text{data}} + \lambda \sum_i |w_i|$. Its gradient magnitude with respect to a weight is constant ($\lambda$, with sign matching the weight), regardless of how small the weight already is — the penalty pulls just as hard near zero as it does far from zero. Combined with the data loss's own gradient, this constant pull is what can drive weights exactly to zero once $\lambda$ is large enough relative to a feature's contribution to reducing data loss, effectively performing feature selection as a side effect of optimization.
from sklearn.linear_model import Ridge, Lasso, ElasticNet
import numpy as np
# L2 (Ridge): shrinks all coefficients smoothly, rarely exactly zero
ridge = Ridge(alpha=1.0).fit(X_train, y_train)
print("Ridge zero coefficients:", np.sum(ridge.coef_ == 0))
# L1 (Lasso): drives some coefficients exactly to zero -> feature selection
lasso = Lasso(alpha=0.1).fit(X_train, y_train)
print("Lasso zero coefficients:", np.sum(lasso.coef_ == 0))
# Increasing lasso's alpha pushes more coefficients to exactly zero
for alpha in [0.01, 0.1, 1.0, 10.0]:
m = Lasso(alpha=alpha).fit(X_train, y_train)
print(f"alpha={alpha}: {np.sum(m.coef_ == 0)} zeroed features")Why L1's geometry produces sparsity and L2's doesn't
| Property | L1 (Lasso) | L2 (Ridge) |
|---|---|---|
| Penalty term | $\lambda \sum |w_i|$ | $\lambda \sum w_i^2$ |
| Produces exact zeros? | Yes | No |
| Behavior with correlated features | Arbitrarily picks one, zeros the rest | Shrinks correlated features together |
| Solution uniqueness | Can be non-unique under high correlation | Always unique (strictly convex penalty) |
| Typical use case | Feature selection, sparse/high-dim data | General-purpose shrinkage, multicollinearity |
Bayesian interpretation: different priors on weights
Both penalties have a clean probabilistic reading: adding an L2 penalty to the loss corresponds exactly to placing a Gaussian prior on the weights and performing maximum a posteriori (MAP) estimation — ridge regression is MAP under a Gaussian prior. An L1 penalty corresponds to a Laplace (double-exponential) prior, which has a sharper peak at zero and heavier tails than a Gaussian — expressing a prior belief that most weights are exactly zero (irrelevant) while a few can be large, which is precisely the sparse-signal assumption L1 encodes.
L1's instability under correlated features
What's Next?
Neither penalty alone is always the right answer — combining both in a single objective is exactly what elastic net does, specifically to get sparsity while handling correlated features more gracefully than lasso alone.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.