Elastic Net Regression
Combining lasso's sparsity with ridge's stability
Elastic net (Zou & Hastie, 2005) was designed to directly address a specific weakness of lasso: when features are highly correlated, lasso tends to arbitrarily select one from the group and zero out the rest, producing unstable, hard-to-interpret selections that can shift dramatically with small data changes. Elastic net combines L1 and L2 penalties in a single objective: $L_{\text{total}} = L_{\text{data}} + \lambda \left[ \alpha \sum_i |w_i| + (1-\alpha) \sum_i w_i^2 \right]$, where $\alpha \in [0,1]$ controls the mix. $\alpha = 1$ recovers pure lasso; $\alpha = 0$ recovers pure ridge; values in between get both sparsity and the smoother, more stable shrinkage of correlated groups that ridge provides.
The grouping effect
Elastic net's key practical property is the grouping effect: when features are highly correlated, elastic net tends to select or shrink them together as a group, assigning them similar coefficients, rather than lasso's tendency to arbitrarily keep one and drop the rest. This happens because the L2 component of the penalty is strictly convex and penalizes large individual coefficients more than several moderate, shared ones — so splitting weight evenly across correlated features is favored over concentrating it all in one, which the pure L1 penalty is indifferent to.
from sklearn.linear_model import ElasticNet, ElasticNetCV
import numpy as np
# l1_ratio corresponds to alpha above: 1.0 = pure lasso, 0.0 = pure ridge
model = ElasticNet(alpha=0.1, l1_ratio=0.5)
model.fit(X_train, y_train)
# ElasticNetCV searches both the overall penalty strength (alpha)
# and the L1/L2 mix (l1_ratio) via cross-validation
cv_model = ElasticNetCV(
l1_ratio=[0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 1.0],
alphas=np.logspace(-3, 1, 50),
cv=5,
max_iter=5000,
)
cv_model.fit(X_train, y_train)
print("best alpha:", cv_model.alpha_, " best l1_ratio:", cv_model.l1_ratio_)Two hyperparameters instead of one
Elastic net trades lasso's single penalty strength $\lambda$ for two interacting hyperparameters — overall penalty strength and the L1/L2 mixing ratio — which means more careful tuning is required to get full benefit, typically via cross-validated grid or random search over both simultaneously, as ElasticNetCV does. This added tuning cost is the main practical tradeoff against elastic net's improved stability: for data without substantial feature correlation, the extra complexity may not buy much over plain lasso or ridge alone.
| Model | l1_ratio equivalent | Produces sparsity? | Stable under correlated features? |
|---|---|---|---|
| Ridge | 0.0 | No | Yes |
| Elastic net (mixed) | 0 < value < 1 | Yes | Yes (grouping effect) |
| Lasso | 1.0 | Yes | No (arbitrary selection among correlated group) |
Elastic net was specifically motivated by genomics data
Don't default to l1_ratio=0.5 without checking
What's Next?
Regularization shapes how a linear model's coefficients behave; the next natural question is how classification models — not just regression — draw their decision boundaries, and what shapes those boundaries beyond regularization alone.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.