Wrapper vs Filter vs Embedded Feature Selection

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Three philosophies for the same goal

Filter, wrapper, and embedded methods all aim to reduce a feature set to the ones that matter, but they differ fundamentally in whether — and how — they involve the downstream model in the selection decision. A filter method never trains the final model during selection; a wrapper method trains it repeatedly, once per candidate subset; an embedded method trains it once, with selection built directly into that single training run. This ordering also roughly tracks cost: filters are cheapest, wrappers are most expensive, embedded methods sit in between.

Filter methods in depth

Filter methods compute a relevance score per feature using only the feature's relationship with the target — no model involved. Common scores: Pearson correlation (linear relationships, numeric features), ANOVA F-value (class separability), chi-squared (categorical feature vs. categorical target independence), and mutual information (captures non-linear dependence, at the cost of needing more data to estimate reliably). Because each feature is scored in isolation, filters are blind to interaction effects — XOR-like relationships, where a feature only matters in combination with another, will often score near zero and get discarded, even though it's genuinely useful in combination.

three_approaches.py

python

from sklearn.feature_selection import SelectKBest, f_classif, RFE
from sklearn.linear_model import LogisticRegression, Lasso
from sklearn.ensemble import RandomForestClassifier

# FILTER: score features independent of any model
filter_selector = SelectKBest(score_func=f_classif, k=15)
X_filtered = filter_selector.fit_transform(X_train, y_train)

# WRAPPER: repeatedly train a model, eliminate weakest features, repeat
estimator = LogisticRegression(max_iter=1000)
wrapper_selector = RFE(estimator, n_features_to_select=15, step=1)
X_wrapped = wrapper_selector.fit_transform(X_train, y_train)

# EMBEDDED: selection happens as a side effect of one training run
lasso = Lasso(alpha=0.01).fit(X_train, y_train)
selected_features = X_train.columns[lasso.coef_ != 0]

rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)
importances = rf.feature_importances_  # usable for post-hoc pruning

Wrapper methods in depth

Recursive Feature Elimination, the most common wrapper, fits a model, ranks features by the model's own importance measure (coefficient magnitude for linear models, impurity reduction for trees), discards the weakest, and refits — repeating until the desired feature count is reached. Because the model is retrained at every step using the actual candidate feature subset, wrappers directly optimize for what you actually care about (held-out performance of the specific model you'll deploy) rather than a proxy statistic. The cost is combinatorial in spirit: with $p$ features, exhaustively trying all subsets is $2^p$, so RFE and similar greedy methods are an approximation, not a guarantee of finding the globally optimal subset.

Embedded methods in depth

Embedded methods fold selection into the training objective itself. L1 regularization (lasso) adds a penalty $\lambda \sum |w_i|$ to the loss, which — unlike L2's smooth shrinkage — has a geometry that drives many coefficients exactly to zero as $\lambda$ increases, performing selection as training proceeds rather than afterward. Tree ensembles don't have an explicit sparsity-inducing penalty, but their natural byproduct (how much each feature reduces impurity across all splits, aggregated over all trees) serves a similar purpose: low-importance features can be pruned using a single trained model, without the repeated retraining a wrapper requires.

Property	Filter	Wrapper	Embedded
Model-aware	No	Yes (uses actual model)	Yes (built into training)
Captures interactions	No	Yes	Partially
Training runs needed	0	Many (one per candidate subset)	1
Risk of overfitting selection	Low	Higher (many evaluations on same val set)	Moderate

A practical sequencing: filter first, wrapper or embedded second

A common and computationally sensible pipeline is to use a cheap filter method to cut an enormous feature set (thousands of columns) down to a more manageable size, then apply a wrapper or embedded method to that reduced set to capture interactions the filter missed. This avoids running an expensive wrapper over every original feature while still getting the interaction-awareness benefit where it matters most.

Wrapper methods can overfit to the validation set with repeated evaluation

Because RFE and similar wrapper methods evaluate many candidate subsets against the same validation data, there's a real risk of selecting a subset that happens to perform well on that specific validation split by chance, not because it generalizes — the same multiple-comparisons problem that affects extensive hyperparameter search. Nested cross-validation, where feature selection happens inside an inner loop and a separate outer loop estimates generalization, is the rigorous (if expensive) way to guard against this.

What's Next?

Embedded methods built on regularization naturally raise the question of how L1 and L2 regularization actually differ mechanically, and why one induces sparsity while the other doesn't — a distinction worth understanding directly rather than taking on faith.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →