Feature Selection Methods

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Why fewer features can mean a better model

More features isn't automatically better: irrelevant or redundant features add noise that models can overfit to, increase training and inference cost, worsen the curse of dimensionality for distance-based methods, and make models harder to interpret and maintain. Feature selection aims to keep only the features that genuinely contribute predictive signal, discarding the rest — distinct from feature *extraction* methods like PCA, which transform features into new combinations rather than selecting a subset of the originals.

Filter methods: score features independently of any model

Filter methods rank features using a statistical measure of their relationship with the target, computed independently of any specific downstream model — correlation coefficient, mutual information, chi-squared test for categorical features, or ANOVA F-value. They're fast (no model training required per feature) and model-agnostic, but they ignore feature interactions: a feature that looks useless in isolation might become highly predictive in combination with another feature, and a pure filter method can't detect that.

filter_methods.py

python

from sklearn.feature_selection import (
    SelectKBest, f_classif, mutual_info_classif
)
import numpy as np

# ANOVA F-value: measures linear separability of classes per feature
selector_f = SelectKBest(score_func=f_classif, k=10)
X_selected = selector_f.fit_transform(X_train, y_train)

# Mutual information: captures non-linear relationships too
selector_mi = SelectKBest(score_func=mutual_info_classif, k=10)
X_mi_selected = selector_mi.fit_transform(X_train, y_train)

# Correlation-based redundancy check: drop one of each highly correlated pair
corr_matrix = np.corrcoef(X_train, rowvar=False)
high_corr_pairs = np.argwhere((np.abs(corr_matrix) > 0.9) & (corr_matrix != 1.0))

Wrapper methods: let the model decide

Wrapper methods evaluate feature subsets by actually training a model on each candidate subset and measuring performance, which captures feature interactions that filter methods miss, at substantially higher computational cost. Recursive Feature Elimination (RFE) trains a model, ranks features by importance (coefficient magnitude, or impurity-based importance for trees), removes the weakest, and repeats until the target number of features remains. Forward/backward selection incrementally adds or removes one feature at a time based on the resulting validation performance change. Both scale poorly to very high feature counts, since the number of model-training runs grows with the number of features considered.

Embedded methods: selection built into training

Embedded methods perform feature selection as an integral part of model fitting rather than as a separate preprocessing step. L1 (lasso) regularization drives some linear model coefficients exactly to zero as a side effect of the optimization, effectively selecting features during training itself. Tree-based models (random forests, gradient boosting) produce feature importance scores as a natural byproduct of how often and how usefully a feature is used for splits, which can be used to prune low-importance features for a subsequent, leaner model even without needing a wrapper's repeated retraining cycles.

Method type	Captures feature interactions?	Computational cost	Example
Filter	No	Low	Mutual information, chi-squared, correlation
Wrapper	Yes	High	Recursive Feature Elimination, forward/backward selection
Embedded	Partially	Moderate (one training run)	Lasso (L1), tree-based feature importance

Feature selection must happen inside cross-validation, not before it

Selecting features using the full dataset's statistics (including what will become test data) before splitting is a specific, common form of leakage — the selected feature subset itself has been chosen with knowledge of data it will later be 'evaluated' against. As with scaling and imputation, feature selection needs to be refit within each cross-validation fold using only that fold's training data, which scikit-learn's Pipeline makes straightforward to enforce.

Correlated features aren't automatically redundant for every model

Highly correlated features can genuinely destabilize linear models (multicollinearity inflates coefficient variance and can flip signs unpredictably), but tree-based models handle correlated features far more gracefully — they'll simply tend to split on whichever correlated feature happens to be slightly more useful at a given node, without the same instability. The need to prune correlated features is model-dependent, not a universal rule.

What's Next?

Choosing between filter, wrapper, and embedded approaches in practice often comes down to a direct tradeoff between computational cost and the ability to capture feature interactions — a tradeoff worth examining on its own, since it shapes which method fits a given dataset size and modeling budget.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →