Feature Scaling: Normalization vs Standardization
Why scale matters at all
Many algorithms are mathematically sensitive to the raw numeric scale of input features, not just their relative information content. A feature measured in dollars (range: 0-500,000) and a feature measured as a ratio (range: 0-1) will dominate a Euclidean-distance calculation, a gradient-descent step, or a regularization penalty very differently depending on which is larger in magnitude — purely as an artifact of units, with no relationship to which feature is actually more predictive. Scaling puts features on comparable footing before the algorithm sees them.
Normalization (min-max scaling)
Min-max normalization rescales a feature linearly into a fixed range, typically $[0,1]$: $x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$. It preserves the original distribution's shape exactly, just rescaled and shifted. Its weakness is sensitivity to outliers — a single extreme value stretches the denominator and compresses every other value toward 0, which can destroy useful resolution among the bulk of normal-range data.
Standardization (z-score scaling)
Standardization centers a feature at mean 0 with unit variance: $x' = \frac{x - \mu}{\sigma}$. Unlike min-max normalization, it doesn't bound values to a fixed range — standardized values can exceed [-3, 3] for outliers — but it's far less distorted by a handful of extreme points, since $\mu$ and $\sigma$ are influenced by, but not dictated by, single outliers the way min and max are. Standardization is the default choice for algorithms that assume roughly Gaussian-distributed inputs (linear/logistic regression with regularization, PCA, SVMs, and most neural network inputs), since it aligns naturally with weight initialization schemes and gradient-based optimization dynamics.
import numpy as np
def min_max_scale(X, feature_min=0, feature_max=1):
X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0))
return X_std * (feature_max - feature_min) + feature_min
def standardize(X):
return (X - X.mean(axis=0)) / X.std(axis=0)
# Critical: fit scaling parameters on TRAIN only, then apply to val/test
# using those same parameters -- never recompute min/max/mean/std on test data.
train_mean, train_std = X_train.mean(axis=0), X_train.std(axis=0)
X_train_scaled = (X_train - train_mean) / train_std
X_test_scaled = (X_test - train_mean) / train_std # reuse train statsFitting the scaler on the full dataset is a leakage bug
Which algorithms actually need scaling
Scaling is not universally necessary — it matters most for algorithms that use distance directly (k-NN, k-means, SVMs), rely on gradient descent where feature scale affects step size per dimension (linear/logistic regression, neural networks), or apply regularization that penalizes coefficient magnitude uniformly across features (ridge, lasso — an unscaled large-magnitude feature gets penalized less per unit of actual predictive contribution). Tree-based models (decision trees, random forests, gradient boosting) are scale-invariant: they split on thresholds, and a monotonic rescaling of a feature doesn't change which splits are optimal, so scaling is unnecessary for them.
| Method | Output range | Outlier sensitivity | Needs scaling? |
|---|---|---|---|
| Min-max normalization | Fixed, e.g. [0,1] | High | k-NN, neural nets, SVM |
| Standardization (z-score) | Unbounded, mean 0 | Moderate | Linear/logistic regression, PCA, SVM |
| Robust scaling (median/IQR) | Unbounded | Low | Data with known heavy outliers |
| Tree-based models | N/A | N/A | No scaling required |
Robust scaling as a middle ground
What's Next?
Scaling handles numeric features; categorical features need a fundamentally different treatment, since there's no inherent numeric order to rescale in the first place.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.