Many of the most important classification problems are naturally imbalanced: fraud is rare, disease is rare, equipment failure is rare, customer churn might be a small minority of accounts. If 99% of transactions are legitimate, a classifier that predicts "legitimate" for everything scores 99% accuracy while catching zero fraud. The model isn't doing anything wrong by the metric you gave it — it's exploiting the fact that the metric doesn't care about the minority class at all.

This is an imbalanced data problem, and it requires deliberate handling at both the data level and the evaluation level, because a model trained naively on imbalanced data will, overwhelmingly, learn to predict the majority class.

Note

On a 99:1 imbalanced dataset, a trivial always-majority classifier gets 99% accuracy. Use precision, recall, F1, and especially the precision-recall curve or ROC-AUC for the minority class instead. Recall tells you what fraction of actual positives you caught; precision tells you how many of your positive predictions were real. Accuracy tells you almost nothing useful.

Undersampling the majority class

Random undersampling removes examples from the majority class until the classes are closer to balanced. It's simple and fast, and it reduces training set size, which speeds up training. The cost is throwing away data — potentially useful majority-class examples near the decision boundary get discarded along with redundant ones far from it, and with severe imbalance you may end up training on a tiny fraction of your available data.

More targeted variants remove majority examples selectively: Tomek links remove majority points that sit directly adjacent to a minority point (likely borderline or mislabeled), and NearMiss keeps majority examples close to the decision boundary while discarding those far from it, preserving the examples that matter most for separating the classes.

Oversampling the minority class

Random oversampling duplicates minority-class examples until the classes balance. It keeps all original data but risks overfitting, since the model sees identical copies of the same minority examples repeatedly and can effectively memorize them rather than learning a generalizable decision boundary.

SMOTE (Synthetic Minority Oversampling Technique) addresses this by generating new, synthetic minority examples rather than duplicating existing ones. For each minority example, SMOTE finds its k nearest minority neighbors in feature space and creates a new synthetic point somewhere along the line segment connecting the example to one of those neighbors. This fills in the feature space around minority examples with plausible interpolated points instead of exact copies, giving the model more varied minority-class signal to learn from.

python

Note

If you oversample before doing a train/test split, synthetic points derived from a minority example can end up correlated with that same example across both sets — leaking information and inflating your reported performance. Always split first, then resample only the training fold, ideally inside a cross-validation-aware pipeline.

Class weighting: no resampling at all

Resampling changes the data; class weighting changes the loss function instead, leaving the data untouched. Most classifiers (logistic regression, SVMs, tree ensembles, neural networks) let you assign a higher weight to the minority class's loss contribution, so misclassifying a minority example costs more during training than misclassifying a majority example.

This avoids both the information loss of undersampling and the duplication/synthesis risks of oversampling, and it's usually the cheapest first thing to try — often a single class_weight="balanced" argument. It doesn't help when the minority class is so sparse that the model simply hasn't seen enough examples to learn its pattern at all; in that regime, resampling or collecting more minority data matters more than reweighting.

python

Resampling and weighting strategies compared

Technique	Changes data or loss?	Risk	Best for
Random undersampling	Data (removes majority)	Discards potentially useful data	Very large majority class, compute constrained
Random oversampling	Data (duplicates minority)	Overfits to duplicated examples	Small datasets where SMOTE's interpolation is unreliable
SMOTE	Data (synthesizes minority)	Can create unrealistic points in sparse/noisy regions	Moderate imbalance with enough minority examples to interpolate between
Class weighting	Loss function only	No effect if minority class is too sparse to learn from	First thing to try; works with almost any model

What's next

Imbalanced data is one failure mode in binary and multi-class classification. A related but distinct problem is when each example can belong to several categories at once rather than exactly one. The next lesson, Multi-Label Classification, covers how that changes both model design and evaluation.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →