The Imbalanced Data Fix That Actually Worked, After Three That Didn't

A fraud detection dataset we worked with had fraud cases at under half a percent of total volume, the kind of imbalance where a model can hit 99.5% accuracy by predicting "not fraud" every single time and still be completely useless. We knew the standard playbook going in. We just went through three rounds of it before anything actually worked.

What didn't work, in order

Random oversampling of the minority class got us a model that overfit hard to the handful of duplicated fraud examples, performing beautifully on validation and falling apart on genuinely new fraud patterns. Class weighting in the loss function nudged the decision boundary slightly but not enough to matter at the precision level the business actually needed. A focal loss variant, which looked promising on paper, mostly just made training unstable without a meaningful recall improvement to show for it.

Every imbalanced-data technique in the standard playbook assumes the imbalance is the whole problem. Sometimes the imbalance is just making a separate, boring problem harder to see.

What actually worked

The breakthrough wasn't a fancier technique, it was going back to the features. SMOTE-based synthetic oversampling, applied carefully in feature space rather than duplicating raw examples, combined with a much lower, business-calibrated decision threshold instead of the default 0.5 cutoff, finally moved recall without destroying precision. The real unlock, though, was realizing some of our existing features were poorly scaled in a way that made the minority class's signal nearly invisible to the model regardless of sampling strategy — fixing that scaling issue did more for recall than any of the three sampling techniques combined.

• SMOTE in feature space, not naive duplication of raw minority examples.
• Threshold tuned against the business's actual precision/recall tradeoff, not left at a default 0.5.
• Feature scaling audited specifically for whether it preserved minority-class signal — this was the real culprit.
• Evaluation metric switched from accuracy to precision-recall AUC early, so we weren't chasing a number that lied to us.

The three failed attempts weren't wasted exactly, but they were all solving the imbalance in isolation from the rest of the pipeline. The actual fix needed us to stop treating imbalance as a standalone technique problem and look at whether the features even carried the signal the sampling strategy was trying to amplify.