Data Preprocessing & Cleaning for Machine Learning

Models fail quietly when data is wrong

There's a well-worn saying in applied machine learning: you spend 80% of your time on data and 20% on modeling. It understates the point — bad preprocessing doesn't just slow you down, it produces a model that looks great in testing and then fails silently in production, because the mistake was baked into how the data was prepared, not into the algorithm. Missing values, outliers, badly encoded categories, and leaked information are the most common causes of a model that scores well in a notebook and badly everywhere else.

Handling missing values

Real datasets are never complete. Before you can train anything, you need a deliberate strategy for missing values — and "just drop the rows" is often the wrong default, because it silently throws away data and can bias your sample if missingness isn't random (e.g., high-income survey respondents skip the income question more often).

Common strategies, roughly from simplest to most sophisticated:

- Deletion — drop rows or columns with missing values. Fine only when missingness is small and random.
- Mean/median/mode imputation — fill numeric gaps with the column mean or median (median is more robust to outliers), and categorical gaps with the most frequent category.
- Model-based imputation — predict the missing value from other columns (e.g., k-nearest-neighbors imputation, or a regression model trained on complete rows).
- Missingness as a feature — add a binary indicator column flagging whether a value was missing at all. Sometimes the fact that data is missing is itself predictive (e.g., a skipped optional field correlating with user behavior).

Outlier detection

Outliers can be genuine extreme values (a legitimate billion-dollar transaction) or data errors (a typo entering age as 999). Treating them the same way is a mistake — investigate before you act.

Statistical methods: the Z-score method flags points more than roughly 3 standard deviations from the mean (works best when data is roughly normal). The IQR method flags points below $Q1 - 1.5 \times IQR$ or above $Q3 + 1.5 \times IQR$ — more robust to skewed distributions since it doesn't assume normality.

Model-based methods: Isolation Forest and Local Outlier Factor learn what "normal" looks like across many dimensions at once and flag points that don't fit, which is essential once you have more than two or three features to reason about jointly.

Once flagged, the decision isn't automatic: remove genuine data errors, but consider capping (clipping to a reasonable range) or leaving true extreme-but-valid values alone, especially if the task is specifically about detecting rare events like fraud.

Categorical encoding basics

Most ML algorithms need numeric input, so categorical columns (city, color, product category) must be converted to numbers — but the wrong encoding introduces false relationships. One-hot encoding creates a separate binary column per category, correctly representing that categories have no inherent order (good for "city" or "color"). Ordinal encoding assigns each category an integer (0, 1, 2, ...), which is appropriate only when categories have a genuine order (small < medium < large) — using it on unordered categories tricks the model into thinking "blue" (2) is somehow "greater than" red (0), which is meaningless and can hurt performance.

For high-cardinality categoricals (thousands of unique values, like zip codes), one-hot encoding explodes dimensionality — consider target encoding, frequency encoding, or embedding-based approaches instead.

One-hot vs. ordinal encoding

Train/test/validation split discipline

The entire point of a held-out test set is to simulate how the model performs on data it has never seen. That guarantee breaks the moment any information from the test set influences training — even indirectly. The standard discipline: split data into training (fit the model), validation (tune hyperparameters and select models), and test (final, untouched evaluation, used exactly once) sets, typically before any other preprocessing happens.

For time-series data, never use a random split — it lets the model "see the future" during training. Use a time-based split instead, where the test set is strictly later in time than training data.

What's next

Clean, correctly split data is the prerequisite for everything downstream. Once your data is ready, Feature Engineering for Machine Learning covers how to construct new, more informative features from it, and training-ml covers how models are actually fit once the data is prepared.