One-Hot vs Label vs Target Encoding
The core problem: categories have no inherent order or magnitude
A categorical feature like 'city' or 'payment_method' doesn't have a meaningful numeric relationship between its values — 'Chicago' isn't twice 'Boston'. Feeding raw integer codes (0, 1, 2...) directly into a model that assumes numeric ordering (like linear regression or distance-based methods) implicitly tells the model that category 2 is 'between' categories 1 and 3 in some meaningful sense, which is usually false and introduces a spurious signal. Encoding schemes exist to convert categories into numeric representations without introducing unwarranted structure — or, in the case of target encoding, to introduce deliberate, useful structure.
Label encoding
Label encoding assigns each category an arbitrary integer: red=0, green=1, blue=2. It's compact and lossless in the sense that it's reversible, but it implies an ordinal relationship that doesn't exist for nominal (unordered) categories. It's appropriate specifically for *ordinal* categorical features where order genuinely matters — small/medium/large, or low/medium/high risk — and should generally be avoided for nominal features fed into linear models or distance-based methods, though tree-based models handle it more gracefully since they split on thresholds rather than treating values as continuous magnitudes.
One-hot encoding
One-hot encoding creates a separate binary column per category, with exactly one column set to 1 for each row. This avoids implying any order, at the cost of dimensionality: a feature with $K$ categories becomes $K$ (or $K-1$, dropping one to avoid perfect multicollinearity in linear models) new columns. For low-to-moderate cardinality features (tens of categories), this is usually the safest default for linear models and neural networks. For high-cardinality features (thousands of categories — zip codes, product IDs), one-hot encoding becomes impractical: it balloons dimensionality, produces extremely sparse feature matrices, and runs straight into the curse of dimensionality.
import pandas as pd
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
df = pd.DataFrame({"city": ["NYC", "LA", "NYC", "SF", "LA"]})
# Label encoding: arbitrary integers, implies false ordering for nominal data
le = LabelEncoder()
df["city_label"] = le.fit_transform(df["city"])
# One-hot encoding: no implied order, but adds one column per category
ohe = OneHotEncoder(sparse_output=False, drop="first")
one_hot = ohe.fit_transform(df[["city"]])
# Target encoding: replace category with the mean target value for that category
# (must be computed with cross-validation / smoothing to avoid leakage)
def target_encode(df, cat_col, target_col, smoothing=10):
global_mean = df[target_col].mean()
agg = df.groupby(cat_col)[target_col].agg(["mean", "count"])
smoothed = (agg["mean"] * agg["count"] + global_mean * smoothing) / (agg["count"] + smoothing)
return df[cat_col].map(smoothed)Target encoding and why it needs special care
Target encoding replaces each category with a statistic of the target variable computed over rows in that category — typically the mean, for regression or binary classification. This handles high-cardinality features gracefully (one numeric column, regardless of category count) and can capture predictive signal that one-hot encoding spreads thin across many sparse columns. The serious risk is target leakage: computing a category's mean target using the same rows you'll train on means each row's encoded value partially encodes its own label, which inflates training performance and can collapse at inference time or generalize poorly. The standard fix is smoothing (blending the category mean with the global mean, weighted by category frequency, as shown above) combined with computing encodings out-of-fold during cross-validation, never on the full dataset at once.
| Encoding | Dimensionality added | Implies order? | Leakage risk |
|---|---|---|---|
| Label encoding | None (1 column) | Yes — use only for true ordinals | Low |
| One-hot encoding | K or K-1 columns | No | None |
| Target encoding | None (1 column) | No | High without smoothing/CV |
| Frequency/count encoding | None (1 column) | No | Low |
Tree-based models are more forgiving of encoding choice
Unseen categories at inference time
What's Next?
Encoding choices only matter once the categorical values themselves are clean and complete — missing categorical values, and missing data more generally, need to be handled as a distinct step in the pipeline, ideally before encoding decisions are finalized.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.