Feature Crossing for Recommenders
When features alone aren't enough
Imagine a recommender predicting whether a user will click an ad. You have two features: the user's country and the app they're browsing in. Individually, neither feature might predict clicks well. But the *combination* — "users in Country X clicking inside App Y" — might be a strong signal, even though neither feature alone reveals it.
This is an interaction effect, and a linear model fed the two raw features separately cannot capture it. A linear model only learns independent weights per feature; it has no way to represent "this feature matters only in combination with that one." Feature crossing fixes this by explicitly constructing a new feature that represents the combination.
What a feature cross actually is
A feature cross multiplies or concatenates two or more features into a single new feature. For categorical features, the simplest cross is a Cartesian product: if country has 50 values and app has 1,000 values, the cross country × app has up to 50,000 possible values — one for every combination seen in the data.
For numeric features, a cross is often a literal product (e.g., price × distance_to_store) that lets a linear model represent a multiplicative relationship it otherwise couldn't express. Either way, the point is the same: hand the model a feature that directly encodes the combination, rather than hoping it infers the interaction from the raw inputs.
Note
Wide & Deep: combining memorization and generalization
Google's Wide & Deep architecture (2016) is the canonical model built around this idea. It combines two components trained jointly: a wide component — a linear model over raw and crossed sparse features, good at *memorizing* specific, narrow co-occurrence patterns seen in training data — and a deep component — a neural network over dense embeddings of the same features, good at *generalizing* to feature combinations never seen during training.
The insight is that these two failure modes are complementary. The wide side nails "I've seen this exact combination work before" signals (great for popular, frequent items); the deep side can say "I've never seen this exact user-item pair, but their embeddings are close to pairs that worked," generalizing to the long tail.
Wide vs. deep component roles
| Component | Input | Strength | Weakness |
|---|---|---|---|
| Wide (linear + crosses) | Sparse raw and crossed categorical features | Memorizes exact, frequent co-occurrence patterns | Cannot generalize to unseen combinations |
| Deep (embeddings + MLP) | Dense learned embeddings per feature | Generalizes to novel feature combinations via similarity | Can over-generalize, blurring rare but real signals |
Note
Practical guidance
Start simple: fit a baseline model on raw features, then look at residual errors for evidence of interaction effects — subgroups where predictions are systematically wrong. Cross only features you have a reason to believe interact (domain knowledge beats blindly crossing everything). Use hashing when cardinality is large, monitor the sparsity of each crossed bucket, and consider dropping or regularizing crosses with very few training examples, since those are the ones most prone to overfitting.
What's next
Feature crossing changes what a model can represent, but it says nothing about whether the resulting model actually improves outcomes for real users. The next lesson covers A/B testing ML-powered products — how to validate a model change through user-facing metrics, not just offline accuracy.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.