LightGBM vs XGBoost vs CatBoost
Three libraries, one algorithm family, different engineering bets
XGBoost, LightGBM, and CatBoost all implement gradient boosted decision trees and often land within a percentage point of each other's accuracy on tabular benchmarks after proper tuning. Where they diverge sharply is in how they grow trees, how they handle categorical features, and how they guard against overfitting by default — choices that matter a lot for training speed, memory use, and how much manual preprocessing you need to do.
Tree growth strategy: level-wise vs. leaf-wise
XGBoost (by default) and CatBoost grow trees level-wise (depth-wise): every leaf at a given depth is split before moving to the next depth, keeping the tree balanced. LightGBM grows trees leaf-wise: it always splits whichever single leaf yields the greatest loss reduction, regardless of depth, producing asymmetric, often deeper trees for the same leaf budget. Leaf-wise growth tends to reach lower training loss faster for a fixed number of leaves, which is a major reason LightGBM is typically faster, but it's also more prone to overfitting on small datasets — which is why max_depth and num_leaves need joint tuning in LightGBM specifically.
Categorical features: the biggest practical difference
XGBoost historically required manual encoding (one-hot or target encoding) of categorical features, though recent versions added native support. LightGBM handles categoricals natively using an optimal-partitioning algorithm for the split search, avoiding the dimensionality blowup of one-hot encoding. CatBoost goes furthest: it was designed around categorical data from the start, using ordered target statistics (a permutation-based scheme) specifically to encode categories without the leakage that naive target-mean encoding introduces. If your dataset is dominated by high-cardinality categorical features, this single difference often matters more than any hyperparameter.
import time
import lightgbm as lgb
import xgboost as xgb
from catboost import CatBoostClassifier
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=50000, n_features=30, random_state=0)
configs = {
"xgboost": lambda: xgb.XGBClassifier(n_estimators=300, max_depth=6, eval_metric="logloss"),
"lightgbm": lambda: lgb.LGBMClassifier(n_estimators=300, num_leaves=63, verbosity=-1),
"catboost": lambda: CatBoostClassifier(iterations=300, depth=6, verbose=False),
}
for name, make_model in configs.items():
model = make_model()
start = time.time()
model.fit(X, y)
print(f"{name:10s} train time: {time.time() - start:.2f}s")
CatBoost's ordered boosting also fights a subtler leakage
| Aspect | XGBoost | LightGBM | CatBoost |
|---|---|---|---|
| Tree growth | Level-wise (default) | Leaf-wise | Level-wise (oblivious trees) |
| Categorical handling | Manual / native (recent) | Native, optimal partitioning | Native, ordered target stats |
| Typical training speed | Moderate | Fast, esp. large data | Slower, strong out-of-the-box defaults |
| Overfitting risk (defaults) | Moderate | Higher on small data | Lower — conservative defaults |
What's next?
Whichever library you pick, the categorical-handling strategy underneath it is solving the same core problem: how to split on a categorical variable without either exploding dimensionality or leaking target information through naive encoding — a problem worth understanding in its own right, independent of any one library's implementation.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.