Gradient Boosting Internals: How Trees Correct Errors
Boosting is sequential, not parallel
Random forests build many independent trees in parallel and average them (variance reduction). Gradient boosting builds trees sequentially, where each new tree is trained specifically to correct the errors of the ensemble built so far (bias reduction). Instead of fitting trees to the raw target $y$, each new tree is fit to the residual error of the current ensemble — the part of $y$ that hasn't been explained yet.
The additive model and the residual-fitting view
Gradient boosting builds a model as a sum of weak learners: $F_M(x) = \sum_{m=1}^{M} \gamma_m h_m(x)$, adding one tree $h_m$ at a time. For squared-error regression, it turns out that fitting a new tree to the negative gradient of the loss with respect to the current predictions is exactly equivalent to fitting it to the residuals $y_i - F_{m-1}(x_i)$. This is the 'gradient' in gradient boosting: at each step, compute the gradient of the loss function with respect to the current predictions, and train the next tree to predict that gradient — a form of gradient descent performed in function space rather than parameter space.
Why this generalizes beyond squared error
The residual-fitting intuition is easiest to see for squared-error loss, but the real algorithm (Friedman's Gradient Boosting Machine) generalizes to *any* differentiable loss function — log loss for classification, quantile loss for quantile regression, Huber loss for robust regression — by always fitting the next tree to the current negative gradient of whatever loss is chosen. This is why gradient boosting frameworks (XGBoost, LightGBM, CatBoost) support dozens of objective functions through a single unified training loop.
import numpy as np
from sklearn.tree import DecisionTreeRegressor
def simple_gradient_boosting(X, y, n_estimators=50, learning_rate=0.1, max_depth=2):
F = np.full(len(y), y.mean()) # initial prediction: the mean
trees, gammas = [], []
for m in range(n_estimators):
residuals = y - F # negative gradient for squared error
tree = DecisionTreeRegressor(max_depth=max_depth)
tree.fit(X, residuals)
update = tree.predict(X)
F = F + learning_rate * update # shrink each tree's contribution
trees.append(tree)
gammas.append(learning_rate)
return trees, gammas, y.mean()
def predict(X, trees, gammas, base):
F = np.full(X.shape[0], base)
for tree, gamma in zip(trees, gammas):
F += gamma * tree.predict(X)
return F
The learning rate is a second regularizer, not just a speed knob
| Property | Random forest (bagging) | Gradient boosting |
|---|---|---|
| Trees built | In parallel, independently | Sequentially, each correcting prior errors |
| Primarily reduces | Variance | Bias |
| Sensitive to overfitting | Less, more trees rarely hurts | Yes — needs tuning (depth, rate, rounds) |
| Typical tree depth | Deep (low-bias trees) | Shallow (high-bias, weak learners) |
What's next?
This sequential, gradient-fitting mechanism is the mathematical core shared by every modern boosting library, but the production frameworks differ heavily in how they regularize, split, and handle categorical data at scale — differences that matter a great deal once you move from understanding the algorithm to actually tuning one.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.