Gradient boosting

The thing that actually wins on tabular data. As of 2026, XGBoost, LightGBM and CatBoost still beat deep learning on most structured problems, and knowing why is a strong interview position.

The idea

Fit a weak model. Look at what it got wrong. Fit another model to those errors. Add it in. Repeat.

F_0(x) = initial_guess                       # e.g. the mean
for m in range(M):
    residuals = -gradient_of_loss(y, F_m(x)) # "pseudo-residuals"
    h_m = fit_tree(X, residuals)             # weak learner fits the errors
    F_{m+1}(x) = F_m(x) + learning_rate * h_m(x)

“Gradient” boosting because the thing each tree fits is the negative gradient of the loss with respect to the current prediction. For squared error that’s literally the residual y - F(x); for other losses it generalises.

The consequence: each tree depends on the previous one, so training is inherently sequential. You can’t parallelise across trees the way a Random Forest can — the parallelism lives inside each tree’s split-finding.

Why shallow trees

Boosting reduces bias by sequentially correcting errors. It needs weak, high-bias learners — depth 3-8 typically, sometimes stumps. Give it deep trees and the first few fit the training data almost perfectly, leaving nothing to correct and overfitting immediately.

Exactly inverted from Random Forest, which wants deep trees to average variance away. That contrast is one of the most reliably asked questions in the classical-ML space.

The parameters that matter

from xgboost import XGBClassifier

model = XGBClassifier(
    n_estimators=5000,          # set high; early stopping picks the real number
    learning_rate=0.05,         # lower = more trees needed, better generalisation
    max_depth=6,                # 3-8; the main capacity knob
    subsample=0.8,              # row sampling per tree - stochastic gradient boosting
    colsample_bytree=0.8,       # feature sampling per tree
    min_child_weight=1,         # minimum sum of instance weight in a leaf
    reg_lambda=1.0,             # L2 on leaf weights
    early_stopping_rounds=50,
    eval_metric="aucpr",
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)

Learning rate and n_estimators trade off directly. Halve the learning rate, roughly double the trees needed. Lower learning rates generalise better; the cost is training time. The standard recipe is: pick a low-ish LR (0.01-0.1), set n_estimators absurdly high, and let early stopping find the number.

Early stopping is not optional. Unlike a Random Forest, more trees will eventually overfit — each one is fitting residuals, and past a point those residuals are noise.

subsample and colsample_bytree add randomness that both regularises and speeds things up. 0.8 for each is a good starting point.

XGBoost vs LightGBM vs CatBoost

LightGBM’s leaf-wise growth splits whichever leaf reduces loss most, rather than completing each level. It reaches lower loss with fewer splits, which is why it’s fast — but it produces deep, unbalanced trees that overfit small datasets. Control it with num_leaves (the real capacity knob in LightGBM, not max_depth) and min_data_in_leaf.

CatBoost’s ordered target encoding is the genuinely clever bit. Naive target encoding leaks: encoding a category by its mean target uses that row’s own label. CatBoost computes each row’s encoding using only rows that came before it in a random permutation, which removes the leak. If your data is category-heavy, CatBoost usually wins with no tuning.

Practical guidance: CatBoost when you have many categoricals or no tuning time; LightGBM when data is large and you want speed; XGBoost when you want the most documented, most stable option. All three are fine; the choice rarely decides a project.

Handling categoricals

# LightGBM - native
model.fit(X, y, categorical_feature=["country", "device"])

# CatBoost - native, and it's the whole point
CatBoostClassifier(cat_features=["country", "device"])

# XGBoost - enable_categorical with pandas category dtype
X["country"] = X["country"].astype("category")
XGBClassifier(enable_categorical=True, tree_method="hist")

Avoid one-hot encoding high-cardinality categoricals for tree models. It creates thousands of sparse binary features, each carrying little signal, and trees split poorly on them. Native handling or target encoding is materially better.

Imbalanced data

XGBClassifier(scale_pos_weight=(y == 0).sum() / (y == 1).sum())

As with logistic regression, re-weighting distorts predicted probabilities. Calibrate afterwards if you need the numbers rather than the ranking.

Why it still beats deep learning on tabular data

Worth being able to articulate:

• Tabular features are heterogeneous — different scales, types, meanings. Trees handle that natively; neural networks need heavy preprocessing.
• Tree splits are irregular and axis-aligned, which suits the piecewise-constant structure of real tabular relationships. Neural networks are biased toward smooth functions.
• Boosting is robust to uninformative features; a neural net has to learn to ignore them.
• Far less tuning, far less data, far less compute.

Deep learning wins on tabular data mainly when there’s genuine high-cardinality relational structure to embed, or when you need to fuse tabular data with text or images in one model.

Interpretability

import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_val)

SHAP gives per-prediction attributions with a solid theoretical grounding (Shapley values from cooperative game theory), and TreeExplainer computes them exactly and fast for tree models. It’s the standard answer to “your model rejected my loan, why?” — and being able to name it, plus the caveat that SHAP explains the model rather than reality, is the senior version of that answer.

Interview angle

• “How does gradient boosting work?” — sequentially fit weak learners to the negative gradient of the loss at the current prediction, adding each scaled by a learning rate. Each tree corrects the ensemble’s remaining errors, which reduces bias.
• “Boosting vs bagging?” — sequential vs parallel; reduces bias vs reduces variance; shallow weak learners vs deep strong ones; overfits with more trees vs doesn’t. Same ensemble family, opposite mechanisms.
• “Relationship between learning rate and number of trees?” — inverse. Lower LR needs more trees and usually generalises better. Set LR low, n_estimators high, and let early stopping choose.
• “XGBoost vs LightGBM?” — level-wise vs leaf-wise growth. LightGBM is faster and reaches lower loss per split, but its deep unbalanced trees overfit small data; tune num_leaves rather than max_depth.
• “Why is CatBoost good with categorical features?” — ordered target statistics. It encodes a category using only rows preceding the current one in a random permutation, which prevents the target leakage that naive mean-encoding introduces.
• “Why does gradient boosting still beat neural networks on tabular data?” — heterogeneous feature types, piecewise-constant relationships, robustness to irrelevant features, and far lower data and tuning requirements. Deep learning wins when you need to embed high-cardinality relations or fuse modalities.
• “How do you explain an individual prediction?” — SHAP with TreeExplainer, exact and fast for trees. Note that it explains the model’s behaviour, not causality.