Random Forest and bagging

The reliable default for tabular data when you want something that works without tuning. Gradient boosting usually beats it on accuracy, but Random Forest is far harder to get wrong.

Bagging

Bootstrap aggregating: train N models on N bootstrap samples (sample with replacement, same size as the original), then average their predictions.

Why it works: averaging N independent estimators with variance σ² gives variance σ²/N. Trees are high-variance and roughly unbiased, so averaging them cuts variance without adding bias — exactly the right medicine. See ../01_ml_foundations/03_bias_variance_tradeoff.md.

The catch is “independent”. Bootstrap samples overlap heavily, so the trees are correlated, and correlation puts a floor on how much averaging helps:

Var(average) = rho * sigma^2 + (1 - rho) * sigma^2 / N

As N -> inf, the second term vanishes but rho * sigma^2 remains. Reducing correlation between trees is therefore more valuable than adding more trees, and that’s precisely what Random Forest adds.

What makes a Random Forest “random”

Two sources of randomness:

1. Bootstrap sampling of rows (bagging).
2. Feature subsampling at every split — consider only a random subset of features when choosing each split.

The second is the key innovation. Without it, if one feature is strongly predictive, every tree splits on it first and they all look alike. Forcing each split to consider a random subset means different trees discover different structure, rho drops, and averaging pays off.

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(
    n_estimators=500,        # more is always better for quality; cost is linear
    max_features="sqrt",     # THE parameter. sqrt(n) for classification
    min_samples_leaf=1,      # deep trees on purpose - we want high variance to average away
    n_jobs=-1,
    random_state=42,
)

max_features is the knob that matters. Lower means more decorrelation but weaker individual trees; higher means stronger trees that agree too much. sqrt(n_features) for classification and n_features/3 for regression are solid defaults.

Base learners should be deep, not shallow. This is the opposite of boosting and a favourite interview question. Bagging removes variance, so you want high-variance, low-bias base learners — fully grown trees. Restricting depth in a Random Forest adds bias that averaging cannot remove.

More trees never overfit

Unlike boosting, adding trees to a Random Forest does not overfit — the average converges. n_estimators is a compute/accuracy trade-off, not a regularisation knob. Set it as high as you can afford, then stop when the OOB score plateaus.

Out-of-bag evaluation

Each bootstrap sample omits roughly 37% of rows (1/e). Those out-of-bag rows are a free validation set:

model = RandomForestClassifier(n_estimators=500, oob_score=True).fit(X, y)
print(model.oob_score_)      # honest estimate without a separate holdout

Genuinely useful with small datasets where you don’t want to sacrifice rows. The 1/e derivation is a common question: the probability a specific row is not picked in n draws is (1 - 1/n)^n -> 1/e ≈ 0.368.

Random Forest vs gradient boosting

The practical rule: Random Forest when you want a strong result with no tuning budget; gradient boosting when you’re chasing the last few points and can afford to tune. For a take-home or a first baseline, Random Forest is the safer choice.

Extremely Randomised Trees

ExtraTreesClassifier goes further: split thresholds are chosen at random rather than optimised, and it uses the whole dataset rather than bootstrap samples. More bias per tree, less variance, and much faster to train since it skips the threshold search. Worth trying as a cheap alternative — sometimes it wins.

Feature importance

Same caveat as single trees, and it matters more because people trust forests: impurity-based importance is biased toward high-cardinality features. Use permutation importance on held-out data, or SHAP.

from sklearn.inspection import permutation_importance
r = permutation_importance(model, X_val, y_val, n_repeats=10, n_jobs=-1)

With correlated features, both methods dilute importance across the group — dropping one barely hurts because the other carries the signal. Interpret groups, not individual features, when collinearity is present.

Practical notes

• Probabilities are poorly calibrated — they’re vote fractions, pushed toward the middle by averaging. Calibrate if you consume the probability numerically.
• Memory grows with n_estimators * tree_size. 500 deep trees on a large dataset is a big object; cap min_samples_leaf if serialised size matters.
• Inference is parallel and cache-unfriendly; boosted models with shallow trees are usually faster to serve.
• class_weight="balanced_subsample" for imbalanced data, applied per bootstrap sample.

Interview angle

• “How does a Random Forest differ from bagged trees?” — bagging randomises rows only; Random Forest also randomises the feature subset considered at each split. That second source decorrelates the trees, and decorrelation is what makes averaging effective.
• “Why deep trees in a forest but shallow in boosting?” — bagging reduces variance and can’t reduce bias, so you want low-bias high-variance learners. Boosting reduces bias sequentially, so it wants high-bias weak learners it can correct.
• “Can you overfit by adding trees?” — not in a Random Forest; the average converges. In gradient boosting, yes — each tree fits the current residuals, so too many will fit noise. That’s why boosting needs early stopping and forests don’t.
• “What is out-of-bag error?” — each bootstrap omits about 37% of rows; scoring each tree on the rows it didn’t see gives a free validation estimate. The 37% is 1/e, the limit of (1-1/n)^n.
• “Your forest and your logistic regression have the same AUC. Which ships?” — the logistic regression, most likely. Equal performance with better interpretability, calibration, latency and model size is an easy call.
• “Feature importance says user_id is the top feature. What’s happening?” — impurity-based importance is biased toward high-cardinality features, and a per-row identifier maximises that. Also check whether it’s genuine leakage. Re-check with permutation importance.