Decision trees

Rarely used alone in production, but you must understand them because Random Forest and gradient boosting are built from them, and every question about those starts here.

How a tree is built

Greedy recursive splitting. At each node, try every feature and every threshold, pick the split that most reduces impurity, recurse.

Is income > 50k?
├── yes: Is age > 30?
│        ├── yes: approve  (n=120, 92% positive)
│        └── no:  review   (n=45,  55% positive)
└── no:  Is credit_score > 700?
         ├── yes: review   (n=30,  48% positive)
         └── no:  reject   (n=200, 8% positive)

The greediness matters: the tree takes the locally best split at every step and never reconsiders. That’s why a small change in data can produce a completely different tree, and why trees are high-variance.

Impurity measures

Gini and entropy almost always produce the same tree. Choosing between them is not a meaningful tuning decision, and saying so is a better answer than inventing a distinction.

Information gain = parent impurity − weighted average of children impurity. The split with the highest gain wins.

Controlling depth

An unconstrained tree grows until every leaf is pure — which means it memorises the training set perfectly and generalises badly.

from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(
    max_depth=6,                # the blunt, effective lever
    min_samples_leaf=20,        # my usual first choice: no leaf on 3 rows
    min_samples_split=50,
    max_features="sqrt",        # also decorrelates, used by Random Forest
    ccp_alpha=0.01,             # cost-complexity pruning
    random_state=42,
)

min_samples_leaf is often more useful than max_depth because it adapts — dense regions can go deep, sparse regions can’t.

Cost-complexity pruning (ccp_alpha) grows the tree fully, then prunes back the branches whose complexity cost outweighs their accuracy gain. Principled, and less arbitrary than picking a depth.

What trees are good at

• No scaling needed. Splits are threshold comparisons, so units are irrelevant. This is a genuine practical advantage over linear models and SVMs.
• Non-linearities and interactions for free. Nested splits are interactions; you don’t engineer them.
• Mixed feature types, including categoricals (natively in LightGBM and CatBoost).
• Monotone-transform invariant. Log-transforming a feature changes nothing, because the ordering is unchanged.
• Readable — a shallow tree is genuinely explainable to a non-technical stakeholder.
• Missing values handled natively by modern implementations.

What they’re bad at

• High variance. Small data perturbations produce very different trees. This is the whole motivation for bagging.
• Cannot extrapolate. A leaf predicts the mean of its training samples, so predictions are bounded by the training target range. Feed a tree a house twice the size of anything it has seen and it predicts the maximum it saw. Linear models don’t have this limitation, and it’s a good interview contrast.
• Axis-aligned splits only. A diagonal boundary needs a staircase of many splits.
• Biased toward high-cardinality features when using impurity-based importance — see below.

Feature importance: the trap

model.feature_importances_       # impurity-based - be careful

Impurity-based importance is biased toward high-cardinality and continuous features, because they offer more possible split points and therefore more chances to reduce impurity by luck. A random ID column can rank surprisingly high.

Use permutation importance instead, computed on held-out data:

from sklearn.inspection import permutation_importance

r = permutation_importance(model, X_val, y_val, n_repeats=10, random_state=42)

It measures how much performance drops when a feature is shuffled — model-agnostic and unbiased by cardinality. Slower, and it splits credit oddly between correlated features, but far more trustworthy.

Knowing that impurity importance is biased is a strong seniority signal, because it’s a default a lot of people trust blindly.

Regression trees

Same algorithm, minimising variance instead of Gini, predicting the leaf mean.

The visible consequence is a step function — predictions are piecewise constant. Plot a regression tree’s predictions against a continuous feature and you get stairs, not a curve. That’s why a single tree is a poor fit for smooth relationships and why ensembles (which average many step functions) look much smoother.

Interview angle

• “How does a decision tree decide where to split?” — greedily: try every feature and threshold, pick the one maximising impurity reduction (Gini or entropy for classification, variance for regression), then recurse. It never revisits earlier splits, which is why it’s high-variance.
• “Gini vs entropy?” — near-identical results in practice; Gini avoids a log and is marginally faster, which is why it’s the default. Not a meaningful tuning knob.
• “Why don’t trees need feature scaling?” — splits are threshold comparisons on one feature at a time, so monotone transformations and units don’t change the tree.
• “Why can’t a tree extrapolate?” — every prediction is the mean of some leaf’s training samples, so outputs are bounded by the observed target range. Linear models extend beyond it; trees cannot.
• “Is feature_importances_ trustworthy?” — not fully. Impurity-based importance favours high-cardinality and continuous features. Prefer permutation importance on a held-out set, or SHAP if you need per-prediction attribution.
• “How do you stop a tree overfitting?” — depth limits, min_samples_leaf / min_samples_split, max_features, cost-complexity pruning — or stop using a single tree and bag or boost them, which is what everyone actually does.
• “Why is a single tree rarely used in production?” — high variance and axis-aligned splits make it weak alone. Its value is as a base learner: bagged into a Random Forest to cut variance, or boosted to cut bias.