Feature selection

Fewer, better features means faster training, cheaper serving, less overfitting, and fewer things to break in production. The interview question is usually “how would you decide which features to keep” — and the strongest answers include the operational cost, not just the statistics.

Why bother

• Overfitting — irrelevant features give the model more chances to fit noise.
• Serving cost — every feature is a lookup, a join, or a computation on the request path.
• Maintenance — every feature is an upstream dependency that can break or drift.
• Interpretability — a 20-feature model can be explained; a 2,000-feature one can’t.

That second and third point are what a senior candidate adds. A feature that improves AUC by 0.001 and requires a new real-time join is a net negative.

The three families

Filter methods

Score features against the target without a model.

from sklearn.feature_selection import (
    SelectKBest, f_classif, mutual_info_classif, VarianceThreshold
)

VarianceThreshold(threshold=0.0)                  # drop constants - always do this
SelectKBest(mutual_info_classif, k=50)            # non-linear dependence

• Correlation / ANOVA F-test — linear relationships only.
• Mutual information — catches non-linear dependence, which is why it’s usually the better filter.
• Chi-squared — categorical features against categorical target, non-negative values only.

Fast, and blind to interactions: two features that are useless alone but predictive together get dropped.

Wrapper methods

from sklearn.feature_selection import RFECV

selector = RFECV(estimator=LogisticRegression(), step=1, cv=5,
                 scoring="average_precision", n_jobs=-1)

Recursive feature elimination fits, drops the weakest, refits, repeats. RFECV uses cross-validation to pick how many to keep. Accurate and slow — cost is roughly n_features model fits.

Embedded methods

Selection as a side effect of training.

from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LassoCV

SelectFromModel(LassoCV())                        # L1 drives coefficients to exactly 0
SelectFromModel(LGBMClassifier(), threshold="median")

L1 regularisation is the cleanest version — it sets coefficients to exactly zero, so selection is built into the fit. See ../01_ml_foundations/04_overfitting_regularization.md.

Importance: use permutation, not impurity

from sklearn.inspection import permutation_importance

r = permutation_importance(model, X_val, y_val, n_repeats=10,
                           scoring="average_precision", n_jobs=-1)

Tree feature_importances_ is impurity-based and biased toward high-cardinality and continuous features, because they offer more split points and therefore more chances to reduce impurity by luck. A random ID column can score surprisingly high.

Permutation importance shuffles one feature on held-out data and measures the performance drop. Model-agnostic, unbiased by cardinality, and computed on data the model didn’t train on.

Two caveats worth knowing:

• Correlated features dilute each other. Shuffle one and the model leans on its twin, so both look unimportant. Group them, or drop one first.
• It measures importance to this model, not to the problem. A different model may use different features.

SHAP gives per-prediction attribution and handles interactions more gracefully:

import shap
shap_values = shap.TreeExplainer(model).shap_values(X_val)

Mean absolute SHAP value per feature is a solid global ranking; the per-row values answer “why was this prediction made”, which is what regulated domains need.

Multicollinearity

Correlated features don’t hurt tree or regularised model predictions much, but they make coefficients uninterpretable and importance rankings unstable.

corr = X.corr().abs()
# find pairs above 0.95 and keep one from each

Keep the one that’s cheaper to compute or more robust upstream — that’s usually a better tie-breaker than a marginal statistical difference.

The practical procedure

1. Drop constants and near-constants. Free.
2. Drop features unavailable at prediction time. This is leakage screening, and it comes before any statistics. See 04_data_leakage.md.
3. Deduplicate highly correlated pairs.
4. Fit a gradient boosting model on everything, rank by permutation importance or SHAP.
5. Cut the tail and re-measure. Stop where validation performance starts to drop.
6. Weigh operational cost. Drop expensive features whose contribution is marginal.

Step 6 is the one that distinguishes the answer.

Selection must live inside cross-validation

Selecting features using the whole dataset and then cross-validating is leakage — the selection saw the validation folds.

from sklearn.pipeline import Pipeline

pipe = Pipeline([
    ("select", SelectFromModel(LassoCV())),
    ("model", GradientBoostingClassifier()),
])
cross_val_score(pipe, X, y, cv=5)     # selection refits per fold

This is a favourite trap question, and it produces impressively wrong results — reported accuracy well above the truth, especially when features vastly outnumber rows.

When not to select

• Gradient boosting is fairly robust to irrelevant features — it simply doesn’t split on them. Aggressive selection often buys little accuracy, though it still buys serving cost and maintenance.
• Deep learning generally prefers more raw signal and learns its own representation.
• When you have far more rows than features, overfitting from extra features is less of a concern.

Selection matters most when features outnumber samples, when serving cost is real, or when the model must be explainable.

Interview angle

• “How do you decide which features to keep?” — leakage screening first, then drop constants and correlated duplicates, then rank with permutation importance or SHAP on a gradient boosting model, cut the tail while watching validation, and finally weigh each surviving feature’s serving and maintenance cost.
• “Filter vs wrapper vs embedded?” — filter scores features independently (fast, misses interactions); wrapper searches subsets by retraining (accurate, expensive); embedded selects during training, L1 being the canonical case.
• “Is feature_importances_ reliable?” — not fully. Impurity-based importance favours high-cardinality and continuous features. Prefer permutation importance on held-out data, or SHAP for per-prediction attribution.
• “Two features are 0.98 correlated. How does that show up in importance?” — both look unimportant under permutation, because shuffling one lets the model use the other. Evaluate them as a group, or drop one before measuring.
• “You selected features on the full dataset then cross-validated. What’s wrong?” — selection saw the validation folds, so the score is optimistic. Put selection inside the pipeline so it refits per fold.
• “A feature adds 0.001 AUC but needs a new real-time join. Keep it?” — no. The accuracy gain is within noise and the operational cost is a permanent dependency on the request path.