Scaling and numeric transforms

Which models need scaling, which don’t, and what to do when a feature is skewed or full of outliers.

Who needs scaling

The rule underneath: if the algorithm computes distances or penalises coefficient magnitude, scale. Trees only compare one feature against a threshold at a time, so units are irrelevant to them.

Getting this wrong on an SVM or kNN doesn’t degrade the model slightly — it lets the largest-magnitude feature dominate the distance metric and effectively deletes the others.

The scalers

from sklearn.preprocessing import (
    StandardScaler, MinMaxScaler, RobustScaler, QuantileTransformer, PowerTransformer
)

Normalizer catches people out: it normalises each row to unit length, not each column. That’s what you want for embeddings before cosine similarity, and almost never what you want for tabular features.

Skew

Right-skewed features (prices, counts, durations, income) hurt linear models and distance metrics. The common fixes:

import numpy as np

X["amount_log"] = np.log1p(X["amount"])     # log1p handles zeros; log(0) is -inf

log1p / expm1 rather than log / exp — the 1p variants are both zero-safe and more numerically accurate for small values.

For the target rather than a feature, log-transforming often fixes heteroscedasticity too. Remember that back-transforming a prediction of the log-mean gives you an estimate of the median, not the mean — a subtle bias worth naming.

PowerTransformer with Yeo-Johnson finds the transform for you and handles zeros and negatives, which plain log can’t.

Outliers

Decide what they are before deciding what to do:

• Data errors (a birth year of 1850, a negative price) — fix or drop.
• Genuine extremes (a whale customer) — often your most important rows. Do not delete.

Options that don’t throw information away:

# Winsorise - cap at percentiles
lo, hi = X["amount"].quantile([0.01, 0.99])
X["amount_capped"] = X["amount"].clip(lo, hi)

# Or keep the signal explicitly
X["is_extreme"] = (X["amount"] > hi).astype(int)

RobustScaler uses the median and IQR, so extreme values don’t move the scaling parameters at all. It’s the usual choice when you want to keep outliers but not let them distort everything else.

Tree models are naturally robust — an outlier just ends up in its own leaf.

Binning

Turning a continuous feature into buckets. Sometimes helps linear models capture non-linearity; usually pointless for trees, which do their own binning by splitting.

from sklearn.preprocessing import KBinsDiscretizer
KBinsDiscretizer(n_bins=10, encode="ordinal", strategy="quantile")

Costs resolution, gains robustness and interpretability. Worth it mainly when the relationship is genuinely non-monotonic and you’re stuck with a linear model.

Missing values

Missingness is information. Check whether it’s random before erasing it.

from sklearn.impute import SimpleImputer

SimpleImputer(strategy="median", add_indicator=True)    # keep the fact it was missing

add_indicator=True is the underused part: it appends a binary column recording that the value was imputed. If missingness correlates with the target — an unfilled income field on a loan application, say — that indicator can be your strongest feature.

That last row is worth stating in an interview: for gradient boosting, imputing is often worse than doing nothing, because the model can learn an optimal default direction per split.

Feature interactions

Trees find interactions implicitly. Linear models need them spelled out.

from sklearn.preprocessing import PolynomialFeatures
PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)

Feature count grows quadratically, so pair it with regularisation. Domain-driven interactions (price_per_sqm, orders_per_active_day) usually beat exhaustive polynomial expansion — they encode knowledge rather than searching blindly.

Doing it correctly in a pipeline

Different columns need different treatment, and everything must fit on training data only:

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

numeric = Pipeline([
    ("impute", SimpleImputer(strategy="median", add_indicator=True)),
    ("scale", RobustScaler()),
])
categorical = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("encode", OneHotEncoder(handle_unknown="infrequent_if_exist")),
])

pre = ColumnTransformer([
    ("num", numeric, numeric_cols),
    ("cat", categorical, categorical_cols),
])

model = Pipeline([("pre", pre), ("clf", LogisticRegression())])

This is the shape to reproduce in an interview whiteboard. It gets the fit-on-train-only property for free inside cross-validation, and it serialises as one artefact so serving can’t drift from training. See 04_data_leakage.md.

Interview angle

• “Which models need feature scaling?” — anything distance-based or gradient-descent-based, plus anything regularised (the penalty is on coefficient magnitude, which depends on units). Trees and tree ensembles don’t.
• “StandardScaler or MinMaxScaler?” — StandardScaler by default. MinMax when you need a bounded range, but it’s dominated by outliers since it uses min and max. RobustScaler when outliers are present and real.
• “Your amount column is heavily right-skewed. What do you do?” — log1p (zero-safe), or PowerTransformer with Yeo-Johnson if there are negatives. For trees, do nothing — monotone transforms don’t change a tree.
• “How do you handle missing values?” — first ask whether missingness is informative. If it is, keep an indicator column. Median-impute numerics, most-frequent categoricals, and for gradient boosting consider leaving NaN alone since those libraries learn a default direction natively.
• “Where do you put the scaler so cross-validation is honest?” — inside a Pipeline, so it refits on each fold’s training portion. Fitting before splitting leaks test statistics into training.
• “Why doesn’t a random forest need scaling?” — splits are threshold comparisons on a single feature, so any monotone transform, including rescaling, produces the same tree.