Data leakage

The single most expensive bug in ML. It doesn’t crash, it doesn’t warn — it makes your metrics better, which is exactly why it survives review and reaches production.

Rule of thumb: if your model is unexpectedly good, assume leakage until you’ve proven otherwise. A senior engineer’s reflex on hearing “we got 0.99 AUC” is suspicion, not celebration.

The two kinds

Target leakage — a feature contains information about the label that won’t be available at prediction time.

Train-test contamination — information from the evaluation set influenced training.

They fail differently. Target leakage produces a model that’s useless in production. Contamination produces a model that’s fine but whose reported metric is a lie.

Target leakage

The feature is a consequence of the label

You’re predicting whether a customer will churn. Your feature set includes account_closed_date. That’s not a predictor, it’s the outcome.

Subtler versions are the ones that get shipped:

The test is always the same question: at the moment I need this prediction, is this value populated, and is it populated with the same distribution as in training?

Timing leakage

The feature exists at prediction time but was computed using future data.

# WRONG - the mean is computed over the whole dataset, including future rows
df["user_avg_order"] = df.groupby("user_id")["amount"].transform("mean")

# RIGHT - only past rows contribute
df = df.sort_values("timestamp")
df["user_avg_order"] = (
    df.groupby("user_id")["amount"]
      .transform(lambda s: s.shift(1).expanding().mean())
)

Any aggregate — rolling means, counts, target encodings — must be computed with a strict cutoff. shift(1) before the window is the pattern; forgetting it means each row’s feature includes its own label.

Target encoding, the classic trap

Encoding a category by its mean target is powerful and leaks by construction, because each row’s encoding includes that row’s own label.

# WRONG
df["city_encoded"] = df.groupby("city")["target"].transform("mean")

Fixes: out-of-fold encoding (compute each fold’s encoding from the other folds), leave-one-out, smoothing toward the global mean, or use CatBoost, whose ordered target statistics solve this natively. See ../02_classical_ml/05_gradient_boosting.md.

Proxy leakage

No single feature leaks, but a combination reconstructs the label. A record_id that was assigned sequentially after case resolution. A file path containing the class name. Row order that happens to be sorted by target.

# If shuffling the index destroys performance, the index was a feature
df = df.sample(frac=1, random_state=0).reset_index(drop=True)

Train-test contamination

Preprocessing fitted before the split

Covered in ../01_ml_foundations/02_train_val_test_split.md, and worth repeating because it’s the most common form:

# WRONG - scaler saw the test set
X_scaled = StandardScaler().fit_transform(X)
X_tr, X_te = train_test_split(X_scaled, ...)

# RIGHT - a Pipeline makes this structurally impossible
pipe = Pipeline([("scale", StandardScaler()), ("model", LogisticRegression())])
cross_val_score(pipe, X, y, cv=5)     # refits the scaler inside every fold

Applies to imputation, feature selection, PCA, target encoding, and resampling.

Duplicates across the split

Scraped datasets, product catalogues and document corpora are full of near-duplicates. One copy in train and one in test is memorisation scored as generalisation. Deduplicate before splitting — exact hash for identical rows, MinHash or embedding similarity for near ones.

Group leakage

Multiple rows per entity. The same patient, user or document in both sets lets the model memorise the entity. Use GroupKFold.

Oversampling before the split

# WRONG - SMOTE creates synthetic points from test rows, then they land in train
X_res, y_res = SMOTE().fit_resample(X, y)
X_tr, X_te = train_test_split(X_res, y_res, ...)

Use imblearn.pipeline.Pipeline, which applies resampling only to the training portion of each fold. The sklearn Pipeline does not do this correctly for samplers.

Tuning against the test set

Not a code bug — a process bug. Every decision made while looking at test performance leaks a little information. After fifty experiments, the test score is optimistic. That’s what a separate validation set is for.

How to detect it

A feature with implausibly high importance. If one feature carries almost all the signal, interrogate it.

from sklearn.inspection import permutation_importance
r = permutation_importance(model, X_val, y_val, n_repeats=10)

Near-perfect metrics. 0.99 AUC on a genuinely hard problem is a bug report, not a result.

Ablate the suspect. Drop the feature and retrain. If performance collapses to near-baseline, that feature was the model.

Sort by time and re-split. If a temporal split performs far worse than a random one, you had timing leakage.

Audit each feature’s availability. For every feature, write down when its value becomes known relative to the prediction moment. Tedious, and it catches most target leakage before you write any code.

Shadow deployment. Run the model on live traffic without acting on it and compare to offline metrics. This is the ultimate check because it uses genuinely-available-at-prediction-time data. See ../01_ml_foundations/05_ml_lifecycle.md.

Prevention, structurally

• Point-in-time correctness. Build features from a timestamped event log with an explicit “as of” cutoff, not from a mutable current-state table. A users table with a status column that’s been overwritten cannot tell you what the status was last March.
• Feature stores enforce this and serve the same definition to training and serving. See 06_feature_stores.md.
• Pipelines for all preprocessing.
• Written feature contracts — a one-line note per feature stating when it’s available.

Interview angle

• “What is data leakage?” — information available at training that won’t be available (or won’t be distributed the same way) at prediction time. It inflates offline metrics and destroys production performance. Split into target leakage and train-test contamination.
• “Give a concrete example.” — predicting loan default with a collections_agency_assigned flag, which is only set after default. Or churn with support_tickets_last_7d, which is a symptom of leaving rather than a predictor.
• “Your model gets 0.99 AUC. Reaction?” — suspicion. Check feature importances for one dominant feature, ablate it, audit availability at prediction time, and re-split temporally. Genuine 0.99 on a hard problem is rare; leakage is common.
• “Why is target encoding dangerous?” — each row’s encoded value includes that row’s own label. Fix with out-of-fold or leave-one-out encoding, smoothing, or CatBoost’s ordered target statistics.
• “You scaled before splitting. What’s the impact?” — the scaler’s statistics came from the full dataset, so test information influenced training. The metric is optimistic. Wrap preprocessing in a Pipeline so it refits per fold.
• “How do you prevent leakage structurally rather than by vigilance?” — point-in-time feature computation from an event log, a feature store serving one definition to both paths, pipelines for preprocessing, and shadow deployment as the final check.