Imbalanced data

Fraud, churn, defects, rare disease, ad clicks — the interesting class is almost always the rare one. Everything about the default ML workflow assumes balance, and each assumption fails here.

Why it breaks things

At 0.1% positives, predicting “always negative” gives you 99.9% accuracy and a useless model. Beyond the metric problem:

• Gradient updates are dominated by the majority class.
• The default 0.5 threshold is far from optimal.
• Cross-validation folds may contain almost no positives, making the metric noisy.

Also: imbalance itself is not the problem — insufficient minority examples is. A dataset with 1% positives out of 10 million rows has 100,000 positives and will train fine. The same ratio out of 5,000 rows has 50 positives and won’t. Ask for the absolute count, not just the ratio. That reframing is a strong interview answer.

Fix the metric first

This is the highest-leverage change and it costs nothing.

from sklearn.metrics import average_precision_score
average_precision_score(y_true, y_score)     # PR-AUC

The ROC-AUC point is the one that gets probed: with 99.9% negatives, adding a thousand false positives barely moves the false-positive rate (it’s divided by an enormous denominator), so ROC-AUC stays high while precision collapses. PR-AUC has no true-negative term and reflects what the reviewer experiences. See ../04_model_evaluation/03_roc_auc_vs_pr_auc.md.

Tune the threshold

Train once, choose the operating point separately. The threshold is a business decision, not a model parameter.

from sklearn.metrics import precision_recall_curve
import numpy as np

precision, recall, thresholds = precision_recall_curve(y_val, scores)

# Highest recall subject to precision >= 0.30 (the review team's floor)
ok = precision[:-1] >= 0.30
best = thresholds[ok][np.argmax(recall[:-1][ok])]

Better still, derive it from costs. If a missed fraud costs $500 and a false alarm costs $5 of review time, the expected-cost-minimising threshold follows directly from that 100:1 ratio.

Class weights

The first thing to try. Cheap, no data manipulation, no leakage risk.

LogisticRegression(class_weight="balanced")
RandomForestClassifier(class_weight="balanced_subsample")
XGBClassifier(scale_pos_weight=(y == 0).sum() / (y == 1).sum())

It re-weights the loss so minority errors count more — equivalent to oversampling, without duplicating rows.

The catch: re-weighting distorts predicted probabilities. The ranking stays useful, but predict_proba no longer reflects the true base rate. If the probability feeds an expected-value calculation, either skip re-weighting or recalibrate afterwards. See ../04_model_evaluation/05_calibration.md.

Resampling

from imblearn.pipeline import Pipeline as ImbPipeline
from imblearn.over_sampling import SMOTE

pipe = ImbPipeline([
    ("smote", SMOTE(random_state=42)),
    ("model", LGBMClassifier()),
])

Use imblearn.pipeline.Pipeline, not sklearn’s. The imblearn version applies resampling only to the training portion of each fold. The sklearn one will resample the validation fold too, which contaminates your metric — a very common and very quiet bug. See 04_data_leakage.md.

Never resample the test set. It must reflect the real distribution, or your reported precision is fiction.

Honest assessment: SMOTE is less useful than its popularity suggests. On tabular data with a strong gradient boosting model, class weights plus threshold tuning usually match or beat it. SMOTE interpolates between minority points, which assumes the space between two fraud cases is also fraud — often false, and worse in high dimensions where interpolation lands in empty regions. Try weights first.

Model choice

Gradient boosting handles imbalance well with scale_pos_weight. Tree ensembles generally cope better than linear models because they can carve out small pure regions.

For extreme imbalance (< 0.1%), consider reframing as anomaly detection — train only on the majority class and flag deviations:

from sklearn.ensemble import IsolationForest
from sklearn.svm import OneClassSVM

This is the right move when positives are so rare, or so varied, that there’s no coherent “positive class” to learn.

Getting more minority data

Often better than any algorithmic fix:

• Targeted labelling. Have reviewers label cases the current model scores as borderline — active learning. Far more efficient than random labelling.
• Longer history for the minority class specifically.
• Related labels. Chargebacks, complaints, and manual reviews are all weak signals of fraud.
• Cost-sensitive framing. Sometimes the answer is not more data but accepting a low-precision, high-recall model feeding a human review queue.

Interview angle

• “How do you handle a 99:1 class imbalance?” — start by asking the absolute number of positives, since 1% of 10M is plenty. Then: switch the metric to PR-AUC, use class weights, tune the threshold from the cost ratio, and only then consider resampling. Never resample the test set.
• “Why not ROC-AUC here?” — the false-positive rate is divided by a huge true-negative count, so a large number of false alarms barely moves it. ROC-AUC stays high while precision is terrible. PR-AUC has no true-negative term.
• “Is SMOTE a good default?” — not really. It assumes the interpolation between two minority points is also minority, which often fails and fails harder in high dimensions. Class weights plus threshold tuning usually do as well with less risk. If you do use it, it must run inside the training fold only.
• “Class weights vs oversampling?” — mathematically similar; weights avoid duplicating data and are simpler to get right. Both distort probability calibration, so recalibrate if the number matters downstream.
• “Where do you set the decision threshold?” — from the cost matrix, on validation data, separately from training. If a miss costs 100x a false alarm, that ratio determines the operating point, not a symmetric metric like F1.
• “Positives are 0.01% and highly varied. Still a classifier?” — consider anomaly detection instead: model normality and flag deviations. With that few positives there may be no coherent positive class to learn.