Calibration

A model is calibrated when its predicted probabilities mean what they say: among cases predicted at 0.7, about 70% are positive. Ranking metrics like AUC don’t measure this at all, so a model can rank perfectly and still lie about probabilities.

When it matters

It matters when you consume the number, not just the order:

• Expected value: p * revenue - (1-p) * cost
• Pricing and bidding — an ad bid of p_click * value needs p_click to be real
• Risk aggregation — expected defaults across a portfolio
• Thresholds set from probabilities rather than from a validation sweep
• Anything shown to a human as “85% confident”

It doesn’t matter when you only rank or threshold: retrieval ordering, triage queues, top-k selection. If you’re sorting and taking the top 200, monotone distortion of the scores changes nothing.

That distinction is the first thing to say when asked about calibration.

Measuring it

from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss

prob_true, prob_pred = calibration_curve(y_val, scores, n_bins=10, strategy="quantile")
brier = brier_score_loss(y_val, scores)

Reliability diagram — plot prob_true against prob_pred. Perfect calibration is the diagonal.

• Curve below the diagonal → over-confident (predicting 0.9 when reality is 0.6).
• Curve above → under-confident.

Use strategy="quantile" rather than the default uniform bins; with skewed score distributions, uniform bins leave some nearly empty and the curve becomes noise.

Brier score = mean squared error on probabilities. It’s a proper scoring rule, meaning it’s minimised by reporting true probabilities — so it captures both calibration and discrimination in one number. Lower is better, and it decomposes into calibration + refinement.

Expected Calibration Error (ECE) — the weighted average gap between confidence and accuracy across bins. Common in the deep learning literature. Useful but bin-count-sensitive, so report the binning.

Who is and isn’t calibrated

Two mechanisms worth being able to explain:

• Random Forest compresses toward 0.5. A prediction of 1.0 requires every tree to agree; averaging makes extremes rare, so the curve is under-confident at both ends.
• Deep networks over-fit confidence. Trained to minimise cross-entropy with high capacity, they push logits large to squeeze out the last of the loss, producing 0.99 predictions that are right 85% of the time.

Fixing it

Fit a small model mapping raw scores to calibrated probabilities, on held-out data.

from sklearn.calibration import CalibratedClassifierCV

cal = CalibratedClassifierCV(base_estimator, method="isotonic", cv=5)
cal.fit(X_train, y_train)

Isotonic is more flexible and overfits on small samples. Platt is constrained and safer when data is scarce.

Temperature scaling is the standard for deep networks:

# Fit a single scalar T on validation data, then apply at inference
calibrated_logits = logits / T

Because it’s a monotone transform with one parameter, it cannot change the argmax — accuracy and AUC are untouched, only the confidence values move. That property is why it’s preferred over anything more expressive.

The calibration set must be separate from the training set. Calibrating on training data fits the model’s over-confidence on data it memorised, which does nothing. CalibratedClassifierCV with cv=5 handles the split internally.

Class weights break calibration

Worth flagging because it’s a routine trap: class_weight="balanced" and scale_pos_weight re-weight the loss, which shifts predicted probabilities away from the true base rate.

You then have a choice:

1. Don’t re-weight; fix the operating point with the threshold instead, and keep calibrated probabilities.
2. Re-weight for training, then recalibrate on unweighted held-out data.

Doing neither and then using predict_proba in an expected-value calculation is a real and common bug. See ../03_feature_engineering/05_imbalanced_data.md.

Drift

Calibration decays faster than discrimination. If the base rate shifts — a fraud wave, a seasonal effect — a model that still ranks well will be systematically mis-calibrated.

Monitor the gap between mean predicted probability and observed positive rate over a rolling window. It’s cheap and it catches base-rate shift before your accuracy metrics do.

drift = scores_last_7d.mean() - y_last_7d.mean()    # should hover near 0

LLM confidence

Adjacent and increasingly asked: an LLM saying “I’m 90% confident” is not calibrated — it’s generating text that looks like a confidence statement. Token log-probabilities are better grounded but still typically over-confident, and RLHF tends to make calibration worse rather than better.

For anything requiring real confidence estimates from an LLM pipeline, use external signals: retrieval scores, self-consistency across samples, or a separate verifier — not the model’s stated confidence. See ../13_evaluation/.

Interview angle

• “What does it mean for a model to be calibrated?” — among predictions of p, roughly a fraction p are positive. Ranking metrics like AUC ignore this entirely, so good AUC and bad calibration coexist easily.
• “When do you care?” — when the probability feeds a downstream calculation: expected value, bidding, risk aggregation, or a number shown to a user. If you only rank or take top-k, monotone distortion is harmless.
• “How do you check it?” — reliability diagram with quantile bins, plus Brier score as a single proper scoring rule. ECE if you want one number in the deep-learning convention.
• “Random Forest probabilities look compressed toward 0.5. Why?” — they’re vote fractions; reaching 1.0 needs unanimity, and averaging makes extremes rare. It’s under-confident at both ends and isotonic regression fixes it well.
• “How do you calibrate a neural network without hurting accuracy?” — temperature scaling: a single scalar dividing the logits, fitted on validation data. Being monotone with one parameter, it can’t change the argmax, so accuracy and AUC are preserved.
• “You used class_weight='balanced' and now your expected-value calculation is off. Why?” — re-weighting shifts predicted probabilities away from the true base rate. Either drop the weighting and tune the threshold instead, or recalibrate on unweighted held-out data.