Monitoring and drift

Models degrade silently. Nothing raises an exception when accuracy falls, which makes monitoring the difference between a system that works and one that used to.

The four layers

You get these in that order of latency. Operational is immediate; outcome may take weeks. The layers above outcome are your early-warning system, and that’s the point of monitoring inputs at all.

The label delay problem

For fraud you might wait weeks for a chargeback; for churn, months. You cannot wait for ground truth to notice a problem.

So you monitor proxies:

• Input drift — the data looks different from training.
• Prediction drift — the model’s output distribution moved.
• Confidence shift — mean predicted probability moving away from the historical base rate.

drift = scores_last_7d.mean() - historical_positive_rate
# sustained non-zero means the model's calibration has shifted

That last one is cheap and catches base-rate change before any accuracy metric could. See ../04_model_evaluation/05_calibration.md.

Data drift vs concept drift

The distinction that gets asked:

Data drift is measurable immediately. Concept drift is only detectable once labels arrive, which is why input monitoring alone is insufficient and why you need a labelling pipeline even when it’s slow.

Measuring drift

from scipy.stats import ks_2samp

def psi(expected, actual, bins=10):
    """Population Stability Index. <0.1 stable, 0.1-0.25 moderate, >0.25 significant."""
    e_pct, edges = np.histogram(expected, bins=bins)
    a_pct, _ = np.histogram(actual, bins=edges)
    e_pct = np.clip(e_pct / e_pct.sum(), 1e-6, None)
    a_pct = np.clip(a_pct / a_pct.sum(), 1e-6, None)
    return float(((a_pct - e_pct) * np.log(a_pct / e_pct)).sum())

Test per feature, and correct for multiple comparisons. Testing 200 features at p<0.05 gives you ten “significant” drifts by chance every run. PSI’s threshold-based interpretation avoids that trap better than repeated hypothesis tests.

Also: not all drift matters. A feature the model barely uses can drift freely. Weight drift alerts by feature importance, or you’ll train the team to ignore the alerts.

Retraining

Scheduled retraining is the pragmatic default. Drift-triggered sounds better and generates false alarms until the thresholds are tuned.

Whatever the trigger, retraining must be gated: the new model has to beat the current one on a held-out set before promotion, ideally with a shadow period first. Automatic retraining that automatically deploys is how a bad data day becomes a bad model.

Monitoring LLM applications

Different signals, same structure:

Cost is a first-class production metric here, unlike classical ML where inference is nearly free. A prompt change that adds 500 tokens to every request is a budget event.

Two LLM-specific alarms worth naming:

• Refusal rate moving in either direction. A spike means over-refusal is damaging usefulness; a drop may mean safety behaviour regressed.
• Provider model change. Pin versions and alert on the version field changing — otherwise your quality moves with no deployment on your side.

Continuous evaluation: sample production traffic daily and score it with the offline judge and rubric. It catches drift a static eval set cannot. See ../13_evaluation/04_online_eval_and_experiments.md.

What to alert on

Alert on things a human should act on tonight:

• SLO breach (latency, error rate)
• Cost anomaly — a sudden multiple of the baseline
• Schema-validation failure rate above threshold
• Drift on an important feature
• Outcome metric decline
• Dead-letter queue depth

Don’t alert on every drifting feature or every low-confidence prediction. Alert fatigue is the failure mode, and a dashboard nobody reads is the same as no monitoring.

Interview angle

• “How do you monitor a model in production?” — four layers: operational telemetry, input data distributions, prediction distributions, and the outcome metric once labels arrive. The upper layers are the early-warning system because outcome data lags by weeks.
• “Data drift vs concept drift?” — data drift is a change in P(X), detectable immediately without labels. Concept drift is a change in P(y|X) — the relationship itself — and only detectable once labels arrive. That asymmetry is why input monitoring alone isn’t enough.
• “How do you detect drift?” — PSI for binned features with its standard thresholds, KS for continuous, chi-squared for categorical, embedding-centroid distance for text. Weight alerts by feature importance, because drift in an unused feature is noise.
• “When do you retrain?” — on a schedule matched to domain velocity is the pragmatic default; drift-triggered is reactive and generates false alarms until tuned. Either way, gate promotion on beating the incumbent on a held-out set, with a shadow period.
• “What’s different about monitoring an LLM feature?” — token spend becomes a first-class metric, you watch schema-validation and refusal rates, you sample production traffic for continuous scoring, and you alert on the provider’s model version changing — because that shifts behaviour without any deploy from you.
• “You have 200 features and drift tests fire constantly. What’s wrong?” — multiple comparisons. At p<0.05 you expect ten false positives per run. Use PSI thresholds rather than repeated hypothesis tests, and gate alerts on feature importance.