Choosing the metric

The decision that quietly determines everything downstream — what you optimise, which model wins, and whether the deployed system helps the business. Interviewers use it to check whether you connect modelling to outcomes.

The chain

Business outcome → decision → error costs → metric → threshold.

Work it in that order. Starting at “which metric is best?” is starting in the middle.

1. What decision does the prediction drive? Block a transaction, order stock, prioritise a queue, show a result.
2. What does each error cost? In money, time or risk. Get a ratio even if you can’t get absolute numbers — “a miss is roughly 100x a false alarm” is enough.
3. Is there a capacity constraint? “The team reviews 200 cases a day” caps recall regardless of the model, and makes precision@200 the natural metric.
4. Do you need the probability or just the order? Determines whether calibration matters.
5. Which segments must not fail? An aggregate metric can hide a broken segment.

The default choices

One primary, several guardrails

Pick one metric to optimise and decide by. Track others as guardrails that must not degrade.

Primary:    PR-AUC
Guardrails: p99 latency < 50ms
            precision >= 0.30 at the deployed threshold
            no segment's recall below 0.60
            inference cost per 1k requests

Multiple co-equal “primary” metrics means no decision rule — when A improves and B degrades, nothing tells you what to do. Naming one primary is a small thing that signals decisiveness.

Offline vs online

They differ, and that gap is a whole discussion:

Offline metrics are a proxy. The proxy is good when a change in it reliably predicts a change in the business metric — which is worth actually validating once, by comparing a few offline deltas against their A/B outcomes.

Classic ways the proxy breaks:

• Feedback loops. A recommender trained on clicks it caused. The offline metric improves because the model is grading its own homework.
• Position bias. Item-1 clicks reflect position, not relevance.
• The metric isn’t the goal. Optimising click-through rate produces clickbait; watch time produces autoplay traps. The measurable proxy diverges from the thing you wanted.

Naming Goodhart’s law here — a measure that becomes a target stops being a good measure — is a legitimate senior point, provided you follow it with the mitigation: guardrail metrics, holdout populations, and long-horizon outcome tracking.

Segment before you decide

for segment, idx in segments.items():
    print(segment, average_precision_score(y[idx], scores[idx]))

An aggregate metric that improved can hide a segment that got worse — new users, a region, a device type, a protected group. Check before shipping, both for quality and fairness reasons.

Make the comparison honest

• Report a confidence interval, not a point estimate. Bootstrap the metric or use cross-validation’s standard deviation.
• Two models within noise of each other are equal; ship the simpler, cheaper one.
• Fix the test set. Comparing models scored on different splits is meaningless.

from sklearn.utils import resample
import numpy as np

boots = [average_precision_score(*resample(y_val, scores, random_state=i))
         for i in range(1000)]
lo, hi = np.percentile(boots, [2.5, 97.5])

A “+0.4% AUC” improvement with a ±1.5% interval is not an improvement.

LLM systems

The same chain applies with different instruments. There’s no single number, so you build a scorecard:

Cost and latency are first-class here, unlike in classical ML where inference is nearly free. “Cost per successfully resolved ticket” is often the metric that actually matters, and it’s the kind of framing that lands well. Detail in ../13_evaluation/.

Interview angle

• “How do you pick an evaluation metric?” — start from the decision the prediction drives and the cost of each error, not from a list of metrics. Capacity constraints and whether you need calibrated probabilities narrow it further. Then pick one primary metric and a set of guardrails.
• “Why one primary metric?” — with two co-equal metrics there’s no decision rule when they disagree. One primary plus explicit guardrails keeps the decision unambiguous.
• “Offline metrics improved but the A/B test was flat. What happened?” — the proxy diverged from the business metric. Common causes: feedback loops in the training data, position bias, a distribution shift between the offline set and live traffic, or the metric measuring something users don’t value.
• “Model A beats Model B by 0.3% AUC. Ship it?” — not without a confidence interval. If the bootstrap interval spans zero they’re equivalent, and you should ship whichever is simpler, cheaper or more explainable.
• “How is metric choice different for an LLM feature?” — no single number. You build a scorecard across correctness, groundedness, format compliance, safety, cost and latency, anchored on a curated eval set. Cost per resolved task is often the metric that decides.
• “Your model improved overall but is worse for new users. Ship it?” — no, or not without a guardrail. Aggregate gains that mask segment regressions are how models quietly damage the experience for the group that matters most.