ROC-AUC vs PR-AUC

A reliable discriminator in interviews, because plenty of people report ROC-AUC by habit without knowing when it flatters a model.

ROC curve

True positive rate against false positive rate, across all thresholds.

TPR = TP / (TP + FN)      # recall - denominator is all actual positives
FPR = FP / (FP + TN)      # denominator is all actual NEGATIVES

from sklearn.metrics import roc_auc_score, roc_curve
roc_auc_score(y_true, scores)

Interpretation: ROC-AUC is the probability that a randomly chosen positive is scored higher than a randomly chosen negative. 0.5 is chance, 1.0 is perfect. That probabilistic reading is the cleanest one-line answer to “what does AUC mean”.

Useful properties: threshold-independent, and invariant to class balance — the baseline is always 0.5 regardless of how rare positives are.

PR curve

Precision against recall, across all thresholds.

from sklearn.metrics import average_precision_score
average_precision_score(y_true, scores)     # PR-AUC / average precision

The baseline is the positive rate, not 0.5. On a 1%-positive dataset, a random classifier scores 0.01. So PR-AUC must always be reported alongside the base rate or it’s uninterpretable.

The difference that matters

Look at the denominators. FPR divides by the total number of actual negatives. When negatives massively outnumber positives, that denominator is enormous, so even a large absolute number of false positives barely moves FPR — and ROC-AUC stays high.

Concretely, 1,000,000 transactions with 1,000 fraudulent:

Model B produces ten times the false alarms. Its FPR moves from 1% to 10% — a change that looks modest on an ROC curve. Its precision collapses from 8% to under 1%, meaning the review team now works through 111 cases to find one fraud instead of 12.

ROC-AUC understates the difference; PR-AUC reflects it. Because PR has no true-negative term anywhere, it can’t be flattered by a big pool of easy negatives.

Which to use

For fraud, churn, defect detection, rare disease, click prediction, retrieval — PR-AUC. That’s most interesting problems.

ROC-AUC’s base-rate independence is genuinely useful for one thing: comparing a model across populations with different prevalence. PR-AUC would change even if the model didn’t.

Threshold-free metrics measure ranking

Both summarise the model across all thresholds, so both measure ranking quality rather than decision quality. A model can have excellent AUC and be useless as deployed because the threshold is wrong, or because its probabilities are miscalibrated.

Consequence: report AUC to compare models, and report precision/recall at your actual operating point to describe what will happen in production. Doing only the first is a common gap.

Related and adjacent

• average_precision_score vs auc(recall, precision) — prefer the former. Trapezoidal interpolation of a PR curve is optimistically biased because PR curves aren’t monotone; average precision uses a step-wise sum that avoids it.
• ROC-AUC for multi-class — roc_auc_score(..., multi_class="ovr", average="macro"). Be explicit about the averaging.
• Partial AUC — when only a region matters (say FPR below 1%), integrate over that region only. Common in security and medical screening.
• Gini = 2 * AUC - 1. Same information; standard in credit scoring.

Interview angle

• “What does ROC-AUC mean?” — the probability that a random positive scores above a random negative. 0.5 is chance regardless of class balance.
• “When is ROC-AUC misleading?” — under heavy class imbalance. FPR’s denominator is all actual negatives, so a huge easy-negative pool absorbs large numbers of false positives with little movement in FPR. The model looks strong while precision is terrible.
• “So which do you report for fraud detection?” — PR-AUC, with the positive base rate stated, plus precision and recall at the deployed threshold. ROC-AUC optionally, for comparison across populations with different prevalence.
• “PR-AUC is 0.35. Is that good?” — meaningless without the base rate. At 35% positives it’s chance; at 1% it’s very strong.
• “Model A has higher AUC but Model B performs better in production. How?” — AUC measures ranking across all thresholds; production uses one. B may rank better in the region you actually operate in, or be better calibrated. Compare at the operating point, or use partial AUC over the relevant range.
• “Can a model have 0.99 AUC and be useless?” — yes. If you need precision at very high recall and the last few percent of positives are ranked poorly, the deployed operating point can still be bad. AUC is an average over regions you may never use.