Confusion matrix and accuracy

Where every classification metric comes from. Get comfortable enough with the four cells that you can derive precision, recall and the rest on a whiteboard rather than recalling formulas.

The matrix

                  Predicted
                  Neg     Pos
Actual   Neg      TN      FP     <- FP = false alarm       (Type I error)
         Pos      FN      TP     <- FN = missed detection  (Type II error)

from sklearn.metrics import confusion_matrix
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()

Note sklearn’s ordering: rows are actual, columns are predicted, and ravel() gives tn, fp, fn, tp. Many textbooks draw it transposed, so always check the axis labels before reading someone else’s matrix.

Naming the errors in domain terms is what makes the discussion useful:

Which is worse is a business question, and every metric choice downstream follows from it.

Accuracy, and why it misleads

accuracy = (TP + TN) / (TP + TN + FP + FN)

Fine when classes are balanced and errors cost roughly the same. Actively harmful otherwise.

# 0.1% fraud
y_true = np.zeros(100_000); y_true[:100] = 1
y_pred = np.zeros(100_000)          # predict "never fraud"
accuracy_score(y_true, y_pred)      # 0.999 - and the model is worthless

The rule: quote the majority-class rate alongside accuracy, or don’t quote accuracy. “99.9% accurate” means nothing until you know the baseline is 99.9%.

Balanced accuracy is the fix when you still want a single accuracy-like number:

from sklearn.metrics import balanced_accuracy_score   # mean of per-class recall

The derived rates

All four cells give rise to a rate, and the confusion is usually about which denominator is which:

The memory hook: precision is a column, recall is a row. Precision divides by a predicted column; recall divides by an actual row.

Trivial extremes worth naming, because they show the metrics need each other:

• Predict positive for everything → recall 1.0, precision equals the base rate.
• Predict positive for the single most confident case → precision likely 1.0, recall ~0.

Multi-class

from sklearn.metrics import classification_report
print(classification_report(y_true, y_pred, digits=3))

The matrix becomes N×N, and the off-diagonal cells are the informative part: which classes get confused with which. That tells you whether to merge two labels, add features that distinguish them, or gather more data for one.

Averaging choices matter a great deal:

With imbalanced classes, macro and weighted can differ dramatically. Macro-F1 dropping while weighted-F1 holds means you’re failing the rare classes — exactly the thing an aggregate hides.

Reading a confusion matrix in review

Practical habits:

1. Normalise by row to see per-class recall: confusion_matrix(..., normalize="true").
2. Look at the largest off-diagonal cells — that’s your biggest fixable error mode.
3. Check whether confusion is symmetric. A confused with B but not B with A suggests a threshold or prior issue rather than genuinely similar classes.
4. Sample actual errors from the biggest cell and read them. Frequently they’re mislabelled rather than mispredicted.

Cost-weighted evaluation

When errors have different costs, evaluate expected cost directly rather than proxying with F1:

COST_FP, COST_FN = 5, 500          # review time vs missed fraud

def expected_cost(y_true, scores, threshold):
    pred = scores >= threshold
    fp = ((pred == 1) & (y_true == 0)).sum()
    fn = ((pred == 0) & (y_true == 1)).sum()
    return fp * COST_FP + fn * COST_FN

best = min(np.linspace(0.01, 0.99, 99), key=lambda t: expected_cost(y_val, scores, t))

This is the honest version of threshold selection, and proposing it unprompted signals that you’ve operated a model rather than only trained one.

Interview angle

• “Define precision and recall.” — precision is TP/(TP+FP), the fraction of positive predictions that are correct; recall is TP/(TP+FN), the fraction of actual positives found. Precision divides by a predicted column, recall by an actual row.
• “Your model is 99% accurate. Is that good?” — unknown without the class balance. If 99% of cases are negative, predicting all-negative scores the same. Ask for the base rate, then move to precision/recall or PR-AUC.
• “Which error is worse, FP or FN?” — a domain question. Name both in the domain’s terms and ask for the cost ratio; that ratio sets the threshold. Refusing to answer in the abstract is the correct answer.
• “Macro vs weighted F1?” — macro averages per-class scores equally so rare classes count fully; weighted averages by support so the result tracks the majority. If macro is much lower, you’re failing the small classes.
• “How do you use a confusion matrix in practice?” — normalise by row for per-class recall, find the largest off-diagonal cell, and read real examples from it. The most common discovery is label noise, not model error.
• “Can precision and recall both be high?” — yes, on an easy problem. They trade off along the threshold, but the whole curve moves up with a better model — that’s what PR-AUC measures.