Model evaluation
Metrics are where modelling meets the business. Most questions here are really “do you understand what your number means to someone who isn’t an ML engineer”.
| # | File | The question it answers |
|---|---|---|
| 01 | 01_confusion_matrix_accuracy.md | the four cells everything derives from; why accuracy misleads |
| 02 | 02_precision_recall_f1.md | the trade-off, and that it’s a threshold choice not a model property |
| 03 | 03_roc_auc_vs_pr_auc.md | the discriminator question — when ROC-AUC flatters a bad model |
| 04 | 04_regression_metrics.md | MAE vs RMSE, why MAPE is a trap, asymmetric costs |
| 05 | 05_calibration.md | when the probability has to be a real probability |
| 06 | 06_choosing_the_metric.md | working backwards from the decision to the metric |
The two ideas that carry most of this
The threshold is not part of the model. Train once, choose the operating point separately from a business constraint — review capacity, a precision floor, or a cost ratio. Leaving it at 0.5 is a decision by default.
ROC-AUC hides false positives under class imbalance. FPR divides by the total actual negatives, so a huge easy-negative pool absorbs enormous numbers of false alarms with barely any movement. PR-AUC has no true-negative term and reflects what the person reviewing the alerts experiences.