Regression metrics

Less contentious than classification metrics, with one genuinely important choice: how much you want large errors punished.

The core three

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

mae  = mean_absolute_error(y_true, y_pred)
rmse = np.sqrt(mean_squared_error(y_true, y_pred))
r2   = r2_score(y_true, y_pred)

MAE vs RMSE is the real decision. RMSE squares errors before averaging, so one error of 10 counts the same as a hundred errors of 1. Use RMSE when large misses are disproportionately bad (delivery time estimates, capacity planning). Use MAE when all errors are equally bad per unit and outliers are present but legitimate.

RMSE ≥ MAE always, and the gap between them tells you how heavy-tailed your errors are. A large gap means a few big misses dominate — worth investigating those rows directly.

MAE is also the number to quote to non-technical stakeholders: “we’re off by £4,200 on average” is understandable; “RMSE is 6,800” is not.

Which does the model optimise

A subtle and useful point: minimising MSE fits the conditional mean; minimising MAE fits the conditional median.

So if you train with MSE and your target is right-skewed, predictions are pulled upward by the tail. Train with MAE and you get the median, which is often what “typical” means to the business.

LGBMRegressor(objective="regression")     # L2 -> predicts the mean
LGBMRegressor(objective="regression_l1")  # L1 -> predicts the median
LGBMRegressor(objective="huber")          # in between

Evaluating on MAE while training on MSE is a mismatch worth noticing — align them unless you have a reason not to.

R², and how it misleads

R² = 1 - SS_res / SS_tot — the fraction of variance explained relative to predicting the mean.

• 1.0 perfect, 0.0 no better than the mean, negative worse than the mean.
• Not comparable across datasets. R² depends on the variance of the target, so an “easy” dataset with high variance inflates it.
• Inflated by trends. On a time series with a strong trend, a model that just follows the trend scores high R² while adding nothing.
• Never decreases when you add features, which is why adjusted R² exists for model comparison at fixed data.

Treat R² as a sanity check, not a headline metric. If someone reports only R², ask for MAE in target units.

Percentage metrics

from sklearn.metrics import mean_absolute_percentage_error
mape = mean_absolute_percentage_error(y_true, y_pred)

MAPE is popular with stakeholders because it’s scale-free, and it has real problems:

• Undefined or explosive when actuals are near zero. One true value of 0.01 with a prediction of 1 contributes 9,900% error.
• Asymmetric — it penalises over-prediction more than under-prediction, which biases models trained or selected on it toward under-forecasting.

Alternatives: sMAPE (symmetric, still awkward near zero), or WAPE — sum(|y - yhat|) / sum(y) — which is scale-free, robust to zeros, and the one to prefer in demand forecasting.

Errors that aren’t symmetric

When over- and under-prediction cost differently — inventory, staffing, capacity — use quantile (pinball) loss:

LGBMRegressor(objective="quantile", alpha=0.9)   # predict the 90th percentile

Predicting the 90th percentile of demand means you stock out 10% of the time, deliberately. This is the honest way to encode asymmetric cost, and mentioning it signals practical experience.

It also gives you prediction intervals: train models at alpha=0.1 and alpha=0.9 and you have an 80% interval, which is usually more useful to a decision-maker than a point estimate.

Log-transformed targets

For right-skewed targets (prices, counts, durations):

model.fit(X, np.log1p(y))
pred = np.expm1(model.predict(X))

Two things to state:

• Errors are then measured multiplicatively — being 2x off is equally bad at any scale, which is usually what you want for prices.
• Back-transforming gives the median, not the mean. exp(E[log y]) != E[y]. If you need an unbiased mean prediction, apply a correction (Duan smearing) or don’t transform.

RMSLE (root mean squared log error) does the same thing inside the metric, and is standard when under-prediction and over-prediction should be penalised proportionally.

Time series specifics

• Never evaluate with a random split. Use rolling-origin / walk-forward evaluation. See ../01_ml_foundations/02_train_val_test_split.md.
• Compare against a naive baseline — “predict the last value” or “predict last week’s same day”. Beating it is a lower bar than it sounds and many models don’t. MASE formalises this as the ratio to naive-forecast error, where below 1 means you beat naive.
• Evaluate at the horizon you actually forecast. Day-1 accuracy and day-30 accuracy are different problems.

Residual analysis

The metric is a summary; the residuals tell you what to fix.

residuals = y_val - y_pred

• Plot residuals against predictions. Structure means missing signal. A fan shape means heteroscedasticity — consider a log transform.
• Plot residuals against each feature. A pattern shows where the model is systematically wrong.
• Check for bias: residuals.mean() should be near zero. A consistent offset is easy to fix and easy to miss.
• Segment the error. Overall MAE can hide a segment where the model is badly wrong.

Interview angle

• “MAE or RMSE?” — RMSE when large errors are disproportionately costly, since squaring emphasises them; MAE when errors scale linearly with cost or when legitimate outliers would distort RMSE. The gap between them tells you how heavy-tailed your errors are.
• “What does minimising MSE actually fit?” — the conditional mean. MAE fits the conditional median. On skewed targets that difference is large, and it’s why an L1 objective often matches “typical error” better.
• “R² is 0.92. Good model?” — insufficient. R² isn’t comparable across datasets, is inflated by trends, and never decreases as you add features. Ask for MAE in target units and check residuals for structure.
• “Why avoid MAPE?” — it explodes near zero actuals and asymmetrically penalises over-prediction, biasing model selection toward under-forecasting. WAPE is the safer scale-free choice.
• “Over-predicting inventory costs 3x under-predicting. What metric?” — quantile/pinball loss at the appropriate percentile, not a symmetric metric. It encodes the asymmetry directly and gives prediction intervals as a bonus.
• “You log-transformed the target. Anything to watch?” — back-transforming the prediction gives the median rather than the mean, so it’s biased low for the mean. Apply a correction if the mean is what’s needed.