Bias-variance trade-off
The framework for answering “my model isn’t good enough — what do I do?” without guessing. It’s asked constantly because it separates people who tune randomly from people who diagnose.
The decomposition
For squared error, expected test error splits into three parts:
E[(y - f_hat(x))^2] = Bias[f_hat]^2 + Var[f_hat] + irreducible_noise| Term | Means | Caused by |
|---|---|---|
| Bias | systematic error — the model can’t represent the truth | too simple a model, wrong assumptions |
| Variance | sensitivity to the particular training sample | too flexible a model, too little data |
| Irreducible | noise in the labels themselves | measurement error, genuine randomness |
You cannot beat the irreducible term. Recognising when you’re near it — and saying so instead of tuning for another two weeks — is a senior behaviour.
Diagnosing from the learning curve
Compare training error against validation error. This one table answers most “what do I do next” questions:
| Train error | Validation error | Diagnosis | Do this |
|---|---|---|---|
| high | high (close to train) | underfitting / high bias | bigger model, better features, train longer, less regularisation |
| low | high (big gap) | overfitting / high variance | more data, more regularisation, simpler model, augmentation |
| low | low | you’re done | ship it |
| high | low | bug | check your split, leakage, or a broken metric |
That last row is not a joke — validation beating training usually means the validation set is easier, dropout is on during training but off at eval, or your splits are wrong.
from sklearn.model_selection import learning_curve
import numpy as np
sizes, train_scores, val_scores = learning_curve(
model, X, y, cv=5,
train_sizes=np.linspace(0.1, 1.0, 10),
scoring="neg_mean_absolute_error",
)Reading the curve is the real skill. If validation error is still falling as you add data, more data will help. If both curves have flattened with a gap between them, more data will help slowly and regularisation will help faster. If they’ve converged to a high value, more data is pointless — you have a bias problem, and you need a better model or better features.
That distinction saves real money: “should we buy more labelled data?” is answerable from a learning curve, not from intuition.
Model complexity and the classical U-curve
As you increase capacity, bias falls and variance rises. Total error traces a U, and the minimum is where you want to sit.
| Lever | Increases capacity | Decreases capacity |
|---|---|---|
| Tree depth | deeper | shallower, min_samples_leaf |
| Polynomial degree | higher | lower |
| Neural network | wider/deeper | smaller, dropout, weight decay |
| kNN | smaller k | larger k |
| Regularisation | weaker (small alpha) | stronger (large alpha) |
Note kNN is inverted: k=1 is maximum variance (memorises every point), large k is maximum bias (predicts near the global mean).
Ensembles attack one term each
This is the crisp way to explain bagging vs boosting, and it’s a favourite follow-up:
- Bagging (Random Forest) trains many high-variance models on bootstrap samples and averages them. Averaging reduces variance without raising bias much — so you deliberately use deep, overfitting trees as the base learner.
- Boosting (XGBoost, LightGBM) trains shallow, high-bias models sequentially, each correcting the previous one’s errors. It reduces bias — so the base learner is a stump or a depth-6 tree, and you control variance with learning rate, subsampling and early stopping.
Same ensemble idea, opposite mechanism, opposite base learner. See ../02_classical_ml/04_random_forest_bagging.md and ../02_classical_ml/05_gradient_boosting.md.
Where the classical picture breaks: double descent
The textbook U-curve says that past a certain capacity, test error rises forever. Modern deep networks don’t behave that way. Past the interpolation threshold — where the model can fit the training data exactly — test error often falls again. That’s “double descent”, and it’s why models with far more parameters than data points generalise well.
The practical consequence: “the model has more parameters than training examples, so it must overfit” is an outdated statement. It was correct for classical models and is not reliable for large neural networks. Being able to say that, with the caveat that regularisation and data scale still matter, signals current knowledge.
Bias-variance in the LLM era
The framing still applies, with different levers:
| Symptom | Analogue | Fix |
|---|---|---|
| Model gives generic, shallow answers | high bias | better prompt, more context, bigger model |
| Answers vary wildly between runs | high variance | lower temperature, structured output, few-shot examples |
| Great on your examples, poor on real traffic | overfitting to the prompt | broader eval set, test on held-out real queries |
| Confidently wrong on facts | irreducible without retrieval | RAG — add grounding rather than tuning |
Few-shot examples in a prompt are a variance-reduction technique: they constrain the output distribution toward the shape you want.
Interview angle
- “Explain the bias-variance trade-off.” — bias is systematic error from a model too simple to capture the pattern; variance is sensitivity to the specific training sample. Increasing capacity trades one for the other, and total error is minimised somewhere in between, above an irreducible noise floor.
- “Training accuracy 0.99, validation 0.72. What now?” — high variance. In order: check for leakage or a broken split first, then more data, stronger regularisation, simpler model, augmentation, early stopping.
- “Both are 0.65. What now?” — high bias. More capacity, better features, longer training, less regularisation. More data will not help, and the learning curve will show you that.
- “How do you know whether more data will help?” — plot the learning curve. If validation error is still declining with training-set size, buy more data. If the curves have converged, spend the money on features or model class instead.
- “How do bagging and boosting differ in this framing?” — bagging averages many high-variance learners to cut variance; boosting sequences many high-bias learners to cut bias. That’s why Random Forest uses deep trees and gradient boosting uses shallow ones.
- “Doesn’t a model with billions of parameters and less data automatically overfit?” — that’s the classical expectation, and modern deep networks violate it. Past the interpolation threshold, test error often descends a second time. The classical U-curve is a good mental model for classical ML, not a law.