Overfitting and regularisation

Overfitting is learning the training set’s noise as if it were signal. Regularisation is any technique that trades a little training accuracy for better generalisation.

Recognising it

The signature is a widening gap between training and validation performance, usually with validation loss turning upward while training loss keeps falling.

Before you reach for regularisation, rule out the cheaper explanations:

1. Leakage — a feature that encodes the answer. Suspiciously good results are leakage until proven otherwise.
2. Broken split — duplicates, group leakage, temporal leakage.
3. Too little data for the capacity you’ve chosen.

Only then is it genuine overfitting.

L1 and L2

Add a penalty on weight magnitude to the loss:

L2 (Ridge):  loss + alpha * sum(w**2)
L1 (Lasso):  loss + alpha * sum(abs(w))

Why L1 produces exact zeros is the follow-up question that separates memorised from understood. L1’s penalty has a constant gradient (±alpha) regardless of how small the weight is, so it keeps pushing until the weight hits zero and stops there. L2’s gradient is 2 * alpha * w, which shrinks as the weight shrinks — asymptotically approaching zero without arriving.

Elastic Net combines both, which handles correlated features better than pure Lasso:

from sklearn.linear_model import ElasticNet
model = ElasticNet(alpha=0.1, l1_ratio=0.5)   # 0 = pure Ridge, 1 = pure Lasso

Regularisation requires scaled features. Penalising sum(w**2) treats all weights alike, so a feature measured in millimetres gets a large coefficient and is penalised far more than the same feature in metres. Always scale before regularising.

Early stopping

Stop when validation stops improving. It’s regularisation by limiting how far optimisation travels from the initialisation.

model = XGBClassifier(n_estimators=10_000, early_stopping_rounds=50)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)])
print(model.best_iteration)

Cheap, effective, and almost free — you were going to hold out a validation set anyway. Set n_estimators deliberately high and let early stopping choose.

Dropout

Randomly zero a fraction of activations during training; disable at inference (frameworks handle the switch when you call model.eval()).

nn.Sequential(
    nn.Linear(768, 256), nn.ReLU(), nn.Dropout(0.3),
    nn.Linear(256, 10),
)

Interpretations worth knowing: it prevents co-adaptation between neurons, and it approximates training an ensemble of exponentially many sub-networks that share weights.

Forgetting model.eval() at inference is a classic bug — dropout stays active and your predictions become non-deterministic and worse. The same call also switches batch norm from batch statistics to running averages.

Typical rates: 0.1-0.3 for transformers, 0.5 for wide fully-connected layers. Modern large models often use very little dropout, relying on data scale instead.

More data, and augmentation

More data is the strongest regulariser there is, and often the cheapest to reason about. When you can’t get more, manufacture variants that preserve the label:

For text in 2026, LLM-generated paraphrases are the practical default — cheaper and higher quality than back-translation, which was the previous standard. Watch for label drift: a paraphrase that changes sentiment is a mislabelled example.

Batch normalisation and layer normalisation

Both stabilise training by normalising activations; both have a mild regularising side effect.

Transformers use LayerNorm precisely because it’s batch-independent, which matters with variable sequence lengths and small per-device batches. RMSNorm — LayerNorm without the mean-centring step — is common in recent LLMs for being slightly cheaper.

Other levers

• Weight decay — L2 applied directly to weights in the optimiser. Use AdamW, not Adam with weight_decay; see ../00_math_foundations/03_calculus_optimization.md.
• Label smoothing — replace one-hot targets with 1-eps and eps/(K-1). Discourages overconfidence and improves calibration.
• Gradient clipping — bounds step size; stabilises rather than regularises, but often bundled in.
• Constrain the model — fewer trees, shallower depth, smaller hidden size. The most direct lever and frequently overlooked.
• Ensembling — averaging several models reduces variance; see 03_bias_variance_tradeoff.md.

Tuning the regularisation strength

Search on a log scale — the useful range spans orders of magnitude.

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
import numpy as np

grid = GridSearchCV(
    LogisticRegression(penalty="l2", max_iter=2000),
    {"C": np.logspace(-4, 4, 20)},          # C = 1/alpha: SMALLER C = STRONGER
    cv=5, scoring="average_precision",
)

The inverted C parameter in scikit-learn’s LogisticRegression and SVC catches people out constantly: smaller C means more regularisation, opposite to alpha in Ridge and Lasso.

Interview angle

• “L1 vs L2?” — L1 gives sparsity and implicit feature selection because its gradient is constant and pushes weights to exactly zero; L2 shrinks smoothly and never quite reaches zero. L2 spreads weight across correlated features; L1 picks one arbitrarily. Elastic Net if you want both.
• “Why does L1 produce exact zeros and L2 doesn’t?” — the gradient of |w| is ±alpha regardless of magnitude, so it keeps pushing all the way to zero. The gradient of w^2 is 2*alpha*w, which vanishes as w does.
• “Do you need to scale features before regularising?” — yes. The penalty is on coefficient magnitude, which depends on feature units, so unscaled features get penalised inconsistently.
• “What does dropout do, and what’s the classic bug?” — randomly zeroes activations during training to prevent co-adaptation, approximating an ensemble of sub-networks. The bug is forgetting model.eval(), leaving dropout active at inference.
• “Why do transformers use LayerNorm rather than BatchNorm?” — LayerNorm normalises within a single sample, so it doesn’t depend on batch size or composition. That matters for variable-length sequences and small per-device batches, and it behaves identically in training and inference.
• “Cheapest regularisation available?” — early stopping. You already have a validation set, and it costs nothing beyond monitoring.
• “Validation loss starts rising while training loss falls. What is that?” — the definition of overfitting. Check for leakage first, then early-stop, regularise, get more data, or reduce capacity.