Linear regression

The simplest useful model and a standard interview warm-up. The questions are rarely “can you fit one” — they’re about assumptions, when it breaks, and why you’d still choose it in 2026.

The model

y_hat = w0 + w1*x1 + w2*x2 + ... + wn*xn

Fit by minimising squared error. There’s a closed-form solution — the normal equation:

w = np.linalg.inv(X.T @ X) @ X.T @ y     # don't actually write this
w = np.linalg.lstsq(X, y, rcond=None)[0] # use this: stable, handles singular X

Never invert X.T @ X explicitly. It’s O(n^3), numerically unstable, and fails outright when features are collinear (the matrix is singular). lstsq uses SVD and degrades gracefully. For large n, use gradient descent or SGDRegressor instead.

from sklearn.linear_model import LinearRegression, Ridge

model = Ridge(alpha=1.0).fit(X_train, y_train)   # Ridge > plain LR by default

Default to Ridge over plain LinearRegression. A small L2 penalty costs almost nothing in bias and buys numerical stability plus resistance to collinearity.

Assumptions, and what violating each does

The plot that answers most of these at once:

residuals = y_val - model.predict(X_val)
plt.scatter(model.predict(X_val), residuals)   # should be a shapeless cloud around 0

A pattern in that plot means your model is missing structure. A fan shape means heteroscedasticity — often fixed by log-transforming the target.

Normality is the assumption people over-weight. It’s not needed for the coefficients to be unbiased or for prediction to work. It matters when you want confidence intervals and p-values on the coefficients.

Multicollinearity

Correlated features don’t hurt predictions much, but they make coefficients meaningless and unstable — small data changes flip signs and magnitudes.

from statsmodels.stats.outliers_influence import variance_inflation_factor

vif = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
# VIF > 5-10 signals a problem

Fixes: drop one of the pair, combine them, use PCA, or apply Ridge — which handles collinearity by spreading weight across correlated features rather than picking arbitrarily.

This matters enormously if anyone will interpret the coefficients, and barely at all if you only care about predictions.

Feature scaling

Plain linear regression doesn’t need it — the closed form is scale-invariant. But you need it for:

• gradient descent (unscaled features make the loss surface elongated and convergence slow),
• any regularisation, since the penalty is on coefficient magnitude,
• comparing coefficient sizes to judge feature importance.

Common extensions

Polynomial features capture curvature while staying a linear model in the parameters:

from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

model = make_pipeline(
    PolynomialFeatures(degree=2),
    StandardScaler(),
    Ridge(alpha=1.0),
)

Degree grows features combinatorially and overfits fast. Degree 2 or 3 with regularisation, and check the learning curve.

Log-transforming the target when it’s right-skewed (prices, counts, durations). It converts multiplicative relationships to additive ones and often fixes heteroscedasticity at the same time. Remember to invert with np.expm1 when predicting, and note that the back-transformed prediction estimates the median, not the mean.

Huber loss when outliers are dragging the fit:

from sklearn.linear_model import HuberRegressor    # quadratic near 0, linear in the tails

Metrics

R² is the most misread of these. It is not “accuracy”. A high R² on a trending time series can come entirely from the trend, and a negative R² simply means you’re worse than predicting the mean.

Why it’s still worth knowing

In an era of gradient boosting and transformers, linear regression survives because:

• Interpretable by construction. Each coefficient is “one unit of this feature moves the prediction by that much, holding others fixed.” Regulators accept it.
• Fast and tiny. Microsecond inference, kilobytes of model.
• A real baseline. If your deep model beats Ridge by 1%, that’s important information.
• Extrapolates. Tree models cannot predict outside the range of their training targets; linear models can — sometimes wrongly, but they can.

That last point is a genuinely good interview answer to “when would you prefer a linear model over gradient boosting”.

Interview angle

• “What are the assumptions of linear regression?” — linearity in parameters, independent errors, constant error variance, approximately normal residuals (only for inference), and no perfect multicollinearity. Say which ones matter for prediction versus for inference — that distinction is the real answer.
• “Does multicollinearity hurt predictions?” — not much. It makes individual coefficients unstable and uninterpretable. If you only need predictions, Ridge handles it; if you need to explain coefficients, you must resolve it.
• “Why not use the normal equation?” — O(n^3), numerically unstable, and undefined when X.T @ X is singular. Use lstsq (SVD-based) or gradient descent at scale.
• “When would you pick linear regression over XGBoost?” — when interpretability is required, when you need to extrapolate beyond the training range, when data is small, when latency is extreme, or as the baseline that tells you whether complexity is earning its keep.
• “Residual plot shows a fan shape. What is it and what do you do?” — heteroscedasticity. Coefficients remain unbiased but inference is invalid. Log-transform the target, use weighted least squares, or use robust standard errors.
• “R² is 0.95. Good model?” — not necessarily. Check it on held-out data, check for a trend inflating it, and check residuals for structure. R² isn’t comparable across datasets and says nothing about whether errors are acceptably sized.