Portfolio optimization

Turning a set of return forecasts into position sizes. The textbook answer is mean-variance; the interview is about why the textbook answer is unstable and what you do instead.

Mean-variance (Markowitz)

Maximise expected return for a given variance, or equivalently minimise variance subject to a return target:

minimize    w' Σ w
subject to  w' μ  >= r_target
            sum(w) = 1,  w >= 0   (long-only)

μ is the vector of expected returns, Σ the covariance matrix, w the weights. Sweeping the return target traces the efficient frontier; adding a risk-free asset gives the tangency portfolio with the highest Sharpe.

Why it fails in practice. The optimizer is extremely sensitive to μ. Expected returns are estimated with huge error, and the optimizer treats a slightly higher estimate as a certainty — so it concentrates into a few assets and the weights swing wildly between rebalances. It has been called an “error maximiser” for exactly this reason: it allocates most to whatever has the largest estimation error in its favour.

Covariance is better estimated than returns but still ill-conditioned: with N assets you estimate N(N+1)/2 parameters, so with 500 assets and two years of daily data the sample covariance matrix is close to singular.

What you actually do

Shrink the covariance matrix. Ledoit-Wolf shrinkage pulls the sample covariance toward a structured target, trading a little bias for a large variance reduction. It is one line in scikit-learn and it materially stabilises the result.

Constrain the weights. Position caps, sector caps, turnover limits, and a minimum position size. Constraints are not a hack around a bad model — they encode the mandate, and they bound the damage from estimation error.

Drop the return estimates. If μ is the unstable input, use methods that do not need it:

Equal weight is the baseline any optimizer must beat net of turnover. Saying that unprompted is a strong signal.

Blend views with the market. Black-Litterman starts from the market-implied equilibrium returns and tilts toward your views in proportion to your confidence. It produces far more stable weights than feeding raw forecasts into a mean-variance optimizer.

In Python

import numpy as np, cvxpy as cp
from sklearn.covariance import LedoitWolf

Sigma = LedoitWolf().fit(returns).covariance_
n = Sigma.shape[0]
w = cp.Variable(n)

problem = cp.Problem(
    cp.Minimize(cp.quad_form(w, cp.psd_wrap(Sigma))),
    [cp.sum(w) == 1, w >= 0, w <= 0.10],          # long-only, 10% position cap
)
problem.solve()

cvxpy for convex formulations, PyPortfolioOpt for a higher-level API over the standard recipes, riskfolio-lib for a wider set of risk measures (CVaR, CDaR) and HRP. scipy.optimize works but you lose the guarantee that a convex solver gives you.

Two practical notes: wrap the covariance so the solver knows it is PSD (numerical noise in an estimated matrix can make it fail otherwise), and add a turnover penalty cp.norm1(w - w_prev) when rebalancing, or the optimizer will churn the book for a basis point of theoretical improvement.

Risk measures beyond variance

Variance penalises upside and downside equally, which is not how anyone experiences risk.

• Semi-variance / downside deviation — only deviations below a threshold. Sortino is the corresponding ratio.
• VaR — the loss not exceeded with probability p. Widely used and not sub-additive, which means it can say a diversified portfolio is riskier than its parts.
• CVaR (Expected Shortfall) — the mean loss beyond VaR. Coherent, optimisable as a linear program, and the regulatory direction of travel.
• Max drawdown — the number that actually gets strategies switched off.

Interview angle

• “Explain mean-variance optimization and its main weakness.” - maximise return per unit of variance given μ and Σ. The weakness is extreme sensitivity to μ: expected returns are badly estimated, and the optimizer treats estimation error as signal, producing concentrated portfolios that swing between rebalances.
• “How do you make it stable?” - shrink the covariance (Ledoit-Wolf), constrain weights and turnover, and prefer methods that do not need return forecasts - minimum variance, risk parity, HRP. Black-Litterman if you genuinely have views.
• “Why is 1/N a serious benchmark?” - it has no estimation error at all. Out of sample, and after turnover costs, optimizers frequently fail to beat it. Any optimizer you propose should be measured against it.
• “VaR or CVaR?” - CVaR. It is coherent (sub-additive, so diversification cannot look like it increases risk), it captures the shape of the tail rather than one quantile, and it can be optimised as a linear program.
• “How do you stop the optimizer churning the portfolio?” - an explicit turnover penalty or constraint in the objective, and rebalancing on a band rather than a schedule - only trade when weights drift beyond a threshold.