Backtesting

Simulating a strategy against historical data. The engineering question is not “does it produce a number” but “is that number believable” — and every design decision below either preserves or destroys that.

Vectorized versus event-driven

The standard workflow uses both: vectorized to explore the parameter landscape cheaply, event-driven to confirm that the survivors still work when fills, costs and capital constraints are modelled honestly. A result that only survives the vectorized pass is not a result.

The off-by-one that invalidates everything

# WRONG - trades on the bar that produced the signal
returns = signal * price.pct_change()

# RIGHT - signal computed on bar t, position held from t+1
returns = signal.shift(1) * price.pct_change()

That single shift(1) is the most common look-ahead bug in the field, and interviewers ask about it directly. If your signal uses the close, you cannot trade at that close.

The same class of error hides in rolling statistics that are centred rather than trailing, in fillna methods that back-fill, and in any normalisation computed over the whole series (a z-score against the full-history mean uses the future).

Costs

A gross backtest is a marketing document. Model, at minimum:

• Commission — per share, per contract, or basis points, whichever the venue charges.
• Spread — buy at the ask, sell at the bid. On a strategy that turns over daily, the spread alone can exceed the edge.
• Slippage — the difference between the decision price and the fill. A fixed number of basis points is the crude model; a square-root impact function scaled by participation rate is the standard one.
• Borrow cost and short availability — a short-side backtest that assumes everything is always shortable is fiction for small caps.
• Financing — for leveraged or futures positions.

State your turnover. A strategy with a 2.0 Sharpe gross and 300% annual turnover is usually a losing strategy net of costs, and that arithmetic is a common follow-up.

Validating that the result generalises

Train/test split alone is not enough for time series — and standard k-fold cross-validation is actively wrong, because shuffling puts future data in the training set.

• Walk-forward analysis: fit on a window, test on the next, roll forward, concatenate the out-of-sample results. This is the credible baseline.
• Purging and embargo: when labels span multiple bars, drop training samples that overlap the test window and leave a gap after it, or information leaks across the boundary.
• Combinatorial purged CV: multiple train/test path combinations, giving a distribution of outcomes rather than a single number.
• Deflated Sharpe: adjust the reported Sharpe for how many variants you tried. Testing 500 parameter sets and reporting the best is a maximum, not an expectation.

The honest framing: out-of-sample stops being out-of-sample the second time you look at it. Keep a holdout you touch once.

What to report

Not just the equity curve:

Drawdown duration is the one people skip and the one that ends strategies. A 20% drawdown lasting three years gets turned off by the humans running it long before it recovers.

Interview angle

• “Walk me through how you would backtest a strategy.” - define the universe point-in-time, generate signals from data available at decision time, shift the position by one bar, apply commission, spread and slippage, then evaluate walk-forward rather than on a single split. Report drawdown and turnover alongside the Sharpe.
• “Your backtest shows a Sharpe of 3. What do you check?” - look-ahead first (does the signal use the bar it trades on, does any normalisation use the full history), then survivorship in the universe, then whether costs were modelled, then how many variants were tried. A Sharpe of 3 on daily data is more often a bug than an edge.
• “Why can’t you use k-fold cross-validation here?” - it shuffles, so the model trains on the future and tests on the past. Time series needs walk-forward, with purging and an embargo when labels overlap.
• “Vectorized or event-driven?” - vectorized to search the parameter space cheaply, event-driven to validate the survivors with realistic fills and capital constraints. Reporting a vectorized result as final is the mistake.
• “How do you model slippage?” - a fixed basis-point haircut as a floor, and a participation-rate-based impact model when order size is material relative to volume. Then run the backtest at 2x your slippage assumption; if the edge disappears, it was never robust.