Market data
The data layer is where most quant interviews actually go. Anyone can call a backtest library; far fewer can explain why their results are optimistic because of how the data was assembled.
Shapes of market data
| Shape | What it is | Where it hurts |
|---|---|---|
| Tick | every trade and quote, timestamped to microseconds | enormous volume; needs columnar storage and careful compression |
| Quote (L1) | best bid and ask | the only honest basis for a fill price; last-trade price is not what you would have paid |
| Order book (L2/L3) | depth per price level, or per order | the input to market-impact and microstructure work; huge and vendor-specific |
| Bar / OHLCV | aggregated open, high, low, close, volume over an interval | most strategy research; hides intra-bar path, which is where fills actually happen |
The interview point on bars: a bar tells you the high and the low but not their order. A backtest that assumes a stop was hit before the target, or the reverse, is guessing. Either model it conservatively (assume the worse of the two) or use tick data for the entry logic.
Asset-class differences that change your code
| Asset class | What is different |
|---|---|
| Equities | corporate actions (splits, dividends), exchange holidays, symbol changes, delistings, multiple venues for the same name |
| Futures | contracts expire, so a continuous series must be stitched; roll method changes your returns materially |
| Forex | no central exchange, so “the” price depends on the venue; near-24/5 with a session boundary, not a daily close |
| Crypto | 24/7 with no session boundary, per-exchange price differences, wildly varying liquidity, exchanges that vanish |
Futures continuation deserves its own answer. When a contract expires you splice the next one in, and the price gap between them is not a return. Back-adjusting (shifting the historical series by the gap) preserves the return series and can produce negative historical prices; ratio-adjusting preserves ratios and distorts absolute levels. Whichever you choose, say that the choice affects the backtest and that you keep the unadjusted series alongside.
The biases that make a backtest lie
These are the questions. Being able to name and mitigate each one is the difference between a junior and a senior answer.
Survivorship bias. A universe built from today’s index constituents excludes everything that was delisted, acquired or went to zero. Every strategy looks better on the survivors. Fix: point-in-time index membership, and a database that keeps dead symbols.
Look-ahead bias. Using information not available at the decision time. The subtle cases: fundamentals timestamped by period rather than by filing date, restated data overwriting the original, and a “close” price used for a signal that also trades at that close.
Point-in-time correctness. Related and broader: the database must be able to answer “what did we know on this date”, not just “what is true now”. Bitemporal storage (an event time and a knowledge time) is the general answer.
Data snooping. Testing hundreds of variants against one dataset and reporting the best. The reported Sharpe is then a maximum over many draws, not an estimate. See 05_ml_for_finance.md.
Storage and access
For anything beyond daily bars, the pattern is columnar files plus a query engine rather than rows in a relational database:
- Parquet partitioned by symbol and date, on object storage. See ../29_data_engineering/04_parquet/01_parquet_fundamentals.md.
- Polars or DuckDB to query it — lazy, multi-threaded, out-of-core. Pandas will hit memory limits on a few years of minute bars across a wide universe. See 07_python_performance_for_quant.md.
- PostgreSQL for reference data (instruments, corporate actions, calendars) and for anything transactional. TimescaleDB adds hypertables and compression if you want time series in Postgres specifically.
Always store timestamps in UTC with an explicit timezone, and keep the exchange calendar separate. Half the reconciliation bugs in this domain are a naive local timestamp meeting daylight saving.
Interview angle
- “Where does market data come from and what do you check on arrival?” - a vendor or exchange feed, and you validate before storing: gaps against the exchange calendar, duplicate timestamps, zero or negative prices, and volumes or ranges outside a sane band. Bad data silently poisons every downstream result, so validation belongs at ingest, not in the strategy.
- “How do you build a continuous futures series?” - splice at roll and adjust for the gap. Back-adjust to preserve returns, ratio-adjust to preserve proportions, and state that the roll rule (by expiry, by open interest, by volume) is itself a research choice that changes the results.
- “What is survivorship bias and how do you eliminate it?” - a universe built from current constituents omits everything that failed. Use point-in-time index membership and keep delisted symbols with their final prices, including the ones that went to zero.
- “Why is last-trade price the wrong fill assumption?” - you buy at the ask and sell at the bid, and both move against a large order. Backtesting at the last trade quietly hands you the spread on every round trip, which is often the whole edge.
- “Bars or ticks?” - bars for research over a broad universe, ticks when execution detail matters: intrabar stops, market impact, or anything holding for minutes. Say what you lose with bars rather than defaulting to them.