Python performance for quantitative work

Where the Python-specific engineering questions land in a quant interview: the data does not fit, the loop is too slow, and the numbers do not reconcile.

The vectorization ladder

Reach down it only as far as the problem requires.

A Python loop over a few million rows costs seconds; the same work as an array operation costs milliseconds. The reason is in ../02_python_core/performance/03_why_python_is_slow.md — per-element interpreter dispatch and boxed objects.

The important caveat: not everything vectorizes. A trailing stop depends on the running maximum since entry, which depends on whether you were still in the position — genuinely sequential. Do not contort it into a chain of shift/cumsum tricks nobody can read; write the loop and put @njit on it.

from numba import njit

@njit(cache=True)
def trailing_stop(prices, entry_idx, trail_pct):
    peak = prices[entry_idx]
    for i in range(entry_idx, len(prices)):
        if prices[i] > peak:
            peak = prices[i]
        if prices[i] < peak * (1 - trail_pct):
            return i
    return -1

Pandas or Polars

For a few years of daily bars, Pandas is fine and its ecosystem wins. For minute or tick data across a wide universe, Pandas hits memory limits and Polars is the answer — build a LazyFrame from a partitioned Parquet dataset, apply filters and aggregations, and let the optimizer push predicates and column selection down to the file scan. See ../29_data_engineering/03_polars/01_polars_fundamentals.md and ../29_data_engineering/02_pandas/03_when_pandas_breaks.md.

import polars as pl

lf = (
    pl.scan_parquet("s3://mkt/bars/symbol=*/date=*/*.parquet")   # lazy, no read yet
      .filter(pl.col("date").is_between(start, end))
      .select(["ts", "symbol", "close", "volume"])
      .with_columns(pl.col("close").pct_change().over("symbol").alias("ret"))
)
df = lf.collect(streaming=True)

.over("symbol") is the piece worth knowing: window functions partitioned by symbol, which is how you avoid a groupby-apply that materialises one frame per instrument.

Storage layout

Partition by the dimensions you filter on — usually symbol and date. A well-partitioned Parquet dataset lets the engine skip entire files, which beats any amount of in-process optimisation. Column pruning does the rest: reading 3 of 40 columns reads 3 columns’ worth of bytes.

Use appropriate dtypes. Prices as float32 halve the memory versus float64, and for most research that precision is sufficient — but not for accounting or position reconciliation, where you use integers of the smallest unit or Decimal. Floating point money is a correctness bug, not a performance trade-off.

Categorical or dictionary-encoded symbol columns save a great deal on wide datasets, since a repeated string per row is the single largest waste in naive market-data storage.

Parallelism

CPU-bound backtests over independent parameter sets or symbols are embarrassingly parallel — a ProcessPoolExecutor over the grid, not threads, because of the GIL. Watch the pickling cost: sending a large DataFrame to each worker can cost more than the computation. Pass a file path and let each worker read its own slice, or use shared memory.

Python 3.14’s free-threaded build changes this story for threads, and NumPy and Polars already release the GIL around native work. See ../04_async_concurrency/01_gil.md.

Reproducibility

A backtest that cannot be reproduced is not evidence.

• Pin dependencies with a lockfile — uv or Poetry. See ../02_python_core/packaging/01_pip_poetry_uv.md.
• Seed every random process, and record the seed with the result.
• Version the data, not just the code. A vendor restating history silently changes yesterday’s result.
• Record the parameters, the code commit and the data snapshot alongside every run. This is the same discipline as an ML experiment tracker, and MLflow works fine for it. See ../../ai_ml/15_mlops_llmops/.

Interview angle

• “Your backtest takes 40 minutes. How do you speed it up?” - profile first. Usually it is a Python loop over rows, a groupby-apply, or repeated reads of the same data. Vectorize, switch to Polars with a lazy scan over partitioned Parquet, and parallelise across symbols or parameters with processes. Numba only for genuinely sequential logic.
• “Pandas or Polars?” - Pandas for daily-frequency research where the ecosystem matters; Polars when data exceeds comfortable memory, because of the lazy optimizer, multi-threading and streaming. They interoperate through Arrow, so it is not an all-or-nothing choice.
• “Why can’t you vectorize a trailing stop?” - it is path-dependent: each step depends on the running peak since entry, which depends on still being in the position. Write it as a loop and compile it with Numba rather than building an unreadable chain of shifts.
• “How do you store money?” - integers of the smallest unit, or Decimal. Floats accumulate error and 0.1 + 0.2 != 0.3, which shows up as a reconciliation break rather than an obvious crash.
• “What makes a backtest reproducible?” - pinned dependencies, a recorded seed, a versioned data snapshot, and the code commit stored with the result. Without the data snapshot, a vendor restatement changes your history and you cannot tell why the number moved.