Profiling Python — cProfile, pstats, timeit

Profile before optimizing. Most Python “slowness” lives in unexpected places — database queries, JSON parsing, redundant computations — not where intuition says.

timeit — micro-benchmarks

For comparing small code snippets:

import timeit

# Compare list comprehension vs map+lambda:
timeit.timeit("[x*2 for x in range(100)]", number=100_000)
# 0.62

timeit.timeit("list(map(lambda x: x*2, range(100)))", number=100_000)
# 1.04

In a script:

from timeit import timeit

setup = "data = list(range(1000))"
print(timeit("sorted(data)", setup=setup, number=10_000))

CLI:

python -m timeit -s "data = list(range(1000))" "sorted(data)"

timeit runs your code N times and reports total. It disables GC during the run for stable numbers — re-enable manually if testing GC-affected code.

Pitfall: cold caches. First runs are slower than steady-state. Use repeat() and take min:

timeit.repeat("expr", number=1000, repeat=5)   # 5 trials of 1000 runs each
# Take min — others may have been disturbed by other processes

cProfile — function-level profiler

For finding which function eats the time in a real workload:

import cProfile

cProfile.run("main()", sort="cumulative")

Or to file:

python -m cProfile -o profile.out my_script.py

Then analyze:

import pstats

p = pstats.Stats("profile.out")
p.strip_dirs().sort_stats("cumulative").print_stats(20)

Output columns:

• ncalls: number of calls
• tottime: total time in this function, excluding sub-calls
• percall: tottime / ncalls
• cumtime: total time including sub-calls
• filename:lineno(function)

Look at cumtime for callers and tottime for hot leaves.

snakeviz — flamegraph viewer for cProfile output

pip install snakeviz
python -m cProfile -o profile.out my_script.py
snakeviz profile.out

Interactive flamegraph in browser. Easier to read than the raw text for deep call trees.

py-spy — sampling profiler, no code changes

cProfile instruments every function call (slows your code 2-5x). py-spy samples the running process via /proc — minimal overhead, can attach to a running server:

pip install py-spy
py-spy record -o flame.svg --pid 12345
py-spy top --pid 12345    # like `top` for Python functions

This is what you reach for in production when you can’t add cProfile.run around the call.

What to look for

• Few functions with high cumtime — usually fixable by caching, batching, or algorithmic change.
• Lots of small calls in tight loops — function call overhead. Inline, vectorize, or use C extension.
• GIL-bound CPU work — consider multiprocessing, NumPy, or native libs.
• Lots of socket.recv or DB driver time — I/O-bound; see 02_async_optimization.md.
• gc.collect appearing in profile → too many container allocations.

Practical workflow

1. Suspect something is slow.
2. Reproduce locally with realistic input.
3. Profile with cProfile (or py-spy for live).
4. Sort by cumtime, find hot path.
5. Drill into hot leaves with tottime.
6. Make a focused change.
7. Re-profile to verify.

Don’t optimize without profiles. Don’t optimize what isn’t hot.

Interview angle

• “How would you find the bottleneck in a slow service?” → cProfile for offline; py-spy to attach to running process.
• “Difference between tottime and cumtime?” (tot = self only, cum = includes called functions.)
• “When would you choose timeit over cProfile?” (timeit for comparing two equivalent micro-implementations; cProfile for finding what to optimize in a larger workload.)