backend / python core / performance / 06_production_debugging.md

Debugging a Live Production Incident

5 interview angles 7 min read source

Debugging a Live Production Incident

The earlier files in this folder profile code you can run locally. Production is different: you can’t cProfile.run() a running server, you can’t add print statements and redeploy mid-incident, and the thing is on fire now. This is the toolkit and the method for diagnosing a live Python process you can’t stop.

The method — before the tools

A senior answer leads with process, not tools:

  1. Characterize the symptom. Slow? Stuck? Leaking memory? Crashing? CPU-pegged? Each points at a different tool. “It’s slow” is not a diagnosis.
  2. Look at metrics/dashboards first. Latency percentiles, error rate, CPU, memory, DB connections, queue depth. The shape over time tells you when it started and what saturated — often that alone localizes it (a deploy, a traffic spike, a slow downstream).
  3. Form one hypothesis, then reach for the tool that confirms or kills it. Don’t randomly attach profilers.
  4. Mitigate, then diagnose. Roll back the deploy / scale out / restart the leaking pod first to stop the bleeding — then debug with the tools below, ideally on a still-affected instance you pulled out of the load balancer.

py-spy — the production profiler

py-spy is the single most important tool here. It’s a sampling profiler that attaches to a running process from the outside — no code changes, no restart, near-zero overhead, reads memory via OS APIs.

# "top" view — which functions are eating CPU, live:
py-spy top --pid 12345

# record a flamegraph over 30s of real traffic:
py-spy record -o flame.svg --pid 12345 --duration 30

# the killer feature — dump the current stack of EVERY thread, instantly:
py-spy dump --pid 12345

py-spy dump is what you reach for when a process is stuck/hung: it shows exactly what every thread is doing right now. If every worker thread is parked in socket.recv on the same DB call, you’ve found it in one command. For CPU-pegged, py-spy top shows the hot function live.

Caveats: it needs ptrace permission — in containers run with --cap-add SYS_PTRACE, in Kubernetes use an ephemeral debug container or shareProcessNamespace. Make sure py-spy is in your prod image or debug sidecar before the incident.

Memory leaks — finding what’s growing

Symptom: RSS climbs steadily, never falls, eventually OOM-kills. (This is what a soak test catches; in prod you catch it from the memory graph.)

  • tracemalloc — built in. If you can afford to enable it (it has overhead), it gives you allocation-site snapshots and diffs: 
    import tracemalloc
tracemalloc.start()
snap1 = tracemalloc.take_snapshot()
# ... let it run ...
snap2 = tracemalloc.take_snapshot()
for stat in snap2.compare_to(snap1, "lineno")[:10]:
    print(stat)            # top growth sites, by file:line
  • objgraph — what type is growing, and what’s keeping it alive: 
    import objgraph
objgraph.show_growth()                       # which types grew since last call
objgraph.show_backrefs([leaked_obj], max_depth=5)   # who references it
  • gc module — gc.get_objects(), len(gc.get_objects()) over time; check gc.garbage for uncollectable cycles.
  • Common Python “leaks” (usually not real leaks — unbounded growth): an ever-growing module-level cache/dict/list, an lru_cache with no maxsize keyed on something high-cardinality, accumulating references in a closure, a logger holding records, __del__ resurrecting objects.

The fastest triage: a memory profiler endpoint or signal handler that dumps objgraph.show_growth() to logs — flip it on, wait, read which type count is climbing.

Getting a foothold in a running process

When you need state, not just stacks:

  • signal handler for an on-demand dump — register a SIGUSR1/SIGUSR2 handler in your app that dumps thread stacks, GC stats, or objgraph growth to the log. faulthandler.register(signal.SIGUSR1) gives you thread tracebacks for free; ship it always-on.
  • faulthandler — faulthandler.enable() (or PYTHONFAULTHANDLER=1) dumps a traceback on a hard crash/segfault, and faulthandler.dump_traceback_later(timeout) dumps if the process hangs past a timeout. Essentially free; enable it in prod by default.
  • manhole / a remote REPL — opens a socket you can connect to and get a live Python prompt inside the process. Powerful, but a security surface — gate it hard.
  • pyrasite / pdb-attach — inject into a running process. Heavier and riskier than py-spy; use when you genuinely need to inspect objects, not just stacks.

Core dumps — post-mortem on a crash

When the process is already dead (segfault, OOM-kill, C-extension crash):

ulimit -c unlimited                  # enable core dumps
gdb python core.12345                # open the core
(gdb) py-bt                          # Python-level backtrace (needs python-gdb extensions)
(gdb) py-list                        # source around the crash

gdb + the CPython python-gdb.py extensions give you a Python-level stack from a C-level core dump — essential when a native extension (numpy, a database driver, lxml) is the thing crashing and the Python traceback alone is useless.

Async-specific

For an asyncio app that’s “stuck,” the question is usually “which coroutines are pending and what are they awaiting”:

  • asyncio.all_tasks() + task.get_stack() — dump every live task’s stack. Wire it to a signal handler.
  • loop.slow_callback_duration + PYTHONASYNCIODEBUG=1 — surfaces a coroutine blocking the event loop (a sync call in async code — the classic async prod bug).
  • py-spy dump still works and shows you the loop thread parked in epoll with the task machinery on the stack.

Distributed tracing — when it’s not this service

Often “our service is slow” is really “a downstream is slow.” A trace (OpenTelemetry → Jaeger/Tempo/X-Ray) breaks one request into spans across services and shows which hop ate the latency. Before deep-profiling your own process, check the trace — the bottleneck may not be in your code at all. See 15_observability/.

Common gotchas

  • Profiling locally and assuming prod matches — prod has real data volume, real concurrency, real network latency, a warm/cold cache. The bottleneck is often prod-only.
  • No py-spy in the prod image — you can’t install it mid-incident in a locked-down container. Bake it into the image or a debug sidecar ahead of time.
  • ptrace blocked in the container — py-spy/gdb need SYS_PTRACE; if it’s not granted you’re blind. Know your container’s capabilities before you need them.
  • Restarting the process before you grab a diagnostic — the restart destroys the evidence. Pull one affected instance out of rotation, keep it alive, debug that.
  • tracemalloc/instrumenting profilers always-on — they have real overhead. py-spy (sampling, external) is the always-safe choice; instrumenting tools are opt-in for a specific investigation.
  • Assuming “memory leak” means a real leak — in Python it’s almost always unbounded growth of a structure you control (cache, list, lru_cache), not a refcount bug. Look for the growing container.
  • Debugging the wrong service — check the distributed trace first; the slow hop may be downstream.

Interview angle

  • “A production service is slow right now. Walk me through what you do.” — Characterize the symptom from dashboards (latency shape, when it started, what saturated), mitigate first (roll back / scale / restart) to stop the bleeding, then on a still-affected instance pulled from the LB, attach py-spy — top for CPU-bound, dump for stuck/hung — to find the hot or blocked code. Check the distributed trace in case it’s a downstream.
  • “How do you profile a process you can’t stop or modify?” — py-spy: a sampling profiler that attaches from outside via OS memory APIs, no code changes, negligible overhead. py-spy top for live hot functions, py-spy record for a flamegraph, py-spy dump to snapshot every thread’s stack instantly.
  • “A service’s memory grows until it gets OOM-killed. How do you find the leak?” — Confirm the shape on the memory graph, then find what type is growing (objgraph.show_growth()) and what’s holding it (objgraph.show_backrefs), or diff tracemalloc snapshots for the allocation site. In Python it’s almost always unbounded growth of a structure you own — an unbounded cache, an lru_cache with no maxsize, an accumulating list — not a true leak.
  • “The process is completely hung — no logs, no response. Now what?” — py-spy dump --pid shows what every thread is doing right now; if they’re all parked on the same lock or socket.recv, that’s your answer. faulthandler (or a SIGUSR1 handler) can dump thread tracebacks from inside. If it’s already dead, a core dump + gdb with py-bt.
  • “A C extension is segfaulting in prod. The Python traceback is useless. What do you do?” — Enable core dumps (ulimit -c unlimited), open the core in gdb with CPython’s python-gdb.py extensions, and use py-bt to get a Python-level backtrace from the C-level crash — that bridges the native crash back to the Python line that triggered it.

Cross-links:

Previous
Async optimization — getting the most out of asyncio