Distributed Tracing

In a single service, a stack trace tells you what happened. In ten services, you need a trace — a tree of spans across processes, tied together by a shared trace ID.

The vocabulary

• Trace — one logical operation end-to-end (e.g., “user places an order”).
• Span — one unit of work inside a trace (e.g., “POST /orders”, “SELECT users”, “publish OrderPlaced”).
• Trace ID — UUID/hex string identifying the whole trace; constant across all spans.
• Span ID — unique to each span; spans have a parent_span_id to form a tree.
• Context propagation — passing trace ID + parent span ID across process boundaries (HTTP headers, message metadata).

The de facto standard: OpenTelemetry (OTel)

W3C traceparent header carries context across HTTP. SDKs for every major language. Backends: Jaeger, Zipkin, Tempo, Honeycomb, Datadog APM, AWS X-Ray.

traceparent: 00-{trace_id}-{span_id}-{flags}
            version  16 bytes hex  8 bytes hex  01=sampled

Python wiring (FastAPI + OTel)

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317"))
)

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)
HTTPXClientInstrumentor().instrument()
SQLAlchemyInstrumentor().instrument(engine=engine)

Auto-instrumentation injects spans for incoming requests, outgoing HTTP, DB queries, queue ops. Manual spans for business logic:

tracer = trace.get_tracer(__name__)

@app.post("/orders")
async def place_order(order: Order):
    with tracer.start_as_current_span("validate_order") as span:
        span.set_attribute("order.user_id", order.user_id)
        span.set_attribute("order.total", order.total)
        validate(order)
    await persist(order)
    await publish_event(order)

Context propagation across boundaries

HTTP — automatic with OTel

OTel HTTPX/requests instrumentations inject traceparent outbound and parse it inbound. You typically don’t write the code.

Celery — needs explicit handling

Celery doesn’t propagate OTel context automatically; it propagates its own task context. The OTel Celery instrumentation does the bridge:

from opentelemetry.instrumentation.celery import CeleryInstrumentor
CeleryInstrumentor().instrument()

What it actually does: serializes the OTel context into the Celery message headers, restores it in the worker before the task runs. Without this, your traces dead-end at task.delay().

Manually, when no instrumentation exists

from opentelemetry.propagate import inject, extract

# Producer
headers = {}
inject(headers)                          # adds traceparent to headers
queue.publish(message, headers=headers)

# Consumer
ctx = extract(message.headers)
with tracer.start_as_current_span("process", context=ctx):
    handle(message)

Async Python gotcha: contextvars

OTel uses contextvars to carry span context across await. This works within one event loop. It breaks if:

• You hand work to loop.run_in_executor without copying context (use contextvars.copy_context().run(...) or rely on the instrumentation).
• You spawn a thread with threading.Thread — context isn’t inherited; pass it explicitly.
• You bridge into a non-asyncio framework (gevent, twisted) without an adapter.

For Celery’s prefork worker, the OTel instrumentation rebuilds the context from the message; for eventlet/gevent pool, monkey-patching ordering matters.

Sampling

You can’t store every trace at scale. Two approaches:

Common config: head-based at 100% in dev, 1-10% in prod, with always_on for traces containing errors via the OTel ParentBased(TraceIdRatioBased) sampler.

What to actually trace

Worth a span:

• Every external network call (HTTP, DB, queue, cache, S3).
• High-value business steps (charge_card, reserve_inventory).
• Long-running loops (set attribute iterations, not a span per iteration).

Not worth a span:

• Tight in-memory loops.
• Pure CPU functions.
• Per-row of a query (auto-instrumentation already covers the query as one span).

Useful span attributes: user.id, tenant.id, order.id, db.statement (parameterized), http.status_code, messaging.message_id. Avoid PII unless you scrub later.

Backends

OTel exports to all of them via OTLP, so swapping backends is config not code.

Correlation IDs in logs

Trace ID should appear in every log line so you can pivot from “this error” to “the whole trace”.

import structlog, logging
from opentelemetry import trace

def inject_trace_context(logger, method_name, event_dict):
    span = trace.get_current_span()
    if span and span.get_span_context().is_valid:
        ctx = span.get_span_context()
        event_dict["trace_id"] = f"{ctx.trace_id:032x}"
        event_dict["span_id"] = f"{ctx.span_id:016x}"
    return event_dict

structlog.configure(processors=[
    inject_trace_context,
    structlog.processors.JSONRenderer(),
])

Interview angle

• “What’s a span, what’s a trace?” — span = one operation; trace = tree of spans sharing a trace ID; parent-child links form the tree.
• “How does the trace ID get from service A to service B over HTTP?” — W3C traceparent header (and optionally tracestate); the OTel HTTP instrumentation injects on the way out and extracts on the way in.
• “How do you propagate a trace into a Celery task?” — Celery doesn’t do it natively; use the OTel Celery instrumentation, which serializes context into the task message and restores it in the worker.
• “What goes wrong with loop.run_in_executor and tracing?” — contextvars don’t transfer to threads automatically; use contextvars.copy_context().run(fn) or let the OTel instrumentation handle it.
• “Head vs tail sampling?” — head: decide at entry, simple but can drop interesting traces. Tail: decide at collector after seeing the whole trace, lets you keep errors/slow ones. Most production: hybrid — always keep errors, sample success.