Jaeger

Open-source distributed tracing system. Originally built at Uber (2015), donated to CNCF (graduated 2019). The reference OSS tracing backend that pairs with OpenTelemetry.

For OpenTelemetry (the modern instrumentation standard that feeds Jaeger), see 07_opentelemetry.md. For the three-pillars context, see 08_three_pillars.md.

What problem tracing solves

Logs and metrics tell you something is wrong. Traces tell you where in a distributed system.

Request /api/checkout takes 4 seconds. Why?

Without tracing: grep logs across 5 services, correlate by request ID, build mental model.
With tracing:    open the trace ID, see a flame graph:

  /api/checkout                                    [████████████████████ 4000ms]
    ├─ auth.verify                                 [█ 50ms]
    ├─ inventory.check                             [██ 200ms]
    ├─ payment.charge                              [█████████████████ 3500ms]  ← the culprit
    │   ├─ payment.bank_api_call                   [████████████████ 3400ms]
    │   └─ payment.fraud_check                     [█ 80ms]
    └─ notification.send_email                     [█ 100ms]

A trace is a tree of spans representing operations. Each span has start/end timestamps, name, tags, optional logs/events. Spans link to a parent; together they form the trace.

Spans, traces, context propagation

Propagation: when service A calls service B, A injects the trace context into the request (HTTP headers, message metadata); B extracts it and creates child spans under that trace.

W3C Trace Context is the standard format:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^ ^^
             version   trace ID                    parent span ID   flags

Older format (B3 from Zipkin/Jaeger):

X-B3-TraceId: 4bf92f3577b34da6
X-B3-SpanId: 00f067aa0ba902b7
X-B3-Sampled: 1

Modern apps use W3C. Jaeger and OpenTelemetry both support it.

Jaeger architecture

                        ┌──────────────────┐
Apps ──spans──>  Agent ──> Collector ──> Storage ──> Query ──> UI
                          (validates,     (Cassandra, (search,    (flame graphs,
                           samples,        Elasticsearch, dependency  service maps)
                           batches)        OpenSearch)    graphs)

Components:

In Jaeger 2.x, “OpenTelemetry Collector” replaces the Jaeger Agent. Apps send OTLP (OpenTelemetry Protocol) to the collector, which writes to Jaeger storage backends. Architecturally simpler.

Sampling — you can’t trace everything

A million requests/day × hundreds of spans each = lots of storage. Sampling reduces volume.

Head-based (constant/probabilistic) is decided at trace start. Easy but loses interesting traces. Tail-based requires buffering full traces — more storage cost upfront, but better signal-to-noise.

Jaeger supports head-based by default. Tail-based requires an OpenTelemetry Collector with the tail-sampling processor.

Recommended starter: 1% baseline + 100% for errors + 100% for slow (>p99 latency). Captures interesting things at low cost.

Querying traces

Jaeger UI lets you search by:

• Service + operation name.
• Tags (http.status_code=500, user_id=42).
• Duration range (find slow traces).
• Time range.

Results show matching traces; click one to see the flame graph + span details.

Programmatic: query API (HTTP) returns JSON.

GET /api/traces?service=checkout&tags={"error":"true"}&lookback=1h

For ad-hoc analysis, the API + a Python script beats clicking around.

Service map / dependency graph

Auto-generated graph of services calling services, derived from trace data.

[ web ] ──> [ api ] ──> [ payments ] ──> [ stripe ]
                  ├──> [ inventory ] ──> [ db ]
                  └──> [ notifications ] ──> [ email ]

Useful for:

• “What does this service depend on?”
• “Who calls this service?”
• “Is there a circular dependency?”
• “What’s the error rate from A → B?”

In Jaeger, the dependency graph is computed periodically from stored traces. For real-time service maps, Datadog and others do better.

Python instrumentation

Two paths: native Jaeger SDK (legacy) or OpenTelemetry → Jaeger (modern).

OpenTelemetry → Jaeger (recommended)

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

resource = Resource(attributes={"service.name": "my-service"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://jaeger:4317", insecure=True)))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

@tracer.start_as_current_span("process_order")
def process_order(order):
    with tracer.start_as_current_span("validate"):
        validate(order)
    with tracer.start_as_current_span("charge"):
        charge(order)

Jaeger 1.35+ supports OTLP directly. Earlier versions need the OpenTelemetry Collector as an intermediary.

For auto-instrumentation:

pip install opentelemetry-distro opentelemetry-instrumentation-flask opentelemetry-instrumentation-sqlalchemy
opentelemetry-bootstrap -a install                  # finds installed libs, installs matching instrumentation
opentelemetry-instrument python app.py              # runs with auto-instrumentation

Most popular libraries (Flask, Django, FastAPI, requests, SQLAlchemy, psycopg, Redis, Celery, …) have instrumentation packages — auto-generated spans without code changes.

Jaeger native client (legacy)

jaeger-client was the old way. Deprecated as of ~2022. Don’t use for new code; migrate to OpenTelemetry.

Trace context propagation in code

For non-HTTP boundaries (queues, custom RPC), manual propagation:

from opentelemetry import trace
from opentelemetry.propagate import inject, extract

# Producer side (sending to Celery)
carrier = {}
inject(carrier)        # injects trace headers into the dict
celery_app.send_task("process", kwargs={"order_id": 42, "_trace_carrier": carrier})

# Consumer side (Celery worker)
def process(order_id, _trace_carrier=None):
    ctx = extract(_trace_carrier or {})
    with tracer.start_as_current_span("process_task", context=ctx):
        ...

For supported queues (Celery, Kafka, RabbitMQ), use the OpenTelemetry instrumentation libraries — they handle this automatically.

Running Jaeger locally

docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \   # UI
  -p 4317:4317 \      # OTLP gRPC
  -p 4318:4318 \      # OTLP HTTP
  jaegertracing/all-in-one:latest

The all-in-one image runs collector + query + UI + in-memory storage in one process. For production, run components separately with Cassandra / Elasticsearch backing.

UI at http://localhost:16686. Point OpenTelemetry exporter at http://localhost:4317 (gRPC) or http://localhost:4318 (HTTP/JSON).

Production storage choices

Trace data is high-volume and write-heavy. Storage is the bottleneck. Plan retention (7-30 days typical) and storage scaling accordingly.

Jaeger vs alternatives

Zipkin is older, simpler, less featured. Tempo (Grafana Labs) is newer; uses object storage for low cost. Datadog APM is the commercial answer if you’ll pay.

For OSS, Jaeger is the default if you want a polished UI with service maps. Tempo if you want cheap storage and have Grafana.

Common pitfalls

• Sampling too aggressively — 1% sample on a 1-error-per-1000-requests bug means you might never see it. Increase sampling for errors specifically.
• No trace context across async boundaries — Celery task started without injecting trace context, so it appears as orphan trace. Use OpenTelemetry’s queue instrumentation.
• High-cardinality tags — putting user_id on every span explodes storage. Better as a span attribute (queryable) than as a separate label.
• Long-running spans — a 1-hour batch job span keeps the trace open for an hour. Either split into many shorter spans or use span events for milestones.
• Trace IDs not in logs — can’t click from a slow trace to its logs. Include trace_id in log records (auto-instrumented in most OpenTelemetry library integrations).
• Production with all-in-one — uses in-memory storage; restart loses data. Run collector + query + storage separately.

Common interview confusions

• “Jaeger and OpenTelemetry are competitors.” — OpenTelemetry is the instrumentation standard (data producers). Jaeger is a backend (data consumer). They cooperate: OTel → Jaeger is the modern pattern.
• “Traces replace logs.” — different roles. Traces show the request path; logs are the event narrative. Best used together (with trace IDs in logs).
• “You should trace 100% of requests.” — too expensive at scale. Sample probabilistically; bias toward errors and slow paths.

Interview angle

• “What is distributed tracing?” — technique for tracking requests across service boundaries. A trace is a tree of spans; each span is one operation. Trace context propagates between services via headers. Lets you see “where did the time go in this 4-second request.”
• “What’s Jaeger?” — open-source distributed tracing system (CNCF graduated). Components: collector (ingests spans), storage (Cassandra/ES), query API, UI. Pairs with OpenTelemetry as the instrumentation source.
• “What’s a span?” — one operation within a trace. Has start/end timestamps, name, tags, parent span ID. Many spans form a tree representing the request flow.
• “How does trace context propagate?” — via headers (W3C traceparent is the standard; older B3 from Zipkin). Service A injects headers when calling B; B extracts and creates child spans under the same trace ID.
• “What’s sampling and why?” — keeping all traces is expensive at scale. Sample N% (head-based) or after-the-fact based on outcome (tail-based: keep errors, slow paths, drop boring). Recommended: 1% baseline + 100% on errors + 100% on slow.
• “Jaeger vs OpenTelemetry?” — not competitors. OTel is the standard for emitting telemetry from apps; Jaeger is one backend that consumes traces. Modern pattern: instrument with OTel, ship to Jaeger.
• “How would you instrument a Python service for Jaeger?” — install OpenTelemetry SDK + OTLP exporter, configure resource with service.name, use BatchSpanProcessor with OTLPSpanExporter pointing at Jaeger. Auto-instrument popular libraries via opentelemetry-instrument CLI.