Observability in practice

Monitoring answers “is it broken”. Observability answers “why” — including for failures nobody predicted. The distinction matters because you can’t add a dashboard for a question you haven’t thought of yet.

Structured logs, always

logger.info(
    "order.created",                          # a stable EVENT NAME, not a sentence
    extra={
        "order_id": str(order.id),
        "user_id": str(user.id),
        "amount_cents": order.total_cents,
        "source": "api",
        "trace_id": trace_id,
    },
)

Not logger.info(f"Created order {order.id} for {user.id}"). The formatted string is unqueryable — you can’t aggregate it, filter it, or alert on it without regex against free text.

Rules that pay off:

• Event name is stable and machine-readable. order.created, not a prose sentence.
• Context as fields, not interpolated into the message.
• One log line per meaningful event, not per function entry and exit.
• Never log secrets, tokens, PII or full request bodies. Redact at the logging boundary so it can’t be forgotten at a call site.

Correlation is the whole point

trace_id = request.headers.get("X-Request-ID") or str(uuid4())
trace_id_var.set(trace_id)          # contextvar - flows through async calls

A single ID propagated through every service, log line, queue message and background job. Without it, debugging a distributed failure means correlating timestamps across seven log streams, which does not work under load.

Propagate it explicitly into queue messages — this is the gap in most systems. A Celery task or Kafka consumer that starts a fresh trace ID breaks the chain exactly where the interesting failures happen.

Use contextvars so it survives await boundaries. See ../../backend/15_observability/10_correlation_ids_async.md.

The three pillars, and what each is for

The working pattern: alert on metrics, diagnose with traces, confirm with logs. Alerting on logs is expensive and noisy; diagnosing from metrics alone is impossible.

OpenTelemetry is the vendor-neutral instrumentation standard covering all three, and it’s what keeps you from re-instrumenting when you change backends. See ../../backend/15_observability/07_opentelemetry.md.

What to measure

The RED method for request-driven services:

USE for resources: Utilisation, Saturation, Errors. Saturation — queue depth, pool waiting — is the leading indicator; it rises before errors do.

Always percentiles, never averages. Mean latency hides the tail entirely: a service at 50ms mean can be timing out for 2% of users. Track p50, p95, p99, and remember that averaging percentiles across instances is mathematically meaningless — aggregate the histogram, not the summary.

Alerting on symptoms

Alert on what users experience, not on internal state:

Every alert must be actionable at the time it fires. An alert nobody acts on trains the team to ignore alerts, and the one that mattered gets ignored with the rest. If the response is “watch it”, it’s a dashboard, not a page.

SLO burn-rate alerting is the mature version: page when you’re consuming the error budget fast enough to exhaust it, rather than on any threshold crossing. Fast burn pages immediately; slow burn opens a ticket. See ../../backend/15_observability/12_slo_sli_sla.md.

Sampling

You cannot afford to trace everything at volume.

• Head sampling — decide at the start, say 1%. Cheap, and it usually discards the interesting traces.
• Tail sampling — decide after completion, keeping all errors and slow requests plus a baseline of normal ones. More infrastructure, far better signal.

Tail sampling is what you want, because the traces worth keeping are exactly the anomalous ones that head sampling drops.

Cardinality — the thing that breaks metrics

# Catastrophic: one time series per user
request_duration.labels(user_id=user.id).observe(elapsed)

# Fine: bounded label values
request_duration.labels(endpoint="/orders", method="POST", status="200").observe(elapsed)

Every distinct label combination is a separate time series. User IDs, request IDs, or raw URLs as labels produce millions of series and take down your metrics backend — and the bill arrives before the outage does.

High-cardinality data belongs in logs and traces, not metric labels. That’s the division of labour between the pillars, and it’s the mistake most likely to cause real damage.

Interview angle

• “Monitoring versus observability?” — monitoring answers predefined questions with dashboards and alerts; observability lets you ask new questions about failures you didn’t anticipate. Structured, high-cardinality event data is what makes the second possible.
• “How do you debug a failure across seven services?” — a correlation ID generated at the edge and propagated through every call, log line, queue message and background job, using contextvars so it survives await boundaries. Without it you’re matching timestamps, which fails under load.
• “What do you alert on?” — user-visible symptoms: error rate, p99 latency, queue depth, saturation. Ideally SLO burn rate rather than static thresholds. Not CPU, not a single restart. Every alert must be actionable when it fires, or the team learns to ignore all of them.
• “Why percentiles rather than averages?” — the mean hides the tail. A 50ms average is compatible with 2% of users timing out. Also, percentiles can’t be averaged across instances; aggregate the underlying histogram.
• “How do you sample traces at volume?” — tail sampling: decide after the request completes, keep all errors and slow requests plus a baseline of normal traffic. Head sampling is cheaper and throws away exactly the traces you needed.
• “What breaks a metrics system?” — cardinality. Using user ID or request ID as a metric label creates a time series per value and will take down the backend. High-cardinality context goes in logs and traces; metrics keep bounded labels.