The Three Pillars of Observability

Logs, metrics, traces. Mentioned in every observability interview. Knowing what each is good and bad at — and where they correlate — is more useful than memorizing definitions.

The framing

┌──────────────┬───────────────────────────┬────────────────────┐
│ Pillar       │ Answers                    │ Cardinality        │
├──────────────┼───────────────────────────┼────────────────────┤
│ Logs         │ "what happened here?"      │ high (per event)   │
│ Metrics      │ "how much / how often?"    │ low (aggregated)   │
│ Traces       │ "where did the time go?"   │ medium (per req)   │
└──────────────┴───────────────────────────┴────────────────────┘

Each pillar has a sweet spot. Trying to do one thing’s job with another’s tool leads to either an expensive bill or poor visibility.

The “three pillars” framing has critics (Charity Majors and others argue “observability is one thing — high-cardinality structured events”), but the model is universal in interviews. Know the framing; know the critique.

Logs

Discrete events recorded at runtime. Free-form text or structured JSON.

logger.info("order placed", extra={
    "order_id": 42,
    "user_id": 99,
    "amount": 100.00,
    "currency": "USD",
})

What logs are good at:

• Reconstructing what happened in a specific request.
• Audit trails.
• Debugging rare cases with full context.
• Errors with stack traces.

What logs aren’t good at:

• “How many 500s in the last 5 minutes?” — that’s a metric.
• “Why is this endpoint slow?” — that’s a trace.
• Asking questions across millions of events without expensive search infrastructure.

Structured logging

# Bad
logger.info(f"User {user_id} placed order {order_id} for ${amount}")

# Good
logger.info("order placed", extra={"user_id": user_id, "order_id": order_id, "amount": amount})

The structured form lets log aggregators (Datadog, ELK, Loki) index fields and answer queries like order placed AND amount > 1000.

Always structured in production. f-string logs become regex hell at scale.

Log levels

Production default: INFO. Bumping to DEBUG temporarily on one host to investigate. INFO that includes every DB query is too noisy; reserve for important business events.

Log volume control

Indexed log storage is expensive ($/GB ingested + $/GB retained). Controls:

• Sampling: keep 1% of routine logs, 100% of errors.
• Filtering: drop health-check logs, debug logs in prod.
• Tiered retention: 7-day searchable + 30-day cold storage for compliance.
• Structured + indexed key fields only: don’t index full message body.

Metrics

Aggregated numerical measurements over time. Cheap to store and query at scale.

request_count.inc(labels={"method": "POST", "endpoint": "/api/users", "status": "200"})
request_duration.observe(elapsed_seconds, labels={"endpoint": "/api/users"})

What metrics are good at:

• Alerting on aggregates (5% error rate, p99 latency >500ms).
• Trends over time.
• Cardinality-bounded slicing (per region, per endpoint, per status code).
• SLOs / SLIs.

What metrics aren’t good at:

• “What was this specific request’s error?” — that’s a log.
• “Per-user request rates” — high cardinality kills metrics.
• Debugging specific incidents.

Metric types (Prometheus model)

For latency, histograms are preferred over summaries — they aggregate across hosts via histogram_quantile().

The RED method

For request-driven services:

• Rate — requests per second.
• Errors — failed requests per second.
• Duration — latency distribution.

Three time series per endpoint covers most of “is this service healthy?” The “RED dashboard” is the standard service overview.

The USE method

For resources (CPU, memory, disk, queue):

• Utilization — % busy.
• Saturation — queue / pending work.
• Errors — error counters.

Use for infrastructure dashboards.

Cardinality is the bill killer

http_requests_total{endpoint="/api/users", status="200"}        # OK: ~100 endpoints × ~10 statuses
http_requests_total{endpoint="/api/users", user_id="42"}        # EXPLOSION: 1M users

Metric cardinality = unique combinations of labels. Each combination is a separate time series in storage. Prometheus / Datadog / others scale poorly past ~1M active series per server.

Rule: metrics labels should be bounded cardinality. For per-user / per-request detail, use logs or traces.

Traces

A trace is a tree of spans showing how one request flowed through services. See 06_jaeger.md for tracing depth; 07_opentelemetry.md for the standard.

Request /api/checkout                              [3000ms]
  ├─ auth.verify                                   [50ms]
  ├─ inventory.check                               [200ms]
  ├─ payment.charge                                [2500ms]  ← slowest
  └─ notification.send                             [100ms]

What traces are good at:

• “Where did the 4 seconds go in this request?”
• “Which downstream service is slowing us down?”
• Identifying service-to-service dependencies.

What traces aren’t good at:

• Long-running batch jobs (a 1-hour span is awkward).
• High-volume aggregate questions (“how many requests today?”).
• Replacing logs (traces are samples; not every detail is captured).

Correlation — the real power

Each pillar alone is limited. Together:

1. Alert fires:    "Error rate spike on /api/checkout" (metric)
2. Dashboard:      "Latency also up; mostly in payment service" (metric)
3. Sample trace:   "payment service spans show 30s timeouts to bank-api" (trace)
4. Logs from trace: "bank-api returns 503 'rate limited'" (log)
5. Root cause:     bank-api rate limit; deploy fix

You jumped from metric → trace → log via shared identifiers (service name, trace ID).

The “modern” stack makes this seamless:

• Logs include trace_id and span_id as fields → click a log line → see the trace.
• Traces include service / endpoint tags → match metric dimensions.
• Metrics with exemplars include sample trace IDs → click metric spike → see example traces.

OpenTelemetry’s design makes this correlation automatic. Vendor stacks (Datadog, NewRelic) do it natively. Self-hosted stacks (Prometheus + Loki + Tempo via Grafana) integrate via Grafana’s data source linking.

When to use which

Symptom: "production is broken"
  ↓ check metrics → which service / endpoint?
  ↓ check traces from that service → where is the slow / failing span?
  ↓ check logs from that span → what's the actual error?

Top-down debugging. Metrics for “is something wrong”; traces for “where is something wrong”; logs for “what is wrong.”

Don’t:

• Use logs to ask “how many users in the last hour” — that’s a metric (active_users_total).
• Use metrics to debug a specific failed request — that’s a trace + log.
• Use traces to monitor service health — that’s a metric (request_rate, error_rate).

The cost dimensions

Per-event cost: traces > logs > metrics. Volume cost: logs > traces > metrics. Cardinality cost: metrics most sensitive.

Practical budget split (rough, varies wildly):

• ~50% on logs.
• ~25% on metrics.
• ~25% on traces.

The fourth pillar — profiles

Some advocates argue continuous profiling (cpu/memory/lock profiles, sampled continuously) is the missing 4th pillar.

• Datadog Continuous Profiler.
• Pyroscope (open source).
• pprof (Go ecosystem standard).
• py-spy / Austin / Pyinstrument (Python).

Useful for “why is my CPU pegged?” and “memory keeps growing” — questions that span and trace can’t answer without modification.

Less mature ecosystem; growing.

The critique — “structured events” alternative

Charity Majors / Honeycomb argue:

The three pillars are a vendor abstraction. The real answer is high-cardinality structured events. Don’t classify them as logs vs metrics vs traces — emit rich events, query them ad-hoc.

The Honeycomb / Lightstep model: every span is a structured event with hundreds of attributes; queries pivot any attribute. Closer to a database than a pre-aggregated time series.

For interview: know the framing. Most teams use the three-pillar model; honeycomb-style is the avant-garde alternative.

Common interview confusions

• “More logs = better observability.” — logs without structure / correlation are just text. Metric and trace correlation matters more than log volume.
• “Traces replace logs.” — they complement. Traces show paths; logs show events. Both are useful.
• “Metrics scale infinitely.” — only with bounded cardinality. High-cardinality labels kill the storage system.

Interview angle

• “What are the three pillars of observability?” — logs (discrete events), metrics (aggregated numbers), traces (request paths through services). Each has a sweet spot; together they let you debug top-down (metric → trace → log).
• “When would you use a metric vs a log?” — metric for “how many / how often / how fast” aggregated over time (alerts, dashboards, SLOs). Log for per-event detail (what happened in this specific request, audit trail, debugging).
• “What’s RED method?” — Rate, Errors, Duration. Three metrics per endpoint that cover “is this service healthy?” Standard service dashboard contents.
• “What’s USE method?” — Utilization, Saturation, Errors. For resources (CPU, memory, disk, queues). Standard infrastructure dashboard contents.
• “Why is metrics cardinality important?” — each unique label combination is a separate time series. High-cardinality labels (user_id, request_id) explode storage. Bounded labels (endpoint, status, region) scale fine.
• “How do traces and logs correlate?” — both include trace_id and span_id as fields. Click a log line → see the originating trace; click a slow span → see its logs. OpenTelemetry instrumentation makes this automatic.
• “Three pillars vs Charity Majors’ ‘one thing’?” — Charity argues observability is one concept: high-cardinality structured events that you query ad-hoc. The three-pillar framing is the operational reality but a simplification. Know both for senior interviews.