Sampling Strategies for Tracing

You can’t store every span from every request at scale — cost and storage. Sampling decides which traces to keep. Where you sample and how matters.

Why sample

A medium-traffic service: 10k requests per second × 10 spans per trace × 1 KB/span = 100 MB/sec of trace data. Few teams can pay for that retention.

At the same time: keeping only 1% means missing 99% of trace data; if the interesting traces (errors, slow requests) are rare, head-based 1% loses them most of the time.

The two dimensions

Plus probabilistic vs deterministic, fixed-rate vs adaptive.

Head-based sampling

The root span (request entry) decides “keep this trace or drop it”. The decision propagates downstream via traceparent (the sampled bit). All services see the same decision; the trace is either fully captured or fully dropped.

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased, ParentBased

# Sample 5% of traces; honor parent decision when present
sampler = ParentBased(root=TraceIdRatioBased(0.05))

ParentBased is critical — without it, a downstream service might re-sample and you’d get partial traces (head present, tail dropped) or vice versa.

Pros and cons

• Cheap. No need to buffer the whole trace.
• Consistent. Trace is whole or absent.
• Can’t sample by outcome. The “this trace had a 5xx error” decision happens at the end, but the keep/drop decision happened at the start.

Tail-based sampling

Collect every span, buffer them at the collector, decide which traces to keep once the whole trace has arrived (or timeout). The collector then drops most of them and forwards the interesting ones.

# OTel Collector config
processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    expected_new_traces_per_sec: 10000
    policies:
    - name: errors
      type: status_code
      status_code: { status_codes: [ERROR] }
    - name: slow
      type: latency
      latency: { threshold_ms: 1000 }
    - name: probabilistic
      type: probabilistic
      probabilistic: { sampling_percentage: 1 }

“Keep all errors, all slow traces (>1s), and a random 1% of the rest.”

Pros and cons

• Sample by outcome — keep errors, slow traces, specific endpoints.
• Visibility into the interesting cases.
• Collector needs to buffer every trace; high memory cost.
• Latency window before the decision (30s here) — your error spans aren’t visible immediately.
• Distributed traces require centralized collection — one collector must see the whole trace.

Adaptive sampling

Probability changes based on traffic / outcomes. “Always keep errors; 100% of slow; rate-limit successful traces to N/sec.”

OTel Collector supports this via tail_sampling policies. Some commercial backends (Datadog APM, Honeycomb) include adaptive sampling at ingest.

Per-endpoint sampling

Sample different rates per route:

class CustomSampler(Sampler):
    def should_sample(self, parent_context, trace_id, name, kind, attributes, links, trace_state):
        attrs = attributes or {}
        if attrs.get("http.target", "").startswith("/healthz"):
            return SamplingResult(Decision.DROP)
        if attrs.get("http.target", "").startswith("/api/v1/orders"):
            return SamplingResult(Decision.RECORD_AND_SAMPLED)
        # default
        return TraceIdRatioBased(0.01).should_sample(...)

Health checks are noisy and uninteresting → drop. Critical endpoints → keep everything.

What you usually want

- 100% errors
- 100% latency outliers (p99+)
- 10-50% of new / unique trace shapes
- 1-5% of routine successful traces
- 0% of health checks / metrics endpoints

This requires tail-based sampling (you don’t know outcome at the start). Most production tracing setups end up here.

For low-volume services (< 100 RPS), head-based 100% is fine and gives you everything for free.

Sampling decisions and bias

If you use sampled traces to compute “what % of traces have error X”, you’ll get a biased answer when you bias the sample toward errors. Track unbiased metrics (sums, counts) at the SDK level instead of inferring from sampled traces.

Logs and metrics alongside

Sampling tracing data doesn’t mean sampling logs and metrics:

• Metrics — always emit. Counters, histograms. Cheap to aggregate.
• Logs — emit all error logs; sample debug-level logs.
• Traces — sample.

Datadog, New Relic, Honeycomb all decouple these. Don’t tail-sample your metrics; that breaks aggregates.

Decision: keep vs drop the trace

When dropping, you can still record the trace’s existence as a counter (for metrics) without storing the spans:

# Drop spans; bump a counter
counter.add(1, attributes={"http.route": route, "status": status})

This way “how many requests” stays accurate even when you’ve dropped trace details.

OTel head-vs-tail summary

Sampling Cognito / Lambda / Step Functions

Each AWS service has its own sampling story. For Lambda: X-Ray Active Tracing has built-in sampling rules (1 req/sec + 5% of additional). OTel via the AWS Distro supports the same.

For a hybrid OTel + AWS X-Ray setup, ensure samplers don’t conflict — one decision per trace.

Production checklist

• Errors always sampled.
• Latency outliers always sampled.
• Health checks NEVER sampled.
• Reasonable baseline rate for the rest.
• Metrics unsampled (don’t bias aggregates).
• ParentBased sampler so all services agree.
• Verify on-call can find recent error traces in the backend.

Interview angle

• “What’s head-based vs tail-based sampling?” — head: decide at trace start (cheap, consistent, can’t sample by outcome). Tail: decide at the collector after the whole trace arrives (can keep errors / slow, but needs buffering and a sampling window).
• “Why do you need ParentBased sampling?” — propagates the sample/don’t-sample decision via traceparent so all services in a trace agree. Without it, downstream services may make a different decision and you get partial traces.
• “How do you ensure you keep error traces if you sample at 1%?” — tail-based. The collector keeps errors regardless of the probabilistic rate. Head-based 1% misses 99% of errors when errors are random.
• “What’s the cost of tail-based sampling?” — the collector must buffer every trace (every span from every service) for decision_wait seconds. Memory and CPU at the collector scale with trace rate. Centralized collection is required — one collector must see the whole trace.
• “How do you sample without biasing your error rate metrics?” — emit metrics at the SDK level (counters, histograms) regardless of trace sampling. Metrics ≠ traces. Don’t compute “what % of traces have errors” from sampled traces — use unsampled metrics for aggregates.
• “What rate would you start with?” — for tail: 100% errors + 100% slow + 1-5% successful. For head: 5-10% baseline, drop health checks. Tune based on storage / cost / debugging needs.