OpenTelemetry

The CNCF-graduated standard for emitting observability data: traces, metrics, and (newer) logs. Replaces OpenTracing + OpenCensus (both merged into OTel in 2019). The vendor-neutral instrumentation layer underneath Jaeger, Tempo, Datadog, New Relic, etc.

The pitch: instrument once, ship anywhere. No vendor lock-in at the SDK level.

The architecture

┌──────────────────────┐
│ App (Python, Go, ...)│
│ + OpenTelemetry SDK   │  ← spans, metrics, log records created here
└──────────────────────┘
          ↓ OTLP (OpenTelemetry Protocol)
┌──────────────────────┐
│ OpenTelemetry Collector │  ← optional; processes, samples, routes
└──────────────────────┘
          ↓ OTLP / vendor protocols
┌──────────────────────┐  ┌──────────────────────┐  ┌──────────────────────┐
│ Jaeger               │  │ Prometheus           │  │ Datadog / NewRelic / │
│ Tempo                │  │ Loki                 │  │ Honeycomb / Splunk   │
└──────────────────────┘  └──────────────────────┘  └──────────────────────┘

Three concepts:

1. API — what your app code calls (tracer.start_as_current_span("...").
2. SDK — the implementation that creates spans, batches, exports.
3. Collector — optional intermediary process that routes, processes, samples.

App code depends only on the API; the SDK and collector can be swapped without changing application code.

OTLP — the wire protocol

OpenTelemetry Protocol. Two transports:

• gRPC (port 4317) — efficient binary, default for Linux servers.
• HTTP/JSON or HTTP/Protobuf (port 4318) — works through HTTP proxies, easier debugging.

Most backends (Jaeger 1.35+, Tempo, Datadog, Honeycomb) accept OTLP directly. Some still require translation via the Collector.

Python instrumentation — the minimum

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

resource = Resource(attributes={
    "service.name": "my-app",
    "service.version": "1.2.3",
    "deployment.environment": "production",
})

provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True)
))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

@tracer.start_as_current_span("checkout")
def checkout(order):
    span = trace.get_current_span()
    span.set_attribute("order.id", order.id)
    span.set_attribute("order.total", float(order.total))
    process(order)

Pattern: get a tracer at module level; use start_as_current_span decorator or context manager around units of work.

Resource attributes — the constants

Resource attributes are attached to every span/metric/log from this process. Standard ones (from the OTel semantic conventions):

Set them once at startup. Don’t repeat per-span — they’re inherited.

Auto-instrumentation

For most popular Python libraries, instrumentation packages exist:

pip install opentelemetry-distro \
  opentelemetry-instrumentation-flask \
  opentelemetry-instrumentation-requests \
  opentelemetry-instrumentation-sqlalchemy \
  opentelemetry-instrumentation-redis \
  opentelemetry-instrumentation-celery

# Or auto-install for everything already in your env:
opentelemetry-bootstrap -a install

Run with auto-instrumentation:

opentelemetry-instrument \
  --service_name=my-app \
  --exporter_otlp_endpoint=http://otel-collector:4317 \
  python app.py

Spans are auto-created for HTTP requests, DB queries, cache calls, queue consumption — no code changes. You add custom spans only for app-specific work the instrumentation doesn’t cover.

Common instrumentation packages:

Spans, attributes, events, links

with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order.id", 42)
    span.set_attribute("order.items", 5)

    try:
        result = charge_card(order)
        span.set_attribute("payment.amount", result.amount)
        span.set_status(trace.Status(trace.StatusCode.OK))
    except PaymentError as e:
        span.record_exception(e)
        span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
        raise

    span.add_event("order_persisted", {"db.row_id": order.row_id})

Semantic conventions

OTel defines standard attribute names so backends can do consistent analysis:

Auto-instrumentation populates these. For manual spans, follow the conventions so dashboards / queries built for the standard work.

Metrics

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter

reader = PeriodicExportingMetricReader(OTLPMetricExporter(endpoint="...:4317"))
metrics.set_meter_provider(MeterProvider(metric_readers=[reader], resource=resource))
meter = metrics.get_meter(__name__)

orders_counter = meter.create_counter("orders.placed", unit="1", description="Orders placed")
order_amount = meter.create_histogram("orders.amount", unit="USD")

orders_counter.add(1, {"tier": "pro", "region": "us-east"})
order_amount.record(99.99, {"currency": "USD"})

Instrument types:

The OTLP metrics exporter sends to a Prometheus-compatible target or to the Collector. The Collector can convert to Prometheus scrape format for backward compatibility.

Logs

The newer pillar; less mature in OTel than traces/metrics. Two approaches:

1. Bridge existing loggers: opentelemetry-instrumentation-logging attaches trace ID / span ID to standard logging records.
2. Native OTel logs API: emit log records via the OTel SDK; ship via OTLP.

Most teams stick with structured logging through their existing logger and use the bridge for trace correlation. See 09_structured_logging_TODO (concept covered in system_design/05_observability/01_observability_in_practice.md).

Sampling

Decision tree:

Head-based, configured in SDK:

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

provider = TracerProvider(
    resource=resource,
    sampler=TraceIdRatioBased(0.1),    # 10% of traces
)

Tail-based requires the OpenTelemetry Collector with the tail_sampling processor:

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors-policy
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-policy
        type: latency
        latency: { threshold_ms: 1000 }
      - name: random-policy
        type: probabilistic
        probabilistic: { sampling_percentage: 1 }

The collector buffers traces, applies policies, samples accordingly. Catches errors + slow + 1% baseline.

The Collector — the routing layer

A standalone process that:

• Receives telemetry from apps (OTLP).
• Processes (sample, redact, transform, batch).
• Exports to one or more backends (Jaeger, Prometheus, Datadog, …).

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc: {}
      http: {}

processors:
  batch: {}
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: insert

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls: { insecure: true }
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Run as a sidecar (one per pod / host) or as a centralized service. Sidecar pattern is common in Kubernetes; centralized for VMs.

Why use it instead of direct app → backend:

• Switch backends without redeploying apps.
• Centralized sampling, redaction, batching.
• One process handles auth and TLS to the backend.
• Buffer when the backend is slow/down.

For small deployments: skip the collector; export directly. For production: run it.

Multi-backend export

Common pattern: emit OTel from apps; collector exports to TWO backends (e.g., Datadog + self-hosted Jaeger). Vendor lock-in hedge.

exporters:
  datadog:
    api: { key: ${DD_API_KEY} }
  otlp/jaeger:
    endpoint: jaeger:4317

service:
  pipelines:
    traces:
      exporters: [datadog, otlp/jaeger]

Cost: traces ingested twice. Benefit: switch primary backend instantly if needed.

Context propagation across boundaries

For HTTP: auto-instrumentation handles W3C traceparent injection/extraction.

For other boundaries (queues, custom protocols), inject/extract manually:

from opentelemetry.propagate import inject, extract

# Producer
carrier = {}
inject(carrier)
celery_app.send_task("work", kwargs={"data": ..., "_trace": carrier})

# Consumer
def work(data, _trace=None):
    ctx = extract(_trace or {})
    with tracer.start_as_current_span("work", context=ctx):
        ...

Many instrumentations (Celery, Kafka, RabbitMQ, gRPC) do this automatically.

Common pitfalls

• Forgetting BatchSpanProcessor: using SimpleSpanProcessor flushes synchronously per span — kills performance.
• Service name typos: dashboards built on service.name=my-app break when one host emits my_app. Set via env var (OTEL_SERVICE_NAME=my-app) for consistency.
• High-cardinality attributes: user.id=42 on every span explodes index size. Fine in spans (queryable); avoid in metric labels (cost).
• Auto-instrumentation creating noisy spans: health checks, metrics endpoints. Filter out via OTEL_PYTHON_FLASK_EXCLUDED_URLS etc.
• Mixing OpenTracing + OpenTelemetry: OpenTracing was the predecessor; coexisting both leads to broken context propagation. Migrate fully to OTel.
• No Collector → app blocks on slow backend: if the backend is down, the SDK’s batch queue fills, and exports block. Use the Collector as a buffer.

OpenTelemetry vs vendor SDKs

Trend: vendors are accepting OTLP as input (Datadog, New Relic, Honeycomb, Splunk). The future is “instrument with OTel, ship anywhere.” Vendor agents are useful when you want their specific features (Datadog Continuous Profiler, NR’s full-stack integrations).

For greenfield: start with OpenTelemetry. Add vendor agent only if you need specific features.

Common interview confusions

• “OpenTelemetry replaces Jaeger.” — OTel is instrumentation; Jaeger is a backend. They cooperate: OTel SDK → OTLP → Jaeger backend.
• “You don’t need a Collector if you’re starting out.” — true in simple setups. Becomes load-bearing when you have multiple backends, sampling policies, or unreliable network paths.
• “OpenTelemetry covers logs the same as traces.” — logs are the newer pillar; less mature. Most teams use existing loggers + trace-ID bridge.

Interview angle

• “What is OpenTelemetry?” — vendor-neutral standard for emitting observability data (traces, metrics, logs). CNCF graduated. Merger of OpenTracing + OpenCensus (2019). Same instrumentation works for any compliant backend (Jaeger, Tempo, Datadog, NewRelic, Honeycomb, …).
• “What’s OTLP?” — OpenTelemetry Protocol; the wire format for shipping telemetry. gRPC (port 4317) or HTTP/Protobuf or HTTP/JSON (port 4318). Native support in most backends.
• “How would you instrument a Python service?” — install opentelemetry-sdk + opentelemetry-exporter-otlp + auto-instrumentation libraries; configure resource with service.name, set up BatchSpanProcessor with OTLPSpanExporter. Run with opentelemetry-instrument python app.py for auto-instrumentation of HTTP/DB/cache.
• “What does the OpenTelemetry Collector do?” — intermediary that receives OTLP, processes (sample, redact, batch), exports to one or more backends. Useful for switching backends, centralized policies, buffering. Optional for simple setups; load-bearing in production.
• “What are semantic conventions?” — standard attribute names so all OTel-instrumented services emit comparable data (http.method, db.statement, messaging.system, etc.). Auto-instrumentation populates them; backends rely on them for cross-service analysis.
• “OpenTelemetry vs vendor SDK like ddtrace?” — OTel is vendor-neutral, ships to any backend, less polished. ddtrace is Datadog-specific, very polished, locks you in. Modern recommendation: OTel for new code; switch only if vendor features (profiler, native integrations) justify the lock-in.
• “Head-based vs tail-based sampling — where do you configure each?” — head-based in the SDK (TraceIdRatioBased(0.1)); tail-based in the Collector (tail_sampling processor with rules: error → keep, slow → keep, random 1% baseline).