Datadog

Commercial SaaS observability platform. One vendor for metrics, logs, APM (traces), real-user monitoring, synthetic tests, security monitoring, and more. The “buy” choice when the alternative is running Prometheus + Grafana + Loki + Jaeger yourself.

For comparison with self-hosted stacks see 09_comparison_choosing.md. For Sentry / Grafana / Prometheus see siblings 01_sentry.md, 02_grafana.md, 03_prometheus.md.

What Datadog provides

The pitch: one vendor, unified UI, correlated data (click from a metric spike to traces to logs to deploys), no infrastructure to run. Cost: per-host, per-million-spans, per-million-log-events. Real bills run into thousands per month for medium teams, six figures for large.

The Datadog Agent

Runs on each host (or as a DaemonSet in Kubernetes). Responsibilities:

• Collects host metrics (CPU, memory, disk, network).
• Receives metrics from apps via statsd-compatible “DogStatsD” protocol.
• Receives traces from apps via the trace agent.
• Tails log files and forwards to the SaaS.
• Auto-discovers services (Postgres, Redis, MySQL, etc.) and scrapes their metrics.

# Agent install (typical Docker)
docker run -d \
  --name datadog-agent \
  -e DD_API_KEY=<key> \
  -e DD_SITE=datadoghq.com \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /proc/:/host/proc/:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  gcr.io/datadoghq/agent:latest

The agent is the data plane; the SaaS is the storage + UI. No data leaves the host without going through the agent.

Custom metrics from Python (DogStatsD)

from datadog import statsd

# counters
statsd.increment("orders.placed", tags=["region:us-east", "tier:pro"])

# gauges
statsd.gauge("queue.depth", queue.size(), tags=["queue:emails"])

# histograms (distribution-aware)
statsd.histogram("request.duration", elapsed_ms, tags=["endpoint:/api/users"])

# distribution (better aggregation across hosts)
statsd.distribution("response.size", size_bytes)

statsd.distribution() is preferred over histogram for cross-host aggregation — histogram percentiles can’t be merged across hosts; distribution can.

Tags are key:value pairs. Datadog’s UI slices by them. Common tags: env, service, version, region, endpoint.

APM — distributed tracing

pip install ddtrace

Auto-instrumentation:

ddtrace-run python myapp.py

ddtrace-run monkey-patches popular libraries (Django, Flask, FastAPI, SQLAlchemy, Redis, requests, etc.). Spans are created automatically for HTTP requests, DB queries, cache calls. No code changes required for most apps.

Manual spans:

from ddtrace import tracer

@tracer.wrap(service="payments", resource="process_order")
def process_order(order):
    with tracer.trace("payments.charge_card", resource=order.card.last4):
        charge_card(order)
    with tracer.trace("payments.send_receipt"):
        send_receipt(order)

Set environment variables once:

export DD_SERVICE=my-app
export DD_ENV=production
export DD_VERSION=1.2.3
export DD_AGENT_HOST=localhost

These become tags on every span. Use them in the UI to scope dashboards and alerts.

Logs

Two ingestion paths:

1. Tail files via agent: log file → agent → SaaS.
2. Direct via the SDK: datadog.logger or any logger configured with a Datadog handler.

For Python:

import logging
import json_log_formatter

formatter = json_log_formatter.JSONFormatter()
handler = logging.StreamHandler()
handler.setFormatter(formatter)
logger = logging.getLogger("myapp")
logger.addHandler(handler)
logger.setLevel(logging.INFO)

logger.info("order processed", extra={
    "order_id": order.id,
    "user_id": user.id,
    "amount": amount,
})

Structured JSON output goes to stdout. The Datadog agent parses it and ships to the SaaS, where you can query by field.

Trace correlation: if you also have ddtrace instrumented, the agent automatically injects dd.trace_id and dd.span_id into log records. Click a log line → see the originating trace.

Dashboards and alerts

Dashboards: drag-and-drop widgets querying metrics, logs, traces. Saved as JSON; can be versioned in git via Terraform or the Datadog API.

Alerts (“monitors”):

metric: avg:trace.web.request.duration{service:my-app}.as_count() > 5
window: last 5 minutes
notify: @slack-oncall, @pagerduty-critical

Monitor types: metric (threshold), anomaly (deviation from baseline), forecast (will it cross threshold?), composite (and/or of multiple), event-based, log-based.

For real alerting hygiene: write monitors in Terraform, review in PRs, avoid clicking around in the UI.

Pricing — the elephant in the room

Datadog bills by:

Real-world examples:

• 100 hosts × $30 = $3000/month infrastructure alone.
• 10 TB logs ingestion = $1000/month — but if you retain, $$$ more.
• High-cardinality tags (e.g. user_id:123 on every metric) explode into hundreds of “custom metrics” each.

The bill grows non-linearly. Companies have horror stories of seven-figure annual Datadog spend.

Cost controls:

• Drop unused metrics via “metrics without limits.”
• Sample APM traces (default 1% in some configs; tune for cost vs coverage).
• Use exclusion filters to drop noisy logs (health checks, etc.).
• Set log indexing rules so verbose-but-rarely-searched logs don’t get indexed.

High-cardinality and the metrics bill

Datadog charges per unique metric+tags combination. So:

statsd.increment("api.requests", tags=[f"user_id:{user.id}"])

If you have 1M users, that’s 1M custom metrics. At $0.05 each, $50k/month for one counter.

Don’t put high-cardinality dimensions (user_id, session_id, trace_id, request_id) in metric tags. Put them in logs and traces, where you pay per-event not per-cardinality.

Tracing — sampling

Tracing every request becomes expensive at high RPS. Datadog’s APM samples:

• Head-based sampling: decision made at trace start. Simple but loses correlation across services.
• Tail-based sampling: decision after the trace completes. Keeps traces with errors / slow paths; drops uninteresting ones. Datadog supports both.

# Head sampling at 10%
from ddtrace import tracer
from ddtrace.sampler import RateSampler
tracer.sampler = RateSampler(sample_rate=0.1)

Or sample based on rules (always keep errors, always keep slow):

from ddtrace.sampling_rule import SamplingRule
tracer.sampler = DatadogSampler(rules=[
    SamplingRule(sample_rate=1.0, service="payments"),     # 100% of payments
    SamplingRule(sample_rate=0.01, service="ads-tracking") # 1% of low-priority
])

Service Map

Datadog’s killer feature for distributed systems. Auto-generated graph of services calling services, based on trace data. Click a service → see its upstreams, downstreams, error rate, latency. Click a connection → see the slow / errored traces.

For microservices, this is the closest thing to “X-ray vision” into your architecture.

Datadog vs Prometheus + Grafana

The standard “buy vs build” trade-off:

Pick Datadog when: you’d otherwise hire 2-3 SREs to run the OSS stack. Pick OSS when: cost matters, you have ops expertise, or you need vendor independence.

Hybrid: many teams emit OpenTelemetry to BOTH Datadog and a self-hosted stack, hedging vendor lock-in. See 07_opentelemetry.md.

Datadog gotchas

Agent missing data after deploy

If DD_AGENT_HOST isn’t set, ddtrace defaults to localhost:8126. In Docker without networking config, that doesn’t reach the agent. Symptoms: traces missing for the new service. Fix: DD_AGENT_HOST=host.docker.internal (Mac/Windows Docker) or proper Kubernetes service.

Metrics not appearing

Custom metrics may take 1-5 minutes to appear after first emission. Tags must be lowercase letters, digits, underscores, hyphens, periods, slashes — but Datadog silently normalizes some characters; build expected tag names and stick to them.

Log volume explosion

Verbose libraries (SQLAlchemy with echo=True, request loggers) push GB/day per host. Bill skyrockets. Audit log levels and add exclusion filters.

High-cardinality custom metrics

statsd.increment("api.requests", tags=[f"user_id:{user.id}"]) → unique metric per user. At scale = bankruptcy. Use logs/traces for high-cardinality.

Trace correlation breaks across queues

Default ddtrace propagates trace context via HTTP headers. Background jobs (Celery, RQ) need manual context injection — or use ddtrace’s queue integrations (which do it automatically for supported brokers).

Sampling decisions diverge

Head-based sampling: at request start, decide. If service A keeps a trace at 1% and service B keeps at 10%, only the union is in Datadog. Configure consistent rates or use tail-based sampling.

When Datadog is worth it

• Small team, lots to monitor: managed service saves SRE time.
• Multi-environment correlation needed: dev / staging / prod / multi-region in one UI.
• Compliance benefits: Datadog provides SOC 2 reports, helps with audit logging.
• Need APM with rich visualizations: service maps, flame graphs, breakdowns by tag.
• Logs + metrics + traces correlated: out-of-box; building this on OSS takes work.

When OSS wins

• Cost-sensitive at scale: $50k/year Datadog bill ≈ 1 senior SRE running OSS.
• Strict data residency: Datadog has regions but your data is still on their infrastructure.
• Need specific custom features: Prometheus + Grafana plugins are vast.
• Vendor lock-in concerns: your dashboards / alerts are in Datadog’s proprietary format.

Interview angle

• “What’s Datadog and why use it?” — SaaS observability platform unifying metrics, logs, APM (traces), RUM, synthetics. Buy-vs-build alternative to running Prometheus + Grafana + Loki + Jaeger. Pay for managed service + correlation between data types.
• “How does the Datadog Agent work?” — runs per-host (or k8s DaemonSet). Collects host metrics, receives app metrics via DogStatsD, traces via the trace agent, tails log files. Forwards everything to Datadog SaaS.
• “How would you instrument a Python service for Datadog APM?” — pip install ddtrace, run with ddtrace-run python app.py for auto-instrumentation of Django/Flask/FastAPI/SQLAlchemy/etc. Set DD_SERVICE, DD_ENV, DD_VERSION env vars.
• “What’s the high-cardinality pitfall in Datadog metrics?” — Datadog bills per metric+tag-combo. Putting user_id as a tag creates one metric per user → bill explodes. Keep high-cardinality dimensions in logs/traces, not metrics.
• “Datadog vs Prometheus + Grafana — when each?” — Datadog when team is small + multiple data types + correlation matters + budget exists. OSS when cost matters, you have ops capacity, vendor independence is a requirement.
• “How do you control Datadog costs?” — drop unused custom metrics, sample APM traces (head or tail), exclude noisy logs (health checks, debug), control log indexing (ingest doesn’t equal index), tag discipline (no high-cardinality in metrics).
• “What’s tail-based vs head-based trace sampling?” — head-based decides at trace start (cheap, may miss errors). Tail-based decides after the trace completes (keeps interesting traces — errors, slow paths — drops boring ones). Datadog supports both.