ELK Stack (Elasticsearch, Logstash, Kibana)

The classic self-hosted log aggregation stack: centralize logs from every service, make them full-text searchable, dashboard them. “ELK” today usually means the wider Elastic Stack — the original trio plus Beats shippers — and often runs as EFK (Fluentd/Fluent Bit instead of Logstash) on Kubernetes.

The components

The pipeline

app (JSON to stdout)
  → Filebeat / Fluent Bit          (per-node DaemonSet, tails container logs)
    → [Kafka buffer — optional]    (absorbs spikes, decouples)
      → Logstash / ingest pipeline (parse, enrich, drop noise)
        → Elasticsearch            (index per day/size)
          → Kibana                 (search & dashboards)

Two decisions define the topology:

• Logstash or not. If apps emit structured JSON already, Filebeat → Elasticsearch ingest pipelines cover light parsing and Logstash is skippable. Logstash earns its cost when you must grok unstructured legacy formats, enrich (geoip, lookups), or route to multiple destinations.
• Buffer or not. A Kafka layer between shippers and indexers keeps log spikes (incident = log storm — exactly when you need logs) from overwhelming Elasticsearch, at the cost of another system.

Structured logs make or break it

Elasticsearch indexes JSON fields. If apps log JSON ({"event": "payment_failed", "order_id": 42, ...}), fields arrive query-ready: event: payment_failed AND amount > 100. If apps log prose, you’re writing brittle grok regexes in Logstash forever. Fix it at the source: 13_structured_logging.md.

Watch mapping explosion: Elasticsearch auto-creates a mapping per new field name. Unbounded field names (user IDs as keys, exception dumps) bloat the cluster. Cap with explicit mappings/templates and a disciplined log schema.

Index management — where ELK clusters die

Logs are time-series; the operational model is time-based indices + lifecycle management (ILM):

• Write to logs-2026.07.08 (or data streams with rollover at ~30–50 GB/shard).
• ILM moves indices through tiers: hot (fast NVMe, being written) → warm/cold (cheaper, query-only) → delete after retention.
• Retention is a cost policy, not a technical limit: keep 7–30 days hot, archive the rest to object storage if compliance demands.

Most “our ELK is on fire” stories are unmanaged indices: too many shards, no rollover, no deletes.

ELK vs the alternatives

Loki’s pitch — “index the labels, not the text” — is the direct response to Elasticsearch’s cost. Broader stack comparison: 09_comparison_choosing.md, 04_datadog.md.

Python integration

Don’t ship logs from Python directly (no python-logstash handlers pointing at Logstash in prod). The 12-factor pattern wins operationally: write JSON to stdout, let the platform ship it (../13_architecture_design/20_twelve_factor_app.md):

# app: structlog → JSON on stdout (see 13_structured_logging.md)
# node: Filebeat/Fluent Bit DaemonSet tails /var/log/containers/*.log
# nothing in app code knows Elasticsearch exists

App-side shipping couples every service to the log pipeline’s availability and loses logs on network failure; node-level shippers buffer and retry on disk.

Common pitfalls

• Logging unstructured text and compensating with grok — fix at the source.
• No ILM: cluster fills, indexing stops, and you lose current logs during the incident that caused the volume.
• Full-bore DEBUG in production “temporarily” — ingest cost is per-GB in every stack; sample or gate noisy logs.
• Treating Kibana as the alerting system of record — metrics alert faster and cheaper (03_prometheus.md); logs explain why after metrics say that (08_three_pillars.md).

Interview angle

• “Design centralized logging for 30 microservices.” — stdout JSON → DaemonSet shipper → (buffer) → store → UI; name retention/ILM and structured logging unprompted.
• “ELK vs Loki?” — full-text indexing cost vs label-index-only; pick by query needs and ops budget.
• “Why did your Elasticsearch cluster fall over?” — mapping explosion, shard sprawl, no lifecycle policy — the triad behind most real incidents.
• “Where do logs fit vs metrics and traces?” — three pillars answer: metrics detect, traces locate, logs explain (08_three_pillars.md).