Elasticsearch — Aggregations

Aggregations are how Elasticsearch becomes “SQL with a search engine attached.” Group, bucket, compute metrics, nest them — all on top of the inverted index.

Two categories

Bucket aggs can nest other aggs inside their buckets. Metric aggs are leaves.

Simple metric

{
  "query": { "match_all": {} },
  "aggs": {
    "total_revenue": { "sum": { "field": "amount" } },
    "avg_price":     { "avg": { "field": "price" } }
  },
  "size": 0
}

size: 0 means “don’t return docs, just aggregations” — typical for dashboards.

Metric agg types: sum, avg, min, max, value_count, cardinality (HyperLogLog approximate count distinct), percentiles, percentile_ranks, stats, extended_stats.

Terms bucket

{
  "aggs": {
    "by_country": {
      "terms": { "field": "country.keyword", "size": 10 },
      "aggs": {
        "avg_order": { "avg": { "field": "amount" } }
      }
    }
  },
  "size": 0
}

Like SQL GROUP BY country. Result: top 10 countries by doc count, with avg order amount each.

Cardinality trap: terms works fine for low-cardinality fields (countries, statuses). For high-cardinality (user_id with millions of values), it loads all buckets into memory. Mitigations:

• size limits result count (default 10, capped at ~10k).
• composite aggregation for paginated grouping over high-cardinality keys.

Composite — paginated grouping

{
  "aggs": {
    "by_user": {
      "composite": {
        "size": 1000,
        "sources": [
          { "user": { "terms": { "field": "user_id" } } }
        ]
      }
    }
  }
}

Returns 1000 buckets at a time. Response includes after_key for the next page:

{
  "composite": {
    "size": 1000,
    "sources": [{"user": {...}}],
    "after": { "user": "last_user_from_previous_page" }
  }
}

Use for: full user-level aggregations on huge datasets without OOM.

date_histogram

{
  "aggs": {
    "per_day": {
      "date_histogram": {
        "field": "timestamp",
        "calendar_interval": "day"
      },
      "aggs": {
        "events": { "value_count": { "field": "_id" } }
      }
    }
  },
  "size": 0
}

Buckets by calendar interval: minute, hour, day, week, month, quarter, year. Or fixed_interval for arbitrary durations (30m, 2h).

Calendar intervals handle DST and month-length variation; fixed intervals don’t.

range and histogram

{
  "aggs": {
    "price_ranges": {
      "range": {
        "field": "price",
        "ranges": [
          { "to": 10 },
          { "from": 10, "to": 50 },
          { "from": 50 }
        ]
      }
    }
  }
}

histogram is uniform buckets (interval: 10); range is custom boundaries.

Nesting aggs

{
  "aggs": {
    "by_country": {
      "terms": { "field": "country.keyword" },
      "aggs": {
        "by_category": {
          "terms": { "field": "category.keyword" },
          "aggs": {
            "total": { "sum": { "field": "amount" } }
          }
        }
      }
    }
  }
}

Country → Category → total. Three nesting levels is common; deeper gets unwieldy.

Filter and filters

{
  "aggs": {
    "high_value": {
      "filter": { "range": { "amount": { "gte": 100 } } },
      "aggs": {
        "total": { "sum": { "field": "amount" } }
      }
    }
  }
}

Compute a metric over a sub-population (without filtering the whole query). For multiple filters in one pass:

{
  "aggs": {
    "buckets": {
      "filters": {
        "filters": {
          "high":   { "range": { "amount": { "gte": 100 } } },
          "medium": { "range": { "amount": { "gte": 10, "lt": 100 } } },
          "low":    { "range": { "amount": { "lt": 10 } } }
        }
      }
    }
  }
}

cardinality — approximate distinct count

{
  "aggs": {
    "unique_users": { "cardinality": { "field": "user_id", "precision_threshold": 10000 } }
  }
}

Uses HyperLogLog. Not exact — accuracy depends on precision_threshold (memory-bounded, default 3000 ≈ ~0.5% error at 1M unique values).

For exact distinct count, you’d need to retrieve all values or precompute. Cardinality + HLL is the standard for “how many unique users?”

Percentiles

{
  "aggs": {
    "latency_p99": { "percentiles": { "field": "latency_ms", "percents": [50, 95, 99, 99.9] } }
  }
}

Uses t-digest by default — approximate but very accurate, especially in the tails. Standard for SLO monitoring.

top_hits — sample docs per bucket

{
  "aggs": {
    "by_user": {
      "terms": { "field": "user_id" },
      "aggs": {
        "latest": {
          "top_hits": {
            "size": 1,
            "sort": [{ "timestamp": "desc" }]
          }
        }
      }
    }
  }
}

“Latest event per user.” Common pattern for “most-recent-row-per-group” queries.

pipeline aggregations — operate on other aggs

{
  "aggs": {
    "per_day":     { "date_histogram": { ..., "calendar_interval": "day" },
                     "aggs": { "revenue": { "sum": { "field": "amount" } } } },
    "max_day":     { "max_bucket": { "buckets_path": "per_day>revenue" } },
    "cumulative":  { "cumulative_sum": { "buckets_path": "per_day>revenue" } }
  }
}

max_bucket, min_bucket, avg_bucket, sum_bucket, cumulative_sum, derivative, moving_avg, bucket_script for ratios. Like SQL window functions over aggregation buckets.

Performance

• Aggregations are computed on the index, not on docs. Sorting / aggregating on a text field doesn’t work — use keyword.
• doc_values (default on for keyword, numeric, date) is the columnar store used for aggs. Disabling saves space but breaks aggs.
• Cardinality: high-cardinality terms aggs load all buckets into memory → OOM risk. Use composite for pagination.
• size: 0 when you don’t need hits, only aggs. Faster (no scoring, no fetch).
• global agg runs over all docs ignoring the query filter. Useful for “X vs the whole dataset” comparisons.

Scripted aggregations

{
  "aggs": {
    "weighted_avg": {
      "scripted_metric": {
        "init_script":    "state.sum=0; state.weight=0",
        "map_script":     "state.sum += doc['value'].value * doc['weight'].value; state.weight += doc['weight'].value",
        "combine_script": "return state",
        "reduce_script":  "double s=0; double w=0; for (s2 in states) {s+=s2.sum; w+=s2.weight;} return s/w"
      }
    }
  }
}

Maximum flexibility, minimum performance. Avoid unless built-in aggs don’t fit. Painless scripts execute per doc; large data sets get expensive.

Common gotchas

• Aggregation on text field — fails or returns weird results. Use text.keyword sub-field.
• Terms agg size: 100 with many shards — each shard returns top 100, coordinator merges. May miss correct top 100 if distribution is uneven. Increase shard_size for accuracy at cost of memory.
• Date histogram with interval: "1d" vs calendar_interval: "day" — fixed vs calendar. Calendar handles DST; fixed doesn’t.
• min_doc_count: 0 in terms agg returns buckets with zero docs (for the documented categories). Default min_doc_count: 1 skips them.
• exists agg — checks the field has a value. Useful for “how many docs have field X?”

Interview angle

• “What’s a terms aggregation?” — GROUP BY on a field. Returns top-N buckets by doc count, with optional nested aggs per bucket. Use composite for paginated grouping over high-cardinality keys.
• “Why is your aggregation on user_id failing?” — user_id is mapped as text. Aggs need keyword / numeric / date fields with doc_values. Use user_id.keyword or remap as keyword.
• “How does cardinality work and what’s the trade-off?” — HyperLogLog approximation. Memory bounded by precision_threshold. Accurate within ~0.5% at default settings; not exact. For exact count, retrieve values.
• “How do you get the latest doc per group?” — terms bucket + nested top_hits agg with sort desc. The standard “top-N-per-group” pattern in Elasticsearch.
• “Date histogram across multiple time zones — gotcha?” — time_zone parameter on the agg shifts bucket boundaries to that TZ. Without it, buckets are in UTC.
• “What’s a pipeline aggregation?” — operates on the output of another agg (cumulative sum, derivatives, moving averages, bucket scripts). Like SQL window functions over already-aggregated buckets.
• “You’re seeing wrong top-10 from a terms agg on a sharded index. Why?” — per-shard top-N may exclude items that aren’t in the top-N on any single shard but would be in the global top-N. Increase shard_size (default 1.5x size) for more accurate results.