Elasticsearch — Query DSL and BM25 Scoring

The Query DSL is JSON. Verbose, expressive, sometimes surprising. The scoring model is BM25 (replaced TF-IDF as default in ES 5.0). Understanding both is what separates senior from junior.

term vs match — the foundational distinction

// term — exact, no analysis on the query side
{ "query": { "term": { "status": "active" } } }

// match — query is analyzed (tokenized, lowercased, stemmed) before matching
{ "query": { "match": { "body": "Running Fast" } } }

term queries the inverted index for the literal value of the query string. Useful for keyword fields, IDs, status enums.

match runs the query through the field’s analyzer first, then looks up the resulting tokens. Useful for text fields — "Running Fast" becomes [run, fast] and matches docs containing those stemmed tokens.

Common bug: running a term query against a text field. The doc indexed "Running fast" as [run, fast], but term: { body: "Running fast" } looks for the literal string. No match.

The main query types

bool — the workhorse

{
  "query": {
    "bool": {
      "must":     [{ "match": { "title": "python" } }],
      "should":   [{ "match": { "tags":  "async"  } }],
      "filter":   [{ "term":  { "language": "en" } }],
      "must_not": [{ "term":  { "draft": true } }]
    }
  }
}

• must — must match; contributes to score.
• should — should match; contributes to score; not required unless minimum_should_match set.
• filter — must match; does NOT contribute to score; cacheable. Faster.
• must_not — must not match.

Rule: use filter for non-scoring binary conditions (status = active, language = en, date range); use must for the actual search terms whose relevance matters. Filters are cached and orders of magnitude faster.

BM25 scoring

When you run a match query, each matching doc gets a _score. The scoring formula is BM25 (Best Matching 25), an evolution of TF-IDF:

score(d, q) = Σ over terms t in q:
    IDF(t) * (tf(t, d) * (k1 + 1)) / (tf(t, d) + k1 * (1 - b + b * (|d| / avgdl)))

Where:

• tf(t, d) = term frequency in doc d.
• IDF(t) = inverse document frequency (rare terms score higher).
• |d| = doc length; avgdl = average doc length.
• k1 (default 1.2) — term frequency saturation; higher = more weight to repeated terms.
• b (default 0.75) — length normalization; higher = penalize longer docs more.

Intuitively:

• Rare terms (low IDF) score higher than common ones.
• Multiple occurrences in a doc help up to a point (saturating at k1).
• Short docs with the term score higher than long docs (length normalization).

You can tune k1 and b per field, but defaults are usually fine. Tuning is for niche cases (very short docs benefit from b=0; bag-of-words like tags from k1=0).

Boosting

Field-level boost: search “title” twice as important as “body”:

{
  "query": {
    "multi_match": {
      "query": "python async",
      "fields": ["title^2", "body"]
    }
  }
}

Per-query boost:

{ "bool": { "should": [
    { "match": { "title": { "query": "python", "boost": 3 } } },
    { "match": { "body":  "python" } }
] } }

Use sparingly. Excessive boosting masks fundamental relevance issues.

function_score — custom relevance

{
  "query": {
    "function_score": {
      "query": { "match": { "title": "python" } },
      "functions": [
        { "field_value_factor": { "field": "popularity", "modifier": "log1p", "missing": 1 } },
        { "gauss": { "created_at": { "origin": "now", "scale": "10d", "decay": 0.5 } } }
      ],
      "score_mode": "multiply"
    }
  }
}

Combine the base text relevance with signals like popularity score, recency decay, geo-distance decay. Production search ranking typically uses function_score.

match_phrase and slop

{ "match_phrase": { "body": "quick brown fox" } }

Requires the three terms in order, adjacent. Useful for exact phrases.

slop allows N positions of flexibility:

{ "match_phrase": { "body": { "query": "quick fox", "slop": 2 } } }

slop: 2 means up to 2 terms can intervene. “quick brown fox” matches with slop=1.

Highlighting

{
  "query": { "match": { "body": "python" } },
  "highlight": { "fields": { "body": {} } }
}

Returns matched snippets with <em> tags around terms. Tunable: pre/post tags, fragment size, number of fragments.

Pagination — and the deep-pagination gotcha

{ "from": 0, "size": 20 }                  // page 1
{ "from": 9980, "size": 20 }               // page 500 — slow!

Each shard returns top (from + size) docs to the coordinator. Page 500 means each shard returns 10,000 docs; the coordinator sorts and discards 9,980. Memory + CPU scale with the offset.

ES enforces a max_result_window of 10,000 by default. Beyond that, you must use search_after:

{ "size": 20, "sort": [{ "_score": "desc" }, { "_id": "asc" }] }
// Response includes "sort" values for the last doc.

// Next page:
{ "size": 20, "sort": ..., "search_after": [last_score, last_id] }

Stateless cursor; doesn’t hit the deep-pagination cost. Standard for “load more” / infinite-scroll UIs.

For long-running scrolls (export, reindex), use _pit (point-in-time) instead — preserves a snapshot.

Filters and caching

Filters (in bool.filter) are:

1. Non-scoring.
2. Cached at the segment level — second use is much faster.

Things to put in filter:

• term, terms, range on keyword / numeric / date fields.
• exists checks.
• Status / type / tenant predicates.

Avoid putting these in must. Both behaviors match the same docs; filter is faster and cacheable.

Common patterns

Pattern: search across multiple fields with field boosting

{
  "query": {
    "multi_match": {
      "query": "django async",
      "fields": ["title^3", "tags^2", "body"],
      "type": "best_fields"
    }
  }
}

best_fields picks the field with the best score per doc. Other modes: most_fields (sum), cross_fields (treat fields as one big field), phrase (require phrase match).

Pattern: autocomplete

{
  "query": {
    "match_phrase_prefix": {
      "name": { "query": "pyt", "max_expansions": 50 }
    }
  }
}

Matches “pyt” as a prefix of the last token. Combine with edge_ngram index analyzer for better performance.

Pattern: fuzzy / typo-tolerant search

{ "query": { "match": { "title": { "query": "djnago", "fuzziness": "AUTO" } } } }

Edit distance. AUTO = 0 for short tokens, 1-2 for longer. Useful for user search; expensive at scale.

Common gotchas

• term on a text field — almost always wrong; use match.
• Sort on a text field — fails or unpredictable. Use a .keyword sub-field.
• Deep pagination beyond max_result_window — use search_after.
• script queries — run per doc, slow. Avoid; precompute fields at index time.
• wildcard, regexp, prefix with leading wildcard — slow. Anchor: foo* ok, *foo slow.
• Default operator is OR in match. Add "operator": "and" if you want all-tokens-required.

Interview angle

• “term vs match?” — term is exact match on the indexed token (no analysis of the query). match runs the query through the field’s analyzer first. Use term for keyword fields and IDs; match for full-text fields.
• “What’s BM25?” — Elasticsearch’s default relevance scoring. TF-IDF evolution with term-frequency saturation (k1) and document-length normalization (b). Rare terms in short docs score highest.
• “must vs filter in a bool query?” — both must match. must contributes to relevance score; filter doesn’t and is cacheable. Use filter for non-scoring conditions (status, date range, tenant) — much faster.
• “How do you paginate beyond 10,000 results?” — search_after with a sort key (typically _score desc + _id for tiebreak). Stateless cursor; no deep-pagination cost. For exports, _pit (point-in-time) with search_after.
• “How do you implement autocomplete?” — edge_ngram at index time + match_phrase_prefix at query time. Or completion suggester for prefix-completion with weights.
• “How do you combine relevance with recency/popularity?” — function_score. Multiply the BM25 score by a popularity factor (field_value_factor) and a date decay (gauss on created_at). Production search ranking lives here.
• “Why is your query returning 0 hits when the docs are there?” — usually mapping (text vs keyword), analyzer mismatch (case sensitivity, stemming), or term against analyzed text. Check via _analyze API to see what tokens are generated.