Elasticsearch — Vector Search and kNN

Since 8.x, Elasticsearch supports dense vector search via HNSW. Same engine for full-text + semantic search; competing directly with Pinecone, Weaviate, Qdrant.

dense_vector mapping

PUT /docs
{
  "mappings": {
    "properties": {
      "content":   { "type": "text" },
      "embedding": {
        "type": "dense_vector",
        "dims": 1536,
        "index": true,
        "similarity": "cosine"
      }
    }
  }
}

Similarity options: cosine, dot_product, l2_norm, max_inner_product.

For most LLM embeddings (OpenAI, Cohere, etc.), cosine is the right pick. Normalized vectors give the same ranking with dot_product (faster).

Indexing

es.index(index="docs", document={
    "content": "Python is a high-level programming language.",
    "embedding": embed("Python is a high-level programming language.")
})

The embedding is computed by your app (OpenAI, sentence-transformers, AWS Bedrock); Elasticsearch stores and indexes it.

kNN query

{
  "knn": {
    "field": "embedding",
    "query_vector": [0.12, 0.03, ...],
    "k": 10,
    "num_candidates": 100
  }
}

Returns the top 10 most similar docs.

• k — number of results to return.
• num_candidates — how many candidates to consider per shard before returning top-k. Higher = better recall, more compute. Typical: 100-200 for k=10.

HNSW under the hood

Hierarchical Navigable Small World — the approximate nearest neighbor algorithm Elasticsearch uses (via Lucene). Builds a multi-layer graph; queries traverse top layers (sparse) then refine in lower layers.

Trade-offs:

• Approximate. Doesn’t guarantee true k-nearest; recall ~98-99% at default settings.
• High memory cost. All vectors must be in memory for fast queries.
• Construction is expensive. Indexing 100M vectors takes hours.

Tuning:

• m (default 16) — number of edges per node in graph. Higher = better recall, more memory.
• ef_construction (default 100) — quality of graph build. Higher = better recall, slower build.
• num_candidates at query time — controls speed/recall trade-off.

Hybrid search — dense + sparse

The killer feature: combine BM25 lexical search with vector semantic search in one query.

{
  "query": { "match": { "content": "python programming" } },
  "knn": {
    "field": "embedding",
    "query_vector": [...],
    "k": 50,
    "num_candidates": 200
  },
  "size": 20,
  "rank": {
    "rrf": { "window_size": 50, "rank_constant": 60 }
  }
}

rank.rrf (Reciprocal Rank Fusion, 8.8+) combines rankings from both: docs ranked highly by either method bubble up. Default fusion that doesn’t require tuning relative weights.

For weighted combination:

{
  "knn": { ..., "boost": 0.5 },
  "query": { "match": { ..., "boost": 1.0 } }
}

Add scores directly. Tuning the boosts is empirical.

Filtering vector search

{
  "knn": {
    "field": "embedding",
    "query_vector": [...],
    "k": 10,
    "num_candidates": 100,
    "filter": { "term": { "tenant_id": "acme" } }
  }
}

Filter pre-applied — graph traversal only considers matching docs. Critical for multi-tenant systems; without filtering, you’d return another tenant’s vectors.

Caveat: aggressive filters that exclude most vectors can hurt recall (HNSW can’t find k neighbors within the filtered subset). For tight filters, increase num_candidates or shard by tenant.

Quantization — save memory

"embedding": {
  "type": "dense_vector",
  "dims": 1536,
  "index": true,
  "similarity": "cosine",
  "index_options": { "type": "int8_hnsw" }
}

int8_hnsw quantizes float32 → int8: 4× memory reduction, minimal recall loss (~1%). Almost always worth enabling for production.

Newer options: bbq_hnsw (1-bit binary quantization) for even smaller memory, larger recall impact (~5-10%); use for big-scale + cost-sensitive workloads.

When ES vs dedicated vector DB

If you already run Elasticsearch: vector search there is the lowest-friction path. If starting from scratch for pure vector workloads at extreme scale, dedicated vector DBs may win on cost.

Embedding storage cost

A dims: 1536 float32 vector is 1536 × 4 = 6144 bytes ≈ 6 KB per doc. With HNSW graph overhead, ~10 KB per doc effective. 10 million docs = ~100 GB just for vectors.

int8 quantization brings this down 4×. bbq (binary) brings it down 32×.

Caveats

• First-class kNN only since 8.0 — older 7.x had dense_vector but no HNSW.
• No multi-vector per field in standard form. Workarounds: nested vectors, separate indices.
• Reindex required to change vector dims. Plan dimensions carefully (don’t change embedding model mid-flight without reindex).
• Approximate. For correctness-critical use cases, exact kNN exists but doesn’t scale.
• No vector arithmetic at query time (yet) for things like Maximal Marginal Relevance (MMR diversification) — implement in your app.

RAG pattern

The standard retrieval pipeline:

def rag_search(query: str, k: int = 5):
    query_embedding = embed(query)

    response = es.search(index="kb", body={
        "knn": {
            "field": "embedding",
            "query_vector": query_embedding,
            "k": k * 4,                  # over-fetch for reranking
            "num_candidates": 200,
            "filter": {"term": {"tenant_id": tenant}}
        },
        "query": {"match": {"content": query}},
        "rank": {"rrf": {}},
        "size": k * 4,
    })

    candidates = [hit["_source"]["content"] for hit in response["hits"]["hits"]]
    reranked = cross_encoder.rerank(query, candidates)[:k]
    return reranked

Hybrid retrieval → cross-encoder reranking → final top-k. Cross-encoder reranking adds ~50-100ms but typically lifts MRR substantially.

Cost / scale considerations

• In-memory. HNSW vectors are kept in RAM for fast queries. 100M docs at 1536 dims → ~100 GB RAM (or 25 GB with int8 quantization).
• Slow indexing. Building the HNSW graph is expensive. Plan for bulk loads via reindex.
• Disk too. Vectors are also stored on disk, doubling the storage cost.
• No quantization for similarity types other than cosine/dot_product (as of recent versions). Check current docs.

Interview angle

• “How does Elasticsearch do vector search?” — dense_vector field type with HNSW index (Lucene 9.x+). At query time, knn clause traverses the graph; num_candidates controls speed/recall. Approximate nearest-neighbor — ~98-99% recall at defaults.
• “What’s hybrid search?” — combining lexical (BM25 match) and semantic (vector knn) retrieval. Elasticsearch 8.8+ supports it natively with RRF (Reciprocal Rank Fusion) to combine rankings. Best results in production RAG systems.
• “How would you filter vector search by tenant?” — knn.filter clause; HNSW traversal restricted to matching docs. Multi-tenant search requires this; without it, you leak vectors across tenants.
• “Memory budget for 10M vectors at 1536 dims?” — float32 = ~100 GB. With int8 quantization (int8_hnsw), ~25 GB at ~1% recall cost. For very large fleets, bbq (binary) brings further reductions at higher recall cost.
• “Elasticsearch vs Pinecone for vector search?” — ES wins if you already have ES infra or need mixed lexical+vector. Dedicated vector DBs (Pinecone, Qdrant) win at pure vector scale with vector-native features. For most RAG apps, ES is sufficient.
• “What’s reciprocal rank fusion?” — rrf_score(doc) = Σ 1/(k + rank_i(doc)) across multiple rankings. Combines lexical and vector rankings without needing to tune relative weights. Default fusion method in modern hybrid search.
• “When would you reach for exact kNN over HNSW?” — when correctness matters (recall = 100%). Doesn’t scale past ~100k vectors. Niche use cases (small reference corpus, regulated workloads).