Hybrid search and reranking

The two upgrades that take a RAG demo to production. As of 2026 the consensus is clear: hybrid retrieval is the minimum viable baseline, and a two-stage pipeline of hybrid retrieval plus neural reranking outperforms any single-stage method by a wide margin.

Why pure vector search isn’t enough

Dense embeddings capture meaning and lose precision on exact tokens.

BM25 excels at exact-match queries — product codes, named entities, rare technical terms — and cannot handle paraphrase. Dense retrieval handles concepts and paraphrase and underweights exact rare-term matches.

Neither is sufficient alone, and the failure cases are complementary. That’s the whole argument for hybrid.

Fusing the two

The problem: BM25 scores and cosine similarities are on incomparable scales, and normalising them is fragile.

Reciprocal Rank Fusion sidesteps it entirely by using only ranks:

def rrf(rankings: list[list[str]], k: int = 60) -> dict[str, float]:
    """Fuse ranked lists by reciprocal rank. Score-scale independent."""
    scores = collections.defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return dict(sorted(scores.items(), key=lambda kv: -kv[1]))

fused = rrf([bm25_search(query, top=50), vector_search(query, top=50)])

RRF is the right default precisely because it avoids score calibration between the two systems. k=60 is the conventional constant; it damps the influence of top ranks slightly so a single system can’t dominate.

The alternative — weighted score fusion after normalisation — can perform better when tuned, and needs retuning whenever either retriever changes. Start with RRF.

Most vector databases now support hybrid natively; Postgres does it with pgvector plus tsvector in one query, which is often the simplest production answer for teams already on Postgres.

Reranking

Retrieval optimises for recall over a large corpus, cheaply. Reranking optimises for precision over a small candidate set, expensively. Retrieval gets you candidates; reranking decides which deserve context-window space.

candidates = hybrid_search(query, top_k=50)      # cheap, recall-oriented
reranked = cross_encoder.rank(query, candidates) # expensive, precision-oriented
context = reranked[:5]                            # what the model actually sees

Bi-encoder vs cross-encoder

The distinction that gets asked:

A cross-encoder sees the query and document in the same forward pass, so attention can relate specific query terms to specific document spans. A bi-encoder must compress each document into a single vector before it knows the query — that’s the information loss reranking recovers.

The cost is why it’s a second stage: you cannot cross-encode a million documents per query, but 50 is trivial.

Reranking is usually the single highest-return improvement to a mediocre RAG system, ahead of better chunking or a better embedding model.

Choosing candidate count

Retrieve 30-100 for reranking, keep 3-10 for context. Tune the first number by measuring recall@k — if the correct chunk is rarely in the top 50, reranking can’t save you and the retrieval stage is the problem.

Keeping too many after reranking actively hurts: extra chunks dilute attention and push relevant content into the weak middle of the context. See ../06_transformers_llm/08_context_windows.md.

Contextual retrieval

A chunk stripped of its document loses meaning. “The policy applies for 30 days” — which policy?

Contextual chunking prepends a short generated description of where the chunk sits before embedding it:

Context: From the 2026 Refund Policy, section on digital goods.
Chunk:   The policy applies for 30 days from purchase.

Both the embedding and the BM25 index see the enriched text. It costs one cheap LLM call per chunk at ingest — one-time, offline, and prefix caching makes it inexpensive since the document is a shared prefix across its chunks.

This addresses the most common RAG failure: retrieval finds a chunk that is topically right and contextually meaningless.

Query-side techniques

Query rewriting is the one that matters most in practice, because multi-turn chat produces queries that are meaningless standalone. Resolving “what about the second one?” against conversation history before retrieval is often a bigger win than any index tuning.

Metadata filtering

Filter before you rank. If the user may only see their own tenant’s documents, that’s a hard constraint, not a scoring signal.

results = collection.query(
    query_embedding=emb,
    filter={"tenant_id": user.tenant_id, "status": "published"},
    top_k=50,
)

Two notes: pre-filtering can degrade ANN index performance (the index isn’t built around your filter), and tenant isolation must be enforced at the query layer, never by hoping the model ignores irrelevant results. Retrieval leaking across tenants is a security incident, not a relevance bug.

The pipeline, assembled

query
  -> rewrite (resolve context, expand)
  -> hybrid retrieve: BM25 (top 50) + vector (top 50)
  -> RRF fuse
  -> metadata filter (hard constraints)
  -> cross-encoder rerank
  -> top 3-10 into context, most relevant at the extremes
  -> generate with citations

Build it in that order and measure at each stage. Adding a reranker before you know your recall@50 is guessing.

Interview angle

• “Why hybrid search rather than pure vector search?” — dense retrieval handles paraphrase and concepts but underweights exact rare tokens; BM25 nails product codes, error codes and proper nouns and misses synonyms. Their failure cases are complementary, which is why hybrid is the 2026 baseline.
• “How do you combine BM25 and vector scores?” — Reciprocal Rank Fusion, because it uses ranks only and avoids calibrating two incomparable score scales. Weighted fusion can beat it when tuned, and needs retuning whenever either retriever changes.
• “Bi-encoder vs cross-encoder?” — a bi-encoder embeds query and document independently so documents can be indexed offline and searched at scale; a cross-encoder processes them together so attention relates query terms to document spans, which is far more accurate but costs a forward pass per candidate. Hence retrieve with one, rerank with the other.
• “Your RAG system retrieves topically relevant but useless chunks. Fix?” — contextual retrieval: prepend a generated summary of the chunk’s place in its document before embedding, so a chunk saying “the policy applies for 30 days” carries which policy.
• “Biggest single improvement to a mediocre RAG system?” — usually adding a reranker, ahead of chunking changes or a new embedding model. Verify recall@50 first; if the right chunk isn’t in the candidate set, reranking can’t help.
• “How do you handle multi-tenant retrieval?” — hard metadata filters at the query layer. Tenant isolation is an authorization boundary enforced in the query, not a relevance signal you hope the ranker respects.