Chunking Strategies and Retrieval Techniques

Two of the biggest quality levers in RAG. Chunking: how you split documents before embedding. Retrieval: how you find the right chunks at query time. Bad chunking creates retrievable but unhelpful chunks; bad retrieval misses the chunks that exist.

For RAG architecture see 03_rag_architecture_patterns.md. For vector DBs see 04_vector_databases.md.

Why chunking matters

LLMs have context limits and the “lost in the middle” problem. You can’t dump a 200-page document into a prompt. So you split — but how?

The trade-off:

• Too small (50 tokens): fragments don’t contain enough context to answer questions.
• Too large (2000 tokens): contains too much; vector embeds become “vague averages”; retrieval surfaces mostly-irrelevant content.

Sweet spot: typically 200-800 tokens for most use cases. But the right answer depends on document type and query patterns.

Chunking strategies

Fixed-size chunks (with overlap)

Split into fixed-token-count chunks with some overlap between adjacent chunks.

def fixed_size_chunks(text, chunk_size=500, overlap=50):
    tokens = tokenizer.encode(text)
    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + chunk_size, len(tokens))
        chunks.append(tokenizer.decode(tokens[start:end]))
        start += chunk_size - overlap
    return chunks

Pros: simple, predictable size. Cons: splits mid-sentence, mid-paragraph; breaks logical units.

Use for: simple docs, prototypes. The default in many tutorials.

Recursive character splitting

Split on a hierarchy of separators: paragraphs first, then sentences, then characters as last resort. Each chunk respects boundaries when possible.

# LangChain's RecursiveCharacterTextSplitter
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(document)

Tries \n\n first (paragraphs). If a paragraph is too long, falls back to \n. Then sentences. Then individual characters.

Pros: respects logical boundaries when possible; simple to configure. Cons: still fixed-ish size; can split semantic units when paragraphs are long.

Use for: most general-purpose text.

Semantic chunking

Split based on semantic similarity between sentences. Adjacent sentences that talk about the same topic stay together.

def semantic_chunks(text, similarity_threshold=0.7):
    sentences = sent_tokenize(text)
    embeddings = embed_model.encode(sentences)
    chunks = [[sentences[0]]]
    for i in range(1, len(sentences)):
        sim = cosine(embeddings[i-1], embeddings[i])
        if sim >= similarity_threshold:
            chunks[-1].append(sentences[i])
        else:
            chunks.append([sentences[i]])
    return [" ".join(c) for c in chunks]

Pros: chunks are semantically coherent units. Cons: chunk sizes vary wildly; computational overhead for embedding every sentence.

Use for: when document quality is heterogeneous and naive splitting produces incoherent chunks.

Document-aware chunking

Use the document’s structure: respect headings, sections, code blocks, tables.

# Markdown-aware
from langchain.text_splitter import MarkdownHeaderTextSplitter

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[
        ("#", "h1"),
        ("##", "h2"),
        ("###", "h3"),
    ]
)
chunks = splitter.split_text(markdown_doc)
# Each chunk preserves heading hierarchy as metadata

For PDFs: use OCR + layout parsing (e.g., Unstructured, LlamaParse) to detect sections, tables, images.

For code: use the language’s AST. Don’t split mid-function; chunk by function/class.

Pros: chunks are natural units of the document. Cons: requires document-specific tooling.

Use for: structured documents (markdown docs, code, technical PDFs, HTML).

Sliding window with context

Each chunk includes a summary of the surrounding context:

Chunk N body: <500 tokens of actual content>
Chunk N metadata: {
    "title": "Section 3.2 — Authentication",
    "previous_summary": "Section 3.1 covered authorization basics...",
    "section_path": "Chapter 3 → 3.2"
}

Stored as metadata; can be added to the prompt for context without bloating the embedding.

Parent-document retrieval

Embed small chunks for precise retrieval; return their larger parent documents for full context.

# Index small chunks but link to parent
def index_with_parents(docs):
    for doc in docs:
        parent_id = uuid()
        store_parent(parent_id, doc.text)
        for chunk in split(doc, chunk_size=200):
            vector_db.add(embed(chunk), metadata={"parent_id": parent_id})

# At query time
def retrieve(query, top_k=3):
    chunk_results = vector_db.search(embed(query), top_k=top_k)
    parents = {r["parent_id"] for r in chunk_results}
    return [load_parent(pid) for pid in parents]

Best of both worlds: precise matching (small chunks) + complete context (parent documents).

Hierarchical / summary indexing

Multiple indexes at different granularities:

Level 1: document summaries (one per doc)
Level 2: section summaries (one per section)
Level 3: paragraph chunks (the actual text)

Query first hits summaries (top-level routing), then drills down to chunks. Used for very large corpora.

Choosing chunk size

Always measure. Build a held-out evaluation set; vary chunk size; check retrieval recall and answer quality.

Overlap — why and how much

Overlap = some content shared between adjacent chunks. Helps when an important fact spans a chunk boundary.

chunk_size=500, overlap=50
→ ~10% of content duplicated

Trade-off:

• More overlap: better recall when content crosses boundaries; more storage; more redundant chunks in top-K.
• Less overlap: cleaner index; risk of missing border-spanning answers.

Rule of thumb: 10-20% overlap. Higher for technical docs where exact wording matters; lower for narrative content.

Retrieval techniques

Dense retrieval (vector similarity)

The “RAG default.” Embed query and chunks; find nearest by cosine similarity.

def dense_retrieve(query, top_k=5):
    q_emb = embed_model.encode(query)
    return vector_db.search(q_emb, top_k=top_k)

Pros: catches semantic similarity (“password reset” matches “how to change password”). Cons: misses exact terms, code, names, IDs, very specific jargon.

Sparse retrieval (BM25)

Classic information retrieval: TF-IDF / BM25 keyword matching.

from rank_bm25 import BM25Okapi

bm25 = BM25Okapi([doc.split() for doc in corpus])

def sparse_retrieve(query, top_k=5):
    tokenized = query.split()
    scores = bm25.get_scores(tokenized)
    return top_k_by_score(corpus, scores, top_k)

Pros: catches exact terms, codes, names. No embedding needed. Cons: misses paraphrases (“car” doesn’t match “vehicle”).

Hybrid retrieval

Combine both. Dense + sparse + fusion.

def hybrid_retrieve(query, top_k=5):
    dense = dense_retrieve(query, top_k=20)
    sparse = sparse_retrieve(query, top_k=20)
    return reciprocal_rank_fusion(dense, sparse, top_k=top_k)

Reciprocal Rank Fusion (RRF): a robust fusion algorithm.

def rrf(dense_results, sparse_results, k=60, top_n=10):
    scores = {}
    for rank, doc in enumerate(dense_results):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank)
    for rank, doc in enumerate(sparse_results):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda x: -x[1])[:top_n]

k=60 is the standard hyperparameter. RRF doesn’t care about score scales — works across retrievers that produce different score distributions.

Most production RAG systems use hybrid retrieval. Pure dense or pure sparse are bested by hybrid almost always.

Multi-query retrieval

Generate variations of the query; retrieve for each; merge.

def multi_query_retrieve(question, top_k=5):
    variations = llm.generate(
        f"Generate 3 different ways to phrase this question:\n{question}",
        max_tokens=200,
    ).split("\n")
    all_results = []
    for variation in [question, *variations]:
        all_results.extend(vector_db.search(embed(variation), top_k=10))
    return dedupe_and_rank(all_results)[:top_k]

Catches more relevant chunks for ambiguous questions. Costs N× LLM calls + N× retrievals.

Reranking

Cheap fast retrieval returns 50 candidates; expensive accurate reranker scores top 5.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def retrieve_with_rerank(query, top_k=5):
    candidates = vector_db.search(embed(query), top_k=50)
    pairs = [(query, c.text) for c in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
    return [c for c, _ in ranked[:top_k]]

Cross-encoders compute query-document attention together; vector embeddings encode independently. Much higher accuracy at the cost of speed. Hence the pattern: fast retrieval → slow rerank.

Common rerankers:

• Cohere Rerank (API).
• BGE Rerank (open source, sentence-transformers).
• Cross-encoders from MS-MARCO collection.

Metadata filtering

Most vector DBs support metadata filters alongside vector search:

results = vector_db.search(
    query_embedding,
    top_k=10,
    filter={
        "must": [
            {"key": "user_id", "match": {"value": user.id}},     # multi-tenancy
            {"key": "lang", "match": {"value": "en"}},
            {"key": "published_at", "range": {"gte": "2024-01-01"}},
        ]
    }
)

Use for: tenant isolation, language, recency, document type, permissions.

Time-aware retrieval

For knowledge bases where recency matters:

def time_weighted_retrieve(query, top_k=5, decay_days=30):
    candidates = vector_db.search(embed(query), top_k=50)
    now = datetime.now()
    for c in candidates:
        age = (now - c.published_at).days
        recency_score = math.exp(-age / decay_days)
        c.score = c.score * recency_score
    return sorted(candidates, key=lambda c: -c.score)[:top_k]

For news, support tickets, documentation: recent chunks often more relevant.

Pre-retrieval query optimization

Query rewriting

LLM transforms the query before embedding:

Original:  "How fix that?"  (no context)
Rewritten: "How do I fix the connection timeout error in PostgreSQL?"

Especially useful in chat — pull context from conversation history.

def rewrite_query(history, current_question):
    prompt = f"Given the conversation:\n{history}\n\nRewrite this question to be standalone:\n{current_question}"
    return llm.generate(prompt, max_tokens=100)

HyDE (Hypothetical Document Embeddings)

Generate a hypothetical answer; embed the answer (not the question).

def hyde(question, top_k=5):
    hypothetical = llm.generate(f"Write a paragraph answering: {question}")
    return vector_db.search(embed(hypothetical), top_k=top_k)

Question and document distributions get closer. Works well for FAQ-like queries; hurts when the question and answer are stylistically very different.

Query decomposition

For complex questions, break into sub-queries.

def decompose(question):
    return llm.generate(
        f"Break this question into 2-4 simpler sub-questions:\n{question}",
        max_tokens=200,
    )

def multi_hop_rag(question):
    sub_questions = decompose(question)
    all_chunks = []
    for sq in sub_questions:
        all_chunks.extend(retrieve(sq))
    return llm.answer(question, dedupe(all_chunks))

Evaluation

Measure retrieval quality on a held-out set:

For RAG end-to-end:

Tools: RAGAS, TruLens, LangSmith, custom evals. Always have an eval set; tune knobs against it.

Common pitfalls

• Default chunk_size=1000 without measuring: copied from a tutorial. Build an eval set; vary chunk size.
• No overlap: borderline content gets cut; recall drops.
• Chunking code as text: function bodies get split mid-statement. Use language-aware splitters.
• Embedding the wrong text: chunk metadata (titles, headers) is more discriminative than body text — sometimes include it in the embedded text.
• No metadata: can’t filter; can’t trace back to source.
• Pure dense or pure sparse retrieval: hybrid almost always wins.
• No reranker: top-5 vector results often include noise; cross-encoder rerank dramatically improves.
• Ignoring lost-in-the-middle: top chunk is at position 5; LLM ignores it. Place top results at start/end of prompt.

Common interview confusions

• “Smaller chunks = better.” — not always. Too small loses context; relevant info is split across chunks not retrieved together. Sweet spot is workload-dependent.
• “Vector search is enough.” — pure dense retrieval misses exact-term queries (codes, names). Hybrid is the production default.
• “Reranking is optional.” — for any quality-critical system, reranking is the cheap big-win — second-most important after good chunking.

Interview angle

• “What chunking strategies have you used?” — fixed-size with overlap (baseline), recursive character splitting (respect paragraph/sentence boundaries), semantic chunking (group by sentence similarity), document-aware (use markdown headers, code structure), parent-document retrieval (embed small, return big).
• “How do you pick chunk size?” — measure. Build a held-out evaluation set; vary chunk size and overlap; track retrieval recall and answer quality. Defaults (300-800 tokens) are starting points, not answers.
• “What’s hybrid retrieval and why?” — combine dense (vector) and sparse (BM25). Dense catches paraphrases; sparse catches exact terms. Fuse with Reciprocal Rank Fusion. Better than either alone.
• “What’s HyDE?” — Hypothetical Document Embeddings. LLM generates a hypothetical answer; embed and search using that. Closes the question-document distribution gap. Common for FAQ-like queries.
• “What’s a cross-encoder reranker?” — model that takes (query, document) pairs and scores them together (unlike bi-encoders which encode each independently). Much more accurate; much slower. Pattern: fast retrieval (50 candidates) → cross-encoder rerank (top 5).
• “How do you handle multi-tenant RAG?” — metadata filter on every query (user_id, tenant_id). Centralize the filter in a wrapper so individual queries can’t bypass it.
• “How do you reduce ‘lost in the middle’ issues?” — rerank to get the most relevant chunks at positions 1 and N; reorder retrieved chunks so important info is at start/end of context.
• “What’s recall@K and why monitor it?” — % of truly relevant documents found in top-K results. Primary retrieval quality metric. Drops in recall@K predict drops in RAG answer quality.
• “How would you debug poor RAG answers?” — log retrieval results, check if the right chunks were retrieved (if not, retrieval problem: chunking, embeddings, query rewriting); if yes but answer is wrong (LLM problem: prompt, instructions, citations).