RAG Architecture Patterns

RAG (Retrieval-Augmented Generation) has evolved past the textbook “retrieve top-K chunks, stuff in prompt” picture. Production RAG is a family of architectures with different trade-offs. Knowing the patterns is what differentiates “I’ve used a RAG tutorial” from “I’ve shipped RAG in production.”

For RAG basics see 01_what_is_rag.md. For embeddings see 02_what_are_embeddings.md. For vector DBs see 04_vector_databases.md. For chunking/retrieval see 05_chunking_and_retrieval.md.

The four evolutionary stages

Most production systems are at “advanced” with elements of modular. Agentic is the frontier.

Naive RAG (the baseline)

Question → Embed → Vector search → Top-K chunks → Stuff in prompt → LLM → Answer

def naive_rag(question: str, top_k: int = 5) -> str:
    query_embedding = embed_model.encode(question)
    chunks = vector_db.search(query_embedding, top_k=top_k)
    context = "\n\n".join(c.text for c in chunks)
    prompt = f"Context:\n{context}\n\nQuestion: {question}\n\nAnswer:"
    return llm.generate(prompt)

What’s wrong:

• Query-document mismatch: questions and documents have different distributions. “How do I reset my password?” doesn’t embed close to “Click Settings → Account → Security → Change Password.”
• Top-K is blunt: low K misses; high K dilutes the prompt with noise.
• No filtering: retrieved irrelevant chunks confuse the LLM.
• No reranking: vector similarity is approximate; the most relevant chunk isn’t always the highest-scored.
• Context window limits: stuffing K=20 chunks may overflow or waste tokens.
• No quality signal: you don’t know if retrieval failed; LLM may hallucinate.

Naive RAG is fine for demos. It’s not production.

Advanced RAG — pre and post retrieval optimization

                  ┌─ pre-retrieval ─┐                      ┌─ post-retrieval ─┐
Question → [query rewrite, decomposition] → Search → [rerank, filter, compress] → Stuff → LLM → Answer

Pre-retrieval techniques

Query expansion / HyDE (Hypothetical Document Embeddings): LLM generates a hypothetical answer; embed the hypothetical answer (not the question); search.

def hyde(question):
    hypo = llm.generate(f"Write a concise paragraph answering: {question}")
    hypo_embedding = embed_model.encode(hypo)
    return vector_db.search(hypo_embedding, top_k=5)

Question and document distributions get closer. Works well for FAQ-like queries.

Query decomposition: break complex questions into sub-questions.

"Compare Python and Go performance for ML workloads."
→ ["Python performance for ML",
   "Go performance for ML",
   "Python vs Go performance comparison"]

Retrieve for each; merge.

Query routing: classify the question, route to the appropriate index / strategy.

def route(question):
    category = classifier.predict(question)
    if category == "technical":
        return docs_index.search(question)
    elif category == "billing":
        return billing_kb.search(question)
    # ...

Post-retrieval techniques

Reranking: vector search gives 50 candidates; a cross-encoder (more expensive but more accurate) reranks down to top 5.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rag_with_rerank(question, top_k=5):
    candidates = vector_db.search(embed(question), top_k=50)
    pairs = [(question, c.text) for c in candidates]
    scores = reranker.predict(pairs)
    ranked = [c for c, s in sorted(zip(candidates, scores), key=lambda x: -x[1])]
    return ranked[:top_k]

Vector search is fast but uses bi-encoder embeddings (independent query/doc encoding). Cross-encoders look at the pair together; much more accurate, much slower. Hence the pattern: fast retrieval → slow rerank.

Filtering: remove obviously irrelevant chunks before sending to LLM. Score thresholds, metadata filters, dedup.

Context compression: summarize or extract only relevant parts of long chunks.

def compress_context(question, chunks):
    compressed = []
    for chunk in chunks:
        extracted = llm.generate(
            f"Extract only the parts of this text relevant to: {question}\n\n{chunk.text}",
            max_tokens=200,
        )
        if extracted.strip():
            compressed.append(extracted)
    return compressed

Trades latency (one LLM call per chunk) for cleaner context. Use sparingly.

Modular RAG

Treat RAG as a directed graph of swappable modules:

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│ Query Aug   │ →  │ Retriever   │ →  │ Reranker    │
└─────────────┘    └─────────────┘    └─────────────┘
                          │                  │
                          ↓                  ↓
                   ┌─────────────┐    ┌─────────────┐
                   │ BM25 Index  │    │ Generator   │
                   └─────────────┘    └─────────────┘

Each module is swappable. Hybrid retrieval combines dense (vector) and sparse (BM25); the fusion module merges scores. Re-rankers slot in or out. Pre-retrieval augmentation routes through different paths.

Frameworks: LlamaIndex, Haystack, DSPy. They expose this as a pipeline configuration.

# LlamaIndex example
from llama_index.core import VectorStoreIndex
from llama_index.core.retrievers import VectorIndexRetriever, BM25Retriever
from llama_index.core.query_engine import RetrieverQueryEngine

vector_retriever = VectorIndexRetriever(index=vector_index, similarity_top_k=10)
bm25_retriever = BM25Retriever.from_defaults(docstore=docstore, similarity_top_k=10)

fused_retriever = HybridRetriever(vector_retriever, bm25_retriever)
query_engine = RetrieverQueryEngine.from_args(fused_retriever, reranker=rerank_module)

Agentic RAG

LLM decides what to retrieve and when, often iteratively. The model is the planner.

LLM: "I need to look up X first."
→ Retrieves X
→ LLM: "Based on X, I also need Y."
→ Retrieves Y
→ LLM: "Now I have enough — final answer."

Built on tool-calling: the LLM emits a “search” tool call; the agent executes; result fed back; LLM continues.

def agentic_rag(question):
    messages = [
        {"role": "system", "content": "You can search the knowledge base. Use it when needed."},
        {"role": "user", "content": question},
    ]
    while True:
        response = llm.chat(messages, tools=[search_tool])
        if response.tool_calls:
            for call in response.tool_calls:
                result = vector_db.search(call.args["query"])
                messages.append({"role": "tool", "content": result, "tool_call_id": call.id})
        else:
            return response.content

Pros:

• Multi-hop reasoning (“first find X, then use X to find Y”).
• Adaptive: simple questions skip retrieval entirely; complex ones iterate.
• Self-correcting: LLM can recognize bad retrieval and search again.

Cons:

• Latency: each iteration is a round trip.
• Cost: many LLM calls + many retrievals.
• Failure modes: infinite loops, getting confused, deciding not to search when it should.

For complex reasoning use cases (research, analysis, coding agents): agentic RAG is the right tool. For simple Q&A: advanced RAG suffices.

See ../10_agents_orchestration/02_the_agent_loop.md.

Specialized patterns

Self-RAG

LLM evaluates its own retrievals: “Is this relevant? Is my answer grounded?” Built-in self-correction.

Retrieve → LLM: "Are these chunks relevant?" → if no, retrieve again
LLM generates → LLM: "Is my answer supported by the chunks?" → if no, regenerate

Reduces hallucination but adds 2-3× LLM calls.

Corrective RAG (CRAG)

A retrieval evaluator scores chunks. If low confidence, fall back to web search or a different index.

def crag(question):
    chunks = vector_db.search(embed(question))
    scores = evaluator.predict(question, chunks)
    if max(scores) < THRESHOLD:
        chunks = web_search(question)
    return generate(question, chunks)

Multi-vector retrieval

Store multiple embeddings per document:

• Summary embedding (for high-level matching).
• Hypothetical question embeddings (for FAQ-like queries).
• Chunk embeddings (for fine-grained retrieval).

Different retrieval strategies hit different embedding indices.

Parent-document retrieval

Embed small chunks (precise matching) but return their larger parent documents (full context):

Embed: each paragraph
Retrieve: top-K paragraphs by similarity
Return: the parent documents (or sections) of those paragraphs

Best of both worlds: precise matching + complete context.

Graph-augmented RAG

For knowledge graph data, retrieve related entities and their relationships, not just text chunks.

Question: "Who manages projects that use Python?"
→ Identify entities (people, projects, technologies)
→ Traverse graph (people MANAGES project USES python)
→ Synthesize result

Used for compliance, recommendation systems, structured Q&A.

Production architecture (typical stack)

User question
    ↓
┌─────────────────┐
│ Query rewriting │ → optional HyDE / decomposition
└─────────────────┘
    ↓
┌─────────────────┐
│ Hybrid retrieval│ → dense (vector) + sparse (BM25), fuse with RRF
│  - vector store │
│  - BM25 index   │
└─────────────────┘
    ↓
┌─────────────────┐
│ Reranker        │ → cross-encoder, top 50 → top 5
└─────────────────┘
    ↓
┌─────────────────┐
│ Filter / dedup  │ → score threshold, metadata filters
└─────────────────┘
    ↓
┌─────────────────┐
│ Compress        │ → only if context too long
└─────────────────┘
    ↓
┌─────────────────┐
│ LLM             │ → prompt with retrieved context + instructions
└─────────────────┘
    ↓
┌─────────────────┐
│ Citation check  │ → optional: verify claims trace to chunks
└─────────────────┘
    ↓
Answer + sources

Most production RAG systems are some subset of this. Start naive, add components as needed.

Failure modes

Cost vs quality knobs

You’re always trading these:

Real systems benchmark each knob against held-out queries; tune for the cost/quality target.

Common interview confusions

• “RAG and fine-tuning are alternatives.” — not quite. RAG injects facts; fine-tuning teaches style or behavior. Often combined: fine-tune for tone, RAG for knowledge.
• “More chunks = better.” — past ~10 chunks, LLMs start ignoring middle content. Reranking + small top-K beats stuffing 50 chunks.
• “Vector search is all you need.” — pure dense retrieval misses keyword-heavy queries. Hybrid (dense + BM25) catches both.

Interview angle

• “What are the RAG architecture patterns?” — four evolutionary stages: naive (retrieve-then-generate), advanced (pre/post retrieval optimization), modular (swappable components in a pipeline), agentic (LLM decides when/what to retrieve, iteratively).
• “What’s wrong with naive RAG and how do you fix it?” — query-document mismatch (HyDE), bad ranking (rerankers), noise in context (filtering + compression), missing recall (hybrid retrieval), no quality signal (eval + grounding checks).
• “What’s HyDE?” — Hypothetical Document Embeddings. LLM generates a hypothetical answer to the question; you embed and search using the hypothetical answer. Question and document distributions get closer; better retrieval.
• “What’s reranking and why use it?” — fast vector search returns 50+ candidates; an expensive cross-encoder rescores them; top 5 go to the LLM. Cross-encoders see the (query, document) pair together — much more accurate than independent embeddings.
• “What’s agentic RAG?” — LLM uses retrieval as a tool, calling it iteratively when needed. Supports multi-hop reasoning, adaptive retrieval (skip if not needed), self-correction. Slower and more expensive but handles complex queries.
• “What’s the ‘lost in the middle’ problem?” — LLMs disproportionately ignore information in the middle of long contexts. Mitigate by reranking and placing the most important chunks at the start or end of the prompt.
• “Hybrid retrieval — what and why?” — combine dense (vector) and sparse (BM25/keyword) retrieval. Dense catches semantic similarity; sparse catches exact terms. Fuse with Reciprocal Rank Fusion (RRF). Better recall than either alone.
• “How do you reduce hallucinations in RAG?” — instruct LLM to cite sources, require citations to be from provided context, add a verification step (LLM checks each claim against chunks), use self-RAG to evaluate own answer, fall back to “I don’t know” if confidence is low.