Agentic RAG and evaluating retrieval

Two things that separate a RAG prototype from a system you’d defend in production: letting the model participate in retrieval, and being able to say how well retrieval works.

Agentic RAG

Classic RAG is one shot: embed the query, retrieve, stuff, generate. It fails when the first retrieval isn’t good enough and there’s no second attempt.

Agentic RAG makes retrieval a tool the model can call repeatedly, judging results and adapting.

@tool
def search_docs(query: str, source: Literal["policy", "technical", "faq"] | None = None) -> str:
    """Search internal documentation. Narrow with `source` when you know the area.
    Returns the top 5 passages with citations."""
    return format_results(hybrid_search(query, filter={"source": source}))

The agent can now reformulate after weak results, decompose a multi-part question into separate searches, search different sources for different sub-questions, and stop when it has enough.

The trade is determinism for capability, the same trade as agents generally. See ../10_agents_orchestration/01_what_is_an_agent.md.

Use classic RAG for straightforward lookup — it’s cheaper, faster and predictable. Use agentic RAG for multi-hop questions, ambiguous queries, or when several sources must be consulted and combined.

Corrective and self-reflective variants

Middle ground between the two, and cheaper than a full agent loop:

• Grade the retrieval. A cheap classifier or model call scores whether retrieved chunks actually answer the question; if not, re-query or fall back to web search.
• Grade the answer. Check the generated answer is supported by the retrieved context before returning it. Catches hallucination at the point it happens.

Both add one cheap call and remove the worst failure — confidently answering from irrelevant context.

Evaluating retrieval

You cannot improve what you don’t measure, and most RAG systems are tuned by vibes.

Build a golden set

~100 queries with known-correct chunks. That’s the minimum viable eval and it’s achievable in a day.

Sources: real user queries from logs (best), questions generated from your documents by an LLM then human-checked, and deliberately hard cases — ambiguous, multi-hop, exact-code lookups, questions with no answer in the corpus.

That last category matters: include queries the corpus cannot answer, so you can measure whether the system says “I don’t know” rather than confabulating.

Retrieval metrics

def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    return len(set(retrieved[:k]) & relevant) / len(relevant)

def mrr(retrieved: list[str], relevant: set[str]) -> float:
    for rank, doc in enumerate(retrieved, 1):
        if doc in relevant:
            return 1.0 / rank
    return 0.0

Measure recall@50 before and after the reranker separately. If recall@50 is low, reranking cannot fix it — the problem is retrieval, and you’re tuning the wrong stage. This diagnostic split is the practically useful part.

End-to-end metrics

Retrieval quality isn’t answer quality. The dimensions that matter:

RAGAS is the common framework implementing these with LLM-as-judge. Useful, with the standard caveat: a judge is a model with its own biases, so validate it against human labels on a sample before trusting it at scale. More in ../13_evaluation/.

Faithfulness is the one to prioritise. A fluent answer unsupported by the context is the failure mode that damages trust, and it’s directly measurable: decompose the answer into claims and check each against the retrieved passages.

Regression testing

Treat the golden set as a test suite. Run it on every change — new embedding model, different chunking, prompt edit, model upgrade — and fail CI on regression.

recall@10:      0.82  (baseline 0.80)  PASS
faithfulness:   0.91  (baseline 0.93)  FAIL - investigate
cost/query:     $0.004 (budget $0.005) PASS

Including cost and latency as tested properties catches regressions no accuracy metric would.

Common failures

Interview angle

• “What is agentic RAG?” — retrieval becomes a tool the model calls repeatedly, so it can reformulate after weak results, decompose multi-hop questions and consult several sources. It trades determinism, cost and latency for capability.
• “How do you evaluate a RAG system?” — a golden set of ~100 queries with known-correct chunks, measured on recall@k and MRR for retrieval, plus faithfulness, answer relevance and context precision end-to-end. Include queries the corpus can’t answer, to test refusal.
• “Your RAG answers are wrong. Where do you look?” — split the stages. Measure recall@50 first: if the correct chunk isn’t in the candidate set, the problem is retrieval and no amount of reranking or prompting fixes it. If it is retrieved but unused, look at context ordering and how many chunks you’re passing.
• “What’s the most important quality metric?” — faithfulness. A fluent answer unsupported by retrieved context is the failure that destroys trust, and it’s measurable by decomposing the answer into claims and checking each against the passages.
• “How do you stop RAG changes regressing quality?” — run the golden set as a CI suite on every change to chunking, embeddings, prompts or model, with cost and latency as tested properties alongside accuracy.
• “Classic or agentic RAG for a documentation chatbot?” — start classic: cheaper, faster, predictable, and adequate for direct lookup. Add corrective grading of retrieval before jumping to a full agent loop, and reserve agentic retrieval for genuinely multi-hop questions.