What is RAG?

Definition

RAG (Retrieval-Augmented Generation) is a technique that combines information retrieval with large language model (LLM) generation to improve the accuracy and relevance of AI-generated responses. Instead of relying solely on the LLM’s pre-trained knowledge, RAG retrieves relevant information from external knowledge bases and uses it as context for generation.

Key Concepts

Core Components

Retrieval System: Searches and retrieves relevant documents/information from a knowledge base
Vector Database: Stores document embeddings for semantic search
Embedding Model: Converts text into vector representations
LLM (Large Language Model): Generates responses based on retrieved context
Knowledge Base: Collection of documents, articles, or data sources

Architecture Flow

User Query
    ↓
Query Embedding
    ↓
Vector Search (Retrieval)
    ↓
Relevant Documents
    ↓
Context + Query → LLM
    ↓
Generated Response

How RAG Works

Step-by-Step Process

Query Processing: User query is converted to an embedding vector
Retrieval: Similar documents are retrieved from the knowledge base using vector similarity
Context Assembly: Retrieved documents are combined with the original query
Generation: LLM generates a response using the retrieved context
Response: Final answer is returned to the user

Basic Example

from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.llms import OpenAI
from langchain.chains import RetrievalQA

# Initialize components
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(documents, embeddings)
llm = OpenAI(temperature=0)

# Create RAG chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever()
)

# Query
response = qa_chain.run("What is machine learning?")

Why RAG is Important

Problems RAG Solves

Hallucination: LLMs sometimes generate incorrect or made-up information
Outdated Knowledge: LLMs have training cutoffs and can’t access recent information
Domain-Specific Knowledge: LLMs may lack specialized domain knowledge
Source Attribution: RAG can cite sources for generated information
Cost Efficiency: Reduces need for fine-tuning on domain-specific data

Benefits

Accuracy: Responses are grounded in retrieved documents
Up-to-date: Can use recent information not in training data
Transparency: Can show source documents
Flexibility: Easy to update knowledge base without retraining
Domain Adaptation: Works with any domain-specific knowledge

RAG Architecture Types

1. Naive RAG (Basic RAG)

def naive_rag(query, knowledge_base, llm):
    # 1. Retrieve relevant documents
    relevant_docs = retrieve_documents(query, knowledge_base, top_k=5)
    
    # 2. Combine context
    context = "\n\n".join([doc.content for doc in relevant_docs])
    prompt = f"Context: {context}\n\nQuestion: {query}\n\nAnswer:"
    
    # 3. Generate response
    response = llm.generate(prompt)
    return response

Characteristics:

Simple retrieval and generation
No query optimization
Basic context assembly

2. Advanced RAG

def advanced_rag(query, knowledge_base, llm):
    # 1. Query rewriting/expansion
    expanded_query = query_expansion(query)
    
    # 2. Hybrid search (semantic + keyword)
    semantic_results = semantic_search(expanded_query, knowledge_base)
    keyword_results = keyword_search(expanded_query, knowledge_base)
    relevant_docs = rerank(semantic_results, keyword_results)
    
    # 3. Context compression
    compressed_context = compress_context(relevant_docs, query)
    
    # 4. Generate with citations
    prompt = build_prompt(compressed_context, query)
    response = llm.generate(prompt)
    
    # 5. Post-processing
    final_response = add_citations(response, relevant_docs)
    return final_response

Characteristics:

Query optimization
Hybrid retrieval
Context compression
Reranking
Citation support

3. Modular RAG

class ModularRAG:
    def __init__(self):
        self.query_processor = QueryProcessor()
        self.retriever = HybridRetriever()
        self.reranker = Reranker()
        self.context_compressor = ContextCompressor()
        self.generator = LLMGenerator()
        self.post_processor = PostProcessor()
    
    def process(self, query):
        # Modular pipeline
        processed_query = self.query_processor.process(query)
        candidates = self.retriever.retrieve(processed_query)
        ranked = self.reranker.rerank(candidates, query)
        context = self.context_compressor.compress(ranked, query)
        response = self.generator.generate(context, query)
        final = self.post_processor.process(response, ranked)
        return final

Characteristics:

Modular components
Easy to swap components
Better control and optimization

Implementation Examples

Simple RAG with LangChain

from langchain.document_loaders import TextLoader
from langchain.text_splitter import CharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import FAISS
from langchain.llms import OpenAI
from langchain.chains import RetrievalQA

# Load documents
loader = TextLoader("documents.txt")
documents = loader.load()

# Split documents
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(documents)

# Create embeddings and vector store
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(texts, embeddings)

# Create retriever
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

# Create LLM
llm = OpenAI(temperature=0)

# Create RAG chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=retriever,
    return_source_documents=True
)

# Query
result = qa_chain({"query": "What is the main topic?"})
print(result["result"])
print(result["source_documents"])

RAG with Custom Retrieval

import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

class CustomRAG:
    def __init__(self, documents, embedding_model="all-MiniLM-L6-v2"):
        self.documents = documents
        self.embedding_model = SentenceTransformer(embedding_model)
        self.document_embeddings = self.embedding_model.encode(documents)
    
    def retrieve(self, query, top_k=5):
        # Encode query
        query_embedding = self.embedding_model.encode([query])
        
        # Calculate similarities
        similarities = cosine_similarity(query_embedding, self.document_embeddings)[0]
        
        # Get top-k documents
        top_indices = np.argsort(similarities)[-top_k:][::-1]
        top_docs = [self.documents[i] for i in top_indices]
        
        return top_docs
    
    def generate(self, query, llm):
        # Retrieve relevant documents
        relevant_docs = self.retrieve(query)
        
        # Build context
        context = "\n\n".join(relevant_docs)
        prompt = f"""Based on the following context, answer the question.

Context:
{context}

Question: {query}

Answer:"""
        
        # Generate response
        response = llm.generate(prompt)
        return response, relevant_docs

# Usage
documents = [
    "Machine learning is a subset of artificial intelligence.",
    "Deep learning uses neural networks with multiple layers.",
    "Natural language processing enables computers to understand text."
]

rag = CustomRAG(documents)
response, sources = rag.generate("What is machine learning?", llm)

RAG with Pinecone Vector Database

from pinecone import Pinecone, ServerlessSpec
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Pinecone as PineconeVectorStore
from langchain.chains import RetrievalQA
from langchain.llms import OpenAI

# Initialize Pinecone
pc = Pinecone(api_key="your-api-key")
index_name = "rag-index"

# Create index if it doesn't exist
if index_name not in pc.list_indexes().names():
    pc.create_index(
        name=index_name,
        dimension=1536,  # OpenAI embedding dimension
        metric="cosine",
        spec=ServerlessSpec(cloud="aws", region="us-east-1")
    )

# Get index
index = pc.Index(index_name)

# Create embeddings
embeddings = OpenAIEmbeddings()

# Create vector store
vectorstore = PineconeVectorStore(index, embeddings, "text")

# Add documents
documents = ["Document 1 content", "Document 2 content"]
vectorstore.add_texts(documents)

# Create RAG chain
llm = OpenAI(temperature=0)
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever()
)

# Query
response = qa_chain.run("Your question here")

Retrieval Strategies

1. Semantic Search (Dense Retrieval)

def semantic_search(query, vectorstore, top_k=5):
    # Convert query to embedding
    query_embedding = embedding_model.encode(query)
    
    # Search in vector space
    results = vectorstore.similarity_search_with_score(
        query_embedding, 
        k=top_k
    )
    return results

Pros: Understands meaning and context Cons: May miss exact keyword matches

2. Keyword Search (Sparse Retrieval)

def keyword_search(query, documents, top_k=5):
    from sklearn.feature_extraction.text import TfidfVectorizer
    
    # Create TF-IDF vectors
    vectorizer = TfidfVectorizer()
    doc_vectors = vectorizer.fit_transform(documents)
    query_vector = vectorizer.transform([query])
    
    # Calculate similarities
    similarities = (doc_vectors * query_vector.T).toarray().flatten()
    
    # Get top-k
    top_indices = np.argsort(similarities)[-top_k:][::-1]
    return [documents[i] for i in top_indices]

Pros: Good for exact matches, fast Cons: Doesn’t understand semantics

3. Hybrid Search

def hybrid_search(query, vectorstore, documents, top_k=5, alpha=0.5):
    # Semantic search
    semantic_results = semantic_search(query, vectorstore, top_k=top_k*2)
    
    # Keyword search
    keyword_results = keyword_search(query, documents, top_k=top_k*2)
    
    # Combine and rerank
    combined = combine_results(semantic_results, keyword_results, alpha)
    reranked = rerank(combined, query)
    
    return reranked[:top_k]

Pros: Best of both worlds Cons: More complex, slower

Common Interview Questions and Answers

Q1: What is RAG and why is it important?

RAG (Retrieval-Augmented Generation) is a technique that enhances LLM responses by retrieving relevant information from external knowledge bases before generation. It’s important because:

Reduces Hallucination: Responses are grounded in retrieved documents
Access to Current Information: Can use information not in training data
Domain Adaptation: Works with specialized knowledge without fine-tuning
Transparency: Can cite sources
Cost-Effective: No need to retrain models for new information

Q2: How does RAG differ from fine-tuning?

Aspect	RAG	Fine-tuning
Knowledge Update	Add/remove documents	Retrain model
Cost	Low (just storage)	High (compute)
Speed	Fast updates	Slow (hours/days)
Flexibility	Easy to change	Requires retraining
Memory	External storage	Model weights
Use Case	Frequently changing data	Stable domain knowledge

RAG: Better for frequently changing, large knowledge bases Fine-tuning: Better for learning specific patterns/styles

Q3: What are the main components of a RAG system?

Document Loader: Loads documents from various sources (PDFs, web, databases)
Text Splitter: Chunks documents into manageable pieces
Embedding Model: Converts text to vector representations
Vector Database: Stores and searches document embeddings
Retriever: Finds relevant documents for queries
LLM: Generates responses based on retrieved context
Prompt Template: Structures the context and query for the LLM

Q4: How do you handle long documents in RAG?

Chunking: Split documents into smaller chunks

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200  # Overlap to maintain context
)
chunks = splitter.split_documents(documents)

Hierarchical Retrieval: Retrieve at multiple levels (sections, paragraphs, sentences)

Context Compression: Compress retrieved chunks to most relevant parts

from langchain.document_compressors import LLMChainExtractor

compressor = LLMChainExtractor.from_llm(llm)
compressed_docs = compressor.compress_documents(docs, query)

Sliding Window: Use overlapping windows to maintain context

Q5: What is the difference between naive RAG and advanced RAG?

Naive RAG:

Simple retrieval → generation
No query optimization
Basic context assembly
Limited reranking

Advanced RAG:

Query rewriting/expansion
Hybrid retrieval (semantic + keyword)
Context compression
Reranking
Post-processing and citations
Better performance and accuracy

Q6: How do you evaluate a RAG system?

Retrieval Metrics:
- Precision@K: Fraction of retrieved docs that are relevant
- Recall@K: Fraction of relevant docs that are retrieved
- MRR (Mean Reciprocal Rank): Average of reciprocal ranks
Generation Metrics:
- BLEU, ROUGE: Compare generated text to reference
- Faithfulness: How well response is grounded in context
- Answer Relevance: How relevant is answer to query
End-to-End Metrics:
- Human evaluation
- Task-specific metrics (e.g., accuracy for QA)

def evaluate_rag(rag_system, test_queries, ground_truth):
    results = []
    for query, expected in zip(test_queries, ground_truth):
        response, sources = rag_system.generate(query)
        
        # Retrieval metrics
        precision = calculate_precision(sources, expected.sources)
        recall = calculate_recall(sources, expected.sources)
        
        # Generation metrics
        faithfulness = check_faithfulness(response, sources)
        relevance = check_relevance(response, query)
        
        results.append({
            "precision": precision,
            "recall": recall,
            "faithfulness": faithfulness,
            "relevance": relevance
        })
    
    return aggregate_metrics(results)

Q7: What are common challenges in RAG systems?

Retrieval Quality:
- Irrelevant documents retrieved
- Missing relevant documents
- Solution: Better embeddings, hybrid search, reranking
Context Window Limits:
- Too many documents exceed context limit
- Solution: Context compression, better chunking, summarization
Chunking Issues:
- Important information split across chunks
- Solution: Overlapping chunks, hierarchical retrieval
Query Understanding:
- Ambiguous queries
- Solution: Query expansion, clarification
Outdated Information:
- Knowledge base becomes stale
- Solution: Regular updates, versioning
Hallucination:
- LLM generates information not in context
- Solution: Better prompting, fact-checking, citations

Q8: How do you implement query expansion in RAG?

def query_expansion(query, llm):
    # Generate related queries
    expansion_prompt = f"""Generate 3 related queries for: {query}
    
Related queries:
1."""
    
    expanded_queries = llm.generate(expansion_prompt)
    
    # Combine original and expanded
    all_queries = [query] + parse_queries(expanded_queries)
    
    # Retrieve for each query
    all_results = []
    for q in all_queries:
        results = retrieve(q, vectorstore)
        all_results.extend(results)
    
    # Deduplicate and rerank
    unique_results = deduplicate(all_results)
    reranked = rerank(unique_results, query)
    
    return reranked

Q9: What is reranking and why is it important?

Reranking improves retrieval quality by using a more sophisticated model to score and reorder retrieved documents.

from sentence_transformers import CrossEncoder

def rerank(query, documents, top_k=5):
    # Use cross-encoder for reranking (more accurate but slower)
    model = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
    
    # Create query-document pairs
    pairs = [[query, doc] for doc in documents]
    
    # Get scores
    scores = model.predict(pairs)
    
    # Sort by score
    ranked_indices = np.argsort(scores)[::-1]
    ranked_docs = [documents[i] for i in ranked_indices[:top_k]]
    
    return ranked_docs

Why Important:

Initial retrieval may miss relevant docs
Reranking uses more context (query + document)
Improves final answer quality

Q10: How do you handle multi-hop reasoning in RAG?

Multi-hop reasoning requires retrieving information from multiple documents to answer a question.

def multi_hop_rag(query, vectorstore, max_hops=3):
    current_query = query
    retrieved_docs = []
    
    for hop in range(max_hops):
        # Retrieve documents
        docs = retrieve(current_query, vectorstore, top_k=5)
        retrieved_docs.extend(docs)
        
        # Check if we have enough information
        if can_answer(query, retrieved_docs):
            break
        
        # Generate sub-question for next hop
        sub_query = generate_sub_question(query, retrieved_docs, llm)
        current_query = sub_query
    
    # Generate final answer
    context = combine_documents(retrieved_docs)
    answer = generate_answer(query, context, llm)
    
    return answer, retrieved_docs

Best Practices

Chunking Strategy: Use appropriate chunk size and overlap
Embedding Model: Choose model suitable for your domain
Hybrid Search: Combine semantic and keyword search
Reranking: Use reranking for better quality
Context Compression: Compress context to fit in window
Evaluation: Regularly evaluate retrieval and generation quality
Monitoring: Track query patterns and update knowledge base
Citations: Always cite sources for transparency
Error Handling: Handle retrieval failures gracefully
Caching: Cache frequent queries for performance

Summary

RAG is a powerful technique that combines:

Retrieval: Finding relevant information from knowledge bases
Augmentation: Using retrieved info as context
Generation: Creating responses with LLMs

Key benefits:

Reduces hallucination
Access to current information
Domain adaptation without fine-tuning
Source attribution
Cost-effective updates

RAG is essential for building accurate, up-to-date AI applications that can leverage external knowledge sources effectively.

Interview angle

“What is RAG and why use it over fine-tuning?” - retrieve relevant context at query time and put it in the prompt. It gives current facts, citations and cheap updates; fine-tuning teaches form and behaviour and cannot be updated or cited. Knowledge is a retrieval problem.
“Walk through the pipeline.” - ingest (parse, chunk, contextualise, embed, index), then query (rewrite, retrieve, rerank, assemble context, generate with citations, validate). Naming the rewrite and rerank stages separates a production answer from a demo one.
“Where do RAG systems usually fail?” - retrieval, not generation. Measure recall@k first: if the right chunk isn’t in the candidate set, no prompt engineering fixes the answer. See 09_agentic_rag_and_evaluation.md.
“Has long context replaced RAG?” - no. RAG wins on cost, latency, unbounded corpora and attribution. Production systems usually retrieve first, then give the model generous context over what survived.