Fine-Tuning vs RAG vs Prompt Engineering

The three ways to make an LLM behave the way you want. Each has different costs, capabilities, and use cases. Interviewers love “when would you use each?” — the answer is rarely all of them; usually one or two combined.

For RAG patterns see ../09_rag_embeddings/03_rag_architecture_patterns.md. For prompt engineering see ../06_transformers_llm/09_prompt_engineering.md.

The three approaches at a glance

The rule of thumb:

Prompt first. RAG when you need facts. Fine-tune when style/behavior matters AND prompting can’t fix it.

Prompt engineering

Cheapest, fastest, most iterative. Start here.

prompt = """You are a SQL expert. Convert the natural-language question to a SELECT query.

Schema:
- users(id, name, email)
- orders(id, user_id, total, created_at)

Examples:
Q: How many orders did Alice place?
A: SELECT COUNT(*) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.name = 'Alice';

Q: Top 5 customers by total spend
A: SELECT u.name, SUM(o.total) FROM orders o JOIN users u ON u.id = o.user_id GROUP BY u.id ORDER BY SUM(o.total) DESC LIMIT 5;

Q: {question}
A:"""

Works when:

• The task is general (the model has seen enough during training).
• A few examples suffice.
• The data you need is bounded (fits in the prompt).

Doesn’t work when:

• The model lacks the underlying capability (no amount of prompting teaches it to do something it can’t do).
• The required knowledge is too large for the context.
• You need consistent specific behavior across thousands of queries.

RAG (Retrieval-Augmented Generation)

Inject relevant chunks of your data at query time.

def rag_answer(question):
    chunks = vector_db.search(embed(question), top_k=5)
    context = "\n\n".join(c.text for c in chunks)
    prompt = f"Answer the question using only the context.\n\nContext:\n{context}\n\nQuestion: {question}"
    return llm.generate(prompt)

Works when:

• You have a large corpus of facts (knowledge base, documentation, product catalog).
• Information changes frequently (RAG is the only one that can stay fresh without retraining).
• You need source attribution / citations.
• The model needs domain-specific facts but the rest of the task is general.

Doesn’t work for:

• Style / tone (RAG retrieves facts; doesn’t change how the model writes).
• Behavioral changes (refusing certain requests, always responding a certain way).
• Pure reasoning tasks unrelated to your data.

For deep coverage see ../09_rag_embeddings/.

Fine-tuning

Modify the model’s weights using labeled examples.

# OpenAI fine-tuning example
training_data = [
    {"messages": [
        {"role": "system", "content": "You are a customer support agent for Acme."},
        {"role": "user", "content": "I need help with my account."},
        {"role": "assistant", "content": "Sure! Could you tell me your account email so I can look that up?"},
    ]},
    # ... thousands of examples ...
]

# Upload + train
job = openai.fine_tuning.jobs.create(
    training_file="training.jsonl",
    model=SMALL_MODEL,   # pin the exact version in config
)
# Returns a fine-tuned model ID, used like a normal model

Works when:

• Specific tone / style / format must be consistent across thousands of queries.
• The base model “almost” does the task but needs nudging.
• You have many labeled examples (typically 500+).
• The base model is too slow / expensive and you can use a smaller fine-tuned model.

Doesn’t work for:

• Adding new factual knowledge (fine-tuning can teach format but not reliably teach facts).
• Frequently changing information.
• Small training sets (< 100 examples — usually prompting suffices).

What fine-tuning teaches well

The famous Karpathy framing: fine-tuning shifts the model’s distribution toward your data. It doesn’t add knowledge so much as bias what’s already there.

The decision tree

Need a behavior change from the base model?
│
├── Is the data static or fits in context?
│   └── Yes → Prompt engineering (try this first)
│       └── Examples in the prompt (few-shot) help
│
├── Is the data large or changes often?
│   └── Yes → RAG
│       └── Combine with prompting for style / format
│
├── Need consistent style / format / tone at scale?
│   └── Yes → Fine-tuning (after prompting fails)
│       └── Often combined with RAG for facts
│
└── Need genuinely new capability?
    └── Probably no model can; rethink the problem

Combining approaches

Real production systems often combine all three:

def production_pipeline(question, user_context):
    # 1. RAG: retrieve relevant facts
    chunks = vector_db.search(embed(question), filters={"user_id": user_context.id})

    # 2. Fine-tuned model: tone, format, calling conventions
    # 3. Prompt engineering: instructions, examples, output format
    response = fine_tuned_model.generate(
        system="You are Acme's customer support agent. Be concise and friendly.",
        context=chunks,
        question=question,
        examples=few_shot_examples,
        response_format={"type": "json_schema", "schema": OUTPUT_SCHEMA},
    )

    return response

The base model handles general capability; fine-tuning handles tone; RAG handles facts; prompting handles structure.

Cost comparison (rough orders of magnitude)

Prompting iterates in seconds. RAG iterates in hours (rebuild index). Fine-tuning iterates in days (data prep + training).

Start with prompting; add complexity only when needed.

Mistakes by approach

Prompt engineering pitfalls

• “We tried prompting and it didn’t work” — often the prompt was 3 sentences. Real prompt engineering takes iteration.
• No eval set — can’t measure improvement.
• One mega-prompt instead of decomposition.

RAG pitfalls

• Naive RAG without reranking, hybrid retrieval, or query rewriting. See ../09_rag_embeddings/03_rag_architecture_patterns.md.
• Treating RAG as a fix for hallucinations without grounding instructions (“Answer ONLY using the context”).
• Stale index when source data changes.

Fine-tuning pitfalls

• “Fine-tune to teach it our company’s knowledge” — fine-tuning doesn’t reliably teach facts. Use RAG.
• Tiny training set (< 100 examples) — usually prompting works as well.
• Training on examples that don’t match production distribution.
• Not evaluating against a held-out test set.
• Fine-tuning on a model that gets deprecated; you have to redo it.

Newer techniques

Parameter-efficient fine-tuning (PEFT, LoRA, QLoRA)

Instead of retraining all model weights (billions of params), train small adapter layers. Much cheaper.

# Hugging Face PEFT
from peft import LoraConfig, get_peft_model

config = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"])
peft_model = get_peft_model(base_model, config)

LoRA fine-tuning costs are 10-100× lower than full fine-tuning, with similar effectiveness for many use cases. The default for open-source fine-tuning.

Instruction tuning vs RLHF

Original models complete text. Instruction tuning trains them to follow instructions. RLHF (Reinforcement Learning from Human Feedback) further aligns them to human preferences.

You probably won’t do RLHF — it’s expensive and complex. Instruction tuning (a flavor of fine-tuning) is more accessible.

DPO (Direct Preference Optimization)

Newer alternative to RLHF: train on pairs of (preferred, rejected) responses. Simpler than RLHF; widely adopted in 2024+.

Function-calling fine-tuning

Fine-tune models to call your specific tools / APIs consistently. Useful when the base model doesn’t reliably emit your function call format.

When to fine-tune (pragmatic guide)

You should consider fine-tuning when:

1. You’ve exhausted prompt engineering.
2. RAG doesn’t apply (the problem isn’t about facts).
3. You have 500+ high-quality labeled examples.
4. The base model is too slow / expensive at scale, and a smaller fine-tuned model would suffice.
5. Specific style/format/tone must be consistent across thousands of queries.

If you only have 50 examples or the issue is “the model doesn’t know about our product,” fine-tuning is the wrong tool.

Open-source vs API models

For most teams: start with APIs (faster iteration, no infra). Move to open-source when cost, privacy, or specific fine-tuning needs demand it.

Common interview confusions

• “Fine-tuning makes the model know your data.” — only partially, and unreliably. RAG is the right tool for “the model needs to know facts.”
• “RAG is just a hack until we can fine-tune.” — for dynamic knowledge, RAG is the correct architecture. Fine-tuning doesn’t keep up with changing data.
• “More fine-tuning data = better model.” — only if it’s high quality. 500 great examples beat 5000 mediocre ones.

Interview angle

• “When do you use RAG vs fine-tuning?” — RAG for knowledge (facts, documents, dynamic data, citations). Fine-tuning for style/behavior/format/persona that needs to be consistent at scale. They’re complementary, not alternatives.
• “Why doesn’t fine-tuning teach facts well?” — fine-tuning shifts the model’s distribution toward your data; doesn’t add discrete factual entries. It can teach call patterns / tone, but specific facts are inconsistently retrievable. RAG gives reliable, citable facts.
• “What’s parameter-efficient fine-tuning (LoRA)?” — train small adapter layers instead of the full model. 10-100× cheaper than full fine-tuning; effective for most use cases. Default for open-source FT.
• “When does prompt engineering suffice (no RAG or FT)?” — task is general, examples-in-context get acceptable quality, data fits in context, no specific style requirement that prompting can’t achieve. Try this first; it’s the cheapest iteration loop.
• “How much data do you need to fine-tune?” — for tone/format: 100-500 examples can work. For task-specific behavior: 500-10000 typical. Less than 100: prompting usually wins.
• “How would you decide between fine-tuning a smaller model vs prompting a bigger one?” — measure: cost, latency, quality. A fine-tuned 7B model can match GPT-4 on narrow tasks at 1/100th the cost. Worth it for high-volume / latency-sensitive cases.
• “Can you combine RAG and fine-tuning?” — yes. Common pattern: fine-tune for company tone and call patterns; use RAG for specific facts. They address different things.
• “What problems can’t be solved with any of these?” — fundamental capability gaps (genuine reasoning beyond the base model), genuinely novel tasks not in training data, things that require external action without tools/agents.