Supervised fine-tuning and instruction tuning

The stage that turns a text-continuation engine into something that answers questions. Also the fine-tuning you’re most likely to actually do.

What SFT does

Same next-token objective as pretraining, but on curated (instruction, response) pairs, with the loss computed only on the response tokens:

# Mask the prompt so the model isn't trained to predict the user's words
labels = input_ids.clone()
labels[:len(prompt_ids)] = -100      # -100 = ignored by CrossEntropyLoss

Forgetting that mask is a real bug: the model spends capacity learning to generate instructions instead of answers, and quality suffers in a way that’s hard to attribute.

Data quality over quantity

The strongest finding in this area: a few thousand high-quality examples beat hundreds of thousands of mediocre ones.

Practically, that means:

• 1,000-10,000 excellent examples is a realistic and effective target.
• Diversity of task and phrasing matters more than raw volume.
• One bad example is worth several good ones, negatively. Contradictory or wrong responses teach the model to be inconsistent.
• Read a random sample by hand before training. Always.

This is good news for application work: curating a thousand examples is achievable in a way that labelling a hundred thousand isn’t.

Format matters

Training data must use the model’s chat template, exactly:

text = tokenizer.apply_chat_template(
    [{"role": "user", "content": instruction},
     {"role": "assistant", "content": response}],
    tokenize=False,
)

A template mismatch between training and inference is a frequent cause of “fine-tuning made it worse”. See ../06_transformers_llm/03_tokenization.md.

Hyperparameters

from trl import SFTTrainer, SFTConfig

config = SFTConfig(
    learning_rate=2e-5,           # 10-100x lower than pretraining
    num_train_epochs=3,           # 2-4; more overfits
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    bf16=True,
)

The two that matter:

• Learning rate 1e-5 to 5e-5 for full fine-tuning, higher (1e-4 to 3e-4) for LoRA since only a small adapter trains. Too high causes catastrophic forgetting — the model loses general capability while learning your task.
• Epochs: 2-4. Overfitting happens fast on small datasets, sometimes within one epoch. Evaluate frequently, not just at epoch boundaries.

LoRA is the default

Full fine-tuning updates every parameter and needs memory for weights, gradients and optimiser state. LoRA trains a small low-rank adapter instead — typically under 1% of parameters.

from peft import LoraConfig

LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    task_type="CAUSAL_LM",
)

Targeting all linear layers (attention and MLP) generally outperforms attention-only, which was the original recommendation. Mechanics and the memory argument are in ../05_deep_learning/07_transfer_learning.md.

Full fine-tuning is worth it only for large datasets with substantial domain shift, and it’s rarely the right first move.

What SFT is good and bad at

Fine-tuning teaches form, not facts. Knowledge injected into weights can’t be updated without retraining, can’t be cited, and competes with everything learned in pretraining. If the requirement is “answer questions about our documentation”, it’s a retrieval problem. See 07_fine_tuning_vs_rag.md.

One underrated benefit: SFT can absorb a long prompt into the weights. If every request carries 2,000 tokens of instructions and examples, fine-tuning that behaviour in cuts per-request cost and latency permanently.

Synthetic data

The 2026 norm: generate training data with a stronger model, filter it, fine-tune a smaller one. This is distillation in practice.

frontier model -> generate 10k responses -> filter/verify -> SFT a small model

It’s how you get near-frontier quality on a narrow task at a fraction of the serving cost. Two cautions: check the provider’s terms on training from outputs, and filter aggressively — synthetic data inherits and can amplify the teacher’s errors. Verifiable filtering (do the tests pass, does the JSON parse) is far better than trusting the teacher.

Catastrophic forgetting

Fine-tuning on a narrow task degrades general capability. Mitigations, in order of practicality:

1. LoRA — the base weights are frozen, so there’s much less to forget.
2. Low learning rate, few epochs.
3. Mix in general instruction data — 10-20% of the training mix.
4. Evaluate on general benchmarks too, not only your task. If you only measure your task, you won’t notice the model got worse at everything else.

Interview angle

• “What is SFT and how does it differ from pretraining?” — same next-token objective, but on curated instruction/response pairs with the loss masked to the response only. Pretraining teaches language; SFT teaches the model to answer rather than continue.
• “How much data do you need?” — usually 1,000-10,000 high-quality examples. Quality and diversity dominate volume, and a handful of contradictory examples does real damage.
• “Why mask the prompt tokens in the loss?” — otherwise the model is trained to generate instructions as well as responses, wasting capacity and degrading answer quality.
• “Fine-tune to teach the model our internal documentation?” — no. Fine-tuning teaches form, not facts; knowledge in weights can’t be updated or cited. Use retrieval. Fine-tune for consistent output structure, tone, or to absorb a long standing prompt.
• “LoRA or full fine-tuning?” — LoRA by default: under 1% of parameters trained, far less memory, adapters swappable per task, mergeable for zero inference overhead, and much less catastrophic forgetting. Full fine-tuning only for large datasets with heavy domain shift.
• “How do you build a training set without human labellers?” — generate with a stronger model, then filter, ideally with a programmatic verifier rather than trusting the teacher. Check the provider’s terms, and evaluate the student independently since synthetic data inherits the teacher’s mistakes.