Pretraining
You will almost certainly never do this. You should understand it anyway, because it explains what the model knows, what it can’t know, and why the fine-tuning options look the way they do.
The objective
Next-token prediction over a very large corpus:
loss = -sum( log P(token_t | token_1 ... token_{t-1}) )Self-supervised: the label at every position is simply the following token, so any text is training data with no annotation. That’s the property that let pretraining scale — see ../01_ml_foundations/01_ml_problem_types.md.
Two things follow, and both are practical:
- The model learns the distribution of its training data, including its biases, errors and cutoff date. It has no notion of truth, only of what typically follows.
- Every position contributes a gradient, unlike masked language modelling which only trains on the masked ~15%. That efficiency is part of why decoder-only won.
Data
Data quality dominates. The rough pipeline:
- Collect — web crawl, code, books, papers, curated sources.
- Filter — language identification, quality classifiers, heuristics (a page that’s 90% navigation links is not training data).
- Deduplicate — exact and near-duplicate removal. Critical: duplicates cause memorisation and waste compute, and duplicated benchmark text leaks into evaluation.
- Decontaminate — remove benchmark test sets, or your evaluations are meaningless.
- Mix — weight sources deliberately. Code improves reasoning even for non-code tasks; over-weighting a domain skews the model.
Deduplication and decontamination are the two steps that get skipped and cause the most damage. A model that memorised its evaluation set looks excellent and isn’t.
Scaling laws
Loss follows a power law in model size, data size and compute. The practical question is how to spend a fixed compute budget.
The Chinchilla result corrected an earlier bias toward large models trained on too little data: for compute-optimal training, scale parameters and tokens together — roughly 20 tokens per parameter.
The important nuance for 2026: compute-optimal is not inference-optimal. Chinchilla optimises training cost. If you’ll serve a model billions of times, it pays to train a smaller model on far more data than compute-optimal, because you buy cheaper inference forever. That’s why current small models are trained on token counts far above the Chinchilla ratio.
Being able to state that distinction — training-optimal vs serving-optimal — is a strong answer.
Cost and infrastructure
Frontier pretraining runs cost tens of millions of dollars across thousands of GPUs for weeks or months. The engineering, briefly:
| Parallelism | Splits |
|---|---|
| Data | the batch across devices; each holds full weights |
| Tensor | individual matrices across devices within a layer |
| Pipeline | layers across devices |
| Expert | MoE experts across devices |
| FSDP / ZeRO | shards weights, gradients and optimiser state |
Real runs combine several. The recurring problems are hardware failure at scale (checkpoint constantly), loss spikes (skip the batch and resume from a checkpoint), and communication overhead dominating as you scale out.
You don’t need depth here unless the role is training infrastructure. Knowing the names and what each splits is enough.
Continued pretraining
The version you might actually do. Take an existing base model and keep training with the same next-token objective on domain data — medical, legal, a specific language, an internal codebase.
| Continued pretraining | Fine-tuning | |
|---|---|---|
| Objective | next-token, unchanged | instruction/preference |
| Data | large unlabelled domain corpus | small labelled examples |
| Teaches | domain vocabulary and patterns | task behaviour and format |
| Volume | billions of tokens | hundreds to thousands of examples |
Use it when the domain vocabulary is genuinely alien to the base model. It’s expensive and requires real data volume, so it’s usually only worth it after prompting and RAG have been tried. See 07_fine_tuning_vs_rag.md.
What pretraining leaves you with
A base model — good at continuing text, bad at following instructions. Ask it a question and it may continue with more questions, because that’s what commonly follows a question in web text.
Turning that into something useful is post-training: supervised fine-tuning then preference optimisation. See 02_sft_instruction_tuning.md and 03_rlhf_and_preference_optimization.md.
Practically, “base” vs “instruct” is a real deployment decision: base models for continuation and as a fine-tuning starting point, instruct models for anything conversational.
Interview angle
- “How are LLMs pretrained?” — next-token prediction over a large filtered corpus. It’s self-supervised, so every token is a label and training scales with raw text rather than annotation.
- “What are scaling laws, and does Chinchilla still apply?” — loss follows a power law in parameters, data and compute; Chinchilla says scale parameters and tokens together, roughly 20:1, for compute-optimal training. But compute-optimal isn’t inference-optimal: if you serve a model heavily, over-train a smaller one to buy cheaper inference.
- “Why does deduplication matter so much?” — duplicates cause memorisation and waste compute, and duplicated benchmark content contaminates evaluation. A model that memorised its test set scores brilliantly and tells you nothing.
- “Base model vs instruct model?” — a base model continues text and doesn’t reliably follow instructions; an instruct model has been through SFT and preference optimisation. Use base models as fine-tuning starting points, instruct models for anything conversational.
- “When would you do continued pretraining?” — when domain vocabulary is genuinely outside the base model’s distribution and you have billions of tokens of domain text. It’s expensive, so it comes after prompting and retrieval have been ruled out.
- “Why is code in the pretraining mix even for non-code models?” — it appears to improve structured reasoning generally. Data mixture is a deliberate lever, not just a collection of whatever was available.