Quantization

Reducing numeric precision to cut memory and increase speed. The most cost-effective single lever in LLM deployment, and a routine interview topic for anyone claiming inference experience.

Why it works

Model weights don’t need 16 bits of precision. Neural networks are robust to weight noise, so representing weights in 8 or 4 bits loses far less quality than the compression ratio suggests.

FP16/BF16   2 bytes/param    70B model -> 140 GB
INT8        1 byte           70B model ->  70 GB
INT4        0.5 bytes        70B model ->  35 GB

The practical effect: 4-bit quantisation is what puts a 70B model on a single 48GB GPU, or a 7B model on a laptop.

Speed improves too, because decode is memory-bandwidth bound — fewer bytes read per token means faster generation, even when the arithmetic is dequantised back to higher precision. See ../06_transformers_llm/05_kv_cache.md.

The formats

bf16 vs fp16: bf16 has fp32’s exponent range with fewer mantissa bits, so it doesn’t underflow and needs no loss scaling. It’s the safer default where hardware supports it. See ../05_deep_learning/03_training_deep_networks.md.

What gets quantised

Three separate decisions, and conflating them is a common muddle:

Weight-only quantisation is the common case: store weights in INT4, dequantise to bf16 for the matmul. You get the memory and bandwidth win without the accuracy risk of quantised arithmetic.

FP8 KV cache is the single best value in serving. It halves cache memory — doubling concurrency — at essentially no quality cost, and it’s a flag rather than a re-quantisation job.

Post-training vs quantization-aware

Post-training quantization (PTQ) — quantise an already-trained model. Fast (minutes to hours), no training data beyond a small calibration set. This is what you’ll use.

Quantization-aware training (QAT) — simulate quantisation during training so the model adapts. Better quality at very low bit-widths, but requires the full training pipeline. Rare outside model producers.

The methods you’ll actually encounter

from transformers import BitsAndBytesConfig

BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,   # dequantise to bf16 for matmul
    bnb_4bit_quant_type="nf4",               # 4-bit NormalFloat
    bnb_4bit_use_double_quant=True,          # quantise the quantisation constants too
)

The shared insight across GPTQ and AWQ: not all weights matter equally. A small fraction of channels carry disproportionate importance (outlier features), and protecting those while aggressively quantising the rest preserves quality far better than uniform quantisation.

Quality impact

Rough expectations for a well-quantised model:

Two things that shift this:

• Smaller models degrade more. A 70B at INT4 holds up better than a 7B at INT4 — there’s more redundancy to lose.
• Reasoning and long-context tasks degrade first. Simple classification barely moves; multi-step maths shows it clearly. Evaluate on your hardest task, not a generic benchmark.

A larger model quantised usually beats a smaller model at full precision for the same memory. A 70B at INT4 (~35GB) generally outperforms a 13B at BF16 (~26GB). That’s the practical rule worth stating.

QLoRA

Quantise the frozen base to 4-bit, train a LoRA adapter in higher precision on top. The base never updates, so quantisation error doesn’t compound through training.

This is what makes fine-tuning a large model on one consumer GPU feasible, and it’s the standard answer to “how would you fine-tune a 70B model without a cluster”.

Distillation as the alternative

Quantisation compresses a model in place. Distillation trains a smaller model to imitate a larger one — usually on the teacher’s output distribution or generated responses.

They compose: distil to a smaller model, then quantise it. For a narrow high-volume task, distilling a frontier model’s outputs into a small fine-tuned model is often the biggest cost win available — see 02_sft_instruction_tuning.md.

Interview angle

• “Why does quantisation speed up generation, not just save memory?” — decode is memory-bandwidth bound. Each token requires reading all the weights, so halving the bytes roughly halves the dominant cost, even if the arithmetic happens in higher precision after dequantisation.
• “INT4 70B or BF16 13B, same memory budget?” — the quantised 70B, generally. Larger models have more redundancy and tolerate quantisation better; the capability gap usually exceeds the quantisation loss.
• “What’s the cheapest quality-preserving win in serving?” — FP8 KV cache. Halves cache memory, roughly doubles concurrency, negligible quality impact, and it’s a configuration flag.
• “GPTQ vs AWQ?” — both 4-bit post-training methods. GPTQ minimises layer-wise reconstruction error using second-order information; AWQ identifies salient channels by activation magnitude and protects them. Both rest on the same insight that a minority of weights carry disproportionate importance.
• “How do you fine-tune a 70B model on one GPU?” — QLoRA: load the base in 4-bit, freeze it, train a LoRA adapter in bf16. The frozen base means quantisation error doesn’t accumulate through training.
• “How do you validate a quantised model?” — evaluate on your hardest task, not a generic benchmark. Reasoning and long-context degrade first while simple classification barely moves, so an aggregate score can hide the regression that matters.