Transformers and LLMs
Rewritten for 2026. The 2017 paper is not what runs today — every frontier model is decoder-only with RoPE, SwiGLU, RMSNorm, pre-norm, GQA or MLA, and usually MoE.
Architecture
| # | File | Covers |
|---|---|---|
| 01 | 01_transformer_architecture.md | the block, self-attention, why decoder-only won, what changed since 2017 |
| 02 | 02_attention_mechanisms.md | MHA → MQA → GQA → MLA, FlashAttention, attention sinks |
| 03 | 03_tokenization.md | BPE, token costs, why arithmetic fails, chat templates |
| 04 | 04_positional_encoding.md | RoPE, and how context windows get extended |
Inference behaviour
| # | File | Covers |
|---|---|---|
| 05 | 05_kv_cache.md | prefill vs decode, cache sizing, PagedAttention, prefix caching |
| 06 | 06_mixture_of_experts.md | total vs active parameters, load balancing, serving implications |
| 07 | 07_reasoning_models.md | test-time compute, RLVR, GRPO, thinking budgets |
| 08 | 08_context_windows.md | lost-in-the-middle, why advertised ≠ usable, long context vs RAG |
Using them
| # | File | Covers |
|---|---|---|
| 09 | 09_prompt_engineering.md | prompting techniques |
| 10 | 10_hallucinations_and_tools.md | why models confabulate, and grounding |
The four answers worth having ready
Why attention is quadratic, and what follows. Every pair of positions gets a score. That one fact drives context pricing, KV caching, FlashAttention, and the entire efficient-attention literature.
Prefill is compute-bound; decode is memory-bandwidth-bound. It explains why output tokens cost more than input tokens, why batching works, and why KV cache size — not FLOPs — caps concurrency.
Total vs active parameters. A 671B MoE with 37B active still needs memory for 671B. It is not “a 37B model”.
Test-time compute is a scaling axis. A smaller model with an adequate thinking budget can beat a much larger one at matched compute. That changed model selection.