Transformer architecture

The 2017 paper is not what runs in 2026. Every frontier model is a decoder-only transformer with RoPE, SwiGLU, RMSNorm, pre-norm, and grouped-query or latent attention. Describing the original encoder-decoder as if it were current is a dating signal.

The block

class Block(nn.Module):
    def forward(self, x):
        x = x + self.attn(self.norm1(x))    # pre-norm + residual
        x = x + self.mlp(self.norm2(x))
        return x

Two sublayers, each wrapped in a residual with normalisation applied before it. Stack N of these, add embeddings at the bottom and a projection to vocabulary at the top. That’s the whole architecture.

• Attention mixes information across positions.
• MLP transforms each position independently, and holds most of the parameters (typically ~2/3).

A useful framing: attention decides what to look at, the MLP decides what to do with it.

Self-attention

scores = (Q @ K.transpose(-2, -1)) / math.sqrt(d_k)   # (seq, seq)
scores = scores.masked_fill(causal_mask, float("-inf"))
weights = scores.softmax(dim=-1)
out = weights @ V

Every token produces a query (what am I looking for), a key (what do I offer), and a value (what I contribute). The query-key dot product scores relevance; softmax turns scores into weights; the output is a weighted sum of values.

Why divide by sqrt(d_k). Dot products of d_k-dimensional random vectors have variance proportional to d_k. Without scaling, large d_k produces large logits, softmax saturates, and the gradient vanishes. This is a reliable interview question with a precise answer.

The causal mask is what makes it a language model: position i may only attend to positions <= i. Mask future positions to -inf before softmax so they receive zero weight. Without it, the model trivially cheats by reading the answer.

Quadratic cost. The score matrix is (seq, seq), so time is O(n² · d) and memory for scores is O(n²). That single fact drives context-window pricing, KV caching, FlashAttention, and every efficient-attention paper. See ../00_math_foundations/01_linear_algebra.md.

Multi-head

Split the hidden dimension into h heads, attend independently, concatenate, project.

d_model = 4096, h = 32  ->  d_head = 128

Different heads specialise — some track syntax, some resolve coreference, some attend to position. It costs nothing extra: h heads of dimension d/h is the same total compute as one head of dimension d, but gives multiple independent attention patterns instead of one averaged one.

Variants that shrink the KV cache (GQA, MLA) are in 02_attention_mechanisms.md.

The MLP

# Classic (2017)
mlp = nn.Sequential(nn.Linear(d, 4*d), nn.GELU(), nn.Linear(4*d, d))

# Modern - SwiGLU, gated
def swiglu_mlp(x, W_gate, W_up, W_down):
    return (F.silu(x @ W_gate) * (x @ W_up)) @ W_down

Expand to roughly 4× the model dimension, apply a non-linearity, project back. The expansion is where most parameters and most of the model’s stored knowledge live.

SwiGLU’s multiplicative gate is more expressive than a plain activation, so the hidden dimension is typically shrunk to ~2/3 to keep parameter count comparable. See ../05_deep_learning/02_activation_functions.md.

Why decoder-only won

Decoder-only won for three reasons worth being able to state:

1. One objective, unlimited data. Next-token prediction turns any text into training data with no labelling. Masked language modelling only trains on the ~15% of positions that are masked, so it’s less sample-efficient per token.
2. One model for everything. Generation, classification and extraction all become “produce the right continuation”, so there’s no architectural split by task.
3. It scales cleanly, and in-context learning emerged from scale — a capability the other architectures didn’t show.

Encoder-only models are still the right tool for embeddings and classification where you don’t need generation: cheaper, faster, and bidirectional context genuinely helps. Most retrieval systems still use one. See ../09_rag_embeddings/.

Assembled

class GPT(nn.Module):
    def forward(self, idx):
        x = self.token_embedding(idx)          # (B, T, d) - RoPE applies inside attention
        for block in self.blocks:
            x = block(x)
        x = self.final_norm(x)                 # final norm after the stack (pre-norm designs)
        return self.lm_head(x)                 # (B, T, vocab) logits

Weight tying — sharing the token embedding matrix with the output projection — is common and saves vocab × d parameters (hundreds of millions at large vocabularies).

Parameter count is dominated by, per layer: 4d² for attention projections plus roughly 8d²-12d² for the MLP. Multiply by layers, add embeddings.

What changed since 2017

Being able to run that table is a compact way to show you’re current rather than reciting the original paper.

Interview angle

• “Walk me through a transformer block.” — pre-norm, then attention with a residual, then pre-norm and an MLP with a residual. Attention mixes across positions, the MLP transforms each position independently and holds most parameters.
• “Why divide attention scores by sqrt(d_k)?” — dot-product variance grows with d_k, so unscaled logits get large, softmax saturates and gradients vanish. Scaling keeps the logits in a workable range.
• “Why is attention quadratic, and what follows from it?” — every pair of positions gets a score, so O(n²) in time and memory. It sets context limits and pricing, and motivates KV caching, FlashAttention and all the efficient-attention work.
• “Why did decoder-only architectures win?” — next-token prediction uses every token as a label so it scales with raw text, one model handles all tasks, and in-context learning emerged from that scale. Encoder-only models remain better for embeddings.
• “What’s a causal mask and why is it needed?” — masks future positions to -inf before softmax so a token can’t attend forward. Without it, next-token training is trivially degenerate.
• “What in a modern LLM differs from the 2017 paper?” — decoder-only, pre-norm, RMSNorm, SwiGLU, RoPE, GQA or MLA, and usually MoE. Almost nothing except the core attention equation is unchanged.