Positional encoding

Attention is permutation-invariant — shuffle the tokens and the output is the same set, reordered. Position has to be injected explicitly. How it’s injected determines whether the model handles sequences longer than it trained on.

The progression

Learned absolute positions cannot extrapolate at all — position 5000 has no embedding if you only trained to 4096. That hard wall is why they were abandoned.

RoPE

Rotary Position Embedding. Instead of adding a position vector, rotate the query and key vectors by an angle proportional to position.

def apply_rope(x, pos, theta=10000.0):
    d = x.shape[-1]
    freqs = 1.0 / (theta ** (torch.arange(0, d, 2) / d))
    angles = pos[:, None] * freqs[None, :]
    cos, sin = angles.cos(), angles.sin()
    x1, x2 = x[..., 0::2], x[..., 1::2]
    return torch.stack([x1 * cos - x2 * sin,
                        x1 * sin + x2 * cos], dim=-1).flatten(-2)

The elegant property: the dot product between a rotated query at position m and a rotated key at position n depends only on m - n. Absolute rotations produce relative attention — the model sees distances, not indices.

Consequences that matter:

• Applied to Q and K inside attention, not to the input embeddings. Values are untouched.
• Relative by construction, so the same weights work at any position.
• Decays naturally with distance, which is a reasonable inductive bias for language.

Each dimension pair rotates at a different frequency: low dimensions rotate fast (fine local position), high dimensions rotate slowly (coarse long-range position). That frequency spread is what the context-extension methods manipulate.

Extending context

You’ve trained at 8k and want 128k. Naive extrapolation fails — the high-frequency dimensions have wrapped around into positions the model never saw, and output degrades sharply.

Position interpolation compresses rather than extrapolates: instead of asking about position 100,000, ask about position 6,250 in a space the model understands. The trade-off is reduced resolution for nearby tokens, which is why a short fine-tune helps.

NTK-aware scaling is smarter — it leaves high-frequency dimensions (local ordering, which matters and is well-learned) mostly alone and stretches low-frequency ones (long-range position). That’s why it often works with no training at all.

The important caveat: a model advertising a 128k window because its RoPE was rescaled is not equally good throughout that window. Always evaluate long-context quality on your own task. See 08_context_windows.md.

ALiBi

Skip position embeddings entirely; add a linear penalty to attention scores based on distance:

score(i, j) = q_i · k_j - m * (i - j)

m is a fixed per-head slope. Nearer tokens get less penalty. Simple, extrapolates well beyond training length without modification, and needs no extra parameters.

It lost to RoPE mainly on quality at scale — the fixed linear decay is a cruder prior than RoPE’s learned frequency structure.

Why not just add position to the embedding

The 2017 approach of adding a sinusoidal or learned vector to the input has a structural weakness: position information must survive every layer of processing while competing with semantic content in the same vector space. RoPE injects position at every attention operation, where it’s actually used, and leaves the residual stream free to carry meaning.

That framing — position belongs where the comparison happens, not in the content representation — is a good answer to “why did RoPE win”.

Multimodal and 2D positions

Vision transformers need 2D positions; video needs 3D. Extensions like 2D RoPE apply rotations along each axis. Worth knowing exists if the role involves multimodal work.

Interview angle

• “Why do transformers need positional encoding at all?” — self-attention is permutation-invariant. Without position, “dog bites man” and “man bites dog” produce identical representations.
• “What is RoPE and why did it win?” — rotate Q and K by an angle proportional to position, so their dot product depends only on relative distance. It’s relative by construction, injects position where attention actually uses it rather than polluting the residual stream, and extrapolates far better than learned absolute embeddings.
• “How do you extend a model’s context window?” — rescale RoPE: position interpolation to compress long positions into the trained range, or NTK-aware/YaRN scaling that stretches low frequencies while preserving high-frequency local ordering. Usually with a short fine-tune. NTK-style methods often work with none.
• “A model claims 128k context. Do you trust it end to end?” — no. If the window came from RoPE rescaling, quality typically degrades well before the stated limit. Evaluate retrieval and reasoning at your actual context lengths.
• “What’s ALiBi and why isn’t it standard?” — a fixed linear distance penalty on attention scores. It extrapolates well and needs no parameters, but its uniform decay is a cruder prior than RoPE’s frequency structure, and it lost on quality at scale.
• “Why not just add positions to the input embeddings?” — that forces position to survive every layer while sharing space with semantics. RoPE applies position at each attention computation, where the comparison happens, leaving the residual stream for meaning.