Attention variants

Almost every attention variant since 2019 exists to shrink the KV cache, because that — not compute — is what limits inference throughput. Understanding that motivation is more valuable than memorising the variants.

The problem they solve

During generation you cache the keys and values of every previous token. With multi-head attention that cache is:

2 (K and V) * layers * heads * d_head * seq_len * batch * bytes_per_value

For a 70B model at 32k context, that’s tens of gigabytes — often larger than the weights. Since decoding is memory-bandwidth bound, cache size directly determines how many concurrent requests fit on a GPU. See 05_kv_cache.md.

MHA → MQA → GQA

Multi-Query Attention shares a single K/V head across all query heads. Enormous cache saving, but quality suffers because all heads must read the same keys and values.

Grouped-Query Attention is the compromise that stuck: partition query heads into groups, one K/V head per group. With 32 query heads and 8 KV heads you cut the cache 4x with essentially no quality loss.

# GQA: repeat each KV head to serve its group of query heads
k = k.repeat_interleave(n_query_heads // n_kv_heads, dim=1)
v = v.repeat_interleave(n_query_heads // n_kv_heads, dim=1)

GQA is the default in 2026 — Llama, Gemma, Qwen and most open models use it. If asked “how do modern models reduce KV cache”, GQA is the expected first answer.

Multi-head Latent Attention

DeepSeek’s approach, now also used in the GLM series. Instead of sharing heads, compress K and V into a low-rank latent vector, cache that, and re-expand per head at use time.

The compression is far more aggressive than GQA — reported around 93% cache reduction, roughly 14x versus MHA — while preserving full quality, because each head still gets its own effective K/V after expansion rather than sharing one.

The cost is more complex kernels and an extra projection at decode time. It trades a little compute for a lot of memory, which is the right trade when you’re bandwidth-bound.

FlashAttention

Not an approximation — exact attention, computed with better memory access.

The naive implementation materialises the full (seq, seq) score matrix in GPU high-bandwidth memory. FlashAttention tiles the computation, keeps tiles in fast on-chip SRAM, and fuses softmax so the full matrix is never written out. It uses the online-softmax trick to accumulate correctly across tiles.

Result: memory drops from O(n²) to O(n), and it’s faster despite recomputing some values during the backward pass, because attention is memory-bound rather than compute-bound.

# Enabled implicitly by the fused kernel
out = F.scaled_dot_product_attention(q, k, v, is_causal=True)

Key point for interviews: FlashAttention changes nothing about the maths. Same outputs, better hardware utilisation. Contrast that with sparse or linear attention, which approximate and can lose quality.

Sparse and linear attention

Attempts to beat the quadratic term by not attending everywhere:

None fully displaced full attention. The practical pattern in 2026 is hybrid: mostly efficient layers with periodic full-attention layers, so long-range dependencies still have a path.

Cross-attention

Queries from one sequence, keys and values from another. Used in encoder-decoder models and in multimodal models where text queries attend to image features. Decoder-only LLMs use self-attention only, which is part of why they’re simpler.

Attention sinks

An empirical finding worth knowing: models allocate a lot of attention to the first few tokens regardless of content, apparently as a place to dump probability mass when nothing is relevant (softmax must sum to 1).

Practical consequence for streaming/long-context serving: if you evict the earliest tokens from the KV cache to save memory, quality collapses. Keeping the first few tokens pinned — “StreamingLLM” — fixes it. It’s a good example of an implementation detail that only shows up in production.

Choosing

You will rarely implement these. What you’re expected to know:

• Serving an open model: GQA models are the mainstream, well-supported choice.
• Long context on limited memory: MLA models cache far less; otherwise quantise the cache to FP8.
• Always use a FlashAttention-backed kernel — it’s free.
• Very long sequences: expect a hybrid architecture, and measure quality on your long-context task rather than trusting the advertised window.

Interview angle

• “Why do GQA and MLA exist?” — to shrink the KV cache. Decoding is memory-bandwidth bound, and cache size caps how many concurrent requests fit on a GPU, so cache reduction translates directly into throughput and cost.
• “MQA vs GQA?” — MQA shares one KV head across all query heads, which cuts the cache dramatically but costs quality. GQA groups query heads with one KV head per group, giving most of the saving with essentially no quality loss. GQA is the current default.
• “What is MLA?” — compress K and V into a low-rank latent that’s cached and re-expanded per head. Roughly 14x cache reduction versus MHA at full quality, at the cost of more complex kernels. Used by DeepSeek and GLM.
• “Does FlashAttention change the model’s output?” — no. It’s exact attention with a tiled, fused, SRAM-friendly implementation that never materialises the full score matrix. Memory goes from O(n²) to O(n) and it’s faster because attention is memory-bound.
• “Why haven’t linear-attention models replaced transformers?” — the quality gap on long-range reasoning hasn’t closed. The practical compromise is hybrid stacks that interleave efficient layers with occasional full-attention layers.
• “You evict old tokens from the KV cache and quality collapses. Why?” — attention sinks. Models dump excess attention mass on the first few tokens; removing them destabilises the softmax distribution. Pin the initial tokens and evict from the middle instead.