Batching and serving engines

How a serving engine turns a bandwidth-bound, sequential decode loop into something that saturates a GPU. This is the part of LLM infrastructure most worth understanding, because the techniques explain the cost numbers.

Why batching is the whole game

Decode reads every model weight to produce one token. Serving one request wastes almost all the GPU’s compute — you’re moving gigabytes to do a trivial amount of arithmetic.

Batch 32 requests and you read the weights once for all 32. Throughput scales close to linearly with batch size until memory runs out; per-request latency rises modestly. That’s why serving economics are dominated by how many concurrent requests you can fit.

And what limits concurrency is the KV cache, not compute — see ../06_transformers_llm/05_kv_cache.md.

Static vs continuous batching

Static batching groups requests, runs them together, returns when all finish. The problem is that generation lengths vary wildly: a batch where one request generates 1,000 tokens and the rest generate 20 leaves the GPU processing mostly-finished sequences, with the batch slot held hostage until the longest completes.

Continuous batching schedules at the iteration level rather than the request level. After every forward pass, finished sequences leave the batch and waiting ones join.

Static:      [A B C D] ---- wait for D ---- [E F G H]
Continuous:  [A B C D] -> A done, E joins -> [E B C D] -> ...

This is the single biggest throughput win in LLM serving, and it’s why vLLM’s launch claimed order-of-magnitude improvements over naive serving. If asked “what makes vLLM fast”, continuous batching plus PagedAttention is the answer.

PagedAttention

KV cache memory has the same problems as process memory: naive allocation reserves a contiguous block sized for each request’s maximum possible length, most of which is never used.

PagedAttention manages the cache as fixed-size blocks with a per-sequence block table, exactly like OS virtual memory. Blocks are allocated on demand and needn’t be contiguous.

Two results:

• Fragmentation and over-allocation nearly vanish, so far more requests fit.
• Blocks can be shared between sequences with identical prefixes, copying only on divergence.

That sharing is what makes prefix caching possible.

Prefix caching

Requests that share a prefix — system prompt, tool definitions, few-shot examples — have identical KV for that prefix. Compute it once, reuse it.

vllm serve MODEL --enable-prefix-caching

Reported as the highest-value single flag for typical workloads: roughly a 30% throughput gain when requests share a system prompt, and considerably more with long shared prefixes.

The actionable design rule: put stable content first, variable content last.

[system prompt][tool definitions][few-shot examples][user query]   <- cacheable
[user query][system prompt]...                                     <- cache miss every time

Interleaving per-user data early destroys the shared prefix. This is a concrete thing to raise in a system-design interview and it costs nothing to get right.

Chunked prefill

A long prompt’s prefill occupies the GPU and stalls every in-flight decode, spiking inter-token latency for everyone else.

Chunked prefill splits a long prefill into pieces and interleaves them with decode steps, trading slightly worse TTFT for that request against much steadier ITL across the batch. Standard in current engines and the right answer to “one user’s 100k-token prompt is hurting everyone else”.

Speculative decoding

Decode is sequential and bandwidth-bound, so the GPU is idle most of the time. Speculative decoding exploits that: a small draft model proposes K tokens, and the large target model verifies all K in a single forward pass.

Verification costs roughly the same as generating one token, because it’s one batched pass. Accepted tokens are free; rejected ones fall back to normal decoding.

vllm serve MODEL --speculative-model SMALL_MODEL --num-speculative-tokens 5

2-3x decode speedup with no quality change when the acceptance rate is high — the verification step guarantees the output distribution matches the target model exactly, which is the property that makes it safe.

Variants: n-gram / prompt lookup (draft from the prompt itself, free, excellent for summarisation and code editing where output echoes input), Medusa/EAGLE (extra heads on the target model instead of a separate draft model).

The trade-off: it consumes memory and compute that could otherwise serve more concurrent requests. It helps latency at low batch sizes and can hurt throughput at high ones.

The engines

vLLM is the reference answer — by 2026 it’s the default serving engine for most open-model deployments. SGLang is the credible alternative, particularly for agent workloads with heavy prefix reuse.

Tuning checklist

Roughly in order of return:

1. --enable-prefix-caching, and structure prompts to exploit it.
2. FP8 KV cache — halves cache memory, doubles concurrency, negligible quality cost.
3. --max-model-len set to what you actually need, not the model’s maximum; it directly caps memory reserved per sequence.
4. --gpu-memory-utilization up to ~0.90-0.95 if nothing else shares the GPU.
5. Chunked prefill if long prompts are disrupting latency.
6. Speculative decoding if latency-bound at low concurrency.
7. Tensor parallelism only when the model genuinely doesn’t fit — it adds communication overhead.

Interview angle

• “What makes vLLM faster than a naive server?” — continuous batching (scheduling per iteration so finished sequences leave and new ones join immediately) and PagedAttention (block-based KV memory eliminating fragmentation and enabling prefix sharing).
• “What is continuous batching?” — iteration-level scheduling. Static batching holds a slot until the longest generation finishes; continuous batching swaps requests in and out every forward pass, so the GPU stays full.
• “How does PagedAttention help?” — treats KV cache as fixed-size pages with a block table, like virtual memory. Removes over-allocation and fragmentation, and lets sequences share physical blocks for common prefixes.
• “Explain speculative decoding.” — a small draft model proposes several tokens; the target model verifies them in one batched forward pass. Verification preserves the target’s exact output distribution, so it’s 2-3x faster with no quality change. It helps latency at low batch sizes and can cost throughput at high ones.
• “One user sends a 100k-token prompt and everyone’s latency spikes. Fix?” — chunked prefill, which interleaves pieces of that prefill with ongoing decode steps so a single long prompt can’t monopolise the GPU.
• “How do you structure prompts for cheaper serving?” — stable content first (system, tools, examples), variable content last, so the shared prefix hits the cache. Then enable prefix caching and quantise the KV cache to FP8.