LLM inference fundamentals

What actually happens when you call a model, why it costs what it costs, and which numbers to quote when someone asks about performance.

Two phases, two bottlenecks

Decode reads every model weight and the whole KV cache to produce a single token. Arithmetic intensity is terrible — huge data movement, trivial computation — which is why batching helps so much: read the weights once, serve many requests.

This asymmetry explains provider pricing (input tokens cheaper than output tokens) and is the first thing to say when asked how LLM serving works. See ../06_transformers_llm/05_kv_cache.md.

The metrics that matter

Latency and throughput trade off directly. Bigger batches raise throughput and raise per-request latency. There’s no configuration that maximises both, and saying so is the right answer to “how do you optimise serving” — you optimise for one under a constraint on the other.

For streamed UX, TTFT dominates perception. A response that starts in 300ms and streams feels faster than one that completes in 2s silently.

Sampling

response = client.chat.completions.create(
    model=MODEL,
    temperature=0.0,       # deterministic-ish for structured output
    top_p=1.0,
    max_tokens=512,        # ALWAYS set this
    stop=["\n\n"],
)

Use temperature or top_p, not both aggressively. For extraction, classification and structured output, set temperature to 0 — non-determinism in a parsed path is a bug, not creativity. Detail in ../00_math_foundations/04_information_theory.md.

Temperature 0 is not fully deterministic on GPU: batching changes floating-point reduction order, so identical requests can differ. If you need reproducibility, you need caching, not a temperature setting.

Streaming

async for chunk in await client.chat.completions.create(..., stream=True):
    if delta := chunk.choices[0].delta.content:
        yield f"data: {json.dumps({'text': delta})}\n\n"

Server-Sent Events is the standard transport — one-directional, works over plain HTTP, auto-reconnects. See ../../backend/12_protocols/sse/01_server_sent_events.md.

Three things that bite in production:

• Proxy buffering. nginx buffers responses by default and your stream arrives all at once. Set proxy_buffering off.
• Client disconnect. Detect it and cancel the upstream request, or you pay for tokens nobody receives.
• Errors mid-stream. You’ve already sent a 200, so failures must be signalled in-band.

Self-hosted vs API

The crossover is usually sustained high volume, data-residency requirements, or heavy fine-tuning. Below that, an API is almost always the right call, and saying so signals judgement rather than enthusiasm.

Cost model

cost = (input_tokens * input_rate) + (output_tokens * output_rate)

Output tokens typically cost several times input tokens, because decode is sequential and can’t be batched as effectively.

Levers, roughly in order of return:

1. Route by difficulty — cheap model for easy requests, frontier only when needed. Usually the biggest single win.
2. Prefix caching — stable content first; providers discount cached input.
3. Cap max_tokens — output is the expensive half.
4. Shorten prompts — or fine-tune the standing instructions into the weights.
5. Cache identical requests outright.
6. Batch offline work — providers discount asynchronous batch APIs substantially.

Reliability

LLM APIs fail more than typical backends: rate limits, timeouts on long generations, occasional malformed output.

• Retry with exponential backoff and jitter, distinguishing 429 (back off longer) from 5xx (retry) from 4xx (don’t).
• Set an explicit timeout — a long generation can hang far beyond your request budget.
• Validate the output, don’t trust it. Retry on schema violation.
• Have a fallback — a smaller model, a cached answer, or a graceful degradation path.
• Idempotency for anything that triggers side effects.

These are ordinary backend patterns; see ../../system_design/02_resilience/. Applying them to LLM calls is the part people forget.

Interview angle

• “What happens when you call an LLM?” — prefill processes the prompt in parallel and is compute-bound; decode generates tokens one at a time and is memory-bandwidth bound. Different bottlenecks, different metrics, and it’s why output tokens cost more than input.
• “Which metrics do you track?” — TTFT and inter-token latency for user experience, throughput and goodput for capacity, tokens and cost per request for economics. Latency and throughput trade off, so you optimise one under a constraint on the other.
• “How would you cut LLM spend by half?” — route by difficulty first, then prefix caching with stable content leading the prompt, cap max_tokens, shorten or fine-tune away the standing prompt, cache repeats, and move offline work to a batch API.
• “When is self-hosting worth it?” — sustained high volume, data-residency requirements, or heavy fine-tuning. Otherwise an API wins: no GPUs to run, no upgrades, and access to frontier models.
• “Temperature 0 — is output deterministic?” — not strictly on GPU, because batching changes floating-point reduction order. Use caching if you need genuine reproducibility.
• “Your streamed responses arrive all at once. Why?” — almost always proxy buffering. Disable proxy_buffering in nginx and confirm nothing else along the path buffers.