Context windows

Advertised context length and usable context length are different numbers. Knowing why, and how to verify it, is a practical seniority signal.

What the number means

The context window is the maximum total tokens the model processes — system prompt, tools, history, retrieved documents, and the generated response. It’s a shared budget, not an input allowance.

available_for_input = context_limit - max_output_tokens - overhead

Filling the window with input leaves no room to answer. The failure mode is a truncated or empty response rather than a clean error, which makes it annoying to diagnose.

Why long context is expensive

Attention is quadratic in sequence length, so doubling the prompt roughly quadruples attention cost at prefill. The KV cache grows linearly, and at long contexts it dominates GPU memory — see 05_kv_cache.md.

Concretely: time-to-first-token grows with prompt length, decode slows as the cache grows, and concurrency drops because fewer requests fit in memory. Long context is not free just because the API accepts it.

Lost in the middle

The finding that matters most in practice: models attend most reliably to the beginning and end of the context, and least reliably to the middle.

Retrieval accuracy plotted against position in the context is U-shaped. A fact placed in the middle of a long prompt is measurably more likely to be missed than the same fact at either end.

Design consequences:

Put the most important material at the start or the end. For RAG, that means the highest-scoring chunks at the extremes, not buried mid-list.
Put the question after the documents, so it’s at the end and adjacent to generation.
Fewer, better chunks beat more chunks. Retrieving 50 documents when 5 suffice actively hurts — it dilutes attention and pushes relevant content into the weak middle region.

That last point contradicts the intuition that more context is safer, and it’s a good thing to say unprompted.

Needle-in-a-haystack, and its limits

The standard long-context test: hide a specific fact in a long document and ask for it. Models pass this at impressive lengths.

It’s a weak test. Retrieving one verbatim string is far easier than:

Aggregating across many positions (“how many times does X appear”)
Reasoning over dispersed facts (“does the contract in section 3 conflict with section 47”)
Ordering or comparing across distant regions

A model that scores 99% on needle retrieval at 128k can still fail badly at multi-fact reasoning over the same 128k. Evaluate on your actual task shape, not on the vendor’s benchmark.

Extended windows are not uniform quality

Most very long windows come from rescaling RoPE after pretraining at a shorter length — position interpolation, NTK-aware scaling, YaRN. See 04_positional_encoding.md.

That extension works, but quality typically degrades as you approach the advertised maximum. A “128k model” pretrained at 8k and rescaled is not equally capable across the whole range.

The practical test: run your own task at 8k, 32k, 64k and 128k and plot accuracy. Where it falls off is your real limit.

Long context vs RAG

A recurring interview question, and “just use long context now” is the wrong answer.

	Long context	RAG
Cost per query	grows with corpus size	roughly flat
Latency	grows with prompt	flat
Corpus size	bounded by the window	unbounded
Freshness	re-send everything	update the index
Attribution	weak	strong — you know what was retrieved
Accuracy on the relevant span	degrades with dilution	good if retrieval is good
Failure mode	silently misses mid-context facts	retrieves the wrong thing, visibly

RAG remains correct for large or changing corpora, for cost control, and for citations. Long context is better for a single bounded document you need reasoned over holistically — a contract, a codebase module, one long transcript.

The 2026 default is both: retrieve to narrow the candidate set, then give the model generous context over what survived. See ../09_rag_embeddings/.

Managing the budget

Strategies when history exceeds the window, with their trade-offs:

Strategy	Loses
Truncate oldest turns	early context, permanently
Summarise older turns	detail, and summarisation errors compound
Re-retrieve relevant history per turn	continuity of implicit context
Hierarchical summary + recent verbatim	least, most complex

Note that summarising or editing history breaks prefix caching from the point of change, so every subsequent token must be re-prefilled. That’s a real cost against the token saving, and worth measuring rather than assuming. Covered further in ../12_context_engineering/.

Interview angle

“What actually limits usable context?” — quadratic attention cost at prefill, linear KV cache growth constraining concurrency, and the model’s degrading ability to attend to the middle of a long prompt. The advertised number is an upper bound on tokens accepted, not on tokens used well.
“What is ‘lost in the middle’?” — retrieval accuracy is U-shaped across context position: strong at the start and end, weakest in the middle. So order retrieved content by importance toward the extremes, place the question last, and don’t pad with marginal chunks.
“Models pass needle-in-a-haystack at 128k. Is long-context solved?” — no. Retrieving one verbatim string is far easier than aggregating or reasoning across dispersed facts. Test on your own task shape at several lengths and find where accuracy actually falls off.
“Long context has replaced RAG — agree?” — no. RAG wins on cost, latency, unbounded and changing corpora, and attribution. Long context wins on a single bounded document needing holistic reasoning. Production systems usually retrieve first, then use generous context on what survives.
“Retrieving more chunks should be safer, right?” — the opposite past a point. Extra chunks dilute attention and push relevant content into the weak middle region. Fewer, better-ranked chunks generally beat more.
“Your chat app slows down as conversations grow. Why, and what do you do?” — the KV cache grows so each decode step reads more, and prefill cost rises. Options are truncation, summarisation, or re-retrieval — noting that rewriting history invalidates the prefix cache and forces re-prefill.