Context engineering

The discipline that replaced “prompt engineering” as the thing that actually determines quality in LLM applications. Prompt engineering is wording; context engineering is deciding what information occupies the window, in what order, at what cost.

The budget

Everything shares one window: system prompt, tool definitions, retrieved documents, conversation history, and the response.

budget = context_limit - max_output_tokens - len(tok(system)) - len(tok(tools))
# whatever remains is split between history and retrieved context

Tool definitions are the line people forget. Twenty tools with detailed schemas can consume thousands of tokens on every single call, before any content.

The ordering rule

Models attend most reliably to the beginning and end of the context and least reliably to the middle. That U-shape has direct consequences:

[system prompt]            <- stable, cacheable, start
[tool definitions]         <- stable, cacheable
[retrieved context]        <- most relevant items at the EXTREMES of this block
[conversation history]     <- oldest compressible
[current question]         <- LAST, adjacent to generation

Put the question at the end. Put the highest-scoring retrieved chunks at the edges of their block, not buried in the middle of a long list. See ../06_transformers_llm/08_context_windows.md.

Less is more

The counterintuitive rule worth stating explicitly: adding marginal context degrades quality. Extra chunks dilute attention and push relevant material into the weak middle region.

Retrieving 5 well-ranked chunks generally beats retrieving 50. If you’re passing everything you retrieved, you’re not doing context engineering — you’re doing context dumping.

Structure the window for caching

Prefix caching reuses the KV cache for identical leading tokens. That makes ordering a cost decision as well as a quality one.

STABLE   [system][tools][few-shot examples]     <- cached across all requests
VARIABLE [retrieved][history][question]         <- recomputed

Anything user-specific placed early destroys the shared prefix for every subsequent token. Roughly a 30% throughput gain sits on getting this right, and it costs nothing but discipline. See ../08_inference_serving/02_batching_and_serving_engines.md.

Managing history

The cache trap: summarising history to save tokens invalidates the prefix cache from the point of change, forcing a full re-prefill. A “token saving” can cost more than it saves. Measure both sides rather than assuming.

The practical pattern: summarise infrequently in large batches rather than continuously, so you pay the re-prefill rarely.

Memory beyond the window

Conversation history is short-term memory. Long-term memory is a separate store the agent reads from and writes to.

# Write: extract durable facts, don't store raw transcripts
facts = extract_facts(conversation)     # "prefers metric units", "on the Pro plan"
memory.upsert(user_id, facts)

# Read: inject only what's relevant to this turn
relevant = memory.search(user_id, query, top_k=5)

Two failure modes to name: memory bloat (storing everything until retrieval becomes noise) and stale memory (a fact that was true in March, asserted confidently in August). Both need a curation policy — extract facts rather than transcripts, timestamp them, and let them expire.

Compaction

When an agent’s context approaches the limit mid-run:

1. Keep the first message — it carries the task. Losing it is how an agent forgets what it was doing.
2. Keep the most recent turns verbatim — that’s the working state.
3. Summarise the middle, preserving decisions, established facts and constraints.
4. Externalise large artefacts — write them to a store, keep a reference.

if tokens(messages) > THRESHOLD:
    head, middle, tail = messages[:1], messages[1:-KEEP], messages[-KEEP:]
    messages = head + [summarise(middle)] + tail

Summarise decisions and constraints, not prose. “Chose Postgres over DynamoDB because of the join requirement” is worth keeping; a paraphrase of the discussion is not.

Sub-agents as context isolation

The cleanest way to keep a context small: delegate a subtask to a fresh context and return only the conclusion.

The parent never sees the sub-agent’s twenty tool calls — just the answer. That’s context isolation, and it’s the most defensible reason to use multiple agents at all. See ../10_agents_orchestration/04_multi_agent_patterns.md.

Interview angle

• “What’s context engineering, as distinct from prompt engineering?” — prompt engineering is how you word the instruction; context engineering is deciding what information occupies the window, in what order, and at what cost. It’s the larger lever in any real application.
• “How do you order the context window?” — stable content first (system, tools, examples) so it caches, the question last so it’s adjacent to generation, and the most relevant retrieved material at the extremes rather than the middle.
• “Should you retrieve more chunks to be safe?” — no. Marginal chunks dilute attention and push relevant content into the weak middle region. Fewer, better-ranked chunks generally win.
• “Your agent runs out of context mid-task. What do you do?” — keep the first message and the most recent turns, summarise the middle preserving decisions and constraints, and externalise large artefacts to a store with a reference in context. Note that this invalidates the prefix cache from the change point.
• “How does history management interact with cost?” — summarising to save tokens breaks prefix caching from that point, forcing re-prefill. Compact infrequently in large batches rather than continuously, and measure the net.
• “How do you give an agent long-term memory?” — a separate store written with extracted facts rather than raw transcripts, timestamped and expiring, retrieved by relevance per turn. The failure modes are bloat, where everything is stored until retrieval is noise, and staleness, where an outdated fact is asserted confidently.