Memory Strategies for LLM Conversations and Agents

How to give an LLM “memory” across turns and sessions. The naive answer is “send the whole history” — works until you hit the context window. Real systems use a layered approach.

The context-window problem

Modern LLMs have 8k–1M-token context windows. “Just include everything” works for short chats; falls apart for:

• Long conversations (1000+ turns).
• Multi-session memory (user comes back tomorrow).
• Agents accumulating tool results.
• Multi-user / multi-tenant context per request.

Cost also matters: a 100k-token prompt to GPT-4 is ~$1 per call. At scale, context bloat is real money.

The memory taxonomy

You’ll mix several in any non-trivial agent.

Strategy 1: Sliding window

Keep the last N turns in the prompt; drop older ones.

def trim_history(messages, max_messages=10):
    return messages[-max_messages:]

Pros: dirt simple. Cons: anything earlier is gone forever. Bad UX when the user references “what we talked about earlier.”

Strategy 2: Token budget

Trim history to fit within a token budget; preserve recency.

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")

def trim_to_budget(messages, max_tokens=4000):
    out = []
    used = 0
    for msg in reversed(messages):
        cost = len(enc.encode(msg["content"])) + 4
        if used + cost > max_tokens:
            break
        out.append(msg)
        used += cost
    return list(reversed(out))

Production-grade variant: weight system messages as high-priority; trim earlier user/assistant pairs first.

Strategy 3: Summarization

When history exceeds threshold, summarize older turns into a single message. Newer turns stay verbatim.

async def maybe_summarize(messages):
    if total_tokens(messages) < THRESHOLD:
        return messages

    keep = messages[-6:]                  # last 6 turns verbatim
    older = messages[:-6]

    summary = await llm.summarize(
        "Summarize this conversation, preserving names, decisions, "
        "and any open questions:\n\n" + format(older)
    )

    return [
        {"role": "system", "content": f"Previous conversation summary: {summary}"},
        *keep,
    ]

LangChain’s ConversationSummaryBufferMemory does this. Trade-off: summary fidelity. Summaries lose detail; multiple rounds of summarizing compound the loss.

Strategy 4: Vector store retrieval

Store each turn (or chunk of turns) as embeddings. At each new turn, retrieve the K most relevant past turns to include.

async def get_relevant_history(query, user_id, k=5):
    query_embedding = await embed(query)
    results = await vector_store.search(
        query_embedding,
        filter={"user_id": user_id},
        top_k=k,
    )
    return [r.metadata["message"] for r in results]

Pros: scales to unbounded history; recall is by relevance, not recency. Cons: misses cross-references the query doesn’t surface (“what was the third item I mentioned?”). Expensive (embedding + vector search per turn).

ConversationVectorStoreMemory in LangChain.

Strategy 5: Hybrid recency + relevance

Combine: keep the last N turns verbatim (recency) + retrieve K relevant older turns (semantic).

async def build_context(query, user_id, history):
    recent = history[-6:]
    relevant_older = await retrieve_relevant_older(query, user_id, history[:-6], k=5)

    return recent + relevant_older

Most production agent memories end up here. Recency for “what just happened”; semantic for “things I should remember.”

Strategy 6: Entity / structured memory

Extract structured facts as the conversation progresses; store them separately.

{
  "user_id": 42,
  "name": "Alice",
  "preferences": {"language": "Python", "tone": "concise"},
  "open_topics": ["payment integration", "deployment"],
  "decisions": [{"topic": "DB", "value": "Postgres", "date": "2024-03-12"}]
}

Extract via LLM:

async def extract_facts(turn):
    prompt = f"Extract any user preferences, decisions, or important facts mentioned. Return JSON.\n\nTurn: {turn}"
    facts = await llm.parse(prompt, output_schema=FactExtraction)
    update_user_state(user_id, facts)

Include the structured state in every prompt:

system_msg = f"""You are an assistant. Here's what we know about this user:
{json.dumps(user_state)}

Conversation:
"""

LangChain’s ConversationEntityMemory operationalizes this. Production agents (Anthropic’s Computer Use, OpenAI’s memory feature) use variants of this pattern.

Strategy 7: Multi-tier memory

Different stores for different lifetimes:

A turn might pull from all four:

• Working: last 6 messages.
• Episodic (short): 3 relevant turns from the past week.
• Episodic (long): structured facts about the user.
• Semantic: 5 chunks from the knowledge base relevant to the question.

Assembled into one prompt within the context budget.

Memory eviction

Vector stores grow forever; need eviction.

Strategies:

• TTL — drop messages older than N days.
• LRU — drop messages with no recent retrieval.
• Importance scoring — LLM-assigned importance per turn; drop low-importance.
• Summarize-and-evict — compress old turns into summaries, evict the originals.

For per-user memory, total size is usually bounded (a user has finite interaction history); the bigger issue is unlimited free trial users accumulating storage.

Multi-user / multi-tenant

Vector store filter by user_id always:

results = await vector_store.search(
    query_embedding,
    filter={"user_id": current_user_id},
    top_k=k,
)

Or one collection per tenant — depending on the store. Don’t accidentally retrieve another user’s memory; that’s a privacy bug.

Memory and tool use

Agent tools (web search, function calls) generate context that should be remembered:

turn_history = [
  {"role": "user", "content": "Look up the current weather in Berlin"},
  {"role": "assistant", "tool_calls": [{"name": "weather", "args": {"city": "Berlin"}}]},
  {"role": "tool", "content": "{\"temp\": 15, \"conditions\": \"cloudy\"}"},
  {"role": "assistant", "content": "It's 15°C and cloudy in Berlin."}
]

You can summarize tool outputs aggressively — usually only the assistant’s interpretation matters, not the raw tool response. Reduces token cost.

LangGraph’s state pattern

LangGraph (agentic workflow framework) uses a typed state passed across nodes. Memory becomes part of the state, with reducer functions:

class State(TypedDict):
    messages: Annotated[list, add_messages]
    user_facts: dict
    decisions: list

# Each node may update state; reducers merge updates
def remember_decision(state):
    return {"decisions": [{"topic": "...", "value": "..."}]}

State persists across turns via checkpointer (Postgres, SQLite). Cleaner than hand-rolled “load memory, run, save memory” loops.

Cost-aware memory

Token cost matters. Strategies:

• Cache the summary — only recompute when history grows; otherwise reuse.
• Smaller model for summarization — use Haiku / GPT-4o-mini for summarizing; bigger model for actual responses.
• Prompt caching — Anthropic + OpenAI support caching static prefixes; put the long system prompt + user facts at the start to benefit.
• Aggressive truncation of tool outputs in history.

Common pitfalls

• Sending the full history every turn. Works until you hit the token limit; then breaks suddenly.
• Summarization losing critical details. The summary forgets “user mentioned they’re allergic to X” → harmful response 10 turns later.
• Vector retrieval missing cross-references. “What was that thing we discussed?” doesn’t match well by embedding.
• No eviction policy. Memory grows forever; cost grows linearly; performance drops.
• Mixing memory across users. Privacy violation. Always filter by user_id.
• Hot summary loop. Summarize on every turn = compounding lossy compression. Trigger only above threshold; cache summary.

Tools

Interview angle

• “How do you handle conversation history that exceeds the LLM’s context window?” — multi-strategy: keep last N turns verbatim (recency); summarize older turns; retrieve semantically relevant past turns via vector store; structured-extract user facts to persist. Layered approach is standard.
• “What’s ConversationSummaryBufferMemory?” — LangChain’s hybrid: keep recent turns verbatim; summarize older turns into a single message when the buffer overflows. Recency + compression.
• “How do you store memory across sessions?” — vector store keyed by user_id (per-user retrieval) + structured DB for extracted facts (preferences, decisions, identities). Both persist across sessions; the vector store handles semantic recall; the structured DB ensures critical facts aren’t lost to summarization.
• “What’s the privacy concern with shared vector stores?” — without proper user_id filtering, a user’s query could match another user’s stored turns; that’s a data leak. Always filter retrievals by tenant/user.
• “How does memory work with tool-using agents?” — turns include tool calls + tool responses. Tool responses are often verbose; summarize aggressively (keep the assistant’s interpretation, drop the raw tool output) to manage context cost.
• “Cost-aware strategies?” — prompt caching for static prefixes; smaller model for summarization; cache the running summary instead of recomputing; aggressively trim tool outputs. Token budget is real money at scale.