Context Window and Memory in LLMs

Every large language model has a context window — the maximum amount of text (measured in tokens) it can process at once. Everything inside this window is the model’s “working memory.” Understanding context windows and their limitations is essential for building reliable LLM-powered applications.

What Is a Context Window?

A context window defines how much text an LLM can “see” at once during a single inference call. It includes the system prompt, conversation history, retrieved documents, and the user’s message. The model has no memory outside this window — every request is stateless.

Token counts for common models (approximate):

One token ≈ 4 characters or ¾ of a word in English.

Why Context Limits Matter

When a conversation or document exceeds the context window:

• Older messages are dropped (truncated), losing earlier context.
• The model cannot reference information from outside the current window.
• Costs scale with token count — large contexts are expensive.

Even within the window, models can struggle with the “lost in the middle” problem: information in the middle of a long context is often recalled less reliably than information at the beginning or end.

Types of Memory in LLM Systems

Since LLMs are stateless, applications must implement memory explicitly. There are four main types:

1. In-Context Memory (Working Memory)

Information stored directly in the current context window — conversation history, retrieved documents, instructions. Fast and reliable but limited by window size and cost.

2. External Memory (RAG)

Large knowledge bases stored in vector databases. Relevant chunks are retrieved and inserted into the context on demand. This is Retrieval-Augmented Generation (RAG). Allows access to virtually unlimited information but requires semantic search to find the right pieces.

3. Episodic Memory

Summaries or records of past conversations stored externally and retrieved when relevant. Lets the model “remember” prior interactions with a user without storing full transcripts.

4. Semantic / Parametric Memory

Knowledge encoded in the model’s weights during training. The model “knows” facts about the world without being told — but this knowledge is static and can be outdated or incorrect.

Strategies for Managing Context

Summarization

Compress old conversation turns into a rolling summary before they would be truncated. The summary is prepended to future context to preserve important information in fewer tokens.

Sliding Window

Keep only the most recent N turns in context, dropping the oldest. Simple but loses information from earlier in the conversation.

RAG (Retrieval-Augmented Generation)

Store all relevant documents in a vector database. Retrieve only the most relevant chunks at query time. Efficient for large, static knowledge bases.

Memory-Enabled Agents

Frameworks like LangChain, LlamaIndex, and MemGPT implement dedicated memory modules that classify, store, and retrieve memories intelligently, simulating long-term memory on top of stateless LLMs.

Practical Implications

• Design prompts to be concise — every token counts.
• Use structured summaries for long conversations.
• Chunk documents appropriately for RAG — too large and retrieval is imprecise, too small and context is fragmented.
• Monitor token usage to control costs and stay within limits.
• Don’t assume the model remembers — always verify what context is present.