5 · Memory and RAG

We’ve said it twice now, so let’s take it seriously: models are stateless. Every request carries everything the model is allowed to know, and when the response comes back, it has already forgotten.

“Memory” is therefore not a feature the model has. It’s a decision you make about what to put back in. This page is about the two different decisions people mean by that word — and confusing them causes most agent memory bugs.

Two kinds of memory

	Working memory	Long-term memory
Lives for	one run	forever
Contains	this conversation, tool results	facts, documents, past runs
In our project	`runs.messages` and the `steps` table	the `memories` table
Grows	fast, within a run	slowly, across runs
Problem it has	fills the context window	finding the relevant bit

Working memory is the agent’s scratchpad — the growing pile of “you asked this, I searched that, here’s what came back” that gets re-sent on every turn. It’s what makes turn six aware of turn one. Its problem is size: it grows every turn, and the context window is finite.

Our project keeps this twice, on purpose. runs.messages is the version the model sees — every question, tool call and result, in the shape it expects back. The steps table is the same story told for humans, and because it’s in a database rather than in RAM, the UI can read the agent’s progress while it’s still running and a crashed run leaves evidence behind.

Long-term memory is anything that should outlive the run — your documents, your notes, what happened last week. Its problem is the opposite one: you have far too much of it to send, so you need to find the relevant slice first.

That “find the relevant slice” job is what RAG does.

RAG, without the mystique

RAG — Retrieval-Augmented Generation — means: find relevant text, paste it into the prompt, ask the question.

That’s genuinely all it is. Once you know the model is stateless, RAG stops sounding like a technology and starts sounding like the obvious move.

The only interesting part is “find relevant”. Keyword search finds documents containing your words. Vector search finds documents that mean something similar — “car” matches “automobile”, because the two end up close together in a space of numbers.

1. Turn each document into a vector (a list of ~768 numbers) — "embedding"
2. Store the vectors
3. Turn the question into a vector too
4. Find the stored vectors closest to it
5. Paste those documents into the prompt

Step 4 is the only step that needs anything special, and it’s what a vector database does. Postgres does it with the pgvector extension, which is why this project doesn’t need a second service:

create extension if not exists vector;

create table if not exists memories (
  id          bigserial primary key,
  content     text not null,
  embedding   vector(768),     -- must match your model's output size
  created_at  timestamptz not null default now()
);

Similarity search then looks like ordinary SQL — <=> is cosine distance, and smaller means closer:

select content, 1 - (embedding <=> '[0.1, 0.2, ...]'::vector) as similarity
from memories
order by embedding <=> '[0.1, 0.2, ...]'::vector
limit 5;

RAG or tools?

These two solve the same problem — getting information the model doesn’t have — from opposite directions, so it’s worth being explicit:

	RAG	Tools
When	before the model runs	during the run
Who chooses	you	the model
Good when	you know what’s relevant	the model must decide what it wants

Our agent uses tools: it searched Wikipedia because it decided to, twice, with different angles. A RAG version would have searched your documents up front and handed the results over without being asked.

Most real systems use both — RAG for the context you know they’ll need, tools for the things only the model can decide it wants.

Working memory, actually built

The demo has a conversation in it, and the whole of “the agent remembers” is two database round trips. Ask a question, then ask “why is that?” — a phrase that means nothing on its own. It works because the server reloads the conversation and replays it to the model, which remembers nothing by itself:

history = await db.get_thread_messages(thread_id)      # load it

result = await agent.run(query, message_history=history, ...)

await db.update_run(                                    # save it back
    run_id, status="done", result=result.output,
    messages=to_jsonable_python(result.all_messages()),
)

all_messages() is the conversation from its very first question — every question, tool call, tool result and answer — so the newest run in a thread always holds the whole of it, and resuming is a one-row read.

Three things worth taking from that:

Memory is a column, not a feature. One run is one question; a conversation is several runs sharing a thread_id. Nothing about the agent changed.

The browser holds none of it. The client’s entire share is: keep the thread_id you were handed, send it with the next question. Which is why reloading the page doesn’t lose your conversation — it was never in the tab. GET /threads/{id} reads it back out of Postgres.

It has to be capped. Every turn re-sends the whole history, so a thread gets more expensive per question the longer it runs, and eventually stops fitting in the context window at all. That’s the “grows fast” row of the table above, arriving in person. MAX_THREAD_TURNS turns it into a 409 that says start a new conversation, which beats a run that dies three minutes in with a token error.

What we left as your extension point

memories and pgvector are set up in the schema and go no further. The workshop is about deployment, and a full RAG pipeline is a session of its own.

If you want to go further: embed your documents, store the vectors, and add a search_memory tool. Notice that the agent loop doesn’t change at all — you’re just giving it one more door, exactly as page 3 described.

database/schema.sql — all three tables, with the dimension warning in place.
app/db.py — add_step() is working memory in four lines; get_thread_messages() is the conversation in ten.
app/agent/research.py — load, run, save.

Where this leaves us

That’s five of the six parts from the anatomy: a model that decides, instructions that shape it, tools that let it act, other clients that can borrow those tools, and memory that carries the run forward.

Which means we’ve built something that can now take real actions in the world, choose its own arguments, and keep going until it’s satisfied — with nobody watching.

Time to talk about what stops it.

Next: Guardrails →