BlogAI agents

AI agent memory explained: short-term, long-term, in between

A language model remembers nothing between calls. Everything we call memory is text that the application stores and puts back into the context window before the next call: the recent conversation, a summary of the older part, and facts saved from earlier sessions. Short-term memory is what the model can see now. Long-term memory is what is kept outside the window, and most of the engineering is in deciding what moves between the two.

A model has no memory of its own

A model’s weights are fixed when training ends. Talking to it does not change them. Each call is a function: text goes in, text comes out, and nothing is kept. OpenAI’s documentation says it directly: each text generation request is ‘independent and stateless’ unless you use the features that carry state for you. The same is true of the other large providers’ basic APIs.

So when an assistant refers to something you said ten minutes ago, the application sent your earlier messages again. When it greets you by name next week, something wrote your name to a database and read it back. Memory belongs to the software around the model, the part we describe in what an agent harness is.

This is more than a technicality. It turns the vague question of whether an assistant ‘remembers’ into three that an engineer can answer: what is stored, where, and what gets put back into the context for this call.

Most systems end up with these layers.

Layer What it holds How long it lasts How it works
Context window Everything the model sees on this call One call Assembled by code each time
Conversation history The turns of the current session One session Earlier messages sent again, or a thread kept by the provider
Summary A condensed version of older turns The rest of the session A summarising call when history grows long
Working notes Progress on a long task The task A file or record the agent writes and rereads
Long-term memory Facts and preferences that outlive sessions Until changed or deleted A store read at the start of a turn

The first two are short-term memory. The last is long-term. Summaries and notes are what sits between, and they are where most of the trouble is.

The context window as working memory

The context window is the only thing the model can see: instructions, tool definitions, retrieved documents, the conversation, and whatever memory the application chose to include. In the taxonomy borrowed from cognitive science, which we go through in types of memory in AI systems, this is working memory.

Take a property manager that looks after 3,000 rented flats and runs a tenant assistant on chat and WhatsApp. A tenant writes on Monday that the kitchen tap is leaking. On that call the window holds the system prompt, the tenant’s message and perhaps their tenancy record, fetched by a tool. That is all the model knows.

The window has three limits. It has a maximum size. Every token in it is paid for on every call. And models get less reliable at using any one detail as the window fills, an effect we cover, with the evidence, in context engineering. The practical consequence is that working memory has to be curated. You cannot keep adding to it.

Conversation history and where it runs out

Within a session, the simplest memory is to send the whole conversation again with each new message. Turn ten pays again for turns one to nine. For a short chat that is fine.

It stops being fine in two cases. One is the long conversation: the tenant who goes back and forth forty times about a deposit dispute. The other is the agent, where most of the history is tool output. A single call that returns a maintenance log can be larger than the whole chat that came before it.

Truncation

The cheapest fix is a sliding window: keep the last few turns and drop the rest. It fails in a predictable way. The start of a conversation usually holds the goal and the constraints, so the assistant keeps the small talk and forgets what it was asked to do.

Summaries and compaction

The better fix is to replace older turns with a summary and keep recent turns word for word. Providers now offer this as an API feature. Anthropic’s compaction, still marked as beta, writes the summary on the server, and OpenAI’s API has an equivalent. You can also do it yourself with one extra model call, which has the advantage that you can read what was kept.

A summary is lossy by design, and what it loses is exact detail: the reference number, the date the contractor promised, the tenant’s own wording of the complaint. Two rules help. Tell the summariser to carry identifiers, amounts, dates and commitments through unchanged. And keep anything that must not be lost outside the conversation altogether.

Working notes

That second rule is the idea behind working notes. An agent on a long task writes its plan and progress to a file or a record, and reads it back after its history has been compacted or the process has restarted. Anthropic’s memory tool builds the habit in. According to its documentation, the API adds an instruction telling the model to assume its context may be reset at any moment and to record progress in its memory directory. We go further into this in long-running AI agents.

Long-term memory: what is kept outside the window

Long-term memory is anything written to storage that a later session can read: a tenant’s preferred contact hours, the fact that they have a dog that must be shut away before a contractor visits, the outcome of last month’s repair.

Two design choices define it.

How it is read. A small set of facts can be loaded into every call, like a profile card. A larger set has to be searched, either by the application before the call or by the model through a tool when it decides it needs something. Most systems do both: a short profile that is always present and a searchable store behind it, often the same vector search that is used for documents.

How it is written. Memories can be saved explicitly, when the user says ‘remember this’ or the model calls a save tool. Or a separate process can read finished conversations and extract what is worth keeping. Explicit saving is predictable and misses things. Automatic extraction catches more and saves some things it should not.

Either way, the store is now a data system. It will hold facts that were true once, guesses recorded as facts, and personal details the user never expected to be kept. How to write, update and expire memories is the subject of our build guide for long-term memory. What you are allowed to keep, and how to delete it, is in AI memory and privacy.

How is memory different from RAG?

Both put retrieved text into the context, and both often use the same search technology, so the two get confused. The difference is where the content comes from and who it belongs to.

Retrieval over documents (RAG) Memory
Source Documents your organisation wrote or approved The assistant’s own conversations and actions
Scope Usually shared across users One user, account or agent
Written by An ingestion pipeline The system itself, during or after use
Typical failure A wrong or outdated passage is retrieved A wrong inference is saved and repeated

In the tenant example, the repairs policy and the tenancy handbook are retrieval-augmented generation: written by the company, the same for everyone. ‘Prefers texts after six, has a dog’ is memory.

There is a third category that is neither, and teams often get it wrong. The status of the open repair ticket lives in the maintenance system. The assistant should look it up with a tool every time. If it memorises ‘plumber booked for Thursday’ and the booking moves, memory now contradicts the system of record, and the model has no way to know which is right. Our rule: if a system of record holds the fact, fetch it, and never store a copy as a memory.

What do the main assistants and platforms offer now?

This is the position at the time of writing, taken from each vendor’s own documentation. These features change often, so check before you rely on a detail.

ChatGPT. OpenAI’s memory documentation describes two mechanisms: saved memories, which are explicit items, and reference to chat history, which draws on past conversations and changes over time. A newer mode keeps a broader summary that is updated continually. Users can review a memory summary, see which sources personalised a response, and correct or delete items. Deleting a chat does not delete a saved memory that was created from it, and temporary chats do not create memories.

Claude. Anthropic’s support page says Claude saves memory as individual topics during chats, not as summaries written afterwards. Each project has its own memory. Users can read, edit and delete topics, pause memory or reset it, and incognito chats stay out of it. Separately, paid plans can search past chats, which the page describes as retrieval-augmented generation that shows up as a tool call.

Gemini. Google’s help page describes a Memory setting, under Personal Intelligence, that lets Gemini learn from past chats. It needs activity history to be switched on and is not offered on work or school accounts. To make Gemini forget something, you delete the chats that contain it.

For builders. Anthropic’s memory tool is client-side: the model asks to view, create, edit or delete files under a memory directory, and your application carries out each request against storage you control. OpenAI’s API offers conversation objects that hold a thread on the server, plus compaction. Google and AWS sell managed memory services that extract memories from sessions, which we compare with the open-source options in the build guide.

Two patterns stand out. ChatGPT and Claude each pair something curated (saved items or topics) with something searched (past chats). And all three give the user a way to correct or remove what is held. Anyone building an assistant now is building for users who expect both.

When does an agent need memory, and when not?

Memory costs engineering time, tokens on every call and a new class of error, so it should earn its place.

It usually does when the same people come back and have stable preferences, when a piece of work runs across several sessions, when users correct the assistant and expect the correction to stick, or when a single task is too long for one context window. The last of these needs working notes, not a user profile.

It usually does not when each request stands alone (classifying an email, extracting fields from an invoice), when users are anonymous, when the facts already live in a system the agent can query, or when a wrong remembered fact would cost more than asking again. A tenant assistant that remembers the wrong flat number is worse than one that asks.

Before adding long-term memory, we ask five questions.

  1. What do users have to repeat today? Read real transcripts and count.
  2. Is each of those facts already in a system we could query instead?
  3. Could ten fixed profile fields cover it, with no automatic extraction?
  4. What happens when a remembered fact is wrong, and how would the user correct it?
  5. Who is allowed to see what is stored, and how is it deleted?

If the first question turns up little, stop there.

How to start

In the first week, pull fifty real conversations and mark every place where a user repeated something or the assistant lost the thread. Sort each into one of three groups: lost within the session, lost between sessions, or available in another system. That tells you which layer to fix.

Fix short-term memory first. Cap history, summarise older turns with a prompt you can read, and keep identifiers unchanged. This is cheap, and it removes most complaints that sound like ‘it forgot’.

Then make the first decision about long-term memory a small one: a short, explicit profile that the user can see and edit, before any automatic extraction. Add extraction only when the profile is plainly not enough. If you would like a second opinion on where your assistant is losing context, this is part of the conversational AI and agent work we do at WeStack AI.

Questions

Asked often.

Do AI models remember previous conversations?

Not by themselves. A model's weights do not change when you talk to it, and each call starts from nothing. An assistant appears to remember because the application stores earlier messages or saved facts and sends the relevant ones back to the model with every new request. Memory is a feature of the software around the model.

What is the difference between short-term and long-term memory in an AI agent?

Short-term memory is what sits in the context window for the current call: the recent conversation, tool results and instructions. It disappears when the session ends or the window fills. Long-term memory is information kept in a database or files outside the window, such as user preferences, and read back in later sessions.

Is agent memory the same as RAG?

No, although both retrieve text and place it in the context. RAG searches documents your organisation wrote, shared across users. Memory holds what the assistant learned from its own interactions, usually about one user or account, and the system writes it itself. That makes memory harder to keep accurate and gives it stricter privacy duties.

Does every AI agent need long-term memory?

No. One-off tasks such as classifying a ticket or extracting fields from an invoice need none, and anonymous help bots rarely do either. Long-term memory earns its cost when the same people return, when work spans several sessions, or when users keep repeating the same preferences. Facts held in a system of record should be looked up, not memorised.

Not on the list? Ask us directly

Keep reading

More notes from the build.

All blogs

Let’s build your stack.

Send a brief and we’ll reply with next steps.