Agent memory: what should persist, and what should not

In December 2025 a benchmark study compared memory systems for AI agents. Among the entrants was a control condition: a plain filesystem storing memories as markdown files, retrieved by reading them.

It scored 74%, beating several purpose-built vector memory systems.

That result should be read carefully rather than as an argument against sophisticated storage. What it says is that the difficulty in agent memory was never keeping bytes. Any system can keep bytes. The difficulty is deciding what is worth keeping, retrieving the right thing later, and knowing when a stored fact has stopped being true.

An agent's memory is a database of assertions about a world that keeps changing, with no expiry dates and no mechanism for noticing contradiction. Stale records and entity conflicts are the leading source of production accuracy degradation, and they get worse the longer a system runs, which means memory is the one component that degrades from success rather than from failure.

The persistence gap

Start with what actually breaks, because it is specific.

Agents typically perform well within a single session and fail across sessions. The reason is structural: working memory, the active context during a conversation, is volatile and discarded when the session ends. Most systems never promote anything from it into durable storage.

The result is an agent that appeared attentive an hour ago and starts from nothing tomorrow. The user experiences this as forgetting. What actually happened is that nothing was ever written down, because deciding what to write down is a design decision nobody made.

The inverse failure is equally common and less visible. Systems that do write everything accumulate an undifferentiated pile in which the important and the incidental have equal standing, and retrieval quality falls as the pile grows.

The four layers

The field converged on a taxonomy borrowed from cognitive science, and it is useful because each layer has a different retention question.

Working memory. The active context during a session. Volatile by design. The question is what to promote before it disappears.

Episodic memory. What happened. This conversation, that decision, the error last Tuesday. Time-stamped and specific. The question is how long an episode stays relevant.

Semantic memory. What is true. The user prefers metric units, this account is on the enterprise plan, that API returns dates in ISO format. Extracted from episodes rather than recorded directly. The question is what happens when a new episode contradicts it.

Procedural memory. How to do things. Conventions, workflows, patterns that worked. In coding agents this is often a plain instruction file in the repository, which turns out to be a durable and maintainable form of it.

The taxonomy matters because most memory failures are a layer confusion. Storing an episode as though it were a fact means an offhand remark becomes a permanent belief. Storing a fact as an episode means it is retrieved only when the conversation resembles the one where it was learned.

The three questions to ask of every stored item

Most memory design goes wrong at the point of writing rather than the point of reading, and three questions asked at write time prevent most of what follows.

Is this an event or a belief? "The user asked for a summary on 3 March" is an event and is permanently true. "The user prefers summaries" is a belief and may stop being true tomorrow. Storing the second as though it were the first is how an offhand remark becomes doctrine. Events are safe to keep forever; beliefs need an expiry and a supersession path.

Would this be wrong if the world changed? Anything about a person's role, a system's configuration, a price, a policy or a preference will eventually be wrong, and the store has no way to learn this. Anything about what physically happened will not. Sorting by this property alone tells you which records need review and which do not.

Who is this true for? A fact about one user is not a fact. A memory written without a scope becomes a memory retrievable by anyone whose query is similar enough, which is the mechanism behind most cross-user leakage. Scope should be written at the same time as the content, not inferred later from the query.

The reason to ask these at write time is that the information required to answer them exists then and does not exist afterwards. A stored assertion with no record of whether it was an event or an inference cannot be classified retrospectively, which is why a store built without these degrades into an undifferentiated pile that can only be trusted wholesale or not at all.

Staleness is the main event

Everything else in this article is secondary to this.

A frequently retrieved memory stays accurate until reality changes, and then it becomes confidently wrong. Nothing in the system notices. There is no expiry, no confidence decay, and no mechanism that flags a contradiction between a six-month-old preference and last week's.

Without explicit versioning and conflict resolution, an agent treats a stale memory as exactly as authoritative as fresh context. Every long-lived agent with a write-once store hits this eventually, and it arrives gradually enough that nobody attributes the degradation to memory.

The practical shape: a user mentioned in March that they preferred summaries. In September they asked for full detail three times. A well-built memory has updated. A typical one has both records, retrieves whichever is more semantically similar to the current query, and produces the wrong format with complete confidence.

Stale records and entity contradictions are reported as the top source of production accuracy degradation in memory-equipped agents. Not retrieval failure, not model capability. Facts that used to be true.

The mitigations are unglamorous and they need to exist from the first deployment rather than being added later: time-to-live on episodic records, explicit versioning of semantic facts with the superseded version retained, conflict detection when a new assertion contradicts a stored one, and periodic audits comparing stored beliefs against reality.

Retrieval hallucination

The second failure is more subtle and produces output that reads as grounded.

Memory retrieval usually works by embedding similarity, and embedding similarity is not factual relevance. A query about Python memory management retrieves a stored note about Python memory profiling tools. Semantically adjacent, factually orthogonal. The agent incorporates it with high confidence and produces an answer that sounds well-sourced and is contaminated.

This is worse than retrieving nothing, because retrieving nothing produces visible uncertainty while retrieving the wrong thing produces invisible error.

There is a specific and widely underused finding here: accuracy tends to peak at roughly three to five retrieved items and degrade beyond that, as noise overwhelms signal. Systems that retrieve twenty memories to be safe are actively worse than systems that retrieve four, and the instinct to retrieve more is exactly backwards.

The countermeasures are ordinary information retrieval practice: filter by metadata before computing similarity to narrow the candidate pool, combine similarity with keyword matching rather than relying on embeddings alone, weight by recency, and re-rank before the results enter the prompt.

Memory as an attack surface

Persistent state creates a category of vulnerability that stateless systems do not have, and the research literature has named several.

Memory poisoning. Injecting content into long-term memory to influence future behaviour. The injection happens once; the effect persists across every subsequent session.

Sleeper poisoning. A variant where a fabricated memory is stored from manipulated context, remains dormant, and resurfaces later to steer behaviour. The gap between injection and effect makes attribution nearly impossible, since the session that caused the problem is long closed.

Temporal contamination. Agent safety measurably degrades as memory accumulates across unrelated tasks. This one is not adversarial. It is what happens by default.

Memory extraction. Black-box attacks that recover private data from an agent's memory store without any access to the model, by querying in ways that cause it to surface what it holds.

The governance consequence is direct: memory contents are subject to the same scrutiny as any other data store, and are frequently more sensitive. They contain what users said, what was inferred about them, and what the agent concluded, which raises the same jurisdictional questions as any other store, which is a richer record than the database the conversation drew from.

Scope isolation is the control that most deployments lack. Can agent A read agent B's memories? Can user X's context reach user Y? These are permission questions rather than storage ones. In multi-agent systems where memory is shared to improve coordination, the answer is often yes by design and nobody has written down the consequences.

Why larger context windows do not solve it

A reasonable objection: context windows now exceed a million tokens, so why not put everything in the prompt?

Because context length and persistence are different properties. Context windows have grown enormously and persistence across sessions remains zero. A million-token window is a larger working memory for one complex session. It is not a memory system, and when the session ends it is gone.

There is also a quality argument. Benchmarks measuring multi-session recall find that as conversation history grows, performance degrades faster than the context grows. More history does not linearly buy more competence; it buys more material to be confused by. Systems that score well on memory accuracy frequently require upwards of 26,000 tokens per query, which is not viable at production volume.

The conclusion the deployment literature reached is that external memory is the right architecture rather than a workaround for small context, and longer windows are best understood as expanded working memory for hard single-session tasks.

The append-only argument, unresolved

One design question has serious people on both sides and is worth stating rather than resolving.

Append-only stores never update or delete, only add. The argument is that mutation creates decoherence, where what the agent believes it remembers diverges from what actually happened, and an immutable log with newer records superseding older ones preserves the audit trail.

Against append-only: without time-to-live policies the store grows without bound, retrieval quality falls as the pile deepens, and every stale record remains retrievable forever. A store that never forgets is a store where every obsolete fact is one similarity match away from being used.

Both are correct about their failure mode. The reconciliation most systems reach is append-only storage with retrieval-time filtering, so history is preserved while only current records are eligible to be returned. Whether that is a synthesis or a compromise that inherits both problems is not established.

What memory costs, which nobody models

Memory is presented as a capability and it is also a line item, in three places that do not appear in the usual estimate.

Every retrieved memory is billed on every step it survives. A memory entering the context at step two of a twelve-step workflow is re-sent ten more times. Retrieving five memories of two hundred tokens each adds a thousand tokens to step two and roughly ten thousand across the run, which is why the three-to-five retrieval finding is an economic result as much as an accuracy one.

Writing costs inference. Deciding what to promote from working memory usually means a model call to summarise or classify, once per session or once per turn depending on the design. At volume this is a second inference workload nobody budgeted, running on every interaction rather than only the ones that produce value.

Hygiene costs continuously. Expiry, conflict detection and accuracy audits all consume compute, and they run forever rather than at deployment. A memory system is the component whose operating cost rises with age, because the store grows and the maintenance scales with it.

The pattern worth noticing: memory is the only part of an agent that gets more expensive the better it works. A system nobody uses accumulates nothing. A successful one accumulates continuously, and the cost curve follows adoption rather than following usage, which makes it the component most likely to surprise a budget built on per-request estimates.

What to actually build

The deployment sequence that the practice literature converges on is deliberately slow.

Start with episodic memory, for one use case. Not all three layers. Failure modes multiply when layers interact, and a system with one layer that works is more valuable than three that half-work.

Measure recall quality and latency for thirty days at real volume. Not on a test set. Real users produce retrieval patterns no test set anticipates, and thirty days is roughly the interval at which staleness begins to appear.

Add semantic memory once episodic is stable. Semantic facts are extracted from episodes, so an unstable episodic layer produces unreliable facts that then persist.

Build hygiene from day one rather than later. Automated expiry, conflict resolution, and periodic accuracy audits. These are the components teams defer because nothing breaks without them, and they are the components that determine whether the system still works in a year.

Consider whether you need a memory system at all. The filesystem result at the top of this article is a real finding. For a modest number of durable facts, a structured file that a human can read and edit is maintainable, auditable, debuggable, and competitive on quality. The case for a vector store begins when the volume exceeds what retrieval-by-reading can handle, and that threshold is higher than most teams assume.

What is unresolved

How to decide what is worth remembering. Current systems mostly store everything or store what a rule matched. Recent work trains agents to write selectively based on whether a memory led to a good outcome, which is promising and unproven at scale. The general problem, judging significance at the time of the event rather than in hindsight, is unsolved.

Whether memory should be shared between agents. Sharing improves coordination and multiplies contamination and privacy surface. Per-agent isolation avoids that and fragments knowledge so each agent operates on a different slice of truth. No principled answer exists.

How to evaluate memory at all. Existing benchmarks show systems performing well on single-hop factual recall and poorly on multi-hop, temporal and open-domain questions, which suggests the benchmarks measure the easy part. What a good memory system should do over months, with contradictions and revisions, is not something current evaluation captures.

Whether forgetting can be principled. Human memory forgets adaptively, and the mechanism is not understood well enough to copy. Time-to-live is a crude proxy that discards recent-but-unimportant material and keeps old-but-critical material only by accident.

The counter-argument

Most agents do not need this. A single-session assistant with no cross-session continuity has no memory problem, and a substantial share of production deployments are exactly that. Building a memory layer for a system that does not require one adds a failure surface for no benefit.

The staleness problem is not new. Caches have had it forever, and the solutions are known: expiry, invalidation, versioning. Framing it as a novel AI challenge can obscure that the fix is ordinary engineering rather than research.

The benchmark evidence is thin. Memory benchmarks are recent, the field disagrees about what they should measure, and the filesystem result, while striking, comes from one study on one set of tasks. It supports scepticism about complexity rather than proving simplicity wins.

And the security findings describe attacks, not incidents. Memory poisoning and extraction have been demonstrated in research settings. Whether they occur at meaningful rates in production is not established, and treating demonstrated capability as observed prevalence overstates the current risk.

The short version

A plain filesystem of markdown files scored 74% on standard memory benchmarks, beating several purpose-built vector systems, which indicates the difficulty was never storage. It is deciding what to keep, retrieving the right thing, and knowing when a stored fact stopped being true.

Agents typically work within a session and fail across sessions, because working memory is volatile by design and most systems never promote anything from it into durable storage. Four layers have a distinct retention question each: working, episodic, semantic and procedural. Most failures are layer confusion, where an offhand remark becomes a permanent belief or a durable fact is retrieved only when the conversation resembles where it was learned.

Staleness is the main event. A frequently retrieved memory stays accurate until reality changes and then becomes confidently wrong, with nothing in the system noticing, and stale records with entity contradictions are reported as the top source of production accuracy degradation. Retrieval hallucination is second: embedding similarity is not factual relevance, and accuracy peaks at roughly three to five retrieved items and degrades beyond that, so retrieving twenty to be safe is actively worse than retrieving four.

Persistent state is also an attack surface, with demonstrated memory poisoning, dormant injections that resurface later, measurable safety degradation as memory accumulates across unrelated tasks, and extraction attacks that recover private data without model access. Larger context windows do not solve any of this: windows exceed a million tokens and cross-session persistence remains zero, while multi-session benchmarks find performance degrading faster than history grows.

The build order is deliberately slow: one layer, one use case, thirty days at real volume, hygiene from day one rather than later. And the prior question is whether a memory system is needed at all, since for a modest number of durable facts a structured file a human can read and edit is maintainable, auditable and competitive on quality.

Common questions

What is AI agent memory? The mechanism by which an agent retains information beyond a single session. It is usually structured in four layers: working memory, the volatile active context during a session; episodic memory, records of what happened at a given time; semantic memory, facts extracted from episodes; and procedural memory, conventions and methods. Each layer has a different retention question, and most failures come from confusing them.

Why do agents forget between sessions? Because working memory is volatile by design and discarded when the session ends, and most systems never promote anything from it into durable storage. The agent that seemed attentive an hour ago starts from nothing tomorrow, not because storage failed but because deciding what to write down is a design decision nobody made.

What is the biggest problem with agent memory? Staleness. A frequently retrieved memory stays accurate until reality changes, then becomes confidently wrong, and nothing in the system notices. Without versioning and conflict resolution an agent treats a six-month-old preference as exactly as authoritative as last week's. Stale records and entity contradictions are reported as the top source of production accuracy degradation in memory-equipped agents, ahead of retrieval failure or model capability.

How many memories should be retrieved per query? Fewer than most systems retrieve. Accuracy tends to peak at roughly three to five retrieved items and degrade beyond that as noise overwhelms signal, so retrieving twenty to be safe is actively worse than retrieving four. Narrow the candidate pool by metadata before computing similarity, combine similarity with keyword matching rather than relying on embeddings alone, weight by recency, and re-rank before results enter the prompt.

Do larger context windows remove the need for a memory system? No. Context length and persistence are different properties. Windows now exceed a million tokens while cross-session persistence remains zero, and a large window is expanded working memory for one hard session rather than a memory system. Multi-session benchmarks also find performance degrading faster than history grows, so more material is not linearly more competence, and high-scoring systems frequently need upwards of 26,000 tokens per query.

What is memory poisoning? Injecting content into an agent's long-term memory to influence future behaviour, where the injection happens once and the effect persists across sessions. A sleeper variant stores a fabricated memory that stays dormant and resurfaces later, which makes attribution nearly impossible since the originating session is long closed. Research has also demonstrated extraction attacks recovering private data from memory stores without access to the model, and measurable safety degradation as memory accumulates across unrelated tasks.

Should memory be shared between agents? Unresolved. Sharing improves coordination and multiplies both contamination surface and privacy exposure, since a poisoned memory reaches every agent that reads the store. Per-agent isolation avoids that and fragments knowledge so each agent operates on a different slice of truth. The control most deployments lack is scope isolation: whether agent A can read agent B's memories, and whether one user's context can reach another's.

Do I need a vector database for agent memory? Often not. A December 2025 benchmark found a plain filesystem storing memories as markdown files scoring 74% on standard tasks, beating several dedicated vector systems. For a modest number of durable facts, a structured file a human can read and edit is maintainable, auditable, debuggable and competitive on quality. The case for a vector store begins when volume exceeds what retrieval-by-reading handles, and that threshold is higher than most teams assume.