Six layers of memory, and what each one forgets on purpose

A memory model is not a database. It is a series of decisions about which past should reach the present. Most teams treat memory as an accumulation problem, then wonder why the system gets worse as it learns more. The forgetting is the work.

We think of an agent system as a small building. Rooms with purpose, doors in the right places. Memory is the building's relationship to its own history: what sits on the desk, what lives in the filing cabinet, what is carved into the cornerstone. Put a record in the wrong room and the building behaves strangely, and it will not tell you why.

Why the big window fails

The tempting alternative is to skip the architecture and buy a bigger context window. It fails for three reasons, and only one of them is cost.

First, attention dilution. Attention is a budget, not a spotlight. Every token you add competes with the tokens that matter, and recall over long contexts degrades unevenly, long before the window is full. A million tokens of undifferentiated history is not memory. It is noise with a search problem attached.

Second, cost, which compounds. You pay for the window on every call, and an agent system makes many calls per task. Prompt caching softens this. It does not repeal it.

Third, and most important: no retention policy is itself a policy. It is the policy that everything is equally important forever, which is false, and the model will act on the falsehood. A window with everything in it makes yesterday's abandoned draft as authoritative as the founding brief. Something will decide which past reaches the present. The only question is whether you decide, or the attention mechanism decides for you.

Six rooms, fastest to slowest

Our model has six layers, ordered from fastest-changing to slowest. Each holds a different kind of past, ages at a different rate, and forgets something specific on purpose. The walls between them are the point.

L0, session: the desk

Holds the current question, the attached documents, whatever context the requester made explicit. Retention: none. Cleared when the task ends, completely, every time.

L0 forgets everything by design, and that is its whole value. Its failure mode is not skipping the layer; it is failing to clear it. Residue from the last task leaks into the next: an assumption about one matter carries into another, a constraint from a dead draft shapes a live decision. This is context bleed, and it is the most common memory bug in production because it never announces itself. The output stays plausible. It is just answering a slightly different question than the one asked.

L1, thread: recent work

Holds a rolling window of related recent work. Days, not months. Retention: compaction on a schedule. The raw record is periodically summarised down; detail is discarded, decisions are kept. What survives is “chose X over Y because Z,” not the forty messages it took to get there.

Skip L1 and every task starts cold: the system re-litigates settled questions and feels like it needs onboarding daily. Merge it into the domain layer and something worse happens: transient noise accumulates as fact. A half-formed idea from Tuesday hardens into domain knowledge by Friday with no curation in between. The compaction schedule is a genuine design decision. Compact too aggressively and you lose the reasoning behind recent choices; too lazily and the thread becomes a second, worse context window.

L2, domain: the library

Holds each agent's own long store: precedents, documents, domain corpora, the material a specialist keeps within reach. Each room has its own shelves. A research agent and a drafting agent do not share a library, because they should not inherit each other's accidents of retrieval.

Retention: L2 forgets nothing quickly but admits new material slowly, through curation rather than ingestion. Nothing writes itself into the library, and from the deliberation layer's perspective L2 is read-only. This is also where the embeddings-versus-structured-store decision lives. Embed the corpora you search by meaning; keep entities, precedents, and their relationships in a structured store you can query exactly. Vector similarity is a poor substitute for “the precedent from this matter, specifically.”

Skip or starve L2 and you get an agent that reasons well about nothing in particular. General capability, no ground.

L3, deliberation: the minutes

Holds the history of decisions. Not just outcomes: dissents, refused premises, rationales, the arguments that lost. Retention: long, append-mostly, heavily indexed, because this is the most audited store in the system.

When a system starts to drift (approving what it used to refuse, hedging where it used to commit), L3 is where you look. Drift is rarely a model problem. It is usually a memory problem: a decision compacted into a caricature of itself, or a refused premise that stopped being retrieved and got quietly re-accepted. Skip L3 and you have a system with opinions but no jurisprudence. It cannot tell you why it believes what it believes, and neither can you.

L4, voice: the cadence

Holds the prose corpus of the person the system writes as. Sentences, not conclusions. It forgets content and keeps cadence: rhythm, diction, how this person opens an argument and how they close one.

The rule that matters: L4 is read at the last step and never used for reasoning. Let voice samples into deliberation and old opinions smuggle themselves in as style; the system starts asserting things the person said years ago because the sentence sounded right. Merge L4 into L2 and you get exactly that, a system that confuses how someone talks with what they currently think. Keep the wall. Reason first, then dress.

L5, seed: the cornerstone

Holds the founding documents: principles, standing positions, the boundaries of the mandate. The smallest store and the most consulted. Read before every significant decision, in full, with no retrieval step, because a retrieval step is a way to accidentally not read something.

Retention: edited by a human, rarely, deliberately, and never by the system itself. This is a safety property, not a workflow preference. Every other layer is downstream of the system's own behaviour, so every other layer can drift with it. If the seed were writable by the system, drift would have a ratchet: each drifted decision would re-anchor the definition of normal, and the anchor would move with the ship. A human-edited seed keeps one fixed point outside the feedback loop. When the L3 audits show divergence, you need something the system diverged from.

Where the bug lives

Production memory bugs sort by layer once you know the signatures.

• Bleed between unrelated tasks. L0 is not being cleared. Check task boundaries and shared scratch state.
• Cold starts and re-litigated decisions. L1 is missing, or compaction is destroying decisions along with detail.
• Confident nonsense inside a specialty. L2. The corpus is thin, retrieval is embedding-only where it needs structure, or ingestion is uncurated and the library is full of drafts.
• Judgment drifting over weeks. L3. Read the record; find where a rationale was lost or a refusal expired.
• Right conclusions, wrong sound, or stale opinions resurfacing as style. L4, almost always a wall breach into reasoning.
• The system arguing with its own mandate. L5 is stale, or something other than a human has been editing it.

Eval each layer, not just the pipeline. An end-to-end eval tells you the building is leaning. A per-layer eval tells you which wall.

The forgetting is the work

Six layers sounds like ceremony until you watch a system without them age. It gets slower, then stranger, then confidently wrong, and nobody can say when the turn happened. The layers exist so that every piece of the past has one room, one retention policy, and one deliberate way to be forgotten. Memory is architecture, not storage. Build the walls first. Then decide, on purpose, what each room may forget.