AI agent memory systems, layer by layer

The short version

An agent without memory is a goldfish with tools: every turn starts from zero. Memory systems give agents continuity — what the user said, what worked, what the codebase looks like — across turns and sessions. The trick is that "memory" isn't one thing; it's a stack of mechanisms with different lifetimes, and most broken agents are broken because they used the wrong layer for the job.

The layers

Working memory is the context window itself: the current conversation, recent tool outputs, the task at hand. It's fast, it's immediate, and it's finite. Everything else exists to decide what deserves to be in the window.

Short-term memory spans a session: rolling summaries of what happened so far. When the transcript gets long, you compress the old turns into a summary and keep going. The art is in what the summary preserves — decisions and state, not chatter.

Long-term memory persists across sessions: user preferences, project facts, learned procedures. This is usually a vector store or a structured note file the agent reads at session start and writes to at session end.

External memory is the world itself: files, databases, issue trackers. The strongest pattern in practice is treating the environment as memory — the agent re-reads the file rather than remembering it. "Memory" you can re-derive beats memory you have to maintain.

What goes wrong

• Everything in the window. Dumping the whole history into context works until it doesn't — then you're paying for 100k tokens to answer a question about line 12. Summarize aggressively.
• Memory that never forgets. Stale facts are worse than no facts. Timestamp everything, and give the agent an explicit way to correct or expire entries.
• The summary ate the state. Rolling summaries love to drop the one detail the next step needed — usually a path, an ID, or a decision. Keep a separate structured "scratchpad" for load-bearing facts, outside the prose summary.
• No write policy. If the agent can't decide what deserves remembering, the memory fills with trivia. Write on: user corrections, discovered facts about the environment, decisions with reasons.

Rule of thumb: prefer re-reading the world over remembering it, summarize the past, and keep load-bearing facts in structured state — never buried in prose the summarizer might eat.