Episodic Memory in AI Agents
Episodic memory lets an agent learn from its own history. Why similarity search is the wrong retrieval strategy for events, how recency and importance scoring works, and when to consolidate.
Episodic memory is an AI agent's record of specific events it participated in, with time as part of the meaning. The deployment failed on Tuesday when we skipped the migration step. The customer escalated after the third transfer. I suggested this refactor in March and the team rejected it.
It is what makes an agent able to learn from its own history rather than only from training data. An agent without episodic memory repeats mistakes indefinitely, because nothing recorded that they were mistakes.
Episodic memory is one of three long-term memory types in the CoALA framework, alongside semantic and procedural memory.
What makes it different from semantic memory
Semantic memory holds claims about how the world is. Episodic memory holds records of what happened. The distinction sounds academic until you consider updates.
A fact that changes should be superseded: replace the old value, keep it for audit, point the new one at it. An event that becomes irrelevant should decay: it happened, it will always have happened, but its relevance to current decisions fades.
Apply the wrong policy and you get one of two failure modes. Decay your semantic facts and you lose knowledge you needed. Supersede your episodes and you have destroyed the history you built the system to keep. This conflation is a known gap in CoALA itself, which files both under a single long-term memory heading with no formal difference in decay behaviour.
Why similarity search is the wrong retrieval strategy
This is the part most implementations get wrong, and it is worth being precise about why.
For semantic memory, relevance to the query is usually the right selection criterion. For episodes it often is not. The most textually similar past event is frequently not the most useful one. If an agent is about to run a migration, the relevant episode is not the migration that went fine; it is the one that broke, which may share little vocabulary with the current task.
The reference implementation here is Generative Agents (Park et al., 2023), which the CoALA paper cites for combining three signals:
- Recency: a rule-based decay, so last week outranks last year
- Importance: a reasoning-based score, generated when the memory is written
- Relevance: embedding similarity to the current situation
Scoring importance at write time is the key move. It costs an extra model call per stored episode and it is almost always worth it, because relevance alone cannot distinguish "the user said hello" from "the user said the last invoice was wrong."
A fourth signal worth adding in practice: outcome. Episodes that ended badly should be retrievable preferentially when the agent is about to do something similar. Failure is more informative than success and most retrieval strategies treat them identically.
Consolidation and when to run it
Episodic memory grows without bound. Every interaction produces events; nothing naturally removes them. Two mechanisms manage this.
Decay reduces retrieval weight over time. Cheap, and it preserves the record.
Consolidation compresses many episodes into a durable pattern. Twelve occasions where the user rejected verbose explanations become one semantic fact: the user prefers terse answers. This is the mechanism by which episodic memory turns into semantic memory, and it is where an agent starts to look like it is learning rather than remembering.
Consolidation has a cost people discover late. Once you have compressed twelve episodes into one summary, deleting the contribution of one of those episodes is hard. If a user exercises a deletion right, can you remove their data from a summary that no longer references them individually? Many systems cannot. Decide your consolidation policy with that in mind, not after.
Storage shape
Episodes want more structure than a text blob. A workable schema:
- What happened, in natural language
- When, as a real timestamp rather than a relative phrase
- Who or what was involved
- Outcome, if the episode has one
- Importance score, assigned at write time
- Source, so the episode is traceable to a session
The timestamp matters more than it looks. "Last Tuesday" stored as text becomes wrong within a week and there is no way to detect that it has.
Related reading
- What is an AI memory layer?
- Semantic memory in AI agents
- Procedural memory in AI agents
- CoALA explained
OctaMem keeps episodic memory in its own layer with decay and consolidation rules separate from semantic storage, and retains provenance through consolidation so individual contributions stay traceable and removable. Docs
Frequently asked questions
What is episodic memory in AI?
An agent's record of specific events it participated in, where time is part of the meaning rather than metadata.
How is episodic memory different from a conversation log?
A log is raw transcript. Episodic memory is extracted, scored and retrievable: the durable content of what happened, not every token of it.
Why not just use vector search for episodes?
Because textual similarity does not identify the most useful past event. Recency, importance and outcome carry information that embedding distance does not.
How do you stop episodic memory growing forever?
Decay reduces retrieval weight over time; consolidation compresses recurring episodes into semantic facts. Most systems need both.
What are the four types of agent memory?
Working, semantic, episodic, and procedural, following CoALA.