What Is an AI Memory Layer?

An AI memory layer is infrastructure that gives an AI agent persistent, queryable state across sessions. It decides what an agent writes down, what it retrieves at inference time, what it updates when facts change, and what it forgets. Without one, every conversation starts from zero.

Language models are stateless. They do not carry information between calls. Everything a model appears to "know" about you within a session is text someone put in the context window, and when that window closes the information is gone. A memory layer is the component that makes the forgetting optional.

Why statelessness is a product problem

The engineering description undersells how bad this feels to a user.

An agent that helped you debug a deployment on Tuesday has no idea who you are on Wednesday. A support agent asks a customer for their account tier for the fourth time. A research assistant that spent an hour learning your codebase's conventions loses all of it when the session ends. A legal AI reviews the same contract clause twice and reaches different conclusions, because nothing recorded what it decided the first time.

The naive fix is to stuff more history into the context window. This fails in three predictable ways.

It gets expensive. You pay per token, every call, for history the model mostly ignores.

It gets slower, and accuracy degrades in the middle of long contexts even in models with large windows.

And it does not actually scale. A window is finite. A customer relationship is not. Two years of interaction history will not fit no matter how large windows get, and even if it did, dumping everything is not the same as retrieving the right thing.

Memory is a retrieval problem wearing a storage problem's clothes.

What a memory layer actually does

Four operations, and most systems that call themselves memory only implement the first two.

Write. Deciding what is worth keeping from an interaction. Not the transcript, but the durable content of it: the user prefers TypeScript, the account is on the enterprise tier, the last migration attempt failed on the foreign key constraint. Extraction is a judgement call, and it is where most of the quality difference between systems lives.

Retrieve. Surfacing the relevant subset at inference time, in a token budget. This is not "search the history." It is deciding, before the model runs, which twelve of forty thousand stored facts belong in this particular prompt.

Update. Handling the fact that stored information goes stale. The user changed jobs. The pricing changed. The bug got fixed. A memory system that only appends will eventually retrieve a confident, well-formatted, wrong answer. Reconciling a new fact against a contradicting old one is genuinely hard, and it is the operation most naive implementations skip entirely.

Forget. Dropping or decaying what no longer matters. Partly for cost and retrieval quality, partly because in a regulated context "delete this person's data" is a legal requirement rather than a feature request.

How it differs from RAG, vector databases, and context windows

These get conflated constantly, including by vendors. The distinctions are real.

A vector database is a storage primitive. It does similarity search over embeddings. It has no opinion about what should be stored, when a stored item is obsolete, or which of two contradicting items is current. Calling a vector database a memory layer is like calling PostgreSQL a CRM. You can build one on it. It is not one.

RAG retrieves from a corpus that exists independently of the conversation: your docs, your knowledge base, your policy PDFs. The corpus is authored by humans and changes on a release cycle. Memory is generated by the interaction itself and changes constantly. Different write paths, different staleness behaviour, different failure modes. Most production systems need both, and confusing them produces architectures where user preferences live in a documentation index and nobody can explain why the agent is wrong.

The context window is working memory. It is where retrieved memory gets used, not where it lives.

The four types of memory

The standard reference here is CoALA, a framework published by Sumers, Yao, Narasimhan and Griffiths in Transactions on Machine Learning Research (arXiv:2309.02427). It borrows a taxonomy from cognitive psychology and has become the vocabulary most of the field uses.

• Working memory: the current decision cycle's active state. In practice, the context window.
• Semantic memory: facts and knowledge. The client is incorporated in Delaware.
• Episodic memory: specific past events. We tried this approach in March and it failed.
• Procedural memory: how to do things. Skills, tool definitions, learned routines.

The distinction matters operationally, not just taxonomically, because the four types want different treatment. A semantic fact should be superseded when better information arrives, and it should keep a record of what it replaced. An episodic event should decay unless something makes it durable. Procedural memory should be versioned like code, because that is what it is.

A recent critique (arXiv:2604.11364) argues CoALA papers over exactly this by filing semantic and episodic memory under a single "long-term memory" heading with no formal difference in update mechanism or decay behaviour. Applying experience-style forgetting to a store of facts produces correctness regressions. That critique is the single most useful thing to understand before you architect a memory system, and we go into it properly in our CoALA breakdown.

There is also a fair counterargument that the taxonomy can be vocabulary churn. If your team already says "context window", "transcript", "RAG index" and "skills file" and everyone understands each other, renaming them adds nothing. The taxonomy earns its keep when it changes how you handle updates and decay. If it does not, skip it.

When you do not need one

Worth saying plainly, because the honest answer is often "not yet."

Single-turn tools do not need memory. A classifier, a summariser, a translation endpoint: no state, no problem. Neither do agents whose entire relevant context fits comfortably in one window and does not need to persist, or internal tools with five users where you can hardcode the five users' preferences.

You need a memory layer when the number of things worth remembering exceeds what you can fit in a prompt, when the information changes over time, or when you have to answer for what the system knew and when. That third one arrives suddenly and usually from your compliance team.

How to evaluate a memory layer

Seven things, roughly in order of how much they will hurt you if you get them wrong.

1. Can you audit a retrieval? When the agent produces an answer, can you see which memories were retrieved, when they were written, and what they were derived from? In a regulated setting this is not a nice-to-have. If a bank's agent tells a customer something wrong, "the vector search returned these embeddings" is not an answer anybody accepts. Ask for a worked example of a retrieval trace before you commit.

2. What happens when facts contradict? Ask the vendor directly. Write "the client is based in London", then write "the client relocated to Dubai", then query. A system that returns both without ranking them, or silently keeps the first, will produce confident wrong answers in production. You want to see supersession with the old value retained and marked.

3. Retrieval latency at your scale, not their demo's. Memory sits on the critical path of every agent turn. A retrieval that takes 400ms is 400ms added to every interaction. Ask for p99, not median, at a memory count resembling your second year.

4. Where does the data live, and can you move it? For anything touching EU personal data, GDPR makes region a hard constraint rather than a preference. Ask whether data residency is configurable, whether a self-hosted or bring-your-own-storage deployment exists, and what the export path looks like. Memory is stickier than most infrastructure. The data is the value, so an export path you have not tested is a lock-in you have not priced.

5. Access control granularity. If two users share an agent, can one retrieve the other's memories? In a multi-tenant product this is the difference between a feature and an incident. Look for scoping at the level your product actually needs, which is usually finer than user-level.

6. Deletion that means deletion. Can you delete one person's memories on request, including derived and consolidated artefacts, and prove it? Systems that consolidate memories into summaries often cannot cleanly remove a single contributor's data from an existing summary. Ask.

7. What it costs at year two. Covered separately in what an AI memory layer costs, because pricing models in this category vary enough that headline numbers are not comparable.

Build or buy

Building a first version is easy, which is the trap. Embeddings into a vector store, similarity search on retrieval, a prompt that extracts facts. A competent engineer ships that in a week and it demos beautifully.

The hard parts arrive later, in a fairly reliable order: contradiction handling, retrieval quality as the store grows past the point where everything is plausibly relevant, deletion and audit for compliance, latency under concurrency, and the operational cost of a system nobody owns.

Build if memory logic is your product's actual differentiator, or if your requirements are unusual enough that generic systems fight you. Buy if memory is infrastructure your product depends on but does not compete on. The tell is whether your team would be proud to demo the memory system itself, or just its effects.

Interfaces you should expect

A memory layer should be reachable from wherever your agents run. In practice that means a REST API for anything, native SDKs for Python and JavaScript, and an MCP server, which is how memory reaches Claude, Cursor, and the rest of the MCP ecosystem without you writing glue.

The MCP path is worth calling out because it changes who can adopt memory. Adding a memory server to Cursor is a config file edit, not an engineering project. If you are doing that, our Cursor permissions reference covers how to allowlist memory tools so they run without an approval prompt on every recall.

How OctaMem approaches this

We build OctaMem as a memory layer for regulated industries, which shaped what we optimised for.

Memory is organised into semantic, episodic, and procedural layers with different update and decay rules for each, rather than one undifferentiated store. Every retrieval carries provenance: which memory was returned, when it was written, what it was derived from, and what it superseded. Data residency is configurable, with EU-region hosting and self-hosted deployment for organisations that cannot send memory to a vendor's cloud. Access control scopes to memory groups rather than only to users.

Available as a REST API, Python and JavaScript SDKs, and an MCP server. Docs · Pricing

Frequently asked questions