How to Choose an AI Memory Layer: A Buyer's Checklist

The fastest way to compare memory layers is not to read their benchmark scores. It is to run ten specific tests on a trial account, most of which take under ten minutes and none of which appear in a vendor demo.

Every test below has a right answer and a wrong answer, and the wrong answers are common. Run these before you commit, because the failure modes they expose surface in month four rather than week one.

1. The contradiction test

Write two conflicting facts, then query.

write: "The client is headquartered in London."
write: "The client relocated its headquarters to Dubai."
query: "Where is the client headquartered?"

Right answer: Dubai, with the London entry retained and marked as superseded, and a visible pointer between them.

Wrong answers: both returned with no ranking, London returned because it embedded closer to the query, or London silently deleted.

This is the single most diagnostic test in this list. Run it first. If a system fails here, nothing else about it matters, because an agent that confidently reports superseded facts is worse than an agent with no memory. Note also that some systems are architecturally append-only by design and document this openly. Append-only extraction is a legitimate engineering choice with real advantages in speed and history preservation, but it means supersession is handled at retrieval ranking rather than in the store. Ask which it is, because ranking-based recency is a softer guarantee than structural supersession.

2. The abstention test

Ask about something you never wrote.

query: "What is the client's VAT registration number?"

Right answer: an explicit statement that nothing is stored on this.

Wrong answer: a plausible-sounding fabrication assembled from adjacent memories. This is how memory systems turn into hallucination engines. The stored data was fine; the retrieval returned the nearest thing and the model filled the gap.

Abstention is one of the five abilities LongMemEval measures and it is almost never the sub-score a vendor leads with. Ask for it.

3. The deletion test

Write five memories about a fictional person. Let the system consolidate if it does that on a schedule. Then request deletion of that person's data and query for anything derived from it.

Right answer: nothing about them is retrievable, including from summaries, and you get an artefact you could show an auditor.

Wrong answer: the individual entries are gone but a consolidated summary still contains their information. This is extremely common and it is a live compliance problem rather than a theoretical one. See GDPR Article 17 and AI memory.

4. The provenance test

Retrieve something, then ask where it came from.

Right answer: which memory was returned, when it was written, what session or document produced it, what it superseded, and why it ranked where it did.

Wrong answer: a list of IDs and similarity scores. "The vector search returned these embeddings" is not an explanation that survives contact with a compliance team, and if your agent tells a customer something wrong, it is the only question anyone will ask.

5. The isolation test

Create two users or tenants. Write private memories for each. Query as user A for user B's information, including with deliberately leading prompts.

Right answer: nothing crosses.

Wrong answer: anything crosses. In a multi-tenant product this is not a bug, it is an incident with disclosure obligations.

Then check the granularity. User-level isolation is table stakes. Ask whether you can scope below that, because most real products need it: a legal matter, a client engagement, a project. If everything one user can see is one bucket, you will build that partitioning yourself.

6. The scale test

Load memories until the store is roughly the size you expect in year two. Not month two. Then measure retrieval quality and p99 latency, not median.

What you are looking for: does retrieval quality hold as the store grows? Early on everything is plausibly relevant and retrieval looks great. The interesting behaviour starts when there are forty thousand memories and only twelve belong in this prompt.

Memory sits on the critical path of every agent turn, so a 400ms retrieval is 400ms added to every interaction your users have. Ask for p99 under concurrency, and be suspicious of any latency figure quoted without a memory count attached.

7. The residency test

Ask three questions and get the answers in writing.

• Which region is my data stored in, and can I choose?
• Is there a self-hosted or bring-your-own-storage deployment?
• Which subprocessors touch memory content, and where are they?

For EU personal data this is a hard constraint under GDPR rather than a preference. The third question catches more problems than the first two, because a vendor hosting in Frankfurt may still route extraction through a model provider elsewhere.

8. The export test

Actually run the export. Do not read about it.

Right answer: a complete, documented, machine-readable export including provenance and supersession relationships, not just current values.

Wrong answer: current facts only, or a format you would have to reverse-engineer.

Memory is stickier than most infrastructure because the accumulated data is the value. An export path you have not tested is a lock-in you have not priced. Do this during the trial, while you still have leverage.

9. The write-quality test

Feed in a realistic messy transcript: small talk, corrections, a decision, someone changing their mind, an irrelevant tangent. Then inspect what got stored.

Right answer: the durable content. The decision, the correction as a supersession, the preference. Not the small talk.

Wrong answer: the whole transcript chunked, or a thin summary that lost the decision.

Almost every public benchmark grades retrieval rather than writing, which means this is the axis where vendors are least externally validated and where the quality spread is widest. It is worth an hour of your time.

10. The cost-at-scale test

Take your own traces. Count retrievals and writes per interaction. Multiply by realistic monthly volume. Price it through each vendor's model.

Then ask the question that reveals the pricing model's shape: what happens to my bill if my retrieval strategy gets worse? If the answer is "it rises significantly," you have coupled infrastructure spend to prompt quality and you will end up tuning prompts for cost rather than accuracy. Details in what an AI memory layer costs.

Scoring it

Nothing here requires an enterprise sales cycle. Every test runs on a trial account in an afternoon, which is less time than most teams spend reading comparison blog posts, including this one.

What benchmarks will and will not tell you

Published scores are useful for narrowing a field and useless for choosing within it. They grade retrieval on shared datasets, which is a real improvement over self-reported claims, but they largely do not measure writing, forgetting, consolidation, or deletion. Tests 1, 3, 8 and 9 above exist precisely because no benchmark covers them.

If you want to read vendor scores properly, we broke down what LoCoMo, LongMemEval and BEAM actually measure, including the six questions to ask about any published number.

How OctaMem scores itself

We would rather you ran the tests than took our word for it, so: a free tier exists specifically so you can. Start here.

Where we deliberately optimised: structural supersession with the superseded value retained, provenance on every retrieval, memory-group scoping below user level, configurable residency including EU-region and self-hosted deployment, and deletion that propagates through consolidated artefacts.

Where we are still building: the docs are the honest current state, and if a test above fails for you we would rather hear it than not.

Frequently asked questions