Agent Memory Benchmarks Explained: LoCoMo, LongMemEval and BEAM

Three benchmarks currently define how agent memory gets measured: LoCoMo tests recall and reasoning across multi-session conversations, LongMemEval tests five specific memory abilities including knowledge updates and abstention, and BEAM tests goal-directed task completion at million-token scale. They measure different things, they are not interchangeable, and a headline score from any one of them tells you less than most vendors imply.

This page explains what each one does, what all three leave out, and how to read a published number.

Why these exist

Before them, memory quality was mostly self-reported. A vendor would describe an architecture, show a demo, and assert that it worked. There was no shared evaluation set, so two systems could not be compared and no claim could be checked.

That was a genuine problem for buyers, and these three benchmarks improved it substantially. They also created a new problem, which is that a number is easier to publish than to interpret.

LoCoMo

Introduced by Maharana et al. in 2024 to evaluate very long-term conversational memory.

Construction. A hybrid pipeline: two LLM-driven agents hold multi-session conversations grounded in persona profiles and temporal event graphs, then humans refine the output for consistency. The result is dialogue that is coherent and causally linked rather than randomly generated.

Scale. Ten conversations, each spanning roughly 27 to 32 sessions. Published descriptions vary on turn and token counts, with figures between about 9K and 17K tokens per conversation appearing across different papers. That variance is itself worth noting: if the literature does not agree on the benchmark's dimensions, treat precise cross-paper comparisons with care.

Task structure. 1,540 question-answering samples in four categories:

Beyond QA, LoCoMo includes event summarisation grounded in event graphs and a multimodal dialogue task.

What makes it hard. The information needed to answer a question is often not semantically similar to the question. That single property is why LoCoMo is a real test of retrieval rather than of embedding quality. A system that only does similarity search struggles here by design, because the useful memory does not look like the query.

What it is weak at. Ten conversations is a small sample, and the multi-hop and open-domain categories are small enough that a handful of items moves the score. It also tops out around 35 sessions, which is not long by the standards of a personal assistant that has been running for a year.

LongMemEval

Introduced by Wu et al. in 2024, aimed at chat assistants.

Construction. 500 curated questions, each embedded in a dynamically assembled chat history. Inspired by needle-in-a-haystack evaluation, but with configurable history length rather than a fixed one.

Scale. The small variant pairs each question with roughly 115K tokens of history. The medium variant extends much further, into hundreds of sessions and beyond a million tokens.

Task structure. Five core abilities, sometimes reported as six when preference recall is split out:

• Information extraction from dialogue
• Multi-session reasoning
• Temporal reasoning
• Knowledge updates
• Abstention

Abstention deserves attention because almost nothing else measures it. A memory system that answers every question confidently, including the ones it has no basis for, is dangerous in exactly the settings where memory matters most. Abstention tests whether the system knows when it does not know. If you are buying memory for a regulated context, this is the sub-score to ask about, and it is rarely the one vendors lead with.

Knowledge updates is the other one that matters disproportionately. It tests whether the system handles a fact changing. Systems that only append tend to do poorly here and the aggregate score can hide it.

BEAM

The newest of the three and the one designed not to be saturated.

BEAM evaluates memory in agentic task completion rather than question answering: whether an agent can use retrieved memory to actually finish a goal-directed task. It runs at 1M and 10M token scales.

Published scores are visibly lower than LoCoMo and LongMemEval headlines. Mem0, for example, reports 92.5 on LoCoMo and 94.4 on LongMemEval but 64.1 at BEAM-1M and 48.6 at BEAM-10M. That gap is the point of the benchmark rather than a criticism of any system. When accuracy on one evaluation is in the nineties and on another is below fifty, the honest reading is that the first evaluation is closer to saturation than the field is to solving memory.

What none of them measure

Four gaps, and they are the ones that will bite you in production.

The write step. Almost every public benchmark grades retrieval. Deciding what is worth keeping out of a conversation that is mostly noise is barely evaluated anywhere. This matters because a system that stores everything and a system that stores selectively look identical on retrieval accuracy. The difference only appears in token cost, latency, and how quickly retrieval quality degrades as the store grows. Retrieval-only benchmarks structurally favour systems that hoard.

Forgetting, eviction and consolidation. Real memory systems must drop stale facts, merge duplicates, and reconcile contradictions. None of the three benchmarks tests this directly. LongMemEval's knowledge-update category is the closest thing available and it is a partial proxy.

Deletion under constraint. No mainstream benchmark tests whether you can remove one person's data and prove it. That is a legal requirement in most jurisdictions and it is architecturally hard once memories have been consolidated. It appears in no score.

Cost and latency at the same time as accuracy. Some evaluations report token consumption alongside accuracy, which is a genuine improvement. But accuracy at unbounded token cost is not a useful result, and a p50 latency figure does not describe what happens under concurrency.

How to read a published score

Six questions. Ask them of any vendor, including us.

1. Which benchmark, which variant, and which date? LongMemEval-S and LongMemEval-M are different difficulty regimes. BEAM-1M and BEAM-10M are different problems. A score without a variant is not a score.

2. Which answering model? Memory benchmarks evaluate a memory system plus a model that answers from retrieved memory. Swap the model and the number moves. A memory score reported without the model that produced it is not reproducible.

3. What was the token budget per query? Accuracy at 7,000 tokens per query and accuracy at 25,000 are different products with different bills.

4. Full run or subset? Subsets are legitimate for iteration and illegitimate for publication. Ask which categories were included, since dropping the hardest ones lifts an aggregate meaningfully.

5. What are the sub-scores? An aggregate hides everything interesting. Temporal reasoning, knowledge updates, and abstention are where architectures actually differ. A system can post a strong aggregate while failing badly at knowledge updates, and knowledge updates are what break in production.

6. Who judged? These benchmarks generally use LLM-as-a-judge with official prompts. That is the accepted method and it introduces variance. A single run is a data point rather than a result.

Read scores in pairs

The most useful discipline: never read a memory number alone.

• Pair accuracy with token cost. High accuracy at high cost may just be a large context window in a trench coat.
• Pair a single-session score with a multi-session score. Long-context ability and cross-session memory are different capabilities.
• Pair a QA benchmark with a task-completion benchmark. Answering a question about the past is easier than using the past to do something.
• Pair a retrieval score with anything at all about writes.

If a vendor publishes one number, the number they chose tells you which axis they are strongest on. That is not dishonest. It is just incomplete, and the incompleteness is doing work.

Beyond the big three

Worth knowing these exist, because they cover gaps:

• EpBench: episodic memory specifically, on synthetic narrative corpora from 100K to 1M tokens
• PersonaMem: long-term personalisation across multi-session interaction
• HaluMem: how hallucinations emerge and propagate through memory management stages
• MemoryAgentBench, PerLTQA, DialSim, and MemoryBank: additional evaluation frameworks

HaluMem is the interesting one for anyone building on memory, because it targets error propagation. A wrong memory does not stay contained; it gets retrieved, reasoned over, and consolidated into something more confident than the original mistake.

Our own position on this

We are running the full LoCoMo evaluation now and have not published results. We are not going to publish partial numbers, a subset run, or a comparison we have not verified against a reproducible baseline.

That is a slower answer than a chart, and it is deliberate. We sell a memory layer whose value proposition is that you can audit what it did. A retracted benchmark claim from a company selling auditability costs more than the claim ever earned.

When the full run is complete and the comparison baseline is confirmed, results will go up with the answering model, token budget, variant, sub-scores, and date attached, so anyone can check them.

In the meantime, the more useful question for a buyer is not which vendor posted the highest LoCoMo score. It is whether the system handles a fact changing, whether it can abstain, and whether you can prove a deletion. Those are answerable today with a thirty-minute test on your own data, and we wrote up how to run it in how to choose an AI memory layer.

Related reading

• How to choose an AI memory layer
• What is an AI memory layer?
• Nine ways agent memory fails

Sources: Maharana et al. (LoCoMo, 2024); Wu et al. (LongMemEval, 2024); Chhikara et al., "Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory", ECAI 2025; published benchmark figures from mem0.ai as of August 2026; benchmark descriptions cross-checked against arXiv:2601.20352, arXiv:2602.13594, arXiv:2604.04853, arXiv:2604.01560. Last updated 10 August 2026.

Frequently asked questions