LoCoMo Benchmark Results 2026: OctaMem reaches 93.51% accuracy.
LoCoMo tests whether a memory system can answer questions about facts buried deep in a long, multi-session conversation, without stuffing the whole history into a prompt. Here is where OctaMem landed — including the part that did not work.
93.51%
1,440 of 1,540 questions answered correctly, computed directly from per-question judgements, with nothing rescaled or smoothed.
The trajectory
This is our third evaluation, and it is not a smooth creep.
It is one fix landing. Run over run, accuracy moved 5.21 percentage points, and almost all of that arrived at once.
| Run | Accuracy | Correct |
|---|---|---|
| Initial evaluation | 88.30% | 1,360 / 1,540 |
| Second evaluation | 90.62% | ≈1,396 / 1,540 |
| Final run (this report) | 93.51% | 1,440 / 1,540 |
Where the fix landed
Temporal reasoning was our worst category. It is now our best.
Both earlier evaluations flagged the same weakness: ordering, duration, and “as of when” questions. Temporal accuracy moved from 83.70% to 97.20%, a 13.5-point jump that accounts for almost the entire run-over-run gain. Every other category held within a point of where it was. The release did one thing, and did it without breaking anything else.
Error concentration
One conversation is carrying almost all our error.
We are not going to bury this, because it is the most useful number in the report. The 100 remaining errors are not spread across the benchmark: 95 of them sit in a single conversation out of ten, which scores 63/158 — 39.9%. Eight of the others score a perfect 100.0% and the ninth scores 96.8%. Outside that one conversation, the run answers 1,377 of 1,382 questions correctly, or 99.64%.
Conversation
A near-perfect run sitting next to one conversation that fails badly is not a diffuse capability gap. It is a signature that points somewhere specific. We are auditing it end to end, and applying the same scrutiny to the eight conversations that scored 100.0%, because a perfect score deserves exactly as much suspicion as a bad one.
What it costs
An accuracy number without a cost figure is half a claim.
Both of our earlier reports flagged missing cost telemetry as a gap. This run closes it.
3,519
Mean tokens / question
2,306
Answer-stage tokens
$0.00076
Mean cost / question
$1.17
Full 1,540-question run
The LoCoMo leaderboard question
There is no reliable LoCoMo leaderboard — here, or anywhere else.
Short answer: LoCoMo has no official leaderboard, and the unofficial rankings you will find are not comparing like with like. Every vendor runs the benchmark on its own harness and publishes its own number. Those numbers are real, but they are not a ranking, and putting them in one chart implies a comparison that was never made.
LoCoMo is the benchmark most memory vendors report against, including Zep, Mem0, Supermemory, and Letta, and it has become a crowded, noisy leaderboard. Before considering a comparison chart, we checked how those numbers are actually sourced. Published LoCoMo scores for the same system can vary by fifteen points or more depending on who measured it and under what harness, and new entrants publish their own “beats everyone” claim every few weeks. Supermemory’s public figure is a different metric altogether: recall rather than overall accuracy, so it cannot sit on the same axis as the others regardless of harness.
Until we have reproduced results against these systems in a shared, neutral setup, we are not putting a ranking chart next to our name. When that work is done, we will publish the comparison, numbers and methodology together.
How to read any published LoCoMo score
Which metric? Overall accuracy, recall, and F1 are not interchangeable. A recall figure will read several points higher than an accuracy figure on the same run, and the two cannot go on one axis.
Which judge? Most LoCoMo scores are graded by an LLM. The judge model and the grading prompt move the result, so a score without a named judge is not reproducible.
All 1,540 questions? Some published runs exclude the adversarial category, or a conversation that went badly. A subset score is fine if it is declared, and misleading if it is not.
What retrieval budget? Accuracy at top-20 retrieved memories is a different claim from accuracy at top-5, and costs more per question. Ask what was retrieved before comparing.
Who ran it? A vendor's score for its own system and a third party's score for that same system routinely differ by fifteen points or more. Self-reported is not disqualifying, but it is not a ranking either.
Run those five checks against our number too. The methodology, the per-question judgements, and the conversation we fail on are all on this page for exactly that reason.
We’d rather be the company whose numbers are never wrong than the company that was first.
Questions
What people ask about this score.
What is the LoCoMo benchmark?
LoCoMo (Long Conversational Memory) tests whether a memory system can answer questions about facts buried deep in long, multi-session conversations without loading the entire history into the prompt. It covers 1,540 questions across ten conversations, in four categories: single-hop, multi-hop, open-domain, and temporal.
What did OctaMem score on LoCoMo?
93.51% overall — 1,440 of 1,540 questions answered correctly. By category: temporal 97.20%, single-hop 94.29%, open-domain 93.75%, and multi-hop 86.88%. The figure is computed directly from per-question judgements, with nothing rescaled or smoothed.
How much does a LoCoMo run cost on OctaMem?
$0.00076 per question, or $1.17 for the full 1,540-question run, at a mean of 3,519 tokens per question with 2,306 of those in the answer stage.
Is there an official LoCoMo leaderboard?
No. LoCoMo ships as a dataset and a paper, not a hosted leaderboard with a submission process, so every score you will find is self-reported by the vendor that ran it. The rankings circulating online are compilations of those self-reported numbers, measured on different harnesses, with different judge models, sometimes on different subsets of the 1,540 questions — which is why the same system can appear fifteen points apart in two different tables.
How do I compare LoCoMo scores between memory vendors?
Check five things before putting two numbers side by side: which metric was reported (accuracy, recall and F1 are not interchangeable), which model graded the answers, whether all 1,540 questions were included, how many memories were retrieved per question, and who ran the evaluation. If any of the five is undeclared, the comparison is not sound. Apply the same five to our 93.51%.
Where do OctaMem's remaining LoCoMo errors come from?
They are highly concentrated. Of 100 remaining errors, 95 sit in a single conversation which scores 63/158, or 39.9%. Excluding that conversation, the run answers 1,377 of 1,382 questions correctly — 99.64%. We are auditing that conversation end to end.
Full methodology and per-question data are available on request. If you are evaluating memory layers, we will walk you through the harness.
Request the methodologyNew to these benchmarks? What LoCoMo, LongMemEval and BEAM actually measure explains what each one tests and how to read a published score. For the buyer’s view, see how to choose an AI memory layer.