94.7% on LoCoMo at 5.0K context tokens — 95.0% when re-graded by Mem0’s own judge. The more useful finding is what we measured on the way there.
The number, with its configuration attached
94.7% on LoCoMo, all 1540 questions, at 5.0K context tokens per question. It is the highest LoCoMo number we are aware of, published or otherwise, and we can hand you the code and the per-question outputs for it (github.com/todoforai/livemem).
The nearest published row is Mem0’s April 2026 algorithm at 92.5, delivered in 6,956 mean context tokens; then MemMachine’s 91.7% for v0.2. So: +2.2 points on about 30% less context.
Per category: open-domain 97.0%, single-hop 93.6%, temporal 93.5%, multi-hop 81.2%. That last one is where the remaining work is, and we’d rather print it than average it away.
How much of that is the grader? This is the first question we would ask, so we
answered it: we re-graded the identical 1540 answers with gpt-4o-mini running Mem0’s own
ACCURACY_PROMPT — the judge model and prompt behind their published LoCoMo numbers — and
got 95.0% (+18 flips to correct, −13 the other way). The two judges agree within 0.3
points on these answers. The lead is not a grading artefact.
How much of it is the ensemble? The answers come from two passes over the same
retrieved memory: gemini-flash and claude-haiku-4-5 answer the identical 5.0K block
independently, then a rule hierarchy (agree > commit > abstain) settles the 82
disagreements. One pass alone scores 92.4%. So +2.3 points are answer-side, bought with
a second model call and ~6 seconds per question instead of one — not with more context.
That distinction turns out to be the theme of this entire post.
And before anyone else says it: LoCoMo at 94.7% is a nearly saturated benchmark. Mem0 say so themselves — “older benchmarks like LoCoMo are less informative today,” because they don’t stress conflicting updates, identity ambiguity or long-running workflows. We agree, and we still publish the row, because a saturated benchmark is exactly the regime where the unreported variables — answerer, judge, context budget — decide the ranking. Two points of headroom and a seven-point answerer effect is not a leaderboard; it’s a measurement problem. The rest of this post is that measurement problem.
The setup
We build TODOforAI, an agent that runs long-lived tasks on your machine. Long-lived means it has to remember: what you decided three weeks ago, what your deploy procedure is, which of two conflicting instructions is the newer one.
So we built a memory system, and then we did the thing you’re supposed to do: benchmarked it against published numbers instead of trusting our own demos.
Two public datasets:
- LongMemEval_S — 500 questions over long, multi-session chat histories. Single-session recall, multi-session aggregation, temporal reasoning, knowledge updates.
- LoCoMo — 1540 questions, heavier on temporal and multi-hop reasoning.
Both run through AMB, an independent harness that fixes the dataset, the answering model and the judge for every memory provider it runs.
LoCoMo went well from early on. LongMemEval was the humbling one, and still is: our first honest full-dataset number was 85.4%, against Mem0’s published 94.4%. We are at 87.8% today — better, still behind, and behind at 4.2K context tokens against their 6.8K, which is the only part of that comparison we’re happy with.
That 9-point hole is what sent us looking for bugs, and the bugs turned out to be more interesting than the number. It is also why the rest of this post spends more time on LongMemEval than on the benchmark we lead.
Chasing it
The obvious hypothesis is that their architecture is better. So we went looking for what we were doing wrong, and we found real things:
Deduplication was deleting evidence. A question like “how many plants did I buy last month?” needs every instance, not the top-k most similar ones. Our dedup merged “bought 2 basil plants” and “bought 3 basil plants” — near-identical sentences — silently destroying an instance the count depended on. Making dedup number-aware (two facts with different digits are never duplicates, however similar the prose) was worth +4.3 points on a 94-question validation subset (78.7% → 83.0%).
Extraction was too tasteful. Our prompt skipped things “an LLM already knows.” Then a question asked “what was that Borges quote you gave me?” — public knowledge, yes, but the specific thing we said in that conversation is not in any model’s weights. The lesson generalizes: extraction should be greedy, because a fact never extracted can never be retrieved. Selectivity belongs to the selector, not the extractor. This fixed three questions on our subset that had been unanswerable.
Some answers don’t live in facts at all. They live in a conversation turn that no summary preserved. Adding a retrieval configuration that mixes entity cards with 3-turn conversation-window excerpts alongside the distilled facts took the full 500-question run from 85.4% to 87.8%.
Real improvements, real points. Still not 94.4%.
The variable we hadn’t controlled
Here’s what we should have checked on day one. We had been answering with the cheapest
model we could — gemini-flash-lite — deliberately, for the reason above.
Mem0’s published harness config uses gpt-5 as the answerer and gpt-5 as the judge.
So we ran the control: same memory system, same extracted facts, same retrieval, same rendered context. Swap only the answering model.
| Config (94-question subset) | Accuracy |
|---|---|
| flash-lite answerer, flash-lite judge | 85.1% |
| gpt-5 answerer, flash-lite judge | 92.6% (+7.4) |
| gpt-5 answerer, gpt-5 judge, same answers re-judged | 90.4% (−2.2) |
+7.4 points from the answerer alone. And the judge is a free variable too — though note the stronger judge was stricter, not more generous: it accepted two hedged answers we’d been denied and rejected four we’d been given.
Be careful what this does and doesn’t show. It does not prove Mem0’s 94.4% is “really” lower, or that the gap between two systems is entirely answerer choice — we never ran their system on our harness. What it shows is narrower and still useful: on the same memory, configuration differences of the size that separate published rows are worth about as many points as the memory architecture itself. Our own comparable-ish number is 90.4%, still short of 94.4%. We just no longer believe the raw distance between two differently-configured rows is a measurement of anything.
What we actually learned
1. Cross-harness memory numbers are not a ranking. The clearest evidence isn’t even ours: in the published literature Mem0 appears at 94.4% (self-reported) and at 67.6% (measured in a third-party paper). Same system, 27 points apart, because the harness, answerer, judge prompt and budget differ. Any table mixing sources — including the one in our repo — should be read as a collection of separately-configured experiments, not a leaderboard.
2. Judges are worth points, not decimals — and not only via the prompt. Our
single-pass answers scored 92.4% under claude-haiku-4-5 and 95.6% under gpt-4o-mini,
on near-identical judge prompts. Nothing about the memory changed between those two
numbers; only which model read the same grading instruction. (On the ensemble answers the
same two judges land within 0.3 points of each other — the size of the judge effect is
itself configuration-dependent, which is the point.)
3. Context size is a hidden axis. We run at ~4–5k tokens per question, measured after rendering. Mem0’s new algorithm reports its 92.5 at 7.0K and its 94.4 at 6.8K; the Hindsight run on the AMB leaderboard reports 43.6k context tokens per question — about 9× ours. Accuracy per token is a different ranking than accuracy, and only one of the two is usually reported. Credit where due: Mem0 do report mean tokens next to every score, which is more than most leaderboard rows offer.
4. Our own most embarrassing finding was in our token accounting. Our “5000-token”
budget was actually delivering 5742 real tokens. Date prefixes (- [2026-08-14] ) are
15 characters but 10 tokens — dates are token-dense — and across a full block that
came to roughly 1.6k tokens we never charged for. After enforcing the real budget, delivered
context dropped to 4182 tokens and the score on our subset dropped from 83.0% to 77.7%. A
rerun with the budget raised so delivered context matched the old 5742 recovered to
81.9% — so the drop was the missing context, not lost capability. The uncounted tokens
had been buying real answers. If you don’t measure delivered tokens, your budget is
decoration.
5. The dominant remaining error family is aggregation. “How many X in total”, “what percentage of Y”. The instances are usually in the retrieved context — the answerer just doesn’t count them reliably. That’s not fixed by retrieving harder. It’s also a direct cost of our weak-answerer choice: flash-lite stresses retrieval honestly, but it fails arithmetic that a stronger model wouldn’t. There’s no free methodology.
Where LoCoMo stops being enough
Mem0’s benchmark post is worth reading precisely because it argues against its own top row, and its list of failure modes matches ours: paraphrase overfitting (systems pass when the query echoes the stored wording), temporal revision (the user changed their mind and both facts are still in the store), identity ambiguity, mixed-granularity scoping (per-user vs per-project vs per-device), and scale — benchmarks run over hundreds of events, production over millions.
Their own numbers show the drop: 92.5 on LoCoMo and 94.4 on LongMemEval_S, but 64.1 on BEAM 1M and 48.6 on BEAM 10M — same system, same year, thirty to forty-five points lower as soon as the benchmark contains interference, updates and long-horizon chains. That is a much more honest picture of where memory actually is than any 9x.x% row, including ours.
We have not run BEAM yet. It’s next, and we’ll publish it the same way — with the answerer, the judge, the delivered token count and the per-question outputs attached — whatever the number turns out to be. Our expectation, stated in advance so it’s falsifiable: our multi-hop score of 81.2% is the honest predictor of how we do there, not our 94.7%.
The metrics they suggest reading instead of raw accuracy — recall@K, end-to-end task success, context inflation, cross-session continuity, stability under paraphrase — are the right list. We already publish one of them (delivered tokens per question, item 4 below is what happens when you don’t) and we’re missing the other four. So is nearly everyone.
The code
We published the reference implementation: github.com/todoforai/livemem
The core is about 200 lines of TypeScript, because the architecture genuinely is small:
conversations ──extract──▶ dated facts + embeddings ──pack──▶ ≤ N-token block
(1 LLM call, (cosine + greedy knapsack,
offline) no LLM on this path)The bet the design makes: extract everything once, select per question, never call an LLM to retrieve. Selection is cosine similarity plus a greedy knapsack under a token budget — tens of milliseconds — so you can afford to re-select for every single question instead of maintaining one static “user profile” that’s stale by definition.
Our hosted API adds the tuned extraction prompting and the retrieval refinements described
above — number-aware dedup, entity cards, conversation-window units — and the benchmark
rows above were run in that configuration; the repo is the same architecture without that
tuning. The benchmark harness in bench/
runs against either one through the same provider, so you can measure the difference
yourself rather than taking our word for it.
The per-question outputs behind our principal rows — answers, judge verdicts, context-token counts — are in that repo, each labelled with the answerer, judge and budget it used. Some intermediate numbers in this post come from validation subsets we ran during development and did not publish as full artifacts; they’re marked as such. Numbers from other systems are linked to their published source and were not re-run by us. If we got something wrong, it’s falsifiable, which is the only claim worth making about a benchmark.
We know where the next points are: multi-hop at 81.2%, aggregation, LongMemEval_S where we still trail Mem0 by 6.6 points, BEAM — which we haven’t run and expect to hurt — and the hosted live path still behind our offline one. We’ll publish the next number the same way — measured against this one, with the artifacts attached.
If you want the memory without running it yourself, it’s what powers agent memory in TODOforAI.