PLATE · Ⅲ Benchmarks Open harness · reproducible

Honest benchmarks

The AI-memory market publishes leaderboard numbers that don't survive independent reproduction. We do it differently: every Mnem figure on this page comes from an open harness you can run yourself in one command — fully offline after the one-time dataset and embedding-model downloads. The only cloud component anywhere below is the benchmark's own judge model, and it is pinned.

Verified performance generated from harness output

†1

LongMemEval end-to-end answer accuracy over all 500 questions — the figure the market quotes — with the official reader and judge prompts, a gpt-4o reader, and the benchmark's own judge pinned.

LongMemEval_s · QA ·

†2

Retrieval Recall@5 on the same dataset: a labeled evidence session sits in Mnem's top-5 recalled sessions. No answering model, no judge — the layer we ship, measured alone.

LongMemEval_s · retrieval · questions

†3

Retrieval Recall@10 — the context window the end-to-end reader actually receives. The gap to 100% is the ceiling on the QA number above.

LongMemEval_s · retrieval · top-10

†4

On-device recall latency at p95, measured through the shipped search path with no network round-trip. The column that decides product fit.

Local harness · recall p95 ·

† Method notes — what was measured, and how

Our rules

Loading…

LongMemEval, end to end open harness

Question type	n	Accuracy

Reproduce it: python benchmarks/run_longmemeval.py --save-context then python benchmarks/run_longmemeval_qa.py --contexts benchmarks/results/longmemeval_contexts_latest.jsonl --out benchmarks/results/qa-gpt4o --base-url https://api.openai.com/v1 --reader-model gpt-4o --judge-model gpt-4o-2024-08-06. The reader and judge prompts are verbatim copies of the benchmark's own; the hypotheses file is in the official layout, so the upstream evaluate_qa.py grades it unchanged.

Reading it against vendor claims

LongMemEval, retrieval stage open harness

Question type	n	any@1	any@5	any@10

Reproduce it: python benchmarks/run_longmemeval.py --download then python benchmarks/run_longmemeval.py — offline after the one-time dataset and embedding-model downloads (the harness aborts rather than silently substituting the non-semantic fallback encoder).

Published reference points on the same dataset

System	Metric	Figure

Mnem, measured locally open harness

Category	Items	Recall@1	Recall@5
Loading results…

Reproduce it: python benchmarks/run_benchmarks.py — dataset, scorer, and config ship with the app's benchmark kit. Offline after the one-time embedding-model download (the harness aborts rather than silently substituting the non-semantic fallback encoder).

What competitors claim vs. what reproductions found

These are not our measurements. Claims are the vendors' own self-reported figures; reproductions are third-party runs (some by competing vendors — noted where known). We list both because, as of 2026, cross-vendor leaderboard numbers are harness-relative and not comparable. The one axis they share with us is the end-to-end LongMemEval figure above — read the caveats there first.

The columns that decide product fit

	Where memories live	Recall path	Portability

Competitor claims and reproductions cited as of the date in each entry. Nothing on this page is hand-edited: figures are generated from harness output.

Run the numbers yourself.

Sub-10ms recall on your machine, with nothing leaving it.

Get Mnem Read the docs