A self-reported benchmark number is worth exactly as much as your ability to check it. Ours says 56.76% on LongMemEval-V2 Small at 0.187 seconds per query — best accuracy in the sub-second regime, hundreds of times faster than the minute-scale agent baselines that score higher. You shouldn't take that on faith. So don't:
curl -LO /results//repro/udi-lmev2-small-reproduce-v0.3.1.tar.gz
tar xzf udi-lmev2-small-reproduce-v0.3.1.tar.gz
python3 reproduce.py
It prints PASS and the exact numbers: 256 correct out of 451, 56.762749%, 0.187 seconds average. Every digit recomputed in front of you.
How it can be honest and sealed at the same time.
Here is the part that matters. The system that produced the answers is protected — its internals are background IP and appear nowhere in the bundle. That sounds like it should block reproduction. It doesn't, because of an order-of-operations that is easy to miss: the sealed system already produced its 451 answers, and the benchmark's own official scorer already judged each one. Those judgments are frozen. Recomputing the headline number from them needs no model, no mechanism — just arithmetic anyone can audit.
You can publish the proof without publishing the machine.
The bundle carries the scoring layer only — each question, our answer, the gold answer, the judge's verdict, the query time. Memory internals, prompts, and geometry are excluded by construction. What you get is enough to check the claim and nothing that rebuilds the system. A content hash binds it: change one byte and it no longer matches.
Where we're weakest, on purpose.
The reproducer also prints a category breakdown, and we left it in because it's the most useful thing on the page. When the honest answer is "the record doesn't say," we fall down:
| Category | Accuracy |
|---|---|
| static (answerable) | 74.6% |
| dynamic (answerable) | 76.7% |
| procedure (answerable) | 71.6% |
| static-abstention | 30.9% |
| static... procedure-abstention | 9.4% |
Answerable questions: low-to-mid seventies. Abstention questions — the ones that require saying "I don't know" instead of inventing an answer — collapse into the single digits and thirties. For a project whose whole thesis is that refusal should be first-class, that is exactly the gap we have to close, and now it's measured instead of hand-waved. It's the next experiment.
What this is, and isn't.
This reproduces the score from the official scorer's frozen judgments. It is not, by itself, an independent re-run of the judge — the README shows that deeper path, which re-grades our answers with the official evaluator and a judge-model key. And it is not an accepted leaderboard entry; it's a self-reported development result, submitted and awaiting review. We say all of that on the page, because a claim you have to soften later isn't worth making.
If the number is wrong, the fastest way to show it is the three lines above. Run them. If you find something, we'll publish what you find — that's the whole point.