N4 — can a decision model do the memory judge's job? 2026-09-21
The memory benchmarks ask their judge one narrow question per item: does this response give the gold answer? No explanation is stored, no partial credit is given, the verdict is binary. That is the shape a decision model is built for, and this is the measurement of whether it can.
Sample N4 (dotnet run -- 103), 42 LongMemEval oracle questions across all six question types,
84 items, 0 errors on either arm. Arm A zai-org/GLM-5.3-Flash@bitdeer through the shipped
LongMemEvalJudge; arm B jev-1.13.0@typesafe through the new DecisionBenchmarkJudge.
The design, and the floor it builds in
A judge-vs-judge study needs responses to judge, and running an agent over LongMemEval first would cost hours and mix the agent's errors into the judges' scores. So each question contributes two items:
- its own gold answer as the response — a judge must say correct;
- a distractor: another question's gold answer, of the same question type — a judge must say wrong.
Both judges see identical items, and the design has a real chance floor of 50%: a judge that says "correct" to everything scores exactly 0.500, as does one that says "wrong" to everything. This measures judge discrimination on memory items. It is not agreement on a real run's messy, partially-correct answers, which is harder and is still owed.
Result
| arm A — GLM-5.3 Flash | arm B — Jev | |
|---|---|---|
| items scored | 80 of 84 | 84 of 84 |
| accuracy (floor 50%) | 96.3% | 100.0% |
| on gold answers | 92.1% | 100.0% |
| on distractors | 100.0% | 100.0% |
| p50 per item | 1,458 ms | 293 ms |
| judge tokens | 19,675 | 36,964 |
Judge-vs-judge agreement: 96.3% on the 80 items both scored.
⚠️ The single-session-preference row is withdrawn. That type's "gold answer" is a rubric describing
what a good personalised answer does, not an answer to match — so feeding the gold back as the correct
response asks a judge whether a rubric satisfies itself. Both numbers in that row measure nothing, and the
sample now excludes the type from its default comparison (--types single-session-preference still runs it
for anyone who wants to look). The overall figures above include those 10 items and are left as they were
measured; excluding them would change arm A's 96.3% and is a re-run, not an edit.
| question type | n | A | B |
|---|---|---|---|
| knowledge-update | 14 | 100.0% | 100.0% |
| multi-session | 14 | 100.0% | 100.0% |
| single-session-assistant | 14 | 100.0% | 100.0% |
| single-session-user | 14 | 100.0% | 100.0% |
| temporal-reasoning | 14 | 100.0% | 100.0% |
100% is an extreme value, so it was checked before it was written down
A perfect score is a wiring fault until proven otherwise. Three checks:
- It is not one answer repeated. A judge saying "correct" to everything scores 50% by construction here, and arm B scored 100% on both halves.
- The probabilities separate cleanly. Gold items: P(yes) 0.640 – 0.990, mean 0.963. Distractors: 0.010 – 0.190, mean 0.021. The lowest gold sits 0.45 above the highest distractor — a gap no threshold in that range could cross.
- The values vary. 14 distinct probabilities across the 84 items, not a constant.
The separation is real. What it means is bounded by the design: these are clean items — a verbatim correct answer against a clearly unrelated one. Real agent answers are partial, hedged and differently worded, and this says nothing about those.
The four items arm A could not score
The generative judge returned 4 unparseable verdicts (status Invalid, all on expected-correct items):
4.8% of its calls produced no usable judgement, and they are excluded from its 96.3%. The decision model
returned zero — a typed protocol cannot produce an unparseable verdict, which is the operational
difference between a JSON-in-free-text contract and a schema. It is the same class of defect as the judge
parser that once vetoed valid "yes" verdicts in this repository.
That is worth more than the accuracy gap: it is not that Jev is a better judge here, it is that 5% of the generative judge's answers needed a retry or a fallback and none of the decision model's did.
What this settles, and what it does not
- Settled: on narrow, clean memory items a decision model discriminates a correct answer from an incorrect one at 100% across the five question types where the gold answer is an answer, at a fifth of the latency, with no parse failures. It is at least as good as the incumbent judge on this task.
- Not settled — and this is the gap that matters: TypedMemEval's shipped judge-agreement bar is 0.999, and nothing here has been measured against it. This study has 84 clean items; that bar is about a judge's agreement with itself and with the incumbent on real runs, at a scale where 0.1% matters.
- Not settled — the abstention lane. LongMemEval marks 30 of its 500 questions as abstention items
(
_abs), where the conversation does not contain the answer and recognising that is the correct behaviour. Four of them were in this run, and both judges scored them correctly — but only because this design feeds the gold answer as the response, so the item reduces to matching identical text. A real abstention response was never judged here. Worse, the adapter as first written sent those items the ordinary rubric, which says a refusal is not a match, and would have scored every correct abstention wrong; it now branches to an abstention rubric, as the shippedLongMemEvalJudgedoes. That branch is unit-tested and has not been measured live. The distractor pool now excludes abstention items as well: once abstention questions are judged by "does the response recognise it cannot answer?", another abstention item's gold text is a correct refusal, so using it as the negative half would put valid answers where the design requires wrong ones and the 50% floor would stop holding. The published run is unaffected — it predates the abstention rubric, so those items were judged by the ordinary "contains the gold answer?" question, under which another abstention explanation is correctly negative. - Not settled: partially-correct answers, which is where memory judging is actually hard and where the documented weaknesses (counting, dates as text, literal reading) would show. 32 of the 500 gold answers are numbers; they are included here and cost nothing, but a verbatim number is not a counting task.
- Therefore: the adapter ships as a comparison instrument. No memory benchmark switches to it, and a per-shape agreement study on a real run against the 0.999 bar remains the gate — exactly as the proposal ordered it.
Reproduce
The sample now excludes single-session-preference from its default comparison (see the withdrawn row
above), so reproducing these figures needs that type asked for explicitly:
# the run this file reports, all six types including the withdrawn preference items
dotnet run -- 103 --limit 42 --types knowledge-update,multi-session,single-session-assistant,single-session-preference,single-session-user,temporal-reasoning
# what the sample does by default now — five types, and NOT the numbers above
dotnet run -- 103 --limit 42
dotnet run -- 103 --dry-run --limit 12 # builds every item, renders the first request, sends nothing