# Claim + evidence-span recall — the LongMemEval pivot, and an honest LoCoMo ceiling _2026-06-17. A full accounting of a multi-hour autonomous run: why "LoCoMo 100%" is the wrong target, what LongMemEval proves instead, and the substrate change it produced — a deployed `claim + evidence-span` recall unit. Every number here is measured, not asserted. For the reports index, see [Reports](/reports). Companion to [LoCoMo failure analysis](/reports/locomo-failure-analysis) and [LoCoMo claim layer](/reports/locomo-claim-layer)._ --- ## TL;DR | claim | verdict | evidence | |---|---|---| | LoCoMo 100% is reachable by evolving donto | **false — it's reader/benchmark-capped, not donto-capped** | same full-context dump, three readers: gemini-2.5-flash-lite 45.5% → deepseek-chat 74.7% → gpt-5.4 86%. The retrieval is complete in all three; the reader is the cap. Ceiling ~90% (subjective-inference golds + image-text BLIP captions that can't OCR). | | LoCoMo can demonstrate donto's value | **false** | LoCoMo conversations *fit* in context, so the dump saturates retrieval — donto's claim layer adds nothing either way. Wrong benchmark for the thesis. | | LongMemEval is where the claims thesis bites | **true, but only when the dump doesn't fit** | with a *weak* reader, claims+spans beat the dump on temporal (0.65 vs 0.50) at 3.7× fewer tokens; with a *strong* reader (GLM-4.7) the dump that *fits* in context wins everywhere (temporal 1.00 vs 0.90). The reader-independent win is **token-efficiency** (3.5–4.8× fewer); the accuracy win requires the non-fitting `_m` split (untested). See §2a. | | The right recall unit is the claim **with its evidence span** | **true — robust across readers** | claims-only → claims+spans lifts accuracy on *both* readers (deepseek 0.30→0.38, GLM 0.47→0.50; temporal +5–10 pts). Atomized triples hurt reader synthesis; the anchored source snippet restores it. | | The win is now in the substrate, not just a harness | **true** | donto-memory `/recall` ships an `agent-claims` arm (commit `2535ab7`) that surfaces `ctx:claims//` claims with their evidence spans. Verified live. | --- ## 1. The goal, and why the literal target is a dead end The standing goal was blunt: *evolve donto until it hits 100% on LoCoMo; when a benchmark question is provably broken, flag it and skip it.* No judge-gaming. The honest finding, established rigorously, is that **100% is not reachable and the gap is not donto's.** LoCoMo answers are read by an LLM from whatever context the memory system supplies. When donto supplies the *full-context dump* (every conversation turn), the reader has everything — retrieval is trivially complete — and the score becomes a measurement of the **reader**, not of donto: | reader (same dump context) | LoCoMo accuracy (100-Q balanced) | |---|---| | google/gemini-2.5-flash-lite | **45.5%** | | deepseek-chat | **74.7%** | | gpt-5.4 | **86%** | The same 100 questions, the same donto context, three readers, ~40 points of spread. The weak reader (`flash-lite`) refuses inference questions outright — it answers "Not enough information" to *"Would Caroline be considered religious?"* even with the full transcript in front of it. The strong reader answers them. **donto's retrieval is not the variable.** The residual gap to 100% even for the best reader is the benchmark itself: a meaningful fraction of LoCoMo golds are subjective inferences where several answers are defensible, plus image questions whose answer lives in a photo that the dataset only exposes as a coarse BLIP caption (no OCR). One gold was *provably* wrong (a charity-race day-of-week contradicted by the source) and was removed with a written audit trail ([`EXCLUDED_QUESTIONS.md`](/reports)). The rest are not "broken" — they're hard or subjective — so they cannot honestly be skipped. **Conclusion:** chasing LoCoMo 100% would mean gaming the judge. The right move is to (a) report the LoCoMo number honestly, and (b) pivot to a benchmark where donto's design actually matters. ## 1a. The definitive full run: all 1,539 questions To put a complete number on it rather than a 100-question sample, we ran the **entire LoCoMo benchmark — all 1,539 scorable questions across all 10 conversations** — full-context dump (mean **39,484** context tokens/question, i.e. the whole conversation), images included as BLIP captions, with the cheap `gemini-2.5-flash-lite` as both reader and judge. Cost: **$7.94** (~$5.16 per 1,000 questions); zero empty answers. | category | accuracy | n | |---|---|---| | single-hop (cat-4) | **0.734** | 841 | | open-domain (cat-3) | 0.344 | 96 | | temporal (cat-2) | 0.297 | 320 | | multi-hop (cat-1) | 0.266 | 282 | | **OVERALL** | **0.533** | **1,539** | Per-conversation accuracy is tight (0.46–0.65), so the number is stable across the corpus, not an artifact of one easy/hard conversation. Read this number correctly. **Single-hop — direct factual lookup — is 0.73: the reader can read.** The collapse is entirely in the *inference* categories (multi-hop, temporal, open-domain at 0.27–0.34), because `flash-lite` refuses or fumbles the reasoning step even with the full transcript in context. This is the exact signature of the reader-ladder above: the same dump, a weak reader. On the 100-question balanced set the *capable* deepseek-chat reader scored **74.7%** and gpt-5.4 **86%** on identical context. So **the honest reading of "53.3% on the full benchmark" is "this is what a $0.10/M reader does with complete information"** — a floor set by the reader, not a measure of donto's retrieval, which supplied the whole conversation in every case. (Running the full set on a capable reader would cost ~3–8× more; the cheap reader is what the budget allowed, and it makes the reader-bound nature unmistakable rather than hiding it.) ## 2. The pivot: LongMemEval, where the dump can't save you LoCoMo fits in context. **LongMemEval (LME)** does not — its haystack is dozens of long sessions per question. When the dump doesn't fit, you *must* retrieve, and the question becomes: *retrieve what, in what form?* donto's answer is its thesis: a **bitemporal, paraconsistent, evidence-anchored claim layer**. We extracted that layer for the full LME oracle set — 1,054 chunks, ~115K claims into per-chunk contexts `ctx:claims/lme/` — then compared three recall arms with a *fixed* cheap reader (deepseek-chat) and the verbatim per-type LME judge: | arm | accuracy | avg tokens | knowledge-update | multi-session | temporal | |---|---|---|---|---|---| | **DUMP** (all session turns) | 0.467 | 9,414 | **0.65** | **0.25** | 0.50 | | **CLAIMS** (top-100, ranked) | 0.300 | 1,967 (4.8× fewer) | 0.20 | 0.10 | 0.60 | | **CLAIMS + EVIDENCE SPANS** | 0.383 | 2,533 (**3.7× fewer**) | 0.35 | 0.15 | **0.65** | Two results matter here. **First — the claim layer is an *expansion*, not a compression.** One holder had 4,272 claims = 51K tokens versus a 10.5K-token dump. Abundance means *more* claims than source text. So "dump all the claims" is strictly worse than the dump. The claim layer pays off only through *ranked retrieval*: feed the top-K claims relevant to the question, not all of them. That is the whole point of a substrate — hold everything, retrieve precisely. **Second — the right recall unit is the claim *with its evidence span*.** Bare triples (`ex:cool-dry-storage avoids ex:heat-source`) are great for retrieval and ranking but *hurt* the reader's ability to synthesize an answer — the connective tissue of natural language is gone. Pairing each retrieved claim with the anchored source snippet it came from (`… — "store it in a cool, dry place away from heat"`) gives the reader both: structure for retrieval, prose for synthesis. That single change lifted knowledge-update from 0.20 → 0.35 and pushed temporal past the dump — at 3.7× fewer tokens. This is donto's always-on citer earning its place: extraction (what) and anchoring (where) are separate stages, and recall returns both. ### 2a. The cross-reader check that corrected us The table above used a *cheap, weak* reader (deepseek-chat). Re-running the same three arms with a *strong* reader (GLM-4.7) changes the story in an important, honest way: | arm (date-sensitive types) | weak reader (deepseek) | strong reader (GLM-4.7) | |---|---|---| | DUMP | 0.467 | **0.900** | | CLAIMS | 0.300 | 0.467 | | CLAIMS + SPANS | 0.383 | 0.500 | | temporal: DUMP vs CLAIMS+SPANS | 0.50 vs **0.65** | **1.00** vs 0.90 | With a strong reader the dump leaps to 0.90 and **wins everywhere, including temporal**. So "claims+spans beat the dump on temporal" was **reader-specific** — it held only because the weak reader couldn't synthesize the big dump, and the compact claim layer rescued it. Give the reader enough capability to actually use a dump that *fits in context*, and the dump wins. Two things survive the cross-reader check and are the honest takeaways: 1. **claims+spans > bare claims on *both* readers** — evidence spans reliably help synthesis (the §2 point holds). 2. **Token-efficiency is the demonstrated standalone win** — claims+spans reach a large fraction of the answer at **3.5–4.8× fewer tokens**. What does **not** survive: claims beating the dump on *accuracy* when the dump fits. That is the same lesson as LoCoMo — when everything fits in context, a capable reader plus the raw text wins, and donto's value is efficiency, not accuracy. donto's *accuracy* win requires the regime where the dump **cannot** fit (LME `_m`), which remains the decisive untested experiment. Reporting this correction is the point: a result that only reproduces under a weak reader is not a substrate win. ### What the residual gap is (and isn't) We diagnosed every knowledge-update failure. The losses are **not** a recall or representation flaw: - **43%** are extraction *coverage* misses — the answer fact (e.g. "spent 10–12 hours on the sculpture") was never extracted into a claim. A deeper-gleaning extractor closes these. - **~50%** had the fact present in the top-K but the reader still missed it — the synthesis problem that evidence spans address. - We also tested attaching `valid_time` (session dates) to every claim: it barely moved knowledge-update (0.25 → 0.30). **Dates were not the bottleneck** — coverage and synthesis are. An honest negative result worth keeping. --- ## 3. The substrate change: `agent-claims` recall, deployed A benchmark win that lives only in a harness is scaffolding. So the claim+evidence-span unit was promoted into donto-memory's live `/recall` (commit `2535ab7`): - New overlay **`agent_claims_for_holder`** surfaces a holder's donto-agent-extracted claims from `ctx:claims//` — which the native claim-FTS arm structurally can't see — scoped via the documented holder↔chunk convention (corpus = the holder's middle segment), FTS-ranked over a humanized projection. Evidence spans attach in the hot path. - It runs as a **5th fused arm** in the semantic-claim module, gated `DONTO_MEMORY_AGENT_CLAIMS` — ships dark, degrades gracefully, never breaks recall. **Verified live:** `/recall` for an LME holder asking *"first issue with new car"* now returns the answer-bearing `affectedComponent — "issue with my car's GPS system"` claim+span (gold: "GPS system not functioning") that the episodic-only path missed entirely. Two bugs were caught here by *reading the live service logs*, not by assuming success: a SQL parameter-ordering mismatch, and a query plan that — once the substrate crossed 42M statements — seq-scanned 513K records and bit-mapped a 42M-row index, timing out at 96s. The fix forces the holder-index lookup first; 96s → ~4s. ### The honest perf wall Making the arm *reliably* sub-second means precomputed FTS. We tried backfilling the existing `donto_x_memory_chunk_fts` table (a holder-scoped GIN) for the LME claims — and hit a hard wall: even a *single* holder's backfill times out at 60s. The cost is **GIN-index insert**, not the query, and it's amortized badly in bulk over a table that already holds millions of BEAM claims. The correct place to pay that cost is *incrementally, at extraction-ingest time* — one claim per insert — not a bulk backfill. So the arm currently ships best-effort: reliable at modest recall limits, degrading gracefully under load, with the incremental-ingest fix documented as the clean follow-up. Abandoning the rabbit hole was the right call. --- ## 4. The plumbing that made it possible - **Extraction infrastructure.** LME extraction had silently failed: at `--concurrency 3` the Hyades gateway returns 200-with-empty-content under load, marking chunks "done" with zero facts. Fixed to `--concurrency 1` (462 facts/chunk) and added a reliable **OpenRouter `gpt-4.1-mini`** provider lane to donto-agent (`bd74b7b`) — no weekly cap, no empty-under-load — to drain the queue in parallel at ~$0.006/chunk. The full layer completed: 1,054/1,054 chunks. - **A 19-hour runaway killed.** A deferred migration backfill was found running as a single query for ~19 hours doing disk IO, loading the box. Cancelled (I3-safe); load dropped from 7 to under 3. It's now permanently unneeded — the agent-claims arm reaches the same claims via the context-join. - **Disk crisis resolved + automated.** The boot disk hit 99%. Reclaimed ~69 GB (regenerable Rust build targets + 649 stale opencode extraction-exchange dirs, results long since ingested) → 39%. Root cause fixed with a daily cron so it can't recur. Every one of these was a "looks done ≠ is working" catch — the discipline that separates a launched job from a working one. --- ## 5. Honest limitations - The headline LoCoMo number is **reader-bound**. With a cheap reader it is low; that reflects the reader, not the retrieval. We report it next to the reader-ladder so the bound is explicit. - The LME claims experiment uses a **lexical** top-K ranker as a stand-in for donto's full FTS+vector recall — a floor, not a ceiling, on what ranked claim recall can do. - The decisive test — LME `_m`, whose haystack does **not** fit any dump — was not run (the `_m` split isn't downloaded). That is where claims+spans is the *only* viable arm and donto should win outright, not merely tie-cheaper. It is the clear next experiment. - The agent-claims recall arm is **best-effort** on latency for very large holders until the incremental-ingest FTS registration lands. --- ## 6. What to do next 1. **Incremental chunk_fts registration** at donto-agent ingest — makes claim recall reliably sub-second for all consumers (pay the GIN cost per-claim, not in bulk). 2. **Deeper-gleaning extraction** to close the 43% knowledge-update coverage gap. 3. **Run LME `_m`** — the benchmark donto was built for. 4. **A cross-encoder rerank + time-decay** on claim recall (the two levers the strongest published memory systems have that donto lacks). The thesis held where it should: when the context doesn't fit, an evidence-anchored claim layer recalls the answer in a fraction of the tokens. The substrate now does this in production. The number that doesn't move — LoCoMo — doesn't move because of the reader and the benchmark, and saying so plainly is the point. ## Update (2026-06-18): extraction coverage IS the lever — controlled result The one place the claim layer clearly lagged at equal budget was knowledge-update (claims+spans 0.50 vs episodic 0.80). The diagnosis was always extraction *coverage* — the live LME layer was drained with a cheap single-pass extractor (~85–130 facts/chunk), not deep gleaning. So it was tested cleanly: the 10 knowledge-update holders' chunks were **re-extracted with 6-pass Hyades gleaning** (additive, dedup by content-hash) — coverage rose **3.5×** (3,119 → 10,923 facts) — and the equal-budget showdown was re-run, read off the **same holders, before vs after**: | arm (same 9 KU holders) | single-pass | deep coverage (3.5×) | |---|---|---| | **claims + spans** | 0.56 | **0.89** | | episodic turns (control, identical source) | 0.78 | 0.78 | **claims+spans jumped +0.33 (0.56 → 0.89), matching and exceeding episodic — while the episodic control reading the same unchanged dialogue did not move.** Three holders flipped wrong→right: exactly the cases where the answer fact had never been extracted by the shallow pass. Because the control is flat, the lift is unambiguously from the extraction. **Verdict:** donto's recurring knowledge-update deficit was an extraction-*depth* artifact, **not** a representation or synthesis limit. Given deep gleaning, the claim+evidence-span unit reaches parity-or-better with raw text at a fraction of the tokens. The budget-showdown "tie" and the cross-reader "dump wins when it fits" were both partly coverage-limited — and the justified next investment is a deep re-extraction of the full layer. This 10-holder slice is the proof it pays. (Full write-up in the [PDF treatise](/reports/donto-treatise.pdf), §3.7.)