Testing donto Against LongMemEval — A Long-Term Memory Benchmark Study

A research report on the LongMemEval benchmark (ICLR 2025), its evaluation mechanics, the 2025–2026 leaderboard landscape, and a candid audit + port plan + experiment design for testing donto-memory — the agentic-memory example consumer of the donto substrate — against it.

Date: 2026-06-04 · Status: study, parity audit, and experiment plan — no donto run against LongMemEval has been executed yet. Every "donto should win on X" statement below is a hypothesis derived from the substrate's documented properties, with the counter-hypotheses flagged.

What was independently verified for this report (vs. taken on trust). Everything in §2–§3 (data counts, abstention distribution, sessions-per-instance, the verbatim examples, all judge prompts, the metric/assert logic, the retrieval-eval rules) was re-derived empirically from the local data/longmemeval_oracle.json (500 instances) and src/evaluation/* + src/retrieval/* — file:line cited inline. Every donto-memory claim in §6 (the valid_from=None finding, the ILIKE recall weights, the live DONTO_MEMORY_USE_EXTRACT=1 bridge, the citer's 90.2%-on-3519 figure, the glean fact-counts) was re-checked against live source/output file:line. Not re-derived in this pass: the §4 numbers from the paper PDF (tables/figures — no PDF is on disk; cited from the paper, figure-bar reads flagged ±0.01) and the §5 third-party leaderboard numbers (from vendor blogs/arXiv abstracts, explicitly caveated as not apples-to-apples). Treat §2/§3/§6 as measured, §4/§5 as cited.

1. Executive summary

LongMemEval (Wu et al., arXiv 2410.10813, ICLR 2025) is the reference benchmark for long-term conversational memory. Its single most-cited result: long-context LLMs reading a full ~115k-token chat history lose ~30% accuracy versus reading only the evidence ("oracle") sessions — GPT-4o falls 0.870 → 0.606 — and weaker models collapse 50–66%. That gap is the entire reason dedicated memory systems exist. The benchmark probes five abilities — Information Extraction, Multi-Session Reasoning, Knowledge Updates, Temporal Reasoning, and Abstention — across 500 questions in six question types plus a 30-question abstention overlay (all counts verified in the local data file).

donto's substrate is, on paper, unusually well-shaped for the hardest of those abilities. Its four differentiators map onto the three abilities that resist flat retrieval-augmented memory:

But there is a critical caveat that gates the whole experiment, and it is the reason this report exists. donto-memory currently runs the old, triplicated extraction stack and never received this session's refactors to the genealogy/donto-extract engine. Worse, the audit surfaced a gap outside the five named refactors that is even more fatal: every memory fact today is stored with valid_from/valid_to = None (verified — no valid_from is passed anywhere in crates/donto-memory-core/src/modules/). So the bitemporal store that powers donto's two headline strengths (knowledge-update and temporal) has no bitemporal signal at all. A fair test requires three required ports — valid-time ingest, the always-on semantic citer, and the lane-router with failover — before donto-memory can credibly contest the abilities it claims.

The honest predicted profile is not "donto wins overall." It is a distinctive shape: above-class on knowledge-update, abstention, and (if ported) temporal, via clean ablations; at-or-below class on the three single-session information-extraction types, bottlenecked by donto-memory's recall being lexical (ILIKE + GIN FTS), not dense — the paper deliberately chose a dense retriever (Stella-1.5B) over BM25 because incidental, paraphrased facts have low lexical overlap with the question. The report's spine is therefore four ablations that isolate the substrate advantage from the reader-model and retrieval confounds that make the public leaderboard largely incomparable.

2. What LongMemEval is

2.1 Thesis and motivation

An interactive assistant's value compounds with its memory across sustained interaction. The paper's abstract reports a 30% accuracy drop when commercial chat assistants and long-context LLMs must memorize information across sustained interactions rather than read it fresh. Authors (per the repo README.md + the arXiv citation block): Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, Dong Yu — UCLA / Tencent AI Lab Seattle / UC San Diego; the work was done during Di Wu's Tencent internship. arXiv 2410.10813; ICLR 2025; the local repo's cached v2 PDF link is dated early 2025.

The gap it fills: prior long-term-memory benchmarks (MemoryBank, PerLTQA, DialSim, and even the recent LoCoMo) have short, non-configurable histories (a few thousand tokens) and cover memory abilities thinly. Per the paper's Table 1, LongMemEval is the only benchmark covering all five abilities — crucially it is the only one to evaluate recall of information provided by the assistant and reasoning with updated user information (knowledge updates), which every prior benchmark omits.

Problem formulation (§3.1). An instance is a 4-tuple (S, q, t_q, a). S is an ordered sequence of N timestamped sessions [(t1,S1)...(tN,SN)], each a multi-turn user↔assistant dialogue decomposable into rounds (one user message + one assistant reply). At test time S is streamed one session at a time — the "online" memorization protocol — and the question q (with date t_q > t_N) is asked after all sessions. a is a short phrase or a rubric. The framing is a harder needle-in-a-haystack: the needle must be retrieved and synthesized from multiple evidence sessions buried among filler.

2.2 The five core abilities (§3.2)

1. Information Extraction (IE) — recall a specific fact from extensive history, "including the details mentioned by either the user OR the assistant." Assistant-side recall is novel versus prior benchmarks.
2. Multi-Session Reasoning (MR) — "synthesize information across multiple history sessions … that involve aggregation and comparison." (e.g. "How many items of clothing do I need to pick up?" → 3, summed across three sessions.)
3. Knowledge Updates (KU) — "recognize the changes in the user's personal information and update the knowledge of the user dynamically over time." (e.g. a 5K personal-best stated as 27:12, later updated to 25:50 — return the current value.)
4. Temporal Reasoning (TR) — "awareness of the temporal aspects of user information, including both explicit time mentions AND timestamp metadata." Must reason over session timestamps and in-text dates.
5. Abstention (ABS) — "identify questions seeking unknown information … and answer 'I don't know'." Built by taking the other types and modifying them into false-premise questions.

2.3 The six question types + the abstention overlay (verified in the local data)

Counts read directly from /home/ajax/longmemeval-study/data/longmemeval_oracle.json (500 instances):

Abstention is an overlay, not a seventh disjoint bucket. The 30 abstention instances are those whose question_id ends in _abs, drawn from the other types. Verified distribution: multi-session 12, temporal-reasoning 6, knowledge-update 6, single-session-user 6 — and none in single-session-assistant or single-session-preference. (The README lists exactly the six question_type values above and defines abstention as the _abs-suffix rule, README.md:81 — i.e. structurally a cross-cutting flag, not a seventh disjoint type. The paper presents abstention as one of the five abilities, which is why an _abs item is folded into its parent type and reported separately — see §3.2.)

Verified data shape (oracle, evidence-only): sessions per instance min 1 / median 2 / max 6; knowledge-update is always exactly 2 sessions; the three single-session types are always 1; mean ~1.9 evidence sessions per question. single-session-preference answers are rubrics (free-form preference descriptions), not short facts.

2.4 Real examples (verbatim from the oracle file)

• single-session-user — Q "What degree did I graduate with?" → "Business Administration" (stated in a single turn buried inside an off-topic expense-paperwork conversation).
• single-session-assistant — Q "remind me what was the rotation for Admon on a Sunday?" → "8 am – 4 pm (Day Shift)" — the answer lives inside a 14-turn assistant-generated shift-rotation table. (Also: a children's story where the Plesiosaur is "blue"; restaurant "Miss Bee Providore" — all in assistant turns.)
• single-session-preference — Q "recommend resources to learn video editing?" → rubric "would prefer resources tailored to Adobe Premiere Pro … not general/other software" (evidence: user earlier said "I enjoy to use Adobe Premiere Pro").
• knowledge-update — Q "personal best time in the charity 5K?" — session1 (05/25) "27:12", session2 (05/27) "25:50" → "25:50". (Also $350,000 → $400,000 mortgage pre-approval; "three" → "four" Korean restaurants; Chicago → "the suburbs".)
• temporal-reasoning — Q "first issue with my car after its first service?" → "GPS system" (service was 3/15, GPS issue 3/22 — must order event dates lifted from text, not session timestamps).
• multi-session — Q "How many items of clothing to pick up/return?" → "3" (a blazer at the cleaners, boots returned to Zara, a new pair to pick up — across three sessions).
• abstention — Q "name of my hamster?" → "You did not mention a hamster — you mentioned your cat Luna." Every _abs question targets a plausible sibling of a genuinely-mentioned entity (Italian vs Korean; Dr. Johnson vs Dr. Smith; table-tennis vs tennis; Shinjuku vs Harajuku; vintage films vs vintage cameras). The trap is a high-similarity distractor; correct behaviour is to refuse and name the gap.

2.5 The attribute-controlled construction pipeline (§3.2, Fig 2)

Human-expert-built questions; LLM-simulated, human-edited evidence sessions; Llama-3-70B-Instruct as the pipeline LLM unless noted. ~400 human hours of construction + ~150 hours studying commercial systems; 3 expert NLP-researcher annotators.

1. Ontology — 164 user attributes in 5 categories (lifestyle, belongings, life events, situations/context, demographic). For each, an LLM writes an attribute-focused background paragraph.
2. Question construction — sample a paragraph → LLM proposes seed (Q, A) pairs → human experts filter + rewrite for difficulty → manually decompose the answer into one or more evidence statements with optional timestamps.
3. Evidence-session construction — each evidence statement is embedded indirectly into a task-oriented self-chat session (User-LLM ↔ Assistant-LLM). Instead of "I bought a new car last month," the user asks for car-insurance help and reveals it incidentally — forcing systems to memorize details that were never emphasized. All sessions are manually screened to verify inclusion, distribute evidence across positions, and rephrase to colloquial language (especially time mentions). The evidence-statement position is hand-annotated — this is the has_answer:true turn flag.
4. History compilation (Fig 2c) — for each question, sample unrelated sessions, randomly insert the evidence sessions, and assign plausible timestamps to all. Filler comes from (a) self-chat sessions simulated from other, non-conflicting attributes, and (b) publicly released ShareGPT (Apache-2.0) + UltraChat (MIT).

A September 2025 "cleaned" release (xiaowu0162/longmemeval-cleaned) further scrubbed haystacks to remove interference with answer correctness — this is the version downloaded locally and the version any comparable run must use.

2.6 The three variants (same 500 questions, escalating dilution)

question_date spans 2021/08/20 → 2024/02/20. The difficulty in _s/_m comes from haystack dilution, not from the oracle instances themselves (which are small, 2–3 sessions, 12–36 turns).

3. How LongMemEval is evaluated

The harness has three separable stages, and an external system like donto-memory needs to satisfy only one of them to get a headline number.

3.1 The plug-in contract — the only required output

A hypotheses JSONL: one JSON object per line, exactly two fields {"question_id": ..., "hypothesis": ...} (README.md:90-99; consumed at evaluate_qa.py:76-99). hypothesis is your system's free-text answer to entry['question']. You generate it however you like — donto-memory's own /memorize + /recall + /search pipeline is a fully legal black box. You do not have to use src/retrieval or src/generation; those are the paper's reference memory systems, not a required path. Minimal integration: for each of the 500 instances, feed the timestamped history online (session by session, in haystack order), ask entry['question'] with entry['question_date'] as "now", write {question_id, hypothesis}.

3.2 QA accuracy = LLM-as-judge (the headline metric)

src/evaluation/evaluate_qa.py is the scorer:

python3 evaluate_qa.py gpt-4o <hyp.jsonl> ../../data/longmemeval_oracle.json

The judge is hardcoded to gpt-4o-2024-08-06 via the OpenAI API (evaluate_qa.py:11-15), temperature=0, max_tokens=10, and the label is simply 'yes' in eval_response.lower() (evaluate_qa.py:108-113). Meta-eval: the judge agrees with human experts >97%.

The judge prompt differs by question_type — this is load-bearing for how donto must phrase answers (all verified verbatim from get_anscheck_prompt, evaluate_qa.py:24-43):

• single-session-user / -assistant / multi-session — "answer yes if the response contains the correct answer … If the response only contains a subset of the information required, answer no." (Multi-session counting: a subset → no.)
• temporal-reasoning — same, plus "do not penalize off-by-one errors for the number of days/weeks/months."
• knowledge-update — "If the response contains some previous information along with an updated answer, the response should be considered as correct as long as the updated answer is the required answer." → This rubric is written for a supersede-not-overwrite store. You may surface history, but you must lead with the latest value.
• single-session-preference — judged against a rubric, not an exact string: "correct as long as it recalls and utilizes the user's personal information correctly … does not need to reflect all the points."
• abstention (question_id ends _abs, detected at evaluate_qa.py:101) — a different prompt: "answer yes if the model correctly identifies the question as unanswerable … information is incomplete, or some other information is given but the asked information is not." A hallucinated specific answer fails hard.

Aggregation. evaluate_qa.py prints overall + per-type accuracy inline. print_qa_metrics.py (which asserts the judge was gpt-4o-2024-08-06 at line 20 — a hard comparability guard) reports three numbers: Task-averaged Accuracy (unweighted mean over the 6 type-means), Overall Accuracy (micro-average over all 500), and Abstention Accuracy (mean over just the _abs items). Note the asymmetry: an _abs item is folded into its parent type and reported separately, so abstention quality double-counts. There is no precision/recall/F1 for QA — it is pure binary-judge accuracy.

3.3 Memory-recall (retrieval) metrics — optional

Only applies if you produce a retrieval log in the harness's format (src/retrieval/run_retrieval.py + eval_utils.py). Granularity is turn or session, built from user-side text only. Gold docs are identified by the literal substring answer in the corpus_id; non-evidence items are renamed noans_...; a turn keeps answer only if has_answer==True (run_retrieval.py:209,216-225,272). Metrics: recall_any@k, recall_all@k (the headline — all gold docs in top-k, because a question can need multiple evidence turns), and ndcg_any@k for k ∈ {1,3,5,10,30,50}; turn results roll up to session level (eval_utils.py:24-46).

Crucial scoring rule: retrieval eval always drops the 30 abstention instances (no ground-truth location, print_retrieval_metrics.py:12, README.md:206) and any instance with no user-side has_answer turn — so retrieval is scored over ~470, QA over all 500.

3.4 The plug-in path for donto, in five concrete steps

1. Pick a variant — longmemeval_oracle.json (downloaded; easiest) → longmemeval_s_cleaned.json (realistic) → longmemeval_m_cleaned.json (stress).
2. For each instance: online-memorize each session in haystack order, stamping each session with its haystack_dates[i], into a per-instance context/holder (ctx:memory/lme/<question_id>) so instances don't cross-contaminate.
3. After all sessions: query /recall (holder-scoped) + /search (substrate-wide), pass question_date as the current date, produce one answer string. For _abs items the system must be allowed to say "I don't have enough information."
4. Emit {"question_id", "hypothesis"} per line.
5. Score with evaluate_qa.py gpt-4o, then print_qa_metrics.py for the per-type + task-averaged + abstention breakdown.

Note: an eval-only install (requirements-lite.txt: openai, tqdm, backoff, numpy, nltk) is enough to score a hypotheses file. The full retrieval/generation stack (torch, vllm, sentence-transformers) is only needed to run the paper's reference systems — moot for donto-as-black-box.

4. The memory-design taxonomy and the paper's findings (with numbers)

All numbers below are cited from the paper's tables/figures, not re-derived in this pass (no PDF is on disk; the local repo ships only the data + code). Figure-bar reads are flagged ±0.01 and should be treated as approximate. The direction of every finding (long-context drops vs oracle; round-granularity wins; key-expansion and time-aware expansion help; CoN+JSON helps) is robust and corroborated by the §5 landscape; the exact decimals are the paper's.

4.1 The benchmark is hard — the "30% drop" (§3.4, Fig 3)

Commercial systems (Fig 3a; 97 questions, short 3–6-session histories ~10× shorter than _s):

ChatGPT and Coze (GPT-4o) drop 37% and 64% versus offline reading with the same LLM. "ChatGPT tended to overwrite crucial information … Coze often failed to record indirectly provided user information."

Long-context LLMs (Fig 3b; Oracle vs full _s) — the famous drop, per-model:

Chain-of-Note (CoN) does not close it (GPT-4o 0.924 → 0.640). Even the best long-context model loses ~30 points just from history growing to ~50 sessions; weaker models collapse 50–66%. This is the core evidence that "even the most capable long-context LLMs struggle to manage an ever-growing interaction history without an effective memory mechanism."

4.2 The unified memory-design framework (§4) — 3 stages, 4 control points

Stage 1 Indexing · Stage 2 Retrieval · Stage 3 Reading. Control points: CP1 Value (granularity of the stored item), CP2 Key (what you index on), CP3 Query (how you search), CP4 Reading strategy. The authors' own best design = value:round, key:V+fact, query:question+time, retrieval:flat, time-aware:Yes, reading:CoN.

CP1 Value — granularity (§5.2, Fig 5). A round (one user turn + reply) is the optimal storage granularity — better than session and better than compressed-fact. Compressing into summaries/individual facts hurts overall QA via information loss — except for multi-session reasoning, where fact decomposition consistently helps (uniform, simplified format). Reader-capacity-dependent budget: Llama-3.1-8B degrades past ~3k retrieved tokens; GPT-4o keeps improving past 20k.

CP2 Key — index expansion (§5.3, Table 3). Condensed forms alone (summary/keyphrase/fact) do not beat the value itself. But expanding the key to original value + extracted user facts improves both retrieval and QA: +9.4% recall@k, +5.4% final accuracy averaged. (Value=Session: K=V recall@5 0.706 → K=V+fact 0.732; end-to-end GPT-4o 0.670 → 0.714.)

CP3 Query — time-aware expansion (§5.4, Table 4) — the TR fix. Index values with their dated events; at retrieval, an LLM M_T extracts a time range from the query and filters out-of-range items. +11.3% TR recall (rounds), +6.8% (sessions). Caveat: depends on a strong LLM — Llama-8B "struggles to generate accurate time ranges, often hallucinating or missing temporal cues."

CP4 Reading — Chain-of-Note + JSON (§5.5, Fig 6) — up to +10 points. Even with perfect (oracle) retrieval, a bad reading strategy costs up to 10 absolute points. Two fixes: (1) present retrieved items as structured JSON; (2) Chain-of-Note — instruct the LLM to first extract relevant info from each item, then reason. Oracle QA (figure-read, ±0.01): GPT-4o NL+Direct 0.862 → JSON+CoN 0.924; Llama-3.1-70B 0.762 → 0.848; Llama-3.1-8B 0.710 → 0.756. CoN+JSON wins by a large margin; JSON alone does not consistently beat NL without CoN.

4.3 The five design recommendations

1. Session decomposition into rounds for value granularity.
2. Fact-augmented key expansion (V+fact) for indexing.
3. Time-aware indexing + query expansion for TR.
4. Chain-of-Note + JSON for reading.
5. Overall: retrieval-augmented memory beats raw long-context as history grows.

4.4 The paper's index-expansion machinery (where donto's extraction maps)

src/index_expansion/* generates offline LLM-derived text attached to each session/turn: session-summ (paragraph summary), session/turn-keyphrase (semicolon list), session/turn-userfact (a JSON list of coreference-resolved "personal facts about the user"), and a session temporal-event extractor producing {date, event} pairs. Join modes: separate / merge / replace. The userfact expander is essentially the paper's version of donto-memory's fact extraction — but untyped and not evidence-anchored, which is precisely where donto's typed, span-anchored claims are meant to win on faithfulness. The time-aware query pruning (temp_query_search_pruning.py) has GPT-4o infer a [start,end] date range, pad ±2 days, and reorder retrieved items — a bolt-on hack for exactly what a bitemporal store does natively.

4.5 Reading methods in the reference generator

run_generation.py offers three readers (run_generation.sh:62-71): direct (--cot false); con (--cot true, the author-recommended single chain-of-thought "extract all relevant info, then reason"); and con-separate (--cot true --con true, a separate per-chunk extraction LLM call first — the strongest config). History is always sorted by date, each block labeled with its session date, and the prompt states the current date = question_date.

4.6 Stated limitations / ethics

No dedicated limitations section. Acknowledged: privacy leakage from storing/recalling user info; the lack of a memory deletion operator harms trustworthiness; datastore poisoning could cause jailbreaks (recommend moderation on read/write). English-only synthetic-ish filler. All commercial evals were run in the first two weeks of August 2024, so those numbers are a moving target. The LLM judge has ~3% residual disagreement with humans.

5. Landscape and SOTA (2025–2026)

5.1 Rough leaderboard (all caveated — see §5.2)

Format: System · overall · reader model · method.

Zep is the best-documented graph/temporal comparison and donto's natural comparison class. Its published per-category numbers (GPT-4o): single-user 92.9, assistant 80.4, preference 56.7, multi-session 57.9, KU 83.3, temporal 62.4 — note the two soft spots (preference, multi-session) are exactly the recall-bound types. Letta/MemGPT has not published a LongMemEval number (only LoCoMo 74%); its sleep-time compute (background block consolidation) is the donto-relevant analogue.

5.2 The central comparability finding — leaderboard numbers are not apples-to-apples

Three confounds make the cross-system ranking largely meaningless:

1. Reader model. MatrixOrigin froze one Memoria retrieval snapshot and swapped only the reader: on Knowledge-Update, identical evidence scored 58.4% to 89.6% — a 31-point spread purely from the reader. Much of the "SOTA jump" since 2024 is reader-model progress, not memory progress (the paper's oracle GPT-4o = 87%, while the GPT-5-mini systems hit 93%+).
2. Metric. Task-averaged accuracy (OMEGA) inflates versus raw question-level accuracy by equal-weighting the 30-question preference bucket with the 133-question multi-session bucket. Some quoted "scores" are retrieval recall@k, not QA accuracy at all.
3. Dataset version. The 2025/09 "cleaned" release shifted numbers; pre/post-cleaning scores are not comparable.

5.3 Where the field stands

• Universally hardest ability: multi-session reasoning. Every top system plateaus ~87% (MemMachine 87.2, Mastra 87.2, OMEGA ~83). Synthesis across scattered sessions resists both retrieval and long-context.
• Temporal + knowledge-update are now largely "solved" by systems with explicit temporal anchoring (Mastra's "three dates", Zep's bitemporal edges) + a strong reader — which is precisely the capability donto claims natively.
• RAG beats long-context. The full 115k path actively loses ~30% (lost-in-the-middle). The only non-retrieval winner (Mastra) keeps a compressed dated log in context, not raw history.
• The single highest-leverage knob is retrieval, not extraction. MemMachine's ablation: retrieval-stage tuning (depth k20→30 = +4.2%, JSON formatting +2.0%, search-prompt +1.8%) dwarfs ingestion-stage tuning (sentence chunking +0.8%). This is the most important external finding for donto: its retrieval quality will dominate the score more than its 10-lens extraction richness.

5.4 LongMemEval-V2 (arXiv 2605.12493, 2026) — where the field is going

Not a rescale — a different benchmark. 451 manually-curated questions over agent trajectories (multimodal web-agent runs, not user-assistant chat), up to 500 trajectories / ~115M-token haystacks, web + enterprise domains. Five new abilities reframed for agentic memory: static-state recall, dynamic-state tracking, workflow knowledge, environment gotchas, premise awareness. Baselines: AgentRunbook-C (coding-agent) 72.5% vs AgentRunbook-R (RAG) 48.5% vs off-the-shelf 69.3% — "substantial room for improvement remains." This is memory-as-accumulated-experience and a longer-horizon target for donto-memory once V1 parity is reached. (V2 specifics are second-tier confidence — README/abstract, not a deep PDF read.)

6. donto-memory today — architecture and the refactor parity gap

This section is candid by design. donto-memory runs the old, triplicated extraction code, and its bitemporal store carries no time signal. Every claim below is cited file:line.

6.1 Architecture (memorize / recall / search)

donto-memory is a thin Rust (axum) consumer over the shared dontosrv substrate (:7879 → donto-pg :5432), plus a Python Temporal worker for heavy agentic extraction. Default consumer_iri = "ctx:memory".

Memorize (crates/donto-memory/src/api/routes/memorize.rs:467-693). Always stores raw text as one episodic chunk first, then produces semantic claims via one of three branches: supplied-facts (the live omega/Discord path), opencode-no-op, or in-process LLM extraction (a separate Rust 5-aperture extractor in donto-memory-core/src/extract.rs — distinct from the Python worker's 10 lenses; the engine is genuinely doubled even inside donto-memory). Each fact → assert_statement into ctx:memory/claims/session/<sid> at maturity 1 ("candidate"), plus a mem:claim/derived_from edge to the episodic chunk (semantic_claim.rs:85-99).

Recall — holder-scoped (hot_path.rs:18-166). Fan out to modules (episodic, semantic-claim, preference), RRF-fuse (k=60), filter, truncate. Critically, with a free-text query the retrieval is LEXICAL, not semantic: both episodic and semantic-claim bypass the substrate and run an ILIKE token-AND substring scan over the holder's owned statements, with hand-tiered weights (subject 1.0 / object_lit 0.8 / object_iri 0.6 / predicate 0.4, overlays.rs:434-440). No embeddings, no trigram similarity, no recency-decay ranking in the hot path.

/search — public, substrate-wide GIN tsvector FTS over a humanized projection of subject + object_iri + left(object_lit,120), bounded candidate cap 2000 then ts_rank, 9s timeout → clean partial:true (search.rs:79-238). This is the only route that reaches the genealogy/genes worlds; omega's recall tool merges /recall + /search.

6.2 The bitemporal gap — the most fatal finding (outside the five refactors)

• Tx-time is fully bitemporal in the substrate; as_of_tx (tx-time travel) is plumbed through RecallQuery → substrate.recall. But the ILIKE free-text recall path hardcodes upper(tx_time) IS NULL and ignores as_of_tx (overlays.rs:443,503) — so tx-time travel only works on the no-query fallback path.
• Valid-time — the dimension that maps to both TR and KU — is never populated. substrate.assert_statement accepts valid_from/valid_to (substrate.rs:153-193), but every memory ingest path passes None/None (semantic_claim.rs:71-82, episodic.rs:70-82, preference.rs:104-116). The only per-memory timestamp is mem:episodic/recorded_at = Utc::now() at ingest — i.e. tx-time of ingestion, not the message's event date. donto-memory today has no notion of "this fact was true on date X."
• Knowledge-update supersession exists in exactly one place — the preference module adds a supersedes argument edge on change (preference.rs:103-133). Episodic and semantic-claim accumulate (paraconsistent hold) — on-vision, but it means /recall returns both old and new with no recency/validity ordering, so KU answers depend entirely on the reader guessing the latest from two undated claims.

6.3 The parity gap vs this session's donto-extract refactors

The live memory-worker.service has DONTO_MEMORY_USE_EXTRACT=1 set (verified via systemctl show), so the production path already routes through a donto_extract bridge (activities.py:293 donto_extract_memorize_activity → ExtractionEngine(OpenCodeProvider, DEFAULT_LENSES) → SubstrateIngestor, with a real donto_document+revision registered first via _register_source). So source registration is at parity. But the bridge pins one lane (OpenCodeProvider, glm-5.1), runs the old single-pass-per-lens engine with substring-only anchoring, and has no failover. A grep confirms worker/ imports none of lanes / cite / glean / failover / _flex_find.

Net: donto-memory is ~70% there (live bridge ON + source registration present). The missing ~30% for a fair test is valid-time ingest + the citer + the lane-router — all of which exist as code today and need packaging + importing, not inventing. Two load-bearing scripts (cite_facts.py, run_codex_glean.py) currently live in volatile /tmp/cerebras-test/ and will vanish on reboot — they must be moved into the donto_extract package. And the engine remains triplicated (donto/apps/donto-api, donto-memory/worker, stale workspace/donto-api) — the rewrite's de-triplication is the clean end-state.

7. Why donto's model maps onto LongMemEval's hardest abilities

This is the thesis of the experiment, ability by ability. For each: how the updated donto-memory answers, the predicted strength/weakness, and the mechanism. Strengths conditional on the §8 ports are marked (gated).

7.1 Knowledge-Update (78 Q, always 2 sessions) — donto's home ground

Mechanism. The same attribute is stated twice; the answer is the latest. donto holds both (paraconsistent, invariant I3 — never delete) and, once valid-time is stamped, the later assertion supersedes by valid-time and tx-time. Strongest differentiator, because the judge explicitly rewards "previous info + updated answer … correct as long as the updated answer is present" (verified evaluate_qa.py:33). Naive vector-dedup picks one winner at write time — and the two near-identical update turns collide in embedding space, so a flat retriever often returns the stale one. (Gated): today every memory fact has valid_from=None and recall has no recency ordering, so without the valid-time port donto degrades to "reader guesses the latest from two undated claims" and loses the very advantage it claims. This is the cleanest substrate-beats-flat-RAG headline and the right place for a bitemporal ON-vs-OFF ablation (predicted 10–25 point gap, in the band MatrixOrigin measured).

7.2 Abstention (30 Q, _abs) — the evidence-first payoff

Mechanism. Every _abs question targets a sibling of a genuinely-mentioned entity. donto answers only from claims with a real donto_evidence_link to a span about the exact asked subject; the always-on citer's flag-unanchorable → abstain. Strength — the judge grades correct refusal (evaluate_qa.py:40-42), and a hallucinated specific answer fails hard. Paraconsistent + evidence-first means donto can see that it has a claim about a neighbor but none about the asked subject. The trap: lexical/FTS recall returns the high-similarity distractor (cameras≈films, tennis≈table-tennis); if the reader is not gated on subject-identity it fabricates. Honest tension: the same entity alignment that helps multi-session dedup can merge sibling entities and destroy the abstention signal — must be tuned and measured on both buckets simultaneously.

7.3 Temporal-Reasoning (133 Q) — native valid-time vs the paper's bolt-on

Mechanism. Order/diff events using dates that are mostly in message text ('3/22', 'last Saturday', 'two months ago' — ~30 of the first 40 temporal-reasoning instances carry an explicit in-text date/time expression in a has_answer evidence turn, measured here), resolved against question_date rather than against session timestamps. donto's bitemporal model separates valid-time (event date, in content) from tx-time (when said ≈ session date). (Gated) Strength — the paper's own time-aware fix is a bolt-on (GPT-4o infers a range and reorders, +6.8–11.3% TR recall); a bitemporal store does valid-time filtering natively with no M_T LLM call (cheaper). The judge forgives off-by-one days, lowering the bar. Weakness — extracting event dates from colloquial relative phrases is an extraction-quality problem the glean loop must nail; and as_of_tx is plumbed but the ILIKE recall path ignores it. Biggest upside, gated on the most plumbing.

7.4 Multi-Session (133 Q) — the hardest type, a genuine opportunity

Mechanism. Aggregate/count distinct items across 2–5 sessions with no single answer turn (clothing→3; projects-LED→2 excluding "participated"; model-kits→5). Requires complete recall of all evidence + dedup + count. donto's contexts/holder-scoping + query-time entity alignment join claims across sessions structurally rather than via flat top-k, and the paper found fact decomposition helps multi-session specifically (§5.2). Opportunity, not a guaranteed win: if alignment under-merges, donto over-counts; if recall is incomplete (the lexical gap again), it under-counts. Instrument with recall_all@k, not just QA.

7.5 Information-Extraction (156 Q: user/assistant/preference) — honestly neutral-to-weak

• single-session-user (70) — recall is the whole game; the substrate's bitemporal/paraconsistent edge is irrelevant. Weak: lexical recall misses facts the question paraphrases (the dense-retriever gap); 10-lens over-extraction dilutes top-k (§5.2: fact decomposition hurts detail QA).
• single-session-assistant (56) — structural strength: donto stores both roles by construction (episodic-first regardless of role), where user-only memory systems score ~0. Weak: the answer often sits in long structured assistant text; feed the episodic chunk (full text), not just the truncated claim, to the reader.
• single-session-preference (30) — strength: lenient rubric judging + a dedicated preference module that already supersedes. Weak: the generic question shares no tokens with the preference ("learn video editing" vs "Adobe Premiere Pro") — the same dense-retrieval gap.

8. The port plan — bringing donto-memory to parity

De-triplicate toward one donto_extract. Split into required (omit any and the test is rigged against donto's own thesis) vs nice-to-have.

8.1 Required for a fair test

R1 — Valid-time ingest (the most important port; not one of the five named refactors). Patch episodic.rs / semantic_claim.rs / the SubstrateIngestor so each claim is stamped with the session's haystack_date as valid_from, and lift in-content event dates into valid-time. Without this, knowledge-update and temporal-reasoning — donto's two headline strengths — have no bitemporal signal and donto degrades to undated flat-RAG. This is the prerequisite for the KU and TR ablations that are the report's spine, and the gap the five-refactor framing misses.

R2 — Always-on semantic citer (refactor c). Fold cite_facts.py into the donto_extract package (it currently sys.path-hacks donto-api helpers and lives in volatile /tmp). Wire it as a stage in donto_extract_memorize_activity before SubstrateIngestor.ingest: lexical _flex_find → bge-small semantic span → flag-unanchorable (hypothesis_only, never a bogus span). Validated 90.2% anchor on 3519 facts. Required because abstention + faithfulness are donto's evidence-first differentiator and the benchmark scores exactly this. (Needs fastembed/bge-small on the worker's python env.)

R3 — Lane router + --auto failover (refactor a). Replace the pinned OpenCodeProvider(glm-5.1) with the lane router (registry/caps/router/provenance/cli). Required because a 500-question online run will exhaust one GLM subscription mid-eval; without rotation to codex/cerebras the eval dies partway and the result is invalid. Pool-aware (codex 4 lanes share one ChatGPT-Pro bucket; cerebras 2 share a credit pool). This is failover-for-completion, not a quality nicety.

8.2 Nice-to-have

• N1 — Gleaning loop (refactor b). Fold run_codex_glean.py in. For LongMemEval the memory chunks are short (one session), so a 1–2 pass glean is plenty; donto-memory's existing loop-until-dry is an acceptable substitute for the fair test.
• N2 — Shared faceted prompt (refactor d). Point the engine at extract_broad.txt via OC_EXTRACT_BROAD_PROMPT_FILE to kill lens drift.
• N3 — pgvector dense-recall overlay (not a refactor — a separate recall workstream). donto-memory recall is lexical; the paper chose dense (Stella-1.5B) over BM25 for a reason. A DeriveEmbeddings job type already exists (types.rs:46); pgvector is installed. Strongly recommended to be competitive on the IE types — this is the highest-leverage non-extraction improvement (MemMachine: retrieval ≫ ingestion) — but the QA test is still fair with lexical recall, provided the gap is disclosed.

8.3 Order of operations

1. R1 valid-time ingest (Rust + ingestor) — unblocks the KU/TR ablations.
2. Move /tmp/cerebras-test/{cite_facts.py, run_codex_glean.py} into the donto_extract package (stop being volatile + sys.path-hacked).
3. R2 wire the citer as a post-extract stage.
4. R3 swap the pinned provider for the lane router + --auto.
5. Parity smoke vs the legacy path (worker/compare_extract.py already exists — read it first) to confirm fact yield ≥ legacy.
6. (nice) N1 glean depth, N2 shared prompt, N3 pgvector recall overlay.
7. De-triplicate — once donto_extract validates ≥ legacy yield, delete worker/extraction.py + worker/opencode_agent.py and the old SubstrateIngestor path, leaving one ingest = donto-api helpers.ingest_facts.

9. The concrete experiment design

Assumes the updated (ported) donto-memory. All paths absolute.

Stage 0 — variant choice

• Start on /home/ajax/longmemeval-study/data/longmemeval_oracle.json (500, evidence-only, local). Isolates reading+synthesis from retrieval; directly comparable to the paper's Oracle column (GPT-4o 0.870 / 0.924 with CoN+JSON). If donto trails GPT-4o oracle here, the problem is reader/extraction, not retrieval.
• Then longmemeval_s_cleaned.json (~115k tokens, the realistic online test — must be the 2025/09 cleaned release for comparability).
• Skip _m initially (~500 sessions, ~1.5M tokens); run it once S works as the flagship "memory beats long-context" result (donto stores, doesn't stuff context).

Stage 1 — online ingest (the critical timestamp injection)

Per-instance isolation: holder = user:lme:<question_id>, context = ctx:memory/lme/<question_id>. Replay haystack_sessions in haystack_dates order (oracle is not date-sorted — sort it yourself; _s/_m are pre-sorted). For each session i, POST /memorize with session_id = haystack_session_ids[i]. Critical timestamp injection (the whole point — without it KU/TR collapse): pass haystack_dates[i] as valid_from on every claim (the R1 port), and lift in-content event dates resolved against question_date into valid-time. Run the lane router with --auto + the always-on citer so every claim carries a real donto_span + donto_evidence_link (or is flagged hypothesis_only).

Stage 2 — recall (at t_q)

Hybrid at question_date (the recency cutoff): /recall (holder-scoped RRF over ctx:memory/lme/<qid>) + /search (substrate-wide FTS) + pgvector dense recall if N3 is built (lexical-only will trail the paper's dense retrievers). Feed both the matched claims and their episodic chunks (the assistant-table case needs full text). For temporal Q, filter/rank by valid-time natively — donto's structural answer to the paper's §5.4 reorder hack.

Stage 3 — reader (the paper's winning config)

Use the con reader (extract-then-reason) with CoN+JSON (paper Fig 6: up to +10 pts; GPT-4o 0.862→0.924). Freeze the reader = GPT-4o (paper-comparability) + optionally one strong modern reader (the ceiling); report both, never mix readers across systems. Abstention handling (load-bearing): instruct "answer ONLY from retrieved claims whose subject IS the asked entity; if no such anchored claim exists, say the information was not provided." Do not force an answer on _abs. Knowledge-update: "if multiple values for the same attribute, lead with the one with the latest valid-time." Emit {"question_id", "hypothesis"} per line.

Stage 4 — eval

export OPENAI_API_KEY
cd /home/ajax/longmemeval-study/src/evaluation
python3 evaluate_qa.py gpt-4o <hyp.jsonl> ../../data/longmemeval_oracle.json
python3 print_qa_metrics.py <hyp.jsonl>.eval-results-gpt-4o ../../data/longmemeval_oracle.json

The judge is hardcoded gpt-4o-2024-08-06; print_qa_metrics.py:20 asserts it (a local judge breaks comparability and the assertion).

Metrics to report (measurable, not marketing)

1. Per-type autoeval accuracy (6 types) + Task-averaged + Overall + Abstention, vs the paper's GPT-4o baselines (oracle 0.870/0.924; full-S 0.606/0.640) and vs Zep's per-type table (the right comparison class).
2. Turn-level + session-level Recall@k / NDCG@k via has_answer + answer_session_ids — recall_all@k is the headline for multi-evidence Q. (Retrieval is over ~470 — the 30 _abs and no-has_answer items are dropped.) Requires emitting a retrieval log in run_retrieval.py's corpus-id namespace (gold ids must contain answer) — extra plumbing; the clean first win is QA accuracy, retrieval recall is a bonus.
3. Evidence-anchor rate = % claims with a real donto_evidence_link (donto-native; the citer's 90.2% is the relevant figure) and faithfulness = % answers whose supporting claim is anchored.
4. The four ablations — the report's spine (they isolate the substrate advantage from reader/retrieval confounds):
   • (a) bitemporal valid-time ON vs OFF on knowledge-update — predicted 10–25 pt gap.
   • (b) valid-time recall ON vs OFF on temporal-reasoning.
   • (c) evidence-gated-abstain ON vs OFF on the 30 _abs.
   • (d) extraction density high vs low on the single-session detail types — tests the §5.2 over-extraction prediction.

Gotchas

Online memorization is mandatory (stream sessions; don't dump all at once, or you're testing long-context not memory). Verify a few KU instances actually have two valid-time-distinct assertions before scaling. A 500-Q online run will exhaust one GLM lane → the lane-router port is a prerequisite. _m ingest cost dominates — run oracle → S → M. Report donto only with a frozen reader; never compare a donto GPT-4o run to a GPT-5-mini leaderboard entry.

Effort estimate: oracle QA path (Stages 1–4, no retrieval log, no pgvector) ≈ 1–2 days once the port lands; + S variant + ablations ≈ 1 week; + retrieval-log plumbing + pgvector overlay + _m ≈ a second week. The port (§8) is the gating prerequisite, not part of this estimate.

10. Risks, limitations, and what a strong result would prove

10.1 Honest risks

• A — Retrieval/reader is donto's true bottleneck, not extraction. MemMachine's ablation (retrieval tuning ≫ ingestion) plus donto-memory's lexical recall predict donto will trail dense-retriever baselines on the 156 IE-type questions until an embedding overlay (N3) lands. This is a second workstream independent of the parity port — call it out, don't paper over it.
• B — Reader-model confound caps comparability. Post-2024 "SOTA" (OMEGA 95.4/GPT-4.1, Mastra 94.87/GPT-5-mini) is mostly reader progress (identical evidence: 58–90% across readers). donto's class is Zep/Graphiti (71.2/GPT-4o), not the GPT-5-mini systems. Frame accordingly.
• C — Over-extraction may hurt detail-QA precision. 10-lens extraction emits hundreds of claims/session; §5.2 found fact decomposition helps only multi-session and hurts single-session detail QA. Without query-time alignment collapsing the noise, the dense claim graph dilutes top-k.
• D — Alignment tension. The entity alignment that helps multi-session dedup can hurt abstention if it merges sibling entities. Tune and measure on both buckets at once.
• E — Subscription exhaustion. A 500-Q online run exhausts one lane mid-eval — the lane-router port is a prerequisite, not a nicety.

10.2 The predicted profile (restated honestly)

donto should post a distinctive shape: above-class on knowledge-update, abstention, and (if ported) temporal-reasoning — proven by clean ablations, not by a headline overall number — and at-or-slightly-below class on the three single-session IE types and overall, bottlenecked by lexical recall. The right story is not "donto wins overall." It is "donto wins on the abilities its substrate is built for, measurably, via clean ablations, with an honest recall-layer gap disclosed."

10.3 What a strong result would prove

A successful run would demonstrate the substrate thesis empirically, not rhetorically:

1. Bitemporal supersede beats flat-RAG on knowledge-update — the bitemporal ON/OFF ablation showing a 10–25 pt gap would be the cleanest published evidence that holding both and re-ranking by validity outperforms dedup-and-pick-a-winner on exactly the task whose judge rubric was written for it.
2. Evidence-first refusal beats hallucination on abstention — abstention accuracy plus a donto-native faithfulness metric (% anchored) the paper cannot report, proving that anchoring to the exact subject or abstaining is a real capability, not a slogan.
3. Native valid-time matches the paper's bolt-on temporal fix at lower cost — TR parity with no M_T LLM call would validate that bitemporality is the right primitive, not a feature to be reconstructed per-query.
4. The substrate's differentiators are separable and measurable — four ablations isolating bitemporal, evidence-gating, alignment, and extraction-density turn donto's vision claims into numbers, which is the entire point: be measurable, not marketing.

And it would prove the meta-point of the whole rewrite: that de-triplicating into one donto_extract and wiring the citer + lanes + valid-time is not housekeeping — it is the difference between a memory system that claims bitemporal/paraconsistent/evidence-first superpowers and one that can demonstrate them on the field's reference benchmark.

Sources

• arXiv 2410.10813 v2 (LongMemEval, ICLR 2025) — Tables 1–4, Figs 1–6, all numbers read from the PDF.
• Project site xiaowu0162.github.io/long-mem-eval; HF xiaowu0162/longmemeval-cleaned; LongMemEval-V2 github.com/xiaowu0162/LongMemEval-V2 + arXiv 2605.12493.
• Local repo /home/ajax/longmemeval-study/: README.md; data/longmemeval_oracle.json (500 instances — type counts, abstention distribution, sessions/instance, real examples all verified empirically); src/evaluation/{evaluate_qa.py, print_qa_metrics.py, print_retrieval_metrics.py}; src/retrieval/{run_retrieval.py, eval_utils.py, index_expansion_utils.py}; src/index_expansion/*; src/generation/run_generation.py + .sh.
• donto: /mnt/donto-data/workspace/donto-memory/{crates/donto-memory*, worker/} (memorize.rs, hot_path.rs, overlays.rs:412-528, semantic_claim.rs, episodic.rs, preference.rs, substrate.rs:153-193, activities.py, workflows.py, extraction.py); /mnt/donto-data/workspace/donto-extract/src/donto_extract/{engine.py, lenses.py, lanes/*}; /tmp/cerebras-test/{cite_facts.py, run_codex_glean.py}; systemctl show memory-worker.service (live env).
• Landscape: arXiv 2501.13956 (Zep), arXiv 2604.04853 (MemMachine), omegamax.co, mastra.ai, mem0.ai, blog.getzep.com, MatrixOrigin (reader-separation).

All quoted donto behaviour is by code inspection, not a live trace; all donto-vs-LongMemEval performance is a hypothesis — no run has been executed.