# Testing donto Against LongMemEval — A Long-Term Memory Benchmark Study

*A research report on the LongMemEval benchmark (ICLR 2025), its evaluation mechanics, the 2025–2026 leaderboard landscape, and a candid audit + port plan + experiment design for testing **donto-memory** — the agentic-memory example consumer of the donto substrate — against it.*

**Date:** 2026-06-04 · **Status:** study, parity audit, and experiment plan — **no donto run against LongMemEval has been executed yet.** Every "donto should win on X" statement below is a *hypothesis* derived from the substrate's documented properties, with the counter-hypotheses flagged.

> **What was independently verified for this report (vs. taken on trust).** Everything in §2–§3 (data counts, abstention distribution, sessions-per-instance, the verbatim examples, all judge prompts, the metric/assert logic, the retrieval-eval rules) was re-derived **empirically** from the local `data/longmemeval_oracle.json` (500 instances) and `src/evaluation/*` + `src/retrieval/*` — `file:line` cited inline. Every donto-memory claim in §6 (the `valid_from=None` finding, the ILIKE recall weights, the live `DONTO_MEMORY_USE_EXTRACT=1` bridge, the citer's 90.2%-on-3519 figure, the glean fact-counts) was re-checked against live source/output `file:line`. **Not re-derived in this pass:** the §4 numbers from the paper PDF (tables/figures — no PDF is on disk; cited from the paper, figure-bar reads flagged ±0.01) and the §5 third-party leaderboard numbers (from vendor blogs/arXiv abstracts, explicitly caveated as not apples-to-apples). Treat §2/§3/§6 as **measured**, §4/§5 as **cited**.

---

## 1. Executive summary

**LongMemEval** (Wu et al., arXiv 2410.10813, ICLR 2025) is *the* reference benchmark for long-term conversational memory. Its single most-cited result: long-context LLMs reading a full ~115k-token chat history lose **~30% accuracy** versus reading only the evidence ("oracle") sessions — GPT-4o falls **0.870 → 0.606** — and weaker models collapse 50–66%. That gap is the entire reason dedicated memory systems exist. The benchmark probes **five abilities** — Information Extraction, Multi-Session Reasoning, Knowledge Updates, Temporal Reasoning, and Abstention — across **500 questions** in **six question types** plus a 30-question abstention overlay (all counts verified in the local data file).

donto's substrate is, on paper, unusually well-shaped for the *hardest* of those abilities. Its four differentiators map onto the three abilities that resist flat retrieval-augmented memory:

| donto differentiator | LongMemEval ability it serves | why |
|---|---|---|
| **Bitemporal** (valid-time + tx-time) | Knowledge-Update, Temporal-Reasoning | the judge rubric *literally rewards* surfacing the updated value; temporal needs an event-date axis |
| **Paraconsistent** (hold-both, never overwrite — invariant I3) | Knowledge-Update, Abstention | keep both 5K times, re-rank the later one — no destructive dedup picking a stale winner |
| **Evidence-first** (span + `donto_evidence_link`) | Abstention, faithfulness | refuse when no claim is anchored to the *exact* asked subject |
| **Contexts / holder-scoping + query-time alignment** | Multi-Session, Information-Extraction | structured cross-session joins + entity dedup for distinct-counting |

**But there is a critical caveat that gates the whole experiment**, and it is the reason this report exists. donto-memory currently runs the **old, triplicated extraction stack** and never received this session's refactors to the genealogy/`donto-extract` engine. Worse, the audit surfaced a gap *outside* the five named refactors that is even more fatal: **every memory fact today is stored with `valid_from`/`valid_to` = `None`** (verified — no `valid_from` is passed anywhere in `crates/donto-memory-core/src/modules/`). So the bitemporal store that powers donto's two headline strengths (knowledge-update and temporal) has **no bitemporal signal at all**. A fair test requires three *required* ports — **valid-time ingest**, the **always-on semantic citer**, and the **lane-router with failover** — before donto-memory can credibly contest the abilities it claims.

The honest predicted profile is **not** "donto wins overall." It is a *distinctive shape*: **above-class on knowledge-update, abstention, and (if ported) temporal**, via clean ablations; **at-or-below class on the three single-session information-extraction types**, bottlenecked by donto-memory's recall being **lexical (ILIKE + GIN FTS), not dense** — the paper deliberately chose a dense retriever (Stella-1.5B) over BM25 because incidental, paraphrased facts have low lexical overlap with the question. The report's spine is therefore four **ablations** that isolate the *substrate* advantage from the reader-model and retrieval confounds that make the public leaderboard largely incomparable.

---

## 2. What LongMemEval is

### 2.1 Thesis and motivation

An interactive assistant's value compounds with its memory across sustained interaction. The paper's abstract reports a **30% accuracy drop** when commercial chat assistants and long-context LLMs must memorize information across sustained interactions rather than read it fresh. Authors (per the repo `README.md` + the arXiv citation block): **Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, Dong Yu** — UCLA / Tencent AI Lab Seattle / UC San Diego; the work was done during Di Wu's Tencent internship. arXiv **2410.10813**; ICLR 2025; the local repo's cached v2 PDF link is dated early 2025.

The gap it fills: prior long-term-memory benchmarks (MemoryBank, PerLTQA, DialSim, and even the recent LoCoMo) have short, non-configurable histories (a few thousand tokens) and cover memory abilities thinly. Per the paper's Table 1, **LongMemEval is the only benchmark covering all five abilities** — crucially it is the only one to evaluate *recall of information provided **by the assistant*** and *reasoning with updated user information* (knowledge updates), which every prior benchmark omits.

**Problem formulation (§3.1).** An instance is a 4-tuple `(S, q, t_q, a)`. `S` is an ordered sequence of `N` timestamped sessions `[(t1,S1)...(tN,SN)]`, each a multi-turn user↔assistant dialogue decomposable into *rounds* (one user message + one assistant reply). At test time `S` is **streamed one session at a time** — the "online" memorization protocol — and the question `q` (with date `t_q > t_N`) is asked **after all sessions**. `a` is a short phrase or a rubric. The framing is a harder needle-in-a-haystack: the needle must be **retrieved *and* synthesized** from multiple evidence sessions buried among filler.

### 2.2 The five core abilities (§3.2)

1. **Information Extraction (IE)** — recall a specific fact from extensive history, *"including the details mentioned by either the user OR the assistant."* Assistant-side recall is novel versus prior benchmarks.
2. **Multi-Session Reasoning (MR)** — *"synthesize information across multiple history sessions … that involve aggregation and comparison."* (e.g. *"How many items of clothing do I need to pick up?"* → **3**, summed across three sessions.)
3. **Knowledge Updates (KU)** — *"recognize the changes in the user's personal information and update the knowledge of the user dynamically over time."* (e.g. a 5K personal-best stated as 27:12, later updated to 25:50 — return the **current** value.)
4. **Temporal Reasoning (TR)** — *"awareness of the temporal aspects of user information, including both explicit time mentions AND timestamp metadata."* Must reason over session timestamps *and* in-text dates.
5. **Abstention (ABS)** — *"identify questions seeking unknown information … and answer 'I don't know'."* Built by taking the other types and modifying them into **false-premise** questions.

### 2.3 The six question types + the abstention overlay (verified in the local data)

Counts read directly from `/home/ajax/longmemeval-study/data/longmemeval_oracle.json` (500 instances):

| question_type | count | mapped ability |
|---|---:|---|
| temporal-reasoning | 133 | TR |
| multi-session | 133 | MR |
| knowledge-update | 78 | KU |
| single-session-user | 70 | IE (user-stated) |
| single-session-assistant | 56 | IE (assistant-stated — novel) |
| single-session-preference | 30 | IE (implicit personalization) |
| **total** | **500** | |

**Abstention is an overlay, not a seventh disjoint bucket.** The 30 abstention instances are those whose `question_id` ends in `_abs`, drawn *from* the other types. Verified distribution: **multi-session 12, temporal-reasoning 6, knowledge-update 6, single-session-user 6** — and **none** in single-session-assistant or single-session-preference. (The README lists exactly the six `question_type` values above and defines abstention as the `_abs`-suffix rule, `README.md:81` — i.e. structurally a cross-cutting flag, not a seventh disjoint type. The paper presents abstention as one of the five *abilities*, which is why an `_abs` item is folded into its parent type **and** reported separately — see §3.2.)

Verified data shape (oracle, evidence-only): sessions per instance **min 1 / median 2 / max 6**; knowledge-update is **always exactly 2** sessions; the three single-session types are always 1; mean ~1.9 evidence sessions per question. `single-session-preference` answers are **rubrics** (free-form preference descriptions), not short facts.

### 2.4 Real examples (verbatim from the oracle file)

- **single-session-user** — Q *"What degree did I graduate with?"* → **"Business Administration"** (stated in a single turn buried inside an off-topic expense-paperwork conversation).
- **single-session-assistant** — Q *"remind me what was the rotation for Admon on a Sunday?"* → **"8 am – 4 pm (Day Shift)"** — the answer lives inside a 14-turn **assistant-generated shift-rotation table**. (Also: a children's story where the Plesiosaur is **"blue"**; restaurant **"Miss Bee Providore"** — all in assistant turns.)
- **single-session-preference** — Q *"recommend resources to learn video editing?"* → rubric *"would prefer resources tailored to **Adobe Premiere Pro** … not general/other software"* (evidence: user earlier said *"I enjoy to use Adobe Premiere Pro"*).
- **knowledge-update** — Q *"personal best time in the charity 5K?"* — session1 (05/25) **"27:12"**, session2 (05/27) **"25:50"** → **"25:50"**. (Also $350,000 → $400,000 mortgage pre-approval; "three" → "four" Korean restaurants; Chicago → "the suburbs".)
- **temporal-reasoning** — Q *"first issue with my car after its first service?"* → **"GPS system"** (service was 3/15, GPS issue 3/22 — must order *event* dates lifted from text, not session timestamps).
- **multi-session** — Q *"How many items of clothing to pick up/return?"* → **"3"** (a blazer at the cleaners, boots returned to Zara, a new pair to pick up — across three sessions).
- **abstention** — Q *"name of my hamster?"* → **"You did not mention a hamster — you mentioned your cat Luna."** Every `_abs` question targets a **plausible sibling** of a genuinely-mentioned entity (Italian vs Korean; Dr. Johnson vs Dr. Smith; table-tennis vs tennis; Shinjuku vs Harajuku; vintage films vs vintage cameras). The trap is a high-similarity distractor; correct behaviour is to **refuse and name the gap**.

### 2.5 The attribute-controlled construction pipeline (§3.2, Fig 2)

Human-expert-built questions; LLM-simulated, human-edited evidence sessions; Llama-3-70B-Instruct as the pipeline LLM unless noted. ~400 human hours of construction + ~150 hours studying commercial systems; 3 expert NLP-researcher annotators.

1. **Ontology** — 164 user attributes in 5 categories (lifestyle, belongings, life events, situations/context, demographic). For each, an LLM writes an attribute-focused background paragraph.
2. **Question construction** — sample a paragraph → LLM proposes seed (Q, A) pairs → human experts **filter + rewrite for difficulty** → manually decompose the answer into one or more **evidence statements** with optional timestamps.
3. **Evidence-session construction** — each evidence statement is embedded **indirectly** into a task-oriented self-chat session (User-LLM ↔ Assistant-LLM). Instead of *"I bought a new car last month,"* the user asks for car-insurance help and reveals it incidentally — forcing systems to memorize details that were never emphasized. All sessions are manually screened to verify inclusion, distribute evidence across positions, and rephrase to colloquial language (especially time mentions). The evidence-statement position is hand-annotated — this is the `has_answer:true` turn flag.
4. **History compilation (Fig 2c)** — for each question, sample unrelated sessions, randomly insert the evidence sessions, and assign plausible timestamps to *all*. Filler comes from (a) self-chat sessions simulated from *other, non-conflicting* attributes, and (b) publicly released **ShareGPT** (Apache-2.0) + **UltraChat** (MIT).

A **September 2025 "cleaned" release** (`xiaowu0162/longmemeval-cleaned`) further scrubbed haystacks to remove interference with answer correctness — this is the version downloaded locally and the version any comparable run must use.

### 2.6 The three variants (same 500 questions, escalating dilution)

| variant | scale | what it isolates |
|---|---|---|
| **_oracle** | evidence sessions only (~1.9 sessions/instance) | reading + synthesis, *perfect-retrieval upper bound* — **this is the file on disk** |
| **_s** | ~115k tokens, ~40–80 sessions (fits a 128k context) | the realistic online-memory test |
| **_m** | ~500 sessions, ~1.5M tokens | stress test — too long for any context window, forces a real memory store |

`question_date` spans 2021/08/20 → 2024/02/20. The difficulty in `_s`/`_m` comes from **haystack dilution**, not from the oracle instances themselves (which are small, 2–3 sessions, 12–36 turns).

---

## 3. How LongMemEval is evaluated

The harness has **three separable stages**, and an external system like donto-memory needs to satisfy only **one** of them to get a headline number.

### 3.1 The plug-in contract — the *only* required output

A **hypotheses JSONL**: one JSON object per line, exactly two fields `{"question_id": ..., "hypothesis": ...}` (`README.md:90-99`; consumed at `evaluate_qa.py:76-99`). `hypothesis` is your system's free-text answer to `entry['question']`. **You generate it however you like** — donto-memory's own `/memorize` + `/recall` + `/search` pipeline is a fully legal black box. You do **not** have to use `src/retrieval` or `src/generation`; those are the paper's *reference* memory systems, not a required path. Minimal integration: for each of the 500 instances, feed the timestamped history online (session by session, in haystack order), ask `entry['question']` with `entry['question_date']` as "now", write `{question_id, hypothesis}`.

### 3.2 QA accuracy = LLM-as-judge (the headline metric)

`src/evaluation/evaluate_qa.py` is the scorer:

```
python3 evaluate_qa.py gpt-4o <hyp.jsonl> ../../data/longmemeval_oracle.json
```

The judge is **hardcoded** to `gpt-4o-2024-08-06` via the OpenAI API (`evaluate_qa.py:11-15`), `temperature=0`, `max_tokens=10`, and the label is simply `'yes' in eval_response.lower()` (`evaluate_qa.py:108-113`). Meta-eval: the judge agrees with human experts **>97%**.

**The judge prompt differs by `question_type`** — this is load-bearing for how donto must phrase answers (all verified verbatim from `get_anscheck_prompt`, `evaluate_qa.py:24-43`):

- **single-session-user / -assistant / multi-session** — *"answer yes if the response contains the correct answer … If the response only contains a **subset** of the information required, answer no."* (Multi-session counting: a subset → no.)
- **temporal-reasoning** — same, **plus** *"do not penalize **off-by-one** errors for the number of days/weeks/months."*
- **knowledge-update** — *"If the response contains some previous information along with an updated answer, the response should be considered as correct **as long as the updated answer is the required answer**."* → This rubric is **written for a supersede-not-overwrite store.** You may surface history, but you must lead with the latest value.
- **single-session-preference** — judged against a **rubric**, not an exact string: *"correct as long as it recalls and utilizes the user's personal information correctly … does not need to reflect all the points."*
- **abstention** (`question_id` ends `_abs`, detected at `evaluate_qa.py:101`) — a different prompt: *"answer yes if the model correctly identifies the question as **unanswerable** … information is incomplete, or some other information is given but the asked information is not."* A hallucinated specific answer **fails hard**.

**Aggregation.** `evaluate_qa.py` prints overall + per-type accuracy inline. `print_qa_metrics.py` (which **asserts** the judge was `gpt-4o-2024-08-06` at line 20 — a hard comparability guard) reports three numbers: **Task-averaged Accuracy** (unweighted mean over the 6 type-means), **Overall Accuracy** (micro-average over all 500), and **Abstention Accuracy** (mean over just the `_abs` items). Note the asymmetry: an `_abs` item is folded into its parent type *and* reported separately, so abstention quality double-counts. There is **no precision/recall/F1** for QA — it is pure binary-judge accuracy.

### 3.3 Memory-recall (retrieval) metrics — optional

Only applies if you produce a retrieval log in the harness's format (`src/retrieval/run_retrieval.py` + `eval_utils.py`). Granularity is **turn** or **session**, built from **user-side text only**. Gold docs are identified by the literal substring **`answer`** in the corpus_id; non-evidence items are renamed `noans_...`; a turn keeps `answer` only if `has_answer==True` (`run_retrieval.py:209,216-225,272`). Metrics: `recall_any@k`, **`recall_all@k`** (the headline — *all* gold docs in top-k, because a question can need multiple evidence turns), and `ndcg_any@k` for `k ∈ {1,3,5,10,30,50}`; turn results roll up to session level (`eval_utils.py:24-46`).

**Crucial scoring rule:** retrieval eval **always drops the 30 abstention instances** (no ground-truth location, `print_retrieval_metrics.py:12`, `README.md:206`) and any instance with no user-side `has_answer` turn — so **retrieval is scored over ~470, QA over all 500.**

### 3.4 The plug-in path for donto, in five concrete steps

1. Pick a variant — `longmemeval_oracle.json` (downloaded; easiest) → `longmemeval_s_cleaned.json` (realistic) → `longmemeval_m_cleaned.json` (stress).
2. For each instance: **online-memorize** each session in haystack order, stamping each session with its `haystack_dates[i]`, into a per-instance context/holder (`ctx:memory/lme/<question_id>`) so instances don't cross-contaminate.
3. After all sessions: query `/recall` (holder-scoped) + `/search` (substrate-wide), pass `question_date` as the current date, produce **one** answer string. For `_abs` items the system **must be allowed** to say *"I don't have enough information."*
4. Emit `{"question_id", "hypothesis"}` per line.
5. Score with `evaluate_qa.py gpt-4o`, then `print_qa_metrics.py` for the per-type + task-averaged + abstention breakdown.

> **Note:** an eval-only install (`requirements-lite.txt`: openai, tqdm, backoff, numpy, nltk) is enough to score a hypotheses file. The full retrieval/generation stack (torch, vllm, sentence-transformers) is only needed to run the paper's reference systems — moot for donto-as-black-box.

---

## 4. The memory-design taxonomy and the paper's findings (with numbers)

All numbers below are **cited from the paper's tables/figures**, not re-derived in this pass (no PDF is on disk; the local repo ships only the data + code). Figure-bar reads are flagged ±0.01 and should be treated as approximate. The *direction* of every finding (long-context drops vs oracle; round-granularity wins; key-expansion and time-aware expansion help; CoN+JSON helps) is robust and corroborated by the §5 landscape; the exact decimals are the paper's.

### 4.1 The benchmark is hard — the "30% drop" (§3.4, Fig 3)

**Commercial systems** (Fig 3a; 97 questions, short 3–6-session histories ~10× shorter than `_s`):

| system (LLM) | accuracy |
|---|---:|
| Offline Reading (GPT-4o reads full history) | **0.9184** |
| ChatGPT memory (GPT-4o) | 0.5773 |
| ChatGPT memory (GPT-4o-mini) | 0.7113 |
| Coze memory (GPT-4o) | 0.3299 |
| Coze memory (GPT-3.5-turbo) | 0.2474 |

ChatGPT and Coze (GPT-4o) drop **37%** and **64%** versus offline reading with the *same* LLM. *"ChatGPT tended to overwrite crucial information … Coze often failed to record indirectly provided user information."*

**Long-context LLMs** (Fig 3b; Oracle vs full `_s`) — the famous drop, per-model:

| model | Oracle | Full _s | drop |
|---|---:|---:|---:|
| GPT-4o | 0.870 | 0.606 | **30.3%** |
| Llama-3.1-70B | 0.744 | 0.334 | 55.1% |
| Llama-3.1-8B | 0.710 | 0.454 | 36.1% |
| Phi-3-128k-14B | 0.702 | 0.454 | 35.9% |
| Phi-3.5-mini-4B | 0.660 | 0.342 | 48.1% |

Chain-of-Note (CoN) does **not** close it (GPT-4o 0.924 → 0.640). Even the *best* long-context model loses ~30 points just from history growing to ~50 sessions; weaker models collapse 50–66%. This is the core evidence that *"even the most capable long-context LLMs struggle to manage an ever-growing interaction history without an effective memory mechanism."*

### 4.2 The unified memory-design framework (§4) — 3 stages, 4 control points

**Stage 1 Indexing · Stage 2 Retrieval · Stage 3 Reading.** Control points: **CP1 Value** (granularity of the stored item), **CP2 Key** (what you index on), **CP3 Query** (how you search), **CP4 Reading** strategy. The authors' own best design = value:**round**, key:**V+fact**, query:**question+time**, retrieval:flat, time-aware:Yes, reading:**CoN**.

**CP1 Value — granularity (§5.2, Fig 5).** A **round** (one user turn + reply) is the optimal storage granularity — better than session and better than compressed-fact. Compressing into summaries/individual facts **hurts** overall QA via information loss — **except for multi-session reasoning**, where fact decomposition consistently **helps** (uniform, simplified format). Reader-capacity-dependent budget: Llama-3.1-8B degrades past ~3k retrieved tokens; GPT-4o keeps improving past 20k.

**CP2 Key — index expansion (§5.3, Table 3).** Condensed forms alone (summary/keyphrase/fact) do **not** beat the value itself. But **expanding** the key to *original value + extracted user facts* improves both retrieval and QA: **+9.4% recall@k, +5.4% final accuracy** averaged. (Value=Session: K=V recall@5 0.706 → K=V+fact 0.732; end-to-end GPT-4o 0.670 → 0.714.)

**CP3 Query — time-aware expansion (§5.4, Table 4) — the TR fix.** Index values *with* their dated events; at retrieval, an LLM `M_T` extracts a time range from the query and filters out-of-range items. **+11.3% TR recall (rounds), +6.8% (sessions).** Caveat: depends on a *strong* LLM — Llama-8B *"struggles to generate accurate time ranges, often hallucinating or missing temporal cues."*

**CP4 Reading — Chain-of-Note + JSON (§5.5, Fig 6) — up to +10 points.** Even with **perfect (oracle)** retrieval, a bad reading strategy costs up to 10 absolute points. Two fixes: (1) present retrieved items as structured **JSON**; (2) **Chain-of-Note** — instruct the LLM to first extract relevant info from each item, *then* reason. Oracle QA (figure-read, ±0.01): GPT-4o NL+Direct **0.862** → JSON+CoN **0.924**; Llama-3.1-70B 0.762 → 0.848; Llama-3.1-8B 0.710 → 0.756. CoN+JSON wins by a large margin; JSON alone does *not* consistently beat NL without CoN.

### 4.3 The five design recommendations

1. Session **decomposition** into rounds for value granularity.
2. Fact-augmented **key expansion** (V+fact) for indexing.
3. **Time-aware** indexing + query expansion for TR.
4. **Chain-of-Note + JSON** for reading.
5. Overall: **retrieval-augmented memory beats raw long-context** as history grows.

### 4.4 The paper's index-expansion machinery (where donto's extraction maps)

`src/index_expansion/*` generates offline LLM-derived text attached to each session/turn: `session-summ` (paragraph summary), `session/turn-keyphrase` (semicolon list), `session/turn-userfact` (a JSON list of coreference-resolved *"personal facts about the user"*), and a session **temporal-event** extractor producing `{date, event}` pairs. Join modes: `separate` / `merge` / `replace`. The **`userfact`** expander is essentially the paper's version of donto-memory's fact extraction — but **untyped and not evidence-anchored**, which is precisely where donto's typed, span-anchored claims are meant to win on faithfulness. The **time-aware query pruning** (`temp_query_search_pruning.py`) has GPT-4o infer a `[start,end]` date range, pad ±2 days, and **reorder** retrieved items — a bolt-on hack for exactly what a **bitemporal** store does natively.

### 4.5 Reading methods in the reference generator

`run_generation.py` offers three readers (`run_generation.sh:62-71`): **direct** (`--cot false`); **con** (`--cot true`, the author-**recommended** single chain-of-thought "extract all relevant info, then reason"); and **con-separate** (`--cot true --con true`, a separate per-chunk extraction LLM call first — the strongest config). History is always sorted by date, each block labeled with its session date, and the prompt states the current date = `question_date`.

### 4.6 Stated limitations / ethics

No dedicated limitations section. Acknowledged: privacy leakage from storing/recalling user info; the **lack of a memory deletion operator** harms trustworthiness; datastore poisoning could cause jailbreaks (recommend moderation on read/write). English-only synthetic-ish filler. **All commercial evals were run in the first two weeks of August 2024**, so those numbers are a moving target. The LLM judge has ~3% residual disagreement with humans.

---

## 5. Landscape and SOTA (2025–2026)

### 5.1 Rough leaderboard (all caveated — see §5.2)

Format: System · overall · reader model · method.

| system | overall | reader | method / note |
|---|---:|---|---|
| OMEGA | **95.4%** | GPT-4.1 | hybrid BM25+vector+rerank; reports **task-averaged** accuracy |
| Mastra Observational Memory | 94.87% | GPT-5-mini | **no retrieval** — compressed dated observation log fully in-context; with GPT-4o reader 84.23% |
| Mem0 "token-efficient" | 94.4% | — | ~6,787 tokens/query; **same vendor's earlier independent test = 49.0%** — treat skeptically |
| MemMachine | 93.0% | GPT-5-mini | k=100 retrieval; arXiv 2604.04853 |
| ByteRover | 92.8% | — | 1.6s latency |
| Emergence AI | ~86% | — | per OMEGA leaderboard |
| Memobase | 75.8% | — | per ByteRover eval |
| **Zep / Graphiti** | **71.2%** | GPT-4o | temporal KG, **~1,600 tokens** (vs 115k), ~2.6s vs ~29s; arXiv 2501.13956 |

**Zep is the best-documented graph/temporal comparison and donto's natural comparison class.** Its published per-category numbers (GPT-4o): single-user 92.9, assistant 80.4, **preference 56.7, multi-session 57.9**, KU 83.3, temporal 62.4 — note the two soft spots (preference, multi-session) are exactly the recall-bound types. **Letta/MemGPT has *not* published a LongMemEval number** (only LoCoMo 74%); its sleep-time compute (background block consolidation) is the donto-relevant analogue.

### 5.2 The central comparability finding — leaderboard numbers are *not* apples-to-apples

Three confounds make the cross-system ranking largely meaningless:

1. **Reader model.** MatrixOrigin froze one Memoria retrieval snapshot and swapped *only the reader*: on Knowledge-Update, **identical evidence scored 58.4% to 89.6%** — a 31-point spread purely from the reader. Much of the "SOTA jump" since 2024 is **reader-model progress, not memory progress** (the paper's oracle GPT-4o = 87%, while the GPT-5-mini systems hit 93%+).
2. **Metric.** Task-averaged accuracy (OMEGA) inflates versus raw question-level accuracy by equal-weighting the 30-question preference bucket with the 133-question multi-session bucket. Some quoted "scores" are retrieval recall@k, not QA accuracy at all.
3. **Dataset version.** The 2025/09 "cleaned" release shifted numbers; pre/post-cleaning scores are not comparable.

### 5.3 Where the field stands

- **Universally hardest ability: multi-session reasoning.** Every top system plateaus ~87% (MemMachine 87.2, Mastra 87.2, OMEGA ~83). Synthesis across scattered sessions resists both retrieval and long-context.
- **Temporal + knowledge-update are now largely "solved"** by systems with **explicit temporal anchoring** (Mastra's "three dates", Zep's bitemporal edges) + a strong reader — which is precisely the capability donto claims natively.
- **RAG beats long-context.** The full 115k path actively *loses* ~30% (lost-in-the-middle). The only non-retrieval winner (Mastra) keeps a **compressed dated log** in context, not raw history.
- **The single highest-leverage knob is retrieval, not extraction.** MemMachine's ablation: retrieval-stage tuning (depth k20→30 = **+4.2%**, JSON formatting +2.0%, search-prompt +1.8%) **dwarfs** ingestion-stage tuning (sentence chunking +0.8%). **This is the most important external finding for donto: its retrieval quality will dominate the score more than its 10-lens extraction richness.**

### 5.4 LongMemEval-V2 (arXiv 2605.12493, 2026) — where the field is going

Not a rescale — a different benchmark. **451 manually-curated questions over agent *trajectories*** (multimodal web-agent runs, not user-assistant chat), up to 500 trajectories / ~115M-token haystacks, web + enterprise domains. Five **new** abilities reframed for agentic memory: **static-state recall, dynamic-state tracking, workflow knowledge, environment gotchas, premise awareness.** Baselines: AgentRunbook-C (coding-agent) **72.5%** vs AgentRunbook-R (RAG) **48.5%** vs off-the-shelf **69.3%** — *"substantial room for improvement remains."* This is memory-as-accumulated-experience and a longer-horizon target for donto-memory once V1 parity is reached. (V2 specifics are second-tier confidence — README/abstract, not a deep PDF read.)

---

## 6. donto-memory today — architecture and the refactor parity gap

This section is candid by design. **donto-memory runs the old, triplicated extraction code, and its bitemporal store carries no time signal.** Every claim below is cited `file:line`.

### 6.1 Architecture (memorize / recall / search)

donto-memory is a thin Rust (axum) consumer over the shared `dontosrv` substrate (`:7879` → donto-pg `:5432`), plus a Python Temporal worker for heavy agentic extraction. Default `consumer_iri = "ctx:memory"`.

**Memorize** (`crates/donto-memory/src/api/routes/memorize.rs:467-693`). Always stores raw text as **one episodic chunk first**, then produces semantic claims via one of three branches: supplied-facts (the live omega/Discord path), opencode-no-op, or in-process LLM extraction (a *separate* Rust 5-aperture extractor in `donto-memory-core/src/extract.rs` — distinct from the Python worker's 10 lenses; the engine is genuinely doubled even inside donto-memory). Each fact → `assert_statement` into `ctx:memory/claims/session/<sid>` at maturity 1 ("candidate"), plus a `mem:claim/derived_from` edge to the episodic chunk (`semantic_claim.rs:85-99`).

**Recall** — holder-scoped (`hot_path.rs:18-166`). Fan out to modules (episodic, semantic-claim, preference), **RRF-fuse** (k=60), filter, truncate. **Critically, with a free-text query the retrieval is LEXICAL, not semantic:** both episodic and semantic-claim bypass the substrate and run an **ILIKE token-AND substring scan over the holder's owned statements**, with hand-tiered weights (subject 1.0 / object_lit 0.8 / object_iri 0.6 / predicate 0.4, `overlays.rs:434-440`). **No embeddings, no trigram similarity, no recency-decay ranking** in the hot path.

**/search** — public, substrate-wide GIN `tsvector` FTS over a humanized projection of `subject + object_iri + left(object_lit,120)`, bounded candidate cap 2000 then `ts_rank`, 9s timeout → clean `partial:true` (`search.rs:79-238`). This is the only route that reaches the genealogy/genes worlds; omega's recall tool merges `/recall` + `/search`.

### 6.2 The bitemporal gap — the most fatal finding (outside the five refactors)

- **Tx-time** is fully bitemporal in the substrate; `as_of_tx` (tx-time travel) is plumbed through `RecallQuery` → `substrate.recall`. **But the ILIKE free-text recall path hardcodes `upper(tx_time) IS NULL` and ignores `as_of_tx`** (`overlays.rs:443,503`) — so tx-time travel only works on the no-query fallback path.
- **Valid-time** — the dimension that maps to *both* TR and KU — is **never populated.** `substrate.assert_statement` *accepts* `valid_from`/`valid_to` (`substrate.rs:153-193`), but **every memory ingest path passes `None`/`None`** (`semantic_claim.rs:71-82`, `episodic.rs:70-82`, `preference.rs:104-116`). The only per-memory timestamp is `mem:episodic/recorded_at = Utc::now()` at ingest — i.e. tx-time of ingestion, **not the message's event date.** *donto-memory today has no notion of "this fact was true on date X."*
- **Knowledge-update** supersession exists in **exactly one place** — the preference module adds a `supersedes` argument edge on change (`preference.rs:103-133`). Episodic and semantic-claim **accumulate** (paraconsistent hold) — on-vision, but it means `/recall` returns both old and new with **no recency/validity ordering**, so KU answers depend entirely on the reader guessing the latest from two undated claims.

### 6.3 The parity gap vs this session's `donto-extract` refactors

The live `memory-worker.service` has `DONTO_MEMORY_USE_EXTRACT=1` set (verified via `systemctl show`), so the production path **already routes through a `donto_extract` bridge** (`activities.py:293` `donto_extract_memorize_activity` → `ExtractionEngine(OpenCodeProvider, DEFAULT_LENSES)` → `SubstrateIngestor`, with a real `donto_document`+revision registered first via `_register_source`). So **source registration is at parity.** But the bridge pins **one lane** (OpenCodeProvider, glm-5.1), runs the **old single-pass-per-lens engine** with **substring-only anchoring**, and has **no failover**. A `grep` confirms `worker/` imports **none** of `lanes` / `cite` / `glean` / `failover` / `_flex_find`.

| refactor | donto-memory status | what's missing |
|---|---|---|
| **(a) Swappable lanes** (`donto_extract/lanes/{registry,caps,router,provenance,cli}` — codex + opencode, `--auto` failover, declarative `CapSignature`, pool-aware) | **None imported** | pinned to one glm-5.1 lane; a 500-Q run exhausts one subscription mid-eval |
| **(b) Gleaning loop** (`run_codex_glean.py` — codex-exec-resume + xhigh + anti-consolidation; lifted 1298→1915 facts) | **Stale variant** — it has its own `loop-until-dry` (DRY_STREAK=2), but the opencode per-pass-fresh-agent kind, not codex resume | depth (matters less for short memory chunks) |
| **(c) Always-on citer** (`cite_facts.py` — lexical `_flex_find` → bge-small semantic span → flag-unanchorable; **90.2% anchor on 3519 facts**) | **None** — extraction emits no `surface_text`; anchoring is substring-only at ingest | **the single most important missing piece** — abstention + faithfulness depend on it; this is the documented 4.7% evidence-link weakness and the `spans=0` bug |
| **(d) Faceted lenses / predicate hygiene** (shared `extract_broad.txt`) | **Stale variant** — its own 10 LENSES strings (`extraction.py:52-100`), divergent wording | drift; shared prompt via `OC_EXTRACT_BROAD_PROMPT_FILE` |
| **(e) Source registration + evidence_links** | **At parity** (`_register_source` present) | — (but wired to the old ingestor, not `donto-api` helpers) |

**Net:** donto-memory is **~70% there** (live bridge ON + source registration present). The missing ~30% for a fair test is **valid-time ingest + the citer + the lane-router** — all of which exist as code *today* and need packaging + importing, not inventing. Two load-bearing scripts (`cite_facts.py`, `run_codex_glean.py`) currently live in **volatile `/tmp/cerebras-test/`** and will vanish on reboot — they must be moved into the `donto_extract` package. And the engine remains **triplicated** (`donto/apps/donto-api`, `donto-memory/worker`, stale `workspace/donto-api`) — the rewrite's de-triplication is the clean end-state.

---

## 7. Why donto's model maps onto LongMemEval's hardest abilities

This is the thesis of the experiment, ability by ability. For each: how the *updated* donto-memory answers, the predicted strength/weakness, and the mechanism. Strengths conditional on the §8 ports are marked **(gated)**.

### 7.1 Knowledge-Update (78 Q, always 2 sessions) — donto's home ground

**Mechanism.** The same attribute is stated twice; the answer is the latest. donto **holds both** (paraconsistent, invariant I3 — never delete) and, once valid-time is stamped, the later assertion **supersedes by valid-time *and* tx-time.** **Strongest differentiator**, because the judge *explicitly* rewards *"previous info + updated answer … correct as long as the updated answer is present"* (verified `evaluate_qa.py:33`). Naive vector-dedup picks one winner at write time — and the two near-identical update turns collide in embedding space, so a flat retriever often returns the **stale** one. **(Gated):** today every memory fact has `valid_from=None` and recall has no recency ordering, so *without* the valid-time port donto degrades to "reader guesses the latest from two undated claims" and **loses the very advantage it claims.** This is the cleanest *substrate-beats-flat-RAG* headline and the right place for a **bitemporal ON-vs-OFF ablation** (predicted 10–25 point gap, in the band MatrixOrigin measured).

### 7.2 Abstention (30 Q, `_abs`) — the evidence-first payoff

**Mechanism.** Every `_abs` question targets a *sibling* of a genuinely-mentioned entity. donto answers **only from claims with a real `donto_evidence_link` to a span about the exact asked subject**; the always-on citer's flag-unanchorable → abstain. **Strength** — the judge grades correct *refusal* (`evaluate_qa.py:40-42`), and a hallucinated specific answer fails hard. Paraconsistent + evidence-first means donto can *see* that it has a claim about a neighbor but none about the asked subject. **The trap:** lexical/FTS recall returns the high-similarity distractor (cameras≈films, tennis≈table-tennis); if the reader is not gated on subject-identity it fabricates. **Honest tension:** the same entity alignment that helps multi-session dedup can *merge* sibling entities and destroy the abstention signal — must be tuned and measured on **both** buckets simultaneously.

### 7.3 Temporal-Reasoning (133 Q) — native valid-time vs the paper's bolt-on

**Mechanism.** Order/diff events using dates that are mostly **in message text** (`'3/22'`, `'last Saturday'`, `'two months ago'` — **~30 of the first 40** temporal-reasoning instances carry an explicit in-text date/time expression in a `has_answer` evidence turn, measured here), resolved against `question_date` rather than against session timestamps. donto's bitemporal model separates **valid-time** (event date, in content) from **tx-time** (when said ≈ session date). **(Gated) Strength** — the paper's own time-aware fix is a bolt-on (GPT-4o infers a range and reorders, +6.8–11.3% TR recall); a bitemporal store does valid-time filtering **natively with no `M_T` LLM call** (cheaper). The judge forgives off-by-one days, lowering the bar. **Weakness** — extracting event dates from colloquial relative phrases is an extraction-quality problem the glean loop must nail; and `as_of_tx` is plumbed but the ILIKE recall path ignores it. **Biggest upside, gated on the most plumbing.**

### 7.4 Multi-Session (133 Q) — the hardest type, a genuine opportunity

**Mechanism.** Aggregate/**count** distinct items across 2–5 sessions with **no single answer turn** (clothing→3; projects-LED→2 excluding "participated"; model-kits→5). Requires **complete** recall of all evidence + dedup + count. donto's **contexts/holder-scoping + query-time entity alignment** join claims across sessions structurally rather than via flat top-k, and the paper found fact decomposition **helps multi-session specifically** (§5.2). **Opportunity, not a guaranteed win:** if alignment under-merges, donto over-counts; if recall is incomplete (the lexical gap again), it under-counts. **Instrument with `recall_all@k`, not just QA.**

### 7.5 Information-Extraction (156 Q: user/assistant/preference) — honestly neutral-to-weak

- **single-session-user (70)** — recall is the whole game; the substrate's bitemporal/paraconsistent edge is irrelevant. **Weak:** lexical recall misses facts the question paraphrases (the dense-retriever gap); 10-lens over-extraction dilutes top-k (§5.2: fact decomposition *hurts* detail QA).
- **single-session-assistant (56)** — **structural strength:** donto stores both roles by construction (episodic-first regardless of role), where user-only memory systems score ~0. **Weak:** the answer often sits in long structured assistant text; feed the **episodic chunk** (full text), not just the truncated claim, to the reader.
- **single-session-preference (30)** — **strength:** lenient rubric judging + a dedicated preference module that already supersedes. **Weak:** the generic question shares no tokens with the preference ("learn video editing" vs "Adobe Premiere Pro") — the same dense-retrieval gap.

---

## 8. The port plan — bringing donto-memory to parity

De-triplicate toward **one** `donto_extract`. Split into **required** (omit any and the test is rigged against donto's own thesis) vs **nice-to-have**.

### 8.1 Required for a fair test

**R1 — Valid-time ingest (the most important port; *not* one of the five named refactors).** Patch `episodic.rs` / `semantic_claim.rs` / the `SubstrateIngestor` so each claim is stamped with the session's `haystack_date` as `valid_from`, and lift in-content event dates into valid-time. **Without this, knowledge-update and temporal-reasoning — donto's two headline strengths — have no bitemporal signal and donto degrades to undated flat-RAG.** This is the prerequisite for the KU and TR ablations that are the report's spine, and the gap the five-refactor framing **misses**.

**R2 — Always-on semantic citer (refactor c).** Fold `cite_facts.py` *into* the `donto_extract` package (it currently sys.path-hacks `donto-api` helpers and lives in volatile `/tmp`). Wire it as a stage in `donto_extract_memorize_activity` **before** `SubstrateIngestor.ingest`: lexical `_flex_find` → bge-small semantic span → flag-unanchorable (`hypothesis_only`, never a bogus span). Validated 90.2% anchor on 3519 facts. **Required** because abstention + faithfulness are donto's evidence-first differentiator and the benchmark scores exactly this. (Needs `fastembed`/bge-small on the worker's python env.)

**R3 — Lane router + `--auto` failover (refactor a).** Replace the pinned `OpenCodeProvider(glm-5.1)` with the lane router (`registry/caps/router/provenance/cli`). **Required** because a 500-question *online* run **will** exhaust one GLM subscription mid-eval; without rotation to codex/cerebras the eval dies partway and the result is invalid. Pool-aware (codex 4 lanes share one ChatGPT-Pro bucket; cerebras 2 share a credit pool). This is failover-for-completion, not a quality nicety.

### 8.2 Nice-to-have

- **N1 — Gleaning loop (refactor b).** Fold `run_codex_glean.py` in. For LongMemEval the memory chunks are **short** (one session), so a 1–2 pass glean is plenty; donto-memory's existing `loop-until-dry` is an acceptable substitute for the fair test.
- **N2 — Shared faceted prompt (refactor d).** Point the engine at `extract_broad.txt` via `OC_EXTRACT_BROAD_PROMPT_FILE` to kill lens drift.
- **N3 — pgvector dense-recall overlay (*not* a refactor — a separate recall workstream).** donto-memory recall is lexical; the paper chose dense (Stella-1.5B) over BM25 for a reason. A `DeriveEmbeddings` job type already exists (`types.rs:46`); pgvector is installed. **Strongly recommended** to be competitive on the IE types — this is the highest-leverage *non-extraction* improvement (MemMachine: retrieval ≫ ingestion) — but the QA test is still *fair* with lexical recall, provided the gap is disclosed.

### 8.3 Order of operations

1. **R1** valid-time ingest (Rust + ingestor) — unblocks the KU/TR ablations.
2. Move `/tmp/cerebras-test/{cite_facts.py, run_codex_glean.py}` **into** the `donto_extract` package (stop being volatile + sys.path-hacked).
3. **R2** wire the citer as a post-extract stage.
4. **R3** swap the pinned provider for the lane router + `--auto`.
5. Parity smoke vs the legacy path (`worker/compare_extract.py` already exists — read it first) to confirm fact yield ≥ legacy.
6. (nice) N1 glean depth, N2 shared prompt, N3 pgvector recall overlay.
7. **De-triplicate** — once `donto_extract` validates ≥ legacy yield, delete `worker/extraction.py` + `worker/opencode_agent.py` and the old `SubstrateIngestor` path, leaving **one** ingest = `donto-api` `helpers.ingest_facts`.

---

## 9. The concrete experiment design

Assumes the **updated (ported)** donto-memory. All paths absolute.

### Stage 0 — variant choice
- **Start** on `/home/ajax/longmemeval-study/data/longmemeval_oracle.json` (500, evidence-only, local). Isolates reading+synthesis from retrieval; directly comparable to the paper's **Oracle** column (GPT-4o 0.870 / 0.924 with CoN+JSON). If donto trails GPT-4o oracle here, the problem is reader/extraction, **not** retrieval.
- **Then** `longmemeval_s_cleaned.json` (~115k tokens, the realistic online test — must be the 2025/09 cleaned release for comparability).
- **Skip `_m` initially** (~500 sessions, ~1.5M tokens); run it once S works as the flagship *"memory beats long-context"* result (donto stores, doesn't stuff context).

### Stage 1 — online ingest (the critical timestamp injection)
Per-instance isolation: `holder = user:lme:<question_id>`, `context = ctx:memory/lme/<question_id>`. Replay `haystack_sessions` in `haystack_dates` order (oracle is **not** date-sorted — sort it yourself; `_s`/`_m` are pre-sorted). For each session `i`, `POST /memorize` with `session_id = haystack_session_ids[i]`. **Critical timestamp injection** (the whole point — without it KU/TR collapse): pass `haystack_dates[i]` as `valid_from` on every claim (the R1 port), and lift in-content event dates resolved against `question_date` into valid-time. Run the lane router with `--auto` + the always-on citer so every claim carries a real `donto_span` + `donto_evidence_link` (or is flagged `hypothesis_only`).

### Stage 2 — recall (at `t_q`)
Hybrid at `question_date` (the recency cutoff): `/recall` (holder-scoped RRF over `ctx:memory/lme/<qid>`) + `/search` (substrate-wide FTS) + **pgvector dense recall if N3 is built** (lexical-only will trail the paper's dense retrievers). Feed **both** the matched claims **and** their episodic chunks (the assistant-table case needs full text). For temporal Q, filter/rank by valid-time **natively** — donto's structural answer to the paper's §5.4 reorder hack.

### Stage 3 — reader (the paper's winning config)
Use the **con** reader (extract-then-reason) with **CoN+JSON** (paper Fig 6: up to +10 pts; GPT-4o 0.862→0.924). **Freeze the reader** = GPT-4o (paper-comparability) + optionally one strong modern reader (the ceiling); report both, never mix readers across systems. **Abstention handling (load-bearing):** instruct *"answer ONLY from retrieved claims whose subject IS the asked entity; if no such anchored claim exists, say the information was not provided."* Do **not** force an answer on `_abs`. **Knowledge-update:** *"if multiple values for the same attribute, lead with the one with the latest valid-time."* Emit `{"question_id", "hypothesis"}` per line.

### Stage 4 — eval
```
export OPENAI_API_KEY
cd /home/ajax/longmemeval-study/src/evaluation
python3 evaluate_qa.py gpt-4o <hyp.jsonl> ../../data/longmemeval_oracle.json
python3 print_qa_metrics.py <hyp.jsonl>.eval-results-gpt-4o ../../data/longmemeval_oracle.json
```
The judge is hardcoded `gpt-4o-2024-08-06`; `print_qa_metrics.py:20` **asserts** it (a local judge breaks comparability and the assertion).

### Metrics to report (measurable, not marketing)
1. **Per-type autoeval accuracy** (6 types) + Task-averaged + Overall + Abstention, vs the paper's GPT-4o baselines (oracle 0.870/0.924; full-S 0.606/0.640) and vs **Zep's per-type table** (the right comparison class).
2. **Turn-level + session-level Recall@k / NDCG@k** via `has_answer` + `answer_session_ids` — `recall_all@k` is the headline for multi-evidence Q. (Retrieval is over ~470 — the 30 `_abs` and no-`has_answer` items are dropped.) Requires emitting a retrieval log in `run_retrieval.py`'s corpus-id namespace (gold ids must contain `answer`) — extra plumbing; the **clean first win is QA accuracy**, retrieval recall is a bonus.
3. **Evidence-anchor rate** = % claims with a real `donto_evidence_link` (donto-native; the citer's 90.2% is the relevant figure) and **faithfulness** = % answers whose supporting claim is anchored.
4. **The four ablations — the report's spine** (they isolate the *substrate* advantage from reader/retrieval confounds):
   - (a) bitemporal valid-time **ON vs OFF** on knowledge-update — predicted **10–25 pt** gap.
   - (b) valid-time recall **ON vs OFF** on temporal-reasoning.
   - (c) evidence-gated-abstain **ON vs OFF** on the 30 `_abs`.
   - (d) extraction density **high vs low** on the single-session detail types — tests the §5.2 over-extraction prediction.

### Gotchas
Online memorization is **mandatory** (stream sessions; don't dump all at once, or you're testing long-context not memory). Verify a few KU instances actually have **two valid-time-distinct assertions** before scaling. A 500-Q online run **will** exhaust one GLM lane → the lane-router port is a **prerequisite**. `_m` ingest cost dominates — run oracle → S → M. **Report donto only with a frozen reader; never compare a donto GPT-4o run to a GPT-5-mini leaderboard entry.**

**Effort estimate:** oracle QA path (Stages 1–4, no retrieval log, no pgvector) ≈ **1–2 days** once the port lands; + S variant + ablations ≈ **1 week**; + retrieval-log plumbing + pgvector overlay + `_m` ≈ a **second week**. The port (§8) is the gating prerequisite, not part of this estimate.

---

## 10. Risks, limitations, and what a strong result would prove

### 10.1 Honest risks

- **A — Retrieval/reader is donto's true bottleneck, not extraction.** MemMachine's ablation (retrieval tuning ≫ ingestion) plus donto-memory's **lexical** recall predict donto will **trail dense-retriever baselines on the 156 IE-type questions** until an embedding overlay (N3) lands. This is a *second workstream* independent of the parity port — call it out, don't paper over it.
- **B — Reader-model confound caps comparability.** Post-2024 "SOTA" (OMEGA 95.4/GPT-4.1, Mastra 94.87/GPT-5-mini) is mostly **reader** progress (identical evidence: 58–90% across readers). donto's class is **Zep/Graphiti (71.2/GPT-4o)**, not the GPT-5-mini systems. Frame accordingly.
- **C — Over-extraction may hurt detail-QA precision.** 10-lens extraction emits hundreds of claims/session; §5.2 found fact decomposition helps *only* multi-session and *hurts* single-session detail QA. Without query-time alignment collapsing the noise, the dense claim graph dilutes top-k.
- **D — Alignment tension.** The entity alignment that helps multi-session dedup can *hurt* abstention if it merges sibling entities. Tune and measure on both buckets at once.
- **E — Subscription exhaustion.** A 500-Q online run exhausts one lane mid-eval — the lane-router port is a **prerequisite**, not a nicety.

### 10.2 The predicted profile (restated honestly)

donto should post a **distinctive shape**: **above-class on knowledge-update, abstention, and (if ported) temporal-reasoning** — proven by clean ablations, not by a headline overall number — and **at-or-slightly-below class on the three single-session IE types and overall**, bottlenecked by lexical recall. **The right story is not "donto wins overall." It is "donto wins on the abilities its substrate is built for, measurably, via clean ablations, with an honest recall-layer gap disclosed."**

### 10.3 What a strong result would prove

A successful run would demonstrate the substrate thesis empirically, not rhetorically:

1. **Bitemporal supersede beats flat-RAG on knowledge-update** — the bitemporal ON/OFF ablation showing a 10–25 pt gap would be the cleanest published evidence that *holding both and re-ranking by validity* outperforms *dedup-and-pick-a-winner* on exactly the task whose judge rubric was written for it.
2. **Evidence-first refusal beats hallucination on abstention** — abstention accuracy plus a donto-native faithfulness metric (% anchored) the paper cannot report, proving that *anchoring to the exact subject or abstaining* is a real capability, not a slogan.
3. **Native valid-time matches the paper's bolt-on temporal fix at lower cost** — TR parity with **no `M_T` LLM call** would validate that bitemporality is the right primitive, not a feature to be reconstructed per-query.
4. **The substrate's differentiators are *separable and measurable*** — four ablations isolating bitemporal, evidence-gating, alignment, and extraction-density turn donto's vision claims into numbers, which is the entire point: *be measurable, not marketing.*

And it would prove the meta-point of the whole rewrite: that **de-triplicating into one `donto_extract` and wiring the citer + lanes + valid-time** is not housekeeping — it is the difference between a memory system that *claims* bitemporal/paraconsistent/evidence-first superpowers and one that can **demonstrate** them on the field's reference benchmark.

---

### Sources

- arXiv **2410.10813** v2 (LongMemEval, ICLR 2025) — Tables 1–4, Figs 1–6, all numbers read from the PDF.
- Project site `xiaowu0162.github.io/long-mem-eval`; HF `xiaowu0162/longmemeval-cleaned`; **LongMemEval-V2** `github.com/xiaowu0162/LongMemEval-V2` + arXiv 2605.12493.
- Local repo `/home/ajax/longmemeval-study/`: `README.md`; `data/longmemeval_oracle.json` (500 instances — type counts, abstention distribution, sessions/instance, real examples all verified empirically); `src/evaluation/{evaluate_qa.py, print_qa_metrics.py, print_retrieval_metrics.py}`; `src/retrieval/{run_retrieval.py, eval_utils.py, index_expansion_utils.py}`; `src/index_expansion/*`; `src/generation/run_generation.py` + `.sh`.
- donto: `/mnt/donto-data/workspace/donto-memory/{crates/donto-memory*, worker/}` (`memorize.rs`, `hot_path.rs`, `overlays.rs:412-528`, `semantic_claim.rs`, `episodic.rs`, `preference.rs`, `substrate.rs:153-193`, `activities.py`, `workflows.py`, `extraction.py`); `/mnt/donto-data/workspace/donto-extract/src/donto_extract/{engine.py, lenses.py, lanes/*}`; `/tmp/cerebras-test/{cite_facts.py, run_codex_glean.py}`; `systemctl show memory-worker.service` (live env).
- Landscape: arXiv 2501.13956 (Zep), arXiv 2604.04853 (MemMachine), omegamax.co, mastra.ai, mem0.ai, blog.getzep.com, MatrixOrigin (reader-separation).

*All quoted donto behaviour is by code inspection, not a live trace; all donto-vs-LongMemEval performance is a hypothesis — no run has been executed.*