Every LoCoMo failure, and why it fails

A per-question post-mortem of donto's best LoCoMo claims-only run (turn-recall, depth 100, 75.8%). The point isn't the score — it's that knowing why each of the 48 failures fails tells you exactly which lever is donto's to pull and which isn't.

Method

Best valid configuration: turns recall (full dialogue turns, dated, in conversation order) + ±1 neighbour adjacency, uncapped reader budget, depth 100 → 75.8% at ~14K tokens (vs episodic stuffing's 83.7% at 19K). For every failed question we line up: the gold answer, what the reader said, and whether the answer was actually in the recalled context. That splits each failure into a root cause — and only some of those causes are donto's recall to fix.

The four failure modes

~60% (29/48) are reader- or judge-bound; ~40% (19/48) are recall-bound. Only the recall-bound ones are donto's to fix without either a stronger reader (off-budget) or judge tuning (off-limits per the fairness rule). Two further patterns hide inside these buckets and matter more than the raw counts:

Pattern 1 — multi-hop failures are almost all list/aggregation, and that's a donto lever

Nearly every multi-hop failure is "list everything X" where the reader returns a subset:

• "What activities does Melanie partake in?" — gold: pottery, camping, painting, swimming · reader: "Swimming, …"
• "What books has Melanie read?" — gold: "Nothing is Impossible", "Charlotte's Web" · reader: "Charlotte's Web"
• "What has Melanie painted?" — gold: Horse, sunset, sunrise · reader: "a lake sunrise, …"

The instances are scattered across sessions; coherent-turn recall surfaces some, the reader can only list what it sees. This is exactly what donto's pre-joined aggregate claims are for — "all activities Melanie does" computed once over the claim layer and handed to the reader as one fact. That's the next recall-side experiment, and it's a general substrate capability (answer-shaping aggregates), not a LoCoMo hack.

Pattern 2 — the judge under-counts: some "failures" are correct answers

• "When did Melanie run a charity race?" — gold: "the Sunday before 25 May 2023" · reader: "May 20, 2023" — which is the Sunday before 25 May. The reader was right; the judge didn't compute the relative-date equivalence.
• "What fields would Caroline pursue?" — gold: "Psychology, counseling certification" · reader: "Counseling or mental health…" — substantially correct, marked wrong on the missing "psychology".

So true accuracy is above 75.8%. We do not touch the judge (the fairness rule forbids tuning it on the test set), but it's important to state that a slice of the "failures" aren't donto's, the reader's, or even wrong.

Pattern 3 — open-domain is the reader's reasoning ceiling

Eleven open-domain failures are inference questions where the facts are recalled and the reader simply won't reason:

• "Would Caroline likely have Dr. Seuss books?" — gold: "Yes, since she collects classic children's books" · reader: "Not enough information."
• "Would Melanie go on another roadtrip soon?" — gold: "Likely no; this one went badly" (and the bad roadtrip is in context) · reader: "Not enough information."

The free holo3.1 reader is instructed to answer only from literal memories and is conservative about the leap. Fixing this needs a stronger reader (per-token, off-budget) or a prompt change (off-limits) — not recall. Honest boundary on the road to 100%.

What's actually donto's to fix (the 19 recall misses)

These are the legitimate levers, by sub-type:

• Aggregation/list (multi-hop) → pre-joined aggregate claims (gather all instances). Highest-value next experiment.
• Specific dated fact not recalled (temporal) — "When did Melanie read 'Nothing is Impossible'?" gold "2022" — the dated claim wasn't surfaced → deeper temporal recall + extraction granularity.
• Specific entity fact not recalled (open-domain) — "What state did Joanna visit in summer 2021?" gold "Indiana"; "Pomodoro technique" — the supporting fact wasn't retrieved → recall depth / finer extraction.
• Reader picks the session date over the event date — "When did Caroline join a mentorship program?" gold "the weekend before 17 July" · reader "July 17, 2023" — the recall should surface the event's resolved occurredAt more prominently than the session timestamp (a representation fix, donto-side).

The full ledger

All 48 failures, sorted by category:

Conclusion

The 75.8% ceiling decomposes cleanly: the next real gains are aggregate-claim recall for list/multi-hop questions and deeper dated-fact recall for the temporal/open-domain misses. The rest — inference refusals and a strict judge — sit with the free reader and the grader, not with donto's retrieval. That's the no-shortcuts map: pull the recall levers, report the reader/judge ceiling honestly, and don't pretend a grader artifact is a substrate win.

This analysis is regenerated from the live run transcripts — drill into any failure in the runs viewer.