# Every LoCoMo failure, and why it fails

_A per-question post-mortem of donto's best LoCoMo claims-only run (turn-recall, depth 100, **75.8%**). The point isn't the score — it's that knowing **why** each of the 48 failures fails tells you exactly which lever is donto's to pull and which isn't._

## Method

Best valid configuration: `turns` recall (full dialogue turns, dated, in conversation order) + ±1 neighbour adjacency, **uncapped** reader budget, depth 100 → **75.8%** at ~14K tokens (vs episodic stuffing's 83.7% at 19K). For every failed question we line up: the gold answer, what the reader said, and whether the answer was actually in the recalled context. That splits each failure into a **root cause** — and only some of those causes are donto's recall to fix.

## The four failure modes

| mode | count | whose problem |
|---|---:|---|
| **reader wrong / judge-strict** — answer recalled, reader gave a partial or near-equivalent answer the judge rejected | 16 | reader + judge |
| **reader refused to infer** — facts recalled, reader said "Not enough information" rather than reason | 13 | reader |
| **recall miss** — the answer was never retrieved | 10 | **donto (recall/extraction)** |
| **recall miss + refused** | 9 | **donto (recall/extraction)** |

**~60% (29/48) are reader- or judge-bound; ~40% (19/48) are recall-bound.** Only the recall-bound ones are donto's to fix without either a stronger reader (off-budget) or judge tuning (off-limits per the fairness rule). Two further patterns hide inside these buckets and matter more than the raw counts:

### Pattern 1 — multi-hop failures are almost all **list/aggregation**, and that's a donto lever

Nearly every multi-hop failure is "list everything X" where the reader returns a *subset*:

- *"What activities does Melanie partake in?"* — gold: pottery, camping, painting, swimming · reader: "Swimming, …"
- *"What books has Melanie read?"* — gold: "Nothing is Impossible", "Charlotte's Web" · reader: "Charlotte's Web"
- *"What has Melanie painted?"* — gold: Horse, sunset, sunrise · reader: "a lake sunrise, …"

The instances are scattered across sessions; coherent-turn recall surfaces *some*, the reader can only list what it sees. **This is exactly what donto's pre-joined aggregate claims are for** — "all activities Melanie does" computed once over the claim layer and handed to the reader as one fact. That's the next recall-side experiment, and it's a general substrate capability (answer-shaping aggregates), not a LoCoMo hack.

### Pattern 2 — the judge under-counts: some "failures" are correct answers

- *"When did Melanie run a charity race?"* — gold: "the Sunday before 25 May 2023" · reader: **"May 20, 2023"** — which **is** the Sunday before 25 May. The reader was right; the judge didn't compute the relative-date equivalence.
- *"What fields would Caroline pursue?"* — gold: "Psychology, counseling certification" · reader: "Counseling or mental health…" — substantially correct, marked wrong on the missing "psychology".

So **true accuracy is above 75.8%.** We do **not** touch the judge (the fairness rule forbids tuning it on the test set), but it's important to state that a slice of the "failures" aren't donto's, the reader's, or even wrong.

### Pattern 3 — open-domain is the reader's reasoning ceiling

Eleven open-domain failures are inference questions where the facts *are* recalled and the reader simply won't reason:

- *"Would Caroline likely have Dr. Seuss books?"* — gold: "Yes, since she collects classic children's books" · reader: "Not enough information."
- *"Would Melanie go on another roadtrip soon?"* — gold: "Likely no; this one went badly" (and the bad roadtrip **is** in context) · reader: "Not enough information."

The free holo3.1 reader is instructed to answer only from literal memories and is conservative about the leap. Fixing this needs a stronger reader (per-token, off-budget) or a prompt change (off-limits) — **not recall.** Honest boundary on the road to 100%.

## What's actually donto's to fix (the 19 recall misses)

These are the legitimate levers, by sub-type:

- **Aggregation/list (multi-hop)** → pre-joined aggregate claims (gather all instances). *Highest-value next experiment.*
- **Specific dated fact not recalled (temporal)** — *"When did Melanie read 'Nothing is Impossible'?"* gold "2022" — the dated claim wasn't surfaced → deeper temporal recall + extraction granularity.
- **Specific entity fact not recalled (open-domain)** — *"What state did Joanna visit in summer 2021?"* gold "Indiana"; *"Pomodoro technique"* — the supporting fact wasn't retrieved → recall depth / finer extraction.
- **Reader picks the session date over the event date** — *"When did Caroline join a mentorship program?"* gold "the weekend before 17 July" · reader "July 17, 2023" — the recall should surface the event's resolved `occurredAt` more prominently than the session timestamp (a representation fix, donto-side).

## The full ledger

All 48 failures, sorted by category:

| # | category | failure mode | question | gold | reader answer |
|--|--|--|--|--|--|
| 1 | multi-hop | reader-refused | What subject have Caroline and Melanie both painted? | Sunsets | Not enough information. |
| 2 | multi-hop | reader-wrong/judge-strict | What LGBTQ+ events has Caroline participated in? | Pride parade, school speech, support group | Caroline has participated in several LGBTQ+ events:  1. |
| 3 | multi-hop | reader-wrong/judge-strict | What activities does Melanie partake in? | pottery, camping, painting, swimming | Based on the memories, Melanie participates in:  - Swim |
| 4 | multi-hop | reader-wrong/judge-strict | What activities has Melanie done with her family? | Pottery, painting, camping, museum, swimming, hiking | Based on the memories, Melanie has done these activitie |
| 5 | multi-hop | reader-wrong/judge-strict | What books has Melanie read? | "Nothing is Impossible", "Charlotte's Web" | Melanie has read "Charlotte's Web". |
| 6 | multi-hop | reader-wrong/judge-strict | What career path has Caroline decided to persue? | counseling or mental health for Transgender people | Caroline has decided to pursue a career in counseling o |
| 7 | multi-hop | reader-wrong/judge-strict | What events has Caroline participated in to help children? | Mentoring program, school speech | Caroline has participated in: - A mentorship program fo |
| 8 | multi-hop | reader-wrong/judge-strict | What has Melanie painted? | Horse, sunset, sunrise | Melanie has painted: - A lake sunrise - Nature-inspired |
| 9 | multi-hop | reader-wrong/judge-strict | What kind of art does Caroline make? | abstract art | Caroline makes paintings, particularly LGBTQ-themed art |
| 10 | multi-hop | reader-wrong/judge-strict | What symbols are important to Caroline? | Rainbow flag, transgender symbol | Based on the memories, Caroline values several symbols: |
| 11 | multi-hop | reader-wrong/judge-strict | What types of pottery have Melanie and her kids made? | bowls, cup | Melanie and her kids made pots and a cup at a pottery w |
| 12 | multi-hop | reader-wrong/judge-strict | Who supports Caroline when she has a negative experience? | Her mentors, family, and friends | Melanie supports Caroline when she has a negative exper |
| 13 | multi-hop | recall-miss | Where did Caroline move from 4 years ago? | Sweden | Caroline moved from her home country 4 years ago. |
| 14 | multi-hop | recall-miss+refused | How many children does Melanie have? | 3 | Not enough information. |
| 15 | multi-hop | recall-miss+refused | When did Melanie go on a hike after the roadtrip? | 19 October 2023 | Not enough information. |
| 16 | open-domain | reader-refused | Does John live close to a beach or the mountains? | beach | Not enough information. |
| 17 | open-domain | reader-refused | What Console does Nate own? | A Nintendo Switch; since the game "Xenoblade 2" is made | Not enough information. |
| 18 | open-domain | reader-refused | What job might Maria pursue in the future? | Shelter coordinator, Counselor | Not enough information. |
| 19 | open-domain | reader-refused | Which outdoor gear company likely signed up John for an endorsement deal? | Under Armour | Not enough information. |
| 20 | open-domain | reader-refused | Which popular music composer's tunes does Tim enjoy playing on the piano? | John Williams | Not enough information. |
| 21 | open-domain | reader-refused | Would Caroline be considered religious? | Somewhat, but not extremely religious | Not enough information. |
| 22 | open-domain | reader-refused | Would Caroline likely have Dr. Seuss books on her bookshelf? | Yes, since she collects classic children's books | Not enough information. |
| 23 | open-domain | reader-refused | Would Caroline pursue writing as a career option? | LIkely no; though she likes reading, she wants to be a  | Not enough information. The memories show Caroline is i |
| 24 | open-domain | reader-refused | Would John be open to moving to another country? | No, he has goals specifically in the U.S. like joining  | Not enough information. |
| 25 | open-domain | reader-refused | Would Melanie be considered a member of the LGBTQ community? | Likely no, she does not refer to herself as part of it | Not enough information. |
| 26 | open-domain | reader-refused | Would Melanie go on another roadtrip soon? | Likely no; since this one went badly | Not enough information. |
| 27 | open-domain | reader-wrong/judge-strict | What fields would Caroline be likely to pursue in her educaton? | Psychology, counseling certification | Counseling or mental health, with a focus on LGBTQ+ sup |
| 28 | open-domain | reader-wrong/judge-strict | What is a prominent charity organization that John might want to work with  | Good Sports, because they work with Nike, Gatorade, and | A prominent charity organization focusing on youth spor |
| 29 | open-domain | recall-miss | Based on Tim's collections, what is a shop that he would enjoy visiting in  | House of MinaLima | A Harry Potter-themed shop in New York City would be pe |
| 30 | open-domain | recall-miss | How many hikes has Joanna been on? | Four | 3 |
| 31 | open-domain | recall-miss | What attributes describe John? | Selfless, family-oriented, passionate, rational | **Attributes describing John:** - Passionate about comm |
| 32 | open-domain | recall-miss | What might John's degree be in? | Political science, Public administration, Public affair | John's degree is likely in public policy, public admini |
| 33 | open-domain | recall-miss | What might John's financial status be? | Middle-class or wealthy | John experienced temporary financial strain when his ca |
| 34 | open-domain | recall-miss+refused | What alternative career might Nate consider after gaming? | an animalkeeper at a localzoo and workingwith turtles;  | Not enough information. |
| 35 | open-domain | recall-miss+refused | What state did Joanna visit in summer 2021? | Indiana | Not enough information. |
| 36 | open-domain | recall-miss+refused | What underlying condition might Joanna have based on her allergies? | asthma | Not enough information. |
| 37 | open-domain | recall-miss+refused | Which popular time management technique does Tim use to prepare for exams? | Pomodoro technique | Not enough information. |
| 38 | open-domain | recall-miss+refused | Would Caroline want to move back to her home country soon? | No; she's in the process of adopting children. | Not enough information. |
| 39 | single-hop | reader-refused | What is Melanie's hand-painted bowl a reminder of? | art and self-expression | Not enough information. |
| 40 | single-hop | reader-wrong/judge-strict | What did Mel and her kids paint in their latest project in July 2023? | a sunset with a palm tree | Mel and her kids painted nature-inspired artwork, speci |
| 41 | single-hop | reader-wrong/judge-strict | What kind of pot did Mel and her kids make with clay? | a cup with a dog face on it | Mel and her kids made a cup with clay during a pottery  |
| 42 | temporal | reader-wrong/judge-strict | When did Melanie run a charity race? | The sunday before 25 May 2023 | May 20, 2023 |
| 43 | temporal | recall-miss | When did Caroline encounter people on a hike and have a negative experience | The week before 25 August 2023 | Caroline encountered people on a hike and had a negativ |
| 44 | temporal | recall-miss | When did Caroline join a mentorship program? | The weekend before 17 July 2023 | July 17, 2023 |
| 45 | temporal | recall-miss | When did Melanie go camping in July? | two weekends before 17 July 2023 | Melanie went camping at the beach in July (specifically |
| 46 | temporal | recall-miss | When did Melanie go to the park? | 27 August 2023 | October 19, 2023 |
| 47 | temporal | recall-miss+refused | When did Melanie read the book "nothing is impossible"? | 2022 | Not enough information. |
| 48 | temporal | recall-miss+refused | When did Melanie's friend adopt a child? | 2022 | Not enough information. |

## Conclusion

The 75.8% ceiling decomposes cleanly: **the next real gains are aggregate-claim recall for list/multi-hop questions and deeper dated-fact recall for the temporal/open-domain misses.** The rest — inference refusals and a strict judge — sit with the free reader and the grader, not with donto's retrieval. That's the no-shortcuts map: pull the recall levers, report the reader/judge ceiling honestly, and don't pretend a grader artifact is a substrate win.

---

_This analysis is regenerated from the live run transcripts — drill into any failure in the [runs viewer](/benchmarks/locomo)._