# donto extraction — checkpoint (2026-06-07)

*A snapshot of the live extraction stack: what's running, what it's producing, the providers it
rotates, the backlog, and how to operate it. Numbers verified against the running box at
`2026-06-07T04:13Z` (read-only `donto_ro` / `psql` against `donto-pg`).*

---

## TL;DR

- **The extraction engine is `donto-agent`** — a Rust crate in the donto Cargo workspace (`apps/donto-agent`), binary at `/home/ajax/bin/donto-agent`. Two modes: `extract <file>` (one-shot) and `work` (drain a queue). It reuses the superb extraction prompt and `donto-client` for evidence-anchored ingest.
- **BEAM-10M is actively extracting right now** via `donto-agent-beam.service` on **Hyades / holo3.1** (no per-token cost), ~**0.045 chunk/s** across **5 concurrent** workers, **6 gleaning passes** per chunk, ~**150–250 claims per chunk**.
- **It writes to per-chunk sub-contexts** `ctx:claims/beam/<chunk-uuid>` — *not* the parent `ctx:claims/beam`. (This bit us during this checkpoint: a query on the exact parent context showed "0 in 24h / stalled"; the truth is ~**9,500 claims/hour** landing in the children. Documented below so it isn't re-misdiagnosed.)
- **Three other extraction services are up**: genealogy (`donto-api` + `donto-api-worker`, Temporal) and memory (`memory-worker`, Temporal).
- **Budget posture holds**: all extraction runs on flat/prepaid lanes (Hyades `holo3.1` primary; GLM + codex available) — **no per-token API spend**.
- **Substrate is at ~40.91M live statements**; BEAM has contributed **338,457** so far and discord **582,250**.

---

## 1. Extraction services (live)

| Service | Role | State (04:13Z) |
|---|---|---|
| **`donto-agent-beam.service`** | Rust `donto-agent work` — BEAM-10M claim extraction on Hyades/holo3.1, thread-nudge gleaning loop-until-dry, `--concurrency 5 --max-passes 6 --max-tokens 2000` | **active, working** (leasing + ingesting now) |
| **`donto-api.service`** | FastAPI — genealogy extract/ingest (`/extract-and-ingest`, opencode lanes) | active |
| **`donto-api-worker.service`** | donto's Python Temporal worker, task queue `donto-extraction` | active |
| **`memory-worker.service`** | Python Temporal worker — memory extraction + `/enqueue` | active |
| **`donto-memory-api.service`** | donto-memory HTTP API (`/memorize`, `/recall`, `/search`, `/jobs`) | active |

Supporting containers: `donto-pg` (the substrate DB), `temporal` + `temporal-ui` (durable queues), `omega-vm-omega-bot-1` + `opencode-runner` (the OpenCode/GLM extraction lane for the genealogy/memory paths).

---

## 2. The engine — `donto-agent` (Rust-first)

`/mnt/donto-data/workspace/donto/apps/donto-agent/` (lib `donto_agent` + bin `donto-agent`). The pipeline per chunk:

1. **Extract** — the chunk is sent to the provider behind the prompt from `donto-api/prompts/extract_broad.txt` (single source of truth), with the source **bookended as inert data** (prompt-injection defense — essential for conversational BEAM chunks, or the model answers the embedded `User:` turn instead of extracting).
2. **Gleaning loop** — `max-passes` thread-nudge passes ("what did you miss?") until saturation/dry; on Hyades this is a server-side thread nudge (`X-Hyades-Thread`) rather than re-sending the prompt.
3. **Ingest, evidence-anchored** — via `donto-client` (`assert` + `create_char_span` + `link_evidence_span`), confidence→maturity capped at **E2** (I5: machine confidence can't self-promote past E2).
4. **Citer** — STATED facts get an anchored span; INTERPRETED facts are honestly flagged, never given a bogus span.

**Subcommands**
```bash
donto-agent extract <file> --context ctx:agent/demo          # one file, extract+ingest
donto-agent extract <file> --dry-run                         # print, no DB write
donto-agent work --context ctx:claims/beam --concurrency 5 --max-passes 6   # drain a queue
```

**Build note** (toolchain is `ajaxdavis`-owned):
```bash
RUSTUP_HOME=/home/ajaxdavis/.rustup CARGO_HOME=/home/ajaxdavis/.cargo \
PATH=/home/ajaxdavis/.cargo/bin:$PATH CARGO_TARGET_DIR=/tmp/da-target \
cargo build -p donto-agent && cp /tmp/da-target/debug/donto-agent /home/ajax/bin/
```

The older Python engine (`donto-extract` package — `gleaning`, `citer`, `lanes`, `ingest`) remains the de-triplicated reference and the genealogy/memory lane; `donto-agent` is the throughput path.

---

## 3. Providers / lanes (flat-rate only — no per-token spend)

| Lane | Model | Use | Cost |
|---|---|---|---|
| **Hyades** (user's gateway, `https://hyades.gnostr.cloud/v1`) | **`holo3.1`** (default) · also `glm-5.1`, `nemotron3-omni`, `openrouter/free` | BEAM bulk extraction (workhorse) | **$0 / no cap** |
| **GLM coding** (z.ai) | `glm-4.7` / `glm-5.1` | discord + genealogy opencode lane | flat subscription |
| **Codex / ChatGPT-Pro** | `gpt-5.3-codex-spark` | best-anchored quality lane | flat subscription |

Hyades parallelizes and persists conversation threads server-side (the gleaning loop nudges the same thread). Reasoning models can return `200` + empty content if reasoning eats the budget → `donto-agent` retries on empty; GLM lanes pass `{"thinking":{"type":"disabled"}}` to avoid it. Keys: `/etc/donto/hyades.key`, `/etc/donto/glm.key`.

---

## 4. Live work — BEAM-10M

The headline run: extract the BEAM-10M conversational corpus (~352K chunks) into evidence-anchored claims.

| Metric | Value (04:13Z) |
|---|---|
| Queue (`donto_extract_queue`) | **352,902 pending · 7,995 done · 5 leased** |
| Throughput | ~**0.045 chunk/s** (5 concurrent, 6 passes each) |
| Claims/chunk | ~**150–250** (e.g. 174, 196, 158, 163 in the last few) |
| Claims landing | ~**9,300 / hour**, newest insert = now (04:24Z) |
| BEAM claims in substrate | **338,457** live, across **3,790** chunk sub-contexts |
| Target contexts | **`ctx:claims/beam/<chunk-uuid>`** (one sub-context per chunk) |
| Per-chunk transcript | `/opencode/logs/beam-conv/<thread-uuid>.log` (source + every turn) |
| Run log | `/opencode/logs/beam-donto-agent-holo31.log` |

### ⚠️ Operational gotcha (verify-it-works)
BEAM claims land in **per-chunk children** `ctx:claims/beam/<uuid>`, so a count on the **exact** parent `ctx:claims/beam` reads ~7K and "frozen since Jun 5" — which *looks* stalled but is not. Always aggregate with `LIKE 'ctx:claims/beam%'`, and measure real recency with `lower(tx_time)` (not the `&&` overlap operator — open-ended live ranges overlap *every* recent window, so `&&` reports the whole substrate as "last hour"). The true live signal is the log's chunk lines + fresh `beam-conv/` transcripts + the queue `done` counter incrementing.

```sql
-- real BEAM throughput
SELECT count(*) FROM donto_statement
 WHERE context LIKE 'ctx:claims/beam%' AND lower(tx_time) > now() - interval '60 min';
-- queue state
SELECT status, count(*) FROM donto_extract_queue GROUP BY status;
```

**Pause/resume** (e.g. for a Hyades restart): `systemctl stop|start donto-agent-beam`.

---

## 5. Recent one-off extractions

- **Seven Sisters / Pleiades myth** → `ctx:mythology/seven-sisters` — **101 claims, 25 entities, 44 predicates, 24 evidence-anchored** (Hyades/holo3.1, 2 passes). Demonstrates paraconsistent cross-cultural holding: one star cluster, one myth, three `MythVariant` nodes (Greek/Aboriginal/Māori) held side-by-side; `isSameAs` as a hypothesis, not a merge.
- **Discord "blah" server** → `ctx:discord/blah/*` — full corpus extracted via `donto-agent` + GLM-4.7 (thinking-disabled), zero cap hits — **582,250 claims** live.

---

## 6. Backlog & next

- **BEAM queue: 352,902 chunks pending.** At ~0.045 chunk/s this is the long pole; throughput is provider-bound (Hyades), so the lever is more concurrency or a second lane via `--key-file` (each key has its own concurrency cap → keys sum).
- **Embedding fabric** (separate from extraction but downstream of it): the distributed coordinator at `donto.org/embed` now embeds predicate/entity/memory aspects; contributor workers (see `donto.org/help`) chew the backlog so query-time alignment keeps up with the claim firehose.
- **Citer coverage**: every lane's output flows through the always-on semantic citer; BEAM claims carry spans where stated, flagged-interpreted otherwise.

---

## 7. Operate it

```bash
# one file, evidence-anchored, into a context:
/home/ajax/bin/donto-agent extract /path/to/source.txt --context ctx:mythology/seven-sisters --max-passes 4

# drain a queue (what the BEAM service runs):
/home/ajax/bin/donto-agent work --context ctx:claims/beam --concurrency 5 --max-passes 6

# verify it landed (no shortcuts — "extracted" = rows in donto_statement):
sudo docker exec donto-pg psql -U donto -d donto -c \
  "SELECT count(*) FROM donto_statement WHERE context LIKE 'ctx:claims/beam%' AND upper(tx_time) IS NULL;"

# watch BEAM live:
tail -f /opencode/logs/beam-donto-agent-holo31.log
```

> **The one hard rule still governs extraction:** writes are asserts (append-only), never `DELETE FROM donto_statement`. A failed chunk resets to `pending` (I3-safe), never dropped.

---

*Checkpoint generated 2026-06-07. Source of truth: the running box (`donto-agent-beam.service`, `donto_extract_queue`, `/opencode/logs/`). Re-run the §4 SQL for current numbers.*