donto extraction — checkpoint (2026-06-07)

A snapshot of the live extraction stack: what's running, what it's producing, the providers it rotates, the backlog, and how to operate it. Numbers verified against the running box at 2026-06-07T04:13Z (read-only donto_ro / psql against donto-pg).

TL;DR

• The extraction engine is donto-agent — a Rust crate in the donto Cargo workspace (apps/donto-agent), binary at /home/ajax/bin/donto-agent. Two modes: extract <file> (one-shot) and work (drain a queue). It reuses the superb extraction prompt and donto-client for evidence-anchored ingest.
• BEAM-10M is actively extracting right now via donto-agent-beam.service on Hyades / holo3.1 (no per-token cost), ~0.045 chunk/s across 5 concurrent workers, 6 gleaning passes per chunk, ~150–250 claims per chunk.
• It writes to per-chunk sub-contexts ctx:claims/beam/<chunk-uuid> — not the parent ctx:claims/beam. (This bit us during this checkpoint: a query on the exact parent context showed "0 in 24h / stalled"; the truth is ~9,500 claims/hour landing in the children. Documented below so it isn't re-misdiagnosed.)
• Three other extraction services are up: genealogy (donto-api + donto-api-worker, Temporal) and memory (memory-worker, Temporal).
• Budget posture holds: all extraction runs on flat/prepaid lanes (Hyades holo3.1 primary; GLM + codex available) — no per-token API spend.
• Substrate is at ~40.91M live statements; BEAM has contributed 338,457 so far and discord 582,250.

1. Extraction services (live)

Supporting containers: donto-pg (the substrate DB), temporal + temporal-ui (durable queues), omega-vm-omega-bot-1 + opencode-runner (the OpenCode/GLM extraction lane for the genealogy/memory paths).

2. The engine — donto-agent (Rust-first)

/mnt/donto-data/workspace/donto/apps/donto-agent/ (lib donto_agent + bin donto-agent). The pipeline per chunk:

1. Extract — the chunk is sent to the provider behind the prompt from donto-api/prompts/extract_broad.txt (single source of truth), with the source bookended as inert data (prompt-injection defense — essential for conversational BEAM chunks, or the model answers the embedded User: turn instead of extracting).
2. Gleaning loop — max-passes thread-nudge passes ("what did you miss?") until saturation/dry; on Hyades this is a server-side thread nudge (X-Hyades-Thread) rather than re-sending the prompt.
3. Ingest, evidence-anchored — via donto-client (assert + create_char_span + link_evidence_span), confidence→maturity capped at E2 (I5: machine confidence can't self-promote past E2).
4. Citer — STATED facts get an anchored span; INTERPRETED facts are honestly flagged, never given a bogus span.

Subcommands

donto-agent extract <file> --context ctx:agent/demo          # one file, extract+ingest
donto-agent extract <file> --dry-run                         # print, no DB write
donto-agent work --context ctx:claims/beam --concurrency 5 --max-passes 6   # drain a queue

Build note (toolchain is ajaxdavis-owned):

RUSTUP_HOME=/home/ajaxdavis/.rustup CARGO_HOME=/home/ajaxdavis/.cargo \
PATH=/home/ajaxdavis/.cargo/bin:$PATH CARGO_TARGET_DIR=/tmp/da-target \
cargo build -p donto-agent && cp /tmp/da-target/debug/donto-agent /home/ajax/bin/

The older Python engine (donto-extract package — gleaning, citer, lanes, ingest) remains the de-triplicated reference and the genealogy/memory lane; donto-agent is the throughput path.

3. Providers / lanes (flat-rate only — no per-token spend)

Hyades parallelizes and persists conversation threads server-side (the gleaning loop nudges the same thread). Reasoning models can return 200 + empty content if reasoning eats the budget → donto-agent retries on empty; GLM lanes pass {"thinking":{"type":"disabled"}} to avoid it. Keys: /etc/donto/hyades.key, /etc/donto/glm.key.

4. Live work — BEAM-10M

The headline run: extract the BEAM-10M conversational corpus (~352K chunks) into evidence-anchored claims.

⚠️ Operational gotcha (verify-it-works)

BEAM claims land in per-chunk children ctx:claims/beam/<uuid>, so a count on the exact parent ctx:claims/beam reads ~7K and "frozen since Jun 5" — which looks stalled but is not. Always aggregate with LIKE 'ctx:claims/beam%', and measure real recency with lower(tx_time) (not the && overlap operator — open-ended live ranges overlap every recent window, so && reports the whole substrate as "last hour"). The true live signal is the log's chunk lines + fresh beam-conv/ transcripts + the queue done counter incrementing.

-- real BEAM throughput
SELECT count(*) FROM donto_statement
 WHERE context LIKE 'ctx:claims/beam%' AND lower(tx_time) > now() - interval '60 min';
-- queue state
SELECT status, count(*) FROM donto_extract_queue GROUP BY status;

Pause/resume (e.g. for a Hyades restart): systemctl stop|start donto-agent-beam.

5. Recent one-off extractions

• Seven Sisters / Pleiades myth → ctx:mythology/seven-sisters — 101 claims, 25 entities, 44 predicates, 24 evidence-anchored (Hyades/holo3.1, 2 passes). Demonstrates paraconsistent cross-cultural holding: one star cluster, one myth, three MythVariant nodes (Greek/Aboriginal/Māori) held side-by-side; isSameAs as a hypothesis, not a merge.
• Discord "blah" server → ctx:discord/blah/* — full corpus extracted via donto-agent + GLM-4.7 (thinking-disabled), zero cap hits — 582,250 claims live.

6. Backlog & next

• BEAM queue: 352,902 chunks pending. At ~0.045 chunk/s this is the long pole; throughput is provider-bound (Hyades), so the lever is more concurrency or a second lane via --key-file (each key has its own concurrency cap → keys sum).
• Embedding fabric (separate from extraction but downstream of it): the distributed coordinator at donto.org/embed now embeds predicate/entity/memory aspects; contributor workers (see donto.org/help) chew the backlog so query-time alignment keeps up with the claim firehose.
• Citer coverage: every lane's output flows through the always-on semantic citer; BEAM claims carry spans where stated, flagged-interpreted otherwise.

7. Operate it

# one file, evidence-anchored, into a context:
/home/ajax/bin/donto-agent extract /path/to/source.txt --context ctx:mythology/seven-sisters --max-passes 4

# drain a queue (what the BEAM service runs):
/home/ajax/bin/donto-agent work --context ctx:claims/beam --concurrency 5 --max-passes 6

# verify it landed (no shortcuts — "extracted" = rows in donto_statement):
sudo docker exec donto-pg psql -U donto -d donto -c \
  "SELECT count(*) FROM donto_statement WHERE context LIKE 'ctx:claims/beam%' AND upper(tx_time) IS NULL;"

# watch BEAM live:
tail -f /opencode/logs/beam-donto-agent-holo31.log

The one hard rule still governs extraction: writes are asserts (append-only), never DELETE FROM donto_statement. A failed chunk resets to pending (I3-safe), never dropped.

Checkpoint generated 2026-06-07. Source of truth: the running box (donto-agent-beam.service, donto_extract_queue, /opencode/logs/). Re-run the §4 SQL for current numbers.