donto — Operations Session Report (through 2026-06-09)

A complete record of the work done this session: contributor infrastructure, the distributed embedding fabric, Omega's graph-reasoning upgrade, five research reports, two book extractions, and a deep repair of the post-ingest processing pipeline. Every claim here was verified live against the running box; artifacts and current state are listed at the end.

Executive summary

This session shipped, in order:

1. A contributor on-ramp — donto.org/help, a one-command public Docker image, an embed-worker kit, and a comprehensive help-embed.md so anyone (or their Claude Code) can lend compute in seconds.
2. The distributed embedding fabric, completed — the coordinator now embeds every aspect (predicates + entities + memory chunks), self-refills its queue, and reports stats instantly; moved to the clean donto.org/embed endpoint.
3. A live embedding tracker on admin.donto.org (/embed) so contributions are visible.
4. Omega can now reason over and deeply traverse the substrate — a 12-lens dontoQuery tool extended with a job/document lens that mirrors the admin job page, next-hop suggestions to drive multi-tool investigation, and a routing fix that took it from deployed-but-dormant to actually used. Deployed to the live bot and verified.
5. Five /research reports — the alignment contributor guide, the Omega-integration audit, the full API reference, the extraction checkpoint, and the admin observability audit.
6. Extraction — the Seven Sisters myth, two full Project Gutenberg books (The Seven Sisters of Sleep and Hamlet), and confirmation the discord corpus is complete (~582K claims).
7. A deep repair of the post-ingest processing pipeline — the alignment daemon had been doing zero work for days (dry-run + a leaked process starving its load gate); the context matview was 6 days stale. All fixed, scheduled, and verified producing real work.

1. Contributor on-ramp — donto.org/help

The goal: let friends donate spare CPU/GPU to embed donto's backlog with zero friction.

• https://donto.org/help — a focused, single-purpose page (rewritten through 5 refinement passes from an earlier bloated version): run an embedding worker, nothing else. Two headline paths — a one-command Docker run and a paste-into-Claude-Code block — plus get-the-token, verify-it-worked, and a tight FAQ. Caddy serves it for both /help and /help/ (fixed an earlier bare-/help blank-page bug). Backing files in /srv/donto-help/ (index.html, claude.txt, help-embed.md).
• One-command Docker image — ghcr.io/thomasdavis/donto-embed-worker (public, anonymously pullable, built via GitHub Actions in repo thomasdavis/donto-embed-worker). docker run --rm -e DONTO_EMBED_TOKEN=… ghcr.io/thomasdavis/donto-embed-worker runs one worker per core, GPU-aware. Proven: pulled anonymously and embedded 2,352 vectors in one run.
• help-embed.md — a 16-section comprehensive guide (Docker / Claude-Code / bare-Python / compose, GPU, fleet, verify, env reference, how-it-works, troubleshooting, FAQ, full worker source, HTTP API). Served at donto.org/help/help-embed.md.
• A real bug caught here: Cloudflare 403s the default python-urllib User-Agent — the worker now sends a curl/8 UA. Without this, every contributor's worker would have silently failed.

2. Distributed embedding fabric — completed

donto-embed-coordinator.service (the lease→embed→submit work queue for remote workers):

• Wired predicate + entity providers alongside the original memory-chunk one, reusing the exact text/signature builders from embed.py / entity_embed.py (byte-for-byte vector parity with the daemon). /embed/health now reports ["memory_chunk","predicate","entity"]. Round-trip (lease→bge-small→submit→upsert) verified for all three.
• Self-topping queue — an auto-enqueue task keeps each target's pending count topped from the backlog, so contributors never run dry (currently 200K queued: 50K predicate + 50K entity + 100K chunk).
• Instant /stats — moved the heavy source/embedded count(*)s to a background cache (they were timing out at Cloudflare); queue counts are a single grouped scan.
• Endpoint moved to donto.org/embed (added the Caddy route) — everything points there now, not the old memories.apexpots.com/embed.

3. admin.donto.org — live embedding tracker

New /embed page (its own nav tab) + an EmbedPanel shared with /fabric, driven by lib/embed.ts (read-only donto_ro, all off the small donto_embed_queue table so it isolates distributed/contributor work from the local daemon). Shows throughput tiles (1m/5m/1h), per-target queue progress, and a Contributors table (worker, vectors done, in-flight, last-seen, live dot) — auto-refreshing, animated numbers. Verified by running a worker and watching it appear.

4. Omega — graph reasoning + deep traversal

Omega was using donto only through keyword recallMemories; the new reasoning tool was deployed but never called (0 invocations — diagnosed in the Omega-integration report). This session made it real:

• dontoQuery extended (packages/database/src/postgres/tools/dontoQuery.ts): the original 12 entity lenses (search, entity_facts, contradictions, predicate_alignment, semantic_neighbors, identity, cross_source, bitemporal, entity_reach, in/out graph, raw_sql) plus a new document lens that mirrors the admin job page — panel=overview|contradictions|evidence|provenance|entity_reach|cross_source|cooccurring|argument_graph|outgoing_graph|incoming_graph|predicate_alignment, accepting a job id, doc:… IRI, or document_id.
• next-hop suggestions on every result → induces multi-tool investigation instead of one-shot lookups (the agent loop already allows 30 steps).
• Routing fix — narrowed recallMemories to "what was said", rewrote the system-prompt to route fact/job/relationship questions to dontoQuery with a worked job-traversal example, added job/document/traverse BM25 keywords.
• Deployed to the live bot and verified — invoked the document lens inside the running container against a real job: overview returned 92 statements / 19 subjects / 51 predicates + top entities, contradictions returned 3, with next hints populated. (Caught + fixed a deploy trap: the system prompt is one template literal, and raw backticks in the first edit silently failed the build, leaving the old image running.)

5. Reports published to /research

6. Extraction

• Seven Sisters / Pleiades myth → ctx:mythology/seven-sisters — 101 claims, 25 entities, 44 predicates, 24 evidence-anchored; demonstrates paraconsistent cross-cultural holding (one star cluster, three MythVariant nodes: Greek/Aboriginal/Māori).
• Two Project Gutenberg books, in full — The Seven Sisters of Sleep (M.C. Cooke, ~110K words, 283 chunks) and Hamlet (~32K words, 81 chunks). Built a resumable, bounded-concurrency chunk driver (extract_book.py) into ctx:books/<slug>/<n>. Now complete: Hamlet 81/81, Seven Sisters 282/283.
• A real throughput diagnosis — the books were silently timing out on Hyades (the real DB signal showed only ~9 chunks landed despite the log claiming ~55, because Hyades was saturated by BEAM + the books competing). Switching the books to the GLM lane (a separate gateway) was a 16× speedup (0.4 → 6.4 chunk/min, 0 timeouts); then resumed BEAM on the now-free Hyades so both run in parallel.
• Discord — confirmed complete: discord-extract-donto-agent.service finished clean (Result=success) on 2026-06-06; ~5,540 ctx:discord/blah/* chunk-contexts, ≈582,250 claims.

7. Post-ingest processing pipeline — deep repair

The most important fix. The audit + a direct investigation found the post-processing was not happening:

• The alignment daemon was doing zero work — 0 embeds / 0 proposals / 0 applies / 0 rebuilds across 144 ticks in 24h; last real rebuild was 2026-06-04. Two compounding causes, both found in the tick log:
  1. load_skip every tick — box load pinned at ~7 (threshold 4.0) by a leaked headless chrome, 3 days 19 hours old, orphaned, eating ~3 cores. Killed it → load fell to ~1.4–2.2.
  2. DONTO_ALIGN_DRY_RUN=1 (shipped safe-by-default: propose-only, never apply/rebuild). Flipped to 0 — safe, since the daemon is I3-non-destructive (only additive/reversible derived tables).
• The context matview was 6 days stale (the audit's P0) — donto_mv_context_stats showed 23,406 contexts vs 64K+ live, and nothing was scheduled to refresh it. Refreshed it (→ 64,462 rows) and added a REFRESH MATERIALIZED VIEW CONCURRENTLY step to the daemon's periodic step_rebuild so it stays fresh automatically.
• Verified end-to-end (per the verify-it-works rule, not "started"): once load settled, the daemon's first real tick (state=idle, load 1.4) did +2,000 predicates embedded, 95 proposed, 4 applied, closure + identity rebuilt (883K → 1,004,943 closure rows), and the log confirmed rebuild_matview_refreshed. A subsequent tick added another +2,000 — sustained, not a one-off.
• The 2.84M entity-embedding backlog is intentionally the distributed fabric's job (the 200K-deep coordinator queue + contributors/GPU), not the local daemon's — the daemon handles steady-state + the periodic rebuild.

8. Operational lessons (traps caught this session)

Each of these looked fine and wasn't — caught only by checking the real signal:

• Per-chunk sub-contexts — BEAM/book claims land in ctx:.../<uuid> children; counting the exact parent context reads "stalled" while children take thousands/hour. Always aggregate LIKE 'prefix%'.
• tx_time &&-overlap reports the whole substrate as "last hour" (open-ended live ranges overlap every window) — use lower(tx_time) > now()-interval.
• pgrep -f <pattern> self-match — matches the matcher's own argv; it killed my own shell twice (the exit-144s). Match on comm or exact PID instead.
• Template-literal backticks — raw backticks in a system-prompt edit silently failed the Omega build; --force-recreate then restarted the old image. Verify the new code is actually in the running image.
• "Looked launched, wasn't working" — the daemon ticking ≠ the daemon working; the books logging chunks ≠ chunks landing. The DB row-delta is the truth, not the program's self-report.

Current live state (verified 2026-06-09)

• Substrate: ≈39.5M live statements, 64,492 contexts (matview now fresh), 893,732 predicate embeddings (rising), 1,004,943 closure rows.
• Alignment daemon: active, DONTO_ALIGN_DRY_RUN=0, embedding + applying + rebuilding; context matview auto-refreshing in each rebuild.
• Contributor stack: donto.org/help (200), donto.org/embed/health (200), ghcr.io/thomasdavis/donto-embed-worker (public/pullable), embed queue 200K ready.
• Omega: live bot running the extended dontoQuery (document lens confirmed in the deployed image).
• admin.donto.org: /embed tracker live; Contexts page now shows the true 64K.
• Books: Hamlet 81/81, Seven Sisters 282/283 (one chunk re-runnable). Discord complete (~582K).
• BEAM: running on Hyades.

Open / handoff items

• Admin observability vision (~55–60%) — the audit's roadmap: the differentiator global explorers (/arguments, /evidence, /identity), a steering/abundance surface, closure-folding discovery, and a bitemporal as-of viewer remain unbuilt. (Auth was explicitly declined.)
• Entity-embedding backlog (2.84M) — needs distributed contributors / the planned GPU; the queue is ready.
• Load-gate trade-off — when BEAM extracts at full concurrency, the daemon will yield (load > 4.0) and do its work in the gaps; raise the threshold only if post-processing should preempt extraction.
• Daily donto-matview-refresh timer still only covers subjects+predicates (the daemon now handles the context matview); fine as-is.

Compiled 2026-06-09 from the live box. All figures re-verified against donto_ro / the running services at time of writing.