# donto — Operations Session Report (through 2026-06-09)

*A complete record of the work done this session: contributor infrastructure, the distributed
embedding fabric, Omega's graph-reasoning upgrade, five research reports, two book extractions, and
a deep repair of the post-ingest processing pipeline. Every claim here was verified live against the
running box; artifacts and current state are listed at the end.*

---

## Executive summary

This session shipped, in order:

1. **A contributor on-ramp** — `donto.org/help`, a one-command public Docker image, an embed-worker kit, and a comprehensive `help-embed.md` so anyone (or their Claude Code) can lend compute in seconds.
2. **The distributed embedding fabric, completed** — the coordinator now embeds *every* aspect (predicates + entities + memory chunks), self-refills its queue, and reports stats instantly; moved to the clean `donto.org/embed` endpoint.
3. **A live embedding tracker on admin.donto.org** (`/embed`) so contributions are visible.
4. **Omega can now reason over and deeply traverse the substrate** — a 12-lens `dontoQuery` tool extended with a job/document lens that mirrors the admin job page, next-hop suggestions to drive multi-tool investigation, and a routing fix that took it from *deployed-but-dormant* to actually used. Deployed to the live bot and verified.
5. **Five /research reports** — the alignment contributor guide, the Omega-integration audit, the full API reference, the extraction checkpoint, and the admin observability audit.
6. **Extraction** — the Seven Sisters myth, two full Project Gutenberg books (*The Seven Sisters of Sleep* and *Hamlet*), and confirmation the discord corpus is complete (~582K claims).
7. **A deep repair of the post-ingest processing pipeline** — the alignment daemon had been doing *zero* work for days (dry-run + a leaked process starving its load gate); the context matview was 6 days stale. All fixed, scheduled, and verified producing real work.

---

## 1. Contributor on-ramp — `donto.org/help`

The goal: let friends donate spare CPU/GPU to embed donto's backlog with zero friction.

- **`https://donto.org/help`** — a focused, single-purpose page (rewritten through 5 refinement passes from an earlier bloated version): run an embedding worker, nothing else. Two headline paths — a one-command Docker run and a paste-into-Claude-Code block — plus get-the-token, verify-it-worked, and a tight FAQ. Caddy serves it for both `/help` and `/help/` (fixed an earlier bare-`/help` blank-page bug). Backing files in `/srv/donto-help/` (`index.html`, `claude.txt`, `help-embed.md`).
- **One-command Docker image** — `ghcr.io/thomasdavis/donto-embed-worker` (public, anonymously pullable, built via GitHub Actions in repo `thomasdavis/donto-embed-worker`). `docker run --rm -e DONTO_EMBED_TOKEN=… ghcr.io/thomasdavis/donto-embed-worker` runs one worker per core, GPU-aware. **Proven**: pulled anonymously and embedded **2,352 vectors** in one run.
- **`help-embed.md`** — a 16-section comprehensive guide (Docker / Claude-Code / bare-Python / compose, GPU, fleet, verify, env reference, how-it-works, troubleshooting, FAQ, full worker source, HTTP API). Served at `donto.org/help/help-embed.md`.
- **A real bug caught here**: Cloudflare 403s the default `python-urllib` User-Agent — the worker now sends a `curl/8` UA. Without this, *every* contributor's worker would have silently failed.

## 2. Distributed embedding fabric — completed

`donto-embed-coordinator.service` (the lease→embed→submit work queue for remote workers):

- **Wired predicate + entity providers** alongside the original memory-chunk one, reusing the exact text/signature builders from `embed.py` / `entity_embed.py` (byte-for-byte vector parity with the daemon). `/embed/health` now reports `["memory_chunk","predicate","entity"]`. Round-trip (lease→bge-small→submit→upsert) verified for all three.
- **Self-topping queue** — an auto-enqueue task keeps each target's pending count topped from the backlog, so contributors never run dry (currently 200K queued: 50K predicate + 50K entity + 100K chunk).
- **Instant `/stats`** — moved the heavy source/embedded `count(*)`s to a background cache (they were timing out at Cloudflare); queue counts are a single grouped scan.
- **Endpoint moved to `donto.org/embed`** (added the Caddy route) — everything points there now, not the old `memories.apexpots.com/embed`.

## 3. admin.donto.org — live embedding tracker

New **`/embed`** page (its own nav tab) + an `EmbedPanel` shared with `/fabric`, driven by `lib/embed.ts` (read-only `donto_ro`, all off the small `donto_embed_queue` table so it isolates *distributed/contributor* work from the local daemon). Shows throughput tiles (1m/5m/1h), per-target queue progress, and a Contributors table (worker, vectors done, in-flight, last-seen, live dot) — auto-refreshing, animated numbers. Verified by running a worker and watching it appear.

## 4. Omega — graph reasoning + deep traversal

Omega was using donto only through keyword `recallMemories`; the new reasoning tool was deployed but **never called** (0 invocations — diagnosed in the Omega-integration report). This session made it real:

- **`dontoQuery` extended** (`packages/database/src/postgres/tools/dontoQuery.ts`): the original 12 entity lenses (search, entity_facts, contradictions, predicate_alignment, semantic_neighbors, identity, cross_source, bitemporal, entity_reach, in/out graph, raw_sql) **plus a new `document` lens** that mirrors the admin job page — `panel=overview|contradictions|evidence|provenance|entity_reach|cross_source|cooccurring|argument_graph|outgoing_graph|incoming_graph|predicate_alignment`, accepting a job id, `doc:…` IRI, or document_id.
- **`next`-hop suggestions** on every result → induces multi-tool investigation instead of one-shot lookups (the agent loop already allows 30 steps).
- **Routing fix** — narrowed `recallMemories` to "what was *said*", rewrote the system-prompt to route fact/job/relationship questions to `dontoQuery` with a worked job-traversal example, added `job/document/traverse` BM25 keywords.
- **Deployed to the live bot and verified** — invoked the `document` lens inside the running container against a real job: overview returned 92 statements / 19 subjects / 51 predicates + top entities, contradictions returned 3, with `next` hints populated. (Caught + fixed a deploy trap: the system prompt is one template literal, and raw backticks in the first edit silently failed the build, leaving the old image running.)

## 5. Reports published to /research

| Report | What |
|---|---|
| `donto-alignment-contribute-2026-06-06` | The alignment engine + how to contribute compute (the worker kit). |
| `omega-donto-integration-2026-06-07` | Grounded audit of how Omega uses donto; found `dontoQuery` dormant (0 calls) + the recall-quality gaps; prioritized fixes (drove §4). |
| `donto-api-reference-2026-06-07` | Full developer API reference — all five HTTP services (~70 dontosrv endpoints) + MCP, live-probed, 5 corrections folded in. |
| `donto-extraction-checkpoint-2026-06-07` | Live status of the extraction stack (donto-agent, providers, BEAM throughput, the per-chunk-sub-context gotcha). |
| `admin-donto-observability-audit-2026-06-09` | 6-agent audit of admin.donto.org — verdict ~55–60% of the full vision; two P0s (no auth; stale matview); ordered roadmap. |

## 6. Extraction

- **Seven Sisters / Pleiades myth** → `ctx:mythology/seven-sisters` — 101 claims, 25 entities, 44 predicates, 24 evidence-anchored; demonstrates paraconsistent cross-cultural holding (one star cluster, three `MythVariant` nodes: Greek/Aboriginal/Māori).
- **Two Project Gutenberg books, in full** — *The Seven Sisters of Sleep* (M.C. Cooke, ~110K words, 283 chunks) and *Hamlet* (~32K words, 81 chunks). Built a resumable, bounded-concurrency chunk driver (`extract_book.py`) into `ctx:books/<slug>/<n>`. **Now complete: Hamlet 81/81, Seven Sisters 282/283.**
- **A real throughput diagnosis** — the books were silently *timing out* on Hyades (the real DB signal showed only ~9 chunks landed despite the log claiming ~55, because Hyades was saturated by BEAM + the books competing). Switching the books to the **GLM lane** (a separate gateway) was a **16× speedup** (0.4 → 6.4 chunk/min, 0 timeouts); then resumed BEAM on the now-free Hyades so both run in parallel.
- **Discord** — confirmed complete: `discord-extract-donto-agent.service` finished clean (Result=success) on 2026-06-06; ~5,540 `ctx:discord/blah/*` chunk-contexts, **≈582,250 claims**.

## 7. Post-ingest processing pipeline — deep repair

The most important fix. The audit + a direct investigation found the post-processing was **not happening**:

- **The alignment daemon was doing zero work** — 0 embeds / 0 proposals / 0 applies / 0 rebuilds across 144 ticks in 24h; last real rebuild was **2026-06-04**. Two compounding causes, both found in the tick log:
  1. **`load_skip` every tick** — box load pinned at ~7 (threshold 4.0) by a **leaked headless chrome, 3 days 19 hours old**, orphaned, eating ~3 cores. Killed it → load fell to ~1.4–2.2.
  2. **`DONTO_ALIGN_DRY_RUN=1`** (shipped safe-by-default: propose-only, never apply/rebuild). Flipped to `0` — safe, since the daemon is I3-non-destructive (only additive/reversible derived tables).
- **The context matview was 6 days stale** (the audit's P0) — `donto_mv_context_stats` showed 23,406 contexts vs 64K+ live, and nothing was scheduled to refresh it. Refreshed it (→ 64,462 rows) **and added a `REFRESH MATERIALIZED VIEW CONCURRENTLY` step to the daemon's periodic `step_rebuild`** so it stays fresh automatically.
- **Verified end-to-end** (per the verify-it-works rule, not "started"): once load settled, the daemon's first real tick (state=`idle`, load 1.4) did **+2,000 predicates embedded, 95 proposed, 4 applied, closure + identity rebuilt** (`883K → 1,004,943` closure rows), and the log confirmed `rebuild_matview_refreshed`. A subsequent tick added another +2,000 — sustained, not a one-off.
- **The 2.84M entity-embedding backlog** is intentionally the *distributed fabric's* job (the 200K-deep coordinator queue + contributors/GPU), not the local daemon's — the daemon handles steady-state + the periodic rebuild.

## 8. Operational lessons (traps caught this session)

Each of these *looked* fine and wasn't — caught only by checking the real signal:

- **Per-chunk sub-contexts** — BEAM/book claims land in `ctx:.../<uuid>` children; counting the exact parent context reads "stalled" while children take thousands/hour. Always aggregate `LIKE 'prefix%'`.
- **`tx_time` `&&`-overlap** reports the *whole* substrate as "last hour" (open-ended live ranges overlap every window) — use `lower(tx_time) > now()-interval`.
- **`pgrep -f <pattern>` self-match** — matches the matcher's own argv; it killed my own shell twice (the exit-144s). Match on `comm` or exact PID instead.
- **Template-literal backticks** — raw backticks in a system-prompt edit silently failed the Omega build; `--force-recreate` then restarted the *old* image. Verify the new code is actually in the running image.
- **"Looked launched, wasn't working"** — the daemon ticking ≠ the daemon working; the books logging chunks ≠ chunks landing. The DB row-delta is the truth, not the program's self-report.

## Current live state (verified 2026-06-09)

- **Substrate**: ≈39.5M live statements, **64,492 contexts** (matview now fresh), 893,732 predicate embeddings (rising), 1,004,943 closure rows.
- **Alignment daemon**: `active`, `DONTO_ALIGN_DRY_RUN=0`, embedding + applying + rebuilding; context matview auto-refreshing in each rebuild.
- **Contributor stack**: `donto.org/help` (200), `donto.org/embed/health` (200), `ghcr.io/thomasdavis/donto-embed-worker` (public/pullable), embed queue 200K ready.
- **Omega**: live bot running the extended `dontoQuery` (document lens confirmed in the deployed image).
- **admin.donto.org**: `/embed` tracker live; Contexts page now shows the true 64K.
- **Books**: Hamlet 81/81, Seven Sisters 282/283 (one chunk re-runnable). Discord complete (~582K).
- **BEAM**: running on Hyades.

## Open / handoff items

- **Admin observability vision (~55–60%)** — the audit's roadmap: the differentiator *global* explorers (`/arguments`, `/evidence`, `/identity`), a steering/abundance surface, closure-folding discovery, and a bitemporal as-of viewer remain unbuilt. (Auth was explicitly declined.)
- **Entity-embedding backlog (2.84M)** — needs distributed contributors / the planned GPU; the queue is ready.
- **Load-gate trade-off** — when BEAM extracts at full concurrency, the daemon will yield (load > 4.0) and do its work in the gaps; raise the threshold only if post-processing should preempt extraction.
- **Daily `donto-matview-refresh` timer** still only covers subjects+predicates (the daemon now handles the context matview); fine as-is.

---

*Compiled 2026-06-09 from the live box. All figures re-verified against `donto_ro` / the running services at time of writing.*