donto box — workers, jobs & capacity (live catalogue, 2026-06-29)

A verified-live inventory of every worker, job, daemon and queue on the donto box, the hardware it runs on, and a safe-concurrency budget (what can run together without overloading it). Built from a 6-way parallel live sweep (systemctl / docker / Temporal / psql / coordinator API), not from the docs — several doc entries were stale and are corrected inline. Raw data: donto-resources/infra/worker-catalogue-raw-2026-06-29.json.

1. The box

Correction to the docs: CLAUDE.md still says 4 vCPU / 16 GB / GCloud 34.135.161.87. The live box is 8 vCPU / 22 GB / OVH 15.235.185.42, with much larger disks.

This box is multi-tenant. Besides donto it runs: the Omega Discord stack (bot/web/jobs/mongo/postgres/deployer), MobTranslate (web/tts/pg), the full tpmjs.com stack (web/playground/tutorial + 2 Deno sandboxes), and the observability stack (OpenObserve/OTel/Umami). That is the always-on baseline every heavy job competes with.

2. ⚠️ The #1 capacity problem right now: a runaway health-view poll

The single biggest thing overloading the box is not a legitimate worker. A donto-role host service polls dontosrv's substrate-health endpoint — select row_to_json(donto_v_substrate_health.*) — on a long-lived connection. That view full-scans the 42M-row substrate (~20+ minutes each), and because the donto role has no statement_timeout, repeated polls pile up: at sweep time there were 9 concurrent copies, the oldest 24 minutes, driving load to 25–33 on 8 vCPU with ~18 GB swap in use.

This single pile is the cause of: the embedding contributor's submit timeouts (Cloudflare 504/524), slow builds, and the test-runner worker-start timeouts seen all session.

Relieved (cancelled the 18 stuck read-only queries — safe, no data touched), but it recurs because the poller keeps issuing it. Recommended durable fix (needs a decision — touches the substrate):

1. Make donto_v_substrate_health cheap — back it with reltuples + the existing matviews instead of live count(DISTINCT …) and a SUM over the 6.3M-row donto_contradiction_pressure matview (the same reltuples trick already used by the admin overview). Migration 0147_substrate_health_fast started this; it needs finishing for today's row counts/load.
2. Throttle / cache the poller (it does not need sub-minute fresh exact counts).
3. Belt-and-suspenders: a generous statement_timeout for that read path (pg_dump sets its own =0, so backups are unaffected; the align daemon already self-caps).

3. Worker / job catalogue (by function)

All VERIFIED active/running unless marked. 30 donto systemd units, 0 failed.

3.1 Substrate & core APIs

3.2 Extraction (Rust + Temporal fleet — replaced donto-agent-beam)

The doc's headline donto-agent-beam.service is gone/inactive. Extraction is now a Temporal Rust pipeline:

3.3 Embedding fabric (bge-small-en-v1.5 / 384-dim)

3.4 Alignment

3.5 Memory

3.6 Web · Bot · Observability · Research

4. Inference lanes — the real constraint

Extraction/adjudication/research throughput is gated by inference capacity, not the box. Live state:

Net usable inference right now = Hyades + codex. Everything that depends on GLM is degraded until 2026-07-01.

5. Queues & backlogs (the work waiting)

6. Safe-concurrency budget

Baseline (idle, healthy): Postgres + Temporal(java) + OTel + OpenObserve + ~15 web/python services ≈ load 4–8 at rest on 8 vCPU. RAM is comfortable (16 GB available); swap is heavily used but cold.

The CPU-bound contenders (each wants a core under load): donto-align-daemon (bge-small), donto-extract-citer (bge-small), donto-memory-rerank (cross-encoder), embed_memory.py backfill, local embed services, and any Next.js build. The extraction lane workers are network/inference-bound (the CPU cost is the citer+ingest, not the model), so extraction throughput is capped by the lane, not the box.

Empirical (measured this session):

• one next build → load +3–6; four parallel builds → load ~25 (serialize them).
• the test runner (vitest) can't boot its workers if load > ~12 (60 s startup timeout).
• the runaway substrate_health poll alone → load +20.

Rules of thumb:

1. Fix §2 first. With the health-view pileup gone, the legit fleet sits at ~load 4–8 and everything else has room.
2. Serialize heavy CPU jobs. Run at most one of {a build, an embed_memory backfill, a CPU-heavy extraction batch} at a time. Two+ pushes load > 15 and starves the daemons.
3. Extraction scales on the lane, not the box. You can run many extract-hyades/extract-codex tasks concurrently against the inference lanes with little box load — as long as the citer + ingest keep up (those are the CPU cost). Hyades parallelizes (~5 concurrent cleanly).
4. Embedding backlog is currently zero, so embed CPU is free — but a big new backfill is single-process CPU-bound (~2–4 chunk/s); budget it as one heavy job and prefer the off-box GPU contributors for volume.
5. Keep ~2 cores free for Postgres + Temporal at all times; they back every worker.

7. Issues found — prioritized action list

1. ⚠️ substrate_health poll overload (§2) — relieved, recurs. Make the view cheap + throttle the poller. Biggest single win.
2. ⚠️ Duplicate Postgres on the same data dir. donto-pg-pre-pgvector is a second live postmaster bind-mounted to the same /mnt/donto-data/pgdata as donto-pg (verified identical PG_VERSION). Two postmasters on one data dir is a corruption hazard — stop/remove it. (No published port, so nothing reaches it — but I3 "never lose data" says fix this.)
3. dontopedia-worker crash-looping. Container "Up" but fatal-looping on TEMPORAL_ADDRESS=10.128.0.3:7233 (a dead GCloud IP). Repoint to temporal:7233.
4. Extraction stalled — 352K backlog, scheduler not draining ("connection not ready" restarts; MAX_STARTS_PER_TICK=4). Route to Hyades+codex once the box is healthy.
5. Memory reconsolidation backlog 44,526 not draining (worker inactive).
6. GLM lane exhausted until 2026-07-01 — omega bot, research, tribunal, align-adjudication all degraded.
7. 7 of 8 matviews drift (no refresh job) — incl. donto_contradiction_pressure (1.7 GB), subject_count (1.26 GB), predicate_idf. Only donto_mv_context_stats auto-refreshes.
8. donto_align_daemon_tick history stopped writing 2026-06-22 (status table still ticks).
9. omega-vm-caddy-1 stuck "Created" (port-80 conflict with host caddy) — harmless leftover.