# GPU acceleration for donto — a brief for an implementing friend

_Reports · 2026-06-14. A practical hand-off: what GPU work donto needs, why, and exactly how to start. Two distinct workloads — a **turnkey** embedding backfill (most of the value, infrastructure already built) and a **research-y** sheaf-training job. Written so someone with a GPU can pick up the first one in an afternoon._

---

## TL;DR

donto runs on a single 4-core CPU box. Two things are bottlenecked on it, and both are embarrassingly GPU-friendly:

1. **Embedding backfill (the big one, ~95% of the value).** The substrate has tens of millions of claims; only a tiny fraction have vector embeddings. Embedding is `bge-small` inference — **~2–4 items/sec on our CPU**, vs **hundreds–thousands/sec on a GPU**. There is already a **distributed work-queue + a turnkey GPU worker** (built for exactly this). A friend with a GPU runs the worker, it leases work, embeds, submits — no donto internals required.
2. **Sheaf model training (smaller, more bespoke).** The Stage-1 Neural Sheaf Diffusion model currently trains on CPU over small scoped subgraphs. A GPU enables substrate-scale training, bigger restriction-map families, and hyperparameter sweeps. This one needs a bit more context.

If your friend only does #1, that's a huge win and it's basically plug-and-play.

---

## Workload 1 — embedding backfill (turnkey, start here)

### What and why
donto aligns and reconciles knowledge by vector similarity (predicate folding, entity resolution, semantic recall, and now the sheaf "stalk" features). That needs an embedding per claim/predicate/entity. The model is **`BAAI/bge-small-en-v1.5`** (384-dim) — small, fast on GPU, the same model everywhere (do **not** swap it without re-embedding the whole fabric).

Coverage today (live):

| target | embedded | of total | gap |
|---|---|---|---|
| predicates | ~1.04M | ~1.04M | ~done |
| entities | ~1.0M+ | (growing) | partial |
| **claims (statements)** | **~57K** | **~42M live** | **~0.14% — the big gap** |

The claim gap is the prize: claim embeddings unlock semantic claim-recall and lift the sheaf analytic's stalks above the cheap `d=1` fallback at substrate scale. On CPU this is **tens of days**; on one decent GPU it's **hours to a couple of days**.

### The infrastructure already exists
This was built ahead of time precisely so an external GPU could plug in:

- **A coordinator** (`donto-embed-coordinator.service`, internal :7930) exposes a work queue (`donto_embed_queue`) with `FOR UPDATE SKIP LOCKED` leasing — billions-scale-safe, no double-work, ~15-min stale-lease reclaim. Endpoints: `/embed/{health,stats,lease,submit,report}`, bearer-token auth.
- **Public ingress:** `https://donto.org/embed/*` (and `memories.apexpots.com/embed/*`).
- **A turnkey worker + paste-and-go page:** `https://donto.org/help` (GPU-first). Canonical worker code is mirrored at **github `thomasdavis/donto-embed-worker`**. The worker is *dumb on purpose*: `lease → fastembed(bge-small, 384) → submit`. It never touches the DB.
- **An admin view** of contributor throughput + error reports at `admin.donto.org/embed`.

### Quick start for the friend (GPU box)
```bash
pip install fastembed-gpu          # GPU build of fastembed (CUDA)
git clone https://github.com/thomasdavis/donto-embed-worker && cd donto-embed-worker
export DONTO_EMBED_URL=https://donto.org/embed
export DONTO_EMBED_TOKEN=<token from Thomas>     # kept out of the public page
export EMBED_CUDA=1
python worker.py --concurrency 8                 # leases, embeds, submits in a loop
```
That's it — it self-paces against the queue, retries with backoff, and reports errors. Scale `--concurrency` to the GPU. Watch progress on `admin.donto.org/embed`.

### Hardware / cost
- **Any modern CUDA GPU works** — bge-small is tiny (~120 MB), so even a 6–8 GB consumer card (e.g. RTX 3060/4060) saturates throughput; VRAM is not the constraint, batch throughput is.
- Inference is short-sequence (the first ~300 tokens carry the signal — `CHUNK_TEXT_CAP` governs), so it's compute-light per item and batches beautifully.
- **Rough budget:** at even ~1,000 claims/sec a single GPU clears the ~42M backlog in ~12 GPU-hours of actual compute; realistically a day or two wall-clock with leasing overhead. A spot/preemptible cloud GPU or a friend's gaming rig overnight both work.
- ⚠️ Known gotcha (already fixed server-side): a pool-deadlock under ≥8 concurrent workers caused Cloudflare 524s — fixed 2026-06-10. If you see "lease error 524," report it; don't just hammer.

---

## Workload 2 — sheaf model training (bespoke, optional, higher-context)

### What and why
The Stage-1 **Neural Sheaf Diffusion** model (the learnable contradiction/identity operator) trains offline in `python/sheaf/` (torch). On CPU it already beats a GCN baseline on scoped subgraphs (~500 nodes), but a GPU enables:
- **Bigger subgraphs** and full-corpus sweeps instead of ≤500-node scopes.
- **Richer restriction-map families** (orthogonal/`O(d)` via Cayley, general `GL(d)`) that are heavier than the diagonal default.
- **Hyperparameter / per-task sweeps** (arg-link, identity-resolution, claim-scoring) that are slow serially on CPU.

### What's there
- `python/sheaf/train_nsd.py` — the trainer (torch). `requirements.txt` installs CPU torch; for GPU, install the CUDA torch build instead.
- Training data is **exported from the substrate** by a Rust tool (`donto analyze sheaf-export --seed arguments|frontier|incompatibilities --out FILE.json`) — leakage-free (bitemporal as-of holdout + self-feedback filter). So the friend never needs DB access: we hand them exported JSON subgraphs, they train, they hand back checkpoints.
- A checkpoint registers as a `donto_sheaf_model` row and the inference operator is **ported to Rust with a parity gate** (Rust == torch within 1e-5), so a GPU-trained checkpoint drops straight into production inference.

### The clean hand-off
1. We export a batch of subgraphs (`sheaf-export`) and send the JSON.
2. Friend runs `train_nsd.py --task arg_link --family orthogonal --in <dir>` on the GPU, sweeps hyperparameters, returns the best `checkpoint.json` + metrics.
3. We register it (`donto_sheaf_model`) and the parity-gated Rust op serves it.

No substrate access needed on the GPU side — exported files in, checkpoints out.

### Hardware
- bigger than #1 but still modest: the cochain space is `nodes × d` (small `d`); a single 12–24 GB GPU (RTX 3090/4090 or a cloud A10/L4) is plenty. This is a "run sweeps overnight" job, not a cluster job.

---

## A note on Hyades (our free LLM GPU lane — and what it can't do)

We also run **Hyades** (`holo3.1`, our own OpenAI-compatible GPU gateway, no per-token cost). It is the right home for the **LLM-judgment** GPU work and is already used by the sheaf disjointness proposer (judging whether two predicates are mutually exclusive). It scales naturally to: corpus-wide disjointness judging, LLM-assisted extraction, the citer micro-verify pass, and alignment adjudication — all free.

**But Hyades is a chat model, not an embedder** — it has no `/v1/embeddings` endpoint (verified). So it **cannot** run the bge-small backfill (Workload 1); that needs raw GPU with `fastembed-gpu`, i.e. the friend's card or the CPU fabric. Keep the two lanes straight: **embeddings → fastembed-gpu (friend) or CPU; LLM judgments → Hyades (free).**

### CPU backfill is already running
We've started the claim backfill on CPU now (2 bounded `fastembed` workers leasing from the same coordinator queue, ~700 MB total, claim coverage climbing). It's slow — that's the point of bringing in a GPU. **The friend's GPU worker leases from the exact same queue**, so it just adds throughput; nothing to coordinate. Run both and the GPU does the heavy lifting while the CPU keeps chipping.

## How to think about the split

- **#1 (embedding) is the high-leverage, low-context job** — it's already productized for contributors; point the friend at `donto.org/help` + the token and they're contributing in minutes. This is what unblocks semantic recall and substrate-scale sheaf stalks.
- **#2 (training) is collaborative + iterative** — best done as a back-and-forth (we export, they train, we register), and only worth it once #1 has populated enough claim embeddings to make `d>1` stalks meaningful.

Recommended order: **do #1 first** (it also feeds #2's stalk features), then pick up #2 as a fun collaboration.

---

## One-paragraph version to send the friend

> donto needs ~42M text snippets embedded with `bge-small` (384-dim) and our CPU box does ~2–4/sec; a GPU does hundreds–thousands. There's a ready-made distributed worker — `pip install fastembed-gpu`, clone `github.com/thomasdavis/donto-embed-worker`, set `DONTO_EMBED_URL=https://donto.org/embed` + a token + `EMBED_CUDA=1`, run it; it leases work, embeds, submits, and you can watch it on `admin.donto.org/embed`. That alone is a massive help. Separately, if you're up for it, there's a small PyTorch model (Neural Sheaf Diffusion) we'd love to train on GPU from exported data files — no DB access needed, just JSON in / checkpoints out.

_See also: the distributed-embedding PRD (genes.apexpots.com/research/donto-distributed-embedding-prd-2026-06-05) for the full worker code + protocol; [sheaf neural networks for donto](/reports/sheaf-neural-networks-for-donto) for what the training feeds._