GPU acceleration for donto — a brief for an implementing friend

Reports · 2026-06-14. A practical hand-off: what GPU work donto needs, why, and exactly how to start. Two distinct workloads — a turnkey embedding backfill (most of the value, infrastructure already built) and a research-y sheaf-training job. Written so someone with a GPU can pick up the first one in an afternoon.

TL;DR

donto runs on a single 4-core CPU box. Two things are bottlenecked on it, and both are embarrassingly GPU-friendly:

1. Embedding backfill (the big one, ~95% of the value). The substrate has tens of millions of claims; only a tiny fraction have vector embeddings. Embedding is bge-small inference — ~2–4 items/sec on our CPU, vs hundreds–thousands/sec on a GPU. There is already a distributed work-queue + a turnkey GPU worker (built for exactly this). A friend with a GPU runs the worker, it leases work, embeds, submits — no donto internals required.
2. Sheaf model training (smaller, more bespoke). The Stage-1 Neural Sheaf Diffusion model currently trains on CPU over small scoped subgraphs. A GPU enables substrate-scale training, bigger restriction-map families, and hyperparameter sweeps. This one needs a bit more context.

If your friend only does #1, that's a huge win and it's basically plug-and-play.

Workload 1 — embedding backfill (turnkey, start here)

What and why

donto aligns and reconciles knowledge by vector similarity (predicate folding, entity resolution, semantic recall, and now the sheaf "stalk" features). That needs an embedding per claim/predicate/entity. The model is BAAI/bge-small-en-v1.5 (384-dim) — small, fast on GPU, the same model everywhere (do not swap it without re-embedding the whole fabric).

Coverage today (live):

The claim gap is the prize: claim embeddings unlock semantic claim-recall and lift the sheaf analytic's stalks above the cheap d=1 fallback at substrate scale. On CPU this is tens of days; on one decent GPU it's hours to a couple of days.

The infrastructure already exists

This was built ahead of time precisely so an external GPU could plug in:

• A coordinator (donto-embed-coordinator.service, internal :7930) exposes a work queue (donto_embed_queue) with FOR UPDATE SKIP LOCKED leasing — billions-scale-safe, no double-work, ~15-min stale-lease reclaim. Endpoints: /embed/{health,stats,lease,submit,report}, bearer-token auth.
• Public ingress: https://donto.org/embed/* (and memories.apexpots.com/embed/*).
• A turnkey worker + paste-and-go page: https://donto.org/help (GPU-first). Canonical worker code is mirrored at github thomasdavis/donto-embed-worker. The worker is dumb on purpose: lease → fastembed(bge-small, 384) → submit. It never touches the DB.
• An admin view of contributor throughput + error reports at admin.donto.org/embed.

Quick start for the friend (GPU box)

pip install fastembed-gpu          # GPU build of fastembed (CUDA)
git clone https://github.com/thomasdavis/donto-embed-worker && cd donto-embed-worker
export DONTO_EMBED_URL=https://donto.org/embed
export DONTO_EMBED_TOKEN=<token from Thomas>     # kept out of the public page
export EMBED_CUDA=1
python worker.py --concurrency 8                 # leases, embeds, submits in a loop

That's it — it self-paces against the queue, retries with backoff, and reports errors. Scale --concurrency to the GPU. Watch progress on admin.donto.org/embed.

Hardware / cost

• Any modern CUDA GPU works — bge-small is tiny (~120 MB), so even a 6–8 GB consumer card (e.g. RTX 3060/4060) saturates throughput; VRAM is not the constraint, batch throughput is.
• Inference is short-sequence (the first ~300 tokens carry the signal — CHUNK_TEXT_CAP governs), so it's compute-light per item and batches beautifully.
• Rough budget: at even ~1,000 claims/sec a single GPU clears the ~42M backlog in ~12 GPU-hours of actual compute; realistically a day or two wall-clock with leasing overhead. A spot/preemptible cloud GPU or a friend's gaming rig overnight both work.
• ⚠️ Known gotcha (already fixed server-side): a pool-deadlock under ≥8 concurrent workers caused Cloudflare 524s — fixed 2026-06-10. If you see "lease error 524," report it; don't just hammer.

Workload 2 — sheaf model training (bespoke, optional, higher-context)

What and why

The Stage-1 Neural Sheaf Diffusion model (the learnable contradiction/identity operator) trains offline in python/sheaf/ (torch). On CPU it already beats a GCN baseline on scoped subgraphs (~500 nodes), but a GPU enables:

• Bigger subgraphs and full-corpus sweeps instead of ≤500-node scopes.
• Richer restriction-map families (orthogonal/O(d) via Cayley, general GL(d)) that are heavier than the diagonal default.
• Hyperparameter / per-task sweeps (arg-link, identity-resolution, claim-scoring) that are slow serially on CPU.

What's there

• python/sheaf/train_nsd.py — the trainer (torch). requirements.txt installs CPU torch; for GPU, install the CUDA torch build instead.
• Training data is exported from the substrate by a Rust tool (donto analyze sheaf-export --seed arguments|frontier|incompatibilities --out FILE.json) — leakage-free (bitemporal as-of holdout + self-feedback filter). So the friend never needs DB access: we hand them exported JSON subgraphs, they train, they hand back checkpoints.
• A checkpoint registers as a donto_sheaf_model row and the inference operator is ported to Rust with a parity gate (Rust == torch within 1e-5), so a GPU-trained checkpoint drops straight into production inference.

The clean hand-off

1. We export a batch of subgraphs (sheaf-export) and send the JSON.
2. Friend runs train_nsd.py --task arg_link --family orthogonal --in <dir> on the GPU, sweeps hyperparameters, returns the best checkpoint.json + metrics.
3. We register it (donto_sheaf_model) and the parity-gated Rust op serves it.

No substrate access needed on the GPU side — exported files in, checkpoints out.

Hardware

• bigger than #1 but still modest: the cochain space is nodes × d (small d); a single 12–24 GB GPU (RTX 3090/4090 or a cloud A10/L4) is plenty. This is a "run sweeps overnight" job, not a cluster job.

A note on Hyades (our free LLM GPU lane — and what it can't do)

We also run Hyades (holo3.1, our own OpenAI-compatible GPU gateway, no per-token cost). It is the right home for the LLM-judgment GPU work and is already used by the sheaf disjointness proposer (judging whether two predicates are mutually exclusive). It scales naturally to: corpus-wide disjointness judging, LLM-assisted extraction, the citer micro-verify pass, and alignment adjudication — all free.

But Hyades is a chat model, not an embedder — it has no /v1/embeddings endpoint (verified). So it cannot run the bge-small backfill (Workload 1); that needs raw GPU with fastembed-gpu, i.e. the friend's card or the CPU fabric. Keep the two lanes straight: embeddings → fastembed-gpu (friend) or CPU; LLM judgments → Hyades (free).

CPU backfill is already running

We've started the claim backfill on CPU now (2 bounded fastembed workers leasing from the same coordinator queue, ~700 MB total, claim coverage climbing). It's slow — that's the point of bringing in a GPU. The friend's GPU worker leases from the exact same queue, so it just adds throughput; nothing to coordinate. Run both and the GPU does the heavy lifting while the CPU keeps chipping.

How to think about the split

• #1 (embedding) is the high-leverage, low-context job — it's already productized for contributors; point the friend at donto.org/help + the token and they're contributing in minutes. This is what unblocks semantic recall and substrate-scale sheaf stalks.
• #2 (training) is collaborative + iterative — best done as a back-and-forth (we export, they train, we register), and only worth it once #1 has populated enough claim embeddings to make d>1 stalks meaningful.

Recommended order: do #1 first (it also feeds #2's stalk features), then pick up #2 as a fun collaboration.

One-paragraph version to send the friend

donto needs ~42M text snippets embedded with bge-small (384-dim) and our CPU box does ~2–4/sec; a GPU does hundreds–thousands. There's a ready-made distributed worker — pip install fastembed-gpu, clone github.com/thomasdavis/donto-embed-worker, set DONTO_EMBED_URL=https://donto.org/embed + a token + EMBED_CUDA=1, run it; it leases work, embeds, submits, and you can watch it on admin.donto.org/embed. That alone is a massive help. Separately, if you're up for it, there's a small PyTorch model (Neural Sheaf Diffusion) we'd love to train on GPU from exported data files — no DB access needed, just JSON in / checkpoints out.

See also: the distributed-embedding PRD (genes.apexpots.com/research/donto-distributed-embedding-prd-2026-06-05) for the full worker code + protocol; sheaf neural networks for donto for what the training feeds.