# Faster Inference for donto: From ~8 Months to ~4 Weeks (Free) to ~13 Days (GPU)

*The substrate is not the bottleneck — token generation is. A measured, prioritized plan to speed up donto's extraction and embedding firehose at $0 per-token, on the gateway you already own.*

2026-06-05

## 1. The bottleneck, measured

donto holds ~39.5M live statements across ~23K contexts and ingests new claims fine — the firehose-bulk path proved 251 chunks/s into the substrate via `COPY`. **The substrate is not the gate. Inference is.** Three measured numbers define the problem:

| Workload | Measured rate | Implication |
|---|---|---|
| **Extraction** (donto-agent, holo3.1, conc=5, 6-pass) | **~60 chunks/hr** | BEAM-10M's ~359K pending chunks → **~8 months** |
| **Hyades gateway** (holo3.1, aggregate) | plateaus **~0.9 req/s** at 60–100 in-flight; 27% HTTP-524 at 150 | The serving ceiling — but you're pulling only ~0.1–0.25/s |
| **Embedding** (bge-small fp32, CPU, 4 vCPU) | **~2–4 chunk/s** | ~352K BEAM + ~2.8M entity backlog = **~10–22 days CPU** |

The decisive fact: **Hyades is the user's OWN OpenAI-compatible gateway** (`hyades.gnostr.cloud/v1`, holo3.1, no per-token cost). That changes everything. Most inference advice assumes you rent tokens from someone else; here the serving stack is yours to reconfigure. Server-side wins — prefix caching, the injected system prompt, speculative decoding, batch ceilings — are **free leverage you control**, not vendor knobs you beg for.

And the live concurrency probe is unambiguous: at the current `--concurrency 5` you are using **~11–28% of a gateway you already own.**

```
Hyades holo3.1 concurrency probe (2026-06-05, ~800-tok calls)
  N=5   → 0.47 ok/s   (you are HERE; service pulls ~0.1-0.25/s end-to-end)
  N=30  → 0.73 ok/s
  N=60  → 0.92 ok/s   ← aggregate ceiling, 0 errors
  N=100 → 0.91 ok/s   ← plateaued
  N=150 → 0.87 ok/s   but 41/150 = 27% HTTP-524 (saturated, Cloudflare cliff)
```

The aggregate req/s is **flat** above ~60 in-flight while per-request latency climbs — the classic signature of a continuous-batching server that's GPU/KV-bound, not a scheduling bug. You have ~3–9× of already-paid-for headroom sitting idle.

## 2. Where the time actually goes

A single BEAM chunk costs `passes × (prefill + decode)`:

- **Passes (6).** One broad pass + up to 5 gleaning nudges (`max_passes=6`, `dry_min=8`, stop after 2 dry passes). Each pass is a full round-trip. On a 359K-chunk firehose, passes 3–6 are deep diminishing returns — the broad pass plus one nudge captures the bulk; the showcase 511→1,915→3,227 numbers were **coverage-maximizing on one article**, not throughput-optimizing on a corpus.
- **The shared ~3K-token prompt, re-prefilled every pass.** `extract_broad.txt` + injection bookends is identical on every chunk *and* every pass. With 6 passes × 359K chunks, that prefix is being recomputed millions of times needlessly — and probes confirm **zero `cached_tokens`**: prefix caching is OFF.
- **A ~1120-token Hyades-injected system prompt, on every call.** A 1-char user message bills 1120 prompt tokens on holo3.1. donto supplies its own prompt, so this is a **~27% prefill tax** paid every pass.
- **The thread header is not KV reuse.** `X-Hyades-Thread` *re-prefills the whole thread* per nudge (measured: warm nudge 10.97s ≥ cold 10.56s, full 2052 prompt_tokens billed). The gleaning loop's core efficiency premise — "nudges are cheap" — is **false as currently served.**
- **Reasoning overhead.** Extraction needs almost none (codex's 511-fact pass spent 587 reasoning tokens); reasoning tokens are pure latency and cause the "200 + empty content" failure when they eat the budget.
- **Embedding: CPU, sequence-length-bound.** Measured: a 512-token chunk runs 0.8/s (bs=1) → 2.8/s (bs=128); a 12-token chunk runs **43–71/s** on the *same model*. The cost is tokens, not the model.

One myth to kill up front, verified in code: **`provider.rs`'s `serialized: true` flag is dead code** — declared, never read. The work loop already spawns N genuinely-concurrent SKIP-LOCKED tokio workers. **Concurrency is purely the CLI arg; no Rust edit is needed to raise it.** The widely-repeated "fix serialized:true" advice is a red herring; the real fix is even simpler.

## 3. The plan, prioritized

Every free/GPU lever below is **$0 per-token, subscription-only.** Only the flagged Cerebras/RunPod bursts cost cash, and those are explicit user decisions for the one finite proof-gated run — never the default firehose.

| # | Lever | Speedup (on real numbers) | Effort | Cost | Risk | Tier |
|---|---|---|---|---|---|---|
| 1 | **Raise concurrency 5 → 35** (CLI arg; `serialized` is dead code) | **~3×** (60 → ~180 chunks/hr) | trivial (~5 min + ramp) | $0 | low — own gateway; 524-cliff only at ≥150 | FREE |
| 2 | **Cut gleaning passes 6 → 2** + deferred high-yield re-sweep | **~3×** (multiplies with #1) | trivial | $0 | low — coverage recovered later | FREE |
| 3 | **Enable + VERIFY real prefix caching** (vLLM `--enable-prefix-caching` / SGLang RadixAttention) | **~2–4×** | low→med | $0 | med — needs Hyades-host control | FREE |
| 4 | **Guided/constrained JSON decoding** (vLLM `guided_json` / xgrammar) | ~1.2–1.5× + kills ramble & injection drift | medium | $0 | low | FREE |
| 5 | **Drop the ~1120-tok injected system prompt** (raw model alias) | ~1.25–1.35× prefill | low | $0 | low | FREE |
| 6 | **N-gram / prompt-lookahead speculative decoding** (`method=ngram`) | **~1.6–2.0×**, output-identical | trivial flag | $0 | none to output | FREE* |
| 7 | **Drop `X-Hyades-Thread`; re-send cache-aligned stack** | makes nudges genuinely cheap (the gleaning premise) | low (Rust) | $0 | low | FREE |
| 8 | **Embedding: truncate to ~96 tok** + batch=128 + threads=2 | **~8–12×** (2–4/s → ~20–30/s) | ~10 min | $0 | low — validate hit@10 | FREE |
| 9 | **Add glm-5.1@Hyades as 2nd lane**; EXCLUDE nemotron3-omni | helps saturate the ~0.9 ceiling + predicate diversity | low | $0 | low | FREE |
| 10 | **Pin `reasoning_effort='none'`** for the extract lane | ~nil on holo3.1; ~1.3–2× on reasoning lanes | trivial | $0 | low | FREE |
| 11 | **GPU embedding worker** vs the live coordinator (`fastembed-gpu`, same bge-small) | **~100–1000×** (2–4/s → ~1,000–3,000/s) | ~1–2h | GPU only | low — vectors compatible, no re-embed | GPU |
| 12 | **Serve official quantized holo3.1** (FP8 / AWQ-INT4 / NVFP4 per arch) + FP8 KV | **~1.5–2× (FP8) / ~2.7–3.3× (INT4)** decode at conc=5 | low (download+flag) | GPU only | low — CITER backstops drift | GPU |
| 13 | **vLLM 7–8B extractor on GPU** as a 2nd lane parallel to Hyades | +~600 chunks/hr (3090) / +~1200 (4090) | 1–2 days | ~$700 used 3090 | med — separate host | GPU |
| 14 | **Distill a 3B extractor** from holo3.1's logged (source→facts) pairs | **~5–15×** over holo3.1; removes the gateway wall | medium | one-time GPU-hrs | med — validate vs CITER | BIGGER BET |
| 15 | **EAGLE-3 self-spec head** for holo3.1's base | **~2.8–4.8×**, output-identical | med→high | $0 if head exists | med — needs base-matched head | BIGGER BET |
| 16 | **Cerebras / RunPod-A100 burst** — finite 2,800-row corpus ONLY | ~10–20× (600–1,200 chunks/hr) | low→med | **CASH** (per-token / ~$50–130) | high vs budget — user decision | BIGGER BET |

\* #3, #6, #7 (and the holo3.1 quant/spec wins) are only actionable **if the user controls the holo3.1 inference engine** behind Cloudflare. The model list shows `openrouter/free` + `z-ai:free` passthroughs alongside `owned_by:hyades` models (holo3.1, glm-5.1, nemotron) — the latter look self-hosted. If holo3.1 is itself proxied, these Hyades-side levers vanish and effort shifts entirely to the donto-agent-side wins (#1, #2, #4, #7, #9, #10) + embedding.

### FREE / this-week wins (the heart)

**#1 — Raise concurrency 5 → 35.** The single highest-ROI move and entirely the user's dial. It's one token in `donto-agent-beam.service`'s `ExecStart` (`--concurrency 5` → `35`). It converts ~28% gateway utilization into ~85%. Ramp **5 → 20 → 35**, measuring the *real* done-count delta over 15 min (not the log's self-reported rate, per CLAUDE.md §0.4); back off to 25 if 524s exceed ~5%. The 524-cliff lives at ≥150 in-flight, far above 35. **conc=5 is the user's explicit setting — it gates nearly every other free win until raised.**

**#2 — Cut gleaning passes 6 → 2.** `--max-passes 2` + raise `dry_min` to ~10 so the single nudge stops fast on thin chunks. Tag high-yield chunks (an `n_claims` threshold in `queue.rs`) to re-sweep when a quality lane (codex-spark, or the GPU) is free. You keep coverage where it pays without paying for it everywhere.

**#3 — Real prefix caching.** donto's ~3K shared prompt is the textbook best case — identical across every chunk and pass. The flag is free; verify it actually fires by calling the same large prompt twice and confirming `usage.prompt_tokens_details.cached_tokens > 0`. **Do not trust the flag.** SGLang's RadixAttention is the strongest fit for the multi-turn nudge pattern if a backend swap is on the table.

**#4 — Guided JSON decoding.** Forces dense, parseable fact arrays, kills markdown ramble, near-zero parse failures, and **structurally defeats the conversation prompt-injection drift** (the model literally cannot emit a coding tutorial if the grammar permits only a fact array).

**#5 — Kill the injected system prompt.** Expose a `holo3.1-raw` alias with an empty system prompt for the extract lane. ~27% less prefill, every pass, and it compounds with #3.

**#6 — N-gram speculative decoding.** Output is 28% repetitive JSON scaffold (`{"s":...,"p":...,"o":...,"c":...}` per line) + ~70% verbatim-source literal objects + ~55 verbatim anchor spans/chunk — ideal for context-wide n-gram drafting. `method:"ngram", num_speculative_tokens:5, prompt_lookup_max:4`. Output-identical, no VRAM, no training, consumes no subscription cap.

**#7 — Drop the thread; re-send a cache-aligned stack.** Once #3 is on, re-send `system(prompt) + user(source) + assistant(prior) + user(nudge)` — content-based caching hits the cached prefix and prefills only the new tail. Cheaper than the thread re-prefill *and* stateless/resumable (better for the SKIP-LOCKED queue).

**#8 — Embedding truncation.** The bottleneck is sequence length, not the model. You already proved raising `CHUNK_TEXT_CAP` gave **zero** recall gain — signal lives in the first ~300 tokens. Truncate to ~96–128 tok (~384–512 chars) before `model.embed()`, set `batch_size=128, threads=2`, and pause the extractor during backfill to de-contend the 4 cores. Validate `_s` pref-recall hit@10 is unchanged.

**#9 — glm-5.1 as a 2nd lane.** Works (~396 comp-tok in ~9s on the same uncapped gateway); adds predicate diversity and helps two model queues push the ~0.9 ceiling. **EXCLUDE nemotron3-omni** — live test: HTTP 200 + empty content + `usage:null` = zero facts. Its "0.18s" is the speed of returning nothing. Add an empty-content liveness check to the router probe.

**#10 — `reasoning_effort='none'`.** ~nil on holo3.1 (honest — the probe showed no visible effect), but dominant the moment you rotate to a reasoning lane (glm-5.1 ~3.9s/call) and it mitigates the "200 + empty" failure. Keep a separate `xhigh` profile for showcase extractions.

### GPU wins (when the card lands)

**#11 — GPU embedding worker — the highest-ceiling, clearest GPU win.** The distributed fabric was *built* for this: `donto_embed_queue` (`FOR UPDATE SKIP LOCKED`) + the live coordinator at `:7930` (`memories.apexpots.com/embed/*`). Install `fastembed-gpu` (CUDA EP) — **same bge-small, same 384-dim, vectors compatible, NO re-embed.** Lease batches, embed, submit; the worker never touches the DB. On GPU, drop the truncation — full 512-tok chunks are nearly free, so the GPU run can maximize recall. Clears the ~3.15M backlog in minutes-to-low-hours.

> **Correction to delete from CLAUDE.md §0.5:** "needs an int8-quantized ONNX model" is a **misconception**. fastembed already loads qdrant's `-onnx-q` int8 model, and this CPU has AVX2 but **no AVX-512-VNNI** — so int8 gets no fast path and re-quantizing buys ~0%. The real embed levers are **truncation (free)** and **GPU**.

**#12 — Quantized holo3.1.** holo3.1 = H Company's **Holo-3.1-35B-A3B** (35B total, ~3B active, Qwen-3.5-derived MoE, 256K ctx). H Co. **ships official FP8 / Q4_K_M-GGUF (21.3 GB) / NVFP4 checkpoints** — download-and-flag, not a project. At conc=5 (low-batch, memory-bandwidth-bound) + A3B sparsity, weight quantization wins exactly where you live. Add `--kv-cache-dtype fp8` regardless (~2× KV space for the long-context nudge pattern, ~free). Format is hardware-gated: Hopper→FP8, Ada/Ampere→AWQ-INT4, Blackwell→NVFP4 — **but A/B NVFP4 vs AWQ-INT4 first** on desktop Blackwell (FP4 underperforms in vLLM today). The always-on CITER backstops INT4 drift, so donto can quant *more* aggressively than typical.

**#13 — vLLM 7–8B extractor as a 2nd lane.** A used **3090 (~$700)** is the value king — skip the 4090 (~2× throughput for ~3× price). Enable prefix-caching (the ~7K prompt is identical → broad passes near-free). The GPU goes in a *separate physical box* (the e2-standard-4 can't take a consumer card) and joins via the same SKIP-LOCKED queue/coordinator pattern. Removes the Hyades concurrency wall — you control the vLLM batch.

### Bigger bets

**#14 — Distill a 3B extractor.** The SFT corpus is **already accumulating** as (source→facts) pairs at `/opencode/logs/beam-conv/<thread>.log`. Once the GPU lands, LoRA-fine-tune Qwen2.5-3B / Llama-3.2-3B-Instruct on the teacher pairs, serve with vLLM (hundreds of tok/s vs holo3.1's ~30 effective). CITER backstops quality. **ACTION NOW ($0): stop pruning `/opencode/logs/beam-conv/`** so the dataset survives.

**#15 — EAGLE-3.** The biggest spec-decode ceiling (~2.8–4.8×, output-identical) but needs a draft head matched to holo3.1's base; train with SpecForge on the GPU if no community head drops in. Use n-gram first; layer EAGLE-3 after.

**#16 — Cerebras / RunPod burst.** ~10–20× (600–1,200 chunks/hr), finishes the **finite 2,800-row frontier corpus in hours**. But Cerebras is currently 402-capped/offline and a top-up is per-token spend; a RunPod Community A100 (own vLLM, `--max-passes 3`) is ~$50–130/run. **Cash — a deliberate user decision for the one proof-gated run, never the default firehose, never spend silently.**

## 4. A concrete throughput model

Starting from the measured **60 chunks/hr**:

```
EXTRACTION
  baseline (conc=5, 6-pass, holo3.1) ......................   60 chunks/hr   → BEAM-10M ~249 days (~8 mo)

  FREE / this-week (compounding):
   × #1 concurrency 5→35  (~3.0×) ........................   180 chunks/hr
   × #2 passes 6→2        (~3.0×) ........................   540 chunks/hr
   × #4+#5 guided JSON + prompt trim (~1.3×) .............  ~700 chunks/hr   ← realistic do-this-week
   ─ honest conservative band (overlap/measurement) ......  ~250–540 chunks/hr → BEAM ~4–8 WEEKS,  $0

  + IF user controls holo3.1 engine (compounding, free):
   × #3 prefix caching    (~2–4×)
   × #6 n-gram spec-dec   (~1.6–2.0×)        → another large multiplier on top

  GPU (vLLM 7–8B 2nd lane parallel to Hyades):
   combined ~19× ........................................  → BEAM-10M in ~13 DAYS,  $0/token
  GPU + distilled 3B (~5–15× over holo3.1) ..............  → removes the gateway wall entirely

EMBEDDING  (~352K BEAM + ~2.8M entity ≈ 3.15M backlog)
  baseline CPU fp32 ....................   2–4/s   → ~10–22 days
  + #8 truncate to ~96 tok (~8–12×) ....  ~20–30/s → BEAM 352K in ~3–5h CPU
  + batch128 + threads2 + pause extr. ..  ~30–50/s → 3.15M in ~18–29h CPU
  GPU worker (#11) .....................  ~1,000–3,000/s → 3.15M in MINUTES-to-low-HOURS
```

**Do-this-week target (free, no hardware): ~250–540 chunks/hr → BEAM-10M from ~8 months to ~4–8 weeks.** With the GPU as a second lane: **~13 days.**

**Honest ceiling:** every free extraction multiplier is bounded by the user's **conc=5 choice** until raised, and by the Hyades **~0.9 req/s plateau** (~1,620 chunks/hr at 2 passes) until the GPU gives a lane the user fully controls.

## 5. Embedding fast-path

The embedding backlog (~352K BEAM chunks + ~2.8M entity vectors) has **no subscription ceiling** — it's pure compute you own — making it the lowest-risk place to win, in two stages:

1. **Today, free, ~10 min — truncate.** In `embed/embed_memory.py` and `donto-align/entity_embed.py`: hard-cap text to ~384 chars *before* `model.embed()`, set `batch_size=128`, construct `TextEmbedding(..., threads=2)`. Pause `donto-agent-beam.service` during the burst to free the 4 cores. Run a bounded 5,000-chunk test and measure the real rate = Δ(`SELECT count(*) FROM donto_x_memory_chunk_embedding`) over a 5-min wall window; confirm `_s` pref-recall hit@10 is unchanged. **Expect ~8–12× → 352K cleared in ~3–5h CPU.** Note `fastembed parallel=0` is broken here (multiprocessing queue Empty) — single process + `threads` + large batch.

2. **GPU, ~100–1000× — the coordinator is ready.** `pip install fastembed-gpu` (onnxruntime-gpu / CUDAExecutionProvider), point the worker at `memories.apexpots.com/embed/*` with the bearer token, lease batches of ~512. **Same model, same 384-dim, same ONNX — vectors are compatible with the existing fabric, so nothing already embedded is re-done.** On GPU, drop the truncation (full chunks are nearly free → max recall). The ~3.15M backlog falls to minutes-to-low-hours. The published GPU-worker PRD (`/research/donto-distributed-embedding-prd-2026-06-05`) has the worker code. Rust-first long-term: an `ort`-crate CUDA worker — but get vectors flowing first.

**The int8-ONNX to-do is a dead end on this box** (already int8, no VNNI). The model stays **pinned** to `BAAI/bge-small-en-v1.5` — none of this changes the model or requires re-embedding.

## 6. The honest caveats

1. **conc=5 is the user's explicit setting.** It gates nearly every free win; until raised, the workload-shape and serving-stack multipliers can't fully express. It's the user's dial to turn.
2. **The Hyades-side wins are conditional on controlling the holo3.1 engine.** Prefix caching, the raw-prompt alias, n-gram/EAGLE spec-decode, SGLang — all live *inside* the inference engine behind Cloudflare. If holo3.1 is itself a proxied/third-party route (the model list shows `:free` passthroughs), those levers vanish and effort shifts to the donto-agent side (concurrency, passes, lanes) + embedding. **Confirm the backend first.**
3. **No per-token budget rules out API scale-out.** Every ranked free/GPU win is $0/token, subscription-only. Only the flagged Cerebras/RunPod bursts cost cash, and those are user decisions for the one finite proof-gated run — never the default firehose, never silent spend. Cerebras is currently 402-capped/offline.
4. **Speed/quality trade-offs in extraction.** Cutting passes 6→2 lowers per-chunk coverage — acceptable for a 359K firehose (meaningful coverage, not exhaustive), recovered by the deferred high-yield re-sweep. INT4 quant adds drift — acceptable because the always-on CITER catches hallucinated/unanchorable facts regardless. Be honest that count-maximizing showcase runs (`xhigh`, 6-pass) and throughput-maximizing firehose runs are different profiles.
5. **The embedding model is pinned, and `nemotron3-omni` is broken for extraction.** Don't change bge-small (would force a full re-embed). Keep nemotron3-omni out of the extraction rotation (live-confirmed empty-content 200s = zero facts) and add the empty-content liveness check to the router probe. The "fix serialized:true" advice is wrong about the mechanism — `serialized` is dead code; concurrency is purely the CLI arg.

Throughout, donto stays **the substrate**: every lever here speeds the *extraction and embedding pipelines that feed it*. The I3 append-only invariant and evidence-first model are untouched — quantization changes only inference numerics, the CITER remains a separate always-on stage, and ingest stays pure inserts.

## 7. Recommended sequence

Free-first, each step measured the donto way (real DB-count delta over a wall-clock window, never the log's self-reported rate) before the next:

1. **TODAY, $0, ~5 min — raise concurrency 5 → 35.** Edit `donto-agent-beam.service` `ExecStart`, `daemon-reload`, `restart`. Ramp 5→20→35, measure Δdone/15 min, watch the 524 rate, back off to 25 if it climbs. *(Master gate — do this first; expect ~3× immediately.)*
2. **TODAY, $0, ~10 min — embedding truncation** (#8) on a bounded 5K-chunk test; validate hit@10; then run the full CPU backfill with the extractor paused. *(Independent of the LLM work — unblocks the embed backlog in ~3–5h.)*
3. **THIS WEEK, $0 — cut passes 6 → 2 + deferred re-sweep tag** (#2), then **`reasoning_effort='none'`** (#10) and **add glm-5.1 as a 2nd lane / exclude nemotron3-omni** (#9). *(Compounds with #1 toward ~250–540 chunks/hr.)*
4. **THIS WEEK, $0 — confirm you control the holo3.1 backend** (`ps aux | grep -E 'vllm|sglang|tgi'` on the Hyades host). If yes: enable + **verify** prefix caching (#3), kill the injected system prompt (#5), turn on n-gram spec-decode (#6), then drop `X-Hyades-Thread` for a cache-aligned stack (#7). If no: skip these, lean harder on #1/#2/#9.
5. **THIS WEEK, $0, medium — guided JSON decoding** (#4) for dense output + structural injection defense.
6. **NOW, $0 — stop pruning `/opencode/logs/beam-conv/`** to preserve the distillation corpus for #14.
7. **WHEN THE GPU LANDS (~$700 used 3090):** GPU embedding worker against the live coordinator (#11) → backlog in minutes; then quantized holo3.1 + FP8 KV (#12); then a vLLM 7–8B 2nd extraction lane (#13). Combined → BEAM-10M in ~13 days.
8. **LATER, GPU-gated bigger bets:** distill the 3B extractor (#14), layer EAGLE-3 (#15).
9. **ONLY if the proof-gate demands it, and only as an explicit user decision:** one Cerebras/RunPod-A100 burst for the finite 2,800-row frontier corpus (#16). Never the default; never silent.

The shortest honest summary: **turn one dial (concurrency 5→35) and truncate the embeddings today, for ~3× extraction and ~10× embedding at $0** — then, if you control the holo3.1 engine, stack prefix caching + fewer passes + n-gram spec-decode for another large multiplier, and let the incoming GPU close the rest. From ~8 months to ~4 weeks free, ~13 days with a $700 card, none of it spending a per-token cent.