Faster Inference for donto: From ~8 Months to ~4 Weeks (Free) to ~13 Days (GPU)

The substrate is not the bottleneck — token generation is. A measured, prioritized plan to speed up donto's extraction and embedding firehose at $0 per-token, on the gateway you already own.

2026-06-05

1. The bottleneck, measured

donto holds ~39.5M live statements across ~23K contexts and ingests new claims fine — the firehose-bulk path proved 251 chunks/s into the substrate via COPY. The substrate is not the gate. Inference is. Three measured numbers define the problem:

The decisive fact: Hyades is the user's OWN OpenAI-compatible gateway (hyades.gnostr.cloud/v1, holo3.1, no per-token cost). That changes everything. Most inference advice assumes you rent tokens from someone else; here the serving stack is yours to reconfigure. Server-side wins — prefix caching, the injected system prompt, speculative decoding, batch ceilings — are free leverage you control, not vendor knobs you beg for.

And the live concurrency probe is unambiguous: at the current --concurrency 5 you are using ~11–28% of a gateway you already own.

Hyades holo3.1 concurrency probe (2026-06-05, ~800-tok calls)
  N=5   → 0.47 ok/s   (you are HERE; service pulls ~0.1-0.25/s end-to-end)
  N=30  → 0.73 ok/s
  N=60  → 0.92 ok/s   ← aggregate ceiling, 0 errors
  N=100 → 0.91 ok/s   ← plateaued
  N=150 → 0.87 ok/s   but 41/150 = 27% HTTP-524 (saturated, Cloudflare cliff)

The aggregate req/s is flat above ~60 in-flight while per-request latency climbs — the classic signature of a continuous-batching server that's GPU/KV-bound, not a scheduling bug. You have ~3–9× of already-paid-for headroom sitting idle.

2. Where the time actually goes

A single BEAM chunk costs passes × (prefill + decode):

• Passes (6). One broad pass + up to 5 gleaning nudges (max_passes=6, dry_min=8, stop after 2 dry passes). Each pass is a full round-trip. On a 359K-chunk firehose, passes 3–6 are deep diminishing returns — the broad pass plus one nudge captures the bulk; the showcase 511→1,915→3,227 numbers were coverage-maximizing on one article, not throughput-optimizing on a corpus.
• The shared ~3K-token prompt, re-prefilled every pass. extract_broad.txt + injection bookends is identical on every chunk and every pass. With 6 passes × 359K chunks, that prefix is being recomputed millions of times needlessly — and probes confirm zero cached_tokens: prefix caching is OFF.
• A ~1120-token Hyades-injected system prompt, on every call. A 1-char user message bills 1120 prompt tokens on holo3.1. donto supplies its own prompt, so this is a ~27% prefill tax paid every pass.
• The thread header is not KV reuse. X-Hyades-Thread re-prefills the whole thread per nudge (measured: warm nudge 10.97s ≥ cold 10.56s, full 2052 prompt_tokens billed). The gleaning loop's core efficiency premise — "nudges are cheap" — is false as currently served.
• Reasoning overhead. Extraction needs almost none (codex's 511-fact pass spent 587 reasoning tokens); reasoning tokens are pure latency and cause the "200 + empty content" failure when they eat the budget.
• Embedding: CPU, sequence-length-bound. Measured: a 512-token chunk runs 0.8/s (bs=1) → 2.8/s (bs=128); a 12-token chunk runs 43–71/s on the same model. The cost is tokens, not the model.

One myth to kill up front, verified in code: provider.rs's serialized: true flag is dead code — declared, never read. The work loop already spawns N genuinely-concurrent SKIP-LOCKED tokio workers. Concurrency is purely the CLI arg; no Rust edit is needed to raise it. The widely-repeated "fix serialized:true" advice is a red herring; the real fix is even simpler.

3. The plan, prioritized

Every free/GPU lever below is $0 per-token, subscription-only. Only the flagged Cerebras/RunPod bursts cost cash, and those are explicit user decisions for the one finite proof-gated run — never the default firehose.

* #3, #6, #7 (and the holo3.1 quant/spec wins) are only actionable if the user controls the holo3.1 inference engine behind Cloudflare. The model list shows openrouter/free + z-ai:free passthroughs alongside owned_by:hyades models (holo3.1, glm-5.1, nemotron) — the latter look self-hosted. If holo3.1 is itself proxied, these Hyades-side levers vanish and effort shifts entirely to the donto-agent-side wins (#1, #2, #4, #7, #9, #10) + embedding.

FREE / this-week wins (the heart)

#1 — Raise concurrency 5 → 35. The single highest-ROI move and entirely the user's dial. It's one token in donto-agent-beam.service's ExecStart (--concurrency 5 → 35). It converts ~28% gateway utilization into ~85%. Ramp 5 → 20 → 35, measuring the real done-count delta over 15 min (not the log's self-reported rate, per CLAUDE.md §0.4); back off to 25 if 524s exceed ~5%. The 524-cliff lives at ≥150 in-flight, far above 35. conc=5 is the user's explicit setting — it gates nearly every other free win until raised.

#2 — Cut gleaning passes 6 → 2. --max-passes 2 + raise dry_min to ~10 so the single nudge stops fast on thin chunks. Tag high-yield chunks (an n_claims threshold in queue.rs) to re-sweep when a quality lane (codex-spark, or the GPU) is free. You keep coverage where it pays without paying for it everywhere.

#3 — Real prefix caching. donto's ~3K shared prompt is the textbook best case — identical across every chunk and pass. The flag is free; verify it actually fires by calling the same large prompt twice and confirming usage.prompt_tokens_details.cached_tokens > 0. Do not trust the flag. SGLang's RadixAttention is the strongest fit for the multi-turn nudge pattern if a backend swap is on the table.

#4 — Guided JSON decoding. Forces dense, parseable fact arrays, kills markdown ramble, near-zero parse failures, and structurally defeats the conversation prompt-injection drift (the model literally cannot emit a coding tutorial if the grammar permits only a fact array).

#5 — Kill the injected system prompt. Expose a holo3.1-raw alias with an empty system prompt for the extract lane. ~27% less prefill, every pass, and it compounds with #3.

#6 — N-gram speculative decoding. Output is 28% repetitive JSON scaffold ({"s":...,"p":...,"o":...,"c":...} per line) + ~70% verbatim-source literal objects + ~55 verbatim anchor spans/chunk — ideal for context-wide n-gram drafting. method:"ngram", num_speculative_tokens:5, prompt_lookup_max:4. Output-identical, no VRAM, no training, consumes no subscription cap.

#7 — Drop the thread; re-send a cache-aligned stack. Once #3 is on, re-send system(prompt) + user(source) + assistant(prior) + user(nudge) — content-based caching hits the cached prefix and prefills only the new tail. Cheaper than the thread re-prefill and stateless/resumable (better for the SKIP-LOCKED queue).

#8 — Embedding truncation. The bottleneck is sequence length, not the model. You already proved raising CHUNK_TEXT_CAP gave zero recall gain — signal lives in the first ~300 tokens. Truncate to ~96–128 tok (~384–512 chars) before model.embed(), set batch_size=128, threads=2, and pause the extractor during backfill to de-contend the 4 cores. Validate _s pref-recall hit@10 is unchanged.

#9 — glm-5.1 as a 2nd lane. Works (~396 comp-tok in ~9s on the same uncapped gateway); adds predicate diversity and helps two model queues push the ~0.9 ceiling. EXCLUDE nemotron3-omni — live test: HTTP 200 + empty content + usage:null = zero facts. Its "0.18s" is the speed of returning nothing. Add an empty-content liveness check to the router probe.

#10 — reasoning_effort='none'. ~nil on holo3.1 (honest — the probe showed no visible effect), but dominant the moment you rotate to a reasoning lane (glm-5.1 ~3.9s/call) and it mitigates the "200 + empty" failure. Keep a separate xhigh profile for showcase extractions.

GPU wins (when the card lands)

#11 — GPU embedding worker — the highest-ceiling, clearest GPU win. The distributed fabric was built for this: donto_embed_queue (FOR UPDATE SKIP LOCKED) + the live coordinator at :7930 (memories.apexpots.com/embed/*). Install fastembed-gpu (CUDA EP) — same bge-small, same 384-dim, vectors compatible, NO re-embed. Lease batches, embed, submit; the worker never touches the DB. On GPU, drop the truncation — full 512-tok chunks are nearly free, so the GPU run can maximize recall. Clears the ~3.15M backlog in minutes-to-low-hours.

Correction to delete from CLAUDE.md §0.5: "needs an int8-quantized ONNX model" is a misconception. fastembed already loads qdrant's -onnx-q int8 model, and this CPU has AVX2 but no AVX-512-VNNI — so int8 gets no fast path and re-quantizing buys ~0%. The real embed levers are truncation (free) and GPU.

#12 — Quantized holo3.1. holo3.1 = H Company's Holo-3.1-35B-A3B (35B total, ~3B active, Qwen-3.5-derived MoE, 256K ctx). H Co. ships official FP8 / Q4_K_M-GGUF (21.3 GB) / NVFP4 checkpoints — download-and-flag, not a project. At conc=5 (low-batch, memory-bandwidth-bound) + A3B sparsity, weight quantization wins exactly where you live. Add --kv-cache-dtype fp8 regardless (~2× KV space for the long-context nudge pattern, ~free). Format is hardware-gated: Hopper→FP8, Ada/Ampere→AWQ-INT4, Blackwell→NVFP4 — but A/B NVFP4 vs AWQ-INT4 first on desktop Blackwell (FP4 underperforms in vLLM today). The always-on CITER backstops INT4 drift, so donto can quant more aggressively than typical.

#13 — vLLM 7–8B extractor as a 2nd lane. A used 3090 (~$700) is the value king — skip the 4090 (~2× throughput for ~3× price). Enable prefix-caching (the ~7K prompt is identical → broad passes near-free). The GPU goes in a separate physical box (the e2-standard-4 can't take a consumer card) and joins via the same SKIP-LOCKED queue/coordinator pattern. Removes the Hyades concurrency wall — you control the vLLM batch.

Bigger bets

#14 — Distill a 3B extractor. The SFT corpus is already accumulating as (source→facts) pairs at /opencode/logs/beam-conv/<thread>.log. Once the GPU lands, LoRA-fine-tune Qwen2.5-3B / Llama-3.2-3B-Instruct on the teacher pairs, serve with vLLM (hundreds of tok/s vs holo3.1's ~30 effective). CITER backstops quality. ACTION NOW ($0): stop pruning /opencode/logs/beam-conv/ so the dataset survives.

#15 — EAGLE-3. The biggest spec-decode ceiling (~2.8–4.8×, output-identical) but needs a draft head matched to holo3.1's base; train with SpecForge on the GPU if no community head drops in. Use n-gram first; layer EAGLE-3 after.

#16 — Cerebras / RunPod burst. ~10–20× (600–1,200 chunks/hr), finishes the finite 2,800-row frontier corpus in hours. But Cerebras is currently 402-capped/offline and a top-up is per-token spend; a RunPod Community A100 (own vLLM, --max-passes 3) is ~$50–130/run. Cash — a deliberate user decision for the one proof-gated run, never the default firehose, never spend silently.

4. A concrete throughput model

Starting from the measured 60 chunks/hr:

EXTRACTION
  baseline (conc=5, 6-pass, holo3.1) ......................   60 chunks/hr   → BEAM-10M ~249 days (~8 mo)

  FREE / this-week (compounding):
   × #1 concurrency 5→35  (~3.0×) ........................   180 chunks/hr
   × #2 passes 6→2        (~3.0×) ........................   540 chunks/hr
   × #4+#5 guided JSON + prompt trim (~1.3×) .............  ~700 chunks/hr   ← realistic do-this-week
   ─ honest conservative band (overlap/measurement) ......  ~250–540 chunks/hr → BEAM ~4–8 WEEKS,  $0

  + IF user controls holo3.1 engine (compounding, free):
   × #3 prefix caching    (~2–4×)
   × #6 n-gram spec-dec   (~1.6–2.0×)        → another large multiplier on top

  GPU (vLLM 7–8B 2nd lane parallel to Hyades):
   combined ~19× ........................................  → BEAM-10M in ~13 DAYS,  $0/token
  GPU + distilled 3B (~5–15× over holo3.1) ..............  → removes the gateway wall entirely

EMBEDDING  (~352K BEAM + ~2.8M entity ≈ 3.15M backlog)
  baseline CPU fp32 ....................   2–4/s   → ~10–22 days
  + #8 truncate to ~96 tok (~8–12×) ....  ~20–30/s → BEAM 352K in ~3–5h CPU
  + batch128 + threads2 + pause extr. ..  ~30–50/s → 3.15M in ~18–29h CPU
  GPU worker (#11) .....................  ~1,000–3,000/s → 3.15M in MINUTES-to-low-HOURS

Do-this-week target (free, no hardware): ~250–540 chunks/hr → BEAM-10M from ~8 months to ~4–8 weeks. With the GPU as a second lane: ~13 days.

Honest ceiling: every free extraction multiplier is bounded by the user's conc=5 choice until raised, and by the Hyades ~0.9 req/s plateau (~1,620 chunks/hr at 2 passes) until the GPU gives a lane the user fully controls.

5. Embedding fast-path

The embedding backlog (~352K BEAM chunks + ~2.8M entity vectors) has no subscription ceiling — it's pure compute you own — making it the lowest-risk place to win, in two stages:

1. Today, free, ~10 min — truncate. In embed/embed_memory.py and donto-align/entity_embed.py: hard-cap text to ~384 chars before model.embed(), set batch_size=128, construct TextEmbedding(..., threads=2). Pause donto-agent-beam.service during the burst to free the 4 cores. Run a bounded 5,000-chunk test and measure the real rate = Δ(SELECT count(*) FROM donto_x_memory_chunk_embedding) over a 5-min wall window; confirm _s pref-recall hit@10 is unchanged. Expect ~8–12× → 352K cleared in ~3–5h CPU. Note fastembed parallel=0 is broken here (multiprocessing queue Empty) — single process + threads + large batch.

2. GPU, ~100–1000× — the coordinator is ready. pip install fastembed-gpu (onnxruntime-gpu / CUDAExecutionProvider), point the worker at memories.apexpots.com/embed/* with the bearer token, lease batches of ~512. Same model, same 384-dim, same ONNX — vectors are compatible with the existing fabric, so nothing already embedded is re-done. On GPU, drop the truncation (full chunks are nearly free → max recall). The ~3.15M backlog falls to minutes-to-low-hours. The published GPU-worker PRD (/research/donto-distributed-embedding-prd-2026-06-05) has the worker code. Rust-first long-term: an ort-crate CUDA worker — but get vectors flowing first.

The int8-ONNX to-do is a dead end on this box (already int8, no VNNI). The model stays pinned to BAAI/bge-small-en-v1.5 — none of this changes the model or requires re-embedding.

6. The honest caveats

1. conc=5 is the user's explicit setting. It gates nearly every free win; until raised, the workload-shape and serving-stack multipliers can't fully express. It's the user's dial to turn.
2. The Hyades-side wins are conditional on controlling the holo3.1 engine. Prefix caching, the raw-prompt alias, n-gram/EAGLE spec-decode, SGLang — all live inside the inference engine behind Cloudflare. If holo3.1 is itself a proxied/third-party route (the model list shows :free passthroughs), those levers vanish and effort shifts to the donto-agent side (concurrency, passes, lanes) + embedding. Confirm the backend first.
3. No per-token budget rules out API scale-out. Every ranked free/GPU win is $0/token, subscription-only. Only the flagged Cerebras/RunPod bursts cost cash, and those are user decisions for the one finite proof-gated run — never the default firehose, never silent spend. Cerebras is currently 402-capped/offline.
4. Speed/quality trade-offs in extraction. Cutting passes 6→2 lowers per-chunk coverage — acceptable for a 359K firehose (meaningful coverage, not exhaustive), recovered by the deferred high-yield re-sweep. INT4 quant adds drift — acceptable because the always-on CITER catches hallucinated/unanchorable facts regardless. Be honest that count-maximizing showcase runs (xhigh, 6-pass) and throughput-maximizing firehose runs are different profiles.
5. The embedding model is pinned, and nemotron3-omni is broken for extraction. Don't change bge-small (would force a full re-embed). Keep nemotron3-omni out of the extraction rotation (live-confirmed empty-content 200s = zero facts) and add the empty-content liveness check to the router probe. The "fix serialized:true" advice is wrong about the mechanism — serialized is dead code; concurrency is purely the CLI arg.

Throughout, donto stays the substrate: every lever here speeds the extraction and embedding pipelines that feed it. The I3 append-only invariant and evidence-first model are untouched — quantization changes only inference numerics, the CITER remains a separate always-on stage, and ingest stays pure inserts.

7. Recommended sequence

Free-first, each step measured the donto way (real DB-count delta over a wall-clock window, never the log's self-reported rate) before the next:

1. TODAY, $0, ~5 min — raise concurrency 5 → 35. Edit donto-agent-beam.service ExecStart, daemon-reload, restart. Ramp 5→20→35, measure Δdone/15 min, watch the 524 rate, back off to 25 if it climbs. (Master gate — do this first; expect ~3× immediately.)
2. TODAY, $0, ~10 min — embedding truncation (#8) on a bounded 5K-chunk test; validate hit@10; then run the full CPU backfill with the extractor paused. (Independent of the LLM work — unblocks the embed backlog in ~3–5h.)
3. THIS WEEK, $0 — cut passes 6 → 2 + deferred re-sweep tag (#2), then reasoning_effort='none' (#10) and add glm-5.1 as a 2nd lane / exclude nemotron3-omni (#9). (Compounds with #1 toward ~250–540 chunks/hr.)
4. THIS WEEK, $0 — confirm you control the holo3.1 backend (ps aux | grep -E 'vllm|sglang|tgi' on the Hyades host). If yes: enable + verify prefix caching (#3), kill the injected system prompt (#5), turn on n-gram spec-decode (#6), then drop X-Hyades-Thread for a cache-aligned stack (#7). If no: skip these, lean harder on #1/#2/#9.
5. THIS WEEK, $0, medium — guided JSON decoding (#4) for dense output + structural injection defense.
6. NOW, $0 — stop pruning /opencode/logs/beam-conv/ to preserve the distillation corpus for #14.
7. WHEN THE GPU LANDS (~$700 used 3090): GPU embedding worker against the live coordinator (#11) → backlog in minutes; then quantized holo3.1 + FP8 KV (#12); then a vLLM 7–8B 2nd extraction lane (#13). Combined → BEAM-10M in ~13 days.
8. LATER, GPU-gated bigger bets: distill the 3B extractor (#14), layer EAGLE-3 (#15).
9. ONLY if the proof-gate demands it, and only as an explicit user decision: one Cerebras/RunPod-A100 burst for the finite 2,800-row frontier corpus (#16). Never the default; never silent.

The shortest honest summary: turn one dial (concurrency 5→35) and truncate the embeddings today, for ~3× extraction and ~10× embedding at $0 — then, if you control the holo3.1 engine, stack prefix caching + fewer passes + n-gram spec-decode for another large multiplier, and let the incoming GPU close the rest. From ~8 months to ~4 weeks free, ~13 days with a $700 card, none of it spending a per-token cent.