donto's Alignment Engine — and How to Help Build It

A contributor's guide to the part of donto that turns an unbounded, contradictory firehose of claims into queryable knowledge — and a one-command way to lend it your compute. Compiled 6 June 2026. All numbers verified live against the running donto-pg instance.

0. TL;DR for contributors

donto extracts knowledge by letting an LLM invent predicates freely — the live store holds 40.8M statements across ~1.07M distinct, freely-minted predicates. Nothing is collapsed at write time. The hard part — folding synonyms (killedBy ↔ assassinatedBy), resolving identities (is this "George Davis" that one?), and ranking by reality — is deferred to query time, powered by a vector embedding fabric.

That fabric is CPU-bound on a single 4-core box (bge-small at ~2–4 chunks/sec), and there's a backlog of roughly:

~2.84M entity vectors · ~63K predicate vectors · ~455K conversation chunks waiting to be embedded.

You can knock that down from your laptop or a spare GPU box. A worker never touches the database — it leases text from a coordinator, embeds it, and submits 384-dim vectors. One command:

export DONTO_EMBED_TOKEN=<ask Thomas>
docker compose up -d --build --scale embed-worker=$(nproc)

The full worker kit (worker.py · Dockerfile · docker-compose.yml · README) is at /research/donto-embed-worker/. The rest of this report explains what you'd be feeding.

1. Why align at QUERY TIME — the abundance bet

For sixty years, generating typed knowledge was the scarce step. That's over: a guided frontier model emits an essentially unbounded, multi-directional space of properties and relations about anything — inventing the predicates as it goes — for ~$0.0001 each. donto's bet is to keep all of it: every variant, every contradiction, anchored to its source, held forever as legal state.

The cost of that bet is a proliferation the system must tame later, not a problem to prevent: ~1.07M distinct predicates, most of them LLM-minted variants of one another (rdfType/rdf:type, locatedIn/located-in, killedBy/murderedBy/wasAssassinatedBy). Collapsing them at write time (pick-a-winner, dedup-on-conflict) is what a vector DB or a normal KG does — and it throws most of the firehose away.

donto does the opposite: it folds synonyms at query time, by similarity — never a hand-maintained synonym list. The rule is inscribed in the source: "There is NO hardcoded synonym list anywhere — the model supplies the semantics." That's the no-brittle-logic rule (embeddings / similarity / learned alignment, never an if/elif ladder over strings). The alignment engine is how the bet pays off.

2. The embedding fabric — pgvector + bge-small

The substrate is Postgres 16 with pgvector 0.8.2. Meaning is carried by BAAI/bge-small-en-v1.5 (via fastembed), 384-dim, pinned. Three embedding targets, each a vector(384) column with an HNSW cosine index:

Every write is additive / I3-safe: embeddings only insert into embedding tables — they propose relationships, they never mutate donto_statement and never merge anything.

Why bge-small, why pinned: it's small enough to run on any CPU (and fly on a GPU), 384-dim keeps the HNSW indexes cheap, and the whole fabric must use one model so vectors are comparable. Changing the model means re-embedding everything — so the model name travels with each unit of work, and workers just load whatever they're told.

3. Query-time predicate folding — ledger → closure → donto_match_aligned

Three layers turn "1.07M predicates" into "a query for occupation also finds worksAs":

1. donto_predicate_alignment — the bitemporal, append-only alignment ledger (valid-time + tx-time; retraction closes a row's tx-time, never deletes). Relations: exact_equivalent, inverse_equivalent, sub_property_of, close_match, decomposition, not_equivalent (+ broader/narrower/value-mapping). Per-row safety flags gate use (safe_for_query_expansion on by default; logical-inference / export are opt-in). 19,837 active alignments today.
2. donto_predicate_closure — a flat, materialized index (predicate, equivalent, relation, swap_direction, confidence), rebuilt periodically by donto_rebuild_predicate_closure() (a cheap TRUNCATE-then-INSERT, never per-query). 883,226 rows: self 873,603 · close_match 8,390 · exact_equivalent 1,083 · inverse_equivalent 109 · sub_property_of 29 · decomposition 12 — i.e. 9,623 real expansion edges over a self-map that lets any predicate resolve in O(1).
3. donto_match_aligned(...) — the query-time fold. It returns (a) direct matches (confidence 1.0), (b) closure equivalents, and (c) swapping inverses — filtered with subject/object swapped, then projected back so the caller always sees a well-formed (subject, predicate, object). Every row is tagged with matched_via (the relation) and alignment_confidence. donto-memory recall calls this directly, which is how "where does he work?" folds occupation ↔ worksAs where lexical FTS scores zero.

Embeddings feed layer 1: nearest-neighbour predicates in the HNSW index become candidate alignments, which (above a similarity threshold, optionally LLM-adjudicated) get written to the ledger. The more predicate vectors exist, the better the folding.

4. Identity as a hypothesis — never a merge

Predicate alignment folds relations; identity resolution folds entities: does symbol A co-refer with symbol B? donto's stance is that identity is a reversible, versioned hypothesis, never a destructive merge.

• donto_identity_edge (124 live) — weighted, bitemporal pairwise assertions: same_referent / possibly_same_referent / distinct_referent / not_enough_information, each with confidence, method (trigram, embedding, neural, human, import, rule) and an evidence trail. Crucially, distinct_referent ("cannot-link") coexists with same_referent as legal state — contradictions are held, not resolved.
• donto_identity_hypothesis (13) + donto_identity_membership (123) — versioned clustering solutions over those edges, each with its own thresholds. Three ship: strict (≥0.98), likely (≥0.85), exploratory (≥0.60). The same data yields different clusters under different policies; a new hypothesis supersedes an old one rather than overwriting it.
• donto_identity_cluster_cache (1,026) — the materialized clusters, rebuilt periodically. donto_identity_proposal (85) — reversible ANN-generated candidates (entity embeddings → nearest neighbours → proposed, never auto-merged).

Entity embeddings are what make this work — they're the candidate generator. The 2.84M unembedded entities are 2.84M referents that can't yet be matched to their duplicates.

5. The continuous alignment daemon

donto-align-daemon.service (healthy, 192 ticks, 600s heartbeat) orchestrates the proven tools on a loop — it doesn't reinvent them, and each step is bounded + wrapped so one failure never kills the tick:

• embed new predicates + entities (bge-small) → propose predicate alignments (ANN + optional LLM adjudication) → propose identity edges → rebuild the closure + cluster caches periodically.
• It's cap-aware (≤300 GLM calls/window, backs off on the weekly cap), load-aware (pauses above load 4.0 / during active extraction), and idempotent. A real fix baked in: entity-embed batches are sorted by IRI so concurrent ticks acquire row locks in the same order (no deadlock). It currently runs dry_run — proposing, not auto-committing — so a human/threshold gate stays in the loop.

The daemon is the brain; it is bottlenecked by the muscle — embedding throughput. That's where you come in.

6. The distributed embedding fabric — coordinator + workers

To scale embedding from one box to many machines, donto runs a tiny coordinator (donto-embed-coordinator.service, aiohttp + asyncpg, 127.0.0.1:7930, public via Caddy at https://donto.org/embed/*). Workers are deliberately dumb and stateless: lease → embed → submit. They hold no DB credentials and see only text.

• Work queue donto_embed_queue — FOR UPDATE SKIP LOCKED leasing, so any number of concurrent workers/machines take disjoint work with zero double-embedding. A crashed worker's in-flight items auto-return after a ~15-min stale-lease TTL — nothing is lost.
• Provider-per-target — v1 serves memory_chunk; predicate / entity / statement providers follow the same shape (server-side only; workers need zero changes — they embed whatever text the lease hands them).
• Auth — Authorization: Bearer <token> on every endpoint except /embed/health. The token is distributed privately; it is not in this report or the worker repo.
• Dimension-validated — the coordinator drops any vector that isn't exactly 384-dim, so a misconfigured worker can't corrupt the fabric.

Worker protocol (what your container speaks)

The loop: lease a batch → fastembed(bge-small) the text fields → submit the 384-dim vectors → repeat; sleep when empty. That's the entire worker.

7. The backlog you'd knock down (live, 6 Jun 2026)

On the single 4-core box, bge-small runs ~2–4 chunks/sec — so this backlog is months of CPU. A handful of contributor cores (or one consumer GPU at hundreds–thousands/sec) collapses it to days or hours. Every vector you add directly improves predicate folding, identity resolution, and memory recall across the whole substrate.

8. Contribute — the worker

Everything you need is in /research/donto-embed-worker/ (worker.py, Dockerfile, docker-compose.yml, README.md).

# 1. get the token from Thomas (sent privately)
export DONTO_EMBED_TOKEN=<token>

# 2. run one worker per core
docker compose up -d --build --scale embed-worker=$(nproc)
docker compose logs -f          # watch  +256 (total …)

# …or bare metal
pip install fastembed           # fastembed-gpu for NVIDIA
DONTO_EMBED_URL=https://donto.org/embed python worker.py

The worker, in full (it really is this small):

batch = post("/lease", {"worker_id": WID, "n": N, "target": TARGET})["batch"]
vecs  = model_for(b["model"]).embed([b["text"] for b in batch])     # bge-small, 384-dim
post("/submit", {"worker_id": WID,
                 "items": [{"target": b["target"], "item_id": b["item_id"],
                            "vector": [float(x) for x in v]} for b, v in zip(batch, vecs)]})

Safety / FAQ. Your worker (a) never connects to Postgres or sees the corpus — only the text it must embed; (b) can be stopped anytime — unsubmitted leases reclaim in ~15 min, nothing double-embeds; (c) must keep the model at BAAI/bge-small-en-v1.5 (the coordinator names it per lease, so just don't override it); (d) needs only Python 3.9+, fastembed, and outbound HTTPS. A GPU (fastembed-gpu, EMBED_N=1024, --gpus all) is 100–1000× a CPU core.

9. Known issues (contributions welcome here too)

• GET /embed/stats currently times out over HTTP — its per-target counts() runs an unbounded count(*) join over the 40M-row statement table. Needs a cached/bounded count. (Use the DB figures in §7 meanwhile.)
• Only the memory_chunk provider is wired into the coordinator's PROVIDERS; the predicate / entity / statement providers are designed but not yet registered, and the queue isn't bulk-enqueued yet — so step one for the first contributor session is the operator running /embed/enqueue (and wiring the entity provider) to materialize the 2.84M-entity backlog into the queue.

If you want to go deeper than embedding — the alignment ledger, the identity hypotheses, the daemon, the closure rebuild — that's all open too. Ping Thomas. We can figure out all understanding together.