# donto's Alignment Engine — and How to Help Build It

*A contributor's guide to the part of donto that turns an unbounded, contradictory firehose of claims into queryable knowledge — and a one-command way to lend it your compute. Compiled 6 June 2026. All numbers verified live against the running `donto-pg` instance.*

---

## 0. TL;DR for contributors

donto extracts knowledge by letting an LLM **invent predicates freely** — the live store holds **40.8M statements** across **~1.07M distinct, freely-minted predicates**. Nothing is collapsed at write time. The hard part — folding synonyms (`killedBy` ↔ `assassinatedBy`), resolving identities (is *this* "George Davis" *that* one?), and ranking by reality — is deferred to **query time**, powered by a vector **embedding fabric**.

That fabric is **CPU-bound on a single 4-core box** (bge-small at ~2–4 chunks/sec), and there's a backlog of roughly:

> **~2.84M entity vectors · ~63K predicate vectors · ~455K conversation chunks** waiting to be embedded.

You can knock that down from your laptop or a spare GPU box. A worker **never touches the database** — it leases text from a coordinator, embeds it, and submits 384-dim vectors. One command:

```bash
export DONTO_EMBED_TOKEN=<ask Thomas>
docker compose up -d --build --scale embed-worker=$(nproc)
```

The full worker kit (worker.py · Dockerfile · docker-compose.yml · README) is at **[/research/donto-embed-worker/](donto-embed-worker/)**. The rest of this report explains what you'd be feeding.

---

## 1. Why align at QUERY TIME — the abundance bet

For sixty years, *generating* typed knowledge was the scarce step. That's over: a guided frontier model emits an essentially unbounded, multi-directional space of properties and relations about anything — **inventing the predicates as it goes** — for ~$0.0001 each. donto's bet is to **keep all of it**: every variant, every contradiction, anchored to its source, held forever as legal state.

The cost of that bet is a *proliferation* the system must tame later, not a problem to prevent: ~1.07M distinct predicates, most of them LLM-minted variants of one another (`rdfType`/`rdf:type`, `locatedIn`/`located-in`, `killedBy`/`murderedBy`/`wasAssassinatedBy`). Collapsing them at write time (pick-a-winner, dedup-on-conflict) is what a vector DB or a normal KG does — and it throws most of the firehose away.

donto does the opposite: it **folds synonyms at query time, by similarity** — never a hand-maintained synonym list. The rule is inscribed in the source: *"There is NO hardcoded synonym list anywhere — the model supplies the semantics."* That's the no-brittle-logic rule (embeddings / similarity / learned alignment, never an `if/elif` ladder over strings). The alignment engine is how the bet pays off.

---

## 2. The embedding fabric — pgvector + bge-small

The substrate is Postgres 16 with **pgvector 0.8.2**. Meaning is carried by **`BAAI/bge-small-en-v1.5`** (via fastembed), **384-dim**, pinned. Three embedding targets, each a `vector(384)` column with an **HNSW cosine index**:

| Target table | Rows embedded (live) | What the text is |
|---|---:|---|
| `donto_predicate_embedding` | **886,832** | the *humanized* predicate IRI — `ex:assassinatedBy` → "assassinated by" (strip prefix, split camel/kebab/snake, lowercase). Structure only; the model supplies meaning. |
| `donto_entity_embedding` | **317,233** | an entity *fingerprint* — label + its most salient outgoing statements, salience weighted by **data-driven IDF** (`donto_predicate_idf`) so plumbing predicates sink and discriminating relations float. A `sig_hash` makes it incremental — only re-embed when the entity's statements actually change. |
| `donto_x_memory_chunk_embedding` | **59,705** | conversation/episodic chunks, for donto-memory's hybrid recall (bge-small + FTS via Reciprocal Rank Fusion). |

Every write is **additive / I3-safe**: embeddings only insert into embedding tables — they *propose* relationships, they never mutate `donto_statement` and never merge anything.

> **Why bge-small, why pinned:** it's small enough to run on any CPU (and fly on a GPU), 384-dim keeps the HNSW indexes cheap, and the whole fabric must use *one* model so vectors are comparable. Changing the model means re-embedding everything — so the model name travels *with* each unit of work, and workers just load whatever they're told.

---

## 3. Query-time predicate folding — ledger → closure → `donto_match_aligned`

Three layers turn "1.07M predicates" into "a query for `occupation` also finds `worksAs`":

1. **`donto_predicate_alignment`** — the bitemporal, append-only **alignment ledger** (valid-time + tx-time; retraction *closes* a row's tx-time, never deletes). Relations: `exact_equivalent`, `inverse_equivalent`, `sub_property_of`, `close_match`, `decomposition`, `not_equivalent` (+ broader/narrower/value-mapping). Per-row safety flags gate use (`safe_for_query_expansion` on by default; logical-inference / export are opt-in). **19,837 active alignments** today.
2. **`donto_predicate_closure`** — a flat, materialized index `(predicate, equivalent, relation, swap_direction, confidence)`, rebuilt periodically by `donto_rebuild_predicate_closure()` (a cheap TRUNCATE-then-INSERT, never per-query). **883,226 rows**: `self` 873,603 · `close_match` 8,390 · `exact_equivalent` 1,083 · `inverse_equivalent` 109 · `sub_property_of` 29 · `decomposition` 12 — i.e. **9,623 real expansion edges** over a self-map that lets any predicate resolve in O(1).
3. **`donto_match_aligned(...)`** — the query-time fold. It returns (a) direct matches (confidence 1.0), (b) closure equivalents, and (c) **swapping inverses** — filtered with subject/object swapped, then *projected back* so the caller always sees a well-formed `(subject, predicate, object)`. Every row is tagged with `matched_via` (the relation) and `alignment_confidence`. donto-memory recall calls this directly, which is how "where does he work?" folds `occupation` ↔ `worksAs` where lexical FTS scores zero.

**Embeddings feed layer 1**: nearest-neighbour predicates in the HNSW index become *candidate* alignments, which (above a similarity threshold, optionally LLM-adjudicated) get written to the ledger. The more predicate vectors exist, the better the folding.

---

## 4. Identity as a hypothesis — never a merge

Predicate alignment folds *relations*; identity resolution folds *entities*: does symbol A co-refer with symbol B? donto's stance is that **identity is a reversible, versioned hypothesis, never a destructive merge.**

- **`donto_identity_edge`** (124 live) — weighted, bitemporal pairwise assertions: `same_referent` / `possibly_same_referent` / `distinct_referent` / `not_enough_information`, each with confidence, `method` (trigram, embedding, neural, human, import, rule) and an evidence trail. Crucially, **`distinct_referent` ("cannot-link") coexists with `same_referent` as legal state** — contradictions are held, not resolved.
- **`donto_identity_hypothesis`** (13) + **`donto_identity_membership`** (123) — *versioned clustering solutions* over those edges, each with its own thresholds. Three ship: **strict** (≥0.98), **likely** (≥0.85), **exploratory** (≥0.60). The same data yields different clusters under different policies; a new hypothesis *supersedes* an old one rather than overwriting it.
- **`donto_identity_cluster_cache`** (1,026) — the materialized clusters, rebuilt periodically. **`donto_identity_proposal`** (85) — reversible ANN-generated candidates (entity embeddings → nearest neighbours → proposed, never auto-merged).

Entity embeddings are what make this work — they're the candidate generator. The **2.84M unembedded entities** are 2.84M referents that *can't yet be matched to their duplicates.*

---

## 5. The continuous alignment daemon

`donto-align-daemon.service` (**healthy**, 192 ticks, 600s heartbeat) orchestrates the proven tools on a loop — it doesn't reinvent them, and each step is bounded + wrapped so one failure never kills the tick:

- **embed** new predicates + entities (bge-small) → **propose** predicate alignments (ANN + optional LLM adjudication) → **propose** identity edges → **rebuild** the closure + cluster caches periodically.
- It's **cap-aware** (≤300 GLM calls/window, backs off on the weekly cap), **load-aware** (pauses above load 4.0 / during active extraction), and **idempotent**. A real fix baked in: entity-embed batches are **sorted by IRI** so concurrent ticks acquire row locks in the same order (no deadlock). It currently runs `dry_run` — proposing, not auto-committing — so a human/threshold gate stays in the loop.

The daemon is the *brain*; it is bottlenecked by the *muscle* — embedding throughput. That's where you come in.

---

## 6. The distributed embedding fabric — coordinator + workers

To scale embedding from one box to many machines, donto runs a tiny **coordinator** (`donto-embed-coordinator.service`, aiohttp + asyncpg, `127.0.0.1:7930`, public via Caddy at **`https://donto.org/embed/*`**). Workers are deliberately **dumb and stateless**: `lease → embed → submit`. They hold no DB credentials and see only text.

- **Work queue** `donto_embed_queue` — `FOR UPDATE SKIP LOCKED` leasing, so any number of concurrent workers/machines take **disjoint** work with zero double-embedding. A crashed worker's in-flight items **auto-return after a ~15-min stale-lease TTL** — nothing is lost.
- **Provider-per-target** — v1 serves `memory_chunk`; `predicate` / `entity` / `statement` providers follow the same shape (server-side only; workers need **zero** changes — they embed whatever text the lease hands them).
- **Auth** — `Authorization: Bearer <token>` on every endpoint except `/embed/health`. The token is distributed privately; it is **not in this report or the worker repo**.
- **Dimension-validated** — the coordinator drops any vector that isn't exactly 384-dim, so a misconfigured worker can't corrupt the fabric.

### Worker protocol (what your container speaks)
| Endpoint | Auth | Purpose |
|---|---|---|
| `GET /embed/health` | no | `{"ok":true,"targets":["memory_chunk"]}` |
| `GET /embed/stats` | yes | per-target `{source, embedded, queue:{pending,leased,done}}` |
| `POST /embed/enqueue` | yes (operator) | fill the queue: `{target, limit}` → `{enqueued}` |
| `POST /embed/lease` | yes | `{worker_id, n, target?}` → `{batch:[{target,item_id,text,model,dim}]}` (n ≤ 1024) |
| `POST /embed/submit` | yes | `{worker_id, items:[{target,item_id,vector:[…384]}]}` → `{upserted}` |

The loop: **lease a batch → `fastembed(bge-small)` the `text` fields → submit the 384-dim vectors → repeat; sleep when empty.** That's the entire worker.

---

## 7. The backlog you'd knock down (live, 6 Jun 2026)

| What's embedded | Count | What's waiting | Count |
|---|---:|---|---:|
| predicate vectors | 886,832 | **unembedded entities** | **2,837,699** |
| entity vectors | 317,233 | **unembedded predicates** | **63,074** |
| memory-chunk vectors | 59,705 | **BEAM conversation chunks** | **~454,648** |
| closure expansion edges | 9,623 | memory-chunk coverage | ~10.7% |

On the single 4-core box, bge-small runs ~2–4 chunks/sec — so this backlog is *months* of CPU. A handful of contributor cores (or one consumer GPU at hundreds–thousands/sec) collapses it to days or hours. **Every vector you add directly improves predicate folding, identity resolution, and memory recall across the whole substrate.**

---

## 8. Contribute — the worker

Everything you need is in **[/research/donto-embed-worker/](donto-embed-worker/)** (`worker.py`, `Dockerfile`, `docker-compose.yml`, `README.md`).

```bash
# 1. get the token from Thomas (sent privately)
export DONTO_EMBED_TOKEN=<token>

# 2. run one worker per core
docker compose up -d --build --scale embed-worker=$(nproc)
docker compose logs -f          # watch  +256 (total …)

# …or bare metal
pip install fastembed           # fastembed-gpu for NVIDIA
DONTO_EMBED_URL=https://donto.org/embed python worker.py
```

The worker, in full (it really is this small):

```python
batch = post("/lease", {"worker_id": WID, "n": N, "target": TARGET})["batch"]
vecs  = model_for(b["model"]).embed([b["text"] for b in batch])     # bge-small, 384-dim
post("/submit", {"worker_id": WID,
                 "items": [{"target": b["target"], "item_id": b["item_id"],
                            "vector": [float(x) for x in v]} for b, v in zip(batch, vecs)]})
```

**Safety / FAQ.** Your worker (a) never connects to Postgres or sees the corpus — only the text it must embed; (b) can be stopped anytime — unsubmitted leases reclaim in ~15 min, nothing double-embeds; (c) must keep the model at `BAAI/bge-small-en-v1.5` (the coordinator names it per lease, so just don't override it); (d) needs only Python 3.9+, `fastembed`, and outbound HTTPS. A **GPU** (`fastembed-gpu`, `EMBED_N=1024`, `--gpus all`) is 100–1000× a CPU core.

---

## 9. Known issues (contributions welcome here too)

- `GET /embed/stats` currently **times out** over HTTP — its per-target `counts()` runs an unbounded `count(*)` join over the 40M-row statement table. Needs a cached/bounded count. (Use the DB figures in §7 meanwhile.)
- Only the `memory_chunk` provider is wired into the coordinator's `PROVIDERS`; the `predicate` / `entity` / `statement` providers are designed but not yet registered, and the queue isn't bulk-enqueued yet — so step one for the first contributor session is the operator running `/embed/enqueue` (and wiring the entity provider) to materialize the 2.84M-entity backlog into the queue.

If you want to go deeper than embedding — the alignment ledger, the identity hypotheses, the daemon, the closure rebuild — that's all open too. Ping Thomas. **We can figure out all understanding together.**