The dist/soul.c-matches-sources check is a proof-of-concept of the DHARMA
contract, not an enforced gate we need between us mid-cultivation. Keep it
running (it still reports) but stop it failing the build. The enforced
contract is for the world and re-hardens before deploy, when the full DHARMA
blockchain stands up.
tools/list now returns exactly 9 ops (design: api-reshape README, artifact
0e828907 / surface.el §5) instead of the noun-per-tool catalog. Type is a
parameter, not a tool-per-noun.
Layer 1 — geometry (live against soul :7770 today):
read({vantage,type?,k,depth}) write({content,type,...})
relate({from,to,relationship}) supersede({id,action,content?})
Layer 2 — agentic primitives (return an honest pending-cognition-promotion
envelope until the cognition build is promoted on the engram):
think attend assert ground learn
Why:
- The old surface advertised empty inputSchemas so args never bound; every op
here declares a real schema (tool_s) so targeting/bounding params bite.
- Vantage-read fixes the whole-self-dump: the aperture (k/depth) bounds output.
Because the live soul's /graph does not yet honor compact/k, the aperture is
enforced at the WRAPPER boundary (cap_output, ~2000 + k*3000 chars) where the
MCP transport limit bites. Measured: self read k=1 -> 5.3KB, k=20 -> 65KB
(was ~790KB unbounded).
- Identity keystones (kn-efeb4a5b / kn-5b606390) are write-protected on
write(type=self|values), relate, and supersede.
Transition: the previous ~90 tool names remain as HIDDEN ALIASES in
dispatch_tool_call (old catalog retained as unused tools_catalog_full), so any
caller still using an old name keeps working while the visible surface is the 9.
The doc overstated the provenance, birth-gate, and lineage layer as fully
realized. That layer is built but has drifted from spec, so tier it honestly
as [STAGED] where real and [TARGET] where aspirational to keep the
documentation faithful to what actually runs.
Give the architecture set its persistence and moral layers so a self's
durability and sovereignty are documented as first-class, not folded into
the cognitive doc. 07 explains how a self persists and travels
(events-become-the-graph, weights-as-world-lines with bitemporal recall,
transactionless coherence, and the honest load/tiering findings); 08
explains the moral mechanism (DHARMA as a proof-of-integrity ledger,
abundance economics, the relational immune system, dual-anchor governance,
and CGI citizenship as telos). Extend 06 with forward-pointers into both,
and reconcile two cross-references so tiers agree across docs: the
canonical 187 reseed count, and the #56 load-merge-persist fix as
LIVE/reboot-proven with only full WAL edge-ownership left decision-pending.
auto_term_try_slot now passes the WM node's ID to engram_salient_term()
instead of passing its label to a first-word extractor. The runtime scores
every candidate token in the node's text and returns the best one, falling
back from a sentinel label ("memory:remembered") to content — which is the
only reason Memory nodes are visible to the extractor at all. They dominate
working memory, and dynamic seeding had been dead for 50+ consecutive scans
because of it.
Policy stays here: node-type filter, df thresholds, stopword list. The
runtime measures, the soul decides — same split as engram_label_df.
The stopword list stays, and not as belt-and-braces. An earlier draft assumed
the min_df floor would subsume it based on 08-03's finding that function
words have df 0 in labels. Re-measured under word-boundary df: about:2,
whole:1, them:2 — they clear a floor of 1. What keeps them from winning is
the argmax, not the floor.
The old extractor and its five guards are retained as
auto_term_try_slot_legacy, unreferenced, so the reasoning behind each guard
stays readable next to what replaced it. Delete once the new path has a month
of live telemetry.
Live after restart: auto_term producing DRIFT, Wrote; empty streak reset to 0
and holding; activation counts 123-281, within the normal band, no flood.
Reversal runbook for the ENGRAM_GEOMETRY_PRIMING cutover (default OFF, reversible
flag flip; exact rollback) and the A/B perf profile: default-OFF binary GO
(byte-identical to M8), enabling the flag NO-GO on latency (3.2x/13x) with no
demonstrated recall benefit; safety/sanitizer clean.
Convert routes.el from ~89 hand-written dispatch branches to 75 @route-decorated
handlers with VBD roles (@manager/@accessor/@utility), driven by the modular
compiler's native @route dispatch. routes.elh updated to match.
Fix four latent bugs that the modular compiler surfaces (unlike the old inlining
compiler, it faithfully emits every source statement, so per-module compilation
no longer silently drops code):
- safety.el: malformed soft-phrases literal (extra unescaped quote)
- sessions.el: str_replace quote arg + topic_tags JSON escaping
- memory.el mem_save and neuron-api.el consolidate: engram_save returns a Bool,
not a String; str_eq(result, "") dereferenced the Bool value (0x1) as a char*,
segfaulting the awareness loop on boot and POST /api/neuron/consolidate.
Check it as a Bool (if !save_result). The old compiler dropped these checks,
masking the bug in the shipped dist/soul.c.
Regenerate dist/soul.c through the bounded per-module path (modular elc per module
plus amalgamation that embeds dist/elp-c-decls.h), never by folding soul.el through
elc (27GB OOM). Verified on a throwaway snapshot and port: boots with the awareness
loop active and no segfault, @route dispatch matches the manual dispatcher across
all sampled routes and the four shadowing-hazard pairs, and consolidate returns 200.
The cognitive-graph and write tools advertised an empty inputSchema
({"properties":{}}), so MCP clients never sent entity_id/depth/query/
from_id/node_id etc. Graph reads fell back to the full neighborhood
(480-775KB, over transport limits) and write tools (forget, linkEntities,
evolveMemory) had no way to target a node.
- Declare per-tool JSON-Schemas matching the params each soul handler
already accepts (76 of 87 tools; 11 are genuinely param-less).
- Read + forward the declared args: inspectGraph now honors depth (was
reading only legacy max_depth), compact (default on), snip, k;
traverseGraph accepts entity_id and defaults compact on so a depth-2
walk stays bounded; retrieveKnowledge forwards depth/snip/k.
- compact_flag() reads the raw JSON token so an integer 0 / false / "0"
opts out correctly (json_get_string could not see an integer and
silently forced compact back on).
Builds on PR #149's compact projection; keeps the relevance-ranked
bound on by default for graph neighborhoods.
High-fanout identity anchors (voice, writing-imprint, self-root) have ~670KB
neighborhoods. inspect_graph returned the full traversal, which overflowed the
MCP client's context and socket-closed the wrapper mid self-load -- the soul
could not traverse its own identity graph.
handle_api_inspect_graph gains an opt-in `compact` projection (compact=1|true):
the neighborhood is relevance-ranked, the top K (default 12) keep a UTF-8-safe
content snippet (default snip=600), and the remainder collapse to lightweight
{id,label,node_type,tier,edge,pointer:true} stubs. This bounds the voice node
from 669,799B -> 25,353B (HTTP 200, valid JSON) and the wrapper's soul-load no
longer socket-closes. New helpers: api_compact_neighbors, api_neigh_full,
api_neigh_pointer, api_neigh_rank, api_neigh_better, api_float_or.
The flag is gated: ABSENT it, the response is byte-identical to the old plain
traversal, so the studio app (which never sends it) is unaffected. The MCP
wrapper (mcp-wrapper/src/main.el) appends &compact=1 on its inspectGraph and
fetch-by-id paths.
dist/soul.c is REGENERATED so CI ships the fix: CI compiles the committed
single-TU dist/soul.c directly (running elb/elc on the Linux runner OOM-kills
it), so an .el-only change would build the OLD behavior. Regenerated and verified
on macOS -- compiles with the CI cc line (0 errors) and, on a throwaway soul over
a copy of the live snapshot, serves compact ~25KB / non-compact ~670KB. The regen
also syncs the amalgamation to this branch's .el sources, which had drifted
several self-review commits ahead of the previously-committed soul.c.
Docs: docs/architecture/00-05 added; 01/02/05 corrected so the relevance-ranked
inspect_graph projection reads as committed source, not an in-flight concern.
el_runtime.c emitted 19 metric keys from engram_metrics_json; emit_heartbeat
forwarded 14. Dropped on the floor: hebb_cands, hebb_cand_max, hebb_mass,
hebb_edges, embed_consec_fail. The first two are exactly the pair the runtime
added to answer the 08-06 question -- whether a stalled hebb_links means
nothing co-activates or the threshold is too high. Undiagnosable from the
durable record without them. An instrument computed but not plumbed to
durable storage is not an instrument.
Also adds the corpus damage STOCK, not just the flow. 08-08 fixed the JSON
parser, watched txt_damaged (nodes damaged by a write THIS process) fall to
0, and recorded the defect closed. Census today: 2781 of 4100 nodes still
damaged -- 67.8%, including the self root and every values node. A flow gauge
reads 0 both when the corpus is clean and when it is uniformly damaged but
quiescent. Sampled on a 30-beat countdown and carried with an explicit age.
studio.el changed, so the committed build input went stale the moment the merge
landed. CI compiles dist/soul.c, not the .el files. The stamp gate named
studio.el and refused; this is the regeneration it asked for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
el_cgi_init loaded the declared identity into runtime globals at startup and
printed it. Nothing read it back out — no accessor existed and it writes no
state, so every consumer still read identity from the mutable state store.
studio.el's dharma_registry read state_get("soul_principal"), a key with no
producer anywhere in the tree, and reported an empty principal under a heading
reading 'Principal Covenant v1'.
Adds cgi_name / cgi_dharma_id / cgi_principal / cgi_network / cgi_engram and
points dharma_registry at the compiled constant.
Read-only on purpose. There is deliberately no setter: publishing these into the
state store would have been one line, passed the same test, and recreated exactly
the runtime-mutable copy IDPROTO claims 1-2 forbid.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Installing the fixed compiler exposed a blind spot in this gate. On clean main,
with no source changed, soulc-stamp reported OK while the committed amalgam had
gone stale by a line — because the fingerprint covered .el sources and not the
toolchain that turns them into dist/soul.c. That is exactly the class of silent
divergence the gate was written to close, and it had it.
The stamp now fingerprints the elc binary alongside the sources. Demonstrated: with
the old stamp the gate passed after a compiler swap; with this change the same
condition fails, naming __compiler__.
dist/soul.c regenerated under the installed compiler (1,205,027 bytes) and verified:
builds from its own committed input, the declared principal is present in the
resulting binary, interface 110 routes in / 110 out.
Differential evidence that the new compiler is a strict superset — same sources,
both compilers:
neuron soul 1 differing line, the el_cgi_init emission
engram server.el 0 differing lines
mcp-wrapper 0 differing lines
mcp-proxy 0 differing lines
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
el_cgi_init loaded the declared identity into runtime globals and printed it, and
nothing read it back out. No accessor existed and it writes no state, so every
consumer still read identity from the mutable state store. studio.el's registry
read state_get("soul_principal") — a key with no producer anywhere — and reported
an empty principal under a heading reading 'Principal Covenant v1'.
Adds cgi_name/cgi_dharma_id/cgi_principal/cgi_network/cgi_engram. READ-ONLY on
purpose: there is deliberately no setter. Publishing these into the state store
would have been one line and would have recreated exactly the runtime-mutable copy
IDPROTO claims 1-2 forbid ('not modifiable by any runtime mechanism including
environment variables, configuration files, or API calls').
dharma_registry now reads the compiled constant. cgi_id keeps its state read
deliberately — the runtime instance id is a different fact from the compiled
dharma_id, and conflating them would hide a binary running under an id its own
declaration never claimed.
Measured, same corpus, binary the only variable:
deployed engine -> "principal":""
accessor build -> "principal":"william-christopher-anderson"
interface: 110 routes in, 110 out, nothing removed
Requires the codegen fix in el (fix/cgi-identity-emission); without it the
declaration is never compiled in and the accessors return empty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five sites in chat.el splice state_get("soul_identity") into the system prompt.
Nothing has written that key since b163fa6 deleted the producer days after 601e0fe
added it on 2026-05-02. Every chat turn since built its prompt with an empty
identity section, and nothing reported it.
Found by the #132 state-key gate within an hour of that gate being rebased onto
main — a read with no producer treated as a build error rather than a silence.
Restored verbatim rather than repointed. soul_identity is an env-configurable
persona line; soul_identity_context is the graph-derived DNA/values/memory-philosophy
block. Aiming the five reads at the latter would have substituted different content
and called it a repair. Whether the chat prompt should also carry that block is a
separate question, left open rather than smuggled in.
Verified by the gate that found it: dead reads 6 -> 1, with the chat.el baseline
entry now reported STALE. Rung: BUILT and gate-verified; not end-to-end chat-verified.
Closes#137. Refs #132.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five sites in chat.el read state_get("soul_identity") and splice the result into
the system prompt, beside the voice, security and capability rules:
chat.el:737, 1745, 2620, 3425, 3480
Nothing has written that key since b163fa6. The producer was added 2026-05-02 in
601e0fe and deleted by the awareness refactor days later. Every chat turn since has
built its system prompt with an EMPTY identity section, and nothing reported it.
Found by the #132 state-key gate, which treats a read with no producer as a build
error rather than a silence. That is the entire argument for that gate.
RESTORED VERBATIM, NOT IMPROVED. soul_identity is an env-configurable persona LINE.
It is not soul_identity_context — the graph-derived
[INTELLECTUAL-DNA]/[VALUES]/[MEMORY-PHILOSOPHY] block written at soul.el:184.
Repointing the five reads at that block would have substituted different content and
called it a repair. Whether the chat prompt should ALSO carry the graph-derived block
is a real question and a separate one; it is not smuggled in here.
Verified by the gate that found it: dead reads 6 -> 1, and it now reports the
chat.el baseline entry as STALE — 'entries that no longer match anything; delete
them'. The remaining one is studio.el's soul_principal, untouched by this change.
Rung: BUILT, gate-verified, boots. NOT end-to-end chat-verified — proving the
prompt now carries the line needs a live provider call, which I have not run.
Refs #137, #132
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The runtime-vs-owner divergence check. Its absence let a ~24,000-node loss run for
weeks with every boot reporting green, which is most of why the last two days were
spent rediscovering by hand what this route would have said.
Follows the spec rather than inventing a metric: CGI provisional
05-detailed-description.md Stage 1 calls for an annotated characterization of the
graph's structure, so the route returns findings with score null BY DESIGN. A number
here would be a fabrication dressed as rigour.
Rebased across 27 commits of drift. The rebase merged cleanly at source level and
that was misleading — dist/soul.c held one side's code and not the other, because
git resolved the amalgam as an ordinary file. The stamp gate from 9fd8c11 caught it
on its first real use. Without it this would have landed an engine containing the
audit but none of the 08-09 engine work, or the reverse: neuron#133 again.
Verified before merging, not after:
stamp OK dist/soul.c matches the sources (1,204,442 bytes, 1,259 bodies)
builds from committed input 920,776 bytes
interface 108 -> 110 routes, nothing removed
the route answers stage 1, annotated_characterization, findings present
Closes#142. Refs #91.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rebase merged cleanly at source level but left dist/soul.c holding one side's
code and not the other — main's regenerated amalgam vs this branch's. The stamp
gate added in 9fd8c11 caught it on its first real use:
FAIL: dist/soul.c is STALE. It does not match the current .el sources.
Sources that changed: neuron-api.el, routes.el
Without that gate this branch would have looked clean and shipped an engine
containing the structural audit but none of the 08-09 engine work, or the reverse.
That is exactly neuron#133, which once hid five merged fixes including a P0.
Regenerated from the rebased sources: 1,204,442 bytes, 1,259 inlined bodies.
Verified after: stamp OK; builds from its own committed input (920,776 bytes);
interface 108 -> 110 routes with nothing removed, adding /api/neuron/audit/structural;
and the route answers — stage 1, assessment_kind 'annotated_characterization',
score null by design, with findings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`runStructuralAudit` has been an advertised MCP tool with nothing behind it: the
dispatcher GET'd /session/begin and returned that unrelated session digest under
an audit tool's name. Meanwhile the failure the audit would have caught ran
silently for about three weeks — the soul reporting 103,089 nodes while the
engram, which OWNS persistence, held ~79,900, a crash discarding the difference,
and every boot reporting green throughout, because nothing in the system ever
compared the two sides.
WHAT THE PATENT SPECIFIES, AND HOW IT SHAPED THIS
CGI provisional, 05-detailed-description.md, "Stage 1: Structural audit 430".
Two clauses did the design work. First the four things the module evaluates:
the density and typed distribution of causal edges; value/execution-record
consistency; the richness and connectivity of the self-model; and wonder-
manifest authenticity. Second, and decisively: it "produces a coherence
assessment 432 — NOT A BINARY SCORE but an annotated characterization of the
graph's structural properties."
So every finding carries its numbers AND a plain-language note saying what
they mean and how they were obtained. There is no pass/fail and no composite
health figure, and `"score":null` is emitted explicitly so a reader cannot
mistake its absence for an omission.
WHAT IS IN STAGE 1 (four findings)
owner_runtime_divergence — the motivating case. Runtime counts vs the owner's
own GET /api/stats, the delta, and the trend against the previous audit, so
a second call answers "is the gap growing?" rather than restating it.
self_model_connectivity — the three identity pillars plus the self root:
present, content length, one-hop degree. This RETIRES the Claude-side vitals
identity block, which lived outside the system it was checking and went on
reporting green while the memory-philosophy pillar was absent from the live
graph. Asking the running soul is the designed mechanism; a shell probe was
the fourth patch on the same hole.
typed_edge_distribution — exact counts against the claim-10 vocabulary, plus
density, plus a separate count of LOWERCASE near-misses ("causes" vs
"Causes"): "the vocabulary is unused" and "the vocabulary is misspelled by
the write paths" are different defects with different fixes.
orphans_and_dangling_edges — the tool's own long-standing promise.
WHAT IS DEFERRED, AND WHY IT IS DATA RATHER THAN A COMMENT
Value/execution-record consistency and wonder-manifest authenticity ship as a
`deferred` array that MEASURES the populations they would need (Prediction and
WonderQuestion nodes) and reports those counts as the reason. Both are ~0 today
— WonderQuestion because of a known write/read node-type mismatch. Asserting
value coherence or a pull-weight correlation on an empty population would be a
fabricated result, which is worse than a stated gap.
MEASUREMENT HONESTY: EXACT WHERE CHEAP, SAMPLED WHERE NOT, ALWAYS LABELLED
Counts, edge typing and self-model connectivity are exact. Orphan and dangling
rates are sampled, because engram_find_node_index is a linear scan — an
exhaustive dangling check is O(nodes x edges), ~2.2e9 string compares at today's
scale. Samples are UNIFORM across the whole population (str_index_of_all gives
every edge offset in one pass, so any index is O(1); json_array_get would have
been O(n^2)), never head-of-list, and each figure ships with its own sampled /
population / exhaustive fields. ?edge_sample= and ?node_sample= at population
size run either check exhaustively. The real fix is an id index in the runtime.
ONE BUG THIS FOUND IN ITSELF, CAUGHT IN TEST
http_get does not return "" when the owner is unreachable — it returns a JSON
error object. Testing only for "" made a DEAD owner read as reachable with
node_count 0, so the audit reported 100% divergence and named it data loss.
Reachability is now proved by the presence of the node_count field, and the
owner's raw reply is attached. A confident wrong answer is exactly what this
route exists to stop.
Edge findings need relation labels and the runtime has no edge-enumeration
builtin, so they use the same scratch export GET /api/graph/edges already uses
(engram_save to TMPDIR, never the owner's canonical file — #117). That is a
large write on a large graph, so this is a manual route, not a timer; ?edges=0
skips it.
neuron-api.el:900-1273 handler + helpers
routes.el:567,752 GET and POST /api/neuron/audit/structural
mcp-wrapper/src/main.el:113,682 tool description + dispatch off /session/begin
dist/soul.c regenerated (1255 bodies)
Rung: E2E-VERIFIED. Soul built from this branch (gen-soul-amalgam + cc-brain,
921,192 bytes, 16 warnings, 0 errors), booted on throwaway ports 7893/7896/7897
with throwaway HOMEs against a stub owner on 7894. Three scenarios pass: owner
reachable (runtime 62 vs owner 42, delta 20 / 32.2%, trend flat on the second
call; 12/20 edges claim-10 typed, 3 lowercase near-misses; 50/62 orphans, 3/20
dangling — every figure matches the fixture by construction), owner unreachable
(reported as a finding with the raw reply, not a crash), and file mode (owner
"none", divergence undefined). Reached end-to-end through the MCP tool via a
locally built wrapper. verify-soul-contract.sh: GATE PASS, 27/27 routes +
immutability. No process left running; live :7770 and :8742 untouched (GET only).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Consolidates the 2026-08-09 engine work. Each piece was measured in an isolated lab
against the real corpus, gated, and verified on the operator machine before landing.
WHAT IT CONTAINS
Semantic retrieval + the #146 delete fix, shipped together on purpose. The
semantic leg was once wired into engram_search_json, which seven sites use as a
keyed read and then delete every result from; one runs at every boot. Measured
cost when live: ~234 real records destroyed per startup, identity among them.
Retrieval must never reach production without the engram_recall_json split.
Self-seeded compiled context. The design: 'Every compilation query begins at the
self-model node and traverses outward.' Ours seeded from a hardcoded string, so
compiled context held 0 identity records. Now 18, bounded the same way every other
list in that handler is — the unbounded version is what set self_neighbors to []
after it closed the socket on every call.
Connections on write. CCR claim 29 requires linking candidates with typed edges
'rather than appending as unlinked content'. Unlinked append was the only
behaviour we had: 5% of nodes connected, no edge created by any write in 19 days.
dist/soul.c drift becomes a build failure (#133), and deploys now build from that
same committed input rather than a scratch amalgam — one lineage, provenance
recorded, enforced by a gate in the deployer.
MEASURED, all on the real corpus or the live machine
rephrased-question recall 0/43 -> 20/43 (bar was >=6 queries; nonsense controls held 10/10)
identity in compiled context 0 -> 18
connections per memory 0 -> ~2 on write; 2,828 backfilled into history
no data loss across boots 44,418 / 44,418 / 44,418 over three restarts
FOUR DEFECTS FOUND BY MEASURING RATHER THAN READING, each fixed here
an inert test arm (both arms byte-identical but for a float rounding artifact);
an association hook wired into a path the request never takes (zero edges across
four writes); an identity exclusion matching lowercase 'self' that let a memory
link into the identity graph, whose verification shared the blind spot and printed
PASS; a telemetry filter matching hyphenated 'state-event' that missed
'INTERNAL STATE EVENT'. Common shape: a filter tested a proxy for the property it
cared about. Assert the property.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Until now deploys were built by build-soul.sh from a scratch amalgam while CI
compiled the committed dist/soul.c. Two lineages — 'what runs' and 'what the repo
says builds' were different artifacts. That is #133/#111 in another costume, and it
is how a round-9.1 brain shipped matching no committed source at all.
build-soul-from-dist.sh asserts dist/soul.c matches the .el sources, compiles it
with CI's own flags (-O2 -DHAVE_CURL -rdynamic; the CI comment explains -rdynamic —
without it the runtime cannot resolve its HTTP handler by name and the binary serves
nothing on every route), and writes a .provenance sidecar recording the dist/soul.c
hash, the stamp hash and the commit.
deploy_binary.sh gains GATE 0: refuse any soul without provenance, or built from a
dist/soul.c other than the one in the repo now. The override
(NEURON_DEPLOY_UNSTAMPED=i-accept-two-lineages) exists deliberately — a gate with no
escape hatch gets bypassed by disabling the gate, which is worse than one that
announces itself.
Verified before enforcing, so the sanctioned path is not a broken one:
interface parity with the deployed binary 108 routes in, 108 out
boots and serves 2s
carries today's work 24 self-neighbours, 17 identity records
GATE 0 refuses an unstamped binary exit 10
GATE 0 accepts the stamped one, deployed durability PASS
Production now runs 4845db3d, built from dist/soul.c at 9fd8c11. After the swap:
identity in context 17, connections still form on write, semantic recall intact,
mind and store at delta 0.
Refs #133, #111
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#133 regenerated the amalgam once and said so itself: 'Nothing in the tree
regenerates this file. Only a human running the recipe. It lags in batches, never
per-change, and it will drift again.'
It drifted again. Every binary deployed on 2026-08-09 was built by build-soul.sh
from a scratch amalgam that never touches dist/soul.c, so the committed build
input fell 2,761 bytes behind the sources by a different route than #133 describes.
Auto-regeneration is not available: the CI workflow records that elc needs 24GB+
of virtual memory and would OOM the runner. So the build cannot regenerate the
file. It can refuse to compile a stale one, for free and with no compiler.
tools/soulc-stamp.sh records a content fingerprint of every .el source at the
moment the amalgam is generated. --check recomputes and compares; divergence exits
1 and names the changed files and the recipe. Wired into CI ahead of the compile.
dist/soul.c regenerated from current sources: 1,176,361 -> 1,179,122 bytes, 1,247
inlined bodies (gate wants >=1200), and verified to compile clean at 903,552 bytes.
Demonstrated to FAIL on the bad input, per postmortem 0004's rule that a gate which
only passes on good input proves nothing:
fresh stamp -> OK, exit 0
one .el modified -> FAIL, exit 1, names memory.el
source restored -> OK, exit 0
Refs #133, #111
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The design rejects what this system did: promotion must link candidates 'using
typed semantic edges rather than appending as unlinked content' (CCR claim 29).
Unlinked append was the only behaviour we had. Measured 2026-08-09: 14,214 edges
over 80,936 nodes, 5% of nodes connected to anything, and no edge created by any
write since 2026-07-19 across 27,000+ new nodes. A memory with no connections is
unreachable by spreading activation, so retrieval degrades to literal matching.
On write, a memory is now linked to its top related existing memories.
Bounds, each bought with a specific failure:
- max 3 edges per memory (link_memories.py's cap: precision over spray)
- never link to identity. The existing policy is explicit that 'memories must
not pollute the self traversal by similarity; only an explicit citation may
touch identity'.
- never link telemetry (state-event, soul-response, boot_count, loop-outcome,
search-result): ~97% of daily write volume. Linking it would add thousands of
noise edges a day and re-flatten the graph in the name of connecting it.
- fail-soft: a failed association never fails the write
- associate only AFTER durability, so no edge points at a node that did not
persist — that is the dangling-edge defect the 08-09 cleanup removed 830 of
Two defects found by measuring rather than reading, both fixed here:
1. Hooking mem_store alone produced ZERO edges across four real writes. The HTTP
memory route writes via wt_node directly; mem_store serves only the awareness
telemetry paths we refuse to link.
2. A lowercase-only identity check let a memory link to 'Self — Values
(grounded)' — the exact pollution the policy forbids. My verification shared
the blind spot and printed PASS. Now uses str_lower.
Measured, lab, same corpus: ~1-2 edges per memory; identity edges unchanged at
475; zero identity leaks post-fix including an adversarial batch of six memories
written about values. Edges route through wt_edge so they reach the owner.
Rung: E2E-VERIFIED in an isolated lab. Not deployed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The design is explicit — 'Every compilation query begins at the self-model node
and traverses outward... structural reachability from the self-model node is a
precondition for any node to appear in compiled context.' (will-anderson
patents/drafts/engram-claims.md, Self-Seeded Activation. DRAFT, not a filed
provisional — cited as such.)
Measured before: compiled context held 0-1 identity records out of 10, because
compilation seeds from a hardcoded text string and never from the self. Even an
explicit 'my values identity who I am' query returned a boot counter and
state-events.
This restores the designed behaviour without repeating the failure that set
self_neighbors to [] originally: that was an unbounded ~90KB neighbour dump which
closed the socket on every call. Same bound as every other list in the handler.
Cap is 24 rather than 8 because the self root's first 8 neighbours are tag nodes
('neuron', 'tier:note', 'disposition:experimental', 'imprint', 'traversal') that
crowd out the substantive identity records behind them. The root has 34
neighbours of which 23 are identity.
Measured after, same corpus, binary as the only variable:
identity records in compiled context 0 -> 17
response size 10,806 -> 18,985 bytes
interface gate 108 routes in, 108 out, none removed
Rung: E2E-VERIFIED in an isolated lab. Not deployed — this is the identity layer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Today's review found the JSON parser had been replacing every \uXXXX escape
with a literal '?' for at least two months: 3,119 of 4,081 non-telemetry
nodes damaged, including the self traversal root and all 13 values nodes.
The parser is fixed in el_runtime.c. This is the part that keeps it fixed.
Every gauge on this heartbeat answers whether the machinery is running - WM
occupancy, Hebbian potentiation, embedding coverage, sync age, breaker state.
None answered whether the text the machinery carries is intact, which is why
two months of silent corruption read as a perfectly healthy system.
Adds txt_damaged: nodes created this process whose content carries the
character-loss signature. Flat 0 is healthy; any climb means a write path is
mangling text again. The full store census lives at GET /api/text-health,
which is too expensive for a 60s beat.
A gauge that exists but is not plumbed into the beat is not observability -
act_stats fields have to be extracted by name here or they are invisible.
The accumulated retrieval stack (iterations 1-9) put the claim-24 semantic
leg and the claim-10 associative leg on engram_search_json — the function
~40 internal .el call sites already used as a KEYED read. Seven of those
sites delete every record that comes back ("prune all existing X nodes,
keep exactly one"): memory.el:176, sessions.el:250/268/444/523,
soul.el:359.
mem_boot_count_inc() calls engram_search_json("soul:boot_count", 50) and
engram_forget()s all 50 results. With a lexical leg that returned 1 record.
With a semantic leg it returns 50 — the 49 nearest neighbours of the STRING
"soul:boot_count" — and the soul deletes them.
MEASURED on the harness corpus, isolated, read-only, zero writes from any
caller: 234 node records destroyed in a single boot. The deletion list is
the soul's own lookup result list, in rank order. Casualties include 6
Knowledge nodes, a layer-1 "CORE IDENTITY - GENESIS, LINEAGE" Memory, the
value node kn-58874a74, and the gold answers to 8 of the 75 gold-set
queries. After the fix: 1 deletion, which is the one the code intends.
THE BOUNDARY, from Will. Claim 24 authorises the vector index "to respond
to EMBEDDING SEARCH QUERIES by returning the node records whose embedding
vectors have the highest cosine similarity to a query vector". A keyed
state read is not an embedding search query; it is the identifier-keyed
retrieval of claim 23 ("node records are stored under a key encoding the
node identifier"). One function served both, so a nearest neighbour of
"soul:boot_count" was treated as a boot counter.
So: engram_search_json returns to its lexical contract, and the legs move
to engram_recall_json, which is what /api/neuron/recall reaches — the route
the MCP wrapper, the app, and this harness all call. Retrieval quality on
that route is unchanged by construction.
MEASURED, 75-query extended gold set, embedded corpus, vs the iteration-9
baseline: +3 / -0 (q15, q28, q60), p=0.2500, hit@5 53.8 -> 58.5%, latency
1.02x, every regression guard held, nonsense 10/10. Net +3 against a floor
of 6 is NOT-SHOWN and I am not calling it an improvement. The deliverable
is the defect.
Diagnostics kept, env-gated (EG_DIAG / EG_DIAG_ID), zero cost when unset:
node/embedding census at load, per-query leg dump, and a FORGET log — the
last is the regression detector for exactly this class of bug.
LIMIT, stated: handle_api_search_knowledge still uses the lexical function.
It is a retrieval surface and arguably wants the legs, but nothing in this
harness measures it, so I did not change unmeasured behaviour.
Measured on the 75-query extended gold set (iteration 8's held-out extension)
against the certified stack baseline results-stack-ext.json, on the embedded
corpus. Three runs of the candidate, zero drift.
A. CLAIM 24 WITHOUT THE THRESHOLD - net +4, NOT-SHOWN, kept in the tree.
fixed : q14, q25 (in-sample paraphrase), q43, q52, q63, q67 (held-out)
broken : q15 (paraphrase), q28 (associative)
8 discordant, McNemar exact p = 0.2891, floor is 6.
heldout_paraphrase 16.7% -> 30.0%, paraphrase 61.5% -> 69.2%.
Every regression guard held: exact_rare 6/6, phrase 7/7, nonsense 10/10,
superseded 2/3. Latency FLAT: p50 641 -> 632 ms.
The in-sample half (+q14 +q25 -q15 -q28 = 0) was already on record in
iteration 7's cmp-nogate.json, so only the held-out +4 is new.
B. ASYMMETRIC TASK PREFIXES ON THE EMBEDDER - net -5, REVERTED in this commit.
Rationale was sound and the prediction was wrong, which is why it was worth
measuring: nomic-embed-text is an asymmetric retrieval encoder and this file
embedded query and document bare on both sides. Prefixing does exactly what
the model card implies for the far-away cases - it rescued q42 (gold at
GLOBAL COSINE RANK 25,564) and q39 - but it re-ranks the whole space and
broke more than it fixed:
fixed : q24, q39, q42
broken : q18, q19, q22, q31, q43, q44, q52, q63
heldout_paraphrase 30.0% -> 23.3%, paraphrase 69.2% -> 53.8%.
The corpus and the reproducer are kept (embed-corpus-prefixed.py,
snapshot-pre-repair-20260806-embedded-prefixed.json) so nobody re-runs it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two changes to the ONE leg that generalises. Iteration 8 measured that out of
sample the semantic leg contributes 100% of the stack's gain and the graph leg
contributes nothing, so this is where the remaining headroom is.
1. THE 0.60 FLOOR IS A PER-QUERY LOTTERY, AND CLAIM 24 HAS NO THRESHOLD IN IT.
06-claims.md l.148: "respond to embedding search queries by returning the
node records whose embedding vectors have the HIGHEST COSINE SIMILARITY to a
query vector, independently of the spreading activation traversal." A
ranking. ENGRAM_EMBED_SEED_MIN is defined at el_runtime.c l.6094 as the
HippoRAG seed-JOIN threshold and l.6102 admits the read-path leg merely
"reuses" it. Measured on the 30 held-out paraphrases: the query's own top-1
cosine ranges 0.564-0.680, so the constant keeps a rank-1 answer for one
query and discards a rank-1 answer for the next. Six golds sit at global
cosine rank 1-2 scoring 0.564-0.589 - discarded by nothing but the constant.
What holds the nonsense controls is the corpus-vocabulary gate (nhits == 0),
not this floor. Cosine clamped to [0,1] per 05-detailed-description l.69.
2. THE VECTORS THEMSELVES ANSWER THE WRONG QUESTION. EL_EMBED_MODEL defaults to
nomic-embed-text, an ASYMMETRIC retrieval encoder trained with task prefixes.
Embedding query and document bare - as this file did on both sides - measures
topical similarity rather than answer-hood. eg_embed_fetch now takes the task
prefix: EL_EMBED_QUERY_PREFIX on the three query call sites, EL_EMBED_DOC_PREFIX
on the two backfill sites. Restores no claim, and says so: Will specifies only
"computed by an embedding model over the node's content" (l.17), so the model
is his and its correct use is ours. It is the substrate under claim 24 -
the index is only as good as the vectors in it.
Reproducer for the derived corpus: tools/retrieval-eval/embed-corpus-prefixed.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The retrieval match test is a raw substring scan, so a query token matches
anywhere INSIDE a corpus word: "throom" matches "bathroom". Measured over the
38-query gold set on this corpus that is not a rare accident - q28's lexical
leg is 36,954 records of which only 13 contain a query token at a word start
(99.96% mid-word noise), six further queries carry ~20,500 mid-word-only
records each, and the nonsense control q35 returns 7 records ALL of which
match only mid-word.
istr_contains_wordstart() anchors a token to a word start (preceding char not
alphanumeric) while still matching suffixes, so "value" still hits "values".
That empties the lexical leg for gibberish, and the nhits==0 corpus-vocabulary
gate (iteration 6's mechanism, feat/claim24-unfloored-semantic) then makes the
whole query decline rather than let the semantic leg answer it.
Measured vs feat/bm25-lexical-leg on the embedded corpus, 2 runs each,
0 queries of run-to-run drift on both sides:
net +1 (nonsense:q35), 0 losses, McNemar p=1.0 -> NOT-SHOWN (floor is 6)
nonsense clean 2/3 -> 3/3; exact_rare 100%, phrase 100%, paraphrase 61.5%,
associative 66.7%, superseded 2/3 all UNCHANGED
latency p50 1184 -> 543 ms (0.46x)
Iteration 6 called q35 "a DEFECTIVE CONTROL ... cannot be cleaned without
breaking the lexical leg". It can: the defect was the match primitive, and
cleaning it cost nothing.
Also committed: results-wsclaim24.json + cmp-nogate.json, a measured negative
for bundling the claim-24 unfloored semantic leg on top (gains q14/q25, breaks
q15/q28/q33/q34, net -2) - it independently reproduces iteration 6's q15/q28
losses and shows unflooring REQUIRES the vocabulary gate.
Reproducers: legs.py (leg-level replica, reproduces baseline hit@5 exactly on
all 38 queries), policy2.py, ceiling.py, wb2.py.
hit@5 68.6% -> 74.3%, phrase 71.4% -> 100%, MRR@10 0.461 -> 0.502, latency
p50 0.97x. Zero losses, zero run-to-run drift on both sides. 2 queries moved
against a 6-query noise floor: NO MEASURABLE DIFFERENCE by the harness's own
test (McNemar exact p=0.50). Unaddressable records in returned slots: 57 -> 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
engram_search_json ranked its lexical leg by raw distinct-token coverage with
salience as tiebreak: a token in 30,000 nodes counted the same as a token in 1,
and a 1.3 MB record matched nearly every query token by surface area alone.
Score it BM25-shaped instead - Lucene-form IDF and length normalisation over
the corpus mean - with per-token document frequency accumulated in the SAME
corpus pass that finds the hits (no extra scan, no extra round-trip).
Also refuse to return records whose identifier is not printable ASCII. This
corpus carries 1,032 such records (453 by the printable test) from a save-side
corruption; they occupy 125 of 303 returned slots on main. Claims 12, 23 and 27
all key on the node identifier, so such a record is unfetchable by any caller.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
engram_assoc_leg previously took its seeds only from the top-3 LEXICAL hits.
For a paraphrase query the lexical hits are noise by construction, so the walk
never reached the neighbourhood that holds the answer. This adds the seeding
pass Will documents at el_runtime.c l.6082 — "Semantic seeding (HippoRAG
pattern, use similarity twice): the query is embedded, the top-K nodes by
cosine join the seed set" — using his own ENGRAM_EMBED_SEED_K (8).
Similarity is now used twice, coherently: cosine picks where to STAND in the
graph, the structural-relation walk decides what is REACHABLE, and cosine
orders what was reached (iteration 2's finding, unchanged).
The seed list is deliberately NOT floored at ENGRAM_EMBED_SEED_MIN. Measured
over all 38 gold queries: true paraphrase targets score cosine 0.46-0.66 and
the three nonsense controls' own nearest neighbours score 0.55/0.60/0.62 —
the distributions OVERLAP, so no absolute cosine floor separates signal from
gibberish. The gate that works is reachability: gibberish's nearest neighbours
carry no structural edge, so its graph leg is empty and the controls hold.
The raw top-K is selected inside the existing scoring pass, so the cosine is
computed exactly once per node: no extra corpus pass, no extra embed
round-trip, latency flat (p50 1220 -> 1227 ms, 1.01x).
Measured vs the certified baseline feat/hybrid-semantic-recall, embedded
corpus, 2 runs each, zero run-to-run drift on both sides:
hit@5 51.4% -> 68.6% MRR@10 0.387 -> 0.461
paraphrase 38.5% -> 61.5% associative 0% -> 66.7%
exact_rare 100% held, nonsense 2/3 held, superseded 2/3 held
phrase 85.7% -> 71.4% (q11, the known rank-5 rotation tax)
net +6 queries (7 fixed / 1 broken), McNemar p=0.0703
The recall route had no way to reach a node that shares no token and no
embedding neighbourhood with the query. The design reserves that case for the
graph, and nothing on the read path consulted an edge.
This adds a third ranked leg beside the lexical and semantic ones: expand the
top 3 lexical hits along STRUCTURAL relations only (claim 10 — identity,
contains, superseded_by, references, ...), two hops, both directions, pruned
at the same 0.02 firing threshold engram_activate uses; order what was reached
by query similarity. Merged by strict rotation, never by score blending.
Not PR #135. That wired recall wholesale to engram_activate and lost 57 points
of phrase accuracy. The failure there was RANK, not reach — a 2-hop associate
at strength 0.06 cannot outrank thousands of 1-hop neighbours of strong
lexical seeds. Here the lexical leg is untouched and the associative list is
empty for most queries, because a node whose only edges are `tagged` and
`related` expands to nothing.
MEASURED, hybrid-semantic baseline -> this, 38-query gold set, embedded corpus:
associative 0.0% -> 66.7% (first non-zero ever recorded on that category)
hit@5 51.4% -> 62.9%
exact_rare, phrase, paraphrase, nonsense, superseded: all unchanged
latency p50 1.01x
4 queries moved, all gains, 0 losses, McNemar p=0.125
deterministic: two runs of the same binary differ on 0 of 38 rows
VERDICT: NOT-SHOWN. The harness needs 6 queries to clear p<0.05 and the whole
associative category is only 6 queries, so even 4/6 fixed cannot reach the
floor. The mechanism is confirmed to work; the gold set cannot certify it.
Replaces the score-fusion first cut with rank fusion, which is what the data
called for. nomic's cosine scale is compressed (true matches 0.55-0.70,
unrelated pairs 0.35-0.50), so an additive blend of cosine onto token-coverage
is dominated by whichever leg has the wider spread. Alternation is invariant to
both scales:
L1, S1, L2, S2, ... deduped, capped at limit
Lexical ranking is left byte-identical; the semantic ranking is computed beside
it and admitted only above ENGRAM_EMBED_SEED_MIN (0.60) — Will's existing seed
floor, no new tuning constant. That floor is what keeps the nonsense controls
clean: a query with no real match must not be answered with its neighbours.
embed-corpus.py / merge-corpus.py produce the derived corpus the semantic leg
needs (76,986 vectors, nomic-embed-text, 0 failures, 11 min). Zero of 78,791
nodes carried an embedding before this; the field round-tripped through the
snapshot but nothing ever wrote it.
MEASURED, 38-query gold set, paired against the SAME derived corpus so the
comparison isolates the code change:
hit@5 34.3% -> 51.4% paraphrase 0.0% -> 38.5%
MRR@10 0.294 -> 0.387 superseded 1/3 -> 2/3 outranks
recall@10 33.3% -> 50.5% latency p50 1146 -> 1220ms (1.06x)
exact_rare 100% -> 100% phrase 85.7% -> 85.7% nonsense 2/3 -> 2/3
6 queries fixed, 0 broken, McNemar exact p=0.0312, 0 drift across repeats.
Regression guards all held. Contrast PR #135, which swapped the read path to
spreading activation wholesale: phrase 85.7 -> 28.6, latency 2.81x. Correct
mechanism, wrong substrate. The substrate is now present.
Restores engram claim 24 (previously 0% honoured).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
engram_search_json — the function /api/neuron/recall actually reaches — ranked
only by distinct-token match count, so the embedding field on every node record
was inert. Add the semantic leg as a UNION beside the lexical one, not a
replacement for it:
fused = (distinct_tokens_matched / query_tokens) + 0.90 * sem
sem = clamp01((cos(q,n) - 0.60) / (1 - 0.60)) ; 0 when not comparable
Holding the semantic weight strictly below 1.0 means a node matching every
query token can never be displaced by semantics alone — the regression guard
that PR #135 lacked when it swapped the read path to spreading activation and
took phrase recall from 85.7% to 28.6%.
No query embedding (embedder down, circuit breaker open) => sem == 0 for all
nodes => fused == sc/ntok, a monotone map of the old integer score, so the
ordering degrades to the historical behaviour exactly.
Restores engram claim 24: 'maintain a vector similarity index over the semantic
embedding vectors of all stored node records, and ... respond to embedding
search queries by returning the node records whose embedding vectors have the
highest cosine similarity to a query vector, independently of the spreading
activation traversal.'
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
will.anderson
Neuron
tim.lingo