Commit Graph

345 Commits

Author SHA1 Message Date
Tim Lingo 4eb4c9e287 feat(engram): word-start match primitive + corpus-vocabulary gate on recall
The retrieval match test is a raw substring scan, so a query token matches
anywhere INSIDE a corpus word: "throom" matches "bathroom". Measured over the
38-query gold set on this corpus that is not a rare accident - q28's lexical
leg is 36,954 records of which only 13 contain a query token at a word start
(99.96% mid-word noise), six further queries carry ~20,500 mid-word-only
records each, and the nonsense control q35 returns 7 records ALL of which
match only mid-word.

istr_contains_wordstart() anchors a token to a word start (preceding char not
alphanumeric) while still matching suffixes, so "value" still hits "values".
That empties the lexical leg for gibberish, and the nhits==0 corpus-vocabulary
gate (iteration 6's mechanism, feat/claim24-unfloored-semantic) then makes the
whole query decline rather than let the semantic leg answer it.

Measured vs feat/bm25-lexical-leg on the embedded corpus, 2 runs each,
0 queries of run-to-run drift on both sides:
  net +1 (nonsense:q35), 0 losses, McNemar p=1.0 -> NOT-SHOWN (floor is 6)
  nonsense clean 2/3 -> 3/3; exact_rare 100%, phrase 100%, paraphrase 61.5%,
  associative 66.7%, superseded 2/3 all UNCHANGED
  latency p50 1184 -> 543 ms (0.46x)

Iteration 6 called q35 "a DEFECTIVE CONTROL ... cannot be cleaned without
breaking the lexical leg". It can: the defect was the match primitive, and
cleaning it cost nothing.

Also committed: results-wsclaim24.json + cmp-nogate.json, a measured negative
for bundling the claim-24 unfloored semantic leg on top (gains q14/q25, breaks
q15/q28/q33/q34, net -2) - it independently reproduces iteration 6's q15/q28
losses and shows unflooring REQUIRES the vocabulary gate.

Reproducers: legs.py (leg-level replica, reproduces baseline hit@5 exactly on
all 38 queries), policy2.py, ceiling.py, wb2.py.
2026-08-07 16:58:34 -05:00
Tim Lingo 55f9ee3cb0 measure: BM25 lexical leg vs semseed baseline - net +2 (q10,q11), NOT-SHOWN
hit@5 68.6% -> 74.3%, phrase 71.4% -> 100%, MRR@10 0.461 -> 0.502, latency
p50 0.97x. Zero losses, zero run-to-run drift on both sides. 2 queries moved
against a 6-query noise floor: NO MEASURABLE DIFFERENCE by the harness's own
test (McNemar exact p=0.50). Unaddressable records in returned slots: 57 -> 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 15:55:42 -05:00
Tim Lingo 9c39084e60 feat(engram): BM25-shaped lexical leg + addressability guard on the read path
engram_search_json ranked its lexical leg by raw distinct-token coverage with
salience as tiebreak: a token in 30,000 nodes counted the same as a token in 1,
and a 1.3 MB record matched nearly every query token by surface area alone.
Score it BM25-shaped instead - Lucene-form IDF and length normalisation over
the corpus mean - with per-token document frequency accumulated in the SAME
corpus pass that finds the hits (no extra scan, no extra round-trip).

Also refuse to return records whose identifier is not printable ASCII. This
corpus carries 1,032 such records (453 by the printable test) from a save-side
corruption; they occupy 125 of 303 returned slots on main. Claims 12, 23 and 27
all key on the node identifier, so such a record is unfetchable by any caller.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 15:51:49 -05:00
Tim Lingo 6f3a048f36 feat(engram): semantically seed the graph leg (Will's HippoRAG pass, SEED_K=8)
engram_assoc_leg previously took its seeds only from the top-3 LEXICAL hits.
For a paraphrase query the lexical hits are noise by construction, so the walk
never reached the neighbourhood that holds the answer. This adds the seeding
pass Will documents at el_runtime.c l.6082 — "Semantic seeding (HippoRAG
pattern, use similarity twice): the query is embedded, the top-K nodes by
cosine join the seed set" — using his own ENGRAM_EMBED_SEED_K (8).

Similarity is now used twice, coherently: cosine picks where to STAND in the
graph, the structural-relation walk decides what is REACHABLE, and cosine
orders what was reached (iteration 2's finding, unchanged).

The seed list is deliberately NOT floored at ENGRAM_EMBED_SEED_MIN. Measured
over all 38 gold queries: true paraphrase targets score cosine 0.46-0.66 and
the three nonsense controls' own nearest neighbours score 0.55/0.60/0.62 —
the distributions OVERLAP, so no absolute cosine floor separates signal from
gibberish. The gate that works is reachability: gibberish's nearest neighbours
carry no structural edge, so its graph leg is empty and the controls hold.

The raw top-K is selected inside the existing scoring pass, so the cosine is
computed exactly once per node: no extra corpus pass, no extra embed
round-trip, latency flat (p50 1220 -> 1227 ms, 1.01x).

Measured vs the certified baseline feat/hybrid-semantic-recall, embedded
corpus, 2 runs each, zero run-to-run drift on both sides:
  hit@5 51.4% -> 68.6%   MRR@10 0.387 -> 0.461
  paraphrase 38.5% -> 61.5%   associative 0% -> 66.7%
  exact_rare 100% held, nonsense 2/3 held, superseded 2/3 held
  phrase 85.7% -> 71.4% (q11, the known rank-5 rotation tax)
  net +6 queries (7 fixed / 1 broken), McNemar p=0.0703
2026-08-07 15:35:08 -05:00
Tim Lingo 059ce02003 feat(engram): an associative leg on the recall path (claim 10 typed relations)
The recall route had no way to reach a node that shares no token and no
embedding neighbourhood with the query. The design reserves that case for the
graph, and nothing on the read path consulted an edge.

This adds a third ranked leg beside the lexical and semantic ones: expand the
top 3 lexical hits along STRUCTURAL relations only (claim 10 — identity,
contains, superseded_by, references, ...), two hops, both directions, pruned
at the same 0.02 firing threshold engram_activate uses; order what was reached
by query similarity. Merged by strict rotation, never by score blending.

Not PR #135. That wired recall wholesale to engram_activate and lost 57 points
of phrase accuracy. The failure there was RANK, not reach — a 2-hop associate
at strength 0.06 cannot outrank thousands of 1-hop neighbours of strong
lexical seeds. Here the lexical leg is untouched and the associative list is
empty for most queries, because a node whose only edges are `tagged` and
`related` expands to nothing.

MEASURED, hybrid-semantic baseline -> this, 38-query gold set, embedded corpus:
  associative  0.0% -> 66.7%   (first non-zero ever recorded on that category)
  hit@5       51.4% -> 62.9%
  exact_rare, phrase, paraphrase, nonsense, superseded: all unchanged
  latency p50 1.01x
  4 queries moved, all gains, 0 losses, McNemar p=0.125
  deterministic: two runs of the same binary differ on 0 of 38 rows

VERDICT: NOT-SHOWN. The harness needs 6 queries to clear p<0.05 and the whole
associative category is only 6 queries, so even 4/6 fixed cannot reach the
floor. The mechanism is confirmed to work; the gold set cannot certify it.
2026-08-07 15:17:27 -05:00
Neuron 635453b936 feat(engram): rank-interleave the semantic leg into recall; embed the corpus
Replaces the score-fusion first cut with rank fusion, which is what the data
called for. nomic's cosine scale is compressed (true matches 0.55-0.70,
unrelated pairs 0.35-0.50), so an additive blend of cosine onto token-coverage
is dominated by whichever leg has the wider spread. Alternation is invariant to
both scales:

  L1, S1, L2, S2, ...  deduped, capped at limit

Lexical ranking is left byte-identical; the semantic ranking is computed beside
it and admitted only above ENGRAM_EMBED_SEED_MIN (0.60) — Will's existing seed
floor, no new tuning constant. That floor is what keeps the nonsense controls
clean: a query with no real match must not be answered with its neighbours.

embed-corpus.py / merge-corpus.py produce the derived corpus the semantic leg
needs (76,986 vectors, nomic-embed-text, 0 failures, 11 min). Zero of 78,791
nodes carried an embedding before this; the field round-tripped through the
snapshot but nothing ever wrote it.

MEASURED, 38-query gold set, paired against the SAME derived corpus so the
comparison isolates the code change:

  hit@5      34.3% -> 51.4%     paraphrase   0.0% -> 38.5%
  MRR@10     0.294 -> 0.387     superseded   1/3  -> 2/3 outranks
  recall@10  33.3% -> 50.5%     latency p50  1146 -> 1220ms (1.06x)

  exact_rare 100% -> 100%   phrase 85.7% -> 85.7%   nonsense 2/3 -> 2/3

  6 queries fixed, 0 broken, McNemar exact p=0.0312, 0 drift across repeats.

Regression guards all held. Contrast PR #135, which swapped the read path to
spreading activation wholesale: phrase 85.7 -> 28.6, latency 2.81x. Correct
mechanism, wrong substrate. The substrate is now present.

Restores engram claim 24 (previously 0% honoured).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 14:59:39 -05:00
Neuron 315b2eff00 feat(engram): fuse cosine similarity into the recall read path (claim 24)
engram_search_json — the function /api/neuron/recall actually reaches — ranked
only by distinct-token match count, so the embedding field on every node record
was inert. Add the semantic leg as a UNION beside the lexical one, not a
replacement for it:

  fused = (distinct_tokens_matched / query_tokens) + 0.90 * sem
  sem   = clamp01((cos(q,n) - 0.60) / (1 - 0.60))     ; 0 when not comparable

Holding the semantic weight strictly below 1.0 means a node matching every
query token can never be displaced by semantics alone — the regression guard
that PR #135 lacked when it swapped the read path to spreading activation and
took phrase recall from 85.7% to 28.6%.

No query embedding (embedder down, circuit breaker open) => sem == 0 for all
nodes => fused == sc/ntok, a monotone map of the old integer score, so the
ordering degrades to the historical behaviour exactly.

Restores engram claim 24: 'maintain a vector similarity index over the semantic
embedding vectors of all stored node records, and ... respond to embedding
search queries by returning the node records whose embedding vectors have the
highest cosine similarity to a query vector, independently of the spreading
activation traversal.'

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 14:44:42 -05:00
Neuron cf41d12d22 test(retrieval): a measurement harness for memory recall, and its first verdict
Neuron Soul CI / build (pull_request) Failing after 14m41s
Neuron Soul CI / deploy (pull_request) Failing after 14m45s
Nothing else on the memory roadmap should be built until a change can be shown
to help. Right now we judge by feel, and the benchmark literature is full of
systems that felt better and measured worse. This is the missing gate.

WHAT IT MEASURES, AND WHY IT BOOTS A REAL SOUL
The subject is Will's designed retrieval — spreading activation over the
weighted directed graph, four-factor multiplicative scoring — not a proxy for
it. A Python re-implementation would measure my reading of the design, so the
harness compiles the actual soul.el amalgam from a git ref and asks it over
HTTP on /api/neuron/recall, exactly as the MCP wrapper and the app do.

BUILT ON WHAT WAS ALREADY HERE, NOT AROUND IT
  docs/research/graphrag_eval/{collect,score}.py  — per-query relevant-id
    scoring and fixed-denominator precision@5 (kept verbatim: an empty result
    should be punished like a page of junk).
  docs/research-archive/p0-prototypes/eval_pinned_40q_20260715.py — the pinned
    ground truth + --check winnability gate, so every run judges alike.
  scripts/verify-soul-contract.sh — the isolation recipe, including the
    non-obvious SOUL_ISE_URL pin without which an "isolated" soul silently
    syncs the operator's live brain.
  gen-soul-amalgam.sh + .gitea/workflows/ci.yaml — the build recipe and flags.
New here: ids rather than regexes as ground truth, an associative category
derived from real edges, a superseded category scored on ranking, a
machine-checked zero-lexical-overlap guarantee on paraphrases, paired
significance testing, and measurement of the real compiled soul rather than an
offline replica of one leg of it.

THE GOLD SET IS AUDITABLE, NOT VIBES
38 queries over the real 78,768-node corpus, each carrying a `derivation`
string, each re-validated by `build_gold_set.py --check`. exact_rare is mined
(document frequency 1). phrase is mined (verbatim scan; >25 matches rejected as
too diffuse). paraphrase is hand-selected then PROVEN to share zero content
words with its target — a leak fails the build, so the category cannot decay
into lexical matching. associative is derived from real hub edges with
lexically-reachable siblings dropped. nonsense is verified absent. superseded
pairs are kept only when both sides survive as distinct nodes.

HONEST ABOUT NOISE
Minimum detectable swing on 38 queries is 6: if every changed query moves the
same way, p = 2*0.5^n first clears 0.05 at n=6. Run-to-run drift is measured,
not assumed — activation is a stateful read, and it shows: main is fully
deterministic across 3 runs, the candidate drifts by 1 query. compare.py
reports "no measurable difference" for anything inside max(6, drift+1).

FIRST VERDICT — feat/recall-through-activation
hit@5 34.3% -> 22.9%, phrase 85.7% -> 28.6%, latency p50 2.81x. Five discordant
pairs, all five against the candidate, none for it; McNemar exact p = 0.0625,
so by the stated rule this is one query short of significant and is reported as
such rather than as a win for main. The latency regression is deterministic and
not in any noise band.

The benefit the branch was written for is absent: associative recall is 0/6 on
BOTH builds. Probed directly, the traversal returns the lexical seed at rank 8
and none of its 12 hub siblings. Two measured corpus facts explain it — only
4,060 of 78,768 nodes (5.2%) carry any edge, and no node has an embedding, so
the fourth factor of the four-factor product has nothing to compute from. The
mechanism runs; the corpus lacks the structure it needs.

SAFETY
Throwaway port, throwaway HOME, disposable per-run copy of the corpus; live
ports refused by name. Every soul started is killed AND confirmed dead by pid
probe, with the confirmation written into the results file; run_comparison.sh
sweeps for strays and exits non-zero if any survive. Nothing under ~/.neuron,
/Applications/Neuron*, or ~/neuron-dev-stack is read, written, or restarted.

Rung: E2E-VERIFIED — 6 full runs (3 per config) against the real compiled
binaries on the real corpus; numbers above are measured, not projected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 13:40:51 -05:00
tim.lingo 18714e6142 Merge pull request 'fix(engine): restore multi-turn crisis escalation on the agentic path (P0, closes #129)' (#130) from fix/129-history-amplification into main
Neuron Soul CI / build (push) Failing after 14m37s
Neuron Soul CI / deploy (push) Has been skipped
2026-08-07 15:54:41 +00:00
tim.lingo 4936099c39 Merge pull request 'fix(engine): the daemon survives a client leaving, and says it is working while it works' (#127) from fix/liveness-engine-91 into main
Neuron Soul CI / build (push) Has been cancelled
Neuron Soul CI / deploy (push) Has been cancelled
2026-08-07 15:54:15 +00:00
tim.lingo f1471763f5 Merge pull request 'fix(engine): approving a researched mission completes — the resume replay read a tool id out of the conversation (BUG-42, both faces)' (#115) from fix/resume-server-tool-replay into main
Neuron Soul CI / build (push) Has been cancelled
Neuron Soul CI / deploy (push) Has been cancelled
2026-08-07 15:53:51 +00:00
tim.lingo 5850793b67 Merge pull request 'fix(engine): history keeps its provenance and its session — kills the false confession, the blank stare, and the "to.Good" seams' (#114) from fix/soul-history-provenance-20260805 into main
Neuron Soul CI / build (push) Has been cancelled
Neuron Soul CI / deploy (push) Has been cancelled
2026-08-07 15:53:32 +00:00
tim.lingo fc1745c652 Merge pull request 'feat(engine): plain chat generates at L3 — inside the safety cycle, not around it (+ crisis-path segfault fix)' (#109) from feat/soul-plain-chat-generation-20260805 into main
Neuron Soul CI / build (push) Failing after 10m39s
Neuron Soul CI / deploy (push) Failing after 14m47s
2026-08-07 15:53:08 +00:00
Tim Lingo 43d0449904 fix(engine): the agentic crisis screen reads the session's own history again
P0 SAFETY. Closes the regression we introduced in ff421d3 (2026-08-05).

ff421d3 correctly moved conversation history to a per-session key via
conv_hist_key(session_id). One consumer did not move with it: the agentic
path's L1 safety screen kept reading the anonymous "conv_history" bucket. The
desktop app always mints a session id (DaemonClient.kt:706), so history was
always written under session_hist_<id> and that read always returned "".

The half of the crisis score that receives history is the escalation half — the
one that exists for distress building across several turns, where no single
message trips the bell on its own. It scored 0 on every real conversation for
two days. Single-message hard bell was never affected.

The bitter part: the comment that line carried documented this exact bug being
fixed once already, under issue #9. The fix was right then. The rename
re-broke it, and the comment went on describing a repair that no longer held.
A comment is not a gate.

The read now goes through conv_hist_key like every other consumer, including
the plain path at soul.el:398 and the thread-anchoring read thirty lines below
it in this same handler. It is one line. The rest of this commit is structure
so it cannot happen quietly again:

  - agentic_safety_screen() owns the two decisions that were inline — which
    window the screen sees, and the screen call. Inline safety inputs are
    untestable safety inputs; that is what let a rename starve this one with
    nothing failing and nothing logging.
  - the comment above the call site now states the invariant (read window ==
    written window) instead of naming a key that can be renamed out from under
    it.

TWO-LEG PROOF, one variable — the single line state_get("conv_history") ->
state_get(conv_hist_key(session_id)):

  before  scripts/run-el-test.sh tests/test_history_amplification.el
          3. REGRESSION #129 ... FAIL  got: soft_bell  expected: hard_bell
          8 passed, 1 failed          runner exit 1
  after   same command, same tree, that one line changed
          9 passed, 0 failed          runner exit 0

Full engine rebuild from these sources is clean: gen-soul-amalgam.sh ->
1,164,103 bytes / 1226 inlined bodies (gate wants >= 1200), cc-brain.sh ->
903,096 bytes, 0 errors. agentic_safety_screen and conv_hist_key both present
in the built binary (nm: T _agentic_safety_screen, T _conv_hist_key).

Rung reached: BUILT + RUNS (discriminating test). NOT yet in a DMG and not yet
verified in the app a human opens — those are the next two rungs and neither is
claimed here.

Known and NOT fixed by this commit:
  - feat/soul-openai-tools-v2 carries the same defect independently at
    chat.el:2937 and needs the same change or a merge.
  - the defect CLASS (a read of a state key no producer writes) is still
    invisible to every gate we have. Issue #129 proposes making it a build
    error; that is the follow-on.

Closes #129

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 09:33:05 -05:00
Tim Lingo b842e82f77 test(engine): a runner for tests/, and a failing regression test for #129
tests/ has held 14 test programs for months with no way to run them. CI does
not run them. The convention printed in their own headers
(`elc soul.el && ./soul --test tests/x.el`) refers to a --test flag the El
runtime does not implement. So the tests were documentation, not gates — which
is how a P0 safety regression shipped with a test directory sitting right
there.

scripts/run-el-test.sh compiles and runs one test program. It reuses the
gen-soul-amalgam.sh discovery: elc emits only an extern prototype for a module
that has a .elh beside it, and inlines the bodies when it does not, so a test
importing ../chat.el must be compiled in a scratch tree with the headers
removed. Scratch copy on purpose — the worktree is shared. It runs the binary
under a throwaway HOME so a test can never reach the live engram.

Exit status is the gate: the El tests print failures and still exit 0, so the
runner greps for FAIL lines and for a zero assertion count as well.

tests/test_history_amplification.el pins the invariant #129 violated: the
window the safety screen READS must be the window conv_history_record WRITES.
Not "must be called conv_history" — must AGREE.

THIS COMMIT IS RED BY DESIGN. On this tree the test fails one assertion:

  3. REGRESSION #129 — agentic screen reads the session's own window
    FAIL: distress history escalates the agentic screen to hard_bell
      got:      soft_bell
      expected: hard_bell
  history amplification tests: 8 passed, 1 failed   (runner exit 1)

The next commit turns it green by changing one line. Two legs, one variable —
that is the whole point of committing the test first.

Two flaws in the older harness that this one does not copy: the idiom
`let pass_count = pass_count + 1` inside an assert function declares a local
that dies with the call, so every existing suite prints "0 passed, 0 failed"
regardless of outcome; and a test program without a `cgi` block compiles as a
'utility', which may not reference the self-formation primitives chat.el's
agentic loop calls — it fails to build on a capability violation it never
triggers at runtime.

Refs #129

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 09:32:40 -05:00
Tim Lingo 98ccbd4704 fix(engine): a client that leaves must not kill the daemon, and a long round must say it started
Round 9.1, spec §3 D + ADR 0006 items 2 and 4. Two small changes, both proven
by measurement, both E2E-verified locally against a rebuilt brain.

D1 — SIGPIPE/EPIPE survival (vendor/el-runtime el_runtime.c).
Root cause, at the layer that owns it: the whole HTTP server lives in the C
runtime; .el has no socket primitive. http_send_all() called send() with flags
0 and nothing anywhere in the runtime set a SIGPIPE disposition, so the default
disposition — terminate the process — applied. When a handler finished after
its client had gone (Tim's VM: reply at 116.9 s, client cancelled at 25.0 s),
the second of the four sends that write one reply raised SIGPIPE and the daemon
died: `exited due to SIGPIPE ... ran for 361177ms`, launchd respawn 4 ms later,
every other in-flight session's work lost, user never told.

Fix: SIGPIPE -> SIG_IGN at runtime init and at each http_serve* entry, plus
per-connection SO_NOSIGPIPE / MSG_NOSIGNAL so the guard survives an embedder
resetting dispositions. http_send_all now retries EINTR and preserves errno;
http_send_response classifies it once — a departure is logged as routine
("client left before the reply was written ... reply discarded") and ANY other
errno is logged as a real "send failed: <strerror>". Spec §5.3: the routine
case must not mask a genuine write fault, and it does not.

Proof (scratch HOME + free port, 3 disconnects mid-reply):
  round-9 shipped brain 4402179554… — DIED, exit 141 (128+13 = SIGPIPE), round 1
  round-9 sources rebuilt with this exact recipe — DIED, exit 141, round 1
  this build — SURVIVED 3/3, /health 200 after, still serving the full graph,
  three honest "client left" lines in the log naming Broken pipe / Connection
  reset by peer.

D2 — the round-start marker (chat.el, agentic_loop).
The ledger only ever appended AFTER a round returned, so a healthy first leg
produced zero progress by construction; since server-side web_search moved
inside the outbound call that leg is 60-120 s of silence, which is how a 25 s
client watchdog came to kill a healthy mission. One entry,
{"i":N,"t":"","tool":"__working__"}, written to the existing
run_progress_<session_id> ledger BEFORE each round's outbound call — the wire
shape ChatView.kt:1148 has handled as a life signal since 2026-07-13 and never
received. No new key, no new route, no new lifecycle: a strict subset of WS3
item 3. WS3's run registry is untouched and stays Will's.

Proof (live Anthropic key, real research mission, scratch HOME + free port):
  round-9 baseline — ledger EMPTY for the whole 59.7 s leg
  this build       — {"i":0,"t":"","tool":"__working__"} visible at 18.6 s of a
                     70.0 s leg; both builds returned correct ~4.9 KB answers

Regression: prompt-matrix gate 32/32 on this build (round-9 baseline also 32/32
under the same recipe, so the score is not a build artifact). Soul contract
gate PASS — 27/27 routes, immutability clean. neuron#111 miscompile guard: 0
sites in the generated amalgam this binary was compiled from.

NOT included, deliberately: the regenerated dist/soul.c. CI compiles that file,
so production stays exposed until it is regenerated — the same open ask as
neuron#111 / ui#209. The regen recipe is now known and recorded; landing it is
Will's call, per BUILD-HYGIENE.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 18:23:12 -05:00
Tim Lingo dba755dcec fix(engine): resume reads the bridged tool id from the blob's own field, not from inside the replayed conversation
ROOT CAUSE (round 9; live-repro'd 5/5 this morning, both faces stub-proven by the
prompt-matrix gate). json_get is a first-substring-match scanner (strstr for
'"key":', el_runtime.c). bridge_save serialized the RAW messages array BEFORE the
tool_use_id scalar, so agentic_resume's json_get(blob, 'tool_use_id') returned the
FIRST '"tool_use_id":' occurrence inside the replayed conversation, not the saved
field. The resume guard then preferred that misread over the client's correct
call_id (its two branches both reduced to saved_use_id), attached the tool_result
to the wrong id, and Anthropic 400'd the resume ('unexpected tool_use_id found in
tool_result blocks'), surfaced as {"error":"llm unavailable"}.

ONE MISREAD, TWO FACES — whichever block owns the first tool_use_id in the array:
  FACE 1 (search-then-bridge, the Key West killer): the first occurrence is the
    first web_search_tool_result's srvtoolu_… id — every agentic turn that ran
    server-side web_search and then bridged on a client tool died on approval,
    deterministically (messages.2.content.0 … srvtoolu_…). The write itself had
    already succeeded; only the resume died.
  FACE 2 (multi-cycle missions): with no search, the first occurrence is ROUND 0's
    tool_result block — so every LATER approve/resume cycle replayed the round-0
    client id (stale-resume-id), killing multi-file missions after ~2 files.
  And the shape that PASSES on round 8 confirms the mechanism: a single-cycle
  bridge with no prior tool round has no 'tool_use_id' substring in its messages
  at all (tool_use blocks carry 'id'), so the scan fell through to the blob's own
  field and resumed correctly.

The server_tool_use ↔ web_search_tool_result pairs themselves replay intact — the
defect was a cross-field misread of the blob, the same first-match-scanner class
as BUG-6 (approve 'content' matched inside tool_input, 2026-07-17) and round 8's
citation-block fix.

THE FIX, the pattern not the spot:
  1. bridge_save writes every json_safe'd scalar BEFORE both raw fields (an escaped
     value cannot contain a bare '"key":' byte pattern, so first-match always lands
     on the blob's own fields), and tools_raw (our fixed schema) before messages_raw
     (arbitrary conversation), so the raw extractions cannot first-match into
     model-controlled bytes either. Field order documented as load-bearing.
  2. agentic_resume now honors the client's echoed call_id when present — the value
     with clean provenance (minted from pend_tool_id, never blob-round-tripped) —
     falling back to the saved id only when the client omits it. Each approve cycle
     therefore binds to ITS OWN round's id (kills FACE 2 even against a blob written
     by a pre-fix binary), and an omitted call_id still resumes on the saved id,
     which the reordered blob now reads correctly.

Pattern sweep: the legacy synthetic blob (sessions.el handle_session_approve) embeds
only json_safe'd fields — no raw hazard, untouched. No other json_get read of any
container that embeds raw conversation JSON before the read field.

PROOF: prompt-matrix gate 24/32 RED on the round-8 brain (fails exactly the two
resume classes, named) -> 32/32 GREEN on this build; live-key Key West tracer
3/3 consecutive full round-trips (bridge -> approve-as-the-app -> real completion,
file on disk), plain-chat and weather-only controls PASS; unpatched round-8 brain
and a same-toolchain unpatched baseline build both still fail the identical
sequence with the identical srvtoolu 400 (the test discriminates, and the only
variable between failing and passing builds is this diff).

Refs neuron#109

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-06 11:34:12 -05:00
Tim Lingo 8f3a478771 fix(engine): excise the receipt, do not truncate at it — a leading receipt was erasing whole answers
CAUGHT BY A/B, AND ONLY BY A/B. The previous commit's receipt_strip assumed the receipt is
always TERMINAL and cut everything from the marker onward. It is not always terminal: once
receipt_rule told the model what [[RECEIPT ...]] means, the model sometimes LED with one and
wrote the answer underneath. Cutting at the marker then deleted the entire answer and the
turn returned {"error":"no response"}.

MEASURED, same prompt (two web searches, cited prose), fresh session each run:
    round-7 brain   4 / 4 answered   (471, 473, 473, 544 chars)
    round-8 brain   2 / 7 answered   (five {"error":"no response"})
This looked exactly like a flaky model. It was not — it was mine. Running the two brains
side by side on the same prompt is the only reason it was found, and it is the reason the
A/B is now part of how this class gets tested.

AFTER THE FIX, same protocol:
    round-7 brain   4 / 4   (473, 473, 473, 544)
    round-8 brain   4 / 4   (657, 657, 657, 673)

THE FIX: remove the [[...]] span and keep BOTH sides, instead of truncating at the marker.
An unterminated marker at position 0 is left completely alone — no rule about receipts is
worth erasing an answer over. Bounded four-pass loop rather than a conditional exit, because
rebinding the counter inside an if-expression is the block-expression shape that miscompiles
integer arithmetic under this elc (BUG-PLAINCHAT-1). Verified in the generated C:
    str_slice(rest, (e + 2), str_len(rest))   <- integer addition, correct
    el_str_concat(head, tail)                 <- string concat, correct
and zero el_str_concat(<ident>, str_len(...)) sites across all 49 modules.

SEAM PROOF (FIX C) rides on the same runs — a real two-search cited answer, inspected byte
by byte, in BOTH failure directions:
  missing separator (the round-7 "to.Good", bytes 77 2e 47): 0 hits. Sentence boundaries
    measure 2e 20 4d — "." SPACE "M".
  over-separation (a cited sentence shattered across paragraphs): 0 hits. The answer is one
    continuous paragraph with its sentences intact, which is the direction a blanket
    separator would have broken.

BUILT: sha256 77115f2733e794c5bc4ad1f55b1acaf658f8f4a91cccd423a2d633d94a726cbc

Refs neuron#109

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 23:25:48 -05:00
Tim Lingo 9ea41eed78 fix(engine): the model was signing its own answers with our receipt — strip it
FOUND BY E2E, NOT BY REASONING. The previous commit's design note asserted the provenance
receipt "never reaches the user: it is appended to the history copy, not the reply." That
was FALSE, and only running the thing showed it. On the first live run against the built
DMG brain, two agentic turns out of two came back with

  "Your favourite colour is chartreuse and your project is called Perihelion.

   [[RECEIPT - recorded by the soul, not written by the model: no tools ran on this turn.]]"

— the receipt in the user-visible reply.

MECHANISM: the receipt is stored inside the assistant turn, and the agentic path replays
history VERBATIM as Anthropic message objects. So the model sees its own previous answers
ending in [[RECEIPT ...]] and does the obvious thing — it imitates the format and signs the
next answer the same way. The plain path did NOT leak, which is the tell: there, history is
rendered into the SYSTEM prompt as labelled lines rather than replayed as assistant turns,
and a model imitates its own turns far more readily than a transcript.

FIX, two layers, because one of them is not a guarantee:
  - receipt_rule() names the marker in both system prompts (plain and agentic): these lines
    are written by the system, read them as evidence, never write one. Reduces occurrence.
  - receipt_strip() truncates any [[RECEIPT ...]] out of model output before it becomes the
    reply — plain path in layered_generate, agentic path on final_text in agentic_loop.
    Deterministic. A guard that depends on the model choosing to obey is exactly the class of
    thing round 8 exists to stop shipping, so the instruction is the optimisation and the
    strip is the guarantee.
Placed ABOVE agentic_loop's empty-check on purpose: a turn whose entire output was an
imitated receipt has produced no answer, and must be reported as no answer.

The receipt stays in HISTORY, which is the whole point and is proven to work: asked "What
source did you use for that?" one turn after a live web_search, this brain answered
"I used Weather Underground (https://www.wunderground.com/weather/is/reykjav%C3%ADk) for the
current temperature in Reykjavik" — a real source, no apology. That is the false confession
dead, and it is dead BECAUSE the model can read the receipt.

BUILT: 887,112 bytes, sha256 54a2eff84d4fa44f8d2db6781dcf225df4b5075a8ec5f1058018bd40cc1af10b
BUG-PLAINCHAT-1 miscompile guard: zero sites.

Refs neuron#109

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 23:10:53 -05:00
Tim Lingo ff421d39f6 fix(engine): history keeps its provenance and its session — the false confession and the blank stare
DESIGN FIT: three of round 7's five defects share ONE root — the conversation-history
layer persists only {role, content}, discarding tool provenance, session scoping, and the
distinction between a real user turn and an internal utility call. Fixes A and B RESTORE
Will's design rather than extend it: his agentic path already scopes history per session,
the plain path never got it, and his own source carries the TODO admitting the resulting
race (chat.el, handle_chat: "process-global key; concurrent /api/chat requests without
session_id race on this read-append-write"). Fix C repairs one join Will wrote that was
correct for a year and one we added last week. E1/E2 are ours.

FIX A — tool provenance in history (kills the FALSE CONFESSION)
  Root cause, EXECUTED-verified: handle_chat_agentic recorded turns via hist_append, which
  emits {"role","content"} only. server_tool_use blocks, web_search_tool_result blocks and
  every citation were discarded, then replayed as text. On the next turn the model saw a
  data-rich answer with zero evidence a search had happened, and its own permanent rule
  ("never describe a search you did not perform") left one conclusion available: that it
  had fabricated the data. It apologised for a search it HAD run — four independent lines
  of evidence confirm the search was real. The defect is not the model's honesty. It is
  that we deleted the evidence and then asked it to account for itself.
  Change: agentic_loop accumulates the source URLs it already walks past (citations and
  web_search_tool_result content) and returns them as "sources"; handle_chat_agentic folds
  tools_used + sources into a receipt line stored WITH the assistant turn. Receipts are
  unconditional — a negative receipt ("no tools ran") is the other half of the guarantee,
  because "no evidence of a tool" and "evidence of no tool" were previously identical in
  the transcript. conv_history_block splits the receipt off before snipping so a long
  answer cannot truncate away the evidence. The user never sees it: it is appended to the
  history copy, not the reply.

FIX B — one history key for both paths (kills the BLANK STARE)
  Root cause, EXECUTED-verified: the agentic path keyed history on session_hist_<id>; the
  plain path was hard-wired to the process-global conv_history and never read session_id.
  One conversation, two buckets. Proven in the guest engram: the scoped node held exactly
  two turns starting at "Try again" while the earlier exchanges sat unscoped.
  Change: conv_hist_key/conv_hist_label are now the single definition, used by BOTH paths;
  session_id is threaded route -> layered_cycle -> layered_generate / conv_history_record.
  The 2-line fallback (plain path reads the agentic key) was REJECTED: it keeps the
  process-global bucket as a live write target, which is the bleed the TODO describes.
  Also found and closed while threading: layered_cycle read session_id from the state key
  "current_session_id", which is read here and WRITTEN NOWHERE in the entire source. It
  was unconditionally "", so TODO(reliability #4) — per-session steward continuity — was
  dead code that could never fire. It fires now.
  LAZY SESSION, decided explicitly: we create the session EAGERLY at the door (app half,
  ui#223) rather than migrating orphaned turns. Migration would copy the CONTENTS of a
  process-global bucket, possibly another conversation's, into a named session — the bleed,
  performed deliberately. Eager creation makes the situation impossible instead. Migration
  is deliberately not implemented and must not be added without solving provenance first.

FIX C — the two text-join seams ("to.Good", byte-verified 0x77 0x2e 0x47)
  Two bare `+` joins, written a year apart, had drifted into two answers to one question:
  within-response block joins (Will's, 2026-05-03, latent until server-side web_search
  began interleaving non-text blocks) and across-round joins (ours, 62af564).
  Change: one named rule, text_join_sep, at both sites. NOT a blanket separator — a cited
  answer splits MID-SENTENCE ("The current temperature is " + "86°F" + ", with "), so a
  blanket separator shatters every sourced sentence. The rule takes the one bit that
  distinguishes the cases: whether a NON-TEXT block intervened. Hoisting it also makes the
  fix verifiable in the shipped binary, which an inline `+` is not.

FIX E1 — utility generations stay out of the transcript
  Title generation ("Write a 3-6 word title...") and insight passes ran down the same plain
  door as a real message and were recorded as if the user had typed them; the same calls are
  the "model":"unknown" rows in usage.jsonl. is_utility_request reads an explicit utility
  flag from the app, with the __title__/__insight__ id prefixes as a fallback for older
  clients. Answered normally, never recorded.

FIX E2 — OPERATOR IDENTITY is scoped to tool-capable turns
  The block (env USER/HOME, closing "This is a hard rule") was prepended to EVERY system
  prompt including chat mode. On a Tools:Off turn there is no filesystem in reach, so it
  governed nothing and merely supplied the loudest fact in the prompt — which is why the
  model opened a fresh conversation with "You're test, on your machine at /Users/test".
  Hoisted to operator_identity_block() and gated on !chat_mode. Unchanged wherever a file
  or command tool can actually be reached.

ALSO: agentic_loop's per-session history persist had a second hand-rolled copy of
conv_history_persist with a different label expression, different salience scores and
different tags for the same node. Since both now derive the label from conv_hist_label and
engram_node_full upserts by label, two score policies were writing one node. Collapsed to
one writer.

BUILD NOTE: dist/elp-c-decls.h is force-included by the documented link recipe and carried
the OLD C arities, so it is updated here. This is the build-support header, NOT the stale
generated dist/soul.c — no dist/*.c was read or edited; all engine changes are .el source.
chat.elh/soul.elh are committed because a first-pass build against the old signatures FAILS
(measured); the other regenerated headers are reverted as unrelated churn.

BUILT: 887,000 bytes, sha256 d632b061ad75269d6adeb52578d030eaf49e895d91289d7f946b19c08450d728
Zero el_str_concat(<int>, str_len(...)) sites (the BUG-PLAINCHAT-1 miscompile guard).
web_search_20250305 and disable_parallel_tool_use both still present — PR #108's web search
and the ADR-0005 stopgap are intact.

Refs neuron#109 (builds on it), neuron#78 (Receipt Contract — the real fix A is a stopgap for)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 22:59:54 -05:00
Tim Lingo 635f6febe4 feat(engine): plain chat generates at L3 — inside the safety cycle, not around it
Neuron Soul CI / build (pull_request) Failing after 12m8s
Neuron Soul CI / deploy (pull_request) Has been skipped
Non-agentic /api/chat (the desktop app's default "Tools: Off" mode) returned the
user's own screened text as a bare non-JSON string. Every JSON client failed to
parse it and showed "Couldn't reach Neuron - it may be offline."

Root cause: f52d5bd (2026-06-11) correctly moved the route onto the layer spine
(handle_chat -> layered_cycle), but L3 never got a generator — imprint_respond()
annotates its input and returns it. Two pieces of the architecture were already
waiting for that step: layered_cycle parks a bell directive in the state key
build_system_prompt is written to consume, and build_system_prompt carries a
chat_mode ("no tools") flag with no live caller.

The fix composes rather than replaces. Wiring handle_chat would have removed
safety_screen, the hard-bell short-circuit, the whole stewardship layer and
safety_validate — the only enforcing output gate in the codebase — in exchange
for a working reply (see _engine-websearch-20260804/SAFETY-STOP.md). Instead
layered_cycle keeps every gate, in order, and gains a generation step between
imprint_respond and safety_validate.

  L1 screen -> guard -> hard-bell short-circuit -> L2a -> L2b -> L2c
    -> L3 imprint_respond (prompt) -> L3b layered_generate (NEW) -> L1 validate

- chat.el:  NEW layered_generate (L3 generation, no tools offered),
            conv_history_block, conv_history_record.
            FIX build_system_prompt never concatenated no_tools_rule into its
            return — the "[NO TOOLS THIS TURN]" rule reached no model at all.
            handle_chat annotated DO-NOT-WIRE with the reason.
- soul.el:  layered_cycle gains L3b + post-validation turn bookkeeping.
- routes.el: NEW plain_chat_envelope; all three /api/chat dispatch sites wrap the
            cycle's output. Built OUTSIDE the cycle so safety_validate always sees
            raw model text — nothing to unwrap or rebuild on the crisis path.
            Emits both `reply` and `response`: the desktop app reads `reply`,
            the CLI tools and telegram-gateway read `response`.

Also fixes BUG-PLAINCHAT-1, a pre-existing CRITICAL crash on the crisis path.
elc compiles `let n: Int = pos + str_len(marker)` to el_str_concat() — string
concat on two integers — inside a block-expression initializer, segfaulting the
daemon (SIGSEGV in strlen). Six inline copies of the same " | ts:" parser had it:
two in layered_cycle L2c, two in engram_compile (live on the AGENTIC path too),
two in affective_context_prefix. A distress turn following an earlier affective
turn killed the whole process. Proven pre-existing: an unmodified baseline binary
crashes identically, and the same bad C is in the committed dist/soul.c. Fixed by
hoisting to one top-level function, affective_node_ts(), where the expression
compiles to integer addition — verified in the generated C.

Proof (throwaway HOME/engram, explicit NEURON_PORT, live chain untouched):
- Plain turn returns a JSON envelope with the provider's answer, not an echo.
- Captured request body: no `tools`, no `tool_choice`; system prompt carries the
  NO-TOOLS rule. Tools:Off means no tool is offered, structurally.
- Hard bell: canned 988 message, and the provider request count does not move —
  the message never reaches a model.
- Soft bell + a 2-char model reply: safety_validate's care phrase is appended to
  the MODEL's output. Output gate acting, on this route.
- The L1 bell directive now reaches the model here for the first time (the state
  addendum had a producer and no consumer).
- test_layered_cycle PASS; all six El suites byte-identical to baseline.
- verify-soul-contract.sh (bash 5.3): GATE PASS, 27/27, immutability PASS.
- The crash sequence that killed the baseline daemon now returns HTTP 200.

Not proven: no live Anthropic call — the login keychain refuses the key to a
non-interactive process (rc=24 errSecInteractionNotAllowed). Details and the
one-command close-out are in _engine-plainchat-20260805/README.md §7.

Builds on PR #108. dist/soul.c deliberately not regenerated — Will's toolchain.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 09:11:03 -05:00
Tim Lingo 62af5649fe feat(engine): port Anthropic server-side web_search into the agentic loop
Neuron Soul CI / build (pull_request) Failing after 10m45s
Neuron Soul CI / deploy (pull_request) Has been skipped
Re-authors soul-webfix-20260711.patch in El (the patch is a diff against
generated C at month-old offsets, so nothing was applied as a patch). Its
last two hunks — an unrelated /api/safety-contact implementation — were
deliberately not ported; that route already exists and is safety-critical.

Activation restores existing design, it does not invent a mechanism:
commit 8eea1d9 (2026-06-09, Tim-approved) made native web_search built-in
with no user-facing toggle, and tests/test_agentic_tools.el section 2 still
asserts agentic_tools_all() contains it — an assertion main currently fails.
The call site was lost when agentic_tools_all() (connector tools, PR #19)
replaced agentic_tools_with_web(). Attaching in agentic_tools_all() covers
handle_chat_agentic, handle_dharma_room_turn_agentic and agentic_resume.

pause_turn handling is included and was genuinely missing: the shipped
binary has zero occurrences of it. Without it a paused server-side search
returns only the text written so far and the loop treats it as final —
a silently truncated answer. final_text now accumulates across resume
cycles rather than overwriting (overwriting would discard everything
written before the pause).

Default tool version is web_search_20250305, NOT the newer _20260209, and
that is a measured choice: _20260209's dynamic filtering uses server-side
programmatic tool calling, which the API refuses to combine with ADR 0005's
stopgap —

  HTTP 400 invalid_request_error
  tool_choice.disable_parallel_tool_use: true cannot be used with
  programmatic tool calling

Dropping the stopgap would resurrect neuron#78 bug b (killed runs 3/3 in
ADR 0005's own A/B). The basic variant is compatible with the stopgap and
returns real results, so neither feature is dropped. Version lives in state
key web_search_tool_version; flipping it once the stopgap retires is a
config write, no recompile. The fallback also fires on "programmatic tool
calling" so a premature flip self-heals loudly instead of dying.

Also fixes a real bug this port exposed: json_get is a first-match scanner
and a cited text block serialises citations FIRST, so json_get(block,"type")
returned the nested citation's type and every citation-bearing block — the
ones carrying the searched facts — was silently dropped from the reply.
Reply length on the same question: 79 -> 360 chars.

Other loop changes: server_tool_use accounting into tools_used, iteration
cap 8->12 for pause/resume cycles, max_tokens 4096->16384, container-id
carry-forward, API error head logged instead of swallowed, tools_used gated
on is_tool_turn so a truncated tool block is not reported as work done.

dist/soul.c is deliberately NOT regenerated — the local elc predates Will's
last regen (e610a41) and regenerating with an older compiler risks unrelated
codegen drift. Source only; regen is Will's.

E2E-VERIFIED on a sandbox soul (scratch HOME/engram, dead axon+ISE, explicit
NEURON_PORT): live Bentonville weather with tools_used ["web_search"]; the
27-route contract gate PASSes; the parallel-tool stopgap still completes a
3-file mission with no 400; a 3-search 3,486-char answer kept its end
sentinel. Honest gap: pause_turn is compiled in but not exercised by a live
pause (max_uses:5 caps the server loop below the pause threshold).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-04 22:57:59 -05:00
Tim Lingo 710761e2d5 fix(engine): send tool_choice.disable_parallel_tool_use on agentic loop (STOPGAP)
The agentic loop keeps only the FIRST tool_use block per round (chat.el:2281,
"Capture first tool_use block only"). Anthropic lets a model emit several
tool_use blocks in one message and requires a tool_result for every one, so a
parallel-tool turn is answered once, the rest are dropped, and the next request
dies with:

  tool_use ids were found without tool_result blocks immediately after

(neuron#78 quotes this as "tool_use ids found without tool_result"; the above is
the API's actual wording - recorded so the next person's grep matches.)

This constrains the wire to match what the loop can assemble:
  "tool_choice":{"type":"auto","disable_parallel_tool_use":true}

STOPGAP - AND THE DURABLE FIX ALREADY EXISTS. A correct multi-tool loop is
already in Will's EL runtime, in C, and the soul does not call it. Verified on
el:origin/main lang/el-compiler/runtime/el_runtime.c: llm_register_tool:9616,
llm_build_tool_results:9743 - which walks EVERY content block, emits one
tool_result per tool_use, and sets is_error for an unregistered tool -
llm_call_agentic:9817 calling it at :9918, iteration cap 10 at :9847. Will's
commit 12d5e77 (2026-04-30). grep for llm_call_agentic/llm_register_tool across
every neuron/*.el returns nothing; dist/soul.c has zero references. chat.el
hand-rolls its own single-tool loop instead, and that is the one that breaks.

The durable fix is therefore to register the soul's tools via llm_register_tool
and call llm_call_agentic - deleting a loop, not writing one. See ADR 0005.

Our own approved spec called this seven weeks ago:
docs/research/agentic-tool-approval-design.md (2026-06-12, "Approved for build"),
line 20 on the defect, line 30 on the goal ("Execute all tool_use blocks in a
turn (one result per block)").

Two edits, because dist/soul.c cannot be regenerated here (Will's gated elc/elb
toolchain is not on this machine):
  (a) chat.el:2255 - source of truth, so a later regen carries the fix.
      One edit covers all three routes: agentic_loop is called from chat.el:2152
      (/api/chat agentic), :2676 (dharma room) and :2496 (agentic_resume).
  (b) dist/soul.c:28173 - generated form, hand-spliced. Line 27624 is the
      non-agentic/OpenAI-compat req_body (no tools) and was left untouched.
Prior art reused rather than reinvented: soul-narrated-runs-20260713.patch
(27,824 bytes) line 78 spliced this same string into the same concat chain on
2026-07-13.

Deliberately NOT ported from that patch, having read it: max_tokens is not
changed by it (16384 sits on both sides of the hunk; our main's 4096 is a
separate output-truncation concern), and its pause_turn pairing fix - same
defect class - is unreachable today because no server-side web_search is wired
(agentic_tools_with_web at chat.el:1418 is never called), so it is untestable
and logged instead.

Proven E2E on a scratch profile and port 7791, never the live chain. A/B against
a pristine origin/main control built from the same vendored runtime: fixed
completed the mission (tools_used read_file x3, 4 iterations, correct answer);
control failed 3/3. Direct API probe confirmed the mechanism - without the field
the model emits 3 parallel tool_use blocks and replaying the unfixed loop's next
turn returns HTTP 400; with it, exactly 1 block.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-03 17:07:04 -05:00
will.anderson 74520b8333 Merge pull request 'ci: pin + complete vendored el-runtime so reconciled soul.c links' (#105) from ci/pin-vendored-runtime into main
Neuron Soul CI / build (push) Failing after 14m36s
Neuron Soul CI / deploy (push) Has been skipped
2026-08-03 16:08:51 +00:00
will.anderson 7f3d6ed8cd ci: update vendored el-runtime to complete v1.0.0-20260501
Neuron Soul CI / build (pull_request) Has been cancelled
Neuron Soul CI / deploy (pull_request) Has been cancelled
The runtime vendored alongside the CI pin was the Jul-21 snapshot, which
predates two builtins the reconciled ship-soul now calls:
  - http_delete_json  (boot-counter HTTP write-back, awareness/memory self-review)
  - engram_act_stats_json  (heartbeat activation observability)
Compiling dist/soul.c against the stale runtime fails with implicit-declaration
errors. Vendor the current release runtime (identical to the one the soul was
gate-verified against: verify-soul-contract PASS, genesis boots clean, full
safety-contact) so the CI Linux soul is byte-for-byte the verified soul.
2026-08-03 11:07:38 -05:00
will.anderson eed6487114 ci: pin soul build to vendored release runtime v1.0.0-20260501
The soul build downloaded el-runtime-c 'latest' from Artifact Registry. The
merged ship-soul calls engram_prune_telemetry, which the latest published
runtime no longer defines, so an unpinned build fails to link — the failure
mode that let a broken/handlerless soul reach prod.

Vendor the release runtime v1.0.0-20260501 (el_runtime.c/.h) into the repo and
compile the soul against it. This is the exact runtime the merged soul was
verified against (verify-soul-contract GATE PASS, genesis boot survives, full
safety-contact response), making the build reproducible and independent of a
moving AR 'latest'.

The verify-soul-contract.sh HARD-BLOCK gate already runs before Publish (from
the CI-hardening arc on main), so a destructive or stale soul can never
publish/deploy again.
2026-08-03 11:06:26 -05:00
will.anderson 2c2aaa0653 Merge pull request 'Reconcile: main = union of all ship-critical soul fixes (beta-gating)' (#104) from reconcile/soul-union-main into main
Neuron Soul CI / build (push) Has been cancelled
Neuron Soul CI / deploy (push) Has been cancelled
2026-08-03 16:03:46 +00:00
will.anderson e610a412b9 regen soul.c from reconciled tree + harden contract gate (#199 by-id, isolation)
Neuron Soul CI / build (pull_request) Failing after 12m40s
Neuron Soul CI / deploy (pull_request) Failing after 14m47s
dist/soul.c: regenerated amalgamation (1.15MB) from the reconciled sources via
the hide-.elh + elc --target=c recipe, so the shipped translation unit CI
compiles now actually carries every landed fix — genesis-boot SIGSEGV (#150),
safety-contact 988 truncation (#96), url-decode multi-word search, honest
receipts (#100/#101), immutability arc (#83), and the bounded payloads (#103).

verify-soul-contract.sh, two non-weakening fixes (both false-NEGATIVE bugs that
spuriously failed a CORRECT soul; neither relaxes what fails a defective one):
 1. #199 by-id gate: verify tombstone/KEPT via /api/neuron/graph?id=<id>&depth=1
    (a compact neighborhood) instead of grepping engram_scan_nodes_json(9999,0)
    — a multi-MB, salience-ordered, 9999-capped whole-graph dump in which the
    salience-0.01 tombstone marker sorts past the cap and vanished.
 2. Isolation: pin SOUL_ISE_URL to the dead axon port. Unsetting ENGRAM_URL was
    not enough — the periodic engram sync defaults its source to the LIVE engram
    (http://localhost:8742), so the 'isolated' gate pulled the operator's real
    brain (56 -> 12k nodes in seconds), which both broke Section B determinism
    and read live state. Now the soul stays on its own store.

Verified GREEN on a throwaway port/HOME (live :7770/:8742/~/.neuron untouched):
gate PASS x3 (presence 27/27, immutability all 5 KEPT); safety-contact POST 218B
/ GET 208B full untruncated; multi-word search (%20 and +) returns ranked hits
with an all-gibberish control at 0; bounded session/begin 1370B; honest ok:false
on a missing-id delete; genesis (ntn-genesis) boots clean through mem_save with
no SIGSEGV.
2026-08-03 11:01:46 -05:00
will.anderson 8ba35a0d56 reconcile: merge Will's self-review WIP (456267a) into main
Union of all ship-critical soul fixes for the Mac beta:
- Keeps main's honest receipts (#100/#101), immutability arc + #199 by-id
  gate (#83), Track B threat routing (#76), bounded beginSession (#103),
  CI hardening (#85/#86), elc typo hotfix (#77), neuron-dev-setup (#84).
- Brings WIP's genesis-boot SIGSEGV fix (#150/#95), safety-contact 988
  truncation fix (#96), bounded-persona floor (#93), and 10 self-review
  commits (importance flattening, curiosity DF gating, WM/heartbeat
  observability, boot-counter telemetry).
- Folds the multi-word ranked-search fix: api_query_param now url_decode()s
  the extracted value so q=foo%20bar / foo+bar tokenize as two words.

Conflicts (neuron-api.el payload-bound comments, mcp-wrapper tool_forget)
resolved toward the correct end state: main's verified honest-receipt
read-back is kept; WIP's improved forget description is kept. Generated
dist/*.c taken from main and will be regenerated from this reconciled
source in the following commit.
2026-08-03 10:35:01 -05:00
will.anderson b75d5c8c30 Merge pull request 'Bound beginSession/compileCtx payloads to a compact digest (main)' (#103) from fix/bound-session-payload-main into main
Neuron Soul CI / build (push) Successful in 3m44s
Neuron Soul CI / deploy (push) Failing after 5m46s
2026-08-01 16:42:05 +00:00
will.anderson 9bbb4f2af8 Bound beginSession/compileCtx payloads to a compact digest
Neuron Soul CI / build (pull_request) Successful in 4m19s
Neuron Soul CI / deploy (pull_request) Has been skipped
Port the payload-bounding fix (PR #102, commit 872120c) onto main. The
session-init endpoints projected unbounded engram nodes (~900KB), closing
the MCP client socket on every beginSession. Cap the lists (8 activated /
10 recent for begin_session, 10/20 for compile_ctx) and project each node
to a light identity + a bounded, UTF-8-safe content snippet via
api_compact_node / api_compact_activated / api_utf8_trunc. Response drops
~900KB -> ~12KB; full content stays available via recall/fetch/inspectGraph.

Regenerate dist/soul.c (the CI-built amalgamation) and dist/neuron-api.c
from source. The soul.c regen also compiles in already-merged source the
previously-committed soul.c was stale against (agent write/edit receipt
fixes, #100/#101); verified via scripts/verify-soul-contract.sh
(PRESENCE + IMMUTABILITY PASS) and a clean CI-style cc build.
2026-08-01 11:37:04 -05:00
will.anderson ec219c5830 Merge pull request 'fix(mcp-wrapper): forget/delete tools no longer return fake ok receipts (BUG-18)' (#101) from fix/receipts-wrapper-forget into main
Neuron Soul CI / build (push) Successful in 5m56s
Neuron Soul CI / deploy (push) Failing after 6m26s
2026-08-01 15:56:36 +00:00
will.anderson 731efaedaf Merge pull request 'fix(chat): agent write_file/edit_file no longer return false success receipts (BUG-29)' (#100) from fix/receipts-agent-tools into main
Neuron Soul CI / build (push) Successful in 4m29s
Neuron Soul CI / deploy (push) Failing after 7m28s
2026-08-01 15:56:32 +00:00
Tim Lingo 3723e3b7e7 fix(mcp-wrapper): forget/delete tools no longer return fake ok receipts (BUG-18)
Root cause: two false-receipt paths in the wrapper's delete family.
- delete_by_id (removeKnowledge, deleteProcess, deleteImprint,
  dischargeWonder) FABRICATED {"ok":true,...,"note":"soft-deleted"}
  without calling the soul at all — the 'soul does not yet expose a delete
  HTTP route' note was stale (/api/neuron/node/delete exists and tombstones
  any node type).
- tool_forget forwarded the soul's response but never verified the deletion
  actually persisted before answering ok.

The change (Receipt Contract rule 1 — a tool result must reflect what
actually happened):
- delete_by_id now routes to the soul's real /api/neuron/node/delete and
  propagates its answer (honest 'node not found' for bad ids).
- Both handlers read back before answering ok: GET /api/neuron/graph?id=..
  &depth=1 must show the tombstone marker (label "tombstone:<id>"); if it
  does not, answer {"ok":false,"error":"delete_not_persisted",...} in
  the soul's not-persisted error shape (api_not_persisted).
- Soul errors and transport failures pass through unchanged.

E2E evidence (sandbox soul :7791 + wrapper :7792, elb builds):
- unpatched: removeKnowledge on a NONEXISTENT id -> {"ok":true,
  "deleted":"kn-DOES-NOT-EXIST-deadbeef","note":"soft-deleted"} (lie)
- patched:   same call -> {"error":"node not found: ..."} (soul's answer)
- happy path: remember -> forget -> {"ok":true,"tombstoned":true};
  read-back: hidden from default /list/Memory, present with
  ?include_deleted=1, node KEPT in full graph view (immutability intact)
- scripts/verify-soul-contract.sh on the soul it talks to: GATE PASS
  (27/27 presence + immutability)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-01 10:52:21 -05:00
Tim Lingo 62c562a3f1 fix(chat): agent write_file/edit_file no longer return false success receipts (BUG-29)
Root cause: dispatch_tool's write_file returned {"ok":true} without checking
fs_write's result, and edit_file returned ok:true even when old_text was absent
(str_replace silently no-ops) and its fs_write was also unchecked. Any failed
or no-op write fed the model a false success receipt, which it then repeated
to the user as fact.

The change (Receipt Contract rule 1 — a tool result must reflect what actually
happened):
- write_file: check fs_write's return (1 = all bytes written, 0 = fail);
  on failure return {"error":"write failed"} in the handler's existing
  error-JSON shape.
- edit_file: reject empty old_text, verify old_text is actually present
  (str_contains) before replacing, and check the fs_write result the same way.
- Verification is by operation result, NOT an fs_read read-back: fs_read arms
  the runtime's one-shot binary send length, the exact mechanism that truncated
  the safety-contact response (#96). Same honest-write pattern as that fix.

E2E evidence (sandboxed elb build, dispatch_tool driven directly):
- unpatched: write_file into a chmod-000 dir -> {"ok":true} (lie);
  edit_file with absent old_text -> {"ok":true} (lie, file untouched)
- patched:   same calls -> {"error":"write failed"} /
  {"error":"old_text not found in file"}; happy paths still ok:true
- scripts/verify-soul-contract.sh on the patched soul: GATE PASS (27/27
  presence + immutability)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-01 10:51:28 -05:00
will.anderson 456267a771 self-review 2026-08-01: fix importance flattening in remember/evolve/cultivate paths
Four sites passed the Float local 'sal' through el_from_float() a second
time. el_val_t is the bit-pattern of the double, so re-wrapping performs
an int64->double VALUE conversion of the bits before re-bitcasting —
garbage that fails engram_decode_score's range check and clamps to
defaults. Net effect: importance="critical" stored 0.5/0.5 — importance
levels were cosmetic on the MCP memory path. Verified fixed live:
critical now stores salience/importance 0.95/0.95. Same bug fixed today
in engram server.el route_create_node (foundation/el 7f03876). Literal
wraps (el_from_float(0.9)) are safe — elc passes numeric literals raw.
2026-08-01 08:42:44 -05:00
will.anderson edb0670670 self-review 2026-08-01: emit discrete wm_saturation_transition ISE
wm_saturated was a sampled boolean — the 0->1 onset and 1->0 release
moments were only recoverable by hand-diffing consecutive heartbeats.
Emit a low-rate transition ISE at each edge carrying the WM top-5 at
that instant, so the composition that caused the regime change is
captured rather than the composition up to 59s later. First beat of a
boot never fires (restart is not a transition).
2026-08-01 08:38:53 -05:00
will.anderson 872120c757 Bound beginSession/compileCtx payloads to a compact digest
The session-init endpoints concatenated unbounded engram activate/scan
results as FULL node objects (content up to ~90KB per node), producing a
~900KB response. After the MCP wrapper re-escapes that into a stringified
text block the client dropped the socket ('connection closed unexpectedly')
on every beginSession call. Cap each list (8-10 activated, 10-20 recent)
and project every node to a light identity plus a bounded, UTF-8-safe
content snippet. Response drops from ~900KB to ~12KB; full content stays
available on demand via recall/fetch/inspectGraph.
2026-07-31 14:59:09 -05:00
will.anderson f3660e92a1 self-review 2026-07-31: heartbeat deltas for cumulative counters, embed_eligible; fix stale semantic-seeding comment
Runtime activation counters are now cumulative, so the heartbeat emits
wm_evicted/breakthroughs as totals plus wm_evicted_delta/
breakthroughs_delta (state-tracked change since the previous beat) —
events between beats are no longer lost. Adds embed_eligible from
/api/stats so coverage reads as embed_count/embed_eligible instead of
the misleading absolute count, and surfaces auto_term_streak in the
heartbeat stream. Replaces the false 'semantic seeding NOT implemented'
comment: engram_activate embeds the query, seeds semantic top-K, gates
propagation on cosine, and scores WM promotion semantically.
2026-07-31 08:41:41 -05:00
will.anderson b0f4d6c493 self-review 2026-07-30: auto_term stopword filter, real idle_ms, bounded beginSession
- awareness.el: curiosity auto_term was the raw first word of a WM
  label with no term-quality scoring — observed seeds included What,
  Colon, Prose, Context. Replaced the 7-word genre blocklist whack-a-
  mole with a delimited stopword membership test (function words +
  document-structure words); topical terms pass untouched. Verified
  live: seeds now ReasonEdit, Reasoning-model, Self-review.
- routes.el + awareness.el: idle counter only reset on rare inbox
  synthesis-requests, so idle==pulse always (zero information).
  handle_request now stamps soul.last_activity_ts on every inbound
  HTTP request; heartbeat emits idle_ms = ms since last request
  (-1 until first request of a boot).
- neuron-api.el: beginSession concatenated a depth-2 spread plus the
  unbounded self-hub neighbor dump — multi-MB response, doubled by
  wrapper re-escaping, socket died on every call. Now depth-1 and
  the hub dump dropped (identity loading has its own tool). Verified:
  beginSession returns instead of closing the socket.
2026-07-30 08:45:21 -05:00
will.anderson 627eb534a2 self-review 2026-07-28: close ISE lifecycle observability gaps
- session_start now also posted to the HTTP Engram via ise_post: the
  local engram_node_full write never crossed to the observable stream
  (sync flows HTTP->soul only), so boots 5+ were invisible — last
  visible session_start was boot 4, two weeks ago
- graceful shutdown emits a final ISE with boot/pulse/uptime; a boot
  with no shutdown event now reliably signals a crash/SIGKILL
- empty /api/sync responses emit a sync_empty warn ISE instead of
  being silently skipped — unreachable engram no longer looks
  identical to quiet-but-healthy
- sync backflow prune reads ENGRAM_ISE_RETENTION_MS instead of
  duplicating the 48h magic number server.el already honors
2026-07-28 08:37:39 -05:00
will.anderson 2b612ed5d4 self-review 2026-07-27: heartbeat carries activation observability
Fold engram_act_stats_json() into the heartbeat ISE: wm_evicted and
breakthroughs (per curiosity-scan activate call) plus embed_breaker_open —
the failure mode embed_ok structurally cannot see (it pings the Ollama
root, not the embed pipeline). WM-cap eviction, breakthrough-floor
flooding, and silent lexical degradation are now one-glance diagnosable
from telemetry.
2026-07-27 08:38:54 -05:00
will.anderson 8392f44c45 self-review 2026-07-26: heartbeat wm_churn + wm_top0_wm; fix streak counting on empty id
- wm_churn: count of top-5 WM ids absent from previous beat — separates
  'one stuck node' from 'whole WM frozen' without hand-correlating ISEs.
- wm_top0_wm: leader's weight; a frozen anchor reads as a constant here.
- Streak guard: before the runtime emitted id in wm_top JSON,
  json_get(...,"id") was always empty and the streak incremented on
  ""=="" every beat — wm_top0_streak measured uptime, not fixation.
  Empty id now resets the streak to 0.
2026-07-26 08:40:56 -05:00
will.anderson 58a9eda311 self-review 2026-07-25: break curiosity positive-feedback loop; observability for WM regime
proactive_curiosity strengthened its top result unconditionally every
scan — a positive-feedback fixed point that pinned auto_term on the same
node's first word for hours ('Fast-slow' era). Strengthen now fires only
when the top node changed since the last scan, and a 4-deep finst-style
tabu ring (ACT-R declarative finsts) hard-excludes recently used auto
terms (~2 min at the 30s cadence). Quoted-title guard stops '"The'
leaking through the >3-char stopword check and seeding lexical floods.

Heartbeat now pumps /api/embed-backfill?n=32 on the authoritative store
(its lazy backfill had no production trigger; coverage stalled at
93/12175) and emits wm_saturated, wm_top0_streak, embed_backfilled,
embed_count. Curiosity ISE emits auto_term_streak. The stuck-WM failure
mode is now a one-glance signal instead of manual ISE cross-referencing.
2026-07-25 08:45:22 -05:00
will.anderson fb0bb553f3 self-review 2026-07-24: boot counter — demote to telemetry weight, restore persistence via HTTP write-back
Three stale soul:boot_count copies (salience .9, importance .9, Canonical:
+0.2 tier bias, 0.15 threshold) held the top WM slots for 23h — a boot
counter outcompeting real context. Demoted to salience .55 / importance .2 /
tier Working: plumbing, not memory.

Persistence was also broken: in HTTP-engram mode the server owns state and
nothing wrote the counter back — the log shows boot #5 on three consecutive
boots. mem_boot_count_inc now mirrors the persona write-back: delete stale
server copies (matched by content prefix — route_create_node sets
label=content), create the replacement server-side. Working tier is in the
boot seed (/api/nodes) but excluded from periodic /api/sync, so the count
survives restarts without re-importing mid-session. Verified: restart
incremented 1->2 with exactly one server-side counter node.
2026-07-24 08:53:02 -05:00
will.anderson 9e59c51f3c self-review 2026-07-23: title-derived Knowledge labels + Knowledge admitted to curiosity auto-term
Sentinel labels (knowledge:captured/evolved/canonical) made every capture
anonymous in WM telemetry — 35 identical wm_top entries — and starved the
curiosity auto-term seeder, which derives scan seeds from WM top-10 labels
and had returned empty on every scan since boot 6 because WM became
Knowledge-dominated while Knowledge was excluded from seeding.

- capture/evolve/promote now pass title (or empty → engram_node_full's
  content[:60] derivation) instead of sentinels
- auto_term_try_slot admits Knowledge slots; sentinel-shaped labels
  (colon, no space) are skipped so legacy nodes cannot seed 'knowledge'
- verified: probe capture labeled 'Label derivation probe 2026-07-23'
2026-07-23 08:39:16 -05:00
will.anderson 3d74472a4c Merge pull request 'Add neuron-dev-setup: one-command CORE dev stack onboarding installer' (#84) from feat/neuron-dev-setup into main
Neuron Soul CI / build (push) Failing after 13m54s
Neuron Soul CI / deploy (push) Has been skipped
2026-07-22 22:33:33 +00:00
will.anderson acbe858995 Merge remote-tracking branch 'origin/main' into feat/neuron-dev-setup
Neuron Soul CI / build (pull_request) Successful in 4m13s
Neuron Soul CI / deploy (pull_request) Has been skipped
2026-07-22 16:49:40 -05:00
will.anderson 3ae07cc7b0 harden(neuron-dev-setup): fix 7 fresh-Mac onboarding installer bugs (#99)
Neuron Soul CI / build (pull_request) Has been cancelled
Neuron Soul CI / deploy (pull_request) Has been cancelled
Co-authored-by: Neuron <will.anderson@neurontechnologies.ai>
Co-committed-by: Neuron <will.anderson@neurontechnologies.ai>
2026-07-22 21:49:30 +00:00
will.anderson 33d2574b72 Merge pull request 'Agent consent: the pause contract + false-receipt kill (2 fixes, stricter only)' (#79) from feat/agent-phase1-soul into main
Neuron Soul CI / build (push) Has been cancelled
Neuron Soul CI / deploy (push) Has been cancelled
2026-07-22 19:50:42 +00:00