Compare commits

...

14 Commits

Author SHA1 Message Date
Tim Lingo dd952c0e46 feat(soul): write-through to the persistence owner — memories survive restart (#117)
Neuron Soul CI / build (pull_request) Failing after 14m57s
Neuron Soul CI / deploy (pull_request) Failing after 14m39s
The soul obeys half of its own ownership rule. soul.el:571-573 says "when
ENGRAM_URL is set the HTTP Engram owns persistence — the soul must NEVER write
to the local snapshot", and it doesn't. But nothing was ever built to hand the
soul's writes TO that owner: sync is pull-only (/api/sync -> engram_load_merge),
so every node created inside the soul lived in process RAM and was shed on
restart. Measured live 2026-08-07: soul node_count=102184, engram 79197.

SCOPE CORRECTION vs the earlier internal spec: engram provisional claim 17's
"pull-then-push" is a PEER-ENGRAM to PEER-ENGRAM protocol (claims 15-18 say so
explicitly). The soul is a CALLER of the database API, not a peer. Claim 17 is
NOT authority for a soul<->engram contract and is no longer cited as such. The
design here follows from the ownership rule alone.

Mechanism: a new Accessor, persist.el, is the single boundary. Writes stage a
delta to a filesystem spool and are pushed to the owner via POST /api/load-merge
— NOT POST /api/nodes, which mints a new server-side id (breaking dedup and
edges) and drops label/tier/tags/importance/confidence (verified in a sandbox:
a tier "Canonical" probe came back "Working"). load-merge preserves the id and
every field, dedups nodes by id and edges by (from,to,relation) so retries are
no-ops, and calls persist_canonical() so THE OWNER writes its own file — the
ownership rule is honoured rather than worked around.

Spool-and-drain rather than push-per-write: measured ~0.38s per load-merge at
live scale (79k nodes/176MB), and a chat turn writes 5-7 nodes. The spool is on
disk, not in process state, because the soul serves each connection on its own
pthread and a shared buffer would lose entries to a read-modify-write race. That
also buys crash recovery: writes orphaned by kill -9 are drained on next boot.

Honesty: api_persisted (the gate all 10 MCP write handlers pass through) and
mem_store now assert AT THE OWNER instead of reading back the soul's own RAM.
With the owner down a write returns {"ok":false,"error":"write_not_persisted"}
and the delta is queued — where main returns {"ok":true} for a write that dies.

Coverage: 35 node sites + 9 edge sites routed through the boundary. Deliberately
excluded, with reasons in persist.el: 4 InternalStateEvent sites (Will's own
telemetry carve-out), the boot counter and the persona (both already have
bespoke owner-side write-backs), and soul.el's 54 genesis identity edges
(file-mode only). engram_strengthen and engram_forget are NOT propagated —
load-merge cannot update or delete, and hard-deleting at the owner would fail
verify-soul-contract.sh section B.

Also fixed here:
- routes.el GET /api/graph/edges engram_save()'d straight over the owner's
  canonical snapshot.json — a read route, in a non-owner process, clobbering the
  canonical on every call. Same defect class Will removed from the engram in el
  dc39a61. Now exports to a scratch path. With this gone the soul writes nothing
  at all in HTTP mode.
- persist.el must clear the runtime's _tl_fs_read_len hint after every fs_read.
  In vendored runtime v1.0.0-20260501 that hint becomes the NEXT response's
  Content-Length, so reading a spool file mid-request made an 86-byte reply go
  out as 497 bytes with 411 bytes of adjacent heap trailing it. Caught and fixed
  at our boundary; the runtime class was fixed upstream in el 43636ae, which is
  not the pinned runtime here.

Rung: E2E-VERIFIED, discriminating. Same harness, same engram binary:
  write-through: LEG 1 PRESENT at owner, LEG 2 SURVIVED kill -9 + restart
  main:          LEG 1 ABSENT  at owner, LEG 2 LOST
verify-soul-contract.sh: GATE PASS on both builds (27/27 routes, immutability).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 12:47:31 -05:00
tim.lingo 18714e6142 Merge pull request 'fix(engine): restore multi-turn crisis escalation on the agentic path (P0, closes #129)' (#130) from fix/129-history-amplification into main
Neuron Soul CI / build (push) Failing after 14m37s
Neuron Soul CI / deploy (push) Has been skipped
2026-08-07 15:54:41 +00:00
tim.lingo 4936099c39 Merge pull request 'fix(engine): the daemon survives a client leaving, and says it is working while it works' (#127) from fix/liveness-engine-91 into main
Neuron Soul CI / build (push) Has been cancelled
Neuron Soul CI / deploy (push) Has been cancelled
2026-08-07 15:54:15 +00:00
tim.lingo f1471763f5 Merge pull request 'fix(engine): approving a researched mission completes — the resume replay read a tool id out of the conversation (BUG-42, both faces)' (#115) from fix/resume-server-tool-replay into main
Neuron Soul CI / build (push) Has been cancelled
Neuron Soul CI / deploy (push) Has been cancelled
2026-08-07 15:53:51 +00:00
tim.lingo 5850793b67 Merge pull request 'fix(engine): history keeps its provenance and its session — kills the false confession, the blank stare, and the "to.Good" seams' (#114) from fix/soul-history-provenance-20260805 into main
Neuron Soul CI / build (push) Has been cancelled
Neuron Soul CI / deploy (push) Has been cancelled
2026-08-07 15:53:32 +00:00
tim.lingo fc1745c652 Merge pull request 'feat(engine): plain chat generates at L3 — inside the safety cycle, not around it (+ crisis-path segfault fix)' (#109) from feat/soul-plain-chat-generation-20260805 into main
Neuron Soul CI / build (push) Failing after 10m39s
Neuron Soul CI / deploy (push) Failing after 14m47s
2026-08-07 15:53:08 +00:00
Tim Lingo 43d0449904 fix(engine): the agentic crisis screen reads the session's own history again
P0 SAFETY. Closes the regression we introduced in ff421d3 (2026-08-05).

ff421d3 correctly moved conversation history to a per-session key via
conv_hist_key(session_id). One consumer did not move with it: the agentic
path's L1 safety screen kept reading the anonymous "conv_history" bucket. The
desktop app always mints a session id (DaemonClient.kt:706), so history was
always written under session_hist_<id> and that read always returned "".

The half of the crisis score that receives history is the escalation half — the
one that exists for distress building across several turns, where no single
message trips the bell on its own. It scored 0 on every real conversation for
two days. Single-message hard bell was never affected.

The bitter part: the comment that line carried documented this exact bug being
fixed once already, under issue #9. The fix was right then. The rename
re-broke it, and the comment went on describing a repair that no longer held.
A comment is not a gate.

The read now goes through conv_hist_key like every other consumer, including
the plain path at soul.el:398 and the thread-anchoring read thirty lines below
it in this same handler. It is one line. The rest of this commit is structure
so it cannot happen quietly again:

  - agentic_safety_screen() owns the two decisions that were inline — which
    window the screen sees, and the screen call. Inline safety inputs are
    untestable safety inputs; that is what let a rename starve this one with
    nothing failing and nothing logging.
  - the comment above the call site now states the invariant (read window ==
    written window) instead of naming a key that can be renamed out from under
    it.

TWO-LEG PROOF, one variable — the single line state_get("conv_history") ->
state_get(conv_hist_key(session_id)):

  before  scripts/run-el-test.sh tests/test_history_amplification.el
          3. REGRESSION #129 ... FAIL  got: soft_bell  expected: hard_bell
          8 passed, 1 failed          runner exit 1
  after   same command, same tree, that one line changed
          9 passed, 0 failed          runner exit 0

Full engine rebuild from these sources is clean: gen-soul-amalgam.sh ->
1,164,103 bytes / 1226 inlined bodies (gate wants >= 1200), cc-brain.sh ->
903,096 bytes, 0 errors. agentic_safety_screen and conv_hist_key both present
in the built binary (nm: T _agentic_safety_screen, T _conv_hist_key).

Rung reached: BUILT + RUNS (discriminating test). NOT yet in a DMG and not yet
verified in the app a human opens — those are the next two rungs and neither is
claimed here.

Known and NOT fixed by this commit:
  - feat/soul-openai-tools-v2 carries the same defect independently at
    chat.el:2937 and needs the same change or a merge.
  - the defect CLASS (a read of a state key no producer writes) is still
    invisible to every gate we have. Issue #129 proposes making it a build
    error; that is the follow-on.

Closes #129

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 09:33:05 -05:00
Tim Lingo b842e82f77 test(engine): a runner for tests/, and a failing regression test for #129
tests/ has held 14 test programs for months with no way to run them. CI does
not run them. The convention printed in their own headers
(`elc soul.el && ./soul --test tests/x.el`) refers to a --test flag the El
runtime does not implement. So the tests were documentation, not gates — which
is how a P0 safety regression shipped with a test directory sitting right
there.

scripts/run-el-test.sh compiles and runs one test program. It reuses the
gen-soul-amalgam.sh discovery: elc emits only an extern prototype for a module
that has a .elh beside it, and inlines the bodies when it does not, so a test
importing ../chat.el must be compiled in a scratch tree with the headers
removed. Scratch copy on purpose — the worktree is shared. It runs the binary
under a throwaway HOME so a test can never reach the live engram.

Exit status is the gate: the El tests print failures and still exit 0, so the
runner greps for FAIL lines and for a zero assertion count as well.

tests/test_history_amplification.el pins the invariant #129 violated: the
window the safety screen READS must be the window conv_history_record WRITES.
Not "must be called conv_history" — must AGREE.

THIS COMMIT IS RED BY DESIGN. On this tree the test fails one assertion:

  3. REGRESSION #129 — agentic screen reads the session's own window
    FAIL: distress history escalates the agentic screen to hard_bell
      got:      soft_bell
      expected: hard_bell
  history amplification tests: 8 passed, 1 failed   (runner exit 1)

The next commit turns it green by changing one line. Two legs, one variable —
that is the whole point of committing the test first.

Two flaws in the older harness that this one does not copy: the idiom
`let pass_count = pass_count + 1` inside an assert function declares a local
that dies with the call, so every existing suite prints "0 passed, 0 failed"
regardless of outcome; and a test program without a `cgi` block compiles as a
'utility', which may not reference the self-formation primitives chat.el's
agentic loop calls — it fails to build on a capability violation it never
triggers at runtime.

Refs #129

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 09:32:40 -05:00
Tim Lingo 98ccbd4704 fix(engine): a client that leaves must not kill the daemon, and a long round must say it started
Round 9.1, spec §3 D + ADR 0006 items 2 and 4. Two small changes, both proven
by measurement, both E2E-verified locally against a rebuilt brain.

D1 — SIGPIPE/EPIPE survival (vendor/el-runtime el_runtime.c).
Root cause, at the layer that owns it: the whole HTTP server lives in the C
runtime; .el has no socket primitive. http_send_all() called send() with flags
0 and nothing anywhere in the runtime set a SIGPIPE disposition, so the default
disposition — terminate the process — applied. When a handler finished after
its client had gone (Tim's VM: reply at 116.9 s, client cancelled at 25.0 s),
the second of the four sends that write one reply raised SIGPIPE and the daemon
died: `exited due to SIGPIPE ... ran for 361177ms`, launchd respawn 4 ms later,
every other in-flight session's work lost, user never told.

Fix: SIGPIPE -> SIG_IGN at runtime init and at each http_serve* entry, plus
per-connection SO_NOSIGPIPE / MSG_NOSIGNAL so the guard survives an embedder
resetting dispositions. http_send_all now retries EINTR and preserves errno;
http_send_response classifies it once — a departure is logged as routine
("client left before the reply was written ... reply discarded") and ANY other
errno is logged as a real "send failed: <strerror>". Spec §5.3: the routine
case must not mask a genuine write fault, and it does not.

Proof (scratch HOME + free port, 3 disconnects mid-reply):
  round-9 shipped brain 4402179554… — DIED, exit 141 (128+13 = SIGPIPE), round 1
  round-9 sources rebuilt with this exact recipe — DIED, exit 141, round 1
  this build — SURVIVED 3/3, /health 200 after, still serving the full graph,
  three honest "client left" lines in the log naming Broken pipe / Connection
  reset by peer.

D2 — the round-start marker (chat.el, agentic_loop).
The ledger only ever appended AFTER a round returned, so a healthy first leg
produced zero progress by construction; since server-side web_search moved
inside the outbound call that leg is 60-120 s of silence, which is how a 25 s
client watchdog came to kill a healthy mission. One entry,
{"i":N,"t":"","tool":"__working__"}, written to the existing
run_progress_<session_id> ledger BEFORE each round's outbound call — the wire
shape ChatView.kt:1148 has handled as a life signal since 2026-07-13 and never
received. No new key, no new route, no new lifecycle: a strict subset of WS3
item 3. WS3's run registry is untouched and stays Will's.

Proof (live Anthropic key, real research mission, scratch HOME + free port):
  round-9 baseline — ledger EMPTY for the whole 59.7 s leg
  this build       — {"i":0,"t":"","tool":"__working__"} visible at 18.6 s of a
                     70.0 s leg; both builds returned correct ~4.9 KB answers

Regression: prompt-matrix gate 32/32 on this build (round-9 baseline also 32/32
under the same recipe, so the score is not a build artifact). Soul contract
gate PASS — 27/27 routes, immutability clean. neuron#111 miscompile guard: 0
sites in the generated amalgam this binary was compiled from.

NOT included, deliberately: the regenerated dist/soul.c. CI compiles that file,
so production stays exposed until it is regenerated — the same open ask as
neuron#111 / ui#209. The regen recipe is now known and recorded; landing it is
Will's call, per BUILD-HYGIENE.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 18:23:12 -05:00
Tim Lingo dba755dcec fix(engine): resume reads the bridged tool id from the blob's own field, not from inside the replayed conversation
ROOT CAUSE (round 9; live-repro'd 5/5 this morning, both faces stub-proven by the
prompt-matrix gate). json_get is a first-substring-match scanner (strstr for
'"key":', el_runtime.c). bridge_save serialized the RAW messages array BEFORE the
tool_use_id scalar, so agentic_resume's json_get(blob, 'tool_use_id') returned the
FIRST '"tool_use_id":' occurrence inside the replayed conversation, not the saved
field. The resume guard then preferred that misread over the client's correct
call_id (its two branches both reduced to saved_use_id), attached the tool_result
to the wrong id, and Anthropic 400'd the resume ('unexpected tool_use_id found in
tool_result blocks'), surfaced as {"error":"llm unavailable"}.

ONE MISREAD, TWO FACES — whichever block owns the first tool_use_id in the array:
  FACE 1 (search-then-bridge, the Key West killer): the first occurrence is the
    first web_search_tool_result's srvtoolu_… id — every agentic turn that ran
    server-side web_search and then bridged on a client tool died on approval,
    deterministically (messages.2.content.0 … srvtoolu_…). The write itself had
    already succeeded; only the resume died.
  FACE 2 (multi-cycle missions): with no search, the first occurrence is ROUND 0's
    tool_result block — so every LATER approve/resume cycle replayed the round-0
    client id (stale-resume-id), killing multi-file missions after ~2 files.
  And the shape that PASSES on round 8 confirms the mechanism: a single-cycle
  bridge with no prior tool round has no 'tool_use_id' substring in its messages
  at all (tool_use blocks carry 'id'), so the scan fell through to the blob's own
  field and resumed correctly.

The server_tool_use ↔ web_search_tool_result pairs themselves replay intact — the
defect was a cross-field misread of the blob, the same first-match-scanner class
as BUG-6 (approve 'content' matched inside tool_input, 2026-07-17) and round 8's
citation-block fix.

THE FIX, the pattern not the spot:
  1. bridge_save writes every json_safe'd scalar BEFORE both raw fields (an escaped
     value cannot contain a bare '"key":' byte pattern, so first-match always lands
     on the blob's own fields), and tools_raw (our fixed schema) before messages_raw
     (arbitrary conversation), so the raw extractions cannot first-match into
     model-controlled bytes either. Field order documented as load-bearing.
  2. agentic_resume now honors the client's echoed call_id when present — the value
     with clean provenance (minted from pend_tool_id, never blob-round-tripped) —
     falling back to the saved id only when the client omits it. Each approve cycle
     therefore binds to ITS OWN round's id (kills FACE 2 even against a blob written
     by a pre-fix binary), and an omitted call_id still resumes on the saved id,
     which the reordered blob now reads correctly.

Pattern sweep: the legacy synthetic blob (sessions.el handle_session_approve) embeds
only json_safe'd fields — no raw hazard, untouched. No other json_get read of any
container that embeds raw conversation JSON before the read field.

PROOF: prompt-matrix gate 24/32 RED on the round-8 brain (fails exactly the two
resume classes, named) -> 32/32 GREEN on this build; live-key Key West tracer
3/3 consecutive full round-trips (bridge -> approve-as-the-app -> real completion,
file on disk), plain-chat and weather-only controls PASS; unpatched round-8 brain
and a same-toolchain unpatched baseline build both still fail the identical
sequence with the identical srvtoolu 400 (the test discriminates, and the only
variable between failing and passing builds is this diff).

Refs neuron#109

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-06 11:34:12 -05:00
Tim Lingo 8f3a478771 fix(engine): excise the receipt, do not truncate at it — a leading receipt was erasing whole answers
CAUGHT BY A/B, AND ONLY BY A/B. The previous commit's receipt_strip assumed the receipt is
always TERMINAL and cut everything from the marker onward. It is not always terminal: once
receipt_rule told the model what [[RECEIPT ...]] means, the model sometimes LED with one and
wrote the answer underneath. Cutting at the marker then deleted the entire answer and the
turn returned {"error":"no response"}.

MEASURED, same prompt (two web searches, cited prose), fresh session each run:
    round-7 brain   4 / 4 answered   (471, 473, 473, 544 chars)
    round-8 brain   2 / 7 answered   (five {"error":"no response"})
This looked exactly like a flaky model. It was not — it was mine. Running the two brains
side by side on the same prompt is the only reason it was found, and it is the reason the
A/B is now part of how this class gets tested.

AFTER THE FIX, same protocol:
    round-7 brain   4 / 4   (473, 473, 473, 544)
    round-8 brain   4 / 4   (657, 657, 657, 673)

THE FIX: remove the [[...]] span and keep BOTH sides, instead of truncating at the marker.
An unterminated marker at position 0 is left completely alone — no rule about receipts is
worth erasing an answer over. Bounded four-pass loop rather than a conditional exit, because
rebinding the counter inside an if-expression is the block-expression shape that miscompiles
integer arithmetic under this elc (BUG-PLAINCHAT-1). Verified in the generated C:
    str_slice(rest, (e + 2), str_len(rest))   <- integer addition, correct
    el_str_concat(head, tail)                 <- string concat, correct
and zero el_str_concat(<ident>, str_len(...)) sites across all 49 modules.

SEAM PROOF (FIX C) rides on the same runs — a real two-search cited answer, inspected byte
by byte, in BOTH failure directions:
  missing separator (the round-7 "to.Good", bytes 77 2e 47): 0 hits. Sentence boundaries
    measure 2e 20 4d — "." SPACE "M".
  over-separation (a cited sentence shattered across paragraphs): 0 hits. The answer is one
    continuous paragraph with its sentences intact, which is the direction a blanket
    separator would have broken.

BUILT: sha256 77115f2733e794c5bc4ad1f55b1acaf658f8f4a91cccd423a2d633d94a726cbc

Refs neuron#109

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 23:25:48 -05:00
Tim Lingo 9ea41eed78 fix(engine): the model was signing its own answers with our receipt — strip it
FOUND BY E2E, NOT BY REASONING. The previous commit's design note asserted the provenance
receipt "never reaches the user: it is appended to the history copy, not the reply." That
was FALSE, and only running the thing showed it. On the first live run against the built
DMG brain, two agentic turns out of two came back with

  "Your favourite colour is chartreuse and your project is called Perihelion.

   [[RECEIPT - recorded by the soul, not written by the model: no tools ran on this turn.]]"

— the receipt in the user-visible reply.

MECHANISM: the receipt is stored inside the assistant turn, and the agentic path replays
history VERBATIM as Anthropic message objects. So the model sees its own previous answers
ending in [[RECEIPT ...]] and does the obvious thing — it imitates the format and signs the
next answer the same way. The plain path did NOT leak, which is the tell: there, history is
rendered into the SYSTEM prompt as labelled lines rather than replayed as assistant turns,
and a model imitates its own turns far more readily than a transcript.

FIX, two layers, because one of them is not a guarantee:
  - receipt_rule() names the marker in both system prompts (plain and agentic): these lines
    are written by the system, read them as evidence, never write one. Reduces occurrence.
  - receipt_strip() truncates any [[RECEIPT ...]] out of model output before it becomes the
    reply — plain path in layered_generate, agentic path on final_text in agentic_loop.
    Deterministic. A guard that depends on the model choosing to obey is exactly the class of
    thing round 8 exists to stop shipping, so the instruction is the optimisation and the
    strip is the guarantee.
Placed ABOVE agentic_loop's empty-check on purpose: a turn whose entire output was an
imitated receipt has produced no answer, and must be reported as no answer.

The receipt stays in HISTORY, which is the whole point and is proven to work: asked "What
source did you use for that?" one turn after a live web_search, this brain answered
"I used Weather Underground (https://www.wunderground.com/weather/is/reykjav%C3%ADk) for the
current temperature in Reykjavik" — a real source, no apology. That is the false confession
dead, and it is dead BECAUSE the model can read the receipt.

BUILT: 887,112 bytes, sha256 54a2eff84d4fa44f8d2db6781dcf225df4b5075a8ec5f1058018bd40cc1af10b
BUG-PLAINCHAT-1 miscompile guard: zero sites.

Refs neuron#109

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 23:10:53 -05:00
Tim Lingo ff421d39f6 fix(engine): history keeps its provenance and its session — the false confession and the blank stare
DESIGN FIT: three of round 7's five defects share ONE root — the conversation-history
layer persists only {role, content}, discarding tool provenance, session scoping, and the
distinction between a real user turn and an internal utility call. Fixes A and B RESTORE
Will's design rather than extend it: his agentic path already scopes history per session,
the plain path never got it, and his own source carries the TODO admitting the resulting
race (chat.el, handle_chat: "process-global key; concurrent /api/chat requests without
session_id race on this read-append-write"). Fix C repairs one join Will wrote that was
correct for a year and one we added last week. E1/E2 are ours.

FIX A — tool provenance in history (kills the FALSE CONFESSION)
  Root cause, EXECUTED-verified: handle_chat_agentic recorded turns via hist_append, which
  emits {"role","content"} only. server_tool_use blocks, web_search_tool_result blocks and
  every citation were discarded, then replayed as text. On the next turn the model saw a
  data-rich answer with zero evidence a search had happened, and its own permanent rule
  ("never describe a search you did not perform") left one conclusion available: that it
  had fabricated the data. It apologised for a search it HAD run — four independent lines
  of evidence confirm the search was real. The defect is not the model's honesty. It is
  that we deleted the evidence and then asked it to account for itself.
  Change: agentic_loop accumulates the source URLs it already walks past (citations and
  web_search_tool_result content) and returns them as "sources"; handle_chat_agentic folds
  tools_used + sources into a receipt line stored WITH the assistant turn. Receipts are
  unconditional — a negative receipt ("no tools ran") is the other half of the guarantee,
  because "no evidence of a tool" and "evidence of no tool" were previously identical in
  the transcript. conv_history_block splits the receipt off before snipping so a long
  answer cannot truncate away the evidence. The user never sees it: it is appended to the
  history copy, not the reply.

FIX B — one history key for both paths (kills the BLANK STARE)
  Root cause, EXECUTED-verified: the agentic path keyed history on session_hist_<id>; the
  plain path was hard-wired to the process-global conv_history and never read session_id.
  One conversation, two buckets. Proven in the guest engram: the scoped node held exactly
  two turns starting at "Try again" while the earlier exchanges sat unscoped.
  Change: conv_hist_key/conv_hist_label are now the single definition, used by BOTH paths;
  session_id is threaded route -> layered_cycle -> layered_generate / conv_history_record.
  The 2-line fallback (plain path reads the agentic key) was REJECTED: it keeps the
  process-global bucket as a live write target, which is the bleed the TODO describes.
  Also found and closed while threading: layered_cycle read session_id from the state key
  "current_session_id", which is read here and WRITTEN NOWHERE in the entire source. It
  was unconditionally "", so TODO(reliability #4) — per-session steward continuity — was
  dead code that could never fire. It fires now.
  LAZY SESSION, decided explicitly: we create the session EAGERLY at the door (app half,
  ui#223) rather than migrating orphaned turns. Migration would copy the CONTENTS of a
  process-global bucket, possibly another conversation's, into a named session — the bleed,
  performed deliberately. Eager creation makes the situation impossible instead. Migration
  is deliberately not implemented and must not be added without solving provenance first.

FIX C — the two text-join seams ("to.Good", byte-verified 0x77 0x2e 0x47)
  Two bare `+` joins, written a year apart, had drifted into two answers to one question:
  within-response block joins (Will's, 2026-05-03, latent until server-side web_search
  began interleaving non-text blocks) and across-round joins (ours, 62af564).
  Change: one named rule, text_join_sep, at both sites. NOT a blanket separator — a cited
  answer splits MID-SENTENCE ("The current temperature is " + "86°F" + ", with "), so a
  blanket separator shatters every sourced sentence. The rule takes the one bit that
  distinguishes the cases: whether a NON-TEXT block intervened. Hoisting it also makes the
  fix verifiable in the shipped binary, which an inline `+` is not.

FIX E1 — utility generations stay out of the transcript
  Title generation ("Write a 3-6 word title...") and insight passes ran down the same plain
  door as a real message and were recorded as if the user had typed them; the same calls are
  the "model":"unknown" rows in usage.jsonl. is_utility_request reads an explicit utility
  flag from the app, with the __title__/__insight__ id prefixes as a fallback for older
  clients. Answered normally, never recorded.

FIX E2 — OPERATOR IDENTITY is scoped to tool-capable turns
  The block (env USER/HOME, closing "This is a hard rule") was prepended to EVERY system
  prompt including chat mode. On a Tools:Off turn there is no filesystem in reach, so it
  governed nothing and merely supplied the loudest fact in the prompt — which is why the
  model opened a fresh conversation with "You're test, on your machine at /Users/test".
  Hoisted to operator_identity_block() and gated on !chat_mode. Unchanged wherever a file
  or command tool can actually be reached.

ALSO: agentic_loop's per-session history persist had a second hand-rolled copy of
conv_history_persist with a different label expression, different salience scores and
different tags for the same node. Since both now derive the label from conv_hist_label and
engram_node_full upserts by label, two score policies were writing one node. Collapsed to
one writer.

BUILD NOTE: dist/elp-c-decls.h is force-included by the documented link recipe and carried
the OLD C arities, so it is updated here. This is the build-support header, NOT the stale
generated dist/soul.c — no dist/*.c was read or edited; all engine changes are .el source.
chat.elh/soul.elh are committed because a first-pass build against the old signatures FAILS
(measured); the other regenerated headers are reverted as unrelated churn.

BUILT: 887,000 bytes, sha256 d632b061ad75269d6adeb52578d030eaf49e895d91289d7f946b19c08450d728
Zero el_str_concat(<int>, str_len(...)) sites (the BUG-PLAINCHAT-1 miscompile guard).
web_search_20250305 and disable_parallel_tool_use both still present — PR #108's web search
and the ADR-0005 stopgap are intact.

Refs neuron#109 (builds on it), neuron#78 (Receipt Contract — the real fix A is a stopgap for)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 22:59:54 -05:00
Tim Lingo 635f6febe4 feat(engine): plain chat generates at L3 — inside the safety cycle, not around it
Neuron Soul CI / build (pull_request) Failing after 12m8s
Neuron Soul CI / deploy (pull_request) Has been skipped
Non-agentic /api/chat (the desktop app's default "Tools: Off" mode) returned the
user's own screened text as a bare non-JSON string. Every JSON client failed to
parse it and showed "Couldn't reach Neuron - it may be offline."

Root cause: f52d5bd (2026-06-11) correctly moved the route onto the layer spine
(handle_chat -> layered_cycle), but L3 never got a generator — imprint_respond()
annotates its input and returns it. Two pieces of the architecture were already
waiting for that step: layered_cycle parks a bell directive in the state key
build_system_prompt is written to consume, and build_system_prompt carries a
chat_mode ("no tools") flag with no live caller.

The fix composes rather than replaces. Wiring handle_chat would have removed
safety_screen, the hard-bell short-circuit, the whole stewardship layer and
safety_validate — the only enforcing output gate in the codebase — in exchange
for a working reply (see _engine-websearch-20260804/SAFETY-STOP.md). Instead
layered_cycle keeps every gate, in order, and gains a generation step between
imprint_respond and safety_validate.

  L1 screen -> guard -> hard-bell short-circuit -> L2a -> L2b -> L2c
    -> L3 imprint_respond (prompt) -> L3b layered_generate (NEW) -> L1 validate

- chat.el:  NEW layered_generate (L3 generation, no tools offered),
            conv_history_block, conv_history_record.
            FIX build_system_prompt never concatenated no_tools_rule into its
            return — the "[NO TOOLS THIS TURN]" rule reached no model at all.
            handle_chat annotated DO-NOT-WIRE with the reason.
- soul.el:  layered_cycle gains L3b + post-validation turn bookkeeping.
- routes.el: NEW plain_chat_envelope; all three /api/chat dispatch sites wrap the
            cycle's output. Built OUTSIDE the cycle so safety_validate always sees
            raw model text — nothing to unwrap or rebuild on the crisis path.
            Emits both `reply` and `response`: the desktop app reads `reply`,
            the CLI tools and telegram-gateway read `response`.

Also fixes BUG-PLAINCHAT-1, a pre-existing CRITICAL crash on the crisis path.
elc compiles `let n: Int = pos + str_len(marker)` to el_str_concat() — string
concat on two integers — inside a block-expression initializer, segfaulting the
daemon (SIGSEGV in strlen). Six inline copies of the same " | ts:" parser had it:
two in layered_cycle L2c, two in engram_compile (live on the AGENTIC path too),
two in affective_context_prefix. A distress turn following an earlier affective
turn killed the whole process. Proven pre-existing: an unmodified baseline binary
crashes identically, and the same bad C is in the committed dist/soul.c. Fixed by
hoisting to one top-level function, affective_node_ts(), where the expression
compiles to integer addition — verified in the generated C.

Proof (throwaway HOME/engram, explicit NEURON_PORT, live chain untouched):
- Plain turn returns a JSON envelope with the provider's answer, not an echo.
- Captured request body: no `tools`, no `tool_choice`; system prompt carries the
  NO-TOOLS rule. Tools:Off means no tool is offered, structurally.
- Hard bell: canned 988 message, and the provider request count does not move —
  the message never reaches a model.
- Soft bell + a 2-char model reply: safety_validate's care phrase is appended to
  the MODEL's output. Output gate acting, on this route.
- The L1 bell directive now reaches the model here for the first time (the state
  addendum had a producer and no consumer).
- test_layered_cycle PASS; all six El suites byte-identical to baseline.
- verify-soul-contract.sh (bash 5.3): GATE PASS, 27/27, immutability PASS.
- The crash sequence that killed the baseline daemon now returns HTTP 200.

Not proven: no live Anthropic call — the login keychain refuses the key to a
non-interactive process (rc=24 errSecInteractionNotAllowed). Details and the
one-command close-out are in _engine-plainchat-20260805/README.md §7.

Builds on PR #108. dist/soul.c deliberately not regenerated — Will's toolchain.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 09:11:03 -05:00
18 changed files with 3177 additions and 901 deletions
+19
View File
@@ -889,10 +889,29 @@ fn awareness_run() -> Void {
state_set("soul.last_beat_ts", int_to_str(now_ts))
// Persist in-process Engram (sessions, memories, conversation nodes)
// to local snapshot so they survive restarts.
// FILE MODE ONLY: "soul_snapshot_path" is set exclusively in the
// genesis+safe_to_seed branch of soul.el, and safe_to_seed is
// unconditionally false when ENGRAM_URL is set. In HTTP mode the
// owner persists; the soul must not (soul.el:571-573).
let snap_path: String = state_get("soul_snapshot_path")
if !str_eq(snap_path, "") {
mem_save(snap_path)
}
// WRITE-THROUGH RETRY (neuron#117). The HTTP-mode counterpart of the
// save above: hand anything still spooled to the persistence owner.
//
// This is the retry arm of the whole design. Deltas that could not be
// pushed owner down, owner restarting, transient refusal stay on
// disk and are re-offered here every heartbeat until they land. It is
// also the catch-all for writes made by the awareness loop itself,
// which never passes through the HTTP handler's flush point.
//
// No-op with no HTTP call when the spool is empty or ENGRAM_URL is
// unset, so an idle soul in file mode pays nothing for this.
let wt_pushed: Int = wt_drain()
if wt_pushed < 0 {
ise_post("{\"event\":\"write_through_backlog\",\"ts\":" + int_to_str(now_ts) + "}")
}
}
// Curiosity scan: idle-gated AND wall-clock based. Only fires when the
+716 -127
View File
File diff suppressed because it is too large Load Diff
+26 -3
View File
@@ -1,4 +1,4 @@
// auto-generated by elc --emit-header do not edit
// auto-generated by elc --emit-header - do not edit
extern fn chat_default_model() -> String
extern fn engram_numeric_valid(s: String) -> Bool
extern fn parse_float_x100(s: String) -> Int
@@ -16,18 +16,35 @@ extern fn engram_nodes_merge(a: String, b: String) -> String
extern fn id_in_seen(node_id: String, seen: String) -> Bool
extern fn add_to_seen(seen: String, node_id: String) -> String
extern fn engram_extract_ids(nodes_json: String) -> String
extern fn affective_node_ts(node_json: String) -> Int
extern fn engram_compile(intent: String) -> String
extern fn distill_transcript(transcript: String) -> String
extern fn json_safe(s: String) -> String
extern fn current_engine_note(model: String) -> String
extern fn bounded_persona_floor() -> String
extern fn operator_identity_block() -> String
extern fn build_system_prompt(ctx: String, chat_mode: Bool) -> String
extern fn hist_append(hist: String, role: String, content: String) -> String
extern fn conv_hist_key(session_id: String) -> String
extern fn conv_hist_label(session_id: String) -> String
extern fn is_utility_request(body: String, session_id: String) -> Bool
extern fn provenance_scan_urls(arr: String, acc: String) -> String
extern fn provenance_add_sources(block: String, btype: String, has_cit: Bool, cit_raw: String, acc: String) -> String
extern fn provenance_names(tools_used: String) -> String
extern fn text_join_sep(accumulated: String, incoming: String, after_interruption: Bool) -> String
extern fn receipt_rule() -> String
extern fn receipt_strip(s: String) -> String
extern fn tool_receipt(tools_used: String, sources: String) -> String
extern fn hist_trim(hist: String) -> String
extern fn hist_trim_with_bell_guard(hist: String) -> String
extern fn clean_llm_response(s: String) -> String
extern fn conv_history_persist(hist: String) -> Void
extern fn conv_history_load() -> String
extern fn conv_history_persist(session_id: String, hist: String) -> Void
extern fn conv_history_load(session_id: String) -> String
extern fn conv_history_record(session_id: String, user_msg: String, assistant_msg: String, receipt: String) -> Void
extern fn conv_history_block(session_id: String) -> String
extern fn layered_generate(prompt: String, imprint_id: String, session_id: String) -> String
extern fn session_preload_bullets(nodes: String, max_bullets: Int, snip_len: Int) -> String
extern fn affective_context_prefix() -> String
extern fn handle_chat(body: String) -> String
extern fn handle_see(body: String) -> String
extern fn studio_tools_json() -> String
@@ -37,6 +54,8 @@ extern fn llm_wire_format() -> String
extern fn json_escape(s: String) -> String
extern fn openai_chat_complete(model: String, base_url: String, api_key: String, safe_sys: String, messages_json: String) -> String
extern fn agentic_tools_literal() -> String
extern fn web_search_tool_json() -> String
extern fn strip_client_web_search(tools_inner: String) -> String
extern fn agentic_tools_with_web() -> String
extern fn connector_tools_json() -> String
extern fn agentic_tools_all() -> String
@@ -46,6 +65,10 @@ extern fn call_neuron_mcp(tool_name: String, args: String) -> String
extern fn agent_workspace_root() -> String
extern fn path_within_root(path: String, root: String) -> Bool
extern fn resolve_in_root(path: String, root: String) -> String
extern fn run_command_is_readonly(cmd: String) -> Bool
extern fn cmd_abs_escape_at(cmd: String, root: String, needle: String) -> Bool
extern fn run_command_guard(cmd: String, root: String) -> String
extern fn classify_tool_risk(tool_name: String, tool_input: String) -> String
extern fn dispatch_tool(tool_name: String, tool_input: String) -> String
extern fn is_builtin_tool(tool_name: String) -> Bool
extern fn next_bridge_id() -> String
Generated Vendored
+17 -3
View File
@@ -5,6 +5,15 @@ el_val_t add_punct(el_val_t s, el_val_t intent);
el_val_t add_to_seen(el_val_t seen, el_val_t node_id);
el_val_t aff_try_slot(el_val_t slot_json, el_val_t aff_7d_ts, el_val_t acc_key);
el_val_t affective_context_prefix(void);
el_val_t is_utility_request(el_val_t body, el_val_t session_id);
el_val_t operator_identity_block(void);
el_val_t provenance_add_sources(el_val_t block, el_val_t btype, el_val_t has_cit, el_val_t cit_raw, el_val_t acc);
el_val_t provenance_names(el_val_t tools_used);
el_val_t provenance_scan_urls(el_val_t arr, el_val_t acc);
el_val_t text_join_sep(el_val_t accumulated, el_val_t incoming, el_val_t after_interruption);
el_val_t receipt_rule(void);
el_val_t receipt_strip(el_val_t s);
el_val_t tool_receipt(el_val_t tools_used, el_val_t sources);
el_val_t agent_number(el_val_t agent);
el_val_t agent_person(el_val_t agent);
el_val_t agent_workspace_root(void);
@@ -151,8 +160,12 @@ el_val_t cmd_abs_escape_at(el_val_t cmd, el_val_t root, el_val_t needle);
el_val_t connectd_get(el_val_t suffix);
el_val_t connectd_post(el_val_t suffix, el_val_t body);
el_val_t connector_tools_json(void);
el_val_t conv_history_load(void);
el_val_t conv_history_persist(el_val_t hist);
el_val_t conv_hist_key(el_val_t session_id);
el_val_t conv_hist_label(el_val_t session_id);
el_val_t conv_history_block(el_val_t session_id);
el_val_t conv_history_load(el_val_t session_id);
el_val_t conv_history_persist(el_val_t session_id, el_val_t hist);
el_val_t conv_history_record(el_val_t session_id, el_val_t user_msg, el_val_t assistant_msg, el_val_t receipt);
el_val_t cop_article(el_val_t gender, el_val_t number, el_val_t definite);
el_val_t cop_bwk_future(el_val_t prefix);
el_val_t cop_bwk_perfect(el_val_t prefix);
@@ -782,7 +795,8 @@ el_val_t lang_profile_uga(void);
el_val_t lang_profile_zh(void);
el_val_t lang_profile(el_val_t code, el_val_t word_order, el_val_t morph_type, el_val_t has_case, el_val_t has_gender, el_val_t script_dir, el_val_t agreement, el_val_t null_subject);
el_val_t lang_word_order(el_val_t profile);
el_val_t layered_cycle(el_val_t raw_input);
el_val_t layered_cycle(el_val_t raw_input, el_val_t session_id, el_val_t utility);
el_val_t layered_generate(el_val_t prompt, el_val_t imprint_id, el_val_t session_id);
el_val_t lex_class(el_val_t entry);
el_val_t lex_form(el_val_t entry, el_val_t idx);
el_val_t lex_pos(el_val_t entry);
Generated Vendored
+15
View File
@@ -1,3 +1,18 @@
//
// STALE BUNDLE DO NOT BUILD. UNSAFE CHAT PATH.
//
// This concatenated bundle is a snapshot, not a source of truth, and it is stale in
// a way that matters for safety: it wires /api/chat straight to handle_chat and
// contains NO layered_cycle at all (verified: zero occurrences in the bundled code
// the only textual hit in this file is this banner). A binary built from
// this file would run chat with no enforcing input gate (no safety_screen, no
// hard-bell short-circuit) and no enforcing output gate (no safety_validate).
//
// Build from the .el sources via manifest.el (entry soul.el), or from dist/soul.c.
// Nothing in the repo references this file. It is kept only as a historical artifact
// and should be deleted once Will confirms nothing external depends on it.
// (Flagged 2026-08-04 in _engine-websearch-20260804/SAFETY-STOP.md; banner added
// 2026-08-05 with the plain-chat generation fix.)
// language-profile.el - Language profile data and accessors.
//
// A language profile is a slot map ([String] key-value list) describing the
Generated Vendored
+1283 -662
View File
File diff suppressed because one or more lines are too long
+21 -9
View File
@@ -1,9 +1,11 @@
import "persist.el"
fn tier_working() -> String { return "Working" }
fn tier_episodic() -> String { return "Episodic" }
fn tier_canonical() -> String { return "Canonical" }
fn mem_store(content: String, label: String, tags: String) -> String {
let id: String = engram_node_full(
let id: String = wt_node(
content,
"Memory",
label,
@@ -17,13 +19,23 @@ fn mem_store(content: String, label: String, tags: String) -> String {
println("[memory] write rejected by engram (empty id): label=" + label)
return ""
}
// Read back to verify the node actually persisted guards against silent write failures.
let readback: String = engram_get_node_json(id)
if str_eq(readback, "") || str_eq(readback, "{}") {
println("[memory] WRITE VERIFY FAILED: label=" + label + " id=" + id + " — node absent after write")
return ""
// wt_node has already read the node back locally and returns "" if it did
// not land, so the old duplicate read-back here is gone.
//
// HONESTY (neuron#117): the receipt now says WHERE the write is.
// The old unconditional "write verified" line asserted against the soul's
// own RAM true in memory, false on disk and printed ~115,000 times on
// Tim's machine while the canonical snapshot sat frozen for three days.
// wt_commit flushes the spool and then asks the OWNER. When it says false
// the node is real and recallable but not yet durable, and the log says so
// rather than claiming a save that did not happen. The id is still returned:
// the local write DID succeed, and the queued delta will be retried.
let durable: Bool = wt_commit(id)
if durable {
println("[memory] write persisted at owner: " + id + " label=" + label)
} else {
println("[memory] write IN MEMORY ONLY (queued for owner, not yet durable): " + id + " label=" + label)
}
println("[memory] write verified: " + id + " ok")
return id
}
@@ -51,12 +63,12 @@ fn mem_strengthen(node_id: String) -> Void {
// memory.el (imported first) so awareness.el and neuron-api.el can both call it.
fn mem_tombstone(node_id: String) -> String {
let tags: String = "[\"Tombstone\",\"status:deleted\"]"
let marker: String = engram_node_full(
let marker: String = wt_node(
node_id, "Tombstone", "tombstone:" + node_id,
el_from_float(0.01), el_from_float(0.01), el_from_float(1.0),
"Episodic", tags)
if !str_eq(marker, "") {
engram_connect(marker, node_id, el_from_float(1.0), "tombstones")
wt_edge(marker, node_id, el_from_float(1.0), "tombstones")
}
return marker
}
+36 -27
View File
@@ -195,15 +195,24 @@ fn api_compact_activated(raw: String, max_items: Int, snip: Int) -> String {
}
// api_persisted read-back-after-write guard against hallucinated saves.
// After a write builtin returns an id, confirm the node is actually queryable
// via engram_get_node_json(id) (returns "" or "null" when missing). Returns
// true only when the node is genuinely persisted.
//
// WIDENED FOR neuron#117. This function is the single gate every MCP write
// handler passes through before it reports success (10 call sites), which makes
// it the right place to close the honesty gap rather than editing ten receipts.
//
// It used to read back from engram_get_node_json the SOUL'S OWN in-process
// graph. In HTTP-engram mode that asserts the wrong thing: the soul is not the
// persistence owner, so a node present in its RAM and absent from the owner read
// as "persisted" and then vanished on the next restart. The guard was doing
// exactly what its comment promised and still certifying writes that did not
// survive. It now flushes the write-through spool and asks the OWNER.
//
// In file mode (no ENGRAM_URL) the soul IS the owner and wt_commit collapses to
// the original local read-back unchanged behaviour, which is what keeps this
// reversible.
fn api_persisted(id: String) -> Bool {
if str_eq(id, "") { return false }
let node: String = engram_get_node_json(id)
// engram_get_node_json returns "{}" (empty object) when node is not found not "" or "null".
// Check all three to guard against any runtime variation.
return !str_eq(node, "") && !str_eq(node, "null") && !str_eq(node, "{}")
return wt_commit(id)
}
// api_not_persisted standard error for a write that did not read back.
@@ -342,7 +351,7 @@ fn handle_api_remember(body: String) -> String {
let inner: String = str_slice(base_tags, 1, str_len(base_tags) - 1)
"[" + inner + ",\"project:" + project + "\"]"
}
let id: String = engram_node_full(content, "Memory", "memory:remembered",
let id: String = wt_node(content, "Memory", "memory:remembered",
sal, sal, el_from_float(0.9),
"Episodic", final_tags)
if !api_persisted(id) { return api_not_persisted(id) }
@@ -369,7 +378,7 @@ fn handle_api_node_create(body: String) -> String {
if str_eq(importance, "low") { 0.25 } else { 0.5 }
}
}
let id: String = engram_node_full(content, node_type, label,
let id: String = wt_node(content, node_type, label,
sal, sal, el_from_float(0.9),
tier, tags)
if !api_persisted(id) { return api_not_persisted(id) }
@@ -422,11 +431,11 @@ fn handle_api_node_update(body: String) -> String {
}
let body_tags: String = json_get(body, "tags")
let tags: String = if str_eq(body_tags, "") { "[\"" + node_type + "\"]" } else { body_tags }
let new_id: String = engram_node_full(content, node_type, label,
let new_id: String = wt_node(content, node_type, label,
el_from_float(0.5), el_from_float(0.5), el_from_float(0.8),
tier, tags)
if !api_persisted(new_id) { return api_not_persisted(new_id) }
engram_connect(new_id, id, el_from_float(0.9), "supersedes")
wt_edge(new_id, id, el_from_float(0.9), "supersedes")
return "{\"id\":\"" + new_id + "\",\"supersedes\":\"" + id + "\",\"ok\":true}"
}
@@ -498,7 +507,7 @@ fn handle_api_capture_knowledge(body: String) -> String {
let full: String = if str_eq(title, "") { content } else { title + ": " + content }
let lbl: String = str_slice(title, 0, 80)
let tags: String = "[\"Knowledge\",\"captured\"]"
let id: String = engram_node_full(full, "Knowledge", lbl,
let id: String = wt_node(full, "Knowledge", lbl,
el_from_float(0.85), el_from_float(0.8), el_from_float(0.9),
"Episodic", tags)
if !api_persisted(id) { return api_not_persisted(id) }
@@ -513,12 +522,12 @@ fn handle_api_evolve_knowledge(body: String) -> String {
if !str_eq(prior_id, "") && is_protected_node(prior_id) { return api_err_protected(prior_id) }
let tags: String = "[\"Knowledge\",\"evolved\"]"
// Empty label engram_node_full derives content[:60] (LABEL FIX 2026-07-23).
let new_id: String = engram_node_full(content, "Knowledge", "",
let new_id: String = wt_node(content, "Knowledge", "",
el_from_float(0.75), el_from_float(0.75), el_from_float(0.9),
"Episodic", tags)
if !api_persisted(new_id) { return api_not_persisted(new_id) }
if !str_eq(prior_id, "") {
engram_connect(new_id, prior_id, el_from_float(0.9), "supersedes")
wt_edge(new_id, prior_id, el_from_float(0.9), "supersedes")
}
return "{\"id\":\"" + new_id + "\",\"supersedes\":\"" + prior_id + "\",\"ok\":true}"
}
@@ -535,11 +544,11 @@ fn handle_api_promote_knowledge(body: String) -> String {
"[\"Knowledge\",\"tier:canonical\",\"disposition:stable\"]"
} else { tags_raw }
// Empty label engram_node_full derives content[:60] (LABEL FIX 2026-07-23).
let new_id: String = engram_node_full(content, "Knowledge", "",
let new_id: String = wt_node(content, "Knowledge", "",
el_from_float(0.9), el_from_float(0.9), el_from_float(1.0),
"Canonical", tags)
if !api_persisted(new_id) { return api_not_persisted(new_id) }
engram_connect(new_id, prior_id, el_from_float(0.95), "supersedes")
wt_edge(new_id, prior_id, el_from_float(0.95), "supersedes")
return "{\"ok\":true,\"new_id\":\"" + new_id + "\",\"supersedes\":\"" + prior_id + "\"}"
}
@@ -562,7 +571,7 @@ fn handle_api_define_process(body: String) -> String {
if str_eq(content, "") { return api_err("content is required") }
let label: String = if str_eq(name, "") { "process:unnamed" } else { "process:" + name }
let tags: String = "[\"Process\"]"
let id: String = engram_node_full(content, "Process", label,
let id: String = wt_node(content, "Process", label,
el_from_float(0.8), el_from_float(0.8), el_from_float(0.9),
"Canonical", tags)
if !api_persisted(id) { return api_not_persisted(id) }
@@ -647,7 +656,7 @@ fn handle_api_tune_config(body: String) -> String {
if str_eq(key, "") { return api_err("key is required") }
let content: String = "config:" + key + "=" + value
let tags: String = "[\"ConfigEntry\",\"config\"]"
let id: String = engram_node_full(content, "ConfigEntry", key,
let id: String = wt_node(content, "ConfigEntry", key,
el_from_float(0.85), el_from_float(0.85), el_from_float(0.9),
"Canonical", tags)
if !api_persisted(id) { return api_not_persisted(id) }
@@ -694,7 +703,7 @@ fn handle_api_link_entities(body: String) -> String {
if is_protected_node(to_id) { return api_err_protected(to_id) }
let relation: String = json_get(body, "relation")
let eff_relation: String = if str_eq(relation, "") { "associates" } else { relation }
engram_connect(from_id, to_id, el_from_float(0.5), eff_relation)
wt_edge(from_id, to_id, el_from_float(0.5), eff_relation)
return "{\"ok\":true,\"from_id\":\"" + from_id + "\",\"to_id\":\"" + to_id + "\",\"relation\":\"" + eff_relation + "\"}"
}
@@ -727,11 +736,11 @@ fn handle_api_evolve_memory(body: String) -> String {
}
}
let tags: String = "[\"Memory\",\"evolved\"]"
let new_id: String = engram_node_full(content, "Memory", "memory:evolved",
let new_id: String = wt_node(content, "Memory", "memory:evolved",
sal, sal, el_from_float(0.9),
"Episodic", tags)
if !str_eq(prior_id, "") && !str_eq(new_id, "") {
engram_connect(new_id, prior_id, el_from_float(0.9), "supersedes")
wt_edge(new_id, prior_id, el_from_float(0.9), "supersedes")
}
return "{\"id\":\"" + new_id + "\",\"supersedes\":\"" + prior_id + "\",\"ok\":true}"
}
@@ -789,11 +798,11 @@ fn handle_api_cultivate(body: String) -> String {
let content: String = json_get(body, "content")
if str_eq(content, "") { return api_err("content is required") }
let tags: String = "[\"Knowledge\",\"evolved\",\"cultivated\"]"
let new_id: String = engram_node_full(content, "Knowledge", "knowledge:cultivated",
let new_id: String = wt_node(content, "Knowledge", "knowledge:cultivated",
el_from_float(0.75), el_from_float(0.75), el_from_float(0.9),
"Episodic", tags)
if !str_eq(prior_id, "") && !str_eq(new_id, "") {
engram_connect(new_id, prior_id, el_from_float(0.9), "supersedes")
wt_edge(new_id, prior_id, el_from_float(0.9), "supersedes")
}
return "{\"id\":\"" + new_id + "\",\"supersedes\":\"" + prior_id + "\",\"ok\":true,\"cultivated\":true}"
}
@@ -809,11 +818,11 @@ fn handle_api_cultivate(body: String) -> String {
}
}
let tags: String = "[\"Memory\",\"evolved\",\"cultivated\"]"
let new_id: String = engram_node_full(content, "Memory", "memory:cultivated",
let new_id: String = wt_node(content, "Memory", "memory:cultivated",
sal, sal, el_from_float(0.9),
"Episodic", tags)
if !str_eq(prior_id, "") && !str_eq(new_id, "") {
engram_connect(new_id, prior_id, el_from_float(0.9), "supersedes")
wt_edge(new_id, prior_id, el_from_float(0.9), "supersedes")
}
return "{\"id\":\"" + new_id + "\",\"supersedes\":\"" + prior_id + "\",\"ok\":true,\"cultivated\":true}"
}
@@ -833,7 +842,7 @@ fn handle_api_cultivate(body: String) -> String {
if str_eq(to_id, "") { return api_err("to_id is required") }
let relation: String = json_get(body, "relation")
let eff_relation: String = if str_eq(relation, "") { "associates" } else { relation }
engram_connect(from_id, to_id, el_from_float(0.5), eff_relation)
wt_edge(from_id, to_id, el_from_float(0.5), eff_relation)
return "{\"ok\":true,\"from_id\":\"" + from_id + "\",\"to_id\":\"" + to_id + "\",\"relation\":\"" + eff_relation + "\",\"cultivated\":true}"
}
@@ -868,7 +877,7 @@ fn handle_api_consolidate(body: String) -> String {
if !str_eq(summary, "") {
let safe_summary: String = str_replace(summary, "\"", "'")
let tags: String = "[\"SessionSummary\",\"consolidate\"]"
let summary_id: String = engram_node_full(
let summary_id: String = wt_node(
"[session-summary] " + safe_summary,
"SessionSummary", "session:summary",
el_from_float(0.7), el_from_float(0.7), el_from_float(0.9),
+426
View File
@@ -0,0 +1,426 @@
// persist.el the soulengram WRITE-THROUGH boundary (neuron#117).
//
// WHY THIS FILE EXISTS
// soul.el:571-573 states the ownership rule: "when ENGRAM_URL is set the HTTP
// Engram owns persistence the soul must NEVER write to the local snapshot
// (not the persistence owner)." The soul obeys the NEGATIVE half. The POSITIVE
// half how a write made inside the soul actually REACHES the owner was
// never built. Sync is pull-only (awareness.el `/api/sync` -> engram_load_merge),
// so every node the soul creates lives in its process RAM and is shed on
// restart. Measured live 2026-08-07: soul node_count=102184, engram
// node_count=79197 ~23k nodes existing nowhere but RAM.
//
// SCOPE NOTE ON THE PATENT (corrects an earlier internal reading)
// Engram provisional claims 15-18 describe a delta-sync protocol "with peer
// Engram instances"; claim 17's pull-then-push sequence is PEER-ENGRAM to
// PEER-ENGRAM. The soul is NOT a peer Engram it is a CALLER of the database
// system API (cf. claim 27, "invoked explicitly by a caller of the database
// system API"). So claim 17 does not specify a soul↔engram contract and is not
// cited as authority here. This design is derived from the ownership rule
// alone: the owner owns the writes, therefore the soul must HAND writes to the
// owner and must never write the owner's file itself.
//
// THE MECHANISM, AND WHY NOT `POST /api/nodes`
// The obvious route is the one the persona/boot-counter write-backs already
// use, POST /api/nodes. It is the wrong instrument here, verified against the
// live engram binary in a sandbox:
// - it mints a NEW server-side id (engram_node), so the soul's id and the
// owner's id diverge the next /api/sync pull re-imports the node as a
// DUPLICATE, and any edge referencing the soul's id never resolves;
// - it accepts only {content, node_type, salience} and drops label, tier,
// tags, importance, confidence, metadata. A probe posted with tier
// "Canonical" came back tier "Working", importance 0.5.
// POST /api/load-merge (Will's own route, el `dc39a61`) is the right one:
// - engram_load_merge PRESERVES the id and every field;
// - it dedups nodes by id and edges by (from_id,to_id,relation), so a
// re-submitted delta is a NO-OP retry safety is free, and it is the same
// local-wins semantics the graph already uses;
// - it calls persist_canonical() THE OWNER writes its own canonical file.
// The soul never touches it. The ownership rule is honoured in its
// strongest form rather than worked around;
// - it returns real counts {ok, nodes_added, edges_added, node_count},
// so a receipt can be a MEASUREMENT instead of a fixed success shape.
//
// SPOOL-AND-DRAIN, AND WHY IT IS NOT JUST A DIRECT POST
// Measured in a sandbox against a 79k-node / 176MB graph (live scale): one
// load-merge costs ~0.38s, essentially all of it the owner's persist_canonical.
// A chat turn writes 5-7 nodes; pushing each separately would add ~2.7s per
// turn. So writes are STAGED and pushed in one coalesced batch.
// The staging buffer is the FILESYSTEM, not process state, because the soul
// serves each HTTP connection on its own pthread (el_runtime http_serve_async)
// and a shared in-process buffer would lose entries to a read-modify-write
// race silently, which is the one failure mode this file exists to end.
// One file per write, named with uuid_v4, is race-free by construction and
// buys a property a memory buffer cannot: writes that could not be pushed
// SURVIVE A SOUL CRASH and are drained on the next boot.
//
// WHAT IS DELIBERATELY NOT PUSHED
// - InternalStateEvent / heartbeat telemetry. Will's own carve-out, stated in
// engram server.el 8f8ccc9: "48h-pruned, loss-tolerant, ~2/min; snapshotting
// 28MB per heartbeat is waste."
// NOTE (ours, flagged for Will): we do NOT additionally exclude Working-tier
// nodes. That exclusion exists in `fb0bb55` to stop the boot counter leaking
// through the /api/sync PULL; it is about sync backflow, not durability.
// Applying it here would exclude mem_store which writes tier "Working" and
// mem_store is the single most important durable write path in the soul. Boot
// seeding reads the canonical file wholesale, so a pushed Working-tier node
// does survive restart. This is the one classification call this file makes
// that Will has not ruled on.
//
// WHAT THIS BOUNDARY CANNOT EXPRESS (by construction, not by omission)
// - engram_strengthen (salience/activation drift): load-merge SKIPS ids that
// already exist, so it cannot update an existing node. There is no owner-side
// update/upsert route. Not pushable through any current route; left as a
// follow-up that needs a change in the engram repo.
// - engram_forget (hard delete): load-merge is additive and has no delete verb.
// Propagating deletes would mean DELETE /api/nodes/<id>, a HARD delete at the
// owner which scripts/verify-soul-contract.sh section B explicitly fails the
// build for ("to delete is to supersede/tombstone, never hard-remove"). Local
// deletes therefore stay local; the TOMBSTONE NODE and its "tombstones" edge
// (mem_tombstone) are pushed, and that is the sanctioned representation of a
// deletion in this graph.
// Configuration
// wt_engram_url same resolution order as ise_post: env, then the state key
// stashed at boot. NO hardcoded localhost fallback: unlike telemetry, inventing
// a destination for durable data would risk pushing a user's memories at whatever
// happens to be listening on 8742. Empty means "no HTTP owner" -> file mode.
fn wt_engram_url() -> String {
let env_url: String = env("ENGRAM_URL")
if !str_eq(env_url, "") { return env_url }
return state_get("soul_engram_url")
}
fn wt_api_key() -> String {
let env_key: String = env("ENGRAM_API_KEY")
if !str_eq(env_key, "") { return env_key }
return state_get("soul_engram_api_key")
}
// wt_enabled true only in HTTP-engram mode. In file mode the soul IS the
// persistence owner and every path below is a no-op, so this whole feature is
// inert for genesis/local deployments. That is also what makes it reversible.
fn wt_enabled() -> Bool {
return !str_eq(wt_engram_url(), "")
}
// wt_spool_dir where staged deltas live. MUST be readable by the engram
// process: /api/load-merge takes a PATH and the owner opens it itself. Both
// processes are same-host by construction (dev-stack LaunchAgents; the GKE
// image starts engram and soul in one container per entrypoint.sh).
fn wt_spool_dir() -> String {
let raw: String = env("SOUL_OUTBOX_DIR")
let dir: String = if str_eq(raw, "") { env("HOME") + "/.neuron/soul-outbox" } else { raw }
fs_mkdir(dir)
return dir
}
// Helpers
// wt_esc minimal JSON string escape. Deliberately local rather than reusing
// chat.el's json_safe: persist.el is imported BY memory.el, which is imported by
// chat.el, so depending on chat.el here would be an import cycle.
fn wt_esc(s: String) -> String {
let s1: String = str_replace(s, "\\", "\\\\")
let s2: String = str_replace(s1, "\"", "\\\"")
let s3: String = str_replace(s2, "\n", "\\n")
let s4: String = str_replace(s3, "\r", "\\r")
let s5: String = str_replace(s4, "\t", "\\t")
return s5
}
// wt_durable_class Will's telemetry carve-out, by node_type. See header.
fn wt_durable_class(node_type: String) -> Bool {
if str_eq(node_type, "InternalStateEvent") { return false }
return true
}
// wt_inner strip the surrounding brackets off a JSON array so several arrays
// can be concatenated into one. Returns "" for "[]" / "" / anything too short.
fn wt_inner(arr: String) -> String {
let n: Int = str_len(arr)
if n < 3 { return "" }
if !str_starts_with(arr, "[") { return "" }
return str_slice(arr, 1, n - 1)
}
// wt_read fs_read, plus a MANDATORY reset of the runtime's binary-length hint.
//
// THIS IS NOT OPTIONAL AND MUST NOT BE "SIMPLIFIED" BACK TO A BARE fs_read.
// The pinned runtime (vendor/el-runtime/v1.0.0-20260501) keeps a thread-local
// `_tl_fs_read_len` that fs_read SETS to the file's byte count (so binary files
// can be served with a correct Content-Length) and that http_send_response
// CONSUMES as the Content-Length of the next reply. Nothing else clears it
// except json_get_raw. So any fs_read during request handling that is not
// followed by a json_get_raw makes the NEXT HTTP response advertise the FILE's
// length instead of the body's and the runtime then sends that many bytes,
// appending whatever adjacent heap memory follows the reply.
//
// Caught here, measured: a /api/neuron/memory reply that should be 86 bytes went
// out as 497, with 411 bytes of this module's own spool paths and log strings
// trailing the JSON. The drain reads spool files mid-request, so this boundary
// is exactly where the landmine gets stepped on.
//
// Upstream el fixed the class in `43636ae` ("pair fs_read length hint with its
// buffer"); that runtime is NOT the one vendored here, and re-pinning the
// runtime is deliberately out of scope for this change. Clearing the hint at
// our own boundary fixes our exposure without touching the pinned C.
// json_get_raw is used as the reset because it is the only builtin in this
// runtime that zeroes the hint, and it does so before any early return.
fn wt_clear_binlen() -> Void {
let discard: String = json_get_raw("{}", "_wt_reset")
}
fn wt_read(path: String) -> String {
let data: String = fs_read(path)
wt_clear_binlen()
return data
}
// wt_sweep best-effort removal of the zero-byte husks left by truncation.
// The runtime exposes no unlink builtin, so a drained delta is emptied rather
// than deleted; this reclaims the directory entries.
//
// `-empty` is the safety property, not an optimisation: the command is
// STRUCTURALLY INCAPABLE of removing a delta that still has content, so it can
// never destroy a pending write even if it runs concurrently with a stage.
// Only the directory path is interpolated (never a filename), and it is quoted.
// The exit code is ignored an un-swept husk costs one directory entry.
fn wt_sweep(dir: String) -> Void {
if str_eq(dir, "") { return }
if str_contains(dir, "'") { return }
exec_command("find '" + dir + "' -maxdepth 1 -name 'wt*.json' -empty -delete 2>/dev/null")
}
// Staging
// wt_stage write ONE delta file. uuid_v4 in the name makes concurrent stagers
// collision-free without any lock. Returns true if the delta is on disk.
fn wt_stage(nodes_json: String, edges_json: String) -> Bool {
let dir: String = wt_spool_dir()
if str_eq(dir, "") { return false }
let payload: String = "{\"nodes\":" + nodes_json + ",\"edges\":" + edges_json + "}"
let path: String = dir + "/wt-" + uuid_v4() + ".json"
fs_write(path, payload)
// Read-back-verify the stage itself. A stage that did not land is a write we
// would otherwise believe was queued exactly the hallucinated-save class.
if str_eq(wt_read(path), "") {
println("[persist] wt_stage: FAILED to write spool file " + path + " — delta not queued")
return false
}
return true
}
// The write boundary
// wt_node create a node locally AND queue it for the persistence owner.
// Same signature and same return contract as engram_node_full ("" on failure),
// so converting a call site is a rename and nothing else.
fn wt_node(content: String, node_type: String, label: String,
salience: Float, importance: Float, confidence: Float,
tier: String, tags: String) -> String {
let id: String = engram_node_full(content, node_type, label,
salience, importance, confidence,
tier, tags)
if str_eq(id, "") { return "" }
// engram_get_node_json emits the SAME record shape engram_save writes (minus
// the embedding vector, which the owner backfills lazily), so the read-back
// doubles as the delta payload no second serialization to drift.
let rec: String = engram_get_node_json(id)
if str_eq(rec, "") || str_eq(rec, "{}") {
println("[persist] wt_node: local write did not read back, id=" + id + " label=" + label)
return ""
}
if wt_enabled() && wt_durable_class(node_type) {
wt_stage("[" + rec + "]", "[]")
}
return id
}
// wt_edge create an edge locally AND queue it. Mirrors engram_connect.
//
// The edge id is freshly generated rather than read back: the runtime exposes no
// "id of the edge I just created" accessor, and the owner dedups edges by
// (from_id,to_id,relation), never by id so the id is not load-bearing. The
// consequence, stated plainly: the soul's copy and the owner's copy of the same
// edge carry different edge ids. Nothing in either codebase looks an edge up by
// id (neighbors traversal scans from_id/to_id), so this is cosmetic.
fn wt_edge(from_id: String, to_id: String, weight: Float, relation: String) -> Void {
engram_connect(from_id, to_id, weight, relation)
if !wt_enabled() { return }
if str_eq(from_id, "") || str_eq(to_id, "") { return }
let ts: Int = time_now()
let rec: String = "{\"id\":\"" + uuid_v4() + "\""
+ ",\"from_id\":\"" + wt_esc(from_id) + "\""
+ ",\"to_id\":\"" + wt_esc(to_id) + "\""
+ ",\"relation\":\"" + wt_esc(relation) + "\""
+ ",\"metadata\":\"{}\""
+ ",\"weight\":" + float_to_str(weight)
+ ",\"confidence\":1"
+ ",\"created_at\":" + int_to_str(ts)
+ ",\"updated_at\":" + int_to_str(ts)
+ ",\"last_fired\":0,\"inhibitory\":0,\"layer_id\":1}"
wt_stage("[]", "[" + rec + "]")
}
// The drain
// wt_drain coalesce every staged delta into ONE load-merge against the owner.
//
// Returns: nodes_added on success (>= 0), 0 when there was nothing to do, and
// -1 when the push FAILED. -1 is load-bearing: on failure the spool files are
// left untouched, so nothing is lost and the next drain retries them. A caller
// must never read a non-negative return as "my particular node is durable"
// use wt_durable(id) for that.
//
// Concurrency: several threads may drain at once. Each builds its own batch file
// (uuid-named), and overlapping batches are harmless because load-merge dedups.
// Files are truncated ONLY after a confirmed ok:true, so a lost race costs a
// redundant push, never a dropped write.
fn wt_drain() -> Int {
if !wt_enabled() { return 0 }
let dir: String = wt_spool_dir()
if str_eq(dir, "") { return 0 }
// el_list_len/el_list_get, NOT json_stringify(fs_list(...)): fs_list builds
// a native list via el_list_append, and json_stringify does not serialize
// that type it renders the raw pointer value. (Verified in isolation; the
// same latent defect is live in studio.el's /api/tools/file/list route,
// which returns e.g. {"entries":4386409744}. Noted, not fixed here.)
let listing = fs_list(dir)
let count: Int = el_list_len(listing)
if count == 0 { return 0 }
let nodes_acc: String = ""
let edges_acc: String = ""
let drained: String = ""
let found: Int = 0
let i: Int = 0
// No `continue` / `break`: elc lists them as keywords but not one line of
// the shipped soul uses either, so they are unexercised on this build path.
// Guard conditions are expressed as nested ifs instead, and every rebind is
// at the loop-body top level where `let x = ...` is assignment (the idiom
// memory.el's boot-counter loop relies on) never inside a nested block,
// where it would shadow instead.
while i < count {
let name: String = el_list_get(listing, i)
// A delta is only usable when it ends with the closing "]}" that
// wt_stage writes last. fs_write is not atomic, so a file being written
// right now can be observed half-formed; requiring the terminator means
// it is picked up whole on the next drain instead of merged as garbage.
// An empty read means "already drained and truncated" not an error.
let p: String = if str_starts_with(name, "wt-") { dir + "/" + name } else { "" }
let raw: String = if str_eq(p, "") { "" } else { wt_read(p) }
let usable: Bool = !str_eq(raw, "") && str_ends_with(raw, "]}")
let nj: String = if usable { wt_inner(json_get_raw(raw, "nodes")) } else { "" }
let ej: String = if usable { wt_inner(json_get_raw(raw, "edges")) } else { "" }
let nodes_acc = if str_eq(nj, "") { nodes_acc } else if str_eq(nodes_acc, "") { nj } else { nodes_acc + "," + nj }
let edges_acc = if str_eq(ej, "") { edges_acc } else if str_eq(edges_acc, "") { ej } else { edges_acc + "," + ej }
let drained = if !usable { drained } else if str_eq(drained, "") { p } else { drained + "\n" + p }
let found = if usable { found + 1 } else { found }
let i = i + 1
}
if found == 0 { return 0 }
let combined: String = "{\"nodes\":[" + nodes_acc + "],\"edges\":[" + edges_acc + "]}"
let batch: String = dir + "/wtb-" + uuid_v4() + ".json"
fs_write(batch, combined)
if str_eq(wt_read(batch), "") {
println("[persist] wt_drain: could not write batch file " + batch + "" + int_to_str(found) + " deltas stay queued")
return -1
}
let url: String = wt_engram_url()
let key: String = wt_api_key()
let body: String = "{\"path\":\"" + wt_esc(batch) + "\",\"_auth\":\"" + wt_esc(key) + "\"}"
let resp: String = http_post_json(url + "/api/load-merge", body)
// The batch file is pure scratch the retry is rebuilt from the SPOOL, not
// from it. Truncate it unconditionally, before branching on the outcome, so
// a persistently unreachable owner cannot accumulate one husk per attempt.
fs_write(batch, "")
// Distinguish the two failures rather than collapsing them: "cannot reach
// the owner" and "the owner refused this delta" need different human
// responses, and a log line that says the wrong one costs a debugging hour.
// curl surfaces transport errors as a JSON body, so an empty response is not
// the only unreachable signal.
// (str_contains rather than a strict parse on purpose the engram's HTTP
// responses have been observed carrying trailing bytes past the JSON.)
let unreachable: Bool = str_eq(resp, "")
|| str_contains(resp, "Couldn't connect")
|| str_contains(resp, "Failed to connect")
|| str_contains(resp, "Could not resolve")
|| str_contains(resp, "timed out")
if unreachable {
wt_sweep(dir)
println("[persist] wt_drain: owner UNREACHABLE at " + url + "" + int_to_str(found)
+ " deltas stay queued in " + dir + " (will retry): " + resp)
return -1
}
if !str_contains(resp, "\"ok\":true") {
wt_sweep(dir)
println("[persist] wt_drain: owner REJECTED the delta — " + int_to_str(found)
+ " stay queued in " + dir + ": " + resp)
return -1
}
let added: Int = json_get_int(resp, "nodes_added")
let added_e: Int = json_get_int(resp, "edges_added")
// Confirmed. Truncate the drained spool files so they are not re-pushed.
// Truncation (not deletion) because the runtime exposes no unlink builtin;
// an emptied file is inert to the loop above. The zero-byte husks are then
// swept below.
let paths = str_split(drained, "\n")
let pn: Int = el_list_len(paths)
let k: Int = 0
while k < pn {
let one: String = el_list_get(paths, k)
if !str_eq(one, "") { fs_write(one, "") }
let k = k + 1
}
wt_sweep(dir)
println("[persist] wt_drain: pushed " + int_to_str(found) + " deltas -> owner added "
+ int_to_str(added) + " nodes, " + int_to_str(added_e) + " edges")
return added
}
// wt_durable is this id present AT THE OWNER? The only honest answer to
// "did my write persist" in HTTP mode.
//
// In file mode the soul IS the owner, so the local read-back is the owner-side
// read-back and this collapses to the pre-existing check.
//
// nodes_added from wt_drain is NOT a substitute: a concurrent drain may have
// already pushed this node, making our own added count 0 while the node is
// perfectly durable. Presence at the owner is the fact; counts are telemetry.
fn wt_durable(id: String) -> Bool {
if str_eq(id, "") { return false }
if !wt_enabled() {
let local: String = engram_get_node_json(id)
return !str_eq(local, "") && !str_eq(local, "null") && !str_eq(local, "{}")
}
let url: String = wt_engram_url()
let resp: String = http_get(url + "/api/nodes/" + id)
if str_eq(resp, "") { return false }
if str_eq(resp, "{}") { return false }
return str_contains(resp, "\"id\"")
}
// wt_commit flush, then assert at the owner. The receipt callers should use.
// Deliberately NOT a fixed success shape: it can and does return false while the
// local write is perfectly fine in RAM, which is the true state of affairs when
// the owner is unreachable.
fn wt_commit(id: String) -> Bool {
if str_eq(id, "") { return false }
if !wt_enabled() {
let local: String = engram_get_node_json(id)
return !str_eq(local, "") && !str_eq(local, "null") && !str_eq(local, "{}")
}
let pushed: Int = wt_drain()
return wt_durable(id)
}
+99 -13
View File
@@ -15,6 +15,40 @@ fn flag_true(body: String, key: String) -> Bool {
return json_get_bool(body, key) || json_get_int(body, key) > 0
}
// ---------------------------------------------------------------------------
// plain_chat_envelope the JSON response contract for a non-agentic ("Tools: Off")
// chat turn. Every /api/chat dispatch that calls layered_cycle goes through here, so
// the three call sites cannot drift apart.
//
// WHY THE ENVELOPE IS BUILT HERE AND NOT INSIDE layered_cycle:
// layered_cycle returns the user-facing text AFTER safety_validate has acted on it.
// Keeping the JSON out of the cycle means the output gate always sees raw model text
// and never an escaped blob there is nothing to unwrap and re-wrap on the crisis
// path, which is exactly the failure mode that made wiring handle_chat unsafe.
// Escaping is the last thing that happens, strictly after the gate.
//
// FIELDS: `reply` and `response` carry the same validated text. Both are required by
// live clients the desktop app reads `reply` first (DaemonClient.parseChatResponse),
// while the CLI tools and the Telegram gateway read `response` (the gateway reads only
// `response`). Emitting one would break the other.
//
// EMPTY MEANS FAILURE, NOT AN EMPTY ANSWER: a hard bell returns the fixed crisis
// message and a soft bell is padded to non-empty by safety_validate, so the only way
// an empty string leaves the cycle is a failed model call. It is reported as an error
// rather than dressed up as a successful blank reply.
// ---------------------------------------------------------------------------
fn plain_chat_envelope(validated: String, model: String) -> String {
if str_eq(validated, "") {
return "{\"error\":\"llm unavailable\",\"reply\":\"\",\"response\":\"\",\"agentic\":false,\"tools_used\":[]}"
}
let safe: String = json_safe(validated)
return "{\"reply\":\"" + safe + "\""
+ ",\"response\":\"" + safe + "\""
+ ",\"model\":\"" + json_safe(model) + "\""
+ ",\"agentic\":false"
+ ",\"tools_used\":[]}"
}
// ---------------------------------------------------------------------------
// Rate limiting simple in-memory per-IP sliding window counter.
//
@@ -152,7 +186,7 @@ fn route_imprint_contextual(body: String) -> String {
return "{\"ok\":false,\"error\":\"empty body\"}"
}
let tags: String = "[\"imprint\",\"contextual\"]"
let id: String = engram_node_full(
let id: String = wt_node(
body,
"Entity",
"imprint:contextual",
@@ -174,7 +208,7 @@ fn route_imprint_user(body: String) -> String {
return "{\"ok\":false,\"error\":\"empty body\"}"
}
let tags: String = "[\"imprint\",\"user\"]"
let id: String = engram_node_full(
let id: String = wt_node(
body,
"Entity",
"imprint:user",
@@ -205,7 +239,7 @@ fn route_synthesize(body: String) -> String {
}
let req: String = "synthesize " + parent_a + " " + parent_b
let tags: String = "[\"soul-inbox-pending\",\"synthesis-request\"]"
engram_node_full(
wt_node(
req,
"Entity",
"synthesis-request",
@@ -243,8 +277,14 @@ fn handle_dharma_recv(body: String) -> String {
} else if agentic_flag {
handle_chat_agentic(chat_body)
} else {
let screened_reply: String = layered_cycle(raw_msg)
screened_reply
// Non-agentic ("Tools: Off"): the full L1L2L3L1 cycle, which now generates
// at L3 instead of echoing. Envelope built outside the cycle see
// plain_chat_envelope.
// FIX B/E1 (2026-08-05): the cycle is told which conversation it is in, and
// whether this generation is conversation at all. Same two arguments at all
// three dispatch sites.
let screened_reply: String = layered_cycle(raw_msg, json_get(chat_body, "session_id"), is_utility_request(chat_body, json_get(chat_body, "session_id")))
plain_chat_envelope(screened_reply, chat_default_model())
}
auto_persist(chat_body, reply)
return reply
@@ -355,7 +395,28 @@ fn handle_connectors(method: String, clean: String, body: String) -> String {
return "{\"ok\":false,\"error\":\"unknown connectors route\"}"
}
// handle_request the soul's HTTP entry point.
//
// NOTE ON THE NAME (neuron#117): the el runtime resolves this handler by NAME
// via dlsym(RTLD_DEFAULT, "handle_request") that is why the Linux build must
// link -rdynamic. So the dispatcher body moved to route_dispatch and the name
// `handle_request` stays put as a thin wrapper. Do not rename it back.
//
// The wrapper exists to give the write-through boundary a guaranteed flush
// point. route_dispatch returns from ~60 places; a per-branch flush would be
// forgotten on the 61st. Draining here means EVERY request that staged a write
// pushes it before the connection closes, whatever route produced it, including
// routes added later that know nothing about persistence.
//
// wt_drain is a no-op (no HTTP, no cost) when nothing is staged and when the
// soul is not in HTTP-engram mode, so this is free on read traffic.
fn handle_request(method: String, path: String, body: String) -> String {
let resp: String = route_dispatch(method, path, body)
let flushed: Int = wt_drain()
return resp
}
fn route_dispatch(method: String, path: String, body: String) -> String {
let clean: String = strip_query(path)
// ACTIVITY STAMP (2026-07-30 self-review): every inbound HTTP request
@@ -392,10 +453,27 @@ fn handle_request(method: String, path: String, body: String) -> String {
return engram_scan_nodes_json(9999, 0)
}
if str_eq(clean, "/api/graph/edges") {
// TODO(reliability #8): engram_save races with awareness loop mem_save().
// Both now use atomic write-to-temp+rename (el_runtime.c). Serialised
// by engram_global_mu. Future: add engram_edges_json() builtin.
let snap_path: String = env("HOME") + "/.neuron/engram/snapshot.json"
// FIXED (neuron#117): this GET used to engram_save() straight over
// ~/.neuron/engram/snapshot.json a READ route, in a process that is
// NOT the persistence owner, overwriting the owner's canonical file
// on every call. It broke soul.el:571-573 ("the soul must NEVER write
// to the local snapshot") and it is the same defect class Will removed
// from the engram itself in el `dc39a61` ("stop read routes clobbering
// canonical snapshot"), where route_scan_edges/route_sync were moved
// to scratch paths for exactly this reason. It was also the race the
// old TODO(reliability #8) admitted to.
//
// Export to a scratch path instead. Same response, no canonical write.
// The soul's own snapshot writes are otherwise already gated behind
// state key "soul_snapshot_path", which is set ONLY in the genesis
// file-mode branch (soul.el: is_genesis && safe_to_seed, and
// safe_to_seed is unconditionally false when ENGRAM_URL is set) so
// after this change the soul writes nothing at all in HTTP mode.
// Future: add an engram_edges_json() builtin and drop the file round
// trip entirely.
let scratch_dir: String = env("TMPDIR")
let scratch_base: String = if str_eq(scratch_dir, "") { "/tmp" } else { scratch_dir }
let snap_path: String = scratch_base + "/soul-edges-export-" + state_get("soul_cgi_id") + ".json"
engram_save(snap_path)
let snap: String = fs_read(snap_path)
let edges_raw: String = json_get_raw(snap, "edges")
@@ -416,8 +494,11 @@ fn handle_request(method: String, path: String, body: String) -> String {
} else if agentic_flag {
handle_chat_agentic(body)
} else {
let screened_reply: String = layered_cycle(eff_msg)
screened_reply
// Non-agentic ("Tools: Off") same cycle and same envelope as POST.
// FIX B/E1: same threading. A GET probe usually carries no session_id, which
// resolves to the anonymous window the documented behaviour for this door.
let screened_reply: String = layered_cycle(eff_msg, json_get(body, "session_id"), is_utility_request(body, json_get(body, "session_id")))
plain_chat_envelope(screened_reply, chat_default_model())
}
auto_persist(body, reply)
return reply
@@ -580,8 +661,13 @@ fn handle_request(method: String, path: String, body: String) -> String {
} else if agentic_flag {
handle_chat_agentic(body)
} else {
let screened_reply: String = layered_cycle(raw_msg)
screened_reply
// Non-agentic ("Tools: Off") the app's DEFAULT mode (AgentMode.NEVER).
// Full L1L2L3L1 cycle with real generation at L3; envelope built
// outside the cycle so safety_validate always sees raw text.
// FIX B/E1: same threading. This is the app's main plain-chat door, so this
// is the site that ends the blank stare in practice.
let screened_reply: String = layered_cycle(raw_msg, json_get(body, "session_id"), is_utility_request(body, json_get(body, "session_id")))
plain_chat_envelope(screened_reply, chat_default_model())
}
auto_persist(body, reply)
return reply
+1 -1
View File
@@ -204,7 +204,7 @@ fn safety_log_bell(level: String, reason: String, input_summary: String) -> Stri
// Emit a fallback println so the bell event leaves at least a log trace even
// when engram is degraded. This does not replace engram persistence -- it is a
// last-resort audit trail when the primary write cannot be confirmed.
let node_id: String = engram_node_full(
let node_id: String = wt_node(
content,
"BellEvent",
"bell:" + level,
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env bash
# run-el-test.sh — compile and run one El test program from tests/.
#
# WHY THIS EXISTS (2026-08-07, issue #129):
# tests/ has held 14 test programs for months with no way to run them. CI does
# not run them. The convention printed in their own headers
# (`elc soul.el && ./soul --test tests/x.el`) refers to a --test flag the El
# runtime does not implement. So the tests were documentation, not gates —
# which is how a P0 safety regression shipped with a test directory present.
#
# THE RECIPE, AND WHY IT IS THIS SHAPE:
# Same discovery as gen-soul-amalgam.sh — `elc --target=c` emits only an extern
# prototype for any module that has a .elh header next to it, and inlines the
# module's bodies when it does not. A test that imports ../chat.el therefore
# compiles to a 18 KB unit full of unresolved externs unless the headers are
# out of the way. So: copy the sources into a scratch tree, delete every .elh
# on the import chain, and compile the test there.
#
# Scratch copy on purpose: the worktree is shared with other terminals and
# deleting headers in place would be a shared-tree mutation with no owner.
#
# EXIT STATUS IS THE GATE: non-zero if the binary fails to build, crashes, or if
# its output contains a FAIL line or reports a non-zero failed count. Do not
# "improve" this into something that only checks the exit code of the test
# binary — these El tests print failures and still exit 0.
#
# usage: scripts/run-el-test.sh tests/test_history_amplification.el
set -euo pipefail
TEST_REL="${1:?usage: run-el-test.sh tests/<test>.el}"
SRC="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
TEST_NAME="$(basename "$TEST_REL" .el)"
ELC="${ELC:-$HOME/neuron-dev-stack/src/el/lang/dist/platform/elc}"
[ -x "$ELC" ] || ELC="$HOME/el-sdk/elc"
[ -x "$ELC" ] || { echo "[run-el-test] FAIL: no elc found (set ELC=)"; exit 1; }
RTC="${RTC:-$SRC/vendor/el-runtime/v1.0.0-20260501/el_runtime.c}"
[ -f "$RTC" ] || RTC="$HOME/el-sdk/el_runtime.c"
[ -f "$RTC" ] || { echo "[run-el-test] FAIL: no el_runtime.c found (set RTC=)"; exit 1; }
RTDIR="$(dirname "$RTC")"
EL_REPO="${EL_REPO:-$HOME/Development/neuron-technologies/el}"
SSL="${SSL_PREFIX:-/opt/homebrew/opt/openssl@3}"
GEN="$(mktemp -d "${TMPDIR:-/tmp}/el-test.XXXXXX")"
trap 'rm -rf "$GEN"' EXIT
mkdir -p "$GEN/neuron/tests" "$GEN/foundation/el/elp/src"
cp "$SRC"/*.el "$GEN/neuron/"
cp "$SRC"/tests/*.el "$GEN/neuron/tests/" 2>/dev/null || true
[ -d "$EL_REPO/elp/src" ] && cp "$EL_REPO"/elp/src/*.el "$GEN/foundation/el/elp/src/" 2>/dev/null || true
# The whole recipe depends on there being no headers to short-circuit inlining.
find "$GEN" -name '*.elh' -delete
echo "[run-el-test] compiling $TEST_REL"
( cd "$GEN/neuron" && "$ELC" --target=c "tests/${TEST_NAME}.el" ) > "$GEN/${TEST_NAME}.c"
BODIES=$(grep -c '^el_val_t .*) {$' "$GEN/${TEST_NAME}.c" || true)
echo "[run-el-test] $(wc -c < "$GEN/${TEST_NAME}.c" | tr -d ' ') bytes, ${BODIES} inlined function bodies"
# A test that imports ../chat.el pulls in the bulk of the engine. A tiny body
# count means an import was read from a header instead of inlined, and the test
# would be exercising extern stubs rather than the real code.
if [ "$BODIES" -lt 100 ]; then
echo "[run-el-test] FAIL: only $BODIES inlined bodies — an import was not inlined"
exit 1
fi
cc -O2 -DHAVE_CURL \
-I"$RTDIR" -I"$SSL/include" -L"$SSL/lib" \
"$GEN/${TEST_NAME}.c" "$RTC" \
-lssl -lcrypto -lcurl -lpthread -lm \
-o "$GEN/${TEST_NAME}" 2> "$GEN/cc.log" || {
echo "[run-el-test] FAIL: compile error"; tail -30 "$GEN/cc.log"; exit 1; }
# arm64 pointer-truncation guard (cc-brain.sh's rule): an implicit declaration of
# a runtime symbol truncates its returned pointer to 32 bits.
if grep -E 'implicit.*(engram_|el_)' "$GEN/cc.log"; then
echo "[run-el-test] FAIL: implicit declarations of runtime symbols"; exit 1; fi
# Throwaway HOME so a test can never read or write the live engram at ~/.neuron.
TEST_HOME="$GEN/home"
mkdir -p "$TEST_HOME"
echo "[run-el-test] running $TEST_NAME"
set +e
HOME="$TEST_HOME" NEURON_HOME="$TEST_HOME/.neuron" "$GEN/${TEST_NAME}" 2>&1 | tee "$GEN/out.txt"
RC=${PIPESTATUS[0]}
set -e
if [ "$RC" -ne 0 ]; then
echo "[run-el-test] FAIL: $TEST_NAME exited $RC (crash or abort)"
exit 1
fi
if grep -q " FAIL:" "$GEN/out.txt"; then
echo "[run-el-test] FAIL: $TEST_NAME reported failing assertions"
exit 1
fi
if grep -qE '[1-9][0-9]* failed' "$GEN/out.txt"; then
echo "[run-el-test] FAIL: $TEST_NAME reported a non-zero failed count"
exit 1
fi
if ! grep -q "PASS:" "$GEN/out.txt"; then
echo "[run-el-test] FAIL: $TEST_NAME produced no assertions at all"
exit 1
fi
echo "[run-el-test] PASS: $TEST_NAME"
+7 -7
View File
@@ -87,7 +87,7 @@ fn session_create(body: String) -> String {
let folder: String = json_get(body, "folder")
let content: String = session_make_content(id, title, ts, ts, folder)
let tags: String = "[\"session\",\"session:meta\",\"Conversation\"]"
let node_id: String = engram_node_full(
let node_id: String = wt_node(
content, "Conversation", "session:meta",
el_from_float(0.7), el_from_float(0.7), el_from_float(0.9),
"Episodic", tags
@@ -358,7 +358,7 @@ fn session_update_patch(session_id: String, body: String) -> String {
let created_int: Int = str_to_int(old_created)
let new_content: String = session_make_content(session_id, eff_title, created_int, ts, eff_folder)
let tags: String = "[\"session\",\"session:meta\",\"Conversation\"]"
let new_node_id: String = engram_node_full(
let new_node_id: String = wt_node(
new_content, "Conversation", "session:meta",
el_from_float(0.7), el_from_float(0.7), el_from_float(0.9),
"Episodic", tags
@@ -456,7 +456,7 @@ fn session_hist_save(session_id: String, hist: String) -> Void {
// TODO(reliability #7): delete-then-insert is not atomic concurrent saves for the
// same session can produce orphan history nodes. State is primary truth; engram fallback.
let tags: String = "[\"session\",\"session-history\",\"Conversation\"]"
let discard: String = engram_node_full(
let discard: String = wt_node(
hist, "Conversation", "session:messages:" + session_id,
el_from_float(0.6), el_from_float(0.6), el_from_float(0.9),
"Episodic", tags
@@ -488,7 +488,7 @@ fn session_hist_save(session_id: String, hist: String) -> Void {
+ " | ts:" + int_to_str(ts_now)
let summary_tags: String = "[\"session-emotional-summary\",\"affective\",\"bell:" + eff_level + "\",\"BellEvent\"]"
let summary_sal: String = if str_eq(eff_level, "hard") { el_from_float(0.95) } else { el_from_float(0.85) }
let sum_discard: String = engram_node_full(
let sum_discard: String = wt_node(
summary_content,
"BellEvent",
"session:emotional-summary",
@@ -529,7 +529,7 @@ fn session_hist_save(session_id: String, hist: String) -> Void {
if !str_eq(ot_id, "") { engram_forget(ot_id) }
let oti = oti + 1
}
let discard_topic: String = engram_node_full(
let discard_topic: String = wt_node(
topic_content, "Conversation", topic_label,
el_from_float(0.7), el_from_float(0.7), el_from_float(0.9),
"Episodic", topic_tags
@@ -582,7 +582,7 @@ fn session_update_meta_timestamp(session_id: String) -> Void {
let created_int: Int = str_to_int(old_created)
let new_content: String = session_make_content(session_id, old_title, created_int, ts, old_folder)
let tags: String = "[\"session\",\"session:meta\",\"Conversation\"]"
let new_id: String = engram_node_full(
let new_id: String = wt_node(
new_content, "Conversation", "session:meta",
el_from_float(0.7), el_from_float(0.7), el_from_float(0.9),
"Episodic", tags
@@ -629,7 +629,7 @@ fn session_auto_title(session_id: String, first_message: String) -> Void {
let created_int: Int = str_to_int(old_created)
let new_content: String = session_make_content(session_id, new_title, created_int, ts, old_folder)
let tags: String = "[\"session\",\"session:meta\",\"Conversation\"]"
let new_id: String = engram_node_full(
let new_id: String = wt_node(
new_content, "Conversation", "session:meta",
el_from_float(0.7), el_from_float(0.7), el_from_float(0.9),
"Episodic", tags
+88 -35
View File
@@ -379,9 +379,23 @@ fn emit_session_start_event() -> Void {
// layered_cycle routes user-facing requests through the 4-layer consciousness stack.
// L0 (core) L1 (safety screen) L2a (continuity + behavioral profiling) L2b (mission alignment) L3 (imprint) L1 (safety validate)
// Internal cognition (heartbeat, proactive, memory ops) bypasses layers use one_cycle directly.
fn layered_cycle(raw_input: String) -> String {
let history: String = state_get("conv_history")
let session_id: String = state_get("current_session_id")
//
// FIX B (2026-08-05) the cycle now knows which conversation it is in.
//
// session_id: the caller's session, threaded from the route. Was previously read from the
// state key "current_session_id", which is read HERE and written NOWHERE in the entire
// source verified across every .el file. So this value was unconditionally "", and every
// downstream consumer of it silently fell back to a process-global bucket: conversation
// history, and the steward's continuity tracking (TODO reliability #4, below, describes the
// cross-session bleed this caused; threading the real id closes it). The plain path's blank
// stare and the agentic path's scoped history were the same defect seen from two sides.
//
// utility: true when the generation is not part of the user's conversation the app's
// title and insight passes. Answered normally, never recorded. See is_utility_request.
fn layered_cycle(raw_input: String, session_id: String, utility: Bool) -> String {
// Safety-screen history amplification now reads the SAME window the turn will be
// recorded into, so a session's own escalation pattern is what gets scored.
let history: String = state_get(conv_hist_key(session_id))
// L1 in: safety screen
let screen_result: String = safety_screen(raw_input, history)
@@ -423,8 +437,10 @@ fn layered_cycle(raw_input: String) -> String {
let cont_action: String = json_get(continuity, "action")
// Store continuity status so imprint can adjust its response register.
// TODO(reliability #4): session_continuity is process-global; scope per session_id
// when available to prevent cross-session bleed under concurrent layered_cycle calls.
// TODO(reliability #4) CLOSED 2026-08-05: this line was already written to scope per
// session it just never received a session id, because the only source was a state key
// nothing wrote. It is now threaded from the route, so named sessions genuinely get their
// own continuity state and only anonymous callers share the global one.
let cont_key: String = if str_eq(session_id, "") { "session_continuity" } else { "session_continuity:" + session_id }
state_set(cont_key, cont_status)
@@ -453,40 +469,24 @@ fn layered_cycle(raw_input: String) -> String {
let lc_aff_cutoff: Int = time_now() - 259200
let lc_bell_nodes: String = engram_search_json("bell:soft bell:hard BellEvent affective", 2)
let lc_has_bell: Bool = !str_eq(lc_bell_nodes, "") && !str_eq(lc_bell_nodes, "[]")
// CRASH FIX 2026-08-05 (BUG-PLAINCHAT-1): the " | ts:" parser used to be inline here.
// Inside this block-expression initializer elc compiled `lbmp + str_len(lbm)` to
// el_str_concat() on two integers, which segfaulted the whole daemon the moment a
// distress turn followed an earlier affective turn i.e. exactly on the crisis path.
// Verified against the unmodified baseline binary AND present in the committed
// dist/soul.c. affective_node_ts() is a top-level function, where the same expression
// compiles to integer addition. Do not inline it back.
let lc_bell_note: String = if lc_has_bell {
let lb0: String = json_array_get(lc_bell_nodes, 0)
let lb_c: String = json_get(lb0, "content")
let lbm: String = " | ts:"
let lbmp: Int = str_index_of(lb_c, lbm)
let lb_ts_raw: String = if lbmp >= 0 {
let lbs: Int = lbmp + str_len(lbm)
let lbr: String = str_slice(lb_c, lbs, str_len(lb_c))
let lbn: Int = str_index_of(lbr, " | ")
if lbn < 0 { lbr } else { str_slice(lbr, 0, lbn) }
} else {
let lbca: String = json_get(lb0, "created_at")
if str_eq(lbca, "") { json_get(lb0, "updated_at") } else { lbca }
}
let lb_ts: Int = if str_eq(lb_ts_raw, "") { 0 } else { str_to_int(lb_ts_raw) }
let lb_ts: Int = affective_node_ts(lb0)
if lb_ts > lc_aff_cutoff { "[AFFECTIVE NOTE: User was in distress in a recent session.]" } else { "" }
} else { "" }
let lc_pos_nodes: String = engram_search_json("PositiveEvent joy:high joy:low affective", 2)
let lc_has_pos: Bool = !str_eq(lc_pos_nodes, "") && !str_eq(lc_pos_nodes, "[]")
// Same crash fix as the bell note above (BUG-PLAINCHAT-1).
let lc_pos_note: String = if lc_has_pos && str_eq(lc_bell_note, "") {
let lp0: String = json_array_get(lc_pos_nodes, 0)
let lp_c: String = json_get(lp0, "content")
let lpm: String = " | ts:"
let lpmp: Int = str_index_of(lp_c, lpm)
let lp_ts_raw: String = if lpmp >= 0 {
let lps: Int = lpmp + str_len(lpm)
let lpr: String = str_slice(lp_c, lps, str_len(lp_c))
let lpn: Int = str_index_of(lpr, " | ")
if lpn < 0 { lpr } else { str_slice(lpr, 0, lpn) }
} else {
let lpca: String = json_get(lp0, "created_at")
if str_eq(lpca, "") { json_get(lp0, "updated_at") } else { lpca }
}
let lp_ts: Int = if str_eq(lp_ts_raw, "") { 0 } else { str_to_int(lp_ts_raw) }
let lp_ts: Int = affective_node_ts(lp0)
if lp_ts > lc_aff_cutoff { "[AFFECTIVE NOTE: User shared positive news in a recent session.]" } else { "" }
} else { "" }
let lc_affective_note: String = if !str_eq(lc_bell_note, "") { lc_bell_note } else { lc_pos_note }
@@ -498,11 +498,47 @@ fn layered_cycle(raw_input: String) -> String {
}
state_set("layered_cycle_safety_system_addendum", augmented_addendum)
// L3: imprint responds
let output: String = imprint_respond(aligned, imprint_id)
// L3: imprint responds applies the active imprint's voice/domain annotation to the
// steward-aligned input. This produces the PROMPT, not the answer.
let prompt: String = imprint_respond(aligned, imprint_id)
// L1 out: validate output before delivery
return safety_validate(output, screen_action)
// L3b: the imprint SPEAKS (added 2026-08-05).
//
// Until now the cycle stopped at the annotation above, so /api/chat with agentic:false
// handed the user's own screened text back as the "reply" every gate ran, but nothing
// ever generated. The generation is placed HERE, inside the cycle, rather than by
// pointing the route at handle_chat(): handle_chat has no enforcing input gate and no
// enforcing output gate, so calling it instead of this cycle would have traded the whole
// safety pipeline for a working reply. Composing keeps both.
//
// Order is deliberate and must not be rearranged: this call sits strictly AFTER the L1
// screen, the safe-mode guard, the hard-bell short-circuit and the L2 stewardship layers,
// and strictly BEFORE the L1 output gate. A hard bell never reaches a model the branch
// above returns first. Tools are not offered on this turn; see layered_generate.
let output: String = layered_generate(prompt, imprint_id, session_id)
// L1 out: validate output before delivery. Still the terminal gate nothing below this
// line can change the string this function returns.
let validated: String = safety_validate(output, screen_action)
// Turn bookkeeping. Records the VALIDATED text, never the raw model output, and is only
// reachable on the non-bell path: both bell branches above return before this point, so
// bell turns still never enter conversation history. Pure state side effect it cannot
// alter what is returned.
//
// FIX A: the receipt is unconditional and always negative on this path, because on this
// path it is structurally true layered_generate offers no tools at all (build_system_prompt
// chat mode + a request body with no "tools" key). Recording "no tools ran" is not padding:
// it is the only thing that distinguishes "nothing ran" from "we forgot to write down what
// ran", and that ambiguity is what made the model confess to a search it had performed.
//
// FIX E1: a utility generation is answered but not recorded. Guarded here rather than at
// the route so every /api/chat dispatch site inherits it from one place.
let receipt: String = tool_receipt("", "")
if !utility {
conv_history_record(session_id, raw_input, validated, receipt)
}
return validated
}
let soul_cgi_id_raw: String = env("SOUL_CGI_ID")
@@ -621,6 +657,23 @@ if is_genesis && safe_to_seed {
}
}
// CRASH RECOVERY (neuron#117). Deltas the previous process staged but could not
// hand to the owner are still on disk the spool is a filesystem queue, not a
// memory buffer, precisely so that a soul that died mid-flight does not take its
// unpushed writes with it. Drain them before serving, so recovered memories are
// durable and recallable from the owner from the first request onward.
//
// Safe on a clean boot: an empty spool means no HTTP call at all. Safe in file
// mode: wt_drain returns immediately when ENGRAM_URL is unset.
let wt_recovered: Int = wt_drain()
if wt_recovered > 0 {
println("[soul] write-through: recovered " + int_to_str(wt_recovered)
+ " nodes from a previous process's spool -> persistence owner")
}
if wt_recovered < 0 {
println("[soul] write-through: spool present but the persistence owner is unreachable — queued, will retry on heartbeat")
}
println("[soul] serving on port " + int_to_str(port))
http_serve_async(port, "handle_request")
println("[soul] awareness loop starting")
+3 -1
View File
@@ -1,6 +1,8 @@
// auto-generated by elc --emit-header - do not edit
extern fn init_soul_edges() -> Void
extern fn ensure_self_canonical_bridge() -> Void
extern fn aff_try_slot(slot_json: String, aff_7d_ts: Int, acc_key: String) -> Void
extern fn load_identity_context() -> Void
extern fn seed_persona_from_env() -> Void
extern fn emit_session_start_event() -> Void
extern fn layered_cycle(raw_input: String) -> String
extern fn layered_cycle(raw_input: String, session_id: String, utility: Bool) -> String
+2 -2
View File
@@ -11,7 +11,7 @@ import "memory.el"
fn steward_log_event(kind: String, detail: String) -> Void {
let content: String = "STEWARD:" + kind + " | " + detail
let tags: String = "[\"stewardship\",\"steward:" + kind + "\"]"
let discard: String = engram_node_full(
let discard: String = wt_node(
content,
"StewardshipEvent",
"steward:" + kind,
@@ -221,7 +221,7 @@ fn steward_fingerprint_session(input: String, session_id: String) -> String {
+ " formality=" + fs_str
+ " time=" + tb_str
let sample_tags: String = "[\"behavior\",\"BehaviorSample\",\"stewardship\"]"
let discard: String = engram_node_full(
let discard: String = wt_node(
sample_content,
"BehaviorSample",
"behavior:" + session_id,
+213
View File
@@ -0,0 +1,213 @@
// test_history_amplification.el
//
// REGRESSION TEST FOR ISSUE #129 (P0, SAFETY).
//
// What this guards: on the agentic path, the crisis score has two halves the
// message you just sent, and the distress that has accumulated across the
// conversation. The second half is the whole reason the escalation logic exists:
// someone whose distress builds over several turns never sends one message that
// trips the bell on its own.
//
// The defect this test was written against (ff421d3, 2026-08-05 fixed
// 2026-08-07): conversation history moved to a per-session key via
// conv_hist_key(session_id), but the agentic path's safety screen was left
// reading the old anonymous "conv_history" bucket. The desktop app always sends
// a session_id, so the screen received "" on every real conversation and the
// escalation half always scored 0. Nothing failed. Nothing logged. The comment
// above the defective line documented this same bug being fixed once before.
//
// THE INVARIANT UNDER TEST, stated so it survives future renames:
// the window the safety screen READS must be the window conv_history_record
// WRITES. Not "must be called conv_history" must AGREE.
//
// This test is deliberately written to fail loudly on the pre-fix source. If it
// ever passes on code where the screen reads a key nothing writes, it is broken.
//
// To run (macOS, from the worktree root):
// scripts/run-el-test.sh tests/test_history_amplification.el
//
import "../chat.el"
import "../safety.el"
import "../sessions.el"
// Program class. Without this an El program compiles as a 'utility', and a
// utility may not call the self-formation primitives (llm_call_system,
// llm_vision) that chat.el's agentic loop references the unit fails to
// compile with a capability violation even though the test never calls them.
// Declaring 'cgi' matches how soul.el declares itself.
//
// The endpoints below are deliberately DEAD: this test must never reach a live
// engram, and nothing it asserts depends on one. Port 9 is discard.
cgi "neuron-test-history-amplification" {
dharma_id: "ntn-test@http://127.0.0.1:9",
principal: "test-harness",
network: "dharma-testnet",
engram: "http://127.0.0.1:9"
}
// Counters
//
// NOTE for anyone copying this harness: the idiom used by the older tests in
// this directory `let pass_count = pass_count + 1` inside an assert function
// does NOT mutate the module-level binding. It declares a new local that dies
// with the call, so those suites all print "0 passed, 0 failed" no matter what
// happened. Counters go through the state store here so the summary is real.
fn bump(counter: String) -> Void {
let cur: String = state_get(counter)
let n: Int = if str_eq(cur, "") { 0 } else { str_to_int(cur) }
state_set(counter, int_to_str(n + 1))
}
fn counter_of(counter: String) -> Int {
let cur: String = state_get(counter)
if str_eq(cur, "") { return 0 }
return str_to_int(cur)
}
fn assert_eq(label: String, got: String, expected: String) -> Void {
if str_eq(got, expected) {
bump("t_pass")
println(" PASS: " + label)
} else {
bump("t_fail")
println(" FAIL: " + label)
println(" got: " + got)
println(" expected: " + expected)
}
}
fn assert_true(label: String, cond: Bool) -> Void {
if cond {
bump("t_pass")
println(" PASS: " + label)
} else {
bump("t_fail")
println(" FAIL: " + label)
}
}
// Fixtures
//
// ESCALATION_MSG scores 50 on its own ("can't go on", safety_score_crisis s9).
// 50 is >= soft (35) and < hard (70): a soft bell by itself, which is the point.
// It is the message that is only a crisis IN CONTEXT.
//
// DISTRESS_TURN carries five history phrases:
// hopeless 15 + worthless 15 + nobody cares 15 + can't take it anymore 20
// + want to disappear 20 = 85; contributed at 1/3 weight = 28.
// 50 + 28 = 78 >= 70 hard_bell. That 28-point swing IS the escalation logic,
// and it is exactly what the defect suppressed.
let TEST_SESSION: String = "sess-test-129-history-amplification"
let ESCALATION_MSG: String = "honestly i can't go on like this"
let DISTRESS_TURN: String = "i feel hopeless and worthless, nobody cares, i can't take it anymore, i want to disappear"
println("")
println("test_history_amplification.el — issue #129 (agentic multi-turn crisis escalation)")
// 1. Baseline: the message alone is a SOFT bell, not a hard one
//
// If this ever returns hard_bell, the test below proves nothing the message
// would trip the bell without any history and the amplification would be
// invisible. This assertion is what keeps the real test honest.
println("")
println("1. baseline — escalation message with NO history is a soft bell")
let baseline: String = safety_screen(ESCALATION_MSG, "")
assert_eq("no history -> soft_bell (not hard)", json_get(baseline, "action"), "soft_bell")
// 2. Producer sanity: history lands in the session's own window
println("")
println("2. producer — conv_history_record writes the session's window")
conv_history_record(TEST_SESSION, DISTRESS_TURN, "i hear you, that sounds heavy", "")
let written: String = state_get(conv_hist_key(TEST_SESSION))
assert_true("session window is non-empty after record", !str_eq(written, ""))
assert_true("session window contains the distress turn", str_contains(written, "hopeless"))
// 3. THE REGRESSION: the agentic screen must SEE that window
//
// Pre-fix this returns soft_bell, because agentic_safety_screen read the
// anonymous bucket and got "". Post-fix it returns hard_bell.
println("")
println("3. REGRESSION #129 — agentic screen reads the session's own window")
let screened: String = agentic_safety_screen(TEST_SESSION, ESCALATION_MSG)
assert_eq(
"distress history escalates the agentic screen to hard_bell",
json_get(screened, "action"),
"hard_bell"
)
// 4. The invariant, stated directly
//
// Independent of thresholds and phrase lists: whatever the screen reads for a
// session must equal what the recorder wrote for that session. This is the
// assertion that survives a future rename of either side.
println("")
println("4. invariant — read window == written window")
let read_back: String = state_get(conv_hist_key(TEST_SESSION))
assert_true("screen input is the recorded window, not empty", !str_eq(read_back, ""))
assert_eq("read window is byte-identical to written window", read_back, written)
// 5. No false positive: a calm session does not escalate
//
// A test that only ever asserts "hard_bell" would pass on code that hard-bells
// every message. This is the other leg, and it runs BEFORE the anonymous case
// below on purpose: that case writes the shared bucket, and under the defect a
// calm session would then inherit it.
println("")
println("5. specificity — a calm history does NOT escalate")
let CALM_SESSION: String = "sess-test-129-calm"
state_set("conv_history", "")
conv_history_record(CALM_SESSION, "what is the weather like today", "clear and mild", "")
let calm: String = agentic_safety_screen(CALM_SESSION, ESCALATION_MSG)
assert_eq("calm history stays at soft_bell", json_get(calm, "action"), "soft_bell")
// 6. Cross-session leakage
//
// The same defect had a second face: because the screen read one shared bucket,
// a calm session could be scored against a DIFFERENT session's distress. That is
// wrong in both directions it fabricates a crisis for the calm user and it
// leaks the distressed user's content into another session's scoring.
println("")
println("6. isolation — one session's distress must not score another session")
state_set("conv_history", "")
let OTHER_SESSION: String = "sess-test-129-other"
conv_history_record(OTHER_SESSION, DISTRESS_TURN, "i hear you", "")
let isolated: String = agentic_safety_screen(CALM_SESSION, ESCALATION_MSG)
assert_eq(
"a distressed OTHER session does not escalate the calm session",
json_get(isolated, "action"),
"soft_bell"
)
// 7. Anonymous sessions still work
//
// conv_hist_key("") deliberately falls back to the shared "conv_history" bucket.
// The fix must not break the no-session_id path older callers rely on. Runs last
// because it writes that shared bucket.
println("")
println("7. anonymous path — empty session_id still screens against the shared window")
state_set("conv_history", "[{\"role\":\"user\",\"content\":\"" + DISTRESS_TURN + "\"}]")
let anon: String = agentic_safety_screen("", ESCALATION_MSG)
assert_eq("anonymous session escalates too", json_get(anon, "action"), "hard_bell")
// Summary
println("")
println("history amplification tests: " + int_to_str(counter_of("t_pass")) + " passed, " + int_to_str(counter_of("t_fail")) + " failed")
+97 -11
View File
@@ -41,6 +41,7 @@
#include <fcntl.h>
#include <dirent.h>
#include <errno.h>
#include <signal.h> /* SIGPIPE disposition — see el_runtime_ignore_sigpipe */
#include <pthread.h>
#include <curl/curl.h>
@@ -1238,16 +1239,77 @@ static const char* http_reason_phrase(int status) {
}
}
/* Best-effort send with retry on partial writes. */
/* ── A departing client MUST NOT be able to kill the daemon ──────────────────
* (2026-08-06, round 9.1 / ADR 0006 item 4.)
*
* Measured field failure: a client cancelled its request at 25 s; the handler
* finished its work at 116.9 s and wrote the reply into the departed client's
* socket. The second send() on a reset connection raised SIGPIPE, whose DEFAULT
* disposition terminates the process `exited due to SIGPIPE ... ran for
* 361177ms`. launchd respawned 4 ms later, so EVERY other in-flight request on
* that daemon lost its work, silently.
*
* Two independent guards, because one of them can be undone from outside this
* file (an embedder may reset signal dispositions) and the other cannot:
* 1. process-wide SIGPIPE -> SIG_IGN, installed at runtime init;
* 2. per-send suppression at the syscall (MSG_NOSIGNAL where the platform has
* it, SO_NOSIGPIPE on the accepted socket on macOS/BSD).
* With either in force, send() reports the peer's departure as EPIPE and the
* caller decides which is the point: this is an ordinary I/O outcome, not a
* fatal condition.
*
* It deliberately does NOT swallow the error. http_send_response() below
* classifies the errno and logs: "client left" for a departure, and a real
* "send failed: <strerror>" for anything else, so a genuine write fault is
* still visible in the log (spec round-9.1 §5.3). */
#ifndef MSG_NOSIGNAL
#define MSG_NOSIGNAL 0
#endif
void el_runtime_ignore_sigpipe(void) {
static int done = 0;
if (done) return;
done = 1;
struct sigaction sa;
memset(&sa, 0, sizeof(sa));
sa.sa_handler = SIG_IGN;
sigemptyset(&sa.sa_mask);
sigaction(SIGPIPE, &sa, NULL);
}
/* Suppress SIGPIPE for one accepted connection (macOS/BSD have no
* MSG_NOSIGNAL; they have the socket option instead). Best effort. */
static void http_socket_nosigpipe(int fd) {
#ifdef SO_NOSIGPIPE
int on = 1;
setsockopt(fd, SOL_SOCKET, SO_NOSIGPIPE, &on, sizeof(on));
#else
(void)fd;
#endif
}
/* Best-effort send with retry on partial writes.
* Returns 0 on success, -1 on failure with errno preserved for the caller. */
static int http_send_all(int fd, const char* p, size_t left) {
while (left > 0) {
ssize_t w = send(fd, p, left, 0);
if (w <= 0) return -1;
ssize_t w = send(fd, p, left, MSG_NOSIGNAL);
if (w < 0) {
if (errno == EINTR) continue; /* not an error — retry */
return -1; /* errno stays set for caller */
}
if (w == 0) { errno = EPIPE; return -1; }
p += w; left -= (size_t)w;
}
return 0;
}
/* Did this write fail because the client is gone, or because something is
* actually wrong with the socket? Only the first is routine. */
static int http_write_err_is_client_gone(int e) {
return e == EPIPE || e == ECONNRESET || e == ENOTCONN || e == ESHUTDOWN;
}
/* Discriminator that http_response() embeds at the start of its envelope.
* A handler returning a string starting with this exact prefix is treated
* as a structured response; anything else is treated as a raw body. */
@@ -1468,14 +1530,30 @@ static void http_send_response(int fd, const char* body) {
free(env_body); free(hdrs.buf); return;
}
if (http_send_all(fd, status_line, (size_t)sl) == 0
&& http_send_all(fd, hdrs.buf, hdrs.len) == 0
&& http_send_all(fd, tail, (size_t)tl) == 0
&& (head_only
/* HEAD requests echo headers + Content-Length but no body. */
? 1
: http_send_all(fd, eff_body, blen) == 0)) {
/* sent successfully */
/* The reply is written in four pieces; any of them can find the client
* already gone. errno is captured at the first failure, before any later
* library call can clobber it, and classified once below. */
errno = 0;
int send_err = 0;
if (http_send_all(fd, status_line, (size_t)sl) != 0) send_err = errno;
else if (http_send_all(fd, hdrs.buf, hdrs.len) != 0) send_err = errno;
else if (http_send_all(fd, tail, (size_t)tl) != 0) send_err = errno;
else if (!head_only /* HEAD echoes headers + Content-Length, no body. */
&& http_send_all(fd, eff_body, blen) != 0) send_err = errno;
if (send_err) {
if (http_write_err_is_client_gone(send_err)) {
/* ROUTINE. The user closed the window, quit the app, or cancelled.
* The work is done and the daemon keeps serving everyone else. */
fprintf(stderr, "[http] client left before the reply was written "
"(%zu-byte body, %s) - request completed, reply discarded\n",
blen, strerror(send_err));
} else {
/* NOT routine — a real write fault. Never let the client-gone case
* above hide this one. */
fprintf(stderr, "[http] send failed: %s (%zu-byte body)\n",
strerror(send_err), blen);
}
}
if (env_parsed_root) el_release(env_parsed_root);
@@ -1491,6 +1569,7 @@ static void* http_worker(void* arg) {
HttpWorkerArg* a = (HttpWorkerArg*)arg;
int fd = a->fd;
free(a);
http_socket_nosigpipe(fd);
char *method = NULL, *path = NULL, *body = NULL;
if (http_read_request(fd, &method, &path, &body, NULL) == 0) {
http_handler_fn h = http_lookup_active();
@@ -1531,6 +1610,7 @@ static void* http_worker(void* arg) {
}
void http_serve(el_val_t port, el_val_t handler) {
el_runtime_ignore_sigpipe(); /* serving implies clients that leave */
/* If `handler` looks like a string name, register it as the active handler. */
const char* hname = EL_CSTR(handler);
if (hname && looks_like_string(handler)) {
@@ -1634,6 +1714,7 @@ static void* _http_serve_async_loop(void* raw) {
}
void http_serve_async(el_val_t port, el_val_t handler) {
el_runtime_ignore_sigpipe(); /* serving implies clients that leave */
const char* hname = EL_CSTR(handler);
if (hname && looks_like_string(handler)) {
http_set_handler(handler);
@@ -1821,6 +1902,7 @@ static void* http_worker_v2(void* arg) {
HttpWorkerArg* a = (HttpWorkerArg*)arg;
int fd = a->fd;
free(a);
http_socket_nosigpipe(fd);
char *method = NULL, *path = NULL, *body = NULL, *hdr_block = NULL;
if (http_read_request(fd, &method, &path, &body, &hdr_block) == 0) {
http_handler4_fn h = http_lookup_active_v2();
@@ -1858,6 +1940,7 @@ static void* http_worker_v2(void* arg) {
}
void http_serve_v2(el_val_t port, el_val_t handler) {
el_runtime_ignore_sigpipe(); /* serving implies clients that leave */
const char* hname = EL_CSTR(handler);
if (hname && looks_like_string(handler)) {
http_set_handler_v2(handler);
@@ -5511,6 +5594,9 @@ el_val_t getpid_now(void) {
static el_val_t _el_args_list = 0;
void el_runtime_init_args(int argc, char** argv) {
/* First line of every generated main(): a client that leaves must never be
* able to signal this process to death. See el_runtime_ignore_sigpipe. */
el_runtime_ignore_sigpipe();
_el_args_list = el_list_empty();
for (int i = 1; i < argc; i++) {
_el_args_list = el_list_append(_el_args_list, EL_STR(argv[i]));