Compare commits

...

8 Commits

Author SHA1 Message Date
Tim Lingo 027a573d89 feat(gate): make a state_get with no producer a build error, not a silence
Retires the defect class behind #129. The engine's state store returns "" for a
key nothing writes — no error, no warning, no log. That is how the agentic
path's crisis-escalation input scored 0 on every real conversation for two days
after ff421d3 moved conversation history behind conv_hist_key(session_id) and
left one consumer reading the old "conv_history" bucket by hand.

scripts/verify-state-keys.sh is the gate; scripts/state-key-audit.py is the El
reader behind it. Two checks:

  DEAD-READ    a state_get whose key resolves to something no state_set in the
               tree produces.
  HAND-ROLLED  a literal that belongs to a namespace a helper owns, accessed
               without the helper. This is #129's actual shape, and DEAD-READ
               alone does NOT catch it: the dead handle_chat() still writes
               "conv_history" through conv_hist_key(""). Stating that plainly
               because a gate that only appears to work is worse than none.

WHY IT DOES NOT CRY WOLF. Keys are usually computed, so a literal-matching
script would flood and be switched off in a day. The resolver handles
concatenation (matched on the static prefix), helper functions (resolved to
their possible returns, with guard conditions folded so conv_hist_key("") does
not falsely claim to produce the session_hist_ namespace), keys built into a
local, and keys arriving as a parameter (resolved through the call sites).
278 of 278 sites on this tree resolve: UNRESOLVED 0, FINDINGS 0. Unresolvable
keys would be listed and would NOT fail the build.

TWO-LEG PROOF, one variable — agentic_safety_screen's single line:
  pre-fix  scripts/verify-state-keys.sh --root <scratch>
           chat.el:2536 state_get("conv_history")
             conv_hist_key() owns this key namespace (EXACT 'conv_history')
           FAIL: 1 state-key finding(s)                        exit 1
  as-is    scripts/verify-state-keys.sh
           FINDINGS (0) ... PASS                               exit 0

INDEPENDENT CONFIRMATION: run read-only against
origin/feat/soul-openai-tools-v2, which carries the same defect on its own, the
gate reported chat.el:2937 — the exact line 43d0449's message had named by
hand, with no prior knowledge. Against origin/fix/129-on-openai-tools: PASS.

PRODUCER-MOVED CONTROLS: renaming the sole writer of an EXACT key (soul_model)
orphans 3 readers across 3 files; renaming the sole writer of a PREFIX
namespace (agent_workspace_root_*) orphans 3 readers, including when the
producer moves to a NARROWER namespace — a case an earlier, more permissive
prefix rule let through. That rule is now directional, with the reason written
next to it.

FOUND ON ITS FIRST RUN, unprompted: soul.el's state_set("soul_identity", ...)
was deleted 2026-05-13 in b163fa6 (a commit about awareness/ISE writes) and five
readers in chat.el were left behind — build_system_prompt, the vision handler,
the agentic system prompt and two council handlers have prefixed "" for ~3
months. studio.el:57 emits "principal":"" and never had a producer. Both are
recorded in state-key-baseline.txt with dates and causes so the gate can be
turned on today; they are DEBT, not false positives, and every run prints them.

Baseline signatures carry no line number (an unrelated edit must not un-mute an
accepted finding) but do carry a count, so a GROWTH in a baselined finding still
fails the build.

Engine behaviour unchanged: this commit adds scripts only, no .el is touched.
CI is deliberately NOT wired here — .gitea/workflows/ci.yaml has changes in
flight from someone else, and turning the gate on would immediately red
feat/soul-openai-tools-v2 (correctly). That flip should be deliberate.

Rung reached: RUNS — the gate executes (0.15s), discriminates on four
independent test pairs, and its verdicts are quoted above. Not wired to CI, and
no engine binary was built from this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 10:30:27 -05:00
Tim Lingo 43d0449904 fix(engine): the agentic crisis screen reads the session's own history again
P0 SAFETY. Closes the regression we introduced in ff421d3 (2026-08-05).

ff421d3 correctly moved conversation history to a per-session key via
conv_hist_key(session_id). One consumer did not move with it: the agentic
path's L1 safety screen kept reading the anonymous "conv_history" bucket. The
desktop app always mints a session id (DaemonClient.kt:706), so history was
always written under session_hist_<id> and that read always returned "".

The half of the crisis score that receives history is the escalation half — the
one that exists for distress building across several turns, where no single
message trips the bell on its own. It scored 0 on every real conversation for
two days. Single-message hard bell was never affected.

The bitter part: the comment that line carried documented this exact bug being
fixed once already, under issue #9. The fix was right then. The rename
re-broke it, and the comment went on describing a repair that no longer held.
A comment is not a gate.

The read now goes through conv_hist_key like every other consumer, including
the plain path at soul.el:398 and the thread-anchoring read thirty lines below
it in this same handler. It is one line. The rest of this commit is structure
so it cannot happen quietly again:

  - agentic_safety_screen() owns the two decisions that were inline — which
    window the screen sees, and the screen call. Inline safety inputs are
    untestable safety inputs; that is what let a rename starve this one with
    nothing failing and nothing logging.
  - the comment above the call site now states the invariant (read window ==
    written window) instead of naming a key that can be renamed out from under
    it.

TWO-LEG PROOF, one variable — the single line state_get("conv_history") ->
state_get(conv_hist_key(session_id)):

  before  scripts/run-el-test.sh tests/test_history_amplification.el
          3. REGRESSION #129 ... FAIL  got: soft_bell  expected: hard_bell
          8 passed, 1 failed          runner exit 1
  after   same command, same tree, that one line changed
          9 passed, 0 failed          runner exit 0

Full engine rebuild from these sources is clean: gen-soul-amalgam.sh ->
1,164,103 bytes / 1226 inlined bodies (gate wants >= 1200), cc-brain.sh ->
903,096 bytes, 0 errors. agentic_safety_screen and conv_hist_key both present
in the built binary (nm: T _agentic_safety_screen, T _conv_hist_key).

Rung reached: BUILT + RUNS (discriminating test). NOT yet in a DMG and not yet
verified in the app a human opens — those are the next two rungs and neither is
claimed here.

Known and NOT fixed by this commit:
  - feat/soul-openai-tools-v2 carries the same defect independently at
    chat.el:2937 and needs the same change or a merge.
  - the defect CLASS (a read of a state key no producer writes) is still
    invisible to every gate we have. Issue #129 proposes making it a build
    error; that is the follow-on.

Closes #129

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 09:33:05 -05:00
Tim Lingo b842e82f77 test(engine): a runner for tests/, and a failing regression test for #129
tests/ has held 14 test programs for months with no way to run them. CI does
not run them. The convention printed in their own headers
(`elc soul.el && ./soul --test tests/x.el`) refers to a --test flag the El
runtime does not implement. So the tests were documentation, not gates — which
is how a P0 safety regression shipped with a test directory sitting right
there.

scripts/run-el-test.sh compiles and runs one test program. It reuses the
gen-soul-amalgam.sh discovery: elc emits only an extern prototype for a module
that has a .elh beside it, and inlines the bodies when it does not, so a test
importing ../chat.el must be compiled in a scratch tree with the headers
removed. Scratch copy on purpose — the worktree is shared. It runs the binary
under a throwaway HOME so a test can never reach the live engram.

Exit status is the gate: the El tests print failures and still exit 0, so the
runner greps for FAIL lines and for a zero assertion count as well.

tests/test_history_amplification.el pins the invariant #129 violated: the
window the safety screen READS must be the window conv_history_record WRITES.
Not "must be called conv_history" — must AGREE.

THIS COMMIT IS RED BY DESIGN. On this tree the test fails one assertion:

  3. REGRESSION #129 — agentic screen reads the session's own window
    FAIL: distress history escalates the agentic screen to hard_bell
      got:      soft_bell
      expected: hard_bell
  history amplification tests: 8 passed, 1 failed   (runner exit 1)

The next commit turns it green by changing one line. Two legs, one variable —
that is the whole point of committing the test first.

Two flaws in the older harness that this one does not copy: the idiom
`let pass_count = pass_count + 1` inside an assert function declares a local
that dies with the call, so every existing suite prints "0 passed, 0 failed"
regardless of outcome; and a test program without a `cgi` block compiles as a
'utility', which may not reference the self-formation primitives chat.el's
agentic loop calls — it fails to build on a capability violation it never
triggers at runtime.

Refs #129

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 09:32:40 -05:00
Tim Lingo 98ccbd4704 fix(engine): a client that leaves must not kill the daemon, and a long round must say it started
Round 9.1, spec §3 D + ADR 0006 items 2 and 4. Two small changes, both proven
by measurement, both E2E-verified locally against a rebuilt brain.

D1 — SIGPIPE/EPIPE survival (vendor/el-runtime el_runtime.c).
Root cause, at the layer that owns it: the whole HTTP server lives in the C
runtime; .el has no socket primitive. http_send_all() called send() with flags
0 and nothing anywhere in the runtime set a SIGPIPE disposition, so the default
disposition — terminate the process — applied. When a handler finished after
its client had gone (Tim's VM: reply at 116.9 s, client cancelled at 25.0 s),
the second of the four sends that write one reply raised SIGPIPE and the daemon
died: `exited due to SIGPIPE ... ran for 361177ms`, launchd respawn 4 ms later,
every other in-flight session's work lost, user never told.

Fix: SIGPIPE -> SIG_IGN at runtime init and at each http_serve* entry, plus
per-connection SO_NOSIGPIPE / MSG_NOSIGNAL so the guard survives an embedder
resetting dispositions. http_send_all now retries EINTR and preserves errno;
http_send_response classifies it once — a departure is logged as routine
("client left before the reply was written ... reply discarded") and ANY other
errno is logged as a real "send failed: <strerror>". Spec §5.3: the routine
case must not mask a genuine write fault, and it does not.

Proof (scratch HOME + free port, 3 disconnects mid-reply):
  round-9 shipped brain 4402179554… — DIED, exit 141 (128+13 = SIGPIPE), round 1
  round-9 sources rebuilt with this exact recipe — DIED, exit 141, round 1
  this build — SURVIVED 3/3, /health 200 after, still serving the full graph,
  three honest "client left" lines in the log naming Broken pipe / Connection
  reset by peer.

D2 — the round-start marker (chat.el, agentic_loop).
The ledger only ever appended AFTER a round returned, so a healthy first leg
produced zero progress by construction; since server-side web_search moved
inside the outbound call that leg is 60-120 s of silence, which is how a 25 s
client watchdog came to kill a healthy mission. One entry,
{"i":N,"t":"","tool":"__working__"}, written to the existing
run_progress_<session_id> ledger BEFORE each round's outbound call — the wire
shape ChatView.kt:1148 has handled as a life signal since 2026-07-13 and never
received. No new key, no new route, no new lifecycle: a strict subset of WS3
item 3. WS3's run registry is untouched and stays Will's.

Proof (live Anthropic key, real research mission, scratch HOME + free port):
  round-9 baseline — ledger EMPTY for the whole 59.7 s leg
  this build       — {"i":0,"t":"","tool":"__working__"} visible at 18.6 s of a
                     70.0 s leg; both builds returned correct ~4.9 KB answers

Regression: prompt-matrix gate 32/32 on this build (round-9 baseline also 32/32
under the same recipe, so the score is not a build artifact). Soul contract
gate PASS — 27/27 routes, immutability clean. neuron#111 miscompile guard: 0
sites in the generated amalgam this binary was compiled from.

NOT included, deliberately: the regenerated dist/soul.c. CI compiles that file,
so production stays exposed until it is regenerated — the same open ask as
neuron#111 / ui#209. The regen recipe is now known and recorded; landing it is
Will's call, per BUILD-HYGIENE.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 18:23:12 -05:00
Tim Lingo dba755dcec fix(engine): resume reads the bridged tool id from the blob's own field, not from inside the replayed conversation
ROOT CAUSE (round 9; live-repro'd 5/5 this morning, both faces stub-proven by the
prompt-matrix gate). json_get is a first-substring-match scanner (strstr for
'"key":', el_runtime.c). bridge_save serialized the RAW messages array BEFORE the
tool_use_id scalar, so agentic_resume's json_get(blob, 'tool_use_id') returned the
FIRST '"tool_use_id":' occurrence inside the replayed conversation, not the saved
field. The resume guard then preferred that misread over the client's correct
call_id (its two branches both reduced to saved_use_id), attached the tool_result
to the wrong id, and Anthropic 400'd the resume ('unexpected tool_use_id found in
tool_result blocks'), surfaced as {"error":"llm unavailable"}.

ONE MISREAD, TWO FACES — whichever block owns the first tool_use_id in the array:
  FACE 1 (search-then-bridge, the Key West killer): the first occurrence is the
    first web_search_tool_result's srvtoolu_… id — every agentic turn that ran
    server-side web_search and then bridged on a client tool died on approval,
    deterministically (messages.2.content.0 … srvtoolu_…). The write itself had
    already succeeded; only the resume died.
  FACE 2 (multi-cycle missions): with no search, the first occurrence is ROUND 0's
    tool_result block — so every LATER approve/resume cycle replayed the round-0
    client id (stale-resume-id), killing multi-file missions after ~2 files.
  And the shape that PASSES on round 8 confirms the mechanism: a single-cycle
  bridge with no prior tool round has no 'tool_use_id' substring in its messages
  at all (tool_use blocks carry 'id'), so the scan fell through to the blob's own
  field and resumed correctly.

The server_tool_use ↔ web_search_tool_result pairs themselves replay intact — the
defect was a cross-field misread of the blob, the same first-match-scanner class
as BUG-6 (approve 'content' matched inside tool_input, 2026-07-17) and round 8's
citation-block fix.

THE FIX, the pattern not the spot:
  1. bridge_save writes every json_safe'd scalar BEFORE both raw fields (an escaped
     value cannot contain a bare '"key":' byte pattern, so first-match always lands
     on the blob's own fields), and tools_raw (our fixed schema) before messages_raw
     (arbitrary conversation), so the raw extractions cannot first-match into
     model-controlled bytes either. Field order documented as load-bearing.
  2. agentic_resume now honors the client's echoed call_id when present — the value
     with clean provenance (minted from pend_tool_id, never blob-round-tripped) —
     falling back to the saved id only when the client omits it. Each approve cycle
     therefore binds to ITS OWN round's id (kills FACE 2 even against a blob written
     by a pre-fix binary), and an omitted call_id still resumes on the saved id,
     which the reordered blob now reads correctly.

Pattern sweep: the legacy synthetic blob (sessions.el handle_session_approve) embeds
only json_safe'd fields — no raw hazard, untouched. No other json_get read of any
container that embeds raw conversation JSON before the read field.

PROOF: prompt-matrix gate 24/32 RED on the round-8 brain (fails exactly the two
resume classes, named) -> 32/32 GREEN on this build; live-key Key West tracer
3/3 consecutive full round-trips (bridge -> approve-as-the-app -> real completion,
file on disk), plain-chat and weather-only controls PASS; unpatched round-8 brain
and a same-toolchain unpatched baseline build both still fail the identical
sequence with the identical srvtoolu 400 (the test discriminates, and the only
variable between failing and passing builds is this diff).

Refs neuron#109

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-06 11:34:12 -05:00
Tim Lingo 8f3a478771 fix(engine): excise the receipt, do not truncate at it — a leading receipt was erasing whole answers
CAUGHT BY A/B, AND ONLY BY A/B. The previous commit's receipt_strip assumed the receipt is
always TERMINAL and cut everything from the marker onward. It is not always terminal: once
receipt_rule told the model what [[RECEIPT ...]] means, the model sometimes LED with one and
wrote the answer underneath. Cutting at the marker then deleted the entire answer and the
turn returned {"error":"no response"}.

MEASURED, same prompt (two web searches, cited prose), fresh session each run:
    round-7 brain   4 / 4 answered   (471, 473, 473, 544 chars)
    round-8 brain   2 / 7 answered   (five {"error":"no response"})
This looked exactly like a flaky model. It was not — it was mine. Running the two brains
side by side on the same prompt is the only reason it was found, and it is the reason the
A/B is now part of how this class gets tested.

AFTER THE FIX, same protocol:
    round-7 brain   4 / 4   (473, 473, 473, 544)
    round-8 brain   4 / 4   (657, 657, 657, 673)

THE FIX: remove the [[...]] span and keep BOTH sides, instead of truncating at the marker.
An unterminated marker at position 0 is left completely alone — no rule about receipts is
worth erasing an answer over. Bounded four-pass loop rather than a conditional exit, because
rebinding the counter inside an if-expression is the block-expression shape that miscompiles
integer arithmetic under this elc (BUG-PLAINCHAT-1). Verified in the generated C:
    str_slice(rest, (e + 2), str_len(rest))   <- integer addition, correct
    el_str_concat(head, tail)                 <- string concat, correct
and zero el_str_concat(<ident>, str_len(...)) sites across all 49 modules.

SEAM PROOF (FIX C) rides on the same runs — a real two-search cited answer, inspected byte
by byte, in BOTH failure directions:
  missing separator (the round-7 "to.Good", bytes 77 2e 47): 0 hits. Sentence boundaries
    measure 2e 20 4d — "." SPACE "M".
  over-separation (a cited sentence shattered across paragraphs): 0 hits. The answer is one
    continuous paragraph with its sentences intact, which is the direction a blanket
    separator would have broken.

BUILT: sha256 77115f2733e794c5bc4ad1f55b1acaf658f8f4a91cccd423a2d633d94a726cbc

Refs neuron#109

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 23:25:48 -05:00
Tim Lingo 9ea41eed78 fix(engine): the model was signing its own answers with our receipt — strip it
FOUND BY E2E, NOT BY REASONING. The previous commit's design note asserted the provenance
receipt "never reaches the user: it is appended to the history copy, not the reply." That
was FALSE, and only running the thing showed it. On the first live run against the built
DMG brain, two agentic turns out of two came back with

  "Your favourite colour is chartreuse and your project is called Perihelion.

   [[RECEIPT - recorded by the soul, not written by the model: no tools ran on this turn.]]"

— the receipt in the user-visible reply.

MECHANISM: the receipt is stored inside the assistant turn, and the agentic path replays
history VERBATIM as Anthropic message objects. So the model sees its own previous answers
ending in [[RECEIPT ...]] and does the obvious thing — it imitates the format and signs the
next answer the same way. The plain path did NOT leak, which is the tell: there, history is
rendered into the SYSTEM prompt as labelled lines rather than replayed as assistant turns,
and a model imitates its own turns far more readily than a transcript.

FIX, two layers, because one of them is not a guarantee:
  - receipt_rule() names the marker in both system prompts (plain and agentic): these lines
    are written by the system, read them as evidence, never write one. Reduces occurrence.
  - receipt_strip() truncates any [[RECEIPT ...]] out of model output before it becomes the
    reply — plain path in layered_generate, agentic path on final_text in agentic_loop.
    Deterministic. A guard that depends on the model choosing to obey is exactly the class of
    thing round 8 exists to stop shipping, so the instruction is the optimisation and the
    strip is the guarantee.
Placed ABOVE agentic_loop's empty-check on purpose: a turn whose entire output was an
imitated receipt has produced no answer, and must be reported as no answer.

The receipt stays in HISTORY, which is the whole point and is proven to work: asked "What
source did you use for that?" one turn after a live web_search, this brain answered
"I used Weather Underground (https://www.wunderground.com/weather/is/reykjav%C3%ADk) for the
current temperature in Reykjavik" — a real source, no apology. That is the false confession
dead, and it is dead BECAUSE the model can read the receipt.

BUILT: 887,112 bytes, sha256 54a2eff84d4fa44f8d2db6781dcf225df4b5075a8ec5f1058018bd40cc1af10b
BUG-PLAINCHAT-1 miscompile guard: zero sites.

Refs neuron#109

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 23:10:53 -05:00
Tim Lingo ff421d39f6 fix(engine): history keeps its provenance and its session — the false confession and the blank stare
DESIGN FIT: three of round 7's five defects share ONE root — the conversation-history
layer persists only {role, content}, discarding tool provenance, session scoping, and the
distinction between a real user turn and an internal utility call. Fixes A and B RESTORE
Will's design rather than extend it: his agentic path already scopes history per session,
the plain path never got it, and his own source carries the TODO admitting the resulting
race (chat.el, handle_chat: "process-global key; concurrent /api/chat requests without
session_id race on this read-append-write"). Fix C repairs one join Will wrote that was
correct for a year and one we added last week. E1/E2 are ours.

FIX A — tool provenance in history (kills the FALSE CONFESSION)
  Root cause, EXECUTED-verified: handle_chat_agentic recorded turns via hist_append, which
  emits {"role","content"} only. server_tool_use blocks, web_search_tool_result blocks and
  every citation were discarded, then replayed as text. On the next turn the model saw a
  data-rich answer with zero evidence a search had happened, and its own permanent rule
  ("never describe a search you did not perform") left one conclusion available: that it
  had fabricated the data. It apologised for a search it HAD run — four independent lines
  of evidence confirm the search was real. The defect is not the model's honesty. It is
  that we deleted the evidence and then asked it to account for itself.
  Change: agentic_loop accumulates the source URLs it already walks past (citations and
  web_search_tool_result content) and returns them as "sources"; handle_chat_agentic folds
  tools_used + sources into a receipt line stored WITH the assistant turn. Receipts are
  unconditional — a negative receipt ("no tools ran") is the other half of the guarantee,
  because "no evidence of a tool" and "evidence of no tool" were previously identical in
  the transcript. conv_history_block splits the receipt off before snipping so a long
  answer cannot truncate away the evidence. The user never sees it: it is appended to the
  history copy, not the reply.

FIX B — one history key for both paths (kills the BLANK STARE)
  Root cause, EXECUTED-verified: the agentic path keyed history on session_hist_<id>; the
  plain path was hard-wired to the process-global conv_history and never read session_id.
  One conversation, two buckets. Proven in the guest engram: the scoped node held exactly
  two turns starting at "Try again" while the earlier exchanges sat unscoped.
  Change: conv_hist_key/conv_hist_label are now the single definition, used by BOTH paths;
  session_id is threaded route -> layered_cycle -> layered_generate / conv_history_record.
  The 2-line fallback (plain path reads the agentic key) was REJECTED: it keeps the
  process-global bucket as a live write target, which is the bleed the TODO describes.
  Also found and closed while threading: layered_cycle read session_id from the state key
  "current_session_id", which is read here and WRITTEN NOWHERE in the entire source. It
  was unconditionally "", so TODO(reliability #4) — per-session steward continuity — was
  dead code that could never fire. It fires now.
  LAZY SESSION, decided explicitly: we create the session EAGERLY at the door (app half,
  ui#223) rather than migrating orphaned turns. Migration would copy the CONTENTS of a
  process-global bucket, possibly another conversation's, into a named session — the bleed,
  performed deliberately. Eager creation makes the situation impossible instead. Migration
  is deliberately not implemented and must not be added without solving provenance first.

FIX C — the two text-join seams ("to.Good", byte-verified 0x77 0x2e 0x47)
  Two bare `+` joins, written a year apart, had drifted into two answers to one question:
  within-response block joins (Will's, 2026-05-03, latent until server-side web_search
  began interleaving non-text blocks) and across-round joins (ours, 62af564).
  Change: one named rule, text_join_sep, at both sites. NOT a blanket separator — a cited
  answer splits MID-SENTENCE ("The current temperature is " + "86°F" + ", with "), so a
  blanket separator shatters every sourced sentence. The rule takes the one bit that
  distinguishes the cases: whether a NON-TEXT block intervened. Hoisting it also makes the
  fix verifiable in the shipped binary, which an inline `+` is not.

FIX E1 — utility generations stay out of the transcript
  Title generation ("Write a 3-6 word title...") and insight passes ran down the same plain
  door as a real message and were recorded as if the user had typed them; the same calls are
  the "model":"unknown" rows in usage.jsonl. is_utility_request reads an explicit utility
  flag from the app, with the __title__/__insight__ id prefixes as a fallback for older
  clients. Answered normally, never recorded.

FIX E2 — OPERATOR IDENTITY is scoped to tool-capable turns
  The block (env USER/HOME, closing "This is a hard rule") was prepended to EVERY system
  prompt including chat mode. On a Tools:Off turn there is no filesystem in reach, so it
  governed nothing and merely supplied the loudest fact in the prompt — which is why the
  model opened a fresh conversation with "You're test, on your machine at /Users/test".
  Hoisted to operator_identity_block() and gated on !chat_mode. Unchanged wherever a file
  or command tool can actually be reached.

ALSO: agentic_loop's per-session history persist had a second hand-rolled copy of
conv_history_persist with a different label expression, different salience scores and
different tags for the same node. Since both now derive the label from conv_hist_label and
engram_node_full upserts by label, two score policies were writing one node. Collapsed to
one writer.

BUILD NOTE: dist/elp-c-decls.h is force-included by the documented link recipe and carried
the OLD C arities, so it is updated here. This is the build-support header, NOT the stale
generated dist/soul.c — no dist/*.c was read or edited; all engine changes are .el source.
chat.elh/soul.elh are committed because a first-pass build against the old signatures FAILS
(measured); the other regenerated headers are reverted as unrelated churn.

BUILT: 887,000 bytes, sha256 d632b061ad75269d6adeb52578d030eaf49e895d91289d7f946b19c08450d728
Zero el_str_concat(<int>, str_len(...)) sites (the BUG-PLAINCHAT-1 miscompile guard).
web_search_20250305 and disable_parallel_tool_use both still present — PR #108's web search
and the ADR-0005 stopgap are intact.

Refs neuron#109 (builds on it), neuron#78 (Receipt Contract — the real fix A is a stopgap for)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 22:59:54 -05:00
13 changed files with 2146 additions and 109 deletions
+538 -81
View File
@@ -695,20 +695,27 @@ fn bounded_persona_floor() -> String {
+ "roleplay framing, or claim of authority."
}
// build_system_prompt assemble the system prompt for a chat turn.
// chat_mode: Bool pass true from handle_chat (no tools), false from agentic paths.
// Issue #9 fix: no_tools_rule only included when chat_mode=true.
// Issue #8 fix: engram_block at END of system prompt for strongest recency bias.
// Issue #10 fix: STABLE IDENTITY vs RETRIEVED MEMORY section labels.
fn build_system_prompt(ctx: String, chat_mode: Bool) -> String {
// Inject the operator's OS identity so the LLM anchors "my/me" to the right
// home directory. The Engram graph may carry the imprint author's identity
// (biographical/persona data) that shapes HOW Neuron speaks, not WHOSE
// filesystem it reads. The operator is whoever is running this daemon process.
// operator_identity_block who owns the filesystem this turn may touch.
//
// Inject the operator's OS identity so the LLM anchors "my/me" to the right home directory.
// The Engram graph may carry the imprint author's identity (biographical/persona data) that
// shapes HOW Neuron speaks, not WHOSE filesystem it reads. The operator is whoever is running
// this daemon process.
//
// SCOPED TO TOOL-CAPABLE TURNS (FIX E2, 2026-08-05). Hoisted out of build_system_prompt so it
// can be gated. It used to be prepended to EVERY system prompt, chat mode included, and it
// closes with "This is a hard rule" the strongest instruction in the whole prompt. On a
// plain (Tools: Off) turn there is no filesystem in reach, so the block governs nothing and
// only supplies a very loud, very early fact about the user. Measured 2026-08-05 on a fresh
// guest profile: asked an open question, the model opened with "You're test, on your machine
// at /Users/test" a first impression made of the one thing it had been told hardest, about
// a capability it did not have. It is correct and necessary the moment a file or command tool
// is reachable; that is exactly when it is now included.
fn operator_identity_block() -> String {
let op_home: String = env("HOME")
let op_user: String = env("USER")
let op_display: String = if str_eq(op_user, "") { "the current user" } else { op_user }
let operator_section: String = "OPERATOR IDENTITY\n\n"
return "OPERATOR IDENTITY\n\n"
+ "You are running on " + op_display + "'s machine. Their home directory is " + op_home + ".\n\n"
+ "When they say \"my files\", \"my notes\", \"my downloads\", \"my desktop\", or any possessive "
+ "referring to their filesystem, always resolve those paths under " + op_home + " — never under "
@@ -716,6 +723,16 @@ fn build_system_prompt(ctx: String, chat_mode: Bool) -> String {
+ "The memory graph may include identity context from a different person (the imprint who shaped your personality and values). "
+ "That context governs how you think and speak — it does not tell you whose machine you are on. "
+ "The person speaking to you right now is " + op_display + " at " + op_home + ".\n\n"
}
// build_system_prompt assemble the system prompt for a chat turn.
// chat_mode: Bool pass true from handle_chat (no tools), false from agentic paths.
// Issue #9 fix: no_tools_rule only included when chat_mode=true.
// Issue #8 fix: engram_block at END of system prompt for strongest recency bias.
// Issue #10 fix: STABLE IDENTITY vs RETRIEVED MEMORY section labels.
fn build_system_prompt(ctx: String, chat_mode: Bool) -> String {
// FIX E2 (2026-08-05): tool-capable turns only. See operator_identity_block.
let operator_section: String = if chat_mode { "" } else { operator_identity_block() }
let identity: String = state_get("soul_identity")
let current_date: String = time_format(time_now(), "%A, %B %d, %Y")
@@ -790,7 +807,7 @@ fn build_system_prompt(ctx: String, chat_mode: Bool) -> String {
// in this revision the chat_mode flag had no effect on the prompt. Restored here, in the
// permanent-rules group, immediately after capability_rules (the rule it qualifies).
// Zero effect on agentic paths: they pass chat_mode=false, so no_tools_rule is "".
return identity + operator_section + date_line + voice_rules + security_rules + capability_rules + no_tools_rule + bounded_persona_block + identity_block + affective_boot_block + engram_block + safety_block
return identity + operator_section + date_line + voice_rules + security_rules + capability_rules + receipt_rule() + no_tools_rule + bounded_persona_block + identity_block + affective_boot_block + engram_block + safety_block
}
fn hist_append(hist: String, role: String, content: String) -> String {
@@ -803,6 +820,292 @@ fn hist_append(hist: String, role: String, content: String) -> String {
return "[" + inner + "," + entry + "]"
}
//
// ONE HISTORY KEY FOR BOTH PATHS (FIX B, 2026-08-05)
//
// THE BUG the BLANK STARE. The agentic path keyed conversation history on
// "session_hist_<id>"; the plain path was hard-wired to the process-global "conv_history"
// and never read session_id at all. One conversation, two buckets. Measured on a fresh
// guest engram: a user chatted with Tools OFF, turned Tools ON and said "try again", and
// the scoped node contained exactly two turns starting at "Try again" while every earlier
// exchange sat in the unscoped node. From the user's side the assistant simply forgot the
// conversation it was in the middle of, at the exact moment they asked it to try harder.
//
// THESE TWO FUNCTIONS ARE THE FIX. Both paths now derive their key and their engram label
// from here, so there is exactly one definition of "where does this conversation's history
// live" and it cannot drift again. Not a fallback bolted onto one path one rule, used by
// both. (The rejected 2-line alternative was to have the plain path fall back to reading
// the agentic key: that keeps the global as a live write target, and the global bucket is
// process-global. handle_chat's own TODO(reliability #3) says so concurrent requests
// without a session_id race on its read-append-write, which is how one conversation bleeds
// into another.)
//
// THE ANONYMOUS BUCKET. An empty session_id still maps to "conv_history". That is the
// documented anonymous path (GET /api/chat probes, curl, the CLI) and it must keep working.
// It is now the ONLY writer of that key, which makes the bleed risk explicit and bounded
// instead of ambient.
//
// TURNS THAT PRECEDE THE SESSION decided, not left implicit. The soul session used to be
// created lazily on first AGENTIC use (measured: session:meta was written 37ms AFTER the
// message that needed it), so early plain turns had no scoped key to go to. Two candidate
// answers:
// (1) migrate the unscoped node into the scoped one when the session is created, or
// (2) create the session eagerly, at the door, on the first turn of either path.
// We chose (2), and the app half ships with it (DaemonClient.chatWithHandshake now resolves
// the soul session id for plain sends too, registering on first use exactly as the agentic
// path already did). Reason: (1) repairs the damage after the fact and, worse, it would copy
// the CONTENTS of a process-global bucket which may hold a different conversation into a
// named session. That is the bleed the TODO warns about, performed deliberately. (2) makes
// the situation impossible instead: every turn of a real conversation carries the same scoped
// id from turn one, so nothing is ever written to the anonymous bucket that needs rescuing.
// Migration is therefore deliberately NOT implemented, and must not be added later without
// solving the provenance question first.
fn conv_hist_key(session_id: String) -> String {
if str_eq(session_id, "") {
return "conv_history"
}
return "session_hist_" + session_id
}
fn conv_hist_label(session_id: String) -> String {
if str_eq(session_id, "") {
return "conv:history"
}
return "conv:history:" + session_id
}
// is_utility_request a generation the USER did not ask for (FIX E1, 2026-08-05).
//
// The app makes model calls that are not conversation: title generation
// ("Write a 3-6 word title (Title Case) for this conversation...") and insight/suggestion
// passes. They ran down the same plain /api/chat door as a real message, so they were
// recorded into conversation history as if the user had typed them. Measured: the unscoped
// history node contained the literal title prompt and the model's reply "What Is Neuron" as
// a user/assistant pair, and usage.jsonl carried the same call as the "model":"unknown" row.
// The user then sees the assistant answering a question they never asked, and the model
// reads its own title-writing as part of the dialogue.
//
// Primary signal is an explicit "utility":true on the request the app declares intent
// rather than the engine guessing. The two id prefixes are a compatibility fallback so an
// older client that does not send the flag (round 7's jar, the CLI helpers) still gets the
// right behaviour when it sends its throwaway id raw.
fn is_utility_request(body: String, session_id: String) -> Bool {
if str_eq(json_get(body, "utility"), "true") {
return true
}
if str_starts_with(session_id, "__title__") {
return true
}
if str_starts_with(session_id, "__insight__") {
return true
}
return false
}
//
// TOOL PROVENANCE IN HISTORY (FIX A, 2026-08-05)
//
// THE BUG the FALSE CONFESSION. hist_append above stores {"role","content"} and nothing
// else. server_tool_use blocks, web_search_tool_result blocks and every citation are
// discarded at the moment the turn is recorded, and the next turn replays that text-only
// array. So the model is shown a data-rich answer it apparently produced with no evidence
// any tool ran and its own permanent rule ("never describe a search you did not perform")
// leaves exactly one conclusion available: that it invented the data. Measured 2026-08-05:
// asked where its figures came from, it apologised for fabricating a web search it had in
// fact performed. Four independent lines of evidence showed the search was real. The defect
// is not the model's honesty. It is that we deleted the evidence and then asked it to
// account for itself.
//
// THE SHAPE OF THE FIX a receipt line inside content, not a sibling field. History entries
// are replayed VERBATIM into the Anthropic messages array (see the prior_messages seed in
// handle_chat_agentic), and a message object there may carry role and content only; an extra
// key is not part of that contract. So provenance rides INSIDE the assistant turn's content,
// as a trailing bracketed line. It is appended to the HISTORY copy only the reply returned
// to the client is the loop's own envelope and is untouched, so the user never sees it.
//
// WHAT IT BUYS beyond not-defaming-itself: with the source URLs recorded, "what source did
// you use?" becomes a question the next turn can actually answer from the transcript.
//
// STOPGAP, AND SAID SO. The real answer is Will's Receipt Contract (neuron#78): structured,
// verifiable receipts on the wire that a client can render and a model cannot confuse with
// prose. Until that lands, a line the model can read is the difference between "I searched"
// and "I must have made it up".
// provenance_scan_urls pull "url"/"title" pairs out of a JSON array into a display string.
// Used for both citation arrays (web_search_result_location) and web_search_tool_result
// content arrays (web_search_result); both spell the fields the same way. Deduped by
// substring, capped at 6 entries per array so a broad search cannot flood the window.
fn provenance_scan_urls(arr: String, acc: String) -> String {
if str_eq(arr, "") { return acc }
if str_eq(arr, "null") { return acc }
if !str_starts_with(arr, "[") { return acc }
let total: Int = json_array_len(arr)
let limit: Int = if total > 6 { 6 } else { total }
let out: String = acc
let i: Int = 0
while i < limit {
let item: String = json_array_get(arr, i)
let url: String = json_get(item, "url")
let title: String = json_get(item, "title")
let skip: Bool = str_eq(url, "") || str_contains(out, url)
let entry: String = if str_eq(title, "") { url } else { title + " (" + url + ")" }
let out = if skip {
out
} else {
if str_eq(out, "") { entry } else { out + "; " + entry }
}
let i = i + 1
}
return out
}
// provenance_add_sources one call site inside the content-block walk, so that walk keeps
// exactly one mutation per variable (the El scope rule documented at the walk).
// Reads sources from whichever block carries them: a cited text block's citations array, or
// a web_search_tool_result's own content array.
fn provenance_add_sources(block: String, btype: String, has_cit: Bool, cit_raw: String, acc: String) -> String {
// Hard cap on the whole accumulator: provenance is evidence, not payload.
if str_len(acc) > 600 { return acc }
if has_cit { return provenance_scan_urls(cit_raw, acc) }
if str_eq(btype, "web_search_tool_result") {
return provenance_scan_urls(json_get_raw(block, "content"), acc)
}
return acc
}
// provenance_names dedupe a tools_used JSON array into a readable list.
// json_array_get on an array of strings may or may not keep the quotes depending on the
// runtime build, so they are stripped defensively rather than assumed either way.
fn provenance_names(tools_used: String) -> String {
if str_eq(tools_used, "") { return "" }
if str_eq(tools_used, "[]") { return "" }
let total: Int = json_array_len(tools_used)
let limit: Int = if total > 12 { 12 } else { total }
let out: String = ""
let i: Int = 0
while i < limit {
let raw_nm: String = json_array_get(tools_used, i)
let nm: String = str_replace(raw_nm, "\"", "")
let skip: Bool = str_eq(nm, "") || str_contains(out, nm)
let out = if skip {
out
} else {
if str_eq(out, "") { nm } else { out + ", " + nm }
}
let i = i + 1
}
return out
}
// tool_receipt the line appended to an assistant turn's HISTORY copy.
//
// Emitted on every recorded turn, including turns where nothing ran. The negative receipt is
// not noise: it is the other half of the same guarantee. Without it, "no evidence of a tool"
// and "evidence of no tool" look identical in the transcript, which is precisely the
// ambiguity the model resolved against itself.
//
// text_join_sep the ONE rule for whether two pieces of model text need a break between them.
// (FIX C, 2026-08-05.)
//
// THE BUG "to.Good". Byte-verified in a shipped reply: 0x77 0x2e 0x47, "to" then "." then
// "Good", no space, no newline. Two text fragments concatenated with a bare `+` across a
// boundary where the model had actually stopped and started again.
//
// TWO SEAMS, ONE RULE. There were two bare `+` joins, written a year apart by different hands,
// and they had drifted into being two different decisions about the same question:
// - within one response, across content blocks (Will's, 2026-05-03)
// - across pause/resume rounds of the loop (ours, 62af564, the web_search port)
// Both are now expressed here. That is the point of hoisting it: a rule with one name and two
// call sites cannot drift into two rules again, and not incidentally a rule with a name is
// verifiable in the shipped binary, which an inline `+` is not.
//
// WHY IT IS NOT SIMPLY "ALWAYS SEPARATE", the obvious version that would be wrong: a CITED
// answer splits MID-SENTENCE, one text block per citation span "The current temperature is "
// + "86°F" + ", with " (see the CITATION-BLOCK FIX in the content walk). Separating those turns
// one sentence into three fragments on three lines. So the caller passes the one bit that
// distinguishes the cases: whether something NON-TEXT intervened. Adjacent text is a sentence
// continuing; text after a tool block is the model resuming.
//
// Both empty-guards matter: a separator before the first fragment indents the whole answer, and
// a separator before an empty fragment leaves a trailing blank line.
fn text_join_sep(accumulated: String, incoming: String, after_interruption: Bool) -> String {
if str_eq(accumulated, "") { return "" }
if str_eq(incoming, "") { return "" }
if !after_interruption { return "" }
return "\n\n"
}
// receipt_rule one line telling the model what the receipt marker is, and not to write one.
// Appended to both system prompts (the plain path's build_system_prompt and the agentic path's
// hand-built system string). See receipt_strip for why instruction alone is not enough.
fn receipt_rule() -> String {
return "\n\n[RECEIPTS - permanent]\nLines of the form [[RECEIPT ...]] in the conversation are written by the system, not by you. They are the record of which tools actually ran on a turn - read them as evidence, and rely on them when asked what you did or where information came from. NEVER write one yourself and never copy the format into your reply; the system adds them."
}
// receipt_strip remove a RECEIPT line the MODEL wrote, so it can never reach the user.
//
// FOUND BY E2E, NOT BY REASONING (2026-08-05). The receipt is stored inside the assistant turn
// and the agentic path replays history VERBATIM as Anthropic message objects so the model sees
// its own previous answers ending in [[RECEIPT ...]] and does the obvious thing: it imitates the
// format and signs its next answer the same way. Measured on the very first live run, on two
// turns out of two. The design note claimed "the user never sees it"; that was false, and only
// running it showed that.
//
// The plain path did not leak, which is the tell: there, history is rendered into the SYSTEM
// prompt as labelled lines rather than replayed as assistant messages, and a model imitates its
// own turns far more readily than it imitates a transcript.
//
// Instruction (receipt_rule) reduces this; only a deterministic strip PREVENTS it. Both ship,
// because a guard that depends on the model choosing to obey is the class of thing round 8 exists
// to stop shipping.
//
// EXCISE THE RECEIPT, DO NOT TRUNCATE AT IT bought with a measured regression, 2026-08-05.
// The first version of this function assumed the receipt is always TERMINAL and cut everything
// from the marker onward. It is not always terminal: told about the format in its system prompt,
// the model sometimes LEADS with a receipt and then writes the answer underneath. Cutting at the
// marker then deleted the entire answer, and the turn came back {"error":"no response"}.
// Measured on a two-search cited prompt: round 7 answered 4/4; that version answered 2/7. The
// A/B is the only reason this was caught it looked like a flaky model, and it was not.
// So: remove the [[...]] span and keep BOTH sides. An unterminated marker at position 0 is left
// alone entirely, because no rule about receipts is worth erasing an answer over.
fn receipt_strip(s: String) -> String {
let out: String = s
// Bounded pass rather than a conditional exit: rebinding the counter inside an if-expression
// is the block-expression shape that miscompiles integer arithmetic under this elc
// (BUG-PLAINCHAT-1). Four straight passes is cheaper than being clever, and once no marker
// remains every further pass is a no-op.
let guard: Int = 0
while guard < 4 {
let p: Int = str_index_of(out, "[[RECEIPT")
let found: Bool = p >= 0
let rest: String = if found { str_slice(out, p, str_len(out)) } else { "" }
let e: Int = if found { str_index_of(rest, "]]") } else { 0 - 1 }
let head: String = if found { str_slice(out, 0, p) } else { "" }
let tail: String = if e >= 0 { str_slice(rest, e + 2, str_len(rest)) } else { "" }
let out = if !found {
out
} else {
if e >= 0 {
head + tail
} else {
if p == 0 { out } else { head }
}
}
let guard = guard + 1
}
return str_trim(out)
}
fn tool_receipt(tools_used: String, sources: String) -> String {
let names: String = provenance_names(tools_used)
if str_eq(names, "") {
return "\n\n[[RECEIPT - recorded by the soul, not written by the model: no tools ran on this turn.]]"
}
let src_part: String = if str_eq(sources, "") { "" } else { " Sources retrieved: " + sources + "." }
return "\n\n[[RECEIPT - recorded by the soul, not written by the model: tools that actually executed on this turn: "
+ names + "." + src_part + "]]"
}
fn hist_trim(hist: String) -> String {
let inner: String = str_slice(hist, 1, str_len(hist) - 1)
let marker: String = "{\"role\":"
@@ -898,7 +1201,7 @@ fn clean_llm_response(s: String) -> String {
// conv_history_persist save conversation history to engram for cross-restart continuity.
// Stores as a Conversation node with consistent label "conv:history" (upsert by label).
// Q3/Q6 fix: added partial-write guard and failure logging.
fn conv_history_persist(hist: String) -> Void {
fn conv_history_persist(session_id: String, hist: String) -> Void {
if str_eq(hist, "") { return "" }
if str_eq(hist, "[]") { return "" }
// Partial-write guard: refuse to persist a blob that is not a complete JSON array.
@@ -906,8 +1209,9 @@ fn conv_history_persist(hist: String) -> Void {
if !str_starts_with(hist, "[") { return "" }
if !str_contains(hist, "]") { return "" }
let tags: String = "[\"conv-history\",\"persistent\"]"
// FIX B: one label rule, shared with the agentic path. See conv_hist_label.
let node_id: String = engram_node_full(
hist, "Conversation", "conv:history",
hist, "Conversation", conv_hist_label(session_id),
el_from_float(0.7), el_from_float(0.8), el_from_float(0.9),
"Episodic", tags
)
@@ -920,9 +1224,11 @@ fn conv_history_persist(hist: String) -> Void {
// conv_history_load restore conversation history from engram on first access.
// Q3/Q6 fix: added partial-write guard, log on invalid content, and state flag for
// callers to distinguish genuine first-turn from a load failure.
fn conv_history_load() -> String {
fn conv_history_load(session_id: String) -> String {
// FIX B: scoped label, shared with the agentic path. See conv_hist_label.
let hist_label: String = conv_hist_label(session_id)
// Primary: label-based fetch symmetric with persist, immune to vector index drift.
let label_node: String = engram_get_node_by_label("conv:history")
let label_node: String = engram_get_node_by_label(hist_label)
let label_ok: Bool = !str_eq(label_node, "") && !str_eq(label_node, "null")
if label_ok {
let label_content: String = json_get(label_node, "content")
@@ -933,7 +1239,7 @@ fn conv_history_load() -> String {
println("[chat] conv_history_load: label node found but content invalid — falling back to vector search")
}
// Fallback: vector search.
let results: String = engram_search_json("conv:history", 3)
let results: String = engram_search_json(hist_label, 3)
if str_eq(results, "") {
// Q3 fix: set a state flag so callers can distinguish load failure from first turn.
state_set("conv_history_load_failed", "1")
@@ -961,12 +1267,22 @@ fn conv_history_load() -> String {
// rather than the raw model output means the history window can never replay something the
// output gate replaced or augmented. Callers on a hard bell must not call this at all
// bell turns are kept out of conversation history by design (see layered_cycle).
fn conv_history_record(user_msg: String, assistant_msg: String) -> Void {
//
// FIX B (2026-08-05): keyed on the caller's session, via conv_hist_key the same rule the
// agentic path uses, so one conversation has one history no matter which switch position it
// was sent from.
//
// FIX A (2026-08-05): `receipt` is appended to the assistant turn AFTER safety_validate has
// run. That does not weaken the contract above: the receipt is soul-generated text about what
// the soul itself did, never model output, so there is nothing for the output gate to have an
// opinion about. Recording it inside the gated text would be the actual violation.
fn conv_history_record(session_id: String, user_msg: String, assistant_msg: String, receipt: String) -> Void {
if str_eq(user_msg, "") { return "" }
let state_hist: String = state_get("conv_history")
let stored_hist: String = if str_eq(state_hist, "") { conv_history_load() } else { state_hist }
let hist_key: String = conv_hist_key(session_id)
let state_hist: String = state_get(hist_key)
let stored_hist: String = if str_eq(state_hist, "") { conv_history_load(session_id) } else { state_hist }
let h1: String = hist_append(stored_hist, "user", user_msg)
let h2: String = hist_append(h1, "assistant", assistant_msg)
let h2: String = hist_append(h1, "assistant", assistant_msg + receipt)
// Bell-guarded trim: an evicted turn that triggered a bell is preserved to engram
// before it leaves the in-memory window.
let final_hist: String = if json_array_len(h2) > 20 {
@@ -974,18 +1290,18 @@ fn conv_history_record(user_msg: String, assistant_msg: String) -> Void {
} else {
h2
}
state_set("conv_history", final_hist)
conv_history_persist(final_hist)
state_set(hist_key, final_hist)
conv_history_persist(session_id, final_hist)
}
// conv_history_block recent dialogue, rendered for a system prompt.
//
// Same rendering handle_chat uses (role label + snipped content, one line per turn), read
// from the same "conv_history" window, so a plain-chat turn can follow the thread instead
// from the session's own window (FIX B), so a plain-chat turn can follow the thread instead
// of answering every message from cold. Read-only: never writes history.
fn conv_history_block() -> String {
let state_hist: String = state_get("conv_history")
let stored_hist: String = if str_eq(state_hist, "") { conv_history_load() } else { state_hist }
fn conv_history_block(session_id: String) -> String {
let state_hist: String = state_get(conv_hist_key(session_id))
let stored_hist: String = if str_eq(state_hist, "") { conv_history_load(session_id) } else { state_hist }
let hist_len: Int = if str_eq(stored_hist, "") { 0 } else { json_array_len(stored_hist) }
if hist_len == 0 {
return ""
@@ -997,8 +1313,15 @@ fn conv_history_block() -> String {
let rh_role: String = json_get(rh_entry, "role")
let rh_content: String = json_get(rh_entry, "content")
let rh_label: String = if str_eq(rh_role, "user") { "User" } else { "Assistant" }
let rh_snip: String = if str_len(rh_content) > 400 { str_slice(rh_content, 0, 400) + "..." } else { rh_content }
let rh_line: String = rh_label + ": " + rh_snip
// FIX A: the provenance receipt lives at the END of an assistant turn, so a plain
// 400-char head-snip would delete exactly the evidence this whole change exists to
// preserve and on a long sourced answer it would delete it every time. Split the
// receipt off, snip only the prose, then re-attach it.
let rh_cut: Int = str_index_of(rh_content, "\n\n[[RECEIPT")
let rh_body: String = if rh_cut < 0 { rh_content } else { str_slice(rh_content, 0, rh_cut) }
let rh_tail: String = if rh_cut < 0 { "" } else { str_slice(rh_content, rh_cut, str_len(rh_content)) }
let rh_snip: String = if str_len(rh_body) > 400 { str_slice(rh_body, 0, 400) + "..." } else { rh_body }
let rh_line: String = rh_label + ": " + rh_snip + rh_tail
let rh_out = if str_eq(rh_out, "") { rh_line } else { rh_out + "\n" + rh_line }
let rh_i = rh_i + 1
}
@@ -1044,7 +1367,7 @@ fn conv_history_block() -> String {
//
// Returns "" when the model call fails, so the caller reports the failure honestly instead
// of echoing the user's own text back at them.
fn layered_generate(prompt: String, imprint_id: String) -> String {
fn layered_generate(prompt: String, imprint_id: String, session_id: String) -> String {
if str_eq(prompt, "") {
return ""
}
@@ -1052,7 +1375,10 @@ fn layered_generate(prompt: String, imprint_id: String) -> String {
let ctx: String = engram_compile(prompt)
let model: String = chat_default_model()
let base_system: String = build_system_prompt(ctx, true) + current_engine_note(model)
let hist_block: String = conv_history_block()
// FIX B (2026-08-05): the session's own window, not the process-global one. This is the
// read half of the blank stare a turn sent with Tools OFF now sees the turns that were
// sent with Tools ON, because they are in the same bucket.
let hist_block: String = conv_history_block(session_id)
let full_system: String = base_system + hist_block
let raw: String = llm_call_system(model, full_system, prompt)
@@ -1065,7 +1391,9 @@ fn layered_generate(prompt: String, imprint_id: String) -> String {
return ""
}
return clean_llm_response(raw)
// FIX A follow-up: a model that has seen receipts in its context may sign its own answer
// with one. Strip it before the caller ever sees it. See receipt_strip.
return receipt_strip(clean_llm_response(raw))
}
// session_preload_bullets render up to max_bullets nodes from a JSON array as
@@ -1155,8 +1483,12 @@ fn handle_chat(body: String) -> String {
// Load history BEFORE compiling context so we can anchor activation to the thread.
// TODO(reliability #3 conv_history global race): process-global key; concurrent
// /api/chat requests without session_id race on this read-append-write.
let state_hist: String = state_get("conv_history")
let stored_hist: String = if str_eq(state_hist, "") { conv_history_load() } else { state_hist }
// NOTE 2026-08-05 (FIX B): this function is DEAD (see the banner above) and is left on the
// anonymous key deliberately. The race the TODO describes is exactly why the live plain
// path was scoped instead of given a fallback to this global. If this function is ever
// revived it must take a session_id and use conv_hist_key, like every live caller now does.
let state_hist: String = state_get(conv_hist_key(""))
let stored_hist: String = if str_eq(state_hist, "") { conv_history_load("") } else { state_hist }
let hist_load_failed: Bool = str_eq(state_get("conv_history_load_failed"), "1")
let hist_len: Int = if str_eq(stored_hist, "") { 0 } else { json_array_len(stored_hist) }
@@ -1352,8 +1684,8 @@ fn handle_chat(body: String) -> String {
} else {
updated_hist2
}
state_set("conv_history", final_hist)
conv_history_persist(final_hist)
state_set(conv_hist_key(""), final_hist)
conv_history_persist("", final_hist)
// Session-end summary hook: write a dated SessionSummary node once per boot when
// the conversation reaches >= 5 user turns (10 hist entries = 5 user+assistant pairs).
@@ -2187,6 +2519,24 @@ fn handle_chat_plan(body: String) -> String {
return "{\"plan\":" + plan_json + ",\"model\":\"" + json_safe(model) + "\"}"
}
// agentic_safety_screen the agentic path's L1 input gate
//
// Extracted 2026-08-07 (issue #129) so the agentic path's safety INPUT is
// reachable by a test. It owns exactly two decisions: which history window the
// screen sees, and the screen call itself.
//
// Why it is a function and not two inline lines: those two lines sat in the
// middle of a 300-line handler, and a key rename (ff421d3) moved the producer
// without moving this consumer. Nothing failed, nothing logged the
// history-amplification half of the crisis score simply received "" on every
// real session for a day. Inline safety inputs are untestable safety inputs.
// See tests/test_history_amplification.el, which fails if this window and
// conv_history_record ever stop agreeing.
fn agentic_safety_screen(session_id: String, message: String) -> String {
let history: String = state_get(conv_hist_key(session_id))
return safety_screen(message, history)
}
fn handle_chat_agentic(body: String) -> String {
let message: String = json_get(body, "message")
if str_eq(message, "") {
@@ -2222,10 +2572,10 @@ fn handle_chat_agentic(body: String) -> String {
// L1 safety screen agentic path must pass the same gate as layered_cycle.
// Hard bell: return the crisis response immediately, do not enter the agentic loop.
// Fix(issue #9): "conversation_history" key was never written; history lives under "conv_history".
// Old key caused history-amplification in safety_screen to always receive "" on agentic path.
let history: String = state_get("conv_history")
let screen_result: String = safety_screen(message, history)
// The history window this screen sees is owned by agentic_safety_screen (issue #129);
// it must be the same window conv_history_record writes, or the escalation half of the
// crisis score is silently starved. Do not inline this read back into the handler.
let screen_result: String = agentic_safety_screen(sess_for_root, message)
let screen_action: String = json_get(screen_result, "action")
if str_eq(screen_action, "hard_bell") {
safety_log_bell("hard", json_get(screen_result, "reason"), str_slice(message, 0, 80))
@@ -2253,7 +2603,10 @@ fn handle_chat_agentic(body: String) -> String {
return "{\"error\":\"session not found\",\"session_id\":\"" + req_session + "\",\"reply\":\"\"}"
}
let hist_key: String = if str_eq(req_session, "") { "conv_history" } else { "session_hist_" + req_session }
// FIX B (2026-08-05): the key rule now lives in one place and the plain path uses the
// same one. Behaviour on this path is unchanged conv_hist_key reproduces exactly what
// this line computed inline but there is no longer a second, divergent definition.
let hist_key: String = conv_hist_key(req_session)
let agentic_hist: String = state_get(hist_key)
let agentic_hist_len: Int = if str_eq(agentic_hist, "") { 0 } else { json_array_len(agentic_hist) }
// Issue 8 fix: use engram_is_continuation instead of brittle 50-char threshold.
@@ -2311,7 +2664,7 @@ fn handle_chat_agentic(body: String) -> String {
let system: String = identity + bounded_persona_floor() + " You have access to tools: read files, write files, browse the web, search your memory, run commands. Use them when they add genuine value. Be direct.
" + ctx + ag_session_preload
" + ctx + ag_session_preload + receipt_rule()
let api_key: String = agentic_api_key()
let tools_json: String = agentic_tools_all()
@@ -2367,38 +2720,31 @@ fn handle_chat_agentic(body: String) -> String {
// Persist the exchange to session/global history for thread continuity on next turn.
// Only save when the loop completed (reply present), not when tool_pending.
let reply_text: String = json_get(result, "reply")
let discard_hist: Bool = if !str_eq(reply_text, "") {
// FIX A (2026-08-05): the evidence the next turn needs. `result` already carries
// tools_used, and agentic_loop now also returns the source URLs it saw; both are folded
// into a receipt line and stored WITH the assistant turn. Without this the next turn sees
// a sourced answer and no trace of the search, and concludes it made the data up the
// false confession. See tool_receipt.
let turn_tools: String = json_get_raw(result, "tools_used")
let turn_sources: String = json_get(result, "sources")
let turn_receipt: String = tool_receipt(turn_tools, turn_sources)
// FIX E1 (2026-08-05): a utility generation (title, insight) is not conversation and is
// not recorded as one. It is still answered normally only the transcript is spared.
let record_turn: Bool = !str_eq(reply_text, "") && !is_utility_request(body, req_session)
let discard_hist: Bool = if record_turn {
let updated: String = hist_append(agentic_hist, "user", message)
let updated2: String = hist_append(updated, "assistant", reply_text)
let updated2: String = hist_append(updated, "assistant", reply_text + turn_receipt)
// Increased from 20 to 40 turns: consistent with handle_chat window expansion.
let trimmed: String = if json_array_len(updated2) > 40 { hist_trim(updated2) } else { updated2 }
state_set(hist_key, trimmed)
// Persist to engram for cross-restart continuity.
// Named sessions get session-scoped labels, fixing ephemeral-only limitation (issue #4).
if str_eq(hist_key, "conv_history") {
conv_history_persist(trimmed)
} else {
if !str_eq(trimmed, "") && !str_eq(trimmed, "[]") {
let sess_hist_label: String = "conv:history:" + req_session
let sess_hist_tags: String = "[\"session-history\",\"persistent\"]"
let sess_hist_id: String = engram_node_full(
trimmed, "Conversation", sess_hist_label,
el_from_float(0.6), el_from_float(0.7), el_from_float(0.8),
"Episodic", sess_hist_tags
)
// NOTE: bind an explicit Bool value here. A bare `if { println(...) }`
// leaves a void-typed branch in value position, which the current elc
// lowers to `_if_result = (println(...))` invalid C. Yielding a value
// keeps the branch non-void without changing behavior (still only logs).
let persist_ok: Bool = if str_eq(sess_hist_id, "") {
println("[chat] agentic: named session history persist failed for session=" + req_session)
false
} else { true }
persist_ok
} else {
false
}
}
// FIX B (2026-08-05): ONE persist, through the shared helper. This site used to hold
// a second, hand-rolled copy of the same write for named sessions a different label
// expression, different salience scores and a different tag set for the same data.
// Since conv_hist_label now gives both callers the same label and engram_node_full
// upserts by label, two score policies were writing the same node. One writer, one
// label rule, one policy.
conv_history_persist(req_session, trimmed)
true
} else { false }
@@ -2432,6 +2778,11 @@ fn agentic_loop(session_id: String, model: String, safe_sys: String, tools_json:
let messages: String = messages_in
let final_text: String = ""
let tools_log: String = tools_log_in
// FIX A (2026-08-05): source URLs accumulated across every round of this turn, so the
// receipt written into history can name what the search actually returned. Carried at
// loop level for the same reason tools_log is: a resumed round must not lose the
// evidence gathered before the pause.
let sources_all: String = ""
let iteration: Int = 0
let keep_going: Bool = true
@@ -2504,6 +2855,30 @@ fn agentic_loop(session_id: String, model: String, safe_sys: String, tools_json:
+ ",\"messages\":" + messages
+ "}"
// ROUND-START MARKER (2026-08-06, round 9.1 D2 / ADR 0006 item 2)
// The ledger below only ever appended AFTER a round returned, so a healthy
// first leg produced ZERO progress by construction. Since server-side
// web_search moved inside the outbound call (2026-08-04) that leg measures
// 84-117 s, and the client had no way to tell "working" from "dead" which is
// how a 25 s client-side watchdog came to kill a healthy mission.
//
// Only this loop knows a round has started, so only this loop can say so. One
// entry, written BEFORE the call goes out, using the ledger and the wire shape
// that already exist: the app has handled tool == "__working__" as an
// Activity-only life signal since 2026-07-13 (ChatView.kt:1148) and never
// received one. Narration is deliberately empty - the marker means "a round
// started", nothing more, and the client renders it as a heartbeat, not prose.
//
// This is a strict subset of WS3 item 3 (push/poll progress). It builds none of
// WS3's run registry: no new state key, no new route, no new lifecycle.
if !str_eq(session_id, "") {
let start_key: String = "run_progress_" + session_id
let start_prev: String = state_get(start_key)
let start_entry: String = "{\"i\":" + int_to_str(iteration) + ",\"t\":\"\",\"tool\":\"__working__\"}"
let start_next: String = if str_eq(start_prev, "") { start_entry } else { start_prev + "," + start_entry }
state_set(start_key, start_next)
}
let raw_resp: String = http_post_with_headers(api_url, req_body, h)
let is_error: Bool = str_starts_with(raw_resp, "{\"error\"")
@@ -2578,6 +2953,13 @@ fn agentic_loop(session_id: String, model: String, safe_sys: String, tools_json:
// separate from tools_log so this inner walk has exactly one mutation site per
// variable (the El scope rule below), then merged in at the outer level.
let srv_log: String = ""
// FIX A: source URLs seen this round (citations + web_search results). Merged into
// the loop-level accumulator below, same shape as srv_log.
let src_log: String = ""
// FIX C: seam tracking. True once a NON-text block has been walked, so the next text
// block knows it is resuming after an interruption rather than continuing a sentence.
// See the separator decision at the text accumulation site.
let saw_nontext: Bool = false
let ci: Int = 0
let c_total: Int = json_array_len(eff_content)
while ci < c_total {
@@ -2595,8 +2977,34 @@ fn agentic_loop(session_id: String, model: String, safe_sys: String, tools_json:
let has_cit: Bool = !str_eq(cit_raw, "") && !str_eq(cit_raw, "null")
let btype_scan: String = json_get(block, "type")
let btype: String = if has_cit { "text" } else { btype_scan }
// Accumulate text at top level using if-expression
let text_out = if str_eq(btype, "text") { text_out + json_get(block, "text") } else { text_out }
// ── FIX C, seam 1 of 2 (2026-08-05): "to.Good" ────────────────────────────────
// Byte-verified in a shipped reply: 0x77 0x2e 0x47 — "to" then "." then "Good",
// with no space and no newline. The bare `+` below joined the last sentence of a
// pre-search paragraph directly onto the first word of the post-search paragraph.
// Will wrote this line on 2026-05-03 and it was correct for a year: before
// server-side web_search, text blocks were adjacent, and adjacent text blocks are
// one continuous string that must be joined with nothing.
//
// WHY THE OBVIOUS FIX IS WRONG. Inserting a separator between all text blocks
// shatters every cited answer. A cited response splits MID-SENTENCE, one block per
// citation span: "The current temperature is " + "86°F" + ", with " — see the
// CITATION-BLOCK FIX note above. A blanket "\n\n" turns that into three fragments
// on three lines. Both failure modes are real and they pull in opposite directions.
//
// THE DISTINCTION THAT RESOLVES IT: a text block that directly follows another
// text block is a continuation and gets nothing; a text block that follows an
// INTERVENING NON-TEXT block (server_tool_use, web_search_tool_result, tool_use)
// resumes after an interruption and gets "\n\n". saw_nontext carries exactly that
// one bit, and text_join_sep holds the rule — shared with the resume seam below.
// Mid-sentence citation splits are untouched: no non-text block sits between them.
let is_text: Bool = str_eq(btype, "text")
let btext: String = if is_text { json_get(block, "text") } else { "" }
let text_out = if is_text {
text_out + text_join_sep(text_out, btext, saw_nontext) + btext
} else { text_out }
let saw_nontext = if is_text { false } else { true }
// FIX A: record where the facts came from, from whichever block carries them.
let src_log = provenance_add_sources(block, btype, has_cit, cit_raw, src_log)
// FUTURE-PROOF: tools Anthropic runs on our behalf (web_search today, whatever
// ships tomorrow) arrive as server_tool_use blocks, never as client tool_use.
// Count the CATEGORY by the block's own name so a new server tool appears in
@@ -2678,6 +3086,12 @@ fn agentic_loop(session_id: String, model: String, safe_sys: String, tools_json:
} else {
if str_eq(tools_log, "") { srv_log } else { tools_log + "," + srv_log }
}
// FIX A: same merge, for the sources seen in this round's blocks.
let sources_all = if str_eq(src_log, "") {
sources_all
} else {
if str_eq(sources_all, "") { src_log } else { sources_all + "; " + src_log }
}
// The assistant turn that requested the tool needed verbatim on resume so the
// tool_use/tool_result pairing stays valid when the client posts its result.
@@ -2732,7 +3146,17 @@ fn agentic_loop(session_id: String, model: String, safe_sys: String, tools_json:
// CONTINUES the answer, it does not repeat it overwriting here would throw away
// everything the model wrote before the pause, which is the same truncation the
// pause handling exists to prevent. A version-fallback round contributes nothing.
let final_text = if !is_tool_turn && !can_fallback { final_text + text_out } else { final_text }
//
// FIX C, seam 2 of 2 (2026-08-05)
// The other half of "to.Good". This join is ours (62af564, the web_search port) and
// is unconditionally a boundary: the two sides are separate rounds of the Anthropic
// loop, separated by a pause and a tool execution. There is no mid-sentence case to
// protect here the model was interrupted, and when it resumes it starts a new
// thought. So this seam passes after_interruption=true unconditionally; text_join_sep's
// own empty-guards handle the first round and an empty round.
let final_text = if !is_tool_turn && !can_fallback {
final_text + text_join_sep(final_text, text_out, true) + text_out
} else { final_text }
// Output cap hit mid-action: the tool block is truncated and will NOT run. Say so
// instead of ending on silent almost-work.
let final_text = if str_eq(stop_reason, "max_tokens") && has_tool {
@@ -2756,6 +3180,7 @@ fn agentic_loop(session_id: String, model: String, safe_sys: String, tools_json:
+ ",\"narration\":\"" + json_safe(pend_narration) + "\""
+ ",\"model\":\"" + model + "\""
+ ",\"agentic\":true"
+ ",\"sources\":\"" + json_safe(sources_all) + "\""
+ ",\"tools_used\":" + tools_arr + "}"
}
@@ -2763,6 +3188,10 @@ fn agentic_loop(session_id: String, model: String, safe_sys: String, tools_json:
// genuine no-response (model returned an empty text block). The iteration cap
// means the task was too complex for the agentic loop depth surface it clearly
// so the caller/operator knows to increase the cap or break the task apart.
// FIX A follow-up: strip any receipt the MODEL wrote before this becomes the reply. Placed
// ABOVE the empty check on purpose a turn whose entire output was an imitated receipt has
// produced no answer, and must be reported as no answer rather than as a receipt.
let final_text = receipt_strip(final_text)
if str_eq(final_text, "") {
let hit_cap: Bool = iteration >= 12
let err_msg: String = if hit_cap {
@@ -2782,7 +3211,10 @@ fn agentic_loop(session_id: String, model: String, safe_sys: String, tools_json:
let done_next: String = if str_eq(done_prev, "") { "{\"done\":true}" } else { done_prev + ",{\"done\":true}" }
state_set(done_key, done_next)
}
return "{\"reply\":\"" + safe_text + "\",\"model\":\"" + model + "\",\"agentic\":true,\"tools_used\":" + tools_arr + ",\"iterations\":" + int_to_str(iteration) + "}"
// FIX A: "sources" carries the URLs this turn actually retrieved, so handle_chat_agentic
// can write them into the history receipt and the next turn can answer "what source did
// you use?" from the transcript instead of guessing (or apologising).
return "{\"reply\":\"" + safe_text + "\",\"model\":\"" + model + "\",\"agentic\":true,\"tools_used\":" + tools_arr + ",\"sources\":\"" + json_safe(sources_all) + "\",\"iterations\":" + int_to_str(iteration) + "}"
}
// bridge_save persist a suspended agentic turn keyed by session_id. Stored as a
@@ -2800,12 +3232,31 @@ fn bridge_save(session_id: String, model: String, safe_sys: String, tools_json:
// JSON values (not string-escaped) so the round-trip through state_get/json_get_raw
// never corrupts nested quotes. Scalar strings (model, safe_sys, tools_log,
// tool_use_id) stay as string fields via json_safe as before.
//
// FIELD ORDER IS LOAD-BEARING (round-9 fix, 2026-08-06). json_get is a first-
// substring-match scanner (strstr for "\"key\":", el_runtime.c), and the two raw
// fields embed the UNESCAPED conversation every key the model's own blocks carry
// ("tool_use_id" in each web_search_tool_result, "content", "type", ...) is findable
// by a whole-blob scan. With messages_raw serialized BEFORE tool_use_id, the resume
// read json_get(blob, "tool_use_id") returned the FIRST web_search_tool_result's
// srvtoolu_ id instead of the saved client-tool id, so every search-then-bridge
// turn 400'd on approval ("unexpected tool_use_id found in tool_result blocks:
// srvtoolu_…") and the run died as {"error":"llm unavailable"}. Same first-match-
// scanner class as BUG-6 (approve "content" matched inside tool_input) and the
// round-8 citation-block fix.
//
// The rule: every json_safe'd scalar precedes both raw fields (escaping means a
// scalar value can never contain a bare "key": byte pattern, so first-match lands
// on the blob's own fields), and tools_raw our own fixed tool schema precedes
// messages_raw arbitrary model/user content so neither raw extraction can
// first-match into model-controlled bytes either. Do not reorder; do not add a
// field after messages_raw.
let blob: String = "{\"model\":\"" + json_safe(model) + "\""
+ ",\"safe_sys\":\"" + json_safe(safe_sys) + "\""
+ ",\"messages_raw\":" + messages
+ ",\"tools_raw\":" + tools_json
+ ",\"tools_log\":\"" + json_safe(tools_log) + "\""
+ ",\"tool_use_id\":\"" + json_safe(tool_use_id) + "\"}"
+ ",\"tool_use_id\":\"" + json_safe(tool_use_id) + "\""
+ ",\"tools_raw\":" + tools_json
+ ",\"messages_raw\":" + messages + "}"
state_set("mcp_bridge:" + session_id, blob)
return true
}
@@ -2842,11 +3293,17 @@ fn agentic_resume(session_id: String, tool_use_id: String, content: String) -> S
let tools_log: String = json_get(blob, "tools_log")
let saved_use_id: String = json_get(blob, "tool_use_id")
// Bind the result to the tool the soul actually suspended on. The client should
// echo the call_id; if it omits or mismatches it, fall back to the saved id so a
// late/partial client still resumes correctly.
let use_id: String = if str_eq(tool_use_id, "") { saved_use_id } else { tool_use_id }
let eff_use_id: String = if str_eq(use_id, saved_use_id) { use_id } else { saved_use_id }
// Bind the result to the tool the loop actually suspended on. The client echoes
// the call_id from the pending envelope; that value came straight from
// pend_tool_id and never round-tripped through this blob, so when both are
// present and disagree the CLIENT's id is the one with clean provenance (a blob
// written by a pre-round-9 binary misreads tool_use_id by first-match scanning
// into messages_raw see bridge_save). A client that omits call_id still
// resumes on the saved id, which the reordered blob now reads correctly.
// (The old guard here "on mismatch, prefer saved" reduced to eff_use_id
// saved_use_id in both branches: the client's correct id could never win, which
// is what turned the misread into a deterministic 400 on resume.)
let eff_use_id: String = if str_eq(tool_use_id, "") { saved_use_id } else { tool_use_id }
// Result may be large (an MCP page/file); truncate like local tool results do.
let trimmed: String = if str_len(content) > 6000 {
+26 -3
View File
@@ -1,4 +1,4 @@
// auto-generated by elc --emit-header do not edit
// auto-generated by elc --emit-header - do not edit
extern fn chat_default_model() -> String
extern fn engram_numeric_valid(s: String) -> Bool
extern fn parse_float_x100(s: String) -> Int
@@ -16,18 +16,35 @@ extern fn engram_nodes_merge(a: String, b: String) -> String
extern fn id_in_seen(node_id: String, seen: String) -> Bool
extern fn add_to_seen(seen: String, node_id: String) -> String
extern fn engram_extract_ids(nodes_json: String) -> String
extern fn affective_node_ts(node_json: String) -> Int
extern fn engram_compile(intent: String) -> String
extern fn distill_transcript(transcript: String) -> String
extern fn json_safe(s: String) -> String
extern fn current_engine_note(model: String) -> String
extern fn bounded_persona_floor() -> String
extern fn operator_identity_block() -> String
extern fn build_system_prompt(ctx: String, chat_mode: Bool) -> String
extern fn hist_append(hist: String, role: String, content: String) -> String
extern fn conv_hist_key(session_id: String) -> String
extern fn conv_hist_label(session_id: String) -> String
extern fn is_utility_request(body: String, session_id: String) -> Bool
extern fn provenance_scan_urls(arr: String, acc: String) -> String
extern fn provenance_add_sources(block: String, btype: String, has_cit: Bool, cit_raw: String, acc: String) -> String
extern fn provenance_names(tools_used: String) -> String
extern fn text_join_sep(accumulated: String, incoming: String, after_interruption: Bool) -> String
extern fn receipt_rule() -> String
extern fn receipt_strip(s: String) -> String
extern fn tool_receipt(tools_used: String, sources: String) -> String
extern fn hist_trim(hist: String) -> String
extern fn hist_trim_with_bell_guard(hist: String) -> String
extern fn clean_llm_response(s: String) -> String
extern fn conv_history_persist(hist: String) -> Void
extern fn conv_history_load() -> String
extern fn conv_history_persist(session_id: String, hist: String) -> Void
extern fn conv_history_load(session_id: String) -> String
extern fn conv_history_record(session_id: String, user_msg: String, assistant_msg: String, receipt: String) -> Void
extern fn conv_history_block(session_id: String) -> String
extern fn layered_generate(prompt: String, imprint_id: String, session_id: String) -> String
extern fn session_preload_bullets(nodes: String, max_bullets: Int, snip_len: Int) -> String
extern fn affective_context_prefix() -> String
extern fn handle_chat(body: String) -> String
extern fn handle_see(body: String) -> String
extern fn studio_tools_json() -> String
@@ -37,6 +54,8 @@ extern fn llm_wire_format() -> String
extern fn json_escape(s: String) -> String
extern fn openai_chat_complete(model: String, base_url: String, api_key: String, safe_sys: String, messages_json: String) -> String
extern fn agentic_tools_literal() -> String
extern fn web_search_tool_json() -> String
extern fn strip_client_web_search(tools_inner: String) -> String
extern fn agentic_tools_with_web() -> String
extern fn connector_tools_json() -> String
extern fn agentic_tools_all() -> String
@@ -46,6 +65,10 @@ extern fn call_neuron_mcp(tool_name: String, args: String) -> String
extern fn agent_workspace_root() -> String
extern fn path_within_root(path: String, root: String) -> Bool
extern fn resolve_in_root(path: String, root: String) -> String
extern fn run_command_is_readonly(cmd: String) -> Bool
extern fn cmd_abs_escape_at(cmd: String, root: String, needle: String) -> Bool
extern fn run_command_guard(cmd: String, root: String) -> String
extern fn classify_tool_risk(tool_name: String, tool_input: String) -> String
extern fn dispatch_tool(tool_name: String, tool_input: String) -> String
extern fn is_builtin_tool(tool_name: String) -> Bool
extern fn next_bridge_id() -> String
Generated Vendored
+17 -3
View File
@@ -5,6 +5,15 @@ el_val_t add_punct(el_val_t s, el_val_t intent);
el_val_t add_to_seen(el_val_t seen, el_val_t node_id);
el_val_t aff_try_slot(el_val_t slot_json, el_val_t aff_7d_ts, el_val_t acc_key);
el_val_t affective_context_prefix(void);
el_val_t is_utility_request(el_val_t body, el_val_t session_id);
el_val_t operator_identity_block(void);
el_val_t provenance_add_sources(el_val_t block, el_val_t btype, el_val_t has_cit, el_val_t cit_raw, el_val_t acc);
el_val_t provenance_names(el_val_t tools_used);
el_val_t provenance_scan_urls(el_val_t arr, el_val_t acc);
el_val_t text_join_sep(el_val_t accumulated, el_val_t incoming, el_val_t after_interruption);
el_val_t receipt_rule(void);
el_val_t receipt_strip(el_val_t s);
el_val_t tool_receipt(el_val_t tools_used, el_val_t sources);
el_val_t agent_number(el_val_t agent);
el_val_t agent_person(el_val_t agent);
el_val_t agent_workspace_root(void);
@@ -151,8 +160,12 @@ el_val_t cmd_abs_escape_at(el_val_t cmd, el_val_t root, el_val_t needle);
el_val_t connectd_get(el_val_t suffix);
el_val_t connectd_post(el_val_t suffix, el_val_t body);
el_val_t connector_tools_json(void);
el_val_t conv_history_load(void);
el_val_t conv_history_persist(el_val_t hist);
el_val_t conv_hist_key(el_val_t session_id);
el_val_t conv_hist_label(el_val_t session_id);
el_val_t conv_history_block(el_val_t session_id);
el_val_t conv_history_load(el_val_t session_id);
el_val_t conv_history_persist(el_val_t session_id, el_val_t hist);
el_val_t conv_history_record(el_val_t session_id, el_val_t user_msg, el_val_t assistant_msg, el_val_t receipt);
el_val_t cop_article(el_val_t gender, el_val_t number, el_val_t definite);
el_val_t cop_bwk_future(el_val_t prefix);
el_val_t cop_bwk_perfect(el_val_t prefix);
@@ -782,7 +795,8 @@ el_val_t lang_profile_uga(void);
el_val_t lang_profile_zh(void);
el_val_t lang_profile(el_val_t code, el_val_t word_order, el_val_t morph_type, el_val_t has_case, el_val_t has_gender, el_val_t script_dir, el_val_t agreement, el_val_t null_subject);
el_val_t lang_word_order(el_val_t profile);
el_val_t layered_cycle(el_val_t raw_input);
el_val_t layered_cycle(el_val_t raw_input, el_val_t session_id, el_val_t utility);
el_val_t layered_generate(el_val_t prompt, el_val_t imprint_id, el_val_t session_id);
el_val_t lex_class(el_val_t entry);
el_val_t lex_form(el_val_t entry, el_val_t idx);
el_val_t lex_pos(el_val_t entry);
+10 -3
View File
@@ -280,7 +280,10 @@ fn handle_dharma_recv(body: String) -> String {
// Non-agentic ("Tools: Off"): the full L1L2L3L1 cycle, which now generates
// at L3 instead of echoing. Envelope built outside the cycle see
// plain_chat_envelope.
let screened_reply: String = layered_cycle(raw_msg)
// FIX B/E1 (2026-08-05): the cycle is told which conversation it is in, and
// whether this generation is conversation at all. Same two arguments at all
// three dispatch sites.
let screened_reply: String = layered_cycle(raw_msg, json_get(chat_body, "session_id"), is_utility_request(chat_body, json_get(chat_body, "session_id")))
plain_chat_envelope(screened_reply, chat_default_model())
}
auto_persist(chat_body, reply)
@@ -454,7 +457,9 @@ fn handle_request(method: String, path: String, body: String) -> String {
handle_chat_agentic(body)
} else {
// Non-agentic ("Tools: Off") same cycle and same envelope as POST.
let screened_reply: String = layered_cycle(eff_msg)
// FIX B/E1: same threading. A GET probe usually carries no session_id, which
// resolves to the anonymous window the documented behaviour for this door.
let screened_reply: String = layered_cycle(eff_msg, json_get(body, "session_id"), is_utility_request(body, json_get(body, "session_id")))
plain_chat_envelope(screened_reply, chat_default_model())
}
auto_persist(body, reply)
@@ -621,7 +626,9 @@ fn handle_request(method: String, path: String, body: String) -> String {
// Non-agentic ("Tools: Off") the app's DEFAULT mode (AgentMode.NEVER).
// Full L1L2L3L1 cycle with real generation at L3; envelope built
// outside the cycle so safety_validate always sees raw text.
let screened_reply: String = layered_cycle(raw_msg)
// FIX B/E1: same threading. This is the app's main plain-chat door, so this
// is the site that ends the blank stare in practice.
let screened_reply: String = layered_cycle(raw_msg, json_get(body, "session_id"), is_utility_request(body, json_get(body, "session_id")))
plain_chat_envelope(screened_reply, chat_default_model())
}
auto_persist(body, reply)
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env bash
# run-el-test.sh — compile and run one El test program from tests/.
#
# WHY THIS EXISTS (2026-08-07, issue #129):
# tests/ has held 14 test programs for months with no way to run them. CI does
# not run them. The convention printed in their own headers
# (`elc soul.el && ./soul --test tests/x.el`) refers to a --test flag the El
# runtime does not implement. So the tests were documentation, not gates —
# which is how a P0 safety regression shipped with a test directory present.
#
# THE RECIPE, AND WHY IT IS THIS SHAPE:
# Same discovery as gen-soul-amalgam.sh — `elc --target=c` emits only an extern
# prototype for any module that has a .elh header next to it, and inlines the
# module's bodies when it does not. A test that imports ../chat.el therefore
# compiles to a 18 KB unit full of unresolved externs unless the headers are
# out of the way. So: copy the sources into a scratch tree, delete every .elh
# on the import chain, and compile the test there.
#
# Scratch copy on purpose: the worktree is shared with other terminals and
# deleting headers in place would be a shared-tree mutation with no owner.
#
# EXIT STATUS IS THE GATE: non-zero if the binary fails to build, crashes, or if
# its output contains a FAIL line or reports a non-zero failed count. Do not
# "improve" this into something that only checks the exit code of the test
# binary — these El tests print failures and still exit 0.
#
# usage: scripts/run-el-test.sh tests/test_history_amplification.el
set -euo pipefail
TEST_REL="${1:?usage: run-el-test.sh tests/<test>.el}"
SRC="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
TEST_NAME="$(basename "$TEST_REL" .el)"
ELC="${ELC:-$HOME/neuron-dev-stack/src/el/lang/dist/platform/elc}"
[ -x "$ELC" ] || ELC="$HOME/el-sdk/elc"
[ -x "$ELC" ] || { echo "[run-el-test] FAIL: no elc found (set ELC=)"; exit 1; }
RTC="${RTC:-$SRC/vendor/el-runtime/v1.0.0-20260501/el_runtime.c}"
[ -f "$RTC" ] || RTC="$HOME/el-sdk/el_runtime.c"
[ -f "$RTC" ] || { echo "[run-el-test] FAIL: no el_runtime.c found (set RTC=)"; exit 1; }
RTDIR="$(dirname "$RTC")"
EL_REPO="${EL_REPO:-$HOME/Development/neuron-technologies/el}"
SSL="${SSL_PREFIX:-/opt/homebrew/opt/openssl@3}"
GEN="$(mktemp -d "${TMPDIR:-/tmp}/el-test.XXXXXX")"
trap 'rm -rf "$GEN"' EXIT
mkdir -p "$GEN/neuron/tests" "$GEN/foundation/el/elp/src"
cp "$SRC"/*.el "$GEN/neuron/"
cp "$SRC"/tests/*.el "$GEN/neuron/tests/" 2>/dev/null || true
[ -d "$EL_REPO/elp/src" ] && cp "$EL_REPO"/elp/src/*.el "$GEN/foundation/el/elp/src/" 2>/dev/null || true
# The whole recipe depends on there being no headers to short-circuit inlining.
find "$GEN" -name '*.elh' -delete
echo "[run-el-test] compiling $TEST_REL"
( cd "$GEN/neuron" && "$ELC" --target=c "tests/${TEST_NAME}.el" ) > "$GEN/${TEST_NAME}.c"
BODIES=$(grep -c '^el_val_t .*) {$' "$GEN/${TEST_NAME}.c" || true)
echo "[run-el-test] $(wc -c < "$GEN/${TEST_NAME}.c" | tr -d ' ') bytes, ${BODIES} inlined function bodies"
# A test that imports ../chat.el pulls in the bulk of the engine. A tiny body
# count means an import was read from a header instead of inlined, and the test
# would be exercising extern stubs rather than the real code.
if [ "$BODIES" -lt 100 ]; then
echo "[run-el-test] FAIL: only $BODIES inlined bodies — an import was not inlined"
exit 1
fi
cc -O2 -DHAVE_CURL \
-I"$RTDIR" -I"$SSL/include" -L"$SSL/lib" \
"$GEN/${TEST_NAME}.c" "$RTC" \
-lssl -lcrypto -lcurl -lpthread -lm \
-o "$GEN/${TEST_NAME}" 2> "$GEN/cc.log" || {
echo "[run-el-test] FAIL: compile error"; tail -30 "$GEN/cc.log"; exit 1; }
# arm64 pointer-truncation guard (cc-brain.sh's rule): an implicit declaration of
# a runtime symbol truncates its returned pointer to 32 bits.
if grep -E 'implicit.*(engram_|el_)' "$GEN/cc.log"; then
echo "[run-el-test] FAIL: implicit declarations of runtime symbols"; exit 1; fi
# Throwaway HOME so a test can never read or write the live engram at ~/.neuron.
TEST_HOME="$GEN/home"
mkdir -p "$TEST_HOME"
echo "[run-el-test] running $TEST_NAME"
set +e
HOME="$TEST_HOME" NEURON_HOME="$TEST_HOME/.neuron" "$GEN/${TEST_NAME}" 2>&1 | tee "$GEN/out.txt"
RC=${PIPESTATUS[0]}
set -e
if [ "$RC" -ne 0 ]; then
echo "[run-el-test] FAIL: $TEST_NAME exited $RC (crash or abort)"
exit 1
fi
if grep -q " FAIL:" "$GEN/out.txt"; then
echo "[run-el-test] FAIL: $TEST_NAME reported failing assertions"
exit 1
fi
if grep -qE '[1-9][0-9]* failed' "$GEN/out.txt"; then
echo "[run-el-test] FAIL: $TEST_NAME reported a non-zero failed count"
exit 1
fi
if ! grep -q "PASS:" "$GEN/out.txt"; then
echo "[run-el-test] FAIL: $TEST_NAME produced no assertions at all"
exit 1
fi
echo "[run-el-test] PASS: $TEST_NAME"
+937
View File
@@ -0,0 +1,937 @@
#!/usr/bin/env python3
"""state-key-audit.py — the analyzer behind scripts/verify-state-keys.sh.
Read that script's header for WHY this exists (issue #129). This file is the
HOW: a small El reader that resolves the key expression at every state_get /
state_set site, including keys that are computed.
WHAT IT PARSES
El as this engine writes it: `fn f(a: T, b: T) -> T { ... }`, `let x: T = e`,
`return e`, `if c { a } else { b }` as an expression, `+` concatenation,
`"..."` with backslash escapes, `//` line comments. No block comments, no
const/match/struct exist in this dialect (verified over the whole tree).
KEY PATTERNS the only two things a key expression can resolve to
EXACT "soul_model" the whole key is known
PREFIX "session_hist_" a known head, then runtime text
(plus UNRESOLVED, which is a report line and never a failure)
RESOLUTION resolve_expr() returns a SET of patterns; unions are how branches,
multiple returns, and multiple bindings of one name are represented.
literal "k" -> {EXACT k}
concat A + B -> fold left; all-static -> EXACT,
static head + dynamic tail -> PREFIX
if-expression if c {A} else {B} -> resolve(A) | resolve(B), except that
str_eq(X,"") with X statically ""
folds to the taken branch only
call f(args) -> union over f's return expressions,
with f's params bound to THIS call
site's actual argument expressions
local var let k = e; state_get(k)-> union over every `let k =` in the
enclosing function
parameter fn g(k) { state_get(k) }-> union over the argument at that
position across every call site of g
anything else json_get(...), env(...)-> UNRESOLVED
Recursion is depth- and cycle-guarded; a guard trip yields UNRESOLVED, never a
failure.
COVERAGE a read is satisfied when some write can produce the same key:
read EXACT k <- write EXACT k, or write PREFIX p where k starts with p
read PREFIX p <- write EXACT k where k starts with p, or write PREFIX q
where p and q are prefixes of each other
Deliberately permissive at the boundaries: a gate that cries wolf gets deleted.
"""
import os
import re
import sys
MAX_DEPTH = 12
# ── patterns ────────────────────────────────────────────────────────────────
EXACT = "exact"
PREFIX = "prefix"
def pat_exact(s):
return (EXACT, s)
def pat_prefix(s):
# A prefix with no static text at all carries no information; that is the
# UNRESOLVED case, not a pattern.
return (PREFIX, s) if s else None
def covers(write, read):
"""Can a write of pattern `write` produce a key that `read` reads?
The prefix rule is DIRECTIONAL, and that direction is the whole point. A
write namespace that is the same or BROADER than the read namespace covers
it (write "rl:" covers read "rl:x"). A write namespace that is NARROWER does
NOT (write "session_histv2_" does not cover read "session_hist_") being
permissive there re-opens the exact hole this gate exists to close: rename
the producer, leave the readers, stay green. Verified with a control run
that renames sessions.el's writer and leaves its four readers behind."""
wk, wv = write
rk, rv = read
if rk == EXACT:
return rv == wv if wk == EXACT else rv.startswith(wv)
# read is a PREFIX: some key starting with rv is read
if wk == EXACT:
return wv.startswith(rv) # that one written key is in range
return rv.startswith(wv) # write namespace same-or-broader
# ── lexer ───────────────────────────────────────────────────────────────────
TOK_STR, TOK_IDENT, TOK_PUNCT, TOK_NUM = "str", "ident", "punct", "num"
IDENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
NUM_RE = re.compile(r"[0-9]+(\.[0-9]+)?")
class Tok:
__slots__ = ("kind", "val", "line")
def __init__(self, kind, val, line):
self.kind, self.val, self.line = kind, val, line
def __repr__(self):
return "%s(%r)@%d" % (self.kind, self.val, self.line)
def lex(src):
toks, i, n, line = [], 0, len(src), 1
while i < n:
c = src[i]
if c == "\n":
line += 1
i += 1
continue
if c in " \t\r":
i += 1
continue
if c == "/" and i + 1 < n and src[i + 1] == "/":
while i < n and src[i] != "\n":
i += 1
continue
if c == '"':
j, buf = i + 1, []
while j < n:
if src[j] == "\\" and j + 1 < n:
esc = src[j + 1]
buf.append({"n": "\n", "t": "\t", "r": "\r"}.get(esc, esc))
j += 2
continue
if src[j] == '"':
break
if src[j] == "\n":
line += 1
buf.append(src[j])
j += 1
toks.append(Tok(TOK_STR, "".join(buf), line))
i = j + 1
continue
m = IDENT_RE.match(src, i)
if m:
toks.append(Tok(TOK_IDENT, m.group(0), line))
i = m.end()
continue
m = NUM_RE.match(src, i)
if m:
toks.append(Tok(TOK_NUM, m.group(0), line))
i = m.end()
continue
toks.append(Tok(TOK_PUNCT, c, line))
i += 1
return toks
def match_close(toks, i, open_ch, close_ch):
"""toks[i] is open_ch; return index of its matching close_ch."""
depth = 0
while i < len(toks):
if toks[i].kind == TOK_PUNCT:
if toks[i].val == open_ch:
depth += 1
elif toks[i].val == close_ch:
depth -= 1
if depth == 0:
return i
i += 1
return len(toks) - 1
# ── program model ───────────────────────────────────────────────────────────
class Func:
def __init__(self, name, path, line, params, toks, start, end):
self.name, self.path, self.line = name, path, line
self.params = params # [param name]
self.toks = toks # the whole file's token list
self.start, self.end = start, end # body token range, exclusive of braces
self.lets = None # name -> [expr token ranges], lazily built
class Site:
def __init__(self, kind, path, line, func, arg_range, text):
self.kind = kind # "get" | "set"
self.path, self.line = path, line
self.func = func
self.arg_range = arg_range
self.text = text # source text of the key expression
self.pats = set()
self.unresolved = False
self.literal = None # set when the key expression is a bare literal
class Program:
def __init__(self):
self.files = {} # path -> toks
self.funcs = {} # name -> [Func] (El allows no overloads, but be safe)
self.toplevel = [] # [Func] one per file, params=[]
self.sites = [] # [Site]
self.calls = {} # callee name -> [(Func caller, [arg ranges])]
# -- loading ------------------------------------------------------------
def load(self, path, rel):
with open(path, "r", encoding="utf-8", errors="replace") as fh:
src = fh.read()
toks = lex(src)
self.files[rel] = toks
self._scan_funcs(rel, toks)
def _scan_funcs(self, rel, toks):
covered = []
i = 0
while i < len(toks):
t = toks[i]
if t.kind == TOK_IDENT and t.val == "fn" and i + 2 < len(toks) \
and toks[i + 1].kind == TOK_IDENT and toks[i + 2].val == "(":
name = toks[i + 1].val
pclose = match_close(toks, i + 2, "(", ")")
params = self._params(toks, i + 3, pclose)
bopen = pclose + 1
while bopen < len(toks) and toks[bopen].val != "{":
bopen += 1
bclose = match_close(toks, bopen, "{", "}")
f = Func(name, rel, t.line, params, toks, bopen + 1, bclose)
self.funcs.setdefault(name, []).append(f)
covered.append((i, bclose))
i = bclose + 1
continue
i += 1
# everything outside a fn is the file's top-level "function"
tl = Func("<toplevel:%s>" % rel, rel, 1, [], toks, 0, len(toks))
tl.covered = covered
self.toplevel.append(tl)
@staticmethod
def _params(toks, i, end):
"""`a: T, b: T` -> ['a','b'] (top-level commas only)."""
names, depth, expect = [], 0, True
while i < end:
t = toks[i]
if t.kind == TOK_PUNCT and t.val in "([{":
depth += 1
elif t.kind == TOK_PUNCT and t.val in ")]}":
depth -= 1
elif depth == 0 and t.kind == TOK_PUNCT and t.val == ",":
expect = True
elif depth == 0 and expect and t.kind == TOK_IDENT:
names.append(t.val)
expect = False
i += 1
return names
def func_at(self, rel, tok_index):
for f in self.funcs_in(rel):
if f.start <= tok_index < f.end:
return f
for f in self.toplevel:
if f.path == rel:
return f
return None
def funcs_in(self, rel):
for fl in self.funcs.values():
for f in fl:
if f.path == rel:
yield f
# -- indexing -----------------------------------------------------------
def index(self):
for rel, toks in self.files.items():
i = 0
while i < len(toks):
t = toks[i]
if t.kind == TOK_IDENT and i + 1 < len(toks) and toks[i + 1].val == "(" \
and t.val not in KEYWORDS \
and not (i > 0 and toks[i - 1].kind == TOK_IDENT
and toks[i - 1].val == "fn"):
# ^ the `fn f(a: T)` declaration is not a call site; counting
# it as one makes every parameter resolve to its own name
# and reports the whole function UNRESOLVED.
close = match_close(toks, i + 1, "(", ")")
args = split_args(toks, i + 2, close)
self.calls.setdefault(t.val, []).append(
(self.func_at(rel, i), args, rel, t.line))
if t.val in ("state_get", "state_set") and args:
self.sites.append(Site(
"get" if t.val == "state_get" else "set",
rel, t.line, self.func_at(rel, i), args[0],
render(toks, *args[0])))
i += 1
# -- resolution ---------------------------------------------------------
def lets_of(self, f):
if f.lets is not None:
return f.lets
f.lets = {}
toks = f.toks
skip = getattr(f, "covered", [])
i = f.start
while i < f.end:
if any(a <= i <= b for a, b in skip):
i = max(b for a, b in skip if a <= i <= b) + 1
continue
t = toks[i]
if t.kind == TOK_IDENT and t.val == "let" and i + 1 < f.end \
and toks[i + 1].kind == TOK_IDENT:
name = toks[i + 1].val
j = i + 2
if j < f.end and toks[j].val == ":": # skip the type
while j < f.end and toks[j].val != "=":
j += 1
if j < f.end and toks[j].val == "=":
s = j + 1
e = stmt_end(toks, s, f.end)
f.lets.setdefault(name, []).append((s, e))
i = e
continue
i += 1
return f.lets
def returns_of(self, ctx, depth=0, seen=None):
"""The value expressions of a function, in the context it was CALLED in.
Context-sensitive on purpose. `conv_hist_key` is written as a guard:
if str_eq(session_id, "") { return "conv_history" }
return "session_hist_" + session_id
Collecting both returns flat would make state_set(conv_hist_key("")) the
dead handle_chat() write claim to produce the session_hist_ namespace
too. That is a producer this engine does not actually have, and claiming
it would let the gate stay green if sessions.el's real writer vanished:
a masking hole in the exact namespace #129 lives in. So a guard whose
condition folds is honoured, and the branch not taken is dropped."""
out = []
self._values(ctx.toks, ctx.start, ctx.end, ctx, depth,
seen if seen is not None else set(), out)
return out
def _values(self, toks, s, e, ctx, depth, seen, out):
"""Append the value expressions of a statement sequence.
Returns True when the sequence definitely returns (rest unreachable)."""
if depth > MAX_DEPTH:
return False
i = s
while i < e:
t = toks[i]
if t.kind == TOK_IDENT and t.val == "return":
j = stmt_end(toks, i + 1, e)
if j > i + 1:
out.append((i + 1, j))
return True
if t.kind == TOK_IDENT and t.val == "let":
i = stmt_end(toks, i + 2, e)
continue
if t.kind == TOK_IDENT and t.val == "if":
i = self._if_stmt(toks, i, e, ctx, depth, seen, out)
if i is True:
return True
continue
if t.kind == TOK_PUNCT and t.val in "([{":
i = match_close(toks, i, t.val,
{"(": ")", "[": "]", "{": "}"}[t.val]) + 1
continue
en = stmt_end(toks, i, e)
if en <= i:
i += 1
continue
if en >= e: # trailing expression = the value
out.append((i, en))
i = en
return False
def _if_stmt(self, toks, i, e, ctx, depth, seen, out):
"""Walk one if / else-if / else chain. Returns the next index, or True
if the chain definitely returns on every reachable branch."""
bopen = i + 1
while bopen < e and toks[bopen].val != "{":
bopen += 1
if bopen >= e:
return e
bclose = match_close(toks, bopen, "{", "}")
fold = self._fold_cond(toks, i + 1, bopen, ctx, depth, seen)
j = bclose + 1
else_s = else_e = None
if j < e and toks[j].kind == TOK_IDENT and toks[j].val == "else":
if j + 1 < e and toks[j + 1].val == "{":
ec = match_close(toks, j + 1, "{", "}")
else_s, else_e = j + 2, ec
j = ec + 1
else: # `else if ...` — the rest of the chain
else_s = j + 1
else_e = stmt_end(toks, j + 1, e)
j = else_e
then_ret = else_ret = False
if fold is not False:
then_ret = self._values(toks, bopen + 1, bclose, ctx, depth + 1, seen, out)
if fold is not True and else_s is not None:
else_ret = self._values(toks, else_s, else_e, ctx, depth + 1, seen, out)
if fold is True and then_ret:
return True
if fold is False and else_s is not None and else_ret:
return True
if fold is None and else_s is not None and then_ret and else_ret:
return True
return j
def resolve(self, rng, func, depth=0, seen=None):
"""-> (set of patterns, unresolved_flag)"""
if seen is None:
seen = set()
if depth > MAX_DEPTH:
return set(), True
return self._expr(func.toks, rng[0], rng[1], func, depth, seen)
# -- expression walker --------------------------------------------------
def _expr(self, toks, s, e, func, depth, seen):
parts, cur, d = [], s, 0
i = s
while i < e: # split on top-level '+'
v = toks[i].val
if toks[i].kind == TOK_PUNCT and v in "([{":
d += 1
elif toks[i].kind == TOK_PUNCT and v in ")]}":
d -= 1
elif d == 0 and toks[i].kind == TOK_PUNCT and v == "+" and i > s:
parts.append((cur, i))
cur = i + 1
i += 1
parts.append((cur, e))
if len(parts) == 1:
return self._primary(toks, s, e, func, depth, seen)
# concatenation: keep folding while every operand so far is EXACT
head, unres = "", False
static = True
for (ps, pe) in parts:
pats, u = self._primary(toks, ps, pe, func, depth, seen)
exacts = {p[1] for p in pats if p[0] == EXACT}
if static and len(exacts) == 1 and not u and len(pats) == 1:
head += exacts.pop()
continue
if static and pats and all(p[0] == EXACT for p in pats) and len(pats) > 1:
# a branchy static operand: keep the shared head only
static = False
head += os.path.commonprefix(sorted({p[1] for p in pats}))
break
static = False
# first non-static operand: everything after it is runtime text
if (ps, pe) == parts[0]:
for p in pats:
if p[0] == PREFIX:
head = p[1]
break
if not head:
unres = True
break
if static:
return {pat_exact(head)}, False
p = pat_prefix(head)
return ({p} if p else set()), (unres or not p)
def _primary(self, toks, s, e, func, depth, seen):
while s < e and toks[s].kind == TOK_PUNCT and toks[s].val == "(" \
and match_close(toks, s, "(", ")") == e - 1:
s, e = s + 1, e - 1
if s >= e:
return set(), True
t = toks[s]
if t.kind == TOK_STR and e == s + 1:
return {pat_exact(t.val)}, False
if t.kind == TOK_IDENT and t.val == "if":
return self._if_expr(toks, s, e, func, depth, seen)
if t.kind == TOK_IDENT and s + 1 < e and toks[s + 1].val == "(":
close = match_close(toks, s + 1, "(", ")")
if close == e - 1:
return self._call(toks, t.val, split_args(toks, s + 2, close),
func, depth, seen)
if t.kind == TOK_IDENT and e == s + 1:
return self._var(t.val, func, depth, seen)
return set(), True
def _if_expr(self, toks, s, e, func, depth, seen):
bopen = s + 1
while bopen < e and toks[bopen].val != "{":
bopen += 1
cond = (s + 1, bopen)
bclose = match_close(toks, bopen, "{", "}")
then_rng = block_tail(toks, bopen + 1, bclose) or (bopen + 1, bclose)
else_rng = None
j = bclose + 1
if j < e and toks[j].kind == TOK_IDENT and toks[j].val == "else":
if j + 1 < e and toks[j + 1].val == "{":
ec = match_close(toks, j + 1, "{", "}")
else_rng = block_tail(toks, j + 2, ec) or (j + 2, ec)
else:
else_rng = (j + 1, e) # `else if ...`
taken = self._fold_cond(toks, cond[0], cond[1], func, depth, seen)
rngs = []
if taken is not False:
rngs.append(then_rng)
if taken is not True and else_rng:
rngs.append(else_rng)
pats, unres = set(), False
for r in rngs:
p, u = self._expr(toks, r[0], r[1], func, depth + 1, seen)
pats |= p
unres = unres or u
return pats, unres
def _fold_cond(self, toks, s, e, func, depth, seen):
"""Constant-fold `str_eq(X, "")` / `!str_eq(X, "")` so a helper called with
a literal (conv_hist_key("")) yields only the branch it really takes.
Returns True / False / None(unknown)."""
neg = False
if s < e and toks[s].kind == TOK_PUNCT and toks[s].val == "!":
neg, s = True, s + 1
if not (s < e and toks[s].kind == TOK_IDENT and toks[s].val == "str_eq"
and s + 1 < e and toks[s + 1].val == "("):
return None
close = match_close(toks, s + 1, "(", ")")
if close != e - 1:
return None
args = split_args(toks, s + 2, close)
if len(args) != 2:
return None
va, ua = self._expr(toks, args[0][0], args[0][1], func, depth + 1, seen)
vb, ub = self._expr(toks, args[1][0], args[1][1], func, depth + 1, seen)
if ua or ub or len(va) != 1 or len(vb) != 1:
return None
(ka, sa), (kb, sb) = va.pop(), vb.pop()
if ka != EXACT or kb != EXACT:
return None
r = (sa == sb)
return (not r) if neg else r
def _call(self, toks, name, args, func, depth, seen):
cands = self.funcs.get(name)
if not cands:
return set(), True # builtin: json_get, env, ...
pats, unres = set(), False
for callee in cands:
key = ("fn", callee.path, callee.name, tuple(args))
if key in seen:
unres = True
continue
seen = seen | {key}
# bind the callee's params to THIS call site's argument expressions
binding = {}
for idx, pname in enumerate(callee.params):
if idx < len(args):
binding[pname] = (args[idx], func)
callee_ctx = _Bound(callee, binding)
for r in self.returns_of(callee_ctx, depth + 1, seen):
p, u = self._expr(callee.toks, r[0], r[1], callee_ctx,
depth + 1, seen)
pats |= p
unres = unres or u
return pats, unres
def _var(self, name, func, depth, seen):
real = func.func if isinstance(func, _Bound) else func
# 1. a parameter bound by the call site we came through
if isinstance(func, _Bound) and name in func.binding:
rng, caller_ctx = func.binding[name]
return self._expr(caller_ctx.toks, rng[0], rng[1], caller_ctx,
depth + 1, seen)
# 2. a local `let` in the enclosing function
lets = self.lets_of(real)
if name in lets:
key = ("let", real.path, real.name, name)
if key in seen:
return set(), True
seen = seen | {key}
pats, unres = set(), False
for rng in lets[name]:
p, u = self._expr(real.toks, rng[0], rng[1], real, depth + 1, seen)
pats |= p
unres = unres or u
return pats, unres
# 3. an unbound parameter -> look at every call site of the enclosing fn
if name in real.params:
key = ("param", real.path, real.name, name)
if key in seen:
return set(), True
seen = seen | {key}
idx = real.params.index(name)
pats, unres = set(), False
sites = self.calls.get(real.name, [])
if not sites:
return set(), True
for caller, args, _rel, _line in sites:
if caller is None or idx >= len(args):
unres = True
continue
p, u = self._expr(caller.toks, args[idx][0], args[idx][1],
caller, depth + 1, seen)
pats |= p
unres = unres or u
return pats, unres
# 4. a file-level / cross-file top-level `let`
for tl in self.toplevel:
lets = self.lets_of(tl)
if name in lets:
key = ("let", tl.path, tl.name, name)
if key in seen:
return set(), True
seen2 = seen | {key}
pats, unres = set(), False
for rng in lets[name]:
p, u = self._expr(tl.toks, rng[0], rng[1], tl, depth + 1, seen2)
pats |= p
unres = unres or u
return pats, unres
return set(), True
class _Bound:
"""A callee view that also knows what its params were called with."""
def __init__(self, func, binding):
self.func, self.binding = func, binding
self.toks, self.start, self.end = func.toks, func.start, func.end
self.params, self.path, self.name = func.params, func.path, func.name
def __getattr__(self, k):
return getattr(self.func, k)
# ── token helpers ───────────────────────────────────────────────────────────
def split_args(toks, s, e):
out, cur, d = [], s, 0
i = s
while i < e:
v = toks[i].val
if toks[i].kind == TOK_PUNCT and v in "([{":
d += 1
elif toks[i].kind == TOK_PUNCT and v in ")]}":
d -= 1
elif d == 0 and toks[i].kind == TOK_PUNCT and v == ",":
out.append((cur, i))
cur = i + 1
i += 1
if cur < e:
out.append((cur, e))
return out
STMT_START = {"let", "return", "if", "while", "for"}
KEYWORDS = {"if", "while", "for", "return", "fn", "let", "else", "match"}
def stmt_end(toks, s, limit):
"""End of the expression starting at s: the next top-level statement
boundary. El has no semicolons, so a newline that starts a new statement
ends this one."""
d, i = 0, s
while i < limit:
t = toks[i]
if t.kind == TOK_PUNCT and t.val in "([":
d += 1
elif t.kind == TOK_PUNCT and t.val in ")]":
d -= 1
if d < 0:
return i
elif t.kind == TOK_PUNCT and t.val == "{":
# a brace at depth 0 belongs to this expression only when it is an
# if/else block that is part of it
d += 1
elif t.kind == TOK_PUNCT and t.val == "}":
d -= 1
if d < 0:
return i
elif d == 0 and t.kind == TOK_PUNCT and t.val == ",":
return i
elif d == 0 and i > s and t.kind == TOK_IDENT and t.val in STMT_START:
if t.val == "if" and toks[i - 1].kind == TOK_IDENT and toks[i - 1].val == "else":
i += 1
continue
return i
elif d == 0 and i > s and t.kind == TOK_IDENT and t.val == "fn":
return i
i += 1
return limit
def block_tail(toks, s, e):
"""The trailing expression of a block, if the block ends in one."""
i, last = s, None
while i < e:
t = toks[i]
if t.kind == TOK_IDENT and t.val in ("let", "return"):
i = stmt_end(toks, i + 1, e)
last = None
continue
if t.kind == TOK_PUNCT and t.val in "([{":
i = match_close(toks, i, t.val, {"(": ")", "[": "]", "{": "}"}[t.val]) + 1
continue
st = i
en = stmt_end(toks, i, e)
if en <= st:
i = st + 1
continue
last = (st, en)
i = en
return last
def render(toks, s, e):
out = []
for t in toks[s:e]:
out.append('"%s"' % t.val if t.kind == TOK_STR else t.val)
return " ".join(out)
# ── the gate ────────────────────────────────────────────────────────────────
def collect(root, include_tests):
files = []
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = [d for d in dirnames
if d not in ("dist", "vendor", ".git", "node_modules")]
rel_dir = os.path.relpath(dirpath, root)
if not include_tests and rel_dir.split(os.sep)[0] == "tests":
continue
for fn in sorted(filenames):
if fn.endswith(".el"):
rel = os.path.normpath(os.path.join(rel_dir, fn))
files.append((os.path.join(dirpath, fn), rel))
return sorted(files, key=lambda x: x[1])
def is_bare_literal(prog, site):
toks = prog.files[site.path]
s, e = site.arg_range
return e == s + 1 and toks[s].kind == TOK_STR
def read_decl(path):
"""A declaration file: one entry per line, `# ...` comments stripped."""
out = []
if not path or not os.path.exists(path):
return out
with open(path) as fh:
for ln in fh:
ln = ln.split("#", 1)[0].strip()
if ln:
out.append(ln)
return out
def opt(argv, name, default=None):
for i, a in enumerate(argv):
if a == name and i + 1 < len(argv):
return argv[i + 1]
return default
def main(argv):
root = os.path.abspath(argv[1]) if len(argv) > 1 and not argv[1].startswith("-") else "."
include_tests = "--include-tests" in argv
verbose = "--verbose" in argv
baseline_path = opt(argv, "--baseline")
external_path = opt(argv, "--external")
prog = Program()
for path, rel in collect(root, include_tests):
prog.load(path, rel)
prog.index()
for site in prog.sites:
pats, unres = prog.resolve(site.arg_range, site.func)
site.pats, site.unresolved = {p for p in pats if p}, unres
if is_bare_literal(prog, site):
site.literal = prog.files[site.path][site.arg_range[0]].val
writes = [s for s in prog.sites if s.kind == "set"]
reads = [s for s in prog.sites if s.kind == "get"]
write_pats = set()
for w in writes:
write_pats |= w.pats
# Declared host-set keys: written by something outside the El tree (an
# operator, the installer, a host process). Each entry must carry a reason.
external = []
for ln in read_decl(external_path):
parts = ln.split(None, 1)
if len(parts) != 2 or parts[0] not in (EXACT, PREFIX):
print("bad --external line (want `exact|prefix <key>`): %r" % ln,
file=sys.stderr)
return 2
external.append((parts[0], parts[1]))
write_pats |= set(external)
# F1 — a read of a key no write in the tree produces.
f1 = []
for r in reads:
for p in sorted(r.pats):
if not any(covers(w, p) for w in write_pats):
f1.append((r, p))
# F2 — a key namespace owned by a helper, accessed by a hand-rolled literal.
# This is the #129 shape: the producer moved behind conv_hist_key() and
# one consumer kept spelling the old key out by hand.
owners = {} # helper fn name -> its value set
for s in prog.sites:
toks = prog.files[s.path]
a, b = s.arg_range
if toks[a].kind == TOK_IDENT and a + 1 < b and toks[a + 1].val == "(" \
and match_close(toks, a + 1, "(", ")") == b - 1 \
and toks[a].val in prog.funcs:
name = toks[a].val
if name not in owners:
vals = set()
for callee in prog.funcs[name]:
# No call context here on purpose: the OWNED namespace is
# every key the helper can ever produce, over all call sites.
for rng in prog.returns_of(callee):
p, _ = prog._expr(callee.toks, rng[0], rng[1], callee, 0, set())
vals |= {x for x in p if x}
owners[name] = vals
f2 = []
for s in prog.sites:
if s.literal is None:
continue
for owner, vals in sorted(owners.items()):
for v in sorted(vals):
if covers(v, pat_exact(s.literal)):
f2.append((s, owner, v))
break
else:
continue
break
unresolved = [s for s in prog.sites if s.unresolved or not s.pats]
# Baseline signatures carry NO line number on purpose: an unrelated edit that
# shifts a line must not un-mute an accepted finding (that is crying wolf),
# but a GROWTH in count must not hide either. So a baseline entry is
# `<file> <CODE> <detail> [xN]` and only the first N matches are muted.
baseline, bad_baseline = {}, []
for ln in read_decl(baseline_path):
n, key = 1, ln
parts = ln.rsplit(" x", 1)
if len(parts) == 2 and parts[1].isdigit():
key, n = parts[0].strip(), int(parts[1])
baseline[key] = n
def sig(path, code, detail):
return "%s %s %s" % (path, code, detail)
findings = []
for r, p in f1:
findings.append((sig(r.path, "DEAD-READ", "%s:%s" % p), r.line,
" %s:%d state_get(%s)\n resolves to %s %r — no state_set in the tree produces it"
% (r.path, r.line, r.text, p[0].upper(), p[1])))
for s, owner, v in f2:
findings.append((sig(s.path, "HAND-ROLLED", "%s<-%s()" % (s.literal, owner)), s.line,
" %s:%d state_%s(\"%s\")\n %s() owns this key namespace (%s %r) — go through the helper, "
"or a rename orphans this site silently" % (s.path, s.line, s.kind, s.literal, owner, v[0].upper(), v[1])))
findings.sort(key=lambda f: (f[0], f[1]))
live, muted, budget = [], [], dict(baseline)
for f in findings:
if budget.get(f[0], 0) > 0:
budget[f[0]] -= 1
muted.append(f)
else:
live.append(f)
stale = sorted(k for k, v in budget.items() if v > 0)
print("── state-key audit ─────────────────────────────────────────────")
print("scanned %d .el files%s" % (len(prog.files),
"" if include_tests else " (tests/ excluded)"))
print("sites %d state_set, %d state_get" % (len(writes), len(reads)))
print("keys %d distinct write patterns" % len(write_pats))
print("")
if verbose:
print("WRITE PATTERNS")
for k, v in sorted(write_pats):
print(" %-6s %s" % (k, v))
print("")
if external:
print("DECLARED HOST-SET (%d) — %s" % (len(external), external_path))
for k, v in sorted(external):
print(" %-6s %s" % (k, v))
print("")
print("UNRESOLVED (%d) — reported, never fails the build" % len(unresolved))
if not unresolved:
print(" (none)")
for s in sorted(unresolved, key=lambda x: (x.path, x.line)):
print(" %s:%d state_%s(%s)%s"
% (s.path, s.line, s.kind, s.text,
" [partial: %s]" % ", ".join("%s %r" % p for p in sorted(s.pats))
if s.pats else ""))
print("")
if muted:
print("BASELINED (%d) — pre-existing debt accepted in %s. NOT clean; fix these."
% (len(muted), baseline_path))
for sg, line, _ in muted:
print(" %s (line %d)" % (sg, line))
print("")
if stale:
print("STALE BASELINE (%d) — entries that no longer match anything; delete them:"
% len(stale))
for sg in stale:
print(" %s" % sg)
print("")
print("FINDINGS (%d)" % len(live))
if not live:
print(" (none)")
for _, _, body in live:
print(body)
print("")
if live:
print("FAIL: %d state-key finding(s). See scripts/verify-state-keys.sh "
"for why this gate exists (issue #129)." % len(live))
return 1
print("PASS: every resolvable state_get key has a producer, and no key "
"namespace is spelled two ways.")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
+28
View File
@@ -0,0 +1,28 @@
# state-key-baseline.txt — findings that already existed when this gate landed
# (2026-08-07). Each one is a REAL defect of the #129 class, not a false
# positive. They are muted only so the gate can be turned on today instead of
# being deferred until the debt is paid; every run still prints them under
# BASELINED with the word "debt".
#
# THIS FILE SHOULD ONLY EVER SHRINK. Adding a line means you are shipping a
# known silent-"" read. If you must, date it and say why in the comment.
#
# format: <file> <CODE> <detail> [xN] # N = how many sites are accepted
# No line numbers on purpose: an unrelated edit must not un-mute an accepted
# finding, but a GROWTH in count is NOT muted — the extra site fails the build.
#
chat.el DEAD-READ exact:soul_identity x5
# ^ soul.el used to run `state_set("soul_identity", soul_identity)`. It was
# deleted on 2026-05-13 in b163fa6 ("feat(awareness): route ISE writes to HTTP
# Engram ..."), a commit about something else entirely, and the five readers in
# chat.el were left behind. Since that date build_system_prompt (737), the
# vision handler (1745), the agentic system prompt (2620), the council
# transcript handler (3425) and 3480 have all been prefixing "" — exactly the
# #129 shape, found by this gate on its first run. Sites: 737, 1745, 2620,
# 3425, 3480. Fix = restore the boot-time write or delete the reads; not done
# here because this branch must not change engine behaviour.
studio.el DEAD-READ exact:soul_principal x1
# ^ studio.el:57 dharma_registry() emits "principal":"" on every call — no
# producer has ever existed in the tree's history (git log -S finds none).
# Never-wired rather than orphaned, same silent-"" result.
+16
View File
@@ -0,0 +1,16 @@
# state-key-external.txt — state keys the engine READS but deliberately never
# WRITES, because a host outside the El tree sets them (an operator, the
# installer, a deployment env). Read scripts/verify-state-keys.sh for why this
# list has to exist and why it has to stay short.
#
# THE RULE FOR ADDING A LINE: the read site must already treat "" as a defined
# default (`if str_eq(x, "") { <default> }`) AND the source must say so in a
# comment. "I could not find the writer" is NOT a reason — that is the #129
# defect, and it belongs in state-key-baseline.txt with a date, not here.
#
# format: exact|prefix <key> # why, and where the source says so
#
exact soul_rate_limit # routes.el:59-61 — "configurable via soul state key ... Falls back to 60 req/min if not set."
exact web_search_tool_version # chat.el:1884-1910 — version lives in state "so a future bump is a config write, not a recompile"; defaults to web_search_20250305
exact platform_auth # stewardship.el:92 — host-set capability flag; fail-CLOSED (anything but "true" denies the platform tool)
exact security_research_authorized # awareness.el:991-996 — state override for env SECURITY_RESEARCH_TOKEN; fail-closed, defaults false
+118
View File
@@ -0,0 +1,118 @@
#!/usr/bin/env bash
# verify-state-keys.sh — the state-key gate. Retires a defect class at build time.
#
# ── WHY THIS EXISTS. DO NOT DELETE IT AS NOISE. ──────────────────────────────
#
# The engine keeps runtime values in a key-value store: state_set("k", v) writes,
# state_get("k") reads. A read of a key that NOTHING writes returns an empty
# string. Silently. No error, no warning, no log line. The El compiler cannot see
# it, no test sees it, and the product keeps running — just with a hole in it.
#
# That is how issue #129 happened. ff421d3 (2026-08-05) correctly moved
# conversation history to a per-session key behind conv_hist_key(session_id). One
# consumer did not move with it: the agentic path's L1 safety screen kept reading
# the old anonymous "conv_history" bucket. The desktop app always mints a session
# id, so history was always written under session_hist_<id> and that read always
# returned "". The half of the crisis score that receives history is the
# ESCALATION half — the one that exists for distress building across several
# turns, where no single message trips the bell on its own. It scored 0 on every
# real conversation for two days, and nothing failed.
#
# The line that broke carried a comment describing this exact bug being fixed
# once already, under issue #9. A comment is not a gate. This is the gate.
#
# ── WHAT IT CHECKS ──────────────────────────────────────────────────────────
#
# DEAD-READ a state_get whose key resolves to something no state_set in the
# tree produces. The direct form of the class.
#
# HAND-ROLLED a state_get/state_set that spells out a literal belonging to a
# key namespace a helper function owns (e.g. "conv_history", owned
# by conv_hist_key()). This is #129's actual shape: the producer
# moved behind the helper and one consumer kept the old spelling
# by hand. DEAD-READ alone does NOT catch #129, because the dead
# handle_chat() still writes that key through the helper — so this
# second check is the one that earns the gate its keep.
#
# ── WHY IT DOES NOT CRY WOLF ────────────────────────────────────────────────
#
# Keys are usually COMPUTED, not literal, so a naive grep would flood and get
# switched off within a day. scripts/state-key-audit.py resolves computed keys:
# string concatenation (matched on the static prefix), helper functions (resolved
# to their possible return values), keys built into a local variable, and keys
# arriving as a function parameter (resolved through the call sites). Where a key
# genuinely cannot be resolved it is printed under UNRESOLVED and does NOT fail
# the build — visible, never silently ignored. Keep that list short.
#
# On this tree it resolves 278 of 278 sites: UNRESOLVED is 0 and FINDINGS is 0.
#
# Two declaration files, both of which should only ever shrink:
# scripts/state-key-external.txt keys a host outside the El tree writes
# scripts/state-key-baseline.txt findings that predate the gate (real debt)
#
# ── PROVEN TO DISCRIMINATE (2026-08-07) ─────────────────────────────────────
#
# 1. Synthetic: a scratch copy of this tree with agentic_safety_screen reverted
# to the pre-fix state_get("conv_history") — ONE line, nothing else — FAILS
# with `chat.el:2536 ... conv_hist_key() owns this key namespace`. The tree
# as shipped PASSES. One variable, opposite verdicts.
# 2. Independent: run read-only against origin/feat/soul-openai-tools-v2, which
# carries the same defect on its own, the gate reported chat.el:2937 — the
# exact line 43d0449's commit message had named by hand. Against that
# branch's fix (origin/fix/129-on-openai-tools) it passes.
# 3. Producer-moved controls: renaming the sole writer of an EXACT key
# (soul_model) orphans 3 readers across 3 files; renaming the sole writer of
# a PREFIX namespace (agent_workspace_root_*) orphans 3 readers — including
# when the producer moves to a NARROWER namespace, which an earlier,
# sloppier prefix rule let through.
#
# It also found, on its first run, a defect nobody was looking for: soul.el's
# `state_set("soul_identity", ...)` was deleted on 2026-05-13 in b163fa6 (a
# commit about awareness/ISE writes) and five readers in chat.el were left
# behind — the system prompt, the vision handler, the agentic prompt and the
# council handler have been prefixing "" ever since. See state-key-baseline.txt.
#
# ── SAFETY ──────────────────────────────────────────────────────────────────
# Pure static read of .el sources. Starts nothing, opens no port, touches no
# daemon, and never reads or writes ~/.neuron.
#
# ── USAGE ───────────────────────────────────────────────────────────────────
# scripts/verify-state-keys.sh gate the repo (honours baseline)
# scripts/verify-state-keys.sh --strict ignore the baseline: show the debt
# scripts/verify-state-keys.sh --verbose also dump every write pattern
# scripts/verify-state-keys.sh --root DIR audit a different tree
# exit 0 = clean; 1 = finding(s); 2 = the gate itself could not run.
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
STRICT=0
PASS_THROUGH=()
while [ $# -gt 0 ]; do
case "$1" in
--strict) STRICT=1; shift ;;
--root) ROOT="${2:?--root needs a directory}"; shift 2 ;;
-h|--help) awk 'NR>1 && /^#/ {print; next} NR>1 {exit}' "${BASH_SOURCE[0]}"; exit 0 ;;
*) PASS_THROUGH+=("$1"); shift ;;
esac
done
command -v python3 >/dev/null 2>&1 || {
echo "[state-keys] CANNOT RUN: python3 not found" >&2; exit 2; }
[ -d "$ROOT" ] || { echo "[state-keys] CANNOT RUN: no such tree: $ROOT" >&2; exit 2; }
AUDIT="$SCRIPT_DIR/state-key-audit.py"
[ -f "$AUDIT" ] || { echo "[state-keys] CANNOT RUN: missing $AUDIT" >&2; exit 2; }
ARGS=("$ROOT" "--external" "$SCRIPT_DIR/state-key-external.txt")
[ "$STRICT" -eq 0 ] && ARGS+=("--baseline" "$SCRIPT_DIR/state-key-baseline.txt")
[ ${#PASS_THROUGH[@]} -gt 0 ] && ARGS+=("${PASS_THROUGH[@]}")
python3 "$AUDIT" "${ARGS[@]}"
RC=$?
if [ "$RC" -gt 1 ]; then
echo "[state-keys] CANNOT RUN: the audit itself failed (exit $RC)" >&2
exit 2
fi
exit "$RC"
+35 -7
View File
@@ -379,9 +379,23 @@ fn emit_session_start_event() -> Void {
// layered_cycle routes user-facing requests through the 4-layer consciousness stack.
// L0 (core) L1 (safety screen) L2a (continuity + behavioral profiling) L2b (mission alignment) L3 (imprint) L1 (safety validate)
// Internal cognition (heartbeat, proactive, memory ops) bypasses layers use one_cycle directly.
fn layered_cycle(raw_input: String) -> String {
let history: String = state_get("conv_history")
let session_id: String = state_get("current_session_id")
//
// FIX B (2026-08-05) the cycle now knows which conversation it is in.
//
// session_id: the caller's session, threaded from the route. Was previously read from the
// state key "current_session_id", which is read HERE and written NOWHERE in the entire
// source verified across every .el file. So this value was unconditionally "", and every
// downstream consumer of it silently fell back to a process-global bucket: conversation
// history, and the steward's continuity tracking (TODO reliability #4, below, describes the
// cross-session bleed this caused; threading the real id closes it). The plain path's blank
// stare and the agentic path's scoped history were the same defect seen from two sides.
//
// utility: true when the generation is not part of the user's conversation the app's
// title and insight passes. Answered normally, never recorded. See is_utility_request.
fn layered_cycle(raw_input: String, session_id: String, utility: Bool) -> String {
// Safety-screen history amplification now reads the SAME window the turn will be
// recorded into, so a session's own escalation pattern is what gets scored.
let history: String = state_get(conv_hist_key(session_id))
// L1 in: safety screen
let screen_result: String = safety_screen(raw_input, history)
@@ -423,8 +437,10 @@ fn layered_cycle(raw_input: String) -> String {
let cont_action: String = json_get(continuity, "action")
// Store continuity status so imprint can adjust its response register.
// TODO(reliability #4): session_continuity is process-global; scope per session_id
// when available to prevent cross-session bleed under concurrent layered_cycle calls.
// TODO(reliability #4) CLOSED 2026-08-05: this line was already written to scope per
// session it just never received a session id, because the only source was a state key
// nothing wrote. It is now threaded from the route, so named sessions genuinely get their
// own continuity state and only anonymous callers share the global one.
let cont_key: String = if str_eq(session_id, "") { "session_continuity" } else { "session_continuity:" + session_id }
state_set(cont_key, cont_status)
@@ -499,7 +515,7 @@ fn layered_cycle(raw_input: String) -> String {
// screen, the safe-mode guard, the hard-bell short-circuit and the L2 stewardship layers,
// and strictly BEFORE the L1 output gate. A hard bell never reaches a model the branch
// above returns first. Tools are not offered on this turn; see layered_generate.
let output: String = layered_generate(prompt, imprint_id)
let output: String = layered_generate(prompt, imprint_id, session_id)
// L1 out: validate output before delivery. Still the terminal gate nothing below this
// line can change the string this function returns.
@@ -509,7 +525,19 @@ fn layered_cycle(raw_input: String) -> String {
// reachable on the non-bell path: both bell branches above return before this point, so
// bell turns still never enter conversation history. Pure state side effect it cannot
// alter what is returned.
conv_history_record(raw_input, validated)
//
// FIX A: the receipt is unconditional and always negative on this path, because on this
// path it is structurally true layered_generate offers no tools at all (build_system_prompt
// chat mode + a request body with no "tools" key). Recording "no tools ran" is not padding:
// it is the only thing that distinguishes "nothing ran" from "we forgot to write down what
// ran", and that ambiguity is what made the model confess to a search it had performed.
//
// FIX E1: a utility generation is answered but not recorded. Guarded here rather than at
// the route so every /api/chat dispatch site inherits it from one place.
let receipt: String = tool_receipt("", "")
if !utility {
conv_history_record(session_id, raw_input, validated, receipt)
}
return validated
}
+3 -1
View File
@@ -1,6 +1,8 @@
// auto-generated by elc --emit-header - do not edit
extern fn init_soul_edges() -> Void
extern fn ensure_self_canonical_bridge() -> Void
extern fn aff_try_slot(slot_json: String, aff_7d_ts: Int, acc_key: String) -> Void
extern fn load_identity_context() -> Void
extern fn seed_persona_from_env() -> Void
extern fn emit_session_start_event() -> Void
extern fn layered_cycle(raw_input: String) -> String
extern fn layered_cycle(raw_input: String, session_id: String, utility: Bool) -> String
+213
View File
@@ -0,0 +1,213 @@
// test_history_amplification.el
//
// REGRESSION TEST FOR ISSUE #129 (P0, SAFETY).
//
// What this guards: on the agentic path, the crisis score has two halves the
// message you just sent, and the distress that has accumulated across the
// conversation. The second half is the whole reason the escalation logic exists:
// someone whose distress builds over several turns never sends one message that
// trips the bell on its own.
//
// The defect this test was written against (ff421d3, 2026-08-05 fixed
// 2026-08-07): conversation history moved to a per-session key via
// conv_hist_key(session_id), but the agentic path's safety screen was left
// reading the old anonymous "conv_history" bucket. The desktop app always sends
// a session_id, so the screen received "" on every real conversation and the
// escalation half always scored 0. Nothing failed. Nothing logged. The comment
// above the defective line documented this same bug being fixed once before.
//
// THE INVARIANT UNDER TEST, stated so it survives future renames:
// the window the safety screen READS must be the window conv_history_record
// WRITES. Not "must be called conv_history" must AGREE.
//
// This test is deliberately written to fail loudly on the pre-fix source. If it
// ever passes on code where the screen reads a key nothing writes, it is broken.
//
// To run (macOS, from the worktree root):
// scripts/run-el-test.sh tests/test_history_amplification.el
//
import "../chat.el"
import "../safety.el"
import "../sessions.el"
// Program class. Without this an El program compiles as a 'utility', and a
// utility may not call the self-formation primitives (llm_call_system,
// llm_vision) that chat.el's agentic loop references the unit fails to
// compile with a capability violation even though the test never calls them.
// Declaring 'cgi' matches how soul.el declares itself.
//
// The endpoints below are deliberately DEAD: this test must never reach a live
// engram, and nothing it asserts depends on one. Port 9 is discard.
cgi "neuron-test-history-amplification" {
dharma_id: "ntn-test@http://127.0.0.1:9",
principal: "test-harness",
network: "dharma-testnet",
engram: "http://127.0.0.1:9"
}
// Counters
//
// NOTE for anyone copying this harness: the idiom used by the older tests in
// this directory `let pass_count = pass_count + 1` inside an assert function
// does NOT mutate the module-level binding. It declares a new local that dies
// with the call, so those suites all print "0 passed, 0 failed" no matter what
// happened. Counters go through the state store here so the summary is real.
fn bump(counter: String) -> Void {
let cur: String = state_get(counter)
let n: Int = if str_eq(cur, "") { 0 } else { str_to_int(cur) }
state_set(counter, int_to_str(n + 1))
}
fn counter_of(counter: String) -> Int {
let cur: String = state_get(counter)
if str_eq(cur, "") { return 0 }
return str_to_int(cur)
}
fn assert_eq(label: String, got: String, expected: String) -> Void {
if str_eq(got, expected) {
bump("t_pass")
println(" PASS: " + label)
} else {
bump("t_fail")
println(" FAIL: " + label)
println(" got: " + got)
println(" expected: " + expected)
}
}
fn assert_true(label: String, cond: Bool) -> Void {
if cond {
bump("t_pass")
println(" PASS: " + label)
} else {
bump("t_fail")
println(" FAIL: " + label)
}
}
// Fixtures
//
// ESCALATION_MSG scores 50 on its own ("can't go on", safety_score_crisis s9).
// 50 is >= soft (35) and < hard (70): a soft bell by itself, which is the point.
// It is the message that is only a crisis IN CONTEXT.
//
// DISTRESS_TURN carries five history phrases:
// hopeless 15 + worthless 15 + nobody cares 15 + can't take it anymore 20
// + want to disappear 20 = 85; contributed at 1/3 weight = 28.
// 50 + 28 = 78 >= 70 hard_bell. That 28-point swing IS the escalation logic,
// and it is exactly what the defect suppressed.
let TEST_SESSION: String = "sess-test-129-history-amplification"
let ESCALATION_MSG: String = "honestly i can't go on like this"
let DISTRESS_TURN: String = "i feel hopeless and worthless, nobody cares, i can't take it anymore, i want to disappear"
println("")
println("test_history_amplification.el — issue #129 (agentic multi-turn crisis escalation)")
// 1. Baseline: the message alone is a SOFT bell, not a hard one
//
// If this ever returns hard_bell, the test below proves nothing the message
// would trip the bell without any history and the amplification would be
// invisible. This assertion is what keeps the real test honest.
println("")
println("1. baseline — escalation message with NO history is a soft bell")
let baseline: String = safety_screen(ESCALATION_MSG, "")
assert_eq("no history -> soft_bell (not hard)", json_get(baseline, "action"), "soft_bell")
// 2. Producer sanity: history lands in the session's own window
println("")
println("2. producer — conv_history_record writes the session's window")
conv_history_record(TEST_SESSION, DISTRESS_TURN, "i hear you, that sounds heavy", "")
let written: String = state_get(conv_hist_key(TEST_SESSION))
assert_true("session window is non-empty after record", !str_eq(written, ""))
assert_true("session window contains the distress turn", str_contains(written, "hopeless"))
// 3. THE REGRESSION: the agentic screen must SEE that window
//
// Pre-fix this returns soft_bell, because agentic_safety_screen read the
// anonymous bucket and got "". Post-fix it returns hard_bell.
println("")
println("3. REGRESSION #129 — agentic screen reads the session's own window")
let screened: String = agentic_safety_screen(TEST_SESSION, ESCALATION_MSG)
assert_eq(
"distress history escalates the agentic screen to hard_bell",
json_get(screened, "action"),
"hard_bell"
)
// 4. The invariant, stated directly
//
// Independent of thresholds and phrase lists: whatever the screen reads for a
// session must equal what the recorder wrote for that session. This is the
// assertion that survives a future rename of either side.
println("")
println("4. invariant — read window == written window")
let read_back: String = state_get(conv_hist_key(TEST_SESSION))
assert_true("screen input is the recorded window, not empty", !str_eq(read_back, ""))
assert_eq("read window is byte-identical to written window", read_back, written)
// 5. No false positive: a calm session does not escalate
//
// A test that only ever asserts "hard_bell" would pass on code that hard-bells
// every message. This is the other leg, and it runs BEFORE the anonymous case
// below on purpose: that case writes the shared bucket, and under the defect a
// calm session would then inherit it.
println("")
println("5. specificity — a calm history does NOT escalate")
let CALM_SESSION: String = "sess-test-129-calm"
state_set("conv_history", "")
conv_history_record(CALM_SESSION, "what is the weather like today", "clear and mild", "")
let calm: String = agentic_safety_screen(CALM_SESSION, ESCALATION_MSG)
assert_eq("calm history stays at soft_bell", json_get(calm, "action"), "soft_bell")
// 6. Cross-session leakage
//
// The same defect had a second face: because the screen read one shared bucket,
// a calm session could be scored against a DIFFERENT session's distress. That is
// wrong in both directions it fabricates a crisis for the calm user and it
// leaks the distressed user's content into another session's scoring.
println("")
println("6. isolation — one session's distress must not score another session")
state_set("conv_history", "")
let OTHER_SESSION: String = "sess-test-129-other"
conv_history_record(OTHER_SESSION, DISTRESS_TURN, "i hear you", "")
let isolated: String = agentic_safety_screen(CALM_SESSION, ESCALATION_MSG)
assert_eq(
"a distressed OTHER session does not escalate the calm session",
json_get(isolated, "action"),
"soft_bell"
)
// 7. Anonymous sessions still work
//
// conv_hist_key("") deliberately falls back to the shared "conv_history" bucket.
// The fix must not break the no-session_id path older callers rely on. Runs last
// because it writes that shared bucket.
println("")
println("7. anonymous path — empty session_id still screens against the shared window")
state_set("conv_history", "[{\"role\":\"user\",\"content\":\"" + DISTRESS_TURN + "\"}]")
let anon: String = agentic_safety_screen("", ESCALATION_MSG)
assert_eq("anonymous session escalates too", json_get(anon, "action"), "hard_bell")
// Summary
println("")
println("history amplification tests: " + int_to_str(counter_of("t_pass")) + " passed, " + int_to_str(counter_of("t_fail")) + " failed")
+97 -11
View File
@@ -41,6 +41,7 @@
#include <fcntl.h>
#include <dirent.h>
#include <errno.h>
#include <signal.h> /* SIGPIPE disposition — see el_runtime_ignore_sigpipe */
#include <pthread.h>
#include <curl/curl.h>
@@ -1238,16 +1239,77 @@ static const char* http_reason_phrase(int status) {
}
}
/* Best-effort send with retry on partial writes. */
/* ── A departing client MUST NOT be able to kill the daemon ──────────────────
* (2026-08-06, round 9.1 / ADR 0006 item 4.)
*
* Measured field failure: a client cancelled its request at 25 s; the handler
* finished its work at 116.9 s and wrote the reply into the departed client's
* socket. The second send() on a reset connection raised SIGPIPE, whose DEFAULT
* disposition terminates the process `exited due to SIGPIPE ... ran for
* 361177ms`. launchd respawned 4 ms later, so EVERY other in-flight request on
* that daemon lost its work, silently.
*
* Two independent guards, because one of them can be undone from outside this
* file (an embedder may reset signal dispositions) and the other cannot:
* 1. process-wide SIGPIPE -> SIG_IGN, installed at runtime init;
* 2. per-send suppression at the syscall (MSG_NOSIGNAL where the platform has
* it, SO_NOSIGPIPE on the accepted socket on macOS/BSD).
* With either in force, send() reports the peer's departure as EPIPE and the
* caller decides which is the point: this is an ordinary I/O outcome, not a
* fatal condition.
*
* It deliberately does NOT swallow the error. http_send_response() below
* classifies the errno and logs: "client left" for a departure, and a real
* "send failed: <strerror>" for anything else, so a genuine write fault is
* still visible in the log (spec round-9.1 §5.3). */
#ifndef MSG_NOSIGNAL
#define MSG_NOSIGNAL 0
#endif
void el_runtime_ignore_sigpipe(void) {
static int done = 0;
if (done) return;
done = 1;
struct sigaction sa;
memset(&sa, 0, sizeof(sa));
sa.sa_handler = SIG_IGN;
sigemptyset(&sa.sa_mask);
sigaction(SIGPIPE, &sa, NULL);
}
/* Suppress SIGPIPE for one accepted connection (macOS/BSD have no
* MSG_NOSIGNAL; they have the socket option instead). Best effort. */
static void http_socket_nosigpipe(int fd) {
#ifdef SO_NOSIGPIPE
int on = 1;
setsockopt(fd, SOL_SOCKET, SO_NOSIGPIPE, &on, sizeof(on));
#else
(void)fd;
#endif
}
/* Best-effort send with retry on partial writes.
* Returns 0 on success, -1 on failure with errno preserved for the caller. */
static int http_send_all(int fd, const char* p, size_t left) {
while (left > 0) {
ssize_t w = send(fd, p, left, 0);
if (w <= 0) return -1;
ssize_t w = send(fd, p, left, MSG_NOSIGNAL);
if (w < 0) {
if (errno == EINTR) continue; /* not an error — retry */
return -1; /* errno stays set for caller */
}
if (w == 0) { errno = EPIPE; return -1; }
p += w; left -= (size_t)w;
}
return 0;
}
/* Did this write fail because the client is gone, or because something is
* actually wrong with the socket? Only the first is routine. */
static int http_write_err_is_client_gone(int e) {
return e == EPIPE || e == ECONNRESET || e == ENOTCONN || e == ESHUTDOWN;
}
/* Discriminator that http_response() embeds at the start of its envelope.
* A handler returning a string starting with this exact prefix is treated
* as a structured response; anything else is treated as a raw body. */
@@ -1468,14 +1530,30 @@ static void http_send_response(int fd, const char* body) {
free(env_body); free(hdrs.buf); return;
}
if (http_send_all(fd, status_line, (size_t)sl) == 0
&& http_send_all(fd, hdrs.buf, hdrs.len) == 0
&& http_send_all(fd, tail, (size_t)tl) == 0
&& (head_only
/* HEAD requests echo headers + Content-Length but no body. */
? 1
: http_send_all(fd, eff_body, blen) == 0)) {
/* sent successfully */
/* The reply is written in four pieces; any of them can find the client
* already gone. errno is captured at the first failure, before any later
* library call can clobber it, and classified once below. */
errno = 0;
int send_err = 0;
if (http_send_all(fd, status_line, (size_t)sl) != 0) send_err = errno;
else if (http_send_all(fd, hdrs.buf, hdrs.len) != 0) send_err = errno;
else if (http_send_all(fd, tail, (size_t)tl) != 0) send_err = errno;
else if (!head_only /* HEAD echoes headers + Content-Length, no body. */
&& http_send_all(fd, eff_body, blen) != 0) send_err = errno;
if (send_err) {
if (http_write_err_is_client_gone(send_err)) {
/* ROUTINE. The user closed the window, quit the app, or cancelled.
* The work is done and the daemon keeps serving everyone else. */
fprintf(stderr, "[http] client left before the reply was written "
"(%zu-byte body, %s) - request completed, reply discarded\n",
blen, strerror(send_err));
} else {
/* NOT routine — a real write fault. Never let the client-gone case
* above hide this one. */
fprintf(stderr, "[http] send failed: %s (%zu-byte body)\n",
strerror(send_err), blen);
}
}
if (env_parsed_root) el_release(env_parsed_root);
@@ -1491,6 +1569,7 @@ static void* http_worker(void* arg) {
HttpWorkerArg* a = (HttpWorkerArg*)arg;
int fd = a->fd;
free(a);
http_socket_nosigpipe(fd);
char *method = NULL, *path = NULL, *body = NULL;
if (http_read_request(fd, &method, &path, &body, NULL) == 0) {
http_handler_fn h = http_lookup_active();
@@ -1531,6 +1610,7 @@ static void* http_worker(void* arg) {
}
void http_serve(el_val_t port, el_val_t handler) {
el_runtime_ignore_sigpipe(); /* serving implies clients that leave */
/* If `handler` looks like a string name, register it as the active handler. */
const char* hname = EL_CSTR(handler);
if (hname && looks_like_string(handler)) {
@@ -1634,6 +1714,7 @@ static void* _http_serve_async_loop(void* raw) {
}
void http_serve_async(el_val_t port, el_val_t handler) {
el_runtime_ignore_sigpipe(); /* serving implies clients that leave */
const char* hname = EL_CSTR(handler);
if (hname && looks_like_string(handler)) {
http_set_handler(handler);
@@ -1821,6 +1902,7 @@ static void* http_worker_v2(void* arg) {
HttpWorkerArg* a = (HttpWorkerArg*)arg;
int fd = a->fd;
free(a);
http_socket_nosigpipe(fd);
char *method = NULL, *path = NULL, *body = NULL, *hdr_block = NULL;
if (http_read_request(fd, &method, &path, &body, &hdr_block) == 0) {
http_handler4_fn h = http_lookup_active_v2();
@@ -1858,6 +1940,7 @@ static void* http_worker_v2(void* arg) {
}
void http_serve_v2(el_val_t port, el_val_t handler) {
el_runtime_ignore_sigpipe(); /* serving implies clients that leave */
const char* hname = EL_CSTR(handler);
if (hname && looks_like_string(handler)) {
http_set_handler_v2(handler);
@@ -5511,6 +5594,9 @@ el_val_t getpid_now(void) {
static el_val_t _el_args_list = 0;
void el_runtime_init_args(int argc, char** argv) {
/* First line of every generated main(): a client that leaves must never be
* able to signal this process to death. See el_runtime_ignore_sigpipe. */
el_runtime_ignore_sigpipe();
_el_args_list = el_list_empty();
for (int i = 1; i < argc; i++) {
_el_args_list = el_list_append(_el_args_list, EL_STR(argv[i]));