Same defect as the release runtime: \uXXXX was skipped and a literal '?'
emitted, destroying every non-ASCII character in JSON entering the runtime.
Two copies of one parser bug is how this class of fault survives a fix, so
it lands in both.
NOTE: this file also carries pre-existing uncommitted work from 2026-07-15/16
that this commit preserves rather than authors - loopback bind hardening
(EL_HTTP_BIND_HOST) and per-install API-key auth (EL_HTTP_AUTH_KEY) for the
shipped desktop build, plus goal-bias and node-json changes. It had been
sitting in the working tree for three weeks. Committing it because
uncommitted work is work that does not survive, which is the same durability
lesson as yesterday's Hebbian write-back finding. It needs review on its own
terms - see the backlog item for reconciling the two runtime copies.
engram search/activate/goal-bias matched the ENTIRE raw query string as a
single case-insensitive substring (istr_contains(field, q)). Multi-word
queries like "windows msi signing" only matched a node containing that exact
contiguous run, so real multi-word queries returned ZERO on a graph saturated
with the answer. This is Ctrl-F, not search — and search is the core of the
engram being useful.
Fix: split the query on whitespace into distinct tokens; a node matches if it
contains ANY token in content/label/tags. Rank by distinct tokens matched
(desc) then salience (desc). istr_contains is kept unchanged as the per-token
primitive. Single-token queries are a strict special case (score 0 or 1) so
the many single-word callers do not regress.
Sites changed (all in el_runtime.c):
- new helpers engram_tokenize_query / engram_node_match_score / engram_rank_cmp
- engram_search (internal el_val_t path)
- engram_search_json (HTTP /api/search path)
- engram_activate seed loop (HTTP /api/activate path; seed activation scaled
by token coverage so full-query matches seed more strongly)
- engram_goal_bias overlap bonus upgraded to graded token coverage
Proof (6591-node snapshot copy, rebuilt binary on :8799, POST JSON path):
windows msi signing 0 -> 20 Will Anderson 0 -> 20
windows msi 0 -> 20 tokenized search fix 0 -> 20
Single-word parity preserved (VBD/volatility/elc capped at limit; unkey = all
matching nodes). Top hits are relevant (e.g. "Will Anderson" surfaces the
Project Design and VBD whitepapers).
Note: GET ?q=a%20b still returns 0 because query_param (server.el) does not
URL-decode — a separate EL-layer bug; the soul's POST-JSON path is fixed here.
Two independent investigations, one runtime, complementary halves:
1. Leak (Jul 2, this machine): JsonBuf buffers returned via el_wrap_str
were raw malloc, never arena-tracked — every engram_*_json call leaked
its output unconditionally. Added jb_finish() arena-tracking across all
~30 return sites. Plus el_arena_push/pop per-tick bracketing support
for the soul's awareness loop (the loop ran outside any request arena,
so even correctly-tracked allocations were permanent — 7.5GB RSS in
under a minute at 1s tick).
2. Corruption (Tim's container soak, docs findings/container-migration):
stored engram node/edge fields (content, node_type, label, tier, tags,
metadata, from/to ids) were arena el_strdup — freed at request end,
leaving dangling pointers that read back as recycled request-buffer
bytes one request later. This is the June corruption root cause and
the mechanism that grew snapshot.json to 18GB of empty-type junk
(21.6M nodes, 3,335 real). 39 sites switched to el_strdup_persist,
plus a latent double-free fix in engram_load metadata fixup.
Interaction note: fix 1's per-tick arena reclamation makes fix 2
mandatory — more aggressive arena recycling widens the use-after-free
window if stored fields still live in the arena. Apply as a pair, never
separately.
Verified live: soul + engram rebuilt from this runtime, booted against
the recovered real snapshot (3,335 nodes/40,146 edges), 5h stable at
<100MB RSS, write-then-next-request field-integrity test passes (the
June corruption fingerprint does not reproduce). engram/dist/engram
binary updated from this build.
Investigation credit: leak diagnosis this machine Jul 2-6; corruption
diagnosis + persist-fix patch by Tim's instance (docs PR #4).
Implements the accumulation layer from the Layered Consciousness architecture
(provisional 64/064,262) and answers the deferred design question. Per the spec
and Will's design: new user-facing nodes (memories, knowledge, conversations) are
created in an accumulation layer at the TOP of the consciousness stack — the engram
the user sees — while the layers below (safety, core-identity, domain, imprint,
suit) shape behavior but are hidden from the user.
- Adds ENGRAM_LAYER_ACCUMULATION (5) + the layer record in engram_init_layers
(activation_priority 50, suppressible, not injectable, transparent=0).
- engram_node and engram_node_full now assign new nodes to ENGRAM_LAYER_ACCUMULATION.
- ENGRAM_LAYER_DEFAULT stays CORE_IDENTITY ON PURPOSE: it is the fallback for LEGACY
nodes loaded from snapshots without a layer_id, so existing data (the originator
corpus) is NEVER migrated. New-nodes-only — the immutable-originator rule.
This is the foundation for fixing the identity-bleed / customer-isolation issue
(user data was landing in Neuron's core-identity layer). The retrieval-side
provenance filter (introspection should compile from accumulation, not the
originator corpus — Persona 64/036,574) is a follow-on, pending the batch-2
Layered Consciousness + Engram spec docs for exact semantics. Compiles clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ports the fixes that until now lived only in the un-versioned el-sdk source the live
macOS soul was hand-built from (captured in the [DO NOT MERGE] live-darwin-runtime
snapshot) FORWARD onto main, faithfully and minimally — without dragging in the
snapshot's deletions of main's newer engram_wm_/engram_load_merge/http_serve_async.
1. UAF (hallucinated/lost-saves root cause): engram_new_id + engram_node_full now use
el_strdup_persist, NOT el_strdup. el_strdup tracks into the per-request arena that
el_request_end() frees when the creating HTTP request completes — leaving stored
nodes with dangling pointers (corrupted ids, 'saved but never listed'). Transplanted
verbatim from the live runtime; el_strdup_persist sites 19->27, matching live.
2. Atomic engram_save: write <path>.tmp, fflush+fsync, rename() over target (atomic on
POSIX) so a booting soul's engram_load never reads a truncated/0-byte snapshot — the
genesis -> nodes=1 -> 63-node-clobber loop. Plus a sparse-write floor: refuse to
overwrite a >200KB snapshot with one < 1/16 its size. (Validated in isolation:
harness 11/11; rebuilt+booted the darwin soul, round-tripped 5113 nodes, no clobber.)
The response-truncation fix is already on main (_tl_fs_read_len binary-safe length).
Compiles clean. For Will to build through CI/elb and deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
llm_call_system / llm_call accepted a model argument and discarded it:
they called llm_chain_call(system, user) with no model, and the legacy
ANTHROPIC_API_KEY fallback passed NULL to llm_provider_request, so every
non-agentic chat was pinned to LLM_DEFAULT_MODEL (claude-sonnet-4-5)
regardless of the caller's selection.
Thread model_pref through llm_chain_call: provider-chain entries still
honor their own NEURON_LLM_N_MODEL override and fall back to the
requested model otherwise; the legacy Anthropic path now uses the
requested model. NULL/empty preserves prior default behavior.
Effect: the soul's model selection (state soul_model / SOUL_LLM_MODEL,
e.g. claude-opus-4-8) now reaches api.anthropic.com. Previously the
chat response echoed the selected model in its label while the request
billed Sonnet 4.5.
Not built locally (no elc/cc toolchain on this checkout); needs stage CI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validate UTF-8 continuation bytes in jb_emit_escaped; pass valid
sequences through and escape orphaned/invalid start bytes as \u00xx.
Pre-existing change found uncommitted in the working tree; committed
here so it is reviewable rather than lost.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
parse_html_children consumed the closing `}` of the outer El function as
HTML text content when a tag was left open across a function boundary
(e.g. `page_open()` opens `<body>` without a closing `</body>`). Fix:
stop the children loop when the current token is RBrace — that token
belongs to the El function, not the HTML tree.
Add html_raw() and html_escape() builtins to el_runtime so templates
can interpolate trusted raw HTML and safely escape user-supplied content.
Rename elc-new.c → elc.c as the canonical compiler source; rebuild
elc binary from it.
Add el_mem_check() to el_runtime.c: reads ELC_MAX_MEM_MB (default 512),
checks RSS via getrusage (macOS bytes / Linux KB normalised to MB), prints
a clear diagnostic to stderr and exits(1) if exceeded.
Wire it into two places:
- compiler.el: upfront check at --emit-header entry point
- codegen.el: per-function check in the streaming loop after each
el_arena_pop, so runaway growth is caught at the earliest function
boundary rather than after the machine is already dying.
el-install.el generates calls to __http_do_map_to_file (HTTP request
with JSON headers map, streaming response to file). Add it to both
the HAVE_CURL implementation and the no-curl stub section.
El compiler generates calls to println, print, exit_program,
http_set_handler, http_serve, http_set_handler_v2, and http_serve_v2
as el_val_t-returning functions. The runtime declared them void,
causing conflicting-type errors when el-install.c was compiled.
Change all seven to return el_val_t (side-effect functions return 0).
Also update el_runtime.h declarations to match.
Add native_str_to_int (El compiler alias for str_to_int) and
http_post_json_with_headers (JSON POST with additional headers map)
which epm.el generates calls to but were absent from el_runtime.c.
0.0/0.0 can produce -nan on Linux/x86_64 (%g gives '-nan'),
causing the no-cycle calendar test to fail. Explicitly check isnan()
and emit 'nan' so behavior is platform-independent.
Implement compile_test() entry point that emits a C test harness instead
of a normal program. Test blocks (previously skipped) now compile to
static functions with per-assertion pass/fail tracking. Assert statement
added to parser and codegen. Runtime extended with now_ns, fs_list_json,
json_build_object, json_build_array, json_escape_string, state_has,
state_get_or. Fix float negation codegen, float equality comparisons,
time_to_parts return type (JSON string), time_format empty-fmt, json_set
raw-value semantics, state_keys JSON array return. All 310 native tests
pass across 9 suites (core, text, string, math, env, state, json, time, fs).
el_strdup tracks pointers in the arena. The BFS arrays in
engram_neighbors_json are manually freed — using el_strdup caused a
double-free when the arena was later popped. Changed to plain strdup
for those allocations.
engram/dist/engram.c rebuilt from engram/src/server.el with current
elc (minor codegen diff: parenthesisation and _argc/_argv rename).
Adds EL_TRUE/EL_FALSE convenience macros to el_runtime.h alongside the
existing EL_NULL, making boolean-returning builtins readable without
raw (el_val_t) casts. Documents all value macros in the header comment.
Also lands el_arena_push/el_arena_pop — a scoped string arena for CLI
programs that never call el_request_start/end. The compiler can push a
mark before a compilation unit and pop it after to free intermediate
strings, reducing peak RSS during long compile runs.
will.anderson
tim.lingo
Tim Lingo