transduce() is now THE single mechanism: one function, no content-type
branch inside it. It never asks whether `source` is prose, JSON, or
raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally:
split on "\n\n" as a universal boundary-marker check, and if that finds
no boundary, fall back to fixed 4096-char windows. Same node/edge wiring
(root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets
a heading/section_of link) regardless of what's inside a chunk. Dedup is
the existing find_existing_by_content path via merge_manifold, applied
uniformly. The old transduce_structured JSON dataset/records/feature-node
interpretation is deleted outright, not just unused — a JSON file now
gets chunked and deduped like anything else, with no pre-computed
structure. All five ingest_* entry points still exist unchanged in name
and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one
transduce() (ingest_stream builds its own turn-nodes directly and never
called either old function, so it's untouched).
This unlocks raw/opaque content (audio, or anything else with no natural
text/JSON shape) without any DSP, LLM call, or external API: transduce()
chunks it exactly like it chunks anything else. There is zero semantic
understanding of audio (or any payload) claimed or built here — any
meaning is expected to emerge later from Neuron's own existing mechanisms
(embedding, spreading activation, dedup) acting on this real geometry
over time.
Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk)
because El strings are NUL-unsafe under strlen-based ops and fs_read()'s
result silently truncates at the first embedded NUL, which is routine in
real binary/audio bytes. ingest_file compares fs_read()'s string length
against a real fs_size() stat() count; on mismatch it rebuilds the
payload as base64-encoded fixed 3072-byte windows read directly off disk
(binary-safe in C, verbatim, no invention), joined with the same "\n\n"
marker transduce()'s boundary scan already looks for. This is a
mechanical fidelity fix, not interpretation of content — transduce()
never learns a fallback happened. Registered both builtins' arity in
codegen.el; did not rebuild the elc compiler binary itself (unrelated,
pre-existing gap: self-hosting elc via el_seed.c fails on this worktree
independent of this change, reproduced with codegen.el reverted) — the
existing elc binary compiles calls to unregistered builtins via its
already-existing arity=-1 passthrough, confirmed by an actual clean
`elc ingest.el` + `cc` build against the modified el_runtime.c.
INGEST_KIND keeps existing only as an acquisition-mechanism selector
(dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a
content-type flag; the redundant "structured" value (an alias for "file"
that hinted the now-deleted JSON branch) is removed. ingest_dir drops its
file-extension filter for the same reason: transduce() takes anything now.
Verification: local manifold construction confirmed correct against a
real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte
real prefix slice) — exact expected node/edge counts both times
(101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching
ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64
content confirmed decoding back to the actual WAV header bytes. Compiles
clean via the real elc + the modified el_runtime.c/engram_*.c (built and
booted an actual sandbox engram off this exact source with `nsbx create
--branch`).
NOT verified this session, disclosed rather than papered over: end-to-end
server-confirmed persistence (a real before/after /api/stats delta, and a
fetched node by id) for the audio, prose, and JSON-fixture cases. Every
local nsbx sandbox engram tried tonight (two stock pre-#109 binaries
hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW
binary built from current dev) took minutes-to indefinitely long on the
final /api/load-merge write's embedding step and hit the client's 60s
HTTP timeout before responding, even for a 5-node write. This is
confirmed as real (if slow) forward progress, not a hang: the sandbox's
WAL file was observed growing steadily across every attempt. The code's
own pre-existing HONESTY GATE correctly refused to report success in
every case, returning "load-merge failed: ..." with a
"nothing below this manifold was confirmed persisted by the server" note
instead — exactly as designed. This is an environment/infrastructure
limitation, not a defect introduced by this change: the engram server
binary itself is untouched by this commit.
Lands feat/reframe-region-setop (PR #109: native set-based reframe_region,
decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency
work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated
seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's
actual current HEAD, plus engram-tiered-storage's still-unique test suite.
RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/
verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/
select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two
branches; #109's copy is a strict superset (adds vindex_harvest_from_store,
used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and
engram_verify.c are also byte-identical. #109's own branch point already
carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing
left to merge into #109 for those files. The one thing engram-tiered-storage
had that #109's tree dropped: its full test suite (test_vindex.c,
test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the
interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh
harnesses) — ported over here unchanged.
WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch
forked from dev on 2026-08-14 15:40 (before restructure-adjacent history
diverged the file's merge-base for `git merge` — it presented as an add/add
conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but
it silently reverted THREE dev fixes landed on 2026-08-14/15, after the
branch point, that the PR's diff had no way to know about:
1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of
the query-aware propagation gate dropped the shift-and-floor rescale
about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate
0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped
around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi]
is computed) is orthogonal to WHAT it gates on and both are kept.
2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes
wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is
diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global).
PR's tree predates this and dropped all three counters + their JSON
stats fields. Restored declarations, all 4 direct increment sites, the
eg_wm_carry_over bll increment, and the act-stats JSON fields --
alongside (not instead of) the PR's own P4 afferent / API-reshape
counters already in that same struct/JSON.
3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev
selects the STRONGEST qualifying candidate for consolidation each call;
PR's tree predates this and reverted to hash-slot order (arbitrary wrt
association strength) for edge formation -- the one path that writes
PERMANENT structure. Restored the strongest-candidate while-loop,
keeping the PR's own genuine improvement at that site
(engram_adj_on_edge_added incremental-index append instead of a bare
adj_dirty=1 full-rebuild flag).
engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env
flags, /api/nodes connected-count in responses) were pure additive: dev's
side was empty, PR's side added the feature. Took PR's side whole.
VERIFIED (nsbx sandbox only, live :8742/:7770 never touched):
- cc -std=c11 -O2, clean link against the real engram/src/server.el via
elc, zero errors.
- vindex_bench (built standalone, read-only harvest) against the real
production store clone (13,671 embedded nodes, 768-dim nomic-embed-text):
recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs
2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s
for the full 13,671-node set -- see the flagged risk below.
- Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned
snapshot of the live store, 13,424 nodes / 37,656 edges) and called
/api/activate for real: first call after boot 41.5s (pays the one-time
HNSW build inline -- matches the standalone bench), second/third calls
356ms/605ms, no crash, correct results, act-stats JSON (including the
restored evict_floor/cap/bll fields) reads correctly.
KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for
this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync
builds the HNSW index synchronously, inline, on the first engram_activate()
call after every process start (or index invalidation). On the real node
count that is a ~46s blocking stall on a single-threaded server -- the first
request after every restart (or its concurrent siblings) waits the full
build. Recommend a background/incremental build (or a bounded per-call build
budget) before this ever reaches the live daemon. See PR description / final
report for the fuller writeup.
Float + previously fell through to string concat (segfault); -, *, /, %
operated on raw IEEE-754 bit patterns as integers (garbage results). Floats
are now tracked via a __float_names typed-binding set (parallel to the
existing int-tracking scheme) and arithmetic is emitted as real C double
ops.
Also fixes math_log, which was wrongly aliased to natural log (duplicating
math_ln) — now uses log10 — and adds the missing <math.h> include. Rebuilt
elc binary included.
Will waived diff review -> build it for real. Add engram_boundary_beat() to the
runtime (afferent counter++ + engram_chrono_tick + engram_strengthen(self-anchor)
+ dharma_emit) and two act-stats counters (aff_boundary_ops, dharma_emits).
codegen cg_fn injects ONE engram_boundary_beat(op) at the entry of every
@manager/@accessor fn (fn_has_decorator, so it fires under @route @manager too) —
a decorated op self-reports with ZERO hand-written instrumentation. Rebuilt elc
self-host + the cognition engram in the worktree; ran it as the clone daemon on
:8900. Proof (/api/boundary-proof, @manager, empty body, 5x): aff_boundary_ops
0->5, dharma_emits 0->5, self activation_count 1510->1513, chrono stamp advanced.
Brought in feat/cognitive-architecture engram runtime+server for the build.
strengthen = activation bump (not content/edge write) -> identity protection
intact. Live :8742 untouched; no push, no cutover.
Ground-truth the three seams (route/telemetry+interoception/bus) with file:line
evidence. Port the tested @route codegen+parser from feat/el-route-decorators
into the worktree elc (decoration synthesizes el_route_dispatch — no hand-written
90-branch handle_request). Rebuild elc self-host; prove decorate->serve end-to-end
(route_proof.el on :8951). Rewrite surface.el as El-native decorated components:
@route + @accessor/@manager, in-process engram_* builtins (not http_get), @manager
ops emit on the real dharma_* bus (same transport as wt/swarm-ccr). Identity
keystones refused in write/relate/supersede. Gate-1 clone recipe (WAL-aside
cold-boot + ENGRAM_WAL=on) proves the FULL op set live on the clone. Boundary
auto-emit (telemetry/interoception/bus) staged as a reviewable cg_fn diff
(SEAM_STAGED.md) — needs the cognition-engram rebuild to verify link. Live :8742
untouched; no push, no cutover.
Resolve 3 conflicts:
- lang/el-compiler/runtime/el_runtime.c: keep deletion (deprecated runtime fork;
single-source-of-truth is lang/runtime/, enforced by scripts/check-single-runtime.sh).
- lang/releases/v1.0.0-20260501/el_runtime.h: keep deletion (releases/ is a generated
artifact folder, not a source path; a release is a git tag, not a folder).
- lang/runtime/el_platform_win.h: union of dev's Windows port (#80: setsockopt optval
wrapper + curl-less libcurl stubs) and our fsync(->_commit) shim needed by engram_store WAL.
Nothing in dev's build consumes the deprecated fork or releases/ folder.
- Wire paged store into the ENGRAM SERVICE (server.el, the authoritative durable
owner): boot->engram_store_boot, persist_canonical->engram_store_checkpoint,
gated by ENGRAM_STORE.
- elc (lang/elc.c + src/parser.el + codegen.el + elc-combined.el): OOB guard in
tok_kind/tok_value + parse_block progress backstop — fixes the pre-existing
unbounded-memory fold hang on sessions.el.
- engram_prune_telemetry mirrors ISE prune to the store (store_forget) so store
live-count tracks resident and stale telemetry stays bounded.
- Deployed live 2026-08-12: engram :8742 on neuron.egm+WAL, count reconciled 11552.
Establish lang/runtime/ as the ONE canonical el runtime (from the active
runtime that carries hebb/emb persistence + the new WAL); repoint the el CI
publish, engram build, elb default, and in-repo build scripts to it; delete
the el-compiler/runtime + lang/releases/ forks; add scripts/check-single-runtime.sh
drift guard.
Fixes a live prod bug: the el CI published el-runtime-c/-h from the LAGGING
el-compiler fork (0 hebb refs), so the shipped soul never persisted Hebbian
edge weights — learned co-activation was wiped on every restart. Publishing
from canonical ships the stranded 'learning that cannot outlive the process'
fix.
WAL storage engine + integrity fixes (DELETE->tombstone + store-layer
protection, safe data-dir default) ride in behind ENGRAM_WAL (default off =
byte-identical to today). Verified: engram elb per-module build clean, WAL
gate 66/66, native smoke ok, drift-guard green.
A cgi block is a top-level declaration, so codegen_streaming classifies it via
is_top_level_decl and releases it. The identity emission then searched
toplevel_exec_stmts for that same block. Declarations are excluded from that list by
construction, so the search could never succeed. A probe printed what it actually
saw for a program whose first statement is a cgi block: [Let, Expr]. It emitted
nothing, silently, with no diagnostic on any channel.
The code documented its own assumption — 'Since cgi blocks are rare and small, they
end up in toplevel_exec_stmts' — and that assumption was false.
Capture the declared values before the release and emit from them. The search is
deleted rather than repaired, so the failure mode is removed rather than relocated.
Proven discriminating (old fails, new passes):
minimal cgi program, old -> 0 el_cgi_init
minimal cgi program, fixed -> el_cgi_init with all four declared values
neuron soul, fixed -> principal present in the compiled binary (0 before),
boots in 2s, interface 110 routes in / 110 out
Consequence: a binary now carries its declared identity as a compiled constant,
which is what the identity protocol requires. Whether the runtime surfaces it to
state_get("soul_principal") is unverified and separate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The loopback/API-key hardening carried in this file since 2026-07-15 called
el_http_request_authorized and el_http_send_401 from http_worker with no
forward declarations, so the calls were implicit and the later static
definitions conflicted. The file did not build. Two prototypes fix it.
Worth naming the pattern: uncommitted work is invisible to every check that
would have caught this. Three weeks of desktop security hardening was neither
committed nor compiling, and nothing reported either fact.
Same defect as the release runtime: \uXXXX was skipped and a literal '?'
emitted, destroying every non-ASCII character in JSON entering the runtime.
Two copies of one parser bug is how this class of fault survives a fix, so
it lands in both.
NOTE: this file also carries pre-existing uncommitted work from 2026-07-15/16
that this commit preserves rather than authors - loopback bind hardening
(EL_HTTP_BIND_HOST) and per-install API-key auth (EL_HTTP_AUTH_KEY) for the
shipped desktop build, plus goal-bias and node-json changes. It had been
sitting in the working tree for three weeks. Committing it because
uncommitted work is work that does not survive, which is the same durability
lesson as yesterday's Hebbian write-back finding. It needs review on its own
terms - see the backlog item for reconciling the two runtime copies.
The binary-safe fs_read length (_tl_fs_read_len) was consumed by the HTTP
response path for ANY body, even when a handler wrapped a smaller file into a
larger reply. Content-Length then lied AND the send stopped short: the
safety-contact (988) routes returned 178 of 208/218 bytes, cut mid-'set_at' —
unparseable JSON. The desktop app read that as failure. On Windows the shipped
brain is an OLD build without even the per-handler workaround, so EVERY reply
truncated: the app can't read confirmations and refuses the new user.
Durable fix: pair the length hint with the exact buffer pointer it describes
(_tl_fs_read_buf). Apply the raw byte count ONLY when the response IS that
buffer (binary file serving stays correct); every wrapped/enveloped/derived
body is measured with strlen. Reset both at request start and in fs_read /
json_get_raw. This also closes the stale-hint heap over-read (a length larger
than a later body would read past it out the socket) that a plain max() leaves
open — so this class of bug dies on every platform, not just where a handler
happened to be patched.
Applied identically to the mainline runtime (lang/el-compiler/runtime) AND the
frozen release runtime (lang/releases/v1.0.0-20260501) the desktop souls
compile against — the release copy still carried the raw leak, which is why the
Windows brain kept truncating. Same proven approach as PR #78 (Tim Lingo),
extended to cover the release runtime and rebased onto current main.
Both runtimes: gcc -fsyntax-only clean.
Two el_runtime portability defects only ever lived in staged local copies
used to hand-build neuron-ui PR #136's curl-enabled Windows neuron.exe.
gcc 15 promotes both to hard errors, so a clean el checkout cannot rebuild
that soul. Upstream the minimal fixes so the build is reproducible:
- http_serve_async: cast setsockopt optval to (const char*). Win32/mingw
setsockopt wants const char*, not int*; the cast is a no-op on POSIX and
matches the four already-cast sites elsewhere in this file.
- engram_save persist path: map fsync -> _commit in the _WIN32-only
el_platform_win.h (io.h already included). Windows has no fsync(); the
POSIX path is untouched.
chat.el calls the runtime native engram_get_node_by_label to fetch
well-known nodes (conv:history, session:summary) by stable label rather
than by ID — immune to vector-index drift across restarts. The current
runtime never defined it, so the regenerated dist/soul.c fails to link.
Backport the function verbatim (idiom-adapted to jb_finish) from release
runtime v1.0.0-20260501 and register it as an EL builtin exactly like its
siblings: runtime definition + prototype, __-prefixed seed wrapper +
prototype, and codegen arity entry. No search-site code is touched.
Lexical istr_contains alone can't surface a node whose words don't appear
in the query. This adds an optional dense-vector layer: node content and the
query are embedded through Ollama (nomic-embed-text), and nodes are ranked by
cosine similarity unioned with lexical hits, so a paraphrase query reaches the
right node.
Wired into all three query entry points in el_runtime.c:
- engram_search_json (HTTP /api/search): collect lexical ∪ semantic
candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine),
rank, emit top-N. Stable sort preserves old order when semantic is off.
- engram_search (internal el_val twin): lexical ∪ semantic union.
- engram_activate seed loop (HTTP /api/activate): a node seeds if it
lexically matches OR clears the cosine threshold; pure-semantic seeds
enter scaled by cosine so paraphrase spreads without overpowering.
Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot
runtime probe. If curl is compiled out, Ollama is unreachable, or
ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers
fall back byte-for-byte to the pre-existing lexical search.
Node embeddings are cached in process memory keyed by node id with an FNV-1a
content hash for invalidation; the query is embedded once per call — so the
graph is not re-embedded on every query. nomic task prefixes
(search_query:/search_document:) are applied for retrieval separation.
Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in
(-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL,
ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).
engram search/activate/goal-bias matched the ENTIRE raw query string as a
single case-insensitive substring (istr_contains(field, q)). Multi-word
queries like "windows msi signing" only matched a node containing that exact
contiguous run, so real multi-word queries returned ZERO on a graph saturated
with the answer. This is Ctrl-F, not search — and search is the core of the
engram being useful.
Fix: split the query on whitespace into distinct tokens; a node matches if it
contains ANY token in content/label/tags. Rank by distinct tokens matched
(desc) then salience (desc). istr_contains is kept unchanged as the per-token
primitive. Single-token queries are a strict special case (score 0 or 1) so
the many single-word callers do not regress.
Sites changed (all in el_runtime.c):
- new helpers engram_tokenize_query / engram_node_match_score / engram_rank_cmp
- engram_search (internal el_val_t path)
- engram_search_json (HTTP /api/search path)
- engram_activate seed loop (HTTP /api/activate path; seed activation scaled
by token coverage so full-query matches seed more strongly)
- engram_goal_bias overlap bonus upgraded to graded token coverage
Proof (6591-node snapshot copy, rebuilt binary on :8799, POST JSON path):
windows msi signing 0 -> 20 Will Anderson 0 -> 20
windows msi 0 -> 20 tokenized search fix 0 -> 20
Single-word parity preserved (VBD/volatility/elc capped at limit; unkey = all
matching nodes). Top hits are relevant (e.g. "Will Anderson" surfaces the
Project Design and VBD whitepapers).
Note: GET ?q=a%20b still returns 0 because query_param (server.el) does not
URL-decode — a separate EL-layer bug; the soul's POST-JSON path is fixed here.
Out-of-range tok_kind/tok_value reads returned runtime null (el_list_get OOB
-> 0) rather than the Eof sentinel, so the inner parse loops (parse_block,
call-arg, array-literal, match-arm) that terminate only on their close
delimiter or k=="Eof" never saw Eof once the cursor ran past the single
trailing Eof token. On unclosed-delimiter input the parser then appended AST
nodes forever -> unbounded allocation -> ~700GB -> OOM (observed compiling
neuron/sessions.el).
Fix at the choke point: tok_kind returns "Eof" and tok_value returns "" for
out-of-range positions, restoring the parser-wide contract that reads at/after
the end yield Eof. expect() no longer steps past the Eof sentinel on mismatch.
This terminates every overrun loop simultaneously; a malformed program now
surfaces as a normal (best-effort) parse end instead of exhausting memory.
Requires a self-hosted bootstrap rebuild of elc to take effect.
Two independent investigations, one runtime, complementary halves:
1. Leak (Jul 2, this machine): JsonBuf buffers returned via el_wrap_str
were raw malloc, never arena-tracked — every engram_*_json call leaked
its output unconditionally. Added jb_finish() arena-tracking across all
~30 return sites. Plus el_arena_push/pop per-tick bracketing support
for the soul's awareness loop (the loop ran outside any request arena,
so even correctly-tracked allocations were permanent — 7.5GB RSS in
under a minute at 1s tick).
2. Corruption (Tim's container soak, docs findings/container-migration):
stored engram node/edge fields (content, node_type, label, tier, tags,
metadata, from/to ids) were arena el_strdup — freed at request end,
leaving dangling pointers that read back as recycled request-buffer
bytes one request later. This is the June corruption root cause and
the mechanism that grew snapshot.json to 18GB of empty-type junk
(21.6M nodes, 3,335 real). 39 sites switched to el_strdup_persist,
plus a latent double-free fix in engram_load metadata fixup.
Interaction note: fix 1's per-tick arena reclamation makes fix 2
mandatory — more aggressive arena recycling widens the use-after-free
window if stored fields still live in the arena. Apply as a pair, never
separately.
Verified live: soul + engram rebuilt from this runtime, booted against
the recovered real snapshot (3,335 nodes/40,146 edges), 5h stable at
<100MB RSS, write-then-next-request field-integrity test passes (the
June corruption fingerprint does not reproduce). engram/dist/engram
binary updated from this build.
Investigation credit: leak diagnosis this machine Jul 2-6; corruption
diagnosis + persist-fix patch by Tim's instance (docs PR #4).
ElBridge.java: Android Java companion to el_android.c — all public methods are
static, dispatches View mutations to the UI thread via runOnUiThread/CountDownLatch,
and exposes native callbacks (nativeOnClick, nativeOnChange, nativeOnSubmit).
PLATFORM_BRIDGE_SPEC.md: authoritative spec for implementing new platform bridges
(slot table contract, required __* functions, callback dispatch pattern).
detect-platforms: shell script that probes for available bridge toolchains and
prints what can be built on the current machine.
new-platform: scaffold generator that creates a new el_<name>.c with all 33
required stubs wired up.
Add seven platform bridge implementations and the shared native target header:
el_native_target.h, el_appkit.m, el_uikit.m, el_android.c, el_gtk4.c,
el_sdl2.c, el_lvgl.c, el_win32.c, el_runtime_win32.c. Each bridge implements
the 33 __widget_* C builtins declared in el_native_target.h for its platform
toolkit. el_runtime_win32.c provides a POSIX-free runtime stub for cross-compiled
Win32 targets.
Implements the accumulation layer from the Layered Consciousness architecture
(provisional 64/064,262) and answers the deferred design question. Per the spec
and Will's design: new user-facing nodes (memories, knowledge, conversations) are
created in an accumulation layer at the TOP of the consciousness stack — the engram
the user sees — while the layers below (safety, core-identity, domain, imprint,
suit) shape behavior but are hidden from the user.
- Adds ENGRAM_LAYER_ACCUMULATION (5) + the layer record in engram_init_layers
(activation_priority 50, suppressible, not injectable, transparent=0).
- engram_node and engram_node_full now assign new nodes to ENGRAM_LAYER_ACCUMULATION.
- ENGRAM_LAYER_DEFAULT stays CORE_IDENTITY ON PURPOSE: it is the fallback for LEGACY
nodes loaded from snapshots without a layer_id, so existing data (the originator
corpus) is NEVER migrated. New-nodes-only — the immutable-originator rule.
This is the foundation for fixing the identity-bleed / customer-isolation issue
(user data was landing in Neuron's core-identity layer). The retrieval-side
provenance filter (introspection should compile from accumulation, not the
originator corpus — Persona 64/036,574) is a follow-on, pending the batch-2
Layered Consciousness + Engram spec docs for exact semantics. Compiles clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ports the fixes that until now lived only in the un-versioned el-sdk source the live
macOS soul was hand-built from (captured in the [DO NOT MERGE] live-darwin-runtime
snapshot) FORWARD onto main, faithfully and minimally — without dragging in the
snapshot's deletions of main's newer engram_wm_/engram_load_merge/http_serve_async.
1. UAF (hallucinated/lost-saves root cause): engram_new_id + engram_node_full now use
el_strdup_persist, NOT el_strdup. el_strdup tracks into the per-request arena that
el_request_end() frees when the creating HTTP request completes — leaving stored
nodes with dangling pointers (corrupted ids, 'saved but never listed'). Transplanted
verbatim from the live runtime; el_strdup_persist sites 19->27, matching live.
2. Atomic engram_save: write <path>.tmp, fflush+fsync, rename() over target (atomic on
POSIX) so a booting soul's engram_load never reads a truncated/0-byte snapshot — the
genesis -> nodes=1 -> 63-node-clobber loop. Plus a sparse-write floor: refuse to
overwrite a >200KB snapshot with one < 1/16 its size. (Validated in isolation:
harness 11/11; rebuilt+booted the darwin soul, round-tripped 5113 nodes, no clobber.)
The response-truncation fix is already on main (_tl_fs_read_len binary-safe length).
Compiles clean. For Will to build through CI/elb and deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
http_handler_fn / http_handler4_fn were defined only inside el_runtime.c, so soul
modules (routes/chat/...) that reference them via cross-module forward declarations
couldn't see the types — which broke the Windows link of every module. Moving the
public function-pointer types to the shared header is the correct home and unblocks
the build on all platforms (identical typedef, C11-safe redefinition in el_runtime.c).
With this, the soul links into a native Windows neuron.exe (mingw, static) that boots
and serves HTTP on :7770 — verified /health → 200 {"status":"alive",...} in a Win11 VM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
llm_call_system / llm_call accepted a model argument and discarded it:
they called llm_chain_call(system, user) with no model, and the legacy
ANTHROPIC_API_KEY fallback passed NULL to llm_provider_request, so every
non-agentic chat was pinned to LLM_DEFAULT_MODEL (claude-sonnet-4-5)
regardless of the caller's selection.
Thread model_pref through llm_chain_call: provider-chain entries still
honor their own NEURON_LLM_N_MODEL override and fall back to the
requested model otherwise; the legacy Anthropic path now uses the
requested model. NULL/empty preserves prior default behavior.
Effect: the soul's model selection (state soul_model / SOUL_LLM_MODEL,
e.g. claude-opus-4-8) now reaches api.anthropic.com. Previously the
chat response echoed the selected model in its label while the request
billed Sonnet 4.5.
Not built locally (no elc/cc toolchain on this checkout); needs stage CI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validate UTF-8 continuation bytes in jb_emit_escaped; pass valid
sequences through and escape orphaned/invalid start bytes as \u00xx.
Pre-existing change found uncommitted in the working tree; committed
here so it is reviewable rather than lost.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
parse_html_children consumed the closing `}` of the outer El function as
HTML text content when a tag was left open across a function boundary
(e.g. `page_open()` opens `<body>` without a closing `</body>`). Fix:
stop the children loop when the current token is RBrace — that token
belongs to the El function, not the HTML tree.
Add html_raw() and html_escape() builtins to el_runtime so templates
can interpolate trusted raw HTML and safely escape user-supplied content.
Rename elc-new.c → elc.c as the canonical compiler source; rebuild
elc binary from it.
Add el_mem_check() to el_runtime.c: reads ELC_MAX_MEM_MB (default 512),
checks RSS via getrusage (macOS bytes / Linux KB normalised to MB), prints
a clear diagnostic to stderr and exits(1) if exceeded.
Wire it into two places:
- compiler.el: upfront check at --emit-header entry point
- codegen.el: per-function check in the streaming loop after each
el_arena_pop, so runaway growth is caught at the earliest function
boundary rather than after the machine is already dying.
The --emit-header path previously called parse() which builds the entire
program AST in memory before writing the .elh file. For checkout.el (~491
lines with HTML template trees and deep BinOp string-concat chains), this
exhausted memory before the header could be written.
Fix: replace parse() + emit_header() with scan_fn_sigs_el() +
emit_header_from_sigs(). The new path tokenises the source once, then
walks the flat token list skipping over function bodies entirely — peak
memory is O(tokens) instead of O(whole-program AST).
New functions in parser.el:
- scan_type_el: reads a type annotation and returns its El source string
- scan_params_el: reads (name: Type, ...) and returns El params string
- scan_fn_sigs_el: token-level scan that collects El-style fn signatures
without building any expression AST nodes
New function in compiler.el:
- emit_header_from_sigs: writes .elh from scan_fn_sigs_el output
Self-hosting check: elc compiled with new elc, diff of outputs is
identical (zero difference).
Smoke test: elc --emit-header checkout.el produces correct three-entry
.elh (previously truncated at two entries due to mid-parse OOM).
The El lexer silently skips '#', so {#each} lexes as LBrace Ident:"each"
and {#if} lexes as LBrace If ... (using the If keyword token, not Hash).
The existing {#each} check used k2=="Hash" which was dead code.
Parser changes (parser.el):
- Add parse_raw_text_content(): collects all tokens as raw text until
</tag_name>, bypassing El expression parsing. Used for <style> and
<script> elements so CSS/JS content isn't parsed as El expressions.
- parse_html_element(): use raw-text mode for <style> and <script> tags.
- parse_html_children(): fix {#each} detection (k2=="Ident", k3=="each"
instead of dead k2=="Hash" check). Add {#if cond}...{#else}...{/if}
support generating HtmlIf AST nodes.
Codegen changes (codegen.el):
- Add cg_html_if(): generates if (cond_c) { then_c } else { else_c }
for HtmlIf nodes.
- cg_html_parts(): dispatch HtmlIf to cg_html_if.
el-install.el generates calls to __http_do_map_to_file (HTTP request
with JSON headers map, streaming response to file). Add it to both
the HAVE_CURL implementation and the no-curl stub section.
El compiler generates calls to println, print, exit_program,
http_set_handler, http_serve, http_set_handler_v2, and http_serve_v2
as el_val_t-returning functions. The runtime declared them void,
causing conflicting-type errors when el-install.c was compiled.
Change all seven to return el_val_t (side-effect functions return 0).
Also update el_runtime.h declarations to match.
Add native_str_to_int (El compiler alias for str_to_int) and
http_post_json_with_headers (JSON POST with additional headers map)
which epm.el generates calls to but were absent from el_runtime.c.
0.0/0.0 can produce -nan on Linux/x86_64 (%g gives '-nan'),
causing the no-cycle calendar test to fail. Explicitly check isnan()
and emit 'nan' so behavior is platform-independent.
Implement compile_test() entry point that emits a C test harness instead
of a normal program. Test blocks (previously skipped) now compile to
static functions with per-assertion pass/fail tracking. Assert statement
added to parser and codegen. Runtime extended with now_ns, fs_list_json,
json_build_object, json_build_array, json_escape_string, state_has,
state_get_or. Fix float negation codegen, float equality comparisons,
time_to_parts return type (JSON string), time_format empty-fmt, json_set
raw-value semantics, state_keys JSON array return. All 310 native tests
pass across 9 suites (core, text, string, math, env, state, json, time, fs).
bigmerge
will.anderson
tim.lingo