- Link only the entry-point .c (monolithic) instead of all module .c
files — prevents duplicate symbol errors from inlined imports
- Strip capability-violation #error guards post-compile; they fire
incorrectly when modules are compiled in isolation but linked into
a CGI program
- Add liboqs (post-quantum) include/lib detection via brew, matching
the existing OpenSSL detection pattern
Two improvements from daily self-review:
1. engram_wm_top_json(n) builtin — returns top-N WM nodes by weight as
compact JSON [{label,node_type,tier,wm},...]. After long uptime all WM
nodes cycle in steady-state decay+re-promotion so wm_promotion ISEs
never fire (only trigger on 0→>0.1 transitions). This gives continuous
visibility into WM composition on every heartbeat.
2. ISE tier-migration exclusion — InternalStateEvent nodes no longer
participate in tier migration. ISEs are activated by curiosity_scan
substring matches and accumulated 50+ activation_count, reaching
Procedural tier (720h decay). This permanently crowded WM with ephemeral
state events, hiding the Knowledge/Memory nodes that should dominate.
Fix: skip ISEs in the TIER MIGRATION pass; they stay on Working-tier
48h decay regardless of activation frequency.
wm_top_json is called from awareness.el emit_heartbeat (top-5 snapshot).
Three targeted improvements based on graph health analysis (29K nodes, 104 edges):
1. ENGRAM_WM_THRESHOLD 0.15 → 0.08: sparse graph means BFS paths carry weak
signals (0.05-0.12 range). Prior threshold gatekept too aggressively. Grounded
in TBRS* cognitive model (θ=0.05); 0.08 is conservative but effective.
2. ENGRAM_INHIBITION_FACTOR 0.1 → 0.2: factor=0.1 (90% suppression) on a sparse
graph almost always fully silences targeted nodes. Factor=0.2 (80% suppression)
maintains strong inhibition while allowing partially-suppressed nodes to remain
faintly active — consistent with partial inhibition in cognitive neuroscience.
3. engram_wm_avg_weight() builtin: computes mean working_memory_weight of all
promoted nodes. Returns float bits via el_from_float for EL float_to_str usage.
Makes activation quality directly observable in heartbeat ISEs, distinguishing
'many weak activations' (sparse, low avg) from 'few strong' (dense, high avg).
Rebuilt engram binary with new runtime.
Two changes:
1. el_runtime.c — engram_activation_dampen(): add floor of 0.35.
ISE nodes with ac=900+ had dampen=0.128, giving effective salience=0.038
which fell below the epist>=0.1 gate in engram_activate. This silently
killed curiosity seeds "self identity values" and "decision pattern lesson"
— the only corpus matches were high-ac ISEs that were then excluded from
results, causing activated=0 on 50% of proactive_curiosity scans.
Floor at 0.35 keeps salience=0.3 nodes at effective_bg=0.105, above the
visibility threshold, without disrupting relative ordering of content nodes.
2. server.el — route_create_node: replace stale inline auto-link with
auto_link_content_node(). The inline logic used the old engram_search_json
(substring, no ISE filter) while the better BM25-based auto_link_content_node
was added in 2026-05-28 and wired to /api/neuron/* routes but not to the
raw /api/nodes POST path. Removes ~40 lines of duplicated logic.
Three changes:
1. Fix checkpoint ISE temporal_decay_rate: engram_emit_ise_internal was
hardcoded to 0.0 (global 168h default) instead of 2.310 (Working-tier
48h). Result: checkpoint ISEs accumulated at 3.5x intended rate.
2. Raise CHECKPOINT_INTERVAL 1→10: checkpoint ISE fires on every single
node write, producing 2:1 checkpoint:content ratio in ISE stream.
MCP routes still call engram_write_binary_el explicitly after each
important write, so no knowledge durability is lost.
3. Add auto_link_content_node to server.el: route_neuron_memory and
route_neuron_knowledge_capture were creating nodes with zero edges —
invisible to BFS traversal, only reachable via lexical/semantic seed.
New helper runs BM25 over top-20 results, skips ISE nodes (which
dominate the 14K-node corpus), connects up to 3 related nodes.
Add engram_auto_link_semantic(): when a node is embedded, scan all
embedded nodes for cosine sim >= 0.72 and create bidirectional
"semantic-similar" edges to the top-3 matches. Runs once at write
time rather than at every query, converting the expensive O(N) scan
from live activation into durable graph structure.
Fixes the core connectivity problem: 63 edges / 5364 nodes (0.012
edges/node). Verified: new Knowledge nodes now auto-link at sim 0.77–
0.95 with correct threshold discrimination. One checkpoint per insert
(not one per edge) bounds the persistence overhead.
Excludes InternalStateEvents and inbox/outbox transient nodes to keep
semantic graph clean.
Two improvements:
1. BM25 search corpus coverage (server.el) — raised scan cap from 500 to
5000 nodes. On the 161K-node graph, 500 was 0.3% coverage; 5000 is 3%.
engram_scan_nodes_json returns nodes sorted by salience DESC, so ISEs
(salience 0.3) fall below Knowledge/Memory (0.5–0.8) naturally — the
effective corpus stays content-dense. Also updated stale comment on the
ISE route (no longer need high offset; recent-first ordering from May 23).
2. Traversal inference guard (el_runtime.c) — two changes:
- INFER_CAP reduced 256→32: proactive curiosity runs engram_activate
every ~30s. At 256 edges/call the soul daemon accumulated 107K edges
in 23h (5× BFS slowdown). At 32 the rate drops 8×.
- Edge count guard: skip inference entirely when snap_ec ≥ 40,000.
At that density most A→C paths are already explicit; marginal inference
value is low and the O(edge_count²) inner-loop cost is high. Self-limits
unbounded accumulation across restarts.
Two improvements:
1. ISE scan ordering — engram_scan_nodes_by_type_json now sorts InternalStateEvent
nodes by created_at DESC (most-recent-first) instead of salience DESC. Old
high-salience ISEs (session-start, wm-promotion) no longer dominate offset 0,
burying recent heartbeat and curiosity_scan events at offsets 20000+. New
behavior: ?limit=10 returns the 10 most recent ISEs regardless of salience.
All other node types retain existing salience-sorted behavior.
2. http_serve_async registered as elc builtin — added to builtin_arity() in both
codegen.el (EL compiler source) and dist/platform/elc.c (compiled C). Also
rebuilt elc binary from updated elc.c. This closes the fragile-patch gap from
2026-05-21: elc previously treated http_serve_async as an unknown identifier,
and the gap description noted elc would 'silently revert to blocking http_serve'
on next soul rebuild. Now http_serve_async has a proper 2-arg arity entry and
will survive all future soul recompiles without a manual neuron.c patch.
The activation persist step was writing wm_weight=0 for every node not reached
by the current BFS fan-out. This destroyed working memory accumulated by
MCP-layer activations within one tick of the awareness loop firing on an empty
inbox. ACT-R and Soar treat spreading activation as additive: absent seeds
contribute zero spread, not a zero override of existing WM state.
Fix: non-reached nodes now decay by ENGRAM_WM_DECAY (0.7) per activation call
rather than being immediately zeroed. A hard floor of 0.005 clears near-zero
values to prevent infinite decay tails. Reached nodes behave unchanged.
Semantic seed floor raised 0.65 → 0.70: field literature (SuperLocalMemory
arXiv:2604.04514) validates 0.70-0.75 as the noise floor for sentence-transformer
embeddings on non-trivial corpora. The 0.65 threshold was admitting false
positives that diluted BFS traversal quality on the 13K-node post-ingestor graph.
Top-30 cap retained — sparse graph (1.26 edges/node) needs more semantic entry
points than a dense graph would.
ACT-R Base-Level Learning insight applied to engram_temporal_decay: current
purely time-based formula treated a node activated 100 times identically to a
node never activated (same decay rate). ACT-R BLL (Anderson 1993) shows
frequently-accessed memories resist temporal decay. Fix: freq_resist multiplier
= 1.0 + 0.1 * log(1 + activation_count). count=0→1.0×, count=9→1.23×,
count=99→1.46×. Existing activation_dampening continues to reduce per-query
novelty for well-known nodes — the two mechanisms are complementary: resist
decay (durable in graph), dampen per-query (don't dominate any single turn).
Three improvements from daily review:
1. Add semantic seed supplement to Pass 1 activation (el_runtime.c).
Previously, engram_cosine_sim was only called in goal_bias (Pass 2) for
nodes that already matched lexically. Nodes semantically close but
lexically disjoint were completely invisible to activation. With 8K+
world-ingestor nodes added overnight, this was a critical gap. Now: after
lexical seeding, scan un-seeded nodes for cosine sim ≥ 0.65 and inject
top-30 as additional seeds. Sem seeds get 85% of full act to stay weaker
than exact lexical matches.
2. Lower WM promotion ISE threshold from >0.5 to >0.1 (el_runtime.c).
Only one wm-promotion ISE was ever logged — the 0.5 floor was too high.
Most practical Knowledge/Memory promotions are in the 0.1-0.5 range.
Lowering to 0.1 makes working memory activity visible in state events.
Three research-grounded improvements:
1. Tier-based temporal decay in el_runtime.c (engram_node_full, engram_node_layered):
Working=48h, Episodic=72h, Semantic=336h, Procedural=720h half-lives.
Grounded in ACT-R literature — differentiated decay by chunk type. The
temporal_decay_rate field existed but was always 0 (global 168h for everything).
New nodes now carry the correct half-life for their tier from creation.
2. Implement route_neuron_knowledge_promote in server.el (was a silent stub):
Reads existing node, creates promoted-tier copy with supersedes edge,
checkpoints. promote_knowledge MCP tool now has real effect.
3. ISE label extraction + offset support in route_neuron_state_events:
POST now extracts 'event' field from content JSON as label (heartbeat,
wm_promotion, etc.) instead of always writing 'state-event'. GET now
accepts ?offset= for pagination to reach recent ISEs.
engram_cosine_sim was defined but never called. Nodes have 768-dim
nomic-embed-text vectors. Now:
- engram_embed_query() embeds the query string once per activate() call
- engram_goal_bias() takes (qvec, qdim) and adds cosine-similarity bonus
up to +0.6 when sim > 0.5 — semantic relevance now augments lexical bias
- engram_wm_count() exposes working-memory-active node count to EL
- el_runtime.h declares engram_wm_count for soul-daemon linking
- Add list_set, math_exp, and float_add/sub/mul/div/gt/lt/eq/gte/lte builtins to
el_runtime.c + el_runtime.h (float arithmetic builtins needed because EL operators
+*/ operate on raw el_val_t bits, not IEEE 754 doubles)
- Remove engram_embed_query() and its forward declaration from el_runtime.c
- Remove Ollama cosine-similarity blend from activation scoring (reverts 9af2482):
drops query_emb/query_edim variables, bias *= (1 + 0.3 * sim) block, and all
free(query_emb) calls from the activation loop
- Implement BM25+ scoring in server.el (k1=1.2, b=0.75, delta=1.0):
bm25_tokenize, bm25_count_term, bm25_score_doc, bm25_search_json
V1 uses n_t=1 approximation (constant IDF per corpus size); acceptable as a
first pass without an inverted index
- Wire /api/bm25/search POST/GET route in server.el dispatcher
- Zero Ollama calls in the activation/search path; embeddings on nodes are
untouched (still written at node-creation time)
engram_cosine_sim() was defined and embeddings were computed per-node
via nomic-embed-text on write, but the function was never called during
activation scoring. The goal_bias computation used only lexical substring
matching, ignoring all stored embedding vectors.
This change adds engram_embed_query() to embed the query string at search
time (5s timeout so Ollama latency never blocks activation), then blends
cosine similarity into the working-memory bias with α=0.3:
bias_final = goal_bias(lexical) * (1 + 0.3 * max(0, cosine_sim))
Nodes with high semantic similarity to the query but low lexical overlap
now receive up to 30% bias boost into working memory promotion. Gracefully
degrades to pure lexical when Ollama is unavailable or node has no embedding.
- Add ML-KEM-1024 + AES-256-GCM binary persistence to el_runtime.c with
two-key scheme (Neuron master + user key); SHAKE-256 key derivation
- Add nomic-embed-text 768-dim float32 embeddings on every node write
via Ollama; graceful fallback when Ollama is not running
- Wire all /api/neuron/* MCP routes directly into Engram (server.el),
eliminating the Kotlin server as the MCP backend
- Set ENGRAM_CHECKPOINT_INTERVAL = 1 (write binary on every node write,
not every 50)
- Add el_runtime.h declarations for engram_write_binary_el and
engram_load_binary_el builtins
elb.el:
- Auto-detect Homebrew OpenSSL (-L$(brew --prefix openssl)/lib) so -lssl
resolves on macOS without manual flags; no-op on Linux
- Add -include elp-c-decls.h when present in out_dir: resolves undeclared
cross-module calls in packages like ELP that lack explicit imports
ELP source:
- Add import "morphology.el" to all 29 language morphology modules
- Add language module imports to morphology.el (all langs it dispatches to)
These were missing since ELP was originally built as a monolithic unit
parse_html_children consumed the closing `}` of the outer El function as
HTML text content when a tag was left open across a function boundary
(e.g. `page_open()` opens `<body>` without a closing `</body>`). Fix:
stop the children loop when the current token is RBrace — that token
belongs to the El function, not the HTML tree.
Add html_raw() and html_escape() builtins to el_runtime so templates
can interpolate trusted raw HTML and safely escape user-supplied content.
Rename elc-new.c → elc.c as the canonical compiler source; rebuild
elc binary from it.
Add `c_source "path"` in manifest.el build block — lets packages link
extra C files (platform stubs, native glue) without touching elb source.
On macOS, homebrew OpenSSL isn't on the default linker path. Detect it
via `brew --prefix` and inject -L/-I flags; no-op on Linux.
Rebuild elb binary; remove elc-new binary (elc is now canonical).
Rebuilt from fix/elc-oom-checkout: scan_fn_sigs_el() --emit-header path
+ el_mem_check() guard. Verified on checkout.el: all 3 sigs in .elh,
clean exit under normal load, exit(1) on memory limit exceeded.
Add el_mem_check() to el_runtime.c: reads ELC_MAX_MEM_MB (default 512),
checks RSS via getrusage (macOS bytes / Linux KB normalised to MB), prints
a clear diagnostic to stderr and exits(1) if exceeded.
Wire it into two places:
- compiler.el: upfront check at --emit-header entry point
- codegen.el: per-function check in the streaming loop after each
el_arena_pop, so runaway growth is caught at the earliest function
boundary rather than after the machine is already dying.
The --emit-header path previously called parse() which builds the entire
program AST in memory before writing the .elh file. For checkout.el (~491
lines with HTML template trees and deep BinOp string-concat chains), this
exhausted memory before the header could be written.
Fix: replace parse() + emit_header() with scan_fn_sigs_el() +
emit_header_from_sigs(). The new path tokenises the source once, then
walks the flat token list skipping over function bodies entirely — peak
memory is O(tokens) instead of O(whole-program AST).
New functions in parser.el:
- scan_type_el: reads a type annotation and returns its El source string
- scan_params_el: reads (name: Type, ...) and returns El params string
- scan_fn_sigs_el: token-level scan that collects El-style fn signatures
without building any expression AST nodes
New function in compiler.el:
- emit_header_from_sigs: writes .elh from scan_fn_sigs_el output
Self-hosting check: elc compiled with new elc, diff of outputs is
identical (zero difference).
Smoke test: elc --emit-header checkout.el produces correct three-entry
.elh (previously truncated at two entries due to mid-parse OOM).
The El lexer silently skips '#', so {#each} lexes as LBrace Ident:"each"
and {#if} lexes as LBrace If ... (using the If keyword token, not Hash).
The existing {#each} check used k2=="Hash" which was dead code.
Parser changes (parser.el):
- Add parse_raw_text_content(): collects all tokens as raw text until
</tag_name>, bypassing El expression parsing. Used for <style> and
<script> elements so CSS/JS content isn't parsed as El expressions.
- parse_html_element(): use raw-text mode for <style> and <script> tags.
- parse_html_children(): fix {#each} detection (k2=="Ident", k3=="each"
instead of dead k2=="Hash" check). Add {#if cond}...{#else}...{/if}
support generating HtmlIf AST nodes.
Codegen changes (codegen.el):
- Add cg_html_if(): generates if (cond_c) { then_c } else { else_c }
for HtmlIf nodes.
- cg_html_parts(): dispatch HtmlIf to cg_html_if.
el-install.el explicitly imported runtime/*.el modules (string, env, fs, exec,
json, http), which elb compiled to .c files in the shared dist/bin out_dir.
Linking those alongside el_runtime.c caused multiple definition errors for
every runtime function (http_get, http_patch, etc.). The runtime .el files are
thin wrappers over seed primitives already compiled into el_runtime.c — no
import needed.
Fixes:
- Remove all explicit runtime imports from el-install.el (root cause)
- Add --clean to every elb invocation in sdk-release.yaml so each build
starts with a clean out_dir (defense-in-depth against stale .c files)
- Add elb build + epm/el-install build steps to ci-dev.yaml and ci-stage.yaml
so linker errors are caught on every PR, not just stage->main
el_runtime.c uses OpenSSL (EVP_*, RAND_bytes) for AEAD encrypt/decrypt.
elb was only linking -lcurl -lpthread -lm, missing the SSL libs.
Matches the explicit flags used in ci-dev.yaml and ci-stage.yaml.
gcc rejects -fbracket-depth=1024 with 'unrecognized command-line option'.
Use shell subshell to probe cc --version and only pass the flag when
the compiler is clang.
el-install.el generates calls to __http_do_map_to_file (HTTP request
with JSON headers map, streaming response to file). Add it to both
the HAVE_CURL implementation and the no-curl stub section.
El compiler generates calls to println, print, exit_program,
http_set_handler, http_serve, http_set_handler_v2, and http_serve_v2
as el_val_t-returning functions. The runtime declared them void,
causing conflicting-type errors when el-install.c was compiled.
Change all seven to return el_val_t (side-effect functions return 0).
Also update el_runtime.h declarations to match.
Add native_str_to_int (El compiler alias for str_to_int) and
http_post_json_with_headers (JSON POST with additional headers map)
which epm.el generates calls to but were absent from el_runtime.c.
0.0/0.0 can produce -nan on Linux/x86_64 (%g gives '-nan'),
causing the no-cycle calendar test to fail. Explicitly check isnan()
and emit 'nan' so behavior is platform-independent.
Implement compile_test() entry point that emits a C test harness instead
of a normal program. Test blocks (previously skipped) now compile to
static functions with per-assertion pass/fail tracking. Assert statement
added to parser and codegen. Runtime extended with now_ns, fs_list_json,
json_build_object, json_build_array, json_escape_string, state_has,
state_get_or. Fix float negation codegen, float equality comparisons,
time_to_parts return type (JSON string), time_format empty-fmt, json_set
raw-value semantics, state_keys JSON array return. All 310 native tests
pass across 9 suites (core, text, string, math, env, state, json, time, fs).
el_strdup tracks pointers in the arena. The BFS arrays in
engram_neighbors_json are manually freed — using el_strdup caused a
double-free when the arena was later popped. Changed to plain strdup
for those allocations.
engram/dist/engram.c rebuilt from engram/src/server.el with current
elc (minor codegen diff: parenthesisation and _argc/_argv rename).
test "name" { ... } blocks were not recognized by the self-hosted
compiler. The body { } was parsed as a Map literal, creating a huge
AST with O(n²) string concatenation in the toplevel_exec_stmts loop
(which had no arena scope). A 272-line test file would consume 400MB+
and a 720-line file importing the full compiler source caused 150GB
usage and crashed the machine.
Two fixes:
1. Skip Test tokens in codegen_streaming before parse_one() —
advance past "name" and skip_to_rbrace on the body block.
Test blocks are never compiled; self-hosted compiler has no test runner.
2. Add per-statement arena scope to toplevel_exec_stmts emission loop,
matching the el_main_body loop. Frees intermediate strings after
each statement to prevent O(n²) accumulation from any unrecognized
construct that reaches that path.
Result: test_string.el (272 lines, 27 test blocks): 0MB peak (was 400MB+).
test_compiler.el (720 lines + 8728 imported): 15MB peak (was 150GB).
Wrapped compile() body in el_arena_push/pop so the arena is active
before lex() and scan_fn_sigs(). Previously both ran with
_tl_arena_active=0, leaking all token and signature strings permanently.
Also prevents inner pop(mark=0) calls from deactivating the arena
between per-function scopes. Verified: self-host PASS, RSS stable.
- Flat token list: lexer emits [kind0, val0, kind1, val1, ...] instead of [{kind,val}, ...]
Eliminates per-token ElMap allocation (~112B × N tokens)
- str_char_code hot loop: char classification via Int codes, no strdup per char
- Batch c_escape: str_slice clean runs instead of char-at per byte
- Parser updated to use tok_at/tok_kind/tok_value stride-2 accessors
Adds EL_TRUE/EL_FALSE convenience macros to el_runtime.h alongside the
existing EL_NULL, making boolean-returning builtins readable without
raw (el_val_t) casts. Documents all value macros in the header comment.
Also lands el_arena_push/el_arena_pop — a scoped string arena for CLI
programs that never call el_request_start/end. The compiler can push a
mark before a compilation unit and pop it after to free intermediate
strings, reducing peak RSS during long compile runs.
Combines two orthogonal optimizations:
1. c_escape batching (from alpha): ASCII runs emitted as str_slice segments instead
of one str_char_at string per byte. O(N) allocs → O(K) where K = special chars.
2. scan_interp_string batching (from beta): char dispatch via str_char_code (Int)
+ clean_start tracking to flush plain runs as str_slice. Eliminates per-char
string allocations in the string-literal scanning hot path.
Result on web/src/main.el: 14.5MB -> 13.4MB peak RSS (-7.6%).
Self-hosting: PASS.
Combines two orthogonal optimizations:
1. Flat token list (from beta): lex() returns [Any] with alternating kind/value
pairs instead of [Map], eliminating one ElMap per token (~3 mallocs each).
Parser updated: tok_kind(t,i) = t[2*i], tok_value(t,i) = t[2*i+1].
2. Char code dispatch (from alpha): lex() hot loop uses str_char_code -> Int
instead of str_char_at -> strdup String for all character classification.
Eliminates ~400K x 16B = 6.4MB of temporary string allocations.
scan_digits and scan_ident also updated to use str_char_code.
Result on main.el: 17.1MB -> 14.4MB peak RSS (-16%).
Self-hosting: PASS.
Replace str_char_at (returns strdup String) with str_char_code (returns Int)
in the main lex() while loop and scan_digits/scan_ident helpers.
For a 400KB combined source, str_char_at was allocating ~400K x 16B = 6.4MB
of transient 2-byte strings for the ch variable alone. str_char_code returns
an integer directly — zero allocation.
Add Int-based helpers: is_digit_code, is_alpha_code, is_ws_code,
is_alnum_or_underscore_code. Rewrite lex() operator dispatch using char
code constants (e.g. '/'=47, '"'=34, '='=61).
Result on main.el: 17.1MB -> 15.4MB peak RSS (-10%).
Self-hosting: PASS.
will.anderson
Will Anderson