epm's sibling modules (registry/install/update) call functions defined in other
modules and in the El runtime (config, read_installed, registry_find,
manifest_deps, manifest_name, registry_latest_version, registry_token,
install_vessel, installed_version) without importing them, so elc emits no C
prototype for those calls. gcc<=13 treated the resulting implicit declarations
as warnings; gcc>=14 and clang reject them as hard errors, which is why the
"Build epm" CI step fails and blocks the whole dev/stage pipeline.
Add `extern fn` forward declarations -- El's own separate-compilation mechanism
-- for each cross-module callee at the top of registry/install/update. This
gives elc the correct C prototype in every generated translation unit, so the
calls compile cleanly and still resolve at link time. Simply suppressing
-Wimplicit-function-declaration would be unsafe: an implicit int return
truncates the 64-bit pointer returns of config/registry_find into a latent
crash, so declaring the true signatures is the correct fix. Localized to epm;
touches neither elc nor the runtime.
Two el_runtime portability defects only ever lived in staged local copies
used to hand-build neuron-ui PR #136's curl-enabled Windows neuron.exe.
gcc 15 promotes both to hard errors, so a clean el checkout cannot rebuild
that soul. Upstream the minimal fixes so the build is reproducible:
- http_serve_async: cast setsockopt optval to (const char*). Win32/mingw
setsockopt wants const char*, not int*; the cast is a no-op on POSIX and
matches the four already-cast sites elsewhere in this file.
- engram_save persist path: map fsync -> _commit in the _WIN32-only
el_platform_win.h (io.h already included). Windows has no fsync(); the
POSIX path is untouched.
chat.el calls the runtime native engram_get_node_by_label to fetch
well-known nodes (conv:history, session:summary) by stable label rather
than by ID — immune to vector-index drift across restarts. The current
runtime never defined it, so the regenerated dist/soul.c fails to link.
Backport the function verbatim (idiom-adapted to jb_finish) from release
runtime v1.0.0-20260501 and register it as an EL builtin exactly like its
siblings: runtime definition + prototype, __-prefixed seed wrapper +
prototype, and codegen arity entry. No search-site code is touched.
Lexical istr_contains alone can't surface a node whose words don't appear
in the query. This adds an optional dense-vector layer: node content and the
query are embedded through Ollama (nomic-embed-text), and nodes are ranked by
cosine similarity unioned with lexical hits, so a paraphrase query reaches the
right node.
Wired into all three query entry points in el_runtime.c:
- engram_search_json (HTTP /api/search): collect lexical ∪ semantic
candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine),
rank, emit top-N. Stable sort preserves old order when semantic is off.
- engram_search (internal el_val twin): lexical ∪ semantic union.
- engram_activate seed loop (HTTP /api/activate): a node seeds if it
lexically matches OR clears the cosine threshold; pure-semantic seeds
enter scaled by cosine so paraphrase spreads without overpowering.
Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot
runtime probe. If curl is compiled out, Ollama is unreachable, or
ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers
fall back byte-for-byte to the pre-existing lexical search.
Node embeddings are cached in process memory keyed by node id with an FNV-1a
content hash for invalidation; the query is embedded once per call — so the
graph is not re-embedded on every query. nomic task prefixes
(search_query:/search_document:) are applied for retrieval separation.
Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in
(-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL,
ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).
engram search/activate/goal-bias matched the ENTIRE raw query string as a
single case-insensitive substring (istr_contains(field, q)). Multi-word
queries like "windows msi signing" only matched a node containing that exact
contiguous run, so real multi-word queries returned ZERO on a graph saturated
with the answer. This is Ctrl-F, not search — and search is the core of the
engram being useful.
Fix: split the query on whitespace into distinct tokens; a node matches if it
contains ANY token in content/label/tags. Rank by distinct tokens matched
(desc) then salience (desc). istr_contains is kept unchanged as the per-token
primitive. Single-token queries are a strict special case (score 0 or 1) so
the many single-word callers do not regress.
Sites changed (all in el_runtime.c):
- new helpers engram_tokenize_query / engram_node_match_score / engram_rank_cmp
- engram_search (internal el_val_t path)
- engram_search_json (HTTP /api/search path)
- engram_activate seed loop (HTTP /api/activate path; seed activation scaled
by token coverage so full-query matches seed more strongly)
- engram_goal_bias overlap bonus upgraded to graded token coverage
Proof (6591-node snapshot copy, rebuilt binary on :8799, POST JSON path):
windows msi signing 0 -> 20 Will Anderson 0 -> 20
windows msi 0 -> 20 tokenized search fix 0 -> 20
Single-word parity preserved (VBD/volatility/elc capped at limit; unkey = all
matching nodes). Top hits are relevant (e.g. "Will Anderson" surfaces the
Project Design and VBD whitepapers).
Note: GET ?q=a%20b still returns 0 because query_param (server.el) does not
URL-decode — a separate EL-layer bug; the soul's POST-JSON path is fixed here.
Out-of-range tok_kind/tok_value reads returned runtime null (el_list_get OOB
-> 0) rather than the Eof sentinel, so the inner parse loops (parse_block,
call-arg, array-literal, match-arm) that terminate only on their close
delimiter or k=="Eof" never saw Eof once the cursor ran past the single
trailing Eof token. On unclosed-delimiter input the parser then appended AST
nodes forever -> unbounded allocation -> ~700GB -> OOM (observed compiling
neuron/sessions.el).
Fix at the choke point: tok_kind returns "Eof" and tok_value returns "" for
out-of-range positions, restoring the parser-wide contract that reads at/after
the end yield Eof. expect() no longer steps past the Eof sentinel on mismatch.
This terminates every overrun loop simultaneously; a malformed program now
surfaces as a normal (best-effort) parse end instead of exhausting memory.
Requires a self-hosted bootstrap rebuild of elc to take effect.
Two independent investigations, one runtime, complementary halves:
1. Leak (Jul 2, this machine): JsonBuf buffers returned via el_wrap_str
were raw malloc, never arena-tracked — every engram_*_json call leaked
its output unconditionally. Added jb_finish() arena-tracking across all
~30 return sites. Plus el_arena_push/pop per-tick bracketing support
for the soul's awareness loop (the loop ran outside any request arena,
so even correctly-tracked allocations were permanent — 7.5GB RSS in
under a minute at 1s tick).
2. Corruption (Tim's container soak, docs findings/container-migration):
stored engram node/edge fields (content, node_type, label, tier, tags,
metadata, from/to ids) were arena el_strdup — freed at request end,
leaving dangling pointers that read back as recycled request-buffer
bytes one request later. This is the June corruption root cause and
the mechanism that grew snapshot.json to 18GB of empty-type junk
(21.6M nodes, 3,335 real). 39 sites switched to el_strdup_persist,
plus a latent double-free fix in engram_load metadata fixup.
Interaction note: fix 1's per-tick arena reclamation makes fix 2
mandatory — more aggressive arena recycling widens the use-after-free
window if stored fields still live in the arena. Apply as a pair, never
separately.
Verified live: soul + engram rebuilt from this runtime, booted against
the recovered real snapshot (3,335 nodes/40,146 edges), 5h stable at
<100MB RSS, write-then-next-request field-integrity test passes (the
June corruption fingerprint does not reproduce). engram/dist/engram
binary updated from this build.
Investigation credit: leak diagnosis this machine Jul 2-6; corruption
diagnosis + persist-fix patch by Tim's instance (docs PR #4).
Runtime now includes engram_load_merge — soul daemon awareness.el calls
this function during its periodic sync refresh cycle. Binary rebuilt from
server.el (unchanged source) + updated el_runtime.c.
engram_load_merge was added to el-compiler/runtime in 35c1897 but never
ported to the released runtime used by Engram and the soul daemon.
awareness.el calls engram_load_merge in its sync refresh cycle; without
this function in lang/releases/v1.0.0-20260501/el_runtime.c the soul
daemon fails to compile.
Also adds header declarations for engram_wm_count, engram_wm_avg_weight,
engram_wm_top_json, and engram_load_merge — all four were added as
implementations (da116b2 / 35c1897) but their prototypes were missing from
el_runtime.h, causing implicit-function-declaration warnings and potential
ABI breakage on stricter compilers.
Identified during self-review 2026-06-30.
Port critical WM fixes from self-review 2026-06-26 branch (f7bd99a) that were
never merged to HEAD. Running binary had these fixes; source did not — rebuild
would have silently regressed all three improvements.
1. ENGRAM_BREAKTHROUGH_WEIGHT 0.25→0.10
With 0.25, naturally-promoted nodes (threshold ≥0.15) decayed below the
breakthrough floor within one activation call and lost their WM slot to
fresh breakthrough candidates. All 524/525 WM nodes were at floor = useless.
Invariant: BREAKTHROUGH_WEIGHT < min(type_thresholds = 0.15 Canonical).
2. ENGRAM_WM_CAP=24 with Pass 4 (per-call) + Pass 5 (global) enforcement
Without cap, broad curiosity seeds promote 500+ nodes simultaneously.
wm_avg_weight collapses, goal-bias differentiation is lost. Verified:
"knowledge" query now promotes exactly 24 nodes (was 525). Cowan (2001)
cognitive basis: WM capacity ~4 chunks; 24 allows rich multi-topic context.
3. ISE exclusion from WM (Pass 2 guard)
InternalStateEvent JSON content ("knowledge", "memory", etc.) triggered
lexical seeding → suppression accumulation → breakthrough at floor. ISEs
are observability-only and must never surface in context compilation.
suppression_count cleared so ISEs never build toward breakthrough.
4. route_create_ise importance fix (0.5→0.3)
Corrects mismatch between HTTP route and awareness.el in-process fallback.
Also adds body comment clarifying auth-exempt rationale.
SYNAPSE (arXiv 2601.02744) validates WM cap design and ISE exclusion principle.
Next priority: cosine similarity seeding to complement lexical BFS.
2026-06-30 08:48:19 -05:00
17 changed files with 1333 additions and 260 deletions
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
will.anderson