Compare commits

..

5 Commits

Author SHA1 Message Date
Neuron 40653d2aa1 fix(codegen): emit the declared cgi identity — it was searched for in a list that cannot contain it
El SDK Release / build-and-release (pull_request) Failing after 10m39s
A cgi block is a top-level declaration, so codegen_streaming classifies it via
is_top_level_decl and releases it. The identity emission then searched
toplevel_exec_stmts for that same block. Declarations are excluded from that list by
construction, so the search could never succeed. A probe printed what it actually
saw for a program whose first statement is a cgi block: [Let, Expr]. It emitted
nothing, silently, with no diagnostic on any channel.

The code documented its own assumption — 'Since cgi blocks are rare and small, they
end up in toplevel_exec_stmts' — and that assumption was false.

Capture the declared values before the release and emit from them. The search is
deleted rather than repaired, so the failure mode is removed rather than relocated.

Proven discriminating (old fails, new passes):
  minimal cgi program, old   -> 0 el_cgi_init
  minimal cgi program, fixed -> el_cgi_init with all four declared values
  neuron soul, fixed         -> principal present in the compiled binary (0 before),
                                boots in 2s, interface 110 routes in / 110 out

Consequence: a binary now carries its declared identity as a compiled constant,
which is what the identity protocol requires. Whether the runtime surfaces it to
state_get("soul_principal") is unverified and separate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 13:36:51 -05:00
Tim Lingo f76ccc0590 engram: ranked BM25+recency search replaces storage-order substring; URL-decode GET query params
Measured on the live container mind (pinned 40-query eval, judged): substring
2/40=5% hit@5 -> ranked 35/40=88%. Multi-word queries stop returning zero; new
memories stop losing to storage order (created_at tiebreak). Transparent-layer
identity filter preserved in both passes; jb_finish (#64) tail preserved.
query_param now url_decode()s values - %XX arrived literal before (pre-existing
GET defect, masked while multi-word substring returned nothing anyway).
E2E-verified in Tim's container deployment 2026-07-14/15; eval harness:
docs repo research-archive/p0-prototypes/eval_pinned_40q_20260715.py.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 07:48:09 -05:00
will.anderson 2b2a1246e7 Merge pull request 'runtime: fix the memory-leak + write-corruption pair in el_runtime.c' (#64) from hotfix/el-runtime-leak-and-persist into main
El SDK Release / build-and-release (push) Failing after 10m52s
2026-07-13 21:23:31 +00:00
will.anderson f78da81aa4 runtime: fix the memory leak + write-corruption pair in el_runtime.c
El SDK Release / build-and-release (pull_request) Failing after 11m58s
Two independent investigations, one runtime, complementary halves:

1. Leak (Jul 2, this machine): JsonBuf buffers returned via el_wrap_str
   were raw malloc, never arena-tracked — every engram_*_json call leaked
   its output unconditionally. Added jb_finish() arena-tracking across all
   ~30 return sites. Plus el_arena_push/pop per-tick bracketing support
   for the soul's awareness loop (the loop ran outside any request arena,
   so even correctly-tracked allocations were permanent — 7.5GB RSS in
   under a minute at 1s tick).

2. Corruption (Tim's container soak, docs findings/container-migration):
   stored engram node/edge fields (content, node_type, label, tier, tags,
   metadata, from/to ids) were arena el_strdup — freed at request end,
   leaving dangling pointers that read back as recycled request-buffer
   bytes one request later. This is the June corruption root cause and
   the mechanism that grew snapshot.json to 18GB of empty-type junk
   (21.6M nodes, 3,335 real). 39 sites switched to el_strdup_persist,
   plus a latent double-free fix in engram_load metadata fixup.

Interaction note: fix 1's per-tick arena reclamation makes fix 2
mandatory — more aggressive arena recycling widens the use-after-free
window if stored fields still live in the arena. Apply as a pair, never
separately.

Verified live: soul + engram rebuilt from this runtime, booted against
the recovered real snapshot (3,335 nodes/40,146 edges), 5h stable at
<100MB RSS, write-then-next-request field-integrity test passes (the
June corruption fingerprint does not reproduce). engram/dist/engram
binary updated from this build.

Investigation credit: leak diagnosis this machine Jul 2-6; corruption
diagnosis + persist-fix patch by Tim's instance (docs PR #4).
2026-07-13 16:22:02 -05:00
will.anderson 2597a092bb Merge pull request 'chore: integrate local main commits' (#63) from integrate/local-main-commits into main
El SDK Release / build-and-release (push) Successful in 10m58s
2026-07-01 16:30:17 +00:00
4 changed files with 263 additions and 127 deletions
BIN
View File
Binary file not shown.
+6 -2
View File
@@ -50,8 +50,12 @@ fn query_param(path: String, key: String) -> String {
if pos < 0 { return "" }
let after: String = str_slice(qs, pos + str_len(needle), str_len(qs))
let amp: Int = str_index_of(after, "&")
if amp < 0 { return after }
str_slice(after, 0, amp)
// SPEC-SEARCH-UPGRADE 2026-07-14: URL-decode the extracted value (%XX and
// '+' were previously passed through literally, so an encoded multi-word
// query arrived as junk tokens pre-existing GET-path defect, masked
// until search could actually rank multi-word queries).
if amp < 0 { return url_decode(after) }
url_decode(str_slice(after, 0, amp))
}
fn query_int(path: String, key: String, default_val: Int) -> Int {
+214 -98
View File
@@ -710,6 +710,7 @@ static void jb_init(JsonBuf* b);
static void jb_putc(JsonBuf* b, char c);
static void jb_puts(JsonBuf* b, const char* s);
static void jb_emit_escaped(JsonBuf* b, const char* s);
static char* jb_finish(JsonBuf* b);
static int looks_like_string(el_val_t v);
static const char* json_find_key(const char* s, const char* key);
static const char* json_skip_value(const char* p);
@@ -2014,7 +2015,7 @@ el_val_t http_response(el_val_t status, el_val_t headers_json, el_val_t body) {
jb_puts(&out, ",\"body\":");
jb_emit_escaped(&out, b);
jb_putc(&out, '}');
return el_wrap_str(out.buf);
return el_wrap_str(jb_finish(&out));
}
/* ── Filesystem ──────────────────────────────────────────────────────────── */
@@ -2121,7 +2122,7 @@ el_val_t exec_capture(el_val_t cmdv) {
char buf[4096];
while (fgets(buf, sizeof(buf), f)) jb_puts(&b, buf);
pclose(f);
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
// exec — run a shell command via /bin/sh, capture stdout, return as String.
@@ -2145,7 +2146,7 @@ el_val_t exec(el_val_t cmdv) {
jb_puts(&b, buf);
}
pclose(f);
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
// exec_bg — run a shell command in background, return PID as String.
@@ -3292,6 +3293,23 @@ static void jb_puts(JsonBuf* b, const char* s) {
b->buf[b->len] = '\0';
}
/* jb_finish — call exactly once, at the point a JsonBuf's buffer is handed
* off as an el_val_t return value (via el_wrap_str). JsonBuf allocates through
* raw malloc/realloc (jb_init/jb_reserve), NOT el_strdup/el_strbuf, so unlike
* those helpers it was never registered with the per-request arena every
* JsonBuf-built JSON string leaked unconditionally, whether or not a request
* arena was active, because nothing ever freed it. el_arena_track() is a
* documented no-op when no arena is active, so this is safe to call from any
* context (HTTP request handler, CLI arena-scope, or no arena at all the
* last case still leaks, same as before, until the caller's context is itself
* arena-scoped). Do NOT call this on a JsonBuf whose ->buf is consumed by
* another builder (jb_puts(&other, x.buf)) and freed manually only on the
* one actually returned via el_wrap_str. */
static char* jb_finish(JsonBuf* b) {
el_arena_track(b->buf);
return b->buf;
}
static void jb_emit_escaped(JsonBuf* b, const char* s) {
jb_putc(b, '"');
const unsigned char* p = (const unsigned char*)s;
@@ -3400,7 +3418,7 @@ static void jb_emit_value(JsonBuf* b, el_val_t v) {
el_val_t json_stringify(el_val_t v) {
JsonBuf b; jb_init(&b);
jb_emit_value(&b, v);
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
/* ── JSON substring accessors ────────────────────────────────────────────── */
@@ -3587,7 +3605,7 @@ el_val_t json_set(el_val_t json_str, el_val_t key, el_val_t value) {
jb_putc(&b, ':');
jb_puts(&b, raw_val);
jb_putc(&b, '}');
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
const char* existing = json_find_key(json, k);
JsonBuf b; jb_init(&b);
@@ -3601,7 +3619,7 @@ el_val_t json_set(el_val_t json_str, el_val_t key, el_val_t value) {
b.buf[b.len] = '\0';
jb_puts(&b, raw_val);
jb_puts(&b, end);
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
/* Insert before closing '}'. Find last '}' */
size_t jl = strlen(json);
@@ -3633,7 +3651,7 @@ el_val_t json_set(el_val_t json_str, el_val_t key, el_val_t value) {
jb_puts(&b, raw_val);
/* Append from close_idx onward */
jb_puts(&b, json + close_idx);
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
el_val_t json_array_len(el_val_t json_str) {
@@ -3771,7 +3789,7 @@ el_val_t json_build_object(el_val_t kvs) {
jb_putc(&b, '"');
}
jb_putc(&b, '}');
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
/* json_build_array — build a JSON array from a list of raw JSON values.
@@ -3789,7 +3807,7 @@ el_val_t json_build_array(el_val_t items) {
jb_puts(&b, vs);
}
jb_putc(&b, ']');
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
/* ── Time ────────────────────────────────────────────────────────────────── */
@@ -5193,7 +5211,7 @@ el_val_t state_keys(void) {
}
jb_putc(&b, ']');
pthread_mutex_unlock(&_state_mu);
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
/* Returns 1 (true) if the key is present in the state store, else 0 (false). */
@@ -5399,7 +5417,7 @@ el_val_t str_format(el_val_t fmt, el_val_t data) {
jb_putc(&b, *p);
p++;
}
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
el_val_t str_lower(el_val_t s) { return str_to_lower(s); }
@@ -5815,7 +5833,7 @@ el_val_t list_join(el_val_t listv, el_val_t sep) {
jb_puts(&b, tmp);
}
}
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
el_val_t list_range(el_val_t start, el_val_t end) {
@@ -6465,12 +6483,12 @@ el_val_t engram_node(el_val_t content, el_val_t node_type, el_val_t salience) {
n->id = engram_new_id();
const char* c = EL_CSTR(content);
const char* nt = EL_CSTR(node_type);
n->content = el_strdup(c ? c : "");
n->node_type = el_strdup(nt && *nt ? nt : "Memory");
n->label = engram_first_n_chars(c, 60);
n->tier = el_strdup("Working");
n->tags = el_strdup("");
n->metadata = el_strdup("{}");
n->content = el_strdup_persist(c ? c : "");
n->node_type = el_strdup_persist(nt && *nt ? nt : "Memory");
{ char* _lb = engram_first_n_chars(c, 60); n->label = el_strdup_persist(_lb); } /* persist: stored field must outlive request arena */
n->tier = el_strdup_persist("Working");
n->tags = el_strdup_persist("");
n->metadata = el_strdup_persist("{}");
n->salience = engram_decode_score(salience);
if (n->salience <= 0.0 || n->salience > 1.0) n->salience = 0.5;
n->importance = 0.5;
@@ -6597,11 +6615,11 @@ el_val_t engram_node_layered(el_val_t content, el_val_t node_type, el_val_t labe
const char* lb = EL_CSTR(label);
const char* tg = EL_CSTR(tags);
const char* st = EL_CSTR(status);
n->content = el_strdup(c ? c : "");
n->node_type = el_strdup(nt && *nt ? nt : "Memory");
n->label = el_strdup(lb && *lb ? lb : (c ? engram_first_n_chars(c, 60) : ""));
n->tier = el_strdup("Working");
n->tags = el_strdup(tg ? tg : "");
n->content = el_strdup_persist(c ? c : "");
n->node_type = el_strdup_persist(nt && *nt ? nt : "Memory");
n->label = el_strdup_persist(lb && *lb ? lb : (c ? engram_first_n_chars(c, 60) : ""));
n->tier = el_strdup_persist("Working");
n->tags = el_strdup_persist(tg ? tg : "");
if (st && *st) {
/* Minimal metadata payload: {"status":"..."}. Keep it cheap so
* callers using `status` don't pay JSON parse cost on every read. */
@@ -6610,7 +6628,7 @@ el_val_t engram_node_layered(el_val_t content, el_val_t node_type, el_val_t labe
snprintf(meta, sl, "{\"status\":\"%s\"}", st);
n->metadata = meta;
} else {
n->metadata = el_strdup("{}");
n->metadata = el_strdup_persist("{}");
}
n->salience = engram_decode_score(salience);
n->importance = engram_decode_score(certainty);
@@ -6808,6 +6826,116 @@ static int istr_contains(const char* hay, const char* needle) {
return 0;
}
/* ---- SPEC-SEARCH-UPGRADE-OURS-2026-07-14: ranked search (BM25 + recency) ----
* Replaces first-N-in-storage-order substring matching (measured 13% hit@5 on
* the 15-query pinned eval; ranked model measured 93% offline). Deterministic,
* local, transparent no model call on the hot path. Multi-word queries score
* per-token (rare+concentrated terms weigh most); ties break newest-first so
* fresh memories stop losing to storage order. The transparent-layer identity
* filter is preserved unchanged: hidden self layers stay invisible here and
* surface only via engram_activate the legitimate path. */
#define ENGRAM_BM25_MAX_QTOK 16
#define ENGRAM_BM25_TOKLEN 48
static int engram_tok_next(const char** ps, char* out, int cap) {
const char* s = *ps;
while (*s && !isalnum((unsigned char)*s)) s++;
if (!*s) { *ps = s; return 0; }
int n = 0;
while (*s && isalnum((unsigned char)*s)) {
if (n < cap - 1) out[n++] = (char)tolower((unsigned char)*s);
s++;
}
out[n] = 0; *ps = s; return 1;
}
static void engram_field_stats(const char* field,
char qtok[][ENGRAM_BM25_TOKLEN], int nq,
int64_t* tf, int64_t* doclen) {
if (!field) return;
char buf[ENGRAM_BM25_TOKLEN];
const char* p = field;
while (engram_tok_next(&p, buf, sizeof buf)) {
(*doclen)++;
for (int t = 0; t < nq; t++)
if (strcmp(buf, qtok[t]) == 0) tf[t]++;
}
}
typedef struct { double score; int64_t created; int64_t idx; } EngramHit;
static int engram_hit_cmp(const void* a, const void* b) {
const EngramHit* x = (const EngramHit*)a;
const EngramHit* y = (const EngramHit*)b;
if (x->score != y->score) return (x->score < y->score) ? 1 : -1;
if (x->created != y->created) return (x->created < y->created) ? 1 : -1;
return 0;
}
/* Scores every visible node against the query; writes ranked hits into `out`
* (caller allocates g->node_count entries). Returns min(hits, lim). */
static int64_t engram_search_ranked(EngramStore* g, const char* q, int64_t lim,
EngramHit* out) {
char qtok[ENGRAM_BM25_MAX_QTOK][ENGRAM_BM25_TOKLEN];
int nq = 0;
{
const char* p = q; char buf[ENGRAM_BM25_TOKLEN];
while (nq < ENGRAM_BM25_MAX_QTOK && engram_tok_next(&p, buf, sizeof buf)) {
int dup = 0;
for (int t = 0; t < nq; t++)
if (strcmp(qtok[t], buf) == 0) { dup = 1; break; }
if (!dup) { strcpy(qtok[nq], buf); nq++; }
}
}
if (nq == 0) return 0;
int64_t N = g->node_count;
int64_t* tfm = (int64_t*)calloc((size_t)(N * nq), sizeof(int64_t));
int64_t* dlen = (int64_t*)calloc((size_t)N, sizeof(int64_t));
if (!tfm || !dlen) { free(tfm); free(dlen); return 0; }
int64_t df[ENGRAM_BM25_MAX_QTOK] = {0};
double total_len = 0.0; int64_t live = 0;
for (int64_t i = 0; i < N; i++) {
EngramNode* n = &g->nodes[i];
if (engram_layer_is_transparent(n->layer_id)) continue;
live++;
int64_t* tf = &tfm[i * nq];
engram_field_stats(n->content, qtok, nq, tf, &dlen[i]);
engram_field_stats(n->label, qtok, nq, tf, &dlen[i]);
engram_field_stats(n->tags, qtok, nq, tf, &dlen[i]);
total_len += (double)dlen[i];
for (int t = 0; t < nq; t++) if (tf[t] > 0) df[t]++;
}
double avg = (live > 0) ? total_len / (double)live : 1.0;
if (avg <= 0.0) avg = 1.0;
const double k1 = 1.2, b = 0.75;
int64_t nhits = 0;
for (int64_t i = 0; i < N; i++) {
EngramNode* n = &g->nodes[i];
if (engram_layer_is_transparent(n->layer_id)) continue;
int64_t* tf = &tfm[i * nq];
double s = 0.0;
for (int t = 0; t < nq; t++) {
if (tf[t] == 0) continue;
double idf = log(((double)live - (double)df[t] + 0.5) /
((double)df[t] + 0.5) + 1.0);
double tfd = (double)tf[t];
s += idf * (tfd * (k1 + 1.0)) /
(tfd + k1 * (1.0 - b + b * (double)dlen[i] / avg));
}
if (s > 0.0) {
out[nhits].score = s;
out[nhits].created = n->created_at;
out[nhits].idx = i;
nhits++;
}
}
free(tfm); free(dlen);
qsort(out, (size_t)nhits, sizeof(EngramHit), engram_hit_cmp);
return (nhits < lim) ? nhits : lim;
}
el_val_t engram_search(el_val_t query, el_val_t limit) {
EngramStore* g = engram_get();
const char* q = EL_CSTR(query);
@@ -6815,21 +6943,13 @@ el_val_t engram_search(el_val_t query, el_val_t limit) {
if (lim <= 0) lim = 100;
el_val_t lst = el_list_empty();
if (!q || !*q) return lst;
int64_t found = 0;
for (int64_t i = 0; i < g->node_count && found < lim; i++) {
EngramNode* n = &g->nodes[i];
/* Filter transparent layers: nodes whose layer is `transparent=1`
* shape output but are invisible to introspection ("what do you
* know about yourself"). They still surface via engram_activate
* + engram_compile_layered_json that's the legitimate path. */
if (engram_layer_is_transparent(n->layer_id)) continue;
if (istr_contains(n->content, q) ||
istr_contains(n->label, q) ||
istr_contains(n->tags, q)) {
lst = el_list_append(lst, engram_node_to_map(n));
found++;
}
}
if (g->node_count == 0) return lst;
EngramHit* hits = (EngramHit*)malloc((size_t)g->node_count * sizeof(EngramHit));
if (!hits) return lst;
int64_t k = engram_search_ranked(g, q, lim, hits);
for (int64_t i = 0; i < k; i++)
lst = el_list_append(lst, engram_node_to_map(&g->nodes[hits[i].idx]));
free(hits);
return lst;
}
@@ -6883,10 +7003,10 @@ void engram_connect(el_val_t from_id, el_val_t to_id, el_val_t weight, el_val_t
EngramEdge* e = &g->edges[g->edge_count];
memset(e, 0, sizeof(*e));
e->id = engram_new_id();
e->from_id = el_strdup(f);
e->to_id = el_strdup(t);
e->relation = el_strdup(r && *r ? r : "associate");
e->metadata = el_strdup("{}");
e->from_id = el_strdup_persist(f);
e->to_id = el_strdup_persist(t);
e->relation = el_strdup_persist(r && *r ? r : "associate");
e->metadata = el_strdup_persist("{}");
e->weight = engram_decode_score(weight);
if (e->weight <= 0.0 || e->weight > 1.0) e->weight = 0.5;
e->confidence = 1.0;
@@ -7519,11 +7639,11 @@ el_val_t engram_save(el_val_t path) {
/* Helper: extract a string field from a JSON object substring. */
static char* eg_get_str_field(const char* obj, const char* key) {
const char* p = json_find_key(obj, key);
if (!p) return el_strdup("");
if (*p != '"') return el_strdup("");
if (!p) return el_strdup_persist("");
if (*p != '"') return el_strdup_persist("");
JsonParser jp = { .p = p, .end = p + strlen(p), .err = 0 };
char* out = jp_parse_string_raw(&jp);
if (jp.err) { free(out); return el_strdup(""); }
if (jp.err) { free(out); return el_strdup_persist(""); }
return out;
}
@@ -7597,7 +7717,7 @@ el_val_t engram_load(el_val_t path) {
nn->tier = eg_get_str_field(obj, "tier");
nn->tags = eg_get_str_field(obj, "tags");
nn->metadata = eg_get_str_field(obj, "metadata");
if (!nn->metadata || !*nn->metadata) { free(nn->metadata); nn->metadata = el_strdup("{}"); }
if (!nn->metadata || !*nn->metadata) { free(nn->metadata); nn->metadata = el_strdup_persist("{}"); }
nn->salience = eg_get_num_field(obj, "salience");
nn->importance = eg_get_num_field(obj, "importance");
nn->confidence = eg_get_num_field(obj, "confidence");
@@ -7648,7 +7768,7 @@ el_val_t engram_load(el_val_t path) {
ee->to_id = eg_get_str_field(obj, "to_id");
ee->relation = eg_get_str_field(obj, "relation");
ee->metadata = eg_get_str_field(obj, "metadata");
if (!ee->metadata || !*ee->metadata) { free(ee->metadata); ee->metadata = el_strdup("{}"); }
if (!ee->metadata || !*ee->metadata) { free(ee->metadata); ee->metadata = el_strdup_persist("{}"); }
ee->weight = eg_get_num_field(obj, "weight");
ee->confidence = eg_get_num_field(obj, "confidence");
ee->created_at = eg_get_int_field(obj, "created_at");
@@ -7740,35 +7860,31 @@ el_val_t engram_get_node_json(el_val_t id) {
if (!n) return el_wrap_str(el_strdup("{}"));
JsonBuf b; jb_init(&b);
engram_emit_node_json(&b, n);
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
el_val_t engram_search_json(el_val_t query, el_val_t limit) {
/* SPEC-SEARCH-UPGRADE 2026-07-14: same ranked BM25+recency core as
* engram_search; transparent-layer identity filter enforced inside it. */
EngramStore* g = engram_get();
const char* q = EL_CSTR(query);
int64_t lim = (int64_t)limit;
if (lim <= 0) lim = 100;
JsonBuf b; jb_init(&b);
jb_putc(&b, '[');
int first = 1;
int64_t found = 0;
if (q && *q) {
for (int64_t i = 0; i < g->node_count && found < lim; i++) {
EngramNode* n = &g->nodes[i];
/* Filter transparent layers — same as engram_search. */
if (engram_layer_is_transparent(n->layer_id)) continue;
if (istr_contains(n->content, q) ||
istr_contains(n->label, q) ||
istr_contains(n->tags, q)) {
if (!first) jb_putc(&b, ',');
engram_emit_node_json(&b, n);
first = 0;
found++;
if (q && *q && g->node_count > 0) {
EngramHit* hits = (EngramHit*)malloc((size_t)g->node_count * sizeof(EngramHit));
if (hits) {
int64_t k = engram_search_ranked(g, q, lim, hits);
for (int64_t i = 0; i < k; i++) {
if (i) jb_putc(&b, ',');
engram_emit_node_json(&b, &g->nodes[hits[i].idx]);
}
free(hits);
}
}
jb_putc(&b, ']');
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
el_val_t engram_scan_nodes_json(el_val_t limit, el_val_t offset) {
@@ -7777,9 +7893,9 @@ el_val_t engram_scan_nodes_json(el_val_t limit, el_val_t offset) {
int64_t off = (int64_t)offset; if (off < 0) off = 0;
JsonBuf b; jb_init(&b);
jb_putc(&b, '[');
if (g->node_count == 0) { jb_putc(&b, ']'); return el_wrap_str(b.buf); }
if (g->node_count == 0) { jb_putc(&b, ']'); return el_wrap_str(jb_finish(&b)); }
int64_t* idx = malloc((size_t)g->node_count * sizeof(int64_t));
if (!idx) { jb_putc(&b, ']'); return el_wrap_str(b.buf); }
if (!idx) { jb_putc(&b, ']'); return el_wrap_str(jb_finish(&b)); }
/* Skip transparent layers — introspection filter, same as engram_scan_nodes. */
int64_t live = 0;
for (int64_t i = 0; i < g->node_count; i++) {
@@ -7797,7 +7913,7 @@ el_val_t engram_scan_nodes_json(el_val_t limit, el_val_t offset) {
}
free(idx);
jb_putc(&b, ']');
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
/* engram_scan_nodes_by_type_json — filter by node_type before paginating.
@@ -7813,9 +7929,9 @@ el_val_t engram_scan_nodes_by_type_json(el_val_t type_v, el_val_t limit, el_val_
int64_t off = (int64_t)offset; if (off < 0) off = 0;
JsonBuf b; jb_init(&b);
jb_putc(&b, '[');
if (g->node_count == 0) { jb_putc(&b, ']'); return el_wrap_str(b.buf); }
if (g->node_count == 0) { jb_putc(&b, ']'); return el_wrap_str(jb_finish(&b)); }
int64_t* idx = malloc((size_t)g->node_count * sizeof(int64_t));
if (!idx) { jb_putc(&b, ']'); return el_wrap_str(b.buf); }
if (!idx) { jb_putc(&b, ']'); return el_wrap_str(jb_finish(&b)); }
int64_t live = 0;
for (int64_t i = 0; i < g->node_count; i++) {
if (engram_layer_is_transparent(g->nodes[i].layer_id)) continue;
@@ -7834,7 +7950,7 @@ el_val_t engram_scan_nodes_by_type_json(el_val_t type_v, el_val_t limit, el_val_
}
free(idx);
jb_putc(&b, ']');
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
el_val_t engram_neighbors_json(el_val_t node_id, el_val_t max_depth, el_val_t direction) {
@@ -7849,7 +7965,7 @@ el_val_t engram_neighbors_json(el_val_t node_id, el_val_t max_depth, el_val_t di
int allow_in = (strcmp(dir, "in") == 0) || (strcmp(dir, "both") == 0);
JsonBuf b; jb_init(&b);
jb_putc(&b, '[');
if (!sid || !*sid) { jb_putc(&b, ']'); return el_wrap_str(b.buf); }
if (!sid || !*sid) { jb_putc(&b, ']'); return el_wrap_str(jb_finish(&b)); }
/* Frontier of (node_id, hops). Cap to a sane size. */
char** frontier = calloc(1024, sizeof(char*));
@@ -7859,7 +7975,7 @@ el_val_t engram_neighbors_json(el_val_t node_id, el_val_t max_depth, el_val_t di
int64_t vc = 0;
if (!frontier || !frontier_h || !visited) {
free(frontier); free(frontier_h); free(visited);
jb_putc(&b, ']'); return el_wrap_str(b.buf);
jb_putc(&b, ']'); return el_wrap_str(jb_finish(&b));
}
/* Use plain strdup (not el_strdup) so arena doesn't track these pointers.
* The BFS loop manually frees them below arena would double-free them. */
@@ -7902,7 +8018,7 @@ el_val_t engram_neighbors_json(el_val_t node_id, el_val_t max_depth, el_val_t di
for (int64_t i = 0; i < vc; i++) free(visited[i]);
free(frontier); free(frontier_h); free(visited);
jb_putc(&b, ']');
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
el_val_t engram_activate_json(el_val_t query, el_val_t depth) {
@@ -7943,7 +8059,7 @@ el_val_t engram_activate_json(el_val_t query, el_val_t depth) {
}
}
jb_putc(&b, ']');
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
el_val_t engram_stats_json(void) {
@@ -7963,7 +8079,7 @@ el_val_t engram_list_layers_json(void) {
jb_putc(&b, '[');
/* Build a sorted index over live layers. */
size_t* idx = malloc((g->layer_count + 1) * sizeof(size_t));
if (!idx) { jb_putc(&b, ']'); return el_wrap_str(b.buf); }
if (!idx) { jb_putc(&b, ']'); return el_wrap_str(jb_finish(&b)); }
size_t live = 0;
for (size_t i = 0; i < g->layer_count; i++) {
if (g->layers[i].name) idx[live++] = i;
@@ -8000,7 +8116,7 @@ el_val_t engram_list_layers_json(void) {
}
free(idx);
jb_putc(&b, ']');
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
/* engram_compile_layered_json — produce a prompt-ready context block split
@@ -8086,7 +8202,7 @@ el_val_t engram_compile_layered_json(el_val_t intent, el_val_t depth) {
free(b.buf);
return el_wrap_str(el_strdup(""));
}
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
/* engram_query_range — temporal range query.
@@ -8131,7 +8247,7 @@ el_val_t engram_query_range(el_val_t start_ms_v, el_val_t end_ms_v) {
}
jb_putc(&b, ']');
free(idx);
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
/* engram_load_merge — like engram_load but WITHOUT resetting the store.
@@ -8382,7 +8498,7 @@ el_val_t engram_wm_top_json(el_val_t n_v) {
}
free(idx);
jb_putc(&b, ']');
return el_wrap_str(b.buf);
return el_wrap_str(jb_finish(&b));
}
#ifdef HAVE_CURL
@@ -8778,13 +8894,13 @@ static int64_t dharma_find_or_create_relation_edge(const char* peer_base, int cr
engram_grow_nodes();
EngramNode* n = &g->nodes[g->node_count];
memset(n, 0, sizeof(*n));
n->id = el_strdup(self_id);
n->content = el_strdup(_el_cgi_dharma_id ? _el_cgi_dharma_id : "(self)");
n->node_type = el_strdup("DharmaSelf");
n->label = el_strdup("dharma:self");
n->tier = el_strdup("Working");
n->tags = el_strdup("dharma");
n->metadata = el_strdup("{}");
n->id = el_strdup_persist(self_id);
n->content = el_strdup_persist(_el_cgi_dharma_id ? _el_cgi_dharma_id : "(self)");
n->node_type = el_strdup_persist("DharmaSelf");
n->label = el_strdup_persist("dharma:self");
n->tier = el_strdup_persist("Working");
n->tags = el_strdup_persist("dharma");
n->metadata = el_strdup_persist("{}");
n->salience = 1.0; n->importance = 1.0; n->confidence = 1.0;
int64_t now = engram_now_ms();
n->created_at = now; n->updated_at = now; n->last_activated = now;
@@ -8795,13 +8911,13 @@ static int64_t dharma_find_or_create_relation_edge(const char* peer_base, int cr
engram_grow_nodes();
EngramNode* n = &g->nodes[g->node_count];
memset(n, 0, sizeof(*n));
n->id = el_strdup(peer_node);
n->content = el_strdup(peer_base);
n->node_type = el_strdup("DharmaPeer");
n->label = el_strdup(peer_node);
n->tier = el_strdup("Working");
n->tags = el_strdup("dharma");
n->metadata = el_strdup("{}");
n->id = el_strdup_persist(peer_node);
n->content = el_strdup_persist(peer_base);
n->node_type = el_strdup_persist("DharmaPeer");
n->label = el_strdup_persist(peer_node);
n->tier = el_strdup_persist("Working");
n->tags = el_strdup_persist("dharma");
n->metadata = el_strdup_persist("{}");
n->salience = 0.5; n->importance = 0.5; n->confidence = 1.0;
int64_t now = engram_now_ms();
n->created_at = now; n->updated_at = now; n->last_activated = now;
@@ -8813,10 +8929,10 @@ static int64_t dharma_find_or_create_relation_edge(const char* peer_base, int cr
EngramEdge* e = &g->edges[g->edge_count];
memset(e, 0, sizeof(*e));
e->id = engram_new_id();
e->from_id = el_strdup(self_id);
e->to_id = el_strdup(peer_node);
e->relation = el_strdup("dharma-relation");
e->metadata = el_strdup("{}");
e->from_id = el_strdup_persist(self_id);
e->to_id = el_strdup_persist(peer_node);
e->relation = el_strdup_persist("dharma-relation");
e->metadata = el_strdup_persist("{}");
e->weight = 0.0;
e->confidence = 1.0;
int64_t now = engram_now_ms();
@@ -9109,7 +9225,7 @@ static el_val_t llm_extract_text(el_val_t resp_val) {
free(obj);
p = end;
}
return el_wrap_str(out.buf);
return el_wrap_str(jb_finish(&out));
}
el_val_t llm_call(el_val_t model, el_val_t prompt) {
+43 -27
View File
@@ -3626,6 +3626,24 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
let pos: Int = 0
let el_main_body: [Map<String, Any>] = native_list_empty()
let toplevel_exec_stmts: [Map<String, Any>] = native_list_empty()
// CGI IDENTITY CAPTURE (2026-08-09). A cgi block is a top-level DECLARATION, so
// the classifier below correctly excludes it from toplevel_exec_stmts and calls
// el_release on it. The identity emission further down then searched
// toplevel_exec_stmts for it a list that structurally can never contain it
// found nothing, and emitted nothing, silently. Measured: that search sees only
// [Let, Expr] for a program whose first statement is a cgi block.
// Fix: copy the values out BEFORE the release (strings, so no dangling reference)
// and emit from these. No search, so the failure mode is removed rather than moved.
let cgi_have: Bool = false
let cgi_name_v: String = ""
let cgi_did_v: String = ""
let cgi_prin_v: String = ""
let cgi_net_v: String = ""
let cgi_eng_v: String = ""
let cgi_has_did: Bool = false
let cgi_has_prin: Bool = false
let cgi_has_net: Bool = false
let cgi_has_eng: Bool = false
let has_toplevel_exec: Bool = false
let stream_running: Bool = true
@@ -3736,6 +3754,20 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
if is_top_level_decl(stmt) {
// Import, TypeDef, EnumDef, CgiBlock, ServiceBlock, ExternFn
// These are no-ops in codegen (forward decls already emitted)
// except a CgiBlock, whose declared identity must survive
// this release to be emitted as a compiled constant.
if str_eq(sk, "CgiBlock") {
let cgi_have = true
let cgi_name_v = stmt["name"]
let cgi_did_v = stmt["dharma_id"]
let cgi_prin_v = stmt["principal"]
let cgi_net_v = stmt["network"]
let cgi_eng_v = stmt["engram"]
let cgi_has_did = stmt["has_dharma_id"]
let cgi_has_prin = stmt["has_principal"]
let cgi_has_net = stmt["has_network"]
let cgi_has_eng = stmt["has_engram"]
}
el_release(stmt)
} else {
if str_eq(sk, "Let") {
@@ -3814,33 +3846,17 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
let sig2 = native_list_get(sigs, si2)
let sk3: String = sig2["kind"]
if str_eq(sk3, "cgi_block") {
// We need the full cgi_block data it was parsed by scan_fn_sigs
// but scan only stored the name. For cgi_init we need dharma_id etc.
// Since cgi blocks are rare and small, they end up in toplevel_exec_stmts.
// Find the CgiBlock in toplevel_exec_stmts.
let tes_n: Int = native_list_len(toplevel_exec_stmts)
let tes_i: Int = 0
while tes_i < tes_n {
let tes = native_list_get(toplevel_exec_stmts, tes_i)
let tes_k: String = tes["stmt"]
if str_eq(tes_k, "CgiBlock") {
let cname2: String = tes["name"]
let cdid2: String = tes["dharma_id"]
let cprin2: String = tes["principal"]
let cnet2: String = tes["network"]
let ceng2: String = tes["engram"]
let has_did2: Bool = tes["has_dharma_id"]
let has_prin2: Bool = tes["has_principal"]
let has_net2: Bool = tes["has_network"]
let has_eng2: Bool = tes["has_engram"]
let arg_name2: String = "EL_STR(" + c_str_lit(cname2) + ")"
let arg_did2: String = cgi_arg(cdid2, has_did2)
let arg_prin2: String = cgi_arg(cprin2, has_prin2)
let arg_net2: String = cgi_arg(cnet2, has_net2)
let arg_eng2: String = cgi_arg(ceng2, has_eng2)
emit_line(" el_cgi_init(" + arg_name2 + ", " + arg_did2 + ", " + arg_prin2 + ", " + arg_net2 + ", " + arg_eng2 + ");")
}
let tes_i = tes_i + 1
// Emit from the values captured before the declaration was released.
// The previous implementation searched toplevel_exec_stmts, which by
// construction never contains a declaration so it emitted nothing and
// said nothing. See the capture block near toplevel_exec_stmts init.
if cgi_have {
let arg_name2: String = "EL_STR(" + c_str_lit(cgi_name_v) + ")"
let arg_did2: String = cgi_arg(cgi_did_v, cgi_has_did)
let arg_prin2: String = cgi_arg(cgi_prin_v, cgi_has_prin)
let arg_net2: String = cgi_arg(cgi_net_v, cgi_has_net)
let arg_eng2: String = cgi_arg(cgi_eng_v, cgi_has_eng)
emit_line(" el_cgi_init(" + arg_name2 + ", " + arg_did2 + ", " + arg_prin2 + ", " + arg_net2 + ", " + arg_eng2 + ");")
}
}
let si2 = si2 + 1