From b4967af13e966bf4eaa048df1a60f498b85dbe46 Mon Sep 17 00:00:00 2001 From: Will Anderson Date: Tue, 14 Jul 2026 18:48:16 -0500 Subject: [PATCH 1/9] =?UTF-8?q?feat(engram):=20semantic=20search=20layer?= =?UTF-8?q?=20via=20nomic-embed-text=20(cosine=20=E2=88=AA=20lexical)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lexical istr_contains alone can't surface a node whose words don't appear in the query. This adds an optional dense-vector layer: node content and the query are embedded through Ollama (nomic-embed-text), and nodes are ranked by cosine similarity unioned with lexical hits, so a paraphrase query reaches the right node. Wired into all three query entry points in el_runtime.c: - engram_search_json (HTTP /api/search): collect lexical ∪ semantic candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine), rank, emit top-N. Stable sort preserves old order when semantic is off. - engram_search (internal el_val twin): lexical ∪ semantic union. - engram_activate seed loop (HTTP /api/activate): a node seeds if it lexically matches OR clears the cosine threshold; pure-semantic seeds enter scaled by cosine so paraphrase spreads without overpowering. Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot runtime probe. If curl is compiled out, Ollama is unreachable, or ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers fall back byte-for-byte to the pre-existing lexical search. Node embeddings are cached in process memory keyed by node id with an FNV-1a content hash for invalidation; the query is embedded once per call — so the graph is not re-embedded on every query. nomic task prefixes (search_query:/search_document:) are applied for retrieval separation. Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in (-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL, ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6). --- engram/.gitea/workflows/ci-dev.yaml | 2 +- engram/.gitea/workflows/ci-stage.yaml | 2 +- engram/.gitea/workflows/engram-release.yaml | 2 +- lang/el-compiler/runtime/el_runtime.c | 331 ++++++++++++++++++-- 4 files changed, 314 insertions(+), 23 deletions(-) diff --git a/engram/.gitea/workflows/ci-dev.yaml b/engram/.gitea/workflows/ci-dev.yaml index 7463aa8..a432869 100644 --- a/engram/.gitea/workflows/ci-dev.yaml +++ b/engram/.gitea/workflows/ci-dev.yaml @@ -81,7 +81,7 @@ jobs: # Link to produce the engram binary - name: Link engram binary run: | - cc -std=c11 -O2 \ + cc -std=c11 -O2 -DHAVE_CURL \ -I /usr/local/lib/el \ -o dist/engram \ dist/engram.c \ diff --git a/engram/.gitea/workflows/ci-stage.yaml b/engram/.gitea/workflows/ci-stage.yaml index 69ed162..a134117 100644 --- a/engram/.gitea/workflows/ci-stage.yaml +++ b/engram/.gitea/workflows/ci-stage.yaml @@ -88,7 +88,7 @@ jobs: # Link to produce the engram binary - name: Link engram binary run: | - cc -std=c11 -O2 \ + cc -std=c11 -O2 -DHAVE_CURL \ -I /usr/local/lib/el \ -o dist/engram \ dist/engram.c \ diff --git a/engram/.gitea/workflows/engram-release.yaml b/engram/.gitea/workflows/engram-release.yaml index bdf3258..aa02e9f 100644 --- a/engram/.gitea/workflows/engram-release.yaml +++ b/engram/.gitea/workflows/engram-release.yaml @@ -62,7 +62,7 @@ jobs: # Link to produce the engram binary - name: Link engram binary run: | - cc -std=c11 -O2 \ + cc -std=c11 -O2 -DHAVE_CURL \ -I /usr/local/lib/el \ -o dist/engram \ dist/engram.c \ diff --git a/lang/el-compiler/runtime/el_runtime.c b/lang/el-compiler/runtime/el_runtime.c index 68e6dca..4e41611 100644 --- a/lang/el-compiler/runtime/el_runtime.c +++ b/lang/el-compiler/runtime/el_runtime.c @@ -6826,6 +6826,243 @@ static int istr_contains(const char* hay, const char* needle) { return 0; } +/* ══════════════════════════════════════════════════════════════════════════ + * SEMANTIC SEARCH LAYER — nomic-embed-text via Ollama /api/embeddings + * ────────────────────────────────────────────────────────────────────────── + * Augments the lexical (istr_contains) matcher with dense-vector retrieval. + * Node content and the query are embedded through a local Ollama server; + * nodes are ranked by cosine similarity and UNIONED with lexical hits. This + * lets a paraphrase query surface a node whose words never appear in it. + * + * DEGRADABLE BY DESIGN. The whole layer is gated on HAVE_CURL plus a one-shot + * runtime probe of the embedding endpoint. If curl is not compiled in, or + * Ollama is unreachable, or ENGRAM_SEMANTIC=0, every entry point returns + * "no semantic signal" and callers fall back to pure lexical behaviour — + * byte-for-byte the pre-existing search. + * + * CACHE. Node embeddings are computed lazily on first use and cached in + * process memory keyed by node id, with an FNV-1a content hash for + * invalidation (edited content re-embeds). The query is embedded once per + * search call. This is what "avoid re-embedding the whole graph every query" + * buys us: a warm cache serves cosine from RAM. (A cold process still pays + * O(N) embed calls the first time each node is scanned — persisting the cache + * to a snapshot sidecar is the documented next step, not done here.) + * + * nomic task prefixes ("search_query:" / "search_document:") are applied + * because nomic-embed-text is trained with them; they materially improve + * retrieval separation (empirically: paraphrase 0.72 vs distractors <0.48). + * + * ENV: + * ENGRAM_SEMANTIC "0" disables; unset/other = auto-probe + * ENGRAM_EMBED_URL default http://localhost:11434/api/embeddings + * ENGRAM_EMBED_MODEL default nomic-embed-text + * ENGRAM_SEMANTIC_MIN cosine threshold for a pure-semantic match (def 0.6) + * ════════════════════════════════════════════════════════════════════════ */ + +static double engram_semantic_min(void) { + static double v = -1.0; + if (v >= 0.0) return v; + const char* s = getenv("ENGRAM_SEMANTIC_MIN"); + double d = 0.6; + if (s && *s) { char* e = NULL; double t = strtod(s, &e); + if (e != s && t >= 0.0 && t <= 1.0) d = t; } + v = d; return v; +} + +#ifdef HAVE_CURL + +typedef struct { char* id; uint64_t hash; float* vec; int dim; } EngramEmbEntry; +static EngramEmbEntry* g_emb_items = NULL; +static int64_t g_emb_count = 0, g_emb_cap = 0; +static int g_emb_state = 0; /* 0=unprobed, 1=available, -1=disabled */ + +static uint64_t engram_fnv1a(const char* s) { + uint64_t h = 1469598103934665603ULL; + if (s) for (const unsigned char* p = (const unsigned char*)s; *p; p++) { + h ^= *p; h *= 1099511628211ULL; + } + return h; +} + +/* Parse "embedding":[f,f,...] from an Ollama response. malloc'd vec, or NULL. */ +static float* engram_parse_embedding(const char* json, int* out_dim) { + if (!json) return NULL; + const char* p = strstr(json, "\"embedding\""); + if (!p) return NULL; + p = strchr(p, '['); + if (!p) return NULL; + p++; + int cap = 1024, n = 0; + float* v = malloc((size_t)cap * sizeof(float)); + if (!v) return NULL; + while (*p && *p != ']') { + while (*p == ' ' || *p == '\t' || *p == '\n' || *p == '\r' || *p == ',') p++; + if (*p == ']' || !*p) break; + char* e = NULL; + double d = strtod(p, &e); + if (e == p) break; + if (n >= cap) { cap *= 2; float* nv = realloc(v, (size_t)cap * sizeof(float)); + if (!nv) { free(v); return NULL; } v = nv; } + v[n++] = (float)d; + p = e; + } + if (n == 0) { free(v); return NULL; } + *out_dim = n; + return v; +} + +/* JSON-escape src into a malloc'd buffer (no surrounding quotes). */ +static char* engram_json_escape(const char* src) { + if (!src) src = ""; + size_t n = strlen(src); + char* out = malloc(n * 2 + 1); + if (!out) return NULL; + size_t j = 0; + for (size_t i = 0; i < n; i++) { + unsigned char c = (unsigned char)src[i]; + if (c == '"') { out[j++] = '\\'; out[j++] = '"'; } + else if (c == '\\') { out[j++] = '\\'; out[j++] = '\\'; } + else if (c == '\n') { out[j++] = '\\'; out[j++] = 'n'; } + else if (c == '\r') { out[j++] = '\\'; out[j++] = 'r'; } + else if (c == '\t') { out[j++] = '\\'; out[j++] = 't'; } + else if (c < 0x20) { /* drop other control bytes */ } + else { out[j++] = (char)c; } + } + out[j] = '\0'; + return out; +} + +/* Embed `prefix+text` via Ollama. Returns malloc'd vec (caller frees), or NULL. */ +static float* engram_embed_raw(const char* prefix, const char* text, int* out_dim) { + if (!text) return NULL; + const char* url = getenv("ENGRAM_EMBED_URL"); + if (!url || !*url) url = "http://localhost:11434/api/embeddings"; + const char* model = getenv("ENGRAM_EMBED_MODEL"); + if (!model || !*model) model = "nomic-embed-text"; + /* Bound content length to keep latency/memory sane on huge nodes. */ + char* trunc = NULL; + size_t maxlen = 8192; + if (strlen(text) > maxlen) { + trunc = malloc(maxlen + 1); + if (trunc) { memcpy(trunc, text, maxlen); trunc[maxlen] = '\0'; text = trunc; } + } + char* esc_prefix = engram_json_escape(prefix ? prefix : ""); + char* esc = engram_json_escape(text); + free(trunc); + if (!esc || !esc_prefix) { free(esc); free(esc_prefix); return NULL; } + size_t blen = strlen(esc) + strlen(esc_prefix) + strlen(model) + 64; + char* body = malloc(blen); + if (!body) { free(esc); free(esc_prefix); return NULL; } + snprintf(body, blen, "{\"model\":\"%s\",\"prompt\":\"%s%s\"}", model, esc_prefix, esc); + free(esc); free(esc_prefix); + + CURL* c = curl_easy_init(); + if (!c) { free(body); return NULL; } + HttpBuf rb; httpbuf_init(&rb); + struct curl_slist* h = curl_slist_append(NULL, "Content-Type: application/json"); + char errbuf[CURL_ERROR_SIZE]; errbuf[0] = '\0'; + curl_easy_setopt(c, CURLOPT_URL, url); + curl_easy_setopt(c, CURLOPT_WRITEFUNCTION, http_write_cb); + curl_easy_setopt(c, CURLOPT_WRITEDATA, &rb); + curl_easy_setopt(c, CURLOPT_POST, 1L); + curl_easy_setopt(c, CURLOPT_POSTFIELDS, body); + curl_easy_setopt(c, CURLOPT_POSTFIELDSIZE, (long)strlen(body)); + curl_easy_setopt(c, CURLOPT_HTTPHEADER, h); + curl_easy_setopt(c, CURLOPT_TIMEOUT_MS, el_http_timeout_ms()); + curl_easy_setopt(c, CURLOPT_NOSIGNAL, 1L); + curl_easy_setopt(c, CURLOPT_ERRORBUFFER, errbuf); + CURLcode rc = curl_easy_perform(c); + curl_slist_free_all(h); + curl_easy_cleanup(c); + free(body); + if (rc != CURLE_OK) { free(rb.data); return NULL; } + float* v = engram_parse_embedding(rb.data, out_dim); + free(rb.data); + return v; +} + +/* One-shot probe: is semantic search available? Caches the verdict. */ +static int engram_semantic_enabled(void) { + if (g_emb_state != 0) return g_emb_state == 1; + const char* s = getenv("ENGRAM_SEMANTIC"); + if (s && strcmp(s, "0") == 0) { g_emb_state = -1; return 0; } + int dim = 0; + float* v = engram_embed_raw("search_query: ", "probe", &dim); + if (v && dim > 0) { free(v); g_emb_state = 1; return 1; } + free(v); + g_emb_state = -1; return 0; +} + +/* Embed the query. Returns malloc'd vec (caller frees), or NULL if semantic off. */ +static float* engram_embed_query(const char* q, int* dim) { + if (!engram_semantic_enabled()) return NULL; + if (!q || !*q) return NULL; + return engram_embed_raw("search_query: ", q, dim); +} + +/* Cached node embedding. Returns a pointer OWNED BY THE CACHE — do not free. */ +static const float* engram_node_vec(EngramNode* n, int* out_dim) { + if (!n || !n->id) return NULL; + uint64_t h = engram_fnv1a(n->content); + for (int64_t i = 0; i < g_emb_count; i++) { + if (g_emb_items[i].id && strcmp(g_emb_items[i].id, n->id) == 0) { + if (g_emb_items[i].hash == h && g_emb_items[i].vec) { + *out_dim = g_emb_items[i].dim; return g_emb_items[i].vec; + } + /* content changed → re-embed in place */ + int dim = 0; + float* v = engram_embed_raw("search_document: ", n->content ? n->content : "", &dim); + if (!v) return NULL; + free(g_emb_items[i].vec); + g_emb_items[i].vec = v; g_emb_items[i].dim = dim; g_emb_items[i].hash = h; + *out_dim = dim; return v; + } + } + int dim = 0; + float* v = engram_embed_raw("search_document: ", n->content ? n->content : "", &dim); + if (!v) return NULL; + if (g_emb_count >= g_emb_cap) { + int64_t nc = g_emb_cap ? g_emb_cap * 2 : 256; + EngramEmbEntry* ni = realloc(g_emb_items, (size_t)nc * sizeof(EngramEmbEntry)); + if (!ni) { free(v); return NULL; } + g_emb_items = ni; g_emb_cap = nc; + } + g_emb_items[g_emb_count].id = strdup(n->id); + g_emb_items[g_emb_count].hash = h; + g_emb_items[g_emb_count].vec = v; + g_emb_items[g_emb_count].dim = dim; + g_emb_count++; + *out_dim = dim; return v; +} + +static double engram_cosine(const float* a, const float* b, int dim) { + double dot = 0, na = 0, nb = 0; + for (int i = 0; i < dim; i++) { dot += (double)a[i] * b[i]; + na += (double)a[i] * a[i]; + nb += (double)b[i] * b[i]; } + if (na <= 0 || nb <= 0) return 0.0; + return dot / (sqrt(na) * sqrt(nb)); +} + +/* Cosine of node n against the query vector; 0 if unavailable / dim mismatch. */ +static double engram_node_cosine(EngramNode* n, const float* qvec, int qdim) { + if (!qvec || qdim <= 0) return 0.0; + int ndim = 0; + const float* nv = engram_node_vec(n, &ndim); + if (!nv || ndim != qdim) return 0.0; + return engram_cosine(qvec, nv, qdim); +} + +#else /* !HAVE_CURL — semantic layer compiled out; callers stay pure-lexical. + * Only the two boundary functions the always-compiled search/activate + * code calls are stubbed; the query embed always yields NULL so every + * cosine is 0 and every caller collapses to lexical-only. */ +static float* engram_embed_query(const char* q, int* dim) { (void)q; (void)dim; return NULL; } +static double engram_node_cosine(EngramNode* n, const float* qvec, int qdim) { + (void)n; (void)qvec; (void)qdim; return 0.0; +} +#endif /* HAVE_CURL */ + el_val_t engram_search(el_val_t query, el_val_t limit) { EngramStore* g = engram_get(); const char* q = EL_CSTR(query); @@ -6833,6 +7070,12 @@ el_val_t engram_search(el_val_t query, el_val_t limit) { if (lim <= 0) lim = 100; el_val_t lst = el_list_empty(); if (!q || !*q) return lst; + /* Semantic augmentation: embed the query once; a node matches if it is a + * lexical hit OR its cosine similarity clears the threshold. qvec is NULL + * (and cosine 0) whenever semantic search is unavailable → pure lexical. */ + int qdim = 0; + float* qvec = engram_embed_query(q, &qdim); + double sem_min = engram_semantic_min(); int64_t found = 0; for (int64_t i = 0; i < g->node_count && found < lim; i++) { EngramNode* n = &g->nodes[i]; @@ -6841,13 +7084,16 @@ el_val_t engram_search(el_val_t query, el_val_t limit) { * know about yourself"). They still surface via engram_activate * + engram_compile_layered_json — that's the legitimate path. */ if (engram_layer_is_transparent(n->layer_id)) continue; - if (istr_contains(n->content, q) || - istr_contains(n->label, q) || - istr_contains(n->tags, q)) { + int lex = istr_contains(n->content, q) || + istr_contains(n->label, q) || + istr_contains(n->tags, q); + double sem = qvec ? engram_node_cosine(n, qvec, qdim) : 0.0; + if (lex || sem >= sem_min) { lst = el_list_append(lst, engram_node_to_map(n)); found++; } } + free(qvec); return lst; } @@ -7193,14 +7439,28 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) { if (!seeds) { free(best_bg); free(best_hops); free(reached); return out; } + /* Semantic seed augmentation: embed the query once; a node becomes a seed + * if it lexically matches OR its cosine clears the threshold. Semantic-only + * seeds enter at reduced strength (scaled by cosine) so pure paraphrase + * matches spread activation without overpowering exact lexical seeds. + * qvec is NULL (cosine 0) when semantic search is unavailable → the seed + * set is exactly the pre-existing lexical one. qvec is freed right after + * this loop so the many downstream early-returns need no cleanup change. */ + int q_dim = 0; + float* q_vec = engram_embed_query(q, &q_dim); + double q_sem_min = engram_semantic_min(); for (int64_t i = 0; i < g->node_count; i++) { EngramNode* n = &g->nodes[i]; - if (istr_contains(n->content, q) || - istr_contains(n->label, q) || - istr_contains(n->tags, q)) { + int lex = istr_contains(n->content, q) || + istr_contains(n->label, q) || + istr_contains(n->tags, q); + double sem = q_vec ? engram_node_cosine(n, q_vec, q_dim) : 0.0; + if (lex || sem >= q_sem_min) { double tdecay = engram_temporal_decay(n, now_ms); double dampen = engram_activation_dampen(n); double act = n->salience * tdecay * dampen; + /* Down-weight pure-semantic seeds by their cosine strength. */ + if (!lex) act *= sem; seeds[seed_count].idx = i; seeds[seed_count].act = act; seeds[seed_count].created_at = n->created_at; @@ -7210,6 +7470,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) { reached[i] = 1; } } + free(q_vec); /* Compute mean seed created_at for temporal proximity bonus. */ int64_t seed_epoch = 0; if (seed_count > 0) { @@ -7768,22 +8029,52 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { if (lim <= 0) lim = 100; JsonBuf b; jb_init(&b); jb_putc(&b, '['); - int first = 1; - int64_t found = 0; - if (q && *q) { - for (int64_t i = 0; i < g->node_count && found < lim; i++) { - EngramNode* n = &g->nodes[i]; - /* Filter transparent layers — same as engram_search. */ - if (engram_layer_is_transparent(n->layer_id)) continue; - if (istr_contains(n->content, q) || - istr_contains(n->label, q) || - istr_contains(n->tags, q)) { - if (!first) jb_putc(&b, ','); - engram_emit_node_json(&b, n); - first = 0; - found++; + if (q && *q && g->node_count > 0) { + /* Collect candidates from the UNION of lexical and semantic matches, + * score each, rank by score, then emit the top `lim`. A node is a + * candidate if it lexically matches OR its query cosine clears the + * threshold. Lexical hits get a base of 1.0 (so they always outrank a + * pure-semantic hit, whose cosine is in [0,1)), refined by cosine; + * pure-semantic hits are scored by cosine alone. + * + * When semantic search is unavailable, qvec is NULL, sem is 0, only + * lexical hits are collected (score 1.0), and the stable insertion + * sort preserves node order — identical to the pre-existing search. */ + int qdim = 0; + float* qvec = engram_embed_query(q, &qdim); + double sem_min = engram_semantic_min(); + typedef struct { int64_t idx; double score; } Cand; + Cand* cand = malloc((size_t)g->node_count * sizeof(Cand)); + if (cand) { + int64_t nc = 0; + for (int64_t i = 0; i < g->node_count; i++) { + EngramNode* n = &g->nodes[i]; + if (engram_layer_is_transparent(n->layer_id)) continue; + int lex = istr_contains(n->content, q) || + istr_contains(n->label, q) || + istr_contains(n->tags, q); + double sem = qvec ? engram_node_cosine(n, qvec, qdim) : 0.0; + if (lex || sem >= sem_min) { + cand[nc].idx = i; + cand[nc].score = (lex ? 1.0 : 0.0) + sem; + nc++; + } } + /* Insertion sort by score desc; stable for equal scores. */ + for (int64_t i = 1; i < nc; i++) { + Cand k = cand[i]; int64_t j = i - 1; + while (j >= 0 && cand[j].score < k.score) { cand[j + 1] = cand[j]; j--; } + cand[j + 1] = k; + } + int first = 1; + for (int64_t i = 0; i < nc && i < lim; i++) { + if (!first) jb_putc(&b, ','); + engram_emit_node_json(&b, &g->nodes[cand[i].idx]); + first = 0; + } + free(cand); } + free(qvec); } jb_putc(&b, ']'); return el_wrap_str(jb_finish(&b)); From 6b9d9e6c4af4fa902deebbd5b49ec46fa5d97d55 Mon Sep 17 00:00:00 2001 From: Will Anderson Date: Wed, 15 Jul 2026 04:07:33 -0500 Subject: [PATCH 2/9] Add engram_get_node_by_label runtime native to unblock soul link MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit chat.el calls the runtime native engram_get_node_by_label to fetch well-known nodes (conv:history, session:summary) by stable label rather than by ID — immune to vector-index drift across restarts. The current runtime never defined it, so the regenerated dist/soul.c fails to link. Backport the function verbatim (idiom-adapted to jb_finish) from release runtime v1.0.0-20260501 and register it as an EL builtin exactly like its siblings: runtime definition + prototype, __-prefixed seed wrapper + prototype, and codegen arity entry. No search-site code is touched. --- lang/el-compiler/runtime/el_runtime.c | 29 +++++++++++++++++++++++++++ lang/el-compiler/runtime/el_runtime.h | 1 + lang/el-compiler/runtime/el_seed.c | 1 + lang/el-compiler/runtime/el_seed.h | 1 + lang/el-compiler/src/codegen.el | 1 + 5 files changed, 33 insertions(+) diff --git a/lang/el-compiler/runtime/el_runtime.c b/lang/el-compiler/runtime/el_runtime.c index 68e6dca..57a9042 100644 --- a/lang/el-compiler/runtime/el_runtime.c +++ b/lang/el-compiler/runtime/el_runtime.c @@ -7761,6 +7761,35 @@ el_val_t engram_get_node_json(el_val_t id) { return el_wrap_str(jb_finish(&b)); } +/* engram_get_node_by_label — find the first node whose label field exactly + * matches the given string. Returns the node as a JSON object string, or "{}" + * if no match is found. + * + * Used by chat.el to retrieve well-known nodes (e.g. "conv:history", + * "session:summary") by their stable label rather than by ID, which is immune + * to vector index drift across restarts. + * + * Exact match (strcmp, not istr_contains) because labels like "conv:history" + * must not collide with nodes whose content happens to contain that substring. + * + * Backported verbatim (idiom-adapted to jb_finish) from release runtime + * v1.0.0-20260501 to unblock the soul regen link: chat.el references this + * native but the current runtime lacked its definition. */ +el_val_t engram_get_node_by_label(el_val_t label) { + const char* lbl = EL_CSTR(label); + if (!lbl || !*lbl) return el_wrap_str(el_strdup("{}")); + EngramStore* g = engram_get(); + for (int64_t i = 0; i < g->node_count; i++) { + EngramNode* n = &g->nodes[i]; + if (n->label && strcmp(n->label, lbl) == 0) { + JsonBuf b; jb_init(&b); + engram_emit_node_json(&b, n); + return el_wrap_str(jb_finish(&b)); + } + } + return el_wrap_str(el_strdup("{}")); +} + el_val_t engram_search_json(el_val_t query, el_val_t limit) { EngramStore* g = engram_get(); const char* q = EL_CSTR(query); diff --git a/lang/el-compiler/runtime/el_runtime.h b/lang/el-compiler/runtime/el_runtime.h index f64bf63..87348f5 100644 --- a/lang/el-compiler/runtime/el_runtime.h +++ b/lang/el-compiler/runtime/el_runtime.h @@ -632,6 +632,7 @@ el_val_t engram_load(el_val_t path); * can pass results straight through without round-tripping ElList/ElMap * through json_stringify. */ el_val_t engram_get_node_json(el_val_t id); +el_val_t engram_get_node_by_label(el_val_t label); el_val_t engram_search_json(el_val_t query, el_val_t limit); el_val_t engram_scan_nodes_json(el_val_t limit, el_val_t offset); el_val_t engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset); diff --git a/lang/el-compiler/runtime/el_seed.c b/lang/el-compiler/runtime/el_seed.c index b509e48..cda893c 100644 --- a/lang/el-compiler/runtime/el_seed.c +++ b/lang/el-compiler/runtime/el_seed.c @@ -1072,6 +1072,7 @@ el_val_t __engram_save(el_val_t path) { return engram_save el_val_t __engram_load(el_val_t path) { return engram_load(path); } el_val_t __engram_get_node_json(el_val_t id) { return engram_get_node_json(id); } +el_val_t __engram_get_node_by_label(el_val_t label) { return engram_get_node_by_label(label); } el_val_t __engram_search_json(el_val_t query, el_val_t limit) { return engram_search_json(query, limit); diff --git a/lang/el-compiler/runtime/el_seed.h b/lang/el-compiler/runtime/el_seed.h index c502f10..7597511 100644 --- a/lang/el-compiler/runtime/el_seed.h +++ b/lang/el-compiler/runtime/el_seed.h @@ -226,6 +226,7 @@ el_val_t __engram_activate(el_val_t query, el_val_t depth); el_val_t __engram_save(el_val_t path); el_val_t __engram_load(el_val_t path); el_val_t __engram_get_node_json(el_val_t id); +el_val_t __engram_get_node_by_label(el_val_t label); el_val_t __engram_search_json(el_val_t query, el_val_t limit); el_val_t __engram_scan_nodes_json(el_val_t limit, el_val_t offset); el_val_t __engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset); diff --git a/lang/el-compiler/src/codegen.el b/lang/el-compiler/src/codegen.el index 917fb31..a20569a 100644 --- a/lang/el-compiler/src/codegen.el +++ b/lang/el-compiler/src/codegen.el @@ -2670,6 +2670,7 @@ fn builtin_arity(name: String) -> Int { if str_eq(name, "engram_save") { return 1 } if str_eq(name, "engram_load") { return 1 } if str_eq(name, "engram_get_node_json") { return 1 } + if str_eq(name, "engram_get_node_by_label") { return 1 } if str_eq(name, "engram_search_json") { return 2 } if str_eq(name, "engram_scan_nodes_json") { return 2 } if str_eq(name, "engram_neighbors_json") { return 3 } From 4773dd0aa2ed868330349f0021c812948f79e9e8 Mon Sep 17 00:00:00 2001 From: Will Anderson Date: Wed, 15 Jul 2026 04:24:08 -0500 Subject: [PATCH 3/9] runtime: make Windows soul reproducible from a clean el checkout Two el_runtime portability defects only ever lived in staged local copies used to hand-build neuron-ui PR #136's curl-enabled Windows neuron.exe. gcc 15 promotes both to hard errors, so a clean el checkout cannot rebuild that soul. Upstream the minimal fixes so the build is reproducible: - http_serve_async: cast setsockopt optval to (const char*). Win32/mingw setsockopt wants const char*, not int*; the cast is a no-op on POSIX and matches the four already-cast sites elsewhere in this file. - engram_save persist path: map fsync -> _commit in the _WIN32-only el_platform_win.h (io.h already included). Windows has no fsync(); the POSIX path is untouched. --- lang/el-compiler/runtime/el_platform_win.h | 1 + lang/el-compiler/runtime/el_runtime.c | 5 +++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/lang/el-compiler/runtime/el_platform_win.h b/lang/el-compiler/runtime/el_platform_win.h index 88204a0..df42c2b 100644 --- a/lang/el-compiler/runtime/el_platform_win.h +++ b/lang/el-compiler/runtime/el_platform_win.h @@ -75,6 +75,7 @@ static inline void* el_win_dlsym(void* handle, const char* name) { #include /* _mkdir */ #define mkdir(path, mode) _mkdir(path) /* POSIX mkdir(path,mode) → _mkdir(path) */ #define timegm _mkgmtime /* UTC tm → time_t */ +#define fsync(fd) _commit(fd) /* no fsync() on Windows; _commit() () is the equiv */ /* setenv/unsetenv: not in the Windows CRT; map to _putenv_s / SetEnvironmentVariable. */ static inline int setenv(const char* name, const char* value, int overwrite) { diff --git a/lang/el-compiler/runtime/el_runtime.c b/lang/el-compiler/runtime/el_runtime.c index 68e6dca..5c9a066 100644 --- a/lang/el-compiler/runtime/el_runtime.c +++ b/lang/el-compiler/runtime/el_runtime.c @@ -1963,8 +1963,9 @@ void http_serve_async(el_val_t port, el_val_t handler) { int sock = socket(AF_INET6, SOCK_STREAM, 0); if (sock < 0) { perror("socket"); return; } int yes = 1; int no = 0; - setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes)); - setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, &no, sizeof(no)); + /* Win32/mingw setsockopt takes optval as (const char*); the cast is portable on POSIX too. */ + setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (const char*)&yes, sizeof(yes)); + setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, (const char*)&no, sizeof(no)); struct sockaddr_in6 addr; memset(&addr, 0, sizeof(addr)); addr.sin6_family = AF_INET6; From 8ce8656de2ede82eacd91351b71f98e7fd7dd339 Mon Sep 17 00:00:00 2001 From: Will Anderson Date: Wed, 15 Jul 2026 10:14:43 -0500 Subject: [PATCH 4/9] epm: declare cross-module callees as extern fn so strict compilers accept generated C epm's sibling modules (registry/install/update) call functions defined in other modules and in the El runtime (config, read_installed, registry_find, manifest_deps, manifest_name, registry_latest_version, registry_token, install_vessel, installed_version) without importing them, so elc emits no C prototype for those calls. gcc<=13 treated the resulting implicit declarations as warnings; gcc>=14 and clang reject them as hard errors, which is why the "Build epm" CI step fails and blocks the whole dev/stage pipeline. Add `extern fn` forward declarations -- El's own separate-compilation mechanism -- for each cross-module callee at the top of registry/install/update. This gives elc the correct C prototype in every generated translation unit, so the calls compile cleanly and still resolve at link time. Simply suppressing -Wimplicit-function-declaration would be unsafe: an implicit int return truncates the 64-bit pointer returns of config/registry_find into a latent crash, so declaring the true signatures is the correct fix. Localized to epm; touches neither elc nor the runtime. --- epm/src/install.el | 10 ++++++++++ epm/src/registry.el | 9 +++++++++ epm/src/update.el | 9 +++++++++ 3 files changed, 28 insertions(+) diff --git a/epm/src/install.el b/epm/src/install.el index 882fc90..1561cf8 100644 --- a/epm/src/install.el +++ b/epm/src/install.el @@ -17,6 +17,16 @@ // 4. Append dep to order after all its transitive deps // 5. Deduplicate: skip already-ordered vessels +// ── Cross-module forward declarations ───────────────────────────────────────── +// Defined in sibling epm modules; resolved at link time. The `extern fn` decls +// give elc the C prototypes so generated install.c compiles cleanly under strict +// compilers (gcc>=14 / clang) that reject implicit function declarations. +extern fn manifest_name(src: String) -> String // manifest.el +extern fn manifest_deps(src: String) -> String // manifest.el +extern fn registry_token() -> String // registry.el +extern fn registry_find(name: String, version: String) -> String // registry.el +extern fn registry_latest_version(name: String) -> String // registry.el + // ── Install paths ───────────────────────────────────────────────────────────── // packages_dir returns the root directory for installed vessels. diff --git a/epm/src/registry.el b/epm/src/registry.el index 8e1b2b5..d6f9730 100644 --- a/epm/src/registry.el +++ b/epm/src/registry.el @@ -14,6 +14,15 @@ // EPM_REGISTRY_ORG — org name that hosts vessel repos (default: neuron-technologies) // EPM_TOKEN — Gitea personal access token (required for publish) +// ── Cross-module forward declarations ───────────────────────────────────────── +// These symbols are defined in sibling epm modules or the El runtime and are +// resolved at link time. The `extern fn` decls give elc the C prototype so the +// generated registry.c compiles cleanly under strict compilers (gcc>=14 / clang) +// that reject implicit function declarations. Signature arity must match the +// definition; return/param types are informational (all lower to el_val_t). +extern fn config(key: String) -> String // El runtime builtin +extern fn read_installed() -> String // install.el + // ── Config helpers ──────────────────────────────────────────────────────────── // registry_api_url returns the Gitea API base URL with no trailing slash. diff --git a/epm/src/update.el b/epm/src/update.el index 42f60a5..3804ab2 100644 --- a/epm/src/update.el +++ b/epm/src/update.el @@ -6,6 +6,15 @@ // Depends on: registry.el (registry_latest_version, registry_find), // install.el (read_installed, install_vessel, installed_version) +// ── Cross-module forward declarations ───────────────────────────────────────── +// Defined in sibling epm modules; resolved at link time. The `extern fn` decls +// give elc the C prototypes so generated update.c compiles cleanly under strict +// compilers (gcc>=14 / clang) that reject implicit function declarations. +extern fn read_installed() -> String // install.el +extern fn installed_version(name: String) -> String // install.el +extern fn install_vessel(name: String, version: String) -> Bool // install.el +extern fn registry_latest_version(name: String) -> String // registry.el + // ── Semver helpers ──────────────────────────────────────────────────────────── // semver_part extracts the Nth dot-separated component from a semver string. From 4696fd68337f20b5b2d3ef38bd68a559820eea70 Mon Sep 17 00:00:00 2001 From: Will Anderson Date: Wed, 15 Jul 2026 11:33:37 -0500 Subject: [PATCH 5/9] ci(dev): make SDK publish fail loudly, decouple ci-base rebuild The dev push build went green-then-red while nothing published: the Publish step had no set -e, so an auth/upload failure exited 0 (silent no-publish), while the ci-base rebuild (set -euo pipefail + Docker on the host-mode runner) hard-failed the job. Add set -euo pipefail + an empty-key guard + active-account echo to the Publish step so failures surface with a retrievable log, and mark the ci-base cache rebuild continue-on-error so the fragile Docker step can never block the actual SDK artifact publish. --- .gitea/workflows/ci-dev.yaml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/.gitea/workflows/ci-dev.yaml b/.gitea/workflows/ci-dev.yaml index f788d8b..092a80e 100644 --- a/.gitea/workflows/ci-dev.yaml +++ b/.gitea/workflows/ci-dev.yaml @@ -214,9 +214,18 @@ jobs: env: GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} run: | + # Fail loudly: previously this step had no `set -e`, so an auth or + # upload failure was swallowed (step exited 0 on the trailing echo) + # and the SDK silently never published. Surface failures now. + set -euo pipefail + if [ -z "${GCP_SA_KEY:-}" ]; then + echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2 + exit 1 + fi echo "${GCP_SA_KEY}" > /tmp/gcp-key.json gcloud auth activate-service-account --key-file=/tmp/gcp-key.json gcloud config set project neuron-785695 + echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)" VERSION="${GITHUB_SHA:0:8}" @@ -268,6 +277,12 @@ jobs: # Patches ci-base:dev in-place: pulls the existing image (which has all # system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly # built El SDK on top. Keeps the full ci-base rebuild fast and incremental. + # + # continue-on-error: this is a CI-cache optimization, NOT the release + # artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE + # runner where DinD/Docker availability is fragile. A failure here must + # never block or redden the job — the SDK publish above is the deliverable. + continue-on-error: true if: github.event_name == 'push' env: GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} From dd7827059aa8707098f4c8026ae3cba196a2b99d Mon Sep 17 00:00:00 2001 From: Will Anderson Date: Wed, 15 Jul 2026 16:14:50 -0500 Subject: [PATCH 6/9] ci(stage,main): decouple ci-base rebuild, make SDK publish fail loudly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mirror the PR #72 fix (applied to ci-dev.yaml) onto ci-stage.yaml and sdk-release.yaml. The stage and prod release jobs reported FAILURE even when the el-runtime-c/-h publish SUCCEEDED, because the ancillary ci-base Docker rebuild (a CI-cache optimization on the fragile host-mode GCE runner) reddened the whole job. - Rebuild ci-base step: continue-on-error: true — never blocks/reddens the job; the SDK publish is the deliverable. - Publish step: set -euo pipefail + empty-key guard + active-account echo so a real publish failure still fails loud and is diagnosable. --- .gitea/workflows/ci-stage.yaml | 15 +++++++++++++++ .gitea/workflows/sdk-release.yaml | 15 +++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/.gitea/workflows/ci-stage.yaml b/.gitea/workflows/ci-stage.yaml index d8b4ee1..281e9bb 100644 --- a/.gitea/workflows/ci-stage.yaml +++ b/.gitea/workflows/ci-stage.yaml @@ -212,12 +212,21 @@ jobs: env: GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} run: | + # Fail loudly: previously this step had no `set -e`, so an auth or + # upload failure was swallowed (step exited 0 on the trailing echo) + # and the SDK silently never published. Surface failures now. + set -euo pipefail + if [ -z "${GCP_SA_KEY:-}" ]; then + echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2 + exit 1 + fi echo "${GCP_SA_KEY}" > /tmp/gcp-key.json apt-get install -y -qq apt-transport-https ca-certificates curl echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list apt-get update -qq && apt-get install -y google-cloud-cli gcloud auth activate-service-account --key-file=/tmp/gcp-key.json gcloud config set project neuron-785695 + echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)" VERSION="${GITHUB_SHA:0:8}" @@ -253,6 +262,12 @@ jobs: # Patches ci-base:stage in-place: pulls the existing image (which has all # system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly # built El SDK on top. Keeps the full ci-base rebuild fast and incremental. + # + # continue-on-error: this is a CI-cache optimization, NOT the release + # artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE + # runner where DinD/Docker availability is fragile. A failure here must + # never block or redden the job — the SDK publish above is the deliverable. + continue-on-error: true if: github.event_name == 'push' env: GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} diff --git a/.gitea/workflows/sdk-release.yaml b/.gitea/workflows/sdk-release.yaml index 35f2fc6..66d9c3a 100644 --- a/.gitea/workflows/sdk-release.yaml +++ b/.gitea/workflows/sdk-release.yaml @@ -288,12 +288,21 @@ jobs: env: GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} run: | + # Fail loudly: previously this step had no `set -e`, so an auth or + # upload failure was swallowed (step exited 0 on the trailing echo) + # and the SDK silently never published. Surface failures now. + set -euo pipefail + if [ -z "${GCP_SA_KEY:-}" ]; then + echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2 + exit 1 + fi echo "${GCP_SA_KEY}" > /tmp/gcp-key.json apt-get install -y -qq apt-transport-https ca-certificates curl echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list apt-get update -qq && apt-get install -y google-cloud-cli gcloud auth activate-service-account --key-file=/tmp/gcp-key.json gcloud config set project neuron-785695 + echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)" VERSION="${GITHUB_SHA:0:8}" @@ -345,6 +354,12 @@ jobs: # Patches ci-base:latest in-place: pulls the existing image (which has all # system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly # built El SDK on top. Keeps the full ci-base rebuild fast and incremental. + # + # continue-on-error: this is a CI-cache optimization, NOT the release + # artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE + # runner where DinD/Docker availability is fragile. A failure here must + # never block or redden the job — the SDK publish above is the deliverable. + continue-on-error: true if: github.event_name == 'push' env: GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} From 43636aed99ae129aa93a8a9cd8505061e8cc4694 Mon Sep 17 00:00:00 2001 From: Will Anderson Date: Wed, 22 Jul 2026 15:04:25 -0500 Subject: [PATCH 7/9] =?UTF-8?q?runtime:=20pair=20fs=5Fread=20length=20hint?= =?UTF-8?q?=20with=20its=20buffer=20in=20BOTH=20runtimes=20=E2=80=94=20kil?= =?UTF-8?q?l=20response=20truncation=20for=20good?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The binary-safe fs_read length (_tl_fs_read_len) was consumed by the HTTP response path for ANY body, even when a handler wrapped a smaller file into a larger reply. Content-Length then lied AND the send stopped short: the safety-contact (988) routes returned 178 of 208/218 bytes, cut mid-'set_at' — unparseable JSON. The desktop app read that as failure. On Windows the shipped brain is an OLD build without even the per-handler workaround, so EVERY reply truncated: the app can't read confirmations and refuses the new user. Durable fix: pair the length hint with the exact buffer pointer it describes (_tl_fs_read_buf). Apply the raw byte count ONLY when the response IS that buffer (binary file serving stays correct); every wrapped/enveloped/derived body is measured with strlen. Reset both at request start and in fs_read / json_get_raw. This also closes the stale-hint heap over-read (a length larger than a later body would read past it out the socket) that a plain max() leaves open — so this class of bug dies on every platform, not just where a handler happened to be patched. Applied identically to the mainline runtime (lang/el-compiler/runtime) AND the frozen release runtime (lang/releases/v1.0.0-20260501) the desktop souls compile against — the release copy still carried the raw leak, which is why the Windows brain kept truncating. Same proven approach as PR #78 (Tim Lingo), extended to cover the release runtime and rebased onto current main. Both runtimes: gcc -fsyntax-only clean. --- lang/el-compiler/runtime/el_runtime.c | 56 ++++++++++++++++++---- lang/releases/v1.0.0-20260501/el_runtime.c | 56 ++++++++++++++++++---- 2 files changed, 92 insertions(+), 20 deletions(-) diff --git a/lang/el-compiler/runtime/el_runtime.c b/lang/el-compiler/runtime/el_runtime.c index 2b75c6f..af0d945 100644 --- a/lang/el-compiler/runtime/el_runtime.c +++ b/lang/el-compiler/runtime/el_runtime.c @@ -82,8 +82,14 @@ static _Thread_local ElArena _tl_arena = {NULL, 0, 0}; static _Thread_local int _tl_arena_active = 0; /* Binary-safe fs_read length — set by fs_read, consumed by http_send_response. - * Allows serving PNGs and other binary files without strlen truncation. */ -static _Thread_local size_t _tl_fs_read_len = 0; + * Allows serving PNGs and other binary files without strlen truncation. + * PAIRED with the buffer pointer it describes: the length may only be applied + * to the exact buffer fs_read returned. Without the pairing, any handler that + * fs_read a file and then WRAPPED it into a larger response had that response + * truncated to the file's length (Content-Length lied AND the send stopped + * short) — the safety-contact onboarding trap, 2026-07-17. */ +static _Thread_local size_t _tl_fs_read_len = 0; +static _Thread_local const char* _tl_fs_read_buf = NULL; static void el_arena_track(char* p) { if (!_tl_arena_active || !p) return; @@ -101,6 +107,8 @@ static void el_arena_track(char* p) { void el_request_start(void) { _tl_arena.count = 0; _tl_arena_active = 1; + _tl_fs_read_len = 0; /* never let a previous request's file length */ + _tl_fs_read_buf = NULL; /* leak into this response's byte accounting */ } /* Called by http_worker after the El handler returns and the response is sent. @@ -1484,11 +1492,14 @@ static void http_send_response(int fd, const char* body) { } const char* eff_body = is_envelope ? env_body : body; - /* Use the real byte count from fs_read if available (handles binary files - * with embedded null bytes — PNG, WOFF2, etc.). Fall back to strlen for - * normal text/JSON responses where _tl_fs_read_len is 0. */ - size_t blen = (_tl_fs_read_len > 0) ? _tl_fs_read_len : strlen(eff_body); + /* Use the real byte count from fs_read ONLY when this body IS the exact + * buffer fs_read returned (binary files with embedded null bytes — PNG, + * WOFF2, etc.). Any other body — wrapped, enveloped, or derived — must be + * measured with strlen, or it is truncated/over-read to the file's size. */ + size_t blen = (_tl_fs_read_len > 0 && eff_body == _tl_fs_read_buf) + ? _tl_fs_read_len : strlen(eff_body); _tl_fs_read_len = 0; /* consume — one-shot per response */ + _tl_fs_read_buf = NULL; int head_only = _tl_http_head_only; JsonBuf hdrs; jb_init(&hdrs); @@ -1568,11 +1579,22 @@ static void* http_worker(void* arg) { const char* rs = EL_CSTR(r); /* Copy response out BEFORE arena teardown. * For binary files, _tl_fs_read_len holds the real byte count — - * use memcpy instead of strdup so null bytes are preserved. */ - size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0); + * use memcpy instead of strdup so null bytes are preserved. + * The stored length applies ONLY when the response IS the exact + * fs_read buffer; a wrapped/derived response must use strlen or + * it gets truncated (or over-read) to the file's length. */ + size_t rlen; + if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) { + rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */ + } else { + rlen = rs ? strlen(rs) : 0; + _tl_fs_read_len = 0; /* hint doesn't describe this body */ + _tl_fs_read_buf = NULL; + } response = malloc(rlen + 1); if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; } else if (response) { response[0] = '\0'; } + if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */ } else { response = el_strdup_persist("el-runtime: no http handler registered"); } @@ -1822,10 +1844,20 @@ static void* http_worker_v2(void* arg) { el_val_t hmap = http_build_headers_map(hdr_block ? hdr_block : ""); el_val_t r = h(EL_STR(dispatch_method), EL_STR(path), hmap, EL_STR(body)); const char* rs = EL_CSTR(r); - size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0); + /* Same pairing rule as the v1 worker: the fs_read length is only + * trustworthy for the exact buffer fs_read returned. */ + size_t rlen; + if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) { + rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */ + } else { + rlen = rs ? strlen(rs) : 0; + _tl_fs_read_len = 0; /* hint doesn't describe this body */ + _tl_fs_read_buf = NULL; + } response = malloc(rlen + 1); if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; } else if (response) { response[0] = '\0'; } + if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */ el_release(hmap); } else { response = el_strdup_persist( @@ -2024,6 +2056,7 @@ el_val_t http_response(el_val_t status, el_val_t headers_json, el_val_t body) { el_val_t fs_read(el_val_t pathv) { const char* path = EL_CSTR(pathv); _tl_fs_read_len = 0; + _tl_fs_read_buf = NULL; if (!path) return el_wrap_str(el_strdup("")); FILE* f = fopen(path, "rb"); if (!f) return el_wrap_str(el_strdup("")); @@ -2035,6 +2068,7 @@ el_val_t fs_read(el_val_t pathv) { size_t got = fread(buf, 1, (size_t)sz, f); buf[got] = '\0'; _tl_fs_read_len = got; /* store real byte count for binary-safe send */ + _tl_fs_read_buf = buf; /* ...valid ONLY for this exact buffer */ fclose(f); return el_wrap_str(buf); } @@ -3577,8 +3611,10 @@ el_val_t json_get_raw(el_val_t json_str, el_val_t key) { const char* k = EL_CSTR(key); const char* p = json_find_key(json, k); /* Clear fs_read binary-length hint — result is a fresh null-terminated - * string, not the raw file bytes, so Content-Length must use strlen. */ + * string, not the raw file bytes, so Content-Length must use strlen. + * (Kept although the pointer pairing now makes this redundant.) */ _tl_fs_read_len = 0; + _tl_fs_read_buf = NULL; if (!p) return el_wrap_str(el_strdup("")); const char* end = json_skip_value(p); size_t n = (size_t)(end - p); diff --git a/lang/releases/v1.0.0-20260501/el_runtime.c b/lang/releases/v1.0.0-20260501/el_runtime.c index f4f605f..971cf56 100644 --- a/lang/releases/v1.0.0-20260501/el_runtime.c +++ b/lang/releases/v1.0.0-20260501/el_runtime.c @@ -71,8 +71,14 @@ static _Thread_local ElArena _tl_arena = {NULL, 0, 0}; static _Thread_local int _tl_arena_active = 0; /* Binary-safe fs_read length — set by fs_read, consumed by http_send_response. - * Allows serving PNGs and other binary files without strlen truncation. */ -static _Thread_local size_t _tl_fs_read_len = 0; + * Allows serving PNGs and other binary files without strlen truncation. + * PAIRED with the buffer pointer it describes: the length may only be applied + * to the exact buffer fs_read returned. Without the pairing, any handler that + * fs_read a file and then WRAPPED it into a larger response had that response + * truncated to the file's length (Content-Length lied AND the send stopped + * short) — the safety-contact onboarding trap, 2026-07-17. */ +static _Thread_local size_t _tl_fs_read_len = 0; +static _Thread_local const char* _tl_fs_read_buf = NULL; static void el_arena_track(char* p) { if (!_tl_arena_active || !p) return; @@ -90,6 +96,8 @@ static void el_arena_track(char* p) { void el_request_start(void) { _tl_arena.count = 0; _tl_arena_active = 1; + _tl_fs_read_len = 0; /* never let a previous request's file length */ + _tl_fs_read_buf = NULL; /* leak into this response's byte accounting */ } /* Called by http_worker after the El handler returns and the response is sent. @@ -1362,11 +1370,14 @@ static void http_send_response(int fd, const char* body) { } const char* eff_body = is_envelope ? env_body : body; - /* Use the real byte count from fs_read if available (handles binary files - * with embedded null bytes — PNG, WOFF2, etc.). Fall back to strlen for - * normal text/JSON responses where _tl_fs_read_len is 0. */ - size_t blen = (_tl_fs_read_len > 0) ? _tl_fs_read_len : strlen(eff_body); + /* Use the real byte count from fs_read ONLY when this body IS the exact + * buffer fs_read returned (binary files with embedded null bytes — PNG, + * WOFF2, etc.). Any other body — wrapped, enveloped, or derived — must be + * measured with strlen, or it is truncated/over-read to the file's size. */ + size_t blen = (_tl_fs_read_len > 0 && eff_body == _tl_fs_read_buf) + ? _tl_fs_read_len : strlen(eff_body); _tl_fs_read_len = 0; /* consume — one-shot per response */ + _tl_fs_read_buf = NULL; int head_only = _tl_http_head_only; JsonBuf hdrs; jb_init(&hdrs); @@ -1438,11 +1449,22 @@ static void* http_worker(void* arg) { const char* rs = EL_CSTR(r); /* Copy response out BEFORE arena teardown. * For binary files, _tl_fs_read_len holds the real byte count — - * use memcpy instead of strdup so null bytes are preserved. */ - size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0); + * use memcpy instead of strdup so null bytes are preserved. + * The stored length applies ONLY when the response IS the exact + * fs_read buffer; a wrapped/derived response must use strlen or + * it gets truncated (or over-read) to the file's length. */ + size_t rlen; + if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) { + rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */ + } else { + rlen = rs ? strlen(rs) : 0; + _tl_fs_read_len = 0; /* hint doesn't describe this body */ + _tl_fs_read_buf = NULL; + } response = malloc(rlen + 1); if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; } else if (response) { response[0] = '\0'; } + if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */ } else { response = el_strdup_persist("el-runtime: no http handler registered"); } @@ -1684,10 +1706,20 @@ static void* http_worker_v2(void* arg) { el_val_t hmap = http_build_headers_map(hdr_block ? hdr_block : ""); el_val_t r = h(EL_STR(dispatch_method), EL_STR(path), hmap, EL_STR(body)); const char* rs = EL_CSTR(r); - size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0); + /* Same pairing rule as the v1 worker: the fs_read length is only + * trustworthy for the exact buffer fs_read returned. */ + size_t rlen; + if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) { + rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */ + } else { + rlen = rs ? strlen(rs) : 0; + _tl_fs_read_len = 0; /* hint doesn't describe this body */ + _tl_fs_read_buf = NULL; + } response = malloc(rlen + 1); if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; } else if (response) { response[0] = '\0'; } + if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */ el_release(hmap); } else { response = el_strdup_persist( @@ -1803,6 +1835,7 @@ el_val_t http_response(el_val_t status, el_val_t headers_json, el_val_t body) { el_val_t fs_read(el_val_t pathv) { const char* path = EL_CSTR(pathv); _tl_fs_read_len = 0; + _tl_fs_read_buf = NULL; if (!path) return el_wrap_str(el_strdup("")); FILE* f = fopen(path, "rb"); if (!f) return el_wrap_str(el_strdup("")); @@ -1814,6 +1847,7 @@ el_val_t fs_read(el_val_t pathv) { size_t got = fread(buf, 1, (size_t)sz, f); buf[got] = '\0'; _tl_fs_read_len = got; /* store real byte count for binary-safe send */ + _tl_fs_read_buf = buf; /* ...valid ONLY for this exact buffer */ fclose(f); return el_wrap_str(buf); } @@ -3184,8 +3218,10 @@ el_val_t json_get_raw(el_val_t json_str, el_val_t key) { const char* k = EL_CSTR(key); const char* p = json_find_key(json, k); /* Clear fs_read binary-length hint — result is a fresh null-terminated - * string, not the raw file bytes, so Content-Length must use strlen. */ + * string, not the raw file bytes, so Content-Length must use strlen. + * (Kept although the pointer pairing now makes this redundant.) */ _tl_fs_read_len = 0; + _tl_fs_read_buf = NULL; if (!p) return el_wrap_str(el_strdup("")); const char* end = json_skip_value(p); size_t n = (size_t)(end - p); From ff577391f2185949228de7b3f21617d621acb8fc Mon Sep 17 00:00:00 2001 From: Will Anderson Date: Wed, 22 Jul 2026 15:56:08 -0500 Subject: [PATCH 8/9] reconcile(release-runtime): Windows-port + complete v1.0.0 release runtime so the desktop soul cross-compiles MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The desktop soul (neuron/dist) compiles against the v1.0.0-20260501 release runtime. After #66 landed the engram natives (tokenized/ranked search, engram_prune_telemetry) and #79 the durable truncation fix into this runtime, two gaps remained before it could cross-compile the Windows brain: 1. Windows OS boundary: the release runtime had no Win32 path. Ported the same _WIN32-guarded shim the mainline runtime carries (#69): #ifdef _WIN32 -> el_platform_win.h (winsock/dlsym/popen + WSAStartup ctor), SOCKET fd guards and el_closesocket() at every socket site, CreateProcessA for exec_bg, the tm_zone/mingw guard, an el_setsockopt optval wrapper (GCC14), and curl-less libcurl stubs. Every change is _WIN32/HAVE_CURL-gated — the POSIX build is byte-identical (gcc -fsyntax-only clean; native behaviour unchanged). 2. Header exports: the release el_runtime.h omitted symbols the soul dist calls that are defined in this runtime's .c — the http_handler_fn/http_handler4_fn typedefs and el_arena_push/pop, engram_prune_telemetry, engram_get_node_by_label. Declaration-only, POSIX-neutral; fixes implicit-declaration/unknown-type errors under the C11 mingw build. Result: x86_64-w64-mingw32-gcc compiles el_runtime.c + all 48 soul modules clean; POSIX gcc -fsyntax-only clean. This is the Windows-port PR the runtime needed on main (the release-runtime counterpart to #69), landed via stage. --- .../v1.0.0-20260501/el_platform_win.h | 186 ++++++++++++++++++ lang/releases/v1.0.0-20260501/el_runtime.c | 80 ++++++-- lang/releases/v1.0.0-20260501/el_runtime.h | 12 ++ 3 files changed, 266 insertions(+), 12 deletions(-) create mode 100644 lang/releases/v1.0.0-20260501/el_platform_win.h diff --git a/lang/releases/v1.0.0-20260501/el_platform_win.h b/lang/releases/v1.0.0-20260501/el_platform_win.h new file mode 100644 index 0000000..1042f44 --- /dev/null +++ b/lang/releases/v1.0.0-20260501/el_platform_win.h @@ -0,0 +1,186 @@ +#ifndef EL_PLATFORM_WIN_H +#define EL_PLATFORM_WIN_H +/* + * el_platform_win.h — Windows OS-boundary shim for el_runtime.c. + * + * Branch: feat/windows-el-runtime. Included ONLY when _WIN32 is defined; the POSIX build is + * untouched. Goal: let el_runtime.c (a BSD-sockets / dlfcn / fork host) compile and link with + * mingw-w64 into a native neuron.exe, with no behavioural change to the Linux/macOS build. + * + * What it maps: + * - sockets : winsock2 (same call names: socket/bind/listen/accept/recv/send/setsockopt). + * Sockets close with closesocket() (see el_closesocket), and the stack must be + * started once with WSAStartup — done automatically via a load-time constructor. + * - dlsym : el_runtime.c uses dlsym(RTLD_DEFAULT, name) to resolve callback/tool symbols + * exported by the main module. Windows equivalent: GetProcAddress on the process + * module. Link the soul with -Wl,--export-all-symbols so the symbols are findable. + * - popen : mapped to _popen/_pclose. + * - threads : UNCHANGED. mingw-w64 ships winpthreads, so + -lpthread just work. + */ + +#ifndef WIN32_LEAN_AND_MEAN +#define WIN32_LEAN_AND_MEAN +#endif +#include +#include +#include +#include +#include + +/* Portable headers mingw-w64 provides (verified present). */ +#include +#include +#include +#include +#include +#include /* strcasecmp */ +#include +#include +#include +#include /* mingw-w64 provides gettimeofday here */ +#include +#include +#include +#include +#include +#include + +/* ── socket close ─────────────────────────────────────────────────────────── */ +/* Winsock closes sockets with closesocket(), not close() (close() is for file fds). The POSIX + build defines the same helper as close() so the call sites are identical across platforms. */ +static inline int el_closesocket(SOCKET s) { return closesocket(s); } + +/* ── setsockopt optval type ───────────────────────────────────────────────── */ +/* Winsock's setsockopt takes optval as (const char*); POSIX takes (const void*), so el_runtime.c + passes &int directly. GCC 14+ makes that an error under -Wincompatible-pointer-types. Wrap it so + the runtime's POSIX-style call sites compile unchanged (defined before the macro so the wrapper + itself resolves to the real winsock setsockopt). */ +static inline int el_setsockopt(SOCKET s, int level, int optname, const void* optval, int optlen) { + return setsockopt(s, level, optname, (const char*)optval, optlen); +} +#define setsockopt(s, l, o, v, n) el_setsockopt((s), (l), (o), (v), (int)(n)) + +/* ── winsock init (once, at load) ─────────────────────────────────────────── */ +static void el__win_net_init(void) { + static int inited = 0; + if (!inited) { WSADATA w; WSAStartup(MAKEWORD(2, 2), &w); inited = 1; } +} +__attribute__((constructor)) static void el__win_ctor(void) { el__win_net_init(); } + +/* ── dlsym → GetProcAddress ───────────────────────────────────────────────── */ +#ifndef RTLD_DEFAULT +#define RTLD_DEFAULT ((void*)0) +#endif +static inline void* el_win_dlsym(void* handle, const char* name) { + (void)handle; + return (void*)(uintptr_t)GetProcAddress(GetModuleHandleA(NULL), name); +} +#define dlsym(h, n) el_win_dlsym((h), (n)) + +/* ── popen / pclose ───────────────────────────────────────────────────────── */ +#define popen _popen +#define pclose _pclose + +/* ── misc POSIX → Win32 shims ─────────────────────────────────────────────── */ +#include /* _mkdir */ +#define mkdir(path, mode) _mkdir(path) /* POSIX mkdir(path,mode) → _mkdir(path) */ +#define timegm _mkgmtime /* UTC tm → time_t */ + +/* setenv/unsetenv: not in the Windows CRT; map to _putenv_s / SetEnvironmentVariable. */ +static inline int setenv(const char* name, const char* value, int overwrite) { + (void)overwrite; + return _putenv_s(name, value ? value : ""); +} +static inline int unsetenv(const char* name) { + /* _putenv_s(name, "") sets VAR="" rather than removing it. + * SetEnvironmentVariableA(name, NULL) truly deletes it from the Win32 + * env block; then we sync the CRT cache with _putenv("NAME="). */ + SetEnvironmentVariableA(name, NULL); + size_t len = strlen(name); + char *buf = (char*)malloc(len + 2); + if (!buf) return -1; + memcpy(buf, name, len); + buf[len] = '='; + buf[len + 1] = '\0'; + _putenv(buf); + free(buf); + return 0; +} + +/* nanosleep — not available in MSVC/UCRT; approximate with Sleep(). */ +static inline int el_nanosleep(const struct timespec *req, struct timespec *rem) { + (void)rem; + DWORD ms = (DWORD)((req->tv_sec * 1000ULL) + (req->tv_nsec / 1000000ULL)); + Sleep(ms ? ms : 1); + return 0; +} +#define nanosleep(req, rem) el_nanosleep((req), (rem)) + +/* localtime_r/gmtime_r: Windows offers localtime_s/gmtime_s with reversed arg order. */ +static inline struct tm* localtime_r(const time_t* t, struct tm* out) { + return localtime_s(out, t) == 0 ? out : (struct tm*)0; +} +static inline struct tm* gmtime_r(const time_t* t, struct tm* out) { + return gmtime_s(out, t) == 0 ? out : (struct tm*)0; +} + +/* ── libcurl: degradable stubs for the curl-less Windows build ─────────────── */ +/* The curl-less validation build (WITH_CURL=0) links no libcurl. el_runtime.c uses libcurl + * unconditionally for its HTTP client / LLM layer; these stubs let it compile and link so the + * runtime, HTTP *server*, graph and memory work natively on Windows. Live outbound HTTP/LLM calls + * degrade to a runtime error (curl_easy_perform returns an error) — matching the documented + * curl-less contract. When HAVE_CURL is defined (WITH_CURL=1) the real is used and + * this whole block is compiled out. POSIX never sees this header, so the POSIX build is untouched. */ +#ifndef HAVE_CURL + +typedef void CURL; +typedef int CURLcode; + +#define CURLE_OK 0 +#define CURLE_HTTP_RETURNED_ERROR 22 +#define CURL_ERROR_SIZE 256 + +/* Option ids: values are irrelevant to the no-op setopt below; kept distinct for readability. */ +#define CURLOPT_URL 10002 +#define CURLOPT_WRITEFUNCTION 20011 +#define CURLOPT_WRITEDATA 10001 +#define CURLOPT_POSTFIELDS 10015 +#define CURLOPT_POSTFIELDSIZE 120 +#define CURLOPT_POST 47 +#define CURLOPT_HTTPHEADER 10023 +#define CURLOPT_TIMEOUT_MS 155 +#define CURLOPT_NOSIGNAL 99 +#define CURLOPT_USERAGENT 10018 +#define CURLOPT_FOLLOWLOCATION 52 +#define CURLOPT_ERRORBUFFER 10010 +#define CURLOPT_CUSTOMREQUEST 10036 +#define CURLOPT_FAILONERROR 45 + +struct curl_slist { char* data; struct curl_slist* next; }; + +static inline struct curl_slist* curl_slist_append(struct curl_slist* list, const char* s) { + struct curl_slist* node = (struct curl_slist*)malloc(sizeof(struct curl_slist)); + if (!node) return list; + node->data = s ? strdup(s) : NULL; + node->next = NULL; + if (!list) return node; + struct curl_slist* p = list; + while (p->next) p = p->next; + p->next = node; + return list; +} +static inline void curl_slist_free_all(struct curl_slist* list) { + while (list) { struct curl_slist* n = list->next; free(list->data); free(list); list = n; } +} + +static inline CURL* curl_easy_init(void) { return (CURL*)malloc(1); } +static inline CURLcode curl_easy_setopt(CURL* h, int opt, ...) { (void)h; (void)opt; return CURLE_OK; } +static inline CURLcode curl_easy_perform(CURL* h) { (void)h; return 7 /* CURLE_COULDNT_CONNECT */; } +static inline void curl_easy_cleanup(CURL* h) { free(h); } +static inline const char* curl_easy_strerror(CURLcode c) { + (void)c; return "libcurl not built in (curl-less build)"; +} + +#endif /* !HAVE_CURL */ + +#endif /* EL_PLATFORM_WIN_H */ diff --git a/lang/releases/v1.0.0-20260501/el_runtime.c b/lang/releases/v1.0.0-20260501/el_runtime.c index d05edfe..4629cc2 100644 --- a/lang/releases/v1.0.0-20260501/el_runtime.c +++ b/lang/releases/v1.0.0-20260501/el_runtime.c @@ -21,6 +21,10 @@ #include "el_runtime.h" +#ifdef _WIN32 +/* Windows OS-boundary shim (winsock/dlsym/popen). Threading stays on (winpthreads). */ +#include "el_platform_win.h" +#else #include #include /* strcasecmp */ #include @@ -42,7 +46,16 @@ #include #include #include +/* On POSIX, sockets close with the same close() as files; el_platform_win.h supplies the Windows + variant. Defined here so the socket call sites are identical across platforms. */ +static inline int el_closesocket(int s) { return close(s); } +#endif +/* libcurl: present on POSIX and on the WITH_CURL Windows build; absent on the curl-less Windows + validation build, where el_platform_win.h supplies degradable stubs. On POSIX (_WIN32 undefined) + this is always taken, so the POSIX build is unchanged. */ +#if !defined(_WIN32) || defined(HAVE_CURL) #include +#endif /* ── Internal allocators ─────────────────────────────────────────────────── */ @@ -1468,12 +1481,20 @@ static void http_send_response(int fd, const char* body) { } typedef struct { +#ifdef _WIN32 + SOCKET fd; +#else int fd; +#endif } HttpWorkerArg; static void* http_worker(void* arg) { HttpWorkerArg* a = (HttpWorkerArg*)arg; +#ifdef _WIN32 + SOCKET fd = a->fd; +#else int fd = a->fd; +#endif free(a); char *method = NULL, *path = NULL, *body = NULL; if (http_read_request(fd, &method, &path, &body, NULL) == 0) { @@ -1516,7 +1537,7 @@ static void* http_worker(void* arg) { free(response); } free(method); free(path); free(body); - close(fd); + el_closesocket(fd); /* release a slot */ pthread_mutex_lock(&_http_conn_mu); _http_conn_active--; @@ -1546,14 +1567,18 @@ void http_serve(el_val_t port, el_val_t handler) { addr.sin6_addr = in6addr_any; addr.sin6_port = htons((uint16_t)p); if (bind(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0) { - perror("bind"); close(sock); return; + perror("bind"); el_closesocket(sock); return; } - if (listen(sock, 64) < 0) { perror("listen"); close(sock); return; } + if (listen(sock, 64) < 0) { perror("listen"); el_closesocket(sock); return; } fprintf(stderr, "[http] listening on [::]:%d (dual-stack)\n", p); while (1) { struct sockaddr_in6 cli; socklen_t clen = sizeof(cli); +#ifdef _WIN32 + SOCKET cfd = accept(sock, (struct sockaddr*)&cli, &clen); +#else int cfd = accept(sock, (struct sockaddr*)&cli, &clen); +#endif if (cfd < 0) { if (errno == EINTR) continue; perror("accept"); break; @@ -1565,11 +1590,11 @@ void http_serve(el_val_t port, el_val_t handler) { _http_conn_active++; pthread_mutex_unlock(&_http_conn_mu); HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg)); - if (!arg) { close(cfd); continue; } + if (!arg) { el_closesocket(cfd); continue; } arg->fd = cfd; pthread_t tid; if (pthread_create(&tid, NULL, http_worker, arg) != 0) { - close(cfd); free(arg); + el_closesocket(cfd); free(arg); pthread_mutex_lock(&_http_conn_mu); _http_conn_active--; pthread_cond_signal(&_http_conn_cv); @@ -1578,7 +1603,7 @@ void http_serve(el_val_t port, el_val_t handler) { } pthread_detach(tid); } - close(sock); + el_closesocket(sock); } /* ── http_serve_async — non-blocking HTTP server ─────────────────────────── */ @@ -1814,7 +1839,11 @@ static el_val_t http_build_headers_map(const char* hdr_block) { static void* http_worker_v2(void* arg) { HttpWorkerArg* a = (HttpWorkerArg*)arg; +#ifdef _WIN32 + SOCKET fd = a->fd; +#else int fd = a->fd; +#endif free(a); char *method = NULL, *path = NULL, *body = NULL, *hdr_block = NULL; if (http_read_request(fd, &method, &path, &body, &hdr_block) == 0) { @@ -1854,7 +1883,7 @@ static void* http_worker_v2(void* arg) { free(response); } free(method); free(path); free(body); free(hdr_block); - close(fd); + el_closesocket(fd); pthread_mutex_lock(&_http_conn_mu); _http_conn_active--; pthread_cond_signal(&_http_conn_cv); @@ -1884,14 +1913,18 @@ void http_serve_v2(el_val_t port, el_val_t handler) { addr.sin6_addr = in6addr_any; addr.sin6_port = htons((uint16_t)p); if (bind(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0) { - perror("bind"); close(sock); return; + perror("bind"); el_closesocket(sock); return; } - if (listen(sock, 64) < 0) { perror("listen"); close(sock); return; } + if (listen(sock, 64) < 0) { perror("listen"); el_closesocket(sock); return; } fprintf(stderr, "[http v2] listening on [::]:%d (dual-stack)\n", p); while (1) { struct sockaddr_in6 cli; socklen_t clen = sizeof(cli); +#ifdef _WIN32 + SOCKET cfd = accept(sock, (struct sockaddr*)&cli, &clen); +#else int cfd = accept(sock, (struct sockaddr*)&cli, &clen); +#endif if (cfd < 0) { if (errno == EINTR) continue; perror("accept"); break; @@ -1903,11 +1936,11 @@ void http_serve_v2(el_val_t port, el_val_t handler) { _http_conn_active++; pthread_mutex_unlock(&_http_conn_mu); HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg)); - if (!arg) { close(cfd); continue; } + if (!arg) { el_closesocket(cfd); continue; } arg->fd = cfd; pthread_t tid; if (pthread_create(&tid, NULL, http_worker_v2, arg) != 0) { - close(cfd); free(arg); + el_closesocket(cfd); free(arg); pthread_mutex_lock(&_http_conn_mu); _http_conn_active--; pthread_cond_signal(&_http_conn_cv); @@ -1916,7 +1949,7 @@ void http_serve_v2(el_val_t port, el_val_t handler) { } pthread_detach(tid); } - close(sock); + el_closesocket(sock); } /* Build the response envelope a 4-arg handler can return. We hand-write @@ -2063,6 +2096,23 @@ el_val_t exec(el_val_t cmdv) { el_val_t exec_bg(el_val_t cmdv) { const char* cmd = EL_CSTR(cmdv); if (!cmd || !*cmd) return el_wrap_str(el_strdup("")); +#ifdef _WIN32 + /* Windows: no fork/exec. Launch a detached `cmd /c ` with no console window via + CreateProcess (DETACHED_PROCESS | CREATE_NO_WINDOW). Returns the PID as a string, "" on fail. + Mirrors the POSIX branch: child runs independently, caller is not blocked. */ + char cmdline[8192]; + snprintf(cmdline, sizeof(cmdline), "cmd.exe /c %s", cmd); + STARTUPINFOA si; ZeroMemory(&si, sizeof(si)); si.cb = sizeof(si); + PROCESS_INFORMATION pi; ZeroMemory(&pi, sizeof(pi)); + BOOL ok = CreateProcessA(NULL, cmdline, NULL, NULL, FALSE, + DETACHED_PROCESS | CREATE_NO_WINDOW, NULL, NULL, &si, &pi); + if (!ok) return el_wrap_str(el_strdup("")); + char pidbuf[32]; + snprintf(pidbuf, sizeof(pidbuf), "%lu", (unsigned long)pi.dwProcessId); + CloseHandle(pi.hProcess); + CloseHandle(pi.hThread); + return el_wrap_str(el_strdup(pidbuf)); +#else pid_t pid = fork(); if (pid < 0) { /* fork failed */ @@ -2085,6 +2135,7 @@ el_val_t exec_bg(el_val_t cmdv) { char pidbuf[32]; snprintf(pidbuf, sizeof(pidbuf), "%d", (int)pid); return el_wrap_str(el_strdup(pidbuf)); +#endif } el_val_t fs_list(el_val_t pathv) { @@ -4214,7 +4265,12 @@ static int _el_decompose_earth(el_caltime_t* ct, struct tm* tm_out, int* abbr_le localtime_r(&s, &tm); *tm_out = tm; if (abbr_buf && abbr_cap > 0) { + /* mingw's struct tm has no tm_zone (BSD/glibc extension); no abbrev available there. */ +#ifdef _WIN32 + const char* z_str = ""; +#else const char* z_str = tm.tm_zone ? tm.tm_zone : ""; +#endif size_t n = strlen(z_str); if (n >= abbr_cap) n = abbr_cap - 1; memcpy(abbr_buf, z_str, n); diff --git a/lang/releases/v1.0.0-20260501/el_runtime.h b/lang/releases/v1.0.0-20260501/el_runtime.h index a3dd683..c3dad8e 100644 --- a/lang/releases/v1.0.0-20260501/el_runtime.h +++ b/lang/releases/v1.0.0-20260501/el_runtime.h @@ -758,6 +758,18 @@ el_val_t trace_span_start(el_val_t name); el_val_t trace_span_end(el_val_t span_handle); el_val_t emit_event(el_val_t name, el_val_t duration_ms); +/* ── Runtime symbols required by the soul modules ──────────────────────────── */ +/* All implemented in el_runtime.c but omitted from this release header; the soul dist modules + * reference them directly, so the public header must export them. Declarations only — mirrors the + * mainline el_runtime.h and is platform-independent (no behavioural change to the POSIX build). */ +typedef el_val_t (*http_handler_fn)(el_val_t method, el_val_t path, el_val_t body); +typedef el_val_t (*http_handler4_fn)(el_val_t method, el_val_t path, el_val_t body, el_val_t headers); +el_val_t el_arena_push(void); +el_val_t el_arena_pop(el_val_t mark); +void http_serve_async(el_val_t port, el_val_t handler); +el_val_t engram_get_node_by_label(el_val_t label); +el_val_t engram_prune_telemetry(el_val_t older_than_ms); + #ifdef __cplusplus } #endif From b97b644799d27e0b91953cefe0b6c41df2ff6a28 Mon Sep 17 00:00:00 2001 From: Andre Botelho Rodrigues Almeida Date: Thu, 23 Jul 2026 16:33:18 -0300 Subject: [PATCH 9/9] Addind readme.md file to start documenting the repo --- README.md | 154 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..ef76ffa --- /dev/null +++ b/README.md @@ -0,0 +1,154 @@ +# El + +**A self-hosting, statically-typed language that compiles to C — built around a graph-native runtime instead of a database driver.** + +El is the execution substrate for the Neuron agent runtime, the DHARMA network, and the Engram knowledge graph. This repository is the monorepo for the whole stack: the language itself, the graph memory engine it's built to talk to natively, and the tools (package manager, IDE, UI framework, diagramming) built on top of it. + +--- + +## Why El exists + +Every other language treats persistent, associative state as something you reach for through a driver — a SQL client, an ORM, a Redis library bolted on from outside. El inverts that: graph operations (`engram_*`) are runtime primitives, on the same footing as string or list operations. There is no separate database driver because the database is not separate. + +El has four defining properties: + +1. **Self-hosting compiler.** The compiler (`lexer.el`, `parser.el`, `codegen.el`, `compiler.el`) is written in El. It compiles El source to C, which `cc` compiles against a fixed runtime into a native binary. A Rust genesis compiler bootstrapped the first iteration; the self-hosted binary at `lang/dist/platform/elc` has been the canonical compiler ever since — every binary in `dist/platform/` was produced by an earlier version of itself compiling `el-compiler/src/`. The chain is auditable: source is the ground truth, not the binary. See [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) for the full recovery path if that binary is ever lost. +2. **C compilation target.** Every compiled program is plain C11. Every El value is `el_val_t` (`int64_t`); strings are heap pointers cast through it. Functions become C functions; top-level statements become `main()`. +3. **Graph-native runtime.** The runtime provides first-class graph operations over an in-process Engram store — no separate DB driver, no ORM. +4. **DHARMA-aware identity.** A `cgi` block declares a program's DHARMA identity at compile time. The runtime resolves identity before user code runs, so `dharma_*` calls have a stable principal and channel surface throughout. + +--- + +## Architecture map + +``` + ┌─────────────┐ + │ lang │ El compiler + C runtime + │ (El itself) │ everything below is written in it, + └──────┬──────┘ or compiles down through it + │ + ┌─────────────┼─────────────┐ + │ │ │ + ┌──────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐ + │ engram │ │ epm │ │ ide │ + │ graph/mem │ │ package │ │ editor + │ + │ substrate │ │ manager │ │ LSP │ + └──────┬─────┘ └───────────┘ └───────────┘ + │ + ┌───────┼────────────────┬─────────────────────┐ + │ │ │ │ +┌─────▼───┐ ┌─▼──────────┐ ┌──▼──────────┐ ┌─────▼──────┐ +│ elp │ │ ql │ │ ui │ │ arbor │ +│ NLG / │ │engram-el. │ |spreading- │ |arbor │ +│ 31 langs│ │studio+tests│ |activation UI│ |diagram lang│ +└─────────┘ └────────────┘ └─────────────┘ └────────────┘ +``` + +`lang` is the foundation — the compiler and C runtime everything else builds on. `engram` is the graph-native memory/state engine that gives El its identity (property 3 above). Everything else is either a tool for working with El (`epm`, `ide`) or a system built on top of Engram's graph model (`elp`, `ql`, `ui`, `arbor`). + +--- + +## Repository layout + +### [lang/](lang/) — the El language + +The compiler and runtime. Self-hosting: `elc-cli.el` → `compiler.el` → `lexer.el` / `parser.el` / `codegen.el` / `codegen-js.el`, textually inlined and compiled in one pass. Compiles to C11 and links against `el-compiler/runtime/el_seed.c`, a hand-maintained OS-boundary layer (libcurl HTTP, pthreads, filesystem, arena allocation) — everything else in the runtime is native El (`runtime/*.el`). + +Two layers to know: **El programs** (`.el` files — where nearly all work belongs) and **the C seed** (`el_seed.c` — edit only for genuine OS-level access; never re-implement what El can already express). + +Current status (single source of truth: [lang/spec/language.md](lang/spec/language.md)): lexer/parser/codegen and the C runtime's core (I/O, strings, math, lists, maps, filesystem, args) are implemented. In flight: `%` operator, match-statement codegen, `?` nil-propagation, `cgi` block parsing + DHARMA identity resolution, VBD role enforcement (`@manager`/`@engine`/`@accessor`), the real `engram_*` and `dharma_*` runtimes (currently stubs), and libcurl-backed `http_get`/`http_post`/`http_serve`. Bitwise operators, `??`, and `as` casts are explicitly **not** in this language. + +Key docs: [AGENTS.md](lang/AGENTS.md) (agent-facing orientation), [BOOTSTRAP.md](lang/BOOTSTRAP.md) (compiler recovery from scratch), [spec/language.md](lang/spec/language.md), [spec/codegen-js.md](lang/spec/codegen-js.md). + +### [engram/](engram/) — graph intelligence substrate + +**A local-first memory substrate for accumulating intelligence**, and the reason El's runtime doesn't need a database driver. Rust core (`engram-core`, `engram-ffi`) exposed to El and other languages (Kotlin, TypeScript/WASM, Go bindings). + +The model: retrieval is **spreading activation**, not query. You name seed nodes and a query embedding; activation propagates outward through weighted edges, attenuating multiplicatively per hop (`strength = parent_strength × edge_weight × target_salience × cosine_sim`), gets pruned below a threshold, and the top-N nodes by activation strength come back. Storage and retrieval are the same structure — the way long-term potentiation works in biological memory, not the way a relational or vector database works. + +Nodes live in four tiers (Working / Episodic / Semantic / Procedural, mirroring prefrontal / hippocampal / neocortical / cerebellar memory) and migrate between them based on **salience decay** — `importance × recency-decay × log(activation_count)`. Forgetting is adaptive pruning, not a bug: unreinforced memories stop competing for attention without being deleted. + +Backed by `sled` (embedded, local-first, no daemon) with flat cosine scan for vector search — deliberately simple until scale demands an HNSW layer. Full API and design rationale in [engram/README.md](engram/README.md). + +### [elp/](elp/) — Engram Language Protocol + +Bidirectional engine mapping between Engram semantic forms and natural-language surface text, across **31 languages** — from Spanish and Japanese through historical/liturgical languages (Old Norse, Sanskrit, Sumerian, Coptic, Akkadian, Ge'ez). Compilation order runs `language-profile` + `vocabulary` → per-language `morphology-*` → `grammar` → `realizer` → `semantics` → `elp`. This is what lets an Engram graph node round-trip to and from readable text in any of those languages. + +### [epm/](epm/) — El Package Manager + +Manages **vessels** (El's package unit): publish, install, resolve dependencies. Vessels are stored in Engram as graph nodes, not files in a registry index — `epm` reads the local `manifest.el`, talks to Engram over HTTP, and writes resolved vessels to `.epm/vessels/`. Source: `registry.el`, `install.el`, `update.el`, `manifest.el`. + +### [ide/](ide/) — El IDE + +Three vessels: **el-ide-server** (HTTP backend — file ops, build/run, LSP bridge, plugin host, settings), **el-lsp** (the language server — completion, hover, diagnostics, outline, format, type graph), and **el-plugin-host** (first-party plugin lifecycle: install/remove/enable/disable). `ide/projects/` and `ide/examples/` hold sample projects, including the canonical `hello-friends` first-program walkthrough. + +### [ql/](ql/) — engram-el + +The El-native integration layer for a *live* Engram server — not a library (no importable modules, no build artifact), a set of standalone `.el` programs run directly via `el run-file`. Three components: **Studio** (`studio/studio.el`, a full terminal graph explorer), a **Hebbian field-model** proof of concept, and El builtin / LLM-builtin smoke test suites. This is the reference for correct patterns when an El program uses Engram as its substrate. Spec: [ql/spec/elql.md](ql/spec/elql.md). + +### [ui/](ui/) — el-ui + +A frontend framework where **component state is an Engram graph and reactivity is spreading activation** — not virtual-DOM diffing (React), Proxy-based dependency tracking (Vue), or compile-time analysis (Svelte). Re-renders are activated and propagated the same way associative memory retrieval works in `engram/`. + +~15 vessels covering the full frontend surface: `el-platform` (env/fs/network/clock abstraction), `el-config`, `el-html` (SSR emit primitives), `el-layout`, `el-style` (design tokens/themes), `el-i18n`, `el-auth` / `el-identity` (JWT, sessions, OAuth PKCE — Engram-native), `el-services` (REST/gRPC/WebSocket bindings), `el-aop` (`@authenticate`/`@authorize`/`@cache`/`@rate_limit` decorators), `el-secrets`, `el-graph` (graph rendering/editor), `el-publish` (App Store / Play Store automation), and `el-ui-compiler` (El→JS component compiler; currently a stub pending a JS backend in `elc`). Spec: [ui/spec/framework.md](ui/spec/framework.md). + +### [arbor/](arbor/) — diagram language + +A `.arbor` diagram language and toolchain: `arbor-core` (NodeId/shape/edge-kind types), `arbor-parse` (recursive-descent parser), `arbor-diagram` (IR + Mermaid serializer + architecture-diagram builders), `arbor-layout` (hierarchical layout — rank assignment, positioning, group bounds), `arbor-render` (SVG renderer), `arbor-cli`. (The architecture map above is the kind of diagram this is for.) + +--- + +## Getting started + +Install the El SDK from the latest release: + +```bash +bash lang/install.sh +# EL_VERSION=v1.0.0 bash lang/install.sh # pin a specific release tag +# EL_PREFIX=/opt/el bash lang/install.sh # custom install prefix +``` + +Or build the compiler from source and verify the self-hosting chain: + +```bash +cd lang +./dist/platform/elc elc-cli.el > elc-new.c +cc -std=c11 -I el-compiler/runtime -lcurl -lpthread \ + -o dist/platform/elc-new \ + elc-new.c el-compiler/runtime/el_seed.c + +# Confirm the new binary reproduces itself exactly +./dist/platform/elc-new elc-cli.el > elc-verify.c +diff elc-new.c elc-verify.c # should be identical + +mv dist/platform/elc-new dist/platform/elc +``` + +Run your first program: + +```bash +./lang/dist/platform/elc lang/examples/hello.el > hello.c +cc -std=c11 -I lang/el-compiler/runtime -lcurl -lpthread \ + -o hello hello.c lang/el-compiler/runtime/el_seed.c +./hello +``` + +More examples in [lang/examples/](lang/examples/), including a full starter project at `lang/examples/hello-project/`. + +If the compiler binary is ever lost or corrupted, [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) is the authoritative recovery path. + +--- + +## Development workflow + +Branching follows `dev → stage → main`: work lands on `dev`, promotes to `stage` for integration testing, and is promoted to `main` for release (visible directly in the git history of this repo). CI is defined per-subproject under `.gitea/workflows/` — `lang`/`epm`/`ide` share the root pipeline; `engram` and `ql` carry their own (`ci-dev`, `ci-stage`, and a release workflow each). + +- Language/runtime specs live at `*/spec/*.md` (`lang/spec/`, `ql/spec/`, `ui/spec/`) and are the single source of truth for implemented-vs-planned status — code and docs are expected to agree with the spec's status markers, not the other way around. +- Agent-facing orientation guides live at `*/AGENTS.md` (currently `lang/AGENTS.md`); more subprojects may grow their own as they need agent-specific conventions documented. +- Tagged releases live under `lang/releases/`, each with its own `RELEASE.md`. + +--- + +## Status + +This is an actively developed, internal monorepo — not yet published under an open license. Treat everything here as proprietary to Neuron Technologies unless told otherwise.