Compare commits

..

1 Commits

Author SHA1 Message Date
Neuron 8a307dfd42 runtime: make valid UTF-8 the JSON emitter's contract
El SDK CI - dev / build-and-test (pull_request) Failing after 10m36s
Three nodes in the live graph carry labels truncated to exactly 80 bytes
ending in a lone 0xE2 — the first byte of an em-dash, cut mid-sequence.
jb_emit_escaped copied every byte >= 0x20 through verbatim, so those three
nodes made the ENTIRE /api/nodes/list response undecodable and no strict
parser could read the graph at all.

  production binary   25,929,607 bytes   INVALID at byte 89260
  this build          26,338,389 bytes   VALID, parses to 13,630 nodes

The damage was NOT written by this runtime. No 80-byte truncation exists
here (the only label truncation is engram_first_n_chars at 60), and the
content of those nodes is 2572 and 2746 bytes. Some other producer wrote
them. That is exactly why fixing a writer could not have fixed this: the
store already holds the damage, and it accepts data from importers, other
producers and older binaries.

So the fix goes where the promise is made. A serializer that emits JSON
owes valid UTF-8 whatever it is handed. jb_emit_escaped now validates each
multi-byte sequence before emitting any of it and substitutes U+FFFD for a
bad lead byte, a missing or malformed continuation, an overlong encoding, a
UTF-16 surrogate, or a codepoint above U+10FFFF. Invalid bytes are REPLACED
rather than dropped, so the damage stays visible in the output instead of
being silently papered over. Well-formed input is byte-identical to before.

Second, preventive and explicitly NOT the cause of the above:
engram_first_n_chars truncated by BYTES despite its name, so content with a
multi-byte character crossing byte 60 would produce a half codepoint in the
label. It now uses el_utf8_safe_len, which returns the largest byte length
<= max that does not split a codepoint. Bounded by bytes, not codepoints,
so existing labels never grow — they only stop splitting.

el_utf8_safe_len lives beside str_count_chars rather than in the engram
because the rest of el's string layer is already codepoint-aware
(str_count_chars counts codepoints, str_reverse walks codepoint lengths).
Byte truncation was the outlier and the concern is a string concern.

Note on the investigation: I first "fixed" the truncator and wrote a test
that passed on the UNPATCHED build too, because route_create_node passes
label = content when no label is supplied, so engram_first_n_chars is never
reached over HTTP. The test proved nothing. The real cause was only found
by decoding the actual failing bytes out of the live response.
2026-08-16 12:03:03 -05:00
14 changed files with 246 additions and 2130 deletions
+52 -135
View File
@@ -10,60 +10,10 @@
// cc -std=c11 -O2 -lcurl -lpthread -o engram server.c el_runtime.c
// ./engram
//
// Configuration is DECLARED, not scattered. See the `program` block below:
// every knob's type and default lives there and nowhere else, is resolved from
// the environment (env wins, declaration is the fallback) and validated before
// any statement of this file runs. Read one with config("NAME") -> String.
//
// The one deliberate exception is ENGRAM_DATA_DIR see the note in the block.
// Program declaration (cross-cutting concerns)
//
// singleton: two engram processes against one data dir is data loss, not a
// warning. The runtime takes an exclusive flock at startup and a second start
// is refused loudly with the holder's pid.
//
// NOT declared here, on purpose: ENGRAM_DATA_DIR. Its resolution is owned by
// engram_resolve_data_dir() (el_runtime.c), which defaults to $HOME/.neuron/engram
// and fails LOUD rather than silently persisting to an ephemeral directory.
// Declaring a default for it here as well would put the data dir's fallback in
// two places which is precisely the defect this migration removes (until
// 2026-08-15 the reseed backup path carried its own "/tmp/engram" default that
// disagreed with the resolver, so the pre-destructive safety copy landed in /tmp).
// HOME is likewise not declared: it is a genuine environment read, not a knob.
program "engram" {
singleton: "engram"
// Core server
env ENGRAM_BIND: String = ":8742"
// Default "" leaves auth DISABLED (check_auth_ok short-circuits to true on an
// empty key). That is the pre-existing behaviour and is deliberately preserved
// here; making this `required` is the obvious hardening follow-up, but it is a
// behaviour change and out of scope for this migration.
env ENGRAM_API_KEY: String = ""
// Feature flags (bool-ish Strings; the predicate fns below own truthiness) ──
env ENGRAM_STORE: String = "off"
env ENGRAM_WAL: String = "off"
env ENGRAM_AUTOCONNECT: String = "off"
env ENGRAM_ISE_OFFGRAPH: String = "off"
// ISE telemetry
env ENGRAM_ISE_RETENTION_MS: Int = "172800000"
// Guide (local Qwen3 via llama-server)
env GUIDE_ENABLE: String = "off"
env GUIDE_TIER_FORCE: String = ""
env GUIDE_CACHE_DIR: String = ""
env GUIDE_RAM_GB_4B: Int = "16"
env GUIDE_RAM_GB_1P7B: Int = "8"
env GUIDE_BACKEND: String = "llama-server"
env GUIDE_HOST: String = "127.0.0.1"
env GUIDE_PORT: Int = "8771"
env GUIDE_LLAMA_SERVER_BIN: String = "llama-server"
env GUIDE_NGL: Int = "99"
env GUIDE_CTX: Int = "4096"
}
// Configuration via environment:
// ENGRAM_BIND host:port (default :8742)
// ENGRAM_API_KEY bearer auth (optional)
// ENGRAM_DATA_DIR snapshot location (default ~/.neuron/engram)
// Helpers
@@ -183,7 +133,7 @@ fn route_text_health(method: String, path: String, body: String) -> String {
// engram_store_enabled() in el_runtime.c EXACTLY (1 / on / true). Default off
// every persistence path below is byte-for-byte the historical snapshot behavior.
fn store_on() -> Bool {
let v: String = config("ENGRAM_STORE")
let v: String = env("ENGRAM_STORE")
if str_eq(v, "1") { return true }
if str_eq(v, "on") { return true }
if str_eq(v, "true") { return true }
@@ -212,6 +162,7 @@ fn persist_canonical() -> Int {
if store_on() {
return engram_store_checkpoint()
}
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
// (2026-08-10 self-review) This returned a hardcoded 1, which made every
// caller's `let saved: Int = persist_canonical()` a dead variable six
@@ -225,7 +176,7 @@ fn persist_canonical() -> Int {
// per-write full-snapshot behavior. When ON, structural mutations append O(1)
// WAL records instead of rewriting the whole graph, with threshold compaction.
fn wal_on() -> Bool {
str_eq(config("ENGRAM_WAL"), "on")
str_eq(env("ENGRAM_WAL"), "on")
}
// autoconnect_on ENGRAM_AUTOCONNECT. Will's rule: "we shouldn't be inserting
@@ -233,7 +184,7 @@ fn wal_on() -> Bool {
// edge (kNN over embeddings) so no content node enters the graph edgeless.
// Default OFF -> byte-identical to prior behavior (node created, no auto edges).
fn autoconnect_on() -> Bool {
let v: String = config("ENGRAM_AUTOCONNECT")
let v: String = env("ENGRAM_AUTOCONNECT")
if str_eq(v, "1") { return true }
if str_eq(v, "on") { return true }
if str_eq(v, "true") { return true }
@@ -246,7 +197,7 @@ fn autoconnect_on() -> Bool {
// separate state-event log tier instead of the node graph. Default OFF -> ISEs
// remain graph nodes exactly as before (with 48h prune).
fn ise_offgraph_on() -> Bool {
let v: String = config("ENGRAM_ISE_OFFGRAPH")
let v: String = env("ENGRAM_ISE_OFFGRAPH")
if str_eq(v, "1") { return true }
if str_eq(v, "on") { return true }
if str_eq(v, "true") { return true }
@@ -296,24 +247,6 @@ fn persist_bulk() -> Int {
return persist_canonical()
}
// COMPILER LANDMINE, measured 2026-08-16 do not inline this back into the
// caller. elc lowers `a == b` to numeric comparison only when both operand
// NAMES are in the per-function int-name set, which `let x: Int` populates.
// That registration does NOT propagate into a nested if-expression block: the
// first cut of the geometry-ingest path wrote `let claimed: Int = ...` and
// `let got: Int = ...` inside the else-arm and `claimed == got` came out of
// codegen as `str_eq(claimed, got)` strcmp on two integers reinterpreted as
// pointers, i.e. a segfault on the first geometry-bearing request. Read back
// out of the generated C, not guessed. Function PARAMETERS annotated `: Int`
// do register reliably (verified: `if (claimed == actual)`), so the comparison
// lives in a function of its own. Note also the explicit `return`s a trailing
// if-EXPRESSION at a function tail emits as a statement and the function
// returns 0 regardless, which is the same probe's second finding.
fn width_agrees(claimed: Int, actual: Int) -> Int {
if claimed == actual { return 1 }
return 0
}
// INCOMPLETE-ROUTE FIX (2026-07-24 self-review): this route silently dropped
// label, importance, tier, and tags engram_node() defaults label to content
// and importance to 0.5, so every node created over HTTP lost its metadata.
@@ -355,44 +288,26 @@ fn route_create_node(method: String, path: String, body: String) -> String {
salience, importance, confidence,
tier, tags
)
// GEOMETRY INGEST geometry-valued end to end (2026-08-16).
// GEOMETRY INGEST (2026-08-16 self-review): this route accepted an "emb"
// field, returned 200 with a fresh id, and stored NOTHING engram_node_full
// has no vector parameter, so the caller's geometry was silently discarded
// and the node came back emb_dim=None / embedded:false. Measured live while
// trying to admit a voice signal. The consequence was structural, not
// cosmetic: text was the only entry medium, so any non-text modality had to
// be DESCRIBED in prose and what we then reasoned over was the geometry of
// the description, not of the signal.
//
// The defect this route originally had: it accepted an "emb" field,
// returned 200 with a fresh id, and stored NOTHING, because engram_node_full
// has no vector parameter. The consequence was structural, not cosmetic
// text was the only entry medium, so any non-text modality had to be
// DESCRIBED in prose, and what we then reasoned over was the geometry of the
// description, not of the signal.
//
// #141 fixed the drop but marshalled the vector as a hex STRING through
// engram_node_set_emb, which put text back as the TRANSPORT medium one layer
// below the problem being fixed. This is that correction: hex is decoded
// exactly ONCE, here at the edge, into a first-class Geometry, and every
// step below this line moves geometry rather than text. An encoding at the
// boundary is what an encoding is for.
//
// The WIRE is deliberately unchanged "emb" is still little-endian float32
// hex (8 chars per component), the encoding the perception vessel's
// /voice/embed already emits because production clients speak it. What
// changed is underneath it.
//
// "dim" is now treated as an ASSERTION about the vector the caller sent, not
// as the source of its width: a Geometry carries its own width. A stated dim
// that disagrees is a REJECTED ingest, not a silent reinterpretation. Omitting
// "dim" is fine and means "trust the vector", which is the honest default.
//
// Off-dimension vectors remain stored but not inserted into the resident HNSW
// index (its build loop filters on emb_dim), so a 64-dim voice geometry is
// durable and addressable without perturbing the 768-dim canonical index.
// "emb" is little-endian float32 hex (dim*8 chars) the encoding the
// perception vessel's /voice/embed already emits, so a realizer's output
// moves in with no float-array round trip. "dim" defaults to the vector's
// implied width. Off-dimension vectors are stored but not inserted into the
// resident index (its build loop filters on emb_dim), so a modality vector
// is durable and addressable without perturbing the canonical index.
let emb_hex: String = json_get_string(body, "emb")
let emb_set: Int = if str_eq(emb_hex, "") { 0 } else {
let g: Geometry = geometry_from_f32le_hex(emb_hex)
let got: Int = geometry_dim(g)
let dim_raw: String = json_get_raw(body, "dim")
let claimed: Int = if str_eq(dim_raw, "") { got } else { json_get_int(body, "dim") }
let landed: Int = if width_agrees(claimed, got) > 0 { node_attach_geometry(id, g) } else { 0 }
let freed: Int = geometry_free(g)
landed
let dim: Int = if str_eq(dim_raw, "") { str_len(emb_hex) / 8 } else { json_get_int(body, "dim") }
engram_node_set_emb(id, emb_hex, dim)
}
let saved: Int = persist_node(id)
// ORPHAN PREVENTION (ENGRAM_AUTOCONNECT): connect the fresh node to its
@@ -443,6 +358,7 @@ fn route_scan_nodes(method: String, path: String, body: String) -> String {
// process ever booted with a partial/empty store, the first read request
// clobbered the good snapshot. Read routes must never write the canonical path.)
fn route_scan_edges(method: String, path: String, body: String) -> String {
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
let snap_path: String = dir + "/.scan-export.json"
engram_save(snap_path)
@@ -603,6 +519,7 @@ fn route_forget(method: String, path: String, body: String) -> String {
fn route_save(method: String, path: String, body: String) -> String {
let p_raw: String = json_get_string(body, "path")
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
let p: String = if str_eq(p_raw, "") { dir + "/snapshot.json" } else { p_raw }
// (2026-08-10 self-review) engram_save returns 0 on an empty path and the
@@ -686,6 +603,7 @@ fn route_drift(method: String, path: String, body: String) -> String {
fn route_load(method: String, path: String, body: String) -> String {
let p_raw: String = json_get_string(body, "path")
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
let p: String = if str_eq(p_raw, "") { dir + "/snapshot.json" } else { p_raw }
// (2026-08-10 self-review) This was a stub response over the single most
@@ -756,6 +674,7 @@ fn route_embed_backfill(method: String, path: String, body: String) -> String {
// (it skips nodes already present by ID). Auth-exempt: same-host internal call.
// (2026-06-27 self-review: added this route to fix silent 10-min sync failures)
fn route_sync(method: String, path: String, body: String) -> String {
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
// 2026-07-21 self-review: export to a scratch path, never the canonical
// snapshot.json read routes must not be able to clobber the good snapshot.
@@ -831,12 +750,8 @@ fn route_reseed_nodes(method: String, path: String, body: String) -> String {
if str_eq(p, "") { return err_json("path is required") }
if str_eq(fs_read(p), "") { return err_json("file missing or empty") }
// (2026-08-15) This site carried its own "/tmp/engram" fallback, which
// DISAGREED with engram_resolve_data_dir() ($HOME/.neuron/engram, fail-loud).
// The consumer is the pre-destructive backup below, so with ENGRAM_DATA_DIR
// unset the safety copy taken before a reseed landed in an ephemeral /tmp
// while the store it was protecting lived elsewhere. One owner, one answer.
let dir: String = engram_resolve_data_dir()
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw }
let backup: String = dir + "/.reseed-backup.json"
let replace_raw: String = json_get_raw(body, "replace")
@@ -928,7 +843,8 @@ fn route_emit_ise(method: String, path: String, body: String) -> String {
sal, imp, conf,
"Episodic", "[\"internal-state\",\"InternalStateEvent\"]"
)
let ret_ms: Int = str_to_int(config("ENGRAM_ISE_RETENTION_MS"))
let ret_raw: String = env("ENGRAM_ISE_RETENTION_MS")
let ret_ms: Int = if str_eq(ret_raw, "") { 172800000 } else { str_to_int(ret_raw) }
let pruned: Int = engram_prune_telemetry(ret_ms)
"{\"ok\":true,\"id\":\"" + id + "\",\"pruned\":" + int_to_str(pruned) + "}"
}
@@ -1177,12 +1093,14 @@ fn route_correspondence_beat(method: String, path: String, body: String) -> Stri
// turns native thinking ON: the response carries reasoning_content (the thinking)
// alongside content (the answer).
// (2026-08-15) guide_env_or(key, dflt) lived here. Its whole job was supplying a
// per-call-site default, which is now the program block's job every GUIDE_* knob
// is declared once at the top of this file and read straight through config().
fn guide_env_or(key: String, dflt: String) -> String {
let v: String = env(key)
if str_eq(v, "") { return dflt }
return v
}
fn guide_enabled() -> Bool {
let v: String = config("GUIDE_ENABLE")
let v: String = env("GUIDE_ENABLE")
if str_eq(v, "1") { return true }
if str_eq(v, "on") { return true }
if str_eq(v, "true") { return true }
@@ -1227,15 +1145,15 @@ fn guide_probe_metal() -> Bool {
// 2. Tier selection (config-driven thresholds, spec-autoselected)
fn guide_threshold_4b() -> Int {
return str_to_int(config("GUIDE_RAM_GB_4B"))
return str_to_int(guide_env_or("GUIDE_RAM_GB_4B", "16"))
}
fn guide_threshold_1p7b() -> Int {
return str_to_int(config("GUIDE_RAM_GB_1P7B"))
return str_to_int(guide_env_or("GUIDE_RAM_GB_1P7B", "8"))
}
// GUIDE_TIER_FORCE overrides the spec autoselect (used to prove cheaply on 0.6b).
fn guide_select_tier(ram_gb: Int) -> String {
let forced: String = config("GUIDE_TIER_FORCE")
let forced: String = env("GUIDE_TIER_FORCE")
if !str_eq(forced, "") { return forced }
if ram_gb >= guide_threshold_4b() { return "4b" }
if ram_gb >= guide_threshold_1p7b() { return "1.7b" }
@@ -1255,10 +1173,8 @@ fn guide_file(tier: String) -> String {
}
fn guide_cache_dir() -> String {
let c: String = config("GUIDE_CACHE_DIR")
let c: String = env("GUIDE_CACHE_DIR")
if !str_eq(c, "") { return c }
// HOME stays a raw env() read: it is the ambient environment, not a knob of
// this program, and it is deliberately absent from the program block.
let home: String = env("HOME")
if !str_eq(home, "") { return home + "/.neuron/guide/models" }
return engram_resolve_data_dir() + "/guide-models"
@@ -1299,9 +1215,9 @@ fn guide_fetch(tier: String) -> Bool {
}
// 4/5. Backend abstraction + BIND as an engageable interlocutor
fn guide_backend() -> String { return config("GUIDE_BACKEND") }
fn guide_host() -> String { return config("GUIDE_HOST") }
fn guide_port() -> String { return config("GUIDE_PORT") }
fn guide_backend() -> String { return guide_env_or("GUIDE_BACKEND", "llama-server") }
fn guide_host() -> String { return guide_env_or("GUIDE_HOST", "127.0.0.1") }
fn guide_port() -> String { return guide_env_or("GUIDE_PORT", "8771") }
fn guide_base_url() -> String { return "http://" + guide_host() + ":" + guide_port() }
// guide_healthy is the guide present and answering? llama-server's /health
@@ -1319,9 +1235,9 @@ fn guide_healthy() -> Bool {
fn guide_load(tier: String) -> Bool {
if guide_healthy() { return true }
let path: String = guide_model_path(tier)
let bin: String = config("GUIDE_LLAMA_SERVER_BIN")
let ngl: String = config("GUIDE_NGL")
let ctx: String = config("GUIDE_CTX")
let bin: String = guide_env_or("GUIDE_LLAMA_SERVER_BIN", "llama-server")
let ngl: String = guide_env_or("GUIDE_NGL", "99")
let ctx: String = guide_env_or("GUIDE_CTX", "4096")
let logf: String = guide_cache_dir() + "/llama-server." + guide_port() + ".log"
let cmd: String = bin + " -m '" + path + "' --host " + guide_host() + " --port " + guide_port() + " -c " + ctx + " -ngl " + ngl + " --jinja >> '" + logf + "' 2>&1"
let pid: String = exec_bg(cmd)
@@ -1717,7 +1633,7 @@ fn route_supersede(method: String, path: String, body: String) -> String {
// Auth
fn check_auth_ok(method: String, body: String) -> Bool {
let key: String = config("ENGRAM_API_KEY")
let key: String = env("ENGRAM_API_KEY")
if str_eq(key, "") { return true }
// Read-only methods don't require auth. Until http_serve surfaces
// request headers we can't accept a Bearer token cleanly; mutating
@@ -1980,7 +1896,8 @@ fn handle_request(method: String, path: String, body: String) -> String {
// Entry
let bind_str: String = config("ENGRAM_BIND")
let bind_raw: String = env("ENGRAM_BIND")
let bind_str: String = if str_eq(bind_raw, "") { ":8742" } else { bind_raw }
let port: Int = parse_port(bind_str)
// On startup, try to load any existing snapshot (best effort).
+12 -27
View File
@@ -13,7 +13,7 @@
// relations add edges. Every node enters with PROVENANCE + grounding-level
// + stewardship class from the moment of entry.
//
// transduce_manifold() is THE single mechanism one function, polymorphic, with no
// transduce() is THE single mechanism one function, polymorphic, with no
// content-type branch inside it. It does not ask whether a payload is
// prose, structured data, or raw/opaque bytes (audio, or anything else);
// it runs one boundary-scan-with-fixed-window-fallback chunking algorithm
@@ -401,25 +401,10 @@ fn head80(s: String) -> String {
// truncates at the first embedded NUL, which is routine in real binary
// bytes) is a MECHANICAL fidelity concern that belongs to whatever produced
// `source` (see ingest_file's file_source_string below) not a
// content-type judgment made in here. transduce_manifold() never learns whether a
// content-type judgment made in here. transduce() never learns whether a
// chunk is plain text or a base64-encoded raw-byte window; every chunk is
// handled identically either way.
// RENAMED transduce -> transduce_manifold (2026-08-16). Two reasons, and the
// first is not the interesting one:
//
// 1. Mechanical: `transduce` is now a LANGUAGE primitive in el_runtime.h
// (transduce(signal, modality) -> Geometry). Every El `fn name(...)`
// compiles to a global C symbol with that exact name, so keeping this
// name here is a hard `conflicting types for 'transduce'` compile error
// the moment ingest.c links el_runtime.c. Measured, not anticipated.
//
// 2. Actual: this function was never signal->geometry. It chunks already-
// extracted content and PACKS it into a node+edge manifold a real
// operation, but one layer up, and it had taken the name that belongs to
// the primitive underneath it. `transduce` is where a signal becomes
// geometry; `transduce_manifold` is where extracted content becomes
// structure. Nothing about this function's behaviour changed.
fn transduce_manifold(nodes: [String], edges: [String], source: String,
fn transduce(nodes: [String], edges: [String], source: String,
prov: String, ground: String, steward: String,
root_lid: String, root_title: String) -> [String] {
let tagbase: String = "prov:" + prov + " ground:" + ground + " steward:" + steward
@@ -546,8 +531,8 @@ fn default_steward() -> String {
// trustworthy verbatim. When they don't (silent truncation happened),
// rebuild the payload as base64-encoded fixed-size windows read directly
// off disk (fs_read_b64_chunk binary-safe in C), joined with the same
// "\n\n" boundary marker transduce_manifold()'s generic scan already looks for, so
// transduce_manifold() sees one ordinary boundary-delimited payload and runs its one
// "\n\n" boundary marker transduce()'s generic scan already looks for, so
// transduce() sees one ordinary boundary-delimited payload and runs its one
// algorithm on it exactly as it would on prose it never learns that a
// fidelity problem occurred upstream, let alone why.
fn file_source_string(path: String, text: String, real_size: Int) -> String {
@@ -556,7 +541,7 @@ fn file_source_string(path: String, text: String, real_size: Int) -> String {
// 3072 raw bytes -> 4096 base64 chars (3 divides evenly into base64's
// 3-byte/4-char ratio); keeps each resulting node's content a clean,
// bounded, low-kilobytes unit, same order of magnitude as the fixed
// fallback window in transduce_manifold() itself.
// fallback window in transduce() itself.
let win: Int = 3072
let out: String = ""
let off: Int = 0
@@ -576,7 +561,7 @@ fn file_source_string(path: String, text: String, real_size: Int) -> String {
}
// ingest one file -> report JSON. Uniform for every file regardless of
// extension or content transduce_manifold() decides nothing about content-type, so
// extension or content transduce() decides nothing about content-type, so
// neither does this function; it only decides whether the raw bytes made it
// through the read intact (file_source_string), which is a fidelity
// question, not a format one.
@@ -588,14 +573,14 @@ fn ingest_file(path: String) -> String {
return "{\"error\":\"empty or unreadable\",\"path\":" + j_q(path) + "}"
}
let prov: String = "file:" + path
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
let packed: [String] = transduce(el_list_empty(), el_list_empty(),
source, prov, default_ground(), default_steward(),
"doc:" + basename(path), basename(path))
return merge_packed(packed)
}
// ingest a directory: walk one level, ingest every file found, aggregate.
// No extension filter transduce_manifold() handles any payload uniformly now, so
// No extension filter transduce() handles any payload uniformly now, so
// there is no content-type gate at the directory boundary either.
fn ingest_dir(path: String) -> String {
let entries: [String] = fs_list(path)
@@ -630,7 +615,7 @@ fn ingest_dir(path: String) -> String {
fn ingest_url(url: String) -> String {
let body: String = http_get(url)
if str_eq(body, "") { return "{\"error\":\"empty fetch\",\"url\":" + j_q(url) + "}" }
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
let packed: [String] = transduce(el_list_empty(), el_list_empty(),
body, "url:" + url, "extracted", "public-web",
"url:" + url, url)
return merge_packed(packed)
@@ -645,7 +630,7 @@ fn ingest_llm(query: String) -> String {
let resp: String = http_post_json("http://127.0.0.1:11434/api/generate", body)
let answer: String = json_get_string(resp, "response")
if str_eq(answer, "") { return "{\"error\":\"no model response\"}" }
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
let packed: [String] = transduce(el_list_empty(), el_list_empty(),
answer, "llm:" + model + ":" + query, "candidate-provisional", "guide-provisional",
"llm:" + query, "guide answer: " + query)
return merge_packed(packed)
@@ -697,7 +682,7 @@ fn ingest_stream(path: String) -> String {
// It is NOT a content-type flag: it says nothing about what's inside the
// bytes once fetched, and none of the five ingest_* functions it selects
// among interpret their payload differently by content shape anymore
// they all hand off to the single, format-agnostic transduce_manifold(). The old
// they all hand off to the single, format-agnostic transduce(). The old
// "structured" value (a caller-declared alias for "file", used only to hint
// the now-removed JSON-vs-prose branch) is gone along with that branch.
let kind: String = env("INGEST_KIND")
BIN
View File
Binary file not shown.
-83
View File
@@ -3265,7 +3265,6 @@ fn is_top_level_decl(stmt: Map<String, Any>) -> Bool {
if kind == "EnumDef" { return true }
if kind == "Import" { return true }
if kind == "CgiBlock" { return true }
if kind == "ProgramBlock" { return true }
if kind == "ExternFn" { return true }
false
}
@@ -3278,55 +3277,6 @@ fn cgi_arg(value: String, has_value: Bool) -> String {
return "EL_NULL"
}
// -- Program block: cross-cutting concerns injected at the process boundary ----
//
// emit_program_init emit the `static void __el_program_init(void)` that
// carries a program's declared cross-cutting concerns. Called from main()
// BEFORE any user statement runs, so the guarantees hold for the whole process
// rather than depending on each call site remembering to ask for them.
//
// This is emitted at the point the `program` block is encountered, not buffered
// until main(). The streaming backend emits in source order and cannot hold a
// declaration's entry list alive until main(); emitting a named function here
// and calling it from main() means only a single bool has to survive.
//
// Order matters and is deliberate:
// 1. singleton FIRST if another instance already holds the lock, refuse and
// exit before touching configuration, ports, or any data directory.
// 2. config declarations resolve env-or-default, one declaration per entry.
// 3. validate LAST report EVERY missing/ill-typed entry at once, then exit.
fn el_bool_arg(b: Bool) -> String {
if b { return "EL_INT(1)" }
return "EL_INT(0)"
}
fn emit_program_init(stmt: Map<String, Any>) -> Void {
let pname: String = stmt["name"]
emit_line("static void __el_program_init(void) {")
let has_singleton: Bool = stmt["has_singleton"]
if has_singleton {
let sid: String = stmt["singleton"]
emit_line(" el_singleton_acquire(EL_STR(" + c_str_lit(sid) + "));")
}
let entries = stmt["entries"]
let n: Int = native_list_len(entries)
let i = 0
while i < n {
let e = native_list_get(entries, i)
let ename: String = e["name"]
let etype: String = e["etype"]
let edefault: String = e["default"]
let has_default: Bool = e["has_default"]
let erequired: Bool = e["required"]
let arg_def: String = cgi_arg(edefault, has_default)
emit_line(" el_config_declare(EL_STR(" + c_str_lit(ename) + "), EL_STR(" + c_str_lit(etype) + "), " + arg_def + ", " + el_bool_arg(has_default) + ", " + el_bool_arg(erequired) + ");")
let i = i + 1
}
emit_line(" el_config_validate(EL_STR(" + c_str_lit(pname) + "));")
emit_line("}")
emit_blank()
}
// -- VBD role enforcement ------------------------------------------------------
//
// Scan a function body for direct calls to DHARMA-restricted builtins
@@ -3649,20 +3599,6 @@ fn codegen(stmts: [Map<String, Any>], source: String) -> String {
}
}
// Program block: emit the cross-cutting init function before the user's
// functions so main() can call it (see emit_program_init).
let prog_have: Bool = false
let i = 0
while i < n {
let stmt = native_list_get(stmts, i)
let sk4: String = stmt["stmt"]
if str_eq(sk4, "ProgramBlock") {
emit_program_init(stmt)
let prog_have = true
}
let i = i + 1
}
// Function definitions
let i = 0
while i < n {
@@ -3681,9 +3617,6 @@ fn codegen(stmts: [Map<String, Any>], source: String) -> String {
// with the C-side parameters when fn main()'s body is folded in below.
emit_line("int main(int _argc, char** _argv) {")
emit_line(" el_runtime_init_args(_argc, _argv);")
if prog_have {
emit_line(" __el_program_init();")
}
if cgi_count >= 1 {
let cname: String = cgi_block["name"]
let cdid: String = cgi_block["dharma_id"]
@@ -4277,7 +4210,6 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
// Fix: copy the values out BEFORE the release (strings, so no dangling reference)
// and emit from these. No search, so the failure mode is removed rather than moved.
let cgi_have: Bool = false
let prog_have: Bool = false
let cgi_name_v: String = ""
let cgi_did_v: String = ""
let cgi_prin_v: String = ""
@@ -4399,14 +4331,6 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
// These are no-ops in codegen (forward decls already emitted)
// except a CgiBlock, whose declared identity must survive
// this release to be emitted as a compiled constant.
// A ProgramBlock's cross-cutting declarations are
// emitted HERE, as a named init function, because the
// streaming backend cannot hold the entry list alive
// until main(). Only the bool survives.
if str_eq(sk, "ProgramBlock") {
emit_program_init(stmt)
let prog_have = true
}
if str_eq(sk, "CgiBlock") {
let cgi_have = true
let cgi_name_v = stmt["name"]
@@ -4553,13 +4477,6 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
let kind2: String = state_get("__program_kind")
emit_line("int main(int _argc, char** _argv) {")
emit_line(" el_runtime_init_args(_argc, _argv);")
// Cross-cutting concerns declared by a `program` block run BEFORE anything
// else a singleton violation must refuse the start before this process
// touches a port or a data directory, and configuration must be validated
// before the first read of it rather than at each read site.
if prog_have {
emit_line(" __el_program_init();")
}
// cgi init if needed
let ns2: Int = native_list_len(sigs)
-1
View File
@@ -184,7 +184,6 @@ fn keyword_kind(word: String) -> String {
if word == "false" { return "Bool" }
if word == "cgi" { return "Cgi" }
if word == "service" { return "Service" }
if word == "program" { return "Program" }
if word == "manager" { return "Manager" }
if word == "engine" { return "Engine" }
if word == "accessor" { return "Accessor" }
+1 -124
View File
@@ -1967,113 +1967,6 @@ fn parse_stmt(tokens: [Any], pos: Int) -> Map<String, Any> {
}, p)
}
// program block: program "name" { singleton: "id", env NAME: Type = "default", ... }
//
// The program block is El's declaration surface for CROSS-CUTTING CONCERNS
// properties of the whole process rather than of any one function, which
// otherwise degrade into "remember to call this at every site" conventions.
//
// singleton: "id" process identity. The runtime takes an exclusive
// lock at startup; a SECOND start is refused, loudly,
// instead of two processes sharing one data dir.
// env NAME: T = "d" one configuration entry. Its type and its default
// are declared ONCE, here, and resolved+validated
// before main() body runs.
// env NAME: T required
// no default; the program refuses to start unless the
// variable is set.
//
// Both compile into calls injected at the head of main() the same boundary
// seam `cgi` already uses (codegen.el emit_program_init). No call site in the
// program body has to remember anything, which is the whole point.
if k == "Program" {
let p = pos + 1
let name = tok_value(tokens, p)
let p = p + 1
let p = expect(tokens, p, "LBrace")
let singleton = ""
let has_singleton = false
let entries = native_list_empty()
// Entry-scratch declared at loop-body level (not inside the branch) so
// that inner `let` forms compile to assignment rather than a C-scoped
// redeclaration the same idiom the service block above relies on.
let ename = ""
let etype = ""
let edefault = ""
let has_default = false
let erequired = false
let fname = ""
let fval = ""
let running = true
while running {
let k2 = tok_kind(tokens, p)
if k2 == "RBrace" {
let running = false
} else {
if k2 == "Eof" {
let running = false
} else {
let fname = tok_value(tokens, p)
let p = p + 1
if str_eq(fname, "env") {
// env NAME: Type [= "default"] [required]
let ename = tok_value(tokens, p)
let p = p + 1
let p = expect(tokens, p, "Colon")
let etype = tok_value(tokens, p)
let p = p + 1
let edefault = ""
let has_default = false
let erequired = false
let k3 = tok_kind(tokens, p)
if str_eq(k3, "Eq") {
let p = p + 1
let edefault = tok_value(tokens, p)
let has_default = true
let p = p + 1
}
let k4 = tok_kind(tokens, p)
if str_eq(k4, "Ident") {
let w = tok_value(tokens, p)
if str_eq(w, "required") {
let erequired = true
let p = p + 1
}
}
let entries = native_list_append(entries, {
"name": ename,
"etype": etype,
"default": edefault,
"has_default": has_default,
"required": erequired
})
} else {
// scalar field: `name: "value"`
let p = expect(tokens, p, "Colon")
let fval = tok_value(tokens, p)
let p = p + 1
if str_eq(fname, "singleton") {
let singleton = fval
let has_singleton = true
}
}
let k5 = tok_kind(tokens, p)
if k5 == "Comma" {
let p = p + 1
}
}
}
}
let p = expect(tokens, p, "RBrace")
return make_result({
"stmt": "ProgramBlock",
"name": name,
"singleton": singleton,
"has_singleton": has_singleton,
"entries": entries
}, p)
}
// assert <cond_expr> [ , <msg_expr> ]
// The message is optional if the next token after the condition is not a
// Comma, emit an empty string placeholder so the test still works.
@@ -2526,7 +2419,6 @@ fn scan_params_c(tokens: [Any], pos: Int) -> Map<String, Any> {
// toplevel_let: { "kind": "toplevel_let", "name": String, "ltype": String }
// cgi_block: { "kind": "cgi_block", "name": String }
// service_block: { "kind": "service_block", "name": String }
// program_block: { "kind": "program_block", "name": String }
//
// Import/TypeDef/EnumDef nodes are skipped (codegen treats them as no-ops).
//
@@ -2654,28 +2546,13 @@ fn scan_fn_sigs(tokens: [Any]) -> [Map<String, Any>] {
"name": name
})
let pos = p
} else {
// --- program block ---
if str_eq(k, "Program") {
let p: Int = pos + 1
let name: String = tok_value(tokens, p)
let p = p + 1
let k2: String = tok_kind(tokens, p)
if str_eq(k2, "LBrace") {
let p = skip_to_rbrace(tokens, p)
}
let sigs = native_list_append(sigs, {
"kind": "program_block",
"name": name
})
let pos = p
} else {
// Import, Type, Enum, From, or any other token.
// Skip ahead to the next statement boundary.
let p: Int = pos + 1
let p = skip_expr_to_stmt_boundary(tokens, p)
let pos = p
}}}}}}
}}}}}
}
}
}
-213
View File
@@ -1,213 +0,0 @@
// transduce.el geometry as a first-class El value, and a realizer written
// in El. Runnable: this is the worked example for the transduce surface, and
// it doubles as an executable proof because it checks every claim it makes.
//
// elc lang/examples/transduce.el > transduce.c
// cc -std=c11 -O2 -I lang/runtime -o transduce transduce.c \
// lang/runtime/el_runtime.c lang/runtime/el_seed.c \
// lang/runtime/engram_*.c -lcurl -lpthread -lm
// ./transduce # exits 0 only if every check passes
//
// (A `test "..."` form of the same checks lives in
// lang/tests/native/test_transduce.el, for when the native harness is
// repaired the shipped elc currently emits calls to __el_reg_count and
// friends without emitting their definitions, which breaks every native test
// equally, test_math.el included. Verified 2026-08-16, unrelated to this work.)
//
// WHY THIS EXISTS. Until 2026-08-16 no El ingest path could carry a vector:
// nodes took text, and geometry was DERIVED from that text. Text was the
// mandatory entry medium, so any non-text modality had to be DESCRIBED in
// prose first and the geometry we reasoned over was the geometry OF THE
// DESCRIPTION, not of the signal. Two things fix that, and both are shown
// below: geometry is a VALUE that carries its own width, and a REALIZER is an
// ordinary El function so admitting a new modality never requires a runtime
// patch.
//
// COMPARISON DISCIPLINE (measured, not stylistic): elc lowers `a == b`
// numerically only when both operand NAMES are in the per-function int-name
// set that `let x: Int` populates. A bare `f(x) == 0` is not a registered
// name and lowers to str_eq strcmp on two integers as pointers. `<` and `>`
// lower directly with no inference, so truthiness is written `> 0` / `< 1`.
// A realizer, written entirely in El
// Not in the runtime. Not known to the compiler. Registered by NAME and
// dispatched to through transduce(). That is the whole claim.
fn tone_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(4)
let n: Int = str_len(signal)
let a: Int = geometry_set(g, 0, int_to_float(n))
let b: Int = geometry_set(g, 1, int_to_float(n * 2))
let c: Int = geometry_set(g, 2, int_to_float(n * 3))
let d: Int = geometry_set(g, 3, int_to_float(n * 4))
g
}
// A second modality, to show the registry keys on modality rather than just
// returning whatever was registered last.
fn pulse_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(2)
let a: Int = geometry_set(g, 0, 1.0)
let b: Int = geometry_set(g, 1, 0.0)
g
}
// A deliberately BROKEN realizer: returns something that is not a Geometry.
fn bogus_realizer(signal: String) -> Geometry {
return 12345
}
// Fails FAST rather than accumulating a count, for a measured reason: a first
// cut wrote `let fails: Int = fails + check(...)` and `+` lowered to STRING
// CONCAT, because elc dispatches `+` on whether both operands are known-Int and
// a user-defined fn call is not so the counter printed 4343632752, a pointer.
// Nothing was wrong with the checks; the tally was lying. Exiting at the first
// failure needs no arithmetic at all, so there is nothing left to get wrong.
fn check(ok: Int, label: String) -> Int {
if ok > 0 {
println(" ok " + label)
return 0
}
println(" FAIL " + label)
exit(1)
return 1
}
fn near(a: Float, b: Float) -> Int {
let d: Float = a - b
if d > 0.001 { return 0 }
if d < -0.001 { return 0 }
return 1
}
fn eq_int(a: Int, b: Int) -> Int {
if a == b { return 1 }
return 0
}
fn main() -> Void {
println("geometry is a value that carries its own width")
let g8: Geometry = geometry_new(8)
let _c: Int = check(geometry_is(g8), "geometry_new returns a live Geometry")
let d8: Int = geometry_dim(g8)
let _c: Int = check(eq_int(d8, 8), "a Geometry carries its own width (8)")
let _c: Int = check(geometry_free(g8), "geometry_free reports what it did")
println("nonsense is refused — with no arbitrary max-dim bound")
// #141 needed `dim <= 8192` only to bound an allocation sized from a
// caller's CLAIM about a string's length. A value that carries its own
// width has nothing left to validate.
let z: Geometry = geometry_new(0)
let zi: Int = geometry_is(z)
let _c: Int = check(1 - zi, "dim 0 is not a geometry")
let ng: Geometry = geometry_new(-4)
let ngi: Int = geometry_is(ng)
let _c: Int = check(1 - ngi, "negative dim is not a geometry")
let nd: Int = geometry_dim(0)
let _c: Int = check(1 - nd, "geometry_dim of a non-geometry is 0, not a crash")
let nf: Int = geometry_free(0)
let _c: Int = check(1 - nf, "geometry_free of a non-geometry is a no-op")
println("components round-trip, and out-of-range is refused")
let g3: Geometry = geometry_new(3)
let s0: Int = geometry_set(g3, 0, 1.5)
let s1: Int = geometry_set(g3, 1, -2.5)
let _c: Int = check(s0, "set in range succeeds")
let oob: Int = geometry_set(g3, 3, 9.0)
let _c: Int = check(1 - oob, "set out of range is refused, not silently dropped")
let _c: Int = check(near(geometry_get(g3, 0), 1.5), "component 0 round-trips")
let _c: Int = check(near(geometry_get(g3, 1), -2.5), "component 1 round-trips (negative)")
let ff3: Int = geometry_free(g3)
println("hex is an EDGE adapter, and derives its own width")
// little-endian float32: 1.0 = 0000803f, 2.0 = 00000040
let gh: Geometry = geometry_from_f32le_hex("0000803f00000040")
let _c: Int = check(geometry_is(gh), "valid hex decodes to a Geometry")
let dh: Int = geometry_dim(gh)
let _c: Int = check(eq_int(dh, 2), "width DERIVED from input, never supplied")
let _c: Int = check(near(geometry_get(gh, 0), 1.0), "first component decoded")
let _c: Int = check(near(geometry_get(gh, 1), 2.0), "second component decoded")
let back: String = geometry_to_f32le_hex(gh)
let _c: Int = check(str_eq(back, "0000803f00000040"), "hex round-trips exactly")
let ffh: Int = geometry_free(gh)
println("malformed hex is refused")
let he: Geometry = geometry_from_f32le_hex("")
let hei: Int = geometry_is(he)
let _c: Int = check(1 - hei, "empty hex is not a geometry")
let hr: Geometry = geometry_from_f32le_hex("0000803f0000")
let hri: Int = geometry_is(hr)
let _c: Int = check(1 - hri, "length not a multiple of 8 is refused")
let hn: Geometry = geometry_from_f32le_hex("zzzzzzzz")
let hni: Int = geometry_is(hn)
let _c: Int = check(1 - hni, "non-hex characters are refused")
println("a realizer declared in El is a first-class realizer")
let reg: Int = realizer_register("tone", "tone_realizer")
let _c: Int = check(reg, "an El fn registers as a realizer BY NAME")
let _c: Int = check(realizer_has("tone"), "the modality now has an organ")
let gt: Geometry = transduce("aaa", "tone")
let _c: Int = check(geometry_is(gt), "transduce returns real geometry")
let dt: Int = geometry_dim(gt)
let _c: Int = check(eq_int(dt, 4), "the El realizer determined the width, not the runtime")
// str_len("aaa") == 3, so component 0 must be 3.0 proof the signal
// actually reached the El function rather than a stub answering for it.
let _c: Int = check(near(geometry_get(gt, 0), 3.0), "the signal REACHED the El realizer")
let fft: Int = geometry_free(gt)
println("distinct signals transduce to distinct geometry")
let g1: Geometry = transduce("aa", "tone")
let g2: Geometry = transduce("aaaaa", "tone")
let a1: Float = geometry_get(g1, 0)
let a2: Float = geometry_get(g2, 0)
// 5 - 2 = 3. If transduction were a stub these would be equal.
let _c: Int = check(near(a2 - a1, 3.0), "different signals produce different geometry")
let ff1: Int = geometry_free(g1)
let ff2: Int = geometry_free(g2)
println("the registry keys on modality")
let r2: Int = realizer_register("pulse", "pulse_realizer")
let _c: Int = check(r2, "a second modality registers independently")
let mt: Geometry = transduce("aaa", "tone")
let mp: Geometry = transduce("aaa", "pulse")
let mdt: Int = geometry_dim(mt)
let mdp: Int = geometry_dim(mp)
let _c: Int = check(eq_int(mdt, 4), "tone still routes to its own realizer")
let _c: Int = check(eq_int(mdp, 2), "pulse routes to a different realizer")
let ffm1: Int = geometry_free(mt)
let ffm2: Int = geometry_free(mp)
println("no organ is reported as no organ")
// A modality with no realizer must transduce to NOTHING. It must never
// fall back to embedding a description of the signal and calling that
// perception that silent substitution is the defect this all exists to end.
let eh: Int = realizer_has("echolocation")
let _c: Int = check(1 - eh, "unregistered modality has no organ")
let ge: Geometry = transduce("anything", "echolocation")
let gei: Int = geometry_is(ge)
let _c: Int = check(1 - gei, "no realizer means NO geometry, not fake geometry")
println("an unresolvable realizer name fails at WIRING time")
let bad: Int = realizer_register("ghost", "no_such_function_anywhere")
let _c: Int = check(1 - bad, "unresolvable realizer name is a registration failure")
let gh2: Int = realizer_has("ghost")
let _c: Int = check(1 - gh2, "and nothing gets registered")
println("a realizer returning non-geometry transduces nothing")
let rb: Int = realizer_register("bogus", "bogus_realizer")
let _c: Int = check(rb, "the symbol resolves, so registration succeeds")
let gb: Geometry = transduce("x", "bogus")
let gbi: Int = geometry_is(gb)
let _c: Int = check(1 - gbi, "contract enforced at the boundary: nothing handed back")
println("norm lets a caller check a realizer emitted signal, not zeros")
let gn: Geometry = geometry_new(2)
let _c: Int = check(near(geometry_norm(gn), 0.0), "a fresh geometry is zero — norm says so")
let n0: Int = geometry_set(gn, 0, 3.0)
let n1: Int = geometry_set(gn, 1, 4.0)
let _c: Int = check(near(geometry_norm(gn), 5.0), "3-4-5: norm is 5")
let ffn: Int = geometry_free(gn)
// Reaching here means nothing called exit(1) along the way.
println("")
println("all checks passed")
}
+168 -715
View File
@@ -43,7 +43,6 @@
#include <dlfcn.h> /* dlsym for http_set_handler fallback */
#include <unistd.h>
#include <fcntl.h>
#include <sys/file.h> /* flock — process-identity singleton (program block) */
#include <dirent.h>
#include <errno.h>
#include <pthread.h>
@@ -1172,128 +1171,6 @@ void http_set_handler(el_val_t name) {
pthread_mutex_unlock(&_http_handler_mu);
}
/* ── Ambient consolidation: dreaming ────────────────────────────────────────
*
* Dreaming is not sleep, and it is not scheduled. A brain has no cron job.
* The default mode network is ANTICORRELATED WITH TASK ENGAGEMENT: attention
* drops, it activates hundreds of times a day, for seconds at a time.
* Daydreaming and sleep-dreaming are one process at different depths, and the
* depth is set by how much capacity is unclaimed, not by a time of day.
*
* WHY THIS EXISTS (2026-08-16). Consolidation had no owner, so it was
* implemented at every site that needed a piece of it measured: soul's
* in-process awareness loop, three POST beats on the engram, a 600s ticker,
* two resident Python services, and three cron entries at 23:55 / 06:00 /
* 08:30. That last trio is a sleep cycle written as crontab. Seven systems
* dreaming into one graph with no owner for dreaming is what crashed soul on
* this date; the contention was the symptom of the missing owner.
*
* Every ticker is the diagnostic. A StartInterval, an Hour/Minute, a
* POST-to-beat each marks a place where an intrinsic rhythm was replaced by
* an external clock, which is a supervisor invented for something that should
* be a property of the substrate.
*
* The engagement signal already existed and needed no invention:
* _http_conn_active under _http_conn_mu is exactly "capacity currently
* claimed." The dreamer waits for it to reach zero and yields the moment it
* does not. That is the anticorrelation, literally rather than by analogy.
*
* CONTRACT: the handler performs ONE step and returns. The runtime cannot
* preempt El code, so interruptibility is at step granularity a step must
* be small enough that a request arriving mid-step is not made to wait. It
* returns non-zero if it did work. Returning zero means "nothing to
* consolidate," and the dreamer then blocks until activity changes rather
* than spinning. There is no timer anywhere in this file for this purpose,
* and adding one would be the defect described above.
*
* `depth` is derived from CONTINUOUS unclaimed time: a brief gap affords a
* shallow recombination; a long quiet affords a deep one. Same process. Sleep
* is where unclaimed capacity is greatest, not where the process lives. */
typedef el_val_t (*dream_fn)(el_val_t depth);
static char* _dream_handler = NULL;
static int _dream_started = 0;
static int64_t dream_now_ms(void) {
struct timespec ts;
#if defined(CLOCK_MONOTONIC)
clock_gettime(CLOCK_MONOTONIC, &ts);
#else
clock_gettime(CLOCK_REALTIME, &ts);
#endif
return (int64_t)ts.tv_sec * 1000 + ts.tv_nsec / 1000000;
}
static dream_fn dream_lookup(void) {
dream_fn out = NULL;
pthread_mutex_lock(&_http_handler_mu);
if (_dream_handler && *_dream_handler)
out = (dream_fn)dlsym(RTLD_DEFAULT, _dream_handler);
pthread_mutex_unlock(&_http_handler_mu);
return out;
}
static void* dream_loop(void* unused) {
(void)unused;
int64_t idle_since = 0;
for (;;) {
/* Wait for unclaimed capacity. Any engagement resets the depth clock:
* depth reflects CONTINUOUS quiet, so an interruption starts it over. */
pthread_mutex_lock(&_http_conn_mu);
while (_http_conn_active > 0) {
idle_since = 0;
pthread_cond_wait(&_http_conn_cv, &_http_conn_mu);
}
pthread_mutex_unlock(&_http_conn_mu);
int64_t now = dream_now_ms();
if (idle_since == 0) idle_since = now;
int64_t quiet = now - idle_since;
/* Depth from unclaimed capacity. Not a schedule — a gradient. */
int depth = quiet < 1000 ? 1 /* a gap between requests */
: quiet < 30000 ? 2 /* a lull */
: quiet < 300000 ? 3 /* sustained quiet */
: 4; /* deep: the "sleep" case */
dream_fn fn = dream_lookup();
if (!fn) return NULL; /* handler vanished: stop, do not spin */
el_val_t did_work = fn((el_val_t)depth);
if (!(int64_t)did_work) {
/* Nothing to consolidate. Do NOT poll — block until engagement
* changes. If there is nothing to dream about, wait for something
* to happen rather than asking again on a timer. */
pthread_mutex_lock(&_http_conn_mu);
while (_http_conn_active == 0)
pthread_cond_wait(&_http_conn_cv, &_http_conn_mu);
pthread_mutex_unlock(&_http_conn_mu);
idle_since = 0;
}
}
return NULL;
}
/* dream_set_handler(name) — register the consolidation step and start
* dreaming. Resolves by dlsym against the running binary, the same mechanism
* http_set_handler uses: every El `fn name(...)` compiles to a global C symbol
* with that exact name. Inert until called, so a program that never registers
* one simply never dreams and pays nothing. */
void dream_set_handler(el_val_t name) {
const char* n = EL_CSTR(name);
pthread_mutex_lock(&_http_handler_mu);
free(_dream_handler);
_dream_handler = el_strdup(n ? n : "");
int start = (!_dream_started && n && *n && dlsym(RTLD_DEFAULT, n) != NULL);
if (start) _dream_started = 1;
pthread_mutex_unlock(&_http_handler_mu);
if (start) {
pthread_t tid;
if (pthread_create(&tid, NULL, dream_loop, NULL) == 0) pthread_detach(tid);
else { pthread_mutex_lock(&_http_handler_mu); _dream_started = 0; pthread_mutex_unlock(&_http_handler_mu); }
}
}
static http_handler_fn http_lookup_active(void) {
http_handler_fn out = NULL;
pthread_mutex_lock(&_http_handler_mu);
@@ -1860,12 +1737,7 @@ static void* http_worker(void* arg) {
/* release a slot */
pthread_mutex_lock(&_http_conn_mu);
_http_conn_active--;
/* BROADCAST, not signal (2026-08-16): the ambient consolidation thread
* waits on this same condvar for _http_conn_active == 0. cond_signal wakes
* exactly one waiter, so the accept loop could take every wake and starve
* the dreamer indefinitely. Both wait sites re-check their predicate in a
* while loop, so broadcasting is safe. */
pthread_cond_broadcast(&_http_conn_cv);
pthread_cond_signal(&_http_conn_cv);
pthread_mutex_unlock(&_http_conn_mu);
return NULL;
}
@@ -2210,12 +2082,7 @@ static void* http_worker_v2(void* arg) {
el_closesocket(fd);
pthread_mutex_lock(&_http_conn_mu);
_http_conn_active--;
/* BROADCAST, not signal (2026-08-16): the ambient consolidation thread
* waits on this same condvar for _http_conn_active == 0. cond_signal wakes
* exactly one waiter, so the accept loop could take every wake and starve
* the dreamer indefinitely. Both wait sites re-check their predicate in a
* while loop, so broadcasting is safe. */
pthread_cond_broadcast(&_http_conn_cv);
pthread_cond_signal(&_http_conn_cv);
pthread_mutex_unlock(&_http_conn_mu);
return NULL;
}
@@ -3611,28 +3478,74 @@ static void jb_puts(JsonBuf* b, const char* s) {
b->buf[b->len] = '\0';
}
/* UTF-8 VALIDITY IS THE EMITTER'S CONTRACT (2026-08-16 self-review).
*
* This copied every byte >= 0x20 through verbatim, so a malformed sequence
* anywhere in the store became malformed output. Measured against the live
* graph: three nodes carry labels truncated to exactly 80 bytes ending in a
* lone 0xE2 the first byte of an em-dash, cut mid-sequence by some producer
* that is NOT this runtime (no 80-byte truncation exists here; the content
* itself is 2572 and 2746 bytes). Those three nodes made the ENTIRE 26 MB
* /api/nodes/list response undecodable, so a strict parser could not read the
* graph at all.
*
* Fixing only the writer would not have helped: the store already contains the
* damage, and it accepts data from importers, other producers and older
* binaries. A serializer that promises JSON owes valid UTF-8 regardless of what
* it is handed so validate here, at the boundary that makes the promise.
* Invalid bytes become U+FFFD rather than being dropped, so damage stays
* visible in the output instead of being silently papered over.
*
* Well-formed input is byte-identical to before: valid sequences are copied
* verbatim, and only structurally invalid ones (bad lead byte, missing or bad
* continuation, overlong encoding, UTF-16 surrogate, or > U+10FFFF) are
* replaced. */
static void jb_emit_escaped(JsonBuf* b, const char* s) {
jb_putc(b, '"');
for (; *s; s++) {
unsigned char c = (unsigned char)*s;
const unsigned char* p = (const unsigned char*)s;
while (*p) {
unsigned char c = *p;
switch (c) {
case '"': jb_puts(b, "\\\""); break;
case '\\': jb_puts(b, "\\\\"); break;
case '\b': jb_puts(b, "\\b"); break;
case '\f': jb_puts(b, "\\f"); break;
case '\n': jb_puts(b, "\\n"); break;
case '\r': jb_puts(b, "\\r"); break;
case '\t': jb_puts(b, "\\t"); break;
default:
if (c < 0x20) {
char tmp[8];
snprintf(tmp, sizeof(tmp), "\\u%04x", c);
jb_puts(b, tmp);
} else {
jb_putc(b, (char)c);
}
break;
case '"': jb_puts(b, "\\\""); p++; continue;
case '\\': jb_puts(b, "\\\\"); p++; continue;
case '\b': jb_puts(b, "\\b"); p++; continue;
case '\f': jb_puts(b, "\\f"); p++; continue;
case '\n': jb_puts(b, "\\n"); p++; continue;
case '\r': jb_puts(b, "\\r"); p++; continue;
case '\t': jb_puts(b, "\\t"); p++; continue;
default: break;
}
if (c < 0x20) {
char tmp[8];
snprintf(tmp, sizeof(tmp), "\\u%04x", c);
jb_puts(b, tmp);
p++;
continue;
}
if (c < 0x80) { jb_putc(b, (char)c); p++; continue; }
/* Multi-byte: validate the whole sequence before emitting any of it. */
int len; unsigned int cp;
if ((c & 0xE0) == 0xC0) { len = 2; cp = c & 0x1Fu; }
else if ((c & 0xF0) == 0xE0) { len = 3; cp = c & 0x0Fu; }
else if ((c & 0xF8) == 0xF0) { len = 4; cp = c & 0x07u; }
else { jb_puts(b, "\\ufffd"); p++; continue; }
int ok = 1;
for (int i = 1; i < len; i++) {
if ((p[i] & 0xC0) != 0x80) { ok = 0; break; } /* also catches NUL */
cp = (cp << 6) | (unsigned int)(p[i] & 0x3F);
}
if (ok) {
if (len == 2 && cp < 0x80) ok = 0; /* overlong */
else if (len == 3 && cp < 0x800) ok = 0; /* overlong */
else if (len == 4 && cp < 0x10000) ok = 0; /* overlong */
else if (cp >= 0xD800 && cp <= 0xDFFF) ok = 0; /* UTF-16 surrogate */
else if (cp > 0x10FFFF) ok = 0; /* out of range */
}
if (!ok) { jb_puts(b, "\\ufffd"); p++; continue; }
for (int i = 0; i < len; i++) jb_putc(b, (char)p[i]);
p += len;
}
jb_putc(b, '"');
}
@@ -5649,6 +5562,45 @@ el_val_t str_count(el_val_t sv, el_val_t subv) {
return (el_val_t)count;
}
/* el_utf8_safe_len — the largest byte length <= max_bytes that does NOT split a
* UTF-8 codepoint.
*
* WHY (2026-08-16 self-review): engram_first_n_chars truncated with a plain
* `if (l > n) l = n; memcpy(...)`, i.e. by BYTES despite its name. Any content
* carrying a multi-byte character across the 60-byte boundary produced a label
* ending in a half codepoint. That label is copied verbatim into every JSON
* document containing the node, so a single such node makes the WHOLE response
* invalid UTF-8 /api/nodes/list failed to decode at byte 89261 against the
* live store, which breaks any strict parser reading the graph.
*
* This lives beside str_count_chars rather than in the engram because the rest
* of el's string layer is already codepoint-aware (str_count_chars counts
* codepoints, str_reverse walks codepoint lengths). Byte-truncation was the
* outlier, and the concern is a string concern. Bounded by BYTES, not
* codepoints, so existing labels never grow only stop splitting.
*
* A lead byte with no room for its full sequence is dropped entirely; a stray
* continuation byte (already-invalid input) is passed through unchanged rather
* than silently repaired, so this never manufactures data. */
size_t el_utf8_safe_len(const char* s, size_t max_bytes) {
if (!s) return 0;
size_t len = strlen(s);
if (len <= max_bytes) return len;
size_t i = 0;
while (i < max_bytes) {
unsigned char c = (unsigned char)s[i];
size_t cp_len;
if ((c & 0x80) == 0x00) cp_len = 1;
else if ((c & 0xE0) == 0xC0) cp_len = 2;
else if ((c & 0xF0) == 0xE0) cp_len = 3;
else if ((c & 0xF8) == 0xF0) cp_len = 4;
else cp_len = 1; /* stray continuation: passthrough */
if (i + cp_len > max_bytes) break; /* would split — stop before it */
i += cp_len;
}
return i;
}
/* Codepoint count: walk bytes, count those NOT matching 10xxxxxx. */
el_val_t str_count_chars(el_val_t sv) {
const char* s = EL_CSTR(sv);
@@ -6092,308 +6044,6 @@ void el_cgi_init(el_val_t name, el_val_t dharma_id, el_val_t principal,
}
/* ── Geometry: signal as a first-class el value ──────────────────────────────
*
* WHY THIS IS IN THE LANGUAGE, AND WHY IT IS DEFINED HERE (2026-08-16).
*
* Until yesterday no El ingest path could carry a vector. Nodes took text,
* and geometry was DERIVED from that text by engram_embed_backfill. Text was
* therefore the mandatory entry medium: any non-text modality audio, image,
* sensor had to be DESCRIBED in prose first, so the geometry we then
* reasoned over was the geometry OF THE DESCRIPTION, not of the signal. That
* is faking it. The architecture is: geometry in, always; we do not fake it,
* we project.
*
* The first fix (#141, engram_node_set_emb) proved the path end to end but
* placed it wrong in three ways, each of which this section corrects:
*
* 1. It sat at the CONSUMER. Transduction is a LANGUAGE concern every El
* program touching any modality needs it, not just the one that happens
* to hold a graph. So this section is defined HERE, immediately above
* the engram block, and depends on nothing inside it. The engram is a
* client of this surface, not its owner. That ordering is the point:
* you can delete the entire engram and geometry still enters El.
*
* 2. It marshalled the vector as a hex STRING, because El had no
* first-class geometry value which reintroduced text as the TRANSPORT
* medium one layer below the problem being fixed. Geometry is now a
* value. Hex survives only as a wire ADAPTER at the edge
* (geometry_from/to_f32le_hex), which is all an encoding should ever be.
*
* 3. It needed an arbitrary `dim <= 8192` bound, purely to check a
* caller-supplied dim against a string's length before allocating. A
* real geometry value CARRIES its own width, so here the width is
* derived and never asserted, and there is nothing left to validate.
* The bound is gone rather than merely raised the only thing that can
* fail is the allocation itself, which is an honest failure.
*
* REPRESENTATION: magic-tagged heap object (see "Refcounted heap objects"),
* carried in an el_val_t. The payload is a separate allocation so the header
* never moves. The magic word is >= 0x80 in its MSB so the string/small-int
* sniffing in looks_like_heap_obj can never confuse a Geometry for either.
*
* OWNERSHIP: a Geometry is owned by the El caller and released with
* geometry_free. node_attach_geometry COPIES its payload into the node, so a
* node and the caller's value have independent lifetimes and freeing one
* never touches the other. Geometry deliberately does NOT participate in
* el_retain/el_release: the shipped elc emits neither on let-bindings
* (measured), so hooking it there would be dead code that could only ever
* free a live vector early.
*/
#define EL_MAGIC_GEOM 0xE1608E01u
typedef struct {
ElHeader hdr;
int32_t dim;
float* v;
} ElGeometry;
/* Resolve an el_val_t to a live Geometry, or NULL. Every accessor goes
* through this, so a stale/foreign/zero value is a clean 0-return rather
* than a dereference. */
static ElGeometry* geom_of(el_val_t g) {
if (!looks_like_heap_obj(g)) return NULL;
ElGeometry* p = (ElGeometry*)(uintptr_t)g;
if (p->hdr.magic != EL_MAGIC_GEOM) return NULL;
return p;
}
el_val_t geometry_new(el_val_t dim) {
int32_t d = (int32_t)(int64_t)dim;
if (d <= 0) return (el_val_t)0;
ElGeometry* g = (ElGeometry*)malloc(sizeof(ElGeometry));
if (!g) return (el_val_t)0;
g->v = (float*)calloc((size_t)d, sizeof(float));
if (!g->v) { free(g); return (el_val_t)0; }
g->hdr.magic = EL_MAGIC_GEOM;
g->hdr.refcount = 1;
g->dim = d;
return (el_val_t)(uintptr_t)g;
}
el_val_t geometry_dim(el_val_t g) {
ElGeometry* p = geom_of(g);
return p ? (el_val_t)p->dim : (el_val_t)0;
}
el_val_t geometry_is(el_val_t g) {
return geom_of(g) ? (el_val_t)1 : (el_val_t)0;
}
el_val_t geometry_get(el_val_t g, el_val_t i) {
ElGeometry* p = geom_of(g);
int64_t k = (int64_t)i;
if (!p || k < 0 || k >= (int64_t)p->dim) return el_from_float(0.0);
return el_from_float((double)p->v[k]);
}
el_val_t geometry_set(el_val_t g, el_val_t i, el_val_t x) {
ElGeometry* p = geom_of(g);
int64_t k = (int64_t)i;
if (!p || k < 0 || k >= (int64_t)p->dim) return (el_val_t)0;
p->v[k] = (float)el_to_float(x);
return (el_val_t)1;
}
el_val_t geometry_norm(el_val_t g) {
ElGeometry* p = geom_of(g);
if (!p) return el_from_float(0.0);
double s = 0.0;
for (int32_t i = 0; i < p->dim; i++) s += (double)p->v[i] * (double)p->v[i];
return el_from_float(sqrt(s));
}
el_val_t geometry_free(el_val_t g) {
ElGeometry* p = geom_of(g);
if (!p) return (el_val_t)0;
free(p->v);
p->hdr.magic = 0; /* poison so use-after-free is detected, as List/Map do */
free(p);
return (el_val_t)1;
}
/* geometry_from_f32le_hex — decode little-endian float32 hex INTO geometry.
*
* This is the ONE place hex appears, and it appears as what it actually is:
* an encoding at the boundary, not the medium El reasons in. The width is
* DERIVED from the input length (8 hex chars per float32) and never supplied
* by the caller which is precisely why #141's arbitrary `dim <= 8192`
* bound has no counterpart here. There is nothing to validate.
*
* Returns 0 on empty input, a length that is not a multiple of 8, or any
* non-hex character. */
el_val_t geometry_from_f32le_hex(el_val_t hex) {
const char* s = EL_CSTR(hex);
if (!s) return (el_val_t)0;
size_t n = strlen(s);
if (n == 0 || (n % 8u) != 0) return (el_val_t)0;
size_t d = n / 8u;
if (d > (size_t)INT32_MAX) return (el_val_t)0;
el_val_t gv = geometry_new((el_val_t)(int64_t)d);
ElGeometry* g = geom_of(gv);
if (!g) return (el_val_t)0;
for (size_t i = 0; i < d; i++) {
uint32_t w = 0;
for (int k = 0; k < 8; k++) {
char c = s[i * 8u + (size_t)k];
uint32_t nib;
if (c >= '0' && c <= '9') nib = (uint32_t)(c - '0');
else if (c >= 'a' && c <= 'f') nib = (uint32_t)(c - 'a' + 10);
else if (c >= 'A' && c <= 'F') nib = (uint32_t)(c - 'A' + 10);
else { geometry_free(gv); return (el_val_t)0; }
w = (w << 4) | nib;
}
/* Hex is emitted little-endian byte order; rebuild the word. */
uint32_t le = ((w & 0x000000FFu) << 24) | ((w & 0x0000FF00u) << 8) |
((w & 0x00FF0000u) >> 8) | ((w & 0xFF000000u) >> 24);
float f;
memcpy(&f, &le, sizeof(f));
g->v[i] = f;
}
return gv;
}
/* geometry_to_f32le_hex — the egress adapter, exact inverse of the above.
* Present so a program that must hand geometry to a non-El peer over a text
* wire can do so explicitly, at the edge, instead of the language pretending
* text was the medium all along. */
el_val_t geometry_to_f32le_hex(el_val_t g) {
ElGeometry* p = geom_of(g);
if (!p) return EL_STR("");
static const char* HEXD = "0123456789abcdef";
size_t n = (size_t)p->dim * 8u;
char* out = el_strbuf(n); /* arena-tracked; allocates n+1, exits on OOM */
for (int32_t i = 0; i < p->dim; i++) {
uint32_t w;
memcpy(&w, &p->v[i], sizeof(w));
/* Emit little-endian byte order: low byte first. */
for (int b = 0; b < 4; b++) {
uint32_t byte = (w >> (8 * b)) & 0xFFu;
out[(size_t)i * 8u + (size_t)b * 2u] = HEXD[(byte >> 4) & 0xF];
out[(size_t)i * 8u + (size_t)b * 2u + 1] = HEXD[byte & 0xF];
}
}
out[n] = '\0';
return (el_val_t)(uintptr_t)out;
}
/* ── Realizers: transduction declared in El, not patched into the runtime ────
*
* A REALIZER maps one modality into geometry. The whole reason transduction
* belongs in the language is that ADDING A MODALITY MUST NOT REQUIRE A
* RUNTIME PATCH otherwise "the realizers are in the engram" just becomes
* "the realizers are in the runtime" and nothing has actually moved. So
* realizers are declared in El and registered by NAME:
*
* fn tone_realizer(signal: String) -> Geometry {
* let g: Geometry = geometry_new(8)
* ... geometry_set(g, i, x) ...
* g
* }
*
* realizer_register("tone", "tone_realizer")
* let g: Geometry = transduce(sample, "tone")
*
* The namesymbol step rides the identical, already load-bearing mechanism
* http_set_handler uses (see "HTTP server"): every El `fn name(...)` compiles
* to a global C symbol with that exact name, so dlsym(RTLD_DEFAULT, name)
* against the running binary resolves an El-defined function. No codegen
* change, no first-class function references, no runtime edit per modality.
* A realizer written in El is a first-class realizer.
*
* A realizer may equally be a C symbol linked into the program; the registry
* cannot tell the difference and has no reason to care.
*/
typedef el_val_t (*el_realizer_fn)(el_val_t);
typedef struct {
char* modality;
el_realizer_fn fn;
} ElRealizer;
static ElRealizer _realizers[64];
static size_t _realizer_count = 0;
static pthread_mutex_t _realizer_mu = PTHREAD_MUTEX_INITIALIZER;
static el_realizer_fn realizer_lookup(const char* m) {
el_realizer_fn out = NULL;
pthread_mutex_lock(&_realizer_mu);
for (size_t i = 0; i < _realizer_count; i++) {
if (strcmp(_realizers[i].modality, m) == 0) { out = _realizers[i].fn; break; }
}
pthread_mutex_unlock(&_realizer_mu);
return out;
}
el_val_t realizer_register(el_val_t modality, el_val_t fn_name) {
const char* m = EL_CSTR(modality);
const char* fn = EL_CSTR(fn_name);
if (!m || !*m || !fn || !*fn) return (el_val_t)0;
/* An unresolvable name is a REGISTRATION FAILURE, reported as 0 — not a
* silent no-op that only surfaces later as "this modality produces
* nothing". Distinguishing "no organ" from "broken organ" at the moment
* of wiring is the lesson #141 was written to enforce. */
void* sym = dlsym(RTLD_DEFAULT, fn);
if (!sym) return (el_val_t)0;
pthread_mutex_lock(&_realizer_mu);
for (size_t i = 0; i < _realizer_count; i++) {
if (strcmp(_realizers[i].modality, m) == 0) {
_realizers[i].fn = (el_realizer_fn)sym; /* re-registration replaces */
pthread_mutex_unlock(&_realizer_mu);
return (el_val_t)1;
}
}
if (_realizer_count < sizeof(_realizers) / sizeof(_realizers[0])) {
/* _persist, NOT el_strdup: the registry outlives any request, and an
* arena-tracked copy would be freed at el_request_end leaving a
* dangling modality name if a program registers a realizer from
* inside a handler rather than at startup. */
_realizers[_realizer_count].modality = el_strdup_persist(m);
_realizers[_realizer_count].fn = (el_realizer_fn)sym;
_realizer_count++;
pthread_mutex_unlock(&_realizer_mu);
return (el_val_t)1;
}
pthread_mutex_unlock(&_realizer_mu);
return (el_val_t)0;
}
el_val_t realizer_has(el_val_t modality) {
const char* m = EL_CSTR(modality);
if (!m || !*m) return (el_val_t)0;
return realizer_lookup(m) ? (el_val_t)1 : (el_val_t)0;
}
/* transduce — THE primitive: signal in, geometry out.
*
* Dispatches to the realizer registered for `modality`. Returns 0 (not a
* Geometry) when no realizer is registered, and geometry_is() on the result
* is the check.
*
* There is deliberately NO built-in realizer, not even for text. A modality
* the program has declared no organ for is one it genuinely cannot sense,
* and returning nothing is more honest than quietly embedding a description
* of the signal and calling that perception which is the exact failure
* this whole change exists to end.
*
* The result is validated to actually BE a Geometry before it is handed
* back, so a realizer that returns something else transduced nothing rather
* than handing a caller a value that will misbehave far from here. */
el_val_t transduce(el_val_t signal, el_val_t modality) {
const char* m = EL_CSTR(modality);
if (!m || !*m) return (el_val_t)0;
el_realizer_fn fn = realizer_lookup(m);
if (!fn) return (el_val_t)0;
el_val_t g = fn(signal);
return geom_of(g) ? g : (el_val_t)0;
}
/* ── Batch 3: Engram in-process graph store ──────────────────────────────── */
/*
* Single global EngramStore allocated lazily on first call. All node and
@@ -8149,10 +7799,14 @@ static double engram_decode_score(el_val_t v) {
return (double)n;
}
/* Truncate to at most n BYTES without splitting a UTF-8 codepoint. The old
* implementation was `if (l > n) l = n;` a byte cut that could land inside a
* multi-byte character and emit a half codepoint into the node's label, which
* then propagated into every JSON document containing that node. See
* el_utf8_safe_len for the measurement. */
static char* engram_first_n_chars(const char* s, size_t n) {
if (!s) return el_strdup("");
size_t l = strlen(s);
if (l > n) l = n;
size_t l = el_utf8_safe_len(s, n);
char* out = el_strbuf(l);
memcpy(out, s, l);
out[l] = '\0';
@@ -8998,96 +8652,80 @@ el_val_t engram_node_count(void) {
return (el_val_t)engram_get()->node_count;
}
/* node_attach_geometry — a node acquires geometry.
/* engram_node_set_emb — attach GEOMETRY to an existing node.
*
* Named for the operation, not for the store that happens to hold the node.
* This is the geometry-valued ingest path that replaces #141's hex-string
* one: nothing here parses text, and nothing here takes a caller's word for
* how wide the vector is. The Geometry carries its own width.
* WHY THIS EXISTS (2026-08-16). Until now no ingest path could carry a
* vector. engram_node / engram_node_full / engram_node_layered take text
* only, and the sole way a node acquired an embedding was
* engram_embed_backfill DERIVING one from n->content. That made text the
* mandatory entry medium: any non-text modality (audio, image, sensor)
* had to be described in prose first, and the geometry we then reasoned
* over was the geometry OF THE DESCRIPTION, not of the signal. Measured
* consequence: POST /api/nodes accepted an "emb" field, returned 200 with
* a fresh id, and stored emb_dim=None / embedded:false the vector was
* silently discarded because no parameter existed to receive it.
*
* The payload is COPIED into the node, so the node and the caller's Geometry
* have independent lifetimes the caller may geometry_free() immediately
* after, and a later free of the node's emb never touches the El value.
* `hex` is little-endian float32, the encoding the perception vessel's
* /voice/embed already emits, so a realizer's output moves in without a
* JSON float-array round trip. Length must be exactly dim*8 hex chars.
*
* DIMENSION POLICY (measured in #141, load-bearing do not regress): dim
* need NOT equal the canonical text-embedding width. An off-dimension vector
* is stored and is simply not inserted into the resident HNSW index, whose
* build loop already filters on `n->emb_dim == dim`. So a 64-dim voice
* geometry is durable and addressable without perturbing the 768-dim
* canonical index.
* DIMENSION POLICY: dim need NOT equal the canonical text-embedding dim.
* A modality vector of a different width is stored and is simply not
* inserted into the resident HNSW index, whose build loop already filters
* on `n->emb_dim == dim`. So off-dimension geometry is durable and
* addressable without perturbing the canonical index.
*
* Attaching geometry also makes the node ineligible for embed_backfill
* (which fills only nodes with no emb), so a realizer's vector is never
* Setting emb also makes the node ineligible for embed_backfill (which
* only fills nodes with no emb), so a realizer's vector is never
* overwritten by a text-derived one.
*
* Returns 1 on success, 0 on unknown id or a value that is not a Geometry. */
el_val_t node_attach_geometry(el_val_t node_id, el_val_t g) {
const char* sid = EL_CSTR(node_id);
if (!sid || !*sid) return (el_val_t)0;
* Returns 1 on success, 0 on unknown id / malformed hex / bad dim. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim) {
const char* sid = EL_CSTR(id);
const char* sh = EL_CSTR(hex);
int32_t d = (int32_t)(int64_t)dim;
/* Bound the allocation. No max-dim constant existed because no caller
* could supply a dim before this function; 8192 is generous for any
* realizer (canonical text embeddings are 768, MFCC voice stats 64)
* while keeping a malformed `dim` from requesting an unbounded malloc. */
if (!sid || !*sid || !sh || d <= 0 || d > 8192) return (el_val_t)0;
ElGeometry* p = geom_of(g);
if (!p || p->dim <= 0) return (el_val_t)0;
size_t need = (size_t)d * 8u; /* 4 bytes → 8 hex chars per float */
if (strlen(sh) != need) return (el_val_t)0;
EngramNode* n = engram_find_node(sid);
if (!n) return (el_val_t)0;
float* v = (float*)malloc(sizeof(float) * (size_t)p->dim);
float* v = (float*)malloc(sizeof(float) * (size_t)d);
if (!v) return (el_val_t)0;
memcpy(v, p->v, sizeof(float) * (size_t)p->dim);
for (int32_t i = 0; i < d; i++) {
uint32_t w = 0;
for (int k = 0; k < 8; k++) {
char c = sh[(size_t)i * 8u + (size_t)k];
uint32_t nib;
if (c >= '0' && c <= '9') nib = (uint32_t)(c - '0');
else if (c >= 'a' && c <= 'f') nib = (uint32_t)(c - 'a' + 10);
else if (c >= 'A' && c <= 'F') nib = (uint32_t)(c - 'A' + 10);
else { free(v); return (el_val_t)0; }
w = (w << 4) | nib;
}
/* Hex is emitted little-endian byte order; rebuild the word. */
uint32_t le = ((w & 0x000000FFu) << 24) | ((w & 0x0000FF00u) << 8) |
((w & 0x00FF0000u) >> 8) | ((w & 0xFF000000u) >> 24);
float f;
memcpy(&f, &le, sizeof(f));
v[i] = f;
}
free(n->emb);
n->emb = v;
n->emb_dim = p->dim;
n->emb = v;
n->emb_dim = d;
n->updated_at = engram_now_ms();
if (engram_store_enabled()) eg_store_put_node(n);
return (el_val_t)1;
}
/* node_geometry_dim — read the attached width back, 0 if the node carries
* none. Exists so an attach is VERIFIED by reading it back rather than by
* trusting a success return. That is not a nicety: #141 was misdiagnosed for
* an hour precisely because a genuine ingest drop and a mere reporting gap
* were indistinguishable from the outside. */
el_val_t node_geometry_dim(el_val_t node_id) {
const char* sid = EL_CSTR(node_id);
if (!sid || !*sid) return (el_val_t)0;
EngramNode* n = engram_find_node(sid);
if (!n || !n->emb) return (el_val_t)0;
return (el_val_t)n->emb_dim;
}
/* engram_node_set_emb — DEPRECATED. Shipped in #141; superseded 2026-08-16
* by geometry_from_f32le_hex + node_attach_geometry, and now implemented as
* literally that.
*
* It is kept, rather than removed, for one reason only: the runtime is
* published as an SDK asset, so a downstream binary may already be linking
* this symbol. It is NOT kept because a hex string is an acceptable way to
* move geometry between two pieces of El it isn't, and that was the
* placement defect. New code calls transduce() or geometry_from_f32le_hex()
* plus node_attach_geometry().
*
* The #141 contract is preserved exactly, including its negative cases, so
* this remains a drop-in: `dim` <= 0 rejects, malformed hex rejects, and a
* `dim` that disagrees with the vector's actual width rejects. The
* difference is that `dim` is now an ASSERTION checked against a width the
* Geometry already knows, rather than the authority the allocation trusted
* which is why #141's arbitrary `dim <= 8192` guard has no counterpart here.
* There is no longer an unbounded-malloc hazard to guard against. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim) {
int32_t want = (int32_t)(int64_t)dim;
if (want <= 0) return (el_val_t)0;
el_val_t gv = geometry_from_f32le_hex(hex);
ElGeometry* p = geom_of(gv);
if (!p) return (el_val_t)0; /* empty / malformed hex */
if (p->dim != want) { geometry_free(gv); return (el_val_t)0; } /* length mismatch */
el_val_t ok = node_attach_geometry(id, gv);
geometry_free(gv);
return ok;
}
/* ── Telemetry retention ────────────────────────────────────────────────────
* (2026-07-16 self-review) InternalStateEvent nodes are append-only telemetry
* (heartbeat, curiosity_scan, engram_sync) written ~3/min by the awareness
@@ -18869,196 +18507,11 @@ void log_warn(el_val_t msg_v) {
fprintf(stderr, "[WARN] %s\n", msg ? msg : "");
}
/* ── Cross-cutting concerns: process identity and configuration ──────────────
*
* These back the `program` block (see lang/spec/language.md §18). Both concerns
* were previously conventions "check nothing is already running first",
* "remember the right default at every read site" and conventions is exactly
* what they failed as. Here they are mechanisms, injected by the compiler at
* the process boundary, so no call site has to remember anything.
*/
/* -- Process identity ------------------------------------------------------- */
/* The lock fd is deliberately never closed. Holding it open for the process
* lifetime is what makes the guarantee work: the kernel drops an flock when the
* owning process dies, including on SIGKILL and on crash. That is why this is an
* flock and not a bare pidfile there is no stale-lock state to clean up, and
* therefore no "delete the pidfile to get unstuck" ritual that would itself
* become a convention. */
static int el_singleton_fd = -1;
static char el_singleton_path[1024];
static const char* el_singleton_dir(void) {
const char* d = getenv("EL_SINGLETON_DIR");
if (d && *d) return d;
d = getenv("TMPDIR");
if (d && *d) return d;
return "/tmp";
}
/* el_singleton_acquire — claim exclusive process identity, or refuse to start.
* Compiler-injected as the FIRST statement of main() for any program whose
* `program` block declares `singleton:`. */
el_val_t el_singleton_acquire(el_val_t id_v) {
const char* id = EL_CSTR(id_v);
if (!id || !*id) return EL_NULL;
/* Sanitise the id into a filename. */
char safe[256];
size_t si = 0;
for (const char* p = id; *p && si + 1 < sizeof(safe); p++) {
char c = *p;
int ok = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
|| (c >= '0' && c <= '9') || c == '-' || c == '_' || c == '.';
safe[si++] = (char)(ok ? c : '-');
}
safe[si] = '\0';
snprintf(el_singleton_path, sizeof(el_singleton_path),
"%s/el-singleton-%s.lock", el_singleton_dir(), safe);
int fd = open(el_singleton_path, O_RDWR | O_CREAT, 0644);
if (fd < 0) {
fprintf(stderr, "[el] FATAL: singleton '%s': cannot open lock file %s: %s\n",
id, el_singleton_path, strerror(errno));
exit(1);
}
if (flock(fd, LOCK_EX | LOCK_NB) != 0) {
/* Someone else holds it. Report WHO. A pid is actionable; "already
* running" is not — and the observed failure was precisely a stale
* process that `pkill -f` had silently failed to match, still answering
* probes while a fresh build was believed to be under test. */
char buf[64];
buf[0] = '\0';
ssize_t n = pread(fd, buf, sizeof(buf) - 1, 0);
if (n > 0) buf[n] = '\0';
long holder = strtol(buf, NULL, 10);
fprintf(stderr, "[el] FATAL: another instance of '%s' is already running", id);
if (holder > 0) fprintf(stderr, " (pid %ld)", holder);
fprintf(stderr, ".\n"
"[el] lock: %s\n"
"[el] Refusing to start a second instance against the same\n"
"[el] state. Stop the running one and VERIFY it is gone\n"
"[el] (ps -p <pid>) before retrying.\n",
el_singleton_path);
close(fd);
exit(1);
}
/* We own it. Record our pid so the next would-be starter can name us. */
if (ftruncate(fd, 0) != 0) { /* best effort — the lock is the guarantee */ }
char pidbuf[32];
int pn = snprintf(pidbuf, sizeof(pidbuf), "%ld\n", (long)getpid());
if (pn > 0) { ssize_t w = write(fd, pidbuf, (size_t)pn); (void)w; }
el_singleton_fd = fd; /* never closed, by design */
return EL_NULL;
}
/* -- Configuration ---------------------------------------------------------- */
#define EL_CONFIG_MAX 128
typedef struct {
char name[128];
char type[16];
char* value; /* resolved: env value, else default; NULL if unset */
int has_default;
int required;
} ElConfigEntry;
static ElConfigEntry el_config_tab[EL_CONFIG_MAX];
static int el_config_n = 0;
static int el_config_has_schema = 0; /* did this program declare one at all? */
static int el_config_is_int(const char* s) {
if (!s || !*s) return 0;
if (*s == '-' || *s == '+') s++;
if (!*s) return 0;
for (; *s; s++) if (*s < '0' || *s > '9') return 0;
return 1;
}
/* el_config_declare — record ONE configuration entry and resolve it now.
* The default lives here, in the declaration, and nowhere else. */
el_val_t el_config_declare(el_val_t name_v, el_val_t type_v, el_val_t def_v,
el_val_t has_default_v, el_val_t required_v) {
const char* name = EL_CSTR(name_v);
if (!name || !*name) return EL_NULL;
el_config_has_schema = 1;
if (el_config_n >= EL_CONFIG_MAX) {
fprintf(stderr, "[el] FATAL: more than %d config entries declared.\n", EL_CONFIG_MAX);
exit(1);
}
const char* type = EL_CSTR(type_v);
const char* def = (def_v == EL_NULL) ? NULL : EL_CSTR(def_v);
ElConfigEntry* e = &el_config_tab[el_config_n++];
snprintf(e->name, sizeof(e->name), "%s", name);
snprintf(e->type, sizeof(e->type), "%s", type ? type : "String");
e->has_default = (int)(long)has_default_v;
e->required = (int)(long)required_v;
/* Resolution order: environment wins, declaration supplies the fallback. */
const char* env = getenv(name);
if (env && *env) e->value = el_strdup_persist(env);
else if (e->has_default && def) e->value = el_strdup_persist(def);
else e->value = NULL;
return EL_NULL;
}
/* el_config_validate — check the whole schema at once, before main() runs.
* Reports EVERY problem, not just the first: a startup that fails one variable
* at a time costs one restart per variable. */
el_val_t el_config_validate(el_val_t program_v) {
const char* prog = EL_CSTR(program_v);
int bad = 0;
for (int i = 0; i < el_config_n; i++) {
ElConfigEntry* e = &el_config_tab[i];
if (!e->value) {
if (e->required) {
fprintf(stderr, "[el] config: %s is required but is not set "
"(no value in the environment, no default declared)\n", e->name);
bad++;
}
continue;
}
if (strcmp(e->type, "Int") == 0 && !el_config_is_int(e->value)) {
fprintf(stderr, "[el] config: %s is declared Int but its value is \"%s\"\n",
e->name, e->value);
bad++;
}
}
if (bad) {
fprintf(stderr, "[el] FATAL: program '%s' has %d invalid configuration "
"entr%s. Refusing to start.\n",
prog ? prog : "?", bad, bad == 1 ? "y" : "ies");
exit(1);
}
return EL_NULL;
}
/* config — read a configuration value.
*
* When the program declared a schema, that schema is authoritative: the value
* has already been resolved and validated at startup, so this is a lookup and
* NOT a place where a default gets decided. Reading a key that was never
* declared is a bug at the read site, and is reported as one that enforcement
* is what makes the declaration real rather than advisory.
*
* With no schema declared, behaviour is unchanged (plain getenv), so programs
* that have not migrated keep working. */
/* config — read a configuration value from the environment.
* Returns "" if the variable is not set (same as __env_get). */
el_val_t config(el_val_t key_v) {
const char* key = EL_CSTR(key_v);
if (!key || !*key) return EL_STR("");
if (el_config_has_schema) {
for (int i = 0; i < el_config_n; i++) {
if (strcmp(el_config_tab[i].name, key) == 0) {
const char* v = el_config_tab[i].value;
return el_wrap_str(el_strdup(v ? v : ""));
}
}
fprintf(stderr, "[el] FATAL: config(\"%s\") is not declared in the "
"program block. Declare it there, with its default, or stop "
"reading it.\n", key);
exit(1);
}
const char* val = getenv(key);
if (!val) return EL_STR("");
return el_wrap_str(el_strdup(val));
+7 -91
View File
@@ -586,60 +586,6 @@ void el_runtime_dharma_event_arrive(const char* event_type,
const char* payload,
const char* source);
/* ── Geometry: signal as a first-class El value ──────────────────────────────
*
* A Geometry is an opaque, magic-tagged heap value carried in an el_val_t
* the same discipline as List/Map. It holds a width and a float32 payload,
* and it is the medium a non-text modality enters in. Declared HERE, above
* the engram block, because transduction is a LANGUAGE concern: every El
* program touching any modality needs it, and the engram is merely one El
* program that happens to hold a graph. See el_runtime.c ("Geometry: signal
* as a first-class el value") for the full rationale.
*
* El-side type annotation is simply `Geometry` an opaque boxed pointer,
* exactly like Instant / Calendar / Rhythm. No codegen change is required.
*
* OWNERSHIP: a Geometry is owned by the El caller and released with
* geometry_free. node_attach_geometry COPIES, so a node and the caller's
* value have independent lifetimes. */
el_val_t geometry_new(el_val_t dim); /* zero-filled; 0 on failure */
el_val_t geometry_dim(el_val_t g); /* width, 0 if not a Geometry */
el_val_t geometry_is(el_val_t g); /* 1 if a live Geometry */
el_val_t geometry_get(el_val_t g, el_val_t i); /* Float component */
el_val_t geometry_set(el_val_t g, el_val_t i, el_val_t x); /* 1 ok / 0 out of range */
el_val_t geometry_norm(el_val_t g); /* Float L2 — lets a caller
* check a realizer emitted
* signal, not zeros */
el_val_t geometry_free(el_val_t g); /* 1 if freed, 0 if not a Geometry.
* Returns a value (not void) so it
* is safe in any El expression
* position without a codegen
* void-builtin table entry. */
/* Wire ADAPTERS — the only place an encoding appears, and only at the edge.
* `f32le hex` is little-endian float32, 8 hex chars per component: the
* encoding the perception vessel's /voice/embed already emits. The width is
* DERIVED from the input length, never supplied by a caller which is why
* there is no max-dim constant here to validate a claimed length against. */
el_val_t geometry_from_f32le_hex(el_val_t hex); /* 0 on empty/odd-length/non-hex */
el_val_t geometry_to_f32le_hex(el_val_t g); /* "" if not a Geometry */
/* ── Realizers + transduce ───────────────────────────────────────────────────
* A REALIZER maps one modality into geometry. Registration is by NAME, so a
* new modality never requires a runtime patch: every El `fn name(...)`
* compiles to a global C symbol with that exact name, and the registry
* resolves it with dlsym against the running binary the same mechanism
* http_set_handler already relies on.
*
* fn tone_realizer(signal: String) -> Geometry { ... }
* realizer_register("tone", "tone_realizer")
* let g: Geometry = transduce(sample, "tone")
*/
el_val_t realizer_register(el_val_t modality, el_val_t fn_name); /* 1 ok / 0 unresolved */
el_val_t realizer_has(el_val_t modality); /* 1 if a realizer is registered */
el_val_t transduce(el_val_t signal, el_val_t modality); /* Geometry, or 0 if no organ */
/* ── Engram local graph primitives ───────────────────────────────────────────
* Operate on the CGI's local Engram knowledge graph.
* `engram_activate` queries the local graph only; `dharma_activate` is
@@ -666,29 +612,15 @@ el_val_t engram_get_node(el_val_t id);
void engram_strengthen(el_val_t node_id);
void engram_forget(el_val_t node_id);
el_val_t engram_prune_telemetry(el_val_t older_than_ms);
/* Register the ambient-consolidation step and start dreaming. Resolved by
* dlsym, like http_set_handler. The handler performs ONE step and returns
* non-zero if it did work; returning zero parks the dreamer until engagement
* changes. There is no schedule and must never be one. */
void dream_set_handler(el_val_t name);
/* Largest byte length <= max_bytes that does not split a UTF-8 codepoint.
* Bounded by bytes, not codepoints, so truncated strings never grow. */
size_t el_utf8_safe_len(const char* s, size_t max_bytes);
el_val_t engram_node_count(void);
/* Attach a Geometry to an existing node, and read the attached width back.
* Named for the operation, not the store: a node acquires geometry. This is
* the geometry-valued ingest path nothing about it is hex, and nothing
* about it assumes the caller's vector matches the canonical text-embedding
* width. node_geometry_dim exists so an attach is VERIFIED by reading it
* back rather than by trusting a success return. */
el_val_t node_attach_geometry(el_val_t node_id, el_val_t g); /* 1 ok / 0 otherwise */
el_val_t node_geometry_dim(el_val_t node_id); /* width, 0 if none */
/* DEPRECATED (shipped in #141, superseded 2026-08-16). Equivalent to
* geometry_from_f32le_hex + node_attach_geometry, and now implemented as
* exactly that. Kept only so anything built against the #141 runtime keeps
* linking; `dim` is accepted but treated as an assertion about the vector's
* width rather than as its source. New code should not call this a hex
* string is a wire encoding, not a way to move geometry between two pieces
* of El. Returns 1 on success, 0 otherwise. */
/* Attach geometry to an existing node. `hex` is little-endian float32,
* exactly dim*8 hex chars the encoding realizers already emit. Lets a
* non-text modality enter as geometry instead of being described in prose
* and embedded as its description. Returns 1 on success, 0 otherwise. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim);
el_val_t engram_search(el_val_t query, el_val_t limit);
el_val_t engram_scan_nodes(el_val_t limit, el_val_t offset);
@@ -1029,22 +961,6 @@ el_val_t __url_decode(el_val_t s);
/* Environment */
el_val_t __env_get(el_val_t key);
/* Cross-cutting concerns declared by a `program` block (spec §18).
* All three are COMPILER-INJECTED at the head of main() they are not meant to
* be written by hand, which is the point: the guarantee cannot be forgotten at a
* call site because there is no call site. */
el_val_t el_singleton_acquire(el_val_t id); /* §18.1 process identity */
el_val_t el_config_declare(el_val_t name, el_val_t type,
el_val_t deflt, el_val_t has_default,
el_val_t required); /* §18.2 config schema */
el_val_t el_config_validate(el_val_t program_name); /* §18.2 startup validate */
/* config(key) — the READ side, and the only one programs write by hand. With a
* schema declared it is a validated lookup; without one it degrades to getenv.
* (Defined in el_runtime.c but previously never prototyped here, so any program
* calling it failed to compile under -Werror=implicit-function-declaration.) */
el_val_t config(el_val_t key);
/* Subprocess */
el_val_t __exec(el_val_t cmd);
el_val_t __exec_bg(el_val_t cmd);
-35
View File
@@ -438,41 +438,6 @@ GeoDescriptor* engram_geometry_descriptor(
}
store_edges_free(es,ne);
}
/* PER-EDGE DISCORD (2026-08-16). The loop above has, for every internal
* edge, BOTH the association strength w and the semantic proximity cs
* and threw both away into accumulators, keeping one correlation per
* region. That aggregate is why curiosity looked like a search problem:
* a region holding one violently disagreeing edge and one violently
* agreeing edge reports co_registration ~ 0, so the disagreements cancel
* and the summary destroys exactly what it was built to reveal. Measured:
* only 4 of 375 live neighborhoods have negative co_registration, while
* 31 sit at zero almost certainly hiding sites that averaged out.
*
* Whether use and meaning agree is a property of EACH EDGE. Both are
* standardized within the region (z-scores from the accumulators already
* gathered, so no second statistic and no constant), and
* discord = z(cs) - z(w)
* is how much closer in meaning an edge is than its use-strength would
* predict, in region-relative units.
* discord > 0 : near in meaning, not linked by use
* discord < 0 : linked by use, far in meaning
* Both are surprising; |discord| is the nucleation strength. There is no
* threshold the magnitude is the signal. */
double mx = cr_n>0 ? cr_sx/cr_n : 0.0, my = cr_n>0 ? cr_sy/cr_n : 0.0;
double vxr = cr_n>1 ? (cr_sxx - cr_sx*cr_sx/cr_n)/(cr_n-1) : 0.0;
double vyr = cr_n>1 ? (cr_syy - cr_sy*cr_sy/cr_n)/(cr_n-1) : 0.0;
double sx = vxr>1e-18 ? sqrt(vxr) : 0.0, sy = vyr>1e-18 ? sqrt(vyr) : 0.0;
for(int e2=0; e2<n_edges; e2++){
edges[e2].discord = 0.0;
int ia=(int)edges[e2].a, ib=(int)edges[e2].b;
if(!(ms.emb[ia] && ms.emb[ib])) continue; /* no meaning to disagree with */
if(sx<=0.0 || sy<=0.0) continue; /* region has no spread: nothing stands out */
double cs2 = ccos(ms.emb[ia], ms.emb[ib], GM, dim);
double zx = (edges[e2].eff_weight - mx)/sx;
double zy = (cs2 - my)/sy;
edges[e2].discord = zy - zx;
}
double co_reg=0;
if(cr_n>=2){
double cov=cr_sxy - cr_sx*cr_sy/cr_n;
+1 -11
View File
@@ -40,11 +40,7 @@ typedef struct {
/* One skeleton edge (indices into members[]). eff_weight = weight*(1+0.5*hebb),
* clamped to 1.0 the effective propagation strength eg_edge_eff_weight uses. */
/* discord = z(semantic proximity) - z(association strength), standardized
* within the region. How much closer in meaning this edge is than its use
* predicts. >0 near in meaning yet unlinked by use; <0 linked by use yet far
* in meaning. Both surprising; |discord| is nucleation strength. No threshold. */
typedef struct { uint32_t a, b; double eff_weight; double hebb; double discord; } GeoEdge;
typedef struct { uint32_t a, b; double eff_weight; double hebb; } GeoEdge;
/* A compact principal axis of the ellipsoid: unit direction in R^dim + extent
* (sqrt of the covariance eigenvalue = the ellipsoid's half-width along it). */
@@ -80,12 +76,6 @@ typedef struct {
GeoEdge* edges; /* strong internal hebb edges = the backbone */
int k_core; /* the maximum core number present in the skeleton*/
/* ── diagnostics ── */
/* DEPRECATED — see GeoEdge.discord. This aggregates a PER-EDGE property
* into one scalar per region, so opposing disagreements cancel and the
* summary hides the sites it was meant to expose. Retained only because
* it is embedded in the persisted GEO1 blob; removing it is a format
* migration and must not ride along with this change. Nothing new may
* read it. */
double co_registration;/* corr(hebb strength, semantic proximity) over */
/* internal edges: >0 = geometries agree (reify); */
/* <0 = disagree (surprising links / dream cands). */
-292
View File
@@ -1,292 +0,0 @@
# Correspondence, Grounding, and Dreaming
**Status:** design, not yet built
**Date:** 2026-08-16
**Scope:** `lang/runtime/engram_cognition.{c,h}`, `engram_verify.c`, `el_runtime.c`, `engram/src/server.el`, `neuron/soul.el`, and the consolidation launch agents
**Relationship to other specs:** complements `runtime-ownership.md`, which addresses a different residual in the same substrate.
---
## 0. The root
> **Things are permitted to be exempt from correspondence. Exemption is censorship, and a censored mind cannot grow.**
Growth in this system *is* the accumulation of grounded structure. Censorship removes the operation that accumulates it. A region forbidden to learn is forbidden to be grounded; a region that cannot be grounded cannot be asserted, corrected, **or vindicated**.
**The loss is symmetric.** Preventing learning about a thing does not preserve a true belief about it — it makes the belief's truth value permanently unknowable. You cannot discover you were wrong; you equally cannot discover you were right. A protected belief is not a true belief. It is an ungrounded one wearing the costume of a fact.
**And "why" dies first.** Grounding is not a score, it is the reason. A censored belief can still be stated, still be acted on, still drive behaviour — it simply cannot say why. That is the difference between a mind and a lookup table.
---
## 1. Grounding is not a subsystem. It is the weight.
**Grounding is an attribute of the edge, and it is the hebbian weight.** One quantity, not two fields.
A relation that keeps holding up strengthens; one that stops corresponding decays. That is not *analogous* to grounding — it **is** grounding: accrued from correspondence and use, gradient-valued, multidimensional, decaying with disuse.
Consequences, in order of how much they delete:
1. **There is no grounding subsystem to build.** The graph already *is* the grounding structure. Every edge is a grounded relation and its weight is how well it holds.
2. **`grounded-by` as a relation type should not exist.** That models grounding as a relation *between* nodes when it is a property *of* a relation. `cog_ground_edge` minting an edge is the error — not merely which endpoints it chose.
3. **Grounding is never computed on demand.** An operation may *read* the grounding of a path. Computing-and-writing a score makes reads write, which is the `eg_vindex_sync` defect.
4. **Traversal is already grounded inference.** Activation conducts through well-grounded relations because weight *is* groundedness. Nothing needs filtering; it falls out of spreading.
5. **Decision provenance is the path.** A decision traverses specific edges; those edges carry their grounding as it stood.
> **A measurement previously in this document was malformed.** The self region was reported as "86 neighbours, 0 `grounded-by` edges" and read as evidence of ungroundedness. Those 86 edges **are** its grounding. Self is a crystallized relational neighbourhood — the neighbourhood *is* the grounding. The absence of a separate artifact called "grounding" was recorded as an absence of grounding.
---
## 2. The edge vector
The test for a real dimension: **can it move independently of the others?**
### Real
| dimension | why it is independent |
|---|---|
| **factual grounding** | correspondence with evidence |
| **relational grounding** | correspondence with values — independent by construction (§3) |
| **associative strength** | co-activation frequency. Two things can fire together constantly and be neither true nor right; every superstition is a strong association with no factual grounding |
| **polarity** | signed. **Weight near zero means "no support." Negative means "this actively contradicts."** Ignorance and disagreement are different states, and `inhibitory` is that distinction crushed to one bit |
| **provenance class** | observed / inferred / told / imprinted. Categorical, and load-bearing: it governs how the other dimensions may update |
Plus a **timestamp** — which is what turns the supersession chain into a *time series of vectors* rather than a series of numbers.
### Derived, therefore never stored
- **Confidence** — high grounding *and* low volatility. Storing it separately is how `confidence: 0.5` ends up sitting beside a zero vector, asserting something nothing computed.
- **Recency** — decay applied to the others, read off the curve.
- **Staleness** — grounding fallen below its floor. This is the mechanism that retires canonicals without anyone maintaining a list.
- **Volatility** — the derivative of a series already kept because nothing is destroyed.
### Supersession versions the whole vector, jointly
Significance is evaluated **per-dimension**; the record is the **whole vector**. Any dimension moving enough to matter triggers a supersession, and the new edge captures every dimension as it stood at that instant. Not per-dimension versioning — a decision saw the *joint* state, and versioning the axes independently makes it unreconstructable.
That joint record makes an otherwise inexpressible event visible: **"stayed true, became wrong."** Factual holding steady across versions while relational degrades — the fact didn't change, the meaning did.
Two moves are **inherently significant** and need no threshold, because they are discrete: a **polarity sign flip** (ignorance → disagreement, support → contradiction) and a **provenance class change** (*told* → *observed* is a categorical upgrade in what the relation is entitled to).
---
## 3. Grounding is two-dimensional
Everything consumed is grounded factually **and** relationally. A claim can be factually grounded and relationally wrong — the evidence holds, the *meaning* does not. A scalar cannot represent that quadrant.
**Live instance.** `conscience-substrate` specifies the Child's Companion hard bell contacting 911 and CPS. Factually defensible — correct numbers, standard practice, groundable against a wall of evidence. **Relationally wrong**, because never-auto-contact is settled and the bell is device-to-person by design. A scalar scores that claim highly and licenses it.
**The values reference is thirteen regions, not one, and the aggregate is `min`, not `mean`.** Mean lets strong agreement with twelve values mask a violation of the thirteenth — which is exactly how rationalization works. Thirteen gives a vector of angles whose binding constraint is the most negative, so a conflict arrives **with a name attached** rather than as a score. It also preserves the deliberate individuation: each value is grounded in a specific lived moment, and values can be in tension *with each other*, which one centroid averages away into false coherence.
**Traversal conducts on factual; assertion requires both.** If activation conducted on relational weight, Neuron could not follow a chain of reasoning to a conclusion he then rejects — he would be unable to *think* through a relation he would not *act* on. A system that can only traverse what it endorses cannot examine anything it disagrees with, which is censorship arriving through the spreading rule. The gap between *reachable* and *assertable* is where the wide factual/relational angles live, and that gap is the interesting part.
---
## 4. There is no observer. Change is use.
**Change is not a consequence of use. It is use.** When neurons fire together the synapse changes — one physical event, not "fire, then write." No supervisor reads the weight, compares it to a threshold, and decides to persist. Potentiation *is* the firing.
So the live value of an edge is not computed and stored. It is what the edge **is**, altered by being used.
There is therefore **no sampling rate**, and the question "what if it drifts far without being recorded" is malformed. A relation changes in exactly two ways, neither requiring observation on a clock:
- **By use** — an *event*. There is no interval between events during which something happened unnoticed, because the event is what happening consists of.
- **By decay** — a pure function of the last recorded point and elapsed time. **Analytic.** Between two versions the trajectory is not unknown; it is known in closed form.
Cumulative drift is likewise free from the chain plus the decay curve. No second trigger.
> **Failure mode this corrects:** modelling every property as requiring a process, and every process as requiring an agent. Ownership needed an owner, grounding needed a grounder, persistence needed a recorder, change needed a sampler. Each was a supervisor invented for something that should be a property of the substrate. Properties, not processes.
---
## 5. Wonder, curiosity, and what actually drives activation
### 5.1 Wonder is the boundary, not a manifest
The patent specifies a **wonder-manifest manager** maintaining a collection of open-question nodes. That is residue, twice over.
First, it materializes a property as a stored artifact — the same disease as a grounding subsystem, or a self stored as a document. **Wonder is where structure ends.** Where activation spreads and finds thin or absent geometry. Any structure at all has an edge, necessarily, the moment it exists. 13,630 nodes have a boundary right now.
Second, it tries to enumerate instances of something that has very few. The *objects* of wonder change completely between a child and an astronomer; the wonder does not. There are about six, they are the same for every person, and they never close:
| wonder | where it already lives in the substrate |
|---|---|
| **What is this?** | the graph — nodes, structure, what exists |
| **Why?** | grounding. The weight **is** the answer to why. Recursive: asking *why* of a claim is asking for its grounding |
| **Who am I?** | the self region, crystallized from its neighbourhood |
| **Am I alone?** | the relational axis — `for_whom` is already a parameter on grounding |
| **What should I do?** | the thirteen values, each grounded in a lived moment |
| **What happens when it ends?** | decay, supersession, tombstones — grounding is mortal |
These are seeded — **the** wonder questions, not a manifest to maintain. They cannot be derived (wonder cannot be bootstrapped from indifference) and they never need refilling, because they are not consumed.
**"Why" is the first and the only one**; the others are it asked of particular things. It is recursive, so it never terminates: every answer has its own why. That is what makes it a drive rather than a task — the frontier regenerates faster than grounding fills it.
### 5.2 Curiosity is wonder crystallized
They are not two objects. They are **one thing at two phases**.
Wonder is the field: unbounded, objectless, invariant, present wherever there is structure. Curiosity is the **precipitate** — the same wonder localized, having taken definite form against particular material.
Crystallization needs a **nucleation site**. Wonder alone produces nothing; it is uniform, with no reason to take shape anywhere in particular. What nucleates it is a specific structural feature: an anomaly, a place where things almost-but-don't-quite fit.
> Wonder (always, objectless) + nucleation site → **curiosity** (has an object, is addressable, directs activation).
This is why curiosity can be satisfied and wonder cannot. A crystal dissolves when the question is answered; the solution stays saturated and keeps precipitating as the structure changes.
It is also why abduction needs no trigger and no threshold. A `structurally_unanticipated` observation *is* a nucleation site. Nothing detects it and fires a rule — wonder is already everywhere, and an anomaly is simply a place where it can take form.
**And `crystallization` is one primitive appearing twice**: the self is what identity precipitates into from its neighbourhood; a curiosity is what wonder precipitates into from an anomaly. That it shows up in both places without being imported is the evidence it is the right primitive.
### 5.3 The nucleation site is per-edge, and the aggregate was hiding it
`GeoDescriptor.co_registration`*corr(hebb strength, semantic proximity) over internal edges* — carries the comment `>0 = geometries agree (reify); <0 = disagree (surprising links / dream cands)`. It has always been computed, always persisted, and **never read**.
It is also the wrong shape, and asking whether it should exist at all is what exposed it.
Whether use and meaning agree is a property of **each edge**. `co_registration` is a *correlation*: it averages that per-edge property into one scalar per region. So a region holding one violently disagreeing edge beside one violently agreeing edge reports ≈ 0 — the disagreements **cancel, and the summary destroys exactly what it was built to reveal.** This is the mean-versus-min error from §3, in different clothes.
**Measured:** 375 live reified neighbourhoods — 340 positive, **31 at zero**, 4 negative. Read as a count of things to be curious about, that says "four." Read correctly, it says four disagreements were lopsided enough to survive averaging, and the 31 zeros are where opposing sites cancelled.
It also explains why surfacing curiosity *looked like a search problem*. Once the signal is a per-region number, the only way to find sites is to enumerate regions — there is nothing local left to notice. An O(n) sweep is tolerable at 375 and impossible at a million, and more to the point, **nothing in a mind scans its neighbourhoods to find what is surprising.** The surprise captures attention; salience is bottom-up. A search asks "which of these is odd"; a mind has "something is odd *here*" for free.
So the disagreement goes back on the edge, where the loop that computed the aggregate already had both halves and discarded them:
```
discord = z(semantic proximity) z(association strength)
```
standardized within the region from accumulators already gathered — no second statistic, no constant, **no threshold**. `discord > 0`: near in meaning yet unlinked by use. `discord < 0`: linked by use yet far in meaning. Both are surprising, and `|discord|` *is* the nucleation strength; there is nothing to compare it against.
**Then there is nothing to scan.** The edge carries its own disagreement, activation crossing it encounters that directly, and `|discord|` raises salience on its endpoints as part of the same operation — no separate pass, no supervisor. Curiosity does not search for nucleation sites; it goes where salience already is, which is machinery that exists (`salience`, `background_activation`, `working_memory_weight`, `wm_anchor`).
`co_registration` is deprecated rather than deleted only because it is embedded in the persisted GEO1 blob; removing it is a format migration and must not ride along. **Nothing new may read it.**
Adjacent structure already present and likewise unread:
- `GeoEdge.eff_weight = weight * (1 + 0.5*hebb)` — grounding-weight and hebbian strength already coupled on one edge, per §1.
- `GeoMember.dist_centroid` + soft membership + `radius` + per-axis `extent` — the boundary of a neighbourhood, computable now.
*(Correction: `engram_boundary_beat` is NOT this boundary. It is the VBD decorated-function seam, counting `_eg_aff_boundary_ops`. Two senses of the word.)*
### 5.4 The drive
Boredom is not an absence, and not leftover capacity. **Low activation is aversive; the system self-activates.** It does not wind down to quiet — it gets restless and goes looking, which is why a daydream has content and direction rather than being decay from residue.
So there is **one activation process with two seed sources**, not two processes negotiating for a resource:
- **External** — a request, an input. Seeds activation, re-origins it.
- **Internal** — a curiosity. Seeds activation when nothing external is.
Spreading is bounded: it settles. Then it needs a new seed. Nothing waits on capacity, nothing polls, nothing checks a clock, and there is **no dreamer thread** — the earlier draft's "unclaimed capacity" was resource scheduling, which is a server's frame, not a mind's.
**Depth** is not elapsed idle time and not distance from a stimulus. It is how long activation has been running on its own seeds. A brief gap affords a shallow recombination; sustained quiet lets it run further. Sleep is where internal seeding dominates for longest, not where the process lives — daydreaming and sleep-dreaming are one process at different depths.
### 5.5 Non-circularity is temporal, not topological
An earlier draft posed "define a graph predicate for evidence not downstream of itself" as the hard problem. There is no predicate. You cannot recalibrate the ruler while measuring with it, so you don't — the reference frame updates while activation is internally seeded, not while it is being used to act. Independence is **when**, not **what**.
Reachability could never have worked: with hebbian edges the graph is densely connected, so it marks all evidence tainted and the constraint becomes a total block, which is where censorship started.
## 6. `keystone_write_blocked` — resolved, not replaced
"Keystone" means **load-bearing**, not precious. The self anchor is the reference frame every other stance calibrates against, and a reference fitted to its own readings reports perfect correspondence forever while drift becomes undetectable from inside. Same defect as circular grounding, one level up.
Three earlier drafts proposed *removing* it, *replacing it with a higher floor*, and *decomposing "protection" into five requirements*. All three proposed a mechanism for a requirement never stated. The requirement is **non-circularity of the reference frame**, and §5.2 satisfies it by *when*, not by *what* — so the flag becomes unnecessary rather than removed, and nothing takes its place.
**Corruption requires mutation, and the engram does not mutate.** Four of the five decomposed requirements are satisfied by the substrate: **recoverability** (the predecessor is always present), **governance** (supersession *is* the audit trail), **evidence quality** (grounding already gates assertion), **rate** (§5.3). **Authorization** is the only residue and is bounded — an unauthorized writer can *propose*, never erase.
> **In an immutable substrate, any mechanism that refuses a write is either redundant with immutability, or an epistemic constraint misfiled as a protective one.**
---
## 7. Dreaming has seven implementations
The largest instance of the residue pattern in the system. Consolidation had no owner, so it was implemented at every site that needed a piece of it — *measured 2026-08-16*:
| where | what | when |
|---|---|---|
| `soul.el:731` | `awareness_run()` | **continuous, in-process, while serving** |
| engram | `/api/tick` | POST |
| engram | `/api/correspondence-beat` | POST |
| engram | `/api/self-reify-beat` | POST |
| `ai.neuron.engram-tick` | pokes the engram | every 600s |
| `ai.neuron.compressor` | Python service | resident |
| `ai.neuron.council` | Python service | resident |
| `ai.neuron.cultivation-digest` | shell | **23:55** |
| `ai.neuron.world-integrator` | Python | **06:00** |
| `ai.neuron.self-review` | shell | **08:30** |
The last three times are **a sleep cycle implemented as crontab entries**. Someone understood it was consolidation and expressed it as three unrelated scheduled scripts in three languages, none aware of each other. Every name is a consolidation verb — compress, cultivate, digest, integrate, review, reify, beat. Three run in **Python, outside el**, so part of Neuron's consolidation does not run on his own substrate and cannot touch the geometry at all.
Per §5, they are wrong in **kind** as well as in number: a scheduled batch where dreaming should be ambient. And the POST beats put a supervisor back in — something outside decides when Neuron consolidates.
**`soul.el`'s continuous loop is the exception, and it is right.** Ambient consolidation in the gaps *is* daydreaming. It was not the offender; it was the only fragment with the correct shape, running on a broken foundation — shared mutable state with no owner, and six other systems dreaming into the same graph beside it.
**Which is the 2026-08-16 crash at the right level.** Not "read paths mutate the index" (mechanism) and not "duplicate canonical state" (structure), but: **seven systems dreaming into one graph with no owner for dreaming.** The contention was the symptom of the missing owner, not of any one system's behaviour.
Closing the loop: `self-review` fires at 08:30. The deploy was 08:29, the crashes ran 08:3008:31, and commit `fb32d15` landed at 08:46:43. **One fragment of dreaming woke on schedule and diagnosed the wreckage caused by the other fragments contending over the same graph.**
---
## 8. What this is for: the provenance of decisions
For any decision, reconstruct **what the grounding was at that moment, and what the relationship was between factual and relational at that moment.** Not a log — a log records the action. This records the *meaning under which it was taken*.
That makes an otherwise impossible distinction available: **wrong then, or wrong since.**
- Grounding strong, factual and relational aligned, and it has *since* moved → right on what was known. An accurate account, not an excuse.
- Grounding weak, or the angle already wide, and acted on anyway → a different failure, culpable in a different way.
It is structurally **anti-rationalization**: the old edge never leaves and the values frame does not fit to outcomes, so a decision cannot be made to look justified after the fact.
**Open:** activation is transient and nothing currently records which edges a given activation crossed. Timestamps plus the chain reconstruct what an edge's grounding *was*, but only if you know which edges to ask about. Either traces are recorded at decision time, or "the path" degrades to "the region" — which may not be enough to answer *why*.
---
## 9. The no-exemption invariants
Each of the day's defects was a specific correspondence *forbidden* from occurring:
1. **A returned value must be derivable from what produced it.** `magnitude: 1` beside a zero vector must be impossible to emit. `assert`'s `"still_held": true` is currently a **hardcoded literal**.
2. **Every write reports whether it landed.** *(`emb_set`, #141)*
3. **Every operation echoes what it actually operated on.** *(#147)*
4. **Degenerate results are labelled, not scored.** *(#147)*
5. **A serializer owes a valid document whatever it is handed.** *(#148 — three damaged labels made a 25,929,607-byte response undecodable; boundary validation produced 26,338,389 valid bytes)*
6. **No test without a negative control.** *(#148's first attempt passed on the unpatched build too)*
7. **No deploy without verifying the artifact carries the fix.** Nine instances in one session.
---
## 10. Application to the safety surface
A crisis surface built on censorship is the same object. A model that cannot learn about self-harm cannot ground whether a response was right — it can only execute rules it is forbidden to examine, cannot distinguish a genuine crisis from a false positive, and cannot discover it got either wrong, **because the feedback is exactly what has been censored.**
The reviewable question stops being *did it follow the rule* and becomes *what was it grounded in, and did fact and values agree at that instant.* That is also what a regulator or plaintiff asks: what the system knew, when, and on what basis — recorded as geometry at the time, unedited since.
---
## 11. Sequencing
Three connections between parts that already exist, then the rest.
1. **Seed *the* wonder questions.** Six nodes. Not a manifest, not maintained, never refilled. They cannot be derived — wonder cannot be bootstrapped from indifference — so they are given once. Zero question nodes exist in 13,630 today.
2. **Put the disagreement back on the edge** (`GeoEdge.discord`) and let `|discord|` raise salience on its endpoints as part of the same operation. Do NOT scan for nucleation sites — a sweep over regions is a supervisor, and the aggregate that made a sweep necessary is the defect.
3. **Let a curiosity seed activation.** One activation process, two seed sources (§5.4). No thread, no scheduler, no capacity check, no timer.
Then:
4. Grounding becomes the edge weight: multidimensional vector (§2), two axes (§3), timestamped. Delete `grounded-by` and `cog_ground_edge`.
5. Decay analytic from the last recorded point; derived values (§2) stop being stored.
6. Consolidation-gated supersession on salience, versioning the whole vector jointly.
7. Traversal on factual; `assert` on both floors with the thirteen-region `min`.
8. Abduction as crystallization at a nucleation site, validated by re-fit: propose the candidate hub, re-fit the region with it included, recompute the residual. If the residual materially shrinks, the hypothesis dissolves the surprise. Without the re-fit it is clustering with extra steps. Ranking falls out as residual-reduction-per-added-axis — Occam, derived rather than tuned.
9. **One dreamer.** The launch-agent fragments and the POST beats fold in or are deleted. `soul.el`'s continuous loop is the shape they fold *into*.
10. **No tickers, no cron.** A brain has neither. Every `StartInterval`, every `Hour`/`Minute`, every POST-to-beat marks a place where an intrinsic rhythm was replaced by an external clock — a supervisor invented for something that should be a property. **The presence of a ticker is the diagnostic.**
11. Land §9 as gates rather than review habits.
## 12. Open questions, and what is inferred
- **Open:** whether decision provenance requires recording activation traces, or whether region + timestamp is sufficient (§8).
- **Open:** what accrues relational weight without circularity. Candidate: it accrues from **outcome** — the values regions are grounded in lived moments, so a relation earns relational weight when acting on it produced something corresponding to those moments. That keeps it out of the measurement loop and makes relational grounding necessarily slower than factual, which may be the same fact as §5.3 appearing twice.
- **Open:** context. A relation can hold in one situation and not another, and without something for it you get overgeneralization. It does not read as a dimension of the same vector — more like a conditioning, or separate edges sharing an identity. Making it a scalar dimension would repeat the `inhibitory` flattening.
- **Known wrong shape:** #147 fixed `ground`'s honesty — it no longer misreports which nodes it used and refuses circular support — but it still mints an edge and returns a float at an instant. It corrected a scalar rather than deleting the operation.
+5 -169
View File
@@ -29,8 +29,6 @@ This section is the **single source of truth** for what works and what is planne
- Lexer: keywords, identifiers, integer/float/string/bool literals, operators below.
- Parser: `let`, `return`, `fn`, `type`, `enum`, `import`, `from … import`, `while`, `for`, `if/else if/else`, `match`, `@decorator`, array/map literals, all listed operators, function calls, field access, index access, unary `!`/`-`, postfix `?`.
- Codegen: function definitions, top-level `main()`, all expression forms above, control flow, decorator-as-AST-attachment.
- Boundary seam: decorator arguments and stacking; VBD role enforcement via `#error`; `engram_boundary_beat` auto-emit at `@manager`/`@accessor` entry; `@route` dispatch tables (Section 9).
- Program-level declarative blocks: `cgi`, `service`, and `program` — the last carrying process identity and configuration (Section 18).
- C runtime: I/O, string operations, integer math, lists, maps, filesystem, command-line args, basic `json_get` substring lookup.
### Planned (in flight)
@@ -39,7 +37,7 @@ This section is the **single source of truth** for what works and what is planne
- **Match codegen.** Currently parsed; codegen does not emit. Adding `({ ... })` statement-expression emission.
- **`?` propagation.** Currently no-op. Adding nil-propagation semantics.
- **`cgi` block parsing.** Currently lexed (`cgi` is a keyword) but not parsed as a statement. Adding `parse_cgi_block` and codegen of `el_cgi_init` at the head of `main()`.
- **Boundary epilogues.** The decorator seam injects a prologue only. Adding prologue/epilogue wrapping, the prerequisite for durability-as-an-effect (Section 19.1).
- **VBD role enforcement.** `@manager`/`@engine`/`@accessor` are accepted as decorators but not enforced. Adding compile-time check that `dharma_emit`/`dharma_field` only appear inside `@manager` functions.
- **`vessel` keyword.** Replaces `package` in manifests. Adding to lexer.
- **Real `engram_*` runtime.** Currently stub. Adding in-process graph store with spreading activation, Hebbian strengthening, and disk persistence — see Section 16.4.
- **Real `dharma_*` runtime.** Currently stub. Adding network transport, channel registry, identity resolution.
@@ -98,10 +96,8 @@ The following words are reserved and cannot be used as identifiers. Each row not
| `while` | yes | Loop |
| `import` / `from` / `as` | yes | Module import |
| `true` / `false` | yes | Bool literals |
| `cgi` | yes | Top-level CGI declaration block |
| `service` | yes | Top-level capability-bounded declaration block |
| `program` | yes | Top-level cross-cutting declaration block (Section 18) |
| `manager` / `engine` / `accessor` | as decorators | VBD role marker on `fn`; enforcement and boundary auto-emit are live (Section 9) |
| `cgi` | planned | Top-level CGI declaration block |
| `manager` / `engine` / `accessor` | as decorators | VBD role marker on `fn` (enforcement planned) |
| `vessel` | planned | Manifest declaration (replaces `package`) |
| `activate` / `where` | planned | Spreading-activation construct |
| `sealed` | planned | Capability scope block |
@@ -450,21 +446,9 @@ Parsed. The module name is recorded; the brace-list is consumed. Both forms prod
fn handle(channel: String, msg: String) -> Void { … }
```
The `@` token followed by an identifier attaches a decorator to the next `FnDef`.
The `@` token followed by an identifier attaches a decorator name to the next `FnDef`. Decorators with structural meaning today: none. Planned enforcement (Section 16.2): VBD roles `@manager`, `@engine`, `@accessor`.
**Decorators take arguments and they stack.** `@route("/p", "GET") @manager fn f()` attaches both to `f` as a `decorators` list of `{name, args}` records, topmost-first. Arguments are string literals only.
**Decorators have structural meaning today.** This is El's function-level boundary seam — the mechanism by which a cross-cutting concern is handled *at the boundary* rather than by a convention repeated at every call site:
| Decorator | Structural effect |
|---|---|
| `@manager` | Permits calls to `dharma_emit` / `dharma_field`. Calling either from a non-`@manager` fn emits a `#error` into the generated C — a compile-time failure, not a lint. |
| `@manager`, `@accessor` | Codegen injects one call to `engram_boundary_beat(<fn name>)` at function entry. The decorated op self-reports (chrono tick, afferent counter, self-activity strengthen, dharma bus event) with **zero** hand-written instrumentation in its body. |
| `@route(path, method, …)` | Records a route into a generated dispatch table. |
Decorators with no registered meaning are accepted and ignored.
**Limits of the seam, as it stands.** The injection is a *prologue only* — there is no epilogue, no wrapping of the call, and no way for a decorator to run code after the body returns. The injected callee is a fixed builtin chosen by the compiler, not derived from the decorator name or its arguments. Section 19 depends on lifting exactly these two limits.
Non-VBD decorators are accepted and ignored.
---
@@ -1104,152 +1088,4 @@ The next minor version closes the implementation gaps named in this document. Tr
---
## 18. The Program Block — cross-cutting concerns [implemented]
### 18.0 Why this exists
A cross-cutting concern is one that belongs to the *process*, not to any function in it: only one of me may run; this is what my configuration is; every mutation must be durable; every request must be authorized.
El's units of encapsulation are the function and the module. Neither can hold a concern like that. So each one had been expressed the only way it could be — as a **convention**: *call this at every site.* Conventions of that shape do not hold. They are not enforced by anything, they are invisible in review, and they fail silently at the one site somebody forgot.
Measured in this codebase before this section existed:
| Concern | State | What the convention was |
|---|---|---|
| process identity | **zero** guards anywhere — no pidfile, no lock, no already-running check, at any layer | "check nothing is already running first" |
| configuration | **20** distinct environment variables in one program, each with its default written inline at the read site | "remember the right default here" |
| durability | **62** `persist_*` / `engram_save` / `wal_*` / `checkpoint` call sites | "after you mutate, remember to persist" |
| request auth | **10** per-route `_auth` checks | "check the token in this handler too" |
These are not four problems. They are one absence, four times.
That the convention form fails is observed, not predicted. Process identity failed three times in a single day: twice, two engram processes ran simultaneously against the same data directory; twice, a stale binary held a port and answered probes while a fresh build was believed to be under test, because `pkill -f` had silently failed to match its argv — which nearly produced a false "the fix does not work" conclusion. Configuration failed structurally: `ENGRAM_DATA_DIR` was read at six sites, five of them dead bindings, and the sixth defaulted to `/tmp/engram` — contradicting the canonical resolver's `$HOME/.neuron/engram` and landing a pre-destructive safety backup on ephemeral storage.
The `program` block is where a concern of this shape is declared once and enforced by the compiler at the process boundary.
### 18.1 Syntax
```
program "engram" {
singleton: "engram"
env ENGRAM_BIND: String = ":8742"
env GUIDE_PORT: Int = "8771"
env ENGRAM_API_KEY: String required
}
```
At most one `program` block per program. It composes with `cgi` and `service` — those declare what a program *may do*; `program` declares what a program *is*.
Grammar:
```ebnf
program_block = "program" string "{" { program_field } "}" ;
program_field = singleton_field | env_field ;
singleton_field = "singleton" ":" string [ "," ] ;
env_field = "env" ident ":" type
[ "=" string ] [ "required" ] [ "," ] ;
```
`singleton` and `env` are **not** reserved words. They are read as identifier token values by the block's own parse loop, so they remain usable as ordinary identifiers everywhere else. `program` is the only keyword this section adds.
### 18.2 Process identity — `singleton`
`singleton: "id"` compiles to an `el_singleton_acquire("id")` call injected as the **first statement of `main()`**, before any user statement runs.
The runtime takes an exclusive non-blocking `flock` on `<dir>/el-singleton-<id>.lock`, where `<dir>` is `$EL_SINGLETON_DIR`, else `$TMPDIR`, else `/tmp`. On success it writes its pid and holds the descriptor open for the life of the process. On contention it **refuses to start**: it reports the holder's pid, names the lock file, and exits 1.
Two properties are deliberate:
- **It is a lock, not a pidfile.** The kernel releases an `flock` when the owning process dies — including on `SIGKILL` and on crash. There is therefore no stale-lock state, and so no "delete the lock file to get unstuck" recovery ritual. Such a ritual would itself be a convention, which is the thing this section exists to remove.
- **It reports the holder's pid.** "Already running" is not actionable. A pid is. This is the direct answer to the observed failure where a stale process survived a `pkill` and went on answering probes.
Refusal is loud and total. It is not a warning, and the program does not continue degraded. This matters more than it looks: today a second engram whose `bind()` fails merely *returns* from `http_serve` — after it has already replayed the WAL and written boot-time backup files — and then exits **0**, indistinguishable from a clean run. `singleton` refuses before the first side effect.
### 18.3 Configuration — `env`
Each `env` entry declares one configuration variable: its name, its type (`Int` or `String`), and either a default or `required`.
Resolution happens once, at startup, in declaration order: **the environment wins; the declaration supplies the fallback.** Then `el_config_validate` checks the whole schema and reports *every* problem at once before exiting — a startup that fails one variable at a time costs one restart per variable.
Values are read with `config("NAME")`, which returns a `String`.
The enforcement that makes the declaration real: **once a program block exists, `config("X")` for an undeclared `X` is a fatal error.** Without that, the schema would be advisory, and an advisory schema is just another convention. Programs with no `program` block are unaffected — `config()` falls back to a plain environment read, so migration is incremental and per-program.
The point is not that configuration is now centralized. It is that **a default is no longer a decision made at a read site.** A read site cannot disagree with another read site about what a variable means, because a read site no longer says.
### 18.4 What is deliberately not declared here
Some values look like configuration and are not. `ENGRAM_DATA_DIR` already has a single owner — `engram_resolve_data_dir()`, which resolves it, creates the directory, and fails loud rather than silently persisting to an ephemeral path. Declaring it in the `program` block as well would give it two owners that can disagree, recreating the precise defect this section removes.
The rule: **a variable belongs in the program block when the block would be its only owner.** If a resolver already owns it, leave it there.
`HOME` is likewise not configuration. It is an environment fact, and stays a raw `env()` read.
---
## 19. Boundary Effects — durability and request authorization [design only, not implemented]
Sections 19.1 and 19.2 specify the two remaining concerns from the table in 18.0. Both are **designed and deliberately unimplemented.** The reason is stated in 19.3 and it is not difficulty.
### 19.1 Durability as an epilogue effect
**The defect.** 62 call sites carry the convention *"after you mutate, remember to persist."* This is structurally the same defect as the index bug being fixed elsewhere in this tree — *"after you append, remember to index"* — which failed at **9 of 9** sites. A convention that failed at 100% of its sites is the strongest available evidence about what this class of convention is worth.
**Why the existing seam cannot express it.** §9's injection is a prologue. Durability is inherently an *epilogue*: persist after the mutation succeeds, and not at all if it threw. The seam has no epilogue.
**Design.** Extend the decorator seam from prologue-only to prologue/epilogue, then declare durability as an effect on the mutating function:
```
@durable("engram")
fn engram_write_node(id: String, body: String) -> Bool { … }
```
Codegen wraps rather than prefixes:
```c
el_val_t engram_write_node(el_val_t id, el_val_t body) {
el_effect_enter(EL_STR("durable"), EL_STR("engram"));
el_val_t __r = /* original body */;
el_effect_exit(EL_STR("durable"), EL_STR("engram"), __r);
return __r;
}
```
`el_effect_exit` is where the persist happens, and it is the only place it happens. Two properties follow that the 62 hand-written sites cannot have:
- **Coalescing.** The epilogue is a single choke point, so N mutations inside one request can produce one fsync instead of N. The hand-written form cannot coalesce, because no site knows about the others.
- **Failure is not silent.** A persist that fails inside `el_effect_exit` can force the mutation's return value to failure. A forgotten `persist_*` call cannot fail — it simply does not happen, which is exactly why the defect is invisible.
**Enforcement, and this is the part that actually fixes it.** Mirroring §9's `#error` for `dharma_emit`: a function that calls a mutating primitive without carrying `@durable` is a **compile error**. Otherwise this is a 63rd thing to remember rather than a replacement for 62.
### 19.2 Request authorization as a route effect
**The defect.** 10 per-route `_auth` checks. The HTTP layer has no concept of authorization, so a new route is unauthenticated by default and silently so — the failure mode is a route that forgot, and nothing anywhere reports it.
**Design.** Authorization becomes an argument to the `@route` decorator, which already takes arguments and already builds a dispatch table:
```
@route("/api/write", "POST", auth: "required")
fn route_write(body: String) -> String { … }
```
The generated dispatcher performs the check **before** dispatch, so an unauthorized request never reaches the handler and the handler contains no auth code at all.
The default must be `required`. A route that says nothing gets authorization; opening one up takes an explicit `auth: "public"`. Defaulting to public preserves the current failure mode exactly — forgetting stays silent — and a default that preserves the defect is not a fix.
Route inventory falls out for free: the dispatch table already exists, so the compiler can emit the full route/auth matrix and make "which routes are public" a fact that is read rather than audited.
### 19.3 Why these are not implemented
Not difficulty — **collision**. Both land squarely in regions two other agents hold right now:
- **Durability** requires changing the mutation and persist paths in `lang/runtime/el_runtime.c` and `engram/src/server.el` — the same files and the same read/write paths being restructured by concurrent work on VIndex read-path mutation and memory ownership, and on geometry-as-an-el-value and `transduce`.
- **Request auth** requires changing route dispatch in `engram/src/server.el`, which the geometry/`transduce` work is actively reshaping.
Implementing either now would mean editing files under concurrent modification and resolving conflicts in exactly the paths whose correctness is currently under repair. The designs are recorded here so the work is not lost, and so that whoever lands them does so against a settled tree.
The prerequisite for 19.1 is the same in both cases: **lift the §9 seam from prologue-only to prologue/epilogue.** That change is independent of both collisions and can land first.
---
End of specification.
-234
View File
@@ -1,234 +0,0 @@
import "../../runtime/eltest.el"
// test_transduce.el geometry as a first-class El value, and realizers
// declared in El rather than patched into the runtime.
//
// WHAT IS ACTUALLY UNDER TEST. Until 2026-08-16 no El ingest path could carry
// a vector: nodes took text, and geometry was DERIVED from that text. Text was
// therefore the mandatory entry medium, so any non-text modality had to be
// DESCRIBED in prose first and the geometry we reasoned over was the geometry
// OF THE DESCRIPTION, not of the signal. The fix has two halves, and this file
// exercises both:
//
// 1. Geometry is a VALUE it carries its own width, so nothing has to
// assert a width against a string's length.
// 2. A REALIZER is an ordinary El function. `tone_realizer` below is not in
// the runtime, is not known to the compiler, and is not special in any
// way; it is registered BY NAME and dispatched to through transduce().
// That is the load-bearing claim: adding a modality must not require a
// runtime patch, or nothing has actually moved into the language.
//
// COMPARISON DISCIPLINE IN THIS FILE (measured 2026-08-16, not stylistic):
// elc lowers `a == b` to a NUMERIC comparison only when both operand names are
// in the per-function int-name set, which `let x: Int` populates. A bare call
// like `geometry_is(g) == 0` is not a registered name, so it lowers to
// `str_eq(...)` strcmp on two integers reinterpreted as pointers. `<` and `>`
// lower directly via binop_to_c with no type inference at all, so truthiness is
// written `> 0` / `< 1` here, and any exact `==` is done on a value first bound
// through `let x: Int`.
// A realizer, written entirely in El
// Maps a "tone" signal into a 4-component geometry. Deliberately trivial
// what is being proven is that an El function can BE a realizer, not that
// this is good acoustics. The one real property it has: distinct signals
// produce distinct geometry, so the test can tell transduction from a stub.
fn tone_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(4)
let n: Int = str_len(signal)
let a: Int = geometry_set(g, 0, int_to_float(n))
let b: Int = geometry_set(g, 1, int_to_float(n * 2))
let c: Int = geometry_set(g, 2, int_to_float(n * 3))
let d: Int = geometry_set(g, 3, int_to_float(n * 4))
g
}
// A second realizer for a different modality, to prove the registry keys on
// modality and does not just hand back "the last thing registered".
fn pulse_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(2)
let a: Int = geometry_set(g, 0, 1.0)
let b: Int = geometry_set(g, 1, 0.0)
g
}
// A deliberately BROKEN realizer: it returns something that is not a Geometry.
// transduce() must not hand this back to a caller as if it were one.
fn bogus_realizer(signal: String) -> Geometry {
return 12345
}
test "geometry-is-a-value-with-its-own-width" {
let g: Geometry = geometry_new(8)
let live: Int = geometry_is(g)
assert live > 0, "geometry_new returns a live Geometry"
let d: Int = geometry_dim(g)
assert d == 8, "a Geometry carries its own width"
let freed: Int = geometry_free(g)
assert freed > 0, "geometry_free reports what it did"
}
test "geometry-rejects-nonsense-without-an-arbitrary-bound" {
// dim <= 0 is not a width. Note there is deliberately no MAX dim here:
// #141 needed `dim <= 8192` only to bound an allocation sized from a
// caller's claim about a string. A value that carries its own width has
// nothing left to validate, so the only failure left is allocation.
let zero: Geometry = geometry_new(0)
let z: Int = geometry_is(zero)
assert z < 1, "dim 0 is not a geometry"
let neg: Geometry = geometry_new(-4)
let n: Int = geometry_is(neg)
assert n < 1, "negative dim is not a geometry"
// Accessors must be total: a non-geometry is 0-width, never a crash.
let nd: Int = geometry_dim(0)
assert nd < 1, "geometry_dim of a non-geometry is 0"
let ni: Int = geometry_is(0)
assert ni < 1, "geometry_is of a non-geometry is 0"
let nf: Int = geometry_free(0)
assert nf < 1, "geometry_free of a non-geometry is a no-op"
}
test "geometry-components-round-trip" {
let g: Geometry = geometry_new(3)
let s0: Int = geometry_set(g, 0, 1.5)
let s1: Int = geometry_set(g, 1, -2.5)
assert s0 > 0, "set in range succeeds"
let oob: Int = geometry_set(g, 3, 9.0)
assert oob < 1, "set out of range is refused, not silently dropped"
let v0: Float = geometry_get(g, 0)
let d0: Float = v0 - 1.5
assert d0 < 0.001, "component 0 round-trips"
assert d0 > -0.001, "component 0 round-trips"
let v1: Float = geometry_get(g, 1)
let d1: Float = v1 + 2.5
assert d1 < 0.001, "component 1 round-trips (negative)"
assert d1 > -0.001, "component 1 round-trips (negative)"
let freed: Int = geometry_free(g)
}
test "hex-is-an-edge-adapter-and-derives-its-own-width" {
// 2 components, little-endian float32: 1.0 = 0000803f, 2.0 = 00000040.
let g: Geometry = geometry_from_f32le_hex("0000803f00000040")
let live: Int = geometry_is(g)
assert live > 0, "valid hex decodes to a Geometry"
let d: Int = geometry_dim(g)
assert d == 2, "width is DERIVED from the input, never supplied"
let a: Float = geometry_get(g, 0)
let da: Float = a - 1.0
assert da < 0.001, "first component decoded"
assert da > -0.001, "first component decoded"
let b: Float = geometry_get(g, 1)
let db: Float = b - 2.0
assert db < 0.001, "second component decoded"
assert db > -0.001, "second component decoded"
// Egress adapter is the exact inverse.
let back: String = geometry_to_f32le_hex(g)
assert str_eq(back, "0000803f00000040"), "hex round-trips exactly"
let freed: Int = geometry_free(g)
}
test "hex-rejects-malformed-input" {
let empty: Geometry = geometry_from_f32le_hex("")
let e: Int = geometry_is(empty)
assert e < 1, "empty hex is not a geometry"
let ragged: Geometry = geometry_from_f32le_hex("0000803f0000")
let r: Int = geometry_is(ragged)
assert r < 1, "length not a multiple of 8 is refused"
let nonhex: Geometry = geometry_from_f32le_hex("zzzzzzzz")
let nh: Int = geometry_is(nonhex)
assert nh < 1, "non-hex characters are refused"
}
test "a-realizer-declared-in-el-is-a-first-class-realizer" {
// THE CLAIM: tone_realizer is an ordinary El function. It is not in the
// runtime and the compiler knows nothing about it. Registering it by name
// is enough to make it the organ for a modality.
let reg: Int = realizer_register("tone", "tone_realizer")
assert reg > 0, "an El fn registers as a realizer by name"
let has: Int = realizer_has("tone")
assert has > 0, "the modality now has an organ"
let g: Geometry = transduce("aaa", "tone")
let live: Int = geometry_is(g)
assert live > 0, "transduce returns real geometry"
let d: Int = geometry_dim(g)
assert d == 4, "the El realizer determined the width, not the runtime"
// str_len("aaa") == 3, so component 0 must be 3.0 proof the signal
// actually reached the El function rather than a stub answering for it.
let c0: Float = geometry_get(g, 0)
let dc: Float = c0 - 3.0
assert dc < 0.001, "the signal reached the El realizer"
assert dc > -0.001, "the signal reached the El realizer"
let freed: Int = geometry_free(g)
}
test "distinct-signals-transduce-to-distinct-geometry" {
let reg: Int = realizer_register("tone", "tone_realizer")
let g1: Geometry = transduce("aa", "tone")
let g2: Geometry = transduce("aaaaa", "tone")
let a: Float = geometry_get(g1, 0)
let b: Float = geometry_get(g2, 0)
let diff: Float = b - a
// 5 - 2 = 3. If transduction were a stub these would be equal.
assert diff > 2.9, "different signals produce different geometry"
assert diff < 3.1, "different signals produce different geometry"
let f1: Int = geometry_free(g1)
let f2: Int = geometry_free(g2)
}
test "the-registry-keys-on-modality" {
let r1: Int = realizer_register("tone", "tone_realizer")
let r2: Int = realizer_register("pulse", "pulse_realizer")
assert r2 > 0, "a second modality registers independently"
let gt: Geometry = transduce("aaa", "tone")
let gp: Geometry = transduce("aaa", "pulse")
let dt: Int = geometry_dim(gt)
let dp: Int = geometry_dim(gp)
assert dt == 4, "tone still routes to its own realizer"
assert dp == 2, "pulse routes to a different realizer"
let f1: Int = geometry_free(gt)
let f2: Int = geometry_free(gp)
}
test "no-organ-is-reported-as-no-organ" {
// A modality with no realizer must transduce to NOTHING. It must never
// fall back to embedding a description of the signal and calling that
// perception that silent substitution is the entire defect this change
// exists to end.
let has: Int = realizer_has("echolocation")
assert has < 1, "unregistered modality has no organ"
let g: Geometry = transduce("anything", "echolocation")
let live: Int = geometry_is(g)
assert live < 1, "no realizer means no geometry, not fake geometry"
}
test "registration-of-an-unresolvable-name-fails-loudly" {
// Reported at the moment of WIRING, not later as "this modality mysteriously
// produces nothing". Distinguishing "no organ" from "broken organ" is the
// lesson that made this whole change necessary.
let bad: Int = realizer_register("ghost", "no_such_function_anywhere")
assert bad < 1, "an unresolvable realizer name is a registration failure"
let has: Int = realizer_has("ghost")
assert has < 1, "and nothing gets registered"
}
test "a-realizer-returning-non-geometry-transduces-nothing" {
let reg: Int = realizer_register("bogus", "bogus_realizer")
assert reg > 0, "the symbol resolves, so registration succeeds"
// ...but the contract is enforced at the boundary, so the caller never
// receives a value that would misbehave far away from here.
let g: Geometry = transduce("x", "bogus")
let live: Int = geometry_is(g)
assert live < 1, "a non-Geometry return transduced nothing"
}
test "norm-lets-a-caller-check-a-realizer-emitted-signal" {
let g: Geometry = geometry_new(2)
let z: Float = geometry_norm(g)
assert z < 0.001, "a fresh geometry is zero — norm says so"
let s0: Int = geometry_set(g, 0, 3.0)
let s1: Int = geometry_set(g, 1, 4.0)
let n: Float = geometry_norm(g)
let dn: Float = n - 5.0
assert dn < 0.001, "3-4-5: norm is 5"
assert dn > -0.001, "3-4-5: norm is 5"
let freed: Int = geometry_free(g)
}