Compare commits

...

1 Commits

Author SHA1 Message Date
bigmerge 3fcc36c2f1 runtime: transduction is a language concern, so move it into the language
El SDK CI - dev / build-and-test (pull_request) Failing after 14m58s
#141 let signal enter as geometry and it worked, but it was placed at the
CONSUMER and said so in its own commit message. This is the correction.

Three defects, all of them placement:

1. It sat in the engram. Ingest is a LANGUAGE concern — every el program
   touching any modality needs it, and the engram is merely one el program
   that happens to hold a graph. The geometry surface is now defined in
   el_runtime.c immediately ABOVE the engram section and depends on nothing
   inside it. Delete the entire engram and geometry still enters el.

2. It marshalled the vector as a hex STRING, because el had no first-class
   geometry value — which reintroduced text as the TRANSPORT medium one layer
   below the problem being fixed. Geometry is now an el value: a magic-tagged
   heap object carried in el_val_t, same discipline as List/Map. Hex survives
   only as an adapter at the edge, which is all an encoding should ever be.

3. It needed an arbitrary `dim <= 8192` bound purely to size an allocation
   from a caller's CLAIM about a string's length. A value carries its own
   width, so the width is derived and never asserted. The bound is gone, not
   raised — there is nothing left to validate.

Language surface, none of it engram-prefixed: geometry_new / _dim / _is /
_get / _set / _norm / _free, geometry_from_f32le_hex + geometry_to_f32le_hex
as the wire adapters, realizer_register(modality, fn_name), realizer_has, and
transduce(signal, modality) -> Geometry.

REALIZERS ARE DECLARABLE IN EL. This is the part that makes the move real
rather than nominal: registration resolves a name with dlsym against the
running binary, the identical mechanism http_set_handler already relies on,
because every el `fn name(...)` compiles to a global C symbol with that exact
name. So an ordinary el function IS a realizer and a new modality needs no
runtime patch. Verified end to end in lang/examples/transduce.el: an el-defined
tone_realizer is registered by name, transduce dispatches to it, and the
signal demonstrably reaches it (distinct signals produce distinct geometry).

A modality with no realizer transduces to NOTHING. There is deliberately no
built-in realizer, not even for text — silently embedding a description of a
signal and calling that perception is the exact defect this ends.

engram/src/server.el is migrated: POST /api/nodes decodes "emb" hex exactly
once, at the edge, into a Geometry, and everything below that line moves
geometry. The wire is unchanged because production clients speak it. "dim" is
now an ASSERTION about the vector, not the source of its width; disagreement
is a rejected ingest, not a silent reinterpretation.

#141's engram_node_set_emb becomes a DEPRECATED WRAPPER over
geometry_from_f32le_hex + node_attach_geometry — kept only because the runtime
ships as an SDK asset and a downstream binary may link the symbol. Its exact
contract, negative cases included, is preserved and re-verified.

ingest.el's `fn transduce` is renamed transduce_manifold. Mechanically it had
to yield the name (duplicate C symbol, a hard compile error, measured). But it
was never signal->geometry: it chunks already-extracted content into a node+edge
manifold, one layer up, and had taken the name belonging to the primitive
underneath it. Behaviour unchanged.

PROPERTIES FROM #141 PRESERVED, each re-measured on a scratch engram (:8971,
never prod :8742):
  - off-dimension vectors stored but NOT indexed — the HNSW build loop still
    filters on n->emb_dim == dim at four sites, so a 64-dim voice vector is
    durable and addressable without perturbing the 768-dim canonical index
  - geometry makes a node ineligible for embed_backfill: after backfill the
    64-dim voice node was still 64-dim while the text control acquired 768
  - the create response reports whether geometry landed, and the node document
    always emits emb_dim and embedded

Read-back with control and negatives, all verified against a PID-confirmed
fresh binary: geometry node emb_dim=64 embedded=true / emb_set=1; text-only
control emb_dim=0 embedded=false / emb_set=0; malformed hex, ragged length,
and dim-disagreement each emb_set=0.

Two compiler landmines found by reading the generated C rather than trusting a
successful build, both documented at their sites: elc lowers `a == b` to
str_eq unless both operand NAMES are in the per-function int-name set (which
does NOT propagate into nested if-expression blocks — the first cut would have
strcmp'd two integers as pointers on the first geometry-bearing request), and
`+` lowers to string concat when either operand is a user-defined call.
2026-08-16 11:37:27 -05:00
6 changed files with 969 additions and 87 deletions
+52 -16
View File
@@ -247,6 +247,24 @@ fn persist_bulk() -> Int {
return persist_canonical() return persist_canonical()
} }
// COMPILER LANDMINE, measured 2026-08-16 do not inline this back into the
// caller. elc lowers `a == b` to numeric comparison only when both operand
// NAMES are in the per-function int-name set, which `let x: Int` populates.
// That registration does NOT propagate into a nested if-expression block: the
// first cut of the geometry-ingest path wrote `let claimed: Int = ...` and
// `let got: Int = ...` inside the else-arm and `claimed == got` came out of
// codegen as `str_eq(claimed, got)` strcmp on two integers reinterpreted as
// pointers, i.e. a segfault on the first geometry-bearing request. Read back
// out of the generated C, not guessed. Function PARAMETERS annotated `: Int`
// do register reliably (verified: `if (claimed == actual)`), so the comparison
// lives in a function of its own. Note also the explicit `return`s a trailing
// if-EXPRESSION at a function tail emits as a statement and the function
// returns 0 regardless, which is the same probe's second finding.
fn width_agrees(claimed: Int, actual: Int) -> Int {
if claimed == actual { return 1 }
return 0
}
// INCOMPLETE-ROUTE FIX (2026-07-24 self-review): this route silently dropped // INCOMPLETE-ROUTE FIX (2026-07-24 self-review): this route silently dropped
// label, importance, tier, and tags engram_node() defaults label to content // label, importance, tier, and tags engram_node() defaults label to content
// and importance to 0.5, so every node created over HTTP lost its metadata. // and importance to 0.5, so every node created over HTTP lost its metadata.
@@ -288,26 +306,44 @@ fn route_create_node(method: String, path: String, body: String) -> String {
salience, importance, confidence, salience, importance, confidence,
tier, tags tier, tags
) )
// GEOMETRY INGEST (2026-08-16 self-review): this route accepted an "emb" // GEOMETRY INGEST geometry-valued end to end (2026-08-16).
// field, returned 200 with a fresh id, and stored NOTHING engram_node_full
// has no vector parameter, so the caller's geometry was silently discarded
// and the node came back emb_dim=None / embedded:false. Measured live while
// trying to admit a voice signal. The consequence was structural, not
// cosmetic: text was the only entry medium, so any non-text modality had to
// be DESCRIBED in prose and what we then reasoned over was the geometry of
// the description, not of the signal.
// //
// "emb" is little-endian float32 hex (dim*8 chars) the encoding the // The defect this route originally had: it accepted an "emb" field,
// perception vessel's /voice/embed already emits, so a realizer's output // returned 200 with a fresh id, and stored NOTHING, because engram_node_full
// moves in with no float-array round trip. "dim" defaults to the vector's // has no vector parameter. The consequence was structural, not cosmetic
// implied width. Off-dimension vectors are stored but not inserted into the // text was the only entry medium, so any non-text modality had to be
// resident index (its build loop filters on emb_dim), so a modality vector // DESCRIBED in prose, and what we then reasoned over was the geometry of the
// is durable and addressable without perturbing the canonical index. // description, not of the signal.
//
// #141 fixed the drop but marshalled the vector as a hex STRING through
// engram_node_set_emb, which put text back as the TRANSPORT medium one layer
// below the problem being fixed. This is that correction: hex is decoded
// exactly ONCE, here at the edge, into a first-class Geometry, and every
// step below this line moves geometry rather than text. An encoding at the
// boundary is what an encoding is for.
//
// The WIRE is deliberately unchanged "emb" is still little-endian float32
// hex (8 chars per component), the encoding the perception vessel's
// /voice/embed already emits because production clients speak it. What
// changed is underneath it.
//
// "dim" is now treated as an ASSERTION about the vector the caller sent, not
// as the source of its width: a Geometry carries its own width. A stated dim
// that disagrees is a REJECTED ingest, not a silent reinterpretation. Omitting
// "dim" is fine and means "trust the vector", which is the honest default.
//
// Off-dimension vectors remain stored but not inserted into the resident HNSW
// index (its build loop filters on emb_dim), so a 64-dim voice geometry is
// durable and addressable without perturbing the 768-dim canonical index.
let emb_hex: String = json_get_string(body, "emb") let emb_hex: String = json_get_string(body, "emb")
let emb_set: Int = if str_eq(emb_hex, "") { 0 } else { let emb_set: Int = if str_eq(emb_hex, "") { 0 } else {
let g: Geometry = geometry_from_f32le_hex(emb_hex)
let got: Int = geometry_dim(g)
let dim_raw: String = json_get_raw(body, "dim") let dim_raw: String = json_get_raw(body, "dim")
let dim: Int = if str_eq(dim_raw, "") { str_len(emb_hex) / 8 } else { json_get_int(body, "dim") } let claimed: Int = if str_eq(dim_raw, "") { got } else { json_get_int(body, "dim") }
engram_node_set_emb(id, emb_hex, dim) let landed: Int = if width_agrees(claimed, got) > 0 { node_attach_geometry(id, g) } else { 0 }
let freed: Int = geometry_free(g)
landed
} }
let saved: Int = persist_node(id) let saved: Int = persist_node(id)
// ORPHAN PREVENTION (ENGRAM_AUTOCONNECT): connect the fresh node to its // ORPHAN PREVENTION (ENGRAM_AUTOCONNECT): connect the fresh node to its
+27 -12
View File
@@ -13,7 +13,7 @@
// relations add edges. Every node enters with PROVENANCE + grounding-level // relations add edges. Every node enters with PROVENANCE + grounding-level
// + stewardship class from the moment of entry. // + stewardship class from the moment of entry.
// //
// transduce() is THE single mechanism one function, polymorphic, with no // transduce_manifold() is THE single mechanism one function, polymorphic, with no
// content-type branch inside it. It does not ask whether a payload is // content-type branch inside it. It does not ask whether a payload is
// prose, structured data, or raw/opaque bytes (audio, or anything else); // prose, structured data, or raw/opaque bytes (audio, or anything else);
// it runs one boundary-scan-with-fixed-window-fallback chunking algorithm // it runs one boundary-scan-with-fixed-window-fallback chunking algorithm
@@ -401,10 +401,25 @@ fn head80(s: String) -> String {
// truncates at the first embedded NUL, which is routine in real binary // truncates at the first embedded NUL, which is routine in real binary
// bytes) is a MECHANICAL fidelity concern that belongs to whatever produced // bytes) is a MECHANICAL fidelity concern that belongs to whatever produced
// `source` (see ingest_file's file_source_string below) not a // `source` (see ingest_file's file_source_string below) not a
// content-type judgment made in here. transduce() never learns whether a // content-type judgment made in here. transduce_manifold() never learns whether a
// chunk is plain text or a base64-encoded raw-byte window; every chunk is // chunk is plain text or a base64-encoded raw-byte window; every chunk is
// handled identically either way. // handled identically either way.
fn transduce(nodes: [String], edges: [String], source: String, // RENAMED transduce -> transduce_manifold (2026-08-16). Two reasons, and the
// first is not the interesting one:
//
// 1. Mechanical: `transduce` is now a LANGUAGE primitive in el_runtime.h
// (transduce(signal, modality) -> Geometry). Every El `fn name(...)`
// compiles to a global C symbol with that exact name, so keeping this
// name here is a hard `conflicting types for 'transduce'` compile error
// the moment ingest.c links el_runtime.c. Measured, not anticipated.
//
// 2. Actual: this function was never signal->geometry. It chunks already-
// extracted content and PACKS it into a node+edge manifold a real
// operation, but one layer up, and it had taken the name that belongs to
// the primitive underneath it. `transduce` is where a signal becomes
// geometry; `transduce_manifold` is where extracted content becomes
// structure. Nothing about this function's behaviour changed.
fn transduce_manifold(nodes: [String], edges: [String], source: String,
prov: String, ground: String, steward: String, prov: String, ground: String, steward: String,
root_lid: String, root_title: String) -> [String] { root_lid: String, root_title: String) -> [String] {
let tagbase: String = "prov:" + prov + " ground:" + ground + " steward:" + steward let tagbase: String = "prov:" + prov + " ground:" + ground + " steward:" + steward
@@ -531,8 +546,8 @@ fn default_steward() -> String {
// trustworthy verbatim. When they don't (silent truncation happened), // trustworthy verbatim. When they don't (silent truncation happened),
// rebuild the payload as base64-encoded fixed-size windows read directly // rebuild the payload as base64-encoded fixed-size windows read directly
// off disk (fs_read_b64_chunk binary-safe in C), joined with the same // off disk (fs_read_b64_chunk binary-safe in C), joined with the same
// "\n\n" boundary marker transduce()'s generic scan already looks for, so // "\n\n" boundary marker transduce_manifold()'s generic scan already looks for, so
// transduce() sees one ordinary boundary-delimited payload and runs its one // transduce_manifold() sees one ordinary boundary-delimited payload and runs its one
// algorithm on it exactly as it would on prose it never learns that a // algorithm on it exactly as it would on prose it never learns that a
// fidelity problem occurred upstream, let alone why. // fidelity problem occurred upstream, let alone why.
fn file_source_string(path: String, text: String, real_size: Int) -> String { fn file_source_string(path: String, text: String, real_size: Int) -> String {
@@ -541,7 +556,7 @@ fn file_source_string(path: String, text: String, real_size: Int) -> String {
// 3072 raw bytes -> 4096 base64 chars (3 divides evenly into base64's // 3072 raw bytes -> 4096 base64 chars (3 divides evenly into base64's
// 3-byte/4-char ratio); keeps each resulting node's content a clean, // 3-byte/4-char ratio); keeps each resulting node's content a clean,
// bounded, low-kilobytes unit, same order of magnitude as the fixed // bounded, low-kilobytes unit, same order of magnitude as the fixed
// fallback window in transduce() itself. // fallback window in transduce_manifold() itself.
let win: Int = 3072 let win: Int = 3072
let out: String = "" let out: String = ""
let off: Int = 0 let off: Int = 0
@@ -561,7 +576,7 @@ fn file_source_string(path: String, text: String, real_size: Int) -> String {
} }
// ingest one file -> report JSON. Uniform for every file regardless of // ingest one file -> report JSON. Uniform for every file regardless of
// extension or content transduce() decides nothing about content-type, so // extension or content transduce_manifold() decides nothing about content-type, so
// neither does this function; it only decides whether the raw bytes made it // neither does this function; it only decides whether the raw bytes made it
// through the read intact (file_source_string), which is a fidelity // through the read intact (file_source_string), which is a fidelity
// question, not a format one. // question, not a format one.
@@ -573,14 +588,14 @@ fn ingest_file(path: String) -> String {
return "{\"error\":\"empty or unreadable\",\"path\":" + j_q(path) + "}" return "{\"error\":\"empty or unreadable\",\"path\":" + j_q(path) + "}"
} }
let prov: String = "file:" + path let prov: String = "file:" + path
let packed: [String] = transduce(el_list_empty(), el_list_empty(), let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
source, prov, default_ground(), default_steward(), source, prov, default_ground(), default_steward(),
"doc:" + basename(path), basename(path)) "doc:" + basename(path), basename(path))
return merge_packed(packed) return merge_packed(packed)
} }
// ingest a directory: walk one level, ingest every file found, aggregate. // ingest a directory: walk one level, ingest every file found, aggregate.
// No extension filter transduce() handles any payload uniformly now, so // No extension filter transduce_manifold() handles any payload uniformly now, so
// there is no content-type gate at the directory boundary either. // there is no content-type gate at the directory boundary either.
fn ingest_dir(path: String) -> String { fn ingest_dir(path: String) -> String {
let entries: [String] = fs_list(path) let entries: [String] = fs_list(path)
@@ -615,7 +630,7 @@ fn ingest_dir(path: String) -> String {
fn ingest_url(url: String) -> String { fn ingest_url(url: String) -> String {
let body: String = http_get(url) let body: String = http_get(url)
if str_eq(body, "") { return "{\"error\":\"empty fetch\",\"url\":" + j_q(url) + "}" } if str_eq(body, "") { return "{\"error\":\"empty fetch\",\"url\":" + j_q(url) + "}" }
let packed: [String] = transduce(el_list_empty(), el_list_empty(), let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
body, "url:" + url, "extracted", "public-web", body, "url:" + url, "extracted", "public-web",
"url:" + url, url) "url:" + url, url)
return merge_packed(packed) return merge_packed(packed)
@@ -630,7 +645,7 @@ fn ingest_llm(query: String) -> String {
let resp: String = http_post_json("http://127.0.0.1:11434/api/generate", body) let resp: String = http_post_json("http://127.0.0.1:11434/api/generate", body)
let answer: String = json_get_string(resp, "response") let answer: String = json_get_string(resp, "response")
if str_eq(answer, "") { return "{\"error\":\"no model response\"}" } if str_eq(answer, "") { return "{\"error\":\"no model response\"}" }
let packed: [String] = transduce(el_list_empty(), el_list_empty(), let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
answer, "llm:" + model + ":" + query, "candidate-provisional", "guide-provisional", answer, "llm:" + model + ":" + query, "candidate-provisional", "guide-provisional",
"llm:" + query, "guide answer: " + query) "llm:" + query, "guide answer: " + query)
return merge_packed(packed) return merge_packed(packed)
@@ -682,7 +697,7 @@ fn ingest_stream(path: String) -> String {
// It is NOT a content-type flag: it says nothing about what's inside the // It is NOT a content-type flag: it says nothing about what's inside the
// bytes once fetched, and none of the five ingest_* functions it selects // bytes once fetched, and none of the five ingest_* functions it selects
// among interpret their payload differently by content shape anymore // among interpret their payload differently by content shape anymore
// they all hand off to the single, format-agnostic transduce(). The old // they all hand off to the single, format-agnostic transduce_manifold(). The old
// "structured" value (a caller-declared alias for "file", used only to hint // "structured" value (a caller-declared alias for "file", used only to hint
// the now-removed JSON-vs-prose branch) is gone along with that branch. // the now-removed JSON-vs-prose branch) is gone along with that branch.
let kind: String = env("INGEST_KIND") let kind: String = env("INGEST_KIND")
+213
View File
@@ -0,0 +1,213 @@
// transduce.el geometry as a first-class El value, and a realizer written
// in El. Runnable: this is the worked example for the transduce surface, and
// it doubles as an executable proof because it checks every claim it makes.
//
// elc lang/examples/transduce.el > transduce.c
// cc -std=c11 -O2 -I lang/runtime -o transduce transduce.c \
// lang/runtime/el_runtime.c lang/runtime/el_seed.c \
// lang/runtime/engram_*.c -lcurl -lpthread -lm
// ./transduce # exits 0 only if every check passes
//
// (A `test "..."` form of the same checks lives in
// lang/tests/native/test_transduce.el, for when the native harness is
// repaired the shipped elc currently emits calls to __el_reg_count and
// friends without emitting their definitions, which breaks every native test
// equally, test_math.el included. Verified 2026-08-16, unrelated to this work.)
//
// WHY THIS EXISTS. Until 2026-08-16 no El ingest path could carry a vector:
// nodes took text, and geometry was DERIVED from that text. Text was the
// mandatory entry medium, so any non-text modality had to be DESCRIBED in
// prose first and the geometry we reasoned over was the geometry OF THE
// DESCRIPTION, not of the signal. Two things fix that, and both are shown
// below: geometry is a VALUE that carries its own width, and a REALIZER is an
// ordinary El function so admitting a new modality never requires a runtime
// patch.
//
// COMPARISON DISCIPLINE (measured, not stylistic): elc lowers `a == b`
// numerically only when both operand NAMES are in the per-function int-name
// set that `let x: Int` populates. A bare `f(x) == 0` is not a registered
// name and lowers to str_eq strcmp on two integers as pointers. `<` and `>`
// lower directly with no inference, so truthiness is written `> 0` / `< 1`.
// A realizer, written entirely in El
// Not in the runtime. Not known to the compiler. Registered by NAME and
// dispatched to through transduce(). That is the whole claim.
fn tone_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(4)
let n: Int = str_len(signal)
let a: Int = geometry_set(g, 0, int_to_float(n))
let b: Int = geometry_set(g, 1, int_to_float(n * 2))
let c: Int = geometry_set(g, 2, int_to_float(n * 3))
let d: Int = geometry_set(g, 3, int_to_float(n * 4))
g
}
// A second modality, to show the registry keys on modality rather than just
// returning whatever was registered last.
fn pulse_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(2)
let a: Int = geometry_set(g, 0, 1.0)
let b: Int = geometry_set(g, 1, 0.0)
g
}
// A deliberately BROKEN realizer: returns something that is not a Geometry.
fn bogus_realizer(signal: String) -> Geometry {
return 12345
}
// Fails FAST rather than accumulating a count, for a measured reason: a first
// cut wrote `let fails: Int = fails + check(...)` and `+` lowered to STRING
// CONCAT, because elc dispatches `+` on whether both operands are known-Int and
// a user-defined fn call is not so the counter printed 4343632752, a pointer.
// Nothing was wrong with the checks; the tally was lying. Exiting at the first
// failure needs no arithmetic at all, so there is nothing left to get wrong.
fn check(ok: Int, label: String) -> Int {
if ok > 0 {
println(" ok " + label)
return 0
}
println(" FAIL " + label)
exit(1)
return 1
}
fn near(a: Float, b: Float) -> Int {
let d: Float = a - b
if d > 0.001 { return 0 }
if d < -0.001 { return 0 }
return 1
}
fn eq_int(a: Int, b: Int) -> Int {
if a == b { return 1 }
return 0
}
fn main() -> Void {
println("geometry is a value that carries its own width")
let g8: Geometry = geometry_new(8)
let _c: Int = check(geometry_is(g8), "geometry_new returns a live Geometry")
let d8: Int = geometry_dim(g8)
let _c: Int = check(eq_int(d8, 8), "a Geometry carries its own width (8)")
let _c: Int = check(geometry_free(g8), "geometry_free reports what it did")
println("nonsense is refused — with no arbitrary max-dim bound")
// #141 needed `dim <= 8192` only to bound an allocation sized from a
// caller's CLAIM about a string's length. A value that carries its own
// width has nothing left to validate.
let z: Geometry = geometry_new(0)
let zi: Int = geometry_is(z)
let _c: Int = check(1 - zi, "dim 0 is not a geometry")
let ng: Geometry = geometry_new(-4)
let ngi: Int = geometry_is(ng)
let _c: Int = check(1 - ngi, "negative dim is not a geometry")
let nd: Int = geometry_dim(0)
let _c: Int = check(1 - nd, "geometry_dim of a non-geometry is 0, not a crash")
let nf: Int = geometry_free(0)
let _c: Int = check(1 - nf, "geometry_free of a non-geometry is a no-op")
println("components round-trip, and out-of-range is refused")
let g3: Geometry = geometry_new(3)
let s0: Int = geometry_set(g3, 0, 1.5)
let s1: Int = geometry_set(g3, 1, -2.5)
let _c: Int = check(s0, "set in range succeeds")
let oob: Int = geometry_set(g3, 3, 9.0)
let _c: Int = check(1 - oob, "set out of range is refused, not silently dropped")
let _c: Int = check(near(geometry_get(g3, 0), 1.5), "component 0 round-trips")
let _c: Int = check(near(geometry_get(g3, 1), -2.5), "component 1 round-trips (negative)")
let ff3: Int = geometry_free(g3)
println("hex is an EDGE adapter, and derives its own width")
// little-endian float32: 1.0 = 0000803f, 2.0 = 00000040
let gh: Geometry = geometry_from_f32le_hex("0000803f00000040")
let _c: Int = check(geometry_is(gh), "valid hex decodes to a Geometry")
let dh: Int = geometry_dim(gh)
let _c: Int = check(eq_int(dh, 2), "width DERIVED from input, never supplied")
let _c: Int = check(near(geometry_get(gh, 0), 1.0), "first component decoded")
let _c: Int = check(near(geometry_get(gh, 1), 2.0), "second component decoded")
let back: String = geometry_to_f32le_hex(gh)
let _c: Int = check(str_eq(back, "0000803f00000040"), "hex round-trips exactly")
let ffh: Int = geometry_free(gh)
println("malformed hex is refused")
let he: Geometry = geometry_from_f32le_hex("")
let hei: Int = geometry_is(he)
let _c: Int = check(1 - hei, "empty hex is not a geometry")
let hr: Geometry = geometry_from_f32le_hex("0000803f0000")
let hri: Int = geometry_is(hr)
let _c: Int = check(1 - hri, "length not a multiple of 8 is refused")
let hn: Geometry = geometry_from_f32le_hex("zzzzzzzz")
let hni: Int = geometry_is(hn)
let _c: Int = check(1 - hni, "non-hex characters are refused")
println("a realizer declared in El is a first-class realizer")
let reg: Int = realizer_register("tone", "tone_realizer")
let _c: Int = check(reg, "an El fn registers as a realizer BY NAME")
let _c: Int = check(realizer_has("tone"), "the modality now has an organ")
let gt: Geometry = transduce("aaa", "tone")
let _c: Int = check(geometry_is(gt), "transduce returns real geometry")
let dt: Int = geometry_dim(gt)
let _c: Int = check(eq_int(dt, 4), "the El realizer determined the width, not the runtime")
// str_len("aaa") == 3, so component 0 must be 3.0 proof the signal
// actually reached the El function rather than a stub answering for it.
let _c: Int = check(near(geometry_get(gt, 0), 3.0), "the signal REACHED the El realizer")
let fft: Int = geometry_free(gt)
println("distinct signals transduce to distinct geometry")
let g1: Geometry = transduce("aa", "tone")
let g2: Geometry = transduce("aaaaa", "tone")
let a1: Float = geometry_get(g1, 0)
let a2: Float = geometry_get(g2, 0)
// 5 - 2 = 3. If transduction were a stub these would be equal.
let _c: Int = check(near(a2 - a1, 3.0), "different signals produce different geometry")
let ff1: Int = geometry_free(g1)
let ff2: Int = geometry_free(g2)
println("the registry keys on modality")
let r2: Int = realizer_register("pulse", "pulse_realizer")
let _c: Int = check(r2, "a second modality registers independently")
let mt: Geometry = transduce("aaa", "tone")
let mp: Geometry = transduce("aaa", "pulse")
let mdt: Int = geometry_dim(mt)
let mdp: Int = geometry_dim(mp)
let _c: Int = check(eq_int(mdt, 4), "tone still routes to its own realizer")
let _c: Int = check(eq_int(mdp, 2), "pulse routes to a different realizer")
let ffm1: Int = geometry_free(mt)
let ffm2: Int = geometry_free(mp)
println("no organ is reported as no organ")
// A modality with no realizer must transduce to NOTHING. It must never
// fall back to embedding a description of the signal and calling that
// perception that silent substitution is the defect this all exists to end.
let eh: Int = realizer_has("echolocation")
let _c: Int = check(1 - eh, "unregistered modality has no organ")
let ge: Geometry = transduce("anything", "echolocation")
let gei: Int = geometry_is(ge)
let _c: Int = check(1 - gei, "no realizer means NO geometry, not fake geometry")
println("an unresolvable realizer name fails at WIRING time")
let bad: Int = realizer_register("ghost", "no_such_function_anywhere")
let _c: Int = check(1 - bad, "unresolvable realizer name is a registration failure")
let gh2: Int = realizer_has("ghost")
let _c: Int = check(1 - gh2, "and nothing gets registered")
println("a realizer returning non-geometry transduces nothing")
let rb: Int = realizer_register("bogus", "bogus_realizer")
let _c: Int = check(rb, "the symbol resolves, so registration succeeds")
let gb: Geometry = transduce("x", "bogus")
let gbi: Int = geometry_is(gb)
let _c: Int = check(1 - gbi, "contract enforced at the boundary: nothing handed back")
println("norm lets a caller check a realizer emitted signal, not zeros")
let gn: Geometry = geometry_new(2)
let _c: Int = check(near(geometry_norm(gn), 0.0), "a fresh geometry is zero — norm says so")
let n0: Int = geometry_set(gn, 0, 3.0)
let n1: Int = geometry_set(gn, 1, 4.0)
let _c: Int = check(near(geometry_norm(gn), 5.0), "3-4-5: norm is 5")
let ffn: Int = geometry_free(gn)
// Reaching here means nothing called exit(1) along the way.
println("")
println("all checks passed")
}
+372 -54
View File
@@ -5959,6 +5959,308 @@ void el_cgi_init(el_val_t name, el_val_t dharma_id, el_val_t principal,
} }
/* ── Geometry: signal as a first-class el value ──────────────────────────────
*
* WHY THIS IS IN THE LANGUAGE, AND WHY IT IS DEFINED HERE (2026-08-16).
*
* Until yesterday no El ingest path could carry a vector. Nodes took text,
* and geometry was DERIVED from that text by engram_embed_backfill. Text was
* therefore the mandatory entry medium: any non-text modality audio, image,
* sensor had to be DESCRIBED in prose first, so the geometry we then
* reasoned over was the geometry OF THE DESCRIPTION, not of the signal. That
* is faking it. The architecture is: geometry in, always; we do not fake it,
* we project.
*
* The first fix (#141, engram_node_set_emb) proved the path end to end but
* placed it wrong in three ways, each of which this section corrects:
*
* 1. It sat at the CONSUMER. Transduction is a LANGUAGE concern every El
* program touching any modality needs it, not just the one that happens
* to hold a graph. So this section is defined HERE, immediately above
* the engram block, and depends on nothing inside it. The engram is a
* client of this surface, not its owner. That ordering is the point:
* you can delete the entire engram and geometry still enters El.
*
* 2. It marshalled the vector as a hex STRING, because El had no
* first-class geometry value which reintroduced text as the TRANSPORT
* medium one layer below the problem being fixed. Geometry is now a
* value. Hex survives only as a wire ADAPTER at the edge
* (geometry_from/to_f32le_hex), which is all an encoding should ever be.
*
* 3. It needed an arbitrary `dim <= 8192` bound, purely to check a
* caller-supplied dim against a string's length before allocating. A
* real geometry value CARRIES its own width, so here the width is
* derived and never asserted, and there is nothing left to validate.
* The bound is gone rather than merely raised the only thing that can
* fail is the allocation itself, which is an honest failure.
*
* REPRESENTATION: magic-tagged heap object (see "Refcounted heap objects"),
* carried in an el_val_t. The payload is a separate allocation so the header
* never moves. The magic word is >= 0x80 in its MSB so the string/small-int
* sniffing in looks_like_heap_obj can never confuse a Geometry for either.
*
* OWNERSHIP: a Geometry is owned by the El caller and released with
* geometry_free. node_attach_geometry COPIES its payload into the node, so a
* node and the caller's value have independent lifetimes and freeing one
* never touches the other. Geometry deliberately does NOT participate in
* el_retain/el_release: the shipped elc emits neither on let-bindings
* (measured), so hooking it there would be dead code that could only ever
* free a live vector early.
*/
#define EL_MAGIC_GEOM 0xE1608E01u
typedef struct {
ElHeader hdr;
int32_t dim;
float* v;
} ElGeometry;
/* Resolve an el_val_t to a live Geometry, or NULL. Every accessor goes
* through this, so a stale/foreign/zero value is a clean 0-return rather
* than a dereference. */
static ElGeometry* geom_of(el_val_t g) {
if (!looks_like_heap_obj(g)) return NULL;
ElGeometry* p = (ElGeometry*)(uintptr_t)g;
if (p->hdr.magic != EL_MAGIC_GEOM) return NULL;
return p;
}
el_val_t geometry_new(el_val_t dim) {
int32_t d = (int32_t)(int64_t)dim;
if (d <= 0) return (el_val_t)0;
ElGeometry* g = (ElGeometry*)malloc(sizeof(ElGeometry));
if (!g) return (el_val_t)0;
g->v = (float*)calloc((size_t)d, sizeof(float));
if (!g->v) { free(g); return (el_val_t)0; }
g->hdr.magic = EL_MAGIC_GEOM;
g->hdr.refcount = 1;
g->dim = d;
return (el_val_t)(uintptr_t)g;
}
el_val_t geometry_dim(el_val_t g) {
ElGeometry* p = geom_of(g);
return p ? (el_val_t)p->dim : (el_val_t)0;
}
el_val_t geometry_is(el_val_t g) {
return geom_of(g) ? (el_val_t)1 : (el_val_t)0;
}
el_val_t geometry_get(el_val_t g, el_val_t i) {
ElGeometry* p = geom_of(g);
int64_t k = (int64_t)i;
if (!p || k < 0 || k >= (int64_t)p->dim) return el_from_float(0.0);
return el_from_float((double)p->v[k]);
}
el_val_t geometry_set(el_val_t g, el_val_t i, el_val_t x) {
ElGeometry* p = geom_of(g);
int64_t k = (int64_t)i;
if (!p || k < 0 || k >= (int64_t)p->dim) return (el_val_t)0;
p->v[k] = (float)el_to_float(x);
return (el_val_t)1;
}
el_val_t geometry_norm(el_val_t g) {
ElGeometry* p = geom_of(g);
if (!p) return el_from_float(0.0);
double s = 0.0;
for (int32_t i = 0; i < p->dim; i++) s += (double)p->v[i] * (double)p->v[i];
return el_from_float(sqrt(s));
}
el_val_t geometry_free(el_val_t g) {
ElGeometry* p = geom_of(g);
if (!p) return (el_val_t)0;
free(p->v);
p->hdr.magic = 0; /* poison so use-after-free is detected, as List/Map do */
free(p);
return (el_val_t)1;
}
/* geometry_from_f32le_hex — decode little-endian float32 hex INTO geometry.
*
* This is the ONE place hex appears, and it appears as what it actually is:
* an encoding at the boundary, not the medium El reasons in. The width is
* DERIVED from the input length (8 hex chars per float32) and never supplied
* by the caller which is precisely why #141's arbitrary `dim <= 8192`
* bound has no counterpart here. There is nothing to validate.
*
* Returns 0 on empty input, a length that is not a multiple of 8, or any
* non-hex character. */
el_val_t geometry_from_f32le_hex(el_val_t hex) {
const char* s = EL_CSTR(hex);
if (!s) return (el_val_t)0;
size_t n = strlen(s);
if (n == 0 || (n % 8u) != 0) return (el_val_t)0;
size_t d = n / 8u;
if (d > (size_t)INT32_MAX) return (el_val_t)0;
el_val_t gv = geometry_new((el_val_t)(int64_t)d);
ElGeometry* g = geom_of(gv);
if (!g) return (el_val_t)0;
for (size_t i = 0; i < d; i++) {
uint32_t w = 0;
for (int k = 0; k < 8; k++) {
char c = s[i * 8u + (size_t)k];
uint32_t nib;
if (c >= '0' && c <= '9') nib = (uint32_t)(c - '0');
else if (c >= 'a' && c <= 'f') nib = (uint32_t)(c - 'a' + 10);
else if (c >= 'A' && c <= 'F') nib = (uint32_t)(c - 'A' + 10);
else { geometry_free(gv); return (el_val_t)0; }
w = (w << 4) | nib;
}
/* Hex is emitted little-endian byte order; rebuild the word. */
uint32_t le = ((w & 0x000000FFu) << 24) | ((w & 0x0000FF00u) << 8) |
((w & 0x00FF0000u) >> 8) | ((w & 0xFF000000u) >> 24);
float f;
memcpy(&f, &le, sizeof(f));
g->v[i] = f;
}
return gv;
}
/* geometry_to_f32le_hex — the egress adapter, exact inverse of the above.
* Present so a program that must hand geometry to a non-El peer over a text
* wire can do so explicitly, at the edge, instead of the language pretending
* text was the medium all along. */
el_val_t geometry_to_f32le_hex(el_val_t g) {
ElGeometry* p = geom_of(g);
if (!p) return EL_STR("");
static const char* HEXD = "0123456789abcdef";
size_t n = (size_t)p->dim * 8u;
char* out = el_strbuf(n); /* arena-tracked; allocates n+1, exits on OOM */
for (int32_t i = 0; i < p->dim; i++) {
uint32_t w;
memcpy(&w, &p->v[i], sizeof(w));
/* Emit little-endian byte order: low byte first. */
for (int b = 0; b < 4; b++) {
uint32_t byte = (w >> (8 * b)) & 0xFFu;
out[(size_t)i * 8u + (size_t)b * 2u] = HEXD[(byte >> 4) & 0xF];
out[(size_t)i * 8u + (size_t)b * 2u + 1] = HEXD[byte & 0xF];
}
}
out[n] = '\0';
return (el_val_t)(uintptr_t)out;
}
/* ── Realizers: transduction declared in El, not patched into the runtime ────
*
* A REALIZER maps one modality into geometry. The whole reason transduction
* belongs in the language is that ADDING A MODALITY MUST NOT REQUIRE A
* RUNTIME PATCH otherwise "the realizers are in the engram" just becomes
* "the realizers are in the runtime" and nothing has actually moved. So
* realizers are declared in El and registered by NAME:
*
* fn tone_realizer(signal: String) -> Geometry {
* let g: Geometry = geometry_new(8)
* ... geometry_set(g, i, x) ...
* g
* }
*
* realizer_register("tone", "tone_realizer")
* let g: Geometry = transduce(sample, "tone")
*
* The namesymbol step rides the identical, already load-bearing mechanism
* http_set_handler uses (see "HTTP server"): every El `fn name(...)` compiles
* to a global C symbol with that exact name, so dlsym(RTLD_DEFAULT, name)
* against the running binary resolves an El-defined function. No codegen
* change, no first-class function references, no runtime edit per modality.
* A realizer written in El is a first-class realizer.
*
* A realizer may equally be a C symbol linked into the program; the registry
* cannot tell the difference and has no reason to care.
*/
typedef el_val_t (*el_realizer_fn)(el_val_t);
typedef struct {
char* modality;
el_realizer_fn fn;
} ElRealizer;
static ElRealizer _realizers[64];
static size_t _realizer_count = 0;
static pthread_mutex_t _realizer_mu = PTHREAD_MUTEX_INITIALIZER;
static el_realizer_fn realizer_lookup(const char* m) {
el_realizer_fn out = NULL;
pthread_mutex_lock(&_realizer_mu);
for (size_t i = 0; i < _realizer_count; i++) {
if (strcmp(_realizers[i].modality, m) == 0) { out = _realizers[i].fn; break; }
}
pthread_mutex_unlock(&_realizer_mu);
return out;
}
el_val_t realizer_register(el_val_t modality, el_val_t fn_name) {
const char* m = EL_CSTR(modality);
const char* fn = EL_CSTR(fn_name);
if (!m || !*m || !fn || !*fn) return (el_val_t)0;
/* An unresolvable name is a REGISTRATION FAILURE, reported as 0 — not a
* silent no-op that only surfaces later as "this modality produces
* nothing". Distinguishing "no organ" from "broken organ" at the moment
* of wiring is the lesson #141 was written to enforce. */
void* sym = dlsym(RTLD_DEFAULT, fn);
if (!sym) return (el_val_t)0;
pthread_mutex_lock(&_realizer_mu);
for (size_t i = 0; i < _realizer_count; i++) {
if (strcmp(_realizers[i].modality, m) == 0) {
_realizers[i].fn = (el_realizer_fn)sym; /* re-registration replaces */
pthread_mutex_unlock(&_realizer_mu);
return (el_val_t)1;
}
}
if (_realizer_count < sizeof(_realizers) / sizeof(_realizers[0])) {
/* _persist, NOT el_strdup: the registry outlives any request, and an
* arena-tracked copy would be freed at el_request_end leaving a
* dangling modality name if a program registers a realizer from
* inside a handler rather than at startup. */
_realizers[_realizer_count].modality = el_strdup_persist(m);
_realizers[_realizer_count].fn = (el_realizer_fn)sym;
_realizer_count++;
pthread_mutex_unlock(&_realizer_mu);
return (el_val_t)1;
}
pthread_mutex_unlock(&_realizer_mu);
return (el_val_t)0;
}
el_val_t realizer_has(el_val_t modality) {
const char* m = EL_CSTR(modality);
if (!m || !*m) return (el_val_t)0;
return realizer_lookup(m) ? (el_val_t)1 : (el_val_t)0;
}
/* transduce — THE primitive: signal in, geometry out.
*
* Dispatches to the realizer registered for `modality`. Returns 0 (not a
* Geometry) when no realizer is registered, and geometry_is() on the result
* is the check.
*
* There is deliberately NO built-in realizer, not even for text. A modality
* the program has declared no organ for is one it genuinely cannot sense,
* and returning nothing is more honest than quietly embedding a description
* of the signal and calling that perception which is the exact failure
* this whole change exists to end.
*
* The result is validated to actually BE a Geometry before it is handed
* back, so a realizer that returns something else transduced nothing rather
* than handing a caller a value that will misbehave far from here. */
el_val_t transduce(el_val_t signal, el_val_t modality) {
const char* m = EL_CSTR(modality);
if (!m || !*m) return (el_val_t)0;
el_realizer_fn fn = realizer_lookup(m);
if (!fn) return (el_val_t)0;
el_val_t g = fn(signal);
return geom_of(g) ? g : (el_val_t)0;
}
/* ── Batch 3: Engram in-process graph store ──────────────────────────────── */ /* ── Batch 3: Engram in-process graph store ──────────────────────────────── */
/* /*
* Single global EngramStore allocated lazily on first call. All node and * Single global EngramStore allocated lazily on first call. All node and
@@ -8563,80 +8865,96 @@ el_val_t engram_node_count(void) {
return (el_val_t)engram_get()->node_count; return (el_val_t)engram_get()->node_count;
} }
/* engram_node_set_emb — attach GEOMETRY to an existing node. /* node_attach_geometry — a node acquires geometry.
* *
* WHY THIS EXISTS (2026-08-16). Until now no ingest path could carry a * Named for the operation, not for the store that happens to hold the node.
* vector. engram_node / engram_node_full / engram_node_layered take text * This is the geometry-valued ingest path that replaces #141's hex-string
* only, and the sole way a node acquired an embedding was * one: nothing here parses text, and nothing here takes a caller's word for
* engram_embed_backfill DERIVING one from n->content. That made text the * how wide the vector is. The Geometry carries its own width.
* mandatory entry medium: any non-text modality (audio, image, sensor)
* had to be described in prose first, and the geometry we then reasoned
* over was the geometry OF THE DESCRIPTION, not of the signal. Measured
* consequence: POST /api/nodes accepted an "emb" field, returned 200 with
* a fresh id, and stored emb_dim=None / embedded:false the vector was
* silently discarded because no parameter existed to receive it.
* *
* `hex` is little-endian float32, the encoding the perception vessel's * The payload is COPIED into the node, so the node and the caller's Geometry
* /voice/embed already emits, so a realizer's output moves in without a * have independent lifetimes the caller may geometry_free() immediately
* JSON float-array round trip. Length must be exactly dim*8 hex chars. * after, and a later free of the node's emb never touches the El value.
* *
* DIMENSION POLICY: dim need NOT equal the canonical text-embedding dim. * DIMENSION POLICY (measured in #141, load-bearing do not regress): dim
* A modality vector of a different width is stored and is simply not * need NOT equal the canonical text-embedding width. An off-dimension vector
* inserted into the resident HNSW index, whose build loop already filters * is stored and is simply not inserted into the resident HNSW index, whose
* on `n->emb_dim == dim`. So off-dimension geometry is durable and * build loop already filters on `n->emb_dim == dim`. So a 64-dim voice
* addressable without perturbing the canonical index. * geometry is durable and addressable without perturbing the 768-dim
* canonical index.
* *
* Setting emb also makes the node ineligible for embed_backfill (which * Attaching geometry also makes the node ineligible for embed_backfill
* only fills nodes with no emb), so a realizer's vector is never * (which fills only nodes with no emb), so a realizer's vector is never
* overwritten by a text-derived one. * overwritten by a text-derived one.
* *
* Returns 1 on success, 0 on unknown id / malformed hex / bad dim. */ * Returns 1 on success, 0 on unknown id or a value that is not a Geometry. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim) { el_val_t node_attach_geometry(el_val_t node_id, el_val_t g) {
const char* sid = EL_CSTR(id); const char* sid = EL_CSTR(node_id);
const char* sh = EL_CSTR(hex); if (!sid || !*sid) return (el_val_t)0;
int32_t d = (int32_t)(int64_t)dim;
/* Bound the allocation. No max-dim constant existed because no caller
* could supply a dim before this function; 8192 is generous for any
* realizer (canonical text embeddings are 768, MFCC voice stats 64)
* while keeping a malformed `dim` from requesting an unbounded malloc. */
if (!sid || !*sid || !sh || d <= 0 || d > 8192) return (el_val_t)0;
size_t need = (size_t)d * 8u; /* 4 bytes → 8 hex chars per float */ ElGeometry* p = geom_of(g);
if (strlen(sh) != need) return (el_val_t)0; if (!p || p->dim <= 0) return (el_val_t)0;
EngramNode* n = engram_find_node(sid); EngramNode* n = engram_find_node(sid);
if (!n) return (el_val_t)0; if (!n) return (el_val_t)0;
float* v = (float*)malloc(sizeof(float) * (size_t)d); float* v = (float*)malloc(sizeof(float) * (size_t)p->dim);
if (!v) return (el_val_t)0; if (!v) return (el_val_t)0;
memcpy(v, p->v, sizeof(float) * (size_t)p->dim);
for (int32_t i = 0; i < d; i++) {
uint32_t w = 0;
for (int k = 0; k < 8; k++) {
char c = sh[(size_t)i * 8u + (size_t)k];
uint32_t nib;
if (c >= '0' && c <= '9') nib = (uint32_t)(c - '0');
else if (c >= 'a' && c <= 'f') nib = (uint32_t)(c - 'a' + 10);
else if (c >= 'A' && c <= 'F') nib = (uint32_t)(c - 'A' + 10);
else { free(v); return (el_val_t)0; }
w = (w << 4) | nib;
}
/* Hex is emitted little-endian byte order; rebuild the word. */
uint32_t le = ((w & 0x000000FFu) << 24) | ((w & 0x0000FF00u) << 8) |
((w & 0x00FF0000u) >> 8) | ((w & 0xFF000000u) >> 24);
float f;
memcpy(&f, &le, sizeof(f));
v[i] = f;
}
free(n->emb); free(n->emb);
n->emb = v; n->emb = v;
n->emb_dim = d; n->emb_dim = p->dim;
n->updated_at = engram_now_ms(); n->updated_at = engram_now_ms();
if (engram_store_enabled()) eg_store_put_node(n); if (engram_store_enabled()) eg_store_put_node(n);
return (el_val_t)1; return (el_val_t)1;
} }
/* node_geometry_dim — read the attached width back, 0 if the node carries
* none. Exists so an attach is VERIFIED by reading it back rather than by
* trusting a success return. That is not a nicety: #141 was misdiagnosed for
* an hour precisely because a genuine ingest drop and a mere reporting gap
* were indistinguishable from the outside. */
el_val_t node_geometry_dim(el_val_t node_id) {
const char* sid = EL_CSTR(node_id);
if (!sid || !*sid) return (el_val_t)0;
EngramNode* n = engram_find_node(sid);
if (!n || !n->emb) return (el_val_t)0;
return (el_val_t)n->emb_dim;
}
/* engram_node_set_emb — DEPRECATED. Shipped in #141; superseded 2026-08-16
* by geometry_from_f32le_hex + node_attach_geometry, and now implemented as
* literally that.
*
* It is kept, rather than removed, for one reason only: the runtime is
* published as an SDK asset, so a downstream binary may already be linking
* this symbol. It is NOT kept because a hex string is an acceptable way to
* move geometry between two pieces of El it isn't, and that was the
* placement defect. New code calls transduce() or geometry_from_f32le_hex()
* plus node_attach_geometry().
*
* The #141 contract is preserved exactly, including its negative cases, so
* this remains a drop-in: `dim` <= 0 rejects, malformed hex rejects, and a
* `dim` that disagrees with the vector's actual width rejects. The
* difference is that `dim` is now an ASSERTION checked against a width the
* Geometry already knows, rather than the authority the allocation trusted
* which is why #141's arbitrary `dim <= 8192` guard has no counterpart here.
* There is no longer an unbounded-malloc hazard to guard against. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim) {
int32_t want = (int32_t)(int64_t)dim;
if (want <= 0) return (el_val_t)0;
el_val_t gv = geometry_from_f32le_hex(hex);
ElGeometry* p = geom_of(gv);
if (!p) return (el_val_t)0; /* empty / malformed hex */
if (p->dim != want) { geometry_free(gv); return (el_val_t)0; } /* length mismatch */
el_val_t ok = node_attach_geometry(id, gv);
geometry_free(gv);
return ok;
}
/* ── Telemetry retention ──────────────────────────────────────────────────── /* ── Telemetry retention ────────────────────────────────────────────────────
* (2026-07-16 self-review) InternalStateEvent nodes are append-only telemetry * (2026-07-16 self-review) InternalStateEvent nodes are append-only telemetry
* (heartbeat, curiosity_scan, engram_sync) written ~3/min by the awareness * (heartbeat, curiosity_scan, engram_sync) written ~3/min by the awareness
+70 -4
View File
@@ -586,6 +586,60 @@ void el_runtime_dharma_event_arrive(const char* event_type,
const char* payload, const char* payload,
const char* source); const char* source);
/* ── Geometry: signal as a first-class El value ──────────────────────────────
*
* A Geometry is an opaque, magic-tagged heap value carried in an el_val_t —
* the same discipline as List/Map. It holds a width and a float32 payload,
* and it is the medium a non-text modality enters in. Declared HERE, above
* the engram block, because transduction is a LANGUAGE concern: every El
* program touching any modality needs it, and the engram is merely one El
* program that happens to hold a graph. See el_runtime.c ("Geometry: signal
* as a first-class el value") for the full rationale.
*
* El-side type annotation is simply `Geometry` — an opaque boxed pointer,
* exactly like Instant / Calendar / Rhythm. No codegen change is required.
*
* OWNERSHIP: a Geometry is owned by the El caller and released with
* geometry_free. node_attach_geometry COPIES, so a node and the caller's
* value have independent lifetimes. */
el_val_t geometry_new(el_val_t dim); /* zero-filled; 0 on failure */
el_val_t geometry_dim(el_val_t g); /* width, 0 if not a Geometry */
el_val_t geometry_is(el_val_t g); /* 1 if a live Geometry */
el_val_t geometry_get(el_val_t g, el_val_t i); /* Float component */
el_val_t geometry_set(el_val_t g, el_val_t i, el_val_t x); /* 1 ok / 0 out of range */
el_val_t geometry_norm(el_val_t g); /* Float L2 — lets a caller
* check a realizer emitted
* signal, not zeros */
el_val_t geometry_free(el_val_t g); /* 1 if freed, 0 if not a Geometry.
* Returns a value (not void) so it
* is safe in any El expression
* position without a codegen
* void-builtin table entry. */
/* Wire ADAPTERS — the only place an encoding appears, and only at the edge.
* `f32le hex` is little-endian float32, 8 hex chars per component: the
* encoding the perception vessel's /voice/embed already emits. The width is
* DERIVED from the input length, never supplied by a caller — which is why
* there is no max-dim constant here to validate a claimed length against. */
el_val_t geometry_from_f32le_hex(el_val_t hex); /* 0 on empty/odd-length/non-hex */
el_val_t geometry_to_f32le_hex(el_val_t g); /* "" if not a Geometry */
/* ── Realizers + transduce ───────────────────────────────────────────────────
* A REALIZER maps one modality into geometry. Registration is by NAME, so a
* new modality never requires a runtime patch: every El `fn name(...)`
* compiles to a global C symbol with that exact name, and the registry
* resolves it with dlsym against the running binary — the same mechanism
* http_set_handler already relies on.
*
* fn tone_realizer(signal: String) -> Geometry { ... }
* realizer_register("tone", "tone_realizer")
* let g: Geometry = transduce(sample, "tone")
*/
el_val_t realizer_register(el_val_t modality, el_val_t fn_name); /* 1 ok / 0 unresolved */
el_val_t realizer_has(el_val_t modality); /* 1 if a realizer is registered */
el_val_t transduce(el_val_t signal, el_val_t modality); /* Geometry, or 0 if no organ */
/* ── Engram local graph primitives ─────────────────────────────────────────── /* ── Engram local graph primitives ───────────────────────────────────────────
* Operate on the CGI's local Engram knowledge graph. * Operate on the CGI's local Engram knowledge graph.
* `engram_activate` queries the local graph only; `dharma_activate` is * `engram_activate` queries the local graph only; `dharma_activate` is
@@ -613,10 +667,22 @@ void engram_strengthen(el_val_t node_id);
void engram_forget(el_val_t node_id); void engram_forget(el_val_t node_id);
el_val_t engram_prune_telemetry(el_val_t older_than_ms); el_val_t engram_prune_telemetry(el_val_t older_than_ms);
el_val_t engram_node_count(void); el_val_t engram_node_count(void);
/* Attach geometry to an existing node. `hex` is little-endian float32, /* Attach a Geometry to an existing node, and read the attached width back.
* exactly dim*8 hex chars — the encoding realizers already emit. Lets a * Named for the operation, not the store: a node acquires geometry. This is
* non-text modality enter as geometry instead of being described in prose * the geometry-valued ingest path — nothing about it is hex, and nothing
* and embedded as its description. Returns 1 on success, 0 otherwise. */ * about it assumes the caller's vector matches the canonical text-embedding
* width. node_geometry_dim exists so an attach is VERIFIED by reading it
* back rather than by trusting a success return. */
el_val_t node_attach_geometry(el_val_t node_id, el_val_t g); /* 1 ok / 0 otherwise */
el_val_t node_geometry_dim(el_val_t node_id); /* width, 0 if none */
/* DEPRECATED (shipped in #141, superseded 2026-08-16). Equivalent to
* geometry_from_f32le_hex + node_attach_geometry, and now implemented as
* exactly that. Kept only so anything built against the #141 runtime keeps
* linking; `dim` is accepted but treated as an assertion about the vector's
* width rather than as its source. New code should not call this — a hex
* string is a wire encoding, not a way to move geometry between two pieces
* of El. Returns 1 on success, 0 otherwise. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim); el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim);
el_val_t engram_search(el_val_t query, el_val_t limit); el_val_t engram_search(el_val_t query, el_val_t limit);
el_val_t engram_scan_nodes(el_val_t limit, el_val_t offset); el_val_t engram_scan_nodes(el_val_t limit, el_val_t offset);
+234
View File
@@ -0,0 +1,234 @@
import "../../runtime/eltest.el"
// test_transduce.el geometry as a first-class El value, and realizers
// declared in El rather than patched into the runtime.
//
// WHAT IS ACTUALLY UNDER TEST. Until 2026-08-16 no El ingest path could carry
// a vector: nodes took text, and geometry was DERIVED from that text. Text was
// therefore the mandatory entry medium, so any non-text modality had to be
// DESCRIBED in prose first and the geometry we reasoned over was the geometry
// OF THE DESCRIPTION, not of the signal. The fix has two halves, and this file
// exercises both:
//
// 1. Geometry is a VALUE it carries its own width, so nothing has to
// assert a width against a string's length.
// 2. A REALIZER is an ordinary El function. `tone_realizer` below is not in
// the runtime, is not known to the compiler, and is not special in any
// way; it is registered BY NAME and dispatched to through transduce().
// That is the load-bearing claim: adding a modality must not require a
// runtime patch, or nothing has actually moved into the language.
//
// COMPARISON DISCIPLINE IN THIS FILE (measured 2026-08-16, not stylistic):
// elc lowers `a == b` to a NUMERIC comparison only when both operand names are
// in the per-function int-name set, which `let x: Int` populates. A bare call
// like `geometry_is(g) == 0` is not a registered name, so it lowers to
// `str_eq(...)` strcmp on two integers reinterpreted as pointers. `<` and `>`
// lower directly via binop_to_c with no type inference at all, so truthiness is
// written `> 0` / `< 1` here, and any exact `==` is done on a value first bound
// through `let x: Int`.
// A realizer, written entirely in El
// Maps a "tone" signal into a 4-component geometry. Deliberately trivial
// what is being proven is that an El function can BE a realizer, not that
// this is good acoustics. The one real property it has: distinct signals
// produce distinct geometry, so the test can tell transduction from a stub.
fn tone_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(4)
let n: Int = str_len(signal)
let a: Int = geometry_set(g, 0, int_to_float(n))
let b: Int = geometry_set(g, 1, int_to_float(n * 2))
let c: Int = geometry_set(g, 2, int_to_float(n * 3))
let d: Int = geometry_set(g, 3, int_to_float(n * 4))
g
}
// A second realizer for a different modality, to prove the registry keys on
// modality and does not just hand back "the last thing registered".
fn pulse_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(2)
let a: Int = geometry_set(g, 0, 1.0)
let b: Int = geometry_set(g, 1, 0.0)
g
}
// A deliberately BROKEN realizer: it returns something that is not a Geometry.
// transduce() must not hand this back to a caller as if it were one.
fn bogus_realizer(signal: String) -> Geometry {
return 12345
}
test "geometry-is-a-value-with-its-own-width" {
let g: Geometry = geometry_new(8)
let live: Int = geometry_is(g)
assert live > 0, "geometry_new returns a live Geometry"
let d: Int = geometry_dim(g)
assert d == 8, "a Geometry carries its own width"
let freed: Int = geometry_free(g)
assert freed > 0, "geometry_free reports what it did"
}
test "geometry-rejects-nonsense-without-an-arbitrary-bound" {
// dim <= 0 is not a width. Note there is deliberately no MAX dim here:
// #141 needed `dim <= 8192` only to bound an allocation sized from a
// caller's claim about a string. A value that carries its own width has
// nothing left to validate, so the only failure left is allocation.
let zero: Geometry = geometry_new(0)
let z: Int = geometry_is(zero)
assert z < 1, "dim 0 is not a geometry"
let neg: Geometry = geometry_new(-4)
let n: Int = geometry_is(neg)
assert n < 1, "negative dim is not a geometry"
// Accessors must be total: a non-geometry is 0-width, never a crash.
let nd: Int = geometry_dim(0)
assert nd < 1, "geometry_dim of a non-geometry is 0"
let ni: Int = geometry_is(0)
assert ni < 1, "geometry_is of a non-geometry is 0"
let nf: Int = geometry_free(0)
assert nf < 1, "geometry_free of a non-geometry is a no-op"
}
test "geometry-components-round-trip" {
let g: Geometry = geometry_new(3)
let s0: Int = geometry_set(g, 0, 1.5)
let s1: Int = geometry_set(g, 1, -2.5)
assert s0 > 0, "set in range succeeds"
let oob: Int = geometry_set(g, 3, 9.0)
assert oob < 1, "set out of range is refused, not silently dropped"
let v0: Float = geometry_get(g, 0)
let d0: Float = v0 - 1.5
assert d0 < 0.001, "component 0 round-trips"
assert d0 > -0.001, "component 0 round-trips"
let v1: Float = geometry_get(g, 1)
let d1: Float = v1 + 2.5
assert d1 < 0.001, "component 1 round-trips (negative)"
assert d1 > -0.001, "component 1 round-trips (negative)"
let freed: Int = geometry_free(g)
}
test "hex-is-an-edge-adapter-and-derives-its-own-width" {
// 2 components, little-endian float32: 1.0 = 0000803f, 2.0 = 00000040.
let g: Geometry = geometry_from_f32le_hex("0000803f00000040")
let live: Int = geometry_is(g)
assert live > 0, "valid hex decodes to a Geometry"
let d: Int = geometry_dim(g)
assert d == 2, "width is DERIVED from the input, never supplied"
let a: Float = geometry_get(g, 0)
let da: Float = a - 1.0
assert da < 0.001, "first component decoded"
assert da > -0.001, "first component decoded"
let b: Float = geometry_get(g, 1)
let db: Float = b - 2.0
assert db < 0.001, "second component decoded"
assert db > -0.001, "second component decoded"
// Egress adapter is the exact inverse.
let back: String = geometry_to_f32le_hex(g)
assert str_eq(back, "0000803f00000040"), "hex round-trips exactly"
let freed: Int = geometry_free(g)
}
test "hex-rejects-malformed-input" {
let empty: Geometry = geometry_from_f32le_hex("")
let e: Int = geometry_is(empty)
assert e < 1, "empty hex is not a geometry"
let ragged: Geometry = geometry_from_f32le_hex("0000803f0000")
let r: Int = geometry_is(ragged)
assert r < 1, "length not a multiple of 8 is refused"
let nonhex: Geometry = geometry_from_f32le_hex("zzzzzzzz")
let nh: Int = geometry_is(nonhex)
assert nh < 1, "non-hex characters are refused"
}
test "a-realizer-declared-in-el-is-a-first-class-realizer" {
// THE CLAIM: tone_realizer is an ordinary El function. It is not in the
// runtime and the compiler knows nothing about it. Registering it by name
// is enough to make it the organ for a modality.
let reg: Int = realizer_register("tone", "tone_realizer")
assert reg > 0, "an El fn registers as a realizer by name"
let has: Int = realizer_has("tone")
assert has > 0, "the modality now has an organ"
let g: Geometry = transduce("aaa", "tone")
let live: Int = geometry_is(g)
assert live > 0, "transduce returns real geometry"
let d: Int = geometry_dim(g)
assert d == 4, "the El realizer determined the width, not the runtime"
// str_len("aaa") == 3, so component 0 must be 3.0 proof the signal
// actually reached the El function rather than a stub answering for it.
let c0: Float = geometry_get(g, 0)
let dc: Float = c0 - 3.0
assert dc < 0.001, "the signal reached the El realizer"
assert dc > -0.001, "the signal reached the El realizer"
let freed: Int = geometry_free(g)
}
test "distinct-signals-transduce-to-distinct-geometry" {
let reg: Int = realizer_register("tone", "tone_realizer")
let g1: Geometry = transduce("aa", "tone")
let g2: Geometry = transduce("aaaaa", "tone")
let a: Float = geometry_get(g1, 0)
let b: Float = geometry_get(g2, 0)
let diff: Float = b - a
// 5 - 2 = 3. If transduction were a stub these would be equal.
assert diff > 2.9, "different signals produce different geometry"
assert diff < 3.1, "different signals produce different geometry"
let f1: Int = geometry_free(g1)
let f2: Int = geometry_free(g2)
}
test "the-registry-keys-on-modality" {
let r1: Int = realizer_register("tone", "tone_realizer")
let r2: Int = realizer_register("pulse", "pulse_realizer")
assert r2 > 0, "a second modality registers independently"
let gt: Geometry = transduce("aaa", "tone")
let gp: Geometry = transduce("aaa", "pulse")
let dt: Int = geometry_dim(gt)
let dp: Int = geometry_dim(gp)
assert dt == 4, "tone still routes to its own realizer"
assert dp == 2, "pulse routes to a different realizer"
let f1: Int = geometry_free(gt)
let f2: Int = geometry_free(gp)
}
test "no-organ-is-reported-as-no-organ" {
// A modality with no realizer must transduce to NOTHING. It must never
// fall back to embedding a description of the signal and calling that
// perception that silent substitution is the entire defect this change
// exists to end.
let has: Int = realizer_has("echolocation")
assert has < 1, "unregistered modality has no organ"
let g: Geometry = transduce("anything", "echolocation")
let live: Int = geometry_is(g)
assert live < 1, "no realizer means no geometry, not fake geometry"
}
test "registration-of-an-unresolvable-name-fails-loudly" {
// Reported at the moment of WIRING, not later as "this modality mysteriously
// produces nothing". Distinguishing "no organ" from "broken organ" is the
// lesson that made this whole change necessary.
let bad: Int = realizer_register("ghost", "no_such_function_anywhere")
assert bad < 1, "an unresolvable realizer name is a registration failure"
let has: Int = realizer_has("ghost")
assert has < 1, "and nothing gets registered"
}
test "a-realizer-returning-non-geometry-transduces-nothing" {
let reg: Int = realizer_register("bogus", "bogus_realizer")
assert reg > 0, "the symbol resolves, so registration succeeds"
// ...but the contract is enforced at the boundary, so the caller never
// receives a value that would misbehave far away from here.
let g: Geometry = transduce("x", "bogus")
let live: Int = geometry_is(g)
assert live < 1, "a non-Geometry return transduced nothing"
}
test "norm-lets-a-caller-check-a-realizer-emitted-signal" {
let g: Geometry = geometry_new(2)
let z: Float = geometry_norm(g)
assert z < 0.001, "a fresh geometry is zero — norm says so"
let s0: Int = geometry_set(g, 0, 3.0)
let s1: Int = geometry_set(g, 1, 4.0)
let n: Float = geometry_norm(g)
let dn: Float = n - 5.0
assert dn < 0.001, "3-4-5: norm is 5"
assert dn > -0.001, "3-4-5: norm is 5"
let freed: Int = geometry_free(g)
}