runtime: transduction is a language concern, so move it into the language
El SDK CI - dev / build-and-test (pull_request) Failing after 14m58s
El SDK CI - dev / build-and-test (pull_request) Failing after 14m58s
#141 let signal enter as geometry and it worked, but it was placed at the CONSUMER and said so in its own commit message. This is the correction. Three defects, all of them placement: 1. It sat in the engram. Ingest is a LANGUAGE concern — every el program touching any modality needs it, and the engram is merely one el program that happens to hold a graph. The geometry surface is now defined in el_runtime.c immediately ABOVE the engram section and depends on nothing inside it. Delete the entire engram and geometry still enters el. 2. It marshalled the vector as a hex STRING, because el had no first-class geometry value — which reintroduced text as the TRANSPORT medium one layer below the problem being fixed. Geometry is now an el value: a magic-tagged heap object carried in el_val_t, same discipline as List/Map. Hex survives only as an adapter at the edge, which is all an encoding should ever be. 3. It needed an arbitrary `dim <= 8192` bound purely to size an allocation from a caller's CLAIM about a string's length. A value carries its own width, so the width is derived and never asserted. The bound is gone, not raised — there is nothing left to validate. Language surface, none of it engram-prefixed: geometry_new / _dim / _is / _get / _set / _norm / _free, geometry_from_f32le_hex + geometry_to_f32le_hex as the wire adapters, realizer_register(modality, fn_name), realizer_has, and transduce(signal, modality) -> Geometry. REALIZERS ARE DECLARABLE IN EL. This is the part that makes the move real rather than nominal: registration resolves a name with dlsym against the running binary, the identical mechanism http_set_handler already relies on, because every el `fn name(...)` compiles to a global C symbol with that exact name. So an ordinary el function IS a realizer and a new modality needs no runtime patch. Verified end to end in lang/examples/transduce.el: an el-defined tone_realizer is registered by name, transduce dispatches to it, and the signal demonstrably reaches it (distinct signals produce distinct geometry). A modality with no realizer transduces to NOTHING. There is deliberately no built-in realizer, not even for text — silently embedding a description of a signal and calling that perception is the exact defect this ends. engram/src/server.el is migrated: POST /api/nodes decodes "emb" hex exactly once, at the edge, into a Geometry, and everything below that line moves geometry. The wire is unchanged because production clients speak it. "dim" is now an ASSERTION about the vector, not the source of its width; disagreement is a rejected ingest, not a silent reinterpretation. #141's engram_node_set_emb becomes a DEPRECATED WRAPPER over geometry_from_f32le_hex + node_attach_geometry — kept only because the runtime ships as an SDK asset and a downstream binary may link the symbol. Its exact contract, negative cases included, is preserved and re-verified. ingest.el's `fn transduce` is renamed transduce_manifold. Mechanically it had to yield the name (duplicate C symbol, a hard compile error, measured). But it was never signal->geometry: it chunks already-extracted content into a node+edge manifold, one layer up, and had taken the name belonging to the primitive underneath it. Behaviour unchanged. PROPERTIES FROM #141 PRESERVED, each re-measured on a scratch engram (:8971, never prod :8742): - off-dimension vectors stored but NOT indexed — the HNSW build loop still filters on n->emb_dim == dim at four sites, so a 64-dim voice vector is durable and addressable without perturbing the 768-dim canonical index - geometry makes a node ineligible for embed_backfill: after backfill the 64-dim voice node was still 64-dim while the text control acquired 768 - the create response reports whether geometry landed, and the node document always emits emb_dim and embedded Read-back with control and negatives, all verified against a PID-confirmed fresh binary: geometry node emb_dim=64 embedded=true / emb_set=1; text-only control emb_dim=0 embedded=false / emb_set=0; malformed hex, ragged length, and dim-disagreement each emb_set=0. Two compiler landmines found by reading the generated C rather than trusting a successful build, both documented at their sites: elc lowers `a == b` to str_eq unless both operand NAMES are in the per-function int-name set (which does NOT propagate into nested if-expression blocks — the first cut would have strcmp'd two integers as pointers on the first geometry-bearing request), and `+` lowers to string concat when either operand is a user-defined call.
This commit is contained in:
@@ -0,0 +1,234 @@
|
||||
import "../../runtime/eltest.el"
|
||||
// test_transduce.el — geometry as a first-class El value, and realizers
|
||||
// declared in El rather than patched into the runtime.
|
||||
//
|
||||
// WHAT IS ACTUALLY UNDER TEST. Until 2026-08-16 no El ingest path could carry
|
||||
// a vector: nodes took text, and geometry was DERIVED from that text. Text was
|
||||
// therefore the mandatory entry medium, so any non-text modality had to be
|
||||
// DESCRIBED in prose first and the geometry we reasoned over was the geometry
|
||||
// OF THE DESCRIPTION, not of the signal. The fix has two halves, and this file
|
||||
// exercises both:
|
||||
//
|
||||
// 1. Geometry is a VALUE — it carries its own width, so nothing has to
|
||||
// assert a width against a string's length.
|
||||
// 2. A REALIZER is an ordinary El function. `tone_realizer` below is not in
|
||||
// the runtime, is not known to the compiler, and is not special in any
|
||||
// way; it is registered BY NAME and dispatched to through transduce().
|
||||
// That is the load-bearing claim: adding a modality must not require a
|
||||
// runtime patch, or nothing has actually moved into the language.
|
||||
//
|
||||
// COMPARISON DISCIPLINE IN THIS FILE (measured 2026-08-16, not stylistic):
|
||||
// elc lowers `a == b` to a NUMERIC comparison only when both operand names are
|
||||
// in the per-function int-name set, which `let x: Int` populates. A bare call
|
||||
// like `geometry_is(g) == 0` is not a registered name, so it lowers to
|
||||
// `str_eq(...)` — strcmp on two integers reinterpreted as pointers. `<` and `>`
|
||||
// lower directly via binop_to_c with no type inference at all, so truthiness is
|
||||
// written `> 0` / `< 1` here, and any exact `==` is done on a value first bound
|
||||
// through `let x: Int`.
|
||||
|
||||
// ── A realizer, written entirely in El ──────────────────────────────────────
|
||||
// Maps a "tone" signal into a 4-component geometry. Deliberately trivial —
|
||||
// what is being proven is that an El function can BE a realizer, not that
|
||||
// this is good acoustics. The one real property it has: distinct signals
|
||||
// produce distinct geometry, so the test can tell transduction from a stub.
|
||||
fn tone_realizer(signal: String) -> Geometry {
|
||||
let g: Geometry = geometry_new(4)
|
||||
let n: Int = str_len(signal)
|
||||
let a: Int = geometry_set(g, 0, int_to_float(n))
|
||||
let b: Int = geometry_set(g, 1, int_to_float(n * 2))
|
||||
let c: Int = geometry_set(g, 2, int_to_float(n * 3))
|
||||
let d: Int = geometry_set(g, 3, int_to_float(n * 4))
|
||||
g
|
||||
}
|
||||
|
||||
// A second realizer for a different modality, to prove the registry keys on
|
||||
// modality and does not just hand back "the last thing registered".
|
||||
fn pulse_realizer(signal: String) -> Geometry {
|
||||
let g: Geometry = geometry_new(2)
|
||||
let a: Int = geometry_set(g, 0, 1.0)
|
||||
let b: Int = geometry_set(g, 1, 0.0)
|
||||
g
|
||||
}
|
||||
|
||||
// A deliberately BROKEN realizer: it returns something that is not a Geometry.
|
||||
// transduce() must not hand this back to a caller as if it were one.
|
||||
fn bogus_realizer(signal: String) -> Geometry {
|
||||
return 12345
|
||||
}
|
||||
|
||||
test "geometry-is-a-value-with-its-own-width" {
|
||||
let g: Geometry = geometry_new(8)
|
||||
let live: Int = geometry_is(g)
|
||||
assert live > 0, "geometry_new returns a live Geometry"
|
||||
let d: Int = geometry_dim(g)
|
||||
assert d == 8, "a Geometry carries its own width"
|
||||
let freed: Int = geometry_free(g)
|
||||
assert freed > 0, "geometry_free reports what it did"
|
||||
}
|
||||
|
||||
test "geometry-rejects-nonsense-without-an-arbitrary-bound" {
|
||||
// dim <= 0 is not a width. Note there is deliberately no MAX dim here:
|
||||
// #141 needed `dim <= 8192` only to bound an allocation sized from a
|
||||
// caller's claim about a string. A value that carries its own width has
|
||||
// nothing left to validate, so the only failure left is allocation.
|
||||
let zero: Geometry = geometry_new(0)
|
||||
let z: Int = geometry_is(zero)
|
||||
assert z < 1, "dim 0 is not a geometry"
|
||||
let neg: Geometry = geometry_new(-4)
|
||||
let n: Int = geometry_is(neg)
|
||||
assert n < 1, "negative dim is not a geometry"
|
||||
// Accessors must be total: a non-geometry is 0-width, never a crash.
|
||||
let nd: Int = geometry_dim(0)
|
||||
assert nd < 1, "geometry_dim of a non-geometry is 0"
|
||||
let ni: Int = geometry_is(0)
|
||||
assert ni < 1, "geometry_is of a non-geometry is 0"
|
||||
let nf: Int = geometry_free(0)
|
||||
assert nf < 1, "geometry_free of a non-geometry is a no-op"
|
||||
}
|
||||
|
||||
test "geometry-components-round-trip" {
|
||||
let g: Geometry = geometry_new(3)
|
||||
let s0: Int = geometry_set(g, 0, 1.5)
|
||||
let s1: Int = geometry_set(g, 1, -2.5)
|
||||
assert s0 > 0, "set in range succeeds"
|
||||
let oob: Int = geometry_set(g, 3, 9.0)
|
||||
assert oob < 1, "set out of range is refused, not silently dropped"
|
||||
let v0: Float = geometry_get(g, 0)
|
||||
let d0: Float = v0 - 1.5
|
||||
assert d0 < 0.001, "component 0 round-trips"
|
||||
assert d0 > -0.001, "component 0 round-trips"
|
||||
let v1: Float = geometry_get(g, 1)
|
||||
let d1: Float = v1 + 2.5
|
||||
assert d1 < 0.001, "component 1 round-trips (negative)"
|
||||
assert d1 > -0.001, "component 1 round-trips (negative)"
|
||||
let freed: Int = geometry_free(g)
|
||||
}
|
||||
|
||||
test "hex-is-an-edge-adapter-and-derives-its-own-width" {
|
||||
// 2 components, little-endian float32: 1.0 = 0000803f, 2.0 = 00000040.
|
||||
let g: Geometry = geometry_from_f32le_hex("0000803f00000040")
|
||||
let live: Int = geometry_is(g)
|
||||
assert live > 0, "valid hex decodes to a Geometry"
|
||||
let d: Int = geometry_dim(g)
|
||||
assert d == 2, "width is DERIVED from the input, never supplied"
|
||||
let a: Float = geometry_get(g, 0)
|
||||
let da: Float = a - 1.0
|
||||
assert da < 0.001, "first component decoded"
|
||||
assert da > -0.001, "first component decoded"
|
||||
let b: Float = geometry_get(g, 1)
|
||||
let db: Float = b - 2.0
|
||||
assert db < 0.001, "second component decoded"
|
||||
assert db > -0.001, "second component decoded"
|
||||
// Egress adapter is the exact inverse.
|
||||
let back: String = geometry_to_f32le_hex(g)
|
||||
assert str_eq(back, "0000803f00000040"), "hex round-trips exactly"
|
||||
let freed: Int = geometry_free(g)
|
||||
}
|
||||
|
||||
test "hex-rejects-malformed-input" {
|
||||
let empty: Geometry = geometry_from_f32le_hex("")
|
||||
let e: Int = geometry_is(empty)
|
||||
assert e < 1, "empty hex is not a geometry"
|
||||
let ragged: Geometry = geometry_from_f32le_hex("0000803f0000")
|
||||
let r: Int = geometry_is(ragged)
|
||||
assert r < 1, "length not a multiple of 8 is refused"
|
||||
let nonhex: Geometry = geometry_from_f32le_hex("zzzzzzzz")
|
||||
let nh: Int = geometry_is(nonhex)
|
||||
assert nh < 1, "non-hex characters are refused"
|
||||
}
|
||||
|
||||
test "a-realizer-declared-in-el-is-a-first-class-realizer" {
|
||||
// THE CLAIM: tone_realizer is an ordinary El function. It is not in the
|
||||
// runtime and the compiler knows nothing about it. Registering it by name
|
||||
// is enough to make it the organ for a modality.
|
||||
let reg: Int = realizer_register("tone", "tone_realizer")
|
||||
assert reg > 0, "an El fn registers as a realizer by name"
|
||||
let has: Int = realizer_has("tone")
|
||||
assert has > 0, "the modality now has an organ"
|
||||
|
||||
let g: Geometry = transduce("aaa", "tone")
|
||||
let live: Int = geometry_is(g)
|
||||
assert live > 0, "transduce returns real geometry"
|
||||
let d: Int = geometry_dim(g)
|
||||
assert d == 4, "the El realizer determined the width, not the runtime"
|
||||
// str_len("aaa") == 3, so component 0 must be 3.0 — proof the signal
|
||||
// actually reached the El function rather than a stub answering for it.
|
||||
let c0: Float = geometry_get(g, 0)
|
||||
let dc: Float = c0 - 3.0
|
||||
assert dc < 0.001, "the signal reached the El realizer"
|
||||
assert dc > -0.001, "the signal reached the El realizer"
|
||||
let freed: Int = geometry_free(g)
|
||||
}
|
||||
|
||||
test "distinct-signals-transduce-to-distinct-geometry" {
|
||||
let reg: Int = realizer_register("tone", "tone_realizer")
|
||||
let g1: Geometry = transduce("aa", "tone")
|
||||
let g2: Geometry = transduce("aaaaa", "tone")
|
||||
let a: Float = geometry_get(g1, 0)
|
||||
let b: Float = geometry_get(g2, 0)
|
||||
let diff: Float = b - a
|
||||
// 5 - 2 = 3. If transduction were a stub these would be equal.
|
||||
assert diff > 2.9, "different signals produce different geometry"
|
||||
assert diff < 3.1, "different signals produce different geometry"
|
||||
let f1: Int = geometry_free(g1)
|
||||
let f2: Int = geometry_free(g2)
|
||||
}
|
||||
|
||||
test "the-registry-keys-on-modality" {
|
||||
let r1: Int = realizer_register("tone", "tone_realizer")
|
||||
let r2: Int = realizer_register("pulse", "pulse_realizer")
|
||||
assert r2 > 0, "a second modality registers independently"
|
||||
let gt: Geometry = transduce("aaa", "tone")
|
||||
let gp: Geometry = transduce("aaa", "pulse")
|
||||
let dt: Int = geometry_dim(gt)
|
||||
let dp: Int = geometry_dim(gp)
|
||||
assert dt == 4, "tone still routes to its own realizer"
|
||||
assert dp == 2, "pulse routes to a different realizer"
|
||||
let f1: Int = geometry_free(gt)
|
||||
let f2: Int = geometry_free(gp)
|
||||
}
|
||||
|
||||
test "no-organ-is-reported-as-no-organ" {
|
||||
// A modality with no realizer must transduce to NOTHING. It must never
|
||||
// fall back to embedding a description of the signal and calling that
|
||||
// perception — that silent substitution is the entire defect this change
|
||||
// exists to end.
|
||||
let has: Int = realizer_has("echolocation")
|
||||
assert has < 1, "unregistered modality has no organ"
|
||||
let g: Geometry = transduce("anything", "echolocation")
|
||||
let live: Int = geometry_is(g)
|
||||
assert live < 1, "no realizer means no geometry, not fake geometry"
|
||||
}
|
||||
|
||||
test "registration-of-an-unresolvable-name-fails-loudly" {
|
||||
// Reported at the moment of WIRING, not later as "this modality mysteriously
|
||||
// produces nothing". Distinguishing "no organ" from "broken organ" is the
|
||||
// lesson that made this whole change necessary.
|
||||
let bad: Int = realizer_register("ghost", "no_such_function_anywhere")
|
||||
assert bad < 1, "an unresolvable realizer name is a registration failure"
|
||||
let has: Int = realizer_has("ghost")
|
||||
assert has < 1, "and nothing gets registered"
|
||||
}
|
||||
|
||||
test "a-realizer-returning-non-geometry-transduces-nothing" {
|
||||
let reg: Int = realizer_register("bogus", "bogus_realizer")
|
||||
assert reg > 0, "the symbol resolves, so registration succeeds"
|
||||
// ...but the contract is enforced at the boundary, so the caller never
|
||||
// receives a value that would misbehave far away from here.
|
||||
let g: Geometry = transduce("x", "bogus")
|
||||
let live: Int = geometry_is(g)
|
||||
assert live < 1, "a non-Geometry return transduced nothing"
|
||||
}
|
||||
|
||||
test "norm-lets-a-caller-check-a-realizer-emitted-signal" {
|
||||
let g: Geometry = geometry_new(2)
|
||||
let z: Float = geometry_norm(g)
|
||||
assert z < 0.001, "a fresh geometry is zero — norm says so"
|
||||
let s0: Int = geometry_set(g, 0, 3.0)
|
||||
let s1: Int = geometry_set(g, 1, 4.0)
|
||||
let n: Float = geometry_norm(g)
|
||||
let dn: Float = n - 5.0
|
||||
assert dn < 0.001, "3-4-5: norm is 5"
|
||||
assert dn > -0.001, "3-4-5: norm is 5"
|
||||
let freed: Int = geometry_free(g)
|
||||
}
|
||||
Reference in New Issue
Block a user