runtime: transduction is a language concern, so move it into the language #144

Merged

will.anderson merged 1 commits from feat/el-geometry-transduce into dev 2026-08-16 16:57:36 +00:00

Conversation 0 Commits 1 Files Changed 6

+969 -87

will.anderson commented 2026-08-16 16:39:06 +00:00

Owner

Copy Link

Copy Source

What this is

#141 let signal enter as geometry and it worked. It was also placed at the consumer, and said so in its own commit message. This is that correction: transduction moves out of the engram and into the el language.

Why it was placed wrong (three defects, all placement)

1. It sat in the engram. Ingest is a language concern — every el program touching any modality needs it, and the engram is merely one el program that happens to hold a graph. The geometry surface is now defined in el_runtime.c immediately above the engram section and depends on nothing inside it. You can delete the entire engram and geometry still enters el. The file ordering is the architectural statement.

2. It marshalled the vector as a hex STRING, because el had no first-class geometry value — reintroducing text as the transport medium one layer below the problem being fixed. Geometry is now an el value: a magic-tagged heap object carried in el_val_t, same discipline as List/Map. Hex survives only as an adapter at the edge, which is all an encoding should ever be.

3. It needed an arbitrary dim <= 8192 bound, purely to size an allocation from a caller's claim about a string's length. A value carries its own width, so the width is now derived and never asserted. The bound is gone, not raised — there is nothing left to validate. The only remaining failure is the allocation itself.

The language surface (none of it engram-prefixed)

geometry_new(dim) -> Geometry        geometry_dim / geometry_is
geometry_get / geometry_set          geometry_norm / geometry_free
geometry_from_f32le_hex(hex)         geometry_to_f32le_hex(g)     # edge adapters only
realizer_register(modality, fn_name) realizer_has(modality)
transduce(signal, modality) -> Geometry
node_attach_geometry(id, g)          node_geometry_dim(id)

Geometry needs zero codegen change — but not for the reason originally assumed. el_type_to_c is dead code (never called anywhere in the compiler); the shipped elc discards type annotations entirely and represents every value as el_val_t. Geometry follows Calendar/Instant: an opaque boxed pointer annotated as a bare identifier.

Realizers are declarable in el — this is what makes the move real rather than nominal

Registration resolves a name with dlsym against the running binary — the identical mechanism http_set_handler already relies on, because every el fn name(...) compiles to a global C symbol with that exact name. So an ordinary el function IS a realizer, and a new modality needs no runtime patch.

fn tone_realizer(signal: String) -> Geometry {
    let g: Geometry = geometry_new(4)
    let n: Int = str_len(signal)
    let a: Int = geometry_set(g, 0, int_to_float(n))
    g
}

realizer_register("tone", "tone_realizer")
let g: Geometry = transduce(sample, "tone")

Proven end to end in lang/examples/transduce.el (36 checks, exits non-zero on any failure; negative-control verified — deliberately breaking one assertion does fail it). The el-defined realizer is registered by name, transduce dispatches to it, and the signal demonstrably reaches it: distinct signals produce distinct geometry.

A modality with no realizer transduces to nothing. There is deliberately no built-in realizer, not even for text — silently embedding a description of a signal and calling that perception is the exact defect this ends. realizer_register on an unresolvable name fails at wiring time rather than surfacing later as "this modality mysteriously produces nothing".

What #141's API becomes: wrapped and deprecated

engram_node_set_emb(id, hex, dim) is now a thin wrapper over geometry_from_f32le_hex + node_attach_geometry — literally that, nothing more. It is kept only because the runtime ships as an SDK asset and a downstream binary may already link the symbol. It is not kept because a hex string is an acceptable way to move geometry between two pieces of el; that was the defect.

Its exact contract is preserved, negative cases included, and re-verified: dim <= 0 rejects, malformed hex rejects, length/dim disagreement rejects, unknown id rejects. The difference is that dim is now an assertion checked against a width the Geometry already knows, rather than the authority the allocation trusted.

Wire compatibility

POST /api/nodes with emb is unchanged — production clients speak it. Hex is decoded exactly once, at the edge, into a Geometry; everything below that line moves geometry. dim is now an assertion about the vector rather than the source of its width, so a disagreement is a rejected ingest, not a silent reinterpretation. Omitting dim means "trust the vector", which is the honest default.

ingest.el: transduce renamed to transduce_manifold

Mechanically it had to yield the name — duplicate C symbol, a hard conflicting types for 'transduce' compile error, measured not anticipated. But it was never signal→geometry: it chunks already-extracted content into a node+edge manifold, one layer up, and had taken the name belonging to the primitive underneath it. Behaviour unchanged.

Properties from #141 preserved — each re-measured, not assumed

Against a scratch engram on :8971 (prod :8742 never touched; PID confirmed against the freshly built binary before every probe, per the stale-binary trap that nearly produced a false result last time):

	result
geometry node, dim omitted	emb_set:1, read-back emb_dim:64 embedded:true
text-only control	emb_set:0, read-back emb_dim:0 embedded:false
malformed hex	emb_set:0
ragged length (not a multiple of 8)	emb_set:0
dim disagrees (claims 32, sends 64)	emb_set:0
dim agrees (claims 64, sends 64)	emb_set:1, emb_dim:64

• Off-dimension vectors stored but NOT indexed — the HNSW build loop still filters on n->emb_dim == dim at four sites, so a 64-dim voice vector is durable and addressable without perturbing the 768-dim canonical index. Search and health verified on the mixed-dim graph.
• Geometry makes a node ineligible for embed_backfill — after backfill the 64-dim voice node was still 64-dim while the text control acquired 768. A realizer's vector is never overwritten by a text-derived one.
• The create response reports whether geometry landed, and the node document always emits emb_dim and embedded.

Two compiler landmines found by reading the generated C

Both documented at their sites. Neither is caused by this change; both would have bitten it.

1. elc lowers a == b to str_eq unless both operand names are in the per-function int-name set — and that registration does not propagate into a nested if-expression block. The first cut of the ingest path emitted str_eq(claimed, got), i.e. strcmp on two integers reinterpreted as pointers: a segfault on the first geometry-bearing request. It built cleanly. Only reading the emitted C caught it. Fixed by moving the comparison into a function whose parameters are : Int (which does register reliably), and verified in the output.
2. + lowers to string concat when either operand is a user-defined call, so a fails + check(...) tally printed 4343632752 — a pointer — while every check passed. The checks were right; the tally was lying.

Not done / known state

• lang/tests/native/test_transduce.el does not run. The shipped elc emits calls to __el_reg_count / __el_reg_invoke / etc. without emitting their definitions, so every native test fails to link — test_math.el included, on unmodified dev. Pre-existing and unrelated; the test file is committed in the conventional form for when the harness is repaired, and lang/examples/transduce.el is the runnable proof in the meantime.
• No __-prefixed el_seed.c wrappers for the new builtins (step 2 of the lang/AGENTS.md recipe). Not needed by anything here — elc emits plain names — and adding them risks a link configuration I could not test.
• No builtin_arity / is_int_call entries in codegen.el. Deliberate: those are optional guards, and touching codegen.el forces an elc rebuild plus a self-host fixpoint check. Consequence: geometry_dim(g) == n written inline will mis-lower — hence width_agrees() and the documented comparison discipline.
• The engram registers no realizers. It gains the geometry-valued ingest path, but wiring an actual modality organ (e.g. the perception vessel's voice realizer) into the engram binary is a separate step. I did not ship a signal+modality route, because with no realizer registered it would be a dead route — better to say so than to ship one.
• Local build only, per instruction; CI not exercised.

Build

lang/dist/platform/elc engram/src/server.el > engram.c
cc -std=c11 -O2 -DHAVE_CURL -I lang/runtime -o engram engram.c \
   lang/runtime/el_runtime.c lang/runtime/el_seed.c lang/runtime/engram_*.c \
   -lcurl -lpthread -lm

engram, ingest, and the transduce example all build clean locally on this branch, rebased onto dev at a6cef4b (#143), and every measurement above was re-run after that rebase.

## What this is #141 let signal enter as geometry and it worked. It was also placed at the **consumer**, and said so in its own commit message. This is that correction: transduction moves out of the engram and into the **el language**. ## Why it was placed wrong (three defects, all placement) 1. **It sat in the engram.** Ingest is a *language* concern — every el program touching any modality needs it, and the engram is merely one el program that happens to hold a graph. The geometry surface is now defined in `el_runtime.c` immediately **above** the engram section and depends on nothing inside it. You can delete the entire engram and geometry still enters el. The file ordering is the architectural statement. 2. **It marshalled the vector as a hex STRING**, because el had no first-class geometry value — reintroducing text as the *transport* medium one layer below the problem being fixed. Geometry is now an el value: a magic-tagged heap object carried in `el_val_t`, same discipline as `List`/`Map`. Hex survives only as an adapter **at the edge**, which is all an encoding should ever be. 3. **It needed an arbitrary `dim <= 8192` bound**, purely to size an allocation from a caller's *claim* about a string's length. A value carries its own width, so the width is now derived and never asserted. **The bound is gone, not raised** — there is nothing left to validate. The only remaining failure is the allocation itself. ## The language surface (none of it engram-prefixed) ``` geometry_new(dim) -> Geometry geometry_dim / geometry_is geometry_get / geometry_set geometry_norm / geometry_free geometry_from_f32le_hex(hex) geometry_to_f32le_hex(g) # edge adapters only realizer_register(modality, fn_name) realizer_has(modality) transduce(signal, modality) -> Geometry node_attach_geometry(id, g) node_geometry_dim(id) ``` `Geometry` needs **zero codegen change** — but not for the reason originally assumed. `el_type_to_c` is dead code (never called anywhere in the compiler); the shipped `elc` discards type annotations entirely and represents every value as `el_val_t`. `Geometry` follows `Calendar`/`Instant`: an opaque boxed pointer annotated as a bare identifier. ## Realizers are declarable in el — this is what makes the move real rather than nominal Registration resolves a name with `dlsym` against the running binary — the identical mechanism `http_set_handler` already relies on, because every el `fn name(...)` compiles to a global C symbol with that exact name. So **an ordinary el function IS a realizer, and a new modality needs no runtime patch.** ```el fn tone_realizer(signal: String) -> Geometry { let g: Geometry = geometry_new(4) let n: Int = str_len(signal) let a: Int = geometry_set(g, 0, int_to_float(n)) g } realizer_register("tone", "tone_realizer") let g: Geometry = transduce(sample, "tone") ``` Proven end to end in `lang/examples/transduce.el` (36 checks, exits non-zero on any failure; negative-control verified — deliberately breaking one assertion does fail it). The el-defined realizer is registered by name, `transduce` dispatches to it, and the signal demonstrably *reaches* it: distinct signals produce distinct geometry. A modality with no realizer transduces to **nothing**. There is deliberately no built-in realizer, not even for text — silently embedding a description of a signal and calling that perception is the exact defect this ends. `realizer_register` on an unresolvable name fails at *wiring* time rather than surfacing later as "this modality mysteriously produces nothing". ## What #141's API becomes: **wrapped and deprecated** `engram_node_set_emb(id, hex, dim)` is now a thin wrapper over `geometry_from_f32le_hex` + `node_attach_geometry` — literally that, nothing more. It is kept **only** because the runtime ships as an SDK asset and a downstream binary may already link the symbol. It is *not* kept because a hex string is an acceptable way to move geometry between two pieces of el; that was the defect. Its exact contract is preserved, negative cases included, and re-verified: `dim <= 0` rejects, malformed hex rejects, length/dim disagreement rejects, unknown id rejects. The difference is that `dim` is now an *assertion* checked against a width the Geometry already knows, rather than the authority the allocation trusted. ## Wire compatibility `POST /api/nodes` with `emb` is **unchanged** — production clients speak it. Hex is decoded exactly once, at the edge, into a `Geometry`; everything below that line moves geometry. `dim` is now an assertion about the vector rather than the source of its width, so a disagreement is a **rejected ingest, not a silent reinterpretation**. Omitting `dim` means "trust the vector", which is the honest default. ## ingest.el: `transduce` renamed to `transduce_manifold` Mechanically it had to yield the name — duplicate C symbol, a hard `conflicting types for 'transduce'` compile error, measured not anticipated. But it was never signal→geometry: it chunks already-extracted content into a node+edge manifold, one layer up, and had taken the name belonging to the primitive underneath it. **Behaviour unchanged.** ## Properties from #141 preserved — each re-measured, not assumed Against a scratch engram on `:8971` (prod `:8742` never touched; PID confirmed against the freshly built binary before every probe, per the stale-binary trap that nearly produced a false result last time): | | result | |---|---| | geometry node, `dim` omitted | `emb_set:1`, read-back `emb_dim:64 embedded:true` | | **text-only control** | `emb_set:0`, read-back `emb_dim:0 embedded:false` | | malformed hex | `emb_set:0` | | ragged length (not a multiple of 8) | `emb_set:0` | | `dim` disagrees (claims 32, sends 64) | `emb_set:0` | | `dim` agrees (claims 64, sends 64) | `emb_set:1`, `emb_dim:64` | - **Off-dimension vectors stored but NOT indexed** — the HNSW build loop still filters on `n->emb_dim == dim` at four sites, so a 64-dim voice vector is durable and addressable without perturbing the 768-dim canonical index. Search and health verified on the mixed-dim graph. - **Geometry makes a node ineligible for `embed_backfill`** — after backfill the 64-dim voice node was *still* 64-dim while the text control acquired 768. A realizer's vector is never overwritten by a text-derived one. - **The create response reports whether geometry landed**, and the node document always emits `emb_dim` and `embedded`. ## Two compiler landmines found by reading the generated C Both documented at their sites. Neither is caused by this change; both would have bitten it. 1. `elc` lowers `a == b` to **`str_eq`** unless both operand *names* are in the per-function int-name set — and that registration does **not** propagate into a nested if-expression block. The first cut of the ingest path emitted `str_eq(claimed, got)`, i.e. `strcmp` on two integers reinterpreted as pointers: a segfault on the first geometry-bearing request. It built cleanly. Only reading the emitted C caught it. Fixed by moving the comparison into a function whose *parameters* are `: Int` (which does register reliably), and verified in the output. 2. `+` lowers to **string concat** when either operand is a user-defined call, so a `fails + check(...)` tally printed `4343632752` — a pointer — while every check passed. The checks were right; the tally was lying. ## Not done / known state - **`lang/tests/native/test_transduce.el` does not run.** The shipped `elc` emits calls to `__el_reg_count` / `__el_reg_invoke` / etc. without emitting their definitions, so **every** native test fails to link — `test_math.el` included, on unmodified `dev`. Pre-existing and unrelated; the test file is committed in the conventional form for when the harness is repaired, and `lang/examples/transduce.el` is the runnable proof in the meantime. - **No `__`-prefixed `el_seed.c` wrappers** for the new builtins (step 2 of the `lang/AGENTS.md` recipe). Not needed by anything here — `elc` emits plain names — and adding them risks a link configuration I could not test. - **No `builtin_arity` / `is_int_call` entries** in `codegen.el`. Deliberate: those are optional guards, and touching `codegen.el` forces an `elc` rebuild plus a self-host fixpoint check. Consequence: `geometry_dim(g) == n` written inline will mis-lower — hence `width_agrees()` and the documented comparison discipline. - **The engram registers no realizers.** It gains the geometry-valued ingest path, but wiring an actual modality organ (e.g. the perception vessel's voice realizer) into the engram binary is a separate step. I did not ship a `signal`+`modality` route, because with no realizer registered it would be a dead route — better to say so than to ship one. - Local build only, per instruction; CI not exercised. ## Build ``` lang/dist/platform/elc engram/src/server.el > engram.c cc -std=c11 -O2 -DHAVE_CURL -I lang/runtime -o engram engram.c \ lang/runtime/el_runtime.c lang/runtime/el_seed.c lang/runtime/engram_*.c \ -lcurl -lpthread -lm ``` engram, ingest, and the transduce example all build clean locally on this branch, rebased onto `dev` at `a6cef4b` (#143), and every measurement above was re-run after that rebase.

will.anderson added 1 commit 2026-08-16 16:39:07 +00:00

runtime: transduction is a language concern, so move it into the language ... 3fcc36c2f1

#141 let signal enter as geometry and it worked, but it was placed at the
CONSUMER and said so in its own commit message. This is the correction.

Three defects, all of them placement:

1. It sat in the engram. Ingest is a LANGUAGE concern — every el program
   touching any modality needs it, and the engram is merely one el program
   that happens to hold a graph. The geometry surface is now defined in
   el_runtime.c immediately ABOVE the engram section and depends on nothing
   inside it. Delete the entire engram and geometry still enters el.

2. It marshalled the vector as a hex STRING, because el had no first-class
   geometry value — which reintroduced text as the TRANSPORT medium one layer
   below the problem being fixed. Geometry is now an el value: a magic-tagged
   heap object carried in el_val_t, same discipline as List/Map. Hex survives
   only as an adapter at the edge, which is all an encoding should ever be.

3. It needed an arbitrary `dim <= 8192` bound purely to size an allocation
   from a caller's CLAIM about a string's length. A value carries its own
   width, so the width is derived and never asserted. The bound is gone, not
   raised — there is nothing left to validate.

Language surface, none of it engram-prefixed: geometry_new / _dim / _is /
_get / _set / _norm / _free, geometry_from_f32le_hex + geometry_to_f32le_hex
as the wire adapters, realizer_register(modality, fn_name), realizer_has, and
transduce(signal, modality) -> Geometry.

REALIZERS ARE DECLARABLE IN EL. This is the part that makes the move real
rather than nominal: registration resolves a name with dlsym against the
running binary, the identical mechanism http_set_handler already relies on,
because every el `fn name(...)` compiles to a global C symbol with that exact
name. So an ordinary el function IS a realizer and a new modality needs no
runtime patch. Verified end to end in lang/examples/transduce.el: an el-defined
tone_realizer is registered by name, transduce dispatches to it, and the
signal demonstrably reaches it (distinct signals produce distinct geometry).

A modality with no realizer transduces to NOTHING. There is deliberately no
built-in realizer, not even for text — silently embedding a description of a
signal and calling that perception is the exact defect this ends.

engram/src/server.el is migrated: POST /api/nodes decodes "emb" hex exactly
once, at the edge, into a Geometry, and everything below that line moves
geometry. The wire is unchanged because production clients speak it. "dim" is
now an ASSERTION about the vector, not the source of its width; disagreement
is a rejected ingest, not a silent reinterpretation.

#141's engram_node_set_emb becomes a DEPRECATED WRAPPER over
geometry_from_f32le_hex + node_attach_geometry — kept only because the runtime
ships as an SDK asset and a downstream binary may link the symbol. Its exact
contract, negative cases included, is preserved and re-verified.

ingest.el's `fn transduce` is renamed transduce_manifold. Mechanically it had
to yield the name (duplicate C symbol, a hard compile error, measured). But it
was never signal->geometry: it chunks already-extracted content into a node+edge
manifold, one layer up, and had taken the name belonging to the primitive
underneath it. Behaviour unchanged.

PROPERTIES FROM #141 PRESERVED, each re-measured on a scratch engram (:8971,
never prod :8742):
  - off-dimension vectors stored but NOT indexed — the HNSW build loop still
    filters on n->emb_dim == dim at four sites, so a 64-dim voice vector is
    durable and addressable without perturbing the 768-dim canonical index
  - geometry makes a node ineligible for embed_backfill: after backfill the
    64-dim voice node was still 64-dim while the text control acquired 768
  - the create response reports whether geometry landed, and the node document
    always emits emb_dim and embedded

Read-back with control and negatives, all verified against a PID-confirmed
fresh binary: geometry node emb_dim=64 embedded=true / emb_set=1; text-only
control emb_dim=0 embedded=false / emb_set=0; malformed hex, ragged length,
and dim-disagreement each emb_set=0.

Two compiler landmines found by reading the generated C rather than trusting a
successful build, both documented at their sites: elc lowers `a == b` to
str_eq unless both operand NAMES are in the per-function int-name set (which
does NOT propagate into nested if-expression blocks — the first cut would have
strcmp'd two integers as pointers on the first geometry-bearing request), and
`+` lowers to string concat when either operand is a user-defined call.

will.anderson merged commit 1a8a966cb3 into dev 2026-08-16 16:57:36 +00:00

will.anderson referenced this pull request 2026-08-16 18:30:12 +00:00

docs: carry the correspondence corrections, because a stale doc builds the wrong thing #152

will.anderson referenced this issue from a commit 2026-08-16 18:34:34 +00:00

docs: carry the correspondence corrections, because a stale doc builds the wrong thing

will.anderson referenced this issue from a commit 2026-08-16 20:11:24 +00:00

runtime: transduction decomposes a signal, it does not convert it

will.anderson referenced this issue from a commit 2026-08-16 20:11:25 +00:00

ingest: name the inversion, and correct the worked example to decomposition

will.anderson referenced this pull request 2026-08-16 20:12:21 +00:00

runtime: transduction decomposes a signal into components and relations, it does not convert it to a point #155

will.anderson referenced this issue from a commit 2026-08-16 20:49:55 +00:00

docs: carry the correspondence corrections, because a stale doc builds the wrong thing

will.anderson referenced this issue from a commit 2026-08-16 20:50:13 +00:00

runtime: transduction decomposes a signal, it does not convert it

will.anderson referenced this issue from a commit 2026-08-16 20:50:14 +00:00

ingest: name the inversion, and correct the worked example to decomposition

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

BETA-CRITICAL

On the beta critical path — must resolve before ship

HELD

Blocked pending an external decision (counsel / dual sign-off)

ORTHOGONAL

Parallel workstream (testing/hardening) — safe to assign off critical path

POST-BETA

Important but not blocking the beta ship

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

andre.almeida (Andre Almeida) tim.lingo (Tim Lingo) will.anderson (Will Anderson)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/el#144