runtime: transduction decomposes a signal into components and relations, it does not convert it to a point #155

2026-08-16T20:12:21Z

will.anderson commented

2026-08-16 20:12:21 +00:00

The premise that was wrong

Transduction is not conversion. #144 implemented it as signal → one vector: take a thing, encode it, store a position. That is a fingerprint, not a transduction. A fingerprint supports exactly two operations — match and rank. It cannot be decomposed, cannot have one part grounded while another is not, and cannot be contradicted in one part while holding in another, because it has no parts.

A song is not a point. It decomposes into pitch, interval, rhythm, harmonic function — components, each with its own geometry, plus the relations between them. The song is the structure of the relations.

transduce(signal, modality) now returns a Manifold: named components carrying geometry, and typed weighted relations among them. Signal in, subgraph out.

Judgement on the `transduce_manifold` inversion

The inversion is real, and the rename enshrined it — but neither layer was doing transduction. Evidence:

	what it returned	verdict
`transduce` (#144, `el_runtime.c`)	`ElGeometry { int32_t dim; float* v; }` — a flat vector, `geometry_get/set(i)`, `geometry_norm`, `to_f32le_hex`. `el_realizer_fn` is `el_val_t ()(el_val_t)`: one value in, one opaque value out. No components. No edges.*	This is encoding, and it took the name.
`transduce_manifold` (`ingest.el`)	nodes + edges (`contains`, `precedes`, `section_of`) — a subgraph	Right shape, so it had the better claim to the name.

So the rename moved the name from the layer with the right output shape to the layer with the wrong output type. That is the bug, exactly as reported.

The correction to the report: transduce_manifold was not simply right. It decomposes by byte boundary — str_split(source, "\n\n"), else fixed 4096-byte windows — and declares "zero semantic understanding of any payload." Its components are positional, not meaningful. Splitting a WAV into 4096-byte windows is not pitch/interval/rhythm. So it is transduction in shape and a modality-blind degenerate case in content.

The real fix therefore is not to swap the names back. It is to give the primitive the subgraph return type, at which point both layers are doing the same kind of operation and the inversion dissolves rather than needing to be re-argued. What remains is a genuine distinction about modality, not layering — so transduce_manifold is renamed transduce_bytes (suffixed by its modality, not demoted by layer), and the comment block that recorded the inverted reasoning is replaced with this account.

The decomposing realizer contract

fn tone_realizer(signal: String) -> Manifold {
    let m: Manifold = manifold_new()
    let i: Int = manifold_add(m, "note:0", "pitch", pitch_geom)
    let j: Int = manifold_add(m, "interval:0-1", "interval", interval_geom)
    let e: Int = manifold_relate(m, "interval:0-1", "spans", "note:0", 0.9)
    m
}
realizer_register("tone", "tone_realizer")
let m: Manifold = transduce("CEG", "tone")

Design decisions, each load-bearing:

Components are addressed by KEY, never by index. The key is what survives persistence: a component becomes a node, and is separately groundable precisely because it is separately named. A positional reference into a decomposition whose arity can change is not a reference. Duplicate keys are refused.
Relation weight IS the grounding — one quantity, no separate score (correspondence-and-censorship.md §1). A realizer states a relation and the weight is the claim; nothing computes or observes a grounding (§4).
A relation to an unknown endpoint is REFUSED, not dropped. A decomposition that silently loses edges is indistinguishable from one that never had them.
manifold_add copies, manifold_geometry returns a copy. No component's vector is aliased in either direction.
The one-part case is manifold_single — a size-1 manifold, visibly degenerate, not a parallel path back to a bare vector.

Evidence, with negative controls

Baseline: test_transduce.el on pristine origin/dev (385c184) → 12 tests, 12 passed. The baseline was green, so these controls are not measuring a pre-broken tree.

Positive: 25 tests, 25 passed, 83 assertions. Worked example lang/examples/transduce.el exits 0.

Control A — the unpatched build cannot express the contract. The same test file, compiled by the same elc against pristine origin/dev: El → C succeeds, then cc fails with call to undeclared function 'manifold_new' / 'manifold_add' / 'manifold_index_of' / 'manifold_relate' / …. No binary is produced.

Control B — same signal, same engram, both builds, measured. "CEG" transduced and persisted (in-memory engram; no server contacted):

measurement	unpatched (#144 `-> Geometry`)	patched (`-> Manifold`)
nodes from one signal	1	5
edges from one signal	0	6
independently groundable parts	1	5
geometry widths	one 8-wide vector	2, 2, 2, 1, 1 — heterogeneous
grounding target for "the C" vs "the G"	same node id	different node ids

Read back from the graph on the patched build, note:0 carries {"relation":"corroborates","weight":0.95} from its evidence node, and that edge is demonstrably absent from note:2 — one part grounded, the sibling untouched, with both geometries intact. On the unpatched build both the corroboration and the contradiction land on one undifferentiated point.

A defect found while proving this — NOT fixed here

The contradiction half of the demonstration does not work, and not because of decomposition. engram_connect at el_runtime.c:9730:

if (e->weight <= 0.0 || e->weight > 1.0) e->weight = 0.5;

A weight of -0.85 is silently coerced to 0.5. A stated contradiction is stored as moderate support, with inhibitory: 0. Verified by read-back, not inferred.

This is precisely the polarity gap named in correspondence-and-censorship.md §2 — "negative means this actively contradicts… inhibitory is that distinction crushed to one bit." It is a separate defect from the one this PR fixes. Decomposition is the precondition for contradicting one part while a sibling holds, and it now exists; a signed edge weight is what is still missing.

What #141 and #144 become

#141 engram_node_set_emb(id, hex, dim) — UNCHANGED, still wire-compatible. It was already superseded in place on 2026-08-16 and reimplemented as geometry_from_f32le_hex + node_attach_geometry. This PR does not touch it, and the "emb" hex wire format on /api/nodes is untouched. Production clients speaking hex keep working.
#144 transduce -> Geometry — SUPERSEDED, deliberately breaking. A realizer still returning a bare Geometry now transduces nothing (transduce returns 0). This is a hard failure by design: "no organ" and "an organ that only fingerprints" must not be indistinguishable — the same distinction realizer_register already draws between an absent and a broken organ. Asserted directly by the test a-fingerprint-realizer-transduces-nothing.
#144's realizer_register / realizer_has / dlsym-by-name — UNCHANGED and vindicated. The dispatch design was right; only the return type was wrong.
Geometry — UNCHANGED, and relocated in meaning. Every #141/#144 geometry test still passes verbatim. A vector was never wrong; it was misplaced. It is the right representation for a component, and was only ever wrong as the representation of a whole signal.

What I did NOT get to

transduce_bytes is still not a registered realizer. It should return a real Manifold and ride the primitive like every other modality, instead of carrying a parallel node+edge implementation against the HTTP API. Renamed and documented, not converted.
No audio realizer. The contract is proven with a chord-of-note-letters realizer that decomposes into notes and intervals. A real acoustic realizer (MFCC per frame, onset detection, harmonic function) is the obvious next consumer and is not written.
Signed edge weight (the :9730 clamp above) — diagnosed, not fixed.
No persistence to a disk-backed engram. All engram evidence is in-process/in-memory. The /api/nodes/emb read-back route on feat/love-origin is untouched and unmerged.
server.el untouched. Nothing in the HTTP surface exposes a Manifold yet; a transduced subgraph currently reaches the graph only through in-process El.

/api/search — characterised, and the reported symptom is wrong

Investigated on an isolated APFS clone (cp -Rc) served on port 8911 with PID/port ownership verified against lsof. Prod :8742 and soul :7770 were never touched.

It does not return the same cluster for every query. Across 17 queries the union was 102 distinct ids and the intersection was 0. Two distinct behaviours: unmatched queries (zzzznonsense, photosynthesis) return 0 results, not a cluster; matched queries often collapse onto the Working-tier skill-curriculum cluster while discarding the query's own exact-label node. /api/search-lexical is correct throughout — the lexical path is fine, the geometric gate is the defect.

Root cause in engram_retrieve_geometric_json, lang/runtime/el_runtime.c (line numbers against 385c184):

:13622-13625 — dominant. The region tag is applied as a hard filter, not a rank boost: if(!keep) continue;. The correct exact-label node is seeded and activated, then discarded for lacking the addressed tag. Only ~38 skill-curriculum nodes carry tags, so the answer collapses onto them.
:13575-13581 — the coarse gate has no concentration threshold (unlike the fine branch at :13570), and truncating at the first : makes dom_family always literally "skill:" — it cannot tell skill:programming from skill:arithmetic.
:13533 + :9288-9300 — engram_node_match_score is case-insensitive substring match with no stopwords and no word boundaries, so "self" matches "give every number times itself". Almost any common word seeds a tagged node and trips the gate.
:13626 — hits[nh].score=0 is hard-coded while engram_rank_cmp sorts on score first, so lexical evidence is discarded from ranking (compare engram_search_json:13435, which sets score = sc correctly).

Not fixed here; out of scope for this PR.

Build

The committed CI recipe does not link on macOS. -lssl -lcrypto are unresolvable, and el_runtime.c alone leaves ~60 undefined symbols — el_seed.c (owns el_request_end) and the six engram_*.c units are required.

ELC=lang/dist/platform/elc; RT=lang/runtime
"$ELC" --test lang/tests/native/test_transduce.el > /tmp/t.c
cc -O2 -I "$RT" /tmp/t.c \
   "$RT/el_runtime.c" "$RT/el_seed.c" \
   "$RT/engram_store.c" "$RT/engram_vindex.c" "$RT/engram_cognition.c" \
   "$RT/engram_geometry.c" "$RT/engram_reason.c" "$RT/engram_verify.c" \
   -lcurl -lpthread -lm -o /tmp/t && /tmp/t

No process was deployed, restarted, or signalled. Prod engram :8742 and soul :7770 were untouched throughout.

## The premise that was wrong **Transduction is not conversion.** #144 implemented it as signal → one vector: take a thing, encode it, store a position. That is a *fingerprint*, not a transduction. A fingerprint supports exactly two operations — match and rank. It cannot be decomposed, cannot have one part grounded while another is not, and cannot be contradicted in one part while holding in another, **because it has no parts**. A song is not a point. It decomposes into pitch, interval, rhythm, harmonic function — components, each with its own geometry, plus the relations between them. **The song is the structure of the relations.** `transduce(signal, modality)` now returns a **Manifold**: named components carrying geometry, and typed weighted relations among them. Signal in, subgraph out. ## Judgement on the `transduce_manifold` inversion **The inversion is real, and the rename enshrined it — but neither layer was doing transduction.** Evidence: | | what it returned | verdict | |---|---|---| | `transduce` (#144, `el_runtime.c`) | `ElGeometry { int32_t dim; float* v; }` — a flat vector, `geometry_get/set(i)`, `geometry_norm`, `to_f32le_hex`. `el_realizer_fn` is `el_val_t (*)(el_val_t)`: one value in, one opaque value out. **No components. No edges.** | This is **encoding**, and it took the name. | | `transduce_manifold` (`ingest.el`) | nodes + edges (`contains`, `precedes`, `section_of`) — a subgraph | Right **shape**, so it had the better claim to the name. | So the rename moved the name from the layer with the right output shape to the layer with the wrong output type. That is the bug, exactly as reported. **The correction to the report:** `transduce_manifold` was not simply *right*. It decomposes by **byte boundary** — `str_split(source, "\n\n")`, else fixed 4096-byte windows — and declares "zero semantic understanding of any payload." Its components are *positional*, not meaningful. Splitting a WAV into 4096-byte windows is not pitch/interval/rhythm. So it is transduction in shape and a modality-blind degenerate case in content. The real fix therefore is not to swap the names back. It is to give the **primitive** the subgraph return type, at which point both layers are doing the same kind of operation and the inversion dissolves rather than needing to be re-argued. What remains is a genuine distinction about **modality, not layering** — so `transduce_manifold` is renamed `transduce_bytes` (suffixed by its modality, not demoted by layer), and the comment block that recorded the inverted reasoning is replaced with this account. ## The decomposing realizer contract ```el fn tone_realizer(signal: String) -> Manifold { let m: Manifold = manifold_new() let i: Int = manifold_add(m, "note:0", "pitch", pitch_geom) let j: Int = manifold_add(m, "interval:0-1", "interval", interval_geom) let e: Int = manifold_relate(m, "interval:0-1", "spans", "note:0", 0.9) m } realizer_register("tone", "tone_realizer") let m: Manifold = transduce("CEG", "tone") ``` Design decisions, each load-bearing: - **Components are addressed by KEY, never by index.** The key is what survives persistence: a component becomes a node, and is separately groundable precisely because it is separately *named*. A positional reference into a decomposition whose arity can change is not a reference. Duplicate keys are refused. - **Relation weight IS the grounding** — one quantity, no separate score (`correspondence-and-censorship.md` §1). A realizer states a relation and the weight is the claim; nothing computes or observes a grounding (§4). - **A relation to an unknown endpoint is REFUSED, not dropped.** A decomposition that silently loses edges is indistinguishable from one that never had them. - **`manifold_add` copies, `manifold_geometry` returns a copy.** No component's vector is aliased in either direction. - **The one-part case is `manifold_single`** — a *size-1 manifold*, visibly degenerate, not a parallel path back to a bare vector. ## Evidence, with negative controls **Baseline:** `test_transduce.el` on pristine `origin/dev` (385c184) → `12 tests, 12 passed`. The baseline was green, so these controls are not measuring a pre-broken tree. **Positive:** `25 tests, 25 passed, 83 assertions`. Worked example `lang/examples/transduce.el` exits 0. **Control A — the unpatched build cannot express the contract.** The same test file, compiled by the same `elc` against pristine `origin/dev`: El → C succeeds, then `cc` fails with `call to undeclared function 'manifold_new' / 'manifold_add' / 'manifold_index_of' / 'manifold_relate' / …`. No binary is produced. **Control B — same signal, same engram, both builds, measured.** `"CEG"` transduced and persisted (in-memory engram; no server contacted): | measurement | unpatched (#144 `-> Geometry`) | patched (`-> Manifold`) | |---|---|---| | nodes from one signal | **1** | **5** | | edges from one signal | **0** | **6** | | independently groundable parts | **1** | **5** | | geometry widths | one 8-wide vector | 2, 2, 2, 1, 1 — heterogeneous | | grounding target for "the C" vs "the G" | **same node id** | **different node ids** | Read back from the graph on the patched build, `note:0` carries `{"relation":"corroborates","weight":0.95}` from its evidence node, and that edge is demonstrably **absent** from `note:2` — one part grounded, the sibling untouched, with both geometries intact. On the unpatched build both the corroboration and the contradiction land on one undifferentiated point. ## A defect found while proving this — NOT fixed here The contradiction half of the demonstration **does not work, and not because of decomposition**. `engram_connect` at `el_runtime.c:9730`: ```c if (e->weight <= 0.0 || e->weight > 1.0) e->weight = 0.5; ``` A weight of `-0.85` is silently coerced to `0.5`. **A stated contradiction is stored as moderate support**, with `inhibitory: 0`. Verified by read-back, not inferred. This is precisely the polarity gap named in `correspondence-and-censorship.md` §2 — *"negative means this actively contradicts… `inhibitory` is that distinction crushed to one bit."* It is a separate defect from the one this PR fixes. Decomposition is the **precondition** for contradicting one part while a sibling holds, and it now exists; a signed edge weight is what is still missing. ## What #141 and #144 become - **#141 `engram_node_set_emb(id, hex, dim)` — UNCHANGED, still wire-compatible.** It was already superseded in place on 2026-08-16 and reimplemented as `geometry_from_f32le_hex` + `node_attach_geometry`. This PR does not touch it, and the `"emb"` hex wire format on `/api/nodes` is untouched. Production clients speaking hex keep working. - **#144 `transduce -> Geometry` — SUPERSEDED, deliberately breaking.** A realizer still returning a bare `Geometry` now transduces **nothing** (`transduce` returns 0). This is a hard failure by design: "no organ" and "an organ that only fingerprints" must not be indistinguishable — the same distinction `realizer_register` already draws between an absent and a broken organ. Asserted directly by the test `a-fingerprint-realizer-transduces-nothing`. - **#144's `realizer_register` / `realizer_has` / dlsym-by-name — UNCHANGED and vindicated.** The dispatch design was right; only the return type was wrong. - **`Geometry` — UNCHANGED, and relocated in meaning.** Every #141/#144 geometry test still passes verbatim. A vector was never wrong; it was misplaced. It is the right representation for a **component**, and was only ever wrong as the representation of a whole signal. ## What I did NOT get to - **`transduce_bytes` is still not a registered realizer.** It should return a real `Manifold` and ride the primitive like every other modality, instead of carrying a parallel node+edge implementation against the HTTP API. Renamed and documented, not converted. - **No audio realizer.** The contract is proven with a chord-of-note-letters realizer that decomposes into notes and intervals. A real acoustic realizer (MFCC per frame, onset detection, harmonic function) is the obvious next consumer and is not written. - **Signed edge weight** (the `:9730` clamp above) — diagnosed, not fixed. - **No persistence to a disk-backed engram.** All engram evidence is in-process/in-memory. The `/api/nodes/emb` read-back route on `feat/love-origin` is untouched and unmerged. - **`server.el` untouched.** Nothing in the HTTP surface exposes a Manifold yet; a transduced subgraph currently reaches the graph only through in-process El. ## /api/search — characterised, and the reported symptom is wrong Investigated on an isolated APFS clone (`cp -Rc`) served on port 8911 with PID/port ownership verified against `lsof`. Prod `:8742` and soul `:7770` were never touched. **It does not return the same cluster for every query.** Across 17 queries the union was 102 distinct ids and the **intersection was 0**. Two distinct behaviours: unmatched queries (`zzzznonsense`, `photosynthesis`) return **0 results**, not a cluster; matched queries often collapse onto the Working-tier skill-curriculum cluster while **discarding the query's own exact-label node**. `/api/search-lexical` is correct throughout — the lexical path is fine, the geometric gate is the defect. Root cause in `engram_retrieve_geometric_json`, `lang/runtime/el_runtime.c` (line numbers against 385c184): 1. **`:13622-13625` — dominant.** The region tag is applied as a **hard filter**, not a rank boost: `if(!keep) continue;`. The correct exact-label node is seeded and activated, then discarded for lacking the addressed tag. Only ~38 skill-curriculum nodes carry tags, so the answer collapses onto them. 2. **`:13575-13581`** — the coarse gate has no concentration threshold (unlike the fine branch at `:13570`), and truncating at the first `:` makes `dom_family` **always literally `"skill:"`** — it cannot tell `skill:programming` from `skill:arithmetic`. 3. **`:13533` + `:9288-9300`** — `engram_node_match_score` is case-insensitive **substring** match with no stopwords and no word boundaries, so `"self"` matches *"give every number times itself"*. Almost any common word seeds a tagged node and trips the gate. 4. **`:13626`** — `hits[nh].score=0` is hard-coded while `engram_rank_cmp` sorts on score first, so lexical evidence is discarded from ranking (compare `engram_search_json:13435`, which sets `score = sc` correctly). Not fixed here; out of scope for this PR. ## Build The committed CI recipe does not link on macOS. `-lssl -lcrypto` are unresolvable, and `el_runtime.c` alone leaves ~60 undefined symbols — `el_seed.c` (owns `el_request_end`) and the six `engram_*.c` units are required. ```bash ELC=lang/dist/platform/elc; RT=lang/runtime "$ELC" --test lang/tests/native/test_transduce.el > /tmp/t.c cc -O2 -I "$RT" /tmp/t.c \ "$RT/el_runtime.c" "$RT/el_seed.c" \ "$RT/engram_store.c" "$RT/engram_vindex.c" "$RT/engram_cognition.c" \ "$RT/engram_geometry.c" "$RT/engram_reason.c" "$RT/engram_verify.c" \ -lcurl -lpthread -lm -o /tmp/t && /tmp/t ``` No process was deployed, restarted, or signalled. Prod engram `:8742` and soul `:7770` were untouched throughout.

will.anderson added 2 commits 2026-08-16 20:50:12 +00:00

runtime: transduction decomposes a signal, it does not convert it d777936ee4

#144 moved transduction into the language and got the dispatch right. It got
the result type wrong: transduce(signal, modality) -> Geometry yields one
vector per signal, and one vector is a fingerprint. A fingerprint can be
matched and ranked; that is all. It cannot be decomposed, cannot have one part
grounded while another is not, and cannot be contradicted in one part while
holding in another, because it has no parts.

A song is not a point. It decomposes into pitch, interval, rhythm, harmonic
function -- components, each with its own geometry, plus the relations among
them. The song IS the structure of the relations.

transduce now returns a Manifold: named components carrying geometry, and
typed weighted relations between them. Signal in, subgraph out.

Components are addressed by key, never by index, because the key is what
survives persistence -- a component becomes a node and is separately groundable
precisely because it is separately named. Relation weight IS the grounding
(correspondence-and-censorship.md 1), so a realizer's relations arrive already
grounded and there is no score computed beside them.

ingest: name the inversion, and correct the worked example to decomposition

El SDK CI - dev / build-and-test (pull_request) Failing after 14m6s

Details

688f24b4c1

ingest.el's transduce() was renamed to transduce_manifold() earlier the same
day on the reasoning that it 'was never signal->geometry -- it chunks
already-extracted content and PACKS it into a node+edge manifold, one layer up,
and it had taken the name that belongs to the primitive underneath it.'

That reasoning was backwards. Producing a node+edge manifold is not a layer
above transduction, it IS transduction. Signal -> one vector is the operation
underneath, and its name is geometry. The layer doing it right was renamed out
of the way so the layer doing it wrong could have the name.

With the primitive corrected to return a Manifold, the two layers do the same
kind of thing and the inversion dissolves. What is left is a real distinction
about MODALITY, not layering: transduce() dispatches to a realizer that knows
its modality and can name its components; transduce_bytes() is the
opaque-bytes realizer, the decomposition available to a reader that knows
nothing about what it is reading. It still yields components and relations,
which is why it is transduction and not packing -- it just cuts on byte
boundaries, so its components are positional rather than meaningful. That is a
limitation of this realizer, not the definition of the operation.

Renamed by modality rather than demoted by layer. A distinct symbol is still
mechanically required: reusing transduce here is a conflicting-types error the
moment ingest.c links el_runtime.c.

lang/examples/transduce.el asserted #144's contract and would now fail, so it
is replaced by the decomposition worked example: transduce a chord, persist the
five components and six relations as real nodes and edges, read each part's
geometry back off its own node, and ground one part while its sibling is
demonstrably untouched.

will.anderson force-pushed fix/transduce-decomposition from c82a4d2f90 to 688f24b4c1

2026-08-16 20:50:12 +00:00

Compare

will.anderson merged commit 95a05109d1 into dev

2026-08-16 20:51:58 +00:00

will.anderson referenced this issue from a commit

2026-08-16 21:34:46 +00:00

engram: intake realizes a signal into a manifold, it does not assume a node

will.anderson referenced this pull request

2026-08-16 21:35:48 +00:00

engram: intake realizes a signal into a manifold, it does not assume a node #158

Sign in to join this conversation.