runtime: transduction is a language concern, so move it into the language
El SDK CI - dev / build-and-test (pull_request) Failing after 14m58s
El SDK CI - dev / build-and-test (pull_request) Failing after 14m58s
#141 let signal enter as geometry and it worked, but it was placed at the CONSUMER and said so in its own commit message. This is the correction. Three defects, all of them placement: 1. It sat in the engram. Ingest is a LANGUAGE concern — every el program touching any modality needs it, and the engram is merely one el program that happens to hold a graph. The geometry surface is now defined in el_runtime.c immediately ABOVE the engram section and depends on nothing inside it. Delete the entire engram and geometry still enters el. 2. It marshalled the vector as a hex STRING, because el had no first-class geometry value — which reintroduced text as the TRANSPORT medium one layer below the problem being fixed. Geometry is now an el value: a magic-tagged heap object carried in el_val_t, same discipline as List/Map. Hex survives only as an adapter at the edge, which is all an encoding should ever be. 3. It needed an arbitrary `dim <= 8192` bound purely to size an allocation from a caller's CLAIM about a string's length. A value carries its own width, so the width is derived and never asserted. The bound is gone, not raised — there is nothing left to validate. Language surface, none of it engram-prefixed: geometry_new / _dim / _is / _get / _set / _norm / _free, geometry_from_f32le_hex + geometry_to_f32le_hex as the wire adapters, realizer_register(modality, fn_name), realizer_has, and transduce(signal, modality) -> Geometry. REALIZERS ARE DECLARABLE IN EL. This is the part that makes the move real rather than nominal: registration resolves a name with dlsym against the running binary, the identical mechanism http_set_handler already relies on, because every el `fn name(...)` compiles to a global C symbol with that exact name. So an ordinary el function IS a realizer and a new modality needs no runtime patch. Verified end to end in lang/examples/transduce.el: an el-defined tone_realizer is registered by name, transduce dispatches to it, and the signal demonstrably reaches it (distinct signals produce distinct geometry). A modality with no realizer transduces to NOTHING. There is deliberately no built-in realizer, not even for text — silently embedding a description of a signal and calling that perception is the exact defect this ends. engram/src/server.el is migrated: POST /api/nodes decodes "emb" hex exactly once, at the edge, into a Geometry, and everything below that line moves geometry. The wire is unchanged because production clients speak it. "dim" is now an ASSERTION about the vector, not the source of its width; disagreement is a rejected ingest, not a silent reinterpretation. #141's engram_node_set_emb becomes a DEPRECATED WRAPPER over geometry_from_f32le_hex + node_attach_geometry — kept only because the runtime ships as an SDK asset and a downstream binary may link the symbol. Its exact contract, negative cases included, is preserved and re-verified. ingest.el's `fn transduce` is renamed transduce_manifold. Mechanically it had to yield the name (duplicate C symbol, a hard compile error, measured). But it was never signal->geometry: it chunks already-extracted content into a node+edge manifold, one layer up, and had taken the name belonging to the primitive underneath it. Behaviour unchanged. PROPERTIES FROM #141 PRESERVED, each re-measured on a scratch engram (:8971, never prod :8742): - off-dimension vectors stored but NOT indexed — the HNSW build loop still filters on n->emb_dim == dim at four sites, so a 64-dim voice vector is durable and addressable without perturbing the 768-dim canonical index - geometry makes a node ineligible for embed_backfill: after backfill the 64-dim voice node was still 64-dim while the text control acquired 768 - the create response reports whether geometry landed, and the node document always emits emb_dim and embedded Read-back with control and negatives, all verified against a PID-confirmed fresh binary: geometry node emb_dim=64 embedded=true / emb_set=1; text-only control emb_dim=0 embedded=false / emb_set=0; malformed hex, ragged length, and dim-disagreement each emb_set=0. Two compiler landmines found by reading the generated C rather than trusting a successful build, both documented at their sites: elc lowers `a == b` to str_eq unless both operand NAMES are in the per-function int-name set (which does NOT propagate into nested if-expression blocks — the first cut would have strcmp'd two integers as pointers on the first geometry-bearing request), and `+` lowers to string concat when either operand is a user-defined call.
This commit is contained in:
+27
-12
@@ -13,7 +13,7 @@
|
||||
// relations add edges. Every node enters with PROVENANCE + grounding-level
|
||||
// + stewardship class from the moment of entry.
|
||||
//
|
||||
// transduce() is THE single mechanism — one function, polymorphic, with no
|
||||
// transduce_manifold() is THE single mechanism — one function, polymorphic, with no
|
||||
// content-type branch inside it. It does not ask whether a payload is
|
||||
// prose, structured data, or raw/opaque bytes (audio, or anything else);
|
||||
// it runs one boundary-scan-with-fixed-window-fallback chunking algorithm
|
||||
@@ -401,10 +401,25 @@ fn head80(s: String) -> String {
|
||||
// truncates at the first embedded NUL, which is routine in real binary
|
||||
// bytes) is a MECHANICAL fidelity concern that belongs to whatever produced
|
||||
// `source` (see ingest_file's file_source_string below) — not a
|
||||
// content-type judgment made in here. transduce() never learns whether a
|
||||
// content-type judgment made in here. transduce_manifold() never learns whether a
|
||||
// chunk is plain text or a base64-encoded raw-byte window; every chunk is
|
||||
// handled identically either way.
|
||||
fn transduce(nodes: [String], edges: [String], source: String,
|
||||
// RENAMED transduce -> transduce_manifold (2026-08-16). Two reasons, and the
|
||||
// first is not the interesting one:
|
||||
//
|
||||
// 1. Mechanical: `transduce` is now a LANGUAGE primitive in el_runtime.h
|
||||
// (transduce(signal, modality) -> Geometry). Every El `fn name(...)`
|
||||
// compiles to a global C symbol with that exact name, so keeping this
|
||||
// name here is a hard `conflicting types for 'transduce'` compile error
|
||||
// the moment ingest.c links el_runtime.c. Measured, not anticipated.
|
||||
//
|
||||
// 2. Actual: this function was never signal->geometry. It chunks already-
|
||||
// extracted content and PACKS it into a node+edge manifold — a real
|
||||
// operation, but one layer up, and it had taken the name that belongs to
|
||||
// the primitive underneath it. `transduce` is where a signal becomes
|
||||
// geometry; `transduce_manifold` is where extracted content becomes
|
||||
// structure. Nothing about this function's behaviour changed.
|
||||
fn transduce_manifold(nodes: [String], edges: [String], source: String,
|
||||
prov: String, ground: String, steward: String,
|
||||
root_lid: String, root_title: String) -> [String] {
|
||||
let tagbase: String = "prov:" + prov + " ground:" + ground + " steward:" + steward
|
||||
@@ -531,8 +546,8 @@ fn default_steward() -> String {
|
||||
// trustworthy verbatim. When they don't (silent truncation happened),
|
||||
// rebuild the payload as base64-encoded fixed-size windows read directly
|
||||
// off disk (fs_read_b64_chunk — binary-safe in C), joined with the same
|
||||
// "\n\n" boundary marker transduce()'s generic scan already looks for, so
|
||||
// transduce() sees one ordinary boundary-delimited payload and runs its one
|
||||
// "\n\n" boundary marker transduce_manifold()'s generic scan already looks for, so
|
||||
// transduce_manifold() sees one ordinary boundary-delimited payload and runs its one
|
||||
// algorithm on it exactly as it would on prose — it never learns that a
|
||||
// fidelity problem occurred upstream, let alone why.
|
||||
fn file_source_string(path: String, text: String, real_size: Int) -> String {
|
||||
@@ -541,7 +556,7 @@ fn file_source_string(path: String, text: String, real_size: Int) -> String {
|
||||
// 3072 raw bytes -> 4096 base64 chars (3 divides evenly into base64's
|
||||
// 3-byte/4-char ratio); keeps each resulting node's content a clean,
|
||||
// bounded, low-kilobytes unit, same order of magnitude as the fixed
|
||||
// fallback window in transduce() itself.
|
||||
// fallback window in transduce_manifold() itself.
|
||||
let win: Int = 3072
|
||||
let out: String = ""
|
||||
let off: Int = 0
|
||||
@@ -561,7 +576,7 @@ fn file_source_string(path: String, text: String, real_size: Int) -> String {
|
||||
}
|
||||
|
||||
// ingest one file -> report JSON. Uniform for every file regardless of
|
||||
// extension or content — transduce() decides nothing about content-type, so
|
||||
// extension or content — transduce_manifold() decides nothing about content-type, so
|
||||
// neither does this function; it only decides whether the raw bytes made it
|
||||
// through the read intact (file_source_string), which is a fidelity
|
||||
// question, not a format one.
|
||||
@@ -573,14 +588,14 @@ fn ingest_file(path: String) -> String {
|
||||
return "{\"error\":\"empty or unreadable\",\"path\":" + j_q(path) + "}"
|
||||
}
|
||||
let prov: String = "file:" + path
|
||||
let packed: [String] = transduce(el_list_empty(), el_list_empty(),
|
||||
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
|
||||
source, prov, default_ground(), default_steward(),
|
||||
"doc:" + basename(path), basename(path))
|
||||
return merge_packed(packed)
|
||||
}
|
||||
|
||||
// ingest a directory: walk one level, ingest every file found, aggregate.
|
||||
// No extension filter — transduce() handles any payload uniformly now, so
|
||||
// No extension filter — transduce_manifold() handles any payload uniformly now, so
|
||||
// there is no content-type gate at the directory boundary either.
|
||||
fn ingest_dir(path: String) -> String {
|
||||
let entries: [String] = fs_list(path)
|
||||
@@ -615,7 +630,7 @@ fn ingest_dir(path: String) -> String {
|
||||
fn ingest_url(url: String) -> String {
|
||||
let body: String = http_get(url)
|
||||
if str_eq(body, "") { return "{\"error\":\"empty fetch\",\"url\":" + j_q(url) + "}" }
|
||||
let packed: [String] = transduce(el_list_empty(), el_list_empty(),
|
||||
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
|
||||
body, "url:" + url, "extracted", "public-web",
|
||||
"url:" + url, url)
|
||||
return merge_packed(packed)
|
||||
@@ -630,7 +645,7 @@ fn ingest_llm(query: String) -> String {
|
||||
let resp: String = http_post_json("http://127.0.0.1:11434/api/generate", body)
|
||||
let answer: String = json_get_string(resp, "response")
|
||||
if str_eq(answer, "") { return "{\"error\":\"no model response\"}" }
|
||||
let packed: [String] = transduce(el_list_empty(), el_list_empty(),
|
||||
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
|
||||
answer, "llm:" + model + ":" + query, "candidate-provisional", "guide-provisional",
|
||||
"llm:" + query, "guide answer: " + query)
|
||||
return merge_packed(packed)
|
||||
@@ -682,7 +697,7 @@ fn ingest_stream(path: String) -> String {
|
||||
// It is NOT a content-type flag: it says nothing about what's inside the
|
||||
// bytes once fetched, and none of the five ingest_* functions it selects
|
||||
// among interpret their payload differently by content shape anymore —
|
||||
// they all hand off to the single, format-agnostic transduce(). The old
|
||||
// they all hand off to the single, format-agnostic transduce_manifold(). The old
|
||||
// "structured" value (a caller-declared alias for "file", used only to hint
|
||||
// the now-removed JSON-vs-prose branch) is gone along with that branch.
|
||||
let kind: String = env("INGEST_KIND")
|
||||
|
||||
Reference in New Issue
Block a user