ingest: name the inversion, and correct the worked example to decomposition
El SDK CI - dev / build-and-test (pull_request) Failing after 11m43s
El SDK CI - dev / build-and-test (pull_request) Failing after 11m43s
ingest.el's transduce() was renamed to transduce_manifold() earlier the same day on the reasoning that it 'was never signal->geometry -- it chunks already-extracted content and PACKS it into a node+edge manifold, one layer up, and it had taken the name that belongs to the primitive underneath it.' That reasoning was backwards. Producing a node+edge manifold is not a layer above transduction, it IS transduction. Signal -> one vector is the operation underneath, and its name is geometry. The layer doing it right was renamed out of the way so the layer doing it wrong could have the name. With the primitive corrected to return a Manifold, the two layers do the same kind of thing and the inversion dissolves. What is left is a real distinction about MODALITY, not layering: transduce() dispatches to a realizer that knows its modality and can name its components; transduce_bytes() is the opaque-bytes realizer, the decomposition available to a reader that knows nothing about what it is reading. It still yields components and relations, which is why it is transduction and not packing -- it just cuts on byte boundaries, so its components are positional rather than meaningful. That is a limitation of this realizer, not the definition of the operation. Renamed by modality rather than demoted by layer. A distinct symbol is still mechanically required: reusing transduce here is a conflicting-types error the moment ingest.c links el_runtime.c. lang/examples/transduce.el asserted #144's contract and would now fail, so it is replaced by the decomposition worked example: transduce a chord, persist the five components and six relations as real nodes and edges, read each part's geometry back off its own node, and ground one part while its sibling is demonstrably untouched.
This commit is contained in:
+54
-25
@@ -13,7 +13,7 @@
|
||||
// relations add edges. Every node enters with PROVENANCE + grounding-level
|
||||
// + stewardship class from the moment of entry.
|
||||
//
|
||||
// transduce_manifold() is THE single mechanism — one function, polymorphic, with no
|
||||
// transduce_bytes() is THE single mechanism — one function, polymorphic, with no
|
||||
// content-type branch inside it. It does not ask whether a payload is
|
||||
// prose, structured data, or raw/opaque bytes (audio, or anything else);
|
||||
// it runs one boundary-scan-with-fixed-window-fallback chunking algorithm
|
||||
@@ -401,25 +401,54 @@ fn head80(s: String) -> String {
|
||||
// truncates at the first embedded NUL, which is routine in real binary
|
||||
// bytes) is a MECHANICAL fidelity concern that belongs to whatever produced
|
||||
// `source` (see ingest_file's file_source_string below) — not a
|
||||
// content-type judgment made in here. transduce_manifold() never learns whether a
|
||||
// content-type judgment made in here. transduce_bytes() never learns whether a
|
||||
// chunk is plain text or a base64-encoded raw-byte window; every chunk is
|
||||
// handled identically either way.
|
||||
// RENAMED transduce -> transduce_manifold (2026-08-16). Two reasons, and the
|
||||
// first is not the interesting one:
|
||||
// NAMING, CORRECTED 2026-08-16 (second pass). This function was renamed
|
||||
// `transduce` -> `transduce_bytes` earlier the same day, on the reasoning
|
||||
// that it "was never signal->geometry — it chunks already-extracted content
|
||||
// and PACKS it into a node+edge manifold, one layer up, and it had taken the
|
||||
// name that belongs to the primitive underneath it."
|
||||
//
|
||||
// 1. Mechanical: `transduce` is now a LANGUAGE primitive in el_runtime.h
|
||||
// (transduce(signal, modality) -> Geometry). Every El `fn name(...)`
|
||||
// compiles to a global C symbol with that exact name, so keeping this
|
||||
// name here is a hard `conflicting types for 'transduce'` compile error
|
||||
// the moment ingest.c links el_runtime.c. Measured, not anticipated.
|
||||
// THAT REASONING WAS BACKWARDS, and it is worth recording why rather than
|
||||
// quietly re-renaming. Producing a node+edge manifold is not a layer above
|
||||
// transduction — it IS transduction. Transduction is not conversion. When you
|
||||
// take in music you do not store the song as one discrete geometry; you break
|
||||
// it into its component parts and store the geometry of each along with the
|
||||
// relations between them. The song is the structure of those relations.
|
||||
// Signal -> one vector is the operation UNDERNEATH transduction, and its name
|
||||
// is encoding, or geometry. So the layer that was doing it right got renamed
|
||||
// out of the way so the layer doing it wrong could have the name.
|
||||
//
|
||||
// 2. Actual: this function was never signal->geometry. It chunks already-
|
||||
// extracted content and PACKS it into a node+edge manifold — a real
|
||||
// operation, but one layer up, and it had taken the name that belongs to
|
||||
// the primitive underneath it. `transduce` is where a signal becomes
|
||||
// geometry; `transduce_manifold` is where extracted content becomes
|
||||
// structure. Nothing about this function's behaviour changed.
|
||||
fn transduce_manifold(nodes: [String], edges: [String], source: String,
|
||||
// The primitive has since been corrected: `transduce(signal, modality)` now
|
||||
// returns a Manifold — components plus relations — not a Geometry
|
||||
// (el_runtime.c, "Manifold"). The two layers are therefore doing the SAME KIND
|
||||
// of thing, and the inversion dissolves rather than needing to be re-argued.
|
||||
//
|
||||
// What is left is a real distinction, and it is about MODALITY, not layering:
|
||||
//
|
||||
// * `transduce(signal, modality)` dispatches to a realizer that KNOWS the
|
||||
// modality and can name its components — for audio: pitch, interval,
|
||||
// rhythm, harmonic function.
|
||||
// * `transduce_bytes` below is the OPAQUE-BYTES realizer: the decomposition
|
||||
// available to a reader that knows nothing about what it is reading. It
|
||||
// still yields components and relations (chunk nodes; contains / precedes
|
||||
// / section_of edges), which is why it is transduction and not packing. It
|
||||
// just cuts on the only structure visible without understanding — byte
|
||||
// boundaries — so its components are positional rather than meaningful.
|
||||
// That is a LIMITATION of this realizer, not the definition of the
|
||||
// operation.
|
||||
//
|
||||
// The name is suffixed by its modality, not demoted to a lesser layer. Keeping
|
||||
// a distinct symbol is also still mechanically required: every El `fn name`
|
||||
// compiles to a global C symbol, so reusing `transduce` here is a hard
|
||||
// `conflicting types` error the moment ingest.c links el_runtime.c.
|
||||
//
|
||||
// WHERE THIS SHOULD GO: this function should become a registered realizer
|
||||
// returning a real Manifold, so ingest rides the same primitive as every other
|
||||
// modality instead of carrying a parallel implementation. Not done here.
|
||||
// Nothing about this function's behaviour changed in this pass.
|
||||
fn transduce_bytes(nodes: [String], edges: [String], source: String,
|
||||
prov: String, ground: String, steward: String,
|
||||
root_lid: String, root_title: String) -> [String] {
|
||||
let tagbase: String = "prov:" + prov + " ground:" + ground + " steward:" + steward
|
||||
@@ -546,8 +575,8 @@ fn default_steward() -> String {
|
||||
// trustworthy verbatim. When they don't (silent truncation happened),
|
||||
// rebuild the payload as base64-encoded fixed-size windows read directly
|
||||
// off disk (fs_read_b64_chunk — binary-safe in C), joined with the same
|
||||
// "\n\n" boundary marker transduce_manifold()'s generic scan already looks for, so
|
||||
// transduce_manifold() sees one ordinary boundary-delimited payload and runs its one
|
||||
// "\n\n" boundary marker transduce_bytes()'s generic scan already looks for, so
|
||||
// transduce_bytes() sees one ordinary boundary-delimited payload and runs its one
|
||||
// algorithm on it exactly as it would on prose — it never learns that a
|
||||
// fidelity problem occurred upstream, let alone why.
|
||||
fn file_source_string(path: String, text: String, real_size: Int) -> String {
|
||||
@@ -556,7 +585,7 @@ fn file_source_string(path: String, text: String, real_size: Int) -> String {
|
||||
// 3072 raw bytes -> 4096 base64 chars (3 divides evenly into base64's
|
||||
// 3-byte/4-char ratio); keeps each resulting node's content a clean,
|
||||
// bounded, low-kilobytes unit, same order of magnitude as the fixed
|
||||
// fallback window in transduce_manifold() itself.
|
||||
// fallback window in transduce_bytes() itself.
|
||||
let win: Int = 3072
|
||||
let out: String = ""
|
||||
let off: Int = 0
|
||||
@@ -576,7 +605,7 @@ fn file_source_string(path: String, text: String, real_size: Int) -> String {
|
||||
}
|
||||
|
||||
// ingest one file -> report JSON. Uniform for every file regardless of
|
||||
// extension or content — transduce_manifold() decides nothing about content-type, so
|
||||
// extension or content — transduce_bytes() decides nothing about content-type, so
|
||||
// neither does this function; it only decides whether the raw bytes made it
|
||||
// through the read intact (file_source_string), which is a fidelity
|
||||
// question, not a format one.
|
||||
@@ -588,14 +617,14 @@ fn ingest_file(path: String) -> String {
|
||||
return "{\"error\":\"empty or unreadable\",\"path\":" + j_q(path) + "}"
|
||||
}
|
||||
let prov: String = "file:" + path
|
||||
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
|
||||
let packed: [String] = transduce_bytes(el_list_empty(), el_list_empty(),
|
||||
source, prov, default_ground(), default_steward(),
|
||||
"doc:" + basename(path), basename(path))
|
||||
return merge_packed(packed)
|
||||
}
|
||||
|
||||
// ingest a directory: walk one level, ingest every file found, aggregate.
|
||||
// No extension filter — transduce_manifold() handles any payload uniformly now, so
|
||||
// No extension filter — transduce_bytes() handles any payload uniformly now, so
|
||||
// there is no content-type gate at the directory boundary either.
|
||||
fn ingest_dir(path: String) -> String {
|
||||
let entries: [String] = fs_list(path)
|
||||
@@ -630,7 +659,7 @@ fn ingest_dir(path: String) -> String {
|
||||
fn ingest_url(url: String) -> String {
|
||||
let body: String = http_get(url)
|
||||
if str_eq(body, "") { return "{\"error\":\"empty fetch\",\"url\":" + j_q(url) + "}" }
|
||||
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
|
||||
let packed: [String] = transduce_bytes(el_list_empty(), el_list_empty(),
|
||||
body, "url:" + url, "extracted", "public-web",
|
||||
"url:" + url, url)
|
||||
return merge_packed(packed)
|
||||
@@ -645,7 +674,7 @@ fn ingest_llm(query: String) -> String {
|
||||
let resp: String = http_post_json("http://127.0.0.1:11434/api/generate", body)
|
||||
let answer: String = json_get_string(resp, "response")
|
||||
if str_eq(answer, "") { return "{\"error\":\"no model response\"}" }
|
||||
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
|
||||
let packed: [String] = transduce_bytes(el_list_empty(), el_list_empty(),
|
||||
answer, "llm:" + model + ":" + query, "candidate-provisional", "guide-provisional",
|
||||
"llm:" + query, "guide answer: " + query)
|
||||
return merge_packed(packed)
|
||||
@@ -697,7 +726,7 @@ fn ingest_stream(path: String) -> String {
|
||||
// It is NOT a content-type flag: it says nothing about what's inside the
|
||||
// bytes once fetched, and none of the five ingest_* functions it selects
|
||||
// among interpret their payload differently by content shape anymore —
|
||||
// they all hand off to the single, format-agnostic transduce_manifold(). The old
|
||||
// they all hand off to the single, format-agnostic transduce_bytes(). The old
|
||||
// "structured" value (a caller-declared alias for "file", used only to hint
|
||||
// the now-removed JSON-vs-prose branch) is gone along with that branch.
|
||||
let kind: String = env("INGEST_KIND")
|
||||
|
||||
Reference in New Issue
Block a user