engram: intake realizes a signal into a manifold, it does not assume a node #158
Reference in New Issue
Block a user
Delete Branch "wire/write-realizes-signal"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The measured defect
transduce(signal, modality)was corrected in #155 to return a Manifold — components plus relations — instead of a single Geometry. That PR touchedingest/src/ingest.el,lang/runtime/el_runtime.c,lang/runtime/el_runtime.h,lang/examples/transduce.eland tests. It never touchedengram/src/server.el.The primitive was corrected and nothing downstream called it:
One line. A comment. The engram's entire HTTP surface never called transduce, never called realize, never mentioned Manifold.
The flattening was one line, in
route_write:A request body is not a manifold. The comment stated the wrong assumption out loud. There is no write node — what arrives at an intake route is a SIGNAL, and a node is an OUTPUT of realization, never an INPUT to it. How many nodes a signal becomes is for the realizer to say, not for the route to assume.
What was wired to what
route_write→intake_signal→transduce(signal, modality)→ components and relations land in the store.Nothing here decomposes anything, and nothing here may ever.
transducedispatches through the dlsym realizer registry (el_runtime.c:6771-6851), so adding a modality is registering a realizer — never an edit toserver.el, never a patch to the runtime.intake_signalonly carries what the primitive returns:node_attach_geometryrather than re-embedding its name as text;manifold_memberstill wires the inserted set into one connected sub-graph, exactly asinsert_manifold_jsonalready did (server.el:1577-1583). Reused, not reinvented.Built general, not special-cased to
route_write:nt/tier/tags/region/reason/rebindare parameters, so the other five intake doors can move onto this one function as they are transitioned.Before / after — raw JSON, same signal
Own port
:19501,cp -Rcclone of~/.neuron/engram. Production untouched.BEFORE (dev @
95a0510):Read back out, not taken on trust:
AFTER, with no organ registered (this branch, unmodified):
Production behaviour is unchanged, and that is the honest result. No realizer is authored in this diff and none is registered, so
transducereturns 0 and the signal is stored flat exactly as before. What changed is that the response says so:"realized":false,"organ":false. Silent flattening was the actual defect — a caller could not distinguish "nothing decomposed me" from "I decomposed into one component".el_runtime.cdraws the same line between an absent organ and a broken one, for the same reason: those two must not look alike.The mechanism is what landed. Writes do not decompose yet.
Proof the seam actually lands components and relations
Since no organ ships here, the insertion path was exercised with a scratch, uncommitted harness: a throwaway copy of
server.elwithtone_realizercopied verbatim fromlang/examples/transduce.eland registered at startup. The committed diff contains no realizer; this only proves the seam carries what a realizer returns.POST /api/write {"_auth":"k","modality":"tone","content":"CEG"}Components read back out of the store (
GET /api/node/<id>):note:0["component","role:pitch","modality:tone"]note:1["component","role:pitch","modality:tone"]note:2["component","role:pitch","modality:tone"]interval:0-1["component","role:interval","modality:tone"]interval:1-2["component","role:interval","modality:tone"]Note the widths: notes are 2-wide, intervals are 1-wide. A single fingerprint vector per signal cannot represent parts of unequal width at all — that difference is the point of a manifold over a point.
Relations read back out of the persisted snapshot (all 10 edges, from
snapshot.jsonafterpersist_canonical):6 realizer relations + 4
manifold_member= 10, matchingedges_added. The weights are exactly what the realizer stated — the grounding rode through untouched.(Read-surface note:
GET /api/neighbors/<id>returns one record per neighbour node, so it collapsed the twosounds_beforeedges behind themanifold_memberedge on the same pair and showed 8 of 10. The edges are all present in the durable snapshot. That is a limitation of the neighbors read route, not of the write — worth a separate fix.)Same cathedral signal through the same seam:
1 node / 0 edges → 137 components / 204 relations.
toneis obviously the wrong organ for an English sentence — its components are per-character and mean nothing. It proves the seam carries N components and M relations into the store; it does not claim the sentence was understood.What still flattens
/api/writeitself, in production. No organ is registered for any modality, so every real write still lands flat.el_runtime.c:6828makes that deliberate: "There is deliberately NO built-in realizer, not even for text." Repo-wide,realizer_register/-> Manifold/manifold_addappear in exactly 5 files: the runtime pair (the primitive),lang/examples/transduce.elandlang/tests/native/test_transduce.el(demo organs), andingest/src/ingest.el(a comment only). Nothing registers an organ anywhere./api/supersede(the identical"[" + body + "]"line),/api/nodes,/api/neuron/knowledge/capture,/api/neuron/state-events,/api/nodes/reseed./api/reframeaccepts a caller-authored manifold — the client does the decomposing, which is the client doing the engram's job. Left alone deliberately: it is currently the only route that carries components and the only working reference./api/nodesis two intakes through one door:content(raw signal) andemb(already-realized geometry from somebody else's realizer) arrive together./api/search,/api/search-lexicaland/api/activatebelong to the same door on the read side. Not touched here — non-mutating, higher blast radius. Related:/api/searchis currently a lexical token-substring scan rather than embedding-nearest, and is broken for multi-token queries — the same disease, a hand-coded path where the primitive belonged./api/neuron/state-eventsleft deliberately: telemetry, auth-exempt, high-volume, and already has an off-graph log mode. Decomposing every internal state event is a volume problem, not a correctness win.The redundancy underneath all of this
"you coded a bunch of shit in engram that belonged in el, and then, when i fixed that, you didn't actually transition engram to use the el primitives."
el_runtime.hdeclares 93 nativeengram_*builtins (counted asel_val_t engram_*declarations, not raw occurrences of the string). Roughly 50 of them already have a route wrapping them 1:1 — argument shuffling and JSON quoting around a function already linked into the same binary. These are deletion candidates, not refactor candidates:write,supersede,reframe,nodes,knowledge/capture,state-eventsall bottom out inengram_node_full, differing only in thenode_type/tier/tagsthey hardcode./api/think,/reason,/induce,/abduce,/relate,/analogize,/planall land onengram_think_jsonviaroute_think+route_faculty. That is literally "there are not 15 transduce endpoints because transduce is one thing", with faculties in place of transduce.route_get_nodeandroute_get_node_singularboth callengram_get_node_json.route_stats→engram_stats_json,route_neighbors→engram_neighbors_json,route_strengthen→engram_strengthen,route_reify→engram_geo_reify_run_json,route_gauge_distance→engram_geo_distance_json,route_recognize→engram_geo_overlap_json,route_discern→engram_geo_subtract_json,route_synthesize→engram_geo_combine_json,route_nearest→engram_nearest_json,route_ground*→engram_ground*_json,route_attend→engram_attend_json,route_similarity→engram_cosine_sim, and ~20 more.One thing that is not redundancy
ingest/src/ingest.elreaches the engram over HTTP (eg_base()→http://127.0.0.1:8902, a port nothing is listening on) to call builtins that are linked into its own binary. That looks like the same redundancy from the other side. It is not, and converting those calls to nativeengram_*calls would silently destroy data.engram_node_fulland friends are implemented inel_runtime.c(9154, 13856, 16223, 16586) under "Batch 3: Engram in-process graph store — Single global EngramStore allocated lazily on first call": a process-global in-memory store. ingest is a separate process — its own header states it "links only el_runtime.c ... it never links el_seed.c or the engram engine." A native call from ingest would mint a node in ingest's own ephemeral store, return a valid-looking id, and lose it on exit — every ingest reporting success and writing nothing. That is the same defect class as the "200 with a fresh id while the vector was discarded" bug already documented atserver.el:358-366. And writing the server's data dir from a second process is corruption, not a shortcut.The HTTP hop between ingest and engram is IPC, not redundancy. The redundancy is entirely inside
server.el, in the same address space. The correct direction is that ingest should not be a separate process at all: the file/url/query/string intake belongs inside the engram, and when it lands there the HTTP client disappears because the client disappears, not because its calls were rewritten. Deletion in the right direction, and safe.Build and test discipline
Local build is the bar; CI is not.
Compiles clean (
elcrc=0; the onlyccwarning is the pre-existing'/*' within block commentinel_runtime.c:14439). Every run used its own port:19501and a freshcp -Rcclone.:8742,:18760,:7770,:7779,:17779were never bound and production was never touched.One measurement worth flagging as a divergence: on this snapshot
embedded_countwas 0 againstembed_eligible_count25,629 before any write — so on this data set the engram had not encoded its eligible nodes at all. It rose to 142 purely from the components this change inserted with attached geometry. That differs from the 1:1embedded_count == embed_eligible_countreported on prod:8742, and the gap between the two data sets is worth a separate look.There is no write node. What arrives at /api/write is a SIGNAL; a node is an OUTPUT of realization, never an INPUT to it. route_write asserted otherwise in one line: let manifold: String = "[" + body + "]" // the body IS a valid manifold node object A request body is not a manifold, and that assertion is the whole defect. It is why every written signal landed as one flat node with zero edges, measured on a clone: {"inserted":1,"nodes_added":1,"edges_added":0} and GET /api/neighbors on the new id returning []. PR #155 corrected transduce(signal, modality) to return a Manifold — components plus relations — but touched only ingest, the runtime and its tests. Nothing downstream called it: grep 'transduce|realize|Manifold|decompos' over engram/src/server.el returned exactly one line, a comment. The primitive was fixed and the engram's entire HTTP surface never reached for it. This wires the intake seam to the primitive that already exists. It decomposes nothing itself and must never: transduce dispatches through the dlsym realizer registry, so adding a modality is registering a realizer, not editing this file and not patching the runtime. intake_signal only carries what the primitive returns into the store — components become nodes carrying their OWN geometry via node_attach_geometry, relations become edges at the weight the realizer stated, and manifold_member still wires the set into one connected sub-graph exactly as insert_manifold_json already did. Built general rather than special-cased: five of the six intake doors (write, supersede, nodes, knowledge/capture, state-events) are the same hand-written "content -> engram_node_full -> one flat node", differing only in the node_type/tier/tags they hardcode. Those are parameters here so each door can move onto this one function. Only /api/write rides it in this pass. When no organ is registered the signal is stored flat exactly as before, but the response now says so ("realized":false,"organ":false,"components":0). Silent flattening was the real defect — a caller could not tell "nothing decomposed me" from "I decomposed into one component". el_runtime.c draws the same line between an absent organ and a broken one, for the same reason. No realizer is authored here and none is registered, so production behaviour is unchanged. The mechanism is what landed.