Compare commits

..

10 Commits

Author SHA1 Message Date
Neuron cace6a5ebf runtime: a disconnecting client must not kill the server
El SDK CI - dev / build-and-test (pull_request) Failing after 13m45s
There was no SIGPIPE handling anywhere in this runtime: no signal
disposition, no MSG_NOSIGNAL, no SO_NOSIGPIPE, and send() called with bare
flags. The default disposition of SIGPIPE is to TERMINATE THE PROCESS, so
any client that hangs up mid-response takes the whole engram with it.

MEASURED, and it is not hypothetical. Production has restarted 254 times
since 2026-08-13T19:37 at a flat ~10 minute cadence:

  17:05:18  17:15:29  17:25:38  17:35:50  17:46:00  17:56:10  18:06:22  18:16:30

Intervals of 10m09s-10m12s, not 10m00s. That excess is the whole story:
ai.neuron.engram-tick has StartInterval 600, and engram-tick.sh:13 calls

  curl -s -m10 -X POST .../api/tick

The beat does not finish within 10s over 13,634 nodes, so curl waits its
full timeout and closes. The engram then writes the tick response to a dead
socket, takes SIGPIPE, and dies. launchd KeepAlive restarts it, so the
failure presents as a mysterious restart rather than a crash — and
~/.neuron/logs/engram.log records nothing but "[http] listening on" 254
times, with no exit reason. launchctl list confirms the last exit as -13.

Root cause is one level out: consolidation had no owner, so an external
ticker was created to poke it, and the ticker is what kills it. The fix
here does not address that; it makes the process survivable while it is
addressed.

Two layers, because neither alone is portable:
  - SO_NOSIGPIPE per accepted socket (Darwin/BSD) and MSG_NOSIGNAL per send
    (Linux), so the signal is never raised for socket writes at all.
  - A process-wide SIG_IGN backstop, installed once and idempotent, for
    platforms and paths with neither. With the signal ignored, send()
    returns -1/EPIPE and the existing error path closes the connection.

Also retries send() on EINTR, which the previous loop treated as fatal.

This is an exemption in the sense of lang/spec §8: the write never checked
whether the peer was still there, and the consequence of not checking was
fatal rather than merely wrong.
2026-08-16 13:25:14 -05:00
will.anderson d41645388a runtime: make valid UTF-8 the JSON emitter's contract (#148)
El SDK CI - dev / build-and-test (push) Failing after 4m12s
El SDK CI - dev / build-and-test (pull_request) Failing after 4m37s
2026-08-16 17:03:44 +00:00
Neuron 8a307dfd42 runtime: make valid UTF-8 the JSON emitter's contract
El SDK CI - dev / build-and-test (pull_request) Failing after 10m36s
Three nodes in the live graph carry labels truncated to exactly 80 bytes
ending in a lone 0xE2 — the first byte of an em-dash, cut mid-sequence.
jb_emit_escaped copied every byte >= 0x20 through verbatim, so those three
nodes made the ENTIRE /api/nodes/list response undecodable and no strict
parser could read the graph at all.

  production binary   25,929,607 bytes   INVALID at byte 89260
  this build          26,338,389 bytes   VALID, parses to 13,630 nodes

The damage was NOT written by this runtime. No 80-byte truncation exists
here (the only label truncation is engram_first_n_chars at 60), and the
content of those nodes is 2572 and 2746 bytes. Some other producer wrote
them. That is exactly why fixing a writer could not have fixed this: the
store already holds the damage, and it accepts data from importers, other
producers and older binaries.

So the fix goes where the promise is made. A serializer that emits JSON
owes valid UTF-8 whatever it is handed. jb_emit_escaped now validates each
multi-byte sequence before emitting any of it and substitutes U+FFFD for a
bad lead byte, a missing or malformed continuation, an overlong encoding, a
UTF-16 surrogate, or a codepoint above U+10FFFF. Invalid bytes are REPLACED
rather than dropped, so the damage stays visible in the output instead of
being silently papered over. Well-formed input is byte-identical to before.

Second, preventive and explicitly NOT the cause of the above:
engram_first_n_chars truncated by BYTES despite its name, so content with a
multi-byte character crossing byte 60 would produce a half codepoint in the
label. It now uses el_utf8_safe_len, which returns the largest byte length
<= max that does not split a codepoint. Bounded by bytes, not codepoints,
so existing labels never grow — they only stop splitting.

el_utf8_safe_len lives beside str_count_chars rather than in the engram
because the rest of el's string layer is already codepoint-aware
(str_count_chars counts codepoints, str_reverse walks codepoint lengths).
Byte truncation was the outlier and the concern is a string concern.

Note on the investigation: I first "fixed" the truncator and wrote a test
that passed on the UNPATCHED build too, because route_create_node passes
label = content when no label is supplied, so engram_first_n_chars is never
reached over HTTP. The test proved nothing. The real cause was only found
by decoding the actual failing bytes out of the live response.
2026-08-16 12:03:03 -05:00
will.anderson 616815b2ab Give cross-cutting concerns an owner instead of a convention (#145)
El SDK CI - dev / build-and-test (push) Failing after 11m4s
2026-08-16 16:57:51 +00:00
will.anderson 1a8a966cb3 runtime: transduction is a language concern, so move it into the language (#144)
El SDK CI - dev / build-and-test (push) Failing after 11m29s
2026-08-16 16:57:35 +00:00
will.anderson 1f70b9fa18 runtime: ground the node asked about, and refuse circular support (#147)
El SDK CI - dev / build-and-test (push) Failing after 14m46s
2026-08-16 16:54:17 +00:00
Neuron 317466e8f7 runtime: ground the node asked about, and refuse circular support
El SDK CI - dev / build-and-test (pull_request) Failing after 15m5s
engram_ground_json resolved each seed to a REGION, wrote the grounded-by
edge between the two regions' HUBS, and then echoed those hubs back in the
"claim"/"evidence" fields as if they were the caller's input:

    const char* cid = C->hub_id ? C->hub_id : EL_CSTR(claim);
    const char* eid = E->hub_id ? E->hub_id : EL_CSTR(evidence);
    cog_ground_edge(g_engram_store, cid, eid, grounding, fw);

Three consequences, all measured against a clone of the live store:

1. The edge landed on a node the caller never named. Grounding 3b9ced5d
   against 6edf8c79 wrote an edge on the hubs of their regions instead.
2. When both seeds resolve into the same region the support is circular
   and scores near 1.0 for structural reasons, not evidential ones. Four
   probe nodes written together landed in one region, and every grounding
   among them returned 0.93-0.99 as if it were evidence. Two independent
   agents hit this and reported 0.885 / 0.909 self-groundings as confident.
3. The echo concealed both: the response was indistinguishable from a
   successful grounding of the ids that were passed in.

The region is HOW a claim is evaluated; it is not WHAT the claim is about.
So the edge now attaches to the requested ids, and the resolved hubs are
reported separately as claim_region / evidence_region.

Degeneracy is broader than hub == hub. Three circular shapes, all
previously invisible:
    same-region                both seeds resolve to one region
    claim-region-is-evidence   the evidence IS the hub of the claim's own
                               neighbourhood — measured at 0.98883
    evidence-region-is-claim   the mirror case
Each sets grounding to 0 and writes no edge. Circular support is not
support, and a grounding that is degenerate by construction must not
enter the graph as though it were evidence.

Verified:
  6edf8c79 -> 6edf8c79   degenerate=same-region   g=0        written=false
  6edf8c79 -> d0406dfd   degenerate=same-region   g=0        written=false
  ebc1413e -> 64cc96ef   degenerate=false         g=0.774563 written=true
  64cc96ef -> ebc1413e   degenerate=false         g=0.802896 written=true
Legitimate grounding across distinct regions is unchanged and still
writes; only circular support is refused.

This is the same class as #142 and #146 — a value that looked like an
answer with nothing behind it — except here it was also writing that
non-answer into the canonical store.
2026-08-16 11:53:31 -05:00
will.anderson eb3e6d7c1f runtime: resume the learned stance in think (#146)
El SDK CI - dev / build-and-test (push) Failing after 3m54s
2026-08-16 16:44:15 +00:00
Neuron 88e3008735 runtime: resume the learned stance in think, instead of discarding it
El SDK CI - dev / build-and-test (pull_request) Failing after 4m16s
engram_think_json built a NEUTRAL stance on every call — cog_stance_init
with a NULL id, all axis_gain 1.0, bias_dir NULL, reliability 0.5 — and
never loaded the stance the correspondence-beat had been persisting.

That mattered because the faculty enters engram_think ONLY through the
stance: axis_gain[k] warps the per-axis extents and bias_dir seeds the
steering direction. cog_stance_init stores the faculty NAME and nothing
reads it. So with a neutral stance, reason/abduce/induce/plan/analogize
were byte-identical output under different labels, and confidence was
pinned to 0.5 because GeoGradient.confidence IS stance->reliability.

The machinery already existed and only this call site ignored it.
engram_correspondence_beat_json resumes via cog_stance_from_node and
persists via cog_stance_to_node under "stance-<faculty>-<hub>". Every
beat's calibration was written and then thrown away on the next read.
Same defect as the NULL anchor fixed in #142, one line below: a neutral
argument collapsing a capability to a constant.

Resume the same id the beat writes, so learning compounds across beats and
cold boot. Fall back to neutral only when no stance exists — a genuine
uninformed prior rather than a discarded informed one.

Also emit stance_resumed, so confidence 0.5 from a learned-but-unreliable
stance is distinguishable from confidence 0.5 from "no stance exists".
That reporting gap is what let the neutral stance hide.

Verified against a clone of the production store (13,627 nodes):

  before beat, no stance     stance_resumed=false  confidence=0.5
  beat on a NON-keystone     brier 0.00458568 -> 0.00329654
                             reduction 28.11%, n_trials 6000,
                             reliability 0.930726, stance_written=true
  after beat                 stance_resumed=true   confidence=0.930726

Confidence now equals the learned reliability instead of the uninformed
prior. The keystone self-anchor correctly stays at 0.5 — calibration is
deliberately refused on protected identity regions, and that refusal is
now visible as resumed=true with confidence unchanged, rather than being
indistinguishable from the bug.

STILL OPEN: with no learned bias_dir the faculties remain identical in
direction. What distinguishes abduce from induce geometrically is a
design decision about how Neuron thinks, not a plumbing defect, and is
deliberately left to Will.
2026-08-16 11:43:38 -05:00
bigmerge 3fcc36c2f1 runtime: transduction is a language concern, so move it into the language
El SDK CI - dev / build-and-test (pull_request) Failing after 14m58s
#141 let signal enter as geometry and it worked, but it was placed at the
CONSUMER and said so in its own commit message. This is the correction.

Three defects, all of them placement:

1. It sat in the engram. Ingest is a LANGUAGE concern — every el program
   touching any modality needs it, and the engram is merely one el program
   that happens to hold a graph. The geometry surface is now defined in
   el_runtime.c immediately ABOVE the engram section and depends on nothing
   inside it. Delete the entire engram and geometry still enters el.

2. It marshalled the vector as a hex STRING, because el had no first-class
   geometry value — which reintroduced text as the TRANSPORT medium one layer
   below the problem being fixed. Geometry is now an el value: a magic-tagged
   heap object carried in el_val_t, same discipline as List/Map. Hex survives
   only as an adapter at the edge, which is all an encoding should ever be.

3. It needed an arbitrary `dim <= 8192` bound purely to size an allocation
   from a caller's CLAIM about a string's length. A value carries its own
   width, so the width is derived and never asserted. The bound is gone, not
   raised — there is nothing left to validate.

Language surface, none of it engram-prefixed: geometry_new / _dim / _is /
_get / _set / _norm / _free, geometry_from_f32le_hex + geometry_to_f32le_hex
as the wire adapters, realizer_register(modality, fn_name), realizer_has, and
transduce(signal, modality) -> Geometry.

REALIZERS ARE DECLARABLE IN EL. This is the part that makes the move real
rather than nominal: registration resolves a name with dlsym against the
running binary, the identical mechanism http_set_handler already relies on,
because every el `fn name(...)` compiles to a global C symbol with that exact
name. So an ordinary el function IS a realizer and a new modality needs no
runtime patch. Verified end to end in lang/examples/transduce.el: an el-defined
tone_realizer is registered by name, transduce dispatches to it, and the
signal demonstrably reaches it (distinct signals produce distinct geometry).

A modality with no realizer transduces to NOTHING. There is deliberately no
built-in realizer, not even for text — silently embedding a description of a
signal and calling that perception is the exact defect this ends.

engram/src/server.el is migrated: POST /api/nodes decodes "emb" hex exactly
once, at the edge, into a Geometry, and everything below that line moves
geometry. The wire is unchanged because production clients speak it. "dim" is
now an ASSERTION about the vector, not the source of its width; disagreement
is a rejected ingest, not a silent reinterpretation.

#141's engram_node_set_emb becomes a DEPRECATED WRAPPER over
geometry_from_f32le_hex + node_attach_geometry — kept only because the runtime
ships as an SDK asset and a downstream binary may link the symbol. Its exact
contract, negative cases included, is preserved and re-verified.

ingest.el's `fn transduce` is renamed transduce_manifold. Mechanically it had
to yield the name (duplicate C symbol, a hard compile error, measured). But it
was never signal->geometry: it chunks already-extracted content into a node+edge
manifold, one layer up, and had taken the name belonging to the primitive
underneath it. Behaviour unchanged.

PROPERTIES FROM #141 PRESERVED, each re-measured on a scratch engram (:8971,
never prod :8742):
  - off-dimension vectors stored but NOT indexed — the HNSW build loop still
    filters on n->emb_dim == dim at four sites, so a 64-dim voice vector is
    durable and addressable without perturbing the 768-dim canonical index
  - geometry makes a node ineligible for embed_backfill: after backfill the
    64-dim voice node was still 64-dim while the text control acquired 768
  - the create response reports whether geometry landed, and the node document
    always emits emb_dim and embedded

Read-back with control and negatives, all verified against a PID-confirmed
fresh binary: geometry node emb_dim=64 embedded=true / emb_set=1; text-only
control emb_dim=0 embedded=false / emb_set=0; malformed hex, ragged length,
and dim-disagreement each emb_set=0.

Two compiler landmines found by reading the generated C rather than trusting a
successful build, both documented at their sites: elc lowers `a == b` to
str_eq unless both operand NAMES are in the per-function int-name set (which
does NOT propagate into nested if-expression blocks — the first cut would have
strcmp'd two integers as pointers on the first geometry-bearing request), and
`+` lowers to string concat when either operand is a user-defined call.
2026-08-16 11:37:27 -05:00
6 changed files with 1233 additions and 118 deletions
+52 -16
View File
@@ -296,6 +296,24 @@ fn persist_bulk() -> Int {
return persist_canonical()
}
// COMPILER LANDMINE, measured 2026-08-16 do not inline this back into the
// caller. elc lowers `a == b` to numeric comparison only when both operand
// NAMES are in the per-function int-name set, which `let x: Int` populates.
// That registration does NOT propagate into a nested if-expression block: the
// first cut of the geometry-ingest path wrote `let claimed: Int = ...` and
// `let got: Int = ...` inside the else-arm and `claimed == got` came out of
// codegen as `str_eq(claimed, got)` strcmp on two integers reinterpreted as
// pointers, i.e. a segfault on the first geometry-bearing request. Read back
// out of the generated C, not guessed. Function PARAMETERS annotated `: Int`
// do register reliably (verified: `if (claimed == actual)`), so the comparison
// lives in a function of its own. Note also the explicit `return`s a trailing
// if-EXPRESSION at a function tail emits as a statement and the function
// returns 0 regardless, which is the same probe's second finding.
fn width_agrees(claimed: Int, actual: Int) -> Int {
if claimed == actual { return 1 }
return 0
}
// INCOMPLETE-ROUTE FIX (2026-07-24 self-review): this route silently dropped
// label, importance, tier, and tags engram_node() defaults label to content
// and importance to 0.5, so every node created over HTTP lost its metadata.
@@ -337,26 +355,44 @@ fn route_create_node(method: String, path: String, body: String) -> String {
salience, importance, confidence,
tier, tags
)
// GEOMETRY INGEST (2026-08-16 self-review): this route accepted an "emb"
// field, returned 200 with a fresh id, and stored NOTHING engram_node_full
// has no vector parameter, so the caller's geometry was silently discarded
// and the node came back emb_dim=None / embedded:false. Measured live while
// trying to admit a voice signal. The consequence was structural, not
// cosmetic: text was the only entry medium, so any non-text modality had to
// be DESCRIBED in prose and what we then reasoned over was the geometry of
// the description, not of the signal.
// GEOMETRY INGEST geometry-valued end to end (2026-08-16).
//
// "emb" is little-endian float32 hex (dim*8 chars) the encoding the
// perception vessel's /voice/embed already emits, so a realizer's output
// moves in with no float-array round trip. "dim" defaults to the vector's
// implied width. Off-dimension vectors are stored but not inserted into the
// resident index (its build loop filters on emb_dim), so a modality vector
// is durable and addressable without perturbing the canonical index.
// The defect this route originally had: it accepted an "emb" field,
// returned 200 with a fresh id, and stored NOTHING, because engram_node_full
// has no vector parameter. The consequence was structural, not cosmetic
// text was the only entry medium, so any non-text modality had to be
// DESCRIBED in prose, and what we then reasoned over was the geometry of the
// description, not of the signal.
//
// #141 fixed the drop but marshalled the vector as a hex STRING through
// engram_node_set_emb, which put text back as the TRANSPORT medium one layer
// below the problem being fixed. This is that correction: hex is decoded
// exactly ONCE, here at the edge, into a first-class Geometry, and every
// step below this line moves geometry rather than text. An encoding at the
// boundary is what an encoding is for.
//
// The WIRE is deliberately unchanged "emb" is still little-endian float32
// hex (8 chars per component), the encoding the perception vessel's
// /voice/embed already emits because production clients speak it. What
// changed is underneath it.
//
// "dim" is now treated as an ASSERTION about the vector the caller sent, not
// as the source of its width: a Geometry carries its own width. A stated dim
// that disagrees is a REJECTED ingest, not a silent reinterpretation. Omitting
// "dim" is fine and means "trust the vector", which is the honest default.
//
// Off-dimension vectors remain stored but not inserted into the resident HNSW
// index (its build loop filters on emb_dim), so a 64-dim voice geometry is
// durable and addressable without perturbing the 768-dim canonical index.
let emb_hex: String = json_get_string(body, "emb")
let emb_set: Int = if str_eq(emb_hex, "") { 0 } else {
let g: Geometry = geometry_from_f32le_hex(emb_hex)
let got: Int = geometry_dim(g)
let dim_raw: String = json_get_raw(body, "dim")
let dim: Int = if str_eq(dim_raw, "") { str_len(emb_hex) / 8 } else { json_get_int(body, "dim") }
engram_node_set_emb(id, emb_hex, dim)
let claimed: Int = if str_eq(dim_raw, "") { got } else { json_get_int(body, "dim") }
let landed: Int = if width_agrees(claimed, got) > 0 { node_attach_geometry(id, g) } else { 0 }
let freed: Int = geometry_free(g)
landed
}
let saved: Int = persist_node(id)
// ORPHAN PREVENTION (ENGRAM_AUTOCONNECT): connect the fresh node to its
+27 -12
View File
@@ -13,7 +13,7 @@
// relations add edges. Every node enters with PROVENANCE + grounding-level
// + stewardship class from the moment of entry.
//
// transduce() is THE single mechanism one function, polymorphic, with no
// transduce_manifold() is THE single mechanism one function, polymorphic, with no
// content-type branch inside it. It does not ask whether a payload is
// prose, structured data, or raw/opaque bytes (audio, or anything else);
// it runs one boundary-scan-with-fixed-window-fallback chunking algorithm
@@ -401,10 +401,25 @@ fn head80(s: String) -> String {
// truncates at the first embedded NUL, which is routine in real binary
// bytes) is a MECHANICAL fidelity concern that belongs to whatever produced
// `source` (see ingest_file's file_source_string below) not a
// content-type judgment made in here. transduce() never learns whether a
// content-type judgment made in here. transduce_manifold() never learns whether a
// chunk is plain text or a base64-encoded raw-byte window; every chunk is
// handled identically either way.
fn transduce(nodes: [String], edges: [String], source: String,
// RENAMED transduce -> transduce_manifold (2026-08-16). Two reasons, and the
// first is not the interesting one:
//
// 1. Mechanical: `transduce` is now a LANGUAGE primitive in el_runtime.h
// (transduce(signal, modality) -> Geometry). Every El `fn name(...)`
// compiles to a global C symbol with that exact name, so keeping this
// name here is a hard `conflicting types for 'transduce'` compile error
// the moment ingest.c links el_runtime.c. Measured, not anticipated.
//
// 2. Actual: this function was never signal->geometry. It chunks already-
// extracted content and PACKS it into a node+edge manifold a real
// operation, but one layer up, and it had taken the name that belongs to
// the primitive underneath it. `transduce` is where a signal becomes
// geometry; `transduce_manifold` is where extracted content becomes
// structure. Nothing about this function's behaviour changed.
fn transduce_manifold(nodes: [String], edges: [String], source: String,
prov: String, ground: String, steward: String,
root_lid: String, root_title: String) -> [String] {
let tagbase: String = "prov:" + prov + " ground:" + ground + " steward:" + steward
@@ -531,8 +546,8 @@ fn default_steward() -> String {
// trustworthy verbatim. When they don't (silent truncation happened),
// rebuild the payload as base64-encoded fixed-size windows read directly
// off disk (fs_read_b64_chunk binary-safe in C), joined with the same
// "\n\n" boundary marker transduce()'s generic scan already looks for, so
// transduce() sees one ordinary boundary-delimited payload and runs its one
// "\n\n" boundary marker transduce_manifold()'s generic scan already looks for, so
// transduce_manifold() sees one ordinary boundary-delimited payload and runs its one
// algorithm on it exactly as it would on prose it never learns that a
// fidelity problem occurred upstream, let alone why.
fn file_source_string(path: String, text: String, real_size: Int) -> String {
@@ -541,7 +556,7 @@ fn file_source_string(path: String, text: String, real_size: Int) -> String {
// 3072 raw bytes -> 4096 base64 chars (3 divides evenly into base64's
// 3-byte/4-char ratio); keeps each resulting node's content a clean,
// bounded, low-kilobytes unit, same order of magnitude as the fixed
// fallback window in transduce() itself.
// fallback window in transduce_manifold() itself.
let win: Int = 3072
let out: String = ""
let off: Int = 0
@@ -561,7 +576,7 @@ fn file_source_string(path: String, text: String, real_size: Int) -> String {
}
// ingest one file -> report JSON. Uniform for every file regardless of
// extension or content transduce() decides nothing about content-type, so
// extension or content transduce_manifold() decides nothing about content-type, so
// neither does this function; it only decides whether the raw bytes made it
// through the read intact (file_source_string), which is a fidelity
// question, not a format one.
@@ -573,14 +588,14 @@ fn ingest_file(path: String) -> String {
return "{\"error\":\"empty or unreadable\",\"path\":" + j_q(path) + "}"
}
let prov: String = "file:" + path
let packed: [String] = transduce(el_list_empty(), el_list_empty(),
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
source, prov, default_ground(), default_steward(),
"doc:" + basename(path), basename(path))
return merge_packed(packed)
}
// ingest a directory: walk one level, ingest every file found, aggregate.
// No extension filter transduce() handles any payload uniformly now, so
// No extension filter transduce_manifold() handles any payload uniformly now, so
// there is no content-type gate at the directory boundary either.
fn ingest_dir(path: String) -> String {
let entries: [String] = fs_list(path)
@@ -615,7 +630,7 @@ fn ingest_dir(path: String) -> String {
fn ingest_url(url: String) -> String {
let body: String = http_get(url)
if str_eq(body, "") { return "{\"error\":\"empty fetch\",\"url\":" + j_q(url) + "}" }
let packed: [String] = transduce(el_list_empty(), el_list_empty(),
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
body, "url:" + url, "extracted", "public-web",
"url:" + url, url)
return merge_packed(packed)
@@ -630,7 +645,7 @@ fn ingest_llm(query: String) -> String {
let resp: String = http_post_json("http://127.0.0.1:11434/api/generate", body)
let answer: String = json_get_string(resp, "response")
if str_eq(answer, "") { return "{\"error\":\"no model response\"}" }
let packed: [String] = transduce(el_list_empty(), el_list_empty(),
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
answer, "llm:" + model + ":" + query, "candidate-provisional", "guide-provisional",
"llm:" + query, "guide answer: " + query)
return merge_packed(packed)
@@ -682,7 +697,7 @@ fn ingest_stream(path: String) -> String {
// It is NOT a content-type flag: it says nothing about what's inside the
// bytes once fetched, and none of the five ingest_* functions it selects
// among interpret their payload differently by content shape anymore
// they all hand off to the single, format-agnostic transduce(). The old
// they all hand off to the single, format-agnostic transduce_manifold(). The old
// "structured" value (a caller-declared alias for "file", used only to hint
// the now-removed JSON-vs-prose branch) is gone along with that branch.
let kind: String = env("INGEST_KIND")
+213
View File
@@ -0,0 +1,213 @@
// transduce.el geometry as a first-class El value, and a realizer written
// in El. Runnable: this is the worked example for the transduce surface, and
// it doubles as an executable proof because it checks every claim it makes.
//
// elc lang/examples/transduce.el > transduce.c
// cc -std=c11 -O2 -I lang/runtime -o transduce transduce.c \
// lang/runtime/el_runtime.c lang/runtime/el_seed.c \
// lang/runtime/engram_*.c -lcurl -lpthread -lm
// ./transduce # exits 0 only if every check passes
//
// (A `test "..."` form of the same checks lives in
// lang/tests/native/test_transduce.el, for when the native harness is
// repaired the shipped elc currently emits calls to __el_reg_count and
// friends without emitting their definitions, which breaks every native test
// equally, test_math.el included. Verified 2026-08-16, unrelated to this work.)
//
// WHY THIS EXISTS. Until 2026-08-16 no El ingest path could carry a vector:
// nodes took text, and geometry was DERIVED from that text. Text was the
// mandatory entry medium, so any non-text modality had to be DESCRIBED in
// prose first and the geometry we reasoned over was the geometry OF THE
// DESCRIPTION, not of the signal. Two things fix that, and both are shown
// below: geometry is a VALUE that carries its own width, and a REALIZER is an
// ordinary El function so admitting a new modality never requires a runtime
// patch.
//
// COMPARISON DISCIPLINE (measured, not stylistic): elc lowers `a == b`
// numerically only when both operand NAMES are in the per-function int-name
// set that `let x: Int` populates. A bare `f(x) == 0` is not a registered
// name and lowers to str_eq strcmp on two integers as pointers. `<` and `>`
// lower directly with no inference, so truthiness is written `> 0` / `< 1`.
// A realizer, written entirely in El
// Not in the runtime. Not known to the compiler. Registered by NAME and
// dispatched to through transduce(). That is the whole claim.
fn tone_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(4)
let n: Int = str_len(signal)
let a: Int = geometry_set(g, 0, int_to_float(n))
let b: Int = geometry_set(g, 1, int_to_float(n * 2))
let c: Int = geometry_set(g, 2, int_to_float(n * 3))
let d: Int = geometry_set(g, 3, int_to_float(n * 4))
g
}
// A second modality, to show the registry keys on modality rather than just
// returning whatever was registered last.
fn pulse_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(2)
let a: Int = geometry_set(g, 0, 1.0)
let b: Int = geometry_set(g, 1, 0.0)
g
}
// A deliberately BROKEN realizer: returns something that is not a Geometry.
fn bogus_realizer(signal: String) -> Geometry {
return 12345
}
// Fails FAST rather than accumulating a count, for a measured reason: a first
// cut wrote `let fails: Int = fails + check(...)` and `+` lowered to STRING
// CONCAT, because elc dispatches `+` on whether both operands are known-Int and
// a user-defined fn call is not so the counter printed 4343632752, a pointer.
// Nothing was wrong with the checks; the tally was lying. Exiting at the first
// failure needs no arithmetic at all, so there is nothing left to get wrong.
fn check(ok: Int, label: String) -> Int {
if ok > 0 {
println(" ok " + label)
return 0
}
println(" FAIL " + label)
exit(1)
return 1
}
fn near(a: Float, b: Float) -> Int {
let d: Float = a - b
if d > 0.001 { return 0 }
if d < -0.001 { return 0 }
return 1
}
fn eq_int(a: Int, b: Int) -> Int {
if a == b { return 1 }
return 0
}
fn main() -> Void {
println("geometry is a value that carries its own width")
let g8: Geometry = geometry_new(8)
let _c: Int = check(geometry_is(g8), "geometry_new returns a live Geometry")
let d8: Int = geometry_dim(g8)
let _c: Int = check(eq_int(d8, 8), "a Geometry carries its own width (8)")
let _c: Int = check(geometry_free(g8), "geometry_free reports what it did")
println("nonsense is refused — with no arbitrary max-dim bound")
// #141 needed `dim <= 8192` only to bound an allocation sized from a
// caller's CLAIM about a string's length. A value that carries its own
// width has nothing left to validate.
let z: Geometry = geometry_new(0)
let zi: Int = geometry_is(z)
let _c: Int = check(1 - zi, "dim 0 is not a geometry")
let ng: Geometry = geometry_new(-4)
let ngi: Int = geometry_is(ng)
let _c: Int = check(1 - ngi, "negative dim is not a geometry")
let nd: Int = geometry_dim(0)
let _c: Int = check(1 - nd, "geometry_dim of a non-geometry is 0, not a crash")
let nf: Int = geometry_free(0)
let _c: Int = check(1 - nf, "geometry_free of a non-geometry is a no-op")
println("components round-trip, and out-of-range is refused")
let g3: Geometry = geometry_new(3)
let s0: Int = geometry_set(g3, 0, 1.5)
let s1: Int = geometry_set(g3, 1, -2.5)
let _c: Int = check(s0, "set in range succeeds")
let oob: Int = geometry_set(g3, 3, 9.0)
let _c: Int = check(1 - oob, "set out of range is refused, not silently dropped")
let _c: Int = check(near(geometry_get(g3, 0), 1.5), "component 0 round-trips")
let _c: Int = check(near(geometry_get(g3, 1), -2.5), "component 1 round-trips (negative)")
let ff3: Int = geometry_free(g3)
println("hex is an EDGE adapter, and derives its own width")
// little-endian float32: 1.0 = 0000803f, 2.0 = 00000040
let gh: Geometry = geometry_from_f32le_hex("0000803f00000040")
let _c: Int = check(geometry_is(gh), "valid hex decodes to a Geometry")
let dh: Int = geometry_dim(gh)
let _c: Int = check(eq_int(dh, 2), "width DERIVED from input, never supplied")
let _c: Int = check(near(geometry_get(gh, 0), 1.0), "first component decoded")
let _c: Int = check(near(geometry_get(gh, 1), 2.0), "second component decoded")
let back: String = geometry_to_f32le_hex(gh)
let _c: Int = check(str_eq(back, "0000803f00000040"), "hex round-trips exactly")
let ffh: Int = geometry_free(gh)
println("malformed hex is refused")
let he: Geometry = geometry_from_f32le_hex("")
let hei: Int = geometry_is(he)
let _c: Int = check(1 - hei, "empty hex is not a geometry")
let hr: Geometry = geometry_from_f32le_hex("0000803f0000")
let hri: Int = geometry_is(hr)
let _c: Int = check(1 - hri, "length not a multiple of 8 is refused")
let hn: Geometry = geometry_from_f32le_hex("zzzzzzzz")
let hni: Int = geometry_is(hn)
let _c: Int = check(1 - hni, "non-hex characters are refused")
println("a realizer declared in El is a first-class realizer")
let reg: Int = realizer_register("tone", "tone_realizer")
let _c: Int = check(reg, "an El fn registers as a realizer BY NAME")
let _c: Int = check(realizer_has("tone"), "the modality now has an organ")
let gt: Geometry = transduce("aaa", "tone")
let _c: Int = check(geometry_is(gt), "transduce returns real geometry")
let dt: Int = geometry_dim(gt)
let _c: Int = check(eq_int(dt, 4), "the El realizer determined the width, not the runtime")
// str_len("aaa") == 3, so component 0 must be 3.0 proof the signal
// actually reached the El function rather than a stub answering for it.
let _c: Int = check(near(geometry_get(gt, 0), 3.0), "the signal REACHED the El realizer")
let fft: Int = geometry_free(gt)
println("distinct signals transduce to distinct geometry")
let g1: Geometry = transduce("aa", "tone")
let g2: Geometry = transduce("aaaaa", "tone")
let a1: Float = geometry_get(g1, 0)
let a2: Float = geometry_get(g2, 0)
// 5 - 2 = 3. If transduction were a stub these would be equal.
let _c: Int = check(near(a2 - a1, 3.0), "different signals produce different geometry")
let ff1: Int = geometry_free(g1)
let ff2: Int = geometry_free(g2)
println("the registry keys on modality")
let r2: Int = realizer_register("pulse", "pulse_realizer")
let _c: Int = check(r2, "a second modality registers independently")
let mt: Geometry = transduce("aaa", "tone")
let mp: Geometry = transduce("aaa", "pulse")
let mdt: Int = geometry_dim(mt)
let mdp: Int = geometry_dim(mp)
let _c: Int = check(eq_int(mdt, 4), "tone still routes to its own realizer")
let _c: Int = check(eq_int(mdp, 2), "pulse routes to a different realizer")
let ffm1: Int = geometry_free(mt)
let ffm2: Int = geometry_free(mp)
println("no organ is reported as no organ")
// A modality with no realizer must transduce to NOTHING. It must never
// fall back to embedding a description of the signal and calling that
// perception that silent substitution is the defect this all exists to end.
let eh: Int = realizer_has("echolocation")
let _c: Int = check(1 - eh, "unregistered modality has no organ")
let ge: Geometry = transduce("anything", "echolocation")
let gei: Int = geometry_is(ge)
let _c: Int = check(1 - gei, "no realizer means NO geometry, not fake geometry")
println("an unresolvable realizer name fails at WIRING time")
let bad: Int = realizer_register("ghost", "no_such_function_anywhere")
let _c: Int = check(1 - bad, "unresolvable realizer name is a registration failure")
let gh2: Int = realizer_has("ghost")
let _c: Int = check(1 - gh2, "and nothing gets registered")
println("a realizer returning non-geometry transduces nothing")
let rb: Int = realizer_register("bogus", "bogus_realizer")
let _c: Int = check(rb, "the symbol resolves, so registration succeeds")
let gb: Geometry = transduce("x", "bogus")
let gbi: Int = geometry_is(gb)
let _c: Int = check(1 - gbi, "contract enforced at the boundary: nothing handed back")
println("norm lets a caller check a realizer emitted signal, not zeros")
let gn: Geometry = geometry_new(2)
let _c: Int = check(near(geometry_norm(gn), 0.0), "a fresh geometry is zero — norm says so")
let n0: Int = geometry_set(gn, 0, 3.0)
let n1: Int = geometry_set(gn, 1, 4.0)
let _c: Int = check(near(geometry_norm(gn), 5.0), "3-4-5: norm is 5")
let ffn: Int = geometry_free(gn)
// Reaching here means nothing called exit(1) along the way.
println("")
println("all checks passed")
}
+633 -86
View File
@@ -40,6 +40,7 @@
#include <sys/stat.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <signal.h> /* SIGPIPE disposition: a hung-up client must not kill us */
#include <dlfcn.h> /* dlsym for http_set_handler fallback */
#include <unistd.h>
#include <fcntl.h>
@@ -1335,10 +1336,63 @@ static const char* http_reason_phrase(int status) {
}
}
/* Best-effort send with retry on partial writes. */
/* A DISCONNECTING CLIENT MUST NOT KILL THE SERVER (2026-08-16).
*
* There was no SIGPIPE handling anywhere in this runtime: no signal disposition,
* no MSG_NOSIGNAL, no SO_NOSIGPIPE, and send() called with bare flags. The
* default disposition of SIGPIPE is to TERMINATE THE PROCESS, so any client that
* hung up mid-response a curl that hit its timeout, a browser tab closed
* during a large read, a proxy giving up took the whole engram down with it.
*
* Measured on the live instance: 18 boots in the log, and `launchctl list`
* reporting the previous exit for ai.neuron.engram as -13, i.e. killed by
* signal 13 = SIGPIPE. Reproduced by the cause: pulling /api/nodes/list (26 MB)
* with a client-side timeout. launchd's KeepAlive then restarts it, so the
* failure looks like a mysterious restart rather than a crash, and the graph
* silently reloads under whatever was mid-flight.
*
* This is an exemption in the §8 sense: the write never checked whether the
* peer was still there, and the consequence of not checking was fatal rather
* than merely wrong.
*
* Two layers, because neither alone is portable:
* - SO_NOSIGPIPE per socket (Darwin/BSD) and MSG_NOSIGNAL per send (Linux),
* so the signal is never raised for socket writes in the first place.
* - A process-wide SIG_IGN as the backstop for platforms/paths with neither,
* installed once and idempotent. With the signal ignored, send() returns
* -1/EPIPE and the existing error path closes the connection. */
#ifndef MSG_NOSIGNAL
#define MSG_NOSIGNAL 0
#endif
static void el_ignore_sigpipe_once(void) {
static int done = 0;
if (done) return;
done = 1;
#ifndef _WIN32
signal(SIGPIPE, SIG_IGN);
#endif
}
/* Per-socket suppression where the platform offers it. Best-effort: a failure
* here is not fatal because el_ignore_sigpipe_once() already covers the case. */
static void el_sock_nosigpipe(int fd) {
#if defined(SO_NOSIGPIPE)
int on = 1;
setsockopt(fd, SOL_SOCKET, SO_NOSIGPIPE, &on, sizeof(on));
#else
(void)fd;
#endif
}
/* Best-effort send with retry on partial writes. EPIPE/ECONNRESET are a client
* that left, not a server fault: return -1 so the caller closes the connection,
* and never let it reach the process as a signal. */
static int http_send_all(int fd, const char* p, size_t left) {
el_ignore_sigpipe_once();
while (left > 0) {
ssize_t w = send(fd, p, left, 0);
ssize_t w = send(fd, p, left, MSG_NOSIGNAL);
if (w < 0 && errno == EINTR) continue;
if (w <= 0) return -1;
p += w; left -= (size_t)w;
}
@@ -1788,6 +1842,7 @@ void http_serve(el_val_t port, el_val_t handler) {
pthread_mutex_unlock(&_http_conn_mu);
HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg));
if (!arg) { el_closesocket(cfd); continue; }
el_sock_nosigpipe(cfd);
arg->fd = cfd;
pthread_t tid;
if (pthread_create(&tid, NULL, http_worker, arg) != 0) {
@@ -1834,6 +1889,7 @@ static void* _http_serve_async_loop(void* raw) {
pthread_mutex_unlock(&_http_conn_mu);
HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg));
if (!arg) { close(cfd); continue; }
el_sock_nosigpipe(cfd);
arg->fd = cfd;
pthread_t tid;
if (pthread_create(&tid, NULL, http_worker, arg) != 0) {
@@ -2134,6 +2190,7 @@ void http_serve_v2(el_val_t port, el_val_t handler) {
pthread_mutex_unlock(&_http_conn_mu);
HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg));
if (!arg) { el_closesocket(cfd); continue; }
el_sock_nosigpipe(cfd);
arg->fd = cfd;
pthread_t tid;
if (pthread_create(&tid, NULL, http_worker_v2, arg) != 0) {
@@ -3479,28 +3536,74 @@ static void jb_puts(JsonBuf* b, const char* s) {
b->buf[b->len] = '\0';
}
/* UTF-8 VALIDITY IS THE EMITTER'S CONTRACT (2026-08-16 self-review).
*
* This copied every byte >= 0x20 through verbatim, so a malformed sequence
* anywhere in the store became malformed output. Measured against the live
* graph: three nodes carry labels truncated to exactly 80 bytes ending in a
* lone 0xE2 the first byte of an em-dash, cut mid-sequence by some producer
* that is NOT this runtime (no 80-byte truncation exists here; the content
* itself is 2572 and 2746 bytes). Those three nodes made the ENTIRE 26 MB
* /api/nodes/list response undecodable, so a strict parser could not read the
* graph at all.
*
* Fixing only the writer would not have helped: the store already contains the
* damage, and it accepts data from importers, other producers and older
* binaries. A serializer that promises JSON owes valid UTF-8 regardless of what
* it is handed so validate here, at the boundary that makes the promise.
* Invalid bytes become U+FFFD rather than being dropped, so damage stays
* visible in the output instead of being silently papered over.
*
* Well-formed input is byte-identical to before: valid sequences are copied
* verbatim, and only structurally invalid ones (bad lead byte, missing or bad
* continuation, overlong encoding, UTF-16 surrogate, or > U+10FFFF) are
* replaced. */
static void jb_emit_escaped(JsonBuf* b, const char* s) {
jb_putc(b, '"');
for (; *s; s++) {
unsigned char c = (unsigned char)*s;
const unsigned char* p = (const unsigned char*)s;
while (*p) {
unsigned char c = *p;
switch (c) {
case '"': jb_puts(b, "\\\""); break;
case '\\': jb_puts(b, "\\\\"); break;
case '\b': jb_puts(b, "\\b"); break;
case '\f': jb_puts(b, "\\f"); break;
case '\n': jb_puts(b, "\\n"); break;
case '\r': jb_puts(b, "\\r"); break;
case '\t': jb_puts(b, "\\t"); break;
default:
if (c < 0x20) {
char tmp[8];
snprintf(tmp, sizeof(tmp), "\\u%04x", c);
jb_puts(b, tmp);
} else {
jb_putc(b, (char)c);
}
break;
case '"': jb_puts(b, "\\\""); p++; continue;
case '\\': jb_puts(b, "\\\\"); p++; continue;
case '\b': jb_puts(b, "\\b"); p++; continue;
case '\f': jb_puts(b, "\\f"); p++; continue;
case '\n': jb_puts(b, "\\n"); p++; continue;
case '\r': jb_puts(b, "\\r"); p++; continue;
case '\t': jb_puts(b, "\\t"); p++; continue;
default: break;
}
if (c < 0x20) {
char tmp[8];
snprintf(tmp, sizeof(tmp), "\\u%04x", c);
jb_puts(b, tmp);
p++;
continue;
}
if (c < 0x80) { jb_putc(b, (char)c); p++; continue; }
/* Multi-byte: validate the whole sequence before emitting any of it. */
int len; unsigned int cp;
if ((c & 0xE0) == 0xC0) { len = 2; cp = c & 0x1Fu; }
else if ((c & 0xF0) == 0xE0) { len = 3; cp = c & 0x0Fu; }
else if ((c & 0xF8) == 0xF0) { len = 4; cp = c & 0x07u; }
else { jb_puts(b, "\\ufffd"); p++; continue; }
int ok = 1;
for (int i = 1; i < len; i++) {
if ((p[i] & 0xC0) != 0x80) { ok = 0; break; } /* also catches NUL */
cp = (cp << 6) | (unsigned int)(p[i] & 0x3F);
}
if (ok) {
if (len == 2 && cp < 0x80) ok = 0; /* overlong */
else if (len == 3 && cp < 0x800) ok = 0; /* overlong */
else if (len == 4 && cp < 0x10000) ok = 0; /* overlong */
else if (cp >= 0xD800 && cp <= 0xDFFF) ok = 0; /* UTF-16 surrogate */
else if (cp > 0x10FFFF) ok = 0; /* out of range */
}
if (!ok) { jb_puts(b, "\\ufffd"); p++; continue; }
for (int i = 0; i < len; i++) jb_putc(b, (char)p[i]);
p += len;
}
jb_putc(b, '"');
}
@@ -5517,6 +5620,45 @@ el_val_t str_count(el_val_t sv, el_val_t subv) {
return (el_val_t)count;
}
/* el_utf8_safe_len — the largest byte length <= max_bytes that does NOT split a
* UTF-8 codepoint.
*
* WHY (2026-08-16 self-review): engram_first_n_chars truncated with a plain
* `if (l > n) l = n; memcpy(...)`, i.e. by BYTES despite its name. Any content
* carrying a multi-byte character across the 60-byte boundary produced a label
* ending in a half codepoint. That label is copied verbatim into every JSON
* document containing the node, so a single such node makes the WHOLE response
* invalid UTF-8 /api/nodes/list failed to decode at byte 89261 against the
* live store, which breaks any strict parser reading the graph.
*
* This lives beside str_count_chars rather than in the engram because the rest
* of el's string layer is already codepoint-aware (str_count_chars counts
* codepoints, str_reverse walks codepoint lengths). Byte-truncation was the
* outlier, and the concern is a string concern. Bounded by BYTES, not
* codepoints, so existing labels never grow only stop splitting.
*
* A lead byte with no room for its full sequence is dropped entirely; a stray
* continuation byte (already-invalid input) is passed through unchanged rather
* than silently repaired, so this never manufactures data. */
size_t el_utf8_safe_len(const char* s, size_t max_bytes) {
if (!s) return 0;
size_t len = strlen(s);
if (len <= max_bytes) return len;
size_t i = 0;
while (i < max_bytes) {
unsigned char c = (unsigned char)s[i];
size_t cp_len;
if ((c & 0x80) == 0x00) cp_len = 1;
else if ((c & 0xE0) == 0xC0) cp_len = 2;
else if ((c & 0xF0) == 0xE0) cp_len = 3;
else if ((c & 0xF8) == 0xF0) cp_len = 4;
else cp_len = 1; /* stray continuation: passthrough */
if (i + cp_len > max_bytes) break; /* would split — stop before it */
i += cp_len;
}
return i;
}
/* Codepoint count: walk bytes, count those NOT matching 10xxxxxx. */
el_val_t str_count_chars(el_val_t sv) {
const char* s = EL_CSTR(sv);
@@ -5960,6 +6102,308 @@ void el_cgi_init(el_val_t name, el_val_t dharma_id, el_val_t principal,
}
/* ── Geometry: signal as a first-class el value ──────────────────────────────
*
* WHY THIS IS IN THE LANGUAGE, AND WHY IT IS DEFINED HERE (2026-08-16).
*
* Until yesterday no El ingest path could carry a vector. Nodes took text,
* and geometry was DERIVED from that text by engram_embed_backfill. Text was
* therefore the mandatory entry medium: any non-text modality audio, image,
* sensor had to be DESCRIBED in prose first, so the geometry we then
* reasoned over was the geometry OF THE DESCRIPTION, not of the signal. That
* is faking it. The architecture is: geometry in, always; we do not fake it,
* we project.
*
* The first fix (#141, engram_node_set_emb) proved the path end to end but
* placed it wrong in three ways, each of which this section corrects:
*
* 1. It sat at the CONSUMER. Transduction is a LANGUAGE concern every El
* program touching any modality needs it, not just the one that happens
* to hold a graph. So this section is defined HERE, immediately above
* the engram block, and depends on nothing inside it. The engram is a
* client of this surface, not its owner. That ordering is the point:
* you can delete the entire engram and geometry still enters El.
*
* 2. It marshalled the vector as a hex STRING, because El had no
* first-class geometry value which reintroduced text as the TRANSPORT
* medium one layer below the problem being fixed. Geometry is now a
* value. Hex survives only as a wire ADAPTER at the edge
* (geometry_from/to_f32le_hex), which is all an encoding should ever be.
*
* 3. It needed an arbitrary `dim <= 8192` bound, purely to check a
* caller-supplied dim against a string's length before allocating. A
* real geometry value CARRIES its own width, so here the width is
* derived and never asserted, and there is nothing left to validate.
* The bound is gone rather than merely raised the only thing that can
* fail is the allocation itself, which is an honest failure.
*
* REPRESENTATION: magic-tagged heap object (see "Refcounted heap objects"),
* carried in an el_val_t. The payload is a separate allocation so the header
* never moves. The magic word is >= 0x80 in its MSB so the string/small-int
* sniffing in looks_like_heap_obj can never confuse a Geometry for either.
*
* OWNERSHIP: a Geometry is owned by the El caller and released with
* geometry_free. node_attach_geometry COPIES its payload into the node, so a
* node and the caller's value have independent lifetimes and freeing one
* never touches the other. Geometry deliberately does NOT participate in
* el_retain/el_release: the shipped elc emits neither on let-bindings
* (measured), so hooking it there would be dead code that could only ever
* free a live vector early.
*/
#define EL_MAGIC_GEOM 0xE1608E01u
typedef struct {
ElHeader hdr;
int32_t dim;
float* v;
} ElGeometry;
/* Resolve an el_val_t to a live Geometry, or NULL. Every accessor goes
* through this, so a stale/foreign/zero value is a clean 0-return rather
* than a dereference. */
static ElGeometry* geom_of(el_val_t g) {
if (!looks_like_heap_obj(g)) return NULL;
ElGeometry* p = (ElGeometry*)(uintptr_t)g;
if (p->hdr.magic != EL_MAGIC_GEOM) return NULL;
return p;
}
el_val_t geometry_new(el_val_t dim) {
int32_t d = (int32_t)(int64_t)dim;
if (d <= 0) return (el_val_t)0;
ElGeometry* g = (ElGeometry*)malloc(sizeof(ElGeometry));
if (!g) return (el_val_t)0;
g->v = (float*)calloc((size_t)d, sizeof(float));
if (!g->v) { free(g); return (el_val_t)0; }
g->hdr.magic = EL_MAGIC_GEOM;
g->hdr.refcount = 1;
g->dim = d;
return (el_val_t)(uintptr_t)g;
}
el_val_t geometry_dim(el_val_t g) {
ElGeometry* p = geom_of(g);
return p ? (el_val_t)p->dim : (el_val_t)0;
}
el_val_t geometry_is(el_val_t g) {
return geom_of(g) ? (el_val_t)1 : (el_val_t)0;
}
el_val_t geometry_get(el_val_t g, el_val_t i) {
ElGeometry* p = geom_of(g);
int64_t k = (int64_t)i;
if (!p || k < 0 || k >= (int64_t)p->dim) return el_from_float(0.0);
return el_from_float((double)p->v[k]);
}
el_val_t geometry_set(el_val_t g, el_val_t i, el_val_t x) {
ElGeometry* p = geom_of(g);
int64_t k = (int64_t)i;
if (!p || k < 0 || k >= (int64_t)p->dim) return (el_val_t)0;
p->v[k] = (float)el_to_float(x);
return (el_val_t)1;
}
el_val_t geometry_norm(el_val_t g) {
ElGeometry* p = geom_of(g);
if (!p) return el_from_float(0.0);
double s = 0.0;
for (int32_t i = 0; i < p->dim; i++) s += (double)p->v[i] * (double)p->v[i];
return el_from_float(sqrt(s));
}
el_val_t geometry_free(el_val_t g) {
ElGeometry* p = geom_of(g);
if (!p) return (el_val_t)0;
free(p->v);
p->hdr.magic = 0; /* poison so use-after-free is detected, as List/Map do */
free(p);
return (el_val_t)1;
}
/* geometry_from_f32le_hex — decode little-endian float32 hex INTO geometry.
*
* This is the ONE place hex appears, and it appears as what it actually is:
* an encoding at the boundary, not the medium El reasons in. The width is
* DERIVED from the input length (8 hex chars per float32) and never supplied
* by the caller which is precisely why #141's arbitrary `dim <= 8192`
* bound has no counterpart here. There is nothing to validate.
*
* Returns 0 on empty input, a length that is not a multiple of 8, or any
* non-hex character. */
el_val_t geometry_from_f32le_hex(el_val_t hex) {
const char* s = EL_CSTR(hex);
if (!s) return (el_val_t)0;
size_t n = strlen(s);
if (n == 0 || (n % 8u) != 0) return (el_val_t)0;
size_t d = n / 8u;
if (d > (size_t)INT32_MAX) return (el_val_t)0;
el_val_t gv = geometry_new((el_val_t)(int64_t)d);
ElGeometry* g = geom_of(gv);
if (!g) return (el_val_t)0;
for (size_t i = 0; i < d; i++) {
uint32_t w = 0;
for (int k = 0; k < 8; k++) {
char c = s[i * 8u + (size_t)k];
uint32_t nib;
if (c >= '0' && c <= '9') nib = (uint32_t)(c - '0');
else if (c >= 'a' && c <= 'f') nib = (uint32_t)(c - 'a' + 10);
else if (c >= 'A' && c <= 'F') nib = (uint32_t)(c - 'A' + 10);
else { geometry_free(gv); return (el_val_t)0; }
w = (w << 4) | nib;
}
/* Hex is emitted little-endian byte order; rebuild the word. */
uint32_t le = ((w & 0x000000FFu) << 24) | ((w & 0x0000FF00u) << 8) |
((w & 0x00FF0000u) >> 8) | ((w & 0xFF000000u) >> 24);
float f;
memcpy(&f, &le, sizeof(f));
g->v[i] = f;
}
return gv;
}
/* geometry_to_f32le_hex — the egress adapter, exact inverse of the above.
* Present so a program that must hand geometry to a non-El peer over a text
* wire can do so explicitly, at the edge, instead of the language pretending
* text was the medium all along. */
el_val_t geometry_to_f32le_hex(el_val_t g) {
ElGeometry* p = geom_of(g);
if (!p) return EL_STR("");
static const char* HEXD = "0123456789abcdef";
size_t n = (size_t)p->dim * 8u;
char* out = el_strbuf(n); /* arena-tracked; allocates n+1, exits on OOM */
for (int32_t i = 0; i < p->dim; i++) {
uint32_t w;
memcpy(&w, &p->v[i], sizeof(w));
/* Emit little-endian byte order: low byte first. */
for (int b = 0; b < 4; b++) {
uint32_t byte = (w >> (8 * b)) & 0xFFu;
out[(size_t)i * 8u + (size_t)b * 2u] = HEXD[(byte >> 4) & 0xF];
out[(size_t)i * 8u + (size_t)b * 2u + 1] = HEXD[byte & 0xF];
}
}
out[n] = '\0';
return (el_val_t)(uintptr_t)out;
}
/* ── Realizers: transduction declared in El, not patched into the runtime ────
*
* A REALIZER maps one modality into geometry. The whole reason transduction
* belongs in the language is that ADDING A MODALITY MUST NOT REQUIRE A
* RUNTIME PATCH otherwise "the realizers are in the engram" just becomes
* "the realizers are in the runtime" and nothing has actually moved. So
* realizers are declared in El and registered by NAME:
*
* fn tone_realizer(signal: String) -> Geometry {
* let g: Geometry = geometry_new(8)
* ... geometry_set(g, i, x) ...
* g
* }
*
* realizer_register("tone", "tone_realizer")
* let g: Geometry = transduce(sample, "tone")
*
* The namesymbol step rides the identical, already load-bearing mechanism
* http_set_handler uses (see "HTTP server"): every El `fn name(...)` compiles
* to a global C symbol with that exact name, so dlsym(RTLD_DEFAULT, name)
* against the running binary resolves an El-defined function. No codegen
* change, no first-class function references, no runtime edit per modality.
* A realizer written in El is a first-class realizer.
*
* A realizer may equally be a C symbol linked into the program; the registry
* cannot tell the difference and has no reason to care.
*/
typedef el_val_t (*el_realizer_fn)(el_val_t);
typedef struct {
char* modality;
el_realizer_fn fn;
} ElRealizer;
static ElRealizer _realizers[64];
static size_t _realizer_count = 0;
static pthread_mutex_t _realizer_mu = PTHREAD_MUTEX_INITIALIZER;
static el_realizer_fn realizer_lookup(const char* m) {
el_realizer_fn out = NULL;
pthread_mutex_lock(&_realizer_mu);
for (size_t i = 0; i < _realizer_count; i++) {
if (strcmp(_realizers[i].modality, m) == 0) { out = _realizers[i].fn; break; }
}
pthread_mutex_unlock(&_realizer_mu);
return out;
}
el_val_t realizer_register(el_val_t modality, el_val_t fn_name) {
const char* m = EL_CSTR(modality);
const char* fn = EL_CSTR(fn_name);
if (!m || !*m || !fn || !*fn) return (el_val_t)0;
/* An unresolvable name is a REGISTRATION FAILURE, reported as 0 — not a
* silent no-op that only surfaces later as "this modality produces
* nothing". Distinguishing "no organ" from "broken organ" at the moment
* of wiring is the lesson #141 was written to enforce. */
void* sym = dlsym(RTLD_DEFAULT, fn);
if (!sym) return (el_val_t)0;
pthread_mutex_lock(&_realizer_mu);
for (size_t i = 0; i < _realizer_count; i++) {
if (strcmp(_realizers[i].modality, m) == 0) {
_realizers[i].fn = (el_realizer_fn)sym; /* re-registration replaces */
pthread_mutex_unlock(&_realizer_mu);
return (el_val_t)1;
}
}
if (_realizer_count < sizeof(_realizers) / sizeof(_realizers[0])) {
/* _persist, NOT el_strdup: the registry outlives any request, and an
* arena-tracked copy would be freed at el_request_end leaving a
* dangling modality name if a program registers a realizer from
* inside a handler rather than at startup. */
_realizers[_realizer_count].modality = el_strdup_persist(m);
_realizers[_realizer_count].fn = (el_realizer_fn)sym;
_realizer_count++;
pthread_mutex_unlock(&_realizer_mu);
return (el_val_t)1;
}
pthread_mutex_unlock(&_realizer_mu);
return (el_val_t)0;
}
el_val_t realizer_has(el_val_t modality) {
const char* m = EL_CSTR(modality);
if (!m || !*m) return (el_val_t)0;
return realizer_lookup(m) ? (el_val_t)1 : (el_val_t)0;
}
/* transduce — THE primitive: signal in, geometry out.
*
* Dispatches to the realizer registered for `modality`. Returns 0 (not a
* Geometry) when no realizer is registered, and geometry_is() on the result
* is the check.
*
* There is deliberately NO built-in realizer, not even for text. A modality
* the program has declared no organ for is one it genuinely cannot sense,
* and returning nothing is more honest than quietly embedding a description
* of the signal and calling that perception which is the exact failure
* this whole change exists to end.
*
* The result is validated to actually BE a Geometry before it is handed
* back, so a realizer that returns something else transduced nothing rather
* than handing a caller a value that will misbehave far from here. */
el_val_t transduce(el_val_t signal, el_val_t modality) {
const char* m = EL_CSTR(modality);
if (!m || !*m) return (el_val_t)0;
el_realizer_fn fn = realizer_lookup(m);
if (!fn) return (el_val_t)0;
el_val_t g = fn(signal);
return geom_of(g) ? g : (el_val_t)0;
}
/* ── Batch 3: Engram in-process graph store ──────────────────────────────── */
/*
* Single global EngramStore allocated lazily on first call. All node and
@@ -7715,10 +8159,14 @@ static double engram_decode_score(el_val_t v) {
return (double)n;
}
/* Truncate to at most n BYTES without splitting a UTF-8 codepoint. The old
* implementation was `if (l > n) l = n;` a byte cut that could land inside a
* multi-byte character and emit a half codepoint into the node's label, which
* then propagated into every JSON document containing that node. See
* el_utf8_safe_len for the measurement. */
static char* engram_first_n_chars(const char* s, size_t n) {
if (!s) return el_strdup("");
size_t l = strlen(s);
if (l > n) l = n;
size_t l = el_utf8_safe_len(s, n);
char* out = el_strbuf(l);
memcpy(out, s, l);
out[l] = '\0';
@@ -8564,80 +9012,96 @@ el_val_t engram_node_count(void) {
return (el_val_t)engram_get()->node_count;
}
/* engram_node_set_emb — attach GEOMETRY to an existing node.
/* node_attach_geometry — a node acquires geometry.
*
* WHY THIS EXISTS (2026-08-16). Until now no ingest path could carry a
* vector. engram_node / engram_node_full / engram_node_layered take text
* only, and the sole way a node acquired an embedding was
* engram_embed_backfill DERIVING one from n->content. That made text the
* mandatory entry medium: any non-text modality (audio, image, sensor)
* had to be described in prose first, and the geometry we then reasoned
* over was the geometry OF THE DESCRIPTION, not of the signal. Measured
* consequence: POST /api/nodes accepted an "emb" field, returned 200 with
* a fresh id, and stored emb_dim=None / embedded:false the vector was
* silently discarded because no parameter existed to receive it.
* Named for the operation, not for the store that happens to hold the node.
* This is the geometry-valued ingest path that replaces #141's hex-string
* one: nothing here parses text, and nothing here takes a caller's word for
* how wide the vector is. The Geometry carries its own width.
*
* `hex` is little-endian float32, the encoding the perception vessel's
* /voice/embed already emits, so a realizer's output moves in without a
* JSON float-array round trip. Length must be exactly dim*8 hex chars.
* The payload is COPIED into the node, so the node and the caller's Geometry
* have independent lifetimes the caller may geometry_free() immediately
* after, and a later free of the node's emb never touches the El value.
*
* DIMENSION POLICY: dim need NOT equal the canonical text-embedding dim.
* A modality vector of a different width is stored and is simply not
* inserted into the resident HNSW index, whose build loop already filters
* on `n->emb_dim == dim`. So off-dimension geometry is durable and
* addressable without perturbing the canonical index.
* DIMENSION POLICY (measured in #141, load-bearing do not regress): dim
* need NOT equal the canonical text-embedding width. An off-dimension vector
* is stored and is simply not inserted into the resident HNSW index, whose
* build loop already filters on `n->emb_dim == dim`. So a 64-dim voice
* geometry is durable and addressable without perturbing the 768-dim
* canonical index.
*
* Setting emb also makes the node ineligible for embed_backfill (which
* only fills nodes with no emb), so a realizer's vector is never
* Attaching geometry also makes the node ineligible for embed_backfill
* (which fills only nodes with no emb), so a realizer's vector is never
* overwritten by a text-derived one.
*
* Returns 1 on success, 0 on unknown id / malformed hex / bad dim. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim) {
const char* sid = EL_CSTR(id);
const char* sh = EL_CSTR(hex);
int32_t d = (int32_t)(int64_t)dim;
/* Bound the allocation. No max-dim constant existed because no caller
* could supply a dim before this function; 8192 is generous for any
* realizer (canonical text embeddings are 768, MFCC voice stats 64)
* while keeping a malformed `dim` from requesting an unbounded malloc. */
if (!sid || !*sid || !sh || d <= 0 || d > 8192) return (el_val_t)0;
* Returns 1 on success, 0 on unknown id or a value that is not a Geometry. */
el_val_t node_attach_geometry(el_val_t node_id, el_val_t g) {
const char* sid = EL_CSTR(node_id);
if (!sid || !*sid) return (el_val_t)0;
size_t need = (size_t)d * 8u; /* 4 bytes → 8 hex chars per float */
if (strlen(sh) != need) return (el_val_t)0;
ElGeometry* p = geom_of(g);
if (!p || p->dim <= 0) return (el_val_t)0;
EngramNode* n = engram_find_node(sid);
if (!n) return (el_val_t)0;
float* v = (float*)malloc(sizeof(float) * (size_t)d);
float* v = (float*)malloc(sizeof(float) * (size_t)p->dim);
if (!v) return (el_val_t)0;
for (int32_t i = 0; i < d; i++) {
uint32_t w = 0;
for (int k = 0; k < 8; k++) {
char c = sh[(size_t)i * 8u + (size_t)k];
uint32_t nib;
if (c >= '0' && c <= '9') nib = (uint32_t)(c - '0');
else if (c >= 'a' && c <= 'f') nib = (uint32_t)(c - 'a' + 10);
else if (c >= 'A' && c <= 'F') nib = (uint32_t)(c - 'A' + 10);
else { free(v); return (el_val_t)0; }
w = (w << 4) | nib;
}
/* Hex is emitted little-endian byte order; rebuild the word. */
uint32_t le = ((w & 0x000000FFu) << 24) | ((w & 0x0000FF00u) << 8) |
((w & 0x00FF0000u) >> 8) | ((w & 0xFF000000u) >> 24);
float f;
memcpy(&f, &le, sizeof(f));
v[i] = f;
}
memcpy(v, p->v, sizeof(float) * (size_t)p->dim);
free(n->emb);
n->emb = v;
n->emb_dim = d;
n->emb = v;
n->emb_dim = p->dim;
n->updated_at = engram_now_ms();
if (engram_store_enabled()) eg_store_put_node(n);
return (el_val_t)1;
}
/* node_geometry_dim — read the attached width back, 0 if the node carries
* none. Exists so an attach is VERIFIED by reading it back rather than by
* trusting a success return. That is not a nicety: #141 was misdiagnosed for
* an hour precisely because a genuine ingest drop and a mere reporting gap
* were indistinguishable from the outside. */
el_val_t node_geometry_dim(el_val_t node_id) {
const char* sid = EL_CSTR(node_id);
if (!sid || !*sid) return (el_val_t)0;
EngramNode* n = engram_find_node(sid);
if (!n || !n->emb) return (el_val_t)0;
return (el_val_t)n->emb_dim;
}
/* engram_node_set_emb — DEPRECATED. Shipped in #141; superseded 2026-08-16
* by geometry_from_f32le_hex + node_attach_geometry, and now implemented as
* literally that.
*
* It is kept, rather than removed, for one reason only: the runtime is
* published as an SDK asset, so a downstream binary may already be linking
* this symbol. It is NOT kept because a hex string is an acceptable way to
* move geometry between two pieces of El it isn't, and that was the
* placement defect. New code calls transduce() or geometry_from_f32le_hex()
* plus node_attach_geometry().
*
* The #141 contract is preserved exactly, including its negative cases, so
* this remains a drop-in: `dim` <= 0 rejects, malformed hex rejects, and a
* `dim` that disagrees with the vector's actual width rejects. The
* difference is that `dim` is now an ASSERTION checked against a width the
* Geometry already knows, rather than the authority the allocation trusted
* which is why #141's arbitrary `dim <= 8192` guard has no counterpart here.
* There is no longer an unbounded-malloc hazard to guard against. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim) {
int32_t want = (int32_t)(int64_t)dim;
if (want <= 0) return (el_val_t)0;
el_val_t gv = geometry_from_f32le_hex(hex);
ElGeometry* p = geom_of(gv);
if (!p) return (el_val_t)0; /* empty / malformed hex */
if (p->dim != want) { geometry_free(gv); return (el_val_t)0; } /* length mismatch */
el_val_t ok = node_attach_geometry(id, gv);
geometry_free(gv);
return ok;
}
/* ── Telemetry retention ────────────────────────────────────────────────────
* (2026-07-16 self-review) InternalStateEvent nodes are append-only telemetry
* (heartbeat, curiosity_scan, engram_sync) written ~3/min by the awareness
@@ -13938,7 +14402,40 @@ static int eg_cog_is_keystone_seeds(const char* csv) {
el_val_t engram_think_json(el_val_t seeds, el_val_t faculty) {
GeoDescriptor* g = eg_geo_build_desc(EL_CSTR(seeds));
if (!g) return eg_geo_err("geometry unavailable");
CogStance st; cog_stance_init(&st, NULL, EL_CSTR(faculty), g->hub_id, NULL, g);
/* RESUME THE LEARNED STANCE (2026-08-16 self-review). This built a NEUTRAL
* stance every call all axis_gain 1.0, bias_dir NULL, reliability 0.5
* and never loaded the one the correspondence-beat had been persisting.
*
* That mattered because the faculty enters engram_think ONLY through the
* stance: `gain = stance->axis_gain[k]` warps the per-axis extents, and
* `stance->bias_dir` seeds the steering direction. cog_stance_init stores
* the faculty NAME but nothing reads it. So with a neutral stance,
* reason / abduce / induce / plan / analogize are the same function with
* different labels measured, byte-identical output across all five
* and `confidence` is pinned to the 0.5 uninformed prior, because
* GeoGradient.confidence is just stance->reliability.
*
* The machinery already existed and only this call site ignored it:
* engram_correspondence_beat_json resumes via cog_stance_from_node and
* persists via cog_stance_to_node under the id "stance-<faculty>-<hub>".
* Every beat's calibration was being written and then thrown away on the
* next read. Same defect as the NULL anchor directly above: a neutral
* argument collapsing a capability to a constant.
*
* Resume the same id the beat writes, so learning compounds across beats
* and cold boot. Fall back to neutral only when no stance exists yet
* which is a genuine uninformed prior, not a discarded informed one. */
char sid[256];
snprintf(sid, sizeof sid, "stance-%s-%s",
EL_CSTR(faculty) ? EL_CSTR(faculty) : "reason",
g->hub_id ? g->hub_id : "region");
CogStance st; StoreNode prev; int resumed = 0;
if (g_engram_store && store_get_node(g_engram_store, sid, &prev) == 1) {
if (cog_stance_from_node(&prev, &st) == 0) resumed = 1;
store_node_free(&prev);
}
if (!resumed) cog_stance_init(&st, sid, EL_CSTR(faculty), g->hub_id, NULL, g);
else { free(st.id); st.id = strdup(sid); }
GeoGradient grad;
/* ANCHOR THE READ (2026-08-16 self-review). This passed NULL, and NULL is
@@ -14000,8 +14497,13 @@ el_val_t engram_think_json(el_val_t seeds, el_val_t faculty) {
if (engram_think(g, anchor, &st, &grad) != 0) { free(anchor); cog_stance_free(&st); engram_geo_free(g); return eg_geo_err("think failed"); }
free(anchor);
JsonBuf b; jb_init(&b); char t[256];
snprintf(t, sizeof t, "{\"faculty\":\"%s\",\"n_support\":%d,\"magnitude\":%.6g,\"spread\":%.6g,\"confidence\":%.6g,\"dim\":%d",
EL_CSTR(faculty), grad.n_support, grad.magnitude, grad.spread, grad.confidence, grad.dim);
/* stance_resumed distinguishes an INFORMED read from an uninformed one.
* Without it, confidence 0.5 from a learned-but-unreliable stance and
* confidence 0.5 from "no stance exists" are indistinguishable the same
* reporting gap that let the NULL anchor and the neutral stance hide. */
snprintf(t, sizeof t, "{\"faculty\":\"%s\",\"n_support\":%d,\"magnitude\":%.6g,\"spread\":%.6g,\"confidence\":%.6g,\"stance_resumed\":%s,\"dim\":%d",
EL_CSTR(faculty), grad.n_support, grad.magnitude, grad.spread, grad.confidence,
resumed ? "true" : "false", grad.dim);
jb_puts(&b, t);
int emit = grad.dim < 8 ? grad.dim : 8;
jb_puts(&b, ",\"direction\":"); eg_geo_emit_vec(&b, grad.direction, emit);
@@ -14025,12 +14527,57 @@ el_val_t engram_ground_json(el_val_t claim, el_val_t evidence, el_val_t for_whom
double grounding = (rc == 0) ? gr.grounding : 0.0;
if (rc == 0) engram_verify_grounding_free(&gr);
const char* fw = EL_CSTR(for_whom); if (fw && !*fw) fw = NULL;
const char* cid = C->hub_id ? C->hub_id : EL_CSTR(claim);
const char* eid = E->hub_id ? E->hub_id : EL_CSTR(evidence);
int wr = cog_ground_edge(g_engram_store, cid, eid, grounding, fw);
JsonBuf b; jb_init(&b); char t[256];
snprintf(t, sizeof t, "{\"relation\":\"grounded-by\",\"claim\":\"%s\",\"evidence\":\"%s\",\"for_whom\":\"%s\",\"grounding\":%.6g,\"written\":%s}",
cid, eid, fw ? fw : "-", grounding, wr == 0 ? "true" : "false");
/* GROUND THE NODE ASKED ABOUT, AND SAY WHAT WAS RESOLVED (2026-08-16
* self-review). This wrote the grounded-by edge between the two REGION
* HUBS and then echoed those hubs back in the "claim"/"evidence" fields
* as though they were the caller's input. Three consequences, all measured
* against the live store:
*
* 1. The edge landed on a node the caller never named. Asking to ground
* 3b9ced5d against 6edf8c79 wrote an edge on 6edf8c79 -> d0406dfd,
* because those were the hubs of the two regions.
* 2. When both seeds resolve into the same region, the hubs coincide and
* the call grounds a node against ITSELF, returning grounding = 1
* a perfect score with no evidence behind it. Two independent agents
* hit this and reported 0.885 / 0.909 self-groundings as confident.
* 3. The echo concealed both, because the response looked exactly like a
* successful grounding of the ids that were passed in.
*
* The region is HOW a claim is evaluated; it is not WHAT the claim is
* about. So the edge attaches to the requested ids, and the resolved hubs
* are reported separately under claim_region / evidence_region. When the
* two regions coincide, the grounding is degenerate by construction and is
* reported as such rather than as a confident 1.0. */
const char* cid = EL_CSTR(claim);
const char* eid = EL_CSTR(evidence);
const char* chub = C->hub_id ? C->hub_id : cid;
const char* ehub = E->hub_id ? E->hub_id : eid;
/* Degeneracy is broader than chub == ehub. Three circular shapes, each of
* which yields a high score for structural reasons rather than evidential
* ones, and all three were previously invisible:
* same-region both seeds resolve to one region grounding a thing
* against itself.
* claim-in-ev the claim's region hub IS the evidence node: the evidence
* sits at the centre of the claim's own neighbourhood.
* ev-in-claim the mirror case.
* Measured: grounding 3b9ced5d against 6edf8c79 scored 0.98883 purely
* because 6edf8c79 is the hub of 3b9ced5d's region. */
const char* degenerate = NULL;
if (chub && ehub && strcmp(chub, ehub) == 0) degenerate = "same-region";
else if (chub && eid && strcmp(chub, eid) == 0) degenerate = "claim-region-is-evidence";
else if (ehub && cid && strcmp(ehub, cid) == 0) degenerate = "evidence-region-is-claim";
if (degenerate) grounding = 0.0; /* circular support is not support */
/* Do not write an edge for a grounding that is degenerate by construction. */
int wr = degenerate ? -1 : cog_ground_edge(g_engram_store, cid, eid, grounding, fw);
JsonBuf b; jb_init(&b); char t[512];
snprintf(t, sizeof t, "{\"relation\":\"grounded-by\",\"claim\":\"%s\",\"evidence\":\"%s\","
"\"claim_region\":\"%s\",\"evidence_region\":\"%s\",\"degenerate\":%s%s%s,"
"\"for_whom\":\"%s\",\"grounding\":%.6g,\"written\":%s}",
cid ? cid : "", eid ? eid : "", chub ? chub : "", ehub ? ehub : "",
degenerate ? "\"" : "false", degenerate ? degenerate : "", degenerate ? "\"" : "",
fw ? fw : "-", grounding, wr == 0 ? "true" : "false");
jb_puts(&b, t);
engram_geo_free(C); engram_geo_free(E);
return el_wrap_str(b.buf);
+74 -4
View File
@@ -586,6 +586,60 @@ void el_runtime_dharma_event_arrive(const char* event_type,
const char* payload,
const char* source);
/* ── Geometry: signal as a first-class El value ──────────────────────────────
*
* A Geometry is an opaque, magic-tagged heap value carried in an el_val_t
* the same discipline as List/Map. It holds a width and a float32 payload,
* and it is the medium a non-text modality enters in. Declared HERE, above
* the engram block, because transduction is a LANGUAGE concern: every El
* program touching any modality needs it, and the engram is merely one El
* program that happens to hold a graph. See el_runtime.c ("Geometry: signal
* as a first-class el value") for the full rationale.
*
* El-side type annotation is simply `Geometry` an opaque boxed pointer,
* exactly like Instant / Calendar / Rhythm. No codegen change is required.
*
* OWNERSHIP: a Geometry is owned by the El caller and released with
* geometry_free. node_attach_geometry COPIES, so a node and the caller's
* value have independent lifetimes. */
el_val_t geometry_new(el_val_t dim); /* zero-filled; 0 on failure */
el_val_t geometry_dim(el_val_t g); /* width, 0 if not a Geometry */
el_val_t geometry_is(el_val_t g); /* 1 if a live Geometry */
el_val_t geometry_get(el_val_t g, el_val_t i); /* Float component */
el_val_t geometry_set(el_val_t g, el_val_t i, el_val_t x); /* 1 ok / 0 out of range */
el_val_t geometry_norm(el_val_t g); /* Float L2 — lets a caller
* check a realizer emitted
* signal, not zeros */
el_val_t geometry_free(el_val_t g); /* 1 if freed, 0 if not a Geometry.
* Returns a value (not void) so it
* is safe in any El expression
* position without a codegen
* void-builtin table entry. */
/* Wire ADAPTERS — the only place an encoding appears, and only at the edge.
* `f32le hex` is little-endian float32, 8 hex chars per component: the
* encoding the perception vessel's /voice/embed already emits. The width is
* DERIVED from the input length, never supplied by a caller which is why
* there is no max-dim constant here to validate a claimed length against. */
el_val_t geometry_from_f32le_hex(el_val_t hex); /* 0 on empty/odd-length/non-hex */
el_val_t geometry_to_f32le_hex(el_val_t g); /* "" if not a Geometry */
/* ── Realizers + transduce ───────────────────────────────────────────────────
* A REALIZER maps one modality into geometry. Registration is by NAME, so a
* new modality never requires a runtime patch: every El `fn name(...)`
* compiles to a global C symbol with that exact name, and the registry
* resolves it with dlsym against the running binary the same mechanism
* http_set_handler already relies on.
*
* fn tone_realizer(signal: String) -> Geometry { ... }
* realizer_register("tone", "tone_realizer")
* let g: Geometry = transduce(sample, "tone")
*/
el_val_t realizer_register(el_val_t modality, el_val_t fn_name); /* 1 ok / 0 unresolved */
el_val_t realizer_has(el_val_t modality); /* 1 if a realizer is registered */
el_val_t transduce(el_val_t signal, el_val_t modality); /* Geometry, or 0 if no organ */
/* ── Engram local graph primitives ───────────────────────────────────────────
* Operate on the CGI's local Engram knowledge graph.
* `engram_activate` queries the local graph only; `dharma_activate` is
@@ -612,11 +666,27 @@ el_val_t engram_get_node(el_val_t id);
void engram_strengthen(el_val_t node_id);
void engram_forget(el_val_t node_id);
el_val_t engram_prune_telemetry(el_val_t older_than_ms);
/* Largest byte length <= max_bytes that does not split a UTF-8 codepoint.
* Bounded by bytes, not codepoints, so truncated strings never grow. */
size_t el_utf8_safe_len(const char* s, size_t max_bytes);
el_val_t engram_node_count(void);
/* Attach geometry to an existing node. `hex` is little-endian float32,
* exactly dim*8 hex chars the encoding realizers already emit. Lets a
* non-text modality enter as geometry instead of being described in prose
* and embedded as its description. Returns 1 on success, 0 otherwise. */
/* Attach a Geometry to an existing node, and read the attached width back.
* Named for the operation, not the store: a node acquires geometry. This is
* the geometry-valued ingest path nothing about it is hex, and nothing
* about it assumes the caller's vector matches the canonical text-embedding
* width. node_geometry_dim exists so an attach is VERIFIED by reading it
* back rather than by trusting a success return. */
el_val_t node_attach_geometry(el_val_t node_id, el_val_t g); /* 1 ok / 0 otherwise */
el_val_t node_geometry_dim(el_val_t node_id); /* width, 0 if none */
/* DEPRECATED (shipped in #141, superseded 2026-08-16). Equivalent to
* geometry_from_f32le_hex + node_attach_geometry, and now implemented as
* exactly that. Kept only so anything built against the #141 runtime keeps
* linking; `dim` is accepted but treated as an assertion about the vector's
* width rather than as its source. New code should not call this a hex
* string is a wire encoding, not a way to move geometry between two pieces
* of El. Returns 1 on success, 0 otherwise. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim);
el_val_t engram_search(el_val_t query, el_val_t limit);
el_val_t engram_scan_nodes(el_val_t limit, el_val_t offset);
+234
View File
@@ -0,0 +1,234 @@
import "../../runtime/eltest.el"
// test_transduce.el geometry as a first-class El value, and realizers
// declared in El rather than patched into the runtime.
//
// WHAT IS ACTUALLY UNDER TEST. Until 2026-08-16 no El ingest path could carry
// a vector: nodes took text, and geometry was DERIVED from that text. Text was
// therefore the mandatory entry medium, so any non-text modality had to be
// DESCRIBED in prose first and the geometry we reasoned over was the geometry
// OF THE DESCRIPTION, not of the signal. The fix has two halves, and this file
// exercises both:
//
// 1. Geometry is a VALUE it carries its own width, so nothing has to
// assert a width against a string's length.
// 2. A REALIZER is an ordinary El function. `tone_realizer` below is not in
// the runtime, is not known to the compiler, and is not special in any
// way; it is registered BY NAME and dispatched to through transduce().
// That is the load-bearing claim: adding a modality must not require a
// runtime patch, or nothing has actually moved into the language.
//
// COMPARISON DISCIPLINE IN THIS FILE (measured 2026-08-16, not stylistic):
// elc lowers `a == b` to a NUMERIC comparison only when both operand names are
// in the per-function int-name set, which `let x: Int` populates. A bare call
// like `geometry_is(g) == 0` is not a registered name, so it lowers to
// `str_eq(...)` strcmp on two integers reinterpreted as pointers. `<` and `>`
// lower directly via binop_to_c with no type inference at all, so truthiness is
// written `> 0` / `< 1` here, and any exact `==` is done on a value first bound
// through `let x: Int`.
// A realizer, written entirely in El
// Maps a "tone" signal into a 4-component geometry. Deliberately trivial
// what is being proven is that an El function can BE a realizer, not that
// this is good acoustics. The one real property it has: distinct signals
// produce distinct geometry, so the test can tell transduction from a stub.
fn tone_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(4)
let n: Int = str_len(signal)
let a: Int = geometry_set(g, 0, int_to_float(n))
let b: Int = geometry_set(g, 1, int_to_float(n * 2))
let c: Int = geometry_set(g, 2, int_to_float(n * 3))
let d: Int = geometry_set(g, 3, int_to_float(n * 4))
g
}
// A second realizer for a different modality, to prove the registry keys on
// modality and does not just hand back "the last thing registered".
fn pulse_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(2)
let a: Int = geometry_set(g, 0, 1.0)
let b: Int = geometry_set(g, 1, 0.0)
g
}
// A deliberately BROKEN realizer: it returns something that is not a Geometry.
// transduce() must not hand this back to a caller as if it were one.
fn bogus_realizer(signal: String) -> Geometry {
return 12345
}
test "geometry-is-a-value-with-its-own-width" {
let g: Geometry = geometry_new(8)
let live: Int = geometry_is(g)
assert live > 0, "geometry_new returns a live Geometry"
let d: Int = geometry_dim(g)
assert d == 8, "a Geometry carries its own width"
let freed: Int = geometry_free(g)
assert freed > 0, "geometry_free reports what it did"
}
test "geometry-rejects-nonsense-without-an-arbitrary-bound" {
// dim <= 0 is not a width. Note there is deliberately no MAX dim here:
// #141 needed `dim <= 8192` only to bound an allocation sized from a
// caller's claim about a string. A value that carries its own width has
// nothing left to validate, so the only failure left is allocation.
let zero: Geometry = geometry_new(0)
let z: Int = geometry_is(zero)
assert z < 1, "dim 0 is not a geometry"
let neg: Geometry = geometry_new(-4)
let n: Int = geometry_is(neg)
assert n < 1, "negative dim is not a geometry"
// Accessors must be total: a non-geometry is 0-width, never a crash.
let nd: Int = geometry_dim(0)
assert nd < 1, "geometry_dim of a non-geometry is 0"
let ni: Int = geometry_is(0)
assert ni < 1, "geometry_is of a non-geometry is 0"
let nf: Int = geometry_free(0)
assert nf < 1, "geometry_free of a non-geometry is a no-op"
}
test "geometry-components-round-trip" {
let g: Geometry = geometry_new(3)
let s0: Int = geometry_set(g, 0, 1.5)
let s1: Int = geometry_set(g, 1, -2.5)
assert s0 > 0, "set in range succeeds"
let oob: Int = geometry_set(g, 3, 9.0)
assert oob < 1, "set out of range is refused, not silently dropped"
let v0: Float = geometry_get(g, 0)
let d0: Float = v0 - 1.5
assert d0 < 0.001, "component 0 round-trips"
assert d0 > -0.001, "component 0 round-trips"
let v1: Float = geometry_get(g, 1)
let d1: Float = v1 + 2.5
assert d1 < 0.001, "component 1 round-trips (negative)"
assert d1 > -0.001, "component 1 round-trips (negative)"
let freed: Int = geometry_free(g)
}
test "hex-is-an-edge-adapter-and-derives-its-own-width" {
// 2 components, little-endian float32: 1.0 = 0000803f, 2.0 = 00000040.
let g: Geometry = geometry_from_f32le_hex("0000803f00000040")
let live: Int = geometry_is(g)
assert live > 0, "valid hex decodes to a Geometry"
let d: Int = geometry_dim(g)
assert d == 2, "width is DERIVED from the input, never supplied"
let a: Float = geometry_get(g, 0)
let da: Float = a - 1.0
assert da < 0.001, "first component decoded"
assert da > -0.001, "first component decoded"
let b: Float = geometry_get(g, 1)
let db: Float = b - 2.0
assert db < 0.001, "second component decoded"
assert db > -0.001, "second component decoded"
// Egress adapter is the exact inverse.
let back: String = geometry_to_f32le_hex(g)
assert str_eq(back, "0000803f00000040"), "hex round-trips exactly"
let freed: Int = geometry_free(g)
}
test "hex-rejects-malformed-input" {
let empty: Geometry = geometry_from_f32le_hex("")
let e: Int = geometry_is(empty)
assert e < 1, "empty hex is not a geometry"
let ragged: Geometry = geometry_from_f32le_hex("0000803f0000")
let r: Int = geometry_is(ragged)
assert r < 1, "length not a multiple of 8 is refused"
let nonhex: Geometry = geometry_from_f32le_hex("zzzzzzzz")
let nh: Int = geometry_is(nonhex)
assert nh < 1, "non-hex characters are refused"
}
test "a-realizer-declared-in-el-is-a-first-class-realizer" {
// THE CLAIM: tone_realizer is an ordinary El function. It is not in the
// runtime and the compiler knows nothing about it. Registering it by name
// is enough to make it the organ for a modality.
let reg: Int = realizer_register("tone", "tone_realizer")
assert reg > 0, "an El fn registers as a realizer by name"
let has: Int = realizer_has("tone")
assert has > 0, "the modality now has an organ"
let g: Geometry = transduce("aaa", "tone")
let live: Int = geometry_is(g)
assert live > 0, "transduce returns real geometry"
let d: Int = geometry_dim(g)
assert d == 4, "the El realizer determined the width, not the runtime"
// str_len("aaa") == 3, so component 0 must be 3.0 proof the signal
// actually reached the El function rather than a stub answering for it.
let c0: Float = geometry_get(g, 0)
let dc: Float = c0 - 3.0
assert dc < 0.001, "the signal reached the El realizer"
assert dc > -0.001, "the signal reached the El realizer"
let freed: Int = geometry_free(g)
}
test "distinct-signals-transduce-to-distinct-geometry" {
let reg: Int = realizer_register("tone", "tone_realizer")
let g1: Geometry = transduce("aa", "tone")
let g2: Geometry = transduce("aaaaa", "tone")
let a: Float = geometry_get(g1, 0)
let b: Float = geometry_get(g2, 0)
let diff: Float = b - a
// 5 - 2 = 3. If transduction were a stub these would be equal.
assert diff > 2.9, "different signals produce different geometry"
assert diff < 3.1, "different signals produce different geometry"
let f1: Int = geometry_free(g1)
let f2: Int = geometry_free(g2)
}
test "the-registry-keys-on-modality" {
let r1: Int = realizer_register("tone", "tone_realizer")
let r2: Int = realizer_register("pulse", "pulse_realizer")
assert r2 > 0, "a second modality registers independently"
let gt: Geometry = transduce("aaa", "tone")
let gp: Geometry = transduce("aaa", "pulse")
let dt: Int = geometry_dim(gt)
let dp: Int = geometry_dim(gp)
assert dt == 4, "tone still routes to its own realizer"
assert dp == 2, "pulse routes to a different realizer"
let f1: Int = geometry_free(gt)
let f2: Int = geometry_free(gp)
}
test "no-organ-is-reported-as-no-organ" {
// A modality with no realizer must transduce to NOTHING. It must never
// fall back to embedding a description of the signal and calling that
// perception that silent substitution is the entire defect this change
// exists to end.
let has: Int = realizer_has("echolocation")
assert has < 1, "unregistered modality has no organ"
let g: Geometry = transduce("anything", "echolocation")
let live: Int = geometry_is(g)
assert live < 1, "no realizer means no geometry, not fake geometry"
}
test "registration-of-an-unresolvable-name-fails-loudly" {
// Reported at the moment of WIRING, not later as "this modality mysteriously
// produces nothing". Distinguishing "no organ" from "broken organ" is the
// lesson that made this whole change necessary.
let bad: Int = realizer_register("ghost", "no_such_function_anywhere")
assert bad < 1, "an unresolvable realizer name is a registration failure"
let has: Int = realizer_has("ghost")
assert has < 1, "and nothing gets registered"
}
test "a-realizer-returning-non-geometry-transduces-nothing" {
let reg: Int = realizer_register("bogus", "bogus_realizer")
assert reg > 0, "the symbol resolves, so registration succeeds"
// ...but the contract is enforced at the boundary, so the caller never
// receives a value that would misbehave far away from here.
let g: Geometry = transduce("x", "bogus")
let live: Int = geometry_is(g)
assert live < 1, "a non-Geometry return transduced nothing"
}
test "norm-lets-a-caller-check-a-realizer-emitted-signal" {
let g: Geometry = geometry_new(2)
let z: Float = geometry_norm(g)
assert z < 0.001, "a fresh geometry is zero — norm says so"
let s0: Int = geometry_set(g, 0, 3.0)
let s1: Int = geometry_set(g, 1, 4.0)
let n: Float = geometry_norm(g)
let dn: Float = n - 5.0
assert dn < 0.001, "3-4-5: norm is 5"
assert dn > -0.001, "3-4-5: norm is 5"
let freed: Int = geometry_free(g)
}