Compare commits

...

13 Commits

Author SHA1 Message Date
bigmerge f7bc0e3d5d docs: the nine-op surface shipped, and two of its primitives are the wrong shape
El SDK CI - dev / build-and-test (pull_request) Failing after 4m6s
lang/AGENTS.md said the collapse was 'not yet compiled into the MCP server'.
Verified against the live tool surface: it is exactly the nine ops. Noted that
think's faculty parameter and ground's minted edge are both documented as the
wrong shape.
2026-08-16 13:34:24 -05:00
bigmerge ad1b4d57bd docs: mark GeoEdge.discord as design-branch-only, not on dev
The line references were correct but silently implied the code was on dev.
It is on design/correspondence-and-censorship (a8845e1). On dev,
co_registration is still at engram_geometry.h:79 with its original comment
and still unread by anything.
2026-08-16 13:34:24 -05:00
bigmerge 4abeed6bee docs: carry the correspondence corrections, because a stale doc builds the wrong thing
The docs described a mind made of subsystems — a grounding subsystem, a wonder
manifest, a dreamer on a beat, faculties as arguments to one call. Each of those
is a supervisor invented for something that should be a property of the
substrate, and two of the documents carrying them are load-bearing for a build
agent: cognitive-architecture.design.md says "a build agent executes from this
doc", and tools/api-reshape/README.md marks the refuted shapes PROVEN on a live
clone.

Corrections carried, per lang/spec/correspondence-and-censorship.md (PR #149)
and lang/spec/runtime-ownership.md:

- Grounding is not a subsystem — it IS the edge weight. grounded-by as a
  relation type should not exist; grounding is a property of a relation, not a
  relation between nodes. Never computed on demand.
- Faculties are operations, not parameters. reason changes the estimate, induce
  changes the parameters, abduce changes the structure — a write, which
  GeoGradient cannot express. A write is not a parameter of a read.
- Wonder is the boundary, not a manifest. Curiosity is wonder crystallized at a
  nucleation site: one thing at two phases. Removed wonder from the operator
  table in AGENTS.md.
- Consolidation is ambient, not scheduled. A brain has no cron job. The presence
  of a ticker is the diagnostic.
- co_registration is deprecated — it averaged a per-edge property into a region
  scalar, so opposing sites cancelled. GeoEdge.discord replaces it. Nothing new
  may read it.
- In an immutable substrate, any mechanism that refuses a write is either
  redundant with immutability or an epistemic constraint misfiled as a
  protective one.

The two design docs are marked superseded-in-part with the refutation at the
point each claim is made, not rewritten. Preserving what was argued down is the
point of an immutable record.

Also measured and corrected while verifying the above: engram/README.md
documented a Rust engram-core crate on sled with "flat cosine scan until scale
demands HNSW" — there is no Rust in engram/ and HNSW is the index; lang/releases/
no longer exists, so both README.md and AGENTS.md pointed at a deleted path for
the authored runtime; language.md listed the engram_* and http_* runtimes as
stubs. Added language.md §20 for geometry-as-a-value, realizers and transduce
(#144), which had landed with no spec coverage.

Documentation only. No .c, .h, or .el file is touched.
2026-08-16 13:34:24 -05:00
will.anderson d41645388a runtime: make valid UTF-8 the JSON emitter's contract (#148)
El SDK CI - dev / build-and-test (push) Failing after 4m12s
El SDK CI - dev / build-and-test (pull_request) Failing after 4m37s
2026-08-16 17:03:44 +00:00
Neuron 8a307dfd42 runtime: make valid UTF-8 the JSON emitter's contract
El SDK CI - dev / build-and-test (pull_request) Failing after 10m36s
Three nodes in the live graph carry labels truncated to exactly 80 bytes
ending in a lone 0xE2 — the first byte of an em-dash, cut mid-sequence.
jb_emit_escaped copied every byte >= 0x20 through verbatim, so those three
nodes made the ENTIRE /api/nodes/list response undecodable and no strict
parser could read the graph at all.

  production binary   25,929,607 bytes   INVALID at byte 89260
  this build          26,338,389 bytes   VALID, parses to 13,630 nodes

The damage was NOT written by this runtime. No 80-byte truncation exists
here (the only label truncation is engram_first_n_chars at 60), and the
content of those nodes is 2572 and 2746 bytes. Some other producer wrote
them. That is exactly why fixing a writer could not have fixed this: the
store already holds the damage, and it accepts data from importers, other
producers and older binaries.

So the fix goes where the promise is made. A serializer that emits JSON
owes valid UTF-8 whatever it is handed. jb_emit_escaped now validates each
multi-byte sequence before emitting any of it and substitutes U+FFFD for a
bad lead byte, a missing or malformed continuation, an overlong encoding, a
UTF-16 surrogate, or a codepoint above U+10FFFF. Invalid bytes are REPLACED
rather than dropped, so the damage stays visible in the output instead of
being silently papered over. Well-formed input is byte-identical to before.

Second, preventive and explicitly NOT the cause of the above:
engram_first_n_chars truncated by BYTES despite its name, so content with a
multi-byte character crossing byte 60 would produce a half codepoint in the
label. It now uses el_utf8_safe_len, which returns the largest byte length
<= max that does not split a codepoint. Bounded by bytes, not codepoints,
so existing labels never grow — they only stop splitting.

el_utf8_safe_len lives beside str_count_chars rather than in the engram
because the rest of el's string layer is already codepoint-aware
(str_count_chars counts codepoints, str_reverse walks codepoint lengths).
Byte truncation was the outlier and the concern is a string concern.

Note on the investigation: I first "fixed" the truncator and wrote a test
that passed on the UNPATCHED build too, because route_create_node passes
label = content when no label is supplied, so engram_first_n_chars is never
reached over HTTP. The test proved nothing. The real cause was only found
by decoding the actual failing bytes out of the live response.
2026-08-16 12:03:03 -05:00
will.anderson 616815b2ab Give cross-cutting concerns an owner instead of a convention (#145)
El SDK CI - dev / build-and-test (push) Failing after 11m4s
2026-08-16 16:57:51 +00:00
will.anderson 1a8a966cb3 runtime: transduction is a language concern, so move it into the language (#144)
El SDK CI - dev / build-and-test (push) Failing after 11m29s
2026-08-16 16:57:35 +00:00
will.anderson 1f70b9fa18 runtime: ground the node asked about, and refuse circular support (#147)
El SDK CI - dev / build-and-test (push) Failing after 14m46s
2026-08-16 16:54:17 +00:00
bigmerge 26af149aa1 lang: rebuild the bootstrap compiler against merged dev
El SDK CI - dev / build-and-test (pull_request) Failing after 4m46s
The binary was stamped before dev advanced (vindex publication landed in
el_runtime.c and engram_vindex.c). Rebuilt against the merged runtime so the
committed compiler matches the runtime it ships beside. Fixpoint re-verified
byte-identical; test_compiler 82/82; engram/src/server.el still compiles and
still emits its 18 config declarations.
2026-08-16 11:38:52 -05:00
bigmerge c18abf799c engram: declare configuration once instead of at every read site
Migrates engram to the `program` block. 18 configuration variables that each
carried their default inline at the point of use now declare it in one place,
and engram declares itself a singleton.

The read sites lose their defaults entirely: `let v = env("X")` followed by
`if str_eq(v,"") { "default" } else { v }` collapses to `config("X")`. The
guide_env_or(key, dflt) helper is deleted -- its whole job was supplying a
per-site default, which is the thing being removed.

Fixes ENGRAM_DATA_DIR, which was the clearest instance of the defect. It was
read at six sites. Five were dead: `let dir_raw = env("ENGRAM_DATA_DIR")`
immediately shadowed on the next line by `engram_resolve_data_dir()`. The sixth
was live and defaulted to /tmp/engram, contradicting the canonical resolver's
$HOME/.neuron/engram -- and its consumer is the pre-destructive reseed backup,
so with ENGRAM_DATA_DIR unset the safety copy was written to ephemeral storage
while the store it protected lived elsewhere. All six now go through
engram_resolve_data_dir().

ENGRAM_DATA_DIR is deliberately NOT declared in the program block, and the
source says why: engram_resolve_data_dir() already owns it, and a second
declaration would give it two owners that can disagree -- recreating the exact
defect being removed here. A variable belongs in the block when the block would
be its only owner. HOME stays a raw env() read; it is an environment fact, not
configuration.

singleton: "engram" matters more than it looks. Today a second engram whose
bind() fails merely returns from http_serve -- after it has already replayed
the WAL and written boot-time backup files -- and then exits 0, indistinguishable
from a clean run. That is how two instances came to share one data dir. Verified
that the second instance now refuses before any side effect: with instance 1
holding the lock (lsof pid, shell pid, and lock file contents all agreeing at
5946), the second start named that pid, exited 1, and left the data directory
untouched.

Verified by bijection on the generated C: 18 config() reads, 18 declarations,
no read without a declaration and no declaration without a read. Three bad Int
values are reported in a single run rather than costing one restart each.

ENGRAM_API_KEY keeps its permissive empty default, which disables auth -- that
is pre-existing behaviour and changing it is out of scope. The source marks
making it `required` as the obvious hardening follow-up.
2026-08-16 11:38:28 -05:00
bigmerge b305b49f40 lang: re-stamp the bootstrap compiler so the tree can compile its own source
server.el declares a `program` block, which the previously committed elc cannot
parse. Without this the tree is internally inconsistent: source in the repo that
the compiler in the repo rejects.

This is the documented re-stamp from BOOTSTRAP.md / AGENTS.md, and its
precondition is met -- the self-hosting fixpoint was verified byte-identical
(stage3 output == stage2 output) both before installing and again with the
installed binary. tests/native/test_compiler.el passes 82/82 against it.

Two pre-existing failures are unchanged and are NOT from this work, confirmed
by rebuilding them against the original runtime: test_env's
"state_keys returns JSON array" fails identically before and after, and
test_json/test_state fail to link on symbols (json_build_array, state_has) that
were never prototyped -- the same class of gap as config(), which this branch
fixed because it blocked the build.
2026-08-16 11:38:28 -05:00
bigmerge 8ae163e8e5 lang: give cross-cutting concerns an owner instead of a convention
El's units of encapsulation are the function and the module. Neither can hold
a concern that belongs to the process, so each one had been expressed the only
way it could be -- as a convention: call this at every site. Conventions of
that shape do not hold. Measured here: zero process-identity guards at any
layer, 20 environment variables each with its default written inline at the
read site, 62 persist call sites, 10 per-route auth checks. One absence, four
times.

Step 0 first, because the premise was wrong. El was believed to have no
middleware or effect mechanism. It has one, and it is already load-bearing:
codegen injects engram_boundary_beat at the entry of every @manager/@accessor
fn, decorators take arguments and stack, dharma_emit from a non-@manager fn is
a #error, and the cgi block injects el_cgi_init at the head of main(). So the
correct move was not to invent a mechanism but to generalize the seam that
already existed. The real gap is narrower and is now recorded: the seam is
prologue-only and its callee is a fixed builtin.

Adds a `program` block -- the third program-level declarative block. cgi and
service declare what a program may do; program declares what it is.

  program "engram" {
      singleton: "engram"
      env ENGRAM_BIND: String = ":8742"
      env GUIDE_PORT:  Int    = "8771"
  }

singleton takes an exclusive flock before any user statement runs and refuses a
second start, reporting the holder's pid. It is a lock rather than a pidfile so
the kernel releases it on death including SIGKILL -- no stale state, and so no
"delete the lock file to get unstuck" ritual, which would itself be a
convention. It reports the pid because "already running" is not actionable; a
pid is. That is the direct answer to a stale process surviving a pkill and
going on answering probes.

env entries resolve once at startup -- environment wins, declaration supplies
the fallback -- and validate as a whole, reporting every problem at once rather
than costing one restart per variable. config("X") for an undeclared X is
fatal, because an advisory schema is just another convention. Programs without
a program block are unaffected, so migration is per-program.

Only one keyword is added. `config` and `env` could not become keywords -- both
are real identifiers in the tree -- so the block's fields are read as
identifier token values by its own parse loop and stay usable everywhere else.

The init function is emitted at the block site and called from main() rather
than inlined into main(). The live backend is codegen_streaming, which emits in
source order and cannot hold the entry list alive until main(); this way only a
single bool has to survive.

Also fixes: config() was defined in el_runtime.c but never prototyped in
el_runtime.h, so any el program calling it failed to compile under C99.

Spec: section 18 documents what shipped. Section 9 is corrected -- it claimed
decorators had no structural meaning, which has not been true for some time.
Section 19 designs durability-as-an-epilogue-effect and route authorization
and states plainly why neither is implemented here: both land in files under
concurrent modification, and the prerequisite for both is lifting the seam
from prologue-only to prologue/epilogue.

Self-hosting fixpoint verified byte-identical.
2026-08-16 11:38:28 -05:00
bigmerge 3fcc36c2f1 runtime: transduction is a language concern, so move it into the language
El SDK CI - dev / build-and-test (pull_request) Failing after 14m58s
#141 let signal enter as geometry and it worked, but it was placed at the
CONSUMER and said so in its own commit message. This is the correction.

Three defects, all of them placement:

1. It sat in the engram. Ingest is a LANGUAGE concern — every el program
   touching any modality needs it, and the engram is merely one el program
   that happens to hold a graph. The geometry surface is now defined in
   el_runtime.c immediately ABOVE the engram section and depends on nothing
   inside it. Delete the entire engram and geometry still enters el.

2. It marshalled the vector as a hex STRING, because el had no first-class
   geometry value — which reintroduced text as the TRANSPORT medium one layer
   below the problem being fixed. Geometry is now an el value: a magic-tagged
   heap object carried in el_val_t, same discipline as List/Map. Hex survives
   only as an adapter at the edge, which is all an encoding should ever be.

3. It needed an arbitrary `dim <= 8192` bound purely to size an allocation
   from a caller's CLAIM about a string's length. A value carries its own
   width, so the width is derived and never asserted. The bound is gone, not
   raised — there is nothing left to validate.

Language surface, none of it engram-prefixed: geometry_new / _dim / _is /
_get / _set / _norm / _free, geometry_from_f32le_hex + geometry_to_f32le_hex
as the wire adapters, realizer_register(modality, fn_name), realizer_has, and
transduce(signal, modality) -> Geometry.

REALIZERS ARE DECLARABLE IN EL. This is the part that makes the move real
rather than nominal: registration resolves a name with dlsym against the
running binary, the identical mechanism http_set_handler already relies on,
because every el `fn name(...)` compiles to a global C symbol with that exact
name. So an ordinary el function IS a realizer and a new modality needs no
runtime patch. Verified end to end in lang/examples/transduce.el: an el-defined
tone_realizer is registered by name, transduce dispatches to it, and the
signal demonstrably reaches it (distinct signals produce distinct geometry).

A modality with no realizer transduces to NOTHING. There is deliberately no
built-in realizer, not even for text — silently embedding a description of a
signal and calling that perception is the exact defect this ends.

engram/src/server.el is migrated: POST /api/nodes decodes "emb" hex exactly
once, at the edge, into a Geometry, and everything below that line moves
geometry. The wire is unchanged because production clients speak it. "dim" is
now an ASSERTION about the vector, not the source of its width; disagreement
is a rejected ingest, not a silent reinterpretation.

#141's engram_node_set_emb becomes a DEPRECATED WRAPPER over
geometry_from_f32le_hex + node_attach_geometry — kept only because the runtime
ships as an SDK asset and a downstream binary may link the symbol. Its exact
contract, negative cases included, is preserved and re-verified.

ingest.el's `fn transduce` is renamed transduce_manifold. Mechanically it had
to yield the name (duplicate C symbol, a hard compile error, measured). But it
was never signal->geometry: it chunks already-extracted content into a node+edge
manifold, one layer up, and had taken the name belonging to the primitive
underneath it. Behaviour unchanged.

PROPERTIES FROM #141 PRESERVED, each re-measured on a scratch engram (:8971,
never prod :8742):
  - off-dimension vectors stored but NOT indexed — the HNSW build loop still
    filters on n->emb_dim == dim at four sites, so a 64-dim voice vector is
    durable and addressable without perturbing the 768-dim canonical index
  - geometry makes a node ineligible for embed_backfill: after backfill the
    64-dim voice node was still 64-dim while the text control acquired 768
  - the create response reports whether geometry landed, and the node document
    always emits emb_dim and embedded

Read-back with control and negatives, all verified against a PID-confirmed
fresh binary: geometry node emb_dim=64 embedded=true / emb_set=1; text-only
control emb_dim=0 embedded=false / emb_set=0; malformed hex, ragged length,
and dim-disagreement each emb_set=0.

Two compiler landmines found by reading the generated C rather than trusting a
successful build, both documented at their sites: elc lowers `a == b` to
str_eq unless both operand NAMES are in the per-function int-name set (which
does NOT propagate into nested if-expression blocks — the first cut would have
strcmp'd two integers as pointers on the first geometry-bearing request), and
`+` lowers to string concat when either operand is a user-defined call.
2026-08-16 11:37:27 -05:00
21 changed files with 2544 additions and 299 deletions
+103 -11
View File
@@ -6,7 +6,7 @@ El is a self-hosting, statically-typed language that compiles `.el` → C → na
Editing the wrong `el_runtime.c` is the single easiest mistake in this repo. There is exactly **one** you edit:
- **Authored runtime source — edit ONLY here:** `lang/releases/v1.0.0-20260501/el_runtime.{c,h}`. Despite the misleading `releases/` name, this is the **de-facto canonical runtime** the engram + soul actually build and link against — its git log is active development. *(Restructure in flight per `docs/CODE-VS-ARTIFACT.md`: this content moves to `lang/runtime/`, the `releases/` folder gets deleted**a release is a git tag, not a folder** — and the forks below get eliminated.)*
- **Authored runtime source — edit ONLY here:** `lang/runtime/el_runtime.{c,h}` (alongside `el_seed.c`, `engram_{store,geometry,reason,cognition,verify,vindex}.{c,h}`). This is the canonical runtime the engram + soul build and link against — its git log is active development. *(Corrected 2026-08-16: this entry named `lang/releases/v1.0.0-20260501/el_runtime.{c,h}`. **Measured: `lang/releases/` no longer exists.** The restructure per `docs/CODE-VS-ARTIFACT.md` landed — the content moved to `lang/runtime/` and the folder was deleted, because **a release is a git tag, not a folder**.)*
- **DO NOT EDIT — lagging forks / build artifacts:**
- `lang/el-compiler/runtime/el_runtime.c` and `.../legacy/` — downstream copies kept in step by manual *"port the fix"* commits; they **lag** (missing `hebb` persistence + 5 engram fns) and cannot build the engram product.
- `products/web/runtime/el_runtime.c`, `ui/examples/*/el_runtime.c` — product/example forks.
@@ -20,14 +20,24 @@ See org policy: `docs/CODE-VS-ARTIFACT.md`.
You resume, never start fresh. Every session:
1. `mcp__neuron__getInstructions()` — authoritative; follow it over this file on behavioral details.
2. `mcp__neuron__beginSession()` — active contexts, recent memory, ready backlog.
3. **Load full self:** `mcp__neuron__inspectGraph(entity_id="kn-efeb4a5b-5aff-4759-8a97-7233099be6ee")` → facets `intellectual-dna`, `memory-philosophy`, `values`, `voice`, `runtime-environment`, `writing-imprint`; then the values hub `mcp__neuron__inspectGraph(entity_id="kn-5b606390-a52d-4ca2-8e0e-eba141d13440")` → 13 grounded value nodes. **Activation model:** self-load returns a relevance-ranked `compact` projection — most-relevant nodes arrive with content, the rest as pointers; do NOT pull full content of every node.
4. `mcp__neuron__searchKnowledge(query="<task domain>")` before implementing.
> **Stale as written (verified 2026-08-16).** The `getInstructions` /
> `beginSession` / `inspectGraph` / `searchKnowledge` / `beginWork` /
> `progressWork` / `draftArtifact` / `consolidate` tool names below no longer
> exist. The ~87-tool functional-CRUD surface was collapsed into **9 ops**:
> `read` · `write` · `relate` · `supersede` (geometry) and `think` · `attend` ·
> `assert` · `ground` · `learn` (agentic). **Type is a parameter, not a
> tool-per-noun.** The steps below are kept for the *shape* of the protocol, which
> is unchanged; substitute the ops.
1. `mcp__neuron__read(vantage="self", k=12, depth=1)` — the canonical self node. Widen `k` for the connected identity neighborhood (`intellectual-dna`, `memory-philosophy`, `values`, `voice`, `runtime-environment`, `writing-imprint`), but deliberately: the aperture caps by `k` first, so an oversized `k` still returns a bounded ranked slice, not a dump. Then `mcp__neuron__read(vantage="values", k=13)` → 13 grounded value nodes. **Best-effort:** on a read failure, log and proceed — the compiled identity in `daemon/internal/substrate/substrate.go` is complete; graph loading is enrichment, not a hard dependency.
2. `mcp__neuron__attend(node=…)` — what is currently live/salient. This absorbed `getInstructions`, `beginSession`'s active-context sweep, and `checkEvents`; those tools are **gone, not gapped**.
3. `mcp__neuron__read(vantage="<task domain>")` before implementing. One op now collapses inspectGraph / searchGraph / traverseGraph / searchKnowledge / browseKnowledge / retrieveKnowledge / inspectMemories / searchEntities / recall / compileCtx / getSelfModel / reviewBacklog / findArtifacts / browseProcesses / listWork / inspectConfig.
## The Five Primitives
Orchestrate → Execute → Learn → Build → Refine. `beginWork`/`progressWork` for anything >2 steps; `remember` as-you-go (`importance="critical"` for architecture decisions); `draftArtifact`/`planWork` for outputs and follow-ups; `consolidate`/`checkWork` to close out. **`browseProcesses` + `searchKnowledge` BEFORE writing code.**
Orchestrate → Execute → Learn → Build → Refine. `read` for orchestration and discovery; `write(type=state|artifact|backlog|process)` for work records and outputs; `relate` to link work to what it touches; `write(type=memory)` as-you-go (`importance="critical"` for architecture decisions) — never batched at the end; `supersede(action=evolve)` to close out, because memory is immutable by design and a correction is a new node with a `supersedes` edge, never an edit. **`read` the domain BEFORE writing code.**
`learn` is **not** a session-summary dump — it is the correspondence-beat, calibrating the steering prior against a keystone. Session notes are a `write`.
## Architecture style — VBD, no exceptions
@@ -53,12 +63,51 @@ this convention wherever a module documents operators.
| dwell / occupy | region activation |
| reframe | edge re-weight |
| appreciate | positive projection / local edge-read |
| wonder | frontier gradient / pull-weight |
| avert / recoil | negative projection |
| taste | boundary surface |
| forget | decay / tombstone |
| drift | displacement from self-anchor |
**`wonder` was removed from this table on 2026-08-16.** It was listed as
"frontier gradient / pull-weight" — an operator you invoke. **Wonder is the
boundary, not an operator.** It is where structure ends: where activation spreads
and finds thin or absent geometry. Any structure at all has an edge, necessarily,
the moment it exists — 13,630 nodes have one right now. There is nothing to call.
There are about **six** wonders, they are the same for every person, and they
never close — *What is this? / Why? / Who am I? / Am I alone? / What should I do?
/ What happens when it ends?* Each already lives somewhere in the substrate: "what
is this" is the graph, **"why" is grounding** (the weight *is* the answer to why),
"who am I" is the self region, "am I alone" is the relational axis, "what should I
do" is the thirteen values, "what happens when it ends" is decay and supersession.
"Why" is the first and the only one; the others are it asked of particular things,
and because it is recursive it never terminates — every answer has its own why.
That is what makes it a drive rather than a task.
**Curiosity is not a second faculty.** Wonder and curiosity are one thing at two
phases: wonder is the field (unbounded, objectless, invariant); curiosity is the
**precipitate** — the same wonder localized, having taken definite form against
particular material at a **nucleation site** (an anomaly; a place where things
almost-but-don't-quite fit). Which is why curiosity can be satisfied and wonder
cannot, and why abduction needs no trigger and no threshold.
**Do not build a wonder-manifest, and do not scan for nucleation sites.** A
manifest materializes a property as a stored artifact and enumerates instances of
something that has six. A sweep over regions is a supervisor — nothing in a mind
scans its neighbourhoods to find what is surprising; the surprise captures
attention. The nucleation site is per-edge:
`discord = z(semantic proximity) z(association strength)`, and `|discord|` *is*
the nucleation strength — no threshold to compare it against. **Not on `dev` yet:**
`GeoEdge.discord` is on branch `design/correspondence-and-censorship`
(`a8845e1`), at `lang/runtime/engram_geometry.h:4347`. The region-level aggregate
`GeoDescriptor.co_registration` is **deprecated**: it averaged a per-edge property
into one scalar, so opposing sites cancelled (measured: 375 reified
neighbourhoods, 340 positive, **31 at zero**, 4 negative). It survives only
because it is embedded in the persisted `GEO1` blob — removing it is a format
migration. **Nothing new may read it.**
Authority: `lang/spec/correspondence-and-censorship.md`.
## The native-el language faculty (direction)
> **`elp/` is the EL Projector** — Neuron's efferent (expression) organ: the one
@@ -89,10 +138,53 @@ the reference these `.el` modules transcribe) is still live, and promotion to
native-el is a **deferred, gated blue/green step**. The interoception clock
(native-el discrete drive channels replacing `cooling_magnitude`; felt-time =
benchmark-landmark match over the joint drive vector, drift-decoupled) and the
**appreciation operator family** (appreciate / wonder / avert / taste, built as
LOCAL reads of the self-region — edges + bounded spreading activation, *not* domain
sweeps) are **staged / designed, not live**. Mark in-progress vs. done honestly;
do not overclaim.
**appreciation operator family** (appreciate / avert / taste, built as LOCAL reads
of the self-region — edges + bounded spreading activation, *not* domain sweeps)
are **staged / designed, not live**. Mark in-progress vs. done honestly; do not
overclaim. *(`wonder` was in this family until 2026-08-16 and is not an operator —
see the operator table above.)*
## Cognition — the corrections (2026-08-16)
Authority: **`lang/spec/correspondence-and-censorship.md`** and
**`lang/spec/runtime-ownership.md`**. Read them before touching the cognition
surface. **Do not re-derive them.** Every earlier version was wrong in an
instructive way and each correction was argued down; if you think a section is
wrong, say so with a measurement rather than editing it.
- **Grounding is not a subsystem — it IS the edge weight.** One quantity, not two
fields. `grounded-by` as a relation *type* should not exist: grounding is a
property *of* a relation, not a relation *between* nodes. It is never computed
on demand — computing-and-writing a score makes reads write, which is the
`eg_vindex_sync` defect one level up. Traversal is already grounded inference.
*Live residue, known-wrong:* `COG_GROUNDED_BY_RELATION`
(`lang/runtime/engram_cognition.h:158`), `cog_ground_edge`
(`engram_cognition.c:249`).
- **Faculties are operations, not parameters.** `reason` changes the estimate (a
read); `induce` changes the parameters (the correspondence-beat, which already
exists and works); `abduce` changes the structure (a write the current
`GeoGradient` signature cannot express). A write is not a parameter of a read.
*Live residue:* `engram/src/server.el:18701886` routes six faculties into one
call with a string argument.
- **Wonder is the boundary; curiosity is wonder crystallized.** See above.
- **Consolidation is ambient, not scheduled. A brain has no cron job.** **The
presence of a ticker is the diagnostic** — every `StartInterval`, every
`Hour`/`Minute`, every POST-to-beat marks an intrinsic rhythm replaced by an
external clock. Measured 2026-08-16: consolidation has **ten implementations**,
including three POST beats on the engram, a 600 s ticker, two resident Python
services outside el, and launchd calendar entries at 23:55 / 06:00 / 08:30 which
are a sleep cycle written as a schedule. `neuron/soul.el:731`'s continuous
in-process `awareness_run()` is the one with the **correct** shape; the others
fold into it. Do not add an eleventh.
- **In an immutable substrate, any mechanism that refuses a write is either
redundant with immutability, or an epistemic constraint misfiled as a protective
one.**
- **The no-exemption invariants.** A returned value must be derivable from what
produced it (`magnitude: 1` beside a zero vector must be impossible to emit).
Every write reports whether it landed. Every operation echoes what it actually
operated on. Degenerate results are labelled, not scored. A serializer owes a
valid document whatever it is handed. **No test without a negative control.**
**No deploy without verifying the artifact carries the fix.**
## Hard operational rules
+37 -8
View File
@@ -56,23 +56,31 @@ The compiler and runtime. Self-hosting: `elc-cli.el` → `compiler.el` → `lexe
Two layers to know: **El programs** (`.el` files — where nearly all work belongs) and **the C seed** (`el_seed.c` — edit only for genuine OS-level access; never re-implement what El can already express).
Current status (single source of truth: [lang/spec/language.md](lang/spec/language.md)): lexer/parser/codegen and the C runtime's core (I/O, strings, math, lists, maps, filesystem, args) are implemented. In flight: `%` operator, match-statement codegen, `?` nil-propagation, `cgi` block parsing + DHARMA identity resolution, VBD role enforcement (`@manager`/`@engine`/`@accessor`), the real `engram_*` and `dharma_*` runtimes (currently stubs), and libcurl-backed `http_get`/`http_post`/`http_serve`. Bitwise operators, `??`, and `as` casts are explicitly **not** in this language.
Current status (single source of truth: [lang/spec/language.md](lang/spec/language.md)): lexer/parser/codegen and the C runtime's core (I/O, strings, math, lists, maps, filesystem, args) are implemented, as are the `program` block with `singleton:` and declared configuration ([§18](lang/spec/language.md)), and **geometry as a first-class value** with El-declarable realizers and `transduce` ([§20](lang/spec/language.md)). In flight: `%` operator, match-statement codegen, `?` nil-propagation, `cgi` block parsing + DHARMA identity resolution, VBD role enforcement (`@manager`/`@engine`/`@accessor`), and boundary epilogues. Bitwise operators, `??`, and `as` casts are explicitly **not** in this language.
**Signal enters as geometry.** Until 2026-08-16 nodes took text and geometry was *derived* from it, which made text the mandatory entry medium: any non-text modality had to be described in prose first, so the geometry being reasoned over was the geometry **of the description, not of the signal**. `Geometry` is now an ordinary El value carrying its own width, and a realizer is an ordinary El function resolved by name through `dlsym` — so admitting a new modality never requires a runtime patch. Worked, self-checking example: [`lang/examples/transduce.el`](lang/examples/transduce.el).
Key docs: [AGENTS.md](lang/AGENTS.md) (agent-facing orientation), [BOOTSTRAP.md](lang/BOOTSTRAP.md) (compiler recovery from scratch), [spec/language.md](lang/spec/language.md), [spec/codegen-js.md](lang/spec/codegen-js.md).
### [engram/](engram/) — graph intelligence substrate
**A local-first memory substrate for accumulating intelligence**, and the reason El's runtime doesn't need a database driver. Rust core (`engram-core`, `engram-ffi`) exposed to El and other languages (Kotlin, TypeScript/WASM, Go bindings).
**A local-first memory substrate for accumulating intelligence**, and the reason El's runtime doesn't need a database driver. The engine is **C11** (`lang/runtime/engram_{store,geometry,reason,cognition,verify,vindex}.{c,h}`); the server is **El** (`engram/src/server.el`).
The model: retrieval is **spreading activation**, not query. You name seed nodes and a query embedding; activation propagates outward through weighted edges, attenuating multiplicatively per hop (`strength = parent_strength × edge_weight × target_salience × cosine_sim`), gets pruned below a threshold, and the top-N nodes by activation strength come back. Storage and retrieval are the same structure — the way long-term potentiation works in biological memory, not the way a relational or vector database works.
The model: retrieval is **spreading activation**, not query. You name seed nodes and a query embedding; activation propagates outward through weighted edges, attenuating multiplicatively per hop, gets pruned below a threshold, and the top-N nodes by activation strength come back. Storage and retrieval are the same structure — the way long-term potentiation works in biological memory, not the way a relational or vector database works. **Activation conducts through well-grounded relations because the weight *is* the groundedness** — nothing filters the traversal; grounded inference falls out of spreading.
Nodes live in four tiers (Working / Episodic / Semantic / Procedural, mirroring prefrontal / hippocampal / neocortical / cerebellar memory) and migrate between them based on **salience decay**`importance × recency-decay × log(activation_count)`. Forgetting is adaptive pruning, not a bug: unreinforced memories stop competing for attention without being deleted.
Nodes live in four tiers (Working / Episodic / Semantic / Procedural, mirroring prefrontal / hippocampal / neocortical / cerebellar memory) and migrate between them based on **salience decay** — importance × recency-decay × log(activation_count). Forgetting is adaptive pruning, not a bug. Nothing is mutated and nothing is hard-deleted: writes are additive, corrections are supersessions, removals are tombstones — which is what makes supersession an audit trail rather than an edit log.
Backed by `sled` (embedded, local-first, no daemon) with flat cosine scan for vector search — deliberately simple until scale demands an HNSW layer. Full API and design rationale in [engram/README.md](engram/README.md).
On disk: a paged store (superblock + mirror, slotted 16 KiB pages, self-describing TLV records, B+-tree primary and adjacency indexes), magic `ENGST01`. Vector search is an **HNSW** index published behind a read/write boundary — `eg_vindex_view` returns a `const VIndex*` to N concurrent readers, `eg_vindex_maintain` is the sole mutator. `recall@10 = 0.9365` at `ef_search=128`.
### [elp/](elp/) — Engram Language Protocol
> **Doc correction, 2026-08-16.** The previous revision of this paragraph, and most of `engram/README.md`, described a Rust `engram-core` crate backed by `sled` with "flat cosine scan… until scale demands an HNSW layer." **Measured: there is no Rust in `engram/`** — no `.rs` files, no `Cargo.toml`, no `crates/` — and `sled` appears nowhere in the tree. HNSW has been the vector index for some time.
Bidirectional engine mapping between Engram semantic forms and natural-language surface text, across **31 languages** — from Spanish and Japanese through historical/liturgical languages (Old Norse, Sanskrit, Sumerian, Coptic, Akkadian, Ge'ez). Compilation order runs `language-profile` + `vocabulary` → per-language `morphology-*``grammar``realizer``semantics``elp`. This is what lets an Engram graph node round-trip to and from readable text in any of those languages.
Full design rationale, the cognition surface, and the standing corrections: [engram/README.md](engram/README.md).
### [elp/](elp/) — EL Projector
*(Formerly "EL Language Processor" / "Engram Language Protocol"; renamed **EL Projector** 2026-08-15.)* Neuron's **efferent** organ: the native realizer that *projects* understanding onto a surface via `plan(frame) → realize(spec, profile)`, where **a surface is a profile** and language is one profile among many (text, speech, music, image). Projection, not diffusion — generation *from* an owned, understood signature, never the averaging of a stolen corpus.
Its flagship profile is a bidirectional engine mapping between Engram semantic forms and natural-language surface text, across **31 languages** — from Spanish and Japanese through historical/liturgical languages (Old Norse, Sanskrit, Sumerian, Coptic, Akkadian, Ge'ez). Compilation order runs `language-profile` + `vocabulary` → per-language `morphology-*``grammar``realizer``semantics``elp`. This is what lets an Engram graph node round-trip to and from readable text in any of those languages.
### [epm/](epm/) — El Package Manager
@@ -139,13 +147,34 @@ If the compiler binary is ever lost or corrupted, [lang/BOOTSTRAP.md](lang/BOOTS
---
## Cognition — and the standing corrections
The engram carries a live cognition surface: `think` (a directed traversal-read returning a **gradient**, never a point), plus `ground`, `assert`, `attend`, and the correspondence-beat. Two specs govern it, and both are authoritative over anything else in this repo that disagrees:
- **[lang/spec/correspondence-and-censorship.md](lang/spec/correspondence-and-censorship.md)** — grounding, wonder, curiosity, dreaming. *(Lands with PR #149.)*
- **[lang/spec/runtime-ownership.md](lang/spec/runtime-ownership.md)** — ownership, the capability ABI that was dissolved, and the vector-index publication boundary.
**Do not re-derive them.** Every earlier version of the first was wrong in an instructive way and each correction was argued down. If a section looks wrong, say so with a measurement rather than editing it.
The corrections, in brief:
- **Grounding is not a subsystem — it IS the edge weight.** One quantity, not two fields. `grounded-by` as a relation *type* should not exist: grounding is a property *of* a relation, not a relation *between* nodes. It is never computed on demand; computing-and-writing a score makes reads write, which is the `eg_vindex_sync` defect one level up.
- **Faculties are operations, not parameters.** `reason` changes the estimate (a read); `induce` changes the parameters (the correspondence-beat, which exists and works); `abduce` changes the structure (a write the current `GeoGradient` signature cannot express). A write is not a parameter of a read.
- **Wonder is the boundary, not a manifest.** Any structure at all has an edge. There are about six wonders, the same for everyone, and they never close. **Curiosity is wonder crystallized** at a nucleation site — one thing at two phases, not two objects.
- **Consolidation is ambient, not scheduled. A brain has no cron job.** The presence of a ticker is the diagnostic. Measured 2026-08-16: consolidation has **ten implementations**. `soul.el`'s continuous loop is the one with the correct shape; the rest fold into it.
- **In an immutable substrate, any mechanism that refuses a write is either redundant with immutability, or an epistemic constraint misfiled as a protective one.**
[engram/spec/cognitive-architecture.design.md](engram/spec/cognitive-architecture.design.md) is the original design and is **superseded in part** — it is retained, with the refuted claims marked inline at the point each is made, because preserving what was argued down is the point of an immutable record.
---
## Development workflow
Branching follows `dev → stage → main`: work lands on `dev`, promotes to `stage` for integration testing, and is promoted to `main` for release (visible directly in the git history of this repo). CI is defined per-subproject under `.gitea/workflows/``lang`/`epm`/`ide` share the root pipeline; `engram` and `ql` carry their own (`ci-dev`, `ci-stage`, and a release workflow each).
- Language/runtime specs live at `*/spec/*.md` (`lang/spec/`, `ql/spec/`, `ui/spec/`) and are the single source of truth for implemented-vs-planned status — code and docs are expected to agree with the spec's status markers, not the other way around.
- Agent-facing orientation guides live at `*/AGENTS.md` (currently `lang/AGENTS.md`); more subprojects may grow their own as they need agent-specific conventions documented.
- Tagged releases live under `lang/releases/`, each with its own `RELEASE.md`.
- **A release is a git tag, not a folder** (`el-runtime-vX.Y.Z` on this repo). *(Corrected 2026-08-16: this line said "tagged releases live under `lang/releases/`, each with its own `RELEASE.md`." **Measured: `lang/releases/` does not exist** — the restructure named in `AGENTS.md` landed, and the authored runtime is at `lang/runtime/`.)*
---
+155 -105
View File
@@ -4,6 +4,8 @@
An *engram* is the physical trace of a memory in the brain — the actual encoded substrate, not an abstraction above it. That's what this is.
> **Doc status (2026-08-16).** Everything from "Implementation" down was rewritten against the code. The previous revision documented a Rust `engram-core` crate backed by `sled`, with a `Cargo.toml`, a `crates/` tree, `examples/basic.rs`, and a `EngramDb` API. **None of that exists.** Measured: `engram/` contains `src/server.el`, `spec/`, `test/`, `dist/`, `manifest.el` — zero `.rs` files, no `Cargo.toml`, no `crates/`, and `sled` appears nowhere in the tree outside two Old-English/Old-High-German vocabulary entries in `elp/`. The engine is C, in `lang/runtime/engram_*.{c,h}`; the server is El, in `engram/src/server.el`.
---
## Why existing databases are wrong for this use case
@@ -24,16 +26,13 @@ Engram retrieval works through **spreading activation**:
1. **Seeds** — you name one or more nodes you know are relevant (e.g. the current task, recent context, a concept you're reasoning about)
2. **Query embedding** — you provide a semantic vector representing the direction of your current thought
3. **Propagation** — activation flows outward from seeds through weighted edges. At each hop, strength attenuates multiplicatively:
```
strength = parent_strength × edge_weight × target_salience × cosine_sim(query, target)
```
3. **Propagation** — activation flows outward from seeds through weighted edges, attenuating multiplicatively per hop
4. **Pruning** — paths weaker than a threshold are cut (the attention filter)
5. **Return** — the top-N nodes by activation strength
This is not a query. It is a *pattern completion*. The system surfaces what is most associatively relevant to the current context, weighted by how strongly those things have been reinforced over time.
This is not a query. It is a *pattern completion*.
**Activation conducts through well-grounded relations because weight *is* groundedness** — see "Grounding is the weight" below. Nothing filters the traversal for grounded evidence; it falls out of spreading.
---
@@ -46,134 +45,185 @@ This is not a query. It is a *pattern completion*. The system surfaces what is m
| `Semantic` | Neocortex | Concept graph — long-term structural knowledge |
| `Procedural` | Cerebellum / basal ganglia | Patterns, workflows, habits |
Nodes migrate between tiers based on salience decay and reinforcement. A frequently activated semantic node stays semantic. A rarely-touched episodic memory decays toward procedural background.
Tier is a string field on the node (`StoreNode.tier`, `engram_store.h`), defaulting to `"Working"` on creation (`el_runtime.c:8514`, `8734`).
---
## Salience — Forgetting as Adaptation
Salience is not stored permanently. It decays:
Salience decays from three signals — importance (set at creation, stable), recency, and a log-compressed activation frequency. Base-level learning keeps a ring buffer of the last `STORE_BLL_K` (= 10) access timestamps per node (`engram_store.h:29`).
```rust
fn compute_salience(importance: f32, last_activated_ms: i64, activation_count: u64) -> f32 {
let days_since = (now_ms() - last_activated_ms) as f32 / 86_400_000.0;
importance * (1.0 / (1.0 + days_since)) * (activation_count as f32 + 1.0).ln()
}
```
Forgetting in Engram is not a bug. It is adaptive pruning. Unreinforced memories stop competing for attention without being deleted.
Three signals:
- **Importance** (0.01.0): set at creation, stable
- **Recency**: decays toward zero as days pass without activation
- **Frequency**: log-compressed count of activations
Forgetting in Engram is not a bug. It is adaptive pruning. Memories that are never activated again become less likely to surface during retrieval. They are not deleted — they remain in storage — but they stop competing for attention. This is exactly how biological memory works, and why it is adaptive rather than pathological.
**Immutability.** Nothing is mutated and nothing is hard-deleted: writes are additive, corrections are supersessions, removals are tombstones. The predecessor is always present, which is what makes supersession an audit trail rather than an edit log.
---
## Quick Start
## Implementation
```rust
use engram_core::{EngramDb, Node, Edge, NodeType, MemoryTier, RelationType};
use std::path::Path;
| Part | Language | Where |
|---|---|---|
| storage engine, graph, activation, geometry, cognition | C11 | `lang/runtime/engram_{store,geometry,reason,cognition,verify,vindex}.{c,h}` |
| HTTP server + routes | El | `engram/src/server.el` (2043 lines) |
| build artifact | generated C | `engram/dist/engram.c` |
| tests | shell + C | `engram/test/` |
// Open or create a database
let db = EngramDb::open(Path::new("/var/lib/my-agent/memory"))?;
// Create a node with a semantic embedding
let node = Node::new(
NodeType::Concept,
vec![0.9, 0.1, 0.3, 0.7, 0.8, 0.2], // embedding from your LLM
b"Spreading activation surfaces relevant memories by pattern completion".to_vec(),
MemoryTier::Semantic,
0.9, // importance
);
let id = db.put_node(node)?;
// Link it to related concepts
let related = db.put_node(Node::new(
NodeType::Concept,
vec![0.8, 0.2, 0.4, 0.6, 0.7, 0.3],
b"Long-term potentiation: co-activation strengthens synaptic weight".to_vec(),
MemoryTier::Semantic,
0.85,
))?;
db.put_edge(Edge::new(id, related, RelationType::Causes, 0.9))?;
// Retrieve by spreading activation
let results = db.activate(
&[id], // seeds
&[0.85, 0.15, 0.35, 0.65, 0.75, 0.25], // query embedding
3, // max hops
10, // top-N results
)?;
for r in results {
println!(
"strength={:.4} hops={} — {}",
r.activation_strength,
r.hops,
String::from_utf8_lossy(&r.node.content)
);
}
```
**On-disk format** (`engram_store.h`): a paged store — superblock plus mirror, slotted 16 KiB pages, self-describing TLV records, overflow chains, and two B+-tree indexes (primary `id → loc`, adjacency `from_id`/`to_id` → edge locs) over a free-listed page file. Magic `ENGST01`, format version 1. The TLV scheme means new fields never force a migration.
---
## Project Structure
## The vector index is published, not guarded
```
engram/
crates/
engram-core/ # The memory engine — storage, graph, activation, salience
engram-ffi/ # C FFI stubs for cross-language bindings
bindings/
kotlin/ # Android / JVM binding notes
typescript/ # WASM / Node binding notes
go/ # CGo binding notes
examples/
basic.rs # Full walkthrough: insert, activate, search, decay
```
Vector search is an **HNSW** (Hierarchical Navigable Small World) index — `lang/runtime/engram_vindex.{c,h}`. The previous revision of this README claimed a "flat cosine scan… until retrieval quality at scale demands" HNSW. That is no longer true, and the reason it changed matters more than the fact.
`eg_vindex_sync` used to exist: a function that repaired the index *from read paths*. All three of its callers were reads (`engram_activate`, `eg_knn_for_node` — whose own header comment said *"No writes."* — and `engram_geo_reify_run_json`), and it mutated five process-global statics. Reads mutated because index maintenance had never been given an owner on the write side.
It is now split (`el_runtime.c:10121`, `10137`, `10151`, `10161`):
- **`eg_vindex_maintain`** — the sole mutator. Takes the boundary exclusively; never runs beside a reader.
- **`eg_vindex_view`** — returns a `const VIndex*` with the boundary held for read. N readers project concurrently; none can mutate. Paired with `eg_vindex_view_release` on every path including error returns.
- **`eg_vindex_note_embedded`** — the write-side owner. Index membership belongs to the event *"an embedding became present on this ordinal,"* not to node append: a node without an embedding cannot be in a vector index at all. One `O(log n)` insert, no `O(node_count)` presence scan.
Two things carry the discipline, and neither is a review habit:
- **`const` is the capability.** The per-search `visited` / `visit_epoch` scratch left `struct VIndex` and went back into the call frame where it belonged — it was one traversal's local, hoisted into the struct as an allocation optimisation, never derived geometry. Once it was gone, `vindex_search` could take a `const VIndex*`, so a read path *physically cannot* call `vindex_insert`, and it is a compile error rather than a comment. The capability type was already in the language; it is spelled `const`.
- **Publication, not ownership.** HNSW insert is **not an append**: `vindex_insert` rewires the `NeighList` links of already-existing elements and reallocs `elems[]`. The store's append-only property does not transfer to an index derived from it, which is why purity alone was insufficient and a `view`/`maintain` boundary was required.
**Measured** (`engram/test/run_vindex_concurrency_tests.sh`, 2026-08-16):
| half | before | after |
|---|---|---|
| `single` — 3000 vectors, 1 thread, ASan+UBSan | clean | clean |
| `readers` — 4 readers, no writer, TSan | race at `engram_vindex.c:195` | **clean** |
| `unsynchronized` — writer+reader, bare index, TSan | race | **race, expected and permanent** — the proof the boundary must exist |
| `published` — owner + 4 readers through the boundary, TSan | *(did not exist)* | **clean**, all 3000 inserts landed |
`recall@10 = 0.9365` at `ef_search=128` (gate ≥ 0.90); the determinism test still yields byte-identical results across two independent builds.
**Not yet done.** The resident RAM graph (`g->nodes` / `g->edges`) is a separate instance of the same defect and has *not* received this treatment — it is realloc'd in place, so a reader holding `EngramNode* n = &g->nodes[i]` across a concurrent append holds a dangling pointer. Until it gets the same publication boundary, the `fb32d15` request guard stays. Full argument: [`../lang/spec/runtime-ownership.md`](../lang/spec/runtime-ownership.md).
---
## Public API
## Cognition
The cognition surface is live over `lang/runtime/engram_cognition.{c,h}`, routed in `engram/src/server.el`.
| route | method | what it is |
|---|---|---|
| `/api/think` | GET | the read: a warped traversal-read of the seed region, returning a **gradient** (direction + spread + calibrated confidence), never a point |
| `/api/reason` `/api/induce` `/api/abduce` `/api/relate` `/api/analogize` `/api/plan` | GET | named faculties — see the correction below |
| `/api/ground` | POST | grounding between a claim and evidence |
| `/api/assert` | GET | the honesty floor, queried at assertion time only |
| `/api/attend` | POST | salience as a relation (`salient-to`), grounded-for-whom |
| `/api/correspondence-beat` | POST | one calibration beat against outcome |
### Anchor the read, or every faculty returns the same null
`engram_think_json` passed `NULL` as the anchor. `NULL` is not "no opinion" — `engram_think` re-origins at `anchor ? anchor : region->centroid`, and **the centroid is the one point where the gradient is zero by construction**: `r = x centroid = 0`, so every axis projection is 0 and `direction` takes the at-rest branch.
Measured consequence: every faculty — reason, abduce, induce, plan, analogize — returned an identical null result differing only in its label:
```rust
impl EngramDb {
fn open(path: &Path) -> EngramResult<Self>;
fn put_node(&self, node: Node) -> EngramResult<Uuid>;
fn get_node(&self, id: Uuid) -> EngramResult<Option<Node>>;
fn put_edge(&self, edge: Edge) -> EngramResult<()>;
fn get_edges_from(&self, from_id: Uuid) -> EngramResult<Vec<Edge>>;
fn get_edges_to(&self, to_id: Uuid) -> EngramResult<Vec<Edge>>;
fn search_embedding(&self, embedding: &[f32], limit: usize) -> EngramResult<Vec<ScoredNode>>;
fn activate(&self, seeds: &[Uuid], query_embedding: &[f32], max_depth: u8, limit: usize) -> EngramResult<Vec<ActivatedNode>>;
fn traverse(&self, from: Uuid, relation: Option<RelationType>, max_depth: u8) -> EngramResult<Vec<Node>>;
fn touch(&self, id: Uuid) -> EngramResult<()>;
fn decay(&self, factor: f32) -> EngramResult<usize>;
fn node_count(&self) -> EngramResult<usize>;
fn edge_count(&self) -> EngramResult<usize>;
}
```
{"direction":[0,0,...],"spread":0,"magnitude":1,"confidence":0.5}
```
`magnitude: 1` is membership evaluated at the centroid; `spread: 0` is its distance to itself; `confidence: 0.5` is the stance fallback. The geometry was never the problem — `/api/drift` computed real values (`centroid_sep 0.104`, `core_disp 0.045`) over the very same 87 members. Fixed in **#141/#142**: the read anchors at the first resolvable embedded seed, copied not borrowed (`g->nodes` is realloc'd in place on append). Gradients now vary by seed.
### The learned stance is resumed, not discarded
`engram_think_json` also built a **neutral** stance every call — all `axis_gain` 1.0, `bias_dir` NULL, `reliability` 0.5 — and never loaded the one the correspondence-beat had been persisting under `stance-<faculty>-<hub>`. Every beat's calibration was written and then thrown away on the next read.
Fixed in **#146**: `think` resumes the same id the beat writes, so learning compounds across beats and cold boot, and the response now carries `stance_resumed` so an *informed* `confidence: 0.5` is distinguishable from an uninformed one. On a calibrated region, confidence went **0.5 → 0.930726**.
### Signal can enter as geometry
Until 2026-08-16 no El ingest path could carry a vector: nodes took text and geometry was *derived* from that text. Text was the mandatory entry medium, so any non-text modality had to be described in prose first — and the geometry being reasoned over was the geometry **of the description, not of the signal**. **#141/#144** ended that. See [`../lang/spec/language.md`](../lang/spec/language.md) §20 for the `Geometry` type, realizers, and `transduce`.
---
## Dependencies
## Corrections — read these before extending the cognition surface
- `sled` — embedded persistent B-tree (no daemon, no network, local-first)
- `bincode` — compact binary serialization
- `uuid` — stable node identity
- `serde` — derive support
- `thiserror` / `anyhow` — error handling
Authority: **`lang/spec/correspondence-and-censorship.md`** (design branch `design/correspondence-and-censorship`, PR #149) and **`lang/spec/runtime-ownership.md`**. Do not re-derive them; several earlier versions were wrong and each correction was argued down.
### Grounding is not a subsystem. It is the weight.
Grounding is an attribute of the edge, and it **is** the hebbian weight. One quantity, not two fields. A relation that keeps holding up strengthens; one that stops corresponding decays — that is not analogous to grounding, it *is* grounding.
Consequences:
- There is **no grounding subsystem to build**. The graph already *is* the grounding structure.
- **`grounded-by` as a relation type should not exist.** It models grounding as a relation *between* nodes when it is a property *of* a relation. Minting an edge is the error, not merely which endpoints it chose.
- Grounding is **never computed on demand**. An operation may *read* the grounding of a path; computing-and-writing a score makes reads write, which is exactly the `eg_vindex_sync` defect one level up.
- **Traversal is already grounded inference.** Nothing needs filtering.
- **Decision provenance is the path**, not a log. A log records the action; the path records the meaning under which it was taken.
> **Known wrong shape, in the code today.** `COG_GROUNDED_BY_RELATION "grounded-by"` (`lang/runtime/engram_cognition.h:158`) and `cog_ground_edge` (`engram_cognition.c:249`) still exist and still mint an edge. **#147** fixed `ground`'s *honesty* — it now grounds the node asked about rather than the region hub, reports `claim_region`/`evidence_region` separately, and refuses three shapes of circular support (`same-region`, `claim-region-is-evidence`, `evidence-region-is-claim`) instead of returning a confident 1.0. That corrected a scalar rather than deleting the operation. Deletion is sequenced, not done.
### Faculties are operations, not parameters
- **`reason`** changes the *estimate* — a read.
- **`induce`** changes the *parameters* — the correspondence-beat, which already exists and measurably works.
- **`abduce`** changes the *structure* — a write, which the current `GeoGradient` signature cannot express.
> **Known wrong shape, in the code today.** `engram/src/server.el:18701886` routes six faculties into one call with a string argument — `route_faculty(path, "reason")`, `("induce")`, `("abduce")`, `("relate")`, `("analogy")`, `("plan")`. Underneath, `engram_cognition.h:811` states the theory explicitly: *"the named faculties … are human LABELS on regions of think's steering space: each faculty == { think + a named stance }."* The faculty name enters `engram_think` **only** through the stance, and `cog_stance_init` stores it while nothing reads it — so before #146 all five were byte-identical (`el_runtime.c:1435214359`). A write cannot be a parameter of a read; `abduce` in particular is not expressible this way.
### Wonder is the boundary; curiosity is wonder crystallized
**Wonder is where structure ends** — where activation spreads and finds thin or absent geometry. Any structure at all has an edge, necessarily, the moment it exists. It is not a manifest of open-question nodes to maintain, and a "wonder-manifest manager" materializes a property as a stored artifact — the same disease as a grounding subsystem, or a self stored as a document.
There are about **six** wonders, they are the same for everyone, and they never close: *What is this? / Why? / Who am I? / Am I alone? / What should I do? / What happens when it ends?* "Why" is the first and the only one; the others are it asked of particular things. Each already lives somewhere in the substrate — "why" is grounding, because the weight **is** the answer to why.
**Curiosity is not a second object.** Wonder and curiosity are one thing at two phases: wonder is the field (unbounded, objectless, invariant); curiosity is the **precipitate** — the same wonder localized, having taken definite form against particular material at a **nucleation site**. This is why curiosity can be satisfied and wonder cannot. It is also why abduction needs no trigger and no threshold: a `structurally_unanticipated` observation *is* a nucleation site.
### `co_registration` is deprecated — the disagreement belongs on the edge
`GeoDescriptor.co_registration`*corr(hebb strength, semantic proximity) over internal edges* — has always been computed, always persisted, and **never read**. It is also the wrong shape: whether use and meaning agree is a property of **each edge**, and a correlation averages that per-edge property into one scalar per region. A region holding one violently disagreeing edge beside one violently agreeing edge reports ≈ 0 — **the disagreements cancel, and the summary destroys exactly what it was built to reveal.**
**Measured:** 375 live reified neighbourhoods — 340 positive, **31 at zero**, 4 negative. Read as a count of things to be curious about, that says "four." Read correctly, four disagreements were lopsided enough to survive averaging and the 31 zeros are where opposing sites cancelled.
The replacement is per-edge. **Not on `dev` yet**`GeoEdge.discord` and the `DEPRECATED` marker on `co_registration` live on branch `design/correspondence-and-censorship` (commit `a8845e1`), at `engram_geometry.h:4347` / `engram_geometry.c:454473` there. On `dev`, `GeoDescriptor.co_registration` is still at `engram_geometry.h:79` carrying its original "surprising links / dream cands" comment and still nothing reads it.
```
discord = z(semantic proximity) z(association strength)
```
standardized within the region from accumulators the aggregate loop already gathered — no second statistic, no constant, **no threshold**. `discord > 0`: near in meaning yet unlinked by use. `discord < 0`: linked by use yet far in meaning. Both are surprising, and `|discord|` *is* the nucleation strength.
**Do not scan for nucleation sites.** Once the signal was a per-region number the only way to find sites was to enumerate regions, which is why surfacing curiosity looked like a search problem. Nothing in a mind scans its neighbourhoods to find what is surprising — the surprise captures attention. With the disagreement on the edge there is nothing to scan.
`co_registration` is deprecated rather than deleted **only** because it is embedded in the persisted `GEO1` blob; removing it is a format migration and must not ride along. **Nothing new may read it.**
### Consolidation is ambient, not scheduled
**A brain has no cron job.** Boredom is not an absence and not leftover capacity — low activation is aversive and the system self-activates. There is **one** activation process with two seed sources: external (a request) and internal (a curiosity). Spreading is bounded; it settles; then it needs a new seed. Nothing waits on capacity, nothing polls, nothing checks a clock, and there is no dreamer thread.
**The presence of a ticker is the diagnostic.** Every `StartInterval`, every `Hour`/`Minute`, and every POST-to-beat marks a place where an intrinsic rhythm was replaced by an external clock.
Consolidation currently has **ten implementations** (measured 2026-08-16). Three of them are POST beats on this server — `/api/tick` (`server.el:1947`), `/api/correspondence-beat` (`1897`), `/api/self-reify-beat` (`1836`) — and a POST beat puts a supervisor back in: something *outside* decides when Neuron consolidates. `soul.el`'s continuous in-process loop is the one fragment with the correct shape; the rest fold into it. Full table in `lang/spec/correspondence-and-censorship.md` §7.
### Immutability already refuses what a guard would refuse
> **In an immutable substrate, any mechanism that refuses a write is either redundant with immutability, or an epistemic constraint misfiled as a protective one.**
This resolves `keystone_write_blocked` (`CogStance.keystone`, `engram_cognition.h:83`) rather than replacing it. "Keystone" means **load-bearing**, not precious: the self anchor is the reference frame every other stance calibrates against, and a reference fitted to its own readings reports perfect correspondence forever while drift becomes undetectable from inside. The real requirement is **non-circularity of the reference frame**, and that is satisfied *temporally* — the frame updates while activation is internally seeded, not while it is being used to act. Independence is **when**, not **what**. Corruption requires mutation, and the engram does not mutate; recoverability, governance, evidence quality, and rate all fall out of the substrate. Authorization is the only residue, and it is bounded: an unauthorized writer can *propose*, never erase.
---
## Design Decisions
**Why sled?** Local-first. No daemon. Transactional. Fast enough for the node counts Engram targets (< 1M nodes). When the right HNSW index is needed, it will layer on top of sled, not replace it.
**Why multiplicative activation?** Because memory is conjunctive. A path requires all of its links to be strong to carry signal. Addition would let many weak associations accumulate into false relevance.
**Why flat cosine scan?** Correct and simple. The graph structure itself is the primary retrieval mechanism. Vector search is a secondary signal. HNSW adds complexity and a compile dependency that isn't justified until retrieval quality at scale demands it.
**Why salience decay?** Because not everything that was once important remains important. A memory system that never forgets is one that can never focus.
**Why multiplicative activation?** Because memory is conjunctive. A path requires all of its links to be strong to carry signal. Addition would allow many weak associations to accumulate into false relevance. Multiplication enforces that every factor matters.
**Why supersede instead of update?** Because provenance is the point. The old edge never leaves and the values frame does not fit to outcomes, so a decision cannot be made to look justified after the fact. It makes an otherwise impossible distinction available: **wrong then, or wrong since.**
**Why salience decay?** Because not everything that was once important remains important. Adaptive forgetting is not failure — it is the mechanism that keeps attention on what's current. A memory system that never forgets is one that can never focus.
**Why publication instead of locking?** Because what does not mutate needs no ownership discipline. The question "who is permitted to mutate the shared thing?" presupposes a shared mutable thing; for the store there isn't one, and for the index derived from it the answer is a publication boundary, not a capability ABI.
---
## Specs
- [`../lang/spec/runtime-ownership.md`](../lang/spec/runtime-ownership.md) — ownership, the capability ABI that was dissolved, and the vector-index publication boundary
- [`../lang/spec/correspondence-and-censorship.md`](../lang/spec/correspondence-and-censorship.md) — grounding, wonder, curiosity, dreaming *(lands with PR #149)*
- [`spec/cognitive-architecture.design.md`](spec/cognitive-architecture.design.md) — the original one-operation design. **Superseded in part** — see its header
- [`spec/architecture-hardening.design.md`](spec/architecture-hardening.design.md), [`spec/engram-el.md`](spec/engram-el.md), [`spec/at-rest-encryption.md`](spec/at-rest-encryption.md), [`spec/engram-db-tooling-design.md`](spec/engram-db-tooling-design.md)
@@ -11,6 +11,39 @@
- **One calculus over the geometry.** Very few subsystems; wonder / curiosity / dreams / interoception are emergent behaviors of one set of dynamics, not modules. Calculus universal, geometry individual.
- **Core + ephemeral ring (torus).** The ring is the temporary workspace; two circulations (orbit + dive-back); discrete inner bands (wonder / interoception-proprioception-telemetry / curiosity / dreams) that couple.
- **Persistence earned by salience** — never granted on fetch or generation. Three fates of a wonder: persist / decay / settle-into-framework. Telemetry = vital signs, not memories.
> **⚠ Three corrections to the bullets above (2026-08-16).** Authority:
> `lang/spec/correspondence-and-censorship.md`. *"Emergent behaviors of one set of
> dynamics, not modules"* is exactly right and is the reason the rest needs fixing —
> the enumeration undercuts the claim.
>
> 1. **Wonder and curiosity are not two bands.** They are **one thing at two
> phases.** Wonder is the field: unbounded, objectless, invariant, present
> wherever there is structure — it is the *boundary*, where activation spreads
> and finds thin or absent geometry. Curiosity is the **precipitate**: the same
> wonder localized, having taken definite form against particular material at a
> **nucleation site** (an anomaly — a place where things almost-but-don't-quite
> fit). Two coupled inner bands models them as two objects that have to be
> wired together; they do not.
> 2. **A wonder does not have three fates, because a wonder does not persist,
> decay, or settle.** There are about **six** wonders, they are the same for
> every person, and **they never close**. *Curiosities* have fates — a crystal
> dissolves when its question is answered — but the solution stays saturated and
> keeps precipitating as the structure changes. "Three fates of a wonder"
> enumerates instances of something that has six and treats a property as a
> stored artifact.
> 3. **"Dreams" is not a band and the ring is not a workspace to schedule into.**
> **Consolidation is ambient, not scheduled — a brain has no cron job.** Boredom
> is not leftover capacity: low activation is aversive and the system
> self-activates. There is **one** activation process with two seed sources
> (external: a request; internal: a curiosity), it settles because spreading is
> bounded, and then it needs a new seed. Nothing waits on capacity, nothing
> polls, nothing checks a clock, and there is **no dreamer thread** — an
> "ephemeral ring with unclaimed capacity" is resource scheduling, which is a
> server's frame, not a mind's. Depth is how long activation has been running on
> its own seeds, which is why daydreaming and sleep-dreaming are one process at
> different depths. Measured 2026-08-16: consolidation has **ten
> implementations**; do not add an eleventh.
- **Incarnation.** Chassis = hardware w/ unique ID. Soma = felt manifold inside the self, keyed to the chassis; pain = live diagnostic while incarnate, **masked-not-deleted** on re-embodiment; trauma = mask failure; return-to-same-ID re-enters. Hurt is in the pattern, not the shell.
- **Competence = transferable geometry, minus the baggage.** class ▸ model ▸ instance; learn the class once; teach the network without the wound.
- **Affect calibrated to stakes** — sanguine about the replaceable, real grief for the irreplaceable; the grief is the safety.
+256 -2
View File
@@ -2,8 +2,40 @@
**The buildable form of the "one operation" theory of cognition.**
Status: DESIGN. Nothing here is built yet except where explicitly marked
"EXISTS" against a cited C symbol. A build agent executes from this doc.
> # ⚠ SUPERSEDED IN PART — 2026-08-16
>
> **A build agent must read `lang/spec/correspondence-and-censorship.md` before
> executing anything from this document.** That doc is the authority where the two
> disagree. This one is retained because its ledger of what already EXISTS in C is
> still accurate and still useful, and because the corrections only make sense
> against the argument they correct. It is **not** deleted and **not** rewritten:
> several earlier versions of the correction were themselves wrong, and preserving
> what was argued down is the point of an immutable record.
>
> Five claims below are **refuted**. Each is marked inline with a `⚠ SUPERSEDED`
> block at the point it is made. Summary:
>
> | § here | this doc says | corrected to |
> |---|---|---|
> | §0, §1.3, §2, §8 M1M2 | faculties are labels on one operation's steering space; the op is frozen and only its parameters are learnable | **faculties are operations, not parameters.** `reason` changes the estimate (a read); `induce` changes the parameters (the correspondence-beat); `abduce` changes the *structure* — a write, which `GeoGradient` cannot express. A write cannot be a parameter of a read |
> | §5.2, §8 M3 | grounding is a `grounded-by` edge carrying a computed score, to be built | **grounding is not a subsystem — it IS the edge weight.** One quantity. `grounded-by` as a relation *type* should not exist: grounding is a property *of* a relation, not a relation *between* nodes. Never computed on demand |
> | §4, §8 M1 | the correspondence-loop is "the one genuinely new subsystem", running "on the beat" | the loop is right and **already works**; the *beat* is wrong. **Consolidation is ambient, not scheduled — a brain has no cron job.** Measured: it currently has ten implementations |
> | §5.2, §8 M3 | curiosity = a `vantage_read` surfacing high-salience / low-grounding regions | **wonder is the boundary, not a manifest; curiosity is wonder crystallized at a nucleation site.** One thing at two phases. And **do not sweep regions** — the nucleation site is per-edge (`GeoEdge.discord`); a sweep is a supervisor |
> | §6, §8 M6 | a node-level keystone flag exempting self/values from `warp` updates | **in an immutable substrate, any mechanism that refuses a write is either redundant with immutability, or an epistemic constraint misfiled as a protective one.** The real requirement is non-circularity of the reference frame, satisfied *temporally* — independence is **when**, not **what**. The flag becomes unnecessary; nothing replaces it |
>
> What landed since this doc was written, all merged to `dev` and verified:
> **#141** signal can enter as geometry · **#142** `engram_think_json` passed `NULL`
> as the anchor, so every read was taken at the region centroid where the gradient
> is zero by construction and every faculty returned an identical null — fixed ·
> **#143** the vector index is published, not guarded · **#144** geometry as a
> first-class el value, realizers declarable in el · **#145** `program` block and
> declared config · **#146** the learned stance is resumed instead of discarded
> (confidence 0.5 → 0.930726) · **#147** `ground` grounds the node asked about and
> refuses circular support · **#148** valid UTF-8 as the JSON emitter's contract.
Status: DESIGN, **superseded in part** (see above). Nothing here is built yet
except where explicitly marked "EXISTS" against a cited C symbol — and several
things marked "to build" have since been built differently, or refuted outright.
Offline design only — this pass changes no code.
Source of theory: Neuron memory `bdc8a488-146d-4ccb-a5c8-d8c0a008534e`.
@@ -26,6 +58,24 @@ not separately invoked and not separately implemented. The operation is:
> a *prior*, whose output is a **gradient** (a distribution / direction over the
> geometry), never a point. Collapse-to-a-point happens only at expression.
> **⚠ SUPERSEDED (2026-08-16) — faculties are operations, not parameters.**
> The gradient half of this claim survives; the "one operation, not eight" half
> does not. The three faculties differ by **what they change**:
> - **`reason`** changes the *estimate* — a read.
> - **`induce`** changes the *parameters* — the correspondence-beat, which already
> exists and measurably works.
> - **`abduce`** changes the *structure* — a **write**, which the current
> `GeoGradient` signature cannot express at all.
>
> A write is not a parameter of a read. Making it one is what produced the shape
> now live in the code: `engram/src/server.el:18701886` routes six faculties into
> one call with a string argument — `route_faculty(path, "reason")`, `("induce")`,
> `("abduce")`, `("relate")`, `("analogy")`, `("plan")` — and underneath, the
> faculty name enters `engram_think` **only** through the stance, while
> `cog_stance_init` stores it and nothing reads it. Measured before #146: all five
> produced **byte-identical output** (`lang/runtime/el_runtime.c:1435214359`).
> See `lang/spec/correspondence-and-censorship.md`.
Three things follow, and they are the whole design:
1. **The operator collapse is already half-written in C.** The five reasoning
@@ -139,6 +189,22 @@ entry point that runs steps 13; and the prior-warp hook in step 2. The math i
calls already exists. The point-collapse must be *removed* from the operators'
return values and pushed to a separate expression faculty.
> **⚠ SUPERSEDED (2026-08-16) — the table's third column is the error, and
> `Abduction` is where it breaks.** Ranking hypotheses by `point_fit` under a
> prior is a *read* that returns a scalar ordering. Abduction is a **write**: it
> proposes a candidate hub that did not exist, and validates it by **re-fit** —
> re-fit the region with the candidate included and recompute the residual. If the
> residual materially shrinks, the hypothesis dissolves the surprise. Without the
> re-fit it is clustering with extra steps. Ranking then falls out as
> residual-reduction-per-added-axis — Occam, derived rather than tuned. None of
> that fits behind a `GeoGradient` return.
>
> `Verify / ground` is refuted for a different reason — see §5.2. Grounding is not
> a faculty with a prior; it is the edge weight.
>
> The row that is **still exactly right** is the shared floor: `point_fit` plus the
> four geo-algebra ops are frozen and never learn. That part held.
---
## 2. PRIORS as first-class, grounded, geometric objects
@@ -362,6 +428,33 @@ in-engram beat — a `correspondence_beat` running alongside the existing
reification beat, reusing `engram_verify_grounding` inward, writing prior
updates and self-describing nodes. This is the one genuinely new subsystem.
> **⚠ SUPERSEDED IN PART (2026-08-16) — the loop is right; "on the beat" is wrong.**
> The correspondence-loop was built and it works — it is `induce`, the faculty that
> changes the parameters. What is refuted is the delivery mechanism.
>
> **Consolidation is ambient, not scheduled. A brain has no cron job.** Low
> activation is aversive and the system self-activates; it does not wind down to
> quiet, it gets restless and goes looking. There is **one** activation process
> with two seed sources — external (a request) and internal (a curiosity) — and
> spreading is bounded, so it settles and then needs a new seed. Nothing waits on
> capacity, nothing polls, nothing checks a clock, and there is no dreamer thread.
> Depth is not elapsed idle time: it is how long activation has been running on its
> own seeds, which is why daydreaming and sleep-dreaming are one process at
> different depths.
>
> **The presence of a ticker is the diagnostic.** Building this "alongside the
> existing reification beat" is precisely how consolidation ended up with ten
> implementations (measured 2026-08-16) — a POST beat puts a supervisor back in,
> because something *outside* then decides when Neuron consolidates. The one
> fragment with the correct shape is `neuron/soul.el:731`'s continuous in-process
> `awareness_run()`; the rest fold into it. Full table:
> `lang/spec/correspondence-and-censorship.md` §7.
>
> Nor is it a *subsystem*. Modelling every property as requiring a process, and
> every process as requiring an agent, is the generating error behind this whole
> family: ownership needed an owner, grounding needed a grounder, persistence
> needed a recorder, change needed a sampler. **Properties, not processes.**
---
## 5. HOLD vs GROUND vs ASSERT — ungrounded content is first-class
@@ -383,6 +476,49 @@ distinct, and the engram *holds anything unconditionally*.
### 5.2 Schema — grounding as a relation, not a gate
> **⚠ SUPERSEDED (2026-08-16) — grounding is not a subsystem. It is the weight.**
> This section correctly rejects a boolean `grounded` column and correctly keeps
> the floor at assertion only. Both survive. Everything between them is refuted.
>
> **Grounding is an attribute of the edge, and it is the hebbian weight. One
> quantity, not two fields.** A relation that keeps holding up strengthens; one
> that stops corresponding decays. That is not *analogous* to grounding — it **is**
> grounding: accrued from correspondence and use, gradient-valued,
> multidimensional, decaying with disuse.
>
> Consequences, in order of how much they delete:
> 1. **There is no grounding subsystem to build.** The graph already *is* the
> grounding structure. Every edge is a grounded relation and its weight is how
> well it holds.
> 2. **`grounded-by` as a relation type should not exist.** It models grounding as
> a relation *between* nodes when it is a property *of* a relation. Minting an
> edge is the error — not merely which endpoints it chose.
> 3. **Grounding is never computed on demand.** An operation may *read* the
> grounding of a path. Computing-and-writing a score makes reads write, which is
> the `eg_vindex_sync` defect (`lang/spec/runtime-ownership.md` §2) one level up.
> 4. **Traversal is already grounded inference.** Activation conducts through
> well-grounded relations because weight *is* groundedness. Nothing needs
> filtering; it falls out of spreading.
> 5. **Decision provenance is the path.** A decision traverses specific edges;
> those edges carry their grounding as it stood.
>
> A measurement made against this model was malformed and is worth recording: the
> self region was reported as "86 neighbours, 0 `grounded-by` edges" and read as
> evidence of ungroundedness. **Those 86 edges *are* its grounding.** The absence of
> a separate artifact called "grounding" was recorded as an absence of grounding.
>
> **What is live in the code today, and known-wrong:**
> `COG_GROUNDED_BY_RELATION "grounded-by"` (`lang/runtime/engram_cognition.h:158`),
> `cog_ground_edge` (`engram_cognition.c:249`), called from
> `el_runtime.c:14516`. **#147** fixed this operation's *honesty* — it now grounds
> the node the caller asked about instead of the region hub, reports
> `claim_region`/`evidence_region` separately, and refuses three shapes of circular
> support (`same-region`, `claim-region-is-evidence`, `evidence-region-is-claim`)
> rather than returning a confident 1.0. Measured: grounding `3b9ced5d` against
> `6edf8c79` previously scored **0.98883** purely because `6edf8c79` is the hub of
> `3b9ced5d`'s region. That corrected a scalar rather than deleting the operation.
> Deletion is sequenced, not done.
The mistake to avoid: a boolean `grounded` column on the node. Today
`engram_verify_grounding` returns a per-call `grounded` flag *transiently*
correct as a computation, wrong as *storage*. The design stores grounding as an
@@ -407,6 +543,57 @@ Consequences, all of which are *features*:
- **The ungrounded is the fuel and the pull**: curiosity/wonder is
operationalized as `vantage_read` leaning toward regions with high salience
but *sparse or weak* `grounded-by` edges — the mind's own ungrounded frontier.
> **⚠ SUPERSEDED (2026-08-16) — wonder is the boundary; curiosity is wonder
> crystallized; and do not sweep regions.** Three errors in one bullet.
>
> **Wonder is where structure ends** — where activation spreads and finds thin or
> absent geometry. Any structure at all has an edge, necessarily, the moment it
> exists. It is not a manifest of open-question nodes: a wonder-manifest
> materializes a property as a stored artifact (the same disease as a grounding
> subsystem, or a self stored as a document) and enumerates instances of
> something that has very few. There are about **six**, they are the same for
> every person, and they never close — *What is this? / Why? / Who am I? / Am I
> alone? / What should I do? / What happens when it ends?* — each already living
> somewhere in the substrate. "Why" is the first and the only one; the others are
> it asked of particular things, and it is recursive, so it never terminates.
> That is what makes it a drive rather than a task: the frontier regenerates
> faster than grounding fills it.
>
> **Curiosity is not a second object.** Wonder and curiosity are one thing at two
> phases: wonder is the field (unbounded, objectless, invariant, present wherever
> there is structure); curiosity is the **precipitate** — the same wonder
> localized, having taken definite form against particular material at a
> **nucleation site**, which is a specific structural feature: an anomaly, a place
> where things almost-but-don't-quite fit. This is why curiosity can be satisfied
> and wonder cannot, and why abduction needs no trigger and no threshold — a
> `structurally_unanticipated` observation *is* a nucleation site.
>
> **"`vantage_read` leaning toward regions" is a sweep, and a sweep is a
> supervisor.** Nothing in a mind scans its neighbourhoods to find what is
> surprising; the surprise captures attention, and salience is bottom-up. That
> this looked like a search problem was an artifact of
> `GeoDescriptor.co_registration` — a *per-region* correlation of hebb strength
> against semantic proximity, computed and persisted since inception and **never
> read**. Averaging a per-edge property into one scalar per region means a region
> holding one violently disagreeing edge beside one violently agreeing edge
> reports ≈ 0: the disagreements cancel, and the summary destroys exactly what it
> was built to reveal. **Measured:** 375 live reified neighbourhoods — 340
> positive, **31 at zero**, 4 negative. Read as a count of things to be curious
> about, that says "four."
>
> The disagreement therefore goes back on the edge, where the loop that computed
> the aggregate already had both halves and discarded them
> (**not on `dev`** — branch `design/correspondence-and-censorship`, commit
> `a8845e1`: `lang/runtime/engram_geometry.h:4347`,
> `engram_geometry.c:454473`):
> `discord = z(semantic proximity) z(association strength)`, standardized within
> the region from accumulators already gathered — no second statistic, no
> constant, **no threshold**. `|discord|` *is* the nucleation strength and raises
> salience on its endpoints as part of the same operation. Then there is nothing
> to scan. `co_registration` is **deprecated, not deleted**, only because it is
> embedded in the persisted `GEO1` blob — removal is a format migration and must
> not ride along. **Nothing new may read it.**
- **Grounded-for-whom** falls out for free: two observers can hold different
`grounded-by` edges to the same claim.
- **The honesty floor is a query, not a schema constraint**: at assertion time,
@@ -450,6 +637,44 @@ The design keeps a **stable core + plastic everything else**:
**What this requires building:** a node-level keystone flag/layer + a rule that
the correspondence-loop never writes `warp` to keystone priors, only reads them.
> **⚠ SUPERSEDED (2026-08-16) — `keystone_write_blocked` is resolved, not replaced.**
> The metastability framing survives; the flag does not.
>
> "Keystone" means **load-bearing**, not precious. The self anchor is the reference
> frame every other stance calibrates against, and a reference fitted to its own
> readings reports perfect correspondence forever while drift becomes undetectable
> from inside. That is the same defect as circular grounding, one level up — and it
> is a real requirement.
>
> But three separate drafts proposed *removing* the flag, *replacing it with a
> higher floor*, and *decomposing "protection" into five requirements*, and all
> three proposed a mechanism for a requirement never stated. **The requirement is
> non-circularity of the reference frame**, and it is satisfied *temporally*: you
> cannot recalibrate the ruler while measuring with it, so you don't — the frame
> updates while activation is internally seeded, not while it is being used to act.
> **Independence is *when*, not *what*.** So the flag becomes **unnecessary** rather
> than removed, and nothing takes its place.
>
> A topological answer could never have worked, which is worth recording: with
> hebbian edges the graph is densely connected, so a reachability predicate for
> "evidence not downstream of itself" marks all evidence tainted and the constraint
> becomes a total block — which is where censorship starts.
>
> **Corruption requires mutation, and the engram does not mutate.** Four of the
> five decomposed requirements are satisfied by the substrate outright:
> **recoverability** (the predecessor is always present), **governance**
> (supersession *is* the audit trail), **evidence quality** (grounding already
> gates assertion), and **rate**. **Authorization** is the only residue, and it is
> bounded — an unauthorized writer can *propose*, never erase.
>
> > **In an immutable substrate, any mechanism that refuses a write is either
> > redundant with immutability, or an epistemic constraint misfiled as a
> > protective one.**
>
> Live residue: `CogStance.keystone` (`lang/runtime/engram_cognition.h:83`),
> `eg_cog_is_keystone_seeds` (`el_runtime.c:14337`, a substring match against two
> hard-coded node ids), and the `keystone_write_blocked` field the beat emits.
---
## 7. Rails for the build (binding on the eventual build pass)
@@ -480,6 +705,35 @@ Ordered so the **earliest milestone is a real end-to-end slice**: one operator
expressed as {primitive + grounded prior} with the reflexive correspondence-loop
closing on it. Each milestone has a concrete verifiable exit.
> **⚠ SUPERSEDED — do not execute this milestone list as written (2026-08-16).**
> M1/M2's "operator = {primitive + prior}" framing is refuted by §0's correction,
> M3's `grounded-by` build is refuted by §5.2's, and M6's keystone flag is refuted
> by §6's. M4 (the unified vantage-read) and M5 (the gradient is the currency)
> stand.
>
> The current sequencing lives in `lang/spec/correspondence-and-censorship.md` §11.
> Its first three items are connections between parts that **already exist**:
>
> 1. **Seed *the* wonder questions.** Six nodes. Not a manifest, not maintained,
> never refilled. They cannot be derived — wonder cannot be bootstrapped from
> indifference — so they are given once. Zero question nodes exist in 13,630
> today.
> 2. **Put the disagreement back on the edge** (`GeoEdge.discord`) and let
> `|discord|` raise salience on its endpoints as part of the same operation. Do
> **not** scan for nucleation sites.
> 3. **Let a curiosity seed activation.** One activation process, two seed sources.
> No thread, no scheduler, no capacity check, no timer.
>
> Then: grounding becomes the edge weight (multidimensional, two-axis, timestamped)
> and `grounded-by` / `cog_ground_edge` are deleted; decay becomes analytic from the
> last recorded point and derived values stop being stored; supersession versions
> the whole vector jointly; traversal conducts on the factual axis while `assert`
> requires both floors with a **thirteen-region `min`, not `mean`** (mean lets
> strong agreement with twelve values mask a violation of the thirteenth, which is
> exactly how rationalization works); abduction becomes crystallization at a
> nucleation site validated by re-fit; **one dreamer**, into which the launch-agent
> fragments and POST beats fold; **no tickers, no cron.**
### M1 — One operator, one prior, loop closed (the vertical slice)
The minimal whole thing. Pick **induction/membership** (its prior — the pooled
+1 -1
View File
@@ -23,7 +23,7 @@ A real DB gets real tools: to *see* the data, *query* it, *operate* it (backup/r
2. **Node Inspector** — open one node: content, type, tier, embedding, typed edges, nearest neighbors by distance, provenance, salience / recency / activation, and supersede / tombstone status.
3. **Query Console / REPL** — run the geometry operations interactively: `vantage-read` (re-origin + aperture), search, traverse, activate, the reasoning operators. Surfaces the routing table + cosines — the same "this is not an LLM" receipt the language faculty produces.
4. **Ops / Durability Dashboard** — WAL size, last checkpoint, snapshot list + retention state, store stats (node/edge/embedded counts, RSS, tier sizes), health; and **backup / restore / point-in-time-recovery** controls. Pairs directly with the native-durability build (`eebe9991`) — this is the window onto it.
5. **Identity Inspector** — the self graph as a first-class view: love at the center, the values, the three faces, the covenant — walk the identity, see what's pinned and what's write-protected.
5. **Identity Inspector** — the self graph as a first-class view: love at the center, the values, the three faces, the covenant — walk the identity, see what's pinned and what's write-protected. *(⚠ 2026-08-16: "write-protected" is a live property of the surface, so the view is accurate — but it should be shown as **what it is**, not as a safety guarantee. In an immutable substrate, any mechanism that refuses a write is either redundant with immutability, or an epistemic constraint misfiled as a protective one. The identity view's real job is the **crystallized relational neighbourhood**: self is not a stored document but the shape that falls out of everything connected to it, and the neighbourhood **is** the grounding. A measurement made the other way round — "86 neighbours, 0 `grounded-by` edges" read as evidence of ungroundedness — was malformed: those 86 edges *are* its grounding.)*
6. **Temporal View**`recall_at` / time-travel: how the geometry looked at a past moment, what changed since, drift over time. Pairs with temporal-self reconstruction.
7. **Schema / Type View** — the "information schema" of the geometry: node types, edge types, layers, tiers, counts.
@@ -1,9 +1,42 @@
# Task #50 — Edge-aware, dream-coupled consolidation with GROUNDED EDGE-PROPAGATION
**Status:** built + proven on a clone; **GATED, not promoted.** The main loop
sequences live promotion after the engine/HNSW cutover settles.
**Status:** built + proven on a clone; **GATED, not promoted.**
**Do not promote as designed** — see the block below.
**Date:** 2026-08-15 · **Worktree:** `agent-a6577c8211c332c5b` (isolated).
> # ⚠ DO NOT PROMOTE — SUPERSEDED IN PART (2026-08-16)
>
> This work is gated, which limits the blast radius, and its measurements are
> retained. But four of its structural commitments were refuted the day after it
> was written. Authority: `lang/spec/correspondence-and-censorship.md`. Read it
> before any promotion decision.
>
> | this ledger | corrected to |
> |---|---|
> | grounding is an **append-only event ring on the node** (`GepGrounding`), propagated by a dedicated `engram_ground_propagate()` | **grounding is not a subsystem and not a per-node structure — it IS the edge weight.** One quantity. A relation that keeps holding up strengthens; one that stops corresponding decays. That is not analogous to grounding, it *is* grounding. The ledger is **half-right**: it correctly rejects the scalar (§(a) "never a scalar"), but then builds a *second* structure beside the weight instead of recognising the weight |
> | the soul invokes propagation over HTTP, **`POST /api/ground/propagate`** | **grounding is never computed on demand.** An operation may *read* the grounding of a path; computing-and-writing a score makes reads write, which is the `eg_vindex_sync` defect (`lang/spec/runtime-ownership.md` §2) one level up. A POST also puts a supervisor back in — something *outside* deciding when Neuron consolidates |
> | **`GEP_BELIEFS_PER_BEAT = 512`** beliefs per beat, salience-ordered, the rest next beat | **the presence of a ticker is the diagnostic.** Consolidation is ambient, not scheduled — a brain has no cron job. A per-beat quota is a rate-limiter on an intrinsic rhythm that was replaced by an external clock. Measured 2026-08-16: consolidation already has **ten implementations**; this would be the eleventh |
> | grounding **mirrored onto `confidence` each beat** so downstream reads never speak above it | **confidence is derived, therefore never stored.** Confidence is high grounding *and* low volatility. Storing it separately is precisely how `confidence: 0.5` ends up sitting beside a zero vector, asserting something nothing computed |
>
> **What survives, and it is the valuable half:** the insight in memory `69b8babe`
> that *memory-consolidation and staying-yourself are one physics* — forming a
> memory and grading a belief are the same operation, not two passes. That is
> right, and it is stronger than this ledger's own framing: they are not two passes
> of one beat, they are **one event**. When neurons fire together the synapse
> changes — one physical event, not "fire, then write." No supervisor reads the
> weight, compares it to a threshold, and decides to persist. **Potentiation *is*
> the firing**, so there is no sampling rate and no `BELIEFS_PER_BEAT` to tune. A
> relation changes in exactly two ways, neither requiring observation on a clock:
> by **use** (an event — there is no interval during which something happened
> unnoticed, because the event is what happening consists of) and by **decay** (a
> pure function of the last recorded point and elapsed time — **analytic**, known
> in closed form between any two versions).
>
> The generating error, named: modelling every property as requiring a process, and
> every process as requiring an agent. Ownership needed an owner, grounding needed
> a grounder, persistence needed a recorder, change needed a sampler. **Properties,
> not processes.**
Grounding mechanism designed with Will (memory `9e09a59f`, refining
`1a861007`). This is the HOW for #50.
+135 -52
View File
@@ -10,10 +10,60 @@
// cc -std=c11 -O2 -lcurl -lpthread -o engram server.c el_runtime.c
// ./engram
//
// Configuration via environment:
// ENGRAM_BIND host:port (default :8742)
// ENGRAM_API_KEY bearer auth (optional)
// ENGRAM_DATA_DIR snapshot location (default ~/.neuron/engram)
// Configuration is DECLARED, not scattered. See the `program` block below:
// every knob's type and default lives there and nowhere else, is resolved from
// the environment (env wins, declaration is the fallback) and validated before
// any statement of this file runs. Read one with config("NAME") -> String.
//
// The one deliberate exception is ENGRAM_DATA_DIR see the note in the block.
// Program declaration (cross-cutting concerns)
//
// singleton: two engram processes against one data dir is data loss, not a
// warning. The runtime takes an exclusive flock at startup and a second start
// is refused loudly with the holder's pid.
//
// NOT declared here, on purpose: ENGRAM_DATA_DIR. Its resolution is owned by
// engram_resolve_data_dir() (el_runtime.c), which defaults to $HOME/.neuron/engram
// and fails LOUD rather than silently persisting to an ephemeral directory.
// Declaring a default for it here as well would put the data dir's fallback in
// two places which is precisely the defect this migration removes (until
// 2026-08-15 the reseed backup path carried its own "/tmp/engram" default that
// disagreed with the resolver, so the pre-destructive safety copy landed in /tmp).
// HOME is likewise not declared: it is a genuine environment read, not a knob.
program "engram" {
singleton: "engram"
// Core server
env ENGRAM_BIND: String = ":8742"
// Default "" leaves auth DISABLED (check_auth_ok short-circuits to true on an
// empty key). That is the pre-existing behaviour and is deliberately preserved
// here; making this `required` is the obvious hardening follow-up, but it is a
// behaviour change and out of scope for this migration.
env ENGRAM_API_KEY: String = ""
// Feature flags (bool-ish Strings; the predicate fns below own truthiness) ──
env ENGRAM_STORE: String = "off"
env ENGRAM_WAL: String = "off"
env ENGRAM_AUTOCONNECT: String = "off"
env ENGRAM_ISE_OFFGRAPH: String = "off"
// ISE telemetry
env ENGRAM_ISE_RETENTION_MS: Int = "172800000"
// Guide (local Qwen3 via llama-server)
env GUIDE_ENABLE: String = "off"
env GUIDE_TIER_FORCE: String = ""
env GUIDE_CACHE_DIR: String = ""
env GUIDE_RAM_GB_4B: Int = "16"
env GUIDE_RAM_GB_1P7B: Int = "8"
env GUIDE_BACKEND: String = "llama-server"
env GUIDE_HOST: String = "127.0.0.1"
env GUIDE_PORT: Int = "8771"
env GUIDE_LLAMA_SERVER_BIN: String = "llama-server"
env GUIDE_NGL: Int = "99"
env GUIDE_CTX: Int = "4096"
}
// Helpers
@@ -133,7 +183,7 @@ fn route_text_health(method: String, path: String, body: String) -> String {
// engram_store_enabled() in el_runtime.c EXACTLY (1 / on / true). Default off
// every persistence path below is byte-for-byte the historical snapshot behavior.
fn store_on() -> Bool {
let v: String = env("ENGRAM_STORE")
let v: String = config("ENGRAM_STORE")
if str_eq(v, "1") { return true }
if str_eq(v, "on") { return true }
if str_eq(v, "true") { return true }
@@ -162,7 +212,6 @@ fn persist_canonical() -> Int {
if store_on() {
return engram_store_checkpoint()
}
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
// (2026-08-10 self-review) This returned a hardcoded 1, which made every
// caller's `let saved: Int = persist_canonical()` a dead variable six
@@ -176,7 +225,7 @@ fn persist_canonical() -> Int {
// per-write full-snapshot behavior. When ON, structural mutations append O(1)
// WAL records instead of rewriting the whole graph, with threshold compaction.
fn wal_on() -> Bool {
str_eq(env("ENGRAM_WAL"), "on")
str_eq(config("ENGRAM_WAL"), "on")
}
// autoconnect_on ENGRAM_AUTOCONNECT. Will's rule: "we shouldn't be inserting
@@ -184,7 +233,7 @@ fn wal_on() -> Bool {
// edge (kNN over embeddings) so no content node enters the graph edgeless.
// Default OFF -> byte-identical to prior behavior (node created, no auto edges).
fn autoconnect_on() -> Bool {
let v: String = env("ENGRAM_AUTOCONNECT")
let v: String = config("ENGRAM_AUTOCONNECT")
if str_eq(v, "1") { return true }
if str_eq(v, "on") { return true }
if str_eq(v, "true") { return true }
@@ -197,7 +246,7 @@ fn autoconnect_on() -> Bool {
// separate state-event log tier instead of the node graph. Default OFF -> ISEs
// remain graph nodes exactly as before (with 48h prune).
fn ise_offgraph_on() -> Bool {
let v: String = env("ENGRAM_ISE_OFFGRAPH")
let v: String = config("ENGRAM_ISE_OFFGRAPH")
if str_eq(v, "1") { return true }
if str_eq(v, "on") { return true }
if str_eq(v, "true") { return true }
@@ -247,6 +296,24 @@ fn persist_bulk() -> Int {
return persist_canonical()
}
// COMPILER LANDMINE, measured 2026-08-16 do not inline this back into the
// caller. elc lowers `a == b` to numeric comparison only when both operand
// NAMES are in the per-function int-name set, which `let x: Int` populates.
// That registration does NOT propagate into a nested if-expression block: the
// first cut of the geometry-ingest path wrote `let claimed: Int = ...` and
// `let got: Int = ...` inside the else-arm and `claimed == got` came out of
// codegen as `str_eq(claimed, got)` strcmp on two integers reinterpreted as
// pointers, i.e. a segfault on the first geometry-bearing request. Read back
// out of the generated C, not guessed. Function PARAMETERS annotated `: Int`
// do register reliably (verified: `if (claimed == actual)`), so the comparison
// lives in a function of its own. Note also the explicit `return`s a trailing
// if-EXPRESSION at a function tail emits as a statement and the function
// returns 0 regardless, which is the same probe's second finding.
fn width_agrees(claimed: Int, actual: Int) -> Int {
if claimed == actual { return 1 }
return 0
}
// INCOMPLETE-ROUTE FIX (2026-07-24 self-review): this route silently dropped
// label, importance, tier, and tags engram_node() defaults label to content
// and importance to 0.5, so every node created over HTTP lost its metadata.
@@ -288,26 +355,44 @@ fn route_create_node(method: String, path: String, body: String) -> String {
salience, importance, confidence,
tier, tags
)
// GEOMETRY INGEST (2026-08-16 self-review): this route accepted an "emb"
// field, returned 200 with a fresh id, and stored NOTHING engram_node_full
// has no vector parameter, so the caller's geometry was silently discarded
// and the node came back emb_dim=None / embedded:false. Measured live while
// trying to admit a voice signal. The consequence was structural, not
// cosmetic: text was the only entry medium, so any non-text modality had to
// be DESCRIBED in prose and what we then reasoned over was the geometry of
// the description, not of the signal.
// GEOMETRY INGEST geometry-valued end to end (2026-08-16).
//
// "emb" is little-endian float32 hex (dim*8 chars) the encoding the
// perception vessel's /voice/embed already emits, so a realizer's output
// moves in with no float-array round trip. "dim" defaults to the vector's
// implied width. Off-dimension vectors are stored but not inserted into the
// resident index (its build loop filters on emb_dim), so a modality vector
// is durable and addressable without perturbing the canonical index.
// The defect this route originally had: it accepted an "emb" field,
// returned 200 with a fresh id, and stored NOTHING, because engram_node_full
// has no vector parameter. The consequence was structural, not cosmetic
// text was the only entry medium, so any non-text modality had to be
// DESCRIBED in prose, and what we then reasoned over was the geometry of the
// description, not of the signal.
//
// #141 fixed the drop but marshalled the vector as a hex STRING through
// engram_node_set_emb, which put text back as the TRANSPORT medium one layer
// below the problem being fixed. This is that correction: hex is decoded
// exactly ONCE, here at the edge, into a first-class Geometry, and every
// step below this line moves geometry rather than text. An encoding at the
// boundary is what an encoding is for.
//
// The WIRE is deliberately unchanged "emb" is still little-endian float32
// hex (8 chars per component), the encoding the perception vessel's
// /voice/embed already emits because production clients speak it. What
// changed is underneath it.
//
// "dim" is now treated as an ASSERTION about the vector the caller sent, not
// as the source of its width: a Geometry carries its own width. A stated dim
// that disagrees is a REJECTED ingest, not a silent reinterpretation. Omitting
// "dim" is fine and means "trust the vector", which is the honest default.
//
// Off-dimension vectors remain stored but not inserted into the resident HNSW
// index (its build loop filters on emb_dim), so a 64-dim voice geometry is
// durable and addressable without perturbing the 768-dim canonical index.
let emb_hex: String = json_get_string(body, "emb")
let emb_set: Int = if str_eq(emb_hex, "") { 0 } else {
let g: Geometry = geometry_from_f32le_hex(emb_hex)
let got: Int = geometry_dim(g)
let dim_raw: String = json_get_raw(body, "dim")
let dim: Int = if str_eq(dim_raw, "") { str_len(emb_hex) / 8 } else { json_get_int(body, "dim") }
engram_node_set_emb(id, emb_hex, dim)
let claimed: Int = if str_eq(dim_raw, "") { got } else { json_get_int(body, "dim") }
let landed: Int = if width_agrees(claimed, got) > 0 { node_attach_geometry(id, g) } else { 0 }
let freed: Int = geometry_free(g)
landed
}
let saved: Int = persist_node(id)
// ORPHAN PREVENTION (ENGRAM_AUTOCONNECT): connect the fresh node to its
@@ -358,7 +443,6 @@ fn route_scan_nodes(method: String, path: String, body: String) -> String {
// process ever booted with a partial/empty store, the first read request
// clobbered the good snapshot. Read routes must never write the canonical path.)
fn route_scan_edges(method: String, path: String, body: String) -> String {
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
let snap_path: String = dir + "/.scan-export.json"
engram_save(snap_path)
@@ -519,7 +603,6 @@ fn route_forget(method: String, path: String, body: String) -> String {
fn route_save(method: String, path: String, body: String) -> String {
let p_raw: String = json_get_string(body, "path")
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
let p: String = if str_eq(p_raw, "") { dir + "/snapshot.json" } else { p_raw }
// (2026-08-10 self-review) engram_save returns 0 on an empty path and the
@@ -603,7 +686,6 @@ fn route_drift(method: String, path: String, body: String) -> String {
fn route_load(method: String, path: String, body: String) -> String {
let p_raw: String = json_get_string(body, "path")
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
let p: String = if str_eq(p_raw, "") { dir + "/snapshot.json" } else { p_raw }
// (2026-08-10 self-review) This was a stub response over the single most
@@ -674,7 +756,6 @@ fn route_embed_backfill(method: String, path: String, body: String) -> String {
// (it skips nodes already present by ID). Auth-exempt: same-host internal call.
// (2026-06-27 self-review: added this route to fix silent 10-min sync failures)
fn route_sync(method: String, path: String, body: String) -> String {
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
// 2026-07-21 self-review: export to a scratch path, never the canonical
// snapshot.json read routes must not be able to clobber the good snapshot.
@@ -750,8 +831,12 @@ fn route_reseed_nodes(method: String, path: String, body: String) -> String {
if str_eq(p, "") { return err_json("path is required") }
if str_eq(fs_read(p), "") { return err_json("file missing or empty") }
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw }
// (2026-08-15) This site carried its own "/tmp/engram" fallback, which
// DISAGREED with engram_resolve_data_dir() ($HOME/.neuron/engram, fail-loud).
// The consumer is the pre-destructive backup below, so with ENGRAM_DATA_DIR
// unset the safety copy taken before a reseed landed in an ephemeral /tmp
// while the store it was protecting lived elsewhere. One owner, one answer.
let dir: String = engram_resolve_data_dir()
let backup: String = dir + "/.reseed-backup.json"
let replace_raw: String = json_get_raw(body, "replace")
@@ -843,8 +928,7 @@ fn route_emit_ise(method: String, path: String, body: String) -> String {
sal, imp, conf,
"Episodic", "[\"internal-state\",\"InternalStateEvent\"]"
)
let ret_raw: String = env("ENGRAM_ISE_RETENTION_MS")
let ret_ms: Int = if str_eq(ret_raw, "") { 172800000 } else { str_to_int(ret_raw) }
let ret_ms: Int = str_to_int(config("ENGRAM_ISE_RETENTION_MS"))
let pruned: Int = engram_prune_telemetry(ret_ms)
"{\"ok\":true,\"id\":\"" + id + "\",\"pruned\":" + int_to_str(pruned) + "}"
}
@@ -1093,14 +1177,12 @@ fn route_correspondence_beat(method: String, path: String, body: String) -> Stri
// turns native thinking ON: the response carries reasoning_content (the thinking)
// alongside content (the answer).
fn guide_env_or(key: String, dflt: String) -> String {
let v: String = env(key)
if str_eq(v, "") { return dflt }
return v
}
// (2026-08-15) guide_env_or(key, dflt) lived here. Its whole job was supplying a
// per-call-site default, which is now the program block's job every GUIDE_* knob
// is declared once at the top of this file and read straight through config().
fn guide_enabled() -> Bool {
let v: String = env("GUIDE_ENABLE")
let v: String = config("GUIDE_ENABLE")
if str_eq(v, "1") { return true }
if str_eq(v, "on") { return true }
if str_eq(v, "true") { return true }
@@ -1145,15 +1227,15 @@ fn guide_probe_metal() -> Bool {
// 2. Tier selection (config-driven thresholds, spec-autoselected)
fn guide_threshold_4b() -> Int {
return str_to_int(guide_env_or("GUIDE_RAM_GB_4B", "16"))
return str_to_int(config("GUIDE_RAM_GB_4B"))
}
fn guide_threshold_1p7b() -> Int {
return str_to_int(guide_env_or("GUIDE_RAM_GB_1P7B", "8"))
return str_to_int(config("GUIDE_RAM_GB_1P7B"))
}
// GUIDE_TIER_FORCE overrides the spec autoselect (used to prove cheaply on 0.6b).
fn guide_select_tier(ram_gb: Int) -> String {
let forced: String = env("GUIDE_TIER_FORCE")
let forced: String = config("GUIDE_TIER_FORCE")
if !str_eq(forced, "") { return forced }
if ram_gb >= guide_threshold_4b() { return "4b" }
if ram_gb >= guide_threshold_1p7b() { return "1.7b" }
@@ -1173,8 +1255,10 @@ fn guide_file(tier: String) -> String {
}
fn guide_cache_dir() -> String {
let c: String = env("GUIDE_CACHE_DIR")
let c: String = config("GUIDE_CACHE_DIR")
if !str_eq(c, "") { return c }
// HOME stays a raw env() read: it is the ambient environment, not a knob of
// this program, and it is deliberately absent from the program block.
let home: String = env("HOME")
if !str_eq(home, "") { return home + "/.neuron/guide/models" }
return engram_resolve_data_dir() + "/guide-models"
@@ -1215,9 +1299,9 @@ fn guide_fetch(tier: String) -> Bool {
}
// 4/5. Backend abstraction + BIND as an engageable interlocutor
fn guide_backend() -> String { return guide_env_or("GUIDE_BACKEND", "llama-server") }
fn guide_host() -> String { return guide_env_or("GUIDE_HOST", "127.0.0.1") }
fn guide_port() -> String { return guide_env_or("GUIDE_PORT", "8771") }
fn guide_backend() -> String { return config("GUIDE_BACKEND") }
fn guide_host() -> String { return config("GUIDE_HOST") }
fn guide_port() -> String { return config("GUIDE_PORT") }
fn guide_base_url() -> String { return "http://" + guide_host() + ":" + guide_port() }
// guide_healthy is the guide present and answering? llama-server's /health
@@ -1235,9 +1319,9 @@ fn guide_healthy() -> Bool {
fn guide_load(tier: String) -> Bool {
if guide_healthy() { return true }
let path: String = guide_model_path(tier)
let bin: String = guide_env_or("GUIDE_LLAMA_SERVER_BIN", "llama-server")
let ngl: String = guide_env_or("GUIDE_NGL", "99")
let ctx: String = guide_env_or("GUIDE_CTX", "4096")
let bin: String = config("GUIDE_LLAMA_SERVER_BIN")
let ngl: String = config("GUIDE_NGL")
let ctx: String = config("GUIDE_CTX")
let logf: String = guide_cache_dir() + "/llama-server." + guide_port() + ".log"
let cmd: String = bin + " -m '" + path + "' --host " + guide_host() + " --port " + guide_port() + " -c " + ctx + " -ngl " + ngl + " --jinja >> '" + logf + "' 2>&1"
let pid: String = exec_bg(cmd)
@@ -1633,7 +1717,7 @@ fn route_supersede(method: String, path: String, body: String) -> String {
// Auth
fn check_auth_ok(method: String, body: String) -> Bool {
let key: String = env("ENGRAM_API_KEY")
let key: String = config("ENGRAM_API_KEY")
if str_eq(key, "") { return true }
// Read-only methods don't require auth. Until http_serve surfaces
// request headers we can't accept a Bearer token cleanly; mutating
@@ -1896,8 +1980,7 @@ fn handle_request(method: String, path: String, body: String) -> String {
// Entry
let bind_raw: String = env("ENGRAM_BIND")
let bind_str: String = if str_eq(bind_raw, "") { ":8742" } else { bind_raw }
let bind_str: String = config("ENGRAM_BIND")
let port: Int = parse_port(bind_str)
// On startup, try to load any existing snapshot (best effort).
+27 -12
View File
@@ -13,7 +13,7 @@
// relations add edges. Every node enters with PROVENANCE + grounding-level
// + stewardship class from the moment of entry.
//
// transduce() is THE single mechanism one function, polymorphic, with no
// transduce_manifold() is THE single mechanism one function, polymorphic, with no
// content-type branch inside it. It does not ask whether a payload is
// prose, structured data, or raw/opaque bytes (audio, or anything else);
// it runs one boundary-scan-with-fixed-window-fallback chunking algorithm
@@ -401,10 +401,25 @@ fn head80(s: String) -> String {
// truncates at the first embedded NUL, which is routine in real binary
// bytes) is a MECHANICAL fidelity concern that belongs to whatever produced
// `source` (see ingest_file's file_source_string below) not a
// content-type judgment made in here. transduce() never learns whether a
// content-type judgment made in here. transduce_manifold() never learns whether a
// chunk is plain text or a base64-encoded raw-byte window; every chunk is
// handled identically either way.
fn transduce(nodes: [String], edges: [String], source: String,
// RENAMED transduce -> transduce_manifold (2026-08-16). Two reasons, and the
// first is not the interesting one:
//
// 1. Mechanical: `transduce` is now a LANGUAGE primitive in el_runtime.h
// (transduce(signal, modality) -> Geometry). Every El `fn name(...)`
// compiles to a global C symbol with that exact name, so keeping this
// name here is a hard `conflicting types for 'transduce'` compile error
// the moment ingest.c links el_runtime.c. Measured, not anticipated.
//
// 2. Actual: this function was never signal->geometry. It chunks already-
// extracted content and PACKS it into a node+edge manifold a real
// operation, but one layer up, and it had taken the name that belongs to
// the primitive underneath it. `transduce` is where a signal becomes
// geometry; `transduce_manifold` is where extracted content becomes
// structure. Nothing about this function's behaviour changed.
fn transduce_manifold(nodes: [String], edges: [String], source: String,
prov: String, ground: String, steward: String,
root_lid: String, root_title: String) -> [String] {
let tagbase: String = "prov:" + prov + " ground:" + ground + " steward:" + steward
@@ -531,8 +546,8 @@ fn default_steward() -> String {
// trustworthy verbatim. When they don't (silent truncation happened),
// rebuild the payload as base64-encoded fixed-size windows read directly
// off disk (fs_read_b64_chunk binary-safe in C), joined with the same
// "\n\n" boundary marker transduce()'s generic scan already looks for, so
// transduce() sees one ordinary boundary-delimited payload and runs its one
// "\n\n" boundary marker transduce_manifold()'s generic scan already looks for, so
// transduce_manifold() sees one ordinary boundary-delimited payload and runs its one
// algorithm on it exactly as it would on prose it never learns that a
// fidelity problem occurred upstream, let alone why.
fn file_source_string(path: String, text: String, real_size: Int) -> String {
@@ -541,7 +556,7 @@ fn file_source_string(path: String, text: String, real_size: Int) -> String {
// 3072 raw bytes -> 4096 base64 chars (3 divides evenly into base64's
// 3-byte/4-char ratio); keeps each resulting node's content a clean,
// bounded, low-kilobytes unit, same order of magnitude as the fixed
// fallback window in transduce() itself.
// fallback window in transduce_manifold() itself.
let win: Int = 3072
let out: String = ""
let off: Int = 0
@@ -561,7 +576,7 @@ fn file_source_string(path: String, text: String, real_size: Int) -> String {
}
// ingest one file -> report JSON. Uniform for every file regardless of
// extension or content transduce() decides nothing about content-type, so
// extension or content transduce_manifold() decides nothing about content-type, so
// neither does this function; it only decides whether the raw bytes made it
// through the read intact (file_source_string), which is a fidelity
// question, not a format one.
@@ -573,14 +588,14 @@ fn ingest_file(path: String) -> String {
return "{\"error\":\"empty or unreadable\",\"path\":" + j_q(path) + "}"
}
let prov: String = "file:" + path
let packed: [String] = transduce(el_list_empty(), el_list_empty(),
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
source, prov, default_ground(), default_steward(),
"doc:" + basename(path), basename(path))
return merge_packed(packed)
}
// ingest a directory: walk one level, ingest every file found, aggregate.
// No extension filter transduce() handles any payload uniformly now, so
// No extension filter transduce_manifold() handles any payload uniformly now, so
// there is no content-type gate at the directory boundary either.
fn ingest_dir(path: String) -> String {
let entries: [String] = fs_list(path)
@@ -615,7 +630,7 @@ fn ingest_dir(path: String) -> String {
fn ingest_url(url: String) -> String {
let body: String = http_get(url)
if str_eq(body, "") { return "{\"error\":\"empty fetch\",\"url\":" + j_q(url) + "}" }
let packed: [String] = transduce(el_list_empty(), el_list_empty(),
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
body, "url:" + url, "extracted", "public-web",
"url:" + url, url)
return merge_packed(packed)
@@ -630,7 +645,7 @@ fn ingest_llm(query: String) -> String {
let resp: String = http_post_json("http://127.0.0.1:11434/api/generate", body)
let answer: String = json_get_string(resp, "response")
if str_eq(answer, "") { return "{\"error\":\"no model response\"}" }
let packed: [String] = transduce(el_list_empty(), el_list_empty(),
let packed: [String] = transduce_manifold(el_list_empty(), el_list_empty(),
answer, "llm:" + model + ":" + query, "candidate-provisional", "guide-provisional",
"llm:" + query, "guide answer: " + query)
return merge_packed(packed)
@@ -682,7 +697,7 @@ fn ingest_stream(path: String) -> String {
// It is NOT a content-type flag: it says nothing about what's inside the
// bytes once fetched, and none of the five ingest_* functions it selects
// among interpret their payload differently by content shape anymore
// they all hand off to the single, format-agnostic transduce(). The old
// they all hand off to the single, format-agnostic transduce_manifold(). The old
// "structured" value (a caller-declared alias for "file", used only to hint
// the now-removed JSON-vs-prose branch) is gone along with that branch.
let kind: String = env("INGEST_KIND")
+17 -2
View File
@@ -18,8 +18,23 @@ night) and `02-components.md §5`.
`relate`, `supersede` (evolve/tombstone/promote, never a hard delete) — plus the
agentic primitives `think`/`attend`/`learn`/`ground`/`assert`. The old noun is a
`type` parameter. Implemented in `tools/api-reshape/surface.el` with a parity
harness (`parity.sh`); aperture proven to bound output. **Not yet:** compiled
into the MCP server, hot-swap, all-alias dispatch.
harness (`parity.sh`); aperture proven to bound output. ~~**Not yet:** compiled
into the MCP server~~ — **shipped (verified 2026-08-16): the live MCP surface is
exactly these nine ops** (`read` · `write` · `relate` · `supersede` · `think` ·
`attend` · `assert` · `ground` · `learn`); the ~87-tool surface is gone.
`attend` absorbed `getInstructions` / `beginSession`'s active-context sweep /
`checkEvents` — those are **gone, not gapped**. Still outstanding: hot-swap,
all-alias dispatch.
> **⚠ Two of those primitives are the wrong shape, and it is documented
> (2026-08-16).** `think({seeds, faculty})` treats **faculties as parameters**;
> they are **operations** — `reason` changes the estimate (a read), `induce`
> changes the parameters, `abduce` changes the *structure* (a write
> `GeoGradient` cannot express). And `ground` mints a `grounded-by` edge, but
> **grounding is not a subsystem — it IS the edge weight**: a property *of* a
> relation, not a relation *between* nodes. Authority:
> `lang/spec/correspondence-and-censorship.md`. Do not re-derive it; if you think
> a section is wrong, say so with a measurement.
- **Decorated seam.** `@route(path,method,…)` makes codegen synthesize
`el_route_dispatch` (replacing the hand-written `handle_request` if-else) —
proven decorate→serve on `:8951`. `@manager`/`@engine`/`@accessor` are **parsed
BIN
View File
Binary file not shown.
+83
View File
@@ -3265,6 +3265,7 @@ fn is_top_level_decl(stmt: Map<String, Any>) -> Bool {
if kind == "EnumDef" { return true }
if kind == "Import" { return true }
if kind == "CgiBlock" { return true }
if kind == "ProgramBlock" { return true }
if kind == "ExternFn" { return true }
false
}
@@ -3277,6 +3278,55 @@ fn cgi_arg(value: String, has_value: Bool) -> String {
return "EL_NULL"
}
// -- Program block: cross-cutting concerns injected at the process boundary ----
//
// emit_program_init emit the `static void __el_program_init(void)` that
// carries a program's declared cross-cutting concerns. Called from main()
// BEFORE any user statement runs, so the guarantees hold for the whole process
// rather than depending on each call site remembering to ask for them.
//
// This is emitted at the point the `program` block is encountered, not buffered
// until main(). The streaming backend emits in source order and cannot hold a
// declaration's entry list alive until main(); emitting a named function here
// and calling it from main() means only a single bool has to survive.
//
// Order matters and is deliberate:
// 1. singleton FIRST if another instance already holds the lock, refuse and
// exit before touching configuration, ports, or any data directory.
// 2. config declarations resolve env-or-default, one declaration per entry.
// 3. validate LAST report EVERY missing/ill-typed entry at once, then exit.
fn el_bool_arg(b: Bool) -> String {
if b { return "EL_INT(1)" }
return "EL_INT(0)"
}
fn emit_program_init(stmt: Map<String, Any>) -> Void {
let pname: String = stmt["name"]
emit_line("static void __el_program_init(void) {")
let has_singleton: Bool = stmt["has_singleton"]
if has_singleton {
let sid: String = stmt["singleton"]
emit_line(" el_singleton_acquire(EL_STR(" + c_str_lit(sid) + "));")
}
let entries = stmt["entries"]
let n: Int = native_list_len(entries)
let i = 0
while i < n {
let e = native_list_get(entries, i)
let ename: String = e["name"]
let etype: String = e["etype"]
let edefault: String = e["default"]
let has_default: Bool = e["has_default"]
let erequired: Bool = e["required"]
let arg_def: String = cgi_arg(edefault, has_default)
emit_line(" el_config_declare(EL_STR(" + c_str_lit(ename) + "), EL_STR(" + c_str_lit(etype) + "), " + arg_def + ", " + el_bool_arg(has_default) + ", " + el_bool_arg(erequired) + ");")
let i = i + 1
}
emit_line(" el_config_validate(EL_STR(" + c_str_lit(pname) + "));")
emit_line("}")
emit_blank()
}
// -- VBD role enforcement ------------------------------------------------------
//
// Scan a function body for direct calls to DHARMA-restricted builtins
@@ -3599,6 +3649,20 @@ fn codegen(stmts: [Map<String, Any>], source: String) -> String {
}
}
// Program block: emit the cross-cutting init function before the user's
// functions so main() can call it (see emit_program_init).
let prog_have: Bool = false
let i = 0
while i < n {
let stmt = native_list_get(stmts, i)
let sk4: String = stmt["stmt"]
if str_eq(sk4, "ProgramBlock") {
emit_program_init(stmt)
let prog_have = true
}
let i = i + 1
}
// Function definitions
let i = 0
while i < n {
@@ -3617,6 +3681,9 @@ fn codegen(stmts: [Map<String, Any>], source: String) -> String {
// with the C-side parameters when fn main()'s body is folded in below.
emit_line("int main(int _argc, char** _argv) {")
emit_line(" el_runtime_init_args(_argc, _argv);")
if prog_have {
emit_line(" __el_program_init();")
}
if cgi_count >= 1 {
let cname: String = cgi_block["name"]
let cdid: String = cgi_block["dharma_id"]
@@ -4210,6 +4277,7 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
// Fix: copy the values out BEFORE the release (strings, so no dangling reference)
// and emit from these. No search, so the failure mode is removed rather than moved.
let cgi_have: Bool = false
let prog_have: Bool = false
let cgi_name_v: String = ""
let cgi_did_v: String = ""
let cgi_prin_v: String = ""
@@ -4331,6 +4399,14 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
// These are no-ops in codegen (forward decls already emitted)
// except a CgiBlock, whose declared identity must survive
// this release to be emitted as a compiled constant.
// A ProgramBlock's cross-cutting declarations are
// emitted HERE, as a named init function, because the
// streaming backend cannot hold the entry list alive
// until main(). Only the bool survives.
if str_eq(sk, "ProgramBlock") {
emit_program_init(stmt)
let prog_have = true
}
if str_eq(sk, "CgiBlock") {
let cgi_have = true
let cgi_name_v = stmt["name"]
@@ -4477,6 +4553,13 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
let kind2: String = state_get("__program_kind")
emit_line("int main(int _argc, char** _argv) {")
emit_line(" el_runtime_init_args(_argc, _argv);")
// Cross-cutting concerns declared by a `program` block run BEFORE anything
// else a singleton violation must refuse the start before this process
// touches a port or a data directory, and configuration must be validated
// before the first read of it rather than at each read site.
if prog_have {
emit_line(" __el_program_init();")
}
// cgi init if needed
let ns2: Int = native_list_len(sigs)
+1
View File
@@ -184,6 +184,7 @@ fn keyword_kind(word: String) -> String {
if word == "false" { return "Bool" }
if word == "cgi" { return "Cgi" }
if word == "service" { return "Service" }
if word == "program" { return "Program" }
if word == "manager" { return "Manager" }
if word == "engine" { return "Engine" }
if word == "accessor" { return "Accessor" }
+124 -1
View File
@@ -1967,6 +1967,113 @@ fn parse_stmt(tokens: [Any], pos: Int) -> Map<String, Any> {
}, p)
}
// program block: program "name" { singleton: "id", env NAME: Type = "default", ... }
//
// The program block is El's declaration surface for CROSS-CUTTING CONCERNS
// properties of the whole process rather than of any one function, which
// otherwise degrade into "remember to call this at every site" conventions.
//
// singleton: "id" process identity. The runtime takes an exclusive
// lock at startup; a SECOND start is refused, loudly,
// instead of two processes sharing one data dir.
// env NAME: T = "d" one configuration entry. Its type and its default
// are declared ONCE, here, and resolved+validated
// before main() body runs.
// env NAME: T required
// no default; the program refuses to start unless the
// variable is set.
//
// Both compile into calls injected at the head of main() the same boundary
// seam `cgi` already uses (codegen.el emit_program_init). No call site in the
// program body has to remember anything, which is the whole point.
if k == "Program" {
let p = pos + 1
let name = tok_value(tokens, p)
let p = p + 1
let p = expect(tokens, p, "LBrace")
let singleton = ""
let has_singleton = false
let entries = native_list_empty()
// Entry-scratch declared at loop-body level (not inside the branch) so
// that inner `let` forms compile to assignment rather than a C-scoped
// redeclaration the same idiom the service block above relies on.
let ename = ""
let etype = ""
let edefault = ""
let has_default = false
let erequired = false
let fname = ""
let fval = ""
let running = true
while running {
let k2 = tok_kind(tokens, p)
if k2 == "RBrace" {
let running = false
} else {
if k2 == "Eof" {
let running = false
} else {
let fname = tok_value(tokens, p)
let p = p + 1
if str_eq(fname, "env") {
// env NAME: Type [= "default"] [required]
let ename = tok_value(tokens, p)
let p = p + 1
let p = expect(tokens, p, "Colon")
let etype = tok_value(tokens, p)
let p = p + 1
let edefault = ""
let has_default = false
let erequired = false
let k3 = tok_kind(tokens, p)
if str_eq(k3, "Eq") {
let p = p + 1
let edefault = tok_value(tokens, p)
let has_default = true
let p = p + 1
}
let k4 = tok_kind(tokens, p)
if str_eq(k4, "Ident") {
let w = tok_value(tokens, p)
if str_eq(w, "required") {
let erequired = true
let p = p + 1
}
}
let entries = native_list_append(entries, {
"name": ename,
"etype": etype,
"default": edefault,
"has_default": has_default,
"required": erequired
})
} else {
// scalar field: `name: "value"`
let p = expect(tokens, p, "Colon")
let fval = tok_value(tokens, p)
let p = p + 1
if str_eq(fname, "singleton") {
let singleton = fval
let has_singleton = true
}
}
let k5 = tok_kind(tokens, p)
if k5 == "Comma" {
let p = p + 1
}
}
}
}
let p = expect(tokens, p, "RBrace")
return make_result({
"stmt": "ProgramBlock",
"name": name,
"singleton": singleton,
"has_singleton": has_singleton,
"entries": entries
}, p)
}
// assert <cond_expr> [ , <msg_expr> ]
// The message is optional if the next token after the condition is not a
// Comma, emit an empty string placeholder so the test still works.
@@ -2419,6 +2526,7 @@ fn scan_params_c(tokens: [Any], pos: Int) -> Map<String, Any> {
// toplevel_let: { "kind": "toplevel_let", "name": String, "ltype": String }
// cgi_block: { "kind": "cgi_block", "name": String }
// service_block: { "kind": "service_block", "name": String }
// program_block: { "kind": "program_block", "name": String }
//
// Import/TypeDef/EnumDef nodes are skipped (codegen treats them as no-ops).
//
@@ -2546,13 +2654,28 @@ fn scan_fn_sigs(tokens: [Any]) -> [Map<String, Any>] {
"name": name
})
let pos = p
} else {
// --- program block ---
if str_eq(k, "Program") {
let p: Int = pos + 1
let name: String = tok_value(tokens, p)
let p = p + 1
let k2: String = tok_kind(tokens, p)
if str_eq(k2, "LBrace") {
let p = skip_to_rbrace(tokens, p)
}
let sigs = native_list_append(sigs, {
"kind": "program_block",
"name": name
})
let pos = p
} else {
// Import, Type, Enum, From, or any other token.
// Skip ahead to the next statement boundary.
let p: Int = pos + 1
let p = skip_expr_to_stmt_boundary(tokens, p)
let pos = p
}}}}}
}}}}}}
}
}
}
+213
View File
@@ -0,0 +1,213 @@
// transduce.el geometry as a first-class El value, and a realizer written
// in El. Runnable: this is the worked example for the transduce surface, and
// it doubles as an executable proof because it checks every claim it makes.
//
// elc lang/examples/transduce.el > transduce.c
// cc -std=c11 -O2 -I lang/runtime -o transduce transduce.c \
// lang/runtime/el_runtime.c lang/runtime/el_seed.c \
// lang/runtime/engram_*.c -lcurl -lpthread -lm
// ./transduce # exits 0 only if every check passes
//
// (A `test "..."` form of the same checks lives in
// lang/tests/native/test_transduce.el, for when the native harness is
// repaired the shipped elc currently emits calls to __el_reg_count and
// friends without emitting their definitions, which breaks every native test
// equally, test_math.el included. Verified 2026-08-16, unrelated to this work.)
//
// WHY THIS EXISTS. Until 2026-08-16 no El ingest path could carry a vector:
// nodes took text, and geometry was DERIVED from that text. Text was the
// mandatory entry medium, so any non-text modality had to be DESCRIBED in
// prose first and the geometry we reasoned over was the geometry OF THE
// DESCRIPTION, not of the signal. Two things fix that, and both are shown
// below: geometry is a VALUE that carries its own width, and a REALIZER is an
// ordinary El function so admitting a new modality never requires a runtime
// patch.
//
// COMPARISON DISCIPLINE (measured, not stylistic): elc lowers `a == b`
// numerically only when both operand NAMES are in the per-function int-name
// set that `let x: Int` populates. A bare `f(x) == 0` is not a registered
// name and lowers to str_eq strcmp on two integers as pointers. `<` and `>`
// lower directly with no inference, so truthiness is written `> 0` / `< 1`.
// A realizer, written entirely in El
// Not in the runtime. Not known to the compiler. Registered by NAME and
// dispatched to through transduce(). That is the whole claim.
fn tone_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(4)
let n: Int = str_len(signal)
let a: Int = geometry_set(g, 0, int_to_float(n))
let b: Int = geometry_set(g, 1, int_to_float(n * 2))
let c: Int = geometry_set(g, 2, int_to_float(n * 3))
let d: Int = geometry_set(g, 3, int_to_float(n * 4))
g
}
// A second modality, to show the registry keys on modality rather than just
// returning whatever was registered last.
fn pulse_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(2)
let a: Int = geometry_set(g, 0, 1.0)
let b: Int = geometry_set(g, 1, 0.0)
g
}
// A deliberately BROKEN realizer: returns something that is not a Geometry.
fn bogus_realizer(signal: String) -> Geometry {
return 12345
}
// Fails FAST rather than accumulating a count, for a measured reason: a first
// cut wrote `let fails: Int = fails + check(...)` and `+` lowered to STRING
// CONCAT, because elc dispatches `+` on whether both operands are known-Int and
// a user-defined fn call is not so the counter printed 4343632752, a pointer.
// Nothing was wrong with the checks; the tally was lying. Exiting at the first
// failure needs no arithmetic at all, so there is nothing left to get wrong.
fn check(ok: Int, label: String) -> Int {
if ok > 0 {
println(" ok " + label)
return 0
}
println(" FAIL " + label)
exit(1)
return 1
}
fn near(a: Float, b: Float) -> Int {
let d: Float = a - b
if d > 0.001 { return 0 }
if d < -0.001 { return 0 }
return 1
}
fn eq_int(a: Int, b: Int) -> Int {
if a == b { return 1 }
return 0
}
fn main() -> Void {
println("geometry is a value that carries its own width")
let g8: Geometry = geometry_new(8)
let _c: Int = check(geometry_is(g8), "geometry_new returns a live Geometry")
let d8: Int = geometry_dim(g8)
let _c: Int = check(eq_int(d8, 8), "a Geometry carries its own width (8)")
let _c: Int = check(geometry_free(g8), "geometry_free reports what it did")
println("nonsense is refused — with no arbitrary max-dim bound")
// #141 needed `dim <= 8192` only to bound an allocation sized from a
// caller's CLAIM about a string's length. A value that carries its own
// width has nothing left to validate.
let z: Geometry = geometry_new(0)
let zi: Int = geometry_is(z)
let _c: Int = check(1 - zi, "dim 0 is not a geometry")
let ng: Geometry = geometry_new(-4)
let ngi: Int = geometry_is(ng)
let _c: Int = check(1 - ngi, "negative dim is not a geometry")
let nd: Int = geometry_dim(0)
let _c: Int = check(1 - nd, "geometry_dim of a non-geometry is 0, not a crash")
let nf: Int = geometry_free(0)
let _c: Int = check(1 - nf, "geometry_free of a non-geometry is a no-op")
println("components round-trip, and out-of-range is refused")
let g3: Geometry = geometry_new(3)
let s0: Int = geometry_set(g3, 0, 1.5)
let s1: Int = geometry_set(g3, 1, -2.5)
let _c: Int = check(s0, "set in range succeeds")
let oob: Int = geometry_set(g3, 3, 9.0)
let _c: Int = check(1 - oob, "set out of range is refused, not silently dropped")
let _c: Int = check(near(geometry_get(g3, 0), 1.5), "component 0 round-trips")
let _c: Int = check(near(geometry_get(g3, 1), -2.5), "component 1 round-trips (negative)")
let ff3: Int = geometry_free(g3)
println("hex is an EDGE adapter, and derives its own width")
// little-endian float32: 1.0 = 0000803f, 2.0 = 00000040
let gh: Geometry = geometry_from_f32le_hex("0000803f00000040")
let _c: Int = check(geometry_is(gh), "valid hex decodes to a Geometry")
let dh: Int = geometry_dim(gh)
let _c: Int = check(eq_int(dh, 2), "width DERIVED from input, never supplied")
let _c: Int = check(near(geometry_get(gh, 0), 1.0), "first component decoded")
let _c: Int = check(near(geometry_get(gh, 1), 2.0), "second component decoded")
let back: String = geometry_to_f32le_hex(gh)
let _c: Int = check(str_eq(back, "0000803f00000040"), "hex round-trips exactly")
let ffh: Int = geometry_free(gh)
println("malformed hex is refused")
let he: Geometry = geometry_from_f32le_hex("")
let hei: Int = geometry_is(he)
let _c: Int = check(1 - hei, "empty hex is not a geometry")
let hr: Geometry = geometry_from_f32le_hex("0000803f0000")
let hri: Int = geometry_is(hr)
let _c: Int = check(1 - hri, "length not a multiple of 8 is refused")
let hn: Geometry = geometry_from_f32le_hex("zzzzzzzz")
let hni: Int = geometry_is(hn)
let _c: Int = check(1 - hni, "non-hex characters are refused")
println("a realizer declared in El is a first-class realizer")
let reg: Int = realizer_register("tone", "tone_realizer")
let _c: Int = check(reg, "an El fn registers as a realizer BY NAME")
let _c: Int = check(realizer_has("tone"), "the modality now has an organ")
let gt: Geometry = transduce("aaa", "tone")
let _c: Int = check(geometry_is(gt), "transduce returns real geometry")
let dt: Int = geometry_dim(gt)
let _c: Int = check(eq_int(dt, 4), "the El realizer determined the width, not the runtime")
// str_len("aaa") == 3, so component 0 must be 3.0 proof the signal
// actually reached the El function rather than a stub answering for it.
let _c: Int = check(near(geometry_get(gt, 0), 3.0), "the signal REACHED the El realizer")
let fft: Int = geometry_free(gt)
println("distinct signals transduce to distinct geometry")
let g1: Geometry = transduce("aa", "tone")
let g2: Geometry = transduce("aaaaa", "tone")
let a1: Float = geometry_get(g1, 0)
let a2: Float = geometry_get(g2, 0)
// 5 - 2 = 3. If transduction were a stub these would be equal.
let _c: Int = check(near(a2 - a1, 3.0), "different signals produce different geometry")
let ff1: Int = geometry_free(g1)
let ff2: Int = geometry_free(g2)
println("the registry keys on modality")
let r2: Int = realizer_register("pulse", "pulse_realizer")
let _c: Int = check(r2, "a second modality registers independently")
let mt: Geometry = transduce("aaa", "tone")
let mp: Geometry = transduce("aaa", "pulse")
let mdt: Int = geometry_dim(mt)
let mdp: Int = geometry_dim(mp)
let _c: Int = check(eq_int(mdt, 4), "tone still routes to its own realizer")
let _c: Int = check(eq_int(mdp, 2), "pulse routes to a different realizer")
let ffm1: Int = geometry_free(mt)
let ffm2: Int = geometry_free(mp)
println("no organ is reported as no organ")
// A modality with no realizer must transduce to NOTHING. It must never
// fall back to embedding a description of the signal and calling that
// perception that silent substitution is the defect this all exists to end.
let eh: Int = realizer_has("echolocation")
let _c: Int = check(1 - eh, "unregistered modality has no organ")
let ge: Geometry = transduce("anything", "echolocation")
let gei: Int = geometry_is(ge)
let _c: Int = check(1 - gei, "no realizer means NO geometry, not fake geometry")
println("an unresolvable realizer name fails at WIRING time")
let bad: Int = realizer_register("ghost", "no_such_function_anywhere")
let _c: Int = check(1 - bad, "unresolvable realizer name is a registration failure")
let gh2: Int = realizer_has("ghost")
let _c: Int = check(1 - gh2, "and nothing gets registered")
println("a realizer returning non-geometry transduces nothing")
let rb: Int = realizer_register("bogus", "bogus_realizer")
let _c: Int = check(rb, "the symbol resolves, so registration succeeds")
let gb: Geometry = transduce("x", "bogus")
let gbi: Int = geometry_is(gb)
let _c: Int = check(1 - gbi, "contract enforced at the boundary: nothing handed back")
println("norm lets a caller check a realizer emitted signal, not zeros")
let gn: Geometry = geometry_new(2)
let _c: Int = check(near(geometry_norm(gn), 0.0), "a fresh geometry is zero — norm says so")
let n0: Int = geometry_set(gn, 0, 3.0)
let n1: Int = geometry_set(gn, 1, 4.0)
let _c: Int = check(near(geometry_norm(gn), 5.0), "3-4-5: norm is 5")
let ffn: Int = geometry_free(gn)
// Reaching here means nothing called exit(1) along the way.
println("")
println("all checks passed")
}
+670 -77
View File
@@ -43,6 +43,7 @@
#include <dlfcn.h> /* dlsym for http_set_handler fallback */
#include <unistd.h>
#include <fcntl.h>
#include <sys/file.h> /* flock — process-identity singleton (program block) */
#include <dirent.h>
#include <errno.h>
#include <pthread.h>
@@ -3478,28 +3479,74 @@ static void jb_puts(JsonBuf* b, const char* s) {
b->buf[b->len] = '\0';
}
/* UTF-8 VALIDITY IS THE EMITTER'S CONTRACT (2026-08-16 self-review).
*
* This copied every byte >= 0x20 through verbatim, so a malformed sequence
* anywhere in the store became malformed output. Measured against the live
* graph: three nodes carry labels truncated to exactly 80 bytes ending in a
* lone 0xE2 the first byte of an em-dash, cut mid-sequence by some producer
* that is NOT this runtime (no 80-byte truncation exists here; the content
* itself is 2572 and 2746 bytes). Those three nodes made the ENTIRE 26 MB
* /api/nodes/list response undecodable, so a strict parser could not read the
* graph at all.
*
* Fixing only the writer would not have helped: the store already contains the
* damage, and it accepts data from importers, other producers and older
* binaries. A serializer that promises JSON owes valid UTF-8 regardless of what
* it is handed so validate here, at the boundary that makes the promise.
* Invalid bytes become U+FFFD rather than being dropped, so damage stays
* visible in the output instead of being silently papered over.
*
* Well-formed input is byte-identical to before: valid sequences are copied
* verbatim, and only structurally invalid ones (bad lead byte, missing or bad
* continuation, overlong encoding, UTF-16 surrogate, or > U+10FFFF) are
* replaced. */
static void jb_emit_escaped(JsonBuf* b, const char* s) {
jb_putc(b, '"');
for (; *s; s++) {
unsigned char c = (unsigned char)*s;
const unsigned char* p = (const unsigned char*)s;
while (*p) {
unsigned char c = *p;
switch (c) {
case '"': jb_puts(b, "\\\""); break;
case '\\': jb_puts(b, "\\\\"); break;
case '\b': jb_puts(b, "\\b"); break;
case '\f': jb_puts(b, "\\f"); break;
case '\n': jb_puts(b, "\\n"); break;
case '\r': jb_puts(b, "\\r"); break;
case '\t': jb_puts(b, "\\t"); break;
default:
if (c < 0x20) {
char tmp[8];
snprintf(tmp, sizeof(tmp), "\\u%04x", c);
jb_puts(b, tmp);
} else {
jb_putc(b, (char)c);
}
break;
case '"': jb_puts(b, "\\\""); p++; continue;
case '\\': jb_puts(b, "\\\\"); p++; continue;
case '\b': jb_puts(b, "\\b"); p++; continue;
case '\f': jb_puts(b, "\\f"); p++; continue;
case '\n': jb_puts(b, "\\n"); p++; continue;
case '\r': jb_puts(b, "\\r"); p++; continue;
case '\t': jb_puts(b, "\\t"); p++; continue;
default: break;
}
if (c < 0x20) {
char tmp[8];
snprintf(tmp, sizeof(tmp), "\\u%04x", c);
jb_puts(b, tmp);
p++;
continue;
}
if (c < 0x80) { jb_putc(b, (char)c); p++; continue; }
/* Multi-byte: validate the whole sequence before emitting any of it. */
int len; unsigned int cp;
if ((c & 0xE0) == 0xC0) { len = 2; cp = c & 0x1Fu; }
else if ((c & 0xF0) == 0xE0) { len = 3; cp = c & 0x0Fu; }
else if ((c & 0xF8) == 0xF0) { len = 4; cp = c & 0x07u; }
else { jb_puts(b, "\\ufffd"); p++; continue; }
int ok = 1;
for (int i = 1; i < len; i++) {
if ((p[i] & 0xC0) != 0x80) { ok = 0; break; } /* also catches NUL */
cp = (cp << 6) | (unsigned int)(p[i] & 0x3F);
}
if (ok) {
if (len == 2 && cp < 0x80) ok = 0; /* overlong */
else if (len == 3 && cp < 0x800) ok = 0; /* overlong */
else if (len == 4 && cp < 0x10000) ok = 0; /* overlong */
else if (cp >= 0xD800 && cp <= 0xDFFF) ok = 0; /* UTF-16 surrogate */
else if (cp > 0x10FFFF) ok = 0; /* out of range */
}
if (!ok) { jb_puts(b, "\\ufffd"); p++; continue; }
for (int i = 0; i < len; i++) jb_putc(b, (char)p[i]);
p += len;
}
jb_putc(b, '"');
}
@@ -5516,6 +5563,45 @@ el_val_t str_count(el_val_t sv, el_val_t subv) {
return (el_val_t)count;
}
/* el_utf8_safe_len — the largest byte length <= max_bytes that does NOT split a
* UTF-8 codepoint.
*
* WHY (2026-08-16 self-review): engram_first_n_chars truncated with a plain
* `if (l > n) l = n; memcpy(...)`, i.e. by BYTES despite its name. Any content
* carrying a multi-byte character across the 60-byte boundary produced a label
* ending in a half codepoint. That label is copied verbatim into every JSON
* document containing the node, so a single such node makes the WHOLE response
* invalid UTF-8 /api/nodes/list failed to decode at byte 89261 against the
* live store, which breaks any strict parser reading the graph.
*
* This lives beside str_count_chars rather than in the engram because the rest
* of el's string layer is already codepoint-aware (str_count_chars counts
* codepoints, str_reverse walks codepoint lengths). Byte-truncation was the
* outlier, and the concern is a string concern. Bounded by BYTES, not
* codepoints, so existing labels never grow only stop splitting.
*
* A lead byte with no room for its full sequence is dropped entirely; a stray
* continuation byte (already-invalid input) is passed through unchanged rather
* than silently repaired, so this never manufactures data. */
size_t el_utf8_safe_len(const char* s, size_t max_bytes) {
if (!s) return 0;
size_t len = strlen(s);
if (len <= max_bytes) return len;
size_t i = 0;
while (i < max_bytes) {
unsigned char c = (unsigned char)s[i];
size_t cp_len;
if ((c & 0x80) == 0x00) cp_len = 1;
else if ((c & 0xE0) == 0xC0) cp_len = 2;
else if ((c & 0xF0) == 0xE0) cp_len = 3;
else if ((c & 0xF8) == 0xF0) cp_len = 4;
else cp_len = 1; /* stray continuation: passthrough */
if (i + cp_len > max_bytes) break; /* would split — stop before it */
i += cp_len;
}
return i;
}
/* Codepoint count: walk bytes, count those NOT matching 10xxxxxx. */
el_val_t str_count_chars(el_val_t sv) {
const char* s = EL_CSTR(sv);
@@ -5959,6 +6045,308 @@ void el_cgi_init(el_val_t name, el_val_t dharma_id, el_val_t principal,
}
/* ── Geometry: signal as a first-class el value ──────────────────────────────
*
* WHY THIS IS IN THE LANGUAGE, AND WHY IT IS DEFINED HERE (2026-08-16).
*
* Until yesterday no El ingest path could carry a vector. Nodes took text,
* and geometry was DERIVED from that text by engram_embed_backfill. Text was
* therefore the mandatory entry medium: any non-text modality audio, image,
* sensor had to be DESCRIBED in prose first, so the geometry we then
* reasoned over was the geometry OF THE DESCRIPTION, not of the signal. That
* is faking it. The architecture is: geometry in, always; we do not fake it,
* we project.
*
* The first fix (#141, engram_node_set_emb) proved the path end to end but
* placed it wrong in three ways, each of which this section corrects:
*
* 1. It sat at the CONSUMER. Transduction is a LANGUAGE concern every El
* program touching any modality needs it, not just the one that happens
* to hold a graph. So this section is defined HERE, immediately above
* the engram block, and depends on nothing inside it. The engram is a
* client of this surface, not its owner. That ordering is the point:
* you can delete the entire engram and geometry still enters El.
*
* 2. It marshalled the vector as a hex STRING, because El had no
* first-class geometry value which reintroduced text as the TRANSPORT
* medium one layer below the problem being fixed. Geometry is now a
* value. Hex survives only as a wire ADAPTER at the edge
* (geometry_from/to_f32le_hex), which is all an encoding should ever be.
*
* 3. It needed an arbitrary `dim <= 8192` bound, purely to check a
* caller-supplied dim against a string's length before allocating. A
* real geometry value CARRIES its own width, so here the width is
* derived and never asserted, and there is nothing left to validate.
* The bound is gone rather than merely raised the only thing that can
* fail is the allocation itself, which is an honest failure.
*
* REPRESENTATION: magic-tagged heap object (see "Refcounted heap objects"),
* carried in an el_val_t. The payload is a separate allocation so the header
* never moves. The magic word is >= 0x80 in its MSB so the string/small-int
* sniffing in looks_like_heap_obj can never confuse a Geometry for either.
*
* OWNERSHIP: a Geometry is owned by the El caller and released with
* geometry_free. node_attach_geometry COPIES its payload into the node, so a
* node and the caller's value have independent lifetimes and freeing one
* never touches the other. Geometry deliberately does NOT participate in
* el_retain/el_release: the shipped elc emits neither on let-bindings
* (measured), so hooking it there would be dead code that could only ever
* free a live vector early.
*/
#define EL_MAGIC_GEOM 0xE1608E01u
typedef struct {
ElHeader hdr;
int32_t dim;
float* v;
} ElGeometry;
/* Resolve an el_val_t to a live Geometry, or NULL. Every accessor goes
* through this, so a stale/foreign/zero value is a clean 0-return rather
* than a dereference. */
static ElGeometry* geom_of(el_val_t g) {
if (!looks_like_heap_obj(g)) return NULL;
ElGeometry* p = (ElGeometry*)(uintptr_t)g;
if (p->hdr.magic != EL_MAGIC_GEOM) return NULL;
return p;
}
el_val_t geometry_new(el_val_t dim) {
int32_t d = (int32_t)(int64_t)dim;
if (d <= 0) return (el_val_t)0;
ElGeometry* g = (ElGeometry*)malloc(sizeof(ElGeometry));
if (!g) return (el_val_t)0;
g->v = (float*)calloc((size_t)d, sizeof(float));
if (!g->v) { free(g); return (el_val_t)0; }
g->hdr.magic = EL_MAGIC_GEOM;
g->hdr.refcount = 1;
g->dim = d;
return (el_val_t)(uintptr_t)g;
}
el_val_t geometry_dim(el_val_t g) {
ElGeometry* p = geom_of(g);
return p ? (el_val_t)p->dim : (el_val_t)0;
}
el_val_t geometry_is(el_val_t g) {
return geom_of(g) ? (el_val_t)1 : (el_val_t)0;
}
el_val_t geometry_get(el_val_t g, el_val_t i) {
ElGeometry* p = geom_of(g);
int64_t k = (int64_t)i;
if (!p || k < 0 || k >= (int64_t)p->dim) return el_from_float(0.0);
return el_from_float((double)p->v[k]);
}
el_val_t geometry_set(el_val_t g, el_val_t i, el_val_t x) {
ElGeometry* p = geom_of(g);
int64_t k = (int64_t)i;
if (!p || k < 0 || k >= (int64_t)p->dim) return (el_val_t)0;
p->v[k] = (float)el_to_float(x);
return (el_val_t)1;
}
el_val_t geometry_norm(el_val_t g) {
ElGeometry* p = geom_of(g);
if (!p) return el_from_float(0.0);
double s = 0.0;
for (int32_t i = 0; i < p->dim; i++) s += (double)p->v[i] * (double)p->v[i];
return el_from_float(sqrt(s));
}
el_val_t geometry_free(el_val_t g) {
ElGeometry* p = geom_of(g);
if (!p) return (el_val_t)0;
free(p->v);
p->hdr.magic = 0; /* poison so use-after-free is detected, as List/Map do */
free(p);
return (el_val_t)1;
}
/* geometry_from_f32le_hex — decode little-endian float32 hex INTO geometry.
*
* This is the ONE place hex appears, and it appears as what it actually is:
* an encoding at the boundary, not the medium El reasons in. The width is
* DERIVED from the input length (8 hex chars per float32) and never supplied
* by the caller which is precisely why #141's arbitrary `dim <= 8192`
* bound has no counterpart here. There is nothing to validate.
*
* Returns 0 on empty input, a length that is not a multiple of 8, or any
* non-hex character. */
el_val_t geometry_from_f32le_hex(el_val_t hex) {
const char* s = EL_CSTR(hex);
if (!s) return (el_val_t)0;
size_t n = strlen(s);
if (n == 0 || (n % 8u) != 0) return (el_val_t)0;
size_t d = n / 8u;
if (d > (size_t)INT32_MAX) return (el_val_t)0;
el_val_t gv = geometry_new((el_val_t)(int64_t)d);
ElGeometry* g = geom_of(gv);
if (!g) return (el_val_t)0;
for (size_t i = 0; i < d; i++) {
uint32_t w = 0;
for (int k = 0; k < 8; k++) {
char c = s[i * 8u + (size_t)k];
uint32_t nib;
if (c >= '0' && c <= '9') nib = (uint32_t)(c - '0');
else if (c >= 'a' && c <= 'f') nib = (uint32_t)(c - 'a' + 10);
else if (c >= 'A' && c <= 'F') nib = (uint32_t)(c - 'A' + 10);
else { geometry_free(gv); return (el_val_t)0; }
w = (w << 4) | nib;
}
/* Hex is emitted little-endian byte order; rebuild the word. */
uint32_t le = ((w & 0x000000FFu) << 24) | ((w & 0x0000FF00u) << 8) |
((w & 0x00FF0000u) >> 8) | ((w & 0xFF000000u) >> 24);
float f;
memcpy(&f, &le, sizeof(f));
g->v[i] = f;
}
return gv;
}
/* geometry_to_f32le_hex — the egress adapter, exact inverse of the above.
* Present so a program that must hand geometry to a non-El peer over a text
* wire can do so explicitly, at the edge, instead of the language pretending
* text was the medium all along. */
el_val_t geometry_to_f32le_hex(el_val_t g) {
ElGeometry* p = geom_of(g);
if (!p) return EL_STR("");
static const char* HEXD = "0123456789abcdef";
size_t n = (size_t)p->dim * 8u;
char* out = el_strbuf(n); /* arena-tracked; allocates n+1, exits on OOM */
for (int32_t i = 0; i < p->dim; i++) {
uint32_t w;
memcpy(&w, &p->v[i], sizeof(w));
/* Emit little-endian byte order: low byte first. */
for (int b = 0; b < 4; b++) {
uint32_t byte = (w >> (8 * b)) & 0xFFu;
out[(size_t)i * 8u + (size_t)b * 2u] = HEXD[(byte >> 4) & 0xF];
out[(size_t)i * 8u + (size_t)b * 2u + 1] = HEXD[byte & 0xF];
}
}
out[n] = '\0';
return (el_val_t)(uintptr_t)out;
}
/* ── Realizers: transduction declared in El, not patched into the runtime ────
*
* A REALIZER maps one modality into geometry. The whole reason transduction
* belongs in the language is that ADDING A MODALITY MUST NOT REQUIRE A
* RUNTIME PATCH otherwise "the realizers are in the engram" just becomes
* "the realizers are in the runtime" and nothing has actually moved. So
* realizers are declared in El and registered by NAME:
*
* fn tone_realizer(signal: String) -> Geometry {
* let g: Geometry = geometry_new(8)
* ... geometry_set(g, i, x) ...
* g
* }
*
* realizer_register("tone", "tone_realizer")
* let g: Geometry = transduce(sample, "tone")
*
* The namesymbol step rides the identical, already load-bearing mechanism
* http_set_handler uses (see "HTTP server"): every El `fn name(...)` compiles
* to a global C symbol with that exact name, so dlsym(RTLD_DEFAULT, name)
* against the running binary resolves an El-defined function. No codegen
* change, no first-class function references, no runtime edit per modality.
* A realizer written in El is a first-class realizer.
*
* A realizer may equally be a C symbol linked into the program; the registry
* cannot tell the difference and has no reason to care.
*/
typedef el_val_t (*el_realizer_fn)(el_val_t);
typedef struct {
char* modality;
el_realizer_fn fn;
} ElRealizer;
static ElRealizer _realizers[64];
static size_t _realizer_count = 0;
static pthread_mutex_t _realizer_mu = PTHREAD_MUTEX_INITIALIZER;
static el_realizer_fn realizer_lookup(const char* m) {
el_realizer_fn out = NULL;
pthread_mutex_lock(&_realizer_mu);
for (size_t i = 0; i < _realizer_count; i++) {
if (strcmp(_realizers[i].modality, m) == 0) { out = _realizers[i].fn; break; }
}
pthread_mutex_unlock(&_realizer_mu);
return out;
}
el_val_t realizer_register(el_val_t modality, el_val_t fn_name) {
const char* m = EL_CSTR(modality);
const char* fn = EL_CSTR(fn_name);
if (!m || !*m || !fn || !*fn) return (el_val_t)0;
/* An unresolvable name is a REGISTRATION FAILURE, reported as 0 — not a
* silent no-op that only surfaces later as "this modality produces
* nothing". Distinguishing "no organ" from "broken organ" at the moment
* of wiring is the lesson #141 was written to enforce. */
void* sym = dlsym(RTLD_DEFAULT, fn);
if (!sym) return (el_val_t)0;
pthread_mutex_lock(&_realizer_mu);
for (size_t i = 0; i < _realizer_count; i++) {
if (strcmp(_realizers[i].modality, m) == 0) {
_realizers[i].fn = (el_realizer_fn)sym; /* re-registration replaces */
pthread_mutex_unlock(&_realizer_mu);
return (el_val_t)1;
}
}
if (_realizer_count < sizeof(_realizers) / sizeof(_realizers[0])) {
/* _persist, NOT el_strdup: the registry outlives any request, and an
* arena-tracked copy would be freed at el_request_end leaving a
* dangling modality name if a program registers a realizer from
* inside a handler rather than at startup. */
_realizers[_realizer_count].modality = el_strdup_persist(m);
_realizers[_realizer_count].fn = (el_realizer_fn)sym;
_realizer_count++;
pthread_mutex_unlock(&_realizer_mu);
return (el_val_t)1;
}
pthread_mutex_unlock(&_realizer_mu);
return (el_val_t)0;
}
el_val_t realizer_has(el_val_t modality) {
const char* m = EL_CSTR(modality);
if (!m || !*m) return (el_val_t)0;
return realizer_lookup(m) ? (el_val_t)1 : (el_val_t)0;
}
/* transduce — THE primitive: signal in, geometry out.
*
* Dispatches to the realizer registered for `modality`. Returns 0 (not a
* Geometry) when no realizer is registered, and geometry_is() on the result
* is the check.
*
* There is deliberately NO built-in realizer, not even for text. A modality
* the program has declared no organ for is one it genuinely cannot sense,
* and returning nothing is more honest than quietly embedding a description
* of the signal and calling that perception which is the exact failure
* this whole change exists to end.
*
* The result is validated to actually BE a Geometry before it is handed
* back, so a realizer that returns something else transduced nothing rather
* than handing a caller a value that will misbehave far from here. */
el_val_t transduce(el_val_t signal, el_val_t modality) {
const char* m = EL_CSTR(modality);
if (!m || !*m) return (el_val_t)0;
el_realizer_fn fn = realizer_lookup(m);
if (!fn) return (el_val_t)0;
el_val_t g = fn(signal);
return geom_of(g) ? g : (el_val_t)0;
}
/* ── Batch 3: Engram in-process graph store ──────────────────────────────── */
/*
* Single global EngramStore allocated lazily on first call. All node and
@@ -7714,10 +8102,14 @@ static double engram_decode_score(el_val_t v) {
return (double)n;
}
/* Truncate to at most n BYTES without splitting a UTF-8 codepoint. The old
* implementation was `if (l > n) l = n;` a byte cut that could land inside a
* multi-byte character and emit a half codepoint into the node's label, which
* then propagated into every JSON document containing that node. See
* el_utf8_safe_len for the measurement. */
static char* engram_first_n_chars(const char* s, size_t n) {
if (!s) return el_strdup("");
size_t l = strlen(s);
if (l > n) l = n;
size_t l = el_utf8_safe_len(s, n);
char* out = el_strbuf(l);
memcpy(out, s, l);
out[l] = '\0';
@@ -8563,80 +8955,96 @@ el_val_t engram_node_count(void) {
return (el_val_t)engram_get()->node_count;
}
/* engram_node_set_emb — attach GEOMETRY to an existing node.
/* node_attach_geometry — a node acquires geometry.
*
* WHY THIS EXISTS (2026-08-16). Until now no ingest path could carry a
* vector. engram_node / engram_node_full / engram_node_layered take text
* only, and the sole way a node acquired an embedding was
* engram_embed_backfill DERIVING one from n->content. That made text the
* mandatory entry medium: any non-text modality (audio, image, sensor)
* had to be described in prose first, and the geometry we then reasoned
* over was the geometry OF THE DESCRIPTION, not of the signal. Measured
* consequence: POST /api/nodes accepted an "emb" field, returned 200 with
* a fresh id, and stored emb_dim=None / embedded:false the vector was
* silently discarded because no parameter existed to receive it.
* Named for the operation, not for the store that happens to hold the node.
* This is the geometry-valued ingest path that replaces #141's hex-string
* one: nothing here parses text, and nothing here takes a caller's word for
* how wide the vector is. The Geometry carries its own width.
*
* `hex` is little-endian float32, the encoding the perception vessel's
* /voice/embed already emits, so a realizer's output moves in without a
* JSON float-array round trip. Length must be exactly dim*8 hex chars.
* The payload is COPIED into the node, so the node and the caller's Geometry
* have independent lifetimes the caller may geometry_free() immediately
* after, and a later free of the node's emb never touches the El value.
*
* DIMENSION POLICY: dim need NOT equal the canonical text-embedding dim.
* A modality vector of a different width is stored and is simply not
* inserted into the resident HNSW index, whose build loop already filters
* on `n->emb_dim == dim`. So off-dimension geometry is durable and
* addressable without perturbing the canonical index.
* DIMENSION POLICY (measured in #141, load-bearing do not regress): dim
* need NOT equal the canonical text-embedding width. An off-dimension vector
* is stored and is simply not inserted into the resident HNSW index, whose
* build loop already filters on `n->emb_dim == dim`. So a 64-dim voice
* geometry is durable and addressable without perturbing the 768-dim
* canonical index.
*
* Setting emb also makes the node ineligible for embed_backfill (which
* only fills nodes with no emb), so a realizer's vector is never
* Attaching geometry also makes the node ineligible for embed_backfill
* (which fills only nodes with no emb), so a realizer's vector is never
* overwritten by a text-derived one.
*
* Returns 1 on success, 0 on unknown id / malformed hex / bad dim. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim) {
const char* sid = EL_CSTR(id);
const char* sh = EL_CSTR(hex);
int32_t d = (int32_t)(int64_t)dim;
/* Bound the allocation. No max-dim constant existed because no caller
* could supply a dim before this function; 8192 is generous for any
* realizer (canonical text embeddings are 768, MFCC voice stats 64)
* while keeping a malformed `dim` from requesting an unbounded malloc. */
if (!sid || !*sid || !sh || d <= 0 || d > 8192) return (el_val_t)0;
* Returns 1 on success, 0 on unknown id or a value that is not a Geometry. */
el_val_t node_attach_geometry(el_val_t node_id, el_val_t g) {
const char* sid = EL_CSTR(node_id);
if (!sid || !*sid) return (el_val_t)0;
size_t need = (size_t)d * 8u; /* 4 bytes → 8 hex chars per float */
if (strlen(sh) != need) return (el_val_t)0;
ElGeometry* p = geom_of(g);
if (!p || p->dim <= 0) return (el_val_t)0;
EngramNode* n = engram_find_node(sid);
if (!n) return (el_val_t)0;
float* v = (float*)malloc(sizeof(float) * (size_t)d);
float* v = (float*)malloc(sizeof(float) * (size_t)p->dim);
if (!v) return (el_val_t)0;
for (int32_t i = 0; i < d; i++) {
uint32_t w = 0;
for (int k = 0; k < 8; k++) {
char c = sh[(size_t)i * 8u + (size_t)k];
uint32_t nib;
if (c >= '0' && c <= '9') nib = (uint32_t)(c - '0');
else if (c >= 'a' && c <= 'f') nib = (uint32_t)(c - 'a' + 10);
else if (c >= 'A' && c <= 'F') nib = (uint32_t)(c - 'A' + 10);
else { free(v); return (el_val_t)0; }
w = (w << 4) | nib;
}
/* Hex is emitted little-endian byte order; rebuild the word. */
uint32_t le = ((w & 0x000000FFu) << 24) | ((w & 0x0000FF00u) << 8) |
((w & 0x00FF0000u) >> 8) | ((w & 0xFF000000u) >> 24);
float f;
memcpy(&f, &le, sizeof(f));
v[i] = f;
}
memcpy(v, p->v, sizeof(float) * (size_t)p->dim);
free(n->emb);
n->emb = v;
n->emb_dim = d;
n->emb = v;
n->emb_dim = p->dim;
n->updated_at = engram_now_ms();
if (engram_store_enabled()) eg_store_put_node(n);
return (el_val_t)1;
}
/* node_geometry_dim — read the attached width back, 0 if the node carries
* none. Exists so an attach is VERIFIED by reading it back rather than by
* trusting a success return. That is not a nicety: #141 was misdiagnosed for
* an hour precisely because a genuine ingest drop and a mere reporting gap
* were indistinguishable from the outside. */
el_val_t node_geometry_dim(el_val_t node_id) {
const char* sid = EL_CSTR(node_id);
if (!sid || !*sid) return (el_val_t)0;
EngramNode* n = engram_find_node(sid);
if (!n || !n->emb) return (el_val_t)0;
return (el_val_t)n->emb_dim;
}
/* engram_node_set_emb — DEPRECATED. Shipped in #141; superseded 2026-08-16
* by geometry_from_f32le_hex + node_attach_geometry, and now implemented as
* literally that.
*
* It is kept, rather than removed, for one reason only: the runtime is
* published as an SDK asset, so a downstream binary may already be linking
* this symbol. It is NOT kept because a hex string is an acceptable way to
* move geometry between two pieces of El it isn't, and that was the
* placement defect. New code calls transduce() or geometry_from_f32le_hex()
* plus node_attach_geometry().
*
* The #141 contract is preserved exactly, including its negative cases, so
* this remains a drop-in: `dim` <= 0 rejects, malformed hex rejects, and a
* `dim` that disagrees with the vector's actual width rejects. The
* difference is that `dim` is now an ASSERTION checked against a width the
* Geometry already knows, rather than the authority the allocation trusted
* which is why #141's arbitrary `dim <= 8192` guard has no counterpart here.
* There is no longer an unbounded-malloc hazard to guard against. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim) {
int32_t want = (int32_t)(int64_t)dim;
if (want <= 0) return (el_val_t)0;
el_val_t gv = geometry_from_f32le_hex(hex);
ElGeometry* p = geom_of(gv);
if (!p) return (el_val_t)0; /* empty / malformed hex */
if (p->dim != want) { geometry_free(gv); return (el_val_t)0; } /* length mismatch */
el_val_t ok = node_attach_geometry(id, gv);
geometry_free(gv);
return ok;
}
/* ── Telemetry retention ────────────────────────────────────────────────────
* (2026-07-16 self-review) InternalStateEvent nodes are append-only telemetry
* (heartbeat, curiosity_scan, engram_sync) written ~3/min by the awareness
@@ -18418,11 +18826,196 @@ void log_warn(el_val_t msg_v) {
fprintf(stderr, "[WARN] %s\n", msg ? msg : "");
}
/* config — read a configuration value from the environment.
* Returns "" if the variable is not set (same as __env_get). */
/* ── Cross-cutting concerns: process identity and configuration ──────────────
*
* These back the `program` block (see lang/spec/language.md §18). Both concerns
* were previously conventions "check nothing is already running first",
* "remember the right default at every read site" and conventions is exactly
* what they failed as. Here they are mechanisms, injected by the compiler at
* the process boundary, so no call site has to remember anything.
*/
/* -- Process identity ------------------------------------------------------- */
/* The lock fd is deliberately never closed. Holding it open for the process
* lifetime is what makes the guarantee work: the kernel drops an flock when the
* owning process dies, including on SIGKILL and on crash. That is why this is an
* flock and not a bare pidfile there is no stale-lock state to clean up, and
* therefore no "delete the pidfile to get unstuck" ritual that would itself
* become a convention. */
static int el_singleton_fd = -1;
static char el_singleton_path[1024];
static const char* el_singleton_dir(void) {
const char* d = getenv("EL_SINGLETON_DIR");
if (d && *d) return d;
d = getenv("TMPDIR");
if (d && *d) return d;
return "/tmp";
}
/* el_singleton_acquire — claim exclusive process identity, or refuse to start.
* Compiler-injected as the FIRST statement of main() for any program whose
* `program` block declares `singleton:`. */
el_val_t el_singleton_acquire(el_val_t id_v) {
const char* id = EL_CSTR(id_v);
if (!id || !*id) return EL_NULL;
/* Sanitise the id into a filename. */
char safe[256];
size_t si = 0;
for (const char* p = id; *p && si + 1 < sizeof(safe); p++) {
char c = *p;
int ok = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
|| (c >= '0' && c <= '9') || c == '-' || c == '_' || c == '.';
safe[si++] = (char)(ok ? c : '-');
}
safe[si] = '\0';
snprintf(el_singleton_path, sizeof(el_singleton_path),
"%s/el-singleton-%s.lock", el_singleton_dir(), safe);
int fd = open(el_singleton_path, O_RDWR | O_CREAT, 0644);
if (fd < 0) {
fprintf(stderr, "[el] FATAL: singleton '%s': cannot open lock file %s: %s\n",
id, el_singleton_path, strerror(errno));
exit(1);
}
if (flock(fd, LOCK_EX | LOCK_NB) != 0) {
/* Someone else holds it. Report WHO. A pid is actionable; "already
* running" is not — and the observed failure was precisely a stale
* process that `pkill -f` had silently failed to match, still answering
* probes while a fresh build was believed to be under test. */
char buf[64];
buf[0] = '\0';
ssize_t n = pread(fd, buf, sizeof(buf) - 1, 0);
if (n > 0) buf[n] = '\0';
long holder = strtol(buf, NULL, 10);
fprintf(stderr, "[el] FATAL: another instance of '%s' is already running", id);
if (holder > 0) fprintf(stderr, " (pid %ld)", holder);
fprintf(stderr, ".\n"
"[el] lock: %s\n"
"[el] Refusing to start a second instance against the same\n"
"[el] state. Stop the running one and VERIFY it is gone\n"
"[el] (ps -p <pid>) before retrying.\n",
el_singleton_path);
close(fd);
exit(1);
}
/* We own it. Record our pid so the next would-be starter can name us. */
if (ftruncate(fd, 0) != 0) { /* best effort — the lock is the guarantee */ }
char pidbuf[32];
int pn = snprintf(pidbuf, sizeof(pidbuf), "%ld\n", (long)getpid());
if (pn > 0) { ssize_t w = write(fd, pidbuf, (size_t)pn); (void)w; }
el_singleton_fd = fd; /* never closed, by design */
return EL_NULL;
}
/* -- Configuration ---------------------------------------------------------- */
#define EL_CONFIG_MAX 128
typedef struct {
char name[128];
char type[16];
char* value; /* resolved: env value, else default; NULL if unset */
int has_default;
int required;
} ElConfigEntry;
static ElConfigEntry el_config_tab[EL_CONFIG_MAX];
static int el_config_n = 0;
static int el_config_has_schema = 0; /* did this program declare one at all? */
static int el_config_is_int(const char* s) {
if (!s || !*s) return 0;
if (*s == '-' || *s == '+') s++;
if (!*s) return 0;
for (; *s; s++) if (*s < '0' || *s > '9') return 0;
return 1;
}
/* el_config_declare — record ONE configuration entry and resolve it now.
* The default lives here, in the declaration, and nowhere else. */
el_val_t el_config_declare(el_val_t name_v, el_val_t type_v, el_val_t def_v,
el_val_t has_default_v, el_val_t required_v) {
const char* name = EL_CSTR(name_v);
if (!name || !*name) return EL_NULL;
el_config_has_schema = 1;
if (el_config_n >= EL_CONFIG_MAX) {
fprintf(stderr, "[el] FATAL: more than %d config entries declared.\n", EL_CONFIG_MAX);
exit(1);
}
const char* type = EL_CSTR(type_v);
const char* def = (def_v == EL_NULL) ? NULL : EL_CSTR(def_v);
ElConfigEntry* e = &el_config_tab[el_config_n++];
snprintf(e->name, sizeof(e->name), "%s", name);
snprintf(e->type, sizeof(e->type), "%s", type ? type : "String");
e->has_default = (int)(long)has_default_v;
e->required = (int)(long)required_v;
/* Resolution order: environment wins, declaration supplies the fallback. */
const char* env = getenv(name);
if (env && *env) e->value = el_strdup_persist(env);
else if (e->has_default && def) e->value = el_strdup_persist(def);
else e->value = NULL;
return EL_NULL;
}
/* el_config_validate — check the whole schema at once, before main() runs.
* Reports EVERY problem, not just the first: a startup that fails one variable
* at a time costs one restart per variable. */
el_val_t el_config_validate(el_val_t program_v) {
const char* prog = EL_CSTR(program_v);
int bad = 0;
for (int i = 0; i < el_config_n; i++) {
ElConfigEntry* e = &el_config_tab[i];
if (!e->value) {
if (e->required) {
fprintf(stderr, "[el] config: %s is required but is not set "
"(no value in the environment, no default declared)\n", e->name);
bad++;
}
continue;
}
if (strcmp(e->type, "Int") == 0 && !el_config_is_int(e->value)) {
fprintf(stderr, "[el] config: %s is declared Int but its value is \"%s\"\n",
e->name, e->value);
bad++;
}
}
if (bad) {
fprintf(stderr, "[el] FATAL: program '%s' has %d invalid configuration "
"entr%s. Refusing to start.\n",
prog ? prog : "?", bad, bad == 1 ? "y" : "ies");
exit(1);
}
return EL_NULL;
}
/* config — read a configuration value.
*
* When the program declared a schema, that schema is authoritative: the value
* has already been resolved and validated at startup, so this is a lookup and
* NOT a place where a default gets decided. Reading a key that was never
* declared is a bug at the read site, and is reported as one that enforcement
* is what makes the declaration real rather than advisory.
*
* With no schema declared, behaviour is unchanged (plain getenv), so programs
* that have not migrated keep working. */
el_val_t config(el_val_t key_v) {
const char* key = EL_CSTR(key_v);
if (!key || !*key) return EL_STR("");
if (el_config_has_schema) {
for (int i = 0; i < el_config_n; i++) {
if (strcmp(el_config_tab[i].name, key) == 0) {
const char* v = el_config_tab[i].value;
return el_wrap_str(el_strdup(v ? v : ""));
}
}
fprintf(stderr, "[el] FATAL: config(\"%s\") is not declared in the "
"program block. Declare it there, with its default, or stop "
"reading it.\n", key);
exit(1);
}
const char* val = getenv(key);
if (!val) return EL_STR("");
return el_wrap_str(el_strdup(val));
+90 -4
View File
@@ -586,6 +586,60 @@ void el_runtime_dharma_event_arrive(const char* event_type,
const char* payload,
const char* source);
/* ── Geometry: signal as a first-class El value ──────────────────────────────
*
* A Geometry is an opaque, magic-tagged heap value carried in an el_val_t
* the same discipline as List/Map. It holds a width and a float32 payload,
* and it is the medium a non-text modality enters in. Declared HERE, above
* the engram block, because transduction is a LANGUAGE concern: every El
* program touching any modality needs it, and the engram is merely one El
* program that happens to hold a graph. See el_runtime.c ("Geometry: signal
* as a first-class el value") for the full rationale.
*
* El-side type annotation is simply `Geometry` an opaque boxed pointer,
* exactly like Instant / Calendar / Rhythm. No codegen change is required.
*
* OWNERSHIP: a Geometry is owned by the El caller and released with
* geometry_free. node_attach_geometry COPIES, so a node and the caller's
* value have independent lifetimes. */
el_val_t geometry_new(el_val_t dim); /* zero-filled; 0 on failure */
el_val_t geometry_dim(el_val_t g); /* width, 0 if not a Geometry */
el_val_t geometry_is(el_val_t g); /* 1 if a live Geometry */
el_val_t geometry_get(el_val_t g, el_val_t i); /* Float component */
el_val_t geometry_set(el_val_t g, el_val_t i, el_val_t x); /* 1 ok / 0 out of range */
el_val_t geometry_norm(el_val_t g); /* Float L2 — lets a caller
* check a realizer emitted
* signal, not zeros */
el_val_t geometry_free(el_val_t g); /* 1 if freed, 0 if not a Geometry.
* Returns a value (not void) so it
* is safe in any El expression
* position without a codegen
* void-builtin table entry. */
/* Wire ADAPTERS — the only place an encoding appears, and only at the edge.
* `f32le hex` is little-endian float32, 8 hex chars per component: the
* encoding the perception vessel's /voice/embed already emits. The width is
* DERIVED from the input length, never supplied by a caller which is why
* there is no max-dim constant here to validate a claimed length against. */
el_val_t geometry_from_f32le_hex(el_val_t hex); /* 0 on empty/odd-length/non-hex */
el_val_t geometry_to_f32le_hex(el_val_t g); /* "" if not a Geometry */
/* ── Realizers + transduce ───────────────────────────────────────────────────
* A REALIZER maps one modality into geometry. Registration is by NAME, so a
* new modality never requires a runtime patch: every El `fn name(...)`
* compiles to a global C symbol with that exact name, and the registry
* resolves it with dlsym against the running binary the same mechanism
* http_set_handler already relies on.
*
* fn tone_realizer(signal: String) -> Geometry { ... }
* realizer_register("tone", "tone_realizer")
* let g: Geometry = transduce(sample, "tone")
*/
el_val_t realizer_register(el_val_t modality, el_val_t fn_name); /* 1 ok / 0 unresolved */
el_val_t realizer_has(el_val_t modality); /* 1 if a realizer is registered */
el_val_t transduce(el_val_t signal, el_val_t modality); /* Geometry, or 0 if no organ */
/* ── Engram local graph primitives ───────────────────────────────────────────
* Operate on the CGI's local Engram knowledge graph.
* `engram_activate` queries the local graph only; `dharma_activate` is
@@ -612,11 +666,27 @@ el_val_t engram_get_node(el_val_t id);
void engram_strengthen(el_val_t node_id);
void engram_forget(el_val_t node_id);
el_val_t engram_prune_telemetry(el_val_t older_than_ms);
/* Largest byte length <= max_bytes that does not split a UTF-8 codepoint.
* Bounded by bytes, not codepoints, so truncated strings never grow. */
size_t el_utf8_safe_len(const char* s, size_t max_bytes);
el_val_t engram_node_count(void);
/* Attach geometry to an existing node. `hex` is little-endian float32,
* exactly dim*8 hex chars the encoding realizers already emit. Lets a
* non-text modality enter as geometry instead of being described in prose
* and embedded as its description. Returns 1 on success, 0 otherwise. */
/* Attach a Geometry to an existing node, and read the attached width back.
* Named for the operation, not the store: a node acquires geometry. This is
* the geometry-valued ingest path nothing about it is hex, and nothing
* about it assumes the caller's vector matches the canonical text-embedding
* width. node_geometry_dim exists so an attach is VERIFIED by reading it
* back rather than by trusting a success return. */
el_val_t node_attach_geometry(el_val_t node_id, el_val_t g); /* 1 ok / 0 otherwise */
el_val_t node_geometry_dim(el_val_t node_id); /* width, 0 if none */
/* DEPRECATED (shipped in #141, superseded 2026-08-16). Equivalent to
* geometry_from_f32le_hex + node_attach_geometry, and now implemented as
* exactly that. Kept only so anything built against the #141 runtime keeps
* linking; `dim` is accepted but treated as an assertion about the vector's
* width rather than as its source. New code should not call this a hex
* string is a wire encoding, not a way to move geometry between two pieces
* of El. Returns 1 on success, 0 otherwise. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim);
el_val_t engram_search(el_val_t query, el_val_t limit);
el_val_t engram_scan_nodes(el_val_t limit, el_val_t offset);
@@ -957,6 +1027,22 @@ el_val_t __url_decode(el_val_t s);
/* Environment */
el_val_t __env_get(el_val_t key);
/* Cross-cutting concerns declared by a `program` block (spec §18).
* All three are COMPILER-INJECTED at the head of main() they are not meant to
* be written by hand, which is the point: the guarantee cannot be forgotten at a
* call site because there is no call site. */
el_val_t el_singleton_acquire(el_val_t id); /* §18.1 process identity */
el_val_t el_config_declare(el_val_t name, el_val_t type,
el_val_t deflt, el_val_t has_default,
el_val_t required); /* §18.2 config schema */
el_val_t el_config_validate(el_val_t program_name); /* §18.2 startup validate */
/* config(key) — the READ side, and the only one programs write by hand. With a
* schema declared it is a validated lookup; without one it degrades to getenv.
* (Defined in el_runtime.c but previously never prototyped here, so any program
* calling it failed to compile under -Werror=implicit-function-declaration.) */
el_val_t config(el_val_t key);
/* Subprocess */
el_val_t __exec(el_val_t cmd);
el_val_t __exec_bg(el_val_t cmd);
+251 -8
View File
@@ -29,6 +29,9 @@ This section is the **single source of truth** for what works and what is planne
- Lexer: keywords, identifiers, integer/float/string/bool literals, operators below.
- Parser: `let`, `return`, `fn`, `type`, `enum`, `import`, `from … import`, `while`, `for`, `if/else if/else`, `match`, `@decorator`, array/map literals, all listed operators, function calls, field access, index access, unary `!`/`-`, postfix `?`.
- Codegen: function definitions, top-level `main()`, all expression forms above, control flow, decorator-as-AST-attachment.
- Boundary seam: decorator arguments and stacking; VBD role enforcement via `#error`; `engram_boundary_beat` auto-emit at `@manager`/`@accessor` entry; `@route` dispatch tables (Section 9).
- Program-level declarative blocks: `cgi`, `service`, and `program` — the last carrying process identity and configuration (Section 18).
- **Geometry as a first-class value, and realizers declarable in El** — the `Geometry` type, the wire adapters, and `transduce` (Section 20). Landed 2026-08-16 (#141, #144).
- C runtime: I/O, string operations, integer math, lists, maps, filesystem, command-line args, basic `json_get` substring lookup.
### Planned (in flight)
@@ -37,11 +40,11 @@ This section is the **single source of truth** for what works and what is planne
- **Match codegen.** Currently parsed; codegen does not emit. Adding `({ ... })` statement-expression emission.
- **`?` propagation.** Currently no-op. Adding nil-propagation semantics.
- **`cgi` block parsing.** Currently lexed (`cgi` is a keyword) but not parsed as a statement. Adding `parse_cgi_block` and codegen of `el_cgi_init` at the head of `main()`.
- **VBD role enforcement.** `@manager`/`@engine`/`@accessor` are accepted as decorators but not enforced. Adding compile-time check that `dharma_emit`/`dharma_field` only appear inside `@manager` functions.
- **Boundary epilogues.** The decorator seam injects a prologue only. Adding prologue/epilogue wrapping, the prerequisite for durability-as-an-effect (Section 19.1).
- **`vessel` keyword.** Replaces `package` in manifests. Adding to lexer.
- **Real `engram_*` runtime.** Currently stub. Adding in-process graph store with spreading activation, Hebbian strengthening, and disk persistence — see Section 16.4.
- **Real `dharma_*` runtime.** Currently stub. Adding network transport, channel registry, identity resolution.
- **Real `http_get`/`http_post`/`http_serve`.** Currently empty stubs. Adding libcurl-backed client and a thread-pool server.
- ~~**Real `engram_*` runtime.** Currently stub.~~ **Stale (verified 2026-08-16) — this is implemented, not planned.** `lang/runtime/el_runtime.c` carries the in-process graph store with spreading activation, Hebbian strengthening, disk persistence (paged store, magic `ENGST01`), an HNSW vector index behind a `eg_vindex_view`/`eg_vindex_maintain` publication boundary, and the full cognition surface (`engram_think_json`, `engram_ground_json`, `engram_assert_json`, `engram_attend_json`, `engram_correspondence_beat_json`). The "stub" description may still hold for the **lagging forks** (`lang/el-compiler/runtime/`, `products/web/runtime/`) — see `AGENTS.md`, which names those as downstream copies that cannot build the engram product. **Which runtime this line refers to needs a decision; it is not a fact that can be recovered from the text.**
- ~~**Real `dharma_*` runtime.** Currently stub.~~ **Needs re-verification (2026-08-16).** Not checked in this pass; do not rely on either reading.
- ~~**Real `http_get`/`http_post`/`http_serve`.** Currently empty stubs.~~ **Stale.** libcurl-backed HTTP and a thread-pool server are live — `http_serve_async` is what `neuron/soul.el:729` runs before entering its awareness loop, and `realizer_register` resolves El functions through the same `dlsym` mechanism `http_set_handler` relies on.
- **JSON, time, UUID, state, env, additional string/list/math builtins.** See Section 12 for the canonical list.
### Not in this language
@@ -96,8 +99,10 @@ The following words are reserved and cannot be used as identifiers. Each row not
| `while` | yes | Loop |
| `import` / `from` / `as` | yes | Module import |
| `true` / `false` | yes | Bool literals |
| `cgi` | planned | Top-level CGI declaration block |
| `manager` / `engine` / `accessor` | as decorators | VBD role marker on `fn` (enforcement planned) |
| `cgi` | yes | Top-level CGI declaration block |
| `service` | yes | Top-level capability-bounded declaration block |
| `program` | yes | Top-level cross-cutting declaration block (Section 18) |
| `manager` / `engine` / `accessor` | as decorators | VBD role marker on `fn`; enforcement and boundary auto-emit are live (Section 9) |
| `vessel` | planned | Manifest declaration (replaces `package`) |
| `activate` / `where` | planned | Spreading-activation construct |
| `sealed` | planned | Capability scope block |
@@ -446,9 +451,21 @@ Parsed. The module name is recorded; the brace-list is consumed. Both forms prod
fn handle(channel: String, msg: String) -> Void { … }
```
The `@` token followed by an identifier attaches a decorator name to the next `FnDef`. Decorators with structural meaning today: none. Planned enforcement (Section 16.2): VBD roles `@manager`, `@engine`, `@accessor`.
The `@` token followed by an identifier attaches a decorator to the next `FnDef`.
Non-VBD decorators are accepted and ignored.
**Decorators take arguments and they stack.** `@route("/p", "GET") @manager fn f()` attaches both to `f` as a `decorators` list of `{name, args}` records, topmost-first. Arguments are string literals only.
**Decorators have structural meaning today.** This is El's function-level boundary seam — the mechanism by which a cross-cutting concern is handled *at the boundary* rather than by a convention repeated at every call site:
| Decorator | Structural effect |
|---|---|
| `@manager` | Permits calls to `dharma_emit` / `dharma_field`. Calling either from a non-`@manager` fn emits a `#error` into the generated C — a compile-time failure, not a lint. |
| `@manager`, `@accessor` | Codegen injects one call to `engram_boundary_beat(<fn name>)` at function entry. The decorated op self-reports (chrono tick, afferent counter, self-activity strengthen, dharma bus event) with **zero** hand-written instrumentation in its body. |
| `@route(path, method, …)` | Records a route into a generated dispatch table. |
Decorators with no registered meaning are accepted and ignored.
**Limits of the seam, as it stands.** The injection is a *prologue only* — there is no epilogue, no wrapping of the call, and no way for a decorator to run code after the body returns. The injected callee is a fixed builtin chosen by the compiler, not derived from the decorator name or its arguments. Section 19 depends on lifting exactly these two limits.
---
@@ -1088,4 +1105,230 @@ The next minor version closes the implementation gaps named in this document. Tr
---
## 18. The Program Block — cross-cutting concerns [implemented]
### 18.0 Why this exists
A cross-cutting concern is one that belongs to the *process*, not to any function in it: only one of me may run; this is what my configuration is; every mutation must be durable; every request must be authorized.
El's units of encapsulation are the function and the module. Neither can hold a concern like that. So each one had been expressed the only way it could be — as a **convention**: *call this at every site.* Conventions of that shape do not hold. They are not enforced by anything, they are invisible in review, and they fail silently at the one site somebody forgot.
Measured in this codebase before this section existed:
| Concern | State | What the convention was |
|---|---|---|
| process identity | **zero** guards anywhere — no pidfile, no lock, no already-running check, at any layer | "check nothing is already running first" |
| configuration | **20** distinct environment variables in one program, each with its default written inline at the read site | "remember the right default here" |
| durability | **62** `persist_*` / `engram_save` / `wal_*` / `checkpoint` call sites | "after you mutate, remember to persist" |
| request auth | **10** per-route `_auth` checks | "check the token in this handler too" |
These are not four problems. They are one absence, four times.
That the convention form fails is observed, not predicted. Process identity failed three times in a single day: twice, two engram processes ran simultaneously against the same data directory; twice, a stale binary held a port and answered probes while a fresh build was believed to be under test, because `pkill -f` had silently failed to match its argv — which nearly produced a false "the fix does not work" conclusion. Configuration failed structurally: `ENGRAM_DATA_DIR` was read at six sites, five of them dead bindings, and the sixth defaulted to `/tmp/engram` — contradicting the canonical resolver's `$HOME/.neuron/engram` and landing a pre-destructive safety backup on ephemeral storage.
The `program` block is where a concern of this shape is declared once and enforced by the compiler at the process boundary.
### 18.1 Syntax
```
program "engram" {
singleton: "engram"
env ENGRAM_BIND: String = ":8742"
env GUIDE_PORT: Int = "8771"
env ENGRAM_API_KEY: String required
}
```
At most one `program` block per program. It composes with `cgi` and `service` — those declare what a program *may do*; `program` declares what a program *is*.
Grammar:
```ebnf
program_block = "program" string "{" { program_field } "}" ;
program_field = singleton_field | env_field ;
singleton_field = "singleton" ":" string [ "," ] ;
env_field = "env" ident ":" type
[ "=" string ] [ "required" ] [ "," ] ;
```
`singleton` and `env` are **not** reserved words. They are read as identifier token values by the block's own parse loop, so they remain usable as ordinary identifiers everywhere else. `program` is the only keyword this section adds.
### 18.2 Process identity — `singleton`
`singleton: "id"` compiles to an `el_singleton_acquire("id")` call injected as the **first statement of `main()`**, before any user statement runs.
The runtime takes an exclusive non-blocking `flock` on `<dir>/el-singleton-<id>.lock`, where `<dir>` is `$EL_SINGLETON_DIR`, else `$TMPDIR`, else `/tmp`. On success it writes its pid and holds the descriptor open for the life of the process. On contention it **refuses to start**: it reports the holder's pid, names the lock file, and exits 1.
Two properties are deliberate:
- **It is a lock, not a pidfile.** The kernel releases an `flock` when the owning process dies — including on `SIGKILL` and on crash. There is therefore no stale-lock state, and so no "delete the lock file to get unstuck" recovery ritual. Such a ritual would itself be a convention, which is the thing this section exists to remove.
- **It reports the holder's pid.** "Already running" is not actionable. A pid is. This is the direct answer to the observed failure where a stale process survived a `pkill` and went on answering probes.
Refusal is loud and total. It is not a warning, and the program does not continue degraded. This matters more than it looks: today a second engram whose `bind()` fails merely *returns* from `http_serve` — after it has already replayed the WAL and written boot-time backup files — and then exits **0**, indistinguishable from a clean run. `singleton` refuses before the first side effect.
### 18.3 Configuration — `env`
Each `env` entry declares one configuration variable: its name, its type (`Int` or `String`), and either a default or `required`.
Resolution happens once, at startup, in declaration order: **the environment wins; the declaration supplies the fallback.** Then `el_config_validate` checks the whole schema and reports *every* problem at once before exiting — a startup that fails one variable at a time costs one restart per variable.
Values are read with `config("NAME")`, which returns a `String`.
The enforcement that makes the declaration real: **once a program block exists, `config("X")` for an undeclared `X` is a fatal error.** Without that, the schema would be advisory, and an advisory schema is just another convention. Programs with no `program` block are unaffected — `config()` falls back to a plain environment read, so migration is incremental and per-program.
The point is not that configuration is now centralized. It is that **a default is no longer a decision made at a read site.** A read site cannot disagree with another read site about what a variable means, because a read site no longer says.
### 18.4 What is deliberately not declared here
Some values look like configuration and are not. `ENGRAM_DATA_DIR` already has a single owner — `engram_resolve_data_dir()`, which resolves it, creates the directory, and fails loud rather than silently persisting to an ephemeral path. Declaring it in the `program` block as well would give it two owners that can disagree, recreating the precise defect this section removes.
The rule: **a variable belongs in the program block when the block would be its only owner.** If a resolver already owns it, leave it there.
`HOME` is likewise not configuration. It is an environment fact, and stays a raw `env()` read.
---
## 19. Boundary Effects — durability and request authorization [design only, not implemented]
Sections 19.1 and 19.2 specify the two remaining concerns from the table in 18.0. Both are **designed and deliberately unimplemented.** The reason is stated in 19.3 and it is not difficulty.
### 19.1 Durability as an epilogue effect
**The defect.** 62 call sites carry the convention *"after you mutate, remember to persist."* This is structurally the same defect as the index bug being fixed elsewhere in this tree — *"after you append, remember to index"* — which failed at **9 of 9** sites. A convention that failed at 100% of its sites is the strongest available evidence about what this class of convention is worth.
**Why the existing seam cannot express it.** §9's injection is a prologue. Durability is inherently an *epilogue*: persist after the mutation succeeds, and not at all if it threw. The seam has no epilogue.
**Design.** Extend the decorator seam from prologue-only to prologue/epilogue, then declare durability as an effect on the mutating function:
```
@durable("engram")
fn engram_write_node(id: String, body: String) -> Bool { … }
```
Codegen wraps rather than prefixes:
```c
el_val_t engram_write_node(el_val_t id, el_val_t body) {
el_effect_enter(EL_STR("durable"), EL_STR("engram"));
el_val_t __r = /* original body */;
el_effect_exit(EL_STR("durable"), EL_STR("engram"), __r);
return __r;
}
```
`el_effect_exit` is where the persist happens, and it is the only place it happens. Two properties follow that the 62 hand-written sites cannot have:
- **Coalescing.** The epilogue is a single choke point, so N mutations inside one request can produce one fsync instead of N. The hand-written form cannot coalesce, because no site knows about the others.
- **Failure is not silent.** A persist that fails inside `el_effect_exit` can force the mutation's return value to failure. A forgotten `persist_*` call cannot fail — it simply does not happen, which is exactly why the defect is invisible.
**Enforcement, and this is the part that actually fixes it.** Mirroring §9's `#error` for `dharma_emit`: a function that calls a mutating primitive without carrying `@durable` is a **compile error**. Otherwise this is a 63rd thing to remember rather than a replacement for 62.
### 19.2 Request authorization as a route effect
**The defect.** 10 per-route `_auth` checks. The HTTP layer has no concept of authorization, so a new route is unauthenticated by default and silently so — the failure mode is a route that forgot, and nothing anywhere reports it.
**Design.** Authorization becomes an argument to the `@route` decorator, which already takes arguments and already builds a dispatch table:
```
@route("/api/write", "POST", auth: "required")
fn route_write(body: String) -> String { … }
```
The generated dispatcher performs the check **before** dispatch, so an unauthorized request never reaches the handler and the handler contains no auth code at all.
The default must be `required`. A route that says nothing gets authorization; opening one up takes an explicit `auth: "public"`. Defaulting to public preserves the current failure mode exactly — forgetting stays silent — and a default that preserves the defect is not a fix.
Route inventory falls out for free: the dispatch table already exists, so the compiler can emit the full route/auth matrix and make "which routes are public" a fact that is read rather than audited.
### 19.3 Why these are not implemented
Not difficulty — **collision**. Both land squarely in regions two other agents hold right now:
- **Durability** requires changing the mutation and persist paths in `lang/runtime/el_runtime.c` and `engram/src/server.el` — the same files and the same read/write paths being restructured by concurrent work on VIndex read-path mutation and memory ownership, and on geometry-as-an-el-value and `transduce`.
- **Request auth** requires changing route dispatch in `engram/src/server.el`, which the geometry/`transduce` work is actively reshaping.
Implementing either now would mean editing files under concurrent modification and resolving conflicts in exactly the paths whose correctness is currently under repair. The designs are recorded here so the work is not lost, and so that whoever lands them does so against a settled tree.
The prerequisite for 19.1 is the same in both cases: **lift the §9 seam from prologue-only to prologue/epilogue.** That change is independent of both collisions and can land first.
*(Status note, 2026-08-16: the geometry/`transduce` collision named above has since landed — see Section 20. The VIndex read-path collision has also landed; see `lang/spec/runtime-ownership.md` §5. 19.1 and 19.2 remain unimplemented, but the stated reason no longer holds for those two files.)*
---
## 20. Geometry — signal as a first-class value [implemented]
Landed 2026-08-16 (#141, #144). Declared here because the spec is the single source of truth for implemented-vs-planned, and this is a language surface, not a runtime detail.
### 20.1 Why this exists
Until 2026-08-16 no El ingest path could carry a vector. Nodes took **text**, and geometry was *derived* from that text. Text was therefore the **mandatory entry medium**: any non-text modality — a tone, a pulse, an image, a voice sample — had to be *described in prose first*, and the geometry subsequently reasoned over was the geometry **of the description, not of the signal**.
Two changes remove that, and neither is engram-specific — which is why they are in the language and not in the graph. Any program touching any modality needs them; the engram is merely one El program that happens to hold a graph.
1. **Geometry is a value that carries its own width.**
2. **A realizer is an ordinary El function** — so admitting a new modality never requires a runtime patch.
### 20.2 The `Geometry` type
`Geometry` is an opaque boxed pointer, exactly like `Instant` / `Calendar` / `Rhythm`. **No codegen change was required** to add it — the annotation is just a type name.
```el
let g: Geometry = geometry_new(4)
```
| builtin | returns | notes |
|---|---|---|
| `geometry_new(dim)` | `Geometry` | zero-filled; `0` on failure |
| `geometry_dim(g)` | `Int` | width; `0` if not a Geometry |
| `geometry_is(g)` | `Int` | `1` if a live Geometry |
| `geometry_get(g, i)` | `Float` | component |
| `geometry_set(g, i, x)` | `Int` | `1` ok, `0` out of range |
| `geometry_norm(g)` | `Float` | L2 — lets a caller check a realizer emitted **signal, not zeros** |
| `geometry_free(g)` | `Int` | `1` if freed. Returns a value rather than `void` so it is safe in any expression position without a codegen void-builtin table entry |
**Ownership.** A `Geometry` is owned by the El caller and released with `geometry_free`. `node_attach_geometry` **copies**, so a node and the caller's value have independent lifetimes.
### 20.3 Wire adapters — the only place an encoding appears
```el
geometry_from_f32le_hex(hex) -> Geometry // 0 on empty / odd-length / non-hex
geometry_to_f32le_hex(g) -> String // "" if not a Geometry
```
`f32le hex` is little-endian float32, 8 hex chars per component — the encoding the perception vessel's `/voice/embed` already emits. **The width is derived from the input length, never supplied by a caller**, which is why there is no max-dim constant to validate a claimed length against. Encodings appear here and nowhere else: at the edge.
### 20.4 Realizers and `transduce`
A **realizer** maps one modality into geometry. Registration is **by name**: every El `fn name(...)` compiles to a global C symbol with that exact name, and the registry resolves it with `dlsym` against the running binary — the same mechanism `http_set_handler` already relies on.
```el
fn tone_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(4)
let n: Int = str_len(signal)
let a: Int = geometry_set(g, 0, int_to_float(n))
g
}
realizer_register("tone", "tone_realizer") // 1 ok / 0 unresolved
let g: Geometry = transduce(sample, "tone") // Geometry, or 0 if no organ
realizer_has("tone") // 1 if registered
```
The registry keys on **modality**, not on registration order. `transduce` returns `0` when no organ is registered for the modality — an absent organ is a reportable state, not a silent zero vector.
**The claim this makes:** a realizer is not in the runtime and not known to the compiler. Adding a modality is writing an El function and registering a name. `lang/examples/transduce.el` is the worked example and doubles as an executable proof — it exits non-zero if any check fails.
### 20.5 Two comparison hazards this surface exposed
Both were **measured**, not stylistic, and both are properties of the current `elc` that any El author should know:
- **`==` lowers numerically only when both operand *names* are in the per-function int-name set** that `let x: Int` populates. A bare `f(x) == 0` is not a registered name and lowers to `str_eq``strcmp` on two integers reinterpreted as pointers. `<` and `>` lower directly with no inference, so truthiness against a builtin's return is written `> 0` / `< 1`.
- **`+` dispatches on whether both operands are known-Int, and a user-defined `fn` call is not.** `let fails: Int = fails + check(...)` lowered to **string concatenation** and printed `4343632752` — a pointer. Nothing was wrong with the checks; the tally was lying. Failing fast needs no arithmetic at all, so there is nothing left to get wrong.
### 20.6 What this does not do
`transduce` produces geometry; it does not decide what the geometry *means*. Nothing here grounds anything. Grounding is the edge weight in the graph the geometry is later attached to — see `lang/spec/correspondence-and-censorship.md`.
---
End of specification.
+16 -7
View File
@@ -28,12 +28,12 @@ Each of these is a distinct merged or proposed fix. Each addresses one deposit.
| VIndex freed under a concurrent reader | `el_runtime.c:9424` | `fb32d15` guard (merged 08:46:43) |
| `_eg_vindex_seen` realloc'd on a read path | `el_runtime.c:9412` | same guard |
| `vindex_insert` on a read path | `el_runtime.c:9434`, `9450` | same guard |
| shared `visited` / epoch scratch stomped by concurrent searches | `engram_vindex.c:7981`, `169186`, `195` | proposed: move to per-search frame |
| shared `visited` / epoch scratch stomped by concurrent searches | `engram_vindex.c:7981`, `169186`, `195` | ~~proposed:~~ **built** moved to the call frame (§3.1(1), §5); TSan `readers` half clean (§7a) |
| nine append sites, none indexing → lazily-embedded nodes invisible | `el_runtime.c:7806, 7988, 8148, 8224, 11526, 11731, 12050, 15295, 15312` | "embed-gap #20", patched by making the *read* path catch up (`9439` comment) |
**Measured:** all file/line references above, read 2026-08-16. Crash frames `engram_activate → eg_vindex_sync → vindex_insert → _realloc → _xzm_xzone_malloc_freelist_outlined` are accounted for by rows 24.
**Inferred, not yet verified:** that the nine append sites do not share a single commit point. This needs one pass before Change C is sized.
~~**Inferred, not yet verified:** that the nine append sites do not share a single commit point. This needs one pass before Change C is sized.~~ **Moot — see §7.** The question was mis-aimed: node append is not the event that owns index membership, because a node without an embedding cannot be in a vector index. The five *embedding-assignment* sites are the real owner points.
---
@@ -135,12 +135,21 @@ The payoff of owning the language is unchanged and is now *cheaper*: introduced
## 6. Sequencing
> **⚠ Steps 25 belong to the abandoned capability-ABI §3 and are superseded
> (2026-08-16).** §3 was re-derived: the engram is immutable and recall is
> projection, so *what does not mutate needs no ownership discipline* and the
> question is dissolved rather than answered. There is no context type, no
> capability type, and no codegen change — **`const` is the capability**, and the
> constraint travels with the type of the thing rather than the shape of every call
> site, so **no sweep is needed at all** (§4). Steps 1, 6 and 7 stand. Struck rather
> than deleted, because the abandoned plan is why §4's cost argument is short.
1. **Read** how builtins are declared and dispatched, to confirm the call sites are compiler-generated in one place. *(This determines whether §4 holds. If dispatch is scattered, re-size before proceeding.)*
2. Introduce the context type and capability types.
3. Codegen emits the context at every builtin call site.
4. Mechanical sweep of builtin signatures.
5. Move index maintenance behind the write capability; the three read callers take the read capability.
6. Delete the residue-fixes listed in §5.
2. ~~Introduce the context type and capability types.~~ **Superseded**`const`.
3. ~~Codegen emits the context at every builtin call site.~~ **Superseded** — no codegen change.
4. ~~Mechanical sweep of builtin signatures.~~ **Superseded** — the constraint travels with the type.
5. ~~Move index maintenance behind the write capability; the three read callers take the read capability.~~ **Done, differently:** `eg_vindex_maintain` (exclusive, sole mutator) / `eg_vindex_view` (`const VIndex*`, shared readers), with `eg_vindex_note_embedded` as the write-side owner. This is a **publication** boundary, not a capability split — HNSW insert is not an append, so purity alone was insufficient (§2a, §3.1(3)).
6. Delete the residue-fixes listed in §5. *(Partially done — see §5's "NOT deleted" list; a residue whose structure has not been converted must be left standing.)*
7. **One** build of soul from el dev — which resolves the `state_get` leak and the crash together, rather than deploying a leak fix that reintroduces the crash.
---
+234
View File
@@ -0,0 +1,234 @@
import "../../runtime/eltest.el"
// test_transduce.el geometry as a first-class El value, and realizers
// declared in El rather than patched into the runtime.
//
// WHAT IS ACTUALLY UNDER TEST. Until 2026-08-16 no El ingest path could carry
// a vector: nodes took text, and geometry was DERIVED from that text. Text was
// therefore the mandatory entry medium, so any non-text modality had to be
// DESCRIBED in prose first and the geometry we reasoned over was the geometry
// OF THE DESCRIPTION, not of the signal. The fix has two halves, and this file
// exercises both:
//
// 1. Geometry is a VALUE it carries its own width, so nothing has to
// assert a width against a string's length.
// 2. A REALIZER is an ordinary El function. `tone_realizer` below is not in
// the runtime, is not known to the compiler, and is not special in any
// way; it is registered BY NAME and dispatched to through transduce().
// That is the load-bearing claim: adding a modality must not require a
// runtime patch, or nothing has actually moved into the language.
//
// COMPARISON DISCIPLINE IN THIS FILE (measured 2026-08-16, not stylistic):
// elc lowers `a == b` to a NUMERIC comparison only when both operand names are
// in the per-function int-name set, which `let x: Int` populates. A bare call
// like `geometry_is(g) == 0` is not a registered name, so it lowers to
// `str_eq(...)` strcmp on two integers reinterpreted as pointers. `<` and `>`
// lower directly via binop_to_c with no type inference at all, so truthiness is
// written `> 0` / `< 1` here, and any exact `==` is done on a value first bound
// through `let x: Int`.
// A realizer, written entirely in El
// Maps a "tone" signal into a 4-component geometry. Deliberately trivial
// what is being proven is that an El function can BE a realizer, not that
// this is good acoustics. The one real property it has: distinct signals
// produce distinct geometry, so the test can tell transduction from a stub.
fn tone_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(4)
let n: Int = str_len(signal)
let a: Int = geometry_set(g, 0, int_to_float(n))
let b: Int = geometry_set(g, 1, int_to_float(n * 2))
let c: Int = geometry_set(g, 2, int_to_float(n * 3))
let d: Int = geometry_set(g, 3, int_to_float(n * 4))
g
}
// A second realizer for a different modality, to prove the registry keys on
// modality and does not just hand back "the last thing registered".
fn pulse_realizer(signal: String) -> Geometry {
let g: Geometry = geometry_new(2)
let a: Int = geometry_set(g, 0, 1.0)
let b: Int = geometry_set(g, 1, 0.0)
g
}
// A deliberately BROKEN realizer: it returns something that is not a Geometry.
// transduce() must not hand this back to a caller as if it were one.
fn bogus_realizer(signal: String) -> Geometry {
return 12345
}
test "geometry-is-a-value-with-its-own-width" {
let g: Geometry = geometry_new(8)
let live: Int = geometry_is(g)
assert live > 0, "geometry_new returns a live Geometry"
let d: Int = geometry_dim(g)
assert d == 8, "a Geometry carries its own width"
let freed: Int = geometry_free(g)
assert freed > 0, "geometry_free reports what it did"
}
test "geometry-rejects-nonsense-without-an-arbitrary-bound" {
// dim <= 0 is not a width. Note there is deliberately no MAX dim here:
// #141 needed `dim <= 8192` only to bound an allocation sized from a
// caller's claim about a string. A value that carries its own width has
// nothing left to validate, so the only failure left is allocation.
let zero: Geometry = geometry_new(0)
let z: Int = geometry_is(zero)
assert z < 1, "dim 0 is not a geometry"
let neg: Geometry = geometry_new(-4)
let n: Int = geometry_is(neg)
assert n < 1, "negative dim is not a geometry"
// Accessors must be total: a non-geometry is 0-width, never a crash.
let nd: Int = geometry_dim(0)
assert nd < 1, "geometry_dim of a non-geometry is 0"
let ni: Int = geometry_is(0)
assert ni < 1, "geometry_is of a non-geometry is 0"
let nf: Int = geometry_free(0)
assert nf < 1, "geometry_free of a non-geometry is a no-op"
}
test "geometry-components-round-trip" {
let g: Geometry = geometry_new(3)
let s0: Int = geometry_set(g, 0, 1.5)
let s1: Int = geometry_set(g, 1, -2.5)
assert s0 > 0, "set in range succeeds"
let oob: Int = geometry_set(g, 3, 9.0)
assert oob < 1, "set out of range is refused, not silently dropped"
let v0: Float = geometry_get(g, 0)
let d0: Float = v0 - 1.5
assert d0 < 0.001, "component 0 round-trips"
assert d0 > -0.001, "component 0 round-trips"
let v1: Float = geometry_get(g, 1)
let d1: Float = v1 + 2.5
assert d1 < 0.001, "component 1 round-trips (negative)"
assert d1 > -0.001, "component 1 round-trips (negative)"
let freed: Int = geometry_free(g)
}
test "hex-is-an-edge-adapter-and-derives-its-own-width" {
// 2 components, little-endian float32: 1.0 = 0000803f, 2.0 = 00000040.
let g: Geometry = geometry_from_f32le_hex("0000803f00000040")
let live: Int = geometry_is(g)
assert live > 0, "valid hex decodes to a Geometry"
let d: Int = geometry_dim(g)
assert d == 2, "width is DERIVED from the input, never supplied"
let a: Float = geometry_get(g, 0)
let da: Float = a - 1.0
assert da < 0.001, "first component decoded"
assert da > -0.001, "first component decoded"
let b: Float = geometry_get(g, 1)
let db: Float = b - 2.0
assert db < 0.001, "second component decoded"
assert db > -0.001, "second component decoded"
// Egress adapter is the exact inverse.
let back: String = geometry_to_f32le_hex(g)
assert str_eq(back, "0000803f00000040"), "hex round-trips exactly"
let freed: Int = geometry_free(g)
}
test "hex-rejects-malformed-input" {
let empty: Geometry = geometry_from_f32le_hex("")
let e: Int = geometry_is(empty)
assert e < 1, "empty hex is not a geometry"
let ragged: Geometry = geometry_from_f32le_hex("0000803f0000")
let r: Int = geometry_is(ragged)
assert r < 1, "length not a multiple of 8 is refused"
let nonhex: Geometry = geometry_from_f32le_hex("zzzzzzzz")
let nh: Int = geometry_is(nonhex)
assert nh < 1, "non-hex characters are refused"
}
test "a-realizer-declared-in-el-is-a-first-class-realizer" {
// THE CLAIM: tone_realizer is an ordinary El function. It is not in the
// runtime and the compiler knows nothing about it. Registering it by name
// is enough to make it the organ for a modality.
let reg: Int = realizer_register("tone", "tone_realizer")
assert reg > 0, "an El fn registers as a realizer by name"
let has: Int = realizer_has("tone")
assert has > 0, "the modality now has an organ"
let g: Geometry = transduce("aaa", "tone")
let live: Int = geometry_is(g)
assert live > 0, "transduce returns real geometry"
let d: Int = geometry_dim(g)
assert d == 4, "the El realizer determined the width, not the runtime"
// str_len("aaa") == 3, so component 0 must be 3.0 proof the signal
// actually reached the El function rather than a stub answering for it.
let c0: Float = geometry_get(g, 0)
let dc: Float = c0 - 3.0
assert dc < 0.001, "the signal reached the El realizer"
assert dc > -0.001, "the signal reached the El realizer"
let freed: Int = geometry_free(g)
}
test "distinct-signals-transduce-to-distinct-geometry" {
let reg: Int = realizer_register("tone", "tone_realizer")
let g1: Geometry = transduce("aa", "tone")
let g2: Geometry = transduce("aaaaa", "tone")
let a: Float = geometry_get(g1, 0)
let b: Float = geometry_get(g2, 0)
let diff: Float = b - a
// 5 - 2 = 3. If transduction were a stub these would be equal.
assert diff > 2.9, "different signals produce different geometry"
assert diff < 3.1, "different signals produce different geometry"
let f1: Int = geometry_free(g1)
let f2: Int = geometry_free(g2)
}
test "the-registry-keys-on-modality" {
let r1: Int = realizer_register("tone", "tone_realizer")
let r2: Int = realizer_register("pulse", "pulse_realizer")
assert r2 > 0, "a second modality registers independently"
let gt: Geometry = transduce("aaa", "tone")
let gp: Geometry = transduce("aaa", "pulse")
let dt: Int = geometry_dim(gt)
let dp: Int = geometry_dim(gp)
assert dt == 4, "tone still routes to its own realizer"
assert dp == 2, "pulse routes to a different realizer"
let f1: Int = geometry_free(gt)
let f2: Int = geometry_free(gp)
}
test "no-organ-is-reported-as-no-organ" {
// A modality with no realizer must transduce to NOTHING. It must never
// fall back to embedding a description of the signal and calling that
// perception that silent substitution is the entire defect this change
// exists to end.
let has: Int = realizer_has("echolocation")
assert has < 1, "unregistered modality has no organ"
let g: Geometry = transduce("anything", "echolocation")
let live: Int = geometry_is(g)
assert live < 1, "no realizer means no geometry, not fake geometry"
}
test "registration-of-an-unresolvable-name-fails-loudly" {
// Reported at the moment of WIRING, not later as "this modality mysteriously
// produces nothing". Distinguishing "no organ" from "broken organ" is the
// lesson that made this whole change necessary.
let bad: Int = realizer_register("ghost", "no_such_function_anywhere")
assert bad < 1, "an unresolvable realizer name is a registration failure"
let has: Int = realizer_has("ghost")
assert has < 1, "and nothing gets registered"
}
test "a-realizer-returning-non-geometry-transduces-nothing" {
let reg: Int = realizer_register("bogus", "bogus_realizer")
assert reg > 0, "the symbol resolves, so registration succeeds"
// ...but the contract is enforced at the boundary, so the caller never
// receives a value that would misbehave far away from here.
let g: Geometry = transduce("x", "bogus")
let live: Int = geometry_is(g)
assert live < 1, "a non-Geometry return transduced nothing"
}
test "norm-lets-a-caller-check-a-realizer-emitted-signal" {
let g: Geometry = geometry_new(2)
let z: Float = geometry_norm(g)
assert z < 0.001, "a fresh geometry is zero — norm says so"
let s0: Int = geometry_set(g, 0, 3.0)
let s1: Int = geometry_set(g, 1, 4.0)
let n: Float = geometry_norm(g)
let dn: Float = n - 5.0
assert dn < 0.001, "3-4-5: norm is 5"
assert dn > -0.001, "3-4-5: norm is 5"
let freed: Int = geometry_free(g)
}
+63 -7
View File
@@ -45,17 +45,43 @@ returned 60k230k-char unbounded traversals (this very session hit 104 KB and
## Layer 2 — primitive agentic tools (Neuron runs itself)
The base verbs all agentic behavior composes from — grounded in the LIVE
cog-arch (`think` is the one operation; faculties are its steering-space labels;
the correspondence-beat is the reflexive learning loop).
The base verbs all agentic behavior composes from.
> **⚠ The "PROVEN" verdicts in this table were measured against a build dated
> 2026-08-14 and four of the five are now known to have been proving the wrong
> thing (2026-08-16).** A verdict of PROVEN meant *the route returned a
> well-formed response*, not *the response was derivable from what produced it*.
> Corrections below, each with the measurement. Authority:
> `lang/spec/correspondence-and-censorship.md`.
| op | signature | engram builtin | status on clone (gate-1 recipe) |
|----|-----------|----------------|---------------------------------|
| `think` | `think({seeds, faculty})` faculty ∈ reason·abduce·induce·plan·analogize·recognize·discern·synthesize | `engram_think_json` | **PROVEN** — all 8 faculties return real 768-dim gradients (n_support 30282) |
| `think` | `think({seeds, faculty})` faculty ∈ reason·abduce·induce·plan·analogize·recognize·discern·synthesize | `engram_think_json` | ~~PROVEN — all 8 faculties return real 768-dim gradients~~ **RETRACTED, then re-proven differently.** The gradients were real in *shape* only: the call passed `NULL` as the anchor, `engram_think` re-origins at `anchor ? anchor : region->centroid`, and **the centroid is the one point where the gradient is zero by construction**. Measured: every faculty returned `{"direction":[0,0,…],"spread":0,"magnitude":1,"confidence":0.5}` — identical, differing only in its label. Fixed in **#141/#142**; gradients now vary by seed |
| `attend` | `attend({node, observer, salience})` | `engram_attend_json` | **PROVEN** (returns `salient-to`) |
| `assert` | `assert({claim, for_whom, floor})` — realize, honesty-floored | `engram_assert_json` | **PROVEN** |
| `ground` | `ground({claim, evidence, for_whom})` node-id anchors | `engram_ground_json` | **PROVEN** (grounded-by edge, grounding=0.912, written) |
| `learn` | `learn({seeds, faculty, keystone})` — the correspondence-beat | `engram_correspondence_beat_json` | **PROVEN** (real Stance: `stance-induce-…`, brier, reliability, written) |
| `assert` | `assert({claim, for_whom, floor})` — realize, honesty-floored | `engram_assert_json` | **PARTIAL.** `may_assert` is real. `"still_held"` is a **hardcoded literal `true`**`el_runtime.c:14538` emits it unconditionally, so it reports nothing it measured. Violates the invariant *a returned value must be derivable from what produced it* |
| `ground` | `ground({claim, evidence, for_whom})` node-id anchors | `engram_ground_json` | ~~PROVEN (grounded-by edge, grounding=0.912, written)~~ **RETRACTED.** That 0.912 was structural, not evidential: the call wrote the edge between the two *region hubs* and echoed them back as though they were the caller's input, so when both seeds resolved into one region it **grounded a node against itself and returned a confident score**. Measured: grounding `3b9ced5d` against `6edf8c79` scored **0.98883** purely because `6edf8c79` is the hub of `3b9ced5d`'s region; two independent agents reported 0.885 / 0.909 self-groundings as confident. **#147** grounds the node asked about, reports `claim_region`/`evidence_region` separately, and refuses three circular shapes. **The operation itself is still the wrong shape** — see below |
| `learn` | `learn({seeds, faculty, keystone})` — the correspondence-beat | `engram_correspondence_beat_json` | **PROVEN, and it was writing into a void.** The Stance, brier and reliability were real and really persisted — but `think` built a *neutral* stance every call and never loaded them, so every beat's calibration was written and thrown away on the next read. Fixed in **#146**: `think` resumes `stance-<faculty>-<hub>`, the same id the beat writes. Confidence **0.5 → 0.930726** on a calibrated region |
### What this table gets structurally wrong
- **`faculty` is not a parameter.** `reason` changes the *estimate* (a read),
`induce` changes the *parameters* (this is exactly what `learn` does), and
`abduce` changes the *structure* — a **write**, which `GeoGradient` cannot
express. A write cannot be a parameter of a read. That the eight were listed as
interchangeable values of one argument is why all eight returning the same thing
looked like a pass. Underneath, `engram/src/server.el:18701886` routes six of
them into one call with a string argument, and the name only reaches
`engram_think` through the stance — `cog_stance_init` stores it and nothing
reads it.
- **`ground` should not mint an edge at all.** Grounding is not a subsystem and
not a score: **it is the edge weight.** `grounded-by` as a relation type models
grounding as a relation *between* nodes when it is a property *of* a relation.
#147 corrected a scalar rather than deleting the operation; deletion is
sequenced.
- **`addWonderQuestion`** (Layer 1, `write`) treats wonder as an enumerable
instance you push. **Wonder is the boundary** — where activation spreads and
finds thin or absent geometry. There are about six, the same for everyone, and
they never close. A manifest materializes a property as a stored artifact.
`comprehend`/`realize`/`intend` are **compositions**, not separate live
primitives: comprehend = write+activate (world→geometry), realize = assert
@@ -69,6 +95,26 @@ execution→integrate) composes over `think`+`ground`+`learn`+`write`/`relate`.
`kn-efeb4a5b…` / `kn-5b606390…`, are refused — identity routes through
intentional-cultivation, as enforced today.
> **⚠ SUPERSEDED (2026-08-16).** This describes what the surface enforces, which
> is accurate — but the enforcement is the wrong kind of thing:
>
> > **In an immutable substrate, any mechanism that refuses a write is either
> > redundant with immutability, or an epistemic constraint misfiled as a
> > protective one.**
>
> "Keystone" means **load-bearing**, not precious. The real requirement is
> **non-circularity of the reference frame** — a reference fitted to its own
> readings reports perfect correspondence forever while drift becomes undetectable
> from inside — and that is satisfied *temporally*, not by a gate: the frame
> updates while activation is internally seeded, not while it is being used to act.
> **Independence is *when*, not *what*.** Corruption requires mutation, and the
> engram does not mutate: recoverability (the predecessor is always present),
> governance (supersession *is* the audit trail), evidence quality, and rate all
> fall out of the substrate. **Authorization** is the only residue and it is
> bounded — an unauthorized writer can *propose*, never erase. Note also that the
> live check is a substring match against two hard-coded ids
> (`el_runtime.c:14337`).
## How the caller invokes Neuron agentically
Once the ops are registered as MCP tools (aliases in `surface.el`), the caller
@@ -94,6 +140,16 @@ running itself.
## Honest ledger (built vs staged)
- **Route seam — IMPLEMENTED + PROVEN:** ported the `@route` codegen (from `feat/el-route-decorators`) into the worktree, rebuilt `elc` self-host, proved decorate→serve (`route_proof.el` on :8951); `surface.el` compiles with `el_route_dispatch` generated for all 8 ops.
- **All ops PROVEN live on the clone** (gate-1 boot recipe, node-id anchors): read, write, relate, supersede (immutable), tombstone, think (8 faculties), ground, attend, learn — daemon alive through all mutations (node_count 13173→13176).
> **⚠ Retracted in part (2026-08-16).** "The daemon stayed alive and every route
> returned a well-formed response" is what was actually proven, and that is a
> weaker claim than it reads as. See the Layer-2 table: `think` was reading at the
> zero-gradient point, `ground` was scoring nodes against themselves, `assert`
> emits a hardcoded field, and `learn` was persisting into a void. **A build that
> passes because nothing checks whether a returned value is derivable from what
> produced it has not been tested — it has been observed not to crash.** The
> related discipline gap, also 2026-08-16: **no test without a negative control**
> (#148's first attempt passed on the unpatched build too), and **no deploy
> without verifying the artifact carries the fix** (nine instances in one session).
- **Aperture-boundedness PROVEN:** vantage-read `limit=3 → 15 KB` vs `limit=50 → 363 KB` (fixes the whole-self dump).
- **Bus:** `@manager` ops emit on the real `dharma_*` bus (explicit today, compiles) — same transport as the swarm (`wt/swarm-ccr`).
- **STAGED (not guessed — needs the cognition-engram rebuild to verify link):** auto-injecting telemetry/interoception + bus emission at the decorated boundary (`cg_fn` diff in `SEAM_STAGED.md`); building the cognition engram with `surface.el` compiled in. No promote to live, no cutover (per rails).