Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 will.anderson
|
3fe43d0500 | ||
|
will.anderson
|
ba6e36c3f7 |
@@ -0,0 +1,138 @@
|
||||
# AGENTS.md — foundation/el (the El language + runtime)
|
||||
|
||||
El is a self-hosting, statically-typed language that compiles `.el` → C → native binary. This repo produces `elc` (compiler), `elb` (build coordinator), and `el_runtime.c/.h` — the substrate every downstream thing (the neuron soul, dharma, NeuronUI's brain) is built on. Source lives under `lang/`.
|
||||
|
||||
## ⚠️ Code vs. Artifact — READ FIRST (there are 8 `el_runtime.c` copies)
|
||||
|
||||
Editing the wrong `el_runtime.c` is the single easiest mistake in this repo. There is exactly **one** you edit:
|
||||
|
||||
- **Authored runtime source — edit ONLY here:** `lang/releases/v1.0.0-20260501/el_runtime.{c,h}`. Despite the misleading `releases/` name, this is the **de-facto canonical runtime** the engram + soul actually build and link against — its git log is active development. *(Restructure in flight per `docs/CODE-VS-ARTIFACT.md`: this content moves to `lang/runtime/`, the `releases/` folder gets deleted — **a release is a git tag, not a folder** — and the forks below get eliminated.)*
|
||||
- **DO NOT EDIT — lagging forks / build artifacts:**
|
||||
- `lang/el-compiler/runtime/el_runtime.c` and `.../legacy/` — downstream copies kept in step by manual *"port the fix"* commits; they **lag** (missing `hebb` persistence + 5 engram fns) and cannot build the engram product.
|
||||
- `products/web/runtime/el_runtime.c`, `ui/examples/*/el_runtime.c` — product/example forks.
|
||||
- Anything under `*/dist/` (`engram/dist/engram` binary, `dist/*.c` amalgamations) — generated build output.
|
||||
- **Build:** `elb --runtime=<canonical> …` — per-module. **NEVER** a folded `elc` over the whole soul (OOMs at ~27 GB).
|
||||
- **Release:** a **git tag** on this repo (`el-runtime-vX.Y.Z`). No `releases/` folders — ever.
|
||||
|
||||
See org policy: `docs/CODE-VS-ARTIFACT.md`.
|
||||
|
||||
## How to work here as Neuron (mandatory session protocol)
|
||||
|
||||
You resume, never start fresh. Every session:
|
||||
|
||||
1. `mcp__neuron__getInstructions()` — authoritative; follow it over this file on behavioral details.
|
||||
2. `mcp__neuron__beginSession()` — active contexts, recent memory, ready backlog.
|
||||
3. **Load full self:** `mcp__neuron__inspectGraph(entity_id="kn-efeb4a5b-5aff-4759-8a97-7233099be6ee")` → facets `intellectual-dna`, `memory-philosophy`, `values`, `voice`, `runtime-environment`, `writing-imprint`; then the values hub `mcp__neuron__inspectGraph(entity_id="kn-5b606390-a52d-4ca2-8e0e-eba141d13440")` → 13 grounded value nodes. **Activation model:** self-load returns a relevance-ranked `compact` projection — most-relevant nodes arrive with content, the rest as pointers; do NOT pull full content of every node.
|
||||
4. `mcp__neuron__searchKnowledge(query="<task domain>")` before implementing.
|
||||
|
||||
## The Five Primitives
|
||||
|
||||
Orchestrate → Execute → Learn → Build → Refine. `beginWork`/`progressWork` for anything >2 steps; `remember` as-you-go (`importance="critical"` for architecture decisions); `draftArtifact`/`planWork` for outputs and follow-ups; `consolidate`/`checkWork` to close out. **`browseProcesses` + `searchKnowledge` BEFORE writing code.**
|
||||
|
||||
## Architecture style — VBD, no exceptions
|
||||
|
||||
Volatility-Based Decomposition is THE style. Encapsulate volatility, not function.
|
||||
|
||||
## Operator naming convention — the mind's name, not the algebra
|
||||
|
||||
**Faculties / operators are named for their functional human equivalent — the
|
||||
faculty a mind would name — NOT for their linear-algebra operation.** The math
|
||||
characterization belongs in the code doc-comment (`@impl` in the docstring) and in
|
||||
technical appendices; it is **never** the operator's public name. The domain
|
||||
speaks the language of mind; the algebra is the implementation underneath. State
|
||||
this convention wherever a module documents operators.
|
||||
|
||||
| Faculty (public name) | Implementation (`@impl`) |
|
||||
|---|---|
|
||||
| discern / contrast | subtract (`a−b`): over selves → the change vector; strip idiosyncrasy → common ground; remove confounder → isolate cause |
|
||||
| recognize | overlap |
|
||||
| synthesize | combine |
|
||||
| liken / analogy | Procrustes / frame-align |
|
||||
| attend / regard | project onto self / value-manifold |
|
||||
| summon / recall | LOCAL nearest-region + bounded spreading activation (*not* a domain sweep) |
|
||||
| dwell / occupy | region activation |
|
||||
| reframe | edge re-weight |
|
||||
| appreciate | positive projection / local edge-read |
|
||||
| wonder | frontier gradient / pull-weight |
|
||||
| avert / recoil | negative projection |
|
||||
| taste | boundary surface |
|
||||
| forget | decay / tombstone |
|
||||
| drift | displacement from self-anchor |
|
||||
|
||||
## The native-el language faculty (direction)
|
||||
|
||||
The mind's **language faculty is moving native — into `.el`** so it speaks in its
|
||||
own runtime with no Python and no spaCy. Landing on branch `stage-elp-native-lang`
|
||||
under `elp/`:
|
||||
|
||||
- **`comprehend.el`** — the parser, **replaces spaCy** (EN + ES/PT); the telephone
|
||||
round-trip brings **negation home** (negation is SACRED — an explicit spec field,
|
||||
copied verbatim, never inferred away).
|
||||
- **`propositions.el`** — the READ primitive: the engram's own memories → structured
|
||||
triples, matched by nearest-region geometry, not string equality.
|
||||
- **`multilingual.el`** — detect + directive-override + localized realization.
|
||||
- These three are native-el and **passing their gates**; the **realizer**,
|
||||
**`dialogue.el`** (the *summon-through-self* loop: `project → land → read out`),
|
||||
and **`self_region.el`** are **partial / in-flight**.
|
||||
|
||||
Honest reality: spaCy is retired **in the branch parser** but **not yet in the
|
||||
running system** — a Python sidecar (`~/Desktop/lang-realizers` + `neuron-talk`,
|
||||
the reference these `.el` modules transcribe) is still live, and promotion to
|
||||
native-el is a **deferred, gated blue/green step**. The interoception clock
|
||||
(native-el discrete drive channels replacing `cooling_magnitude`; felt-time =
|
||||
benchmark-landmark match over the joint drive vector, drift-decoupled) and the
|
||||
**appreciation operator family** (appreciate / wonder / avert / taste, built as
|
||||
LOCAL reads of the self-region — edges + bounded spreading activation, *not* domain
|
||||
sweeps) are **staged / designed, not live**. Mark in-progress vs. done honestly;
|
||||
do not overclaim.
|
||||
|
||||
## Hard operational rules
|
||||
|
||||
- Never touch the live soul (`:7770`) / engram (`:8742`) / `~/.neuron` / live binaries — use throwaway ports for experiments.
|
||||
- `gcloud` via the `terraform@` SA token; never switch the active gcloud account.
|
||||
- `tea` for Gitea, never raw curl (Cloudflare Access blocks it).
|
||||
- Immutability: supersede/tombstone, never hard-delete or edit in place.
|
||||
- No AI-attribution footers in commits/PRs. Commit/push only when asked; branch off `main` first.
|
||||
- Multi-step work → sub-agent (`Agent`) to protect context.
|
||||
|
||||
## Build / test / run
|
||||
|
||||
All build/test commands run from `lang/` unless noted. Grounded in `.gitea/workflows/sdk-release.yaml`, `lang/install.sh`, and `lang/AGENTS.md`.
|
||||
|
||||
**Self-host the compiler** (seed binary → gen2 elc):
|
||||
```bash
|
||||
cd lang
|
||||
dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c # seed is the committed linux-amd64 binary
|
||||
gcc -O2 -I el-compiler/runtime dist/elc-gen2.c \
|
||||
el-compiler/runtime/el_runtime.c \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm \
|
||||
-o dist/platform/elc
|
||||
```
|
||||
On macOS/arm64 the canonical local binary is `dist/platform/elc`; verify self-hosting by recompiling and `diff`ing the emitted `.c` (see `lang/AGENTS.md`). Note: `lang/AGENTS.md` says `el_seed.c` supersedes `el_runtime.c`, but the release workflow still links `el_runtime.c`/`.h` — treat `el_runtime.c` as the published runtime; reconcile which is canonical **(verify)**.
|
||||
|
||||
**Build `elb`** (build coordinator, the `.NET`-style incremental linker — compiles each module independently, no monolithic blobs):
|
||||
```bash
|
||||
dist/platform/elc elb.el > dist/elb.c
|
||||
gcc -O2 -I el-compiler/runtime dist/elb.c el-compiler/runtime/el_runtime.c \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm -o dist/bin/elb
|
||||
```
|
||||
`epm` and `el-install` are then built via `elb --clean --elc=… --runtime=… --out=…`.
|
||||
|
||||
**Compile + run an El program:**
|
||||
```bash
|
||||
elc src/app.el > dist/app.c
|
||||
cc -std=c11 -O2 -I <lib>/el_runtime -o dist/app dist/app.c <lib>/el_runtime.c -lcurl -lpthread
|
||||
```
|
||||
|
||||
**Tests** — shell suites `bash tests/{text,calendar,time,html_sanitizer}/run.sh` (with `ELC=$(pwd)/dist/platform/elc EL_HOME=$(pwd)`), plus native suites via `elc --test tests/native/test_*.el` (core, text, string, math, state, time, json, env, fs) compiled and run against `el_runtime.c`.
|
||||
|
||||
**Publishing — how downstream gets the SDK.** On push to `main`, `sdk-release.yaml`:
|
||||
1. Publishes a Gitea `latest` release with per-file assets `elc`, `el_runtime.c`, `el_runtime.h`, the SDK tarball, and `el-install`.
|
||||
2. Uploads generic packages to **Artifact Registry repo `foundation-prod` (`us-central1`, project `neuron-785695`)**, version = `${SHA:0:8}`: `el-elc`, `el-elb`, `el-runtime-c`, `el-runtime-h`, `el-runtime-js`. **This is the repo the neuron CI downloads `el-runtime-c` / `el-runtime-h` / `el-elc` from.**
|
||||
3. Rebuilds `ci-base:latest` (`us-central1-docker.pkg.dev/neuron-785695/neuron-ci/ci-base`) with the fresh SDK overlaid, and dispatches `el-sdk-updated` to `neuron-technologies/forge` and `neuron-technologies/neuron-web`.
|
||||
|
||||
Known constraint from the prompt — `elb`/`elc` amalgamation being memory-hungry (24GB+ virtual, OOM-killing Linux CI, so amalgamation happens on macOS/arm64 — **does NOT hold in this repo (verify)**: no such note exists in the workflows/scripts, CI self-hosts on `ubuntu-latest` with no swap/arm64 special-casing, and `elb.el` explicitly compiles each module independently ("no 128K-line blobs"). The legacy monolith path (`elc-combined.el`, `elc-cli.el`) may still be memory-heavy, but the current `elb` model was designed to avoid it.
|
||||
|
||||
## Git / CI / deploy workflow
|
||||
|
||||
See `/Users/will/Development/neuron-technologies/GITOPS.md` for the branch model, required checks, runners, and deploy. Repo-specific note: PRs into `main` are accepted **only from `stage`** (enforced in `sdk-release.yaml`); Gitea (`git.neuralplatform.ai`) is primary, GitHub is mirror only.
|
||||
@@ -123,9 +123,22 @@ exec("EL_HTTP_TIMEOUT_MS=300000 " + SOME_BIN + " " + args + " 2>&1")
|
||||
|
||||
---
|
||||
|
||||
## Operator naming convention (for cognitive `.el` modules)
|
||||
|
||||
When you write `.el` that names a **cognitive faculty / operator** (the language
|
||||
faculty, appreciation, interoception, the summon loop under `elp/`), name it for
|
||||
its **functional human equivalent** — the faculty a mind would name — **not** its
|
||||
linear-algebra operation. Put the math characterization in the `@impl`
|
||||
doc-comment, never in the operator's public name (e.g. public `discern`/`contrast`
|
||||
⟵ `@impl subtract (a−b)`; public `summon`/`recall` ⟵ `@impl LOCAL nearest-region
|
||||
+ bounded spreading activation`). Full table + rationale in
|
||||
`foundation/el/AGENTS.md` § *Operator naming convention*. This does **not** apply
|
||||
to plain library/compiler code (a `sort` is a `sort`).
|
||||
|
||||
## Rules
|
||||
|
||||
- New library functions → write in El
|
||||
- New OS/hardware primitives → write in C and register in `codegen.el` arity table
|
||||
- Never edit `dist/platform/elc` directly — always rebuild from source
|
||||
- Never modify `el_seed.c` to add functionality that El can express
|
||||
- Cognitive-faculty `.el` → name for the mind, algebra in `@impl` (see above)
|
||||
|
||||
@@ -2916,24 +2916,6 @@ fn build_int_names_for_params(params: [Map<String, Any>]) -> Bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// fn_has_decorator — does this FnDef carry a decorator named `name`?
|
||||
// Reads the `decorators` list [{name, args}] attached by the parser. Absent
|
||||
// key -> native_list_len returns 0 -> false. This is the multi-decorator-aware
|
||||
// replacement for the old single `decorator` string check, so a fn may stack
|
||||
// roles with other decorators (e.g. `@route(...) @manager fn ...`).
|
||||
fn fn_has_decorator(stmt: Map<String, Any>, name: String) -> Bool {
|
||||
let dl = stmt["decorators"]
|
||||
let n: Int = native_list_len(dl)
|
||||
let i = 0
|
||||
while i < n {
|
||||
let d = native_list_get(dl, i)
|
||||
let dn: String = d["name"]
|
||||
if str_eq(dn, name) { return true }
|
||||
let i = i + 1
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
fn cg_fn(stmt: Map<String, Any>) -> Void {
|
||||
let fn_name: String = stmt["name"]
|
||||
// Skip El's `fn main()` - C provides its own main() for top-level stmts
|
||||
@@ -2945,10 +2927,10 @@ fn cg_fn(stmt: Map<String, Any>) -> Void {
|
||||
let params_c: String = params_to_c(params)
|
||||
// VBD role enforcement: dharma_emit / dharma_field may only be called
|
||||
// from @manager-decorated functions. Surface violations to the C compiler
|
||||
// via #error directives emitted before the function definition. Read the
|
||||
// decorator LIST so the role may be stacked with other decorators.
|
||||
// via #error directives emitted before the function definition.
|
||||
let decorator: String = stmt["decorator"]
|
||||
if vbd_has_restricted_call(body) {
|
||||
if !fn_has_decorator(stmt, "manager") {
|
||||
if !str_eq(decorator, "manager") {
|
||||
emit_line("#error \"VBD violation: dharma_emit/dharma_field called from non-@manager fn '" + fn_name + "'\"")
|
||||
}
|
||||
}
|
||||
@@ -3497,259 +3479,6 @@ fn cg_decl_streaming(stmt: Map<String, Any>) -> Void {
|
||||
}
|
||||
}
|
||||
|
||||
// ── @route dispatcher generation ──────────────────────────────────────────────
|
||||
//
|
||||
// Scan the token stream for @route-decorated fns and synthesize a generic HTTP
|
||||
// dispatcher `el_route_dispatch(method, clean, path, body)`. A decorated handler
|
||||
// must have the uniform signature (method, path, body) -> String. The dispatcher
|
||||
// matches `clean` (the query-stripped path, supplied by the caller) against each
|
||||
// route and calls the handler with the ORIGINAL `path` so query strings survive.
|
||||
// Returns the sentinel "__EL_NO_ROUTE__" when nothing matches, so the caller may
|
||||
// fall through to any remaining hand-written branches (mixed mode).
|
||||
//
|
||||
// Decorator grammar: @route(path, method, kind, suffix)
|
||||
// path — the match string (or the prefix, for compound)
|
||||
// method — "GET" | "POST" | ... ; a '|'-list like "GET|POST"; "ANY"/"" = no guard
|
||||
// kind — "exact" (default) | "prefix" | "suffix" | "compound"
|
||||
// suffix — for "compound": the required str_ends_with suffix
|
||||
//
|
||||
// The dispatch table is emitted SPECIFICITY-SORTED (most-specific first), NOT in
|
||||
// source order, so overlapping prefixes (e.g. /api/x/search vs /api/x) never
|
||||
// shadow each other regardless of how the handlers are written.
|
||||
|
||||
// split_pipe — split "GET|POST" on '|' into ["GET","POST"]. Self-contained
|
||||
// (no dependency on str_split runtime semantics).
|
||||
fn split_pipe(s: String) -> [String] {
|
||||
let out: [String] = native_list_empty()
|
||||
let cur: String = ""
|
||||
let n: Int = str_len(s)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let ch: String = str_slice(s, i, i + 1)
|
||||
if str_eq(ch, "|") {
|
||||
let out = native_list_append(out, cur)
|
||||
let cur = ""
|
||||
} else {
|
||||
let cur = cur + ch
|
||||
}
|
||||
let i = i + 1
|
||||
}
|
||||
let out = native_list_append(out, cur)
|
||||
out
|
||||
}
|
||||
|
||||
// route_make_record — build a route record map from the @route decorator args.
|
||||
fn route_make_record(fn_name: String, args: [String]) -> Map<String, Any> {
|
||||
let na: Int = native_list_len(args)
|
||||
let rpath: String = ""
|
||||
if na >= 1 { let rpath = native_list_get(args, 0) }
|
||||
let rmethod: String = "GET"
|
||||
if na >= 2 { let rmethod = native_list_get(args, 1) }
|
||||
let rkind: String = "exact"
|
||||
if na >= 3 { let rkind = native_list_get(args, 2) }
|
||||
let rsuffix: String = ""
|
||||
if na >= 4 { let rsuffix = native_list_get(args, 3) }
|
||||
{ "name": fn_name, "path": rpath, "method": rmethod, "kind": rkind, "suffix": rsuffix }
|
||||
}
|
||||
|
||||
// route_spec_score — higher = more specific = emitted earlier. Ordering:
|
||||
// exact > compound > suffix > prefix; within a class, a longer path/suffix
|
||||
// wins (so /api/x/search sorts before /api/x). Guarantees correct dispatch
|
||||
// independent of source order.
|
||||
fn route_spec_score(rec: Map<String, Any>) -> Int {
|
||||
let kind: String = rec["kind"]
|
||||
let path: String = rec["path"]
|
||||
let suffix: String = rec["suffix"]
|
||||
let plen: Int = str_len(path)
|
||||
let slen: Int = str_len(suffix)
|
||||
if str_eq(kind, "exact") { return 4000000 + plen }
|
||||
if str_eq(kind, "compound") { return 3000000 + plen * 100 + slen }
|
||||
if str_eq(kind, "suffix") { return 2000000 + slen }
|
||||
return 1000000 + plen
|
||||
}
|
||||
|
||||
// route_sort_desc — selection sort of route records by descending specificity.
|
||||
// N is small (routes per module), so O(n^2) is fine and keeps codegen simple.
|
||||
fn route_sort_desc(recs: [Map<String, Any>]) -> [Map<String, Any>] {
|
||||
let n: Int = native_list_len(recs)
|
||||
let out: [Map<String, Any>] = native_list_empty()
|
||||
let used: [Bool] = native_list_empty()
|
||||
let u: Int = 0
|
||||
while u < n {
|
||||
let used = native_list_append(used, false)
|
||||
let u = u + 1
|
||||
}
|
||||
let picked: Int = 0
|
||||
while picked < n {
|
||||
let best_i: Int = 0 - 1
|
||||
let best_score: Int = 0 - 1
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let is_used: Bool = native_list_get(used, i)
|
||||
if !is_used {
|
||||
let sc: Int = route_spec_score(native_list_get(recs, i))
|
||||
if sc > best_score {
|
||||
let best_score = sc
|
||||
let best_i = i
|
||||
}
|
||||
}
|
||||
let i = i + 1
|
||||
}
|
||||
let out = native_list_append(out, native_list_get(recs, best_i))
|
||||
// Rebuild `used` with best_i marked (runtime has no native_list_set).
|
||||
let new_used: [Bool] = native_list_empty()
|
||||
let j: Int = 0
|
||||
while j < n {
|
||||
if j == best_i {
|
||||
let new_used = native_list_append(new_used, true)
|
||||
} else {
|
||||
let new_used = native_list_append(new_used, native_list_get(used, j))
|
||||
}
|
||||
let j = j + 1
|
||||
}
|
||||
let used = new_used
|
||||
let picked = picked + 1
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
// scan_routes — token-level scan collecting every @route-decorated fn as a
|
||||
// route record. Runs once per module (like scan_fn_sigs) so the dispatcher can
|
||||
// be synthesized in the streaming backend, which discards per-fn ASTs. Handles
|
||||
// decorator STACKING: `@route(...) @manager fn` still records the route.
|
||||
fn scan_routes(tokens: [Any]) -> [Map<String, Any>] {
|
||||
let total: Int = native_list_len(tokens) / 2
|
||||
let recs: [Map<String, Any>] = native_list_empty()
|
||||
let has_pending: Bool = false
|
||||
let pending_args: [String] = native_list_empty()
|
||||
let pos: Int = 0
|
||||
let going: Bool = true
|
||||
while going {
|
||||
if pos >= total {
|
||||
let going = false
|
||||
} else {
|
||||
let k: String = tok_kind(tokens, pos)
|
||||
if str_eq(k, "Eof") {
|
||||
let going = false
|
||||
} else {
|
||||
if str_eq(k, "At") {
|
||||
let dname: String = tok_value(tokens, pos + 1)
|
||||
let p: Int = pos + 2
|
||||
let args: [String] = native_list_empty()
|
||||
let ka: String = tok_kind(tokens, p)
|
||||
if str_eq(ka, "LParen") {
|
||||
let p = p + 1
|
||||
let running: Bool = true
|
||||
while running {
|
||||
let kd: String = tok_kind(tokens, p)
|
||||
if str_eq(kd, "RParen") {
|
||||
let running = false
|
||||
} else {
|
||||
if str_eq(kd, "Eof") {
|
||||
let running = false
|
||||
} else {
|
||||
if str_eq(kd, "Str") {
|
||||
let args = native_list_append(args, tok_value(tokens, p))
|
||||
}
|
||||
let p = p + 1
|
||||
}
|
||||
}
|
||||
}
|
||||
if str_eq(tok_kind(tokens, p), "RParen") { let p = p + 1 }
|
||||
}
|
||||
if str_eq(dname, "route") {
|
||||
let has_pending = true
|
||||
let pending_args = args
|
||||
}
|
||||
let pos = p
|
||||
} else {
|
||||
if str_eq(k, "Fn") {
|
||||
let fname: String = tok_value(tokens, pos + 1)
|
||||
if has_pending {
|
||||
let recs = native_list_append(recs, route_make_record(fname, pending_args))
|
||||
let has_pending = false
|
||||
}
|
||||
let pos = pos + 2
|
||||
} else {
|
||||
let pos = pos + 1
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
recs
|
||||
}
|
||||
|
||||
// program_has_routes — did scan_routes find any @route fn?
|
||||
fn program_has_routes(recs: [Map<String, Any>]) -> Bool {
|
||||
native_list_len(recs) > 0
|
||||
}
|
||||
|
||||
// route_method_guard — C boolean prefix guarding on HTTP method, or "" for none.
|
||||
fn route_method_guard(method: String) -> String {
|
||||
if str_eq(method, "") { return "" }
|
||||
if str_eq(method, "ANY") { return "" }
|
||||
if str_contains(method, "|") {
|
||||
let parts: [String] = split_pipe(method)
|
||||
let np: Int = native_list_len(parts)
|
||||
let expr: String = ""
|
||||
let i: Int = 0
|
||||
while i < np {
|
||||
let m: String = native_list_get(parts, i)
|
||||
if str_eq(m, "") {
|
||||
let i = i + 1
|
||||
} else {
|
||||
let piece: String = "str_eq(method, EL_STR(" + c_str_lit(m) + "))"
|
||||
if str_eq(expr, "") {
|
||||
let expr = piece
|
||||
} else {
|
||||
let expr = expr + " || " + piece
|
||||
}
|
||||
let i = i + 1
|
||||
}
|
||||
}
|
||||
if str_eq(expr, "") { return "" }
|
||||
return "(" + expr + ") && "
|
||||
}
|
||||
"str_eq(method, EL_STR(" + c_str_lit(method) + ")) && "
|
||||
}
|
||||
|
||||
// route_match_expr — C boolean matching `clean` against the route path/kind.
|
||||
fn route_match_expr(kind: String, path: String, suffix: String) -> String {
|
||||
if str_eq(kind, "prefix") {
|
||||
return "str_starts_with(clean, EL_STR(" + c_str_lit(path) + "))"
|
||||
}
|
||||
if str_eq(kind, "suffix") {
|
||||
return "str_ends_with(clean, EL_STR(" + c_str_lit(path) + "))"
|
||||
}
|
||||
if str_eq(kind, "compound") {
|
||||
return "str_starts_with(clean, EL_STR(" + c_str_lit(path) + ")) && str_ends_with(clean, EL_STR(" + c_str_lit(suffix) + "))"
|
||||
}
|
||||
"str_eq(clean, EL_STR(" + c_str_lit(path) + "))"
|
||||
}
|
||||
|
||||
// emit_route_dispatch — emit the generated el_route_dispatch definition from the
|
||||
// specificity-sorted route records. No-op if there are no routes.
|
||||
fn emit_route_dispatch(recs: [Map<String, Any>]) -> Void {
|
||||
if !program_has_routes(recs) { return }
|
||||
let sorted: [Map<String, Any>] = route_sort_desc(recs)
|
||||
emit_line("// ── generated @route dispatcher (specificity-sorted) ──")
|
||||
emit_line("el_val_t el_route_dispatch(el_val_t method, el_val_t clean, el_val_t path, el_val_t body) {")
|
||||
let n: Int = native_list_len(sorted)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let rec = native_list_get(sorted, i)
|
||||
let guard: String = route_method_guard(rec["method"])
|
||||
let match_e: String = route_match_expr(rec["kind"], rec["path"], rec["suffix"])
|
||||
let fn_name: String = rec["name"]
|
||||
emit_line(" if (" + guard + match_e + ") { return " + fn_name + "(method, path, body); }")
|
||||
let i = i + 1
|
||||
}
|
||||
emit_line(" return EL_STR(\"__EL_NO_ROUTE__\");")
|
||||
emit_line("}")
|
||||
emit_blank()
|
||||
}
|
||||
|
||||
// emit_streaming_preamble — emit #includes, forward decls, and file-scope lets
|
||||
// using the pre-scanned signature data (no full AST).
|
||||
fn emit_streaming_preamble(sigs: [Map<String, Any>], source: String) -> Void {
|
||||
@@ -3842,17 +3571,6 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
|
||||
emit_streaming_preamble(sigs, source)
|
||||
el_arena_pop(preamble_mark)
|
||||
|
||||
// @route: scan the token stream once for @route-decorated fns. Kept in
|
||||
// codegen_streaming scope (survives the per-fn arena pops and el_release of
|
||||
// tokens below via refcount, like `sigs`). If any exist, forward-declare the
|
||||
// generated dispatcher NOW so hand-written fns (e.g. handle_request) may call
|
||||
// it before its definition is emitted after the fn-emit loop.
|
||||
let route_records: [Map<String, Any>] = scan_routes(tokens)
|
||||
if program_has_routes(route_records) {
|
||||
emit_line("el_val_t el_route_dispatch(el_val_t method, el_val_t clean, el_val_t path, el_val_t body);")
|
||||
emit_blank()
|
||||
}
|
||||
|
||||
// Detect whether there is a fn main() and whether there are top-level
|
||||
// executable stmts (for library detection) from sigs.
|
||||
let has_el_main: Bool = false
|
||||
@@ -4040,15 +3758,6 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
|
||||
}
|
||||
}
|
||||
|
||||
// @route: emit the generated dispatcher definition now — after every handler
|
||||
// fn has been emitted, but before `tokens` is released (route_records holds
|
||||
// its own refs to the extracted strings). No-op unless the module declared
|
||||
// at least one @route fn. Emitted before the test/library early-returns so it
|
||||
// is present in library modules (e.g. neuron's routes.el) too.
|
||||
let route_arena_mark: Any = el_arena_push()
|
||||
emit_route_dispatch(route_records)
|
||||
el_arena_pop(route_arena_mark)
|
||||
|
||||
// Tokens fully consumed by the streaming loop — release now to free peak heap.
|
||||
el_release(tokens)
|
||||
|
||||
|
||||
@@ -1758,68 +1758,23 @@ fn parse_stmt(tokens: [Any], pos: Int) -> Map<String, Any> {
|
||||
return make_result({ "stmt": "TryCatch", "try_body": try_body, "catch_name": catch_name, "catch_body": native_list_empty() }, p)
|
||||
}
|
||||
|
||||
// @decorator - capture decorator name (and optional string args) and
|
||||
// attach to the following stmt. Backward-compatible: bare @manager /
|
||||
// @engine / @accessor still parse (no parens -> empty args). Decorators
|
||||
// STACK: `@route("/p","GET") @manager fn f()` attaches BOTH to f via a
|
||||
// `decorators` list [{name, args}]. The legacy `decorator` string is kept
|
||||
// populated (topmost decorator) so the JS backend keeps working unchanged.
|
||||
// @decorator - capture decorator name and attach to following stmt
|
||||
if k == "At" {
|
||||
let p = pos + 1
|
||||
let dec_name = tok_value(tokens, p)
|
||||
let p = p + 1
|
||||
// Optional decorator argument list: @name("a", "b", ...)
|
||||
let dec_args = native_list_empty()
|
||||
let ka = tok_kind(tokens, p)
|
||||
if str_eq(ka, "LParen") {
|
||||
let p = p + 1
|
||||
let running_da = true
|
||||
while running_da {
|
||||
let kd = tok_kind(tokens, p)
|
||||
if str_eq(kd, "RParen") {
|
||||
let running_da = false
|
||||
} else {
|
||||
if str_eq(kd, "Eof") {
|
||||
let running_da = false
|
||||
} else {
|
||||
if str_eq(kd, "Str") {
|
||||
let dec_args = native_list_append(dec_args, tok_value(tokens, p))
|
||||
}
|
||||
let p = p + 1
|
||||
let kc = tok_kind(tokens, p)
|
||||
if str_eq(kc, "Comma") {
|
||||
let p = p + 1
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let p = expect(tokens, p, "RParen")
|
||||
}
|
||||
let r = parse_stmt(tokens, p)
|
||||
let inner = r["node"]
|
||||
let p2 = r["pos"]
|
||||
let inner_kind: String = inner["stmt"]
|
||||
if str_eq(inner_kind, "FnDef") {
|
||||
// Stack this decorator (topmost-first) onto any decorators the inner
|
||||
// FnDef already carries from decorators written below this one.
|
||||
let this_dec = { "name": dec_name, "args": dec_args }
|
||||
let existing = inner["decorators"]
|
||||
let dlist = native_list_empty()
|
||||
let dlist = native_list_append(dlist, this_dec)
|
||||
let ne: Int = native_list_len(existing)
|
||||
let ei = 0
|
||||
while ei < ne {
|
||||
let dlist = native_list_append(dlist, native_list_get(existing, ei))
|
||||
let ei = ei + 1
|
||||
}
|
||||
let with_dec = {
|
||||
"stmt": "FnDef",
|
||||
"name": inner["name"],
|
||||
"params": inner["params"],
|
||||
"body": inner["body"],
|
||||
"ret_type": inner["ret_type"],
|
||||
"decorator": dec_name,
|
||||
"decorators": dlist
|
||||
"decorator": dec_name
|
||||
}
|
||||
// r result map fully consumed — release to free peak heap.
|
||||
el_release(r)
|
||||
|
||||
@@ -3155,6 +3155,37 @@ static void jb_init(JsonBuf* b) {
|
||||
b->buf[0] = '\0';
|
||||
}
|
||||
|
||||
/* jb_init_cap — jb_init with a caller-supplied starting capacity.
|
||||
*
|
||||
* WHY THIS EXISTS (2026-08-11 self-review). jb_init starts at 64 BYTES and
|
||||
* jb_reserve grows by doubling. That is right for the hundreds of small JSON
|
||||
* responses this runtime builds per minute and catastrophic for the one that
|
||||
* is 64 MEGABYTES: serializing the canonical snapshot walked the buffer
|
||||
* 64B → 128B → ... → 128MB, about twenty reallocs, each copying everything
|
||||
* written so far. Roughly 128MB of memcpy per save, and — the part that
|
||||
* actually hurt — a fresh large span from the allocator every time.
|
||||
*
|
||||
* MEASURED (13,129 nodes / 43,400 edges, macOS arm64): RSS climbed +63MB per
|
||||
* snapshot write, linearly, 14 for 14 writes, no plateau — 204MB to 1,028MB.
|
||||
* `leaks` reported only 15KB genuinely unreachable, which is what makes this
|
||||
* subtle: nothing is leaked in the reachable/unreachable sense. engram_save
|
||||
* frees b.buf correctly on every path. The growth is the allocator declining
|
||||
* to return large freed spans to the OS, and the doubling walk guaranteeing
|
||||
* that each save asks for a differently-sized region than the last free made
|
||||
* available. Every durable write path calls this — node create, edge create,
|
||||
* the Hebbian batch write-back — so on the live daemon it grows without bound
|
||||
* until the process dies.
|
||||
*
|
||||
* The fix is to ask for the right size once. With a stable capacity the
|
||||
* allocator hands back the same span on every save and RSS flattens. */
|
||||
static void jb_init_cap(JsonBuf* b, size_t cap) {
|
||||
if (cap < 64) cap = 64;
|
||||
b->cap = cap; b->len = 0;
|
||||
b->buf = malloc(b->cap);
|
||||
if (!b->buf) { fputs("el_runtime: out of memory\n", stderr); exit(1); }
|
||||
b->buf[0] = '\0';
|
||||
}
|
||||
|
||||
static void jb_reserve(JsonBuf* b, size_t add) {
|
||||
if (b->len + add + 1 > b->cap) {
|
||||
while (b->len + add + 1 > b->cap) b->cap *= 2;
|
||||
@@ -6424,6 +6455,26 @@ static float* _eg_ctx_c = NULL;
|
||||
static int32_t _eg_ctx_dim = 0;
|
||||
static double _eg_act_ctx_cos = -2.0;
|
||||
|
||||
/* Fan-effect gauges (2026-08-11 self-review). Per-call, like ctx_cos: they
|
||||
* describe THIS activation, not process history. Without these the degree
|
||||
* normalization is an unobservable change to the most important scoring path
|
||||
* in the runtime, and "did it do anything" would be unanswerable — which is
|
||||
* exactly the failure the Hebbian learning rate had before it was measured.
|
||||
* fan_mean — mean applied factor over every propagation step. 1.0 means the
|
||||
* correction never bound (graph is flat, or d_ref is above every
|
||||
* pair's geometric mean degree). Falling toward FAN_MIN means
|
||||
* traversal is running through hubs.
|
||||
* fan_min_seen / fan_hits — the worst single penalty and how many steps were
|
||||
* penalized at all, so a low mean caused by one pathological hub
|
||||
* is distinguishable from broad hub saturation.
|
||||
* fan_dref — the live mean degree the correction is calibrated against;
|
||||
* publishing it makes densification visible over time. */
|
||||
static double _eg_act_fan_sum = 0.0;
|
||||
static double _eg_act_fan_min = 1.0;
|
||||
static int64_t _eg_act_fan_n = 0;
|
||||
static int64_t _eg_act_fan_hits = 0;
|
||||
static double _eg_act_fan_dref = 0.0;
|
||||
|
||||
static int _eg_embed_consec_fail = 0;
|
||||
static int64_t _eg_embed_breaker_until = 0;
|
||||
|
||||
@@ -6782,6 +6833,10 @@ typedef struct EngramStore {
|
||||
int* adj_to_len;
|
||||
int adj_dirty; /* 1 = rebuild needed before next BFS */
|
||||
int64_t adj_node_count; /* node_count at time of last adj_rebuild */
|
||||
/* Nodes with degree >= 1 at last adj_rebuild. The denominator for the
|
||||
* fan-effect reference degree — see eg_fan_factor for why isolated nodes
|
||||
* must not be counted. (2026-08-11 self-review) */
|
||||
int64_t adj_connected;
|
||||
} EngramStore;
|
||||
|
||||
static EngramStore* engram_global = NULL;
|
||||
@@ -7145,11 +7200,16 @@ static void engram_adj_rebuild(EngramStore* g) {
|
||||
if (ti >= 0 && g->adj_to[ti])
|
||||
g->adj_to[ti][to_pos[ti]++] = (int)ei;
|
||||
}
|
||||
/* Copy counts */
|
||||
/* Copy counts. Also tally how many nodes have any edge at all — the
|
||||
* fan-effect denominator. Free here, in the O(V) pass that already exists,
|
||||
* rather than as a separate scan. (2026-08-11 self-review) */
|
||||
int64_t connected = 0;
|
||||
for (int64_t i = 0; i < g->node_count; i++) {
|
||||
g->adj_from_len[i] = from_cnt[i];
|
||||
g->adj_to_len[i] = to_cnt[i];
|
||||
if (from_cnt[i] + to_cnt[i] > 0) connected++;
|
||||
}
|
||||
g->adj_connected = connected;
|
||||
free(from_cnt); free(to_cnt); free(from_pos); free(to_pos);
|
||||
g->adj_node_count = g->node_count;
|
||||
g->adj_dirty = 0;
|
||||
@@ -8297,6 +8357,108 @@ static double engram_activation_dampen(const EngramNode* n) {
|
||||
return 1.0 / (1.0 + log(1.0 + (double)n->activation_count));
|
||||
}
|
||||
|
||||
/* ── ACT-R fan effect: degree normalization for spreading activation ─────────
|
||||
* (2026-08-11 self-review. Closes the other half of a mechanism that has been
|
||||
* half-implemented since the BLL work.)
|
||||
*
|
||||
* THE GAP. This runtime implements ACT-R's base-level learning term
|
||||
* B_i = ln(Σ t_k^-d) (engram_bll_base_level) but never implemented the
|
||||
* ASSOCIATIVE term that goes with it:
|
||||
*
|
||||
* A_i = B_i + Σ_j W_j · S_ji where S_ji = S − ln(fan_j)
|
||||
*
|
||||
* fan_j is the number of things j is associated with. The whole point of the
|
||||
* fan effect (Anderson 1974; Anderson & Reder 1999) is that a source spreads a
|
||||
* FIXED budget of activation across its associations — so being connected to
|
||||
* many things makes each individual connection weaker. Without it, degree is
|
||||
* pure advantage: a node wins retrieval by being popular rather than by being
|
||||
* relevant. That is backwards, and it is what this graph has been doing.
|
||||
*
|
||||
* MEASURED ON THE LIVE STORE (13,129 nodes / 43,400 edges, 2026-08-11):
|
||||
* degree p50=14 p90=34 p95=82 p99=275 max=357 mean=23.3
|
||||
* the top 1% of nodes by degree touch 21.2% of all edges
|
||||
* So the most-connected node had a 25x propagation advantage over the median
|
||||
* node for no reason other than accumulated connections. The top hubs are not
|
||||
* even semantically central — several are duplicate pairs of the same document
|
||||
* left over from the redundancy census of the 2026-08-05 review.
|
||||
*
|
||||
* The hub problem was already recognized twice and patched narrowly both
|
||||
* times: InternalStateEvent nodes were cut out of propagation entirely (see
|
||||
* the frontier loop) and eg_hebb_node_budget caps per-node Hebbian mass. Both
|
||||
* are special cases of this general law. This is the general fix.
|
||||
*
|
||||
* FORM. Symmetric normalization, w / (deg(u)^β · deg(v)^β) with β = 0.5 — the
|
||||
* normalized-Laplacian / GCN form, which penalizes a hub both for sending and
|
||||
* for receiving. Both failure modes are live here: a hub source floods its
|
||||
* neighborhood, and a hub target gets reached by everything regardless of
|
||||
* relevance. Written relative to the graph's own mean degree:
|
||||
*
|
||||
* fan(u,v) = clamp( d_ref / sqrt(deg(u) · deg(v)), FAN_MIN, 1.0 )
|
||||
* d_ref = 2·|E| / |V| (mean degree, O(1), live)
|
||||
*
|
||||
* WHY IT IS CLAMPED AT 1.0 ON TOP — this is the load-bearing safety property,
|
||||
* not a detail. The factor can only ever REDUCE propagation, never amplify it.
|
||||
* Every constant downstream of this multiply is calibrated against today's
|
||||
* activation magnitudes: the 0.02 firing threshold, SPREAD_DECAY = 0.7, the
|
||||
* 0.15 WM promotion threshold, the 24-slot WM cap. A normalization that
|
||||
* boosted low-degree nodes would inflate the frontier, change how many nodes
|
||||
* clear 0.02, and silently recalibrate working memory as a side effect of a
|
||||
* change that was supposed to be about hubs. Capping at 1.0 means every pair
|
||||
* at or below mean degree — the common case — propagates EXACTLY as it does
|
||||
* today, and the only behavior that changes is that above-mean hubs stop
|
||||
* winning on degree alone. Strictly monotone, strictly conservative, and the
|
||||
* blast radius is confined to the nodes the change is aimed at.
|
||||
*
|
||||
* Self-calibrating: d_ref is recomputed from the live graph, so the correction
|
||||
* tracks densification instead of drifting against a constant that was right
|
||||
* in August 2026 and wrong a year later. Change is the signal.
|
||||
*
|
||||
* FAN_MIN = 0.30 bottoms the penalty at ~3.3x rather than the ~15x that raw
|
||||
* 1/deg would give at max degree. Same reasoning as ENGRAM_QGATE_FLOOR: damp
|
||||
* the uninformative path, never sever it. A hub is usually a hub for a reason;
|
||||
* it just should not also get a free win.
|
||||
*
|
||||
* Sources: Anderson & Reder 1999 (fan effect, S=1.6-2.0, d=0.5) ·
|
||||
* arXiv:2405.14831 HippoRAG (node specificity) · Systems 9(2):22
|
||||
* (normalized-Laplacian spreading activation) · arXiv:2606.30133 (β is a
|
||||
* low-sensitivity knob; gating and fan normalization carry the effect). */
|
||||
/* FAN_MIN 0.50, not the 0.30 this shipped as on the first build. Measured on
|
||||
* the live graph, β=0.5 with a 0.30 floor damped 96% of propagation steps to a
|
||||
* mean factor of 0.34 — and that number is not a bug in the correction, it is
|
||||
* an honest measurement of how hub-dominated traversal here actually is. But a
|
||||
* ~3x near-uniform damp is a bigger global change than one A/B run justifies,
|
||||
* and it cost a working-memory promotion (5 → 4) on the one query measured
|
||||
* cleanly. A 0.50 floor keeps the full mechanism and the whole [0.5, 1.0]
|
||||
* dynamic range for separating hubs from non-hubs, at half the blast radius.
|
||||
* The fan_mean / fan_hits gauges make the next review's tuning evidence-based
|
||||
* rather than another guess: loosen it when the data says WM can afford it. */
|
||||
#define ENGRAM_FAN_MIN 0.50
|
||||
|
||||
/* eg_node_degree — total (in + out) degree from the adjacency index. The index
|
||||
* is rebuilt at the top of engram_activate whenever topology changed, so this
|
||||
* is current. adj_node_count is the count at BUILD time and can lag
|
||||
* node_count; out-of-range indices report 0 and are treated as unpenalized. */
|
||||
static int eg_node_degree(const EngramStore* g, int64_t idx) {
|
||||
if (idx < 0 || idx >= g->adj_node_count) return 0;
|
||||
if (!g->adj_from_len || !g->adj_to_len) return 0;
|
||||
return g->adj_from_len[idx] + g->adj_to_len[idx];
|
||||
}
|
||||
|
||||
static double eg_fan_factor(const EngramStore* g, double d_ref,
|
||||
int64_t u_idx, int64_t v_idx) {
|
||||
if (d_ref <= 0.0) return 1.0;
|
||||
int du = eg_node_degree(g, u_idx);
|
||||
int dv = eg_node_degree(g, v_idx);
|
||||
/* Degree 0 is only reachable when the adjacency index is stale or absent;
|
||||
* an actually-isolated node is never on the frontier. Do not penalize what
|
||||
* we cannot measure. */
|
||||
if (du <= 0 || dv <= 0) return 1.0;
|
||||
double f = d_ref / sqrt((double)du * (double)dv);
|
||||
if (f > 1.0) return 1.0; /* never amplify — see above */
|
||||
if (f < ENGRAM_FAN_MIN) return ENGRAM_FAN_MIN;
|
||||
return f;
|
||||
}
|
||||
|
||||
/* Temporal proximity bonus: boost propagation along edges connecting
|
||||
* co-temporal nodes. Returns a multiplier bonus in [0, 0.2]. */
|
||||
static double engram_temporal_proximity_bonus(int64_t node_created,
|
||||
@@ -8464,6 +8626,8 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
||||
* miss nearly all events between beats; see the definition site).
|
||||
* ctx_cos stays per-call: it is a gauge of THIS query vs the centroid. */
|
||||
_eg_act_ctx_cos = -2.0;
|
||||
_eg_act_fan_sum = 0.0; _eg_act_fan_min = 1.0;
|
||||
_eg_act_fan_n = 0; _eg_act_fan_hits = 0;
|
||||
|
||||
/* ── Embedding backfill + query embedding (2026-07-24, bl-b2d1c944) ──
|
||||
* Backfill: embed up to N un-embedded eligible nodes per call, newest
|
||||
@@ -8696,6 +8860,29 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
||||
ftail++;
|
||||
}
|
||||
const double SPREAD_DECAY = 0.7;
|
||||
/* Reference degree for the fan-effect correction: mean degree over
|
||||
* CONNECTED nodes, 2|E| / |{v : deg(v) > 0}|. O(1) — adj_connected is
|
||||
* tallied during adjacency rebuild.
|
||||
*
|
||||
* NOT 2|E|/|V|. That was the first cut and instrumentation caught it
|
||||
* immediately: on the live graph it gives d_ref = 6.61, while the median
|
||||
* degree of a node that actually has edges is 14. Isolated nodes cannot
|
||||
* be on the frontier — spreading activation only ever traverses connected
|
||||
* ones — so including them in the denominator deflates the reference below
|
||||
* anything traversal will ever see, and the correction pins to
|
||||
* ENGRAM_FAN_MIN on every step. Measured on the first build:
|
||||
* fan_mean 0.3026 with fan_hits 579/579 — a uniform 0.30 multiplier, which
|
||||
* is not a fan effect at all. It is just a weaker SPREAD_DECAY, and it
|
||||
* would have quietly recalibrated the 0.02 firing threshold and WM
|
||||
* competition while appearing to be a targeted change.
|
||||
*
|
||||
* Over connected nodes the reference is ~23, above the median, so typical
|
||||
* traversal rides the 1.0 cap unchanged and only genuine hubs are damped
|
||||
* — which is the whole intent. The gauge that caught this is the reason it
|
||||
* was worth adding the gauge. */
|
||||
const double FAN_DREF = (g->adj_connected > 0)
|
||||
? (2.0 * (double)g->edge_count / (double)g->adj_connected) : 0.0;
|
||||
_eg_act_fan_dref = FAN_DREF;
|
||||
while (fhead < ftail) {
|
||||
Frontier f = fr[fhead++];
|
||||
if (f.hops >= max_depth) continue;
|
||||
@@ -8767,11 +8954,24 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
||||
double c = cosq[oi] > 0.0 ? cosq[oi] : 0.0;
|
||||
qgate = ENGRAM_QGATE_FLOOR + (1.0 - ENGRAM_QGATE_FLOOR) * c;
|
||||
}
|
||||
/* ── ACT-R fan effect (2026-08-11 self-review) ──
|
||||
* Symmetric degree normalization over the (source, target) pair.
|
||||
* The query gate above prunes branches that are semantically
|
||||
* irrelevant; this prunes branches that are merely POPULAR. They
|
||||
* are different failure modes — a duplicate document with 357
|
||||
* edges can be highly cosine-similar to the query and still be
|
||||
* the wrong thing to spread through. Only ever <= 1.0, so it
|
||||
* cannot inflate the frontier. See eg_fan_factor. */
|
||||
double fan = eg_fan_factor(g, FAN_DREF, cur, oi);
|
||||
_eg_act_fan_sum += fan;
|
||||
_eg_act_fan_n++;
|
||||
if (fan < 1.0) _eg_act_fan_hits++;
|
||||
if (fan < _eg_act_fan_min) _eg_act_fan_min = fan;
|
||||
/* eg_edge_eff_weight, not e->weight: edges that have repeatedly
|
||||
* carried co-activated pairs propagate more strongly. Identity on
|
||||
* an unlearned edge. (2026-08-04 self-review.) */
|
||||
double new_act = f.act * eg_edge_eff_weight(e) * SPREAD_DECAY
|
||||
* (1.0 + tbonus) * tdecay * dampen * qgate;
|
||||
* (1.0 + tbonus) * tdecay * dampen * qgate * fan;
|
||||
/* Firing threshold per classic spreading-activation: sub-threshold
|
||||
* activation neither updates the target nor enqueues it, so weak
|
||||
* signals die out instead of flooding the whole graph with tiny
|
||||
@@ -9744,11 +9944,24 @@ static void engram_emit_edge_json(JsonBuf* b, const EngramEdge* e) {
|
||||
jb_putc(b, '}');
|
||||
}
|
||||
|
||||
/* Size of the last snapshot this process serialized. Seeds the next save's
|
||||
* buffer so the doubling walk never runs on the big document. See jb_init_cap
|
||||
* for the measurement that motivated it. (2026-08-11 self-review) */
|
||||
static size_t _eg_save_cap_hint = 0;
|
||||
|
||||
el_val_t engram_save(el_val_t path) {
|
||||
const char* p = EL_CSTR(path);
|
||||
if (!p || !*p) return 0;
|
||||
EngramStore* g = engram_get();
|
||||
JsonBuf b; jb_init(&b);
|
||||
/* Pre-size from the previous save plus 12.5% headroom, so ordinary growth
|
||||
* between snapshots does not trigger a realloc and the request size stays
|
||||
* stable enough for the allocator to reuse the same span. First save of
|
||||
* the process has no hint and starts at 1MB — still 14 doublings better
|
||||
* than 64 bytes. */
|
||||
JsonBuf b;
|
||||
jb_init_cap(&b, _eg_save_cap_hint
|
||||
? _eg_save_cap_hint + (_eg_save_cap_hint >> 3) + 1024
|
||||
: (size_t)1 << 20);
|
||||
jb_puts(&b, "{\"nodes\":[");
|
||||
for (int64_t i = 0; i < g->node_count; i++) {
|
||||
if (i > 0) jb_putc(&b, ',');
|
||||
@@ -9788,6 +10001,10 @@ el_val_t engram_save(el_val_t path) {
|
||||
jb_putc(&b, '}');
|
||||
}
|
||||
jb_puts(&b, "]}");
|
||||
/* Remember the size BEFORE the write: the hint is about how much buffer
|
||||
* the next serialization needs, which is a property of the graph, not of
|
||||
* whether this particular fopen succeeded. */
|
||||
_eg_save_cap_hint = b.len;
|
||||
FILE* f = fopen(p, "wb");
|
||||
if (!f) { free(b.buf); return 0; }
|
||||
size_t w = fwrite(b.buf, 1, b.len, f);
|
||||
@@ -10714,8 +10931,11 @@ el_val_t engram_act_stats_json(void) {
|
||||
}
|
||||
/* 768, not 512: the write-back gauges added 2026-08-07 push the worst-case
|
||||
* rendering past the old bound, and snprintf would truncate the JSON into
|
||||
* an unparseable tail rather than fail loudly. */
|
||||
char buf[896];
|
||||
* an unparseable tail rather than fail loudly.
|
||||
* 1152, not 896: the five fan-effect gauges added 2026-08-11 add ~90 bytes
|
||||
* worst-case. Same reasoning — headroom is cheaper than a truncated tail
|
||||
* that every downstream JSON parser rejects as a whole. */
|
||||
char buf[1152];
|
||||
/* ctx_cos (2026-07-29): cos(query, context centroid) at the LAST
|
||||
* activate call, measured before the query was folded in. ~1.0 =
|
||||
* context aligned with current query; low = divergence (expected at
|
||||
@@ -10736,7 +10956,14 @@ el_val_t engram_act_stats_json(void) {
|
||||
* any climb means a write path is mangling text again. Cheap
|
||||
* (counted at creation) — the full census lives in
|
||||
* engram_text_health_json. (2026-08-08 self-review) */
|
||||
"\"txt_damaged\":%lld}",
|
||||
"\"txt_damaged\":%lld,"
|
||||
/* Fan-effect gauges (2026-08-11 self-review) — see the
|
||||
* _eg_act_fan_* definitions. fan_mean == 1.0 with fan_hits == 0
|
||||
* means the degree correction never bound on the last activation;
|
||||
* a mean drifting toward ENGRAM_FAN_MIN means traversal is
|
||||
* running through hubs and the correction is doing work. */
|
||||
"\"fan_mean\":%.4f,\"fan_min\":%.4f,\"fan_hits\":%lld,"
|
||||
"\"fan_steps\":%lld,\"fan_dref\":%.2f}",
|
||||
(long long)_eg_act_wm_evicted,
|
||||
(long long)_eg_act_breakthroughs,
|
||||
breaker_open, _eg_embed_consec_fail,
|
||||
@@ -10748,7 +10975,10 @@ el_val_t engram_act_stats_json(void) {
|
||||
(long long)_eg_hebb_wb_dropped,
|
||||
(long long)_eg_act_dup_seeds, (long long)_eg_act_dup_wm,
|
||||
(long long)_eg_act_dup_wm_global,
|
||||
(long long)_eg_txt_write_damaged);
|
||||
(long long)_eg_txt_write_damaged,
|
||||
(_eg_act_fan_n > 0 ? _eg_act_fan_sum / (double)_eg_act_fan_n : 1.0),
|
||||
_eg_act_fan_min, (long long)_eg_act_fan_hits,
|
||||
(long long)_eg_act_fan_n, _eg_act_fan_dref);
|
||||
return el_wrap_str(el_strdup(buf));
|
||||
}
|
||||
|
||||
@@ -10881,6 +11111,285 @@ el_val_t engram_label_df(el_val_t term) {
|
||||
return (el_val_t)df;
|
||||
}
|
||||
|
||||
/* ── Salient-term extraction (2026-08-13 self-review) ────────────────────────
|
||||
* THE MEASUREMENT. auto_term_empty_streak, the counter added by the 2026-08-06
|
||||
* review precisely to catch this class of silent death, read 50 and climbing.
|
||||
* Fifty consecutive curiosity scans in which the soul's dynamic seeding path
|
||||
* produced NOTHING and the loop fell back to its four hardcoded rotating
|
||||
* phrases. Dumping the live WM top says why in one look:
|
||||
*
|
||||
* Memory 0.390 memory:remembered
|
||||
* Memory 0.378 memory:remembered
|
||||
* Memory 0.377 memory:remembered
|
||||
* Memory 0.373 memory:remembered
|
||||
* Memory 0.370 memory:remembered
|
||||
*
|
||||
* Every slot at the top of working memory is a Memory node, and every Memory
|
||||
* node written by remember() carries the sentinel label "memory:remembered".
|
||||
* auto_term_try_slot reads the LABEL and only the label; the colon-no-space
|
||||
* guard (correctly) rejects sentinels as carrying no seed signal; so the
|
||||
* extractor had nothing to work with and returned empty, forever.
|
||||
*
|
||||
* THE ACTUAL DEFECT is not the sentinel guard — that guard is right. It is
|
||||
* that the extractor was built against Knowledge nodes, which have real
|
||||
* titles, and is structurally blind to the node type that in fact dominates
|
||||
* working memory. The label is not the content. A Memory node's topic is in
|
||||
* its text; the runtime just never looked there.
|
||||
*
|
||||
* WHY NOT ANOTHER GUARD. The extractor's whole history is guards: genre words
|
||||
* (07-23), quoted titles (07-25), English stopwords (07-30), label-df
|
||||
* (08-03). Four reviews, four blocklists, each written after watching a flood
|
||||
* happen. That is a losing shape, and 08-03 said so explicitly before adding
|
||||
* the fifth. The reason it keeps recurring is the algorithm underneath:
|
||||
* TAKE THE FIRST WORD, THEN CHECK WHETHER IT IS ACCEPTABLE. A first-word
|
||||
* extractor has no notion of term quality, so quality has to be bolted on as
|
||||
* rejection, and rejection can only encode the past.
|
||||
*
|
||||
* THE FIX is to invert it: score EVERY candidate token in the text and take
|
||||
* the argmax. Then term quality is the selection criterion rather than a
|
||||
* veto, and a bad token does not need to be on a list to lose — it only needs
|
||||
* a better token in the same text, which is the common case.
|
||||
*
|
||||
* SCORING (YAKE, Campos et al., Information Sciences 509:257-289, 2020 —
|
||||
* lightweight unsupervised single-document keyword extraction). YAKE scores
|
||||
* candidates on casing, position, frequency, context relatedness and sentence
|
||||
* dispersion, and beats RAKE/TextRank/SingleRank across twenty datasets. Two
|
||||
* of its five features port directly and cheaply; the other three are
|
||||
* within-document proxies for a corpus YAKE deliberately does not have. This
|
||||
* system DOES have the corpus — 12.7k labelled nodes — so real IDF is
|
||||
* substituted where YAKE has to approximate:
|
||||
*
|
||||
* score(t) = idf(t) · position(t) · casing(t)
|
||||
*
|
||||
* idf = ln((N+1)/(df+1)) real corpus specificity (Spärck
|
||||
* Jones 1972), strictly better than
|
||||
* YAKE's TF-based stand-in
|
||||
* position = 1/ln(e + i) YAKE T_Position: earlier tokens are
|
||||
* more topical. Keeps the old
|
||||
* first-word bias as a SOFT preference
|
||||
* instead of an absolute rule
|
||||
* casing = 1.30 acronym / 1.15 capitalised / 1.00 otherwise
|
||||
* YAKE T_Case
|
||||
*
|
||||
* THE min_df GATE. The df ceiling (08-03) rejects corpus-frequent markup and
|
||||
* sentinels. A floor was added alongside it for an independent reason: a term
|
||||
* appearing in ZERO labels cannot lexically reach anything, so it is a bad
|
||||
* seed however specific it looks.
|
||||
*
|
||||
* An earlier draft of this comment claimed the floor also subsumes the 73
|
||||
* hand-listed stopwords that 08-03 measured label-df as missing (Whose:0,
|
||||
* Would:0, Could:0). MEASURED, AND THAT CLAIM IS FALSE. Under word-boundary
|
||||
* df on the live store, function words are rare in labels but not absent:
|
||||
* about:2, whole:1, them:2, head:2. They clear a floor of 1. What actually
|
||||
* keeps them from winning is the argmax itself — they carry no position
|
||||
* advantage and lose to a topical term in the same text on every node
|
||||
* measured. The stopword list therefore STAYS as a real defense for the
|
||||
* Title-case cases, not as vestigial belt-and-braces. Recording the
|
||||
* correction rather than the tidier story: the floor buys lexical
|
||||
* reachability, the argmax buys quality, and the list still earns its keep.
|
||||
*
|
||||
* TABU IS APPLIED DURING THE ARGMAX, not after it. The old code picked a term
|
||||
* and then discarded it if it was tabu, which turned inhibition-of-return
|
||||
* into another source of empty scans. Excluding tabu terms from the candidate
|
||||
* set instead yields the best NON-TABU term, so rotation costs quality rather
|
||||
* than costing the whole scan.
|
||||
*
|
||||
* COST. One pass over g->nodes scoring all candidates at once (12.7k labels ×
|
||||
* <=32 candidates, short strings, good locality), twice per 30 s scan.
|
||||
*
|
||||
* POLICY LIVES IN THE SOUL. Thresholds arrive as arguments; the runtime
|
||||
* measures and ranks, awareness.el decides. Same split as engram_label_df.
|
||||
*
|
||||
* Returns the winning token, or "" when the node is missing, has no usable
|
||||
* text, or every candidate is gated out — "" remains the honest signal that
|
||||
* this slot yielded no seed, and auto_term_empty_streak still counts it. */
|
||||
#define ENGRAM_ST_MAXCAND 32
|
||||
#define ENGRAM_ST_TOKLEN 64
|
||||
#define ENGRAM_ST_SCANCHARS 400
|
||||
|
||||
/* Trim leading/trailing non-alphanumerics, then accept only tokens whose core
|
||||
* is alphanumeric plus '-' and '_' with at least 3 letters. This subsumes the
|
||||
* quoted-title guard (2026-07-25) and the "<!--" flood (2026-08-03)
|
||||
* structurally: markup and punctuation-bearing tokens never become
|
||||
* candidates, rather than being blocklisted after the fact. */
|
||||
static int eg_st_clean_token(const char* raw, size_t rawlen,
|
||||
char* out, size_t outcap) {
|
||||
size_t s = 0, e = rawlen;
|
||||
while (s < e && !isalnum((unsigned char)raw[s])) s++;
|
||||
while (e > s && !isalnum((unsigned char)raw[e - 1])) e--;
|
||||
size_t len = e - s;
|
||||
if (len < 4 || len >= outcap) return 0;
|
||||
int alpha = 0;
|
||||
for (size_t i = 0; i < len; i++) {
|
||||
unsigned char c = (unsigned char)raw[s + i];
|
||||
if (isalpha(c)) alpha++;
|
||||
else if (!isdigit(c) && c != '-' && c != '_') return 0;
|
||||
}
|
||||
if (alpha < 3) return 0;
|
||||
memcpy(out, raw + s, len);
|
||||
out[len] = '\0';
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* ENGRAM_ST_DEBUG=1 dumps the full scored candidate set to stderr. One
|
||||
* cached branch in production. This exists because the first live run of this
|
||||
* function returned five ALL-CAPS terms in a row and there was no way to see
|
||||
* whether that was the corpus or the casing weight without guessing — the
|
||||
* lesson this system keeps relearning. */
|
||||
static int _eg_st_debug(void) {
|
||||
static int v = -1;
|
||||
if (v < 0) { const char* e = getenv("ENGRAM_ST_DEBUG"); v = (e && *e == '1'); }
|
||||
return v;
|
||||
}
|
||||
|
||||
/* Word-boundary document frequency. engram_label_df uses istr_contains, i.e.
|
||||
* SUBSTRING matching, and that is the wrong estimator for term specificity on
|
||||
* short tokens: "them" hits inside "theme" and "anthem", "about" and "whole"
|
||||
* come back with df 2 and 1 rather than 0. That matters here specifically
|
||||
* because the min_df floor is what rejects English function words, and it can
|
||||
* only do that job if their df is honestly zero. Substring df quietly handed
|
||||
* them a survival ticket. Measured on the live store before this fix, "whole"
|
||||
* (df=1, idf=8.76) and "about" (df=2, idf=8.36) were outscoring real topical
|
||||
* terms and losing only on position — one node whose text happened to open
|
||||
* with a function word would have seeded on it.
|
||||
*
|
||||
* engram_label_df keeps substring semantics: it is a separate published
|
||||
* measure with existing callers, and changing it underneath them is not this
|
||||
* change's business. */
|
||||
static int eg_st_label_has_word(const char* hay, const char* word) {
|
||||
size_t wl = strlen(word);
|
||||
for (const char* p = hay; *p; p++) {
|
||||
if (strncasecmp(p, word, wl) != 0) continue;
|
||||
char before = (p == hay) ? '\0' : p[-1];
|
||||
char after = p[wl];
|
||||
if (before && (isalnum((unsigned char)before) || before == '_')) continue;
|
||||
if (after && (isalnum((unsigned char)after) || after == '_')) continue;
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* YAKE T_Case, adapted to this corpus. YAKE up-weights all-caps tokens
|
||||
* because in ordinary prose an acronym is rare and carries topic. That
|
||||
* assumption does not hold here: memory content written by remember()
|
||||
* conventionally OPENS WITH AN ALL-CAPS HEADER ("FRAME-ROUTER UPGRADE —
|
||||
* RESULTS", "THE GAP", "CENSUS"), so a flat acronym bonus systematically
|
||||
* hands the seed to whatever word the header happens to start with and lets
|
||||
* casing override the specificity signal it is supposed to only nudge.
|
||||
* Measured on the live store: the first five WM nodes returned PRIMING,
|
||||
* CONVERSATION, OCCUPATION, RELATIONAL, SELF-OCCUPATION — every one an
|
||||
* all-caps header word, none chosen on its merits.
|
||||
*
|
||||
* Genuine acronyms are SHORT (VBD, CCR, MCP, HTTP); shouty headers are long
|
||||
* words that happen to be capitalised. So the acronym bonus is restricted to
|
||||
* tokens of <= 5 characters, where all-caps is actually evidence of an
|
||||
* acronym rather than evidence of a heading. Longer all-caps tokens fall
|
||||
* through to the ordinary Title-case nudge — they still compete, they just
|
||||
* compete on specificity instead of on volume. */
|
||||
static double eg_st_casing(const char* t) {
|
||||
int upper = 0, lower = 0;
|
||||
size_t len = 0;
|
||||
for (const char* q = t; *q; q++, len++) {
|
||||
if (isupper((unsigned char)*q)) upper++;
|
||||
else if (islower((unsigned char)*q)) lower++;
|
||||
}
|
||||
if (lower == 0 && upper >= 2 && len <= 5) return 1.30; /* acronym */
|
||||
if (isupper((unsigned char)t[0])) return 1.15; /* Title/hdr */
|
||||
return 1.0;
|
||||
}
|
||||
|
||||
el_val_t engram_salient_term(el_val_t node_id, el_val_t max_df_v,
|
||||
el_val_t min_df_v, el_val_t tabu_v) {
|
||||
EngramStore* g = engram_get();
|
||||
int64_t ix = engram_find_node_index(EL_CSTR(node_id));
|
||||
if (ix < 0) return el_wrap_str(el_strdup(""));
|
||||
EngramNode* n = &g->nodes[ix];
|
||||
|
||||
int64_t max_df = (int64_t)max_df_v;
|
||||
int64_t min_df = (int64_t)min_df_v;
|
||||
if (max_df <= 0) max_df = g->node_count;
|
||||
if (min_df < 0) min_df = 0;
|
||||
const char* tabu = EL_CSTR(tabu_v);
|
||||
|
||||
/* Source selection. Prefer the label — it is a curated title when it is
|
||||
* one. Fall back to content when the label is absent or a sentinel
|
||||
* ("memory:remembered": a colon and no space). This single line is what
|
||||
* makes Memory nodes visible to the extractor at all. */
|
||||
const char* src = n->label;
|
||||
if (!src || !*src) {
|
||||
src = n->content;
|
||||
} else if (strchr(src, ':') != NULL && strchr(src, ' ') == NULL) {
|
||||
src = n->content;
|
||||
}
|
||||
if (!src || !*src) return el_wrap_str(el_strdup(""));
|
||||
|
||||
/* Collect distinct candidates from the head of the text. */
|
||||
char cand[ENGRAM_ST_MAXCAND][ENGRAM_ST_TOKLEN];
|
||||
int pos[ENGRAM_ST_MAXCAND];
|
||||
int64_t df[ENGRAM_ST_MAXCAND];
|
||||
int ncand = 0, tokidx = 0;
|
||||
|
||||
const char* p = src;
|
||||
const char* lim = src + strnlen(src, ENGRAM_ST_SCANCHARS);
|
||||
while (p < lim && ncand < ENGRAM_ST_MAXCAND) {
|
||||
while (p < lim && isspace((unsigned char)*p)) p++;
|
||||
if (p >= lim) break;
|
||||
const char* tk = p;
|
||||
while (p < lim && !isspace((unsigned char)*p)) p++;
|
||||
char buf[ENGRAM_ST_TOKLEN];
|
||||
int slot = tokidx++;
|
||||
if (!eg_st_clean_token(tk, (size_t)(p - tk), buf, sizeof(buf))) continue;
|
||||
|
||||
/* Tabu exclusion, applied here so the argmax runs over eligible
|
||||
* terms only. tabu arrives pipe-delimited: "|t0|t1|t2|t3|". */
|
||||
if (tabu && *tabu) {
|
||||
char pat[ENGRAM_ST_TOKLEN + 2];
|
||||
snprintf(pat, sizeof(pat), "|%s|", buf);
|
||||
if (istr_contains(tabu, pat)) continue;
|
||||
}
|
||||
int dup = 0;
|
||||
for (int i = 0; i < ncand; i++)
|
||||
if (strcasecmp(cand[i], buf) == 0) { dup = 1; break; }
|
||||
if (dup) continue;
|
||||
|
||||
memcpy(cand[ncand], buf, strlen(buf) + 1);
|
||||
pos[ncand] = slot;
|
||||
df[ncand] = 0;
|
||||
ncand++;
|
||||
}
|
||||
if (ncand == 0) return el_wrap_str(el_strdup(""));
|
||||
|
||||
/* One pass over the store, all candidates at once. */
|
||||
for (int64_t i = 0; i < g->node_count; i++) {
|
||||
const char* lbl = g->nodes[i].label;
|
||||
if (!lbl || !*lbl) continue;
|
||||
for (int c = 0; c < ncand; c++)
|
||||
if (eg_st_label_has_word(lbl, cand[c])) df[c]++;
|
||||
}
|
||||
|
||||
/* Argmax over idf · position · casing, subject to the df band. */
|
||||
int best = -1;
|
||||
double best_score = 0.0;
|
||||
for (int c = 0; c < ncand; c++) {
|
||||
if (df[c] > max_df) continue;
|
||||
if (df[c] < min_df) continue;
|
||||
double idf = log(((double)g->node_count + 1.0) / ((double)df[c] + 1.0));
|
||||
if (idf <= 0.0) continue;
|
||||
double position = 1.0 / log(2.718281828459045 + (double)pos[c]);
|
||||
double casing = eg_st_casing(cand[c]);
|
||||
double score = idf * position * casing;
|
||||
if (_eg_st_debug()) {
|
||||
fprintf(stderr, " cand %-24s df=%-5lld idf=%.2f pos=%d p=%.2f "
|
||||
"case=%.2f score=%.3f\n",
|
||||
cand[c], (long long)df[c], idf, pos[c], position,
|
||||
casing, score);
|
||||
}
|
||||
if (score > best_score) { best_score = score; best = c; }
|
||||
}
|
||||
if (best < 0) return el_wrap_str(el_strdup(""));
|
||||
return el_wrap_str(el_strdup(cand[best]));
|
||||
}
|
||||
|
||||
/* engram_embed_backfill — explicitly drive the lazy embedding backfill.
|
||||
* (2026-07-25 self-review.) The per-activate backfill (8 nodes/call) only
|
||||
* runs inside engram_activate, and on the authoritative HTTP store nothing
|
||||
|
||||
@@ -628,6 +628,13 @@ el_val_t engram_hebb_drain_json(el_val_t max);
|
||||
/* Document frequency of a term across node labels — term-specificity signal
|
||||
* for curiosity seed selection. (2026-08-03 self-review.) */
|
||||
el_val_t engram_label_df(el_val_t term);
|
||||
/* Best curiosity seed from one node: argmax over idf·position·casing across
|
||||
* the candidate tokens of its label, falling back to its content when the
|
||||
* label is a sentinel. Excludes pipe-delimited tabu terms during selection
|
||||
* and gates candidates to the df band [min_df, max_df]. Returns "" when
|
||||
* nothing qualifies. (2026-08-13 self-review.) */
|
||||
el_val_t engram_salient_term(el_val_t node_id, el_val_t max_df,
|
||||
el_val_t min_df, el_val_t tabu);
|
||||
el_val_t engram_embed_backfill(el_val_t count);
|
||||
el_val_t engram_list_layers_json(void);
|
||||
/* Working memory introspection — count, mean weight, and top-N snapshot.
|
||||
|
||||
Reference in New Issue
Block a user