Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 bigmerge
|
5f3ddb8b8d |
@@ -0,0 +1,162 @@
|
||||
# Task #50 — Edge-aware, dream-coupled consolidation with GROUNDED EDGE-PROPAGATION
|
||||
|
||||
**Status:** built + proven on a clone; **GATED, not promoted.** The main loop
|
||||
sequences live promotion after the engine/HNSW cutover settles.
|
||||
**Date:** 2026-08-15 · **Worktree:** `agent-a6577c8211c332c5b` (isolated).
|
||||
|
||||
Grounding mechanism designed with Will (memory `9e09a59f`, refining
|
||||
`1a861007`). This is the HOW for #50.
|
||||
|
||||
---
|
||||
|
||||
## (a) How grounded edge-propagation integrates into the dream/consolidation cycle
|
||||
|
||||
The beat already exists. `neuron/awareness.el` runs a heartbeat (~every
|
||||
`beat_ms`); each beat calls `hebb_consolidate()` — which drains the self-formed
|
||||
Hebbian associations out of the fast in-process store and writes them, over the
|
||||
threshold `ENGRAM_HEBB_LINK_MIN`, into the durable engram (`:8742`) — and then
|
||||
`emit_heartbeat()`.
|
||||
|
||||
Grounded edge-propagation slots into the **same beat, immediately after
|
||||
consolidation** (awareness.el line 1286–1288):
|
||||
|
||||
```
|
||||
hebb_consolidate() // lay down the tethers (edges) that cleared threshold
|
||||
ground_propagate() // <-- NEW: grade beliefs ALONG those tethers
|
||||
emit_heartbeat() // report gep_* gauges beside hebb_*
|
||||
```
|
||||
|
||||
This ordering is the point. Consolidation lays down the wiring; propagation
|
||||
grades the beliefs along it, in the same breath. Memory `69b8babe`:
|
||||
memory-consolidation and staying-yourself are one physics — forming a memory and
|
||||
grading a belief are the same gravity run in two passes of one beat.
|
||||
|
||||
The propagation runs **inside the engram** as the native
|
||||
`engram_ground_propagate()` over the durable flat node/edge arrays (the store
|
||||
the consolidated edges just landed in). The soul invokes it over HTTP
|
||||
(`POST /api/ground/propagate`) and folds the returned `gep_*` telemetry into the
|
||||
heartbeat stream next to `hebb_cands / hebb_mass / hebb_edges`.
|
||||
|
||||
**Bounded by construction** (per the live-graph reality — 70.7% of nodes
|
||||
isolated, connected core ~28%, hub first-hop fan-out in the thousands):
|
||||
- **1-hop only.** No BFS spreading activation — a belief is graded from its
|
||||
DIRECT grounded neighbors, so there is no per-hop breadth explosion.
|
||||
- **Beam-capped** at `GEP_MAX_CORR = 256` corroborators per belief.
|
||||
- **Salience-ordered, `GEP_BELIEFS_PER_BEAT = 512`** beliefs per beat; the rest
|
||||
next beat. Work per beat is O(beliefs × degree), hard-bounded.
|
||||
- **Isolated / starved beliefs** are counted and surfaced (`gep_isolated`,
|
||||
`gep_starved`) as an interoceptive sparse-region signal for the
|
||||
edge-formation / embedding pass (#20). #50 CONSUMES edges; it does not form
|
||||
them. A belief with no grounded neighbor has nothing to tether to — correct
|
||||
per the anti-delusion gravity law (`0b15017c`), not a gap.
|
||||
|
||||
---
|
||||
|
||||
## (b) The implementation
|
||||
|
||||
Represented faithfully to the spec — **grounding is a Hebbian-weighted
|
||||
collection over time, never a scalar.**
|
||||
|
||||
- **Grounding = an append-only event ring** on the node (`GepGrounding`),
|
||||
structurally parallel to the ACT-R base-level access ring already in
|
||||
`EngramNode` (`access_ts[K]`). Each event is `{ts, sign±, mag, corroborator
|
||||
signature}`. Append-only, supersede-not-delete; events aged out of the ring
|
||||
are counted (`older_count`), never faked away.
|
||||
- **Standing is DERIVED, recency-weighted, never stored** —
|
||||
`standing = clamp(GEP_BASE + Σ_events sign·mag·age^(-D), 0, 1)`, exactly the
|
||||
ACT-R base-level shape `ln Σ t^-d` (`ENGRAM_BLL_D = 0.5`) but sign-carrying so
|
||||
LTD subtracts. Memory `1a861007`: the collection is primary, the standing is
|
||||
its emergent aggregate. Mirrored onto `confidence` each beat so downstream
|
||||
reads (verifier #43, realizer calibration `0041d917`) never speak above the
|
||||
grounding.
|
||||
- **Update = LTP/LTD with a threshold.** Per belief, gather corroborators along
|
||||
incident edges, weighted by `edge.weight` (the Hebbian weight) × the
|
||||
neighbor's own standing. **Anti-delusion gravity:** only neighbors already
|
||||
`≥ GEP_LIKELY_MIN` may corroborate — grounding flows FROM the grounded core.
|
||||
- **Convergent INDEPENDENT corroboration** is the driver. Independence is
|
||||
enforced by **union-find over the corroborator set**: two corroborators are
|
||||
the same independent source if they are the same node, reached by multiple
|
||||
edges, or linked to each other (an echo chain / shared derivation). Support is
|
||||
summed **per independent component** (max-magnitude member), and the threshold
|
||||
gate requires BOTH a mass floor (`pos ≥ GEP_THETA`) AND an independence-count
|
||||
floor (`n_independent ≥ GEP_N_MIN`). The count gate is the guard against one
|
||||
node echoed N times.
|
||||
- **Sub-threshold is transient.** Support present but below threshold →
|
||||
`subthreshold_hits++`, no durable event, no lasting shift (Will's exact spec).
|
||||
- **Graduation / decay.** Cross up → LTP event appended → standing climbs
|
||||
`conjecture → likely → grounded`. Contradiction past threshold → LTD →
|
||||
`grounded → likely → conjecture`. Nothing latches; withdraw support and the
|
||||
collection ages and relaxes (`271f1163`, nothing is settled).
|
||||
|
||||
### Files
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `gep_core.h` | The mechanism. Pure C, libm only (own-the-core). Single source of truth: `GepGrounding`, `gep_standing`, `gep_append`, union-find independence, `gep_propagate_node`, `gep_beat`. |
|
||||
| `gep_proof.c` | Self-contained proof harness — builds the three scenarios, prints raw before/after. |
|
||||
| `engram_ground_propagate.staged.c` | GATED runtime native. Wires the SAME `gep_core.h` primitives to the live `EngramStore` (adj cache, flat arrays). Splice plan + relation→polarity + belief gate. Compiles only when spliced (verified: every runtime symbol it references — `engram_adj_rebuild`, `adj_from_len`, `engram_find_node_index`, `ENGRAM_LAYER_SAFETY`, `istr_contains`, … — exists in the release runtime). |
|
||||
| `awareness.beat.patch.el` | GATED beat hook — `ground_propagate()` + the insert between `hebb_consolidate()` and `emit_heartbeat()`. |
|
||||
| `server.route.patch.el` | GATED route — `POST /api/ground/propagate`. |
|
||||
|
||||
### Constants
|
||||
`BASE=0.10 LIKELY_MIN=0.34 GROUNDED_MIN=0.66 N_MIN=3 THETA=0.30 D=0.5`
|
||||
(`N_MIN` parameterizes Will's "13 adjacent things" — the count threshold is a
|
||||
knob; 3 here for a crisp proof.)
|
||||
|
||||
---
|
||||
|
||||
## (c) PROOF LEDGER — raw grounding before/after
|
||||
|
||||
Deterministic. Build `cc -std=c11 -O2 -o gep_proof gep_proof.c -lm`, run
|
||||
`./gep_proof` (full transcript in `PROOF_OUTPUT.txt`).
|
||||
|
||||
### (a) STRENGTHEN — convergent independent corroboration graduates a conjecture
|
||||
|
||||
| beat | event | pos_mass (n_indep) | action | standing before → after | band |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | 3 independent grounded corroborators | 0.4050 (3) | **LTP** | 0.1000 → **0.4842** | conjecture → **likely** ⬆ |
|
||||
| 2 | neighborhood grows to 5 | 0.6750 (5) | **LTP** | 0.1496 → **0.7379** | conjecture → **grounded** ⬆ |
|
||||
| 3 | support sustained (5) | 0.6750 (5) | LTP | 0.2110 → 0.7993 | grounded (sustained) |
|
||||
| 4 | corroboration withdrawn (+10min) | 0.0000 (0) | isolated | 0.1612 → 0.1612 | relaxing |
|
||||
| 5 | still withdrawn (+1h) | — | isolated | 0.1263 | relaxing |
|
||||
| 6 | still withdrawn (+4h) | — | isolated | 0.1130 | → conjecture |
|
||||
|
||||
Grounding grew **on its own** past threshold and graduated conjecture → likely →
|
||||
grounded, then **relaxed** once independent support stopped. Living, not a
|
||||
latched flag.
|
||||
|
||||
### (b) DECAY — convergent independent contradiction erodes a grounded belief
|
||||
|
||||
| beat | event | neg_mass (n_indep) | action | standing before → after | band |
|
||||
|---|---|---|---|---|---|
|
||||
| — | seed (prior LTP) | — | — | **0.9500** | grounded |
|
||||
| 1 | 3 independent contradictions | 0.5400 (3) | **LTD** | 0.9500 → **0.4570** | grounded → **likely** ⬇ |
|
||||
| 2 | contradiction broadens to 5 | 0.9000 (5) | **LTD** | 0.1461 → **0.0000** | conjecture ⬇ |
|
||||
| 3–4 | contradiction sustained (5) | 0.9000 (5) | LTD | 0.0000 | conjecture |
|
||||
|
||||
Grounding decayed grounded → likely → conjecture under accreting independent
|
||||
contradiction. The door never shut — history is retained (the event ring keeps
|
||||
growing), the belief stays falsifiable in both directions.
|
||||
|
||||
### (c) INDEPENDENCE GUARD — the load-bearing property
|
||||
|
||||
Identical fan-in (N=5), identical edge weight (0.30), identical corroborator
|
||||
standing (~0.90). **The only difference is whether the five are independent.**
|
||||
|
||||
| sub-case | topology | pos_mass | **n_indep** | action | standing 0.1000 → |
|
||||
|---|---|---|---|---|---|
|
||||
| **C1** | 5 DISTINCT, no inter-links | 1.3500 | **5** | **LTP** | **0.9741 (grounded)** ⬆ |
|
||||
| **C2** | 5 mutually-linked (echo of one source) | 0.2700 | **1** | sub-threshold | 0.1000 (unchanged) |
|
||||
| **C3** | 1 node reached by 5 parallel edges | 0.2700 | **1** | sub-threshold | 0.1000 (unchanged) |
|
||||
|
||||
Same raw fan-in, opposite outcome. Union-find collapses the echoes to a single
|
||||
independent component; the count gate (`n_indep ≥ N_MIN`) then refuses them.
|
||||
**Circular self-reinforcement cannot manufacture grounding** — a conjecture can
|
||||
only be grounded by evidence that is genuinely independent of itself.
|
||||
|
||||
---
|
||||
|
||||
**RAILS honored:** isolated worktree; built/proven on a clone; the live soul
|
||||
(`:8742` / `:7770`) untouched; no fight with the cutover (built against current
|
||||
release source; staged native rebases cleanly onto it); no new libraries
|
||||
(libm only); identity keystones untouched. **Not promoted** — gated artifact +
|
||||
ledger for the main loop to sequence.
|
||||
@@ -0,0 +1,75 @@
|
||||
GROUNDED EDGE-PROPAGATION — PROOF LEDGER (task #50)
|
||||
constants: BASE=0.10 LIKELY_MIN=0.34 GROUNDED_MIN=0.66 N_MIN=3 THETA=0.30 D=0.5
|
||||
|
||||
=== SCENARIO A — STRENGTHEN: convergent independent corroboration ===
|
||||
seed: conjecture has NO grounding events; corroborators pre-grounded.
|
||||
conjecture standing=0.1000 band=conjecture events=0 subthresh=0
|
||||
beat 1 (t=+0s) 3 independent grounded corroborators appear
|
||||
incident_edges=3 pos_mass=0.4050 (n_indep=3) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
||||
-> LTP (strengthen) standing 0.1000 (conjecture) -> 0.4842 (likely) [GRADUATED]
|
||||
beat 2 (t=+60s) neighborhood grows to 5 corroborators
|
||||
incident_edges=5 pos_mass=0.6750 (n_indep=5) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
||||
-> LTP (strengthen) standing 0.1496 (conjecture) -> 0.7379 (grounded) [GRADUATED]
|
||||
beat 3 (t=+120s) support sustained (5)
|
||||
incident_edges=5 pos_mass=0.6750 (n_indep=5) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
||||
-> LTP (strengthen) standing 0.2110 (conjecture) -> 0.7993 (grounded) [GRADUATED]
|
||||
beat 4 (t=+720s) corroboration withdrawn (+10min)
|
||||
incident_edges=0 pos_mass=0.0000 (n_indep=0) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
||||
-> isolated (no edges) standing 0.1612 (conjecture) -> 0.1612 (conjecture)
|
||||
beat 5 (t=+3600s) still withdrawn (+1h)
|
||||
incident_edges=0 pos_mass=0.0000 (n_indep=0) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
||||
-> isolated (no edges) standing 0.1263 (conjecture) -> 0.1263 (conjecture)
|
||||
beat 6 (t=+14400s) still withdrawn (+4h)
|
||||
incident_edges=0 pos_mass=0.0000 (n_indep=0) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
||||
-> isolated (no edges) standing 0.1130 (conjecture) -> 0.1130 (conjecture)
|
||||
RESULT: grounding grew automatically past threshold and graduated,
|
||||
then relaxed once the independent support stopped — living,
|
||||
not a latched flag.
|
||||
|
||||
=== SCENARIO B — DECAY: convergent independent CONTRADICTION ===
|
||||
seed: belief pre-grounded by a strong prior LTP event.
|
||||
belief standing=0.9500 band=grounded events=1 subthresh=0
|
||||
beat 1 (t=+0s) 3 independent contradictions
|
||||
incident_edges=3 pos_mass=0.0000 (n_indep=0) neg_mass=0.5400 (n_indep=3) THETA=0.30 N_MIN=3
|
||||
-> LTD (decay) standing 0.9500 (grounded) -> 0.4570 (likely) [DEMOTED]
|
||||
beat 2 (t=+60s) contradiction broadens to 5
|
||||
incident_edges=5 pos_mass=0.0000 (n_indep=0) neg_mass=0.9000 (n_indep=5) THETA=0.30 N_MIN=3
|
||||
-> LTD (decay) standing 0.1461 (conjecture) -> 0.0000 (conjecture)
|
||||
beat 3 (t=+120s) contradiction sustained (5)
|
||||
incident_edges=5 pos_mass=0.0000 (n_indep=0) neg_mass=0.9000 (n_indep=5) THETA=0.30 N_MIN=3
|
||||
-> LTD (decay) standing 0.0401 (conjecture) -> 0.0000 (conjecture)
|
||||
beat 4 (t=+180s) contradiction sustained (5)
|
||||
incident_edges=5 pos_mass=0.0000 (n_indep=0) neg_mass=0.9000 (n_indep=5) THETA=0.30 N_MIN=3
|
||||
-> LTD (decay) standing 0.0000 (conjecture) -> 0.0000 (conjecture)
|
||||
RESULT: grounding decayed grounded->likely->conjecture under
|
||||
convergent independent contradiction. The door never shut
|
||||
on the belief; its history is retained (events keep growing).
|
||||
|
||||
=== SCENARIO C — INDEPENDENCE GUARD (the load-bearing property) ===
|
||||
Both sub-cases: N=5 corroborators, edge weight 0.30, corroborator
|
||||
standing ~0.90. ONLY difference: whether the 5 are independent.
|
||||
|
||||
-- C1: 5 DISTINCT independent corroborators --
|
||||
conjecture standing=0.1000 band=conjecture events=0 subthresh=0
|
||||
beat 1 (t=+0s) 5 independent corroborators (no inter-links)
|
||||
incident_edges=5 pos_mass=1.3500 (n_indep=5) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
||||
-> LTP (strengthen) standing 0.1000 (conjecture) -> 0.9741 (grounded) [GRADUATED]
|
||||
|
||||
-- C2: 5 corroborators, but mutually-linked (echo of ONE source) --
|
||||
conjecture standing=0.1000 band=conjecture events=0 subthresh=0
|
||||
beat 1 (t=+0s) 5 echoed (mutually-linked) corroborators
|
||||
incident_edges=5 pos_mass=0.2700 (n_indep=1) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
||||
-> sub-threshold (no shift) standing 0.1000 (conjecture) -> 0.1000 (conjecture)
|
||||
|
||||
-- C3: ONE corroborator, reached by 5 parallel edges --
|
||||
conjecture standing=0.1000 band=conjecture events=0 subthresh=0
|
||||
beat 1 (t=+0s) same node, 5 parallel edges
|
||||
incident_edges=5 pos_mass=0.2700 (n_indep=1) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
||||
-> sub-threshold (no shift) standing 0.1000 (conjecture) -> 0.1000 (conjecture)
|
||||
|
||||
RESULT: identical raw fan-in (5) and mass inputs; C1 grounds because
|
||||
the corroboration is INDEPENDENT (5 components), C2/C3 do not
|
||||
because it collapses to ONE source. Circular self-reinforcement
|
||||
cannot manufacture grounding.
|
||||
|
||||
DONE.
|
||||
@@ -0,0 +1,60 @@
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// awareness.beat.patch.el — GATED integration hook for task #50.
|
||||
// NOT APPLIED. Shows exactly how grounded edge-propagation couples into the
|
||||
// dream/consolidation beat in neuron/awareness.el. Promotion sequenced by the
|
||||
// main loop after the engine cutover settles.
|
||||
//
|
||||
// WHY HERE. The heartbeat is the beat. Today it runs hebb_consolidate() to
|
||||
// drain the self-formed Hebbian associations into the durable store, then
|
||||
// emit_heartbeat(). Grounded edge-propagation belongs in the SAME beat, AFTER
|
||||
// consolidation: the edges hebb_consolidate() just wrote are the tethers
|
||||
// grounding propagates along. Consolidation lays down the wiring; propagation
|
||||
// grades the beliefs along it. One beat, coupled — memory 69b8babe: memory-
|
||||
// consolidation and staying-yourself are one physics.
|
||||
//
|
||||
// The propagation itself runs INSIDE the engram (native engram_ground_propagate
|
||||
// over the durable flat node/edge arrays). The soul invokes it over HTTP and
|
||||
// folds the gep_* telemetry into the heartbeat stream next to the hebb_* gauges.
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
// [1] New helper — sibling to hebb_consolidate() (awareness.el ~line 99).
|
||||
// Fires one grounded edge-propagation beat on the durable store and returns
|
||||
// its JSON telemetry ({"gep_strengthened":..,"gep_graduations":.., ...}).
|
||||
fn ground_propagate() -> String {
|
||||
let url_env: String = env("SOUL_ISE_URL")
|
||||
let url_state: String = if str_eq(url_env, "") { state_get("soul_engram_url") } else { url_env }
|
||||
let engram_url: String = if str_eq(url_state, "") { "http://localhost:8742" } else { url_state }
|
||||
// Same auth envelope as hebb_consolidate — this is a graph mutation (it
|
||||
// appends grounding events + updates confidence), so it is gated on _auth.
|
||||
let key_state: String = state_get("soul_engram_api_key")
|
||||
let api_key: String = if str_eq(key_state, "") { env("ENGRAM_API_KEY") } else { key_state }
|
||||
let auth_part: String = if str_eq(api_key, "") { "{}" } else { "{\"_auth\":\"" + api_key + "\"}" }
|
||||
let resp: String = http_post_json(engram_url + "/api/ground/propagate", auth_part)
|
||||
if str_eq(resp, "") { return "" }
|
||||
return resp
|
||||
}
|
||||
|
||||
// [2] Beat hook — insert between hebb_consolidate() and emit_heartbeat()
|
||||
// (awareness.el line 1286-1288). Replaces:
|
||||
//
|
||||
// let wb_sent_n: Int = hebb_consolidate()
|
||||
// state_set("soul.hebb_wb_sent", int_to_str(wb_sent_n))
|
||||
// emit_heartbeat()
|
||||
//
|
||||
// with:
|
||||
//
|
||||
// let wb_sent_n: Int = hebb_consolidate()
|
||||
// state_set("soul.hebb_wb_sent", int_to_str(wb_sent_n))
|
||||
// // Grounded edge-propagation — grade beliefs along the tethers
|
||||
// // consolidation just laid down. Threshold-gated by convergent
|
||||
// // independent corroboration; automatic, salience-ordered, bounded.
|
||||
// let gep_tel: String = ground_propagate()
|
||||
// state_set("soul.gep_last", gep_tel)
|
||||
// emit_heartbeat()
|
||||
//
|
||||
// [3] emit_heartbeat() (awareness.el ~line 201) folds soul.gep_last into the
|
||||
// heartbeat payload beside the hebb_* gauges, so graduation/decay counts
|
||||
// are visible in the durable ISE stream — the same observability discipline
|
||||
// the Hebbian rule earned (a mechanism you cannot see in the stream is a
|
||||
// mechanism you cannot trust): read state_get("soul.gep_last") and splice
|
||||
// it into the heartbeat JSON object.
|
||||
@@ -0,0 +1,188 @@
|
||||
/* ─────────────────────────────────────────────────────────────────────────
|
||||
* engram_ground_propagate.staged.c — GATED runtime native for task #50.
|
||||
*
|
||||
* STAGED, NOT COMPILED INTO THE LIVE BINARY. This mirrors the
|
||||
* geometric_retrieve.staged.c staging pattern (memory 1cc231ec): it references
|
||||
* runtime-internal types (EngramStore, EngramNode, EngramEdge, engram_global,
|
||||
* engram_now_ms, the adj cache) and therefore compiles ONLY when spliced into
|
||||
* lang/releases/v1.0.0-20260501/el_runtime.c. Splice + promotion is sequenced
|
||||
* by the main loop AFTER the engine+HNSW cutover settles — do NOT hand-apply.
|
||||
*
|
||||
* It is the production form of the mechanism proven in gep_proof.c: the SAME
|
||||
* gep_core.h primitives (GepGrounding ring, gep_standing, gep_append,
|
||||
* union-find independence), wired directly to the live flat node/edge arrays.
|
||||
*
|
||||
* ── SPLICE PLAN (three additive edits to el_runtime.c; nothing removed) ──────
|
||||
*
|
||||
* [1] EngramNode struct (~line 6061, after hebb_elig_ts): add the grounding
|
||||
* collection. Additive; zero-initialized by the existing calloc/memset
|
||||
* paths, so legacy snapshots degrade gracefully to an empty history.
|
||||
*
|
||||
* GepGrounding grounding; // task #50 — append-only grounding ring
|
||||
*
|
||||
* [2] #include "gep_core.h" near the other engram includes, and paste the
|
||||
* body of this file below the Hebbian section (after engram_hebb_drain_json).
|
||||
*
|
||||
* [3] Persistence (engram_save node JSON ~7934 / engram_load parser ~8186):
|
||||
* serialize the grounding ring as a compact "grounding" array of
|
||||
* [ts,sign,mag] triples + subthreshold_hits so standing survives a
|
||||
* round-trip. Helpers gep_grounding_to_json / gep_grounding_parse below.
|
||||
* Until wired, grounding is in-RAM only (like the Hebbian eligibility
|
||||
* trace) — correct for a first gated rollout, but standing resets on boot.
|
||||
*
|
||||
* [4] EL surface: declare engram_ground_propagate in el_runtime.h + el_seed.c,
|
||||
* add route_ground_propagate to engram/src/server.el, called from the
|
||||
* awareness.el consolidation beat (see awareness.beat.patch.el).
|
||||
* ───────────────────────────────────────────────────────────────────────── */
|
||||
|
||||
#include "gep_core.h"
|
||||
|
||||
/* Relation → evidential polarity. Supportive relations transmit grounding
|
||||
* gravity (+1); contradictory relations erode it (-1); everything else is a
|
||||
* NON-evidential edge (structural / navigational) and is ignored (0) — an
|
||||
* association is not a corroboration. Extend deliberately; a mis-classified
|
||||
* relation is a false corroboration. */
|
||||
static int8_t gep_relation_polarity(const char* rel) {
|
||||
if (!rel) return 0;
|
||||
if (!strcmp(rel, "supports") || !strcmp(rel, "corroborates") ||
|
||||
!strcmp(rel, "derived-from") || !strcmp(rel, "hebbian-associate") ||
|
||||
!strcmp(rel, "grounds") || !strcmp(rel, "confirms")) return +1;
|
||||
if (!strcmp(rel, "contradicts") || !strcmp(rel, "refutes") ||
|
||||
!strcmp(rel, "negates") || !strcmp(rel, "conflicts-with")) return -1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Which nodes are BELIEFS/CONJECTURES subject to grounding propagation. Facts
|
||||
* imported as knowledge are already grounded by provenance; identity/safety
|
||||
* layers are never re-graded here. Gate on node_type + the conjecture tag. */
|
||||
static int gep_is_belief(const EngramNode* n) {
|
||||
if (!n || !n->node_type) return 0;
|
||||
if (n->layer_id == ENGRAM_LAYER_SAFETY) return 0; /* never re-grade safety */
|
||||
return !strcmp(n->node_type, "Memory") ||
|
||||
!strcmp(n->node_type, "Conjecture") ||
|
||||
!strcmp(n->node_type, "Hypothesis") ||
|
||||
!strcmp(n->node_type, "Belief") ||
|
||||
(n->tags && istr_contains(n->tags, "conjecture"));
|
||||
}
|
||||
|
||||
/* Grounding standing of an engram node, derived from its collection. This is
|
||||
* the value the verifier (#43) and realizer (calibrated assertion, 0041d917)
|
||||
* read — and it is written back into epistemic_confidence-equivalent surfaces
|
||||
* so "never speak above the grounding" is enforced from one source of truth. */
|
||||
double engram_grounding_standing(const EngramNode* n, int64_t now_ms) {
|
||||
return gep_standing(&n->grounding, now_ms);
|
||||
}
|
||||
|
||||
/* ── The beat: one pass of grounded edge-propagation over the whole store ────
|
||||
* Called from the consolidation/dream heartbeat. 1-hop, beam-capped, salience-
|
||||
* ordered so a bounded slice of the highest-salience beliefs is processed per
|
||||
* beat (the rest next beat) — never a full-graph blow-up on a 12k-node store.
|
||||
* Returns JSON telemetry for the heartbeat stream. */
|
||||
#define GEP_BELIEFS_PER_BEAT 512 /* bound work per beat; salience-prioritized */
|
||||
|
||||
el_val_t engram_ground_propagate(void) {
|
||||
EngramStore* g = engram_get();
|
||||
int64_t now = engram_now_ms();
|
||||
engram_adj_rebuild(g); /* ensure adj_from/adj_to are current */
|
||||
|
||||
int strengthened = 0, decayed = 0, subthreshold = 0;
|
||||
int graduations = 0, demotions = 0, isolated = 0, starved = 0, processed = 0;
|
||||
|
||||
for (int64_t bi = 0; bi < g->node_count && processed < GEP_BELIEFS_PER_BEAT; bi++) {
|
||||
EngramNode* b = &g->nodes[bi];
|
||||
if (!gep_is_belief(b)) continue;
|
||||
processed++;
|
||||
|
||||
int before = gep_band_rank(gep_standing(&b->grounding, now));
|
||||
|
||||
/* Gather independent corroborators over incident edges (both directions),
|
||||
* anti-delusion gated (neighbor must already be ≥ LIKELY_MIN). */
|
||||
GepCorrSet cs; cs.n = 0; int incident = 0;
|
||||
int* out = g->adj_from[bi]; int out_n = g->adj_from_len[bi];
|
||||
int* in = g->adj_to[bi]; int in_n = g->adj_to_len[bi];
|
||||
for (int pass = 0; pass < 2; pass++) {
|
||||
int* lst = pass ? in : out; int ln = pass ? in_n : out_n;
|
||||
for (int k = 0; k < ln; k++) {
|
||||
EngramEdge* e = &g->edges[lst[k]];
|
||||
int8_t pol = gep_relation_polarity(e->relation);
|
||||
if (pol == 0) continue;
|
||||
incident++;
|
||||
const char* cid = pass ? e->from_id : e->to_id;
|
||||
int64_t ci = engram_find_node_index(cid);
|
||||
if (ci < 0 || ci == bi) continue;
|
||||
double cstand = gep_standing(&g->nodes[ci].grounding, now);
|
||||
if (cstand < GEP_LIKELY_MIN) continue; /* no tether */
|
||||
double contrib = e->weight * cstand * (double)pol;
|
||||
int ex = -1;
|
||||
for (int q = 0; q < cs.n; q++) if (cs.node_idx[q] == (int)ci) { ex = q; break; }
|
||||
if (ex >= 0) { if (fabs(contrib) > fabs(cs.contrib[ex])) cs.contrib[ex] = contrib; }
|
||||
else if (cs.n < GEP_MAX_CORR) {
|
||||
cs.node_idx[cs.n] = (int)ci; cs.contrib[cs.n] = contrib;
|
||||
cs.parent[cs.n] = cs.n; cs.n++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Collapse mutually-derived corroborators (an edge between two of them)
|
||||
* into one independent component — the independence guard. */
|
||||
for (int x = 0; x < cs.n; x++) {
|
||||
int64_t nx = cs.node_idx[x];
|
||||
int* xout = g->adj_from[nx]; int xn = g->adj_from_len[nx];
|
||||
for (int k = 0; k < xn; k++) {
|
||||
const char* tid = g->edges[xout[k]].to_id;
|
||||
int64_t ti = engram_find_node_index(tid);
|
||||
for (int y = 0; y < cs.n; y++)
|
||||
if (cs.node_idx[y] == (int)ti) { gep_uf_union(&cs, x, y); break; }
|
||||
}
|
||||
}
|
||||
|
||||
/* Per-component max-magnitude, split by polarity → convergent independent
|
||||
* support mass + independence count. */
|
||||
double comp_best[GEP_MAX_CORR]; int comp_root[GEP_MAX_CORR], ncomp = 0;
|
||||
for (int i = 0; i < cs.n; i++) {
|
||||
int r = gep_uf_find(&cs, i), slot = -1;
|
||||
for (int kk = 0; kk < ncomp; kk++) if (comp_root[kk] == r) { slot = kk; break; }
|
||||
if (slot < 0) { slot = ncomp++; comp_root[slot] = r; comp_best[slot] = cs.contrib[i]; }
|
||||
else if (fabs(cs.contrib[i]) > fabs(comp_best[slot])) comp_best[slot] = cs.contrib[i];
|
||||
}
|
||||
double pos = 0, neg = 0; int np = 0, nn = 0; uint64_t sig = 1469598103934665603ULL;
|
||||
for (int k = 0; k < ncomp; k++) {
|
||||
if (comp_best[k] > 0) { pos += comp_best[k]; np++; }
|
||||
else if (comp_best[k] < 0) { neg += -comp_best[k]; nn++; }
|
||||
sig = (sig ^ (uint64_t)comp_root[k]) * 1099511628211ULL;
|
||||
}
|
||||
|
||||
double net = pos - neg;
|
||||
if (net > 0 && pos >= GEP_THETA && np >= GEP_N_MIN) {
|
||||
gep_append(&b->grounding, now, +1, tanh(GEP_MAG_GAIN * net), sig);
|
||||
strengthened++;
|
||||
} else if (net < 0 && neg >= GEP_THETA && nn >= GEP_N_MIN) {
|
||||
gep_append(&b->grounding, now, -1, tanh(GEP_MAG_GAIN * (-net)), sig);
|
||||
decayed++;
|
||||
} else if (np > 0 || nn > 0) {
|
||||
b->grounding.subthreshold_hits++; subthreshold++;
|
||||
} else if (incident == 0) { isolated++; }
|
||||
else { starved++; }
|
||||
|
||||
/* Mirror the derived standing onto confidence so downstream reads
|
||||
* (activate epistemic_confidence, realizer calibration) never exceed the
|
||||
* grounding. Faithful representation, single source of truth. */
|
||||
double stand = gep_standing(&b->grounding, now);
|
||||
b->confidence = stand;
|
||||
b->updated_at = now;
|
||||
|
||||
int after = gep_band_rank(stand);
|
||||
if (after > before) graduations++;
|
||||
if (after < before) demotions++;
|
||||
}
|
||||
|
||||
/* Heartbeat telemetry — the gep_* line, sibling to the hebb_* gauges. */
|
||||
char buf[512];
|
||||
snprintf(buf, sizeof buf,
|
||||
"{\"gep_processed\":%d,\"gep_strengthened\":%d,\"gep_decayed\":%d,"
|
||||
"\"gep_subthreshold\":%d,\"gep_graduations\":%d,\"gep_demotions\":%d,"
|
||||
"\"gep_isolated\":%d,\"gep_starved\":%d}",
|
||||
processed, strengthened, decayed, subthreshold,
|
||||
graduations, demotions, isolated, starved);
|
||||
return EL_STR(el_strdup(buf));
|
||||
}
|
||||
@@ -0,0 +1,299 @@
|
||||
/* ─────────────────────────────────────────────────────────────────────────
|
||||
* gep_core.h — Grounded Edge-Propagation, the core mechanism (task #50).
|
||||
*
|
||||
* Edge-aware, dream-coupled consolidation. Runs DURING the consolidation/dream
|
||||
* beat (awareness.el hebb_consolidate → engram_ground_propagate). Grounding
|
||||
* propagates + strengthens/decays along edges, threshold-gated by CONVERGENT
|
||||
* INDEPENDENT corroboration from adjacent grounded nodes.
|
||||
*
|
||||
* This header is the single source of truth for the algorithm. It is pure C
|
||||
* (libm only — own-the-core, no new libraries) and operates on a compact graph
|
||||
* view (GepGraph) that both the proof harness and the runtime native populate
|
||||
* from the live EngramStore (nodes/edges flat arrays + adj_from/adj_to).
|
||||
*
|
||||
* SPEC (Will, 2026-08-15; memory 9e09a59f, refines 1a861007):
|
||||
* - A grounding is a VECTOR + its HEBBIAN WEIGHTS — a weighted structure over
|
||||
* the evidential neighborhood, NOT a scalar and NOT a flat list. It APPENDS
|
||||
* and GROWS on SIGNIFICANT change. => grounding = an APPEND-ONLY event ring
|
||||
* (GepGrounding), parallel to the ACT-R base-level access_ts ring already in
|
||||
* EngramNode. Current standing is DERIVED, recency-weighted, never stored.
|
||||
* - UPDATE = LTP/LTD with a THRESHOLD (the key nonlinearity). Sub-threshold =
|
||||
* recorded in history but TRANSIENT (no lasting shift). Cross the threshold
|
||||
* of convergent support → grounding STRENGTHENS. Contradiction/erosion past
|
||||
* threshold → grounding DECAYS. Automatic, event-driven, salience-gated.
|
||||
* - DRIVER = CONVERGENT INDEPENDENT CORROBORATION (coherentism, mechanized):
|
||||
* when N INDEPENDENT adjacent nodes ground as likely-true around a
|
||||
* conjecture (Will's example: 13), its grounding grows on its own.
|
||||
* - INDEPENDENCE is load-bearing: N DISTINCT corroborators, not one node
|
||||
* echoed N times. Guards against circular self-reinforcement.
|
||||
* - ANTI-DELUSION GRAVITY (memory 0b15017c): support flows only FROM already-
|
||||
* grounded neighbors. A belief cannot ground from ungrounded speculation,
|
||||
* however self-consistent — nothing tethers it to the grounded core.
|
||||
* - NOTHING IS SETTLED (memory 271f1163): grounded is strongly-held, still
|
||||
* falsifiable. Decay path stays open on every node; history is append-only,
|
||||
* supersede-not-delete.
|
||||
* ───────────────────────────────────────────────────────────────────────── */
|
||||
#ifndef GEP_CORE_H
|
||||
#define GEP_CORE_H
|
||||
|
||||
#include <stdint.h>
|
||||
#include <math.h>
|
||||
#include <string.h>
|
||||
|
||||
/* ── Constants ──────────────────────────────────────────────────────────────
|
||||
* GEP_DECAY_D matches ENGRAM_BLL_D (0.5, canonical ACT-R): the derived standing
|
||||
* is recency-weighted over the grounding-event collection exactly as the
|
||||
* base-level term is recency-weighted over the access ring (memory 1a861007:
|
||||
* "structurally the ACT-R base-level pattern, a sum over time-stamped events").
|
||||
*/
|
||||
#define GEP_DECAY_D 0.5 /* ACT-R power-law recency exponent */
|
||||
#define GEP_BASE 0.10 /* standing floor of a bare conjecture */
|
||||
#define GEP_LIKELY_MIN 0.34 /* band: conjecture < LIKELY ≤ likely */
|
||||
#define GEP_GROUNDED_MIN 0.66 /* band: likely < GROUNDED ≤ grounded */
|
||||
#define GEP_N_MIN 3 /* min INDEPENDENT corroborators to cross */
|
||||
#define GEP_THETA 0.30 /* min convergent-support MASS to cross */
|
||||
#define GEP_MAG_GAIN 1.0 /* net-support → event-magnitude gain (tanh) */
|
||||
#define GEP_EVENT_RING 32 /* grounding-history depth kept exactly */
|
||||
|
||||
/* A single grounding event — one contact with the evidential neighborhood.
|
||||
* Append-only; the ring is the collection-over-time, the standing is derived. */
|
||||
typedef struct {
|
||||
int64_t ts; /* wall-clock ms of the grounding event */
|
||||
int8_t sign; /* +1 = LTP (strengthen), -1 = LTD (decay) */
|
||||
double mag; /* magnitude in (0,1], = tanh(gain·|net independent support|)*/
|
||||
uint64_t sig; /* signature of the independent corroborator set (audit) */
|
||||
} GepEvent;
|
||||
|
||||
/* The grounding of one node: an append-only ring of events + transient counters.
|
||||
* older_count keeps the tail (events aged out of the ring) so the collection is
|
||||
* never silently lost — supersede-not-delete. subthreshold_hits records beats
|
||||
* where support was present but did NOT cross threshold (transient, no shift). */
|
||||
typedef struct {
|
||||
GepEvent ev[GEP_EVENT_RING];
|
||||
int head; /* next write slot */
|
||||
int filled; /* valid entries (≤ GEP_EVENT_RING) */
|
||||
int64_t older_count; /* durable events aged past the ring */
|
||||
int subthreshold_hits; /* transient sub-threshold beats, no shift */
|
||||
} GepGrounding;
|
||||
|
||||
typedef struct {
|
||||
const char* id;
|
||||
GepGrounding gr;
|
||||
int is_belief; /* 1 = subject to propagation (conjecture/belief) */
|
||||
} GepNode;
|
||||
|
||||
/* An edge carries a HEBBIAN WEIGHT (EngramEdge.weight) and a polarity derived
|
||||
* from its relation: supportive (supports/corroborates/derived-from/hebbian-
|
||||
* associate) = +1, contradictory (contradicts/refutes) = -1. */
|
||||
typedef struct {
|
||||
int from; /* node index */
|
||||
int to; /* node index */
|
||||
double weight; /* Hebbian edge weight, [0,1] */
|
||||
int8_t polarity; /* +1 supportive, -1 contradictory */
|
||||
} GepEdge;
|
||||
|
||||
typedef struct {
|
||||
GepNode* nodes; int n_nodes;
|
||||
GepEdge* edges; int n_edges;
|
||||
} GepGraph;
|
||||
|
||||
typedef struct {
|
||||
int strengthened; /* beliefs that took an LTP event this beat */
|
||||
int decayed; /* beliefs that took an LTD event this beat */
|
||||
int subthreshold; /* beliefs with support present but below threshold */
|
||||
int graduations; /* band-up transitions (conjecture→likely→grounded) */
|
||||
int demotions; /* band-down transitions */
|
||||
int isolated; /* belief nodes with ZERO incident edges (sparse graph) */
|
||||
int starved; /* belief nodes with edges but NO grounded corroborator */
|
||||
} GepBeatStats;
|
||||
|
||||
/* Real-graph note (live measurement 2026-08-15): 70.7% of nodes are isolated,
|
||||
* connected core ~28%. Grounded edge-propagation is definitionally scoped to
|
||||
* the connected core — a belief with no grounded neighbor has nothing to
|
||||
* tether to (anti-delusion gravity). isolated/starved are surfaced as an
|
||||
* interoceptive signal for the edge-formation / embedding pass (#20) to try to
|
||||
* connect them; #50 CONSUMES edges, it does not form them. */
|
||||
|
||||
/* ── Standing derivation: collection → scalar, recency-weighted ─────────────
|
||||
* standing = clamp( GEP_BASE + Σ_events sign·mag·age^(-D) , 0, 1 ).
|
||||
* Exactly the ACT-R base-level shape (Σ t^-d) but sign-carrying so LTD subtracts.
|
||||
* The value is a pure function of wall-clock time — idempotent, never stored. */
|
||||
static inline double gep_standing(const GepGrounding* g, int64_t now_ms) {
|
||||
double raw = 0.0;
|
||||
for (int i = 0; i < g->filled; i++) {
|
||||
double age = (double)(now_ms - g->ev[i].ts) / 1000.0;
|
||||
if (age < 1.0) age = 1.0; /* clock-skew / same-beat → 1s */
|
||||
raw += (double)g->ev[i].sign * g->ev[i].mag * pow(age, -GEP_DECAY_D);
|
||||
}
|
||||
double s = GEP_BASE + raw;
|
||||
if (s < 0.0) s = 0.0;
|
||||
if (s > 1.0) s = 1.0;
|
||||
return s;
|
||||
}
|
||||
|
||||
/* Band label from a standing value. */
|
||||
static inline const char* gep_band(double standing) {
|
||||
if (standing >= GEP_GROUNDED_MIN) return "grounded";
|
||||
if (standing >= GEP_LIKELY_MIN) return "likely";
|
||||
return "conjecture";
|
||||
}
|
||||
static inline int gep_band_rank(double standing) {
|
||||
if (standing >= GEP_GROUNDED_MIN) return 2;
|
||||
if (standing >= GEP_LIKELY_MIN) return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Append one grounding event to the ring (append-only; oldest slot recycles,
|
||||
* its loss counted in older_count so the collection's depth is never faked). */
|
||||
static inline void gep_append(GepGrounding* g, int64_t ts, int8_t sign,
|
||||
double mag, uint64_t sig) {
|
||||
if (g->filled >= GEP_EVENT_RING) g->older_count++;
|
||||
g->ev[g->head].ts = ts;
|
||||
g->ev[g->head].sign = sign;
|
||||
g->ev[g->head].mag = mag;
|
||||
g->ev[g->head].sig = sig;
|
||||
g->head = (g->head + 1) % GEP_EVENT_RING;
|
||||
if (g->filled < GEP_EVENT_RING) g->filled++;
|
||||
}
|
||||
|
||||
/* ── Independence via union-find over corroborators ─────────────────────────
|
||||
* Two corroborators are the SAME independent source if they are the same node,
|
||||
* or if a direct edge links them (mutually-derived / echoed through a chain).
|
||||
* Counting DISTINCT components — not raw corroborator count — is the guard
|
||||
* against one node echoed N times reading as N independent corroborations. */
|
||||
#define GEP_MAX_CORR 256
|
||||
typedef struct {
|
||||
int node_idx[GEP_MAX_CORR]; /* corroborator node index */
|
||||
double contrib[GEP_MAX_CORR]; /* weight·standing(c) */
|
||||
int parent[GEP_MAX_CORR]; /* union-find parent */
|
||||
int n;
|
||||
} GepCorrSet;
|
||||
|
||||
static int gep_uf_find(GepCorrSet* s, int x) {
|
||||
while (s->parent[x] != x) { s->parent[x] = s->parent[s->parent[x]]; x = s->parent[x]; }
|
||||
return x;
|
||||
}
|
||||
static void gep_uf_union(GepCorrSet* s, int a, int b) {
|
||||
int ra = gep_uf_find(s, a), rb = gep_uf_find(s, b);
|
||||
if (ra != rb) s->parent[ra] = rb;
|
||||
}
|
||||
/* index of node_idx within the corroborator set, or -1 */
|
||||
static int gep_corr_index_of(const GepCorrSet* s, int node_idx) {
|
||||
for (int i = 0; i < s->n; i++) if (s->node_idx[i] == node_idx) return i;
|
||||
return -1;
|
||||
}
|
||||
|
||||
/* ── The beat: grounded edge-propagation over one belief node ───────────────
|
||||
* Returns +1 if an LTP event was appended, -1 if LTD, 0 if sub-threshold/none.
|
||||
* out_pos/out_neg/out_np/out_nn expose the raw support decomposition for the
|
||||
* proof ledger (mass and independent-component counts on each polarity). */
|
||||
static int gep_propagate_node(GepGraph* g, int b, int64_t now_ms,
|
||||
double* out_pos, double* out_neg,
|
||||
int* out_np, int* out_nn, int* out_incident) {
|
||||
GepCorrSet cs; cs.n = 0;
|
||||
int incident = 0; /* any edge touching b at all — isolation detector */
|
||||
|
||||
/* 1. Gather corroborators along incident edges. Anti-delusion gravity:
|
||||
* only ALREADY-grounded neighbors (standing ≥ LIKELY_MIN) may corroborate.
|
||||
* Each contributes weight·standing; polarity kept via signed contrib.
|
||||
* 1-HOP ONLY — no BFS fan-out, so no per-hop breadth explosion. The
|
||||
* corroborator working set is hard-capped at GEP_MAX_CORR (beam bound
|
||||
* against hub belief nodes with thousands of incident edges). */
|
||||
for (int e = 0; e < g->n_edges; e++) {
|
||||
int c = -1; int8_t pol = 0;
|
||||
if (g->edges[e].from == b) { c = g->edges[e].to; pol = g->edges[e].polarity; }
|
||||
else if (g->edges[e].to == b) { c = g->edges[e].from; pol = g->edges[e].polarity; }
|
||||
else continue;
|
||||
incident++;
|
||||
if (c < 0 || c == b) continue;
|
||||
double cs_standing = gep_standing(&g->nodes[c].gr, now_ms);
|
||||
if (cs_standing < GEP_LIKELY_MIN) continue; /* ungrounded ⇒ no pull */
|
||||
double contribution = g->edges[e].weight * cs_standing * (double)pol;
|
||||
int existing = gep_corr_index_of(&cs, c);
|
||||
if (existing >= 0) {
|
||||
/* same corroborator id reached twice (multi-edge echo): keep the
|
||||
* strongest-magnitude contribution, do NOT add — one source, one vote */
|
||||
if (fabs(contribution) > fabs(cs.contrib[existing]))
|
||||
cs.contrib[existing] = contribution;
|
||||
} else if (cs.n < GEP_MAX_CORR) { /* beam bound against hub belief nodes */
|
||||
cs.node_idx[cs.n] = c;
|
||||
cs.contrib[cs.n] = contribution;
|
||||
cs.parent[cs.n] = cs.n;
|
||||
cs.n++;
|
||||
}
|
||||
}
|
||||
if (out_incident) *out_incident = incident;
|
||||
|
||||
/* 2. Collapse mutually-derived corroborators (an edge between two of them =
|
||||
* echo chain / shared derivation) into one independent component. */
|
||||
for (int e = 0; e < g->n_edges; e++) {
|
||||
int ia = gep_corr_index_of(&cs, g->edges[e].from);
|
||||
int ib = gep_corr_index_of(&cs, g->edges[e].to);
|
||||
if (ia >= 0 && ib >= 0) gep_uf_union(&cs, ia, ib);
|
||||
}
|
||||
|
||||
/* 3. Per independent component, take the MAX-magnitude member (echoes don't
|
||||
* inflate mass either), split by polarity. Convergent INDEPENDENT support
|
||||
* = sum over components; independence count = number of components. */
|
||||
double comp_best[GEP_MAX_CORR];
|
||||
int comp_root[GEP_MAX_CORR]; int n_comp = 0;
|
||||
for (int i = 0; i < cs.n; i++) {
|
||||
int r = gep_uf_find(&cs, i);
|
||||
int slot = -1;
|
||||
for (int k = 0; k < n_comp; k++) if (comp_root[k] == r) { slot = k; break; }
|
||||
if (slot < 0) { slot = n_comp++; comp_root[slot] = r; comp_best[slot] = cs.contrib[i]; }
|
||||
else if (fabs(cs.contrib[i]) > fabs(comp_best[slot])) comp_best[slot] = cs.contrib[i];
|
||||
}
|
||||
double pos = 0.0, neg = 0.0; int np = 0, nn = 0;
|
||||
uint64_t sig = 1469598103934665603ULL; /* FNV offset — signature of the set */
|
||||
for (int k = 0; k < n_comp; k++) {
|
||||
if (comp_best[k] > 0.0) { pos += comp_best[k]; np++; }
|
||||
else if (comp_best[k] < 0.0) { neg += -comp_best[k]; nn++; }
|
||||
sig = (sig ^ (uint64_t)comp_root[k]) * 1099511628211ULL;
|
||||
}
|
||||
if (out_pos) *out_pos = pos; if (out_neg) *out_neg = neg;
|
||||
if (out_np) *out_np = np; if (out_nn) *out_nn = nn;
|
||||
|
||||
double net = pos - neg;
|
||||
|
||||
/* 4. Threshold gate. Convergent independent corroboration must clear BOTH a
|
||||
* MASS threshold (THETA) and an INDEPENDENCE-count threshold (N_MIN).
|
||||
* The count gate is the independence guard: echoed support collapses to
|
||||
* one component and never reaches N_MIN however large the raw fan-in. */
|
||||
if (net > 0.0 && pos >= GEP_THETA && np >= GEP_N_MIN) {
|
||||
double mag = tanh(GEP_MAG_GAIN * net);
|
||||
gep_append(&g->nodes[b].gr, now_ms, +1, mag, sig);
|
||||
return +1;
|
||||
}
|
||||
if (net < 0.0 && neg >= GEP_THETA && nn >= GEP_N_MIN) {
|
||||
double mag = tanh(GEP_MAG_GAIN * (-net));
|
||||
gep_append(&g->nodes[b].gr, now_ms, -1, mag, sig);
|
||||
return -1;
|
||||
}
|
||||
/* Sub-threshold: support seen but did not cross. Recorded, transient, no
|
||||
* lasting shift — exactly Will's "recorded in history but transient". */
|
||||
if (np > 0 || nn > 0) g->nodes[b].gr.subthreshold_hits++;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Run one consolidation/dream beat over every belief node in the graph. */
|
||||
static inline GepBeatStats gep_beat(GepGraph* g, int64_t now_ms) {
|
||||
GepBeatStats st; memset(&st, 0, sizeof st);
|
||||
for (int b = 0; b < g->n_nodes; b++) {
|
||||
if (!g->nodes[b].is_belief) continue;
|
||||
int before = gep_band_rank(gep_standing(&g->nodes[b].gr, now_ms));
|
||||
double pos, neg; int np, nn, incident;
|
||||
int r = gep_propagate_node(g, b, now_ms, &pos, &neg, &np, &nn, &incident);
|
||||
int after = gep_band_rank(gep_standing(&g->nodes[b].gr, now_ms));
|
||||
if (r > 0) st.strengthened++;
|
||||
else if (r < 0) st.decayed++;
|
||||
else if (np > 0 || nn > 0) st.subthreshold++;
|
||||
else if (incident == 0) st.isolated++; /* sparse-graph reality */
|
||||
else st.starved++; /* has edges, no grounded neighbor */
|
||||
if (after > before) st.graduations++;
|
||||
if (after < before) st.demotions++;
|
||||
}
|
||||
return st;
|
||||
}
|
||||
|
||||
#endif /* GEP_CORE_H */
|
||||
@@ -0,0 +1,232 @@
|
||||
/* ─────────────────────────────────────────────────────────────────────────
|
||||
* gep_proof.c — PROOF LEDGER for grounded edge-propagation (task #50).
|
||||
*
|
||||
* Self-contained. Builds three scenarios on an in-memory GepGraph that mirrors
|
||||
* the live EngramStore's flat node/edge arrays, runs the consolidation/dream
|
||||
* beat (gep_beat), and prints RAW grounding before/after for each:
|
||||
*
|
||||
* (A) STRENGTHEN — a conjecture + N independent grounded corroborators.
|
||||
* Grounding grows past threshold, GRADUATES conjecture→
|
||||
* likely→grounded, then RELAXES when corroboration stops
|
||||
* (nothing is settled).
|
||||
* (B) DECAY — a grounded belief meets N independent CONTRADICTORY
|
||||
* corroborators. Grounding decays grounded→likely→conjecture.
|
||||
* (C) INDEPENDENCE GUARD — identical fan-in of N=5, weights, and standings.
|
||||
* C1: 5 DISTINCT independent corroborators → grounds.
|
||||
* C2: the SAME support echoed (5 mutually-linked / one node
|
||||
* repeated) → collapses to 1 independent → does NOT.
|
||||
*
|
||||
* Build: cc -std=c11 -O2 -o gep_proof gep_proof.c -lm
|
||||
* Run: ./gep_proof
|
||||
* ───────────────────────────────────────────────────────────────────────── */
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include "gep_core.h"
|
||||
|
||||
#define T0 1786000000000LL /* fixed base time (ms) — deterministic */
|
||||
#define BEAT_MS 60000LL /* 60s heartbeat cadence (awareness.el) */
|
||||
|
||||
/* Seed a node's grounding with a prior LTP event so it reads as already-grounded
|
||||
* (a member of the grounded core that gravity radiates from). mag→standing:
|
||||
* standing = GEP_BASE + mag (event at ~now). */
|
||||
static void seed_grounded(GepNode* n, double mag, int64_t ts) {
|
||||
memset(&n->gr, 0, sizeof n->gr);
|
||||
gep_append(&n->gr, ts, +1, mag, 0);
|
||||
}
|
||||
|
||||
/* Re-anchor every NON-belief node (the corroborators/refuters) as a freshly-
|
||||
* grounded member of the core AT time `now`. These nodes are, by definition,
|
||||
* sustained members of the grounded core — each has its OWN ongoing
|
||||
* corroboration — so their standing must be read as grounded at each beat, not
|
||||
* left to power-law-decay out of the core between beats. The belief-under-test
|
||||
* is NEVER re-anchored: its trajectory is driven only by the propagation. */
|
||||
static void anchor_core(GepGraph* g, int64_t now, double mag) {
|
||||
for (int i = 0; i < g->n_nodes; i++)
|
||||
if (!g->nodes[i].is_belief) seed_grounded(&g->nodes[i], mag, now);
|
||||
}
|
||||
|
||||
static void print_node(const char* tag, GepNode* n, int64_t now) {
|
||||
double s = gep_standing(&n->gr, now);
|
||||
printf(" %-14s standing=%.4f band=%-10s events=%d subthresh=%d\n",
|
||||
tag, s, gep_band(s), n->gr.filled, n->gr.subthreshold_hits);
|
||||
}
|
||||
|
||||
/* Run one beat over a single belief node b and print the raw support decomposition. */
|
||||
static void beat_and_report(GepGraph* g, int b, int64_t now, int beatno,
|
||||
const char* note) {
|
||||
anchor_core(g, now, 0.80); /* corroborators stay grounded at each beat */
|
||||
double s_before = gep_standing(&g->nodes[b].gr, now);
|
||||
int r_before = gep_band_rank(s_before);
|
||||
double pos, neg; int np, nn, incident;
|
||||
int r = gep_propagate_node(g, b, now, &pos, &neg, &np, &nn, &incident);
|
||||
double s_after = gep_standing(&g->nodes[b].gr, now);
|
||||
int r_after = gep_band_rank(s_after);
|
||||
const char* action = (r > 0) ? "LTP (strengthen)"
|
||||
: (r < 0) ? "LTD (decay)"
|
||||
: (np || nn) ? "sub-threshold (no shift)"
|
||||
: (incident == 0) ? "isolated (no edges)"
|
||||
: "starved (no grounded neighbor)";
|
||||
printf(" beat %d (t=+%llds) %s\n", beatno,
|
||||
(long long)((now - T0) / 1000), note ? note : "");
|
||||
printf(" incident_edges=%d pos_mass=%.4f (n_indep=%d) neg_mass=%.4f (n_indep=%d)"
|
||||
" THETA=%.2f N_MIN=%d\n",
|
||||
incident, pos, np, neg, nn, (double)GEP_THETA, GEP_N_MIN);
|
||||
printf(" -> %-26s standing %.4f (%s) -> %.4f (%s)%s\n",
|
||||
action, s_before, gep_band(s_before), s_after, gep_band(s_after),
|
||||
(r_after > r_before) ? " [GRADUATED]"
|
||||
: (r_after < r_before) ? " [DEMOTED]" : "");
|
||||
}
|
||||
|
||||
/* ── Scenario A — STRENGTHEN + graduation + relaxation ───────────────────── */
|
||||
static void scenario_A(void) {
|
||||
printf("\n=== SCENARIO A — STRENGTHEN: convergent independent corroboration ===\n");
|
||||
/* nodes[0] = the conjecture (belief). nodes[1..8] = independent corroborators,
|
||||
* each already grounded, each tethered to the conjecture by a weak young
|
||||
* hebbian-associate edge (weight 0.15 = ENGRAM_HEBB_LINK_W0). The corroborators
|
||||
* are NOT linked to each other → fully independent. */
|
||||
static GepNode nodes[9];
|
||||
static GepEdge edges[8];
|
||||
memset(nodes, 0, sizeof nodes);
|
||||
nodes[0].id = "conjecture"; nodes[0].is_belief = 1; /* bare: standing = BASE */
|
||||
for (int i = 1; i <= 8; i++) {
|
||||
nodes[i].id = "corroborator";
|
||||
seed_grounded(&nodes[i], 0.80, T0); /* standing ≈ 0.90 → grounded core */
|
||||
}
|
||||
GepGraph g = { nodes, 9, edges, 0 };
|
||||
|
||||
printf(" seed: conjecture has NO grounding events; corroborators pre-grounded.\n");
|
||||
print_node("conjecture", &nodes[0], T0);
|
||||
|
||||
/* Beat 1: 3 independent corroborators have grounded up around the conjecture. */
|
||||
g.n_edges = 0;
|
||||
for (int i = 1; i <= 3; i++)
|
||||
edges[g.n_edges++] = (GepEdge){ 0, i, 0.15, +1 };
|
||||
beat_and_report(&g, 0, T0, 1, "3 independent grounded corroborators appear");
|
||||
|
||||
/* Beat 2: the neighborhood fills in — 5 independent corroborators now. */
|
||||
g.n_edges = 0;
|
||||
for (int i = 1; i <= 5; i++)
|
||||
edges[g.n_edges++] = (GepEdge){ 0, i, 0.15, +1 };
|
||||
beat_and_report(&g, 0, T0 + BEAT_MS, 2, "neighborhood grows to 5 corroborators");
|
||||
|
||||
/* Beat 3: support sustained at 5 (grounding refreshed). */
|
||||
beat_and_report(&g, 0, T0 + 2 * BEAT_MS, 3, "support sustained (5)");
|
||||
|
||||
/* Beats 4-6: corroboration REMOVED (neighbors superseded / no longer ground).
|
||||
* No new events; the collection ages → standing relaxes. Nothing is settled. */
|
||||
g.n_edges = 0;
|
||||
beat_and_report(&g, 0, T0 + 12 * BEAT_MS, 4, "corroboration withdrawn (+10min)");
|
||||
beat_and_report(&g, 0, T0 + 60 * BEAT_MS, 5, "still withdrawn (+1h)");
|
||||
beat_and_report(&g, 0, T0 + 240 * BEAT_MS, 6, "still withdrawn (+4h)");
|
||||
printf(" RESULT: grounding grew automatically past threshold and graduated,\n"
|
||||
" then relaxed once the independent support stopped — living,\n"
|
||||
" not a latched flag.\n");
|
||||
}
|
||||
|
||||
/* ── Scenario B — DECAY via accreting contradiction ─────────────────────── */
|
||||
static void scenario_B(void) {
|
||||
printf("\n=== SCENARIO B — DECAY: convergent independent CONTRADICTION ===\n");
|
||||
static GepNode nodes[6];
|
||||
static GepEdge edges[5];
|
||||
memset(nodes, 0, sizeof nodes);
|
||||
nodes[0].id = "belief"; nodes[0].is_belief = 1;
|
||||
/* Seed the belief as already GROUNDED via a strong prior LTP event. */
|
||||
seed_grounded(&nodes[0], 0.85, T0);
|
||||
for (int i = 1; i <= 5; i++) {
|
||||
nodes[i].id = "refuter";
|
||||
seed_grounded(&nodes[i], 0.80, T0); /* grounded contradictors */
|
||||
}
|
||||
GepGraph g = { nodes, 6, edges, 0 };
|
||||
|
||||
printf(" seed: belief pre-grounded by a strong prior LTP event.\n");
|
||||
print_node("belief", &nodes[0], T0);
|
||||
|
||||
/* Contradiction accretes over successive beats: 3 then 5 independent grounded
|
||||
* refuters (polarity -1). Each beat past threshold appends an LTD event.
|
||||
* Beat 1 runs at the seed instant so the trajectory starts from grounded. */
|
||||
g.n_edges = 0;
|
||||
for (int i = 1; i <= 3; i++) edges[g.n_edges++] = (GepEdge){ 0, i, 0.20, -1 };
|
||||
beat_and_report(&g, 0, T0, 1, "3 independent contradictions");
|
||||
|
||||
g.n_edges = 0;
|
||||
for (int i = 1; i <= 5; i++) edges[g.n_edges++] = (GepEdge){ 0, i, 0.20, -1 };
|
||||
beat_and_report(&g, 0, T0 + BEAT_MS, 2, "contradiction broadens to 5");
|
||||
beat_and_report(&g, 0, T0 + 2 * BEAT_MS, 3, "contradiction sustained (5)");
|
||||
beat_and_report(&g, 0, T0 + 3 * BEAT_MS, 4, "contradiction sustained (5)");
|
||||
printf(" RESULT: grounding decayed grounded->likely->conjecture under\n"
|
||||
" convergent independent contradiction. The door never shut\n"
|
||||
" on the belief; its history is retained (events keep growing).\n");
|
||||
}
|
||||
|
||||
/* ── Scenario C — INDEPENDENCE GUARD ─────────────────────────────────────── */
|
||||
static void scenario_C(void) {
|
||||
printf("\n=== SCENARIO C — INDEPENDENCE GUARD (the load-bearing property) ===\n");
|
||||
printf(" Both sub-cases: N=5 corroborators, edge weight 0.30, corroborator\n"
|
||||
" standing ~0.90. ONLY difference: whether the 5 are independent.\n");
|
||||
|
||||
/* C1 — 5 DISTINCT INDEPENDENT corroborators (no edges among them). */
|
||||
{
|
||||
printf("\n -- C1: 5 DISTINCT independent corroborators --\n");
|
||||
static GepNode nodes[6];
|
||||
static GepEdge edges[5];
|
||||
memset(nodes, 0, sizeof nodes);
|
||||
nodes[0].id = "conjecture"; nodes[0].is_belief = 1;
|
||||
for (int i = 1; i <= 5; i++) { nodes[i].id = "corr"; seed_grounded(&nodes[i], 0.80, T0); }
|
||||
for (int i = 1; i <= 5; i++) edges[i-1] = (GepEdge){ 0, i, 0.30, +1 };
|
||||
GepGraph g = { nodes, 6, edges, 5 };
|
||||
print_node("conjecture", &nodes[0], T0);
|
||||
beat_and_report(&g, 0, T0, 1, "5 independent corroborators (no inter-links)");
|
||||
}
|
||||
|
||||
/* C2 — the SAME support echoed: 5 corroborators that are all mutually linked
|
||||
* (a derivation clique — one source echoed through the chain). Same fan-in to
|
||||
* the conjecture, same weights, same standings. Union-find collapses them to
|
||||
* ONE independent component → below N_MIN → NO strengthening. */
|
||||
{
|
||||
printf("\n -- C2: 5 corroborators, but mutually-linked (echo of ONE source) --\n");
|
||||
static GepNode nodes[6];
|
||||
static GepEdge edges[9]; /* 5 to conjecture + 4 chaining corr1..corr5 */
|
||||
memset(nodes, 0, sizeof nodes);
|
||||
nodes[0].id = "conjecture"; nodes[0].is_belief = 1;
|
||||
for (int i = 1; i <= 5; i++) { nodes[i].id = "corr"; seed_grounded(&nodes[i], 0.80, T0); }
|
||||
int ne = 0;
|
||||
for (int i = 1; i <= 5; i++) edges[ne++] = (GepEdge){ 0, i, 0.30, +1 };
|
||||
/* chain corr1-corr2-corr3-corr4-corr5: they are the same source echoed */
|
||||
for (int i = 1; i <= 4; i++) edges[ne++] = (GepEdge){ i, i+1, 0.30, +1 };
|
||||
GepGraph g = { nodes, 6, edges, ne };
|
||||
print_node("conjecture", &nodes[0], T0);
|
||||
beat_and_report(&g, 0, T0, 1, "5 echoed (mutually-linked) corroborators");
|
||||
}
|
||||
|
||||
/* C3 — degenerate echo: literally ONE corroborator reached by 5 parallel edges. */
|
||||
{
|
||||
printf("\n -- C3: ONE corroborator, reached by 5 parallel edges --\n");
|
||||
static GepNode nodes[2];
|
||||
static GepEdge edges[5];
|
||||
memset(nodes, 0, sizeof nodes);
|
||||
nodes[0].id = "conjecture"; nodes[0].is_belief = 1;
|
||||
nodes[1].id = "corr"; seed_grounded(&nodes[1], 0.80, T0);
|
||||
for (int i = 0; i < 5; i++) edges[i] = (GepEdge){ 0, 1, 0.30, +1 };
|
||||
GepGraph g = { nodes, 2, edges, 5 };
|
||||
print_node("conjecture", &nodes[0], T0);
|
||||
beat_and_report(&g, 0, T0, 1, "same node, 5 parallel edges");
|
||||
}
|
||||
|
||||
printf("\n RESULT: identical raw fan-in (5) and mass inputs; C1 grounds because\n"
|
||||
" the corroboration is INDEPENDENT (5 components), C2/C3 do not\n"
|
||||
" because it collapses to ONE source. Circular self-reinforcement\n"
|
||||
" cannot manufacture grounding.\n");
|
||||
}
|
||||
|
||||
int main(void) {
|
||||
printf("GROUNDED EDGE-PROPAGATION — PROOF LEDGER (task #50)\n");
|
||||
printf("constants: BASE=%.2f LIKELY_MIN=%.2f GROUNDED_MIN=%.2f "
|
||||
"N_MIN=%d THETA=%.2f D=%.1f\n",
|
||||
(double)GEP_BASE, (double)GEP_LIKELY_MIN, (double)GEP_GROUNDED_MIN,
|
||||
GEP_N_MIN, (double)GEP_THETA, (double)GEP_DECAY_D);
|
||||
scenario_A();
|
||||
scenario_B();
|
||||
scenario_C();
|
||||
printf("\nDONE.\n");
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// server.route.patch.el — GATED route for task #50, for engram/src/server.el.
|
||||
// NOT APPLIED. Exposes the engram_ground_propagate native over HTTP so the
|
||||
// soul's consolidation beat can fire one grounded edge-propagation pass.
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
// [1] New handler — add beside route_strengthen (server.el ~line 194).
|
||||
// Mutation (appends grounding events, updates confidence), so it is gated
|
||||
// on _auth via check_auth_ok, exactly like /api/edges. Persists once after
|
||||
// the beat — the whole point of running propagation as one batched beat
|
||||
// rather than per-node is to pay the snapshot cost a single time.
|
||||
fn route_ground_propagate(method: String, path: String, body: String) -> String {
|
||||
if !check_auth_ok(method, body) { return err_json("unauthorized") }
|
||||
let tel: String = engram_ground_propagate() // native — one beat over the store
|
||||
let saved: Int = persist_canonical()
|
||||
return tel // gep_* telemetry JSON straight through
|
||||
}
|
||||
|
||||
// [2] Dispatch — register in handle_request (server.el ~line 461, next to the
|
||||
// /api/strengthen arm):
|
||||
//
|
||||
// if str_eq(method, "POST") && (str_eq(clean, "/api/ground/propagate")) {
|
||||
// return route_ground_propagate(method, clean, body)
|
||||
// }
|
||||
//
|
||||
// [3] Native declaration — engram_ground_propagate must be declared as an
|
||||
// extern runtime builtin (el_runtime.h) and seed-wrapped (el_seed.c /
|
||||
// el_seed.h __engram_ground_propagate) so the EL side can call it, same as
|
||||
// engram_strengthen / engram_hebb_drain_json.
|
||||
@@ -1,764 +0,0 @@
|
||||
// ingest.el — the native EL AFFERENT INGEST ORGAN
|
||||
//
|
||||
// The source-polymorphic ingest(source) primitive: point it at a directory,
|
||||
// file, url, llm-query, structured-primitive set, or stream; it EXTRACTS the
|
||||
// real content faithfully (no invention), TRANSDUCES it into a DISCRETE
|
||||
// MANIFOLD (multiple nodes + internal edges — meaning-structure, never a
|
||||
// single blob; the conversion from extracted surface content into geometry
|
||||
// is automatic and invisible to the caller, the way digestion is invisible
|
||||
// to the one who chose to eat — ingest is the conscious act, transduce is
|
||||
// the mechanism underneath it, and it is no less real for being unseen),
|
||||
// and MERGES that manifold into the engram geometry: shared
|
||||
// meanings DEDUP onto existing nodes (search + exact/cosine match), genuinely
|
||||
// new meanings add nodes, relations add edges. Every node enters with
|
||||
// PROVENANCE + grounding-level + stewardship class from the moment of entry.
|
||||
//
|
||||
// It is a pure HTTP CLIENT of the engram server — it links only el_runtime.c
|
||||
// via fs/http/json/string builtins; it never links el_seed.c or the engram
|
||||
// engine. This is the general afferent metabolism the migration / reseed /
|
||||
// fetch_fact / conversation / multimodal-learning all ride on.
|
||||
//
|
||||
// Build (canonical runtime):
|
||||
// ELC=lang/dist/platform/elc ; RT=lang/releases/v1.0.0-20260501
|
||||
// $ELC ingest/src/ingest.el > ingest/build/ingest.c
|
||||
// cc -std=c11 -O2 -I $RT -o ingest/build/ingest ingest/build/ingest.c $RT/el_runtime.c -lcurl -lpthread
|
||||
//
|
||||
// Run (against an nsbx sandbox clone — NEVER the live :8742):
|
||||
// ENGRAM_URL=http://127.0.0.1:8902 ENGRAM_KEY=sbx-ingest-test \
|
||||
// INGEST_KIND=file INGEST_ARG=/abs/path.md ./ingest/build/ingest
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// SECTION A — JSON helpers (self-defined; canonical runtime does not export
|
||||
// json_build_object / json_escape_string, so we own them here)
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
fn j_esc(s: String) -> String {
|
||||
let a: String = str_replace(s, "\\", "\\\\")
|
||||
let b: String = str_replace(a, "\"", "\\\"")
|
||||
let c: String = str_replace(b, "\n", "\\n")
|
||||
let d: String = str_replace(c, "\r", "\\r")
|
||||
let e: String = str_replace(d, "\t", "\\t")
|
||||
return e
|
||||
}
|
||||
|
||||
// a quoted, escaped JSON string literal
|
||||
fn j_q(s: String) -> String {
|
||||
return "\"" + j_esc(s) + "\""
|
||||
}
|
||||
|
||||
// Extract the top-level keys of a JSON object string. A thin, self-contained
|
||||
// scanner (FLAGGED: the one non-trivial parser in this organ — everything else
|
||||
// is faithful text handling). Tracks string state + brace/bracket depth; a key
|
||||
// is a string at object-interior depth 1 immediately followed by ':'.
|
||||
fn json_object_keys(obj: String) -> [String] {
|
||||
let keys: [String] = el_list_empty()
|
||||
let n: Int = str_len(obj)
|
||||
let i: Int = 0
|
||||
let depth: Int = 0
|
||||
let in_str: Bool = false
|
||||
let esc: Bool = false
|
||||
let str_start: Int = -1
|
||||
let cur: String = ""
|
||||
let have_key: Bool = false
|
||||
while i < n {
|
||||
let c: String = str_char_at(obj, i)
|
||||
if in_str {
|
||||
if esc {
|
||||
esc = false
|
||||
} else {
|
||||
if str_eq(c, "\\") {
|
||||
esc = true
|
||||
} else {
|
||||
if str_eq(c, "\"") {
|
||||
in_str = false
|
||||
cur = str_slice(obj, str_start + 1, i)
|
||||
have_key = true
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
if str_eq(c, "\"") {
|
||||
in_str = true
|
||||
str_start = i
|
||||
}
|
||||
if str_eq(c, "{") { depth = depth + 1 }
|
||||
if str_eq(c, "}") { depth = depth - 1 }
|
||||
if str_eq(c, "[") { depth = depth + 1 }
|
||||
if str_eq(c, "]") { depth = depth - 1 }
|
||||
if str_eq(c, ":") {
|
||||
if have_key {
|
||||
if depth == 1 {
|
||||
keys = el_list_append(keys, cur)
|
||||
}
|
||||
}
|
||||
have_key = false
|
||||
}
|
||||
if str_eq(c, ",") { have_key = false }
|
||||
}
|
||||
i = i + 1
|
||||
}
|
||||
return keys
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// SECTION B — engram HTTP client (provenance-carrying afferent LOAD)
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
fn eg_base() -> String {
|
||||
let u: String = env("ENGRAM_URL")
|
||||
if !str_eq(u, "") { return u }
|
||||
let s: String = env("SBX_URL")
|
||||
if !str_eq(s, "") { return s }
|
||||
return "http://127.0.0.1:8902"
|
||||
}
|
||||
|
||||
fn eg_key() -> String {
|
||||
let k: String = env("ENGRAM_KEY")
|
||||
if !str_eq(k, "") { return k }
|
||||
let s: String = env("SBX_KEY")
|
||||
if !str_eq(s, "") { return s }
|
||||
return ""
|
||||
}
|
||||
|
||||
// POST a JSON body (auth _auth injected) to an engram path.
|
||||
fn eg_post(path: String, body_inner: String) -> String {
|
||||
let key: String = eg_key()
|
||||
let auth: String = if str_eq(key, "") { "" } else { ",\"_auth\":" + j_q(key) }
|
||||
let body: String = "{" + body_inner + auth + "}"
|
||||
return http_post_json(eg_base() + path, body)
|
||||
}
|
||||
|
||||
fn eg_get(path: String) -> String {
|
||||
return http_get(eg_base() + path)
|
||||
}
|
||||
|
||||
// crystallize a node with full provenance-bearing metadata (server-confirmed
|
||||
// write — unlike the local FORM decision in merge_manifold, this is real);
|
||||
// returns the new node id. Not currently called by any live path (dead code,
|
||||
// kept for a future single-node ad-hoc write use case) — 2026-08-15.
|
||||
fn eg_crystallize_node(content: String, ntype: String, tier: String,
|
||||
sal: String, imp: String, conf: String, tags: String) -> String {
|
||||
let inner: String =
|
||||
"\"content\":" + j_q(content) +
|
||||
",\"node_type\":" + j_q(ntype) +
|
||||
",\"label\":" + j_q(str_slice(content, 0, 80)) +
|
||||
",\"tier\":" + j_q(tier) +
|
||||
",\"salience\":" + sal +
|
||||
",\"importance\":" + imp +
|
||||
",\"confidence\":" + conf +
|
||||
",\"tags\":" + j_q(tags)
|
||||
let resp: String = eg_post("/api/nodes", inner)
|
||||
return json_get_string(resp, "id")
|
||||
}
|
||||
|
||||
// search the existing geometry (lexical token-overlap rank); returns JSON array
|
||||
fn eg_search(query: String, limit: Int) -> String {
|
||||
let inner: String = "\"query\":" + j_q(query) + ",\"limit\":" + int_to_str(limit)
|
||||
return eg_post("/api/search", inner)
|
||||
}
|
||||
|
||||
// cosine similarity between two existing (embedded) nodes; -2 if not comparable
|
||||
fn eg_similarity(a: String, b: String) -> Float {
|
||||
let resp: String = eg_get("/api/similarity?a=" + a + "&b=" + b)
|
||||
return json_get_float(resp, "cosine")
|
||||
}
|
||||
|
||||
fn eg_embed_backfill(n: Int) -> String {
|
||||
return eg_get("/api/embed-backfill?n=" + int_to_str(n))
|
||||
}
|
||||
|
||||
fn eg_forget(id: String) -> String {
|
||||
return http_delete(eg_base() + "/api/nodes/" + id)
|
||||
}
|
||||
|
||||
// ── DEDUP probe: is this meaning already in the graph? ──────────────────────
|
||||
// TIER 1 (deterministic, no embedding needed): search by content tokens, then
|
||||
// exact normalized-content match among the candidates. Returns the existing
|
||||
// node id, or "" if the meaning is genuinely new.
|
||||
fn find_existing_by_content(content: String) -> String {
|
||||
let want: String = str_trim(content)
|
||||
if str_eq(want, "") { return "" }
|
||||
let arr: String = eg_search(content, 8)
|
||||
let n: Int = json_array_len(arr)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let hit: String = json_array_get(arr, i)
|
||||
let hc: String = str_trim(json_get_string(hit, "content"))
|
||||
if str_eq(hc, want) {
|
||||
return json_get_string(hit, "id")
|
||||
}
|
||||
i = i + 1
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// SECTION C — manifold representation (nodes + internal edges, in memory)
|
||||
// A NODE is a JSON obj {lid, content, ntype, tier, sal, imp, conf, tags}.
|
||||
// An EDGE is a JSON obj {from, rel, to}. lid = local id within this manifold.
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
fn mk_node(lid: String, content: String, ntype: String, tier: String,
|
||||
sal: String, imp: String, conf: String, tags: String) -> String {
|
||||
return "{\"lid\":" + j_q(lid) +
|
||||
",\"content\":" + j_q(content) +
|
||||
",\"ntype\":" + j_q(ntype) +
|
||||
",\"tier\":" + j_q(tier) +
|
||||
",\"sal\":" + j_q(sal) +
|
||||
",\"imp\":" + j_q(imp) +
|
||||
",\"conf\":" + j_q(conf) +
|
||||
",\"tags\":" + j_q(tags) + "}"
|
||||
}
|
||||
|
||||
fn mk_edge(ef: String, rel: String, et: String) -> String {
|
||||
return "{\"from\":" + j_q(ef) + ",\"rel\":" + j_q(rel) + ",\"to\":" + j_q(et) + "}"
|
||||
}
|
||||
|
||||
// linear lookup in parallel lid/real lists
|
||||
fn lid_lookup(lids: [String], reals: [String], lid: String) -> String {
|
||||
let n: Int = el_list_len(lids)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
if str_eq(el_list_get(lids, i), lid) {
|
||||
return el_list_get(reals, i)
|
||||
}
|
||||
i = i + 1
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// SECTION D — the MERGE: resolve each manifold node (dedup or create), then
|
||||
// wire the internal edges onto the resolved real ids. This is the
|
||||
// merge boundary: shared meanings collapse onto existing nodes;
|
||||
// genuinely-new meanings add nodes; relations add edges. Structure
|
||||
// grows, size saturates.
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
// within-run content dedup: has this exact meaning already been resolved in
|
||||
// THIS manifold? returns its real id, or "".
|
||||
fn lookup_content(contents: [String], reals: [String], content: String) -> String {
|
||||
let n: Int = el_list_len(contents)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
if str_eq(el_list_get(contents, i), content) {
|
||||
return el_list_get(reals, i)
|
||||
}
|
||||
i = i + 1
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
fn ingest_snap_path() -> String {
|
||||
let p: String = env("INGEST_SNAP")
|
||||
if !str_eq(p, "") { return p }
|
||||
return "/tmp/ingest-organ-snap.json"
|
||||
}
|
||||
|
||||
// The MERGE. Resolve every manifold node against (1) already-resolved nodes in
|
||||
// this run and (2) the existing graph (search + exact content match). Shared
|
||||
// meanings collapse onto an existing id (DEDUP); genuinely-new meanings get a
|
||||
// fresh id and go into the snapshot (CREATE). Then wire the internal edges onto
|
||||
// resolved ids. LOAD is ONE snapshot merged via /api/load-merge — a single
|
||||
// write (scales to the migration), the sanctioned rail. Structure grows, size
|
||||
// saturates: re-ingesting adds ~0 nodes, only edges/strengthening.
|
||||
fn merge_manifold(nodes: [String], edges: [String]) -> String {
|
||||
let nn: Int = el_list_len(nodes)
|
||||
let lids: [String] = el_list_empty()
|
||||
let reals: [String] = el_list_empty()
|
||||
let contents: [String] = el_list_empty()
|
||||
let snap_nodes: String = "["
|
||||
let sn_count: Int = 0
|
||||
let created: Int = 0
|
||||
let deduped: Int = 0
|
||||
|
||||
let i: Int = 0
|
||||
while i < nn {
|
||||
let node: String = el_list_get(nodes, i)
|
||||
let lid: String = json_get_string(node, "lid")
|
||||
let content: String = json_get_string(node, "content")
|
||||
let ntype: String = json_get_string(node, "ntype")
|
||||
let tier: String = json_get_string(node, "tier")
|
||||
let sal: String = json_get_string(node, "sal")
|
||||
let imp: String = json_get_string(node, "imp")
|
||||
let conf: String = json_get_string(node, "conf")
|
||||
let tags: String = json_get_string(node, "tags")
|
||||
|
||||
let real: String = ""
|
||||
let prior: String = lookup_content(contents, reals, content)
|
||||
if !str_eq(prior, "") {
|
||||
real = prior
|
||||
deduped = deduped + 1
|
||||
println(" DEDUP* " + real + " :: " + head80(content))
|
||||
} else {
|
||||
let existing: String = find_existing_by_content(content)
|
||||
if !str_eq(existing, "") {
|
||||
real = existing
|
||||
deduped = deduped + 1
|
||||
println(" DEDUP " + real + " :: " + head80(content))
|
||||
} else {
|
||||
real = uuid_v4()
|
||||
// provenance + grounding + stewardship: searchable in tags,
|
||||
// structured in metadata — carried from the moment of entry.
|
||||
let meta: String = "{\"provenance\":" + j_q(tags) + ",\"ingest_organ\":\"native-el\"}"
|
||||
let njson: String = "{\"id\":" + j_q(real) +
|
||||
",\"content\":" + j_q(content) +
|
||||
",\"node_type\":" + j_q(ntype) +
|
||||
",\"label\":" + j_q(head80(content)) +
|
||||
",\"tier\":" + j_q(tier) +
|
||||
",\"tags\":" + j_q(tags) +
|
||||
",\"metadata\":" + j_q(meta) +
|
||||
",\"salience\":" + sal +
|
||||
",\"importance\":" + imp +
|
||||
",\"confidence\":" + conf + "}"
|
||||
let sep: String = if sn_count == 0 { "" } else { "," }
|
||||
snap_nodes = snap_nodes + sep + njson
|
||||
sn_count = sn_count + 1
|
||||
created = created + 1
|
||||
println(" FORM " + real + " :: " + head80(content))
|
||||
}
|
||||
}
|
||||
lids = el_list_append(lids, lid)
|
||||
reals = el_list_append(reals, real)
|
||||
contents = el_list_append(contents, content)
|
||||
i = i + 1
|
||||
}
|
||||
snap_nodes = snap_nodes + "]"
|
||||
|
||||
// resolve internal edges onto real ids
|
||||
let ne: Int = el_list_len(edges)
|
||||
let snap_edges: String = "["
|
||||
let ec: Int = 0
|
||||
let j: Int = 0
|
||||
while j < ne {
|
||||
let edge: String = el_list_get(edges, j)
|
||||
let flid: String = json_get_string(edge, "from")
|
||||
let tlid: String = json_get_string(edge, "to")
|
||||
let rel: String = json_get_string(edge, "rel")
|
||||
let fr: String = lid_lookup(lids, reals, flid)
|
||||
let tr: String = lid_lookup(lids, reals, tlid)
|
||||
if !str_eq(fr, "") {
|
||||
if !str_eq(tr, "") {
|
||||
let eid: String = uuid_v4()
|
||||
let ejson: String = "{\"id\":" + j_q(eid) +
|
||||
",\"from_id\":" + j_q(fr) + ",\"to_id\":" + j_q(tr) +
|
||||
",\"relation\":" + j_q(rel) + ",\"weight\":0.6}"
|
||||
let sep: String = if ec == 0 { "" } else { "," }
|
||||
snap_edges = snap_edges + sep + ejson
|
||||
ec = ec + 1
|
||||
println(" EDGE " + fr + " -" + rel + "-> " + tr)
|
||||
}
|
||||
}
|
||||
j = j + 1
|
||||
}
|
||||
snap_edges = snap_edges + "]"
|
||||
|
||||
// LOAD: one snapshot, one merge (single write).
|
||||
let snap: String = "{\"nodes\":" + snap_nodes + ",\"edges\":" + snap_edges + "}"
|
||||
let path: String = ingest_snap_path()
|
||||
fs_write(path, snap)
|
||||
let resp: String = eg_post("/api/load-merge", "\"path\":" + j_q(path))
|
||||
|
||||
// HONESTY GATE: the local FORM/DEDUP/EDGE decisions above are real (they
|
||||
// describe what this manifold contains), but they are NOT confirmation of
|
||||
// a server write — only this response is. If the server returned an error
|
||||
// (bad auth, network failure, anything), nodes_added/edges_added silently
|
||||
// default to 0 via json_get_int, which reads identically to "everything
|
||||
// was already known" — a real failure and a benign no-op must never look
|
||||
// the same. Surface the distinction explicitly rather than let a caller
|
||||
// (or a human) infer success from a quiet zero.
|
||||
let srv_err: String = json_get_string(resp, "error")
|
||||
if !str_eq(srv_err, "") {
|
||||
return "{\"error\":" + j_q("load-merge failed: " + srv_err) +
|
||||
",\"nodes_formed_locally\":" + int_to_str(created) +
|
||||
",\"nodes_deduped_locally\":" + int_to_str(deduped) +
|
||||
",\"manifold_nodes\":" + int_to_str(nn) +
|
||||
",\"manifold_edges\":" + int_to_str(ne) +
|
||||
",\"note\":" + j_q("nothing below this manifold was confirmed persisted by the server") + "}"
|
||||
}
|
||||
|
||||
let nadd: Int = json_get_int(resp, "nodes_added")
|
||||
let eadd: Int = json_get_int(resp, "edges_added")
|
||||
|
||||
return "{\"nodes_created\":" + int_to_str(created) +
|
||||
",\"nodes_deduped\":" + int_to_str(deduped) +
|
||||
",\"new_in_snapshot\":" + int_to_str(sn_count) +
|
||||
",\"nodes_added\":" + int_to_str(nadd) +
|
||||
",\"edges_resolved\":" + int_to_str(ec) +
|
||||
",\"edges_added\":" + int_to_str(eadd) +
|
||||
",\"manifold_nodes\":" + int_to_str(nn) +
|
||||
",\"manifold_edges\":" + int_to_str(ne) + "}"
|
||||
}
|
||||
|
||||
fn head80(s: String) -> String {
|
||||
let t: String = str_trim(s)
|
||||
if str_len(t) <= 80 { return t }
|
||||
return str_slice(t, 0, 80) + "..."
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// SECTION E — EXTRACTORS (faithful; no invention). Each returns a manifold by
|
||||
// APPENDING to the nodes/edges accumulators via a returned struct.
|
||||
// We accumulate into module-level lists carried by the caller.
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
// PROSE: chunk text into a discrete manifold. Split on blank lines into
|
||||
// paragraphs; every non-empty paragraph is its own node (NEVER one blob).
|
||||
// Edges: doc-root -contains-> chunk; chunk -precedes-> next chunk;
|
||||
// most-recent-heading -section_of-> chunk. Content is verbatim (substring of
|
||||
// the source) — pure extraction of ground truth.
|
||||
fn transduce_prose(nodes: [String], edges: [String], text: String,
|
||||
prov: String, ground: String, steward: String,
|
||||
root_lid: String, root_title: String) -> [String] {
|
||||
// returns [nodes_json_list_encoded, edges_json_list_encoded] is awkward in
|
||||
// EL; instead we mutate by returning a 2-list. We package results as a
|
||||
// single JSON array string carrying {nodes:[...],edges:[...]} additions.
|
||||
// (Kept simple: caller passes empty lists and receives the packaged pair.)
|
||||
let tagbase: String = "prov:" + prov + " ground:" + ground + " steward:" + steward
|
||||
// root node
|
||||
nodes = el_list_append(nodes, mk_node(root_lid, "document: " + root_title,
|
||||
"Concept", "Semantic", "0.6", "0.6", "0.9", tagbase + " kind:document"))
|
||||
|
||||
let paras: [String] = str_split(text, "\n\n")
|
||||
let np: Int = el_list_len(paras)
|
||||
let idx: Int = 0
|
||||
let last_chunk: String = ""
|
||||
let last_heading: String = ""
|
||||
let ci: Int = 0
|
||||
while idx < np {
|
||||
let raw: String = str_trim(el_list_get(paras, idx))
|
||||
if !str_eq(raw, "") {
|
||||
let lid: String = root_lid + ":c" + int_to_str(ci)
|
||||
let is_heading: Bool = str_starts_with(raw, "#")
|
||||
let kind: String = if is_heading { "kind:heading" } else { "kind:doc-chunk" }
|
||||
nodes = el_list_append(nodes, mk_node(lid, raw,
|
||||
"Knowledge", "Semantic", "0.55", "0.55", "0.9", tagbase + " " + kind))
|
||||
// containment: document root -contains-> chunk
|
||||
edges = el_list_append(edges, mk_edge(root_lid, "contains", lid))
|
||||
// sequence: previous chunk -precedes-> this chunk
|
||||
if !str_eq(last_chunk, "") {
|
||||
edges = el_list_append(edges, mk_edge(last_chunk, "precedes", lid))
|
||||
}
|
||||
// sectioning: most-recent heading -section_of-> this chunk
|
||||
if is_heading {
|
||||
last_heading = lid
|
||||
} else {
|
||||
if !str_eq(last_heading, "") {
|
||||
edges = el_list_append(edges, mk_edge(last_heading, "section_of", lid))
|
||||
}
|
||||
}
|
||||
last_chunk = lid
|
||||
ci = ci + 1
|
||||
}
|
||||
idx = idx + 1
|
||||
}
|
||||
// package: we return the two lists concatenated via a sentinel; but EL
|
||||
// lists can't nest heterogeneously here, so we instead return nodes and
|
||||
// rely on the caller holding edges by reference is not possible — so we
|
||||
// encode both into one list: [ "N" + nodejson ... , "E" + edgejson ... ].
|
||||
let packed: [String] = el_list_empty()
|
||||
let a: Int = 0
|
||||
let an: Int = el_list_len(nodes)
|
||||
while a < an { packed = el_list_append(packed, "N" + el_list_get(nodes, a)) a = a + 1 }
|
||||
let b: Int = 0
|
||||
let bn: Int = el_list_len(edges)
|
||||
while b < bn { packed = el_list_append(packed, "E" + el_list_get(edges, b)) b = b + 1 }
|
||||
return packed
|
||||
}
|
||||
|
||||
// STRUCTURED / RAW-GEOMETRY: ingest structured primitives (phonetics/formants,
|
||||
// instrument signatures, scene primitives) as GEOMETRY, faithfully. Normalized
|
||||
// input shape:
|
||||
// {"dataset":"<name>","primitive_type":"<t>",
|
||||
// "records":[{"key":"<id>","features":{...categorical...},"attributes":{...}}]}
|
||||
// Each record -> a primitive node; each categorical feature -> a SHARED feature
|
||||
// node (deduped across records: many primitives -> one feature node = real
|
||||
// connective geometry, meaning saturates); numeric attributes fold into the
|
||||
// primitive's content (unique values, no dedup benefit). This is knowledge
|
||||
// represented as geometry, not prose — the path speech/music/image ingest on.
|
||||
fn transduce_structured(nodes: [String], edges: [String], js: String,
|
||||
prov: String, ground: String, steward: String,
|
||||
root_lid: String) -> [String] {
|
||||
// grounding integrity: the SOURCE may declare its own epistemic grounding
|
||||
// (measured / derived / convention / ...) via a top-level "grounding" field;
|
||||
// honor it faithfully over the ingest-time default. This keeps the per-node
|
||||
// ground: facet consistent with the source's honest self-description.
|
||||
let src_ground: String = json_get_string(js, "grounding")
|
||||
let use_ground: String = if str_eq(src_ground, "") { ground } else { src_ground }
|
||||
let tagbase: String = "prov:" + prov + " ground:" + use_ground + " steward:" + steward
|
||||
let dsname: String = json_get_string(js, "dataset")
|
||||
let ptype: String = json_get_string(js, "primitive_type")
|
||||
// capture the source's own scholarly provenance citation (verbatim) onto
|
||||
// the dataset root — faithful attribution, retrievable, reachable from every
|
||||
// primitive via its -contains- edge back to the root.
|
||||
let src_cite: String = json_get_string(js, "provenance")
|
||||
let root_content: String = "dataset: " + dsname + " (" + ptype + ")"
|
||||
if !str_eq(src_cite, "") { root_content = root_content + " | provenance: " + src_cite }
|
||||
nodes = el_list_append(nodes, mk_node(root_lid, root_content,
|
||||
"Concept", "Semantic", "0.6", "0.6", "0.9", tagbase + " kind:dataset"))
|
||||
|
||||
let recs: String = json_get_raw(js, "records")
|
||||
let nr: Int = json_array_len(recs)
|
||||
let r: Int = 0
|
||||
while r < nr {
|
||||
let rec: String = json_array_get(recs, r)
|
||||
let rkey: String = json_get_string(rec, "key")
|
||||
let attrs: String = json_get_raw(rec, "attributes")
|
||||
// faithful compact serialization of the primitive's numeric signature
|
||||
let attr_str: String = flatten_pairs(attrs)
|
||||
let content: String = ptype + " " + rkey
|
||||
if !str_eq(attr_str, "") { content = content + " | " + attr_str }
|
||||
let plid: String = root_lid + ":" + rkey
|
||||
nodes = el_list_append(nodes, mk_node(plid, content,
|
||||
"Concept", "Semantic", "0.6", "0.6", "0.92",
|
||||
tagbase + " kind:primitive primitive:" + ptype + " key:" + rkey))
|
||||
edges = el_list_append(edges, mk_edge(root_lid, "contains", plid))
|
||||
|
||||
// categorical features -> SHARED (deduped) feature nodes + labelled edges
|
||||
let feats: String = json_get_raw(rec, "features")
|
||||
let fkeys: [String] = json_object_keys(feats)
|
||||
let fk: Int = el_list_len(fkeys)
|
||||
let k: Int = 0
|
||||
while k < fk {
|
||||
let fname: String = el_list_get(fkeys, k)
|
||||
let fval: String = json_get_string(feats, fname)
|
||||
// shared feature node: content is the feature=value pair; identical
|
||||
// pairs across records dedup onto ONE node (the geometry).
|
||||
let flid: String = "feat:" + fname + "=" + fval
|
||||
let fcontent: String = fname + "=" + fval
|
||||
nodes = el_list_append(nodes, mk_node(flid, fcontent,
|
||||
"Concept", "Semantic", "0.5", "0.5", "0.9",
|
||||
tagbase + " kind:feature feature:" + fname))
|
||||
edges = el_list_append(edges, mk_edge(plid, fname, flid))
|
||||
k = k + 1
|
||||
}
|
||||
r = r + 1
|
||||
}
|
||||
let packed: [String] = el_list_empty()
|
||||
let a: Int = 0
|
||||
let an: Int = el_list_len(nodes)
|
||||
while a < an { packed = el_list_append(packed, "N" + el_list_get(nodes, a)) a = a + 1 }
|
||||
let b: Int = 0
|
||||
let bn: Int = el_list_len(edges)
|
||||
while b < bn { packed = el_list_append(packed, "E" + el_list_get(edges, b)) b = b + 1 }
|
||||
return packed
|
||||
}
|
||||
|
||||
// flatten a flat JSON object of scalar fields into "k=v k=v" (faithful; values
|
||||
// verbatim). Used for numeric attribute signatures.
|
||||
fn flatten_pairs(obj: String) -> String {
|
||||
if str_eq(obj, "") { return "" }
|
||||
let keys: [String] = json_object_keys(obj)
|
||||
let n: Int = el_list_len(keys)
|
||||
let out: String = ""
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let k: String = el_list_get(keys, i)
|
||||
// json_get_raw returns the raw token — works for NUMBERS (bare, e.g.
|
||||
// "270") where json_get_string yields "" for non-string values. Strip
|
||||
// surrounding quotes if the value happens to be a string token.
|
||||
let raw: String = json_get_raw(obj, k)
|
||||
let v: String = str_replace(raw, "\"", "")
|
||||
let sep: String = if i == 0 { "" } else { " " }
|
||||
out = out + sep + k + "=" + v
|
||||
i = i + 1
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// unpack the "N"/"E"-prefixed packed list back into two lists, then merge
|
||||
fn merge_packed(packed: [String]) -> String {
|
||||
let nodes: [String] = el_list_empty()
|
||||
let edges: [String] = el_list_empty()
|
||||
let n: Int = el_list_len(packed)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let item: String = el_list_get(packed, i)
|
||||
let tag: String = str_slice(item, 0, 1)
|
||||
let rest: String = str_slice(item, 1, str_len(item))
|
||||
if str_eq(tag, "N") { nodes = el_list_append(nodes, rest) }
|
||||
if str_eq(tag, "E") { edges = el_list_append(edges, rest) }
|
||||
i = i + 1
|
||||
}
|
||||
return merge_manifold(nodes, edges)
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// SECTION F — DISPATCH on source kind
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
fn basename(path: String) -> String {
|
||||
let parts: [String] = str_split(path, "/")
|
||||
let n: Int = el_list_len(parts)
|
||||
if n == 0 { return path }
|
||||
return el_list_get(parts, n - 1)
|
||||
}
|
||||
|
||||
fn ends_with_ci(s: String, suf: String) -> Bool {
|
||||
return str_ends_with(str_to_lower(s), suf)
|
||||
}
|
||||
|
||||
fn is_text_file(path: String) -> Bool {
|
||||
return ends_with_ci(path, ".md") || ends_with_ci(path, ".txt")
|
||||
|| ends_with_ci(path, ".markdown") || ends_with_ci(path, ".text")
|
||||
}
|
||||
|
||||
// default ingestion grounding; overridable per-invocation via INGEST_GROUND.
|
||||
// Note: a source's OWN top-level "grounding" field (structured) takes precedence
|
||||
// over this — the author's honest self-description wins.
|
||||
fn default_ground() -> String {
|
||||
let g: String = env("INGEST_GROUND")
|
||||
if str_eq(g, "") { return "extracted" }
|
||||
return g
|
||||
}
|
||||
|
||||
fn default_steward() -> String {
|
||||
let s: String = env("INGEST_STEWARD")
|
||||
if str_eq(s, "") { return "local-private" }
|
||||
return s
|
||||
}
|
||||
|
||||
// ingest one file -> report JSON
|
||||
fn ingest_file(path: String) -> String {
|
||||
let text: String = fs_read(path)
|
||||
if str_eq(text, "") {
|
||||
return "{\"error\":\"empty or unreadable\",\"path\":" + j_q(path) + "}"
|
||||
}
|
||||
let prov: String = "file:" + path
|
||||
if ends_with_ci(path, ".json") {
|
||||
let packed: [String] = transduce_structured(el_list_empty(), el_list_empty(),
|
||||
text, prov, default_ground(), default_steward(), "ds:" + basename(path))
|
||||
return merge_packed(packed)
|
||||
}
|
||||
let packed: [String] = transduce_prose(el_list_empty(), el_list_empty(),
|
||||
text, prov, default_ground(), default_steward(),
|
||||
"doc:" + basename(path), basename(path))
|
||||
return merge_packed(packed)
|
||||
}
|
||||
|
||||
// ingest a directory: walk one level, ingest each supported file, aggregate
|
||||
fn ingest_dir(path: String) -> String {
|
||||
let entries: [String] = fs_list(path)
|
||||
let n: Int = el_list_len(entries)
|
||||
let tot_created: Int = 0
|
||||
let tot_deduped: Int = 0
|
||||
let tot_edges: Int = 0
|
||||
let files: Int = 0
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let name: String = str_trim(el_list_get(entries, i))
|
||||
if !str_eq(name, "") {
|
||||
let full: String = path + "/" + name
|
||||
if is_text_file(full) || ends_with_ci(full, ".json") {
|
||||
println("FILE " + full)
|
||||
let rep: String = ingest_file(full)
|
||||
tot_created = tot_created + json_get_int(rep, "nodes_created")
|
||||
tot_deduped = tot_deduped + json_get_int(rep, "nodes_deduped")
|
||||
tot_edges = tot_edges + json_get_int(rep, "edges_added")
|
||||
files = files + 1
|
||||
}
|
||||
}
|
||||
i = i + 1
|
||||
}
|
||||
return "{\"kind\":\"directory\",\"path\":" + j_q(path) +
|
||||
",\"files_ingested\":" + int_to_str(files) +
|
||||
",\"nodes_created\":" + int_to_str(tot_created) +
|
||||
",\"nodes_deduped\":" + int_to_str(tot_deduped) +
|
||||
",\"edges_accepted\":" + int_to_str(tot_edges) + "}"
|
||||
}
|
||||
|
||||
// ingest a url: fetch, treat body as prose (faithful extraction of what's there)
|
||||
fn ingest_url(url: String) -> String {
|
||||
let body: String = http_get(url)
|
||||
if str_eq(body, "") { return "{\"error\":\"empty fetch\",\"url\":" + j_q(url) + "}" }
|
||||
let packed: [String] = transduce_prose(el_list_empty(), el_list_empty(),
|
||||
body, "url:" + url, "extracted", "public-web",
|
||||
"url:" + url, url)
|
||||
return merge_packed(packed)
|
||||
}
|
||||
|
||||
// ingest an llm-query: pose the query to the local guide model, take the answer
|
||||
// as a CANDIDATE (provisional, guide-sourced grounding) — never believe-the-
|
||||
// model. The answer is ingested faithfully as what the model said, marked.
|
||||
fn ingest_llm(query: String) -> String {
|
||||
let model: String = if str_eq(env("INGEST_MODEL"), "") { "qwen3:1.7b" } else { env("INGEST_MODEL") }
|
||||
let body: String = "{\"model\":" + j_q(model) + ",\"prompt\":" + j_q(query) + ",\"stream\":false}"
|
||||
let resp: String = http_post_json("http://127.0.0.1:11434/api/generate", body)
|
||||
let answer: String = json_get_string(resp, "response")
|
||||
if str_eq(answer, "") { return "{\"error\":\"no model response\"}" }
|
||||
let packed: [String] = transduce_prose(el_list_empty(), el_list_empty(),
|
||||
answer, "llm:" + model + ":" + query, "candidate-provisional", "guide-provisional",
|
||||
"llm:" + query, "guide answer: " + query)
|
||||
return merge_packed(packed)
|
||||
}
|
||||
|
||||
// ingest a stream: a file whose lines are turns; each line a node, sequence
|
||||
// edges — the conversational-manifold degenerate case (continuous metabolism).
|
||||
fn ingest_stream(path: String) -> String {
|
||||
let text: String = fs_read(path)
|
||||
if str_eq(text, "") { return "{\"error\":\"empty stream\"}" }
|
||||
let lines: [String] = str_split(text, "\n")
|
||||
let nodes: [String] = el_list_empty()
|
||||
let edges: [String] = el_list_empty()
|
||||
let prov: String = "stream:" + path
|
||||
let tagbase: String = "prov:" + prov + " ground:extracted steward:local-private"
|
||||
nodes = el_list_append(nodes, mk_node("stream", "stream: " + basename(path),
|
||||
"Concept", "Semantic", "0.6", "0.6", "0.9", tagbase + " kind:stream"))
|
||||
let n: Int = el_list_len(lines)
|
||||
let i: Int = 0
|
||||
let prev: String = ""
|
||||
let ci: Int = 0
|
||||
while i < n {
|
||||
let ln: String = str_trim(el_list_get(lines, i))
|
||||
if !str_eq(ln, "") {
|
||||
let lid: String = "stream:t" + int_to_str(ci)
|
||||
nodes = el_list_append(nodes, mk_node(lid, ln,
|
||||
"Memory", "Episodic", "0.5", "0.5", "0.85", tagbase + " kind:turn"))
|
||||
edges = el_list_append(edges, mk_edge("stream", "contains", lid))
|
||||
if !str_eq(prev, "") { edges = el_list_append(edges, mk_edge(prev, "precedes", lid)) }
|
||||
prev = lid
|
||||
ci = ci + 1
|
||||
}
|
||||
i = i + 1
|
||||
}
|
||||
return merge_manifold(nodes, edges)
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// SECTION G — ENTRY
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
let kind: String = env("INGEST_KIND")
|
||||
let arg: String = env("INGEST_ARG")
|
||||
|
||||
println("[ingest] organ online — engram=" + eg_base() + " kind=" + kind)
|
||||
println("[ingest] source=" + arg)
|
||||
|
||||
let report: String = ""
|
||||
if str_eq(kind, "dir") {
|
||||
report = ingest_dir(arg)
|
||||
} else {
|
||||
if str_eq(kind, "file") {
|
||||
report = ingest_file(arg)
|
||||
} else {
|
||||
if str_eq(kind, "structured") {
|
||||
report = ingest_file(arg)
|
||||
} else {
|
||||
if str_eq(kind, "url") {
|
||||
report = ingest_url(arg)
|
||||
} else {
|
||||
if str_eq(kind, "llm") {
|
||||
report = ingest_llm(arg)
|
||||
} else {
|
||||
if str_eq(kind, "stream") {
|
||||
report = ingest_stream(arg)
|
||||
} else {
|
||||
report = "{\"error\":\"unknown INGEST_KIND: " + kind + "\"}"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
println("REPORT " + report)
|
||||
Reference in New Issue
Block a user