self-review 2026-08-07: learning that cannot outlive the process is not learning
Yesterday's eligibility-trace fix made Hebbian consolidation numerically real: hebb_max 0.000799 -> 0.4725, and 1,198 hebbian-associate edges formed in 23h48m. This morning's census found where they went: nowhere. soul daemon (in-process graph): 42,426 edges, 1,198 hebbian engram server (:8742, durable): 41,213 edges, 49 hebbian Two processes, two graphs, one direction of travel. The soul pulls from the server every 10 min (GET /api/sync) and never pushes. It cannot fall back on saving its own copy either: soul.el sets soul_snapshot_path only inside `if is_genesis && safe_to_seed`, and safe_to_seed is unconditionally false whenever ENGRAM_URL is set -- because the server owns persistence and a soul writing snapshot.json would clobber it. That guard is correct. The consequence was not: mem_save() has never once executed. The soul is the ONLY process running idle cognition, so it is where essentially all co-activation happens -- and it was throwing away every association it learned, every restart, silently. The mechanism worked and the learning still evaporated. Consolidation is now a message, not a file. Fast volatile store hands each newly-formed association to the slow durable store over the API the server already exposes; only edges past ENGRAM_HEBB_LINK_MIN are ever queued, so what crosses the process boundary already earned it. - el_runtime.c: 512-slot overwrite-oldest write-back ring; enqueue at edge formation; engram_hebb_drain_json() pops a postable JSON batch. Drops and drains are counted, not silent -- a consolidation path that quietly discards is the exact failure this entry exists to correct. - server.el: POST /api/edges/batch. persist_canonical() writes the full 60MB snapshot per call, and route_create_edge calls it per edge -- correct for one interactive edge, ruinous for bulk (~840MB/beat to persist 14 associations). Batch connects all, snapshots once. Same durability, 1/N the writes. - act-stats: hebb_wb_pending / _drained / _dropped. pending climbing with drained flat = drain not called; drained climbing with sent 0 = POST refused. Both failure modes are now visible in the stream instead of in an autopsy. Verified live: batch route accepts valid entries, skips malformed ones without aborting the batch, and enforces _auth. All 1,256 learned associations are now in the canonical store; the soul booted at 42,431 edges with hebb_max 0.4941 carried across the restart for the first time.
This commit is contained in:
@@ -234,6 +234,55 @@ fn route_create_edge(method: String, path: String, body: String) -> String {
|
||||
"{\"ok\":true,\"from_id\":\"" + from_id + "\",\"to_id\":\"" + to_id + "\",\"relation\":\"" + relation + "\"}"
|
||||
}
|
||||
|
||||
// route_create_edges_batch — POST /api/edges/batch {"edges":[{from_id,to_id,relation,weight}, ...]}
|
||||
//
|
||||
// WHY THIS EXISTS (2026-08-07 self-review). persist_canonical() writes the
|
||||
// FULL canonical snapshot — 60MB at current graph size — and route_create_edge
|
||||
// calls it once per edge. That is correct for the interactive one-edge case and
|
||||
// ruinous for any bulk write: the soul's Hebbian consolidation path delivers
|
||||
// ~14 associations per 8-minute heartbeat, which through the single-edge route
|
||||
// would be ~840MB of disk writes per beat, ~150GB/day, to persist 14 edges.
|
||||
//
|
||||
// The fix is not to weaken durability — it is to make the unit of durability
|
||||
// the BATCH. Connect every edge, then snapshot exactly once. Same guarantee
|
||||
// (nothing acknowledged is lost to a restart), 1/N the writes. Empty or
|
||||
// malformed entries are skipped rather than aborting the batch: a consolidation
|
||||
// payload is best-effort by design, and one bad id should not cost the other 13.
|
||||
//
|
||||
// Returns the accepted count so the caller can tell delivery from silence.
|
||||
fn route_create_edges_batch(method: String, path: String, body: String) -> String {
|
||||
let arr: String = json_get_raw(body, "edges")
|
||||
if str_eq(arr, "") { return err_json("missing edges array") }
|
||||
let n: Int = json_array_len(arr)
|
||||
if n == 0 { return "{\"ok\":true,\"accepted\":0,\"skipped\":0}" }
|
||||
let i: Int = 0
|
||||
let accepted: Int = 0
|
||||
let skipped: Int = 0
|
||||
while i < n {
|
||||
let item: String = json_array_get(arr, i)
|
||||
let from_id: String = json_get_string(item, "from_id")
|
||||
let to_id: String = json_get_string(item, "to_id")
|
||||
if str_eq(from_id, "") || str_eq(to_id, "") {
|
||||
let skipped = skipped + 1
|
||||
} else {
|
||||
let rel_raw: String = json_get_string(item, "relation")
|
||||
let relation: String = if str_eq(rel_raw, "") { "associates" } else { rel_raw }
|
||||
let w_present: String = json_get_raw(item, "weight")
|
||||
let weight: Float = if str_eq(w_present, "") { 0.5 } else { json_get_float(item, "weight") }
|
||||
engram_connect(from_id, to_id, weight, relation)
|
||||
let accepted = accepted + 1
|
||||
}
|
||||
let i = i + 1
|
||||
}
|
||||
// ONE snapshot for the whole batch — the entire point of this route.
|
||||
// Skip it when nothing was accepted: an all-malformed payload must not
|
||||
// trigger a 60MB write.
|
||||
if accepted > 0 {
|
||||
let saved: Int = persist_canonical()
|
||||
}
|
||||
return "{\"ok\":true,\"accepted\":" + int_to_str(accepted) + ",\"skipped\":" + int_to_str(skipped) + "}"
|
||||
}
|
||||
|
||||
fn route_neighbors(method: String, path: String, body: String) -> String {
|
||||
let id: String = extract_id(path, "/api/neighbors/")
|
||||
if str_eq(id, "") { return err_json("missing id") }
|
||||
@@ -546,6 +595,13 @@ fn handle_request(method: String, path: String, body: String) -> String {
|
||||
if str_eq(method, "POST") && (str_eq(clean, "/api/edges") || str_eq(clean, "/edges")) {
|
||||
return route_create_edge(method, path, body)
|
||||
}
|
||||
// Batch edge write — one snapshot for the whole payload. Must be tested
|
||||
// BEFORE nothing else claims it; the exact-match on "/api/edges" above
|
||||
// does not catch "/api/edges/batch", so order is not load-bearing here,
|
||||
// but keeping the two adjacent keeps them from drifting apart.
|
||||
if str_eq(method, "POST") && (str_eq(clean, "/api/edges/batch") || str_eq(clean, "/edges/batch")) {
|
||||
return route_create_edges_batch(method, path, body)
|
||||
}
|
||||
if str_eq(method, "GET") && str_starts_with(clean, "/api/neighbors/") {
|
||||
return route_neighbors(method, path, body)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user