self-review 2026-05-24: BM25 scan cap 500→5000 + traversal inference guard

Two improvements:

1. BM25 search corpus coverage (server.el) — raised scan cap from 500 to
   5000 nodes. On the 161K-node graph, 500 was 0.3% coverage; 5000 is 3%.
   engram_scan_nodes_json returns nodes sorted by salience DESC, so ISEs
   (salience 0.3) fall below Knowledge/Memory (0.5–0.8) naturally — the
   effective corpus stays content-dense. Also updated stale comment on the
   ISE route (no longer need high offset; recent-first ordering from May 23).

2. Traversal inference guard (el_runtime.c) — two changes:
   - INFER_CAP reduced 256→32: proactive curiosity runs engram_activate
     every ~30s. At 256 edges/call the soul daemon accumulated 107K edges
     in 23h (5× BFS slowdown). At 32 the rate drops 8×.
   - Edge count guard: skip inference entirely when snap_ec ≥ 40,000.
     At that density most A→C paths are already explicit; marginal inference
     value is low and the O(edge_count²) inner-loop cost is high. Self-limits
     unbounded accumulation across restarts.
This commit is contained in:
2026-05-24 08:43:22 -05:00
parent 34249b39a3
commit ef1db34846
4 changed files with 30 additions and 7 deletions
BIN
View File
Binary file not shown.
+2 -2
View File
@@ -130,8 +130,8 @@ el_val_t bm25_search_json(el_val_t query, el_val_t limit) {
if (scan_limit < 200) {
scan_limit = 200;
}
if (scan_limit > 500) {
scan_limit = 500;
if (scan_limit > 5000) {
scan_limit = 5000;
}
el_val_t nodes_json = engram_scan_nodes_json(scan_limit, 0);
el_val_t n = json_array_len(nodes_json);
+10 -3
View File
@@ -123,9 +123,15 @@ fn bm25_score_doc(doc_content: String, query_tokens: String, corpus_size: Int, a
fn bm25_search_json(query: String, limit: Int) -> String {
// 1. Determine scan size: floor at 200 so small `limit` values still scan
// enough of the corpus to find relevant nodes.
// Cap raised from 500 5000 (2026-05-24 self-review): 500 was 0.3% of the
// 161K-node corpus. At 5000 we cover the top-3% by salience still fast
// (pure C scan, no Ollama calls) and 10x better recall for content search.
// engram_scan_nodes_json returns nodes sorted by salience DESC, so ISEs
// (salience 0.3) naturally fall below Knowledge/Memory (0.50.8), keeping
// the effective search corpus content-dense.
let scan_limit: Int = limit * 10
if scan_limit < 200 { let scan_limit = 200 }
if scan_limit > 500 { let scan_limit = 500 }
if scan_limit > 5000 { let scan_limit = 5000 }
// 2. Fetch node sample
let nodes_json: String = engram_scan_nodes_json(scan_limit, 0)
@@ -753,8 +759,9 @@ fn route_neuron_config(method: String, path: String, body: String) -> String {
// route_neuron_state_events GET lists ISEs, POST logs a new one.
// GET supports ?limit=N&offset=M for pagination; ?label=X to extract label
// from the ISE content's "event" field.
// Use a high offset to skip to recent ISEs (ISEs fill quickly and sort by
// salience then insertion order older entries dominate the front of scans).
// ISEs sort by created_at DESC (most-recent-first) as of 2026-05-23 fix.
// ?limit=10 returns the 10 most recent ISEs. Offset for pagination, not for
// skipping to recent events (that was the pre-fix behavior; no longer needed).
fn route_neuron_state_events(method: String, path: String, body: String) -> String {
if str_eq(method, "GET") {
let limit_str: String = query_param(path, "limit")