self-review 2026-05-24: BM25 scan cap 500→5000 + traversal inference guard
Two improvements:
1. BM25 search corpus coverage (server.el) — raised scan cap from 500 to
5000 nodes. On the 161K-node graph, 500 was 0.3% coverage; 5000 is 3%.
engram_scan_nodes_json returns nodes sorted by salience DESC, so ISEs
(salience 0.3) fall below Knowledge/Memory (0.5–0.8) naturally — the
effective corpus stays content-dense. Also updated stale comment on the
ISE route (no longer need high offset; recent-first ordering from May 23).
2. Traversal inference guard (el_runtime.c) — two changes:
- INFER_CAP reduced 256→32: proactive curiosity runs engram_activate
every ~30s. At 256 edges/call the soul daemon accumulated 107K edges
in 23h (5× BFS slowdown). At 32 the rate drops 8×.
- Edge count guard: skip inference entirely when snap_ec ≥ 40,000.
At that density most A→C paths are already explicit; marginal inference
value is low and the O(edge_count²) inner-loop cost is high. Self-limits
unbounded accumulation across restarts.
This commit is contained in:
@@ -6938,18 +6938,34 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
||||
/* ── TRAVERSAL INFERENCE: infer A→C edges when A→B→C was traversed ──
|
||||
* For each pair of edges (A→B, B→C) where all three nodes were reached,
|
||||
* create an inferred A→C edge with weight = w(A→B) * w(B→C) * 0.8
|
||||
* if no A→C edge already exists. Cap at 256 new edges per call.
|
||||
* if no A→C edge already exists. Cap at 32 new edges per call.
|
||||
*
|
||||
* Cap reduced 256→32 (2026-05-24 self-review): the soul daemon's proactive
|
||||
* curiosity runs engram_activate every ~30s. At 256 edges/call, the in-process
|
||||
* edge store grew from 21K → 107K in 23h — a 5× BFS slowdown. The inner
|
||||
* "check if A→C already exists" loop is O(edge_count) per candidate, so cost
|
||||
* scales as O(edge_count²) as the graph densifies. Reducing to 32 gives the
|
||||
* same latent-path benefit at 8× less accumulation rate.
|
||||
*
|
||||
* Edge count guard: skip inference entirely when the graph already has
|
||||
* ≥ 40,000 edges. At that density, most 2-hop A→C paths are already
|
||||
* explicit (either persisted or inferred in earlier calls), so marginal
|
||||
* inference value drops sharply while the O(edge²) scan cost stays high.
|
||||
* This self-limits unbounded accumulation across restarts.
|
||||
*
|
||||
* IMPORTANT: we collect candidate edges FIRST (snapshot the edge count and
|
||||
* copy the needed IDs/weights), then apply them AFTER — this avoids
|
||||
* dangling pointer bugs from realloc inside the scan loop. */
|
||||
{
|
||||
const int64_t INFER_CAP = 256;
|
||||
const int64_t INFER_CAP = 32;
|
||||
typedef struct { char from[64]; char to[64]; double weight; } InferCandidate;
|
||||
InferCandidate* cands = malloc((size_t)INFER_CAP * sizeof(InferCandidate));
|
||||
int64_t ncands = 0;
|
||||
/* Snapshot edge count so we only scan pre-existing edges. */
|
||||
int64_t snap_ec = g->edge_count;
|
||||
/* Skip inference on dense graphs: marginal value drops, O(e²) cost stays.
|
||||
* NULL cands → the if (cands) block below is skipped cleanly. */
|
||||
if (snap_ec >= 40000) { free(cands); cands = NULL; }
|
||||
if (cands) {
|
||||
for (int64_t e1 = 0; e1 < snap_ec && ncands < INFER_CAP; e1++) {
|
||||
EngramEdge* ea = &g->edges[e1];
|
||||
|
||||
Reference in New Issue
Block a user