runtime: engram_edges_json — kill the whole-graph file round trip #127

2026-08-16T01:11:06Z

will.anderson commented

2026-08-16 01:11:06 +00:00

/api/graph/edges answered a read query by serializing the entire graph to disk (128 MB) and reading it back. Two defects in one line, both of which bit production tonight:

The path it wrote was ~/.neuron/engram/snapshot.json — the engram's CANONICAL store. A read route overwriting the persistence owner's file. This had been fixed once (export moved to a scratch path); it came back when the hand-written dispatch block was replaced by @route dispatch and the unfixed copy survived the merge.
A full snapshot write + 128 MB read + whole-graph parse, per request, to return a bounded slice.

Calling it tonight overwrote the canonical snapshot and immediately preceded an engram crash loop.

engram_edges_json(limit, offset) is the builtin that route's own TODO asked for — walks g->edges directly, emits every persisted field, no file touched.

limit <= 0 defaults to 1000, not unbounded — this is the endpoint that fell over, and an unbounded default would preserve the failure mode under a new name.

Registered in codegen.el's builtin_arity (plain + __) and wrapped in el_seed.c per the project's C-builtin recipe. Verified: elc rebuilds clean, soul amalgam regenerates and links.

`/api/graph/edges` answered a **read** query by serializing the entire graph to disk (128 MB) and reading it back. Two defects in one line, both of which bit production tonight: 1. **The path it wrote was `~/.neuron/engram/snapshot.json` — the engram's CANONICAL store.** A read route overwriting the persistence owner's file. This had been fixed once (export moved to a scratch path); it came back when the hand-written dispatch block was replaced by `@route` dispatch and the *unfixed* copy survived the merge. 2. A full snapshot write + 128 MB read + whole-graph parse, per request, to return a bounded slice. Calling it tonight overwrote the canonical snapshot and immediately preceded an engram crash loop. `engram_edges_json(limit, offset)` is the builtin that route's own TODO asked for — walks `g->edges` directly, emits every persisted field, no file touched. `limit <= 0` defaults to **1000, not unbounded** — this is the endpoint that fell over, and an unbounded default would preserve the failure mode under a new name. Registered in `codegen.el`'s `builtin_arity` (plain + `__`) and wrapped in `el_seed.c` per the project's C-builtin recipe. Verified: `elc` rebuilds clean, soul amalgam regenerates and links.

will.anderson added 1 commit 2026-08-16 01:11:07 +00:00

runtime: engram_edges_json — read edges without a whole-graph file round trip

El SDK CI - dev / build-and-test (pull_request) Failing after 13m4s

Details

4e24d7d3f1

/api/graph/edges answered a read query by calling engram_save() to serialize
the ENTIRE graph to disk (128 MB) and then fs_read-ing it back. Two defects in
one line, and both bit production on 2026-08-15:

  1. The path it wrote was ~/.neuron/engram/snapshot.json — the engram
     server's CANONICAL store. A READ route overwriting the persistence
     owner's canonical file. This defect had been fixed once (export moved to
     a scratch path); it came back when the hand-written dispatch block was
     replaced by @route dispatch and the unfixed copy is the one that
     survived the merge.
  2. Cost: a full snapshot write, a 128 MB read, and a parse of the whole
     graph, per request, to return a bounded slice.

Calling it tonight overwrote the canonical snapshot and immediately preceded
an engram crash loop.

engram_edges_json(limit, offset) is the builtin that route's own TODO asked
for ("Future: add an engram_edges_json() builtin and drop the file round trip
entirely"). It walks g->edges directly and emits every persisted field.

limit <= 0 defaults to 1000, not unbounded: this is the endpoint that fell
over, and an unbounded default would preserve the failure mode under a new
name. Callers page explicitly.

Registered in codegen.el's builtin_arity (both plain and __ spellings) and
wrapped in el_seed.c per the project's C-builtin recipe.

will.anderson merged commit c21074b547 into dev

2026-08-16 01:13:45 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/el#127