soul: the engram is the canonical store, not a fallback #162
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: neuron-technologies/neuron#162
Reference in New Issue
Block a user
Delete Branch "fix/engram-is-canonical"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The soul preferred its own local snapshot over the engram:
The copy outranked the store. Every one of these is a cost of that one inversion, and all were live tonight:
/api/graph/edgesserialized 128 MB to answer a read, because the soul's copy wasn't the engram'ssnapshot.jsonwith the soul's divergent copyThe engram had already reached this conclusion for its own boot: "the durable owner is the paged store... snapshot.json is never read again as the ongoing store. This closes the 'restart reverted to a 17h-old snapshot' data-loss window." The soul kept booting the way the engram had abandoned, and inherited exactly that window.
Now: in HTTP-engram mode the soul seeds from the engram every boot and never reads a local snapshot, present or not. It already never wrote one in this mode (gated behind
is_genesis && safe_to_seed), so this supplies the missing half. It refuses to boot on an empty seed rather than rebuilding a divergent graph from nothing;KeepAlive+ThrottleInterval=10makes that a 10s retry until the engram is up. File mode is untouched — there the soul genuinely is the owner.Verified in isolation (empty local snapshot planted): booted in ~30s with 13,446 nodes — the engram's contents, not the empty file.
Verified in production after deploy:
In sync, and 1.6 GB of duplicated graph reclaimed.
The soul preferred its own local snapshot over the engram: // Always try local snapshot first ... HTTP Engram is only used for the // very first boot (empty/absent snapshot). The copy outranked the store. Every one of these is a cost of that inversion, and all of them were live tonight: - the graphs drifted: 31,795 nodes / 75,241 edges in the soul against 13,439 / 37,670 in the engram — more than twice the edges, silently - write-through exists only to reconcile them, and had never once run - /api/graph/edges serialized 128 MB to answer a read, because the soul's copy was not the engram's - a read route overwrote the engram's canonical snapshot.json with the soul's divergent copy - three resident copies of one graph (soul, engram, Neuron.app) — about 7.2 GB of RAM for a store that is 2.2 GB on disk, which is what pushed the host into swap None of those are features. They are reconciliation debt from one decision. The engram had already reached this conclusion for its own boot path — "the durable owner is the paged store (neuron.egm + neuron.wal) ... snapshot.json is never read again as the ongoing store. This closes the 'restart reverted to a 17h-old snapshot' data-loss window." The soul kept booting the legacy way the engram had abandoned, and inherited exactly that data-loss window. So in HTTP-engram mode the soul now seeds from the engram on EVERY boot and never reads a local snapshot, present or not — a stale copy that outranks the store is the bug, not a fallback. It already never wrote one in this mode (gated behind is_genesis && safe_to_seed, and safe_to_seed requires !using_http_engram), so this supplies the missing half. It also refuses to boot on an empty seed rather than silently rebuilding a divergent graph from nothing. launchd KeepAlive with ThrottleInterval=10 turns that into a retry every 10s until the engram is up — self-healing, no spin. File mode (no ENGRAM_URL) is untouched: there the soul genuinely is the owner. Verified before deploy: with a deliberately empty local snapshot planted, the soul booted in ~30s reporting 13,446 nodes / 37,675 edges — the engram's contents, not the empty local file.