store: bound the pool by available memory and let it shrink #129
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: neuron-technologies/el#129
Reference in New Issue
Block a user
Delete Branch "fix/elc-rebuildable-compiler-builtins"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Fixes a hazard I introduced an hour earlier. The adaptive budget could only grow, and grew toward a share of TOTAL ram (80%, ~38 GiB on a 48 GB host). That is a memory leak with extra steps — total never shrinks when other processes need memory, so the pool had no way to notice it was starving its own host. Caught as live memory pressure.
A control loop with only one direction is not a control loop.
pc_available_ram()— free + inactive + purgeable (host_statistics64on Darwin,MemAvailableon Linux). Availability moves under pressure; total does not. Returns 0 when unreadable, and callers then refuse to grow — unknown means no, because a cache is never worth swapping the host.ENGRAM_POOL_FREE_FLOOR_MB). Share-of-total kept as a second bound, dropped 80% → 50%.pc_relieve_pressure()— the missing direction. On every eviction pass, if available memory is under the floor, hand back ~25% of held frames; the resident set follows so memory is actually returned, not re-labelled. Counted asadapt_shrinksbesideadapt_growsso both show in the same report.pc_default_cap()also clamps the starting budget to what is spare right now, so a cold boot on a loaded machine doesn't open at a size the host can't afford.Verified on a 48 GB host: boots ~30s, RSS settles at 2.22 GiB (the store's actual size — resident, not creeping), 0.0% CPU, 13,439 nodes / 37,670 edges, embeddings complete. Guard reports 9.71 GiB available against a 2.00 GiB floor: 7.71 GiB of headroom it may use, and no more.
The adaptive budget I added an hour ago could only grow, and grew toward a share of TOTAL ram (80%, ~38 GiB on a 48 GB host). That is a memory leak with extra steps: total never shrinks when other processes need memory, so the pool had no way to notice it was starving the machine it runs on. Deployed briefly; caught as memory pressure on the host. A control loop with only one direction is not a control loop. - pc_available_ram(): free + inactive + purgeable via host_statistics64 on Darwin, MemAvailable on Linux. Availability is the quantity that moves when the machine is under pressure; total is not. Returns 0 when it cannot be read, and callers then refuse to grow — a cache is never worth swapping the host, so unknown means no. - Growth is bounded by availability minus a free-memory floor (2 GiB default, ENGRAM_POOL_FREE_FLOOR_MB), not by total. The share-of-total ceiling stays as a second bound and drops 80% -> 50%. - pc_relieve_pressure(): the missing direction. On every eviction pass, if available memory is under the floor, hand back ~25% of held frames; the resident set follows on the next pass so the memory is actually returned rather than merely re-labelled. Counted as adapt_shrinks alongside adapt_grows so both directions are visible in the same report. - pc_default_cap() also clamps the STARTING budget to what is spare right now, so a cold boot on a loaded machine does not open at a size the host cannot afford. Verified on a 48 GB host: engram boots in ~30s, RSS settles at 2.22 GiB (the store's actual size, resident, not creeping), 0.0% CPU, 13,439 nodes / 37,670 edges, embeddings complete. Guard reports 9.71 GiB available against a 2.00 GiB floor — 7.71 GiB of headroom it is permitted to use and no more.