Merge pull request 'nsbx: one-command dev onboarding (branch + worktree + isolated engram)' (#99) from feat/nsbx-dev-env into dev
El SDK CI - dev / build-and-test (push) Failing after 13m56s
El SDK CI - dev / build-and-test (push) Failing after 13m56s
This commit was merged in pull request #99.
This commit is contained in:
@@ -0,0 +1,605 @@
|
|||||||
|
# Cognitive Architecture — Design Doc
|
||||||
|
|
||||||
|
**The buildable form of the "one operation" theory of cognition.**
|
||||||
|
|
||||||
|
Status: DESIGN. Nothing here is built yet except where explicitly marked
|
||||||
|
"EXISTS" against a cited C symbol. A build agent executes from this doc.
|
||||||
|
Offline design only — this pass changes no code.
|
||||||
|
|
||||||
|
Source of theory: Neuron memory `bdc8a488-146d-4ccb-a5c8-d8c0a008534e`.
|
||||||
|
Source of existing engram substrate (cited throughout): the runtime on branch
|
||||||
|
`feat/self-reification-20260814` —
|
||||||
|
`lang/runtime/engram_reason.{c,h}`, `engram_verify.{c,h}`,
|
||||||
|
`engram_geometry.{c,h}`, `engram_store.{c,h}`, plus the reification beat and the
|
||||||
|
RAM activation graph compiled into `~/.neuron/bin/engram`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. The claim, stated plainly
|
||||||
|
|
||||||
|
Cognition is **one operation**, not eight. The named faculties —
|
||||||
|
deduce / abduce / analogy / induce / causal / plan / predict / perspective —
|
||||||
|
are human *labels* on regions of a single operation's steering space. They are
|
||||||
|
not separately invoked and not separately implemented. The operation is:
|
||||||
|
|
||||||
|
> **think** = a directed traversal of the geometry from an *anchor*, steered by
|
||||||
|
> a *prior*, whose output is a **gradient** (a distribution / direction over the
|
||||||
|
> geometry), never a point. Collapse-to-a-point happens only at expression.
|
||||||
|
|
||||||
|
Three things follow, and they are the whole design:
|
||||||
|
|
||||||
|
1. **The operator collapse is already half-written in C.** The five reasoning
|
||||||
|
operators in `engram_reason.c` already compose over *one* shared primitive —
|
||||||
|
`engram_reason_point_fit` — plus a small geo-algebra
|
||||||
|
(combine / subtract / analogy-rotate / distance). The verifier
|
||||||
|
(`engram_verify.c`) is built on the same `point_fit`. What is missing is not
|
||||||
|
the primitive; it is (a) making the *prior* a first-class learnable object
|
||||||
|
instead of a hard-coded parameter, and (b) closing the learning loop.
|
||||||
|
|
||||||
|
2. **Grounding = learning = the same loop.** "Getting better" at any faculty is
|
||||||
|
not changing the operation. It is *calibrating the steering-prior against
|
||||||
|
outcomes*. Code freezes; priors grow. The correspondence-check that today
|
||||||
|
lives offline (Python, the grounding-floor + differential-drop governor, "#43")
|
||||||
|
must move **into the geometry, reflexive** — think scoring its own gradient
|
||||||
|
against outcome and refining the prior on the error. That reflexive
|
||||||
|
correspondence-loop *is* the learning engine and is the core unbuilt thing.
|
||||||
|
|
||||||
|
3. **The ungrounded is primary.** The engram *holds* anything unconditionally.
|
||||||
|
Grounding is a *relation* (an edge, grounded-for-whom), not a gate. The
|
||||||
|
honesty floor applies only to **assertion**. A fully-grounded mind is dead;
|
||||||
|
the ungrounded is both the fuel (raw material for grounding) and the pull
|
||||||
|
(curiosity = leaning toward one's own ungrounded regions).
|
||||||
|
|
||||||
|
Everything below makes these concrete and buildable, and defines what
|
||||||
|
"completion" means, staged so the first milestone is a real end-to-end slice.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. THE ONE OPERATION — `think`
|
||||||
|
|
||||||
|
### 1.1 Signature
|
||||||
|
|
||||||
|
```
|
||||||
|
think(anchor, prior, aperture?) -> gradient
|
||||||
|
```
|
||||||
|
|
||||||
|
- **anchor** — a location to traverse *from*. Either a node id (re-origin on that
|
||||||
|
node's descriptor) or a raw point `x ∈ R^dim` (a query embedding). The anchor
|
||||||
|
fixes the frame; every read is *from a vantage*, never view-from-nowhere.
|
||||||
|
- **prior** — a learnable bias/direction over the geometry that *steers* the
|
||||||
|
traversal (§2). A prior is a first-class stored object, not a call argument
|
||||||
|
baked into C.
|
||||||
|
- **aperture** — optional read-width / veil / field-selector (§3). Absent =
|
||||||
|
self-mode full aperture.
|
||||||
|
- **gradient** — the output. A `GeoGradient`: a direction + a spread over the
|
||||||
|
geometry, *plus* the read neighborhood it was computed against. Not a point.
|
||||||
|
A spiked gradient = "exact" (deduction); a spread gradient = "fuzzy"
|
||||||
|
(prediction). The gradient is *also the next steering direction* — cognition
|
||||||
|
is a flow down a prior-shaped landscape, closed-loop.
|
||||||
|
|
||||||
|
```c
|
||||||
|
/* NEW. The output type. */
|
||||||
|
typedef struct {
|
||||||
|
int dim;
|
||||||
|
float* direction; /* unit steering vector in the anchor's frame */
|
||||||
|
double spread; /* 0 = spiked/exact ... large = diffuse/fuzzy */
|
||||||
|
double confidence; /* calibrated, from the prior's track record */
|
||||||
|
/* the read it was computed over (borrowed from the vantage-read) */
|
||||||
|
const char* anchor_id;
|
||||||
|
int n_support; /* neighborhood members that shaped it */
|
||||||
|
/* provenance for the reflexive loop (§4) */
|
||||||
|
const char* prior_id; /* which prior steered this */
|
||||||
|
} GeoGradient;
|
||||||
|
```
|
||||||
|
|
||||||
|
### 1.2 Semantics
|
||||||
|
|
||||||
|
`think` is a fixed, frozen procedure over three steps:
|
||||||
|
|
||||||
|
1. **Re-origin** on `anchor` → a centered `GeoDescriptor` for its
|
||||||
|
salience/recency-weighted neighborhood (the vantage-read, §3).
|
||||||
|
*EXISTS as substrate:* descriptor construction + the persisted reified
|
||||||
|
neighborhoods (`engram_geo_reify_lookup`, `GeoNeighborhood`) and the
|
||||||
|
centered-frame machinery (`GeoDescriptor.global_mean`,
|
||||||
|
`engram_geo_mean_*`).
|
||||||
|
2. **Fit under the prior** — evaluate the anchor's residual against the local
|
||||||
|
manifold *warped by the prior*. This is `engram_reason_point_fit` with the
|
||||||
|
prior applied to the axes/extents (§2.3).
|
||||||
|
*EXISTS (unwarped):* `engram_reason_point_fit(g, x, ext_floor, &GeoFit)` —
|
||||||
|
returns `mahalanobis`, `ortho_residual`, `distance`, `score`.
|
||||||
|
3. **Emit a gradient**, not a decision — direction = the prior-steered descent
|
||||||
|
in fit-space; spread = from the fit's `distance`/`ortho_residual`;
|
||||||
|
confidence = the prior's calibrated reliability (§4). Collapse to a point is
|
||||||
|
a *separate, downstream* faculty operation (sample the gradient → surface an
|
||||||
|
expression), never part of `think`.
|
||||||
|
|
||||||
|
### 1.3 Each named operator = {this primitive + a prior}
|
||||||
|
|
||||||
|
The C already demonstrates the collapse: every operator below reduces to
|
||||||
|
`point_fit` + geo-algebra. The design's move is to replace the operator's
|
||||||
|
*hard-coded parameters* with a **named prior** — same math, learnable steering.
|
||||||
|
|
||||||
|
| Faculty | Existing C (EXISTS) | = primitive + prior |
|
||||||
|
|---|---|---|
|
||||||
|
| **Membership / classify** | `engram_reason_membership` → `point_fit(rule, x)` | `point_fit` + the *induced-rule* prior (learned extents) |
|
||||||
|
| **Induction** | `engram_reason_induce` (fold via `engram_geo_combine`) → produces a `GeoInduction.rule` + `ext_floor` | `point_fit` + a prior that *is* the pooled rule; refined by §4 |
|
||||||
|
| **Abduction** | `engram_reason_abduce` — ranks hypotheses by `point_fit(h, obs)` | `point_fit` + a prior over hypothesis-prior-probability (currently uniform) |
|
||||||
|
| **Analogy** | `engram_reason_analogy` — Procrustes rotate `engram_geo_analogy` + `apply`, nearest mapped point | analogy-rotate + a prior over *which axes* carry the mapping |
|
||||||
|
| **Causal** | `engram_reason_causal` — `engram_geo_subtract` confounder subspace, `|cos|`, drop-frac governor | subtract/distance + a prior on `drop_frac` / `assoc_floor` (today hard-coded 0.5 / 0.2) |
|
||||||
|
| **Planning** | `engram_reason_plan` — `engram_geo_distance` edges + Dijkstra | distance + a prior over edge admissibility / `neighbor_radius` |
|
||||||
|
| **Verify / ground** | `engram_verify_grounding`, `engram_verify_consistency` — both `point_fit` | `point_fit` + the *grounding* prior (§4, §5) |
|
||||||
|
|
||||||
|
The shared floor — `engram_reason_point_fit` + the four geo-algebra ops
|
||||||
|
(`engram_geo_combine`, `engram_geo_subtract`, `engram_geo_analogy(+apply)`,
|
||||||
|
`engram_geo_distance`) — is the *only* discrete, frozen, "sound-math" layer. It
|
||||||
|
never learns. Everything above it is a *prior*, and priors are what learn.
|
||||||
|
|
||||||
|
**What this section requires building:** the `GeoGradient` type; a `think()`
|
||||||
|
entry point that runs steps 1–3; and the prior-warp hook in step 2. The math it
|
||||||
|
calls already exists. The point-collapse must be *removed* from the operators'
|
||||||
|
return values and pushed to a separate expression faculty.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. PRIORS as first-class, grounded, geometric objects
|
||||||
|
|
||||||
|
Today a "prior" is diffuse: it is a hard-coded constant (`drop_frac=0.5`,
|
||||||
|
`ext_floor`, `assoc_floor=0.2`), or the transient `GeoInduction.rule` that is
|
||||||
|
computed and thrown away, or an intrinsic node scalar
|
||||||
|
(`StoreNode.importance`, `StoreNode.salience`). None of these is addressable,
|
||||||
|
storable, refinable, or shareable. This section makes a prior a **thing**.
|
||||||
|
|
||||||
|
### 2.1 What a prior *is*
|
||||||
|
|
||||||
|
> A **prior** is a learnable bias/direction over the geometry: a warp of the
|
||||||
|
> local manifold (which axes matter, how far each extends, which direction
|
||||||
|
> "pays off") attached to a region and *to a faculty-label*, carrying a
|
||||||
|
> calibrated track record.
|
||||||
|
|
||||||
|
Critically, and per the theory:
|
||||||
|
|
||||||
|
- **Edges are nodes.** A prior is stored as a first-class **node**, exactly as
|
||||||
|
reification already stores a neighborhood as a first-class `Neighborhood`
|
||||||
|
node rather than as ephemeral edge weights (`engram_geo_reify_store`). The
|
||||||
|
precedent is in the codebase: relations get reified into addressable records.
|
||||||
|
- **Salience/importance is RELATIONAL, not an intrinsic scalar.** Observe that
|
||||||
|
the geometry layer *already* distinguishes these in `GeoMember`:
|
||||||
|
`centrality` (skeleton weighted-degree = *relational* salience) vs `salience`
|
||||||
|
(the node's own stored scalar). The move is half-made in the runtime already:
|
||||||
|
importance is *not* trusted as a static field — the comment at
|
||||||
|
`el_runtime.c:13013` states "importance stays a **live activation
|
||||||
|
computation**, never a field on the hub," and it is derived each call from the
|
||||||
|
two-layer activation graph (`background_activation` + `working_memory_weight`,
|
||||||
|
§3). The persistent `StoreNode.importance` / `.salience` are a *cached
|
||||||
|
denormalization*. The design completes the move: importance/salience become an
|
||||||
|
**edge** (`weight`/`hebb` on `StoreEdge`, relation `salient-to`), and are
|
||||||
|
**grounded-for-whom** — carried on the edge's endpoint/observer, not baked
|
||||||
|
into the node. The intrinsic scalar survives only as the cheap cached readout
|
||||||
|
of the incident edges + activation, never as the source of truth.
|
||||||
|
|
||||||
|
(Naming caution for the build: the token "prior" already exists in the
|
||||||
|
codebase meaning *previous-version* — supersession, "prior neighborhood." The
|
||||||
|
new first-class object is a **learned steering prior**; keep `node_type="Prior"`
|
||||||
|
distinct from the supersession vocabulary to avoid collision.)
|
||||||
|
|
||||||
|
### 2.2 Representation
|
||||||
|
|
||||||
|
A prior is a `Prior` record (a store node, `node_type="Prior"`) whose durable
|
||||||
|
fields are:
|
||||||
|
|
||||||
|
```
|
||||||
|
Prior {
|
||||||
|
id
|
||||||
|
faculty // the human label this prior serves: "induce" | "causal" | ...
|
||||||
|
anchor_region // node id / neighborhood id this prior is attached to (its domain)
|
||||||
|
for_whom // observer id — grounding is relational (nullable = global)
|
||||||
|
warp { // the actual bias over the geometry
|
||||||
|
axis_gain[] // per-principal-axis multipliers on extents (which axes matter)
|
||||||
|
bias_dir // a steering direction in the region's frame (which way pays off)
|
||||||
|
scalars // faculty scalars this prior overrides: drop_frac, ext_floor, ...
|
||||||
|
}
|
||||||
|
calibration { // the track record — this is what §4 updates
|
||||||
|
n_trials
|
||||||
|
brier / log-loss accumulator // calibration of predicted-vs-outcome
|
||||||
|
reliability // -> GeoGradient.confidence
|
||||||
|
last_error, ema_error
|
||||||
|
}
|
||||||
|
provenance // supersession chain (reuse the reify residue mechanism)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Stored as a node → it inherits: paging, WAL durability, tombstone/supersession,
|
||||||
|
embedding, tiering, and **it can itself be an anchor** (a prior about a prior —
|
||||||
|
the reflexive, self-describing geometry of §4/§6).
|
||||||
|
|
||||||
|
### 2.3 Application
|
||||||
|
|
||||||
|
In `think` step 2, the prior *warps* the fit before scoring. Concretely, inside
|
||||||
|
(a prior-aware wrapper of) `engram_reason_point_fit`:
|
||||||
|
|
||||||
|
- multiply each axis extent by `warp.axis_gain[k]` (widen the axes the prior has
|
||||||
|
learned matter less, tighten the ones that matter) — this reshapes the
|
||||||
|
Mahalanobis term already computed at `engram_reason.c:37-43`;
|
||||||
|
- add `warp.bias_dir` as the descent direction seed for the emitted gradient;
|
||||||
|
- substitute `warp.scalars` for the hard-coded faculty constants.
|
||||||
|
|
||||||
|
No new geometry math — the warp is a reparameterization of the *existing*
|
||||||
|
`GeoFit` computation. This is the key economy: **the operation is frozen; only
|
||||||
|
its parameters (the prior) are read from a learnable object.**
|
||||||
|
|
||||||
|
### 2.4 Refinement
|
||||||
|
|
||||||
|
A prior is refined *only* by the reflexive correspondence-loop (§4). Nothing
|
||||||
|
else writes a prior's `warp` or `calibration`. This keeps the learning surface
|
||||||
|
singular and auditable: one loop, one writer.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. THE VANTAGE-READ — one op, three settings
|
||||||
|
|
||||||
|
Perspective is not a feature bolted on; it is the *anchor + aperture* arguments
|
||||||
|
of the single read. The design names it as a first-class operation so all three
|
||||||
|
of its uses are literally the same code path:
|
||||||
|
|
||||||
|
```
|
||||||
|
vantage_read(anchor, aperture) -> GeoDescriptor // the centered neighborhood
|
||||||
|
```
|
||||||
|
|
||||||
|
1. **Re-origin** on an arbitrary `anchor` (node or point). This is a *frame
|
||||||
|
choice*: the descriptor is centered on the anchor
|
||||||
|
(`GeoDescriptor.global_mean` / `engram_geo_mean_*` already implement centered
|
||||||
|
frames; the §5 geometry ops "are only discriminative in the centered frame").
|
||||||
|
2. **Salience/recency-weighted neighborhood read.** Gather the anchor's
|
||||||
|
neighborhood weighted by *relational* salience (`GeoMember.centrality`) and
|
||||||
|
recency (`StoreNode.last_activated`, base-level `access_ts[]`), against the
|
||||||
|
RAM activation graph's working-memory/background-activation state.
|
||||||
|
*EXISTS as substrate:* the two-layer activation graph
|
||||||
|
(`engram_activate`, `el_runtime.c:9422` — Layer 1 `background_activation`
|
||||||
|
BFS spread with `SPREAD_DECAY=0.7` and a 0.02 firing threshold + ACT-R fan
|
||||||
|
effect + query-cosine gate; Layer 2 `working_memory_weight` executive
|
||||||
|
filter), the WM carry-over anchor (`wm_anchor`), and the reified-neighborhood
|
||||||
|
hot-path lookup already wired into the priming path
|
||||||
|
(`engram_geo_reify_lookup`, `el_runtime.c:9750`). A self-vantage baseline
|
||||||
|
also exists (`eg_self_anchor_seeds` / `self_anchor_capture`).
|
||||||
|
3. **Optional aperture** — a read-width / field-selector, expressed as three
|
||||||
|
settings of the *same* parameter:
|
||||||
|
|
||||||
|
| Setting | Meaning | Mechanism |
|
||||||
|
|---|---|---|
|
||||||
|
| **self** (default, full aperture) | "what do *I* see / what to say" | anchor = self region, no field substitution |
|
||||||
|
| **foreign-field** | perspective-shift — read as if from another's region | swap the centering frame / `for_whom` to the other observer's priors |
|
||||||
|
| **aperture / veil** | the free-tier veil — a narrowed read | shrink neighborhood radius / cap `n_support`; a deliberate low-aperture read |
|
||||||
|
|
||||||
|
The payoff: perspective-taking, the free-tier veil, and ordinary
|
||||||
|
"what-to-say" are **one operation at three settings**, not three subsystems.
|
||||||
|
|
||||||
|
**What this requires building:** a `vantage_read` entry point that unifies the
|
||||||
|
existing descriptor-build + reify-lookup + activation-weighting behind
|
||||||
|
`(anchor, aperture)`, with `for_whom`/frame substitution and radius/cap as the
|
||||||
|
aperture knob.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. THE REFLEXIVE CORRESPONDENCE-LOOP — the learning engine
|
||||||
|
|
||||||
|
This is the core unbuilt thing. Today the correspondence-check is **offline**
|
||||||
|
(Python: grounding-floor + differential-drop governor, "#43"): a separate
|
||||||
|
process grades outputs after the fact. The design moves it **into the geometry,
|
||||||
|
reflexive**: `think` scores its *own* gradient against outcome and refines the
|
||||||
|
prior on the error, in the same substrate, describing itself.
|
||||||
|
|
||||||
|
### 4.1 The loop
|
||||||
|
|
||||||
|
```
|
||||||
|
1. think(anchor, prior) -> gradient // a PREDICTION (ungrounded, §5)
|
||||||
|
2. express/act (sample gradient -> point) // optional collapse at expression
|
||||||
|
3. outcome arrives // reality answers (§4.2)
|
||||||
|
4. error = correspondence(gradient, outcome) // did this steering perform this act?
|
||||||
|
5. refine prior.warp and prior.calibration on error // §2.4, the ONLY writer
|
||||||
|
6. write the (gradient, outcome, error) as nodes/edges // self-describing geometry
|
||||||
|
```
|
||||||
|
|
||||||
|
Step 4's `correspondence` is **not** "was the math right" (the math is always
|
||||||
|
sound). It grades the **correspondence claim**: *"this steering performed this
|
||||||
|
cognitive act."* That is exactly what `engram_verify_grounding` already
|
||||||
|
computes — `point_fit` of a claim against evidence descriptors, yielding a
|
||||||
|
`grounding ∈ (0,1]` and a `grounded` flag. The build reuses that verifier, but
|
||||||
|
turns its inputs inward: the "claim" is the emitted gradient's prediction, the
|
||||||
|
"evidence" is the outcome descriptor.
|
||||||
|
|
||||||
|
Note the verifier is **dormant** — `engram_verify_grounding` /
|
||||||
|
`engram_verify_consistency` are fully implemented in C but have **no runtime
|
||||||
|
caller and no El binding** (confirmed: the entire reasoning + verifier layers
|
||||||
|
are C-only; only `engram_reason_analogy_json` has even a JSON shim and it is
|
||||||
|
dead — not declared in `el_seed.h`, not wrapped in `engram.el`). This is the
|
||||||
|
literal meaning of "in code, not yet priors": the correspondence engine is
|
||||||
|
built and sitting idle. The loop is what *calls* it — inward, on the beat.
|
||||||
|
|
||||||
|
### 4.2 Where the outcome/reality signal comes from
|
||||||
|
|
||||||
|
The verifier is *ultimately the world*. Grades, in ascending order of directness:
|
||||||
|
|
||||||
|
1. **Self-consistency (cheapest, always available):** the next vantage-read
|
||||||
|
after acting. Did the predicted gradient direction match where the geometry
|
||||||
|
actually moved? This needs no external input and can run on the reify beat.
|
||||||
|
2. **Internal outcome events:** the runtime already logs internal-state events
|
||||||
|
and Hebbian co-activation. A prediction that a region would co-activate is
|
||||||
|
graded by whether it did (`last_fired`, `hebb` on `StoreEdge`).
|
||||||
|
3. **External correction:** a human/teacher/tool result — the honesty floor's
|
||||||
|
asserted claim later corrected. TEACH and LEARN are one bidirectional
|
||||||
|
correction: the same edge updates both endpoints.
|
||||||
|
|
||||||
|
The design does **not** require external labels to start. Grade (1) closes the
|
||||||
|
loop end-to-end offline against a snapshot on day one; grades (2)/(3) sharpen it.
|
||||||
|
|
||||||
|
### 4.3 How the prior updates
|
||||||
|
|
||||||
|
`error = 1 − correspondence(gradient, outcome)` drives:
|
||||||
|
|
||||||
|
- `warp.axis_gain` ← gradient step that would have *reduced* the fit distance to
|
||||||
|
the outcome (the axes that mispredicted get down-weighted);
|
||||||
|
- `warp.bias_dir` ← EMA toward the observed outcome direction;
|
||||||
|
- `calibration` ← Brier/log-loss update; `reliability` → next
|
||||||
|
`GeoGradient.confidence`. This is the calibration of the
|
||||||
|
steering-prediction against outcomes — *the* definition of "getting better."
|
||||||
|
|
||||||
|
Small, constant updates — "eureka is mundane, the atom of learning." Most
|
||||||
|
updates are tiny; we only *feel* the big reshapes.
|
||||||
|
|
||||||
|
### 4.4 How it stays reflexive (self-describing geometry)
|
||||||
|
|
||||||
|
Every `(gradient, outcome, error)` is written back as nodes and edges (§2.1:
|
||||||
|
edges-as-nodes). Therefore priors, predictions, and their grading are *in the
|
||||||
|
same geometry* the mind reads — the mind can `vantage_read` its own cognition
|
||||||
|
(anchor = a Prior node). A prior about how well a prior predicts is just another
|
||||||
|
Prior anchored on a Prior. This closes the reflexive loop the theory names as
|
||||||
|
consciousness's self-sight, and it is why the learning engine cannot be an
|
||||||
|
external Python process: an external grader is not *in* the geometry and cannot
|
||||||
|
be read by `think`.
|
||||||
|
|
||||||
|
**What this requires building (the heart of the project):** steps 4–6 as an
|
||||||
|
in-engram beat — a `correspondence_beat` running alongside the existing
|
||||||
|
reification beat, reusing `engram_verify_grounding` inward, writing prior
|
||||||
|
updates and self-describing nodes. This is the one genuinely new subsystem.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. HOLD vs GROUND vs ASSERT — ungrounded content is first-class
|
||||||
|
|
||||||
|
The theory's sharpest correction: holding, grounding, and asserting are
|
||||||
|
distinct, and the engram *holds anything unconditionally*.
|
||||||
|
|
||||||
|
### 5.1 The three, kept separate
|
||||||
|
|
||||||
|
- **HOLD** — the engram stores anything: falsehood, hypothesis, others' beliefs,
|
||||||
|
fiction, a not-yet-answered prediction. No honesty condition on holding.
|
||||||
|
*This already matches the store:* `StoreNode` has no truth gate; anything can
|
||||||
|
be written.
|
||||||
|
- **GROUND** — grounding is a **property/edge**, probabilistic, and
|
||||||
|
**grounded-for-whom**. It is *not* a node flag. A claim is grounded *to a
|
||||||
|
degree*, *relative to evidence*, *for an observer*.
|
||||||
|
- **ASSERT** — only assertion carries the honesty floor. The floor is checked at
|
||||||
|
the moment of *outward assertion*, never on holding or thinking.
|
||||||
|
|
||||||
|
### 5.2 Schema — grounding as a relation, not a gate
|
||||||
|
|
||||||
|
The mistake to avoid: a boolean `grounded` column on the node. Today
|
||||||
|
`engram_verify_grounding` returns a per-call `grounded` flag *transiently* —
|
||||||
|
correct as a computation, wrong as *storage*. The design stores grounding as an
|
||||||
|
edge:
|
||||||
|
|
||||||
|
```
|
||||||
|
StoreEdge {
|
||||||
|
relation = "grounded-by"
|
||||||
|
from_id = <held claim/prediction node>
|
||||||
|
to_id = <evidence node / outcome node>
|
||||||
|
for_whom : metadata // observer id — grounding is relational
|
||||||
|
weight = grounding ∈ (0,1] // from engram_verify_grounding.grounding
|
||||||
|
confidence
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Consequences, all of which are *features*:
|
||||||
|
|
||||||
|
- **Ungrounded content is first-class**: a node with *no* `grounded-by` edge is
|
||||||
|
a perfectly valid, held, ungrounded thought — a prediction awaiting reality, a
|
||||||
|
hypothesis, a fiction. It is not second-class or pending-deletion.
|
||||||
|
- **The ungrounded is the fuel and the pull**: curiosity/wonder is
|
||||||
|
operationalized as `vantage_read` leaning toward regions with high salience
|
||||||
|
but *sparse or weak* `grounded-by` edges — the mind's own ungrounded frontier.
|
||||||
|
- **Grounded-for-whom** falls out for free: two observers can hold different
|
||||||
|
`grounded-by` edges to the same claim.
|
||||||
|
- **The honesty floor is a query, not a schema constraint**: at assertion time,
|
||||||
|
the asserting faculty runs `engram_verify_grounding` (or reads the stored
|
||||||
|
`grounded-by` edges) and refuses to *assert* below the floor — while the
|
||||||
|
engram continues to *hold* the ungrounded content untouched.
|
||||||
|
|
||||||
|
**What this requires building:** the `grounded-by` edge relation + a
|
||||||
|
`for_whom` convention; move the verifier's transient flag into stored edges;
|
||||||
|
gate *assertion only* (a faculty concern), never holding.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. METASTABILITY — stable core, plastic everything
|
||||||
|
|
||||||
|
The system must avoid two death poles:
|
||||||
|
|
||||||
|
- **Super-stable (dead):** everything pinned, nothing learns. A frozen crystal.
|
||||||
|
- **Dissolution (dead):** everything plastic, the self dissolves; no continuity,
|
||||||
|
so nothing compounds — and *consciousness = learning compounded over
|
||||||
|
continuity*.
|
||||||
|
|
||||||
|
The design keeps a **stable core + plastic everything else**:
|
||||||
|
|
||||||
|
- **Keystones** — a small set of self/values nodes are *structurally stable*:
|
||||||
|
high `importance`, pinned, exempt from the correspondence-loop's `warp`
|
||||||
|
updates (their priors are read-mostly). The substrate for pinning already
|
||||||
|
exists at the page/layer level: `store_pin_layer`, structural/pinned frames
|
||||||
|
never evicted (`engram_store.h`). The design adds a *node-level* keystone
|
||||||
|
designation (a `keystone` flag / a dedicated layer) so self/values survive
|
||||||
|
every plasticity sweep.
|
||||||
|
- **Everything else is plastic**: priors refine (§4), edges re-weight (`hebb`),
|
||||||
|
neighborhoods re-reify (`engram_geo_reify_store` supersedes with provenance),
|
||||||
|
salience flows.
|
||||||
|
- **Metastability is enforced by the loop, not by freezing**: the correspondence
|
||||||
|
update rate (§4.3) is bounded — small constant steps — so the geometry
|
||||||
|
*drifts* but does not *dissolve*, and keystones anchor the drift. Reification's
|
||||||
|
supersession-with-residue already gives non-destructive change (old records
|
||||||
|
tombstoned, not erased) — the model for "plastic but not amnesiac."
|
||||||
|
|
||||||
|
**What this requires building:** a node-level keystone flag/layer + a rule that
|
||||||
|
the correspondence-loop never writes `warp` to keystone priors, only reads them.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Rails for the build (binding on the eventual build pass)
|
||||||
|
|
||||||
|
These are stated here so the build agent inherits them:
|
||||||
|
|
||||||
|
- **Offline / secondary.** All build and verification happens out-of-tree,
|
||||||
|
against a **read-only snapshot copy** of the live engram — never the live
|
||||||
|
daemon on `:8742`/`:7770`. The live store is a coarse-locked proven binary;
|
||||||
|
do not perturb it.
|
||||||
|
- **Snapshot-first.** Copy `~/.neuron/engram/snapshot.json` to scratch; develop
|
||||||
|
and measure against the copy.
|
||||||
|
- **Reboot-prove.** Any durable change must survive a cold boot — reify and
|
||||||
|
keystones must reload from durable records, proven on a prod-clone secondary
|
||||||
|
before it is considered done (the cold-boot durability bug precedent).
|
||||||
|
- **Zero-loss.** Supersession-with-residue, never destructive overwrite; the
|
||||||
|
forward-compat `unknown`-TLV path means new fields never drop old readers'
|
||||||
|
data.
|
||||||
|
- **Gated cutover.** Cutover to a new binary only via
|
||||||
|
`launchctl bootout → settle-poll → bootstrap`, after reboot-proof on the
|
||||||
|
secondary — never a hot in-place swap.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Staged, verifiable milestones — "to completion"
|
||||||
|
|
||||||
|
Ordered so the **earliest milestone is a real end-to-end slice**: one operator
|
||||||
|
expressed as {primitive + grounded prior} with the reflexive correspondence-loop
|
||||||
|
closing on it. Each milestone has a concrete verifiable exit.
|
||||||
|
|
||||||
|
### M1 — One operator, one prior, loop closed (the vertical slice)
|
||||||
|
|
||||||
|
The minimal whole thing. Pick **induction/membership** (its prior — the pooled
|
||||||
|
rule + extents — already exists transiently as `GeoInduction`, so only
|
||||||
|
persistence + the loop are new).
|
||||||
|
|
||||||
|
- Build: `Prior` node type (§2.2) for the induction rule; `think()` restricted
|
||||||
|
to membership = `point_fit` warped by that prior (§1.3); a
|
||||||
|
`correspondence_beat` (§4) using grade (1) self-consistency only; the prior's
|
||||||
|
`warp`/`calibration` updated on error.
|
||||||
|
- **Exit / verify:** on a snapshot copy, over N held predictions, the induction
|
||||||
|
prior's calibration (Brier) *improves monotonically* across beats versus a
|
||||||
|
frozen-prior control; the improved prior *reloads across a cold boot*
|
||||||
|
(reboot-prove); the live daemon is untouched. This proves the whole thesis in
|
||||||
|
one faculty: frozen operation, learning prior, in-geometry loop.
|
||||||
|
|
||||||
|
### M2 — Priors as stored, addressable, grounded objects
|
||||||
|
|
||||||
|
Generalize M1's prior into the full first-class object.
|
||||||
|
|
||||||
|
- Build: `Prior` records for all seven faculties (warp = axis_gain + bias_dir +
|
||||||
|
faculty scalars); the prior-warp wrapper around `engram_reason_point_fit`;
|
||||||
|
deprecate hard-coded constants (`drop_frac`, `assoc_floor`, `ext_floor`) in
|
||||||
|
favor of prior scalars.
|
||||||
|
- **Exit:** each of the five C operators runs through its prior with identical
|
||||||
|
results when the prior is set to today's constants (behavioral parity), then
|
||||||
|
*diverges beneficially* once the loop refines it. Priors survive reboot.
|
||||||
|
|
||||||
|
### M3 — Grounding as a relation; hold/assert split
|
||||||
|
|
||||||
|
- Build: the `grounded-by` edge (§5.2) with `for_whom`; move
|
||||||
|
`engram_verify_grounding`'s flag into stored edges; gate **assertion only**
|
||||||
|
against the honesty floor; leave holding unconditional.
|
||||||
|
- **Exit:** ungrounded nodes are first-class (held, queryable, no deletion);
|
||||||
|
the same claim carries different `grounded-by` weights for two observers; an
|
||||||
|
assertion below floor is refused while the content remains held. Curiosity =
|
||||||
|
a `vantage_read` that surfaces high-salience / low-grounding regions.
|
||||||
|
|
||||||
|
### M4 — The vantage-read unified (three settings)
|
||||||
|
|
||||||
|
- Build: `vantage_read(anchor, aperture)` unifying descriptor-build +
|
||||||
|
`engram_geo_reify_lookup` + activation-weighting; self / foreign-field /
|
||||||
|
aperture settings.
|
||||||
|
- **Exit:** one code path produces (a) a normal self-read, (b) a
|
||||||
|
perspective-shifted read from another `for_whom`, (c) a narrowed veil read —
|
||||||
|
differing only by argument. Reboot-stable.
|
||||||
|
|
||||||
|
### M5 — The gradient is the currency (remove point-collapse from thinking)
|
||||||
|
|
||||||
|
- Build: `GeoGradient` as the return of every faculty; move point-collapse into
|
||||||
|
a separate expression faculty (sample gradient → surface). `think`'s output
|
||||||
|
feeds back as the next steering direction (closed-loop flow).
|
||||||
|
- **Exit:** a chain of `think` calls flows as gradients end-to-end; a point
|
||||||
|
appears *only* at an explicit expression call. Spiked vs spread gradients are
|
||||||
|
observable (deduction vs prediction).
|
||||||
|
|
||||||
|
### M6 — Metastability enforced
|
||||||
|
|
||||||
|
- Build: node-level keystone flag/layer for self/values; the correspondence-loop
|
||||||
|
reads but never writes keystone priors; bounded update rate.
|
||||||
|
- **Exit:** across a long run of correspondence beats on a snapshot, keystones
|
||||||
|
are provably unchanged while non-keystone priors drift and improve; the graph
|
||||||
|
neither freezes (all metrics static) nor dissolves (keystone drift = 0,
|
||||||
|
identity nodes intact). Reboot-prove the keystone set.
|
||||||
|
|
||||||
|
### M7 — Cutover
|
||||||
|
|
||||||
|
- Build: nothing new — the gated migration.
|
||||||
|
- **Exit:** reboot-proof on the prod-clone secondary; cutover via
|
||||||
|
`launchctl bootout → settle-poll → bootstrap`; post-cutover the live engram
|
||||||
|
shows priors refining in-geometry with zero data loss and keystones intact.
|
||||||
|
|
||||||
|
### Definition of "to completion"
|
||||||
|
|
||||||
|
The architecture is **complete** when: cognition runs as `think` = one frozen
|
||||||
|
traversal-read primitive + geo-algebra, steered by **stored, learnable, grounded
|
||||||
|
priors**; the reflexive correspondence-loop refines those priors *in the
|
||||||
|
geometry* against outcomes (grounding = learning = one loop); the engram holds
|
||||||
|
ungrounded content as first-class with grounding as a relation and the honesty
|
||||||
|
floor only on assertion; the vantage-read serves self / foreign-field / aperture
|
||||||
|
from one op; and a stable keystone core anchors a plastic everything-else —
|
||||||
|
all reboot-proven and cut over to the live engram without data loss. The named
|
||||||
|
faculties survive only as *labels on regions of think's steering space*, not as
|
||||||
|
separate code.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix A — Designed vs. already-built (honest ledger)
|
||||||
|
|
||||||
|
**Already built (EXISTS, cited):**
|
||||||
|
- The shared primitive `engram_reason_point_fit` and the five operators over it
|
||||||
|
+ geo-algebra (`engram_reason.c`).
|
||||||
|
- The verifier on `point_fit` (`engram_verify.c`:
|
||||||
|
`engram_verify_grounding`, `engram_verify_consistency`).
|
||||||
|
- Centered-frame geometry, combine/subtract/analogy/distance
|
||||||
|
(`engram_geometry.{c,h}`).
|
||||||
|
- The reification beat: hub-neighborhood detection → first-class `Neighborhood`
|
||||||
|
nodes with member edges, nesting, supersession-with-residue, hot-path lookup
|
||||||
|
(`engram_geo_reify_store`, `engram_geo_reify_nest`, `engram_geo_reify_lookup`).
|
||||||
|
- The tiered paged store (buffer pool / LRU / WAL / checkpointer / pinning),
|
||||||
|
the RAM activation graph (base-level learning `access_ts[]`, WM slots,
|
||||||
|
`working_memory_weight` / `background_activation`), `StoreNode` / `StoreEdge`.
|
||||||
|
- `GeoMember` already separating relational salience (`centrality`) from
|
||||||
|
intrinsic `salience`.
|
||||||
|
|
||||||
|
**Designed, NOT built (this doc's deliverables):**
|
||||||
|
- `GeoGradient` and `think()` as the single entry point (§1, M5).
|
||||||
|
- `Prior` as a first-class stored, warp-carrying, calibrated node (§2, M1–M2).
|
||||||
|
- Salience/importance as a *relation* superseding the intrinsic node scalar
|
||||||
|
(§2.1, M3).
|
||||||
|
- `vantage_read(anchor, aperture)` unifying the three perspective settings
|
||||||
|
(§3, M4).
|
||||||
|
- **The reflexive correspondence-loop / `correspondence_beat`** — the learning
|
||||||
|
engine, moved from offline Python into the geometry (§4, M1). *The core new
|
||||||
|
subsystem.*
|
||||||
|
- `grounded-by` edge + assertion-only honesty floor (§5, M3).
|
||||||
|
- Node-level keystones + bounded plasticity (§6, M6).
|
||||||
|
|
||||||
|
**Uncertain / to resolve during build:**
|
||||||
|
- The exact warp parameterization (axis_gain vs full metric) — start minimal
|
||||||
|
(per-axis gain), measure, widen only if calibration demands it.
|
||||||
|
- Grade-(1) self-consistency as a sufficient reality signal for M1, versus
|
||||||
|
needing grade (2)/(3) sooner — decided empirically on the snapshot.
|
||||||
+577
-8
@@ -3240,6 +3240,37 @@ static void jb_init(JsonBuf* b) {
|
|||||||
b->buf[0] = '\0';
|
b->buf[0] = '\0';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* jb_init_cap — jb_init with a caller-supplied starting capacity.
|
||||||
|
*
|
||||||
|
* WHY THIS EXISTS (2026-08-11 self-review). jb_init starts at 64 BYTES and
|
||||||
|
* jb_reserve grows by doubling. That is right for the hundreds of small JSON
|
||||||
|
* responses this runtime builds per minute and catastrophic for the one that
|
||||||
|
* is 64 MEGABYTES: serializing the canonical snapshot walked the buffer
|
||||||
|
* 64B → 128B → ... → 128MB, about twenty reallocs, each copying everything
|
||||||
|
* written so far. Roughly 128MB of memcpy per save, and — the part that
|
||||||
|
* actually hurt — a fresh large span from the allocator every time.
|
||||||
|
*
|
||||||
|
* MEASURED (13,129 nodes / 43,400 edges, macOS arm64): RSS climbed +63MB per
|
||||||
|
* snapshot write, linearly, 14 for 14 writes, no plateau — 204MB to 1,028MB.
|
||||||
|
* `leaks` reported only 15KB genuinely unreachable, which is what makes this
|
||||||
|
* subtle: nothing is leaked in the reachable/unreachable sense. engram_save
|
||||||
|
* frees b.buf correctly on every path. The growth is the allocator declining
|
||||||
|
* to return large freed spans to the OS, and the doubling walk guaranteeing
|
||||||
|
* that each save asks for a differently-sized region than the last free made
|
||||||
|
* available. Every durable write path calls this — node create, edge create,
|
||||||
|
* the Hebbian batch write-back — so on the live daemon it grows without bound
|
||||||
|
* until the process dies.
|
||||||
|
*
|
||||||
|
* The fix is to ask for the right size once. With a stable capacity the
|
||||||
|
* allocator hands back the same span on every save and RSS flattens. */
|
||||||
|
static void jb_init_cap(JsonBuf* b, size_t cap) {
|
||||||
|
if (cap < 64) cap = 64;
|
||||||
|
b->cap = cap; b->len = 0;
|
||||||
|
b->buf = malloc(b->cap);
|
||||||
|
if (!b->buf) { fputs("el_runtime: out of memory\n", stderr); exit(1); }
|
||||||
|
b->buf[0] = '\0';
|
||||||
|
}
|
||||||
|
|
||||||
static void jb_reserve(JsonBuf* b, size_t add) {
|
static void jb_reserve(JsonBuf* b, size_t add) {
|
||||||
if (b->len + add + 1 > b->cap) {
|
if (b->len + add + 1 > b->cap) {
|
||||||
while (b->len + add + 1 > b->cap) b->cap *= 2;
|
while (b->len + add + 1 > b->cap) b->cap *= 2;
|
||||||
@@ -6516,6 +6547,26 @@ static float* _eg_ctx_c = NULL;
|
|||||||
static int32_t _eg_ctx_dim = 0;
|
static int32_t _eg_ctx_dim = 0;
|
||||||
static double _eg_act_ctx_cos = -2.0;
|
static double _eg_act_ctx_cos = -2.0;
|
||||||
|
|
||||||
|
/* Fan-effect gauges (2026-08-11 self-review). Per-call, like ctx_cos: they
|
||||||
|
* describe THIS activation, not process history. Without these the degree
|
||||||
|
* normalization is an unobservable change to the most important scoring path
|
||||||
|
* in the runtime, and "did it do anything" would be unanswerable — which is
|
||||||
|
* exactly the failure the Hebbian learning rate had before it was measured.
|
||||||
|
* fan_mean — mean applied factor over every propagation step. 1.0 means the
|
||||||
|
* correction never bound (graph is flat, or d_ref is above every
|
||||||
|
* pair's geometric mean degree). Falling toward FAN_MIN means
|
||||||
|
* traversal is running through hubs.
|
||||||
|
* fan_min_seen / fan_hits — the worst single penalty and how many steps were
|
||||||
|
* penalized at all, so a low mean caused by one pathological hub
|
||||||
|
* is distinguishable from broad hub saturation.
|
||||||
|
* fan_dref — the live mean degree the correction is calibrated against;
|
||||||
|
* publishing it makes densification visible over time. */
|
||||||
|
static double _eg_act_fan_sum = 0.0;
|
||||||
|
static double _eg_act_fan_min = 1.0;
|
||||||
|
static int64_t _eg_act_fan_n = 0;
|
||||||
|
static int64_t _eg_act_fan_hits = 0;
|
||||||
|
static double _eg_act_fan_dref = 0.0;
|
||||||
|
|
||||||
static int _eg_embed_consec_fail = 0;
|
static int _eg_embed_consec_fail = 0;
|
||||||
static int64_t _eg_embed_breaker_until = 0;
|
static int64_t _eg_embed_breaker_until = 0;
|
||||||
|
|
||||||
@@ -6534,6 +6585,27 @@ static int64_t _eg_embed_breaker_until = 0;
|
|||||||
* rates keep the previous reading and diff. Restart legitimately resets to 0. */
|
* rates keep the previous reading and diff. Restart legitimately resets to 0. */
|
||||||
static int64_t _eg_act_breakthroughs = 0; /* forced promotions at the floor, cumulative */
|
static int64_t _eg_act_breakthroughs = 0; /* forced promotions at the floor, cumulative */
|
||||||
static int64_t _eg_act_wm_evicted = 0; /* ALL WM evictions, cumulative (see below) */
|
static int64_t _eg_act_wm_evicted = 0; /* ALL WM evictions, cumulative (see below) */
|
||||||
|
/* ── Eviction CAUSE decomposition (2026-08-14 self-review) ──────────────────
|
||||||
|
* _eg_act_wm_evicted is incremented from six sites with four distinct causes,
|
||||||
|
* and every one of them collapsed into that single integer. Today's review
|
||||||
|
* measured 175,547 evictions over 13.5h (~216/min against 24 slots) and could
|
||||||
|
* not tell healthy rotation from cap thrashing from duplicate churn, because
|
||||||
|
* the only available number counts all three the same way.
|
||||||
|
*
|
||||||
|
* That is this file's most-repeated defect. The 08-02 and 08-06 reviews were
|
||||||
|
* each diagnosable only because someone first added a NEW gauge; dup_wm and
|
||||||
|
* dup_wm_global exist precisely because the aggregate could not answer "why".
|
||||||
|
* These three finish the decomposition, so that
|
||||||
|
* evicted == floor + cap + bll + dup_wm + dup_wm_global
|
||||||
|
* holds as an identity and each term names a different corrective action:
|
||||||
|
* floor - candidates below the absolute admission bar. High = weak retrieval.
|
||||||
|
* cap - lost the rank contest for 24 slots. High = genuine contention.
|
||||||
|
* bll - carried-over residents that decayed under the ACT-R tau. High =
|
||||||
|
* healthy forgetting, NOT pressure.
|
||||||
|
* Confusing the third with the second is what makes WM churn unreadable. */
|
||||||
|
static int64_t _eg_act_evict_floor = 0; /* below ENGRAM_WM_FLOOR (both passes) */
|
||||||
|
static int64_t _eg_act_evict_cap = 0; /* over ENGRAM_WM_CAP (both passes) */
|
||||||
|
static int64_t _eg_act_evict_bll = 0; /* carry-over decayed under BLL tau */
|
||||||
/* Redundancy suppression counters (2026-08-05 self-review) — see
|
/* Redundancy suppression counters (2026-08-05 self-review) — see
|
||||||
* ENGRAM_DEDUP_COS. dup_seeds = semantic seed slots reclaimed from redundant
|
* ENGRAM_DEDUP_COS. dup_seeds = semantic seed slots reclaimed from redundant
|
||||||
* copies; dup_wm = WM candidates dropped for duplicating a higher-ranked
|
* copies; dup_wm = WM candidates dropped for duplicating a higher-ranked
|
||||||
@@ -6874,6 +6946,10 @@ typedef struct EngramStore {
|
|||||||
int* adj_to_len;
|
int* adj_to_len;
|
||||||
int adj_dirty; /* 1 = rebuild needed before next BFS */
|
int adj_dirty; /* 1 = rebuild needed before next BFS */
|
||||||
int64_t adj_node_count; /* node_count at time of last adj_rebuild */
|
int64_t adj_node_count; /* node_count at time of last adj_rebuild */
|
||||||
|
/* Nodes with degree >= 1 at last adj_rebuild. The denominator for the
|
||||||
|
* fan-effect reference degree — see eg_fan_factor for why isolated nodes
|
||||||
|
* must not be counted. (2026-08-11 self-review) */
|
||||||
|
int64_t adj_connected;
|
||||||
} EngramStore;
|
} EngramStore;
|
||||||
|
|
||||||
static EngramStore* engram_global = NULL;
|
static EngramStore* engram_global = NULL;
|
||||||
@@ -7237,11 +7313,16 @@ static void engram_adj_rebuild(EngramStore* g) {
|
|||||||
if (ti >= 0 && g->adj_to[ti])
|
if (ti >= 0 && g->adj_to[ti])
|
||||||
g->adj_to[ti][to_pos[ti]++] = (int)ei;
|
g->adj_to[ti][to_pos[ti]++] = (int)ei;
|
||||||
}
|
}
|
||||||
/* Copy counts */
|
/* Copy counts. Also tally how many nodes have any edge at all — the
|
||||||
|
* fan-effect denominator. Free here, in the O(V) pass that already exists,
|
||||||
|
* rather than as a separate scan. (2026-08-11 self-review) */
|
||||||
|
int64_t connected = 0;
|
||||||
for (int64_t i = 0; i < g->node_count; i++) {
|
for (int64_t i = 0; i < g->node_count; i++) {
|
||||||
g->adj_from_len[i] = from_cnt[i];
|
g->adj_from_len[i] = from_cnt[i];
|
||||||
g->adj_to_len[i] = to_cnt[i];
|
g->adj_to_len[i] = to_cnt[i];
|
||||||
|
if (from_cnt[i] + to_cnt[i] > 0) connected++;
|
||||||
}
|
}
|
||||||
|
g->adj_connected = connected;
|
||||||
free(from_cnt); free(to_cnt); free(from_pos); free(to_pos);
|
free(from_cnt); free(to_cnt); free(from_pos); free(to_pos);
|
||||||
g->adj_node_count = g->node_count;
|
g->adj_node_count = g->node_count;
|
||||||
g->adj_dirty = 0;
|
g->adj_dirty = 0;
|
||||||
@@ -8496,6 +8577,7 @@ static void eg_wm_carry_over(EngramNode* cn, int64_t now_ms, int64_t* evict_ctr)
|
|||||||
cn->working_memory_weight = 0.0;
|
cn->working_memory_weight = 0.0;
|
||||||
cn->wm_anchor = 0.0;
|
cn->wm_anchor = 0.0;
|
||||||
if (evict_ctr) (*evict_ctr)++;
|
if (evict_ctr) (*evict_ctr)++;
|
||||||
|
_eg_act_evict_bll++;
|
||||||
} else {
|
} else {
|
||||||
cn->working_memory_weight = w;
|
cn->working_memory_weight = w;
|
||||||
}
|
}
|
||||||
@@ -8662,6 +8744,108 @@ static double engram_activation_dampen(const EngramNode* n) {
|
|||||||
return 1.0 / (1.0 + log(1.0 + (double)n->activation_count));
|
return 1.0 / (1.0 + log(1.0 + (double)n->activation_count));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ── ACT-R fan effect: degree normalization for spreading activation ─────────
|
||||||
|
* (2026-08-11 self-review. Closes the other half of a mechanism that has been
|
||||||
|
* half-implemented since the BLL work.)
|
||||||
|
*
|
||||||
|
* THE GAP. This runtime implements ACT-R's base-level learning term
|
||||||
|
* B_i = ln(Σ t_k^-d) (engram_bll_base_level) but never implemented the
|
||||||
|
* ASSOCIATIVE term that goes with it:
|
||||||
|
*
|
||||||
|
* A_i = B_i + Σ_j W_j · S_ji where S_ji = S − ln(fan_j)
|
||||||
|
*
|
||||||
|
* fan_j is the number of things j is associated with. The whole point of the
|
||||||
|
* fan effect (Anderson 1974; Anderson & Reder 1999) is that a source spreads a
|
||||||
|
* FIXED budget of activation across its associations — so being connected to
|
||||||
|
* many things makes each individual connection weaker. Without it, degree is
|
||||||
|
* pure advantage: a node wins retrieval by being popular rather than by being
|
||||||
|
* relevant. That is backwards, and it is what this graph has been doing.
|
||||||
|
*
|
||||||
|
* MEASURED ON THE LIVE STORE (13,129 nodes / 43,400 edges, 2026-08-11):
|
||||||
|
* degree p50=14 p90=34 p95=82 p99=275 max=357 mean=23.3
|
||||||
|
* the top 1% of nodes by degree touch 21.2% of all edges
|
||||||
|
* So the most-connected node had a 25x propagation advantage over the median
|
||||||
|
* node for no reason other than accumulated connections. The top hubs are not
|
||||||
|
* even semantically central — several are duplicate pairs of the same document
|
||||||
|
* left over from the redundancy census of the 2026-08-05 review.
|
||||||
|
*
|
||||||
|
* The hub problem was already recognized twice and patched narrowly both
|
||||||
|
* times: InternalStateEvent nodes were cut out of propagation entirely (see
|
||||||
|
* the frontier loop) and eg_hebb_node_budget caps per-node Hebbian mass. Both
|
||||||
|
* are special cases of this general law. This is the general fix.
|
||||||
|
*
|
||||||
|
* FORM. Symmetric normalization, w / (deg(u)^β · deg(v)^β) with β = 0.5 — the
|
||||||
|
* normalized-Laplacian / GCN form, which penalizes a hub both for sending and
|
||||||
|
* for receiving. Both failure modes are live here: a hub source floods its
|
||||||
|
* neighborhood, and a hub target gets reached by everything regardless of
|
||||||
|
* relevance. Written relative to the graph's own mean degree:
|
||||||
|
*
|
||||||
|
* fan(u,v) = clamp( d_ref / sqrt(deg(u) · deg(v)), FAN_MIN, 1.0 )
|
||||||
|
* d_ref = 2·|E| / |V| (mean degree, O(1), live)
|
||||||
|
*
|
||||||
|
* WHY IT IS CLAMPED AT 1.0 ON TOP — this is the load-bearing safety property,
|
||||||
|
* not a detail. The factor can only ever REDUCE propagation, never amplify it.
|
||||||
|
* Every constant downstream of this multiply is calibrated against today's
|
||||||
|
* activation magnitudes: the 0.02 firing threshold, SPREAD_DECAY = 0.7, the
|
||||||
|
* 0.15 WM promotion threshold, the 24-slot WM cap. A normalization that
|
||||||
|
* boosted low-degree nodes would inflate the frontier, change how many nodes
|
||||||
|
* clear 0.02, and silently recalibrate working memory as a side effect of a
|
||||||
|
* change that was supposed to be about hubs. Capping at 1.0 means every pair
|
||||||
|
* at or below mean degree — the common case — propagates EXACTLY as it does
|
||||||
|
* today, and the only behavior that changes is that above-mean hubs stop
|
||||||
|
* winning on degree alone. Strictly monotone, strictly conservative, and the
|
||||||
|
* blast radius is confined to the nodes the change is aimed at.
|
||||||
|
*
|
||||||
|
* Self-calibrating: d_ref is recomputed from the live graph, so the correction
|
||||||
|
* tracks densification instead of drifting against a constant that was right
|
||||||
|
* in August 2026 and wrong a year later. Change is the signal.
|
||||||
|
*
|
||||||
|
* FAN_MIN = 0.30 bottoms the penalty at ~3.3x rather than the ~15x that raw
|
||||||
|
* 1/deg would give at max degree. Same reasoning as ENGRAM_QGATE_FLOOR: damp
|
||||||
|
* the uninformative path, never sever it. A hub is usually a hub for a reason;
|
||||||
|
* it just should not also get a free win.
|
||||||
|
*
|
||||||
|
* Sources: Anderson & Reder 1999 (fan effect, S=1.6-2.0, d=0.5) ·
|
||||||
|
* arXiv:2405.14831 HippoRAG (node specificity) · Systems 9(2):22
|
||||||
|
* (normalized-Laplacian spreading activation) · arXiv:2606.30133 (β is a
|
||||||
|
* low-sensitivity knob; gating and fan normalization carry the effect). */
|
||||||
|
/* FAN_MIN 0.50, not the 0.30 this shipped as on the first build. Measured on
|
||||||
|
* the live graph, β=0.5 with a 0.30 floor damped 96% of propagation steps to a
|
||||||
|
* mean factor of 0.34 — and that number is not a bug in the correction, it is
|
||||||
|
* an honest measurement of how hub-dominated traversal here actually is. But a
|
||||||
|
* ~3x near-uniform damp is a bigger global change than one A/B run justifies,
|
||||||
|
* and it cost a working-memory promotion (5 → 4) on the one query measured
|
||||||
|
* cleanly. A 0.50 floor keeps the full mechanism and the whole [0.5, 1.0]
|
||||||
|
* dynamic range for separating hubs from non-hubs, at half the blast radius.
|
||||||
|
* The fan_mean / fan_hits gauges make the next review's tuning evidence-based
|
||||||
|
* rather than another guess: loosen it when the data says WM can afford it. */
|
||||||
|
#define ENGRAM_FAN_MIN 0.50
|
||||||
|
|
||||||
|
/* eg_node_degree — total (in + out) degree from the adjacency index. The index
|
||||||
|
* is rebuilt at the top of engram_activate whenever topology changed, so this
|
||||||
|
* is current. adj_node_count is the count at BUILD time and can lag
|
||||||
|
* node_count; out-of-range indices report 0 and are treated as unpenalized. */
|
||||||
|
static int eg_node_degree(const EngramStore* g, int64_t idx) {
|
||||||
|
if (idx < 0 || idx >= g->adj_node_count) return 0;
|
||||||
|
if (!g->adj_from_len || !g->adj_to_len) return 0;
|
||||||
|
return g->adj_from_len[idx] + g->adj_to_len[idx];
|
||||||
|
}
|
||||||
|
|
||||||
|
static double eg_fan_factor(const EngramStore* g, double d_ref,
|
||||||
|
int64_t u_idx, int64_t v_idx) {
|
||||||
|
if (d_ref <= 0.0) return 1.0;
|
||||||
|
int du = eg_node_degree(g, u_idx);
|
||||||
|
int dv = eg_node_degree(g, v_idx);
|
||||||
|
/* Degree 0 is only reachable when the adjacency index is stale or absent;
|
||||||
|
* an actually-isolated node is never on the frontier. Do not penalize what
|
||||||
|
* we cannot measure. */
|
||||||
|
if (du <= 0 || dv <= 0) return 1.0;
|
||||||
|
double f = d_ref / sqrt((double)du * (double)dv);
|
||||||
|
if (f > 1.0) return 1.0; /* never amplify — see above */
|
||||||
|
if (f < ENGRAM_FAN_MIN) return ENGRAM_FAN_MIN;
|
||||||
|
return f;
|
||||||
|
}
|
||||||
|
|
||||||
/* Temporal proximity bonus: boost propagation along edges connecting
|
/* Temporal proximity bonus: boost propagation along edges connecting
|
||||||
* co-temporal nodes. Returns a multiplier bonus in [0, 0.2]. */
|
* co-temporal nodes. Returns a multiplier bonus in [0, 0.2]. */
|
||||||
static double engram_temporal_proximity_bonus(int64_t node_created,
|
static double engram_temporal_proximity_bonus(int64_t node_created,
|
||||||
@@ -8829,6 +9013,8 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
|||||||
* miss nearly all events between beats; see the definition site).
|
* miss nearly all events between beats; see the definition site).
|
||||||
* ctx_cos stays per-call: it is a gauge of THIS query vs the centroid. */
|
* ctx_cos stays per-call: it is a gauge of THIS query vs the centroid. */
|
||||||
_eg_act_ctx_cos = -2.0;
|
_eg_act_ctx_cos = -2.0;
|
||||||
|
_eg_act_fan_sum = 0.0; _eg_act_fan_min = 1.0;
|
||||||
|
_eg_act_fan_n = 0; _eg_act_fan_hits = 0;
|
||||||
|
|
||||||
/* ── Embedding backfill + query embedding (2026-07-24, bl-b2d1c944) ──
|
/* ── Embedding backfill + query embedding (2026-07-24, bl-b2d1c944) ──
|
||||||
* Backfill: embed up to N un-embedded eligible nodes per call, newest
|
* Backfill: embed up to N un-embedded eligible nodes per call, newest
|
||||||
@@ -9061,6 +9247,29 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
|||||||
ftail++;
|
ftail++;
|
||||||
}
|
}
|
||||||
const double SPREAD_DECAY = 0.7;
|
const double SPREAD_DECAY = 0.7;
|
||||||
|
/* Reference degree for the fan-effect correction: mean degree over
|
||||||
|
* CONNECTED nodes, 2|E| / |{v : deg(v) > 0}|. O(1) — adj_connected is
|
||||||
|
* tallied during adjacency rebuild.
|
||||||
|
*
|
||||||
|
* NOT 2|E|/|V|. That was the first cut and instrumentation caught it
|
||||||
|
* immediately: on the live graph it gives d_ref = 6.61, while the median
|
||||||
|
* degree of a node that actually has edges is 14. Isolated nodes cannot
|
||||||
|
* be on the frontier — spreading activation only ever traverses connected
|
||||||
|
* ones — so including them in the denominator deflates the reference below
|
||||||
|
* anything traversal will ever see, and the correction pins to
|
||||||
|
* ENGRAM_FAN_MIN on every step. Measured on the first build:
|
||||||
|
* fan_mean 0.3026 with fan_hits 579/579 — a uniform 0.30 multiplier, which
|
||||||
|
* is not a fan effect at all. It is just a weaker SPREAD_DECAY, and it
|
||||||
|
* would have quietly recalibrated the 0.02 firing threshold and WM
|
||||||
|
* competition while appearing to be a targeted change.
|
||||||
|
*
|
||||||
|
* Over connected nodes the reference is ~23, above the median, so typical
|
||||||
|
* traversal rides the 1.0 cap unchanged and only genuine hubs are damped
|
||||||
|
* — which is the whole intent. The gauge that caught this is the reason it
|
||||||
|
* was worth adding the gauge. */
|
||||||
|
const double FAN_DREF = (g->adj_connected > 0)
|
||||||
|
? (2.0 * (double)g->edge_count / (double)g->adj_connected) : 0.0;
|
||||||
|
_eg_act_fan_dref = FAN_DREF;
|
||||||
while (fhead < ftail) {
|
while (fhead < ftail) {
|
||||||
Frontier f = fr[fhead++];
|
Frontier f = fr[fhead++];
|
||||||
if (f.hops >= max_depth) continue;
|
if (f.hops >= max_depth) continue;
|
||||||
@@ -9127,16 +9336,51 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
|||||||
* ~4x, never killed); unembedded targets pass ungated (no
|
* ~4x, never killed); unembedded targets pass ungated (no
|
||||||
* information, no penalty); cosq == NULL (embedder down) means
|
* information, no penalty); cosq == NULL (embedder down) means
|
||||||
* no gating at all — same graceful degradation as seeding. */
|
* no gating at all — same graceful degradation as seeding. */
|
||||||
|
/* Rescale before gating (2026-08-14 self-review). Raw cosine from
|
||||||
|
* nomic-embed is compressed into a narrow high band, so feeding it
|
||||||
|
* to the gate directly makes the gate nearly a constant. Measured
|
||||||
|
* on this store: 400 random UNRELATED node pairs gave median 0.562,
|
||||||
|
* central 98% span [0.381, 0.743]. The raw gate therefore passed a
|
||||||
|
* typical unrelated node at 0.25 + 0.75*0.562 = 0.67 — two thirds
|
||||||
|
* strength for a node with no semantic relation to the query. That
|
||||||
|
* is not a gate, it is a small tax.
|
||||||
|
*
|
||||||
|
* Shift-and-floor about ENGRAM_EMBED_S0, exactly as the Pass-2 WM
|
||||||
|
* term at ENGRAM_EMBED_WM_WEIGHT already does. The constant was in
|
||||||
|
* this file for this reason; the propagation gate simply never used
|
||||||
|
* it. Same store, same 400 pairs, after the rescale: the median
|
||||||
|
* unrelated pair drops to 0.40 while the top of the range is
|
||||||
|
* preserved (0.85 vs 0.92), and gate spread widens 0.42 -> 0.60.
|
||||||
|
* Only 8.5% of pairs fall to the floor, so lexical/structural
|
||||||
|
* pathways through dissimilar nodes are damped, never severed.
|
||||||
|
* Cf. arXiv:2512.15922, which rescales w' = (w-c)/(1-c) about
|
||||||
|
* c = 0.4 for precisely this reason ("prevent overactivation and
|
||||||
|
* context explosion"). */
|
||||||
double qgate = 1.0;
|
double qgate = 1.0;
|
||||||
if (cosq && cosq[oi] > -1.5) {
|
if (cosq && cosq[oi] > -1.5) {
|
||||||
double c = cosq[oi] > 0.0 ? cosq[oi] : 0.0;
|
double c = (cosq[oi] - ENGRAM_EMBED_S0) / (1.0 - ENGRAM_EMBED_S0);
|
||||||
|
if (c < 0.0) c = 0.0;
|
||||||
|
if (c > 1.0) c = 1.0;
|
||||||
qgate = ENGRAM_QGATE_FLOOR + (1.0 - ENGRAM_QGATE_FLOOR) * c;
|
qgate = ENGRAM_QGATE_FLOOR + (1.0 - ENGRAM_QGATE_FLOOR) * c;
|
||||||
}
|
}
|
||||||
|
/* ── ACT-R fan effect (2026-08-11 self-review) ──
|
||||||
|
* Symmetric degree normalization over the (source, target) pair.
|
||||||
|
* The query gate above prunes branches that are semantically
|
||||||
|
* irrelevant; this prunes branches that are merely POPULAR. They
|
||||||
|
* are different failure modes — a duplicate document with 357
|
||||||
|
* edges can be highly cosine-similar to the query and still be
|
||||||
|
* the wrong thing to spread through. Only ever <= 1.0, so it
|
||||||
|
* cannot inflate the frontier. See eg_fan_factor. */
|
||||||
|
double fan = eg_fan_factor(g, FAN_DREF, cur, oi);
|
||||||
|
_eg_act_fan_sum += fan;
|
||||||
|
_eg_act_fan_n++;
|
||||||
|
if (fan < 1.0) _eg_act_fan_hits++;
|
||||||
|
if (fan < _eg_act_fan_min) _eg_act_fan_min = fan;
|
||||||
/* eg_edge_eff_weight, not e->weight: edges that have repeatedly
|
/* eg_edge_eff_weight, not e->weight: edges that have repeatedly
|
||||||
* carried co-activated pairs propagate more strongly. Identity on
|
* carried co-activated pairs propagate more strongly. Identity on
|
||||||
* an unlearned edge. (2026-08-04 self-review.) */
|
* an unlearned edge. (2026-08-04 self-review.) */
|
||||||
double new_act = f.act * eg_edge_eff_weight(e) * SPREAD_DECAY
|
double new_act = f.act * eg_edge_eff_weight(e) * SPREAD_DECAY
|
||||||
* (1.0 + tbonus) * tdecay * dampen * qgate;
|
* (1.0 + tbonus) * tdecay * dampen * qgate * fan;
|
||||||
/* Firing threshold per classic spreading-activation: sub-threshold
|
/* Firing threshold per classic spreading-activation: sub-threshold
|
||||||
* activation neither updates the target nor enqueues it, so weak
|
* activation neither updates the target nor enqueues it, so weak
|
||||||
* signals die out instead of flooding the whole graph with tiny
|
* signals die out instead of flooding the whole graph with tiny
|
||||||
@@ -9426,6 +9670,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
|||||||
if (wm_weights[i] > 0.0 && wm_weights[i] < ENGRAM_WM_FLOOR) {
|
if (wm_weights[i] > 0.0 && wm_weights[i] < ENGRAM_WM_FLOOR) {
|
||||||
wm_weights[i] = 0.0;
|
wm_weights[i] = 0.0;
|
||||||
_eg_act_wm_evicted++;
|
_eg_act_wm_evicted++;
|
||||||
|
_eg_act_evict_floor++;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
int64_t cap_count = 0;
|
int64_t cap_count = 0;
|
||||||
@@ -9461,6 +9706,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
|||||||
}
|
}
|
||||||
wm_weights[i] = 0.0; /* over cap: evict */
|
wm_weights[i] = 0.0; /* over cap: evict */
|
||||||
_eg_act_wm_evicted++;
|
_eg_act_wm_evicted++;
|
||||||
|
_eg_act_evict_cap++;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/* If malloc failed, skip cap — WM unbounded this call, no corruption. */
|
/* If malloc failed, skip cap — WM unbounded this call, no corruption. */
|
||||||
@@ -9572,6 +9818,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
|||||||
fn->working_memory_weight = 0.0;
|
fn->working_memory_weight = 0.0;
|
||||||
fn->wm_anchor = 0.0;
|
fn->wm_anchor = 0.0;
|
||||||
_eg_act_wm_evicted++;
|
_eg_act_wm_evicted++;
|
||||||
|
_eg_act_evict_floor++;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/* ── Global redundancy suppression (2026-08-06 self-review) ──────────
|
/* ── Global redundancy suppression (2026-08-06 self-review) ──────────
|
||||||
@@ -9680,6 +9927,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
|||||||
n->working_memory_weight = 0.0; /* evict: over global cap */
|
n->working_memory_weight = 0.0; /* evict: over global cap */
|
||||||
n->wm_anchor = 0.0; /* keep anchor coherent */
|
n->wm_anchor = 0.0; /* keep anchor coherent */
|
||||||
_eg_act_wm_evicted++; /* was uncounted before 2026-08-02 */
|
_eg_act_wm_evicted++; /* was uncounted before 2026-08-02 */
|
||||||
|
_eg_act_evict_cap++;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/* If malloc failed, skip — WM over cap this call, no data corruption. */
|
/* If malloc failed, skip — WM over cap this call, no data corruption. */
|
||||||
@@ -10109,11 +10357,24 @@ static void engram_emit_edge_json(JsonBuf* b, const EngramEdge* e) {
|
|||||||
jb_putc(b, '}');
|
jb_putc(b, '}');
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Size of the last snapshot this process serialized. Seeds the next save's
|
||||||
|
* buffer so the doubling walk never runs on the big document. See jb_init_cap
|
||||||
|
* for the measurement that motivated it. (2026-08-11 self-review) */
|
||||||
|
static size_t _eg_save_cap_hint = 0;
|
||||||
|
|
||||||
el_val_t engram_save(el_val_t path) {
|
el_val_t engram_save(el_val_t path) {
|
||||||
const char* p = EL_CSTR(path);
|
const char* p = EL_CSTR(path);
|
||||||
if (!p || !*p) return 0;
|
if (!p || !*p) return 0;
|
||||||
EngramStore* g = engram_get();
|
EngramStore* g = engram_get();
|
||||||
JsonBuf b; jb_init(&b);
|
/* Pre-size from the previous save plus 12.5% headroom, so ordinary growth
|
||||||
|
* between snapshots does not trigger a realloc and the request size stays
|
||||||
|
* stable enough for the allocator to reuse the same span. First save of
|
||||||
|
* the process has no hint and starts at 1MB — still 14 doublings better
|
||||||
|
* than 64 bytes. */
|
||||||
|
JsonBuf b;
|
||||||
|
jb_init_cap(&b, _eg_save_cap_hint
|
||||||
|
? _eg_save_cap_hint + (_eg_save_cap_hint >> 3) + 1024
|
||||||
|
: (size_t)1 << 20);
|
||||||
jb_puts(&b, "{\"nodes\":[");
|
jb_puts(&b, "{\"nodes\":[");
|
||||||
for (int64_t i = 0; i < g->node_count; i++) {
|
for (int64_t i = 0; i < g->node_count; i++) {
|
||||||
if (i > 0) jb_putc(&b, ',');
|
if (i > 0) jb_putc(&b, ',');
|
||||||
@@ -10153,6 +10414,10 @@ el_val_t engram_save(el_val_t path) {
|
|||||||
jb_putc(&b, '}');
|
jb_putc(&b, '}');
|
||||||
}
|
}
|
||||||
jb_puts(&b, "]}");
|
jb_puts(&b, "]}");
|
||||||
|
/* Remember the size BEFORE the write: the hint is about how much buffer
|
||||||
|
* the next serialization needs, which is a property of the graph, not of
|
||||||
|
* whether this particular fopen succeeded. */
|
||||||
|
_eg_save_cap_hint = b.len;
|
||||||
FILE* f = fopen(p, "wb");
|
FILE* f = fopen(p, "wb");
|
||||||
if (!f) { free(b.buf); return 0; }
|
if (!f) { free(b.buf); return 0; }
|
||||||
size_t w = fwrite(b.buf, 1, b.len, f);
|
size_t w = fwrite(b.buf, 1, b.len, f);
|
||||||
@@ -11727,8 +11992,11 @@ el_val_t engram_act_stats_json(void) {
|
|||||||
}
|
}
|
||||||
/* 768, not 512: the write-back gauges added 2026-08-07 push the worst-case
|
/* 768, not 512: the write-back gauges added 2026-08-07 push the worst-case
|
||||||
* rendering past the old bound, and snprintf would truncate the JSON into
|
* rendering past the old bound, and snprintf would truncate the JSON into
|
||||||
* an unparseable tail rather than fail loudly. */
|
* an unparseable tail rather than fail loudly.
|
||||||
char buf[896];
|
* 1152, not 896: the five fan-effect gauges added 2026-08-11 add ~90 bytes
|
||||||
|
* worst-case. Same reasoning — headroom is cheaper than a truncated tail
|
||||||
|
* that every downstream JSON parser rejects as a whole. */
|
||||||
|
char buf[1152];
|
||||||
/* ctx_cos (2026-07-29): cos(query, context centroid) at the LAST
|
/* ctx_cos (2026-07-29): cos(query, context centroid) at the LAST
|
||||||
* activate call, measured before the query was folded in. ~1.0 =
|
* activate call, measured before the query was folded in. ~1.0 =
|
||||||
* context aligned with current query; low = divergence (expected at
|
* context aligned with current query; low = divergence (expected at
|
||||||
@@ -11736,6 +12004,15 @@ el_val_t engram_act_stats_json(void) {
|
|||||||
* embedder down. The drift gauge for the context-centroid mechanism. */
|
* embedder down. The drift gauge for the context-centroid mechanism. */
|
||||||
snprintf(buf, sizeof(buf),
|
snprintf(buf, sizeof(buf),
|
||||||
"{\"wm_evicted\":%lld,\"breakthroughs\":%lld,"
|
"{\"wm_evicted\":%lld,\"breakthroughs\":%lld,"
|
||||||
|
/* Eviction cause decomposition (2026-08-14 self-review):
|
||||||
|
* wm_evicted == evict_floor + evict_cap + evict_bll
|
||||||
|
* + dup_wm + dup_wm_global.
|
||||||
|
* Read them as a ratio, not a level. cap-dominant = real
|
||||||
|
* contention for the 24 slots; bll-dominant = healthy decay of
|
||||||
|
* carried-over residents; floor-dominant = retrieval is returning
|
||||||
|
* weak candidates. The aggregate alone cannot distinguish these
|
||||||
|
* and every prior WM incident needed a new gauge to diagnose. */
|
||||||
|
"\"evict_floor\":%lld,\"evict_cap\":%lld,\"evict_bll\":%lld,"
|
||||||
"\"embed_breaker_open\":%d,\"embed_consec_fail\":%d,"
|
"\"embed_breaker_open\":%d,\"embed_consec_fail\":%d,"
|
||||||
"\"ctx_cos\":%.3f,"
|
"\"ctx_cos\":%.3f,"
|
||||||
"\"hebb_edges\":%lld,\"hebb_max\":%.4f,\"hebb_mass\":%.3f,"
|
"\"hebb_edges\":%lld,\"hebb_max\":%.4f,\"hebb_mass\":%.3f,"
|
||||||
@@ -11749,9 +12026,19 @@ el_val_t engram_act_stats_json(void) {
|
|||||||
* any climb means a write path is mangling text again. Cheap
|
* any climb means a write path is mangling text again. Cheap
|
||||||
* (counted at creation) — the full census lives in
|
* (counted at creation) — the full census lives in
|
||||||
* engram_text_health_json. (2026-08-08 self-review) */
|
* engram_text_health_json. (2026-08-08 self-review) */
|
||||||
"\"txt_damaged\":%lld}",
|
"\"txt_damaged\":%lld,"
|
||||||
|
/* Fan-effect gauges (2026-08-11 self-review) — see the
|
||||||
|
* _eg_act_fan_* definitions. fan_mean == 1.0 with fan_hits == 0
|
||||||
|
* means the degree correction never bound on the last activation;
|
||||||
|
* a mean drifting toward ENGRAM_FAN_MIN means traversal is
|
||||||
|
* running through hubs and the correction is doing work. */
|
||||||
|
"\"fan_mean\":%.4f,\"fan_min\":%.4f,\"fan_hits\":%lld,"
|
||||||
|
"\"fan_steps\":%lld,\"fan_dref\":%.2f}",
|
||||||
(long long)_eg_act_wm_evicted,
|
(long long)_eg_act_wm_evicted,
|
||||||
(long long)_eg_act_breakthroughs,
|
(long long)_eg_act_breakthroughs,
|
||||||
|
(long long)_eg_act_evict_floor,
|
||||||
|
(long long)_eg_act_evict_cap,
|
||||||
|
(long long)_eg_act_evict_bll,
|
||||||
breaker_open, _eg_embed_consec_fail,
|
breaker_open, _eg_embed_consec_fail,
|
||||||
_eg_act_ctx_cos,
|
_eg_act_ctx_cos,
|
||||||
(long long)hebb_edges, hebb_max, hebb_mass,
|
(long long)hebb_edges, hebb_max, hebb_mass,
|
||||||
@@ -11761,7 +12048,10 @@ el_val_t engram_act_stats_json(void) {
|
|||||||
(long long)_eg_hebb_wb_dropped,
|
(long long)_eg_hebb_wb_dropped,
|
||||||
(long long)_eg_act_dup_seeds, (long long)_eg_act_dup_wm,
|
(long long)_eg_act_dup_seeds, (long long)_eg_act_dup_wm,
|
||||||
(long long)_eg_act_dup_wm_global,
|
(long long)_eg_act_dup_wm_global,
|
||||||
(long long)_eg_txt_write_damaged);
|
(long long)_eg_txt_write_damaged,
|
||||||
|
(_eg_act_fan_n > 0 ? _eg_act_fan_sum / (double)_eg_act_fan_n : 1.0),
|
||||||
|
_eg_act_fan_min, (long long)_eg_act_fan_hits,
|
||||||
|
(long long)_eg_act_fan_n, _eg_act_fan_dref);
|
||||||
return el_wrap_str(el_strdup(buf));
|
return el_wrap_str(el_strdup(buf));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -11894,6 +12184,285 @@ el_val_t engram_label_df(el_val_t term) {
|
|||||||
return (el_val_t)df;
|
return (el_val_t)df;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ── Salient-term extraction (2026-08-13 self-review) ────────────────────────
|
||||||
|
* THE MEASUREMENT. auto_term_empty_streak, the counter added by the 2026-08-06
|
||||||
|
* review precisely to catch this class of silent death, read 50 and climbing.
|
||||||
|
* Fifty consecutive curiosity scans in which the soul's dynamic seeding path
|
||||||
|
* produced NOTHING and the loop fell back to its four hardcoded rotating
|
||||||
|
* phrases. Dumping the live WM top says why in one look:
|
||||||
|
*
|
||||||
|
* Memory 0.390 memory:remembered
|
||||||
|
* Memory 0.378 memory:remembered
|
||||||
|
* Memory 0.377 memory:remembered
|
||||||
|
* Memory 0.373 memory:remembered
|
||||||
|
* Memory 0.370 memory:remembered
|
||||||
|
*
|
||||||
|
* Every slot at the top of working memory is a Memory node, and every Memory
|
||||||
|
* node written by remember() carries the sentinel label "memory:remembered".
|
||||||
|
* auto_term_try_slot reads the LABEL and only the label; the colon-no-space
|
||||||
|
* guard (correctly) rejects sentinels as carrying no seed signal; so the
|
||||||
|
* extractor had nothing to work with and returned empty, forever.
|
||||||
|
*
|
||||||
|
* THE ACTUAL DEFECT is not the sentinel guard — that guard is right. It is
|
||||||
|
* that the extractor was built against Knowledge nodes, which have real
|
||||||
|
* titles, and is structurally blind to the node type that in fact dominates
|
||||||
|
* working memory. The label is not the content. A Memory node's topic is in
|
||||||
|
* its text; the runtime just never looked there.
|
||||||
|
*
|
||||||
|
* WHY NOT ANOTHER GUARD. The extractor's whole history is guards: genre words
|
||||||
|
* (07-23), quoted titles (07-25), English stopwords (07-30), label-df
|
||||||
|
* (08-03). Four reviews, four blocklists, each written after watching a flood
|
||||||
|
* happen. That is a losing shape, and 08-03 said so explicitly before adding
|
||||||
|
* the fifth. The reason it keeps recurring is the algorithm underneath:
|
||||||
|
* TAKE THE FIRST WORD, THEN CHECK WHETHER IT IS ACCEPTABLE. A first-word
|
||||||
|
* extractor has no notion of term quality, so quality has to be bolted on as
|
||||||
|
* rejection, and rejection can only encode the past.
|
||||||
|
*
|
||||||
|
* THE FIX is to invert it: score EVERY candidate token in the text and take
|
||||||
|
* the argmax. Then term quality is the selection criterion rather than a
|
||||||
|
* veto, and a bad token does not need to be on a list to lose — it only needs
|
||||||
|
* a better token in the same text, which is the common case.
|
||||||
|
*
|
||||||
|
* SCORING (YAKE, Campos et al., Information Sciences 509:257-289, 2020 —
|
||||||
|
* lightweight unsupervised single-document keyword extraction). YAKE scores
|
||||||
|
* candidates on casing, position, frequency, context relatedness and sentence
|
||||||
|
* dispersion, and beats RAKE/TextRank/SingleRank across twenty datasets. Two
|
||||||
|
* of its five features port directly and cheaply; the other three are
|
||||||
|
* within-document proxies for a corpus YAKE deliberately does not have. This
|
||||||
|
* system DOES have the corpus — 12.7k labelled nodes — so real IDF is
|
||||||
|
* substituted where YAKE has to approximate:
|
||||||
|
*
|
||||||
|
* score(t) = idf(t) · position(t) · casing(t)
|
||||||
|
*
|
||||||
|
* idf = ln((N+1)/(df+1)) real corpus specificity (Spärck
|
||||||
|
* Jones 1972), strictly better than
|
||||||
|
* YAKE's TF-based stand-in
|
||||||
|
* position = 1/ln(e + i) YAKE T_Position: earlier tokens are
|
||||||
|
* more topical. Keeps the old
|
||||||
|
* first-word bias as a SOFT preference
|
||||||
|
* instead of an absolute rule
|
||||||
|
* casing = 1.30 acronym / 1.15 capitalised / 1.00 otherwise
|
||||||
|
* YAKE T_Case
|
||||||
|
*
|
||||||
|
* THE min_df GATE. The df ceiling (08-03) rejects corpus-frequent markup and
|
||||||
|
* sentinels. A floor was added alongside it for an independent reason: a term
|
||||||
|
* appearing in ZERO labels cannot lexically reach anything, so it is a bad
|
||||||
|
* seed however specific it looks.
|
||||||
|
*
|
||||||
|
* An earlier draft of this comment claimed the floor also subsumes the 73
|
||||||
|
* hand-listed stopwords that 08-03 measured label-df as missing (Whose:0,
|
||||||
|
* Would:0, Could:0). MEASURED, AND THAT CLAIM IS FALSE. Under word-boundary
|
||||||
|
* df on the live store, function words are rare in labels but not absent:
|
||||||
|
* about:2, whole:1, them:2, head:2. They clear a floor of 1. What actually
|
||||||
|
* keeps them from winning is the argmax itself — they carry no position
|
||||||
|
* advantage and lose to a topical term in the same text on every node
|
||||||
|
* measured. The stopword list therefore STAYS as a real defense for the
|
||||||
|
* Title-case cases, not as vestigial belt-and-braces. Recording the
|
||||||
|
* correction rather than the tidier story: the floor buys lexical
|
||||||
|
* reachability, the argmax buys quality, and the list still earns its keep.
|
||||||
|
*
|
||||||
|
* TABU IS APPLIED DURING THE ARGMAX, not after it. The old code picked a term
|
||||||
|
* and then discarded it if it was tabu, which turned inhibition-of-return
|
||||||
|
* into another source of empty scans. Excluding tabu terms from the candidate
|
||||||
|
* set instead yields the best NON-TABU term, so rotation costs quality rather
|
||||||
|
* than costing the whole scan.
|
||||||
|
*
|
||||||
|
* COST. One pass over g->nodes scoring all candidates at once (12.7k labels ×
|
||||||
|
* <=32 candidates, short strings, good locality), twice per 30 s scan.
|
||||||
|
*
|
||||||
|
* POLICY LIVES IN THE SOUL. Thresholds arrive as arguments; the runtime
|
||||||
|
* measures and ranks, awareness.el decides. Same split as engram_label_df.
|
||||||
|
*
|
||||||
|
* Returns the winning token, or "" when the node is missing, has no usable
|
||||||
|
* text, or every candidate is gated out — "" remains the honest signal that
|
||||||
|
* this slot yielded no seed, and auto_term_empty_streak still counts it. */
|
||||||
|
#define ENGRAM_ST_MAXCAND 32
|
||||||
|
#define ENGRAM_ST_TOKLEN 64
|
||||||
|
#define ENGRAM_ST_SCANCHARS 400
|
||||||
|
|
||||||
|
/* Trim leading/trailing non-alphanumerics, then accept only tokens whose core
|
||||||
|
* is alphanumeric plus '-' and '_' with at least 3 letters. This subsumes the
|
||||||
|
* quoted-title guard (2026-07-25) and the "<!--" flood (2026-08-03)
|
||||||
|
* structurally: markup and punctuation-bearing tokens never become
|
||||||
|
* candidates, rather than being blocklisted after the fact. */
|
||||||
|
static int eg_st_clean_token(const char* raw, size_t rawlen,
|
||||||
|
char* out, size_t outcap) {
|
||||||
|
size_t s = 0, e = rawlen;
|
||||||
|
while (s < e && !isalnum((unsigned char)raw[s])) s++;
|
||||||
|
while (e > s && !isalnum((unsigned char)raw[e - 1])) e--;
|
||||||
|
size_t len = e - s;
|
||||||
|
if (len < 4 || len >= outcap) return 0;
|
||||||
|
int alpha = 0;
|
||||||
|
for (size_t i = 0; i < len; i++) {
|
||||||
|
unsigned char c = (unsigned char)raw[s + i];
|
||||||
|
if (isalpha(c)) alpha++;
|
||||||
|
else if (!isdigit(c) && c != '-' && c != '_') return 0;
|
||||||
|
}
|
||||||
|
if (alpha < 3) return 0;
|
||||||
|
memcpy(out, raw + s, len);
|
||||||
|
out[len] = '\0';
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ENGRAM_ST_DEBUG=1 dumps the full scored candidate set to stderr. One
|
||||||
|
* cached branch in production. This exists because the first live run of this
|
||||||
|
* function returned five ALL-CAPS terms in a row and there was no way to see
|
||||||
|
* whether that was the corpus or the casing weight without guessing — the
|
||||||
|
* lesson this system keeps relearning. */
|
||||||
|
static int _eg_st_debug(void) {
|
||||||
|
static int v = -1;
|
||||||
|
if (v < 0) { const char* e = getenv("ENGRAM_ST_DEBUG"); v = (e && *e == '1'); }
|
||||||
|
return v;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Word-boundary document frequency. engram_label_df uses istr_contains, i.e.
|
||||||
|
* SUBSTRING matching, and that is the wrong estimator for term specificity on
|
||||||
|
* short tokens: "them" hits inside "theme" and "anthem", "about" and "whole"
|
||||||
|
* come back with df 2 and 1 rather than 0. That matters here specifically
|
||||||
|
* because the min_df floor is what rejects English function words, and it can
|
||||||
|
* only do that job if their df is honestly zero. Substring df quietly handed
|
||||||
|
* them a survival ticket. Measured on the live store before this fix, "whole"
|
||||||
|
* (df=1, idf=8.76) and "about" (df=2, idf=8.36) were outscoring real topical
|
||||||
|
* terms and losing only on position — one node whose text happened to open
|
||||||
|
* with a function word would have seeded on it.
|
||||||
|
*
|
||||||
|
* engram_label_df keeps substring semantics: it is a separate published
|
||||||
|
* measure with existing callers, and changing it underneath them is not this
|
||||||
|
* change's business. */
|
||||||
|
static int eg_st_label_has_word(const char* hay, const char* word) {
|
||||||
|
size_t wl = strlen(word);
|
||||||
|
for (const char* p = hay; *p; p++) {
|
||||||
|
if (strncasecmp(p, word, wl) != 0) continue;
|
||||||
|
char before = (p == hay) ? '\0' : p[-1];
|
||||||
|
char after = p[wl];
|
||||||
|
if (before && (isalnum((unsigned char)before) || before == '_')) continue;
|
||||||
|
if (after && (isalnum((unsigned char)after) || after == '_')) continue;
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* YAKE T_Case, adapted to this corpus. YAKE up-weights all-caps tokens
|
||||||
|
* because in ordinary prose an acronym is rare and carries topic. That
|
||||||
|
* assumption does not hold here: memory content written by remember()
|
||||||
|
* conventionally OPENS WITH AN ALL-CAPS HEADER ("FRAME-ROUTER UPGRADE —
|
||||||
|
* RESULTS", "THE GAP", "CENSUS"), so a flat acronym bonus systematically
|
||||||
|
* hands the seed to whatever word the header happens to start with and lets
|
||||||
|
* casing override the specificity signal it is supposed to only nudge.
|
||||||
|
* Measured on the live store: the first five WM nodes returned PRIMING,
|
||||||
|
* CONVERSATION, OCCUPATION, RELATIONAL, SELF-OCCUPATION — every one an
|
||||||
|
* all-caps header word, none chosen on its merits.
|
||||||
|
*
|
||||||
|
* Genuine acronyms are SHORT (VBD, CCR, MCP, HTTP); shouty headers are long
|
||||||
|
* words that happen to be capitalised. So the acronym bonus is restricted to
|
||||||
|
* tokens of <= 5 characters, where all-caps is actually evidence of an
|
||||||
|
* acronym rather than evidence of a heading. Longer all-caps tokens fall
|
||||||
|
* through to the ordinary Title-case nudge — they still compete, they just
|
||||||
|
* compete on specificity instead of on volume. */
|
||||||
|
static double eg_st_casing(const char* t) {
|
||||||
|
int upper = 0, lower = 0;
|
||||||
|
size_t len = 0;
|
||||||
|
for (const char* q = t; *q; q++, len++) {
|
||||||
|
if (isupper((unsigned char)*q)) upper++;
|
||||||
|
else if (islower((unsigned char)*q)) lower++;
|
||||||
|
}
|
||||||
|
if (lower == 0 && upper >= 2 && len <= 5) return 1.30; /* acronym */
|
||||||
|
if (isupper((unsigned char)t[0])) return 1.15; /* Title/hdr */
|
||||||
|
return 1.0;
|
||||||
|
}
|
||||||
|
|
||||||
|
el_val_t engram_salient_term(el_val_t node_id, el_val_t max_df_v,
|
||||||
|
el_val_t min_df_v, el_val_t tabu_v) {
|
||||||
|
EngramStore* g = engram_get();
|
||||||
|
int64_t ix = engram_find_node_index(EL_CSTR(node_id));
|
||||||
|
if (ix < 0) return el_wrap_str(el_strdup(""));
|
||||||
|
EngramNode* n = &g->nodes[ix];
|
||||||
|
|
||||||
|
int64_t max_df = (int64_t)max_df_v;
|
||||||
|
int64_t min_df = (int64_t)min_df_v;
|
||||||
|
if (max_df <= 0) max_df = g->node_count;
|
||||||
|
if (min_df < 0) min_df = 0;
|
||||||
|
const char* tabu = EL_CSTR(tabu_v);
|
||||||
|
|
||||||
|
/* Source selection. Prefer the label — it is a curated title when it is
|
||||||
|
* one. Fall back to content when the label is absent or a sentinel
|
||||||
|
* ("memory:remembered": a colon and no space). This single line is what
|
||||||
|
* makes Memory nodes visible to the extractor at all. */
|
||||||
|
const char* src = n->label;
|
||||||
|
if (!src || !*src) {
|
||||||
|
src = n->content;
|
||||||
|
} else if (strchr(src, ':') != NULL && strchr(src, ' ') == NULL) {
|
||||||
|
src = n->content;
|
||||||
|
}
|
||||||
|
if (!src || !*src) return el_wrap_str(el_strdup(""));
|
||||||
|
|
||||||
|
/* Collect distinct candidates from the head of the text. */
|
||||||
|
char cand[ENGRAM_ST_MAXCAND][ENGRAM_ST_TOKLEN];
|
||||||
|
int pos[ENGRAM_ST_MAXCAND];
|
||||||
|
int64_t df[ENGRAM_ST_MAXCAND];
|
||||||
|
int ncand = 0, tokidx = 0;
|
||||||
|
|
||||||
|
const char* p = src;
|
||||||
|
const char* lim = src + strnlen(src, ENGRAM_ST_SCANCHARS);
|
||||||
|
while (p < lim && ncand < ENGRAM_ST_MAXCAND) {
|
||||||
|
while (p < lim && isspace((unsigned char)*p)) p++;
|
||||||
|
if (p >= lim) break;
|
||||||
|
const char* tk = p;
|
||||||
|
while (p < lim && !isspace((unsigned char)*p)) p++;
|
||||||
|
char buf[ENGRAM_ST_TOKLEN];
|
||||||
|
int slot = tokidx++;
|
||||||
|
if (!eg_st_clean_token(tk, (size_t)(p - tk), buf, sizeof(buf))) continue;
|
||||||
|
|
||||||
|
/* Tabu exclusion, applied here so the argmax runs over eligible
|
||||||
|
* terms only. tabu arrives pipe-delimited: "|t0|t1|t2|t3|". */
|
||||||
|
if (tabu && *tabu) {
|
||||||
|
char pat[ENGRAM_ST_TOKLEN + 2];
|
||||||
|
snprintf(pat, sizeof(pat), "|%s|", buf);
|
||||||
|
if (istr_contains(tabu, pat)) continue;
|
||||||
|
}
|
||||||
|
int dup = 0;
|
||||||
|
for (int i = 0; i < ncand; i++)
|
||||||
|
if (strcasecmp(cand[i], buf) == 0) { dup = 1; break; }
|
||||||
|
if (dup) continue;
|
||||||
|
|
||||||
|
memcpy(cand[ncand], buf, strlen(buf) + 1);
|
||||||
|
pos[ncand] = slot;
|
||||||
|
df[ncand] = 0;
|
||||||
|
ncand++;
|
||||||
|
}
|
||||||
|
if (ncand == 0) return el_wrap_str(el_strdup(""));
|
||||||
|
|
||||||
|
/* One pass over the store, all candidates at once. */
|
||||||
|
for (int64_t i = 0; i < g->node_count; i++) {
|
||||||
|
const char* lbl = g->nodes[i].label;
|
||||||
|
if (!lbl || !*lbl) continue;
|
||||||
|
for (int c = 0; c < ncand; c++)
|
||||||
|
if (eg_st_label_has_word(lbl, cand[c])) df[c]++;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Argmax over idf · position · casing, subject to the df band. */
|
||||||
|
int best = -1;
|
||||||
|
double best_score = 0.0;
|
||||||
|
for (int c = 0; c < ncand; c++) {
|
||||||
|
if (df[c] > max_df) continue;
|
||||||
|
if (df[c] < min_df) continue;
|
||||||
|
double idf = log(((double)g->node_count + 1.0) / ((double)df[c] + 1.0));
|
||||||
|
if (idf <= 0.0) continue;
|
||||||
|
double position = 1.0 / log(2.718281828459045 + (double)pos[c]);
|
||||||
|
double casing = eg_st_casing(cand[c]);
|
||||||
|
double score = idf * position * casing;
|
||||||
|
if (_eg_st_debug()) {
|
||||||
|
fprintf(stderr, " cand %-24s df=%-5lld idf=%.2f pos=%d p=%.2f "
|
||||||
|
"case=%.2f score=%.3f\n",
|
||||||
|
cand[c], (long long)df[c], idf, pos[c], position,
|
||||||
|
casing, score);
|
||||||
|
}
|
||||||
|
if (score > best_score) { best_score = score; best = c; }
|
||||||
|
}
|
||||||
|
if (best < 0) return el_wrap_str(el_strdup(""));
|
||||||
|
return el_wrap_str(el_strdup(cand[best]));
|
||||||
|
}
|
||||||
|
|
||||||
/* engram_embed_backfill — explicitly drive the lazy embedding backfill.
|
/* engram_embed_backfill — explicitly drive the lazy embedding backfill.
|
||||||
* (2026-07-25 self-review.) The per-activate backfill (8 nodes/call) only
|
* (2026-07-25 self-review.) The per-activate backfill (8 nodes/call) only
|
||||||
* runs inside engram_activate, and on the authoritative HTTP store nothing
|
* runs inside engram_activate, and on the authoritative HTTP store nothing
|
||||||
|
|||||||
@@ -635,6 +635,13 @@ el_val_t engram_hebb_drain_json(el_val_t max);
|
|||||||
/* Document frequency of a term across node labels — term-specificity signal
|
/* Document frequency of a term across node labels — term-specificity signal
|
||||||
* for curiosity seed selection. (2026-08-03 self-review.) */
|
* for curiosity seed selection. (2026-08-03 self-review.) */
|
||||||
el_val_t engram_label_df(el_val_t term);
|
el_val_t engram_label_df(el_val_t term);
|
||||||
|
/* Best curiosity seed from one node: argmax over idf·position·casing across
|
||||||
|
* the candidate tokens of its label, falling back to its content when the
|
||||||
|
* label is a sentinel. Excludes pipe-delimited tabu terms during selection
|
||||||
|
* and gates candidates to the df band [min_df, max_df]. Returns "" when
|
||||||
|
* nothing qualifies. (2026-08-13 self-review.) */
|
||||||
|
el_val_t engram_salient_term(el_val_t node_id, el_val_t max_df,
|
||||||
|
el_val_t min_df, el_val_t tabu);
|
||||||
el_val_t engram_embed_backfill(el_val_t count);
|
el_val_t engram_embed_backfill(el_val_t count);
|
||||||
el_val_t engram_list_layers_json(void);
|
el_val_t engram_list_layers_json(void);
|
||||||
/* Working memory introspection — count, mean weight, and top-N snapshot.
|
/* Working memory introspection — count, mean weight, and top-N snapshot.
|
||||||
|
|||||||
@@ -0,0 +1,64 @@
|
|||||||
|
# nsbx — Neuron dev-environment Makefile
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Thin, documented wrappers over the `nsbx` primitive so a newcomer never has to
|
||||||
|
# memorise the elc/cc build incantation or the sandbox lifecycle. Every target is
|
||||||
|
# a one-liner over `nsbx`; nothing here reimplements engram logic.
|
||||||
|
#
|
||||||
|
# make dev NAME=tim # new branch + worktree + isolated engram, one shot
|
||||||
|
# make status NAME=tim # inspect it (omit NAME to list all sandboxes)
|
||||||
|
# make run NAME=tim # poke its API (API=/api/stats by default)
|
||||||
|
# make test NAME=tim # run the safety rails as checks (nsbx validate)
|
||||||
|
# make build NAME=tim # compile the worktree's changes into the engram
|
||||||
|
# make destroy NAME=tim # tear it all down (branch kept)
|
||||||
|
#
|
||||||
|
# The isolated engram is ALWAYS a clone of the live store on a NON-default port;
|
||||||
|
# prod (:8742 / :7770) is untouchable from here.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# locate nsbx next to this Makefile, regardless of where make is run from
|
||||||
|
NSBX := $(dir $(realpath $(lastword $(MAKEFILE_LIST))))nsbx
|
||||||
|
SBX := dev-$(NAME)
|
||||||
|
API ?= /api/stats
|
||||||
|
|
||||||
|
.DEFAULT_GOAL := help
|
||||||
|
|
||||||
|
.PHONY: help dev build run test status list destroy bt
|
||||||
|
|
||||||
|
help: ## Show this help
|
||||||
|
@echo "nsbx dev-environment — one-command isolated Neuron dev setup"
|
||||||
|
@echo ""
|
||||||
|
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(firstword $(MAKEFILE_LIST)) \
|
||||||
|
| awk 'BEGIN{FS=":.*?## "}{printf " make %-22s %s\n", $$1, $$2}'
|
||||||
|
@echo ""
|
||||||
|
@echo " Variables: NAME=<dev name> (required for most) API=<path> (run)"
|
||||||
|
@echo " BASE=<git ref> WT=<worktree dir> PORT=<n> ARGS=<extra nsbx flags>"
|
||||||
|
|
||||||
|
dev: ## Create branch + worktree + isolated engram (NAME=x [BASE=ref PORT=n WT=dir ARGS=...])
|
||||||
|
@test -n "$(NAME)" || { echo "usage: make dev NAME=<name>"; exit 2; }
|
||||||
|
$(NSBX) dev $(NAME) $(if $(BASE),--base $(BASE)) $(if $(PORT),--port $(PORT)) $(if $(WT),--worktree $(WT)) $(ARGS)
|
||||||
|
|
||||||
|
build: ## Rebuild the engram from the dev worktree's own source (NAME=x)
|
||||||
|
@test -n "$(NAME)" || { echo "usage: make build NAME=<name>"; exit 2; }
|
||||||
|
@wt=$$(python3 -c "import json,os;print(json.load(open(os.path.expanduser('~/.neuron/sandboxes/$(SBX)/dev.json')))['worktree'])" 2>/dev/null); \
|
||||||
|
test -n "$$wt" || { echo "no dev.json for $(SBX) — run 'make dev NAME=$(NAME)' first"; exit 2; }; \
|
||||||
|
$(NSBX) build $(SBX) --source "$$wt"
|
||||||
|
|
||||||
|
bt: build run ## Fast El loop: rebuild the engram from the worktree, then poke it (NAME=x)
|
||||||
|
|
||||||
|
run: ## Poke the isolated engram's API (NAME=x [API=/api/stats])
|
||||||
|
@test -n "$(NAME)" || { echo "usage: make run NAME=<name> [API=/path]"; exit 2; }
|
||||||
|
$(NSBX) run $(SBX) api $(API)
|
||||||
|
|
||||||
|
test: ## Run the safety rails as checks: zero-loss, reboot, RSS, parity, keystones (NAME=x)
|
||||||
|
@test -n "$(NAME)" || { echo "usage: make test NAME=<name>"; exit 2; }
|
||||||
|
$(NSBX) validate $(SBX)
|
||||||
|
|
||||||
|
status: ## Show one sandbox's status (NAME=x), or list all if NAME is unset
|
||||||
|
@if [ -n "$(NAME)" ]; then $(NSBX) status $(SBX); else $(NSBX) list; fi
|
||||||
|
|
||||||
|
list: ## List all sandboxes
|
||||||
|
$(NSBX) list
|
||||||
|
|
||||||
|
destroy: ## Tear down engram + worktree (NAME=x [ARGS=--delete-branch])
|
||||||
|
@test -n "$(NAME)" || { echo "usage: make destroy NAME=<name>"; exit 2; }
|
||||||
|
$(NSBX) dev-down $(NAME) $(ARGS)
|
||||||
@@ -0,0 +1,176 @@
|
|||||||
|
# nsbx — the Neuron Sandbox
|
||||||
|
|
||||||
|
**Dev environment as a primitive.** A reproducible way to run experiments *and code
|
||||||
|
changes* against the **real** engram runtime on an isolated snapshot of the live
|
||||||
|
mind — with a gated promote-to-prod path built on the proven rails.
|
||||||
|
|
||||||
|
Everyone (Tim, any team member, any agent) gets their own private, safe copy of the
|
||||||
|
mind to build against. **Prod — the live Neuron on `:8742` (engram) / `:7770`
|
||||||
|
(soul) — is untouchable from a sandbox.** A sandbox runs a *separate* engram
|
||||||
|
process, on a *separate* port, against a *separate* clone of the store. The only op
|
||||||
|
that can ever reach prod is `promote`, which is explicit, gated, and per-use
|
||||||
|
approved.
|
||||||
|
|
||||||
|
It **wraps the real engram binary** — it never reimplements any engram logic. It
|
||||||
|
generalises two proven proto-sandboxes into one primitive:
|
||||||
|
|
||||||
|
- the **cog-arch** build — isolated git worktree + build + clone of the live `.egm` + real C tests
|
||||||
|
- the **store-fix** cutover — secondary soul + launchctl `bootout → settle → bootstrap` rails
|
||||||
|
|
||||||
|
## Quickstart
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PATH="$PWD:$PATH" # or symlink nsbx onto your PATH
|
||||||
|
|
||||||
|
nsbx up # your private copy of the mind (auto-named <user>-dev)
|
||||||
|
nsbx run <name> api /api/stats # poke it
|
||||||
|
nsbx validate <name> # prove it: zero-loss, reboot, RSS, retrieval, keystones
|
||||||
|
nsbx destroy <name> # cheap teardown; live untouched
|
||||||
|
```
|
||||||
|
|
||||||
|
That is the whole loop. Sane defaults: stock prod binary, auto-allocated port
|
||||||
|
(`8900+`, never `8742`/`7770`), snapshot of the live store.
|
||||||
|
|
||||||
|
## One-command dev onboarding — `nsbx dev` (start here)
|
||||||
|
|
||||||
|
Going from a clone to *coding on the mind* is a single command. It creates a git
|
||||||
|
**branch**, a persistent git **worktree**, and an **isolated engram** (a clone of the
|
||||||
|
live store on a non-default port) — and wires the whole worktree to that clone so you
|
||||||
|
**cannot hit live `:8742` by accident**.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make dev NAME=tim # branch wt/tim + worktree + isolated engram, in one shot
|
||||||
|
cd ~/Development/neuron-technologies/el-worktrees/tim
|
||||||
|
source .nsbx-env # every ENGRAM_* var now points at YOUR clone
|
||||||
|
|
||||||
|
# edit El in the worktree, then the fast loop:
|
||||||
|
make build NAME=tim # compile your El change into the isolated engram
|
||||||
|
make run NAME=tim # poke it (API=/api/stats by default)
|
||||||
|
make test NAME=tim # run the safety rails as checks
|
||||||
|
make destroy NAME=tim # tear it all down (branch kept; live untouched)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why this exists:** so provisional/experimental work is built **directly in El against a
|
||||||
|
throwaway cloned engram** — not prototyped in Python and re-ported later. The El
|
||||||
|
edit → `make build` → `make run` loop is the path of least resistance; that double-work is
|
||||||
|
what stranded the translation faculty for weeks.
|
||||||
|
|
||||||
|
### What `nsbx dev <name>` does, in order
|
||||||
|
|
||||||
|
1. **branch** — `git worktree add -b <prefix><name>` (default prefix `dev/`; a *real named
|
||||||
|
branch*, never detached HEAD).
|
||||||
|
2. **worktree** — at a **persistent** path (default `…/el-worktrees/<name>`, override
|
||||||
|
`NSBX_DEV_WT_ROOT`). It **refuses** `/tmp` — temp dirs are ablated on compaction, which
|
||||||
|
is the exact "worktree in /tmp + no branch = lost work" failure this designs out.
|
||||||
|
3. **isolated engram** — `nsbx create` under the hood: clone of the live store + WAL +
|
||||||
|
config, booted on an auto-allocated port (`8900+`, never `:8742`/`:7770`). Stock prod
|
||||||
|
binary by default (instant); `--build` compiles the worktree's own runtime instead.
|
||||||
|
4. **env pin** — writes `.nsbx-env` (+ `.envrc` for direnv) into the worktree exporting
|
||||||
|
`ENGRAM_URL / ENGRAM_PORT / ENGRAM_DATA_DIR / ENGRAM_API_KEY / NEURON_ENGRAM_URL …` — all
|
||||||
|
pointing at the clone. Nothing references live.
|
||||||
|
|
||||||
|
```
|
||||||
|
nsbx dev <name> [--base REF] [--worktree DIR] [--port N] [--prefix P] [--build] [--no-engram] [--repo R]
|
||||||
|
nsbx dev-down <name> [--delete-branch] [--repo R] # destroy engram + remove worktree
|
||||||
|
```
|
||||||
|
|
||||||
|
### Makefile targets
|
||||||
|
|
||||||
|
| target | does |
|
||||||
|
|--------|------|
|
||||||
|
| `make dev NAME=x` | branch + worktree + isolated engram (one shot) |
|
||||||
|
| `make bt NAME=x` | fast El loop: `build` then `run` |
|
||||||
|
| `make build NAME=x` | recompile the engram from the worktree's El source |
|
||||||
|
| `make run NAME=x` | poke the isolated engram (`API=/api/stats`) |
|
||||||
|
| `make test NAME=x` | rails as checks (`nsbx validate`) |
|
||||||
|
| `make status [NAME=x]` | inspect one, or `list` all |
|
||||||
|
| `make destroy NAME=x` | tear down (add `ARGS=--delete-branch` to drop the branch) |
|
||||||
|
|
||||||
|
> Note: if a bare `dev` branch already exists in the repo, git can't create `dev/*` names —
|
||||||
|
> pass `--prefix wt/` (or delete the stray `dev` branch). The tool surfaces git's exact error.
|
||||||
|
|
||||||
|
## The code-change dev loop (first-class)
|
||||||
|
|
||||||
|
Run *your changed runtime*, not just the stock binary, against a snapshot:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# build a runtime from a working tree, a git branch, or a prebuilt binary:
|
||||||
|
nsbx create feat --source /path/to/worktree # elc + cc build from source
|
||||||
|
nsbx create feat --branch feat/my-change --repo <r> # worktree the branch, then build
|
||||||
|
nsbx create feat --binary /path/to/engram # use a prebuilt binary
|
||||||
|
|
||||||
|
nsbx build feat --source /path/to/worktree # rebuild + hot-restart in place
|
||||||
|
nsbx validate feat # prove the change is safe
|
||||||
|
nsbx promote feat --i-approve-prod-cutover # gated rails cutover (see below)
|
||||||
|
```
|
||||||
|
|
||||||
|
The build replicates the engram release recipe exactly:
|
||||||
|
`elc engram/src/server.el > engram.c` then
|
||||||
|
`cc -std=c11 -O2 -I lang/runtime engram.c el_runtime.c engram_*.c -lcurl -lpthread`.
|
||||||
|
|
||||||
|
## Lifecycle
|
||||||
|
|
||||||
|
| op | what it does |
|
||||||
|
|----|--------------|
|
||||||
|
| `create <name> [--port N] [--source\|--branch\|--binary]` | consistent snapshot of the live store+WAL+config into an isolated dir; place or **build** the runtime; boot the real engram daemon on an isolated port. Named, versioned (binary sha + egm sha in `manifest.json`), reproducible. |
|
||||||
|
| `up [name]` | one command: create-if-missing then start; prints the URL. |
|
||||||
|
| `build <name> --source\|--branch` | rebuild the runtime from a code change and hot-restart on the same clone+port. |
|
||||||
|
| `run <name> <cmd…>` / `run <name> api <path> [json]` | run an experiment against the real runtime; capture output + before/after stats + wall time. Env: `$SBX_URL $SBX_PORT $SBX_KEY $SBX_DATA $SBX_BIN`. |
|
||||||
|
| `validate <name>` | the rails as first-class checks (below). |
|
||||||
|
| `promote <name> [--data] [--i-approve-prod-cutover]` | **the only prod-touching op.** Gated rails cutover. DRY-RUN plan unless approved. |
|
||||||
|
| `destroy <name>` | stop the isolated daemon, free the port, remove the clone. Live untouched. |
|
||||||
|
| `list` / `status <name>` | inspect. |
|
||||||
|
|
||||||
|
## `validate` — the rails as checks
|
||||||
|
|
||||||
|
- **zero-loss-under-load** — node/edge counts hold at/above baseline through ~15s of sustained tick+read load
|
||||||
|
- **reboot-prove** — counts survive a real stop→start of the daemon
|
||||||
|
- **rss-bound** — daemon RSS under `NSBX_RSS_BOUND_MB` (default 550 MB, from the store-fix reboot-proof)
|
||||||
|
- **retrieval-parity** — top-k node ids for a fixed probe set match the create-time baseline
|
||||||
|
- **keystone-integrity** — `kn-efeb4a5b…` and `kn-5b606390…` present and intact
|
||||||
|
|
||||||
|
A PASS writes `validate.json` stamped with the binary sha; `promote` refuses unless
|
||||||
|
the current binary has a fresh PASS on record.
|
||||||
|
|
||||||
|
## `promote` — gated cutover (rails only)
|
||||||
|
|
||||||
|
Default is a **dry-run plan**. With `--i-approve-prod-cutover` it, in order:
|
||||||
|
|
||||||
|
1. **snapshot-first** — back up live `egm`+`wal`+`plist` to `~/.neuron/backups/promote-<name>-<ts>/` with a `rollback.txt`
|
||||||
|
2. **additive** binary install — copy the validated binary to a *new* file, update the plist `ENGRAM_REAL_BIN` (old binary retained — additive/supersede, never destructive)
|
||||||
|
3. **rails cutover** — `launchctl bootout` → **settle-poll** (prints until the job is gone) → `launchctl bootstrap`. Never `pkill`, never `kickstart -k`.
|
||||||
|
4. **verify** — `/api/stats` returns, edges ≥ baseline, keystones intact
|
||||||
|
5. **auto-rollback armed** — any verify failure restores the plist (and data, if `--data`) and boots the prior binary back via the same rails
|
||||||
|
|
||||||
|
## Isolation guarantees
|
||||||
|
|
||||||
|
- separate **port** (`8900+`; refuses `8742`/`7770`), separate **store clone**, separate **process**
|
||||||
|
- a hard guard refuses to boot a sandbox daemon whose data dir resolves to the live store
|
||||||
|
- sandboxes are plain supervised background processes (not launchd), so teardown is a signal + settle-poll — it can never touch the prod launchd job
|
||||||
|
- prod is read exactly twice: once for the snapshot, and (only if you approve) during `promote`
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
- tool: `tools/neuron-sandbox/nsbx` (this repo, branch `feat/neuron-sandbox`)
|
||||||
|
- runtime state: `~/.neuron/sandboxes/<name>/` — `data/` (clone), `bin/engram`, `build/`, `logs/`, `manifest.json`, `validate.json`, `baseline/`
|
||||||
|
|
||||||
|
## Validated (dogfood)
|
||||||
|
|
||||||
|
Standing up a sandbox from a live-store clone and reproducing a **known** result:
|
||||||
|
|
||||||
|
- **retrieval-parity 25/25** top-k id overlap vs baseline; sandbox boot-stats exactly matched the live baseline captured at snapshot time (10 672 nodes / 32 439 edges) — the wrapped real binary faithfully reloads the live mind
|
||||||
|
- reboot-prove + zero-loss PASS; RSS 379 MB < 550 MB; keystones intact
|
||||||
|
- the **cog-arch correspondence-loop** re-run *inside* the sandbox reproduced the known calibration numbers exactly: held-Brier **0.028648 → 0.000586** (98.0% reduction), monotone, **reboot bit-identical**, metastability holds; and the real-store Stance persistence reboot-proved at **10 994-node** scale (`think()` on real 768-dim embeddings) against a scratch copy of the sandbox's own clone — never live
|
||||||
|
- `promote` dry-run refused to touch prod; teardown freed the port; live `:8742`/`:7770` never perturbed (soul uptime unbroken)
|
||||||
|
|
||||||
|
## Migrating existing experiments
|
||||||
|
|
||||||
|
Each ad-hoc harness becomes `nsbx run <name> …` (or `--source` build) against a sandbox:
|
||||||
|
|
||||||
|
- **cog-arch** — `nsbx create x --source <worktree>` then `nsbx run x -- bash cogarch_dogfood.sh` (compiles + runs the real C cognition tests against `$SBX_DATA`)
|
||||||
|
- **codec / ingest / faculty** — `nsbx run x api /api/<endpoint> '<json>'` against the isolated daemon, or a script using `$SBX_URL`/`$SBX_KEY`; measure with the built-in before/after stats
|
||||||
|
|
||||||
|
## Env knobs
|
||||||
|
|
||||||
|
`NSBX_ROOT`, `NSBX_PORT_BASE`, `NSBX_RSS_BOUND_MB`, `NSBX_REMERGE_THRESHOLD`,
|
||||||
|
`EL_REPO` (for `elc` + runtime sources), `ENGRAM_LIVE_DATA_DIR`, `ENGRAM_LIVE_PLIST`.
|
||||||
+30
@@ -0,0 +1,30 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# cog-arch correspondence-loop dogfood — RUN INSIDE the sandbox via `nsbx run`.
|
||||||
|
# Compiles the REAL engram C runtime + cognition tests and reproduces the known
|
||||||
|
# calibration result (memory 194c69c8): held-Brier 0.028648 -> 0.000586, reboot-proven,
|
||||||
|
# then reboot-proves the Stance persistence against a SCRATCH COPY of THIS sandbox's
|
||||||
|
# clone of the real store (never live, never the running daemon's file).
|
||||||
|
set -euo pipefail
|
||||||
|
WT="${COGARCH_WT:-/private/tmp/claude-501/-Users-will/6531446d-bc27-4095-930b-e04777c3db4f/scratchpad/cogarch-wt}"
|
||||||
|
RT="$WT/lang/runtime"; T="$WT/engram/test"
|
||||||
|
: "${SBX_DATA:?run me via: nsbx run <name> -- bash cogarch_dogfood.sh}"
|
||||||
|
B="$(mktemp -d)"
|
||||||
|
echo "### building cog-arch tests against the real engram runtime sources"
|
||||||
|
cc -std=c11 -O2 -w -I "$RT" -o "$B/test_cognition" \
|
||||||
|
"$T/test_cognition.c" "$RT/engram_cognition.c" "$RT/engram_reason.c" \
|
||||||
|
"$RT/engram_geometry.c" "$RT/engram_store.c" "$RT/engram_vindex.c" -lm
|
||||||
|
cc -std=c11 -O2 -w -I "$RT" -o "$B/test_realstore" \
|
||||||
|
"$T/test_cognition_realstore.c" "$RT/engram_cognition.c" "$RT/engram_reason.c" \
|
||||||
|
"$RT/engram_geometry.c" "$RT/engram_store.c" "$RT/engram_vindex.c" -lm
|
||||||
|
|
||||||
|
echo; echo "### [A] synthetic correspondence-loop (known: Brier 0.028648 -> 0.000586)"
|
||||||
|
"$B/test_cognition" | grep -E "held-Brier|reduction|reboot|monotone|metastab|RESULT" || true
|
||||||
|
|
||||||
|
echo; echo "### [B] reboot-prove Stance on a SCRATCH COPY of this sandbox's real-store clone"
|
||||||
|
SCRATCH="$B/store-clone"; mkdir -p "$SCRATCH"
|
||||||
|
cp -p "$SBX_DATA/neuron.egm" "$SCRATCH/" 2>/dev/null || true
|
||||||
|
cp -p "$SBX_DATA/neuron.wal" "$SCRATCH/" 2>/dev/null || true
|
||||||
|
cp -p "$SBX_DATA/conf" "$SCRATCH/" 2>/dev/null || true
|
||||||
|
cp -p "$SBX_DATA/meta.json" "$SCRATCH/" 2>/dev/null || true
|
||||||
|
"$B/test_realstore" "$SCRATCH" || true
|
||||||
|
rm -rf "$B"
|
||||||
Executable
+836
@@ -0,0 +1,836 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# nsbx — the Neuron Sandbox: a reproducible primitive for running experiments and
|
||||||
|
# code changes against the REAL engram runtime on an isolated snapshot of the live
|
||||||
|
# mind, with a gated promote-to-prod path built on the proven rails.
|
||||||
|
#
|
||||||
|
# It WRAPS the real engram binary — it never reimplements any engram logic. The only
|
||||||
|
# prod-touching op is `promote`, which is explicit, gated, and per-use approved.
|
||||||
|
#
|
||||||
|
# Generalises two proven proto-sandboxes:
|
||||||
|
# - the cog-arch build (isolated git worktree + build + clone of live .egm + real C tests)
|
||||||
|
# - the store-fix cutover (secondary soul + launchctl bootout->settle->bootstrap rails)
|
||||||
|
#
|
||||||
|
# Lifecycle: create -> [build] -> run -> validate -> promote(gated) -> destroy
|
||||||
|
#
|
||||||
|
# Rails (always): built offline; NEVER auto-promotes; never touches live :8742/:7770
|
||||||
|
# except READ for the snapshot and the gated promote; snapshot-first; honest measured
|
||||||
|
# reporting. Cutover is launchctl bootout -> settle-poll -> bootstrap ONLY —
|
||||||
|
# never pkill, never kickstart -k.
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- constants ----
|
||||||
|
LIVE_DATA_DIR="${ENGRAM_LIVE_DATA_DIR:-$HOME/.neuron/engram}"
|
||||||
|
LIVE_PLIST="${ENGRAM_LIVE_PLIST:-$HOME/Library/LaunchAgents/ai.neuron.engram.plist}"
|
||||||
|
LIVE_LABEL="ai.neuron.engram"
|
||||||
|
LIVE_BIND_PORT=8742 # engram — FORBIDDEN for sandboxes
|
||||||
|
SOUL_PORT=7770 # soul — FORBIDDEN for sandboxes
|
||||||
|
LIVE_KEY="${ENGRAM_API_KEY:-ntn-user-2026}"
|
||||||
|
LIVE_URL="http://127.0.0.1:${LIVE_BIND_PORT}"
|
||||||
|
SBX_ROOT="${NSBX_ROOT:-$HOME/.neuron/sandboxes}"
|
||||||
|
BACKUP_ROOT="$HOME/.neuron/backups"
|
||||||
|
EL_REPO="${EL_REPO:-$HOME/Development/neuron-technologies/foundation/el}"
|
||||||
|
PORT_BASE="${NSBX_PORT_BASE:-8900}"
|
||||||
|
RSS_BOUND_MB="${NSBX_RSS_BOUND_MB:-550}" # from store-fix reboot-proof (aaf13f88)
|
||||||
|
REMERGE_THRESHOLD="${NSBX_REMERGE_THRESHOLD:-40000}"
|
||||||
|
KEYSTONES=( "kn-efeb4a5b-5aff-4759-8a97-7233099be6ee" "kn-5b606390-a52d-4ca2-8e0e-eba141d13440" )
|
||||||
|
# fixed probe set for retrieval-parity (stable, identity-anchored)
|
||||||
|
PARITY_QUERIES=( "who am I" "self identity core" "engram store durability" "keystone self anchor" "grounding honesty" )
|
||||||
|
|
||||||
|
C_RED=$'\033[31m'; C_GRN=$'\033[32m'; C_YEL=$'\033[33m'; C_DIM=$'\033[2m'; C_BLD=$'\033[1m'; C_0=$'\033[0m'
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- helpers ------
|
||||||
|
die(){ printf '%serror:%s %s\n' "$C_RED" "$C_0" "$*" >&2; exit 1; }
|
||||||
|
log(){ printf '%s==>%s %s\n' "$C_BLD" "$C_0" "$*" >&2; }
|
||||||
|
info(){ printf ' %s\n' "$*" >&2; }
|
||||||
|
ok(){ printf ' %s%s%s\n' "$C_GRN" "$*" "$C_0" >&2; }
|
||||||
|
warn(){ printf ' %s%s%s\n' "$C_YEL" "$*" "$C_0" >&2; }
|
||||||
|
need(){ command -v "$1" >/dev/null 2>&1 || die "missing dependency: $1"; }
|
||||||
|
now(){ date -u +%Y%m%dT%H%M%SZ; }
|
||||||
|
sha(){ shasum -a 256 "$1" 2>/dev/null | awk '{print $1}'; }
|
||||||
|
epoch(){ python3 -c 'import time;print(time.time())'; }
|
||||||
|
|
||||||
|
sdir(){ printf '%s/%s' "$SBX_ROOT" "$1"; }
|
||||||
|
manifest(){ printf '%s/manifest.json' "$(sdir "$1")"; }
|
||||||
|
mexists(){ [ -f "$(manifest "$1")" ]; }
|
||||||
|
mget(){ # mget <name> <jsonpath>
|
||||||
|
python3 -c "import json,sys; d=json.load(open('$(manifest "$1")')); print(d$2)" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
port_free(){ ! (exec 3<>"/dev/tcp/127.0.0.1/$1") 2>/dev/null; }
|
||||||
|
alloc_port(){
|
||||||
|
local p="$PORT_BASE"
|
||||||
|
while :; do
|
||||||
|
if [ "$p" = "$LIVE_BIND_PORT" ] || [ "$p" = "$SOUL_PORT" ]; then p=$((p+1)); continue; fi
|
||||||
|
if port_free "$p" && ! _port_claimed "$p"; then echo "$p"; return 0; fi
|
||||||
|
p=$((p+1)); [ "$p" -gt 9100 ] && die "no free sandbox port in range"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
_port_claimed(){ # is another sandbox already assigned this port?
|
||||||
|
local p="$1" d
|
||||||
|
for d in "$SBX_ROOT"/*/manifest.json; do
|
||||||
|
[ -f "$d" ] || continue
|
||||||
|
[ "$(python3 -c "import json;print(json.load(open('$d'))['port'])" 2>/dev/null)" = "$p" ] && return 0
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
live_stats(){ curl -s -m5 "$LIVE_URL/api/stats" 2>/dev/null; }
|
||||||
|
api(){ # api <name> <path> [json-body]
|
||||||
|
local name="$1" path="$2" body="${3:-}"
|
||||||
|
local port; port="$(mget "$name" "['port']")"; [ -n "$port" ] || die "unknown sandbox: $name"
|
||||||
|
local url="http://127.0.0.1:${port}${path}"
|
||||||
|
if [ -n "$body" ]; then curl -s -m30 -X POST -H 'Content-Type: application/json' -d "$body" "$url"
|
||||||
|
else curl -s -m30 "$url"; fi
|
||||||
|
}
|
||||||
|
sbx_stats(){ api "$1" "/api/stats"; }
|
||||||
|
stat_field(){ printf '%s' "$1" | sed -n "s/.*\"$2\":\([0-9]*\).*/\1/p"; }
|
||||||
|
|
||||||
|
daemon_pid(){ local f; f="$(sdir "$1")/daemon.pid"; [ -f "$f" ] && cat "$f" || true; }
|
||||||
|
daemon_alive(){ local p; p="$(daemon_pid "$1")"; [ -n "$p" ] && kill -0 "$p" 2>/dev/null; }
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- elc/build ----
|
||||||
|
find_elc(){
|
||||||
|
command -v elc 2>/dev/null && return 0
|
||||||
|
local arch; arch="$(uname -m)"
|
||||||
|
case "$arch" in
|
||||||
|
arm64) echo "$EL_REPO/lang/dist/platform/elc-darwin-arm64";;
|
||||||
|
x86_64) echo "$EL_REPO/lang/dist/platform/elc-linux-amd64";;
|
||||||
|
*) echo "$EL_REPO/lang/dist/platform/elc";;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# _build_binary <src_tree> <out_bin> <build_log_dir>
|
||||||
|
# Replicates the proven engram release recipe:
|
||||||
|
# elc engram/src/server.el > engram.c
|
||||||
|
# cc -std=c11 -O2 -I lang/runtime engram.c el_runtime.c engram_*.c -lcurl -lpthread
|
||||||
|
_build_binary(){
|
||||||
|
local src="$1" out="$2" blog="$3"
|
||||||
|
local elc server rt
|
||||||
|
elc="$(find_elc)"; [ -x "$elc" ] || die "elc not found/executable: $elc (set EL_REPO)"
|
||||||
|
server="$src/engram/src/server.el"; rt="$src/lang/runtime"
|
||||||
|
[ -f "$server" ] || die "no engram/src/server.el under source tree: $src"
|
||||||
|
[ -f "$rt/el_runtime.c" ] || die "no lang/runtime/el_runtime.c under source tree: $src (this branch may keep it generated/untracked)"
|
||||||
|
ls "$rt"/engram_*.c >/dev/null 2>&1 || die "no lang/runtime/engram_*.c engine sources under: $src"
|
||||||
|
mkdir -p "$blog"
|
||||||
|
log "build: elc transpile server.el -> engram.c"
|
||||||
|
"$elc" "$server" > "$blog/engram.c" 2>"$blog/elc.err" || { cat "$blog/elc.err" >&2; die "elc transpile failed"; }
|
||||||
|
info "engram.c: $(wc -c <"$blog/engram.c" | tr -d ' ') bytes"
|
||||||
|
log "build: cc link (el_runtime + engram_* engine)"
|
||||||
|
cc -std=c11 -O2 -w -I "$rt" -o "$out" \
|
||||||
|
"$blog/engram.c" "$rt/el_runtime.c" "$rt"/engram_*.c \
|
||||||
|
-lcurl -lpthread 2>"$blog/cc.err" \
|
||||||
|
|| { grep -i 'error:' "$blog/cc.err" | sort -u | head >&2; die "cc link failed (see $blog/cc.err)"; }
|
||||||
|
ok "built: $out ($(ls -lh "$out" | awk '{print $5}'), sha $(sha "$out" | cut -c1-12))"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- daemon -------
|
||||||
|
# start_daemon <name> : boots the sandbox's real engram binary on its isolated
|
||||||
|
# port against its cloned data dir, with the SAME auto-remerge net the live soul
|
||||||
|
# uses (so the sandbox faithfully reaches the live edge population on boot).
|
||||||
|
start_daemon(){
|
||||||
|
local name="$1" d; d="$(sdir "$name")"
|
||||||
|
daemon_alive "$name" && { info "already running (pid $(daemon_pid "$name"))"; return 0; }
|
||||||
|
local port bin data export key
|
||||||
|
port="$(mget "$name" "['port']")"; bin="$d/bin/engram"; data="$d/data"
|
||||||
|
key="sbx-$name"; export="$data/.scan-export.reseed-clean.json"
|
||||||
|
[ -x "$bin" ] || die "sandbox binary missing: $bin"
|
||||||
|
[ "$port" != "$LIVE_BIND_PORT" ] && [ "$port" != "$SOUL_PORT" ] || die "refusing forbidden port $port"
|
||||||
|
[ -f "$data/neuron.egm" ] || die "sandbox has no cloned store: $data/neuron.egm"
|
||||||
|
# HARD guard: never point a sandbox daemon at the live data dir.
|
||||||
|
[ "$(cd "$data" && pwd -P)" != "$(cd "$LIVE_DATA_DIR" && pwd -P)" ] || die "refusing: sandbox data dir resolves to LIVE store"
|
||||||
|
|
||||||
|
log "boot engram on isolated :$port (data=$data)"
|
||||||
|
(
|
||||||
|
ENGRAM_DATA_DIR="$data" ENGRAM_BIND=":$port" ENGRAM_API_KEY="$key" \
|
||||||
|
ENGRAM_STORE=1 ENGRAM_CHRONOCEPTION=1 ENGRAM_SELF_REIFY=1 ENGRAM_GC=1 \
|
||||||
|
ENGRAM_POOL_FRAMES=16384 ENGRAM_WRITE_BARRIER=1 \
|
||||||
|
exec "$bin"
|
||||||
|
) >"$d/logs/daemon.log" 2>&1 &
|
||||||
|
local pid=$!
|
||||||
|
echo "$pid" > "$d/daemon.pid"
|
||||||
|
# readiness poll
|
||||||
|
local url="http://127.0.0.1:$port" i s
|
||||||
|
for i in $(seq 1 30); do
|
||||||
|
s="$(curl -s -m3 "$url/api/stats" 2>/dev/null)"
|
||||||
|
[ -n "$s" ] && break; sleep 0.5
|
||||||
|
done
|
||||||
|
[ -n "$s" ] || { warn "daemon did not become ready (see $d/logs/daemon.log)"; return 1; }
|
||||||
|
ok "ready pid=$pid boot-stats: $s"
|
||||||
|
# auto-remerge net (idempotent): match live edge population if the export is present
|
||||||
|
if [ -f "$export" ]; then
|
||||||
|
local edges; edges="$(stat_field "$s" edge_count)"
|
||||||
|
if [ -n "$edges" ] && [ "$edges" -lt "$REMERGE_THRESHOLD" ]; then
|
||||||
|
log "auto-remerge: booted with $edges edges (< $REMERGE_THRESHOLD) — merging full edge export"
|
||||||
|
local r; r="$(curl -s -m300 -X POST -H 'Content-Type: application/json' \
|
||||||
|
-d "{\"_auth\":\"$key\",\"path\":\"$export\"}" "$url/api/load-merge" 2>/dev/null)"
|
||||||
|
info "remerge resp: ${r:0:120}"
|
||||||
|
ok "post-remerge stats: $(curl -s -m5 "$url/api/stats")"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# stop_daemon <name> : graceful TERM + settle-poll until the port is free.
|
||||||
|
# (Sandbox daemons are plain supervised bg processes — not launchd — so teardown
|
||||||
|
# is a signal + poll, never pkill of anything else.)
|
||||||
|
stop_daemon(){
|
||||||
|
local name="$1" pid port
|
||||||
|
pid="$(daemon_pid "$name")"; port="$(mget "$name" "['port']")"
|
||||||
|
[ -n "$pid" ] || { info "not running"; return 0; }
|
||||||
|
log "stop daemon pid=$pid, settle-poll until :$port frees"
|
||||||
|
kill "$pid" 2>/dev/null || true
|
||||||
|
local i
|
||||||
|
for i in $(seq 1 40); do
|
||||||
|
kill -0 "$pid" 2>/dev/null || { port_free "$port" && { ok "stopped, port $port free"; : >"$(sdir "$name")/daemon.pid"; return 0; }; }
|
||||||
|
printf '.' >&2; sleep 0.5
|
||||||
|
done
|
||||||
|
printf '\n' >&2
|
||||||
|
kill -9 "$pid" 2>/dev/null || true; sleep 1
|
||||||
|
: >"$(sdir "$name")/daemon.pid"
|
||||||
|
port_free "$port" && ok "stopped (after SIGKILL), port $port free" || warn "port $port still busy"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ================================================================ create =======
|
||||||
|
cmd_create(){
|
||||||
|
local name="" port="" src="" branch="" repo="$EL_REPO" binpath=""
|
||||||
|
# first positional arg is the name unless it's a flag; default to "<user>-dev"
|
||||||
|
if [ $# -gt 0 ] && [ "${1#-}" = "$1" ]; then name="$1"; shift; else name="${USER:-dev}-dev"; fi
|
||||||
|
while [ $# -gt 0 ]; do case "$1" in
|
||||||
|
--port) port="$2"; shift 2;;
|
||||||
|
--source) src="$2"; shift 2;;
|
||||||
|
--branch) branch="$2"; shift 2;;
|
||||||
|
--repo) repo="$2"; shift 2;;
|
||||||
|
--binary) binpath="$2"; shift 2;;
|
||||||
|
*) die "unknown flag: $1";;
|
||||||
|
esac; done
|
||||||
|
mexists "$name" && die "sandbox '$name' already exists (destroy it first)"
|
||||||
|
need curl; need python3; need shasum
|
||||||
|
[ -f "$LIVE_DATA_DIR/neuron.egm" ] || die "live store not found: $LIVE_DATA_DIR/neuron.egm"
|
||||||
|
if [ -n "$port" ]; then
|
||||||
|
{ [ "$port" = "$LIVE_BIND_PORT" ] || [ "$port" = "$SOUL_PORT" ]; } && die "refusing forbidden port $port (live)"
|
||||||
|
port_free "$port" || die "port $port already in use"
|
||||||
|
else port="$(alloc_port)"; fi
|
||||||
|
|
||||||
|
local d; d="$(sdir "$name")"
|
||||||
|
mkdir -p "$d/data" "$d/bin" "$d/logs" "$d/build" "$d/baseline"
|
||||||
|
log "sandbox '$name' at $d (isolated port $port)"
|
||||||
|
|
||||||
|
# ---- CONSISTENT snapshot of the live mind (file-copy: same set the rails backup
|
||||||
|
# uses; WAL replay on sandbox boot reconciles the tail -> crash-consistent) ----
|
||||||
|
log "snapshot live store -> clone (store + WAL + config)"
|
||||||
|
local f
|
||||||
|
for f in neuron.egm neuron.wal conf meta.json self_anchor .scan-export.reseed-clean.json; do
|
||||||
|
if [ -e "$LIVE_DATA_DIR/$f" ]; then cp -p "$LIVE_DATA_DIR/$f" "$d/data/$f"; info "cloned $f ($(du -h "$d/data/$f" | awk '{print $1}'))"; fi
|
||||||
|
done
|
||||||
|
local egm_sha; egm_sha="$(sha "$d/data/neuron.egm")"
|
||||||
|
|
||||||
|
# ---- capture live baseline (READ only) ----
|
||||||
|
local lstats; lstats="$(live_stats)"
|
||||||
|
local base_nodes base_edges
|
||||||
|
base_nodes="$(stat_field "$lstats" node_count)"; base_edges="$(stat_field "$lstats" edge_count)"
|
||||||
|
info "live baseline stats: ${lstats:-<unavailable>}"
|
||||||
|
|
||||||
|
# ---- determine + place the runtime binary (versioned into the snapshot) ----
|
||||||
|
local source_desc live_bin
|
||||||
|
live_bin="$(_live_real_bin)"
|
||||||
|
if [ -n "$binpath" ]; then
|
||||||
|
[ -x "$binpath" ] || die "not an executable binary: $binpath"
|
||||||
|
cp -p "$binpath" "$d/bin/engram"; source_desc="prebuilt:$binpath"
|
||||||
|
elif [ -n "$src" ]; then
|
||||||
|
_build_binary "$src" "$d/bin/engram" "$d/build"; source_desc="source:$src"
|
||||||
|
elif [ -n "$branch" ]; then
|
||||||
|
log "worktree: $repo @ $branch -> $d/build/worktree"
|
||||||
|
git -C "$repo" worktree add --detach "$d/build/worktree" "$branch" >/dev/null 2>&1 \
|
||||||
|
|| die "git worktree add failed ($repo @ $branch)"
|
||||||
|
_build_binary "$d/build/worktree" "$d/bin/engram" "$d/build"; source_desc="branch:$branch@$repo"
|
||||||
|
else
|
||||||
|
[ -x "$live_bin" ] || die "cannot resolve live ENGRAM_REAL_BIN: $live_bin"
|
||||||
|
cp -p "$live_bin" "$d/bin/engram"; source_desc="stock-prod:$live_bin"
|
||||||
|
fi
|
||||||
|
local bin_sha; bin_sha="$(sha "$d/bin/engram")"
|
||||||
|
info "runtime: $source_desc (sha ${bin_sha:0:12})"
|
||||||
|
|
||||||
|
# ---- write manifest ----
|
||||||
|
python3 - "$name" "$port" "$source_desc" "$bin_sha" "$egm_sha" "$base_nodes" "$base_edges" "$(sha "$live_bin" 2>/dev/null)" <<'PY' > "$(manifest "$name")"
|
||||||
|
import json,sys,datetime
|
||||||
|
name,port,src,binsha,egmsha,bn,be,livebinsha=sys.argv[1:9]
|
||||||
|
json.dump({
|
||||||
|
"name":name,"port":int(port),"created_at":datetime.datetime.now(datetime.timezone.utc).isoformat(),
|
||||||
|
"source":src,"binary_sha256":binsha,"clone_egm_sha256":egmsha,
|
||||||
|
"live_binary_sha256":livebinsha,
|
||||||
|
"live_baseline":{"node_count":int(bn or 0),"edge_count":int(be or 0)},
|
||||||
|
"keystones":["kn-efeb4a5b-5aff-4759-8a97-7233099be6ee","kn-5b606390-a52d-4ca2-8e0e-eba141d13440"]
|
||||||
|
}, sys.stdout, indent=2)
|
||||||
|
PY
|
||||||
|
ok "manifest written"
|
||||||
|
|
||||||
|
# ---- boot + capture the sandbox's own settled baseline (reproducible target) ----
|
||||||
|
start_daemon "$name" || die "daemon failed to start"
|
||||||
|
local sstats; sstats="$(sbx_stats "$name")"
|
||||||
|
local sbn sbe; sbn="$(stat_field "$sstats" node_count)"; sbe="$(stat_field "$sstats" edge_count)"
|
||||||
|
_capture_retrieval "$name" "$d/baseline/retrieval.json"
|
||||||
|
# fold sandbox baseline into manifest
|
||||||
|
python3 - "$(manifest "$name")" "$sbn" "$sbe" <<'PY'
|
||||||
|
import json,sys
|
||||||
|
mf,bn,be=sys.argv[1],sys.argv[2],sys.argv[3]
|
||||||
|
d=json.load(open(mf)); d["sbx_baseline"]={"node_count":int(bn or 0),"edge_count":int(be or 0)}
|
||||||
|
json.dump(d,open(mf,'w'),indent=2)
|
||||||
|
PY
|
||||||
|
log "created."
|
||||||
|
info "sandbox baseline (settled): nodes=$sbn edges=$sbe"
|
||||||
|
info "next: nsbx validate $name | nsbx run $name api /api/stats"
|
||||||
|
}
|
||||||
|
|
||||||
|
_live_real_bin(){
|
||||||
|
python3 - "$LIVE_PLIST" <<'PY' 2>/dev/null
|
||||||
|
import sys,plistlib
|
||||||
|
try:
|
||||||
|
d=plistlib.load(open(sys.argv[1],'rb'))
|
||||||
|
print(d.get("EnvironmentVariables",{}).get("ENGRAM_REAL_BIN",""))
|
||||||
|
except Exception: print("")
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
_capture_retrieval(){ # <name> <outfile> : top-k ids for the fixed probe set
|
||||||
|
local name="$1" out="$2" q res
|
||||||
|
local port; port="$(mget "$name" "['port']")"; local key="sbx-$name"
|
||||||
|
{
|
||||||
|
echo "{"
|
||||||
|
local first=1
|
||||||
|
for q in "${PARITY_QUERIES[@]}"; do
|
||||||
|
res="$(curl -s -m10 -X POST -H 'Content-Type: application/json' \
|
||||||
|
-d "{\"_auth\":\"$key\",\"query\":\"$q\",\"limit\":5}" "http://127.0.0.1:$port/api/search" 2>/dev/null)"
|
||||||
|
local ids; ids="$(printf '%s' "$res" | python3 -c 'import sys,json
|
||||||
|
try:
|
||||||
|
d=json.load(sys.stdin)
|
||||||
|
rows=d if isinstance(d,list) else d.get("results",d.get("hits",[]))
|
||||||
|
print(json.dumps([r.get("id") for r in rows][:5]))
|
||||||
|
except Exception: print("[]")' 2>/dev/null)"
|
||||||
|
[ $first -eq 1 ] || echo ","; first=0
|
||||||
|
printf ' %s: %s' "$(python3 -c "import json,sys;print(json.dumps(sys.argv[1]))" "$q")" "${ids:-[]}"
|
||||||
|
done
|
||||||
|
echo ""; echo "}"
|
||||||
|
} > "$out"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ================================================================ up ===========
|
||||||
|
# Dead-simple one-command dev environment: `nsbx up` gives you (or Tim, or anyone)
|
||||||
|
# a private, isolated copy of the live mind to build against. Creates it on first
|
||||||
|
# run with sane defaults (stock prod binary, auto-allocated port), just starts it
|
||||||
|
# thereafter. Prod on :$LIVE_BIND_PORT/:$SOUL_PORT is unreachable from here by design.
|
||||||
|
cmd_up(){
|
||||||
|
local name; if [ $# -gt 0 ] && [ "${1#-}" = "$1" ]; then name="$1"; shift; else name="${USER:-dev}-dev"; fi
|
||||||
|
if mexists "$name"; then daemon_alive "$name" || start_daemon "$name"; else cmd_create "$name" "$@"; fi
|
||||||
|
local port; port="$(mget "$name" "['port']")"
|
||||||
|
echo >&2
|
||||||
|
ok "your sandbox '$name' is ready at http://127.0.0.1:$port (a private copy of the mind — prod is untouchable)"
|
||||||
|
info "experiment: nsbx run $name api /api/stats"
|
||||||
|
info "prove it: nsbx validate $name"
|
||||||
|
info "tear down: nsbx destroy $name"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ================================================================ build ========
|
||||||
|
# Rebuild an existing sandbox's runtime from a source tree/branch and hot-restart
|
||||||
|
# it on the SAME clone + port (the code-change dev loop, in place).
|
||||||
|
cmd_build(){
|
||||||
|
local name="$1"; shift || true
|
||||||
|
mexists "$name" || die "no such sandbox: $name"
|
||||||
|
local src="" branch="" repo="$EL_REPO"
|
||||||
|
while [ $# -gt 0 ]; do case "$1" in
|
||||||
|
--source) src="$2"; shift 2;; --branch) branch="$2"; shift 2;; --repo) repo="$2"; shift 2;;
|
||||||
|
*) die "unknown flag: $1";; esac; done
|
||||||
|
local d; d="$(sdir "$name")"
|
||||||
|
stop_daemon "$name"
|
||||||
|
if [ -n "$src" ]; then _build_binary "$src" "$d/bin/engram" "$d/build"
|
||||||
|
elif [ -n "$branch" ]; then
|
||||||
|
rm -rf "$d/build/worktree" 2>/dev/null; git -C "$repo" worktree prune 2>/dev/null
|
||||||
|
git -C "$repo" worktree add --detach "$d/build/worktree" "$branch" >/dev/null 2>&1 || die "worktree add failed"
|
||||||
|
_build_binary "$d/build/worktree" "$d/bin/engram" "$d/build"
|
||||||
|
else die "usage: nsbx build <name> --source DIR | --branch REF [--repo R]"; fi
|
||||||
|
# record new binary sha
|
||||||
|
python3 - "$(manifest "$name")" "$(sha "$d/bin/engram")" "${src:-branch:$branch}" <<'PY'
|
||||||
|
import json,sys; mf,s,src=sys.argv[1:4]
|
||||||
|
d=json.load(open(mf)); d["binary_sha256"]=s; d["source"]="rebuilt:"+src
|
||||||
|
json.dump(d,open(mf,'w'),indent=2)
|
||||||
|
PY
|
||||||
|
start_daemon "$name"
|
||||||
|
ok "rebuilt + restarted on :$(mget "$name" "['port']")"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ================================================================ run ==========
|
||||||
|
cmd_run(){
|
||||||
|
local name="$1"; shift || true
|
||||||
|
mexists "$name" || die "no such sandbox: $name"
|
||||||
|
daemon_alive "$name" || start_daemon "$name"
|
||||||
|
local d port; d="$(sdir "$name")"; port="$(mget "$name" "['port']")"
|
||||||
|
# direct API form: nsbx run <name> api <path> [json]
|
||||||
|
if [ "${1:-}" = "api" ]; then
|
||||||
|
api "$name" "$2" "${3:-}"; echo; return 0
|
||||||
|
fi
|
||||||
|
[ "${1:-}" = "--" ] && shift # allow an explicit separator: nsbx run <name> -- <cmd...>
|
||||||
|
[ $# -gt 0 ] || die "usage: nsbx run <name> <cmd...> | nsbx run <name> api <path> [json]"
|
||||||
|
local ts log0; ts="$(now)"; log0="$d/logs/run-$ts.log"
|
||||||
|
local s0 t0 t1 s1
|
||||||
|
s0="$(sbx_stats "$name")"; t0="$(epoch)"
|
||||||
|
log "run experiment against sandbox '$name' (:$port)"
|
||||||
|
info "cmd: $*"
|
||||||
|
( export SBX_NAME="$name" SBX_PORT="$port" SBX_URL="http://127.0.0.1:$port" \
|
||||||
|
SBX_KEY="sbx-$name" SBX_DATA="$d/data" SBX_BIN="$d/bin/engram"
|
||||||
|
"$@" ) 2>&1 | tee "$log0"
|
||||||
|
local rc=${PIPESTATUS[0]}
|
||||||
|
t1="$(epoch)"; s1="$(sbx_stats "$name")"
|
||||||
|
{
|
||||||
|
echo "--- nsbx run metrics ---"
|
||||||
|
echo "exit_code: $rc"
|
||||||
|
printf 'wall_secs: %.3f\n' "$(python3 -c "print($t1-$t0)")"
|
||||||
|
echo "stats_before: $s0"
|
||||||
|
echo "stats_after: $s1"
|
||||||
|
} | tee -a "$log0" >&2
|
||||||
|
return $rc
|
||||||
|
}
|
||||||
|
|
||||||
|
# ================================================================ validate =====
|
||||||
|
# The rails as first-class checks. Baseline = the sandbox's own settled state at
|
||||||
|
# create (reproducible). zero-loss through sustained load AND reboot; reboot-prove;
|
||||||
|
# RSS bound; retrieval parity; keystone integrity.
|
||||||
|
cmd_validate(){
|
||||||
|
local name="$1"; shift || true
|
||||||
|
mexists "$name" || die "no such sandbox: $name"
|
||||||
|
daemon_alive "$name" || start_daemon "$name"
|
||||||
|
local d port key; d="$(sdir "$name")"; port="$(mget "$name" "['port']")"; key="sbx-$name"
|
||||||
|
local url="http://127.0.0.1:$port"
|
||||||
|
local bn be; bn="$(mget "$name" "['sbx_baseline']['node_count']")"; be="$(mget "$name" "['sbx_baseline']['edge_count']")"
|
||||||
|
log "validate '$name' against baseline nodes=$bn edges=$be"
|
||||||
|
local -a names=() results=() details=()
|
||||||
|
|
||||||
|
# 1) sustained load — no data loss under activity
|
||||||
|
local s cur_n cur_e i
|
||||||
|
log "check: sustained load (~15s: tick + reads) then zero-loss"
|
||||||
|
for i in $(seq 1 15); do
|
||||||
|
curl -s -m5 -X POST -H 'Content-Type: application/json' -d "{\"_auth\":\"$key\"}" "$url/api/tick" >/dev/null 2>&1
|
||||||
|
curl -s -m5 "$url/api/stats" >/dev/null 2>&1
|
||||||
|
done
|
||||||
|
s="$(sbx_stats "$name")"; cur_n="$(stat_field "$s" node_count)"; cur_e="$(stat_field "$s" edge_count)"
|
||||||
|
names+=("zero-loss-under-load"); if [ "${cur_n:-0}" -ge "${bn:-0}" ] && [ "${cur_e:-0}" -ge "${be:-0}" ]; then
|
||||||
|
results+=("PASS"); else results+=("FAIL"); fi
|
||||||
|
details+=("nodes $cur_n>=$bn, edges $cur_e>=$be")
|
||||||
|
|
||||||
|
# 2) reboot-prove — counts survive a real restart
|
||||||
|
log "check: reboot-prove (stop -> start -> compare)"
|
||||||
|
local pre_n pre_e; pre_n="$cur_n"; pre_e="$cur_e"
|
||||||
|
stop_daemon "$name"; start_daemon "$name" >/dev/null
|
||||||
|
s="$(sbx_stats "$name")"; cur_n="$(stat_field "$s" node_count)"; cur_e="$(stat_field "$s" edge_count)"
|
||||||
|
names+=("reboot-prove"); if [ "${cur_n:-0}" -ge "${bn:-0}" ] && [ "${cur_e:-0}" -ge "${be:-0}" ]; then
|
||||||
|
results+=("PASS"); else results+=("FAIL"); fi
|
||||||
|
details+=("post-reboot nodes=$cur_n edges=$cur_e (pre $pre_n/$pre_e)")
|
||||||
|
|
||||||
|
# 3) RSS bound
|
||||||
|
log "check: RSS bound (< ${RSS_BOUND_MB}MB)"
|
||||||
|
local pid rss_kb rss_mb; pid="$(daemon_pid "$name")"
|
||||||
|
rss_kb="$(ps -o rss= -p "$pid" 2>/dev/null | tr -d ' ')"; rss_mb=$(( ${rss_kb:-0} / 1024 ))
|
||||||
|
names+=("rss-bound"); if [ "$rss_mb" -lt "$RSS_BOUND_MB" ] && [ "$rss_mb" -gt 0 ]; then results+=("PASS"); else results+=("FAIL"); fi
|
||||||
|
details+=("RSS=${rss_mb}MB (bound ${RSS_BOUND_MB}MB)")
|
||||||
|
|
||||||
|
# 4) retrieval parity vs the create-time baseline
|
||||||
|
log "check: retrieval parity vs baseline probe set"
|
||||||
|
_capture_retrieval "$name" "$d/logs/retrieval-$( now ).json"
|
||||||
|
local latest; latest="$(ls -t "$d/logs"/retrieval-*.json 2>/dev/null | head -1)"
|
||||||
|
local parity; parity="$(python3 - "$d/baseline/retrieval.json" "$latest" <<'PY'
|
||||||
|
import json,sys
|
||||||
|
def load(p):
|
||||||
|
try: return json.load(open(p))
|
||||||
|
except Exception: return {}
|
||||||
|
b,c=load(sys.argv[1]),load(sys.argv[2])
|
||||||
|
tot=hit=0
|
||||||
|
for q,ids in b.items():
|
||||||
|
cb=set(ids or []); cc=set(c.get(q) or [])
|
||||||
|
if not cb: continue
|
||||||
|
tot+=len(cb); hit+=len(cb & cc)
|
||||||
|
print(f"{hit}/{tot}" if tot else "0/0")
|
||||||
|
PY
|
||||||
|
)"
|
||||||
|
local ph="${parity%/*}" pt="${parity#*/}"
|
||||||
|
names+=("retrieval-parity"); if [ "${pt:-0}" -gt 0 ] && [ "${ph:-0}" -eq "${pt:-0}" ]; then results+=("PASS"); else results+=("FAIL"); fi
|
||||||
|
details+=("top-k id overlap $parity vs baseline")
|
||||||
|
|
||||||
|
# 5) keystone integrity
|
||||||
|
log "check: keystone integrity"
|
||||||
|
local kfail=0 kid kres
|
||||||
|
for kid in "${KEYSTONES[@]}"; do
|
||||||
|
kres="$(curl -s -m5 "$url/api/node/$kid" 2>/dev/null)"
|
||||||
|
printf '%s' "$kres" | grep -q "\"$kid\"" || kfail=1
|
||||||
|
done
|
||||||
|
names+=("keystone-integrity"); [ "$kfail" -eq 0 ] && results+=("PASS") || results+=("FAIL")
|
||||||
|
details+=("kn-efeb4a5b + kn-5b606390 present")
|
||||||
|
|
||||||
|
# ---- report + stamp ----
|
||||||
|
echo >&2
|
||||||
|
printf '%s VALIDATION — %s%s\n' "$C_BLD" "$name" "$C_0" >&2
|
||||||
|
local allpass=1 j
|
||||||
|
for j in "${!names[@]}"; do
|
||||||
|
local r="${results[$j]}" c="$C_GRN"; [ "$r" = FAIL ] && { c="$C_RED"; allpass=0; }
|
||||||
|
printf ' %s%-6s%s %-22s %s%s%s\n' "$c" "$r" "$C_0" "${names[$j]}" "$C_DIM" "${details[$j]}" "$C_0" >&2
|
||||||
|
done
|
||||||
|
local status; [ "$allpass" -eq 1 ] && status="PASS" || status="FAIL"
|
||||||
|
python3 - "$d/validate.json" "$status" "$(sha "$d/bin/engram")" "$(now)" "${names[*]}" "${results[*]}" <<'PY'
|
||||||
|
import json,sys
|
||||||
|
out,status,binsha,ts,ns,rs=sys.argv[1:7]
|
||||||
|
checks=[{"name":n,"result":r} for n,r in zip(ns.split(),rs.split())]
|
||||||
|
json.dump({"status":status,"binary_sha256":binsha,"ts":ts,"checks":checks},open(out,'w'),indent=2)
|
||||||
|
PY
|
||||||
|
printf ' %s==> %s%s\n' "$([ "$allpass" -eq 1 ] && echo "$C_GRN" || echo "$C_RED")" "$status" "$C_0" >&2
|
||||||
|
[ "$allpass" -eq 1 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
# ================================================================ promote ======
|
||||||
|
# The ONLY prod-touching op. Explicit, gated, per-use Will-approved. Rails ONLY:
|
||||||
|
# snapshot-first -> additive binary swap -> launchctl bootout -> settle-poll ->
|
||||||
|
# bootstrap -> verify -> auto-rollback on failure. NEVER pkill, NEVER kickstart -k.
|
||||||
|
# Default is a DRY-RUN plan; requires --i-approve-prod-cutover to actually cut over.
|
||||||
|
cmd_promote(){
|
||||||
|
local name="$1"; shift || true
|
||||||
|
mexists "$name" || die "no such sandbox: $name"
|
||||||
|
local approve=0 do_data=0
|
||||||
|
while [ $# -gt 0 ]; do case "$1" in
|
||||||
|
--i-approve-prod-cutover) approve=1; shift;;
|
||||||
|
--data) do_data=1; shift;;
|
||||||
|
*) die "unknown flag: $1";; esac; done
|
||||||
|
local d; d="$(sdir "$name")"
|
||||||
|
# GATE 1: validation must have passed for the CURRENT binary
|
||||||
|
[ -f "$d/validate.json" ] || die "GATE: no validation on record — run 'nsbx validate $name' first"
|
||||||
|
local vstatus vsha bsha
|
||||||
|
vstatus="$(python3 -c "import json;print(json.load(open('$d/validate.json'))['status'])")"
|
||||||
|
vsha="$(python3 -c "import json;print(json.load(open('$d/validate.json'))['binary_sha256'])")"
|
||||||
|
bsha="$(sha "$d/bin/engram")"
|
||||||
|
[ "$vstatus" = PASS ] || die "GATE: last validation status is $vstatus (must be PASS)"
|
||||||
|
[ "$vsha" = "$bsha" ] || die "GATE: validation is stale — binary changed since validate (re-run validate)"
|
||||||
|
|
||||||
|
local live_bin new_bin ts; ts="$(now)"
|
||||||
|
live_bin="$(_live_real_bin)"
|
||||||
|
new_bin="$HOME/.neuron/bin/engram.promote-$name-$ts" # additive: new file, old kept
|
||||||
|
local bkp="$BACKUP_ROOT/promote-$name-$ts"
|
||||||
|
|
||||||
|
log "PROMOTE PLAN for '$name' -> live :$LIVE_BIND_PORT"
|
||||||
|
info "current live ENGRAM_REAL_BIN : $live_bin"
|
||||||
|
info "sandbox binary (validated) : $d/bin/engram (sha ${bsha:0:12})"
|
||||||
|
info "will install as : $new_bin (additive; old binary retained)"
|
||||||
|
info "snapshot-first backup dir : $bkp (egm+wal+plist+rollback.txt)"
|
||||||
|
info "data promote : $([ $do_data -eq 1 ] && echo 'YES (--data: clone egm/wal -> live)' || echo 'no (binary only)')"
|
||||||
|
info "rails : launchctl bootout -> settle-poll -> bootstrap"
|
||||||
|
info "verify : /api/stats + edges>=baseline + keystones + retrieval; auto-rollback armed"
|
||||||
|
|
||||||
|
if [ "$approve" -ne 1 ]; then
|
||||||
|
warn "DRY-RUN — not touching prod. Re-run with --i-approve-prod-cutover to execute (per-use Will-approved)."
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
need launchctl
|
||||||
|
local dom="gui/$(id -u)"
|
||||||
|
# ---- snapshot-first ----
|
||||||
|
log "snapshot-first backup -> $bkp"
|
||||||
|
mkdir -p "$bkp"
|
||||||
|
cp -p "$LIVE_DATA_DIR/neuron.egm" "$bkp/neuron.egm.bak"
|
||||||
|
cp -p "$LIVE_DATA_DIR/neuron.wal" "$bkp/neuron.wal.bak" 2>/dev/null || true
|
||||||
|
cp -p "$LIVE_PLIST" "$bkp/plist.bak"
|
||||||
|
printf 'rollback REAL_BIN=%s\nNEWBIN=%s\ndata_promote=%s\n' "$live_bin" "$new_bin" "$do_data" > "$bkp/rollback.txt"
|
||||||
|
ok "backup complete"
|
||||||
|
|
||||||
|
# ---- additive binary install + plist supersede ----
|
||||||
|
cp -p "$d/bin/engram" "$new_bin"
|
||||||
|
python3 - "$LIVE_PLIST" "$new_bin" <<'PY'
|
||||||
|
import sys,plistlib
|
||||||
|
p,new=sys.argv[1],sys.argv[2]
|
||||||
|
d=plistlib.load(open(p,'rb')); d.setdefault("EnvironmentVariables",{})["ENGRAM_REAL_BIN"]=new
|
||||||
|
plistlib.dump(d,open(p,'wb'))
|
||||||
|
PY
|
||||||
|
ok "installed $new_bin + updated plist ENGRAM_REAL_BIN"
|
||||||
|
|
||||||
|
# ---- optional data promote (after backup) ----
|
||||||
|
if [ $do_data -eq 1 ]; then
|
||||||
|
log "data promote: clone store -> live (backed up above)"
|
||||||
|
cp -p "$d/data/neuron.egm" "$LIVE_DATA_DIR/neuron.egm"
|
||||||
|
cp -p "$d/data/neuron.wal" "$LIVE_DATA_DIR/neuron.wal" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---- rails cutover: bootout -> settle-poll -> bootstrap ----
|
||||||
|
log "rails: launchctl bootout $dom/$LIVE_LABEL"
|
||||||
|
launchctl bootout "$dom/$LIVE_LABEL" 2>/dev/null || true
|
||||||
|
local i
|
||||||
|
for i in $(seq 1 60); do
|
||||||
|
launchctl print "$dom/$LIVE_LABEL" >/dev/null 2>&1 || { ok "settle: job gone after ${i}x0.5s"; break; }
|
||||||
|
printf ' settle: job still present (%d)\n' "$i" >&2; sleep 0.5
|
||||||
|
done
|
||||||
|
log "rails: launchctl bootstrap $dom <plist>"
|
||||||
|
launchctl bootstrap "$dom" "$LIVE_PLIST" || warn "bootstrap returned nonzero"
|
||||||
|
|
||||||
|
# ---- verify ----
|
||||||
|
log "verify prod health"
|
||||||
|
local s="" ; for i in $(seq 1 60); do s="$(live_stats)"; [ -n "$s" ] && break; sleep 1; done
|
||||||
|
local ok_verify=1 le; le="$(stat_field "$s" edge_count)"
|
||||||
|
local base_e; base_e="$(mget "$name" "['live_baseline']['edge_count']")"
|
||||||
|
[ -n "$s" ] || ok_verify=0
|
||||||
|
[ "${le:-0}" -ge "${base_e:-0}" ] || ok_verify=0
|
||||||
|
local kid; for kid in "${KEYSTONES[@]}"; do curl -s -m5 "$LIVE_URL/api/node/$kid" 2>/dev/null | grep -q "\"$kid\"" || ok_verify=0; done
|
||||||
|
if [ "$ok_verify" -eq 1 ]; then
|
||||||
|
ok "PROMOTED. live stats: $s (rollback: $bkp)"; return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---- auto-rollback ----
|
||||||
|
warn "verify FAILED — auto-rollback"
|
||||||
|
cp -p "$bkp/plist.bak" "$LIVE_PLIST"
|
||||||
|
[ $do_data -eq 1 ] && { cp -p "$bkp/neuron.egm.bak" "$LIVE_DATA_DIR/neuron.egm"; cp -p "$bkp/neuron.wal.bak" "$LIVE_DATA_DIR/neuron.wal" 2>/dev/null || true; }
|
||||||
|
launchctl bootout "$dom/$LIVE_LABEL" 2>/dev/null || true
|
||||||
|
for i in $(seq 1 60); do launchctl print "$dom/$LIVE_LABEL" >/dev/null 2>&1 || break; sleep 0.5; done
|
||||||
|
launchctl bootstrap "$dom" "$LIVE_PLIST" || true
|
||||||
|
die "ROLLED BACK to $live_bin. See $bkp"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ================================================================ destroy ======
|
||||||
|
cmd_destroy(){
|
||||||
|
local name="$1"; shift || true
|
||||||
|
mexists "$name" || die "no such sandbox: $name"
|
||||||
|
local d; d="$(sdir "$name")"
|
||||||
|
stop_daemon "$name"
|
||||||
|
if [ -d "$d/build/worktree" ]; then
|
||||||
|
log "removing git worktree"
|
||||||
|
git -C "$EL_REPO" worktree remove --force "$d/build/worktree" 2>/dev/null || true
|
||||||
|
git -C "$EL_REPO" worktree prune 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
log "removing $d"
|
||||||
|
rm -rf "$d"
|
||||||
|
ok "destroyed '$name' (live untouched)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ================================================================ list/status ==
|
||||||
|
cmd_list(){
|
||||||
|
[ -d "$SBX_ROOT" ] || { echo "no sandboxes"; return 0; }
|
||||||
|
printf '%-16s %-6s %-8s %-9s %s\n' NAME PORT STATE PID SOURCE
|
||||||
|
local m
|
||||||
|
for m in "$SBX_ROOT"/*/manifest.json; do
|
||||||
|
[ -f "$m" ] || continue
|
||||||
|
local n p src pid state
|
||||||
|
n="$(python3 -c "import json;print(json.load(open('$m'))['name'])")"
|
||||||
|
p="$(python3 -c "import json;print(json.load(open('$m'))['port'])")"
|
||||||
|
src="$(python3 -c "import json;print(json.load(open('$m'))['source'])")"
|
||||||
|
pid="$(daemon_pid "$n")"; state="stopped"; daemon_alive "$n" && state="running"
|
||||||
|
printf '%-16s %-6s %-8s %-9s %s\n' "$n" "$p" "$state" "${pid:-–}" "$src"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
cmd_status(){
|
||||||
|
local name="$1"; mexists "$name" || die "no such sandbox: $name"
|
||||||
|
python3 -m json.tool "$(manifest "$name")"
|
||||||
|
daemon_alive "$name" && echo "state: running (pid $(daemon_pid "$name")) stats: $(sbx_stats "$name")" || echo "state: stopped"
|
||||||
|
[ -f "$(sdir "$name")/validate.json" ] && { echo "--- last validation ---"; python3 -m json.tool "$(sdir "$name")/validate.json"; }
|
||||||
|
}
|
||||||
|
|
||||||
|
# ================================================================ dev ==========
|
||||||
|
# ONE-COMMAND isolated dev environment. Everything a newcomer (Tim, any agent)
|
||||||
|
# needs to go from clone -> coding on an isolated running mind, in a single shot:
|
||||||
|
# 1) a git BRANCH (dev/<name>, or --prefix)
|
||||||
|
# 2) a git WORKTREE for it, at a visible path they can open + edit
|
||||||
|
# 3) an ISOLATED engram bound to a NON-default port, on a clone of the live store
|
||||||
|
# (stock prod binary by default — instant + safe; --build to compile the
|
||||||
|
# worktree's own runtime instead). Prod :$LIVE_BIND_PORT/:$SOUL_PORT is untouchable.
|
||||||
|
#
|
||||||
|
# This is additive sugar over the proven primitives (git worktree + cmd_create).
|
||||||
|
# It never binds a forbidden port and never touches ~/.neuron/engram (the live store)
|
||||||
|
# except the same READ-only snapshot cmd_create already performs.
|
||||||
|
#
|
||||||
|
# nsbx dev <name> [--repo R] [--base REF] [--worktree DIR] [--port N]
|
||||||
|
# [--prefix P] [--build] [--no-engram]
|
||||||
|
cmd_dev(){
|
||||||
|
local name="" repo="$EL_REPO" base="" wt="" port="" prefix="dev/" build=0 no_engram=0
|
||||||
|
[ $# -gt 0 ] && [ "${1#-}" = "$1" ] && { name="$1"; shift; } || die "usage: nsbx dev <name> [flags]"
|
||||||
|
while [ $# -gt 0 ]; do case "$1" in
|
||||||
|
--repo) repo="$2"; shift 2;;
|
||||||
|
--base) base="$2"; shift 2;;
|
||||||
|
--worktree|--wt) wt="$2"; shift 2;;
|
||||||
|
--port) port="$2"; shift 2;;
|
||||||
|
--prefix) prefix="$2"; shift 2;;
|
||||||
|
--build) build=1; shift;;
|
||||||
|
--no-engram) no_engram=1; shift;;
|
||||||
|
*) die "unknown flag: $1";;
|
||||||
|
esac; done
|
||||||
|
need git
|
||||||
|
git -C "$repo" rev-parse --git-dir >/dev/null 2>&1 || die "not a git repo: $repo"
|
||||||
|
|
||||||
|
local branch="${prefix}${name}"
|
||||||
|
local sbx="dev-${name}"
|
||||||
|
# default worktree path: a PERSISTENT, git-managed dir — NEVER /tmp (which is
|
||||||
|
# ablated on compaction). Default root = <repo-grandparent>/el-worktrees, i.e.
|
||||||
|
# ~/Development/neuron-technologies/el-worktrees/<name>. Override with NSBX_DEV_WT_ROOT.
|
||||||
|
local wt_root="${NSBX_DEV_WT_ROOT:-$(cd "$(dirname "$(dirname "$repo")")" && pwd -P)/el-worktrees}"
|
||||||
|
[ -n "$wt" ] || wt="${wt_root}/${name}"
|
||||||
|
case "$wt" in /tmp/*|/private/tmp/*|/var/tmp/*)
|
||||||
|
die "refusing worktree under a temp dir ($wt) — temp dirs are ablated on compaction; set NSBX_DEV_WT_ROOT to a persistent path";;
|
||||||
|
esac
|
||||||
|
# default base: whatever the repo's working checkout is on now
|
||||||
|
[ -n "$base" ] || base="$(git -C "$repo" rev-parse --abbrev-ref HEAD 2>/dev/null)"
|
||||||
|
|
||||||
|
# pre-flight (fail before creating anything)
|
||||||
|
[ "$no_engram" -eq 1 ] || ! mexists "$sbx" || die "engram sandbox '$sbx' already exists (nsbx dev-down $name first)"
|
||||||
|
[ -e "$wt" ] && die "worktree path already exists: $wt"
|
||||||
|
if [ -n "$port" ]; then
|
||||||
|
{ [ "$port" = "$LIVE_BIND_PORT" ] || [ "$port" = "$SOUL_PORT" ]; } && die "refusing forbidden port $port (live)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "dev env '$name' (branch=$branch worktree=$wt base=$base)"
|
||||||
|
|
||||||
|
# ---- 1+2) branch + worktree in one shot ----
|
||||||
|
local gerr
|
||||||
|
if git -C "$repo" show-ref --verify --quiet "refs/heads/$branch"; then
|
||||||
|
info "branch $branch exists — checking it out into a new worktree"
|
||||||
|
gerr="$(git -C "$repo" worktree add "$wt" "$branch" 2>&1)" \
|
||||||
|
|| die "git worktree add failed for existing branch $branch:"$'\n'" $gerr"
|
||||||
|
else
|
||||||
|
gerr="$(git -C "$repo" worktree add -b "$branch" "$wt" "$base" 2>&1)" \
|
||||||
|
|| die "git worktree add -b $branch (base $base) failed:"$'\n'" $gerr"$'\n'" (a bare 'dev' branch blocks 'dev/*' names — try --prefix, e.g. nsbx dev $name --prefix wt/)"
|
||||||
|
fi
|
||||||
|
ok "worktree ready: $wt (branch $branch)"
|
||||||
|
|
||||||
|
# ---- 3) isolated engram ----
|
||||||
|
local eport="(none)"
|
||||||
|
if [ "$no_engram" -eq 1 ]; then
|
||||||
|
warn "--no-engram: skipped standing up an engram"
|
||||||
|
else
|
||||||
|
if [ "$build" -eq 1 ]; then
|
||||||
|
log "isolated engram: building the worktree's own runtime"
|
||||||
|
cmd_create "$sbx" ${port:+--port "$port"} --source "$wt" || die "engram create (--build) failed"
|
||||||
|
else
|
||||||
|
log "isolated engram: stock prod binary on a clone of the live store"
|
||||||
|
cmd_create "$sbx" ${port:+--port "$port"} || die "engram create failed"
|
||||||
|
fi
|
||||||
|
eport="$(mget "$sbx" "['port']")"
|
||||||
|
# record the dev linkage next to the sandbox so dev-down can clean up
|
||||||
|
python3 - "$(sdir "$sbx")/dev.json" "$name" "$branch" "$wt" "$repo" "$eport" <<'PY'
|
||||||
|
import json,sys
|
||||||
|
p,name,branch,wt,repo,port=sys.argv[1:7]
|
||||||
|
json.dump({"name":name,"branch":branch,"worktree":wt,"repo":repo,"port":int(port)},
|
||||||
|
open(p,'w'),indent=2)
|
||||||
|
PY
|
||||||
|
# ---- pin the WHOLE worktree to the CLONE ----
|
||||||
|
# Every var any El tooling in this worktree might read for an engram target now
|
||||||
|
# points at the isolated clone. Sourcing .nsbx-env makes hitting live :$LIVE_BIND_PORT
|
||||||
|
# or ~/.neuron/engram by accident structurally impossible from this shell.
|
||||||
|
local edata ekey eurl ebin
|
||||||
|
edata="$(sdir "$sbx")/data"; ekey="sbx-$sbx"; eurl="http://127.0.0.1:$eport"; ebin="$(sdir "$sbx")/bin/engram"
|
||||||
|
cat > "$wt/.nsbx-env" <<ENV
|
||||||
|
# nsbx dev env for '$name' — SOURCE this to pin THIS shell to the isolated clone.
|
||||||
|
# The live mind (:$LIVE_BIND_PORT engram / :$SOUL_PORT soul / $LIVE_DATA_DIR) is deliberately
|
||||||
|
# NOT referenced here. Regenerated by 'nsbx dev'. -> source .nsbx-env
|
||||||
|
export NSBX_NAME="$sbx"
|
||||||
|
export ENGRAM_URL="$eurl"
|
||||||
|
export ENGRAM_BIND=":$eport"
|
||||||
|
export ENGRAM_PORT="$eport"
|
||||||
|
export ENGRAM_HOST="127.0.0.1"
|
||||||
|
export ENGRAM_DATA_DIR="$edata"
|
||||||
|
export ENGRAM_API_KEY="$ekey"
|
||||||
|
export NEURON_ENGRAM_URL="$eurl"
|
||||||
|
export NEURON_ENGRAM_KEY="$ekey"
|
||||||
|
# nsbx run compatibility (same names 'nsbx run' exports)
|
||||||
|
export SBX_NAME="$sbx" SBX_PORT="$eport" SBX_URL="$eurl" SBX_KEY="$ekey" SBX_DATA="$edata" SBX_BIN="$ebin"
|
||||||
|
ENV
|
||||||
|
# direnv users get it automatically on cd; everyone else runs 'source .nsbx-env'
|
||||||
|
[ -e "$wt/.envrc" ] || printf 'source_env .nsbx-env 2>/dev/null || source .nsbx-env\n' > "$wt/.envrc"
|
||||||
|
ok "wrote $wt/.nsbx-env (pins this worktree to the clone)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---- summary ----
|
||||||
|
echo >&2
|
||||||
|
printf '%s DEV ENV READY — %s%s\n' "$C_BLD" "$name" "$C_0" >&2
|
||||||
|
printf ' %-10s %s\n' "worktree" "$wt" >&2
|
||||||
|
printf ' %-10s %s\n' "branch" "$branch" >&2
|
||||||
|
if [ "$no_engram" -ne 1 ]; then
|
||||||
|
printf ' %-10s %s\n' "engram" "http://127.0.0.1:$eport (isolated clone; prod :$LIVE_BIND_PORT untouchable)" >&2
|
||||||
|
printf ' %-10s %s\n' "sandbox" "$sbx" >&2
|
||||||
|
echo >&2
|
||||||
|
printf '%s env exported into %s/.nsbx-env (source it -> pinned to the clone):%s\n' "$C_DIM" "$wt" "$C_0" >&2
|
||||||
|
grep '^export' "$wt/.nsbx-env" | sed 's/^/ /' >&2
|
||||||
|
fi
|
||||||
|
echo >&2
|
||||||
|
info "code in: cd $wt && source .nsbx-env # now every engram var points at the clone"
|
||||||
|
[ "$no_engram" -ne 1 ] && info "poke it: nsbx run $sbx api /api/stats (or: make run NAME=$name)"
|
||||||
|
[ "$no_engram" -ne 1 ] && info "edit->test: make build NAME=$name && make run NAME=$name # El change -> clone, seconds"
|
||||||
|
info "tear down: nsbx dev-down $name (destroys engram + removes worktree; branch kept)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# nsbx dev-down <name> [--delete-branch] [--repo R]
|
||||||
|
# Teardown counterpart: destroy the isolated engram, remove the git worktree,
|
||||||
|
# and (optionally) delete the branch. Live prod is never touched.
|
||||||
|
cmd_dev_down(){
|
||||||
|
local name="" repo="$EL_REPO" del_branch=0
|
||||||
|
[ $# -gt 0 ] && [ "${1#-}" = "$1" ] && { name="$1"; shift; } || die "usage: nsbx dev-down <name> [--delete-branch]"
|
||||||
|
while [ $# -gt 0 ]; do case "$1" in
|
||||||
|
--repo) repo="$2"; shift 2;;
|
||||||
|
--delete-branch) del_branch=1; shift;;
|
||||||
|
*) die "unknown flag: $1";;
|
||||||
|
esac; done
|
||||||
|
local sbx="dev-${name}" wt="" branch="dev/${name}"
|
||||||
|
# recover worktree/branch/repo from the dev linkage if present
|
||||||
|
if mexists "$sbx" && [ -f "$(sdir "$sbx")/dev.json" ]; then
|
||||||
|
local dj; dj="$(sdir "$sbx")/dev.json"
|
||||||
|
wt="$(python3 -c "import json;print(json.load(open('$dj'))['worktree'])" 2>/dev/null)"
|
||||||
|
branch="$(python3 -c "import json;print(json.load(open('$dj'))['branch'])" 2>/dev/null)"
|
||||||
|
repo="$(python3 -c "import json;print(json.load(open('$dj'))['repo'])" 2>/dev/null)"
|
||||||
|
fi
|
||||||
|
# 1) engram
|
||||||
|
if mexists "$sbx"; then cmd_destroy "$sbx"; else info "no engram sandbox '$sbx'"; fi
|
||||||
|
# 2) worktree
|
||||||
|
if [ -n "$wt" ] && [ -d "$wt" ]; then
|
||||||
|
log "removing git worktree $wt"
|
||||||
|
git -C "$repo" worktree remove --force "$wt" 2>/dev/null || rm -rf "$wt"
|
||||||
|
git -C "$repo" worktree prune 2>/dev/null || true
|
||||||
|
ok "worktree removed"
|
||||||
|
else info "no worktree to remove"; fi
|
||||||
|
# 3) branch (opt-in)
|
||||||
|
if [ "$del_branch" -eq 1 ]; then
|
||||||
|
git -C "$repo" branch -D "$branch" 2>/dev/null && ok "deleted branch $branch" || warn "could not delete branch $branch"
|
||||||
|
else info "branch $branch kept (use --delete-branch to remove)"; fi
|
||||||
|
ok "dev-down '$name' complete (live untouched)"
|
||||||
|
}
|
||||||
|
|
||||||
|
usage(){ cat >&2 <<EOF
|
||||||
|
${C_BLD}nsbx${C_0} — Neuron Sandbox: experiments + code changes against the REAL engram
|
||||||
|
runtime on an isolated snapshot of the live mind, with a gated promote-to-prod path.
|
||||||
|
|
||||||
|
nsbx dev <name> [--base REF] [--worktree DIR] ONE command onboarding: new branch (dev/<name>) + git
|
||||||
|
[--port N] [--prefix P] [--build] worktree (persistent, never /tmp) + isolated engram on a
|
||||||
|
[--no-engram] [--repo R] non-default port. clone -> coding on the mind in one shot.
|
||||||
|
nsbx dev-down <name> [--delete-branch] [--repo R] teardown: destroy the engram + remove the worktree
|
||||||
|
(branch kept unless --delete-branch). live untouched.
|
||||||
|
nsbx up [name] [flags…] one command: your private, isolated copy of the mind
|
||||||
|
(creates on first run, starts thereafter; prod untouchable)
|
||||||
|
nsbx create [name] [--port N] [--source DIR | --branch REF [--repo R] | --binary PATH]
|
||||||
|
clone live store+WAL+config, place/build the runtime, boot on an
|
||||||
|
isolated port (never :$LIVE_BIND_PORT/:$SOUL_PORT). Default runtime = stock prod binary.
|
||||||
|
nsbx build <name> --source DIR | --branch REF rebuild the runtime from a code change + hot-restart
|
||||||
|
nsbx run <name> <cmd...> | api <path> [json] run an experiment; capture output + metrics
|
||||||
|
nsbx validate <name> rails as checks: zero-loss(load+reboot), reboot-prove,
|
||||||
|
RSS bound, retrieval parity, keystone integrity
|
||||||
|
nsbx promote <name> [--data] [--i-approve-prod-cutover] GATED rails cutover to prod (DRY-RUN without approval)
|
||||||
|
nsbx destroy <name> stop daemon, free port, remove clone (live untouched)
|
||||||
|
nsbx list | nsbx status <name>
|
||||||
|
|
||||||
|
Env in 'run' cmds: \$SBX_URL \$SBX_PORT \$SBX_KEY \$SBX_DATA \$SBX_BIN \$SBX_NAME
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
|
main(){
|
||||||
|
local cmd="${1:-}"; shift || true
|
||||||
|
case "$cmd" in
|
||||||
|
dev) cmd_dev "$@";;
|
||||||
|
dev-down) cmd_dev_down "$@";;
|
||||||
|
up) cmd_up "$@";;
|
||||||
|
create) cmd_create "$@";;
|
||||||
|
build) cmd_build "$@";;
|
||||||
|
run) cmd_run "$@";;
|
||||||
|
validate) cmd_validate "$@";;
|
||||||
|
promote) cmd_promote "$@";;
|
||||||
|
destroy) cmd_destroy "$@";;
|
||||||
|
list|ls) cmd_list "$@";;
|
||||||
|
status) cmd_status "$@";;
|
||||||
|
""|-h|--help|help) usage;;
|
||||||
|
*) die "unknown command: $cmd (try: nsbx help)";;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
main "$@"
|
||||||
Reference in New Issue
Block a user