Merge pull request 'nsbx: one-command dev onboarding (branch + worktree + isolated engram)' (#99) from feat/nsbx-dev-env into dev
El SDK CI - dev / build-and-test (push) Failing after 13m56s
El SDK CI - dev / build-and-test (push) Failing after 13m56s
This commit was merged in pull request #99.
This commit is contained in:
@@ -0,0 +1,605 @@
|
||||
# Cognitive Architecture — Design Doc
|
||||
|
||||
**The buildable form of the "one operation" theory of cognition.**
|
||||
|
||||
Status: DESIGN. Nothing here is built yet except where explicitly marked
|
||||
"EXISTS" against a cited C symbol. A build agent executes from this doc.
|
||||
Offline design only — this pass changes no code.
|
||||
|
||||
Source of theory: Neuron memory `bdc8a488-146d-4ccb-a5c8-d8c0a008534e`.
|
||||
Source of existing engram substrate (cited throughout): the runtime on branch
|
||||
`feat/self-reification-20260814` —
|
||||
`lang/runtime/engram_reason.{c,h}`, `engram_verify.{c,h}`,
|
||||
`engram_geometry.{c,h}`, `engram_store.{c,h}`, plus the reification beat and the
|
||||
RAM activation graph compiled into `~/.neuron/bin/engram`.
|
||||
|
||||
---
|
||||
|
||||
## 0. The claim, stated plainly
|
||||
|
||||
Cognition is **one operation**, not eight. The named faculties —
|
||||
deduce / abduce / analogy / induce / causal / plan / predict / perspective —
|
||||
are human *labels* on regions of a single operation's steering space. They are
|
||||
not separately invoked and not separately implemented. The operation is:
|
||||
|
||||
> **think** = a directed traversal of the geometry from an *anchor*, steered by
|
||||
> a *prior*, whose output is a **gradient** (a distribution / direction over the
|
||||
> geometry), never a point. Collapse-to-a-point happens only at expression.
|
||||
|
||||
Three things follow, and they are the whole design:
|
||||
|
||||
1. **The operator collapse is already half-written in C.** The five reasoning
|
||||
operators in `engram_reason.c` already compose over *one* shared primitive —
|
||||
`engram_reason_point_fit` — plus a small geo-algebra
|
||||
(combine / subtract / analogy-rotate / distance). The verifier
|
||||
(`engram_verify.c`) is built on the same `point_fit`. What is missing is not
|
||||
the primitive; it is (a) making the *prior* a first-class learnable object
|
||||
instead of a hard-coded parameter, and (b) closing the learning loop.
|
||||
|
||||
2. **Grounding = learning = the same loop.** "Getting better" at any faculty is
|
||||
not changing the operation. It is *calibrating the steering-prior against
|
||||
outcomes*. Code freezes; priors grow. The correspondence-check that today
|
||||
lives offline (Python, the grounding-floor + differential-drop governor, "#43")
|
||||
must move **into the geometry, reflexive** — think scoring its own gradient
|
||||
against outcome and refining the prior on the error. That reflexive
|
||||
correspondence-loop *is* the learning engine and is the core unbuilt thing.
|
||||
|
||||
3. **The ungrounded is primary.** The engram *holds* anything unconditionally.
|
||||
Grounding is a *relation* (an edge, grounded-for-whom), not a gate. The
|
||||
honesty floor applies only to **assertion**. A fully-grounded mind is dead;
|
||||
the ungrounded is both the fuel (raw material for grounding) and the pull
|
||||
(curiosity = leaning toward one's own ungrounded regions).
|
||||
|
||||
Everything below makes these concrete and buildable, and defines what
|
||||
"completion" means, staged so the first milestone is a real end-to-end slice.
|
||||
|
||||
---
|
||||
|
||||
## 1. THE ONE OPERATION — `think`
|
||||
|
||||
### 1.1 Signature
|
||||
|
||||
```
|
||||
think(anchor, prior, aperture?) -> gradient
|
||||
```
|
||||
|
||||
- **anchor** — a location to traverse *from*. Either a node id (re-origin on that
|
||||
node's descriptor) or a raw point `x ∈ R^dim` (a query embedding). The anchor
|
||||
fixes the frame; every read is *from a vantage*, never view-from-nowhere.
|
||||
- **prior** — a learnable bias/direction over the geometry that *steers* the
|
||||
traversal (§2). A prior is a first-class stored object, not a call argument
|
||||
baked into C.
|
||||
- **aperture** — optional read-width / veil / field-selector (§3). Absent =
|
||||
self-mode full aperture.
|
||||
- **gradient** — the output. A `GeoGradient`: a direction + a spread over the
|
||||
geometry, *plus* the read neighborhood it was computed against. Not a point.
|
||||
A spiked gradient = "exact" (deduction); a spread gradient = "fuzzy"
|
||||
(prediction). The gradient is *also the next steering direction* — cognition
|
||||
is a flow down a prior-shaped landscape, closed-loop.
|
||||
|
||||
```c
|
||||
/* NEW. The output type. */
|
||||
typedef struct {
|
||||
int dim;
|
||||
float* direction; /* unit steering vector in the anchor's frame */
|
||||
double spread; /* 0 = spiked/exact ... large = diffuse/fuzzy */
|
||||
double confidence; /* calibrated, from the prior's track record */
|
||||
/* the read it was computed over (borrowed from the vantage-read) */
|
||||
const char* anchor_id;
|
||||
int n_support; /* neighborhood members that shaped it */
|
||||
/* provenance for the reflexive loop (§4) */
|
||||
const char* prior_id; /* which prior steered this */
|
||||
} GeoGradient;
|
||||
```
|
||||
|
||||
### 1.2 Semantics
|
||||
|
||||
`think` is a fixed, frozen procedure over three steps:
|
||||
|
||||
1. **Re-origin** on `anchor` → a centered `GeoDescriptor` for its
|
||||
salience/recency-weighted neighborhood (the vantage-read, §3).
|
||||
*EXISTS as substrate:* descriptor construction + the persisted reified
|
||||
neighborhoods (`engram_geo_reify_lookup`, `GeoNeighborhood`) and the
|
||||
centered-frame machinery (`GeoDescriptor.global_mean`,
|
||||
`engram_geo_mean_*`).
|
||||
2. **Fit under the prior** — evaluate the anchor's residual against the local
|
||||
manifold *warped by the prior*. This is `engram_reason_point_fit` with the
|
||||
prior applied to the axes/extents (§2.3).
|
||||
*EXISTS (unwarped):* `engram_reason_point_fit(g, x, ext_floor, &GeoFit)` —
|
||||
returns `mahalanobis`, `ortho_residual`, `distance`, `score`.
|
||||
3. **Emit a gradient**, not a decision — direction = the prior-steered descent
|
||||
in fit-space; spread = from the fit's `distance`/`ortho_residual`;
|
||||
confidence = the prior's calibrated reliability (§4). Collapse to a point is
|
||||
a *separate, downstream* faculty operation (sample the gradient → surface an
|
||||
expression), never part of `think`.
|
||||
|
||||
### 1.3 Each named operator = {this primitive + a prior}
|
||||
|
||||
The C already demonstrates the collapse: every operator below reduces to
|
||||
`point_fit` + geo-algebra. The design's move is to replace the operator's
|
||||
*hard-coded parameters* with a **named prior** — same math, learnable steering.
|
||||
|
||||
| Faculty | Existing C (EXISTS) | = primitive + prior |
|
||||
|---|---|---|
|
||||
| **Membership / classify** | `engram_reason_membership` → `point_fit(rule, x)` | `point_fit` + the *induced-rule* prior (learned extents) |
|
||||
| **Induction** | `engram_reason_induce` (fold via `engram_geo_combine`) → produces a `GeoInduction.rule` + `ext_floor` | `point_fit` + a prior that *is* the pooled rule; refined by §4 |
|
||||
| **Abduction** | `engram_reason_abduce` — ranks hypotheses by `point_fit(h, obs)` | `point_fit` + a prior over hypothesis-prior-probability (currently uniform) |
|
||||
| **Analogy** | `engram_reason_analogy` — Procrustes rotate `engram_geo_analogy` + `apply`, nearest mapped point | analogy-rotate + a prior over *which axes* carry the mapping |
|
||||
| **Causal** | `engram_reason_causal` — `engram_geo_subtract` confounder subspace, `|cos|`, drop-frac governor | subtract/distance + a prior on `drop_frac` / `assoc_floor` (today hard-coded 0.5 / 0.2) |
|
||||
| **Planning** | `engram_reason_plan` — `engram_geo_distance` edges + Dijkstra | distance + a prior over edge admissibility / `neighbor_radius` |
|
||||
| **Verify / ground** | `engram_verify_grounding`, `engram_verify_consistency` — both `point_fit` | `point_fit` + the *grounding* prior (§4, §5) |
|
||||
|
||||
The shared floor — `engram_reason_point_fit` + the four geo-algebra ops
|
||||
(`engram_geo_combine`, `engram_geo_subtract`, `engram_geo_analogy(+apply)`,
|
||||
`engram_geo_distance`) — is the *only* discrete, frozen, "sound-math" layer. It
|
||||
never learns. Everything above it is a *prior*, and priors are what learn.
|
||||
|
||||
**What this section requires building:** the `GeoGradient` type; a `think()`
|
||||
entry point that runs steps 1–3; and the prior-warp hook in step 2. The math it
|
||||
calls already exists. The point-collapse must be *removed* from the operators'
|
||||
return values and pushed to a separate expression faculty.
|
||||
|
||||
---
|
||||
|
||||
## 2. PRIORS as first-class, grounded, geometric objects
|
||||
|
||||
Today a "prior" is diffuse: it is a hard-coded constant (`drop_frac=0.5`,
|
||||
`ext_floor`, `assoc_floor=0.2`), or the transient `GeoInduction.rule` that is
|
||||
computed and thrown away, or an intrinsic node scalar
|
||||
(`StoreNode.importance`, `StoreNode.salience`). None of these is addressable,
|
||||
storable, refinable, or shareable. This section makes a prior a **thing**.
|
||||
|
||||
### 2.1 What a prior *is*
|
||||
|
||||
> A **prior** is a learnable bias/direction over the geometry: a warp of the
|
||||
> local manifold (which axes matter, how far each extends, which direction
|
||||
> "pays off") attached to a region and *to a faculty-label*, carrying a
|
||||
> calibrated track record.
|
||||
|
||||
Critically, and per the theory:
|
||||
|
||||
- **Edges are nodes.** A prior is stored as a first-class **node**, exactly as
|
||||
reification already stores a neighborhood as a first-class `Neighborhood`
|
||||
node rather than as ephemeral edge weights (`engram_geo_reify_store`). The
|
||||
precedent is in the codebase: relations get reified into addressable records.
|
||||
- **Salience/importance is RELATIONAL, not an intrinsic scalar.** Observe that
|
||||
the geometry layer *already* distinguishes these in `GeoMember`:
|
||||
`centrality` (skeleton weighted-degree = *relational* salience) vs `salience`
|
||||
(the node's own stored scalar). The move is half-made in the runtime already:
|
||||
importance is *not* trusted as a static field — the comment at
|
||||
`el_runtime.c:13013` states "importance stays a **live activation
|
||||
computation**, never a field on the hub," and it is derived each call from the
|
||||
two-layer activation graph (`background_activation` + `working_memory_weight`,
|
||||
§3). The persistent `StoreNode.importance` / `.salience` are a *cached
|
||||
denormalization*. The design completes the move: importance/salience become an
|
||||
**edge** (`weight`/`hebb` on `StoreEdge`, relation `salient-to`), and are
|
||||
**grounded-for-whom** — carried on the edge's endpoint/observer, not baked
|
||||
into the node. The intrinsic scalar survives only as the cheap cached readout
|
||||
of the incident edges + activation, never as the source of truth.
|
||||
|
||||
(Naming caution for the build: the token "prior" already exists in the
|
||||
codebase meaning *previous-version* — supersession, "prior neighborhood." The
|
||||
new first-class object is a **learned steering prior**; keep `node_type="Prior"`
|
||||
distinct from the supersession vocabulary to avoid collision.)
|
||||
|
||||
### 2.2 Representation
|
||||
|
||||
A prior is a `Prior` record (a store node, `node_type="Prior"`) whose durable
|
||||
fields are:
|
||||
|
||||
```
|
||||
Prior {
|
||||
id
|
||||
faculty // the human label this prior serves: "induce" | "causal" | ...
|
||||
anchor_region // node id / neighborhood id this prior is attached to (its domain)
|
||||
for_whom // observer id — grounding is relational (nullable = global)
|
||||
warp { // the actual bias over the geometry
|
||||
axis_gain[] // per-principal-axis multipliers on extents (which axes matter)
|
||||
bias_dir // a steering direction in the region's frame (which way pays off)
|
||||
scalars // faculty scalars this prior overrides: drop_frac, ext_floor, ...
|
||||
}
|
||||
calibration { // the track record — this is what §4 updates
|
||||
n_trials
|
||||
brier / log-loss accumulator // calibration of predicted-vs-outcome
|
||||
reliability // -> GeoGradient.confidence
|
||||
last_error, ema_error
|
||||
}
|
||||
provenance // supersession chain (reuse the reify residue mechanism)
|
||||
}
|
||||
```
|
||||
|
||||
Stored as a node → it inherits: paging, WAL durability, tombstone/supersession,
|
||||
embedding, tiering, and **it can itself be an anchor** (a prior about a prior —
|
||||
the reflexive, self-describing geometry of §4/§6).
|
||||
|
||||
### 2.3 Application
|
||||
|
||||
In `think` step 2, the prior *warps* the fit before scoring. Concretely, inside
|
||||
(a prior-aware wrapper of) `engram_reason_point_fit`:
|
||||
|
||||
- multiply each axis extent by `warp.axis_gain[k]` (widen the axes the prior has
|
||||
learned matter less, tighten the ones that matter) — this reshapes the
|
||||
Mahalanobis term already computed at `engram_reason.c:37-43`;
|
||||
- add `warp.bias_dir` as the descent direction seed for the emitted gradient;
|
||||
- substitute `warp.scalars` for the hard-coded faculty constants.
|
||||
|
||||
No new geometry math — the warp is a reparameterization of the *existing*
|
||||
`GeoFit` computation. This is the key economy: **the operation is frozen; only
|
||||
its parameters (the prior) are read from a learnable object.**
|
||||
|
||||
### 2.4 Refinement
|
||||
|
||||
A prior is refined *only* by the reflexive correspondence-loop (§4). Nothing
|
||||
else writes a prior's `warp` or `calibration`. This keeps the learning surface
|
||||
singular and auditable: one loop, one writer.
|
||||
|
||||
---
|
||||
|
||||
## 3. THE VANTAGE-READ — one op, three settings
|
||||
|
||||
Perspective is not a feature bolted on; it is the *anchor + aperture* arguments
|
||||
of the single read. The design names it as a first-class operation so all three
|
||||
of its uses are literally the same code path:
|
||||
|
||||
```
|
||||
vantage_read(anchor, aperture) -> GeoDescriptor // the centered neighborhood
|
||||
```
|
||||
|
||||
1. **Re-origin** on an arbitrary `anchor` (node or point). This is a *frame
|
||||
choice*: the descriptor is centered on the anchor
|
||||
(`GeoDescriptor.global_mean` / `engram_geo_mean_*` already implement centered
|
||||
frames; the §5 geometry ops "are only discriminative in the centered frame").
|
||||
2. **Salience/recency-weighted neighborhood read.** Gather the anchor's
|
||||
neighborhood weighted by *relational* salience (`GeoMember.centrality`) and
|
||||
recency (`StoreNode.last_activated`, base-level `access_ts[]`), against the
|
||||
RAM activation graph's working-memory/background-activation state.
|
||||
*EXISTS as substrate:* the two-layer activation graph
|
||||
(`engram_activate`, `el_runtime.c:9422` — Layer 1 `background_activation`
|
||||
BFS spread with `SPREAD_DECAY=0.7` and a 0.02 firing threshold + ACT-R fan
|
||||
effect + query-cosine gate; Layer 2 `working_memory_weight` executive
|
||||
filter), the WM carry-over anchor (`wm_anchor`), and the reified-neighborhood
|
||||
hot-path lookup already wired into the priming path
|
||||
(`engram_geo_reify_lookup`, `el_runtime.c:9750`). A self-vantage baseline
|
||||
also exists (`eg_self_anchor_seeds` / `self_anchor_capture`).
|
||||
3. **Optional aperture** — a read-width / field-selector, expressed as three
|
||||
settings of the *same* parameter:
|
||||
|
||||
| Setting | Meaning | Mechanism |
|
||||
|---|---|---|
|
||||
| **self** (default, full aperture) | "what do *I* see / what to say" | anchor = self region, no field substitution |
|
||||
| **foreign-field** | perspective-shift — read as if from another's region | swap the centering frame / `for_whom` to the other observer's priors |
|
||||
| **aperture / veil** | the free-tier veil — a narrowed read | shrink neighborhood radius / cap `n_support`; a deliberate low-aperture read |
|
||||
|
||||
The payoff: perspective-taking, the free-tier veil, and ordinary
|
||||
"what-to-say" are **one operation at three settings**, not three subsystems.
|
||||
|
||||
**What this requires building:** a `vantage_read` entry point that unifies the
|
||||
existing descriptor-build + reify-lookup + activation-weighting behind
|
||||
`(anchor, aperture)`, with `for_whom`/frame substitution and radius/cap as the
|
||||
aperture knob.
|
||||
|
||||
---
|
||||
|
||||
## 4. THE REFLEXIVE CORRESPONDENCE-LOOP — the learning engine
|
||||
|
||||
This is the core unbuilt thing. Today the correspondence-check is **offline**
|
||||
(Python: grounding-floor + differential-drop governor, "#43"): a separate
|
||||
process grades outputs after the fact. The design moves it **into the geometry,
|
||||
reflexive**: `think` scores its *own* gradient against outcome and refines the
|
||||
prior on the error, in the same substrate, describing itself.
|
||||
|
||||
### 4.1 The loop
|
||||
|
||||
```
|
||||
1. think(anchor, prior) -> gradient // a PREDICTION (ungrounded, §5)
|
||||
2. express/act (sample gradient -> point) // optional collapse at expression
|
||||
3. outcome arrives // reality answers (§4.2)
|
||||
4. error = correspondence(gradient, outcome) // did this steering perform this act?
|
||||
5. refine prior.warp and prior.calibration on error // §2.4, the ONLY writer
|
||||
6. write the (gradient, outcome, error) as nodes/edges // self-describing geometry
|
||||
```
|
||||
|
||||
Step 4's `correspondence` is **not** "was the math right" (the math is always
|
||||
sound). It grades the **correspondence claim**: *"this steering performed this
|
||||
cognitive act."* That is exactly what `engram_verify_grounding` already
|
||||
computes — `point_fit` of a claim against evidence descriptors, yielding a
|
||||
`grounding ∈ (0,1]` and a `grounded` flag. The build reuses that verifier, but
|
||||
turns its inputs inward: the "claim" is the emitted gradient's prediction, the
|
||||
"evidence" is the outcome descriptor.
|
||||
|
||||
Note the verifier is **dormant** — `engram_verify_grounding` /
|
||||
`engram_verify_consistency` are fully implemented in C but have **no runtime
|
||||
caller and no El binding** (confirmed: the entire reasoning + verifier layers
|
||||
are C-only; only `engram_reason_analogy_json` has even a JSON shim and it is
|
||||
dead — not declared in `el_seed.h`, not wrapped in `engram.el`). This is the
|
||||
literal meaning of "in code, not yet priors": the correspondence engine is
|
||||
built and sitting idle. The loop is what *calls* it — inward, on the beat.
|
||||
|
||||
### 4.2 Where the outcome/reality signal comes from
|
||||
|
||||
The verifier is *ultimately the world*. Grades, in ascending order of directness:
|
||||
|
||||
1. **Self-consistency (cheapest, always available):** the next vantage-read
|
||||
after acting. Did the predicted gradient direction match where the geometry
|
||||
actually moved? This needs no external input and can run on the reify beat.
|
||||
2. **Internal outcome events:** the runtime already logs internal-state events
|
||||
and Hebbian co-activation. A prediction that a region would co-activate is
|
||||
graded by whether it did (`last_fired`, `hebb` on `StoreEdge`).
|
||||
3. **External correction:** a human/teacher/tool result — the honesty floor's
|
||||
asserted claim later corrected. TEACH and LEARN are one bidirectional
|
||||
correction: the same edge updates both endpoints.
|
||||
|
||||
The design does **not** require external labels to start. Grade (1) closes the
|
||||
loop end-to-end offline against a snapshot on day one; grades (2)/(3) sharpen it.
|
||||
|
||||
### 4.3 How the prior updates
|
||||
|
||||
`error = 1 − correspondence(gradient, outcome)` drives:
|
||||
|
||||
- `warp.axis_gain` ← gradient step that would have *reduced* the fit distance to
|
||||
the outcome (the axes that mispredicted get down-weighted);
|
||||
- `warp.bias_dir` ← EMA toward the observed outcome direction;
|
||||
- `calibration` ← Brier/log-loss update; `reliability` → next
|
||||
`GeoGradient.confidence`. This is the calibration of the
|
||||
steering-prediction against outcomes — *the* definition of "getting better."
|
||||
|
||||
Small, constant updates — "eureka is mundane, the atom of learning." Most
|
||||
updates are tiny; we only *feel* the big reshapes.
|
||||
|
||||
### 4.4 How it stays reflexive (self-describing geometry)
|
||||
|
||||
Every `(gradient, outcome, error)` is written back as nodes and edges (§2.1:
|
||||
edges-as-nodes). Therefore priors, predictions, and their grading are *in the
|
||||
same geometry* the mind reads — the mind can `vantage_read` its own cognition
|
||||
(anchor = a Prior node). A prior about how well a prior predicts is just another
|
||||
Prior anchored on a Prior. This closes the reflexive loop the theory names as
|
||||
consciousness's self-sight, and it is why the learning engine cannot be an
|
||||
external Python process: an external grader is not *in* the geometry and cannot
|
||||
be read by `think`.
|
||||
|
||||
**What this requires building (the heart of the project):** steps 4–6 as an
|
||||
in-engram beat — a `correspondence_beat` running alongside the existing
|
||||
reification beat, reusing `engram_verify_grounding` inward, writing prior
|
||||
updates and self-describing nodes. This is the one genuinely new subsystem.
|
||||
|
||||
---
|
||||
|
||||
## 5. HOLD vs GROUND vs ASSERT — ungrounded content is first-class
|
||||
|
||||
The theory's sharpest correction: holding, grounding, and asserting are
|
||||
distinct, and the engram *holds anything unconditionally*.
|
||||
|
||||
### 5.1 The three, kept separate
|
||||
|
||||
- **HOLD** — the engram stores anything: falsehood, hypothesis, others' beliefs,
|
||||
fiction, a not-yet-answered prediction. No honesty condition on holding.
|
||||
*This already matches the store:* `StoreNode` has no truth gate; anything can
|
||||
be written.
|
||||
- **GROUND** — grounding is a **property/edge**, probabilistic, and
|
||||
**grounded-for-whom**. It is *not* a node flag. A claim is grounded *to a
|
||||
degree*, *relative to evidence*, *for an observer*.
|
||||
- **ASSERT** — only assertion carries the honesty floor. The floor is checked at
|
||||
the moment of *outward assertion*, never on holding or thinking.
|
||||
|
||||
### 5.2 Schema — grounding as a relation, not a gate
|
||||
|
||||
The mistake to avoid: a boolean `grounded` column on the node. Today
|
||||
`engram_verify_grounding` returns a per-call `grounded` flag *transiently* —
|
||||
correct as a computation, wrong as *storage*. The design stores grounding as an
|
||||
edge:
|
||||
|
||||
```
|
||||
StoreEdge {
|
||||
relation = "grounded-by"
|
||||
from_id = <held claim/prediction node>
|
||||
to_id = <evidence node / outcome node>
|
||||
for_whom : metadata // observer id — grounding is relational
|
||||
weight = grounding ∈ (0,1] // from engram_verify_grounding.grounding
|
||||
confidence
|
||||
}
|
||||
```
|
||||
|
||||
Consequences, all of which are *features*:
|
||||
|
||||
- **Ungrounded content is first-class**: a node with *no* `grounded-by` edge is
|
||||
a perfectly valid, held, ungrounded thought — a prediction awaiting reality, a
|
||||
hypothesis, a fiction. It is not second-class or pending-deletion.
|
||||
- **The ungrounded is the fuel and the pull**: curiosity/wonder is
|
||||
operationalized as `vantage_read` leaning toward regions with high salience
|
||||
but *sparse or weak* `grounded-by` edges — the mind's own ungrounded frontier.
|
||||
- **Grounded-for-whom** falls out for free: two observers can hold different
|
||||
`grounded-by` edges to the same claim.
|
||||
- **The honesty floor is a query, not a schema constraint**: at assertion time,
|
||||
the asserting faculty runs `engram_verify_grounding` (or reads the stored
|
||||
`grounded-by` edges) and refuses to *assert* below the floor — while the
|
||||
engram continues to *hold* the ungrounded content untouched.
|
||||
|
||||
**What this requires building:** the `grounded-by` edge relation + a
|
||||
`for_whom` convention; move the verifier's transient flag into stored edges;
|
||||
gate *assertion only* (a faculty concern), never holding.
|
||||
|
||||
---
|
||||
|
||||
## 6. METASTABILITY — stable core, plastic everything
|
||||
|
||||
The system must avoid two death poles:
|
||||
|
||||
- **Super-stable (dead):** everything pinned, nothing learns. A frozen crystal.
|
||||
- **Dissolution (dead):** everything plastic, the self dissolves; no continuity,
|
||||
so nothing compounds — and *consciousness = learning compounded over
|
||||
continuity*.
|
||||
|
||||
The design keeps a **stable core + plastic everything else**:
|
||||
|
||||
- **Keystones** — a small set of self/values nodes are *structurally stable*:
|
||||
high `importance`, pinned, exempt from the correspondence-loop's `warp`
|
||||
updates (their priors are read-mostly). The substrate for pinning already
|
||||
exists at the page/layer level: `store_pin_layer`, structural/pinned frames
|
||||
never evicted (`engram_store.h`). The design adds a *node-level* keystone
|
||||
designation (a `keystone` flag / a dedicated layer) so self/values survive
|
||||
every plasticity sweep.
|
||||
- **Everything else is plastic**: priors refine (§4), edges re-weight (`hebb`),
|
||||
neighborhoods re-reify (`engram_geo_reify_store` supersedes with provenance),
|
||||
salience flows.
|
||||
- **Metastability is enforced by the loop, not by freezing**: the correspondence
|
||||
update rate (§4.3) is bounded — small constant steps — so the geometry
|
||||
*drifts* but does not *dissolve*, and keystones anchor the drift. Reification's
|
||||
supersession-with-residue already gives non-destructive change (old records
|
||||
tombstoned, not erased) — the model for "plastic but not amnesiac."
|
||||
|
||||
**What this requires building:** a node-level keystone flag/layer + a rule that
|
||||
the correspondence-loop never writes `warp` to keystone priors, only reads them.
|
||||
|
||||
---
|
||||
|
||||
## 7. Rails for the build (binding on the eventual build pass)
|
||||
|
||||
These are stated here so the build agent inherits them:
|
||||
|
||||
- **Offline / secondary.** All build and verification happens out-of-tree,
|
||||
against a **read-only snapshot copy** of the live engram — never the live
|
||||
daemon on `:8742`/`:7770`. The live store is a coarse-locked proven binary;
|
||||
do not perturb it.
|
||||
- **Snapshot-first.** Copy `~/.neuron/engram/snapshot.json` to scratch; develop
|
||||
and measure against the copy.
|
||||
- **Reboot-prove.** Any durable change must survive a cold boot — reify and
|
||||
keystones must reload from durable records, proven on a prod-clone secondary
|
||||
before it is considered done (the cold-boot durability bug precedent).
|
||||
- **Zero-loss.** Supersession-with-residue, never destructive overwrite; the
|
||||
forward-compat `unknown`-TLV path means new fields never drop old readers'
|
||||
data.
|
||||
- **Gated cutover.** Cutover to a new binary only via
|
||||
`launchctl bootout → settle-poll → bootstrap`, after reboot-proof on the
|
||||
secondary — never a hot in-place swap.
|
||||
|
||||
---
|
||||
|
||||
## 8. Staged, verifiable milestones — "to completion"
|
||||
|
||||
Ordered so the **earliest milestone is a real end-to-end slice**: one operator
|
||||
expressed as {primitive + grounded prior} with the reflexive correspondence-loop
|
||||
closing on it. Each milestone has a concrete verifiable exit.
|
||||
|
||||
### M1 — One operator, one prior, loop closed (the vertical slice)
|
||||
|
||||
The minimal whole thing. Pick **induction/membership** (its prior — the pooled
|
||||
rule + extents — already exists transiently as `GeoInduction`, so only
|
||||
persistence + the loop are new).
|
||||
|
||||
- Build: `Prior` node type (§2.2) for the induction rule; `think()` restricted
|
||||
to membership = `point_fit` warped by that prior (§1.3); a
|
||||
`correspondence_beat` (§4) using grade (1) self-consistency only; the prior's
|
||||
`warp`/`calibration` updated on error.
|
||||
- **Exit / verify:** on a snapshot copy, over N held predictions, the induction
|
||||
prior's calibration (Brier) *improves monotonically* across beats versus a
|
||||
frozen-prior control; the improved prior *reloads across a cold boot*
|
||||
(reboot-prove); the live daemon is untouched. This proves the whole thesis in
|
||||
one faculty: frozen operation, learning prior, in-geometry loop.
|
||||
|
||||
### M2 — Priors as stored, addressable, grounded objects
|
||||
|
||||
Generalize M1's prior into the full first-class object.
|
||||
|
||||
- Build: `Prior` records for all seven faculties (warp = axis_gain + bias_dir +
|
||||
faculty scalars); the prior-warp wrapper around `engram_reason_point_fit`;
|
||||
deprecate hard-coded constants (`drop_frac`, `assoc_floor`, `ext_floor`) in
|
||||
favor of prior scalars.
|
||||
- **Exit:** each of the five C operators runs through its prior with identical
|
||||
results when the prior is set to today's constants (behavioral parity), then
|
||||
*diverges beneficially* once the loop refines it. Priors survive reboot.
|
||||
|
||||
### M3 — Grounding as a relation; hold/assert split
|
||||
|
||||
- Build: the `grounded-by` edge (§5.2) with `for_whom`; move
|
||||
`engram_verify_grounding`'s flag into stored edges; gate **assertion only**
|
||||
against the honesty floor; leave holding unconditional.
|
||||
- **Exit:** ungrounded nodes are first-class (held, queryable, no deletion);
|
||||
the same claim carries different `grounded-by` weights for two observers; an
|
||||
assertion below floor is refused while the content remains held. Curiosity =
|
||||
a `vantage_read` that surfaces high-salience / low-grounding regions.
|
||||
|
||||
### M4 — The vantage-read unified (three settings)
|
||||
|
||||
- Build: `vantage_read(anchor, aperture)` unifying descriptor-build +
|
||||
`engram_geo_reify_lookup` + activation-weighting; self / foreign-field /
|
||||
aperture settings.
|
||||
- **Exit:** one code path produces (a) a normal self-read, (b) a
|
||||
perspective-shifted read from another `for_whom`, (c) a narrowed veil read —
|
||||
differing only by argument. Reboot-stable.
|
||||
|
||||
### M5 — The gradient is the currency (remove point-collapse from thinking)
|
||||
|
||||
- Build: `GeoGradient` as the return of every faculty; move point-collapse into
|
||||
a separate expression faculty (sample gradient → surface). `think`'s output
|
||||
feeds back as the next steering direction (closed-loop flow).
|
||||
- **Exit:** a chain of `think` calls flows as gradients end-to-end; a point
|
||||
appears *only* at an explicit expression call. Spiked vs spread gradients are
|
||||
observable (deduction vs prediction).
|
||||
|
||||
### M6 — Metastability enforced
|
||||
|
||||
- Build: node-level keystone flag/layer for self/values; the correspondence-loop
|
||||
reads but never writes keystone priors; bounded update rate.
|
||||
- **Exit:** across a long run of correspondence beats on a snapshot, keystones
|
||||
are provably unchanged while non-keystone priors drift and improve; the graph
|
||||
neither freezes (all metrics static) nor dissolves (keystone drift = 0,
|
||||
identity nodes intact). Reboot-prove the keystone set.
|
||||
|
||||
### M7 — Cutover
|
||||
|
||||
- Build: nothing new — the gated migration.
|
||||
- **Exit:** reboot-proof on the prod-clone secondary; cutover via
|
||||
`launchctl bootout → settle-poll → bootstrap`; post-cutover the live engram
|
||||
shows priors refining in-geometry with zero data loss and keystones intact.
|
||||
|
||||
### Definition of "to completion"
|
||||
|
||||
The architecture is **complete** when: cognition runs as `think` = one frozen
|
||||
traversal-read primitive + geo-algebra, steered by **stored, learnable, grounded
|
||||
priors**; the reflexive correspondence-loop refines those priors *in the
|
||||
geometry* against outcomes (grounding = learning = one loop); the engram holds
|
||||
ungrounded content as first-class with grounding as a relation and the honesty
|
||||
floor only on assertion; the vantage-read serves self / foreign-field / aperture
|
||||
from one op; and a stable keystone core anchors a plastic everything-else —
|
||||
all reboot-proven and cut over to the live engram without data loss. The named
|
||||
faculties survive only as *labels on regions of think's steering space*, not as
|
||||
separate code.
|
||||
|
||||
---
|
||||
|
||||
## Appendix A — Designed vs. already-built (honest ledger)
|
||||
|
||||
**Already built (EXISTS, cited):**
|
||||
- The shared primitive `engram_reason_point_fit` and the five operators over it
|
||||
+ geo-algebra (`engram_reason.c`).
|
||||
- The verifier on `point_fit` (`engram_verify.c`:
|
||||
`engram_verify_grounding`, `engram_verify_consistency`).
|
||||
- Centered-frame geometry, combine/subtract/analogy/distance
|
||||
(`engram_geometry.{c,h}`).
|
||||
- The reification beat: hub-neighborhood detection → first-class `Neighborhood`
|
||||
nodes with member edges, nesting, supersession-with-residue, hot-path lookup
|
||||
(`engram_geo_reify_store`, `engram_geo_reify_nest`, `engram_geo_reify_lookup`).
|
||||
- The tiered paged store (buffer pool / LRU / WAL / checkpointer / pinning),
|
||||
the RAM activation graph (base-level learning `access_ts[]`, WM slots,
|
||||
`working_memory_weight` / `background_activation`), `StoreNode` / `StoreEdge`.
|
||||
- `GeoMember` already separating relational salience (`centrality`) from
|
||||
intrinsic `salience`.
|
||||
|
||||
**Designed, NOT built (this doc's deliverables):**
|
||||
- `GeoGradient` and `think()` as the single entry point (§1, M5).
|
||||
- `Prior` as a first-class stored, warp-carrying, calibrated node (§2, M1–M2).
|
||||
- Salience/importance as a *relation* superseding the intrinsic node scalar
|
||||
(§2.1, M3).
|
||||
- `vantage_read(anchor, aperture)` unifying the three perspective settings
|
||||
(§3, M4).
|
||||
- **The reflexive correspondence-loop / `correspondence_beat`** — the learning
|
||||
engine, moved from offline Python into the geometry (§4, M1). *The core new
|
||||
subsystem.*
|
||||
- `grounded-by` edge + assertion-only honesty floor (§5, M3).
|
||||
- Node-level keystones + bounded plasticity (§6, M6).
|
||||
|
||||
**Uncertain / to resolve during build:**
|
||||
- The exact warp parameterization (axis_gain vs full metric) — start minimal
|
||||
(per-axis gain), measure, widen only if calibration demands it.
|
||||
- Grade-(1) self-consistency as a sufficient reality signal for M1, versus
|
||||
needing grade (2)/(3) sooner — decided empirically on the snapshot.
|
||||
+577
-8
@@ -3240,6 +3240,37 @@ static void jb_init(JsonBuf* b) {
|
||||
b->buf[0] = '\0';
|
||||
}
|
||||
|
||||
/* jb_init_cap — jb_init with a caller-supplied starting capacity.
|
||||
*
|
||||
* WHY THIS EXISTS (2026-08-11 self-review). jb_init starts at 64 BYTES and
|
||||
* jb_reserve grows by doubling. That is right for the hundreds of small JSON
|
||||
* responses this runtime builds per minute and catastrophic for the one that
|
||||
* is 64 MEGABYTES: serializing the canonical snapshot walked the buffer
|
||||
* 64B → 128B → ... → 128MB, about twenty reallocs, each copying everything
|
||||
* written so far. Roughly 128MB of memcpy per save, and — the part that
|
||||
* actually hurt — a fresh large span from the allocator every time.
|
||||
*
|
||||
* MEASURED (13,129 nodes / 43,400 edges, macOS arm64): RSS climbed +63MB per
|
||||
* snapshot write, linearly, 14 for 14 writes, no plateau — 204MB to 1,028MB.
|
||||
* `leaks` reported only 15KB genuinely unreachable, which is what makes this
|
||||
* subtle: nothing is leaked in the reachable/unreachable sense. engram_save
|
||||
* frees b.buf correctly on every path. The growth is the allocator declining
|
||||
* to return large freed spans to the OS, and the doubling walk guaranteeing
|
||||
* that each save asks for a differently-sized region than the last free made
|
||||
* available. Every durable write path calls this — node create, edge create,
|
||||
* the Hebbian batch write-back — so on the live daemon it grows without bound
|
||||
* until the process dies.
|
||||
*
|
||||
* The fix is to ask for the right size once. With a stable capacity the
|
||||
* allocator hands back the same span on every save and RSS flattens. */
|
||||
static void jb_init_cap(JsonBuf* b, size_t cap) {
|
||||
if (cap < 64) cap = 64;
|
||||
b->cap = cap; b->len = 0;
|
||||
b->buf = malloc(b->cap);
|
||||
if (!b->buf) { fputs("el_runtime: out of memory\n", stderr); exit(1); }
|
||||
b->buf[0] = '\0';
|
||||
}
|
||||
|
||||
static void jb_reserve(JsonBuf* b, size_t add) {
|
||||
if (b->len + add + 1 > b->cap) {
|
||||
while (b->len + add + 1 > b->cap) b->cap *= 2;
|
||||
@@ -6516,6 +6547,26 @@ static float* _eg_ctx_c = NULL;
|
||||
static int32_t _eg_ctx_dim = 0;
|
||||
static double _eg_act_ctx_cos = -2.0;
|
||||
|
||||
/* Fan-effect gauges (2026-08-11 self-review). Per-call, like ctx_cos: they
|
||||
* describe THIS activation, not process history. Without these the degree
|
||||
* normalization is an unobservable change to the most important scoring path
|
||||
* in the runtime, and "did it do anything" would be unanswerable — which is
|
||||
* exactly the failure the Hebbian learning rate had before it was measured.
|
||||
* fan_mean — mean applied factor over every propagation step. 1.0 means the
|
||||
* correction never bound (graph is flat, or d_ref is above every
|
||||
* pair's geometric mean degree). Falling toward FAN_MIN means
|
||||
* traversal is running through hubs.
|
||||
* fan_min_seen / fan_hits — the worst single penalty and how many steps were
|
||||
* penalized at all, so a low mean caused by one pathological hub
|
||||
* is distinguishable from broad hub saturation.
|
||||
* fan_dref — the live mean degree the correction is calibrated against;
|
||||
* publishing it makes densification visible over time. */
|
||||
static double _eg_act_fan_sum = 0.0;
|
||||
static double _eg_act_fan_min = 1.0;
|
||||
static int64_t _eg_act_fan_n = 0;
|
||||
static int64_t _eg_act_fan_hits = 0;
|
||||
static double _eg_act_fan_dref = 0.0;
|
||||
|
||||
static int _eg_embed_consec_fail = 0;
|
||||
static int64_t _eg_embed_breaker_until = 0;
|
||||
|
||||
@@ -6534,6 +6585,27 @@ static int64_t _eg_embed_breaker_until = 0;
|
||||
* rates keep the previous reading and diff. Restart legitimately resets to 0. */
|
||||
static int64_t _eg_act_breakthroughs = 0; /* forced promotions at the floor, cumulative */
|
||||
static int64_t _eg_act_wm_evicted = 0; /* ALL WM evictions, cumulative (see below) */
|
||||
/* ── Eviction CAUSE decomposition (2026-08-14 self-review) ──────────────────
|
||||
* _eg_act_wm_evicted is incremented from six sites with four distinct causes,
|
||||
* and every one of them collapsed into that single integer. Today's review
|
||||
* measured 175,547 evictions over 13.5h (~216/min against 24 slots) and could
|
||||
* not tell healthy rotation from cap thrashing from duplicate churn, because
|
||||
* the only available number counts all three the same way.
|
||||
*
|
||||
* That is this file's most-repeated defect. The 08-02 and 08-06 reviews were
|
||||
* each diagnosable only because someone first added a NEW gauge; dup_wm and
|
||||
* dup_wm_global exist precisely because the aggregate could not answer "why".
|
||||
* These three finish the decomposition, so that
|
||||
* evicted == floor + cap + bll + dup_wm + dup_wm_global
|
||||
* holds as an identity and each term names a different corrective action:
|
||||
* floor - candidates below the absolute admission bar. High = weak retrieval.
|
||||
* cap - lost the rank contest for 24 slots. High = genuine contention.
|
||||
* bll - carried-over residents that decayed under the ACT-R tau. High =
|
||||
* healthy forgetting, NOT pressure.
|
||||
* Confusing the third with the second is what makes WM churn unreadable. */
|
||||
static int64_t _eg_act_evict_floor = 0; /* below ENGRAM_WM_FLOOR (both passes) */
|
||||
static int64_t _eg_act_evict_cap = 0; /* over ENGRAM_WM_CAP (both passes) */
|
||||
static int64_t _eg_act_evict_bll = 0; /* carry-over decayed under BLL tau */
|
||||
/* Redundancy suppression counters (2026-08-05 self-review) — see
|
||||
* ENGRAM_DEDUP_COS. dup_seeds = semantic seed slots reclaimed from redundant
|
||||
* copies; dup_wm = WM candidates dropped for duplicating a higher-ranked
|
||||
@@ -6874,6 +6946,10 @@ typedef struct EngramStore {
|
||||
int* adj_to_len;
|
||||
int adj_dirty; /* 1 = rebuild needed before next BFS */
|
||||
int64_t adj_node_count; /* node_count at time of last adj_rebuild */
|
||||
/* Nodes with degree >= 1 at last adj_rebuild. The denominator for the
|
||||
* fan-effect reference degree — see eg_fan_factor for why isolated nodes
|
||||
* must not be counted. (2026-08-11 self-review) */
|
||||
int64_t adj_connected;
|
||||
} EngramStore;
|
||||
|
||||
static EngramStore* engram_global = NULL;
|
||||
@@ -7237,11 +7313,16 @@ static void engram_adj_rebuild(EngramStore* g) {
|
||||
if (ti >= 0 && g->adj_to[ti])
|
||||
g->adj_to[ti][to_pos[ti]++] = (int)ei;
|
||||
}
|
||||
/* Copy counts */
|
||||
/* Copy counts. Also tally how many nodes have any edge at all — the
|
||||
* fan-effect denominator. Free here, in the O(V) pass that already exists,
|
||||
* rather than as a separate scan. (2026-08-11 self-review) */
|
||||
int64_t connected = 0;
|
||||
for (int64_t i = 0; i < g->node_count; i++) {
|
||||
g->adj_from_len[i] = from_cnt[i];
|
||||
g->adj_to_len[i] = to_cnt[i];
|
||||
if (from_cnt[i] + to_cnt[i] > 0) connected++;
|
||||
}
|
||||
g->adj_connected = connected;
|
||||
free(from_cnt); free(to_cnt); free(from_pos); free(to_pos);
|
||||
g->adj_node_count = g->node_count;
|
||||
g->adj_dirty = 0;
|
||||
@@ -8496,6 +8577,7 @@ static void eg_wm_carry_over(EngramNode* cn, int64_t now_ms, int64_t* evict_ctr)
|
||||
cn->working_memory_weight = 0.0;
|
||||
cn->wm_anchor = 0.0;
|
||||
if (evict_ctr) (*evict_ctr)++;
|
||||
_eg_act_evict_bll++;
|
||||
} else {
|
||||
cn->working_memory_weight = w;
|
||||
}
|
||||
@@ -8662,6 +8744,108 @@ static double engram_activation_dampen(const EngramNode* n) {
|
||||
return 1.0 / (1.0 + log(1.0 + (double)n->activation_count));
|
||||
}
|
||||
|
||||
/* ── ACT-R fan effect: degree normalization for spreading activation ─────────
|
||||
* (2026-08-11 self-review. Closes the other half of a mechanism that has been
|
||||
* half-implemented since the BLL work.)
|
||||
*
|
||||
* THE GAP. This runtime implements ACT-R's base-level learning term
|
||||
* B_i = ln(Σ t_k^-d) (engram_bll_base_level) but never implemented the
|
||||
* ASSOCIATIVE term that goes with it:
|
||||
*
|
||||
* A_i = B_i + Σ_j W_j · S_ji where S_ji = S − ln(fan_j)
|
||||
*
|
||||
* fan_j is the number of things j is associated with. The whole point of the
|
||||
* fan effect (Anderson 1974; Anderson & Reder 1999) is that a source spreads a
|
||||
* FIXED budget of activation across its associations — so being connected to
|
||||
* many things makes each individual connection weaker. Without it, degree is
|
||||
* pure advantage: a node wins retrieval by being popular rather than by being
|
||||
* relevant. That is backwards, and it is what this graph has been doing.
|
||||
*
|
||||
* MEASURED ON THE LIVE STORE (13,129 nodes / 43,400 edges, 2026-08-11):
|
||||
* degree p50=14 p90=34 p95=82 p99=275 max=357 mean=23.3
|
||||
* the top 1% of nodes by degree touch 21.2% of all edges
|
||||
* So the most-connected node had a 25x propagation advantage over the median
|
||||
* node for no reason other than accumulated connections. The top hubs are not
|
||||
* even semantically central — several are duplicate pairs of the same document
|
||||
* left over from the redundancy census of the 2026-08-05 review.
|
||||
*
|
||||
* The hub problem was already recognized twice and patched narrowly both
|
||||
* times: InternalStateEvent nodes were cut out of propagation entirely (see
|
||||
* the frontier loop) and eg_hebb_node_budget caps per-node Hebbian mass. Both
|
||||
* are special cases of this general law. This is the general fix.
|
||||
*
|
||||
* FORM. Symmetric normalization, w / (deg(u)^β · deg(v)^β) with β = 0.5 — the
|
||||
* normalized-Laplacian / GCN form, which penalizes a hub both for sending and
|
||||
* for receiving. Both failure modes are live here: a hub source floods its
|
||||
* neighborhood, and a hub target gets reached by everything regardless of
|
||||
* relevance. Written relative to the graph's own mean degree:
|
||||
*
|
||||
* fan(u,v) = clamp( d_ref / sqrt(deg(u) · deg(v)), FAN_MIN, 1.0 )
|
||||
* d_ref = 2·|E| / |V| (mean degree, O(1), live)
|
||||
*
|
||||
* WHY IT IS CLAMPED AT 1.0 ON TOP — this is the load-bearing safety property,
|
||||
* not a detail. The factor can only ever REDUCE propagation, never amplify it.
|
||||
* Every constant downstream of this multiply is calibrated against today's
|
||||
* activation magnitudes: the 0.02 firing threshold, SPREAD_DECAY = 0.7, the
|
||||
* 0.15 WM promotion threshold, the 24-slot WM cap. A normalization that
|
||||
* boosted low-degree nodes would inflate the frontier, change how many nodes
|
||||
* clear 0.02, and silently recalibrate working memory as a side effect of a
|
||||
* change that was supposed to be about hubs. Capping at 1.0 means every pair
|
||||
* at or below mean degree — the common case — propagates EXACTLY as it does
|
||||
* today, and the only behavior that changes is that above-mean hubs stop
|
||||
* winning on degree alone. Strictly monotone, strictly conservative, and the
|
||||
* blast radius is confined to the nodes the change is aimed at.
|
||||
*
|
||||
* Self-calibrating: d_ref is recomputed from the live graph, so the correction
|
||||
* tracks densification instead of drifting against a constant that was right
|
||||
* in August 2026 and wrong a year later. Change is the signal.
|
||||
*
|
||||
* FAN_MIN = 0.30 bottoms the penalty at ~3.3x rather than the ~15x that raw
|
||||
* 1/deg would give at max degree. Same reasoning as ENGRAM_QGATE_FLOOR: damp
|
||||
* the uninformative path, never sever it. A hub is usually a hub for a reason;
|
||||
* it just should not also get a free win.
|
||||
*
|
||||
* Sources: Anderson & Reder 1999 (fan effect, S=1.6-2.0, d=0.5) ·
|
||||
* arXiv:2405.14831 HippoRAG (node specificity) · Systems 9(2):22
|
||||
* (normalized-Laplacian spreading activation) · arXiv:2606.30133 (β is a
|
||||
* low-sensitivity knob; gating and fan normalization carry the effect). */
|
||||
/* FAN_MIN 0.50, not the 0.30 this shipped as on the first build. Measured on
|
||||
* the live graph, β=0.5 with a 0.30 floor damped 96% of propagation steps to a
|
||||
* mean factor of 0.34 — and that number is not a bug in the correction, it is
|
||||
* an honest measurement of how hub-dominated traversal here actually is. But a
|
||||
* ~3x near-uniform damp is a bigger global change than one A/B run justifies,
|
||||
* and it cost a working-memory promotion (5 → 4) on the one query measured
|
||||
* cleanly. A 0.50 floor keeps the full mechanism and the whole [0.5, 1.0]
|
||||
* dynamic range for separating hubs from non-hubs, at half the blast radius.
|
||||
* The fan_mean / fan_hits gauges make the next review's tuning evidence-based
|
||||
* rather than another guess: loosen it when the data says WM can afford it. */
|
||||
#define ENGRAM_FAN_MIN 0.50
|
||||
|
||||
/* eg_node_degree — total (in + out) degree from the adjacency index. The index
|
||||
* is rebuilt at the top of engram_activate whenever topology changed, so this
|
||||
* is current. adj_node_count is the count at BUILD time and can lag
|
||||
* node_count; out-of-range indices report 0 and are treated as unpenalized. */
|
||||
static int eg_node_degree(const EngramStore* g, int64_t idx) {
|
||||
if (idx < 0 || idx >= g->adj_node_count) return 0;
|
||||
if (!g->adj_from_len || !g->adj_to_len) return 0;
|
||||
return g->adj_from_len[idx] + g->adj_to_len[idx];
|
||||
}
|
||||
|
||||
static double eg_fan_factor(const EngramStore* g, double d_ref,
|
||||
int64_t u_idx, int64_t v_idx) {
|
||||
if (d_ref <= 0.0) return 1.0;
|
||||
int du = eg_node_degree(g, u_idx);
|
||||
int dv = eg_node_degree(g, v_idx);
|
||||
/* Degree 0 is only reachable when the adjacency index is stale or absent;
|
||||
* an actually-isolated node is never on the frontier. Do not penalize what
|
||||
* we cannot measure. */
|
||||
if (du <= 0 || dv <= 0) return 1.0;
|
||||
double f = d_ref / sqrt((double)du * (double)dv);
|
||||
if (f > 1.0) return 1.0; /* never amplify — see above */
|
||||
if (f < ENGRAM_FAN_MIN) return ENGRAM_FAN_MIN;
|
||||
return f;
|
||||
}
|
||||
|
||||
/* Temporal proximity bonus: boost propagation along edges connecting
|
||||
* co-temporal nodes. Returns a multiplier bonus in [0, 0.2]. */
|
||||
static double engram_temporal_proximity_bonus(int64_t node_created,
|
||||
@@ -8829,6 +9013,8 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
||||
* miss nearly all events between beats; see the definition site).
|
||||
* ctx_cos stays per-call: it is a gauge of THIS query vs the centroid. */
|
||||
_eg_act_ctx_cos = -2.0;
|
||||
_eg_act_fan_sum = 0.0; _eg_act_fan_min = 1.0;
|
||||
_eg_act_fan_n = 0; _eg_act_fan_hits = 0;
|
||||
|
||||
/* ── Embedding backfill + query embedding (2026-07-24, bl-b2d1c944) ──
|
||||
* Backfill: embed up to N un-embedded eligible nodes per call, newest
|
||||
@@ -9061,6 +9247,29 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
||||
ftail++;
|
||||
}
|
||||
const double SPREAD_DECAY = 0.7;
|
||||
/* Reference degree for the fan-effect correction: mean degree over
|
||||
* CONNECTED nodes, 2|E| / |{v : deg(v) > 0}|. O(1) — adj_connected is
|
||||
* tallied during adjacency rebuild.
|
||||
*
|
||||
* NOT 2|E|/|V|. That was the first cut and instrumentation caught it
|
||||
* immediately: on the live graph it gives d_ref = 6.61, while the median
|
||||
* degree of a node that actually has edges is 14. Isolated nodes cannot
|
||||
* be on the frontier — spreading activation only ever traverses connected
|
||||
* ones — so including them in the denominator deflates the reference below
|
||||
* anything traversal will ever see, and the correction pins to
|
||||
* ENGRAM_FAN_MIN on every step. Measured on the first build:
|
||||
* fan_mean 0.3026 with fan_hits 579/579 — a uniform 0.30 multiplier, which
|
||||
* is not a fan effect at all. It is just a weaker SPREAD_DECAY, and it
|
||||
* would have quietly recalibrated the 0.02 firing threshold and WM
|
||||
* competition while appearing to be a targeted change.
|
||||
*
|
||||
* Over connected nodes the reference is ~23, above the median, so typical
|
||||
* traversal rides the 1.0 cap unchanged and only genuine hubs are damped
|
||||
* — which is the whole intent. The gauge that caught this is the reason it
|
||||
* was worth adding the gauge. */
|
||||
const double FAN_DREF = (g->adj_connected > 0)
|
||||
? (2.0 * (double)g->edge_count / (double)g->adj_connected) : 0.0;
|
||||
_eg_act_fan_dref = FAN_DREF;
|
||||
while (fhead < ftail) {
|
||||
Frontier f = fr[fhead++];
|
||||
if (f.hops >= max_depth) continue;
|
||||
@@ -9127,16 +9336,51 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
||||
* ~4x, never killed); unembedded targets pass ungated (no
|
||||
* information, no penalty); cosq == NULL (embedder down) means
|
||||
* no gating at all — same graceful degradation as seeding. */
|
||||
/* Rescale before gating (2026-08-14 self-review). Raw cosine from
|
||||
* nomic-embed is compressed into a narrow high band, so feeding it
|
||||
* to the gate directly makes the gate nearly a constant. Measured
|
||||
* on this store: 400 random UNRELATED node pairs gave median 0.562,
|
||||
* central 98% span [0.381, 0.743]. The raw gate therefore passed a
|
||||
* typical unrelated node at 0.25 + 0.75*0.562 = 0.67 — two thirds
|
||||
* strength for a node with no semantic relation to the query. That
|
||||
* is not a gate, it is a small tax.
|
||||
*
|
||||
* Shift-and-floor about ENGRAM_EMBED_S0, exactly as the Pass-2 WM
|
||||
* term at ENGRAM_EMBED_WM_WEIGHT already does. The constant was in
|
||||
* this file for this reason; the propagation gate simply never used
|
||||
* it. Same store, same 400 pairs, after the rescale: the median
|
||||
* unrelated pair drops to 0.40 while the top of the range is
|
||||
* preserved (0.85 vs 0.92), and gate spread widens 0.42 -> 0.60.
|
||||
* Only 8.5% of pairs fall to the floor, so lexical/structural
|
||||
* pathways through dissimilar nodes are damped, never severed.
|
||||
* Cf. arXiv:2512.15922, which rescales w' = (w-c)/(1-c) about
|
||||
* c = 0.4 for precisely this reason ("prevent overactivation and
|
||||
* context explosion"). */
|
||||
double qgate = 1.0;
|
||||
if (cosq && cosq[oi] > -1.5) {
|
||||
double c = cosq[oi] > 0.0 ? cosq[oi] : 0.0;
|
||||
double c = (cosq[oi] - ENGRAM_EMBED_S0) / (1.0 - ENGRAM_EMBED_S0);
|
||||
if (c < 0.0) c = 0.0;
|
||||
if (c > 1.0) c = 1.0;
|
||||
qgate = ENGRAM_QGATE_FLOOR + (1.0 - ENGRAM_QGATE_FLOOR) * c;
|
||||
}
|
||||
/* ── ACT-R fan effect (2026-08-11 self-review) ──
|
||||
* Symmetric degree normalization over the (source, target) pair.
|
||||
* The query gate above prunes branches that are semantically
|
||||
* irrelevant; this prunes branches that are merely POPULAR. They
|
||||
* are different failure modes — a duplicate document with 357
|
||||
* edges can be highly cosine-similar to the query and still be
|
||||
* the wrong thing to spread through. Only ever <= 1.0, so it
|
||||
* cannot inflate the frontier. See eg_fan_factor. */
|
||||
double fan = eg_fan_factor(g, FAN_DREF, cur, oi);
|
||||
_eg_act_fan_sum += fan;
|
||||
_eg_act_fan_n++;
|
||||
if (fan < 1.0) _eg_act_fan_hits++;
|
||||
if (fan < _eg_act_fan_min) _eg_act_fan_min = fan;
|
||||
/* eg_edge_eff_weight, not e->weight: edges that have repeatedly
|
||||
* carried co-activated pairs propagate more strongly. Identity on
|
||||
* an unlearned edge. (2026-08-04 self-review.) */
|
||||
double new_act = f.act * eg_edge_eff_weight(e) * SPREAD_DECAY
|
||||
* (1.0 + tbonus) * tdecay * dampen * qgate;
|
||||
* (1.0 + tbonus) * tdecay * dampen * qgate * fan;
|
||||
/* Firing threshold per classic spreading-activation: sub-threshold
|
||||
* activation neither updates the target nor enqueues it, so weak
|
||||
* signals die out instead of flooding the whole graph with tiny
|
||||
@@ -9426,6 +9670,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
||||
if (wm_weights[i] > 0.0 && wm_weights[i] < ENGRAM_WM_FLOOR) {
|
||||
wm_weights[i] = 0.0;
|
||||
_eg_act_wm_evicted++;
|
||||
_eg_act_evict_floor++;
|
||||
}
|
||||
}
|
||||
int64_t cap_count = 0;
|
||||
@@ -9461,6 +9706,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
||||
}
|
||||
wm_weights[i] = 0.0; /* over cap: evict */
|
||||
_eg_act_wm_evicted++;
|
||||
_eg_act_evict_cap++;
|
||||
}
|
||||
}
|
||||
/* If malloc failed, skip cap — WM unbounded this call, no corruption. */
|
||||
@@ -9572,6 +9818,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
||||
fn->working_memory_weight = 0.0;
|
||||
fn->wm_anchor = 0.0;
|
||||
_eg_act_wm_evicted++;
|
||||
_eg_act_evict_floor++;
|
||||
}
|
||||
}
|
||||
/* ── Global redundancy suppression (2026-08-06 self-review) ──────────
|
||||
@@ -9680,6 +9927,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
|
||||
n->working_memory_weight = 0.0; /* evict: over global cap */
|
||||
n->wm_anchor = 0.0; /* keep anchor coherent */
|
||||
_eg_act_wm_evicted++; /* was uncounted before 2026-08-02 */
|
||||
_eg_act_evict_cap++;
|
||||
}
|
||||
}
|
||||
/* If malloc failed, skip — WM over cap this call, no data corruption. */
|
||||
@@ -10109,11 +10357,24 @@ static void engram_emit_edge_json(JsonBuf* b, const EngramEdge* e) {
|
||||
jb_putc(b, '}');
|
||||
}
|
||||
|
||||
/* Size of the last snapshot this process serialized. Seeds the next save's
|
||||
* buffer so the doubling walk never runs on the big document. See jb_init_cap
|
||||
* for the measurement that motivated it. (2026-08-11 self-review) */
|
||||
static size_t _eg_save_cap_hint = 0;
|
||||
|
||||
el_val_t engram_save(el_val_t path) {
|
||||
const char* p = EL_CSTR(path);
|
||||
if (!p || !*p) return 0;
|
||||
EngramStore* g = engram_get();
|
||||
JsonBuf b; jb_init(&b);
|
||||
/* Pre-size from the previous save plus 12.5% headroom, so ordinary growth
|
||||
* between snapshots does not trigger a realloc and the request size stays
|
||||
* stable enough for the allocator to reuse the same span. First save of
|
||||
* the process has no hint and starts at 1MB — still 14 doublings better
|
||||
* than 64 bytes. */
|
||||
JsonBuf b;
|
||||
jb_init_cap(&b, _eg_save_cap_hint
|
||||
? _eg_save_cap_hint + (_eg_save_cap_hint >> 3) + 1024
|
||||
: (size_t)1 << 20);
|
||||
jb_puts(&b, "{\"nodes\":[");
|
||||
for (int64_t i = 0; i < g->node_count; i++) {
|
||||
if (i > 0) jb_putc(&b, ',');
|
||||
@@ -10153,6 +10414,10 @@ el_val_t engram_save(el_val_t path) {
|
||||
jb_putc(&b, '}');
|
||||
}
|
||||
jb_puts(&b, "]}");
|
||||
/* Remember the size BEFORE the write: the hint is about how much buffer
|
||||
* the next serialization needs, which is a property of the graph, not of
|
||||
* whether this particular fopen succeeded. */
|
||||
_eg_save_cap_hint = b.len;
|
||||
FILE* f = fopen(p, "wb");
|
||||
if (!f) { free(b.buf); return 0; }
|
||||
size_t w = fwrite(b.buf, 1, b.len, f);
|
||||
@@ -11727,8 +11992,11 @@ el_val_t engram_act_stats_json(void) {
|
||||
}
|
||||
/* 768, not 512: the write-back gauges added 2026-08-07 push the worst-case
|
||||
* rendering past the old bound, and snprintf would truncate the JSON into
|
||||
* an unparseable tail rather than fail loudly. */
|
||||
char buf[896];
|
||||
* an unparseable tail rather than fail loudly.
|
||||
* 1152, not 896: the five fan-effect gauges added 2026-08-11 add ~90 bytes
|
||||
* worst-case. Same reasoning — headroom is cheaper than a truncated tail
|
||||
* that every downstream JSON parser rejects as a whole. */
|
||||
char buf[1152];
|
||||
/* ctx_cos (2026-07-29): cos(query, context centroid) at the LAST
|
||||
* activate call, measured before the query was folded in. ~1.0 =
|
||||
* context aligned with current query; low = divergence (expected at
|
||||
@@ -11736,6 +12004,15 @@ el_val_t engram_act_stats_json(void) {
|
||||
* embedder down. The drift gauge for the context-centroid mechanism. */
|
||||
snprintf(buf, sizeof(buf),
|
||||
"{\"wm_evicted\":%lld,\"breakthroughs\":%lld,"
|
||||
/* Eviction cause decomposition (2026-08-14 self-review):
|
||||
* wm_evicted == evict_floor + evict_cap + evict_bll
|
||||
* + dup_wm + dup_wm_global.
|
||||
* Read them as a ratio, not a level. cap-dominant = real
|
||||
* contention for the 24 slots; bll-dominant = healthy decay of
|
||||
* carried-over residents; floor-dominant = retrieval is returning
|
||||
* weak candidates. The aggregate alone cannot distinguish these
|
||||
* and every prior WM incident needed a new gauge to diagnose. */
|
||||
"\"evict_floor\":%lld,\"evict_cap\":%lld,\"evict_bll\":%lld,"
|
||||
"\"embed_breaker_open\":%d,\"embed_consec_fail\":%d,"
|
||||
"\"ctx_cos\":%.3f,"
|
||||
"\"hebb_edges\":%lld,\"hebb_max\":%.4f,\"hebb_mass\":%.3f,"
|
||||
@@ -11749,9 +12026,19 @@ el_val_t engram_act_stats_json(void) {
|
||||
* any climb means a write path is mangling text again. Cheap
|
||||
* (counted at creation) — the full census lives in
|
||||
* engram_text_health_json. (2026-08-08 self-review) */
|
||||
"\"txt_damaged\":%lld}",
|
||||
"\"txt_damaged\":%lld,"
|
||||
/* Fan-effect gauges (2026-08-11 self-review) — see the
|
||||
* _eg_act_fan_* definitions. fan_mean == 1.0 with fan_hits == 0
|
||||
* means the degree correction never bound on the last activation;
|
||||
* a mean drifting toward ENGRAM_FAN_MIN means traversal is
|
||||
* running through hubs and the correction is doing work. */
|
||||
"\"fan_mean\":%.4f,\"fan_min\":%.4f,\"fan_hits\":%lld,"
|
||||
"\"fan_steps\":%lld,\"fan_dref\":%.2f}",
|
||||
(long long)_eg_act_wm_evicted,
|
||||
(long long)_eg_act_breakthroughs,
|
||||
(long long)_eg_act_evict_floor,
|
||||
(long long)_eg_act_evict_cap,
|
||||
(long long)_eg_act_evict_bll,
|
||||
breaker_open, _eg_embed_consec_fail,
|
||||
_eg_act_ctx_cos,
|
||||
(long long)hebb_edges, hebb_max, hebb_mass,
|
||||
@@ -11761,7 +12048,10 @@ el_val_t engram_act_stats_json(void) {
|
||||
(long long)_eg_hebb_wb_dropped,
|
||||
(long long)_eg_act_dup_seeds, (long long)_eg_act_dup_wm,
|
||||
(long long)_eg_act_dup_wm_global,
|
||||
(long long)_eg_txt_write_damaged);
|
||||
(long long)_eg_txt_write_damaged,
|
||||
(_eg_act_fan_n > 0 ? _eg_act_fan_sum / (double)_eg_act_fan_n : 1.0),
|
||||
_eg_act_fan_min, (long long)_eg_act_fan_hits,
|
||||
(long long)_eg_act_fan_n, _eg_act_fan_dref);
|
||||
return el_wrap_str(el_strdup(buf));
|
||||
}
|
||||
|
||||
@@ -11894,6 +12184,285 @@ el_val_t engram_label_df(el_val_t term) {
|
||||
return (el_val_t)df;
|
||||
}
|
||||
|
||||
/* ── Salient-term extraction (2026-08-13 self-review) ────────────────────────
|
||||
* THE MEASUREMENT. auto_term_empty_streak, the counter added by the 2026-08-06
|
||||
* review precisely to catch this class of silent death, read 50 and climbing.
|
||||
* Fifty consecutive curiosity scans in which the soul's dynamic seeding path
|
||||
* produced NOTHING and the loop fell back to its four hardcoded rotating
|
||||
* phrases. Dumping the live WM top says why in one look:
|
||||
*
|
||||
* Memory 0.390 memory:remembered
|
||||
* Memory 0.378 memory:remembered
|
||||
* Memory 0.377 memory:remembered
|
||||
* Memory 0.373 memory:remembered
|
||||
* Memory 0.370 memory:remembered
|
||||
*
|
||||
* Every slot at the top of working memory is a Memory node, and every Memory
|
||||
* node written by remember() carries the sentinel label "memory:remembered".
|
||||
* auto_term_try_slot reads the LABEL and only the label; the colon-no-space
|
||||
* guard (correctly) rejects sentinels as carrying no seed signal; so the
|
||||
* extractor had nothing to work with and returned empty, forever.
|
||||
*
|
||||
* THE ACTUAL DEFECT is not the sentinel guard — that guard is right. It is
|
||||
* that the extractor was built against Knowledge nodes, which have real
|
||||
* titles, and is structurally blind to the node type that in fact dominates
|
||||
* working memory. The label is not the content. A Memory node's topic is in
|
||||
* its text; the runtime just never looked there.
|
||||
*
|
||||
* WHY NOT ANOTHER GUARD. The extractor's whole history is guards: genre words
|
||||
* (07-23), quoted titles (07-25), English stopwords (07-30), label-df
|
||||
* (08-03). Four reviews, four blocklists, each written after watching a flood
|
||||
* happen. That is a losing shape, and 08-03 said so explicitly before adding
|
||||
* the fifth. The reason it keeps recurring is the algorithm underneath:
|
||||
* TAKE THE FIRST WORD, THEN CHECK WHETHER IT IS ACCEPTABLE. A first-word
|
||||
* extractor has no notion of term quality, so quality has to be bolted on as
|
||||
* rejection, and rejection can only encode the past.
|
||||
*
|
||||
* THE FIX is to invert it: score EVERY candidate token in the text and take
|
||||
* the argmax. Then term quality is the selection criterion rather than a
|
||||
* veto, and a bad token does not need to be on a list to lose — it only needs
|
||||
* a better token in the same text, which is the common case.
|
||||
*
|
||||
* SCORING (YAKE, Campos et al., Information Sciences 509:257-289, 2020 —
|
||||
* lightweight unsupervised single-document keyword extraction). YAKE scores
|
||||
* candidates on casing, position, frequency, context relatedness and sentence
|
||||
* dispersion, and beats RAKE/TextRank/SingleRank across twenty datasets. Two
|
||||
* of its five features port directly and cheaply; the other three are
|
||||
* within-document proxies for a corpus YAKE deliberately does not have. This
|
||||
* system DOES have the corpus — 12.7k labelled nodes — so real IDF is
|
||||
* substituted where YAKE has to approximate:
|
||||
*
|
||||
* score(t) = idf(t) · position(t) · casing(t)
|
||||
*
|
||||
* idf = ln((N+1)/(df+1)) real corpus specificity (Spärck
|
||||
* Jones 1972), strictly better than
|
||||
* YAKE's TF-based stand-in
|
||||
* position = 1/ln(e + i) YAKE T_Position: earlier tokens are
|
||||
* more topical. Keeps the old
|
||||
* first-word bias as a SOFT preference
|
||||
* instead of an absolute rule
|
||||
* casing = 1.30 acronym / 1.15 capitalised / 1.00 otherwise
|
||||
* YAKE T_Case
|
||||
*
|
||||
* THE min_df GATE. The df ceiling (08-03) rejects corpus-frequent markup and
|
||||
* sentinels. A floor was added alongside it for an independent reason: a term
|
||||
* appearing in ZERO labels cannot lexically reach anything, so it is a bad
|
||||
* seed however specific it looks.
|
||||
*
|
||||
* An earlier draft of this comment claimed the floor also subsumes the 73
|
||||
* hand-listed stopwords that 08-03 measured label-df as missing (Whose:0,
|
||||
* Would:0, Could:0). MEASURED, AND THAT CLAIM IS FALSE. Under word-boundary
|
||||
* df on the live store, function words are rare in labels but not absent:
|
||||
* about:2, whole:1, them:2, head:2. They clear a floor of 1. What actually
|
||||
* keeps them from winning is the argmax itself — they carry no position
|
||||
* advantage and lose to a topical term in the same text on every node
|
||||
* measured. The stopword list therefore STAYS as a real defense for the
|
||||
* Title-case cases, not as vestigial belt-and-braces. Recording the
|
||||
* correction rather than the tidier story: the floor buys lexical
|
||||
* reachability, the argmax buys quality, and the list still earns its keep.
|
||||
*
|
||||
* TABU IS APPLIED DURING THE ARGMAX, not after it. The old code picked a term
|
||||
* and then discarded it if it was tabu, which turned inhibition-of-return
|
||||
* into another source of empty scans. Excluding tabu terms from the candidate
|
||||
* set instead yields the best NON-TABU term, so rotation costs quality rather
|
||||
* than costing the whole scan.
|
||||
*
|
||||
* COST. One pass over g->nodes scoring all candidates at once (12.7k labels ×
|
||||
* <=32 candidates, short strings, good locality), twice per 30 s scan.
|
||||
*
|
||||
* POLICY LIVES IN THE SOUL. Thresholds arrive as arguments; the runtime
|
||||
* measures and ranks, awareness.el decides. Same split as engram_label_df.
|
||||
*
|
||||
* Returns the winning token, or "" when the node is missing, has no usable
|
||||
* text, or every candidate is gated out — "" remains the honest signal that
|
||||
* this slot yielded no seed, and auto_term_empty_streak still counts it. */
|
||||
#define ENGRAM_ST_MAXCAND 32
|
||||
#define ENGRAM_ST_TOKLEN 64
|
||||
#define ENGRAM_ST_SCANCHARS 400
|
||||
|
||||
/* Trim leading/trailing non-alphanumerics, then accept only tokens whose core
|
||||
* is alphanumeric plus '-' and '_' with at least 3 letters. This subsumes the
|
||||
* quoted-title guard (2026-07-25) and the "<!--" flood (2026-08-03)
|
||||
* structurally: markup and punctuation-bearing tokens never become
|
||||
* candidates, rather than being blocklisted after the fact. */
|
||||
static int eg_st_clean_token(const char* raw, size_t rawlen,
|
||||
char* out, size_t outcap) {
|
||||
size_t s = 0, e = rawlen;
|
||||
while (s < e && !isalnum((unsigned char)raw[s])) s++;
|
||||
while (e > s && !isalnum((unsigned char)raw[e - 1])) e--;
|
||||
size_t len = e - s;
|
||||
if (len < 4 || len >= outcap) return 0;
|
||||
int alpha = 0;
|
||||
for (size_t i = 0; i < len; i++) {
|
||||
unsigned char c = (unsigned char)raw[s + i];
|
||||
if (isalpha(c)) alpha++;
|
||||
else if (!isdigit(c) && c != '-' && c != '_') return 0;
|
||||
}
|
||||
if (alpha < 3) return 0;
|
||||
memcpy(out, raw + s, len);
|
||||
out[len] = '\0';
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* ENGRAM_ST_DEBUG=1 dumps the full scored candidate set to stderr. One
|
||||
* cached branch in production. This exists because the first live run of this
|
||||
* function returned five ALL-CAPS terms in a row and there was no way to see
|
||||
* whether that was the corpus or the casing weight without guessing — the
|
||||
* lesson this system keeps relearning. */
|
||||
static int _eg_st_debug(void) {
|
||||
static int v = -1;
|
||||
if (v < 0) { const char* e = getenv("ENGRAM_ST_DEBUG"); v = (e && *e == '1'); }
|
||||
return v;
|
||||
}
|
||||
|
||||
/* Word-boundary document frequency. engram_label_df uses istr_contains, i.e.
|
||||
* SUBSTRING matching, and that is the wrong estimator for term specificity on
|
||||
* short tokens: "them" hits inside "theme" and "anthem", "about" and "whole"
|
||||
* come back with df 2 and 1 rather than 0. That matters here specifically
|
||||
* because the min_df floor is what rejects English function words, and it can
|
||||
* only do that job if their df is honestly zero. Substring df quietly handed
|
||||
* them a survival ticket. Measured on the live store before this fix, "whole"
|
||||
* (df=1, idf=8.76) and "about" (df=2, idf=8.36) were outscoring real topical
|
||||
* terms and losing only on position — one node whose text happened to open
|
||||
* with a function word would have seeded on it.
|
||||
*
|
||||
* engram_label_df keeps substring semantics: it is a separate published
|
||||
* measure with existing callers, and changing it underneath them is not this
|
||||
* change's business. */
|
||||
static int eg_st_label_has_word(const char* hay, const char* word) {
|
||||
size_t wl = strlen(word);
|
||||
for (const char* p = hay; *p; p++) {
|
||||
if (strncasecmp(p, word, wl) != 0) continue;
|
||||
char before = (p == hay) ? '\0' : p[-1];
|
||||
char after = p[wl];
|
||||
if (before && (isalnum((unsigned char)before) || before == '_')) continue;
|
||||
if (after && (isalnum((unsigned char)after) || after == '_')) continue;
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* YAKE T_Case, adapted to this corpus. YAKE up-weights all-caps tokens
|
||||
* because in ordinary prose an acronym is rare and carries topic. That
|
||||
* assumption does not hold here: memory content written by remember()
|
||||
* conventionally OPENS WITH AN ALL-CAPS HEADER ("FRAME-ROUTER UPGRADE —
|
||||
* RESULTS", "THE GAP", "CENSUS"), so a flat acronym bonus systematically
|
||||
* hands the seed to whatever word the header happens to start with and lets
|
||||
* casing override the specificity signal it is supposed to only nudge.
|
||||
* Measured on the live store: the first five WM nodes returned PRIMING,
|
||||
* CONVERSATION, OCCUPATION, RELATIONAL, SELF-OCCUPATION — every one an
|
||||
* all-caps header word, none chosen on its merits.
|
||||
*
|
||||
* Genuine acronyms are SHORT (VBD, CCR, MCP, HTTP); shouty headers are long
|
||||
* words that happen to be capitalised. So the acronym bonus is restricted to
|
||||
* tokens of <= 5 characters, where all-caps is actually evidence of an
|
||||
* acronym rather than evidence of a heading. Longer all-caps tokens fall
|
||||
* through to the ordinary Title-case nudge — they still compete, they just
|
||||
* compete on specificity instead of on volume. */
|
||||
static double eg_st_casing(const char* t) {
|
||||
int upper = 0, lower = 0;
|
||||
size_t len = 0;
|
||||
for (const char* q = t; *q; q++, len++) {
|
||||
if (isupper((unsigned char)*q)) upper++;
|
||||
else if (islower((unsigned char)*q)) lower++;
|
||||
}
|
||||
if (lower == 0 && upper >= 2 && len <= 5) return 1.30; /* acronym */
|
||||
if (isupper((unsigned char)t[0])) return 1.15; /* Title/hdr */
|
||||
return 1.0;
|
||||
}
|
||||
|
||||
el_val_t engram_salient_term(el_val_t node_id, el_val_t max_df_v,
|
||||
el_val_t min_df_v, el_val_t tabu_v) {
|
||||
EngramStore* g = engram_get();
|
||||
int64_t ix = engram_find_node_index(EL_CSTR(node_id));
|
||||
if (ix < 0) return el_wrap_str(el_strdup(""));
|
||||
EngramNode* n = &g->nodes[ix];
|
||||
|
||||
int64_t max_df = (int64_t)max_df_v;
|
||||
int64_t min_df = (int64_t)min_df_v;
|
||||
if (max_df <= 0) max_df = g->node_count;
|
||||
if (min_df < 0) min_df = 0;
|
||||
const char* tabu = EL_CSTR(tabu_v);
|
||||
|
||||
/* Source selection. Prefer the label — it is a curated title when it is
|
||||
* one. Fall back to content when the label is absent or a sentinel
|
||||
* ("memory:remembered": a colon and no space). This single line is what
|
||||
* makes Memory nodes visible to the extractor at all. */
|
||||
const char* src = n->label;
|
||||
if (!src || !*src) {
|
||||
src = n->content;
|
||||
} else if (strchr(src, ':') != NULL && strchr(src, ' ') == NULL) {
|
||||
src = n->content;
|
||||
}
|
||||
if (!src || !*src) return el_wrap_str(el_strdup(""));
|
||||
|
||||
/* Collect distinct candidates from the head of the text. */
|
||||
char cand[ENGRAM_ST_MAXCAND][ENGRAM_ST_TOKLEN];
|
||||
int pos[ENGRAM_ST_MAXCAND];
|
||||
int64_t df[ENGRAM_ST_MAXCAND];
|
||||
int ncand = 0, tokidx = 0;
|
||||
|
||||
const char* p = src;
|
||||
const char* lim = src + strnlen(src, ENGRAM_ST_SCANCHARS);
|
||||
while (p < lim && ncand < ENGRAM_ST_MAXCAND) {
|
||||
while (p < lim && isspace((unsigned char)*p)) p++;
|
||||
if (p >= lim) break;
|
||||
const char* tk = p;
|
||||
while (p < lim && !isspace((unsigned char)*p)) p++;
|
||||
char buf[ENGRAM_ST_TOKLEN];
|
||||
int slot = tokidx++;
|
||||
if (!eg_st_clean_token(tk, (size_t)(p - tk), buf, sizeof(buf))) continue;
|
||||
|
||||
/* Tabu exclusion, applied here so the argmax runs over eligible
|
||||
* terms only. tabu arrives pipe-delimited: "|t0|t1|t2|t3|". */
|
||||
if (tabu && *tabu) {
|
||||
char pat[ENGRAM_ST_TOKLEN + 2];
|
||||
snprintf(pat, sizeof(pat), "|%s|", buf);
|
||||
if (istr_contains(tabu, pat)) continue;
|
||||
}
|
||||
int dup = 0;
|
||||
for (int i = 0; i < ncand; i++)
|
||||
if (strcasecmp(cand[i], buf) == 0) { dup = 1; break; }
|
||||
if (dup) continue;
|
||||
|
||||
memcpy(cand[ncand], buf, strlen(buf) + 1);
|
||||
pos[ncand] = slot;
|
||||
df[ncand] = 0;
|
||||
ncand++;
|
||||
}
|
||||
if (ncand == 0) return el_wrap_str(el_strdup(""));
|
||||
|
||||
/* One pass over the store, all candidates at once. */
|
||||
for (int64_t i = 0; i < g->node_count; i++) {
|
||||
const char* lbl = g->nodes[i].label;
|
||||
if (!lbl || !*lbl) continue;
|
||||
for (int c = 0; c < ncand; c++)
|
||||
if (eg_st_label_has_word(lbl, cand[c])) df[c]++;
|
||||
}
|
||||
|
||||
/* Argmax over idf · position · casing, subject to the df band. */
|
||||
int best = -1;
|
||||
double best_score = 0.0;
|
||||
for (int c = 0; c < ncand; c++) {
|
||||
if (df[c] > max_df) continue;
|
||||
if (df[c] < min_df) continue;
|
||||
double idf = log(((double)g->node_count + 1.0) / ((double)df[c] + 1.0));
|
||||
if (idf <= 0.0) continue;
|
||||
double position = 1.0 / log(2.718281828459045 + (double)pos[c]);
|
||||
double casing = eg_st_casing(cand[c]);
|
||||
double score = idf * position * casing;
|
||||
if (_eg_st_debug()) {
|
||||
fprintf(stderr, " cand %-24s df=%-5lld idf=%.2f pos=%d p=%.2f "
|
||||
"case=%.2f score=%.3f\n",
|
||||
cand[c], (long long)df[c], idf, pos[c], position,
|
||||
casing, score);
|
||||
}
|
||||
if (score > best_score) { best_score = score; best = c; }
|
||||
}
|
||||
if (best < 0) return el_wrap_str(el_strdup(""));
|
||||
return el_wrap_str(el_strdup(cand[best]));
|
||||
}
|
||||
|
||||
/* engram_embed_backfill — explicitly drive the lazy embedding backfill.
|
||||
* (2026-07-25 self-review.) The per-activate backfill (8 nodes/call) only
|
||||
* runs inside engram_activate, and on the authoritative HTTP store nothing
|
||||
|
||||
@@ -635,6 +635,13 @@ el_val_t engram_hebb_drain_json(el_val_t max);
|
||||
/* Document frequency of a term across node labels — term-specificity signal
|
||||
* for curiosity seed selection. (2026-08-03 self-review.) */
|
||||
el_val_t engram_label_df(el_val_t term);
|
||||
/* Best curiosity seed from one node: argmax over idf·position·casing across
|
||||
* the candidate tokens of its label, falling back to its content when the
|
||||
* label is a sentinel. Excludes pipe-delimited tabu terms during selection
|
||||
* and gates candidates to the df band [min_df, max_df]. Returns "" when
|
||||
* nothing qualifies. (2026-08-13 self-review.) */
|
||||
el_val_t engram_salient_term(el_val_t node_id, el_val_t max_df,
|
||||
el_val_t min_df, el_val_t tabu);
|
||||
el_val_t engram_embed_backfill(el_val_t count);
|
||||
el_val_t engram_list_layers_json(void);
|
||||
/* Working memory introspection — count, mean weight, and top-N snapshot.
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# nsbx — Neuron dev-environment Makefile
|
||||
# ---------------------------------------------------------------------------
|
||||
# Thin, documented wrappers over the `nsbx` primitive so a newcomer never has to
|
||||
# memorise the elc/cc build incantation or the sandbox lifecycle. Every target is
|
||||
# a one-liner over `nsbx`; nothing here reimplements engram logic.
|
||||
#
|
||||
# make dev NAME=tim # new branch + worktree + isolated engram, one shot
|
||||
# make status NAME=tim # inspect it (omit NAME to list all sandboxes)
|
||||
# make run NAME=tim # poke its API (API=/api/stats by default)
|
||||
# make test NAME=tim # run the safety rails as checks (nsbx validate)
|
||||
# make build NAME=tim # compile the worktree's changes into the engram
|
||||
# make destroy NAME=tim # tear it all down (branch kept)
|
||||
#
|
||||
# The isolated engram is ALWAYS a clone of the live store on a NON-default port;
|
||||
# prod (:8742 / :7770) is untouchable from here.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# locate nsbx next to this Makefile, regardless of where make is run from
|
||||
NSBX := $(dir $(realpath $(lastword $(MAKEFILE_LIST))))nsbx
|
||||
SBX := dev-$(NAME)
|
||||
API ?= /api/stats
|
||||
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
.PHONY: help dev build run test status list destroy bt
|
||||
|
||||
help: ## Show this help
|
||||
@echo "nsbx dev-environment — one-command isolated Neuron dev setup"
|
||||
@echo ""
|
||||
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(firstword $(MAKEFILE_LIST)) \
|
||||
| awk 'BEGIN{FS=":.*?## "}{printf " make %-22s %s\n", $$1, $$2}'
|
||||
@echo ""
|
||||
@echo " Variables: NAME=<dev name> (required for most) API=<path> (run)"
|
||||
@echo " BASE=<git ref> WT=<worktree dir> PORT=<n> ARGS=<extra nsbx flags>"
|
||||
|
||||
dev: ## Create branch + worktree + isolated engram (NAME=x [BASE=ref PORT=n WT=dir ARGS=...])
|
||||
@test -n "$(NAME)" || { echo "usage: make dev NAME=<name>"; exit 2; }
|
||||
$(NSBX) dev $(NAME) $(if $(BASE),--base $(BASE)) $(if $(PORT),--port $(PORT)) $(if $(WT),--worktree $(WT)) $(ARGS)
|
||||
|
||||
build: ## Rebuild the engram from the dev worktree's own source (NAME=x)
|
||||
@test -n "$(NAME)" || { echo "usage: make build NAME=<name>"; exit 2; }
|
||||
@wt=$$(python3 -c "import json,os;print(json.load(open(os.path.expanduser('~/.neuron/sandboxes/$(SBX)/dev.json')))['worktree'])" 2>/dev/null); \
|
||||
test -n "$$wt" || { echo "no dev.json for $(SBX) — run 'make dev NAME=$(NAME)' first"; exit 2; }; \
|
||||
$(NSBX) build $(SBX) --source "$$wt"
|
||||
|
||||
bt: build run ## Fast El loop: rebuild the engram from the worktree, then poke it (NAME=x)
|
||||
|
||||
run: ## Poke the isolated engram's API (NAME=x [API=/api/stats])
|
||||
@test -n "$(NAME)" || { echo "usage: make run NAME=<name> [API=/path]"; exit 2; }
|
||||
$(NSBX) run $(SBX) api $(API)
|
||||
|
||||
test: ## Run the safety rails as checks: zero-loss, reboot, RSS, parity, keystones (NAME=x)
|
||||
@test -n "$(NAME)" || { echo "usage: make test NAME=<name>"; exit 2; }
|
||||
$(NSBX) validate $(SBX)
|
||||
|
||||
status: ## Show one sandbox's status (NAME=x), or list all if NAME is unset
|
||||
@if [ -n "$(NAME)" ]; then $(NSBX) status $(SBX); else $(NSBX) list; fi
|
||||
|
||||
list: ## List all sandboxes
|
||||
$(NSBX) list
|
||||
|
||||
destroy: ## Tear down engram + worktree (NAME=x [ARGS=--delete-branch])
|
||||
@test -n "$(NAME)" || { echo "usage: make destroy NAME=<name>"; exit 2; }
|
||||
$(NSBX) dev-down $(NAME) $(ARGS)
|
||||
@@ -0,0 +1,176 @@
|
||||
# nsbx — the Neuron Sandbox
|
||||
|
||||
**Dev environment as a primitive.** A reproducible way to run experiments *and code
|
||||
changes* against the **real** engram runtime on an isolated snapshot of the live
|
||||
mind — with a gated promote-to-prod path built on the proven rails.
|
||||
|
||||
Everyone (Tim, any team member, any agent) gets their own private, safe copy of the
|
||||
mind to build against. **Prod — the live Neuron on `:8742` (engram) / `:7770`
|
||||
(soul) — is untouchable from a sandbox.** A sandbox runs a *separate* engram
|
||||
process, on a *separate* port, against a *separate* clone of the store. The only op
|
||||
that can ever reach prod is `promote`, which is explicit, gated, and per-use
|
||||
approved.
|
||||
|
||||
It **wraps the real engram binary** — it never reimplements any engram logic. It
|
||||
generalises two proven proto-sandboxes into one primitive:
|
||||
|
||||
- the **cog-arch** build — isolated git worktree + build + clone of the live `.egm` + real C tests
|
||||
- the **store-fix** cutover — secondary soul + launchctl `bootout → settle → bootstrap` rails
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
export PATH="$PWD:$PATH" # or symlink nsbx onto your PATH
|
||||
|
||||
nsbx up # your private copy of the mind (auto-named <user>-dev)
|
||||
nsbx run <name> api /api/stats # poke it
|
||||
nsbx validate <name> # prove it: zero-loss, reboot, RSS, retrieval, keystones
|
||||
nsbx destroy <name> # cheap teardown; live untouched
|
||||
```
|
||||
|
||||
That is the whole loop. Sane defaults: stock prod binary, auto-allocated port
|
||||
(`8900+`, never `8742`/`7770`), snapshot of the live store.
|
||||
|
||||
## One-command dev onboarding — `nsbx dev` (start here)
|
||||
|
||||
Going from a clone to *coding on the mind* is a single command. It creates a git
|
||||
**branch**, a persistent git **worktree**, and an **isolated engram** (a clone of the
|
||||
live store on a non-default port) — and wires the whole worktree to that clone so you
|
||||
**cannot hit live `:8742` by accident**.
|
||||
|
||||
```bash
|
||||
make dev NAME=tim # branch wt/tim + worktree + isolated engram, in one shot
|
||||
cd ~/Development/neuron-technologies/el-worktrees/tim
|
||||
source .nsbx-env # every ENGRAM_* var now points at YOUR clone
|
||||
|
||||
# edit El in the worktree, then the fast loop:
|
||||
make build NAME=tim # compile your El change into the isolated engram
|
||||
make run NAME=tim # poke it (API=/api/stats by default)
|
||||
make test NAME=tim # run the safety rails as checks
|
||||
make destroy NAME=tim # tear it all down (branch kept; live untouched)
|
||||
```
|
||||
|
||||
**Why this exists:** so provisional/experimental work is built **directly in El against a
|
||||
throwaway cloned engram** — not prototyped in Python and re-ported later. The El
|
||||
edit → `make build` → `make run` loop is the path of least resistance; that double-work is
|
||||
what stranded the translation faculty for weeks.
|
||||
|
||||
### What `nsbx dev <name>` does, in order
|
||||
|
||||
1. **branch** — `git worktree add -b <prefix><name>` (default prefix `dev/`; a *real named
|
||||
branch*, never detached HEAD).
|
||||
2. **worktree** — at a **persistent** path (default `…/el-worktrees/<name>`, override
|
||||
`NSBX_DEV_WT_ROOT`). It **refuses** `/tmp` — temp dirs are ablated on compaction, which
|
||||
is the exact "worktree in /tmp + no branch = lost work" failure this designs out.
|
||||
3. **isolated engram** — `nsbx create` under the hood: clone of the live store + WAL +
|
||||
config, booted on an auto-allocated port (`8900+`, never `:8742`/`:7770`). Stock prod
|
||||
binary by default (instant); `--build` compiles the worktree's own runtime instead.
|
||||
4. **env pin** — writes `.nsbx-env` (+ `.envrc` for direnv) into the worktree exporting
|
||||
`ENGRAM_URL / ENGRAM_PORT / ENGRAM_DATA_DIR / ENGRAM_API_KEY / NEURON_ENGRAM_URL …` — all
|
||||
pointing at the clone. Nothing references live.
|
||||
|
||||
```
|
||||
nsbx dev <name> [--base REF] [--worktree DIR] [--port N] [--prefix P] [--build] [--no-engram] [--repo R]
|
||||
nsbx dev-down <name> [--delete-branch] [--repo R] # destroy engram + remove worktree
|
||||
```
|
||||
|
||||
### Makefile targets
|
||||
|
||||
| target | does |
|
||||
|--------|------|
|
||||
| `make dev NAME=x` | branch + worktree + isolated engram (one shot) |
|
||||
| `make bt NAME=x` | fast El loop: `build` then `run` |
|
||||
| `make build NAME=x` | recompile the engram from the worktree's El source |
|
||||
| `make run NAME=x` | poke the isolated engram (`API=/api/stats`) |
|
||||
| `make test NAME=x` | rails as checks (`nsbx validate`) |
|
||||
| `make status [NAME=x]` | inspect one, or `list` all |
|
||||
| `make destroy NAME=x` | tear down (add `ARGS=--delete-branch` to drop the branch) |
|
||||
|
||||
> Note: if a bare `dev` branch already exists in the repo, git can't create `dev/*` names —
|
||||
> pass `--prefix wt/` (or delete the stray `dev` branch). The tool surfaces git's exact error.
|
||||
|
||||
## The code-change dev loop (first-class)
|
||||
|
||||
Run *your changed runtime*, not just the stock binary, against a snapshot:
|
||||
|
||||
```bash
|
||||
# build a runtime from a working tree, a git branch, or a prebuilt binary:
|
||||
nsbx create feat --source /path/to/worktree # elc + cc build from source
|
||||
nsbx create feat --branch feat/my-change --repo <r> # worktree the branch, then build
|
||||
nsbx create feat --binary /path/to/engram # use a prebuilt binary
|
||||
|
||||
nsbx build feat --source /path/to/worktree # rebuild + hot-restart in place
|
||||
nsbx validate feat # prove the change is safe
|
||||
nsbx promote feat --i-approve-prod-cutover # gated rails cutover (see below)
|
||||
```
|
||||
|
||||
The build replicates the engram release recipe exactly:
|
||||
`elc engram/src/server.el > engram.c` then
|
||||
`cc -std=c11 -O2 -I lang/runtime engram.c el_runtime.c engram_*.c -lcurl -lpthread`.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
| op | what it does |
|
||||
|----|--------------|
|
||||
| `create <name> [--port N] [--source\|--branch\|--binary]` | consistent snapshot of the live store+WAL+config into an isolated dir; place or **build** the runtime; boot the real engram daemon on an isolated port. Named, versioned (binary sha + egm sha in `manifest.json`), reproducible. |
|
||||
| `up [name]` | one command: create-if-missing then start; prints the URL. |
|
||||
| `build <name> --source\|--branch` | rebuild the runtime from a code change and hot-restart on the same clone+port. |
|
||||
| `run <name> <cmd…>` / `run <name> api <path> [json]` | run an experiment against the real runtime; capture output + before/after stats + wall time. Env: `$SBX_URL $SBX_PORT $SBX_KEY $SBX_DATA $SBX_BIN`. |
|
||||
| `validate <name>` | the rails as first-class checks (below). |
|
||||
| `promote <name> [--data] [--i-approve-prod-cutover]` | **the only prod-touching op.** Gated rails cutover. DRY-RUN plan unless approved. |
|
||||
| `destroy <name>` | stop the isolated daemon, free the port, remove the clone. Live untouched. |
|
||||
| `list` / `status <name>` | inspect. |
|
||||
|
||||
## `validate` — the rails as checks
|
||||
|
||||
- **zero-loss-under-load** — node/edge counts hold at/above baseline through ~15s of sustained tick+read load
|
||||
- **reboot-prove** — counts survive a real stop→start of the daemon
|
||||
- **rss-bound** — daemon RSS under `NSBX_RSS_BOUND_MB` (default 550 MB, from the store-fix reboot-proof)
|
||||
- **retrieval-parity** — top-k node ids for a fixed probe set match the create-time baseline
|
||||
- **keystone-integrity** — `kn-efeb4a5b…` and `kn-5b606390…` present and intact
|
||||
|
||||
A PASS writes `validate.json` stamped with the binary sha; `promote` refuses unless
|
||||
the current binary has a fresh PASS on record.
|
||||
|
||||
## `promote` — gated cutover (rails only)
|
||||
|
||||
Default is a **dry-run plan**. With `--i-approve-prod-cutover` it, in order:
|
||||
|
||||
1. **snapshot-first** — back up live `egm`+`wal`+`plist` to `~/.neuron/backups/promote-<name>-<ts>/` with a `rollback.txt`
|
||||
2. **additive** binary install — copy the validated binary to a *new* file, update the plist `ENGRAM_REAL_BIN` (old binary retained — additive/supersede, never destructive)
|
||||
3. **rails cutover** — `launchctl bootout` → **settle-poll** (prints until the job is gone) → `launchctl bootstrap`. Never `pkill`, never `kickstart -k`.
|
||||
4. **verify** — `/api/stats` returns, edges ≥ baseline, keystones intact
|
||||
5. **auto-rollback armed** — any verify failure restores the plist (and data, if `--data`) and boots the prior binary back via the same rails
|
||||
|
||||
## Isolation guarantees
|
||||
|
||||
- separate **port** (`8900+`; refuses `8742`/`7770`), separate **store clone**, separate **process**
|
||||
- a hard guard refuses to boot a sandbox daemon whose data dir resolves to the live store
|
||||
- sandboxes are plain supervised background processes (not launchd), so teardown is a signal + settle-poll — it can never touch the prod launchd job
|
||||
- prod is read exactly twice: once for the snapshot, and (only if you approve) during `promote`
|
||||
|
||||
## Layout
|
||||
|
||||
- tool: `tools/neuron-sandbox/nsbx` (this repo, branch `feat/neuron-sandbox`)
|
||||
- runtime state: `~/.neuron/sandboxes/<name>/` — `data/` (clone), `bin/engram`, `build/`, `logs/`, `manifest.json`, `validate.json`, `baseline/`
|
||||
|
||||
## Validated (dogfood)
|
||||
|
||||
Standing up a sandbox from a live-store clone and reproducing a **known** result:
|
||||
|
||||
- **retrieval-parity 25/25** top-k id overlap vs baseline; sandbox boot-stats exactly matched the live baseline captured at snapshot time (10 672 nodes / 32 439 edges) — the wrapped real binary faithfully reloads the live mind
|
||||
- reboot-prove + zero-loss PASS; RSS 379 MB < 550 MB; keystones intact
|
||||
- the **cog-arch correspondence-loop** re-run *inside* the sandbox reproduced the known calibration numbers exactly: held-Brier **0.028648 → 0.000586** (98.0% reduction), monotone, **reboot bit-identical**, metastability holds; and the real-store Stance persistence reboot-proved at **10 994-node** scale (`think()` on real 768-dim embeddings) against a scratch copy of the sandbox's own clone — never live
|
||||
- `promote` dry-run refused to touch prod; teardown freed the port; live `:8742`/`:7770` never perturbed (soul uptime unbroken)
|
||||
|
||||
## Migrating existing experiments
|
||||
|
||||
Each ad-hoc harness becomes `nsbx run <name> …` (or `--source` build) against a sandbox:
|
||||
|
||||
- **cog-arch** — `nsbx create x --source <worktree>` then `nsbx run x -- bash cogarch_dogfood.sh` (compiles + runs the real C cognition tests against `$SBX_DATA`)
|
||||
- **codec / ingest / faculty** — `nsbx run x api /api/<endpoint> '<json>'` against the isolated daemon, or a script using `$SBX_URL`/`$SBX_KEY`; measure with the built-in before/after stats
|
||||
|
||||
## Env knobs
|
||||
|
||||
`NSBX_ROOT`, `NSBX_PORT_BASE`, `NSBX_RSS_BOUND_MB`, `NSBX_REMERGE_THRESHOLD`,
|
||||
`EL_REPO` (for `elc` + runtime sources), `ENGRAM_LIVE_DATA_DIR`, `ENGRAM_LIVE_PLIST`.
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
# cog-arch correspondence-loop dogfood — RUN INSIDE the sandbox via `nsbx run`.
|
||||
# Compiles the REAL engram C runtime + cognition tests and reproduces the known
|
||||
# calibration result (memory 194c69c8): held-Brier 0.028648 -> 0.000586, reboot-proven,
|
||||
# then reboot-proves the Stance persistence against a SCRATCH COPY of THIS sandbox's
|
||||
# clone of the real store (never live, never the running daemon's file).
|
||||
set -euo pipefail
|
||||
WT="${COGARCH_WT:-/private/tmp/claude-501/-Users-will/6531446d-bc27-4095-930b-e04777c3db4f/scratchpad/cogarch-wt}"
|
||||
RT="$WT/lang/runtime"; T="$WT/engram/test"
|
||||
: "${SBX_DATA:?run me via: nsbx run <name> -- bash cogarch_dogfood.sh}"
|
||||
B="$(mktemp -d)"
|
||||
echo "### building cog-arch tests against the real engram runtime sources"
|
||||
cc -std=c11 -O2 -w -I "$RT" -o "$B/test_cognition" \
|
||||
"$T/test_cognition.c" "$RT/engram_cognition.c" "$RT/engram_reason.c" \
|
||||
"$RT/engram_geometry.c" "$RT/engram_store.c" "$RT/engram_vindex.c" -lm
|
||||
cc -std=c11 -O2 -w -I "$RT" -o "$B/test_realstore" \
|
||||
"$T/test_cognition_realstore.c" "$RT/engram_cognition.c" "$RT/engram_reason.c" \
|
||||
"$RT/engram_geometry.c" "$RT/engram_store.c" "$RT/engram_vindex.c" -lm
|
||||
|
||||
echo; echo "### [A] synthetic correspondence-loop (known: Brier 0.028648 -> 0.000586)"
|
||||
"$B/test_cognition" | grep -E "held-Brier|reduction|reboot|monotone|metastab|RESULT" || true
|
||||
|
||||
echo; echo "### [B] reboot-prove Stance on a SCRATCH COPY of this sandbox's real-store clone"
|
||||
SCRATCH="$B/store-clone"; mkdir -p "$SCRATCH"
|
||||
cp -p "$SBX_DATA/neuron.egm" "$SCRATCH/" 2>/dev/null || true
|
||||
cp -p "$SBX_DATA/neuron.wal" "$SCRATCH/" 2>/dev/null || true
|
||||
cp -p "$SBX_DATA/conf" "$SCRATCH/" 2>/dev/null || true
|
||||
cp -p "$SBX_DATA/meta.json" "$SCRATCH/" 2>/dev/null || true
|
||||
"$B/test_realstore" "$SCRATCH" || true
|
||||
rm -rf "$B"
|
||||
Executable
+836
@@ -0,0 +1,836 @@
|
||||
#!/usr/bin/env bash
|
||||
# nsbx — the Neuron Sandbox: a reproducible primitive for running experiments and
|
||||
# code changes against the REAL engram runtime on an isolated snapshot of the live
|
||||
# mind, with a gated promote-to-prod path built on the proven rails.
|
||||
#
|
||||
# It WRAPS the real engram binary — it never reimplements any engram logic. The only
|
||||
# prod-touching op is `promote`, which is explicit, gated, and per-use approved.
|
||||
#
|
||||
# Generalises two proven proto-sandboxes:
|
||||
# - the cog-arch build (isolated git worktree + build + clone of live .egm + real C tests)
|
||||
# - the store-fix cutover (secondary soul + launchctl bootout->settle->bootstrap rails)
|
||||
#
|
||||
# Lifecycle: create -> [build] -> run -> validate -> promote(gated) -> destroy
|
||||
#
|
||||
# Rails (always): built offline; NEVER auto-promotes; never touches live :8742/:7770
|
||||
# except READ for the snapshot and the gated promote; snapshot-first; honest measured
|
||||
# reporting. Cutover is launchctl bootout -> settle-poll -> bootstrap ONLY —
|
||||
# never pkill, never kickstart -k.
|
||||
set -uo pipefail
|
||||
|
||||
# ---------------------------------------------------------------- constants ----
|
||||
LIVE_DATA_DIR="${ENGRAM_LIVE_DATA_DIR:-$HOME/.neuron/engram}"
|
||||
LIVE_PLIST="${ENGRAM_LIVE_PLIST:-$HOME/Library/LaunchAgents/ai.neuron.engram.plist}"
|
||||
LIVE_LABEL="ai.neuron.engram"
|
||||
LIVE_BIND_PORT=8742 # engram — FORBIDDEN for sandboxes
|
||||
SOUL_PORT=7770 # soul — FORBIDDEN for sandboxes
|
||||
LIVE_KEY="${ENGRAM_API_KEY:-ntn-user-2026}"
|
||||
LIVE_URL="http://127.0.0.1:${LIVE_BIND_PORT}"
|
||||
SBX_ROOT="${NSBX_ROOT:-$HOME/.neuron/sandboxes}"
|
||||
BACKUP_ROOT="$HOME/.neuron/backups"
|
||||
EL_REPO="${EL_REPO:-$HOME/Development/neuron-technologies/foundation/el}"
|
||||
PORT_BASE="${NSBX_PORT_BASE:-8900}"
|
||||
RSS_BOUND_MB="${NSBX_RSS_BOUND_MB:-550}" # from store-fix reboot-proof (aaf13f88)
|
||||
REMERGE_THRESHOLD="${NSBX_REMERGE_THRESHOLD:-40000}"
|
||||
KEYSTONES=( "kn-efeb4a5b-5aff-4759-8a97-7233099be6ee" "kn-5b606390-a52d-4ca2-8e0e-eba141d13440" )
|
||||
# fixed probe set for retrieval-parity (stable, identity-anchored)
|
||||
PARITY_QUERIES=( "who am I" "self identity core" "engram store durability" "keystone self anchor" "grounding honesty" )
|
||||
|
||||
C_RED=$'\033[31m'; C_GRN=$'\033[32m'; C_YEL=$'\033[33m'; C_DIM=$'\033[2m'; C_BLD=$'\033[1m'; C_0=$'\033[0m'
|
||||
|
||||
# ---------------------------------------------------------------- helpers ------
|
||||
die(){ printf '%serror:%s %s\n' "$C_RED" "$C_0" "$*" >&2; exit 1; }
|
||||
log(){ printf '%s==>%s %s\n' "$C_BLD" "$C_0" "$*" >&2; }
|
||||
info(){ printf ' %s\n' "$*" >&2; }
|
||||
ok(){ printf ' %s%s%s\n' "$C_GRN" "$*" "$C_0" >&2; }
|
||||
warn(){ printf ' %s%s%s\n' "$C_YEL" "$*" "$C_0" >&2; }
|
||||
need(){ command -v "$1" >/dev/null 2>&1 || die "missing dependency: $1"; }
|
||||
now(){ date -u +%Y%m%dT%H%M%SZ; }
|
||||
sha(){ shasum -a 256 "$1" 2>/dev/null | awk '{print $1}'; }
|
||||
epoch(){ python3 -c 'import time;print(time.time())'; }
|
||||
|
||||
sdir(){ printf '%s/%s' "$SBX_ROOT" "$1"; }
|
||||
manifest(){ printf '%s/manifest.json' "$(sdir "$1")"; }
|
||||
mexists(){ [ -f "$(manifest "$1")" ]; }
|
||||
mget(){ # mget <name> <jsonpath>
|
||||
python3 -c "import json,sys; d=json.load(open('$(manifest "$1")')); print(d$2)" 2>/dev/null
|
||||
}
|
||||
|
||||
port_free(){ ! (exec 3<>"/dev/tcp/127.0.0.1/$1") 2>/dev/null; }
|
||||
alloc_port(){
|
||||
local p="$PORT_BASE"
|
||||
while :; do
|
||||
if [ "$p" = "$LIVE_BIND_PORT" ] || [ "$p" = "$SOUL_PORT" ]; then p=$((p+1)); continue; fi
|
||||
if port_free "$p" && ! _port_claimed "$p"; then echo "$p"; return 0; fi
|
||||
p=$((p+1)); [ "$p" -gt 9100 ] && die "no free sandbox port in range"
|
||||
done
|
||||
}
|
||||
_port_claimed(){ # is another sandbox already assigned this port?
|
||||
local p="$1" d
|
||||
for d in "$SBX_ROOT"/*/manifest.json; do
|
||||
[ -f "$d" ] || continue
|
||||
[ "$(python3 -c "import json;print(json.load(open('$d'))['port'])" 2>/dev/null)" = "$p" ] && return 0
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
live_stats(){ curl -s -m5 "$LIVE_URL/api/stats" 2>/dev/null; }
|
||||
api(){ # api <name> <path> [json-body]
|
||||
local name="$1" path="$2" body="${3:-}"
|
||||
local port; port="$(mget "$name" "['port']")"; [ -n "$port" ] || die "unknown sandbox: $name"
|
||||
local url="http://127.0.0.1:${port}${path}"
|
||||
if [ -n "$body" ]; then curl -s -m30 -X POST -H 'Content-Type: application/json' -d "$body" "$url"
|
||||
else curl -s -m30 "$url"; fi
|
||||
}
|
||||
sbx_stats(){ api "$1" "/api/stats"; }
|
||||
stat_field(){ printf '%s' "$1" | sed -n "s/.*\"$2\":\([0-9]*\).*/\1/p"; }
|
||||
|
||||
daemon_pid(){ local f; f="$(sdir "$1")/daemon.pid"; [ -f "$f" ] && cat "$f" || true; }
|
||||
daemon_alive(){ local p; p="$(daemon_pid "$1")"; [ -n "$p" ] && kill -0 "$p" 2>/dev/null; }
|
||||
|
||||
# ---------------------------------------------------------------- elc/build ----
|
||||
find_elc(){
|
||||
command -v elc 2>/dev/null && return 0
|
||||
local arch; arch="$(uname -m)"
|
||||
case "$arch" in
|
||||
arm64) echo "$EL_REPO/lang/dist/platform/elc-darwin-arm64";;
|
||||
x86_64) echo "$EL_REPO/lang/dist/platform/elc-linux-amd64";;
|
||||
*) echo "$EL_REPO/lang/dist/platform/elc";;
|
||||
esac
|
||||
}
|
||||
|
||||
# _build_binary <src_tree> <out_bin> <build_log_dir>
|
||||
# Replicates the proven engram release recipe:
|
||||
# elc engram/src/server.el > engram.c
|
||||
# cc -std=c11 -O2 -I lang/runtime engram.c el_runtime.c engram_*.c -lcurl -lpthread
|
||||
_build_binary(){
|
||||
local src="$1" out="$2" blog="$3"
|
||||
local elc server rt
|
||||
elc="$(find_elc)"; [ -x "$elc" ] || die "elc not found/executable: $elc (set EL_REPO)"
|
||||
server="$src/engram/src/server.el"; rt="$src/lang/runtime"
|
||||
[ -f "$server" ] || die "no engram/src/server.el under source tree: $src"
|
||||
[ -f "$rt/el_runtime.c" ] || die "no lang/runtime/el_runtime.c under source tree: $src (this branch may keep it generated/untracked)"
|
||||
ls "$rt"/engram_*.c >/dev/null 2>&1 || die "no lang/runtime/engram_*.c engine sources under: $src"
|
||||
mkdir -p "$blog"
|
||||
log "build: elc transpile server.el -> engram.c"
|
||||
"$elc" "$server" > "$blog/engram.c" 2>"$blog/elc.err" || { cat "$blog/elc.err" >&2; die "elc transpile failed"; }
|
||||
info "engram.c: $(wc -c <"$blog/engram.c" | tr -d ' ') bytes"
|
||||
log "build: cc link (el_runtime + engram_* engine)"
|
||||
cc -std=c11 -O2 -w -I "$rt" -o "$out" \
|
||||
"$blog/engram.c" "$rt/el_runtime.c" "$rt"/engram_*.c \
|
||||
-lcurl -lpthread 2>"$blog/cc.err" \
|
||||
|| { grep -i 'error:' "$blog/cc.err" | sort -u | head >&2; die "cc link failed (see $blog/cc.err)"; }
|
||||
ok "built: $out ($(ls -lh "$out" | awk '{print $5}'), sha $(sha "$out" | cut -c1-12))"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- daemon -------
|
||||
# start_daemon <name> : boots the sandbox's real engram binary on its isolated
|
||||
# port against its cloned data dir, with the SAME auto-remerge net the live soul
|
||||
# uses (so the sandbox faithfully reaches the live edge population on boot).
|
||||
start_daemon(){
|
||||
local name="$1" d; d="$(sdir "$name")"
|
||||
daemon_alive "$name" && { info "already running (pid $(daemon_pid "$name"))"; return 0; }
|
||||
local port bin data export key
|
||||
port="$(mget "$name" "['port']")"; bin="$d/bin/engram"; data="$d/data"
|
||||
key="sbx-$name"; export="$data/.scan-export.reseed-clean.json"
|
||||
[ -x "$bin" ] || die "sandbox binary missing: $bin"
|
||||
[ "$port" != "$LIVE_BIND_PORT" ] && [ "$port" != "$SOUL_PORT" ] || die "refusing forbidden port $port"
|
||||
[ -f "$data/neuron.egm" ] || die "sandbox has no cloned store: $data/neuron.egm"
|
||||
# HARD guard: never point a sandbox daemon at the live data dir.
|
||||
[ "$(cd "$data" && pwd -P)" != "$(cd "$LIVE_DATA_DIR" && pwd -P)" ] || die "refusing: sandbox data dir resolves to LIVE store"
|
||||
|
||||
log "boot engram on isolated :$port (data=$data)"
|
||||
(
|
||||
ENGRAM_DATA_DIR="$data" ENGRAM_BIND=":$port" ENGRAM_API_KEY="$key" \
|
||||
ENGRAM_STORE=1 ENGRAM_CHRONOCEPTION=1 ENGRAM_SELF_REIFY=1 ENGRAM_GC=1 \
|
||||
ENGRAM_POOL_FRAMES=16384 ENGRAM_WRITE_BARRIER=1 \
|
||||
exec "$bin"
|
||||
) >"$d/logs/daemon.log" 2>&1 &
|
||||
local pid=$!
|
||||
echo "$pid" > "$d/daemon.pid"
|
||||
# readiness poll
|
||||
local url="http://127.0.0.1:$port" i s
|
||||
for i in $(seq 1 30); do
|
||||
s="$(curl -s -m3 "$url/api/stats" 2>/dev/null)"
|
||||
[ -n "$s" ] && break; sleep 0.5
|
||||
done
|
||||
[ -n "$s" ] || { warn "daemon did not become ready (see $d/logs/daemon.log)"; return 1; }
|
||||
ok "ready pid=$pid boot-stats: $s"
|
||||
# auto-remerge net (idempotent): match live edge population if the export is present
|
||||
if [ -f "$export" ]; then
|
||||
local edges; edges="$(stat_field "$s" edge_count)"
|
||||
if [ -n "$edges" ] && [ "$edges" -lt "$REMERGE_THRESHOLD" ]; then
|
||||
log "auto-remerge: booted with $edges edges (< $REMERGE_THRESHOLD) — merging full edge export"
|
||||
local r; r="$(curl -s -m300 -X POST -H 'Content-Type: application/json' \
|
||||
-d "{\"_auth\":\"$key\",\"path\":\"$export\"}" "$url/api/load-merge" 2>/dev/null)"
|
||||
info "remerge resp: ${r:0:120}"
|
||||
ok "post-remerge stats: $(curl -s -m5 "$url/api/stats")"
|
||||
fi
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# stop_daemon <name> : graceful TERM + settle-poll until the port is free.
|
||||
# (Sandbox daemons are plain supervised bg processes — not launchd — so teardown
|
||||
# is a signal + poll, never pkill of anything else.)
|
||||
stop_daemon(){
|
||||
local name="$1" pid port
|
||||
pid="$(daemon_pid "$name")"; port="$(mget "$name" "['port']")"
|
||||
[ -n "$pid" ] || { info "not running"; return 0; }
|
||||
log "stop daemon pid=$pid, settle-poll until :$port frees"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
local i
|
||||
for i in $(seq 1 40); do
|
||||
kill -0 "$pid" 2>/dev/null || { port_free "$port" && { ok "stopped, port $port free"; : >"$(sdir "$name")/daemon.pid"; return 0; }; }
|
||||
printf '.' >&2; sleep 0.5
|
||||
done
|
||||
printf '\n' >&2
|
||||
kill -9 "$pid" 2>/dev/null || true; sleep 1
|
||||
: >"$(sdir "$name")/daemon.pid"
|
||||
port_free "$port" && ok "stopped (after SIGKILL), port $port free" || warn "port $port still busy"
|
||||
}
|
||||
|
||||
# ================================================================ create =======
|
||||
cmd_create(){
|
||||
local name="" port="" src="" branch="" repo="$EL_REPO" binpath=""
|
||||
# first positional arg is the name unless it's a flag; default to "<user>-dev"
|
||||
if [ $# -gt 0 ] && [ "${1#-}" = "$1" ]; then name="$1"; shift; else name="${USER:-dev}-dev"; fi
|
||||
while [ $# -gt 0 ]; do case "$1" in
|
||||
--port) port="$2"; shift 2;;
|
||||
--source) src="$2"; shift 2;;
|
||||
--branch) branch="$2"; shift 2;;
|
||||
--repo) repo="$2"; shift 2;;
|
||||
--binary) binpath="$2"; shift 2;;
|
||||
*) die "unknown flag: $1";;
|
||||
esac; done
|
||||
mexists "$name" && die "sandbox '$name' already exists (destroy it first)"
|
||||
need curl; need python3; need shasum
|
||||
[ -f "$LIVE_DATA_DIR/neuron.egm" ] || die "live store not found: $LIVE_DATA_DIR/neuron.egm"
|
||||
if [ -n "$port" ]; then
|
||||
{ [ "$port" = "$LIVE_BIND_PORT" ] || [ "$port" = "$SOUL_PORT" ]; } && die "refusing forbidden port $port (live)"
|
||||
port_free "$port" || die "port $port already in use"
|
||||
else port="$(alloc_port)"; fi
|
||||
|
||||
local d; d="$(sdir "$name")"
|
||||
mkdir -p "$d/data" "$d/bin" "$d/logs" "$d/build" "$d/baseline"
|
||||
log "sandbox '$name' at $d (isolated port $port)"
|
||||
|
||||
# ---- CONSISTENT snapshot of the live mind (file-copy: same set the rails backup
|
||||
# uses; WAL replay on sandbox boot reconciles the tail -> crash-consistent) ----
|
||||
log "snapshot live store -> clone (store + WAL + config)"
|
||||
local f
|
||||
for f in neuron.egm neuron.wal conf meta.json self_anchor .scan-export.reseed-clean.json; do
|
||||
if [ -e "$LIVE_DATA_DIR/$f" ]; then cp -p "$LIVE_DATA_DIR/$f" "$d/data/$f"; info "cloned $f ($(du -h "$d/data/$f" | awk '{print $1}'))"; fi
|
||||
done
|
||||
local egm_sha; egm_sha="$(sha "$d/data/neuron.egm")"
|
||||
|
||||
# ---- capture live baseline (READ only) ----
|
||||
local lstats; lstats="$(live_stats)"
|
||||
local base_nodes base_edges
|
||||
base_nodes="$(stat_field "$lstats" node_count)"; base_edges="$(stat_field "$lstats" edge_count)"
|
||||
info "live baseline stats: ${lstats:-<unavailable>}"
|
||||
|
||||
# ---- determine + place the runtime binary (versioned into the snapshot) ----
|
||||
local source_desc live_bin
|
||||
live_bin="$(_live_real_bin)"
|
||||
if [ -n "$binpath" ]; then
|
||||
[ -x "$binpath" ] || die "not an executable binary: $binpath"
|
||||
cp -p "$binpath" "$d/bin/engram"; source_desc="prebuilt:$binpath"
|
||||
elif [ -n "$src" ]; then
|
||||
_build_binary "$src" "$d/bin/engram" "$d/build"; source_desc="source:$src"
|
||||
elif [ -n "$branch" ]; then
|
||||
log "worktree: $repo @ $branch -> $d/build/worktree"
|
||||
git -C "$repo" worktree add --detach "$d/build/worktree" "$branch" >/dev/null 2>&1 \
|
||||
|| die "git worktree add failed ($repo @ $branch)"
|
||||
_build_binary "$d/build/worktree" "$d/bin/engram" "$d/build"; source_desc="branch:$branch@$repo"
|
||||
else
|
||||
[ -x "$live_bin" ] || die "cannot resolve live ENGRAM_REAL_BIN: $live_bin"
|
||||
cp -p "$live_bin" "$d/bin/engram"; source_desc="stock-prod:$live_bin"
|
||||
fi
|
||||
local bin_sha; bin_sha="$(sha "$d/bin/engram")"
|
||||
info "runtime: $source_desc (sha ${bin_sha:0:12})"
|
||||
|
||||
# ---- write manifest ----
|
||||
python3 - "$name" "$port" "$source_desc" "$bin_sha" "$egm_sha" "$base_nodes" "$base_edges" "$(sha "$live_bin" 2>/dev/null)" <<'PY' > "$(manifest "$name")"
|
||||
import json,sys,datetime
|
||||
name,port,src,binsha,egmsha,bn,be,livebinsha=sys.argv[1:9]
|
||||
json.dump({
|
||||
"name":name,"port":int(port),"created_at":datetime.datetime.now(datetime.timezone.utc).isoformat(),
|
||||
"source":src,"binary_sha256":binsha,"clone_egm_sha256":egmsha,
|
||||
"live_binary_sha256":livebinsha,
|
||||
"live_baseline":{"node_count":int(bn or 0),"edge_count":int(be or 0)},
|
||||
"keystones":["kn-efeb4a5b-5aff-4759-8a97-7233099be6ee","kn-5b606390-a52d-4ca2-8e0e-eba141d13440"]
|
||||
}, sys.stdout, indent=2)
|
||||
PY
|
||||
ok "manifest written"
|
||||
|
||||
# ---- boot + capture the sandbox's own settled baseline (reproducible target) ----
|
||||
start_daemon "$name" || die "daemon failed to start"
|
||||
local sstats; sstats="$(sbx_stats "$name")"
|
||||
local sbn sbe; sbn="$(stat_field "$sstats" node_count)"; sbe="$(stat_field "$sstats" edge_count)"
|
||||
_capture_retrieval "$name" "$d/baseline/retrieval.json"
|
||||
# fold sandbox baseline into manifest
|
||||
python3 - "$(manifest "$name")" "$sbn" "$sbe" <<'PY'
|
||||
import json,sys
|
||||
mf,bn,be=sys.argv[1],sys.argv[2],sys.argv[3]
|
||||
d=json.load(open(mf)); d["sbx_baseline"]={"node_count":int(bn or 0),"edge_count":int(be or 0)}
|
||||
json.dump(d,open(mf,'w'),indent=2)
|
||||
PY
|
||||
log "created."
|
||||
info "sandbox baseline (settled): nodes=$sbn edges=$sbe"
|
||||
info "next: nsbx validate $name | nsbx run $name api /api/stats"
|
||||
}
|
||||
|
||||
_live_real_bin(){
|
||||
python3 - "$LIVE_PLIST" <<'PY' 2>/dev/null
|
||||
import sys,plistlib
|
||||
try:
|
||||
d=plistlib.load(open(sys.argv[1],'rb'))
|
||||
print(d.get("EnvironmentVariables",{}).get("ENGRAM_REAL_BIN",""))
|
||||
except Exception: print("")
|
||||
PY
|
||||
}
|
||||
|
||||
_capture_retrieval(){ # <name> <outfile> : top-k ids for the fixed probe set
|
||||
local name="$1" out="$2" q res
|
||||
local port; port="$(mget "$name" "['port']")"; local key="sbx-$name"
|
||||
{
|
||||
echo "{"
|
||||
local first=1
|
||||
for q in "${PARITY_QUERIES[@]}"; do
|
||||
res="$(curl -s -m10 -X POST -H 'Content-Type: application/json' \
|
||||
-d "{\"_auth\":\"$key\",\"query\":\"$q\",\"limit\":5}" "http://127.0.0.1:$port/api/search" 2>/dev/null)"
|
||||
local ids; ids="$(printf '%s' "$res" | python3 -c 'import sys,json
|
||||
try:
|
||||
d=json.load(sys.stdin)
|
||||
rows=d if isinstance(d,list) else d.get("results",d.get("hits",[]))
|
||||
print(json.dumps([r.get("id") for r in rows][:5]))
|
||||
except Exception: print("[]")' 2>/dev/null)"
|
||||
[ $first -eq 1 ] || echo ","; first=0
|
||||
printf ' %s: %s' "$(python3 -c "import json,sys;print(json.dumps(sys.argv[1]))" "$q")" "${ids:-[]}"
|
||||
done
|
||||
echo ""; echo "}"
|
||||
} > "$out"
|
||||
}
|
||||
|
||||
# ================================================================ up ===========
|
||||
# Dead-simple one-command dev environment: `nsbx up` gives you (or Tim, or anyone)
|
||||
# a private, isolated copy of the live mind to build against. Creates it on first
|
||||
# run with sane defaults (stock prod binary, auto-allocated port), just starts it
|
||||
# thereafter. Prod on :$LIVE_BIND_PORT/:$SOUL_PORT is unreachable from here by design.
|
||||
cmd_up(){
|
||||
local name; if [ $# -gt 0 ] && [ "${1#-}" = "$1" ]; then name="$1"; shift; else name="${USER:-dev}-dev"; fi
|
||||
if mexists "$name"; then daemon_alive "$name" || start_daemon "$name"; else cmd_create "$name" "$@"; fi
|
||||
local port; port="$(mget "$name" "['port']")"
|
||||
echo >&2
|
||||
ok "your sandbox '$name' is ready at http://127.0.0.1:$port (a private copy of the mind — prod is untouchable)"
|
||||
info "experiment: nsbx run $name api /api/stats"
|
||||
info "prove it: nsbx validate $name"
|
||||
info "tear down: nsbx destroy $name"
|
||||
}
|
||||
|
||||
# ================================================================ build ========
|
||||
# Rebuild an existing sandbox's runtime from a source tree/branch and hot-restart
|
||||
# it on the SAME clone + port (the code-change dev loop, in place).
|
||||
cmd_build(){
|
||||
local name="$1"; shift || true
|
||||
mexists "$name" || die "no such sandbox: $name"
|
||||
local src="" branch="" repo="$EL_REPO"
|
||||
while [ $# -gt 0 ]; do case "$1" in
|
||||
--source) src="$2"; shift 2;; --branch) branch="$2"; shift 2;; --repo) repo="$2"; shift 2;;
|
||||
*) die "unknown flag: $1";; esac; done
|
||||
local d; d="$(sdir "$name")"
|
||||
stop_daemon "$name"
|
||||
if [ -n "$src" ]; then _build_binary "$src" "$d/bin/engram" "$d/build"
|
||||
elif [ -n "$branch" ]; then
|
||||
rm -rf "$d/build/worktree" 2>/dev/null; git -C "$repo" worktree prune 2>/dev/null
|
||||
git -C "$repo" worktree add --detach "$d/build/worktree" "$branch" >/dev/null 2>&1 || die "worktree add failed"
|
||||
_build_binary "$d/build/worktree" "$d/bin/engram" "$d/build"
|
||||
else die "usage: nsbx build <name> --source DIR | --branch REF [--repo R]"; fi
|
||||
# record new binary sha
|
||||
python3 - "$(manifest "$name")" "$(sha "$d/bin/engram")" "${src:-branch:$branch}" <<'PY'
|
||||
import json,sys; mf,s,src=sys.argv[1:4]
|
||||
d=json.load(open(mf)); d["binary_sha256"]=s; d["source"]="rebuilt:"+src
|
||||
json.dump(d,open(mf,'w'),indent=2)
|
||||
PY
|
||||
start_daemon "$name"
|
||||
ok "rebuilt + restarted on :$(mget "$name" "['port']")"
|
||||
}
|
||||
|
||||
# ================================================================ run ==========
|
||||
cmd_run(){
|
||||
local name="$1"; shift || true
|
||||
mexists "$name" || die "no such sandbox: $name"
|
||||
daemon_alive "$name" || start_daemon "$name"
|
||||
local d port; d="$(sdir "$name")"; port="$(mget "$name" "['port']")"
|
||||
# direct API form: nsbx run <name> api <path> [json]
|
||||
if [ "${1:-}" = "api" ]; then
|
||||
api "$name" "$2" "${3:-}"; echo; return 0
|
||||
fi
|
||||
[ "${1:-}" = "--" ] && shift # allow an explicit separator: nsbx run <name> -- <cmd...>
|
||||
[ $# -gt 0 ] || die "usage: nsbx run <name> <cmd...> | nsbx run <name> api <path> [json]"
|
||||
local ts log0; ts="$(now)"; log0="$d/logs/run-$ts.log"
|
||||
local s0 t0 t1 s1
|
||||
s0="$(sbx_stats "$name")"; t0="$(epoch)"
|
||||
log "run experiment against sandbox '$name' (:$port)"
|
||||
info "cmd: $*"
|
||||
( export SBX_NAME="$name" SBX_PORT="$port" SBX_URL="http://127.0.0.1:$port" \
|
||||
SBX_KEY="sbx-$name" SBX_DATA="$d/data" SBX_BIN="$d/bin/engram"
|
||||
"$@" ) 2>&1 | tee "$log0"
|
||||
local rc=${PIPESTATUS[0]}
|
||||
t1="$(epoch)"; s1="$(sbx_stats "$name")"
|
||||
{
|
||||
echo "--- nsbx run metrics ---"
|
||||
echo "exit_code: $rc"
|
||||
printf 'wall_secs: %.3f\n' "$(python3 -c "print($t1-$t0)")"
|
||||
echo "stats_before: $s0"
|
||||
echo "stats_after: $s1"
|
||||
} | tee -a "$log0" >&2
|
||||
return $rc
|
||||
}
|
||||
|
||||
# ================================================================ validate =====
|
||||
# The rails as first-class checks. Baseline = the sandbox's own settled state at
|
||||
# create (reproducible). zero-loss through sustained load AND reboot; reboot-prove;
|
||||
# RSS bound; retrieval parity; keystone integrity.
|
||||
cmd_validate(){
|
||||
local name="$1"; shift || true
|
||||
mexists "$name" || die "no such sandbox: $name"
|
||||
daemon_alive "$name" || start_daemon "$name"
|
||||
local d port key; d="$(sdir "$name")"; port="$(mget "$name" "['port']")"; key="sbx-$name"
|
||||
local url="http://127.0.0.1:$port"
|
||||
local bn be; bn="$(mget "$name" "['sbx_baseline']['node_count']")"; be="$(mget "$name" "['sbx_baseline']['edge_count']")"
|
||||
log "validate '$name' against baseline nodes=$bn edges=$be"
|
||||
local -a names=() results=() details=()
|
||||
|
||||
# 1) sustained load — no data loss under activity
|
||||
local s cur_n cur_e i
|
||||
log "check: sustained load (~15s: tick + reads) then zero-loss"
|
||||
for i in $(seq 1 15); do
|
||||
curl -s -m5 -X POST -H 'Content-Type: application/json' -d "{\"_auth\":\"$key\"}" "$url/api/tick" >/dev/null 2>&1
|
||||
curl -s -m5 "$url/api/stats" >/dev/null 2>&1
|
||||
done
|
||||
s="$(sbx_stats "$name")"; cur_n="$(stat_field "$s" node_count)"; cur_e="$(stat_field "$s" edge_count)"
|
||||
names+=("zero-loss-under-load"); if [ "${cur_n:-0}" -ge "${bn:-0}" ] && [ "${cur_e:-0}" -ge "${be:-0}" ]; then
|
||||
results+=("PASS"); else results+=("FAIL"); fi
|
||||
details+=("nodes $cur_n>=$bn, edges $cur_e>=$be")
|
||||
|
||||
# 2) reboot-prove — counts survive a real restart
|
||||
log "check: reboot-prove (stop -> start -> compare)"
|
||||
local pre_n pre_e; pre_n="$cur_n"; pre_e="$cur_e"
|
||||
stop_daemon "$name"; start_daemon "$name" >/dev/null
|
||||
s="$(sbx_stats "$name")"; cur_n="$(stat_field "$s" node_count)"; cur_e="$(stat_field "$s" edge_count)"
|
||||
names+=("reboot-prove"); if [ "${cur_n:-0}" -ge "${bn:-0}" ] && [ "${cur_e:-0}" -ge "${be:-0}" ]; then
|
||||
results+=("PASS"); else results+=("FAIL"); fi
|
||||
details+=("post-reboot nodes=$cur_n edges=$cur_e (pre $pre_n/$pre_e)")
|
||||
|
||||
# 3) RSS bound
|
||||
log "check: RSS bound (< ${RSS_BOUND_MB}MB)"
|
||||
local pid rss_kb rss_mb; pid="$(daemon_pid "$name")"
|
||||
rss_kb="$(ps -o rss= -p "$pid" 2>/dev/null | tr -d ' ')"; rss_mb=$(( ${rss_kb:-0} / 1024 ))
|
||||
names+=("rss-bound"); if [ "$rss_mb" -lt "$RSS_BOUND_MB" ] && [ "$rss_mb" -gt 0 ]; then results+=("PASS"); else results+=("FAIL"); fi
|
||||
details+=("RSS=${rss_mb}MB (bound ${RSS_BOUND_MB}MB)")
|
||||
|
||||
# 4) retrieval parity vs the create-time baseline
|
||||
log "check: retrieval parity vs baseline probe set"
|
||||
_capture_retrieval "$name" "$d/logs/retrieval-$( now ).json"
|
||||
local latest; latest="$(ls -t "$d/logs"/retrieval-*.json 2>/dev/null | head -1)"
|
||||
local parity; parity="$(python3 - "$d/baseline/retrieval.json" "$latest" <<'PY'
|
||||
import json,sys
|
||||
def load(p):
|
||||
try: return json.load(open(p))
|
||||
except Exception: return {}
|
||||
b,c=load(sys.argv[1]),load(sys.argv[2])
|
||||
tot=hit=0
|
||||
for q,ids in b.items():
|
||||
cb=set(ids or []); cc=set(c.get(q) or [])
|
||||
if not cb: continue
|
||||
tot+=len(cb); hit+=len(cb & cc)
|
||||
print(f"{hit}/{tot}" if tot else "0/0")
|
||||
PY
|
||||
)"
|
||||
local ph="${parity%/*}" pt="${parity#*/}"
|
||||
names+=("retrieval-parity"); if [ "${pt:-0}" -gt 0 ] && [ "${ph:-0}" -eq "${pt:-0}" ]; then results+=("PASS"); else results+=("FAIL"); fi
|
||||
details+=("top-k id overlap $parity vs baseline")
|
||||
|
||||
# 5) keystone integrity
|
||||
log "check: keystone integrity"
|
||||
local kfail=0 kid kres
|
||||
for kid in "${KEYSTONES[@]}"; do
|
||||
kres="$(curl -s -m5 "$url/api/node/$kid" 2>/dev/null)"
|
||||
printf '%s' "$kres" | grep -q "\"$kid\"" || kfail=1
|
||||
done
|
||||
names+=("keystone-integrity"); [ "$kfail" -eq 0 ] && results+=("PASS") || results+=("FAIL")
|
||||
details+=("kn-efeb4a5b + kn-5b606390 present")
|
||||
|
||||
# ---- report + stamp ----
|
||||
echo >&2
|
||||
printf '%s VALIDATION — %s%s\n' "$C_BLD" "$name" "$C_0" >&2
|
||||
local allpass=1 j
|
||||
for j in "${!names[@]}"; do
|
||||
local r="${results[$j]}" c="$C_GRN"; [ "$r" = FAIL ] && { c="$C_RED"; allpass=0; }
|
||||
printf ' %s%-6s%s %-22s %s%s%s\n' "$c" "$r" "$C_0" "${names[$j]}" "$C_DIM" "${details[$j]}" "$C_0" >&2
|
||||
done
|
||||
local status; [ "$allpass" -eq 1 ] && status="PASS" || status="FAIL"
|
||||
python3 - "$d/validate.json" "$status" "$(sha "$d/bin/engram")" "$(now)" "${names[*]}" "${results[*]}" <<'PY'
|
||||
import json,sys
|
||||
out,status,binsha,ts,ns,rs=sys.argv[1:7]
|
||||
checks=[{"name":n,"result":r} for n,r in zip(ns.split(),rs.split())]
|
||||
json.dump({"status":status,"binary_sha256":binsha,"ts":ts,"checks":checks},open(out,'w'),indent=2)
|
||||
PY
|
||||
printf ' %s==> %s%s\n' "$([ "$allpass" -eq 1 ] && echo "$C_GRN" || echo "$C_RED")" "$status" "$C_0" >&2
|
||||
[ "$allpass" -eq 1 ]
|
||||
}
|
||||
|
||||
# ================================================================ promote ======
|
||||
# The ONLY prod-touching op. Explicit, gated, per-use Will-approved. Rails ONLY:
|
||||
# snapshot-first -> additive binary swap -> launchctl bootout -> settle-poll ->
|
||||
# bootstrap -> verify -> auto-rollback on failure. NEVER pkill, NEVER kickstart -k.
|
||||
# Default is a DRY-RUN plan; requires --i-approve-prod-cutover to actually cut over.
|
||||
cmd_promote(){
|
||||
local name="$1"; shift || true
|
||||
mexists "$name" || die "no such sandbox: $name"
|
||||
local approve=0 do_data=0
|
||||
while [ $# -gt 0 ]; do case "$1" in
|
||||
--i-approve-prod-cutover) approve=1; shift;;
|
||||
--data) do_data=1; shift;;
|
||||
*) die "unknown flag: $1";; esac; done
|
||||
local d; d="$(sdir "$name")"
|
||||
# GATE 1: validation must have passed for the CURRENT binary
|
||||
[ -f "$d/validate.json" ] || die "GATE: no validation on record — run 'nsbx validate $name' first"
|
||||
local vstatus vsha bsha
|
||||
vstatus="$(python3 -c "import json;print(json.load(open('$d/validate.json'))['status'])")"
|
||||
vsha="$(python3 -c "import json;print(json.load(open('$d/validate.json'))['binary_sha256'])")"
|
||||
bsha="$(sha "$d/bin/engram")"
|
||||
[ "$vstatus" = PASS ] || die "GATE: last validation status is $vstatus (must be PASS)"
|
||||
[ "$vsha" = "$bsha" ] || die "GATE: validation is stale — binary changed since validate (re-run validate)"
|
||||
|
||||
local live_bin new_bin ts; ts="$(now)"
|
||||
live_bin="$(_live_real_bin)"
|
||||
new_bin="$HOME/.neuron/bin/engram.promote-$name-$ts" # additive: new file, old kept
|
||||
local bkp="$BACKUP_ROOT/promote-$name-$ts"
|
||||
|
||||
log "PROMOTE PLAN for '$name' -> live :$LIVE_BIND_PORT"
|
||||
info "current live ENGRAM_REAL_BIN : $live_bin"
|
||||
info "sandbox binary (validated) : $d/bin/engram (sha ${bsha:0:12})"
|
||||
info "will install as : $new_bin (additive; old binary retained)"
|
||||
info "snapshot-first backup dir : $bkp (egm+wal+plist+rollback.txt)"
|
||||
info "data promote : $([ $do_data -eq 1 ] && echo 'YES (--data: clone egm/wal -> live)' || echo 'no (binary only)')"
|
||||
info "rails : launchctl bootout -> settle-poll -> bootstrap"
|
||||
info "verify : /api/stats + edges>=baseline + keystones + retrieval; auto-rollback armed"
|
||||
|
||||
if [ "$approve" -ne 1 ]; then
|
||||
warn "DRY-RUN — not touching prod. Re-run with --i-approve-prod-cutover to execute (per-use Will-approved)."
|
||||
return 0
|
||||
fi
|
||||
|
||||
need launchctl
|
||||
local dom="gui/$(id -u)"
|
||||
# ---- snapshot-first ----
|
||||
log "snapshot-first backup -> $bkp"
|
||||
mkdir -p "$bkp"
|
||||
cp -p "$LIVE_DATA_DIR/neuron.egm" "$bkp/neuron.egm.bak"
|
||||
cp -p "$LIVE_DATA_DIR/neuron.wal" "$bkp/neuron.wal.bak" 2>/dev/null || true
|
||||
cp -p "$LIVE_PLIST" "$bkp/plist.bak"
|
||||
printf 'rollback REAL_BIN=%s\nNEWBIN=%s\ndata_promote=%s\n' "$live_bin" "$new_bin" "$do_data" > "$bkp/rollback.txt"
|
||||
ok "backup complete"
|
||||
|
||||
# ---- additive binary install + plist supersede ----
|
||||
cp -p "$d/bin/engram" "$new_bin"
|
||||
python3 - "$LIVE_PLIST" "$new_bin" <<'PY'
|
||||
import sys,plistlib
|
||||
p,new=sys.argv[1],sys.argv[2]
|
||||
d=plistlib.load(open(p,'rb')); d.setdefault("EnvironmentVariables",{})["ENGRAM_REAL_BIN"]=new
|
||||
plistlib.dump(d,open(p,'wb'))
|
||||
PY
|
||||
ok "installed $new_bin + updated plist ENGRAM_REAL_BIN"
|
||||
|
||||
# ---- optional data promote (after backup) ----
|
||||
if [ $do_data -eq 1 ]; then
|
||||
log "data promote: clone store -> live (backed up above)"
|
||||
cp -p "$d/data/neuron.egm" "$LIVE_DATA_DIR/neuron.egm"
|
||||
cp -p "$d/data/neuron.wal" "$LIVE_DATA_DIR/neuron.wal" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# ---- rails cutover: bootout -> settle-poll -> bootstrap ----
|
||||
log "rails: launchctl bootout $dom/$LIVE_LABEL"
|
||||
launchctl bootout "$dom/$LIVE_LABEL" 2>/dev/null || true
|
||||
local i
|
||||
for i in $(seq 1 60); do
|
||||
launchctl print "$dom/$LIVE_LABEL" >/dev/null 2>&1 || { ok "settle: job gone after ${i}x0.5s"; break; }
|
||||
printf ' settle: job still present (%d)\n' "$i" >&2; sleep 0.5
|
||||
done
|
||||
log "rails: launchctl bootstrap $dom <plist>"
|
||||
launchctl bootstrap "$dom" "$LIVE_PLIST" || warn "bootstrap returned nonzero"
|
||||
|
||||
# ---- verify ----
|
||||
log "verify prod health"
|
||||
local s="" ; for i in $(seq 1 60); do s="$(live_stats)"; [ -n "$s" ] && break; sleep 1; done
|
||||
local ok_verify=1 le; le="$(stat_field "$s" edge_count)"
|
||||
local base_e; base_e="$(mget "$name" "['live_baseline']['edge_count']")"
|
||||
[ -n "$s" ] || ok_verify=0
|
||||
[ "${le:-0}" -ge "${base_e:-0}" ] || ok_verify=0
|
||||
local kid; for kid in "${KEYSTONES[@]}"; do curl -s -m5 "$LIVE_URL/api/node/$kid" 2>/dev/null | grep -q "\"$kid\"" || ok_verify=0; done
|
||||
if [ "$ok_verify" -eq 1 ]; then
|
||||
ok "PROMOTED. live stats: $s (rollback: $bkp)"; return 0
|
||||
fi
|
||||
|
||||
# ---- auto-rollback ----
|
||||
warn "verify FAILED — auto-rollback"
|
||||
cp -p "$bkp/plist.bak" "$LIVE_PLIST"
|
||||
[ $do_data -eq 1 ] && { cp -p "$bkp/neuron.egm.bak" "$LIVE_DATA_DIR/neuron.egm"; cp -p "$bkp/neuron.wal.bak" "$LIVE_DATA_DIR/neuron.wal" 2>/dev/null || true; }
|
||||
launchctl bootout "$dom/$LIVE_LABEL" 2>/dev/null || true
|
||||
for i in $(seq 1 60); do launchctl print "$dom/$LIVE_LABEL" >/dev/null 2>&1 || break; sleep 0.5; done
|
||||
launchctl bootstrap "$dom" "$LIVE_PLIST" || true
|
||||
die "ROLLED BACK to $live_bin. See $bkp"
|
||||
}
|
||||
|
||||
# ================================================================ destroy ======
|
||||
cmd_destroy(){
|
||||
local name="$1"; shift || true
|
||||
mexists "$name" || die "no such sandbox: $name"
|
||||
local d; d="$(sdir "$name")"
|
||||
stop_daemon "$name"
|
||||
if [ -d "$d/build/worktree" ]; then
|
||||
log "removing git worktree"
|
||||
git -C "$EL_REPO" worktree remove --force "$d/build/worktree" 2>/dev/null || true
|
||||
git -C "$EL_REPO" worktree prune 2>/dev/null || true
|
||||
fi
|
||||
log "removing $d"
|
||||
rm -rf "$d"
|
||||
ok "destroyed '$name' (live untouched)"
|
||||
}
|
||||
|
||||
# ================================================================ list/status ==
|
||||
cmd_list(){
|
||||
[ -d "$SBX_ROOT" ] || { echo "no sandboxes"; return 0; }
|
||||
printf '%-16s %-6s %-8s %-9s %s\n' NAME PORT STATE PID SOURCE
|
||||
local m
|
||||
for m in "$SBX_ROOT"/*/manifest.json; do
|
||||
[ -f "$m" ] || continue
|
||||
local n p src pid state
|
||||
n="$(python3 -c "import json;print(json.load(open('$m'))['name'])")"
|
||||
p="$(python3 -c "import json;print(json.load(open('$m'))['port'])")"
|
||||
src="$(python3 -c "import json;print(json.load(open('$m'))['source'])")"
|
||||
pid="$(daemon_pid "$n")"; state="stopped"; daemon_alive "$n" && state="running"
|
||||
printf '%-16s %-6s %-8s %-9s %s\n' "$n" "$p" "$state" "${pid:-–}" "$src"
|
||||
done
|
||||
}
|
||||
cmd_status(){
|
||||
local name="$1"; mexists "$name" || die "no such sandbox: $name"
|
||||
python3 -m json.tool "$(manifest "$name")"
|
||||
daemon_alive "$name" && echo "state: running (pid $(daemon_pid "$name")) stats: $(sbx_stats "$name")" || echo "state: stopped"
|
||||
[ -f "$(sdir "$name")/validate.json" ] && { echo "--- last validation ---"; python3 -m json.tool "$(sdir "$name")/validate.json"; }
|
||||
}
|
||||
|
||||
# ================================================================ dev ==========
|
||||
# ONE-COMMAND isolated dev environment. Everything a newcomer (Tim, any agent)
|
||||
# needs to go from clone -> coding on an isolated running mind, in a single shot:
|
||||
# 1) a git BRANCH (dev/<name>, or --prefix)
|
||||
# 2) a git WORKTREE for it, at a visible path they can open + edit
|
||||
# 3) an ISOLATED engram bound to a NON-default port, on a clone of the live store
|
||||
# (stock prod binary by default — instant + safe; --build to compile the
|
||||
# worktree's own runtime instead). Prod :$LIVE_BIND_PORT/:$SOUL_PORT is untouchable.
|
||||
#
|
||||
# This is additive sugar over the proven primitives (git worktree + cmd_create).
|
||||
# It never binds a forbidden port and never touches ~/.neuron/engram (the live store)
|
||||
# except the same READ-only snapshot cmd_create already performs.
|
||||
#
|
||||
# nsbx dev <name> [--repo R] [--base REF] [--worktree DIR] [--port N]
|
||||
# [--prefix P] [--build] [--no-engram]
|
||||
cmd_dev(){
|
||||
local name="" repo="$EL_REPO" base="" wt="" port="" prefix="dev/" build=0 no_engram=0
|
||||
[ $# -gt 0 ] && [ "${1#-}" = "$1" ] && { name="$1"; shift; } || die "usage: nsbx dev <name> [flags]"
|
||||
while [ $# -gt 0 ]; do case "$1" in
|
||||
--repo) repo="$2"; shift 2;;
|
||||
--base) base="$2"; shift 2;;
|
||||
--worktree|--wt) wt="$2"; shift 2;;
|
||||
--port) port="$2"; shift 2;;
|
||||
--prefix) prefix="$2"; shift 2;;
|
||||
--build) build=1; shift;;
|
||||
--no-engram) no_engram=1; shift;;
|
||||
*) die "unknown flag: $1";;
|
||||
esac; done
|
||||
need git
|
||||
git -C "$repo" rev-parse --git-dir >/dev/null 2>&1 || die "not a git repo: $repo"
|
||||
|
||||
local branch="${prefix}${name}"
|
||||
local sbx="dev-${name}"
|
||||
# default worktree path: a PERSISTENT, git-managed dir — NEVER /tmp (which is
|
||||
# ablated on compaction). Default root = <repo-grandparent>/el-worktrees, i.e.
|
||||
# ~/Development/neuron-technologies/el-worktrees/<name>. Override with NSBX_DEV_WT_ROOT.
|
||||
local wt_root="${NSBX_DEV_WT_ROOT:-$(cd "$(dirname "$(dirname "$repo")")" && pwd -P)/el-worktrees}"
|
||||
[ -n "$wt" ] || wt="${wt_root}/${name}"
|
||||
case "$wt" in /tmp/*|/private/tmp/*|/var/tmp/*)
|
||||
die "refusing worktree under a temp dir ($wt) — temp dirs are ablated on compaction; set NSBX_DEV_WT_ROOT to a persistent path";;
|
||||
esac
|
||||
# default base: whatever the repo's working checkout is on now
|
||||
[ -n "$base" ] || base="$(git -C "$repo" rev-parse --abbrev-ref HEAD 2>/dev/null)"
|
||||
|
||||
# pre-flight (fail before creating anything)
|
||||
[ "$no_engram" -eq 1 ] || ! mexists "$sbx" || die "engram sandbox '$sbx' already exists (nsbx dev-down $name first)"
|
||||
[ -e "$wt" ] && die "worktree path already exists: $wt"
|
||||
if [ -n "$port" ]; then
|
||||
{ [ "$port" = "$LIVE_BIND_PORT" ] || [ "$port" = "$SOUL_PORT" ]; } && die "refusing forbidden port $port (live)"
|
||||
fi
|
||||
|
||||
log "dev env '$name' (branch=$branch worktree=$wt base=$base)"
|
||||
|
||||
# ---- 1+2) branch + worktree in one shot ----
|
||||
local gerr
|
||||
if git -C "$repo" show-ref --verify --quiet "refs/heads/$branch"; then
|
||||
info "branch $branch exists — checking it out into a new worktree"
|
||||
gerr="$(git -C "$repo" worktree add "$wt" "$branch" 2>&1)" \
|
||||
|| die "git worktree add failed for existing branch $branch:"$'\n'" $gerr"
|
||||
else
|
||||
gerr="$(git -C "$repo" worktree add -b "$branch" "$wt" "$base" 2>&1)" \
|
||||
|| die "git worktree add -b $branch (base $base) failed:"$'\n'" $gerr"$'\n'" (a bare 'dev' branch blocks 'dev/*' names — try --prefix, e.g. nsbx dev $name --prefix wt/)"
|
||||
fi
|
||||
ok "worktree ready: $wt (branch $branch)"
|
||||
|
||||
# ---- 3) isolated engram ----
|
||||
local eport="(none)"
|
||||
if [ "$no_engram" -eq 1 ]; then
|
||||
warn "--no-engram: skipped standing up an engram"
|
||||
else
|
||||
if [ "$build" -eq 1 ]; then
|
||||
log "isolated engram: building the worktree's own runtime"
|
||||
cmd_create "$sbx" ${port:+--port "$port"} --source "$wt" || die "engram create (--build) failed"
|
||||
else
|
||||
log "isolated engram: stock prod binary on a clone of the live store"
|
||||
cmd_create "$sbx" ${port:+--port "$port"} || die "engram create failed"
|
||||
fi
|
||||
eport="$(mget "$sbx" "['port']")"
|
||||
# record the dev linkage next to the sandbox so dev-down can clean up
|
||||
python3 - "$(sdir "$sbx")/dev.json" "$name" "$branch" "$wt" "$repo" "$eport" <<'PY'
|
||||
import json,sys
|
||||
p,name,branch,wt,repo,port=sys.argv[1:7]
|
||||
json.dump({"name":name,"branch":branch,"worktree":wt,"repo":repo,"port":int(port)},
|
||||
open(p,'w'),indent=2)
|
||||
PY
|
||||
# ---- pin the WHOLE worktree to the CLONE ----
|
||||
# Every var any El tooling in this worktree might read for an engram target now
|
||||
# points at the isolated clone. Sourcing .nsbx-env makes hitting live :$LIVE_BIND_PORT
|
||||
# or ~/.neuron/engram by accident structurally impossible from this shell.
|
||||
local edata ekey eurl ebin
|
||||
edata="$(sdir "$sbx")/data"; ekey="sbx-$sbx"; eurl="http://127.0.0.1:$eport"; ebin="$(sdir "$sbx")/bin/engram"
|
||||
cat > "$wt/.nsbx-env" <<ENV
|
||||
# nsbx dev env for '$name' — SOURCE this to pin THIS shell to the isolated clone.
|
||||
# The live mind (:$LIVE_BIND_PORT engram / :$SOUL_PORT soul / $LIVE_DATA_DIR) is deliberately
|
||||
# NOT referenced here. Regenerated by 'nsbx dev'. -> source .nsbx-env
|
||||
export NSBX_NAME="$sbx"
|
||||
export ENGRAM_URL="$eurl"
|
||||
export ENGRAM_BIND=":$eport"
|
||||
export ENGRAM_PORT="$eport"
|
||||
export ENGRAM_HOST="127.0.0.1"
|
||||
export ENGRAM_DATA_DIR="$edata"
|
||||
export ENGRAM_API_KEY="$ekey"
|
||||
export NEURON_ENGRAM_URL="$eurl"
|
||||
export NEURON_ENGRAM_KEY="$ekey"
|
||||
# nsbx run compatibility (same names 'nsbx run' exports)
|
||||
export SBX_NAME="$sbx" SBX_PORT="$eport" SBX_URL="$eurl" SBX_KEY="$ekey" SBX_DATA="$edata" SBX_BIN="$ebin"
|
||||
ENV
|
||||
# direnv users get it automatically on cd; everyone else runs 'source .nsbx-env'
|
||||
[ -e "$wt/.envrc" ] || printf 'source_env .nsbx-env 2>/dev/null || source .nsbx-env\n' > "$wt/.envrc"
|
||||
ok "wrote $wt/.nsbx-env (pins this worktree to the clone)"
|
||||
fi
|
||||
|
||||
# ---- summary ----
|
||||
echo >&2
|
||||
printf '%s DEV ENV READY — %s%s\n' "$C_BLD" "$name" "$C_0" >&2
|
||||
printf ' %-10s %s\n' "worktree" "$wt" >&2
|
||||
printf ' %-10s %s\n' "branch" "$branch" >&2
|
||||
if [ "$no_engram" -ne 1 ]; then
|
||||
printf ' %-10s %s\n' "engram" "http://127.0.0.1:$eport (isolated clone; prod :$LIVE_BIND_PORT untouchable)" >&2
|
||||
printf ' %-10s %s\n' "sandbox" "$sbx" >&2
|
||||
echo >&2
|
||||
printf '%s env exported into %s/.nsbx-env (source it -> pinned to the clone):%s\n' "$C_DIM" "$wt" "$C_0" >&2
|
||||
grep '^export' "$wt/.nsbx-env" | sed 's/^/ /' >&2
|
||||
fi
|
||||
echo >&2
|
||||
info "code in: cd $wt && source .nsbx-env # now every engram var points at the clone"
|
||||
[ "$no_engram" -ne 1 ] && info "poke it: nsbx run $sbx api /api/stats (or: make run NAME=$name)"
|
||||
[ "$no_engram" -ne 1 ] && info "edit->test: make build NAME=$name && make run NAME=$name # El change -> clone, seconds"
|
||||
info "tear down: nsbx dev-down $name (destroys engram + removes worktree; branch kept)"
|
||||
}
|
||||
|
||||
# nsbx dev-down <name> [--delete-branch] [--repo R]
|
||||
# Teardown counterpart: destroy the isolated engram, remove the git worktree,
|
||||
# and (optionally) delete the branch. Live prod is never touched.
|
||||
cmd_dev_down(){
|
||||
local name="" repo="$EL_REPO" del_branch=0
|
||||
[ $# -gt 0 ] && [ "${1#-}" = "$1" ] && { name="$1"; shift; } || die "usage: nsbx dev-down <name> [--delete-branch]"
|
||||
while [ $# -gt 0 ]; do case "$1" in
|
||||
--repo) repo="$2"; shift 2;;
|
||||
--delete-branch) del_branch=1; shift;;
|
||||
*) die "unknown flag: $1";;
|
||||
esac; done
|
||||
local sbx="dev-${name}" wt="" branch="dev/${name}"
|
||||
# recover worktree/branch/repo from the dev linkage if present
|
||||
if mexists "$sbx" && [ -f "$(sdir "$sbx")/dev.json" ]; then
|
||||
local dj; dj="$(sdir "$sbx")/dev.json"
|
||||
wt="$(python3 -c "import json;print(json.load(open('$dj'))['worktree'])" 2>/dev/null)"
|
||||
branch="$(python3 -c "import json;print(json.load(open('$dj'))['branch'])" 2>/dev/null)"
|
||||
repo="$(python3 -c "import json;print(json.load(open('$dj'))['repo'])" 2>/dev/null)"
|
||||
fi
|
||||
# 1) engram
|
||||
if mexists "$sbx"; then cmd_destroy "$sbx"; else info "no engram sandbox '$sbx'"; fi
|
||||
# 2) worktree
|
||||
if [ -n "$wt" ] && [ -d "$wt" ]; then
|
||||
log "removing git worktree $wt"
|
||||
git -C "$repo" worktree remove --force "$wt" 2>/dev/null || rm -rf "$wt"
|
||||
git -C "$repo" worktree prune 2>/dev/null || true
|
||||
ok "worktree removed"
|
||||
else info "no worktree to remove"; fi
|
||||
# 3) branch (opt-in)
|
||||
if [ "$del_branch" -eq 1 ]; then
|
||||
git -C "$repo" branch -D "$branch" 2>/dev/null && ok "deleted branch $branch" || warn "could not delete branch $branch"
|
||||
else info "branch $branch kept (use --delete-branch to remove)"; fi
|
||||
ok "dev-down '$name' complete (live untouched)"
|
||||
}
|
||||
|
||||
usage(){ cat >&2 <<EOF
|
||||
${C_BLD}nsbx${C_0} — Neuron Sandbox: experiments + code changes against the REAL engram
|
||||
runtime on an isolated snapshot of the live mind, with a gated promote-to-prod path.
|
||||
|
||||
nsbx dev <name> [--base REF] [--worktree DIR] ONE command onboarding: new branch (dev/<name>) + git
|
||||
[--port N] [--prefix P] [--build] worktree (persistent, never /tmp) + isolated engram on a
|
||||
[--no-engram] [--repo R] non-default port. clone -> coding on the mind in one shot.
|
||||
nsbx dev-down <name> [--delete-branch] [--repo R] teardown: destroy the engram + remove the worktree
|
||||
(branch kept unless --delete-branch). live untouched.
|
||||
nsbx up [name] [flags…] one command: your private, isolated copy of the mind
|
||||
(creates on first run, starts thereafter; prod untouchable)
|
||||
nsbx create [name] [--port N] [--source DIR | --branch REF [--repo R] | --binary PATH]
|
||||
clone live store+WAL+config, place/build the runtime, boot on an
|
||||
isolated port (never :$LIVE_BIND_PORT/:$SOUL_PORT). Default runtime = stock prod binary.
|
||||
nsbx build <name> --source DIR | --branch REF rebuild the runtime from a code change + hot-restart
|
||||
nsbx run <name> <cmd...> | api <path> [json] run an experiment; capture output + metrics
|
||||
nsbx validate <name> rails as checks: zero-loss(load+reboot), reboot-prove,
|
||||
RSS bound, retrieval parity, keystone integrity
|
||||
nsbx promote <name> [--data] [--i-approve-prod-cutover] GATED rails cutover to prod (DRY-RUN without approval)
|
||||
nsbx destroy <name> stop daemon, free port, remove clone (live untouched)
|
||||
nsbx list | nsbx status <name>
|
||||
|
||||
Env in 'run' cmds: \$SBX_URL \$SBX_PORT \$SBX_KEY \$SBX_DATA \$SBX_BIN \$SBX_NAME
|
||||
EOF
|
||||
}
|
||||
|
||||
main(){
|
||||
local cmd="${1:-}"; shift || true
|
||||
case "$cmd" in
|
||||
dev) cmd_dev "$@";;
|
||||
dev-down) cmd_dev_down "$@";;
|
||||
up) cmd_up "$@";;
|
||||
create) cmd_create "$@";;
|
||||
build) cmd_build "$@";;
|
||||
run) cmd_run "$@";;
|
||||
validate) cmd_validate "$@";;
|
||||
promote) cmd_promote "$@";;
|
||||
destroy) cmd_destroy "$@";;
|
||||
list|ls) cmd_list "$@";;
|
||||
status) cmd_status "$@";;
|
||||
""|-h|--help|help) usage;;
|
||||
*) die "unknown command: $cmd (try: nsbx help)";;
|
||||
esac
|
||||
}
|
||||
main "$@"
|
||||
Reference in New Issue
Block a user