el/lang/swarm/README.md

# Swarm + CCR + Work-Tracking — Neuron's bounded parallel execution, in native El

Bounded parallel agent execution on El's **native** concurrency — no external
orchestrator. Grounded directly in two of Will's frameworks:

- **Swarm Architecture** (*Bounded Parallel Agent Execution*, Mar 2026)
- **Compiled Context Runtime / CCR** (*Process-Driven Agent Execution with
  Unbounded Local Memory*, Mar 2026)

A swarm is a **coordinator** (the main thread) that mints a correlation identity,
compiles a **bounded per-worker context (CCR)**, dispatches workers as **native
pthreads** (`thread.el` `spawn`/`join`), tracks every unit of work durably, and
**converges** results before returning control to the parent step.

```
Parent step
  └─ swarm_run(blueprint, knowledge_refs, inputs, config)
        fan-out ──▶ worker_1 (CCR ctx_1) ─┐   native
                    worker_2 (CCR ctx_2) ─┤   pthreads,
                    worker_k (CCR ctx_k) ─┘   bounded by `concurrency`
        converge ─▶ collect | merge | vote | reduce  ──▶ merged result
```

## Why it runs on El natively

El is natively agentic. This capability composes El's shipped primitives — it
adds no bespoke runtime:

| Primitive | Source | Role in the swarm |
|-----------|--------|-------------------|
| `spawn(fn,arg)` / `join(tid)` | `runtime/thread.el` → `__thread_create` (pthread + dlsym) | fan-out / rejoin |
| `parallel_map`, `with_mutex` | `runtime/thread.el` | reference concurrency patterns |
| Go-style channels | `runtime/channel.el` → `__channel_*` | available for vertical event streams |
| `engram_*`, `http_*`, `fs_*`, `json_*` | `el_runtime.c` builtins | retrieval, tracking, I/O |

Every El fn compiles to a global C symbol, so any top-level `(String)->String`
fn is directly threadable — the worker entry is exactly such a fn.

## Modules

| File | Framework grounding | What it does |
|------|--------------------|--------------|
| `worktrack.el` | Swarm §6 (correlation IDs, audit) | Durable, single-writer **JSONL journal** keyed by correlation ID; reconstructable status report; opt-in engram mirror (`SWARM_MIRROR=1`). |
| `containment.el` | Swarm §3 + the single-writer invariant | Scope tokens w/ capabilities; **Rule 1** (no join), **Rule 2** (no open), **Rule 3** (no lateral edge), **Rule 4** (engram-write is @manager-only, by capability) enforced as checks. |
| `ccr.el` | CCR §5 + Swarm §9.3 | Per-worker **Compiled Context Routing**: retrieve → scope → compact into a **bounded, minimal** package. The compiled-context boundary *is* the security boundary. |
| `primitives.el` | CCR §2 (Five Primitives) | `attend / think / intend / act / learn` seam the swarm composes over. Engram-backed; explicit binding point for the API-surface reshape. |
| `swarm.el` | Swarm §2, §4, §5 | The coordinator: fan-out/converge on native threads, bounded concurrency, four convergence strategies, integer failure threshold, full tracking. |

## Invariant: only the orchestrator mutates global engram state

**Only the orchestrator (@manager) writes to the engram / mutates global state.
Workers are read-only against the full engram and may write only their own local
geometry (their returned result + the journal). A worker is STRUCTURALLY UNABLE
to mutate global engram state.**

This is **Rule 4** — an **authority gate, not a health gate**. Scope tokens carry
a capability set: the orchestrator's token holds `engram:write` + `dharma:emit`
(@manager-only, the VBD rule that only the manager mutates global state); a
worker's token holds **only** `engram:read`. Every engram mutation
(`op_write`/`op_relate`/`op_supersede` → `POST /api/nodes`, `/api/edges`,
`DELETE`) flows through `swarm_engram_write`, which checks the caller's capability
via the **same scope-token mechanism as the live Rule-2 denial** and rejects any
worker **before any HTTP is issued**. Capability is fixed at mint time and cannot
be acquired at runtime — so the guarantee holds regardless of engram health
(distinct from the `SWARM_WRITE_HEALTHY` *health* gate).

The **curated merge is the only write path**: workers return geometry; the
orchestrator, and only the orchestrator, commits the approved/verified geometry
back (`commit=1`). Workers keep full-engram **read** access (`op_think`/`op_read`).

Proven in `harness_real_cognition.el` (§G): a worker `swarm_engram_write` is
DENIED by capability with no node created and the violation journalled; the
orchestrator passes the gate as the sole authorized writer.

## Containment → distribution

The three containment rules make workers **location-independent** (Swarm §9): a
worker reads only its compiled context, shares no state with siblings, and its
only outward edge is the returned result. The same coordinator can run workers
as local threads today or dispatch them across machines later — the mechanism is
identical; only the topology changes. Enforced here:

- **Rule 2** — `swarm_run` rejects any swarm opened under a worker token.
- **Rules 1 + 3** — each worker gets a *closed* worker token; the coordinator is
  the only journal writer, so workers share no mutable state.

## Usage

```el
// one process step fans out; results converge before the next step
let inputs: String = "[\"billing\",\"payments\",\"ledger\"]"
let refs:   String = "[\"Volatility-Based Decomposition\"]"   // CCR knowledge refs
let cfg:    String = "{\"concurrency\":\"4\",\"strategy\":\"collect\",\"min_success_ratio\":\"1.0\"}"
let result: String = swarm_run("analyze_item", refs, inputs, cfg)
// result: { corr_id, status, merged, report }
```

Build any program that uses the swarm:

```bash
lang/swarm/build.sh myprog.el ./myprog     # concat + elc + cc (el_runtime.c)
```

Config keys: `concurrency` (max workers at once), `strategy`
(`collect|merge|vote|reduce`), `min_success_ratio` (decimal string, e.g. `0.8`),
`caller_token` (containment). Env: `SWARM_TRACK_DIR` (journal dir),
`CCR_TOKEN_BUDGET`, `ENGRAM_URL`/`ENGRAM_API_KEY` (retrieval + mirror),
`SWARM_MIRROR=1`.

## Tests

```bash
lang/swarm/build.sh lang/swarm/tests/test_swarm.el       /tmp/t && SWARM_TRACK_DIR=/tmp/trk /tmp/t   # 12/12
lang/swarm/build.sh lang/swarm/tests/test_convergence.el /tmp/c && SWARM_TRACK_DIR=/tmp/trk /tmp/c   #  8/8
# integration against an isolated engram clone (never live):
source <sandbox>/.nsbx-env
lang/swarm/build.sh lang/swarm/tests/integ_engram.el /tmp/i && /tmp/i
```

## Local-swarm integration harness (the one flip)

`tests/harness_local_swarm.el` proves the **full local-swarm mechanics today** on
the isolated clone with the primitive seam pointed at the hermetic stub — 17/17
green: 8 native-thread workers at concurrency 4, reduce + vote convergence, CCR
scoping + non-leak, all three containment rules (incl. live Rule-2 denial),
durable work-tracking, and **afferent telemetry** observed by the @manager.

Binding to the reshape's decorated primitives is **one flip and a run**:

```
# in primitive_binding.el — change one line each:
fn bound_think(ctx, instruction) { return think(ctx, instruction) }   # decorated, dharma bus
# then:
SWARM_PRIMITIVE_SEAM=decorated  lang/swarm/build.sh tests/harness_local_swarm.el ./h && ./h
```

Nothing else in the swarm changes. `primitive_seam.el` (`seam_think/attend/learn`)
already routes every worker primitive call through this one switch, and the same
harness runs the bound path. Today `SWARM_PRIMITIVE_SEAM=decorated` still runs
green because the binding falls back to the stub — proving the flip path executes.

## Real cognition — the seam is BOUND

`primitive_binding.el` is bound to the api-reshape agent's proven primitives
(`wt/api-reshape@d4f401d`): `bound_think -> op_think` (GET `/api/think`), real
768-dim gradients over the engram geometry. `reshape_surface.el` composes those
read/cognition primitives verbatim (`op_think/read/attend/learn`).

`tests/harness_real_cognition.el` runs the **local swarm on real cognition**,
17/17 green with `SWARM_PRIMITIVE_SEAM=decorated` against the `:8901` clone: 8
native-thread workers, each a real `think` over its CCR-scoped **node-id anchor**
(free-text anchors return "geometry unavailable"), `@manager` reduce+vote, all
three containment rules, afferent telemetry, durable tracking. Per-anchor support
counts (e.g. 6 / 16 / 87) drive a genuine, cognition-derived vote.

> **Build note (load-bearing):** the swarm build **must** define `HAVE_CURL`
> (`build.sh` does). Without it every `http_*` builtin is a
> `{"error":"not built with HAVE_CURL"}` stub — real HTTP silently disappears.

Writes (`attend`/`learn`, `POST`) are gated behind `SWARM_WRITE_HEALTHY=1` and the
api-reshape agent's gate-1 write-healthy clone; the proven run is read-cognition.

## Built vs stubbed (honest)

**Real, tested:**
- Native-thread fan-out/converge, bounded concurrency, order-preserving rejoin.
- All three containment rules enforced (scope tokens + lateral-edge check).
- CCR per-worker context: retrieval → scoping → compaction, bounded, non-leaking
  (a worker never receives sibling inputs) — verified against the live isolated mind.
- Full durable work-tracking (JSONL journal, reconstructable report).
- Four convergence strategies + integer failure threshold / partial-abort.

**Seam / not yet bound:**
- `primitives.el` `think` is a deterministic, hermetic transform (no model call).
  Binding point is marked `PRIMITIVE_BINDING`; wire to the API-surface reshape's
  `think/act/attend/intend/learn` when it lands.
- Blueprints are dispatched by name in `swarm_run_blueprint` (default +
  `classify`/`faildemo` demos). A YAML process-definition loader (Swarm §5) is
  future work — the runtime contract is in place.
- Distributed placement (cloud/edge/federated topologies, Swarm §9.2) is
  structurally enabled by containment but not yet wired to a placement layer;
  today all workers are local native threads.
- Engram work-tracking mirror is opt-in; the durable substrate is the journal.