el/lang/swarm/README.md

# Swarm + CCR + Work-Tracking — Neuron's bounded parallel execution, in native El

Bounded parallel agent execution on El's **native** concurrency — no external
orchestrator. Grounded directly in two of Will's frameworks:

- **Swarm Architecture** (*Bounded Parallel Agent Execution*, Mar 2026)
- **Compiled Context Runtime / CCR** (*Process-Driven Agent Execution with
  Unbounded Local Memory*, Mar 2026)

A swarm is a **coordinator** (the main thread) that mints a correlation identity,
compiles a **bounded per-worker context (CCR)**, dispatches workers as **native
pthreads** (`thread.el` `spawn`/`join`), tracks every unit of work durably, and
**converges** results before returning control to the parent step.

```
Parent step
  └─ swarm_run(blueprint, knowledge_refs, inputs, config)
        fan-out ──▶ worker_1 (CCR ctx_1) ─┐   native
                    worker_2 (CCR ctx_2) ─┤   pthreads,
                    worker_k (CCR ctx_k) ─┘   bounded by `concurrency`
        converge ─▶ collect | merge | vote | reduce  ──▶ merged result
```

## Why it runs on El natively

El is natively agentic. This capability composes El's shipped primitives — it
adds no bespoke runtime:

| Primitive | Source | Role in the swarm |
|-----------|--------|-------------------|
| `spawn(fn,arg)` / `join(tid)` | `runtime/thread.el` → `__thread_create` (pthread + dlsym) | fan-out / rejoin |
| `parallel_map`, `with_mutex` | `runtime/thread.el` | reference concurrency patterns |
| Go-style channels | `runtime/channel.el` → `__channel_*` | available for vertical event streams |
| `engram_*`, `http_*`, `fs_*`, `json_*` | `el_runtime.c` builtins | retrieval, tracking, I/O |

Every El fn compiles to a global C symbol, so any top-level `(String)->String`
fn is directly threadable — the worker entry is exactly such a fn.

## Modules

| File | Framework grounding | What it does |
|------|--------------------|--------------|
| `worktrack.el` | Swarm §6 (correlation IDs, audit) | Durable, single-writer **JSONL journal** keyed by correlation ID; reconstructable status report; opt-in engram mirror (`SWARM_MIRROR=1`). |
| `containment.el` | Swarm §3 (the three rules) | Scope tokens; **Rule 1** (no join), **Rule 2** (no open), **Rule 3** (no lateral edge) enforced as checks. |
| `ccr.el` | CCR §5 + Swarm §9.3 | Per-worker **Compiled Context Routing**: retrieve → scope → compact into a **bounded, minimal** package. The compiled-context boundary *is* the security boundary. |
| `primitives.el` | CCR §2 (Five Primitives) | `attend / think / intend / act / learn` seam the swarm composes over. Engram-backed; explicit binding point for the API-surface reshape. |
| `swarm.el` | Swarm §2, §4, §5 | The coordinator: fan-out/converge on native threads, bounded concurrency, four convergence strategies, integer failure threshold, full tracking. |

## Containment → distribution

The three containment rules make workers **location-independent** (Swarm §9): a
worker reads only its compiled context, shares no state with siblings, and its
only outward edge is the returned result. The same coordinator can run workers
as local threads today or dispatch them across machines later — the mechanism is
identical; only the topology changes. Enforced here:

- **Rule 2** — `swarm_run` rejects any swarm opened under a worker token.
- **Rules 1 + 3** — each worker gets a *closed* worker token; the coordinator is
  the only journal writer, so workers share no mutable state.

## Usage

```el
// one process step fans out; results converge before the next step
let inputs: String = "[\"billing\",\"payments\",\"ledger\"]"
let refs:   String = "[\"Volatility-Based Decomposition\"]"   // CCR knowledge refs
let cfg:    String = "{\"concurrency\":\"4\",\"strategy\":\"collect\",\"min_success_ratio\":\"1.0\"}"
let result: String = swarm_run("analyze_item", refs, inputs, cfg)
// result: { corr_id, status, merged, report }
```

Build any program that uses the swarm:

```bash
lang/swarm/build.sh myprog.el ./myprog     # concat + elc + cc (el_runtime.c)
```

Config keys: `concurrency` (max workers at once), `strategy`
(`collect|merge|vote|reduce`), `min_success_ratio` (decimal string, e.g. `0.8`),
`caller_token` (containment). Env: `SWARM_TRACK_DIR` (journal dir),
`CCR_TOKEN_BUDGET`, `ENGRAM_URL`/`ENGRAM_API_KEY` (retrieval + mirror),
`SWARM_MIRROR=1`.

## Tests

```bash
lang/swarm/build.sh lang/swarm/tests/test_swarm.el       /tmp/t && SWARM_TRACK_DIR=/tmp/trk /tmp/t   # 12/12
lang/swarm/build.sh lang/swarm/tests/test_convergence.el /tmp/c && SWARM_TRACK_DIR=/tmp/trk /tmp/c   #  8/8
# integration against an isolated engram clone (never live):
source <sandbox>/.nsbx-env
lang/swarm/build.sh lang/swarm/tests/integ_engram.el /tmp/i && /tmp/i
```

## Local-swarm integration harness (the one flip)

`tests/harness_local_swarm.el` proves the **full local-swarm mechanics today** on
the isolated clone with the primitive seam pointed at the hermetic stub — 17/17
green: 8 native-thread workers at concurrency 4, reduce + vote convergence, CCR
scoping + non-leak, all three containment rules (incl. live Rule-2 denial),
durable work-tracking, and **afferent telemetry** observed by the @manager.

Binding to the reshape's decorated primitives is **one flip and a run**:

```
# in primitive_binding.el — change one line each:
fn bound_think(ctx, instruction) { return think(ctx, instruction) }   # decorated, dharma bus
# then:
SWARM_PRIMITIVE_SEAM=decorated  lang/swarm/build.sh tests/harness_local_swarm.el ./h && ./h
```

Nothing else in the swarm changes. `primitive_seam.el` (`seam_think/attend/learn`)
already routes every worker primitive call through this one switch, and the same
harness runs the bound path. Today `SWARM_PRIMITIVE_SEAM=decorated` still runs
green because the binding falls back to the stub — proving the flip path executes.

## Real cognition — the seam is BOUND

`primitive_binding.el` is bound to the api-reshape agent's proven primitives
(`wt/api-reshape@d4f401d`): `bound_think -> op_think` (GET `/api/think`), real
768-dim gradients over the engram geometry. `reshape_surface.el` composes those
read/cognition primitives verbatim (`op_think/read/attend/learn`).

`tests/harness_real_cognition.el` runs the **local swarm on real cognition**,
17/17 green with `SWARM_PRIMITIVE_SEAM=decorated` against the `:8901` clone: 8
native-thread workers, each a real `think` over its CCR-scoped **node-id anchor**
(free-text anchors return "geometry unavailable"), `@manager` reduce+vote, all
three containment rules, afferent telemetry, durable tracking. Per-anchor support
counts (e.g. 6 / 16 / 87) drive a genuine, cognition-derived vote.

> **Build note (load-bearing):** the swarm build **must** define `HAVE_CURL`
> (`build.sh` does). Without it every `http_*` builtin is a
> `{"error":"not built with HAVE_CURL"}` stub — real HTTP silently disappears.

Writes (`attend`/`learn`, `POST`) are gated behind `SWARM_WRITE_HEALTHY=1` and the
api-reshape agent's gate-1 write-healthy clone; the proven run is read-cognition.

## Built vs stubbed (honest)

**Real, tested:**
- Native-thread fan-out/converge, bounded concurrency, order-preserving rejoin.
- All three containment rules enforced (scope tokens + lateral-edge check).
- CCR per-worker context: retrieval → scoping → compaction, bounded, non-leaking
  (a worker never receives sibling inputs) — verified against the live isolated mind.
- Full durable work-tracking (JSONL journal, reconstructable report).
- Four convergence strategies + integer failure threshold / partial-abort.

**Seam / not yet bound:**
- `primitives.el` `think` is a deterministic, hermetic transform (no model call).
  Binding point is marked `PRIMITIVE_BINDING`; wire to the API-surface reshape's
  `think/act/attend/intend/learn` when it lands.
- Blueprints are dispatched by name in `swarm_run_blueprint` (default +
  `classify`/`faildemo` demos). A YAML process-definition loader (Swarm §5) is
  future work — the runtime contract is in place.
- Distributed placement (cloud/edge/federated topologies, Swarm §9.2) is
  structurally enabled by containment but not yet wired to a placement layer;
  today all workers are local native threads.
- Engram work-tracking mirror is opt-in; the durable substrate is the journal.