diff --git a/lang/swarm/README.md b/lang/swarm/README.md new file mode 100644 index 0000000..9fe4a65 --- /dev/null +++ b/lang/swarm/README.md @@ -0,0 +1,115 @@ +# Swarm + CCR + Work-Tracking — Neuron's bounded parallel execution, in native El + +Bounded parallel agent execution on El's **native** concurrency — no external +orchestrator. Grounded directly in two of Will's frameworks: + +- **Swarm Architecture** (*Bounded Parallel Agent Execution*, Mar 2026) +- **Compiled Context Runtime / CCR** (*Process-Driven Agent Execution with + Unbounded Local Memory*, Mar 2026) + +A swarm is a **coordinator** (the main thread) that mints a correlation identity, +compiles a **bounded per-worker context (CCR)**, dispatches workers as **native +pthreads** (`thread.el` `spawn`/`join`), tracks every unit of work durably, and +**converges** results before returning control to the parent step. + +``` +Parent step + └─ swarm_run(blueprint, knowledge_refs, inputs, config) + fan-out ──▶ worker_1 (CCR ctx_1) ─┐ native + worker_2 (CCR ctx_2) ─┤ pthreads, + worker_k (CCR ctx_k) ─┘ bounded by `concurrency` + converge ─▶ collect | merge | vote | reduce ──▶ merged result +``` + +## Why it runs on El natively + +El is natively agentic. This capability composes El's shipped primitives — it +adds no bespoke runtime: + +| Primitive | Source | Role in the swarm | +|-----------|--------|-------------------| +| `spawn(fn,arg)` / `join(tid)` | `runtime/thread.el` → `__thread_create` (pthread + dlsym) | fan-out / rejoin | +| `parallel_map`, `with_mutex` | `runtime/thread.el` | reference concurrency patterns | +| Go-style channels | `runtime/channel.el` → `__channel_*` | available for vertical event streams | +| `engram_*`, `http_*`, `fs_*`, `json_*` | `el_runtime.c` builtins | retrieval, tracking, I/O | + +Every El fn compiles to a global C symbol, so any top-level `(String)->String` +fn is directly threadable — the worker entry is exactly such a fn. + +## Modules + +| File | Framework grounding | What it does | +|------|--------------------|--------------| +| `worktrack.el` | Swarm §6 (correlation IDs, audit) | Durable, single-writer **JSONL journal** keyed by correlation ID; reconstructable status report; opt-in engram mirror (`SWARM_MIRROR=1`). | +| `containment.el` | Swarm §3 (the three rules) | Scope tokens; **Rule 1** (no join), **Rule 2** (no open), **Rule 3** (no lateral edge) enforced as checks. | +| `ccr.el` | CCR §5 + Swarm §9.3 | Per-worker **Compiled Context Routing**: retrieve → scope → compact into a **bounded, minimal** package. The compiled-context boundary *is* the security boundary. | +| `primitives.el` | CCR §2 (Five Primitives) | `attend / think / intend / act / learn` seam the swarm composes over. Engram-backed; explicit binding point for the API-surface reshape. | +| `swarm.el` | Swarm §2, §4, §5 | The coordinator: fan-out/converge on native threads, bounded concurrency, four convergence strategies, integer failure threshold, full tracking. | + +## Containment → distribution + +The three containment rules make workers **location-independent** (Swarm §9): a +worker reads only its compiled context, shares no state with siblings, and its +only outward edge is the returned result. The same coordinator can run workers +as local threads today or dispatch them across machines later — the mechanism is +identical; only the topology changes. Enforced here: + +- **Rule 2** — `swarm_run` rejects any swarm opened under a worker token. +- **Rules 1 + 3** — each worker gets a *closed* worker token; the coordinator is + the only journal writer, so workers share no mutable state. + +## Usage + +```el +// one process step fans out; results converge before the next step +let inputs: String = "[\"billing\",\"payments\",\"ledger\"]" +let refs: String = "[\"Volatility-Based Decomposition\"]" // CCR knowledge refs +let cfg: String = "{\"concurrency\":\"4\",\"strategy\":\"collect\",\"min_success_ratio\":\"1.0\"}" +let result: String = swarm_run("analyze_item", refs, inputs, cfg) +// result: { corr_id, status, merged, report } +``` + +Build any program that uses the swarm: + +```bash +lang/swarm/build.sh myprog.el ./myprog # concat + elc + cc (el_runtime.c) +``` + +Config keys: `concurrency` (max workers at once), `strategy` +(`collect|merge|vote|reduce`), `min_success_ratio` (decimal string, e.g. `0.8`), +`caller_token` (containment). Env: `SWARM_TRACK_DIR` (journal dir), +`CCR_TOKEN_BUDGET`, `ENGRAM_URL`/`ENGRAM_API_KEY` (retrieval + mirror), +`SWARM_MIRROR=1`. + +## Tests + +```bash +lang/swarm/build.sh lang/swarm/tests/test_swarm.el /tmp/t && SWARM_TRACK_DIR=/tmp/trk /tmp/t # 12/12 +lang/swarm/build.sh lang/swarm/tests/test_convergence.el /tmp/c && SWARM_TRACK_DIR=/tmp/trk /tmp/c # 8/8 +# integration against an isolated engram clone (never live): +source /.nsbx-env +lang/swarm/build.sh lang/swarm/tests/integ_engram.el /tmp/i && /tmp/i +``` + +## Built vs stubbed (honest) + +**Real, tested:** +- Native-thread fan-out/converge, bounded concurrency, order-preserving rejoin. +- All three containment rules enforced (scope tokens + lateral-edge check). +- CCR per-worker context: retrieval → scoping → compaction, bounded, non-leaking + (a worker never receives sibling inputs) — verified against the live isolated mind. +- Full durable work-tracking (JSONL journal, reconstructable report). +- Four convergence strategies + integer failure threshold / partial-abort. + +**Seam / not yet bound:** +- `primitives.el` `think` is a deterministic, hermetic transform (no model call). + Binding point is marked `PRIMITIVE_BINDING`; wire to the API-surface reshape's + `think/act/attend/intend/learn` when it lands. +- Blueprints are dispatched by name in `swarm_run_blueprint` (default + + `classify`/`faildemo` demos). A YAML process-definition loader (Swarm §5) is + future work — the runtime contract is in place. +- Distributed placement (cloud/edge/federated topologies, Swarm §9.2) is + structurally enabled by containment but not yet wired to a placement layer; + today all workers are local native threads. +- Engram work-tracking mirror is opt-in; the durable substrate is the journal. +