el/lang/swarm/README.md

Swarm + CCR + Work-Tracking — Neuron's bounded parallel execution, in native El

Bounded parallel agent execution on El's native concurrency — no external orchestrator. Grounded directly in two of Will's frameworks:

• Swarm Architecture (Bounded Parallel Agent Execution, Mar 2026)
• Compiled Context Runtime / CCR (Process-Driven Agent Execution with Unbounded Local Memory, Mar 2026)

A swarm is a coordinator (the main thread) that mints a correlation identity, compiles a bounded per-worker context (CCR), dispatches workers as native pthreads (thread.el spawn/join), tracks every unit of work durably, and converges results before returning control to the parent step.

Parent step
  └─ swarm_run(blueprint, knowledge_refs, inputs, config)
        fan-out ──▶ worker_1 (CCR ctx_1) ─┐   native
                    worker_2 (CCR ctx_2) ─┤   pthreads,
                    worker_k (CCR ctx_k) ─┘   bounded by `concurrency`
        converge ─▶ collect | merge | vote | reduce  ──▶ merged result

Why it runs on El natively

El is natively agentic. This capability composes El's shipped primitives — it adds no bespoke runtime:

Primitive	Source	Role in the swarm
spawn(fn,arg) / join(tid)	runtime/thread.el → __thread_create (pthread + dlsym)	fan-out / rejoin
parallel_map, with_mutex	runtime/thread.el	reference concurrency patterns
Go-style channels	runtime/channel.el → __channel_*	available for vertical event streams
engram_*, http_*, fs_*, json_*	el_runtime.c builtins	retrieval, tracking, I/O

Every El fn compiles to a global C symbol, so any top-level (String)->String fn is directly threadable — the worker entry is exactly such a fn.

Modules

File	Framework grounding	What it does
worktrack.el	Swarm §6 (correlation IDs, audit)	Durable, single-writer JSONL journal keyed by correlation ID; reconstructable status report; opt-in engram mirror (SWARM_MIRROR=1).
containment.el	Swarm §3 + the single-writer invariant	Scope tokens w/ capabilities; Rule 1 (no join), Rule 2 (no open), Rule 3 (no lateral edge), Rule 4 (engram-write is @manager-only, by capability) enforced as checks.
ccr.el	CCR §5 + Swarm §9.3	Per-worker Compiled Context Routing: retrieve → scope → compact into a bounded, minimal package. The compiled-context boundary is the security boundary.
primitives.el	CCR §2 (Five Primitives)	attend / think / intend / act / learn seam the swarm composes over. Engram-backed; explicit binding point for the API-surface reshape.
swarm.el	Swarm §2, §4, §5	The coordinator: fan-out/converge on native threads, bounded concurrency, four convergence strategies, integer failure threshold, full tracking.

Invariant: only the orchestrator mutates global engram state

Only the orchestrator (@manager) writes to the engram / mutates global state. Workers are read-only against the full engram and may write only their own local geometry (their returned result + the journal). A worker is STRUCTURALLY UNABLE to mutate global engram state.

This is Rule 4 — an authority gate, not a health gate. Scope tokens carry a capability set: the orchestrator's token holds engram:write + dharma:emit (@manager-only, the VBD rule that only the manager mutates global state); a worker's token holds only engram:read. Every engram mutation (op_write/op_relate/op_supersede → POST /api/nodes, /api/edges, DELETE) flows through swarm_engram_write, which checks the caller's capability via the same scope-token mechanism as the live Rule-2 denial and rejects any worker before any HTTP is issued. Capability is fixed at mint time and cannot be acquired at runtime — so the guarantee holds regardless of engram health (distinct from the SWARM_WRITE_HEALTHY health gate).

The curated merge is the only write path: workers return geometry; the orchestrator, and only the orchestrator, commits the approved/verified geometry back (commit=1). Workers keep full-engram read access (op_think/op_read).

Proven in harness_real_cognition.el (§G): a worker swarm_engram_write is DENIED by capability with no node created and the violation journalled; the orchestrator passes the gate as the sole authorized writer.

Containment → distribution

The three containment rules make workers location-independent (Swarm §9): a worker reads only its compiled context, shares no state with siblings, and its only outward edge is the returned result. The same coordinator can run workers as local threads today or dispatch them across machines later — the mechanism is identical; only the topology changes. Enforced here:

• Rule 2 — swarm_run rejects any swarm opened under a worker token.
• Rules 1 + 3 — each worker gets a closed worker token; the coordinator is the only journal writer, so workers share no mutable state.

Usage

// one process step fans out; results converge before the next step
let inputs: String = "[\"billing\",\"payments\",\"ledger\"]"
let refs:   String = "[\"Volatility-Based Decomposition\"]"   // CCR knowledge refs
let cfg:    String = "{\"concurrency\":\"4\",\"strategy\":\"collect\",\"min_success_ratio\":\"1.0\"}"
let result: String = swarm_run("analyze_item", refs, inputs, cfg)
// result: { corr_id, status, merged, report }

Build any program that uses the swarm:

lang/swarm/build.sh myprog.el ./myprog     # concat + elc + cc (el_runtime.c)

Config keys: concurrency (max workers at once), strategy (collect|merge|vote|reduce), min_success_ratio (decimal string, e.g. 0.8), caller_token (containment). Env: SWARM_TRACK_DIR (journal dir), CCR_TOKEN_BUDGET, ENGRAM_URL/ENGRAM_API_KEY (retrieval + mirror), SWARM_MIRROR=1.

Tests

lang/swarm/build.sh lang/swarm/tests/test_swarm.el       /tmp/t && SWARM_TRACK_DIR=/tmp/trk /tmp/t   # 12/12
lang/swarm/build.sh lang/swarm/tests/test_convergence.el /tmp/c && SWARM_TRACK_DIR=/tmp/trk /tmp/c   #  8/8
# integration against an isolated engram clone (never live):
source <sandbox>/.nsbx-env
lang/swarm/build.sh lang/swarm/tests/integ_engram.el /tmp/i && /tmp/i

Local-swarm integration harness (the one flip)

tests/harness_local_swarm.el proves the full local-swarm mechanics today on the isolated clone with the primitive seam pointed at the hermetic stub — 17/17 green: 8 native-thread workers at concurrency 4, reduce + vote convergence, CCR scoping + non-leak, all three containment rules (incl. live Rule-2 denial), durable work-tracking, and afferent telemetry observed by the @manager.

Binding to the reshape's decorated primitives is one flip and a run:

# in primitive_binding.el — change one line each:
fn bound_think(ctx, instruction) { return think(ctx, instruction) }   # decorated, dharma bus
# then:
SWARM_PRIMITIVE_SEAM=decorated  lang/swarm/build.sh tests/harness_local_swarm.el ./h && ./h

Nothing else in the swarm changes. primitive_seam.el (seam_think/attend/learn) already routes every worker primitive call through this one switch, and the same harness runs the bound path. Today SWARM_PRIMITIVE_SEAM=decorated still runs green because the binding falls back to the stub — proving the flip path executes.

Real cognition — the seam is BOUND

primitive_binding.el is bound to the api-reshape agent's proven primitives (wt/api-reshape@d4f401d): bound_think -> op_think (GET /api/think), real 768-dim gradients over the engram geometry. reshape_surface.el composes those read/cognition primitives verbatim (op_think/read/attend/learn).

tests/harness_real_cognition.el runs the local swarm on real cognition, 17/17 green with SWARM_PRIMITIVE_SEAM=decorated against the :8901 clone: 8 native-thread workers, each a real think over its CCR-scoped node-id anchor (free-text anchors return "geometry unavailable"), @manager reduce+vote, all three containment rules, afferent telemetry, durable tracking. Per-anchor support counts (e.g. 6 / 16 / 87) drive a genuine, cognition-derived vote.

Build note (load-bearing): the swarm build must define HAVE_CURL (build.sh does). Without it every http_* builtin is a {"error":"not built with HAVE_CURL"} stub — real HTTP silently disappears.

Writes (attend/learn, POST) are gated behind SWARM_WRITE_HEALTHY=1 and the api-reshape agent's gate-1 write-healthy clone; the proven run is read-cognition.

Built vs stubbed (honest)

Real, tested:

• Native-thread fan-out/converge, bounded concurrency, order-preserving rejoin.
• All three containment rules enforced (scope tokens + lateral-edge check).
• CCR per-worker context: retrieval → scoping → compaction, bounded, non-leaking (a worker never receives sibling inputs) — verified against the live isolated mind.
• Full durable work-tracking (JSONL journal, reconstructable report).
• Four convergence strategies + integer failure threshold / partial-abort.

Seam / not yet bound:

• primitives.el think is a deterministic, hermetic transform (no model call). Binding point is marked PRIMITIVE_BINDING; wire to the API-surface reshape's think/act/attend/intend/learn when it lands.
• Blueprints are dispatched by name in swarm_run_blueprint (default + classify/faildemo demos). A YAML process-definition loader (Swarm §5) is future work — the runtime contract is in place.
• Distributed placement (cloud/edge/federated topologies, Swarm §9.2) is structurally enabled by containment but not yet wired to a placement layer; today all workers are local native threads.
• Engram work-tracking mirror is opt-in; the durable substrate is the journal.