el/lang/swarm

Swarm + CCR + Work-Tracking — Neuron's bounded parallel execution, in native El

Bounded parallel agent execution on El's native concurrency — no external orchestrator. Grounded directly in two of Will's frameworks:

• Swarm Architecture (Bounded Parallel Agent Execution, Mar 2026)
• Compiled Context Runtime / CCR (Process-Driven Agent Execution with Unbounded Local Memory, Mar 2026)

A swarm is a coordinator (the main thread) that mints a correlation identity, compiles a bounded per-worker context (CCR), dispatches workers as native pthreads (thread.el spawn/join), tracks every unit of work durably, and converges results before returning control to the parent step.

Parent step
  └─ swarm_run(blueprint, knowledge_refs, inputs, config)
        fan-out ──▶ worker_1 (CCR ctx_1) ─┐   native
                    worker_2 (CCR ctx_2) ─┤   pthreads,
                    worker_k (CCR ctx_k) ─┘   bounded by `concurrency`
        converge ─▶ collect | merge | vote | reduce  ──▶ merged result

Why it runs on El natively

El is natively agentic. This capability composes El's shipped primitives — it adds no bespoke runtime:

Primitive	Source	Role in the swarm
spawn(fn,arg) / join(tid)	runtime/thread.el → __thread_create (pthread + dlsym)	fan-out / rejoin
parallel_map, with_mutex	runtime/thread.el	reference concurrency patterns
Go-style channels	runtime/channel.el → __channel_*	available for vertical event streams
engram_*, http_*, fs_*, json_*	el_runtime.c builtins	retrieval, tracking, I/O

Every El fn compiles to a global C symbol, so any top-level (String)->String fn is directly threadable — the worker entry is exactly such a fn.

Modules

File	Framework grounding	What it does
worktrack.el	Swarm §6 (correlation IDs, audit)	Durable, single-writer JSONL journal keyed by correlation ID; reconstructable status report; opt-in engram mirror (SWARM_MIRROR=1).
containment.el	Swarm §3 (the three rules)	Scope tokens; Rule 1 (no join), Rule 2 (no open), Rule 3 (no lateral edge) enforced as checks.
ccr.el	CCR §5 + Swarm §9.3	Per-worker Compiled Context Routing: retrieve → scope → compact into a bounded, minimal package. The compiled-context boundary is the security boundary.
primitives.el	CCR §2 (Five Primitives)	attend / think / intend / act / learn seam the swarm composes over. Engram-backed; explicit binding point for the API-surface reshape.
swarm.el	Swarm §2, §4, §5	The coordinator: fan-out/converge on native threads, bounded concurrency, four convergence strategies, integer failure threshold, full tracking.

Containment → distribution

The three containment rules make workers location-independent (Swarm §9): a worker reads only its compiled context, shares no state with siblings, and its only outward edge is the returned result. The same coordinator can run workers as local threads today or dispatch them across machines later — the mechanism is identical; only the topology changes. Enforced here:

• Rule 2 — swarm_run rejects any swarm opened under a worker token.
• Rules 1 + 3 — each worker gets a closed worker token; the coordinator is the only journal writer, so workers share no mutable state.

Usage

// one process step fans out; results converge before the next step
let inputs: String = "[\"billing\",\"payments\",\"ledger\"]"
let refs:   String = "[\"Volatility-Based Decomposition\"]"   // CCR knowledge refs
let cfg:    String = "{\"concurrency\":\"4\",\"strategy\":\"collect\",\"min_success_ratio\":\"1.0\"}"
let result: String = swarm_run("analyze_item", refs, inputs, cfg)
// result: { corr_id, status, merged, report }

Build any program that uses the swarm:

lang/swarm/build.sh myprog.el ./myprog     # concat + elc + cc (el_runtime.c)

Config keys: concurrency (max workers at once), strategy (collect|merge|vote|reduce), min_success_ratio (decimal string, e.g. 0.8), caller_token (containment). Env: SWARM_TRACK_DIR (journal dir), CCR_TOKEN_BUDGET, ENGRAM_URL/ENGRAM_API_KEY (retrieval + mirror), SWARM_MIRROR=1.

Tests

lang/swarm/build.sh lang/swarm/tests/test_swarm.el       /tmp/t && SWARM_TRACK_DIR=/tmp/trk /tmp/t   # 12/12
lang/swarm/build.sh lang/swarm/tests/test_convergence.el /tmp/c && SWARM_TRACK_DIR=/tmp/trk /tmp/c   #  8/8
# integration against an isolated engram clone (never live):
source <sandbox>/.nsbx-env
lang/swarm/build.sh lang/swarm/tests/integ_engram.el /tmp/i && /tmp/i

Local-swarm integration harness (the one flip)

tests/harness_local_swarm.el proves the full local-swarm mechanics today on the isolated clone with the primitive seam pointed at the hermetic stub — 17/17 green: 8 native-thread workers at concurrency 4, reduce + vote convergence, CCR scoping + non-leak, all three containment rules (incl. live Rule-2 denial), durable work-tracking, and afferent telemetry observed by the @manager.

Binding to the reshape's decorated primitives is one flip and a run:

# in primitive_binding.el — change one line each:
fn bound_think(ctx, instruction) { return think(ctx, instruction) }   # decorated, dharma bus
# then:
SWARM_PRIMITIVE_SEAM=decorated  lang/swarm/build.sh tests/harness_local_swarm.el ./h && ./h

Nothing else in the swarm changes. primitive_seam.el (seam_think/attend/learn) already routes every worker primitive call through this one switch, and the same harness runs the bound path. Today SWARM_PRIMITIVE_SEAM=decorated still runs green because the binding falls back to the stub — proving the flip path executes.

Real cognition — the seam is BOUND

primitive_binding.el is bound to the api-reshape agent's proven primitives (wt/api-reshape@d4f401d): bound_think -> op_think (GET /api/think), real 768-dim gradients over the engram geometry. reshape_surface.el composes those read/cognition primitives verbatim (op_think/read/attend/learn).

tests/harness_real_cognition.el runs the local swarm on real cognition, 17/17 green with SWARM_PRIMITIVE_SEAM=decorated against the :8901 clone: 8 native-thread workers, each a real think over its CCR-scoped node-id anchor (free-text anchors return "geometry unavailable"), @manager reduce+vote, all three containment rules, afferent telemetry, durable tracking. Per-anchor support counts (e.g. 6 / 16 / 87) drive a genuine, cognition-derived vote.

Build note (load-bearing): the swarm build must define HAVE_CURL (build.sh does). Without it every http_* builtin is a {"error":"not built with HAVE_CURL"} stub — real HTTP silently disappears.

Writes (attend/learn, POST) are gated behind SWARM_WRITE_HEALTHY=1 and the api-reshape agent's gate-1 write-healthy clone; the proven run is read-cognition.

Built vs stubbed (honest)

Real, tested:

• Native-thread fan-out/converge, bounded concurrency, order-preserving rejoin.
• All three containment rules enforced (scope tokens + lateral-edge check).
• CCR per-worker context: retrieval → scoping → compaction, bounded, non-leaking (a worker never receives sibling inputs) — verified against the live isolated mind.
• Full durable work-tracking (JSONL journal, reconstructable report).
• Four convergence strategies + integer failure threshold / partial-abort.

Seam / not yet bound:

• primitives.el think is a deterministic, hermetic transform (no model call). Binding point is marked PRIMITIVE_BINDING; wire to the API-surface reshape's think/act/attend/intend/learn when it lands.
• Blueprints are dispatched by name in swarm_run_blueprint (default + classify/faildemo demos). A YAML process-definition loader (Swarm §5) is future work — the runtime contract is in place.
• Distributed placement (cloud/edge/federated topologies, Swarm §9.2) is structurally enabled by containment but not yet wired to a placement layer; today all workers are local native threads.
• Engram work-tracking mirror is opt-in; the durable substrate is the journal.