Swarm + CCR + Work-Tracking — Neuron's bounded parallel execution, in native El

Bounded parallel agent execution on El's native concurrency — no external orchestrator. Grounded directly in two of Will's frameworks:

Swarm Architecture (Bounded Parallel Agent Execution, Mar 2026)
Compiled Context Runtime / CCR (Process-Driven Agent Execution with Unbounded Local Memory, Mar 2026)

A swarm is a coordinator (the main thread) that mints a correlation identity, compiles a bounded per-worker context (CCR), dispatches workers as native pthreads (thread.el spawn/join), tracks every unit of work durably, and converges results before returning control to the parent step.

Parent step
  └─ swarm_run(blueprint, knowledge_refs, inputs, config)
        fan-out ──▶ worker_1 (CCR ctx_1) ─┐   native
                    worker_2 (CCR ctx_2) ─┤   pthreads,
                    worker_k (CCR ctx_k) ─┘   bounded by `concurrency`
        converge ─▶ collect | merge | vote | reduce  ──▶ merged result

Why it runs on El natively

El is natively agentic. This capability composes El's shipped primitives — it adds no bespoke runtime:

Primitive	Source	Role in the swarm
`spawn(fn,arg)` / `join(tid)`	`runtime/thread.el` → `__thread_create` (pthread + dlsym)	fan-out / rejoin
`parallel_map`, `with_mutex`	`runtime/thread.el`	reference concurrency patterns
Go-style channels	`runtime/channel.el` → `__channel_*`	available for vertical event streams
`engram_`, `http_`, `fs_`, `json_`	`el_runtime.c` builtins	retrieval, tracking, I/O

Every El fn compiles to a global C symbol, so any top-level (String)->String fn is directly threadable — the worker entry is exactly such a fn.

Modules

File	Framework grounding	What it does
`worktrack.el`	Swarm §6 (correlation IDs, audit)	Durable, single-writer JSONL journal keyed by correlation ID; reconstructable status report; opt-in engram mirror (`SWARM_MIRROR=1`).
`containment.el`	Swarm §3 (the three rules)	Scope tokens; Rule 1 (no join), Rule 2 (no open), Rule 3 (no lateral edge) enforced as checks.
`ccr.el`	CCR §5 + Swarm §9.3	Per-worker Compiled Context Routing: retrieve → scope → compact into a bounded, minimal package. The compiled-context boundary is the security boundary.
`primitives.el`	CCR §2 (Five Primitives)	`attend / think / intend / act / learn` seam the swarm composes over. Engram-backed; explicit binding point for the API-surface reshape.
`swarm.el`	Swarm §2, §4, §5	The coordinator: fan-out/converge on native threads, bounded concurrency, four convergence strategies, integer failure threshold, full tracking.

Containment → distribution

The three containment rules make workers location-independent (Swarm §9): a worker reads only its compiled context, shares no state with siblings, and its only outward edge is the returned result. The same coordinator can run workers as local threads today or dispatch them across machines later — the mechanism is identical; only the topology changes. Enforced here:

Rule 2 — swarm_run rejects any swarm opened under a worker token.
Rules 1 + 3 — each worker gets a closed worker token; the coordinator is the only journal writer, so workers share no mutable state.

Usage

// one process step fans out; results converge before the next step
let inputs: String = "[\"billing\",\"payments\",\"ledger\"]"
let refs:   String = "[\"Volatility-Based Decomposition\"]"   // CCR knowledge refs
let cfg:    String = "{\"concurrency\":\"4\",\"strategy\":\"collect\",\"min_success_ratio\":\"1.0\"}"
let result: String = swarm_run("analyze_item", refs, inputs, cfg)
// result: { corr_id, status, merged, report }

Build any program that uses the swarm:

lang/swarm/build.sh myprog.el ./myprog     # concat + elc + cc (el_runtime.c)

Config keys: concurrency (max workers at once), strategy (collect|merge|vote|reduce), min_success_ratio (decimal string, e.g. 0.8), caller_token (containment). Env: SWARM_TRACK_DIR (journal dir), CCR_TOKEN_BUDGET, ENGRAM_URL/ENGRAM_API_KEY (retrieval + mirror), SWARM_MIRROR=1.

Tests

lang/swarm/build.sh lang/swarm/tests/test_swarm.el       /tmp/t && SWARM_TRACK_DIR=/tmp/trk /tmp/t   # 12/12
lang/swarm/build.sh lang/swarm/tests/test_convergence.el /tmp/c && SWARM_TRACK_DIR=/tmp/trk /tmp/c   #  8/8
# integration against an isolated engram clone (never live):
source <sandbox>/.nsbx-env
lang/swarm/build.sh lang/swarm/tests/integ_engram.el /tmp/i && /tmp/i

Built vs stubbed (honest)

Real, tested:

Native-thread fan-out/converge, bounded concurrency, order-preserving rejoin.
All three containment rules enforced (scope tokens + lateral-edge check).
CCR per-worker context: retrieval → scoping → compaction, bounded, non-leaking (a worker never receives sibling inputs) — verified against the live isolated mind.
Full durable work-tracking (JSONL journal, reconstructable report).
Four convergence strategies + integer failure threshold / partial-abort.

Seam / not yet bound:

primitives.el think is a deterministic, hermetic transform (no model call). Binding point is marked PRIMITIVE_BINDING; wire to the API-surface reshape's think/act/attend/intend/learn when it lands.
Blueprints are dispatched by name in swarm_run_blueprint (default + classify/faildemo demos). A YAML process-definition loader (Swarm §5) is future work — the runtime contract is in place.
Distributed placement (cloud/edge/federated topologies, Swarm §9.2) is structurally enabled by containment but not yet wired to a placement layer; today all workers are local native threads.
Engram work-tracking mirror is opt-in; the durable substrate is the journal.

5.9 KiB Raw Blame History