swarm: land wt/swarm-ccr onto dev (clean re-merge) #122

2026-08-15T23:17:50Z

will.anderson commented

2026-08-15 23:17:50 +00:00

Re-does the PR #102 merge (dev <- wt/swarm-ccr) clean off current dev, now that #118 landed the missing _channel* seed primitives that were the real root cause of the original conflict.

Fixes applied on top of a straight merge:

el_runtime.c/.h: swarm-ccr's own copy of threading/mutex/channel primitives and the entire __-prefixed alias layer (I/O, string, filesystem, JSON, state, HTTP server) duplicated what el_seed.c already owns as of #118 — deleted (~330 + ~115 lines), matching the file's own established weak-symbol convention for the few that needed to stay (__http_do).
Fixed a real type bug: __http_serve/__http_serve_v2 tried to return el_val_t from void C functions.
Fixed dead #ifdef HAVE_CURL guards (never defined on POSIX) so http_patch/http_post_engram/http_get_engram/__http_do_map/__http_do_map_to_file actually compile and run on Mac instead of silently never existing.
Removed an orphaned #endif with no matching #ifdef (pre-existing bug in swarm-ccr's own OTLP section).

Verified: el_runtime.c compiles clean standalone, el_seed.c compiles clean standalone, and el_seed.o + el_runtime.o + all engram_*.o link together with zero new duplicate symbols (one pre-existing duplicate, el_request_start/el_request_end, was found — confirmed byte-identical on origin/dev already, unrelated to this merge, separately flagged, not fixed here).

Re-does the PR #102 merge (dev <- wt/swarm-ccr) clean off current dev, now that #118 landed the missing __channel_* seed primitives that were the real root cause of the original conflict. Fixes applied on top of a straight merge: - el_runtime.c/.h: swarm-ccr's own copy of threading/mutex/channel primitives and the entire __-prefixed alias layer (I/O, string, filesystem, JSON, state, HTTP server) duplicated what el_seed.c already owns as of #118 — deleted (~330 + ~115 lines), matching the file's own established weak-symbol convention for the few that needed to stay (__http_do). - Fixed a real type bug: __http_serve/__http_serve_v2 tried to return el_val_t from void C functions. - Fixed dead #ifdef HAVE_CURL guards (never defined on POSIX) so http_patch/http_post_engram/http_get_engram/__http_do_map/__http_do_map_to_file actually compile and run on Mac instead of silently never existing. - Removed an orphaned #endif with no matching #ifdef (pre-existing bug in swarm-ccr's own OTLP section). Verified: el_runtime.c compiles clean standalone, el_seed.c compiles clean standalone, and el_seed.o + el_runtime.o + all engram_*.o link together with zero new duplicate symbols (one pre-existing duplicate, el_request_start/el_request_end, was found — confirmed byte-identical on origin/dev already, unrelated to this merge, separately flagged, not fixed here).

will.anderson added 14 commits 2026-08-15 23:17:51 +00:00

el runtime: prototype + fix channel/mutex seed ABI for modern clang b2aac4bf89

el_runtime.h declared only __thread_create/__thread_join; the mutex and
channel seed primitives (__mutex_*, __channel_*) were defined in
el_runtime.c but never prototyped. Under Apple clang 21 (C11) the missing
prototypes became implicit-declaration errors, and the void-returning
__channel_send/__channel_close mis-typed el_val_t (long long) returns,
so any El program using runtime/channel.el failed to compile.

- add prototypes for __mutex_new/lock/unlock and all __channel_* to el_runtime.h
- make __channel_send/__channel_close return el_val_t nil so elc's
  trailing-expression codegen for the void El wrappers type-checks

Additive; unbreaks native channels for every downstream El program.

swarm: durable, inspectable work-tracking journal (worktrack.el) d5411fb58a

Single-writer append-only JSONL journal keyed by correlation ID: swarm +
worker + convergence records, reconstructable into a status report. Optional
engram mirror via POST /api/node when ENGRAM_URL is set. Coordinator is the
only writer (workers return structured results), which is race-free and
enforces Swarm containment rule 3 by construction.

Also prototype now_millis/now_ns in el_runtime.h (defined in el_runtime.c but
unprototyped — blocked any El program needing a real ms clock under clang 21).

Test proves durability + inspectability end-to-end.

swarm: orchestrator, CCR context compilation, containment rules, primitive seam 40bb6ff579

- swarm.el: coordinator running fan-out/converge on El NATIVE threads
  (thread.el spawn/join) in bounded concurrency waves, order-preserving;
  convergence strategies collect/merge/vote/reduce; integer per-mille failure
  threshold (El float division is unreliable — avoided deliberately).
- ccr.el: per-worker Compiled Context Routing — retrieval/scoping/compaction
  into a bounded, minimal package; the compiled-context boundary is the
  security boundary (a worker cannot receive or leak sibling inputs).
- containment.el: the three Swarm containment rules enforced via scope tokens
  (Rule 1 no join, Rule 2 no open, Rule 3 no lateral edge) + execution-tree
  lateral-edge check.
- primitives.el: attend/think/intend/act/learn seam the swarm composes over,
  with engram-backed fallbacks and an explicit binding point for the reshape.
- prototype json_array_push in el_runtime.h (defined but unprototyped).

test_swarm: 12/12 — native fan-out/converge, bounded concurrency, durable
tracking, CCR bounding + non-leak, and all three containment rules.

swarm: convergence strategies + failure threshold, hardened El JSON usage d4e82d3d56

- vote/merge/reduce/collect convergence proven end-to-end; failure threshold
  aborts a swarm below min_success_ratio (integer per-mille) and completes
  when failures are within tolerance, with worker.failed + swarm.aborted
  tracked durably.
- worked around three El runtime/codegen semantics surfaced during the build:
  json_set inserts RAW (use json_set_str for string values); json_set cannot
  update an existing key (vote tallies via list rescanning); json_array_get
  keeps quotes (use json_array_get_string). Also: float division is unreliable
  (swarm uses integer math), and a let-rebind in a deeply nested if/else does
  not propagate outward (accumulators kept at one block level).

test_convergence: 8/8; test_swarm: 12/12.

swarm: HTTP-backed primitive retrieval + live-engram integration test 447d042022

- primitive_attend retrieves over HTTP (POST /api/search) when ENGRAM_URL is
  set — the location-independent worker model — falling back to the in-process
  store otherwise. Proven against the isolated :8901 clone: CCR compiled a
  bounded context from REAL mind content (VBD/intellectual-dna).
- gate the engram work-tracking mirror behind SWARM_MIRROR=1; the durable
  substrate is always the JSONL journal, so a swarm never depends on the mind
  to track its work. (Repeated POST /api/nodes mirror writes were observed to
  crash the isolated daemon — a daemon-side write-path robustness issue;
  retrieval POST /api/search is solid. Prod :8742 never touched.)
- integ_engram: CCR real-retrieval + full swarm completion against live clone.

swarm: capability README — architecture, framework grounding, built vs stubbed b0a78c5737

swarm: build harness executable + module load order ed722b9e2e

swarm: local-swarm integration harness + one-flip primitive seam + telemetry 373265c05d

- primitive_seam.el: SWARM_PRIMITIVE_SEAM selects stub (default, hermetic) vs
  decorated (reshape's dharma-bus primitives). Every seam call is an afferent
  signal; telemetry (seam_mode + afferent tick) rides the vertical result path.
- primitive_binding.el: THE ONE FLIP POINT — bound_think/attend/learn today fall
  back to the stub; when the reshape's decorated primitives land, flip one line
  each and set SWARM_PRIMITIVE_SEAM=decorated. No other change anywhere.
- swarm.el: default blueprint routes think through the seam; the @manager
  aggregates afferent counters from worker results (containment-safe, no shared
  bus register) and journals a swarm.telemetry record; telemetry in the return.
- harness_local_swarm.el: 17/17 GREEN on :8901 with the stub — 8 native-thread
  workers at concurrency 4, reduce+vote convergence, CCR scoping+non-leak, all
  three containment rules (incl. live Rule-2 denial), durable work-tracking,
  afferent telemetry observed. Runs identically under seam=decorated today
  (binding fallback), proving the flip path executes.

Engram writes stay opt-in (durable journal is the substrate); daemon healthy.

swarm: document local-swarm harness + one-flip seam in README 70982498e0

swarm: bind reshape's proven primitives — REAL-COGNITION local swarm end-to-end 20bd9ed00b

Binds the api-reshape surface at wt/api-reshape@d4f401d (op_think/read/attend/
learn, verified against engram.cognition-20260814) into the swarm:
- reshape_surface.el composes the reshape's proven read/cognition primitives
  verbatim (write ops omitted — they need the gate-1 write-healthy clone).
- primitive_binding.el: bound_think -> op_think over the worker's NODE-ID
  anchor (ctx.input); attend/learn bound behind SWARM_WRITE_HEALTHY.
- cognize blueprint derives the vote verdict from the REAL gradient's n_support
  (json_get_int) — per-anchor diversity (6/16/87 support) drives a genuine vote.
- build.sh now defines HAVE_CURL. CRITICAL FIX: without it every http_* was a
  '{"error":"not built with HAVE_CURL"}' stub, so prior 'live engram'
  retrieval was a false positive (matched the ref string, not real content).
  With HAVE_CURL the swarm genuinely hits /api/think on the :8901 clone.

harness_real_cognition.el: 17/17 GREEN with seam=decorated — 8 native-thread
workers each a REAL think (768-dim gradient) over its CCR-scoped node-id anchor,
@manager reduce+vote convergence, all 3 containment rules incl. live Rule-2
denial, afferent telemetry (8 real think signals), durable work-tracking. Reads
only — daemon stays healthy; writes stay gated on the gate-1 clone. Prod :8742
untouched.

swarm: document real-cognition binding + HAVE_CURL build note in README b53b5b4e8a

swarm: Rule 4 — engram-write is @manager-ONLY, enforced by capability e5c80359a8

New hard invariant (Will): only the orchestrator mutates global engram state;
workers are read-only against the full engram + write only their own local
geometry. This is an AUTHORITY gate (capability), not a health gate — a worker
is STRUCTURALLY UNABLE to mutate global engram state regardless of engram health.

- containment.el: scope tokens now carry a caps set. Orchestrator token holds
  engram:write + dharma:emit (@manager-only, the VBD rule that only the manager
  mutates global state); worker token holds ONLY engram:read. Rule 4:
  containment_check_engram_write / _dharma_emit reject any caller lacking the
  capability — same scope-token mechanism as the live Rule-2 denial.
- swarm.el: swarm_engram_write is the ONLY engram write path, gated by Rule 4;
  a worker token is denied before any HTTP is issued (no mutation). The curated
  merge (commit=1) is the sole writer: the orchestrator commits approved
  geometry via its write-capable token. Workers' full-engram READ stays intact.
- reshape_surface.el: compose op_write (json_escape_string) for the commit path.
- harness: Rule-4 suite proven — worker engram-write DENIED by capability, no
  node created, violation journalled; orchestrator passes the gate as sole
  writer. 24/24 green on the :8901 clone with real cognition.

Authority gate holds independent of daemon write-health (proven with daemon
both alive and, earlier, crashed). Prod :8742 untouched.

swarm: document the single-writer invariant (Rule 4) in README

El SDK CI - dev / build-and-test (pull_request) Failing after 12m1s

Details

ff37835ae5

Merge remote-tracking branch 'origin/wt/swarm-ccr' into merge-swarm-ccr-v2

El SDK CI - dev / build-and-test (pull_request) Failing after 4m13s

Details

9883aa7564

# Conflicts:
#	lang/runtime/el_runtime.c
#	lang/runtime/el_runtime.h

will.anderson merged commit 15b66c8b1a into dev

2026-08-15 23:17:57 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/el#122