neuron-technologies/el
3
0
Fork 0
Code Issues 4 Pull Requests 6 Actions Packages Projects Releases 5 Wiki Activity

engram: real Metal batch-cosine kernel for vindex_bench's brute-force oracle #114

Closed
will.anderson wants to merge 0 commits from feat/engram-metal-cosine-batch into feat/reframe-region-setop
pull from: feat/engram-metal-cosine-batch
merge into: :feat/reframe-region-setop
Conversation 1 Commits - Files Changed -
+805 -5
will.anderson commented 2026-08-15 21:48:31 +00:00
Owner
Copy Link
Copy Source

Relationship to #109 / #105

This branches from #109 (feat/reframe-region-setop), not dev directly — the only genuine batch-cosine call site (vindex_bench.c) exists solely on that branch. That's a deliberate deviation from the original brief's "PR against dev," flagged here explicitly: this PR should land after/alongside #109, not independently against dev. No overlap with #105 (separate latency fix, different files).

Why this target, not engram_activate

The original ask was to GPU-accelerate engram_activate's O(N·D) cosine prescan. #109 (this PR's base) already retires that loop algorithmically — HNSW seed selection (vindex_search) + a lazy-memoized eg_cosq_at() replace the full scan with per-candidate lookups. GPU-accelerating a loop that's being deleted isn't real work, so I dropped that target rather than force it.

Re-investigated for a genuine remaining fit, with code, not assumption:

  • HNSW insert's candidate-list work (bounded degree M=24-48 per hop, sequential/adaptive best-first search) — too fine-grained for a GPU dispatch to pay off against CPU SIMD. Real cost (confirmed ENGRAM_SELF_REIFY=1 is live in prod, route_tick rebuilds the whole vindex from scratch every 10 min), wrong shape for this primitive.
  • O(N²) pairwise dedup — doesn't exist. ENGRAM_DEDUP_COS only checks against the K=8 already-selected seed slots, O(K) not O(N²).
  • Concurrent multi-query batching — doesn't exist. server.el: "nothing in production calls /api/activate on this store — the soul's curiosity loop activates its own in-process graph." Single serial caller.
  • vindex_bench.c's brute_topk oracle — real, genuine, unforced fit: 1-query-vs-N-vectors, no adaptivity, embarrassingly parallel. It's the correctness oracle this same PR (#109) adds to validate HNSW recall against ground truth.

What's added

  • eg_cosine_batch.metal — batched cosine kernel (single- and multi-query variants), same -2.0 dim-mismatch/zero-norm sentinel as eg_cosine().
  • eg_metal_cosine.h / .m — C-callable Objective-C bridge. Lazy one-time device/pipeline init, MTLResourceStorageModeShared buffers (unified memory, no explicit upload step). Returns false on any failure — no device, compile error, alloc failure — so callers fall back to the scalar CPU loop unconditionally. Never partial, never throws.
  • eg_metal_cosine_stub.c — zero-dependency CPU-only implementation for non-Darwin (Linux CI). Same symbols, always returns false. No #ifdef needed at any call site — platform selection happens at the build/link step.
  • build_vindex_bench.sh — one-command build: real bridge + -framework Metal -framework Foundation on Darwin, stub everywhere else.
  • vindex_bench.c — brute_topk_metal / brute_topk_metal_batch call the bridge, falling back to the existing CPU brute_topk on failure or EL_METAL_COSINE=0.

Not wired into engram_activate or the daemon build (nsbx's _build_binary) — vindex_bench is a standalone offline tool, not part of the request-serving binary. No server-latency claim is made here; the bridge is a reusable primitive (eg_cosine_batch_metal + batched eg_cosine_batch_metal_multi) other call sites can adopt later without re-deriving any of this.

Correctness + performance — real numbers, real data

Measured against an nsbx-sandboxed clone of the live store (isolated port, never :8742/:7770, destroyed after), 13,671 real embedded nodes, dim=768, 300 real queries:

BRUTE-FORCE (CPU):    2.013 ms/query
BRUTE-METAL (GPU):    0.117 ms/query   (17.2x)
id-recall vs CPU oracle: 0.9990 over 300 queries
same-rank |Δdist|: max 2.98e-07, mean 7.53e-08  (float32 rounding, not a bug)

The first version of this was measured slower than CPU (re-uploaded the full N×D matrix on every single query). Fixed by adding a real multi-query batched kernel that uploads the matrix once per query batch — see commit message for the before/after.

Synthetic scaling sweep (13k → 50k nodes, same dim/queries, no real data touched) shows the GPU speedup holding (~11x) as N grows toward the mathematical-foundations doc's 1.3M-node target, CPU brute-force cost growing linearly as expected.

Open items for Will

  • dev itself has none of this code (no engram_vindex.c, no engram_geometry.c, no self-reify) — the live production binary and #109 both descend from an unmerged private worktree lineage. Worth knowing regardless of this PR.
  • ENGRAM_SELF_REIFY=1 is live on the production launchd job right now, rebuilding the entire HNSW index from scratch every 10-minute tick — real, currently-running cost, but the wrong shape for this GPU primitive (see above). Flagging for awareness, not fixing here.
  • Do not merge this PR myself — for review.
## Relationship to #109 / #105 This branches from **#109** (`feat/reframe-region-setop`), not `dev` directly — the only genuine batch-cosine call site (`vindex_bench.c`) exists solely on that branch. That's a deliberate deviation from the original brief's "PR against dev," flagged here explicitly: this PR should land after/alongside #109, not independently against dev. No overlap with #105 (separate latency fix, different files). ## Why this target, not engram_activate The original ask was to GPU-accelerate `engram_activate`'s O(N·D) cosine prescan. #109 (this PR's base) already retires that loop algorithmically — HNSW seed selection (`vindex_search`) + a lazy-memoized `eg_cosq_at()` replace the full scan with per-candidate lookups. GPU-accelerating a loop that's being deleted isn't real work, so I dropped that target rather than force it. Re-investigated for a genuine remaining fit, with code, not assumption: - **HNSW insert's candidate-list work** (bounded degree M=24-48 per hop, sequential/adaptive best-first search) — too fine-grained for a GPU dispatch to pay off against CPU SIMD. Real cost (confirmed `ENGRAM_SELF_REIFY=1` is live in prod, `route_tick` rebuilds the whole vindex from scratch every 10 min), wrong shape for this primitive. - **O(N²) pairwise dedup** — doesn't exist. `ENGRAM_DEDUP_COS` only checks against the K=8 already-selected seed slots, O(K) not O(N²). - **Concurrent multi-query batching** — doesn't exist. `server.el`: "nothing in production calls /api/activate on this store — the soul's curiosity loop activates its own in-process graph." Single serial caller. - **`vindex_bench.c`'s `brute_topk` oracle** — real, genuine, unforced fit: 1-query-vs-N-vectors, no adaptivity, embarrassingly parallel. It's the correctness oracle this same PR (#109) adds to validate HNSW recall against ground truth. ## What's added - `eg_cosine_batch.metal` — batched cosine kernel (single- and multi-query variants), same `-2.0` dim-mismatch/zero-norm sentinel as `eg_cosine()`. - `eg_metal_cosine.h` / `.m` — C-callable Objective-C bridge. Lazy one-time device/pipeline init, `MTLResourceStorageModeShared` buffers (unified memory, no explicit upload step). Returns `false` on *any* failure — no device, compile error, alloc failure — so callers fall back to the scalar CPU loop unconditionally. Never partial, never throws. - `eg_metal_cosine_stub.c` — zero-dependency CPU-only implementation for non-Darwin (Linux CI). Same symbols, always returns `false`. No `#ifdef` needed at any call site — platform selection happens at the build/link step. - `build_vindex_bench.sh` — one-command build: real bridge + `-framework Metal -framework Foundation` on Darwin, stub everywhere else. - `vindex_bench.c` — `brute_topk_metal` / `brute_topk_metal_batch` call the bridge, falling back to the existing CPU `brute_topk` on failure or `EL_METAL_COSINE=0`. Not wired into `engram_activate` or the daemon build (nsbx's `_build_binary`) — `vindex_bench` is a standalone offline tool, not part of the request-serving binary. No server-latency claim is made here; the bridge is a reusable primitive (`eg_cosine_batch_metal` + batched `eg_cosine_batch_metal_multi`) other call sites can adopt later without re-deriving any of this. ## Correctness + performance — real numbers, real data Measured against an `nsbx`-sandboxed clone of the live store (isolated port, never `:8742`/`:7770`, destroyed after), 13,671 real embedded nodes, dim=768, 300 real queries: ``` BRUTE-FORCE (CPU): 2.013 ms/query BRUTE-METAL (GPU): 0.117 ms/query (17.2x) id-recall vs CPU oracle: 0.9990 over 300 queries same-rank |Δdist|: max 2.98e-07, mean 7.53e-08 (float32 rounding, not a bug) ``` The first version of this was measured *slower* than CPU (re-uploaded the full N×D matrix on every single query). Fixed by adding a real multi-query batched kernel that uploads the matrix once per query batch — see commit message for the before/after. Synthetic scaling sweep (13k → 50k nodes, same dim/queries, no real data touched) shows the GPU speedup holding (~11x) as N grows toward the mathematical-foundations doc's 1.3M-node target, CPU brute-force cost growing linearly as expected. ## Open items for Will - `dev` itself has none of this code (no `engram_vindex.c`, no `engram_geometry.c`, no self-reify) — the live production binary and #109 both descend from an unmerged private worktree lineage. Worth knowing regardless of this PR. - `ENGRAM_SELF_REIFY=1` is live on the production launchd job right now, rebuilding the entire HNSW index from scratch every 10-minute tick — real, currently-running cost, but the wrong shape for this GPU primitive (see above). Flagging for awareness, not fixing here. - Do not merge this PR myself — for review.
will.anderson added 1 commit 2026-08-15 21:48:32 +00:00
engram: real Metal batch-cosine kernel, wired into vindex_bench's brute-force oracle 728207aabf 
The original brief targeted engram_activate's O(N*D) cosq prescan, but #109
(this branch) already retires that loop algorithmically (HNSW seed selection
+ lazy memoized cosine) — GPU-accelerating a loop being deleted isn't real
work, so that target was dropped rather than forced.

Re-investigated for a genuine remaining GPU-shaped call site (not a
manufactured one): HNSW insert's candidate-list distance work is bounded-
degree (M=24-48) and sequential/adaptive — too fine-grained for a GPU
dispatch to pay off. No O(N^2) pairwise cosine pass exists (dedup only
checks the K=8 already-selected seed slots). No concurrent multi-query
traffic exists (server.el: the soul's curiosity loop is a single in-process
caller). vindex_bench.c's brute_topk — the correctness oracle this same PR
adds to validate HNSW recall — is the one real, unforced fit: genuine
1-query-vs-N-vectors, embarrassingly parallel, no adaptivity.

Adds:
  - eg_cosine_batch.metal: batched cosine kernel, single- and multi-query
    variants, same -2.0 dim-mismatch/zero-norm sentinel as eg_cosine().
  - eg_metal_cosine.h/.m: C-callable Objective-C bridge. Lazy one-time
    device/pipeline init, MTLResourceStorageModeShared buffers, returns
    false on ANY failure so callers fall back to the scalar CPU loop
    unconditionally — never partial, never throws.
  - eg_metal_cosine_stub.c: zero-dependency CPU-only implementation for
    non-Darwin builds (Linux CI) — same symbols, always returns false, no
    #ifdef needed at any call site.
  - build_vindex_bench.sh: one-command build, real bridge + Metal frameworks
    on Darwin, stub everywhere else.

vindex_bench.c: brute_topk_metal / brute_topk_metal_batch call the bridge,
falling back to the existing CPU brute_topk on any failure or
EL_METAL_COSINE=0. The multi-query batched path exists because the first
version (one GPU call per query) measured SLOWER than CPU at N~13.7k — it
re-uploaded the full N*D matrix every query. Fixed by uploading the matrix
once per query batch.

Measured against a real nsbx-sandboxed clone of the live store (never
:8742/:7770), 13,671 real embedded nodes, dim=768, 300 real queries:
  BRUTE-FORCE (CPU):    2.013 ms/query
  BRUTE-METAL (GPU):    0.117 ms/query   (17.2x)
  id-recall vs CPU oracle: 0.9990 over 300 queries
  same-rank |Δdist|: max 2.98e-07, mean 7.53e-08 (float32 rounding, not a bug)

Synthetic scaling sweep (13k -> 50k nodes, same dim/queries) shows the GPU
speedup holding (~11x) as N grows toward the mathematical-foundations doc's
1.3M-node target, with CPU brute-force cost growing linearly as expected.

Not wired into engram_activate or the daemon build (nsbx's _build_binary) —
vindex_bench is a standalone offline tool, not part of the request-serving
binary, so no engram_activate/server-latency claim is made here. The bridge
is a reusable primitive (single eg_cosine_batch_metal + batched
eg_cosine_batch_metal_multi) other call sites can adopt later without
re-deriving any of this.

Based on feat/reframe-region-setop (PR #109), not dev directly: the only
genuine batch-cosine call site (vindex_bench.c) exists solely on this
branch. Flagged explicitly in the PR description as a deliberate deviation
from the original "base off dev" instruction.
will.anderson commented 2026-08-15 22:21:02 +00:00
Author
Owner
Copy Link
Copy Source

Superseded by #116, which restructures this work into an Adapter/Strategy/Factory (rather than a hand-rolled-Metal-only bridge) and adds ggml as the preferred backend strategy, per Will's directive to stop hand-rolling GPU kernels. This branch's content is NOT discarded — eg_cosine_batch.metal and eg_metal_cosine.m are preserved almost verbatim in #116 as one of three selectable strategies (eg_cosine_batch_strategy_metal_hand.m), still real, still verified, still selectable via EL_COSINE_BATCH_STRATEGY=metal. #116 also lands this work onto current dev (this branch predates #109's HNSW merge and a lot of other dev history since). Closing this PR in favor of #116: #116

Superseded by #116, which restructures this work into an Adapter/Strategy/Factory (rather than a hand-rolled-Metal-only bridge) and adds ggml as the preferred backend strategy, per Will's directive to stop hand-rolling GPU kernels. This branch's content is NOT discarded — eg_cosine_batch.metal and eg_metal_cosine.m are preserved almost verbatim in #116 as one of three selectable strategies (eg_cosine_batch_strategy_metal_hand.m), still real, still verified, still selectable via EL_COSINE_BATCH_STRATEGY=metal. #116 also lands this work onto current dev (this branch predates #109's HNSW merge and a lot of other dev history since). Closing this PR in favor of #116: https://git.neuralplatform.ai/neuron-technologies/el/pulls/116
will.anderson closed this pull request 2026-08-15 22:21:07 +00:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
BETA-CRITICAL
On the beta critical path — must resolve before ship
 HELD
Blocked pending an external decision (counsel / dual sign-off)
 ORTHOGONAL
Parallel workstream (testing/hardening) — safe to assign off critical path
 POST-BETA
Important but not blocking the beta ship
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
andre.almeida (Andre Almeida) tim.lingo (Tim Lingo) will.anderson (Will Anderson)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: neuron-technologies/el#114
Delete Branch "feat/engram-metal-cosine-batch"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?