engram: fix lazy-embed index gap (#20) and make activate's cosine scan lazy; extract vindex harvest primitive with a bench/oracle harness
El SDK CI - dev / build-and-test (pull_request) Successful in 6m42s

Adds an O(1) "seen" bitmap so lazily-embedded older nodes get picked up
incrementally instead of only on a full rebuild (embed-gap #20).

Replaces engram_activate's O(N*D) cosine prescan with a lazy-memoized
cosine cache (eg_cosq_at), proven bit-identical to the old path.

Extracts a clean vindex_harvest_from_store primitive (read-only vector
harvest, careful malloc/ownership/error-path handling) reused by both
index-build and the new vindex_bench.c — a read-only proof harness
comparing brute-force vs HNSW recall/latency on both the real store and
synthetic data.

.nsbx-env intentionally excluded — local sandbox config (ports, paths,
dev-only placeholder key), not checked in.
This commit is contained in:
bigmerge
2026-08-15 14:26:16 -05:00
parent 6621a4dbc5
commit 08cbcef5d9
5 changed files with 426 additions and 43 deletions
+12
View File
@@ -73,6 +73,18 @@ void vindex_free(VIndex* idx);
int vindex_build_from_store(VIndex* idx, const char* store_path,
char*** ids_out, int* n_out);
/* Read-only harvest of the raw (un-normalised) emb vectors from a paged store,
* applying the SAME filtering vindex_build_from_store does (live records only,
* deduped by store id, emb present with emb_dim == `dim`), in insertion order.
* On success sets *vecs_out to a malloc'd float[n*dim] (row i == the i-th kept
* vector) and *n_out to n; if `ids_out` is non-NULL, sets it to a malloc'd array
* of n strdup'd store ids (ids_out[i] == the id of row i). Caller frees *vecs_out,
* each id string, and the id array. Returns n, or <0 on error. Used both by
* vindex_build_from_store (which then inserts each row) and by benchmarks/oracles
* that need the same vector set the index holds. */
int vindex_harvest_from_store(const char* store_path, int dim,
float** vecs_out, char*** ids_out, int* n_out);
/* Optional persistence (index is rebuildable from the store; provided for
* convenience). vindex_save writes a self-describing snapshot; vindex_load
* reconstructs an index from one. Return 0 / non-NULL on success. */