engram: fix lazy-embed index gap (#20) and make activate's cosine scan lazy; extract vindex harvest primitive with a bench/oracle harness
El SDK CI - dev / build-and-test (pull_request) Successful in 6m42s
El SDK CI - dev / build-and-test (pull_request) Successful in 6m42s
Adds an O(1) "seen" bitmap so lazily-embedded older nodes get picked up incrementally instead of only on a full rebuild (embed-gap #20). Replaces engram_activate's O(N*D) cosine prescan with a lazy-memoized cosine cache (eg_cosq_at), proven bit-identical to the old path. Extracts a clean vindex_harvest_from_store primitive (read-only vector harvest, careful malloc/ownership/error-path handling) reused by both index-build and the new vindex_bench.c — a read-only proof harness comparing brute-force vs HNSW recall/latency on both the real store and synthetic data. .nsbx-env intentionally excluded — local sandbox config (ports, paths, dev-only placeholder key), not checked in.
This commit is contained in:
@@ -73,6 +73,18 @@ void vindex_free(VIndex* idx);
|
||||
int vindex_build_from_store(VIndex* idx, const char* store_path,
|
||||
char*** ids_out, int* n_out);
|
||||
|
||||
/* Read-only harvest of the raw (un-normalised) emb vectors from a paged store,
|
||||
* applying the SAME filtering vindex_build_from_store does (live records only,
|
||||
* deduped by store id, emb present with emb_dim == `dim`), in insertion order.
|
||||
* On success sets *vecs_out to a malloc'd float[n*dim] (row i == the i-th kept
|
||||
* vector) and *n_out to n; if `ids_out` is non-NULL, sets it to a malloc'd array
|
||||
* of n strdup'd store ids (ids_out[i] == the id of row i). Caller frees *vecs_out,
|
||||
* each id string, and the id array. Returns n, or <0 on error. Used both by
|
||||
* vindex_build_from_store (which then inserts each row) and by benchmarks/oracles
|
||||
* that need the same vector set the index holds. */
|
||||
int vindex_harvest_from_store(const char* store_path, int dim,
|
||||
float** vecs_out, char*** ids_out, int* n_out);
|
||||
|
||||
/* Optional persistence (index is rebuildable from the store; provided for
|
||||
* convenience). vindex_save writes a self-describing snapshot; vindex_load
|
||||
* reconstructs an index from one. Return 0 / non-NULL on success. */
|
||||
|
||||
Reference in New Issue
Block a user