Neuron
8e9d88fc01
El SDK CI - dev / build-and-test (pull_request) Failing after 10m59s
The crash (SIGTRAP in engram_activate -> eg_vindex_sync -> vindex_insert -> _realloc) had three read paths mutating five process-global statics. engram_activate, eg_knn_for_node (whose own comment says "No writes.") and engram_geo_reify_run_json all called eg_vindex_sync, which frees the index, reallocs the seen-map and inserts — on a read. Three moves, in decreasing order of how much they dissolve: 1. Misfiled scratch is not shared state. visited/visit_epoch/visited_cap were never owned by the index; they are one traversal's local, hoisted into struct VIndex as an allocation optimisation. They want neither a lock nor a capability nor a pool — just to go back in the call frame. Two concurrent READS stomped each other purely because of this. 2. const IS the capability. Once the scratch leaves the struct, search reads and nothing else, so vindex_search takes a const VIndex*. That is exactly what a capability-pointer ABI would have bought — a read path physically cannot call vindex_insert, enforced by the compiler on every future caller — for one qualifier instead of an ABI swept across hundreds of builtins. 3. What survives is publication, not ownership. HNSW insert is NOT an append: it rewires the neighbour links of already-existing elements and reallocs elems[], so the store's append-only property does not transfer to the index derived from it. eg_vindex_sync therefore splits into eg_vindex_maintain (exclusive, sole mutator) and eg_vindex_view (shared, returns const VIndex*). A read path may demand that a current snapshot exist — a request to the owner, not a mutation by the reader. Write-side owner: eg_vindex_note_embedded hooks the embedding-ASSIGNMENT sites rather than the append sites, because a node with no embedding cannot be in a vector index — embedding assignment is the event that owns index membership. One O(log n) insert, no O(node_count) presence scan. This also retires the "STALENESS (honest tradeoff)" note where a lazily-embedded older node stayed invisible to route_nearest/autoconnect until a full rebuild (the embed-gap #20 shape). Evidence. The existing harness conflated two hazards, which is why fixing half of it read as failure. Split into four: single (3000 vec, ASan+UBSan) clean -> clean readers (4 readers, no writer, TSan) RACE -> clean unsynchronized (writer+reader, bare) race -> race, expected forever published (owner + 4 readers) n/a -> clean, 3000/3000 landed RESULT: PASS. recall@10 = 0.9365 at ef_search=128 (gate >= 0.90); determinism byte-identical across two independent builds. The unsynchronized half is now permanently expected to race, deliberately: it is the executable proof that the boundary must live above the data structure, not inside it. fb32d15's guard is KEPT, correcting this design's own section 5. Measured, it guards TWO structures and only one was converted here: g->nodes/g->edges are realloc'd in place (el_runtime.c:7618,7629) and engram_activate_inner's embed-backfill writes n->emb through exactly such a borrowed pointer. Deleting the guard reintroduces a measured 11171->9579 edge loss. Its comment is narrowed to the RAM graph and the deletion precondition named. That corrects the ordering claim too: the residual is not one ABI that dissolves everything at once, it is a PROPERTY applied per structure. Residues evaporate in the order the property is applied, and a residue whose structure has not been converted must be left standing.
102 lines
5.3 KiB
C
102 lines
5.3 KiB
C
/* engram_vindex.h — M8 of the engram query engine: an approximate-nearest-
|
|
* neighbour (ANN) vector index over the node embedding vectors, for fast
|
|
* activation-seed selection.
|
|
*
|
|
* Replaces the O(n) cosine scan over emb vectors (design §9 M8; backlog #20)
|
|
* with an HNSW (Hierarchical Navigable Small World) graph that returns
|
|
* high-recall top-k seeds in ~O(log n).
|
|
*
|
|
* Standalone module: plain C11, stdlib + libm only. It does NOT modify the
|
|
* store format or engram_store.{c,h}; vindex_build_from_store() decodes the
|
|
* PERMANENT on-disk node format (design §2.4) read-only to harvest emb vectors.
|
|
*
|
|
* Similarity metric: cosine. Vectors are L2-normalised on insert/query, so
|
|
* cosine similarity == dot product. Reported distance = 1 - cosine_similarity
|
|
* (range [0,2]); smaller == closer. A query equal to an indexed vector scores
|
|
* distance ~0 against it.
|
|
*
|
|
* The index is fully rebuildable from the store, so persistence is optional for
|
|
* this milestone (see vindex_save/vindex_load below — provided as a convenience;
|
|
* boot may simply rebuild via vindex_build_from_store()).
|
|
*/
|
|
#ifndef ENGRAM_VINDEX_H
|
|
#define ENGRAM_VINDEX_H
|
|
|
|
#include <stddef.h>
|
|
#include <stdint.h>
|
|
|
|
/* Tuned defaults (rationale in engram_vindex.c). Pass 0 to vindex_create for
|
|
* M / ef_construction to take these; pass ef_search<=0 to vindex_search for
|
|
* VINDEX_DEFAULT_EF_SEARCH. */
|
|
#define VINDEX_DEFAULT_M 24
|
|
#define VINDEX_DEFAULT_EF_CONSTRUCTION 200
|
|
#define VINDEX_DEFAULT_EF_SEARCH 128
|
|
|
|
typedef struct VIndex VIndex;
|
|
|
|
/* Create an index over `dim`-dimensional f32 vectors.
|
|
* M — max neighbours per node on upper layers (2*M on layer 0).
|
|
* ef_construction — candidate-list width during insert (recall/build cost).
|
|
* Pass M<=0 or ef_construction<=0 to use the VINDEX_DEFAULT_* above.
|
|
* Returns NULL on bad args / OOM. */
|
|
VIndex* vindex_create(int dim, int M, int ef_construction);
|
|
|
|
/* Insert one vector under an opaque caller-defined node_id (need not be unique,
|
|
* but the caller is responsible for meaning). `vec` has `dim` floats; it is
|
|
* copied and L2-normalised internally. A zero vector is accepted (it simply has
|
|
* distance ~1 to everything; never produces NaN). Returns 0 on success, <0 on
|
|
* error (bad args / OOM). */
|
|
int vindex_insert(VIndex* idx, uint64_t node_id, const float* vec);
|
|
|
|
/* Top-k search by cosine similarity. Writes up to k results (fewer if the index
|
|
* holds fewer than k elements) into node_id_out[] / dist_out[], ordered nearest
|
|
* first (ascending distance). Either out array may be NULL to skip it.
|
|
* ef_search — search-time candidate width; larger == higher recall, slower.
|
|
* Pass <=0 for VINDEX_DEFAULT_EF_SEARCH. Internally clamped to >=k.
|
|
* Returns the number of results written, or <0 on error.
|
|
*
|
|
* `idx` is const BY CONTRACT AND BY TYPE: search does not mutate the index. The
|
|
* traversal's visited set is owned by the call frame, so N threads may search one
|
|
* index concurrently. Concurrent search against a vindex_insert on the same index
|
|
* is still unsafe — insert rewires existing elements' neighbour lists and reallocs
|
|
* elems[] — so the index's owner must not extend a published index under a live
|
|
* reader. See eg_vindex_view / eg_vindex_maintain in el_runtime.c. */
|
|
int vindex_search(const VIndex* idx, const float* query, int k, int ef_search,
|
|
uint64_t* node_id_out, float* dist_out);
|
|
|
|
/* Number of vectors currently indexed. */
|
|
size_t vindex_size(const VIndex* idx);
|
|
|
|
void vindex_free(VIndex* idx);
|
|
|
|
/* Build an index by scanning every live node record in the paged store at
|
|
* `store_path` (the on-disk format is decoded read-only; the store need not be
|
|
* open). Nodes without an emb vector, or whose emb_dim != idx->dim, are skipped.
|
|
* Each inserted node is assigned node_id = its 0-based insertion ordinal; if
|
|
* `ids_out`/`n_out` are non-NULL, *ids_out is set to a malloc'd array of that
|
|
* many strdup'd string ids (ids_out[node_id] == the store id) and *n_out to the
|
|
* count — the caller frees each string and the array. Returns the number of
|
|
* vectors inserted, or <0 on error. */
|
|
int vindex_build_from_store(VIndex* idx, const char* store_path,
|
|
char*** ids_out, int* n_out);
|
|
|
|
/* Read-only harvest of the raw (un-normalised) emb vectors from a paged store,
|
|
* applying the SAME filtering vindex_build_from_store does (live records only,
|
|
* deduped by store id, emb present with emb_dim == `dim`), in insertion order.
|
|
* On success sets *vecs_out to a malloc'd float[n*dim] (row i == the i-th kept
|
|
* vector) and *n_out to n; if `ids_out` is non-NULL, sets it to a malloc'd array
|
|
* of n strdup'd store ids (ids_out[i] == the id of row i). Caller frees *vecs_out,
|
|
* each id string, and the id array. Returns n, or <0 on error. Used both by
|
|
* vindex_build_from_store (which then inserts each row) and by benchmarks/oracles
|
|
* that need the same vector set the index holds. */
|
|
int vindex_harvest_from_store(const char* store_path, int dim,
|
|
float** vecs_out, char*** ids_out, int* n_out);
|
|
|
|
/* Optional persistence (index is rebuildable from the store; provided for
|
|
* convenience). vindex_save writes a self-describing snapshot; vindex_load
|
|
* reconstructs an index from one. Return 0 / non-NULL on success. */
|
|
int vindex_save(const VIndex* idx, const char* path);
|
|
VIndex* vindex_load(const char* path);
|
|
|
|
#endif /* ENGRAM_VINDEX_H */
|