neuron-technologies/el
3
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases 5 Wiki Activity

feat: HNSW index, consolidation engine, Kotlin/TS/Go bindings, SQLite migration connector

Browse Source 
- vector.rs: replace flat O(n) scan with instant-distance HNSW for stores
  >= 100 nodes; flat scan retained as fallback for small graphs; dirty-flag
  persistence in sled triggers index rebuild only when nodes are added

- consolidation.rs: Episodic → Semantic promotion based on activation_count
  and salience_floor thresholds; global decay pass after each cycle;
  ConsolidationConfig + ConsolidationReport types; 8 tests

- migration.rs: reads Neuron SQLite (memory_nodes, knowledge_entries,
  graph_edges) and writes to Engram sled; placeholder unit-vector embeddings
  with TODO for ONNX; 5 tests including full in-memory DB roundtrip

- crates/engram-migrate: CLI binary (engram-migrate --sqlite / --output)

- crates/engram-jni: JNI cdylib exposing open/close/put_node/get_node/
  activate/search_embedding/touch/decay/node_count/edge_count via
  Java_ai_neuron_engram_EngramDb_* entry points; 6 tests

- bindings/kotlin: EngramDb.kt (AutoCloseable JNI wrapper), EngramNode,
  EngramEdge, ActivatedNode, EngramTypes; build.gradle.kts; settings.gradle.kts

- bindings/typescript: engram-wasm crate (wasm-bindgen, serde-wasm-bindgen);
  WasmEngramDb with in-memory backend (sled not available in WASM);
  TypeScript wrapper (index.ts, types.ts, package.json, tsconfig.json)

- bindings/go: engram.go (CGo wrapper), engram.h (C header), engram_test.go
  (4 tests covering open/close/put_node/get_node/node_count/decay); go.mod

- engram-core: wasm feature gate for in-memory backend; mem_storage.rs;
  activation.activate_mem for WASM path; Node::with_id helper;
  salience.rs doctest fixed (text block)

- examples/basic.rs: consolidation section added
- examples/migrate.rs: migration API demonstration

Build: cargo build --workspace -- zero warnings, zero errors
Tests: 38 pass (25 engram-core + 7 engram-ffi + 6 engram-jni)
This commit is contained in:
Will Anderson
2026-04-27 16:00:47 -05:00
parent 1a609502c8
commit 2454c83e82
37 changed files with 4573 additions and 237 deletions
Download Patch File Download Diff File Expand all files Collapse all files
engram/crates/engram-ffi/src/lib.rs
+392 -41
View File
@@ -1,32 +1,40 @@
/// C FFI stubs for engram-core.
/// C FFI for engram-core.
///
/// These are minimal stubs for v0.1 — enough to link from Kotlin, TypeScript (via WASM
/// or Node native addon), and Go. Full binding generation will use cbindgen in v0.2.
///
/// All pointers passed across the FFI boundary must remain valid for the duration of
/// the call. Strings are null-terminated UTF-8. The caller owns all returned heap memory
/// and must free it via the corresponding `engram_free_*` function.
/// These functions form the stable ABI that Go (via CGo), Python (via ctypes),
/// and other native callers use. All pointers must remain valid for the duration
/// of the call. Strings are null-terminated UTF-8. The caller must free any
/// returned heap-allocated C string with `engram_free_string`.
///
/// # Safety
/// All functions in this module are `unsafe` because they accept raw pointers.
/// Callers are responsible for ensuring pointer validity and correct lifetimes.
use engram_core::EngramDb;
/// Every function in this module accepts raw pointers and is therefore `unsafe`.
/// Callers must ensure:
/// - All handle pointers came from `engram_open` and have not been freed.
/// - All string pointers are valid null-terminated UTF-8.
/// - Returned C strings are freed exactly once via `engram_free_string`.
use engram_core::{
ActivatedNode, EngramDb, MemoryTier, Node, NodeType,
};
use std::ffi::{CStr, CString};
use std::os::raw::c_char;
use std::path::Path;
use uuid::Uuid;
/// Opaque handle to an open EngramDb instance.
// ── Handle type ───────────────────────────────────────────────────────────────
/// Opaque handle wrapping an open `EngramDb`.
pub struct EngramHandle {
db: EngramDb,
}
/// Open an engram database at the given path.
// ── Lifecycle ─────────────────────────────────────────────────────────────────
/// Open or create an engram database at `path`.
///
/// Returns a heap-allocated handle on success, or null on failure.
/// The caller must eventually call `engram_close` to free the handle.
/// Returns a heap-allocated `EngramHandle` on success, null on error.
/// Must be freed with `engram_close`.
///
/// # Safety
/// `path` must be a valid, null-terminated UTF-8 string.
/// `path` must be a valid, non-null, null-terminated UTF-8 string.
#[no_mangle]
pub unsafe extern "C" fn engram_open(path: *const c_char) -> *mut EngramHandle {
if path.is_null() {
@@ -42,9 +50,9 @@ pub unsafe extern "C" fn engram_open(path: *const c_char) -> *mut EngramHandle {
}
}
/// Close and free an engram database handle.
/// Close and free an engram handle.
///
/// After this call, `handle` is invalid and must not be used.
/// After this call `handle` is invalid.
///
/// # Safety
/// `handle` must have been returned by `engram_open` and not yet freed.
@@ -55,64 +63,407 @@ pub unsafe extern "C" fn engram_close(handle: *mut EngramHandle) {
}
}
/// Return the number of nodes in the database.
///
/// Returns -1 on error.
// ── Statistics ────────────────────────────────────────────────────────────────
/// Return the total number of nodes. Returns -1 on error.
///
/// # Safety
/// `handle` must be a valid, non-null pointer from `engram_open`.
/// `handle` must be a valid non-null pointer from `engram_open`.
#[no_mangle]
pub unsafe extern "C" fn engram_node_count(handle: *const EngramHandle) -> i64 {
if handle.is_null() {
return -1;
}
match (*handle).db.node_count() {
Ok(n) => n as i64,
Err(_) => -1,
}
(*handle).db.node_count().map(|n| n as i64).unwrap_or(-1)
}
/// Return the number of edges in the database.
///
/// Returns -1 on error.
/// Return the total number of edges. Returns -1 on error.
///
/// # Safety
/// `handle` must be a valid, non-null pointer from `engram_open`.
/// `handle` must be a valid non-null pointer from `engram_open`.
#[no_mangle]
pub unsafe extern "C" fn engram_edge_count(handle: *const EngramHandle) -> i64 {
if handle.is_null() {
return -1;
}
match (*handle).db.edge_count() {
Ok(n) => n as i64,
Err(_) => -1,
}
(*handle).db.edge_count().map(|n| n as i64).unwrap_or(-1)
}
/// Apply salience decay across all nodes.
// ── Salience management ───────────────────────────────────────────────────────
/// Apply multiplicative decay to all node saliences.
///
/// Returns the number of nodes updated, or -1 on error.
/// `factor` should be in (0.0, 1.0). Returns nodes updated, or -1 on error.
///
/// # Safety
/// `handle` must be a valid, non-null pointer from `engram_open`.
/// `handle` must be a valid non-null pointer from `engram_open`.
#[no_mangle]
pub unsafe extern "C" fn engram_decay(handle: *mut EngramHandle, factor: f32) -> i64 {
if handle.is_null() {
return -1;
}
match (*handle).db.decay(factor) {
Ok(n) => n as i64,
Err(_) => -1,
(*handle).db.decay(factor).map(|n| n as i64).unwrap_or(-1)
}
// ── Node operations ───────────────────────────────────────────────────────────
/// Store a node from a JSON representation.
///
/// `json` must be a UTF-8 JSON object with at least:
/// `{ "content": "...", "node_type": "Memory"|"Concept"|..., "tier": "Episodic"|...,
/// "importance": 0.8, "embedding": [f32, ...] }`
///
/// Returns a heap-allocated UUID string on success, null on error.
/// Caller must free with `engram_free_string`.
///
/// # Safety
/// `handle` and `json` must be valid non-null pointers.
#[no_mangle]
pub unsafe extern "C" fn engram_put_node(
handle: *mut EngramHandle,
json: *const c_char,
) -> *mut c_char {
if handle.is_null() || json.is_null() {
return std::ptr::null_mut();
}
let json_str = match CStr::from_ptr(json).to_str() {
Ok(s) => s,
Err(_) => return std::ptr::null_mut(),
};
let node = match node_from_json(json_str) {
Some(n) => n,
None => return std::ptr::null_mut(),
};
match (*handle).db.put_node(node) {
Ok(id) => match CString::new(id.to_string()) {
Ok(s) => s.into_raw(),
Err(_) => std::ptr::null_mut(),
},
Err(_) => std::ptr::null_mut(),
}
}
/// Free a C string returned by engram FFI functions.
/// Retrieve a node by UUID and return it as JSON.
///
/// `id` must be a UUID string. Returns heap-allocated JSON on success, null if
/// not found or on error. Caller must free with `engram_free_string`.
///
/// # Safety
/// `s` must have been allocated by an engram FFI function, not by the caller.
/// `handle` and `id` must be valid non-null pointers.
#[no_mangle]
pub unsafe extern "C" fn engram_get_node(
handle: *const EngramHandle,
id: *const c_char,
) -> *mut c_char {
if handle.is_null() || id.is_null() {
return std::ptr::null_mut();
}
let id_str = match CStr::from_ptr(id).to_str() {
Ok(s) => s,
Err(_) => return std::ptr::null_mut(),
};
let uuid = match id_str.parse::<Uuid>() {
Ok(u) => u,
Err(_) => return std::ptr::null_mut(),
};
match (*handle).db.get_node(uuid) {
Ok(Some(node)) => match CString::new(node_to_json(&node)) {
Ok(s) => s.into_raw(),
Err(_) => std::ptr::null_mut(),
},
_ => std::ptr::null_mut(),
}
}
// ── Spreading activation ──────────────────────────────────────────────────────
/// Run spreading activation and return results as JSON.
///
/// `req_json` must be:
/// `{ "seeds": ["uuid", ...], "query_embedding": [f32, ...],
/// "max_depth": 3, "limit": 10 }`
///
/// Returns heap-allocated JSON array of `ActivatedNode` objects, or null.
/// Caller must free with `engram_free_string`.
///
/// # Safety
/// `handle` and `req_json` must be valid non-null pointers.
#[no_mangle]
pub unsafe extern "C" fn engram_activate(
handle: *const EngramHandle,
req_json: *const c_char,
) -> *mut c_char {
if handle.is_null() || req_json.is_null() {
return std::ptr::null_mut();
}
let json_str = match CStr::from_ptr(req_json).to_str() {
Ok(s) => s,
Err(_) => return std::ptr::null_mut(),
};
let (seeds, query_emb, max_depth, limit) = match parse_activate_request(json_str) {
Some(r) => r,
None => return std::ptr::null_mut(),
};
match (*handle).db.activate(&seeds, &query_emb, max_depth, limit) {
Ok(results) => {
let json = activated_nodes_to_json(&results);
match CString::new(json) {
Ok(s) => s.into_raw(),
Err(_) => std::ptr::null_mut(),
}
}
Err(_) => std::ptr::null_mut(),
}
}
/// Free a C string returned by any engram FFI function.
///
/// # Safety
/// `s` must have been allocated by an engram FFI function. Do not call twice.
#[no_mangle]
pub unsafe extern "C" fn engram_free_string(s: *mut c_char) {
if !s.is_null() {
drop(CString::from_raw(s));
}
}
// ── JSON helpers ──────────────────────────────────────────────────────────────
// Minimal hand-rolled JSON to avoid adding serde_json as a dependency.
// These are intentionally simple — they handle the subset we need.
fn node_from_json(json: &str) -> Option<Node> {
// Extract fields with simple string scanning.
let content = extract_string(json, "content").unwrap_or_default();
let node_type_str = extract_string(json, "node_type").unwrap_or_else(|| "Memory".into());
let tier_str = extract_string(json, "tier").unwrap_or_else(|| "Episodic".into());
let importance: f32 = extract_number(json, "importance").unwrap_or(0.5);
let embedding = extract_float_array(json, "embedding").unwrap_or_default();
let node_type = match node_type_str.as_str() {
"Concept" => NodeType::Concept,
"Event" => NodeType::Event,
"Entity" => NodeType::Entity,
"Process" => NodeType::Process,
"InternalState" => NodeType::InternalState,
_ => NodeType::Memory,
};
let tier = match tier_str.as_str() {
"Working" => MemoryTier::Working,
"Semantic" => MemoryTier::Semantic,
"Procedural" => MemoryTier::Procedural,
_ => MemoryTier::Episodic,
};
Some(Node::new(node_type, embedding, content.into_bytes(), tier, importance))
}
fn node_to_json(node: &Node) -> String {
let content = String::from_utf8_lossy(&node.content);
let node_type = format!("{:?}", node.node_type);
let tier = format!("{:?}", node.tier);
let emb_str = node
.embedding
.iter()
.map(|f| format!("{:.6}", f))
.collect::<Vec<_>>()
.join(",");
format!(
r#"{{"id":"{}","node_type":"{}","tier":"{}","content":"{}","salience":{:.6},"importance":{:.6},"activation_count":{},"embedding":[{}]}}"#,
node.id,
node_type,
tier,
content.replace('"', "\\\""),
node.salience,
node.importance,
node.activation_count,
emb_str,
)
}
fn activated_nodes_to_json(nodes: &[ActivatedNode]) -> String {
let items: Vec<String> = nodes
.iter()
.map(|a| {
format!(
r#"{{"node":{},"activation_strength":{:.6},"hops":{}}}"#,
node_to_json(&a.node),
a.activation_strength,
a.hops,
)
})
.collect();
format!("[{}]", items.join(","))
}
fn parse_activate_request(json: &str) -> Option<(Vec<Uuid>, Vec<f32>, u8, usize)> {
let seeds_raw = extract_string_array(json, "seeds")?;
let seeds: Vec<Uuid> = seeds_raw
.iter()
.filter_map(|s| s.parse::<Uuid>().ok())
.collect();
let query_emb = extract_float_array(json, "query_embedding")?;
let max_depth = extract_number(json, "max_depth").unwrap_or(3.0) as u8;
let limit = extract_number(json, "limit").unwrap_or(10.0) as usize;
Some((seeds, query_emb, max_depth, limit))
}
// ── Tiny JSON field extractors ────────────────────────────────────────────────
fn extract_string(json: &str, key: &str) -> Option<String> {
let needle = format!("\"{}\":", key);
let start = json.find(&needle)? + needle.len();
let rest = json[start..].trim_start();
if !rest.starts_with('"') {
return None;
}
let inner = &rest[1..];
let end = inner.find('"')?;
Some(inner[..end].to_string())
}
fn extract_number(json: &str, key: &str) -> Option<f32> {
let needle = format!("\"{}\":", key);
let start = json.find(&needle)? + needle.len();
let rest = json[start..].trim_start();
let end = rest
.find(|c: char| c == ',' || c == '}' || c == ']')
.unwrap_or(rest.len());
rest[..end].trim().parse::<f32>().ok()
}
fn extract_float_array(json: &str, key: &str) -> Option<Vec<f32>> {
let needle = format!("\"{}\":", key);
let start = json.find(&needle)? + needle.len();
let rest = json[start..].trim_start();
if !rest.starts_with('[') {
return None;
}
let end = rest.find(']')?;
let inner = &rest[1..end];
let floats: Vec<f32> = inner
.split(',')
.filter_map(|s| s.trim().parse::<f32>().ok())
.collect();
Some(floats)
}
fn extract_string_array(json: &str, key: &str) -> Option<Vec<String>> {
let needle = format!("\"{}\":", key);
let start = json.find(&needle)? + needle.len();
let rest = json[start..].trim_start();
if !rest.starts_with('[') {
return None;
}
let end = rest.find(']')?;
let inner = &rest[1..end];
let strings: Vec<String> = inner
.split(',')
.filter_map(|s| {
let s = s.trim();
if s.starts_with('"') && s.ends_with('"') {
Some(s[1..s.len() - 1].to_string())
} else {
None
}
})
.collect();
Some(strings)
}
// ── Tests ─────────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
use std::ffi::CString;
#[test]
fn open_and_close() {
let dir = tempfile::tempdir().unwrap();
let path = CString::new(dir.path().to_str().unwrap()).unwrap();
unsafe {
let handle = engram_open(path.as_ptr());
assert!(!handle.is_null());
engram_close(handle);
}
}
#[test]
fn null_path_returns_null() {
unsafe {
let handle = engram_open(std::ptr::null());
assert!(handle.is_null());
}
}
#[test]
fn put_and_get_node_roundtrip() {
let dir = tempfile::tempdir().unwrap();
let path = CString::new(dir.path().to_str().unwrap()).unwrap();
let json = CString::new(
r#"{"content":"hello","node_type":"Memory","tier":"Episodic","importance":0.8,"embedding":[0.1,0.2,0.3]}"#,
)
.unwrap();
unsafe {
let handle = engram_open(path.as_ptr());
assert!(!handle.is_null());
let uuid_ptr = engram_put_node(handle, json.as_ptr());
assert!(!uuid_ptr.is_null());
let uuid_str = CStr::from_ptr(uuid_ptr).to_str().unwrap().to_string();
engram_free_string(uuid_ptr);
// Now get the node back.
let id_cstr = CString::new(uuid_str).unwrap();
let node_json_ptr = engram_get_node(handle, id_cstr.as_ptr());
assert!(!node_json_ptr.is_null());
let node_json = CStr::from_ptr(node_json_ptr).to_str().unwrap().to_string();
assert!(node_json.contains("hello"));
engram_free_string(node_json_ptr);
assert_eq!(engram_node_count(handle), 1);
engram_close(handle);
}
}
#[test]
fn node_count_and_edge_count() {
let dir = tempfile::tempdir().unwrap();
let path = CString::new(dir.path().to_str().unwrap()).unwrap();
unsafe {
let handle = engram_open(path.as_ptr());
assert_eq!(engram_node_count(handle), 0);
assert_eq!(engram_edge_count(handle), 0);
engram_close(handle);
}
}
#[test]
fn extract_string_works() {
let json = r#"{"content":"hello world","importance":0.5}"#;
assert_eq!(extract_string(json, "content"), Some("hello world".into()));
}
#[test]
fn extract_number_works() {
let json = r#"{"importance":0.75,"other":1}"#;
let v = extract_number(json, "importance").unwrap();
assert!((v - 0.75).abs() < 1e-4);
}
#[test]
fn extract_float_array_works() {
let json = r#"{"embedding":[0.1,0.2,0.3]}"#;
let arr = extract_float_array(json, "embedding").unwrap();
assert_eq!(arr.len(), 3);
assert!((arr[0] - 0.1).abs() < 1e-4);
}
}