Compare commits

..

1 Commits

Author SHA1 Message Date
Andre Botelho Rodrigues Almeida b97b644799 Addind readme.md file to start documenting the repo
El SDK CI - dev / build-and-test (pull_request) Successful in 8m18s
2026-07-23 16:41:51 -03:00
2 changed files with 202 additions and 140 deletions
+154
View File
@@ -0,0 +1,154 @@
# El
**A self-hosting, statically-typed language that compiles to C — built around a graph-native runtime instead of a database driver.**
El is the execution substrate for the Neuron agent runtime, the DHARMA network, and the Engram knowledge graph. This repository is the monorepo for the whole stack: the language itself, the graph memory engine it's built to talk to natively, and the tools (package manager, IDE, UI framework, diagramming) built on top of it.
---
## Why El exists
Every other language treats persistent, associative state as something you reach for through a driver — a SQL client, an ORM, a Redis library bolted on from outside. El inverts that: graph operations (`engram_*`) are runtime primitives, on the same footing as string or list operations. There is no separate database driver because the database is not separate.
El has four defining properties:
1. **Self-hosting compiler.** The compiler (`lexer.el`, `parser.el`, `codegen.el`, `compiler.el`) is written in El. It compiles El source to C, which `cc` compiles against a fixed runtime into a native binary. A Rust genesis compiler bootstrapped the first iteration; the self-hosted binary at `lang/dist/platform/elc` has been the canonical compiler ever since — every binary in `dist/platform/` was produced by an earlier version of itself compiling `el-compiler/src/`. The chain is auditable: source is the ground truth, not the binary. See [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) for the full recovery path if that binary is ever lost.
2. **C compilation target.** Every compiled program is plain C11. Every El value is `el_val_t` (`int64_t`); strings are heap pointers cast through it. Functions become C functions; top-level statements become `main()`.
3. **Graph-native runtime.** The runtime provides first-class graph operations over an in-process Engram store — no separate DB driver, no ORM.
4. **DHARMA-aware identity.** A `cgi` block declares a program's DHARMA identity at compile time. The runtime resolves identity before user code runs, so `dharma_*` calls have a stable principal and channel surface throughout.
---
## Architecture map
```
┌─────────────┐
│ lang │ El compiler + C runtime
│ (El itself) │ everything below is written in it,
└──────┬──────┘ or compiles down through it
┌─────────────┼─────────────┐
│ │ │
┌──────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐
│ engram │ │ epm │ │ ide │
│ graph/mem │ │ package │ │ editor + │
│ substrate │ │ manager │ │ LSP │
└──────┬─────┘ └───────────┘ └───────────┘
┌───────┼────────────────┬─────────────────────┐
│ │ │ │
┌─────▼───┐ ┌─▼──────────┐ ┌──▼──────────┐ ┌─────▼──────┐
│ elp │ │ ql │ │ ui │ │ arbor │
│ NLG / │ │engram-el. │ |spreading- │ |arbor │
│ 31 langs│ │studio+tests│ |activation UI│ |diagram lang│
└─────────┘ └────────────┘ └─────────────┘ └────────────┘
```
`lang` is the foundation — the compiler and C runtime everything else builds on. `engram` is the graph-native memory/state engine that gives El its identity (property 3 above). Everything else is either a tool for working with El (`epm`, `ide`) or a system built on top of Engram's graph model (`elp`, `ql`, `ui`, `arbor`).
---
## Repository layout
### [lang/](lang/) — the El language
The compiler and runtime. Self-hosting: `elc-cli.el``compiler.el``lexer.el` / `parser.el` / `codegen.el` / `codegen-js.el`, textually inlined and compiled in one pass. Compiles to C11 and links against `el-compiler/runtime/el_seed.c`, a hand-maintained OS-boundary layer (libcurl HTTP, pthreads, filesystem, arena allocation) — everything else in the runtime is native El (`runtime/*.el`).
Two layers to know: **El programs** (`.el` files — where nearly all work belongs) and **the C seed** (`el_seed.c` — edit only for genuine OS-level access; never re-implement what El can already express).
Current status (single source of truth: [lang/spec/language.md](lang/spec/language.md)): lexer/parser/codegen and the C runtime's core (I/O, strings, math, lists, maps, filesystem, args) are implemented. In flight: `%` operator, match-statement codegen, `?` nil-propagation, `cgi` block parsing + DHARMA identity resolution, VBD role enforcement (`@manager`/`@engine`/`@accessor`), the real `engram_*` and `dharma_*` runtimes (currently stubs), and libcurl-backed `http_get`/`http_post`/`http_serve`. Bitwise operators, `??`, and `as` casts are explicitly **not** in this language.
Key docs: [AGENTS.md](lang/AGENTS.md) (agent-facing orientation), [BOOTSTRAP.md](lang/BOOTSTRAP.md) (compiler recovery from scratch), [spec/language.md](lang/spec/language.md), [spec/codegen-js.md](lang/spec/codegen-js.md).
### [engram/](engram/) — graph intelligence substrate
**A local-first memory substrate for accumulating intelligence**, and the reason El's runtime doesn't need a database driver. Rust core (`engram-core`, `engram-ffi`) exposed to El and other languages (Kotlin, TypeScript/WASM, Go bindings).
The model: retrieval is **spreading activation**, not query. You name seed nodes and a query embedding; activation propagates outward through weighted edges, attenuating multiplicatively per hop (`strength = parent_strength × edge_weight × target_salience × cosine_sim`), gets pruned below a threshold, and the top-N nodes by activation strength come back. Storage and retrieval are the same structure — the way long-term potentiation works in biological memory, not the way a relational or vector database works.
Nodes live in four tiers (Working / Episodic / Semantic / Procedural, mirroring prefrontal / hippocampal / neocortical / cerebellar memory) and migrate between them based on **salience decay**`importance × recency-decay × log(activation_count)`. Forgetting is adaptive pruning, not a bug: unreinforced memories stop competing for attention without being deleted.
Backed by `sled` (embedded, local-first, no daemon) with flat cosine scan for vector search — deliberately simple until scale demands an HNSW layer. Full API and design rationale in [engram/README.md](engram/README.md).
### [elp/](elp/) — Engram Language Protocol
Bidirectional engine mapping between Engram semantic forms and natural-language surface text, across **31 languages** — from Spanish and Japanese through historical/liturgical languages (Old Norse, Sanskrit, Sumerian, Coptic, Akkadian, Ge'ez). Compilation order runs `language-profile` + `vocabulary` → per-language `morphology-*``grammar``realizer``semantics``elp`. This is what lets an Engram graph node round-trip to and from readable text in any of those languages.
### [epm/](epm/) — El Package Manager
Manages **vessels** (El's package unit): publish, install, resolve dependencies. Vessels are stored in Engram as graph nodes, not files in a registry index — `epm` reads the local `manifest.el`, talks to Engram over HTTP, and writes resolved vessels to `.epm/vessels/`. Source: `registry.el`, `install.el`, `update.el`, `manifest.el`.
### [ide/](ide/) — El IDE
Three vessels: **el-ide-server** (HTTP backend — file ops, build/run, LSP bridge, plugin host, settings), **el-lsp** (the language server — completion, hover, diagnostics, outline, format, type graph), and **el-plugin-host** (first-party plugin lifecycle: install/remove/enable/disable). `ide/projects/` and `ide/examples/` hold sample projects, including the canonical `hello-friends` first-program walkthrough.
### [ql/](ql/) — engram-el
The El-native integration layer for a *live* Engram server — not a library (no importable modules, no build artifact), a set of standalone `.el` programs run directly via `el run-file`. Three components: **Studio** (`studio/studio.el`, a full terminal graph explorer), a **Hebbian field-model** proof of concept, and El builtin / LLM-builtin smoke test suites. This is the reference for correct patterns when an El program uses Engram as its substrate. Spec: [ql/spec/elql.md](ql/spec/elql.md).
### [ui/](ui/) — el-ui
A frontend framework where **component state is an Engram graph and reactivity is spreading activation** — not virtual-DOM diffing (React), Proxy-based dependency tracking (Vue), or compile-time analysis (Svelte). Re-renders are activated and propagated the same way associative memory retrieval works in `engram/`.
~15 vessels covering the full frontend surface: `el-platform` (env/fs/network/clock abstraction), `el-config`, `el-html` (SSR emit primitives), `el-layout`, `el-style` (design tokens/themes), `el-i18n`, `el-auth` / `el-identity` (JWT, sessions, OAuth PKCE — Engram-native), `el-services` (REST/gRPC/WebSocket bindings), `el-aop` (`@authenticate`/`@authorize`/`@cache`/`@rate_limit` decorators), `el-secrets`, `el-graph` (graph rendering/editor), `el-publish` (App Store / Play Store automation), and `el-ui-compiler` (El→JS component compiler; currently a stub pending a JS backend in `elc`). Spec: [ui/spec/framework.md](ui/spec/framework.md).
### [arbor/](arbor/) — diagram language
A `.arbor` diagram language and toolchain: `arbor-core` (NodeId/shape/edge-kind types), `arbor-parse` (recursive-descent parser), `arbor-diagram` (IR + Mermaid serializer + architecture-diagram builders), `arbor-layout` (hierarchical layout — rank assignment, positioning, group bounds), `arbor-render` (SVG renderer), `arbor-cli`. (The architecture map above is the kind of diagram this is for.)
---
## Getting started
Install the El SDK from the latest release:
```bash
bash lang/install.sh
# EL_VERSION=v1.0.0 bash lang/install.sh # pin a specific release tag
# EL_PREFIX=/opt/el bash lang/install.sh # custom install prefix
```
Or build the compiler from source and verify the self-hosting chain:
```bash
cd lang
./dist/platform/elc elc-cli.el > elc-new.c
cc -std=c11 -I el-compiler/runtime -lcurl -lpthread \
-o dist/platform/elc-new \
elc-new.c el-compiler/runtime/el_seed.c
# Confirm the new binary reproduces itself exactly
./dist/platform/elc-new elc-cli.el > elc-verify.c
diff elc-new.c elc-verify.c # should be identical
mv dist/platform/elc-new dist/platform/elc
```
Run your first program:
```bash
./lang/dist/platform/elc lang/examples/hello.el > hello.c
cc -std=c11 -I lang/el-compiler/runtime -lcurl -lpthread \
-o hello hello.c lang/el-compiler/runtime/el_seed.c
./hello
```
More examples in [lang/examples/](lang/examples/), including a full starter project at `lang/examples/hello-project/`.
If the compiler binary is ever lost or corrupted, [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) is the authoritative recovery path.
---
## Development workflow
Branching follows `dev → stage → main`: work lands on `dev`, promotes to `stage` for integration testing, and is promoted to `main` for release (visible directly in the git history of this repo). CI is defined per-subproject under `.gitea/workflows/``lang`/`epm`/`ide` share the root pipeline; `engram` and `ql` carry their own (`ci-dev`, `ci-stage`, and a release workflow each).
- Language/runtime specs live at `*/spec/*.md` (`lang/spec/`, `ql/spec/`, `ui/spec/`) and are the single source of truth for implemented-vs-planned status — code and docs are expected to agree with the spec's status markers, not the other way around.
- Agent-facing orientation guides live at `*/AGENTS.md` (currently `lang/AGENTS.md`); more subprojects may grow their own as they need agent-specific conventions documented.
- Tagged releases live under `lang/releases/`, each with its own `RELEASE.md`.
---
## Status
This is an actively developed, internal monorepo — not yet published under an open license. Treat everything here as proprietary to Neuron Technologies unless told otherwise.
+48 -140
View File
@@ -7056,16 +7056,10 @@ static float* engram_embed_raw(const char* prefix, const char* text, int* out_di
char* esc = engram_json_escape(text);
free(trunc);
if (!esc || !esc_prefix) { free(esc); free(esc_prefix); return NULL; }
size_t blen = strlen(esc) + strlen(esc_prefix) + strlen(model) + 96;
size_t blen = strlen(esc) + strlen(esc_prefix) + strlen(model) + 64;
char* body = malloc(blen);
if (!body) { free(esc); free(esc_prefix); return NULL; }
/* keep_alive:-1 pins the embed model resident in Ollama indefinitely.
* Without it the tiny embed model is evicted whenever a large generation
* model loads (unified-memory pressure), so the NEXT search pays a cold
* model reload the dominant search-latency cost (measured cold reload
* up to ~2.2s vs ~0.02-0.05s warm). Pinning makes cold reload impossible. */
snprintf(body, blen, "{\"model\":\"%s\",\"keep_alive\":-1,\"prompt\":\"%s%s\"}",
model, esc_prefix, esc);
snprintf(body, blen, "{\"model\":\"%s\",\"prompt\":\"%s%s\"}", model, esc_prefix, esc);
free(esc); free(esc_prefix);
CURL* c = curl_easy_init();
@@ -7105,52 +7099,11 @@ static int engram_semantic_enabled(void) {
g_emb_state = -1; return 0;
}
/* ── Query-embedding cache ──────────────────────────────────────────────────
* The node embeddings are cached (engram_node_vec) but the QUERY was re-embedded
* on every search/activate call a blocking Ollama round-trip each time. Query
* embeddings are deterministic for a given model, so we cache them keyed by an
* FNV-1a hash of the query string (with a full strcmp to reject hash
* collisions). A repeated query then costs zero network round-trips. This makes
* warm search latency independent of Ollama entirely, and directly serves the
* curiosity loop, which reseeds the same query terms repeatedly. Direct-mapped,
* fixed-size, process-lifetime. */
#define ENGRAM_QCACHE_SIZE 1024
typedef struct { char* q; uint64_t hash; float* vec; int dim; } EngramQCacheEntry;
static EngramQCacheEntry g_qcache[ENGRAM_QCACHE_SIZE];
/* Returns a malloc'd COPY of the cached vector (caller frees), or NULL on miss —
* preserving engram_embed_query's "caller frees" contract. */
static float* engram_qcache_get(const char* q, uint64_t h, int* dim) {
EngramQCacheEntry* e = &g_qcache[h & (ENGRAM_QCACHE_SIZE - 1)];
if (e->vec && e->hash == h && e->q && strcmp(e->q, q) == 0 && e->dim > 0) {
float* copy = malloc((size_t)e->dim * sizeof(float));
if (!copy) return NULL;
memcpy(copy, e->vec, (size_t)e->dim * sizeof(float));
*dim = e->dim; return copy;
}
return NULL;
}
static void engram_qcache_put(const char* q, uint64_t h, const float* vec, int dim) {
if (!vec || dim <= 0) return;
EngramQCacheEntry* e = &g_qcache[h & (ENGRAM_QCACHE_SIZE - 1)];
float* stored = malloc((size_t)dim * sizeof(float));
char* qcopy = el_strdup(q);
if (!stored || !qcopy) { free(stored); free(qcopy); return; }
memcpy(stored, vec, (size_t)dim * sizeof(float));
free(e->q); free(e->vec); /* evict prior occupant of this slot */
e->q = qcopy; e->hash = h; e->vec = stored; e->dim = dim;
}
/* Embed the query. Returns malloc'd vec (caller frees), or NULL if semantic off. */
static float* engram_embed_query(const char* q, int* dim) {
if (!engram_semantic_enabled()) return NULL;
if (!q || !*q) return NULL;
uint64_t h = engram_fnv1a(q);
float* hit = engram_qcache_get(q, h, dim);
if (hit) return hit;
float* v = engram_embed_raw("search_query: ", q, dim);
if (v && *dim > 0) engram_qcache_put(q, h, v, *dim);
return v;
return engram_embed_raw("search_query: ", q, dim);
}
/* Cached node embedding. Returns a pointer OWNED BY THE CACHE — do not free. */
@@ -7584,39 +7537,6 @@ static double engram_goal_bias(const EngramNode* n, const char* query) {
return bias;
}
/* ── Beam cap for engram_activate spreading activation ──────────────────────
* Bounds the number of frontier nodes expanded PER HOP. Without it a single
* high-degree hub enqueues thousands of successors, each re-scanning the whole
* edge list, and dense cycles re-enqueue them repeatedly so capping DEPTH
* does not bound work (measured: depth-2/3 in the multi-second range, depth-3
* can crash). With the cap, only the top-BEAM highest-activation nodes at each
* level spread further. Every reached node is still recorded and returned, so
* recall is preserved the cap bounds only associative spread, never the
* direct seed matches or the reported set. Tunable via ENGRAM_ACTIVATE_BEAM
* (default 128); set very high to restore unbounded behaviour. */
static int64_t engram_activate_beam(void) {
static int64_t v = -1;
if (v >= 0) return v;
const char* s = getenv("ENGRAM_ACTIVATE_BEAM");
int64_t d = 128;
if (s && *s) { char* e = NULL; long t = strtol(s, &e, 10); if (e != s && t > 0) d = (int64_t)t; }
v = d; return v;
}
/* Partition the k highest-`score` entries of idx[0..n) to the front (order
* within the top-k is unspecified). O(k*n) partial selection k is the small
* beam width, so this is cheap relative to a hop's edge scan. */
static void engram_beam_select(int64_t* idx, int64_t n, int64_t k, const double* score) {
if (k >= n) return;
for (int64_t i = 0; i < k; i++) {
int64_t best = i;
for (int64_t j = i + 1; j < n; j++)
if (score[idx[j]] > score[idx[best]]) best = j;
if (best != i) { int64_t t = idx[i]; idx[i] = idx[best]; idx[best] = t; }
}
}
el_val_t engram_activate(el_val_t query, el_val_t depth) {
EngramStore* g = engram_get();
const char* q = EL_CSTR(query);
@@ -7686,65 +7606,53 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
for (int64_t s = 1; s < seed_count; s++)
seed_epoch = (seed_epoch + seeds[s].created_at) / 2;
}
/* ── Beam-capped, level-synchronous BFS ────────────────────────────────
* Expand the graph hop-by-hop; at each hop expand only the top-`beam`
* nodes by current best background activation (engram_beam_select). This
* replaces the old unbounded FIFO frontier, which let a hub enqueue
* thousands of successors and dense cycles re-enqueue them without limit
* (the breadth explosion). `reached` / `best_bg` / `best_hops` keep the
* exact same meaning, so the downstream executive/override passes and the
* reported result set are unchanged only how far weak spread propagates
* is bounded. `cur`/`nxt` hold node indices for this/next level; `in_nxt`
* dedups a node to at most one entry per level. */
const int64_t beam = engram_activate_beam();
const double SPREAD_DECAY = 0.7;
int64_t* cur = malloc((size_t)g->node_count * sizeof(int64_t));
int64_t* nxt = malloc((size_t)g->node_count * sizeof(int64_t));
int* in_nxt = calloc((size_t)g->node_count, sizeof(int));
if (!cur || !nxt || !in_nxt) {
free(cur); free(nxt); free(in_nxt);
typedef struct { int64_t idx; int64_t hops; double act; } Frontier;
Frontier* fr = malloc((size_t)(g->node_count * (max_depth + 1)) * sizeof(Frontier) + 16 * sizeof(Frontier));
if (!fr) {
free(best_bg); free(best_hops); free(reached); free(seeds); return out;
}
int64_t cur_n = 0;
for (int64_t s = 0; s < seed_count && cur_n < g->node_count; s++)
cur[cur_n++] = seeds[s].idx;
for (int64_t hop = 0; hop < max_depth && cur_n > 0; hop++) {
if (cur_n > beam) { engram_beam_select(cur, cur_n, beam, best_bg); cur_n = beam; }
int64_t nxt_n = 0;
for (int64_t ci = 0; ci < cur_n; ci++) {
int64_t fidx = cur[ci];
double f_act = best_bg[fidx];
const char* cur_id = g->nodes[fidx].id;
for (int64_t ei = 0; ei < g->edge_count; ei++) {
EngramEdge* e = &g->edges[ei];
const char* other = NULL;
if (e->from_id && strcmp(e->from_id, cur_id) == 0) other = e->to_id;
else if (e->to_id && strcmp(e->to_id, cur_id) == 0) other = e->from_id;
else continue;
int64_t oi = engram_find_node_index(other);
if (oi < 0) continue;
EngramNode* on = &g->nodes[oi];
double tbonus = engram_temporal_proximity_bonus(on->created_at, seed_epoch);
double tdecay = engram_temporal_decay(on, now_ms);
double dampen = engram_activation_dampen(on);
double new_act = f_act * e->weight * SPREAD_DECAY * (1.0 + tbonus)
* tdecay * dampen;
if (!reached[oi] || new_act > best_bg[oi]) {
best_bg[oi] = new_act;
best_hops[oi] = hop + 1;
reached[oi] = 1;
if (!in_nxt[oi] && nxt_n < g->node_count) {
in_nxt[oi] = 1;
nxt[nxt_n++] = oi;
}
int64_t fhead = 0, ftail = 0;
int64_t fcap = (int64_t)((size_t)(g->node_count * (max_depth + 1)) + 16);
for (int64_t s = 0; s < seed_count; s++) {
if (ftail >= fcap) break;
fr[ftail].idx = seeds[s].idx;
fr[ftail].hops = 0;
fr[ftail].act = seeds[s].act;
ftail++;
}
const double SPREAD_DECAY = 0.7;
while (fhead < ftail) {
Frontier f = fr[fhead++];
if (f.hops >= max_depth) continue;
const char* cur_id = g->nodes[f.idx].id;
for (int64_t ei = 0; ei < g->edge_count; ei++) {
EngramEdge* e = &g->edges[ei];
const char* other = NULL;
if (e->from_id && strcmp(e->from_id, cur_id) == 0) other = e->to_id;
else if (e->to_id && strcmp(e->to_id, cur_id) == 0) other = e->from_id;
else continue;
int64_t oi = engram_find_node_index(other);
if (oi < 0) continue;
EngramNode* on = &g->nodes[oi];
double tbonus = engram_temporal_proximity_bonus(on->created_at, seed_epoch);
double tdecay = engram_temporal_decay(on, now_ms);
double dampen = engram_activation_dampen(on);
double new_act = f.act * e->weight * SPREAD_DECAY * (1.0 + tbonus)
* tdecay * dampen;
int64_t new_hops = f.hops + 1;
if (!reached[oi] || new_act > best_bg[oi]) {
best_bg[oi] = new_act;
best_hops[oi] = new_hops;
reached[oi] = 1;
if (ftail < fcap) {
fr[ftail].idx = oi;
fr[ftail].hops = new_hops;
fr[ftail].act = new_act;
ftail++;
}
}
}
for (int64_t k = 0; k < nxt_n; k++) in_nxt[nxt[k]] = 0;
int64_t* tmp = cur; cur = nxt; nxt = tmp;
cur_n = nxt_n;
}
free(cur); free(nxt); free(in_nxt);
/* Persist layer-1 background_activation to node store. */
for (int64_t i = 0; i < g->node_count; i++) {
g->nodes[i].background_activation = reached[i] ? best_bg[i] : 0.0;
@@ -7758,7 +7666,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
* memory weight cannot be silenced by attentional suppression. */
double* inhibition = calloc((size_t)g->node_count, sizeof(double));
if (!inhibition) {
free(best_bg); free(best_hops); free(reached); free(seeds);
free(best_bg); free(best_hops); free(reached); free(seeds); free(fr);
return out;
}
for (int64_t ei = 0; ei < g->edge_count; ei++) {
@@ -7784,7 +7692,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
double* wm_weights = calloc((size_t)g->node_count, sizeof(double));
if (!wm_weights) {
free(best_bg); free(best_hops); free(reached); free(seeds);
free(inhibition); return out;
free(fr); free(inhibition); return out;
}
for (int64_t i = 0; i < g->node_count; i++) {
if (!reached[i] || best_bg[i] <= 0.0) continue;
@@ -7854,7 +7762,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
int64_t rcount = 0;
if (!results) {
free(best_bg); free(best_hops); free(reached); free(seeds);
free(inhibition); free(wm_weights); return out;
free(fr); free(inhibition); free(wm_weights); return out;
}
for (int64_t i = 0; i < g->node_count; i++) {
if (!reached[i]) continue;
@@ -7898,7 +7806,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
out = el_list_append(out, entry);
}
free(best_bg); free(best_hops); free(reached);
free(seeds); free(inhibition); free(wm_weights); free(results);
free(seeds); free(fr); free(inhibition); free(wm_weights); free(results);
return out;
}