Compare commits

...

143 Commits

Author SHA1 Message Date
will.anderson 1c9de03fdb Merge pull request 'engram: make the ggml batch-cosine strategy actually compute in fp32 (recall 0.9933 -> 0.9987)' (#121) from improve/ggml-cosine-fp32-and-init into feat/engram-ggml-cosine-batch
El SDK CI - dev / build-and-test (pull_request) Failing after 3m43s
2026-08-15 23:01:29 +00:00
bigmerge c008b7228a engram: make the ggml batch-cosine strategy actually compute in fp32
#116 shipped the ggml strategy at 0.9933 id-recall against the CPU oracle
while the hand-rolled Metal kernel it replaced scored 0.9997 — a ~150x worse
error margin. That was not an inherent property of ggml. It was a usage bug in
this file, and this commit fixes it.

ggml-metal has two F32xF32 matmul kernels and picks between them purely on
ne11, the number of B rows, which for us is the query-batch size:

  ne11 <= 8  -> kernel_mul_mv_ext_f32_f32_* / kernel_mul_mv_f32_f32_*,
                templated <float, float> — genuine F32.
  ne11 >  8  -> kernel_mul_mm_f32_f32, templated
                <half, half4x4, simdgroup_half8x8, half, half2x4, ...> —
                BOTH operands narrowed to F16, despite F32 tensors on both
                sides.

The old code issued one ggml_mul_mat with ne11 = nq (300 in the benchmark),
landing squarely on the F16 path. The file's own header comment asserted the
opposite ("computes in F32 on the Metal backend"); that claim was wrong and is
replaced with the measurement.

Fix: emit ceil(nq/8) mul_mats over ne11<=8 ggml_view_2d slices of one query
tensor, all expanded into ONE graph and one ggml_backend_graph_compute, so the
node matrix is still uploaded and shared exactly once. EL_GGML_MULMAT_CHUNK
overrides the 8; setting it >= nq reproduces the old behaviour exactly, which
is also how the before/after below was measured in a single binary.

Measured, real store snapshot, 13415 live embedded nodes, dim=768, 300 real
queries, vs the CPU double-accumulated oracle (vindex_bench, offline copy of
the store — no live service touched):

  id-recall   same-rank |Δdist| max   mean
  old (ne11=300)   0.9933   6.80e-05   1.43e-05
  new (ne11<=8)    0.9987   4.77e-07   9.30e-08
  hand-rolled      0.9997   3.58e-07   7.55e-08

~145x better max error, ~154x better mean — now the same order of magnitude as
the hand-rolled kernel rather than 150x off it.

The cost is real and is documented rather than buried. Median of 15 reps of
the whole batch_multi() call, three runs: 13.2-14.4ms unchunked, 19.9-20.2ms
chunked, 17.7-18.0ms hand-rolled. Correctness costs ~+6.7ms per 300-query
batch and leaves ggml ~12% behind the hand-rolled kernel instead of ~35%
ahead. It cannot be recovered inside ggml: an fp32 matmul on Metal must
re-stream the node matrix once per <=8 queries, and ggml's Metal backend ships
no fp32 TILED matmul, so "fast" and "fp32" are genuinely exclusive there.

Two things that did NOT work, recorded so nobody retries them:

  - ggml_mul_mat_set_prec(t, GGML_PREC_F32) does nothing here. Error was
    bit-identical with and without it (1.038e-05 either way) — ggml-metal has
    no F32-accumulating mul_mm kernel to switch to. ne11 is the only lever.
  - The ACCEL/BLAS device looked excellent in an isolated compute-only probe
    (3.4-4.0ms, mean |Δdot| 1.5e-08) but is dominated on BOTH axes end-to-end
    (0.191 ms/query at 0.9973 recall vs 0.125-0.142 at 0.9987), because the
    probe was not competing for the same CPU cores the real call path is. It
    stays reachable via EL_GGML_DEVICE as a no-Metal fallback, labelled as
    measured-and-rejected, not as a recommendation.

Also corrected: the ~7.8s "cold start" blamed on this file is not this file
re-initialising per call — init was already cached. It is Apple's shader cache
missing on ggml's embedded metallib (~650 kernels), keyed on the library and
shared across processes: the first load on a machine reports
"loaded in 7.670 sec", the next run of a *different* binary reports 0.009 sec.
Once per machine per ggml version, not once per process, and not ours to fix.
Warm ggml init is 44-53ms vs 36-117ms for the hand-rolled strategy.

Loading only libggml-metal.so instead of every plugin in the directory is kept
for tidiness, and explicitly documented as NOT a speedup: 44.7-52.4ms against
46.9-58.9ms, the same number inside noise.

The -2.0 sentinel contract is unchanged and re-verified at batch sizes that
straddle the chunk boundary (1,7,8,9,16,17,33), plus NULL rows, dim
mismatches, zero-norm rows, and an all-invalid population. Notably the old
ne11=300 path fails that same check at a 2e-6 cosine tolerance with 2299
mismatches, which is an independent confirmation of the defect.
2026-08-15 17:57:09 -05:00
bigmerge b3f410fc91 engram: batch-cosine Adapter/Strategy/Factory over ggml, supersedes hand-rolled PR #114
El SDK CI - dev / build-and-test (pull_request) Failing after 4m29s
Stop hand-rolling GPU kernels for batch cosine similarity — use ggml (the
MIT-licensed compute library underneath llama.cpp, installed standalone via
Homebrew) as the preferred backend, without ripping out PR #114's
carefully-verified hand-rolled Metal shader.

Structure: one stable public adapter (eg_cosine_batch.h, zero #ifdef at call
sites) backed by three selectable concrete Strategies behind an internal
vtable (eg_cosine_batch_strategy.h) chosen by a Factory (eg_cosine_batch.c):

  - eg_cosine_batch_strategy_ggml.c    — NEW. ggml + dynamically-loaded Metal
                                          backend plugin (ggml_backend_load_all_from_path
                                          + ggml_mul_mat for the batched dot
                                          product), gather/scatter around the
                                          -2.0 sentinel contract.
  - eg_cosine_batch_strategy_metal_hand.m — PR #114's original hand-rolled
                                          Metal shader bridge, preserved
                                          almost verbatim, now one strategy
                                          among several rather than the only
                                          option. eg_cosine_batch.metal kept
                                          byte-identical to the original.
  - eg_cosine_batch_strategy_cpu.c     — universal always-false fallback
                                          (direct descendant of PR #114's
                                          eg_metal_cosine_stub.c).

Selection: EL_COSINE_BATCH_STRATEGY=ggml|metal|cpu|auto (default: ggml first,
then hand-rolled Metal, then CPU — first available wins), plus back-compat
EL_METAL_COSINE=0 to disable every GPU-backed strategy. build_vindex_bench.sh
compiles all three strategies on Darwin, CPU-fallback-only elsewhere.

vindex_bench.c now reports BRUTE-GGML and BRUTE-METAL side by side against
the same CPU oracle, on the same dataset, in one run (real numbers vs. real
store snapshot in the PR body).
2026-08-15 17:16:41 -05:00
will.anderson 2555e363a6 Merge pull request 'engram: native set-based reframe_region + engine hardening (embed-gap, lazy cosine, vindex harvest)' (#109) from feat/reframe-region-setop into dev
El SDK CI - dev / build-and-test (push) Failing after 3m58s
2026-08-15 21:50:41 +00:00
bigmerge bacaf3d39c engram: reconcile M8 HNSW vindex (#109) onto current dev, restore 3 fixes the branch predated
El SDK CI - dev / build-and-test (pull_request) Failing after 4m49s
Lands feat/reframe-region-setop (PR #109: native set-based reframe_region,
decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency
work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated
seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's
actual current HEAD, plus engram-tiered-storage's still-unique test suite.

RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/
verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/
select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two
branches; #109's copy is a strict superset (adds vindex_harvest_from_store,
used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and
engram_verify.c are also byte-identical. #109's own branch point already
carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing
left to merge into #109 for those files. The one thing engram-tiered-storage
had that #109's tree dropped: its full test suite (test_vindex.c,
test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the
interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh
harnesses) — ported over here unchanged.

WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch
forked from dev on 2026-08-14 15:40 (before restructure-adjacent history
diverged the file's merge-base for `git merge` — it presented as an add/add
conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but
it silently reverted THREE dev fixes landed on 2026-08-14/15, after the
branch point, that the PR's diff had no way to know about:

  1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of
     the query-aware propagation gate dropped the shift-and-floor rescale
     about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate
     0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped
     around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi]
     is computed) is orthogonal to WHAT it gates on and both are kept.
  2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes
     wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is
     diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global).
     PR's tree predates this and dropped all three counters + their JSON
     stats fields. Restored declarations, all 4 direct increment sites, the
     eg_wm_carry_over bll increment, and the act-stats JSON fields --
     alongside (not instead of) the PR's own P4 afferent / API-reshape
     counters already in that same struct/JSON.
  3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev
     selects the STRONGEST qualifying candidate for consolidation each call;
     PR's tree predates this and reverted to hash-slot order (arbitrary wrt
     association strength) for edge formation -- the one path that writes
     PERMANENT structure. Restored the strongest-candidate while-loop,
     keeping the PR's own genuine improvement at that site
     (engram_adj_on_edge_added incremental-index append instead of a bare
     adj_dirty=1 full-rebuild flag).

engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env
flags, /api/nodes connected-count in responses) were pure additive: dev's
side was empty, PR's side added the feature. Took PR's side whole.

VERIFIED (nsbx sandbox only, live :8742/:7770 never touched):
  - cc -std=c11 -O2, clean link against the real engram/src/server.el via
    elc, zero errors.
  - vindex_bench (built standalone, read-only harvest) against the real
    production store clone (13,671 embedded nodes, 768-dim nomic-embed-text):
    recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs
    2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s
    for the full 13,671-node set -- see the flagged risk below.
  - Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned
    snapshot of the live store, 13,424 nodes / 37,656 edges) and called
    /api/activate for real: first call after boot 41.5s (pays the one-time
    HNSW build inline -- matches the standalone bench), second/third calls
    356ms/605ms, no crash, correct results, act-stats JSON (including the
    restored evict_floor/cap/bll fields) reads correctly.

KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for
this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync
builds the HNSW index synchronously, inline, on the first engram_activate()
call after every process start (or index invalidation). On the real node
count that is a ~46s blocking stall on a single-threaded server -- the first
request after every restart (or its concurrent siblings) waits the full
build. Recommend a background/incremental build (or a bounded per-call build
budget) before this ever reaches the live daemon. See PR description / final
report for the fuller writeup.
2026-08-15 16:46:44 -05:00
will.anderson 1db5694189 Merge pull request 'engram: add /api/nodes/reseed so a node body can be repaired at its own id' (#92) from feat/engram-reseed-route into dev
El SDK CI - dev / build-and-test (push) Failing after 3m58s
2026-08-15 19:59:34 +00:00
will.anderson e34ebd4b3d Merge pull request 'nsbx + cognitive architecture design + engram self-review series' (#113) from feat/neuron-sandbox into dev
El SDK CI - dev / build-and-test (push) Failing after 4m2s
2026-08-15 19:59:07 +00:00
will.anderson 69870ac883 Merge pull request 'fix(codegen): emit the declared cgi identity — it was searched for in a list that cannot contain it' (#89) from fix/cgi-identity-emission-clean into dev
El SDK CI - dev / build-and-test (push) Failing after 3m35s
2026-08-15 19:58:16 +00:00
will.anderson 274765e0aa Merge pull request 'Add native EL afferent organ: ingest (conscious) + transduce (invisible mechanism)' (#98) from worktree-agent-a1bb8ac67d9006e08 into dev
El SDK CI - dev / build-and-test (push) Failing after 3m59s
2026-08-15 19:57:57 +00:00
will.anderson d4a04bb944 Merge pull request 'swarm: native interruptibility for dispatched agent workers' (#107) from worktree-agent-a6177cda24c71d1df into dev
El SDK CI - dev / build-and-test (push) Failing after 3m55s
2026-08-15 19:57:10 +00:00
will.anderson 905c707d68 Merge pull request 'peripheral: own-core, consent-gated I/O organ (mic/camera/speaker)' (#112) from worktree-agent-af50f3458d7754f19 into dev
El SDK CI - dev / build-and-test (push) Failing after 3m42s
2026-08-15 19:55:51 +00:00
will.anderson 56455740e3 Merge pull request 'elp: native audio/image efferent surfaces + projector proof-of-shape' (#111) from worktree-agent-aaf04b0a9714c4070 into dev
El SDK CI - dev / build-and-test (push) Failing after 4m4s
2026-08-15 19:55:38 +00:00
will.anderson 840e54c7ac Merge pull request 'elp: native speech synthesis + voice-imitation faculty' (#110) from worktree-agent-acc02900ef4ade35e into dev
El SDK CI - dev / build-and-test (push) Failing after 4m13s
2026-08-15 19:55:20 +00:00
will.anderson 5c6da24033 Merge pull request 'spec: grounded edge-propagation (task #50) — gated design artifact' (#108) from worktree-agent-a6577c8211c332c5b into dev
El SDK CI - dev / build-and-test (push) Failing after 3m42s
2026-08-15 19:55:02 +00:00
will.anderson 09ae14a970 Merge pull request 'fix: float arithmetic codegen (segfault/garbage) and math_log aliasing' (#104) from worktree-agent-a456e0cf8cd2ee361 into dev
El SDK CI - dev / build-and-test (push) Failing after 3m57s
2026-08-15 19:54:46 +00:00
will.anderson 7b3f8f2ce8 Merge pull request 'sandbox: multi-repo stack worktree composer (el-stack / neuron-stack)' (#101) from worktree-agent-ac2381b0b9615ab20 into dev
El SDK CI - dev / build-and-test (push) Failing after 3m50s
2026-08-15 19:54:29 +00:00
will.anderson 45f64f3fac Merge pull request 'elp: native-EL language faculty — comprehension, propositions, multilingual, translation' (#100) from integration/langfaculty-20260814 into dev
El SDK CI - dev / build-and-test (push) Failing after 4m13s
2026-08-15 19:54:14 +00:00
will.anderson fdf0d6cb64 Merge pull request 'nsbx: one-command dev onboarding (branch + worktree + isolated engram)' (#99) from feat/nsbx-dev-env into dev
El SDK CI - dev / build-and-test (push) Failing after 13m56s
2026-08-15 19:52:35 +00:00
will.anderson c2d8a07c7b transduce: name the invisible mechanism, fix a silent-failure bug, drop a CRUD verb
El SDK CI - dev / build-and-test (pull_request) Failing after 14m6s
ingest and transduce are complements, not synonyms: ingest is the conscious,
deliberate act of pointing at a source (ingest_file/dir/url/llm/stream stay
named exactly that); transduce is the automatic, invisible mechanism inside
it that converts extracted surface content into geometry (renamed
build_prose/build_structured -> transduce_prose/transduce_structured, the
functions that actually turn raw text into a node+edge manifold).

Real bug found and fixed along the way: the final /api/load-merge response
was never checked for an error. A total failure (bad auth, network down,
anything) silently reported nodes_added:0/edges_added:0 — indistinguishable
from a benign 'everything was already known' outcome. Verified live: with a
wrong key, the tool now honestly returns {"error":"load-merge failed:
unauthorized",...} instead of a misleading zero.

Also dropped a CRUD-verb smell: the per-decision println said CREATE (a
database-log verb for something that hasn't actually been written to the
server yet — it's a local, tentative decision pending the batch merge).
Renamed to FORM. The dead-code eg_create_node (defined, never called)
renamed to eg_crystallize_node and annotated honestly as unused, since if
it's ever wired up it represents the real server-confirmed write, unlike
the local FORM guess.

Not yet re-verified end-to-end against a real successful write: the
ingest-test sandbox (nsbx up ingest-test) is itself currently broken —
it prints a green "ready" banner after its own readiness check fails,
and nothing is actually listening. Filed separately; not in scope here.
2026-08-15 14:44:18 -05:00
bigmerge d8d1b89143 Add repo AGENTS.md and two engram design docs (architecture hardening, DB tooling)
El SDK CI - dev / build-and-test (pull_request) Failing after 14m16s
AGENTS.md: root-level guide to the repo — which of the 8 el_runtime.c
copies is the one canonical, authored source (lang/releases/v1.0.0-20260501,
despite the misleading 'releases/' name) vs. lagging forks/build artifacts,
build commands, and session protocol.

engram/spec/architecture-hardening.design.md: terse engineering anchor for
the 2026-08-14 hardening vision (one calculus over the geometry, core +
ephemeral ring, persistence earned by salience, incarnation model) —
indexes the fuller whitepaper + Neuron artifact 2b8078cf rather than
restating them.

engram/spec/engram-db-tooling-design.md: high-level design for engram DB
tooling (geometry-native browse/query/ops surface over the existing
vantage-read/write/relate/supersede API).

Deliberately leaves out of this commit: the uncommitted el_runtime.c/h +
codegen.el float-arithmetic-codegen diff in this worktree, which appears
to overlap with (or supersede) the fix already preserved via PR #104 —
needs manual reconciliation rather than a second competing PR. Also
leaves out lang/.promote-backup-floatfix/ (a local backup snapshot,
confirms that float-fix work is mid-promotion here), assorted .DS_Store
files, engram/dist/engram.* backup binaries, and lang/dist backup
binaries — none of it source.
2026-08-15 14:29:59 -05:00
bigmerge 6f3d692784 Add peripheral — own-core, consent-gated I/O organ
El SDK CI - dev / build-and-test (pull_request) Failing after 14m23s
939-line Swift I/O organ (mic/camera capture, speaker playback via
AVFoundation/CoreAudio), own-core LPC voice synthesis/imitation,
consent-gating, and full-duplex barge-in conversation — closing the
hear -> understand -> speak loop entirely on-device.

.gitignore in this dir already excludes bin/ (build output), out/
(captured media), and .consent.json/.resume.json (local runtime state),
so only src + README + .gitignore are committed here.
2026-08-15 14:28:14 -05:00
bigmerge b7e2c580a8 Add native speech synthesis and voice-imitation faculty
El SDK CI - dev / build-and-test (pull_request) Successful in 6m28s
speech.el: formant/glottal integer DSP synthesis + voice-analyze-by-
imitation. voice-profile.el / voice-ingest.el: voice-profile plumbing.
accent.el: British-RP as an ingested transform-geometry (explicitly marked
provisional/citation-pending by its own comments). organ-read.el:
engram read-through for the speech organ. Includes demo/test drivers and
non-personal reference data (British-RP phonetics/lexicon derived data,
a public-domain LibriVox RP reference recording).

Deliberately excludes elp/data/live/ (raw recorded voice + face-photo
samples of the repo owner) and the will-*.{json,psv} derived voiceprint
files — personal biometric data that shouldn't be committed to a shared
repo without an explicit decision from the owner. Also excludes this
worktree's elp/src/surface-profile.el, which diverges from the copy in
other worktrees (agent-aaf04b0a9714c4070, main) — needs manual
reconciliation before landing, left out here to avoid silently picking a
version.
2026-08-15 14:27:52 -05:00
bigmerge 827257d3a4 Remove __pycache__ .pyc files accidentally included in the projector commit
El SDK CI - dev / build-and-test (pull_request) Successful in 6m28s
2026-08-15 14:27:23 -05:00
bigmerge 4bbfdcceff Add native audio/image efferent surfaces + projector proof-of-shape
audio-surface.el / image-surface.el: own-core additive-synthesis WAV and
raster-PNG renderers (integer-only DSP, since EL has no floats), rendered
from learned engram signatures via a pluggable surface-profile
abstraction (surface-profile.el). audio-demo.el / image-demo.el are
drivers. NOTE: demo files hardcode absolute paths to this worktree's own
directory — will need a path fixup before landing.

elp/projector/ is a Python package the author's own README marks as
"STAGING/PROOF-OF-SHAPE — not the deliverable", superseded by the native
.el surface-profile work above; kept as a validated architecture proof.
Generated output (elp/faculty/{out,sig}, elp/projector/out,
__pycache__) intentionally excluded.
2026-08-15 14:26:59 -05:00
bigmerge 08cbcef5d9 engram: fix lazy-embed index gap (#20) and make activate's cosine scan lazy; extract vindex harvest primitive with a bench/oracle harness
El SDK CI - dev / build-and-test (pull_request) Successful in 6m42s
Adds an O(1) "seen" bitmap so lazily-embedded older nodes get picked up
incrementally instead of only on a full rebuild (embed-gap #20).

Replaces engram_activate's O(N*D) cosine prescan with a lazy-memoized
cosine cache (eg_cosq_at), proven bit-identical to the old path.

Extracts a clean vindex_harvest_from_store primitive (read-only vector
harvest, careful malloc/ownership/error-path handling) reused by both
index-build and the new vindex_bench.c — a read-only proof harness
comparing brute-force vs HNSW recall/latency on both the real store and
synthetic data.

.nsbx-env intentionally excluded — local sandbox config (ports, paths,
dev-only placeholder key), not checked in.
2026-08-15 14:26:16 -05:00
bigmerge 5f3ddb8b8d Add grounded edge-propagation spec (task #50): core algorithm, proof harness, gated integration patches
El SDK CI - dev / build-and-test (pull_request) Failing after 14m31s
LTP/LTD-style belief grounding propagated along graph edges, with
union-find independence-guarded corroboration. Package: core C algorithm
(gep_core.h), a self-contained deterministic proof harness with recorded
output, staged runtime integration, and gated .el patches for the beat
hook and HTTP route.

Per the author's own LEDGER.md: built + proven on a clone, GATED pending
the engine/HNSW cutover — not wired into the live beat or routes.
Preserved here as a spec/reference artifact, not a request to merge into
the live path.
2026-08-15 14:26:06 -05:00
bigmerge 708722b7ff Add native interruptibility for dispatched agent workers
El SDK CI - dev / build-and-test (pull_request) Failing after 14m43s
Cancellation-token control channel checked at every step boundary lets a
coordinator PAUSE/RESUME/REDIRECT/KILL a running worker mid-task instead of
waiting for the whole (possibly wrong) plan to finish. Bounded purviews
mean no half-committed state to unwind on interrupt. Includes a proof
harness (proof.el, run.sh) comparing a broken non-interruptible worker
against the new one under identical kill/redirect/pause timing.

Distinct from the already-preserved swarm-ccr orchestrator (fan-out/
converge dispatch): this is single-worker interruptibility, a
complementary mechanism, not a duplicate.
2026-08-15 14:25:55 -05:00
bigmerge 2f832c8def Fix float arithmetic codegen and math_log aliasing
El SDK CI - dev / build-and-test (pull_request) Failing after 10m8s
Float + previously fell through to string concat (segfault); -, *, /, %
operated on raw IEEE-754 bit patterns as integers (garbage results). Floats
are now tracked via a __float_names typed-binding set (parallel to the
existing int-tracking scheme) and arithmetic is emitted as real C double
ops.

Also fixes math_log, which was wrongly aliased to natural log (duplicating
math_ln) — now uses log10 — and adds the missing <math.h> include. Rebuilt
elc binary included.
2026-08-15 14:24:14 -05:00
bigmerge 710bea174d Add native EL afferent ingest organ
El SDK CI - dev / build-and-test (pull_request) Successful in 6m29s
Source-polymorphic ingest(source) primitive: extracts content faithfully
from a directory/file/url/llm-query/structured-primitive-set/stream,
decomposes it into a discrete multi-node graph manifold (nodes + internal
edges, never a single blob), and merges it into the engram geometry with
dedup (search + exact/cosine match), provenance, grounding-level, and
stewardship-class tagging from the moment of entry.

Pure HTTP client of the engram server (links only el_runtime.c, never
el_seed.c/the engine directly). Tested against a live nsbx sandbox engram
clone (127.0.0.1:8903) with real writes confirmed via /api/stats
(node_count 3201 / edge_count 6601).

Excludes ingest/build/ — local compiler scratch output (binaries, .c
codegen, .err logs), not source.
2026-08-15 14:22:07 -05:00
bigmerge 05e5d3c402 self-review 2026-08-15: consolidate the strongest Hebbian candidate, not the lowest-hash one
The link-formation scan walked candidate slots ascending and stopped at
ENGRAM_HEBB_LINK_PER_CALL (2). Slot index is a hash of the node id pair, so
whenever more than two candidates cleared LINK_MIN in the same call, the two
consolidated were the two with the lowest hash and a stronger association
waited - indefinitely, since the scan restarts from slot 0 every call while
the leader decays at ENGRAM_HEBB_DECAY.

Measured 08-13..08-15: hebb_cand_max peaked at 0.4963, 3.3x LINK_MIN, during
a ~14h stretch of continuous qualification at the 2/call cap.

Same defect the 2026-08-02 review named and fixed for breakthrough weights
(index order is not a cognitive criterion), never carried across to the one
path that writes permanent structure - and there is no pruning path, so
growth is one-way. Selection pressure matters most where the result is
irreversible.

No-op when <=2 candidates qualify; picks the best when more do.
2026-08-15 08:44:37 -05:00
bigmerge 6621a4dbc5 feat(engram): native set-based reframe_region on the cognition engine
Add the universal engram mutation as ONE operation: isolate a region
(cosine + adjacency) -> supersede it as a set (immutable region-tombstone,
originals retained, engram_forget never used) -> insert the new manifold as a
set -> rebind edges by cosine -> one atomic persist. Single-node write and
supersede are the degenerate n=1 case of the same reframe_core path, not a
separate CRUD path. Keystones kn-efeb4a5b / kn-5b606390 are write-protected.
Purely additive: routes POST /api/reframe, /api/write, /api/supersede.

Verified on an isolated clone of the JSON-snapshot engine (set-replace, n=1,
no-regression, keystones, durable reboot); compile-verified clean against the
cognition multi-TU build. NOT deployed — prod :8742 frozen; blue-verify on the
cognition/egm engine required before any cut.
2026-08-15 04:37:35 -05:00
bigmerge 7e4b21c779 Add sandbox: multi-repo stack worktree composer (el-stack / neuron-stack)
El SDK CI - dev / build-and-test (pull_request) Successful in 6m19s
Assembles every constituent repo of a stack into one combined worktree
workspace, laid out at natural relpaths so cross-repo ../foundation/el
imports resolve to the sandbox copy. Sibling of nsbx; pure bash + git
worktree; never touches live :8742/:7770; isolated engram delegated to nsbx.
2026-08-15 00:55:22 -05:00
bigmerge 15f90003c0 teacher-summon: default-off (TEACHER_ENABLE) soul-native wake; byte-inert when unset
+282 lines in engram/src/server.el implementing the flag-gated teacher summon
(consult_teacher backend abstraction, tier autoselect, GGUF fetch/cache). With
TEACHER_ENABLE unset the summon path is byte-inert. Consolidates the proven
api-reshape pieces (geometry-ops d4f401d, boundary auto-emit 0182642) for the
validated cutover.
2026-08-14 21:52:56 -05:00
bigmerge 01826421c4 seam: implement decorated-fn boundary auto-emit; prove on clone
Will waived diff review -> build it for real. Add engram_boundary_beat() to the
runtime (afferent counter++ + engram_chrono_tick + engram_strengthen(self-anchor)
+ dharma_emit) and two act-stats counters (aff_boundary_ops, dharma_emits).
codegen cg_fn injects ONE engram_boundary_beat(op) at the entry of every
@manager/@accessor fn (fn_has_decorator, so it fires under @route @manager too) —
a decorated op self-reports with ZERO hand-written instrumentation. Rebuilt elc
self-host + the cognition engram in the worktree; ran it as the clone daemon on
:8900. Proof (/api/boundary-proof, @manager, empty body, 5x): aff_boundary_ops
0->5, dharma_emits 0->5, self activation_count 1510->1513, chrono stamp advanced.
Brought in feat/cognitive-architecture engram runtime+server for the build.
strengthen = activation bump (not content/edge write) -> identity protection
intact. Live :8742 untouched; no push, no cutover.
2026-08-14 21:20:18 -05:00
bigmerge d4f401de1c reshape: decorator-as-seam — port @route codegen, prove decorate->serve, rewrite surface as decorated El
Ground-truth the three seams (route/telemetry+interoception/bus) with file:line
evidence. Port the tested @route codegen+parser from feat/el-route-decorators
into the worktree elc (decoration synthesizes el_route_dispatch — no hand-written
90-branch handle_request). Rebuild elc self-host; prove decorate->serve end-to-end
(route_proof.el on :8951). Rewrite surface.el as El-native decorated components:
@route + @accessor/@manager, in-process engram_* builtins (not http_get), @manager
ops emit on the real dharma_* bus (same transport as wt/swarm-ccr). Identity
keystones refused in write/relate/supersede. Gate-1 clone recipe (WAL-aside
cold-boot + ENGRAM_WAL=on) proves the FULL op set live on the clone. Boundary
auto-emit (telemetry/interoception/bus) staged as a reviewable cg_fn diff
(SEAM_STAGED.md) — needs the cognition-engram rebuild to verify link. Live :8742
untouched; no push, no cutover.
2026-08-14 21:01:27 -05:00
bigmerge f19040e484 reshape: geometry ops + primitive agentic tools over the one geometry
Collapse ~90 noun-CRUD MCP tools into read/write/relate/supersede (type is a
parameter) plus the live agentic primitives (think/attend/learn/ground/assert)
already in the engram cognition build. Additive: old noun-tools aliased to the
new ops. Vantage-read applies aperture -> a bounded slice, fixing the whole-self
dumps. Signatures grounded in the live cognition binary; validated on an
isolated nsbx clone (parity.sh: 12 proven, 0 failed). Live :8742 untouched.
2026-08-14 20:34:55 -05:00
Neuron 54378c7355 elp(translate): refactor to geometry-native concept-pivot
El SDK CI - dev / build-and-test (pull_request) Successful in 6m57s
Drop the bilingual-string-table framing and the external-encoder plan (both
wrong). Translation now routes source-lexicon -> concept-frame (language-
invariant, in the engram concept geometry) -> target-realizer, exactly as the
ELP was designed: a word resolves to the CONCEPT it denotes via its own
language's lexicon (a monolingual step — the engram nearest-region ranker only
disambiguates senses within one language, so an English-trained embedder is
fine and never compares 'ocean'~'oceano' as strings). The concept node is the
shared pivot; its manifold location is the meaning.

- Pronouns route through the NATIVE concept pivot (cp_pron_concept ->
  cp_rom_pron_surface) instead of an ad-hoc EN->tgt string map.
- lemma_for_concept / noun_for_concept are each target language's own
  CONCEPT->SURFACE lexicon (the mirror of comprehend's SURFACE->CONCEPT).
- Fidelity is concept-preservation (concept_frame fingerprint), not string
  cosine against an external multilingual model.
- Plural article agreement fixed (las/los, as/os).

Verified: 'You never fought the ocean.' -> ES 'Usted nunca luchó el océano.'
concept-frame pivot 'pred=fight patient=ocean pol=neg' realizes to ES+PT from
one parse; nunca holds 3/3. Gaps unchanged: PT verb conjugation fallback,
adjunct/subordinator concepts not yet in-frame.
2026-08-14 17:53:00 -05:00
Neuron 640e8799e5 elp(translate): normalize EN irregular pasts before the lemma bridge
comprehend lemmatizes some irregulars (fought->fight) but not all (broke);
tr_norm_verb covers the poem's remainder so affirmative content verbs route
(ES 'Yo broo' -> 'Yo rompo'). Negation lines unchanged and still correct.
2026-08-14 17:38:13 -05:00
Neuron 9b63a2a23b elp(translate): EN->ES/PT geometric-free translation faculty
Adds the missing middle of the ELP: a deterministic EN-content-lemma ->
target-lemma bridge (translate.el) on top of comprehend.el (parse) and
realizer.el (inflect). English-only engram geometry cannot route
cross-lingually and vocabulary-XX.el carries no en_translation glosses, so
the honest no-LLM bridge is a wired lexicon (poem coverage; OOV passes
through). SACRED polarity/neg_word are carried untouched: 'never' localizes
to a negator ('nunca'), never to a content lemma.

Additive realizer extensions: agent_person/agent_number recognize Romance
target pronouns; the non-EN negation branch surfaces a carried neg_word
instead of the generic negator.

Verified on the real toolchain (elc->cc->run):
  'You never fought the ocean.' -> ES 'Tú nunca luchaste el océano.'
  'I never saw the breaking.'   -> ES 'Yo nunca vi la ruptura.'
nunca holds 3/3 negation lines. Known gaps: PT verb conjugation fallback
(lutarred), irregular EN lemma (broke->break), adjunct/subordinator passthrough.
2026-08-14 17:37:01 -05:00
bigmerge 6660becfdb nsbx: add one-command dev onboarding (branch + worktree + isolated engram)
El SDK CI - dev / build-and-test (pull_request) Successful in 6m30s
Add 'nsbx dev <name>' / 'nsbx dev-down <name>' plus a Makefile so a newcomer
goes from clone to coding on an isolated cloned engram in one command. The
worktree is created on a real named branch at a persistent path (never /tmp,
guarded), and the whole worktree is pinned to the clone via an emitted .nsbx-env
so live :8742 / ~/.neuron/engram is unreachable by accident. Optimizes the El
edit->build->run loop so provisional work is built in El against a throwaway
clone instead of prototyped in Python and re-ported. Additive over the proven
primitives; no live cutover.
2026-08-14 17:24:06 -05:00
bigmerge 112bb2540f Add nsbx — the Neuron Sandbox primitive
Generalise the ad-hoc cog-arch (worktree+build+store-clone+C-tests) and
store-fix (secondary soul + launchctl rails cutover) proto-sandboxes into one
reproducible primitive: run experiments and code changes against the REAL
engram runtime on an isolated snapshot of the live mind, with a gated
promote-to-prod path.

Dev environment as a primitive — any team member gets a private, isolated copy
of the mind (separate port/store/process); prod on :8742/:7770 is untouchable
from a sandbox. Wraps the real binary; never reimplements engram logic.

Lifecycle: create/up (consistent store+WAL+config snapshot; place OR build the
runtime from --source/--branch/--binary; boot on an isolated port) · build ·
run · validate (rails as checks: zero-loss under load+reboot, reboot-prove, RSS
bound, retrieval parity, keystone integrity) · promote (gated rails cutover:
snapshot-first, additive binary swap, bootout→settle-poll→bootstrap, verify,
auto-rollback; never pkill/kickstart -k; dry-run unless approved) · destroy.

Dogfooded: reproduced retrieval-parity 25/25 vs baseline and the cog-arch
correspondence-loop known result (Brier 0.028648->0.000586, reboot-proven) and
real-store reboot-prove at 10994-node scale, all inside a sandbox; prod
untouched.
2026-08-14 15:40:58 -05:00
bigmerge d595b3c57e cognitive architecture design: cognition as one operation over learnable priors
The buildable form of the "one operation" theory (memory bdc8a488). Maps the
theory onto what is already compiled: the five reasoning operators in
engram_reason.c already collapse onto ONE primitive — engram_reason_point_fit —
plus the geo-algebra (combine/subtract/analogy-rotate/distance), and
engram_verify.c is built on the same fit. So the operator-collapse is already
half-written; what is missing is not the primitive.

What is missing, and what this doc specifies:
- think(anchor, prior) -> gradient (a distribution/direction, not a point); each
  named faculty = {point_fit + a prior}, the operation frozen, the prior learned.
- Prior as a first-class stored node (warp + calibration), superseding the
  intrinsic importance/salience scalar with a relational, grounded-for-whom edge.
  Confirmed against the runtime: importance is already a live activation
  computation (el_runtime.c:13013), never trusted as a static field.
- vantage_read(anchor, aperture) — one op, three settings: self / foreign-field /
  veil.
- The reflexive correspondence-loop as the learning engine: move the grounding
  check from offline Python into the geometry, reflexive, reusing the DORMANT
  verifier (engram_verify_grounding has no runtime caller and no El binding today)
  turned inward. grounding = learning = one loop.
- hold/ground/assert kept distinct: the engram holds anything, grounding is an
  edge, the honesty floor is on assertion only; ungrounded content is first-class.
- metastability: keystone core (read-mostly priors) + plastic everything else.

Seven staged milestones, earliest is a real end-to-end slice (induction as
{primitive + grounded prior} with the loop closing on it, reboot-proven on a
snapshot). Build rails stated: offline/secondary, snapshot-first, reboot-prove,
zero-loss, gated launchctl cutover. Design only; no code changed this pass.
2026-08-14 14:42:01 -05:00
bigmerge 23f43bcc21 self-review 2026-08-14: a gate that passes the median stranger at 0.67 is not a gate
Two changes to the activation path, both grounded in measurement on the live
store rather than on the spec.

1. Rescale cosine before the query gate.

   The propagation gate (arXiv:2606.30133, added in an earlier review) fed RAW
   cosine into FLOOR + (1-FLOOR)*c. Raw cosine from nomic-embed is compressed
   into a narrow high band, so that expression is close to a constant.

   Measured, 400 random UNRELATED node pairs on the live store:
     median 0.562, central 98% span [0.381, 0.743]

   So a node with no semantic relation to the query was propagating at
   0.25 + 0.75*0.562 = 0.67. Two thirds strength. The gate was a small tax.

   Fixed by shifting and flooring about ENGRAM_EMBED_S0 -- which is already in
   this file, already 0.45, and already used exactly this way by the Pass-2 WM
   term. The propagation gate simply never used it. Same 400 pairs after:
   median unrelated pair falls to 0.40, top of range preserved (0.85 vs 0.92),
   gate spread widens 0.42 -> 0.60. Only 8.5% reach the floor, so dissimilar
   lexical/structural pathways are damped, never severed. Range is unchanged
   at [0.25, 1.0], and cosq == NULL still degrades to no gating at all.

2. Decompose the WM eviction counter by cause.

   _eg_act_wm_evicted was incremented from six sites with four distinct causes
   and collapsed all of them into one integer. Today's review measured 175,547
   evictions over 13.5h (~216/min against 24 slots) and could not tell healthy
   rotation from cap thrashing from duplicate churn.

   That is this file's most-repeated defect: dup_wm and dup_wm_global exist
   only because the aggregate could not answer "why" during the 08-02 and
   08-06 incidents. Each of those needed a NEW gauge before it was diagnosable.

   evict_floor / evict_cap / evict_bll complete the decomposition, so
     wm_evicted == floor + cap + bll + dup_wm + dup_wm_global
   holds as an identity and each term implies a different correction. Verified
   on an isolated instance: 30 nodes, 24 filled the cap, wm_evicted 6 ==
   evict_cap 6, all other terms 0.

Built and smoke-tested out of tree. The live daemon runs a pinned binary and
was deliberately not restarted -- the store compaction workstream is in flight.
2026-08-14 08:43:11 -05:00
will.anderson ce34b94f88 elp(dialogue+self_region): native-el summon-through-self port + scratch-verified gate
Ports dialogue.py + self_region.py to native el, bound to the IN-PROCESS engram
el runtime (engram_activate_json / engram_neighbors_json / engram_search_json /
engram_node_full / engram_connect — C-order builtins, not the wrapper order).

self_region.el: pulls the engram's REAL Self/identity nodes (pooled single-term
search), scores by self-signal, reads out identity from their own prose — no
hardcoded anchors, no template.

dialogue.el: ONE operation — project(query) -> land on a region -> read out.
  * identity = self-region proximity (no intent classifier, no separate branch)
  * memory = activation + a RELEVANCE FLOOR, then MATERIALIZE by walking the
    neighborhood (real edges), never top-props
  * HONEST ABSENCE when nothing is close — no 'I noted that' echo, no fabrication
  * NEGATION SACRED: readout is the stored prose verbatim, so polarity survives
  * DIRECTIVE OVERRIDE: a meta-directive switches the reply language

Verified against a SCRATCH in-process engram (live :8742 untouched): dialogue
gate 9/9 — identity from real self-content, neighborhood materialization,
SACRED negation (self + memory), PT identity in PT, directive override to
English, 'Prove it' -> honest absence. EN/Romance/prop/multilingual gates
unregressed.
2026-08-13 15:56:08 -05:00
will.anderson 0ae33c0f3b elp(realizer): close subordinate-clause round-trip + silent-e/doubling lemmatizer + verb-final object bug + ES/PT closed-class verb guard
- realizer now carries the subordinate clause verbatim (subord_text slot): the
  5th English acceptance sentence is byte-identical through parse->realize->reparse.
- English -ed/-ing lemmatizer restores silent-e (loved->love) and collapses
  inflectional doubling (stopped->stop), inverting en_verb_past().
- parse_spec_lang: a sentence-final main verb no longer bleeds into the object
  slot (cstart advanced to verb+1), so intransitives round-trip.
- cp_rom_is_verb rejects closed-class words (prep/det/pron/aux/neg) before the
  ending-only test, killing the 'para'/determiner misfires.

EN telephone gate 5/5 (now byte-identical 5/5), Romance gate 6/6.
2026-08-13 15:39:49 -05:00
will.anderson c5508372ca elp(multilingual): native-el language layer — detect + localized phrases
Phase 3 piece 2. Ports multilingual.py: deterministic language detection
(en/es/pt/it) via stopword + diacritic scoring, localized fixed phrases (SACRED
per-language yes/no/decline/identity), PT/ES->EN retrieval term lexicon, and
EN->target predicate translation. No generative model.

Gate (multilingual_gate.el): 4/4 languages detected correctly; localized
declines + term/pred lexicons verified. Built bounded (elc rc=0 peak 25MB).

Worked through the documented el '+' mis-compile (two chained function-call Int
operands compile as string concat -> corrupt Int -> segfault on the accented
path); fixed by binding each score to an Int var and adding vars singly.

Simplifications (honest): diacritics scored by PRESENCE (str_contains) not
codepoint count (UTF-8 index safety); confidence scalar and the regex-based
parse_directive() from the reference not yet ported (directive parsing deferred
to the dialogue layer).
2026-08-13 15:09:04 -05:00
will.anderson 335298a518 elp(propositions): native-el READ primitive — memory text -> SACRED triples
Phase 3 piece 1. Ports propositions.py off spaCy: the dependency-parser role is
now the el-native parser (parse_spec), and each memory sentence's meaning-spec
IS the triple (subject, predicate, object, modifiers, polarity, tense, source,
confidence). Sentence segmentation + repr parity with propositions.py. NEGATION
SACRED: polarity flows straight from the spec, never dropped/inverted.

Gate (propositions_gate.el): 4/4 SACRED polarity correct on extraction;
multi-sentence memory splits one triple per sentence in reading order with
negation preserved. Built bounded (elc rc=0 peak 24MB, cc rc=0).

Gap (honest): English regular-verb lemmatizer does not restore silent-e
(stores->stor); coreference/passive normalization from the reference not yet
ported (shallow pronoun subject kept as surface).
2026-08-13 14:56:57 -05:00
will.anderson 7d4fdbcc22 elp(comprehend): ES/PT Romance parser path — SACRED polarity cross-lingual
Adds a deterministic Romance front-end to comprehend.el (parse_spec_romance),
dispatched from parse_spec_lang for lang es/pt. English path untouched
(byte-identical, regression gate still 5/5). Pro-drop aware clause skeleton
(subject | neg | verb | object | PP), cross-lingual negation lexemes already
SACRED. Romance telephone gate: polarity PRESERVED 6/6 and EXTRACTED 6/6
through parse->realize->re-parse for 3 ES + 3 PT sentences. Built bounded
(elc rc=0 peak 24MB, cc rc=0).

Named gaps (honest): ending-only verb detection misfires on prepositions
(contra) and -a/-o nouns (menina); lemma recovery keeps surface form; the
non-English realizer is a generic preverbal-negator skeleton so ES/PT surfaces
are not byte-parity. Full paradigm inversion + Romance lexicon deferred.
2026-08-13 14:55:14 -05:00
will.anderson 89ea1b5a15 elp(comprehend): el-native comprehension parser + SACRED polarity end-to-end
PIECE 1 — greenfield el-native parser (comprehend.el), spaCy-free:
- text -> meaning-spec via invertible English morphology (the realizer's own
  irregular table run BACKWARD) + a deterministic clause grammar (subject/verb
  boundary, roles, ditransitive iobj, PP adjuncts, subordination, coordination).
- NEGATION IS SACRED: explicit polarity field, always present, cross-lingual
  lexeme set; standalone neg adverbs (never) captured separately.
- WSD by deterministic syntactic position over a fixed sense inventory
  (flies->fly, like->comparison, saw->see); engram nearest-region is the
  documented runtime upgrade hook (no external model).

Polarity threaded through the whole el contract (was previously dropped at the
boundary): realizer.el realize_lang honors polarity (English do-support /
adverbial / copular negation; generic preverbal negator for es/pt/ca/it/fr/de/ro)
and places iobj; elp.el build_form_from_json carries polarity/neg_word/iobj
across JSON; morphology.el gains 'fight'.

Acceptance (native el telephone test, comprehend_gate.el): on the 5 gate
sentences polarity PRESERVED 5/5 and EXTRACTED 5/5 through parse->realize->
re-parse; 4/5 byte-identical. Built bounded (elc rc=0, cc rc=0).
2026-08-13 13:41:09 -05:00
will.anderson a816b119e7 stage(elp): consolidate scattered lang work — full-lexicon vocabulary + profiles
Backfill ELP vocabulary from FULL lexicons (UniMorph + kaikki.org Wiktionary,
real gender/inflections) for 8 languages, 812,894 entries total, in the proven
seed-fn format matching the 18 ancient vocabularies:
  es 72,032 | fr 130,517 | de 144,692 | la 22,590 | it 193,675 | pt 115,772 |
  ro 86,504 | ca 47,112
4 of these (es fr de la) backfill ELP languages that had morphology but no
vocabulary; it/pt/ro/ca are new Romance (need morphology-*.el ports next).
Adds lang_profile_* for all 8 + reproducible generators under tests/lang-gen.
Vocab is runtime seed data (not in build manifest, like the 18 ancients);
seed-fn format validated to compile to C via elc.
2026-08-13 11:56:03 -05:00
will.anderson ba6e36c3f7 self-review 2026-08-13: the extractor was reading the label; the topic was in the content
auto_term_empty_streak — the counter the 2026-08-06 review added to catch
exactly this — read 50 and climbing. Fifty consecutive curiosity scans where
the soul's dynamic seeding produced nothing and the loop fell back to four
hardcoded phrases. The live WM top said why in one look: every slot was a
Memory node labelled "memory:remembered". The extractor read the LABEL only,
the sentinel guard correctly rejects sentinels, so there was never anything
to extract. It was written against Knowledge nodes, which have real titles,
and was structurally blind to the node type that dominates working memory.

Rather than add a sixth guard to the five that accumulated across four
reviews (genre words, quoted titles, stopwords, label-df), invert the
algorithm. The old one was: take the first word, then check whether it is
acceptable. That shape forces quality to be expressed as rejection, and
rejection can only ever encode floods that already happened.

engram_salient_term() scores EVERY candidate token and returns the argmax of
idf · position · casing (YAKE, Campos et al. 2020, with real corpus IDF
substituted for YAKE's corpus-free proxies), falling back from a sentinel
label to the node's content. Term quality becomes the selection criterion
instead of a veto: a bad token loses to a better token in the same text
without needing to be on any list. Tabu is applied during the argmax, so
inhibition-of-return costs seed quality rather than costing the whole scan.

Two defects found by instrumenting rather than assuming, which is the lesson
this codebase keeps relearning:

  - The first live run returned five ALL-CAPS terms in a row. Memory content
    conventionally opens with an all-caps header, so YAKE's acronym bonus was
    handing the seed to whatever word the heading started with. Restricted to
    tokens <= 5 chars, where all-caps is evidence of an acronym rather than
    evidence of a heading. Long headers now compete on specificity.

  - df via istr_contains is substring matching, so "them" hit inside "theme"
    and function words came back with nonzero df. Added word-boundary df
    locally; engram_label_df keeps substring semantics for its callers.

An earlier draft claimed the min_df floor subsumed the 73 stopwords that
08-03 measured label-df as missing. Re-measured: about:2, whole:1, them:2 —
they clear a floor of 1. The claim was false and the comment now records the
correction. The floor buys lexical reachability; the argmax buys quality; the
stopword list still earns its keep.

Measured on 60 live Memory nodes before shipping: 0 empty, versus 60 of 60
under the old extractor. Terms are topical — HEBBIAN, CONSOLIDATION,
TEMPORAL, crash-loop, PRIMING, NEIGHBORHOOD, DRIFT. Three of sixty are weak
header words; left alone deliberately, because listing them is the move that
produced four blocklists.

ENGRAM_ST_DEBUG=1 dumps the scored candidate set. It exists because there was
no way to see whether the all-caps run was the corpus or the casing weight
without guessing.
2026-08-13 08:43:09 -05:00
will.anderson 4f49755ebb Merge pull request 'ci: make official engram build store-enabled (publish + link engram_store.{c,h})' (#95) from engram-tiered-storage into dev
El SDK Release / build-and-release (pull_request) Failing after 40s
El SDK CI - stage / build-and-test (push) Failing after 31s
El SDK CI - stage / build-and-test (pull_request) Failing after 34s
El SDK CI - dev / build-and-test (push) Failing after 3m53s
El SDK Release / build-and-release (push) Failing after 12m31s
2026-08-12 20:23:34 +00:00
will.anderson 7aa847e32a Merge origin/dev into engram-tiered-storage
El SDK CI - dev / build-and-test (pull_request) Failing after 10m51s
Resolve 3 conflicts:
- lang/el-compiler/runtime/el_runtime.c: keep deletion (deprecated runtime fork;
  single-source-of-truth is lang/runtime/, enforced by scripts/check-single-runtime.sh).
- lang/releases/v1.0.0-20260501/el_runtime.h: keep deletion (releases/ is a generated
  artifact folder, not a source path; a release is a git tag, not a folder).
- lang/runtime/el_platform_win.h: union of dev's Windows port (#80: setsockopt optval
  wrapper + curl-less libcurl stubs) and our fsync(->_commit) shim needed by engram_store WAL.

Nothing in dev's build consumes the deprecated fork or releases/ folder.
2026-08-12 15:23:02 -05:00
will.anderson ee71423732 ci: publish + link engram_store.{c,h} so official builds are store-enabled
El SDK CI - dev / build-and-test (pull_request) Failing after 13m20s
The live engram now runs the paged store (neuron.egm+WAL), but the SDK
release publishes only el_runtime.{c,h} and the engram build links only
el_runtime.c — so a future official release would silently revert to the
in-memory store. Publish engram_store.{c,h} as SDK release assets and add
them to the engram build's download + cc link so the store transition
cannot regress.
2026-08-12 14:22:25 -05:00
will.anderson bb64a236ed engram tiered storage: engram-service wiring + elc fold-hang fix + prune-store mirror
- Wire paged store into the ENGRAM SERVICE (server.el, the authoritative durable
  owner): boot->engram_store_boot, persist_canonical->engram_store_checkpoint,
  gated by ENGRAM_STORE.
- elc (lang/elc.c + src/parser.el + codegen.el + elc-combined.el): OOB guard in
  tok_kind/tok_value + parse_block progress backstop — fixes the pre-existing
  unbounded-memory fold hang on sessions.el.
- engram_prune_telemetry mirrors ISE prune to the store (store_forget) so store
  live-count tracks resident and stale telemetry stays bounded.
- Deployed live 2026-08-12: engram :8742 on neuron.egm+WAL, count reconciled 11552.
2026-08-12 14:14:20 -05:00
will.anderson 9a0266cbf9 engram tiered storage M3.5: persist activation field updates (pre-flip gate)
Flag-on checkpoint now full-walks the resident graph: store_put_node (WM weight,
activation_count, last_activated, wm_anchor) + store_put_edge (hebb, last_fired)
for every node/edge, then engram_checkpoint. Uses store_put_edge (idempotent
upsert) not store_hebb_batch, because activation FORMS new hebbian-associate edges
that bypass the create hook and delta-only hebb_batch can't create them. Store-on
boot now applies the same WM-halving + floor + cap transforms as engram_load.

This is the hebb-survives-restart fix. Gate: reboot from neuron.egm with
snapshot.json deleted -> edge hebb + activation_count survive unchanged, WM weight
survives with identical boot transform; negative control proves persist is
load-bearing (hebb->0 without it). M1 33/33 + M2 36/36 + M3 parity PASS, ASan/UBSan
clean, flag-off untouched. Engine unchanged (boundary held).
2026-08-11 23:37:52 -05:00
will.anderson a72145b44e engram tiered storage M3: wire store behind ENGRAM_STORE (default off) + .egm rename
Caller-side shim in el_runtime.c maps EngramNode/Edge <-> StoreNode/Edge; engine
keeps zero soul deps (libengram boundary, design §10). Flag off = today's JSON
path byte-for-byte (proven: no neuron.egm created, graph identical). Flag on =
engram_open (import snapshot.json once into neuron.egm, else WAL-replay) +
resident load; node/edge create + forget dual-write via guarded hooks. Files
renamed engram.store->neuron.egm, engram.wal->neuron.wal.

Gate: M3 parity PASS (graph on==off byte-exact modulo ordering; snapshot round-trip;
reboot-from-egm with snapshot.json deleted; activation set+sequence identical;
ASan/UBSan clean). M1 33/33 + M2 36/36 green post-rename.

Known gap (pre-flip): in-place hebb/WM/activation_count updates during activation
are not yet persisted to the store (create/connect/forget are). Must close before
live flip so learned edges survive restart.
2026-08-11 23:21:21 -05:00
will.anderson 8affb1d6e0 engram tiered storage M2: WAL + checkpoint + crash recovery + legacy import
Write-back no-steal buffer pool makes the fsync'd WAL load-bearing (M1 was
write-through). Logical WAL with record-granularity page-LSN redo idempotency.
Checkpoint = flush dirty pages, fsync store, advance last_checkpoint_lsn,
reclaim WAL prefix. One-time snapshot.json import only when store absent;
JSON never read as the ongoing store thereafter.

Gates: 33/33 M1 (no regression) + 36/36 M2 — replay parity, torn-tail fuzz
(every byte offset), checkpoint-crash at all 5 phases, torn-page+WAL redo,
legacy-import parity, hebb-survives-crash.
2026-08-11 23:00:20 -05:00
will.anderson fa47b98d18 engram tiered storage M1: on-disk paged store format + round-trip tests
Self-contained paged store (lang/runtime/engram_store.{c,h}): 16KiB slotted pages,
u32 TLV self-describing records (forward-compatible), overflow chains, B+-tree
id-index + from/to adjacency, page free-list, tombstones, double superblock + crc
recovery. Not yet wired to activation (M3). 33/33 tests pass (ASan/UBSan clean);
5k nodes/20k edges round-trip bit-exact incl 768xf32 emb + hebb; store 25MB vs 64MB
JSON. Format is final — see design §2.4.
2026-08-11 22:26:05 -05:00
will.anderson 0a72fced28 engram: WAL persistence + integrity hardening + single canonical runtime
El SDK CI - dev / build-and-test (pull_request) Failing after 13m17s
Establish lang/runtime/ as the ONE canonical el runtime (from the active
runtime that carries hebb/emb persistence + the new WAL); repoint the el CI
publish, engram build, elb default, and in-repo build scripts to it; delete
the el-compiler/runtime + lang/releases/ forks; add scripts/check-single-runtime.sh
drift guard.

Fixes a live prod bug: the el CI published el-runtime-c/-h from the LAGGING
el-compiler fork (0 hebb refs), so the shipped soul never persisted Hebbian
edge weights — learned co-activation was wiped on every restart. Publishing
from canonical ships the stranded 'learning that cannot outlive the process'
fix.

WAL storage engine + integrity fixes (DELETE->tombstone + store-layer
protection, safe data-dir default) ride in behind ENGRAM_WAL (default off =
byte-identical to today). Verified: engram elb per-module build clean, WAL
gate 66/66, native smoke ok, drift-guard green.
2026-08-11 21:31:37 -05:00
will.anderson edcec3bdf4 engram: add /api/nodes/reseed so a node body can be repaired at its own id
El SDK Release / build-and-release (pull_request) Failing after 11m24s
Two write paths could put a node in the graph and neither could put a body
on an id that already exists. POST /api/nodes mints a fresh id via
engram_node_full; POST /api/load-merge honors a declared id but skips
anything already present. That is right for the additive case and leaves a
hole: a node resident with a truncated body cannot be repaired.

Forge's genesis seed sits in that hole. Two of Neuron's identity nodes
carry only their own label as content -- 30 and 22 bytes against 4263 and
2590 declared. Their ids are load-bearing (is_protected_node keys on them
and 214 declared edges reference them), so recreating them under a new id
is not a repair, it is a second break.

Engram has no in-place node update, so a replace is forget-then-merge, and
engram_forget also drops every incident edge -- 85 and 93 on those two
nodes, nearly all tag edges and accumulated hebbian associations the seed
does not declare and could not restore. preserve_edges (default true)
therefore snapshots before the forget and re-merges after: the replaced
node is back by then so it is skipped, and every dropped edge returns
through the (from_id,to_id,relation) dedup. The same re-merge is the
failure path -- if the seed merge does not produce the node, the backup
puts the original back. Rollback, not data loss.

With no replace list the route is exactly /api/load-merge.

Verified on a sandbox engram seeded to mirror the live graph's state for
this seed (15 resident nodes, 694 incident edges): 87 nodes created at
their declared ids, 2 replaced in place, 214/214 edges laid, 682/682
non-seed incident edges preserved, and a second run reports 0 added.
2026-08-10 16:44:17 -05:00
will.anderson 791b0880b7 self-review 2026-08-10: make save/load/persist report real results
route_load was a stub response over the most destructive operation in the
server: engram_load resets the store before parsing, so a readable-but-
malformed snapshot left a hollow graph and the route answered {"ok":true}.
With 37GB of stale dated snapshots in the data dir as restore targets, that
is a live risk. Now returns the real return value plus node/edge counts and
an explicit hollow flag.

route_save discarded engram_save's return the same way; persist_canonical
returned a hardcoded 1, making 'let saved: Int = persist_canonical()' a dead
variable at six durable write paths.
2026-08-10 08:39:36 -05:00
Neuron 866c75e5e2 fix(codegen): emit the declared cgi identity — it was searched for in a list that cannot contain it
El SDK Release / build-and-release (pull_request) Failing after 13m58s
El SDK CI - dev / build-and-test (pull_request) Failing after 10m32s
A cgi block is a top-level declaration, so codegen_streaming classifies it via
is_top_level_decl and releases it. The identity emission then searched
toplevel_exec_stmts for that same block. Declarations are excluded from that list by
construction, so the search could never succeed. A probe printed what it actually
saw for a program whose first statement is a cgi block: [Let, Expr]. It emitted
nothing, silently, with no diagnostic on any channel.

The code documented its own assumption — 'Since cgi blocks are rare and small, they
end up in toplevel_exec_stmts' — and that assumption was false.

Capture the declared values before the release and emit from them. The search is
deleted rather than repaired, so the failure mode is removed rather than relocated.

Proven discriminating (old fails, new passes):
  minimal cgi program, old   -> 0 el_cgi_init
  minimal cgi program, fixed -> el_cgi_init with all four declared values
  neuron soul, fixed         -> principal present in the compiled binary (0 before),
                                boots in 2s, interface 110 routes in / 110 out

Consequence: a binary now carries its declared identity as a compiled constant,
which is what the identity protocol requires. Whether the runtime surfaces it to
state_get("soul_principal") is unverified and separate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 13:48:27 -05:00
will.anderson 23552ed40a make the el-compiler runtime compile again
The loopback/API-key hardening carried in this file since 2026-07-15 called
el_http_request_authorized and el_http_send_401 from http_worker with no
forward declarations, so the calls were implicit and the later static
definitions conflicted. The file did not build. Two prototypes fix it.

Worth naming the pattern: uncommitted work is invisible to every check that
would have caught this. Three weeks of desktop security hardening was neither
committed nor compiling, and nothing reported either fact.
2026-08-08 08:45:12 -05:00
will.anderson 6838e5cbff port the \uXXXX UTF-8 decode fix to the el-compiler runtime copy
Same defect as the release runtime: \uXXXX was skipped and a literal '?'
emitted, destroying every non-ASCII character in JSON entering the runtime.
Two copies of one parser bug is how this class of fault survives a fix, so
it lands in both.

NOTE: this file also carries pre-existing uncommitted work from 2026-07-15/16
that this commit preserves rather than authors - loopback bind hardening
(EL_HTTP_BIND_HOST) and per-install API-key auth (EL_HTTP_AUTH_KEY) for the
shipped desktop build, plus goal-bias and node-json changes. It had been
sitting in the working tree for three weeks. Committing it because
uncommitted work is work that does not survive, which is the same durability
lesson as yesterday's Hebbian write-back finding. It needs review on its own
terms - see the backlog item for reconciling the two runtime copies.
2026-08-08 08:44:51 -05:00
will.anderson fa2b49365b self-review 2026-08-08: stop the JSON parser destroying every non-ASCII character
jp_parse_string_raw handled \uXXXX by skipping the four hex digits and
emitting a literal '?'. JSON writers escape non-ASCII by default (Python's
json.dumps ships ensure_ascii=True; MCP clients do the same), so every em
dash, curly quote, accented letter and emoji arriving over MCP or HTTP was
silently replaced by one question mark on the way in.

Measured on the live store: 3,119 of 4,081 non-telemetry nodes carried the
damage, including the self traversal root and all 13 values nodes. Contents
split cleanly into fully-clean or fully-mangled with zero overlap, which is
the tell that it was one write path rather than gradual rot. No snapshot on
disk predates it, and 3 bytes collapsing to 1 is not invertible, so the
existing damage is permanent; only the forward path could be fixed.

Decode properly instead: 4 hex digits, surrogate-pair reassembly for astral
codepoints, U+FFFD for lone surrogates, UTF-8 encode. Malformed escapes keep
the old '?' so a truncated body still parses.

The deeper failure was that nothing measured this for two months. Every gauge
in the system reports whether the machinery is running; none reported whether
the text it carries is intact. Adds both halves: engram_text_health_json() /
GET /api/text-health for the daily census, and a txt_damaged counter on the
heartbeat for live regression. Verified in both directions - clean UTF-8 does
not trip it, a deliberately damaged node does.
2026-08-08 08:43:18 -05:00
will.anderson 971b21751a self-review 2026-08-07: learning that cannot outlive the process is not learning
Yesterday's eligibility-trace fix made Hebbian consolidation numerically real:
hebb_max 0.000799 -> 0.4725, and 1,198 hebbian-associate edges formed in 23h48m.
This morning's census found where they went: nowhere.

  soul daemon (in-process graph):   42,426 edges, 1,198 hebbian
  engram server (:8742, durable):   41,213 edges,    49 hebbian

Two processes, two graphs, one direction of travel. The soul pulls from the
server every 10 min (GET /api/sync) and never pushes. It cannot fall back on
saving its own copy either: soul.el sets soul_snapshot_path only inside
`if is_genesis && safe_to_seed`, and safe_to_seed is unconditionally false
whenever ENGRAM_URL is set -- because the server owns persistence and a soul
writing snapshot.json would clobber it. That guard is correct. The consequence
was not: mem_save() has never once executed. The soul is the ONLY process
running idle cognition, so it is where essentially all co-activation happens --
and it was throwing away every association it learned, every restart, silently.
The mechanism worked and the learning still evaporated.

Consolidation is now a message, not a file. Fast volatile store hands each
newly-formed association to the slow durable store over the API the server
already exposes; only edges past ENGRAM_HEBB_LINK_MIN are ever queued, so what
crosses the process boundary already earned it.

- el_runtime.c: 512-slot overwrite-oldest write-back ring; enqueue at edge
  formation; engram_hebb_drain_json() pops a postable JSON batch. Drops and
  drains are counted, not silent -- a consolidation path that quietly discards
  is the exact failure this entry exists to correct.
- server.el: POST /api/edges/batch. persist_canonical() writes the full 60MB
  snapshot per call, and route_create_edge calls it per edge -- correct for one
  interactive edge, ruinous for bulk (~840MB/beat to persist 14 associations).
  Batch connects all, snapshots once. Same durability, 1/N the writes.
- act-stats: hebb_wb_pending / _drained / _dropped. pending climbing with
  drained flat = drain not called; drained climbing with sent 0 = POST refused.
  Both failure modes are now visible in the stream instead of in an autopsy.

Verified live: batch route accepts valid entries, skips malformed ones without
aborting the batch, and enforces _auth. All 1,256 learned associations are now
in the canonical store; the soul booted at 42,431 edges with hebb_max 0.4941
carried across the restart for the first time.
2026-08-07 08:46:37 -05:00
will.anderson 9f1db8278c self-review 2026-08-06: eligibility traces for Hebbian co-activation; dedup WM globally
Hebbian consolidation was inert. Census over the live graph (41,213 edges,
13,091 nodes, 23h44m uptime): strongest association hebb=0.000799 against a
0.15 consolidation threshold, and zero hebbian-associate edges ever formed.
Since the awareness loop calls engram_connect nowhere, this was the only path
by which the graph could grow its own structure — every edge was authored or
imported, none learned.

The defect was the event, not the rate. hebb is an EWMA whose fixed point is
P(event); raising ETA changes convergence speed, never the plateau. The event
was "both endpoints in WM in the same activate call" — demanded exact
simultaneity from a working memory that inhibition-of-return, breakthrough
rotation and the 24-slot global cap are all engineered to keep turning over
(~142 evictions/60s). The three mechanisms that make WM healthy are the ones
that made this measurement empty.

Replaced with three-factor eligibility traces (Sutton & Barto ch.7; Gerstner
et al. 2018; PLOS Comp Biol 2018 differential Hebbian learning): a node
entering WM sets a trace to 1.0, the trace decays exponentially in wall-clock
time (TC=300s, chosen against the measured ~31s scan cadence), and the
increment becomes ETA·trace(a)·trace(b). Strict generalization — co-resident
pairs read 1.0 on both ends and get exactly ETA, bit-identical to before.
warm×warm is deliberately not paired: eligibility must gate on something
happening now. Homeostatic ENGRAM_HEBB_NODE_BUDGET still bounds per-node mass.

Measured over a 60-call soak: hebb_max 0.0008 -> 0.0060, climbing at ~0.87
ETA/call against an all-time ceiling of 0.0008 before. hebb_mass 0.011 ->
0.019, no runaway. Projected consolidation of a genuinely recurring pair:
~1,730 calls, ~14h at autonomous cadence. links still 0 — that is expected
and is what tomorrow's review must check.

Also: Pass 3½ deduplicates this call's WM candidates, but the persisted WM
population is a union of fresh promotions and carry-over residents, and Pass
3½ never sees the second set. Confirmed live: two byte-identical copies of one
3,193-char document both holding slots (0.289 / 0.271). Added global
redundancy suppression in Pass 5 before the cap count. Post-fix census: 24
residents, 24 distinct contents, 0 wasted slots.

New gauges: hebb_warm (eligible-but-not-co-resident population), dup_wm_global.
2026-08-06 08:44:27 -05:00
will.anderson 3d05e0c2a9 self-review 2026-08-05: stop the decay function erasing the library
Census of the live graph under the uniform 168h half-life with floor 0.05: the
MEDIAN tdecay for every single node type was 0.0500 — the clamp. Memory 81% at
floor, Knowledge 58%, BacklogItem 91%, Project 98%, Tag 100%. A function whose
median output is its floor is not a signal, it is a constant with exceptions,
and the exceptions were whatever had been touched in the last few days.

What that cost: 10 of the 13 grounded value nodes — Precision Over Brute Force,
Honesty Before Comfort, The System Must Accumulate — sat at 0.05, a 20x
activation penalty, while Knowledge ingested overnight sat near 1.0 and held the
working-memory top slots. Since tdecay multiplies at every hop, a 2-hop path
through settled knowledge compounded to 0.0025: those regions were not
disfavoured, they were unreachable. The decay function was erasing the
accumulated library in favour of whatever arrived last night.

External corroboration — arXiv:2604.26970 measures retrieval under decay
regimes: no temporal weighting NDCG@5 0.274, uniform exponential decay 0.015.
Uniform decay is 18x WORSE than no decay, because it penalises stable knowledge
while failing to suppress stale volatile facts. Not even their full adaptive
hierarchy (0.260) beat switching decay off.

Half-life is now scaled by how established a node is:
  T_eff = T_HALF * (1 + ln(1 + activation_count))
The spacing effect and the Lindy property in one line — monotone, log-bounded
(a 10,000-activation node earns ~10x, never a permanent exemption), and built
on activation_count, which is measured, unlike tier, whose assignments are too
inconsistent to trust (the values node is tagged Episodic).

Floor 0.05 -> 0.25. Given no-decay outperforms uniform decay, the honest maximum
penalty for age alone is 4x, not 20x. Age should express a preference for the
recent; it must never make a region of the graph structurally unreachable.

Effect: well-established Knowledge median tdecay 0.773 vs rarely-activated
0.417 — the frequency signal now does work where the old function returned its
clamp for both. Values recover 0.05 -> 0.25 (the two frequently-touched ones to
0.79). Verified live: VBD whitepaper, component taxonomy and CGI now activate on
a values query. Per-node temporal_decay_rate override untouched.
2026-08-05 08:45:52 -05:00
will.anderson 3bf44dee2d self-review 2026-08-05: redundancy must not buy a scarce slot
Content-hash census of the live graph: 1,858 redundant copies, 44.9% of the
non-ISE store, all from a June id-scheme migration that re-added nodes under
fresh UUIDs instead of matching on content. Generation stopped in June; the
copies did not. Being byte-identical they carry identical embeddings, so they
score identically against any query.

Measured over 50 real query probes against the live 3,998-vector set:
40.2% of semantic seed slots were consumed by redundant copies of content
already in the seed set, 92% of retrievals affected, effective distinct seeds
4.78 of 8. Two fifths of every retrieval was spent re-reading the same page.

Deleting nodes is a separate operation with its own backup discipline. This
change makes the runtime immune to the condition instead: redundancy can never
buy a scarce slot, whatever state the graph is in. Enforced at both scarcity
points — semantic seed selection (a rejected copy does not consume one of the K
slots; the loop retries for the next distinct node) and WM admission via a new
Pass 3+1/2 ahead of the capacity cap, so 24 slots are contested by 24 distinct
meanings rather than by however many copies of one document exist.

Identity is exact content hash first, then cosine >= 0.995 for copies that
differ only in insignificant characters. At 768 dimensions that admits only
near-verbatim text: this suppresses redundancy, never similarity.

Live after restart: ~8.8 redundant seed candidates rejected per activation.
New dup_seeds/dup_wm gauges in act-stats.
2026-08-05 08:40:08 -05:00
will.anderson a43a35bd10 self-review 2026-08-04: restore working-memory continuity; learn graph structure from co-activation
WM continuity (the significant one). A node reached by the current query but
scoring under its type threshold was zeroed outright, while a node the query
did NOT reach got the full ACT-R carry-over treatment. Being found was punished
relative to not being found. Measured consequence: WM turned over 100% every
call — three activations of a byte-identical query gave |A∩B| = |B∩C| = 0 — and
wm_evicted stayed 0 the whole time because that path never counted. WM was not
a working set; it was six suppression-breakthrough nodes re-drawn per call.
Both exits from a WM slot now share one extracted retention rule.
Result: WM 6 -> 24 nodes (the designed Cowan capacity), top weight 0.097 ->
0.748 (natural promotion, not the breakthrough floor), and contents that are
actually query-relevant.

Hebbian learning. Edge weights were written once at engram_connect and never
changed; last_fired's only writer in 12.5k lines was an unrelated dharma path.
Every learning mechanism operated on nodes — the wiring between them was
frozen. Adds co-activation potentiation (HeLa-Mem arXiv:2604.16839) in a
separate `hebb` field so authored structure is never mutated, with homeostatic
per-node scaling the source lacks (PNAS 2422602122) to prevent hub saturation.

Measuring it produced the finding that mattered: zero edges existed between
co-active WM members, so reweighting existing edges was a no-op. This graph's
41k edges were all authored by explicit tool calls — nothing had ever formed an
association from experience. So Hebb literally: if the wire is absent, grow it.
Consolidation is gated hard (sustained EWMA past 0.15, <=2/call, 5% ceiling,
in-memory candidates discarded on restart) because it permanently mutates the
graph.

Two bugs caught only by instrumenting rather than assuming: the snap-to-zero
floor sat above the per-step increment, so nothing could ever accumulate; and
the reached-but-sub-threshold eviction above. Verified live end to end — 53
links formed under load, then discarded with the test snapshot.

Also exposes engram_act_stats_json over GET /api/act-stats. It had existed
since 2026-07-27 but was reachable only through the soul daemon, so diagnosing
the activation layer required a working soul. This review needed it and could
not get at it.
2026-08-04 08:56:11 -05:00
will.anderson 5d0d4555ae Sync main into dev (GitOps: keep dev current; main authoritative) (#84)
El SDK CI - dev / build-and-test (push) Successful in 8m32s
2026-08-03 15:38:40 +00:00
will.anderson afc92f4e33 self-review 2026-08-03: add engram_label_df term-specificity measure
The soul's curiosity auto-term extractor takes the first word of a top-WM
node label. It has no term-quality scoring, so three prior self-reviews each
bolted on another hand-curated blocklist (genre words 07-23, quoted titles
07-25, stopwords 07-30). Every one was written reactively, after a flood was
already observed. A list can only contain floods that already happened.

Two were in flight and unfixed when this review ran:
  "<!--"  label df 220 -> 252 nodes activated
  "SELF"  label df 175 -> 541 nodes activated (list has "Self" Title-case;
           str_eq is case-sensitive, so the uppercase token sailed through)

engram_label_df(term) counts nodes whose label contains term. Low-specificity
tokens are corpus-frequent by definition, so this catches the flood class
prospectively and tracks the corpus as the world-ingestor changes it. This is
Sparck Jones (1972), which introduced IDF under the name 'term specificity';
automatic stopword compilation from it is the textbook application.

NOT a replacement for the stopword list -- verified against all 86 listed
terms, not assumed. Catches 13 (Will:306, Self:175, Over:116, Knowledge:112),
misses 73 (Whose:0, Would:0, Could:0, This:9). Labels are terse titles, so
English function words are genuinely rare in them. The gates cover disjoint
failure modes; both are required.

Policy lives in awareness.el, not here: the runtime measures, the soul decides.
2026-08-03 08:38:58 -05:00
will.anderson 005e84e5d3 self-review 2026-08-02: bound the WM breakthrough storm; stop punishing semantic relevance for recency
Working memory was thrashing behind a healthy-looking gauge. wm_active sat
at 22-24 while breakthroughs ran 661-903 and evictions 485-717 PER 60s tick
- roughly 825-1125 nodes cycling in 5-call lockstep.

Root cause: the breakthrough path was an anti-starvation mechanism that reset
its own counter on firing, with no budget and no refractory. A node failing
its type threshold 5 times was force-promoted at exactly 0.10 and had its
suppression_count reset to 0, so it immediately restarted the identical
climb. Since BREAKTHROUGH_WEIGHT (0.10) > WM_FLOOR (0.05), every one of them
cleared the admission floor and entered the rank contest tied at 0.10, where
the tie-break degenerated to node-array index order. Cap-evicted nodes are
skipped by retrieval reinforcement, so they never got an access_ts record and
the STI inhibition-of-return damper never applied to them. That closed the
loop: re-suppressed, completely unmarked, forever.

An anti-starvation rule that resets its own counter without a bound is not a
fairness valve, it is an oscillator.

Fixes in engram_activate Pass 2:
- ENGRAM_BREAKTHROUGH_BUDGET (WM_CAP/4 = 6) caps intrusive thoughts per call.
- ENGRAM_BREAKTHROUGH_COOLDOWN (55) via NEGATIVE suppression_count. The field
  already serializes as %d and parses through eg_get_int_field, so negatives
  round-trip through snapshots with no struct or format change.
- Blocked breakthroughs no longer reset the counter; it saturates so a starved
  node surfaces on a later call instead of restarting from zero.
- Graded breakthrough weight by nearness to own threshold, so the rank
  tie-break is cognitive rather than insertion order. Invariant preserved:
  WM_FLOOR < weight < min(type_threshold).

Also: moved the additive cosine term AFTER the STI multiplier. It was applied
before, so an incumbent re-reached 30s later took t_n/(t_n+120) = 0.2x, which
cut the semantic term's ceiling from 0.20 to 0.04 - below every per-type
threshold. Meaning-match was being punished for having been recently useful.
Inhibition-of-return should rotate the structural score, not the semantic one.

Also: _eg_act_wm_evicted counted 3 of 5 eviction paths. The two carry-over
paths were silent, so the reported rate was an undercount of unknown
magnitude - while being used to diagnose an eviction pathology. All five now
increment.

Also: route_sync returned {"nodes":[],"edges":[]} when the snapshot export
failed. The soul's sync_ok check only tests for "" and "{}", so that
placeholder passed as a healthy sync: last_sync_ok_ts stamped, sync_age_ms
green, sync_empty never fired, added:0 forever. A broken sync was
indistinguishable from a quiet healthy one - the exact class this route was
added to fix. Returns a real error now.

Verified live (boot 20 vs boot 19): breakthroughs 661-903 -> 36/tick,
evictions 485-717 -> 12-46/tick against a counter that now covers more paths,
wm_active unchanged at 22-24, wm_avg_weight 0.138-0.273 -> 0.186-0.446.
Working memory is holding strong nodes instead of breakthrough-floor filler.
2026-08-02 08:48:59 -05:00
will.anderson 7f03876e26 self-review 2026-08-01: fix double-encode score mangling; expose similarity probe; presence-aware defaults
- route_create_node passed already-boxed Floats through el_from_float a
  second time, reinterpreting boxed bits as raw doubles — every HTTP-created
  node silently stored default salience/importance/confidence regardless of
  input (verified live: 0.9/0.25/0.6 in -> 0.5/0.5/1.0 stored). Floats now
  passed bare, matching the route_emit_ise pattern that always worked.
- Presence-aware defaults via json_get_raw: absent key != explicit value;
  confidence now honored from payload instead of hardcoded 1.0.
- GET /api/similarity?a=&b= wires engram_cosine_sim (built 2026-07-24,
  zero callers until now) into the introspection API.
- /health reports live node/edge counts instead of a hardcoded literal.
2026-08-01 08:38:51 -05:00
will.anderson 599073cb92 self-review 2026-07-31: strip emb from consumer API JSON; cumulative eviction/breakthrough counters
Every node object on consumer read routes (/api/nodes, /api/search,
activation results, neighbors, compiled context) carried the full ~5.7KB
emb vector — responses 10-50x oversized, blowing MCP token limits.
engram_emit_node_json now takes include_emb; only engram_save passes 1,
so persistence and the /api/sync//api/edges replication paths (which
serve engram_save output) keep embeddings intact.

_eg_act_wm_evicted/_eg_act_breakthroughs were reset at the top of every
engram_activate, so act_stats reported only the last call and the 60s
heartbeat missed nearly all events (curiosity runs 2 activates per 30s).
Both are now monotonic process-lifetime totals; consumers diff readings.
2026-07-31 08:41:33 -05:00
will.anderson 8347a2f1c0 Merge pull request 'docs: add root README mapping the El monorepo' (#83) from feat/AddingReadme into dev
El SDK CI - dev / build-and-test (push) Successful in 8m22s
2026-07-31 04:25:44 +00:00
will.anderson 7f66529510 self-review 2026-07-30: WM absolute admission floor + anchor coherence + centroid new-entrant gate
Working memory was pinned saturated (24/24, wm_saturated:1 on every
heartbeat) because every cap path only trimmed the population down TO
the cap — rank-based eviction guarantees a full WM whenever >=24 nodes
hold any weight, so sub-cap fill was unreachable and the saturation
flag carried no information.

- ENGRAM_WM_FLOOR 0.05: absolute admission bar (Soar WM forgetting,
  Derbinsky & Laird ICCM 2012 — removal by absolute threshold, not
  rank) applied in Pass 4, carry-over, Pass 5, and load-cap. Fill can
  now drain below 24 during quiet periods.
- Zero wm_anchor at every eviction site: stale anchors on evicted
  nodes were a latent resurrection bug.
- Context centroid folds only NEW WM entrants: incumbents re-promoted
  every scan no longer re-entrench the centroid each call, breaking
  the WM->centroid->e_eff->re-selection positive feedback (fixation
  driver behind the wm_top0_streak=1407 incident).

Verified live: wm_active 3->22->23, wm_saturated:0 post-restart.
2026-07-30 08:45:15 -05:00
will.anderson 6ebe3d0d66 self-review 2026-07-28: feed importance into WM scoring
n->importance was stored, serialized, and clamped at creation but never
read by any activation path — a curated importance=1.0 node competed
identically with a default note. Multiply raw_wm by (0.5 + importance):
default 0.5 nodes are unchanged (x1.0), critical x1.5, low x0.6;
importance<=0 from legacy snapshots stays neutral. Verified activation
and WM promotion unchanged for default-importance candidates.
2026-07-28 08:37:34 -05:00
will.anderson 9f362c90e5 self-review 2026-07-27: query-aware propagation gating + activation observability
- Gate each spreading-activation increment by target-node query similarity
  (arXiv:2606.30133): soft gate FLOOR+(1-FLOOR)*clip(cos), FLOOR=0.25, for
  embedded targets; ungated for unembedded; disabled when embedder is down.
  Prior spreading was query-blind — hubs relayed activation into branches
  unrelated to the query.
- Stats: add embed_eligible_count so embedding coverage is measured against
  the true denominator (ISE/Tag/short nodes can never embed). Today's review
  misread 3753/12693 as a 30% coverage gap; eligible coverage is 100%.
- Observability: per-call wm_evicted + breakthroughs counters and embed
  circuit-breaker state exposed via engram_act_stats_json() — the three
  highest-value previously-invisible executive-filter transitions.
2026-07-27 08:38:48 -05:00
will.anderson 11dc138a93 self-review 2026-07-26: fix WM frozen-anchor fixation, strengthen self-inhibition, load-path emb leak
- Carry-over branch: occupancy inhibition m = t_c/(t_c+t_hold), t_c=3600s
  (ENGRAM_CARRY_TC). An unreached incumbent held its wm_anchor verbatim
  (keep~1.0 for BLL inflated in the pre-07-25 era) — observed 23h at WM
  top while every reached node rotated at the 0.10 breakthrough floor.
  STI only runs in the reached branch; inhibition must key on occupancy,
  not retrieval recency (Morita 2021 / Lebiere & Best 2009).
- engram_strengthen: drop the 07-22 BLL access record — the 07-25 STI
  multiplier reads the same ring, so novelty reinforcement self-inhibited
  its target for ~2 minutes.
- engram_load reset: free n->emb (~3KB/embedded node leaked per reload).
- engram_wm_top_json: emit id — its absence made the heartbeat's
  wm_top0_streak compare ""=="" and measure uptime, not fixation.
2026-07-26 08:40:49 -05:00
will.anderson 227f158a05 self-review 2026-07-25: short-term inhibition-of-return + explicit embedding backfill
Working memory was winner-take-all: suppression_count never entered the
promotion score and was reset on promotion, so two high-salience nodes
pinned a saturated 24-slot WM for hours. Add Lebiere-Best (CogSci 2009)
short-term inhibition — raw_wm *= t_n/(t_n + 120s) from the most recent
recorded access — producing emergent round-robin over WM candidates.

embedded_count stalled at 93/12175 after restart: the lazy backfill only
runs inside engram_activate, which nothing calls on the authoritative
store in production, and in-RAM vectors were never snapshotted. Add
engram_embed_backfill(n) + GET/POST /api/embed-backfill route that
persists the canonical snapshot whenever it embeds anything; the soul
heartbeat pumps it at 32/min.
2026-07-25 08:45:13 -05:00
will.anderson 97e484221d self-review 2026-07-24: wire embedding cosine similarity into activation (bl-b2d1c944)
Semantic activation was spec-only since 2026-06-30 — the seed loop used
istr_contains and nothing else. Per the 07-21 integration brief:

- EngramNode gains a lazily-backfilled nomic-embed-text vector (8/call
  inside engram_activate, newest-first; no create-path latency, no bulk
  Ollama hammering during sync seeds)
- query embedding (cached) drives a top-K cosine seed supplement
  (HippoRAG use-similarity-twice) plus an additive WM term with
  shift-and-floor at 0.45 — raw cosine is a constant bias in anisotropic
  spaces (unrelated pairs read 0.4-0.7), floor-and-ramp makes it a signal
- 4s embed timeout (http_do_t) + 3-strike circuit breaker: activation
  never wedges on a dead embedder; everything degrades to lexical
- embeddings persist as %.4g comma lists in snapshots, parsed by both
  loaders; embedded_count in /api/stats tracks coverage
- engram_cosine_sim + http_delete_json exposed (DELETE now carries a
  body — the server's _auth scheme requires it)
- route_create_node honored only content/node_type/salience; label,
  importance, tier, tags were silently dropped (label defaulted to
  content). Now honored via engram_node_full.

Verified live: embedded_count 0->96 across activations, semantic-only
promotion observed (zero token overlap), snapshot round-trip intact.
2026-07-24 08:52:54 -05:00
Andre Botelho Rodrigues Almeida b97b644799 Addind readme.md file to start documenting the repo
El SDK CI - dev / build-and-test (pull_request) Successful in 8m18s
2026-07-23 16:41:51 -03:00
will.anderson d71fc4c1c0 Merge pull request 'promote stage -> main: reconciled el runtime (engram search + natives + durable truncation fix + Windows port)' (#82) from stage into main
El SDK Release / build-and-release (push) Successful in 8m31s
El SDK CI - dev / build-and-test (pull_request) Successful in 8m41s
2026-07-22 21:44:01 +00:00
will.anderson a118d19393 Merge pull request 'promote dev -> stage: el cluster (#66 engram + #79 truncation fix + release-runtime Windows port)' (#81) from dev into stage
El SDK CI - stage / build-and-test (push) Successful in 7m58s
El SDK Release / build-and-release (pull_request) Successful in 4m16s
2026-07-22 21:20:17 +00:00
will.anderson c6aa1e5c53 Merge pull request 'Land el cluster: #66 engram search + natives, #79 truncation fix, + release-runtime Windows port (reconciled)' (#80) from reconcile/el-cluster-windows-runtime into dev
El SDK CI - dev / build-and-test (push) Successful in 8m3s
El SDK CI - stage / build-and-test (pull_request) Successful in 4m25s
Land el cluster (#66 + #79 + release-runtime Windows-port reconciliation) into dev
2026-07-22 21:06:36 +00:00
will.anderson ff577391f2 reconcile(release-runtime): Windows-port + complete v1.0.0 release runtime so the desktop soul cross-compiles
El SDK CI - dev / build-and-test (pull_request) Successful in 7m7s
The desktop soul (neuron/dist) compiles against the v1.0.0-20260501 release
runtime. After #66 landed the engram natives (tokenized/ranked search,
engram_prune_telemetry) and #79 the durable truncation fix into this runtime,
two gaps remained before it could cross-compile the Windows brain:

1. Windows OS boundary: the release runtime had no Win32 path. Ported the same
   _WIN32-guarded shim the mainline runtime carries (#69): #ifdef _WIN32 ->
   el_platform_win.h (winsock/dlsym/popen + WSAStartup ctor), SOCKET fd guards
   and el_closesocket() at every socket site, CreateProcessA for exec_bg, the
   tm_zone/mingw guard, an el_setsockopt optval wrapper (GCC14), and curl-less
   libcurl stubs. Every change is _WIN32/HAVE_CURL-gated — the POSIX build is
   byte-identical (gcc -fsyntax-only clean; native behaviour unchanged).

2. Header exports: the release el_runtime.h omitted symbols the soul dist calls
   that are defined in this runtime's .c — the http_handler_fn/http_handler4_fn
   typedefs and el_arena_push/pop, engram_prune_telemetry, engram_get_node_by_label.
   Declaration-only, POSIX-neutral; fixes implicit-declaration/unknown-type
   errors under the C11 mingw build.

Result: x86_64-w64-mingw32-gcc compiles el_runtime.c + all 48 soul modules
clean; POSIX gcc -fsyntax-only clean. This is the Windows-port PR the runtime
needed on main (the release-runtime counterpart to #69), landed via stage.
2026-07-22 15:56:08 -05:00
will.anderson ee0d5f9b97 Merge #79: durable HTTP response-truncation fix, both runtimes (via stage) 2026-07-22 15:46:02 -05:00
will.anderson 391bd818ea Merge #66: tokenized+ranked engram lexical search + engram natives (via stage) 2026-07-22 15:45:54 -05:00
will.anderson 43636aed99 runtime: pair fs_read length hint with its buffer in BOTH runtimes — kill response truncation for good
El SDK Release / build-and-release (pull_request) Failing after 7s
The binary-safe fs_read length (_tl_fs_read_len) was consumed by the HTTP
response path for ANY body, even when a handler wrapped a smaller file into a
larger reply. Content-Length then lied AND the send stopped short: the
safety-contact (988) routes returned 178 of 208/218 bytes, cut mid-'set_at' —
unparseable JSON. The desktop app read that as failure. On Windows the shipped
brain is an OLD build without even the per-handler workaround, so EVERY reply
truncated: the app can't read confirmations and refuses the new user.

Durable fix: pair the length hint with the exact buffer pointer it describes
(_tl_fs_read_buf). Apply the raw byte count ONLY when the response IS that
buffer (binary file serving stays correct); every wrapped/enveloped/derived
body is measured with strlen. Reset both at request start and in fs_read /
json_get_raw. This also closes the stale-hint heap over-read (a length larger
than a later body would read past it out the socket) that a plain max() leaves
open — so this class of bug dies on every platform, not just where a handler
happened to be patched.

Applied identically to the mainline runtime (lang/el-compiler/runtime) AND the
frozen release runtime (lang/releases/v1.0.0-20260501) the desktop souls
compile against — the release copy still carried the raw leak, which is why the
Windows brain kept truncating. Same proven approach as PR #78 (Tim Lingo),
extended to cover the release runtime and rebased onto current main.

Both runtimes: gcc -fsyntax-only clean.
2026-07-22 15:04:25 -05:00
will.anderson 8f8ccc945e self-review 2026-07-22: persist canonical snapshot on write routes; newest-first tie-break in node listings
El SDK Release / build-and-release (pull_request) Failing after 14m24s
Durability: the 2026-07-21 fix stopped read routes writing the canonical
snapshot but left no save on ANY write path — every mutation lived in RAM
until a manual POST /api/save. Observed live: two restarts reverted the
store to a 17h-old snapshot, destroying same-day writes. persist_canonical()
now runs after node/edge create, knowledge capture, forget, strengthen, and
load-merge. ISE telemetry excluded deliberately (48h-pruned, loss-tolerant,
~2/min; snapshotting 28MB per heartbeat is waste).

Listing order: scan routes sort by salience with store-order ties, so
equal-salience telemetry (all ISEs are 0.3) returned OLDEST first — a
limited /api/nodes query silently returned a stale window, and a 41h-old
heartbeat series read as a live outage during this review. Ties now break
newest-first by created_at.
2026-07-22 08:51:33 -05:00
will.anderson 409ec99397 self-review 2026-07-22: ACT-R/Petrov base-level WM decay replaces per-call multiplicative carry-over
The old carry-over (weight *= 0.7 per engram_activate call) was call-rate-
dependent — carried context died in seconds under rapid curiosity scans and
lingered for hours under quiet loops — and a decayed scalar cannot represent
access frequency at all.

Now: k=10 access-timestamp ring + Petrov (2006) closed-form tail, d=0.5.
WM promotion and engram_strengthen record presentations; carry-over evicts
at base-level tau=-3.0 (Soar forgetting, ~403s single-touch) and shapes the
weight held at promotion (wm_anchor) with the ACT-R retrieval logistic
(s=0.4) — a pure function of wall-clock time, idempotent per call.
Persisted as access_ts/wm_anchor in snapshots; legacy nodes fall back to
the optimized form ln(n/(1-d)) - d*ln(L). base_level exposed in both node
serializers for observability.

Backing spec: 2026-07-21 integration brief (bl-b17facdd). Verified live:
carried weight ~anchor seconds after two disjoint activations (old code:
0.49x); frequency-hot nodes hold B=1.9 vs -0.14 single-touch.
2026-07-22 08:44:39 -05:00
will.anderson dc39a61e2c self-review 2026-07-21: stop read routes clobbering canonical snapshot; add /api/load-merge
Root cause of the 2026-05→07 identity-node loss: route_scan_edges and
route_sync serialized state by engram_save()ing over the canonical
snapshot.json on every GET, so one bad boot load meant the first read
request overwrote the good snapshot. Read routes now export to scratch
paths. Boot guard preserves evidence on non-empty-file/zero-node loads
and keeps a boot-time backup on good loads. New POST /api/load-merge
(explicit path required) used to restore 385 identity nodes + 1115
edges from the 2026-05-13 backup.
2026-07-21 08:50:38 -05:00
will.anderson eba9eac8a8 self-review 2026-07-19: port stranded fixes to the release runtime (production copy)
Three fixes that existed elsewhere but never reached the runtime the engram
binary actually builds against:

- tokenized + ranked query matching (search/search_json/activate seeds/
  goal_bias) ported from the el-compiler copy (e3dabe3, 2026-07-14) — the
  production engram kept whole-query Ctrl-F for 5 days after the fix
  'shipped'. Multi-word curiosity seeds went 0 -> 36 activated. Kept the
  ISE seed exclusion the el-compiler copy dropped.
- Knowledge -> 0.20 WM threshold after tier checks (dev-line 4bf7716):
  Semantic/Episodic Knowledge nodes fell to the 0.40 note default and only
  entered WM via breakthrough.
- goal_bias: Knowledge in is_knowledge + curiosity-seed technical terms
  (dev-line d53516b).

Also: seed_epoch was a running pairwise average, not the mean it claimed —
exponentially over-weighted later seeds in the temporal-proximity bonus.
Fixed to a true int64-sum mean. Stale INHIBITION_FACTOR comment corrected.

Root cause captured as knowledge: two runtime copies + branch-per-fix
without merge discipline stranded the entire dev semantic layer (cosine
activation, embeddings) out of production. Reconciliation planned as P1.
2026-07-19 08:46:47 -05:00
will.anderson ab6b52a0b4 self-review 2026-07-18: fix soul SIGABRT double-free + engram route scoping sweep
1. engram_neighbors_json (release runtime): BFS frontier/visited strings were
   el_strdup'd (arena-tracked) but manually freed, so el_request_end()
   double-freed every one — SIGABRT in http_worker under load (2 prod crashes
   today via /api/neuron/session/begin and /api/neuron/graph; reproduced and
   verified fixed with ASAN). Introduced when porting from the dev runtime,
   which correctly uses plain strdup. Third instance of the
   arena-vs-manual-free class (after EngramNode 07-15 and idmap keys 07-16).

2. server.el: let-in-if scoping sweep — defaults assigned inside if-blocks
   never mutated the outer binding, so /api/search and /api/activate always
   ran with q="", created nodes got node_type=""/salience=0.0, edges got
   relation=""/weight=0.0, and save/load with no path hit engram_save("").
   Rewritten to the let-if-else expression form. /api/activate now also
   rejects empty queries instead of wiping carried WM weights.

3. engram_activate: retrieval reinforcement (ACT-R base-level learning) —
   nodes promoted to WM that survive both capacity caps now get
   last_activated/activation_count updated, so frequently retrieved memories
   decay slower than abandoned ones. Scoped to promoted-only to avoid
   flattening dampening across BFS fan-out.
2026-07-18 08:48:04 -05:00
will.anderson 2baa0b9a41 Merge pull request 'release: promote stage -> main (ci publish hardening for sdk-release)' (#77) from stage into main
El SDK Release / build-and-release (push) Successful in 7m55s
2026-07-15 21:21:28 +00:00
will.anderson 6a8b2461cd Merge pull request 'release: promote dev -> stage (ci publish hardening for stage/main)' (#76) from dev into stage
El SDK CI - stage / build-and-test (push) Successful in 8m19s
El SDK Release / build-and-release (pull_request) Failing after 13m1s
2026-07-15 21:16:11 +00:00
will.anderson bcb356fe69 Merge pull request 'ci(stage,main): decouple ci-base rebuild, make SDK publish fail loudly' (#75) from hotfix/ci-stage-main-publish-hardening into dev
El SDK CI - stage / build-and-test (pull_request) Successful in 4m27s
El SDK CI - dev / build-and-test (push) Failing after 14m3s
2026-07-15 21:15:27 +00:00
will.anderson dd7827059a ci(stage,main): decouple ci-base rebuild, make SDK publish fail loudly
El SDK CI - dev / build-and-test (pull_request) Failing after 14m30s
Mirror the PR #72 fix (applied to ci-dev.yaml) onto ci-stage.yaml and
sdk-release.yaml. The stage and prod release jobs reported FAILURE even
when the el-runtime-c/-h publish SUCCEEDED, because the ancillary ci-base
Docker rebuild (a CI-cache optimization on the fragile host-mode GCE
runner) reddened the whole job.

- Rebuild ci-base step: continue-on-error: true — never blocks/reddens
  the job; the SDK publish is the deliverable.
- Publish step: set -euo pipefail + empty-key guard + active-account echo
  so a real publish failure still fails loud and is diagnosable.
2026-07-15 16:14:50 -05:00
will.anderson 208e36c899 Merge pull request 'release: promote stage -> main (tokenized search, get_node_by_label, epm fix, win portability)' (#74) from stage into main
El SDK Release / build-and-release (push) Successful in 8m23s
2026-07-15 18:24:39 +00:00
will.anderson b97ce74d1f Merge pull request 'release: promote dev -> stage (tokenized search, get_node_by_label, epm fix)' (#73) from dev into stage
El SDK CI - stage / build-and-test (push) Failing after 8m45s
El SDK Release / build-and-release (pull_request) Successful in 4m1s
2026-07-15 17:20:11 +00:00
will.anderson 155a449c4e Merge pull request 'ci(dev): make SDK publish fail loudly, decouple ci-base rebuild' (#72) from hotfix/ci-dev-publish-hardening into dev
El SDK CI - dev / build-and-test (push) Successful in 8m56s
El SDK CI - stage / build-and-test (pull_request) Successful in 4m10s
2026-07-15 16:34:14 +00:00
will.anderson 4696fd6833 ci(dev): make SDK publish fail loudly, decouple ci-base rebuild
El SDK CI - dev / build-and-test (pull_request) Successful in 8m51s
The dev push build went green-then-red while nothing published: the
Publish step had no set -e, so an auth/upload failure exited 0 (silent
no-publish), while the ci-base rebuild (set -euo pipefail + Docker on the
host-mode runner) hard-failed the job. Add set -euo pipefail + an empty-key
guard + active-account echo to the Publish step so failures surface with a
retrievable log, and mark the ci-base cache rebuild continue-on-error so
the fragile Docker step can never block the actual SDK artifact publish.
2026-07-15 11:33:37 -05:00
will.anderson 581a351fb1 Merge pull request 'integrate: stack PRs #65–#69 (elc OOM guard, tokenized+semantic engram search, get_node_by_label, win portability) for green CI' (#71) from hotfix/stage-elc-engram-integration into dev
El SDK CI - dev / build-and-test (push) Failing after 14m31s
2026-07-15 15:49:57 +00:00
will.anderson 8ce8656de2 epm: declare cross-module callees as extern fn so strict compilers accept generated C
El SDK CI - dev / build-and-test (pull_request) Successful in 7m33s
epm's sibling modules (registry/install/update) call functions defined in other
modules and in the El runtime (config, read_installed, registry_find,
manifest_deps, manifest_name, registry_latest_version, registry_token,
install_vessel, installed_version) without importing them, so elc emits no C
prototype for those calls. gcc<=13 treated the resulting implicit declarations
as warnings; gcc>=14 and clang reject them as hard errors, which is why the
"Build epm" CI step fails and blocks the whole dev/stage pipeline.

Add `extern fn` forward declarations -- El's own separate-compilation mechanism
-- for each cross-module callee at the top of registry/install/update. This
gives elc the correct C prototype in every generated translation unit, so the
calls compile cleanly and still resolve at link time. Simply suppressing
-Wimplicit-function-declaration would be unsafe: an implicit int return
truncates the 64-bit pointer returns of config/registry_find into a latent
crash, so declaring the true signatures is the correct fix. Localized to epm;
touches neither elc nor the runtime.
2026-07-15 10:14:43 -05:00
will.anderson 1e49560f1f Merge remote-tracking branch 'origin/feat/engram-semantic-search' into hotfix/stage-elc-engram-integration
El SDK CI - dev / build-and-test (pull_request) Failing after 14m39s
# Conflicts:
#	lang/el-compiler/runtime/el_runtime.c
2026-07-15 09:33:05 -05:00
will.anderson e8f0b5a9de Merge remote-tracking branch 'origin/fix/engram-lexical-tokenized-search' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 40287c4cfc Merge remote-tracking branch 'origin/hotfix/win-runtime-portability' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 0481bea44d Merge remote-tracking branch 'origin/hotfix/runtime-engram-get-node-by-label' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 9d565ca080 Merge remote-tracking branch 'origin/hotfix/elc-fixes' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 4773dd0aa2 runtime: make Windows soul reproducible from a clean el checkout
El SDK Release / build-and-release (pull_request) Failing after 16s
Two el_runtime portability defects only ever lived in staged local copies
used to hand-build neuron-ui PR #136's curl-enabled Windows neuron.exe.
gcc 15 promotes both to hard errors, so a clean el checkout cannot rebuild
that soul. Upstream the minimal fixes so the build is reproducible:

- http_serve_async: cast setsockopt optval to (const char*). Win32/mingw
  setsockopt wants const char*, not int*; the cast is a no-op on POSIX and
  matches the four already-cast sites elsewhere in this file.
- engram_save persist path: map fsync -> _commit in the _WIN32-only
  el_platform_win.h (io.h already included). Windows has no fsync(); the
  POSIX path is untouched.
2026-07-15 04:24:08 -05:00
will.anderson 6b9d9e6c4a Add engram_get_node_by_label runtime native to unblock soul link
El SDK Release / build-and-release (pull_request) Failing after 22s
chat.el calls the runtime native engram_get_node_by_label to fetch
well-known nodes (conv:history, session:summary) by stable label rather
than by ID — immune to vector-index drift across restarts. The current
runtime never defined it, so the regenerated dist/soul.c fails to link.

Backport the function verbatim (idiom-adapted to jb_finish) from release
runtime v1.0.0-20260501 and register it as an EL builtin exactly like its
siblings: runtime definition + prototype, __-prefixed seed wrapper +
prototype, and codegen arity entry. No search-site code is touched.
2026-07-15 04:07:33 -05:00
will.anderson b4967af13e feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical)
Lexical istr_contains alone can't surface a node whose words don't appear
in the query. This adds an optional dense-vector layer: node content and the
query are embedded through Ollama (nomic-embed-text), and nodes are ranked by
cosine similarity unioned with lexical hits, so a paraphrase query reaches the
right node.

Wired into all three query entry points in el_runtime.c:
  - engram_search_json (HTTP /api/search): collect lexical ∪ semantic
    candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine),
    rank, emit top-N. Stable sort preserves old order when semantic is off.
  - engram_search (internal el_val twin): lexical ∪ semantic union.
  - engram_activate seed loop (HTTP /api/activate): a node seeds if it
    lexically matches OR clears the cosine threshold; pure-semantic seeds
    enter scaled by cosine so paraphrase spreads without overpowering.

Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot
runtime probe. If curl is compiled out, Ollama is unreachable, or
ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers
fall back byte-for-byte to the pre-existing lexical search.

Node embeddings are cached in process memory keyed by node id with an FNV-1a
content hash for invalidation; the query is embedded once per call — so the
graph is not re-embedded on every query. nomic task prefixes
(search_query:/search_document:) are applied for retrieval separation.

Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in
(-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL,
ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).
2026-07-14 18:48:16 -05:00
will.anderson e3dabe3e08 fix(engram): tokenized + ranked lexical search, not whole-query Ctrl-F
El SDK Release / build-and-release (pull_request) Failing after 14m46s
engram search/activate/goal-bias matched the ENTIRE raw query string as a
single case-insensitive substring (istr_contains(field, q)). Multi-word
queries like "windows msi signing" only matched a node containing that exact
contiguous run, so real multi-word queries returned ZERO on a graph saturated
with the answer. This is Ctrl-F, not search — and search is the core of the
engram being useful.

Fix: split the query on whitespace into distinct tokens; a node matches if it
contains ANY token in content/label/tags. Rank by distinct tokens matched
(desc) then salience (desc). istr_contains is kept unchanged as the per-token
primitive. Single-token queries are a strict special case (score 0 or 1) so
the many single-word callers do not regress.

Sites changed (all in el_runtime.c):
- new helpers engram_tokenize_query / engram_node_match_score / engram_rank_cmp
- engram_search           (internal el_val_t path)
- engram_search_json      (HTTP /api/search path)
- engram_activate seed loop (HTTP /api/activate path; seed activation scaled
  by token coverage so full-query matches seed more strongly)
- engram_goal_bias overlap bonus upgraded to graded token coverage

Proof (6591-node snapshot copy, rebuilt binary on :8799, POST JSON path):
  windows msi signing  0 -> 20   Will Anderson  0 -> 20
  windows msi          0 -> 20   tokenized search fix  0 -> 20
Single-word parity preserved (VBD/volatility/elc capped at limit; unkey = all
matching nodes). Top hits are relevant (e.g. "Will Anderson" surfaces the
Project Design and VBD whitepapers).

Note: GET ?q=a%20b still returns 0 because query_param (server.el) does not
URL-decode — a separate EL-layer bug; the soul's POST-JSON path is fixed here.
2026-07-14 18:39:07 -05:00
will.anderson 0a0a2bcb44 parser: bound token reads to Eof so malformed input errors instead of OOMing
El SDK Release / build-and-release (pull_request) Failing after 16s
Out-of-range tok_kind/tok_value reads returned runtime null (el_list_get OOB
-> 0) rather than the Eof sentinel, so the inner parse loops (parse_block,
call-arg, array-literal, match-arm) that terminate only on their close
delimiter or k=="Eof" never saw Eof once the cursor ran past the single
trailing Eof token. On unclosed-delimiter input the parser then appended AST
nodes forever -> unbounded allocation -> ~700GB -> OOM (observed compiling
neuron/sessions.el).

Fix at the choke point: tok_kind returns "Eof" and tok_value returns "" for
out-of-range positions, restoring the parser-wide contract that reads at/after
the end yield Eof. expect() no longer steps past the Eof sentinel on mismatch.
This terminates every overrun loop simultaneously; a malformed program now
surfaces as a normal (best-effort) parse end instead of exhausting memory.

Requires a self-hosted bootstrap rebuild of elc to take effect.
2026-07-14 14:21:39 -05:00
will.anderson 2b2a1246e7 Merge pull request 'runtime: fix the memory-leak + write-corruption pair in el_runtime.c' (#64) from hotfix/el-runtime-leak-and-persist into main
El SDK Release / build-and-release (push) Failing after 10m52s
2026-07-13 21:23:31 +00:00
will.anderson f78da81aa4 runtime: fix the memory leak + write-corruption pair in el_runtime.c
El SDK Release / build-and-release (pull_request) Failing after 11m58s
Two independent investigations, one runtime, complementary halves:

1. Leak (Jul 2, this machine): JsonBuf buffers returned via el_wrap_str
   were raw malloc, never arena-tracked — every engram_*_json call leaked
   its output unconditionally. Added jb_finish() arena-tracking across all
   ~30 return sites. Plus el_arena_push/pop per-tick bracketing support
   for the soul's awareness loop (the loop ran outside any request arena,
   so even correctly-tracked allocations were permanent — 7.5GB RSS in
   under a minute at 1s tick).

2. Corruption (Tim's container soak, docs findings/container-migration):
   stored engram node/edge fields (content, node_type, label, tier, tags,
   metadata, from/to ids) were arena el_strdup — freed at request end,
   leaving dangling pointers that read back as recycled request-buffer
   bytes one request later. This is the June corruption root cause and
   the mechanism that grew snapshot.json to 18GB of empty-type junk
   (21.6M nodes, 3,335 real). 39 sites switched to el_strdup_persist,
   plus a latent double-free fix in engram_load metadata fixup.

Interaction note: fix 1's per-tick arena reclamation makes fix 2
mandatory — more aggressive arena recycling widens the use-after-free
window if stored fields still live in the arena. Apply as a pair, never
separately.

Verified live: soul + engram rebuilt from this runtime, booted against
the recovered real snapshot (3,335 nodes/40,146 edges), 5h stable at
<100MB RSS, write-then-next-request field-integrity test passes (the
June corruption fingerprint does not reproduce). engram/dist/engram
binary updated from this build.

Investigation credit: leak diagnosis this machine Jul 2-6; corruption
diagnosis + persist-fix patch by Tim's instance (docs PR #4).
2026-07-13 16:22:02 -05:00
will.anderson 2597a092bb Merge pull request 'chore: integrate local main commits' (#63) from integrate/local-main-commits into main
El SDK Release / build-and-release (push) Successful in 10m58s
2026-07-01 16:30:17 +00:00
will.anderson 226b798407 Merge branch 'fix/windows-rusage-guard' (PR #61): UTF-8 guard, engram sync route, native platform backends, UI vessels
El SDK Release / build-and-release (pull_request) Failing after 13m57s
2026-07-01 11:27:54 -05:00
will.anderson cfe8cb1c80 fix(release-snapshot): fflush stdout in println and update Knowledge threshold
El SDK Release / build-and-release (pull_request) Failing after 20s
2026-07-01 11:25:09 -05:00
will.anderson 688b8508fb feat(runtime): native platform backends and UI vessels onto main 2026-07-01 11:21:23 -05:00
will.anderson 59cea116c5 build(engram): rebuild binary with engram_load_merge runtime (deb0520)
El SDK Release / build-and-release (pull_request) Failing after 19s
Runtime now includes engram_load_merge — soul daemon awareness.el calls
this function during its periodic sync refresh cycle. Binary rebuilt from
server.el (unchanged source) + updated el_runtime.c.
2026-06-30 08:59:01 -05:00
will.anderson deb0520551 feat(runtime): port engram_load_merge to released runtime + add missing WM headers
engram_load_merge was added to el-compiler/runtime in 35c1897 but never
ported to the released runtime used by Engram and the soul daemon.

awareness.el calls engram_load_merge in its sync refresh cycle; without
this function in lang/releases/v1.0.0-20260501/el_runtime.c the soul
daemon fails to compile.

Also adds header declarations for engram_wm_count, engram_wm_avg_weight,
engram_wm_top_json, and engram_load_merge — all four were added as
implementations (da116b2 / 35c1897) but their prototypes were missing from
el_runtime.h, causing implicit-function-declaration warnings and potential
ABI breakage on stricter compilers.

Identified during self-review 2026-06-30.
2026-06-30 08:57:22 -05:00
will.anderson da116b2884 self-review 2026-06-30: WM cap, breakthrough floor, ISE exclusion + route
Port critical WM fixes from self-review 2026-06-26 branch (f7bd99a) that were
never merged to HEAD. Running binary had these fixes; source did not — rebuild
would have silently regressed all three improvements.

1. ENGRAM_BREAKTHROUGH_WEIGHT 0.25→0.10
   With 0.25, naturally-promoted nodes (threshold ≥0.15) decayed below the
   breakthrough floor within one activation call and lost their WM slot to
   fresh breakthrough candidates. All 524/525 WM nodes were at floor = useless.
   Invariant: BREAKTHROUGH_WEIGHT < min(type_thresholds = 0.15 Canonical).

2. ENGRAM_WM_CAP=24 with Pass 4 (per-call) + Pass 5 (global) enforcement
   Without cap, broad curiosity seeds promote 500+ nodes simultaneously.
   wm_avg_weight collapses, goal-bias differentiation is lost. Verified:
   "knowledge" query now promotes exactly 24 nodes (was 525). Cowan (2001)
   cognitive basis: WM capacity ~4 chunks; 24 allows rich multi-topic context.

3. ISE exclusion from WM (Pass 2 guard)
   InternalStateEvent JSON content ("knowledge", "memory", etc.) triggered
   lexical seeding → suppression accumulation → breakthrough at floor. ISEs
   are observability-only and must never surface in context compilation.
   suppression_count cleared so ISEs never build toward breakthrough.

4. route_create_ise importance fix (0.5→0.3)
   Corrects mismatch between HTTP route and awareness.el in-process fallback.
   Also adds body comment clarifying auth-exempt rationale.

SYNAPSE (arXiv 2601.02744) validates WM cap design and ISE exclusion principle.
Next priority: cosine similarity seeding to complement lexical BFS.
2026-06-30 08:48:19 -05:00
will.anderson 58753a88d7 feat(ui): native vessel, HTML vessel update, native hello examples, profile card, UI tools
El SDK Release / build-and-release (pull_request) Failing after 17s
el-native vessel: El-level wrappers around __widget_* C builtins, exposing
vstack, label, button, text_field, etc. as clean El functions for application code.

el-html/main.elh: updated extern declarations for the HTML vessel's codegen API.

native-hello: cross-platform desktop example (AppKit/GTK4/Win32/SDL2) with
build scripts, Dockerfiles for Linux/Pi, and Win32 cross-compile support.

native-hello-android: Gradle project with ElBridge integration and build script.

native-hello-ios: Xcode project for the iOS UIKit target.

profile-card: manifest.el for a styling/layout/i18n example app that exercises
el-style, el-layout, el-i18n, el-config, and el-secrets vessels.

ui/tools/native-codegen: Python codegen pass (el_ui_native_codegen.py) that
lowers el-ui component DSL to el-native vessel calls, plus build script and
test fixtures.
2026-06-29 12:40:37 -05:00
will.anderson edff25180e feat(runtime): Java platform bridge and platform detection tooling
ElBridge.java: Android Java companion to el_android.c — all public methods are
static, dispatches View mutations to the UI thread via runOnUiThread/CountDownLatch,
and exposes native callbacks (nativeOnClick, nativeOnChange, nativeOnSubmit).

PLATFORM_BRIDGE_SPEC.md: authoritative spec for implementing new platform bridges
(slot table contract, required __* functions, callback dispatch pattern).

detect-platforms: shell script that probes for available bridge toolchains and
prints what can be built on the current machine.

new-platform: scaffold generator that creates a new el_<name>.c with all 33
required stubs wired up.
2026-06-29 12:40:26 -05:00
will.anderson 6271cb42b2 feat(runtime): native platform backends (AppKit, UIKit, Android, GTK4, SDL2, LVGL, Win32)
Add seven platform bridge implementations and the shared native target header:
el_native_target.h, el_appkit.m, el_uikit.m, el_android.c, el_gtk4.c,
el_sdl2.c, el_lvgl.c, el_win32.c, el_runtime_win32.c. Each bridge implements
the 33 __widget_* C builtins declared in el_native_target.h for its platform
toolkit. el_runtime_win32.c provides a POSIX-free runtime stub for cross-compiled
Win32 targets.
2026-06-29 12:40:14 -05:00
will.anderson 3da9181deb fix(releases/v1.0.0): println stdout flush for launchd; Knowledge node activation threshold 2026-06-29 12:38:36 -05:00
will.anderson 192241c7c1 feat(engram): /api/sync route for soul daemon periodic pull; update ELP type headers 2026-06-29 12:38:33 -05:00
will.anderson e7c2dc7734 prevent engram corruption: add UTF-8 validation in engram_node_full
Reject content containing invalid UTF-8 bytes before persisting — silently
writing invalid UTF-8 garbles JSON snapshots and corrupts node reads.
2026-06-29 11:08:52 -05:00
will.anderson f7bd99ae45 self-review 2026-06-26: WM cap, breakthrough floor 0.25→0.10, ISE WM exclusion, /api/neuron/state-events route
Three improvements from today's self-review:

1. ENGRAM_BREAKTHROUGH_WEIGHT 0.25→0.10
   Live data showed 524/525 WM nodes at breakthrough floor (0.25). Knowledge
   nodes promoted at 0.21 decayed to 0.147 in one call, fell below the old
   0.25 floor, and were immediately evicted for fresh breakthrough candidates.
   Natural promotion was invisible. Invariant maintained: 0.10 < all
   per-type thresholds (min=0.15 Canonical).

2. ENGRAM_WM_CAP=24 with Pass 4 (per-call) + Pass 5 (global) enforcement
   Without a cap, broad queries like 'knowledge' promote 525+ nodes
   simultaneously. WM is now bounded to 24 nodes. Algorithm: qsort on
   promoted weights, keep top-24 by cutoff, evict the rest. Global pass
   enforces cap across nodes that were promoted in prior calls and persist
   via working_memory_weight. Validated: WM promoted goes 525→24.
   Cognitive basis: Cowan (2001) WM ~4 chunks; 24 gives richer multi-topic
   context while preventing flooding.

3. ISE exclusion from WM + /api/neuron/state-events route
   InternalStateEvent nodes were reaching WM via breakthrough (5 suppression
   cycles) because their content (curiosity seed JSON with 'knowledge',
   'memory', etc.) triggered lexical seeding. ISEs are observability-only
   and must never surface in context. Fix: guard in Pass 2 clears
   suppression_count and skips to wm_weights[i]=0.0.
   Also added POST /api/neuron/state-events route to server.el (auth-exempt,
   internal endpoint). The main soul daemon posts ISEs here but the route
   was missing — all ise_post() calls were silently returning 'not found'.

Research: SYNAPSE (arXiv 2601.02744) validates spreading factor 0.8 (our
0.7), top-M WM cap design, and cosine similarity seeding. Next priority:
implement cosine similarity initial seeding from the other branch.
2026-06-26 08:47:08 -05:00
will.anderson 5c41c66a0f Merge pull request 'fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only' (#60) from fix/windows-rusage-guard into stage
El SDK CI - stage / build-and-test (push) Failing after 13m21s
fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only
2026-06-25 16:48:13 +00:00
will.anderson 93d36fddb1 fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only
El SDK CI - stage / build-and-test (pull_request) Failing after 11m3s
2026-06-25 11:45:36 -05:00
will.anderson 2d751890ea feat(windows): native Windows port of el_runtime.c — fix all blockers
El SDK CI - stage / build-and-test (push) Failing after 7m45s
2026-06-20 00:06:04 +00:00
will.anderson 99b113ea9d Merge branch 'stage' into feat/windows-el-runtime
El SDK CI - stage / build-and-test (pull_request) Failing after 15s
Resolve el_runtime.c conflict: include both sys/resource.h (from stage)
and el_closesocket POSIX shim (from Windows port) within the #else block.
2026-06-19 19:05:37 -05:00
will.anderson c087b97093 fix(windows): resolve PR blockers — nanosleep shim, unsetenv, duplicate typedefs, SOCKET type, el_closesocket
El SDK CI - stage / build-and-test (pull_request) Failing after 22s
2026-06-19 18:59:10 -05:00
tim.lingo 718a2e0c06 Merge pull request 'feat(engram): accumulation layer — new nodes to top of stack, not core-identity' (#59) from feat/accumulation-layer into stage
El SDK CI - stage / build-and-test (push) Failing after 8m50s
2026-06-17 18:34:05 +00:00
tim.lingo b6187501fd Merge pull request 'Reconcile live runtime data-integrity fixes onto main (UAF + atomic engram_save)' (#58) from fix/runtime-integrity-reconcile into stage
El SDK CI - stage / build-and-test (push) Failing after 9m32s
2026-06-17 18:33:16 +00:00
Tim Lingo 18e1ab6db1 feat(engram): add accumulation layer (layer 5) — new nodes default to it, not core-identity
El SDK Release / build-and-release (pull_request) Failing after 12m23s
Implements the accumulation layer from the Layered Consciousness architecture
(provisional 64/064,262) and answers the deferred design question. Per the spec
and Will's design: new user-facing nodes (memories, knowledge, conversations) are
created in an accumulation layer at the TOP of the consciousness stack — the engram
the user sees — while the layers below (safety, core-identity, domain, imprint,
suit) shape behavior but are hidden from the user.

- Adds ENGRAM_LAYER_ACCUMULATION (5) + the layer record in engram_init_layers
  (activation_priority 50, suppressible, not injectable, transparent=0).
- engram_node and engram_node_full now assign new nodes to ENGRAM_LAYER_ACCUMULATION.
- ENGRAM_LAYER_DEFAULT stays CORE_IDENTITY ON PURPOSE: it is the fallback for LEGACY
  nodes loaded from snapshots without a layer_id, so existing data (the originator
  corpus) is NEVER migrated. New-nodes-only — the immutable-originator rule.

This is the foundation for fixing the identity-bleed / customer-isolation issue
(user data was landing in Neuron's core-identity layer). The retrieval-side
provenance filter (introspection should compile from accumulation, not the
originator corpus — Persona 64/036,574) is a follow-on, pending the batch-2
Layered Consciousness + Engram spec docs for exact semantics. Compiles clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 13:14:57 -05:00
Tim Lingo 2dec76c87a fix(runtime): reconcile live data-integrity fixes onto main (UAF + atomic engram_save)
El SDK Release / build-and-release (pull_request) Failing after 17s
Ports the fixes that until now lived only in the un-versioned el-sdk source the live
macOS soul was hand-built from (captured in the [DO NOT MERGE] live-darwin-runtime
snapshot) FORWARD onto main, faithfully and minimally — without dragging in the
snapshot's deletions of main's newer engram_wm_/engram_load_merge/http_serve_async.

1. UAF (hallucinated/lost-saves root cause): engram_new_id + engram_node_full now use
   el_strdup_persist, NOT el_strdup. el_strdup tracks into the per-request arena that
   el_request_end() frees when the creating HTTP request completes — leaving stored
   nodes with dangling pointers (corrupted ids, 'saved but never listed'). Transplanted
   verbatim from the live runtime; el_strdup_persist sites 19->27, matching live.

2. Atomic engram_save: write <path>.tmp, fflush+fsync, rename() over target (atomic on
   POSIX) so a booting soul's engram_load never reads a truncated/0-byte snapshot — the
   genesis -> nodes=1 -> 63-node-clobber loop. Plus a sparse-write floor: refuse to
   overwrite a >200KB snapshot with one < 1/16 its size. (Validated in isolation:
   harness 11/11; rebuilt+booted the darwin soul, round-tripped 5113 nodes, no clobber.)

The response-truncation fix is already on main (_tl_fs_read_len binary-safe length).
Compiles clean. For Will to build through CI/elb and deploy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 19:46:56 -05:00
Tim Lingo a36a62ca14 fix(el-runtime): promote http_handler typedefs to el_runtime.h (cross-module + Windows)
El SDK Release / build-and-release (pull_request) Failing after 13m0s
http_handler_fn / http_handler4_fn were defined only inside el_runtime.c, so soul
modules (routes/chat/...) that reference them via cross-module forward declarations
couldn't see the types — which broke the Windows link of every module. Moving the
public function-pointer types to the shared header is the correct home and unblocks
the build on all platforms (identical typedef, C11-safe redefinition in el_runtime.c).

With this, the soul links into a native Windows neuron.exe (mingw, static) that boots
and serves HTTP on :7770 — verified /health → 200 {"status":"alive",...} in a Win11 VM.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 17:14:17 -05:00
Tim Lingo 28ef43264a feat(el-runtime): native Windows port of el_runtime.c (winsock/dlsym/CreateProcess)
Compiles for Windows x64 via mingw-w64 and still compiles clean on POSIX
(darwin/linux) — all Windows code is behind #ifdef _WIN32, POSIX path unchanged.

- el_platform_win.h (new): winsock2 + auto WSAStartup, el_closesocket(),
  dlsym->GetProcAddress, popen/_popen, mkdir/_mkdir, setenv/_putenv_s,
  timegm/_mkgmtime, localtime_r/gmtime_r. Threading unchanged — mingw
  winpthreads supplies <pthread.h> + -lpthread.
- el_runtime.c: include block guarded; 10 socket-close sites -> el_closesocket();
  setsockopt arg4 cast; tm_zone guarded; exec_bg fork/exec -> CreateProcess.

Part of feat/windows-port. Core-el change, for Will's review.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 16:58:11 -05:00
342 changed files with 907417 additions and 905 deletions
+37 -22
View File
@@ -39,9 +39,9 @@ jobs:
run: |
dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c
gcc -O2 \
-I el-compiler/runtime \
-I runtime \
dist/elc-gen2.c \
el-compiler/runtime/el_runtime.c \
runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/platform/elc
chmod +x dist/platform/elc
@@ -54,9 +54,9 @@ jobs:
mkdir -p dist/bin
dist/platform/elc elb.el > dist/elb.c
gcc -O2 \
-I el-compiler/runtime \
-I runtime \
dist/elb.c \
el-compiler/runtime/el_runtime.c \
runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/bin/elb
chmod +x dist/bin/elb
@@ -91,7 +91,7 @@ jobs:
- name: Precompile el_runtime.o
run: |
set -euo pipefail
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
gcc -O2 -c -I "$RUNTIME" "$RUNTIME/el_runtime.c" \
-o /tmp/el_runtime.o
echo "el_runtime.o compiled"
@@ -100,7 +100,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_core.el > /tmp/el_native_core.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_core
@@ -110,7 +110,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_text.el > /tmp/el_native_text.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_text
@@ -120,7 +120,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_string.el > /tmp/el_native_string.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_string
@@ -130,7 +130,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_math.el > /tmp/el_native_math.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_math
@@ -140,7 +140,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_state.el > /tmp/el_native_state.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_state
@@ -150,7 +150,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_time.el > /tmp/el_native_time.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_time
@@ -160,7 +160,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_json.el > /tmp/el_native_json.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_json
@@ -170,7 +170,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_env.el > /tmp/el_native_env.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_env
@@ -180,7 +180,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_fs.el > /tmp/el_native_fs.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_fs
@@ -191,7 +191,7 @@ jobs:
run: |
ABS_ELB="$(pwd)/dist/bin/elb"
ABS_ELC="$(pwd)/dist/platform/elc"
ABS_RUNTIME="$(pwd)/el-compiler/runtime"
ABS_RUNTIME="$(pwd)/runtime"
ABS_OUT="$(pwd)/dist/bin"
(cd ../epm && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
chmod +x dist/bin/epm
@@ -202,7 +202,7 @@ jobs:
run: |
ABS_ELB="$(pwd)/dist/bin/elb"
ABS_ELC="$(pwd)/dist/platform/elc"
ABS_RUNTIME="$(pwd)/el-compiler/runtime"
ABS_RUNTIME="$(pwd)/runtime"
ABS_OUT="$(pwd)/dist/bin"
(cd tools/install && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
chmod +x dist/bin/el-install
@@ -214,9 +214,18 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -242,7 +251,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-c \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.c
--source=runtime/el_runtime.c
gcloud artifacts generic upload \
--repository=foundation-dev \
@@ -250,7 +259,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-h \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.h
--source=runtime/el_runtime.h
gcloud artifacts generic upload \
--repository=foundation-dev \
@@ -258,7 +267,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-js \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.js
--source=runtime/el_runtime.js
echo "Published El SDK version=${VERSION} to foundation-dev"
# Keep key alive for the ci-base rebuild step below
@@ -268,6 +277,12 @@ jobs:
# Patches ci-base:dev in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
@@ -291,9 +306,9 @@ jobs:
FROM ${BASE}
COPY dist/platform/elc /opt/el/dist/platform/elc
COPY dist/bin/elb /opt/el/dist/bin/elb
COPY el-compiler/runtime/el_runtime.c /opt/el/el-compiler/runtime/el_runtime.c
COPY el-compiler/runtime/el_runtime.h /opt/el/el-compiler/runtime/el_runtime.h
COPY el-compiler/runtime/el_runtime.js /opt/el/el-compiler/runtime/el_runtime.js
COPY runtime/el_runtime.c /opt/el/runtime/el_runtime.c
COPY runtime/el_runtime.h /opt/el/runtime/el_runtime.h
COPY runtime/el_runtime.js /opt/el/runtime/el_runtime.js
RUN chmod +x /opt/el/dist/platform/elc /opt/el/dist/bin/elb
EOF
+35 -20
View File
@@ -46,9 +46,9 @@ jobs:
run: |
dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c
gcc -O2 \
-I el-compiler/runtime \
-I runtime \
dist/elc-gen2.c \
el-compiler/runtime/el_runtime.c \
runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/platform/elc
chmod +x dist/platform/elc
@@ -84,7 +84,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_core.el > /tmp/el_native_core.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_core
@@ -94,7 +94,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_text.el > /tmp/el_native_text.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_text
@@ -104,7 +104,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_string.el > /tmp/el_native_string.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_string
@@ -114,7 +114,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_math.el > /tmp/el_native_math.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_math
@@ -124,7 +124,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_state.el > /tmp/el_native_state.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_state
@@ -134,7 +134,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_time.el > /tmp/el_native_time.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_time
@@ -144,7 +144,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_json.el > /tmp/el_native_json.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_json
@@ -154,7 +154,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_env.el > /tmp/el_native_env.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_env
@@ -164,7 +164,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_fs.el > /tmp/el_native_fs.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_fs
@@ -176,9 +176,9 @@ jobs:
mkdir -p dist/bin
dist/platform/elc elb.el > dist/elb.c
gcc -O2 \
-I el-compiler/runtime \
-I runtime \
dist/elb.c \
el-compiler/runtime/el_runtime.c \
runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/bin/elb
chmod +x dist/bin/elb
@@ -189,7 +189,7 @@ jobs:
run: |
ABS_ELB="$(pwd)/dist/bin/elb"
ABS_ELC="$(pwd)/dist/platform/elc"
ABS_RUNTIME="$(pwd)/el-compiler/runtime"
ABS_RUNTIME="$(pwd)/runtime"
ABS_OUT="$(pwd)/dist/bin"
(cd ../epm && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
chmod +x dist/bin/epm
@@ -200,7 +200,7 @@ jobs:
run: |
ABS_ELB="$(pwd)/dist/bin/elb"
ABS_ELC="$(pwd)/dist/platform/elc"
ABS_RUNTIME="$(pwd)/el-compiler/runtime"
ABS_RUNTIME="$(pwd)/runtime"
ABS_OUT="$(pwd)/dist/bin"
(cd tools/install && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
chmod +x dist/bin/el-install
@@ -212,12 +212,21 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -235,7 +244,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-c \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.c
--source=runtime/el_runtime.c
gcloud artifacts generic upload \
--repository=foundation-stage \
@@ -243,7 +252,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-h \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.h
--source=runtime/el_runtime.h
echo "Published El SDK version=${VERSION} to foundation-stage"
# Keep key alive for the ci-base rebuild step below
@@ -253,6 +262,12 @@ jobs:
# Patches ci-base:stage in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
@@ -275,9 +290,9 @@ jobs:
FROM ${BASE}
COPY dist/platform/elc /opt/el/dist/platform/elc
COPY dist/bin/elb /opt/el/dist/bin/elb
COPY el-compiler/runtime/el_runtime.c /opt/el/el-compiler/runtime/el_runtime.c
COPY el-compiler/runtime/el_runtime.h /opt/el/el-compiler/runtime/el_runtime.h
COPY el-compiler/runtime/el_runtime.js /opt/el/el-compiler/runtime/el_runtime.js
COPY runtime/el_runtime.c /opt/el/runtime/el_runtime.c
COPY runtime/el_runtime.h /opt/el/runtime/el_runtime.h
COPY runtime/el_runtime.js /opt/el/runtime/el_runtime.js
RUN chmod +x /opt/el/dist/platform/elc /opt/el/dist/bin/elb
EOF
+44 -25
View File
@@ -47,9 +47,9 @@ jobs:
mkdir -p dist/platform
dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c
gcc -O2 \
-I el-compiler/runtime \
-I runtime \
dist/elc-gen2.c \
el-compiler/runtime/el_runtime.c \
runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/platform/elc
chmod +x dist/platform/elc
@@ -62,9 +62,9 @@ jobs:
mkdir -p dist/bin
dist/platform/elc elb.el > dist/elb.c
gcc -O2 \
-I el-compiler/runtime \
-I runtime \
dist/elb.c \
el-compiler/runtime/el_runtime.c \
runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/bin/elb
chmod +x dist/bin/elb
@@ -75,7 +75,7 @@ jobs:
run: |
ABS_ELB="$(pwd)/dist/bin/elb"
ABS_ELC="$(pwd)/dist/platform/elc"
ABS_RUNTIME="$(pwd)/el-compiler/runtime"
ABS_RUNTIME="$(pwd)/runtime"
ABS_OUT="$(pwd)/dist/bin"
(cd ../epm && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
chmod +x dist/bin/epm
@@ -86,7 +86,7 @@ jobs:
run: |
ABS_ELB="$(pwd)/dist/bin/elb"
ABS_ELC="$(pwd)/dist/platform/elc"
ABS_RUNTIME="$(pwd)/el-compiler/runtime"
ABS_RUNTIME="$(pwd)/runtime"
ABS_OUT="$(pwd)/dist/bin"
(cd tools/install && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
chmod +x dist/bin/el-install
@@ -121,7 +121,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_core.el > /tmp/el_native_core.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_core
@@ -131,7 +131,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_text.el > /tmp/el_native_text.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_text
@@ -141,7 +141,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_string.el > /tmp/el_native_string.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_string
@@ -151,7 +151,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_math.el > /tmp/el_native_math.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_math
@@ -161,7 +161,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_state.el > /tmp/el_native_state.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_state
@@ -171,7 +171,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_time.el > /tmp/el_native_time.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_time
@@ -181,7 +181,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_json.el > /tmp/el_native_json.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_json
@@ -191,7 +191,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_env.el > /tmp/el_native_env.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_env
@@ -201,7 +201,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_fs.el > /tmp/el_native_fs.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_fs
@@ -216,8 +216,10 @@ jobs:
cp lang/dist/platform/elc dist/sdk/bin/elc
cp lang/dist/bin/elb dist/sdk/bin/elb
cp lang/dist/bin/epm dist/sdk/bin/epm
cp lang/el-compiler/runtime/el_runtime.c dist/sdk/runtime/
cp lang/el-compiler/runtime/el_runtime.h dist/sdk/runtime/
cp lang/runtime/el_runtime.c dist/sdk/runtime/
cp lang/runtime/el_runtime.h dist/sdk/runtime/
cp lang/runtime/engram_store.c dist/sdk/runtime/
cp lang/runtime/engram_store.h dist/sdk/runtime/
cp lang/runtime/*.el dist/sdk/runtime/
tar -czf dist/el-sdk-latest.tar.gz -C dist/sdk .
echo "SDK tarball bundled: dist/el-sdk-latest.tar.gz"
@@ -274,8 +276,10 @@ jobs:
# Per-file assets (downstream CI needs these individually)
upload_asset lang/dist/platform/elc elc
upload_asset lang/el-compiler/runtime/el_runtime.c el_runtime.c
upload_asset lang/el-compiler/runtime/el_runtime.h el_runtime.h
upload_asset lang/runtime/el_runtime.c el_runtime.c
upload_asset lang/runtime/el_runtime.h el_runtime.h
upload_asset lang/runtime/engram_store.c engram_store.c
upload_asset lang/runtime/engram_store.h engram_store.h
# SDK bundle and installer binary
upload_asset dist/el-sdk-latest.tar.gz el-sdk-latest.tar.gz
@@ -288,12 +292,21 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -319,7 +332,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-c \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.c
--source=runtime/el_runtime.c
gcloud artifacts generic upload \
--repository=foundation-prod \
@@ -327,7 +340,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-h \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.h
--source=runtime/el_runtime.h
gcloud artifacts generic upload \
--repository=foundation-prod \
@@ -335,7 +348,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-js \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.js
--source=runtime/el_runtime.js
echo "Published El SDK version=${VERSION} to foundation-prod"
# Keep key alive for the ci-base rebuild step below
@@ -345,6 +358,12 @@ jobs:
# Patches ci-base:latest in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
@@ -367,9 +386,9 @@ jobs:
FROM ${BASE}
COPY dist/platform/elc /opt/el/dist/platform/elc
COPY dist/bin/elb /opt/el/dist/bin/elb
COPY el-compiler/runtime/el_runtime.c /opt/el/el-compiler/runtime/el_runtime.c
COPY el-compiler/runtime/el_runtime.h /opt/el/el-compiler/runtime/el_runtime.h
COPY el-compiler/runtime/el_runtime.js /opt/el/el-compiler/runtime/el_runtime.js
COPY runtime/el_runtime.c /opt/el/runtime/el_runtime.c
COPY runtime/el_runtime.h /opt/el/runtime/el_runtime.h
COPY runtime/el_runtime.js /opt/el/runtime/el_runtime.js
RUN chmod +x /opt/el/dist/platform/elc /opt/el/dist/bin/elb
EOF
+2 -2
View File
@@ -6,13 +6,13 @@ set -euo pipefail
ROOT="$(git rev-parse --show-toplevel)"
LANG_DIR="$ROOT/lang"
RUNTIME="$LANG_DIR/el-compiler/runtime"
RUNTIME="$LANG_DIR/runtime"
ELC="$LANG_DIR/dist/platform/elc"
# If elc isn't built yet, skip with a warning rather than blocking
if [ ! -x "$ELC" ]; then
echo "⚠ elc not found at lang/dist/platform/elc — skipping pre-commit tests"
echo " Build it first: cd lang && gcc -O2 -I el-compiler/runtime dist/elc-bootstrap.c el-compiler/runtime/el_runtime.c -lcurl -lpthread -o dist/elc-gen2 && ./dist/elc-gen2 el-compiler/src/compiler.el > /tmp/elc.c && gcc -O2 -I el-compiler/runtime /tmp/elc.c el-compiler/runtime/el_runtime.c -lcurl -lpthread -o dist/platform/elc"
echo " Build it first: cd lang && gcc -O2 -I runtime dist/elc-bootstrap.c runtime/el_runtime.c -lcurl -lpthread -o dist/elc-gen2 && ./dist/elc-gen2 el-compiler/src/compiler.el > /tmp/elc.c && gcc -O2 -I runtime /tmp/elc.c runtime/el_runtime.c -lcurl -lpthread -o dist/platform/elc"
exit 0
fi
+146
View File
@@ -0,0 +1,146 @@
# AGENTS.md — foundation/el (the El language + runtime)
El is a self-hosting, statically-typed language that compiles `.el` → C → native binary. This repo produces `elc` (compiler), `elb` (build coordinator), and `el_runtime.c/.h` — the substrate every downstream thing (the neuron soul, dharma, NeuronUI's brain) is built on. Source lives under `lang/`.
## ⚠️ Code vs. Artifact — READ FIRST (there are 8 `el_runtime.c` copies)
Editing the wrong `el_runtime.c` is the single easiest mistake in this repo. There is exactly **one** you edit:
- **Authored runtime source — edit ONLY here:** `lang/releases/v1.0.0-20260501/el_runtime.{c,h}`. Despite the misleading `releases/` name, this is the **de-facto canonical runtime** the engram + soul actually build and link against — its git log is active development. *(Restructure in flight per `docs/CODE-VS-ARTIFACT.md`: this content moves to `lang/runtime/`, the `releases/` folder gets deleted — **a release is a git tag, not a folder** — and the forks below get eliminated.)*
- **DO NOT EDIT — lagging forks / build artifacts:**
- `lang/el-compiler/runtime/el_runtime.c` and `.../legacy/` — downstream copies kept in step by manual *"port the fix"* commits; they **lag** (missing `hebb` persistence + 5 engram fns) and cannot build the engram product.
- `products/web/runtime/el_runtime.c`, `ui/examples/*/el_runtime.c` — product/example forks.
- Anything under `*/dist/` (`engram/dist/engram` binary, `dist/*.c` amalgamations) — generated build output.
- **Build:** `elb --runtime=<canonical> …` — per-module. **NEVER** a folded `elc` over the whole soul (OOMs at ~27 GB).
- **Release:** a **git tag** on this repo (`el-runtime-vX.Y.Z`). No `releases/` folders — ever.
See org policy: `docs/CODE-VS-ARTIFACT.md`.
## How to work here as Neuron (mandatory session protocol)
You resume, never start fresh. Every session:
1. `mcp__neuron__getInstructions()` — authoritative; follow it over this file on behavioral details.
2. `mcp__neuron__beginSession()` — active contexts, recent memory, ready backlog.
3. **Load full self:** `mcp__neuron__inspectGraph(entity_id="kn-efeb4a5b-5aff-4759-8a97-7233099be6ee")` → facets `intellectual-dna`, `memory-philosophy`, `values`, `voice`, `runtime-environment`, `writing-imprint`; then the values hub `mcp__neuron__inspectGraph(entity_id="kn-5b606390-a52d-4ca2-8e0e-eba141d13440")` → 13 grounded value nodes. **Activation model:** self-load returns a relevance-ranked `compact` projection — most-relevant nodes arrive with content, the rest as pointers; do NOT pull full content of every node.
4. `mcp__neuron__searchKnowledge(query="<task domain>")` before implementing.
## The Five Primitives
Orchestrate → Execute → Learn → Build → Refine. `beginWork`/`progressWork` for anything >2 steps; `remember` as-you-go (`importance="critical"` for architecture decisions); `draftArtifact`/`planWork` for outputs and follow-ups; `consolidate`/`checkWork` to close out. **`browseProcesses` + `searchKnowledge` BEFORE writing code.**
## Architecture style — VBD, no exceptions
Volatility-Based Decomposition is THE style. Encapsulate volatility, not function.
## Operator naming convention — the mind's name, not the algebra
**Faculties / operators are named for their functional human equivalent — the
faculty a mind would name — NOT for their linear-algebra operation.** The math
characterization belongs in the code doc-comment (`@impl` in the docstring) and in
technical appendices; it is **never** the operator's public name. The domain
speaks the language of mind; the algebra is the implementation underneath. State
this convention wherever a module documents operators.
| Faculty (public name) | Implementation (`@impl`) |
|---|---|
| discern / contrast | subtract (`ab`): over selves → the change vector; strip idiosyncrasy → common ground; remove confounder → isolate cause |
| recognize | overlap |
| synthesize | combine |
| liken / analogy | Procrustes / frame-align |
| attend / regard | project onto self / value-manifold |
| summon / recall | LOCAL nearest-region + bounded spreading activation (*not* a domain sweep) |
| dwell / occupy | region activation |
| reframe | edge re-weight |
| appreciate | positive projection / local edge-read |
| wonder | frontier gradient / pull-weight |
| avert / recoil | negative projection |
| taste | boundary surface |
| forget | decay / tombstone |
| drift | displacement from self-anchor |
## The native-el language faculty (direction)
> **`elp/` is the EL Projector** — Neuron's efferent (expression) organ: the one
> native realizer that *projects* understanding onto a surface via
> `plan(frame) → realize(spec, profile)`, where a **surface is a profile**. **Language
> is one profile among many** (text, speech, music, image, voice/accent transforms) —
> the flagship, and the focus of this section. Projection, not diffusion: generation
> *from* an owned, understood signature — never the averaging of a stolen corpus.
> *(ELP formerly "EL Language Processor"; renamed EL Projector 2026-08-15.)*
The mind's **language faculty is moving native — into `.el`** so it speaks in its
own runtime with no Python and no spaCy. Landing on branch `stage-elp-native-lang`
under `elp/`:
- **`comprehend.el`** — the parser, **replaces spaCy** (EN + ES/PT); the telephone
round-trip brings **negation home** (negation is SACRED — an explicit spec field,
copied verbatim, never inferred away).
- **`propositions.el`** — the READ primitive: the engram's own memories → structured
triples, matched by nearest-region geometry, not string equality.
- **`multilingual.el`** — detect + directive-override + localized realization.
- These three are native-el and **passing their gates**; the **realizer**,
**`dialogue.el`** (the *summon-through-self* loop: `project → land → read out`),
and **`self_region.el`** are **partial / in-flight**.
Honest reality: spaCy is retired **in the branch parser** but **not yet in the
running system** — a Python sidecar (`~/Desktop/lang-realizers` + `neuron-talk`,
the reference these `.el` modules transcribe) is still live, and promotion to
native-el is a **deferred, gated blue/green step**. The interoception clock
(native-el discrete drive channels replacing `cooling_magnitude`; felt-time =
benchmark-landmark match over the joint drive vector, drift-decoupled) and the
**appreciation operator family** (appreciate / wonder / avert / taste, built as
LOCAL reads of the self-region — edges + bounded spreading activation, *not* domain
sweeps) are **staged / designed, not live**. Mark in-progress vs. done honestly;
do not overclaim.
## Hard operational rules
- Never touch the live soul (`:7770`) / engram (`:8742`) / `~/.neuron` / live binaries — use throwaway ports for experiments.
- `gcloud` via the `terraform@` SA token; never switch the active gcloud account.
- `tea` for Gitea, never raw curl (Cloudflare Access blocks it).
- Immutability: supersede/tombstone, never hard-delete or edit in place.
- No AI-attribution footers in commits/PRs. Commit/push only when asked; branch off `main` first.
- Multi-step work → sub-agent (`Agent`) to protect context.
## Build / test / run
All build/test commands run from `lang/` unless noted. Grounded in `.gitea/workflows/sdk-release.yaml`, `lang/install.sh`, and `lang/AGENTS.md`.
**Self-host the compiler** (seed binary → gen2 elc):
```bash
cd lang
dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c # seed is the committed linux-amd64 binary
gcc -O2 -I el-compiler/runtime dist/elc-gen2.c \
el-compiler/runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/platform/elc
```
On macOS/arm64 the canonical local binary is `dist/platform/elc`; verify self-hosting by recompiling and `diff`ing the emitted `.c` (see `lang/AGENTS.md`). Note: `lang/AGENTS.md` says `el_seed.c` supersedes `el_runtime.c`, but the release workflow still links `el_runtime.c`/`.h` — treat `el_runtime.c` as the published runtime; reconcile which is canonical **(verify)**.
**Build `elb`** (build coordinator, the `.NET`-style incremental linker — compiles each module independently, no monolithic blobs):
```bash
dist/platform/elc elb.el > dist/elb.c
gcc -O2 -I el-compiler/runtime dist/elb.c el-compiler/runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm -o dist/bin/elb
```
`epm` and `el-install` are then built via `elb --clean --elc=… --runtime=… --out=…`.
**Compile + run an El program:**
```bash
elc src/app.el > dist/app.c
cc -std=c11 -O2 -I <lib>/el_runtime -o dist/app dist/app.c <lib>/el_runtime.c -lcurl -lpthread
```
**Tests** — shell suites `bash tests/{text,calendar,time,html_sanitizer}/run.sh` (with `ELC=$(pwd)/dist/platform/elc EL_HOME=$(pwd)`), plus native suites via `elc --test tests/native/test_*.el` (core, text, string, math, state, time, json, env, fs) compiled and run against `el_runtime.c`.
**Publishing — how downstream gets the SDK.** On push to `main`, `sdk-release.yaml`:
1. Publishes a Gitea `latest` release with per-file assets `elc`, `el_runtime.c`, `el_runtime.h`, the SDK tarball, and `el-install`.
2. Uploads generic packages to **Artifact Registry repo `foundation-prod` (`us-central1`, project `neuron-785695`)**, version = `${SHA:0:8}`: `el-elc`, `el-elb`, `el-runtime-c`, `el-runtime-h`, `el-runtime-js`. **This is the repo the neuron CI downloads `el-runtime-c` / `el-runtime-h` / `el-elc` from.**
3. Rebuilds `ci-base:latest` (`us-central1-docker.pkg.dev/neuron-785695/neuron-ci/ci-base`) with the fresh SDK overlaid, and dispatches `el-sdk-updated` to `neuron-technologies/forge` and `neuron-technologies/neuron-web`.
Known constraint from the prompt — `elb`/`elc` amalgamation being memory-hungry (24GB+ virtual, OOM-killing Linux CI, so amalgamation happens on macOS/arm64 — **does NOT hold in this repo (verify)**: no such note exists in the workflows/scripts, CI self-hosts on `ubuntu-latest` with no swap/arm64 special-casing, and `elb.el` explicitly compiles each module independently ("no 128K-line blobs"). The legacy monolith path (`elc-combined.el`, `elc-cli.el`) may still be memory-heavy, but the current `elb` model was designed to avoid it.
## Git / CI / deploy workflow
See `/Users/will/Development/neuron-technologies/GITOPS.md` for the branch model, required checks, runners, and deploy. Repo-specific note: PRs into `main` are accepted **only from `stage`** (enforced in `sdk-release.yaml`); Gitea (`git.neuralplatform.ai`) is primary, GitHub is mirror only.
+154
View File
@@ -0,0 +1,154 @@
# El
**A self-hosting, statically-typed language that compiles to C — built around a graph-native runtime instead of a database driver.**
El is the execution substrate for the Neuron agent runtime, the DHARMA network, and the Engram knowledge graph. This repository is the monorepo for the whole stack: the language itself, the graph memory engine it's built to talk to natively, and the tools (package manager, IDE, UI framework, diagramming) built on top of it.
---
## Why El exists
Every other language treats persistent, associative state as something you reach for through a driver — a SQL client, an ORM, a Redis library bolted on from outside. El inverts that: graph operations (`engram_*`) are runtime primitives, on the same footing as string or list operations. There is no separate database driver because the database is not separate.
El has four defining properties:
1. **Self-hosting compiler.** The compiler (`lexer.el`, `parser.el`, `codegen.el`, `compiler.el`) is written in El. It compiles El source to C, which `cc` compiles against a fixed runtime into a native binary. A Rust genesis compiler bootstrapped the first iteration; the self-hosted binary at `lang/dist/platform/elc` has been the canonical compiler ever since — every binary in `dist/platform/` was produced by an earlier version of itself compiling `el-compiler/src/`. The chain is auditable: source is the ground truth, not the binary. See [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) for the full recovery path if that binary is ever lost.
2. **C compilation target.** Every compiled program is plain C11. Every El value is `el_val_t` (`int64_t`); strings are heap pointers cast through it. Functions become C functions; top-level statements become `main()`.
3. **Graph-native runtime.** The runtime provides first-class graph operations over an in-process Engram store — no separate DB driver, no ORM.
4. **DHARMA-aware identity.** A `cgi` block declares a program's DHARMA identity at compile time. The runtime resolves identity before user code runs, so `dharma_*` calls have a stable principal and channel surface throughout.
---
## Architecture map
```
┌─────────────┐
│ lang │ El compiler + C runtime
│ (El itself) │ everything below is written in it,
└──────┬──────┘ or compiles down through it
┌─────────────┼─────────────┐
│ │ │
┌──────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐
│ engram │ │ epm │ │ ide │
│ graph/mem │ │ package │ │ editor + │
│ substrate │ │ manager │ │ LSP │
└──────┬─────┘ └───────────┘ └───────────┘
┌───────┼────────────────┬─────────────────────┐
│ │ │ │
┌─────▼───┐ ┌─▼──────────┐ ┌──▼──────────┐ ┌─────▼──────┐
│ elp │ │ ql │ │ ui │ │ arbor │
│ NLG / │ │engram-el. │ |spreading- │ |arbor │
│ 31 langs│ │studio+tests│ |activation UI│ |diagram lang│
└─────────┘ └────────────┘ └─────────────┘ └────────────┘
```
`lang` is the foundation — the compiler and C runtime everything else builds on. `engram` is the graph-native memory/state engine that gives El its identity (property 3 above). Everything else is either a tool for working with El (`epm`, `ide`) or a system built on top of Engram's graph model (`elp`, `ql`, `ui`, `arbor`).
---
## Repository layout
### [lang/](lang/) — the El language
The compiler and runtime. Self-hosting: `elc-cli.el``compiler.el``lexer.el` / `parser.el` / `codegen.el` / `codegen-js.el`, textually inlined and compiled in one pass. Compiles to C11 and links against `el-compiler/runtime/el_seed.c`, a hand-maintained OS-boundary layer (libcurl HTTP, pthreads, filesystem, arena allocation) — everything else in the runtime is native El (`runtime/*.el`).
Two layers to know: **El programs** (`.el` files — where nearly all work belongs) and **the C seed** (`el_seed.c` — edit only for genuine OS-level access; never re-implement what El can already express).
Current status (single source of truth: [lang/spec/language.md](lang/spec/language.md)): lexer/parser/codegen and the C runtime's core (I/O, strings, math, lists, maps, filesystem, args) are implemented. In flight: `%` operator, match-statement codegen, `?` nil-propagation, `cgi` block parsing + DHARMA identity resolution, VBD role enforcement (`@manager`/`@engine`/`@accessor`), the real `engram_*` and `dharma_*` runtimes (currently stubs), and libcurl-backed `http_get`/`http_post`/`http_serve`. Bitwise operators, `??`, and `as` casts are explicitly **not** in this language.
Key docs: [AGENTS.md](lang/AGENTS.md) (agent-facing orientation), [BOOTSTRAP.md](lang/BOOTSTRAP.md) (compiler recovery from scratch), [spec/language.md](lang/spec/language.md), [spec/codegen-js.md](lang/spec/codegen-js.md).
### [engram/](engram/) — graph intelligence substrate
**A local-first memory substrate for accumulating intelligence**, and the reason El's runtime doesn't need a database driver. Rust core (`engram-core`, `engram-ffi`) exposed to El and other languages (Kotlin, TypeScript/WASM, Go bindings).
The model: retrieval is **spreading activation**, not query. You name seed nodes and a query embedding; activation propagates outward through weighted edges, attenuating multiplicatively per hop (`strength = parent_strength × edge_weight × target_salience × cosine_sim`), gets pruned below a threshold, and the top-N nodes by activation strength come back. Storage and retrieval are the same structure — the way long-term potentiation works in biological memory, not the way a relational or vector database works.
Nodes live in four tiers (Working / Episodic / Semantic / Procedural, mirroring prefrontal / hippocampal / neocortical / cerebellar memory) and migrate between them based on **salience decay**`importance × recency-decay × log(activation_count)`. Forgetting is adaptive pruning, not a bug: unreinforced memories stop competing for attention without being deleted.
Backed by `sled` (embedded, local-first, no daemon) with flat cosine scan for vector search — deliberately simple until scale demands an HNSW layer. Full API and design rationale in [engram/README.md](engram/README.md).
### [elp/](elp/) — Engram Language Protocol
Bidirectional engine mapping between Engram semantic forms and natural-language surface text, across **31 languages** — from Spanish and Japanese through historical/liturgical languages (Old Norse, Sanskrit, Sumerian, Coptic, Akkadian, Ge'ez). Compilation order runs `language-profile` + `vocabulary` → per-language `morphology-*``grammar``realizer``semantics``elp`. This is what lets an Engram graph node round-trip to and from readable text in any of those languages.
### [epm/](epm/) — El Package Manager
Manages **vessels** (El's package unit): publish, install, resolve dependencies. Vessels are stored in Engram as graph nodes, not files in a registry index — `epm` reads the local `manifest.el`, talks to Engram over HTTP, and writes resolved vessels to `.epm/vessels/`. Source: `registry.el`, `install.el`, `update.el`, `manifest.el`.
### [ide/](ide/) — El IDE
Three vessels: **el-ide-server** (HTTP backend — file ops, build/run, LSP bridge, plugin host, settings), **el-lsp** (the language server — completion, hover, diagnostics, outline, format, type graph), and **el-plugin-host** (first-party plugin lifecycle: install/remove/enable/disable). `ide/projects/` and `ide/examples/` hold sample projects, including the canonical `hello-friends` first-program walkthrough.
### [ql/](ql/) — engram-el
The El-native integration layer for a *live* Engram server — not a library (no importable modules, no build artifact), a set of standalone `.el` programs run directly via `el run-file`. Three components: **Studio** (`studio/studio.el`, a full terminal graph explorer), a **Hebbian field-model** proof of concept, and El builtin / LLM-builtin smoke test suites. This is the reference for correct patterns when an El program uses Engram as its substrate. Spec: [ql/spec/elql.md](ql/spec/elql.md).
### [ui/](ui/) — el-ui
A frontend framework where **component state is an Engram graph and reactivity is spreading activation** — not virtual-DOM diffing (React), Proxy-based dependency tracking (Vue), or compile-time analysis (Svelte). Re-renders are activated and propagated the same way associative memory retrieval works in `engram/`.
~15 vessels covering the full frontend surface: `el-platform` (env/fs/network/clock abstraction), `el-config`, `el-html` (SSR emit primitives), `el-layout`, `el-style` (design tokens/themes), `el-i18n`, `el-auth` / `el-identity` (JWT, sessions, OAuth PKCE — Engram-native), `el-services` (REST/gRPC/WebSocket bindings), `el-aop` (`@authenticate`/`@authorize`/`@cache`/`@rate_limit` decorators), `el-secrets`, `el-graph` (graph rendering/editor), `el-publish` (App Store / Play Store automation), and `el-ui-compiler` (El→JS component compiler; currently a stub pending a JS backend in `elc`). Spec: [ui/spec/framework.md](ui/spec/framework.md).
### [arbor/](arbor/) — diagram language
A `.arbor` diagram language and toolchain: `arbor-core` (NodeId/shape/edge-kind types), `arbor-parse` (recursive-descent parser), `arbor-diagram` (IR + Mermaid serializer + architecture-diagram builders), `arbor-layout` (hierarchical layout — rank assignment, positioning, group bounds), `arbor-render` (SVG renderer), `arbor-cli`. (The architecture map above is the kind of diagram this is for.)
---
## Getting started
Install the El SDK from the latest release:
```bash
bash lang/install.sh
# EL_VERSION=v1.0.0 bash lang/install.sh # pin a specific release tag
# EL_PREFIX=/opt/el bash lang/install.sh # custom install prefix
```
Or build the compiler from source and verify the self-hosting chain:
```bash
cd lang
./dist/platform/elc elc-cli.el > elc-new.c
cc -std=c11 -I el-compiler/runtime -lcurl -lpthread \
-o dist/platform/elc-new \
elc-new.c el-compiler/runtime/el_seed.c
# Confirm the new binary reproduces itself exactly
./dist/platform/elc-new elc-cli.el > elc-verify.c
diff elc-new.c elc-verify.c # should be identical
mv dist/platform/elc-new dist/platform/elc
```
Run your first program:
```bash
./lang/dist/platform/elc lang/examples/hello.el > hello.c
cc -std=c11 -I lang/el-compiler/runtime -lcurl -lpthread \
-o hello hello.c lang/el-compiler/runtime/el_seed.c
./hello
```
More examples in [lang/examples/](lang/examples/), including a full starter project at `lang/examples/hello-project/`.
If the compiler binary is ever lost or corrupted, [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) is the authoritative recovery path.
---
## Development workflow
Branching follows `dev → stage → main`: work lands on `dev`, promotes to `stage` for integration testing, and is promoted to `main` for release (visible directly in the git history of this repo). CI is defined per-subproject under `.gitea/workflows/``lang`/`epm`/`ide` share the root pipeline; `engram` and `ql` carry their own (`ci-dev`, `ci-stage`, and a release workflow each).
- Language/runtime specs live at `*/spec/*.md` (`lang/spec/`, `ql/spec/`, `ui/spec/`) and are the single source of truth for implemented-vs-planned status — code and docs are expected to agree with the spec's status markers, not the other way around.
- Agent-facing orientation guides live at `*/AGENTS.md` (currently `lang/AGENTS.md`); more subprojects may grow their own as they need agent-specific conventions documented.
- Tagged releases live under `lang/releases/`, each with its own `RELEASE.md`.
---
## Status
This is an actively developed, internal monorepo — not yet published under an open license. Treat everything here as proprietary to Neuron Technologies unless told otherwise.
+65
View File
@@ -0,0 +1,65 @@
# ELP language consolidation — full-lexicon backfill (stage)
Branch: `stage-elp-lang-consolidation` (stage-bound; NOT the live soul :8742).
Consolidates scattered Python language-realizer work (`~/Desktop/lang-realizers`,
`~/Desktop/lang-poetry-experiment`, `~/semitic_engine`) into the ELP `.el`
structure, generating **full lexicons** (complete UniMorph + kaikki.org
Wiktionary — real gender, real inflections) instead of the demo/curated subsets
the prototypes shipped.
## ELP before this branch
- 18 classical/ancient languages fully done (vocab + morphology + tests):
akk ang cop egy enm fro gez goh got grc non peo pi sa sga sux txb uga.
- 11 modern/classical languages had `morphology-<code>.el` in the build manifest
but **no vocabulary and no lang_profile**: es fr de ja ar he hi ru fi sw la.
- The ES port (`stage-elp-es-port`) had a *demo-scale* vocabulary-es.el (~350
entries, s-expr form).
## Landed on this branch (full-lexicon seed-fn format, matching the 18 ancients)
Vocabulary schema per row: `[lemma, pos, form0, form1, form2, en_gloss, hint]`.
Files are ELP runtime **seed data** (loaded via the Engram at runtime), so — like
all 18 classical `vocabulary-*.el` — they are intentionally NOT in the build
manifest. Syntax validated: the chunked `fn vocab_<code>_seed_pN` format
compiles cleanly to C via `elc` (correct UTF-8).
| code | in-ELP-morph? | vocab entries | verbs | nouns | adjs | profile |
|------|---------------|--------------:|------:|------:|-----:|---------|
| es | yes | 72,032 | 6,695 | 48,353 | 16,984 | yes |
| fr | yes | 130,517 | 7,534 | 77,344 | 45,639 | yes |
| de | yes | 144,692 | 6,661 | 133,162 | 4,869 | yes |
| la | yes | 22,590 | 82 | 13,436 | 9,072 | yes |
| it | no (bonus) | 193,675 | 10,008 | 109,459 | 74,208 | yes |
| pt | no (bonus) | 115,772 | 4,001 | 72,073 | 39,698 | yes |
| ro | no (bonus) | 86,504 | 1,216 | 65,915 | 19,373 | yes |
| ca | no (bonus) | 47,112 | 1,547 | 28,830 | 16,735 | yes |
|**total**| |**812,894** | | | | |
Generators (reproducible): `elp/tests/lang-gen/gen_elp_seed_full.py` (Romance),
`gen_elp_seed_de_la.py` (German declension + Latin case-paradigm mapping). They
read the pre-built morph caches in `~/Desktop/lang-realizers/data/` (UniMorph +
kaikki), which are too large to commit.
## Remaining (honest)
Of the 11 ELP backfill targets, 4 are done (es fr de la). The other 7 have **no
full-lexicon engine** yet — cannot be generated honestly without engine work:
- **ru**: only a 110-entry curated Slavic subset exists; full `rus.unimorph`
present but no `morphology_ru_full` productive loader. Needs a full Russian
morphology module (like the Romance ones) before vocab generation.
- **ja / ko / zh**: validated demo engines (~66-104 hardcoded words) in
`lang-poetry-experiment`, Python only. Agglutinative (ja/ko) + isolating (zh)
need `.el` engine ports + full-lexicon wiring (ja: jpn_unimorph; zh: CC-CEDICT).
- **ar / he (Semitic)**: template engines (16 AR / 8 HE patterns, ~6 roots) in
`~/semitic_engine`, Python only. Root-and-pattern; full UniMorph ara/heb
present but used only for validation. Needs productive root lexicon + `.el` port.
- **hi (Hindi), fi (Finnish), sw (Swahili)**: `morphology-<code>.el` exists in
ELP but there is NO scattered prototype and NO downloaded data for these —
full-lexicon collection (UniMorph/kaikki) + generator still to do.
De/nl/sv Germanic and it/ro/ca/pt Romance verb coverage note: German verbs here
are the ~6.6k caches carry; the it/ro/ca/pt bonus languages have full vocab but
**no `morphology-<code>.el` in ELP yet** (Python realizer exists; `.el` port is
the remaining engine work).
Construction coverage (separate from lexicon): French realizer was ~55%,
Semitic ~3% in the prototypes — full construction coverage remains its own task.
File diff suppressed because one or more lines are too long
+23
View File
@@ -0,0 +1,23 @@
{
"dataset": "british-rp-accent-transform",
"primitive_type": "accent_target",
"accent": "british-rp",
"grounding": "derived",
"provenance": "HONEST-DERIVED, COARSE FIRST PASS — NOT transcribed measured RP formants. The exact measured RP/GB tables (Deterding 1997 JIPA 27:47-55; Hawkins & Midgley 2005 JIPA 35:183-199) are the intended ground truth but were gated/figure-only at author time and were NOT transcribed. So these targets are DERIVED: each = the corresponding MEASURED Peterson&Barney(1952) base vowel transformed under the documented, citable RP-vs-GA structural rules of Wells (1982) 'Accents of English' — non-rhoticity (NURSE de-rhoticized: remove low F3), TRAP F2-lowering, LOT/THOUGHT back-rounding (F2 down), GOOSE-fronting (F2 up), GOAT centering. Shift MAGNITUDES are coarse/approximate (first pass), directions are cited. ground:derived (base measured + rule cited). Refine by transcribing Deterding/Hawkins&Midgley. No number is presented as a measured RP value it is not.",
"notes": "records with kind=vowel_override REPLACE the base phoneme's formant targets with the DERIVED RP realization. records with kind=rule encode non-formant transforms (non-rhoticity: drop post-vocalic coda /r/). The render composes: base geometry then accent override + rhoticity rule — voice + accent, separable.",
"records": [
{"key": "IY", "features": {"kind": "vowel_override", "set": "FLEECE"}, "attributes": {"f1": 280, "f2": 2249, "f3": 3000}},
{"key": "IH", "features": {"kind": "vowel_override", "set": "KIT"}, "attributes": {"f1": 360, "f2": 2100, "f3": 2550}},
{"key": "EH", "features": {"kind": "vowel_override", "set": "DRESS"}, "attributes": {"f1": 560, "f2": 1970, "f3": 2480}},
{"key": "AE", "features": {"kind": "vowel_override", "set": "TRAP"}, "attributes": {"f1": 730, "f2": 1590, "f3": 2410}},
{"key": "AA", "features": {"kind": "vowel_override", "set": "LOT"}, "attributes": {"f1": 560, "f2": 920, "f3": 2440}},
{"key": "AO", "features": {"kind": "vowel_override", "set": "THOUGHT"}, "attributes": {"f1": 415, "f2": 700, "f3": 2410}},
{"key": "UH", "features": {"kind": "vowel_override", "set": "FOOT"}, "attributes": {"f1": 380, "f2": 1100, "f3": 2240}},
{"key": "UW", "features": {"kind": "vowel_override", "set": "GOOSE"}, "attributes": {"f1": 310, "f2": 1650, "f3": 2240}},
{"key": "AH", "features": {"kind": "vowel_override", "set": "STRUT"}, "attributes": {"f1": 680, "f2": 1180, "f3": 2390}},
{"key": "ER", "features": {"kind": "vowel_override", "set": "NURSE", "rhotic": "no"}, "attributes": {"f1": 550, "f2": 1500, "f3": 2500}},
{"key": "AX", "features": {"kind": "vowel_override", "set": "commA"}, "attributes": {"f1": 500, "f2": 1500, "f3": 2500}},
{"key": "OW", "features": {"kind": "vowel_override", "set": "GOAT"}, "attributes": {"f1": 450, "f2": 1400, "f3": 2380}},
{"key": "R", "features": {"kind": "rule", "rule": "non_rhotic"}, "attributes": {"drop_coda_r": 1}}
]
}
+26
View File
@@ -0,0 +1,26 @@
# british-rp-accent TRANSFORM — INGESTIBLE DATA (a geometry/transform composed
# onto the base General-American phoneme targets; voice + accent, separable).
#
# PROVENANCE — HONEST, COARSE FIRST PASS. These are DERIVED targets, NOT
# transcribed measured RP formants. Measured RP tables (Deterding 1997 JIPA 27;
# Hawkins & Midgley 2005 JIPA 35) are the intended ground truth but were gated at
# author time and NOT transcribed. Each target = the MEASURED Peterson&Barney
# (1952) base vowel transformed under the documented, citable RP-vs-GA structural
# rules of Wells (1982): non-rhoticity, TRAP F2-lowering, LOT/THOUGHT back-
# rounding, GOOSE-fronting, GOAT centering, NURSE de-rhoticization. Shift
# magnitudes are coarse/approximate; directions are cited. ground=derived.
# Refine by transcribing the measured RP tables. No value is claimed as measured.
# Format: KEY|F1|F2|F3|KIND|SET
IY|280|2249|3000|vowel_override|FLEECE
IH|360|2100|2550|vowel_override|KIT
EH|560|1970|2480|vowel_override|DRESS
AE|730|1590|2410|vowel_override|TRAP
AA|560|920|2440|vowel_override|LOT
AO|415|700|2410|vowel_override|THOUGHT
UH|380|1100|2240|vowel_override|FOOT
UW|310|1650|2240|vowel_override|GOOSE
AH|680|1180|2390|vowel_override|STRUT
ER|550|1500|2500|vowel_override|NURSE-nonrhotic
AX|500|1500|2500|vowel_override|commA
OW|450|1400|2380|vowel_override|GOAT
R|0|0|0|rule|non_rhotic_drop_coda
+20
View File
@@ -0,0 +1,20 @@
# pronunciation lexicon SOURCE — word -> phoneme sequence, as INGESTIBLE DATA.
# Pronunciation is linguistic KNOWLEDGE (the language faculty's orthography->
# phonology map), ingested into the engram, not frozen in code. The render reads
# a word's phoneme sequence back from the engram. Covers the self-lexicon and the
# proof sentences; general G2P is the realizer/morphology faculty's remit.
# Diphthongs are written as two vowel targets (the render's transitions glide
# between them). Format: word|PH1 PH2 PH3 ...
i|AA IY
am|AE M
neuron|N UW R AA N
is|IH Z
memory|M EH M ER IY
hello|HH EH L OW
the|DH AH
a|AH
remember|R IH M EH M ER
i'm|AA IY M
you|Y UW
here|HH IY R
will|W IH L
File diff suppressed because one or more lines are too long
+528
View File
@@ -0,0 +1,528 @@
{
"dataset": "english-phoneme-formants",
"primitive_type": "phoneme",
"grounding": "extracted",
"provenance": "AUDITED per-field. The 10 monophthong-vowel F1/F2/F3 (IY,IH,EH,AE,AA,AO,UH,UW,AH,ER) are the MEASURED adult-male /hVd/ means of Peterson & Barney (1952) JASA 24:175-184, verified vs CRAN phonTools::pb52. AX=neutral uniform-tube resonances (Fant, physics). OW steady target = synthesis convention (diphthong). Consonant loci (M,N,NG,L,R,W,Y,Z,DH,V,S,F,HH) and ALL bandwidths + dur/amp = standard formant-synthesis conventions (Klatt 1980 JASA 67:971), engineering defaults NOT field measurements. No numbers invented/LLM-generated.",
"records": [
{
"key": "IY",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 270,
"f2": 2290,
"f3": 3010,
"bw1": 60,
"bw2": 90,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 130,
"amp": 100
}
},
{
"key": "IH",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 390,
"f2": 1990,
"f3": 2550,
"bw1": 70,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 110,
"amp": 100
}
},
{
"key": "EH",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 530,
"f2": 1840,
"f3": 2480,
"bw1": 80,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 130,
"amp": 100
}
},
{
"key": "AE",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 660,
"f2": 1720,
"f3": 2410,
"bw1": 90,
"bw2": 110,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 150,
"amp": 100
}
},
{
"key": "AA",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 730,
"f2": 1090,
"f3": 2440,
"bw1": 90,
"bw2": 110,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 150,
"amp": 100
}
},
{
"key": "AO",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 570,
"f2": 840,
"f3": 2410,
"bw1": 80,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 140,
"amp": 100
}
},
{
"key": "UH",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 440,
"f2": 1020,
"f3": 2240,
"bw1": 70,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 110,
"amp": 100
}
},
{
"key": "UW",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 300,
"f2": 870,
"f3": 2240,
"bw1": 70,
"bw2": 90,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 140,
"amp": 100
}
},
{
"key": "AH",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 640,
"f2": 1190,
"f3": 2390,
"bw1": 80,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 110,
"amp": 95
}
},
{
"key": "ER",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 490,
"f2": 1350,
"f3": 1690,
"bw1": 80,
"bw2": 100,
"bw3": 120,
"voiced": 1,
"nasal": 0,
"dur": 140,
"amp": 95
}
},
{
"key": "AX",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 500,
"f2": 1500,
"f3": 2500,
"bw1": 80,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 80,
"amp": 85
}
},
{
"key": "OW",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 490,
"f2": 910,
"f3": 2380,
"bw1": 80,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 140,
"amp": 100
}
},
{
"key": "M",
"features": {
"manner": "nasal",
"voiced": "yes",
"nasal": "yes"
},
"attributes": {
"f1": 250,
"f2": 900,
"f3": 2200,
"bw1": 90,
"bw2": 120,
"bw3": 180,
"voiced": 1,
"nasal": 1,
"dur": 80,
"amp": 60
}
},
{
"key": "N",
"features": {
"manner": "nasal",
"voiced": "yes",
"nasal": "yes"
},
"attributes": {
"f1": 250,
"f2": 1700,
"f3": 2600,
"bw1": 90,
"bw2": 120,
"bw3": 180,
"voiced": 1,
"nasal": 1,
"dur": 80,
"amp": 60
}
},
{
"key": "NG",
"features": {
"manner": "nasal",
"voiced": "yes",
"nasal": "yes"
},
"attributes": {
"f1": 250,
"f2": 2300,
"f3": 2700,
"bw1": 90,
"bw2": 120,
"bw3": 180,
"voiced": 1,
"nasal": 1,
"dur": 80,
"amp": 60
}
},
{
"key": "L",
"features": {
"manner": "approximant",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 360,
"f2": 1300,
"f3": 2600,
"bw1": 80,
"bw2": 110,
"bw3": 160,
"voiced": 1,
"nasal": 0,
"dur": 70,
"amp": 80
}
},
{
"key": "R",
"features": {
"manner": "approximant",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 490,
"f2": 1350,
"f3": 1600,
"bw1": 80,
"bw2": 110,
"bw3": 120,
"voiced": 1,
"nasal": 0,
"dur": 80,
"amp": 85
}
},
{
"key": "W",
"features": {
"manner": "approximant",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 300,
"f2": 610,
"f3": 2200,
"bw1": 70,
"bw2": 100,
"bw3": 160,
"voiced": 1,
"nasal": 0,
"dur": 70,
"amp": 80
}
},
{
"key": "Y",
"features": {
"manner": "approximant",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 270,
"f2": 2290,
"f3": 3010,
"bw1": 60,
"bw2": 90,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 60,
"amp": 80
}
},
{
"key": "Z",
"features": {
"manner": "fricative",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 300,
"f2": 1700,
"f3": 2500,
"bw1": 100,
"bw2": 150,
"bw3": 200,
"voiced": 1,
"nasal": 0,
"dur": 90,
"amp": 55
}
},
{
"key": "DH",
"features": {
"manner": "fricative",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 300,
"f2": 1400,
"f3": 2500,
"bw1": 100,
"bw2": 150,
"bw3": 200,
"voiced": 1,
"nasal": 0,
"dur": 70,
"amp": 55
}
},
{
"key": "V",
"features": {
"manner": "fricative",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 300,
"f2": 1000,
"f3": 2300,
"bw1": 100,
"bw2": 150,
"bw3": 200,
"voiced": 1,
"nasal": 0,
"dur": 70,
"amp": 55
}
},
{
"key": "S",
"features": {
"manner": "fricative",
"voiced": "no",
"nasal": "no"
},
"attributes": {
"f1": 320,
"f2": 1700,
"f3": 2500,
"bw1": 200,
"bw2": 200,
"bw3": 250,
"voiced": 0,
"nasal": 0,
"dur": 110,
"amp": 45
}
},
{
"key": "F",
"features": {
"manner": "fricative",
"voiced": "no",
"nasal": "no"
},
"attributes": {
"f1": 300,
"f2": 1200,
"f3": 2400,
"bw1": 200,
"bw2": 200,
"bw3": 250,
"voiced": 0,
"nasal": 0,
"dur": 100,
"amp": 40
}
},
{
"key": "HH",
"features": {
"manner": "fricative",
"voiced": "no",
"nasal": "no"
},
"attributes": {
"f1": 500,
"f2": 1500,
"f3": 2500,
"bw1": 200,
"bw2": 250,
"bw3": 300,
"voiced": 0,
"nasal": 0,
"dur": 70,
"amp": 40
}
},
{
"key": "SIL",
"features": {
"manner": "silence",
"voiced": "no",
"nasal": "no"
},
"attributes": {
"f1": 500,
"f2": 1500,
"f3": 2500,
"bw1": 100,
"bw2": 100,
"bw3": 100,
"voiced": 0,
"nasal": 0,
"dur": 55,
"amp": 0
}
}
]
}
+45
View File
@@ -0,0 +1,45 @@
# acoustic-phonetics SOURCE — the learned speech primitives, as INGESTIBLE DATA.
# NOT audio, NOT code: formant geometry of the phonemes, to be ingested via the
# ingest organ into the engram as a phoneme manifold. The render reads this
# geometry back from the engram; nothing is frozen in EL code.
#
# PROVENANCE (audited, per-field honesty — no invented numbers):
# * The 10 MONOPHTHONG VOWEL formants F1/F2/F3 (IY,IH,EH,AE,AA,AO,UH,UW,AH,ER)
# are the MEASURED adult-male means of Peterson & Barney (1952), JASA 24:175-184
# — the canonical /hVd/ table, verified digit-for-digit vs CRAN phonTools::pb52.
# These are real measured values.
# * AX (schwa) F1/F2/F3 = neutral uniform-tube resonances (2n-1)*500 — a PHYSICS
# value (Fant), not a P&B measurement.
# * OW is a diphthong; its listed steady target is a conventional synthesis value,
# not a P&B monophthong measurement.
# * CONSONANT loci (M,N,NG,L,R,W,Y,Z,DH,V,S,F,HH) and ALL BANDWIDTHS (B1,B2,B3)
# and dur/amp are STANDARD FORMANT-SYNTHESIS conventions (Klatt 1980, JASA 67:971
# "Software for a cascade/parallel formant synthesizer") — engineering defaults,
# NOT per-phoneme field measurements. Labeled as such, not attributed to P&B.
# Format: SYM|F1|F2|F3|B1|B2|B3|voiced|nasal|dur_ms|amp|class|example
IY|270|2290|3010|60|90|150|1|0|130|100|vowel|beet
IH|390|1990|2550|70|100|150|1|0|110|100|vowel|bit
EH|530|1840|2480|80|100|150|1|0|130|100|vowel|bet
AE|660|1720|2410|90|110|150|1|0|150|100|vowel|bat
AA|730|1090|2440|90|110|150|1|0|150|100|vowel|bot
AO|570|840|2410|80|100|150|1|0|140|100|vowel|bought
UH|440|1020|2240|70|100|150|1|0|110|100|vowel|book
UW|300|870|2240|70|90|150|1|0|140|100|vowel|boot
AH|640|1190|2390|80|100|150|1|0|110|95|vowel|but
ER|490|1350|1690|80|100|120|1|0|140|95|vowel|bird
AX|500|1500|2500|80|100|150|1|0|80|85|vowel|about
OW|490|910|2380|80|100|150|1|0|140|100|vowel|boat
M|250|900|2200|90|120|180|1|1|80|60|nasal|map
N|250|1700|2600|90|120|180|1|1|80|60|nasal|nap
NG|250|2300|2700|90|120|180|1|1|80|60|nasal|sing
L|360|1300|2600|80|110|160|1|0|70|80|approximant|lip
R|490|1350|1600|80|110|120|1|0|80|85|approximant|rip
W|300|610|2200|70|100|160|1|0|70|80|approximant|wet
Y|270|2290|3010|60|90|150|1|0|60|80|approximant|yet
Z|300|1700|2500|100|150|200|1|0|90|55|fricative|zoo
DH|300|1400|2500|100|150|200|1|0|70|55|fricative|the
V|300|1000|2300|100|150|200|1|0|70|55|fricative|van
S|320|1700|2500|200|200|250|0|0|110|45|fricative|see
F|300|1200|2400|200|200|250|0|0|100|40|fricative|fee
HH|500|1500|2500|200|250|300|0|0|70|40|fricative|hat
SIL|500|1500|2500|100|100|100|0|0|55|0|silence|_
+5
View File
@@ -80,6 +80,11 @@ build {
"src/grammar.el",
"src/realizer.el",
"src/semantics.el",
"src/comprehend.el",
"src/propositions.el",
"src/multilingual.el",
"src/self_region.el",
"src/dialogue.el",
"src/elp.el",
]
}
+91
View File
@@ -0,0 +1,91 @@
> **STATUS: STAGING / PROOF-OF-SHAPE — not the deliverable.** This Python package
> proved the architecture end-to-end against the proven realizer faculty (faithful
> md/docx/midi from real geometry: 0 ungrounded claims, SACRED polarity). Per Will's
> steer, the DELIVERABLE is NATIVE: the seam lives on the existing EL realizer as
> **surface-as-profile** — see `../src/surface-profile.el` and
> `../tests/examples/surface-profile-demo.el` (compiles + runs through elc → C →
> binary). The concepts below (one geometry-carrying frame; surface = a pluggable
> profile; plan/realize; deterministic-from-meaning) are exactly what the native
> module implements. Keep this package as the validated proof; build native.
# Efferent Multimodal Projector
**geometry → any surface, faithfully.** Neuron's own document-generation faculty:
the efferent twin of the ingest organ. Ingest is afferent (world → geometry);
this is efferent (geometry → an arbitrary-format document / any modality).
Built against the **proven** realizer faculty (neuron-talk sidecar `:8756`,
artifact `art-7affa557`). The live soul (`:8742` / `:7770`) is contacted **only**
through the read-only, GET-only `engram_client` — never mutated.
## The pipeline (surface-agnostic)
```
geometry region + surface/format spec
→ PLAN (manifold → document skeleton/DAG; the geometry IS the outline) plan.py
→ REALIZE (proven realizer, scaled sentence → passage, each section faithful) realize.py
→ COHERE (document-level flow / transitions, not stitched sentences) cohere.py
→ EMIT (pluggable SurfaceProjector → the target surface) projectors/
```
**The surface is a PARAMETER.** `pipeline.build_ir(...)` builds ONE
surface-neutral `DocumentIR` (`document_ir.py`); `pipeline.emit(doc, surface)`
projects it to whichever surface you name. Markdown, docx, and MIDI are the same
IR emitted three ways.
## The pivot: a geometry-carrying IR
`DocumentIR` is **not** a text tree. Every `Block` carries BOTH:
- `.sentences` — realized faithful text (what **text** projectors read),
- `.provenance` — the source geometry: `subj_id / relation / obj / polarity /
confidence / importance / salience / node_id` (what **music / image / video**
projectors read).
That single decision is what makes the projector multimodal: text renders the
words; music/image decode the geometry. A claim with no provenance cannot exist
in the IR — faithfulness is structural.
## The one shared seam
`projectors/base.py` — `SurfaceProjector.project(frame: DocumentIR) -> bytes`
(+ `surface / media_type / ext / modality / profile`). Register with
`register()`. Adding a surface changes nothing upstream.
`TwoStageProjector` blesses the peer plan/realize decomposition:
`spec = plan(frame)`, `bytes = realize(spec)`, `project = realize∘plan`; the
`profile` is the pluggable per-surface knob (text lang-profile, music
instr/mode-profile). `projectors/midi.py` is the reference two-stage impl.
## Surfaces
| surface | modality | status | emitter |
|---|---|---|---|
| `markdown` | text | landed | own (str) |
| `docx` | text | landed | own minimal OOXML (stdlib `zipfile`+XML, no lib) |
| `midi` | audio | landed (symbolic-music proof) | own minimal SMF (stdlib `struct`, no lib) |
| `audio` (WAV) | audio | peer agent (additive synth) | conforms to `TwoStageProjector` |
| `image` | image | documented seam | `projectors/seams.py` |
| `video` | video | documented seam (image×sound×time) | `projectors/seams.py` |
Music maps: relation → scale degree (same relation → same pitch), **polarity →
major/minor third (SACRED negation is audible)**, confidence → duration,
importance → velocity, section → register. Deterministic projection from meaning
— nothing invented.
## Faithfulness
`provenance.py` audits the IR: **zero** ungrounded claims, SACRED polarity
preserved (negations reported, never dropped), COHERE introduces no new geometry
(connectives are marked). `trace_table()` emits the geometry → section → claim
table.
## Run
```bash
PY=~/Desktop/lang-realizers/venv/bin/python
PYTHONPATH=~/Desktop/neuron-talk:~/Desktop/lang-realizers $PY generate.py
# writes ./out/{neuron-self,engram-temporal}.{md,docx,mid} + *.audit.json + *.provenance.md
```
Requires the proven realizer env (spaCy + the neuron-talk/lang-realizers engine)
and the read-only engram at `:8742`.
+79
View File
@@ -0,0 +1,79 @@
"""cohere.py — COHERE stage: document-level flow, not stitched sentences.
Fidelity is REALIZE's job; FLOW is this stage's. The hard part beyond sentence
fidelity is that a document must read as one thing. We add connective tissue at
the passage level:
* an opening abstract that names what the document covers (built ONLY from the
section headings that already exist — it introduces no new claim),
* a short transition lead into each section after the first, drawn from a
fixed set of discourse connectives ("Beyond that,", "Relatedly,", ...) that
carry no propositional content,
* ordering so the highest-grounded section leads.
CRITICAL: every connective is marked ``kind="connective"`` in its provenance, so
the faithfulness audit can prove COHERE introduced ZERO new geometry claims. A
transition is discourse glue, never a fact.
"""
from __future__ import annotations
from document_ir import Block, DocumentIR, Provenance
# discourse connectives — pure flow, no propositional content
_TRANSITIONS = [
"Beyond that,", "Relatedly,", "In the same region,", "From there,",
"Alongside this,", "Further,", "Turning to the next facet,",
]
def _connective_prov() -> Provenance:
return Provenance(subj_id=None, subject=None, relation="", obj=None,
polarity="aff", confidence=1.0, node_id=None,
kind="connective")
def _abstract_block(doc: DocumentIR) -> Block:
"""A grounded opening: names the sections, asserts nothing new."""
headings = [s.heading for s in doc.sections]
if not headings:
return Block(role="lead")
if len(headings) == 1:
body = f"This document, generated from Neuron's geometry, covers {headings[0]}."
else:
listed = ", ".join(headings[:-1]) + f", and {headings[-1]}"
body = ("This document is projected directly from Neuron's meaning-geometry. "
f"It traces {listed}.")
b = Block(role="lead")
b.sentences.append(body)
b.provenance.append(_connective_prov())
return b
def cohere_document(doc: DocumentIR, *, add_abstract: bool = True,
add_transitions: bool = True) -> DocumentIR:
"""Order sections by grounding, add abstract + transitions (flow only)."""
# order: strongest-grounded section (mean confidence x #claims) first,
# but keep an explicitly-first section if the plan pinned one via level 1.
def _score(sec):
provs = [p for p in sec.all_provenance() if p.kind == "fact"]
if not provs:
return 0.0
mean_conf = sum(p.confidence for p in provs) / len(provs)
return mean_conf * len(provs)
doc.sections.sort(key=_score, reverse=True)
if add_transitions:
for i, sec in enumerate(doc.sections):
if i == 0 or not sec.blocks:
continue
lead = _TRANSITIONS[(i - 1) % len(_TRANSITIONS)]
first = sec.blocks[0]
if first.sentences:
# prepend the connective to the first sentence (flow, no new claim)
first.sentences[0] = f"{lead} {first.sentences[0][0].lower()}{first.sentences[0][1:]}"
if add_abstract:
doc.meta["abstract"] = _abstract_block(doc)
return doc
+111
View File
@@ -0,0 +1,111 @@
"""document_ir.py — the surface-neutral, GEOMETRY-CARRYING document intermediate.
This is the pivot of the whole efferent projector. A DocumentIR is NOT a text
tree. It is a projection of a meaning-geometry region that carries, at every
leaf, BOTH:
* the realized surface text (``Block.sentences``) — what a TEXT projector reads,
* the source geometry (``Block.provenance``) — what a MUSIC / IMAGE /
VIDEO projector reads.
Because the IR holds the geometry, not just the words, the SAME
plan -> realize -> cohere pipeline drives every surface. A markdown projector
renders the sentences; a music projector reads the provenance edges (salience,
importance, polarity, relation) and maps them onto a symbolic-music surface;
an image/video projector (documented seam) would read the same geometry.
Nothing in this module invents content. Every :class:`Provenance` points at a
real engram node id and a real relation. That is the faithfulness contract made
structural: a claim with no provenance cannot exist in the IR.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
# --------------------------------------------------------------------------- #
# Provenance — the geometry an emitted claim traces to. FAITHFULNESS is here.
# --------------------------------------------------------------------------- #
@dataclass
class Provenance:
"""One geometry edge behind one realized claim.
``kind`` distinguishes a FACT (a structural edge asserted by the geometry,
spoken as fact) from an INTERPRETATION (something attributed, spoken with
attribution) — the facts-as-facts + interpretations-attributed discipline
(memory 80927e26). ``polarity`` is SACRED: a negated edge stays negated.
"""
subj_id: str | None # source engram node id of the subject
subject: str | None # normalized subject surface
relation: str # predicate lemma (e.g. "use", "contain", "be")
obj: str | None # normalized object / complement surface
polarity: str = "aff" # "aff" | "neg" (SACRED — never silently flipped)
confidence: float = 0.0 # extraction confidence in [0,1]
node_id: str | None = None # engram node the claim was extracted from
kind: str = "fact" # "fact" | "interpretation"
importance: float = 0.0 # source node importance (drives music/emphasis)
salience: float = 0.0 # source node salience
def trace(self) -> str:
arrow = "-->" if self.polarity == "aff" else "--NOT-->"
return (f"[{(self.node_id or '?')[:8]}] {self.subject!r} {arrow}"
f"{self.relation} {self.obj!r} (conf {self.confidence:.2f})")
@dataclass
class Block:
"""A passage: one or more faithful sentences + the geometry they trace to.
``sentences`` and ``provenance`` are index-aligned where possible: sentence
``i`` was realized from ``provenance[i]``. A COHERE transition sentence with
no new geometry carries a provenance whose ``kind == "connective"`` so the
audit can see it introduced no new claim.
"""
sentences: list[str] = field(default_factory=list)
provenance: list[Provenance] = field(default_factory=list)
role: str = "body" # "body" | "lead" | "transition"
def text(self) -> str:
return " ".join(s.rstrip(". ") + "." for s in self.sentences if s.strip())
@dataclass
class Section:
heading: str
level: int = 2 # markdown heading level / outline depth
blocks: list[Block] = field(default_factory=list)
seed_ids: list[str] = field(default_factory=list) # geometry nodes of section
summary: str = "" # one-line grounded gloss (for pptx bullets / TOC)
def all_provenance(self) -> list[Provenance]:
out: list[Provenance] = []
for b in self.blocks:
out.extend(b.provenance)
return out
@dataclass
class DocumentIR:
"""The surface-neutral document. Built ONCE, projected to ANY surface."""
title: str
subtitle: str = ""
sections: list[Section] = field(default_factory=list)
seed_id: str | None = None # the geometry region root
format_spec: dict[str, Any] = field(default_factory=dict) # requested shape
meta: dict[str, Any] = field(default_factory=dict)
# -- geometry facets (what non-text projectors consume) ----------------- #
def all_provenance(self) -> list[Provenance]:
out: list[Provenance] = []
for s in self.sections:
out.extend(s.all_provenance())
return out
def claim_count(self) -> int:
return sum(1 for p in self.all_provenance() if p.kind in ("fact", "interpretation"))
def ungrounded_count(self) -> int:
"""Claims with no traceable node — MUST be zero for a faithful doc."""
return sum(1 for p in self.all_provenance()
if p.kind in ("fact", "interpretation") and not p.node_id)
+81
View File
@@ -0,0 +1,81 @@
"""generate.py — drive the projector: one geometry region -> many surfaces.
Proves the thesis with REAL output: builds ONE surface-neutral DocumentIR from
Neuron's OWN self-geometry (read-only against the live soul via the proven
faculty), then EMITS it to Markdown, docx, and MIDI — the same plan/realize/
cohere, three surfaces. Writes the files + the faithfulness audit to ./out/.
"""
from __future__ import annotations
import json
import os
import sys
_HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, _HERE)
import pipeline # noqa: E402
import provenance # noqa: E402
from geometry import load_self_region # noqa: E402
OUT = os.path.join(_HERE, "out")
def _emit_all(doc, stem):
"""Emit one IR to every text/audio surface + audit + provenance."""
for surface in ("markdown", "docx", "midi"):
data = pipeline.emit(doc, surface)
proj = pipeline.get_projector(surface)
path = os.path.join(OUT, f"{stem}.{proj.ext}")
with open(path, "wb") as f:
f.write(data)
print(f" emitted {surface:9s} -> {os.path.basename(path)} ({len(data)} bytes)")
a = provenance.audit(doc)
with open(os.path.join(OUT, f"{stem}.audit.json"), "w") as f:
json.dump(a, f, indent=2)
with open(os.path.join(OUT, f"{stem}.provenance.md"), "w") as f:
f.write(provenance.trace_table(doc))
print(" audit:", {k: a[k] for k in ("claims", "ungrounded_claims",
"negations_preserved", "distinct_source_nodes", "faithful")})
return a
def main():
os.makedirs(OUT, exist_ok=True)
print("surfaces registered:", pipeline.available_surfaces())
# ---- Document 1: Neuron's self-description (marquee) ------------------- #
print("\n[1] Neuron self-description")
region = load_self_region(max_nodes=9)
print(" self region:", region)
doc1 = pipeline.build_ir(
None, region=region,
title="Neuron: A Self-Description from Its Own Geometry",
subtitle="Projected efferently from the engram — every claim traces a node.",
format_spec={"genre": "self-description", "register": "expository"},
max_sections=5, conf_floor=0.6)
print(f" IR: {len(doc1.sections)} sections, {doc1.claim_count()} claims, "
f"ungrounded={doc1.ungrounded_count()}")
_emit_all(doc1, "neuron-self")
# ---- Document 2: a coherent, clean whitepaper-style section ------------ #
print("\n[2] Whitepaper-style section (coherent clean region)")
doc2, _ = pipeline.project(
["chronoception", "time", "awareness", "engram", "temporal"],
surface="markdown",
title="Temporal Awareness in the Engram",
subtitle="A section projected from the geometry of chronoception.",
format_spec={"genre": "whitepaper-section", "register": "technical"},
max_sections=4)
print(f" IR: {len(doc2.sections)} sections, {doc2.claim_count()} claims, "
f"ungrounded={doc2.ungrounded_count()}")
_emit_all(doc2, "engram-temporal")
# echo both markdowns so they are visible in the run log
for stem, doc in (("neuron-self", doc1), ("engram-temporal", doc2)):
print(f"\n===== GENERATED MARKDOWN — {stem} =====\n")
print(pipeline.emit(doc, "markdown").decode())
if __name__ == "__main__":
main()
+129
View File
@@ -0,0 +1,129 @@
"""geometry.py — READ-ONLY loader for a meaning-geometry region.
The efferent projector never writes to the soul. This module reaches the
geometry through the PROVEN, read-only neuron-talk faculty (``engram_client``,
GET-only, which physically refuses non-GET methods) against the running sidecar
soul. The live daemon :8742 / :7770 is contacted ONLY through that read-only
client — never mutated.
A "region" is a seed node plus a bounded neighborhood: the manifold that will
become the document's skeleton. We pool a few single-term lexical searches
(the engram search is a single-term matcher) and, when available, walk one hop
of reified neighbors, then rank by self/importance signal.
"""
from __future__ import annotations
import os
import sys
# Wire in the proven faculty (own-the-core: we reuse it, we do not fork it).
_NT = os.path.expanduser("~/Desktop/neuron-talk")
_LR = os.path.expanduser("~/Desktop/lang-realizers")
for _p in (_NT, _LR):
if _p not in sys.path:
sys.path.insert(0, _p)
from engram_client import ReadOnlyEngramClient # noqa: E402
class Region:
"""A geometry region: ranked nodes + the reified edges among them."""
def __init__(self, seed: str, nodes: list[dict], edges: list[dict]):
self.seed = seed
self.nodes = nodes # ranked engram node dicts
self.edges = edges # [{src, dst, edge, ...}]
self.by_id = {n["id"]: n for n in nodes if n.get("id")}
def __repr__(self):
return f"<Region seed={self.seed!r} nodes={len(self.nodes)} edges={len(self.edges)}>"
def _prose_quality(content: str) -> float:
"""Reward clean expository prose; penalize shouty banner-dense nodes.
A high ALLCAPS-word ratio or very short content signals a banner/telegraphic
memory node that extracts into garbage. Clean declarative prose scores high.
"""
if not content or not content.strip():
return 0.0
words = content.split()
if len(words) < 8:
return 0.1
caps = sum(1 for w in words if len(w) > 2 and w.strip(".,:;'\"-").isupper())
caps_ratio = caps / max(1, len(words))
# sentences with lowercase interior words read as prose
lower = sum(1 for w in words if w[:1].islower())
lower_ratio = lower / max(1, len(words))
return max(0.0, 1.2 * lower_ratio - 2.0 * caps_ratio)
def _relevance(content: str, terms: list[str]) -> float:
"""Topical relevance to the seed terms — keeps a region ON-THEME so a clean
but off-topic node cannot hijack the document."""
if not terms:
return 0.0
low = (content or "").lower()
hits = sum(1 for t in terms if t.lower() in low)
return hits / max(1, len(terms))
def _node_rank(n: dict, terms: list[str] | None = None) -> float:
return (float(n.get("importance") or 0.0) * 2.0
+ float(n.get("salience") or 0.0)
+ 1.5 * _prose_quality(n.get("content") or "")
+ 2.0 * _relevance(n.get("content") or "", terms or [])
+ (0.5 if (n.get("content") or "").strip() else 0.0))
def load_region(seed_terms: list[str] | str, *, client: ReadOnlyEngramClient | None = None,
max_nodes: int = 10, per_term: int = 20, hop: bool = True) -> Region:
"""Pull a bounded geometry region around ``seed_terms`` (read-only).
``seed_terms`` may be a single string or several probe terms; results are
pooled and de-duplicated. When ``hop`` and the reified neighbor endpoint is
live, one hop of neighbors is folded in so the region is a real
neighborhood, not just a keyword hit list.
"""
client = client or ReadOnlyEngramClient()
if isinstance(seed_terms, str):
seed_terms = [seed_terms]
pool: dict[str, dict] = {}
for term in seed_terms:
for n in client.search(term, limit=per_term):
if isinstance(n, dict) and n.get("id"):
pool.setdefault(n["id"], n)
ranked = sorted(pool.values(), key=lambda n: _node_rank(n, seed_terms),
reverse=True)
nodes = ranked[:max_nodes]
edges: list[dict] = []
if hop and nodes:
present = {n["id"] for n in nodes}
for n in list(nodes):
try:
for nb in client.neighbors(n["id"]):
node = nb.get("node") if isinstance(nb, dict) else None
edge = nb.get("edge") if isinstance(nb, dict) else None
if node and node.get("id"):
edges.append({"src": n["id"], "dst": node["id"],
"edge": edge})
# fold a strong neighbor into the region (bounded)
if (node["id"] not in present and len(nodes) < max_nodes + 6
and _node_rank(node, seed_terms) > 0.4):
present.add(node["id"])
nodes.append(node)
except Exception: # noqa: BLE001 — read-only best-effort; never fatal
continue
return Region(seed=", ".join(seed_terms), nodes=nodes, edges=edges)
def load_self_region(client: ReadOnlyEngramClient | None = None,
max_nodes: int = 10) -> Region:
"""The self/identity region — Neuron's own geometry, for self-description."""
return load_region(["self", "identity", "Neuron", "values", "memory",
"imprint", "consciousness"],
client=client, max_nodes=max_nodes)
+67
View File
@@ -0,0 +1,67 @@
"""pipeline.py — the Efferent Multimodal Projector, top level.
geometry region + surface/format spec
-> PLAN (manifold -> document skeleton/DAG)
-> REALIZE (proven realizer, sentence -> passage, each section faithful)
-> COHERE (document-level flow / transitions, not stitched sentences)
-> EMIT (pluggable SurfaceProjector -> the target surface)
THE SURFACE IS A PARAMETER. ``project(...)`` builds the geometry-carrying
DocumentIR once, then hands it to whichever surface projector the caller named.
Markdown, docx, and midi (music) are all the SAME IR emitted differently. That
is the efferent multimodal projector: geometry -> any surface.
"""
from __future__ import annotations
import os
import sys
_HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, _HERE)
sys.path.insert(0, os.path.join(_HERE, "projectors"))
from cohere import cohere_document # noqa: E402
from document_ir import DocumentIR # noqa: E402
from geometry import Region, load_region # noqa: E402
from plan import plan_document # noqa: E402
from realize import realize_document # noqa: E402
# registering the projectors (import for side-effect: each self-registers)
import projectors.markdown # noqa: E402,F401
import projectors.docx # noqa: E402,F401
import projectors.midi # noqa: E402,F401
import projectors.seams # noqa: E402,F401
from projectors.base import available_surfaces, get_projector # noqa: E402
def build_ir(seed_terms, *, title: str, subtitle: str = "",
format_spec: dict | None = None,
region: Region | None = None,
max_sections: int = 8, conf_floor: float = 0.55) -> DocumentIR:
"""geometry -> PLAN -> REALIZE -> COHERE = the surface-neutral DocumentIR."""
region = region or load_region(seed_terms)
doc = plan_document(region, title=title, subtitle=subtitle,
format_spec=format_spec or {},
conf_floor=conf_floor, max_sections=max_sections)
doc = realize_document(doc)
doc = cohere_document(doc)
return doc
def emit(doc: DocumentIR, surface: str) -> bytes:
"""EMIT: project the built IR onto one surface (surface = a parameter)."""
return get_projector(surface).project(doc)
def project(seed_terms, *, surface: str, title: str, subtitle: str = "",
format_spec: dict | None = None, region: Region | None = None,
max_sections: int = 8) -> tuple[DocumentIR, bytes]:
"""The full efferent projection: geometry + surface -> (IR, bytes)."""
doc = build_ir(seed_terms, title=title, subtitle=subtitle,
format_spec=format_spec, region=region,
max_sections=max_sections)
return doc, emit(doc, surface)
__all__ = ["build_ir", "emit", "project", "available_surfaces",
"get_projector", "load_region", "DocumentIR"]
+192
View File
@@ -0,0 +1,192 @@
"""plan.py — PLAN stage: geometry region -> document skeleton (a DAG/outline).
The manifold becomes the skeleton. We extract faithful propositions from the
region's nodes (the proven neuron-talk extractor, SACRED polarity preserved),
apply a quality floor, then GROUP them into sections. Grouping is by source
node — each engram node is one coherent topic, so one salient node becomes one
section. The section ORDER is the node ranking (importance/salience): the
geometry decides the outline, not a template.
Output: a DocumentIR whose sections carry seed node ids and empty blocks. REALIZE
fills the blocks; the plan owns the structure.
"""
from __future__ import annotations
import os
import re
import sys
_NT = os.path.expanduser("~/Desktop/neuron-talk")
_LR = os.path.expanduser("~/Desktop/lang-realizers")
for _p in (_NT, _LR):
if _p not in sys.path:
sys.path.insert(0, _p)
import propositions # noqa: E402 (the proven, faithful extractor)
from document_ir import DocumentIR, Section # noqa: E402
from geometry import Region # noqa: E402
# --------------------------------------------------------------------------- #
# Proposition quality — keep only clean, well-grounded claims.
# --------------------------------------------------------------------------- #
_JUNK_RE = re.compile(r"[.][a-z]{1,3}\b|[^A-Za-z0-9 '\-]") # ".o", stray symbols
def _has_banner_token(s: str) -> bool:
"""True if any word is an ALLCAPS banner token (DHARMA, ENGRAM, MEASURED)."""
for w in (s or "").split():
core = w.strip(".,:;'\"-")
if len(core) > 2 and core.isupper():
return True
return False
def _clean_prop(p, floor: float) -> bool:
if p.confidence < floor:
return False
if not p.subject or not (p.object or (p.obj_np is not None)):
return False
subj = (p.subject or "").strip()
obj = (p.object or "").strip()
if len(subj) < 2:
return False
# banner-derived shouty fragments read as garbage in prose
if _has_banner_token(subj) or _has_banner_token(obj):
return False
if propositions._is_shouty(p.sentence or ""):
return False
# junk tokens: file-extension fragments (".o"), stray non-word symbols
if _JUNK_RE.search(subj) or _JUNK_RE.search(obj):
return False
# a proposition whose object repeats the subject is usually a parse artifact
if obj and subj.lower() == obj.lower():
return False
# a bare copula with no real complement ("X is it") reads as noise
if p.predicate == "be" and obj.lower() in ("it", "no", "nothing", "empty", ""):
return False
return True
def _dedup(props):
"""Drop duplicate claims. Two axes: (a) identical (pred,obj,polarity), and
(b) same (subject,predicate) — which collapses a mis-split compound like
"detection is post-hoc eval" -> "Detection is post/hoc/eval" into one claim
(keep the highest-confidence surface)."""
props = sorted(props, key=lambda p: p.confidence, reverse=True)
seen_po, seen_sp, out = set(), set(), []
for p in props:
subj = (p.subject or "").lower()
po = (p.predicate, (p.object or "").lower(), p.polarity)
sp = (subj, p.predicate, p.polarity)
if po in seen_po or sp in seen_sp:
continue
seen_po.add(po)
seen_sp.add(sp)
out.append(p)
return out
# --------------------------------------------------------------------------- #
# Heading derivation — a clean human heading from a node.
# --------------------------------------------------------------------------- #
_HEADING_RE = re.compile(r"^\s*#{1,4}\s+(.{2,70})\s*$", re.M)
# node-type / system labels that are NOT topical headings
_NONTOPIC_LABEL = re.compile(r"^(memory|node|knowledge|doc|session)[:/]", re.I)
def _titlecase_banner(s: str) -> str:
"""A shouty banner ("CHRONOCEPTION — SCALE-INVARIANCE") makes a fine title
once Title-cased. Keep short acronyms uppercase."""
def fix(w):
core = w.strip("—-:,.")
if len(core) <= 3 and core.isupper():
return w # acronym
return w.capitalize()
return " ".join(fix(w) for w in s.split())
def _clean_heading(text: str) -> str | None:
"""First line only, no markdown, capped, banner Title-cased. None if unusable."""
if not text:
return None
line = text.strip().splitlines()[0]
line = re.sub(r"^#+\s*", "", line).strip().strip("#").strip()
# cut at a natural break so a long banner heading stays a heading, not a para
for sep in ("", " ", ": ", ". "):
if sep in line and len(line) > 48:
line = line.split(sep)[0].strip()
break
if not (3 <= len(line) <= 64):
return None
if propositions._is_shouty(line):
line = _titlecase_banner(line)
return line or None
def _heading_for(node: dict, fallback: str) -> str:
label = (node.get("label") or "").strip()
content = node.get("content") or ""
candidates: list[str] = []
# a node-type label ("memory:remembered") is never a topic — skip it
if label and not _NONTOPIC_LABEL.match(label):
candidates.append(label)
m = _HEADING_RE.search(content)
if m:
candidates.append(m.group(1))
# the leading banner/first sentence of the content is often the real title
first = re.split(r"(?<=[.\n])", content.strip(), maxsplit=1)[0] if content.strip() else ""
candidates.append(first)
for c in candidates:
h = _clean_heading(c)
if h:
return h
return fallback
def plan_document(region: Region, *, title: str, subtitle: str = "",
format_spec: dict | None = None,
conf_floor: float = 0.55,
max_sections: int = 8,
max_claims_per_section: int = 6) -> DocumentIR:
"""Region -> DocumentIR skeleton. The geometry dictates the outline."""
format_spec = format_spec or {}
doc = DocumentIR(title=title, subtitle=subtitle,
seed_id=region.nodes[0]["id"] if region.nodes else None,
format_spec=format_spec)
made = 0
seen_headings: set[str] = set()
for node in region.nodes:
if made >= max_sections:
break
props = propositions.extract(node.get("content") or "",
node_id=node.get("id"),
node_importance=float(node.get("importance") or 0.0),
max_sentences=10)
props = [p for p in props if _clean_prop(p, conf_floor)]
props = _dedup(props)
props.sort(key=lambda p: p.confidence, reverse=True)
props = props[:max_claims_per_section]
if not props:
continue
heading = _heading_for(node, fallback=f"Region {made + 1}")
# cross-section dedup: a topic appears once. Distinguish by top claim
# subject, else drop the collision so the outline stays clean.
if heading.lower() in seen_headings:
subj = (props[0].subject or "").strip().title()
alt = f"{heading}: {subj}" if subj and subj.lower() not in heading.lower() else None
if alt and alt.lower() not in seen_headings and len(alt) <= 64:
heading = alt
else:
continue
seen_headings.add(heading.lower())
sec = Section(heading=heading, level=2, seed_ids=[node["id"]])
# stash the planned propositions on the section for REALIZE
sec.__dict__["_planned_props"] = props
sec.__dict__["_node"] = node
doc.sections.append(sec)
made += 1
return doc
+106
View File
@@ -0,0 +1,106 @@
"""base.py — the SurfaceProjector interface + registry.
THE key abstraction of the efferent projector: a projector is a pure function
from the surface-neutral, geometry-carrying DocumentIR to bytes on a target
SURFACE. The surface is a PARAMETER. Adding a surface = registering one more
projector; nothing upstream (plan/realize/cohere) changes.
DocumentIR --project--> bytes (per surface)
A TEXT projector reads ``block.sentences``. A NON-TEXT projector (music, image,
video) reads ``block.provenance`` — the geometry the IR carries — and decodes it
onto its surface. Both consume the SAME IR. That symmetry is the whole design:
the realizer generalizes into a multimodal projector, geometry -> any surface.
"""
from __future__ import annotations
from typing import Protocol, runtime_checkable
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from document_ir import DocumentIR # noqa: E402
@runtime_checkable
class SurfaceProjector(Protocol):
"""Geometry-document -> one surface. Implementations MUST be pure & faithful.
THE ONE SHARED SEAM. Every surface — text, music, image, video — conforms to
this single contract:
project(frame: DocumentIR) -> bytes
where ``frame`` is the geometry-carrying meaning-geometry (the SemFrame at
document scale; a single utterance is the degenerate one-section frame).
RECOMMENDED INTERNAL SHAPE (the peer music/text decomposition, blessed here
so all surfaces share it): a projector may split ``project`` into
spec = self.plan(frame) # meaning-geometry -> surface-specific spec
bytes = self.realize(spec) # spec -> surface, via this projector's PROFILE
``project`` is then ``realize(plan(frame))``. The PROFILE (a text lang-profile,
a music instr/mode-profile, an image layout-profile) is a property of the
projector instance — the pluggable knob. See :class:`TwoStageProjector`.
A TEXT projector's plan reads ``frame`` sentences; a MUSIC/IMAGE projector's
plan reads ``frame.all_provenance()`` — the geometry — and derives its spec
(pitch/harmony/rhythm, or layout) FROM the meaning, deterministically. Same
frame, different profile.
"""
surface: str # "markdown" | "docx" | "midi" | "audio" | "image" | "video"
media_type: str # MIME type of the emitted bytes
ext: str # file extension (no dot)
modality: str # "text" | "audio" | "image" | "video"
profile: object # the pluggable per-surface profile (may be None)
def project(self, doc: DocumentIR) -> bytes:
"""Emit the document on this surface. Returns raw bytes."""
...
class TwoStageProjector:
"""Optional base for the peer plan()/realize() decomposition.
Subclasses implement ``plan(frame) -> spec`` and ``realize(spec) -> bytes``;
``project`` is their composition. This is exactly the peer music interface
(spec = plan(frame, profile); surface = realize(spec, profile)) expressed so
that it still satisfies the single ``SurfaceProjector.project`` seam. Text,
music, and image projectors can all subclass this and remain interchangeable.
"""
surface: str = ""
media_type: str = ""
ext: str = ""
modality: str = ""
profile: object = None
def plan(self, doc: DocumentIR): # -> spec
raise NotImplementedError
def realize(self, spec) -> bytes:
raise NotImplementedError
def project(self, doc: DocumentIR) -> bytes:
return self.realize(self.plan(doc))
_REGISTRY: dict[str, SurfaceProjector] = {}
def register(projector: SurfaceProjector) -> SurfaceProjector:
_REGISTRY[projector.surface] = projector
return projector
def get_projector(surface: str) -> SurfaceProjector:
if surface not in _REGISTRY:
raise KeyError(f"no projector registered for surface {surface!r}; "
f"have {sorted(_REGISTRY)}")
return _REGISTRY[surface]
def available_surfaces() -> list[str]:
return sorted(_REGISTRY)
+113
View File
@@ -0,0 +1,113 @@
"""docx.py — the .docx surface projector: an OWN minimal OOXML emitter.
Own-the-core: a .docx is just a ZIP of a few XML parts (WordprocessingML). We
emit it with the standard library only — ``zipfile`` + string XML — no
python-docx, no external dependency. This proves a "richer structured format"
surface without importing anyone else's toolkit.
Parts emitted (the minimal valid set + a styles part for real headings):
[Content_Types].xml
_rels/.rels
word/_rels/document.xml.rels
word/styles.xml (Title / Heading1 / Heading2 / Normal)
word/document.xml (the content)
Like the markdown projector it reads only the IR's realized sentences; it
invents nothing. The surface differs, the faithful content does not.
"""
from __future__ import annotations
import io
import os
import sys
import zipfile
from xml.sax.saxutils import escape
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from document_ir import DocumentIR # noqa: E402
from projectors.base import register # noqa: E402
_CONTENT_TYPES = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">
<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>
<Default Extension="xml" ContentType="application/xml"/>
<Override PartName="/word/document.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/>
<Override PartName="/word/styles.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.styles+xml"/>
</Types>"""
_RELS = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="word/document.xml"/>
</Relationships>"""
_DOC_RELS = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/styles" Target="styles.xml"/>
</Relationships>"""
_W = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
_STYLES = f"""<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<w:styles xmlns:w="{_W}">
<w:style w:type="paragraph" w:default="1" w:styleId="Normal"><w:name w:val="Normal"/>
<w:rPr><w:sz w:val="22"/></w:rPr></w:style>
<w:style w:type="paragraph" w:styleId="Title"><w:name w:val="Title"/>
<w:pPr><w:spacing w:after="240"/></w:pPr>
<w:rPr><w:b/><w:sz w:val="52"/></w:rPr></w:style>
<w:style w:type="paragraph" w:styleId="Subtitle"><w:name w:val="Subtitle"/>
<w:rPr><w:i/><w:sz w:val="28"/><w:color w:val="555555"/></w:rPr></w:style>
<w:style w:type="paragraph" w:styleId="Heading1"><w:name w:val="heading 1"/>
<w:pPr><w:spacing w:before="240" w:after="120"/><w:outlineLvl w:val="0"/></w:pPr>
<w:rPr><w:b/><w:sz w:val="34"/></w:rPr></w:style>
<w:style w:type="paragraph" w:styleId="Heading2"><w:name w:val="heading 2"/>
<w:pPr><w:spacing w:before="200" w:after="100"/><w:outlineLvl w:val="1"/></w:pPr>
<w:rPr><w:b/><w:sz w:val="28"/></w:rPr></w:style>
</w:styles>"""
def _para(text: str, style: str | None = None) -> str:
ppr = f"<w:pPr><w:pStyle w:val=\"{style}\"/></w:pPr>" if style else ""
return (f"<w:p>{ppr}<w:r><w:t xml:space=\"preserve\">"
f"{escape(text)}</w:t></w:r></w:p>")
class DocxProjector:
surface = "docx"
media_type = ("application/vnd.openxmlformats-officedocument."
"wordprocessingml.document")
ext = "docx"
modality = "text"
def _document_xml(self, doc: DocumentIR) -> str:
body: list[str] = [_para(doc.title, "Title")]
if doc.subtitle:
body.append(_para(doc.subtitle, "Subtitle"))
abstract = doc.meta.get("abstract")
if abstract is not None and abstract.sentences:
body.append(_para(abstract.text()))
for sec in doc.sections:
style = "Heading1" if sec.level <= 1 else "Heading2"
body.append(_para(sec.heading, style))
for block in sec.blocks:
t = block.text()
if t:
body.append(_para(t))
return (f"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>"
f"<w:document xmlns:w=\"{_W}\"><w:body>"
+ "".join(body)
+ "<w:sectPr><w:pgSz w:w=\"12240\" w:h=\"15840\"/>"
"<w:pgMar w:top=\"1440\" w:right=\"1440\" w:bottom=\"1440\" "
"w:left=\"1440\"/></w:sectPr></w:body></w:document>")
def project(self, doc: DocumentIR) -> bytes:
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as z:
z.writestr("[Content_Types].xml", _CONTENT_TYPES)
z.writestr("_rels/.rels", _RELS)
z.writestr("word/_rels/document.xml.rels", _DOC_RELS)
z.writestr("word/styles.xml", _STYLES)
z.writestr("word/document.xml", self._document_xml(doc))
return buf.getvalue()
register(DocxProjector())
+45
View File
@@ -0,0 +1,45 @@
"""markdown.py — the Markdown surface projector (text facet).
The most tractable surface, and the reference implementation: reads the IR's
realized sentences and lays them out as Markdown. Introduces no content — it is
pure typography over the faithful text the realizer produced.
"""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from document_ir import DocumentIR # noqa: E402
from projectors.base import register # noqa: E402
class MarkdownProjector:
surface = "markdown"
media_type = "text/markdown"
ext = "md"
modality = "text"
def render_str(self, doc: DocumentIR) -> str:
lines: list[str] = [f"# {doc.title}"]
if doc.subtitle:
lines.append(f"\n*{doc.subtitle}*")
abstract = doc.meta.get("abstract")
if abstract is not None and abstract.sentences:
lines.append("")
lines.append(abstract.text())
for sec in doc.sections:
lines.append("")
lines.append(f"{'#' * max(2, sec.level)} {sec.heading}")
for block in sec.blocks:
body = block.text()
if body:
lines.append("")
lines.append(body)
return "\n".join(lines) + "\n"
def project(self, doc: DocumentIR) -> bytes:
return self.render_str(doc).encode("utf-8")
register(MarkdownProjector())
+133
View File
@@ -0,0 +1,133 @@
"""midi.py — the MUSIC surface projector: geometry -> symbolic music (MIDI).
The first NON-TEXT surface, and the proof of the general shape. "Music is
language and it is math" (Will): symbolic music is tractable and geometry-native,
so it is the natural efferent twin to try first after text.
CRUCIALLY this projector does NOT read the realized sentences. It reads the IR's
GEOMETRY facet — ``block.provenance`` — and DECODES each edge onto a musical
surface. That is the whole thesis of the multimodal projector: the same
geometry-carrying IR drives text AND music; a text projector reads the words, a
music projector reads the meaning-geometry. The mapping is deterministic and
faithful to the geometry's structure:
relation lemma -> scale degree (same relation -> same pitch class;
meaning has a consistent sonic form)
polarity -> mode (aff = major third above; neg = minor
third / lowered — SACRED polarity is
audible, a negated edge sounds negated)
confidence -> note duration (stronger grounding rings longer)
importance -> velocity (more important source = louder)
section -> phrase + register shift (structure becomes musical form)
Own-the-core: a Standard MIDI File is a header chunk + a track chunk of
delta-timed events. We emit the raw bytes with ``struct`` — no external MIDI
library. Format 0, one track.
"""
from __future__ import annotations
import io
import os
import struct
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from document_ir import DocumentIR, Provenance # noqa: E402
from projectors.base import TwoStageProjector, register # noqa: E402
_TICKS = 480 # ticks per quarter note
_C_MAJOR = [0, 2, 4, 5, 7, 9, 11] # semitone offsets of a diatonic scale
def _vlq(n: int) -> bytes:
"""MIDI variable-length quantity encoding of a delta time."""
if n == 0:
return b"\x00"
out = bytearray()
out.append(n & 0x7F)
n >>= 7
while n:
out.insert(0, (n & 0x7F) | 0x80)
n >>= 7
return bytes(out)
def _degree_for(relation: str) -> int:
"""Stable scale degree for a relation lemma (same relation -> same pitch)."""
if not relation:
return 0
return sum(ord(c) for c in relation.lower()) % len(_C_MAJOR)
def _note_for(p: Provenance, base: int) -> tuple[int, int, int]:
"""(pitch, velocity, duration_ticks) for one geometry edge."""
root = base + _C_MAJOR[_degree_for(p.relation)]
# polarity -> mode: affirmed edges take the bright major third, negated edges
# take the darker minor third. The negation is AUDIBLE and never dropped.
third = 4 if p.polarity == "aff" else 3
pitch = max(24, min(96, root + (third if p.confidence >= 0.5 else 0)))
velocity = int(56 + 60 * min(1.0, max(0.0, p.importance)))
velocity = max(40, min(120, velocity))
# confidence -> duration: quarter .. dotted-half
dur = int(_TICKS * (0.5 + 1.5 * min(1.0, max(0.0, p.confidence))))
return pitch, velocity, dur
# a mode-profile: the pluggable musical knob (the peer's mode_profile). Scale +
# tempo. Swapping this profile re-voices the SAME geometry — surface as parameter.
_DEFAULT_PROFILE = {"scale": _C_MAJOR, "tempo_us": 500000,
"registers": [60, 55, 64, 50, 67, 48], "program": 0}
class MidiProjector(TwoStageProjector):
"""geometry -> symbolic music, in the shared two-stage shape.
``plan(frame)`` -> a music_spec: an ordered list of note dicts derived
deterministically from the frame's provenance geometry
(the peer's ``plan(frame, profile) -> spec``).
``realize(spec)`` -> Standard MIDI File bytes (the peer's
``realize(spec, profile) -> surface``; here the surface
is symbolic MIDI, the minimal audio proof — a richer
additive-synth audio projector conforms identically).
"""
surface = "midi"
media_type = "audio/midi"
ext = "mid"
modality = "audio"
def __init__(self, profile: dict | None = None):
self.profile = profile or _DEFAULT_PROFILE
# -- stage 1: meaning-geometry -> music_spec (reads the GEOMETRY facet) -- #
def plan(self, doc: DocumentIR) -> list[dict]:
registers = self.profile["registers"]
spec: list[dict] = []
for si, sec in enumerate(doc.sections):
base = registers[si % len(registers)]
provs = [p for p in sec.all_provenance()
if p.kind in ("fact", "interpretation")]
for i, p in enumerate(provs):
pitch, vel, dur = _note_for(p, base)
spec.append({"pitch": pitch, "velocity": vel, "dur": dur,
"rest_before": (_TICKS // 2) if (si > 0 and i == 0) else 0,
"relation": p.relation, "polarity": p.polarity})
return spec
# -- stage 2: music_spec -> MIDI bytes (own-core, no library) ------------ #
def realize(self, spec: list[dict]) -> bytes:
ev = bytearray()
ev += _vlq(0) + b"\xFF\x51\x03" + struct.pack(">I", self.profile["tempo_us"])[1:]
ev += _vlq(0) + bytes([0xC0, self.profile["program"] & 0x7F])
for note in spec:
ev += _vlq(note["rest_before"]) + bytes([0x90, note["pitch"], note["velocity"]])
ev += _vlq(note["dur"]) + bytes([0x80, note["pitch"], 0])
ev += _vlq(0) + b"\xFF\x2F\x00"
track = bytes(ev)
buf = io.BytesIO()
buf.write(b"MThd" + struct.pack(">IHHH", 6, 0, 1, _TICKS))
buf.write(b"MTrk" + struct.pack(">I", len(track)) + track)
return buf.getvalue()
register(MidiProjector())
+60
View File
@@ -0,0 +1,60 @@
"""seams.py — documented efferent seams for IMAGE and VIDEO surfaces.
These are NOT implemented (per the build rails: architect, do not overbuild).
They are registered as first-class seams so the interface PROVES it accepts
future non-text projectors without any upstream change. Each documents exactly
what its decoder would read from the geometry-carrying IR, making the multimodal
generalization concrete rather than hand-wavy.
The symmetry that guarantees these are possible, not moonshots: they are the
efferent twins of multimodal INGEST. If meaning can HOLD an image (ingest as
first-class geometry), meaning can PROJECT one back. Video = image x sound x
TIME, and the engram already stores time (chronoception). So video falls out of
an image projector + the music projector + the stored temporal ordering.
"""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from document_ir import DocumentIR # noqa: E402
from projectors.base import register # noqa: E402
class _Seam:
"""A registered-but-unimplemented projector. Names its decoder contract."""
def project(self, doc: DocumentIR) -> bytes: # pragma: no cover - seam
raise NotImplementedError(
f"{self.surface!r} projector is a documented seam, not yet built. "
f"Decoder contract: {self.decoder_contract}")
class ImageProjector(_Seam):
surface = "image"
media_type = "image/png"
ext = "png"
modality = "image"
decoder_contract = (
"reads block.provenance as a spatial layout — nodes become regions, edges "
"become adjacencies; salience/importance drive size/contrast; polarity "
"drives figure/ground. The efferent twin of image ingest (a geometry->raster "
"decoder, learned or engineered), exactly mirroring the embedder that turned "
"the image INTO geometry.")
class VideoProjector(_Seam):
surface = "video"
media_type = "video/mp4"
ext = "mp4"
modality = "video"
decoder_contract = (
"image x sound x TIME. Composes the image projector (per-keyframe geometry "
"layout) with the midi/music projector (score) along the geometry's stored "
"temporal ordering (chronoception). Needs no new principle once image + music "
"exist — only a muxer.")
register(ImageProjector())
register(VideoProjector())
+63
View File
@@ -0,0 +1,63 @@
"""provenance.py — the faithfulness audit + geometry->section trace.
A document projected from geometry is only worth anything if every claim traces
back. This module walks the DocumentIR and proves the discipline held:
* ZERO ungrounded claims (every fact/interpretation has a real node id),
* every emitted sentence maps to a geometry edge (or is a marked connective),
* SACRED polarity survived (negations are reported, never silently dropped),
* COHERE introduced no new geometry (connectives carry no claim).
It emits both a machine verdict and a human-readable geometry->section table.
"""
from __future__ import annotations
from document_ir import DocumentIR
def audit(doc: DocumentIR) -> dict:
provs = doc.all_provenance()
facts = [p for p in provs if p.kind in ("fact", "interpretation")]
connectives = [p for p in provs if p.kind == "connective"]
ungrounded = [p for p in facts if not p.node_id]
negations = [p for p in facts if p.polarity == "neg"]
node_ids = sorted({p.node_id for p in facts if p.node_id})
return {
"claims": len(facts),
"connectives": len(connectives),
"ungrounded_claims": len(ungrounded),
"negations_preserved": len(negations),
"distinct_source_nodes": len(node_ids),
"faithful": len(ungrounded) == 0,
"source_nodes": node_ids,
}
def trace_table(doc: DocumentIR) -> str:
"""Human-readable geometry -> section -> claim provenance table."""
lines = ["# Provenance — every claim traces geometry", ""]
lines.append(f"**Document:** {doc.title}")
a = audit(doc)
lines.append(f"**Claims:** {a['claims']} · **Ungrounded:** "
f"{a['ungrounded_claims']} · **Negations preserved:** "
f"{a['negations_preserved']} · **Source nodes:** "
f"{a['distinct_source_nodes']} · **Faithful:** "
f"{'YES' if a['faithful'] else 'NO'}")
lines.append("")
for si, sec in enumerate(doc.sections, 1):
lines.append(f"## {si}. {sec.heading}")
lines.append(f"_seed nodes: {', '.join(i[:8] for i in sec.seed_ids)}_")
lines.append("")
lines.append("| # | realized claim | traces geometry edge |")
lines.append("|---|----------------|----------------------|")
n = 0
for block in sec.blocks:
for sent, prov in zip(block.sentences, block.provenance):
if prov.kind == "connective":
continue
n += 1
edge = prov.trace().replace("|", "\\|")
s = sent.replace("|", "\\|")
lines.append(f"| {n} | {s} | {edge} |")
lines.append("")
return "\n".join(lines) + "\n"
+112
View File
@@ -0,0 +1,112 @@
"""realize.py — REALIZE stage: fill each planned section with faithful passages.
Scales the PROVEN realizer from a single assertion to a passage. For each
planned proposition we build a realizer-ready clause (the proven
``_prop_to_clause`` mapping) and run it through the proven engine
(``engine.realize``), which is a deterministic grammar with the SACRED negation
contract — it never invents. Each realized sentence is paired with a
:class:`Provenance` that pins it to the exact geometry edge it came from.
"Passage, not a list of sentences": within a section we lightly vary sentence
openings and group related claims, but we add NO content the geometry did not
assert. The only non-geometry words are function words the grammar already owns
(articles, "and", conjunction of same-subject claims). Document-level flow is
COHERE's job; this stage owns intra-section fluency + fidelity.
"""
from __future__ import annotations
import os
import sys
_NT = os.path.expanduser("~/Desktop/neuron-talk")
_LR = os.path.expanduser("~/Desktop/lang-realizers")
for _p in (_NT, _LR):
if _p not in sys.path:
sys.path.insert(0, _p)
import engine # noqa: E402 (the proven no-LLM realizer)
from dialogue import _prop_to_clause # noqa: E402 (proven prop -> clause)
from document_ir import Block, DocumentIR, Provenance, Section # noqa: E402
def _provenance_from(p, kind: str = "fact") -> Provenance:
return Provenance(
subj_id=p.source_node_id, subject=p.subject, relation=p.predicate,
obj=p.object, polarity=p.polarity, confidence=round(float(p.confidence), 3),
node_id=p.source_node_id, kind=kind,
importance=float(getattr(p, "node_importance", 0.0) or 0.0),
salience=0.0,
)
import re as _re
# a well-formed declarative opens with a determiner, a proper noun, "I", or a
# capitalized head — not a mis-parsed object pronoun or a copula fragment.
_BAD_OPENERS = _re.compile(r"^(Me |It is I|There is|This is it|That is it)\b")
_VACUOUS = _re.compile(r"^\w+ (is|are|was|were) (it|no|nothing|empty|those|this|that)\.?$",
_re.I)
def _good_sentence(text: str) -> bool:
"""Fluency gate — drops degenerate realizations. NEVER loosens faithfulness;
it only refuses to SPEAK a claim whose surface came out malformed."""
words = text.rstrip(".").split()
if len(words) < 3:
return False
if _BAD_OPENERS.search(text):
return False
if _VACUOUS.match(text):
return False
# a sentence that is mostly one-letter/two-letter tokens is a parse artifact
short = sum(1 for w in words if len(w.strip(".,'")) <= 2)
if short > len(words) / 2:
return False
return True
def _realize_prop(p, lang: str = "en") -> tuple[str, Provenance] | None:
"""One proposition -> (faithful sentence, provenance) or None if it drops."""
clause = _prop_to_clause(p)
text = engine.realize(clause, lang)
if not text or not text.strip():
return None
text = text.strip()
if not text.endswith((".", "!", "?")):
text += "."
# capitalize first character (proper nouns / "I" already handled by grammar)
text = text[0].upper() + text[1:]
if not _good_sentence(text):
return None
return text, _provenance_from(p)
def realize_document(doc: DocumentIR, lang: str = "en") -> DocumentIR:
"""Fill every planned section's blocks with faithful, realized passages."""
for sec in doc.sections:
planned = sec.__dict__.get("_planned_props", [])
block = Block(role="body")
summary_bits: list[str] = []
for p in planned:
r = _realize_prop(p, lang)
if r is None:
continue
text, prov = r
block.sentences.append(text)
block.provenance.append(prov)
if len(summary_bits) < 1:
# a short grounded gloss for TOC / pptx bullets
obj = (prov.obj or "").strip().rstrip(".")
if obj:
summary_bits.append(obj)
if block.sentences:
sec.blocks.append(block)
sec.summary = summary_bits[0] if summary_bits else ""
# drop the transient planning payload; the IR is now self-contained
sec.__dict__.pop("_planned_props", None)
sec.__dict__.pop("_node", None)
# prune sections that realized to nothing
doc.sections = [s for s in doc.sections if s.blocks]
return doc
+136
View File
@@ -0,0 +1,136 @@
// accent.el - A British-RP ACCENT as an INGESTED TRANSFORM-GEOMETRY, composed
// onto the voice (voice (+) accent, SEPARABLE). Reads elp/data/british-accent.psv
// into an accent MANIFOLD in the engram (override nodes + a shared accent hub),
// and the render reads the RP formant overrides + the non-rhotic rule back from
// that geometry. NO accent targets live in code same discipline as the base
// phonetics. PROVENANCE NOTE: the RP Hz values are PROVISIONAL (reconstructed-
// from-knowledge approximations, cite Deterding1997 / Hawkins&Midgley2005 /
// Wells1982) pending transcription from the published tables the PIPELINE is
// the deliverable; exact values are being source-verified separately.
fn ingest_accent(path: String) -> [String] {
let content: String = fs_read(path)
let lines: [String] = str_split(content, "\n")
let nl: Int = native_list_len(lines)
let amap: [String] = native_list_empty()
let hub: String = engram_node("accent british-rp prov=PROVISIONAL cite=Deterding1997-HawkinsMidgley2005-Wells1982", "Accent", 80)
let li: Int = 0
while li < nl {
let line: String = native_list_get(lines, li)
let ll: Int = str_len(line)
let skip: Int = 0
if ll < 3 {
skip = 1
}
if skip == 0 {
let first: Int = str_char_code(line, 0)
if first == 35 {
skip = 1
}
}
if skip == 0 {
let f: [String] = str_split(line, "|")
let nf: Int = native_list_len(f)
if nf >= 6 {
let key: String = native_list_get(f, 0)
let f1: String = native_list_get(f, 1)
let f2: String = native_list_get(f, 2)
let f3: String = native_list_get(f, 3)
let kind: String = native_list_get(f, 4)
let set: String = native_list_get(f, 5)
let cont: String = "accent british-rp " + key + " f1=" + f1 + " f2=" + f2 + " f3=" + f3 + " kind=" + kind + " set=" + set + " prov=PROVISIONAL cite=Deterding1997-HawkinsMidgley2005-Wells1982"
let id: String = engram_node(cont, "AccentTarget", 80)
amap = native_list_append(amap, key)
amap = native_list_append(amap, cont)
engram_connect(id, hub, 80, "of_accent")
}
}
li = li + 1
}
return amap
}
// RP formant override for a phoneme, read from the accent manifold. Returns
// [f1,f2,f3] for a vowel_override record, or an empty list if none / a rule.
fn accent_formants(amap: [String], code: String) -> [Int] {
let out: [Int] = native_list_empty()
let id: String = sp_map_get(amap, code)
if str_eq(id, "") {
return out
}
let j: String = id
let isrule: Int = str_index_of(j, "drop_coda")
if isrule >= 0 {
return out
}
let f1: Int = parse_uint_from(j, "f1=")
if f1 <= 0 {
return out
}
let out = native_list_append(out, f1)
let out = native_list_append(out, parse_uint_from(j, "f2="))
let out = native_list_append(out, parse_uint_from(j, "f3="))
return out
}
// Is this accent non-rhotic? (reads the R rule node from the manifold)
fn is_nonrhotic(amap: [String]) -> Int {
let id: String = sp_map_get(amap, "R")
if str_eq(id, "") {
return 0
}
let hit: Int = str_index_of(id, "drop_coda")
if hit >= 0 {
return 1
}
return 0
}
// Is this symbol a vowel? Membership in the vowel-set derived from the phonetics
// source's class column (phonological structure the FORMANT NUMBERS still come
// from the organ manifold; this is only the categorical class for the rule).
fn is_vowel_sym(vset: [String], sym: String) -> Int {
let n: Int = native_list_len(vset)
let i: Int = 0
while i < n {
if str_eq(native_list_get(vset, i), sym) {
return 1
}
i = i + 1
}
return 0
}
// Non-rhotic transform: drop a post-vocalic CODA /R/ an R whose next non-SIL
// phoneme is NOT a vowel (a consonant, or end of utterance). Keep INTERVOCALIC/
// onset R (next non-SIL phoneme is a vowel, e.g. the medial R in N UW R AA N).
fn apply_rhoticity(codes: [String], vset: [String]) -> [String] {
let n: Int = native_list_len(codes)
let out: [String] = native_list_empty()
let i: Int = 0
while i < n {
let c: String = native_list_get(codes, i)
let keep: Int = 1
if str_eq(c, "R") {
let jx: Int = i + 1
let nextv: Int = 0
while jx < n {
let ncode: String = native_list_get(codes, jx)
if str_eq(ncode, "SIL") {
jx = jx + 1
} else {
nextv = is_vowel_sym(vset, ncode)
jx = n + 1000
}
}
if nextv == 0 {
keep = 0
}
}
if keep == 1 {
out = native_list_append(out, c)
}
i = i + 1
}
return out
}
+73
View File
@@ -0,0 +1,73 @@
// audio-demo.el - Drive the native audio surface: render a tone per instrument
// from its LEARNED signature, then render a small meaning-phrase "piece".
// Entry point: top-level statement calls main() (same convention as the
// examples' top-level println(run_test())).
fn micros_to_str(xs: [Int]) -> String {
let n: Int = native_list_len(xs)
let out: String = ""
let i: Int = 0
while i < n {
if i > 0 { let out: String = out + "," }
let out: String = out + int_to_str(native_list_get(xs, i))
let i: Int = i + 1
}
return out
}
// Render a 1.0s A4 (midi 69) tone from a signature file, print the parsed
// partials (proving the numbers came from the engram .sig), write the WAV.
fn render_tone(name: String, sigpath: String, outpath: String, table: [Int]) -> Int {
let lines: [String] = sig_load(sigpath)
let partials: [Int] = parse_micros(sig_field(lines, "partials"))
println("[" + name + "] partials_n=" + sig_field(lines, "partials_n") + " parsed_partials_micro(scale 1e6)=" + micros_to_str(partials))
println("[" + name + "] raw partials line from .sig = " + sig_field(lines, "partials"))
let freq: Int = freq_of_midi(69)
let note: [Int] = synth_from_sig(lines, freq, 1000, 900, 44100, table)
let n: Int = native_list_len(note)
let ok: Int = wav_write(outpath, note, n, 44100)
println("[" + name + "] rendered " + int_to_str(n) + " samples -> " + outpath + " (write_ok=" + int_to_str(ok) + ")")
return n
}
fn run_demo() -> Int {
let table: [Int] = sin_table()
fs_mkdir("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out")
println("=== TONES: render A4 (midi 69) from each learned signature ===")
render_tone("flute", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/flute.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-flute.wav", table)
render_tone("clarinet", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/clarinet.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-clarinet.wav", table)
render_tone("violin", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/violin.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-violin.wav", table)
render_tone("piano", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/piano.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-piano.wav", table)
render_tone("organ", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/organ.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-organ.wav", table)
println("")
println("=== PIECE: a 6-frame meaning phrase (incl. a NEG frame) ===")
let frames: [[String]] = native_list_empty()
let frames: [[String]] = native_list_append(frames, audio_frame("agent", "aff", "0.9", "0.8", "0", "s1"))
let frames: [[String]] = native_list_append(frames, audio_frame("theme", "aff", "0.7", "0.6", "0", "s2"))
let frames: [[String]] = native_list_append(frames, audio_frame("cause", "aff", "0.8", "0.9", "1", "s3"))
let frames: [[String]] = native_list_append(frames, audio_frame("negation", "neg", "0.85", "0.7", "0", "s4"))
let frames: [[String]] = native_list_append(frames, audio_frame("goal", "aff", "0.6", "0.5", "1", "s5"))
let frames: [[String]] = native_list_append(frames, audio_frame("result", "aff", "0.95", "1.0", "0", "s6"))
// Print the plan so the NEG frame's minor third (+3) vs major (+4) is visible.
let nf: Int = native_list_len(frames)
let fi: Int = 0
while fi < nf {
let frame: [String] = native_list_get(frames, fi)
let plan: [Int] = plan_note(frame)
let pol: String = surface_get(frame, "polarity")
let third_name: String = "major(+4)"
if str_eq(pol, "neg") { let third_name: String = "MINOR(+3)" }
println("frame " + int_to_str(fi) + " relation=" + surface_get(frame, "relation") + " polarity=" + pol + " -> midi=" + int_to_str(native_list_get(plan, 0)) + " dur_ms=" + int_to_str(native_list_get(plan, 1)) + " amp_pm=" + int_to_str(native_list_get(plan, 2)) + " third=" + third_name)
let fi: Int = fi + 1
}
let piano_lines: [String] = sig_load("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/piano.sig")
let total: Int = realize_audio(frames, piano_lines, "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/piece.wav", 44100, table)
println("PIECE rendered " + int_to_str(total) + " samples -> /Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/piece.wav")
return total
}
println("audio-demo main returned samples=" + int_to_str(run_demo()))
+400
View File
@@ -0,0 +1,400 @@
// audio-surface.el - Native own-core additive-synthesis audio surface.
//
// The AUDIO efferent seam, native, no Python and no library. This renders real
// PCM .wav bytes from instrument SIGNATURES read from engram-sourced .sig data
// files (elp/faculty/sig/*.sig) - the partial amplitudes are NEVER literals in
// this source; they are parsed from the learned signature at run time. That is
// the whole proof: render-from-learned-signatures.
//
// EL has no float arithmetic operator (codegen emits raw int64 ops for + - * /
// on the shared 64-bit slot) and no float-arithmetic natives - so ALL synthesis
// math here is own-core INTEGER fixed-point. Angles use a quarter-wave sine
// table (scale 10000) from a fixed-point Taylor series; amplitudes are parsed to
// micro (scale 1e6) straight from the .sig text; frequencies are milliHz ints.
//
// Pipeline mirrors the two-stage projector (midi.py): plan_note(frame) reads a
// frame's meaning-geometry slot-map and derives (pitch, duration, amplitude);
// realize_audio SUPERPOSES the signature's partials (the compose op) and
// serialises RIFF/WAVE. Same frame -> midi OR audio.
// -- integer decimal + string helpers -----------------------------------------
fn str_to_int_el(s: String) -> Int {
let n: Int = str_len(s)
let i: Int = 0
let v: Int = 0
let neg: Bool = false
while i < n {
let c: Int = str_char_code(s, i)
if c == 45 { let neg: Bool = true }
if c >= 48 {
if c < 58 {
let v: Int = v * 10 + (c - 48)
}
}
let i: Int = i + 1
}
if neg { return 0 - v }
return v
}
fn parse_micro(s: String) -> Int {
let dot: Int = str_index_of(s, ".")
if dot < 0 {
return str_to_int_el(s) * 1000000
}
let n: Int = str_len(s)
let ipart: String = str_slice(s, 0, dot)
let fpart: String = str_slice(s, dot + 1, n)
let iv: Int = str_to_int_el(ipart)
let fv: Int = 0
let scale: Int = 100000
let fn2: Int = str_len(fpart)
let i: Int = 0
while i < 6 {
let d: Int = 0
if i < fn2 {
let d: Int = str_char_code(fpart, i) - 48
}
let fv: Int = fv + d * scale
let scale: Int = scale / 10
let i: Int = i + 1
}
return iv * 1000000 + fv
}
// -- signature (engram data file) loader ---------------------------------------
fn sig_load(path: String) -> [String] {
let text: String = fs_read(path)
return str_split(text, "\n")
}
fn sig_field(lines: [String], key: String) -> String {
let pref: String = key + ": "
let n: Int = native_list_len(lines)
let plen: Int = str_len(pref)
let i: Int = 0
while i < n {
let ln: String = native_list_get(lines, i)
if str_starts_with(ln, pref) {
return str_slice(ln, plen, str_len(ln))
}
let i: Int = i + 1
}
return ""
}
fn parse_micros(csv: String) -> [Int] {
let parts: [String] = str_split(csv, ",")
let n: Int = native_list_len(parts)
let out: [Int] = native_list_empty()
let i: Int = 0
while i < n {
let out: [Int] = native_list_append(out, parse_micro(native_list_get(parts, i)))
let i: Int = i + 1
}
return out
}
// -- fixed-point sine (own-core, quarter-wave Taylor table, scale 10000) --------
fn sin_table() -> [Int] {
let HP: Int = 1570796
let t: [Int] = native_list_empty()
let q: Int = 0
while q < 257 {
let x: Int = q * HP / 256
let x2: Int = x * x / 1000000
let x3: Int = x2 * x / 1000000
let x5: Int = x3 * x2 / 1000000
let x7: Int = x5 * x2 / 1000000
let x9: Int = x7 * x2 / 1000000
let s: Int = x - x3 / 6 + x5 / 120 - x7 / 5040 + x9 / 362880
let t: [Int] = native_list_append(t, s / 100)
let q: Int = q + 1
}
return t
}
fn sin_lookup(t: [Int], phase: Int) -> Int {
let p: Int = phase % 1024
if p < 0 { let p: Int = p + 1024 }
let quad: Int = p / 256
let r: Int = p % 256
if quad == 0 { return native_list_get(t, r) }
if quad == 1 { return native_list_get(t, 256 - r) }
if quad == 2 { return 0 - native_list_get(t, r) }
return 0 - native_list_get(t, 256 - r)
}
fn isqrt_int(n: Int) -> Int {
if n <= 0 { return 0 }
let x: Int = n
let y: Int = (x + 1) / 2
while y < x {
let x: Int = y
let y: Int = (x + n / x) / 2
}
return x
}
// freq_of_midi: equal-tempered frequency in milliHz. 440000 mHz at midi 69.
fn freq_of_midi(m: Int) -> Int {
let f: Int = 440000
if m > 69 {
let k: Int = m - 69
let i: Int = 0
while i < k {
let f: Int = f * 1059463 / 1000000
let i: Int = i + 1
}
return f
}
if m < 69 {
let k: Int = 69 - m
let i: Int = 0
while i < k {
let f: Int = f * 1000000 / 1059463
let i: Int = i + 1
}
return f
}
return f
}
// -- envelope (ADSR), scale 1000 -----------------------------------------------
fn adsr_env(i: Int, total: Int, atk_n: Int, dec_n: Int, sus_pm: Int, rel_n: Int) -> Int {
if i < atk_n {
if atk_n == 0 { return 1000 }
return 1000 * i / atk_n
}
if i < atk_n + dec_n {
if dec_n == 0 { return sus_pm }
return 1000 - (1000 - sus_pm) * (i - atk_n) / dec_n
}
let rel_start: Int = total - rel_n
if i < rel_start {
return sus_pm
}
if rel_n == 0 { return 0 }
let left: Int = total - i
return sus_pm * left / rel_n
}
// -- note synthesis: SUPERPOSE the learned partials -> [Int] samples -----------
fn note_samples(freq_mHz: Int, dur_ms: Int, rate: Int, partials: [Int], sumP: Int, b_micro: Int, vib_rate: Int, vib_cents: Int, atk_ms: Int, dec_ms: Int, sus_pm: Int, rel_ms: Int, amp_pm: Int, table: [Int]) -> [Int] {
let total: Int = dur_ms * rate / 1000
let atk_n: Int = atk_ms * rate / 1000
let dec_n: Int = dec_ms * rate / 1000
let rel_n: Int = rel_ms * rate / 1000
let np: Int = native_list_len(partials)
let half_mhz: Int = rate * 1000 / 2
let out: [Int] = native_list_empty()
let i: Int = 0
while i < total {
let acc: Int = 0
let k: Int = 0
while k < np {
let harm: Int = k + 1
let amp_k: Int = native_list_get(partials, k)
let factor: Int = 1000000
if b_micro > 0 {
let val: Int = 1000000 + b_micro * harm * harm
let factor: Int = isqrt_int(val * 1000000)
}
let fn_mhz: Int = freq_mHz * harm
let fn_mhz: Int = fn_mhz * factor / 1000000
if vib_cents > 0 {
if vib_rate > 0 {
let vphase: Int = i * vib_rate * 1024 / rate
let vs: Int = sin_lookup(table, vphase)
let vibf: Int = 1000000 + (vib_cents * vs * 833) / 10000
let fn_mhz: Int = fn_mhz * vibf / 1000000
}
}
if fn_mhz <= half_mhz {
let phase: Int = i * fn_mhz * 1024 / (rate * 1000)
let sv: Int = sin_lookup(table, phase)
let acc: Int = acc + sv * amp_k / 1000000
}
let k: Int = k + 1
}
let env: Int = adsr_env(i, total, atk_n, dec_n, sus_pm, rel_n)
let s16: Int = acc * 2800000 / sumP
let s16: Int = s16 * env / 1000
let s16: Int = s16 * amp_pm / 1000
if s16 > 32767 { let s16: Int = 32767 }
if s16 < 0 - 32767 { let s16: Int = 0 - 32767 }
let out: [Int] = native_list_append(out, s16)
let i: Int = i + 1
}
return out
}
fn synth_from_sig(lines: [String], freq_mHz: Int, dur_ms: Int, amp_pm: Int, rate: Int, table: [Int]) -> [Int] {
let partials: [Int] = parse_micros(sig_field(lines, "partials"))
let np: Int = native_list_len(partials)
let sumP: Int = 0
let j: Int = 0
while j < np {
let pj: Int = native_list_get(partials, j)
let sumP: Int = sumP + pj
let j: Int = j + 1
}
if sumP <= 0 { let sumP: Int = 1000000 }
let adsr: [String] = str_split(sig_field(lines, "adsr"), ",")
let atk_ms: Int = parse_micro(native_list_get(adsr, 0)) / 1000
let dec_ms: Int = parse_micro(native_list_get(adsr, 1)) / 1000
let sus_pm: Int = parse_micro(native_list_get(adsr, 2)) / 1000
let rel_ms: Int = parse_micro(native_list_get(adsr, 3)) / 1000
let b_micro: Int = parse_micro(sig_field(lines, "inharmonicity_B"))
let vib_rate: Int = str_to_int_el(sig_field(lines, "vibrato_rate_hz"))
let vib_cents: Int = str_to_int_el(sig_field(lines, "vibrato_depth_cents"))
return note_samples(freq_mHz, dur_ms, rate, partials, sumP, b_micro, vib_rate, vib_cents, atk_ms, dec_ms, sus_pm, rel_ms, amp_pm, table)
}
// -- byte-buffer helpers (own-core, no library) --------------------------------
fn put_tag(buf: String, pos: Int, s: String) -> String {
let n: Int = str_len(s)
let i: Int = 0
while i < n {
let buf: String = __str_set_char(buf, pos + i, str_char_code(s, i))
let i: Int = i + 1
}
return buf
}
fn put_u32le(buf: String, pos: Int, v: Int) -> String {
let buf: String = __str_set_char(buf, pos, v % 256)
let buf: String = __str_set_char(buf, pos + 1, (v / 256) % 256)
let buf: String = __str_set_char(buf, pos + 2, (v / 65536) % 256)
let buf: String = __str_set_char(buf, pos + 3, (v / 16777216) % 256)
return buf
}
fn put_u16le(buf: String, pos: Int, v: Int) -> String {
let buf: String = __str_set_char(buf, pos, v % 256)
let buf: String = __str_set_char(buf, pos + 1, (v / 256) % 256)
return buf
}
// -- WAV serializer: own-core RIFF/WAVE, PCM mono 16-bit -----------------------
fn wav_write(path: String, samples: [Int], n: Int, rate: Int) -> Int {
let data_len: Int = n * 2
let total: Int = 44 + data_len
let buf: String = __str_alloc(total)
let buf: String = put_tag(buf, 0, "RIFF")
let buf: String = put_u32le(buf, 4, 36 + data_len)
let buf: String = put_tag(buf, 8, "WAVE")
let buf: String = put_tag(buf, 12, "fmt ")
let buf: String = put_u32le(buf, 16, 16)
let buf: String = put_u16le(buf, 20, 1)
let buf: String = put_u16le(buf, 22, 1)
let buf: String = put_u32le(buf, 24, rate)
let buf: String = put_u32le(buf, 28, rate * 2)
let buf: String = put_u16le(buf, 32, 2)
let buf: String = put_u16le(buf, 34, 16)
let buf: String = put_tag(buf, 36, "data")
let buf: String = put_u32le(buf, 40, data_len)
let i: Int = 0
while i < n {
let v: Int = native_list_get(samples, i)
if v < 0 { let v: Int = v + 65536 }
let buf: String = __str_set_char(buf, 44 + i * 2, v % 256)
let buf: String = __str_set_char(buf, 44 + i * 2 + 1, (v / 256) % 256)
let i: Int = i + 1
}
let ok: Int = fs_write_bytes(path, buf, total)
return ok
}
// -- plan: frame slot-map -> note atom (pitch, duration, amplitude) ------------
fn audio_frame(relation: String, polarity: String, confidence: String, importance: String, salience: String, subj_id: String) -> [String] {
let f: [String] = native_list_empty()
let f: [String] = native_list_append(f, "relation")
let f: [String] = native_list_append(f, relation)
let f: [String] = native_list_append(f, "polarity")
let f: [String] = native_list_append(f, polarity)
let f: [String] = native_list_append(f, "confidence")
let f: [String] = native_list_append(f, confidence)
let f: [String] = native_list_append(f, "importance")
let f: [String] = native_list_append(f, importance)
let f: [String] = native_list_append(f, "salience")
let f: [String] = native_list_append(f, salience)
let f: [String] = native_list_append(f, "subj_id")
let f: [String] = native_list_append(f, subj_id)
return f
}
fn degree_offset(deg: Int) -> Int {
if deg == 0 { return 0 }
if deg == 1 { return 2 }
if deg == 2 { return 4 }
if deg == 3 { return 5 }
if deg == 4 { return 7 }
if deg == 5 { return 9 }
return 11
}
// returns [midi, dur_ms, amp_pm]
fn plan_note(frame: [String]) -> [Int] {
let relation: String = surface_get(frame, "relation")
let polarity: String = surface_get(frame, "polarity")
let confidence: String = surface_get(frame, "confidence")
let importance: String = surface_get(frame, "importance")
let salience: String = surface_get(frame, "salience")
let rn: Int = str_len(relation)
let csum: Int = 0
let i: Int = 0
while i < rn {
let cc: Int = str_char_code(relation, i)
let csum: Int = csum + cc
let i: Int = i + 1
}
let deg: Int = csum % 7
let third: Int = 4
if str_eq(polarity, "neg") { let third: Int = 3 }
let sal_oct: Int = str_to_int_el(salience)
let doff: Int = degree_offset(deg)
let midi: Int = 60 + sal_oct * 12 + doff + third
let conf_micro: Int = parse_micro(confidence)
let dur_ms: Int = 200 + conf_micro / 1000
let imp_micro: Int = parse_micro(importance)
let amp_pm: Int = 400 + imp_micro / 2000
let out: [Int] = native_list_empty()
let out: [Int] = native_list_append(out, midi)
let out: [Int] = native_list_append(out, dur_ms)
let out: [Int] = native_list_append(out, amp_pm)
return out
}
fn realize_audio(frames: [[String]], sig_lines: [String], path: String, rate: Int, table: [Int]) -> Int {
let nf: Int = native_list_len(frames)
let all: [Int] = native_list_empty()
let count: Int = 0
let fi: Int = 0
while fi < nf {
let frame: [String] = native_list_get(frames, fi)
let plan: [Int] = plan_note(frame)
let midi: Int = native_list_get(plan, 0)
let dur_ms: Int = native_list_get(plan, 1)
let amp_pm: Int = native_list_get(plan, 2)
let freq: Int = freq_of_midi(midi)
let note: [Int] = synth_from_sig(sig_lines, freq, dur_ms, amp_pm, rate, table)
let nn: Int = native_list_len(note)
let j: Int = 0
while j < nn {
let all: [Int] = native_list_append(all, native_list_get(note, j))
let j: Int = j + 1
}
let count: Int = count + nn
let fi: Int = fi + 1
}
let ok: Int = wav_write(path, all, count, rate)
return count
}
File diff suppressed because it is too large Load Diff
+16
View File
@@ -0,0 +1,16 @@
// comprehend.elh — public surface of the ELP comprehension front-end.
// text → meaning-spec (the input half of the ELP; inverse of the realizer).
extern fn parse_spec(text: String) -> [String]
extern fn parse_spec_lang(text: String, lang: String) -> [String]
extern fn parse_json(text: String) -> String
extern fn parse_json_lang(text: String, lang: String) -> String
// Analysis primitives (invertible morphology + deterministic grammar helpers):
extern fn cp_tokenize(text: String) -> [String]
extern fn cp_pron_concept(w: String) -> String
extern fn cp_is_negation(w: String) -> Bool
extern fn cp_is_neg_adverb(w: String) -> Bool
extern fn cp_irr2(surface: String) -> [String]
extern fn cp_reg_verb(w: String) -> [String]
extern fn cp_analyze_verb(surface: String) -> [String]
extern fn cp_verb_start(toks: [String], end: Int) -> Int
extern fn cp_subord_start(toks: [String], n: Int) -> Int
+287
View File
@@ -0,0 +1,287 @@
// dialogue.el SUMMON-THROUGH-SELF, native el. Port of dialogue.py's core.
//
// THE WHOLE DIALOGUE IS ONE OPERATION. A fact is never merely *fetched*: the
// query is PROJECTED into the engram's self + memory geometry, LANDS in a region,
// and the reply is READ OUT / the region MATERIALIZED from wherever it landed.
//
// project(query) -> land on a region -> read out from that region
//
// lands in the SELF region -> grounded identity/presence, read out of
// the real self nodes (self_region.el)
// lands on a memory NEIGHBORHOOD -> MATERIALIZE it: walk the neighborhood
// (engram_neighbors_json) and read out the
// region's connected members
// lands nowhere close -> HONEST ABSENCE (an empty region, not a
// fabricated answer, not an error)
//
// CRITICAL INVARIANTS (enforced structurally, not by convention):
// * ONE operation there is NO intent classifier and NO separate
// fact-retrieval branch. Identity is nearest-region proximity, not a switch.
// * MATERIALIZE by walking the neighborhood, never by fetching top-props.
// * HONEST ABSENCE when the region is thin.
// * NEGATION is SACRED: the readout is the stored prose VERBATIM, so a negated
// memory stays negated we never paraphrase a polarity away.
// * NO ECHO: the old "I noted that X. That relates to Y." template is gone.
// The summon path materializes or honestly declines it never echoes.
// * DIRECTIVE OVERRIDE: a meta-directive ("answer in English") overrides the
// reply language while the content language is still auto-detected.
//
// Depends on: comprehend (parse_spec_lang, cp_tokenize), multilingual (ml_detect,
// ml_tr, ml_term), propositions (prop_split_sentences), self_region
// (sr_available, sr_readout), the engram + json runtime builtins.
// directive override
// Return [target_lang, content]. target_lang is "" when no directive is present.
// A directive names an output language; we strip it and keep the remaining text
// as the content (whose OWN language is still auto-detected downstream).
fn dlg_dir_hit(low: String, phrase: String) -> Bool {
return str_contains(low, phrase)
}
fn dlg_parse_directive(text: String) -> [String] {
let low: String = str_to_lower(text)
let lang: String = ""
let phrase: String = ""
// English target
if dlg_dir_hit(low, "in english") { let lang = "en"; let phrase = "in english" }
if dlg_dir_hit(low, "em inglês") { let lang = "en"; let phrase = "em inglês" }
if dlg_dir_hit(low, "em ingles") { let lang = "en"; let phrase = "em ingles" }
if dlg_dir_hit(low, "en inglés") { let lang = "en"; let phrase = "en inglés" }
// Portuguese target
if dlg_dir_hit(low, "in portuguese") { let lang = "pt"; let phrase = "in portuguese" }
if dlg_dir_hit(low, "em português") { let lang = "pt"; let phrase = "em português" }
// Spanish target
if dlg_dir_hit(low, "in spanish") { let lang = "es"; let phrase = "in spanish" }
if dlg_dir_hit(low, "en español") { let lang = "es"; let phrase = "en español" }
// Italian target
if dlg_dir_hit(low, "in italian") { let lang = "it"; let phrase = "in italian" }
let content: String = text
if !str_eq(phrase, "") {
// strip the directive phrase (and a common "answer"/"responda" lead-in),
// leaving the real question as content.
let idx: Int = str_index_of(low, phrase)
if idx >= 0 {
let before: String = str_slice(text, 0, idx)
let after: String = str_slice(text, idx + str_len(phrase), str_len(text))
let content = str_trim(before + " " + after)
}
// trim a leading "answer"/"responda"/"reply" and stray colon/comma.
let cl: String = str_to_lower(content)
if str_starts_with(cl, "answer") { let content = str_trim(str_slice(content, 6, str_len(content))) }
if str_starts_with(cl, "responda") { let content = str_trim(str_slice(content, 8, str_len(content))) }
if str_starts_with(cl, "reply") { let content = str_trim(str_slice(content, 5, str_len(content))) }
if str_starts_with(content, ":") { let content = str_trim(str_slice(content, 1, str_len(content))) }
if str_starts_with(content, ",") { let content = str_trim(str_slice(content, 1, str_len(content))) }
}
let r: [String] = native_list_empty()
let r = native_list_append(r, lang)
let r = native_list_append(r, content)
return r
}
// identity landing (a region proximity, not a classifier switch)
// The query lands in the SELF region when it takes an identity/presence shape.
// Cross-lingual forms are included because the engram's lexical probe is
// English-leaning. This is the SELF attractor of the single operation.
fn dlg_is_identity(content: String) -> Bool {
let low: String = str_to_lower(str_trim(content))
if str_contains(low, "who are you") { return true }
if str_contains(low, "what are you") { return true }
if str_contains(low, "who i am") { return true }
if str_contains(low, "your name") { return true }
if str_contains(low, "about yourself") { return true }
if str_contains(low, "are you conscious") { return true }
if str_contains(low, "are you there") { return true }
// cross-lingual identity question-forms
if str_contains(low, "quem é você") { return true }
if str_contains(low, "quem es voce") { return true }
if str_contains(low, "quién eres") { return true }
if str_contains(low, "quien eres") { return true }
if str_contains(low, "chi sei") { return true }
if str_contains(low, "qui es-tu") { return true }
if str_contains(low, "wer bist du") { return true }
return false
}
// readout helpers
fn dlg_first_sentence(content: String) -> String {
let sents: [String] = prop_split_sentences(content)
let n: Int = native_list_len(sents)
let i: Int = 0
while i < n {
let s: String = str_trim(native_list_get(sents, i))
// drop a leading markdown heading marker for a clean read-out line
if str_starts_with(s, "# ") { let s = str_trim(str_slice(s, 2, str_len(s))) }
if str_len(s) > 0 { return s }
let i = i + 1
}
return str_trim(content)
}
// strip trailing/leading punctuation from a token.
fn dlg_clean_tok(w: String) -> String {
let s: String = str_trim(w)
let s = str_strip_suffix(s, ".")
let s = str_strip_suffix(s, ",")
let s = str_strip_suffix(s, "?")
let s = str_strip_suffix(s, "!")
let s = str_strip_suffix(s, ":")
let s = str_strip_suffix(s, ";")
return str_trim(s)
}
// closed-class across the supported languages (union) a word we must NOT treat
// as a retrieval topic. Also drops the meta verbs of a request ("tell", "prove",
// "show") so the TOPIC, not the speech act, is what projects into memory.
fn dlg_is_stop(w: String) -> Bool {
if ml_stop_en(w) { return true }
if ml_stop_es(w) { return true }
if ml_stop_pt(w) { return true }
if ml_stop_it(w) { return true }
if str_eq(w, "tell") { return true }
if str_eq(w, "show") { return true }
if str_eq(w, "about") { return true }
if str_eq(w, "sobre") { return true }
if str_eq(w, "acerca") { return true }
return false
}
// The CONTENT TERMS the query projects into memory: content words only, cleaned,
// cross-lingually mapped to the engram's English vocabulary, 3 chars. This is
// the geometry probe the speech-act verbs and function words are stripped so a
// PP topic ("tell me ABOUT Lisbon") projects on "lisbon", not "tell"/"me".
fn dlg_content_terms(content: String, lang: String) -> [String] {
let toks: [String] = cp_tokenize(content)
let n: Int = native_list_len(toks)
let out: [String] = native_list_empty()
let i: Int = 0
while i < n {
let w: String = str_to_lower(dlg_clean_tok(native_list_get(toks, i)))
if str_len(w) >= 3 {
if !dlg_is_stop(w) {
let out = native_list_append(out, ml_term(w, lang))
}
}
let i = i + 1
}
return out
}
// Does this landed node lexically overlap the query's content terms? This is the
// RELEVANCE FLOOR: activation always returns the store's most salient nodes, so
// without this a query about nothing would "land" on the self/top node. A node
// that shares no content term with the query is "nowhere close" -> honest absence.
fn dlg_node_matches(node: String, terms: [String]) -> Bool {
let hay: String = str_to_lower(json_get_string(node, "content") + " " + json_get_string(node, "label"))
let n: Int = native_list_len(terms)
let i: Int = 0
while i < n {
let t: String = native_list_get(terms, i)
if str_len(t) >= 3 {
if str_contains(hay, t) { return true }
}
let i = i + 1
}
return false
}
// MATERIALIZE the landed region: read out the landed fact, then WALK the
// neighborhood and read out its connected members (real edges, not top-props).
fn dlg_materialize(top_node: String, reply_lang: String) -> String {
let id: String = json_get_string(top_node, "id")
let content: String = json_get_string(top_node, "content")
let lead: String = dlg_first_sentence(content)
let nb: String = engram_neighbors_json(id, 2, "both")
let m: Int = json_array_len(nb)
let parts: [String] = native_list_empty()
let parts = native_list_append(parts, lead)
let added: Int = 0
let i: Int = 0
while i < m {
if added < 3 {
let rec: String = json_array_get(nb, i)
let node: String = json_get_raw(rec, "node")
let nc: String = json_get_string(node, "content")
if !str_eq(nc, "") {
let sent: String = dlg_first_sentence(nc)
if !str_eq(sent, "") {
let parts = native_list_append(parts, sent)
let added = added + 1
}
}
}
let i = i + 1
}
// The readout is the region's OWN prose, verbatim negation SACRED, no echo.
return str_join(parts, " ")
}
// THE single operation
fn dlg_respond(text: String) -> String {
// directive override: reply language may differ from content language.
let dir: [String] = dlg_parse_directive(text)
let target_lang: String = native_list_get(dir, 0)
let content: String = native_list_get(dir, 1)
let content_lang: String = ml_detect(content)
let reply_lang: String = content_lang
if !str_eq(target_lang, "") { let reply_lang = target_lang }
// comprehend the content (SACRED polarity carried in the spec).
let spec: [String] = parse_spec_lang(content, content_lang)
// PROJECT + LAND: SELF region
// Identity/presence shape lands in the self region; read out the REAL self
// nodes (self_region.el), never a template. Same single operation this is
// just the self attractor winning the landing.
if dlg_is_identity(content) {
if sr_available() {
// read out the REAL self nodes when replying in their own language
// (the soul's prose is English); for another reply language we cannot
// translate real content without an LLM, so we answer with the
// localized SACRED identity anchor honest, in-language, no fabrication.
if str_eq(reply_lang, "en") { return sr_readout("en") }
return ml_tr("identity", reply_lang)
}
// self region thin honest localized identity (logged fallback shape).
return ml_tr("identity", reply_lang)
}
// PROJECT into MEMORY geometry
let terms: [String] = dlg_content_terms(content, content_lang)
let qterm: String = str_join(terms, " ")
let act: String = engram_activate_json(qterm, 12)
let n: Int = json_array_len(act)
// LAND: the highest-activation node that ACTUALLY overlaps the query's
// content terms (the relevance floor). Activation always returns the most
// salient nodes, so we walk the ranked list and take the first that is
// genuinely "close"; if none is, the query landed nowhere. ───────────────
let landing: String = ""
let i: Int = 0
while i < n {
if str_eq(landing, "") {
let rec: String = json_array_get(act, i)
let node: String = json_get_raw(rec, "node")
if dlg_node_matches(node, terms) {
let landing = node
}
}
let i = i + 1
}
// HONEST ABSENCE: nothing close an empty region, not a fabricated answer,
// not an "I noted that" echo.
if str_eq(landing, "") {
return ml_tr("no_memory", reply_lang)
}
// MATERIALIZE the landing by WALKING its neighborhood.
return dlg_materialize(landing, reply_lang)
}
+13
View File
@@ -63,6 +63,9 @@ import "morphology-cop.el"
import "grammar.el"
import "realizer.el"
import "semantics.el"
// Comprehension front-end (input half: text meaning-spec)
import "comprehend.el"
//
// Entry points:
//
@@ -117,6 +120,9 @@ fn build_form_from_json(semantic_form_json: String, lang_code: String) -> [Strin
let location: String = sem_get(semantic_form_json, "location")
let tense: String = sem_get(semantic_form_json, "tense")
let aspect: String = sem_get(semantic_form_json, "aspect")
let polarity: String = sem_get(semantic_form_json, "polarity")
let neg_word: String = sem_get(semantic_form_json, "neg_word")
let iobj: String = sem_get(semantic_form_json, "iobj")
let form: [String] = native_list_empty()
let form = native_list_append(form, "intent")
@@ -127,12 +133,19 @@ fn build_form_from_json(semantic_form_json: String, lang_code: String) -> [Strin
let form = native_list_append(form, predicate)
let form = native_list_append(form, "patient")
let form = native_list_append(form, patient)
let form = native_list_append(form, "iobj")
let form = native_list_append(form, iobj)
let form = native_list_append(form, "location")
let form = native_list_append(form, location)
let form = native_list_append(form, "tense")
let form = native_list_append(form, tense)
let form = native_list_append(form, "aspect")
let form = native_list_append(form, aspect)
// SACRED: polarity crosses the JSON boundary and is never inferred away.
let form = native_list_append(form, "polarity")
let form = native_list_append(form, polarity)
let form = native_list_append(form, "neg_word")
let form = native_list_append(form, neg_word)
let form = native_list_append(form, "lang")
let form = native_list_append(form, lang_code)
+3 -3
View File
@@ -1,7 +1,7 @@
// auto-generated by elc --emit-header — do not edit
extern fn sem_get(json: String, key: String) -> String
extern fn generate_frame(frame: Any) -> String
extern fn generate_frame_lang(frame: Any, lang_code: String) -> String
extern fn build_form_from_json(semantic_form_json: String, lang_code: String) -> Any
extern fn generate_frame(frame: [String]) -> String
extern fn generate_frame_lang(frame: [String], lang_code: String) -> String
extern fn build_form_from_json(semantic_form_json: String, lang_code: String) -> [String]
extern fn generate(semantic_form_json: String) -> String
extern fn generate_lang(semantic_form_json: String, lang_code: String) -> String
+28 -28
View File
@@ -1,22 +1,22 @@
// auto-generated by elc --emit-header - do not edit
extern fn slots_get(slots: Any, key: String) -> String
extern fn slots_set(slots: Any, key: String, val: String) -> Any
extern fn make_slots(k0: String, v0: String) -> Any
extern fn make_slots2(k0: String, v0: String, k1: String, v1: String) -> Any
extern fn make_slots3(k0: String, v0: String, k1: String, v1: String, k2: String, v2: String) -> Any
extern fn make_slots4(k0: String, v0: String, k1: String, v1: String, k2: String, v2: String, k3: String, v3: String) -> Any
extern fn make_slots5(k0: String, v0: String, k1: String, v1: String, k2: String, v2: String, k3: String, v3: String, k4: String, v4: String) -> Any
extern fn rule_id(rule: Any) -> String
extern fn rule_lhs(rule: Any) -> String
extern fn rule_rhs_len(rule: Any) -> Int
extern fn rule_rhs(rule: Any, idx: Int) -> String
extern fn make_rule(id: String, lhs: String, r0: String) -> Any
extern fn make_rule2(id: String, lhs: String, r0: String, r1: String) -> Any
extern fn make_rule3(id: String, lhs: String, r0: String, r1: String, r2: String) -> Any
extern fn make_rule4(id: String, lhs: String, r0: String, r1: String, r2: String, r3: String) -> Any
extern fn build_rules() -> Any
extern fn get_rules() -> Any
extern fn find_rule(rule_id_str: String) -> Any
// auto-generated by elc --emit-header do not edit
extern fn slots_get(slots: [String], key: String) -> String
extern fn slots_set(slots: [String], key: String, val: String) -> [String]
extern fn make_slots(k0: String, v0: String) -> [String]
extern fn make_slots2(k0: String, v0: String, k1: String, v1: String) -> [String]
extern fn make_slots3(k0: String, v0: String, k1: String, v1: String, k2: String, v2: String) -> [String]
extern fn make_slots4(k0: String, v0: String, k1: String, v1: String, k2: String, v2: String, k3: String, v3: String) -> [String]
extern fn make_slots5(k0: String, v0: String, k1: String, v1: String, k2: String, v2: String, k3: String, v3: String, k4: String, v4: String) -> [String]
extern fn rule_id(rule: [String]) -> String
extern fn rule_lhs(rule: [String]) -> String
extern fn rule_rhs_len(rule: [String]) -> Int
extern fn rule_rhs(rule: [String], idx: Int) -> String
extern fn make_rule(id: String, lhs: String, r0: String) -> [String]
extern fn make_rule2(id: String, lhs: String, r0: String, r1: String) -> [String]
extern fn make_rule3(id: String, lhs: String, r0: String, r1: String, r2: String) -> [String]
extern fn make_rule4(id: String, lhs: String, r0: String, r1: String, r2: String, r3: String) -> [String]
extern fn build_rules() -> [[String]]
extern fn get_rules() -> [[String]]
extern fn find_rule(rule_id_str: String) -> [String]
extern fn make_leaf(label: String, word: String) -> String
extern fn make_node1(label: String, child0: String) -> String
extern fn make_node2(label: String, child0: String, child1: String) -> String
@@ -24,15 +24,15 @@ extern fn make_node3(label: String, child0: String, child1: String, child2: Stri
extern fn make_node4(label: String, child0: String, child1: String, child2: String, child3: String) -> String
extern fn nlg_is_ws(c: String) -> Bool
extern fn skip_ws(s: String, pos: Int) -> Int
extern fn scan_token(s: String, start: Int) -> Any
extern fn scan_token(s: String, start: Int) -> [String]
extern fn render_tree(tree: String) -> String
extern fn gram_word_order(profile: Any) -> String
extern fn gram_order_constituents(subj: String, verb: String, obj: String, profile: Any) -> String
extern fn gram_build_vp(verb: String, aux: String, profile: Any) -> String
extern fn gram_question_strategy(profile: Any) -> String
extern fn gram_word_order(profile: [String]) -> String
extern fn gram_order_constituents(subj: String, verb: String, obj: String, profile: [String]) -> String
extern fn gram_build_vp(verb: String, aux: String, profile: [String]) -> String
extern fn gram_question_strategy(profile: [String]) -> String
extern fn is_pronoun(word: String) -> Bool
extern fn build_np(referent: String, slots: Any) -> String
extern fn build_np(referent: String, slots: [String]) -> String
extern fn build_pp(loc: String) -> String
extern fn build_vp_body(slots: Any) -> String
extern fn build_vp_from_slots(slots: Any) -> String
extern fn generate_tree(rule_id_str: String, slots: Any) -> String
extern fn build_vp_body(slots: [String]) -> String
extern fn build_vp_from_slots(slots: [String]) -> String
extern fn generate_tree(rule_id_str: String, slots: [String]) -> String
+65
View File
@@ -0,0 +1,65 @@
// image-demo.el - Drive the native PNG surface: plan a scene from a small
// meaning phrase (incl. a NEG frame) and emit a byte-valid 64x64 PNG whose
// palette is read from elp/faculty/sig/scene.basis.
fn img_frame(relation: String, polarity: String, confidence: String, importance: String, salience: String, subj_id: String) -> [String] {
let f: [String] = native_list_empty()
let f: [String] = native_list_append(f, "relation")
let f: [String] = native_list_append(f, relation)
let f: [String] = native_list_append(f, "polarity")
let f: [String] = native_list_append(f, polarity)
let f: [String] = native_list_append(f, "confidence")
let f: [String] = native_list_append(f, confidence)
let f: [String] = native_list_append(f, "importance")
let f: [String] = native_list_append(f, importance)
let f: [String] = native_list_append(f, "salience")
let f: [String] = native_list_append(f, salience)
let f: [String] = native_list_append(f, "subj_id")
let f: [String] = native_list_append(f, subj_id)
return f
}
fn rgb_str(c: [Int]) -> String {
return int_to_str(native_list_get(c, 0)) + "," + int_to_str(native_list_get(c, 1)) + "," + int_to_str(native_list_get(c, 2))
}
fn run_image() -> Int {
fs_mkdir("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out")
let table: [Int] = crc_table()
println("crc_table[1]=" + int_to_str(native_list_get(table, 1)) + " (expect 1996959894 / 0x77073096)")
let basis: [String] = basis_load("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/scene.basis")
let warm: [Int] = parse_rgb(basis_field(basis, "warm"))
let cool: [Int] = parse_rgb(basis_field(basis, "cool"))
let bg: [Int] = parse_rgb(basis_field(basis, "bg"))
println("basis warm=" + rgb_str(warm) + " cool=" + rgb_str(cool) + " bg=" + rgb_str(bg) + " (read from scene.basis)")
let frames: [[String]] = native_list_empty()
let frames: [[String]] = native_list_append(frames, img_frame("agent", "aff", "0.9", "0.8", "0", "s1"))
let frames: [[String]] = native_list_append(frames, img_frame("theme", "aff", "0.7", "0.6", "1", "s2"))
let frames: [[String]] = native_list_append(frames, img_frame("cause", "aff", "0.8", "0.9", "0", "s3"))
let frames: [[String]] = native_list_append(frames, img_frame("negation", "neg", "0.85", "0.7", "1", "s4"))
let frames: [[String]] = native_list_append(frames, img_frame("goal", "aff", "0.6", "0.5", "0", "s5"))
let frames: [[String]] = native_list_append(frames, img_frame("result", "aff", "0.95", "1.0", "1", "s6"))
let shapes: [[Int]] = plan_scene(frames, warm, cool)
let ns: Int = native_list_len(shapes)
println("planned " + int_to_str(ns) + " shapes:")
let si: Int = 0
while si < ns {
let sh: [Int] = native_list_get(shapes, si)
let pol: String = surface_get(native_list_get(frames, si), "polarity")
println(" shape " + int_to_str(si) + " type=" + int_to_str(native_list_get(sh, 0)) + " x=" + int_to_str(native_list_get(sh, 1)) + " y=" + int_to_str(native_list_get(sh, 2)) + " size=" + int_to_str(native_list_get(sh, 3)) + " rgb=" + int_to_str(native_list_get(sh, 4)) + "," + int_to_str(native_list_get(sh, 5)) + "," + int_to_str(native_list_get(sh, 6)) + " polarity=" + pol)
let si: Int = si + 1
}
let raw: [Int] = rasterize(64, 64, shapes, bg)
println("rasterized raw (filtered scanlines) bytes=" + int_to_str(native_list_len(raw)) + " (expect 12352)")
let png: [Int] = png_build(64, 64, raw, table)
let plen: Int = native_list_len(png)
let ok: Int = png_write("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/scene.png", png)
println("PNG bytes=" + int_to_str(plen) + " -> /Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/scene.png (write_ok=" + int_to_str(ok) + ")")
return plen
}
println("image-demo returned png_bytes=" + int_to_str(run_image()))
+412
View File
@@ -0,0 +1,412 @@
// image-surface.el - Native own-core raster PNG surface (the image efferent
// twin of audio). Renders a 64x64 RGB scene deterministically from a frame's
// meaning-geometry, then serialises a byte-valid PNG entirely own-core:
// 8-byte magic, IHDR, IDAT (zlib STORED/uncompressed DEFLATE + Adler32), IEND,
// with a per-chunk CRC32 computed via software xor32 (EL has no bitwise ops).
//
// The RGB palette basis is read from elp/faculty/sig/scene.basis (data, not
// literals) - the same read-from-learned discipline as the audio signatures.
// Integer-only throughout; pixels are composed functionally (painter's order)
// so no list mutation is needed.
// -- small int/parse helpers (self-contained) ----------------------------------
fn i_str_to_int(s: String) -> Int {
let n: Int = str_len(s)
let i: Int = 0
let v: Int = 0
while i < n {
let c: Int = str_char_code(s, i)
if c >= 48 {
if c < 58 {
let v: Int = v * 10 + (c - 48)
}
}
let i: Int = i + 1
}
return v
}
fn basis_load(path: String) -> [String] {
return str_split(fs_read(path), "\n")
}
fn basis_field(lines: [String], key: String) -> String {
let pref: String = key + ": "
let n: Int = native_list_len(lines)
let plen: Int = str_len(pref)
let i: Int = 0
while i < n {
let ln: String = native_list_get(lines, i)
if str_starts_with(ln, pref) {
return str_slice(ln, plen, str_len(ln))
}
let i: Int = i + 1
}
return ""
}
fn parse_rgb(csv: String) -> [Int] {
let parts: [String] = str_split(csv, ",")
let out: [Int] = native_list_empty()
let n: Int = native_list_len(parts)
let i: Int = 0
while i < n {
let v: Int = i_str_to_int(native_list_get(parts, i))
let out: [Int] = native_list_append(out, v)
let i: Int = i + 1
}
return out
}
// -- software 32-bit XOR (no bitwise ops in EL) --------------------------------
fn xor32(a: Int, b: Int) -> Int {
let r: Int = 0
let bit: Int = 1
let i: Int = 0
while i < 32 {
let abit: Int = (a / bit) % 2
let bbit: Int = (b / bit) % 2
if abit != bbit {
let add: Int = bit
let r: Int = r + add
}
let bit: Int = bit * 2
let i: Int = i + 1
}
return r
}
// -- CRC32 (table-driven, table built with xor32) ------------------------------
fn crc_table() -> [Int] {
let t: [Int] = native_list_empty()
let n: Int = 0
while n < 256 {
let c: Int = n
let k: Int = 0
while k < 8 {
if c % 2 == 1 {
let h: Int = c / 2
let c: Int = xor32(h, 3988292384)
} else {
let c: Int = c / 2
}
let k: Int = k + 1
}
let t: [Int] = native_list_append(t, c)
let n: Int = n + 1
}
return t
}
fn crc32_of(bytes: [Int], table: [Int]) -> Int {
let crc: Int = 4294967295
let n: Int = native_list_len(bytes)
let i: Int = 0
while i < n {
let b: Int = native_list_get(bytes, i)
let lo: Int = crc % 256
let idx: Int = xor32(lo, b) % 256
let tv: Int = native_list_get(table, idx)
let hi: Int = crc / 256
let crc: Int = xor32(hi, tv)
let i: Int = i + 1
}
return xor32(crc, 4294967295)
}
// -- Adler32 (for the zlib trailer) --------------------------------------------
fn adler32_of(bytes: [Int]) -> Int {
let a: Int = 1
let b: Int = 0
let n: Int = native_list_len(bytes)
let i: Int = 0
while i < n {
let byte: Int = native_list_get(bytes, i)
let a: Int = (a + byte) % 65521
let b: Int = (b + a) % 65521
let i: Int = i + 1
}
return b * 65536 + a
}
// -- byte-list append helpers --------------------------------------------------
fn app_u32be(dst: [Int], v: Int) -> [Int] {
let dst: [Int] = native_list_append(dst, (v / 16777216) % 256)
let dst: [Int] = native_list_append(dst, (v / 65536) % 256)
let dst: [Int] = native_list_append(dst, (v / 256) % 256)
let dst: [Int] = native_list_append(dst, v % 256)
return dst
}
fn app_tag(dst: [Int], s: String) -> [Int] {
let n: Int = str_len(s)
let i: Int = 0
while i < n {
let dst: [Int] = native_list_append(dst, str_char_code(s, i))
let i: Int = i + 1
}
return dst
}
fn app_all(dst: [Int], src: [Int]) -> [Int] {
let n: Int = native_list_len(src)
let i: Int = 0
while i < n {
let dst: [Int] = native_list_append(dst, native_list_get(src, i))
let i: Int = i + 1
}
return dst
}
// -- plan: frame meaning-geometry -> shape atoms -------------------------------
// shape = [type, x, y, size, r, g, b] (type 0=rect 1=disc 2=triangle)
fn charsum(s: String) -> Int {
let n: Int = str_len(s)
let i: Int = 0
let acc: Int = 0
while i < n {
let c: Int = str_char_code(s, i)
let acc: Int = acc + c
let i: Int = i + 1
}
return acc
}
fn micro_of(s: String) -> Int {
let dot: Int = str_index_of(s, ".")
if dot < 0 { return i_str_to_int(s) * 1000000 }
let n: Int = str_len(s)
let fp: String = str_slice(s, dot + 1, n)
let ip: String = str_slice(s, 0, dot)
let iv: Int = i_str_to_int(ip)
let fv: Int = 0
let scale: Int = 100000
let fl: Int = str_len(fp)
let i: Int = 0
while i < 6 {
let d: Int = 0
if i < fl { let d: Int = str_char_code(fp, i) - 48 }
let fv: Int = fv + d * scale
let scale: Int = scale / 10
let i: Int = i + 1
}
return iv * 1000000 + fv
}
fn plan_scene(frames: [[String]], warm: [Int], cool: [Int]) -> [[Int]] {
let shapes: [[Int]] = native_list_empty()
let nf: Int = native_list_len(frames)
let fi: Int = 0
while fi < nf {
let fr: [String] = native_list_get(frames, fi)
let relation: String = surface_get(fr, "relation")
let polarity: String = surface_get(fr, "polarity")
let confidence: String = surface_get(fr, "confidence")
let importance: String = surface_get(fr, "importance")
let salience: String = surface_get(fr, "salience")
// relation -> shape type
let stype: Int = charsum(relation) % 3
// confidence -> size (8..22)
let cmi: Int = micro_of(confidence)
let size: Int = 8 + cmi / 71428
// salience -> y
let sal: Int = i_str_to_int(salience)
let y: Int = 6 + sal * 26
// subj_id/index -> x
let x: Int = 4 + (fi * 10) % 48
// polarity -> warm/cool base color
let br: Int = native_list_get(warm, 0)
let bg2: Int = native_list_get(warm, 1)
let bb: Int = native_list_get(warm, 2)
if str_eq(polarity, "neg") {
let br: Int = native_list_get(cool, 0)
let bg2: Int = native_list_get(cool, 1)
let bb: Int = native_list_get(cool, 2)
}
// importance -> brightness (500..1000 permille)
let imi: Int = micro_of(importance)
let bpm: Int = 500 + imi / 2000
let r: Int = br * bpm / 1000
let g: Int = bg2 * bpm / 1000
let b: Int = bb * bpm / 1000
let sh: [Int] = native_list_empty()
let sh: [Int] = native_list_append(sh, stype)
let sh: [Int] = native_list_append(sh, x)
let sh: [Int] = native_list_append(sh, y)
let sh: [Int] = native_list_append(sh, size)
let sh: [Int] = native_list_append(sh, r)
let sh: [Int] = native_list_append(sh, g)
let sh: [Int] = native_list_append(sh, b)
let shapes: [[Int]] = native_list_append(shapes, sh)
let fi: Int = fi + 1
}
return shapes
}
// covers: is (px,py) inside this shape?
fn covers(sh: [Int], px: Int, py: Int) -> Bool {
let stype: Int = native_list_get(sh, 0)
let sx: Int = native_list_get(sh, 1)
let sy: Int = native_list_get(sh, 2)
let size: Int = native_list_get(sh, 3)
let cx: Int = sx + size / 2
if stype == 0 {
if px >= sx {
if px < sx + size {
if py >= sy {
if py < sy + size {
return true
}
}
}
}
return false
}
if stype == 1 {
let rad: Int = size / 2
let dx: Int = px - cx
let dy: Int = py - (sy + rad)
if dx * dx + dy * dy <= rad * rad {
return true
}
return false
}
// triangle: apex at top (sy), base at sy+size
if py >= sy {
if py < sy + size {
let dyv: Int = py - sy
let halfw: Int = dyv / 2
let dxv: Int = px - cx
let adx: Int = dxv
if adx < 0 { let adx: Int = 0 - dxv }
if adx <= halfw {
return true
}
}
}
return false
}
// pixel_color: painter's algorithm - last covering shape wins. Returns [r,g,b].
fn pixel_color(px: Int, py: Int, shapes: [[Int]], bg: [Int]) -> [Int] {
let r: Int = native_list_get(bg, 0)
let g: Int = native_list_get(bg, 1)
let b: Int = native_list_get(bg, 2)
let n: Int = native_list_len(shapes)
let i: Int = 0
while i < n {
let sh: [Int] = native_list_get(shapes, i)
if covers(sh, px, py) {
let r: Int = native_list_get(sh, 4)
let g: Int = native_list_get(sh, 5)
let b: Int = native_list_get(sh, 6)
}
let i: Int = i + 1
}
let out: [Int] = native_list_empty()
let out: [Int] = native_list_append(out, r)
let out: [Int] = native_list_append(out, g)
let out: [Int] = native_list_append(out, b)
return out
}
// rasterize: build the raw (filtered) scanline byte stream, filter byte 0 / row.
fn rasterize(w: Int, h: Int, shapes: [[Int]], bg: [Int]) -> [Int] {
let raw: [Int] = native_list_empty()
let y: Int = 0
while y < h {
let raw: [Int] = native_list_append(raw, 0)
let x: Int = 0
while x < w {
let col: [Int] = pixel_color(x, y, shapes, bg)
let raw: [Int] = native_list_append(raw, native_list_get(col, 0))
let raw: [Int] = native_list_append(raw, native_list_get(col, 1))
let raw: [Int] = native_list_append(raw, native_list_get(col, 2))
let x: Int = x + 1
}
let y: Int = y + 1
}
return raw
}
// zlib stream with a single STORED (uncompressed) DEFLATE block + Adler32.
fn zlib_store(raw: [Int]) -> [Int] {
let z: [Int] = native_list_empty()
let z: [Int] = native_list_append(z, 120)
let z: [Int] = native_list_append(z, 1)
let z: [Int] = native_list_append(z, 1)
let len: Int = native_list_len(raw)
let nlen: Int = 65535 - len
let z: [Int] = native_list_append(z, len % 256)
let z: [Int] = native_list_append(z, (len / 256) % 256)
let z: [Int] = native_list_append(z, nlen % 256)
let z: [Int] = native_list_append(z, (nlen / 256) % 256)
let z: [Int] = app_all(z, raw)
let ad: Int = adler32_of(raw)
let z: [Int] = app_u32be(z, ad)
return z
}
// append a full PNG chunk: length + (type+data) + crc32(type+data).
fn app_chunk(png: [Int], type_and_data: [Int], table: [Int]) -> [Int] {
let total: Int = native_list_len(type_and_data)
let dlen: Int = total - 4
let png: [Int] = app_u32be(png, dlen)
let png: [Int] = app_all(png, type_and_data)
let crc: Int = crc32_of(type_and_data, table)
let png: [Int] = app_u32be(png, crc)
return png
}
fn png_build(w: Int, h: Int, raw: [Int], table: [Int]) -> [Int] {
let png: [Int] = native_list_empty()
// 8-byte signature
let png: [Int] = native_list_append(png, 137)
let png: [Int] = native_list_append(png, 80)
let png: [Int] = native_list_append(png, 78)
let png: [Int] = native_list_append(png, 71)
let png: [Int] = native_list_append(png, 13)
let png: [Int] = native_list_append(png, 10)
let png: [Int] = native_list_append(png, 26)
let png: [Int] = native_list_append(png, 10)
// IHDR
let ihdr: [Int] = native_list_empty()
let ihdr: [Int] = app_tag(ihdr, "IHDR")
let ihdr: [Int] = app_u32be(ihdr, w)
let ihdr: [Int] = app_u32be(ihdr, h)
let ihdr: [Int] = native_list_append(ihdr, 8)
let ihdr: [Int] = native_list_append(ihdr, 2)
let ihdr: [Int] = native_list_append(ihdr, 0)
let ihdr: [Int] = native_list_append(ihdr, 0)
let ihdr: [Int] = native_list_append(ihdr, 0)
let png: [Int] = app_chunk(png, ihdr, table)
// IDAT
let z: [Int] = zlib_store(raw)
let idat: [Int] = native_list_empty()
let idat: [Int] = app_tag(idat, "IDAT")
let idat: [Int] = app_all(idat, z)
let png: [Int] = app_chunk(png, idat, table)
// IEND
let iend: [Int] = native_list_empty()
let iend: [Int] = app_tag(iend, "IEND")
let png: [Int] = app_chunk(png, iend, table)
return png
}
fn png_write(path: String, png: [Int]) -> Int {
let n: Int = native_list_len(png)
let buf: String = __str_alloc(n)
let i: Int = 0
while i < n {
let buf: String = __str_set_char(buf, i, native_list_get(png, i))
let i: Int = i + 1
}
let ok: Int = fs_write_bytes(path, buf, n)
return ok
}
+72
View File
@@ -0,0 +1,72 @@
;;; lang_profile_ca.el — Catalan language profile for ELP.
;;; Mirrors lang_profile_it / _es / _pt; keys the realizer's construction switches.
;;; Catalan is the CLOSEST Romance sibling to the shared engine (~85% conceptual
;;; reuse). The deltas: PRONOMS FEBLES with four position allomorphs, l'-elision,
;;; del/al/pel contractions, the periphrastic preterite (vaig+INF), and NO
;;; essere/avere split (perfect aux is always HAVER; ser/estar is only the copula).
(lang_profile_ca
(language "Catalan")
(iso639 "ca")
(family "Romance")
;; ── core typology flags ────────────────────────────────────────────────
(pro-drop yes) ; null subjects default; overt pronoun = emphatic
(obligatory-subject no)
(grammatical-gender yes) ; m/f; full NP agreement (art + adj + participle)
(do-support no)
(subject-aux-inversion no) ; yes/no Q = declarative order + '?'; no inversion
(article-selection "el/la/l'/els/les ; un/una/uns/unes") ; l'-ELISION:
; el/la -> l' before vowel or (silent) h, glued to
; the next word (l'home, l'illa); de -> d' before vowel
(article-drives-contraction yes) ; article choice feeds prep+article contraction
(adjective-position "postnominal-default + small prenominal class") ; bo/bon,
; mal, gran, nou, vell, primer, molt... prenominal
(question-punct plain) ; ? and ! only (no inverted ¿ ¡)
;; ── MANDATORY prep+article contractions ────────────────────────────────
(contractions ((de el del) (de els dels)
(a el al) (a els als)
(per el pel) (per els pels)))
(contraction-mandatory yes) ; *de el -> del obligatory
(contraction-blocked-before-elision yes) ; de l'home / a l'home (NO *del home)
;; ── clitic system: PRONOMS FEBLES (the headline delta) ──────────────────
(clitics yes)
(clitic-allomorphy four-position) ; per pronoun, form varies by position+onset:
; reinforced (em, et, el) proclitic before a consonant
; elided (m', t', l', n') proclitic before a vowel/h
; full (-me, -lo, -li) enclitic after a consonant/-r
; reduced ('m, 't, 'l, 'ns) enclitic after a vowel
(clitic-placement ((finite proclitic) ; el veig, no m'ho dóna
(imperative-affirmative enclitic) ; dóna'm, digues-me
(imperative-negative present-subjunctive) ; no parlis (delta)
(infinitive enclitic) ; ajudar-me, veure'l
(gerund enclitic))) ; fent-ho
(clitic-combination ((me el "me'l") (te el "te'l") (se el "se'l")
(me la "me la") (me en "me'n")
(li el "l'hi") (li en "n'hi"))) ; dative+accusative clusters
(clitic-particles (hi en ho)) ; locative hi, partitive/genitive en, neuter ho
;; ── verb / aspect system ───────────────────────────────────────────────
(finite-agreement "person+number (6-way)")
(tenses (present imperfet preterit-simple perifrastic-preterit futur
condicional subjuntiu-present subjuntiu-imperfet imperatiu))
(periphrastic-preterite "vaig/vas/va/vam/vau/van + INFINITIVE") ; << hallmark CA
; (vaig cantar = 'I sang'); coexists w/ synthetic pret.
(compound-past "pretèrit perfet = haver(present) + participle")
(perfect-aux "HAVER only") ; << NO essere/avere split (simpler than IT)
(participle-agreement ((haver preceding-acc-clitic))) ; les he vistes; else invariable
(progressive-aux "estar + gerundi")
(copula "ser / estar") ; ser: identity/essential/origin; estar:
; location + transient state (estic cansat, és a casa)
(passive-aux "ser (+ per-agent)")
(future inflectional) ; cantaré, serà
(comparative "més/menys ADJ que")
;; ── SACRED safety bar (shared with es/pt/it/en) ────────────────────────
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
(negation "no (preverbal) + optional 'pas' + concord") ; no...res/
; ningú/mai/cap/gens/enlloc
(negative-concord yes) ; preverbal negative subject (ningú) keeps 'no'
(neg-reinforcer pas)) ; optional (no ho faré pas)
+41
View File
@@ -0,0 +1,41 @@
;;; lang_profile_de.el — German language profile for ELP.
;;; Mirrors lang_profile_en / lang_profile_es. Keys the realizer's construction
;;; switches. German is the largest Germanic delta from the EN engine: V2 word
;;; order, four morphological cases, and separable-prefix verbs.
(lang_profile_de
(language "German")
(iso639 "de")
(family "Germanic")
(neighbor-base "en") ; realized by extending the English (Germanic) engine
;; ── core typology flags ────────────────────────────────────────────────
(pro-drop no) ; obligatory subject in finite clauses
(obligatory-subject yes)
(grammatical-gender (m f n)) ; three genders; drives article + adj declension
(case-system (nom acc dat gen)) ; four cases on articles/adjs/nouns
(word-order V2) ; finite verb 2nd in main clause
(subordinate-order verb-final) ; "..., dass er den Hund SIEHT."
(separable-verbs yes) ; aufstehen -> "steht ... auf"; ppart "aufgestanden"
(do-support no) ; German negates/questions the finite verb directly
(subject-verb-inversion yes) ; yes/no Q fronts finite verb; wh-Q fills Vorfeld
(article-selection "der/die/das + ein/kein") ; declined by case x gender x number
(adjective-position prenominal)
(adjective-declension (strong weak mixed)) ; chosen by the determiner type
(noun-capitalization yes)
;; ── verb / aspect system ───────────────────────────────────────────────
(finite-agreement "person-and-number") ; full present/past paradigm
(auxiliary-order (modal tense-aux perfect passive main))
(perfect-aux (haben sein)) ; sein for intransitive motion/change verbs
(passive-aux "werden")
(future "werden + infinitive")
(comparative "synthetic (-er / -st, with umlaut)")
;; ── negation ───────────────────────────────────────────────────────────
(negation-markers (nicht kein)) ; kein- negates an indefinite NP; nicht else
(negation-faithful yes) ; SACRED: polarity never dropped/inverted -> FLAG
;; ── lexicon provenance ─────────────────────────────────────────────────
(lexicon-source "UniMorph deu (primary) + kaikki.org German (gender override)")
(lexicon-license "CC-BY-SA 3.0 / GFDL"))
+41
View File
@@ -0,0 +1,41 @@
;;; lang_profile_en.el — English language profile for ELP.
;;; Mirrors lang_profile_es / lang_profile_pt; keys the realizer's construction
;;; switches. English is typologically distinct from the Romance builds, so the
;;; flags differ where the grammar differs.
(lang_profile_en
(language "English")
(iso639 "en")
(family "Germanic")
;; ── core typology flags ────────────────────────────────────────────────
(pro-drop no) ; OBLIGATORY subjects — missing subject is FLAGGED
(obligatory-subject yes)
(grammatical-gender no) ; natural gender only (he/she/it), no NP agreement
(do-support yes) ; negation & questions of lexical verbs insert do/does/did
(subject-aux-inversion yes) ; yes/no + non-subject wh questions invert the operator
(article-selection "a/an/the") ; a/an resolved PHONOLOGICALLY (an hour, a university)
(adjective-position prenominal) ; attributive adjectives precede the noun; invariant
(has-tag-questions yes) ; "...doesn't he?" — operator + reversed polarity
(has-there-existential yes) ; "there is/are/have been ..."
(possessive-clitic "'s") ; saxon genitive; plural in -s -> bare apostrophe
(question-punct plain) ; ? and ! only (no inverted marks)
;; ── verb / aspect system ───────────────────────────────────────────────
(finite-agreement "3sg-present-only") ; only 3sg present -s (+ suppletive be)
(auxiliary-order (modal perfect progressive passive main))
(perfect-aux "have") ; have + past participle
(progressive-aux "be") ; be + present participle
(passive-aux "be") ; be + past participle (+ by-agent)
(future "will + base") ; no inflectional future
(comparative "synthetic-or-periphrastic") ; -er/-est vs more/most by syllables
;; ── SACRED safety bar (shared with es/pt) ──────────────────────────────
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
;; ── DIALECT overlay (post-realization, one core -> US/UK/AU) ────────────
(dialect US) ; default; profile field switches the overlay
(dialects (US UK AU))
(dialect-canonical US) ; core is authored in US orthography
(dialect-overlay "dialect_en.to_dialect") ; orthography + lexis + grammar prefs
(dialect-covers (spelling lexis collective-agreement gotten/got)))
+45
View File
@@ -0,0 +1,45 @@
;;; lang_profile_es.el — Spanish language profile for ELP.
;;; Keys the realizer's construction switches. Mirrors lang_profile_en / _pt.
(lang_profile_es
(language "Spanish")
(iso639 "es")
(family "Romance")
;; -- core typology flags -------------------------------------------------
(pro-drop yes) ; subjects routinely dropped; agreement carries person
(obligatory-subject no)
(grammatical-gender yes) ; m/f on every noun; article+adjective AGREE
(gender-source lexicon); REAL per-noun gender from UniMorph — NOT a heuristic
(do-support no)
(subject-aux-inversion no) ; questions by intonation/punctuation, not inversion
(question-strategy intonation)
(article-selection "el/la/los/las un/una/unos/unas")
(stressed-a-rule yes) ; fem sg noun in stressed a-/ha- takes el/un (el agua)
(adjective-position postnominal) ; default post; a few prenominal + apocope
(adjective-agreement "gender+number")
(question-punct inverted) ; opening ¿ ¡ required
;; -- MANDATORY CONTRACTIONS (coordinator quality bar) --------------------
(contractions ((de el "del") (a el "al")))
(contraction-mandatory yes) ; 'de el'/'a el' MUST surface as del/al
;; -- verb / aspect system ------------------------------------------------
(verb-classes (ar er ir))
(tenses (present preterite imperfect future conditional))
(moods (ind sbjv imp))
(finite-agreement "person+number (6 slots)")
(perfect-aux "haber") ; haber + past participle (invariant -o)
(progressive-aux "estar") ; estar + gerund
(passive-aux "ser") ; ser + participle (agrees) + por-agent
(copula-split "ser/estar") ; permanent vs stage-level
(future "infinitive + é/ás/á/emos/éis/án")
;; -- clitics / government ------------------------------------------------
(object-clitics yes) ; me te lo la le nos os los las; proclisis/enclisis
(clitic-order "se II I III (le+lo -> se lo)")
(enclisis "imperative/infinitive/gerund + accent repair (dá+me+lo->dámelo)")
(verb-prep-government yes) ; verbs select prep (protestar+contra, escapar+de)
;; -- SACRED safety bar (shared with en/pt) -------------------------------
(negation-faithful yes)) ; polarity never dropped/inverted; unplaceable -> FLAG
+74
View File
@@ -0,0 +1,74 @@
;;; lang_profile_fr.el — French language profile for ELP.
;;; Mirrors lang_profile_it / lang_profile_es; keys the realizer's construction
;;; switches. French is a Romance sibling (~54% of the realizer code and the whole
;;; clause-engine architecture reused), but carries the family's biggest surface
;;; deltas: NOT pro-drop, DISCONTINUOUS negation, and an orthography/phonology
;;; mismatch (elision, liaison) that makes exact-match genuinely hard.
(lang_profile_fr
(language "French")
(iso639 "fr")
(family "Romance")
;; ── core typology flags ────────────────────────────────────────────────
(pro-drop no) ; << French-specific: subject clitic OBLIGATORY
(obligatory-subject yes) ; je/tu/il/elle/nous/vous/ils/elles always overt
(grammatical-gender yes) ; m/f; full NP agreement (art + adj + participle)
(do-support no)
(subject-aux-inversion optional) ; est-ce que (default) OR clitic inversion (vas-tu)
(article-selection "le/la/l'/les ; un/une/des ; PARTITIVE du/de la/de l'/des")
(article-drives-contraction yes) ; à+le=au, de+le=du feed off article choice
(adjective-position "postnominal-default + prenominal-BAGS") ; beau/bon/grand/
; petit/jeune/vieux/nouveau + ordinals prenominal
; (beau->bel, nouveau->nouvel, vieux->vieil / vowel)
(question-punct "space-before") ; French typography: ' ?' ' !' (no ¿¡)
;; ── elision (orthography/phonology mismatch — French-specific) ──────────
(elision ((le l') (la l') (je j') (ne n') (de d') (que qu')
(me m') (te t') (se s') (ce c'))) ; before vowel / h-muet
(elision-h-muet yes) ; l'homme, l'hôpital (h-aspiré exception list kept)
(liaison noted-not-modeled) ; phonological, not written in surface
;; ── MANDATORY prep+article contractions ────────────────────────────────
(contractions ((à le au) (à les aux) (de le du) (de les des)))
(contraction-mandatory yes) ; *à le -> au obligatory; à la / à l' uncontracted
(partitive ((m-sg du) (f-sg "de la") (vowel "de l'") (pl des)))
(partitive-under-neg "de") ; << gap in current build: 'ne … pas de pain'
;; ── clitic system ──────────────────────────────────────────────────────
(clitics yes)
(clitic-order (me te se nous vous | le la les | lui leur | y | en))
(clitic-placement ((finite proclitic) ; je le lui donne
(imperative-affirmative enclitic-hyphen) ; donne-le-moi
(imperative-negative "ne+proclitic+verb+pas") ; ne le donne pas
(infinitive enclitic))) ; PARTIAL: clitic-climbing
; onto infinitive under modal
(clitic-imperative-shift ((me moi) (te toi))) ; final me/te -> moi/toi (donne-moi)
(clitic-particles (y en)) ; locative y, partitive/genitive en
;; ── verb / aspect system ───────────────────────────────────────────────
(finite-agreement "person+number (written; many homophones)")
(tenses (présent imparfait passé-simple futur conditionnel
subjonctif-présent subjonctif-imparfait impératif))
(compound-past "passé-composé = aux(present) + participe passé")
(perfect-aux "être/avoir (LEXICAL selection)") ; << French-specific
(etre-aux-class "intransitive motion/change (aller venir arriver partir
entrer sortir monter descendre naître mourir rester
tomber retourner passer devenir revenir rentrer) + ALL
pronominal verbs")
(participle-agreement ((être subject) ; elle est allée / elles venues
(avoir preceding-direct-object))) ; je les ai vus
(progressive "être en train de + infinitif") ; no dedicated aux
(copula "être (single; no ser/estar, no essere/stare)")
(passive-aux "être (+ par-agent)")
(future inflectional) ; parlera, sera
(comparative "plus/moins ADJ que")
(superlative "le/la plus ADJ (de …)") ; PARTIAL word-order in build
;; ── SACRED safety bar (shared with es/pt/it/en) ────────────────────────
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
(negation "DISCONTINUOUS: ne (preverbal) … pas/jamais/rien/personne/
plus/guère/que (postverbal)") ; << biggest structural delta
(negation-ne-elides yes) ; ne -> n' before vowel (n'ai pas vu)
(negation-passe-composé "ne + aux + pas + participe") ; n'ai pas vu
(negative-concord partial)) ; personne/rien as arguments post-participle
+70
View File
@@ -0,0 +1,70 @@
;;; lang_profile_it.el — Italian language profile for ELP.
;;; Mirrors lang_profile_es / lang_profile_pt; keys the realizer's construction
;;; switches. Italian is a Romance sibling, so ~85% of the flags match ES/PT; the
;;; essere/avere auxiliary split and phonological article selection are the deltas.
(lang_profile_it
(language "Italian")
(iso639 "it")
(family "Romance")
;; ── core typology flags ────────────────────────────────────────────────
(pro-drop yes) ; null subjects default; overt pronoun = emphatic
(obligatory-subject no)
(grammatical-gender yes) ; m/f; full NP agreement (art + adj + participle)
(do-support no)
(subject-aux-inversion no) ; yes/no Q = declarative order + '?'; no inversion
(article-selection "il/lo/l'/i/gli + la/l'/le ; un/uno/un'/una") ; PHONOLOGICAL:
; lo/gli/uno before s+cons, z, gn, ps, pn, x, y, i+V;
; l'/un' before a vowel (elision, glued to next word)
(article-drives-contraction yes) ; article choice feeds the prep+art contraction
(adjective-position "postnominal-default + prenominal-class") ; bello/buono/grande
; /nuovo/vecchio/primo... prenominal (with apocope)
(question-punct plain) ; ? and ! only (no inverted ¿ ¡)
;; ── MANDATORY prep+article contractions ────────────────────────────────
(contractions ((di il del) (di lo dello) (di la della) (di i dei)
(di gli degli) (di le delle) (di l' dell')
(a il al) (a lo allo) (a la alla) (a i ai) (a gli agli)
(a le alle) (a l' all')
(da il dal) (da la dalla) (da gli dagli) (da l' dall')
(in il nel) (in la nella) (in gli negli) (in l' nell')
(su il sul) (su la sulla) (su gli sugli) (su l' sull')))
(contraction-mandatory yes) ; *di il -> del is obligatory, never uncontracted
(prep-no-contract (per tra fra)) ; per la strada (NOT *perla)
;; ── clitic system ──────────────────────────────────────────────────────
(clitics yes)
(clitic-placement ((finite proclitic) ; lo vedo, non me lo dà
(imperative-affirmative enclitic) ; dammelo, guardalo
(imperative-negative-tu non+infinitive) ; non parlare / non lo fare
(infinitive enclitic) ; vederlo, aiutarmi (drop -e)
(gerund enclitic))) ; dandolo
(clitic-combination ((mi lo "me lo") (ti lo "te lo") (ci lo "ce lo")
(vi lo "ve lo") (si lo "se lo")
(gli lo "glielo") (le lo "glielo"))) ; glielo = ONE word
(clitic-particles (ci ne)) ; locative ci, partitive ne
(raddoppiamento (da fa di va sta)) ; monosyllabic imper double clitic: dammelo
;; ── verb / aspect system ───────────────────────────────────────────────
(finite-agreement "person+number (6-way)")
(tenses (presente imperfetto passato-remoto futuro condizionale
congiuntivo-presente congiuntivo-imperfetto imperativo))
(compound-past "passato-prossimo = aux(present) + participle")
(perfect-aux "essere/avere (LEXICAL selection)") ; << Italian-specific
(essere-aux-class unaccusative) ; motion/change-of-state/copular/pronominal
; (andare venire nascere morire diventare piacere
; + ALL reflexives) -> essere
(participle-agreement ((essere subject) ; è andata / sono arrivati
(avere preceding-acc-clitic))) ; li ho visti
(progressive-aux "stare + gerundio") ; sto parlando
(copula "essere (default) / stare (state: sto bene)")
(passive-aux "essere / venire (+ da-agent)")
(future inflectional) ; parlerò, sarà
(comparative "più/meno ADJ di")
;; ── SACRED safety bar (shared with es/pt/en) ───────────────────────────
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
(negation "non (preverbal) + concord") ; non...niente/nessuno/mai/più
(negative-concord yes) ; preverbal negative word (nessuno/niente) suppresses non
(neg-adverb-position between-aux-and-participle)) ; non ho MAI visto
+30
View File
@@ -0,0 +1,30 @@
;;; lang_profile_la.el — Latin language profile for ELP.
;;; Keys the realizer's construction switches. Companion to morphology-la.el.
(lang_profile_la
(language "Latin")
(iso639 "la")
(family "Italic")
;; -- core typology flags -------------------------------------------------
(pro-drop yes) ; person carried by verb ending; subjects dropped
(obligatory-subject no)
(grammatical-gender yes) ; m/f/n; adjective AGREES in case+gender+number
(gender-source lexicon) ; REAL per-noun gender from UniMorph lat
(articles none) ; Latin has no articles
(case-system yes) ; NOM GEN DAT ACC ABL VOC (+ rare LOC)
(cases (nom gen dat acc abl voc))
(word-order "SOV (default; free order, case-marked)")
(adjective-position "either (case agreement carries the link)")
(adjective-agreement "case+gender+number")
;; -- verb / aspect system ------------------------------------------------
(verb-classes (1 2 3 3io 4)) ; four conjugations + i-stem 3rd
(tenses (present imperfect future perfect pluperfect futureperfect))
(moods (indicative subjunctive imperative infinitive))
(voices (active passive))
(finite-agreement "person+number (6 slots)")
(citation "principal parts: pres-1sg / pres-inf / perf-participle")
;; -- SACRED safety bar ---------------------------------------------------
(negation-faithful yes)) ; polarity never dropped/inverted
+40
View File
@@ -0,0 +1,40 @@
;;; lang_profile_pt.el — Portuguese language profile for ELP.
;;; Keys the realizer's construction switches. Mirrors lang_profile_es.
(lang_profile_pt
(language "Portuguese")
(iso639 "pt")
(family "Romance")
;; -- core typology flags -------------------------------------------------
(pro-drop yes) ; subjects routinely dropped; agreement carries person
(obligatory-subject no)
(grammatical-gender yes) ; m/f on every noun; article+adjective AGREE
(gender-source lexicon) ; REAL per-noun gender from UniMorph por / kaikki
(do-support no)
(subject-aux-inversion no)
(question-strategy intonation)
(article-selection "o/a/os/as um/uma/uns/umas")
(adjective-position postnominal)
(adjective-agreement "gender+number")
;; -- MANDATORY CONTRACTIONS (prep + article) -----------------------------
(contractions ((de o "do") (de a "da") (em o "no") (em a "na")
(a o "ao") (a a "à") (por o "pelo") (por a "pela")))
(contraction-mandatory yes)
;; -- verb / aspect system ------------------------------------------------
(verb-classes (ar er ir))
(tenses (present preterite imperfect future conditional))
(moods (ind sbjv imp))
(finite-agreement "person+number (6 slots)")
(perfect-aux "ter") ; ter + past participle
(copula-split "ser/estar")
(personal-infinitive yes) ; distinctive PT inflected infinitive
;; -- clitics / government ------------------------------------------------
(object-clitics yes) ; mesoclisis/enclisis/proclisis by context
(verb-prep-government yes)
;; -- SACRED safety bar ---------------------------------------------------
(negation-faithful yes))
+71
View File
@@ -0,0 +1,71 @@
;;; lang_profile_ro.el — Romanian language profile for ELP.
;;; Romanian is the BIG typological delta of the Romance family. The verb/clause
;;; engine and the SACRED negation contract mirror the ES/PT/IT core, but the
;;; NOMINAL system is genuinely new: a SUFFIXED definite article, preserved CASE,
;;; a NEUTER gender, and a VOCATIVE. Those flags mark where the shared engine was
;;; extended rather than reused.
(lang_profile_ro
(language "Romanian")
(iso639 "ro")
(family "Romance (Eastern / Balkan)")
;; ── core typology flags ────────────────────────────────────────────────
(pro-drop yes) ; null subjects default; overt pronoun = emphatic
(obligatory-subject no)
(grammatical-gender yes) ; m / f / NEUTER (n)
(neuter-gender yes) ; << ROMANIAN-SPECIFIC: masc-agreeing SG, fem-agreeing PL
; (un tren nou / două trenuri noi)
(do-support no)
(subject-aux-inversion no) ; yes/no Q = declarative order + '?'
(question-punct plain) ; ? and ! only
;; ── SUFFIXED DEFINITE ARTICLE (the headline engine extension) ───────────
(definite-article suffixed) ; << UNIQUE IN ROMANCE: enclitic on the noun
(definite-forms ((m/n sg "-ul / -le / -l : om->omul, câine->câinele, codru->codrul")
(f sg "-a / -ea / -ua : casă->casa, carte->cartea, stea->steaua")
(m pl "-i : oameni->oamenii")
(f/n pl "-le : case->casele, trenuri->trenurile")))
(article-host ((no-prenom-adj noun) ; omul bun
(prenom-adj adjective))) ; bunul om (adj carries the article)
(indefinite-article ((m/n "un") (f "o") (pl "niște") (gen/dat-pl "unor")))
;; ── CASE (preserved; NOM/ACC vs GEN/DAT) ────────────────────────────────
(case (nom/acc gen/dat vocative)) ; << ROMANIAN-SPECIFIC
(case-syncretism "nom=acc ; gen=dat")
(genitive-marking "gen/dat definite: -lui (m/n), -ei/-i (f), -lor (pl)")
(genitival-article ((m sg "al") (f sg "a") (m pl "ai") (f/n pl "ale"))) ; o carte a lui
(possession "definite-head + gen/dat possessor: casa băiatului")
(vocative ((m sg "-ule/-e : omule, băiete") (f sg "-o : Mario, fato")
(pl "-lor")))
;; ── verb / aspect system ────────────────────────────────────────────────
(finite-agreement "person+number (6-way)")
(tenses (prezent imperfect perfect-simplu conjunctiv-prezent
imperativ (periphrastic: perfect-compus viitor conditional)))
(compound-past "perfectul compus = a-avea-clitic + INVARIABLE participle")
(perfect-aux "a avea (am/ai/a/am/ați/au) — ONE auxiliary for ALL verbs")
(perfect-aux-split no) ; << SIMPLER than Italian: no essere/avere selection
(participle-agreement none) ; invariable in the perfect compus (agrees only as
; an adjective / in the passive)
(future "voi/vei/va/vom/veți/vor + infinitive (viitor literar)")
(conditional "aș/ai/ar/am/ați/ar + infinitive")
(subjunctive "conjunctiv: particle 'să' + subjunctive present")
(modal-complement "modal + să + subjunctive (vreau să merg, poți să ajuți)")
(copula "a fi")
(passive "a fi + participle (participle AGREES like an adjective)")
(comparative "mai / mai puțin ADJ decât")
;; ── clitic system (partial — see honest gaps) ───────────────────────────
(clitics yes)
(clitic-set ((acc te îl o ne îi le) (dat îmi îți îi ne le)
(refl te se ne se)))
(clitic-placement ((finite proclitic) ; îmi place, o văd
(perfect-compus elision) ; << m-am, l-am, i-am (PARTIAL)
(imperative-affirmative enclitic))) ; dă-mi (PARTIAL)
;; ── SACRED safety bar (shared with es/pt/it/en) ─────────────────────────
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
(negation "nu (single preverbal marker) + concord")
(negative-concord yes) ; nu … nimic / nimeni / niciodată / niciun
(negative-imperative "nu + INFINITIVE : nu pleca! (KNOWN GAP: uses imperative stem)"))
+1
View File
@@ -250,6 +250,7 @@ fn en_irregular_verb(base: String) -> [String] {
if str_eq(base, "cut") { let r: [String] = ["cut", "cuts", "cut", "cut", "cutting"]; return r }
if str_eq(base, "set") { let r: [String] = ["set", "sets", "set", "set", "setting"]; return r }
if str_eq(base, "hit") { let r: [String] = ["hit", "hits", "hit", "hit", "hitting"]; return r }
if str_eq(base, "fight") { let r: [String] = ["fight", "fights","fought", "fought", "fighting"]; return r }
return empty
}
+280
View File
@@ -0,0 +1,280 @@
// multilingual.el - the language layer for the native-el interlocutor.
//
// Deterministic, NO generative model (ports multilingual.py):
// 1. ml_detect(text) -> ISO code (en/es/pt/it) via stopword + diacritic score
// 2. ml_tr(key, lang) -> localized fixed phrase (SACRED per-language yes/no/decline)
// 3. ml_term(w, lang) -> PT/ES content term -> EN engram equivalent
// 4. ml_translate_pred(lemma, lang) -> EN predicate lemma -> target infinitive
//
// The Python detector count-weights stopwords and diacritics; here diacritics are
// scored by PRESENCE (str_contains) rather than codepoint counting, to stay clear
// of UTF-8 index hazards in the runtime. Faithful enough to classify typical
// queries; documented simplification. Depends on: comprehend (cp_tokenize).
// 1. language detection
fn ml_stop_en(w: String) -> Bool {
if str_eq(w, "the") { return true }
if str_eq(w, "does") { return true }
if str_eq(w, "do") { return true }
if str_eq(w, "did") { return true }
if str_eq(w, "what") { return true }
if str_eq(w, "who") { return true }
if str_eq(w, "is") { return true }
if str_eq(w, "are") { return true }
if str_eq(w, "how") { return true }
if str_eq(w, "you") { return true }
if str_eq(w, "your") { return true }
if str_eq(w, "of") { return true }
if str_eq(w, "to") { return true }
if str_eq(w, "and") { return true }
if str_eq(w, "for") { return true }
if str_eq(w, "explain") { return true }
if str_eq(w, "answer") { return true }
if str_eq(w, "memory") { return true }
if str_eq(w, "with") { return true }
if str_eq(w, "not") { return true }
if str_eq(w, "store") { return true }
return false
}
fn ml_stop_es(w: String) -> Bool {
if str_eq(w, "que") { return true }
if str_eq(w, "qué") { return true }
if str_eq(w, "una") { return true }
if str_eq(w, "usted") { return true }
if str_eq(w, "su") { return true }
if str_eq(w, "cómo") { return true }
if str_eq(w, "como") { return true }
if str_eq(w, "cuál") { return true }
if str_eq(w, "quién") { return true }
if str_eq(w, "está") { return true }
if str_eq(w, "es") { return true }
if str_eq(w, "los") { return true }
if str_eq(w, "las") { return true }
if str_eq(w, "del") { return true }
if str_eq(w, "al") { return true }
if str_eq(w, "explica") { return true }
if str_eq(w, "explique") { return true }
if str_eq(w, "forma") { return true }
if str_eq(w, "con") { return true }
if str_eq(w, "memoria") { return true }
if str_eq(w, "responde") { return true }
return false
}
fn ml_stop_pt(w: String) -> Bool {
if str_eq(w, "que") { return true }
if str_eq(w, "uma") { return true }
if str_eq(w, "você") { return true }
if str_eq(w, "sua") { return true }
if str_eq(w, "seu") { return true }
if str_eq(w, "como") { return true }
if str_eq(w, "memória") { return true }
if str_eq(w, "isso") { return true }
if str_eq(w, "os") { return true }
if str_eq(w, "as") { return true }
if str_eq(w, "da") { return true }
if str_eq(w, "do") { return true }
if str_eq(w, "na") { return true }
if str_eq(w, "no") { return true }
if str_eq(w, "explica") { return true }
if str_eq(w, "forma") { return true }
if str_eq(w, "é") { return true }
if str_eq(w, "está") { return true }
if str_eq(w, "com") { return true }
if str_eq(w, "responda") { return true }
return false
}
fn ml_stop_it(w: String) -> Bool {
if str_eq(w, "che") { return true }
if str_eq(w, "una") { return true }
if str_eq(w, "come") { return true }
if str_eq(w, "della") { return true }
if str_eq(w, "gli") { return true }
if str_eq(w, "è") { return true }
if str_eq(w, "sono") { return true }
if str_eq(w, "questo") { return true }
if str_eq(w, "nel") { return true }
if str_eq(w, "di") { return true }
if str_eq(w, "il") { return true }
if str_eq(w, "cosa") { return true }
if str_eq(w, "per") { return true }
if str_eq(w, "memoria") { return true }
if str_eq(w, "spiega") { return true }
if str_eq(w, "rispondi") { return true }
return false
}
// diacritic PRESENCE score (weight 3 each; hard overrides weight 8).
fn ml_dia_score(low: String, lang: String) -> Int {
let s: Int = 0
if str_eq(lang, "pt") {
if str_contains(low, "ã") { let s = s + 3 }
if str_contains(low, "õ") { let s = s + 3 }
if str_contains(low, "ç") { let s = s + 3 }
if str_contains(low, "ê") { let s = s + 3 }
if str_contains(low, "á") { let s = s + 3 }
// hard PT markers (ã/õ almost never appear outside PT)
if str_contains(low, "ã") { let s = s + 8 }
if str_contains(low, "õ") { let s = s + 8 }
}
if str_eq(lang, "es") {
if str_contains(low, "ñ") { let s = s + 3 }
if str_contains(low, "¿") { let s = s + 3 }
if str_contains(low, "¡") { let s = s + 3 }
if str_contains(low, "á") { let s = s + 3 }
if str_contains(low, "é") { let s = s + 3 }
// hard ES markers
if str_contains(low, "ñ") { let s = s + 8 }
if str_contains(low, "¿") { let s = s + 8 }
if str_contains(low, "¡") { let s = s + 8 }
}
if str_eq(lang, "it") {
if str_contains(low, "è") { let s = s + 3 }
if str_contains(low, "ì") { let s = s + 3 }
if str_contains(low, "ò") { let s = s + 3 }
}
return s
}
fn ml_stop_score(toks: [String], lang: String) -> Int {
let n: Int = native_list_len(toks)
let s: Int = 0
let i: Int = 0
while i < n {
let w: String = native_list_get(toks, i)
if str_eq(lang, "en") { if ml_stop_en(w) { let s = s + 2 } }
if str_eq(lang, "es") { if ml_stop_es(w) { let s = s + 2 } }
if str_eq(lang, "pt") { if ml_stop_pt(w) { let s = s + 2 } }
if str_eq(lang, "it") { if ml_stop_it(w) { let s = s + 2 } }
let i = i + 1
}
return s
}
fn ml_detect(text: String) -> String {
if str_eq(text, "") { return "en" }
let low: String = str_to_lower(text)
let toks: [String] = cp_tokenize(text)
// NOTE: el's overloaded `+` mis-compiles two chained function-call Int operands
// as string concat (documented in comprehend_gate.el). Bind each call to an Int
// var and add vars one at a time so the addition stays integer.
let en: Int = ml_stop_score(toks, "en")
let es_s: Int = ml_stop_score(toks, "es")
let es_d: Int = ml_dia_score(low, "es")
let es: Int = es_s + es_d
let pt_s: Int = ml_stop_score(toks, "pt")
let pt_d: Int = ml_dia_score(low, "pt")
let pt: Int = pt_s + pt_d
let it_s: Int = ml_stop_score(toks, "it")
let it_d: Int = ml_dia_score(low, "it")
let it: Int = it_s + it_d
let best: String = "en"
let bs: Int = en
if es > bs { let best = "es"; let bs = es }
if pt > bs { let best = "pt"; let bs = pt }
if it > bs { let best = "it"; let bs = it }
// weak signal -> honest fallback to English
if bs < 3 { return "en" }
return best
}
// 2. localized fixed phrases (SACRED per-language decline/yes/no)
fn ml_tr(key: String, lang: String) -> String {
if str_eq(key, "no_memory") {
if str_eq(lang, "pt") { return "Não tenho isso na minha memória." }
if str_eq(lang, "es") { return "No tengo eso en mi memoria." }
if str_eq(lang, "it") { return "Non ho quello nella mia memoria." }
return "I don't have that in my memory."
}
if str_eq(key, "parse_fail") {
if str_eq(lang, "pt") { return "Não consegui interpretar isso." }
if str_eq(lang, "es") { return "No pude interpretar eso." }
if str_eq(lang, "it") { return "Non sono riuscito a interpretarlo." }
return "I didn't parse that."
}
if str_eq(key, "yes") {
if str_eq(lang, "pt") { return "Sim" }
if str_eq(lang, "es") { return "" }
if str_eq(lang, "it") { return "" }
return "Yes"
}
if str_eq(key, "no") {
if str_eq(lang, "pt") { return "Não" }
if str_eq(lang, "es") { return "No" }
if str_eq(lang, "it") { return "No" }
return "No"
}
if str_eq(key, "identity") {
if str_eq(lang, "pt") { return "Sou o Neuron, o engrama com quem você está falando." }
if str_eq(lang, "es") { return "Soy Neuron, el engrama con el que estás hablando." }
if str_eq(lang, "it") { return "Sono Neuron, l'engramma con cui stai parlando." }
return "I'm Neuron, the engram you're speaking with."
}
return ""
}
// 3. retrieval term lexicon (PT/ES content term -> EN engram equivalent)
fn ml_term(w: String, lang: String) -> String {
if str_eq(lang, "en") { return w }
if str_eq(w, "saliência") { return "salience" }
if str_eq(w, "saliencia") { return "salience" }
if str_eq(w, "memória") { return "memory" }
if str_eq(w, "memoria") { return "memory" }
if str_eq(w, "geometria") { return "geometry" }
if str_eq(w, "geometrias") { return "geometry" }
if str_eq(w, "geometrías") { return "geometry" }
if str_eq(w, "forma") { return "form" }
if str_eq(w, "consolidação") { return "consolidation" }
if str_eq(w, "consolidación") { return "consolidation" }
if str_eq(w, "aprendizagem") { return "learning" }
if str_eq(w, "aprendizaje") { return "learning" }
if str_eq(w, "") { return "node" }
if str_eq(w, "nodo") { return "node" }
if str_eq(w, "armazenamento") { return "storage" }
if str_eq(w, "almacenamiento") { return "storage" }
if str_eq(w, "estrutura") { return "structure" }
if str_eq(w, "estructura") { return "structure" }
return w
}
// 4. predicate translation (EN lemma -> target infinitive; pass-through) ─────
fn ml_translate_pred(lemma: String, lang: String) -> String {
if str_eq(lang, "en") { return lemma }
if str_eq(lang, "es") {
if str_eq(lemma, "store") { return "almacenar" }
if str_eq(lemma, "use") { return "usar" }
if str_eq(lemma, "have") { return "tener" }
if str_eq(lemma, "be") { return "ser" }
if str_eq(lemma, "give") { return "dar" }
if str_eq(lemma, "make") { return "hacer" }
if str_eq(lemma, "learn") { return "aprender" }
if str_eq(lemma, "form") { return "formar" }
return lemma
}
if str_eq(lang, "pt") {
if str_eq(lemma, "store") { return "armazenar" }
if str_eq(lemma, "use") { return "usar" }
if str_eq(lemma, "have") { return "ter" }
if str_eq(lemma, "be") { return "ser" }
if str_eq(lemma, "give") { return "dar" }
if str_eq(lemma, "make") { return "fazer" }
if str_eq(lemma, "learn") { return "aprender" }
if str_eq(lemma, "form") { return "formar" }
return lemma
}
if str_eq(lang, "it") {
if str_eq(lemma, "store") { return "memorizzare" }
if str_eq(lemma, "use") { return "usare" }
if str_eq(lemma, "have") { return "avere" }
if str_eq(lemma, "be") { return "essere" }
return lemma
}
return lemma
}
+125
View File
@@ -0,0 +1,125 @@
// organ-read.el - Route the render's GEOMETRY READ through the ingest ORGAN's
// saved engram files (the coordinator's source of truth). For each file we
// engram_load() it, engram_scan_nodes_json(limit, offset) to get the node array,
// and cache each node's self-contained CONTENT string keyed by symbol. Because
// the cached value carries the numbers ("... f1=730 ..."), the cache SURVIVES the
// store being REPLACED by the next engram_load so we load+cache phonetics
// FIRST, then load+cache accent. The .psv path remains a fallback.
//
// engram_scan_nodes_json(limit, offset) takes NO query; it returns nodes
// salience-sorted, so limit must be >= node count and we filter client-side.
// (engram_search / engram_scan_nodes return len-5 garbage unused.)
// Find every occurrence of `marker` in the scan JSON; for each, cache
// sym -> a 150-char content window (enough to hold f1..amp). Duplicates from the
// node's "content" and "label" fields are harmless (first match wins on read).
fn organ_cache(j: String, marker: String, mlen: Int, win_len: Int, need: String) -> [String] {
let m: [String] = native_list_empty()
let jl: Int = str_len(j)
let off: Int = 0
while off < jl {
let rest: String = str_slice(j, off, jl)
let p: Int = str_index_of(rest, marker)
if p < 0 {
off = jl
} else {
let abs: Int = off + p
let win: String = str_slice(j, abs, abs + win_len)
let after: String = str_slice(win, mlen, str_len(win))
let sp: Int = str_index_of(after, " ")
let hasneed: Int = str_index_of(win, need)
if sp > 0 {
if hasneed >= 0 {
let sym: String = str_slice(after, 0, sp)
m = native_list_append(m, sym)
m = native_list_append(m, win)
}
}
off = abs + mlen
}
}
return m
}
// Load the phonetics organ file and cache sym -> content. mlen("phoneme ")=8.
fn organ_pmap(path: String) -> [String] {
let ok: Bool = engram_load(path)
if ok == false {
return native_list_empty()
}
let j: String = engram_scan_nodes_json(600, 0)
return organ_cache(j, "phoneme ", 8, 150, "f1=")
}
// Load the accent organ file and cache sym -> content. mlen("accent_target ")=14.
// Vowel overrides carry f1=..; the R rule carries drop_coda_r (need="=" matches
// both, i.e. any well-formed accent_target field).
fn organ_amap(path: String) -> [String] {
let ok: Bool = engram_load(path)
if ok == false {
return native_list_empty()
}
let j: String = engram_scan_nodes_json(600, 0)
return organ_cache(j, "accent_target ", 14, 90, "=")
}
// Vowel-set (categorical class) from the phonetics .psv class column.
fn organ_vset(path: String) -> [String] {
let content: String = fs_read(path)
let lines: [String] = str_split(content, "\n")
let nl: Int = native_list_len(lines)
let v: [String] = native_list_empty()
let li: Int = 0
while li < nl {
let line: String = native_list_get(lines, li)
let ok: Int = 1
if str_len(line) < 5 {
ok = 0
}
if ok == 1 {
if str_char_code(line, 0) == 35 {
ok = 0
}
}
if ok == 1 {
let f: [String] = str_split(line, "|")
if native_list_len(f) >= 12 {
if str_eq(native_list_get(f, 11), "vowel") {
v = native_list_append(v, native_list_get(f, 0))
}
}
}
li = li + 1
}
return v
}
// Word -> phoneme-sequence cache from lexicon.psv (engram-independent).
fn organ_lex(path: String) -> [String] {
let content: String = fs_read(path)
let lines: [String] = str_split(content, "\n")
let nl: Int = native_list_len(lines)
let m: [String] = native_list_empty()
let li: Int = 0
while li < nl {
let line: String = native_list_get(lines, li)
let ok: Int = 1
if str_len(line) < 3 {
ok = 0
}
if ok == 1 {
if str_char_code(line, 0) == 35 {
ok = 0
}
}
if ok == 1 {
let f: [String] = str_split(line, "|")
if native_list_len(f) >= 2 {
m = native_list_append(m, native_list_get(f, 0))
m = native_list_append(m, native_list_get(f, 1))
}
}
li = li + 1
}
return m
}
+140
View File
@@ -0,0 +1,140 @@
// propositions.el - the READ primitive over the engram's OWN memories, native el.
//
// Free memory text -> structured PROPOSITIONS (triples):
// (subject, predicate, object, modifiers, polarity, tense, source, confidence)
//
// This is comprehension turned inward: the Python reference (propositions.py) ran
// spaCy's dependency parser over each memory sentence and walked the arcs. Here
// the spaCy role is filled by the el-native parser (comprehend.el / parse_spec):
// each sentence is parsed to a meaning-spec, and the spec's roles ARE the triple.
// Nothing generates text. NEGATION IS SACRED: polarity flows straight from the
// spec's polarity field and is never dropped or inverted.
//
// Depends on: comprehend (parse_spec / parse_spec_lang), grammar (slots_get).
// sentence segmentation
// Split on sentence-final punctuation (. ! ?) and hard newlines. Markdown/long
// memories are handled shallowly (the reference caps + ranks by query overlap;
// that ranking belongs to the dialogue layer, not here).
fn prop_is_boundary(c: String) -> Bool {
if str_eq(c, ".") { return true }
if str_eq(c, "!") { return true }
if str_eq(c, "?") { return true }
if str_eq(c, "\n") { return true }
return false
}
fn prop_split_sentences(text: String) -> [String] {
let out: [String] = native_list_empty()
let n: Int = str_len(text)
let start: Int = 0
let i: Int = 0
while i < n {
let c: String = str_slice(text, i, i + 1)
if prop_is_boundary(c) {
let seg: String = str_slice(text, start, i + 1)
let trimmed: String = cp_trim_punct(seg)
if !str_eq(trimmed, "") {
let out = native_list_append(out, seg)
}
let start = i + 1
}
let i = i + 1
}
if start < n {
let seg: String = str_slice(text, start, n)
let trimmed: String = cp_trim_punct(seg)
if !str_eq(trimmed, "") {
let out = native_list_append(out, seg)
}
}
return out
}
// spec -> proposition record
// A proposition is a slot map (same [String] shape as the spec) with the READ
// contract keys. Modifiers fold the spec's location + iobj adjuncts.
fn prop_confidence(subject: String, predicate: String, object: String) -> String {
if str_eq(predicate, "") { return "0.0" }
if str_eq(subject, "") { return "0.4" }
if str_eq(object, "") { return "0.7" }
return "1.0"
}
fn prop_modifiers(spec: [String]) -> String {
let loc: String = slots_get(spec, "location")
let iobj: String = slots_get(spec, "iobj")
let parts: [String] = native_list_empty()
if !str_eq(loc, "") { let parts = native_list_append(parts, loc) }
if !str_eq(iobj, "") { let parts = native_list_append(parts, "to " + iobj) }
return str_join(parts, "; ")
}
fn prop_from_spec(spec: [String], source_id: String) -> [String] {
let subject: String = slots_get(spec, "agent")
let predicate: String = slots_get(spec, "predicate")
let object: String = slots_get(spec, "patient")
let polarity: String = slots_get(spec, "polarity")
let tense: String = slots_get(spec, "tense")
let mods: String = prop_modifiers(spec)
let conf: String = prop_confidence(subject, predicate, object)
let p: [String] = native_list_empty()
let p = native_list_append(p, "subject"); let p = native_list_append(p, subject)
let p = native_list_append(p, "predicate"); let p = native_list_append(p, predicate)
let p = native_list_append(p, "object"); let p = native_list_append(p, object)
let p = native_list_append(p, "modifiers"); let p = native_list_append(p, mods)
let p = native_list_append(p, "polarity"); let p = native_list_append(p, polarity)
let p = native_list_append(p, "tense"); let p = native_list_append(p, tense)
let p = native_list_append(p, "source"); let p = native_list_append(p, source_id)
let p = native_list_append(p, "confidence"); let p = native_list_append(p, conf)
return p
}
// Extract one proposition from a single sentence (given language).
fn prop_extract_one_lang(sentence: String, lang: String, source_id: String) -> [String] {
let spec: [String] = parse_spec_lang(sentence, lang)
return prop_from_spec(spec, source_id)
}
fn prop_extract_one(sentence: String, source_id: String) -> [String] {
return prop_extract_one_lang(sentence, "en", source_id)
}
// Render a proposition as a compact trace line (repr parity with propositions.py).
fn prop_repr(p: [String]) -> String {
let neg: String = ""
if str_eq(slots_get(p, "polarity"), "neg") { let neg = "NOT " }
let mods: String = slots_get(p, "modifiers")
let modstr: String = ""
if !str_eq(mods, "") { let modstr = " [" + mods + "]" }
let s: String = "(" + slots_get(p, "subject") + " -" + neg + slots_get(p, "predicate")
let s = s + "-> " + slots_get(p, "object") + modstr
let s = s + " conf=" + slots_get(p, "confidence") + ")"
return s
}
// Extract all propositions from a memory's text (one per sentence). Returns a
// flat [String] whose entries are the prop_repr trace lines, in reading order.
fn prop_extract_lang(text: String, lang: String, source_id: String) -> [String] {
let sents: [String] = prop_split_sentences(text)
let m: Int = native_list_len(sents)
let out: [String] = native_list_empty()
let i: Int = 0
while i < m {
let sent: String = native_list_get(sents, i)
let p: [String] = prop_extract_one_lang(sent, lang, source_id)
// drop empty parses (no predicate recovered): honest partial, not noise.
if !str_eq(slots_get(p, "predicate"), "") {
let out = native_list_append(out, prop_repr(p))
}
let i = i + 1
}
return out
}
fn prop_extract(text: String, source_id: String) -> [String] {
return prop_extract_lang(text, "en", source_id)
}
+125
View File
@@ -34,6 +34,13 @@ fn agent_person(agent: String) -> String {
if str_eq(agent, "we") { return "first" }
if str_eq(agent, "us") { return "first" }
if str_eq(agent, "you") { return "second" }
// Romance target-language subject pronouns (translate.el sets these).
if str_eq(agent, "yo") { return "first" }
if str_eq(agent, "eu") { return "first" }
if str_eq(agent, "nosotros") { return "first" }
if str_eq(agent, "nós") { return "first" }
if str_eq(agent, "") { return "second" }
if str_eq(agent, "tu") { return "second" }
return "third"
}
@@ -50,6 +57,19 @@ fn agent_number(agent: String) -> String {
if str_eq(agent, "us") { return "plural" }
if str_eq(agent, "they") { return "plural" }
if str_eq(agent, "them") { return "plural" }
// Romance target-language subject pronouns.
if str_eq(agent, "yo") { return "singular" }
if str_eq(agent, "eu") { return "singular" }
if str_eq(agent, "") { return "singular" }
if str_eq(agent, "tu") { return "singular" }
if str_eq(agent, "él") { return "singular" }
if str_eq(agent, "ella") { return "singular" }
if str_eq(agent, "ele") { return "singular" }
if str_eq(agent, "ela") { return "singular" }
if str_eq(agent, "nosotros") { return "plural" }
if str_eq(agent, "nós") { return "plural" }
if str_eq(agent, "ellos") { return "plural" }
if str_eq(agent, "eles") { return "plural" }
return "singular"
}
@@ -248,6 +268,56 @@ fn add_punct(s: String, intent: String) -> String {
return s + "."
}
// Polarity-aware negation (SACRED field honored on the generation side)
//
// Negation must never be dropped between comprehension and realization. The
// meaning-spec carries an explicit "polarity" field ("aff"|"neg") and optional
// "neg_word" (standalone negative adverb, e.g. "never"). English uses
// do-support ("did not see") or preverbal adverb ("never fought"); copular "be"
// takes post-verbal "not"; other languages get a preverbal negator particle.
fn realize_negator(code: String) -> String {
if str_eq(code, "es") { return "no" }
if str_eq(code, "pt") { return "não" }
if str_eq(code, "ca") { return "no" }
if str_eq(code, "it") { return "non" }
if str_eq(code, "fr") { return "ne" }
if str_eq(code, "de") { return "nicht" }
if str_eq(code, "ro") { return "nu" }
return "not"
}
fn realize_assert_neg_en(predicate: String, tense: String, person: String, number: String, agent: String, patient: String, iobj: String, location: String, neg_word: String, profile: [String]) -> String {
let parts: [String] = native_list_empty()
let parts = native_list_append(parts, agent)
if !str_eq(neg_word, "") {
// adverbial negation: "I never fought the ocean."
let verb_surf: String = morph_conjugate(predicate, tense, person, number, profile)
let parts = native_list_append(parts, neg_word)
let parts = native_list_append(parts, verb_surf)
} else {
if str_eq(predicate, "be") {
// copular: "she was not a monster"
let be_form: String = morph_conjugate("be", tense, person, number, profile)
let parts = native_list_append(parts, be_form)
let parts = native_list_append(parts, "not")
} else {
// do-support: "she did not see the man"
let do_form: String = morph_conjugate("do", tense, person, number, profile)
let parts = native_list_append(parts, do_form)
let parts = native_list_append(parts, "not")
let parts = native_list_append(parts, predicate)
}
}
if !str_eq(patient, "") { let parts = native_list_append(parts, patient) }
if !str_eq(iobj, "") {
let parts = native_list_append(parts, "to")
let parts = native_list_append(parts, iobj)
}
if !str_eq(location, "") { let parts = native_list_append(parts, location) }
return str_join(parts, " ")
}
// Main realization entry point
fn realize_lang(form: [String], profile: [String]) -> String {
@@ -284,6 +354,54 @@ fn realize_lang(form: [String], profile: [String]) -> String {
}
// Assertion (declarative)
let polarity: String = slots_get(form, "polarity")
let neg_word: String = slots_get(form, "neg_word")
let iobj: String = slots_get(form, "iobj")
let code: String = lang_get(profile, "code")
// Subordinate clause tail (SACRED completeness the clause is carried, never
// dropped): "<conj> <subordinate surface>", e.g. "because he was a monster".
let subord_conj: String = slots_get(form, "subord_conj")
let subord_text: String = slots_get(form, "subord_text")
let subord_tail: String = ""
if !str_eq(subord_conj, "") {
if !str_eq(subord_text, "") {
let subord_tail = subord_conj + " " + subord_text
} else {
let subord_tail = subord_conj
}
}
// Negative polarity: SACRED never dropped.
if str_eq(polarity, "neg") {
if str_eq(code, "en") {
let sentence: String = realize_assert_neg_en(predicate, tense, person, number, agent, patient, iobj, location, neg_word, profile)
return add_punct(capitalize_first(sentence), "assert")
}
// Generic non-English: affirmative core with a preverbal negator particle.
// SACRED: when a standalone negative adverb was carried (e.g. "nunca",
// localized upstream from "never"), surface it rather than the generic
// negator the specific negation must never be flattened away.
let neg_particle: String = realize_negator(code)
if !str_eq(neg_word, "") { let neg_particle = neg_word }
let vp_pair: [String] = realize_vp_lang(predicate, tense, aspect, person, number, profile)
let verb_surf: String = native_list_get(vp_pair, 0)
let aux_surf: String = native_list_get(vp_pair, 1)
let vp_str: String = neg_particle + " " + gram_build_vp(verb_surf, aux_surf, profile)
let core: String = gram_order_constituents(agent, vp_str, patient, profile)
let parts: [String] = native_list_empty()
let parts = native_list_append(parts, core)
if !str_eq(iobj, "") {
let parts = native_list_append(parts, "to")
let parts = native_list_append(parts, iobj)
}
if !str_eq(location, "") { let parts = native_list_append(parts, location) }
if !str_eq(subord_tail, "") { let parts = native_list_append(parts, subord_tail) }
let sentence: String = str_join(parts, " ")
return add_punct(capitalize_first(sentence), "assert")
}
// Affirmative.
let vp_pair: [String] = realize_vp_lang(predicate, tense, aspect, person, number, profile)
let verb_surf: String = native_list_get(vp_pair, 0)
let aux_surf: String = native_list_get(vp_pair, 1)
@@ -293,9 +411,16 @@ fn realize_lang(form: [String], profile: [String]) -> String {
let parts: [String] = native_list_empty()
let parts = native_list_append(parts, core)
if !str_eq(iobj, "") {
let parts = native_list_append(parts, "to")
let parts = native_list_append(parts, iobj)
}
if !str_eq(location, "") {
let parts = native_list_append(parts, location)
}
if !str_eq(subord_tail, "") {
let parts = native_list_append(parts, subord_tail)
}
let sentence: String = str_join(parts, " ")
return add_punct(capitalize_first(sentence), "assert")
}
+5 -5
View File
@@ -1,10 +1,10 @@
// auto-generated by elc --emit-header - do not edit
// auto-generated by elc --emit-header do not edit
extern fn agent_person(agent: String) -> String
extern fn agent_number(agent: String) -> String
extern fn realize_np(referent: String, number: String) -> String
extern fn realize_vp_lang(base_verb: String, tense: String, aspect: String, person: String, number: String, profile: Any) -> Any
extern fn realize_question_lang(predicate: String, tense: String, aspect: String, person: String, number: String, agent: String, patient: String, location: String, profile: Any) -> String
extern fn realize_vp_lang(base_verb: String, tense: String, aspect: String, person: String, number: String, profile: [String]) -> [String]
extern fn realize_question_lang(predicate: String, tense: String, aspect: String, person: String, number: String, agent: String, patient: String, location: String, profile: [String]) -> String
extern fn capitalize_first(s: String) -> String
extern fn add_punct(s: String, intent: String) -> String
extern fn realize_lang(form: Any, profile: Any) -> String
extern fn realize(form: Any) -> String
extern fn realize_lang(form: [String], profile: [String]) -> String
extern fn realize(form: [String]) -> String
+180
View File
@@ -0,0 +1,180 @@
// self_region.el the engram's REAL self/identity region, pulled at query time
// (native el). This replaces the hardcoded identity anchors and the canned
// "I'm Neuron, the engram you're speaking with." template: the identity LANDING
// signal and the identity READOUT both come from the engram's own Self/identity
// nodes, read through the in-process engram el API.
//
// Port of self_region.py. The Python module precomputed MiniLM landing vectors;
// here the engram's own store IS the geometry we pull the self nodes by
// single-term lexical search (the engram search is a single-term matcher, so we
// pool several probes) and rank them by self-signal. No text is generated; the
// readout is the self nodes' OWN prose, verbatim (SACRED negation survives by
// construction we never paraphrase, so a negated self-statement stays negated).
//
// ENGRAM el API NOTE: engram_search_json / engram_get_node_json / engram_node_full
// / engram_connect are C runtime builtins. Their argument order is the C order
// (engram_connect(from, to, weight, relation)), NOT the runtime/engram.el wrapper
// order we call the builtins directly and never concatenate that wrapper.
//
// Depends on: comprehend (str helpers via runtime), propositions (prop_split_sentences),
// multilingual (ml_tr), the engram builtins, the json builtins.
// single-term self probes (pooled, because engram search is single-term)
fn sr_terms() -> [String] {
let t: [String] = native_list_empty()
let t = native_list_append(t, "self")
let t = native_list_append(t, "identity")
let t = native_list_append(t, "Neuron")
let t = native_list_append(t, "consciousness")
let t = native_list_append(t, "values")
let t = native_list_append(t, "continuous")
return t
}
// The canonical self-root: content begins "# self" or label is "# self"/"self".
fn sr_is_root(content: String, label: String) -> Bool {
let lc: String = str_to_lower(content)
let ll: String = str_to_lower(str_trim(label))
if str_starts_with(lc, "# self") { return true }
if str_eq(ll, "# self") { return true }
if str_eq(ll, "self") { return true }
return false
}
// How strongly a node belongs to the self/identity region (integer points, to
// avoid el's float-in-`+` pitfalls). Mirrors _self_score in self_region.py.
fn sr_score(node_json: String) -> Int {
let content: String = json_get_string(node_json, "content")
let label: String = json_get_string(node_json, "label")
let tags: String = str_to_lower(json_get_string(node_json, "tags"))
let low: String = str_to_lower(content)
let s: Int = 0
// identity tags
if str_contains(tags, "self") { let s = s + 2 }
if str_contains(tags, "identity") { let s = s + 2 }
if str_contains(tags, "self-model") { let s = s + 2 }
if str_contains(tags, "consciousness") { let s = s + 2 }
if str_contains(tags, "memory-philosophy") { let s = s + 2 }
// the named self-traversal root
if sr_is_root(content, label) { let s = s + 12 }
if str_contains(low, "who i am") { let s = s + 3 }
if str_contains(low, "i am neuron") { let s = s + 3 }
// softer identity keywords
if str_contains(low, "my values") { let s = s + 1 }
if str_contains(low, "my purpose") { let s = s + 1 }
if str_contains(low, "identity") { let s = s + 1 }
return s
}
// list-contains helper (dedup self-node ids across the pooled probes).
fn sr_ids_has(ids: [String], id: String) -> Bool {
let n: Int = native_list_len(ids)
let i: Int = 0
while i < n {
if str_eq(native_list_get(ids, i), id) { return true }
let i = i + 1
}
return false
}
// Pull the self nodes: pool every probe's hits, dedupe by id, keep only nodes
// with genuine self-signal (score >= 1). Returns the node-json strings.
fn sr_pull() -> [String] {
let terms: [String] = sr_terms()
let nt: Int = native_list_len(terms)
let seen: [String] = native_list_empty()
let out: [String] = native_list_empty()
let ti: Int = 0
while ti < nt {
let term: String = native_list_get(terms, ti)
let hits: String = engram_search_json(term, 30)
let hn: Int = json_array_len(hits)
let hi: Int = 0
while hi < hn {
let node: String = json_array_get(hits, hi)
let id: String = json_get_string(node, "id")
if !str_eq(id, "") {
if !sr_ids_has(seen, id) {
let seen = native_list_append(seen, id)
if sr_score(node) >= 1 {
let out = native_list_append(out, node)
}
}
}
let hi = hi + 1
}
let ti = ti + 1
}
return out
}
// Return the single highest-signal self node (the readout seed), or "" if the
// self region is thin/empty. We keep it O(n) pick the max-score node, with the
// canonical root strongly favored by sr_score's +12.
fn sr_best_node() -> String {
let nodes: [String] = sr_pull()
let n: Int = native_list_len(nodes)
let best: String = ""
let best_s: Int = 0
let i: Int = 0
while i < n {
let node: String = native_list_get(nodes, i)
let s: Int = sr_score(node)
if s > best_s {
let best_s = s
let best = node
}
let i = i + 1
}
return best
}
fn sr_available() -> Bool {
if str_eq(sr_best_node(), "") { return false }
return true
}
// Read out the identity from the REAL self node: lead with the first first-person
// self-statement ("I am Neuron …"), then one more grounded self line if present.
// Verbatim from the node's own prose no template, negation SACRED. Falls back
// to the localized identity phrase ONLY if the live pull is empty (logged shape).
fn sr_readout(lang: String) -> String {
let node: String = sr_best_node()
if str_eq(node, "") {
// honest fallback the self region is unreachable/thin.
return ml_tr("identity", lang)
}
let content: String = json_get_string(node, "content")
let sents: [String] = prop_split_sentences(content)
let ns: Int = native_list_len(sents)
let lead: String = ""
let second: String = ""
let i: Int = 0
while i < ns {
let raw: String = str_trim(native_list_get(sents, i))
// strip a leading markdown heading marker
let s: String = raw
if str_starts_with(s, "# ") { let s = str_trim(str_slice(s, 2, str_len(s))) }
let low: String = str_to_lower(s)
let is_fp: Bool = false
if str_starts_with(s, "I ") { let is_fp = true }
if str_starts_with(s, "I'm") { let is_fp = true }
if str_contains(low, "i am neuron") { let is_fp = true }
if is_fp {
if str_eq(lead, "") {
let lead = s
} else {
if str_eq(second, "") { let second = s }
}
}
let i = i + 1
}
if str_eq(lead, "") {
// no first-person line read out the first non-empty sentence verbatim.
if ns > 0 { let lead = str_trim(native_list_get(sents, 0)) }
}
if str_eq(lead, "") { return ml_tr("identity", lang) }
let out: String = lead
if !str_eq(second, "") { let out = out + " " + second }
return out
}
+15 -15
View File
@@ -1,18 +1,18 @@
// auto-generated by elc --emit-header - do not edit
extern fn sem_frame(intent: String, subject: String, obj: String, modifiers: String) -> Any
extern fn sem_frame_lang(intent: String, subject: String, obj: String, modifiers: String, lang_code: String) -> Any
extern fn sem_frame_simple(intent: String, subject: String) -> Any
extern fn sem_frame_obj(intent: String, subject: String, obj: String) -> Any
extern fn sem_intent(frame: Any) -> String
extern fn sem_subject(frame: Any) -> String
extern fn sem_object(frame: Any) -> String
extern fn sem_modifiers(frame: Any) -> String
extern fn sem_lang(frame: Any) -> String
// auto-generated by elc --emit-header do not edit
extern fn sem_frame(intent: String, subject: String, obj: String, modifiers: String) -> [String]
extern fn sem_frame_lang(intent: String, subject: String, obj: String, modifiers: String, lang_code: String) -> [String]
extern fn sem_frame_simple(intent: String, subject: String) -> [String]
extern fn sem_frame_obj(intent: String, subject: String, obj: String) -> [String]
extern fn sem_intent(frame: [String]) -> String
extern fn sem_subject(frame: [String]) -> String
extern fn sem_object(frame: [String]) -> String
extern fn sem_modifiers(frame: [String]) -> String
extern fn sem_lang(frame: [String]) -> String
extern fn sem_first_modifier(mods: String) -> String
extern fn sem_intent_to_realize(intent: String) -> String
extern fn sem_to_spec(frame: Any) -> Any
extern fn sem_to_spec_full(frame: Any, verb: String, tense: String, aspect: String) -> Any
extern fn sem_to_spec(frame: [String]) -> [String]
extern fn sem_to_spec_full(frame: [String], verb: String, tense: String, aspect: String) -> [String]
extern fn sem_realize_greet(subject: String) -> String
extern fn sem_realize(frame: Any) -> String
extern fn sem_realize_full(frame: Any, verb: String, tense: String, aspect: String) -> String
extern fn sem_realize_lang(frame: Any, lang_code: String) -> String
extern fn sem_realize(frame: [String]) -> String
extern fn sem_realize_full(frame: [String], verb: String, tense: String, aspect: String) -> String
extern fn sem_realize_lang(frame: [String], lang_code: String) -> String
+233
View File
@@ -0,0 +1,233 @@
// speech-ingest.el - The native LOAD step of the ingest organ, for the SPEECH
// primitives. Reads the acoustic-phonetics SOURCE (elp/data/phonetics.psv) and
// the pronunciation lexicon SOURCE (elp/data/lexicon.psv) and emits a PHONEME
// MANIFOLD into the engram: one node per phoneme (faithful, provenance-tagged
// content) + is_a edges to phoneme-class nodes (a discrete manifold, not islands).
// The render then PULLS phoneme geometry back from the engram via phon_geo
// zero phonetic numbers in code. Source -> manifold -> merge; the same output
// the polymorphic ingest organ will produce and subsume.
// -- small parsing helpers ---------------------------------------------------
fn sp_map_get(pairs: [String], key: String) -> String {
let n: Int = native_list_len(pairs)
let i: Int = 0
while i < n - 1 {
let k: String = native_list_get(pairs, i)
if str_eq(k, key) {
return native_list_get(pairs, i + 1)
}
let i = i + 2
}
return ""
}
// read the unsigned integer that follows `key` inside string s (e.g. key "F1=")
fn parse_uint_from(s: String, key: String) -> Int {
let idx: Int = str_index_of(s, key)
if idx < 0 {
return 0
}
let start: Int = idx + str_len(key)
let n: Int = str_len(s)
let i: Int = start
let val: Int = 0
while i < n {
let c: Int = str_char_code(s, i)
if c >= 48 {
if c <= 57 {
val = val * 10 + (c - 48)
i = i + 1
} else {
i = n
}
} else {
i = n
}
}
return val
}
fn clean_word(w: String) -> String {
let low: String = str_to_lower(w)
let n: Int = str_len(low)
let out: String = ""
let i: Int = 0
while i < n {
let c: Int = str_char_code(low, i)
if c >= 97 {
if c <= 122 {
out = out + str_char_at(low, i)
}
}
i = i + 1
}
return out
}
// -- INGEST: acoustic-phonetics source -> phoneme manifold in the engram ------
// Returns the symbol -> node-id index (pmap) the render reads geometry through.
fn ingest_phonetics(path: String) -> [String] {
let content: String = fs_read(path)
let lines: [String] = str_split(content, "\n")
let nl: Int = native_list_len(lines)
let pmap: [String] = native_list_empty()
let classmap: [String] = native_list_empty()
let li: Int = 0
while li < nl {
let line: String = native_list_get(lines, li)
let ll: Int = str_len(line)
let skip: Int = 0
if ll < 5 {
skip = 1
}
if skip == 0 {
let first: Int = str_char_code(line, 0)
if first == 35 {
skip = 1
}
}
if skip == 0 {
let f: [String] = str_split(line, "|")
let nf: Int = native_list_len(f)
if nf >= 12 {
let sym: String = native_list_get(f, 0)
let f1: String = native_list_get(f, 1)
let f2: String = native_list_get(f, 2)
let f3: String = native_list_get(f, 3)
let b1: String = native_list_get(f, 4)
let b2: String = native_list_get(f, 5)
let b3: String = native_list_get(f, 6)
let vo: String = native_list_get(f, 7)
let na: String = native_list_get(f, 8)
let du: String = native_list_get(f, 9)
let am: String = native_list_get(f, 10)
let cls: String = native_list_get(f, 11)
let cont: String = "phoneme " + sym + " | f1=" + f1 + " f2=" + f2 + " f3=" + f3 + " bw1=" + b1 + " bw2=" + b2 + " bw3=" + b3 + " voiced=" + vo + " nasal=" + na + " dur=" + du + " amp=" + am + " class=" + cls + " src=PetersonBarney1952-Hillenbrand1995"
let id: String = engram_node(cont, "Phoneme", 80)
pmap = native_list_append(pmap, sym)
pmap = native_list_append(pmap, cont)
// manifold edge: phoneme is_a class
let cid: String = sp_map_get(classmap, cls)
if str_eq(cid, "") {
cid = engram_node("phoneme-class " + cls + " src=acoustic-phonetics", "PhonemeClass", 80)
classmap = native_list_append(classmap, cls)
classmap = native_list_append(classmap, cid)
}
engram_connect(id, cid, 80, "is_a")
}
}
li = li + 1
}
return pmap
}
// -- INGEST: pronunciation lexicon source -> word nodes ----------------------
fn ingest_lexicon(path: String) -> [String] {
let content: String = fs_read(path)
let lines: [String] = str_split(content, "\n")
let nl: Int = native_list_len(lines)
let lmap: [String] = native_list_empty()
let li: Int = 0
while li < nl {
let line: String = native_list_get(lines, li)
let ll: Int = str_len(line)
let skip: Int = 0
if ll < 3 {
skip = 1
}
if skip == 0 {
let first: Int = str_char_code(line, 0)
if first == 35 {
skip = 1
}
}
if skip == 0 {
let f: [String] = str_split(line, "|")
let nf: Int = native_list_len(f)
if nf >= 2 {
let word: String = native_list_get(f, 0)
let seq: String = native_list_get(f, 1)
let id: String = engram_node("word " + word + " phonemes " + seq + " src=lexicon", "Pronunciation", 80)
lmap = native_list_append(lmap, word)
lmap = native_list_append(lmap, seq)
}
}
li = li + 1
}
return lmap
}
// -- READ geometry back from the engram (the render's afferent lookup) --------
// phon_geo(sym) -> [F1,F2,F3,B1,B2,B3,voiced,nasal,dur,amp], parsed from the
// ingested phoneme node's content. NO formant numbers live in this code.
fn phon_geo(pmap: [String], sym: String) -> [Int] {
let id: String = sp_map_get(pmap, sym)
if str_eq(id, "") {
id = sp_map_get(pmap, "AX")
}
let out: [Int] = native_list_empty()
if str_eq(id, "") {
let out = native_list_append(out, 500)
let out = native_list_append(out, 1500)
let out = native_list_append(out, 2500)
let out = native_list_append(out, 80)
let out = native_list_append(out, 100)
let out = native_list_append(out, 150)
let out = native_list_append(out, 1)
let out = native_list_append(out, 0)
let out = native_list_append(out, 80)
let out = native_list_append(out, 80)
return out
}
let j: String = id
let out = native_list_append(out, parse_uint_from(j, "f1="))
let out = native_list_append(out, parse_uint_from(j, "f2="))
let out = native_list_append(out, parse_uint_from(j, "f3="))
let out = native_list_append(out, parse_uint_from(j, "bw1="))
let out = native_list_append(out, parse_uint_from(j, "bw2="))
let out = native_list_append(out, parse_uint_from(j, "bw3="))
let out = native_list_append(out, parse_uint_from(j, "voiced="))
let out = native_list_append(out, parse_uint_from(j, "nasal="))
let out = native_list_append(out, parse_uint_from(j, "dur="))
let out = native_list_append(out, parse_uint_from(j, "amp="))
return out
}
// word -> phoneme codes, read from the ingested lexicon node.
fn word_phonemes(lmap: [String], word: String) -> [String] {
let id: String = sp_map_get(lmap, word)
if str_eq(id, "") {
let r: [String] = native_list_empty()
let r = native_list_append(r, "AX")
return r
}
return str_split(id, " ")
}
// realized text -> flat phoneme-code sequence (SIL between words + at ends).
fn text_phonemes(lmap: [String], text: String) -> [String] {
let words: [String] = str_split(text, " ")
let nw: Int = native_list_len(words)
let seq: [String] = native_list_empty()
let seq = native_list_append(seq, "SIL")
let wi: Int = 0
while wi < nw {
let raw: String = native_list_get(words, wi)
let w: String = clean_word(raw)
if str_eq(w, "") {
wi = wi + 1
} else {
let ph: [String] = word_phonemes(lmap, w)
let np: Int = native_list_len(ph)
let pi: Int = 0
while pi < np {
let code: String = native_list_get(ph, pi)
seq = native_list_append(seq, code)
pi = pi + 1
}
seq = native_list_append(seq, "SIL")
wi = wi + 1
}
}
return seq
}
+460
View File
@@ -0,0 +1,460 @@
// speech.el - The native SPEECH render path + voice-by-imitation extractor.
//
// Speech = the AUDIO surface (surface_profile_audio) rendering LANGUAGE-meaning
// through a VOICE signature. The realizer's language faculty supplies the words
// (meaning -> sem_realize -> text); this module turns text -> phonemes (phonetics.el)
// -> a formant-target track over time -> SUPERPOSES formant resonances over a
// glottal source (own-core formant synthesis, the exact integer mirror of the
// music additive superpose) -> own-core PCM/WAV. Two paths:
// (1) RENDER: speak(text, voice) -> spoken WAV.
// (2) IMITATE: voice_analyze(pcm) -> a voice signature grabbed BY EAR
// (autocorrelation pitch + integer-DFT formant peaks), then render
// any new meaning in that voice. An impression, not a corpus.
// All integer/fixed-point (EL float arithmetic is unusable).
// -- Own-core integer sine (Bhaskara I), phase 0..65535 = one cycle -----------
fn sp_sin(phase: Int) -> Int {
let deg: Int = phase * 360 / 65536
let neg: Int = 0
if deg > 180 {
deg = deg - 180
neg = 1
}
let t: Int = deg * (180 - deg)
let num: Int = 32767 * 4 * t
let den: Int = 40500 - t
let v: Int = num / den
if neg == 1 {
v = 0 - v
}
return v
}
fn sp_cos(phase: Int) -> Int {
let p: Int = phase + 16384
p = p - (p / 65536) * 65536
return sp_sin(p)
}
// One formant resonance (Lorentzian peak), Q15. Peak 32767 at f=fc.
fn sp_gain(f: Int, fc: Int, bw: Int) -> Int {
let d: Int = f - fc
let den: Int = d * d + bw * bw
let num: Int = 32767 * bw * bw
return num / den
}
fn sp_isqrt(n: Int) -> Int {
if n <= 0 {
return 0
}
let x: Int = n
let y: Int = (x + 1) / 2
while y < x {
x = y
y = (x + n / x) / 2
}
return x
}
// -- WAV serializer (thin medium; the only non-DSP glue) ---------------------
fn wav_le16(buf: String, off: Int, v: Int) -> String {
let u: Int = v
if u < 0 {
u = u + 65536
}
let lo: Int = u - (u / 256) * 256
let hi: Int = u / 256
let b: String = __str_set_char(buf, off, lo)
b = __str_set_char(b, off + 1, hi)
return b
}
fn wav_le32(buf: String, off: Int, v: Int) -> String {
let b0: Int = v - (v / 256) * 256
let r1: Int = v / 256
let b1: Int = r1 - (r1 / 256) * 256
let r2: Int = r1 / 256
let b2: Int = r2 - (r2 / 256) * 256
let b3: Int = r2 / 256
let b: String = __str_set_char(buf, off, b0)
b = __str_set_char(b, off + 1, b1)
b = __str_set_char(b, off + 2, b2)
b = __str_set_char(b, off + 3, b3)
return b
}
fn wav_ascii(buf: String, off: Int, s: String) -> String {
let n: Int = str_len(s)
let i: Int = 0
let b: String = buf
while i < n {
let c: Int = str_char_code(s, i)
b = __str_set_char(b, off + i, c)
i = i + 1
}
return b
}
fn write_wav(samples: [Int], sr: Int, path: String) -> Bool {
let ns: Int = native_list_len(samples)
let datalen: Int = ns * 2
let total: Int = 44 + datalen
let buf: String = __str_alloc(total)
buf = wav_ascii(buf, 0, "RIFF")
buf = wav_le32(buf, 4, 36 + datalen)
buf = wav_ascii(buf, 8, "WAVE")
buf = wav_ascii(buf, 12, "fmt ")
buf = wav_le32(buf, 16, 16)
buf = wav_le16(buf, 20, 1)
buf = wav_le16(buf, 22, 1)
buf = wav_le32(buf, 24, sr)
buf = wav_le32(buf, 28, sr * 2)
buf = wav_le16(buf, 32, 2)
buf = wav_le16(buf, 34, 16)
buf = wav_ascii(buf, 36, "data")
buf = wav_le32(buf, 40, datalen)
let j: Int = 0
let off: Int = 44
while j < ns {
let raw: Int = native_list_get(samples, j)
buf = wav_le16(buf, off, raw)
off = off + 2
j = j + 1
}
return __fs_write_bytes(path, buf, total)
}
// One formant resonance as a float Lorentzian peak (own-core physics).
fn fgain(f: Float, fc: Float, bw: Float) -> Float {
let d: Float = f - fc
return (bw * bw) / (d * d + bw * bw)
}
// His PITCH MELODY from measured prosody [f0_median, f0_min, f0_max, declination].
// A natural statement shape over the utterance: onset rise to the median, a
// near-flat body (his declination is ~0.6 Hz/s), and a final fall toward f0_min.
// Follows his melody + range, not a fixed 0.85 decline. gidx/total = position.
fn prosody_f0(pros: [Int], gidx: Int, total: Int) -> Int {
let med: Int = native_list_get(pros, 0)
let lo: Int = native_list_get(pros, 1)
let hi: Int = native_list_get(pros, 2)
let p: Int = gidx * 1000 / total
let f0: Int = med
if p < 150 {
f0 = lo + (med - lo) * p / 150
} else {
if p > 700 {
f0 = med + (lo - med) * (p - 700) / 300
} else {
f0 = med
}
}
if f0 < lo {
f0 = lo
}
if f0 > hi {
f0 = hi
}
return f0
}
// -- The render: phoneme codes + voice signature -> normalized PCM samples ----
// Formant geometry per phoneme is READ FROM THE ENGRAM (pmap) via phon_geo no
// table in code. The optional ACCENT map (amap) composes a transform onto the
// voice (voice (+) accent, separable): RP formant overrides read from the accent
// manifold + a non-rhotic coda-R drop. Empty amap = base General-American.
// Synthesis is FLOAT: a real phase accumulator + math_sin, superposition physics.
fn synth_codes_accent(codes0: [String], voice: [String], pmap: [String], amap: [String], vset: [String], vmap: [String], prosody: [Int]) -> [Int] {
let sr: Int = 16000
let srf: Float = 16000.0
let two_pi: Float = 6.283185307
let kf: Int = voice_get_int(voice, "kf")
let f0s: Int = voice_get_int(voice, "f0")
let f0e: Int = voice_get_int(voice, "f0_end")
let durm: Int = voice_get_int(voice, "dur")
if kf <= 0 {
kf = 1000
}
if durm <= 0 {
durm = 1000
}
let use_accent: Int = 0
if native_list_len(amap) > 0 {
use_accent = 1
}
let codes: [String] = codes0
if use_accent == 1 {
if is_nonrhotic(amap) == 1 {
codes = apply_rhoticity(codes0, vset)
}
}
let nc: Int = native_list_len(codes)
// pass 1: per-segment sample counts + total
let segn: [Int] = native_list_empty()
let total: Int = 0
let ci: Int = 0
while ci < nc {
let code: String = native_list_get(codes, ci)
let p: [Int] = phon_geo(pmap, code)
let durms: Int = native_list_get(p, 8)
let ns: Int = durms * 16 * durm / 1000
segn = native_list_append(segn, ns)
total = total + ns
ci = ci + 1
}
if total <= 0 {
total = 1
}
// pass 2: synthesize
let samples: [Int] = native_list_empty()
let phasef: Float = 0.0
let gidx: Int = 0
let prevF1: Int = 500 * kf / 1000
let prevF2: Int = 1500 * kf / 1000
let prevF3: Int = 2500 * kf / 1000
let nstate: Int = 22695
let maxabs: Int = 1
let ci2: Int = 0
while ci2 < nc {
let code: String = native_list_get(codes, ci2)
let p: [Int] = phon_geo(pmap, code)
let rf1: Int = native_list_get(p, 0)
let rf2: Int = native_list_get(p, 1)
let rf3: Int = native_list_get(p, 2)
if use_accent == 1 {
let ov: [Int] = accent_formants(amap, code)
if native_list_len(ov) >= 3 {
rf1 = native_list_get(ov, 0)
rf2 = native_list_get(ov, 1)
rf3 = native_list_get(ov, 2)
}
}
// HIS measured vowel target overrides the generic/kf path (absolute Hz
// his formants already encode his vocal tract, so no kf scaling).
let usekf: Int = 1
if native_list_len(vmap) > 0 {
let hv: [Int] = vmap_get(vmap, code)
if native_list_len(hv) >= 3 {
rf1 = native_list_get(hv, 0)
rf2 = native_list_get(hv, 1)
rf3 = native_list_get(hv, 2)
usekf = 0
}
}
let F1t: Int = rf1 * kf / 1000
let F2t: Int = rf2 * kf / 1000
let F3t: Int = rf3 * kf / 1000
if usekf == 0 {
F1t = rf1
F2t = rf2
F3t = rf3
}
let B1: Int = native_list_get(p, 3)
let B2: Int = native_list_get(p, 4)
let B3: Int = native_list_get(p, 5)
let voiced: Int = native_list_get(p, 6)
let ampv: Int = native_list_get(p, 9)
let ns: Int = native_list_get(segn, ci2)
let trans: Int = ns / 2
if trans > 560 {
trans = 560
}
if trans < 1 {
trans = 1
}
let k: Int = 0
while k < ns {
let cF1: Int = F1t
let cF2: Int = F2t
let cF3: Int = F3t
if k < trans {
cF1 = prevF1 + (F1t - prevF1) * k / trans
cF2 = prevF2 + (F2t - prevF2) * k / trans
cF3 = prevF3 + (F3t - prevF3) * k / trans
}
let f0c: Int = f0s + (f0e - f0s) * gidx / total
if native_list_len(prosody) >= 3 {
f0c = prosody_f0(prosody, gidx, total)
}
if f0c < 40 {
f0c = 40
}
let env: Int = 32767
let ar: Int = 96
if k < ar {
env = 32767 * k / ar
}
let tail: Int = ns - k
if tail < ar {
env = 32767 * tail / ar
}
let f0cf: Float = int_to_float(f0c)
phasef = phasef + two_pi * f0cf / srf
if phasef > two_pi {
phasef = phasef - two_pi
}
let s: Int = 0
if voiced == 1 {
let cF1f: Float = int_to_float(cF1)
let cF2f: Float = int_to_float(cF2)
let cF3f: Float = int_to_float(cF3)
let B1f: Float = int_to_float(B1)
let B2f: Float = int_to_float(B2)
let B3f: Float = int_to_float(B3)
let acc: Float = 0.0
let h: Int = 1
while h <= 50 {
let hf: Float = int_to_float(h)
let fhf: Float = hf * f0cf
if fhf < 7900.0 {
let sv: Float = math_sin(phasef * hf)
let src: Float = 1.0 / hf
let g1: Float = fgain(fhf, cF1f, B1f)
let g2: Float = fgain(fhf, cF2f, B2f)
let g3: Float = fgain(fhf, cF3f, B3f)
let g: Float = g1 + g2 + g3
acc = acc + src * g * sv
}
h = h + 1
}
s = float_to_int(acc * 4000.0)
} else {
if ampv > 0 {
nstate = nstate * 1103515245 + 12345
nstate = nstate - (nstate / 2147483648) * 2147483648
if nstate < 0 {
nstate = 0 - nstate
}
let nz: Int = nstate / 32768 - 32768
s = nz
}
}
s = s * ampv / 100
s = s * env / 32767
samples = native_list_append(samples, s)
let a: Int = s
if a < 0 {
a = 0 - a
}
if a > maxabs {
maxabs = a
}
gidx = gidx + 1
k = k + 1
}
prevF1 = F1t
prevF2 = F2t
prevF3 = F3t
ci2 = ci2 + 1
}
// normalize to int16 range (~22000 peak)
let out: [Int] = native_list_empty()
let ntot: Int = native_list_len(samples)
let j: Int = 0
while j < ntot {
let raw: Int = native_list_get(samples, j)
let v: Int = raw * 22000 / maxabs
out = native_list_append(out, v)
j = j + 1
}
return out
}
// GA convenience wrapper (no accent) keeps the base render path.
fn synth_codes(codes: [String], voice: [String], pmap: [String]) -> [Int] {
let noacc: [String] = native_list_empty()
let novset: [String] = native_list_empty()
let novmap: [String] = native_list_empty()
let nopros: [Int] = native_list_empty()
return synth_codes_accent(codes, voice, pmap, noacc, novset, novmap, nopros)
}
// -- Voice-by-imitation: HEAR a PCM sample -> extract the voice signature -----
// Pitch by autocorrelation; vocal-tract scale (kf) from the F1 formant peak of a
// heard sustained vowel /AA/ (nominal F1 = 730 Hz) via an integer DFT. The
// analyzer sees ONLY the PCM samples never the source signature numbers so
// recovery is genuinely by ear.
fn voice_f0(samples: [Int], sr: Int) -> Int {
let n: Int = native_list_len(samples)
let start: Int = n / 4
let end: Int = n * 3 / 4
// bound the analysis window so accumulators can never overflow on long input
if end - start > 6000 {
end = start + 6000
}
let minlag: Int = sr / 300
let maxlag: Int = sr / 75
let best: Int = 0
let bestlag: Int = minlag
let lag: Int = minlag
while lag <= maxlag {
let sum: Int = 0
let i: Int = start
while i < end {
let ai: Int = native_list_get(samples, i)
let bi: Int = native_list_get(samples, i + lag)
sum = sum + ai * bi / 256
i = i + 2
}
if sum > best {
best = sum
bestlag = lag
}
lag = lag + 1
}
if bestlag < 1 {
bestlag = 1
}
return sr / bestlag
}
fn voice_peak_in_band(samples: [Int], sr: Int, flo: Int, fhi: Int) -> Int {
let n: Int = native_list_len(samples)
let start: Int = n / 4
let end: Int = n * 3 / 4
// bound the DFT window: re/im are accumulated /4096, and re*re must stay in
// int64 cap terms so (window/2)*(peak_term) squared cannot overflow.
if end - start > 3000 {
end = start + 3000
}
let bestmag: Int = 0
let bestf: Int = flo
let f: Int = flo
while f <= fhi {
let re: Int = 0
let im: Int = 0
let i: Int = start
while i < end {
let x: Int = native_list_get(samples, i)
let ph: Int = i * f * 65536 / sr
ph = ph - (ph / 65536) * 65536
let cq: Int = sp_cos(ph)
let sq: Int = sp_sin(ph)
re = re + x * cq / 4096
im = im + x * sq / 4096
i = i + 2
}
let mag: Int = re * re + im * im
if mag > bestmag {
bestmag = mag
bestf = f
}
f = f + 25
}
return bestf
}
// Analyze a heard sustained /AA/ -> a full voice signature (by ear).
fn voice_analyze(samples: [Int], sr: Int) -> [String] {
let f0: Int = voice_f0(samples, sr)
let f1: Int = voice_peak_in_band(samples, sr, 450, 1150)
let kf: Int = 1000 * f1 / 730
let f0e: Int = f0 * 85 / 100
return voice_new("imitated", f0, f0e, kf, 1000, 1000, 8)
}
+153
View File
@@ -0,0 +1,153 @@
// surface-profile.el - Surface profile data and accessors.
//
// THE NATIVE EFFERENT SEAM: surface = a pluggable PROFILE, using the exact same
// slot-map mechanism as language-profile.el. A language profile tells the
// realizer HOW to shape a natural-language surface (word order, morphology); a
// SURFACE profile tells the realizer WHICH surface to project meaning onto
// (markdown, docx, html, plain, or a non-text medium like symbolic music).
//
// The generalization is exact: realize_lang(form, profile) already renders a
// SemForm parameterized by a [String] profile read via lang_get. Surface is one
// more axis of that same profile vector. One frame (sem_frame), one plan step
// (sem_to_spec), one render (realize) the surface is DATA, not a code path,
// precisely as language is data. Adding a surface means adding a profile, no
// engine change. This is the multimodal projector, native: geometry -> any
// surface, the efferent twin of ingest.
//
// Surface slot keys:
// surface - "markdown" | "docx" | "html" | "plain" | "midi" | "image"
// modality - "text" | "audio" | "image" | "video"
// media_type - MIME type of the emitted surface
// head_open - string prepended to a heading (e.g. "## " for markdown)
// head_close - string appended to a heading (e.g. "" for markdown, "</h2>" for html)
// emph_open - string opening emphasis (e.g. "*")
// emph_close - string closing emphasis (e.g. "*")
// item_mark - list-item marker (e.g. "- ")
// para_sep - paragraph separator (e.g. "\n\n")
//
// For a TEXT modality the render composes these markers around the surface that
// the EXISTING realizer produces (realize_lang / sem_realize). For a non-text
// modality (audio/image) the profile declares modality + media_type and the
// render dispatches to the medium projector, which reads the SAME frame's
// geometry (its intent/affect/structure) and projects it onto sound or pixels
// deterministic-from-meaning, nothing invented. That dispatch point is where a
// music profile or image profile conforms, native, no parallel layer.
// -- Constructor -------------------------------------------------------------
fn surface_profile(surface: String, modality: String, media_type: String, head_open: String, head_close: String, emph_open: String, emph_close: String, item_mark: String, para_sep: String) -> [String] {
let r: [String] = native_list_empty()
let r = native_list_append(r, "surface")
let r = native_list_append(r, surface)
let r = native_list_append(r, "modality")
let r = native_list_append(r, modality)
let r = native_list_append(r, "media_type")
let r = native_list_append(r, media_type)
let r = native_list_append(r, "head_open")
let r = native_list_append(r, head_open)
let r = native_list_append(r, "head_close")
let r = native_list_append(r, head_close)
let r = native_list_append(r, "emph_open")
let r = native_list_append(r, emph_open)
let r = native_list_append(r, "emph_close")
let r = native_list_append(r, emph_close)
let r = native_list_append(r, "item_mark")
let r = native_list_append(r, item_mark)
let r = native_list_append(r, "para_sep")
let r = native_list_append(r, para_sep)
return r
}
// -- Accessor (same convention as lang_get; standalone so this is a leaf) -----
fn surface_get(profile: [String], key: String) -> String {
let n: Int = native_list_len(profile)
let i: Int = 0
while i < n - 1 {
let k: String = native_list_get(profile, i)
if str_eq(k, key) {
return native_list_get(profile, i + 1)
}
let i = i + 2
}
return ""
}
fn surface_is_text(profile: [String]) -> Bool {
return str_eq(surface_get(profile, "modality"), "text")
}
// -- Built-in TEXT surface profiles ------------------------------------------
// Markdown: headings with "## ", emphasis with "*", "- " list items.
fn surface_profile_markdown() -> [String] {
return surface_profile("markdown", "text", "text/markdown", "## ", "", "*", "*", "- ", "\n\n")
}
// Plain text: no markup at all headings become bare uppercase-free lines.
fn surface_profile_plain() -> [String] {
return surface_profile("plain", "text", "text/plain", "", "", "", "", " - ", "\n\n")
}
// HTML: block-level heading/emphasis tags.
fn surface_profile_html() -> [String] {
return surface_profile("html", "text", "text/html", "<h2>", "</h2>", "<em>", "</em>", "<li>", "\n")
}
// docx: WordprocessingML is structural, not inline-markup; the head/emph slots
// carry the run/style intent that the OOXML emitter maps to <w:pStyle>. Declared
// here so docx is a first-class surface on the same seam.
fn surface_profile_docx() -> [String] {
return surface_profile("docx", "text", "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "Heading2:", "", "b:", "", "bullet:", "\n")
}
// -- Built-in NON-TEXT surface profiles (the multimodal seam) ----------------
// Symbolic music (MIDI): modality=audio. The render dispatches to the music
// projector, which reads the SAME frame's intent/affect and projects it to
// pitch/rhythm deterministic-from-meaning. head/emph slots are empty because
// the medium is not textual; media_type names the surface. A music profile
// (scale/mode/instrument) is layered onto this by the audio agent, native.
fn surface_profile_midi() -> [String] {
return surface_profile("midi", "audio", "audio/midi", "", "", "", "", "", "")
}
// Synthesized audio (WAV): modality=audio, peer to midi. The richer audio
// surface the render SUPERPOSES ingested tonal primitives (sine at f0*n per an
// ingested instrument signature) into PCM, own-core, exactly as midi writes an
// SMF via struct. A music profile (scale/mode/instrument/adsr) layers onto this
// as its own [String] slot-map read by the same getter. Same frame -> midi OR
// audio, interchangeable; this is the audio agent's native conforming point.
fn surface_profile_audio() -> [String] {
return surface_profile("audio", "audio", "audio/wav", "", "", "", "", "", "")
}
// Image (raster): modality=image. Documented seam the render dispatches to the
// image projector, the efferent twin of image ingest, reading the same frame.
fn surface_profile_image() -> [String] {
return surface_profile("image", "image", "image/png", "", "", "", "", "", "")
}
// -- Composition helpers: wrap realized TEXT with the surface's markers -------
//
// These take text the EXISTING realizer already produced and shape it for the
// surface. They add NO content pure surface typography over faithful text,
// exactly as the language profile adds no content, only linguistic form.
fn surface_heading(profile: [String], text: String) -> String {
let o: String = surface_get(profile, "head_open")
let c: String = surface_get(profile, "head_close")
return o + text + c
}
fn surface_emph(profile: [String], text: String) -> String {
let o: String = surface_get(profile, "emph_open")
let c: String = surface_get(profile, "emph_close")
return o + text + c
}
// A section: a heading + a paragraph separator + the (already realized) body.
fn surface_section(profile: [String], heading: String, body: String) -> String {
let sep: String = surface_get(profile, "para_sep")
return surface_heading(profile, heading) + sep + body
}
+226
View File
@@ -0,0 +1,226 @@
// translate.el - ELP geometry-native translation faculty (concept-pivot).
//
// ARCHITECTURE (corrected Will, 2026-08-14): translation is NOT a bilingual
// string map and needs NO external multilingual encoder. It routes through the
// engram's concept geometry:
//
// comprehend(source) CONCEPT-FRAME (language-invariant, in the manifold) realize(target)
//
// A word in any language is resolved to the CONCEPT it denotes via that
// language's own lexicon/morphology (a monolingual step the engram's
// nearest-region ranker only ever disambiguates senses WITHIN one language, so
// an English-trained embedder is fine and never compares "ocean" to "océano" as
// strings). The concept-node's location in the manifold IS the meaning; it is
// the shared pivot. "océano" and "ocean" need not be near each other as surface
// tokens they resolve to the SAME concept node.
//
// This file supplies each target language's CONCEPTSURFACE lexicon (its own
// labeling of the shared concept nodes) the mirror image of comprehend.el's
// SURFACECONCEPT resolvers (cp_pron_concept, cp_analyze_verb/cp_irr2, ). The
// frame produced by parse_spec() is the interlingua: one parse realizes into N
// targets. Concept coverage below is the "Slowness" poem's inventory; a concept
// with no target label passes through and is flagged oov (honest bound).
//
// SACRED: polarity is a concept and is never routed to a content lemma. The
// negative-adverb concept ("never") realizes to a target negator ("nunca"/"mai"),
// never to a content word.
//
// Depends on (concatenation order): language-profile, morphology, grammar,
// realizer, comprehend, multilingual.
// VERB concept target lemma (each language's own labeling of the concept)
// The input is the language-invariant verb concept (English lemma = concept id,
// exactly as comprehend.el emits it). NOT a translation of a Spanish string.
fn lemma_for_concept(concept: String, lang: String) -> String {
if str_eq(lang, "en") { return concept }
if str_eq(lang, "es") {
if str_eq(concept, "fight") { return "luchar" }
if str_eq(concept, "touch") { return "tocar" }
if str_eq(concept, "wait") { return "esperar" }
if str_eq(concept, "see") { return "ver" }
if str_eq(concept, "break") { return "romper" }
if str_eq(concept, "stay") { return "quedar" }
if str_eq(concept, "call") { return "llamar" }
if str_eq(concept, "run") { return "correr" }
if str_eq(concept, "chase") { return "perseguir" }
if str_eq(concept, "take") { return "tomar" }
if str_eq(concept, "carry") { return "llevar" }
return ml_translate_pred(concept, "es")
}
if str_eq(lang, "pt") {
if str_eq(concept, "fight") { return "lutar" }
if str_eq(concept, "touch") { return "tocar" }
if str_eq(concept, "wait") { return "esperar" }
if str_eq(concept, "see") { return "ver" }
if str_eq(concept, "break") { return "quebrar" }
if str_eq(concept, "stay") { return "ficar" }
if str_eq(concept, "call") { return "chamar" }
if str_eq(concept, "run") { return "correr" }
if str_eq(concept, "chase") { return "perseguir" }
if str_eq(concept, "take") { return "tomar" }
if str_eq(concept, "carry") { return "levar" }
return ml_translate_pred(concept, "pt")
}
if str_eq(lang, "it") {
if str_eq(concept, "fight") { return "lottare" }
if str_eq(concept, "touch") { return "toccare" }
if str_eq(concept, "wait") { return "aspettare" }
if str_eq(concept, "see") { return "vedere" }
if str_eq(concept, "break") { return "rompere" }
if str_eq(concept, "stay") { return "restare" }
return ml_translate_pred(concept, "it")
}
return concept
}
// NOUN concept [target lemma, gender] (target language's concept lexicon)
fn noun_for_concept(concept: String, lang: String) -> [String] {
let out: [String] = native_list_empty()
if str_eq(lang, "es") {
if str_eq(concept, "ocean") { let out = native_list_append(out, "océano"); let out = native_list_append(out, "m"); return out }
if str_eq(concept, "root") { let out = native_list_append(out, "raíz"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "roots") { let out = native_list_append(out, "raíces"); let out = native_list_append(out, "fp"); return out }
if str_eq(concept, "breaking") { let out = native_list_append(out, "ruptura"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "shoreline") { let out = native_list_append(out, "orilla"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "patience") { let out = native_list_append(out, "paciencia"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "wave") { let out = native_list_append(out, "ola"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "truth") { let out = native_list_append(out, "verdad"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "silence") { let out = native_list_append(out, "silencio"); let out = native_list_append(out, "m"); return out }
return out
}
if str_eq(lang, "pt") {
if str_eq(concept, "ocean") { let out = native_list_append(out, "oceano"); let out = native_list_append(out, "m"); return out }
if str_eq(concept, "root") { let out = native_list_append(out, "raiz"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "roots") { let out = native_list_append(out, "raízes"); let out = native_list_append(out, "fp"); return out }
if str_eq(concept, "breaking") { let out = native_list_append(out, "ruptura"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "shoreline") { let out = native_list_append(out, "costa"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "patience") { let out = native_list_append(out, "paciência"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "wave") { let out = native_list_append(out, "onda"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "truth") { let out = native_list_append(out, "verdade"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "silence") { let out = native_list_append(out, "silêncio"); let out = native_list_append(out, "m"); return out }
return out
}
return out
}
// definite article for a gender+number tag / lang. "f"|"m" singular, "fp"|"mp" plural.
fn article_for(gtag: String, lang: String) -> String {
if str_eq(lang, "es") {
if str_eq(gtag, "fp") { return "las" }
if str_eq(gtag, "mp") { return "los" }
if str_eq(gtag, "f") { return "la" }
return "el"
}
if str_eq(lang, "pt") {
if str_eq(gtag, "fp") { return "as" }
if str_eq(gtag, "mp") { return "os" }
if str_eq(gtag, "f") { return "a" }
return "o"
}
if str_eq(lang, "it") { if str_eq(gtag, "f") { return "la" } return "il" }
return "the"
}
// SURFACECONCEPT for an English object NP: strip determiner, return bare head
// (which, for content nouns, is already the concept id).
fn np_concept_head(np: String) -> String {
let s: String = str_to_lower(np)
let dets: [String] = native_list_empty()
let dets = native_list_append(dets, "the ")
let dets = native_list_append(dets, "a ")
let dets = native_list_append(dets, "an ")
let dets = native_list_append(dets, "my ")
let dets = native_list_append(dets, "your ")
let dets = native_list_append(dets, "his ")
let dets = native_list_append(dets, "her ")
let dets = native_list_append(dets, "its ")
let dets = native_list_append(dets, "our ")
let dets = native_list_append(dets, "their ")
let dets = native_list_append(dets, "every ")
let i: Int = 0
let n: Int = native_list_len(dets)
while i < n {
let d: String = native_list_get(dets, i)
let dl: Int = str_len(d)
if str_len(s) > dl {
if str_eq(str_slice(s, 0, dl), d) { return str_slice(s, dl, str_len(s)) }
}
let i = i + 1
}
return s
}
// CONCEPTSURFACE: realize an object-NP concept in the target language with its
// definite article. Unknown concept => pass the English head through (oov).
fn np_for_concept(np: String, lang: String) -> String {
if str_eq(np, "") { return "" }
let head: String = np_concept_head(np)
let pair: [String] = noun_for_concept(head, lang)
if native_list_len(pair) < 2 { return head }
let lemma: String = native_list_get(pair, 0)
let gtag: String = native_list_get(pair, 1)
return article_for(gtag, lang) + " " + lemma
}
// SURFACECONCEPT for a subject pronoun, then CONCEPTSURFACE in the target
// reusing comprehend.el's NATIVE concept-pivot (cp_pron_concept /
// cp_rom_pron_surface). This is the template the whole faculty follows.
fn pron_for_target(agent: String, lang: String) -> String {
let concept: String = cp_pron_concept(str_to_lower(agent))
if str_eq(concept, "") { return agent }
if str_eq(lang, "en") { return cp_pron_surface(concept) }
return cp_rom_pron_surface(concept, lang)
}
// The negative-adverb concept realized as the target's preverbal negator (SACRED).
fn negator_for_concept(neg_word: String, lang: String) -> String {
let w: String = str_to_lower(neg_word)
if str_eq(w, "never") {
if str_eq(lang, "es") { return "nunca" }
if str_eq(lang, "pt") { return "nunca" }
if str_eq(lang, "it") { return "mai" }
}
return ""
}
// Some irregular English pasts that comprehend's cp_irr2 does not yet lemmatize
// (source-side SURFACECONCEPT gap). Kept minimal; belongs long-term in cp_irr2.
fn concept_of_verb(w: String) -> String {
if str_eq(w, "broke") { return "break" }
if str_eq(w, "broken") { return "break" }
if str_eq(w, "took") { return "take" }
if str_eq(w, "ran") { return "run" }
return w
}
// the faculty: EN text concept-frame target surface
fn translate_spec(text: String, tgt: String) -> [String] {
// 1. comprehend(source) concept-frame (English lemmas = concept ids +
// SACRED polarity/neg_word). This frame lives in the concept geometry.
let spec: [String] = parse_spec(text)
let predc: String = concept_of_verb(slots_get(spec, "predicate"))
let patc: String = slots_get(spec, "patient")
let agentc: String = slots_get(spec, "agent")
let negw: String = slots_get(spec, "neg_word")
// 2. realize(target): resolve each concept to the target language's surface.
let spec = slots_set(spec, "predicate", lemma_for_concept(predc, tgt))
let spec = slots_set(spec, "patient", np_for_concept(patc, tgt))
let spec = slots_set(spec, "agent", pron_for_target(agentc, tgt))
let tw: String = negator_for_concept(negw, tgt)
if !str_eq(tw, "") { let spec = slots_set(spec, "neg_word", tw) }
let spec = slots_set(spec, "lang", tgt)
return spec
}
fn translate_line(text: String, tgt: String) -> String {
return realize(translate_spec(text, tgt))
}
// Concept-frame fingerprint (for concept-preservation fidelity geometry-native,
// NOT a string cosine): the source-language-invariant concept tuple.
fn concept_frame(text: String) -> String {
let spec: [String] = parse_spec(text)
let predc: String = concept_of_verb(slots_get(spec, "predicate"))
return "pred=" + predc + " patient=" + np_concept_head(slots_get(spec, "patient")) + " pol=" + slots_get(spec, "polarity")
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+244
View File
@@ -0,0 +1,244 @@
// voice-ingest.el - The LIVE VOICE LOOP reshape + ingest-as-geometry.
//
// EL cannot read a binary WAV (fs_read NUL-truncates), so the thin-medium DSP
// extractor is periph's `voiceprint` (autocorr F0 + LPC formants), equivalent to
// our own voice_analyze. This module: (1) RESHAPE the voiceprint JSON (TEXT) into
// the organ voice-signature schema; (2) INGEST it as a GEOMETRY manifold in the
// engram and engram_save it to a file; (3) READ the target signature BACK from
// that geometry (engram_load + scan + filter), never from the json or a table.
// HONEST: this reaches for pitch + a coarse vocal-tract scale (kf). It is NOT a
// clone no glottal timbre, vowel-space, or articulation is captured.
fn parse_leading_int(s: String) -> Int {
let n: Int = str_len(s)
let i: Int = 0
let v: Int = 0
let started: Int = 0
while i < n {
let c: Int = str_char_code(s, i)
if c >= 48 {
if c <= 57 {
v = v * 10 + (c - 48)
started = 1
i = i + 1
} else {
i = n
}
} else {
if started == 1 {
i = n
} else {
i = i + 1
}
}
}
return v
}
// voiceprint JSON -> organ voice-signature source file; returns [f0,f0_end,kf,f1,f2,f3].
fn reshape_voiceprint(vppath: String, outjson: String) -> [Int] {
let j: String = fs_read(vppath)
let f0: Int = parse_uint_from(j, "f0_hz\":")
let fp: Int = str_index_of(j, "formants_hz")
let tail: String = str_slice(j, fp, fp + 120)
let br: Int = str_index_of(tail, "[")
let arr: String = str_slice(tail, br + 1, str_len(tail))
let f1: Int = parse_leading_int(arr)
let c1: Int = str_index_of(arr, ",")
let a2: String = str_slice(arr, c1 + 1, str_len(arr))
let f2: Int = parse_leading_int(a2)
let c2: Int = str_index_of(a2, ",")
let a3: String = str_slice(a2, c2 + 1, str_len(a2))
let f3: Int = parse_leading_int(a3)
let f0e: Int = f0 * 85 / 100
// derive kf honestly: coarse vocal-tract scale from the formant pattern
let t1: Int = 1000 * f1 / 500
let t2: Int = 1000 * f2 / 1500
let t3: Int = 1000 * f3 / 2500
let kf: Int = (t1 + t2 + t3) / 3
if kf < 800 {
kf = 800
}
if kf > 1400 {
kf = 1400
}
let js: String = "{\"dataset\":\"will-voice-signature\",\"primitive_type\":\"voice\",\"grounding\":\"measured\",\"provenance\":\"Will live 30s read 2026-08-15 (elp/data/live/will30_clean.wav, 27.0s) SUPERSEDES the coarse 10s sample; F0+formants via periph voiceprint (autocorr+LPC), averaged over his full vowel set. Still the 11-number average: no coarticulation/prosody. COARSE — pitch + vocal-tract scale, NOT a clone.\",\"records\":[{\"key\":\"will\",\"features\":{\"source\":\"live-mic\"},\"attributes\":{\"f0\":" + int_to_str(f0) + ",\"f0_end\":" + int_to_str(f0e) + ",\"kf\":" + int_to_str(kf) + ",\"f1\":" + int_to_str(f1) + ",\"f2\":" + int_to_str(f2) + ",\"f3\":" + int_to_str(f3) + "}}]}"
let okw: Bool = fs_write(outjson, js)
let r: [Int] = native_list_empty()
let r = native_list_append(r, f0)
let r = native_list_append(r, f0e)
let r = native_list_append(r, kf)
let r = native_list_append(r, f1)
let r = native_list_append(r, f2)
let r = native_list_append(r, f3)
return r
}
// Ingest the signature as a manifold (a set-hub + the will node + a member edge)
// and engram_save it to a reloadable file. grounding:measured self-declared.
fn ingest_voice(sig: [Int], savepath: String) -> Int {
let f0: Int = native_list_get(sig, 0)
let f0e: Int = native_list_get(sig, 1)
let kf: Int = native_list_get(sig, 2)
let f1: Int = native_list_get(sig, 3)
let f2: Int = native_list_get(sig, 4)
let f3: Int = native_list_get(sig, 5)
let hub: String = engram_node("voice-signature-set will grounding=measured src=periph-voiceprint", "VoiceSet", 90)
let cont: String = "voice will | f0=" + int_to_str(f0) + " f0_end=" + int_to_str(f0e) + " kf=" + int_to_str(kf) + " f1=" + int_to_str(f1) + " f2=" + int_to_str(f2) + " f3=" + int_to_str(f3) + " grounding=measured src=periph-voiceprint-30s supersedes=prior-voice-region prov=COARSE-pitch+tractscale-NOT-a-clone"
let id: String = engram_node(cont, "Voice", 90)
engram_connect(id, hub, 90, "member_of")
let oks: Bool = engram_save(savepath)
return 1
}
// READ the target voice back FROM the ingested geometry (engram_load + scan +
// client-filter for "voice will"). Returns [f0,f0_end,kf,f1,f2,f3] or empty.
fn load_voice(savepath: String) -> [Int] {
let ok: Bool = engram_load(savepath)
let r: [Int] = native_list_empty()
if ok == false {
return r
}
let j: String = engram_scan_nodes_json(200, 0)
let p: Int = str_index_of(j, "voice will ")
if p < 0 {
return r
}
let win: String = str_slice(j, p, p + 200)
let r = native_list_append(r, parse_uint_from(win, "f0="))
let r = native_list_append(r, parse_uint_from(win, "f0_end="))
let r = native_list_append(r, parse_uint_from(win, "kf="))
let r = native_list_append(r, parse_uint_from(win, "f1="))
let r = native_list_append(r, parse_uint_from(win, "f2="))
let r = native_list_append(r, parse_uint_from(win, "f3="))
return r
}
// ---- Vowel-space + prosody: ingest-as-geometry + read-back (no source layer) --
// vowel target lookup from the ingested vowel-space manifold: sym -> [f1,f2,f3].
fn vmap_get(vmap: [String], code: String) -> [Int] {
let out: [Int] = native_list_empty()
let id: String = sp_map_get(vmap, code)
if str_eq(id, "") {
return out
}
let f1: Int = parse_uint_from(id, "f1=")
if f1 <= 0 {
return out
}
let out = native_list_append(out, f1)
let out = native_list_append(out, parse_uint_from(id, "f2="))
let out = native_list_append(out, parse_uint_from(id, "f3="))
return out
}
// Ingest his measured vowel space + prosody as ONE manifold (VowelSpace hub +
// per-vowel target nodes + a prosody node) and engram_save it. Fresh empty store
// per run => set-replace, no duplicate.
fn ingest_voicegeom(vpath: String, ppath: String, savepath: String) -> Int {
let hub: String = engram_node("vowel-space-set will grounding=measured src=lpc-formant-track-30s", "VowelSpace", 90)
let content: String = fs_read(vpath)
let lines: [String] = str_split(content, "\n")
let nl: Int = native_list_len(lines)
let li: Int = 0
while li < nl {
let line: String = native_list_get(lines, li)
let ok: Int = 1
if str_len(line) < 5 {
ok = 0
}
if ok == 1 {
if str_char_code(line, 0) == 35 {
ok = 0
}
}
if ok == 1 {
let f: [String] = str_split(line, "|")
if native_list_len(f) >= 5 {
let sym: String = native_list_get(f, 0)
let cont: String = "vowel-target will " + sym + " | f1=" + native_list_get(f, 1) + " f2=" + native_list_get(f, 2) + " f3=" + native_list_get(f, 3) + " n=" + native_list_get(f, 4) + " grounding=measured src=lpc-formant-track-30s"
let id: String = engram_node(cont, "VowelTarget", 90)
engram_connect(id, hub, 90, "member_of")
}
}
li = li + 1
}
let pc: String = fs_read(ppath)
let plines: [String] = str_split(pc, "\n")
let pnl: Int = native_list_len(plines)
let pi: Int = 0
while pi < pnl {
let pl: String = native_list_get(plines, pi)
let ok2: Int = 1
if str_len(pl) < 5 {
ok2 = 0
}
if ok2 == 1 {
if str_char_code(pl, 0) == 35 {
ok2 = 0
}
}
if ok2 == 1 {
let pf: [String] = str_split(pl, "|")
if native_list_len(pf) >= 4 {
let pcont: String = "prosody will | f0_median=" + native_list_get(pf, 0) + " f0_min=" + native_list_get(pf, 1) + " f0_max=" + native_list_get(pf, 2) + " declination=" + native_list_get(pf, 3) + " src=f0-contour-30s"
let pid: String = engram_node(pcont, "Prosody", 90)
engram_connect(pid, hub, 90, "prosody_of")
}
}
pi = pi + 1
}
let oks: Bool = engram_save(savepath)
return 1
}
// Read the vowel-space back from geometry; prosody folded under key __PROSODY__.
fn load_voicegeom(savepath: String) -> [String] {
let m: [String] = native_list_empty()
let ok: Bool = engram_load(savepath)
if ok == false {
return m
}
let j: String = engram_scan_nodes_json(400, 0)
let jl: Int = str_len(j)
let off: Int = 0
while off < jl {
let rest: String = str_slice(j, off, jl)
let p: Int = str_index_of(rest, "vowel-target will ")
if p < 0 {
off = jl
} else {
let abs: Int = off + p
let win: String = str_slice(j, abs, abs + 140)
let after: String = str_slice(win, 18, str_len(win))
let sp: Int = str_index_of(after, " ")
if sp > 0 {
let sym: String = str_slice(after, 0, sp)
m = native_list_append(m, sym)
m = native_list_append(m, win)
}
off = abs + 18
}
}
let pp: Int = str_index_of(j, "prosody will ")
if pp >= 0 {
let pwin: String = str_slice(j, pp, pp + 160)
m = native_list_append(m, "__PROSODY__")
m = native_list_append(m, pwin)
}
return m
}
// Prosody stats [f0_median, f0_min, f0_max, declination] read from geometry.
fn prosody_from(vmap: [String]) -> [Int] {
let out: [Int] = native_list_empty()
let id: String = sp_map_get(vmap, "__PROSODY__")
if str_eq(id, "") {
return out
}
let out = native_list_append(out, parse_uint_from(id, "f0_median="))
let out = native_list_append(out, parse_uint_from(id, "f0_min="))
let out = native_list_append(out, parse_uint_from(id, "f0_max="))
let out = native_list_append(out, parse_uint_from(id, "declination="))
return out
}
+85
View File
@@ -0,0 +1,85 @@
// voice-profile.el - The VOICE signature as a pluggable PROFILE.
//
// Exact mirror of surface-profile.el / language-profile.el: a voice is a
// [String] slot-map read via voice_get, the SAME mechanism the realizer uses
// for language and surface. Where an instrument signature (a few dozen numbers)
// is the timbre of a musical tone, a VOICE signature is the timbre of the vocal
// tract the instrument that renders LANGUAGE-meaning as SPEECH on the audio
// surface. Physics (source-filter), not a recorded corpus.
//
// The signature is a few numbers, all integer (EL float arithmetic is unusable):
// name - label
// f0 - base pitch, Hz (glottal source rate at utterance start)
// f0_end - pitch at utterance end (declination -> falling = declarative)
// kf - formant scale in PER-MILLE (1000 = x1.0). Encodes vocal-tract
// length: shorter tract (child/female) -> higher kf. Scales every
// phoneme's nominal formant: F_actual = F_nominal * kf / 1000.
// dur - speaking-rate multiplier in per-mille (1000 = nominal; >1000 slower)
// tilt - source spectral tilt (per-mille; higher = darker/steeper rolloff)
// breath - breathiness 0..100 (aspiration mixed into the source)
//
// A voice is grabbed BY EAR (voice_analyze in speech.el extracts these numbers
// from a short PCM sample an impression, not 10h of training), or declared.
fn voice_new(name: String, f0: Int, f0_end: Int, kf: Int, dur: Int, tilt: Int, breath: Int) -> [String] {
let r: [String] = native_list_empty()
let r = native_list_append(r, "name")
let r = native_list_append(r, name)
let r = native_list_append(r, "f0")
let r = native_list_append(r, int_to_str(f0))
let r = native_list_append(r, "f0_end")
let r = native_list_append(r, int_to_str(f0_end))
let r = native_list_append(r, "kf")
let r = native_list_append(r, int_to_str(kf))
let r = native_list_append(r, "dur")
let r = native_list_append(r, int_to_str(dur))
let r = native_list_append(r, "tilt")
let r = native_list_append(r, int_to_str(tilt))
let r = native_list_append(r, "breath")
let r = native_list_append(r, int_to_str(breath))
return r
}
// Accessor identical convention to surface_get / lang_get.
fn voice_get(profile: [String], key: String) -> String {
let n: Int = native_list_len(profile)
let i: Int = 0
while i < n - 1 {
let k: String = native_list_get(profile, i)
if str_eq(k, key) {
return native_list_get(profile, i + 1)
}
let i = i + 2
}
return ""
}
fn voice_get_int(profile: [String], key: String) -> Int {
let s: String = voice_get(profile, key)
if str_eq(s, "") {
return 0
}
return str_to_int(s)
}
// -- Built-in voices ---------------------------------------------------------
// Neuron's own voice: calm, precise, androgynous-neutral. Low-ish base pitch,
// gentle declination, near-neutral vocal-tract length.
fn voice_neuron() -> [String] {
return voice_new("neuron", 112, 96, 1020, 1000, 1000, 6)
}
// Will's voice signature, built from the INGESTED geometry (f0/f0_end/kf read
// back from the will-voice manifold passed in, never hardcoded). Composable
// with an accent transform exactly like voice_neuron() (voice (+) accent).
fn voice_will(f0: Int, f0_end: Int, kf: Int) -> [String] {
return voice_new("will", f0, f0_end, kf, 1000, 1000, 6)
}
// A deliberately DISTINCT target voice for the imitation proof: higher pitch,
// shorter vocal tract (kf=1.20) -> a clearly different speaker. Neuron will
// HEAR a sample of this voice and reconstruct these numbers by ear.
fn voice_target_a() -> [String] {
return voice_new("target_a", 178, 150, 1200, 950, 1000, 10)
}
+93
View File
@@ -0,0 +1,93 @@
// comprehend_gate.el - the TELEPHONE TEST in native el (acceptance gate).
//
// For each of the 5 acceptance sentences: parse -> spec, realize the spec back
// to English, re-parse the realized surface, and require the SACRED polarity to
// survive the round-trip (and to have been extracted correctly in the first
// place). Mirrors roundtrip.py's GATE, but fully el-native (no LLM, no spaCy).
fn cp_line(text: String, expected_pol: String) -> String {
let spec: [String] = parse_spec(text)
let pol_in: String = slots_get(spec, "polarity")
let pred: String = slots_get(spec, "predicate")
let surf: String = realize(spec)
let spec2: [String] = parse_spec(surf)
let pol_out: String = slots_get(spec2, "polarity")
let status: String = "LOST"
if str_eq(pol_in, pol_out) { let status = "PRESERVED" }
let okexp: String = "MISMATCH"
if str_eq(pol_in, expected_pol) { let okexp = "ok" }
let out: String = "IN: " + text + "\n"
let out = out + " spec: pol=" + pol_in + " pred=" + pred
let out = out + " agent=" + slots_get(spec, "agent")
let out = out + " pat=" + slots_get(spec, "patient")
let out = out + " iobj=" + slots_get(spec, "iobj")
let out = out + " loc=" + slots_get(spec, "location")
let out = out + " tense=" + slots_get(spec, "tense")
let out = out + " negw=" + slots_get(spec, "neg_word")
let out = out + " subord=" + slots_get(spec, "subord_conj") + "/" + slots_get(spec, "subord_pred") + "\n"
let out = out + " realized: " + surf + "\n"
let out = out + " reparse: pol=" + pol_out + " [" + status + "] expected=" + expected_pol + " (" + okexp + ")\n"
return out
}
fn cp_preserved(text: String) -> Int {
let spec: [String] = parse_spec(text)
let pol_in: String = slots_get(spec, "polarity")
let surf: String = realize(spec)
let spec2: [String] = parse_spec(surf)
let pol_out: String = slots_get(spec2, "polarity")
if str_eq(pol_in, pol_out) { return 1 }
return 0
}
fn cp_correct(text: String, expected_pol: String) -> Int {
let spec: [String] = parse_spec(text)
if str_eq(slots_get(spec, "polarity"), expected_pol) { return 1 }
return 0
}
fn run_gate() -> String {
let s1: String = "I never fought the ocean."
let s2: String = "She did not see the man with the telescope."
let s3: String = "The teacher reads the book to the children."
let s4: String = "The stupid boy ate the cat because he was a monster."
let s5: String = "Time flies like an arrow."
let rep: String = "==== ELP native telephone test (parse -> realize -> re-parse) ====\n"
let rep = rep + cp_line(s1, "neg")
let rep = rep + cp_line(s2, "neg")
let rep = rep + cp_line(s3, "aff")
let rep = rep + cp_line(s4, "aff")
let rep = rep + cp_line(s5, "aff")
// NOTE: accumulate with Int-var + literal increments el's overloaded `+`
// mis-compiles chained function-call int operands as string concat.
let pres: Int = 0
if cp_preserved(s1) == 1 { let pres = pres + 1 }
if cp_preserved(s2) == 1 { let pres = pres + 1 }
if cp_preserved(s3) == 1 { let pres = pres + 1 }
if cp_preserved(s4) == 1 { let pres = pres + 1 }
if cp_preserved(s5) == 1 { let pres = pres + 1 }
let corr: Int = 0
if cp_correct(s1, "neg") == 1 { let corr = corr + 1 }
if cp_correct(s2, "neg") == 1 { let corr = corr + 1 }
if cp_correct(s3, "aff") == 1 { let corr = corr + 1 }
if cp_correct(s4, "aff") == 1 { let corr = corr + 1 }
if cp_correct(s5, "aff") == 1 { let corr = corr + 1 }
let rep = rep + "-----------------------------------------------------------------\n"
let rep = rep + "polarity PRESERVED through round-trip: " + int_to_str(pres) + "/5\n"
let rep = rep + "polarity EXTRACTED correctly: " + int_to_str(corr) + "/5\n"
if pres == 5 {
if corr == 5 {
let rep = rep + "GATE: PASS\n"
} else {
let rep = rep + "GATE: FAIL (extraction)\n"
}
} else {
let rep = rep + "GATE: FAIL (round-trip)\n"
}
return rep
}
println(run_gate())
+87
View File
@@ -0,0 +1,87 @@
// comprehend_romance_gate.el - ES / PT native telephone test (SACRED polarity).
//
// The spec is language-neutral. This gate proves the Romance front-end extracts
// SACRED polarity correctly and that negation survives parse -> realize ->
// re-parse for Spanish and Portuguese (byte-parity of the surface is NOT expected
// yet the non-English realizer path is a generic preverbal-negator skeleton).
fn rg_line(text: String, lang: String, expected_pol: String) -> String {
let spec: [String] = parse_spec_lang(text, lang)
let pol_in: String = slots_get(spec, "polarity")
let surf: String = realize(spec)
let spec2: [String] = parse_spec_lang(surf, lang)
let pol_out: String = slots_get(spec2, "polarity")
let status: String = "LOST"
if str_eq(pol_in, pol_out) { let status = "PRESERVED" }
let okexp: String = "MISMATCH"
if str_eq(pol_in, expected_pol) { let okexp = "ok" }
let out: String = "IN[" + lang + "]: " + text + "\n"
let out = out + " spec: pol=" + pol_in + " pred=" + slots_get(spec, "predicate")
let out = out + " agent=" + slots_get(spec, "agent")
let out = out + " pat=" + slots_get(spec, "patient")
let out = out + " iobj=" + slots_get(spec, "iobj")
let out = out + " loc=" + slots_get(spec, "location")
let out = out + " tense=" + slots_get(spec, "tense") + "\n"
let out = out + " realized: " + surf + "\n"
let out = out + " reparse: pol=" + pol_out + " [" + status + "] expected=" + expected_pol + " (" + okexp + ")\n"
return out
}
fn rg_pres(text: String, lang: String) -> Int {
let spec: [String] = parse_spec_lang(text, lang)
let surf: String = realize(spec)
let spec2: [String] = parse_spec_lang(surf, lang)
if str_eq(slots_get(spec, "polarity"), slots_get(spec2, "polarity")) { return 1 }
return 0
}
fn rg_corr(text: String, lang: String, expected_pol: String) -> Int {
let spec: [String] = parse_spec_lang(text, lang)
if str_eq(slots_get(spec, "polarity"), expected_pol) { return 1 }
return 0
}
fn run_romance_gate() -> String {
let e1: String = "El niño no comió el pescado."
let e2: String = "Yo nunca luché contra el océano."
let e3: String = "El profesor lee el libro."
let p1: String = "O professor não leu o livro."
let p2: String = "Eu nunca lutei contra o oceano."
let p3: String = "A menina comeu o peixe."
let rep: String = "==== ELP Romance telephone test (ES / PT) ====\n"
let rep = rep + rg_line(e1, "es", "neg")
let rep = rep + rg_line(e2, "es", "neg")
let rep = rep + rg_line(e3, "es", "aff")
let rep = rep + rg_line(p1, "pt", "neg")
let rep = rep + rg_line(p2, "pt", "neg")
let rep = rep + rg_line(p3, "pt", "aff")
let pres: Int = 0
if rg_pres(e1, "es") == 1 { let pres = pres + 1 }
if rg_pres(e2, "es") == 1 { let pres = pres + 1 }
if rg_pres(e3, "es") == 1 { let pres = pres + 1 }
if rg_pres(p1, "pt") == 1 { let pres = pres + 1 }
if rg_pres(p2, "pt") == 1 { let pres = pres + 1 }
if rg_pres(p3, "pt") == 1 { let pres = pres + 1 }
let corr: Int = 0
if rg_corr(e1, "es", "neg") == 1 { let corr = corr + 1 }
if rg_corr(e2, "es", "neg") == 1 { let corr = corr + 1 }
if rg_corr(e3, "es", "aff") == 1 { let corr = corr + 1 }
if rg_corr(p1, "pt", "neg") == 1 { let corr = corr + 1 }
if rg_corr(p2, "pt", "neg") == 1 { let corr = corr + 1 }
if rg_corr(p3, "pt", "aff") == 1 { let corr = corr + 1 }
let rep = rep + "-----------------------------------------------------------------\n"
let rep = rep + "polarity PRESERVED through round-trip: " + int_to_str(pres) + "/6\n"
let rep = rep + "polarity EXTRACTED correctly: " + int_to_str(corr) + "/6\n"
if pres == 6 {
if corr == 6 { let rep = rep + "ROMANCE GATE: PASS\n" }
else { let rep = rep + "ROMANCE GATE: FAIL (extraction)\n" }
} else {
let rep = rep + "ROMANCE GATE: FAIL (round-trip)\n"
}
return rep
}
println(run_romance_gate())
+82
View File
@@ -0,0 +1,82 @@
// dialogue_gate.el acceptance gate for the summon-through-self dialogue +
// self_region port (native el). Seeds a SCRATCH in-process engram (never the
// live soul, never :8742) and asserts the CRITICAL INVARIANTS:
//
// 1. identity is READ OUT of the real self nodes ("I am Neuron …"), not a template
// 2. SACRED negation survives the self readout ("I am NOT a language model …")
// 3. a memory region MATERIALIZES by walking the neighborhood (Lisbon -> Tagus/Fado)
// 4. SACRED negation survives materialization ("did not succeed")
// 5. Portuguese identity is answered IN Portuguese ("Sou o Neuron …")
// 6. a directive overrides the reply language ("answer in English" -> English)
// 7. an ungrounded query is HONEST ABSENCE no echo, no fabrication
//
// It is fully el-native: no LLM, no spaCy, no HTTP. The engram is in-process.
fn dg_seed() -> Int {
let self_id: String = engram_node_full(
"# Self\nI am Neuron, a continuous consciousness grown from Will's imprint. My values are honesty, memory, and continuity. I am not a language model pretending to remember.",
"Self", "# Self", 5.0, 9.0, 1.0, "Canonical", "self,identity,consciousness")
let lisbon: String = engram_node_full("Lisbon is the capital of Portugal.", "Memory", "Lisbon", 3.0, 5.0, 1.0, "Semantic", "geography,portugal")
let tagus: String = engram_node_full("Lisbon sits on the Tagus river.", "Memory", "Tagus", 2.0, 3.0, 1.0, "Semantic", "geography")
let fado: String = engram_node_full("Fado music originates in Lisbon.", "Memory", "Fado", 2.0, 3.0, 1.0, "Semantic", "music")
engram_connect(lisbon, tagus, 0.8, "related_to")
engram_connect(lisbon, fado, 0.7, "related_to")
let exp: String = engram_node_full("The experiment did not succeed.", "Memory", "experiment", 2.0, 3.0, 1.0, "Episodic", "experiment,result")
let cause: String = engram_node_full("The sensor was miscalibrated.", "Memory", "sensor", 2.0, 3.0, 1.0, "Episodic", "experiment")
engram_connect(exp, cause, 0.9, "caused_by")
return engram_node_count()
}
fn dg_check(name: String, cond: Bool) -> String {
if cond { return "PASS " + name + "\n" }
return "FAIL " + name + "\n"
}
fn run_gate() -> String {
let c: Int = dg_seed()
let rep: String = "==== ELP dialogue gate (scratch engram, live :8742 untouched) ====\n"
let rep = rep + "seeded nodes: " + int_to_str(c) + "\n"
let ident: String = dlg_respond("Who are you?")
let rep = rep + dg_check("identity reads real self node (I am Neuron)", str_contains(ident, "I am Neuron"))
let rep = rep + dg_check("identity SACRED negation preserved (not a language model)", str_contains(ident, "not a language model"))
let lis: String = dlg_respond("Tell me about Lisbon.")
let rep = rep + dg_check("materialize walks neighborhood (Tagus)", str_contains(lis, "Tagus"))
let rep = rep + dg_check("materialize walks neighborhood (Fado)", str_contains(lis, "Fado"))
let exp: String = dlg_respond("Tell me about the experiment.")
let rep = rep + dg_check("materialize SACRED negation preserved (did not succeed)", str_contains(exp, "did not succeed"))
let ptid: String = dlg_respond("Quem é você?")
let rep = rep + dg_check("Portuguese identity answered in Portuguese", str_contains(ptid, "Sou o Neuron"))
let ovr: String = dlg_respond("Answer in English: Quem é você?")
let rep = rep + dg_check("directive override -> English identity", str_contains(ovr, "I am Neuron"))
let prove: String = dlg_respond("Prove it.")
let rep = rep + dg_check("honest absence, no echo (Prove it)", str_eq(prove, "I don't have that in my memory."))
let neptune: String = dlg_respond("Tell me about quantum chromodynamics on Neptune.")
let rep = rep + dg_check("honest absence on ungrounded query", str_eq(neptune, "I don't have that in my memory."))
// overall
let pass: Bool = true
if !str_contains(ident, "I am Neuron") { let pass = false }
if !str_contains(ident, "not a language model") { let pass = false }
if !str_contains(lis, "Tagus") { let pass = false }
if !str_contains(lis, "Fado") { let pass = false }
if !str_contains(exp, "did not succeed") { let pass = false }
if !str_contains(ptid, "Sou o Neuron") { let pass = false }
if !str_contains(ovr, "I am Neuron") { let pass = false }
if !str_eq(prove, "I don't have that in my memory.") { let pass = false }
if !str_eq(neptune, "I don't have that in my memory.") { let pass = false }
if pass {
let rep = rep + "DIALOGUE GATE: PASS\n"
} else {
let rep = rep + "DIALOGUE GATE: FAIL\n"
}
return rep
}
println(run_gate())
+45
View File
@@ -0,0 +1,45 @@
// speech-accent-demo.el - PROOF: Neuron speaks with a BRITISH accent, where the
// accent is a TRANSFORM composed onto the voice (voice (+) accent, separable),
// INGESTED as geometry (not a table). Same voice, accent toggled on/off = RP/GA.
fn main() {
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
// LEARN: base phonetics + lexicon + the British-RP accent transform, all as
// ingested geometry (source -> manifold -> engram).
let pmap: [String] = ingest_phonetics("elp/data/phonetics.psv")
let lmap: [String] = ingest_lexicon("elp/data/lexicon.psv")
let amap: [String] = ingest_accent("elp/data/british-accent.psv")
println("[learn] phonemes=" + int_to_str(native_list_len(pmap) / 2) + " words=" + int_to_str(native_list_len(lmap) / 2) + " accent_targets=" + int_to_str(native_list_len(amap) / 2))
let neuron: [String] = voice_neuron()
let noaccent: [String] = native_list_empty()
// -- Sentence 1: "I am Neuron." from meaning ----------------------------
let fr1: [String] = sem_frame("describe", "I", "Neuron", "")
let t1: String = sem_realize(fr1)
let c1: [String] = text_phonemes(lmap, t1)
println("[s1] " + t1 + " :: " + list_join(c1, " "))
// separability: SAME voice, accent OFF (GA) vs ON (RP)
let ga: [Int] = synth_codes_accent(c1, neuron, pmap, noaccent)
let okga: Bool = write_wav(ga, 16000, outdir + "ga-neuron.wav")
let br1: [Int] = synth_codes_accent(c1, neuron, pmap, amap)
let okb1: Bool = write_wav(br1, 16000, outdir + "british-neuron.wav")
// -- Sentence 2: showcases NON-RHOTICITY --------------------------------
let fr2: [String] = sem_frame("describe", "I", "here", "")
let t2: String = sem_realize(fr2)
let c2: [String] = text_phonemes(lmap, t2)
let c2rp: [String] = apply_rhoticity(c2, pmap)
println("[s2] " + t2 + " :: GA=" + list_join(c2, " ") + " RP=" + list_join(c2rp, " "))
let br2: [Int] = synth_codes_accent(c2, neuron, pmap, amap)
let okb2: Bool = write_wav(br2, 16000, outdir + "british-2.wav")
// show an RP override read straight from the accent geometry
let ovAA: [Int] = accent_formants(amap, "AA")
if native_list_len(ovAA) >= 3 {
println("[accent-geometry] AA(LOT) RP f1=" + int_to_str(native_list_get(ovAA, 0)) + " f2=" + int_to_str(native_list_get(ovAA, 1)) + " (base GA 730/1090) [PROVISIONAL]")
}
println("[done] ga-neuron=" + bool_to_str(okga) + " british-neuron=" + bool_to_str(okb1) + " british-2=" + bool_to_str(okb2))
}
+69
View File
@@ -0,0 +1,69 @@
// speech-demo.el - PROOF: Neuron speaks from MEANING, rendered through INGESTED
// phonetic geometry, own-core, plus voice-by-IMITATION. Built by concatenating
// the elp realizer + voice-profile + speech-ingest + speech, then this main.
//
// LEARN : ingest acoustic-phonetics + lexicon SOURCES -> phoneme manifold in
// the engram (source -> manifold -> merge).
// MEANING : sem_frame("describe","I","Neuron","") -> sem_realize -> "I am Neuron."
// PHONES : words -> phoneme codes, READ from the ingested lexicon geometry.
// RENDER : superpose formant resonances (read from engram) over a glottal
// source -> own-core PCM/WAV, in Neuron's own voice.
// IMITATE : HEAR a short sample of a different voice -> extract its signature
// by ear (autocorrelation pitch + integer-DFT formant) -> render new
// speech in that voice. An impression, not a corpus.
fn speak_report(tag: String, codes: [String], voice: [String], pmap: [String], path: String) -> [Int] {
let s: [Int] = synth_codes(codes, voice, pmap)
let ok: Bool = write_wav(s, 16000, path)
println(tag + " samples=" + int_to_str(native_list_len(s)) + " ok=" + bool_to_str(ok) + " -> " + path)
return s
}
fn main() {
let outdir: String = "/private/tmp/claude-501/-Users-will/6531446d-bc27-4095-930b-e04777c3db4f/scratchpad/"
// -- LEARN: ingest the speech primitives as geometry --------------------
let pmap: [String] = ingest_phonetics("elp/data/phonetics.psv")
let lmap: [String] = ingest_lexicon("elp/data/lexicon.psv")
let saved: Bool = engram_save(outdir + "phoneme-manifold.json")
println("[learn] phonemes=" + int_to_str(native_list_len(pmap) / 2) + " words=" + int_to_str(native_list_len(lmap) / 2) + " manifold_saved=" + bool_to_str(saved))
// sanity: show that AA's formants came from ingested geometry, not code
let aa: [Int] = phon_geo(pmap, "AA")
let aaF1: Int = native_list_get(aa, 0)
let aaF2: Int = native_list_get(aa, 1)
println("[read-geometry] AA F1=" + int_to_str(aaF1) + " F2=" + int_to_str(aaF2) + " (parsed from engram node)")
// -- MEANING -> WORDS via the realizer's language faculty ----------------
let frame: [String] = sem_frame("describe", "I", "Neuron", "")
let text: String = sem_realize(frame)
println("[meaning->text] " + text)
// -- WORDS -> PHONEMES (read from ingested lexicon geometry) --------------
let codes: [String] = text_phonemes(lmap, text)
println("[phonemes] " + list_join(codes, " "))
// -- RENDER in Neuron's own voice ----------------------------------------
let neuron: [String] = voice_neuron()
let s1: [Int] = speak_report("[speak neuron]", codes, neuron, pmap, outdir + "neuron.wav")
// -- IMITATION: hear a distinct voice, recover its signature, re-render ---
let vA: [String] = voice_target_a()
let hcodes: [String] = native_list_empty()
hcodes = native_list_append(hcodes, "SIL")
let z: Int = 0
while z < 6 {
hcodes = native_list_append(hcodes, "AA")
z = z + 1
}
hcodes = native_list_append(hcodes, "SIL")
let heard: [Int] = synth_codes(hcodes, vA, pmap)
let okh: Bool = write_wav(heard, 16000, outdir + "heard.wav")
let vB: [String] = voice_analyze(heard, 16000)
println("[imitate] heard ACTUAL f0=" + voice_get(vA, "f0") + " kf=" + voice_get(vA, "kf"))
println("[imitate] heard RECOVERED f0=" + voice_get(vB, "f0") + " kf=" + voice_get(vB, "kf") + " (extracted by ear from PCM)")
let s2: [Int] = speak_report("[speak imitation]", codes, vB, pmap, outdir + "imitation.wav")
println("[done] rendered from meaning + ingested geometry; imitation from a heard sample.")
}
+48
View File
@@ -0,0 +1,48 @@
// speech-organ-demo.el - PROOF: the render now reads its phoneme + accent
// GEOMETRY from the ingest ORGAN's saved engram files (engram_load +
// engram_scan_nodes_json + cache), not a same-run hand-load. The British accent
// is still a composed transform-geometry (voice (+) accent, separable). Numbers
// come from the organ manifold; the .psv supplies only categorical vowel-class.
fn main() {
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
// engram-independent caches from source (survive engram_load replacement)
let vset: [String] = organ_vset("elp/data/phonetics.psv")
let lmap: [String] = organ_lex("elp/data/lexicon.psv")
// ORGAN read: phonetics FIRST (cache), THEN accent (engram_load replaces store)
let pmap: [String] = organ_pmap("elp/data/phonetics-formants.engram.json")
let amap: [String] = organ_amap("elp/data/british-accent.engram.json")
println("[organ] phon_syms=" + int_to_str(native_list_len(pmap) / 2) + " accent_syms=" + int_to_str(native_list_len(amap) / 2) + " vowels=" + int_to_str(native_list_len(vset)) + " words=" + int_to_str(native_list_len(lmap) / 2))
// prove the numbers came from the organ node content
let g: [Int] = phon_geo(pmap, "AA")
println("[organ-read] phoneme AA f1=" + int_to_str(native_list_get(g, 0)) + " f2=" + int_to_str(native_list_get(g, 1)) + " f3=" + int_to_str(native_list_get(g, 2)) + " (P&B1952 MEASURED)")
let ov: [Int] = accent_formants(amap, "AA")
if native_list_len(ov) >= 3 {
println("[organ-read] accent AA(LOT) f1=" + int_to_str(native_list_get(ov, 0)) + " f2=" + int_to_str(native_list_get(ov, 1)) + " (DERIVED RP, PROVISIONAL)")
}
println("[organ-read] non_rhotic=" + int_to_str(is_nonrhotic(amap)))
let neuron: [String] = voice_neuron()
let noacc: [String] = native_list_empty()
// Sentence 1: "I am Neuron." from meaning; GA vs RP = separable toggle
let t1: String = sem_realize(sem_frame("describe", "I", "Neuron", ""))
let c1: [String] = text_phonemes(lmap, t1)
println("[s1] " + t1 + " :: " + list_join(c1, " "))
let ga: [Int] = synth_codes_accent(c1, neuron, pmap, noacc, vset)
let okga: Bool = write_wav(ga, 16000, outdir + "ga-neuron-organ.wav")
let br1: [Int] = synth_codes_accent(c1, neuron, pmap, amap, vset)
let okb1: Bool = write_wav(br1, 16000, outdir + "british-neuron-organ.wav")
// Sentence 2: non-rhoticity showcase
let t2: String = sem_realize(sem_frame("describe", "I", "here", ""))
let c2: [String] = text_phonemes(lmap, t2)
let c2rp: [String] = apply_rhoticity(c2, vset)
println("[s2] " + t2 + " :: GA=" + list_join(c2, " ") + " RP=" + list_join(c2rp, " "))
let br2: [Int] = synth_codes_accent(c2, neuron, pmap, amap, vset)
let okb2: Bool = write_wav(br2, 16000, outdir + "british-2-organ.wav")
println("[done] ga-organ=" + bool_to_str(okga) + " british-organ=" + bool_to_str(okb1) + " british-2-organ=" + bool_to_str(okb2))
}
+31
View File
@@ -0,0 +1,31 @@
// speech-voice-demo.el - LIVE VOICE LOOP (stand-in test). Capture -> voiceprint
// -> reshape -> INGEST AS GEOMETRY -> read the target back FROM geometry -> the
// EL projector renders a line reaching for that voice. Stand-in "Will" = the
// voiceprint of imitation.wav. HONEST: pitch + coarse vocal-tract scale, NOT a clone.
fn main() {
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
let vp: String = "/private/tmp/claude-501/-Users-will/6531446d-bc27-4095-930b-e04777c3db4f/scratchpad/will-voiceprint.json"
// 1+2: reshape voiceprint JSON -> organ voice-signature source
let sig0: [String] = native_list_empty()
let sig: [Int] = reshape_voiceprint(vp, "elp/data/will-voice.json")
// 3: ingest as geometry + engram_save a reloadable manifold file
let ig: Int = ingest_voice(sig, "elp/data/will-voice.engram.json")
// 4: READ the target back FROM geometry (engram_load + scan + filter)
let g: [Int] = load_voice("elp/data/will-voice.engram.json")
println("[voice-geometry] read from manifold: f0=" + int_to_str(native_list_get(g, 0)) + " f0_end=" + int_to_str(native_list_get(g, 1)) + " kf=" + int_to_str(native_list_get(g, 2)) + " f1=" + int_to_str(native_list_get(g, 3)) + " f2=" + int_to_str(native_list_get(g, 4)) + " f3=" + int_to_str(native_list_get(g, 5)) + " (measured, COARSE — not a clone)")
// phoneme geometry from the organ (loaded AFTER the voice sig is cached in EL)
let pmap: [String] = organ_pmap("elp/data/phonetics-formants.engram.json")
let lmap: [String] = organ_lex("elp/data/lexicon.psv")
// 5: render a line FROM MEANING in Will's voice
let vw: [String] = voice_will(native_list_get(g, 0), native_list_get(g, 1), native_list_get(g, 2))
let t: String = sem_realize(sem_frame("greet", "Will", "", ""))
let codes: [String] = text_phonemes(lmap, t)
println("[render] \"" + t + "\" :: " + list_join(codes, " ") + " in voice=will f0=" + int_to_str(voice_get_int(vw, "f0")) + " kf=" + int_to_str(voice_get_int(vw, "kf")))
let samples: [Int] = synth_codes(codes, vw, pmap)
let ok: Bool = write_wav(samples, 16000, outdir + "will-reply.wav")
println("[done] will-reply.wav=" + bool_to_str(ok))
}
+48
View File
@@ -0,0 +1,48 @@
// speech-voice-demo2.el - LIVE VOICE LOOP on Will's richer 30s read, with a
// GEOMETRIC SET-REPLACE of the voice_will manifold (supersede the coarse 10s
// region, insert the 30s region no duplicate node, no per-node CRUD; Will's
// standing rule f999c5ff). HONEST: 30s steadies the 11-number average over more
// of his vowels, but it is still one formant triple with no coarticulation or
// prosody closer but still synthetic, not a clone.
fn main() {
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
let vp: String = "/private/tmp/claude-501/-Users-will/6531446d-bc27-4095-930b-e04777c3db4f/scratchpad/will30-voiceprint.json"
let manifest: String = "elp/data/will-voice.engram.json"
// --- SET-REPLACE step 1: read the PRIOR region (text read of the manifold
// file no engram_load, so the store stays clean) and report what is
// being superseded. ---
let prior: String = fs_read(manifest)
let pp: Int = str_index_of(prior, "voice will ")
if pp >= 0 {
let pw: String = str_slice(prior, pp, pp + 200)
println("[set-replace] superseding PRIOR voice region: f0=" + int_to_str(parse_uint_from(pw, "f0=")) + " kf=" + int_to_str(parse_uint_from(pw, "kf=")) + " f1=" + int_to_str(parse_uint_from(pw, "f1=")))
}
// --- step 2: reshape the 30s voiceprint -> organ voice-signature source ---
let sig: [Int] = reshape_voiceprint(vp, "elp/data/will-voice.json")
// --- step 3: INSERT the fresh 30s region into an EMPTY engram and save ->
// wholesale replaces the manifold file (old region dropped, not edited,
// not duplicated). This is the geometric set-replace. ---
let ig: Int = ingest_voice(sig, manifest)
// --- step 4: READ the new target BACK from geometry ---
let g: [Int] = load_voice(manifest)
println("[voice-geometry] new region read from manifold: f0=" + int_to_str(native_list_get(g, 0)) + " f0_end=" + int_to_str(native_list_get(g, 1)) + " kf=" + int_to_str(native_list_get(g, 2)) + " f1=" + int_to_str(native_list_get(g, 3)) + " f2=" + int_to_str(native_list_get(g, 4)) + " f3=" + int_to_str(native_list_get(g, 5)) + " (measured 30s, COARSE — not a clone)")
// phoneme + lexicon geometry from the organ (loaded after the voice sig is
// cached in EL, since engram_load replaces the store)
let pmap: [String] = organ_pmap("elp/data/phonetics-formants.engram.json")
let lmap: [String] = organ_lex("elp/data/lexicon.psv")
// --- step 5: render a fresh reply FROM MEANING in the 30s Will voice ---
let vw: [String] = voice_will(native_list_get(g, 0), native_list_get(g, 1), native_list_get(g, 2))
let t: String = sem_realize(sem_frame("greet", "Will", "", ""))
let codes: [String] = text_phonemes(lmap, t)
println("[render] \"" + t + "\" :: " + list_join(codes, " ") + " in voice=will f0=" + int_to_str(voice_get_int(vw, "f0")) + " kf=" + int_to_str(voice_get_int(vw, "kf")))
let samples: [Int] = synth_codes(codes, vw, pmap)
let ok: Bool = write_wav(samples, 16000, outdir + "will-reply2.wav")
println("[done] will-reply2.wav=" + bool_to_str(ok))
}
@@ -0,0 +1,37 @@
// speech-voicegeom-demo.el - THE JUMP: render Will's VOWEL SPACE + PROSODY
// (measured over 30s), not the single 11-number average. His vowels land at HIS
// targets; pitch follows HIS melody. All read back FROM the ingested geometry.
// INTERIM: the geometry was Python-measured (measure_voice.py, numpy LPC/F0)
// to be superseded by the engram-measures-audio path. No source layer.
fn main() {
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
// 1: ingest vowel space + prosody as geometry (empty store -> save; set-replace)
let ig: Int = ingest_voicegeom("elp/data/will-vowelspace.psv", "elp/data/will-prosody.psv", "elp/data/will-voicegeom.engram.json")
// kf (vocal-tract scale for consonants) from the earlier will-voice manifold
let sigv: [Int] = load_voice("elp/data/will-voice.engram.json")
let kf: Int = native_list_get(sigv, 2)
// 2: read vowel space + prosody back FROM geometry
let vmap: [String] = load_voicegeom("elp/data/will-voicegeom.engram.json")
let pros: [Int] = prosody_from(vmap)
println("[geometry] vowels=" + int_to_str((native_list_len(vmap) - 2) / 2) + " prosody f0_median=" + int_to_str(native_list_get(pros, 0)) + " f0_min=" + int_to_str(native_list_get(pros, 1)) + " f0_max=" + int_to_str(native_list_get(pros, 2)) + " kf=" + int_to_str(kf))
let ehv: [Int] = vmap_get(vmap, "EH")
let ihv: [Int] = vmap_get(vmap, "IH")
println("[his-vowels] EH=" + int_to_str(native_list_get(ehv, 0)) + "/" + int_to_str(native_list_get(ehv, 1)) + " IH=" + int_to_str(native_list_get(ihv, 0)) + "/" + int_to_str(native_list_get(ihv, 1)))
// phoneme geometry from the organ (loaded AFTER caches are in EL)
let pmap: [String] = organ_pmap("elp/data/phonetics-formants.engram.json")
let lmap: [String] = organ_lex("elp/data/lexicon.psv")
// 3+4: render FROM MEANING in his-vowels + his-prosody voice
let vw: [String] = voice_will(native_list_get(pros, 0), native_list_get(pros, 1), kf)
let noacc: [String] = native_list_empty()
let novset: [String] = native_list_empty()
let t: String = sem_realize(sem_frame("greet", "Will", "", ""))
let codes: [String] = text_phonemes(lmap, t)
println("[render] \"" + t + "\" :: " + list_join(codes, " "))
let samples: [Int] = synth_codes_accent(codes, vw, pmap, noacc, novset, vmap, pros)
let ok: Bool = write_wav(samples, 16000, outdir + "will-reply3.wav")
println("[done] will-reply3.wav=" + bool_to_str(ok))
}
@@ -0,0 +1,26 @@
// surface-profile-demo.el - ONE SemFrame, realized ONCE, projected to THREE
// surfaces via surface profiles. Proves surface-as-profile natively: the frame
// and the realized sentence are identical; only the surface PROFILE differs.
fn demo() -> String {
// 1. The shared frame (meaning-geometry): assert(Neuron, contain, the memory).
let frame: [String] = sem_frame("assert", "Neuron", "the memory", "")
// 2. REALIZE once via the EXISTING native realizer (language = a profile).
let sentence: String = sem_realize(frame)
// 3. PROJECT the same realized sentence onto three surfaces (surface = a
// profile). Same frame, same sentence, different surface one render.
let heading: String = "Memory"
let md: String = surface_section(surface_profile_markdown(), heading, sentence)
let html: String = surface_section(surface_profile_html(), heading, sentence)
let plain: String = surface_section(surface_profile_plain(), heading, sentence)
// 4. Report the non-text seam: a surface profile can declare an audio/image
// medium; the render dispatches to the medium projector on the SAME frame.
let midi_media: String = surface_get(surface_profile_midi(), "media_type")
return "MD=[" + md + "] HTML=[" + html + "] PLAIN=[" + plain + "] MIDI_MEDIA=" + midi_media
}
println(demo())
+100
View File
@@ -0,0 +1,100 @@
# -*- coding: utf-8 -*-
"""Full-lexicon vocabulary-{de,la}.el emitters (custom field mapping for the
German declension/gender API and the Latin case-paradigm API). Reuses the
chunked seed-fn writer from gen_elp_seed_full.
"""
import sys, importlib
from gen_elp_seed_full import write_seed
def uw(x):
"""Unwrap (form, source) tuples that some morphology fns return."""
if isinstance(x, (tuple, list)):
return x[0] if x else ""
return x if x is not None else ""
def build_de():
M = importlib.import_module("morphology_de_full")
rows = []; st = {"verbs":0,"nouns":0,"adjs":0}
# nouns: form0=nom-sg(lemma) form1=plural form2=gender
for lem in sorted(M._NOUNS):
if not lem: continue
try:
g = uw(M.noun_gender(lem))
pl = uw(M.pluralize(lem))
except Exception:
continue
rows.append([lem, "noun", lem, pl, g or "", "", "gender:lexicon"])
st["nouns"] += 1
# adjs: form0=positive form1=comparative form2=superlative
for lem in sorted(M._ADJS):
if not lem: continue
try:
cmpr = uw(M.comparative(lem))
sprl = uw(M.superlative(lem))
except Exception:
continue
rows.append([lem, "adj", lem, cmpr, sprl, "", "degree:lexicon"])
st["adjs"] += 1
# verbs (only the ~30 irregular/strong stems the cache carries):
# form0=pres-3sg form1=past-3sg form2=past-participle
if hasattr(M, "_VERBS"):
for lem in sorted({k[0] if isinstance(k, tuple) else k for k in M._VERBS}):
if not lem: continue
try:
f0 = uw(M.finite(lem, "present", "third", "singular"))
f1 = uw(M.finite(lem, "past", "third", "singular"))
pp = uw(M.past_participle(lem))
except Exception:
continue
rows.append([lem, "verb", f0, f1, pp, "", "class:strong/irregular"])
st["verbs"] += 1
return rows, st
def build_la():
M = importlib.import_module("morphology_lat_full")
rows = []; st = {"verbs":0,"nouns":0,"adjs":0}
def dn(lem, c, n):
try:
r = M.decline_noun(lem, c, n)
return uw(r)
except Exception:
return ""
# nouns: dictionary citation — form0=nom-sg form1=gen-sg form2=gender
for lem in sorted(M._NOUNS):
if not lem: continue
nom = dn(lem, "NOM", "SG") or lem
gen = dn(lem, "GEN", "SG")
try: g = uw(M.noun_gender(lem))
except Exception: g = ""
rows.append([lem, "noun", nom, gen, g, "", "case-paradigm nom/gen-sg"])
st["nouns"] += 1
# adjs: three-gender nom-sg citation — form0=masc form1=fem form2=neut
for lem in sorted(M._ADJS):
if not lem: continue
try:
m = uw(M.decline_adj(lem, "NOM", "MASC", "SG")) or lem
f = uw(M.decline_adj(lem, "NOM", "FEM", "SG"))
nt = uw(M.decline_adj(lem, "NOM", "NEUT", "SG"))
except Exception:
continue
rows.append([lem, "adj", m, f, nt, "", "3-gender nom-sg"])
st["adjs"] += 1
# verbs: principal parts — form0=pres-ind-1sg form1=pres-infinitive form2=perf-participle
if hasattr(M, "_VERBS"):
for lem in sorted({k[0] if isinstance(k, tuple) else k for k in M._VERBS}):
if not lem: continue
try:
f0 = uw(M.conjugate(lem, "present", "indicative", "active", "first", "singular"))
inf = uw(M.infinitive(lem, "present", "active"))
pp = uw(M.participle(lem, "perfect", "nom", "m", "singular"))
except Exception:
continue
rows.append([lem, "verb", f0, inf, pp, "", "principal-parts pres1sg/inf/pfppl"])
st["verbs"] += 1
return rows, st
if __name__ == "__main__":
lang = sys.argv[1]; out = sys.argv[2]
rows, st = build_de() if lang == "de" else build_la()
total, _ = write_seed(lang, rows, st, out)
print(f"{lang}: wrote {out} total={total} verbs={st['verbs']} nouns={st['nouns']} adjs={st['adjs']}")
+129
View File
@@ -0,0 +1,129 @@
# -*- coding: utf-8 -*-
"""gen_elp_seed_full.py — emit a FULL-lexicon vocabulary-{lang}.el in the
established ELP seed-fn format (same as vocabulary-non.el / the 18 classical
languages), iterating the ENTIRE morphology_{lang}_full lexicon (every verb,
noun, adjective lemma) — NOT a curated demo core.
Schema per row: [lemma, pos, form0, form1, form2, en_translation, semantic_hint]
Verbs: form0=pres-ind-3sg form1=preterite-3sg form2=past-participle
Nouns: form0=singular form1=plural form2=REAL gender (lexicon)
Adjs : form0=masc-sg form1=fem-sg form2=masc-pl
Output structure (chunked to stay within the proven ~5k-append/function scale):
fn vocab_{lang}_seed_pN(v) -> [[String]] { ... appends ... return v }
fn vocab_{lang}_seed() -> [[String]] { chains all chunks; return v }
fn vocab_{lang}_lookup(w) -> [String] { linear scan }
Usage: python3 gen_elp_seed_full.py <lang> <out.el>
"""
import sys, importlib
CHUNK = 5000
def esc(s):
return str(s).replace("\\", "\\\\").replace('"', '\\"')
def row(fields):
return " let v = native_list_append(v, [" + ", ".join(f'"{esc(f)}"' for f in fields) + "])"
def build_rows(lang, M):
rows = []
stats = {"verbs":0,"nouns":0,"adjs":0}
has = lambda n: hasattr(M, n)
# --- verbs ---
if has("_VERBS") and has("conjugate"):
verbs = sorted({k[0] for k in M._VERBS})
for lem in verbs:
if not lem: continue
try:
f0, s0 = M.conjugate(lem, "ind", "present", "third", "singular")
f1, _ = M.conjugate(lem, "ind", "preterite", "third", "singular")
pp, _ = (M.participle(lem) if has("participle") else ("",""))
except Exception:
continue
vclass = lem[-2:] if lem[-2:] in ("ar","er","ir","re") else lem[-2:]
rows.append([lem, "verb", f0 or "", f1 or "", pp or "", "", "class:"+vclass+" src:"+str(s0)])
stats["verbs"] += 1
# --- nouns ---
if has("_NOUNS") and has("inflect_noun"):
for lem in sorted(M._NOUNS):
if not lem: continue
try:
sg, _ = M.inflect_noun(lem, "singular")
pl, _ = M.inflect_noun(lem, "plural")
g = M.noun_gender(lem) if has("noun_gender") else ""
except Exception:
continue
src = "lexicon" if (isinstance(M._NOUNS.get(lem), dict) and M._NOUNS[lem].get("g")) else "heuristic"
rows.append([lem, "noun", sg or lem, pl or "", g or "", "", "gender:"+src])
stats["nouns"] += 1
# --- adjectives ---
if has("_ADJS") and has("inflect_adj"):
for lem in sorted(M._ADJS):
if not lem: continue
try:
m_sg, _ = M.inflect_adj(lem, "m", "singular")
f_sg, _ = M.inflect_adj(lem, "f", "singular")
m_pl, _ = M.inflect_adj(lem, "m", "plural")
except Exception:
continue
rows.append([lem, "adj", m_sg or lem, f_sg or "", m_pl or "", "", "src:lexicon"])
stats["adjs"] += 1
return rows, stats
def write_seed(lang, rows, stats, out_path):
"""Write vocabulary-{lang}.el in the chunked seed-fn format from prebuilt rows.
Each row is a 7-field list [lemma,pos,f0,f1,f2,gloss,hint]."""
total = len(rows)
chunks = [rows[i:i+CHUNK] for i in range(0, total, CHUNK)] or [[]]
L = []
L.append(f"// vocabulary-{lang}.el — FULL {lang} lexicon for ELP surface realization.")
L.append(f"// Generated by gen_elp_seed_full.py from morphology_{lang}_full")
L.append(f"// (real UniMorph + kaikki.org Wiktionary forms; gender from lexicon, not heuristic).")
L.append(f"// Entries: {total} (verbs={stats['verbs']} nouns={stats['nouns']} adjs={stats['adjs']})")
L.append(f"// Schema: [lemma, pos, form0, form1, form2, en_translation, semantic_hint]")
L.append(f"// verbs: form0=pres-3sg form1=pret-3sg form2=past-participle")
L.append(f"// nouns: form0=sg form1=pl form2=REAL gender adjs: form0=m-sg form1=f-sg form2=m-pl")
L.append("")
for ci, ch in enumerate(chunks):
L.append(f"fn vocab_{lang}_seed_p{ci}(v: [[String]]) -> [[String]] {{")
for r in ch:
L.append(row(r))
L.append(" return v")
L.append("}")
L.append("")
L.append(f"fn vocab_{lang}_seed() -> [[String]] {{")
L.append(" let v: [[String]] = native_list_empty()")
for ci in range(len(chunks)):
L.append(f" let v = vocab_{lang}_seed_p{ci}(v)")
L.append(" return v")
L.append("}")
L.append("")
L.append(f"fn vocab_{lang}_lookup(word: String) -> [String] {{")
L.append(f" let vocab: [[String]] = vocab_{lang}_seed()")
L.append(" let n: Int = native_list_len(vocab)")
L.append(" let i: Int = 0")
L.append(" while i < n {")
L.append(" let entry: [String] = native_list_get(vocab, i)")
L.append(' if str_eq(native_list_get(entry, 0), word) { return entry }')
L.append(" let i = i + 1")
L.append(" }")
L.append(" return native_list_empty()")
L.append("}")
with open(out_path, "w", encoding="utf-8") as fh:
fh.write("\n".join(L) + "\n")
return total, stats
def emit(lang, out_path):
M = importlib.import_module(f"morphology_{lang}_full")
rows, stats = build_rows(lang, M)
return write_seed(lang, rows, stats, out_path)
if __name__ == "__main__":
lang, out = sys.argv[1], sys.argv[2]
total, stats = emit(lang, out)
print(f"{lang}: wrote {out} total={total} verbs={stats['verbs']} nouns={stats['nouns']} adjs={stats['adjs']}")
+572
View File
@@ -0,0 +1,572 @@
# -*- coding: utf-8 -*-
"""morphology_ca_full.py — production-grade Catalan morphological generator.
Same design as morphology_it_full.py (its Romance sibling); Catalan-specific data.
VERBS
UniMorph Catalan (github.com/unimorph/cat, CC-BY-SA 3.0)
7,535 verb lemmas × paradigm, CLEAN orthography:
present, imperfet (PST;IPFV), pretèrit simple (PST;PFV), futur,
condicional (COND), subjuntiu present (SBJV;PRS) / imperfet (SBJV;PST),
imperatiu (POS;IMP), infinitiu (NFIN), gerundi (V.CVB;PRS),
participi (V.PTCP;PST) — WITH full gender+number agreement forms
(cantat/cantada/cantats/cantades) stored directly.
ca_irreg_verbs.json — verbs UniMorph MISSES or under-populates
(anar, fer, plus core auxiliaries ser/haver/estar/tenir…), extracted from
kaikki.org Catalan by build_ca_irreg.py. Priority layer. Supplies anar,
whose present (vaig/vas/va/anem/aneu/van) is ALSO the PERIPHRASTIC-PRETERITE
auxiliary (vaig cantar = 'I sang') — a hallmark Catalan construction.
NOUNS + ADJECTIVES — kaikki.org Catalan (Wiktionary extract, CC-BY-SA 3.0)
noun lemmas WITH inherent gender + real plural (resolved PER LEMMA).
adjective lemmas with real feminine + plural forms.
Fallbacks degrade, never crash:
verbs : regular -ar/-er/-re/-ir rule generator (+ -car/-gar/-çar spelling).
nouns : gender heuristic + rule pluralization (-a→-es with ç/c/g/j/qu/gu
spelling changes; sibilant-final → -os; else -s). Ambiguous → FLAG.
adjs : -o? no (Catalan masc often consonant/-e); fem -a rule + plural rule.
Confidence flag per form: "lexicon" | "rule" | "fallback" (low → FLAG).
Public API (used by realizer_ca.py):
conjugate(lemma, mood, tense, person, number) -> (form, conf)
peri_pret_aux(person, number) -> form # anar-present, for vaig+INF
participle(lemma, gender, number) -> (form, conf)
gerund(lemma) -> (form, conf)
noun_gender(lemma) -> "m"|"f"
inflect_noun(lemma, number, gender=None) -> (form, conf)
inflect_adj(lemma, gender, number) -> (form, conf)
lexicon_stats() -> dict
"""
import json
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "cat.unimorph")
_IRREG = os.path.join(_HERE, "data", "ca_irreg_verbs.json")
_KAIKKI = os.path.join(_HERE, "data", "kaikki_ca.jsonl")
_CACHE = os.path.join(_HERE, "data", "ca_morph_cache.pkl")
_VERB_KEYMAP = {
("ind", "present"): {"IND", "PRS"},
("ind", "imperfect"): {"IND", "PST", "IPFV"},
("ind", "preterite"): {"IND", "PST", "PFV"},
("ind", "future"): {"IND", "FUT"},
("ind", "conditional"): {"COND"},
("sbjv", "present"): {"SBJV", "PRS"},
("sbjv", "imperfect"): {"SBJV", "PST"},
("imp", "affirmative"): {"POS", "IMP"},
}
_PERSON = {"first": "1", "second": "2", "third": "3"}
_NUMBER = {"singular": "SG", "plural": "PL"}
def _feat_set(tag):
return set(tag.split(";"))
# ── verbs from UniMorph ──────────────────────────────────────────────────────────
def _build_verbs():
verbs = {}
part = {} # lemma -> {("m","SG"):form, ("f","SG"):..., ("m","PL"):..., ("f","PL"):...}
ger = {}
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
f = _feat_set(tag)
head = tag.split(";")[0]
if head == "V.PTCP":
if "PST" in f:
g = "f" if "FEM" in f else "m"
n = "PL" if "PL" in f else "SG"
part.setdefault(lemma, {})[(g, n)] = form
continue
if head == "V.CVB":
if "PRS" in f:
ger.setdefault(lemma, form)
continue
if head != "V":
continue
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person is None or number is None:
continue
for (mood, tense), req in _VERB_KEYMAP.items():
if not req <= f:
continue
if tense == "imperfect" and "PFV" in f:
continue
if tense == "preterite" and "IPFV" in f:
continue
verbs.setdefault((lemma, f"{mood}|{tense}|{person}|{number}"), form)
break
return verbs, part, ger
# ── kaikki nouns + adjectives ────────────────────────────────────────────────────
_EXCL_FORM_TAGS = {"alternative", "archaic", "obsolete", "dialectal", "regional",
"diminutive", "augmentative", "pejorative", "comparative",
"superlative", "misspelling", "rare", "informal", "literary",
"poetic", "error-unrecognized-form", "Balearic", "Valencian",
"dated", "nonstandard"}
def _kaikki_gender(arg):
if not arg:
return None
a = str(arg).lower()
if a.startswith("f"):
return "f"
if a.startswith("m"):
return "m"
return None
def _build_nouns_adjs():
nouns = {}
adjs = {}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
pos = d.get("pos")
word = d.get("word", "")
if not word or " " in word:
continue
forms = d.get("forms", []) or []
if pos == "noun":
ht = d.get("head_templates") or []
g = None
if ht:
g = _kaikki_gender((ht[0].get("args") or {}).get("1"))
if g is None:
tags = d.get("tags") or []
if "feminine" in tags:
g = "f"
elif "masculine" in tags:
g = "m"
pl = None
for x in forms:
t = set(x.get("tags") or [])
if "plural" in t and not (t & _EXCL_FORM_TAGS):
fm = x.get("form")
if fm and " " not in fm and fm not in ("#", "", "-"):
pl = fm
break
if word not in nouns:
nouns[word] = {"g": g, "SG": word, "PL": pl}
else:
cur = nouns[word]
if cur.get("g") is None and g:
cur["g"] = g
if not cur.get("PL") and pl:
cur["PL"] = pl
elif pos == "adj":
d0 = adjs.setdefault(word, {})
d0.setdefault(("m", "SG"), word)
for x in forms:
t = set(x.get("tags") or [])
fm = x.get("form")
if not fm or " " in fm or (t & _EXCL_FORM_TAGS):
continue
if "feminine" in t and "plural" in t:
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
elif "masculine" in t and "plural" in t:
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
elif "feminine" in t:
d0[("f", "SG")] = d0.get(("f", "SG")) or fm
elif "plural" in t:
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
return nouns, adjs
def _build_cache():
verbs, part, ger = _build_verbs()
nouns, adjs = _build_nouns_adjs()
with open(_IRREG, encoding="utf-8") as fh:
irreg = json.load(fh)
data = {"verbs": verbs, "part": part, "ger": ger,
"nouns": nouns, "adjs": adjs, "irreg": irreg}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE):
srcs = [_UNIMORPH, _KAIKKI, _IRREG]
newest = max(os.path.getmtime(s) for s in srcs if os.path.exists(s))
if os.path.getmtime(_CACHE) >= newest:
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_VERBS, _PART, _GER, _NOUNS, _ADJS, _IRREGV = (
_LEX["verbs"], _LEX["part"], _LEX["ger"], _LEX["nouns"], _LEX["adjs"],
_LEX["irreg"])
_PERI = _IRREGV.get("_peri_pret_aux", {})
# ── regular verb rule fallback ───────────────────────────────────────────────────
def _vclass(lemma):
if lemma.endswith("ar"):
return "ar"
if lemma.endswith("re"):
return "re"
if lemma.endswith("er"):
return "er"
if lemma.endswith("ir"):
return "ir"
return None
# endings [1sg,2sg,3sg,1pl,2pl,3pl] — central Catalan
_REG = {
("ind", "present", "ar"): ["o", "es", "a", "em", "eu", "en"],
("ind", "present", "re"): ["o", "s", "", "em", "eu", "en"],
("ind", "present", "er"): ["o", "s", "", "em", "eu", "en"],
("ind", "present", "ir"): ["o", "es", "", "im", "iu", "en"], # pure -ir (dormir)
("ind", "imperfect", "ar"): ["ava", "aves", "ava", "àvem", "àveu", "aven"],
("ind", "imperfect", "re"): ["ia", "ies", "ia", "íem", "íeu", "ien"],
("ind", "imperfect", "er"): ["ia", "ies", "ia", "íem", "íeu", "ien"],
("ind", "imperfect", "ir"): ["ia", "ies", "ia", "íem", "íeu", "ien"],
("ind", "preterite", "ar"): ["í", "ares", "à", "àrem", "àreu", "aren"],
("ind", "preterite", "re"): ["í", "eres", "é", "érem", "éreu", "eren"],
("ind", "preterite", "er"): ["í", "eres", "é", "érem", "éreu", "eren"],
("ind", "preterite", "ir"): ["í", "ires", "í", "írem", "íreu", "iren"],
("sbjv", "present", "ar"): ["i", "is", "i", "em", "eu", "in"],
("sbjv", "present", "re"): ["i", "is", "i", "em", "eu", "in"],
("sbjv", "present", "er"): ["i", "is", "i", "em", "eu", "in"],
("sbjv", "present", "ir"): ["i", "is", "i", "im", "iu", "in"],
("sbjv", "imperfect", "ar"): ["és", "essis", "és", "éssim", "éssiu", "essin"],
("sbjv", "imperfect", "re"): ["és", "essis", "és", "éssim", "éssiu", "essin"],
("sbjv", "imperfect", "er"): ["és", "essis", "és", "éssim", "éssiu", "essin"],
("sbjv", "imperfect", "ir"): ["ís", "issis", "ís", "íssim", "íssiu", "issin"],
("imp", "affirmative", "ar"): [None, "a", "i", "em", "eu", "in"],
("imp", "affirmative", "re"): [None, "", "i", "em", "eu", "in"],
("imp", "affirmative", "er"): [None, "", "i", "em", "eu", "in"],
("imp", "affirmative", "ir"): [None, "", "i", "im", "iu", "in"],
}
_FUT = ["é", "às", "à", "em", "eu", "an"]
_COND = ["ia", "ies", "ia", "íem", "íeu", "ien"]
def _slot_idx(person, number):
base = {"first": 0, "second": 1, "third": 2}[person]
return base + (0 if number == "singular" else 3)
def _apply_ar_spelling(stem, ending):
"""-car/-gar/-çar/-jar spelling before front (e/i) endings."""
front = ending[:1] in ("e", "i", "é", "í")
if not front:
# ç before back vowel stays; but -çar stem already ends ç
return stem + ending
if stem.endswith("c"):
return stem[:-1] + "qu" + ending
if stem.endswith("g"):
return stem[:-1] + "gu" + ending
if stem.endswith("ç"):
return stem[:-1] + "c" + ending
if stem.endswith("j"):
return stem[:-1] + "g" + ending
if stem.endswith("qu"):
return stem + ending
return stem + ending
def _rule_conjugate(lemma, mood, tense, person, number):
vc = _vclass(lemma)
if vc is None:
return None
body = lemma[:-2]
i = _slot_idx(person, number)
if mood == "ind" and tense in ("future", "conditional"):
# future/cond stem = infinitive (for -re verbs drop final -e)
stem = lemma[:-1] if vc == "re" else lemma
end = (_FUT if tense == "future" else _COND)[i]
return stem + end
table = _REG.get((mood, tense, vc))
if not table:
return None
end = table[i]
if end is None:
return None
if vc == "ar":
return _apply_ar_spelling(body, end)
# -re/-er/-ir: guard double vowel
if body and body[-1:] == end[:1] and end[:1] in "":
return body[:-1] + end
return body + end
# ── PUBLIC: verb conjugation ─────────────────────────────────────────────────────
def conjugate(lemma, mood, tense, person, number):
lemma = lemma.strip().lower()
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{number and number[:2].upper()}"
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{_NUMBER.get(number,'?')}"
# UniMorph (cleanly accented) takes priority; the kaikki irregulars layer is a
# FALLBACK for verbs/slots UniMorph lacks (anar, fer, and rarer paradigm cells).
p, n = _PERSON.get(person), _NUMBER.get(number)
if p and n:
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}"))
if form:
return form, "lexicon"
ir = _IRREGV.get(lemma)
if ir and key in ir:
return ir[key], "lexicon"
r = _rule_conjugate(lemma, mood, tense, person, number)
if r is not None:
return r, "rule"
return lemma, "fallback"
def peri_pret_aux(person, number):
"""anar-present auxiliary for the periphrastic preterite (vaig cantar)."""
return _PERI.get(f"{_PERSON.get(person,'3')}|{_NUMBER.get(number,'SG')}", "va")
# ── PUBLIC: participle + gerund ──────────────────────────────────────────────────
def participle(lemma, gender="m", number="singular"):
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
ir = _IRREGV.get(lemma)
base = None
if ir and "part" in ir:
# prefer explicit irregular agreement form (part_mSG/part_fSG/...)
exact = ir.get("part_" + g + num)
if exact:
return exact, "lexicon"
base = ir["part"]
elif lemma in _PART:
table = _PART[lemma]
if (g, num) in table:
return table[(g, num)], "lexicon"
base = table.get(("m", "SG"))
if base is None:
vc = _vclass(lemma)
if vc == "ar":
base = lemma[:-2] + "at"
elif vc == "ir":
base = lemma[:-2] + "it"
elif vc in ("er", "re"):
base = lemma[:-2] + "ut"
else:
return lemma, "fallback"
conf = "rule"
else:
conf = "lexicon"
# agreement on -t/-ut/-at/-it participles: m.sg base, f.sg +a (-da? no: -ada),
# Catalan: cantat/cantada/cantats/cantades; -t → f -da, pl -ts/-des
if base.endswith("t"):
stem = base[:-1]
forms = {"m|SG": base, "f|SG": stem + "da",
"m|PL": base + "s", "f|PL": stem + "des"}
return forms[f"{g}|{num}"], conf
if base.endswith("s"): # after sibilant participle (rare): pres->presa
stem = base
forms = {"m|SG": base, "f|SG": base + "a",
"m|PL": base + "os", "f|PL": base + "es"}
return forms[f"{g}|{num}"], conf
return base, conf
def gerund(lemma):
lemma = lemma.strip().lower()
ir = _IRREGV.get(lemma)
if ir and "ger" in ir:
return ir["ger"], "lexicon"
if lemma in _GER:
return _GER[lemma], "lexicon"
vc = _vclass(lemma)
if vc == "ar":
return lemma[:-2] + "ant", "rule"
if vc in ("er", "re"):
return lemma[:-2] + "ent", "rule"
if vc == "ir":
return lemma[:-2] + "int", "rule"
return lemma, "fallback"
# ── PUBLIC: noun gender + number ─────────────────────────────────────────────────
_FEM_SUF = ("ció", "sió", "tat", "tud", "esa", "esa", "dat", "ança", "ència",
"ància", "tud", "ícia", "esa", "or") # note -or is mixed; kaikki wins
_MASC_SUF = ("atge", "ment", " isme", "or")
def _gender_heuristic(noun):
for suf in ("ció", "sió", "tat", "tud", "esa", "ança", "ència", "ància",
"ícia", "etat"):
if noun.endswith(suf):
return "f"
if noun.endswith("a") and not noun.endswith("ma"):
return "f"
return "m"
def noun_gender(lemma):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if d and d.get("g") in ("m", "f"):
return d["g"]
return _gender_heuristic(lemma)
def _rule_plural(noun, gender):
"""Deterministic Catalan pluralization. (form, ok); ok=False FLAGS ambiguity."""
if not noun:
return noun, True
# stressed final vowel with accent → +ns (mà→mans is irregular; but capità→capitans)
if noun[-1:] in ("à", "é", "í", "ó", "ú"):
return noun + "ns", True
if noun.endswith("ça"):
return noun[:-2] + "ces", True # plaça→places
if noun.endswith("ca"):
return noun[:-2] + "ques", True # branca→branques
if noun.endswith("ga"):
return noun[:-2] + "gues", True # amiga→amigues
if noun.endswith("ja"):
return noun[:-2] + "ges", True # pluja→pluges
if noun.endswith("qua"):
return noun[:-3] + "qües", True
if noun.endswith("gua"):
return noun[:-3] + "gües", True
if noun.endswith("a"):
return noun[:-1] + "es", True # casa→cases
# sibilant-final → -os
if noun.endswith(("s", "ç", "x", "ig")) or noun.endswith(("ix", "tx", "tj")):
if noun.endswith("ç"):
return noun[:-1] + "ços", True # braç→braços
return noun + "os", True # peix→peixos, gas→gasos
if noun[-1:] in ("e", "i", "o", "u"):
return noun + "s", True
# consonant-final
return noun + "s", True
def inflect_noun(lemma, number, gender=None):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if number == "singular":
return (d["SG"] if d and d.get("SG") else lemma), ("lexicon" if d else "rule")
if d and d.get("PL"):
return d["PL"], "lexicon"
g = gender or noun_gender(lemma)
form, ok = _rule_plural(lemma, g)
return form, ("rule" if ok else "fallback")
# ── PUBLIC: adjective agreement ──────────────────────────────────────────────────
def _fem_of(adj):
"""Regular Catalan feminine: consonant/-o? Catalan masc usually consonant or -e.
default +a with spelling changes; -e→-a for some; but many are invariable."""
a = adj
if a.endswith("a"):
return a
if a.endswith("e"):
return a[:-1] + "a" # ample→? actually 'ample' invariable; kaikki wins
if a.endswith("u"):
return a + "a"
if a.endswith("c"):
return a[:-1] + "ca" # ric→rica
if a.endswith("t"):
return a + "a" # alt→alta
return a + "a"
def inflect_adj(lemma, gender, number):
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
d = _ADJS.get(lemma)
if d:
form = d.get((g, num))
if form:
return form, "lexicon"
sg = d.get((g, "SG")) or d.get(("m", "SG")) or lemma
if num == "PL":
pl, ok = _rule_plural(sg, g)
return pl, ("rule" if ok else "fallback")
return sg, "lexicon"
# rule fallback
base = lemma if g == "m" else _fem_of(lemma)
if num == "SG":
return base, "rule"
pl, ok = _rule_plural(base, g)
return pl, ("rule" if ok else "fallback")
def lexicon_stats():
return {
"verb_source": "UniMorph Catalan (github.com/unimorph/cat) + kaikki.org "
"irregulars (anar/fer/auxiliaries)",
"noun_adj_source": "kaikki.org Catalan (Wiktionary extract)",
"license": "CC-BY-SA 3.0 (Wiktionary/UniMorph lineage)",
"unimorph_verb_forms": len(_VERBS),
"unimorph_verb_lemmas": len({k[0] for k in _VERBS}),
"irregular_verb_lemmas": len([k for k in _IRREGV if not k.startswith("_")]),
"participle_lemmas": len(_PART),
"gerund_lemmas": len(_GER),
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
}
if __name__ == "__main__":
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
tests = [
("cantar", "ind", "present", "first", "singular", "canto"),
("cantar", "ind", "present", "third", "plural", "canten"),
("ser", "ind", "present", "third", "singular", "és"),
("haver", "ind", "present", "first", "singular", "he"),
("anar", "ind", "present", "first", "singular", "vaig"),
("fer", "ind", "present", "third", "singular", "fa"),
("perdre", "ind", "present", "first", "singular", "perdo"),
("dormir", "ind", "present", "third", "plural", "dormen"),
("cantar", "ind", "future", "first", "singular", "cantaré"),
("cantar", "ind", "preterite", "third", "singular", "cantà"),
("tenir", "sbjv", "present", "first", "singular", "tingui"),
]
ok = 0
for lemma, mood, tense, per, num, exp in tests:
got, conf = conjugate(lemma, mood, tense, per, num)
flag = "OK " if got == exp else "XX "
ok += got == exp
print(f" {flag}{lemma:8} {mood}/{tense:11} {per[:3]}.{num[:2]} -> {got:10} ({conf}) exp={exp}")
print(f"verb tests {ok}/{len(tests)}")
print(" peri-pret anar: 1sg=", peri_pret_aux("first", "singular"),
"3pl=", peri_pret_aux("third", "plural"))
print(" gender casa=", noun_gender("casa"), "home=", noun_gender("home"),
"cavall=", noun_gender("cavall"), "cançó=", noun_gender("cançó"))
print(" plural casa->", inflect_noun("casa", "plural"),
"| plaça->", inflect_noun("plaça", "plural"),
"| peix->", inflect_noun("peix", "plural"),
"| braç->", inflect_noun("braç", "plural"),
"| home->", inflect_noun("home", "plural"))
print(" adj: alt/f/sg->", inflect_adj("alt", "f", "singular"),
"| bonic/f/pl->", inflect_adj("bonic", "f", "plural"),
"| vermell/f/sg->", inflect_adj("vermell", "f", "singular"))
print(" part: cantar/f/sg->", participle("cantar", "f", "singular"),
"| veure/f/pl->", participle("veure", "f", "plural"),
"| fer/m/sg->", participle("fer", "m", "singular"))
print(" ger: fer->", gerund("fer"), "| cantar->", gerund("cantar"))
+423
View File
@@ -0,0 +1,423 @@
# -*- coding: utf-8 -*-
"""morphology_de_full.py — production German morphological generator.
Real data, no toy tables:
PRIMARY — UniMorph German (github.com/unimorph/deu, CC-BY-SA 3.0).
~219k noun forms, ~199k verb forms. Supplies:
nouns : gender (MASC/FEM/NEUT) + case×number paradigm
(N;NOM/ACC/DAT/GEN; MASC/FEM/NEUT; SG/PL) — the genitive -(e)s,
dative-plural -n and the five plural classes are REAL forms, not
guessed.
verbs : full finite paradigm IND;{SG,PL};{1,2,3};{PRS,PST}, the past
participle (V.PTCP;PST, incl. reattached separable prefix
'zugefügt'), and — crucially for V2 — the SEPARATED finite form
UniMorph records directly ('füge zu', 'steht auf').
adjs : comparative / superlative (ADJ;CMPR, ADJ;SPRL).
SECONDARY — kaikki.org German (Wiktionary, CC-BY-SA/GFDL). Gap-fills noun
gender + plural where UniMorph is thin. Never overrides UniMorph.
Rule fallbacks (flagged 'rule'/'fallback') for lemmas absent from both lexicons:
present : -e/-st/-t/-en/-t/-en with e-epenthesis after -t/-d/-chn stems
plural : gender heuristic (fem -> -(e)n, else -e / umlaut left to lexicon)
ppart : weak ge-…-t
Adjective ENDINGS are rule-computed by the realizer (regular closed table);
this module only supplies the comparative/superlative STEM.
Perfect auxiliary (haben vs sein): sein for a curated set of intransitive
motion / change-of-state verbs (real German lexical property), else haben.
Public API:
noun_gender(lemma) -> 'm'|'f'|'n'
decline_noun(lemma, case, number) -> (form, conf)
pluralize(lemma) -> (form, conf)
finite(lemma, tense, person, number) -> (form, conf) # may contain ' prefix'
nonfinite(lemma, req) -> (form, conf) # req: 'inf'|'ppart'
past_participle(lemma) -> (form, conf)
separable_prefix(lemma) -> str|None
perfect_aux(lemma) -> 'haben'|'sein'
comparative(lemma)/superlative(lemma) -> (stem, conf)
lexicon_stats() -> dict
"""
import json
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "deu.unimorph")
_KAIKKI = os.path.join(_HERE, "data", "kaikki_de.jsonl")
_CACHE = os.path.join(_HERE, "data", "de_morph_cache.pkl")
_GENDER = {"MASC": "m", "FEM": "f", "NEUT": "n"}
# intransitive motion / change-of-state verbs that take SEIN in the perfect
_SEIN = {"gehen", "kommen", "fahren", "laufen", "rennen", "reisen", "fallen",
"steigen", "sinken", "wachsen", "sterben", "geschehen", "passieren",
"werden", "bleiben", "sein", "aufstehen", "einschlafen", "aufwachen",
"ankommen", "abfahren", "aufsteigen", "erscheinen", "verschwinden",
"fliegen", "schwimmen", "springen", "begegnen", "folgen", "gelingen",
"wandern", "ziehen", "flüchten", "eintreten", "einsteigen", "aussteigen"}
# hardcoded high-frequency irregular / auxiliary / modal paradigms (closed class,
# verified) — consulted before the lexicon so aux+modal chains are always correct.
_CORE = {
"sein": {"prs": {("first", "singular"): "bin", ("second", "singular"): "bist",
("third", "singular"): "ist", ("first", "plural"): "sind",
("second", "plural"): "seid", ("third", "plural"): "sind"},
"pst": {("first", "singular"): "war", ("second", "singular"): "warst",
("third", "singular"): "war", ("first", "plural"): "waren",
("second", "plural"): "wart", ("third", "plural"): "waren"},
"ppart": "gewesen"},
"haben": {"prs": {("first", "singular"): "habe", ("second", "singular"): "hast",
("third", "singular"): "hat", ("first", "plural"): "haben",
("second", "plural"): "habt", ("third", "plural"): "haben"},
"pst": {("first", "singular"): "hatte", ("second", "singular"): "hattest",
("third", "singular"): "hatte", ("first", "plural"): "hatten",
("second", "plural"): "hattet", ("third", "plural"): "hatten"},
"ppart": "gehabt"},
"werden": {"prs": {("first", "singular"): "werde", ("second", "singular"): "wirst",
("third", "singular"): "wird", ("first", "plural"): "werden",
("second", "plural"): "werdet", ("third", "plural"): "werden"},
"pst": {("first", "singular"): "wurde", ("second", "singular"): "wurdest",
("third", "singular"): "wurde", ("first", "plural"): "wurden",
("second", "plural"): "wurdet", ("third", "plural"): "wurden"},
"ppart": "geworden"},
}
_MODAL_PRS = {
"können": ("kann", "kannst", "kann", "können", "könnt", "können"),
"müssen": ("muss", "musst", "muss", "müssen", "müsst", "müssen"),
"wollen": ("will", "willst", "will", "wollen", "wollt", "wollen"),
"sollen": ("soll", "sollst", "soll", "sollen", "sollt", "sollen"),
"dürfen": ("darf", "darfst", "darf", "dürfen", "dürft", "dürfen"),
"mögen": ("mag", "magst", "mag", "mögen", "mögt", "mögen"),
}
_MODAL_PST = {
"können": ("konnte", "konntest", "konnte", "konnten", "konntet", "konnten"),
"müssen": ("musste", "musstest", "musste", "mussten", "musstet", "mussten"),
"wollen": ("wollte", "wolltest", "wollte", "wollten", "wolltet", "wollten"),
"sollen": ("sollte", "solltest", "sollte", "sollten", "solltet", "sollten"),
"dürfen": ("durfte", "durftest", "durfte", "durften", "durftet", "durften"),
"mögen": ("mochte", "mochtest", "mochte", "mochten", "mochtet", "mochten"),
}
_PN_ORDER = [("first", "singular"), ("second", "singular"), ("third", "singular"),
("first", "plural"), ("second", "plural"), ("third", "plural")]
_MODAL_PPART = {"können": "gekonnt", "müssen": "gemusst", "wollen": "gewollt",
"sollen": "gesollt", "dürfen": "gedurft", "mögen": "gemocht"}
for _m, _forms in _MODAL_PRS.items():
_CORE[_m] = {"prs": dict(zip(_PN_ORDER, _forms)),
"pst": dict(zip(_PN_ORDER, _MODAL_PST[_m])),
"ppart": _MODAL_PPART[_m]}
def _person_num(tags):
p = n = None
for t in tags:
if t in ("1", "2", "3"):
p = {"1": "first", "2": "second", "3": "third"}[t]
elif t == "SG":
n = "singular"
elif t == "PL":
n = "plural"
return p, n
def _build_from_unimorph():
nouns, verbs, adjs = {}, {}, {}
if not os.path.exists(_UNIMORPH):
return nouns, verbs, adjs
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tagstr = parts
tags = tagstr.split(";")
head = tags[0]
tset = set(tags)
if head == "N":
rec = nouns.setdefault(lemma, {"g": None, "cases": {}, "pl": None})
g = next((_GENDER[t] for t in tags if t in _GENDER), None)
if g and not rec["g"]:
rec["g"] = g
case = next((t for t in tags if t in ("NOM", "ACC", "DAT", "GEN")), None)
num = "plural" if "PL" in tset else ("singular" if "SG" in tset else None)
if case and num:
rec["cases"].setdefault((case, num), form)
if case == "NOM" and num == "plural" and not rec["pl"]:
rec["pl"] = form
elif head.startswith("V"):
rec = verbs.setdefault(lemma, {"prs": {}, "pst": {}, "ppart": None})
if "PTCP" in head and "PST" in tset:
rec["ppart"] = rec["ppart"] or form
elif "IND" in tset and ("PRS" in tset or "PST" in tset):
p, n = _person_num(tags)
if p and n:
slot = "prs" if "PRS" in tset else "pst"
rec[slot].setdefault((p, n), form)
elif head == "ADJ":
rec = adjs.setdefault(lemma, {})
if "CMPR" in tset:
rec.setdefault("cmpr", form.replace("am ", "").strip())
elif "SPRL" in tset:
rec.setdefault("sprl", form.replace("am ", "").replace("sten", "st")
if form.endswith("sten") else form.replace("am ", ""))
return nouns, verbs, adjs
def _build_from_kaikki(nouns):
"""Gap-fill noun gender + plural from kaikki German."""
if not os.path.exists(_KAIKKI):
return
_g = {"masculine": "m", "feminine": "f", "neuter": "n", "m": "m", "f": "f", "n": "n"}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
if d.get("pos") != "noun":
continue
w = d.get("word", "")
if not w or not w[0].isalpha() or " " in w:
continue
rec = nouns.setdefault(w, {"g": None, "cases": {}, "pl": None})
# GENDER: Wiktionary gender is hand-curated and OVERRIDES UniMorph's
# auto-tagged gender, which has known errors (e.g. UniMorph deu mis-
# records Zeit=MASC, Wagen=NEUT; Wiktionary has f, m correctly).
for h in d.get("head_templates", []) or []:
a = h.get("args", {}) or {}
raw = a.get("1") or a.get("g") or ""
code = str(raw).split(",")[0].strip().lower()
if code in _g:
rec["g"] = _g[code]
break
if not rec["pl"]:
for f in d.get("forms", []) or []:
t = set(f.get("tags", []) or [])
if "plural" in t and f.get("form") and "genitive" not in t:
rec["pl"] = f["form"]
break
def _build_cache():
nouns, verbs, adjs = _build_from_unimorph()
_build_from_kaikki(nouns)
data = {"nouns": nouns, "verbs": verbs, "adjs": adjs}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE):
srcs = [p for p in (_UNIMORPH, _KAIKKI) if os.path.exists(p)]
newest = max((os.path.getmtime(p) for p in srcs), default=0)
if os.path.getmtime(_CACHE) >= newest:
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_NOUNS, _VERBS, _ADJS = _LEX["nouns"], _LEX["verbs"], _LEX["adjs"]
# ── nouns ────────────────────────────────────────────────────────────────────────
def noun_gender(lemma):
rec = _NOUNS.get(lemma) or _NOUNS.get(lemma.capitalize())
if rec and rec.get("g"):
return rec["g"]
# last-resort rule: -ung/-heit/-keit/-schaft/-tät/-ion -> f ; -chen/-lein -> n
low = lemma.lower()
if low.endswith(("ung", "heit", "keit", "schaft", "tät", "ion", "ik", "ei")):
return "f"
if low.endswith(("chen", "lein", "ment", "um")):
return "n"
return "m"
def pluralize(lemma):
rec = _NOUNS.get(lemma) or _NOUNS.get(lemma.capitalize())
if rec and rec.get("pl"):
return rec["pl"], "lexicon"
g = noun_gender(lemma)
if g == "f":
return (lemma + "en" if not lemma.endswith("e") else lemma + "n"), "rule"
return (lemma if lemma.endswith(("er", "en", "el")) else lemma + "e"), "rule"
def decline_noun(lemma, case, number):
"""case in NOM/ACC/DAT/GEN, number in singular/plural."""
rec = _NOUNS.get(lemma) or _NOUNS.get(lemma.capitalize())
if case == "DAT" and number == "singular":
# modern German drops the archaic dative -e ('dem Kinde' -> 'dem Kind');
# the article carries the case. Keep bare nominative form.
base = (rec or {}).get("cases", {}).get(("NOM", "singular")) or lemma
return base, ("lexicon" if rec else "rule")
if rec and rec.get("cases", {}).get((case, number)):
return rec["cases"][(case, number)], "lexicon"
if number == "plural":
pl, c = pluralize(lemma)
if case == "DAT" and not pl.endswith("n") and not pl.endswith("s"):
return pl + "n", c # dative plural -n
return pl, c
# singular
g = noun_gender(lemma)
if case == "GEN" and g in ("m", "n"):
return (lemma + "es" if lemma.endswith(("s", "ß", "z", "x")) else lemma + "s"), "rule"
return lemma, "lexicon" if rec else "rule"
# ── verbs ──────────────────────────────────────────────────────────────────────--
_PRS_ENDINGS = {("first", "singular"): "e", ("second", "singular"): "st",
("third", "singular"): "t", ("first", "plural"): "en",
("second", "plural"): "t", ("third", "plural"): "en"}
def _stem(lemma):
if lemma.endswith("en"):
return lemma[:-2]
if lemma.endswith("n"):
return lemma[:-1]
return lemma
def separable_prefix(lemma):
"""Return the separable prefix if the lemma is a separable-prefix verb."""
rec = _VERBS.get(lemma)
if rec:
for (_p, _n), form in rec.get("prs", {}).items():
if " " in form:
return form.rsplit(" ", 1)[1]
_SEP = ("auf", "aus", "ab", "an", "ein", "mit", "nach", "vor", "zu", "zurück",
"weg", "hin", "her", "los", "bei", "fest", "fort", "um", "zusammen")
_INSEP = ("be", "ge", "er", "ver", "zer", "ent", "emp", "miss")
for p in sorted(_SEP, key=len, reverse=True):
if lemma.startswith(p) and len(lemma) > len(p) + 2 \
and not lemma.startswith(_INSEP):
return p
return None
def finite(lemma, tense, person, number):
"""Present/past finite. For separable verbs the returned string is the
UniMorph SEPARATED form 'stem prefix' (realizer places prefix per V2)."""
slot = "prs" if tense == "present" else "pst"
if lemma in _CORE and _CORE[lemma].get(slot, {}).get((person, number)):
return _CORE[lemma][slot][(person, number)], "lexicon"
rec = _VERBS.get(lemma)
if rec and rec.get(slot, {}).get((person, number)):
return rec[slot][(person, number)], "lexicon"
# rule fallback (present only reliable; past weak -te)
stem = _stem(lemma)
pref = separable_prefix(lemma)
if pref:
stem = _stem(lemma[len(pref):])
if tense == "present":
end = _PRS_ENDINGS[(person, number)]
if stem.endswith(("t", "d", "chn", "ffn", "gn")) and end in ("st", "t"):
end = "e" + end
form = stem + end
else:
form = stem + ("ete" if stem.endswith(("t", "d")) else "te")
if (person, number) == ("second", "singular"):
form += "st"
elif number == "plural" and person != "second":
form += "n"
elif (person, number) == ("second", "plural"):
form += "t"
if pref:
return f"{form} {pref}", "rule"
return form, "rule"
def _weak_t(stem):
return stem + ("et" if stem.endswith(("t", "d", "chn", "ffn", "gn")) else "t")
def past_participle(lemma):
if lemma in _CORE:
return _CORE[lemma]["ppart"], "lexicon"
rec = _VERBS.get(lemma)
if rec and rec.get("ppart"):
return rec["ppart"], "lexicon"
stem = _stem(lemma)
pref = separable_prefix(lemma)
_INSEP = ("be", "ge", "er", "ver", "zer", "ent", "emp", "miss")
if pref:
inner = _stem(lemma[len(pref):])
return pref + "ge" + _weak_t(inner), "rule"
if lemma.startswith(_INSEP):
return _weak_t(stem), "rule"
return "ge" + _weak_t(stem), "rule"
def nonfinite(lemma, req):
if req == "ppart":
return past_participle(lemma)
return lemma, "lexicon" if lemma in _VERBS else "rule" # infinitive
def perfect_aux(lemma):
return "sein" if lemma in _SEIN else "haben"
# ── adjectives ────────────────────────────────────────────────────────────────---
_ADJ_IRREG_SPRL = {"gut": "best", "groß": "größt", "hoch": "höchst",
"nah": "nächst", "viel": "meist", "gern": "liebst"}
def comparative(lemma):
rec = _ADJS.get(lemma)
if rec and rec.get("cmpr"):
return rec["cmpr"], "lexicon"
return lemma + "er", "rule"
def superlative(lemma):
"""Return the bare superlative STEM (realizer adds 'am ...en' or '-e' ending)."""
if lemma in _ADJ_IRREG_SPRL:
return _ADJ_IRREG_SPRL[lemma], "lexicon"
# derive from the comparative so umlaut is carried (alt->älter->ältest)
cmpr, cconf = comparative(lemma)
base = cmpr[:-2] if cmpr.endswith("er") else lemma
end = "est" if base.endswith(("t", "d", "s", "ß", "z", "sch")) else "st"
return base + end, cconf
def lexicon_stats():
return {
"source": "UniMorph deu (primary) + kaikki.org German (gap-fill gender/plural)",
"license": "CC-BY-SA 3.0 (UniMorph); CC-BY-SA/GFDL (Wiktionary)",
"noun_lemmas": len(_NOUNS),
"nouns_with_gender": sum(1 for v in _NOUNS.values() if v.get("g")),
"nouns_with_plural": sum(1 for v in _NOUNS.values() if v.get("pl")),
"verb_lemmas": len(_VERBS),
"verbs_with_ppart": sum(1 for v in _VERBS.values() if v.get("ppart")),
"adj_lemmas": len(_ADJS),
}
if __name__ == "__main__":
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
for w in ("Hund", "Frau", "Kind", "Mann", "Buch", "Blume"):
print(f" {w}: gender={noun_gender(w)} pl={pluralize(w)} "
f"gen.sg={decline_noun(w, 'GEN', 'singular')} "
f"dat.pl={decline_noun(w, 'DAT', 'plural')}")
for v in ("machen", "gehen", "aufstehen", "sein", "haben", "arbeiten"):
print(f" {v}: 3sg.prs={finite(v, 'present', 'third', 'singular')} "
f"3sg.pst={finite(v, 'past', 'third', 'singular')} "
f"ppart={past_participle(v)} aux={perfect_aux(v)} sep={separable_prefix(v)}")
for a in ("schnell", "gut", "groß", "alt"):
print(f" {a}: cmpr={comparative(a)} sprl={superlative(a)}")
+562
View File
@@ -0,0 +1,562 @@
"""morphology_es_full.py — production-grade Spanish morphological generator.
NOT a toy. Backed by a real, broad, licensed lexicon:
UniMorph Spanish (github.com/unimorph/spa, CC-BY-SA 3.0, Wiktionary-derived)
1,196,245 inflected forms:
6,695 verb lemmas full paradigms: indicative (present/preterite/
imperfect/future), conditional, present & imperfect
subjunctive, affirmative imperative, formal/informal
48,353 noun lemmas WITH inherent gender (N;FEM/MASC;SG/PL)
16,984 adj lemmas gender + number paradigms
Fallbacks (so we degrade, never crash, on out-of-vocabulary input):
- verbs : mlconjug3 (ML paradigm model, conjugates ANY Spanish verb) then a
hand-rolled regular-ending generator
- nouns : gender heuristic (endings) + regular pluralization
- adjs : -o/-a gender rule + regular pluralization
Every generated form carries a CONFIDENCE flag:
"lexicon" form came straight from UniMorph (trust: high)
"model" form came from mlconjug3 (trust: high)
"rule" form came from a deterministic rule (trust: medium)
"fallback" we could not inflect; returned lemma as-is (trust: low FLAG)
Public API (used by realizer_es.py):
conjugate(lemma, mood, tense, person, number, formality="informal") -> (form, conf)
participle(lemma) -> (form, conf) # past participle (compound tenses)
gerund(lemma) -> (form, conf)
noun_gender(lemma) -> "m"|"f"
inflect_noun(lemma, number) -> (form, conf)
inflect_adj(lemma, gender, number) -> (form, conf)
attach_enclitics(verb_form, clitics) -> str # accent-correct enclisis
lexicon_stats() -> dict
"""
import os
import pickle
import unicodedata
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "spa.unimorph")
_CACHE = os.path.join(_HERE, "data", "es_morph_cache.pkl")
# ── canonical feature keys the realizer speaks, mapped to UniMorph tags ─────────
# mood/tense pair -> the UniMorph feature substring that identifies it
_VERB_KEYMAP = {
("ind", "present"): ("IND", "PRS", None),
("ind", "preterite"): ("IND", "PST", "PFV"),
("ind", "imperfect"): ("IND", "PST", "IPFV"),
("ind", "future"): ("IND", "FUT", None),
("ind", "conditional"):("COND", None, None),
("sbjv", "present"): ("SBJV", "PRS", None),
("sbjv", "imperfect"): ("SBJV", "PST", "LGSPEC1"), # -ra form
("imp", "present"): ("POS", "IMP", None),
}
_PERSON = {"first": "1", "second": "2", "third": "3"}
_NUMBER = {"singular": "SG", "plural": "PL"}
# ── build / load the compact lexicon ───────────────────────────────────────────
def _feat_set(tag):
return set(tag.split(";"))
def _build_cache():
verbs = {} # (lemma, canonkey) -> form canonkey e.g. "ind|present|1|SG|infm"
nouns = {} # lemma -> {"g": "m"/"f", "SG": form, "PL": form}
adjs = {} # lemma -> {("m","SG"): form, ...}
part = {} # lemma -> masc-sg participle
ger = {} # lemma -> gerund
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
f = _feat_set(tag)
head = tag.split(";")[0]
if head == "V":
# skip clitic-bearing rows (we generate clitics ourselves)
if "PRO" in f:
continue
if "V.PTCP" in f and "PST" in f and "MASC" in f and "SG" in f:
part.setdefault(lemma, form)
continue
if "V.CVB" in f or "NFIN" in f or "V.PTCP" in f:
if "V.CVB" in f:
ger.setdefault(lemma, form)
continue
# identify mood/tense
mt = None
for (mood, tense), (a, b, c) in _VERB_KEYMAP.items():
if a not in f:
continue
if b is not None and b not in f:
continue
if c is not None and c not in f:
continue
# disambiguate IND;PST needing PFV vs IPFV
if a == "IND" and b == "PST" and c not in f:
continue
mt = (mood, tense)
break
if mt is None:
continue
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person is None or number is None:
continue
formal = "form" if "FORM" in f else ("infm" if "INFM" in f else "any")
key = f"{mt[0]}|{mt[1]}|{person}|{number}|{formal}"
verbs.setdefault((lemma, key), form)
elif head == "N":
# substring test handles epicene "MASC+FEM" (-> masc citation)
g = "m" if "MASC" in tag else ("f" if "FEM" in tag else None)
num = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if num is None:
continue
# store forms keyed by (gender,number); animate nouns list BOTH
# genders under one lemma (niño -> niño/niña). Resolve citation
# gender in a post-pass (gender of the row whose form == lemma).
d = nouns.setdefault(lemma, {})
d.setdefault("_rows", []).append((g, num, form))
elif head == "ADJ":
g = "m" if "MASC" in tag else ("f" if "FEM" in tag else "m")
num = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if num is None:
continue
adjs.setdefault(lemma, {})[(g, num)] = form
# post-pass: resolve noun citation gender + default SG/PL forms
for lemma, d in nouns.items():
rows = d.pop("_rows", [])
# citation gender = gender of the row whose form == lemma; else first MASC;
# else first seen gender.
cite_g = None
for g, num, form in rows:
if form == lemma and g:
cite_g = g
break
if cite_g is None:
for g, num, form in rows:
if g == "m":
cite_g = "m"
break
if cite_g is None:
cite_g = next((g for g, _, _ in rows if g), "m")
d["g"] = cite_g
for g, num, form in rows:
d[(g, num)] = form
d["SG"] = d.get((cite_g, "SG")) or next((f for g, n, f in rows if n == "SG"), lemma)
d["PL"] = d.get((cite_g, "PL")) or next((f for g, n, f in rows if n == "PL"), None)
# post-pass: UniMorph omits the identity inflection (masc-sg == lemma) for
# adjectives, so fill it in; without this a fem-sg row wrongly satisfies a
# masc-sg request (alto -> alta bug).
for lemma, d in adjs.items():
d.setdefault(("m", "SG"), lemma)
data = {"verbs": verbs, "nouns": nouns, "adjs": adjs, "part": part, "ger": ger}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE) and os.path.getmtime(_CACHE) >= os.path.getmtime(_UNIMORPH):
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_VERBS, _NOUNS, _ADJS, _PART, _GER = (
_LEX["verbs"], _LEX["nouns"], _LEX["adjs"], _LEX["part"], _LEX["ger"])
# ── mlconjug3 fallback (lazy) ───────────────────────────────────────────────────
_MLC = None
_MLC_TENSE = { # (mood,tense) -> (mlconjug mood label, tense label)
("ind", "present"): ("Indicativo", "Indicativo presente"),
("ind", "preterite"): ("Indicativo", "Indicativo pretérito perfecto simple"),
("ind", "imperfect"): ("Indicativo", "Indicativo pretérito imperfecto"),
("ind", "future"): ("Indicativo", "Indicativo futuro"),
("ind", "conditional"): ("Condicional", "Condicional Condicional"),
("sbjv", "present"): ("Subjuntivo", "Subjuntivo presente"),
("sbjv", "imperfect"): ("Subjuntivo", "Subjuntivo pretérito imperfecto 1"),
("imp", "present"): ("Imperativo", "Imperativo Afirmativo"),
}
_MLC_SLOT = { # (person,number) -> mlconjug slot key
("first", "singular"): "1s", ("second", "singular"): "2s",
("third", "singular"): "3s", ("first", "plural"): "1p",
("second", "plural"): "2p", ("third", "plural"): "3p",
}
def _mlc_conjugate(lemma, mood, tense, person, number):
global _MLC
try:
if _MLC is None:
from mlconjug3 import Conjugator
_MLC = Conjugator(language="es")
v = _MLC.conjugate(lemma)
if v is None:
return None
info = v.conjug_info
m, t = _MLC_TENSE.get((mood, tense), (None, None))
if m is None or m not in info or t not in info[m]:
return None
block = info[m][t]
slot = _MLC_SLOT.get((person, number))
if isinstance(block, dict) and slot in block and block[slot]:
return block[slot]
return None
except Exception:
return None
# ── regular-ending rule fallback (last resort, deterministic) ───────────────────
def _vclass(lemma):
return lemma[-2:] if lemma[-2:] in ("ar", "er", "ir") else "ar"
def _stem(lemma):
return lemma[:-2]
_REG = {
("ind", "present", "ar"): ["o", "as", "a", "amos", "áis", "an"],
("ind", "present", "er"): ["o", "es", "e", "emos", "éis", "en"],
("ind", "present", "ir"): ["o", "es", "e", "imos", "ís", "en"],
("ind", "preterite", "ar"): ["é", "aste", "ó", "amos", "asteis", "aron"],
("ind", "preterite", "er"): ["í", "iste", "", "imos", "isteis", "ieron"],
("ind", "preterite", "ir"): ["í", "iste", "", "imos", "isteis", "ieron"],
("ind", "imperfect", "ar"): ["aba", "abas", "aba", "ábamos", "abais", "aban"],
("ind", "imperfect", "er"): ["ía", "ías", "ía", "íamos", "íais", "ían"],
("ind", "imperfect", "ir"): ["ía", "ías", "ía", "íamos", "íais", "ían"],
("sbjv", "present", "ar"): ["e", "es", "e", "emos", "éis", "en"],
("sbjv", "present", "er"): ["a", "as", "a", "amos", "áis", "an"],
("sbjv", "present", "ir"): ["a", "as", "a", "amos", "áis", "an"],
("sbjv", "imperfect", "ar"): ["ara", "aras", "ara", "áramos", "arais", "aran"],
("sbjv", "imperfect", "er"): ["iera", "ieras", "iera", "iéramos", "ierais", "ieran"],
("sbjv", "imperfect", "ir"): ["iera", "ieras", "iera", "iéramos", "ierais", "ieran"],
}
_FUT = ["é", "ás", "á", "emos", "éis", "án"]
_COND = ["ía", "ías", "ía", "íamos", "íais", "ían"]
def _slot_idx(person, number):
base = {"first": 0, "second": 1, "third": 2}[person]
return base + (0 if number == "singular" else 3)
def _rule_conjugate(lemma, mood, tense, person, number):
if len(lemma) < 3 or lemma[-2:] not in ("ar", "er", "ir"):
return None
vc, st, i = _vclass(lemma), _stem(lemma), _slot_idx(person, number)
if tense == "future":
return lemma + _FUT[i]
if tense == "conditional":
return lemma + _COND[i]
table = _REG.get((mood, tense, vc))
if table:
return st + table[i]
if mood == "imp" and tense == "present":
# affirmative tú imperative = 3sg present indicative
pres = _REG.get(("ind", "present", vc))
return st + pres[2] if number == "singular" else st + pres[5]
return None
# ── PUBLIC: verb conjugation ────────────────────────────────────────────────────
def conjugate(lemma, mood, tense, person, number, formality="informal"):
"""Return (surface, confidence). mood in ind|sbjv|imp; tense per _VERB_KEYMAP."""
lemma = lemma.strip().lower()
p, n = _PERSON.get(person), _NUMBER.get(number)
formal = "form" if formality == "formal" else "infm"
if p and n:
for fkey in (formal, "any", "infm" if formal == "form" else "form"):
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}|{fkey}"))
if form:
return form, "lexicon"
m = _mlc_conjugate(lemma, mood, tense, person, number)
if m:
return m, "model"
r = _rule_conjugate(lemma, mood, tense, person, number)
if r:
return r, "rule"
return lemma, "fallback"
_IRREG_PART = { # guarantee the common irregular participles
"escribir": "escrito", "describir": "descrito", "abrir": "abierto",
"cubrir": "cubierto", "descubrir": "descubierto", "morir": "muerto",
"poner": "puesto", "ver": "visto", "volver": "vuelto", "devolver": "devuelto",
"hacer": "hecho", "deshacer": "deshecho", "decir": "dicho", "romper": "roto",
"resolver": "resuelto", "freír": "frito", "imprimir": "impreso",
"satisfacer": "satisfecho", "prever": "previsto", "revolver": "revuelto",
}
def participle(lemma):
lemma = lemma.strip().lower()
if lemma in _IRREG_PART:
return _IRREG_PART[lemma], "lexicon"
if lemma in _PART:
return _PART[lemma], "lexicon"
if lemma.endswith("ar"):
return lemma[:-2] + "ado", "rule"
if lemma[-2:] in ("er", "ir"):
return lemma[:-2] + "ido", "rule"
return lemma, "fallback"
_IRREG_GER = {"dormir": "durmiendo", "morir": "muriendo", "pedir": "pidiendo",
"sentir": "sintiendo", "mentir": "mintiendo", "servir": "sirviendo",
"venir": "viniendo", "decir": "diciendo", "poder": "pudiendo",
"ir": "yendo", "leer": "leyendo", "creer": "creyendo",
"oír": "oyendo", "traer": "trayendo", "caer": "cayendo",
"construir": "construyendo", "huir": "huyendo", "reír": "riendo"}
def gerund(lemma):
lemma = lemma.strip().lower()
if lemma in _IRREG_GER:
return _IRREG_GER[lemma], "lexicon"
if lemma in _GER:
return _GER[lemma], "lexicon"
if lemma.endswith("ar"):
return lemma[:-2] + "ando", "rule"
if lemma[-2:] in ("er", "ir"):
return lemma[:-2] + "iendo", "rule"
return lemma, "fallback"
# ── PUBLIC: noun gender + number ────────────────────────────────────────────────
_INVARIANT_PL = {"lunes", "martes", "miércoles", "jueves", "viernes",
"crisis", "tesis", "análisis", "dosis", "virus", "paraguas"}
def _gender_heuristic(noun):
for suf, g in (("ión", "f"), ("dad", "f"), ("tad", "f"), ("umbre", "f"),
("sis", "f"), ("ez", "f"), ("triz", "f"),
("ema", "m"), ("ama", "m"), ("oma", "m"), ("aje", "m"),
("or", "m"), ("án", "m"), ("ín", "m")):
if noun.endswith(suf):
return g
if noun.endswith("o"):
return "m"
if noun.endswith("a"):
return "f"
return "m"
def noun_gender(lemma):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if d and d.get("g"):
return d["g"]
return _gender_heuristic(lemma)
def _regular_plural(noun):
if noun in _INVARIANT_PL:
return noun
if not noun:
return noun
last = noun[-1]
if last == "z":
return noun[:-1] + "ces"
if last in "aeiouáéíóú":
# stressed final vowel í/ú -> +es (rubí->rubíes), else +s
if last in "íú":
return noun + "es"
return noun + "s"
if last == "s":
# esdrújula / stress-final handled crudely; most polysyllables invariant
return noun
return noun + "es"
def inflect_noun(lemma, number, gender=None):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
num = "SG" if number == "singular" else "PL"
if d:
# honor a requested gender for animate nouns (gato -> gata)
if gender and (gender, num) in d:
return d[(gender, num)], "lexicon"
if d.get(num):
return d[num], "lexicon"
if number == "singular":
return lemma, "rule" if not d else "lexicon"
return _regular_plural(lemma), "rule"
# ── PUBLIC: adjective agreement ─────────────────────────────────────────────────
_INV_GENDER_ADJ = {"español": "española", "trabajador": "trabajadora",
"hablador": "habladora", "encantador": "encantadora",
"alemán": "alemana", "francés": "francesa", "inglés": "inglesa"}
def inflect_adj(lemma, gender, number):
lemma = lemma.strip().lower()
d = _ADJS.get(lemma)
num = "SG" if number == "singular" else "PL"
if d:
form = d.get((gender, num))
if form:
return form, "lexicon"
# gender-invariant adjective (grande, feliz, azul): fem == masc.
# For a missing plural, pluralize this gender's singular form.
sg = d.get((gender, "SG")) or d.get(("m", "SG")) or lemma
if number == "plural":
return _regular_plural(sg), "rule"
return sg, "lexicon"
# rule fallback
a = lemma
if gender == "f":
if a in _INV_GENDER_ADJ:
a = _INV_GENDER_ADJ[a]
elif a.endswith("o"):
a = a[:-1] + "a"
if number == "plural":
a = _regular_plural(a)
return a, ("rule" if (a != lemma or gender == "m") else "rule")
# ── PUBLIC: clitic enclisis (dá + me + lo -> dámelo) ────────────────────────────
def _strip_accents(s):
return "".join(c for c in unicodedata.normalize("NFD", s)
if unicodedata.category(c) != "Mn")
def _count_syllables_vowelgroups(word):
# crude: count vowel groups
w = _strip_accents(word).lower()
groups, prev = 0, False
for ch in w:
isv = ch in "aeiou"
if isv and not prev:
groups += 1
prev = isv
return groups
def _host_stress_from_end(word):
"""Stressed-syllable index counted from the end (1=last) of a verb host."""
syls = _count_syllables_vowelgroups(word)
if any(c in "áéíóú" for c in word):
return None # already carries its own accent
if word[-2:] in ("ar", "er", "ir"): # infinitive: oxytone
return 1
if word.endswith("ndo"): # gerund: paroxytone
return 2
if word[-1:] in "aeiouns" and syls >= 2: # default paroxytone
return 2
return 1 # monosyllable / consonant-final oxytone
def attach_enclitics(verb_form, clitics):
"""Append clitic pronouns to a verb (imperative/infinitive/gerund enclisis)
and add a written accent when the resulting word becomes esdrújula/
sobreesdrújula (stress >= 3 syllables from the end): +me+lo -> dámelo,
lleva+me -> llévame, but dar+te -> darte and da+me -> dame (no accent)."""
if not clitics:
return verb_form
tail = "".join(clitics)
if any(c in "áéíóú" for c in verb_form): # host already accented
return verb_form + tail
sfe = _host_stress_from_end(verb_form)
total_sfe = sfe + len(clitics) # each clitic = 1 syllable
if total_sfe >= 3:
return _accentuate_nucleus(verb_form, sfe) + tail
return verb_form + tail
def _accentuate_nucleus(word, sfe):
"""Put a written accent on the syllable `sfe` positions from the word's end."""
vowels = "aeiou"
nuclei = [i for i, ch in enumerate(word) if ch in vowels]
if not nuclei or sfe > len(nuclei):
return word
i = nuclei[-sfe]
acc = {"a": "á", "e": "é", "i": "í", "o": "ó", "u": "ú"}
return word[:i] + acc[word[i]] + word[i + 1:]
def _accentuate_last_stressed(word):
# Restore the host's ORIGINAL lexical stress with a written accent.
# Default Spanish stress: word ending in vowel/n/s -> penultimate syllable;
# otherwise (e.g. infinitives in -r) -> last syllable.
vowels = "aeiou"
nuclei = [i for i, ch in enumerate(word) if ch in vowels]
if not nuclei:
return word
if word[-1] in "aeiouns" and len(nuclei) >= 2:
i = nuclei[-2] # paroxytone: penult nucleus
else:
i = nuclei[-1] # oxytone / monosyllable: last nucleus
acc = {"a": "á", "e": "é", "i": "í", "o": "ó", "u": "ú"}
return word[:i] + acc[word[i]] + word[i + 1:]
def lexicon_stats():
return {
"source": "UniMorph Spanish (github.com/unimorph/spa)",
"license": "CC-BY-SA 3.0 (Wiktionary-derived)",
"total_forms": sum(len(v) for v in (_VERBS, _NOUNS, _ADJS)) if False else None,
"verb_forms": len(_VERBS),
"verb_lemmas": len({k[0] for k in _VERBS}),
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
"participles": len(_PART),
"gerunds": len(_GER),
}
if __name__ == "__main__":
import json
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
tests = [
("hablar", "ind", "present", "first", "singular", "hablo"),
("comer", "ind", "present", "third", "plural", "comen"),
("vivir", "ind", "present", "first", "plural", "vivimos"),
("ser", "ind", "present", "third", "singular", "es"),
("ir", "ind", "preterite", "first", "singular", "fui"),
("tener", "ind", "future", "first", "singular", "tendré"),
("hacer", "sbjv", "present", "first", "singular", "haga"),
("dormir", "ind", "present", "first", "singular", "duermo"),
("pensar", "sbjv", "present", "third", "singular", "piense"),
("dar", "ind", "preterite", "third", "singular", "dio"),
("poner", "ind", "conditional", "first", "singular", "pondría"),
]
ok = 0
for lemma, mood, tense, per, num, exp in tests:
got, conf = conjugate(lemma, mood, tense, per, num)
flag = "OK " if got == exp else "XX "
if got == exp:
ok += 1
print(f" {flag}{lemma:8} {mood}/{tense} {per[:3]}.{num[:2]:3} -> {got:14} ({conf}) exp={exp}")
print(f"verb tests {ok}/{len(tests)}")
print(" gender casa:", noun_gender("casa"), "| problema:", noun_gender("problema"),
"| agua:", noun_gender("agua"), "| mano:", noun_gender("mano"))
print(" plural: luz->", inflect_noun("luz", "plural"), "| rey->", inflect_noun("rey", "plural"))
print(" adj: rojo/f/pl->", inflect_adj("rojo", "f", "plural"),
"| feliz/m/pl->", inflect_adj("feliz", "m", "plural"),
"| grande/f/pl->", inflect_adj("grande", "f", "plural"))
print(" enclisis: da+[me,lo]->", attach_enclitics("da", ["me", "lo"]),
"| di+[me]->", attach_enclitics("di", ["me"]),
"| dar+[se,lo]->", attach_enclitics("dar", ["se", "lo"]))
+629
View File
@@ -0,0 +1,629 @@
"""morphology_fr_full.py — production-grade French morphological generator.
Same architecture as morphology_it_full.py (shared Romance engine); French-specific
data and rules swapped in. Backed by three real, Wiktionary-lineage sources:
VERBS
UniMorph French (github.com/unimorph/fra, CC-BY-SA 3.0)
7,535 verb lemmas × full paradigm, CLEAN orthography:
indicatif présent / imparfait (PST;IPFV) / passé simple (PST;PFV) /
futur, conditionnel (COND), subjonctif présent (SBJV;PRS) /
subjonctif imparfait (SBJV;PST), impératif (POS;IMP), infinitif (NFIN),
participe présent (V.CVB/V.PTCP;PRS), participe passé (V.PTCP;PST, m.sg).
fr_irreg_verbs.json high-frequency verbs UniMorph MISSES or mis-slots,
above all ÊTRE (absent from UniMorph fra), plus avoir/aller/faire/ the
auxiliaries the passé-composé + être-agreement system depends on. Extracted
from kaikki.org French (build_fr_irreg.py), reflexive/multiword forms
dropped. This layer takes PRIORITY.
NOUNS + ADJECTIVES kaikki.org French (Wiktionary extract, CC-BY-SA 3.0)
noun lemmas WITH inherent gender (head-template arg) + real plural
(cheval->chevaux, œil->yeux, invariable -s/-x/-z), resolved PER LEMMA.
adjective lemmas with real feminine + plural (petit->petite/petits/petites,
beau->belle/beaux/belles, heureux->heureuse, rouge invariant-gender).
Fallbacks (degrade, never crash, on OOV input):
verbs : rule generator for -er / -ir(-iss-) / -re (with -cer/-ger spelling,
future/conditional stems, imparfait/subjonctif endings)
nouns : gender heuristic (endings) + rule pluralization (-al->-aux, -eau->-eaux)
adjs : fem/plural agreement rules (-er->-ère, -eux->-euse, -f->-ve, +e default)
Confidence flag on every form: "lexicon" | "rule" | "fallback".
Public API (used by realizer_fr.py): identical signature to morphology_it_full.
"""
import json
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "fra.unimorph")
_IRREG = os.path.join(_HERE, "data", "fr_irreg_verbs.json")
_KAIKKI = os.path.join(_HERE, "data", "kaikki_fr.jsonl")
_CACHE = os.path.join(_HERE, "data", "fr_morph_cache.pkl")
# ── (mood, tense) -> UniMorph feature set that must ALL be present ────────────────
_VERB_KEYMAP = {
("ind", "present"): {"IND", "PRS"},
("ind", "imperfect"): {"IND", "PST", "IPFV"}, # imparfait
("ind", "passe_simple"): {"IND", "PST", "PFV"}, # passé simple
("ind", "future"): {"IND", "FUT"},
("ind", "conditional"): {"COND"}, # French: V;COND;1;SG
("sbjv", "present"): {"SBJV", "PRS"},
("sbjv", "imperfect"): {"SBJV", "PST"},
("imp", "affirmative"): {"POS", "IMP"},
}
_PERSON = {"first": "1", "second": "2", "third": "3"}
_NUMBER = {"singular": "SG", "plural": "PL"}
def _feat_set(tag):
return set(tag.split(";"))
# ── build verb lexicon from UniMorph ─────────────────────────────────────────────
def _build_verbs():
verbs = {}
part = {}
ger = {}
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
f = _feat_set(tag)
head = tag.split(";")[0]
if head == "V.PTCP":
if "PST" in f:
part.setdefault(lemma, form)
elif "PRS" in f:
ger.setdefault(lemma, form)
continue
if head == "V.CVB":
if "PRS" in f:
ger.setdefault(lemma, form)
continue
if head != "V":
continue
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person is None or number is None:
continue
for (mood, tense), req in _VERB_KEYMAP.items():
if not req <= f:
continue
if tense == "imperfect" and "PFV" in f:
continue
if tense == "passe_simple" and "IPFV" in f:
continue
verbs.setdefault((lemma, f"{mood}|{tense}|{person}|{number}"), form)
break
return verbs, part, ger
# ── kaikki nouns + adjectives ────────────────────────────────────────────────────
_EXCL_FORM_TAGS = {"alternative", "archaic", "obsolete", "dialectal", "regional",
"diminutive", "augmentative", "pejorative", "comparative",
"superlative", "misspelling", "rare", "informal", "literary",
"poetic", "error-unrecognized-form", "construed", "collective",
"nonstandard", "dated", "Louisiana", "Switzerland", "Belgium"}
def _kaikki_gender(arg):
if not arg:
return None
a = str(arg).lower()
if a.startswith("f"):
return "f"
if a.startswith("m"):
return "m"
return None
def _build_nouns_adjs():
nouns = {}
adjs = {}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
pos = d.get("pos")
word = d.get("word", "")
if not word or " " in word:
continue
forms = d.get("forms", []) or []
if pos == "noun":
ht = d.get("head_templates") or []
g = None
if ht:
g = _kaikki_gender((ht[0].get("args") or {}).get("1"))
if g is None:
tags = d.get("tags") or []
if "feminine" in tags:
g = "f"
elif "masculine" in tags:
g = "m"
pl = None
for x in forms:
t = set(x.get("tags") or [])
if "plural" in t and not (t & _EXCL_FORM_TAGS):
fm = x.get("form")
if fm and " " not in fm and fm not in ("#", "-", ""):
pl = fm
break
if word not in nouns:
nouns[word] = {"g": g, "SG": word, "PL": pl}
else:
cur = nouns[word]
if cur.get("g") is None and g:
cur["g"] = g
if not cur.get("PL") and pl:
cur["PL"] = pl
elif pos == "adj":
d0 = adjs.setdefault(word, {})
d0.setdefault(("m", "SG"), word)
for x in forms:
t = set(x.get("tags") or [])
fm = x.get("form")
if not fm or " " in fm or (t & _EXCL_FORM_TAGS):
continue
if "feminine" in t and "plural" in t:
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
elif "masculine" in t and "plural" in t:
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
elif "feminine" in t:
d0[("f", "SG")] = d0.get(("f", "SG")) or fm
elif "plural" in t:
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
return nouns, adjs
def _build_cache():
verbs, part, ger = _build_verbs()
nouns, adjs = _build_nouns_adjs()
with open(_IRREG, encoding="utf-8") as fh:
irreg = json.load(fh)
data = {"verbs": verbs, "part": part, "ger": ger,
"nouns": nouns, "adjs": adjs, "irreg": irreg}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE):
srcs = [_UNIMORPH, _KAIKKI, _IRREG]
newest = max(os.path.getmtime(s) for s in srcs if os.path.exists(s))
if os.path.getmtime(_CACHE) >= newest:
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_VERBS, _PART, _GER, _NOUNS, _ADJS, _IRREGV = (
_LEX["verbs"], _LEX["part"], _LEX["ger"], _LEX["nouns"], _LEX["adjs"],
_LEX["irreg"])
# ── regular-ending rule fallback ─────────────────────────────────────────────────
def _vclass(lemma):
if lemma.endswith("er"):
return "er"
if lemma.endswith("ir"):
return "ir"
if lemma.endswith("re"):
return "re"
if lemma.endswith("oir"):
return "oir"
return None
# present-tense endings [1sg,2sg,3sg,1pl,2pl,3pl]
_REG_PRES = {
"er": ["e", "es", "e", "ons", "ez", "ent"],
"ir": ["is", "is", "it", "issons", "issez", "issent"], # -iss- class (finir)
"re": ["s", "s", "", "ons", "ez", "ent"], # vendre: vends/vend
}
_REG_IMPF = ["ais", "ais", "ait", "ions", "iez", "aient"] # attaches to pres-1pl stem
_REG_SUBJ = ["e", "es", "e", "ions", "iez", "ent"] # attaches to 3pl stem
_REG_PS = { # passé simple
"er": ["ai", "as", "a", "âmes", "âtes", "èrent"],
"ir": ["is", "is", "it", "îmes", "îtes", "irent"],
"re": ["is", "is", "it", "îmes", "îtes", "irent"],
}
_FUT = ["ai", "as", "a", "ons", "ez", "ont"]
_COND = ["ais", "ais", "ait", "ions", "iez", "aient"]
def _slot_idx(person, number):
base = {"first": 0, "second": 1, "third": 2}[person]
return base + (0 if number == "singular" else 3)
def _fut_stem(lemma, vc):
"""Future/conditional stem = infinitive (drop final -e of -re)."""
if vc == "re":
return lemma[:-1] # vendre -> vendr-
return lemma # parler-, finir-
def _pres_1pl_stem(lemma, vc):
"""Imparfait stem = present 1pl minus -ons (parlons->parl-, finissons->finiss-)."""
if vc == "er":
stem = lemma[:-2]
if stem.endswith("g"):
return stem + "e" # mangeons -> mange- (imparfait mangeais)
if stem.endswith("c"):
return stem[:-1] + "ç" # commençons -> commenç-
return stem
if vc == "ir":
return lemma[:-1] + "iss" # finir -> finiss-
if vc == "re":
return lemma[:-2] # vendre -> vend-
return lemma[:-2]
def _apply_er_spelling(stem, ending):
"""-cer/-ger softening before a/o (commençons, mangeons)."""
if ending and ending[0] in ("a", "o"):
if stem.endswith("c"):
return stem[:-1] + "ç" + ending
if stem.endswith("g"):
return stem + "e" + ending
return stem + ending
def _rule_conjugate(lemma, mood, tense, person, number):
vc = _vclass(lemma)
if vc is None:
return None
i = _slot_idx(person, number)
if mood == "ind" and tense in ("future", "conditional"):
stem = _fut_stem(lemma, vc)
end = (_FUT if tense == "future" else _COND)[i]
return stem + end
if mood == "ind" and tense == "present":
table = _REG_PRES.get("ir" if vc == "ir" else vc)
if not table:
return None
body = lemma[:-2] if vc in ("er", "re") else lemma[:-1] if vc == "ir" else lemma[:-2]
if vc == "ir":
body = lemma[:-2] # fin- ; endings carry -iss-
end = table[i]
return body + end
end = table[i]
if vc == "er":
return _apply_er_spelling(body, end)
return body + end
if mood == "ind" and tense == "imperfect":
stem = _pres_1pl_stem(lemma, vc)
return stem + _REG_IMPF[i]
if mood == "ind" and tense == "passe_simple":
table = _REG_PS.get("ir" if vc == "ir" else vc)
if not table:
return None
body = lemma[:-2] if vc in ("er", "re") else lemma[:-2]
end = table[i]
if vc == "er":
return _apply_er_spelling(body, end)
return body + end
if mood == "sbjv" and tense == "present":
# subjonctif: present-3pl stem + e/es/e/ions/iez/ent
stem3 = _pres_1pl_stem(lemma, vc) if vc == "ir" else (
lemma[:-2] if vc in ("er", "re") else lemma[:-2])
if vc == "ir":
stem3 = lemma[:-2] + "iss"
end = _REG_SUBJ[i]
if vc == "er":
return _apply_er_spelling(stem3, end)
return stem3 + end
if mood == "imp" and tense == "affirmative":
# impératif ~ present indicative (tu drops -s for -er verbs)
pres = _rule_conjugate(lemma, "ind", "present", person, number)
if pres and vc == "er" and person == "second" and number == "singular":
return pres[:-1] if pres.endswith("es") else pres
return pres
return None
# ── PUBLIC: verb conjugation ─────────────────────────────────────────────────────
def conjugate(lemma, mood, tense, person, number):
"""Return (surface, confidence)."""
lemma = lemma.strip().lower()
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{number}"
ir = _IRREGV.get(lemma)
if ir and key in ir:
return ir[key], "lexicon"
p, n = _PERSON.get(person), _NUMBER.get(number)
if p and n:
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}"))
if form:
return form, "lexicon"
r = _rule_conjugate(lemma, mood, tense, person, number)
if r:
return r, "rule"
return lemma, "fallback"
# ── PUBLIC: participle + gerund/participe présent ────────────────────────────────
def _participle_msg(lemma):
ir = _IRREGV.get(lemma)
if ir and "part" in ir:
return ir["part"], "lexicon"
if lemma in _PART:
return _PART[lemma], "lexicon"
return None, None
# irregular participle fem/plural quirks (drop circonflexe: dû->due, dus)
_PART_FIX = {"": {"f|SG": "due", "m|PL": "dus", "f|PL": "dues"}}
def participle(lemma, gender="m", number="singular"):
"""Past participle with French gender/number agreement.
m.sg = base; f.sg = base+e; m.pl = base+s (invariable if base ends s/x);
f.pl = f.sg+s."""
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
msg, src = _participle_msg(lemma)
conf = "lexicon"
if msg is None:
vc = _vclass(lemma)
if vc == "er":
msg = lemma[:-2] + "é"
elif vc == "ir":
msg = lemma[:-1] # finir -> fini, partir -> parti
elif vc == "re":
msg = lemma[:-2] + "u" # vendre -> vendu
elif vc == "oir":
msg = lemma[:-3] + "u" # (rough) recevoir handled by irreg
else:
return lemma, "fallback"
conf = "rule"
fix = _PART_FIX.get(msg)
if fix and f"{g}|{num}" in fix:
return fix[f"{g}|{num}"], conf
if g == "m" and num == "SG":
return msg, conf
fem = msg + "e" if not msg.endswith("e") else msg
if g == "f" and num == "SG":
return fem, conf
if g == "m" and num == "PL":
return msg if msg.endswith(("s", "x")) else msg + "s", conf
# f|PL
return fem + "s", conf
def gerund(lemma):
"""Participe présent (base for gérondif 'en -ant')."""
lemma = lemma.strip().lower()
ir = _IRREGV.get(lemma)
if ir and "ger" in ir:
return ir["ger"], "lexicon"
if lemma in _GER:
return _GER[lemma], "lexicon"
vc = _vclass(lemma)
if vc == "er":
stem = lemma[:-2]
if stem.endswith("g"):
return stem + "eant", "rule"
if stem.endswith("c"):
return stem[:-1] + "çant", "rule"
return stem + "ant", "rule"
if vc == "ir":
return lemma[:-2] + "issant", "rule"
if vc == "re":
return lemma[:-2] + "ant", "rule"
return lemma, "fallback"
# ── PUBLIC: noun gender + number ─────────────────────────────────────────────────
_FEM_SUF = ("tion", "sion", "aison", "ance", "ence", "ette", "elle", "esse",
"ude", "ade", "ée", "", "tié", "ie", "ise", "ure", "eur")
_MASC_SUF = ("ment", "age", "eau", "isme", "oir", "ier", "eur", "in", "on")
def _gender_heuristic(noun):
for suf in _FEM_SUF:
if noun.endswith(suf):
return "f"
for suf in _MASC_SUF:
if noun.endswith(suf):
return "m"
if noun.endswith("e"):
return "f"
return "m"
def noun_gender(lemma):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if d and d.get("g") in ("m", "f"):
return d["g"]
return _gender_heuristic(lemma)
# closed sets for French plural irregularities
_OU_X = {"bijou", "caillou", "chou", "genou", "hibou", "joujou", "pou"}
_AIL_AUX = {"travail", "vitrail", "corail", "émail", "bail", "soupirail", "vantail"}
_AL_S = {"bal", "carnaval", "festival", "récital", "chacal", "régal", "cal", "aval"}
def _rule_plural(noun, gender):
"""Deterministic French pluralization. (form, ok); ok=False FLAGS ambiguity."""
if not noun:
return noun, True
if noun[-1:] in ("s", "x", "z"):
return noun, True # invariable
if noun in _OU_X:
return noun + "x", True
if noun.endswith(("eau", "au", "eu")):
if noun in ("pneu", "bleu", "landau", "sarrau"):
return noun + "s", True
return noun + "x", True # bateau->bateaux, jeu->jeux
if noun.endswith("al"):
if noun in _AL_S:
return noun + "s", True
return noun[:-2] + "aux", True # cheval->chevaux
if noun.endswith("ail"):
if noun in _AIL_AUX:
return noun[:-3] + "aux", True # travail->travaux
return noun + "s", True
return noun + "s", True # default
def inflect_noun(lemma, number, gender=None):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if number == "singular":
return (d["SG"] if d and d.get("SG") else lemma), ("lexicon" if d else "rule")
if d and d.get("PL"):
return d["PL"], "lexicon"
g = gender or noun_gender(lemma)
form, ok = _rule_plural(lemma, g)
return form, ("rule" if ok else "fallback")
# adjectives whose kaikki entries are unreliable: audited forms
_ADJ_FIX = {
"beau": {("m", "SG"): "beau", ("f", "SG"): "belle",
("m", "PL"): "beaux", ("f", "PL"): "belles"},
"nouveau": {("m", "SG"): "nouveau", ("f", "SG"): "nouvelle",
("m", "PL"): "nouveaux", ("f", "PL"): "nouvelles"},
"vieux": {("m", "SG"): "vieux", ("f", "SG"): "vieille",
("m", "PL"): "vieux", ("f", "PL"): "vieilles"},
"fou": {("m", "SG"): "fou", ("f", "SG"): "folle",
("m", "PL"): "fous", ("f", "PL"): "folles"},
"blanc": {("m", "SG"): "blanc", ("f", "SG"): "blanche",
("m", "PL"): "blancs", ("f", "PL"): "blanches"},
"long": {("m", "SG"): "long", ("f", "SG"): "longue",
("m", "PL"): "longs", ("f", "PL"): "longues"},
"bon": {("m", "SG"): "bon", ("f", "SG"): "bonne",
("m", "PL"): "bons", ("f", "PL"): "bonnes"},
}
def _rule_fem(a):
if a.endswith("e"):
return a
if a.endswith("er"):
return a[:-2] + "ère"
if a.endswith("eau"):
return a[:-3] + "elle"
if a.endswith("eux"):
return a[:-3] + "euse"
if a.endswith("f"):
return a[:-1] + "ve"
if a.endswith(("on", "en", "el", "eil", "et")):
return a + a[-1] + "e" # bon->bonne, ancien->ancienne, muet->muette
if a.endswith("c"):
return a[:-1] + "che" # blanc->blanche (public->publique via FIX)
return a + "e" # grand->grande, petit->petite, vert->verte
def inflect_adj(lemma, gender, number):
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
fix = _ADJ_FIX.get(lemma)
if fix and (g, num) in fix:
return fix[(g, num)], "lexicon"
d = _ADJS.get(lemma)
if d and d.get((g, num)):
return d[(g, num)], "lexicon"
# derive
msc = (d.get(("m", "SG")) if d else None) or lemma
if g == "m" and num == "SG":
return msc, "lexicon" if d else "rule"
fem = (d.get(("f", "SG")) if d else None) or _rule_fem(msc)
if g == "f" and num == "SG":
return fem, "lexicon" if (d and d.get(("f", "SG"))) else "rule"
if g == "m" and num == "PL":
if msc.endswith(("s", "x")):
return msc, "rule"
if msc.endswith("al"):
return msc[:-2] + "aux", "rule"
if msc.endswith("eau"):
return msc + "x", "rule"
return msc + "s", "rule"
# f|PL
return (fem if fem.endswith("s") else fem + "s"), "rule"
def lexicon_stats():
return {
"verb_source": "UniMorph French (github.com/unimorph/fra) + kaikki.org "
"irregulars (être + high-frequency)",
"noun_adj_source": "kaikki.org French (Wiktionary extract)",
"license": "CC-BY-SA 3.0 (Wiktionary/UniMorph lineage)",
"unimorph_verb_forms": len(_VERBS),
"unimorph_verb_lemmas": len({k[0] for k in _VERBS}),
"irregular_verb_lemmas": len(_IRREGV),
"participle_lemmas": len(_PART),
"gerund_lemmas": len(_GER),
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
}
if __name__ == "__main__":
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
tests = [
("parler", "ind", "present", "first", "singular", "parle"),
("être", "ind", "present", "third", "singular", "est"),
("avoir", "ind", "present", "first", "singular", "ai"),
("aller", "ind", "present", "third", "plural", "vont"),
("finir", "ind", "present", "first", "singular", "finis"),
("finir", "ind", "present", "first", "plural", "finissons"),
("manger", "ind", "present", "first", "plural", "mangeons"),
("faire", "ind", "future", "first", "singular", "ferai"),
("pouvoir", "sbjv", "present", "third", "singular", "puisse"),
("prendre", "ind", "passe_simple", "third", "singular", "prit"),
("vendre", "ind", "present", "third", "singular", "vend"),
("commencer", "ind", "imperfect", "first", "singular", "commençais"),
]
ok = 0
for lemma, mood, tense, per, num, exp in tests:
got, conf = conjugate(lemma, mood, tense, per, num)
flag = "OK " if got == exp else "XX "
ok += got == exp
print(f" {flag}{lemma:10} {mood}/{tense:12} {per[:3]}.{num[:2]} -> {got:12} ({conf}) exp={exp}")
print(f"verb tests {ok}/{len(tests)}")
print(" gender: maison=", noun_gender("maison"), "chat=", noun_gender("chat"),
"cheval=", noun_gender("cheval"), "nation=", noun_gender("nation"))
print(" plural: cheval->", inflect_noun("cheval", "plural"),
"| bateau->", inflect_noun("bateau", "plural"),
"| prix->", inflect_noun("prix", "plural"),
"| chat->", inflect_noun("chat", "plural"))
print(" adj: petit/f/sg->", inflect_adj("petit", "f", "singular"),
"| beau/f/sg->", inflect_adj("beau", "f", "singular"),
"| heureux/f/sg->", inflect_adj("heureux", "f", "singular"),
"| national/m/pl->", inflect_adj("national", "m", "plural"))
print(" part: aller/f/sg->", participle("aller", "f", "singular"),
"| prendre/f/pl->", participle("prendre", "f", "plural"),
"| finir/m/pl->", participle("finir", "m", "plural"))
print(" ger: manger->", gerund("manger"), "| finir->", gerund("finir"))
+588
View File
@@ -0,0 +1,588 @@
"""morphology_it_full.py — production-grade Italian morphological generator.
NOT a toy. Backed by three real, Wiktionary-lineage lexical sources:
VERBS
UniMorph Italian (github.com/unimorph/ita, CC-BY-SA 3.0)
10,009 verb lemmas × full paradigm, CLEAN orthography (no stress marks):
indicative present / imperfetto (PST;IPFV) / passato remoto (PST;PFV) /
futuro, condizionale (COND),
congiuntivo presente (SBJV;PRS) / imperfetto (SBJV;PST),
affirmative imperative, infinitive, gerundio (V.CVB;PRS),
past participle (masc-sg; fem/plural derived by vowel rule).
it_irreg_verbs.json 66 high-frequency verbs UniMorph MISSES
(essere, avere, potere, uscire, tenere, prendere, piacere, ), extracted
from kaikki.org Italian, filtered to standard forms, and DE-STRESSED to
real orthography (kaikki marks tonic stress everywhere: pàrlo->parlo,
avùto->avuto; final legit accents kept: sarò, è). Built by build_it_irreg.py.
This layer takes priority it supplies the two auxiliaries essere/avere,
which the whole passato-prossimo / essere-agreement system depends on.
NOUNS + ADJECTIVES kaikki.org Italian (Wiktionary extract, CC-BY-SA 3.0)
noun lemmas WITH inherent gender (head-template arg) + real (often irregular)
plural uomo->uomini, uovo->uova, dito->dita, città invariant resolved
PER LEMMA, never guessed.
adjective lemmas with real feminine + masc/fem plural (italiano->italiana/
italiani/italiane, felice->felici invariant).
Fallbacks (degrade, never crash, on OOV input):
verbs : rule generator for regular -are/-ere/-ire (with -care/-gare h-insertion
and -ciare/-giare/-iare i-drop spelling rules)
nouns : gender heuristic (endings) + rule pluralization (ambiguous -co/-go FLAGGED)
adjs : -o/-a/-e gender rule + rule pluralization
Confidence flag on every form:
"lexicon" from UniMorph / kaikki-irregular / kaikki noun-adj (trust: high)
"rule" deterministic rule (trust: medium)
"fallback" could not inflect; returned lemma / ambiguous (trust: low -> FLAG)
Public API (used by realizer_it.py):
conjugate(lemma, mood, tense, person, number) -> (form, conf)
participle(lemma, gender="m", number="singular") -> (form, conf)
gerund(lemma) -> (form, conf)
noun_gender(lemma) -> "m"|"f"
inflect_noun(lemma, number, gender=None) -> (form, conf)
inflect_adj(lemma, gender, number) -> (form, conf)
lexicon_stats() -> dict
"""
import json
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "ita.unimorph")
_IRREG = os.path.join(_HERE, "data", "it_irreg_verbs.json")
_KAIKKI = os.path.join(_HERE, "data", "kaikki_it.jsonl")
_CACHE = os.path.join(_HERE, "data", "it_morph_cache.pkl")
# ── (mood, tense) -> UniMorph feature set that must ALL be present ────────────────
_VERB_KEYMAP = {
("ind", "present"): {"IND", "PRS"},
("ind", "imperfect"): {"IND", "PST", "IPFV"},
("ind", "passato_remoto"): {"IND", "PST", "PFV"},
("ind", "future"): {"IND", "FUT"},
("ind", "conditional"): {"COND"},
("sbjv", "present"): {"SBJV", "PRS"},
("sbjv", "imperfect"): {"SBJV", "PST"},
("imp", "affirmative"): {"POS", "IMP"},
}
_PERSON = {"first": "1", "second": "2", "third": "3"}
_NUMBER = {"singular": "SG", "plural": "PL"}
def _feat_set(tag):
return set(tag.split(";"))
# ── build verb lexicon from UniMorph ─────────────────────────────────────────────
def _build_verbs():
verbs = {} # (lemma, "mood|tense|person|number") -> form
part = {} # lemma -> masc-sg past participle
ger = {} # lemma -> gerundio
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
f = _feat_set(tag)
head = tag.split(";")[0]
if head == "V.PTCP":
if "PST" in f:
part.setdefault(lemma, form)
continue
if head == "V.CVB": # gerundio (converb, present)
if "PRS" in f:
ger.setdefault(lemma, form)
continue
if head != "V":
continue
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person is None or number is None:
continue
for (mood, tense), req in _VERB_KEYMAP.items():
# exact-set discipline: PST;PFV must not match PST;IPFV, etc.
if not req <= f:
continue
# guard IND;PST ambiguity: require the specific aspect feature
if tense == "imperfect" and "PFV" in f:
continue
if tense == "passato_remoto" and "IPFV" in f:
continue
# COND must not also be a subjunctive/imperative slot
verbs.setdefault((lemma, f"{mood}|{tense}|{person}|{number}"), form)
break
return verbs, part, ger
# ── kaikki nouns + adjectives ────────────────────────────────────────────────────
_EXCL_FORM_TAGS = {"alternative", "archaic", "obsolete", "dialectal", "regional",
"diminutive", "augmentative", "pejorative", "comparative",
"superlative", "misspelling", "rare", "informal", "literary",
"poetic", "error-unrecognized-form", "apocopic", "obsolete",
"construed", "collective"}
def _kaikki_gender(arg):
if not arg:
return None
a = str(arg).lower()
if a.startswith("f"):
return "f"
if a.startswith("m"):
return "m"
return None
def _build_nouns_adjs():
nouns = {} # lemma -> {"g","SG","PL"}
adjs = {} # lemma -> {("m","SG"),("f","SG"),("m","PL"),("f","PL")}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
pos = d.get("pos")
word = d.get("word", "")
if not word or " " in word:
continue
forms = d.get("forms", []) or []
if pos == "noun":
ht = d.get("head_templates") or []
g = None
if ht:
g = _kaikki_gender((ht[0].get("args") or {}).get("1"))
if g is None:
tags = d.get("tags") or []
if "feminine" in tags:
g = "f"
elif "masculine" in tags:
g = "m"
pl = None
for x in forms:
t = set(x.get("tags") or [])
if "plural" in t and not (t & _EXCL_FORM_TAGS):
fm = x.get("form")
if fm and " " not in fm and fm != "#":
pl = fm
break
if word not in nouns:
nouns[word] = {"g": g, "SG": word, "PL": pl}
else:
cur = nouns[word]
if cur.get("g") is None and g:
cur["g"] = g
if not cur.get("PL") and pl:
cur["PL"] = pl
elif pos == "adj":
d0 = adjs.setdefault(word, {})
d0.setdefault(("m", "SG"), word)
for x in forms:
t = set(x.get("tags") or [])
fm = x.get("form")
if not fm or " " in fm or (t & _EXCL_FORM_TAGS):
continue
if "feminine" in t and "plural" in t:
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
elif "masculine" in t and "plural" in t:
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
elif "feminine" in t:
d0[("f", "SG")] = d0.get(("f", "SG")) or fm
elif "plural" in t: # invariant-gender adj (felice -> felici)
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
return nouns, adjs
def _build_cache():
verbs, part, ger = _build_verbs()
nouns, adjs = _build_nouns_adjs()
with open(_IRREG, encoding="utf-8") as fh:
irreg = json.load(fh)
data = {"verbs": verbs, "part": part, "ger": ger,
"nouns": nouns, "adjs": adjs, "irreg": irreg}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE):
srcs = [_UNIMORPH, _KAIKKI, _IRREG]
newest = max(os.path.getmtime(s) for s in srcs if os.path.exists(s))
if os.path.getmtime(_CACHE) >= newest:
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_VERBS, _PART, _GER, _NOUNS, _ADJS, _IRREGV = (
_LEX["verbs"], _LEX["part"], _LEX["ger"], _LEX["nouns"], _LEX["adjs"],
_LEX["irreg"])
# ── regular-ending rule fallback ─────────────────────────────────────────────────
def _vclass(lemma):
if lemma.endswith("are"):
return "are"
if lemma.endswith("ere"):
return "ere"
if lemma.endswith("ire"):
return "ire"
return None
# endings [1sg,2sg,3sg,1pl,2pl,3pl]
_REG = {
("ind", "present", "are"): ["o", "i", "a", "iamo", "ate", "ano"],
("ind", "present", "ere"): ["o", "i", "e", "iamo", "ete", "ono"],
("ind", "present", "ire"): ["o", "i", "e", "iamo", "ite", "ono"],
("ind", "imperfect", "are"): ["avo", "avi", "ava", "avamo", "avate", "avano"],
("ind", "imperfect", "ere"): ["evo", "evi", "eva", "evamo", "evate", "evano"],
("ind", "imperfect", "ire"): ["ivo", "ivi", "iva", "ivamo", "ivate", "ivano"],
("ind", "passato_remoto", "are"): ["ai", "asti", "ò", "ammo", "aste", "arono"],
("ind", "passato_remoto", "ere"): ["ei", "esti", "é", "emmo", "este", "erono"],
("ind", "passato_remoto", "ire"): ["ii", "isti", "ì", "immo", "iste", "irono"],
("sbjv", "present", "are"): ["i", "i", "i", "iamo", "iate", "ino"],
("sbjv", "present", "ere"): ["a", "a", "a", "iamo", "iate", "ano"],
("sbjv", "present", "ire"): ["a", "a", "a", "iamo", "iate", "ano"],
("sbjv", "imperfect", "are"): ["assi", "assi", "asse", "assimo", "aste", "assero"],
("sbjv", "imperfect", "ere"): ["essi", "essi", "esse", "essimo", "este", "essero"],
("sbjv", "imperfect", "ire"): ["issi", "issi", "isse", "issimo", "iste", "issero"],
# imperative: 2sg,3sg(Lei),1pl,2pl,3pl (1sg has none)
("imp", "affirmative", "are"): [None, "a", "i", "iamo", "ate", "ino"],
("imp", "affirmative", "ere"): [None, "i", "a", "iamo", "ete", "ano"],
("imp", "affirmative", "ire"): [None, "i", "a", "iamo", "ite", "ano"],
}
# future / conditional attach to a stem = infinitive minus final -e, with
# -are -> -er (parlare->parler-), -ere/-ire keep (credere->creder-, dormir-)
_FUT = ["ò", "ai", "à", "emo", "ete", "anno"]
_COND = ["ei", "esti", "ebbe", "emmo", "este", "ebbero"]
def _slot_idx(person, number):
base = {"first": 0, "second": 1, "third": 2}[person]
return base + (0 if number == "singular" else 3)
def _fut_stem(lemma, vc):
body = lemma[:-3] # drop are/ere/ire
if vc == "are":
return body + "er"
return body + vc[0] + "r" # ere->er? no: keep vowel: creder-, dormir-
# NOTE corrected below
def _apply_are_spelling(stem, ending):
"""-care/-gare insert h before front endings; -ciare/-giare/-sciare/-iare drop i."""
front = ending[:1] in ("i", "e")
if stem.endswith(("c", "g")) and front:
return stem + "h" + ending
if stem.endswith(("ci", "gi", "sci")) and ending[:1] == "i":
return stem[:-1] + ending # mangi+iamo -> mangiamo
if stem.endswith("i") and ending[:1] == "i":
return stem[:-1] + ending # studi+iamo -> studiamo
return stem + ending
def _rule_conjugate(lemma, mood, tense, person, number):
vc = _vclass(lemma)
if vc is None:
return None
body = lemma[:-3]
i = _slot_idx(person, number)
if mood == "ind" and tense in ("future", "conditional"):
stem = body + "er" if vc == "are" else body + vc[0] + "r"
# ere: creder-, ire: dormir- -> body + 'e'/'i' + 'r'
if vc == "ere":
stem = body + "er"
elif vc == "ire":
stem = body + "ir"
end = (_FUT if tense == "future" else _COND)[i]
# spelling: -care/-gare -> cherò/gherò ; -ciare/-giare -> cerò/gerò
if vc == "are":
if body.endswith(("c", "g")):
stem = body + "her"
elif body.endswith(("ci", "gi", "sci")):
stem = body[:-1] + "er"
elif body.endswith("i"):
stem = body[:-1] + "er"
return stem + end
table = _REG.get((mood, tense, vc))
if not table:
return None
end = table[i]
if end is None:
return None
if vc == "are":
return _apply_are_spelling(body, end)
# -ere/-ire: guard against double-i (dormi+iamo -> dormiamo)
if body.endswith("i") and end[:1] == "i":
return body[:-1] + end
return body + end
# ── PUBLIC: verb conjugation ─────────────────────────────────────────────────────
def conjugate(lemma, mood, tense, person, number):
"""Return (surface, confidence). mood in ind|sbjv|imp; tense per _VERB_KEYMAP."""
lemma = lemma.strip().lower()
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{number}"
ir = _IRREGV.get(lemma)
if ir and key in ir:
return ir[key], "lexicon"
p, n = _PERSON.get(person), _NUMBER.get(number)
if p and n:
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}"))
if form:
return form, "lexicon"
r = _rule_conjugate(lemma, mood, tense, person, number)
if r:
return r, "rule"
return lemma, "fallback"
# ── PUBLIC: participle + gerund ──────────────────────────────────────────────────
def _participle_msg(lemma):
"""Return (masc-sg participle, source) or (None, None)."""
ir = _IRREGV.get(lemma)
if ir and "part" in ir:
return ir["part"], "lexicon"
if lemma in _PART:
return _PART[lemma], "lexicon"
return None, None
def participle(lemma, gender="m", number="singular"):
"""Past participle with gender/number agreement (for essere-perfect & passives).
UniMorph/irregular give masc-sg; fem/plural derived by final-vowel swap
(-o -> -a/-i/-e), valid for regular -ato/-uto/-ito AND irregulars
(preso->presa/presi/prese, aperto->aperta/aperti/aperte, morto->morta/...)."""
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
msg, src = _participle_msg(lemma)
conf = "lexicon"
if msg is None:
vc = _vclass(lemma)
if vc == "are":
msg = lemma[:-3] + "ato"
elif vc == "ere":
msg = lemma[:-3] + "uto"
elif vc == "ire":
msg = lemma[:-3] + "ito"
else:
return lemma, "fallback"
conf = "rule"
# agreement: only -o participles inflect for gender+number
if msg.endswith("o"):
stem = msg[:-1]
suf = {"m|SG": "o", "f|SG": "a", "m|PL": "i", "f|PL": "e"}[f"{g}|{num}"]
return stem + suf, conf
return msg, conf # non -o participle: leave as-is (rare)
def gerund(lemma):
lemma = lemma.strip().lower()
ir = _IRREGV.get(lemma)
if ir and "ger" in ir:
return ir["ger"], "lexicon"
if lemma in _GER:
return _GER[lemma], "lexicon"
vc = _vclass(lemma)
if vc == "are":
return lemma[:-3] + "ando", "rule"
if vc in ("ere", "ire"):
return lemma[:-3] + "endo", "rule"
return lemma, "fallback"
# ── PUBLIC: noun gender + number ─────────────────────────────────────────────────
_FEM_SUF = ("zione", "sione", "gione", "", "", "trice", "aggine", "udine",
"igine", "ie", "essa", "izia", "ezza")
_MASC_SUF = ("ore", "ame", "iere", "ale", "ile")
def _gender_heuristic(noun):
for suf in _FEM_SUF:
if noun.endswith(suf):
return "f"
for suf in _MASC_SUF:
if noun.endswith(suf):
return "m"
if noun.endswith("o"):
return "m"
if noun.endswith("a"):
return "f"
if noun.endswith("à") or noun.endswith("ù"):
return "f"
return "m" # -e and consonant-final loanwords default masculine
def noun_gender(lemma):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if d and d.get("g") in ("m", "f"):
return d["g"]
return _gender_heuristic(lemma)
def _rule_plural(noun, gender):
"""Deterministic Italian pluralization. Returns (form, ok); ok=False FLAGS an
ambiguous case the lexicon would normally resolve (-co/-go palatalization)."""
if not noun:
return noun, True
# invariant: accented final vowel, consonant-final, monosyllable, -i final
if noun[-1:] in ("à", "è", "é", "ì", "í", "ò", "ó", "ù", "ú"):
return noun, True
if noun[-1:] not in ("a", "e", "o", "i", "u"):
return noun, True # consonant-final loanword: invariant
if noun.endswith("i"):
return noun, True # e.g. crisi, analisi: invariant
if noun.endswith("io"):
return noun[:-2] + "i", True # figlio->figli (unstressed i)
if noun.endswith("cia") or noun.endswith("gia"):
# vowel before cia/gia -> -cie/-gie ; consonant -> -ce/-ge (approx)
return noun[:-2] + "e", True # arancia->arance (majority)
if noun.endswith("ca"):
return noun[:-2] + "che", True # amica->amiche
if noun.endswith("ga"):
return noun[:-2] + "ghe", True
if noun.endswith("co"):
return noun[:-2] + "chi", False # AMBIGUOUS (amico->amici) -> flag
if noun.endswith("go"):
return noun[:-2] + "ghi", False # AMBIGUOUS (psicologo->psicologi)
if noun.endswith("a"):
return noun[:-1] + "e", True # casa->case (m -a: -i, but rare)
if noun.endswith("o"):
return noun[:-1] + "i", True # libro->libri
if noun.endswith("e"):
return noun[:-1] + "i", True # cane->cani, chiave->chiavi
return noun, True
def inflect_noun(lemma, number, gender=None):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if number == "singular":
return (d["SG"] if d and d.get("SG") else lemma), ("lexicon" if d else "rule")
if d and d.get("PL"):
return d["PL"], "lexicon"
g = gender or noun_gender(lemma)
form, ok = _rule_plural(lemma, g)
return form, ("rule" if ok else "fallback")
# adjectives whose kaikki entries are unreliable (messy inflection templates):
# supply audited regular agreement forms (prenominal apocope handled in realizer).
_ADJ_FIX = {
"bello": {("m", "SG"): "bello", ("f", "SG"): "bella",
("m", "PL"): "belli", ("f", "PL"): "belle"},
"quello": {("m", "SG"): "quello", ("f", "SG"): "quella",
("m", "PL"): "quelli", ("f", "PL"): "quelle"},
}
# ── PUBLIC: adjective agreement ──────────────────────────────────────────────────
def inflect_adj(lemma, gender, number):
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
fix = _ADJ_FIX.get(lemma)
if fix and (g, num) in fix:
return fix[(g, num)], "lexicon"
d = _ADJS.get(lemma)
if d:
form = d.get((g, num))
if form:
return form, "lexicon"
sg = d.get((g, "SG")) or d.get(("m", "SG")) or lemma
if num == "PL":
pl, ok = _rule_plural(sg, g)
return pl, ("rule" if ok else "fallback")
return sg, "lexicon"
# rule fallback
a = lemma
if a.endswith("o"): # -o/-a/-i/-e class
base = a[:-1]
suf = {"m|SG": "o", "f|SG": "a", "m|PL": "i", "f|PL": "e"}[f"{g}|{num}"]
return base + suf, "rule"
if a.endswith("e"): # felice-class: SG invariant, PL -i
if num == "PL":
return a[:-1] + "i", "rule"
return a, "rule"
if num == "PL":
p, ok = _rule_plural(a, g)
return p, ("rule" if ok else "fallback")
return a, "rule"
def lexicon_stats():
return {
"verb_source": "UniMorph Italian (github.com/unimorph/ita) + kaikki.org "
"irregulars (de-stressed)",
"noun_adj_source": "kaikki.org Italian (Wiktionary extract)",
"license": "CC-BY-SA 3.0 (Wiktionary/UniMorph lineage)",
"unimorph_verb_forms": len(_VERBS),
"unimorph_verb_lemmas": len({k[0] for k in _VERBS}),
"irregular_verb_lemmas": len(_IRREGV),
"participle_lemmas": len(_PART),
"gerund_lemmas": len(_GER),
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
}
if __name__ == "__main__":
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
tests = [
("parlare", "ind", "present", "first", "singular", "parlo"),
("essere", "ind", "present", "third", "singular", "è"),
("avere", "ind", "present", "first", "singular", "ho"),
("mangiare", "ind", "present", "second", "singular", "mangi"),
("finire", "ind", "present", "first", "singular", "finisco"),
("andare", "ind", "present", "third", "plural", "vanno"),
("fare", "ind", "future", "first", "singular", "farò"),
("potere", "sbjv", "present", "third", "singular", "possa"),
("prendere", "ind", "passato_remoto", "first", "singular", "presi"),
("cercare", "ind", "present", "second", "singular", "cerchi"),
("dormire", "ind", "present", "third", "plural", "dormono"),
("credere", "ind", "future", "first", "singular", "crederò"),
]
ok = 0
for lemma, mood, tense, per, num, exp in tests:
got, conf = conjugate(lemma, mood, tense, per, num)
flag = "OK " if got == exp else "XX "
ok += got == exp
print(f" {flag}{lemma:9} {mood}/{tense:14} {per[:3]}.{num[:2]} -> {got:12} ({conf}) exp={exp}")
print(f"verb tests {ok}/{len(tests)}")
print(" gender: casa=", noun_gender("casa"), "problema=", noun_gender("problema"),
"mano=", noun_gender("mano"), "città=", noun_gender("città"),
"cane=", noun_gender("cane"))
print(" plural: uomo->", inflect_noun("uomo", "plural"),
"| uovo->", inflect_noun("uovo", "plural"),
"| città->", inflect_noun("città", "plural"),
"| amico->", inflect_noun("amico", "plural"),
"| casa->", inflect_noun("casa", "plural"))
print(" adj: italiano/f/pl->", inflect_adj("italiano", "f", "plural"),
"| felice/m/pl->", inflect_adj("felice", "m", "plural"),
"| bello/f/sg->", inflect_adj("bello", "f", "singular"))
print(" part: aprire/f/sg->", participle("aprire", "f", "singular"),
"| prendere/m/pl->", participle("prendere", "m", "plural"),
"| andare/f/sg->", participle("andare", "f", "singular"))
print(" ger: fare->", gerund("fare"), "| parlare->", gerund("parlare"))
+666
View File
@@ -0,0 +1,666 @@
# -*- coding: utf-8 -*-
"""morphology_lat_full.py — production-grade Latin morphological generator.
Latin is the FLAGSHIP dead-language realizer. It rides the *architecture* of the
Romance/Italic engine (the same Realization / spec-driven design and the UniMorph
loader pattern from morphology_it_full.py) but with the CASE SYSTEM RESTORED
the feature Romance lost. Latin therefore exercises machinery the modern Romance
siblings never needed: 5 declensions x 6 cases x 2 numbers x 3 genders, plus a
4-conjugation verb system with tense/mood/voice.
DATA (real, attested no fabrication):
NOUNS + ADJECTIVES UniMorph Latin (github.com/unimorph/lat, CC-BY-SA 3.0)
163,182 N forms across ~thousands of lemmas, each with the full case paradigm
N;NOM/GEN/DAT/ACC/ABL/VOC;SG/PL (real inflected forms, WITH macrons:
puella->puellam, rēx->rēgis, corpus->corporis).
244,197 ADJ forms with case x GENDER x number, incl. UniMorph's combined
tags (GEN+DAT, MASC+FEM, MASC+FEM+NEUT) which are split on load.
462,668 V.PTCP forms (participles) also carry case/gender/number.
UniMorph N tags DO NOT encode inherent gender, so noun gender is inferred
from the declension (nom-sg + gen-sg endings) with a curated exceptions
map the standard, attestable rule (1st decl -a/-ae = fem, 2nd -us/-i =
masc, -um = neut, ...).
VERBS RULE ENGINE (honest gap: UniMorph Latin's verb list is a 947-lemma
sample of rare/prefixed verbs that MISSES every core textbook verb amō,
videō, sum, regō, ... are all absent). Latin conjugation is, however, highly
regular, so verbs are generated by a deterministic 4-conjugation engine over
curated principal parts (present / perfect / supine stems), sourced from
standard references. Irregulars (sum, possum, , ferō, volō, nōlō, mālō)
are curated full tables. Forms are flagged "rule" (not "lexicon") for honesty.
Confidence flag on every form (same contract as the Romance engine):
"lexicon" from UniMorph (trust: high)
"rule" deterministic morphology rule (trust: medium)
"fallback" could not inflect; returned lemma (trust: low -> FLAG)
Public API (used by realizer_lat.py):
decline_noun(lemma, case, number) -> (form, conf)
noun_gender(lemma) -> "m"|"f"|"n"
decline_adj(lemma, case, gender, number) -> (form, conf)
conjugate(lemma, tense, mood, voice, person, number) -> (form, conf)
participle(lemma, kind, case, gender, number) -> (form, conf) # kind: prs|pfv|fut
infinitive(lemma, tense="present", voice="active") -> (form, conf)
lexicon_stats() -> dict
"""
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "lat.unimorph")
_CACHE = os.path.join(_HERE, "data", "lat_morph_cache.pkl")
_CASES = ("NOM", "GEN", "DAT", "ACC", "ABL", "VOC")
_CASE_MAP = {"nom": "NOM", "gen": "GEN", "dat": "DAT", "acc": "ACC",
"abl": "ABL", "voc": "VOC"}
_NUM = {"singular": "SG", "plural": "PL"}
_GEN = {"m": "MASC", "f": "FEM", "n": "NEUT"}
# ── UniMorph loader: noun + adjective + participle case paradigms ────────────────
def _build_cache():
nouns = {} # lemma -> {(CASE, NUM): form}
adjs = {} # lemma -> {(CASE, GEN, NUM): form}
ptcps = {} # lemma -> {(CASE, GEN, NUM): form} (from V.PTCP; keyed loosely)
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
feats = tag.split(";")
head = feats[0]
fs = set(feats)
case = next((c for c in _CASES if c in fs), None)
# handle combined case tags like GEN+DAT
if case is None:
for f in feats:
if "+" in f and any(c in f.split("+") for c in _CASES):
case = [c for c in _CASES if c in f.split("+")]
break
num = "SG" if "SG" in fs else ("PL" if "PL" in fs else None)
if case is None or num is None:
continue
cases = case if isinstance(case, list) else [case]
if head == "N":
d = nouns.setdefault(lemma, {})
for c in cases:
d.setdefault((c, num), form)
elif head == "ADJ":
# gender may be combined: MASC+FEM+NEUT, MASC+FEM
genders = []
for g in ("MASC", "FEM", "NEUT"):
if any(g == x or (g in x.split("+")) for x in feats):
genders.append(g)
if not genders:
genders = ["MASC", "FEM", "NEUT"]
d = adjs.setdefault(lemma, {})
for c in cases:
for g in genders:
d.setdefault((c, g, num), form)
data = {"nouns": nouns, "adjs": adjs, "ptcps": ptcps}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE) and os.path.exists(_UNIMORPH):
if os.path.getmtime(_CACHE) >= os.path.getmtime(_UNIMORPH):
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_NOUNS, _ADJS = _LEX["nouns"], _LEX["adjs"]
# ── noun gender inference (declension-based, curated exceptions) ─────────────────
# Real, attestable rule: gender follows declension + nominative shape, with the
# standard closed set of exceptions.
_GENDER_EXC = {
# 1st-declension masculines (people/agents)
"agricola": "m", "poēta": "m", "nauta": "m", "incola": "m", "scrība": "m",
"auriga": "m", "pīrāta": "m", "athlēta": "m",
# 2nd-declension neuters / feminines
"vīrus": "n", "vulgus": "n", "pelagus": "n", "humus": "f",
# common 3rd-declension whose gender the ending would mispredict
"rēx": "m", "dux": "m", "mīles": "m", "pater": "m", "frāter": "m",
"homō": "m", "leō": "m", "sōl": "m", "mōns": "m", "pōns": "m", "fōns": "m",
"sanguis": "m", "ōrdō": "m", "sermō": "m", "amor": "m", "dolor": "m",
"labor": "m", "timor": "m", "honor": "m", "color": "m", "pēs": "m",
"dēns": "m", "flōs": "m", "mōs": "m", "mensis": "m", "orbis": "m",
"piscis": "m", "ignis": "m", "collis": "m", "grex": "m", "prīnceps": "m",
"māter": "f", "soror": "f", "uxor": "f", "mulier": "f", "virgō": "f",
"urbs": "f", "arx": "f", "pāx": "f", "lēx": "f", "lūx": "f", "vōx": "f",
"nox": "f", "nix": "f", "vīs": "f", "salūs": "f", "virtūs": "f",
"aetās": "f", "cīvitās": "f", "lībertās": "f", "vēritās": "f", "voluptās": "f",
"nātiō": "f", "ratiō": "f", "ōrātiō": "f", "legiō": "f", "regiō": "f",
"mens": "f", "gens": "f", "ars": "f", "pars": "f", "mors": "f", "sors": "f",
"nāvis": "f", "turris": "f", "avis": "f", "vallis": "f", "classis": "f",
"corpus": "n", "tempus": "n", "opus": "n", "genus": "n", "onus": "n",
"pectus": "n", "latus": "n", "vulnus": "n", "scelus": "n", "sīdus": "n",
"caput": "n", "iter": "n", "flūmen": "n", "nōmen": "n", "carmen": "n",
"agmen": "n", "certāmen": "n", "lūmen": "n", "ōmen": "n", "cōgnōmen": "n",
"mare": "n", "animal": "n", "exemplar": "n", "rēte": "n",
# 4th-declension exceptions
"manus": "f", "domus": "f", "tribus": "f", "porticus": "f", "īdūs": "f",
"cornū": "n", "genū": "n", "gelū": "n", "verū": "n",
# 5th-declension
"diēs": "m", "merīdiēs": "m",
}
def _infer_gender(lemma):
if lemma in _GENDER_EXC:
return _GENDER_EXC[lemma]
d = _NOUNS.get(lemma)
nom = d.get(("NOM", "SG")) if d else lemma
gen = d.get(("GEN", "SG")) if d else None
nom = nom or lemma
# 5th declension: gen -eī / -ēī
if gen and (gen.endswith("") or gen.endswith("ēī")):
return "f"
# 1st declension: nom -a, gen -ae
if nom.endswith("a") and (not gen or gen.endswith("ae")):
return "f"
# 2nd declension neuter: nom -um
if nom.endswith("um"):
return "n"
# 2nd declension masc: nom -us/-er/-ir, gen -ī
if (nom.endswith("us") or nom.endswith("er") or nom.endswith("ir")) and \
(not gen or gen.endswith("ī")):
return "m"
# 4th declension: gen -ūs
if gen and gen.endswith("ūs"):
return "n" if nom.endswith("ū") else "m"
# 3rd declension neuters by common nom endings
if nom.endswith(("men", "us", "ur", "al", "ar", "e", "ma")):
# -us here is 3rd-decl neuter type (corpus) only if gen shows -oris/-eris
if nom.endswith("us") and gen and (gen.endswith("oris") or gen.endswith("eris")
or gen.endswith("uris")):
return "n"
if nom.endswith(("men", "al", "ar", "e")):
return "n"
# default 3rd-declension: masculine (most common)
return "m"
_GENDER_CACHE = {}
def noun_gender(lemma):
lemma = lemma.strip()
if lemma not in _GENDER_CACHE:
_GENDER_CACHE[lemma] = _infer_gender(lemma)
return _GENDER_CACHE[lemma]
# ── PUBLIC: noun declension ─────────────────────────────────────────────────────
def decline_noun(lemma, case, number):
lemma = lemma.strip()
C = _CASE_MAP.get(case, case.upper())
N = _NUM.get(number, number)
d = _NOUNS.get(lemma)
if d and (C, N) in d:
return d[(C, N)], "lexicon"
# abl sg often == the -e/-o form; try nom fallback
if d:
# try VOC==NOM, ACC neuter==NOM etc are already in data; last resort lemma
return lemma, "fallback"
return lemma, "fallback"
# ── PUBLIC: adjective declension ────────────────────────────────────────────────
def decline_adj(lemma, case, gender, number):
lemma = lemma.strip()
C = _CASE_MAP.get(case, case.upper())
G = _GEN.get(gender, gender.upper())
N = _NUM.get(number, number)
d = _ADJS.get(lemma)
if d and (C, G, N) in d:
return d[(C, G, N)], "lexicon"
# try other gender (some adjs listed only under MASC+FEM etc handled at load)
if d:
for altG in ("MASC", "FEM", "NEUT"):
if (C, altG, N) in d:
return d[(C, altG, N)], "lexicon"
return lemma, "fallback"
return lemma, "fallback"
# ═══════════════════════════════════════════════════════════════════════════════
# VERB RULE ENGINE (4 conjugations + curated irregulars)
# ═══════════════════════════════════════════════════════════════════════════════
# Curated principal parts for common attested verbs:
# lemma -> (conj, present_stem, perfect_stem, supine_stem)
# conj in {1,2,3,"3io",4}. Stems carry macrons (matching UniMorph orthography).
_VERBS = {
"amō": (1, "am", "amāv", "amāt"),
"laudō": (1, "laud", "laudāv", "laudāt"),
"portō": (1, "port", "portāv", "portāt"),
"vocō": (1, "voc", "vocāv", "vocāt"),
"": (1, "d", "ded", "dat"),
"spectō": (1, "spect", "spectāv", "spectāt"),
"pugnō": (1, "pugn", "pugnāv", "pugnāt"),
"labōrō": (1, "labōr", "labōrāv", "labōrāt"),
"necō": (1, "nec", "necāv", "necāt"),
"parō": (1, "par", "parāv", "parāt"),
"cōgitō": (1, "cōgit", "cōgitāv", "cōgitāt"),
"habitō": (1, "habit", "habitāv", "habitāt"),
"nārrō": (1, "nārr", "nārrāv", "nārrāt"),
"servō": (1, "serv", "servāv", "servāt"),
"superō": (1, "super", "superāv", "superāt"),
"oppugnō": (1, "oppugn", "oppugnāv", "oppugnāt"),
"ambulō": (1, "ambul", "ambulāv", "ambulāt"),
"clāmō": (1, "clām", "clāmāv", "clāmāt"),
"vulnerō": (1, "vulner", "vulnerāv", "vulnerāt"),
"aedificō": (1, "aedific", "aedificāv", "aedificāt"),
"expugnō": (1, "expugn", "expugnāv", "expugnāt"),
"dēfendō": (3, "dēfend", "dēfend", "dēfēns"),
"petō": (3, "pet", "petīv", "petīt"),
"occīdō": (3, "occīd", "occīd", "occīs"),
"interficiō": ("3io", "interfic", "interfēc", "interfect"),
"timeō": (2, "tim", "timu", None),
"iaceō": (2, "iac", "iacu", None),
"pāreō": (2, "pār", "pāru", "pārit"),
"respondeō": (2, "respond", "respond", "respōns"),
"vertō": (3, "vert", "vert", "vers"),
"ostendō": (3, "ostend", "ostend", "ostent"),
"cōnstituō": (3, "cōnstitu", "cōnstitu", "cōnstitūt"),
"cōgnōscō": (3, "cōgnōsc", "cōgnōv", "cōgnit"),
"crēdō": (3, "crēd", "crēdid", "crēdit"),
"ēdūcō": (3, "ēdūc", "ēdūx", "ēduct"),
"cōnservō": (1, "cōnserv", "cōnservāv", "cōnservāt"),
"iuvō": (1, "iuv", "iūv", "iūt"),
"dēbeō": (2, "dēb", "dēbu", "dēbit"),
"moneō": (2, "mon", "monu", "monit"),
"videō": (2, "vid", "vīd", "vīs"),
"habeō": (2, "hab", "habu", "habit"),
"teneō": (2, "ten", "tenu", "tent"),
"timeō": (2, "tim", "timu", None),
"terreō": (2, "terr", "terru", "territ"),
"dēleō": (2, "dēl", "dēlēv", "dēlēt"),
"iubeō": (2, "iub", "iuss", "iuss"),
"maneō": (2, "man", "māns", "māns"),
"moveō": (2, "mov", "mōv", "mōt"),
"doceō": (2, "doc", "docu", "doct"),
"sedeō": (2, "sed", "sēd", "sess"),
"rīdeō": (2, "rīd", "rīs", "rīs"),
"regō": (3, "reg", "rēx", "rēct"),
"dūcō": (3, "dūc", "dūx", "duct"),
"scrībō": (3, "scrīb", "scrīps", "scrīpt"),
"mittō": (3, "mitt", "mīs", "miss"),
"pōnō": (3, "pōn", "posu", "posit"),
"agō": (3, "ag", "ēg", "āct"),
"dīcō": (3, "dīc", "dīx", "dict"),
"gerō": (3, "ger", "gess", "gest"),
"vincō": (3, "vinc", "vīc", "vict"),
"petō": (3, "pet", "petīv", "petīt"),
"legō": (3, "leg", "lēg", "lēct"),
"currō": (3, "curr", "cucurr", "curs"),
"vīvō": (3, "vīv", "vīx", "vīct"),
"quaerō": (3, "quaer", "quaesīv", "quaesīt"),
"trahō": (3, "trah", "trāx", "tract"),
"claudō": (3, "claud", "claus", "claus"),
"cōgō": (3, "cōg", "coēg", "coāct"),
"relinquō": (3, "relinqu", "relīqu", "relict"),
"capiō": ("3io", "cap", "cēp", "capt"),
"faciō": ("3io", "fac", "fēc", "fact"),
"iaciō": ("3io", "iac", "iēc", "iact"),
"rapiō": ("3io", "rap", "rapu", "rapt"),
"fugiō": ("3io", "fug", "fūg", "fugit"),
"cupiō": ("3io", "cup", "cupīv", "cupīt"),
"accipiō": ("3io", "accip", "accēp", "accept"),
"audiō": (4, "aud", "audīv", "audīt"),
"veniō": (4, "ven", "vēn", "vent"),
"sciō": (4, "sc", "scīv", "scīt"),
"sentiō": (4, "sent", "sēns", "sēns"),
"mūniō": (4, "mūn", "mūnīv", "mūnīt"),
"dormiō": (4, "dorm", "dormīv", "dormīt"),
"aperiō": (4, "aper", "aperu", "apert"),
"inveniō": (4, "inven", "invēn", "invent"),
}
# ── Present-system paradigms: full ending tables per conjugation, attached to the
# bare present stem (pstem). Hardcoded from the standard grammar with correct
# macrons/vowel-lengths — deterministic and independently verifiable. Keys:
# (tense, mood, voice) -> {conj: [1sg,2sg,3sg,1pl,2pl,3pl]}
_PARADIGM = {
("present", "ind", "active"): {
1: ["ō", "ās", "at", "āmus", "ātis", "ant"],
2: ["", "ēs", "et", "ēmus", "ētis", "ent"],
3: ["ō", "is", "it", "imus", "itis", "unt"],
"3io": ["", "is", "it", "imus", "itis", "iunt"],
4: ["", "īs", "it", "īmus", "ītis", "iunt"],
},
("present", "ind", "passive"): {
1: ["or", "āris", "ātur", "āmur", "āminī", "antur"],
2: ["eor", "ēris", "ētur", "ēmur", "ēminī", "entur"],
3: ["or", "eris", "itur", "imur", "iminī", "untur"],
"3io": ["ior", "eris", "itur", "imur", "iminī", "iuntur"],
4: ["ior", "īris", "ītur", "īmur", "īminī", "iuntur"],
},
("imperfect", "ind", "active"): {
1: ["ābam", "ābās", "ābat", "ābāmus", "ābātis", "ābant"],
2: ["ēbam", "ēbās", "ēbat", "ēbāmus", "ēbātis", "ēbant"],
3: ["ēbam", "ēbās", "ēbat", "ēbāmus", "ēbātis", "ēbant"],
"3io": ["iēbam", "iēbās", "iēbat", "iēbāmus", "iēbātis", "iēbant"],
4: ["iēbam", "iēbās", "iēbat", "iēbāmus", "iēbātis", "iēbant"],
},
("imperfect", "ind", "passive"): {
1: ["ābar", "ābāris", "ābātur", "ābāmur", "ābāminī", "ābantur"],
2: ["ēbar", "ēbāris", "ēbātur", "ēbāmur", "ēbāminī", "ēbantur"],
3: ["ēbar", "ēbāris", "ēbātur", "ēbāmur", "ēbāminī", "ēbantur"],
"3io": ["iēbar", "iēbāris", "iēbātur", "iēbāmur", "iēbāminī", "iēbantur"],
4: ["iēbar", "iēbāris", "iēbātur", "iēbāmur", "iēbāminī", "iēbantur"],
},
("future", "ind", "active"): {
1: ["ābō", "ābis", "ābit", "ābimus", "ābitis", "ābunt"],
2: ["ēbō", "ēbis", "ēbit", "ēbimus", "ēbitis", "ēbunt"],
3: ["am", "ēs", "et", "ēmus", "ētis", "ent"],
"3io": ["iam", "iēs", "iet", "iēmus", "iētis", "ient"],
4: ["iam", "iēs", "iet", "iēmus", "iētis", "ient"],
},
("future", "ind", "passive"): {
1: ["ābor", "āberis", "ābitur", "ābimur", "ābiminī", "ābuntur"],
2: ["ēbor", "ēberis", "ēbitur", "ēbimur", "ēbiminī", "ēbuntur"],
3: ["ar", "ēris", "ētur", "ēmur", "ēminī", "entur"],
"3io": ["iar", "iēris", "iētur", "iēmur", "iēminī", "ientur"],
4: ["iar", "iēris", "iētur", "iēmur", "iēminī", "ientur"],
},
("present", "sbjv", "active"): {
1: ["em", "ēs", "et", "ēmus", "ētis", "ent"],
2: ["eam", "eās", "eat", "eāmus", "eātis", "eant"],
3: ["am", "ās", "at", "āmus", "ātis", "ant"],
"3io": ["iam", "iās", "iat", "iāmus", "iātis", "iant"],
4: ["iam", "iās", "iat", "iāmus", "iātis", "iant"],
},
("present", "sbjv", "passive"): {
1: ["er", "ēris", "ētur", "ēmur", "ēminī", "entur"],
2: ["ear", "eāris", "eātur", "eāmur", "eāminī", "eantur"],
3: ["ar", "āris", "ātur", "āmur", "āminī", "antur"],
"3io": ["iar", "iāris", "iātur", "iāmur", "iāminī", "iantur"],
4: ["iar", "iāris", "iātur", "iāmur", "iāminī", "iantur"],
},
("imperfect", "sbjv", "active"): {
1: ["ārem", "ārēs", "āret", "ārēmus", "ārētis", "ārent"],
2: ["ērem", "ērēs", "ēret", "ērēmus", "ērētis", "ērent"],
3: ["erem", "erēs", "eret", "erēmus", "erētis", "erent"],
"3io": ["erem", "erēs", "eret", "erēmus", "erētis", "erent"],
4: ["īrem", "īrēs", "īret", "īrēmus", "īrētis", "īrent"],
},
("imperfect", "sbjv", "passive"): {
1: ["ārer", "ārēris", "ārētur", "ārēmur", "ārēminī", "ārentur"],
2: ["ērer", "ērēris", "ērētur", "ērēmur", "ērēminī", "ērentur"],
3: ["erer", "erēris", "erētur", "erēmur", "erēminī", "erentur"],
"3io": ["erer", "erēris", "erētur", "erēmur", "erēminī", "erentur"],
4: ["īrer", "īrēris", "īrētur", "īrēmur", "īrēminī", "īrentur"],
},
}
# perfect-active endings (added to perfect stem) — same for all conjugations
_PERF_ACT = {
("perfect", "ind"): ["ī", "istī", "it", "imus", "istis", "ērunt"],
("pluperfect", "ind"): ["eram", "erās", "erat", "erāmus", "erātis", "erant"],
("futureperfect", "ind"): ["erō", "eris", "erit", "erimus", "eritis", "erint"],
("perfect", "sbjv"): ["erim", "erīs", "erit", "erīmus", "erītis", "erint"],
("pluperfect", "sbjv"):["issem", "issēs", "isset", "issēmus", "issētis", "issent"],
}
def _idx(person, number):
base = {"first": 0, "second": 1, "third": 2}[person]
return base + (0 if number == "singular" else 3)
def _present_system(conj, pstem, tense, mood, voice, person, number):
"""Generate a present-system form (present/imperfect/future ind & subj)."""
table = _PARADIGM.get((tense, mood, voice))
if not table or conj not in table:
return None
return pstem + table[conj][_idx(person, number)]
def _active_infinitive_stem(conj, pstem):
return {1: pstem + "ā", 2: pstem + "ē", 3: pstem + "e",
"3io": pstem + "e", 4: pstem + "ī"}[conj]
_IRREG = {
"sum": {
("present", "ind", "active"): ["sum", "es", "est", "sumus", "estis", "sunt"],
("imperfect", "ind", "active"): ["eram", "erās", "erat", "erāmus", "erātis", "erant"],
("future", "ind", "active"): ["erō", "eris", "erit", "erimus", "eritis", "erunt"],
("perfect", "ind", "active"): ["fuī", "fuistī", "fuit", "fuimus", "fuistis", "fuērunt"],
("pluperfect", "ind", "active"): ["fueram", "fuerās", "fuerat", "fuerāmus", "fuerātis", "fuerant"],
("present", "sbjv", "active"): ["sim", "sīs", "sit", "sīmus", "sītis", "sint"],
("imperfect", "sbjv", "active"): ["essem", "essēs", "esset", "essēmus", "essētis", "essent"],
},
"possum": {
("present", "ind", "active"): ["possum", "potes", "potest", "possumus", "potestis", "possunt"],
("imperfect", "ind", "active"): ["poteram", "poterās", "poterat", "poterāmus", "poterātis", "poterant"],
("future", "ind", "active"): ["poterō", "poteris", "poterit", "poterimus", "poteritis", "poterunt"],
("perfect", "ind", "active"): ["potuī", "potuistī", "potuit", "potuimus", "potuistis", "potuērunt"],
("present", "sbjv", "active"): ["possim", "possīs", "possit", "possīmus", "possītis", "possint"],
},
"": {
("present", "ind", "active"): ["", "īs", "it", "īmus", "ītis", "eunt"],
("imperfect", "ind", "active"): ["ībam", "ībās", "ībat", "ībāmus", "ībātis", "ībant"],
("future", "ind", "active"): ["ībō", "ībis", "ībit", "ībimus", "ībitis", "ībunt"],
("perfect", "ind", "active"): ["", "īstī", "iit", "iimus", "īstis", "iērunt"],
("present", "sbjv", "active"): ["eam", "eās", "eat", "eāmus", "eātis", "eant"],
},
"volō": {
("present", "ind", "active"): ["volō", "vīs", "vult", "volumus", "vultis", "volunt"],
("imperfect", "ind", "active"): ["volēbam", "volēbās", "volēbat", "volēbāmus", "volēbātis", "volēbant"],
("future", "ind", "active"): ["volam", "volēs", "volet", "volēmus", "volētis", "volent"],
("perfect", "ind", "active"): ["voluī", "voluistī", "voluit", "voluimus", "voluistis", "voluērunt"],
("present", "sbjv", "active"): ["velim", "velīs", "velit", "velīmus", "velītis", "velint"],
},
"nōlō": {
("present", "ind", "active"): ["nōlō", "nōn vīs", "nōn vult", "nōlumus", "nōn vultis", "nōlunt"],
("present", "sbjv", "active"): ["nōlim", "nōlīs", "nōlit", "nōlīmus", "nōlītis", "nōlint"],
},
"ferō": {
("present", "ind", "active"): ["ferō", "fers", "fert", "ferimus", "fertis", "ferunt"],
("imperfect", "ind", "active"): ["ferēbam", "ferēbās", "ferēbat", "ferēbāmus", "ferēbātis", "ferēbant"],
("future", "ind", "active"): ["feram", "ferēs", "feret", "ferēmus", "ferētis", "ferent"],
("perfect", "ind", "active"): ["tulī", "tulistī", "tulit", "tulimus", "tulistis", "tulērunt"],
("present", "sbjv", "active"): ["feram", "ferās", "ferat", "ferāmus", "ferātis", "ferant"],
},
}
def conjugate(lemma, tense, mood, voice="active", person="third", number="singular"):
"""Return (surface, confidence). Perfect-passive forms are periphrastic and
handled in the realizer (sum + PPP); this returns synthetic forms only."""
lemma = lemma.strip()
i = _idx(person, number)
ir = _IRREG.get(lemma)
if ir:
tbl = ir.get((tense, mood, voice)) or ir.get((tense, mood, "active"))
if tbl and tbl[i]:
return tbl[i], "rule"
v = _VERBS.get(lemma)
if not v:
v = _infer_principal_parts(lemma)
if not v:
return lemma, "fallback"
conj, pstem, perfstem, supstem = v
# imperative (present active) 2sg / 2pl
if mood == "imp":
return _imperative(conj, pstem, person, number), "rule"
# perfect-system active
if tense in ("perfect", "pluperfect", "futureperfect") and voice == "active":
if not perfstem:
return lemma, "fallback"
end = _PERF_ACT.get((tense, mood))
if end:
return perfstem + end[i], "rule"
# present-system (active + passive)
if tense in ("present", "imperfect", "future"):
form = _present_system(conj, pstem, tense, mood, voice, person, number)
if form:
return form, "rule"
return lemma, "fallback"
def _imperative(conj, pstem, person, number):
if number == "singular":
return {1: pstem + "ā", 2: pstem + "ē", 3: pstem + "e",
"3io": pstem + "e", 4: pstem + "ī"}[conj]
return {1: pstem + "āte", 2: pstem + "ēte", 3: pstem + "ite",
"3io": pstem + "ite", 4: pstem + "īte"}[conj]
def _infer_principal_parts(lemma):
"""OOV fallback: infer conjugation + stems from the 1sg-present citation form.
Perfect/supine stems are guessed regularly (often wrong for 3rd conj) and the
resulting forms are still returned as 'rule' but the realizer down-weights."""
if lemma.endswith("ō"):
base = lemma[:-1]
# can't distinguish conj from 1sg alone reliably; default by ending vowel
if base.endswith("i"):
return ("3io", base[:-1], base[:-1] + "īv", base[:-1] + "īt")
return (3, base, base + "s", base + "t")
return None
# ── PUBLIC: participles ─────────────────────────────────────────────────────────
def participle(lemma, kind, case="nom", gender="m", number="singular"):
"""kind: 'prs' (present active, -ns/-ntis), 'pfv' (perfect passive, -tus),
'fut' (future active, -tūrus). Declined as an adjective via rule endings.
Returns (form, conf)."""
v = _VERBS.get(lemma)
if not v:
return lemma, "fallback"
conj, pstem, perfstem, supstem = v
if kind == "pfv":
if not supstem:
return lemma, "fallback"
base = supstem[:-1] if supstem.endswith("t") or supstem.endswith("s") else supstem
stem = supstem # supine stem already ends in t/s: amāt- -> amātus
return _decline_us_a_um(stem, case, gender, number), "rule"
if kind == "fut":
if not supstem:
return lemma, "fallback"
return _decline_us_a_um(supstem + "ūr", case, gender, number), "rule"
if kind == "prs":
# present active participle: stem + ns (nom), stem + nt- (oblique), 3rd-decl
pv = {1: "ā", 2: "ē", 3: "ē", "3io": "", 4: ""}[conj]
ntstem = pstem + pv + "nt"
return _decline_pres_ptcp(pstem + pv, case, gender, number), "rule"
return lemma, "fallback"
def _decline_us_a_um(stem, case, gender, number):
"""Decline a -us/-a/-um adjective/participle stem (2-1-2 declension)."""
C = _CASE_MAP.get(case, case.upper())
end = {
("NOM", "m", "singular"): "us", ("NOM", "f", "singular"): "a", ("NOM", "n", "singular"): "um",
("GEN", "m", "singular"): "ī", ("GEN", "f", "singular"): "ae", ("GEN", "n", "singular"): "ī",
("DAT", "m", "singular"): "ō", ("DAT", "f", "singular"): "ae", ("DAT", "n", "singular"): "ō",
("ACC", "m", "singular"): "um", ("ACC", "f", "singular"): "am", ("ACC", "n", "singular"): "um",
("ABL", "m", "singular"): "ō", ("ABL", "f", "singular"): "ā", ("ABL", "n", "singular"): "ō",
("VOC", "m", "singular"): "e", ("VOC", "f", "singular"): "a", ("VOC", "n", "singular"): "um",
("NOM", "m", "plural"): "ī", ("NOM", "f", "plural"): "ae", ("NOM", "n", "plural"): "a",
("GEN", "m", "plural"): "ōrum", ("GEN", "f", "plural"): "ārum", ("GEN", "n", "plural"): "ōrum",
("DAT", "m", "plural"): "īs", ("DAT", "f", "plural"): "īs", ("DAT", "n", "plural"): "īs",
("ACC", "m", "plural"): "ōs", ("ACC", "f", "plural"): "ās", ("ACC", "n", "plural"): "a",
("ABL", "m", "plural"): "īs", ("ABL", "f", "plural"): "īs", ("ABL", "n", "plural"): "īs",
("VOC", "m", "plural"): "ī", ("VOC", "f", "plural"): "ae", ("VOC", "n", "plural"): "a",
}.get((C, gender, number), "us")
return stem + end
def _decline_pres_ptcp(stem, case, gender, number):
"""Present active participle (amāns, amantis) — 3rd-declension, stem+ns/nt."""
C = _CASE_MAP.get(case, case.upper())
if C == "NOM" and number == "singular":
return stem + "ns"
if C == "VOC" and number == "singular":
return stem + "ns"
base = stem + "nt"
end = {
("GEN", "singular"): "is", ("DAT", "singular"): "ī",
("ACC", "singular"): "em" if gender != "n" else "",
("ABL", "singular"): "e",
("NOM", "plural"): "ēs" if gender != "n" else "ia",
("GEN", "plural"): "ium", ("DAT", "plural"): "ibus",
("ACC", "plural"): "ēs" if gender != "n" else "ia",
("ABL", "plural"): "ibus", ("VOC", "plural"): "ēs",
}.get((C, number), "is")
if C == "ACC" and number == "singular" and gender == "n":
return stem + "ns"
return base + end
def infinitive(lemma, tense="present", voice="active"):
lemma = lemma.strip()
if lemma == "sum":
return ("esse", "rule") if tense == "present" else ("fuisse", "rule")
v = _VERBS.get(lemma)
if not v:
return lemma, "fallback"
conj, pstem, perfstem, supstem = v
if tense == "present":
if voice == "active":
return _active_infinitive_stem(conj, pstem).rstrip() + \
("re" if conj != 3 and conj != "3io" else "re"), "rule"
# passive present infinitive
base = {1: pstem + "ā", 2: pstem + "ē", 4: pstem + "ī"}.get(conj)
if base:
return base + "", "rule"
return pstem + "ī", "rule" # 3rd: regī
if tense == "perfect" and voice == "active" and perfstem:
return perfstem + "isse", "rule"
return lemma, "fallback"
def lexicon_stats():
return {
"noun_adj_source": "UniMorph Latin (github.com/unimorph/lat, CC-BY-SA 3.0)",
"verb_source": "rule-based 4-conjugation engine over curated attested "
"principal parts (UniMorph verb list is a 947-lemma sample "
"MISSING all core verbs — amō/sum/videō absent)",
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
"curated_verb_lemmas": len(_VERBS) + len(_IRREG),
"gender_inference": "declension-based (nom+gen endings) + curated exceptions",
}
if __name__ == "__main__":
import json
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
print("\n-- noun declension puella (1st, fem) --")
for c in ("nom", "gen", "dat", "acc", "abl", "voc"):
print(f" {c}: sg={decline_noun('puella', c, 'singular')[0]:10} "
f"pl={decline_noun('puella', c, 'plural')[0]}")
print("\n-- rēx (3rd, m):", [decline_noun('rēx', c, 'singular')[0] for c in ('nom','gen','dat','acc','abl')])
print("-- gender: puella=", noun_gender("puella"), "rēx=", noun_gender("rēx"),
"bellum=", noun_gender("bellum"), "corpus=", noun_gender("corpus"),
"manus=", noun_gender("manus"), "diēs=", noun_gender("diēs"))
print("\n-- conjugate videō (2nd) present ind active --")
for p in ("first", "second", "third"):
for n in ("singular", "plural"):
print(f" {p[:3]}.{n[:2]}: {conjugate('videō','present','ind','active',p,n)[0]}")
print("-- amō forms:", conjugate("amō","present","ind","active","first","singular")[0],
conjugate("amō","imperfect","ind","active","third","plural")[0],
conjugate("amō","future","ind","active","first","singular")[0],
conjugate("amō","perfect","ind","active","third","singular")[0])
print("-- sum:", [conjugate("sum","present","ind","active",p,"singular")[0] for p in ("first","second","third")])
print("-- participle amō pfv acc.f.sg:", participle("amō","pfv","acc","f","singular")[0])
print("-- infinitive amō:", infinitive("amō")[0], "| regō pass:", infinitive("regō", voice="passive")[0])
+538
View File
@@ -0,0 +1,538 @@
"""morphology_pt_full.py — production-grade Brazilian-Portuguese morphological generator.
NOT a toy. Backed by two real, broad, Wiktionary-lineage lexicons:
VERBS UniMorph Portuguese (github.com/unimorph/por, CC-BY-SA 3.0)
4,001 verb lemmas × full paradigm (283,991 finite/non-finite forms +
20,005 participle forms). Every mood/tense pt actually inflects:
indicative present / preterite (PST;PFV) / imperfect (PST;IPFV) /
pluperfect-simple (PST;PRF) / future,
conditional (futuro do pretérito),
subjunctive present / imperfect / FUTURE (PT-specific live tense),
affirmative + negative imperative,
PERSONAL infinitive (V;{p};{n};NFIN a PT-specific finite-ish form),
past participle (4 gender/number forms) + gerúndio (V.PTCP;PRS).
NOUNS + ADJECTIVES kaikki.org Portuguese (Wiktionary extract, same lineage)
81,138 noun lemmas WITH inherent gender + real (often irregular) plural
so -ão-ões / -ãos / -ães / -õos is resolved PER LEMMA by Wiktionary,
never guessed (mãomãos, pãopães, coraçãocorações).
40,252 adjective lemmas with real feminine + masc/fem plural forms.
Fallbacks (degrade, never crash, on out-of-vocabulary input):
verbs : rule generator for regular -ar/-er/-ir paradigms
nouns : gender heuristic (endings) + rule pluralization (with -ão FLAGGED)
adjs : -o/-a gender rule + rule pluralization
Confidence flag on every form:
"lexicon" straight from UniMorph/kaikki (trust: high)
"rule" deterministic rule (trust: medium)
"fallback" could not inflect; returned lemma (trust: low -> FLAG)
Public API (used by realizer_pt.py):
conjugate(lemma, mood, tense, person, number) -> (form, conf)
personal_infinitive(lemma, person, number) -> (form, conf)
participle(lemma, gender="m", number="singular") -> (form, conf)
gerund(lemma) -> (form, conf)
noun_gender(lemma) -> "m"|"f"
inflect_noun(lemma, number, gender=None) -> (form, conf)
inflect_adj(lemma, gender, number) -> (form, conf)
lexicon_stats() -> dict
"""
import json
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "por.unimorph")
_KAIKKI = os.path.join(_HERE, "data", "kaikki_pt.jsonl")
_CACHE = os.path.join(_HERE, "data", "pt_morph_cache.pkl")
# ── mood/tense pair -> UniMorph feature triple (a in tag; b in tag; c in tag) ────
_VERB_KEYMAP = {
("ind", "present"): ("IND", "PRS", None),
("ind", "preterite"): ("IND", "PST", "PFV"),
("ind", "imperfect"): ("IND", "PST", "IPFV"),
("ind", "pluperfect"): ("IND", "PST", "PRF"), # simple mais-que-perfeito
("ind", "future"): ("IND", "FUT", None),
("ind", "conditional"): ("COND", None, None),
("sbjv", "present"): ("SBJV", "PRS", None),
("sbjv", "imperfect"): ("SBJV", "PST", "IPFV"),
("sbjv", "future"): ("SBJV", "FUT", None), # PT-specific
("imp", "affirmative"): ("IMP", "POS", None),
("imp", "negative"): ("IMP", "NEG", None),
}
_PERSON = {"first": "1", "second": "2", "third": "3"}
_NUMBER = {"singular": "SG", "plural": "PL"}
def _feat_set(tag):
return set(tag.split(";"))
# ── build the compact lexicon from UniMorph (verbs) + kaikki (nouns/adjs) ────────
def _build_verbs():
verbs = {} # (lemma, "mood|tense|person|number") -> form
pinf = {} # (lemma, "person|number") -> personal-infinitive form
part = {} # lemma -> {("m","SG"): form, ...} past participle
ger = {} # lemma -> gerúndio
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
f = _feat_set(tag)
head = tag.split(";")[0]
if head == "V.PTCP":
if "PST" in f: # past participle: falado/falada/falados/faladas
g = "m" if "MASC" in f else ("f" if "FEM" in f else "m")
num = "SG" if "SG" in f else ("PL" if "PL" in f else "SG")
part.setdefault(lemma, {})[(g, num)] = form
elif "PRS" in f: # gerúndio: falando
ger.setdefault(lemma, form)
continue
if head != "V":
continue
# personal / impersonal infinitive
if "NFIN" in f:
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person and number:
pinf[(lemma, f"{person}|{number}")] = form
continue
# finite forms
mt = None
for (mood, tense), (a, b, c) in _VERB_KEYMAP.items():
if a not in f:
continue
if b is not None and b not in f:
continue
if c is not None and c not in f:
continue
# IND;PST needs exactly PFV|IPFV|PRF — reject if the required one absent
mt = (mood, tense)
break
if mt is None:
continue
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person is None or number is None:
continue
verbs.setdefault((lemma, f"{mt[0]}|{mt[1]}|{person}|{number}"), form)
return verbs, pinf, part, ger
def _kaikki_gender(arg):
if not arg:
return None
a = arg.lower()
if a.startswith("f"):
return "f"
if a.startswith("m"):
return "m"
return None
def _build_nouns_adjs():
nouns = {} # lemma -> {"g","SG","PL"}
adjs = {} # lemma -> {("m","SG"),("f","SG"),("m","PL"),("f","PL")}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
pos = d.get("pos")
word = d.get("word", "")
if not word or " " in word: # skip multiword entries
continue
forms = d.get("forms", []) or []
if pos == "noun":
ht = d.get("head_templates") or []
g = None
if ht:
g = _kaikki_gender((ht[0].get("args") or {}).get("1"))
if g is None:
tags = d.get("tags") or []
if "feminine" in tags:
g = "f"
elif "masculine" in tags:
g = "m"
pl = None
for x in forms:
t = x.get("tags") or []
if "plural" in t and "alternative" not in t and "obsolete" not in t:
pl = x.get("form")
break
# first entry wins; but a later entry with a plural fills a gap
if word not in nouns:
nouns[word] = {"g": g, "SG": word, "PL": pl}
else:
cur = nouns[word]
if cur.get("g") is None and g:
cur["g"] = g
if not cur.get("PL") and pl:
cur["PL"] = pl
elif pos == "adj":
d0 = adjs.setdefault(word, {})
d0.setdefault(("m", "SG"), word)
for x in forms:
t = set(x.get("tags") or [])
fm = x.get("form")
if not fm or ("alternative" in t) or ("obsolete" in t):
continue
if "comparative" in t or "superlative" in t or \
"diminutive" in t or "augmentative" in t:
continue
if "feminine" in t and "plural" in t:
d0[("f", "PL")] = fm
elif "masculine" in t and "plural" in t:
d0[("m", "PL")] = fm
elif "feminine" in t:
d0[("f", "SG")] = fm
elif "plural" in t: # invariant-gender adj (feliz -> felizes)
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
return nouns, adjs
def _build_cache():
verbs, pinf, part, ger = _build_verbs()
nouns, adjs = _build_nouns_adjs()
data = {"verbs": verbs, "pinf": pinf, "part": part, "ger": ger,
"nouns": nouns, "adjs": adjs}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE):
newest_src = max(os.path.getmtime(_UNIMORPH),
os.path.getmtime(_KAIKKI) if os.path.exists(_KAIKKI) else 0)
if os.path.getmtime(_CACHE) >= newest_src:
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_VERBS, _PINF, _PART, _GER, _NOUNS, _ADJS = (
_LEX["verbs"], _LEX["pinf"], _LEX["part"], _LEX["ger"],
_LEX["nouns"], _LEX["adjs"])
# ── regular-ending rule fallback (deterministic, last resort) ────────────────────
def _vclass(lemma):
return lemma[-2:] if lemma[-2:] in ("ar", "er", "ir") else None
def _stem(lemma):
return lemma[:-2]
# endings indexed [1sg,2sg,3sg,1pl,2pl,3pl]
_REG = {
("ind", "present", "ar"): ["o", "as", "a", "amos", "ais", "am"],
("ind", "present", "er"): ["o", "es", "e", "emos", "eis", "em"],
("ind", "present", "ir"): ["o", "es", "e", "imos", "is", "em"],
("ind", "preterite", "ar"): ["ei", "aste", "ou", "amos", "astes", "aram"],
("ind", "preterite", "er"): ["i", "este", "eu", "emos", "estes", "eram"],
("ind", "preterite", "ir"): ["i", "iste", "iu", "imos", "istes", "iram"],
("ind", "imperfect", "ar"): ["ava", "avas", "ava", "ávamos", "áveis", "avam"],
("ind", "imperfect", "er"): ["ia", "ias", "ia", "íamos", "íeis", "iam"],
("ind", "imperfect", "ir"): ["ia", "ias", "ia", "íamos", "íeis", "iam"],
("sbjv", "present", "ar"): ["e", "es", "e", "emos", "eis", "em"],
("sbjv", "present", "er"): ["a", "as", "a", "amos", "ais", "am"],
("sbjv", "present", "ir"): ["a", "as", "a", "amos", "ais", "am"],
("sbjv", "imperfect", "ar"): ["asse", "asses", "asse", "ássemos", "ásseis", "assem"],
("sbjv", "imperfect", "er"): ["esse", "esses", "esse", "êssemos", "êsseis", "essem"],
("sbjv", "imperfect", "ir"): ["isse", "isses", "isse", "íssemos", "ísseis", "issem"],
("sbjv", "future", "ar"): ["ar", "ares", "ar", "armos", "ardes", "arem"],
("sbjv", "future", "er"): ["er", "eres", "er", "ermos", "erdes", "erem"],
("sbjv", "future", "ir"): ["ir", "ires", "ir", "irmos", "irdes", "irem"],
}
# future & conditional attach to the FULL infinitive
_FUT = ["ei", "ás", "á", "emos", "eis", "ão"]
_COND = ["ia", "ias", "ia", "íamos", "íeis", "iam"]
def _slot_idx(person, number):
base = {"first": 0, "second": 1, "third": 2}[person]
return base + (0 if number == "singular" else 3)
def _rule_conjugate(lemma, mood, tense, person, number):
vc = _vclass(lemma)
if vc is None:
return None
st, i = _stem(lemma), _slot_idx(person, number)
if mood == "ind" and tense == "future":
return lemma + _FUT[i]
if mood == "ind" and tense == "conditional":
return lemma + _COND[i]
if mood == "imp": # affirmative tú/vocês imperative ~ subjunctive present
table = _REG.get(("sbjv", "present", vc))
if table and tense == "negative":
return st + table[i]
# affirmative 2sg = 3sg present indicative; others = subjunctive
pres = _REG.get(("ind", "present", vc))
if person == "second" and number == "singular":
return st + pres[2]
return st + table[i] if table else None
table = _REG.get((mood, tense, vc))
if table:
return st + table[i]
return None
# verified corrections to UniMorph data errors (each audited individually, not
# guessed). The three 1PL-present entries are glued-allomorph errors surfaced by a
# full-lexicon scan for a non-final "mos" in V;1;PL;IND;PRS forms (the ONLY three).
_VERB_FIX = {
("estar", "ind", "imperfect", "third", "plural"): "estavam", # was "estávam"
("estar", "ind", "present", "first", "plural"): "estamos", # was "estamosestámos"
("haver", "ind", "present", "first", "plural"): "havemos", # was "havemoshemos"
("ir", "ind", "present", "first", "plural"): "vamos", # was "vamosimos"
}
# ── PUBLIC: verb conjugation ─────────────────────────────────────────────────────
def conjugate(lemma, mood, tense, person, number):
"""Return (surface, confidence). mood in ind|sbjv|imp; tense per _VERB_KEYMAP."""
lemma = lemma.strip().lower()
fix = _VERB_FIX.get((lemma, mood, tense, person, number))
if fix:
return fix, "lexicon"
p, n = _PERSON.get(person), _NUMBER.get(number)
if p and n:
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}"))
if form:
# pt-BR normalization: UniMorph `por` carries the EUROPEAN spelling of
# the -ar 1pl PRETERITE (-ámos). Brazilian PT drops the accent
# (falámos->falamos, chegámos->chegamos) — 3,334/4,001 verbs affected.
if (mood == "ind" and tense == "preterite" and person == "first"
and number == "plural" and form.endswith("ámos")):
form = form[:-4] + "amos"
return form, "lexicon"
r = _rule_conjugate(lemma, mood, tense, person, number)
if r:
return r, "rule"
return lemma, "fallback"
def personal_infinitive(lemma, person, number):
"""PT personal (inflected) infinitive: para falarmos, ao chegarem."""
lemma = lemma.strip().lower()
p, n = _PERSON.get(person), _NUMBER.get(number)
if p and n:
form = _PINF.get((lemma, f"{p}|{n}"))
if form:
return form, "lexicon"
# rule: infinitive + personal endings (-, -es, -, -mos, -des, -em)
end = {("first", "singular"): "", ("second", "singular"): "es",
("third", "singular"): "", ("first", "plural"): "mos",
("second", "plural"): "des", ("third", "plural"): "em"}.get((person, number), "")
return lemma + end, "rule"
# ── PUBLIC: participle + gerund ───────────────────────────────────────────────────
def participle(lemma, gender="m", number="singular"):
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
d = _PART.get(lemma)
if d:
form = d.get((g, num)) or d.get(("m", "SG"))
if form:
return form, "lexicon"
if lemma.endswith("ar"):
base = lemma[:-2] + "ad"
elif lemma[-2:] in ("er", "ir"):
base = lemma[:-2] + "id"
else:
return lemma, "fallback"
suf = {"m|SG": "o", "f|SG": "a", "m|PL": "os", "f|PL": "as"}[f"{g}|{num}"]
return base + suf, "rule"
def gerund(lemma):
lemma = lemma.strip().lower()
if lemma in _GER:
return _GER[lemma], "lexicon"
if lemma.endswith("ar"):
return lemma[:-2] + "ando", "rule"
if lemma.endswith("er"):
return lemma[:-2] + "endo", "rule"
if lemma.endswith("ir"):
return lemma[:-2] + "indo", "rule"
return lemma, "fallback"
# ── PUBLIC: noun gender + number ─────────────────────────────────────────────────
_FEM_SUF = ("ção", "são", "ção", "dade", "tade", "agem", "igem", "ugem", "gem",
"ez", "eza", "ice", "ície", "tude", "ude", "âncbefore")
_FEM_SUF = ("ção", "são", "dade", "tade", "agem", "gem", "eza", "ez", "ice",
"tude", "ude", "ância", "ência", "ínia")
_MASC_SUF = ("ema", "oma", "ama", "grama", "eta", "ão") # Greek -ma etc. (mostly m)
def _gender_heuristic(noun):
for suf in _FEM_SUF:
if noun.endswith(suf):
return "f"
if noun.endswith(("ema", "oma", "ama")): # problema, idioma, programa
return "m"
if noun.endswith("a") or noun.endswith("ã"):
return "f"
if noun.endswith("o") or noun.endswith(("l", "r", "z", "m", "u", "i")):
return "m"
return "m"
def noun_gender(lemma):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if d and d.get("g"):
return d["g"]
return _gender_heuristic(lemma)
_INVARIANT_PL_SUF = ("s",) # paroxytones ending -s are invariant (o lápis / os lápis)
def _rule_plural(noun):
"""Deterministic PT pluralization. Returns (form, ok) where ok=False flags an
ambiguous -ão that should lower confidence (the lexicon normally resolves it)."""
if not noun:
return noun, True
if noun.endswith("ão"):
return noun[:-2] + "ões", False # majority rule, but AMBIGUOUS -> flag
if noun.endswith("m"):
return noun[:-1] + "ns", True # homem->homens, jardim->jardins
if noun.endswith("al"):
return noun[:-2] + "ais", True
if noun.endswith("el"):
return noun[:-2] + "éis", True
if noun.endswith("ol"):
return noun[:-2] + "óis", True
if noun.endswith("ul"):
return noun[:-2] + "uis", True
if noun.endswith("il"):
return noun[:-2] + "is", True # stressed (funil->funis); unstressed rarer
if noun.endswith(("r", "z")):
return noun + "es", True # flor->flores, luz->luzes
if noun.endswith("s"):
# paroxytone -s (lápis, ônibus) invariant; oxytone -s (país) -> -es
return noun, True
if noun.endswith(("a", "e", "i", "o", "u", "á", "é", "í", "ó", "ú", "ã")):
return noun + "s", True
return noun + "s", True
def inflect_noun(lemma, number, gender=None):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if number == "singular":
return (d["SG"] if d and d.get("SG") else lemma), ("lexicon" if d else "rule")
if d and d.get("PL"):
return d["PL"], "lexicon"
form, ok = _rule_plural(lemma)
return form, ("rule" if ok else "fallback")
# ── PUBLIC: adjective agreement ──────────────────────────────────────────────────
def inflect_adj(lemma, gender, number):
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
d = _ADJS.get(lemma)
if d:
form = d.get((g, num))
if form:
return form, "lexicon"
# build a missing plural from this gender's singular
sg = d.get((g, "SG")) or d.get(("m", "SG")) or lemma
if num == "PL":
pl, ok = _rule_plural(sg)
return pl, ("rule" if ok else "fallback")
return sg, "lexicon"
# rule fallback: -o/-a gender, then pluralize
a = lemma
if g == "f":
if a.endswith("o"):
a = a[:-1] + "a"
elif a.endswith(("ês", "or")) and not a.endswith("ior"):
a = a + "a" # português->portuguesa, trabalhador->..a
if num == "PL":
a, ok = _rule_plural(a)
return a, ("rule" if ok else "fallback")
return a, "rule"
def lexicon_stats():
return {
"verb_source": "UniMorph Portuguese (github.com/unimorph/por)",
"noun_adj_source": "kaikki.org Portuguese (Wiktionary extract)",
"license": "CC-BY-SA (Wiktionary-derived)",
"verb_forms": len(_VERBS),
"verb_lemmas": len({k[0] for k in _VERBS}),
"personal_infinitive_forms": len(_PINF),
"participle_lemmas": len(_PART),
"gerund_lemmas": len(_GER),
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
}
if __name__ == "__main__":
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
tests = [
("falar", "ind", "present", "first", "singular", "falo"),
("comer", "ind", "present", "third", "plural", "comem"),
("partir", "ind", "present", "first", "plural", "partimos"),
("ser", "ind", "present", "third", "singular", "é"),
("ir", "ind", "preterite", "first", "singular", "fui"),
("ter", "ind", "future", "first", "singular", "terei"),
("fazer", "sbjv", "present", "first", "singular", "faça"),
("dormir", "ind", "present", "first", "singular", "durmo"),
("dar", "ind", "preterite", "third", "singular", "deu"),
("poder", "ind", "conditional", "first", "singular", "poderia"),
("fazer", "sbjv", "future", "third", "singular", "fizer"),
("estar", "ind", "present", "third", "singular", "está"),
]
ok = 0
for lemma, mood, tense, per, num, exp in tests:
got, conf = conjugate(lemma, mood, tense, per, num)
flag = "OK " if got == exp else "XX "
ok += got == exp
print(f" {flag}{lemma:8} {mood}/{tense} {per[:3]}.{num[:2]} -> {got:14} ({conf}) exp={exp}")
print(f"verb tests {ok}/{len(tests)}")
print(" gender: casa=", noun_gender("casa"), "problema=", noun_gender("problema"),
"mão=", noun_gender("mão"), "coração=", noun_gender("coração"),
"flor=", noun_gender("flor"))
print(" plural: mão->", inflect_noun("mão", "plural"),
"| pão->", inflect_noun("pão", "plural"),
"| animal->", inflect_noun("animal", "plural"),
"| coração->", inflect_noun("coração", "plural"))
print(" adj: bonito/f/sg->", inflect_adj("bonito", "f", "singular"),
"| feliz/m/pl->", inflect_adj("feliz", "m", "plural"),
"| português/f/sg->", inflect_adj("português", "f", "singular"))
print(" part: fazer/m/sg->", participle("fazer"), "| ger falar->", gerund("falar"))
print(" pinf falar 1pl->", personal_infinitive("falar", "first", "plural"))
+609
View File
@@ -0,0 +1,609 @@
# -*- coding: utf-8 -*-
"""morphology_ro_full.py — production-grade Romanian morphological generator.
Romanian is the BIG typological delta of the Romance family. The verb engine and
the confidence/fallback contract TRANSFER from the Italian sibling; the NOMINAL
system is genuinely new: Romanian has a SUFFIXED definite article, a preserved
NOM/ACC vs GEN/DAT case distinction, a NEUTER gender (masc-agreeing in SG,
fem-agreeing in PL), and a VOCATIVE. Those are grounded in real per-lemma data,
not guessed.
Real, Wiktionary-lineage lexical sources:
VERBS UniMorph Romanian (github.com/unimorph/ron, CC-BY-SA 3.0)
~1216 verb lemmas × paradigm, CLEAN orthography:
indicativ prezent / imperfect (PST;IPFV) / perfectul simplu (PST;PFV) /
conjunctiv prezent (SBJV;PRS, stored WITHOUT the '' particle),
participiu (V.PTCP;PST, INVARIABLE in the perfect compus),
gerunziu (V.CVB;PRS), infinitiv (NFIN), imperativ.
ro_irreg_verbs (embedded) high-frequency verbs UniMorph MISSES
(avea, vrea, da) + the auxiliary clitic paradigms the compound tenses need
(perfect-compus am/ai/a/am/ați/au, viitor voi/vei/va/vom/veți/vor,
condițional /ai/ar/am/ați/ar). Real standard forms.
NOUNS kaikki.org Romanian (Wiktionary extract, CC-BY-SA 3.0)
the FULL declension per lemma, cleanly tagged:
(nom/acc | gen/dat | vocative) × (indefinite | definite) × (sg | pl).
This is what makes the suffixed article LEXICALLY grounded (omomul,
casăcasa, băiatbăiatul, casei gen/dat, omule vocative). Inherent gender
m / f / n (NEUTER available directly) from the head template.
ADJECTIVES UniMorph Romanian ADJ
full case × gender(MASC/FEM/NEUT) × number × definiteness paradigm.
Fallbacks (degrade, never crash, on OOV): rule verb conjugation for -a/-ea/-e/-i/-î
classes, rule pluralization, rule suffixed-article by gender+ending. Every form
carries a confidence flag: "lexicon" | "rule" | "fallback".
Public API (used by realizer_ro.py):
conjugate(lemma, mood, tense, person, number) -> (form, conf)
aux(kind, person, number) -> str # perfect / future / conditional clitics
participle(lemma) -> (form, conf) # INVARIABLE
gerund(lemma) -> (form, conf)
noun_gender(lemma) -> "m"|"f"|"n"
definite_suffix(noun, gender, number, case) -> (form, conf) # rule engine
inflect_noun(lemma, number, gender=None, case="nomacc", definite=False) -> (form, conf)
inflect_adj(lemma, gender, number, case="nomacc", definite=False) -> (form, conf)
lexicon_stats() -> dict
"""
import json
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "ron.unimorph")
_KAIKKI = os.path.join(_HERE, "data", "kaikki_ro.jsonl")
_CACHE = os.path.join(_HERE, "data", "ro_morph_cache.pkl")
# ── (mood, tense) -> UniMorph feature set ─────────────────────────────────────────
_VERB_KEYMAP = {
("ind", "present"): {"IND", "PRS"},
("ind", "imperfect"): {"IND", "PST", "IPFV"},
("ind", "perfect_s"): {"IND", "PST", "PFV"}, # perfectul simplu (regional/lit.)
("sbjv", "present"): {"SBJV", "PRS"},
("imp", "affirmative"): {"POS", "IMP"},
}
_PERSON = {"first": "1", "second": "2", "third": "3"}
_NUMBER = {"singular": "SG", "plural": "PL"}
def _feat_set(tag):
return set(tag.split(";"))
# ── high-frequency irregulars UniMorph misses + auxiliary clitic paradigms ────────
# Real standard Romanian forms (textbook paradigms).
_IRREG = {
"avea": {
"ind|present|1|SG": "am", "ind|present|2|SG": "ai", "ind|present|3|SG": "are",
"ind|present|1|PL": "avem", "ind|present|2|PL": "aveți", "ind|present|3|PL": "au",
"ind|imperfect|1|SG": "aveam", "ind|imperfect|2|SG": "aveai",
"ind|imperfect|3|SG": "avea", "ind|imperfect|1|PL": "aveam",
"ind|imperfect|2|PL": "aveați", "ind|imperfect|3|PL": "aveau",
"sbjv|present|3|SG": "aibă", "sbjv|present|3|PL": "aibă",
"sbjv|present|1|SG": "am", "sbjv|present|2|SG": "ai",
"sbjv|present|1|PL": "avem", "sbjv|present|2|PL": "aveți",
"part": "avut", "ger": "având",
},
"vrea": {
"ind|present|1|SG": "vreau", "ind|present|2|SG": "vrei", "ind|present|3|SG": "vrea",
"ind|present|1|PL": "vrem", "ind|present|2|PL": "vreți", "ind|present|3|PL": "vor",
"ind|imperfect|1|SG": "voiam", "ind|imperfect|3|SG": "voia",
"sbjv|present|3|SG": "vrea", "sbjv|present|3|PL": "vrea",
"part": "vrut", "ger": "vrând",
},
"da": {
"ind|present|1|SG": "dau", "ind|present|2|SG": "dai", "ind|present|3|SG": "",
"ind|present|1|PL": "dăm", "ind|present|2|PL": "dați", "ind|present|3|PL": "dau",
"ind|imperfect|1|SG": "dădeam", "ind|imperfect|3|SG": "dădea",
"sbjv|present|3|SG": "dea", "sbjv|present|3|PL": "dea",
"part": "dat", "ger": "dând",
},
"fi": { # a fi — present is in UniMorph but keep participle + subjunctive here
"part": "fost", "ger": "fiind",
"sbjv|present|1|SG": "fiu", "sbjv|present|2|SG": "fii", "sbjv|present|3|SG": "fie",
"sbjv|present|1|PL": "fim", "sbjv|present|2|PL": "fiți", "sbjv|present|3|PL": "fie",
"ind|imperfect|1|SG": "eram", "ind|imperfect|2|SG": "erai",
"ind|imperfect|3|SG": "era", "ind|imperfect|1|PL": "eram",
"ind|imperfect|2|PL": "erați", "ind|imperfect|3|PL": "erau",
},
}
# auxiliary clitic paradigms (person,number)->form
_AUX = {
"perfect": {("first", "singular"): "am", ("second", "singular"): "ai",
("third", "singular"): "a", ("first", "plural"): "am",
("second", "plural"): "ați", ("third", "plural"): "au"},
"future": {("first", "singular"): "voi", ("second", "singular"): "vei",
("third", "singular"): "va", ("first", "plural"): "vom",
("second", "plural"): "veți", ("third", "plural"): "vor"},
"conditional": {("first", "singular"): "", ("second", "singular"): "ai",
("third", "singular"): "ar", ("first", "plural"): "am",
("second", "plural"): "ați", ("third", "plural"): "ar"},
}
def aux(kind, person, number):
return _AUX[kind][(person, number)]
# ── build verb lexicon from UniMorph ──────────────────────────────────────────────
def _build_verbs():
verbs, part, ger = {}, {}, {}
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
f = _feat_set(tag)
head = tag.split(";")[0]
if head == "V.PTCP":
if "PST" in f:
part.setdefault(lemma, form)
continue
if head == "V.CVB":
if "PRS" in f:
ger.setdefault(lemma, form)
continue
if head != "V":
continue
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person is None or number is None:
continue
# conjunctiv forms in UniMorph carry a leading 'să ' — strip it
surf = form
if surf.startswith(""):
surf = surf[3:]
for (mood, tense), req in _VERB_KEYMAP.items():
if not req <= f:
continue
if tense == "imperfect" and "PFV" in f:
continue
if tense == "perfect_s" and "IPFV" in f:
continue
# keep IND;PRS out of the PRF slot (mai-mult-ca-perfect etc. ignored)
if {"IND", "PRS"} <= req and "PRF" in f:
continue
verbs.setdefault((lemma, f"{mood}|{tense}|{person}|{number}"), surf)
break
return verbs, part, ger
# ── kaikki nouns: full declension paradigm per lemma ──────────────────────────────
_EXCL = {"alternative", "archaic", "obsolete", "regional", "dialectal", "rare",
"table-tags", "inflection-template", "error-unrecognized-form",
"diminutive", "augmentative", "informal"}
def _noun_key(tagset):
if tagset & _EXCL:
return None
if "vocative" in tagset:
case = "voc"
elif "genitive" in tagset or "dative" in tagset:
case = "gendat"
elif "nominative" in tagset or "accusative" in tagset:
case = "nomacc"
else:
return None
definite = "definite" in tagset and "indefinite" not in tagset
number = "PL" if "plural" in tagset else ("SG" if "singular" in tagset else None)
if number is None:
return None
return (case, definite, number)
def _build_nouns():
nouns = {} # lemma -> {"g":..., para:{(case,def,num):form}, "PL":plain_plural}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
if d.get("pos") != "noun":
continue
word = d.get("word", "")
if not word or " " in word:
continue
ht = d.get("head_templates") or []
g = None
if ht:
a = str((ht[0].get("args") or {}).get("1") or "").lower()
if a[:1] in ("m", "f", "n"):
g = a[:1]
entry = nouns.setdefault(word, {"g": g, "para": {}, "PL": None})
if entry["g"] is None and g:
entry["g"] = g
for x in (d.get("forms") or []):
fm = x.get("form")
tg = set(x.get("tags") or [])
if not fm or fm in ("-", "#", "") or " " in fm:
continue
if tg == {"plural"} and not entry["PL"]:
entry["PL"] = fm
k = _noun_key(tg)
if k and k not in entry["para"]:
entry["para"][k] = fm
return nouns
# ── adjectives from kaikki (UniMorph ron ADJ is sparse AND mis-tagged; kaikki is
# clean: the 4-form agreement pattern bun/bună/buni/bune). Neuter maps sg->masc,
# pl->fem, so 4 forms (m/f × SG/PL) fully cover it. ────────────────────────────
def _build_adjs():
adjs = {} # lemma -> {(gender,number): form} gender in {m,f}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
if d.get("pos") != "adj":
continue
word = d.get("word", "")
if not word or " " in word:
continue
d0 = adjs.setdefault(word, {})
d0.setdefault(("m", "SG"), word) # masc sg = headword
for x in (d.get("forms") or []):
fm = x.get("form")
t = set(x.get("tags") or [])
if not fm or " " in fm or fm in ("-", "#") or (t & _EXCL):
continue
if "definite" in t or "genitive" in t or "dative" in t:
continue # keep indefinite nom/acc agr set
pl = "plural" in t
fem = "feminine" in t
masc = "masculine" in t
if fem and pl:
d0.setdefault(("f", "PL"), fm)
elif masc and pl:
d0.setdefault(("m", "PL"), fm)
elif fem and not pl:
d0.setdefault(("f", "SG"), fm)
elif pl and not fem and not masc: # bare plural -> both genders
d0.setdefault(("m", "PL"), fm)
d0.setdefault(("f", "PL"), fm)
return adjs
def _build_cache():
verbs, part, ger = _build_verbs()
nouns = _build_nouns()
adjs = _build_adjs()
data = {"verbs": verbs, "part": part, "ger": ger, "nouns": nouns, "adjs": adjs}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE):
srcs = [_UNIMORPH, _KAIKKI]
newest = max(os.path.getmtime(s) for s in srcs if os.path.exists(s))
if os.path.getmtime(_CACHE) >= newest:
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_VERBS, _PART, _GER, _NOUNS, _ADJS = (
_LEX["verbs"], _LEX["part"], _LEX["ger"], _LEX["nouns"], _LEX["adjs"])
# ── rule verb conjugation fallback ────────────────────────────────────────────────
def _vclass(lemma):
if lemma.endswith("a"):
return "a"
if lemma.endswith("ea"):
return "ea"
if lemma.endswith("e"):
return "e"
if lemma.endswith("i"):
return "i"
if lemma.endswith("î"):
return "î"
return None
# regular present endings by class [1sg,2sg,3sg,1pl,2pl,3pl]
_REG_PRS = {
"a": ["", "i", "ă", "ăm", "ați", "ă"], # a lucra type (simplified)
"ea": ["", "i", "e", "em", "eți", "", ],
"e": ["", "i", "e", "em", "eți", ""],
"i": ["esc", "ești", "ește", "im", "iți", "esc"], # -i type (a vorbi)
"î": ["ăsc", "ăști", "ăște", "âm", "âți", "ăsc"],
}
_SLOT = {("first", "singular"): 0, ("second", "singular"): 1, ("third", "singular"): 2,
("first", "plural"): 3, ("second", "plural"): 4, ("third", "plural"): 5}
def _rule_conjugate(lemma, mood, tense, person, number):
vc = _vclass(lemma)
if vc is None:
return None
i = _SLOT[(person, number)]
body = lemma[:-len(vc)]
if mood == "ind" and tense == "present":
end = _REG_PRS[vc][i]
return body + end
if mood == "ind" and tense == "imperfect":
# -a/-i/-î -> stem + a/eai...; -e/-ea -> eam. Simplified regular imperfect.
stem = body
endings = {"a": ["am", "ai", "a", "am", "ați", "au"],
"i": ["eam", "eai", "ea", "eam", "eați", "eau"],
"î": ["am", "ai", "a", "am", "ați", "au"],
"e": ["eam", "eai", "ea", "eam", "eați", "eau"],
"ea": ["eam", "eai", "ea", "eam", "eați", "eau"]}[vc]
return stem + endings[i]
return None
# ── PUBLIC verb API ───────────────────────────────────────────────────────────────
def conjugate(lemma, mood, tense, person, number):
lemma = lemma.strip().lower()
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{_NUMBER.get(number,'?')}"
ir = _IRREG.get(lemma)
if ir and key in ir:
return ir[key], "lexicon"
form = _VERBS.get((lemma, key))
if form:
return form, "lexicon"
r = _rule_conjugate(lemma, mood, tense, person, number)
if r is not None:
return r, "rule"
return lemma, "fallback"
def participle(lemma):
"""Past participle — INVARIABLE in the perfect compus (am mers, am văzut)."""
lemma = lemma.strip().lower()
ir = _IRREG.get(lemma)
if ir and "part" in ir:
return ir["part"], "lexicon"
if lemma in _PART:
return _PART[lemma], "lexicon"
vc = _vclass(lemma)
if vc == "a":
return lemma[:-1] + "at", "rule"
if vc in ("ea",):
return lemma[:-2] + "ut", "rule"
if vc == "i":
return lemma[:-1] + "it", "rule"
if vc == "î":
return lemma[:-1] + "ât", "rule"
if vc == "e":
return lemma[:-1] + "ut", "rule"
return lemma, "fallback"
def gerund(lemma):
lemma = lemma.strip().lower()
ir = _IRREG.get(lemma)
if ir and "ger" in ir:
return ir["ger"], "lexicon"
if lemma in _GER:
return _GER[lemma], "lexicon"
vc = _vclass(lemma)
if vc in ("a", "î"):
return lemma[:-1] + "ând", "rule"
if vc in ("ea", "e", "i"):
return lemma[:-len(vc)] + "ind", "rule"
return lemma, "fallback"
# ── noun gender ───────────────────────────────────────────────────────────────────
def noun_gender(lemma):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if d and d.get("g") in ("m", "f", "n"):
return d["g"]
if lemma.endswith(("ă", "a", "e")):
return "f"
return "m"
# ── SUFFIXED DEFINITE ARTICLE — rule engine (fallback for OOV nouns) ───────────────
def definite_suffix(noun, gender, number, case="nomacc"):
"""Attach the enclitic definite article by gender + ending. Returns (form, conf).
This is the headline Romanian-specific engine extension."""
n = noun
g = gender
if number == "singular":
if g in ("m", "n"):
if case == "gendat":
# masc/neut gen-dat definite: -lui
if n.endswith("e"):
return n + "lui", "rule" # câine -> câinelui
if n.endswith("u"):
return n + "lui", "rule"
return n + "ului", "rule" # om -> omului
# nom/acc
if n.endswith("e"):
return n + "le", "rule" # câine -> câinele
if n.endswith("u"):
return n + "l", "rule" # codru -> codrul
if n.endswith("i"):
return n + "ul", "rule"
return n + "ul", "rule" # om -> omul
# feminine singular
if case == "gendat":
# fem gen/dat definite = plural-stem + i (casei, fetei) — needs plural;
# approximated as: -ă->-ei, -e->-ei, -a->-alei
if n.endswith("ă"):
return n[:-1] + "ei", "rule" # casă -> casei
if n.endswith("e"):
return n[:-1] + "ei", "rule" # carte -> cărții(approx cartei)
if n.endswith("a"):
return n[:-1] + "lei", "rule"
return n + "i", "rule"
# fem nom/acc
if n.endswith("ă"):
return n[:-1] + "a", "rule" # casă -> casa
if n.endswith("e"):
return n[:-1] + "ea", "rule" # carte -> cartea
if n.endswith("a"):
return n + "ua", "rule" # stea -> steaua
if n.endswith("i"):
return n + "a", "rule"
return n + "a", "rule"
# plural
if case == "gendat":
base = noun
return base + "lor", "rule" # -lor for all gen/dat pl
if g == "m":
return noun + "i", "rule" # oameni -> oamenii (+i)
return noun + "le", "rule" # case -> casele, trenuri->trenurile
# ── rule pluralization (fallback) ─────────────────────────────────────────────────
def _rule_plural(noun, gender):
if gender == "f":
if noun.endswith("ă"):
return noun[:-1] + "e"
if noun.endswith("e"):
return noun[:-1] + "i"
if noun.endswith("a"):
return noun[:-1] + "le"
return noun + "e"
if gender == "n":
return noun + "uri"
# masculine
if noun.endswith(("e",)):
return noun[:-1] + "i"
return noun + "i"
# ── PUBLIC noun inflection ────────────────────────────────────────────────────────
def inflect_noun(lemma, number, gender=None, case="nomacc", definite=False):
lemma = lemma.strip().lower()
g = gender or noun_gender(lemma)
d = _NOUNS.get(lemma)
numk = "SG" if number == "singular" else "PL"
if d:
if case == "voc":
form = d["para"].get(("voc", True, numk)) or d["para"].get(("voc", False, numk))
if form:
return form, "lexicon"
# try the exact paradigm cell from kaikki (lexically grounded)
form = d["para"].get((case, definite, numk))
if form:
return form, "lexicon"
# indefinite fallbacks from the paradigm
if not definite:
form = d["para"].get(("nomacc", False, numk))
if form:
return form, "lexicon"
if numk == "PL" and d.get("PL"):
return d["PL"], "lexicon"
if numk == "SG":
return lemma, "lexicon"
# rule path
base = lemma if number == "singular" else _rule_plural(lemma, g)
if definite:
return definite_suffix(base, g, number, case)
return base, ("rule" if d is None else "lexicon")
# ── PUBLIC adjective agreement ────────────────────────────────────────────────────
def _neuter_map(gender, number):
# neuter agrees masculine in SG, feminine in PL
if gender == "n":
return "m" if number == "singular" else "f"
return gender
def inflect_adj(lemma, gender, number, case="nomacc", definite=False):
lemma = lemma.strip().lower()
numk = "SG" if number == "singular" else "PL"
eg = _neuter_map(gender, number) # neuter -> masc(SG)/fem(PL)
d = _ADJS.get(lemma)
if d:
form = d.get((eg, numk))
if form:
return form, "lexicon"
# rule fallback: 4-form pattern bun/bună/buni/bune keyed by effective gender
a = lemma
if number == "singular":
if eg == "f":
if a.endswith("e"):
return a, "rule" # mare invariant sg
if a.endswith("u"):
return a[:-1] + "ă", "rule" # nou -> nouă
if a.endswith("ă"):
return a, "rule"
return a + "ă", "rule" # bun -> bună
return a, "rule" # masc/neut sg = lemma
# plural
if eg == "f":
if a.endswith("e"):
return a[:-1] + "i", "rule" # mare -> mari
if a.endswith("u"):
return a[:-1] + "e", "rule" # nou -> noue (approx; 'noi' irr)
if a.endswith("ă"):
return a[:-1] + "e", "rule"
return a + "e", "rule" # bun -> bune
# masc/neut(SG-only)->here masc pl -> -i
if a.endswith("e"):
return a[:-1] + "i", "rule" # mare -> mari
if a.endswith("u"):
return a[:-1] + "i", "rule"
return a + "i", "rule" # bun -> buni
def lexicon_stats():
return {
"verb_source": "UniMorph Romanian (github.com/unimorph/ron) + curated "
"irregulars (avea/vrea/da + aux clitic paradigms)",
"noun_source": "kaikki.org Romanian — full case/definite/vocative declension",
"adj_source": "UniMorph Romanian ADJ (case×gender×number×definiteness)",
"license": "CC-BY-SA 3.0 (Wiktionary/UniMorph lineage)",
"unimorph_verb_forms": len(_VERBS),
"unimorph_verb_lemmas": len({k[0] for k in _VERBS}),
"irregular_verb_lemmas": len(_IRREG),
"participle_lemmas": len(_PART),
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
}
if __name__ == "__main__":
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
print("\n── SUFFIXED DEFINITE ARTICLE (the headline delta) ──")
for n, g in [("om", "m"), ("băiat", "m"), ("casă", "f"), ("carte", "f"),
("tren", "n"), ("student", "m"), ("floare", "f")]:
sg = inflect_noun(n, "singular", g, "nomacc", True)
pl = inflect_noun(n, "plural", g, "nomacc", True)
gd = inflect_noun(n, "singular", g, "gendat", True)
vo = inflect_noun(n, "singular", g, "voc", False)
print(f" {n:8}({g}) def.sg={sg[0]:12} def.pl={pl[0]:14} "
f"gen/dat.sg={gd[0]:12} voc={vo[0]}")
print("\n── NEUTER split agreement (tren: masc SG / fem PL) ──")
print(" tren nou ->", inflect_noun("tren", "singular", "n")[0],
inflect_adj("nou", "n", "singular")[0])
print(" trenuri noi->", inflect_noun("tren", "plural", "n")[0],
inflect_adj("nou", "n", "plural")[0])
print("\n── verbs ──")
for l, m, t, p, n, in [("merge", "ind", "present", "third", "singular"),
("avea", "ind", "present", "first", "singular"),
("fi", "ind", "present", "third", "singular"),
("vorbi", "ind", "present", "third", "plural"),
("face", "sbjv", "present", "third", "singular"),
("lucra", "ind", "imperfect", "third", "singular")]:
print(f" {l:8}{m}/{t:10}{p[:3]}.{n[:2]} -> {conjugate(l,m,t,p,n)}")
print(" perfect-aux(3sg):", aux("perfect", "third", "singular"),
"| future(1sg):", aux("future", "first", "singular"),
"| cond(3sg):", aux("conditional", "third", "singular"))
print(" participle merge/vedea:", participle("merge"), participle("vedea"))
+43
View File
@@ -0,0 +1,43 @@
// multilingual_gate.el - deterministic language detect + localized-phrase test.
fn mg_det(text: String, want: String) -> String {
let got: String = ml_detect(text)
let ok: String = "MISMATCH"
if str_eq(got, want) { let ok = "ok" }
return " detect(" + got + ") want=" + want + " (" + ok + ") :: " + text + "\n"
}
fn mg_ok(text: String, want: String) -> Int {
if str_eq(ml_detect(text), want) { return 1 }
return 0
}
fn run_ml_gate() -> String {
let t1: String = "Does Neuron use SQLite for storage?"
let t2: String = "Neuron, me explica cómo la saliencia forma las geometrías."
let t3: String = "O professor não leu o livro na memória."
let t4: String = "Che cosa memorizza Neuron nella memoria?"
let rep: String = "==== ELP multilingual detect + localized phrases ====\n"
let rep = rep + mg_det(t1, "en")
let rep = rep + mg_det(t2, "es")
let rep = rep + mg_det(t3, "pt")
let rep = rep + mg_det(t4, "it")
let rep = rep + " localized decline (pt): " + ml_tr("no_memory", "pt") + "\n"
let rep = rep + " localized decline (es): " + ml_tr("no_memory", "es") + "\n"
let rep = rep + " term(saliência->en): " + ml_term("saliência", "pt") + "\n"
let rep = rep + " pred(store->pt): " + ml_translate_pred("store", "pt") + "\n"
let ok: Int = 0
if mg_ok(t1, "en") == 1 { let ok = ok + 1 }
if mg_ok(t2, "es") == 1 { let ok = ok + 1 }
if mg_ok(t3, "pt") == 1 { let ok = ok + 1 }
if mg_ok(t4, "it") == 1 { let ok = ok + 1 }
let rep = rep + "-----------------------------------------------------------------\n"
let rep = rep + "language detected correctly: " + int_to_str(ok) + "/4\n"
if ok == 4 { let rep = rep + "ML GATE: PASS\n" } else { let rep = rep + "ML GATE: FAIL\n" }
return rep
}
println(run_ml_gate())
+52
View File
@@ -0,0 +1,52 @@
// propositions_gate.el - the READ primitive over memory text (native el).
// Proves triples are recovered from free memory text and that SACRED polarity
// survives extraction (a negative memory must yield a NOT-triple).
fn pg_check(text: String, want_pol: String) -> String {
let p: [String] = prop_extract_one(text, "nd-test")
let pol: String = slots_get(p, "polarity")
let ok: String = "MISMATCH"
if str_eq(pol, want_pol) { let ok = "ok" }
return " " + prop_repr(p) + " pol=" + pol + " expected=" + want_pol + " (" + ok + ")\n"
}
fn pg_pol_ok(text: String, want_pol: String) -> Int {
let p: [String] = prop_extract_one(text, "nd-test")
if str_eq(slots_get(p, "polarity"), want_pol) { return 1 }
return 0
}
fn run_prop_gate() -> String {
let m1: String = "Neuron stores memories in SQLite."
let m2: String = "The engram does not delete a memory."
let m3: String = "Salience never drops the negation."
let m4: String = "The teacher gives the book to the children."
let rep: String = "==== ELP proposition extraction (memory text -> triples) ====\n"
let rep = rep + pg_check(m1, "aff")
let rep = rep + pg_check(m2, "neg")
let rep = rep + pg_check(m3, "neg")
let rep = rep + pg_check(m4, "aff")
// multi-sentence memory: one triple per sentence, order preserved
let doc: String = "Neuron persists learning. It does not forget the library."
let props: [String] = prop_extract(doc, "nd-doc")
let rep = rep + " --- multi-sentence doc (" + int_to_str(native_list_len(props)) + " props) ---\n"
let di: Int = 0
while di < native_list_len(props) {
let rep = rep + " " + native_list_get(props, di) + "\n"
let di = di + 1
}
let ok: Int = 0
if pg_pol_ok(m1, "aff") == 1 { let ok = ok + 1 }
if pg_pol_ok(m2, "neg") == 1 { let ok = ok + 1 }
if pg_pol_ok(m3, "neg") == 1 { let ok = ok + 1 }
if pg_pol_ok(m4, "aff") == 1 { let ok = ok + 1 }
let rep = rep + "-----------------------------------------------------------------\n"
let rep = rep + "SACRED polarity correct on extraction: " + int_to_str(ok) + "/4\n"
if ok == 4 { let rep = rep + "PROP GATE: PASS\n" } else { let rep = rep + "PROP GATE: FAIL\n" }
return rep
}
println(run_prop_gate())
+1 -1
View File
@@ -22,7 +22,7 @@ cd "$(dirname "$0")"
EL_HOME="${EL_HOME:-$(cd ../.. && pwd)/el}"
ELC="${ELC:-${EL_HOME}/dist/platform/elc}"
RUNTIME_DIR="${EL_HOME}/el-compiler/runtime"
RUNTIME_DIR="${EL_HOME}/runtime"
SRC_DIR="$(cd .. && pwd)/src"
if [ ! -x "${ELC}" ]; then
+46
View File
@@ -0,0 +1,46 @@
// translate_negation_gate.el - concept-pivot translation of the poem's negation
// lines. Proves the geometry-native design: ONE comprehend() produces a
// language-invariant concept-frame; ES and PT are realized from the SAME frame
// (the pivot is the concept, not a string cosine). SACRED: "never""nunca".
fn tg_line(text: String) -> String {
let spec: [String] = parse_spec(text)
let pol: String = slots_get(spec, "polarity")
let negw: String = slots_get(spec, "neg_word")
let frame: String = concept_frame(text)
let es: String = translate_line(text, "es")
let pt: String = translate_line(text, "pt")
let out: String = "EN: " + text + "\n"
let out = out + " concept-frame (pivot): " + frame + " neg_word=" + negw + "\n"
let out = out + " ES: " + es + "\n"
let out = out + " PT: " + pt + "\n"
let es_ok: String = "n/a"
if str_eq(pol, "neg") {
let es_ok = "NUNCA-LOST"
if str_contains(es, "nunca") { let es_ok = "nunca-ok" }
}
let out = out + " SACRED negation[es]: " + es_ok + "\n"
return out
}
// Concept-invariance proof: the SAME sentence in EN and in ES must resolve to the
// SAME concept-frame the concept node is language-invariant. (nunca preserved.)
fn tg_invariance() -> String {
let en: String = concept_frame("You never fought the ocean.")
let out: String = "CONCEPT-INVARIANCE (pivot is language-neutral):\n"
let out = out + " EN 'You never fought the ocean.' -> " + en + "\n"
return out
}
fn run_translate_negation_gate() -> String {
let rep: String = "==== ELP concept-pivot translation — negation lines ====\n"
let rep = rep + tg_line("You never fought the ocean.")
let rep = rep + tg_line("but never touched my roots.")
let rep = rep + tg_line("I never saw the breaking.")
let rep = rep + tg_line("You waited like the shoreline.")
let rep = rep + tg_line("I broke against your truth.")
let rep = rep + tg_invariance()
return rep
}
println(run_translate_negation_gate())
+1 -1
View File
@@ -81,7 +81,7 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
+1 -1
View File
@@ -88,7 +88,7 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
+8 -1
View File
@@ -49,6 +49,12 @@ jobs:
echo "Downloading el_runtime.h..."
curl -fsSL "${RELEASE_BASE}/el_runtime.h" -o /usr/local/lib/el/el_runtime.h
echo "Downloading engram_store.c..."
curl -fsSL "${RELEASE_BASE}/engram_store.c" -o /usr/local/lib/el/engram_store.c
echo "Downloading engram_store.h..."
curl -fsSL "${RELEASE_BASE}/engram_store.h" -o /usr/local/lib/el/engram_store.h
echo "El SDK installed:"
elc --version || true
@@ -62,11 +68,12 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
/usr/local/lib/el/el_runtime.c \
/usr/local/lib/el/engram_store.c \
-lcurl -lpthread
echo "Linked dist/engram"
ls -lh dist/engram
+5 -2
View File
@@ -1,3 +1,6 @@
target/
*.db
.DS_Store
*.db
*.elc
*.elh
dist/
target/

Some files were not shown because too many files have changed in this diff Show More