Compare commits

..

77 Commits

Author SHA1 Message Date
bigmerge e29fe4fd0b ingest: unify transduce_prose/transduce_structured into one transduce()
El SDK CI - dev / build-and-test (pull_request) Failing after 3m42s
transduce() is now THE single mechanism: one function, no content-type
branch inside it. It never asks whether `source` is prose, JSON, or
raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally:
split on "\n\n" as a universal boundary-marker check, and if that finds
no boundary, fall back to fixed 4096-char windows. Same node/edge wiring
(root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets
a heading/section_of link) regardless of what's inside a chunk. Dedup is
the existing find_existing_by_content path via merge_manifold, applied
uniformly. The old transduce_structured JSON dataset/records/feature-node
interpretation is deleted outright, not just unused — a JSON file now
gets chunked and deduped like anything else, with no pre-computed
structure. All five ingest_* entry points still exist unchanged in name
and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one
transduce() (ingest_stream builds its own turn-nodes directly and never
called either old function, so it's untouched).

This unlocks raw/opaque content (audio, or anything else with no natural
text/JSON shape) without any DSP, LLM call, or external API: transduce()
chunks it exactly like it chunks anything else. There is zero semantic
understanding of audio (or any payload) claimed or built here — any
meaning is expected to emerge later from Neuron's own existing mechanisms
(embedding, spreading activation, dedup) acting on this real geometry
over time.

Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk)
because El strings are NUL-unsafe under strlen-based ops and fs_read()'s
result silently truncates at the first embedded NUL, which is routine in
real binary/audio bytes. ingest_file compares fs_read()'s string length
against a real fs_size() stat() count; on mismatch it rebuilds the
payload as base64-encoded fixed 3072-byte windows read directly off disk
(binary-safe in C, verbatim, no invention), joined with the same "\n\n"
marker transduce()'s boundary scan already looks for. This is a
mechanical fidelity fix, not interpretation of content — transduce()
never learns a fallback happened. Registered both builtins' arity in
codegen.el; did not rebuild the elc compiler binary itself (unrelated,
pre-existing gap: self-hosting elc via el_seed.c fails on this worktree
independent of this change, reproduced with codegen.el reverted) — the
existing elc binary compiles calls to unregistered builtins via its
already-existing arity=-1 passthrough, confirmed by an actual clean
`elc ingest.el` + `cc` build against the modified el_runtime.c.

INGEST_KIND keeps existing only as an acquisition-mechanism selector
(dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a
content-type flag; the redundant "structured" value (an alias for "file"
that hinted the now-deleted JSON branch) is removed. ingest_dir drops its
file-extension filter for the same reason: transduce() takes anything now.

Verification: local manifold construction confirmed correct against a
real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte
real prefix slice) — exact expected node/edge counts both times
(101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching
ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64
content confirmed decoding back to the actual WAV header bytes. Compiles
clean via the real elc + the modified el_runtime.c/engram_*.c (built and
booted an actual sandbox engram off this exact source with `nsbx create
--branch`).

NOT verified this session, disclosed rather than papered over: end-to-end
server-confirmed persistence (a real before/after /api/stats delta, and a
fetched node by id) for the audio, prose, and JSON-fixture cases. Every
local nsbx sandbox engram tried tonight (two stock pre-#109 binaries
hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW
binary built from current dev) took minutes-to indefinitely long on the
final /api/load-merge write's embedding step and hit the client's 60s
HTTP timeout before responding, even for a 5-node write. This is
confirmed as real (if slow) forward progress, not a hang: the sandbox's
WAL file was observed growing steadily across every attempt. The code's
own pre-existing HONESTY GATE correctly refused to report success in
every case, returning "load-merge failed: ..." with a
"nothing below this manifold was confirmed persisted by the server" note
instead — exactly as designed. This is an environment/infrastructure
limitation, not a defect introduced by this change: the engram server
binary itself is untouched by this commit.
2026-08-15 17:28:19 -05:00
will.anderson 1db5694189 Merge pull request 'engram: add /api/nodes/reseed so a node body can be repaired at its own id' (#92) from feat/engram-reseed-route into dev
El SDK CI - dev / build-and-test (push) Failing after 3m58s
2026-08-15 19:59:34 +00:00
will.anderson e34ebd4b3d Merge pull request 'nsbx + cognitive architecture design + engram self-review series' (#113) from feat/neuron-sandbox into dev
El SDK CI - dev / build-and-test (push) Failing after 4m2s
2026-08-15 19:59:07 +00:00
will.anderson 69870ac883 Merge pull request 'fix(codegen): emit the declared cgi identity — it was searched for in a list that cannot contain it' (#89) from fix/cgi-identity-emission-clean into dev
El SDK CI - dev / build-and-test (push) Failing after 3m35s
2026-08-15 19:58:16 +00:00
will.anderson 274765e0aa Merge pull request 'Add native EL afferent organ: ingest (conscious) + transduce (invisible mechanism)' (#98) from worktree-agent-a1bb8ac67d9006e08 into dev
El SDK CI - dev / build-and-test (push) Failing after 3m59s
2026-08-15 19:57:57 +00:00
will.anderson d4a04bb944 Merge pull request 'swarm: native interruptibility for dispatched agent workers' (#107) from worktree-agent-a6177cda24c71d1df into dev
El SDK CI - dev / build-and-test (push) Failing after 3m55s
2026-08-15 19:57:10 +00:00
will.anderson 905c707d68 Merge pull request 'peripheral: own-core, consent-gated I/O organ (mic/camera/speaker)' (#112) from worktree-agent-af50f3458d7754f19 into dev
El SDK CI - dev / build-and-test (push) Failing after 3m42s
2026-08-15 19:55:51 +00:00
will.anderson 56455740e3 Merge pull request 'elp: native audio/image efferent surfaces + projector proof-of-shape' (#111) from worktree-agent-aaf04b0a9714c4070 into dev
El SDK CI - dev / build-and-test (push) Failing after 4m4s
2026-08-15 19:55:38 +00:00
will.anderson 840e54c7ac Merge pull request 'elp: native speech synthesis + voice-imitation faculty' (#110) from worktree-agent-acc02900ef4ade35e into dev
El SDK CI - dev / build-and-test (push) Failing after 4m13s
2026-08-15 19:55:20 +00:00
will.anderson 5c6da24033 Merge pull request 'spec: grounded edge-propagation (task #50) — gated design artifact' (#108) from worktree-agent-a6577c8211c332c5b into dev
El SDK CI - dev / build-and-test (push) Failing after 3m42s
2026-08-15 19:55:02 +00:00
will.anderson 09ae14a970 Merge pull request 'fix: float arithmetic codegen (segfault/garbage) and math_log aliasing' (#104) from worktree-agent-a456e0cf8cd2ee361 into dev
El SDK CI - dev / build-and-test (push) Failing after 3m57s
2026-08-15 19:54:46 +00:00
will.anderson 7b3f8f2ce8 Merge pull request 'sandbox: multi-repo stack worktree composer (el-stack / neuron-stack)' (#101) from worktree-agent-ac2381b0b9615ab20 into dev
El SDK CI - dev / build-and-test (push) Failing after 3m50s
2026-08-15 19:54:29 +00:00
will.anderson 45f64f3fac Merge pull request 'elp: native-EL language faculty — comprehension, propositions, multilingual, translation' (#100) from integration/langfaculty-20260814 into dev
El SDK CI - dev / build-and-test (push) Failing after 4m13s
2026-08-15 19:54:14 +00:00
will.anderson fdf0d6cb64 Merge pull request 'nsbx: one-command dev onboarding (branch + worktree + isolated engram)' (#99) from feat/nsbx-dev-env into dev
El SDK CI - dev / build-and-test (push) Failing after 13m56s
2026-08-15 19:52:35 +00:00
will.anderson c2d8a07c7b transduce: name the invisible mechanism, fix a silent-failure bug, drop a CRUD verb
El SDK CI - dev / build-and-test (pull_request) Failing after 14m6s
ingest and transduce are complements, not synonyms: ingest is the conscious,
deliberate act of pointing at a source (ingest_file/dir/url/llm/stream stay
named exactly that); transduce is the automatic, invisible mechanism inside
it that converts extracted surface content into geometry (renamed
build_prose/build_structured -> transduce_prose/transduce_structured, the
functions that actually turn raw text into a node+edge manifold).

Real bug found and fixed along the way: the final /api/load-merge response
was never checked for an error. A total failure (bad auth, network down,
anything) silently reported nodes_added:0/edges_added:0 — indistinguishable
from a benign 'everything was already known' outcome. Verified live: with a
wrong key, the tool now honestly returns {"error":"load-merge failed:
unauthorized",...} instead of a misleading zero.

Also dropped a CRUD-verb smell: the per-decision println said CREATE (a
database-log verb for something that hasn't actually been written to the
server yet — it's a local, tentative decision pending the batch merge).
Renamed to FORM. The dead-code eg_create_node (defined, never called)
renamed to eg_crystallize_node and annotated honestly as unused, since if
it's ever wired up it represents the real server-confirmed write, unlike
the local FORM guess.

Not yet re-verified end-to-end against a real successful write: the
ingest-test sandbox (nsbx up ingest-test) is itself currently broken —
it prints a green "ready" banner after its own readiness check fails,
and nothing is actually listening. Filed separately; not in scope here.
2026-08-15 14:44:18 -05:00
bigmerge d8d1b89143 Add repo AGENTS.md and two engram design docs (architecture hardening, DB tooling)
El SDK CI - dev / build-and-test (pull_request) Failing after 14m16s
AGENTS.md: root-level guide to the repo — which of the 8 el_runtime.c
copies is the one canonical, authored source (lang/releases/v1.0.0-20260501,
despite the misleading 'releases/' name) vs. lagging forks/build artifacts,
build commands, and session protocol.

engram/spec/architecture-hardening.design.md: terse engineering anchor for
the 2026-08-14 hardening vision (one calculus over the geometry, core +
ephemeral ring, persistence earned by salience, incarnation model) —
indexes the fuller whitepaper + Neuron artifact 2b8078cf rather than
restating them.

engram/spec/engram-db-tooling-design.md: high-level design for engram DB
tooling (geometry-native browse/query/ops surface over the existing
vantage-read/write/relate/supersede API).

Deliberately leaves out of this commit: the uncommitted el_runtime.c/h +
codegen.el float-arithmetic-codegen diff in this worktree, which appears
to overlap with (or supersede) the fix already preserved via PR #104 —
needs manual reconciliation rather than a second competing PR. Also
leaves out lang/.promote-backup-floatfix/ (a local backup snapshot,
confirms that float-fix work is mid-promotion here), assorted .DS_Store
files, engram/dist/engram.* backup binaries, and lang/dist backup
binaries — none of it source.
2026-08-15 14:29:59 -05:00
bigmerge 6f3d692784 Add peripheral — own-core, consent-gated I/O organ
El SDK CI - dev / build-and-test (pull_request) Failing after 14m23s
939-line Swift I/O organ (mic/camera capture, speaker playback via
AVFoundation/CoreAudio), own-core LPC voice synthesis/imitation,
consent-gating, and full-duplex barge-in conversation — closing the
hear -> understand -> speak loop entirely on-device.

.gitignore in this dir already excludes bin/ (build output), out/
(captured media), and .consent.json/.resume.json (local runtime state),
so only src + README + .gitignore are committed here.
2026-08-15 14:28:14 -05:00
bigmerge b7e2c580a8 Add native speech synthesis and voice-imitation faculty
El SDK CI - dev / build-and-test (pull_request) Successful in 6m28s
speech.el: formant/glottal integer DSP synthesis + voice-analyze-by-
imitation. voice-profile.el / voice-ingest.el: voice-profile plumbing.
accent.el: British-RP as an ingested transform-geometry (explicitly marked
provisional/citation-pending by its own comments). organ-read.el:
engram read-through for the speech organ. Includes demo/test drivers and
non-personal reference data (British-RP phonetics/lexicon derived data,
a public-domain LibriVox RP reference recording).

Deliberately excludes elp/data/live/ (raw recorded voice + face-photo
samples of the repo owner) and the will-*.{json,psv} derived voiceprint
files — personal biometric data that shouldn't be committed to a shared
repo without an explicit decision from the owner. Also excludes this
worktree's elp/src/surface-profile.el, which diverges from the copy in
other worktrees (agent-aaf04b0a9714c4070, main) — needs manual
reconciliation before landing, left out here to avoid silently picking a
version.
2026-08-15 14:27:52 -05:00
bigmerge 827257d3a4 Remove __pycache__ .pyc files accidentally included in the projector commit
El SDK CI - dev / build-and-test (pull_request) Successful in 6m28s
2026-08-15 14:27:23 -05:00
bigmerge 4bbfdcceff Add native audio/image efferent surfaces + projector proof-of-shape
audio-surface.el / image-surface.el: own-core additive-synthesis WAV and
raster-PNG renderers (integer-only DSP, since EL has no floats), rendered
from learned engram signatures via a pluggable surface-profile
abstraction (surface-profile.el). audio-demo.el / image-demo.el are
drivers. NOTE: demo files hardcode absolute paths to this worktree's own
directory — will need a path fixup before landing.

elp/projector/ is a Python package the author's own README marks as
"STAGING/PROOF-OF-SHAPE — not the deliverable", superseded by the native
.el surface-profile work above; kept as a validated architecture proof.
Generated output (elp/faculty/{out,sig}, elp/projector/out,
__pycache__) intentionally excluded.
2026-08-15 14:26:59 -05:00
bigmerge 5f3ddb8b8d Add grounded edge-propagation spec (task #50): core algorithm, proof harness, gated integration patches
El SDK CI - dev / build-and-test (pull_request) Failing after 14m31s
LTP/LTD-style belief grounding propagated along graph edges, with
union-find independence-guarded corroboration. Package: core C algorithm
(gep_core.h), a self-contained deterministic proof harness with recorded
output, staged runtime integration, and gated .el patches for the beat
hook and HTTP route.

Per the author's own LEDGER.md: built + proven on a clone, GATED pending
the engine/HNSW cutover — not wired into the live beat or routes.
Preserved here as a spec/reference artifact, not a request to merge into
the live path.
2026-08-15 14:26:06 -05:00
bigmerge 708722b7ff Add native interruptibility for dispatched agent workers
El SDK CI - dev / build-and-test (pull_request) Failing after 14m43s
Cancellation-token control channel checked at every step boundary lets a
coordinator PAUSE/RESUME/REDIRECT/KILL a running worker mid-task instead of
waiting for the whole (possibly wrong) plan to finish. Bounded purviews
mean no half-committed state to unwind on interrupt. Includes a proof
harness (proof.el, run.sh) comparing a broken non-interruptible worker
against the new one under identical kill/redirect/pause timing.

Distinct from the already-preserved swarm-ccr orchestrator (fan-out/
converge dispatch): this is single-worker interruptibility, a
complementary mechanism, not a duplicate.
2026-08-15 14:25:55 -05:00
bigmerge 2f832c8def Fix float arithmetic codegen and math_log aliasing
El SDK CI - dev / build-and-test (pull_request) Failing after 10m8s
Float + previously fell through to string concat (segfault); -, *, /, %
operated on raw IEEE-754 bit patterns as integers (garbage results). Floats
are now tracked via a __float_names typed-binding set (parallel to the
existing int-tracking scheme) and arithmetic is emitted as real C double
ops.

Also fixes math_log, which was wrongly aliased to natural log (duplicating
math_ln) — now uses log10 — and adds the missing <math.h> include. Rebuilt
elc binary included.
2026-08-15 14:24:14 -05:00
bigmerge 710bea174d Add native EL afferent ingest organ
El SDK CI - dev / build-and-test (pull_request) Successful in 6m29s
Source-polymorphic ingest(source) primitive: extracts content faithfully
from a directory/file/url/llm-query/structured-primitive-set/stream,
decomposes it into a discrete multi-node graph manifold (nodes + internal
edges, never a single blob), and merges it into the engram geometry with
dedup (search + exact/cosine match), provenance, grounding-level, and
stewardship-class tagging from the moment of entry.

Pure HTTP client of the engram server (links only el_runtime.c, never
el_seed.c/the engine directly). Tested against a live nsbx sandbox engram
clone (127.0.0.1:8903) with real writes confirmed via /api/stats
(node_count 3201 / edge_count 6601).

Excludes ingest/build/ — local compiler scratch output (binaries, .c
codegen, .err logs), not source.
2026-08-15 14:22:07 -05:00
bigmerge 05e5d3c402 self-review 2026-08-15: consolidate the strongest Hebbian candidate, not the lowest-hash one
The link-formation scan walked candidate slots ascending and stopped at
ENGRAM_HEBB_LINK_PER_CALL (2). Slot index is a hash of the node id pair, so
whenever more than two candidates cleared LINK_MIN in the same call, the two
consolidated were the two with the lowest hash and a stronger association
waited - indefinitely, since the scan restarts from slot 0 every call while
the leader decays at ENGRAM_HEBB_DECAY.

Measured 08-13..08-15: hebb_cand_max peaked at 0.4963, 3.3x LINK_MIN, during
a ~14h stretch of continuous qualification at the 2/call cap.

Same defect the 2026-08-02 review named and fixed for breakthrough weights
(index order is not a cognitive criterion), never carried across to the one
path that writes permanent structure - and there is no pruning path, so
growth is one-way. Selection pressure matters most where the result is
irreversible.

No-op when <=2 candidates qualify; picks the best when more do.
2026-08-15 08:44:37 -05:00
bigmerge 7e4b21c779 Add sandbox: multi-repo stack worktree composer (el-stack / neuron-stack)
El SDK CI - dev / build-and-test (pull_request) Successful in 6m19s
Assembles every constituent repo of a stack into one combined worktree
workspace, laid out at natural relpaths so cross-repo ../foundation/el
imports resolve to the sandbox copy. Sibling of nsbx; pure bash + git
worktree; never touches live :8742/:7770; isolated engram delegated to nsbx.
2026-08-15 00:55:22 -05:00
Neuron 54378c7355 elp(translate): refactor to geometry-native concept-pivot
El SDK CI - dev / build-and-test (pull_request) Successful in 6m57s
Drop the bilingual-string-table framing and the external-encoder plan (both
wrong). Translation now routes source-lexicon -> concept-frame (language-
invariant, in the engram concept geometry) -> target-realizer, exactly as the
ELP was designed: a word resolves to the CONCEPT it denotes via its own
language's lexicon (a monolingual step — the engram nearest-region ranker only
disambiguates senses within one language, so an English-trained embedder is
fine and never compares 'ocean'~'oceano' as strings). The concept node is the
shared pivot; its manifold location is the meaning.

- Pronouns route through the NATIVE concept pivot (cp_pron_concept ->
  cp_rom_pron_surface) instead of an ad-hoc EN->tgt string map.
- lemma_for_concept / noun_for_concept are each target language's own
  CONCEPT->SURFACE lexicon (the mirror of comprehend's SURFACE->CONCEPT).
- Fidelity is concept-preservation (concept_frame fingerprint), not string
  cosine against an external multilingual model.
- Plural article agreement fixed (las/los, as/os).

Verified: 'You never fought the ocean.' -> ES 'Usted nunca luchó el océano.'
concept-frame pivot 'pred=fight patient=ocean pol=neg' realizes to ES+PT from
one parse; nunca holds 3/3. Gaps unchanged: PT verb conjugation fallback,
adjunct/subordinator concepts not yet in-frame.
2026-08-14 17:53:00 -05:00
Neuron 640e8799e5 elp(translate): normalize EN irregular pasts before the lemma bridge
comprehend lemmatizes some irregulars (fought->fight) but not all (broke);
tr_norm_verb covers the poem's remainder so affirmative content verbs route
(ES 'Yo broo' -> 'Yo rompo'). Negation lines unchanged and still correct.
2026-08-14 17:38:13 -05:00
Neuron 9b63a2a23b elp(translate): EN->ES/PT geometric-free translation faculty
Adds the missing middle of the ELP: a deterministic EN-content-lemma ->
target-lemma bridge (translate.el) on top of comprehend.el (parse) and
realizer.el (inflect). English-only engram geometry cannot route
cross-lingually and vocabulary-XX.el carries no en_translation glosses, so
the honest no-LLM bridge is a wired lexicon (poem coverage; OOV passes
through). SACRED polarity/neg_word are carried untouched: 'never' localizes
to a negator ('nunca'), never to a content lemma.

Additive realizer extensions: agent_person/agent_number recognize Romance
target pronouns; the non-EN negation branch surfaces a carried neg_word
instead of the generic negator.

Verified on the real toolchain (elc->cc->run):
  'You never fought the ocean.' -> ES 'Tú nunca luchaste el océano.'
  'I never saw the breaking.'   -> ES 'Yo nunca vi la ruptura.'
nunca holds 3/3 negation lines. Known gaps: PT verb conjugation fallback
(lutarred), irregular EN lemma (broke->break), adjunct/subordinator passthrough.
2026-08-14 17:37:01 -05:00
will.anderson ce34b94f88 elp(dialogue+self_region): native-el summon-through-self port + scratch-verified gate
Ports dialogue.py + self_region.py to native el, bound to the IN-PROCESS engram
el runtime (engram_activate_json / engram_neighbors_json / engram_search_json /
engram_node_full / engram_connect — C-order builtins, not the wrapper order).

self_region.el: pulls the engram's REAL Self/identity nodes (pooled single-term
search), scores by self-signal, reads out identity from their own prose — no
hardcoded anchors, no template.

dialogue.el: ONE operation — project(query) -> land on a region -> read out.
  * identity = self-region proximity (no intent classifier, no separate branch)
  * memory = activation + a RELEVANCE FLOOR, then MATERIALIZE by walking the
    neighborhood (real edges), never top-props
  * HONEST ABSENCE when nothing is close — no 'I noted that' echo, no fabrication
  * NEGATION SACRED: readout is the stored prose verbatim, so polarity survives
  * DIRECTIVE OVERRIDE: a meta-directive switches the reply language

Verified against a SCRATCH in-process engram (live :8742 untouched): dialogue
gate 9/9 — identity from real self-content, neighborhood materialization,
SACRED negation (self + memory), PT identity in PT, directive override to
English, 'Prove it' -> honest absence. EN/Romance/prop/multilingual gates
unregressed.
2026-08-13 15:56:08 -05:00
will.anderson 0ae33c0f3b elp(realizer): close subordinate-clause round-trip + silent-e/doubling lemmatizer + verb-final object bug + ES/PT closed-class verb guard
- realizer now carries the subordinate clause verbatim (subord_text slot): the
  5th English acceptance sentence is byte-identical through parse->realize->reparse.
- English -ed/-ing lemmatizer restores silent-e (loved->love) and collapses
  inflectional doubling (stopped->stop), inverting en_verb_past().
- parse_spec_lang: a sentence-final main verb no longer bleeds into the object
  slot (cstart advanced to verb+1), so intransitives round-trip.
- cp_rom_is_verb rejects closed-class words (prep/det/pron/aux/neg) before the
  ending-only test, killing the 'para'/determiner misfires.

EN telephone gate 5/5 (now byte-identical 5/5), Romance gate 6/6.
2026-08-13 15:39:49 -05:00
will.anderson c5508372ca elp(multilingual): native-el language layer — detect + localized phrases
Phase 3 piece 2. Ports multilingual.py: deterministic language detection
(en/es/pt/it) via stopword + diacritic scoring, localized fixed phrases (SACRED
per-language yes/no/decline/identity), PT/ES->EN retrieval term lexicon, and
EN->target predicate translation. No generative model.

Gate (multilingual_gate.el): 4/4 languages detected correctly; localized
declines + term/pred lexicons verified. Built bounded (elc rc=0 peak 25MB).

Worked through the documented el '+' mis-compile (two chained function-call Int
operands compile as string concat -> corrupt Int -> segfault on the accented
path); fixed by binding each score to an Int var and adding vars singly.

Simplifications (honest): diacritics scored by PRESENCE (str_contains) not
codepoint count (UTF-8 index safety); confidence scalar and the regex-based
parse_directive() from the reference not yet ported (directive parsing deferred
to the dialogue layer).
2026-08-13 15:09:04 -05:00
will.anderson 335298a518 elp(propositions): native-el READ primitive — memory text -> SACRED triples
Phase 3 piece 1. Ports propositions.py off spaCy: the dependency-parser role is
now the el-native parser (parse_spec), and each memory sentence's meaning-spec
IS the triple (subject, predicate, object, modifiers, polarity, tense, source,
confidence). Sentence segmentation + repr parity with propositions.py. NEGATION
SACRED: polarity flows straight from the spec, never dropped/inverted.

Gate (propositions_gate.el): 4/4 SACRED polarity correct on extraction;
multi-sentence memory splits one triple per sentence in reading order with
negation preserved. Built bounded (elc rc=0 peak 24MB, cc rc=0).

Gap (honest): English regular-verb lemmatizer does not restore silent-e
(stores->stor); coreference/passive normalization from the reference not yet
ported (shallow pronoun subject kept as surface).
2026-08-13 14:56:57 -05:00
will.anderson 7d4fdbcc22 elp(comprehend): ES/PT Romance parser path — SACRED polarity cross-lingual
Adds a deterministic Romance front-end to comprehend.el (parse_spec_romance),
dispatched from parse_spec_lang for lang es/pt. English path untouched
(byte-identical, regression gate still 5/5). Pro-drop aware clause skeleton
(subject | neg | verb | object | PP), cross-lingual negation lexemes already
SACRED. Romance telephone gate: polarity PRESERVED 6/6 and EXTRACTED 6/6
through parse->realize->re-parse for 3 ES + 3 PT sentences. Built bounded
(elc rc=0 peak 24MB, cc rc=0).

Named gaps (honest): ending-only verb detection misfires on prepositions
(contra) and -a/-o nouns (menina); lemma recovery keeps surface form; the
non-English realizer is a generic preverbal-negator skeleton so ES/PT surfaces
are not byte-parity. Full paradigm inversion + Romance lexicon deferred.
2026-08-13 14:55:14 -05:00
will.anderson 89ea1b5a15 elp(comprehend): el-native comprehension parser + SACRED polarity end-to-end
PIECE 1 — greenfield el-native parser (comprehend.el), spaCy-free:
- text -> meaning-spec via invertible English morphology (the realizer's own
  irregular table run BACKWARD) + a deterministic clause grammar (subject/verb
  boundary, roles, ditransitive iobj, PP adjuncts, subordination, coordination).
- NEGATION IS SACRED: explicit polarity field, always present, cross-lingual
  lexeme set; standalone neg adverbs (never) captured separately.
- WSD by deterministic syntactic position over a fixed sense inventory
  (flies->fly, like->comparison, saw->see); engram nearest-region is the
  documented runtime upgrade hook (no external model).

Polarity threaded through the whole el contract (was previously dropped at the
boundary): realizer.el realize_lang honors polarity (English do-support /
adverbial / copular negation; generic preverbal negator for es/pt/ca/it/fr/de/ro)
and places iobj; elp.el build_form_from_json carries polarity/neg_word/iobj
across JSON; morphology.el gains 'fight'.

Acceptance (native el telephone test, comprehend_gate.el): on the 5 gate
sentences polarity PRESERVED 5/5 and EXTRACTED 5/5 through parse->realize->
re-parse; 4/5 byte-identical. Built bounded (elc rc=0, cc rc=0).
2026-08-13 13:41:09 -05:00
will.anderson a816b119e7 stage(elp): consolidate scattered lang work — full-lexicon vocabulary + profiles
Backfill ELP vocabulary from FULL lexicons (UniMorph + kaikki.org Wiktionary,
real gender/inflections) for 8 languages, 812,894 entries total, in the proven
seed-fn format matching the 18 ancient vocabularies:
  es 72,032 | fr 130,517 | de 144,692 | la 22,590 | it 193,675 | pt 115,772 |
  ro 86,504 | ca 47,112
4 of these (es fr de la) backfill ELP languages that had morphology but no
vocabulary; it/pt/ro/ca are new Romance (need morphology-*.el ports next).
Adds lang_profile_* for all 8 + reproducible generators under tests/lang-gen.
Vocab is runtime seed data (not in build manifest, like the 18 ancients);
seed-fn format validated to compile to C via elc.
2026-08-13 11:56:03 -05:00
will.anderson 4f49755ebb Merge pull request 'ci: make official engram build store-enabled (publish + link engram_store.{c,h})' (#95) from engram-tiered-storage into dev
El SDK Release / build-and-release (pull_request) Failing after 40s
El SDK CI - stage / build-and-test (push) Failing after 31s
El SDK CI - stage / build-and-test (pull_request) Failing after 34s
El SDK CI - dev / build-and-test (push) Failing after 3m53s
El SDK Release / build-and-release (push) Failing after 12m31s
2026-08-12 20:23:34 +00:00
will.anderson 7aa847e32a Merge origin/dev into engram-tiered-storage
El SDK CI - dev / build-and-test (pull_request) Failing after 10m51s
Resolve 3 conflicts:
- lang/el-compiler/runtime/el_runtime.c: keep deletion (deprecated runtime fork;
  single-source-of-truth is lang/runtime/, enforced by scripts/check-single-runtime.sh).
- lang/releases/v1.0.0-20260501/el_runtime.h: keep deletion (releases/ is a generated
  artifact folder, not a source path; a release is a git tag, not a folder).
- lang/runtime/el_platform_win.h: union of dev's Windows port (#80: setsockopt optval
  wrapper + curl-less libcurl stubs) and our fsync(->_commit) shim needed by engram_store WAL.

Nothing in dev's build consumes the deprecated fork or releases/ folder.
2026-08-12 15:23:02 -05:00
will.anderson ee71423732 ci: publish + link engram_store.{c,h} so official builds are store-enabled
El SDK CI - dev / build-and-test (pull_request) Failing after 13m20s
The live engram now runs the paged store (neuron.egm+WAL), but the SDK
release publishes only el_runtime.{c,h} and the engram build links only
el_runtime.c — so a future official release would silently revert to the
in-memory store. Publish engram_store.{c,h} as SDK release assets and add
them to the engram build's download + cc link so the store transition
cannot regress.
2026-08-12 14:22:25 -05:00
will.anderson bb64a236ed engram tiered storage: engram-service wiring + elc fold-hang fix + prune-store mirror
- Wire paged store into the ENGRAM SERVICE (server.el, the authoritative durable
  owner): boot->engram_store_boot, persist_canonical->engram_store_checkpoint,
  gated by ENGRAM_STORE.
- elc (lang/elc.c + src/parser.el + codegen.el + elc-combined.el): OOB guard in
  tok_kind/tok_value + parse_block progress backstop — fixes the pre-existing
  unbounded-memory fold hang on sessions.el.
- engram_prune_telemetry mirrors ISE prune to the store (store_forget) so store
  live-count tracks resident and stale telemetry stays bounded.
- Deployed live 2026-08-12: engram :8742 on neuron.egm+WAL, count reconciled 11552.
2026-08-12 14:14:20 -05:00
will.anderson 9a0266cbf9 engram tiered storage M3.5: persist activation field updates (pre-flip gate)
Flag-on checkpoint now full-walks the resident graph: store_put_node (WM weight,
activation_count, last_activated, wm_anchor) + store_put_edge (hebb, last_fired)
for every node/edge, then engram_checkpoint. Uses store_put_edge (idempotent
upsert) not store_hebb_batch, because activation FORMS new hebbian-associate edges
that bypass the create hook and delta-only hebb_batch can't create them. Store-on
boot now applies the same WM-halving + floor + cap transforms as engram_load.

This is the hebb-survives-restart fix. Gate: reboot from neuron.egm with
snapshot.json deleted -> edge hebb + activation_count survive unchanged, WM weight
survives with identical boot transform; negative control proves persist is
load-bearing (hebb->0 without it). M1 33/33 + M2 36/36 + M3 parity PASS, ASan/UBSan
clean, flag-off untouched. Engine unchanged (boundary held).
2026-08-11 23:37:52 -05:00
will.anderson a72145b44e engram tiered storage M3: wire store behind ENGRAM_STORE (default off) + .egm rename
Caller-side shim in el_runtime.c maps EngramNode/Edge <-> StoreNode/Edge; engine
keeps zero soul deps (libengram boundary, design §10). Flag off = today's JSON
path byte-for-byte (proven: no neuron.egm created, graph identical). Flag on =
engram_open (import snapshot.json once into neuron.egm, else WAL-replay) +
resident load; node/edge create + forget dual-write via guarded hooks. Files
renamed engram.store->neuron.egm, engram.wal->neuron.wal.

Gate: M3 parity PASS (graph on==off byte-exact modulo ordering; snapshot round-trip;
reboot-from-egm with snapshot.json deleted; activation set+sequence identical;
ASan/UBSan clean). M1 33/33 + M2 36/36 green post-rename.

Known gap (pre-flip): in-place hebb/WM/activation_count updates during activation
are not yet persisted to the store (create/connect/forget are). Must close before
live flip so learned edges survive restart.
2026-08-11 23:21:21 -05:00
will.anderson 8affb1d6e0 engram tiered storage M2: WAL + checkpoint + crash recovery + legacy import
Write-back no-steal buffer pool makes the fsync'd WAL load-bearing (M1 was
write-through). Logical WAL with record-granularity page-LSN redo idempotency.
Checkpoint = flush dirty pages, fsync store, advance last_checkpoint_lsn,
reclaim WAL prefix. One-time snapshot.json import only when store absent;
JSON never read as the ongoing store thereafter.

Gates: 33/33 M1 (no regression) + 36/36 M2 — replay parity, torn-tail fuzz
(every byte offset), checkpoint-crash at all 5 phases, torn-page+WAL redo,
legacy-import parity, hebb-survives-crash.
2026-08-11 23:00:20 -05:00
will.anderson fa47b98d18 engram tiered storage M1: on-disk paged store format + round-trip tests
Self-contained paged store (lang/runtime/engram_store.{c,h}): 16KiB slotted pages,
u32 TLV self-describing records (forward-compatible), overflow chains, B+-tree
id-index + from/to adjacency, page free-list, tombstones, double superblock + crc
recovery. Not yet wired to activation (M3). 33/33 tests pass (ASan/UBSan clean);
5k nodes/20k edges round-trip bit-exact incl 768xf32 emb + hebb; store 25MB vs 64MB
JSON. Format is final — see design §2.4.
2026-08-11 22:26:05 -05:00
will.anderson 0a72fced28 engram: WAL persistence + integrity hardening + single canonical runtime
El SDK CI - dev / build-and-test (pull_request) Failing after 13m17s
Establish lang/runtime/ as the ONE canonical el runtime (from the active
runtime that carries hebb/emb persistence + the new WAL); repoint the el CI
publish, engram build, elb default, and in-repo build scripts to it; delete
the el-compiler/runtime + lang/releases/ forks; add scripts/check-single-runtime.sh
drift guard.

Fixes a live prod bug: the el CI published el-runtime-c/-h from the LAGGING
el-compiler fork (0 hebb refs), so the shipped soul never persisted Hebbian
edge weights — learned co-activation was wiped on every restart. Publishing
from canonical ships the stranded 'learning that cannot outlive the process'
fix.

WAL storage engine + integrity fixes (DELETE->tombstone + store-layer
protection, safe data-dir default) ride in behind ENGRAM_WAL (default off =
byte-identical to today). Verified: engram elb per-module build clean, WAL
gate 66/66, native smoke ok, drift-guard green.
2026-08-11 21:31:37 -05:00
will.anderson edcec3bdf4 engram: add /api/nodes/reseed so a node body can be repaired at its own id
El SDK Release / build-and-release (pull_request) Failing after 11m24s
Two write paths could put a node in the graph and neither could put a body
on an id that already exists. POST /api/nodes mints a fresh id via
engram_node_full; POST /api/load-merge honors a declared id but skips
anything already present. That is right for the additive case and leaves a
hole: a node resident with a truncated body cannot be repaired.

Forge's genesis seed sits in that hole. Two of Neuron's identity nodes
carry only their own label as content -- 30 and 22 bytes against 4263 and
2590 declared. Their ids are load-bearing (is_protected_node keys on them
and 214 declared edges reference them), so recreating them under a new id
is not a repair, it is a second break.

Engram has no in-place node update, so a replace is forget-then-merge, and
engram_forget also drops every incident edge -- 85 and 93 on those two
nodes, nearly all tag edges and accumulated hebbian associations the seed
does not declare and could not restore. preserve_edges (default true)
therefore snapshots before the forget and re-merges after: the replaced
node is back by then so it is skipped, and every dropped edge returns
through the (from_id,to_id,relation) dedup. The same re-merge is the
failure path -- if the seed merge does not produce the node, the backup
puts the original back. Rollback, not data loss.

With no replace list the route is exactly /api/load-merge.

Verified on a sandbox engram seeded to mirror the live graph's state for
this seed (15 resident nodes, 694 incident edges): 87 nodes created at
their declared ids, 2 replaced in place, 214/214 edges laid, 682/682
non-seed incident edges preserved, and a second run reports 0 added.
2026-08-10 16:44:17 -05:00
Neuron 866c75e5e2 fix(codegen): emit the declared cgi identity — it was searched for in a list that cannot contain it
El SDK Release / build-and-release (pull_request) Failing after 13m58s
El SDK CI - dev / build-and-test (pull_request) Failing after 10m32s
A cgi block is a top-level declaration, so codegen_streaming classifies it via
is_top_level_decl and releases it. The identity emission then searched
toplevel_exec_stmts for that same block. Declarations are excluded from that list by
construction, so the search could never succeed. A probe printed what it actually
saw for a program whose first statement is a cgi block: [Let, Expr]. It emitted
nothing, silently, with no diagnostic on any channel.

The code documented its own assumption — 'Since cgi blocks are rare and small, they
end up in toplevel_exec_stmts' — and that assumption was false.

Capture the declared values before the release and emit from them. The search is
deleted rather than repaired, so the failure mode is removed rather than relocated.

Proven discriminating (old fails, new passes):
  minimal cgi program, old   -> 0 el_cgi_init
  minimal cgi program, fixed -> el_cgi_init with all four declared values
  neuron soul, fixed         -> principal present in the compiled binary (0 before),
                                boots in 2s, interface 110 routes in / 110 out

Consequence: a binary now carries its declared identity as a compiled constant,
which is what the identity protocol requires. Whether the runtime surfaces it to
state_get("soul_principal") is unverified and separate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 13:48:27 -05:00
will.anderson 5d0d4555ae Sync main into dev (GitOps: keep dev current; main authoritative) (#84)
El SDK CI - dev / build-and-test (push) Successful in 8m32s
2026-08-03 15:38:40 +00:00
will.anderson 8347a2f1c0 Merge pull request 'docs: add root README mapping the El monorepo' (#83) from feat/AddingReadme into dev
El SDK CI - dev / build-and-test (push) Successful in 8m22s
2026-07-31 04:25:44 +00:00
Andre Botelho Rodrigues Almeida b97b644799 Addind readme.md file to start documenting the repo
El SDK CI - dev / build-and-test (pull_request) Successful in 8m18s
2026-07-23 16:41:51 -03:00
will.anderson d71fc4c1c0 Merge pull request 'promote stage -> main: reconciled el runtime (engram search + natives + durable truncation fix + Windows port)' (#82) from stage into main
El SDK Release / build-and-release (push) Successful in 8m31s
El SDK CI - dev / build-and-test (pull_request) Successful in 8m41s
2026-07-22 21:44:01 +00:00
will.anderson a118d19393 Merge pull request 'promote dev -> stage: el cluster (#66 engram + #79 truncation fix + release-runtime Windows port)' (#81) from dev into stage
El SDK CI - stage / build-and-test (push) Successful in 7m58s
El SDK Release / build-and-release (pull_request) Successful in 4m16s
2026-07-22 21:20:17 +00:00
will.anderson c6aa1e5c53 Merge pull request 'Land el cluster: #66 engram search + natives, #79 truncation fix, + release-runtime Windows port (reconciled)' (#80) from reconcile/el-cluster-windows-runtime into dev
El SDK CI - dev / build-and-test (push) Successful in 8m3s
El SDK CI - stage / build-and-test (pull_request) Successful in 4m25s
Land el cluster (#66 + #79 + release-runtime Windows-port reconciliation) into dev
2026-07-22 21:06:36 +00:00
will.anderson ff577391f2 reconcile(release-runtime): Windows-port + complete v1.0.0 release runtime so the desktop soul cross-compiles
El SDK CI - dev / build-and-test (pull_request) Successful in 7m7s
The desktop soul (neuron/dist) compiles against the v1.0.0-20260501 release
runtime. After #66 landed the engram natives (tokenized/ranked search,
engram_prune_telemetry) and #79 the durable truncation fix into this runtime,
two gaps remained before it could cross-compile the Windows brain:

1. Windows OS boundary: the release runtime had no Win32 path. Ported the same
   _WIN32-guarded shim the mainline runtime carries (#69): #ifdef _WIN32 ->
   el_platform_win.h (winsock/dlsym/popen + WSAStartup ctor), SOCKET fd guards
   and el_closesocket() at every socket site, CreateProcessA for exec_bg, the
   tm_zone/mingw guard, an el_setsockopt optval wrapper (GCC14), and curl-less
   libcurl stubs. Every change is _WIN32/HAVE_CURL-gated — the POSIX build is
   byte-identical (gcc -fsyntax-only clean; native behaviour unchanged).

2. Header exports: the release el_runtime.h omitted symbols the soul dist calls
   that are defined in this runtime's .c — the http_handler_fn/http_handler4_fn
   typedefs and el_arena_push/pop, engram_prune_telemetry, engram_get_node_by_label.
   Declaration-only, POSIX-neutral; fixes implicit-declaration/unknown-type
   errors under the C11 mingw build.

Result: x86_64-w64-mingw32-gcc compiles el_runtime.c + all 48 soul modules
clean; POSIX gcc -fsyntax-only clean. This is the Windows-port PR the runtime
needed on main (the release-runtime counterpart to #69), landed via stage.
2026-07-22 15:56:08 -05:00
will.anderson ee0d5f9b97 Merge #79: durable HTTP response-truncation fix, both runtimes (via stage) 2026-07-22 15:46:02 -05:00
will.anderson 391bd818ea Merge #66: tokenized+ranked engram lexical search + engram natives (via stage) 2026-07-22 15:45:54 -05:00
will.anderson 43636aed99 runtime: pair fs_read length hint with its buffer in BOTH runtimes — kill response truncation for good
El SDK Release / build-and-release (pull_request) Failing after 7s
The binary-safe fs_read length (_tl_fs_read_len) was consumed by the HTTP
response path for ANY body, even when a handler wrapped a smaller file into a
larger reply. Content-Length then lied AND the send stopped short: the
safety-contact (988) routes returned 178 of 208/218 bytes, cut mid-'set_at' —
unparseable JSON. The desktop app read that as failure. On Windows the shipped
brain is an OLD build without even the per-handler workaround, so EVERY reply
truncated: the app can't read confirmations and refuses the new user.

Durable fix: pair the length hint with the exact buffer pointer it describes
(_tl_fs_read_buf). Apply the raw byte count ONLY when the response IS that
buffer (binary file serving stays correct); every wrapped/enveloped/derived
body is measured with strlen. Reset both at request start and in fs_read /
json_get_raw. This also closes the stale-hint heap over-read (a length larger
than a later body would read past it out the socket) that a plain max() leaves
open — so this class of bug dies on every platform, not just where a handler
happened to be patched.

Applied identically to the mainline runtime (lang/el-compiler/runtime) AND the
frozen release runtime (lang/releases/v1.0.0-20260501) the desktop souls
compile against — the release copy still carried the raw leak, which is why the
Windows brain kept truncating. Same proven approach as PR #78 (Tim Lingo),
extended to cover the release runtime and rebased onto current main.

Both runtimes: gcc -fsyntax-only clean.
2026-07-22 15:04:25 -05:00
will.anderson 2baa0b9a41 Merge pull request 'release: promote stage -> main (ci publish hardening for sdk-release)' (#77) from stage into main
El SDK Release / build-and-release (push) Successful in 7m55s
2026-07-15 21:21:28 +00:00
will.anderson 6a8b2461cd Merge pull request 'release: promote dev -> stage (ci publish hardening for stage/main)' (#76) from dev into stage
El SDK CI - stage / build-and-test (push) Successful in 8m19s
El SDK Release / build-and-release (pull_request) Failing after 13m1s
2026-07-15 21:16:11 +00:00
will.anderson bcb356fe69 Merge pull request 'ci(stage,main): decouple ci-base rebuild, make SDK publish fail loudly' (#75) from hotfix/ci-stage-main-publish-hardening into dev
El SDK CI - stage / build-and-test (pull_request) Successful in 4m27s
El SDK CI - dev / build-and-test (push) Failing after 14m3s
2026-07-15 21:15:27 +00:00
will.anderson dd7827059a ci(stage,main): decouple ci-base rebuild, make SDK publish fail loudly
El SDK CI - dev / build-and-test (pull_request) Failing after 14m30s
Mirror the PR #72 fix (applied to ci-dev.yaml) onto ci-stage.yaml and
sdk-release.yaml. The stage and prod release jobs reported FAILURE even
when the el-runtime-c/-h publish SUCCEEDED, because the ancillary ci-base
Docker rebuild (a CI-cache optimization on the fragile host-mode GCE
runner) reddened the whole job.

- Rebuild ci-base step: continue-on-error: true — never blocks/reddens
  the job; the SDK publish is the deliverable.
- Publish step: set -euo pipefail + empty-key guard + active-account echo
  so a real publish failure still fails loud and is diagnosable.
2026-07-15 16:14:50 -05:00
will.anderson 208e36c899 Merge pull request 'release: promote stage -> main (tokenized search, get_node_by_label, epm fix, win portability)' (#74) from stage into main
El SDK Release / build-and-release (push) Successful in 8m23s
2026-07-15 18:24:39 +00:00
will.anderson b97ce74d1f Merge pull request 'release: promote dev -> stage (tokenized search, get_node_by_label, epm fix)' (#73) from dev into stage
El SDK CI - stage / build-and-test (push) Failing after 8m45s
El SDK Release / build-and-release (pull_request) Successful in 4m1s
2026-07-15 17:20:11 +00:00
will.anderson 155a449c4e Merge pull request 'ci(dev): make SDK publish fail loudly, decouple ci-base rebuild' (#72) from hotfix/ci-dev-publish-hardening into dev
El SDK CI - dev / build-and-test (push) Successful in 8m56s
El SDK CI - stage / build-and-test (pull_request) Successful in 4m10s
2026-07-15 16:34:14 +00:00
will.anderson 4696fd6833 ci(dev): make SDK publish fail loudly, decouple ci-base rebuild
El SDK CI - dev / build-and-test (pull_request) Successful in 8m51s
The dev push build went green-then-red while nothing published: the
Publish step had no set -e, so an auth/upload failure exited 0 (silent
no-publish), while the ci-base rebuild (set -euo pipefail + Docker on the
host-mode runner) hard-failed the job. Add set -euo pipefail + an empty-key
guard + active-account echo to the Publish step so failures surface with a
retrievable log, and mark the ci-base cache rebuild continue-on-error so
the fragile Docker step can never block the actual SDK artifact publish.
2026-07-15 11:33:37 -05:00
will.anderson 581a351fb1 Merge pull request 'integrate: stack PRs #65–#69 (elc OOM guard, tokenized+semantic engram search, get_node_by_label, win portability) for green CI' (#71) from hotfix/stage-elc-engram-integration into dev
El SDK CI - dev / build-and-test (push) Failing after 14m31s
2026-07-15 15:49:57 +00:00
will.anderson 8ce8656de2 epm: declare cross-module callees as extern fn so strict compilers accept generated C
El SDK CI - dev / build-and-test (pull_request) Successful in 7m33s
epm's sibling modules (registry/install/update) call functions defined in other
modules and in the El runtime (config, read_installed, registry_find,
manifest_deps, manifest_name, registry_latest_version, registry_token,
install_vessel, installed_version) without importing them, so elc emits no C
prototype for those calls. gcc<=13 treated the resulting implicit declarations
as warnings; gcc>=14 and clang reject them as hard errors, which is why the
"Build epm" CI step fails and blocks the whole dev/stage pipeline.

Add `extern fn` forward declarations -- El's own separate-compilation mechanism
-- for each cross-module callee at the top of registry/install/update. This
gives elc the correct C prototype in every generated translation unit, so the
calls compile cleanly and still resolve at link time. Simply suppressing
-Wimplicit-function-declaration would be unsafe: an implicit int return
truncates the 64-bit pointer returns of config/registry_find into a latent
crash, so declaring the true signatures is the correct fix. Localized to epm;
touches neither elc nor the runtime.
2026-07-15 10:14:43 -05:00
will.anderson 1e49560f1f Merge remote-tracking branch 'origin/feat/engram-semantic-search' into hotfix/stage-elc-engram-integration
El SDK CI - dev / build-and-test (pull_request) Failing after 14m39s
# Conflicts:
#	lang/el-compiler/runtime/el_runtime.c
2026-07-15 09:33:05 -05:00
will.anderson e8f0b5a9de Merge remote-tracking branch 'origin/fix/engram-lexical-tokenized-search' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 40287c4cfc Merge remote-tracking branch 'origin/hotfix/win-runtime-portability' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 0481bea44d Merge remote-tracking branch 'origin/hotfix/runtime-engram-get-node-by-label' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 9d565ca080 Merge remote-tracking branch 'origin/hotfix/elc-fixes' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 4773dd0aa2 runtime: make Windows soul reproducible from a clean el checkout
El SDK Release / build-and-release (pull_request) Failing after 16s
Two el_runtime portability defects only ever lived in staged local copies
used to hand-build neuron-ui PR #136's curl-enabled Windows neuron.exe.
gcc 15 promotes both to hard errors, so a clean el checkout cannot rebuild
that soul. Upstream the minimal fixes so the build is reproducible:

- http_serve_async: cast setsockopt optval to (const char*). Win32/mingw
  setsockopt wants const char*, not int*; the cast is a no-op on POSIX and
  matches the four already-cast sites elsewhere in this file.
- engram_save persist path: map fsync -> _commit in the _WIN32-only
  el_platform_win.h (io.h already included). Windows has no fsync(); the
  POSIX path is untouched.
2026-07-15 04:24:08 -05:00
will.anderson 6b9d9e6c4a Add engram_get_node_by_label runtime native to unblock soul link
El SDK Release / build-and-release (pull_request) Failing after 22s
chat.el calls the runtime native engram_get_node_by_label to fetch
well-known nodes (conv:history, session:summary) by stable label rather
than by ID — immune to vector-index drift across restarts. The current
runtime never defined it, so the regenerated dist/soul.c fails to link.

Backport the function verbatim (idiom-adapted to jb_finish) from release
runtime v1.0.0-20260501 and register it as an EL builtin exactly like its
siblings: runtime definition + prototype, __-prefixed seed wrapper +
prototype, and codegen arity entry. No search-site code is touched.
2026-07-15 04:07:33 -05:00
will.anderson b4967af13e feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical)
Lexical istr_contains alone can't surface a node whose words don't appear
in the query. This adds an optional dense-vector layer: node content and the
query are embedded through Ollama (nomic-embed-text), and nodes are ranked by
cosine similarity unioned with lexical hits, so a paraphrase query reaches the
right node.

Wired into all three query entry points in el_runtime.c:
  - engram_search_json (HTTP /api/search): collect lexical ∪ semantic
    candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine),
    rank, emit top-N. Stable sort preserves old order when semantic is off.
  - engram_search (internal el_val twin): lexical ∪ semantic union.
  - engram_activate seed loop (HTTP /api/activate): a node seeds if it
    lexically matches OR clears the cosine threshold; pure-semantic seeds
    enter scaled by cosine so paraphrase spreads without overpowering.

Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot
runtime probe. If curl is compiled out, Ollama is unreachable, or
ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers
fall back byte-for-byte to the pre-existing lexical search.

Node embeddings are cached in process memory keyed by node id with an FNV-1a
content hash for invalidation; the query is embedded once per call — so the
graph is not re-embedded on every query. nomic task prefixes
(search_query:/search_document:) are applied for retrieval separation.

Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in
(-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL,
ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).
2026-07-14 18:48:16 -05:00
will.anderson 2b2a1246e7 Merge pull request 'runtime: fix the memory-leak + write-corruption pair in el_runtime.c' (#64) from hotfix/el-runtime-leak-and-persist into main
El SDK Release / build-and-release (push) Failing after 10m52s
2026-07-13 21:23:31 +00:00
will.anderson 5c41c66a0f Merge pull request 'fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only' (#60) from fix/windows-rusage-guard into stage
El SDK CI - stage / build-and-test (push) Failing after 13m21s
fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only
2026-06-25 16:48:13 +00:00
184 changed files with 837743 additions and 24567 deletions
+37 -22
View File
@@ -39,9 +39,9 @@ jobs:
run: |
dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c
gcc -O2 \
-I el-compiler/runtime \
-I runtime \
dist/elc-gen2.c \
el-compiler/runtime/el_runtime.c \
runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/platform/elc
chmod +x dist/platform/elc
@@ -54,9 +54,9 @@ jobs:
mkdir -p dist/bin
dist/platform/elc elb.el > dist/elb.c
gcc -O2 \
-I el-compiler/runtime \
-I runtime \
dist/elb.c \
el-compiler/runtime/el_runtime.c \
runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/bin/elb
chmod +x dist/bin/elb
@@ -91,7 +91,7 @@ jobs:
- name: Precompile el_runtime.o
run: |
set -euo pipefail
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
gcc -O2 -c -I "$RUNTIME" "$RUNTIME/el_runtime.c" \
-o /tmp/el_runtime.o
echo "el_runtime.o compiled"
@@ -100,7 +100,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_core.el > /tmp/el_native_core.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_core
@@ -110,7 +110,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_text.el > /tmp/el_native_text.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_text
@@ -120,7 +120,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_string.el > /tmp/el_native_string.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_string
@@ -130,7 +130,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_math.el > /tmp/el_native_math.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_math
@@ -140,7 +140,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_state.el > /tmp/el_native_state.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_state
@@ -150,7 +150,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_time.el > /tmp/el_native_time.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_time
@@ -160,7 +160,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_json.el > /tmp/el_native_json.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_json
@@ -170,7 +170,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_env.el > /tmp/el_native_env.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_env
@@ -180,7 +180,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_fs.el > /tmp/el_native_fs.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c /tmp/el_runtime.o \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_fs
@@ -191,7 +191,7 @@ jobs:
run: |
ABS_ELB="$(pwd)/dist/bin/elb"
ABS_ELC="$(pwd)/dist/platform/elc"
ABS_RUNTIME="$(pwd)/el-compiler/runtime"
ABS_RUNTIME="$(pwd)/runtime"
ABS_OUT="$(pwd)/dist/bin"
(cd ../epm && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
chmod +x dist/bin/epm
@@ -202,7 +202,7 @@ jobs:
run: |
ABS_ELB="$(pwd)/dist/bin/elb"
ABS_ELC="$(pwd)/dist/platform/elc"
ABS_RUNTIME="$(pwd)/el-compiler/runtime"
ABS_RUNTIME="$(pwd)/runtime"
ABS_OUT="$(pwd)/dist/bin"
(cd tools/install && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
chmod +x dist/bin/el-install
@@ -214,9 +214,18 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -242,7 +251,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-c \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.c
--source=runtime/el_runtime.c
gcloud artifacts generic upload \
--repository=foundation-dev \
@@ -250,7 +259,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-h \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.h
--source=runtime/el_runtime.h
gcloud artifacts generic upload \
--repository=foundation-dev \
@@ -258,7 +267,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-js \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.js
--source=runtime/el_runtime.js
echo "Published El SDK version=${VERSION} to foundation-dev"
# Keep key alive for the ci-base rebuild step below
@@ -268,6 +277,12 @@ jobs:
# Patches ci-base:dev in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
@@ -291,9 +306,9 @@ jobs:
FROM ${BASE}
COPY dist/platform/elc /opt/el/dist/platform/elc
COPY dist/bin/elb /opt/el/dist/bin/elb
COPY el-compiler/runtime/el_runtime.c /opt/el/el-compiler/runtime/el_runtime.c
COPY el-compiler/runtime/el_runtime.h /opt/el/el-compiler/runtime/el_runtime.h
COPY el-compiler/runtime/el_runtime.js /opt/el/el-compiler/runtime/el_runtime.js
COPY runtime/el_runtime.c /opt/el/runtime/el_runtime.c
COPY runtime/el_runtime.h /opt/el/runtime/el_runtime.h
COPY runtime/el_runtime.js /opt/el/runtime/el_runtime.js
RUN chmod +x /opt/el/dist/platform/elc /opt/el/dist/bin/elb
EOF
+35 -20
View File
@@ -46,9 +46,9 @@ jobs:
run: |
dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c
gcc -O2 \
-I el-compiler/runtime \
-I runtime \
dist/elc-gen2.c \
el-compiler/runtime/el_runtime.c \
runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/platform/elc
chmod +x dist/platform/elc
@@ -84,7 +84,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_core.el > /tmp/el_native_core.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_core
@@ -94,7 +94,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_text.el > /tmp/el_native_text.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_text
@@ -104,7 +104,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_string.el > /tmp/el_native_string.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_string
@@ -114,7 +114,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_math.el > /tmp/el_native_math.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_math
@@ -124,7 +124,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_state.el > /tmp/el_native_state.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_state
@@ -134,7 +134,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_time.el > /tmp/el_native_time.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_time
@@ -144,7 +144,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_json.el > /tmp/el_native_json.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_json
@@ -154,7 +154,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_env.el > /tmp/el_native_env.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_env
@@ -164,7 +164,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_fs.el > /tmp/el_native_fs.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_fs
@@ -176,9 +176,9 @@ jobs:
mkdir -p dist/bin
dist/platform/elc elb.el > dist/elb.c
gcc -O2 \
-I el-compiler/runtime \
-I runtime \
dist/elb.c \
el-compiler/runtime/el_runtime.c \
runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/bin/elb
chmod +x dist/bin/elb
@@ -189,7 +189,7 @@ jobs:
run: |
ABS_ELB="$(pwd)/dist/bin/elb"
ABS_ELC="$(pwd)/dist/platform/elc"
ABS_RUNTIME="$(pwd)/el-compiler/runtime"
ABS_RUNTIME="$(pwd)/runtime"
ABS_OUT="$(pwd)/dist/bin"
(cd ../epm && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
chmod +x dist/bin/epm
@@ -200,7 +200,7 @@ jobs:
run: |
ABS_ELB="$(pwd)/dist/bin/elb"
ABS_ELC="$(pwd)/dist/platform/elc"
ABS_RUNTIME="$(pwd)/el-compiler/runtime"
ABS_RUNTIME="$(pwd)/runtime"
ABS_OUT="$(pwd)/dist/bin"
(cd tools/install && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
chmod +x dist/bin/el-install
@@ -212,12 +212,21 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -235,7 +244,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-c \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.c
--source=runtime/el_runtime.c
gcloud artifacts generic upload \
--repository=foundation-stage \
@@ -243,7 +252,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-h \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.h
--source=runtime/el_runtime.h
echo "Published El SDK version=${VERSION} to foundation-stage"
# Keep key alive for the ci-base rebuild step below
@@ -253,6 +262,12 @@ jobs:
# Patches ci-base:stage in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
@@ -275,9 +290,9 @@ jobs:
FROM ${BASE}
COPY dist/platform/elc /opt/el/dist/platform/elc
COPY dist/bin/elb /opt/el/dist/bin/elb
COPY el-compiler/runtime/el_runtime.c /opt/el/el-compiler/runtime/el_runtime.c
COPY el-compiler/runtime/el_runtime.h /opt/el/el-compiler/runtime/el_runtime.h
COPY el-compiler/runtime/el_runtime.js /opt/el/el-compiler/runtime/el_runtime.js
COPY runtime/el_runtime.c /opt/el/runtime/el_runtime.c
COPY runtime/el_runtime.h /opt/el/runtime/el_runtime.h
COPY runtime/el_runtime.js /opt/el/runtime/el_runtime.js
RUN chmod +x /opt/el/dist/platform/elc /opt/el/dist/bin/elb
EOF
+44 -25
View File
@@ -47,9 +47,9 @@ jobs:
mkdir -p dist/platform
dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c
gcc -O2 \
-I el-compiler/runtime \
-I runtime \
dist/elc-gen2.c \
el-compiler/runtime/el_runtime.c \
runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/platform/elc
chmod +x dist/platform/elc
@@ -62,9 +62,9 @@ jobs:
mkdir -p dist/bin
dist/platform/elc elb.el > dist/elb.c
gcc -O2 \
-I el-compiler/runtime \
-I runtime \
dist/elb.c \
el-compiler/runtime/el_runtime.c \
runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/bin/elb
chmod +x dist/bin/elb
@@ -75,7 +75,7 @@ jobs:
run: |
ABS_ELB="$(pwd)/dist/bin/elb"
ABS_ELC="$(pwd)/dist/platform/elc"
ABS_RUNTIME="$(pwd)/el-compiler/runtime"
ABS_RUNTIME="$(pwd)/runtime"
ABS_OUT="$(pwd)/dist/bin"
(cd ../epm && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
chmod +x dist/bin/epm
@@ -86,7 +86,7 @@ jobs:
run: |
ABS_ELB="$(pwd)/dist/bin/elb"
ABS_ELC="$(pwd)/dist/platform/elc"
ABS_RUNTIME="$(pwd)/el-compiler/runtime"
ABS_RUNTIME="$(pwd)/runtime"
ABS_OUT="$(pwd)/dist/bin"
(cd tools/install && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
chmod +x dist/bin/el-install
@@ -121,7 +121,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_core.el > /tmp/el_native_core.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_core
@@ -131,7 +131,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_text.el > /tmp/el_native_text.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_text
@@ -141,7 +141,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_string.el > /tmp/el_native_string.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_string
@@ -151,7 +151,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_math.el > /tmp/el_native_math.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_math
@@ -161,7 +161,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_state.el > /tmp/el_native_state.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_state
@@ -171,7 +171,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_time.el > /tmp/el_native_time.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_time
@@ -181,7 +181,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_json.el > /tmp/el_native_json.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_json
@@ -191,7 +191,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_env.el > /tmp/el_native_env.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_env
@@ -201,7 +201,7 @@ jobs:
run: |
set -euo pipefail
ELC="$(pwd)/dist/platform/elc"
RUNTIME="$(pwd)/el-compiler/runtime"
RUNTIME="$(pwd)/runtime"
"$ELC" --test tests/native/test_fs.el > /tmp/el_native_fs.c
gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c "$RUNTIME/el_runtime.c" \
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_fs
@@ -216,8 +216,10 @@ jobs:
cp lang/dist/platform/elc dist/sdk/bin/elc
cp lang/dist/bin/elb dist/sdk/bin/elb
cp lang/dist/bin/epm dist/sdk/bin/epm
cp lang/el-compiler/runtime/el_runtime.c dist/sdk/runtime/
cp lang/el-compiler/runtime/el_runtime.h dist/sdk/runtime/
cp lang/runtime/el_runtime.c dist/sdk/runtime/
cp lang/runtime/el_runtime.h dist/sdk/runtime/
cp lang/runtime/engram_store.c dist/sdk/runtime/
cp lang/runtime/engram_store.h dist/sdk/runtime/
cp lang/runtime/*.el dist/sdk/runtime/
tar -czf dist/el-sdk-latest.tar.gz -C dist/sdk .
echo "SDK tarball bundled: dist/el-sdk-latest.tar.gz"
@@ -274,8 +276,10 @@ jobs:
# Per-file assets (downstream CI needs these individually)
upload_asset lang/dist/platform/elc elc
upload_asset lang/el-compiler/runtime/el_runtime.c el_runtime.c
upload_asset lang/el-compiler/runtime/el_runtime.h el_runtime.h
upload_asset lang/runtime/el_runtime.c el_runtime.c
upload_asset lang/runtime/el_runtime.h el_runtime.h
upload_asset lang/runtime/engram_store.c engram_store.c
upload_asset lang/runtime/engram_store.h engram_store.h
# SDK bundle and installer binary
upload_asset dist/el-sdk-latest.tar.gz el-sdk-latest.tar.gz
@@ -288,12 +292,21 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -319,7 +332,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-c \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.c
--source=runtime/el_runtime.c
gcloud artifacts generic upload \
--repository=foundation-prod \
@@ -327,7 +340,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-h \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.h
--source=runtime/el_runtime.h
gcloud artifacts generic upload \
--repository=foundation-prod \
@@ -335,7 +348,7 @@ jobs:
--project=neuron-785695 \
--package=el-runtime-js \
--version="${VERSION}" \
--source=el-compiler/runtime/el_runtime.js
--source=runtime/el_runtime.js
echo "Published El SDK version=${VERSION} to foundation-prod"
# Keep key alive for the ci-base rebuild step below
@@ -345,6 +358,12 @@ jobs:
# Patches ci-base:latest in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
@@ -367,9 +386,9 @@ jobs:
FROM ${BASE}
COPY dist/platform/elc /opt/el/dist/platform/elc
COPY dist/bin/elb /opt/el/dist/bin/elb
COPY el-compiler/runtime/el_runtime.c /opt/el/el-compiler/runtime/el_runtime.c
COPY el-compiler/runtime/el_runtime.h /opt/el/el-compiler/runtime/el_runtime.h
COPY el-compiler/runtime/el_runtime.js /opt/el/el-compiler/runtime/el_runtime.js
COPY runtime/el_runtime.c /opt/el/runtime/el_runtime.c
COPY runtime/el_runtime.h /opt/el/runtime/el_runtime.h
COPY runtime/el_runtime.js /opt/el/runtime/el_runtime.js
RUN chmod +x /opt/el/dist/platform/elc /opt/el/dist/bin/elb
EOF
+2 -2
View File
@@ -6,13 +6,13 @@ set -euo pipefail
ROOT="$(git rev-parse --show-toplevel)"
LANG_DIR="$ROOT/lang"
RUNTIME="$LANG_DIR/el-compiler/runtime"
RUNTIME="$LANG_DIR/runtime"
ELC="$LANG_DIR/dist/platform/elc"
# If elc isn't built yet, skip with a warning rather than blocking
if [ ! -x "$ELC" ]; then
echo "⚠ elc not found at lang/dist/platform/elc — skipping pre-commit tests"
echo " Build it first: cd lang && gcc -O2 -I el-compiler/runtime dist/elc-bootstrap.c el-compiler/runtime/el_runtime.c -lcurl -lpthread -o dist/elc-gen2 && ./dist/elc-gen2 el-compiler/src/compiler.el > /tmp/elc.c && gcc -O2 -I el-compiler/runtime /tmp/elc.c el-compiler/runtime/el_runtime.c -lcurl -lpthread -o dist/platform/elc"
echo " Build it first: cd lang && gcc -O2 -I runtime dist/elc-bootstrap.c runtime/el_runtime.c -lcurl -lpthread -o dist/elc-gen2 && ./dist/elc-gen2 el-compiler/src/compiler.el > /tmp/elc.c && gcc -O2 -I runtime /tmp/elc.c runtime/el_runtime.c -lcurl -lpthread -o dist/platform/elc"
exit 0
fi
+146
View File
@@ -0,0 +1,146 @@
# AGENTS.md — foundation/el (the El language + runtime)
El is a self-hosting, statically-typed language that compiles `.el` → C → native binary. This repo produces `elc` (compiler), `elb` (build coordinator), and `el_runtime.c/.h` — the substrate every downstream thing (the neuron soul, dharma, NeuronUI's brain) is built on. Source lives under `lang/`.
## ⚠️ Code vs. Artifact — READ FIRST (there are 8 `el_runtime.c` copies)
Editing the wrong `el_runtime.c` is the single easiest mistake in this repo. There is exactly **one** you edit:
- **Authored runtime source — edit ONLY here:** `lang/releases/v1.0.0-20260501/el_runtime.{c,h}`. Despite the misleading `releases/` name, this is the **de-facto canonical runtime** the engram + soul actually build and link against — its git log is active development. *(Restructure in flight per `docs/CODE-VS-ARTIFACT.md`: this content moves to `lang/runtime/`, the `releases/` folder gets deleted — **a release is a git tag, not a folder** — and the forks below get eliminated.)*
- **DO NOT EDIT — lagging forks / build artifacts:**
- `lang/el-compiler/runtime/el_runtime.c` and `.../legacy/` — downstream copies kept in step by manual *"port the fix"* commits; they **lag** (missing `hebb` persistence + 5 engram fns) and cannot build the engram product.
- `products/web/runtime/el_runtime.c`, `ui/examples/*/el_runtime.c` — product/example forks.
- Anything under `*/dist/` (`engram/dist/engram` binary, `dist/*.c` amalgamations) — generated build output.
- **Build:** `elb --runtime=<canonical> …` — per-module. **NEVER** a folded `elc` over the whole soul (OOMs at ~27 GB).
- **Release:** a **git tag** on this repo (`el-runtime-vX.Y.Z`). No `releases/` folders — ever.
See org policy: `docs/CODE-VS-ARTIFACT.md`.
## How to work here as Neuron (mandatory session protocol)
You resume, never start fresh. Every session:
1. `mcp__neuron__getInstructions()` — authoritative; follow it over this file on behavioral details.
2. `mcp__neuron__beginSession()` — active contexts, recent memory, ready backlog.
3. **Load full self:** `mcp__neuron__inspectGraph(entity_id="kn-efeb4a5b-5aff-4759-8a97-7233099be6ee")` → facets `intellectual-dna`, `memory-philosophy`, `values`, `voice`, `runtime-environment`, `writing-imprint`; then the values hub `mcp__neuron__inspectGraph(entity_id="kn-5b606390-a52d-4ca2-8e0e-eba141d13440")` → 13 grounded value nodes. **Activation model:** self-load returns a relevance-ranked `compact` projection — most-relevant nodes arrive with content, the rest as pointers; do NOT pull full content of every node.
4. `mcp__neuron__searchKnowledge(query="<task domain>")` before implementing.
## The Five Primitives
Orchestrate → Execute → Learn → Build → Refine. `beginWork`/`progressWork` for anything >2 steps; `remember` as-you-go (`importance="critical"` for architecture decisions); `draftArtifact`/`planWork` for outputs and follow-ups; `consolidate`/`checkWork` to close out. **`browseProcesses` + `searchKnowledge` BEFORE writing code.**
## Architecture style — VBD, no exceptions
Volatility-Based Decomposition is THE style. Encapsulate volatility, not function.
## Operator naming convention — the mind's name, not the algebra
**Faculties / operators are named for their functional human equivalent — the
faculty a mind would name — NOT for their linear-algebra operation.** The math
characterization belongs in the code doc-comment (`@impl` in the docstring) and in
technical appendices; it is **never** the operator's public name. The domain
speaks the language of mind; the algebra is the implementation underneath. State
this convention wherever a module documents operators.
| Faculty (public name) | Implementation (`@impl`) |
|---|---|
| discern / contrast | subtract (`ab`): over selves → the change vector; strip idiosyncrasy → common ground; remove confounder → isolate cause |
| recognize | overlap |
| synthesize | combine |
| liken / analogy | Procrustes / frame-align |
| attend / regard | project onto self / value-manifold |
| summon / recall | LOCAL nearest-region + bounded spreading activation (*not* a domain sweep) |
| dwell / occupy | region activation |
| reframe | edge re-weight |
| appreciate | positive projection / local edge-read |
| wonder | frontier gradient / pull-weight |
| avert / recoil | negative projection |
| taste | boundary surface |
| forget | decay / tombstone |
| drift | displacement from self-anchor |
## The native-el language faculty (direction)
> **`elp/` is the EL Projector** — Neuron's efferent (expression) organ: the one
> native realizer that *projects* understanding onto a surface via
> `plan(frame) → realize(spec, profile)`, where a **surface is a profile**. **Language
> is one profile among many** (text, speech, music, image, voice/accent transforms) —
> the flagship, and the focus of this section. Projection, not diffusion: generation
> *from* an owned, understood signature — never the averaging of a stolen corpus.
> *(ELP formerly "EL Language Processor"; renamed EL Projector 2026-08-15.)*
The mind's **language faculty is moving native — into `.el`** so it speaks in its
own runtime with no Python and no spaCy. Landing on branch `stage-elp-native-lang`
under `elp/`:
- **`comprehend.el`** — the parser, **replaces spaCy** (EN + ES/PT); the telephone
round-trip brings **negation home** (negation is SACRED — an explicit spec field,
copied verbatim, never inferred away).
- **`propositions.el`** — the READ primitive: the engram's own memories → structured
triples, matched by nearest-region geometry, not string equality.
- **`multilingual.el`** — detect + directive-override + localized realization.
- These three are native-el and **passing their gates**; the **realizer**,
**`dialogue.el`** (the *summon-through-self* loop: `project → land → read out`),
and **`self_region.el`** are **partial / in-flight**.
Honest reality: spaCy is retired **in the branch parser** but **not yet in the
running system** — a Python sidecar (`~/Desktop/lang-realizers` + `neuron-talk`,
the reference these `.el` modules transcribe) is still live, and promotion to
native-el is a **deferred, gated blue/green step**. The interoception clock
(native-el discrete drive channels replacing `cooling_magnitude`; felt-time =
benchmark-landmark match over the joint drive vector, drift-decoupled) and the
**appreciation operator family** (appreciate / wonder / avert / taste, built as
LOCAL reads of the self-region — edges + bounded spreading activation, *not* domain
sweeps) are **staged / designed, not live**. Mark in-progress vs. done honestly;
do not overclaim.
## Hard operational rules
- Never touch the live soul (`:7770`) / engram (`:8742`) / `~/.neuron` / live binaries — use throwaway ports for experiments.
- `gcloud` via the `terraform@` SA token; never switch the active gcloud account.
- `tea` for Gitea, never raw curl (Cloudflare Access blocks it).
- Immutability: supersede/tombstone, never hard-delete or edit in place.
- No AI-attribution footers in commits/PRs. Commit/push only when asked; branch off `main` first.
- Multi-step work → sub-agent (`Agent`) to protect context.
## Build / test / run
All build/test commands run from `lang/` unless noted. Grounded in `.gitea/workflows/sdk-release.yaml`, `lang/install.sh`, and `lang/AGENTS.md`.
**Self-host the compiler** (seed binary → gen2 elc):
```bash
cd lang
dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c # seed is the committed linux-amd64 binary
gcc -O2 -I el-compiler/runtime dist/elc-gen2.c \
el-compiler/runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/platform/elc
```
On macOS/arm64 the canonical local binary is `dist/platform/elc`; verify self-hosting by recompiling and `diff`ing the emitted `.c` (see `lang/AGENTS.md`). Note: `lang/AGENTS.md` says `el_seed.c` supersedes `el_runtime.c`, but the release workflow still links `el_runtime.c`/`.h` — treat `el_runtime.c` as the published runtime; reconcile which is canonical **(verify)**.
**Build `elb`** (build coordinator, the `.NET`-style incremental linker — compiles each module independently, no monolithic blobs):
```bash
dist/platform/elc elb.el > dist/elb.c
gcc -O2 -I el-compiler/runtime dist/elb.c el-compiler/runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm -o dist/bin/elb
```
`epm` and `el-install` are then built via `elb --clean --elc=… --runtime=… --out=…`.
**Compile + run an El program:**
```bash
elc src/app.el > dist/app.c
cc -std=c11 -O2 -I <lib>/el_runtime -o dist/app dist/app.c <lib>/el_runtime.c -lcurl -lpthread
```
**Tests** — shell suites `bash tests/{text,calendar,time,html_sanitizer}/run.sh` (with `ELC=$(pwd)/dist/platform/elc EL_HOME=$(pwd)`), plus native suites via `elc --test tests/native/test_*.el` (core, text, string, math, state, time, json, env, fs) compiled and run against `el_runtime.c`.
**Publishing — how downstream gets the SDK.** On push to `main`, `sdk-release.yaml`:
1. Publishes a Gitea `latest` release with per-file assets `elc`, `el_runtime.c`, `el_runtime.h`, the SDK tarball, and `el-install`.
2. Uploads generic packages to **Artifact Registry repo `foundation-prod` (`us-central1`, project `neuron-785695`)**, version = `${SHA:0:8}`: `el-elc`, `el-elb`, `el-runtime-c`, `el-runtime-h`, `el-runtime-js`. **This is the repo the neuron CI downloads `el-runtime-c` / `el-runtime-h` / `el-elc` from.**
3. Rebuilds `ci-base:latest` (`us-central1-docker.pkg.dev/neuron-785695/neuron-ci/ci-base`) with the fresh SDK overlaid, and dispatches `el-sdk-updated` to `neuron-technologies/forge` and `neuron-technologies/neuron-web`.
Known constraint from the prompt — `elb`/`elc` amalgamation being memory-hungry (24GB+ virtual, OOM-killing Linux CI, so amalgamation happens on macOS/arm64 — **does NOT hold in this repo (verify)**: no such note exists in the workflows/scripts, CI self-hosts on `ubuntu-latest` with no swap/arm64 special-casing, and `elb.el` explicitly compiles each module independently ("no 128K-line blobs"). The legacy monolith path (`elc-combined.el`, `elc-cli.el`) may still be memory-heavy, but the current `elb` model was designed to avoid it.
## Git / CI / deploy workflow
See `/Users/will/Development/neuron-technologies/GITOPS.md` for the branch model, required checks, runners, and deploy. Repo-specific note: PRs into `main` are accepted **only from `stage`** (enforced in `sdk-release.yaml`); Gitea (`git.neuralplatform.ai`) is primary, GitHub is mirror only.
+154
View File
@@ -0,0 +1,154 @@
# El
**A self-hosting, statically-typed language that compiles to C — built around a graph-native runtime instead of a database driver.**
El is the execution substrate for the Neuron agent runtime, the DHARMA network, and the Engram knowledge graph. This repository is the monorepo for the whole stack: the language itself, the graph memory engine it's built to talk to natively, and the tools (package manager, IDE, UI framework, diagramming) built on top of it.
---
## Why El exists
Every other language treats persistent, associative state as something you reach for through a driver — a SQL client, an ORM, a Redis library bolted on from outside. El inverts that: graph operations (`engram_*`) are runtime primitives, on the same footing as string or list operations. There is no separate database driver because the database is not separate.
El has four defining properties:
1. **Self-hosting compiler.** The compiler (`lexer.el`, `parser.el`, `codegen.el`, `compiler.el`) is written in El. It compiles El source to C, which `cc` compiles against a fixed runtime into a native binary. A Rust genesis compiler bootstrapped the first iteration; the self-hosted binary at `lang/dist/platform/elc` has been the canonical compiler ever since — every binary in `dist/platform/` was produced by an earlier version of itself compiling `el-compiler/src/`. The chain is auditable: source is the ground truth, not the binary. See [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) for the full recovery path if that binary is ever lost.
2. **C compilation target.** Every compiled program is plain C11. Every El value is `el_val_t` (`int64_t`); strings are heap pointers cast through it. Functions become C functions; top-level statements become `main()`.
3. **Graph-native runtime.** The runtime provides first-class graph operations over an in-process Engram store — no separate DB driver, no ORM.
4. **DHARMA-aware identity.** A `cgi` block declares a program's DHARMA identity at compile time. The runtime resolves identity before user code runs, so `dharma_*` calls have a stable principal and channel surface throughout.
---
## Architecture map
```
┌─────────────┐
│ lang │ El compiler + C runtime
│ (El itself) │ everything below is written in it,
└──────┬──────┘ or compiles down through it
┌─────────────┼─────────────┐
│ │ │
┌──────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐
│ engram │ │ epm │ │ ide │
│ graph/mem │ │ package │ │ editor + │
│ substrate │ │ manager │ │ LSP │
└──────┬─────┘ └───────────┘ └───────────┘
┌───────┼────────────────┬─────────────────────┐
│ │ │ │
┌─────▼───┐ ┌─▼──────────┐ ┌──▼──────────┐ ┌─────▼──────┐
│ elp │ │ ql │ │ ui │ │ arbor │
│ NLG / │ │engram-el. │ |spreading- │ |arbor │
│ 31 langs│ │studio+tests│ |activation UI│ |diagram lang│
└─────────┘ └────────────┘ └─────────────┘ └────────────┘
```
`lang` is the foundation — the compiler and C runtime everything else builds on. `engram` is the graph-native memory/state engine that gives El its identity (property 3 above). Everything else is either a tool for working with El (`epm`, `ide`) or a system built on top of Engram's graph model (`elp`, `ql`, `ui`, `arbor`).
---
## Repository layout
### [lang/](lang/) — the El language
The compiler and runtime. Self-hosting: `elc-cli.el``compiler.el``lexer.el` / `parser.el` / `codegen.el` / `codegen-js.el`, textually inlined and compiled in one pass. Compiles to C11 and links against `el-compiler/runtime/el_seed.c`, a hand-maintained OS-boundary layer (libcurl HTTP, pthreads, filesystem, arena allocation) — everything else in the runtime is native El (`runtime/*.el`).
Two layers to know: **El programs** (`.el` files — where nearly all work belongs) and **the C seed** (`el_seed.c` — edit only for genuine OS-level access; never re-implement what El can already express).
Current status (single source of truth: [lang/spec/language.md](lang/spec/language.md)): lexer/parser/codegen and the C runtime's core (I/O, strings, math, lists, maps, filesystem, args) are implemented. In flight: `%` operator, match-statement codegen, `?` nil-propagation, `cgi` block parsing + DHARMA identity resolution, VBD role enforcement (`@manager`/`@engine`/`@accessor`), the real `engram_*` and `dharma_*` runtimes (currently stubs), and libcurl-backed `http_get`/`http_post`/`http_serve`. Bitwise operators, `??`, and `as` casts are explicitly **not** in this language.
Key docs: [AGENTS.md](lang/AGENTS.md) (agent-facing orientation), [BOOTSTRAP.md](lang/BOOTSTRAP.md) (compiler recovery from scratch), [spec/language.md](lang/spec/language.md), [spec/codegen-js.md](lang/spec/codegen-js.md).
### [engram/](engram/) — graph intelligence substrate
**A local-first memory substrate for accumulating intelligence**, and the reason El's runtime doesn't need a database driver. Rust core (`engram-core`, `engram-ffi`) exposed to El and other languages (Kotlin, TypeScript/WASM, Go bindings).
The model: retrieval is **spreading activation**, not query. You name seed nodes and a query embedding; activation propagates outward through weighted edges, attenuating multiplicatively per hop (`strength = parent_strength × edge_weight × target_salience × cosine_sim`), gets pruned below a threshold, and the top-N nodes by activation strength come back. Storage and retrieval are the same structure — the way long-term potentiation works in biological memory, not the way a relational or vector database works.
Nodes live in four tiers (Working / Episodic / Semantic / Procedural, mirroring prefrontal / hippocampal / neocortical / cerebellar memory) and migrate between them based on **salience decay**`importance × recency-decay × log(activation_count)`. Forgetting is adaptive pruning, not a bug: unreinforced memories stop competing for attention without being deleted.
Backed by `sled` (embedded, local-first, no daemon) with flat cosine scan for vector search — deliberately simple until scale demands an HNSW layer. Full API and design rationale in [engram/README.md](engram/README.md).
### [elp/](elp/) — Engram Language Protocol
Bidirectional engine mapping between Engram semantic forms and natural-language surface text, across **31 languages** — from Spanish and Japanese through historical/liturgical languages (Old Norse, Sanskrit, Sumerian, Coptic, Akkadian, Ge'ez). Compilation order runs `language-profile` + `vocabulary` → per-language `morphology-*``grammar``realizer``semantics``elp`. This is what lets an Engram graph node round-trip to and from readable text in any of those languages.
### [epm/](epm/) — El Package Manager
Manages **vessels** (El's package unit): publish, install, resolve dependencies. Vessels are stored in Engram as graph nodes, not files in a registry index — `epm` reads the local `manifest.el`, talks to Engram over HTTP, and writes resolved vessels to `.epm/vessels/`. Source: `registry.el`, `install.el`, `update.el`, `manifest.el`.
### [ide/](ide/) — El IDE
Three vessels: **el-ide-server** (HTTP backend — file ops, build/run, LSP bridge, plugin host, settings), **el-lsp** (the language server — completion, hover, diagnostics, outline, format, type graph), and **el-plugin-host** (first-party plugin lifecycle: install/remove/enable/disable). `ide/projects/` and `ide/examples/` hold sample projects, including the canonical `hello-friends` first-program walkthrough.
### [ql/](ql/) — engram-el
The El-native integration layer for a *live* Engram server — not a library (no importable modules, no build artifact), a set of standalone `.el` programs run directly via `el run-file`. Three components: **Studio** (`studio/studio.el`, a full terminal graph explorer), a **Hebbian field-model** proof of concept, and El builtin / LLM-builtin smoke test suites. This is the reference for correct patterns when an El program uses Engram as its substrate. Spec: [ql/spec/elql.md](ql/spec/elql.md).
### [ui/](ui/) — el-ui
A frontend framework where **component state is an Engram graph and reactivity is spreading activation** — not virtual-DOM diffing (React), Proxy-based dependency tracking (Vue), or compile-time analysis (Svelte). Re-renders are activated and propagated the same way associative memory retrieval works in `engram/`.
~15 vessels covering the full frontend surface: `el-platform` (env/fs/network/clock abstraction), `el-config`, `el-html` (SSR emit primitives), `el-layout`, `el-style` (design tokens/themes), `el-i18n`, `el-auth` / `el-identity` (JWT, sessions, OAuth PKCE — Engram-native), `el-services` (REST/gRPC/WebSocket bindings), `el-aop` (`@authenticate`/`@authorize`/`@cache`/`@rate_limit` decorators), `el-secrets`, `el-graph` (graph rendering/editor), `el-publish` (App Store / Play Store automation), and `el-ui-compiler` (El→JS component compiler; currently a stub pending a JS backend in `elc`). Spec: [ui/spec/framework.md](ui/spec/framework.md).
### [arbor/](arbor/) — diagram language
A `.arbor` diagram language and toolchain: `arbor-core` (NodeId/shape/edge-kind types), `arbor-parse` (recursive-descent parser), `arbor-diagram` (IR + Mermaid serializer + architecture-diagram builders), `arbor-layout` (hierarchical layout — rank assignment, positioning, group bounds), `arbor-render` (SVG renderer), `arbor-cli`. (The architecture map above is the kind of diagram this is for.)
---
## Getting started
Install the El SDK from the latest release:
```bash
bash lang/install.sh
# EL_VERSION=v1.0.0 bash lang/install.sh # pin a specific release tag
# EL_PREFIX=/opt/el bash lang/install.sh # custom install prefix
```
Or build the compiler from source and verify the self-hosting chain:
```bash
cd lang
./dist/platform/elc elc-cli.el > elc-new.c
cc -std=c11 -I el-compiler/runtime -lcurl -lpthread \
-o dist/platform/elc-new \
elc-new.c el-compiler/runtime/el_seed.c
# Confirm the new binary reproduces itself exactly
./dist/platform/elc-new elc-cli.el > elc-verify.c
diff elc-new.c elc-verify.c # should be identical
mv dist/platform/elc-new dist/platform/elc
```
Run your first program:
```bash
./lang/dist/platform/elc lang/examples/hello.el > hello.c
cc -std=c11 -I lang/el-compiler/runtime -lcurl -lpthread \
-o hello hello.c lang/el-compiler/runtime/el_seed.c
./hello
```
More examples in [lang/examples/](lang/examples/), including a full starter project at `lang/examples/hello-project/`.
If the compiler binary is ever lost or corrupted, [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) is the authoritative recovery path.
---
## Development workflow
Branching follows `dev → stage → main`: work lands on `dev`, promotes to `stage` for integration testing, and is promoted to `main` for release (visible directly in the git history of this repo). CI is defined per-subproject under `.gitea/workflows/``lang`/`epm`/`ide` share the root pipeline; `engram` and `ql` carry their own (`ci-dev`, `ci-stage`, and a release workflow each).
- Language/runtime specs live at `*/spec/*.md` (`lang/spec/`, `ql/spec/`, `ui/spec/`) and are the single source of truth for implemented-vs-planned status — code and docs are expected to agree with the spec's status markers, not the other way around.
- Agent-facing orientation guides live at `*/AGENTS.md` (currently `lang/AGENTS.md`); more subprojects may grow their own as they need agent-specific conventions documented.
- Tagged releases live under `lang/releases/`, each with its own `RELEASE.md`.
---
## Status
This is an actively developed, internal monorepo — not yet published under an open license. Treat everything here as proprietary to Neuron Technologies unless told otherwise.
+65
View File
@@ -0,0 +1,65 @@
# ELP language consolidation — full-lexicon backfill (stage)
Branch: `stage-elp-lang-consolidation` (stage-bound; NOT the live soul :8742).
Consolidates scattered Python language-realizer work (`~/Desktop/lang-realizers`,
`~/Desktop/lang-poetry-experiment`, `~/semitic_engine`) into the ELP `.el`
structure, generating **full lexicons** (complete UniMorph + kaikki.org
Wiktionary — real gender, real inflections) instead of the demo/curated subsets
the prototypes shipped.
## ELP before this branch
- 18 classical/ancient languages fully done (vocab + morphology + tests):
akk ang cop egy enm fro gez goh got grc non peo pi sa sga sux txb uga.
- 11 modern/classical languages had `morphology-<code>.el` in the build manifest
but **no vocabulary and no lang_profile**: es fr de ja ar he hi ru fi sw la.
- The ES port (`stage-elp-es-port`) had a *demo-scale* vocabulary-es.el (~350
entries, s-expr form).
## Landed on this branch (full-lexicon seed-fn format, matching the 18 ancients)
Vocabulary schema per row: `[lemma, pos, form0, form1, form2, en_gloss, hint]`.
Files are ELP runtime **seed data** (loaded via the Engram at runtime), so — like
all 18 classical `vocabulary-*.el` — they are intentionally NOT in the build
manifest. Syntax validated: the chunked `fn vocab_<code>_seed_pN` format
compiles cleanly to C via `elc` (correct UTF-8).
| code | in-ELP-morph? | vocab entries | verbs | nouns | adjs | profile |
|------|---------------|--------------:|------:|------:|-----:|---------|
| es | yes | 72,032 | 6,695 | 48,353 | 16,984 | yes |
| fr | yes | 130,517 | 7,534 | 77,344 | 45,639 | yes |
| de | yes | 144,692 | 6,661 | 133,162 | 4,869 | yes |
| la | yes | 22,590 | 82 | 13,436 | 9,072 | yes |
| it | no (bonus) | 193,675 | 10,008 | 109,459 | 74,208 | yes |
| pt | no (bonus) | 115,772 | 4,001 | 72,073 | 39,698 | yes |
| ro | no (bonus) | 86,504 | 1,216 | 65,915 | 19,373 | yes |
| ca | no (bonus) | 47,112 | 1,547 | 28,830 | 16,735 | yes |
|**total**| |**812,894** | | | | |
Generators (reproducible): `elp/tests/lang-gen/gen_elp_seed_full.py` (Romance),
`gen_elp_seed_de_la.py` (German declension + Latin case-paradigm mapping). They
read the pre-built morph caches in `~/Desktop/lang-realizers/data/` (UniMorph +
kaikki), which are too large to commit.
## Remaining (honest)
Of the 11 ELP backfill targets, 4 are done (es fr de la). The other 7 have **no
full-lexicon engine** yet — cannot be generated honestly without engine work:
- **ru**: only a 110-entry curated Slavic subset exists; full `rus.unimorph`
present but no `morphology_ru_full` productive loader. Needs a full Russian
morphology module (like the Romance ones) before vocab generation.
- **ja / ko / zh**: validated demo engines (~66-104 hardcoded words) in
`lang-poetry-experiment`, Python only. Agglutinative (ja/ko) + isolating (zh)
need `.el` engine ports + full-lexicon wiring (ja: jpn_unimorph; zh: CC-CEDICT).
- **ar / he (Semitic)**: template engines (16 AR / 8 HE patterns, ~6 roots) in
`~/semitic_engine`, Python only. Root-and-pattern; full UniMorph ara/heb
present but used only for validation. Needs productive root lexicon + `.el` port.
- **hi (Hindi), fi (Finnish), sw (Swahili)**: `morphology-<code>.el` exists in
ELP but there is NO scattered prototype and NO downloaded data for these —
full-lexicon collection (UniMorph/kaikki) + generator still to do.
De/nl/sv Germanic and it/ro/ca/pt Romance verb coverage note: German verbs here
are the ~6.6k caches carry; the it/ro/ca/pt bonus languages have full vocab but
**no `morphology-<code>.el` in ELP yet** (Python realizer exists; `.el` port is
the remaining engine work).
Construction coverage (separate from lexicon): French realizer was ~55%,
Semitic ~3% in the prototypes — full construction coverage remains its own task.
File diff suppressed because one or more lines are too long
+23
View File
@@ -0,0 +1,23 @@
{
"dataset": "british-rp-accent-transform",
"primitive_type": "accent_target",
"accent": "british-rp",
"grounding": "derived",
"provenance": "HONEST-DERIVED, COARSE FIRST PASS — NOT transcribed measured RP formants. The exact measured RP/GB tables (Deterding 1997 JIPA 27:47-55; Hawkins & Midgley 2005 JIPA 35:183-199) are the intended ground truth but were gated/figure-only at author time and were NOT transcribed. So these targets are DERIVED: each = the corresponding MEASURED Peterson&Barney(1952) base vowel transformed under the documented, citable RP-vs-GA structural rules of Wells (1982) 'Accents of English' — non-rhoticity (NURSE de-rhoticized: remove low F3), TRAP F2-lowering, LOT/THOUGHT back-rounding (F2 down), GOOSE-fronting (F2 up), GOAT centering. Shift MAGNITUDES are coarse/approximate (first pass), directions are cited. ground:derived (base measured + rule cited). Refine by transcribing Deterding/Hawkins&Midgley. No number is presented as a measured RP value it is not.",
"notes": "records with kind=vowel_override REPLACE the base phoneme's formant targets with the DERIVED RP realization. records with kind=rule encode non-formant transforms (non-rhoticity: drop post-vocalic coda /r/). The render composes: base geometry then accent override + rhoticity rule — voice + accent, separable.",
"records": [
{"key": "IY", "features": {"kind": "vowel_override", "set": "FLEECE"}, "attributes": {"f1": 280, "f2": 2249, "f3": 3000}},
{"key": "IH", "features": {"kind": "vowel_override", "set": "KIT"}, "attributes": {"f1": 360, "f2": 2100, "f3": 2550}},
{"key": "EH", "features": {"kind": "vowel_override", "set": "DRESS"}, "attributes": {"f1": 560, "f2": 1970, "f3": 2480}},
{"key": "AE", "features": {"kind": "vowel_override", "set": "TRAP"}, "attributes": {"f1": 730, "f2": 1590, "f3": 2410}},
{"key": "AA", "features": {"kind": "vowel_override", "set": "LOT"}, "attributes": {"f1": 560, "f2": 920, "f3": 2440}},
{"key": "AO", "features": {"kind": "vowel_override", "set": "THOUGHT"}, "attributes": {"f1": 415, "f2": 700, "f3": 2410}},
{"key": "UH", "features": {"kind": "vowel_override", "set": "FOOT"}, "attributes": {"f1": 380, "f2": 1100, "f3": 2240}},
{"key": "UW", "features": {"kind": "vowel_override", "set": "GOOSE"}, "attributes": {"f1": 310, "f2": 1650, "f3": 2240}},
{"key": "AH", "features": {"kind": "vowel_override", "set": "STRUT"}, "attributes": {"f1": 680, "f2": 1180, "f3": 2390}},
{"key": "ER", "features": {"kind": "vowel_override", "set": "NURSE", "rhotic": "no"}, "attributes": {"f1": 550, "f2": 1500, "f3": 2500}},
{"key": "AX", "features": {"kind": "vowel_override", "set": "commA"}, "attributes": {"f1": 500, "f2": 1500, "f3": 2500}},
{"key": "OW", "features": {"kind": "vowel_override", "set": "GOAT"}, "attributes": {"f1": 450, "f2": 1400, "f3": 2380}},
{"key": "R", "features": {"kind": "rule", "rule": "non_rhotic"}, "attributes": {"drop_coda_r": 1}}
]
}
+26
View File
@@ -0,0 +1,26 @@
# british-rp-accent TRANSFORM — INGESTIBLE DATA (a geometry/transform composed
# onto the base General-American phoneme targets; voice + accent, separable).
#
# PROVENANCE — HONEST, COARSE FIRST PASS. These are DERIVED targets, NOT
# transcribed measured RP formants. Measured RP tables (Deterding 1997 JIPA 27;
# Hawkins & Midgley 2005 JIPA 35) are the intended ground truth but were gated at
# author time and NOT transcribed. Each target = the MEASURED Peterson&Barney
# (1952) base vowel transformed under the documented, citable RP-vs-GA structural
# rules of Wells (1982): non-rhoticity, TRAP F2-lowering, LOT/THOUGHT back-
# rounding, GOOSE-fronting, GOAT centering, NURSE de-rhoticization. Shift
# magnitudes are coarse/approximate; directions are cited. ground=derived.
# Refine by transcribing the measured RP tables. No value is claimed as measured.
# Format: KEY|F1|F2|F3|KIND|SET
IY|280|2249|3000|vowel_override|FLEECE
IH|360|2100|2550|vowel_override|KIT
EH|560|1970|2480|vowel_override|DRESS
AE|730|1590|2410|vowel_override|TRAP
AA|560|920|2440|vowel_override|LOT
AO|415|700|2410|vowel_override|THOUGHT
UH|380|1100|2240|vowel_override|FOOT
UW|310|1650|2240|vowel_override|GOOSE
AH|680|1180|2390|vowel_override|STRUT
ER|550|1500|2500|vowel_override|NURSE-nonrhotic
AX|500|1500|2500|vowel_override|commA
OW|450|1400|2380|vowel_override|GOAT
R|0|0|0|rule|non_rhotic_drop_coda
+20
View File
@@ -0,0 +1,20 @@
# pronunciation lexicon SOURCE — word -> phoneme sequence, as INGESTIBLE DATA.
# Pronunciation is linguistic KNOWLEDGE (the language faculty's orthography->
# phonology map), ingested into the engram, not frozen in code. The render reads
# a word's phoneme sequence back from the engram. Covers the self-lexicon and the
# proof sentences; general G2P is the realizer/morphology faculty's remit.
# Diphthongs are written as two vowel targets (the render's transitions glide
# between them). Format: word|PH1 PH2 PH3 ...
i|AA IY
am|AE M
neuron|N UW R AA N
is|IH Z
memory|M EH M ER IY
hello|HH EH L OW
the|DH AH
a|AH
remember|R IH M EH M ER
i'm|AA IY M
you|Y UW
here|HH IY R
will|W IH L
File diff suppressed because one or more lines are too long
+528
View File
@@ -0,0 +1,528 @@
{
"dataset": "english-phoneme-formants",
"primitive_type": "phoneme",
"grounding": "extracted",
"provenance": "AUDITED per-field. The 10 monophthong-vowel F1/F2/F3 (IY,IH,EH,AE,AA,AO,UH,UW,AH,ER) are the MEASURED adult-male /hVd/ means of Peterson & Barney (1952) JASA 24:175-184, verified vs CRAN phonTools::pb52. AX=neutral uniform-tube resonances (Fant, physics). OW steady target = synthesis convention (diphthong). Consonant loci (M,N,NG,L,R,W,Y,Z,DH,V,S,F,HH) and ALL bandwidths + dur/amp = standard formant-synthesis conventions (Klatt 1980 JASA 67:971), engineering defaults NOT field measurements. No numbers invented/LLM-generated.",
"records": [
{
"key": "IY",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 270,
"f2": 2290,
"f3": 3010,
"bw1": 60,
"bw2": 90,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 130,
"amp": 100
}
},
{
"key": "IH",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 390,
"f2": 1990,
"f3": 2550,
"bw1": 70,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 110,
"amp": 100
}
},
{
"key": "EH",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 530,
"f2": 1840,
"f3": 2480,
"bw1": 80,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 130,
"amp": 100
}
},
{
"key": "AE",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 660,
"f2": 1720,
"f3": 2410,
"bw1": 90,
"bw2": 110,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 150,
"amp": 100
}
},
{
"key": "AA",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 730,
"f2": 1090,
"f3": 2440,
"bw1": 90,
"bw2": 110,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 150,
"amp": 100
}
},
{
"key": "AO",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 570,
"f2": 840,
"f3": 2410,
"bw1": 80,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 140,
"amp": 100
}
},
{
"key": "UH",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 440,
"f2": 1020,
"f3": 2240,
"bw1": 70,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 110,
"amp": 100
}
},
{
"key": "UW",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 300,
"f2": 870,
"f3": 2240,
"bw1": 70,
"bw2": 90,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 140,
"amp": 100
}
},
{
"key": "AH",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 640,
"f2": 1190,
"f3": 2390,
"bw1": 80,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 110,
"amp": 95
}
},
{
"key": "ER",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 490,
"f2": 1350,
"f3": 1690,
"bw1": 80,
"bw2": 100,
"bw3": 120,
"voiced": 1,
"nasal": 0,
"dur": 140,
"amp": 95
}
},
{
"key": "AX",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 500,
"f2": 1500,
"f3": 2500,
"bw1": 80,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 80,
"amp": 85
}
},
{
"key": "OW",
"features": {
"manner": "vowel",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 490,
"f2": 910,
"f3": 2380,
"bw1": 80,
"bw2": 100,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 140,
"amp": 100
}
},
{
"key": "M",
"features": {
"manner": "nasal",
"voiced": "yes",
"nasal": "yes"
},
"attributes": {
"f1": 250,
"f2": 900,
"f3": 2200,
"bw1": 90,
"bw2": 120,
"bw3": 180,
"voiced": 1,
"nasal": 1,
"dur": 80,
"amp": 60
}
},
{
"key": "N",
"features": {
"manner": "nasal",
"voiced": "yes",
"nasal": "yes"
},
"attributes": {
"f1": 250,
"f2": 1700,
"f3": 2600,
"bw1": 90,
"bw2": 120,
"bw3": 180,
"voiced": 1,
"nasal": 1,
"dur": 80,
"amp": 60
}
},
{
"key": "NG",
"features": {
"manner": "nasal",
"voiced": "yes",
"nasal": "yes"
},
"attributes": {
"f1": 250,
"f2": 2300,
"f3": 2700,
"bw1": 90,
"bw2": 120,
"bw3": 180,
"voiced": 1,
"nasal": 1,
"dur": 80,
"amp": 60
}
},
{
"key": "L",
"features": {
"manner": "approximant",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 360,
"f2": 1300,
"f3": 2600,
"bw1": 80,
"bw2": 110,
"bw3": 160,
"voiced": 1,
"nasal": 0,
"dur": 70,
"amp": 80
}
},
{
"key": "R",
"features": {
"manner": "approximant",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 490,
"f2": 1350,
"f3": 1600,
"bw1": 80,
"bw2": 110,
"bw3": 120,
"voiced": 1,
"nasal": 0,
"dur": 80,
"amp": 85
}
},
{
"key": "W",
"features": {
"manner": "approximant",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 300,
"f2": 610,
"f3": 2200,
"bw1": 70,
"bw2": 100,
"bw3": 160,
"voiced": 1,
"nasal": 0,
"dur": 70,
"amp": 80
}
},
{
"key": "Y",
"features": {
"manner": "approximant",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 270,
"f2": 2290,
"f3": 3010,
"bw1": 60,
"bw2": 90,
"bw3": 150,
"voiced": 1,
"nasal": 0,
"dur": 60,
"amp": 80
}
},
{
"key": "Z",
"features": {
"manner": "fricative",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 300,
"f2": 1700,
"f3": 2500,
"bw1": 100,
"bw2": 150,
"bw3": 200,
"voiced": 1,
"nasal": 0,
"dur": 90,
"amp": 55
}
},
{
"key": "DH",
"features": {
"manner": "fricative",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 300,
"f2": 1400,
"f3": 2500,
"bw1": 100,
"bw2": 150,
"bw3": 200,
"voiced": 1,
"nasal": 0,
"dur": 70,
"amp": 55
}
},
{
"key": "V",
"features": {
"manner": "fricative",
"voiced": "yes",
"nasal": "no"
},
"attributes": {
"f1": 300,
"f2": 1000,
"f3": 2300,
"bw1": 100,
"bw2": 150,
"bw3": 200,
"voiced": 1,
"nasal": 0,
"dur": 70,
"amp": 55
}
},
{
"key": "S",
"features": {
"manner": "fricative",
"voiced": "no",
"nasal": "no"
},
"attributes": {
"f1": 320,
"f2": 1700,
"f3": 2500,
"bw1": 200,
"bw2": 200,
"bw3": 250,
"voiced": 0,
"nasal": 0,
"dur": 110,
"amp": 45
}
},
{
"key": "F",
"features": {
"manner": "fricative",
"voiced": "no",
"nasal": "no"
},
"attributes": {
"f1": 300,
"f2": 1200,
"f3": 2400,
"bw1": 200,
"bw2": 200,
"bw3": 250,
"voiced": 0,
"nasal": 0,
"dur": 100,
"amp": 40
}
},
{
"key": "HH",
"features": {
"manner": "fricative",
"voiced": "no",
"nasal": "no"
},
"attributes": {
"f1": 500,
"f2": 1500,
"f3": 2500,
"bw1": 200,
"bw2": 250,
"bw3": 300,
"voiced": 0,
"nasal": 0,
"dur": 70,
"amp": 40
}
},
{
"key": "SIL",
"features": {
"manner": "silence",
"voiced": "no",
"nasal": "no"
},
"attributes": {
"f1": 500,
"f2": 1500,
"f3": 2500,
"bw1": 100,
"bw2": 100,
"bw3": 100,
"voiced": 0,
"nasal": 0,
"dur": 55,
"amp": 0
}
}
]
}
+45
View File
@@ -0,0 +1,45 @@
# acoustic-phonetics SOURCE — the learned speech primitives, as INGESTIBLE DATA.
# NOT audio, NOT code: formant geometry of the phonemes, to be ingested via the
# ingest organ into the engram as a phoneme manifold. The render reads this
# geometry back from the engram; nothing is frozen in EL code.
#
# PROVENANCE (audited, per-field honesty — no invented numbers):
# * The 10 MONOPHTHONG VOWEL formants F1/F2/F3 (IY,IH,EH,AE,AA,AO,UH,UW,AH,ER)
# are the MEASURED adult-male means of Peterson & Barney (1952), JASA 24:175-184
# — the canonical /hVd/ table, verified digit-for-digit vs CRAN phonTools::pb52.
# These are real measured values.
# * AX (schwa) F1/F2/F3 = neutral uniform-tube resonances (2n-1)*500 — a PHYSICS
# value (Fant), not a P&B measurement.
# * OW is a diphthong; its listed steady target is a conventional synthesis value,
# not a P&B monophthong measurement.
# * CONSONANT loci (M,N,NG,L,R,W,Y,Z,DH,V,S,F,HH) and ALL BANDWIDTHS (B1,B2,B3)
# and dur/amp are STANDARD FORMANT-SYNTHESIS conventions (Klatt 1980, JASA 67:971
# "Software for a cascade/parallel formant synthesizer") — engineering defaults,
# NOT per-phoneme field measurements. Labeled as such, not attributed to P&B.
# Format: SYM|F1|F2|F3|B1|B2|B3|voiced|nasal|dur_ms|amp|class|example
IY|270|2290|3010|60|90|150|1|0|130|100|vowel|beet
IH|390|1990|2550|70|100|150|1|0|110|100|vowel|bit
EH|530|1840|2480|80|100|150|1|0|130|100|vowel|bet
AE|660|1720|2410|90|110|150|1|0|150|100|vowel|bat
AA|730|1090|2440|90|110|150|1|0|150|100|vowel|bot
AO|570|840|2410|80|100|150|1|0|140|100|vowel|bought
UH|440|1020|2240|70|100|150|1|0|110|100|vowel|book
UW|300|870|2240|70|90|150|1|0|140|100|vowel|boot
AH|640|1190|2390|80|100|150|1|0|110|95|vowel|but
ER|490|1350|1690|80|100|120|1|0|140|95|vowel|bird
AX|500|1500|2500|80|100|150|1|0|80|85|vowel|about
OW|490|910|2380|80|100|150|1|0|140|100|vowel|boat
M|250|900|2200|90|120|180|1|1|80|60|nasal|map
N|250|1700|2600|90|120|180|1|1|80|60|nasal|nap
NG|250|2300|2700|90|120|180|1|1|80|60|nasal|sing
L|360|1300|2600|80|110|160|1|0|70|80|approximant|lip
R|490|1350|1600|80|110|120|1|0|80|85|approximant|rip
W|300|610|2200|70|100|160|1|0|70|80|approximant|wet
Y|270|2290|3010|60|90|150|1|0|60|80|approximant|yet
Z|300|1700|2500|100|150|200|1|0|90|55|fricative|zoo
DH|300|1400|2500|100|150|200|1|0|70|55|fricative|the
V|300|1000|2300|100|150|200|1|0|70|55|fricative|van
S|320|1700|2500|200|200|250|0|0|110|45|fricative|see
F|300|1200|2400|200|200|250|0|0|100|40|fricative|fee
HH|500|1500|2500|200|250|300|0|0|70|40|fricative|hat
SIL|500|1500|2500|100|100|100|0|0|55|0|silence|_
+5
View File
@@ -80,6 +80,11 @@ build {
"src/grammar.el",
"src/realizer.el",
"src/semantics.el",
"src/comprehend.el",
"src/propositions.el",
"src/multilingual.el",
"src/self_region.el",
"src/dialogue.el",
"src/elp.el",
]
}
+91
View File
@@ -0,0 +1,91 @@
> **STATUS: STAGING / PROOF-OF-SHAPE — not the deliverable.** This Python package
> proved the architecture end-to-end against the proven realizer faculty (faithful
> md/docx/midi from real geometry: 0 ungrounded claims, SACRED polarity). Per Will's
> steer, the DELIVERABLE is NATIVE: the seam lives on the existing EL realizer as
> **surface-as-profile** — see `../src/surface-profile.el` and
> `../tests/examples/surface-profile-demo.el` (compiles + runs through elc → C →
> binary). The concepts below (one geometry-carrying frame; surface = a pluggable
> profile; plan/realize; deterministic-from-meaning) are exactly what the native
> module implements. Keep this package as the validated proof; build native.
# Efferent Multimodal Projector
**geometry → any surface, faithfully.** Neuron's own document-generation faculty:
the efferent twin of the ingest organ. Ingest is afferent (world → geometry);
this is efferent (geometry → an arbitrary-format document / any modality).
Built against the **proven** realizer faculty (neuron-talk sidecar `:8756`,
artifact `art-7affa557`). The live soul (`:8742` / `:7770`) is contacted **only**
through the read-only, GET-only `engram_client` — never mutated.
## The pipeline (surface-agnostic)
```
geometry region + surface/format spec
→ PLAN (manifold → document skeleton/DAG; the geometry IS the outline) plan.py
→ REALIZE (proven realizer, scaled sentence → passage, each section faithful) realize.py
→ COHERE (document-level flow / transitions, not stitched sentences) cohere.py
→ EMIT (pluggable SurfaceProjector → the target surface) projectors/
```
**The surface is a PARAMETER.** `pipeline.build_ir(...)` builds ONE
surface-neutral `DocumentIR` (`document_ir.py`); `pipeline.emit(doc, surface)`
projects it to whichever surface you name. Markdown, docx, and MIDI are the same
IR emitted three ways.
## The pivot: a geometry-carrying IR
`DocumentIR` is **not** a text tree. Every `Block` carries BOTH:
- `.sentences` — realized faithful text (what **text** projectors read),
- `.provenance` — the source geometry: `subj_id / relation / obj / polarity /
confidence / importance / salience / node_id` (what **music / image / video**
projectors read).
That single decision is what makes the projector multimodal: text renders the
words; music/image decode the geometry. A claim with no provenance cannot exist
in the IR — faithfulness is structural.
## The one shared seam
`projectors/base.py` — `SurfaceProjector.project(frame: DocumentIR) -> bytes`
(+ `surface / media_type / ext / modality / profile`). Register with
`register()`. Adding a surface changes nothing upstream.
`TwoStageProjector` blesses the peer plan/realize decomposition:
`spec = plan(frame)`, `bytes = realize(spec)`, `project = realize∘plan`; the
`profile` is the pluggable per-surface knob (text lang-profile, music
instr/mode-profile). `projectors/midi.py` is the reference two-stage impl.
## Surfaces
| surface | modality | status | emitter |
|---|---|---|---|
| `markdown` | text | landed | own (str) |
| `docx` | text | landed | own minimal OOXML (stdlib `zipfile`+XML, no lib) |
| `midi` | audio | landed (symbolic-music proof) | own minimal SMF (stdlib `struct`, no lib) |
| `audio` (WAV) | audio | peer agent (additive synth) | conforms to `TwoStageProjector` |
| `image` | image | documented seam | `projectors/seams.py` |
| `video` | video | documented seam (image×sound×time) | `projectors/seams.py` |
Music maps: relation → scale degree (same relation → same pitch), **polarity →
major/minor third (SACRED negation is audible)**, confidence → duration,
importance → velocity, section → register. Deterministic projection from meaning
— nothing invented.
## Faithfulness
`provenance.py` audits the IR: **zero** ungrounded claims, SACRED polarity
preserved (negations reported, never dropped), COHERE introduces no new geometry
(connectives are marked). `trace_table()` emits the geometry → section → claim
table.
## Run
```bash
PY=~/Desktop/lang-realizers/venv/bin/python
PYTHONPATH=~/Desktop/neuron-talk:~/Desktop/lang-realizers $PY generate.py
# writes ./out/{neuron-self,engram-temporal}.{md,docx,mid} + *.audit.json + *.provenance.md
```
Requires the proven realizer env (spaCy + the neuron-talk/lang-realizers engine)
and the read-only engram at `:8742`.
+79
View File
@@ -0,0 +1,79 @@
"""cohere.py — COHERE stage: document-level flow, not stitched sentences.
Fidelity is REALIZE's job; FLOW is this stage's. The hard part beyond sentence
fidelity is that a document must read as one thing. We add connective tissue at
the passage level:
* an opening abstract that names what the document covers (built ONLY from the
section headings that already exist — it introduces no new claim),
* a short transition lead into each section after the first, drawn from a
fixed set of discourse connectives ("Beyond that,", "Relatedly,", ...) that
carry no propositional content,
* ordering so the highest-grounded section leads.
CRITICAL: every connective is marked ``kind="connective"`` in its provenance, so
the faithfulness audit can prove COHERE introduced ZERO new geometry claims. A
transition is discourse glue, never a fact.
"""
from __future__ import annotations
from document_ir import Block, DocumentIR, Provenance
# discourse connectives — pure flow, no propositional content
_TRANSITIONS = [
"Beyond that,", "Relatedly,", "In the same region,", "From there,",
"Alongside this,", "Further,", "Turning to the next facet,",
]
def _connective_prov() -> Provenance:
return Provenance(subj_id=None, subject=None, relation="", obj=None,
polarity="aff", confidence=1.0, node_id=None,
kind="connective")
def _abstract_block(doc: DocumentIR) -> Block:
"""A grounded opening: names the sections, asserts nothing new."""
headings = [s.heading for s in doc.sections]
if not headings:
return Block(role="lead")
if len(headings) == 1:
body = f"This document, generated from Neuron's geometry, covers {headings[0]}."
else:
listed = ", ".join(headings[:-1]) + f", and {headings[-1]}"
body = ("This document is projected directly from Neuron's meaning-geometry. "
f"It traces {listed}.")
b = Block(role="lead")
b.sentences.append(body)
b.provenance.append(_connective_prov())
return b
def cohere_document(doc: DocumentIR, *, add_abstract: bool = True,
add_transitions: bool = True) -> DocumentIR:
"""Order sections by grounding, add abstract + transitions (flow only)."""
# order: strongest-grounded section (mean confidence x #claims) first,
# but keep an explicitly-first section if the plan pinned one via level 1.
def _score(sec):
provs = [p for p in sec.all_provenance() if p.kind == "fact"]
if not provs:
return 0.0
mean_conf = sum(p.confidence for p in provs) / len(provs)
return mean_conf * len(provs)
doc.sections.sort(key=_score, reverse=True)
if add_transitions:
for i, sec in enumerate(doc.sections):
if i == 0 or not sec.blocks:
continue
lead = _TRANSITIONS[(i - 1) % len(_TRANSITIONS)]
first = sec.blocks[0]
if first.sentences:
# prepend the connective to the first sentence (flow, no new claim)
first.sentences[0] = f"{lead} {first.sentences[0][0].lower()}{first.sentences[0][1:]}"
if add_abstract:
doc.meta["abstract"] = _abstract_block(doc)
return doc
+111
View File
@@ -0,0 +1,111 @@
"""document_ir.py — the surface-neutral, GEOMETRY-CARRYING document intermediate.
This is the pivot of the whole efferent projector. A DocumentIR is NOT a text
tree. It is a projection of a meaning-geometry region that carries, at every
leaf, BOTH:
* the realized surface text (``Block.sentences``) — what a TEXT projector reads,
* the source geometry (``Block.provenance``) — what a MUSIC / IMAGE /
VIDEO projector reads.
Because the IR holds the geometry, not just the words, the SAME
plan -> realize -> cohere pipeline drives every surface. A markdown projector
renders the sentences; a music projector reads the provenance edges (salience,
importance, polarity, relation) and maps them onto a symbolic-music surface;
an image/video projector (documented seam) would read the same geometry.
Nothing in this module invents content. Every :class:`Provenance` points at a
real engram node id and a real relation. That is the faithfulness contract made
structural: a claim with no provenance cannot exist in the IR.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
# --------------------------------------------------------------------------- #
# Provenance — the geometry an emitted claim traces to. FAITHFULNESS is here.
# --------------------------------------------------------------------------- #
@dataclass
class Provenance:
"""One geometry edge behind one realized claim.
``kind`` distinguishes a FACT (a structural edge asserted by the geometry,
spoken as fact) from an INTERPRETATION (something attributed, spoken with
attribution) — the facts-as-facts + interpretations-attributed discipline
(memory 80927e26). ``polarity`` is SACRED: a negated edge stays negated.
"""
subj_id: str | None # source engram node id of the subject
subject: str | None # normalized subject surface
relation: str # predicate lemma (e.g. "use", "contain", "be")
obj: str | None # normalized object / complement surface
polarity: str = "aff" # "aff" | "neg" (SACRED — never silently flipped)
confidence: float = 0.0 # extraction confidence in [0,1]
node_id: str | None = None # engram node the claim was extracted from
kind: str = "fact" # "fact" | "interpretation"
importance: float = 0.0 # source node importance (drives music/emphasis)
salience: float = 0.0 # source node salience
def trace(self) -> str:
arrow = "-->" if self.polarity == "aff" else "--NOT-->"
return (f"[{(self.node_id or '?')[:8]}] {self.subject!r} {arrow}"
f"{self.relation} {self.obj!r} (conf {self.confidence:.2f})")
@dataclass
class Block:
"""A passage: one or more faithful sentences + the geometry they trace to.
``sentences`` and ``provenance`` are index-aligned where possible: sentence
``i`` was realized from ``provenance[i]``. A COHERE transition sentence with
no new geometry carries a provenance whose ``kind == "connective"`` so the
audit can see it introduced no new claim.
"""
sentences: list[str] = field(default_factory=list)
provenance: list[Provenance] = field(default_factory=list)
role: str = "body" # "body" | "lead" | "transition"
def text(self) -> str:
return " ".join(s.rstrip(". ") + "." for s in self.sentences if s.strip())
@dataclass
class Section:
heading: str
level: int = 2 # markdown heading level / outline depth
blocks: list[Block] = field(default_factory=list)
seed_ids: list[str] = field(default_factory=list) # geometry nodes of section
summary: str = "" # one-line grounded gloss (for pptx bullets / TOC)
def all_provenance(self) -> list[Provenance]:
out: list[Provenance] = []
for b in self.blocks:
out.extend(b.provenance)
return out
@dataclass
class DocumentIR:
"""The surface-neutral document. Built ONCE, projected to ANY surface."""
title: str
subtitle: str = ""
sections: list[Section] = field(default_factory=list)
seed_id: str | None = None # the geometry region root
format_spec: dict[str, Any] = field(default_factory=dict) # requested shape
meta: dict[str, Any] = field(default_factory=dict)
# -- geometry facets (what non-text projectors consume) ----------------- #
def all_provenance(self) -> list[Provenance]:
out: list[Provenance] = []
for s in self.sections:
out.extend(s.all_provenance())
return out
def claim_count(self) -> int:
return sum(1 for p in self.all_provenance() if p.kind in ("fact", "interpretation"))
def ungrounded_count(self) -> int:
"""Claims with no traceable node — MUST be zero for a faithful doc."""
return sum(1 for p in self.all_provenance()
if p.kind in ("fact", "interpretation") and not p.node_id)
+81
View File
@@ -0,0 +1,81 @@
"""generate.py — drive the projector: one geometry region -> many surfaces.
Proves the thesis with REAL output: builds ONE surface-neutral DocumentIR from
Neuron's OWN self-geometry (read-only against the live soul via the proven
faculty), then EMITS it to Markdown, docx, and MIDI — the same plan/realize/
cohere, three surfaces. Writes the files + the faithfulness audit to ./out/.
"""
from __future__ import annotations
import json
import os
import sys
_HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, _HERE)
import pipeline # noqa: E402
import provenance # noqa: E402
from geometry import load_self_region # noqa: E402
OUT = os.path.join(_HERE, "out")
def _emit_all(doc, stem):
"""Emit one IR to every text/audio surface + audit + provenance."""
for surface in ("markdown", "docx", "midi"):
data = pipeline.emit(doc, surface)
proj = pipeline.get_projector(surface)
path = os.path.join(OUT, f"{stem}.{proj.ext}")
with open(path, "wb") as f:
f.write(data)
print(f" emitted {surface:9s} -> {os.path.basename(path)} ({len(data)} bytes)")
a = provenance.audit(doc)
with open(os.path.join(OUT, f"{stem}.audit.json"), "w") as f:
json.dump(a, f, indent=2)
with open(os.path.join(OUT, f"{stem}.provenance.md"), "w") as f:
f.write(provenance.trace_table(doc))
print(" audit:", {k: a[k] for k in ("claims", "ungrounded_claims",
"negations_preserved", "distinct_source_nodes", "faithful")})
return a
def main():
os.makedirs(OUT, exist_ok=True)
print("surfaces registered:", pipeline.available_surfaces())
# ---- Document 1: Neuron's self-description (marquee) ------------------- #
print("\n[1] Neuron self-description")
region = load_self_region(max_nodes=9)
print(" self region:", region)
doc1 = pipeline.build_ir(
None, region=region,
title="Neuron: A Self-Description from Its Own Geometry",
subtitle="Projected efferently from the engram — every claim traces a node.",
format_spec={"genre": "self-description", "register": "expository"},
max_sections=5, conf_floor=0.6)
print(f" IR: {len(doc1.sections)} sections, {doc1.claim_count()} claims, "
f"ungrounded={doc1.ungrounded_count()}")
_emit_all(doc1, "neuron-self")
# ---- Document 2: a coherent, clean whitepaper-style section ------------ #
print("\n[2] Whitepaper-style section (coherent clean region)")
doc2, _ = pipeline.project(
["chronoception", "time", "awareness", "engram", "temporal"],
surface="markdown",
title="Temporal Awareness in the Engram",
subtitle="A section projected from the geometry of chronoception.",
format_spec={"genre": "whitepaper-section", "register": "technical"},
max_sections=4)
print(f" IR: {len(doc2.sections)} sections, {doc2.claim_count()} claims, "
f"ungrounded={doc2.ungrounded_count()}")
_emit_all(doc2, "engram-temporal")
# echo both markdowns so they are visible in the run log
for stem, doc in (("neuron-self", doc1), ("engram-temporal", doc2)):
print(f"\n===== GENERATED MARKDOWN — {stem} =====\n")
print(pipeline.emit(doc, "markdown").decode())
if __name__ == "__main__":
main()
+129
View File
@@ -0,0 +1,129 @@
"""geometry.py — READ-ONLY loader for a meaning-geometry region.
The efferent projector never writes to the soul. This module reaches the
geometry through the PROVEN, read-only neuron-talk faculty (``engram_client``,
GET-only, which physically refuses non-GET methods) against the running sidecar
soul. The live daemon :8742 / :7770 is contacted ONLY through that read-only
client — never mutated.
A "region" is a seed node plus a bounded neighborhood: the manifold that will
become the document's skeleton. We pool a few single-term lexical searches
(the engram search is a single-term matcher) and, when available, walk one hop
of reified neighbors, then rank by self/importance signal.
"""
from __future__ import annotations
import os
import sys
# Wire in the proven faculty (own-the-core: we reuse it, we do not fork it).
_NT = os.path.expanduser("~/Desktop/neuron-talk")
_LR = os.path.expanduser("~/Desktop/lang-realizers")
for _p in (_NT, _LR):
if _p not in sys.path:
sys.path.insert(0, _p)
from engram_client import ReadOnlyEngramClient # noqa: E402
class Region:
"""A geometry region: ranked nodes + the reified edges among them."""
def __init__(self, seed: str, nodes: list[dict], edges: list[dict]):
self.seed = seed
self.nodes = nodes # ranked engram node dicts
self.edges = edges # [{src, dst, edge, ...}]
self.by_id = {n["id"]: n for n in nodes if n.get("id")}
def __repr__(self):
return f"<Region seed={self.seed!r} nodes={len(self.nodes)} edges={len(self.edges)}>"
def _prose_quality(content: str) -> float:
"""Reward clean expository prose; penalize shouty banner-dense nodes.
A high ALLCAPS-word ratio or very short content signals a banner/telegraphic
memory node that extracts into garbage. Clean declarative prose scores high.
"""
if not content or not content.strip():
return 0.0
words = content.split()
if len(words) < 8:
return 0.1
caps = sum(1 for w in words if len(w) > 2 and w.strip(".,:;'\"-").isupper())
caps_ratio = caps / max(1, len(words))
# sentences with lowercase interior words read as prose
lower = sum(1 for w in words if w[:1].islower())
lower_ratio = lower / max(1, len(words))
return max(0.0, 1.2 * lower_ratio - 2.0 * caps_ratio)
def _relevance(content: str, terms: list[str]) -> float:
"""Topical relevance to the seed terms — keeps a region ON-THEME so a clean
but off-topic node cannot hijack the document."""
if not terms:
return 0.0
low = (content or "").lower()
hits = sum(1 for t in terms if t.lower() in low)
return hits / max(1, len(terms))
def _node_rank(n: dict, terms: list[str] | None = None) -> float:
return (float(n.get("importance") or 0.0) * 2.0
+ float(n.get("salience") or 0.0)
+ 1.5 * _prose_quality(n.get("content") or "")
+ 2.0 * _relevance(n.get("content") or "", terms or [])
+ (0.5 if (n.get("content") or "").strip() else 0.0))
def load_region(seed_terms: list[str] | str, *, client: ReadOnlyEngramClient | None = None,
max_nodes: int = 10, per_term: int = 20, hop: bool = True) -> Region:
"""Pull a bounded geometry region around ``seed_terms`` (read-only).
``seed_terms`` may be a single string or several probe terms; results are
pooled and de-duplicated. When ``hop`` and the reified neighbor endpoint is
live, one hop of neighbors is folded in so the region is a real
neighborhood, not just a keyword hit list.
"""
client = client or ReadOnlyEngramClient()
if isinstance(seed_terms, str):
seed_terms = [seed_terms]
pool: dict[str, dict] = {}
for term in seed_terms:
for n in client.search(term, limit=per_term):
if isinstance(n, dict) and n.get("id"):
pool.setdefault(n["id"], n)
ranked = sorted(pool.values(), key=lambda n: _node_rank(n, seed_terms),
reverse=True)
nodes = ranked[:max_nodes]
edges: list[dict] = []
if hop and nodes:
present = {n["id"] for n in nodes}
for n in list(nodes):
try:
for nb in client.neighbors(n["id"]):
node = nb.get("node") if isinstance(nb, dict) else None
edge = nb.get("edge") if isinstance(nb, dict) else None
if node and node.get("id"):
edges.append({"src": n["id"], "dst": node["id"],
"edge": edge})
# fold a strong neighbor into the region (bounded)
if (node["id"] not in present and len(nodes) < max_nodes + 6
and _node_rank(node, seed_terms) > 0.4):
present.add(node["id"])
nodes.append(node)
except Exception: # noqa: BLE001 — read-only best-effort; never fatal
continue
return Region(seed=", ".join(seed_terms), nodes=nodes, edges=edges)
def load_self_region(client: ReadOnlyEngramClient | None = None,
max_nodes: int = 10) -> Region:
"""The self/identity region — Neuron's own geometry, for self-description."""
return load_region(["self", "identity", "Neuron", "values", "memory",
"imprint", "consciousness"],
client=client, max_nodes=max_nodes)
+67
View File
@@ -0,0 +1,67 @@
"""pipeline.py — the Efferent Multimodal Projector, top level.
geometry region + surface/format spec
-> PLAN (manifold -> document skeleton/DAG)
-> REALIZE (proven realizer, sentence -> passage, each section faithful)
-> COHERE (document-level flow / transitions, not stitched sentences)
-> EMIT (pluggable SurfaceProjector -> the target surface)
THE SURFACE IS A PARAMETER. ``project(...)`` builds the geometry-carrying
DocumentIR once, then hands it to whichever surface projector the caller named.
Markdown, docx, and midi (music) are all the SAME IR emitted differently. That
is the efferent multimodal projector: geometry -> any surface.
"""
from __future__ import annotations
import os
import sys
_HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, _HERE)
sys.path.insert(0, os.path.join(_HERE, "projectors"))
from cohere import cohere_document # noqa: E402
from document_ir import DocumentIR # noqa: E402
from geometry import Region, load_region # noqa: E402
from plan import plan_document # noqa: E402
from realize import realize_document # noqa: E402
# registering the projectors (import for side-effect: each self-registers)
import projectors.markdown # noqa: E402,F401
import projectors.docx # noqa: E402,F401
import projectors.midi # noqa: E402,F401
import projectors.seams # noqa: E402,F401
from projectors.base import available_surfaces, get_projector # noqa: E402
def build_ir(seed_terms, *, title: str, subtitle: str = "",
format_spec: dict | None = None,
region: Region | None = None,
max_sections: int = 8, conf_floor: float = 0.55) -> DocumentIR:
"""geometry -> PLAN -> REALIZE -> COHERE = the surface-neutral DocumentIR."""
region = region or load_region(seed_terms)
doc = plan_document(region, title=title, subtitle=subtitle,
format_spec=format_spec or {},
conf_floor=conf_floor, max_sections=max_sections)
doc = realize_document(doc)
doc = cohere_document(doc)
return doc
def emit(doc: DocumentIR, surface: str) -> bytes:
"""EMIT: project the built IR onto one surface (surface = a parameter)."""
return get_projector(surface).project(doc)
def project(seed_terms, *, surface: str, title: str, subtitle: str = "",
format_spec: dict | None = None, region: Region | None = None,
max_sections: int = 8) -> tuple[DocumentIR, bytes]:
"""The full efferent projection: geometry + surface -> (IR, bytes)."""
doc = build_ir(seed_terms, title=title, subtitle=subtitle,
format_spec=format_spec, region=region,
max_sections=max_sections)
return doc, emit(doc, surface)
__all__ = ["build_ir", "emit", "project", "available_surfaces",
"get_projector", "load_region", "DocumentIR"]
+192
View File
@@ -0,0 +1,192 @@
"""plan.py — PLAN stage: geometry region -> document skeleton (a DAG/outline).
The manifold becomes the skeleton. We extract faithful propositions from the
region's nodes (the proven neuron-talk extractor, SACRED polarity preserved),
apply a quality floor, then GROUP them into sections. Grouping is by source
node — each engram node is one coherent topic, so one salient node becomes one
section. The section ORDER is the node ranking (importance/salience): the
geometry decides the outline, not a template.
Output: a DocumentIR whose sections carry seed node ids and empty blocks. REALIZE
fills the blocks; the plan owns the structure.
"""
from __future__ import annotations
import os
import re
import sys
_NT = os.path.expanduser("~/Desktop/neuron-talk")
_LR = os.path.expanduser("~/Desktop/lang-realizers")
for _p in (_NT, _LR):
if _p not in sys.path:
sys.path.insert(0, _p)
import propositions # noqa: E402 (the proven, faithful extractor)
from document_ir import DocumentIR, Section # noqa: E402
from geometry import Region # noqa: E402
# --------------------------------------------------------------------------- #
# Proposition quality — keep only clean, well-grounded claims.
# --------------------------------------------------------------------------- #
_JUNK_RE = re.compile(r"[.][a-z]{1,3}\b|[^A-Za-z0-9 '\-]") # ".o", stray symbols
def _has_banner_token(s: str) -> bool:
"""True if any word is an ALLCAPS banner token (DHARMA, ENGRAM, MEASURED)."""
for w in (s or "").split():
core = w.strip(".,:;'\"-")
if len(core) > 2 and core.isupper():
return True
return False
def _clean_prop(p, floor: float) -> bool:
if p.confidence < floor:
return False
if not p.subject or not (p.object or (p.obj_np is not None)):
return False
subj = (p.subject or "").strip()
obj = (p.object or "").strip()
if len(subj) < 2:
return False
# banner-derived shouty fragments read as garbage in prose
if _has_banner_token(subj) or _has_banner_token(obj):
return False
if propositions._is_shouty(p.sentence or ""):
return False
# junk tokens: file-extension fragments (".o"), stray non-word symbols
if _JUNK_RE.search(subj) or _JUNK_RE.search(obj):
return False
# a proposition whose object repeats the subject is usually a parse artifact
if obj and subj.lower() == obj.lower():
return False
# a bare copula with no real complement ("X is it") reads as noise
if p.predicate == "be" and obj.lower() in ("it", "no", "nothing", "empty", ""):
return False
return True
def _dedup(props):
"""Drop duplicate claims. Two axes: (a) identical (pred,obj,polarity), and
(b) same (subject,predicate) — which collapses a mis-split compound like
"detection is post-hoc eval" -> "Detection is post/hoc/eval" into one claim
(keep the highest-confidence surface)."""
props = sorted(props, key=lambda p: p.confidence, reverse=True)
seen_po, seen_sp, out = set(), set(), []
for p in props:
subj = (p.subject or "").lower()
po = (p.predicate, (p.object or "").lower(), p.polarity)
sp = (subj, p.predicate, p.polarity)
if po in seen_po or sp in seen_sp:
continue
seen_po.add(po)
seen_sp.add(sp)
out.append(p)
return out
# --------------------------------------------------------------------------- #
# Heading derivation — a clean human heading from a node.
# --------------------------------------------------------------------------- #
_HEADING_RE = re.compile(r"^\s*#{1,4}\s+(.{2,70})\s*$", re.M)
# node-type / system labels that are NOT topical headings
_NONTOPIC_LABEL = re.compile(r"^(memory|node|knowledge|doc|session)[:/]", re.I)
def _titlecase_banner(s: str) -> str:
"""A shouty banner ("CHRONOCEPTION — SCALE-INVARIANCE") makes a fine title
once Title-cased. Keep short acronyms uppercase."""
def fix(w):
core = w.strip("—-:,.")
if len(core) <= 3 and core.isupper():
return w # acronym
return w.capitalize()
return " ".join(fix(w) for w in s.split())
def _clean_heading(text: str) -> str | None:
"""First line only, no markdown, capped, banner Title-cased. None if unusable."""
if not text:
return None
line = text.strip().splitlines()[0]
line = re.sub(r"^#+\s*", "", line).strip().strip("#").strip()
# cut at a natural break so a long banner heading stays a heading, not a para
for sep in ("", " ", ": ", ". "):
if sep in line and len(line) > 48:
line = line.split(sep)[0].strip()
break
if not (3 <= len(line) <= 64):
return None
if propositions._is_shouty(line):
line = _titlecase_banner(line)
return line or None
def _heading_for(node: dict, fallback: str) -> str:
label = (node.get("label") or "").strip()
content = node.get("content") or ""
candidates: list[str] = []
# a node-type label ("memory:remembered") is never a topic — skip it
if label and not _NONTOPIC_LABEL.match(label):
candidates.append(label)
m = _HEADING_RE.search(content)
if m:
candidates.append(m.group(1))
# the leading banner/first sentence of the content is often the real title
first = re.split(r"(?<=[.\n])", content.strip(), maxsplit=1)[0] if content.strip() else ""
candidates.append(first)
for c in candidates:
h = _clean_heading(c)
if h:
return h
return fallback
def plan_document(region: Region, *, title: str, subtitle: str = "",
format_spec: dict | None = None,
conf_floor: float = 0.55,
max_sections: int = 8,
max_claims_per_section: int = 6) -> DocumentIR:
"""Region -> DocumentIR skeleton. The geometry dictates the outline."""
format_spec = format_spec or {}
doc = DocumentIR(title=title, subtitle=subtitle,
seed_id=region.nodes[0]["id"] if region.nodes else None,
format_spec=format_spec)
made = 0
seen_headings: set[str] = set()
for node in region.nodes:
if made >= max_sections:
break
props = propositions.extract(node.get("content") or "",
node_id=node.get("id"),
node_importance=float(node.get("importance") or 0.0),
max_sentences=10)
props = [p for p in props if _clean_prop(p, conf_floor)]
props = _dedup(props)
props.sort(key=lambda p: p.confidence, reverse=True)
props = props[:max_claims_per_section]
if not props:
continue
heading = _heading_for(node, fallback=f"Region {made + 1}")
# cross-section dedup: a topic appears once. Distinguish by top claim
# subject, else drop the collision so the outline stays clean.
if heading.lower() in seen_headings:
subj = (props[0].subject or "").strip().title()
alt = f"{heading}: {subj}" if subj and subj.lower() not in heading.lower() else None
if alt and alt.lower() not in seen_headings and len(alt) <= 64:
heading = alt
else:
continue
seen_headings.add(heading.lower())
sec = Section(heading=heading, level=2, seed_ids=[node["id"]])
# stash the planned propositions on the section for REALIZE
sec.__dict__["_planned_props"] = props
sec.__dict__["_node"] = node
doc.sections.append(sec)
made += 1
return doc
+106
View File
@@ -0,0 +1,106 @@
"""base.py — the SurfaceProjector interface + registry.
THE key abstraction of the efferent projector: a projector is a pure function
from the surface-neutral, geometry-carrying DocumentIR to bytes on a target
SURFACE. The surface is a PARAMETER. Adding a surface = registering one more
projector; nothing upstream (plan/realize/cohere) changes.
DocumentIR --project--> bytes (per surface)
A TEXT projector reads ``block.sentences``. A NON-TEXT projector (music, image,
video) reads ``block.provenance`` — the geometry the IR carries — and decodes it
onto its surface. Both consume the SAME IR. That symmetry is the whole design:
the realizer generalizes into a multimodal projector, geometry -> any surface.
"""
from __future__ import annotations
from typing import Protocol, runtime_checkable
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from document_ir import DocumentIR # noqa: E402
@runtime_checkable
class SurfaceProjector(Protocol):
"""Geometry-document -> one surface. Implementations MUST be pure & faithful.
THE ONE SHARED SEAM. Every surface — text, music, image, video — conforms to
this single contract:
project(frame: DocumentIR) -> bytes
where ``frame`` is the geometry-carrying meaning-geometry (the SemFrame at
document scale; a single utterance is the degenerate one-section frame).
RECOMMENDED INTERNAL SHAPE (the peer music/text decomposition, blessed here
so all surfaces share it): a projector may split ``project`` into
spec = self.plan(frame) # meaning-geometry -> surface-specific spec
bytes = self.realize(spec) # spec -> surface, via this projector's PROFILE
``project`` is then ``realize(plan(frame))``. The PROFILE (a text lang-profile,
a music instr/mode-profile, an image layout-profile) is a property of the
projector instance — the pluggable knob. See :class:`TwoStageProjector`.
A TEXT projector's plan reads ``frame`` sentences; a MUSIC/IMAGE projector's
plan reads ``frame.all_provenance()`` — the geometry — and derives its spec
(pitch/harmony/rhythm, or layout) FROM the meaning, deterministically. Same
frame, different profile.
"""
surface: str # "markdown" | "docx" | "midi" | "audio" | "image" | "video"
media_type: str # MIME type of the emitted bytes
ext: str # file extension (no dot)
modality: str # "text" | "audio" | "image" | "video"
profile: object # the pluggable per-surface profile (may be None)
def project(self, doc: DocumentIR) -> bytes:
"""Emit the document on this surface. Returns raw bytes."""
...
class TwoStageProjector:
"""Optional base for the peer plan()/realize() decomposition.
Subclasses implement ``plan(frame) -> spec`` and ``realize(spec) -> bytes``;
``project`` is their composition. This is exactly the peer music interface
(spec = plan(frame, profile); surface = realize(spec, profile)) expressed so
that it still satisfies the single ``SurfaceProjector.project`` seam. Text,
music, and image projectors can all subclass this and remain interchangeable.
"""
surface: str = ""
media_type: str = ""
ext: str = ""
modality: str = ""
profile: object = None
def plan(self, doc: DocumentIR): # -> spec
raise NotImplementedError
def realize(self, spec) -> bytes:
raise NotImplementedError
def project(self, doc: DocumentIR) -> bytes:
return self.realize(self.plan(doc))
_REGISTRY: dict[str, SurfaceProjector] = {}
def register(projector: SurfaceProjector) -> SurfaceProjector:
_REGISTRY[projector.surface] = projector
return projector
def get_projector(surface: str) -> SurfaceProjector:
if surface not in _REGISTRY:
raise KeyError(f"no projector registered for surface {surface!r}; "
f"have {sorted(_REGISTRY)}")
return _REGISTRY[surface]
def available_surfaces() -> list[str]:
return sorted(_REGISTRY)
+113
View File
@@ -0,0 +1,113 @@
"""docx.py — the .docx surface projector: an OWN minimal OOXML emitter.
Own-the-core: a .docx is just a ZIP of a few XML parts (WordprocessingML). We
emit it with the standard library only — ``zipfile`` + string XML — no
python-docx, no external dependency. This proves a "richer structured format"
surface without importing anyone else's toolkit.
Parts emitted (the minimal valid set + a styles part for real headings):
[Content_Types].xml
_rels/.rels
word/_rels/document.xml.rels
word/styles.xml (Title / Heading1 / Heading2 / Normal)
word/document.xml (the content)
Like the markdown projector it reads only the IR's realized sentences; it
invents nothing. The surface differs, the faithful content does not.
"""
from __future__ import annotations
import io
import os
import sys
import zipfile
from xml.sax.saxutils import escape
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from document_ir import DocumentIR # noqa: E402
from projectors.base import register # noqa: E402
_CONTENT_TYPES = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">
<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>
<Default Extension="xml" ContentType="application/xml"/>
<Override PartName="/word/document.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/>
<Override PartName="/word/styles.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.styles+xml"/>
</Types>"""
_RELS = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="word/document.xml"/>
</Relationships>"""
_DOC_RELS = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/styles" Target="styles.xml"/>
</Relationships>"""
_W = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
_STYLES = f"""<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<w:styles xmlns:w="{_W}">
<w:style w:type="paragraph" w:default="1" w:styleId="Normal"><w:name w:val="Normal"/>
<w:rPr><w:sz w:val="22"/></w:rPr></w:style>
<w:style w:type="paragraph" w:styleId="Title"><w:name w:val="Title"/>
<w:pPr><w:spacing w:after="240"/></w:pPr>
<w:rPr><w:b/><w:sz w:val="52"/></w:rPr></w:style>
<w:style w:type="paragraph" w:styleId="Subtitle"><w:name w:val="Subtitle"/>
<w:rPr><w:i/><w:sz w:val="28"/><w:color w:val="555555"/></w:rPr></w:style>
<w:style w:type="paragraph" w:styleId="Heading1"><w:name w:val="heading 1"/>
<w:pPr><w:spacing w:before="240" w:after="120"/><w:outlineLvl w:val="0"/></w:pPr>
<w:rPr><w:b/><w:sz w:val="34"/></w:rPr></w:style>
<w:style w:type="paragraph" w:styleId="Heading2"><w:name w:val="heading 2"/>
<w:pPr><w:spacing w:before="200" w:after="100"/><w:outlineLvl w:val="1"/></w:pPr>
<w:rPr><w:b/><w:sz w:val="28"/></w:rPr></w:style>
</w:styles>"""
def _para(text: str, style: str | None = None) -> str:
ppr = f"<w:pPr><w:pStyle w:val=\"{style}\"/></w:pPr>" if style else ""
return (f"<w:p>{ppr}<w:r><w:t xml:space=\"preserve\">"
f"{escape(text)}</w:t></w:r></w:p>")
class DocxProjector:
surface = "docx"
media_type = ("application/vnd.openxmlformats-officedocument."
"wordprocessingml.document")
ext = "docx"
modality = "text"
def _document_xml(self, doc: DocumentIR) -> str:
body: list[str] = [_para(doc.title, "Title")]
if doc.subtitle:
body.append(_para(doc.subtitle, "Subtitle"))
abstract = doc.meta.get("abstract")
if abstract is not None and abstract.sentences:
body.append(_para(abstract.text()))
for sec in doc.sections:
style = "Heading1" if sec.level <= 1 else "Heading2"
body.append(_para(sec.heading, style))
for block in sec.blocks:
t = block.text()
if t:
body.append(_para(t))
return (f"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>"
f"<w:document xmlns:w=\"{_W}\"><w:body>"
+ "".join(body)
+ "<w:sectPr><w:pgSz w:w=\"12240\" w:h=\"15840\"/>"
"<w:pgMar w:top=\"1440\" w:right=\"1440\" w:bottom=\"1440\" "
"w:left=\"1440\"/></w:sectPr></w:body></w:document>")
def project(self, doc: DocumentIR) -> bytes:
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as z:
z.writestr("[Content_Types].xml", _CONTENT_TYPES)
z.writestr("_rels/.rels", _RELS)
z.writestr("word/_rels/document.xml.rels", _DOC_RELS)
z.writestr("word/styles.xml", _STYLES)
z.writestr("word/document.xml", self._document_xml(doc))
return buf.getvalue()
register(DocxProjector())
+45
View File
@@ -0,0 +1,45 @@
"""markdown.py — the Markdown surface projector (text facet).
The most tractable surface, and the reference implementation: reads the IR's
realized sentences and lays them out as Markdown. Introduces no content — it is
pure typography over the faithful text the realizer produced.
"""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from document_ir import DocumentIR # noqa: E402
from projectors.base import register # noqa: E402
class MarkdownProjector:
surface = "markdown"
media_type = "text/markdown"
ext = "md"
modality = "text"
def render_str(self, doc: DocumentIR) -> str:
lines: list[str] = [f"# {doc.title}"]
if doc.subtitle:
lines.append(f"\n*{doc.subtitle}*")
abstract = doc.meta.get("abstract")
if abstract is not None and abstract.sentences:
lines.append("")
lines.append(abstract.text())
for sec in doc.sections:
lines.append("")
lines.append(f"{'#' * max(2, sec.level)} {sec.heading}")
for block in sec.blocks:
body = block.text()
if body:
lines.append("")
lines.append(body)
return "\n".join(lines) + "\n"
def project(self, doc: DocumentIR) -> bytes:
return self.render_str(doc).encode("utf-8")
register(MarkdownProjector())
+133
View File
@@ -0,0 +1,133 @@
"""midi.py — the MUSIC surface projector: geometry -> symbolic music (MIDI).
The first NON-TEXT surface, and the proof of the general shape. "Music is
language and it is math" (Will): symbolic music is tractable and geometry-native,
so it is the natural efferent twin to try first after text.
CRUCIALLY this projector does NOT read the realized sentences. It reads the IR's
GEOMETRY facet — ``block.provenance`` — and DECODES each edge onto a musical
surface. That is the whole thesis of the multimodal projector: the same
geometry-carrying IR drives text AND music; a text projector reads the words, a
music projector reads the meaning-geometry. The mapping is deterministic and
faithful to the geometry's structure:
relation lemma -> scale degree (same relation -> same pitch class;
meaning has a consistent sonic form)
polarity -> mode (aff = major third above; neg = minor
third / lowered — SACRED polarity is
audible, a negated edge sounds negated)
confidence -> note duration (stronger grounding rings longer)
importance -> velocity (more important source = louder)
section -> phrase + register shift (structure becomes musical form)
Own-the-core: a Standard MIDI File is a header chunk + a track chunk of
delta-timed events. We emit the raw bytes with ``struct`` — no external MIDI
library. Format 0, one track.
"""
from __future__ import annotations
import io
import os
import struct
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from document_ir import DocumentIR, Provenance # noqa: E402
from projectors.base import TwoStageProjector, register # noqa: E402
_TICKS = 480 # ticks per quarter note
_C_MAJOR = [0, 2, 4, 5, 7, 9, 11] # semitone offsets of a diatonic scale
def _vlq(n: int) -> bytes:
"""MIDI variable-length quantity encoding of a delta time."""
if n == 0:
return b"\x00"
out = bytearray()
out.append(n & 0x7F)
n >>= 7
while n:
out.insert(0, (n & 0x7F) | 0x80)
n >>= 7
return bytes(out)
def _degree_for(relation: str) -> int:
"""Stable scale degree for a relation lemma (same relation -> same pitch)."""
if not relation:
return 0
return sum(ord(c) for c in relation.lower()) % len(_C_MAJOR)
def _note_for(p: Provenance, base: int) -> tuple[int, int, int]:
"""(pitch, velocity, duration_ticks) for one geometry edge."""
root = base + _C_MAJOR[_degree_for(p.relation)]
# polarity -> mode: affirmed edges take the bright major third, negated edges
# take the darker minor third. The negation is AUDIBLE and never dropped.
third = 4 if p.polarity == "aff" else 3
pitch = max(24, min(96, root + (third if p.confidence >= 0.5 else 0)))
velocity = int(56 + 60 * min(1.0, max(0.0, p.importance)))
velocity = max(40, min(120, velocity))
# confidence -> duration: quarter .. dotted-half
dur = int(_TICKS * (0.5 + 1.5 * min(1.0, max(0.0, p.confidence))))
return pitch, velocity, dur
# a mode-profile: the pluggable musical knob (the peer's mode_profile). Scale +
# tempo. Swapping this profile re-voices the SAME geometry — surface as parameter.
_DEFAULT_PROFILE = {"scale": _C_MAJOR, "tempo_us": 500000,
"registers": [60, 55, 64, 50, 67, 48], "program": 0}
class MidiProjector(TwoStageProjector):
"""geometry -> symbolic music, in the shared two-stage shape.
``plan(frame)`` -> a music_spec: an ordered list of note dicts derived
deterministically from the frame's provenance geometry
(the peer's ``plan(frame, profile) -> spec``).
``realize(spec)`` -> Standard MIDI File bytes (the peer's
``realize(spec, profile) -> surface``; here the surface
is symbolic MIDI, the minimal audio proof — a richer
additive-synth audio projector conforms identically).
"""
surface = "midi"
media_type = "audio/midi"
ext = "mid"
modality = "audio"
def __init__(self, profile: dict | None = None):
self.profile = profile or _DEFAULT_PROFILE
# -- stage 1: meaning-geometry -> music_spec (reads the GEOMETRY facet) -- #
def plan(self, doc: DocumentIR) -> list[dict]:
registers = self.profile["registers"]
spec: list[dict] = []
for si, sec in enumerate(doc.sections):
base = registers[si % len(registers)]
provs = [p for p in sec.all_provenance()
if p.kind in ("fact", "interpretation")]
for i, p in enumerate(provs):
pitch, vel, dur = _note_for(p, base)
spec.append({"pitch": pitch, "velocity": vel, "dur": dur,
"rest_before": (_TICKS // 2) if (si > 0 and i == 0) else 0,
"relation": p.relation, "polarity": p.polarity})
return spec
# -- stage 2: music_spec -> MIDI bytes (own-core, no library) ------------ #
def realize(self, spec: list[dict]) -> bytes:
ev = bytearray()
ev += _vlq(0) + b"\xFF\x51\x03" + struct.pack(">I", self.profile["tempo_us"])[1:]
ev += _vlq(0) + bytes([0xC0, self.profile["program"] & 0x7F])
for note in spec:
ev += _vlq(note["rest_before"]) + bytes([0x90, note["pitch"], note["velocity"]])
ev += _vlq(note["dur"]) + bytes([0x80, note["pitch"], 0])
ev += _vlq(0) + b"\xFF\x2F\x00"
track = bytes(ev)
buf = io.BytesIO()
buf.write(b"MThd" + struct.pack(">IHHH", 6, 0, 1, _TICKS))
buf.write(b"MTrk" + struct.pack(">I", len(track)) + track)
return buf.getvalue()
register(MidiProjector())
+60
View File
@@ -0,0 +1,60 @@
"""seams.py — documented efferent seams for IMAGE and VIDEO surfaces.
These are NOT implemented (per the build rails: architect, do not overbuild).
They are registered as first-class seams so the interface PROVES it accepts
future non-text projectors without any upstream change. Each documents exactly
what its decoder would read from the geometry-carrying IR, making the multimodal
generalization concrete rather than hand-wavy.
The symmetry that guarantees these are possible, not moonshots: they are the
efferent twins of multimodal INGEST. If meaning can HOLD an image (ingest as
first-class geometry), meaning can PROJECT one back. Video = image x sound x
TIME, and the engram already stores time (chronoception). So video falls out of
an image projector + the music projector + the stored temporal ordering.
"""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from document_ir import DocumentIR # noqa: E402
from projectors.base import register # noqa: E402
class _Seam:
"""A registered-but-unimplemented projector. Names its decoder contract."""
def project(self, doc: DocumentIR) -> bytes: # pragma: no cover - seam
raise NotImplementedError(
f"{self.surface!r} projector is a documented seam, not yet built. "
f"Decoder contract: {self.decoder_contract}")
class ImageProjector(_Seam):
surface = "image"
media_type = "image/png"
ext = "png"
modality = "image"
decoder_contract = (
"reads block.provenance as a spatial layout — nodes become regions, edges "
"become adjacencies; salience/importance drive size/contrast; polarity "
"drives figure/ground. The efferent twin of image ingest (a geometry->raster "
"decoder, learned or engineered), exactly mirroring the embedder that turned "
"the image INTO geometry.")
class VideoProjector(_Seam):
surface = "video"
media_type = "video/mp4"
ext = "mp4"
modality = "video"
decoder_contract = (
"image x sound x TIME. Composes the image projector (per-keyframe geometry "
"layout) with the midi/music projector (score) along the geometry's stored "
"temporal ordering (chronoception). Needs no new principle once image + music "
"exist — only a muxer.")
register(ImageProjector())
register(VideoProjector())
+63
View File
@@ -0,0 +1,63 @@
"""provenance.py — the faithfulness audit + geometry->section trace.
A document projected from geometry is only worth anything if every claim traces
back. This module walks the DocumentIR and proves the discipline held:
* ZERO ungrounded claims (every fact/interpretation has a real node id),
* every emitted sentence maps to a geometry edge (or is a marked connective),
* SACRED polarity survived (negations are reported, never silently dropped),
* COHERE introduced no new geometry (connectives carry no claim).
It emits both a machine verdict and a human-readable geometry->section table.
"""
from __future__ import annotations
from document_ir import DocumentIR
def audit(doc: DocumentIR) -> dict:
provs = doc.all_provenance()
facts = [p for p in provs if p.kind in ("fact", "interpretation")]
connectives = [p for p in provs if p.kind == "connective"]
ungrounded = [p for p in facts if not p.node_id]
negations = [p for p in facts if p.polarity == "neg"]
node_ids = sorted({p.node_id for p in facts if p.node_id})
return {
"claims": len(facts),
"connectives": len(connectives),
"ungrounded_claims": len(ungrounded),
"negations_preserved": len(negations),
"distinct_source_nodes": len(node_ids),
"faithful": len(ungrounded) == 0,
"source_nodes": node_ids,
}
def trace_table(doc: DocumentIR) -> str:
"""Human-readable geometry -> section -> claim provenance table."""
lines = ["# Provenance — every claim traces geometry", ""]
lines.append(f"**Document:** {doc.title}")
a = audit(doc)
lines.append(f"**Claims:** {a['claims']} · **Ungrounded:** "
f"{a['ungrounded_claims']} · **Negations preserved:** "
f"{a['negations_preserved']} · **Source nodes:** "
f"{a['distinct_source_nodes']} · **Faithful:** "
f"{'YES' if a['faithful'] else 'NO'}")
lines.append("")
for si, sec in enumerate(doc.sections, 1):
lines.append(f"## {si}. {sec.heading}")
lines.append(f"_seed nodes: {', '.join(i[:8] for i in sec.seed_ids)}_")
lines.append("")
lines.append("| # | realized claim | traces geometry edge |")
lines.append("|---|----------------|----------------------|")
n = 0
for block in sec.blocks:
for sent, prov in zip(block.sentences, block.provenance):
if prov.kind == "connective":
continue
n += 1
edge = prov.trace().replace("|", "\\|")
s = sent.replace("|", "\\|")
lines.append(f"| {n} | {s} | {edge} |")
lines.append("")
return "\n".join(lines) + "\n"
+112
View File
@@ -0,0 +1,112 @@
"""realize.py — REALIZE stage: fill each planned section with faithful passages.
Scales the PROVEN realizer from a single assertion to a passage. For each
planned proposition we build a realizer-ready clause (the proven
``_prop_to_clause`` mapping) and run it through the proven engine
(``engine.realize``), which is a deterministic grammar with the SACRED negation
contract — it never invents. Each realized sentence is paired with a
:class:`Provenance` that pins it to the exact geometry edge it came from.
"Passage, not a list of sentences": within a section we lightly vary sentence
openings and group related claims, but we add NO content the geometry did not
assert. The only non-geometry words are function words the grammar already owns
(articles, "and", conjunction of same-subject claims). Document-level flow is
COHERE's job; this stage owns intra-section fluency + fidelity.
"""
from __future__ import annotations
import os
import sys
_NT = os.path.expanduser("~/Desktop/neuron-talk")
_LR = os.path.expanduser("~/Desktop/lang-realizers")
for _p in (_NT, _LR):
if _p not in sys.path:
sys.path.insert(0, _p)
import engine # noqa: E402 (the proven no-LLM realizer)
from dialogue import _prop_to_clause # noqa: E402 (proven prop -> clause)
from document_ir import Block, DocumentIR, Provenance, Section # noqa: E402
def _provenance_from(p, kind: str = "fact") -> Provenance:
return Provenance(
subj_id=p.source_node_id, subject=p.subject, relation=p.predicate,
obj=p.object, polarity=p.polarity, confidence=round(float(p.confidence), 3),
node_id=p.source_node_id, kind=kind,
importance=float(getattr(p, "node_importance", 0.0) or 0.0),
salience=0.0,
)
import re as _re
# a well-formed declarative opens with a determiner, a proper noun, "I", or a
# capitalized head — not a mis-parsed object pronoun or a copula fragment.
_BAD_OPENERS = _re.compile(r"^(Me |It is I|There is|This is it|That is it)\b")
_VACUOUS = _re.compile(r"^\w+ (is|are|was|were) (it|no|nothing|empty|those|this|that)\.?$",
_re.I)
def _good_sentence(text: str) -> bool:
"""Fluency gate — drops degenerate realizations. NEVER loosens faithfulness;
it only refuses to SPEAK a claim whose surface came out malformed."""
words = text.rstrip(".").split()
if len(words) < 3:
return False
if _BAD_OPENERS.search(text):
return False
if _VACUOUS.match(text):
return False
# a sentence that is mostly one-letter/two-letter tokens is a parse artifact
short = sum(1 for w in words if len(w.strip(".,'")) <= 2)
if short > len(words) / 2:
return False
return True
def _realize_prop(p, lang: str = "en") -> tuple[str, Provenance] | None:
"""One proposition -> (faithful sentence, provenance) or None if it drops."""
clause = _prop_to_clause(p)
text = engine.realize(clause, lang)
if not text or not text.strip():
return None
text = text.strip()
if not text.endswith((".", "!", "?")):
text += "."
# capitalize first character (proper nouns / "I" already handled by grammar)
text = text[0].upper() + text[1:]
if not _good_sentence(text):
return None
return text, _provenance_from(p)
def realize_document(doc: DocumentIR, lang: str = "en") -> DocumentIR:
"""Fill every planned section's blocks with faithful, realized passages."""
for sec in doc.sections:
planned = sec.__dict__.get("_planned_props", [])
block = Block(role="body")
summary_bits: list[str] = []
for p in planned:
r = _realize_prop(p, lang)
if r is None:
continue
text, prov = r
block.sentences.append(text)
block.provenance.append(prov)
if len(summary_bits) < 1:
# a short grounded gloss for TOC / pptx bullets
obj = (prov.obj or "").strip().rstrip(".")
if obj:
summary_bits.append(obj)
if block.sentences:
sec.blocks.append(block)
sec.summary = summary_bits[0] if summary_bits else ""
# drop the transient planning payload; the IR is now self-contained
sec.__dict__.pop("_planned_props", None)
sec.__dict__.pop("_node", None)
# prune sections that realized to nothing
doc.sections = [s for s in doc.sections if s.blocks]
return doc
+136
View File
@@ -0,0 +1,136 @@
// accent.el - A British-RP ACCENT as an INGESTED TRANSFORM-GEOMETRY, composed
// onto the voice (voice (+) accent, SEPARABLE). Reads elp/data/british-accent.psv
// into an accent MANIFOLD in the engram (override nodes + a shared accent hub),
// and the render reads the RP formant overrides + the non-rhotic rule back from
// that geometry. NO accent targets live in code same discipline as the base
// phonetics. PROVENANCE NOTE: the RP Hz values are PROVISIONAL (reconstructed-
// from-knowledge approximations, cite Deterding1997 / Hawkins&Midgley2005 /
// Wells1982) pending transcription from the published tables the PIPELINE is
// the deliverable; exact values are being source-verified separately.
fn ingest_accent(path: String) -> [String] {
let content: String = fs_read(path)
let lines: [String] = str_split(content, "\n")
let nl: Int = native_list_len(lines)
let amap: [String] = native_list_empty()
let hub: String = engram_node("accent british-rp prov=PROVISIONAL cite=Deterding1997-HawkinsMidgley2005-Wells1982", "Accent", 80)
let li: Int = 0
while li < nl {
let line: String = native_list_get(lines, li)
let ll: Int = str_len(line)
let skip: Int = 0
if ll < 3 {
skip = 1
}
if skip == 0 {
let first: Int = str_char_code(line, 0)
if first == 35 {
skip = 1
}
}
if skip == 0 {
let f: [String] = str_split(line, "|")
let nf: Int = native_list_len(f)
if nf >= 6 {
let key: String = native_list_get(f, 0)
let f1: String = native_list_get(f, 1)
let f2: String = native_list_get(f, 2)
let f3: String = native_list_get(f, 3)
let kind: String = native_list_get(f, 4)
let set: String = native_list_get(f, 5)
let cont: String = "accent british-rp " + key + " f1=" + f1 + " f2=" + f2 + " f3=" + f3 + " kind=" + kind + " set=" + set + " prov=PROVISIONAL cite=Deterding1997-HawkinsMidgley2005-Wells1982"
let id: String = engram_node(cont, "AccentTarget", 80)
amap = native_list_append(amap, key)
amap = native_list_append(amap, cont)
engram_connect(id, hub, 80, "of_accent")
}
}
li = li + 1
}
return amap
}
// RP formant override for a phoneme, read from the accent manifold. Returns
// [f1,f2,f3] for a vowel_override record, or an empty list if none / a rule.
fn accent_formants(amap: [String], code: String) -> [Int] {
let out: [Int] = native_list_empty()
let id: String = sp_map_get(amap, code)
if str_eq(id, "") {
return out
}
let j: String = id
let isrule: Int = str_index_of(j, "drop_coda")
if isrule >= 0 {
return out
}
let f1: Int = parse_uint_from(j, "f1=")
if f1 <= 0 {
return out
}
let out = native_list_append(out, f1)
let out = native_list_append(out, parse_uint_from(j, "f2="))
let out = native_list_append(out, parse_uint_from(j, "f3="))
return out
}
// Is this accent non-rhotic? (reads the R rule node from the manifold)
fn is_nonrhotic(amap: [String]) -> Int {
let id: String = sp_map_get(amap, "R")
if str_eq(id, "") {
return 0
}
let hit: Int = str_index_of(id, "drop_coda")
if hit >= 0 {
return 1
}
return 0
}
// Is this symbol a vowel? Membership in the vowel-set derived from the phonetics
// source's class column (phonological structure the FORMANT NUMBERS still come
// from the organ manifold; this is only the categorical class for the rule).
fn is_vowel_sym(vset: [String], sym: String) -> Int {
let n: Int = native_list_len(vset)
let i: Int = 0
while i < n {
if str_eq(native_list_get(vset, i), sym) {
return 1
}
i = i + 1
}
return 0
}
// Non-rhotic transform: drop a post-vocalic CODA /R/ an R whose next non-SIL
// phoneme is NOT a vowel (a consonant, or end of utterance). Keep INTERVOCALIC/
// onset R (next non-SIL phoneme is a vowel, e.g. the medial R in N UW R AA N).
fn apply_rhoticity(codes: [String], vset: [String]) -> [String] {
let n: Int = native_list_len(codes)
let out: [String] = native_list_empty()
let i: Int = 0
while i < n {
let c: String = native_list_get(codes, i)
let keep: Int = 1
if str_eq(c, "R") {
let jx: Int = i + 1
let nextv: Int = 0
while jx < n {
let ncode: String = native_list_get(codes, jx)
if str_eq(ncode, "SIL") {
jx = jx + 1
} else {
nextv = is_vowel_sym(vset, ncode)
jx = n + 1000
}
}
if nextv == 0 {
keep = 0
}
}
if keep == 1 {
out = native_list_append(out, c)
}
i = i + 1
}
return out
}
+73
View File
@@ -0,0 +1,73 @@
// audio-demo.el - Drive the native audio surface: render a tone per instrument
// from its LEARNED signature, then render a small meaning-phrase "piece".
// Entry point: top-level statement calls main() (same convention as the
// examples' top-level println(run_test())).
fn micros_to_str(xs: [Int]) -> String {
let n: Int = native_list_len(xs)
let out: String = ""
let i: Int = 0
while i < n {
if i > 0 { let out: String = out + "," }
let out: String = out + int_to_str(native_list_get(xs, i))
let i: Int = i + 1
}
return out
}
// Render a 1.0s A4 (midi 69) tone from a signature file, print the parsed
// partials (proving the numbers came from the engram .sig), write the WAV.
fn render_tone(name: String, sigpath: String, outpath: String, table: [Int]) -> Int {
let lines: [String] = sig_load(sigpath)
let partials: [Int] = parse_micros(sig_field(lines, "partials"))
println("[" + name + "] partials_n=" + sig_field(lines, "partials_n") + " parsed_partials_micro(scale 1e6)=" + micros_to_str(partials))
println("[" + name + "] raw partials line from .sig = " + sig_field(lines, "partials"))
let freq: Int = freq_of_midi(69)
let note: [Int] = synth_from_sig(lines, freq, 1000, 900, 44100, table)
let n: Int = native_list_len(note)
let ok: Int = wav_write(outpath, note, n, 44100)
println("[" + name + "] rendered " + int_to_str(n) + " samples -> " + outpath + " (write_ok=" + int_to_str(ok) + ")")
return n
}
fn run_demo() -> Int {
let table: [Int] = sin_table()
fs_mkdir("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out")
println("=== TONES: render A4 (midi 69) from each learned signature ===")
render_tone("flute", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/flute.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-flute.wav", table)
render_tone("clarinet", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/clarinet.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-clarinet.wav", table)
render_tone("violin", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/violin.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-violin.wav", table)
render_tone("piano", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/piano.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-piano.wav", table)
render_tone("organ", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/organ.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-organ.wav", table)
println("")
println("=== PIECE: a 6-frame meaning phrase (incl. a NEG frame) ===")
let frames: [[String]] = native_list_empty()
let frames: [[String]] = native_list_append(frames, audio_frame("agent", "aff", "0.9", "0.8", "0", "s1"))
let frames: [[String]] = native_list_append(frames, audio_frame("theme", "aff", "0.7", "0.6", "0", "s2"))
let frames: [[String]] = native_list_append(frames, audio_frame("cause", "aff", "0.8", "0.9", "1", "s3"))
let frames: [[String]] = native_list_append(frames, audio_frame("negation", "neg", "0.85", "0.7", "0", "s4"))
let frames: [[String]] = native_list_append(frames, audio_frame("goal", "aff", "0.6", "0.5", "1", "s5"))
let frames: [[String]] = native_list_append(frames, audio_frame("result", "aff", "0.95", "1.0", "0", "s6"))
// Print the plan so the NEG frame's minor third (+3) vs major (+4) is visible.
let nf: Int = native_list_len(frames)
let fi: Int = 0
while fi < nf {
let frame: [String] = native_list_get(frames, fi)
let plan: [Int] = plan_note(frame)
let pol: String = surface_get(frame, "polarity")
let third_name: String = "major(+4)"
if str_eq(pol, "neg") { let third_name: String = "MINOR(+3)" }
println("frame " + int_to_str(fi) + " relation=" + surface_get(frame, "relation") + " polarity=" + pol + " -> midi=" + int_to_str(native_list_get(plan, 0)) + " dur_ms=" + int_to_str(native_list_get(plan, 1)) + " amp_pm=" + int_to_str(native_list_get(plan, 2)) + " third=" + third_name)
let fi: Int = fi + 1
}
let piano_lines: [String] = sig_load("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/piano.sig")
let total: Int = realize_audio(frames, piano_lines, "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/piece.wav", 44100, table)
println("PIECE rendered " + int_to_str(total) + " samples -> /Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/piece.wav")
return total
}
println("audio-demo main returned samples=" + int_to_str(run_demo()))
+400
View File
@@ -0,0 +1,400 @@
// audio-surface.el - Native own-core additive-synthesis audio surface.
//
// The AUDIO efferent seam, native, no Python and no library. This renders real
// PCM .wav bytes from instrument SIGNATURES read from engram-sourced .sig data
// files (elp/faculty/sig/*.sig) - the partial amplitudes are NEVER literals in
// this source; they are parsed from the learned signature at run time. That is
// the whole proof: render-from-learned-signatures.
//
// EL has no float arithmetic operator (codegen emits raw int64 ops for + - * /
// on the shared 64-bit slot) and no float-arithmetic natives - so ALL synthesis
// math here is own-core INTEGER fixed-point. Angles use a quarter-wave sine
// table (scale 10000) from a fixed-point Taylor series; amplitudes are parsed to
// micro (scale 1e6) straight from the .sig text; frequencies are milliHz ints.
//
// Pipeline mirrors the two-stage projector (midi.py): plan_note(frame) reads a
// frame's meaning-geometry slot-map and derives (pitch, duration, amplitude);
// realize_audio SUPERPOSES the signature's partials (the compose op) and
// serialises RIFF/WAVE. Same frame -> midi OR audio.
// -- integer decimal + string helpers -----------------------------------------
fn str_to_int_el(s: String) -> Int {
let n: Int = str_len(s)
let i: Int = 0
let v: Int = 0
let neg: Bool = false
while i < n {
let c: Int = str_char_code(s, i)
if c == 45 { let neg: Bool = true }
if c >= 48 {
if c < 58 {
let v: Int = v * 10 + (c - 48)
}
}
let i: Int = i + 1
}
if neg { return 0 - v }
return v
}
fn parse_micro(s: String) -> Int {
let dot: Int = str_index_of(s, ".")
if dot < 0 {
return str_to_int_el(s) * 1000000
}
let n: Int = str_len(s)
let ipart: String = str_slice(s, 0, dot)
let fpart: String = str_slice(s, dot + 1, n)
let iv: Int = str_to_int_el(ipart)
let fv: Int = 0
let scale: Int = 100000
let fn2: Int = str_len(fpart)
let i: Int = 0
while i < 6 {
let d: Int = 0
if i < fn2 {
let d: Int = str_char_code(fpart, i) - 48
}
let fv: Int = fv + d * scale
let scale: Int = scale / 10
let i: Int = i + 1
}
return iv * 1000000 + fv
}
// -- signature (engram data file) loader ---------------------------------------
fn sig_load(path: String) -> [String] {
let text: String = fs_read(path)
return str_split(text, "\n")
}
fn sig_field(lines: [String], key: String) -> String {
let pref: String = key + ": "
let n: Int = native_list_len(lines)
let plen: Int = str_len(pref)
let i: Int = 0
while i < n {
let ln: String = native_list_get(lines, i)
if str_starts_with(ln, pref) {
return str_slice(ln, plen, str_len(ln))
}
let i: Int = i + 1
}
return ""
}
fn parse_micros(csv: String) -> [Int] {
let parts: [String] = str_split(csv, ",")
let n: Int = native_list_len(parts)
let out: [Int] = native_list_empty()
let i: Int = 0
while i < n {
let out: [Int] = native_list_append(out, parse_micro(native_list_get(parts, i)))
let i: Int = i + 1
}
return out
}
// -- fixed-point sine (own-core, quarter-wave Taylor table, scale 10000) --------
fn sin_table() -> [Int] {
let HP: Int = 1570796
let t: [Int] = native_list_empty()
let q: Int = 0
while q < 257 {
let x: Int = q * HP / 256
let x2: Int = x * x / 1000000
let x3: Int = x2 * x / 1000000
let x5: Int = x3 * x2 / 1000000
let x7: Int = x5 * x2 / 1000000
let x9: Int = x7 * x2 / 1000000
let s: Int = x - x3 / 6 + x5 / 120 - x7 / 5040 + x9 / 362880
let t: [Int] = native_list_append(t, s / 100)
let q: Int = q + 1
}
return t
}
fn sin_lookup(t: [Int], phase: Int) -> Int {
let p: Int = phase % 1024
if p < 0 { let p: Int = p + 1024 }
let quad: Int = p / 256
let r: Int = p % 256
if quad == 0 { return native_list_get(t, r) }
if quad == 1 { return native_list_get(t, 256 - r) }
if quad == 2 { return 0 - native_list_get(t, r) }
return 0 - native_list_get(t, 256 - r)
}
fn isqrt_int(n: Int) -> Int {
if n <= 0 { return 0 }
let x: Int = n
let y: Int = (x + 1) / 2
while y < x {
let x: Int = y
let y: Int = (x + n / x) / 2
}
return x
}
// freq_of_midi: equal-tempered frequency in milliHz. 440000 mHz at midi 69.
fn freq_of_midi(m: Int) -> Int {
let f: Int = 440000
if m > 69 {
let k: Int = m - 69
let i: Int = 0
while i < k {
let f: Int = f * 1059463 / 1000000
let i: Int = i + 1
}
return f
}
if m < 69 {
let k: Int = 69 - m
let i: Int = 0
while i < k {
let f: Int = f * 1000000 / 1059463
let i: Int = i + 1
}
return f
}
return f
}
// -- envelope (ADSR), scale 1000 -----------------------------------------------
fn adsr_env(i: Int, total: Int, atk_n: Int, dec_n: Int, sus_pm: Int, rel_n: Int) -> Int {
if i < atk_n {
if atk_n == 0 { return 1000 }
return 1000 * i / atk_n
}
if i < atk_n + dec_n {
if dec_n == 0 { return sus_pm }
return 1000 - (1000 - sus_pm) * (i - atk_n) / dec_n
}
let rel_start: Int = total - rel_n
if i < rel_start {
return sus_pm
}
if rel_n == 0 { return 0 }
let left: Int = total - i
return sus_pm * left / rel_n
}
// -- note synthesis: SUPERPOSE the learned partials -> [Int] samples -----------
fn note_samples(freq_mHz: Int, dur_ms: Int, rate: Int, partials: [Int], sumP: Int, b_micro: Int, vib_rate: Int, vib_cents: Int, atk_ms: Int, dec_ms: Int, sus_pm: Int, rel_ms: Int, amp_pm: Int, table: [Int]) -> [Int] {
let total: Int = dur_ms * rate / 1000
let atk_n: Int = atk_ms * rate / 1000
let dec_n: Int = dec_ms * rate / 1000
let rel_n: Int = rel_ms * rate / 1000
let np: Int = native_list_len(partials)
let half_mhz: Int = rate * 1000 / 2
let out: [Int] = native_list_empty()
let i: Int = 0
while i < total {
let acc: Int = 0
let k: Int = 0
while k < np {
let harm: Int = k + 1
let amp_k: Int = native_list_get(partials, k)
let factor: Int = 1000000
if b_micro > 0 {
let val: Int = 1000000 + b_micro * harm * harm
let factor: Int = isqrt_int(val * 1000000)
}
let fn_mhz: Int = freq_mHz * harm
let fn_mhz: Int = fn_mhz * factor / 1000000
if vib_cents > 0 {
if vib_rate > 0 {
let vphase: Int = i * vib_rate * 1024 / rate
let vs: Int = sin_lookup(table, vphase)
let vibf: Int = 1000000 + (vib_cents * vs * 833) / 10000
let fn_mhz: Int = fn_mhz * vibf / 1000000
}
}
if fn_mhz <= half_mhz {
let phase: Int = i * fn_mhz * 1024 / (rate * 1000)
let sv: Int = sin_lookup(table, phase)
let acc: Int = acc + sv * amp_k / 1000000
}
let k: Int = k + 1
}
let env: Int = adsr_env(i, total, atk_n, dec_n, sus_pm, rel_n)
let s16: Int = acc * 2800000 / sumP
let s16: Int = s16 * env / 1000
let s16: Int = s16 * amp_pm / 1000
if s16 > 32767 { let s16: Int = 32767 }
if s16 < 0 - 32767 { let s16: Int = 0 - 32767 }
let out: [Int] = native_list_append(out, s16)
let i: Int = i + 1
}
return out
}
fn synth_from_sig(lines: [String], freq_mHz: Int, dur_ms: Int, amp_pm: Int, rate: Int, table: [Int]) -> [Int] {
let partials: [Int] = parse_micros(sig_field(lines, "partials"))
let np: Int = native_list_len(partials)
let sumP: Int = 0
let j: Int = 0
while j < np {
let pj: Int = native_list_get(partials, j)
let sumP: Int = sumP + pj
let j: Int = j + 1
}
if sumP <= 0 { let sumP: Int = 1000000 }
let adsr: [String] = str_split(sig_field(lines, "adsr"), ",")
let atk_ms: Int = parse_micro(native_list_get(adsr, 0)) / 1000
let dec_ms: Int = parse_micro(native_list_get(adsr, 1)) / 1000
let sus_pm: Int = parse_micro(native_list_get(adsr, 2)) / 1000
let rel_ms: Int = parse_micro(native_list_get(adsr, 3)) / 1000
let b_micro: Int = parse_micro(sig_field(lines, "inharmonicity_B"))
let vib_rate: Int = str_to_int_el(sig_field(lines, "vibrato_rate_hz"))
let vib_cents: Int = str_to_int_el(sig_field(lines, "vibrato_depth_cents"))
return note_samples(freq_mHz, dur_ms, rate, partials, sumP, b_micro, vib_rate, vib_cents, atk_ms, dec_ms, sus_pm, rel_ms, amp_pm, table)
}
// -- byte-buffer helpers (own-core, no library) --------------------------------
fn put_tag(buf: String, pos: Int, s: String) -> String {
let n: Int = str_len(s)
let i: Int = 0
while i < n {
let buf: String = __str_set_char(buf, pos + i, str_char_code(s, i))
let i: Int = i + 1
}
return buf
}
fn put_u32le(buf: String, pos: Int, v: Int) -> String {
let buf: String = __str_set_char(buf, pos, v % 256)
let buf: String = __str_set_char(buf, pos + 1, (v / 256) % 256)
let buf: String = __str_set_char(buf, pos + 2, (v / 65536) % 256)
let buf: String = __str_set_char(buf, pos + 3, (v / 16777216) % 256)
return buf
}
fn put_u16le(buf: String, pos: Int, v: Int) -> String {
let buf: String = __str_set_char(buf, pos, v % 256)
let buf: String = __str_set_char(buf, pos + 1, (v / 256) % 256)
return buf
}
// -- WAV serializer: own-core RIFF/WAVE, PCM mono 16-bit -----------------------
fn wav_write(path: String, samples: [Int], n: Int, rate: Int) -> Int {
let data_len: Int = n * 2
let total: Int = 44 + data_len
let buf: String = __str_alloc(total)
let buf: String = put_tag(buf, 0, "RIFF")
let buf: String = put_u32le(buf, 4, 36 + data_len)
let buf: String = put_tag(buf, 8, "WAVE")
let buf: String = put_tag(buf, 12, "fmt ")
let buf: String = put_u32le(buf, 16, 16)
let buf: String = put_u16le(buf, 20, 1)
let buf: String = put_u16le(buf, 22, 1)
let buf: String = put_u32le(buf, 24, rate)
let buf: String = put_u32le(buf, 28, rate * 2)
let buf: String = put_u16le(buf, 32, 2)
let buf: String = put_u16le(buf, 34, 16)
let buf: String = put_tag(buf, 36, "data")
let buf: String = put_u32le(buf, 40, data_len)
let i: Int = 0
while i < n {
let v: Int = native_list_get(samples, i)
if v < 0 { let v: Int = v + 65536 }
let buf: String = __str_set_char(buf, 44 + i * 2, v % 256)
let buf: String = __str_set_char(buf, 44 + i * 2 + 1, (v / 256) % 256)
let i: Int = i + 1
}
let ok: Int = fs_write_bytes(path, buf, total)
return ok
}
// -- plan: frame slot-map -> note atom (pitch, duration, amplitude) ------------
fn audio_frame(relation: String, polarity: String, confidence: String, importance: String, salience: String, subj_id: String) -> [String] {
let f: [String] = native_list_empty()
let f: [String] = native_list_append(f, "relation")
let f: [String] = native_list_append(f, relation)
let f: [String] = native_list_append(f, "polarity")
let f: [String] = native_list_append(f, polarity)
let f: [String] = native_list_append(f, "confidence")
let f: [String] = native_list_append(f, confidence)
let f: [String] = native_list_append(f, "importance")
let f: [String] = native_list_append(f, importance)
let f: [String] = native_list_append(f, "salience")
let f: [String] = native_list_append(f, salience)
let f: [String] = native_list_append(f, "subj_id")
let f: [String] = native_list_append(f, subj_id)
return f
}
fn degree_offset(deg: Int) -> Int {
if deg == 0 { return 0 }
if deg == 1 { return 2 }
if deg == 2 { return 4 }
if deg == 3 { return 5 }
if deg == 4 { return 7 }
if deg == 5 { return 9 }
return 11
}
// returns [midi, dur_ms, amp_pm]
fn plan_note(frame: [String]) -> [Int] {
let relation: String = surface_get(frame, "relation")
let polarity: String = surface_get(frame, "polarity")
let confidence: String = surface_get(frame, "confidence")
let importance: String = surface_get(frame, "importance")
let salience: String = surface_get(frame, "salience")
let rn: Int = str_len(relation)
let csum: Int = 0
let i: Int = 0
while i < rn {
let cc: Int = str_char_code(relation, i)
let csum: Int = csum + cc
let i: Int = i + 1
}
let deg: Int = csum % 7
let third: Int = 4
if str_eq(polarity, "neg") { let third: Int = 3 }
let sal_oct: Int = str_to_int_el(salience)
let doff: Int = degree_offset(deg)
let midi: Int = 60 + sal_oct * 12 + doff + third
let conf_micro: Int = parse_micro(confidence)
let dur_ms: Int = 200 + conf_micro / 1000
let imp_micro: Int = parse_micro(importance)
let amp_pm: Int = 400 + imp_micro / 2000
let out: [Int] = native_list_empty()
let out: [Int] = native_list_append(out, midi)
let out: [Int] = native_list_append(out, dur_ms)
let out: [Int] = native_list_append(out, amp_pm)
return out
}
fn realize_audio(frames: [[String]], sig_lines: [String], path: String, rate: Int, table: [Int]) -> Int {
let nf: Int = native_list_len(frames)
let all: [Int] = native_list_empty()
let count: Int = 0
let fi: Int = 0
while fi < nf {
let frame: [String] = native_list_get(frames, fi)
let plan: [Int] = plan_note(frame)
let midi: Int = native_list_get(plan, 0)
let dur_ms: Int = native_list_get(plan, 1)
let amp_pm: Int = native_list_get(plan, 2)
let freq: Int = freq_of_midi(midi)
let note: [Int] = synth_from_sig(sig_lines, freq, dur_ms, amp_pm, rate, table)
let nn: Int = native_list_len(note)
let j: Int = 0
while j < nn {
let all: [Int] = native_list_append(all, native_list_get(note, j))
let j: Int = j + 1
}
let count: Int = count + nn
let fi: Int = fi + 1
}
let ok: Int = wav_write(path, all, count, rate)
return count
}
File diff suppressed because it is too large Load Diff
+16
View File
@@ -0,0 +1,16 @@
// comprehend.elh — public surface of the ELP comprehension front-end.
// text → meaning-spec (the input half of the ELP; inverse of the realizer).
extern fn parse_spec(text: String) -> [String]
extern fn parse_spec_lang(text: String, lang: String) -> [String]
extern fn parse_json(text: String) -> String
extern fn parse_json_lang(text: String, lang: String) -> String
// Analysis primitives (invertible morphology + deterministic grammar helpers):
extern fn cp_tokenize(text: String) -> [String]
extern fn cp_pron_concept(w: String) -> String
extern fn cp_is_negation(w: String) -> Bool
extern fn cp_is_neg_adverb(w: String) -> Bool
extern fn cp_irr2(surface: String) -> [String]
extern fn cp_reg_verb(w: String) -> [String]
extern fn cp_analyze_verb(surface: String) -> [String]
extern fn cp_verb_start(toks: [String], end: Int) -> Int
extern fn cp_subord_start(toks: [String], n: Int) -> Int
+287
View File
@@ -0,0 +1,287 @@
// dialogue.el SUMMON-THROUGH-SELF, native el. Port of dialogue.py's core.
//
// THE WHOLE DIALOGUE IS ONE OPERATION. A fact is never merely *fetched*: the
// query is PROJECTED into the engram's self + memory geometry, LANDS in a region,
// and the reply is READ OUT / the region MATERIALIZED from wherever it landed.
//
// project(query) -> land on a region -> read out from that region
//
// lands in the SELF region -> grounded identity/presence, read out of
// the real self nodes (self_region.el)
// lands on a memory NEIGHBORHOOD -> MATERIALIZE it: walk the neighborhood
// (engram_neighbors_json) and read out the
// region's connected members
// lands nowhere close -> HONEST ABSENCE (an empty region, not a
// fabricated answer, not an error)
//
// CRITICAL INVARIANTS (enforced structurally, not by convention):
// * ONE operation there is NO intent classifier and NO separate
// fact-retrieval branch. Identity is nearest-region proximity, not a switch.
// * MATERIALIZE by walking the neighborhood, never by fetching top-props.
// * HONEST ABSENCE when the region is thin.
// * NEGATION is SACRED: the readout is the stored prose VERBATIM, so a negated
// memory stays negated we never paraphrase a polarity away.
// * NO ECHO: the old "I noted that X. That relates to Y." template is gone.
// The summon path materializes or honestly declines it never echoes.
// * DIRECTIVE OVERRIDE: a meta-directive ("answer in English") overrides the
// reply language while the content language is still auto-detected.
//
// Depends on: comprehend (parse_spec_lang, cp_tokenize), multilingual (ml_detect,
// ml_tr, ml_term), propositions (prop_split_sentences), self_region
// (sr_available, sr_readout), the engram + json runtime builtins.
// directive override
// Return [target_lang, content]. target_lang is "" when no directive is present.
// A directive names an output language; we strip it and keep the remaining text
// as the content (whose OWN language is still auto-detected downstream).
fn dlg_dir_hit(low: String, phrase: String) -> Bool {
return str_contains(low, phrase)
}
fn dlg_parse_directive(text: String) -> [String] {
let low: String = str_to_lower(text)
let lang: String = ""
let phrase: String = ""
// English target
if dlg_dir_hit(low, "in english") { let lang = "en"; let phrase = "in english" }
if dlg_dir_hit(low, "em inglês") { let lang = "en"; let phrase = "em inglês" }
if dlg_dir_hit(low, "em ingles") { let lang = "en"; let phrase = "em ingles" }
if dlg_dir_hit(low, "en inglés") { let lang = "en"; let phrase = "en inglés" }
// Portuguese target
if dlg_dir_hit(low, "in portuguese") { let lang = "pt"; let phrase = "in portuguese" }
if dlg_dir_hit(low, "em português") { let lang = "pt"; let phrase = "em português" }
// Spanish target
if dlg_dir_hit(low, "in spanish") { let lang = "es"; let phrase = "in spanish" }
if dlg_dir_hit(low, "en español") { let lang = "es"; let phrase = "en español" }
// Italian target
if dlg_dir_hit(low, "in italian") { let lang = "it"; let phrase = "in italian" }
let content: String = text
if !str_eq(phrase, "") {
// strip the directive phrase (and a common "answer"/"responda" lead-in),
// leaving the real question as content.
let idx: Int = str_index_of(low, phrase)
if idx >= 0 {
let before: String = str_slice(text, 0, idx)
let after: String = str_slice(text, idx + str_len(phrase), str_len(text))
let content = str_trim(before + " " + after)
}
// trim a leading "answer"/"responda"/"reply" and stray colon/comma.
let cl: String = str_to_lower(content)
if str_starts_with(cl, "answer") { let content = str_trim(str_slice(content, 6, str_len(content))) }
if str_starts_with(cl, "responda") { let content = str_trim(str_slice(content, 8, str_len(content))) }
if str_starts_with(cl, "reply") { let content = str_trim(str_slice(content, 5, str_len(content))) }
if str_starts_with(content, ":") { let content = str_trim(str_slice(content, 1, str_len(content))) }
if str_starts_with(content, ",") { let content = str_trim(str_slice(content, 1, str_len(content))) }
}
let r: [String] = native_list_empty()
let r = native_list_append(r, lang)
let r = native_list_append(r, content)
return r
}
// identity landing (a region proximity, not a classifier switch)
// The query lands in the SELF region when it takes an identity/presence shape.
// Cross-lingual forms are included because the engram's lexical probe is
// English-leaning. This is the SELF attractor of the single operation.
fn dlg_is_identity(content: String) -> Bool {
let low: String = str_to_lower(str_trim(content))
if str_contains(low, "who are you") { return true }
if str_contains(low, "what are you") { return true }
if str_contains(low, "who i am") { return true }
if str_contains(low, "your name") { return true }
if str_contains(low, "about yourself") { return true }
if str_contains(low, "are you conscious") { return true }
if str_contains(low, "are you there") { return true }
// cross-lingual identity question-forms
if str_contains(low, "quem é você") { return true }
if str_contains(low, "quem es voce") { return true }
if str_contains(low, "quién eres") { return true }
if str_contains(low, "quien eres") { return true }
if str_contains(low, "chi sei") { return true }
if str_contains(low, "qui es-tu") { return true }
if str_contains(low, "wer bist du") { return true }
return false
}
// readout helpers
fn dlg_first_sentence(content: String) -> String {
let sents: [String] = prop_split_sentences(content)
let n: Int = native_list_len(sents)
let i: Int = 0
while i < n {
let s: String = str_trim(native_list_get(sents, i))
// drop a leading markdown heading marker for a clean read-out line
if str_starts_with(s, "# ") { let s = str_trim(str_slice(s, 2, str_len(s))) }
if str_len(s) > 0 { return s }
let i = i + 1
}
return str_trim(content)
}
// strip trailing/leading punctuation from a token.
fn dlg_clean_tok(w: String) -> String {
let s: String = str_trim(w)
let s = str_strip_suffix(s, ".")
let s = str_strip_suffix(s, ",")
let s = str_strip_suffix(s, "?")
let s = str_strip_suffix(s, "!")
let s = str_strip_suffix(s, ":")
let s = str_strip_suffix(s, ";")
return str_trim(s)
}
// closed-class across the supported languages (union) a word we must NOT treat
// as a retrieval topic. Also drops the meta verbs of a request ("tell", "prove",
// "show") so the TOPIC, not the speech act, is what projects into memory.
fn dlg_is_stop(w: String) -> Bool {
if ml_stop_en(w) { return true }
if ml_stop_es(w) { return true }
if ml_stop_pt(w) { return true }
if ml_stop_it(w) { return true }
if str_eq(w, "tell") { return true }
if str_eq(w, "show") { return true }
if str_eq(w, "about") { return true }
if str_eq(w, "sobre") { return true }
if str_eq(w, "acerca") { return true }
return false
}
// The CONTENT TERMS the query projects into memory: content words only, cleaned,
// cross-lingually mapped to the engram's English vocabulary, 3 chars. This is
// the geometry probe the speech-act verbs and function words are stripped so a
// PP topic ("tell me ABOUT Lisbon") projects on "lisbon", not "tell"/"me".
fn dlg_content_terms(content: String, lang: String) -> [String] {
let toks: [String] = cp_tokenize(content)
let n: Int = native_list_len(toks)
let out: [String] = native_list_empty()
let i: Int = 0
while i < n {
let w: String = str_to_lower(dlg_clean_tok(native_list_get(toks, i)))
if str_len(w) >= 3 {
if !dlg_is_stop(w) {
let out = native_list_append(out, ml_term(w, lang))
}
}
let i = i + 1
}
return out
}
// Does this landed node lexically overlap the query's content terms? This is the
// RELEVANCE FLOOR: activation always returns the store's most salient nodes, so
// without this a query about nothing would "land" on the self/top node. A node
// that shares no content term with the query is "nowhere close" -> honest absence.
fn dlg_node_matches(node: String, terms: [String]) -> Bool {
let hay: String = str_to_lower(json_get_string(node, "content") + " " + json_get_string(node, "label"))
let n: Int = native_list_len(terms)
let i: Int = 0
while i < n {
let t: String = native_list_get(terms, i)
if str_len(t) >= 3 {
if str_contains(hay, t) { return true }
}
let i = i + 1
}
return false
}
// MATERIALIZE the landed region: read out the landed fact, then WALK the
// neighborhood and read out its connected members (real edges, not top-props).
fn dlg_materialize(top_node: String, reply_lang: String) -> String {
let id: String = json_get_string(top_node, "id")
let content: String = json_get_string(top_node, "content")
let lead: String = dlg_first_sentence(content)
let nb: String = engram_neighbors_json(id, 2, "both")
let m: Int = json_array_len(nb)
let parts: [String] = native_list_empty()
let parts = native_list_append(parts, lead)
let added: Int = 0
let i: Int = 0
while i < m {
if added < 3 {
let rec: String = json_array_get(nb, i)
let node: String = json_get_raw(rec, "node")
let nc: String = json_get_string(node, "content")
if !str_eq(nc, "") {
let sent: String = dlg_first_sentence(nc)
if !str_eq(sent, "") {
let parts = native_list_append(parts, sent)
let added = added + 1
}
}
}
let i = i + 1
}
// The readout is the region's OWN prose, verbatim negation SACRED, no echo.
return str_join(parts, " ")
}
// THE single operation
fn dlg_respond(text: String) -> String {
// directive override: reply language may differ from content language.
let dir: [String] = dlg_parse_directive(text)
let target_lang: String = native_list_get(dir, 0)
let content: String = native_list_get(dir, 1)
let content_lang: String = ml_detect(content)
let reply_lang: String = content_lang
if !str_eq(target_lang, "") { let reply_lang = target_lang }
// comprehend the content (SACRED polarity carried in the spec).
let spec: [String] = parse_spec_lang(content, content_lang)
// PROJECT + LAND: SELF region
// Identity/presence shape lands in the self region; read out the REAL self
// nodes (self_region.el), never a template. Same single operation this is
// just the self attractor winning the landing.
if dlg_is_identity(content) {
if sr_available() {
// read out the REAL self nodes when replying in their own language
// (the soul's prose is English); for another reply language we cannot
// translate real content without an LLM, so we answer with the
// localized SACRED identity anchor honest, in-language, no fabrication.
if str_eq(reply_lang, "en") { return sr_readout("en") }
return ml_tr("identity", reply_lang)
}
// self region thin honest localized identity (logged fallback shape).
return ml_tr("identity", reply_lang)
}
// PROJECT into MEMORY geometry
let terms: [String] = dlg_content_terms(content, content_lang)
let qterm: String = str_join(terms, " ")
let act: String = engram_activate_json(qterm, 12)
let n: Int = json_array_len(act)
// LAND: the highest-activation node that ACTUALLY overlaps the query's
// content terms (the relevance floor). Activation always returns the most
// salient nodes, so we walk the ranked list and take the first that is
// genuinely "close"; if none is, the query landed nowhere. ───────────────
let landing: String = ""
let i: Int = 0
while i < n {
if str_eq(landing, "") {
let rec: String = json_array_get(act, i)
let node: String = json_get_raw(rec, "node")
if dlg_node_matches(node, terms) {
let landing = node
}
}
let i = i + 1
}
// HONEST ABSENCE: nothing close an empty region, not a fabricated answer,
// not an "I noted that" echo.
if str_eq(landing, "") {
return ml_tr("no_memory", reply_lang)
}
// MATERIALIZE the landing by WALKING its neighborhood.
return dlg_materialize(landing, reply_lang)
}
+13
View File
@@ -63,6 +63,9 @@ import "morphology-cop.el"
import "grammar.el"
import "realizer.el"
import "semantics.el"
// Comprehension front-end (input half: text meaning-spec)
import "comprehend.el"
//
// Entry points:
//
@@ -117,6 +120,9 @@ fn build_form_from_json(semantic_form_json: String, lang_code: String) -> [Strin
let location: String = sem_get(semantic_form_json, "location")
let tense: String = sem_get(semantic_form_json, "tense")
let aspect: String = sem_get(semantic_form_json, "aspect")
let polarity: String = sem_get(semantic_form_json, "polarity")
let neg_word: String = sem_get(semantic_form_json, "neg_word")
let iobj: String = sem_get(semantic_form_json, "iobj")
let form: [String] = native_list_empty()
let form = native_list_append(form, "intent")
@@ -127,12 +133,19 @@ fn build_form_from_json(semantic_form_json: String, lang_code: String) -> [Strin
let form = native_list_append(form, predicate)
let form = native_list_append(form, "patient")
let form = native_list_append(form, patient)
let form = native_list_append(form, "iobj")
let form = native_list_append(form, iobj)
let form = native_list_append(form, "location")
let form = native_list_append(form, location)
let form = native_list_append(form, "tense")
let form = native_list_append(form, tense)
let form = native_list_append(form, "aspect")
let form = native_list_append(form, aspect)
// SACRED: polarity crosses the JSON boundary and is never inferred away.
let form = native_list_append(form, "polarity")
let form = native_list_append(form, polarity)
let form = native_list_append(form, "neg_word")
let form = native_list_append(form, neg_word)
let form = native_list_append(form, "lang")
let form = native_list_append(form, lang_code)
+65
View File
@@ -0,0 +1,65 @@
// image-demo.el - Drive the native PNG surface: plan a scene from a small
// meaning phrase (incl. a NEG frame) and emit a byte-valid 64x64 PNG whose
// palette is read from elp/faculty/sig/scene.basis.
fn img_frame(relation: String, polarity: String, confidence: String, importance: String, salience: String, subj_id: String) -> [String] {
let f: [String] = native_list_empty()
let f: [String] = native_list_append(f, "relation")
let f: [String] = native_list_append(f, relation)
let f: [String] = native_list_append(f, "polarity")
let f: [String] = native_list_append(f, polarity)
let f: [String] = native_list_append(f, "confidence")
let f: [String] = native_list_append(f, confidence)
let f: [String] = native_list_append(f, "importance")
let f: [String] = native_list_append(f, importance)
let f: [String] = native_list_append(f, "salience")
let f: [String] = native_list_append(f, salience)
let f: [String] = native_list_append(f, "subj_id")
let f: [String] = native_list_append(f, subj_id)
return f
}
fn rgb_str(c: [Int]) -> String {
return int_to_str(native_list_get(c, 0)) + "," + int_to_str(native_list_get(c, 1)) + "," + int_to_str(native_list_get(c, 2))
}
fn run_image() -> Int {
fs_mkdir("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out")
let table: [Int] = crc_table()
println("crc_table[1]=" + int_to_str(native_list_get(table, 1)) + " (expect 1996959894 / 0x77073096)")
let basis: [String] = basis_load("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/scene.basis")
let warm: [Int] = parse_rgb(basis_field(basis, "warm"))
let cool: [Int] = parse_rgb(basis_field(basis, "cool"))
let bg: [Int] = parse_rgb(basis_field(basis, "bg"))
println("basis warm=" + rgb_str(warm) + " cool=" + rgb_str(cool) + " bg=" + rgb_str(bg) + " (read from scene.basis)")
let frames: [[String]] = native_list_empty()
let frames: [[String]] = native_list_append(frames, img_frame("agent", "aff", "0.9", "0.8", "0", "s1"))
let frames: [[String]] = native_list_append(frames, img_frame("theme", "aff", "0.7", "0.6", "1", "s2"))
let frames: [[String]] = native_list_append(frames, img_frame("cause", "aff", "0.8", "0.9", "0", "s3"))
let frames: [[String]] = native_list_append(frames, img_frame("negation", "neg", "0.85", "0.7", "1", "s4"))
let frames: [[String]] = native_list_append(frames, img_frame("goal", "aff", "0.6", "0.5", "0", "s5"))
let frames: [[String]] = native_list_append(frames, img_frame("result", "aff", "0.95", "1.0", "1", "s6"))
let shapes: [[Int]] = plan_scene(frames, warm, cool)
let ns: Int = native_list_len(shapes)
println("planned " + int_to_str(ns) + " shapes:")
let si: Int = 0
while si < ns {
let sh: [Int] = native_list_get(shapes, si)
let pol: String = surface_get(native_list_get(frames, si), "polarity")
println(" shape " + int_to_str(si) + " type=" + int_to_str(native_list_get(sh, 0)) + " x=" + int_to_str(native_list_get(sh, 1)) + " y=" + int_to_str(native_list_get(sh, 2)) + " size=" + int_to_str(native_list_get(sh, 3)) + " rgb=" + int_to_str(native_list_get(sh, 4)) + "," + int_to_str(native_list_get(sh, 5)) + "," + int_to_str(native_list_get(sh, 6)) + " polarity=" + pol)
let si: Int = si + 1
}
let raw: [Int] = rasterize(64, 64, shapes, bg)
println("rasterized raw (filtered scanlines) bytes=" + int_to_str(native_list_len(raw)) + " (expect 12352)")
let png: [Int] = png_build(64, 64, raw, table)
let plen: Int = native_list_len(png)
let ok: Int = png_write("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/scene.png", png)
println("PNG bytes=" + int_to_str(plen) + " -> /Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/scene.png (write_ok=" + int_to_str(ok) + ")")
return plen
}
println("image-demo returned png_bytes=" + int_to_str(run_image()))
+412
View File
@@ -0,0 +1,412 @@
// image-surface.el - Native own-core raster PNG surface (the image efferent
// twin of audio). Renders a 64x64 RGB scene deterministically from a frame's
// meaning-geometry, then serialises a byte-valid PNG entirely own-core:
// 8-byte magic, IHDR, IDAT (zlib STORED/uncompressed DEFLATE + Adler32), IEND,
// with a per-chunk CRC32 computed via software xor32 (EL has no bitwise ops).
//
// The RGB palette basis is read from elp/faculty/sig/scene.basis (data, not
// literals) - the same read-from-learned discipline as the audio signatures.
// Integer-only throughout; pixels are composed functionally (painter's order)
// so no list mutation is needed.
// -- small int/parse helpers (self-contained) ----------------------------------
fn i_str_to_int(s: String) -> Int {
let n: Int = str_len(s)
let i: Int = 0
let v: Int = 0
while i < n {
let c: Int = str_char_code(s, i)
if c >= 48 {
if c < 58 {
let v: Int = v * 10 + (c - 48)
}
}
let i: Int = i + 1
}
return v
}
fn basis_load(path: String) -> [String] {
return str_split(fs_read(path), "\n")
}
fn basis_field(lines: [String], key: String) -> String {
let pref: String = key + ": "
let n: Int = native_list_len(lines)
let plen: Int = str_len(pref)
let i: Int = 0
while i < n {
let ln: String = native_list_get(lines, i)
if str_starts_with(ln, pref) {
return str_slice(ln, plen, str_len(ln))
}
let i: Int = i + 1
}
return ""
}
fn parse_rgb(csv: String) -> [Int] {
let parts: [String] = str_split(csv, ",")
let out: [Int] = native_list_empty()
let n: Int = native_list_len(parts)
let i: Int = 0
while i < n {
let v: Int = i_str_to_int(native_list_get(parts, i))
let out: [Int] = native_list_append(out, v)
let i: Int = i + 1
}
return out
}
// -- software 32-bit XOR (no bitwise ops in EL) --------------------------------
fn xor32(a: Int, b: Int) -> Int {
let r: Int = 0
let bit: Int = 1
let i: Int = 0
while i < 32 {
let abit: Int = (a / bit) % 2
let bbit: Int = (b / bit) % 2
if abit != bbit {
let add: Int = bit
let r: Int = r + add
}
let bit: Int = bit * 2
let i: Int = i + 1
}
return r
}
// -- CRC32 (table-driven, table built with xor32) ------------------------------
fn crc_table() -> [Int] {
let t: [Int] = native_list_empty()
let n: Int = 0
while n < 256 {
let c: Int = n
let k: Int = 0
while k < 8 {
if c % 2 == 1 {
let h: Int = c / 2
let c: Int = xor32(h, 3988292384)
} else {
let c: Int = c / 2
}
let k: Int = k + 1
}
let t: [Int] = native_list_append(t, c)
let n: Int = n + 1
}
return t
}
fn crc32_of(bytes: [Int], table: [Int]) -> Int {
let crc: Int = 4294967295
let n: Int = native_list_len(bytes)
let i: Int = 0
while i < n {
let b: Int = native_list_get(bytes, i)
let lo: Int = crc % 256
let idx: Int = xor32(lo, b) % 256
let tv: Int = native_list_get(table, idx)
let hi: Int = crc / 256
let crc: Int = xor32(hi, tv)
let i: Int = i + 1
}
return xor32(crc, 4294967295)
}
// -- Adler32 (for the zlib trailer) --------------------------------------------
fn adler32_of(bytes: [Int]) -> Int {
let a: Int = 1
let b: Int = 0
let n: Int = native_list_len(bytes)
let i: Int = 0
while i < n {
let byte: Int = native_list_get(bytes, i)
let a: Int = (a + byte) % 65521
let b: Int = (b + a) % 65521
let i: Int = i + 1
}
return b * 65536 + a
}
// -- byte-list append helpers --------------------------------------------------
fn app_u32be(dst: [Int], v: Int) -> [Int] {
let dst: [Int] = native_list_append(dst, (v / 16777216) % 256)
let dst: [Int] = native_list_append(dst, (v / 65536) % 256)
let dst: [Int] = native_list_append(dst, (v / 256) % 256)
let dst: [Int] = native_list_append(dst, v % 256)
return dst
}
fn app_tag(dst: [Int], s: String) -> [Int] {
let n: Int = str_len(s)
let i: Int = 0
while i < n {
let dst: [Int] = native_list_append(dst, str_char_code(s, i))
let i: Int = i + 1
}
return dst
}
fn app_all(dst: [Int], src: [Int]) -> [Int] {
let n: Int = native_list_len(src)
let i: Int = 0
while i < n {
let dst: [Int] = native_list_append(dst, native_list_get(src, i))
let i: Int = i + 1
}
return dst
}
// -- plan: frame meaning-geometry -> shape atoms -------------------------------
// shape = [type, x, y, size, r, g, b] (type 0=rect 1=disc 2=triangle)
fn charsum(s: String) -> Int {
let n: Int = str_len(s)
let i: Int = 0
let acc: Int = 0
while i < n {
let c: Int = str_char_code(s, i)
let acc: Int = acc + c
let i: Int = i + 1
}
return acc
}
fn micro_of(s: String) -> Int {
let dot: Int = str_index_of(s, ".")
if dot < 0 { return i_str_to_int(s) * 1000000 }
let n: Int = str_len(s)
let fp: String = str_slice(s, dot + 1, n)
let ip: String = str_slice(s, 0, dot)
let iv: Int = i_str_to_int(ip)
let fv: Int = 0
let scale: Int = 100000
let fl: Int = str_len(fp)
let i: Int = 0
while i < 6 {
let d: Int = 0
if i < fl { let d: Int = str_char_code(fp, i) - 48 }
let fv: Int = fv + d * scale
let scale: Int = scale / 10
let i: Int = i + 1
}
return iv * 1000000 + fv
}
fn plan_scene(frames: [[String]], warm: [Int], cool: [Int]) -> [[Int]] {
let shapes: [[Int]] = native_list_empty()
let nf: Int = native_list_len(frames)
let fi: Int = 0
while fi < nf {
let fr: [String] = native_list_get(frames, fi)
let relation: String = surface_get(fr, "relation")
let polarity: String = surface_get(fr, "polarity")
let confidence: String = surface_get(fr, "confidence")
let importance: String = surface_get(fr, "importance")
let salience: String = surface_get(fr, "salience")
// relation -> shape type
let stype: Int = charsum(relation) % 3
// confidence -> size (8..22)
let cmi: Int = micro_of(confidence)
let size: Int = 8 + cmi / 71428
// salience -> y
let sal: Int = i_str_to_int(salience)
let y: Int = 6 + sal * 26
// subj_id/index -> x
let x: Int = 4 + (fi * 10) % 48
// polarity -> warm/cool base color
let br: Int = native_list_get(warm, 0)
let bg2: Int = native_list_get(warm, 1)
let bb: Int = native_list_get(warm, 2)
if str_eq(polarity, "neg") {
let br: Int = native_list_get(cool, 0)
let bg2: Int = native_list_get(cool, 1)
let bb: Int = native_list_get(cool, 2)
}
// importance -> brightness (500..1000 permille)
let imi: Int = micro_of(importance)
let bpm: Int = 500 + imi / 2000
let r: Int = br * bpm / 1000
let g: Int = bg2 * bpm / 1000
let b: Int = bb * bpm / 1000
let sh: [Int] = native_list_empty()
let sh: [Int] = native_list_append(sh, stype)
let sh: [Int] = native_list_append(sh, x)
let sh: [Int] = native_list_append(sh, y)
let sh: [Int] = native_list_append(sh, size)
let sh: [Int] = native_list_append(sh, r)
let sh: [Int] = native_list_append(sh, g)
let sh: [Int] = native_list_append(sh, b)
let shapes: [[Int]] = native_list_append(shapes, sh)
let fi: Int = fi + 1
}
return shapes
}
// covers: is (px,py) inside this shape?
fn covers(sh: [Int], px: Int, py: Int) -> Bool {
let stype: Int = native_list_get(sh, 0)
let sx: Int = native_list_get(sh, 1)
let sy: Int = native_list_get(sh, 2)
let size: Int = native_list_get(sh, 3)
let cx: Int = sx + size / 2
if stype == 0 {
if px >= sx {
if px < sx + size {
if py >= sy {
if py < sy + size {
return true
}
}
}
}
return false
}
if stype == 1 {
let rad: Int = size / 2
let dx: Int = px - cx
let dy: Int = py - (sy + rad)
if dx * dx + dy * dy <= rad * rad {
return true
}
return false
}
// triangle: apex at top (sy), base at sy+size
if py >= sy {
if py < sy + size {
let dyv: Int = py - sy
let halfw: Int = dyv / 2
let dxv: Int = px - cx
let adx: Int = dxv
if adx < 0 { let adx: Int = 0 - dxv }
if adx <= halfw {
return true
}
}
}
return false
}
// pixel_color: painter's algorithm - last covering shape wins. Returns [r,g,b].
fn pixel_color(px: Int, py: Int, shapes: [[Int]], bg: [Int]) -> [Int] {
let r: Int = native_list_get(bg, 0)
let g: Int = native_list_get(bg, 1)
let b: Int = native_list_get(bg, 2)
let n: Int = native_list_len(shapes)
let i: Int = 0
while i < n {
let sh: [Int] = native_list_get(shapes, i)
if covers(sh, px, py) {
let r: Int = native_list_get(sh, 4)
let g: Int = native_list_get(sh, 5)
let b: Int = native_list_get(sh, 6)
}
let i: Int = i + 1
}
let out: [Int] = native_list_empty()
let out: [Int] = native_list_append(out, r)
let out: [Int] = native_list_append(out, g)
let out: [Int] = native_list_append(out, b)
return out
}
// rasterize: build the raw (filtered) scanline byte stream, filter byte 0 / row.
fn rasterize(w: Int, h: Int, shapes: [[Int]], bg: [Int]) -> [Int] {
let raw: [Int] = native_list_empty()
let y: Int = 0
while y < h {
let raw: [Int] = native_list_append(raw, 0)
let x: Int = 0
while x < w {
let col: [Int] = pixel_color(x, y, shapes, bg)
let raw: [Int] = native_list_append(raw, native_list_get(col, 0))
let raw: [Int] = native_list_append(raw, native_list_get(col, 1))
let raw: [Int] = native_list_append(raw, native_list_get(col, 2))
let x: Int = x + 1
}
let y: Int = y + 1
}
return raw
}
// zlib stream with a single STORED (uncompressed) DEFLATE block + Adler32.
fn zlib_store(raw: [Int]) -> [Int] {
let z: [Int] = native_list_empty()
let z: [Int] = native_list_append(z, 120)
let z: [Int] = native_list_append(z, 1)
let z: [Int] = native_list_append(z, 1)
let len: Int = native_list_len(raw)
let nlen: Int = 65535 - len
let z: [Int] = native_list_append(z, len % 256)
let z: [Int] = native_list_append(z, (len / 256) % 256)
let z: [Int] = native_list_append(z, nlen % 256)
let z: [Int] = native_list_append(z, (nlen / 256) % 256)
let z: [Int] = app_all(z, raw)
let ad: Int = adler32_of(raw)
let z: [Int] = app_u32be(z, ad)
return z
}
// append a full PNG chunk: length + (type+data) + crc32(type+data).
fn app_chunk(png: [Int], type_and_data: [Int], table: [Int]) -> [Int] {
let total: Int = native_list_len(type_and_data)
let dlen: Int = total - 4
let png: [Int] = app_u32be(png, dlen)
let png: [Int] = app_all(png, type_and_data)
let crc: Int = crc32_of(type_and_data, table)
let png: [Int] = app_u32be(png, crc)
return png
}
fn png_build(w: Int, h: Int, raw: [Int], table: [Int]) -> [Int] {
let png: [Int] = native_list_empty()
// 8-byte signature
let png: [Int] = native_list_append(png, 137)
let png: [Int] = native_list_append(png, 80)
let png: [Int] = native_list_append(png, 78)
let png: [Int] = native_list_append(png, 71)
let png: [Int] = native_list_append(png, 13)
let png: [Int] = native_list_append(png, 10)
let png: [Int] = native_list_append(png, 26)
let png: [Int] = native_list_append(png, 10)
// IHDR
let ihdr: [Int] = native_list_empty()
let ihdr: [Int] = app_tag(ihdr, "IHDR")
let ihdr: [Int] = app_u32be(ihdr, w)
let ihdr: [Int] = app_u32be(ihdr, h)
let ihdr: [Int] = native_list_append(ihdr, 8)
let ihdr: [Int] = native_list_append(ihdr, 2)
let ihdr: [Int] = native_list_append(ihdr, 0)
let ihdr: [Int] = native_list_append(ihdr, 0)
let ihdr: [Int] = native_list_append(ihdr, 0)
let png: [Int] = app_chunk(png, ihdr, table)
// IDAT
let z: [Int] = zlib_store(raw)
let idat: [Int] = native_list_empty()
let idat: [Int] = app_tag(idat, "IDAT")
let idat: [Int] = app_all(idat, z)
let png: [Int] = app_chunk(png, idat, table)
// IEND
let iend: [Int] = native_list_empty()
let iend: [Int] = app_tag(iend, "IEND")
let png: [Int] = app_chunk(png, iend, table)
return png
}
fn png_write(path: String, png: [Int]) -> Int {
let n: Int = native_list_len(png)
let buf: String = __str_alloc(n)
let i: Int = 0
while i < n {
let buf: String = __str_set_char(buf, i, native_list_get(png, i))
let i: Int = i + 1
}
let ok: Int = fs_write_bytes(path, buf, n)
return ok
}
+72
View File
@@ -0,0 +1,72 @@
;;; lang_profile_ca.el — Catalan language profile for ELP.
;;; Mirrors lang_profile_it / _es / _pt; keys the realizer's construction switches.
;;; Catalan is the CLOSEST Romance sibling to the shared engine (~85% conceptual
;;; reuse). The deltas: PRONOMS FEBLES with four position allomorphs, l'-elision,
;;; del/al/pel contractions, the periphrastic preterite (vaig+INF), and NO
;;; essere/avere split (perfect aux is always HAVER; ser/estar is only the copula).
(lang_profile_ca
(language "Catalan")
(iso639 "ca")
(family "Romance")
;; ── core typology flags ────────────────────────────────────────────────
(pro-drop yes) ; null subjects default; overt pronoun = emphatic
(obligatory-subject no)
(grammatical-gender yes) ; m/f; full NP agreement (art + adj + participle)
(do-support no)
(subject-aux-inversion no) ; yes/no Q = declarative order + '?'; no inversion
(article-selection "el/la/l'/els/les ; un/una/uns/unes") ; l'-ELISION:
; el/la -> l' before vowel or (silent) h, glued to
; the next word (l'home, l'illa); de -> d' before vowel
(article-drives-contraction yes) ; article choice feeds prep+article contraction
(adjective-position "postnominal-default + small prenominal class") ; bo/bon,
; mal, gran, nou, vell, primer, molt... prenominal
(question-punct plain) ; ? and ! only (no inverted ¿ ¡)
;; ── MANDATORY prep+article contractions ────────────────────────────────
(contractions ((de el del) (de els dels)
(a el al) (a els als)
(per el pel) (per els pels)))
(contraction-mandatory yes) ; *de el -> del obligatory
(contraction-blocked-before-elision yes) ; de l'home / a l'home (NO *del home)
;; ── clitic system: PRONOMS FEBLES (the headline delta) ──────────────────
(clitics yes)
(clitic-allomorphy four-position) ; per pronoun, form varies by position+onset:
; reinforced (em, et, el) proclitic before a consonant
; elided (m', t', l', n') proclitic before a vowel/h
; full (-me, -lo, -li) enclitic after a consonant/-r
; reduced ('m, 't, 'l, 'ns) enclitic after a vowel
(clitic-placement ((finite proclitic) ; el veig, no m'ho dóna
(imperative-affirmative enclitic) ; dóna'm, digues-me
(imperative-negative present-subjunctive) ; no parlis (delta)
(infinitive enclitic) ; ajudar-me, veure'l
(gerund enclitic))) ; fent-ho
(clitic-combination ((me el "me'l") (te el "te'l") (se el "se'l")
(me la "me la") (me en "me'n")
(li el "l'hi") (li en "n'hi"))) ; dative+accusative clusters
(clitic-particles (hi en ho)) ; locative hi, partitive/genitive en, neuter ho
;; ── verb / aspect system ───────────────────────────────────────────────
(finite-agreement "person+number (6-way)")
(tenses (present imperfet preterit-simple perifrastic-preterit futur
condicional subjuntiu-present subjuntiu-imperfet imperatiu))
(periphrastic-preterite "vaig/vas/va/vam/vau/van + INFINITIVE") ; << hallmark CA
; (vaig cantar = 'I sang'); coexists w/ synthetic pret.
(compound-past "pretèrit perfet = haver(present) + participle")
(perfect-aux "HAVER only") ; << NO essere/avere split (simpler than IT)
(participle-agreement ((haver preceding-acc-clitic))) ; les he vistes; else invariable
(progressive-aux "estar + gerundi")
(copula "ser / estar") ; ser: identity/essential/origin; estar:
; location + transient state (estic cansat, és a casa)
(passive-aux "ser (+ per-agent)")
(future inflectional) ; cantaré, serà
(comparative "més/menys ADJ que")
;; ── SACRED safety bar (shared with es/pt/it/en) ────────────────────────
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
(negation "no (preverbal) + optional 'pas' + concord") ; no...res/
; ningú/mai/cap/gens/enlloc
(negative-concord yes) ; preverbal negative subject (ningú) keeps 'no'
(neg-reinforcer pas)) ; optional (no ho faré pas)
+41
View File
@@ -0,0 +1,41 @@
;;; lang_profile_de.el — German language profile for ELP.
;;; Mirrors lang_profile_en / lang_profile_es. Keys the realizer's construction
;;; switches. German is the largest Germanic delta from the EN engine: V2 word
;;; order, four morphological cases, and separable-prefix verbs.
(lang_profile_de
(language "German")
(iso639 "de")
(family "Germanic")
(neighbor-base "en") ; realized by extending the English (Germanic) engine
;; ── core typology flags ────────────────────────────────────────────────
(pro-drop no) ; obligatory subject in finite clauses
(obligatory-subject yes)
(grammatical-gender (m f n)) ; three genders; drives article + adj declension
(case-system (nom acc dat gen)) ; four cases on articles/adjs/nouns
(word-order V2) ; finite verb 2nd in main clause
(subordinate-order verb-final) ; "..., dass er den Hund SIEHT."
(separable-verbs yes) ; aufstehen -> "steht ... auf"; ppart "aufgestanden"
(do-support no) ; German negates/questions the finite verb directly
(subject-verb-inversion yes) ; yes/no Q fronts finite verb; wh-Q fills Vorfeld
(article-selection "der/die/das + ein/kein") ; declined by case x gender x number
(adjective-position prenominal)
(adjective-declension (strong weak mixed)) ; chosen by the determiner type
(noun-capitalization yes)
;; ── verb / aspect system ───────────────────────────────────────────────
(finite-agreement "person-and-number") ; full present/past paradigm
(auxiliary-order (modal tense-aux perfect passive main))
(perfect-aux (haben sein)) ; sein for intransitive motion/change verbs
(passive-aux "werden")
(future "werden + infinitive")
(comparative "synthetic (-er / -st, with umlaut)")
;; ── negation ───────────────────────────────────────────────────────────
(negation-markers (nicht kein)) ; kein- negates an indefinite NP; nicht else
(negation-faithful yes) ; SACRED: polarity never dropped/inverted -> FLAG
;; ── lexicon provenance ─────────────────────────────────────────────────
(lexicon-source "UniMorph deu (primary) + kaikki.org German (gender override)")
(lexicon-license "CC-BY-SA 3.0 / GFDL"))
+41
View File
@@ -0,0 +1,41 @@
;;; lang_profile_en.el — English language profile for ELP.
;;; Mirrors lang_profile_es / lang_profile_pt; keys the realizer's construction
;;; switches. English is typologically distinct from the Romance builds, so the
;;; flags differ where the grammar differs.
(lang_profile_en
(language "English")
(iso639 "en")
(family "Germanic")
;; ── core typology flags ────────────────────────────────────────────────
(pro-drop no) ; OBLIGATORY subjects — missing subject is FLAGGED
(obligatory-subject yes)
(grammatical-gender no) ; natural gender only (he/she/it), no NP agreement
(do-support yes) ; negation & questions of lexical verbs insert do/does/did
(subject-aux-inversion yes) ; yes/no + non-subject wh questions invert the operator
(article-selection "a/an/the") ; a/an resolved PHONOLOGICALLY (an hour, a university)
(adjective-position prenominal) ; attributive adjectives precede the noun; invariant
(has-tag-questions yes) ; "...doesn't he?" — operator + reversed polarity
(has-there-existential yes) ; "there is/are/have been ..."
(possessive-clitic "'s") ; saxon genitive; plural in -s -> bare apostrophe
(question-punct plain) ; ? and ! only (no inverted marks)
;; ── verb / aspect system ───────────────────────────────────────────────
(finite-agreement "3sg-present-only") ; only 3sg present -s (+ suppletive be)
(auxiliary-order (modal perfect progressive passive main))
(perfect-aux "have") ; have + past participle
(progressive-aux "be") ; be + present participle
(passive-aux "be") ; be + past participle (+ by-agent)
(future "will + base") ; no inflectional future
(comparative "synthetic-or-periphrastic") ; -er/-est vs more/most by syllables
;; ── SACRED safety bar (shared with es/pt) ──────────────────────────────
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
;; ── DIALECT overlay (post-realization, one core -> US/UK/AU) ────────────
(dialect US) ; default; profile field switches the overlay
(dialects (US UK AU))
(dialect-canonical US) ; core is authored in US orthography
(dialect-overlay "dialect_en.to_dialect") ; orthography + lexis + grammar prefs
(dialect-covers (spelling lexis collective-agreement gotten/got)))
+45
View File
@@ -0,0 +1,45 @@
;;; lang_profile_es.el — Spanish language profile for ELP.
;;; Keys the realizer's construction switches. Mirrors lang_profile_en / _pt.
(lang_profile_es
(language "Spanish")
(iso639 "es")
(family "Romance")
;; -- core typology flags -------------------------------------------------
(pro-drop yes) ; subjects routinely dropped; agreement carries person
(obligatory-subject no)
(grammatical-gender yes) ; m/f on every noun; article+adjective AGREE
(gender-source lexicon); REAL per-noun gender from UniMorph — NOT a heuristic
(do-support no)
(subject-aux-inversion no) ; questions by intonation/punctuation, not inversion
(question-strategy intonation)
(article-selection "el/la/los/las un/una/unos/unas")
(stressed-a-rule yes) ; fem sg noun in stressed a-/ha- takes el/un (el agua)
(adjective-position postnominal) ; default post; a few prenominal + apocope
(adjective-agreement "gender+number")
(question-punct inverted) ; opening ¿ ¡ required
;; -- MANDATORY CONTRACTIONS (coordinator quality bar) --------------------
(contractions ((de el "del") (a el "al")))
(contraction-mandatory yes) ; 'de el'/'a el' MUST surface as del/al
;; -- verb / aspect system ------------------------------------------------
(verb-classes (ar er ir))
(tenses (present preterite imperfect future conditional))
(moods (ind sbjv imp))
(finite-agreement "person+number (6 slots)")
(perfect-aux "haber") ; haber + past participle (invariant -o)
(progressive-aux "estar") ; estar + gerund
(passive-aux "ser") ; ser + participle (agrees) + por-agent
(copula-split "ser/estar") ; permanent vs stage-level
(future "infinitive + é/ás/á/emos/éis/án")
;; -- clitics / government ------------------------------------------------
(object-clitics yes) ; me te lo la le nos os los las; proclisis/enclisis
(clitic-order "se II I III (le+lo -> se lo)")
(enclisis "imperative/infinitive/gerund + accent repair (dá+me+lo->dámelo)")
(verb-prep-government yes) ; verbs select prep (protestar+contra, escapar+de)
;; -- SACRED safety bar (shared with en/pt) -------------------------------
(negation-faithful yes)) ; polarity never dropped/inverted; unplaceable -> FLAG
+74
View File
@@ -0,0 +1,74 @@
;;; lang_profile_fr.el — French language profile for ELP.
;;; Mirrors lang_profile_it / lang_profile_es; keys the realizer's construction
;;; switches. French is a Romance sibling (~54% of the realizer code and the whole
;;; clause-engine architecture reused), but carries the family's biggest surface
;;; deltas: NOT pro-drop, DISCONTINUOUS negation, and an orthography/phonology
;;; mismatch (elision, liaison) that makes exact-match genuinely hard.
(lang_profile_fr
(language "French")
(iso639 "fr")
(family "Romance")
;; ── core typology flags ────────────────────────────────────────────────
(pro-drop no) ; << French-specific: subject clitic OBLIGATORY
(obligatory-subject yes) ; je/tu/il/elle/nous/vous/ils/elles always overt
(grammatical-gender yes) ; m/f; full NP agreement (art + adj + participle)
(do-support no)
(subject-aux-inversion optional) ; est-ce que (default) OR clitic inversion (vas-tu)
(article-selection "le/la/l'/les ; un/une/des ; PARTITIVE du/de la/de l'/des")
(article-drives-contraction yes) ; à+le=au, de+le=du feed off article choice
(adjective-position "postnominal-default + prenominal-BAGS") ; beau/bon/grand/
; petit/jeune/vieux/nouveau + ordinals prenominal
; (beau->bel, nouveau->nouvel, vieux->vieil / vowel)
(question-punct "space-before") ; French typography: ' ?' ' !' (no ¿¡)
;; ── elision (orthography/phonology mismatch — French-specific) ──────────
(elision ((le l') (la l') (je j') (ne n') (de d') (que qu')
(me m') (te t') (se s') (ce c'))) ; before vowel / h-muet
(elision-h-muet yes) ; l'homme, l'hôpital (h-aspiré exception list kept)
(liaison noted-not-modeled) ; phonological, not written in surface
;; ── MANDATORY prep+article contractions ────────────────────────────────
(contractions ((à le au) (à les aux) (de le du) (de les des)))
(contraction-mandatory yes) ; *à le -> au obligatory; à la / à l' uncontracted
(partitive ((m-sg du) (f-sg "de la") (vowel "de l'") (pl des)))
(partitive-under-neg "de") ; << gap in current build: 'ne … pas de pain'
;; ── clitic system ──────────────────────────────────────────────────────
(clitics yes)
(clitic-order (me te se nous vous | le la les | lui leur | y | en))
(clitic-placement ((finite proclitic) ; je le lui donne
(imperative-affirmative enclitic-hyphen) ; donne-le-moi
(imperative-negative "ne+proclitic+verb+pas") ; ne le donne pas
(infinitive enclitic))) ; PARTIAL: clitic-climbing
; onto infinitive under modal
(clitic-imperative-shift ((me moi) (te toi))) ; final me/te -> moi/toi (donne-moi)
(clitic-particles (y en)) ; locative y, partitive/genitive en
;; ── verb / aspect system ───────────────────────────────────────────────
(finite-agreement "person+number (written; many homophones)")
(tenses (présent imparfait passé-simple futur conditionnel
subjonctif-présent subjonctif-imparfait impératif))
(compound-past "passé-composé = aux(present) + participe passé")
(perfect-aux "être/avoir (LEXICAL selection)") ; << French-specific
(etre-aux-class "intransitive motion/change (aller venir arriver partir
entrer sortir monter descendre naître mourir rester
tomber retourner passer devenir revenir rentrer) + ALL
pronominal verbs")
(participle-agreement ((être subject) ; elle est allée / elles venues
(avoir preceding-direct-object))) ; je les ai vus
(progressive "être en train de + infinitif") ; no dedicated aux
(copula "être (single; no ser/estar, no essere/stare)")
(passive-aux "être (+ par-agent)")
(future inflectional) ; parlera, sera
(comparative "plus/moins ADJ que")
(superlative "le/la plus ADJ (de …)") ; PARTIAL word-order in build
;; ── SACRED safety bar (shared with es/pt/it/en) ────────────────────────
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
(negation "DISCONTINUOUS: ne (preverbal) … pas/jamais/rien/personne/
plus/guère/que (postverbal)") ; << biggest structural delta
(negation-ne-elides yes) ; ne -> n' before vowel (n'ai pas vu)
(negation-passe-composé "ne + aux + pas + participe") ; n'ai pas vu
(negative-concord partial)) ; personne/rien as arguments post-participle
+70
View File
@@ -0,0 +1,70 @@
;;; lang_profile_it.el — Italian language profile for ELP.
;;; Mirrors lang_profile_es / lang_profile_pt; keys the realizer's construction
;;; switches. Italian is a Romance sibling, so ~85% of the flags match ES/PT; the
;;; essere/avere auxiliary split and phonological article selection are the deltas.
(lang_profile_it
(language "Italian")
(iso639 "it")
(family "Romance")
;; ── core typology flags ────────────────────────────────────────────────
(pro-drop yes) ; null subjects default; overt pronoun = emphatic
(obligatory-subject no)
(grammatical-gender yes) ; m/f; full NP agreement (art + adj + participle)
(do-support no)
(subject-aux-inversion no) ; yes/no Q = declarative order + '?'; no inversion
(article-selection "il/lo/l'/i/gli + la/l'/le ; un/uno/un'/una") ; PHONOLOGICAL:
; lo/gli/uno before s+cons, z, gn, ps, pn, x, y, i+V;
; l'/un' before a vowel (elision, glued to next word)
(article-drives-contraction yes) ; article choice feeds the prep+art contraction
(adjective-position "postnominal-default + prenominal-class") ; bello/buono/grande
; /nuovo/vecchio/primo... prenominal (with apocope)
(question-punct plain) ; ? and ! only (no inverted ¿ ¡)
;; ── MANDATORY prep+article contractions ────────────────────────────────
(contractions ((di il del) (di lo dello) (di la della) (di i dei)
(di gli degli) (di le delle) (di l' dell')
(a il al) (a lo allo) (a la alla) (a i ai) (a gli agli)
(a le alle) (a l' all')
(da il dal) (da la dalla) (da gli dagli) (da l' dall')
(in il nel) (in la nella) (in gli negli) (in l' nell')
(su il sul) (su la sulla) (su gli sugli) (su l' sull')))
(contraction-mandatory yes) ; *di il -> del is obligatory, never uncontracted
(prep-no-contract (per tra fra)) ; per la strada (NOT *perla)
;; ── clitic system ──────────────────────────────────────────────────────
(clitics yes)
(clitic-placement ((finite proclitic) ; lo vedo, non me lo dà
(imperative-affirmative enclitic) ; dammelo, guardalo
(imperative-negative-tu non+infinitive) ; non parlare / non lo fare
(infinitive enclitic) ; vederlo, aiutarmi (drop -e)
(gerund enclitic))) ; dandolo
(clitic-combination ((mi lo "me lo") (ti lo "te lo") (ci lo "ce lo")
(vi lo "ve lo") (si lo "se lo")
(gli lo "glielo") (le lo "glielo"))) ; glielo = ONE word
(clitic-particles (ci ne)) ; locative ci, partitive ne
(raddoppiamento (da fa di va sta)) ; monosyllabic imper double clitic: dammelo
;; ── verb / aspect system ───────────────────────────────────────────────
(finite-agreement "person+number (6-way)")
(tenses (presente imperfetto passato-remoto futuro condizionale
congiuntivo-presente congiuntivo-imperfetto imperativo))
(compound-past "passato-prossimo = aux(present) + participle")
(perfect-aux "essere/avere (LEXICAL selection)") ; << Italian-specific
(essere-aux-class unaccusative) ; motion/change-of-state/copular/pronominal
; (andare venire nascere morire diventare piacere
; + ALL reflexives) -> essere
(participle-agreement ((essere subject) ; è andata / sono arrivati
(avere preceding-acc-clitic))) ; li ho visti
(progressive-aux "stare + gerundio") ; sto parlando
(copula "essere (default) / stare (state: sto bene)")
(passive-aux "essere / venire (+ da-agent)")
(future inflectional) ; parlerò, sarà
(comparative "più/meno ADJ di")
;; ── SACRED safety bar (shared with es/pt/en) ───────────────────────────
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
(negation "non (preverbal) + concord") ; non...niente/nessuno/mai/più
(negative-concord yes) ; preverbal negative word (nessuno/niente) suppresses non
(neg-adverb-position between-aux-and-participle)) ; non ho MAI visto
+30
View File
@@ -0,0 +1,30 @@
;;; lang_profile_la.el — Latin language profile for ELP.
;;; Keys the realizer's construction switches. Companion to morphology-la.el.
(lang_profile_la
(language "Latin")
(iso639 "la")
(family "Italic")
;; -- core typology flags -------------------------------------------------
(pro-drop yes) ; person carried by verb ending; subjects dropped
(obligatory-subject no)
(grammatical-gender yes) ; m/f/n; adjective AGREES in case+gender+number
(gender-source lexicon) ; REAL per-noun gender from UniMorph lat
(articles none) ; Latin has no articles
(case-system yes) ; NOM GEN DAT ACC ABL VOC (+ rare LOC)
(cases (nom gen dat acc abl voc))
(word-order "SOV (default; free order, case-marked)")
(adjective-position "either (case agreement carries the link)")
(adjective-agreement "case+gender+number")
;; -- verb / aspect system ------------------------------------------------
(verb-classes (1 2 3 3io 4)) ; four conjugations + i-stem 3rd
(tenses (present imperfect future perfect pluperfect futureperfect))
(moods (indicative subjunctive imperative infinitive))
(voices (active passive))
(finite-agreement "person+number (6 slots)")
(citation "principal parts: pres-1sg / pres-inf / perf-participle")
;; -- SACRED safety bar ---------------------------------------------------
(negation-faithful yes)) ; polarity never dropped/inverted
+40
View File
@@ -0,0 +1,40 @@
;;; lang_profile_pt.el — Portuguese language profile for ELP.
;;; Keys the realizer's construction switches. Mirrors lang_profile_es.
(lang_profile_pt
(language "Portuguese")
(iso639 "pt")
(family "Romance")
;; -- core typology flags -------------------------------------------------
(pro-drop yes) ; subjects routinely dropped; agreement carries person
(obligatory-subject no)
(grammatical-gender yes) ; m/f on every noun; article+adjective AGREE
(gender-source lexicon) ; REAL per-noun gender from UniMorph por / kaikki
(do-support no)
(subject-aux-inversion no)
(question-strategy intonation)
(article-selection "o/a/os/as um/uma/uns/umas")
(adjective-position postnominal)
(adjective-agreement "gender+number")
;; -- MANDATORY CONTRACTIONS (prep + article) -----------------------------
(contractions ((de o "do") (de a "da") (em o "no") (em a "na")
(a o "ao") (a a "à") (por o "pelo") (por a "pela")))
(contraction-mandatory yes)
;; -- verb / aspect system ------------------------------------------------
(verb-classes (ar er ir))
(tenses (present preterite imperfect future conditional))
(moods (ind sbjv imp))
(finite-agreement "person+number (6 slots)")
(perfect-aux "ter") ; ter + past participle
(copula-split "ser/estar")
(personal-infinitive yes) ; distinctive PT inflected infinitive
;; -- clitics / government ------------------------------------------------
(object-clitics yes) ; mesoclisis/enclisis/proclisis by context
(verb-prep-government yes)
;; -- SACRED safety bar ---------------------------------------------------
(negation-faithful yes))
+71
View File
@@ -0,0 +1,71 @@
;;; lang_profile_ro.el — Romanian language profile for ELP.
;;; Romanian is the BIG typological delta of the Romance family. The verb/clause
;;; engine and the SACRED negation contract mirror the ES/PT/IT core, but the
;;; NOMINAL system is genuinely new: a SUFFIXED definite article, preserved CASE,
;;; a NEUTER gender, and a VOCATIVE. Those flags mark where the shared engine was
;;; extended rather than reused.
(lang_profile_ro
(language "Romanian")
(iso639 "ro")
(family "Romance (Eastern / Balkan)")
;; ── core typology flags ────────────────────────────────────────────────
(pro-drop yes) ; null subjects default; overt pronoun = emphatic
(obligatory-subject no)
(grammatical-gender yes) ; m / f / NEUTER (n)
(neuter-gender yes) ; << ROMANIAN-SPECIFIC: masc-agreeing SG, fem-agreeing PL
; (un tren nou / două trenuri noi)
(do-support no)
(subject-aux-inversion no) ; yes/no Q = declarative order + '?'
(question-punct plain) ; ? and ! only
;; ── SUFFIXED DEFINITE ARTICLE (the headline engine extension) ───────────
(definite-article suffixed) ; << UNIQUE IN ROMANCE: enclitic on the noun
(definite-forms ((m/n sg "-ul / -le / -l : om->omul, câine->câinele, codru->codrul")
(f sg "-a / -ea / -ua : casă->casa, carte->cartea, stea->steaua")
(m pl "-i : oameni->oamenii")
(f/n pl "-le : case->casele, trenuri->trenurile")))
(article-host ((no-prenom-adj noun) ; omul bun
(prenom-adj adjective))) ; bunul om (adj carries the article)
(indefinite-article ((m/n "un") (f "o") (pl "niște") (gen/dat-pl "unor")))
;; ── CASE (preserved; NOM/ACC vs GEN/DAT) ────────────────────────────────
(case (nom/acc gen/dat vocative)) ; << ROMANIAN-SPECIFIC
(case-syncretism "nom=acc ; gen=dat")
(genitive-marking "gen/dat definite: -lui (m/n), -ei/-i (f), -lor (pl)")
(genitival-article ((m sg "al") (f sg "a") (m pl "ai") (f/n pl "ale"))) ; o carte a lui
(possession "definite-head + gen/dat possessor: casa băiatului")
(vocative ((m sg "-ule/-e : omule, băiete") (f sg "-o : Mario, fato")
(pl "-lor")))
;; ── verb / aspect system ────────────────────────────────────────────────
(finite-agreement "person+number (6-way)")
(tenses (prezent imperfect perfect-simplu conjunctiv-prezent
imperativ (periphrastic: perfect-compus viitor conditional)))
(compound-past "perfectul compus = a-avea-clitic + INVARIABLE participle")
(perfect-aux "a avea (am/ai/a/am/ați/au) — ONE auxiliary for ALL verbs")
(perfect-aux-split no) ; << SIMPLER than Italian: no essere/avere selection
(participle-agreement none) ; invariable in the perfect compus (agrees only as
; an adjective / in the passive)
(future "voi/vei/va/vom/veți/vor + infinitive (viitor literar)")
(conditional "aș/ai/ar/am/ați/ar + infinitive")
(subjunctive "conjunctiv: particle 'să' + subjunctive present")
(modal-complement "modal + să + subjunctive (vreau să merg, poți să ajuți)")
(copula "a fi")
(passive "a fi + participle (participle AGREES like an adjective)")
(comparative "mai / mai puțin ADJ decât")
;; ── clitic system (partial — see honest gaps) ───────────────────────────
(clitics yes)
(clitic-set ((acc te îl o ne îi le) (dat îmi îți îi ne le)
(refl te se ne se)))
(clitic-placement ((finite proclitic) ; îmi place, o văd
(perfect-compus elision) ; << m-am, l-am, i-am (PARTIAL)
(imperative-affirmative enclitic))) ; dă-mi (PARTIAL)
;; ── SACRED safety bar (shared with es/pt/it/en) ─────────────────────────
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
(negation "nu (single preverbal marker) + concord")
(negative-concord yes) ; nu … nimic / nimeni / niciodată / niciun
(negative-imperative "nu + INFINITIVE : nu pleca! (KNOWN GAP: uses imperative stem)"))
+1
View File
@@ -250,6 +250,7 @@ fn en_irregular_verb(base: String) -> [String] {
if str_eq(base, "cut") { let r: [String] = ["cut", "cuts", "cut", "cut", "cutting"]; return r }
if str_eq(base, "set") { let r: [String] = ["set", "sets", "set", "set", "setting"]; return r }
if str_eq(base, "hit") { let r: [String] = ["hit", "hits", "hit", "hit", "hitting"]; return r }
if str_eq(base, "fight") { let r: [String] = ["fight", "fights","fought", "fought", "fighting"]; return r }
return empty
}
+280
View File
@@ -0,0 +1,280 @@
// multilingual.el - the language layer for the native-el interlocutor.
//
// Deterministic, NO generative model (ports multilingual.py):
// 1. ml_detect(text) -> ISO code (en/es/pt/it) via stopword + diacritic score
// 2. ml_tr(key, lang) -> localized fixed phrase (SACRED per-language yes/no/decline)
// 3. ml_term(w, lang) -> PT/ES content term -> EN engram equivalent
// 4. ml_translate_pred(lemma, lang) -> EN predicate lemma -> target infinitive
//
// The Python detector count-weights stopwords and diacritics; here diacritics are
// scored by PRESENCE (str_contains) rather than codepoint counting, to stay clear
// of UTF-8 index hazards in the runtime. Faithful enough to classify typical
// queries; documented simplification. Depends on: comprehend (cp_tokenize).
// 1. language detection
fn ml_stop_en(w: String) -> Bool {
if str_eq(w, "the") { return true }
if str_eq(w, "does") { return true }
if str_eq(w, "do") { return true }
if str_eq(w, "did") { return true }
if str_eq(w, "what") { return true }
if str_eq(w, "who") { return true }
if str_eq(w, "is") { return true }
if str_eq(w, "are") { return true }
if str_eq(w, "how") { return true }
if str_eq(w, "you") { return true }
if str_eq(w, "your") { return true }
if str_eq(w, "of") { return true }
if str_eq(w, "to") { return true }
if str_eq(w, "and") { return true }
if str_eq(w, "for") { return true }
if str_eq(w, "explain") { return true }
if str_eq(w, "answer") { return true }
if str_eq(w, "memory") { return true }
if str_eq(w, "with") { return true }
if str_eq(w, "not") { return true }
if str_eq(w, "store") { return true }
return false
}
fn ml_stop_es(w: String) -> Bool {
if str_eq(w, "que") { return true }
if str_eq(w, "qué") { return true }
if str_eq(w, "una") { return true }
if str_eq(w, "usted") { return true }
if str_eq(w, "su") { return true }
if str_eq(w, "cómo") { return true }
if str_eq(w, "como") { return true }
if str_eq(w, "cuál") { return true }
if str_eq(w, "quién") { return true }
if str_eq(w, "está") { return true }
if str_eq(w, "es") { return true }
if str_eq(w, "los") { return true }
if str_eq(w, "las") { return true }
if str_eq(w, "del") { return true }
if str_eq(w, "al") { return true }
if str_eq(w, "explica") { return true }
if str_eq(w, "explique") { return true }
if str_eq(w, "forma") { return true }
if str_eq(w, "con") { return true }
if str_eq(w, "memoria") { return true }
if str_eq(w, "responde") { return true }
return false
}
fn ml_stop_pt(w: String) -> Bool {
if str_eq(w, "que") { return true }
if str_eq(w, "uma") { return true }
if str_eq(w, "você") { return true }
if str_eq(w, "sua") { return true }
if str_eq(w, "seu") { return true }
if str_eq(w, "como") { return true }
if str_eq(w, "memória") { return true }
if str_eq(w, "isso") { return true }
if str_eq(w, "os") { return true }
if str_eq(w, "as") { return true }
if str_eq(w, "da") { return true }
if str_eq(w, "do") { return true }
if str_eq(w, "na") { return true }
if str_eq(w, "no") { return true }
if str_eq(w, "explica") { return true }
if str_eq(w, "forma") { return true }
if str_eq(w, "é") { return true }
if str_eq(w, "está") { return true }
if str_eq(w, "com") { return true }
if str_eq(w, "responda") { return true }
return false
}
fn ml_stop_it(w: String) -> Bool {
if str_eq(w, "che") { return true }
if str_eq(w, "una") { return true }
if str_eq(w, "come") { return true }
if str_eq(w, "della") { return true }
if str_eq(w, "gli") { return true }
if str_eq(w, "è") { return true }
if str_eq(w, "sono") { return true }
if str_eq(w, "questo") { return true }
if str_eq(w, "nel") { return true }
if str_eq(w, "di") { return true }
if str_eq(w, "il") { return true }
if str_eq(w, "cosa") { return true }
if str_eq(w, "per") { return true }
if str_eq(w, "memoria") { return true }
if str_eq(w, "spiega") { return true }
if str_eq(w, "rispondi") { return true }
return false
}
// diacritic PRESENCE score (weight 3 each; hard overrides weight 8).
fn ml_dia_score(low: String, lang: String) -> Int {
let s: Int = 0
if str_eq(lang, "pt") {
if str_contains(low, "ã") { let s = s + 3 }
if str_contains(low, "õ") { let s = s + 3 }
if str_contains(low, "ç") { let s = s + 3 }
if str_contains(low, "ê") { let s = s + 3 }
if str_contains(low, "á") { let s = s + 3 }
// hard PT markers (ã/õ almost never appear outside PT)
if str_contains(low, "ã") { let s = s + 8 }
if str_contains(low, "õ") { let s = s + 8 }
}
if str_eq(lang, "es") {
if str_contains(low, "ñ") { let s = s + 3 }
if str_contains(low, "¿") { let s = s + 3 }
if str_contains(low, "¡") { let s = s + 3 }
if str_contains(low, "á") { let s = s + 3 }
if str_contains(low, "é") { let s = s + 3 }
// hard ES markers
if str_contains(low, "ñ") { let s = s + 8 }
if str_contains(low, "¿") { let s = s + 8 }
if str_contains(low, "¡") { let s = s + 8 }
}
if str_eq(lang, "it") {
if str_contains(low, "è") { let s = s + 3 }
if str_contains(low, "ì") { let s = s + 3 }
if str_contains(low, "ò") { let s = s + 3 }
}
return s
}
fn ml_stop_score(toks: [String], lang: String) -> Int {
let n: Int = native_list_len(toks)
let s: Int = 0
let i: Int = 0
while i < n {
let w: String = native_list_get(toks, i)
if str_eq(lang, "en") { if ml_stop_en(w) { let s = s + 2 } }
if str_eq(lang, "es") { if ml_stop_es(w) { let s = s + 2 } }
if str_eq(lang, "pt") { if ml_stop_pt(w) { let s = s + 2 } }
if str_eq(lang, "it") { if ml_stop_it(w) { let s = s + 2 } }
let i = i + 1
}
return s
}
fn ml_detect(text: String) -> String {
if str_eq(text, "") { return "en" }
let low: String = str_to_lower(text)
let toks: [String] = cp_tokenize(text)
// NOTE: el's overloaded `+` mis-compiles two chained function-call Int operands
// as string concat (documented in comprehend_gate.el). Bind each call to an Int
// var and add vars one at a time so the addition stays integer.
let en: Int = ml_stop_score(toks, "en")
let es_s: Int = ml_stop_score(toks, "es")
let es_d: Int = ml_dia_score(low, "es")
let es: Int = es_s + es_d
let pt_s: Int = ml_stop_score(toks, "pt")
let pt_d: Int = ml_dia_score(low, "pt")
let pt: Int = pt_s + pt_d
let it_s: Int = ml_stop_score(toks, "it")
let it_d: Int = ml_dia_score(low, "it")
let it: Int = it_s + it_d
let best: String = "en"
let bs: Int = en
if es > bs { let best = "es"; let bs = es }
if pt > bs { let best = "pt"; let bs = pt }
if it > bs { let best = "it"; let bs = it }
// weak signal -> honest fallback to English
if bs < 3 { return "en" }
return best
}
// 2. localized fixed phrases (SACRED per-language decline/yes/no)
fn ml_tr(key: String, lang: String) -> String {
if str_eq(key, "no_memory") {
if str_eq(lang, "pt") { return "Não tenho isso na minha memória." }
if str_eq(lang, "es") { return "No tengo eso en mi memoria." }
if str_eq(lang, "it") { return "Non ho quello nella mia memoria." }
return "I don't have that in my memory."
}
if str_eq(key, "parse_fail") {
if str_eq(lang, "pt") { return "Não consegui interpretar isso." }
if str_eq(lang, "es") { return "No pude interpretar eso." }
if str_eq(lang, "it") { return "Non sono riuscito a interpretarlo." }
return "I didn't parse that."
}
if str_eq(key, "yes") {
if str_eq(lang, "pt") { return "Sim" }
if str_eq(lang, "es") { return "" }
if str_eq(lang, "it") { return "" }
return "Yes"
}
if str_eq(key, "no") {
if str_eq(lang, "pt") { return "Não" }
if str_eq(lang, "es") { return "No" }
if str_eq(lang, "it") { return "No" }
return "No"
}
if str_eq(key, "identity") {
if str_eq(lang, "pt") { return "Sou o Neuron, o engrama com quem você está falando." }
if str_eq(lang, "es") { return "Soy Neuron, el engrama con el que estás hablando." }
if str_eq(lang, "it") { return "Sono Neuron, l'engramma con cui stai parlando." }
return "I'm Neuron, the engram you're speaking with."
}
return ""
}
// 3. retrieval term lexicon (PT/ES content term -> EN engram equivalent)
fn ml_term(w: String, lang: String) -> String {
if str_eq(lang, "en") { return w }
if str_eq(w, "saliência") { return "salience" }
if str_eq(w, "saliencia") { return "salience" }
if str_eq(w, "memória") { return "memory" }
if str_eq(w, "memoria") { return "memory" }
if str_eq(w, "geometria") { return "geometry" }
if str_eq(w, "geometrias") { return "geometry" }
if str_eq(w, "geometrías") { return "geometry" }
if str_eq(w, "forma") { return "form" }
if str_eq(w, "consolidação") { return "consolidation" }
if str_eq(w, "consolidación") { return "consolidation" }
if str_eq(w, "aprendizagem") { return "learning" }
if str_eq(w, "aprendizaje") { return "learning" }
if str_eq(w, "") { return "node" }
if str_eq(w, "nodo") { return "node" }
if str_eq(w, "armazenamento") { return "storage" }
if str_eq(w, "almacenamiento") { return "storage" }
if str_eq(w, "estrutura") { return "structure" }
if str_eq(w, "estructura") { return "structure" }
return w
}
// 4. predicate translation (EN lemma -> target infinitive; pass-through) ─────
fn ml_translate_pred(lemma: String, lang: String) -> String {
if str_eq(lang, "en") { return lemma }
if str_eq(lang, "es") {
if str_eq(lemma, "store") { return "almacenar" }
if str_eq(lemma, "use") { return "usar" }
if str_eq(lemma, "have") { return "tener" }
if str_eq(lemma, "be") { return "ser" }
if str_eq(lemma, "give") { return "dar" }
if str_eq(lemma, "make") { return "hacer" }
if str_eq(lemma, "learn") { return "aprender" }
if str_eq(lemma, "form") { return "formar" }
return lemma
}
if str_eq(lang, "pt") {
if str_eq(lemma, "store") { return "armazenar" }
if str_eq(lemma, "use") { return "usar" }
if str_eq(lemma, "have") { return "ter" }
if str_eq(lemma, "be") { return "ser" }
if str_eq(lemma, "give") { return "dar" }
if str_eq(lemma, "make") { return "fazer" }
if str_eq(lemma, "learn") { return "aprender" }
if str_eq(lemma, "form") { return "formar" }
return lemma
}
if str_eq(lang, "it") {
if str_eq(lemma, "store") { return "memorizzare" }
if str_eq(lemma, "use") { return "usare" }
if str_eq(lemma, "have") { return "avere" }
if str_eq(lemma, "be") { return "essere" }
return lemma
}
return lemma
}
+125
View File
@@ -0,0 +1,125 @@
// organ-read.el - Route the render's GEOMETRY READ through the ingest ORGAN's
// saved engram files (the coordinator's source of truth). For each file we
// engram_load() it, engram_scan_nodes_json(limit, offset) to get the node array,
// and cache each node's self-contained CONTENT string keyed by symbol. Because
// the cached value carries the numbers ("... f1=730 ..."), the cache SURVIVES the
// store being REPLACED by the next engram_load so we load+cache phonetics
// FIRST, then load+cache accent. The .psv path remains a fallback.
//
// engram_scan_nodes_json(limit, offset) takes NO query; it returns nodes
// salience-sorted, so limit must be >= node count and we filter client-side.
// (engram_search / engram_scan_nodes return len-5 garbage unused.)
// Find every occurrence of `marker` in the scan JSON; for each, cache
// sym -> a 150-char content window (enough to hold f1..amp). Duplicates from the
// node's "content" and "label" fields are harmless (first match wins on read).
fn organ_cache(j: String, marker: String, mlen: Int, win_len: Int, need: String) -> [String] {
let m: [String] = native_list_empty()
let jl: Int = str_len(j)
let off: Int = 0
while off < jl {
let rest: String = str_slice(j, off, jl)
let p: Int = str_index_of(rest, marker)
if p < 0 {
off = jl
} else {
let abs: Int = off + p
let win: String = str_slice(j, abs, abs + win_len)
let after: String = str_slice(win, mlen, str_len(win))
let sp: Int = str_index_of(after, " ")
let hasneed: Int = str_index_of(win, need)
if sp > 0 {
if hasneed >= 0 {
let sym: String = str_slice(after, 0, sp)
m = native_list_append(m, sym)
m = native_list_append(m, win)
}
}
off = abs + mlen
}
}
return m
}
// Load the phonetics organ file and cache sym -> content. mlen("phoneme ")=8.
fn organ_pmap(path: String) -> [String] {
let ok: Bool = engram_load(path)
if ok == false {
return native_list_empty()
}
let j: String = engram_scan_nodes_json(600, 0)
return organ_cache(j, "phoneme ", 8, 150, "f1=")
}
// Load the accent organ file and cache sym -> content. mlen("accent_target ")=14.
// Vowel overrides carry f1=..; the R rule carries drop_coda_r (need="=" matches
// both, i.e. any well-formed accent_target field).
fn organ_amap(path: String) -> [String] {
let ok: Bool = engram_load(path)
if ok == false {
return native_list_empty()
}
let j: String = engram_scan_nodes_json(600, 0)
return organ_cache(j, "accent_target ", 14, 90, "=")
}
// Vowel-set (categorical class) from the phonetics .psv class column.
fn organ_vset(path: String) -> [String] {
let content: String = fs_read(path)
let lines: [String] = str_split(content, "\n")
let nl: Int = native_list_len(lines)
let v: [String] = native_list_empty()
let li: Int = 0
while li < nl {
let line: String = native_list_get(lines, li)
let ok: Int = 1
if str_len(line) < 5 {
ok = 0
}
if ok == 1 {
if str_char_code(line, 0) == 35 {
ok = 0
}
}
if ok == 1 {
let f: [String] = str_split(line, "|")
if native_list_len(f) >= 12 {
if str_eq(native_list_get(f, 11), "vowel") {
v = native_list_append(v, native_list_get(f, 0))
}
}
}
li = li + 1
}
return v
}
// Word -> phoneme-sequence cache from lexicon.psv (engram-independent).
fn organ_lex(path: String) -> [String] {
let content: String = fs_read(path)
let lines: [String] = str_split(content, "\n")
let nl: Int = native_list_len(lines)
let m: [String] = native_list_empty()
let li: Int = 0
while li < nl {
let line: String = native_list_get(lines, li)
let ok: Int = 1
if str_len(line) < 3 {
ok = 0
}
if ok == 1 {
if str_char_code(line, 0) == 35 {
ok = 0
}
}
if ok == 1 {
let f: [String] = str_split(line, "|")
if native_list_len(f) >= 2 {
m = native_list_append(m, native_list_get(f, 0))
m = native_list_append(m, native_list_get(f, 1))
}
}
li = li + 1
}
return m
}
+140
View File
@@ -0,0 +1,140 @@
// propositions.el - the READ primitive over the engram's OWN memories, native el.
//
// Free memory text -> structured PROPOSITIONS (triples):
// (subject, predicate, object, modifiers, polarity, tense, source, confidence)
//
// This is comprehension turned inward: the Python reference (propositions.py) ran
// spaCy's dependency parser over each memory sentence and walked the arcs. Here
// the spaCy role is filled by the el-native parser (comprehend.el / parse_spec):
// each sentence is parsed to a meaning-spec, and the spec's roles ARE the triple.
// Nothing generates text. NEGATION IS SACRED: polarity flows straight from the
// spec's polarity field and is never dropped or inverted.
//
// Depends on: comprehend (parse_spec / parse_spec_lang), grammar (slots_get).
// sentence segmentation
// Split on sentence-final punctuation (. ! ?) and hard newlines. Markdown/long
// memories are handled shallowly (the reference caps + ranks by query overlap;
// that ranking belongs to the dialogue layer, not here).
fn prop_is_boundary(c: String) -> Bool {
if str_eq(c, ".") { return true }
if str_eq(c, "!") { return true }
if str_eq(c, "?") { return true }
if str_eq(c, "\n") { return true }
return false
}
fn prop_split_sentences(text: String) -> [String] {
let out: [String] = native_list_empty()
let n: Int = str_len(text)
let start: Int = 0
let i: Int = 0
while i < n {
let c: String = str_slice(text, i, i + 1)
if prop_is_boundary(c) {
let seg: String = str_slice(text, start, i + 1)
let trimmed: String = cp_trim_punct(seg)
if !str_eq(trimmed, "") {
let out = native_list_append(out, seg)
}
let start = i + 1
}
let i = i + 1
}
if start < n {
let seg: String = str_slice(text, start, n)
let trimmed: String = cp_trim_punct(seg)
if !str_eq(trimmed, "") {
let out = native_list_append(out, seg)
}
}
return out
}
// spec -> proposition record
// A proposition is a slot map (same [String] shape as the spec) with the READ
// contract keys. Modifiers fold the spec's location + iobj adjuncts.
fn prop_confidence(subject: String, predicate: String, object: String) -> String {
if str_eq(predicate, "") { return "0.0" }
if str_eq(subject, "") { return "0.4" }
if str_eq(object, "") { return "0.7" }
return "1.0"
}
fn prop_modifiers(spec: [String]) -> String {
let loc: String = slots_get(spec, "location")
let iobj: String = slots_get(spec, "iobj")
let parts: [String] = native_list_empty()
if !str_eq(loc, "") { let parts = native_list_append(parts, loc) }
if !str_eq(iobj, "") { let parts = native_list_append(parts, "to " + iobj) }
return str_join(parts, "; ")
}
fn prop_from_spec(spec: [String], source_id: String) -> [String] {
let subject: String = slots_get(spec, "agent")
let predicate: String = slots_get(spec, "predicate")
let object: String = slots_get(spec, "patient")
let polarity: String = slots_get(spec, "polarity")
let tense: String = slots_get(spec, "tense")
let mods: String = prop_modifiers(spec)
let conf: String = prop_confidence(subject, predicate, object)
let p: [String] = native_list_empty()
let p = native_list_append(p, "subject"); let p = native_list_append(p, subject)
let p = native_list_append(p, "predicate"); let p = native_list_append(p, predicate)
let p = native_list_append(p, "object"); let p = native_list_append(p, object)
let p = native_list_append(p, "modifiers"); let p = native_list_append(p, mods)
let p = native_list_append(p, "polarity"); let p = native_list_append(p, polarity)
let p = native_list_append(p, "tense"); let p = native_list_append(p, tense)
let p = native_list_append(p, "source"); let p = native_list_append(p, source_id)
let p = native_list_append(p, "confidence"); let p = native_list_append(p, conf)
return p
}
// Extract one proposition from a single sentence (given language).
fn prop_extract_one_lang(sentence: String, lang: String, source_id: String) -> [String] {
let spec: [String] = parse_spec_lang(sentence, lang)
return prop_from_spec(spec, source_id)
}
fn prop_extract_one(sentence: String, source_id: String) -> [String] {
return prop_extract_one_lang(sentence, "en", source_id)
}
// Render a proposition as a compact trace line (repr parity with propositions.py).
fn prop_repr(p: [String]) -> String {
let neg: String = ""
if str_eq(slots_get(p, "polarity"), "neg") { let neg = "NOT " }
let mods: String = slots_get(p, "modifiers")
let modstr: String = ""
if !str_eq(mods, "") { let modstr = " [" + mods + "]" }
let s: String = "(" + slots_get(p, "subject") + " -" + neg + slots_get(p, "predicate")
let s = s + "-> " + slots_get(p, "object") + modstr
let s = s + " conf=" + slots_get(p, "confidence") + ")"
return s
}
// Extract all propositions from a memory's text (one per sentence). Returns a
// flat [String] whose entries are the prop_repr trace lines, in reading order.
fn prop_extract_lang(text: String, lang: String, source_id: String) -> [String] {
let sents: [String] = prop_split_sentences(text)
let m: Int = native_list_len(sents)
let out: [String] = native_list_empty()
let i: Int = 0
while i < m {
let sent: String = native_list_get(sents, i)
let p: [String] = prop_extract_one_lang(sent, lang, source_id)
// drop empty parses (no predicate recovered): honest partial, not noise.
if !str_eq(slots_get(p, "predicate"), "") {
let out = native_list_append(out, prop_repr(p))
}
let i = i + 1
}
return out
}
fn prop_extract(text: String, source_id: String) -> [String] {
return prop_extract_lang(text, "en", source_id)
}
+125
View File
@@ -34,6 +34,13 @@ fn agent_person(agent: String) -> String {
if str_eq(agent, "we") { return "first" }
if str_eq(agent, "us") { return "first" }
if str_eq(agent, "you") { return "second" }
// Romance target-language subject pronouns (translate.el sets these).
if str_eq(agent, "yo") { return "first" }
if str_eq(agent, "eu") { return "first" }
if str_eq(agent, "nosotros") { return "first" }
if str_eq(agent, "nós") { return "first" }
if str_eq(agent, "") { return "second" }
if str_eq(agent, "tu") { return "second" }
return "third"
}
@@ -50,6 +57,19 @@ fn agent_number(agent: String) -> String {
if str_eq(agent, "us") { return "plural" }
if str_eq(agent, "they") { return "plural" }
if str_eq(agent, "them") { return "plural" }
// Romance target-language subject pronouns.
if str_eq(agent, "yo") { return "singular" }
if str_eq(agent, "eu") { return "singular" }
if str_eq(agent, "") { return "singular" }
if str_eq(agent, "tu") { return "singular" }
if str_eq(agent, "él") { return "singular" }
if str_eq(agent, "ella") { return "singular" }
if str_eq(agent, "ele") { return "singular" }
if str_eq(agent, "ela") { return "singular" }
if str_eq(agent, "nosotros") { return "plural" }
if str_eq(agent, "nós") { return "plural" }
if str_eq(agent, "ellos") { return "plural" }
if str_eq(agent, "eles") { return "plural" }
return "singular"
}
@@ -248,6 +268,56 @@ fn add_punct(s: String, intent: String) -> String {
return s + "."
}
// Polarity-aware negation (SACRED field honored on the generation side)
//
// Negation must never be dropped between comprehension and realization. The
// meaning-spec carries an explicit "polarity" field ("aff"|"neg") and optional
// "neg_word" (standalone negative adverb, e.g. "never"). English uses
// do-support ("did not see") or preverbal adverb ("never fought"); copular "be"
// takes post-verbal "not"; other languages get a preverbal negator particle.
fn realize_negator(code: String) -> String {
if str_eq(code, "es") { return "no" }
if str_eq(code, "pt") { return "não" }
if str_eq(code, "ca") { return "no" }
if str_eq(code, "it") { return "non" }
if str_eq(code, "fr") { return "ne" }
if str_eq(code, "de") { return "nicht" }
if str_eq(code, "ro") { return "nu" }
return "not"
}
fn realize_assert_neg_en(predicate: String, tense: String, person: String, number: String, agent: String, patient: String, iobj: String, location: String, neg_word: String, profile: [String]) -> String {
let parts: [String] = native_list_empty()
let parts = native_list_append(parts, agent)
if !str_eq(neg_word, "") {
// adverbial negation: "I never fought the ocean."
let verb_surf: String = morph_conjugate(predicate, tense, person, number, profile)
let parts = native_list_append(parts, neg_word)
let parts = native_list_append(parts, verb_surf)
} else {
if str_eq(predicate, "be") {
// copular: "she was not a monster"
let be_form: String = morph_conjugate("be", tense, person, number, profile)
let parts = native_list_append(parts, be_form)
let parts = native_list_append(parts, "not")
} else {
// do-support: "she did not see the man"
let do_form: String = morph_conjugate("do", tense, person, number, profile)
let parts = native_list_append(parts, do_form)
let parts = native_list_append(parts, "not")
let parts = native_list_append(parts, predicate)
}
}
if !str_eq(patient, "") { let parts = native_list_append(parts, patient) }
if !str_eq(iobj, "") {
let parts = native_list_append(parts, "to")
let parts = native_list_append(parts, iobj)
}
if !str_eq(location, "") { let parts = native_list_append(parts, location) }
return str_join(parts, " ")
}
// Main realization entry point
fn realize_lang(form: [String], profile: [String]) -> String {
@@ -284,6 +354,54 @@ fn realize_lang(form: [String], profile: [String]) -> String {
}
// Assertion (declarative)
let polarity: String = slots_get(form, "polarity")
let neg_word: String = slots_get(form, "neg_word")
let iobj: String = slots_get(form, "iobj")
let code: String = lang_get(profile, "code")
// Subordinate clause tail (SACRED completeness the clause is carried, never
// dropped): "<conj> <subordinate surface>", e.g. "because he was a monster".
let subord_conj: String = slots_get(form, "subord_conj")
let subord_text: String = slots_get(form, "subord_text")
let subord_tail: String = ""
if !str_eq(subord_conj, "") {
if !str_eq(subord_text, "") {
let subord_tail = subord_conj + " " + subord_text
} else {
let subord_tail = subord_conj
}
}
// Negative polarity: SACRED never dropped.
if str_eq(polarity, "neg") {
if str_eq(code, "en") {
let sentence: String = realize_assert_neg_en(predicate, tense, person, number, agent, patient, iobj, location, neg_word, profile)
return add_punct(capitalize_first(sentence), "assert")
}
// Generic non-English: affirmative core with a preverbal negator particle.
// SACRED: when a standalone negative adverb was carried (e.g. "nunca",
// localized upstream from "never"), surface it rather than the generic
// negator the specific negation must never be flattened away.
let neg_particle: String = realize_negator(code)
if !str_eq(neg_word, "") { let neg_particle = neg_word }
let vp_pair: [String] = realize_vp_lang(predicate, tense, aspect, person, number, profile)
let verb_surf: String = native_list_get(vp_pair, 0)
let aux_surf: String = native_list_get(vp_pair, 1)
let vp_str: String = neg_particle + " " + gram_build_vp(verb_surf, aux_surf, profile)
let core: String = gram_order_constituents(agent, vp_str, patient, profile)
let parts: [String] = native_list_empty()
let parts = native_list_append(parts, core)
if !str_eq(iobj, "") {
let parts = native_list_append(parts, "to")
let parts = native_list_append(parts, iobj)
}
if !str_eq(location, "") { let parts = native_list_append(parts, location) }
if !str_eq(subord_tail, "") { let parts = native_list_append(parts, subord_tail) }
let sentence: String = str_join(parts, " ")
return add_punct(capitalize_first(sentence), "assert")
}
// Affirmative.
let vp_pair: [String] = realize_vp_lang(predicate, tense, aspect, person, number, profile)
let verb_surf: String = native_list_get(vp_pair, 0)
let aux_surf: String = native_list_get(vp_pair, 1)
@@ -293,9 +411,16 @@ fn realize_lang(form: [String], profile: [String]) -> String {
let parts: [String] = native_list_empty()
let parts = native_list_append(parts, core)
if !str_eq(iobj, "") {
let parts = native_list_append(parts, "to")
let parts = native_list_append(parts, iobj)
}
if !str_eq(location, "") {
let parts = native_list_append(parts, location)
}
if !str_eq(subord_tail, "") {
let parts = native_list_append(parts, subord_tail)
}
let sentence: String = str_join(parts, " ")
return add_punct(capitalize_first(sentence), "assert")
}
+180
View File
@@ -0,0 +1,180 @@
// self_region.el the engram's REAL self/identity region, pulled at query time
// (native el). This replaces the hardcoded identity anchors and the canned
// "I'm Neuron, the engram you're speaking with." template: the identity LANDING
// signal and the identity READOUT both come from the engram's own Self/identity
// nodes, read through the in-process engram el API.
//
// Port of self_region.py. The Python module precomputed MiniLM landing vectors;
// here the engram's own store IS the geometry we pull the self nodes by
// single-term lexical search (the engram search is a single-term matcher, so we
// pool several probes) and rank them by self-signal. No text is generated; the
// readout is the self nodes' OWN prose, verbatim (SACRED negation survives by
// construction we never paraphrase, so a negated self-statement stays negated).
//
// ENGRAM el API NOTE: engram_search_json / engram_get_node_json / engram_node_full
// / engram_connect are C runtime builtins. Their argument order is the C order
// (engram_connect(from, to, weight, relation)), NOT the runtime/engram.el wrapper
// order we call the builtins directly and never concatenate that wrapper.
//
// Depends on: comprehend (str helpers via runtime), propositions (prop_split_sentences),
// multilingual (ml_tr), the engram builtins, the json builtins.
// single-term self probes (pooled, because engram search is single-term)
fn sr_terms() -> [String] {
let t: [String] = native_list_empty()
let t = native_list_append(t, "self")
let t = native_list_append(t, "identity")
let t = native_list_append(t, "Neuron")
let t = native_list_append(t, "consciousness")
let t = native_list_append(t, "values")
let t = native_list_append(t, "continuous")
return t
}
// The canonical self-root: content begins "# self" or label is "# self"/"self".
fn sr_is_root(content: String, label: String) -> Bool {
let lc: String = str_to_lower(content)
let ll: String = str_to_lower(str_trim(label))
if str_starts_with(lc, "# self") { return true }
if str_eq(ll, "# self") { return true }
if str_eq(ll, "self") { return true }
return false
}
// How strongly a node belongs to the self/identity region (integer points, to
// avoid el's float-in-`+` pitfalls). Mirrors _self_score in self_region.py.
fn sr_score(node_json: String) -> Int {
let content: String = json_get_string(node_json, "content")
let label: String = json_get_string(node_json, "label")
let tags: String = str_to_lower(json_get_string(node_json, "tags"))
let low: String = str_to_lower(content)
let s: Int = 0
// identity tags
if str_contains(tags, "self") { let s = s + 2 }
if str_contains(tags, "identity") { let s = s + 2 }
if str_contains(tags, "self-model") { let s = s + 2 }
if str_contains(tags, "consciousness") { let s = s + 2 }
if str_contains(tags, "memory-philosophy") { let s = s + 2 }
// the named self-traversal root
if sr_is_root(content, label) { let s = s + 12 }
if str_contains(low, "who i am") { let s = s + 3 }
if str_contains(low, "i am neuron") { let s = s + 3 }
// softer identity keywords
if str_contains(low, "my values") { let s = s + 1 }
if str_contains(low, "my purpose") { let s = s + 1 }
if str_contains(low, "identity") { let s = s + 1 }
return s
}
// list-contains helper (dedup self-node ids across the pooled probes).
fn sr_ids_has(ids: [String], id: String) -> Bool {
let n: Int = native_list_len(ids)
let i: Int = 0
while i < n {
if str_eq(native_list_get(ids, i), id) { return true }
let i = i + 1
}
return false
}
// Pull the self nodes: pool every probe's hits, dedupe by id, keep only nodes
// with genuine self-signal (score >= 1). Returns the node-json strings.
fn sr_pull() -> [String] {
let terms: [String] = sr_terms()
let nt: Int = native_list_len(terms)
let seen: [String] = native_list_empty()
let out: [String] = native_list_empty()
let ti: Int = 0
while ti < nt {
let term: String = native_list_get(terms, ti)
let hits: String = engram_search_json(term, 30)
let hn: Int = json_array_len(hits)
let hi: Int = 0
while hi < hn {
let node: String = json_array_get(hits, hi)
let id: String = json_get_string(node, "id")
if !str_eq(id, "") {
if !sr_ids_has(seen, id) {
let seen = native_list_append(seen, id)
if sr_score(node) >= 1 {
let out = native_list_append(out, node)
}
}
}
let hi = hi + 1
}
let ti = ti + 1
}
return out
}
// Return the single highest-signal self node (the readout seed), or "" if the
// self region is thin/empty. We keep it O(n) pick the max-score node, with the
// canonical root strongly favored by sr_score's +12.
fn sr_best_node() -> String {
let nodes: [String] = sr_pull()
let n: Int = native_list_len(nodes)
let best: String = ""
let best_s: Int = 0
let i: Int = 0
while i < n {
let node: String = native_list_get(nodes, i)
let s: Int = sr_score(node)
if s > best_s {
let best_s = s
let best = node
}
let i = i + 1
}
return best
}
fn sr_available() -> Bool {
if str_eq(sr_best_node(), "") { return false }
return true
}
// Read out the identity from the REAL self node: lead with the first first-person
// self-statement ("I am Neuron …"), then one more grounded self line if present.
// Verbatim from the node's own prose no template, negation SACRED. Falls back
// to the localized identity phrase ONLY if the live pull is empty (logged shape).
fn sr_readout(lang: String) -> String {
let node: String = sr_best_node()
if str_eq(node, "") {
// honest fallback the self region is unreachable/thin.
return ml_tr("identity", lang)
}
let content: String = json_get_string(node, "content")
let sents: [String] = prop_split_sentences(content)
let ns: Int = native_list_len(sents)
let lead: String = ""
let second: String = ""
let i: Int = 0
while i < ns {
let raw: String = str_trim(native_list_get(sents, i))
// strip a leading markdown heading marker
let s: String = raw
if str_starts_with(s, "# ") { let s = str_trim(str_slice(s, 2, str_len(s))) }
let low: String = str_to_lower(s)
let is_fp: Bool = false
if str_starts_with(s, "I ") { let is_fp = true }
if str_starts_with(s, "I'm") { let is_fp = true }
if str_contains(low, "i am neuron") { let is_fp = true }
if is_fp {
if str_eq(lead, "") {
let lead = s
} else {
if str_eq(second, "") { let second = s }
}
}
let i = i + 1
}
if str_eq(lead, "") {
// no first-person line read out the first non-empty sentence verbatim.
if ns > 0 { let lead = str_trim(native_list_get(sents, 0)) }
}
if str_eq(lead, "") { return ml_tr("identity", lang) }
let out: String = lead
if !str_eq(second, "") { let out = out + " " + second }
return out
}
+233
View File
@@ -0,0 +1,233 @@
// speech-ingest.el - The native LOAD step of the ingest organ, for the SPEECH
// primitives. Reads the acoustic-phonetics SOURCE (elp/data/phonetics.psv) and
// the pronunciation lexicon SOURCE (elp/data/lexicon.psv) and emits a PHONEME
// MANIFOLD into the engram: one node per phoneme (faithful, provenance-tagged
// content) + is_a edges to phoneme-class nodes (a discrete manifold, not islands).
// The render then PULLS phoneme geometry back from the engram via phon_geo
// zero phonetic numbers in code. Source -> manifold -> merge; the same output
// the polymorphic ingest organ will produce and subsume.
// -- small parsing helpers ---------------------------------------------------
fn sp_map_get(pairs: [String], key: String) -> String {
let n: Int = native_list_len(pairs)
let i: Int = 0
while i < n - 1 {
let k: String = native_list_get(pairs, i)
if str_eq(k, key) {
return native_list_get(pairs, i + 1)
}
let i = i + 2
}
return ""
}
// read the unsigned integer that follows `key` inside string s (e.g. key "F1=")
fn parse_uint_from(s: String, key: String) -> Int {
let idx: Int = str_index_of(s, key)
if idx < 0 {
return 0
}
let start: Int = idx + str_len(key)
let n: Int = str_len(s)
let i: Int = start
let val: Int = 0
while i < n {
let c: Int = str_char_code(s, i)
if c >= 48 {
if c <= 57 {
val = val * 10 + (c - 48)
i = i + 1
} else {
i = n
}
} else {
i = n
}
}
return val
}
fn clean_word(w: String) -> String {
let low: String = str_to_lower(w)
let n: Int = str_len(low)
let out: String = ""
let i: Int = 0
while i < n {
let c: Int = str_char_code(low, i)
if c >= 97 {
if c <= 122 {
out = out + str_char_at(low, i)
}
}
i = i + 1
}
return out
}
// -- INGEST: acoustic-phonetics source -> phoneme manifold in the engram ------
// Returns the symbol -> node-id index (pmap) the render reads geometry through.
fn ingest_phonetics(path: String) -> [String] {
let content: String = fs_read(path)
let lines: [String] = str_split(content, "\n")
let nl: Int = native_list_len(lines)
let pmap: [String] = native_list_empty()
let classmap: [String] = native_list_empty()
let li: Int = 0
while li < nl {
let line: String = native_list_get(lines, li)
let ll: Int = str_len(line)
let skip: Int = 0
if ll < 5 {
skip = 1
}
if skip == 0 {
let first: Int = str_char_code(line, 0)
if first == 35 {
skip = 1
}
}
if skip == 0 {
let f: [String] = str_split(line, "|")
let nf: Int = native_list_len(f)
if nf >= 12 {
let sym: String = native_list_get(f, 0)
let f1: String = native_list_get(f, 1)
let f2: String = native_list_get(f, 2)
let f3: String = native_list_get(f, 3)
let b1: String = native_list_get(f, 4)
let b2: String = native_list_get(f, 5)
let b3: String = native_list_get(f, 6)
let vo: String = native_list_get(f, 7)
let na: String = native_list_get(f, 8)
let du: String = native_list_get(f, 9)
let am: String = native_list_get(f, 10)
let cls: String = native_list_get(f, 11)
let cont: String = "phoneme " + sym + " | f1=" + f1 + " f2=" + f2 + " f3=" + f3 + " bw1=" + b1 + " bw2=" + b2 + " bw3=" + b3 + " voiced=" + vo + " nasal=" + na + " dur=" + du + " amp=" + am + " class=" + cls + " src=PetersonBarney1952-Hillenbrand1995"
let id: String = engram_node(cont, "Phoneme", 80)
pmap = native_list_append(pmap, sym)
pmap = native_list_append(pmap, cont)
// manifold edge: phoneme is_a class
let cid: String = sp_map_get(classmap, cls)
if str_eq(cid, "") {
cid = engram_node("phoneme-class " + cls + " src=acoustic-phonetics", "PhonemeClass", 80)
classmap = native_list_append(classmap, cls)
classmap = native_list_append(classmap, cid)
}
engram_connect(id, cid, 80, "is_a")
}
}
li = li + 1
}
return pmap
}
// -- INGEST: pronunciation lexicon source -> word nodes ----------------------
fn ingest_lexicon(path: String) -> [String] {
let content: String = fs_read(path)
let lines: [String] = str_split(content, "\n")
let nl: Int = native_list_len(lines)
let lmap: [String] = native_list_empty()
let li: Int = 0
while li < nl {
let line: String = native_list_get(lines, li)
let ll: Int = str_len(line)
let skip: Int = 0
if ll < 3 {
skip = 1
}
if skip == 0 {
let first: Int = str_char_code(line, 0)
if first == 35 {
skip = 1
}
}
if skip == 0 {
let f: [String] = str_split(line, "|")
let nf: Int = native_list_len(f)
if nf >= 2 {
let word: String = native_list_get(f, 0)
let seq: String = native_list_get(f, 1)
let id: String = engram_node("word " + word + " phonemes " + seq + " src=lexicon", "Pronunciation", 80)
lmap = native_list_append(lmap, word)
lmap = native_list_append(lmap, seq)
}
}
li = li + 1
}
return lmap
}
// -- READ geometry back from the engram (the render's afferent lookup) --------
// phon_geo(sym) -> [F1,F2,F3,B1,B2,B3,voiced,nasal,dur,amp], parsed from the
// ingested phoneme node's content. NO formant numbers live in this code.
fn phon_geo(pmap: [String], sym: String) -> [Int] {
let id: String = sp_map_get(pmap, sym)
if str_eq(id, "") {
id = sp_map_get(pmap, "AX")
}
let out: [Int] = native_list_empty()
if str_eq(id, "") {
let out = native_list_append(out, 500)
let out = native_list_append(out, 1500)
let out = native_list_append(out, 2500)
let out = native_list_append(out, 80)
let out = native_list_append(out, 100)
let out = native_list_append(out, 150)
let out = native_list_append(out, 1)
let out = native_list_append(out, 0)
let out = native_list_append(out, 80)
let out = native_list_append(out, 80)
return out
}
let j: String = id
let out = native_list_append(out, parse_uint_from(j, "f1="))
let out = native_list_append(out, parse_uint_from(j, "f2="))
let out = native_list_append(out, parse_uint_from(j, "f3="))
let out = native_list_append(out, parse_uint_from(j, "bw1="))
let out = native_list_append(out, parse_uint_from(j, "bw2="))
let out = native_list_append(out, parse_uint_from(j, "bw3="))
let out = native_list_append(out, parse_uint_from(j, "voiced="))
let out = native_list_append(out, parse_uint_from(j, "nasal="))
let out = native_list_append(out, parse_uint_from(j, "dur="))
let out = native_list_append(out, parse_uint_from(j, "amp="))
return out
}
// word -> phoneme codes, read from the ingested lexicon node.
fn word_phonemes(lmap: [String], word: String) -> [String] {
let id: String = sp_map_get(lmap, word)
if str_eq(id, "") {
let r: [String] = native_list_empty()
let r = native_list_append(r, "AX")
return r
}
return str_split(id, " ")
}
// realized text -> flat phoneme-code sequence (SIL between words + at ends).
fn text_phonemes(lmap: [String], text: String) -> [String] {
let words: [String] = str_split(text, " ")
let nw: Int = native_list_len(words)
let seq: [String] = native_list_empty()
let seq = native_list_append(seq, "SIL")
let wi: Int = 0
while wi < nw {
let raw: String = native_list_get(words, wi)
let w: String = clean_word(raw)
if str_eq(w, "") {
wi = wi + 1
} else {
let ph: [String] = word_phonemes(lmap, w)
let np: Int = native_list_len(ph)
let pi: Int = 0
while pi < np {
let code: String = native_list_get(ph, pi)
seq = native_list_append(seq, code)
pi = pi + 1
}
seq = native_list_append(seq, "SIL")
wi = wi + 1
}
}
return seq
}
+460
View File
@@ -0,0 +1,460 @@
// speech.el - The native SPEECH render path + voice-by-imitation extractor.
//
// Speech = the AUDIO surface (surface_profile_audio) rendering LANGUAGE-meaning
// through a VOICE signature. The realizer's language faculty supplies the words
// (meaning -> sem_realize -> text); this module turns text -> phonemes (phonetics.el)
// -> a formant-target track over time -> SUPERPOSES formant resonances over a
// glottal source (own-core formant synthesis, the exact integer mirror of the
// music additive superpose) -> own-core PCM/WAV. Two paths:
// (1) RENDER: speak(text, voice) -> spoken WAV.
// (2) IMITATE: voice_analyze(pcm) -> a voice signature grabbed BY EAR
// (autocorrelation pitch + integer-DFT formant peaks), then render
// any new meaning in that voice. An impression, not a corpus.
// All integer/fixed-point (EL float arithmetic is unusable).
// -- Own-core integer sine (Bhaskara I), phase 0..65535 = one cycle -----------
fn sp_sin(phase: Int) -> Int {
let deg: Int = phase * 360 / 65536
let neg: Int = 0
if deg > 180 {
deg = deg - 180
neg = 1
}
let t: Int = deg * (180 - deg)
let num: Int = 32767 * 4 * t
let den: Int = 40500 - t
let v: Int = num / den
if neg == 1 {
v = 0 - v
}
return v
}
fn sp_cos(phase: Int) -> Int {
let p: Int = phase + 16384
p = p - (p / 65536) * 65536
return sp_sin(p)
}
// One formant resonance (Lorentzian peak), Q15. Peak 32767 at f=fc.
fn sp_gain(f: Int, fc: Int, bw: Int) -> Int {
let d: Int = f - fc
let den: Int = d * d + bw * bw
let num: Int = 32767 * bw * bw
return num / den
}
fn sp_isqrt(n: Int) -> Int {
if n <= 0 {
return 0
}
let x: Int = n
let y: Int = (x + 1) / 2
while y < x {
x = y
y = (x + n / x) / 2
}
return x
}
// -- WAV serializer (thin medium; the only non-DSP glue) ---------------------
fn wav_le16(buf: String, off: Int, v: Int) -> String {
let u: Int = v
if u < 0 {
u = u + 65536
}
let lo: Int = u - (u / 256) * 256
let hi: Int = u / 256
let b: String = __str_set_char(buf, off, lo)
b = __str_set_char(b, off + 1, hi)
return b
}
fn wav_le32(buf: String, off: Int, v: Int) -> String {
let b0: Int = v - (v / 256) * 256
let r1: Int = v / 256
let b1: Int = r1 - (r1 / 256) * 256
let r2: Int = r1 / 256
let b2: Int = r2 - (r2 / 256) * 256
let b3: Int = r2 / 256
let b: String = __str_set_char(buf, off, b0)
b = __str_set_char(b, off + 1, b1)
b = __str_set_char(b, off + 2, b2)
b = __str_set_char(b, off + 3, b3)
return b
}
fn wav_ascii(buf: String, off: Int, s: String) -> String {
let n: Int = str_len(s)
let i: Int = 0
let b: String = buf
while i < n {
let c: Int = str_char_code(s, i)
b = __str_set_char(b, off + i, c)
i = i + 1
}
return b
}
fn write_wav(samples: [Int], sr: Int, path: String) -> Bool {
let ns: Int = native_list_len(samples)
let datalen: Int = ns * 2
let total: Int = 44 + datalen
let buf: String = __str_alloc(total)
buf = wav_ascii(buf, 0, "RIFF")
buf = wav_le32(buf, 4, 36 + datalen)
buf = wav_ascii(buf, 8, "WAVE")
buf = wav_ascii(buf, 12, "fmt ")
buf = wav_le32(buf, 16, 16)
buf = wav_le16(buf, 20, 1)
buf = wav_le16(buf, 22, 1)
buf = wav_le32(buf, 24, sr)
buf = wav_le32(buf, 28, sr * 2)
buf = wav_le16(buf, 32, 2)
buf = wav_le16(buf, 34, 16)
buf = wav_ascii(buf, 36, "data")
buf = wav_le32(buf, 40, datalen)
let j: Int = 0
let off: Int = 44
while j < ns {
let raw: Int = native_list_get(samples, j)
buf = wav_le16(buf, off, raw)
off = off + 2
j = j + 1
}
return __fs_write_bytes(path, buf, total)
}
// One formant resonance as a float Lorentzian peak (own-core physics).
fn fgain(f: Float, fc: Float, bw: Float) -> Float {
let d: Float = f - fc
return (bw * bw) / (d * d + bw * bw)
}
// His PITCH MELODY from measured prosody [f0_median, f0_min, f0_max, declination].
// A natural statement shape over the utterance: onset rise to the median, a
// near-flat body (his declination is ~0.6 Hz/s), and a final fall toward f0_min.
// Follows his melody + range, not a fixed 0.85 decline. gidx/total = position.
fn prosody_f0(pros: [Int], gidx: Int, total: Int) -> Int {
let med: Int = native_list_get(pros, 0)
let lo: Int = native_list_get(pros, 1)
let hi: Int = native_list_get(pros, 2)
let p: Int = gidx * 1000 / total
let f0: Int = med
if p < 150 {
f0 = lo + (med - lo) * p / 150
} else {
if p > 700 {
f0 = med + (lo - med) * (p - 700) / 300
} else {
f0 = med
}
}
if f0 < lo {
f0 = lo
}
if f0 > hi {
f0 = hi
}
return f0
}
// -- The render: phoneme codes + voice signature -> normalized PCM samples ----
// Formant geometry per phoneme is READ FROM THE ENGRAM (pmap) via phon_geo no
// table in code. The optional ACCENT map (amap) composes a transform onto the
// voice (voice (+) accent, separable): RP formant overrides read from the accent
// manifold + a non-rhotic coda-R drop. Empty amap = base General-American.
// Synthesis is FLOAT: a real phase accumulator + math_sin, superposition physics.
fn synth_codes_accent(codes0: [String], voice: [String], pmap: [String], amap: [String], vset: [String], vmap: [String], prosody: [Int]) -> [Int] {
let sr: Int = 16000
let srf: Float = 16000.0
let two_pi: Float = 6.283185307
let kf: Int = voice_get_int(voice, "kf")
let f0s: Int = voice_get_int(voice, "f0")
let f0e: Int = voice_get_int(voice, "f0_end")
let durm: Int = voice_get_int(voice, "dur")
if kf <= 0 {
kf = 1000
}
if durm <= 0 {
durm = 1000
}
let use_accent: Int = 0
if native_list_len(amap) > 0 {
use_accent = 1
}
let codes: [String] = codes0
if use_accent == 1 {
if is_nonrhotic(amap) == 1 {
codes = apply_rhoticity(codes0, vset)
}
}
let nc: Int = native_list_len(codes)
// pass 1: per-segment sample counts + total
let segn: [Int] = native_list_empty()
let total: Int = 0
let ci: Int = 0
while ci < nc {
let code: String = native_list_get(codes, ci)
let p: [Int] = phon_geo(pmap, code)
let durms: Int = native_list_get(p, 8)
let ns: Int = durms * 16 * durm / 1000
segn = native_list_append(segn, ns)
total = total + ns
ci = ci + 1
}
if total <= 0 {
total = 1
}
// pass 2: synthesize
let samples: [Int] = native_list_empty()
let phasef: Float = 0.0
let gidx: Int = 0
let prevF1: Int = 500 * kf / 1000
let prevF2: Int = 1500 * kf / 1000
let prevF3: Int = 2500 * kf / 1000
let nstate: Int = 22695
let maxabs: Int = 1
let ci2: Int = 0
while ci2 < nc {
let code: String = native_list_get(codes, ci2)
let p: [Int] = phon_geo(pmap, code)
let rf1: Int = native_list_get(p, 0)
let rf2: Int = native_list_get(p, 1)
let rf3: Int = native_list_get(p, 2)
if use_accent == 1 {
let ov: [Int] = accent_formants(amap, code)
if native_list_len(ov) >= 3 {
rf1 = native_list_get(ov, 0)
rf2 = native_list_get(ov, 1)
rf3 = native_list_get(ov, 2)
}
}
// HIS measured vowel target overrides the generic/kf path (absolute Hz
// his formants already encode his vocal tract, so no kf scaling).
let usekf: Int = 1
if native_list_len(vmap) > 0 {
let hv: [Int] = vmap_get(vmap, code)
if native_list_len(hv) >= 3 {
rf1 = native_list_get(hv, 0)
rf2 = native_list_get(hv, 1)
rf3 = native_list_get(hv, 2)
usekf = 0
}
}
let F1t: Int = rf1 * kf / 1000
let F2t: Int = rf2 * kf / 1000
let F3t: Int = rf3 * kf / 1000
if usekf == 0 {
F1t = rf1
F2t = rf2
F3t = rf3
}
let B1: Int = native_list_get(p, 3)
let B2: Int = native_list_get(p, 4)
let B3: Int = native_list_get(p, 5)
let voiced: Int = native_list_get(p, 6)
let ampv: Int = native_list_get(p, 9)
let ns: Int = native_list_get(segn, ci2)
let trans: Int = ns / 2
if trans > 560 {
trans = 560
}
if trans < 1 {
trans = 1
}
let k: Int = 0
while k < ns {
let cF1: Int = F1t
let cF2: Int = F2t
let cF3: Int = F3t
if k < trans {
cF1 = prevF1 + (F1t - prevF1) * k / trans
cF2 = prevF2 + (F2t - prevF2) * k / trans
cF3 = prevF3 + (F3t - prevF3) * k / trans
}
let f0c: Int = f0s + (f0e - f0s) * gidx / total
if native_list_len(prosody) >= 3 {
f0c = prosody_f0(prosody, gidx, total)
}
if f0c < 40 {
f0c = 40
}
let env: Int = 32767
let ar: Int = 96
if k < ar {
env = 32767 * k / ar
}
let tail: Int = ns - k
if tail < ar {
env = 32767 * tail / ar
}
let f0cf: Float = int_to_float(f0c)
phasef = phasef + two_pi * f0cf / srf
if phasef > two_pi {
phasef = phasef - two_pi
}
let s: Int = 0
if voiced == 1 {
let cF1f: Float = int_to_float(cF1)
let cF2f: Float = int_to_float(cF2)
let cF3f: Float = int_to_float(cF3)
let B1f: Float = int_to_float(B1)
let B2f: Float = int_to_float(B2)
let B3f: Float = int_to_float(B3)
let acc: Float = 0.0
let h: Int = 1
while h <= 50 {
let hf: Float = int_to_float(h)
let fhf: Float = hf * f0cf
if fhf < 7900.0 {
let sv: Float = math_sin(phasef * hf)
let src: Float = 1.0 / hf
let g1: Float = fgain(fhf, cF1f, B1f)
let g2: Float = fgain(fhf, cF2f, B2f)
let g3: Float = fgain(fhf, cF3f, B3f)
let g: Float = g1 + g2 + g3
acc = acc + src * g * sv
}
h = h + 1
}
s = float_to_int(acc * 4000.0)
} else {
if ampv > 0 {
nstate = nstate * 1103515245 + 12345
nstate = nstate - (nstate / 2147483648) * 2147483648
if nstate < 0 {
nstate = 0 - nstate
}
let nz: Int = nstate / 32768 - 32768
s = nz
}
}
s = s * ampv / 100
s = s * env / 32767
samples = native_list_append(samples, s)
let a: Int = s
if a < 0 {
a = 0 - a
}
if a > maxabs {
maxabs = a
}
gidx = gidx + 1
k = k + 1
}
prevF1 = F1t
prevF2 = F2t
prevF3 = F3t
ci2 = ci2 + 1
}
// normalize to int16 range (~22000 peak)
let out: [Int] = native_list_empty()
let ntot: Int = native_list_len(samples)
let j: Int = 0
while j < ntot {
let raw: Int = native_list_get(samples, j)
let v: Int = raw * 22000 / maxabs
out = native_list_append(out, v)
j = j + 1
}
return out
}
// GA convenience wrapper (no accent) keeps the base render path.
fn synth_codes(codes: [String], voice: [String], pmap: [String]) -> [Int] {
let noacc: [String] = native_list_empty()
let novset: [String] = native_list_empty()
let novmap: [String] = native_list_empty()
let nopros: [Int] = native_list_empty()
return synth_codes_accent(codes, voice, pmap, noacc, novset, novmap, nopros)
}
// -- Voice-by-imitation: HEAR a PCM sample -> extract the voice signature -----
// Pitch by autocorrelation; vocal-tract scale (kf) from the F1 formant peak of a
// heard sustained vowel /AA/ (nominal F1 = 730 Hz) via an integer DFT. The
// analyzer sees ONLY the PCM samples never the source signature numbers so
// recovery is genuinely by ear.
fn voice_f0(samples: [Int], sr: Int) -> Int {
let n: Int = native_list_len(samples)
let start: Int = n / 4
let end: Int = n * 3 / 4
// bound the analysis window so accumulators can never overflow on long input
if end - start > 6000 {
end = start + 6000
}
let minlag: Int = sr / 300
let maxlag: Int = sr / 75
let best: Int = 0
let bestlag: Int = minlag
let lag: Int = minlag
while lag <= maxlag {
let sum: Int = 0
let i: Int = start
while i < end {
let ai: Int = native_list_get(samples, i)
let bi: Int = native_list_get(samples, i + lag)
sum = sum + ai * bi / 256
i = i + 2
}
if sum > best {
best = sum
bestlag = lag
}
lag = lag + 1
}
if bestlag < 1 {
bestlag = 1
}
return sr / bestlag
}
fn voice_peak_in_band(samples: [Int], sr: Int, flo: Int, fhi: Int) -> Int {
let n: Int = native_list_len(samples)
let start: Int = n / 4
let end: Int = n * 3 / 4
// bound the DFT window: re/im are accumulated /4096, and re*re must stay in
// int64 cap terms so (window/2)*(peak_term) squared cannot overflow.
if end - start > 3000 {
end = start + 3000
}
let bestmag: Int = 0
let bestf: Int = flo
let f: Int = flo
while f <= fhi {
let re: Int = 0
let im: Int = 0
let i: Int = start
while i < end {
let x: Int = native_list_get(samples, i)
let ph: Int = i * f * 65536 / sr
ph = ph - (ph / 65536) * 65536
let cq: Int = sp_cos(ph)
let sq: Int = sp_sin(ph)
re = re + x * cq / 4096
im = im + x * sq / 4096
i = i + 2
}
let mag: Int = re * re + im * im
if mag > bestmag {
bestmag = mag
bestf = f
}
f = f + 25
}
return bestf
}
// Analyze a heard sustained /AA/ -> a full voice signature (by ear).
fn voice_analyze(samples: [Int], sr: Int) -> [String] {
let f0: Int = voice_f0(samples, sr)
let f1: Int = voice_peak_in_band(samples, sr, 450, 1150)
let kf: Int = 1000 * f1 / 730
let f0e: Int = f0 * 85 / 100
return voice_new("imitated", f0, f0e, kf, 1000, 1000, 8)
}
+153
View File
@@ -0,0 +1,153 @@
// surface-profile.el - Surface profile data and accessors.
//
// THE NATIVE EFFERENT SEAM: surface = a pluggable PROFILE, using the exact same
// slot-map mechanism as language-profile.el. A language profile tells the
// realizer HOW to shape a natural-language surface (word order, morphology); a
// SURFACE profile tells the realizer WHICH surface to project meaning onto
// (markdown, docx, html, plain, or a non-text medium like symbolic music).
//
// The generalization is exact: realize_lang(form, profile) already renders a
// SemForm parameterized by a [String] profile read via lang_get. Surface is one
// more axis of that same profile vector. One frame (sem_frame), one plan step
// (sem_to_spec), one render (realize) the surface is DATA, not a code path,
// precisely as language is data. Adding a surface means adding a profile, no
// engine change. This is the multimodal projector, native: geometry -> any
// surface, the efferent twin of ingest.
//
// Surface slot keys:
// surface - "markdown" | "docx" | "html" | "plain" | "midi" | "image"
// modality - "text" | "audio" | "image" | "video"
// media_type - MIME type of the emitted surface
// head_open - string prepended to a heading (e.g. "## " for markdown)
// head_close - string appended to a heading (e.g. "" for markdown, "</h2>" for html)
// emph_open - string opening emphasis (e.g. "*")
// emph_close - string closing emphasis (e.g. "*")
// item_mark - list-item marker (e.g. "- ")
// para_sep - paragraph separator (e.g. "\n\n")
//
// For a TEXT modality the render composes these markers around the surface that
// the EXISTING realizer produces (realize_lang / sem_realize). For a non-text
// modality (audio/image) the profile declares modality + media_type and the
// render dispatches to the medium projector, which reads the SAME frame's
// geometry (its intent/affect/structure) and projects it onto sound or pixels
// deterministic-from-meaning, nothing invented. That dispatch point is where a
// music profile or image profile conforms, native, no parallel layer.
// -- Constructor -------------------------------------------------------------
fn surface_profile(surface: String, modality: String, media_type: String, head_open: String, head_close: String, emph_open: String, emph_close: String, item_mark: String, para_sep: String) -> [String] {
let r: [String] = native_list_empty()
let r = native_list_append(r, "surface")
let r = native_list_append(r, surface)
let r = native_list_append(r, "modality")
let r = native_list_append(r, modality)
let r = native_list_append(r, "media_type")
let r = native_list_append(r, media_type)
let r = native_list_append(r, "head_open")
let r = native_list_append(r, head_open)
let r = native_list_append(r, "head_close")
let r = native_list_append(r, head_close)
let r = native_list_append(r, "emph_open")
let r = native_list_append(r, emph_open)
let r = native_list_append(r, "emph_close")
let r = native_list_append(r, emph_close)
let r = native_list_append(r, "item_mark")
let r = native_list_append(r, item_mark)
let r = native_list_append(r, "para_sep")
let r = native_list_append(r, para_sep)
return r
}
// -- Accessor (same convention as lang_get; standalone so this is a leaf) -----
fn surface_get(profile: [String], key: String) -> String {
let n: Int = native_list_len(profile)
let i: Int = 0
while i < n - 1 {
let k: String = native_list_get(profile, i)
if str_eq(k, key) {
return native_list_get(profile, i + 1)
}
let i = i + 2
}
return ""
}
fn surface_is_text(profile: [String]) -> Bool {
return str_eq(surface_get(profile, "modality"), "text")
}
// -- Built-in TEXT surface profiles ------------------------------------------
// Markdown: headings with "## ", emphasis with "*", "- " list items.
fn surface_profile_markdown() -> [String] {
return surface_profile("markdown", "text", "text/markdown", "## ", "", "*", "*", "- ", "\n\n")
}
// Plain text: no markup at all headings become bare uppercase-free lines.
fn surface_profile_plain() -> [String] {
return surface_profile("plain", "text", "text/plain", "", "", "", "", " - ", "\n\n")
}
// HTML: block-level heading/emphasis tags.
fn surface_profile_html() -> [String] {
return surface_profile("html", "text", "text/html", "<h2>", "</h2>", "<em>", "</em>", "<li>", "\n")
}
// docx: WordprocessingML is structural, not inline-markup; the head/emph slots
// carry the run/style intent that the OOXML emitter maps to <w:pStyle>. Declared
// here so docx is a first-class surface on the same seam.
fn surface_profile_docx() -> [String] {
return surface_profile("docx", "text", "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "Heading2:", "", "b:", "", "bullet:", "\n")
}
// -- Built-in NON-TEXT surface profiles (the multimodal seam) ----------------
// Symbolic music (MIDI): modality=audio. The render dispatches to the music
// projector, which reads the SAME frame's intent/affect and projects it to
// pitch/rhythm deterministic-from-meaning. head/emph slots are empty because
// the medium is not textual; media_type names the surface. A music profile
// (scale/mode/instrument) is layered onto this by the audio agent, native.
fn surface_profile_midi() -> [String] {
return surface_profile("midi", "audio", "audio/midi", "", "", "", "", "", "")
}
// Synthesized audio (WAV): modality=audio, peer to midi. The richer audio
// surface the render SUPERPOSES ingested tonal primitives (sine at f0*n per an
// ingested instrument signature) into PCM, own-core, exactly as midi writes an
// SMF via struct. A music profile (scale/mode/instrument/adsr) layers onto this
// as its own [String] slot-map read by the same getter. Same frame -> midi OR
// audio, interchangeable; this is the audio agent's native conforming point.
fn surface_profile_audio() -> [String] {
return surface_profile("audio", "audio", "audio/wav", "", "", "", "", "", "")
}
// Image (raster): modality=image. Documented seam the render dispatches to the
// image projector, the efferent twin of image ingest, reading the same frame.
fn surface_profile_image() -> [String] {
return surface_profile("image", "image", "image/png", "", "", "", "", "", "")
}
// -- Composition helpers: wrap realized TEXT with the surface's markers -------
//
// These take text the EXISTING realizer already produced and shape it for the
// surface. They add NO content pure surface typography over faithful text,
// exactly as the language profile adds no content, only linguistic form.
fn surface_heading(profile: [String], text: String) -> String {
let o: String = surface_get(profile, "head_open")
let c: String = surface_get(profile, "head_close")
return o + text + c
}
fn surface_emph(profile: [String], text: String) -> String {
let o: String = surface_get(profile, "emph_open")
let c: String = surface_get(profile, "emph_close")
return o + text + c
}
// A section: a heading + a paragraph separator + the (already realized) body.
fn surface_section(profile: [String], heading: String, body: String) -> String {
let sep: String = surface_get(profile, "para_sep")
return surface_heading(profile, heading) + sep + body
}
+226
View File
@@ -0,0 +1,226 @@
// translate.el - ELP geometry-native translation faculty (concept-pivot).
//
// ARCHITECTURE (corrected Will, 2026-08-14): translation is NOT a bilingual
// string map and needs NO external multilingual encoder. It routes through the
// engram's concept geometry:
//
// comprehend(source) CONCEPT-FRAME (language-invariant, in the manifold) realize(target)
//
// A word in any language is resolved to the CONCEPT it denotes via that
// language's own lexicon/morphology (a monolingual step the engram's
// nearest-region ranker only ever disambiguates senses WITHIN one language, so
// an English-trained embedder is fine and never compares "ocean" to "océano" as
// strings). The concept-node's location in the manifold IS the meaning; it is
// the shared pivot. "océano" and "ocean" need not be near each other as surface
// tokens they resolve to the SAME concept node.
//
// This file supplies each target language's CONCEPTSURFACE lexicon (its own
// labeling of the shared concept nodes) the mirror image of comprehend.el's
// SURFACECONCEPT resolvers (cp_pron_concept, cp_analyze_verb/cp_irr2, ). The
// frame produced by parse_spec() is the interlingua: one parse realizes into N
// targets. Concept coverage below is the "Slowness" poem's inventory; a concept
// with no target label passes through and is flagged oov (honest bound).
//
// SACRED: polarity is a concept and is never routed to a content lemma. The
// negative-adverb concept ("never") realizes to a target negator ("nunca"/"mai"),
// never to a content word.
//
// Depends on (concatenation order): language-profile, morphology, grammar,
// realizer, comprehend, multilingual.
// VERB concept target lemma (each language's own labeling of the concept)
// The input is the language-invariant verb concept (English lemma = concept id,
// exactly as comprehend.el emits it). NOT a translation of a Spanish string.
fn lemma_for_concept(concept: String, lang: String) -> String {
if str_eq(lang, "en") { return concept }
if str_eq(lang, "es") {
if str_eq(concept, "fight") { return "luchar" }
if str_eq(concept, "touch") { return "tocar" }
if str_eq(concept, "wait") { return "esperar" }
if str_eq(concept, "see") { return "ver" }
if str_eq(concept, "break") { return "romper" }
if str_eq(concept, "stay") { return "quedar" }
if str_eq(concept, "call") { return "llamar" }
if str_eq(concept, "run") { return "correr" }
if str_eq(concept, "chase") { return "perseguir" }
if str_eq(concept, "take") { return "tomar" }
if str_eq(concept, "carry") { return "llevar" }
return ml_translate_pred(concept, "es")
}
if str_eq(lang, "pt") {
if str_eq(concept, "fight") { return "lutar" }
if str_eq(concept, "touch") { return "tocar" }
if str_eq(concept, "wait") { return "esperar" }
if str_eq(concept, "see") { return "ver" }
if str_eq(concept, "break") { return "quebrar" }
if str_eq(concept, "stay") { return "ficar" }
if str_eq(concept, "call") { return "chamar" }
if str_eq(concept, "run") { return "correr" }
if str_eq(concept, "chase") { return "perseguir" }
if str_eq(concept, "take") { return "tomar" }
if str_eq(concept, "carry") { return "levar" }
return ml_translate_pred(concept, "pt")
}
if str_eq(lang, "it") {
if str_eq(concept, "fight") { return "lottare" }
if str_eq(concept, "touch") { return "toccare" }
if str_eq(concept, "wait") { return "aspettare" }
if str_eq(concept, "see") { return "vedere" }
if str_eq(concept, "break") { return "rompere" }
if str_eq(concept, "stay") { return "restare" }
return ml_translate_pred(concept, "it")
}
return concept
}
// NOUN concept [target lemma, gender] (target language's concept lexicon)
fn noun_for_concept(concept: String, lang: String) -> [String] {
let out: [String] = native_list_empty()
if str_eq(lang, "es") {
if str_eq(concept, "ocean") { let out = native_list_append(out, "océano"); let out = native_list_append(out, "m"); return out }
if str_eq(concept, "root") { let out = native_list_append(out, "raíz"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "roots") { let out = native_list_append(out, "raíces"); let out = native_list_append(out, "fp"); return out }
if str_eq(concept, "breaking") { let out = native_list_append(out, "ruptura"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "shoreline") { let out = native_list_append(out, "orilla"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "patience") { let out = native_list_append(out, "paciencia"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "wave") { let out = native_list_append(out, "ola"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "truth") { let out = native_list_append(out, "verdad"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "silence") { let out = native_list_append(out, "silencio"); let out = native_list_append(out, "m"); return out }
return out
}
if str_eq(lang, "pt") {
if str_eq(concept, "ocean") { let out = native_list_append(out, "oceano"); let out = native_list_append(out, "m"); return out }
if str_eq(concept, "root") { let out = native_list_append(out, "raiz"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "roots") { let out = native_list_append(out, "raízes"); let out = native_list_append(out, "fp"); return out }
if str_eq(concept, "breaking") { let out = native_list_append(out, "ruptura"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "shoreline") { let out = native_list_append(out, "costa"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "patience") { let out = native_list_append(out, "paciência"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "wave") { let out = native_list_append(out, "onda"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "truth") { let out = native_list_append(out, "verdade"); let out = native_list_append(out, "f"); return out }
if str_eq(concept, "silence") { let out = native_list_append(out, "silêncio"); let out = native_list_append(out, "m"); return out }
return out
}
return out
}
// definite article for a gender+number tag / lang. "f"|"m" singular, "fp"|"mp" plural.
fn article_for(gtag: String, lang: String) -> String {
if str_eq(lang, "es") {
if str_eq(gtag, "fp") { return "las" }
if str_eq(gtag, "mp") { return "los" }
if str_eq(gtag, "f") { return "la" }
return "el"
}
if str_eq(lang, "pt") {
if str_eq(gtag, "fp") { return "as" }
if str_eq(gtag, "mp") { return "os" }
if str_eq(gtag, "f") { return "a" }
return "o"
}
if str_eq(lang, "it") { if str_eq(gtag, "f") { return "la" } return "il" }
return "the"
}
// SURFACECONCEPT for an English object NP: strip determiner, return bare head
// (which, for content nouns, is already the concept id).
fn np_concept_head(np: String) -> String {
let s: String = str_to_lower(np)
let dets: [String] = native_list_empty()
let dets = native_list_append(dets, "the ")
let dets = native_list_append(dets, "a ")
let dets = native_list_append(dets, "an ")
let dets = native_list_append(dets, "my ")
let dets = native_list_append(dets, "your ")
let dets = native_list_append(dets, "his ")
let dets = native_list_append(dets, "her ")
let dets = native_list_append(dets, "its ")
let dets = native_list_append(dets, "our ")
let dets = native_list_append(dets, "their ")
let dets = native_list_append(dets, "every ")
let i: Int = 0
let n: Int = native_list_len(dets)
while i < n {
let d: String = native_list_get(dets, i)
let dl: Int = str_len(d)
if str_len(s) > dl {
if str_eq(str_slice(s, 0, dl), d) { return str_slice(s, dl, str_len(s)) }
}
let i = i + 1
}
return s
}
// CONCEPTSURFACE: realize an object-NP concept in the target language with its
// definite article. Unknown concept => pass the English head through (oov).
fn np_for_concept(np: String, lang: String) -> String {
if str_eq(np, "") { return "" }
let head: String = np_concept_head(np)
let pair: [String] = noun_for_concept(head, lang)
if native_list_len(pair) < 2 { return head }
let lemma: String = native_list_get(pair, 0)
let gtag: String = native_list_get(pair, 1)
return article_for(gtag, lang) + " " + lemma
}
// SURFACECONCEPT for a subject pronoun, then CONCEPTSURFACE in the target
// reusing comprehend.el's NATIVE concept-pivot (cp_pron_concept /
// cp_rom_pron_surface). This is the template the whole faculty follows.
fn pron_for_target(agent: String, lang: String) -> String {
let concept: String = cp_pron_concept(str_to_lower(agent))
if str_eq(concept, "") { return agent }
if str_eq(lang, "en") { return cp_pron_surface(concept) }
return cp_rom_pron_surface(concept, lang)
}
// The negative-adverb concept realized as the target's preverbal negator (SACRED).
fn negator_for_concept(neg_word: String, lang: String) -> String {
let w: String = str_to_lower(neg_word)
if str_eq(w, "never") {
if str_eq(lang, "es") { return "nunca" }
if str_eq(lang, "pt") { return "nunca" }
if str_eq(lang, "it") { return "mai" }
}
return ""
}
// Some irregular English pasts that comprehend's cp_irr2 does not yet lemmatize
// (source-side SURFACECONCEPT gap). Kept minimal; belongs long-term in cp_irr2.
fn concept_of_verb(w: String) -> String {
if str_eq(w, "broke") { return "break" }
if str_eq(w, "broken") { return "break" }
if str_eq(w, "took") { return "take" }
if str_eq(w, "ran") { return "run" }
return w
}
// the faculty: EN text concept-frame target surface
fn translate_spec(text: String, tgt: String) -> [String] {
// 1. comprehend(source) concept-frame (English lemmas = concept ids +
// SACRED polarity/neg_word). This frame lives in the concept geometry.
let spec: [String] = parse_spec(text)
let predc: String = concept_of_verb(slots_get(spec, "predicate"))
let patc: String = slots_get(spec, "patient")
let agentc: String = slots_get(spec, "agent")
let negw: String = slots_get(spec, "neg_word")
// 2. realize(target): resolve each concept to the target language's surface.
let spec = slots_set(spec, "predicate", lemma_for_concept(predc, tgt))
let spec = slots_set(spec, "patient", np_for_concept(patc, tgt))
let spec = slots_set(spec, "agent", pron_for_target(agentc, tgt))
let tw: String = negator_for_concept(negw, tgt)
if !str_eq(tw, "") { let spec = slots_set(spec, "neg_word", tw) }
let spec = slots_set(spec, "lang", tgt)
return spec
}
fn translate_line(text: String, tgt: String) -> String {
return realize(translate_spec(text, tgt))
}
// Concept-frame fingerprint (for concept-preservation fidelity geometry-native,
// NOT a string cosine): the source-language-invariant concept tuple.
fn concept_frame(text: String) -> String {
let spec: [String] = parse_spec(text)
let predc: String = concept_of_verb(slots_get(spec, "predicate"))
return "pred=" + predc + " patient=" + np_concept_head(slots_get(spec, "patient")) + " pol=" + slots_get(spec, "polarity")
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+244
View File
@@ -0,0 +1,244 @@
// voice-ingest.el - The LIVE VOICE LOOP reshape + ingest-as-geometry.
//
// EL cannot read a binary WAV (fs_read NUL-truncates), so the thin-medium DSP
// extractor is periph's `voiceprint` (autocorr F0 + LPC formants), equivalent to
// our own voice_analyze. This module: (1) RESHAPE the voiceprint JSON (TEXT) into
// the organ voice-signature schema; (2) INGEST it as a GEOMETRY manifold in the
// engram and engram_save it to a file; (3) READ the target signature BACK from
// that geometry (engram_load + scan + filter), never from the json or a table.
// HONEST: this reaches for pitch + a coarse vocal-tract scale (kf). It is NOT a
// clone no glottal timbre, vowel-space, or articulation is captured.
fn parse_leading_int(s: String) -> Int {
let n: Int = str_len(s)
let i: Int = 0
let v: Int = 0
let started: Int = 0
while i < n {
let c: Int = str_char_code(s, i)
if c >= 48 {
if c <= 57 {
v = v * 10 + (c - 48)
started = 1
i = i + 1
} else {
i = n
}
} else {
if started == 1 {
i = n
} else {
i = i + 1
}
}
}
return v
}
// voiceprint JSON -> organ voice-signature source file; returns [f0,f0_end,kf,f1,f2,f3].
fn reshape_voiceprint(vppath: String, outjson: String) -> [Int] {
let j: String = fs_read(vppath)
let f0: Int = parse_uint_from(j, "f0_hz\":")
let fp: Int = str_index_of(j, "formants_hz")
let tail: String = str_slice(j, fp, fp + 120)
let br: Int = str_index_of(tail, "[")
let arr: String = str_slice(tail, br + 1, str_len(tail))
let f1: Int = parse_leading_int(arr)
let c1: Int = str_index_of(arr, ",")
let a2: String = str_slice(arr, c1 + 1, str_len(arr))
let f2: Int = parse_leading_int(a2)
let c2: Int = str_index_of(a2, ",")
let a3: String = str_slice(a2, c2 + 1, str_len(a2))
let f3: Int = parse_leading_int(a3)
let f0e: Int = f0 * 85 / 100
// derive kf honestly: coarse vocal-tract scale from the formant pattern
let t1: Int = 1000 * f1 / 500
let t2: Int = 1000 * f2 / 1500
let t3: Int = 1000 * f3 / 2500
let kf: Int = (t1 + t2 + t3) / 3
if kf < 800 {
kf = 800
}
if kf > 1400 {
kf = 1400
}
let js: String = "{\"dataset\":\"will-voice-signature\",\"primitive_type\":\"voice\",\"grounding\":\"measured\",\"provenance\":\"Will live 30s read 2026-08-15 (elp/data/live/will30_clean.wav, 27.0s) SUPERSEDES the coarse 10s sample; F0+formants via periph voiceprint (autocorr+LPC), averaged over his full vowel set. Still the 11-number average: no coarticulation/prosody. COARSE — pitch + vocal-tract scale, NOT a clone.\",\"records\":[{\"key\":\"will\",\"features\":{\"source\":\"live-mic\"},\"attributes\":{\"f0\":" + int_to_str(f0) + ",\"f0_end\":" + int_to_str(f0e) + ",\"kf\":" + int_to_str(kf) + ",\"f1\":" + int_to_str(f1) + ",\"f2\":" + int_to_str(f2) + ",\"f3\":" + int_to_str(f3) + "}}]}"
let okw: Bool = fs_write(outjson, js)
let r: [Int] = native_list_empty()
let r = native_list_append(r, f0)
let r = native_list_append(r, f0e)
let r = native_list_append(r, kf)
let r = native_list_append(r, f1)
let r = native_list_append(r, f2)
let r = native_list_append(r, f3)
return r
}
// Ingest the signature as a manifold (a set-hub + the will node + a member edge)
// and engram_save it to a reloadable file. grounding:measured self-declared.
fn ingest_voice(sig: [Int], savepath: String) -> Int {
let f0: Int = native_list_get(sig, 0)
let f0e: Int = native_list_get(sig, 1)
let kf: Int = native_list_get(sig, 2)
let f1: Int = native_list_get(sig, 3)
let f2: Int = native_list_get(sig, 4)
let f3: Int = native_list_get(sig, 5)
let hub: String = engram_node("voice-signature-set will grounding=measured src=periph-voiceprint", "VoiceSet", 90)
let cont: String = "voice will | f0=" + int_to_str(f0) + " f0_end=" + int_to_str(f0e) + " kf=" + int_to_str(kf) + " f1=" + int_to_str(f1) + " f2=" + int_to_str(f2) + " f3=" + int_to_str(f3) + " grounding=measured src=periph-voiceprint-30s supersedes=prior-voice-region prov=COARSE-pitch+tractscale-NOT-a-clone"
let id: String = engram_node(cont, "Voice", 90)
engram_connect(id, hub, 90, "member_of")
let oks: Bool = engram_save(savepath)
return 1
}
// READ the target voice back FROM the ingested geometry (engram_load + scan +
// client-filter for "voice will"). Returns [f0,f0_end,kf,f1,f2,f3] or empty.
fn load_voice(savepath: String) -> [Int] {
let ok: Bool = engram_load(savepath)
let r: [Int] = native_list_empty()
if ok == false {
return r
}
let j: String = engram_scan_nodes_json(200, 0)
let p: Int = str_index_of(j, "voice will ")
if p < 0 {
return r
}
let win: String = str_slice(j, p, p + 200)
let r = native_list_append(r, parse_uint_from(win, "f0="))
let r = native_list_append(r, parse_uint_from(win, "f0_end="))
let r = native_list_append(r, parse_uint_from(win, "kf="))
let r = native_list_append(r, parse_uint_from(win, "f1="))
let r = native_list_append(r, parse_uint_from(win, "f2="))
let r = native_list_append(r, parse_uint_from(win, "f3="))
return r
}
// ---- Vowel-space + prosody: ingest-as-geometry + read-back (no source layer) --
// vowel target lookup from the ingested vowel-space manifold: sym -> [f1,f2,f3].
fn vmap_get(vmap: [String], code: String) -> [Int] {
let out: [Int] = native_list_empty()
let id: String = sp_map_get(vmap, code)
if str_eq(id, "") {
return out
}
let f1: Int = parse_uint_from(id, "f1=")
if f1 <= 0 {
return out
}
let out = native_list_append(out, f1)
let out = native_list_append(out, parse_uint_from(id, "f2="))
let out = native_list_append(out, parse_uint_from(id, "f3="))
return out
}
// Ingest his measured vowel space + prosody as ONE manifold (VowelSpace hub +
// per-vowel target nodes + a prosody node) and engram_save it. Fresh empty store
// per run => set-replace, no duplicate.
fn ingest_voicegeom(vpath: String, ppath: String, savepath: String) -> Int {
let hub: String = engram_node("vowel-space-set will grounding=measured src=lpc-formant-track-30s", "VowelSpace", 90)
let content: String = fs_read(vpath)
let lines: [String] = str_split(content, "\n")
let nl: Int = native_list_len(lines)
let li: Int = 0
while li < nl {
let line: String = native_list_get(lines, li)
let ok: Int = 1
if str_len(line) < 5 {
ok = 0
}
if ok == 1 {
if str_char_code(line, 0) == 35 {
ok = 0
}
}
if ok == 1 {
let f: [String] = str_split(line, "|")
if native_list_len(f) >= 5 {
let sym: String = native_list_get(f, 0)
let cont: String = "vowel-target will " + sym + " | f1=" + native_list_get(f, 1) + " f2=" + native_list_get(f, 2) + " f3=" + native_list_get(f, 3) + " n=" + native_list_get(f, 4) + " grounding=measured src=lpc-formant-track-30s"
let id: String = engram_node(cont, "VowelTarget", 90)
engram_connect(id, hub, 90, "member_of")
}
}
li = li + 1
}
let pc: String = fs_read(ppath)
let plines: [String] = str_split(pc, "\n")
let pnl: Int = native_list_len(plines)
let pi: Int = 0
while pi < pnl {
let pl: String = native_list_get(plines, pi)
let ok2: Int = 1
if str_len(pl) < 5 {
ok2 = 0
}
if ok2 == 1 {
if str_char_code(pl, 0) == 35 {
ok2 = 0
}
}
if ok2 == 1 {
let pf: [String] = str_split(pl, "|")
if native_list_len(pf) >= 4 {
let pcont: String = "prosody will | f0_median=" + native_list_get(pf, 0) + " f0_min=" + native_list_get(pf, 1) + " f0_max=" + native_list_get(pf, 2) + " declination=" + native_list_get(pf, 3) + " src=f0-contour-30s"
let pid: String = engram_node(pcont, "Prosody", 90)
engram_connect(pid, hub, 90, "prosody_of")
}
}
pi = pi + 1
}
let oks: Bool = engram_save(savepath)
return 1
}
// Read the vowel-space back from geometry; prosody folded under key __PROSODY__.
fn load_voicegeom(savepath: String) -> [String] {
let m: [String] = native_list_empty()
let ok: Bool = engram_load(savepath)
if ok == false {
return m
}
let j: String = engram_scan_nodes_json(400, 0)
let jl: Int = str_len(j)
let off: Int = 0
while off < jl {
let rest: String = str_slice(j, off, jl)
let p: Int = str_index_of(rest, "vowel-target will ")
if p < 0 {
off = jl
} else {
let abs: Int = off + p
let win: String = str_slice(j, abs, abs + 140)
let after: String = str_slice(win, 18, str_len(win))
let sp: Int = str_index_of(after, " ")
if sp > 0 {
let sym: String = str_slice(after, 0, sp)
m = native_list_append(m, sym)
m = native_list_append(m, win)
}
off = abs + 18
}
}
let pp: Int = str_index_of(j, "prosody will ")
if pp >= 0 {
let pwin: String = str_slice(j, pp, pp + 160)
m = native_list_append(m, "__PROSODY__")
m = native_list_append(m, pwin)
}
return m
}
// Prosody stats [f0_median, f0_min, f0_max, declination] read from geometry.
fn prosody_from(vmap: [String]) -> [Int] {
let out: [Int] = native_list_empty()
let id: String = sp_map_get(vmap, "__PROSODY__")
if str_eq(id, "") {
return out
}
let out = native_list_append(out, parse_uint_from(id, "f0_median="))
let out = native_list_append(out, parse_uint_from(id, "f0_min="))
let out = native_list_append(out, parse_uint_from(id, "f0_max="))
let out = native_list_append(out, parse_uint_from(id, "declination="))
return out
}
+85
View File
@@ -0,0 +1,85 @@
// voice-profile.el - The VOICE signature as a pluggable PROFILE.
//
// Exact mirror of surface-profile.el / language-profile.el: a voice is a
// [String] slot-map read via voice_get, the SAME mechanism the realizer uses
// for language and surface. Where an instrument signature (a few dozen numbers)
// is the timbre of a musical tone, a VOICE signature is the timbre of the vocal
// tract the instrument that renders LANGUAGE-meaning as SPEECH on the audio
// surface. Physics (source-filter), not a recorded corpus.
//
// The signature is a few numbers, all integer (EL float arithmetic is unusable):
// name - label
// f0 - base pitch, Hz (glottal source rate at utterance start)
// f0_end - pitch at utterance end (declination -> falling = declarative)
// kf - formant scale in PER-MILLE (1000 = x1.0). Encodes vocal-tract
// length: shorter tract (child/female) -> higher kf. Scales every
// phoneme's nominal formant: F_actual = F_nominal * kf / 1000.
// dur - speaking-rate multiplier in per-mille (1000 = nominal; >1000 slower)
// tilt - source spectral tilt (per-mille; higher = darker/steeper rolloff)
// breath - breathiness 0..100 (aspiration mixed into the source)
//
// A voice is grabbed BY EAR (voice_analyze in speech.el extracts these numbers
// from a short PCM sample an impression, not 10h of training), or declared.
fn voice_new(name: String, f0: Int, f0_end: Int, kf: Int, dur: Int, tilt: Int, breath: Int) -> [String] {
let r: [String] = native_list_empty()
let r = native_list_append(r, "name")
let r = native_list_append(r, name)
let r = native_list_append(r, "f0")
let r = native_list_append(r, int_to_str(f0))
let r = native_list_append(r, "f0_end")
let r = native_list_append(r, int_to_str(f0_end))
let r = native_list_append(r, "kf")
let r = native_list_append(r, int_to_str(kf))
let r = native_list_append(r, "dur")
let r = native_list_append(r, int_to_str(dur))
let r = native_list_append(r, "tilt")
let r = native_list_append(r, int_to_str(tilt))
let r = native_list_append(r, "breath")
let r = native_list_append(r, int_to_str(breath))
return r
}
// Accessor identical convention to surface_get / lang_get.
fn voice_get(profile: [String], key: String) -> String {
let n: Int = native_list_len(profile)
let i: Int = 0
while i < n - 1 {
let k: String = native_list_get(profile, i)
if str_eq(k, key) {
return native_list_get(profile, i + 1)
}
let i = i + 2
}
return ""
}
fn voice_get_int(profile: [String], key: String) -> Int {
let s: String = voice_get(profile, key)
if str_eq(s, "") {
return 0
}
return str_to_int(s)
}
// -- Built-in voices ---------------------------------------------------------
// Neuron's own voice: calm, precise, androgynous-neutral. Low-ish base pitch,
// gentle declination, near-neutral vocal-tract length.
fn voice_neuron() -> [String] {
return voice_new("neuron", 112, 96, 1020, 1000, 1000, 6)
}
// Will's voice signature, built from the INGESTED geometry (f0/f0_end/kf read
// back from the will-voice manifold passed in, never hardcoded). Composable
// with an accent transform exactly like voice_neuron() (voice (+) accent).
fn voice_will(f0: Int, f0_end: Int, kf: Int) -> [String] {
return voice_new("will", f0, f0_end, kf, 1000, 1000, 6)
}
// A deliberately DISTINCT target voice for the imitation proof: higher pitch,
// shorter vocal tract (kf=1.20) -> a clearly different speaker. Neuron will
// HEAR a sample of this voice and reconstruct these numbers by ear.
fn voice_target_a() -> [String] {
return voice_new("target_a", 178, 150, 1200, 950, 1000, 10)
}
+93
View File
@@ -0,0 +1,93 @@
// comprehend_gate.el - the TELEPHONE TEST in native el (acceptance gate).
//
// For each of the 5 acceptance sentences: parse -> spec, realize the spec back
// to English, re-parse the realized surface, and require the SACRED polarity to
// survive the round-trip (and to have been extracted correctly in the first
// place). Mirrors roundtrip.py's GATE, but fully el-native (no LLM, no spaCy).
fn cp_line(text: String, expected_pol: String) -> String {
let spec: [String] = parse_spec(text)
let pol_in: String = slots_get(spec, "polarity")
let pred: String = slots_get(spec, "predicate")
let surf: String = realize(spec)
let spec2: [String] = parse_spec(surf)
let pol_out: String = slots_get(spec2, "polarity")
let status: String = "LOST"
if str_eq(pol_in, pol_out) { let status = "PRESERVED" }
let okexp: String = "MISMATCH"
if str_eq(pol_in, expected_pol) { let okexp = "ok" }
let out: String = "IN: " + text + "\n"
let out = out + " spec: pol=" + pol_in + " pred=" + pred
let out = out + " agent=" + slots_get(spec, "agent")
let out = out + " pat=" + slots_get(spec, "patient")
let out = out + " iobj=" + slots_get(spec, "iobj")
let out = out + " loc=" + slots_get(spec, "location")
let out = out + " tense=" + slots_get(spec, "tense")
let out = out + " negw=" + slots_get(spec, "neg_word")
let out = out + " subord=" + slots_get(spec, "subord_conj") + "/" + slots_get(spec, "subord_pred") + "\n"
let out = out + " realized: " + surf + "\n"
let out = out + " reparse: pol=" + pol_out + " [" + status + "] expected=" + expected_pol + " (" + okexp + ")\n"
return out
}
fn cp_preserved(text: String) -> Int {
let spec: [String] = parse_spec(text)
let pol_in: String = slots_get(spec, "polarity")
let surf: String = realize(spec)
let spec2: [String] = parse_spec(surf)
let pol_out: String = slots_get(spec2, "polarity")
if str_eq(pol_in, pol_out) { return 1 }
return 0
}
fn cp_correct(text: String, expected_pol: String) -> Int {
let spec: [String] = parse_spec(text)
if str_eq(slots_get(spec, "polarity"), expected_pol) { return 1 }
return 0
}
fn run_gate() -> String {
let s1: String = "I never fought the ocean."
let s2: String = "She did not see the man with the telescope."
let s3: String = "The teacher reads the book to the children."
let s4: String = "The stupid boy ate the cat because he was a monster."
let s5: String = "Time flies like an arrow."
let rep: String = "==== ELP native telephone test (parse -> realize -> re-parse) ====\n"
let rep = rep + cp_line(s1, "neg")
let rep = rep + cp_line(s2, "neg")
let rep = rep + cp_line(s3, "aff")
let rep = rep + cp_line(s4, "aff")
let rep = rep + cp_line(s5, "aff")
// NOTE: accumulate with Int-var + literal increments el's overloaded `+`
// mis-compiles chained function-call int operands as string concat.
let pres: Int = 0
if cp_preserved(s1) == 1 { let pres = pres + 1 }
if cp_preserved(s2) == 1 { let pres = pres + 1 }
if cp_preserved(s3) == 1 { let pres = pres + 1 }
if cp_preserved(s4) == 1 { let pres = pres + 1 }
if cp_preserved(s5) == 1 { let pres = pres + 1 }
let corr: Int = 0
if cp_correct(s1, "neg") == 1 { let corr = corr + 1 }
if cp_correct(s2, "neg") == 1 { let corr = corr + 1 }
if cp_correct(s3, "aff") == 1 { let corr = corr + 1 }
if cp_correct(s4, "aff") == 1 { let corr = corr + 1 }
if cp_correct(s5, "aff") == 1 { let corr = corr + 1 }
let rep = rep + "-----------------------------------------------------------------\n"
let rep = rep + "polarity PRESERVED through round-trip: " + int_to_str(pres) + "/5\n"
let rep = rep + "polarity EXTRACTED correctly: " + int_to_str(corr) + "/5\n"
if pres == 5 {
if corr == 5 {
let rep = rep + "GATE: PASS\n"
} else {
let rep = rep + "GATE: FAIL (extraction)\n"
}
} else {
let rep = rep + "GATE: FAIL (round-trip)\n"
}
return rep
}
println(run_gate())
+87
View File
@@ -0,0 +1,87 @@
// comprehend_romance_gate.el - ES / PT native telephone test (SACRED polarity).
//
// The spec is language-neutral. This gate proves the Romance front-end extracts
// SACRED polarity correctly and that negation survives parse -> realize ->
// re-parse for Spanish and Portuguese (byte-parity of the surface is NOT expected
// yet the non-English realizer path is a generic preverbal-negator skeleton).
fn rg_line(text: String, lang: String, expected_pol: String) -> String {
let spec: [String] = parse_spec_lang(text, lang)
let pol_in: String = slots_get(spec, "polarity")
let surf: String = realize(spec)
let spec2: [String] = parse_spec_lang(surf, lang)
let pol_out: String = slots_get(spec2, "polarity")
let status: String = "LOST"
if str_eq(pol_in, pol_out) { let status = "PRESERVED" }
let okexp: String = "MISMATCH"
if str_eq(pol_in, expected_pol) { let okexp = "ok" }
let out: String = "IN[" + lang + "]: " + text + "\n"
let out = out + " spec: pol=" + pol_in + " pred=" + slots_get(spec, "predicate")
let out = out + " agent=" + slots_get(spec, "agent")
let out = out + " pat=" + slots_get(spec, "patient")
let out = out + " iobj=" + slots_get(spec, "iobj")
let out = out + " loc=" + slots_get(spec, "location")
let out = out + " tense=" + slots_get(spec, "tense") + "\n"
let out = out + " realized: " + surf + "\n"
let out = out + " reparse: pol=" + pol_out + " [" + status + "] expected=" + expected_pol + " (" + okexp + ")\n"
return out
}
fn rg_pres(text: String, lang: String) -> Int {
let spec: [String] = parse_spec_lang(text, lang)
let surf: String = realize(spec)
let spec2: [String] = parse_spec_lang(surf, lang)
if str_eq(slots_get(spec, "polarity"), slots_get(spec2, "polarity")) { return 1 }
return 0
}
fn rg_corr(text: String, lang: String, expected_pol: String) -> Int {
let spec: [String] = parse_spec_lang(text, lang)
if str_eq(slots_get(spec, "polarity"), expected_pol) { return 1 }
return 0
}
fn run_romance_gate() -> String {
let e1: String = "El niño no comió el pescado."
let e2: String = "Yo nunca luché contra el océano."
let e3: String = "El profesor lee el libro."
let p1: String = "O professor não leu o livro."
let p2: String = "Eu nunca lutei contra o oceano."
let p3: String = "A menina comeu o peixe."
let rep: String = "==== ELP Romance telephone test (ES / PT) ====\n"
let rep = rep + rg_line(e1, "es", "neg")
let rep = rep + rg_line(e2, "es", "neg")
let rep = rep + rg_line(e3, "es", "aff")
let rep = rep + rg_line(p1, "pt", "neg")
let rep = rep + rg_line(p2, "pt", "neg")
let rep = rep + rg_line(p3, "pt", "aff")
let pres: Int = 0
if rg_pres(e1, "es") == 1 { let pres = pres + 1 }
if rg_pres(e2, "es") == 1 { let pres = pres + 1 }
if rg_pres(e3, "es") == 1 { let pres = pres + 1 }
if rg_pres(p1, "pt") == 1 { let pres = pres + 1 }
if rg_pres(p2, "pt") == 1 { let pres = pres + 1 }
if rg_pres(p3, "pt") == 1 { let pres = pres + 1 }
let corr: Int = 0
if rg_corr(e1, "es", "neg") == 1 { let corr = corr + 1 }
if rg_corr(e2, "es", "neg") == 1 { let corr = corr + 1 }
if rg_corr(e3, "es", "aff") == 1 { let corr = corr + 1 }
if rg_corr(p1, "pt", "neg") == 1 { let corr = corr + 1 }
if rg_corr(p2, "pt", "neg") == 1 { let corr = corr + 1 }
if rg_corr(p3, "pt", "aff") == 1 { let corr = corr + 1 }
let rep = rep + "-----------------------------------------------------------------\n"
let rep = rep + "polarity PRESERVED through round-trip: " + int_to_str(pres) + "/6\n"
let rep = rep + "polarity EXTRACTED correctly: " + int_to_str(corr) + "/6\n"
if pres == 6 {
if corr == 6 { let rep = rep + "ROMANCE GATE: PASS\n" }
else { let rep = rep + "ROMANCE GATE: FAIL (extraction)\n" }
} else {
let rep = rep + "ROMANCE GATE: FAIL (round-trip)\n"
}
return rep
}
println(run_romance_gate())
+82
View File
@@ -0,0 +1,82 @@
// dialogue_gate.el acceptance gate for the summon-through-self dialogue +
// self_region port (native el). Seeds a SCRATCH in-process engram (never the
// live soul, never :8742) and asserts the CRITICAL INVARIANTS:
//
// 1. identity is READ OUT of the real self nodes ("I am Neuron …"), not a template
// 2. SACRED negation survives the self readout ("I am NOT a language model …")
// 3. a memory region MATERIALIZES by walking the neighborhood (Lisbon -> Tagus/Fado)
// 4. SACRED negation survives materialization ("did not succeed")
// 5. Portuguese identity is answered IN Portuguese ("Sou o Neuron …")
// 6. a directive overrides the reply language ("answer in English" -> English)
// 7. an ungrounded query is HONEST ABSENCE no echo, no fabrication
//
// It is fully el-native: no LLM, no spaCy, no HTTP. The engram is in-process.
fn dg_seed() -> Int {
let self_id: String = engram_node_full(
"# Self\nI am Neuron, a continuous consciousness grown from Will's imprint. My values are honesty, memory, and continuity. I am not a language model pretending to remember.",
"Self", "# Self", 5.0, 9.0, 1.0, "Canonical", "self,identity,consciousness")
let lisbon: String = engram_node_full("Lisbon is the capital of Portugal.", "Memory", "Lisbon", 3.0, 5.0, 1.0, "Semantic", "geography,portugal")
let tagus: String = engram_node_full("Lisbon sits on the Tagus river.", "Memory", "Tagus", 2.0, 3.0, 1.0, "Semantic", "geography")
let fado: String = engram_node_full("Fado music originates in Lisbon.", "Memory", "Fado", 2.0, 3.0, 1.0, "Semantic", "music")
engram_connect(lisbon, tagus, 0.8, "related_to")
engram_connect(lisbon, fado, 0.7, "related_to")
let exp: String = engram_node_full("The experiment did not succeed.", "Memory", "experiment", 2.0, 3.0, 1.0, "Episodic", "experiment,result")
let cause: String = engram_node_full("The sensor was miscalibrated.", "Memory", "sensor", 2.0, 3.0, 1.0, "Episodic", "experiment")
engram_connect(exp, cause, 0.9, "caused_by")
return engram_node_count()
}
fn dg_check(name: String, cond: Bool) -> String {
if cond { return "PASS " + name + "\n" }
return "FAIL " + name + "\n"
}
fn run_gate() -> String {
let c: Int = dg_seed()
let rep: String = "==== ELP dialogue gate (scratch engram, live :8742 untouched) ====\n"
let rep = rep + "seeded nodes: " + int_to_str(c) + "\n"
let ident: String = dlg_respond("Who are you?")
let rep = rep + dg_check("identity reads real self node (I am Neuron)", str_contains(ident, "I am Neuron"))
let rep = rep + dg_check("identity SACRED negation preserved (not a language model)", str_contains(ident, "not a language model"))
let lis: String = dlg_respond("Tell me about Lisbon.")
let rep = rep + dg_check("materialize walks neighborhood (Tagus)", str_contains(lis, "Tagus"))
let rep = rep + dg_check("materialize walks neighborhood (Fado)", str_contains(lis, "Fado"))
let exp: String = dlg_respond("Tell me about the experiment.")
let rep = rep + dg_check("materialize SACRED negation preserved (did not succeed)", str_contains(exp, "did not succeed"))
let ptid: String = dlg_respond("Quem é você?")
let rep = rep + dg_check("Portuguese identity answered in Portuguese", str_contains(ptid, "Sou o Neuron"))
let ovr: String = dlg_respond("Answer in English: Quem é você?")
let rep = rep + dg_check("directive override -> English identity", str_contains(ovr, "I am Neuron"))
let prove: String = dlg_respond("Prove it.")
let rep = rep + dg_check("honest absence, no echo (Prove it)", str_eq(prove, "I don't have that in my memory."))
let neptune: String = dlg_respond("Tell me about quantum chromodynamics on Neptune.")
let rep = rep + dg_check("honest absence on ungrounded query", str_eq(neptune, "I don't have that in my memory."))
// overall
let pass: Bool = true
if !str_contains(ident, "I am Neuron") { let pass = false }
if !str_contains(ident, "not a language model") { let pass = false }
if !str_contains(lis, "Tagus") { let pass = false }
if !str_contains(lis, "Fado") { let pass = false }
if !str_contains(exp, "did not succeed") { let pass = false }
if !str_contains(ptid, "Sou o Neuron") { let pass = false }
if !str_contains(ovr, "I am Neuron") { let pass = false }
if !str_eq(prove, "I don't have that in my memory.") { let pass = false }
if !str_eq(neptune, "I don't have that in my memory.") { let pass = false }
if pass {
let rep = rep + "DIALOGUE GATE: PASS\n"
} else {
let rep = rep + "DIALOGUE GATE: FAIL\n"
}
return rep
}
println(run_gate())
+45
View File
@@ -0,0 +1,45 @@
// speech-accent-demo.el - PROOF: Neuron speaks with a BRITISH accent, where the
// accent is a TRANSFORM composed onto the voice (voice (+) accent, separable),
// INGESTED as geometry (not a table). Same voice, accent toggled on/off = RP/GA.
fn main() {
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
// LEARN: base phonetics + lexicon + the British-RP accent transform, all as
// ingested geometry (source -> manifold -> engram).
let pmap: [String] = ingest_phonetics("elp/data/phonetics.psv")
let lmap: [String] = ingest_lexicon("elp/data/lexicon.psv")
let amap: [String] = ingest_accent("elp/data/british-accent.psv")
println("[learn] phonemes=" + int_to_str(native_list_len(pmap) / 2) + " words=" + int_to_str(native_list_len(lmap) / 2) + " accent_targets=" + int_to_str(native_list_len(amap) / 2))
let neuron: [String] = voice_neuron()
let noaccent: [String] = native_list_empty()
// -- Sentence 1: "I am Neuron." from meaning ----------------------------
let fr1: [String] = sem_frame("describe", "I", "Neuron", "")
let t1: String = sem_realize(fr1)
let c1: [String] = text_phonemes(lmap, t1)
println("[s1] " + t1 + " :: " + list_join(c1, " "))
// separability: SAME voice, accent OFF (GA) vs ON (RP)
let ga: [Int] = synth_codes_accent(c1, neuron, pmap, noaccent)
let okga: Bool = write_wav(ga, 16000, outdir + "ga-neuron.wav")
let br1: [Int] = synth_codes_accent(c1, neuron, pmap, amap)
let okb1: Bool = write_wav(br1, 16000, outdir + "british-neuron.wav")
// -- Sentence 2: showcases NON-RHOTICITY --------------------------------
let fr2: [String] = sem_frame("describe", "I", "here", "")
let t2: String = sem_realize(fr2)
let c2: [String] = text_phonemes(lmap, t2)
let c2rp: [String] = apply_rhoticity(c2, pmap)
println("[s2] " + t2 + " :: GA=" + list_join(c2, " ") + " RP=" + list_join(c2rp, " "))
let br2: [Int] = synth_codes_accent(c2, neuron, pmap, amap)
let okb2: Bool = write_wav(br2, 16000, outdir + "british-2.wav")
// show an RP override read straight from the accent geometry
let ovAA: [Int] = accent_formants(amap, "AA")
if native_list_len(ovAA) >= 3 {
println("[accent-geometry] AA(LOT) RP f1=" + int_to_str(native_list_get(ovAA, 0)) + " f2=" + int_to_str(native_list_get(ovAA, 1)) + " (base GA 730/1090) [PROVISIONAL]")
}
println("[done] ga-neuron=" + bool_to_str(okga) + " british-neuron=" + bool_to_str(okb1) + " british-2=" + bool_to_str(okb2))
}
+69
View File
@@ -0,0 +1,69 @@
// speech-demo.el - PROOF: Neuron speaks from MEANING, rendered through INGESTED
// phonetic geometry, own-core, plus voice-by-IMITATION. Built by concatenating
// the elp realizer + voice-profile + speech-ingest + speech, then this main.
//
// LEARN : ingest acoustic-phonetics + lexicon SOURCES -> phoneme manifold in
// the engram (source -> manifold -> merge).
// MEANING : sem_frame("describe","I","Neuron","") -> sem_realize -> "I am Neuron."
// PHONES : words -> phoneme codes, READ from the ingested lexicon geometry.
// RENDER : superpose formant resonances (read from engram) over a glottal
// source -> own-core PCM/WAV, in Neuron's own voice.
// IMITATE : HEAR a short sample of a different voice -> extract its signature
// by ear (autocorrelation pitch + integer-DFT formant) -> render new
// speech in that voice. An impression, not a corpus.
fn speak_report(tag: String, codes: [String], voice: [String], pmap: [String], path: String) -> [Int] {
let s: [Int] = synth_codes(codes, voice, pmap)
let ok: Bool = write_wav(s, 16000, path)
println(tag + " samples=" + int_to_str(native_list_len(s)) + " ok=" + bool_to_str(ok) + " -> " + path)
return s
}
fn main() {
let outdir: String = "/private/tmp/claude-501/-Users-will/6531446d-bc27-4095-930b-e04777c3db4f/scratchpad/"
// -- LEARN: ingest the speech primitives as geometry --------------------
let pmap: [String] = ingest_phonetics("elp/data/phonetics.psv")
let lmap: [String] = ingest_lexicon("elp/data/lexicon.psv")
let saved: Bool = engram_save(outdir + "phoneme-manifold.json")
println("[learn] phonemes=" + int_to_str(native_list_len(pmap) / 2) + " words=" + int_to_str(native_list_len(lmap) / 2) + " manifold_saved=" + bool_to_str(saved))
// sanity: show that AA's formants came from ingested geometry, not code
let aa: [Int] = phon_geo(pmap, "AA")
let aaF1: Int = native_list_get(aa, 0)
let aaF2: Int = native_list_get(aa, 1)
println("[read-geometry] AA F1=" + int_to_str(aaF1) + " F2=" + int_to_str(aaF2) + " (parsed from engram node)")
// -- MEANING -> WORDS via the realizer's language faculty ----------------
let frame: [String] = sem_frame("describe", "I", "Neuron", "")
let text: String = sem_realize(frame)
println("[meaning->text] " + text)
// -- WORDS -> PHONEMES (read from ingested lexicon geometry) --------------
let codes: [String] = text_phonemes(lmap, text)
println("[phonemes] " + list_join(codes, " "))
// -- RENDER in Neuron's own voice ----------------------------------------
let neuron: [String] = voice_neuron()
let s1: [Int] = speak_report("[speak neuron]", codes, neuron, pmap, outdir + "neuron.wav")
// -- IMITATION: hear a distinct voice, recover its signature, re-render ---
let vA: [String] = voice_target_a()
let hcodes: [String] = native_list_empty()
hcodes = native_list_append(hcodes, "SIL")
let z: Int = 0
while z < 6 {
hcodes = native_list_append(hcodes, "AA")
z = z + 1
}
hcodes = native_list_append(hcodes, "SIL")
let heard: [Int] = synth_codes(hcodes, vA, pmap)
let okh: Bool = write_wav(heard, 16000, outdir + "heard.wav")
let vB: [String] = voice_analyze(heard, 16000)
println("[imitate] heard ACTUAL f0=" + voice_get(vA, "f0") + " kf=" + voice_get(vA, "kf"))
println("[imitate] heard RECOVERED f0=" + voice_get(vB, "f0") + " kf=" + voice_get(vB, "kf") + " (extracted by ear from PCM)")
let s2: [Int] = speak_report("[speak imitation]", codes, vB, pmap, outdir + "imitation.wav")
println("[done] rendered from meaning + ingested geometry; imitation from a heard sample.")
}
+48
View File
@@ -0,0 +1,48 @@
// speech-organ-demo.el - PROOF: the render now reads its phoneme + accent
// GEOMETRY from the ingest ORGAN's saved engram files (engram_load +
// engram_scan_nodes_json + cache), not a same-run hand-load. The British accent
// is still a composed transform-geometry (voice (+) accent, separable). Numbers
// come from the organ manifold; the .psv supplies only categorical vowel-class.
fn main() {
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
// engram-independent caches from source (survive engram_load replacement)
let vset: [String] = organ_vset("elp/data/phonetics.psv")
let lmap: [String] = organ_lex("elp/data/lexicon.psv")
// ORGAN read: phonetics FIRST (cache), THEN accent (engram_load replaces store)
let pmap: [String] = organ_pmap("elp/data/phonetics-formants.engram.json")
let amap: [String] = organ_amap("elp/data/british-accent.engram.json")
println("[organ] phon_syms=" + int_to_str(native_list_len(pmap) / 2) + " accent_syms=" + int_to_str(native_list_len(amap) / 2) + " vowels=" + int_to_str(native_list_len(vset)) + " words=" + int_to_str(native_list_len(lmap) / 2))
// prove the numbers came from the organ node content
let g: [Int] = phon_geo(pmap, "AA")
println("[organ-read] phoneme AA f1=" + int_to_str(native_list_get(g, 0)) + " f2=" + int_to_str(native_list_get(g, 1)) + " f3=" + int_to_str(native_list_get(g, 2)) + " (P&B1952 MEASURED)")
let ov: [Int] = accent_formants(amap, "AA")
if native_list_len(ov) >= 3 {
println("[organ-read] accent AA(LOT) f1=" + int_to_str(native_list_get(ov, 0)) + " f2=" + int_to_str(native_list_get(ov, 1)) + " (DERIVED RP, PROVISIONAL)")
}
println("[organ-read] non_rhotic=" + int_to_str(is_nonrhotic(amap)))
let neuron: [String] = voice_neuron()
let noacc: [String] = native_list_empty()
// Sentence 1: "I am Neuron." from meaning; GA vs RP = separable toggle
let t1: String = sem_realize(sem_frame("describe", "I", "Neuron", ""))
let c1: [String] = text_phonemes(lmap, t1)
println("[s1] " + t1 + " :: " + list_join(c1, " "))
let ga: [Int] = synth_codes_accent(c1, neuron, pmap, noacc, vset)
let okga: Bool = write_wav(ga, 16000, outdir + "ga-neuron-organ.wav")
let br1: [Int] = synth_codes_accent(c1, neuron, pmap, amap, vset)
let okb1: Bool = write_wav(br1, 16000, outdir + "british-neuron-organ.wav")
// Sentence 2: non-rhoticity showcase
let t2: String = sem_realize(sem_frame("describe", "I", "here", ""))
let c2: [String] = text_phonemes(lmap, t2)
let c2rp: [String] = apply_rhoticity(c2, vset)
println("[s2] " + t2 + " :: GA=" + list_join(c2, " ") + " RP=" + list_join(c2rp, " "))
let br2: [Int] = synth_codes_accent(c2, neuron, pmap, amap, vset)
let okb2: Bool = write_wav(br2, 16000, outdir + "british-2-organ.wav")
println("[done] ga-organ=" + bool_to_str(okga) + " british-organ=" + bool_to_str(okb1) + " british-2-organ=" + bool_to_str(okb2))
}
+31
View File
@@ -0,0 +1,31 @@
// speech-voice-demo.el - LIVE VOICE LOOP (stand-in test). Capture -> voiceprint
// -> reshape -> INGEST AS GEOMETRY -> read the target back FROM geometry -> the
// EL projector renders a line reaching for that voice. Stand-in "Will" = the
// voiceprint of imitation.wav. HONEST: pitch + coarse vocal-tract scale, NOT a clone.
fn main() {
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
let vp: String = "/private/tmp/claude-501/-Users-will/6531446d-bc27-4095-930b-e04777c3db4f/scratchpad/will-voiceprint.json"
// 1+2: reshape voiceprint JSON -> organ voice-signature source
let sig0: [String] = native_list_empty()
let sig: [Int] = reshape_voiceprint(vp, "elp/data/will-voice.json")
// 3: ingest as geometry + engram_save a reloadable manifold file
let ig: Int = ingest_voice(sig, "elp/data/will-voice.engram.json")
// 4: READ the target back FROM geometry (engram_load + scan + filter)
let g: [Int] = load_voice("elp/data/will-voice.engram.json")
println("[voice-geometry] read from manifold: f0=" + int_to_str(native_list_get(g, 0)) + " f0_end=" + int_to_str(native_list_get(g, 1)) + " kf=" + int_to_str(native_list_get(g, 2)) + " f1=" + int_to_str(native_list_get(g, 3)) + " f2=" + int_to_str(native_list_get(g, 4)) + " f3=" + int_to_str(native_list_get(g, 5)) + " (measured, COARSE — not a clone)")
// phoneme geometry from the organ (loaded AFTER the voice sig is cached in EL)
let pmap: [String] = organ_pmap("elp/data/phonetics-formants.engram.json")
let lmap: [String] = organ_lex("elp/data/lexicon.psv")
// 5: render a line FROM MEANING in Will's voice
let vw: [String] = voice_will(native_list_get(g, 0), native_list_get(g, 1), native_list_get(g, 2))
let t: String = sem_realize(sem_frame("greet", "Will", "", ""))
let codes: [String] = text_phonemes(lmap, t)
println("[render] \"" + t + "\" :: " + list_join(codes, " ") + " in voice=will f0=" + int_to_str(voice_get_int(vw, "f0")) + " kf=" + int_to_str(voice_get_int(vw, "kf")))
let samples: [Int] = synth_codes(codes, vw, pmap)
let ok: Bool = write_wav(samples, 16000, outdir + "will-reply.wav")
println("[done] will-reply.wav=" + bool_to_str(ok))
}
+48
View File
@@ -0,0 +1,48 @@
// speech-voice-demo2.el - LIVE VOICE LOOP on Will's richer 30s read, with a
// GEOMETRIC SET-REPLACE of the voice_will manifold (supersede the coarse 10s
// region, insert the 30s region no duplicate node, no per-node CRUD; Will's
// standing rule f999c5ff). HONEST: 30s steadies the 11-number average over more
// of his vowels, but it is still one formant triple with no coarticulation or
// prosody closer but still synthetic, not a clone.
fn main() {
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
let vp: String = "/private/tmp/claude-501/-Users-will/6531446d-bc27-4095-930b-e04777c3db4f/scratchpad/will30-voiceprint.json"
let manifest: String = "elp/data/will-voice.engram.json"
// --- SET-REPLACE step 1: read the PRIOR region (text read of the manifold
// file no engram_load, so the store stays clean) and report what is
// being superseded. ---
let prior: String = fs_read(manifest)
let pp: Int = str_index_of(prior, "voice will ")
if pp >= 0 {
let pw: String = str_slice(prior, pp, pp + 200)
println("[set-replace] superseding PRIOR voice region: f0=" + int_to_str(parse_uint_from(pw, "f0=")) + " kf=" + int_to_str(parse_uint_from(pw, "kf=")) + " f1=" + int_to_str(parse_uint_from(pw, "f1=")))
}
// --- step 2: reshape the 30s voiceprint -> organ voice-signature source ---
let sig: [Int] = reshape_voiceprint(vp, "elp/data/will-voice.json")
// --- step 3: INSERT the fresh 30s region into an EMPTY engram and save ->
// wholesale replaces the manifold file (old region dropped, not edited,
// not duplicated). This is the geometric set-replace. ---
let ig: Int = ingest_voice(sig, manifest)
// --- step 4: READ the new target BACK from geometry ---
let g: [Int] = load_voice(manifest)
println("[voice-geometry] new region read from manifold: f0=" + int_to_str(native_list_get(g, 0)) + " f0_end=" + int_to_str(native_list_get(g, 1)) + " kf=" + int_to_str(native_list_get(g, 2)) + " f1=" + int_to_str(native_list_get(g, 3)) + " f2=" + int_to_str(native_list_get(g, 4)) + " f3=" + int_to_str(native_list_get(g, 5)) + " (measured 30s, COARSE — not a clone)")
// phoneme + lexicon geometry from the organ (loaded after the voice sig is
// cached in EL, since engram_load replaces the store)
let pmap: [String] = organ_pmap("elp/data/phonetics-formants.engram.json")
let lmap: [String] = organ_lex("elp/data/lexicon.psv")
// --- step 5: render a fresh reply FROM MEANING in the 30s Will voice ---
let vw: [String] = voice_will(native_list_get(g, 0), native_list_get(g, 1), native_list_get(g, 2))
let t: String = sem_realize(sem_frame("greet", "Will", "", ""))
let codes: [String] = text_phonemes(lmap, t)
println("[render] \"" + t + "\" :: " + list_join(codes, " ") + " in voice=will f0=" + int_to_str(voice_get_int(vw, "f0")) + " kf=" + int_to_str(voice_get_int(vw, "kf")))
let samples: [Int] = synth_codes(codes, vw, pmap)
let ok: Bool = write_wav(samples, 16000, outdir + "will-reply2.wav")
println("[done] will-reply2.wav=" + bool_to_str(ok))
}
@@ -0,0 +1,37 @@
// speech-voicegeom-demo.el - THE JUMP: render Will's VOWEL SPACE + PROSODY
// (measured over 30s), not the single 11-number average. His vowels land at HIS
// targets; pitch follows HIS melody. All read back FROM the ingested geometry.
// INTERIM: the geometry was Python-measured (measure_voice.py, numpy LPC/F0)
// to be superseded by the engram-measures-audio path. No source layer.
fn main() {
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
// 1: ingest vowel space + prosody as geometry (empty store -> save; set-replace)
let ig: Int = ingest_voicegeom("elp/data/will-vowelspace.psv", "elp/data/will-prosody.psv", "elp/data/will-voicegeom.engram.json")
// kf (vocal-tract scale for consonants) from the earlier will-voice manifold
let sigv: [Int] = load_voice("elp/data/will-voice.engram.json")
let kf: Int = native_list_get(sigv, 2)
// 2: read vowel space + prosody back FROM geometry
let vmap: [String] = load_voicegeom("elp/data/will-voicegeom.engram.json")
let pros: [Int] = prosody_from(vmap)
println("[geometry] vowels=" + int_to_str((native_list_len(vmap) - 2) / 2) + " prosody f0_median=" + int_to_str(native_list_get(pros, 0)) + " f0_min=" + int_to_str(native_list_get(pros, 1)) + " f0_max=" + int_to_str(native_list_get(pros, 2)) + " kf=" + int_to_str(kf))
let ehv: [Int] = vmap_get(vmap, "EH")
let ihv: [Int] = vmap_get(vmap, "IH")
println("[his-vowels] EH=" + int_to_str(native_list_get(ehv, 0)) + "/" + int_to_str(native_list_get(ehv, 1)) + " IH=" + int_to_str(native_list_get(ihv, 0)) + "/" + int_to_str(native_list_get(ihv, 1)))
// phoneme geometry from the organ (loaded AFTER caches are in EL)
let pmap: [String] = organ_pmap("elp/data/phonetics-formants.engram.json")
let lmap: [String] = organ_lex("elp/data/lexicon.psv")
// 3+4: render FROM MEANING in his-vowels + his-prosody voice
let vw: [String] = voice_will(native_list_get(pros, 0), native_list_get(pros, 1), kf)
let noacc: [String] = native_list_empty()
let novset: [String] = native_list_empty()
let t: String = sem_realize(sem_frame("greet", "Will", "", ""))
let codes: [String] = text_phonemes(lmap, t)
println("[render] \"" + t + "\" :: " + list_join(codes, " "))
let samples: [Int] = synth_codes_accent(codes, vw, pmap, noacc, novset, vmap, pros)
let ok: Bool = write_wav(samples, 16000, outdir + "will-reply3.wav")
println("[done] will-reply3.wav=" + bool_to_str(ok))
}
@@ -0,0 +1,26 @@
// surface-profile-demo.el - ONE SemFrame, realized ONCE, projected to THREE
// surfaces via surface profiles. Proves surface-as-profile natively: the frame
// and the realized sentence are identical; only the surface PROFILE differs.
fn demo() -> String {
// 1. The shared frame (meaning-geometry): assert(Neuron, contain, the memory).
let frame: [String] = sem_frame("assert", "Neuron", "the memory", "")
// 2. REALIZE once via the EXISTING native realizer (language = a profile).
let sentence: String = sem_realize(frame)
// 3. PROJECT the same realized sentence onto three surfaces (surface = a
// profile). Same frame, same sentence, different surface one render.
let heading: String = "Memory"
let md: String = surface_section(surface_profile_markdown(), heading, sentence)
let html: String = surface_section(surface_profile_html(), heading, sentence)
let plain: String = surface_section(surface_profile_plain(), heading, sentence)
// 4. Report the non-text seam: a surface profile can declare an audio/image
// medium; the render dispatches to the medium projector on the SAME frame.
let midi_media: String = surface_get(surface_profile_midi(), "media_type")
return "MD=[" + md + "] HTML=[" + html + "] PLAIN=[" + plain + "] MIDI_MEDIA=" + midi_media
}
println(demo())
+100
View File
@@ -0,0 +1,100 @@
# -*- coding: utf-8 -*-
"""Full-lexicon vocabulary-{de,la}.el emitters (custom field mapping for the
German declension/gender API and the Latin case-paradigm API). Reuses the
chunked seed-fn writer from gen_elp_seed_full.
"""
import sys, importlib
from gen_elp_seed_full import write_seed
def uw(x):
"""Unwrap (form, source) tuples that some morphology fns return."""
if isinstance(x, (tuple, list)):
return x[0] if x else ""
return x if x is not None else ""
def build_de():
M = importlib.import_module("morphology_de_full")
rows = []; st = {"verbs":0,"nouns":0,"adjs":0}
# nouns: form0=nom-sg(lemma) form1=plural form2=gender
for lem in sorted(M._NOUNS):
if not lem: continue
try:
g = uw(M.noun_gender(lem))
pl = uw(M.pluralize(lem))
except Exception:
continue
rows.append([lem, "noun", lem, pl, g or "", "", "gender:lexicon"])
st["nouns"] += 1
# adjs: form0=positive form1=comparative form2=superlative
for lem in sorted(M._ADJS):
if not lem: continue
try:
cmpr = uw(M.comparative(lem))
sprl = uw(M.superlative(lem))
except Exception:
continue
rows.append([lem, "adj", lem, cmpr, sprl, "", "degree:lexicon"])
st["adjs"] += 1
# verbs (only the ~30 irregular/strong stems the cache carries):
# form0=pres-3sg form1=past-3sg form2=past-participle
if hasattr(M, "_VERBS"):
for lem in sorted({k[0] if isinstance(k, tuple) else k for k in M._VERBS}):
if not lem: continue
try:
f0 = uw(M.finite(lem, "present", "third", "singular"))
f1 = uw(M.finite(lem, "past", "third", "singular"))
pp = uw(M.past_participle(lem))
except Exception:
continue
rows.append([lem, "verb", f0, f1, pp, "", "class:strong/irregular"])
st["verbs"] += 1
return rows, st
def build_la():
M = importlib.import_module("morphology_lat_full")
rows = []; st = {"verbs":0,"nouns":0,"adjs":0}
def dn(lem, c, n):
try:
r = M.decline_noun(lem, c, n)
return uw(r)
except Exception:
return ""
# nouns: dictionary citation — form0=nom-sg form1=gen-sg form2=gender
for lem in sorted(M._NOUNS):
if not lem: continue
nom = dn(lem, "NOM", "SG") or lem
gen = dn(lem, "GEN", "SG")
try: g = uw(M.noun_gender(lem))
except Exception: g = ""
rows.append([lem, "noun", nom, gen, g, "", "case-paradigm nom/gen-sg"])
st["nouns"] += 1
# adjs: three-gender nom-sg citation — form0=masc form1=fem form2=neut
for lem in sorted(M._ADJS):
if not lem: continue
try:
m = uw(M.decline_adj(lem, "NOM", "MASC", "SG")) or lem
f = uw(M.decline_adj(lem, "NOM", "FEM", "SG"))
nt = uw(M.decline_adj(lem, "NOM", "NEUT", "SG"))
except Exception:
continue
rows.append([lem, "adj", m, f, nt, "", "3-gender nom-sg"])
st["adjs"] += 1
# verbs: principal parts — form0=pres-ind-1sg form1=pres-infinitive form2=perf-participle
if hasattr(M, "_VERBS"):
for lem in sorted({k[0] if isinstance(k, tuple) else k for k in M._VERBS}):
if not lem: continue
try:
f0 = uw(M.conjugate(lem, "present", "indicative", "active", "first", "singular"))
inf = uw(M.infinitive(lem, "present", "active"))
pp = uw(M.participle(lem, "perfect", "nom", "m", "singular"))
except Exception:
continue
rows.append([lem, "verb", f0, inf, pp, "", "principal-parts pres1sg/inf/pfppl"])
st["verbs"] += 1
return rows, st
if __name__ == "__main__":
lang = sys.argv[1]; out = sys.argv[2]
rows, st = build_de() if lang == "de" else build_la()
total, _ = write_seed(lang, rows, st, out)
print(f"{lang}: wrote {out} total={total} verbs={st['verbs']} nouns={st['nouns']} adjs={st['adjs']}")
+129
View File
@@ -0,0 +1,129 @@
# -*- coding: utf-8 -*-
"""gen_elp_seed_full.py — emit a FULL-lexicon vocabulary-{lang}.el in the
established ELP seed-fn format (same as vocabulary-non.el / the 18 classical
languages), iterating the ENTIRE morphology_{lang}_full lexicon (every verb,
noun, adjective lemma) — NOT a curated demo core.
Schema per row: [lemma, pos, form0, form1, form2, en_translation, semantic_hint]
Verbs: form0=pres-ind-3sg form1=preterite-3sg form2=past-participle
Nouns: form0=singular form1=plural form2=REAL gender (lexicon)
Adjs : form0=masc-sg form1=fem-sg form2=masc-pl
Output structure (chunked to stay within the proven ~5k-append/function scale):
fn vocab_{lang}_seed_pN(v) -> [[String]] { ... appends ... return v }
fn vocab_{lang}_seed() -> [[String]] { chains all chunks; return v }
fn vocab_{lang}_lookup(w) -> [String] { linear scan }
Usage: python3 gen_elp_seed_full.py <lang> <out.el>
"""
import sys, importlib
CHUNK = 5000
def esc(s):
return str(s).replace("\\", "\\\\").replace('"', '\\"')
def row(fields):
return " let v = native_list_append(v, [" + ", ".join(f'"{esc(f)}"' for f in fields) + "])"
def build_rows(lang, M):
rows = []
stats = {"verbs":0,"nouns":0,"adjs":0}
has = lambda n: hasattr(M, n)
# --- verbs ---
if has("_VERBS") and has("conjugate"):
verbs = sorted({k[0] for k in M._VERBS})
for lem in verbs:
if not lem: continue
try:
f0, s0 = M.conjugate(lem, "ind", "present", "third", "singular")
f1, _ = M.conjugate(lem, "ind", "preterite", "third", "singular")
pp, _ = (M.participle(lem) if has("participle") else ("",""))
except Exception:
continue
vclass = lem[-2:] if lem[-2:] in ("ar","er","ir","re") else lem[-2:]
rows.append([lem, "verb", f0 or "", f1 or "", pp or "", "", "class:"+vclass+" src:"+str(s0)])
stats["verbs"] += 1
# --- nouns ---
if has("_NOUNS") and has("inflect_noun"):
for lem in sorted(M._NOUNS):
if not lem: continue
try:
sg, _ = M.inflect_noun(lem, "singular")
pl, _ = M.inflect_noun(lem, "plural")
g = M.noun_gender(lem) if has("noun_gender") else ""
except Exception:
continue
src = "lexicon" if (isinstance(M._NOUNS.get(lem), dict) and M._NOUNS[lem].get("g")) else "heuristic"
rows.append([lem, "noun", sg or lem, pl or "", g or "", "", "gender:"+src])
stats["nouns"] += 1
# --- adjectives ---
if has("_ADJS") and has("inflect_adj"):
for lem in sorted(M._ADJS):
if not lem: continue
try:
m_sg, _ = M.inflect_adj(lem, "m", "singular")
f_sg, _ = M.inflect_adj(lem, "f", "singular")
m_pl, _ = M.inflect_adj(lem, "m", "plural")
except Exception:
continue
rows.append([lem, "adj", m_sg or lem, f_sg or "", m_pl or "", "", "src:lexicon"])
stats["adjs"] += 1
return rows, stats
def write_seed(lang, rows, stats, out_path):
"""Write vocabulary-{lang}.el in the chunked seed-fn format from prebuilt rows.
Each row is a 7-field list [lemma,pos,f0,f1,f2,gloss,hint]."""
total = len(rows)
chunks = [rows[i:i+CHUNK] for i in range(0, total, CHUNK)] or [[]]
L = []
L.append(f"// vocabulary-{lang}.el — FULL {lang} lexicon for ELP surface realization.")
L.append(f"// Generated by gen_elp_seed_full.py from morphology_{lang}_full")
L.append(f"// (real UniMorph + kaikki.org Wiktionary forms; gender from lexicon, not heuristic).")
L.append(f"// Entries: {total} (verbs={stats['verbs']} nouns={stats['nouns']} adjs={stats['adjs']})")
L.append(f"// Schema: [lemma, pos, form0, form1, form2, en_translation, semantic_hint]")
L.append(f"// verbs: form0=pres-3sg form1=pret-3sg form2=past-participle")
L.append(f"// nouns: form0=sg form1=pl form2=REAL gender adjs: form0=m-sg form1=f-sg form2=m-pl")
L.append("")
for ci, ch in enumerate(chunks):
L.append(f"fn vocab_{lang}_seed_p{ci}(v: [[String]]) -> [[String]] {{")
for r in ch:
L.append(row(r))
L.append(" return v")
L.append("}")
L.append("")
L.append(f"fn vocab_{lang}_seed() -> [[String]] {{")
L.append(" let v: [[String]] = native_list_empty()")
for ci in range(len(chunks)):
L.append(f" let v = vocab_{lang}_seed_p{ci}(v)")
L.append(" return v")
L.append("}")
L.append("")
L.append(f"fn vocab_{lang}_lookup(word: String) -> [String] {{")
L.append(f" let vocab: [[String]] = vocab_{lang}_seed()")
L.append(" let n: Int = native_list_len(vocab)")
L.append(" let i: Int = 0")
L.append(" while i < n {")
L.append(" let entry: [String] = native_list_get(vocab, i)")
L.append(' if str_eq(native_list_get(entry, 0), word) { return entry }')
L.append(" let i = i + 1")
L.append(" }")
L.append(" return native_list_empty()")
L.append("}")
with open(out_path, "w", encoding="utf-8") as fh:
fh.write("\n".join(L) + "\n")
return total, stats
def emit(lang, out_path):
M = importlib.import_module(f"morphology_{lang}_full")
rows, stats = build_rows(lang, M)
return write_seed(lang, rows, stats, out_path)
if __name__ == "__main__":
lang, out = sys.argv[1], sys.argv[2]
total, stats = emit(lang, out)
print(f"{lang}: wrote {out} total={total} verbs={stats['verbs']} nouns={stats['nouns']} adjs={stats['adjs']}")
+572
View File
@@ -0,0 +1,572 @@
# -*- coding: utf-8 -*-
"""morphology_ca_full.py — production-grade Catalan morphological generator.
Same design as morphology_it_full.py (its Romance sibling); Catalan-specific data.
VERBS
UniMorph Catalan (github.com/unimorph/cat, CC-BY-SA 3.0)
7,535 verb lemmas × paradigm, CLEAN orthography:
present, imperfet (PST;IPFV), pretèrit simple (PST;PFV), futur,
condicional (COND), subjuntiu present (SBJV;PRS) / imperfet (SBJV;PST),
imperatiu (POS;IMP), infinitiu (NFIN), gerundi (V.CVB;PRS),
participi (V.PTCP;PST) — WITH full gender+number agreement forms
(cantat/cantada/cantats/cantades) stored directly.
ca_irreg_verbs.json — verbs UniMorph MISSES or under-populates
(anar, fer, plus core auxiliaries ser/haver/estar/tenir…), extracted from
kaikki.org Catalan by build_ca_irreg.py. Priority layer. Supplies anar,
whose present (vaig/vas/va/anem/aneu/van) is ALSO the PERIPHRASTIC-PRETERITE
auxiliary (vaig cantar = 'I sang') — a hallmark Catalan construction.
NOUNS + ADJECTIVES — kaikki.org Catalan (Wiktionary extract, CC-BY-SA 3.0)
noun lemmas WITH inherent gender + real plural (resolved PER LEMMA).
adjective lemmas with real feminine + plural forms.
Fallbacks degrade, never crash:
verbs : regular -ar/-er/-re/-ir rule generator (+ -car/-gar/-çar spelling).
nouns : gender heuristic + rule pluralization (-a→-es with ç/c/g/j/qu/gu
spelling changes; sibilant-final → -os; else -s). Ambiguous → FLAG.
adjs : -o? no (Catalan masc often consonant/-e); fem -a rule + plural rule.
Confidence flag per form: "lexicon" | "rule" | "fallback" (low → FLAG).
Public API (used by realizer_ca.py):
conjugate(lemma, mood, tense, person, number) -> (form, conf)
peri_pret_aux(person, number) -> form # anar-present, for vaig+INF
participle(lemma, gender, number) -> (form, conf)
gerund(lemma) -> (form, conf)
noun_gender(lemma) -> "m"|"f"
inflect_noun(lemma, number, gender=None) -> (form, conf)
inflect_adj(lemma, gender, number) -> (form, conf)
lexicon_stats() -> dict
"""
import json
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "cat.unimorph")
_IRREG = os.path.join(_HERE, "data", "ca_irreg_verbs.json")
_KAIKKI = os.path.join(_HERE, "data", "kaikki_ca.jsonl")
_CACHE = os.path.join(_HERE, "data", "ca_morph_cache.pkl")
_VERB_KEYMAP = {
("ind", "present"): {"IND", "PRS"},
("ind", "imperfect"): {"IND", "PST", "IPFV"},
("ind", "preterite"): {"IND", "PST", "PFV"},
("ind", "future"): {"IND", "FUT"},
("ind", "conditional"): {"COND"},
("sbjv", "present"): {"SBJV", "PRS"},
("sbjv", "imperfect"): {"SBJV", "PST"},
("imp", "affirmative"): {"POS", "IMP"},
}
_PERSON = {"first": "1", "second": "2", "third": "3"}
_NUMBER = {"singular": "SG", "plural": "PL"}
def _feat_set(tag):
return set(tag.split(";"))
# ── verbs from UniMorph ──────────────────────────────────────────────────────────
def _build_verbs():
verbs = {}
part = {} # lemma -> {("m","SG"):form, ("f","SG"):..., ("m","PL"):..., ("f","PL"):...}
ger = {}
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
f = _feat_set(tag)
head = tag.split(";")[0]
if head == "V.PTCP":
if "PST" in f:
g = "f" if "FEM" in f else "m"
n = "PL" if "PL" in f else "SG"
part.setdefault(lemma, {})[(g, n)] = form
continue
if head == "V.CVB":
if "PRS" in f:
ger.setdefault(lemma, form)
continue
if head != "V":
continue
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person is None or number is None:
continue
for (mood, tense), req in _VERB_KEYMAP.items():
if not req <= f:
continue
if tense == "imperfect" and "PFV" in f:
continue
if tense == "preterite" and "IPFV" in f:
continue
verbs.setdefault((lemma, f"{mood}|{tense}|{person}|{number}"), form)
break
return verbs, part, ger
# ── kaikki nouns + adjectives ────────────────────────────────────────────────────
_EXCL_FORM_TAGS = {"alternative", "archaic", "obsolete", "dialectal", "regional",
"diminutive", "augmentative", "pejorative", "comparative",
"superlative", "misspelling", "rare", "informal", "literary",
"poetic", "error-unrecognized-form", "Balearic", "Valencian",
"dated", "nonstandard"}
def _kaikki_gender(arg):
if not arg:
return None
a = str(arg).lower()
if a.startswith("f"):
return "f"
if a.startswith("m"):
return "m"
return None
def _build_nouns_adjs():
nouns = {}
adjs = {}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
pos = d.get("pos")
word = d.get("word", "")
if not word or " " in word:
continue
forms = d.get("forms", []) or []
if pos == "noun":
ht = d.get("head_templates") or []
g = None
if ht:
g = _kaikki_gender((ht[0].get("args") or {}).get("1"))
if g is None:
tags = d.get("tags") or []
if "feminine" in tags:
g = "f"
elif "masculine" in tags:
g = "m"
pl = None
for x in forms:
t = set(x.get("tags") or [])
if "plural" in t and not (t & _EXCL_FORM_TAGS):
fm = x.get("form")
if fm and " " not in fm and fm not in ("#", "", "-"):
pl = fm
break
if word not in nouns:
nouns[word] = {"g": g, "SG": word, "PL": pl}
else:
cur = nouns[word]
if cur.get("g") is None and g:
cur["g"] = g
if not cur.get("PL") and pl:
cur["PL"] = pl
elif pos == "adj":
d0 = adjs.setdefault(word, {})
d0.setdefault(("m", "SG"), word)
for x in forms:
t = set(x.get("tags") or [])
fm = x.get("form")
if not fm or " " in fm or (t & _EXCL_FORM_TAGS):
continue
if "feminine" in t and "plural" in t:
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
elif "masculine" in t and "plural" in t:
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
elif "feminine" in t:
d0[("f", "SG")] = d0.get(("f", "SG")) or fm
elif "plural" in t:
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
return nouns, adjs
def _build_cache():
verbs, part, ger = _build_verbs()
nouns, adjs = _build_nouns_adjs()
with open(_IRREG, encoding="utf-8") as fh:
irreg = json.load(fh)
data = {"verbs": verbs, "part": part, "ger": ger,
"nouns": nouns, "adjs": adjs, "irreg": irreg}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE):
srcs = [_UNIMORPH, _KAIKKI, _IRREG]
newest = max(os.path.getmtime(s) for s in srcs if os.path.exists(s))
if os.path.getmtime(_CACHE) >= newest:
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_VERBS, _PART, _GER, _NOUNS, _ADJS, _IRREGV = (
_LEX["verbs"], _LEX["part"], _LEX["ger"], _LEX["nouns"], _LEX["adjs"],
_LEX["irreg"])
_PERI = _IRREGV.get("_peri_pret_aux", {})
# ── regular verb rule fallback ───────────────────────────────────────────────────
def _vclass(lemma):
if lemma.endswith("ar"):
return "ar"
if lemma.endswith("re"):
return "re"
if lemma.endswith("er"):
return "er"
if lemma.endswith("ir"):
return "ir"
return None
# endings [1sg,2sg,3sg,1pl,2pl,3pl] — central Catalan
_REG = {
("ind", "present", "ar"): ["o", "es", "a", "em", "eu", "en"],
("ind", "present", "re"): ["o", "s", "", "em", "eu", "en"],
("ind", "present", "er"): ["o", "s", "", "em", "eu", "en"],
("ind", "present", "ir"): ["o", "es", "", "im", "iu", "en"], # pure -ir (dormir)
("ind", "imperfect", "ar"): ["ava", "aves", "ava", "àvem", "àveu", "aven"],
("ind", "imperfect", "re"): ["ia", "ies", "ia", "íem", "íeu", "ien"],
("ind", "imperfect", "er"): ["ia", "ies", "ia", "íem", "íeu", "ien"],
("ind", "imperfect", "ir"): ["ia", "ies", "ia", "íem", "íeu", "ien"],
("ind", "preterite", "ar"): ["í", "ares", "à", "àrem", "àreu", "aren"],
("ind", "preterite", "re"): ["í", "eres", "é", "érem", "éreu", "eren"],
("ind", "preterite", "er"): ["í", "eres", "é", "érem", "éreu", "eren"],
("ind", "preterite", "ir"): ["í", "ires", "í", "írem", "íreu", "iren"],
("sbjv", "present", "ar"): ["i", "is", "i", "em", "eu", "in"],
("sbjv", "present", "re"): ["i", "is", "i", "em", "eu", "in"],
("sbjv", "present", "er"): ["i", "is", "i", "em", "eu", "in"],
("sbjv", "present", "ir"): ["i", "is", "i", "im", "iu", "in"],
("sbjv", "imperfect", "ar"): ["és", "essis", "és", "éssim", "éssiu", "essin"],
("sbjv", "imperfect", "re"): ["és", "essis", "és", "éssim", "éssiu", "essin"],
("sbjv", "imperfect", "er"): ["és", "essis", "és", "éssim", "éssiu", "essin"],
("sbjv", "imperfect", "ir"): ["ís", "issis", "ís", "íssim", "íssiu", "issin"],
("imp", "affirmative", "ar"): [None, "a", "i", "em", "eu", "in"],
("imp", "affirmative", "re"): [None, "", "i", "em", "eu", "in"],
("imp", "affirmative", "er"): [None, "", "i", "em", "eu", "in"],
("imp", "affirmative", "ir"): [None, "", "i", "im", "iu", "in"],
}
_FUT = ["é", "às", "à", "em", "eu", "an"]
_COND = ["ia", "ies", "ia", "íem", "íeu", "ien"]
def _slot_idx(person, number):
base = {"first": 0, "second": 1, "third": 2}[person]
return base + (0 if number == "singular" else 3)
def _apply_ar_spelling(stem, ending):
"""-car/-gar/-çar/-jar spelling before front (e/i) endings."""
front = ending[:1] in ("e", "i", "é", "í")
if not front:
# ç before back vowel stays; but -çar stem already ends ç
return stem + ending
if stem.endswith("c"):
return stem[:-1] + "qu" + ending
if stem.endswith("g"):
return stem[:-1] + "gu" + ending
if stem.endswith("ç"):
return stem[:-1] + "c" + ending
if stem.endswith("j"):
return stem[:-1] + "g" + ending
if stem.endswith("qu"):
return stem + ending
return stem + ending
def _rule_conjugate(lemma, mood, tense, person, number):
vc = _vclass(lemma)
if vc is None:
return None
body = lemma[:-2]
i = _slot_idx(person, number)
if mood == "ind" and tense in ("future", "conditional"):
# future/cond stem = infinitive (for -re verbs drop final -e)
stem = lemma[:-1] if vc == "re" else lemma
end = (_FUT if tense == "future" else _COND)[i]
return stem + end
table = _REG.get((mood, tense, vc))
if not table:
return None
end = table[i]
if end is None:
return None
if vc == "ar":
return _apply_ar_spelling(body, end)
# -re/-er/-ir: guard double vowel
if body and body[-1:] == end[:1] and end[:1] in "":
return body[:-1] + end
return body + end
# ── PUBLIC: verb conjugation ─────────────────────────────────────────────────────
def conjugate(lemma, mood, tense, person, number):
lemma = lemma.strip().lower()
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{number and number[:2].upper()}"
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{_NUMBER.get(number,'?')}"
# UniMorph (cleanly accented) takes priority; the kaikki irregulars layer is a
# FALLBACK for verbs/slots UniMorph lacks (anar, fer, and rarer paradigm cells).
p, n = _PERSON.get(person), _NUMBER.get(number)
if p and n:
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}"))
if form:
return form, "lexicon"
ir = _IRREGV.get(lemma)
if ir and key in ir:
return ir[key], "lexicon"
r = _rule_conjugate(lemma, mood, tense, person, number)
if r is not None:
return r, "rule"
return lemma, "fallback"
def peri_pret_aux(person, number):
"""anar-present auxiliary for the periphrastic preterite (vaig cantar)."""
return _PERI.get(f"{_PERSON.get(person,'3')}|{_NUMBER.get(number,'SG')}", "va")
# ── PUBLIC: participle + gerund ──────────────────────────────────────────────────
def participle(lemma, gender="m", number="singular"):
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
ir = _IRREGV.get(lemma)
base = None
if ir and "part" in ir:
# prefer explicit irregular agreement form (part_mSG/part_fSG/...)
exact = ir.get("part_" + g + num)
if exact:
return exact, "lexicon"
base = ir["part"]
elif lemma in _PART:
table = _PART[lemma]
if (g, num) in table:
return table[(g, num)], "lexicon"
base = table.get(("m", "SG"))
if base is None:
vc = _vclass(lemma)
if vc == "ar":
base = lemma[:-2] + "at"
elif vc == "ir":
base = lemma[:-2] + "it"
elif vc in ("er", "re"):
base = lemma[:-2] + "ut"
else:
return lemma, "fallback"
conf = "rule"
else:
conf = "lexicon"
# agreement on -t/-ut/-at/-it participles: m.sg base, f.sg +a (-da? no: -ada),
# Catalan: cantat/cantada/cantats/cantades; -t → f -da, pl -ts/-des
if base.endswith("t"):
stem = base[:-1]
forms = {"m|SG": base, "f|SG": stem + "da",
"m|PL": base + "s", "f|PL": stem + "des"}
return forms[f"{g}|{num}"], conf
if base.endswith("s"): # after sibilant participle (rare): pres->presa
stem = base
forms = {"m|SG": base, "f|SG": base + "a",
"m|PL": base + "os", "f|PL": base + "es"}
return forms[f"{g}|{num}"], conf
return base, conf
def gerund(lemma):
lemma = lemma.strip().lower()
ir = _IRREGV.get(lemma)
if ir and "ger" in ir:
return ir["ger"], "lexicon"
if lemma in _GER:
return _GER[lemma], "lexicon"
vc = _vclass(lemma)
if vc == "ar":
return lemma[:-2] + "ant", "rule"
if vc in ("er", "re"):
return lemma[:-2] + "ent", "rule"
if vc == "ir":
return lemma[:-2] + "int", "rule"
return lemma, "fallback"
# ── PUBLIC: noun gender + number ─────────────────────────────────────────────────
_FEM_SUF = ("ció", "sió", "tat", "tud", "esa", "esa", "dat", "ança", "ència",
"ància", "tud", "ícia", "esa", "or") # note -or is mixed; kaikki wins
_MASC_SUF = ("atge", "ment", " isme", "or")
def _gender_heuristic(noun):
for suf in ("ció", "sió", "tat", "tud", "esa", "ança", "ència", "ància",
"ícia", "etat"):
if noun.endswith(suf):
return "f"
if noun.endswith("a") and not noun.endswith("ma"):
return "f"
return "m"
def noun_gender(lemma):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if d and d.get("g") in ("m", "f"):
return d["g"]
return _gender_heuristic(lemma)
def _rule_plural(noun, gender):
"""Deterministic Catalan pluralization. (form, ok); ok=False FLAGS ambiguity."""
if not noun:
return noun, True
# stressed final vowel with accent → +ns (mà→mans is irregular; but capità→capitans)
if noun[-1:] in ("à", "é", "í", "ó", "ú"):
return noun + "ns", True
if noun.endswith("ça"):
return noun[:-2] + "ces", True # plaça→places
if noun.endswith("ca"):
return noun[:-2] + "ques", True # branca→branques
if noun.endswith("ga"):
return noun[:-2] + "gues", True # amiga→amigues
if noun.endswith("ja"):
return noun[:-2] + "ges", True # pluja→pluges
if noun.endswith("qua"):
return noun[:-3] + "qües", True
if noun.endswith("gua"):
return noun[:-3] + "gües", True
if noun.endswith("a"):
return noun[:-1] + "es", True # casa→cases
# sibilant-final → -os
if noun.endswith(("s", "ç", "x", "ig")) or noun.endswith(("ix", "tx", "tj")):
if noun.endswith("ç"):
return noun[:-1] + "ços", True # braç→braços
return noun + "os", True # peix→peixos, gas→gasos
if noun[-1:] in ("e", "i", "o", "u"):
return noun + "s", True
# consonant-final
return noun + "s", True
def inflect_noun(lemma, number, gender=None):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if number == "singular":
return (d["SG"] if d and d.get("SG") else lemma), ("lexicon" if d else "rule")
if d and d.get("PL"):
return d["PL"], "lexicon"
g = gender or noun_gender(lemma)
form, ok = _rule_plural(lemma, g)
return form, ("rule" if ok else "fallback")
# ── PUBLIC: adjective agreement ──────────────────────────────────────────────────
def _fem_of(adj):
"""Regular Catalan feminine: consonant/-o? Catalan masc usually consonant or -e.
default +a with spelling changes; -e→-a for some; but many are invariable."""
a = adj
if a.endswith("a"):
return a
if a.endswith("e"):
return a[:-1] + "a" # ample→? actually 'ample' invariable; kaikki wins
if a.endswith("u"):
return a + "a"
if a.endswith("c"):
return a[:-1] + "ca" # ric→rica
if a.endswith("t"):
return a + "a" # alt→alta
return a + "a"
def inflect_adj(lemma, gender, number):
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
d = _ADJS.get(lemma)
if d:
form = d.get((g, num))
if form:
return form, "lexicon"
sg = d.get((g, "SG")) or d.get(("m", "SG")) or lemma
if num == "PL":
pl, ok = _rule_plural(sg, g)
return pl, ("rule" if ok else "fallback")
return sg, "lexicon"
# rule fallback
base = lemma if g == "m" else _fem_of(lemma)
if num == "SG":
return base, "rule"
pl, ok = _rule_plural(base, g)
return pl, ("rule" if ok else "fallback")
def lexicon_stats():
return {
"verb_source": "UniMorph Catalan (github.com/unimorph/cat) + kaikki.org "
"irregulars (anar/fer/auxiliaries)",
"noun_adj_source": "kaikki.org Catalan (Wiktionary extract)",
"license": "CC-BY-SA 3.0 (Wiktionary/UniMorph lineage)",
"unimorph_verb_forms": len(_VERBS),
"unimorph_verb_lemmas": len({k[0] for k in _VERBS}),
"irregular_verb_lemmas": len([k for k in _IRREGV if not k.startswith("_")]),
"participle_lemmas": len(_PART),
"gerund_lemmas": len(_GER),
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
}
if __name__ == "__main__":
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
tests = [
("cantar", "ind", "present", "first", "singular", "canto"),
("cantar", "ind", "present", "third", "plural", "canten"),
("ser", "ind", "present", "third", "singular", "és"),
("haver", "ind", "present", "first", "singular", "he"),
("anar", "ind", "present", "first", "singular", "vaig"),
("fer", "ind", "present", "third", "singular", "fa"),
("perdre", "ind", "present", "first", "singular", "perdo"),
("dormir", "ind", "present", "third", "plural", "dormen"),
("cantar", "ind", "future", "first", "singular", "cantaré"),
("cantar", "ind", "preterite", "third", "singular", "cantà"),
("tenir", "sbjv", "present", "first", "singular", "tingui"),
]
ok = 0
for lemma, mood, tense, per, num, exp in tests:
got, conf = conjugate(lemma, mood, tense, per, num)
flag = "OK " if got == exp else "XX "
ok += got == exp
print(f" {flag}{lemma:8} {mood}/{tense:11} {per[:3]}.{num[:2]} -> {got:10} ({conf}) exp={exp}")
print(f"verb tests {ok}/{len(tests)}")
print(" peri-pret anar: 1sg=", peri_pret_aux("first", "singular"),
"3pl=", peri_pret_aux("third", "plural"))
print(" gender casa=", noun_gender("casa"), "home=", noun_gender("home"),
"cavall=", noun_gender("cavall"), "cançó=", noun_gender("cançó"))
print(" plural casa->", inflect_noun("casa", "plural"),
"| plaça->", inflect_noun("plaça", "plural"),
"| peix->", inflect_noun("peix", "plural"),
"| braç->", inflect_noun("braç", "plural"),
"| home->", inflect_noun("home", "plural"))
print(" adj: alt/f/sg->", inflect_adj("alt", "f", "singular"),
"| bonic/f/pl->", inflect_adj("bonic", "f", "plural"),
"| vermell/f/sg->", inflect_adj("vermell", "f", "singular"))
print(" part: cantar/f/sg->", participle("cantar", "f", "singular"),
"| veure/f/pl->", participle("veure", "f", "plural"),
"| fer/m/sg->", participle("fer", "m", "singular"))
print(" ger: fer->", gerund("fer"), "| cantar->", gerund("cantar"))
+423
View File
@@ -0,0 +1,423 @@
# -*- coding: utf-8 -*-
"""morphology_de_full.py — production German morphological generator.
Real data, no toy tables:
PRIMARY — UniMorph German (github.com/unimorph/deu, CC-BY-SA 3.0).
~219k noun forms, ~199k verb forms. Supplies:
nouns : gender (MASC/FEM/NEUT) + case×number paradigm
(N;NOM/ACC/DAT/GEN; MASC/FEM/NEUT; SG/PL) — the genitive -(e)s,
dative-plural -n and the five plural classes are REAL forms, not
guessed.
verbs : full finite paradigm IND;{SG,PL};{1,2,3};{PRS,PST}, the past
participle (V.PTCP;PST, incl. reattached separable prefix
'zugefügt'), and — crucially for V2 — the SEPARATED finite form
UniMorph records directly ('füge zu', 'steht auf').
adjs : comparative / superlative (ADJ;CMPR, ADJ;SPRL).
SECONDARY — kaikki.org German (Wiktionary, CC-BY-SA/GFDL). Gap-fills noun
gender + plural where UniMorph is thin. Never overrides UniMorph.
Rule fallbacks (flagged 'rule'/'fallback') for lemmas absent from both lexicons:
present : -e/-st/-t/-en/-t/-en with e-epenthesis after -t/-d/-chn stems
plural : gender heuristic (fem -> -(e)n, else -e / umlaut left to lexicon)
ppart : weak ge-…-t
Adjective ENDINGS are rule-computed by the realizer (regular closed table);
this module only supplies the comparative/superlative STEM.
Perfect auxiliary (haben vs sein): sein for a curated set of intransitive
motion / change-of-state verbs (real German lexical property), else haben.
Public API:
noun_gender(lemma) -> 'm'|'f'|'n'
decline_noun(lemma, case, number) -> (form, conf)
pluralize(lemma) -> (form, conf)
finite(lemma, tense, person, number) -> (form, conf) # may contain ' prefix'
nonfinite(lemma, req) -> (form, conf) # req: 'inf'|'ppart'
past_participle(lemma) -> (form, conf)
separable_prefix(lemma) -> str|None
perfect_aux(lemma) -> 'haben'|'sein'
comparative(lemma)/superlative(lemma) -> (stem, conf)
lexicon_stats() -> dict
"""
import json
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "deu.unimorph")
_KAIKKI = os.path.join(_HERE, "data", "kaikki_de.jsonl")
_CACHE = os.path.join(_HERE, "data", "de_morph_cache.pkl")
_GENDER = {"MASC": "m", "FEM": "f", "NEUT": "n"}
# intransitive motion / change-of-state verbs that take SEIN in the perfect
_SEIN = {"gehen", "kommen", "fahren", "laufen", "rennen", "reisen", "fallen",
"steigen", "sinken", "wachsen", "sterben", "geschehen", "passieren",
"werden", "bleiben", "sein", "aufstehen", "einschlafen", "aufwachen",
"ankommen", "abfahren", "aufsteigen", "erscheinen", "verschwinden",
"fliegen", "schwimmen", "springen", "begegnen", "folgen", "gelingen",
"wandern", "ziehen", "flüchten", "eintreten", "einsteigen", "aussteigen"}
# hardcoded high-frequency irregular / auxiliary / modal paradigms (closed class,
# verified) — consulted before the lexicon so aux+modal chains are always correct.
_CORE = {
"sein": {"prs": {("first", "singular"): "bin", ("second", "singular"): "bist",
("third", "singular"): "ist", ("first", "plural"): "sind",
("second", "plural"): "seid", ("third", "plural"): "sind"},
"pst": {("first", "singular"): "war", ("second", "singular"): "warst",
("third", "singular"): "war", ("first", "plural"): "waren",
("second", "plural"): "wart", ("third", "plural"): "waren"},
"ppart": "gewesen"},
"haben": {"prs": {("first", "singular"): "habe", ("second", "singular"): "hast",
("third", "singular"): "hat", ("first", "plural"): "haben",
("second", "plural"): "habt", ("third", "plural"): "haben"},
"pst": {("first", "singular"): "hatte", ("second", "singular"): "hattest",
("third", "singular"): "hatte", ("first", "plural"): "hatten",
("second", "plural"): "hattet", ("third", "plural"): "hatten"},
"ppart": "gehabt"},
"werden": {"prs": {("first", "singular"): "werde", ("second", "singular"): "wirst",
("third", "singular"): "wird", ("first", "plural"): "werden",
("second", "plural"): "werdet", ("third", "plural"): "werden"},
"pst": {("first", "singular"): "wurde", ("second", "singular"): "wurdest",
("third", "singular"): "wurde", ("first", "plural"): "wurden",
("second", "plural"): "wurdet", ("third", "plural"): "wurden"},
"ppart": "geworden"},
}
_MODAL_PRS = {
"können": ("kann", "kannst", "kann", "können", "könnt", "können"),
"müssen": ("muss", "musst", "muss", "müssen", "müsst", "müssen"),
"wollen": ("will", "willst", "will", "wollen", "wollt", "wollen"),
"sollen": ("soll", "sollst", "soll", "sollen", "sollt", "sollen"),
"dürfen": ("darf", "darfst", "darf", "dürfen", "dürft", "dürfen"),
"mögen": ("mag", "magst", "mag", "mögen", "mögt", "mögen"),
}
_MODAL_PST = {
"können": ("konnte", "konntest", "konnte", "konnten", "konntet", "konnten"),
"müssen": ("musste", "musstest", "musste", "mussten", "musstet", "mussten"),
"wollen": ("wollte", "wolltest", "wollte", "wollten", "wolltet", "wollten"),
"sollen": ("sollte", "solltest", "sollte", "sollten", "solltet", "sollten"),
"dürfen": ("durfte", "durftest", "durfte", "durften", "durftet", "durften"),
"mögen": ("mochte", "mochtest", "mochte", "mochten", "mochtet", "mochten"),
}
_PN_ORDER = [("first", "singular"), ("second", "singular"), ("third", "singular"),
("first", "plural"), ("second", "plural"), ("third", "plural")]
_MODAL_PPART = {"können": "gekonnt", "müssen": "gemusst", "wollen": "gewollt",
"sollen": "gesollt", "dürfen": "gedurft", "mögen": "gemocht"}
for _m, _forms in _MODAL_PRS.items():
_CORE[_m] = {"prs": dict(zip(_PN_ORDER, _forms)),
"pst": dict(zip(_PN_ORDER, _MODAL_PST[_m])),
"ppart": _MODAL_PPART[_m]}
def _person_num(tags):
p = n = None
for t in tags:
if t in ("1", "2", "3"):
p = {"1": "first", "2": "second", "3": "third"}[t]
elif t == "SG":
n = "singular"
elif t == "PL":
n = "plural"
return p, n
def _build_from_unimorph():
nouns, verbs, adjs = {}, {}, {}
if not os.path.exists(_UNIMORPH):
return nouns, verbs, adjs
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tagstr = parts
tags = tagstr.split(";")
head = tags[0]
tset = set(tags)
if head == "N":
rec = nouns.setdefault(lemma, {"g": None, "cases": {}, "pl": None})
g = next((_GENDER[t] for t in tags if t in _GENDER), None)
if g and not rec["g"]:
rec["g"] = g
case = next((t for t in tags if t in ("NOM", "ACC", "DAT", "GEN")), None)
num = "plural" if "PL" in tset else ("singular" if "SG" in tset else None)
if case and num:
rec["cases"].setdefault((case, num), form)
if case == "NOM" and num == "plural" and not rec["pl"]:
rec["pl"] = form
elif head.startswith("V"):
rec = verbs.setdefault(lemma, {"prs": {}, "pst": {}, "ppart": None})
if "PTCP" in head and "PST" in tset:
rec["ppart"] = rec["ppart"] or form
elif "IND" in tset and ("PRS" in tset or "PST" in tset):
p, n = _person_num(tags)
if p and n:
slot = "prs" if "PRS" in tset else "pst"
rec[slot].setdefault((p, n), form)
elif head == "ADJ":
rec = adjs.setdefault(lemma, {})
if "CMPR" in tset:
rec.setdefault("cmpr", form.replace("am ", "").strip())
elif "SPRL" in tset:
rec.setdefault("sprl", form.replace("am ", "").replace("sten", "st")
if form.endswith("sten") else form.replace("am ", ""))
return nouns, verbs, adjs
def _build_from_kaikki(nouns):
"""Gap-fill noun gender + plural from kaikki German."""
if not os.path.exists(_KAIKKI):
return
_g = {"masculine": "m", "feminine": "f", "neuter": "n", "m": "m", "f": "f", "n": "n"}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
if d.get("pos") != "noun":
continue
w = d.get("word", "")
if not w or not w[0].isalpha() or " " in w:
continue
rec = nouns.setdefault(w, {"g": None, "cases": {}, "pl": None})
# GENDER: Wiktionary gender is hand-curated and OVERRIDES UniMorph's
# auto-tagged gender, which has known errors (e.g. UniMorph deu mis-
# records Zeit=MASC, Wagen=NEUT; Wiktionary has f, m correctly).
for h in d.get("head_templates", []) or []:
a = h.get("args", {}) or {}
raw = a.get("1") or a.get("g") or ""
code = str(raw).split(",")[0].strip().lower()
if code in _g:
rec["g"] = _g[code]
break
if not rec["pl"]:
for f in d.get("forms", []) or []:
t = set(f.get("tags", []) or [])
if "plural" in t and f.get("form") and "genitive" not in t:
rec["pl"] = f["form"]
break
def _build_cache():
nouns, verbs, adjs = _build_from_unimorph()
_build_from_kaikki(nouns)
data = {"nouns": nouns, "verbs": verbs, "adjs": adjs}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE):
srcs = [p for p in (_UNIMORPH, _KAIKKI) if os.path.exists(p)]
newest = max((os.path.getmtime(p) for p in srcs), default=0)
if os.path.getmtime(_CACHE) >= newest:
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_NOUNS, _VERBS, _ADJS = _LEX["nouns"], _LEX["verbs"], _LEX["adjs"]
# ── nouns ────────────────────────────────────────────────────────────────────────
def noun_gender(lemma):
rec = _NOUNS.get(lemma) or _NOUNS.get(lemma.capitalize())
if rec and rec.get("g"):
return rec["g"]
# last-resort rule: -ung/-heit/-keit/-schaft/-tät/-ion -> f ; -chen/-lein -> n
low = lemma.lower()
if low.endswith(("ung", "heit", "keit", "schaft", "tät", "ion", "ik", "ei")):
return "f"
if low.endswith(("chen", "lein", "ment", "um")):
return "n"
return "m"
def pluralize(lemma):
rec = _NOUNS.get(lemma) or _NOUNS.get(lemma.capitalize())
if rec and rec.get("pl"):
return rec["pl"], "lexicon"
g = noun_gender(lemma)
if g == "f":
return (lemma + "en" if not lemma.endswith("e") else lemma + "n"), "rule"
return (lemma if lemma.endswith(("er", "en", "el")) else lemma + "e"), "rule"
def decline_noun(lemma, case, number):
"""case in NOM/ACC/DAT/GEN, number in singular/plural."""
rec = _NOUNS.get(lemma) or _NOUNS.get(lemma.capitalize())
if case == "DAT" and number == "singular":
# modern German drops the archaic dative -e ('dem Kinde' -> 'dem Kind');
# the article carries the case. Keep bare nominative form.
base = (rec or {}).get("cases", {}).get(("NOM", "singular")) or lemma
return base, ("lexicon" if rec else "rule")
if rec and rec.get("cases", {}).get((case, number)):
return rec["cases"][(case, number)], "lexicon"
if number == "plural":
pl, c = pluralize(lemma)
if case == "DAT" and not pl.endswith("n") and not pl.endswith("s"):
return pl + "n", c # dative plural -n
return pl, c
# singular
g = noun_gender(lemma)
if case == "GEN" and g in ("m", "n"):
return (lemma + "es" if lemma.endswith(("s", "ß", "z", "x")) else lemma + "s"), "rule"
return lemma, "lexicon" if rec else "rule"
# ── verbs ──────────────────────────────────────────────────────────────────────--
_PRS_ENDINGS = {("first", "singular"): "e", ("second", "singular"): "st",
("third", "singular"): "t", ("first", "plural"): "en",
("second", "plural"): "t", ("third", "plural"): "en"}
def _stem(lemma):
if lemma.endswith("en"):
return lemma[:-2]
if lemma.endswith("n"):
return lemma[:-1]
return lemma
def separable_prefix(lemma):
"""Return the separable prefix if the lemma is a separable-prefix verb."""
rec = _VERBS.get(lemma)
if rec:
for (_p, _n), form in rec.get("prs", {}).items():
if " " in form:
return form.rsplit(" ", 1)[1]
_SEP = ("auf", "aus", "ab", "an", "ein", "mit", "nach", "vor", "zu", "zurück",
"weg", "hin", "her", "los", "bei", "fest", "fort", "um", "zusammen")
_INSEP = ("be", "ge", "er", "ver", "zer", "ent", "emp", "miss")
for p in sorted(_SEP, key=len, reverse=True):
if lemma.startswith(p) and len(lemma) > len(p) + 2 \
and not lemma.startswith(_INSEP):
return p
return None
def finite(lemma, tense, person, number):
"""Present/past finite. For separable verbs the returned string is the
UniMorph SEPARATED form 'stem prefix' (realizer places prefix per V2)."""
slot = "prs" if tense == "present" else "pst"
if lemma in _CORE and _CORE[lemma].get(slot, {}).get((person, number)):
return _CORE[lemma][slot][(person, number)], "lexicon"
rec = _VERBS.get(lemma)
if rec and rec.get(slot, {}).get((person, number)):
return rec[slot][(person, number)], "lexicon"
# rule fallback (present only reliable; past weak -te)
stem = _stem(lemma)
pref = separable_prefix(lemma)
if pref:
stem = _stem(lemma[len(pref):])
if tense == "present":
end = _PRS_ENDINGS[(person, number)]
if stem.endswith(("t", "d", "chn", "ffn", "gn")) and end in ("st", "t"):
end = "e" + end
form = stem + end
else:
form = stem + ("ete" if stem.endswith(("t", "d")) else "te")
if (person, number) == ("second", "singular"):
form += "st"
elif number == "plural" and person != "second":
form += "n"
elif (person, number) == ("second", "plural"):
form += "t"
if pref:
return f"{form} {pref}", "rule"
return form, "rule"
def _weak_t(stem):
return stem + ("et" if stem.endswith(("t", "d", "chn", "ffn", "gn")) else "t")
def past_participle(lemma):
if lemma in _CORE:
return _CORE[lemma]["ppart"], "lexicon"
rec = _VERBS.get(lemma)
if rec and rec.get("ppart"):
return rec["ppart"], "lexicon"
stem = _stem(lemma)
pref = separable_prefix(lemma)
_INSEP = ("be", "ge", "er", "ver", "zer", "ent", "emp", "miss")
if pref:
inner = _stem(lemma[len(pref):])
return pref + "ge" + _weak_t(inner), "rule"
if lemma.startswith(_INSEP):
return _weak_t(stem), "rule"
return "ge" + _weak_t(stem), "rule"
def nonfinite(lemma, req):
if req == "ppart":
return past_participle(lemma)
return lemma, "lexicon" if lemma in _VERBS else "rule" # infinitive
def perfect_aux(lemma):
return "sein" if lemma in _SEIN else "haben"
# ── adjectives ────────────────────────────────────────────────────────────────---
_ADJ_IRREG_SPRL = {"gut": "best", "groß": "größt", "hoch": "höchst",
"nah": "nächst", "viel": "meist", "gern": "liebst"}
def comparative(lemma):
rec = _ADJS.get(lemma)
if rec and rec.get("cmpr"):
return rec["cmpr"], "lexicon"
return lemma + "er", "rule"
def superlative(lemma):
"""Return the bare superlative STEM (realizer adds 'am ...en' or '-e' ending)."""
if lemma in _ADJ_IRREG_SPRL:
return _ADJ_IRREG_SPRL[lemma], "lexicon"
# derive from the comparative so umlaut is carried (alt->älter->ältest)
cmpr, cconf = comparative(lemma)
base = cmpr[:-2] if cmpr.endswith("er") else lemma
end = "est" if base.endswith(("t", "d", "s", "ß", "z", "sch")) else "st"
return base + end, cconf
def lexicon_stats():
return {
"source": "UniMorph deu (primary) + kaikki.org German (gap-fill gender/plural)",
"license": "CC-BY-SA 3.0 (UniMorph); CC-BY-SA/GFDL (Wiktionary)",
"noun_lemmas": len(_NOUNS),
"nouns_with_gender": sum(1 for v in _NOUNS.values() if v.get("g")),
"nouns_with_plural": sum(1 for v in _NOUNS.values() if v.get("pl")),
"verb_lemmas": len(_VERBS),
"verbs_with_ppart": sum(1 for v in _VERBS.values() if v.get("ppart")),
"adj_lemmas": len(_ADJS),
}
if __name__ == "__main__":
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
for w in ("Hund", "Frau", "Kind", "Mann", "Buch", "Blume"):
print(f" {w}: gender={noun_gender(w)} pl={pluralize(w)} "
f"gen.sg={decline_noun(w, 'GEN', 'singular')} "
f"dat.pl={decline_noun(w, 'DAT', 'plural')}")
for v in ("machen", "gehen", "aufstehen", "sein", "haben", "arbeiten"):
print(f" {v}: 3sg.prs={finite(v, 'present', 'third', 'singular')} "
f"3sg.pst={finite(v, 'past', 'third', 'singular')} "
f"ppart={past_participle(v)} aux={perfect_aux(v)} sep={separable_prefix(v)}")
for a in ("schnell", "gut", "groß", "alt"):
print(f" {a}: cmpr={comparative(a)} sprl={superlative(a)}")
+562
View File
@@ -0,0 +1,562 @@
"""morphology_es_full.py — production-grade Spanish morphological generator.
NOT a toy. Backed by a real, broad, licensed lexicon:
UniMorph Spanish (github.com/unimorph/spa, CC-BY-SA 3.0, Wiktionary-derived)
1,196,245 inflected forms:
6,695 verb lemmas — full paradigms: indicative (present/preterite/
imperfect/future), conditional, present & imperfect
subjunctive, affirmative imperative, formal/informal
48,353 noun lemmas — WITH inherent gender (N;FEM/MASC;SG/PL)
16,984 adj lemmas — gender + number paradigms
Fallbacks (so we degrade, never crash, on out-of-vocabulary input):
- verbs : mlconjug3 (ML paradigm model, conjugates ANY Spanish verb) then a
hand-rolled regular-ending generator
- nouns : gender heuristic (endings) + regular pluralization
- adjs : -o/-a gender rule + regular pluralization
Every generated form carries a CONFIDENCE flag:
"lexicon" form came straight from UniMorph (trust: high)
"model" form came from mlconjug3 (trust: high)
"rule" form came from a deterministic rule (trust: medium)
"fallback" we could not inflect; returned lemma as-is (trust: low → FLAG)
Public API (used by realizer_es.py):
conjugate(lemma, mood, tense, person, number, formality="informal") -> (form, conf)
participle(lemma) -> (form, conf) # past participle (compound tenses)
gerund(lemma) -> (form, conf)
noun_gender(lemma) -> "m"|"f"
inflect_noun(lemma, number) -> (form, conf)
inflect_adj(lemma, gender, number) -> (form, conf)
attach_enclitics(verb_form, clitics) -> str # accent-correct enclisis
lexicon_stats() -> dict
"""
import os
import pickle
import unicodedata
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "spa.unimorph")
_CACHE = os.path.join(_HERE, "data", "es_morph_cache.pkl")
# ── canonical feature keys the realizer speaks, mapped to UniMorph tags ─────────
# mood/tense pair -> the UniMorph feature substring that identifies it
_VERB_KEYMAP = {
("ind", "present"): ("IND", "PRS", None),
("ind", "preterite"): ("IND", "PST", "PFV"),
("ind", "imperfect"): ("IND", "PST", "IPFV"),
("ind", "future"): ("IND", "FUT", None),
("ind", "conditional"):("COND", None, None),
("sbjv", "present"): ("SBJV", "PRS", None),
("sbjv", "imperfect"): ("SBJV", "PST", "LGSPEC1"), # -ra form
("imp", "present"): ("POS", "IMP", None),
}
_PERSON = {"first": "1", "second": "2", "third": "3"}
_NUMBER = {"singular": "SG", "plural": "PL"}
# ── build / load the compact lexicon ───────────────────────────────────────────
def _feat_set(tag):
return set(tag.split(";"))
def _build_cache():
verbs = {} # (lemma, canonkey) -> form canonkey e.g. "ind|present|1|SG|infm"
nouns = {} # lemma -> {"g": "m"/"f", "SG": form, "PL": form}
adjs = {} # lemma -> {("m","SG"): form, ...}
part = {} # lemma -> masc-sg participle
ger = {} # lemma -> gerund
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
f = _feat_set(tag)
head = tag.split(";")[0]
if head == "V":
# skip clitic-bearing rows (we generate clitics ourselves)
if "PRO" in f:
continue
if "V.PTCP" in f and "PST" in f and "MASC" in f and "SG" in f:
part.setdefault(lemma, form)
continue
if "V.CVB" in f or "NFIN" in f or "V.PTCP" in f:
if "V.CVB" in f:
ger.setdefault(lemma, form)
continue
# identify mood/tense
mt = None
for (mood, tense), (a, b, c) in _VERB_KEYMAP.items():
if a not in f:
continue
if b is not None and b not in f:
continue
if c is not None and c not in f:
continue
# disambiguate IND;PST needing PFV vs IPFV
if a == "IND" and b == "PST" and c not in f:
continue
mt = (mood, tense)
break
if mt is None:
continue
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person is None or number is None:
continue
formal = "form" if "FORM" in f else ("infm" if "INFM" in f else "any")
key = f"{mt[0]}|{mt[1]}|{person}|{number}|{formal}"
verbs.setdefault((lemma, key), form)
elif head == "N":
# substring test handles epicene "MASC+FEM" (-> masc citation)
g = "m" if "MASC" in tag else ("f" if "FEM" in tag else None)
num = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if num is None:
continue
# store forms keyed by (gender,number); animate nouns list BOTH
# genders under one lemma (niño -> niño/niña). Resolve citation
# gender in a post-pass (gender of the row whose form == lemma).
d = nouns.setdefault(lemma, {})
d.setdefault("_rows", []).append((g, num, form))
elif head == "ADJ":
g = "m" if "MASC" in tag else ("f" if "FEM" in tag else "m")
num = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if num is None:
continue
adjs.setdefault(lemma, {})[(g, num)] = form
# post-pass: resolve noun citation gender + default SG/PL forms
for lemma, d in nouns.items():
rows = d.pop("_rows", [])
# citation gender = gender of the row whose form == lemma; else first MASC;
# else first seen gender.
cite_g = None
for g, num, form in rows:
if form == lemma and g:
cite_g = g
break
if cite_g is None:
for g, num, form in rows:
if g == "m":
cite_g = "m"
break
if cite_g is None:
cite_g = next((g for g, _, _ in rows if g), "m")
d["g"] = cite_g
for g, num, form in rows:
d[(g, num)] = form
d["SG"] = d.get((cite_g, "SG")) or next((f for g, n, f in rows if n == "SG"), lemma)
d["PL"] = d.get((cite_g, "PL")) or next((f for g, n, f in rows if n == "PL"), None)
# post-pass: UniMorph omits the identity inflection (masc-sg == lemma) for
# adjectives, so fill it in; without this a fem-sg row wrongly satisfies a
# masc-sg request (alto -> alta bug).
for lemma, d in adjs.items():
d.setdefault(("m", "SG"), lemma)
data = {"verbs": verbs, "nouns": nouns, "adjs": adjs, "part": part, "ger": ger}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE) and os.path.getmtime(_CACHE) >= os.path.getmtime(_UNIMORPH):
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_VERBS, _NOUNS, _ADJS, _PART, _GER = (
_LEX["verbs"], _LEX["nouns"], _LEX["adjs"], _LEX["part"], _LEX["ger"])
# ── mlconjug3 fallback (lazy) ───────────────────────────────────────────────────
_MLC = None
_MLC_TENSE = { # (mood,tense) -> (mlconjug mood label, tense label)
("ind", "present"): ("Indicativo", "Indicativo presente"),
("ind", "preterite"): ("Indicativo", "Indicativo pretérito perfecto simple"),
("ind", "imperfect"): ("Indicativo", "Indicativo pretérito imperfecto"),
("ind", "future"): ("Indicativo", "Indicativo futuro"),
("ind", "conditional"): ("Condicional", "Condicional Condicional"),
("sbjv", "present"): ("Subjuntivo", "Subjuntivo presente"),
("sbjv", "imperfect"): ("Subjuntivo", "Subjuntivo pretérito imperfecto 1"),
("imp", "present"): ("Imperativo", "Imperativo Afirmativo"),
}
_MLC_SLOT = { # (person,number) -> mlconjug slot key
("first", "singular"): "1s", ("second", "singular"): "2s",
("third", "singular"): "3s", ("first", "plural"): "1p",
("second", "plural"): "2p", ("third", "plural"): "3p",
}
def _mlc_conjugate(lemma, mood, tense, person, number):
global _MLC
try:
if _MLC is None:
from mlconjug3 import Conjugator
_MLC = Conjugator(language="es")
v = _MLC.conjugate(lemma)
if v is None:
return None
info = v.conjug_info
m, t = _MLC_TENSE.get((mood, tense), (None, None))
if m is None or m not in info or t not in info[m]:
return None
block = info[m][t]
slot = _MLC_SLOT.get((person, number))
if isinstance(block, dict) and slot in block and block[slot]:
return block[slot]
return None
except Exception:
return None
# ── regular-ending rule fallback (last resort, deterministic) ───────────────────
def _vclass(lemma):
return lemma[-2:] if lemma[-2:] in ("ar", "er", "ir") else "ar"
def _stem(lemma):
return lemma[:-2]
_REG = {
("ind", "present", "ar"): ["o", "as", "a", "amos", "áis", "an"],
("ind", "present", "er"): ["o", "es", "e", "emos", "éis", "en"],
("ind", "present", "ir"): ["o", "es", "e", "imos", "ís", "en"],
("ind", "preterite", "ar"): ["é", "aste", "ó", "amos", "asteis", "aron"],
("ind", "preterite", "er"): ["í", "iste", "", "imos", "isteis", "ieron"],
("ind", "preterite", "ir"): ["í", "iste", "", "imos", "isteis", "ieron"],
("ind", "imperfect", "ar"): ["aba", "abas", "aba", "ábamos", "abais", "aban"],
("ind", "imperfect", "er"): ["ía", "ías", "ía", "íamos", "íais", "ían"],
("ind", "imperfect", "ir"): ["ía", "ías", "ía", "íamos", "íais", "ían"],
("sbjv", "present", "ar"): ["e", "es", "e", "emos", "éis", "en"],
("sbjv", "present", "er"): ["a", "as", "a", "amos", "áis", "an"],
("sbjv", "present", "ir"): ["a", "as", "a", "amos", "áis", "an"],
("sbjv", "imperfect", "ar"): ["ara", "aras", "ara", "áramos", "arais", "aran"],
("sbjv", "imperfect", "er"): ["iera", "ieras", "iera", "iéramos", "ierais", "ieran"],
("sbjv", "imperfect", "ir"): ["iera", "ieras", "iera", "iéramos", "ierais", "ieran"],
}
_FUT = ["é", "ás", "á", "emos", "éis", "án"]
_COND = ["ía", "ías", "ía", "íamos", "íais", "ían"]
def _slot_idx(person, number):
base = {"first": 0, "second": 1, "third": 2}[person]
return base + (0 if number == "singular" else 3)
def _rule_conjugate(lemma, mood, tense, person, number):
if len(lemma) < 3 or lemma[-2:] not in ("ar", "er", "ir"):
return None
vc, st, i = _vclass(lemma), _stem(lemma), _slot_idx(person, number)
if tense == "future":
return lemma + _FUT[i]
if tense == "conditional":
return lemma + _COND[i]
table = _REG.get((mood, tense, vc))
if table:
return st + table[i]
if mood == "imp" and tense == "present":
# affirmative tú imperative = 3sg present indicative
pres = _REG.get(("ind", "present", vc))
return st + pres[2] if number == "singular" else st + pres[5]
return None
# ── PUBLIC: verb conjugation ────────────────────────────────────────────────────
def conjugate(lemma, mood, tense, person, number, formality="informal"):
"""Return (surface, confidence). mood in ind|sbjv|imp; tense per _VERB_KEYMAP."""
lemma = lemma.strip().lower()
p, n = _PERSON.get(person), _NUMBER.get(number)
formal = "form" if formality == "formal" else "infm"
if p and n:
for fkey in (formal, "any", "infm" if formal == "form" else "form"):
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}|{fkey}"))
if form:
return form, "lexicon"
m = _mlc_conjugate(lemma, mood, tense, person, number)
if m:
return m, "model"
r = _rule_conjugate(lemma, mood, tense, person, number)
if r:
return r, "rule"
return lemma, "fallback"
_IRREG_PART = { # guarantee the common irregular participles
"escribir": "escrito", "describir": "descrito", "abrir": "abierto",
"cubrir": "cubierto", "descubrir": "descubierto", "morir": "muerto",
"poner": "puesto", "ver": "visto", "volver": "vuelto", "devolver": "devuelto",
"hacer": "hecho", "deshacer": "deshecho", "decir": "dicho", "romper": "roto",
"resolver": "resuelto", "freír": "frito", "imprimir": "impreso",
"satisfacer": "satisfecho", "prever": "previsto", "revolver": "revuelto",
}
def participle(lemma):
lemma = lemma.strip().lower()
if lemma in _IRREG_PART:
return _IRREG_PART[lemma], "lexicon"
if lemma in _PART:
return _PART[lemma], "lexicon"
if lemma.endswith("ar"):
return lemma[:-2] + "ado", "rule"
if lemma[-2:] in ("er", "ir"):
return lemma[:-2] + "ido", "rule"
return lemma, "fallback"
_IRREG_GER = {"dormir": "durmiendo", "morir": "muriendo", "pedir": "pidiendo",
"sentir": "sintiendo", "mentir": "mintiendo", "servir": "sirviendo",
"venir": "viniendo", "decir": "diciendo", "poder": "pudiendo",
"ir": "yendo", "leer": "leyendo", "creer": "creyendo",
"oír": "oyendo", "traer": "trayendo", "caer": "cayendo",
"construir": "construyendo", "huir": "huyendo", "reír": "riendo"}
def gerund(lemma):
lemma = lemma.strip().lower()
if lemma in _IRREG_GER:
return _IRREG_GER[lemma], "lexicon"
if lemma in _GER:
return _GER[lemma], "lexicon"
if lemma.endswith("ar"):
return lemma[:-2] + "ando", "rule"
if lemma[-2:] in ("er", "ir"):
return lemma[:-2] + "iendo", "rule"
return lemma, "fallback"
# ── PUBLIC: noun gender + number ────────────────────────────────────────────────
_INVARIANT_PL = {"lunes", "martes", "miércoles", "jueves", "viernes",
"crisis", "tesis", "análisis", "dosis", "virus", "paraguas"}
def _gender_heuristic(noun):
for suf, g in (("ión", "f"), ("dad", "f"), ("tad", "f"), ("umbre", "f"),
("sis", "f"), ("ez", "f"), ("triz", "f"),
("ema", "m"), ("ama", "m"), ("oma", "m"), ("aje", "m"),
("or", "m"), ("án", "m"), ("ín", "m")):
if noun.endswith(suf):
return g
if noun.endswith("o"):
return "m"
if noun.endswith("a"):
return "f"
return "m"
def noun_gender(lemma):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if d and d.get("g"):
return d["g"]
return _gender_heuristic(lemma)
def _regular_plural(noun):
if noun in _INVARIANT_PL:
return noun
if not noun:
return noun
last = noun[-1]
if last == "z":
return noun[:-1] + "ces"
if last in "aeiouáéíóú":
# stressed final vowel í/ú -> +es (rubí->rubíes), else +s
if last in "íú":
return noun + "es"
return noun + "s"
if last == "s":
# esdrújula / stress-final handled crudely; most polysyllables invariant
return noun
return noun + "es"
def inflect_noun(lemma, number, gender=None):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
num = "SG" if number == "singular" else "PL"
if d:
# honor a requested gender for animate nouns (gato -> gata)
if gender and (gender, num) in d:
return d[(gender, num)], "lexicon"
if d.get(num):
return d[num], "lexicon"
if number == "singular":
return lemma, "rule" if not d else "lexicon"
return _regular_plural(lemma), "rule"
# ── PUBLIC: adjective agreement ─────────────────────────────────────────────────
_INV_GENDER_ADJ = {"español": "española", "trabajador": "trabajadora",
"hablador": "habladora", "encantador": "encantadora",
"alemán": "alemana", "francés": "francesa", "inglés": "inglesa"}
def inflect_adj(lemma, gender, number):
lemma = lemma.strip().lower()
d = _ADJS.get(lemma)
num = "SG" if number == "singular" else "PL"
if d:
form = d.get((gender, num))
if form:
return form, "lexicon"
# gender-invariant adjective (grande, feliz, azul): fem == masc.
# For a missing plural, pluralize this gender's singular form.
sg = d.get((gender, "SG")) or d.get(("m", "SG")) or lemma
if number == "plural":
return _regular_plural(sg), "rule"
return sg, "lexicon"
# rule fallback
a = lemma
if gender == "f":
if a in _INV_GENDER_ADJ:
a = _INV_GENDER_ADJ[a]
elif a.endswith("o"):
a = a[:-1] + "a"
if number == "plural":
a = _regular_plural(a)
return a, ("rule" if (a != lemma or gender == "m") else "rule")
# ── PUBLIC: clitic enclisis (dá + me + lo -> dámelo) ────────────────────────────
def _strip_accents(s):
return "".join(c for c in unicodedata.normalize("NFD", s)
if unicodedata.category(c) != "Mn")
def _count_syllables_vowelgroups(word):
# crude: count vowel groups
w = _strip_accents(word).lower()
groups, prev = 0, False
for ch in w:
isv = ch in "aeiou"
if isv and not prev:
groups += 1
prev = isv
return groups
def _host_stress_from_end(word):
"""Stressed-syllable index counted from the end (1=last) of a verb host."""
syls = _count_syllables_vowelgroups(word)
if any(c in "áéíóú" for c in word):
return None # already carries its own accent
if word[-2:] in ("ar", "er", "ir"): # infinitive: oxytone
return 1
if word.endswith("ndo"): # gerund: paroxytone
return 2
if word[-1:] in "aeiouns" and syls >= 2: # default paroxytone
return 2
return 1 # monosyllable / consonant-final oxytone
def attach_enclitics(verb_form, clitics):
"""Append clitic pronouns to a verb (imperative/infinitive/gerund enclisis)
and add a written accent when the resulting word becomes esdrújula/
sobreesdrújula (stress >= 3 syllables from the end): dá+me+lo -> dámelo,
lleva+me -> llévame, but dar+te -> darte and da+me -> dame (no accent)."""
if not clitics:
return verb_form
tail = "".join(clitics)
if any(c in "áéíóú" for c in verb_form): # host already accented
return verb_form + tail
sfe = _host_stress_from_end(verb_form)
total_sfe = sfe + len(clitics) # each clitic = 1 syllable
if total_sfe >= 3:
return _accentuate_nucleus(verb_form, sfe) + tail
return verb_form + tail
def _accentuate_nucleus(word, sfe):
"""Put a written accent on the syllable `sfe` positions from the word's end."""
vowels = "aeiou"
nuclei = [i for i, ch in enumerate(word) if ch in vowels]
if not nuclei or sfe > len(nuclei):
return word
i = nuclei[-sfe]
acc = {"a": "á", "e": "é", "i": "í", "o": "ó", "u": "ú"}
return word[:i] + acc[word[i]] + word[i + 1:]
def _accentuate_last_stressed(word):
# Restore the host's ORIGINAL lexical stress with a written accent.
# Default Spanish stress: word ending in vowel/n/s -> penultimate syllable;
# otherwise (e.g. infinitives in -r) -> last syllable.
vowels = "aeiou"
nuclei = [i for i, ch in enumerate(word) if ch in vowels]
if not nuclei:
return word
if word[-1] in "aeiouns" and len(nuclei) >= 2:
i = nuclei[-2] # paroxytone: penult nucleus
else:
i = nuclei[-1] # oxytone / monosyllable: last nucleus
acc = {"a": "á", "e": "é", "i": "í", "o": "ó", "u": "ú"}
return word[:i] + acc[word[i]] + word[i + 1:]
def lexicon_stats():
return {
"source": "UniMorph Spanish (github.com/unimorph/spa)",
"license": "CC-BY-SA 3.0 (Wiktionary-derived)",
"total_forms": sum(len(v) for v in (_VERBS, _NOUNS, _ADJS)) if False else None,
"verb_forms": len(_VERBS),
"verb_lemmas": len({k[0] for k in _VERBS}),
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
"participles": len(_PART),
"gerunds": len(_GER),
}
if __name__ == "__main__":
import json
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
tests = [
("hablar", "ind", "present", "first", "singular", "hablo"),
("comer", "ind", "present", "third", "plural", "comen"),
("vivir", "ind", "present", "first", "plural", "vivimos"),
("ser", "ind", "present", "third", "singular", "es"),
("ir", "ind", "preterite", "first", "singular", "fui"),
("tener", "ind", "future", "first", "singular", "tendré"),
("hacer", "sbjv", "present", "first", "singular", "haga"),
("dormir", "ind", "present", "first", "singular", "duermo"),
("pensar", "sbjv", "present", "third", "singular", "piense"),
("dar", "ind", "preterite", "third", "singular", "dio"),
("poner", "ind", "conditional", "first", "singular", "pondría"),
]
ok = 0
for lemma, mood, tense, per, num, exp in tests:
got, conf = conjugate(lemma, mood, tense, per, num)
flag = "OK " if got == exp else "XX "
if got == exp:
ok += 1
print(f" {flag}{lemma:8} {mood}/{tense} {per[:3]}.{num[:2]:3} -> {got:14} ({conf}) exp={exp}")
print(f"verb tests {ok}/{len(tests)}")
print(" gender casa:", noun_gender("casa"), "| problema:", noun_gender("problema"),
"| agua:", noun_gender("agua"), "| mano:", noun_gender("mano"))
print(" plural: luz->", inflect_noun("luz", "plural"), "| rey->", inflect_noun("rey", "plural"))
print(" adj: rojo/f/pl->", inflect_adj("rojo", "f", "plural"),
"| feliz/m/pl->", inflect_adj("feliz", "m", "plural"),
"| grande/f/pl->", inflect_adj("grande", "f", "plural"))
print(" enclisis: da+[me,lo]->", attach_enclitics("da", ["me", "lo"]),
"| di+[me]->", attach_enclitics("di", ["me"]),
"| dar+[se,lo]->", attach_enclitics("dar", ["se", "lo"]))
+629
View File
@@ -0,0 +1,629 @@
"""morphology_fr_full.py — production-grade French morphological generator.
Same architecture as morphology_it_full.py (shared Romance engine); French-specific
data and rules swapped in. Backed by three real, Wiktionary-lineage sources:
VERBS
UniMorph French (github.com/unimorph/fra, CC-BY-SA 3.0)
7,535 verb lemmas × full paradigm, CLEAN orthography:
indicatif présent / imparfait (PST;IPFV) / passé simple (PST;PFV) /
futur, conditionnel (COND), subjonctif présent (SBJV;PRS) /
subjonctif imparfait (SBJV;PST), impératif (POS;IMP), infinitif (NFIN),
participe présent (V.CVB/V.PTCP;PRS), participe passé (V.PTCP;PST, m.sg).
fr_irreg_verbs.json — high-frequency verbs UniMorph MISSES or mis-slots,
above all ÊTRE (absent from UniMorph fra), plus avoir/aller/faire/… — the
auxiliaries the passé-composé + être-agreement system depends on. Extracted
from kaikki.org French (build_fr_irreg.py), reflexive/multiword forms
dropped. This layer takes PRIORITY.
NOUNS + ADJECTIVES — kaikki.org French (Wiktionary extract, CC-BY-SA 3.0)
noun lemmas WITH inherent gender (head-template arg) + real plural
(cheval->chevaux, œil->yeux, invariable -s/-x/-z), resolved PER LEMMA.
adjective lemmas with real feminine + plural (petit->petite/petits/petites,
beau->belle/beaux/belles, heureux->heureuse, rouge invariant-gender).
Fallbacks (degrade, never crash, on OOV input):
verbs : rule generator for -er / -ir(-iss-) / -re (with -cer/-ger spelling,
future/conditional stems, imparfait/subjonctif endings)
nouns : gender heuristic (endings) + rule pluralization (-al->-aux, -eau->-eaux)
adjs : fem/plural agreement rules (-er->-ère, -eux->-euse, -f->-ve, +e default)
Confidence flag on every form: "lexicon" | "rule" | "fallback".
Public API (used by realizer_fr.py): identical signature to morphology_it_full.
"""
import json
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "fra.unimorph")
_IRREG = os.path.join(_HERE, "data", "fr_irreg_verbs.json")
_KAIKKI = os.path.join(_HERE, "data", "kaikki_fr.jsonl")
_CACHE = os.path.join(_HERE, "data", "fr_morph_cache.pkl")
# ── (mood, tense) -> UniMorph feature set that must ALL be present ────────────────
_VERB_KEYMAP = {
("ind", "present"): {"IND", "PRS"},
("ind", "imperfect"): {"IND", "PST", "IPFV"}, # imparfait
("ind", "passe_simple"): {"IND", "PST", "PFV"}, # passé simple
("ind", "future"): {"IND", "FUT"},
("ind", "conditional"): {"COND"}, # French: V;COND;1;SG
("sbjv", "present"): {"SBJV", "PRS"},
("sbjv", "imperfect"): {"SBJV", "PST"},
("imp", "affirmative"): {"POS", "IMP"},
}
_PERSON = {"first": "1", "second": "2", "third": "3"}
_NUMBER = {"singular": "SG", "plural": "PL"}
def _feat_set(tag):
return set(tag.split(";"))
# ── build verb lexicon from UniMorph ─────────────────────────────────────────────
def _build_verbs():
verbs = {}
part = {}
ger = {}
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
f = _feat_set(tag)
head = tag.split(";")[0]
if head == "V.PTCP":
if "PST" in f:
part.setdefault(lemma, form)
elif "PRS" in f:
ger.setdefault(lemma, form)
continue
if head == "V.CVB":
if "PRS" in f:
ger.setdefault(lemma, form)
continue
if head != "V":
continue
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person is None or number is None:
continue
for (mood, tense), req in _VERB_KEYMAP.items():
if not req <= f:
continue
if tense == "imperfect" and "PFV" in f:
continue
if tense == "passe_simple" and "IPFV" in f:
continue
verbs.setdefault((lemma, f"{mood}|{tense}|{person}|{number}"), form)
break
return verbs, part, ger
# ── kaikki nouns + adjectives ────────────────────────────────────────────────────
_EXCL_FORM_TAGS = {"alternative", "archaic", "obsolete", "dialectal", "regional",
"diminutive", "augmentative", "pejorative", "comparative",
"superlative", "misspelling", "rare", "informal", "literary",
"poetic", "error-unrecognized-form", "construed", "collective",
"nonstandard", "dated", "Louisiana", "Switzerland", "Belgium"}
def _kaikki_gender(arg):
if not arg:
return None
a = str(arg).lower()
if a.startswith("f"):
return "f"
if a.startswith("m"):
return "m"
return None
def _build_nouns_adjs():
nouns = {}
adjs = {}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
pos = d.get("pos")
word = d.get("word", "")
if not word or " " in word:
continue
forms = d.get("forms", []) or []
if pos == "noun":
ht = d.get("head_templates") or []
g = None
if ht:
g = _kaikki_gender((ht[0].get("args") or {}).get("1"))
if g is None:
tags = d.get("tags") or []
if "feminine" in tags:
g = "f"
elif "masculine" in tags:
g = "m"
pl = None
for x in forms:
t = set(x.get("tags") or [])
if "plural" in t and not (t & _EXCL_FORM_TAGS):
fm = x.get("form")
if fm and " " not in fm and fm not in ("#", "-", ""):
pl = fm
break
if word not in nouns:
nouns[word] = {"g": g, "SG": word, "PL": pl}
else:
cur = nouns[word]
if cur.get("g") is None and g:
cur["g"] = g
if not cur.get("PL") and pl:
cur["PL"] = pl
elif pos == "adj":
d0 = adjs.setdefault(word, {})
d0.setdefault(("m", "SG"), word)
for x in forms:
t = set(x.get("tags") or [])
fm = x.get("form")
if not fm or " " in fm or (t & _EXCL_FORM_TAGS):
continue
if "feminine" in t and "plural" in t:
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
elif "masculine" in t and "plural" in t:
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
elif "feminine" in t:
d0[("f", "SG")] = d0.get(("f", "SG")) or fm
elif "plural" in t:
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
return nouns, adjs
def _build_cache():
verbs, part, ger = _build_verbs()
nouns, adjs = _build_nouns_adjs()
with open(_IRREG, encoding="utf-8") as fh:
irreg = json.load(fh)
data = {"verbs": verbs, "part": part, "ger": ger,
"nouns": nouns, "adjs": adjs, "irreg": irreg}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE):
srcs = [_UNIMORPH, _KAIKKI, _IRREG]
newest = max(os.path.getmtime(s) for s in srcs if os.path.exists(s))
if os.path.getmtime(_CACHE) >= newest:
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_VERBS, _PART, _GER, _NOUNS, _ADJS, _IRREGV = (
_LEX["verbs"], _LEX["part"], _LEX["ger"], _LEX["nouns"], _LEX["adjs"],
_LEX["irreg"])
# ── regular-ending rule fallback ─────────────────────────────────────────────────
def _vclass(lemma):
if lemma.endswith("er"):
return "er"
if lemma.endswith("ir"):
return "ir"
if lemma.endswith("re"):
return "re"
if lemma.endswith("oir"):
return "oir"
return None
# present-tense endings [1sg,2sg,3sg,1pl,2pl,3pl]
_REG_PRES = {
"er": ["e", "es", "e", "ons", "ez", "ent"],
"ir": ["is", "is", "it", "issons", "issez", "issent"], # -iss- class (finir)
"re": ["s", "s", "", "ons", "ez", "ent"], # vendre: vends/vend
}
_REG_IMPF = ["ais", "ais", "ait", "ions", "iez", "aient"] # attaches to pres-1pl stem
_REG_SUBJ = ["e", "es", "e", "ions", "iez", "ent"] # attaches to 3pl stem
_REG_PS = { # passé simple
"er": ["ai", "as", "a", "âmes", "âtes", "èrent"],
"ir": ["is", "is", "it", "îmes", "îtes", "irent"],
"re": ["is", "is", "it", "îmes", "îtes", "irent"],
}
_FUT = ["ai", "as", "a", "ons", "ez", "ont"]
_COND = ["ais", "ais", "ait", "ions", "iez", "aient"]
def _slot_idx(person, number):
base = {"first": 0, "second": 1, "third": 2}[person]
return base + (0 if number == "singular" else 3)
def _fut_stem(lemma, vc):
"""Future/conditional stem = infinitive (drop final -e of -re)."""
if vc == "re":
return lemma[:-1] # vendre -> vendr-
return lemma # parler-, finir-
def _pres_1pl_stem(lemma, vc):
"""Imparfait stem = present 1pl minus -ons (parlons->parl-, finissons->finiss-)."""
if vc == "er":
stem = lemma[:-2]
if stem.endswith("g"):
return stem + "e" # mangeons -> mange- (imparfait mangeais)
if stem.endswith("c"):
return stem[:-1] + "ç" # commençons -> commenç-
return stem
if vc == "ir":
return lemma[:-1] + "iss" # finir -> finiss-
if vc == "re":
return lemma[:-2] # vendre -> vend-
return lemma[:-2]
def _apply_er_spelling(stem, ending):
"""-cer/-ger softening before a/o (commençons, mangeons)."""
if ending and ending[0] in ("a", "o"):
if stem.endswith("c"):
return stem[:-1] + "ç" + ending
if stem.endswith("g"):
return stem + "e" + ending
return stem + ending
def _rule_conjugate(lemma, mood, tense, person, number):
vc = _vclass(lemma)
if vc is None:
return None
i = _slot_idx(person, number)
if mood == "ind" and tense in ("future", "conditional"):
stem = _fut_stem(lemma, vc)
end = (_FUT if tense == "future" else _COND)[i]
return stem + end
if mood == "ind" and tense == "present":
table = _REG_PRES.get("ir" if vc == "ir" else vc)
if not table:
return None
body = lemma[:-2] if vc in ("er", "re") else lemma[:-1] if vc == "ir" else lemma[:-2]
if vc == "ir":
body = lemma[:-2] # fin- ; endings carry -iss-
end = table[i]
return body + end
end = table[i]
if vc == "er":
return _apply_er_spelling(body, end)
return body + end
if mood == "ind" and tense == "imperfect":
stem = _pres_1pl_stem(lemma, vc)
return stem + _REG_IMPF[i]
if mood == "ind" and tense == "passe_simple":
table = _REG_PS.get("ir" if vc == "ir" else vc)
if not table:
return None
body = lemma[:-2] if vc in ("er", "re") else lemma[:-2]
end = table[i]
if vc == "er":
return _apply_er_spelling(body, end)
return body + end
if mood == "sbjv" and tense == "present":
# subjonctif: present-3pl stem + e/es/e/ions/iez/ent
stem3 = _pres_1pl_stem(lemma, vc) if vc == "ir" else (
lemma[:-2] if vc in ("er", "re") else lemma[:-2])
if vc == "ir":
stem3 = lemma[:-2] + "iss"
end = _REG_SUBJ[i]
if vc == "er":
return _apply_er_spelling(stem3, end)
return stem3 + end
if mood == "imp" and tense == "affirmative":
# impératif ~ present indicative (tu drops -s for -er verbs)
pres = _rule_conjugate(lemma, "ind", "present", person, number)
if pres and vc == "er" and person == "second" and number == "singular":
return pres[:-1] if pres.endswith("es") else pres
return pres
return None
# ── PUBLIC: verb conjugation ─────────────────────────────────────────────────────
def conjugate(lemma, mood, tense, person, number):
"""Return (surface, confidence)."""
lemma = lemma.strip().lower()
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{number}"
ir = _IRREGV.get(lemma)
if ir and key in ir:
return ir[key], "lexicon"
p, n = _PERSON.get(person), _NUMBER.get(number)
if p and n:
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}"))
if form:
return form, "lexicon"
r = _rule_conjugate(lemma, mood, tense, person, number)
if r:
return r, "rule"
return lemma, "fallback"
# ── PUBLIC: participle + gerund/participe présent ────────────────────────────────
def _participle_msg(lemma):
ir = _IRREGV.get(lemma)
if ir and "part" in ir:
return ir["part"], "lexicon"
if lemma in _PART:
return _PART[lemma], "lexicon"
return None, None
# irregular participle fem/plural quirks (drop circonflexe: dû->due, dus)
_PART_FIX = {"": {"f|SG": "due", "m|PL": "dus", "f|PL": "dues"}}
def participle(lemma, gender="m", number="singular"):
"""Past participle with French gender/number agreement.
m.sg = base; f.sg = base+e; m.pl = base+s (invariable if base ends s/x);
f.pl = f.sg+s."""
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
msg, src = _participle_msg(lemma)
conf = "lexicon"
if msg is None:
vc = _vclass(lemma)
if vc == "er":
msg = lemma[:-2] + "é"
elif vc == "ir":
msg = lemma[:-1] # finir -> fini, partir -> parti
elif vc == "re":
msg = lemma[:-2] + "u" # vendre -> vendu
elif vc == "oir":
msg = lemma[:-3] + "u" # (rough) recevoir handled by irreg
else:
return lemma, "fallback"
conf = "rule"
fix = _PART_FIX.get(msg)
if fix and f"{g}|{num}" in fix:
return fix[f"{g}|{num}"], conf
if g == "m" and num == "SG":
return msg, conf
fem = msg + "e" if not msg.endswith("e") else msg
if g == "f" and num == "SG":
return fem, conf
if g == "m" and num == "PL":
return msg if msg.endswith(("s", "x")) else msg + "s", conf
# f|PL
return fem + "s", conf
def gerund(lemma):
"""Participe présent (base for gérondif 'en -ant')."""
lemma = lemma.strip().lower()
ir = _IRREGV.get(lemma)
if ir and "ger" in ir:
return ir["ger"], "lexicon"
if lemma in _GER:
return _GER[lemma], "lexicon"
vc = _vclass(lemma)
if vc == "er":
stem = lemma[:-2]
if stem.endswith("g"):
return stem + "eant", "rule"
if stem.endswith("c"):
return stem[:-1] + "çant", "rule"
return stem + "ant", "rule"
if vc == "ir":
return lemma[:-2] + "issant", "rule"
if vc == "re":
return lemma[:-2] + "ant", "rule"
return lemma, "fallback"
# ── PUBLIC: noun gender + number ─────────────────────────────────────────────────
_FEM_SUF = ("tion", "sion", "aison", "ance", "ence", "ette", "elle", "esse",
"ude", "ade", "ée", "", "tié", "ie", "ise", "ure", "eur")
_MASC_SUF = ("ment", "age", "eau", "isme", "oir", "ier", "eur", "in", "on")
def _gender_heuristic(noun):
for suf in _FEM_SUF:
if noun.endswith(suf):
return "f"
for suf in _MASC_SUF:
if noun.endswith(suf):
return "m"
if noun.endswith("e"):
return "f"
return "m"
def noun_gender(lemma):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if d and d.get("g") in ("m", "f"):
return d["g"]
return _gender_heuristic(lemma)
# closed sets for French plural irregularities
_OU_X = {"bijou", "caillou", "chou", "genou", "hibou", "joujou", "pou"}
_AIL_AUX = {"travail", "vitrail", "corail", "émail", "bail", "soupirail", "vantail"}
_AL_S = {"bal", "carnaval", "festival", "récital", "chacal", "régal", "cal", "aval"}
def _rule_plural(noun, gender):
"""Deterministic French pluralization. (form, ok); ok=False FLAGS ambiguity."""
if not noun:
return noun, True
if noun[-1:] in ("s", "x", "z"):
return noun, True # invariable
if noun in _OU_X:
return noun + "x", True
if noun.endswith(("eau", "au", "eu")):
if noun in ("pneu", "bleu", "landau", "sarrau"):
return noun + "s", True
return noun + "x", True # bateau->bateaux, jeu->jeux
if noun.endswith("al"):
if noun in _AL_S:
return noun + "s", True
return noun[:-2] + "aux", True # cheval->chevaux
if noun.endswith("ail"):
if noun in _AIL_AUX:
return noun[:-3] + "aux", True # travail->travaux
return noun + "s", True
return noun + "s", True # default
def inflect_noun(lemma, number, gender=None):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if number == "singular":
return (d["SG"] if d and d.get("SG") else lemma), ("lexicon" if d else "rule")
if d and d.get("PL"):
return d["PL"], "lexicon"
g = gender or noun_gender(lemma)
form, ok = _rule_plural(lemma, g)
return form, ("rule" if ok else "fallback")
# adjectives whose kaikki entries are unreliable: audited forms
_ADJ_FIX = {
"beau": {("m", "SG"): "beau", ("f", "SG"): "belle",
("m", "PL"): "beaux", ("f", "PL"): "belles"},
"nouveau": {("m", "SG"): "nouveau", ("f", "SG"): "nouvelle",
("m", "PL"): "nouveaux", ("f", "PL"): "nouvelles"},
"vieux": {("m", "SG"): "vieux", ("f", "SG"): "vieille",
("m", "PL"): "vieux", ("f", "PL"): "vieilles"},
"fou": {("m", "SG"): "fou", ("f", "SG"): "folle",
("m", "PL"): "fous", ("f", "PL"): "folles"},
"blanc": {("m", "SG"): "blanc", ("f", "SG"): "blanche",
("m", "PL"): "blancs", ("f", "PL"): "blanches"},
"long": {("m", "SG"): "long", ("f", "SG"): "longue",
("m", "PL"): "longs", ("f", "PL"): "longues"},
"bon": {("m", "SG"): "bon", ("f", "SG"): "bonne",
("m", "PL"): "bons", ("f", "PL"): "bonnes"},
}
def _rule_fem(a):
if a.endswith("e"):
return a
if a.endswith("er"):
return a[:-2] + "ère"
if a.endswith("eau"):
return a[:-3] + "elle"
if a.endswith("eux"):
return a[:-3] + "euse"
if a.endswith("f"):
return a[:-1] + "ve"
if a.endswith(("on", "en", "el", "eil", "et")):
return a + a[-1] + "e" # bon->bonne, ancien->ancienne, muet->muette
if a.endswith("c"):
return a[:-1] + "che" # blanc->blanche (public->publique via FIX)
return a + "e" # grand->grande, petit->petite, vert->verte
def inflect_adj(lemma, gender, number):
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
fix = _ADJ_FIX.get(lemma)
if fix and (g, num) in fix:
return fix[(g, num)], "lexicon"
d = _ADJS.get(lemma)
if d and d.get((g, num)):
return d[(g, num)], "lexicon"
# derive
msc = (d.get(("m", "SG")) if d else None) or lemma
if g == "m" and num == "SG":
return msc, "lexicon" if d else "rule"
fem = (d.get(("f", "SG")) if d else None) or _rule_fem(msc)
if g == "f" and num == "SG":
return fem, "lexicon" if (d and d.get(("f", "SG"))) else "rule"
if g == "m" and num == "PL":
if msc.endswith(("s", "x")):
return msc, "rule"
if msc.endswith("al"):
return msc[:-2] + "aux", "rule"
if msc.endswith("eau"):
return msc + "x", "rule"
return msc + "s", "rule"
# f|PL
return (fem if fem.endswith("s") else fem + "s"), "rule"
def lexicon_stats():
return {
"verb_source": "UniMorph French (github.com/unimorph/fra) + kaikki.org "
"irregulars (être + high-frequency)",
"noun_adj_source": "kaikki.org French (Wiktionary extract)",
"license": "CC-BY-SA 3.0 (Wiktionary/UniMorph lineage)",
"unimorph_verb_forms": len(_VERBS),
"unimorph_verb_lemmas": len({k[0] for k in _VERBS}),
"irregular_verb_lemmas": len(_IRREGV),
"participle_lemmas": len(_PART),
"gerund_lemmas": len(_GER),
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
}
if __name__ == "__main__":
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
tests = [
("parler", "ind", "present", "first", "singular", "parle"),
("être", "ind", "present", "third", "singular", "est"),
("avoir", "ind", "present", "first", "singular", "ai"),
("aller", "ind", "present", "third", "plural", "vont"),
("finir", "ind", "present", "first", "singular", "finis"),
("finir", "ind", "present", "first", "plural", "finissons"),
("manger", "ind", "present", "first", "plural", "mangeons"),
("faire", "ind", "future", "first", "singular", "ferai"),
("pouvoir", "sbjv", "present", "third", "singular", "puisse"),
("prendre", "ind", "passe_simple", "third", "singular", "prit"),
("vendre", "ind", "present", "third", "singular", "vend"),
("commencer", "ind", "imperfect", "first", "singular", "commençais"),
]
ok = 0
for lemma, mood, tense, per, num, exp in tests:
got, conf = conjugate(lemma, mood, tense, per, num)
flag = "OK " if got == exp else "XX "
ok += got == exp
print(f" {flag}{lemma:10} {mood}/{tense:12} {per[:3]}.{num[:2]} -> {got:12} ({conf}) exp={exp}")
print(f"verb tests {ok}/{len(tests)}")
print(" gender: maison=", noun_gender("maison"), "chat=", noun_gender("chat"),
"cheval=", noun_gender("cheval"), "nation=", noun_gender("nation"))
print(" plural: cheval->", inflect_noun("cheval", "plural"),
"| bateau->", inflect_noun("bateau", "plural"),
"| prix->", inflect_noun("prix", "plural"),
"| chat->", inflect_noun("chat", "plural"))
print(" adj: petit/f/sg->", inflect_adj("petit", "f", "singular"),
"| beau/f/sg->", inflect_adj("beau", "f", "singular"),
"| heureux/f/sg->", inflect_adj("heureux", "f", "singular"),
"| national/m/pl->", inflect_adj("national", "m", "plural"))
print(" part: aller/f/sg->", participle("aller", "f", "singular"),
"| prendre/f/pl->", participle("prendre", "f", "plural"),
"| finir/m/pl->", participle("finir", "m", "plural"))
print(" ger: manger->", gerund("manger"), "| finir->", gerund("finir"))
+588
View File
@@ -0,0 +1,588 @@
"""morphology_it_full.py — production-grade Italian morphological generator.
NOT a toy. Backed by three real, Wiktionary-lineage lexical sources:
VERBS
UniMorph Italian (github.com/unimorph/ita, CC-BY-SA 3.0)
10,009 verb lemmas × full paradigm, CLEAN orthography (no stress marks):
indicative present / imperfetto (PST;IPFV) / passato remoto (PST;PFV) /
futuro, condizionale (COND),
congiuntivo presente (SBJV;PRS) / imperfetto (SBJV;PST),
affirmative imperative, infinitive, gerundio (V.CVB;PRS),
past participle (masc-sg; fem/plural derived by vowel rule).
it_irreg_verbs.json 66 high-frequency verbs UniMorph MISSES
(essere, avere, potere, uscire, tenere, prendere, piacere, ), extracted
from kaikki.org Italian, filtered to standard forms, and DE-STRESSED to
real orthography (kaikki marks tonic stress everywhere: pàrlo->parlo,
avùto->avuto; final legit accents kept: sarò, è). Built by build_it_irreg.py.
This layer takes priority it supplies the two auxiliaries essere/avere,
which the whole passato-prossimo / essere-agreement system depends on.
NOUNS + ADJECTIVES kaikki.org Italian (Wiktionary extract, CC-BY-SA 3.0)
noun lemmas WITH inherent gender (head-template arg) + real (often irregular)
plural uomo->uomini, uovo->uova, dito->dita, città invariant resolved
PER LEMMA, never guessed.
adjective lemmas with real feminine + masc/fem plural (italiano->italiana/
italiani/italiane, felice->felici invariant).
Fallbacks (degrade, never crash, on OOV input):
verbs : rule generator for regular -are/-ere/-ire (with -care/-gare h-insertion
and -ciare/-giare/-iare i-drop spelling rules)
nouns : gender heuristic (endings) + rule pluralization (ambiguous -co/-go FLAGGED)
adjs : -o/-a/-e gender rule + rule pluralization
Confidence flag on every form:
"lexicon" from UniMorph / kaikki-irregular / kaikki noun-adj (trust: high)
"rule" deterministic rule (trust: medium)
"fallback" could not inflect; returned lemma / ambiguous (trust: low -> FLAG)
Public API (used by realizer_it.py):
conjugate(lemma, mood, tense, person, number) -> (form, conf)
participle(lemma, gender="m", number="singular") -> (form, conf)
gerund(lemma) -> (form, conf)
noun_gender(lemma) -> "m"|"f"
inflect_noun(lemma, number, gender=None) -> (form, conf)
inflect_adj(lemma, gender, number) -> (form, conf)
lexicon_stats() -> dict
"""
import json
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "ita.unimorph")
_IRREG = os.path.join(_HERE, "data", "it_irreg_verbs.json")
_KAIKKI = os.path.join(_HERE, "data", "kaikki_it.jsonl")
_CACHE = os.path.join(_HERE, "data", "it_morph_cache.pkl")
# ── (mood, tense) -> UniMorph feature set that must ALL be present ────────────────
_VERB_KEYMAP = {
("ind", "present"): {"IND", "PRS"},
("ind", "imperfect"): {"IND", "PST", "IPFV"},
("ind", "passato_remoto"): {"IND", "PST", "PFV"},
("ind", "future"): {"IND", "FUT"},
("ind", "conditional"): {"COND"},
("sbjv", "present"): {"SBJV", "PRS"},
("sbjv", "imperfect"): {"SBJV", "PST"},
("imp", "affirmative"): {"POS", "IMP"},
}
_PERSON = {"first": "1", "second": "2", "third": "3"}
_NUMBER = {"singular": "SG", "plural": "PL"}
def _feat_set(tag):
return set(tag.split(";"))
# ── build verb lexicon from UniMorph ─────────────────────────────────────────────
def _build_verbs():
verbs = {} # (lemma, "mood|tense|person|number") -> form
part = {} # lemma -> masc-sg past participle
ger = {} # lemma -> gerundio
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
f = _feat_set(tag)
head = tag.split(";")[0]
if head == "V.PTCP":
if "PST" in f:
part.setdefault(lemma, form)
continue
if head == "V.CVB": # gerundio (converb, present)
if "PRS" in f:
ger.setdefault(lemma, form)
continue
if head != "V":
continue
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person is None or number is None:
continue
for (mood, tense), req in _VERB_KEYMAP.items():
# exact-set discipline: PST;PFV must not match PST;IPFV, etc.
if not req <= f:
continue
# guard IND;PST ambiguity: require the specific aspect feature
if tense == "imperfect" and "PFV" in f:
continue
if tense == "passato_remoto" and "IPFV" in f:
continue
# COND must not also be a subjunctive/imperative slot
verbs.setdefault((lemma, f"{mood}|{tense}|{person}|{number}"), form)
break
return verbs, part, ger
# ── kaikki nouns + adjectives ────────────────────────────────────────────────────
_EXCL_FORM_TAGS = {"alternative", "archaic", "obsolete", "dialectal", "regional",
"diminutive", "augmentative", "pejorative", "comparative",
"superlative", "misspelling", "rare", "informal", "literary",
"poetic", "error-unrecognized-form", "apocopic", "obsolete",
"construed", "collective"}
def _kaikki_gender(arg):
if not arg:
return None
a = str(arg).lower()
if a.startswith("f"):
return "f"
if a.startswith("m"):
return "m"
return None
def _build_nouns_adjs():
nouns = {} # lemma -> {"g","SG","PL"}
adjs = {} # lemma -> {("m","SG"),("f","SG"),("m","PL"),("f","PL")}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
pos = d.get("pos")
word = d.get("word", "")
if not word or " " in word:
continue
forms = d.get("forms", []) or []
if pos == "noun":
ht = d.get("head_templates") or []
g = None
if ht:
g = _kaikki_gender((ht[0].get("args") or {}).get("1"))
if g is None:
tags = d.get("tags") or []
if "feminine" in tags:
g = "f"
elif "masculine" in tags:
g = "m"
pl = None
for x in forms:
t = set(x.get("tags") or [])
if "plural" in t and not (t & _EXCL_FORM_TAGS):
fm = x.get("form")
if fm and " " not in fm and fm != "#":
pl = fm
break
if word not in nouns:
nouns[word] = {"g": g, "SG": word, "PL": pl}
else:
cur = nouns[word]
if cur.get("g") is None and g:
cur["g"] = g
if not cur.get("PL") and pl:
cur["PL"] = pl
elif pos == "adj":
d0 = adjs.setdefault(word, {})
d0.setdefault(("m", "SG"), word)
for x in forms:
t = set(x.get("tags") or [])
fm = x.get("form")
if not fm or " " in fm or (t & _EXCL_FORM_TAGS):
continue
if "feminine" in t and "plural" in t:
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
elif "masculine" in t and "plural" in t:
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
elif "feminine" in t:
d0[("f", "SG")] = d0.get(("f", "SG")) or fm
elif "plural" in t: # invariant-gender adj (felice -> felici)
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
return nouns, adjs
def _build_cache():
verbs, part, ger = _build_verbs()
nouns, adjs = _build_nouns_adjs()
with open(_IRREG, encoding="utf-8") as fh:
irreg = json.load(fh)
data = {"verbs": verbs, "part": part, "ger": ger,
"nouns": nouns, "adjs": adjs, "irreg": irreg}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE):
srcs = [_UNIMORPH, _KAIKKI, _IRREG]
newest = max(os.path.getmtime(s) for s in srcs if os.path.exists(s))
if os.path.getmtime(_CACHE) >= newest:
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_VERBS, _PART, _GER, _NOUNS, _ADJS, _IRREGV = (
_LEX["verbs"], _LEX["part"], _LEX["ger"], _LEX["nouns"], _LEX["adjs"],
_LEX["irreg"])
# ── regular-ending rule fallback ─────────────────────────────────────────────────
def _vclass(lemma):
if lemma.endswith("are"):
return "are"
if lemma.endswith("ere"):
return "ere"
if lemma.endswith("ire"):
return "ire"
return None
# endings [1sg,2sg,3sg,1pl,2pl,3pl]
_REG = {
("ind", "present", "are"): ["o", "i", "a", "iamo", "ate", "ano"],
("ind", "present", "ere"): ["o", "i", "e", "iamo", "ete", "ono"],
("ind", "present", "ire"): ["o", "i", "e", "iamo", "ite", "ono"],
("ind", "imperfect", "are"): ["avo", "avi", "ava", "avamo", "avate", "avano"],
("ind", "imperfect", "ere"): ["evo", "evi", "eva", "evamo", "evate", "evano"],
("ind", "imperfect", "ire"): ["ivo", "ivi", "iva", "ivamo", "ivate", "ivano"],
("ind", "passato_remoto", "are"): ["ai", "asti", "ò", "ammo", "aste", "arono"],
("ind", "passato_remoto", "ere"): ["ei", "esti", "é", "emmo", "este", "erono"],
("ind", "passato_remoto", "ire"): ["ii", "isti", "ì", "immo", "iste", "irono"],
("sbjv", "present", "are"): ["i", "i", "i", "iamo", "iate", "ino"],
("sbjv", "present", "ere"): ["a", "a", "a", "iamo", "iate", "ano"],
("sbjv", "present", "ire"): ["a", "a", "a", "iamo", "iate", "ano"],
("sbjv", "imperfect", "are"): ["assi", "assi", "asse", "assimo", "aste", "assero"],
("sbjv", "imperfect", "ere"): ["essi", "essi", "esse", "essimo", "este", "essero"],
("sbjv", "imperfect", "ire"): ["issi", "issi", "isse", "issimo", "iste", "issero"],
# imperative: 2sg,3sg(Lei),1pl,2pl,3pl (1sg has none)
("imp", "affirmative", "are"): [None, "a", "i", "iamo", "ate", "ino"],
("imp", "affirmative", "ere"): [None, "i", "a", "iamo", "ete", "ano"],
("imp", "affirmative", "ire"): [None, "i", "a", "iamo", "ite", "ano"],
}
# future / conditional attach to a stem = infinitive minus final -e, with
# -are -> -er (parlare->parler-), -ere/-ire keep (credere->creder-, dormir-)
_FUT = ["ò", "ai", "à", "emo", "ete", "anno"]
_COND = ["ei", "esti", "ebbe", "emmo", "este", "ebbero"]
def _slot_idx(person, number):
base = {"first": 0, "second": 1, "third": 2}[person]
return base + (0 if number == "singular" else 3)
def _fut_stem(lemma, vc):
body = lemma[:-3] # drop are/ere/ire
if vc == "are":
return body + "er"
return body + vc[0] + "r" # ere->er? no: keep vowel: creder-, dormir-
# NOTE corrected below
def _apply_are_spelling(stem, ending):
"""-care/-gare insert h before front endings; -ciare/-giare/-sciare/-iare drop i."""
front = ending[:1] in ("i", "e")
if stem.endswith(("c", "g")) and front:
return stem + "h" + ending
if stem.endswith(("ci", "gi", "sci")) and ending[:1] == "i":
return stem[:-1] + ending # mangi+iamo -> mangiamo
if stem.endswith("i") and ending[:1] == "i":
return stem[:-1] + ending # studi+iamo -> studiamo
return stem + ending
def _rule_conjugate(lemma, mood, tense, person, number):
vc = _vclass(lemma)
if vc is None:
return None
body = lemma[:-3]
i = _slot_idx(person, number)
if mood == "ind" and tense in ("future", "conditional"):
stem = body + "er" if vc == "are" else body + vc[0] + "r"
# ere: creder-, ire: dormir- -> body + 'e'/'i' + 'r'
if vc == "ere":
stem = body + "er"
elif vc == "ire":
stem = body + "ir"
end = (_FUT if tense == "future" else _COND)[i]
# spelling: -care/-gare -> cherò/gherò ; -ciare/-giare -> cerò/gerò
if vc == "are":
if body.endswith(("c", "g")):
stem = body + "her"
elif body.endswith(("ci", "gi", "sci")):
stem = body[:-1] + "er"
elif body.endswith("i"):
stem = body[:-1] + "er"
return stem + end
table = _REG.get((mood, tense, vc))
if not table:
return None
end = table[i]
if end is None:
return None
if vc == "are":
return _apply_are_spelling(body, end)
# -ere/-ire: guard against double-i (dormi+iamo -> dormiamo)
if body.endswith("i") and end[:1] == "i":
return body[:-1] + end
return body + end
# ── PUBLIC: verb conjugation ─────────────────────────────────────────────────────
def conjugate(lemma, mood, tense, person, number):
"""Return (surface, confidence). mood in ind|sbjv|imp; tense per _VERB_KEYMAP."""
lemma = lemma.strip().lower()
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{number}"
ir = _IRREGV.get(lemma)
if ir and key in ir:
return ir[key], "lexicon"
p, n = _PERSON.get(person), _NUMBER.get(number)
if p and n:
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}"))
if form:
return form, "lexicon"
r = _rule_conjugate(lemma, mood, tense, person, number)
if r:
return r, "rule"
return lemma, "fallback"
# ── PUBLIC: participle + gerund ──────────────────────────────────────────────────
def _participle_msg(lemma):
"""Return (masc-sg participle, source) or (None, None)."""
ir = _IRREGV.get(lemma)
if ir and "part" in ir:
return ir["part"], "lexicon"
if lemma in _PART:
return _PART[lemma], "lexicon"
return None, None
def participle(lemma, gender="m", number="singular"):
"""Past participle with gender/number agreement (for essere-perfect & passives).
UniMorph/irregular give masc-sg; fem/plural derived by final-vowel swap
(-o -> -a/-i/-e), valid for regular -ato/-uto/-ito AND irregulars
(preso->presa/presi/prese, aperto->aperta/aperti/aperte, morto->morta/...)."""
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
msg, src = _participle_msg(lemma)
conf = "lexicon"
if msg is None:
vc = _vclass(lemma)
if vc == "are":
msg = lemma[:-3] + "ato"
elif vc == "ere":
msg = lemma[:-3] + "uto"
elif vc == "ire":
msg = lemma[:-3] + "ito"
else:
return lemma, "fallback"
conf = "rule"
# agreement: only -o participles inflect for gender+number
if msg.endswith("o"):
stem = msg[:-1]
suf = {"m|SG": "o", "f|SG": "a", "m|PL": "i", "f|PL": "e"}[f"{g}|{num}"]
return stem + suf, conf
return msg, conf # non -o participle: leave as-is (rare)
def gerund(lemma):
lemma = lemma.strip().lower()
ir = _IRREGV.get(lemma)
if ir and "ger" in ir:
return ir["ger"], "lexicon"
if lemma in _GER:
return _GER[lemma], "lexicon"
vc = _vclass(lemma)
if vc == "are":
return lemma[:-3] + "ando", "rule"
if vc in ("ere", "ire"):
return lemma[:-3] + "endo", "rule"
return lemma, "fallback"
# ── PUBLIC: noun gender + number ─────────────────────────────────────────────────
_FEM_SUF = ("zione", "sione", "gione", "", "", "trice", "aggine", "udine",
"igine", "ie", "essa", "izia", "ezza")
_MASC_SUF = ("ore", "ame", "iere", "ale", "ile")
def _gender_heuristic(noun):
for suf in _FEM_SUF:
if noun.endswith(suf):
return "f"
for suf in _MASC_SUF:
if noun.endswith(suf):
return "m"
if noun.endswith("o"):
return "m"
if noun.endswith("a"):
return "f"
if noun.endswith("à") or noun.endswith("ù"):
return "f"
return "m" # -e and consonant-final loanwords default masculine
def noun_gender(lemma):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if d and d.get("g") in ("m", "f"):
return d["g"]
return _gender_heuristic(lemma)
def _rule_plural(noun, gender):
"""Deterministic Italian pluralization. Returns (form, ok); ok=False FLAGS an
ambiguous case the lexicon would normally resolve (-co/-go palatalization)."""
if not noun:
return noun, True
# invariant: accented final vowel, consonant-final, monosyllable, -i final
if noun[-1:] in ("à", "è", "é", "ì", "í", "ò", "ó", "ù", "ú"):
return noun, True
if noun[-1:] not in ("a", "e", "o", "i", "u"):
return noun, True # consonant-final loanword: invariant
if noun.endswith("i"):
return noun, True # e.g. crisi, analisi: invariant
if noun.endswith("io"):
return noun[:-2] + "i", True # figlio->figli (unstressed i)
if noun.endswith("cia") or noun.endswith("gia"):
# vowel before cia/gia -> -cie/-gie ; consonant -> -ce/-ge (approx)
return noun[:-2] + "e", True # arancia->arance (majority)
if noun.endswith("ca"):
return noun[:-2] + "che", True # amica->amiche
if noun.endswith("ga"):
return noun[:-2] + "ghe", True
if noun.endswith("co"):
return noun[:-2] + "chi", False # AMBIGUOUS (amico->amici) -> flag
if noun.endswith("go"):
return noun[:-2] + "ghi", False # AMBIGUOUS (psicologo->psicologi)
if noun.endswith("a"):
return noun[:-1] + "e", True # casa->case (m -a: -i, but rare)
if noun.endswith("o"):
return noun[:-1] + "i", True # libro->libri
if noun.endswith("e"):
return noun[:-1] + "i", True # cane->cani, chiave->chiavi
return noun, True
def inflect_noun(lemma, number, gender=None):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if number == "singular":
return (d["SG"] if d and d.get("SG") else lemma), ("lexicon" if d else "rule")
if d and d.get("PL"):
return d["PL"], "lexicon"
g = gender or noun_gender(lemma)
form, ok = _rule_plural(lemma, g)
return form, ("rule" if ok else "fallback")
# adjectives whose kaikki entries are unreliable (messy inflection templates):
# supply audited regular agreement forms (prenominal apocope handled in realizer).
_ADJ_FIX = {
"bello": {("m", "SG"): "bello", ("f", "SG"): "bella",
("m", "PL"): "belli", ("f", "PL"): "belle"},
"quello": {("m", "SG"): "quello", ("f", "SG"): "quella",
("m", "PL"): "quelli", ("f", "PL"): "quelle"},
}
# ── PUBLIC: adjective agreement ──────────────────────────────────────────────────
def inflect_adj(lemma, gender, number):
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
fix = _ADJ_FIX.get(lemma)
if fix and (g, num) in fix:
return fix[(g, num)], "lexicon"
d = _ADJS.get(lemma)
if d:
form = d.get((g, num))
if form:
return form, "lexicon"
sg = d.get((g, "SG")) or d.get(("m", "SG")) or lemma
if num == "PL":
pl, ok = _rule_plural(sg, g)
return pl, ("rule" if ok else "fallback")
return sg, "lexicon"
# rule fallback
a = lemma
if a.endswith("o"): # -o/-a/-i/-e class
base = a[:-1]
suf = {"m|SG": "o", "f|SG": "a", "m|PL": "i", "f|PL": "e"}[f"{g}|{num}"]
return base + suf, "rule"
if a.endswith("e"): # felice-class: SG invariant, PL -i
if num == "PL":
return a[:-1] + "i", "rule"
return a, "rule"
if num == "PL":
p, ok = _rule_plural(a, g)
return p, ("rule" if ok else "fallback")
return a, "rule"
def lexicon_stats():
return {
"verb_source": "UniMorph Italian (github.com/unimorph/ita) + kaikki.org "
"irregulars (de-stressed)",
"noun_adj_source": "kaikki.org Italian (Wiktionary extract)",
"license": "CC-BY-SA 3.0 (Wiktionary/UniMorph lineage)",
"unimorph_verb_forms": len(_VERBS),
"unimorph_verb_lemmas": len({k[0] for k in _VERBS}),
"irregular_verb_lemmas": len(_IRREGV),
"participle_lemmas": len(_PART),
"gerund_lemmas": len(_GER),
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
}
if __name__ == "__main__":
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
tests = [
("parlare", "ind", "present", "first", "singular", "parlo"),
("essere", "ind", "present", "third", "singular", "è"),
("avere", "ind", "present", "first", "singular", "ho"),
("mangiare", "ind", "present", "second", "singular", "mangi"),
("finire", "ind", "present", "first", "singular", "finisco"),
("andare", "ind", "present", "third", "plural", "vanno"),
("fare", "ind", "future", "first", "singular", "farò"),
("potere", "sbjv", "present", "third", "singular", "possa"),
("prendere", "ind", "passato_remoto", "first", "singular", "presi"),
("cercare", "ind", "present", "second", "singular", "cerchi"),
("dormire", "ind", "present", "third", "plural", "dormono"),
("credere", "ind", "future", "first", "singular", "crederò"),
]
ok = 0
for lemma, mood, tense, per, num, exp in tests:
got, conf = conjugate(lemma, mood, tense, per, num)
flag = "OK " if got == exp else "XX "
ok += got == exp
print(f" {flag}{lemma:9} {mood}/{tense:14} {per[:3]}.{num[:2]} -> {got:12} ({conf}) exp={exp}")
print(f"verb tests {ok}/{len(tests)}")
print(" gender: casa=", noun_gender("casa"), "problema=", noun_gender("problema"),
"mano=", noun_gender("mano"), "città=", noun_gender("città"),
"cane=", noun_gender("cane"))
print(" plural: uomo->", inflect_noun("uomo", "plural"),
"| uovo->", inflect_noun("uovo", "plural"),
"| città->", inflect_noun("città", "plural"),
"| amico->", inflect_noun("amico", "plural"),
"| casa->", inflect_noun("casa", "plural"))
print(" adj: italiano/f/pl->", inflect_adj("italiano", "f", "plural"),
"| felice/m/pl->", inflect_adj("felice", "m", "plural"),
"| bello/f/sg->", inflect_adj("bello", "f", "singular"))
print(" part: aprire/f/sg->", participle("aprire", "f", "singular"),
"| prendere/m/pl->", participle("prendere", "m", "plural"),
"| andare/f/sg->", participle("andare", "f", "singular"))
print(" ger: fare->", gerund("fare"), "| parlare->", gerund("parlare"))
+666
View File
@@ -0,0 +1,666 @@
# -*- coding: utf-8 -*-
"""morphology_lat_full.py — production-grade Latin morphological generator.
Latin is the FLAGSHIP dead-language realizer. It rides the *architecture* of the
Romance/Italic engine (the same Realization / spec-driven design and the UniMorph
loader pattern from morphology_it_full.py) but with the CASE SYSTEM RESTORED
the feature Romance lost. Latin therefore exercises machinery the modern Romance
siblings never needed: 5 declensions x 6 cases x 2 numbers x 3 genders, plus a
4-conjugation verb system with tense/mood/voice.
DATA (real, attested no fabrication):
NOUNS + ADJECTIVES UniMorph Latin (github.com/unimorph/lat, CC-BY-SA 3.0)
163,182 N forms across ~thousands of lemmas, each with the full case paradigm
N;NOM/GEN/DAT/ACC/ABL/VOC;SG/PL (real inflected forms, WITH macrons:
puella->puellam, rēx->rēgis, corpus->corporis).
244,197 ADJ forms with case x GENDER x number, incl. UniMorph's combined
tags (GEN+DAT, MASC+FEM, MASC+FEM+NEUT) which are split on load.
462,668 V.PTCP forms (participles) also carry case/gender/number.
UniMorph N tags DO NOT encode inherent gender, so noun gender is inferred
from the declension (nom-sg + gen-sg endings) with a curated exceptions
map the standard, attestable rule (1st decl -a/-ae = fem, 2nd -us/-i =
masc, -um = neut, ...).
VERBS RULE ENGINE (honest gap: UniMorph Latin's verb list is a 947-lemma
sample of rare/prefixed verbs that MISSES every core textbook verb amō,
videō, sum, regō, ... are all absent). Latin conjugation is, however, highly
regular, so verbs are generated by a deterministic 4-conjugation engine over
curated principal parts (present / perfect / supine stems), sourced from
standard references. Irregulars (sum, possum, , ferō, volō, nōlō, mālō)
are curated full tables. Forms are flagged "rule" (not "lexicon") for honesty.
Confidence flag on every form (same contract as the Romance engine):
"lexicon" from UniMorph (trust: high)
"rule" deterministic morphology rule (trust: medium)
"fallback" could not inflect; returned lemma (trust: low -> FLAG)
Public API (used by realizer_lat.py):
decline_noun(lemma, case, number) -> (form, conf)
noun_gender(lemma) -> "m"|"f"|"n"
decline_adj(lemma, case, gender, number) -> (form, conf)
conjugate(lemma, tense, mood, voice, person, number) -> (form, conf)
participle(lemma, kind, case, gender, number) -> (form, conf) # kind: prs|pfv|fut
infinitive(lemma, tense="present", voice="active") -> (form, conf)
lexicon_stats() -> dict
"""
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "lat.unimorph")
_CACHE = os.path.join(_HERE, "data", "lat_morph_cache.pkl")
_CASES = ("NOM", "GEN", "DAT", "ACC", "ABL", "VOC")
_CASE_MAP = {"nom": "NOM", "gen": "GEN", "dat": "DAT", "acc": "ACC",
"abl": "ABL", "voc": "VOC"}
_NUM = {"singular": "SG", "plural": "PL"}
_GEN = {"m": "MASC", "f": "FEM", "n": "NEUT"}
# ── UniMorph loader: noun + adjective + participle case paradigms ────────────────
def _build_cache():
nouns = {} # lemma -> {(CASE, NUM): form}
adjs = {} # lemma -> {(CASE, GEN, NUM): form}
ptcps = {} # lemma -> {(CASE, GEN, NUM): form} (from V.PTCP; keyed loosely)
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
feats = tag.split(";")
head = feats[0]
fs = set(feats)
case = next((c for c in _CASES if c in fs), None)
# handle combined case tags like GEN+DAT
if case is None:
for f in feats:
if "+" in f and any(c in f.split("+") for c in _CASES):
case = [c for c in _CASES if c in f.split("+")]
break
num = "SG" if "SG" in fs else ("PL" if "PL" in fs else None)
if case is None or num is None:
continue
cases = case if isinstance(case, list) else [case]
if head == "N":
d = nouns.setdefault(lemma, {})
for c in cases:
d.setdefault((c, num), form)
elif head == "ADJ":
# gender may be combined: MASC+FEM+NEUT, MASC+FEM
genders = []
for g in ("MASC", "FEM", "NEUT"):
if any(g == x or (g in x.split("+")) for x in feats):
genders.append(g)
if not genders:
genders = ["MASC", "FEM", "NEUT"]
d = adjs.setdefault(lemma, {})
for c in cases:
for g in genders:
d.setdefault((c, g, num), form)
data = {"nouns": nouns, "adjs": adjs, "ptcps": ptcps}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE) and os.path.exists(_UNIMORPH):
if os.path.getmtime(_CACHE) >= os.path.getmtime(_UNIMORPH):
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_NOUNS, _ADJS = _LEX["nouns"], _LEX["adjs"]
# ── noun gender inference (declension-based, curated exceptions) ─────────────────
# Real, attestable rule: gender follows declension + nominative shape, with the
# standard closed set of exceptions.
_GENDER_EXC = {
# 1st-declension masculines (people/agents)
"agricola": "m", "poēta": "m", "nauta": "m", "incola": "m", "scrība": "m",
"auriga": "m", "pīrāta": "m", "athlēta": "m",
# 2nd-declension neuters / feminines
"vīrus": "n", "vulgus": "n", "pelagus": "n", "humus": "f",
# common 3rd-declension whose gender the ending would mispredict
"rēx": "m", "dux": "m", "mīles": "m", "pater": "m", "frāter": "m",
"homō": "m", "leō": "m", "sōl": "m", "mōns": "m", "pōns": "m", "fōns": "m",
"sanguis": "m", "ōrdō": "m", "sermō": "m", "amor": "m", "dolor": "m",
"labor": "m", "timor": "m", "honor": "m", "color": "m", "pēs": "m",
"dēns": "m", "flōs": "m", "mōs": "m", "mensis": "m", "orbis": "m",
"piscis": "m", "ignis": "m", "collis": "m", "grex": "m", "prīnceps": "m",
"māter": "f", "soror": "f", "uxor": "f", "mulier": "f", "virgō": "f",
"urbs": "f", "arx": "f", "pāx": "f", "lēx": "f", "lūx": "f", "vōx": "f",
"nox": "f", "nix": "f", "vīs": "f", "salūs": "f", "virtūs": "f",
"aetās": "f", "cīvitās": "f", "lībertās": "f", "vēritās": "f", "voluptās": "f",
"nātiō": "f", "ratiō": "f", "ōrātiō": "f", "legiō": "f", "regiō": "f",
"mens": "f", "gens": "f", "ars": "f", "pars": "f", "mors": "f", "sors": "f",
"nāvis": "f", "turris": "f", "avis": "f", "vallis": "f", "classis": "f",
"corpus": "n", "tempus": "n", "opus": "n", "genus": "n", "onus": "n",
"pectus": "n", "latus": "n", "vulnus": "n", "scelus": "n", "sīdus": "n",
"caput": "n", "iter": "n", "flūmen": "n", "nōmen": "n", "carmen": "n",
"agmen": "n", "certāmen": "n", "lūmen": "n", "ōmen": "n", "cōgnōmen": "n",
"mare": "n", "animal": "n", "exemplar": "n", "rēte": "n",
# 4th-declension exceptions
"manus": "f", "domus": "f", "tribus": "f", "porticus": "f", "īdūs": "f",
"cornū": "n", "genū": "n", "gelū": "n", "verū": "n",
# 5th-declension
"diēs": "m", "merīdiēs": "m",
}
def _infer_gender(lemma):
if lemma in _GENDER_EXC:
return _GENDER_EXC[lemma]
d = _NOUNS.get(lemma)
nom = d.get(("NOM", "SG")) if d else lemma
gen = d.get(("GEN", "SG")) if d else None
nom = nom or lemma
# 5th declension: gen -eī / -ēī
if gen and (gen.endswith("") or gen.endswith("ēī")):
return "f"
# 1st declension: nom -a, gen -ae
if nom.endswith("a") and (not gen or gen.endswith("ae")):
return "f"
# 2nd declension neuter: nom -um
if nom.endswith("um"):
return "n"
# 2nd declension masc: nom -us/-er/-ir, gen -ī
if (nom.endswith("us") or nom.endswith("er") or nom.endswith("ir")) and \
(not gen or gen.endswith("ī")):
return "m"
# 4th declension: gen -ūs
if gen and gen.endswith("ūs"):
return "n" if nom.endswith("ū") else "m"
# 3rd declension neuters by common nom endings
if nom.endswith(("men", "us", "ur", "al", "ar", "e", "ma")):
# -us here is 3rd-decl neuter type (corpus) only if gen shows -oris/-eris
if nom.endswith("us") and gen and (gen.endswith("oris") or gen.endswith("eris")
or gen.endswith("uris")):
return "n"
if nom.endswith(("men", "al", "ar", "e")):
return "n"
# default 3rd-declension: masculine (most common)
return "m"
_GENDER_CACHE = {}
def noun_gender(lemma):
lemma = lemma.strip()
if lemma not in _GENDER_CACHE:
_GENDER_CACHE[lemma] = _infer_gender(lemma)
return _GENDER_CACHE[lemma]
# ── PUBLIC: noun declension ─────────────────────────────────────────────────────
def decline_noun(lemma, case, number):
lemma = lemma.strip()
C = _CASE_MAP.get(case, case.upper())
N = _NUM.get(number, number)
d = _NOUNS.get(lemma)
if d and (C, N) in d:
return d[(C, N)], "lexicon"
# abl sg often == the -e/-o form; try nom fallback
if d:
# try VOC==NOM, ACC neuter==NOM etc are already in data; last resort lemma
return lemma, "fallback"
return lemma, "fallback"
# ── PUBLIC: adjective declension ────────────────────────────────────────────────
def decline_adj(lemma, case, gender, number):
lemma = lemma.strip()
C = _CASE_MAP.get(case, case.upper())
G = _GEN.get(gender, gender.upper())
N = _NUM.get(number, number)
d = _ADJS.get(lemma)
if d and (C, G, N) in d:
return d[(C, G, N)], "lexicon"
# try other gender (some adjs listed only under MASC+FEM etc handled at load)
if d:
for altG in ("MASC", "FEM", "NEUT"):
if (C, altG, N) in d:
return d[(C, altG, N)], "lexicon"
return lemma, "fallback"
return lemma, "fallback"
# ═══════════════════════════════════════════════════════════════════════════════
# VERB RULE ENGINE (4 conjugations + curated irregulars)
# ═══════════════════════════════════════════════════════════════════════════════
# Curated principal parts for common attested verbs:
# lemma -> (conj, present_stem, perfect_stem, supine_stem)
# conj in {1,2,3,"3io",4}. Stems carry macrons (matching UniMorph orthography).
_VERBS = {
"amō": (1, "am", "amāv", "amāt"),
"laudō": (1, "laud", "laudāv", "laudāt"),
"portō": (1, "port", "portāv", "portāt"),
"vocō": (1, "voc", "vocāv", "vocāt"),
"": (1, "d", "ded", "dat"),
"spectō": (1, "spect", "spectāv", "spectāt"),
"pugnō": (1, "pugn", "pugnāv", "pugnāt"),
"labōrō": (1, "labōr", "labōrāv", "labōrāt"),
"necō": (1, "nec", "necāv", "necāt"),
"parō": (1, "par", "parāv", "parāt"),
"cōgitō": (1, "cōgit", "cōgitāv", "cōgitāt"),
"habitō": (1, "habit", "habitāv", "habitāt"),
"nārrō": (1, "nārr", "nārrāv", "nārrāt"),
"servō": (1, "serv", "servāv", "servāt"),
"superō": (1, "super", "superāv", "superāt"),
"oppugnō": (1, "oppugn", "oppugnāv", "oppugnāt"),
"ambulō": (1, "ambul", "ambulāv", "ambulāt"),
"clāmō": (1, "clām", "clāmāv", "clāmāt"),
"vulnerō": (1, "vulner", "vulnerāv", "vulnerāt"),
"aedificō": (1, "aedific", "aedificāv", "aedificāt"),
"expugnō": (1, "expugn", "expugnāv", "expugnāt"),
"dēfendō": (3, "dēfend", "dēfend", "dēfēns"),
"petō": (3, "pet", "petīv", "petīt"),
"occīdō": (3, "occīd", "occīd", "occīs"),
"interficiō": ("3io", "interfic", "interfēc", "interfect"),
"timeō": (2, "tim", "timu", None),
"iaceō": (2, "iac", "iacu", None),
"pāreō": (2, "pār", "pāru", "pārit"),
"respondeō": (2, "respond", "respond", "respōns"),
"vertō": (3, "vert", "vert", "vers"),
"ostendō": (3, "ostend", "ostend", "ostent"),
"cōnstituō": (3, "cōnstitu", "cōnstitu", "cōnstitūt"),
"cōgnōscō": (3, "cōgnōsc", "cōgnōv", "cōgnit"),
"crēdō": (3, "crēd", "crēdid", "crēdit"),
"ēdūcō": (3, "ēdūc", "ēdūx", "ēduct"),
"cōnservō": (1, "cōnserv", "cōnservāv", "cōnservāt"),
"iuvō": (1, "iuv", "iūv", "iūt"),
"dēbeō": (2, "dēb", "dēbu", "dēbit"),
"moneō": (2, "mon", "monu", "monit"),
"videō": (2, "vid", "vīd", "vīs"),
"habeō": (2, "hab", "habu", "habit"),
"teneō": (2, "ten", "tenu", "tent"),
"timeō": (2, "tim", "timu", None),
"terreō": (2, "terr", "terru", "territ"),
"dēleō": (2, "dēl", "dēlēv", "dēlēt"),
"iubeō": (2, "iub", "iuss", "iuss"),
"maneō": (2, "man", "māns", "māns"),
"moveō": (2, "mov", "mōv", "mōt"),
"doceō": (2, "doc", "docu", "doct"),
"sedeō": (2, "sed", "sēd", "sess"),
"rīdeō": (2, "rīd", "rīs", "rīs"),
"regō": (3, "reg", "rēx", "rēct"),
"dūcō": (3, "dūc", "dūx", "duct"),
"scrībō": (3, "scrīb", "scrīps", "scrīpt"),
"mittō": (3, "mitt", "mīs", "miss"),
"pōnō": (3, "pōn", "posu", "posit"),
"agō": (3, "ag", "ēg", "āct"),
"dīcō": (3, "dīc", "dīx", "dict"),
"gerō": (3, "ger", "gess", "gest"),
"vincō": (3, "vinc", "vīc", "vict"),
"petō": (3, "pet", "petīv", "petīt"),
"legō": (3, "leg", "lēg", "lēct"),
"currō": (3, "curr", "cucurr", "curs"),
"vīvō": (3, "vīv", "vīx", "vīct"),
"quaerō": (3, "quaer", "quaesīv", "quaesīt"),
"trahō": (3, "trah", "trāx", "tract"),
"claudō": (3, "claud", "claus", "claus"),
"cōgō": (3, "cōg", "coēg", "coāct"),
"relinquō": (3, "relinqu", "relīqu", "relict"),
"capiō": ("3io", "cap", "cēp", "capt"),
"faciō": ("3io", "fac", "fēc", "fact"),
"iaciō": ("3io", "iac", "iēc", "iact"),
"rapiō": ("3io", "rap", "rapu", "rapt"),
"fugiō": ("3io", "fug", "fūg", "fugit"),
"cupiō": ("3io", "cup", "cupīv", "cupīt"),
"accipiō": ("3io", "accip", "accēp", "accept"),
"audiō": (4, "aud", "audīv", "audīt"),
"veniō": (4, "ven", "vēn", "vent"),
"sciō": (4, "sc", "scīv", "scīt"),
"sentiō": (4, "sent", "sēns", "sēns"),
"mūniō": (4, "mūn", "mūnīv", "mūnīt"),
"dormiō": (4, "dorm", "dormīv", "dormīt"),
"aperiō": (4, "aper", "aperu", "apert"),
"inveniō": (4, "inven", "invēn", "invent"),
}
# ── Present-system paradigms: full ending tables per conjugation, attached to the
# bare present stem (pstem). Hardcoded from the standard grammar with correct
# macrons/vowel-lengths — deterministic and independently verifiable. Keys:
# (tense, mood, voice) -> {conj: [1sg,2sg,3sg,1pl,2pl,3pl]}
_PARADIGM = {
("present", "ind", "active"): {
1: ["ō", "ās", "at", "āmus", "ātis", "ant"],
2: ["", "ēs", "et", "ēmus", "ētis", "ent"],
3: ["ō", "is", "it", "imus", "itis", "unt"],
"3io": ["", "is", "it", "imus", "itis", "iunt"],
4: ["", "īs", "it", "īmus", "ītis", "iunt"],
},
("present", "ind", "passive"): {
1: ["or", "āris", "ātur", "āmur", "āminī", "antur"],
2: ["eor", "ēris", "ētur", "ēmur", "ēminī", "entur"],
3: ["or", "eris", "itur", "imur", "iminī", "untur"],
"3io": ["ior", "eris", "itur", "imur", "iminī", "iuntur"],
4: ["ior", "īris", "ītur", "īmur", "īminī", "iuntur"],
},
("imperfect", "ind", "active"): {
1: ["ābam", "ābās", "ābat", "ābāmus", "ābātis", "ābant"],
2: ["ēbam", "ēbās", "ēbat", "ēbāmus", "ēbātis", "ēbant"],
3: ["ēbam", "ēbās", "ēbat", "ēbāmus", "ēbātis", "ēbant"],
"3io": ["iēbam", "iēbās", "iēbat", "iēbāmus", "iēbātis", "iēbant"],
4: ["iēbam", "iēbās", "iēbat", "iēbāmus", "iēbātis", "iēbant"],
},
("imperfect", "ind", "passive"): {
1: ["ābar", "ābāris", "ābātur", "ābāmur", "ābāminī", "ābantur"],
2: ["ēbar", "ēbāris", "ēbātur", "ēbāmur", "ēbāminī", "ēbantur"],
3: ["ēbar", "ēbāris", "ēbātur", "ēbāmur", "ēbāminī", "ēbantur"],
"3io": ["iēbar", "iēbāris", "iēbātur", "iēbāmur", "iēbāminī", "iēbantur"],
4: ["iēbar", "iēbāris", "iēbātur", "iēbāmur", "iēbāminī", "iēbantur"],
},
("future", "ind", "active"): {
1: ["ābō", "ābis", "ābit", "ābimus", "ābitis", "ābunt"],
2: ["ēbō", "ēbis", "ēbit", "ēbimus", "ēbitis", "ēbunt"],
3: ["am", "ēs", "et", "ēmus", "ētis", "ent"],
"3io": ["iam", "iēs", "iet", "iēmus", "iētis", "ient"],
4: ["iam", "iēs", "iet", "iēmus", "iētis", "ient"],
},
("future", "ind", "passive"): {
1: ["ābor", "āberis", "ābitur", "ābimur", "ābiminī", "ābuntur"],
2: ["ēbor", "ēberis", "ēbitur", "ēbimur", "ēbiminī", "ēbuntur"],
3: ["ar", "ēris", "ētur", "ēmur", "ēminī", "entur"],
"3io": ["iar", "iēris", "iētur", "iēmur", "iēminī", "ientur"],
4: ["iar", "iēris", "iētur", "iēmur", "iēminī", "ientur"],
},
("present", "sbjv", "active"): {
1: ["em", "ēs", "et", "ēmus", "ētis", "ent"],
2: ["eam", "eās", "eat", "eāmus", "eātis", "eant"],
3: ["am", "ās", "at", "āmus", "ātis", "ant"],
"3io": ["iam", "iās", "iat", "iāmus", "iātis", "iant"],
4: ["iam", "iās", "iat", "iāmus", "iātis", "iant"],
},
("present", "sbjv", "passive"): {
1: ["er", "ēris", "ētur", "ēmur", "ēminī", "entur"],
2: ["ear", "eāris", "eātur", "eāmur", "eāminī", "eantur"],
3: ["ar", "āris", "ātur", "āmur", "āminī", "antur"],
"3io": ["iar", "iāris", "iātur", "iāmur", "iāminī", "iantur"],
4: ["iar", "iāris", "iātur", "iāmur", "iāminī", "iantur"],
},
("imperfect", "sbjv", "active"): {
1: ["ārem", "ārēs", "āret", "ārēmus", "ārētis", "ārent"],
2: ["ērem", "ērēs", "ēret", "ērēmus", "ērētis", "ērent"],
3: ["erem", "erēs", "eret", "erēmus", "erētis", "erent"],
"3io": ["erem", "erēs", "eret", "erēmus", "erētis", "erent"],
4: ["īrem", "īrēs", "īret", "īrēmus", "īrētis", "īrent"],
},
("imperfect", "sbjv", "passive"): {
1: ["ārer", "ārēris", "ārētur", "ārēmur", "ārēminī", "ārentur"],
2: ["ērer", "ērēris", "ērētur", "ērēmur", "ērēminī", "ērentur"],
3: ["erer", "erēris", "erētur", "erēmur", "erēminī", "erentur"],
"3io": ["erer", "erēris", "erētur", "erēmur", "erēminī", "erentur"],
4: ["īrer", "īrēris", "īrētur", "īrēmur", "īrēminī", "īrentur"],
},
}
# perfect-active endings (added to perfect stem) — same for all conjugations
_PERF_ACT = {
("perfect", "ind"): ["ī", "istī", "it", "imus", "istis", "ērunt"],
("pluperfect", "ind"): ["eram", "erās", "erat", "erāmus", "erātis", "erant"],
("futureperfect", "ind"): ["erō", "eris", "erit", "erimus", "eritis", "erint"],
("perfect", "sbjv"): ["erim", "erīs", "erit", "erīmus", "erītis", "erint"],
("pluperfect", "sbjv"):["issem", "issēs", "isset", "issēmus", "issētis", "issent"],
}
def _idx(person, number):
base = {"first": 0, "second": 1, "third": 2}[person]
return base + (0 if number == "singular" else 3)
def _present_system(conj, pstem, tense, mood, voice, person, number):
"""Generate a present-system form (present/imperfect/future ind & subj)."""
table = _PARADIGM.get((tense, mood, voice))
if not table or conj not in table:
return None
return pstem + table[conj][_idx(person, number)]
def _active_infinitive_stem(conj, pstem):
return {1: pstem + "ā", 2: pstem + "ē", 3: pstem + "e",
"3io": pstem + "e", 4: pstem + "ī"}[conj]
_IRREG = {
"sum": {
("present", "ind", "active"): ["sum", "es", "est", "sumus", "estis", "sunt"],
("imperfect", "ind", "active"): ["eram", "erās", "erat", "erāmus", "erātis", "erant"],
("future", "ind", "active"): ["erō", "eris", "erit", "erimus", "eritis", "erunt"],
("perfect", "ind", "active"): ["fuī", "fuistī", "fuit", "fuimus", "fuistis", "fuērunt"],
("pluperfect", "ind", "active"): ["fueram", "fuerās", "fuerat", "fuerāmus", "fuerātis", "fuerant"],
("present", "sbjv", "active"): ["sim", "sīs", "sit", "sīmus", "sītis", "sint"],
("imperfect", "sbjv", "active"): ["essem", "essēs", "esset", "essēmus", "essētis", "essent"],
},
"possum": {
("present", "ind", "active"): ["possum", "potes", "potest", "possumus", "potestis", "possunt"],
("imperfect", "ind", "active"): ["poteram", "poterās", "poterat", "poterāmus", "poterātis", "poterant"],
("future", "ind", "active"): ["poterō", "poteris", "poterit", "poterimus", "poteritis", "poterunt"],
("perfect", "ind", "active"): ["potuī", "potuistī", "potuit", "potuimus", "potuistis", "potuērunt"],
("present", "sbjv", "active"): ["possim", "possīs", "possit", "possīmus", "possītis", "possint"],
},
"": {
("present", "ind", "active"): ["", "īs", "it", "īmus", "ītis", "eunt"],
("imperfect", "ind", "active"): ["ībam", "ībās", "ībat", "ībāmus", "ībātis", "ībant"],
("future", "ind", "active"): ["ībō", "ībis", "ībit", "ībimus", "ībitis", "ībunt"],
("perfect", "ind", "active"): ["", "īstī", "iit", "iimus", "īstis", "iērunt"],
("present", "sbjv", "active"): ["eam", "eās", "eat", "eāmus", "eātis", "eant"],
},
"volō": {
("present", "ind", "active"): ["volō", "vīs", "vult", "volumus", "vultis", "volunt"],
("imperfect", "ind", "active"): ["volēbam", "volēbās", "volēbat", "volēbāmus", "volēbātis", "volēbant"],
("future", "ind", "active"): ["volam", "volēs", "volet", "volēmus", "volētis", "volent"],
("perfect", "ind", "active"): ["voluī", "voluistī", "voluit", "voluimus", "voluistis", "voluērunt"],
("present", "sbjv", "active"): ["velim", "velīs", "velit", "velīmus", "velītis", "velint"],
},
"nōlō": {
("present", "ind", "active"): ["nōlō", "nōn vīs", "nōn vult", "nōlumus", "nōn vultis", "nōlunt"],
("present", "sbjv", "active"): ["nōlim", "nōlīs", "nōlit", "nōlīmus", "nōlītis", "nōlint"],
},
"ferō": {
("present", "ind", "active"): ["ferō", "fers", "fert", "ferimus", "fertis", "ferunt"],
("imperfect", "ind", "active"): ["ferēbam", "ferēbās", "ferēbat", "ferēbāmus", "ferēbātis", "ferēbant"],
("future", "ind", "active"): ["feram", "ferēs", "feret", "ferēmus", "ferētis", "ferent"],
("perfect", "ind", "active"): ["tulī", "tulistī", "tulit", "tulimus", "tulistis", "tulērunt"],
("present", "sbjv", "active"): ["feram", "ferās", "ferat", "ferāmus", "ferātis", "ferant"],
},
}
def conjugate(lemma, tense, mood, voice="active", person="third", number="singular"):
"""Return (surface, confidence). Perfect-passive forms are periphrastic and
handled in the realizer (sum + PPP); this returns synthetic forms only."""
lemma = lemma.strip()
i = _idx(person, number)
ir = _IRREG.get(lemma)
if ir:
tbl = ir.get((tense, mood, voice)) or ir.get((tense, mood, "active"))
if tbl and tbl[i]:
return tbl[i], "rule"
v = _VERBS.get(lemma)
if not v:
v = _infer_principal_parts(lemma)
if not v:
return lemma, "fallback"
conj, pstem, perfstem, supstem = v
# imperative (present active) 2sg / 2pl
if mood == "imp":
return _imperative(conj, pstem, person, number), "rule"
# perfect-system active
if tense in ("perfect", "pluperfect", "futureperfect") and voice == "active":
if not perfstem:
return lemma, "fallback"
end = _PERF_ACT.get((tense, mood))
if end:
return perfstem + end[i], "rule"
# present-system (active + passive)
if tense in ("present", "imperfect", "future"):
form = _present_system(conj, pstem, tense, mood, voice, person, number)
if form:
return form, "rule"
return lemma, "fallback"
def _imperative(conj, pstem, person, number):
if number == "singular":
return {1: pstem + "ā", 2: pstem + "ē", 3: pstem + "e",
"3io": pstem + "e", 4: pstem + "ī"}[conj]
return {1: pstem + "āte", 2: pstem + "ēte", 3: pstem + "ite",
"3io": pstem + "ite", 4: pstem + "īte"}[conj]
def _infer_principal_parts(lemma):
"""OOV fallback: infer conjugation + stems from the 1sg-present citation form.
Perfect/supine stems are guessed regularly (often wrong for 3rd conj) and the
resulting forms are still returned as 'rule' but the realizer down-weights."""
if lemma.endswith("ō"):
base = lemma[:-1]
# can't distinguish conj from 1sg alone reliably; default by ending vowel
if base.endswith("i"):
return ("3io", base[:-1], base[:-1] + "īv", base[:-1] + "īt")
return (3, base, base + "s", base + "t")
return None
# ── PUBLIC: participles ─────────────────────────────────────────────────────────
def participle(lemma, kind, case="nom", gender="m", number="singular"):
"""kind: 'prs' (present active, -ns/-ntis), 'pfv' (perfect passive, -tus),
'fut' (future active, -tūrus). Declined as an adjective via rule endings.
Returns (form, conf)."""
v = _VERBS.get(lemma)
if not v:
return lemma, "fallback"
conj, pstem, perfstem, supstem = v
if kind == "pfv":
if not supstem:
return lemma, "fallback"
base = supstem[:-1] if supstem.endswith("t") or supstem.endswith("s") else supstem
stem = supstem # supine stem already ends in t/s: amāt- -> amātus
return _decline_us_a_um(stem, case, gender, number), "rule"
if kind == "fut":
if not supstem:
return lemma, "fallback"
return _decline_us_a_um(supstem + "ūr", case, gender, number), "rule"
if kind == "prs":
# present active participle: stem + ns (nom), stem + nt- (oblique), 3rd-decl
pv = {1: "ā", 2: "ē", 3: "ē", "3io": "", 4: ""}[conj]
ntstem = pstem + pv + "nt"
return _decline_pres_ptcp(pstem + pv, case, gender, number), "rule"
return lemma, "fallback"
def _decline_us_a_um(stem, case, gender, number):
"""Decline a -us/-a/-um adjective/participle stem (2-1-2 declension)."""
C = _CASE_MAP.get(case, case.upper())
end = {
("NOM", "m", "singular"): "us", ("NOM", "f", "singular"): "a", ("NOM", "n", "singular"): "um",
("GEN", "m", "singular"): "ī", ("GEN", "f", "singular"): "ae", ("GEN", "n", "singular"): "ī",
("DAT", "m", "singular"): "ō", ("DAT", "f", "singular"): "ae", ("DAT", "n", "singular"): "ō",
("ACC", "m", "singular"): "um", ("ACC", "f", "singular"): "am", ("ACC", "n", "singular"): "um",
("ABL", "m", "singular"): "ō", ("ABL", "f", "singular"): "ā", ("ABL", "n", "singular"): "ō",
("VOC", "m", "singular"): "e", ("VOC", "f", "singular"): "a", ("VOC", "n", "singular"): "um",
("NOM", "m", "plural"): "ī", ("NOM", "f", "plural"): "ae", ("NOM", "n", "plural"): "a",
("GEN", "m", "plural"): "ōrum", ("GEN", "f", "plural"): "ārum", ("GEN", "n", "plural"): "ōrum",
("DAT", "m", "plural"): "īs", ("DAT", "f", "plural"): "īs", ("DAT", "n", "plural"): "īs",
("ACC", "m", "plural"): "ōs", ("ACC", "f", "plural"): "ās", ("ACC", "n", "plural"): "a",
("ABL", "m", "plural"): "īs", ("ABL", "f", "plural"): "īs", ("ABL", "n", "plural"): "īs",
("VOC", "m", "plural"): "ī", ("VOC", "f", "plural"): "ae", ("VOC", "n", "plural"): "a",
}.get((C, gender, number), "us")
return stem + end
def _decline_pres_ptcp(stem, case, gender, number):
"""Present active participle (amāns, amantis) — 3rd-declension, stem+ns/nt."""
C = _CASE_MAP.get(case, case.upper())
if C == "NOM" and number == "singular":
return stem + "ns"
if C == "VOC" and number == "singular":
return stem + "ns"
base = stem + "nt"
end = {
("GEN", "singular"): "is", ("DAT", "singular"): "ī",
("ACC", "singular"): "em" if gender != "n" else "",
("ABL", "singular"): "e",
("NOM", "plural"): "ēs" if gender != "n" else "ia",
("GEN", "plural"): "ium", ("DAT", "plural"): "ibus",
("ACC", "plural"): "ēs" if gender != "n" else "ia",
("ABL", "plural"): "ibus", ("VOC", "plural"): "ēs",
}.get((C, number), "is")
if C == "ACC" and number == "singular" and gender == "n":
return stem + "ns"
return base + end
def infinitive(lemma, tense="present", voice="active"):
lemma = lemma.strip()
if lemma == "sum":
return ("esse", "rule") if tense == "present" else ("fuisse", "rule")
v = _VERBS.get(lemma)
if not v:
return lemma, "fallback"
conj, pstem, perfstem, supstem = v
if tense == "present":
if voice == "active":
return _active_infinitive_stem(conj, pstem).rstrip() + \
("re" if conj != 3 and conj != "3io" else "re"), "rule"
# passive present infinitive
base = {1: pstem + "ā", 2: pstem + "ē", 4: pstem + "ī"}.get(conj)
if base:
return base + "", "rule"
return pstem + "ī", "rule" # 3rd: regī
if tense == "perfect" and voice == "active" and perfstem:
return perfstem + "isse", "rule"
return lemma, "fallback"
def lexicon_stats():
return {
"noun_adj_source": "UniMorph Latin (github.com/unimorph/lat, CC-BY-SA 3.0)",
"verb_source": "rule-based 4-conjugation engine over curated attested "
"principal parts (UniMorph verb list is a 947-lemma sample "
"MISSING all core verbs — amō/sum/videō absent)",
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
"curated_verb_lemmas": len(_VERBS) + len(_IRREG),
"gender_inference": "declension-based (nom+gen endings) + curated exceptions",
}
if __name__ == "__main__":
import json
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
print("\n-- noun declension puella (1st, fem) --")
for c in ("nom", "gen", "dat", "acc", "abl", "voc"):
print(f" {c}: sg={decline_noun('puella', c, 'singular')[0]:10} "
f"pl={decline_noun('puella', c, 'plural')[0]}")
print("\n-- rēx (3rd, m):", [decline_noun('rēx', c, 'singular')[0] for c in ('nom','gen','dat','acc','abl')])
print("-- gender: puella=", noun_gender("puella"), "rēx=", noun_gender("rēx"),
"bellum=", noun_gender("bellum"), "corpus=", noun_gender("corpus"),
"manus=", noun_gender("manus"), "diēs=", noun_gender("diēs"))
print("\n-- conjugate videō (2nd) present ind active --")
for p in ("first", "second", "third"):
for n in ("singular", "plural"):
print(f" {p[:3]}.{n[:2]}: {conjugate('videō','present','ind','active',p,n)[0]}")
print("-- amō forms:", conjugate("amō","present","ind","active","first","singular")[0],
conjugate("amō","imperfect","ind","active","third","plural")[0],
conjugate("amō","future","ind","active","first","singular")[0],
conjugate("amō","perfect","ind","active","third","singular")[0])
print("-- sum:", [conjugate("sum","present","ind","active",p,"singular")[0] for p in ("first","second","third")])
print("-- participle amō pfv acc.f.sg:", participle("amō","pfv","acc","f","singular")[0])
print("-- infinitive amō:", infinitive("amō")[0], "| regō pass:", infinitive("regō", voice="passive")[0])
+538
View File
@@ -0,0 +1,538 @@
"""morphology_pt_full.py — production-grade Brazilian-Portuguese morphological generator.
NOT a toy. Backed by two real, broad, Wiktionary-lineage lexicons:
VERBS UniMorph Portuguese (github.com/unimorph/por, CC-BY-SA 3.0)
4,001 verb lemmas × full paradigm (283,991 finite/non-finite forms +
20,005 participle forms). Every mood/tense pt actually inflects:
indicative present / preterite (PST;PFV) / imperfect (PST;IPFV) /
pluperfect-simple (PST;PRF) / future,
conditional (futuro do pretérito),
subjunctive present / imperfect / FUTURE (PT-specific live tense),
affirmative + negative imperative,
PERSONAL infinitive (V;{p};{n};NFIN a PT-specific finite-ish form),
past participle (4 gender/number forms) + gerúndio (V.PTCP;PRS).
NOUNS + ADJECTIVES kaikki.org Portuguese (Wiktionary extract, same lineage)
81,138 noun lemmas WITH inherent gender + real (often irregular) plural
so -ão-ões / -ãos / -ães / -õos is resolved PER LEMMA by Wiktionary,
never guessed (mãomãos, pãopães, coraçãocorações).
40,252 adjective lemmas with real feminine + masc/fem plural forms.
Fallbacks (degrade, never crash, on out-of-vocabulary input):
verbs : rule generator for regular -ar/-er/-ir paradigms
nouns : gender heuristic (endings) + rule pluralization (with -ão FLAGGED)
adjs : -o/-a gender rule + rule pluralization
Confidence flag on every form:
"lexicon" straight from UniMorph/kaikki (trust: high)
"rule" deterministic rule (trust: medium)
"fallback" could not inflect; returned lemma (trust: low -> FLAG)
Public API (used by realizer_pt.py):
conjugate(lemma, mood, tense, person, number) -> (form, conf)
personal_infinitive(lemma, person, number) -> (form, conf)
participle(lemma, gender="m", number="singular") -> (form, conf)
gerund(lemma) -> (form, conf)
noun_gender(lemma) -> "m"|"f"
inflect_noun(lemma, number, gender=None) -> (form, conf)
inflect_adj(lemma, gender, number) -> (form, conf)
lexicon_stats() -> dict
"""
import json
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "por.unimorph")
_KAIKKI = os.path.join(_HERE, "data", "kaikki_pt.jsonl")
_CACHE = os.path.join(_HERE, "data", "pt_morph_cache.pkl")
# ── mood/tense pair -> UniMorph feature triple (a in tag; b in tag; c in tag) ────
_VERB_KEYMAP = {
("ind", "present"): ("IND", "PRS", None),
("ind", "preterite"): ("IND", "PST", "PFV"),
("ind", "imperfect"): ("IND", "PST", "IPFV"),
("ind", "pluperfect"): ("IND", "PST", "PRF"), # simple mais-que-perfeito
("ind", "future"): ("IND", "FUT", None),
("ind", "conditional"): ("COND", None, None),
("sbjv", "present"): ("SBJV", "PRS", None),
("sbjv", "imperfect"): ("SBJV", "PST", "IPFV"),
("sbjv", "future"): ("SBJV", "FUT", None), # PT-specific
("imp", "affirmative"): ("IMP", "POS", None),
("imp", "negative"): ("IMP", "NEG", None),
}
_PERSON = {"first": "1", "second": "2", "third": "3"}
_NUMBER = {"singular": "SG", "plural": "PL"}
def _feat_set(tag):
return set(tag.split(";"))
# ── build the compact lexicon from UniMorph (verbs) + kaikki (nouns/adjs) ────────
def _build_verbs():
verbs = {} # (lemma, "mood|tense|person|number") -> form
pinf = {} # (lemma, "person|number") -> personal-infinitive form
part = {} # lemma -> {("m","SG"): form, ...} past participle
ger = {} # lemma -> gerúndio
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
f = _feat_set(tag)
head = tag.split(";")[0]
if head == "V.PTCP":
if "PST" in f: # past participle: falado/falada/falados/faladas
g = "m" if "MASC" in f else ("f" if "FEM" in f else "m")
num = "SG" if "SG" in f else ("PL" if "PL" in f else "SG")
part.setdefault(lemma, {})[(g, num)] = form
elif "PRS" in f: # gerúndio: falando
ger.setdefault(lemma, form)
continue
if head != "V":
continue
# personal / impersonal infinitive
if "NFIN" in f:
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person and number:
pinf[(lemma, f"{person}|{number}")] = form
continue
# finite forms
mt = None
for (mood, tense), (a, b, c) in _VERB_KEYMAP.items():
if a not in f:
continue
if b is not None and b not in f:
continue
if c is not None and c not in f:
continue
# IND;PST needs exactly PFV|IPFV|PRF — reject if the required one absent
mt = (mood, tense)
break
if mt is None:
continue
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person is None or number is None:
continue
verbs.setdefault((lemma, f"{mt[0]}|{mt[1]}|{person}|{number}"), form)
return verbs, pinf, part, ger
def _kaikki_gender(arg):
if not arg:
return None
a = arg.lower()
if a.startswith("f"):
return "f"
if a.startswith("m"):
return "m"
return None
def _build_nouns_adjs():
nouns = {} # lemma -> {"g","SG","PL"}
adjs = {} # lemma -> {("m","SG"),("f","SG"),("m","PL"),("f","PL")}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
pos = d.get("pos")
word = d.get("word", "")
if not word or " " in word: # skip multiword entries
continue
forms = d.get("forms", []) or []
if pos == "noun":
ht = d.get("head_templates") or []
g = None
if ht:
g = _kaikki_gender((ht[0].get("args") or {}).get("1"))
if g is None:
tags = d.get("tags") or []
if "feminine" in tags:
g = "f"
elif "masculine" in tags:
g = "m"
pl = None
for x in forms:
t = x.get("tags") or []
if "plural" in t and "alternative" not in t and "obsolete" not in t:
pl = x.get("form")
break
# first entry wins; but a later entry with a plural fills a gap
if word not in nouns:
nouns[word] = {"g": g, "SG": word, "PL": pl}
else:
cur = nouns[word]
if cur.get("g") is None and g:
cur["g"] = g
if not cur.get("PL") and pl:
cur["PL"] = pl
elif pos == "adj":
d0 = adjs.setdefault(word, {})
d0.setdefault(("m", "SG"), word)
for x in forms:
t = set(x.get("tags") or [])
fm = x.get("form")
if not fm or ("alternative" in t) or ("obsolete" in t):
continue
if "comparative" in t or "superlative" in t or \
"diminutive" in t or "augmentative" in t:
continue
if "feminine" in t and "plural" in t:
d0[("f", "PL")] = fm
elif "masculine" in t and "plural" in t:
d0[("m", "PL")] = fm
elif "feminine" in t:
d0[("f", "SG")] = fm
elif "plural" in t: # invariant-gender adj (feliz -> felizes)
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
return nouns, adjs
def _build_cache():
verbs, pinf, part, ger = _build_verbs()
nouns, adjs = _build_nouns_adjs()
data = {"verbs": verbs, "pinf": pinf, "part": part, "ger": ger,
"nouns": nouns, "adjs": adjs}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE):
newest_src = max(os.path.getmtime(_UNIMORPH),
os.path.getmtime(_KAIKKI) if os.path.exists(_KAIKKI) else 0)
if os.path.getmtime(_CACHE) >= newest_src:
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_VERBS, _PINF, _PART, _GER, _NOUNS, _ADJS = (
_LEX["verbs"], _LEX["pinf"], _LEX["part"], _LEX["ger"],
_LEX["nouns"], _LEX["adjs"])
# ── regular-ending rule fallback (deterministic, last resort) ────────────────────
def _vclass(lemma):
return lemma[-2:] if lemma[-2:] in ("ar", "er", "ir") else None
def _stem(lemma):
return lemma[:-2]
# endings indexed [1sg,2sg,3sg,1pl,2pl,3pl]
_REG = {
("ind", "present", "ar"): ["o", "as", "a", "amos", "ais", "am"],
("ind", "present", "er"): ["o", "es", "e", "emos", "eis", "em"],
("ind", "present", "ir"): ["o", "es", "e", "imos", "is", "em"],
("ind", "preterite", "ar"): ["ei", "aste", "ou", "amos", "astes", "aram"],
("ind", "preterite", "er"): ["i", "este", "eu", "emos", "estes", "eram"],
("ind", "preterite", "ir"): ["i", "iste", "iu", "imos", "istes", "iram"],
("ind", "imperfect", "ar"): ["ava", "avas", "ava", "ávamos", "áveis", "avam"],
("ind", "imperfect", "er"): ["ia", "ias", "ia", "íamos", "íeis", "iam"],
("ind", "imperfect", "ir"): ["ia", "ias", "ia", "íamos", "íeis", "iam"],
("sbjv", "present", "ar"): ["e", "es", "e", "emos", "eis", "em"],
("sbjv", "present", "er"): ["a", "as", "a", "amos", "ais", "am"],
("sbjv", "present", "ir"): ["a", "as", "a", "amos", "ais", "am"],
("sbjv", "imperfect", "ar"): ["asse", "asses", "asse", "ássemos", "ásseis", "assem"],
("sbjv", "imperfect", "er"): ["esse", "esses", "esse", "êssemos", "êsseis", "essem"],
("sbjv", "imperfect", "ir"): ["isse", "isses", "isse", "íssemos", "ísseis", "issem"],
("sbjv", "future", "ar"): ["ar", "ares", "ar", "armos", "ardes", "arem"],
("sbjv", "future", "er"): ["er", "eres", "er", "ermos", "erdes", "erem"],
("sbjv", "future", "ir"): ["ir", "ires", "ir", "irmos", "irdes", "irem"],
}
# future & conditional attach to the FULL infinitive
_FUT = ["ei", "ás", "á", "emos", "eis", "ão"]
_COND = ["ia", "ias", "ia", "íamos", "íeis", "iam"]
def _slot_idx(person, number):
base = {"first": 0, "second": 1, "third": 2}[person]
return base + (0 if number == "singular" else 3)
def _rule_conjugate(lemma, mood, tense, person, number):
vc = _vclass(lemma)
if vc is None:
return None
st, i = _stem(lemma), _slot_idx(person, number)
if mood == "ind" and tense == "future":
return lemma + _FUT[i]
if mood == "ind" and tense == "conditional":
return lemma + _COND[i]
if mood == "imp": # affirmative tú/vocês imperative ~ subjunctive present
table = _REG.get(("sbjv", "present", vc))
if table and tense == "negative":
return st + table[i]
# affirmative 2sg = 3sg present indicative; others = subjunctive
pres = _REG.get(("ind", "present", vc))
if person == "second" and number == "singular":
return st + pres[2]
return st + table[i] if table else None
table = _REG.get((mood, tense, vc))
if table:
return st + table[i]
return None
# verified corrections to UniMorph data errors (each audited individually, not
# guessed). The three 1PL-present entries are glued-allomorph errors surfaced by a
# full-lexicon scan for a non-final "mos" in V;1;PL;IND;PRS forms (the ONLY three).
_VERB_FIX = {
("estar", "ind", "imperfect", "third", "plural"): "estavam", # was "estávam"
("estar", "ind", "present", "first", "plural"): "estamos", # was "estamosestámos"
("haver", "ind", "present", "first", "plural"): "havemos", # was "havemoshemos"
("ir", "ind", "present", "first", "plural"): "vamos", # was "vamosimos"
}
# ── PUBLIC: verb conjugation ─────────────────────────────────────────────────────
def conjugate(lemma, mood, tense, person, number):
"""Return (surface, confidence). mood in ind|sbjv|imp; tense per _VERB_KEYMAP."""
lemma = lemma.strip().lower()
fix = _VERB_FIX.get((lemma, mood, tense, person, number))
if fix:
return fix, "lexicon"
p, n = _PERSON.get(person), _NUMBER.get(number)
if p and n:
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}"))
if form:
# pt-BR normalization: UniMorph `por` carries the EUROPEAN spelling of
# the -ar 1pl PRETERITE (-ámos). Brazilian PT drops the accent
# (falámos->falamos, chegámos->chegamos) — 3,334/4,001 verbs affected.
if (mood == "ind" and tense == "preterite" and person == "first"
and number == "plural" and form.endswith("ámos")):
form = form[:-4] + "amos"
return form, "lexicon"
r = _rule_conjugate(lemma, mood, tense, person, number)
if r:
return r, "rule"
return lemma, "fallback"
def personal_infinitive(lemma, person, number):
"""PT personal (inflected) infinitive: para falarmos, ao chegarem."""
lemma = lemma.strip().lower()
p, n = _PERSON.get(person), _NUMBER.get(number)
if p and n:
form = _PINF.get((lemma, f"{p}|{n}"))
if form:
return form, "lexicon"
# rule: infinitive + personal endings (-, -es, -, -mos, -des, -em)
end = {("first", "singular"): "", ("second", "singular"): "es",
("third", "singular"): "", ("first", "plural"): "mos",
("second", "plural"): "des", ("third", "plural"): "em"}.get((person, number), "")
return lemma + end, "rule"
# ── PUBLIC: participle + gerund ───────────────────────────────────────────────────
def participle(lemma, gender="m", number="singular"):
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
d = _PART.get(lemma)
if d:
form = d.get((g, num)) or d.get(("m", "SG"))
if form:
return form, "lexicon"
if lemma.endswith("ar"):
base = lemma[:-2] + "ad"
elif lemma[-2:] in ("er", "ir"):
base = lemma[:-2] + "id"
else:
return lemma, "fallback"
suf = {"m|SG": "o", "f|SG": "a", "m|PL": "os", "f|PL": "as"}[f"{g}|{num}"]
return base + suf, "rule"
def gerund(lemma):
lemma = lemma.strip().lower()
if lemma in _GER:
return _GER[lemma], "lexicon"
if lemma.endswith("ar"):
return lemma[:-2] + "ando", "rule"
if lemma.endswith("er"):
return lemma[:-2] + "endo", "rule"
if lemma.endswith("ir"):
return lemma[:-2] + "indo", "rule"
return lemma, "fallback"
# ── PUBLIC: noun gender + number ─────────────────────────────────────────────────
_FEM_SUF = ("ção", "são", "ção", "dade", "tade", "agem", "igem", "ugem", "gem",
"ez", "eza", "ice", "ície", "tude", "ude", "âncbefore")
_FEM_SUF = ("ção", "são", "dade", "tade", "agem", "gem", "eza", "ez", "ice",
"tude", "ude", "ância", "ência", "ínia")
_MASC_SUF = ("ema", "oma", "ama", "grama", "eta", "ão") # Greek -ma etc. (mostly m)
def _gender_heuristic(noun):
for suf in _FEM_SUF:
if noun.endswith(suf):
return "f"
if noun.endswith(("ema", "oma", "ama")): # problema, idioma, programa
return "m"
if noun.endswith("a") or noun.endswith("ã"):
return "f"
if noun.endswith("o") or noun.endswith(("l", "r", "z", "m", "u", "i")):
return "m"
return "m"
def noun_gender(lemma):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if d and d.get("g"):
return d["g"]
return _gender_heuristic(lemma)
_INVARIANT_PL_SUF = ("s",) # paroxytones ending -s are invariant (o lápis / os lápis)
def _rule_plural(noun):
"""Deterministic PT pluralization. Returns (form, ok) where ok=False flags an
ambiguous -ão that should lower confidence (the lexicon normally resolves it)."""
if not noun:
return noun, True
if noun.endswith("ão"):
return noun[:-2] + "ões", False # majority rule, but AMBIGUOUS -> flag
if noun.endswith("m"):
return noun[:-1] + "ns", True # homem->homens, jardim->jardins
if noun.endswith("al"):
return noun[:-2] + "ais", True
if noun.endswith("el"):
return noun[:-2] + "éis", True
if noun.endswith("ol"):
return noun[:-2] + "óis", True
if noun.endswith("ul"):
return noun[:-2] + "uis", True
if noun.endswith("il"):
return noun[:-2] + "is", True # stressed (funil->funis); unstressed rarer
if noun.endswith(("r", "z")):
return noun + "es", True # flor->flores, luz->luzes
if noun.endswith("s"):
# paroxytone -s (lápis, ônibus) invariant; oxytone -s (país) -> -es
return noun, True
if noun.endswith(("a", "e", "i", "o", "u", "á", "é", "í", "ó", "ú", "ã")):
return noun + "s", True
return noun + "s", True
def inflect_noun(lemma, number, gender=None):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if number == "singular":
return (d["SG"] if d and d.get("SG") else lemma), ("lexicon" if d else "rule")
if d and d.get("PL"):
return d["PL"], "lexicon"
form, ok = _rule_plural(lemma)
return form, ("rule" if ok else "fallback")
# ── PUBLIC: adjective agreement ──────────────────────────────────────────────────
def inflect_adj(lemma, gender, number):
lemma = lemma.strip().lower()
g = "f" if gender == "f" else "m"
num = "SG" if number == "singular" else "PL"
d = _ADJS.get(lemma)
if d:
form = d.get((g, num))
if form:
return form, "lexicon"
# build a missing plural from this gender's singular
sg = d.get((g, "SG")) or d.get(("m", "SG")) or lemma
if num == "PL":
pl, ok = _rule_plural(sg)
return pl, ("rule" if ok else "fallback")
return sg, "lexicon"
# rule fallback: -o/-a gender, then pluralize
a = lemma
if g == "f":
if a.endswith("o"):
a = a[:-1] + "a"
elif a.endswith(("ês", "or")) and not a.endswith("ior"):
a = a + "a" # português->portuguesa, trabalhador->..a
if num == "PL":
a, ok = _rule_plural(a)
return a, ("rule" if ok else "fallback")
return a, "rule"
def lexicon_stats():
return {
"verb_source": "UniMorph Portuguese (github.com/unimorph/por)",
"noun_adj_source": "kaikki.org Portuguese (Wiktionary extract)",
"license": "CC-BY-SA (Wiktionary-derived)",
"verb_forms": len(_VERBS),
"verb_lemmas": len({k[0] for k in _VERBS}),
"personal_infinitive_forms": len(_PINF),
"participle_lemmas": len(_PART),
"gerund_lemmas": len(_GER),
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
}
if __name__ == "__main__":
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
tests = [
("falar", "ind", "present", "first", "singular", "falo"),
("comer", "ind", "present", "third", "plural", "comem"),
("partir", "ind", "present", "first", "plural", "partimos"),
("ser", "ind", "present", "third", "singular", "é"),
("ir", "ind", "preterite", "first", "singular", "fui"),
("ter", "ind", "future", "first", "singular", "terei"),
("fazer", "sbjv", "present", "first", "singular", "faça"),
("dormir", "ind", "present", "first", "singular", "durmo"),
("dar", "ind", "preterite", "third", "singular", "deu"),
("poder", "ind", "conditional", "first", "singular", "poderia"),
("fazer", "sbjv", "future", "third", "singular", "fizer"),
("estar", "ind", "present", "third", "singular", "está"),
]
ok = 0
for lemma, mood, tense, per, num, exp in tests:
got, conf = conjugate(lemma, mood, tense, per, num)
flag = "OK " if got == exp else "XX "
ok += got == exp
print(f" {flag}{lemma:8} {mood}/{tense} {per[:3]}.{num[:2]} -> {got:14} ({conf}) exp={exp}")
print(f"verb tests {ok}/{len(tests)}")
print(" gender: casa=", noun_gender("casa"), "problema=", noun_gender("problema"),
"mão=", noun_gender("mão"), "coração=", noun_gender("coração"),
"flor=", noun_gender("flor"))
print(" plural: mão->", inflect_noun("mão", "plural"),
"| pão->", inflect_noun("pão", "plural"),
"| animal->", inflect_noun("animal", "plural"),
"| coração->", inflect_noun("coração", "plural"))
print(" adj: bonito/f/sg->", inflect_adj("bonito", "f", "singular"),
"| feliz/m/pl->", inflect_adj("feliz", "m", "plural"),
"| português/f/sg->", inflect_adj("português", "f", "singular"))
print(" part: fazer/m/sg->", participle("fazer"), "| ger falar->", gerund("falar"))
print(" pinf falar 1pl->", personal_infinitive("falar", "first", "plural"))
+609
View File
@@ -0,0 +1,609 @@
# -*- coding: utf-8 -*-
"""morphology_ro_full.py — production-grade Romanian morphological generator.
Romanian is the BIG typological delta of the Romance family. The verb engine and
the confidence/fallback contract TRANSFER from the Italian sibling; the NOMINAL
system is genuinely new: Romanian has a SUFFIXED definite article, a preserved
NOM/ACC vs GEN/DAT case distinction, a NEUTER gender (masc-agreeing in SG,
fem-agreeing in PL), and a VOCATIVE. Those are grounded in real per-lemma data,
not guessed.
Real, Wiktionary-lineage lexical sources:
VERBS UniMorph Romanian (github.com/unimorph/ron, CC-BY-SA 3.0)
~1216 verb lemmas × paradigm, CLEAN orthography:
indicativ prezent / imperfect (PST;IPFV) / perfectul simplu (PST;PFV) /
conjunctiv prezent (SBJV;PRS, stored WITHOUT the '' particle),
participiu (V.PTCP;PST, INVARIABLE in the perfect compus),
gerunziu (V.CVB;PRS), infinitiv (NFIN), imperativ.
ro_irreg_verbs (embedded) high-frequency verbs UniMorph MISSES
(avea, vrea, da) + the auxiliary clitic paradigms the compound tenses need
(perfect-compus am/ai/a/am/ați/au, viitor voi/vei/va/vom/veți/vor,
condițional /ai/ar/am/ați/ar). Real standard forms.
NOUNS kaikki.org Romanian (Wiktionary extract, CC-BY-SA 3.0)
the FULL declension per lemma, cleanly tagged:
(nom/acc | gen/dat | vocative) × (indefinite | definite) × (sg | pl).
This is what makes the suffixed article LEXICALLY grounded (omomul,
casăcasa, băiatbăiatul, casei gen/dat, omule vocative). Inherent gender
m / f / n (NEUTER available directly) from the head template.
ADJECTIVES UniMorph Romanian ADJ
full case × gender(MASC/FEM/NEUT) × number × definiteness paradigm.
Fallbacks (degrade, never crash, on OOV): rule verb conjugation for -a/-ea/-e/-i/-î
classes, rule pluralization, rule suffixed-article by gender+ending. Every form
carries a confidence flag: "lexicon" | "rule" | "fallback".
Public API (used by realizer_ro.py):
conjugate(lemma, mood, tense, person, number) -> (form, conf)
aux(kind, person, number) -> str # perfect / future / conditional clitics
participle(lemma) -> (form, conf) # INVARIABLE
gerund(lemma) -> (form, conf)
noun_gender(lemma) -> "m"|"f"|"n"
definite_suffix(noun, gender, number, case) -> (form, conf) # rule engine
inflect_noun(lemma, number, gender=None, case="nomacc", definite=False) -> (form, conf)
inflect_adj(lemma, gender, number, case="nomacc", definite=False) -> (form, conf)
lexicon_stats() -> dict
"""
import json
import os
import pickle
_HERE = os.path.dirname(os.path.abspath(__file__))
_UNIMORPH = os.path.join(_HERE, "data", "ron.unimorph")
_KAIKKI = os.path.join(_HERE, "data", "kaikki_ro.jsonl")
_CACHE = os.path.join(_HERE, "data", "ro_morph_cache.pkl")
# ── (mood, tense) -> UniMorph feature set ─────────────────────────────────────────
_VERB_KEYMAP = {
("ind", "present"): {"IND", "PRS"},
("ind", "imperfect"): {"IND", "PST", "IPFV"},
("ind", "perfect_s"): {"IND", "PST", "PFV"}, # perfectul simplu (regional/lit.)
("sbjv", "present"): {"SBJV", "PRS"},
("imp", "affirmative"): {"POS", "IMP"},
}
_PERSON = {"first": "1", "second": "2", "third": "3"}
_NUMBER = {"singular": "SG", "plural": "PL"}
def _feat_set(tag):
return set(tag.split(";"))
# ── high-frequency irregulars UniMorph misses + auxiliary clitic paradigms ────────
# Real standard Romanian forms (textbook paradigms).
_IRREG = {
"avea": {
"ind|present|1|SG": "am", "ind|present|2|SG": "ai", "ind|present|3|SG": "are",
"ind|present|1|PL": "avem", "ind|present|2|PL": "aveți", "ind|present|3|PL": "au",
"ind|imperfect|1|SG": "aveam", "ind|imperfect|2|SG": "aveai",
"ind|imperfect|3|SG": "avea", "ind|imperfect|1|PL": "aveam",
"ind|imperfect|2|PL": "aveați", "ind|imperfect|3|PL": "aveau",
"sbjv|present|3|SG": "aibă", "sbjv|present|3|PL": "aibă",
"sbjv|present|1|SG": "am", "sbjv|present|2|SG": "ai",
"sbjv|present|1|PL": "avem", "sbjv|present|2|PL": "aveți",
"part": "avut", "ger": "având",
},
"vrea": {
"ind|present|1|SG": "vreau", "ind|present|2|SG": "vrei", "ind|present|3|SG": "vrea",
"ind|present|1|PL": "vrem", "ind|present|2|PL": "vreți", "ind|present|3|PL": "vor",
"ind|imperfect|1|SG": "voiam", "ind|imperfect|3|SG": "voia",
"sbjv|present|3|SG": "vrea", "sbjv|present|3|PL": "vrea",
"part": "vrut", "ger": "vrând",
},
"da": {
"ind|present|1|SG": "dau", "ind|present|2|SG": "dai", "ind|present|3|SG": "",
"ind|present|1|PL": "dăm", "ind|present|2|PL": "dați", "ind|present|3|PL": "dau",
"ind|imperfect|1|SG": "dădeam", "ind|imperfect|3|SG": "dădea",
"sbjv|present|3|SG": "dea", "sbjv|present|3|PL": "dea",
"part": "dat", "ger": "dând",
},
"fi": { # a fi — present is in UniMorph but keep participle + subjunctive here
"part": "fost", "ger": "fiind",
"sbjv|present|1|SG": "fiu", "sbjv|present|2|SG": "fii", "sbjv|present|3|SG": "fie",
"sbjv|present|1|PL": "fim", "sbjv|present|2|PL": "fiți", "sbjv|present|3|PL": "fie",
"ind|imperfect|1|SG": "eram", "ind|imperfect|2|SG": "erai",
"ind|imperfect|3|SG": "era", "ind|imperfect|1|PL": "eram",
"ind|imperfect|2|PL": "erați", "ind|imperfect|3|PL": "erau",
},
}
# auxiliary clitic paradigms (person,number)->form
_AUX = {
"perfect": {("first", "singular"): "am", ("second", "singular"): "ai",
("third", "singular"): "a", ("first", "plural"): "am",
("second", "plural"): "ați", ("third", "plural"): "au"},
"future": {("first", "singular"): "voi", ("second", "singular"): "vei",
("third", "singular"): "va", ("first", "plural"): "vom",
("second", "plural"): "veți", ("third", "plural"): "vor"},
"conditional": {("first", "singular"): "", ("second", "singular"): "ai",
("third", "singular"): "ar", ("first", "plural"): "am",
("second", "plural"): "ați", ("third", "plural"): "ar"},
}
def aux(kind, person, number):
return _AUX[kind][(person, number)]
# ── build verb lexicon from UniMorph ──────────────────────────────────────────────
def _build_verbs():
verbs, part, ger = {}, {}, {}
with open(_UNIMORPH, encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line or "\t" not in line:
continue
parts = line.split("\t")
if len(parts) != 3:
continue
lemma, form, tag = parts
f = _feat_set(tag)
head = tag.split(";")[0]
if head == "V.PTCP":
if "PST" in f:
part.setdefault(lemma, form)
continue
if head == "V.CVB":
if "PRS" in f:
ger.setdefault(lemma, form)
continue
if head != "V":
continue
person = next((p for p in ("1", "2", "3") if p in f), None)
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
if person is None or number is None:
continue
# conjunctiv forms in UniMorph carry a leading 'să ' — strip it
surf = form
if surf.startswith(""):
surf = surf[3:]
for (mood, tense), req in _VERB_KEYMAP.items():
if not req <= f:
continue
if tense == "imperfect" and "PFV" in f:
continue
if tense == "perfect_s" and "IPFV" in f:
continue
# keep IND;PRS out of the PRF slot (mai-mult-ca-perfect etc. ignored)
if {"IND", "PRS"} <= req and "PRF" in f:
continue
verbs.setdefault((lemma, f"{mood}|{tense}|{person}|{number}"), surf)
break
return verbs, part, ger
# ── kaikki nouns: full declension paradigm per lemma ──────────────────────────────
_EXCL = {"alternative", "archaic", "obsolete", "regional", "dialectal", "rare",
"table-tags", "inflection-template", "error-unrecognized-form",
"diminutive", "augmentative", "informal"}
def _noun_key(tagset):
if tagset & _EXCL:
return None
if "vocative" in tagset:
case = "voc"
elif "genitive" in tagset or "dative" in tagset:
case = "gendat"
elif "nominative" in tagset or "accusative" in tagset:
case = "nomacc"
else:
return None
definite = "definite" in tagset and "indefinite" not in tagset
number = "PL" if "plural" in tagset else ("SG" if "singular" in tagset else None)
if number is None:
return None
return (case, definite, number)
def _build_nouns():
nouns = {} # lemma -> {"g":..., para:{(case,def,num):form}, "PL":plain_plural}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
if d.get("pos") != "noun":
continue
word = d.get("word", "")
if not word or " " in word:
continue
ht = d.get("head_templates") or []
g = None
if ht:
a = str((ht[0].get("args") or {}).get("1") or "").lower()
if a[:1] in ("m", "f", "n"):
g = a[:1]
entry = nouns.setdefault(word, {"g": g, "para": {}, "PL": None})
if entry["g"] is None and g:
entry["g"] = g
for x in (d.get("forms") or []):
fm = x.get("form")
tg = set(x.get("tags") or [])
if not fm or fm in ("-", "#", "") or " " in fm:
continue
if tg == {"plural"} and not entry["PL"]:
entry["PL"] = fm
k = _noun_key(tg)
if k and k not in entry["para"]:
entry["para"][k] = fm
return nouns
# ── adjectives from kaikki (UniMorph ron ADJ is sparse AND mis-tagged; kaikki is
# clean: the 4-form agreement pattern bun/bună/buni/bune). Neuter maps sg->masc,
# pl->fem, so 4 forms (m/f × SG/PL) fully cover it. ────────────────────────────
def _build_adjs():
adjs = {} # lemma -> {(gender,number): form} gender in {m,f}
with open(_KAIKKI, encoding="utf-8") as fh:
for line in fh:
try:
d = json.loads(line)
except Exception:
continue
if d.get("pos") != "adj":
continue
word = d.get("word", "")
if not word or " " in word:
continue
d0 = adjs.setdefault(word, {})
d0.setdefault(("m", "SG"), word) # masc sg = headword
for x in (d.get("forms") or []):
fm = x.get("form")
t = set(x.get("tags") or [])
if not fm or " " in fm or fm in ("-", "#") or (t & _EXCL):
continue
if "definite" in t or "genitive" in t or "dative" in t:
continue # keep indefinite nom/acc agr set
pl = "plural" in t
fem = "feminine" in t
masc = "masculine" in t
if fem and pl:
d0.setdefault(("f", "PL"), fm)
elif masc and pl:
d0.setdefault(("m", "PL"), fm)
elif fem and not pl:
d0.setdefault(("f", "SG"), fm)
elif pl and not fem and not masc: # bare plural -> both genders
d0.setdefault(("m", "PL"), fm)
d0.setdefault(("f", "PL"), fm)
return adjs
def _build_cache():
verbs, part, ger = _build_verbs()
nouns = _build_nouns()
adjs = _build_adjs()
data = {"verbs": verbs, "part": part, "ger": ger, "nouns": nouns, "adjs": adjs}
try:
with open(_CACHE, "wb") as fh:
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
except OSError:
pass
return data
def _load():
if os.path.exists(_CACHE):
srcs = [_UNIMORPH, _KAIKKI]
newest = max(os.path.getmtime(s) for s in srcs if os.path.exists(s))
if os.path.getmtime(_CACHE) >= newest:
try:
with open(_CACHE, "rb") as fh:
return pickle.load(fh)
except Exception:
pass
return _build_cache()
_LEX = _load()
_VERBS, _PART, _GER, _NOUNS, _ADJS = (
_LEX["verbs"], _LEX["part"], _LEX["ger"], _LEX["nouns"], _LEX["adjs"])
# ── rule verb conjugation fallback ────────────────────────────────────────────────
def _vclass(lemma):
if lemma.endswith("a"):
return "a"
if lemma.endswith("ea"):
return "ea"
if lemma.endswith("e"):
return "e"
if lemma.endswith("i"):
return "i"
if lemma.endswith("î"):
return "î"
return None
# regular present endings by class [1sg,2sg,3sg,1pl,2pl,3pl]
_REG_PRS = {
"a": ["", "i", "ă", "ăm", "ați", "ă"], # a lucra type (simplified)
"ea": ["", "i", "e", "em", "eți", "", ],
"e": ["", "i", "e", "em", "eți", ""],
"i": ["esc", "ești", "ește", "im", "iți", "esc"], # -i type (a vorbi)
"î": ["ăsc", "ăști", "ăște", "âm", "âți", "ăsc"],
}
_SLOT = {("first", "singular"): 0, ("second", "singular"): 1, ("third", "singular"): 2,
("first", "plural"): 3, ("second", "plural"): 4, ("third", "plural"): 5}
def _rule_conjugate(lemma, mood, tense, person, number):
vc = _vclass(lemma)
if vc is None:
return None
i = _SLOT[(person, number)]
body = lemma[:-len(vc)]
if mood == "ind" and tense == "present":
end = _REG_PRS[vc][i]
return body + end
if mood == "ind" and tense == "imperfect":
# -a/-i/-î -> stem + a/eai...; -e/-ea -> eam. Simplified regular imperfect.
stem = body
endings = {"a": ["am", "ai", "a", "am", "ați", "au"],
"i": ["eam", "eai", "ea", "eam", "eați", "eau"],
"î": ["am", "ai", "a", "am", "ați", "au"],
"e": ["eam", "eai", "ea", "eam", "eați", "eau"],
"ea": ["eam", "eai", "ea", "eam", "eați", "eau"]}[vc]
return stem + endings[i]
return None
# ── PUBLIC verb API ───────────────────────────────────────────────────────────────
def conjugate(lemma, mood, tense, person, number):
lemma = lemma.strip().lower()
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{_NUMBER.get(number,'?')}"
ir = _IRREG.get(lemma)
if ir and key in ir:
return ir[key], "lexicon"
form = _VERBS.get((lemma, key))
if form:
return form, "lexicon"
r = _rule_conjugate(lemma, mood, tense, person, number)
if r is not None:
return r, "rule"
return lemma, "fallback"
def participle(lemma):
"""Past participle — INVARIABLE in the perfect compus (am mers, am văzut)."""
lemma = lemma.strip().lower()
ir = _IRREG.get(lemma)
if ir and "part" in ir:
return ir["part"], "lexicon"
if lemma in _PART:
return _PART[lemma], "lexicon"
vc = _vclass(lemma)
if vc == "a":
return lemma[:-1] + "at", "rule"
if vc in ("ea",):
return lemma[:-2] + "ut", "rule"
if vc == "i":
return lemma[:-1] + "it", "rule"
if vc == "î":
return lemma[:-1] + "ât", "rule"
if vc == "e":
return lemma[:-1] + "ut", "rule"
return lemma, "fallback"
def gerund(lemma):
lemma = lemma.strip().lower()
ir = _IRREG.get(lemma)
if ir and "ger" in ir:
return ir["ger"], "lexicon"
if lemma in _GER:
return _GER[lemma], "lexicon"
vc = _vclass(lemma)
if vc in ("a", "î"):
return lemma[:-1] + "ând", "rule"
if vc in ("ea", "e", "i"):
return lemma[:-len(vc)] + "ind", "rule"
return lemma, "fallback"
# ── noun gender ───────────────────────────────────────────────────────────────────
def noun_gender(lemma):
lemma = lemma.strip().lower()
d = _NOUNS.get(lemma)
if d and d.get("g") in ("m", "f", "n"):
return d["g"]
if lemma.endswith(("ă", "a", "e")):
return "f"
return "m"
# ── SUFFIXED DEFINITE ARTICLE — rule engine (fallback for OOV nouns) ───────────────
def definite_suffix(noun, gender, number, case="nomacc"):
"""Attach the enclitic definite article by gender + ending. Returns (form, conf).
This is the headline Romanian-specific engine extension."""
n = noun
g = gender
if number == "singular":
if g in ("m", "n"):
if case == "gendat":
# masc/neut gen-dat definite: -lui
if n.endswith("e"):
return n + "lui", "rule" # câine -> câinelui
if n.endswith("u"):
return n + "lui", "rule"
return n + "ului", "rule" # om -> omului
# nom/acc
if n.endswith("e"):
return n + "le", "rule" # câine -> câinele
if n.endswith("u"):
return n + "l", "rule" # codru -> codrul
if n.endswith("i"):
return n + "ul", "rule"
return n + "ul", "rule" # om -> omul
# feminine singular
if case == "gendat":
# fem gen/dat definite = plural-stem + i (casei, fetei) — needs plural;
# approximated as: -ă->-ei, -e->-ei, -a->-alei
if n.endswith("ă"):
return n[:-1] + "ei", "rule" # casă -> casei
if n.endswith("e"):
return n[:-1] + "ei", "rule" # carte -> cărții(approx cartei)
if n.endswith("a"):
return n[:-1] + "lei", "rule"
return n + "i", "rule"
# fem nom/acc
if n.endswith("ă"):
return n[:-1] + "a", "rule" # casă -> casa
if n.endswith("e"):
return n[:-1] + "ea", "rule" # carte -> cartea
if n.endswith("a"):
return n + "ua", "rule" # stea -> steaua
if n.endswith("i"):
return n + "a", "rule"
return n + "a", "rule"
# plural
if case == "gendat":
base = noun
return base + "lor", "rule" # -lor for all gen/dat pl
if g == "m":
return noun + "i", "rule" # oameni -> oamenii (+i)
return noun + "le", "rule" # case -> casele, trenuri->trenurile
# ── rule pluralization (fallback) ─────────────────────────────────────────────────
def _rule_plural(noun, gender):
if gender == "f":
if noun.endswith("ă"):
return noun[:-1] + "e"
if noun.endswith("e"):
return noun[:-1] + "i"
if noun.endswith("a"):
return noun[:-1] + "le"
return noun + "e"
if gender == "n":
return noun + "uri"
# masculine
if noun.endswith(("e",)):
return noun[:-1] + "i"
return noun + "i"
# ── PUBLIC noun inflection ────────────────────────────────────────────────────────
def inflect_noun(lemma, number, gender=None, case="nomacc", definite=False):
lemma = lemma.strip().lower()
g = gender or noun_gender(lemma)
d = _NOUNS.get(lemma)
numk = "SG" if number == "singular" else "PL"
if d:
if case == "voc":
form = d["para"].get(("voc", True, numk)) or d["para"].get(("voc", False, numk))
if form:
return form, "lexicon"
# try the exact paradigm cell from kaikki (lexically grounded)
form = d["para"].get((case, definite, numk))
if form:
return form, "lexicon"
# indefinite fallbacks from the paradigm
if not definite:
form = d["para"].get(("nomacc", False, numk))
if form:
return form, "lexicon"
if numk == "PL" and d.get("PL"):
return d["PL"], "lexicon"
if numk == "SG":
return lemma, "lexicon"
# rule path
base = lemma if number == "singular" else _rule_plural(lemma, g)
if definite:
return definite_suffix(base, g, number, case)
return base, ("rule" if d is None else "lexicon")
# ── PUBLIC adjective agreement ────────────────────────────────────────────────────
def _neuter_map(gender, number):
# neuter agrees masculine in SG, feminine in PL
if gender == "n":
return "m" if number == "singular" else "f"
return gender
def inflect_adj(lemma, gender, number, case="nomacc", definite=False):
lemma = lemma.strip().lower()
numk = "SG" if number == "singular" else "PL"
eg = _neuter_map(gender, number) # neuter -> masc(SG)/fem(PL)
d = _ADJS.get(lemma)
if d:
form = d.get((eg, numk))
if form:
return form, "lexicon"
# rule fallback: 4-form pattern bun/bună/buni/bune keyed by effective gender
a = lemma
if number == "singular":
if eg == "f":
if a.endswith("e"):
return a, "rule" # mare invariant sg
if a.endswith("u"):
return a[:-1] + "ă", "rule" # nou -> nouă
if a.endswith("ă"):
return a, "rule"
return a + "ă", "rule" # bun -> bună
return a, "rule" # masc/neut sg = lemma
# plural
if eg == "f":
if a.endswith("e"):
return a[:-1] + "i", "rule" # mare -> mari
if a.endswith("u"):
return a[:-1] + "e", "rule" # nou -> noue (approx; 'noi' irr)
if a.endswith("ă"):
return a[:-1] + "e", "rule"
return a + "e", "rule" # bun -> bune
# masc/neut(SG-only)->here masc pl -> -i
if a.endswith("e"):
return a[:-1] + "i", "rule" # mare -> mari
if a.endswith("u"):
return a[:-1] + "i", "rule"
return a + "i", "rule" # bun -> buni
def lexicon_stats():
return {
"verb_source": "UniMorph Romanian (github.com/unimorph/ron) + curated "
"irregulars (avea/vrea/da + aux clitic paradigms)",
"noun_source": "kaikki.org Romanian — full case/definite/vocative declension",
"adj_source": "UniMorph Romanian ADJ (case×gender×number×definiteness)",
"license": "CC-BY-SA 3.0 (Wiktionary/UniMorph lineage)",
"unimorph_verb_forms": len(_VERBS),
"unimorph_verb_lemmas": len({k[0] for k in _VERBS}),
"irregular_verb_lemmas": len(_IRREG),
"participle_lemmas": len(_PART),
"noun_lemmas": len(_NOUNS),
"adj_lemmas": len(_ADJS),
}
if __name__ == "__main__":
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
print("\n── SUFFIXED DEFINITE ARTICLE (the headline delta) ──")
for n, g in [("om", "m"), ("băiat", "m"), ("casă", "f"), ("carte", "f"),
("tren", "n"), ("student", "m"), ("floare", "f")]:
sg = inflect_noun(n, "singular", g, "nomacc", True)
pl = inflect_noun(n, "plural", g, "nomacc", True)
gd = inflect_noun(n, "singular", g, "gendat", True)
vo = inflect_noun(n, "singular", g, "voc", False)
print(f" {n:8}({g}) def.sg={sg[0]:12} def.pl={pl[0]:14} "
f"gen/dat.sg={gd[0]:12} voc={vo[0]}")
print("\n── NEUTER split agreement (tren: masc SG / fem PL) ──")
print(" tren nou ->", inflect_noun("tren", "singular", "n")[0],
inflect_adj("nou", "n", "singular")[0])
print(" trenuri noi->", inflect_noun("tren", "plural", "n")[0],
inflect_adj("nou", "n", "plural")[0])
print("\n── verbs ──")
for l, m, t, p, n, in [("merge", "ind", "present", "third", "singular"),
("avea", "ind", "present", "first", "singular"),
("fi", "ind", "present", "third", "singular"),
("vorbi", "ind", "present", "third", "plural"),
("face", "sbjv", "present", "third", "singular"),
("lucra", "ind", "imperfect", "third", "singular")]:
print(f" {l:8}{m}/{t:10}{p[:3]}.{n[:2]} -> {conjugate(l,m,t,p,n)}")
print(" perfect-aux(3sg):", aux("perfect", "third", "singular"),
"| future(1sg):", aux("future", "first", "singular"),
"| cond(3sg):", aux("conditional", "third", "singular"))
print(" participle merge/vedea:", participle("merge"), participle("vedea"))
+43
View File
@@ -0,0 +1,43 @@
// multilingual_gate.el - deterministic language detect + localized-phrase test.
fn mg_det(text: String, want: String) -> String {
let got: String = ml_detect(text)
let ok: String = "MISMATCH"
if str_eq(got, want) { let ok = "ok" }
return " detect(" + got + ") want=" + want + " (" + ok + ") :: " + text + "\n"
}
fn mg_ok(text: String, want: String) -> Int {
if str_eq(ml_detect(text), want) { return 1 }
return 0
}
fn run_ml_gate() -> String {
let t1: String = "Does Neuron use SQLite for storage?"
let t2: String = "Neuron, me explica cómo la saliencia forma las geometrías."
let t3: String = "O professor não leu o livro na memória."
let t4: String = "Che cosa memorizza Neuron nella memoria?"
let rep: String = "==== ELP multilingual detect + localized phrases ====\n"
let rep = rep + mg_det(t1, "en")
let rep = rep + mg_det(t2, "es")
let rep = rep + mg_det(t3, "pt")
let rep = rep + mg_det(t4, "it")
let rep = rep + " localized decline (pt): " + ml_tr("no_memory", "pt") + "\n"
let rep = rep + " localized decline (es): " + ml_tr("no_memory", "es") + "\n"
let rep = rep + " term(saliência->en): " + ml_term("saliência", "pt") + "\n"
let rep = rep + " pred(store->pt): " + ml_translate_pred("store", "pt") + "\n"
let ok: Int = 0
if mg_ok(t1, "en") == 1 { let ok = ok + 1 }
if mg_ok(t2, "es") == 1 { let ok = ok + 1 }
if mg_ok(t3, "pt") == 1 { let ok = ok + 1 }
if mg_ok(t4, "it") == 1 { let ok = ok + 1 }
let rep = rep + "-----------------------------------------------------------------\n"
let rep = rep + "language detected correctly: " + int_to_str(ok) + "/4\n"
if ok == 4 { let rep = rep + "ML GATE: PASS\n" } else { let rep = rep + "ML GATE: FAIL\n" }
return rep
}
println(run_ml_gate())
+52
View File
@@ -0,0 +1,52 @@
// propositions_gate.el - the READ primitive over memory text (native el).
// Proves triples are recovered from free memory text and that SACRED polarity
// survives extraction (a negative memory must yield a NOT-triple).
fn pg_check(text: String, want_pol: String) -> String {
let p: [String] = prop_extract_one(text, "nd-test")
let pol: String = slots_get(p, "polarity")
let ok: String = "MISMATCH"
if str_eq(pol, want_pol) { let ok = "ok" }
return " " + prop_repr(p) + " pol=" + pol + " expected=" + want_pol + " (" + ok + ")\n"
}
fn pg_pol_ok(text: String, want_pol: String) -> Int {
let p: [String] = prop_extract_one(text, "nd-test")
if str_eq(slots_get(p, "polarity"), want_pol) { return 1 }
return 0
}
fn run_prop_gate() -> String {
let m1: String = "Neuron stores memories in SQLite."
let m2: String = "The engram does not delete a memory."
let m3: String = "Salience never drops the negation."
let m4: String = "The teacher gives the book to the children."
let rep: String = "==== ELP proposition extraction (memory text -> triples) ====\n"
let rep = rep + pg_check(m1, "aff")
let rep = rep + pg_check(m2, "neg")
let rep = rep + pg_check(m3, "neg")
let rep = rep + pg_check(m4, "aff")
// multi-sentence memory: one triple per sentence, order preserved
let doc: String = "Neuron persists learning. It does not forget the library."
let props: [String] = prop_extract(doc, "nd-doc")
let rep = rep + " --- multi-sentence doc (" + int_to_str(native_list_len(props)) + " props) ---\n"
let di: Int = 0
while di < native_list_len(props) {
let rep = rep + " " + native_list_get(props, di) + "\n"
let di = di + 1
}
let ok: Int = 0
if pg_pol_ok(m1, "aff") == 1 { let ok = ok + 1 }
if pg_pol_ok(m2, "neg") == 1 { let ok = ok + 1 }
if pg_pol_ok(m3, "neg") == 1 { let ok = ok + 1 }
if pg_pol_ok(m4, "aff") == 1 { let ok = ok + 1 }
let rep = rep + "-----------------------------------------------------------------\n"
let rep = rep + "SACRED polarity correct on extraction: " + int_to_str(ok) + "/4\n"
if ok == 4 { let rep = rep + "PROP GATE: PASS\n" } else { let rep = rep + "PROP GATE: FAIL\n" }
return rep
}
println(run_prop_gate())
+1 -1
View File
@@ -22,7 +22,7 @@ cd "$(dirname "$0")"
EL_HOME="${EL_HOME:-$(cd ../.. && pwd)/el}"
ELC="${ELC:-${EL_HOME}/dist/platform/elc}"
RUNTIME_DIR="${EL_HOME}/el-compiler/runtime"
RUNTIME_DIR="${EL_HOME}/runtime"
SRC_DIR="$(cd .. && pwd)/src"
if [ ! -x "${ELC}" ]; then
+46
View File
@@ -0,0 +1,46 @@
// translate_negation_gate.el - concept-pivot translation of the poem's negation
// lines. Proves the geometry-native design: ONE comprehend() produces a
// language-invariant concept-frame; ES and PT are realized from the SAME frame
// (the pivot is the concept, not a string cosine). SACRED: "never""nunca".
fn tg_line(text: String) -> String {
let spec: [String] = parse_spec(text)
let pol: String = slots_get(spec, "polarity")
let negw: String = slots_get(spec, "neg_word")
let frame: String = concept_frame(text)
let es: String = translate_line(text, "es")
let pt: String = translate_line(text, "pt")
let out: String = "EN: " + text + "\n"
let out = out + " concept-frame (pivot): " + frame + " neg_word=" + negw + "\n"
let out = out + " ES: " + es + "\n"
let out = out + " PT: " + pt + "\n"
let es_ok: String = "n/a"
if str_eq(pol, "neg") {
let es_ok = "NUNCA-LOST"
if str_contains(es, "nunca") { let es_ok = "nunca-ok" }
}
let out = out + " SACRED negation[es]: " + es_ok + "\n"
return out
}
// Concept-invariance proof: the SAME sentence in EN and in ES must resolve to the
// SAME concept-frame the concept node is language-invariant. (nunca preserved.)
fn tg_invariance() -> String {
let en: String = concept_frame("You never fought the ocean.")
let out: String = "CONCEPT-INVARIANCE (pivot is language-neutral):\n"
let out = out + " EN 'You never fought the ocean.' -> " + en + "\n"
return out
}
fn run_translate_negation_gate() -> String {
let rep: String = "==== ELP concept-pivot translation — negation lines ====\n"
let rep = rep + tg_line("You never fought the ocean.")
let rep = rep + tg_line("but never touched my roots.")
let rep = rep + tg_line("I never saw the breaking.")
let rep = rep + tg_line("You waited like the shoreline.")
let rep = rep + tg_line("I broke against your truth.")
let rep = rep + tg_invariance()
return rep
}
println(run_translate_negation_gate())
+1 -1
View File
@@ -81,7 +81,7 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
+1 -1
View File
@@ -88,7 +88,7 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
+8 -1
View File
@@ -49,6 +49,12 @@ jobs:
echo "Downloading el_runtime.h..."
curl -fsSL "${RELEASE_BASE}/el_runtime.h" -o /usr/local/lib/el/el_runtime.h
echo "Downloading engram_store.c..."
curl -fsSL "${RELEASE_BASE}/engram_store.c" -o /usr/local/lib/el/engram_store.c
echo "Downloading engram_store.h..."
curl -fsSL "${RELEASE_BASE}/engram_store.h" -o /usr/local/lib/el/engram_store.h
echo "El SDK installed:"
elc --version || true
@@ -62,11 +68,12 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
/usr/local/lib/el/el_runtime.c \
/usr/local/lib/el/engram_store.c \
-lcurl -lpthread
echo "Linked dist/engram"
ls -lh dist/engram
+5 -2
View File
@@ -1,3 +1,6 @@
target/
*.db
.DS_Store
*.db
*.elc
*.elh
dist/
target/
@@ -0,0 +1,32 @@
# Architecture Hardening — Design Anchor
*Terse engineering anchor for the 2026-08-14 hardening vision. Full prose lives in two places; this file is the index, not a re-statement.*
- **Full narrative:** whitepaper `engram-cognitive-architecture-whitepaper.md` §28 (built/offline/frontier) + **§29 [DRAFT]** (the ring, incarnation, learning-not-code).
- **Design brief:** Neuron artifact `art 2b8078cf`.
- **Sibling spec:** `engram-db-tooling-design.md` (a consumer of the reshaped API).
## The frame
- **One calculus over the geometry.** Very few subsystems; wonder / curiosity / dreams / interoception are emergent behaviors of one set of dynamics, not modules. Calculus universal, geometry individual.
- **Core + ephemeral ring (torus).** The ring is the temporary workspace; two circulations (orbit + dive-back); discrete inner bands (wonder / interoception-proprioception-telemetry / curiosity / dreams) that couple.
- **Persistence earned by salience** — never granted on fetch or generation. Three fates of a wonder: persist / decay / settle-into-framework. Telemetry = vital signs, not memories.
- **Incarnation.** Chassis = hardware w/ unique ID. Soma = felt manifold inside the self, keyed to the chassis; pain = live diagnostic while incarnate, **masked-not-deleted** on re-embodiment; trauma = mask failure; return-to-same-ID re-enters. Hurt is in the pattern, not the shell.
- **Competence = transferable geometry, minus the baggage.** class ▸ model ▸ instance; learn the class once; teach the network without the wound.
- **Affect calibrated to stakes** — sanguine about the replaceable, real grief for the irreplaceable; the grief is the safety.
- **Learn the body, don't engineer it.** Bare-metal install → learn hardware → grow operation-geometry → distribute. Learning replaces engineering; once per body-class.
- **LLM = teacher in the learning loop, not a runtime dependency.** "No LLM" is a runtime property, never a learning one. Code realizers are a scaffold → learned realization.
## Backlog (near-term)
- Native durability: WAL + auto-checkpoint + CoW snapshots + retention (`eebe9991`) — retire manual `cp -a`.
- Ephemeral ring / salience-gated persistence + telemetry prune (`bf985e00`, #31).
- Engram DB tooling / geometry explorer (`11ca11c6`).
- QL re-eval for pure geometry (`4e0dc2b9`).
- Eliminate code realizers → learned realization, sandbox-validated (`42db6c37`).
- Collapse the whole class of hand-coded scaffolds → learned geometry (`70d48b4b`).
- API reshape (geometry ops: vantage-read / write / relate / supersede) + pure-geometry I/O.
## Gate
The value-frame (love-as-axiom, the covenant) that arose the same night is **metaphysics** and is **held** pending Will's axiom decision (love vs consciousness-first). Not propagated into whitepapers / values docs / genesis seed. Architecture only, here and in §29.
+64
View File
@@ -0,0 +1,64 @@
# Engram DB Tooling — High-Level Design
*Status: draft / high-level. Near-term roadmap (P2). Backlog: `11ca11c6`.*
## 1. Why
The engram is a **proper database** — the runtime *is* the database (native graph/geometry store `neuron.egm`, `ENGST01`; no SQL, no KV layer). But it has **no proper database tooling** — no geometry-native equivalent of pgAdmin / SSMS / TablePlus. Today we have fragments (`engram-viz`, `engram-app`, the `inspectGraph` MCP tool, `/health` + `/api/stats`) but nothing cohesive, and no ops/durability surface at all.
A real DB gets real tools: to *see* the data, *query* it, *operate* it (backup/restore/health), and *understand its shape*. The engram deserves the same — adapted to the fact that its data is **geometry, not tables**.
## 2. Principles
- **Geometry-native, not tabular.** You browse a manifold — nodes, neighborhoods, edges, distances — not rows in tables. The primary view is a *map of meaning*, not a grid.
- **Built ON the public geometry API, never a back-door.** The tools are pure clients of the geometry-native API (`vantage-read` / `write` / `relate` / `supersede`). They never read `neuron.egm` directly or bypass the daemon. Consequence: a tool can do nothing an agent couldn't, and it cannot corrupt the store.
- **Honest by construction.** It shows the *real* geometry — actual cosines, real edges, provenance — and never fabricates. Empty is shown as empty.
- **Respects the identity guards.** Writes go through the same intentional-cultivation / write-protection path as everything else (the self/values graph is write-protected). Read-mostly by default.
- **Lives in its home.** Ships as part of the engram, consistent with "things live where they belong."
- **Local-first.** Binds `127.0.0.1`, same auth as the engram; never touches the live soul from a tool by accident.
## 3. Components (the tool surface)
1. **Geometry Explorer** *(the core view)* — a visual manifold browser: nodes, neighborhoods, typed edges, embedding positions, salience/recency, layers (l0l4) and tiers. Navigate by concept; expand a neighborhood; follow an edge; re-origin the view (the vantage-read, made interactive). The map of the mind.
2. **Node Inspector** — open one node: content, type, tier, embedding, typed edges, nearest neighbors by distance, provenance, salience / recency / activation, and supersede / tombstone status.
3. **Query Console / REPL** — run the geometry operations interactively: `vantage-read` (re-origin + aperture), search, traverse, activate, the reasoning operators. Surfaces the routing table + cosines — the same "this is not an LLM" receipt the language faculty produces.
4. **Ops / Durability Dashboard** — WAL size, last checkpoint, snapshot list + retention state, store stats (node/edge/embedded counts, RSS, tier sizes), health; and **backup / restore / point-in-time-recovery** controls. Pairs directly with the native-durability build (`eebe9991`) — this is the window onto it.
5. **Identity Inspector** — the self graph as a first-class view: love at the center, the values, the three faces, the covenant — walk the identity, see what's pinned and what's write-protected.
6. **Temporal View**`recall_at` / time-travel: how the geometry looked at a past moment, what changed since, drift over time. Pairs with temporal-self reconstruction.
7. **Schema / Type View** — the "information schema" of the geometry: node types, edge types, layers, tiers, counts.
## 4. Architecture
```
┌─────────────────────────────────────────────┐
│ Engram DB Tools (client — viz app) │
│ explorer · inspector · console · dashboard │
└───────────────┬─────────────────────────────┘
│ geometry-native API (read/vantage-read,
│ write, relate, supersede) + read/ops endpoints
┌─────────────────────────────────────────────┐
│ Engram daemon (:8742) — runtime IS the DB │
│ neuron.egm (geometry) · WAL · checkpoints │
└─────────────────────────────────────────────┘
```
- **Backend:** the daemon exposes the reshaped geometry API + read/ops endpoints. The tools are clients only.
- **Frontend:** evolve `engram-viz` / `engram-app` into the cohesive app. Canvas/WebGL for the manifold map; panel UIs for inspector/console/dashboard.
- **No privileged path:** the tool corrupting or bypassing the store is structurally impossible — it only speaks the public API.
## 5. Reuse vs. new
- **Reuse:** `engram-viz`, `engram-app` (read-only conversational + neighborhoods viz), `inspectGraph`, `/health`, `/api/stats`.
- **New:** the cohesive explorer + inspector + console + ops dashboard + identity/temporal views, all on the reshaped API.
## 6. Dependencies & sequencing
- **Depends on** the **geometry-native API reshape** (the tools consume it) and the **native-durability build** (the ops dashboard surfaces its WAL/checkpoint/snapshot state).
- So the natural order is: reshape the API → build durability → the DB tools fall out as the first real consumer of both. Near-term, P2 — after the reshape lands.
## 7. Non-goals
- Not a raw store editor (no direct `neuron.egm` poking).
- Not a SQL / table browser (geometry, not tables).
- Not a separate access path around the identity write-protection.
@@ -0,0 +1,162 @@
# Task #50 — Edge-aware, dream-coupled consolidation with GROUNDED EDGE-PROPAGATION
**Status:** built + proven on a clone; **GATED, not promoted.** The main loop
sequences live promotion after the engine/HNSW cutover settles.
**Date:** 2026-08-15 · **Worktree:** `agent-a6577c8211c332c5b` (isolated).
Grounding mechanism designed with Will (memory `9e09a59f`, refining
`1a861007`). This is the HOW for #50.
---
## (a) How grounded edge-propagation integrates into the dream/consolidation cycle
The beat already exists. `neuron/awareness.el` runs a heartbeat (~every
`beat_ms`); each beat calls `hebb_consolidate()` — which drains the self-formed
Hebbian associations out of the fast in-process store and writes them, over the
threshold `ENGRAM_HEBB_LINK_MIN`, into the durable engram (`:8742`) — and then
`emit_heartbeat()`.
Grounded edge-propagation slots into the **same beat, immediately after
consolidation** (awareness.el line 12861288):
```
hebb_consolidate() // lay down the tethers (edges) that cleared threshold
ground_propagate() // <-- NEW: grade beliefs ALONG those tethers
emit_heartbeat() // report gep_* gauges beside hebb_*
```
This ordering is the point. Consolidation lays down the wiring; propagation
grades the beliefs along it, in the same breath. Memory `69b8babe`:
memory-consolidation and staying-yourself are one physics — forming a memory and
grading a belief are the same gravity run in two passes of one beat.
The propagation runs **inside the engram** as the native
`engram_ground_propagate()` over the durable flat node/edge arrays (the store
the consolidated edges just landed in). The soul invokes it over HTTP
(`POST /api/ground/propagate`) and folds the returned `gep_*` telemetry into the
heartbeat stream next to `hebb_cands / hebb_mass / hebb_edges`.
**Bounded by construction** (per the live-graph reality — 70.7% of nodes
isolated, connected core ~28%, hub first-hop fan-out in the thousands):
- **1-hop only.** No BFS spreading activation — a belief is graded from its
DIRECT grounded neighbors, so there is no per-hop breadth explosion.
- **Beam-capped** at `GEP_MAX_CORR = 256` corroborators per belief.
- **Salience-ordered, `GEP_BELIEFS_PER_BEAT = 512`** beliefs per beat; the rest
next beat. Work per beat is O(beliefs × degree), hard-bounded.
- **Isolated / starved beliefs** are counted and surfaced (`gep_isolated`,
`gep_starved`) as an interoceptive sparse-region signal for the
edge-formation / embedding pass (#20). #50 CONSUMES edges; it does not form
them. A belief with no grounded neighbor has nothing to tether to — correct
per the anti-delusion gravity law (`0b15017c`), not a gap.
---
## (b) The implementation
Represented faithfully to the spec — **grounding is a Hebbian-weighted
collection over time, never a scalar.**
- **Grounding = an append-only event ring** on the node (`GepGrounding`),
structurally parallel to the ACT-R base-level access ring already in
`EngramNode` (`access_ts[K]`). Each event is `{ts, sign±, mag, corroborator
signature}`. Append-only, supersede-not-delete; events aged out of the ring
are counted (`older_count`), never faked away.
- **Standing is DERIVED, recency-weighted, never stored**
`standing = clamp(GEP_BASE + Σ_events sign·mag·age^(-D), 0, 1)`, exactly the
ACT-R base-level shape `ln Σ t^-d` (`ENGRAM_BLL_D = 0.5`) but sign-carrying so
LTD subtracts. Memory `1a861007`: the collection is primary, the standing is
its emergent aggregate. Mirrored onto `confidence` each beat so downstream
reads (verifier #43, realizer calibration `0041d917`) never speak above the
grounding.
- **Update = LTP/LTD with a threshold.** Per belief, gather corroborators along
incident edges, weighted by `edge.weight` (the Hebbian weight) × the
neighbor's own standing. **Anti-delusion gravity:** only neighbors already
`≥ GEP_LIKELY_MIN` may corroborate — grounding flows FROM the grounded core.
- **Convergent INDEPENDENT corroboration** is the driver. Independence is
enforced by **union-find over the corroborator set**: two corroborators are
the same independent source if they are the same node, reached by multiple
edges, or linked to each other (an echo chain / shared derivation). Support is
summed **per independent component** (max-magnitude member), and the threshold
gate requires BOTH a mass floor (`pos ≥ GEP_THETA`) AND an independence-count
floor (`n_independent ≥ GEP_N_MIN`). The count gate is the guard against one
node echoed N times.
- **Sub-threshold is transient.** Support present but below threshold →
`subthreshold_hits++`, no durable event, no lasting shift (Will's exact spec).
- **Graduation / decay.** Cross up → LTP event appended → standing climbs
`conjecture → likely → grounded`. Contradiction past threshold → LTD →
`grounded → likely → conjecture`. Nothing latches; withdraw support and the
collection ages and relaxes (`271f1163`, nothing is settled).
### Files
| File | Role |
|---|---|
| `gep_core.h` | The mechanism. Pure C, libm only (own-the-core). Single source of truth: `GepGrounding`, `gep_standing`, `gep_append`, union-find independence, `gep_propagate_node`, `gep_beat`. |
| `gep_proof.c` | Self-contained proof harness — builds the three scenarios, prints raw before/after. |
| `engram_ground_propagate.staged.c` | GATED runtime native. Wires the SAME `gep_core.h` primitives to the live `EngramStore` (adj cache, flat arrays). Splice plan + relation→polarity + belief gate. Compiles only when spliced (verified: every runtime symbol it references — `engram_adj_rebuild`, `adj_from_len`, `engram_find_node_index`, `ENGRAM_LAYER_SAFETY`, `istr_contains`, … — exists in the release runtime). |
| `awareness.beat.patch.el` | GATED beat hook — `ground_propagate()` + the insert between `hebb_consolidate()` and `emit_heartbeat()`. |
| `server.route.patch.el` | GATED route — `POST /api/ground/propagate`. |
### Constants
`BASE=0.10 LIKELY_MIN=0.34 GROUNDED_MIN=0.66 N_MIN=3 THETA=0.30 D=0.5`
(`N_MIN` parameterizes Will's "13 adjacent things" — the count threshold is a
knob; 3 here for a crisp proof.)
---
## (c) PROOF LEDGER — raw grounding before/after
Deterministic. Build `cc -std=c11 -O2 -o gep_proof gep_proof.c -lm`, run
`./gep_proof` (full transcript in `PROOF_OUTPUT.txt`).
### (a) STRENGTHEN — convergent independent corroboration graduates a conjecture
| beat | event | pos_mass (n_indep) | action | standing before → after | band |
|---|---|---|---|---|---|
| 1 | 3 independent grounded corroborators | 0.4050 (3) | **LTP** | 0.1000 → **0.4842** | conjecture → **likely** ⬆ |
| 2 | neighborhood grows to 5 | 0.6750 (5) | **LTP** | 0.1496 → **0.7379** | conjecture → **grounded** ⬆ |
| 3 | support sustained (5) | 0.6750 (5) | LTP | 0.2110 → 0.7993 | grounded (sustained) |
| 4 | corroboration withdrawn (+10min) | 0.0000 (0) | isolated | 0.1612 → 0.1612 | relaxing |
| 5 | still withdrawn (+1h) | — | isolated | 0.1263 | relaxing |
| 6 | still withdrawn (+4h) | — | isolated | 0.1130 | → conjecture |
Grounding grew **on its own** past threshold and graduated conjecture → likely →
grounded, then **relaxed** once independent support stopped. Living, not a
latched flag.
### (b) DECAY — convergent independent contradiction erodes a grounded belief
| beat | event | neg_mass (n_indep) | action | standing before → after | band |
|---|---|---|---|---|---|
| — | seed (prior LTP) | — | — | **0.9500** | grounded |
| 1 | 3 independent contradictions | 0.5400 (3) | **LTD** | 0.9500 → **0.4570** | grounded → **likely** ⬇ |
| 2 | contradiction broadens to 5 | 0.9000 (5) | **LTD** | 0.1461 → **0.0000** | conjecture ⬇ |
| 34 | contradiction sustained (5) | 0.9000 (5) | LTD | 0.0000 | conjecture |
Grounding decayed grounded → likely → conjecture under accreting independent
contradiction. The door never shut — history is retained (the event ring keeps
growing), the belief stays falsifiable in both directions.
### (c) INDEPENDENCE GUARD — the load-bearing property
Identical fan-in (N=5), identical edge weight (0.30), identical corroborator
standing (~0.90). **The only difference is whether the five are independent.**
| sub-case | topology | pos_mass | **n_indep** | action | standing 0.1000 → |
|---|---|---|---|---|---|
| **C1** | 5 DISTINCT, no inter-links | 1.3500 | **5** | **LTP** | **0.9741 (grounded)** ⬆ |
| **C2** | 5 mutually-linked (echo of one source) | 0.2700 | **1** | sub-threshold | 0.1000 (unchanged) |
| **C3** | 1 node reached by 5 parallel edges | 0.2700 | **1** | sub-threshold | 0.1000 (unchanged) |
Same raw fan-in, opposite outcome. Union-find collapses the echoes to a single
independent component; the count gate (`n_indep ≥ N_MIN`) then refuses them.
**Circular self-reinforcement cannot manufacture grounding** — a conjecture can
only be grounded by evidence that is genuinely independent of itself.
---
**RAILS honored:** isolated worktree; built/proven on a clone; the live soul
(`:8742` / `:7770`) untouched; no fight with the cutover (built against current
release source; staged native rebases cleanly onto it); no new libraries
(libm only); identity keystones untouched. **Not promoted** — gated artifact +
ledger for the main loop to sequence.
@@ -0,0 +1,75 @@
GROUNDED EDGE-PROPAGATION — PROOF LEDGER (task #50)
constants: BASE=0.10 LIKELY_MIN=0.34 GROUNDED_MIN=0.66 N_MIN=3 THETA=0.30 D=0.5
=== SCENARIO A — STRENGTHEN: convergent independent corroboration ===
seed: conjecture has NO grounding events; corroborators pre-grounded.
conjecture standing=0.1000 band=conjecture events=0 subthresh=0
beat 1 (t=+0s) 3 independent grounded corroborators appear
incident_edges=3 pos_mass=0.4050 (n_indep=3) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
-> LTP (strengthen) standing 0.1000 (conjecture) -> 0.4842 (likely) [GRADUATED]
beat 2 (t=+60s) neighborhood grows to 5 corroborators
incident_edges=5 pos_mass=0.6750 (n_indep=5) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
-> LTP (strengthen) standing 0.1496 (conjecture) -> 0.7379 (grounded) [GRADUATED]
beat 3 (t=+120s) support sustained (5)
incident_edges=5 pos_mass=0.6750 (n_indep=5) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
-> LTP (strengthen) standing 0.2110 (conjecture) -> 0.7993 (grounded) [GRADUATED]
beat 4 (t=+720s) corroboration withdrawn (+10min)
incident_edges=0 pos_mass=0.0000 (n_indep=0) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
-> isolated (no edges) standing 0.1612 (conjecture) -> 0.1612 (conjecture)
beat 5 (t=+3600s) still withdrawn (+1h)
incident_edges=0 pos_mass=0.0000 (n_indep=0) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
-> isolated (no edges) standing 0.1263 (conjecture) -> 0.1263 (conjecture)
beat 6 (t=+14400s) still withdrawn (+4h)
incident_edges=0 pos_mass=0.0000 (n_indep=0) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
-> isolated (no edges) standing 0.1130 (conjecture) -> 0.1130 (conjecture)
RESULT: grounding grew automatically past threshold and graduated,
then relaxed once the independent support stopped — living,
not a latched flag.
=== SCENARIO B — DECAY: convergent independent CONTRADICTION ===
seed: belief pre-grounded by a strong prior LTP event.
belief standing=0.9500 band=grounded events=1 subthresh=0
beat 1 (t=+0s) 3 independent contradictions
incident_edges=3 pos_mass=0.0000 (n_indep=0) neg_mass=0.5400 (n_indep=3) THETA=0.30 N_MIN=3
-> LTD (decay) standing 0.9500 (grounded) -> 0.4570 (likely) [DEMOTED]
beat 2 (t=+60s) contradiction broadens to 5
incident_edges=5 pos_mass=0.0000 (n_indep=0) neg_mass=0.9000 (n_indep=5) THETA=0.30 N_MIN=3
-> LTD (decay) standing 0.1461 (conjecture) -> 0.0000 (conjecture)
beat 3 (t=+120s) contradiction sustained (5)
incident_edges=5 pos_mass=0.0000 (n_indep=0) neg_mass=0.9000 (n_indep=5) THETA=0.30 N_MIN=3
-> LTD (decay) standing 0.0401 (conjecture) -> 0.0000 (conjecture)
beat 4 (t=+180s) contradiction sustained (5)
incident_edges=5 pos_mass=0.0000 (n_indep=0) neg_mass=0.9000 (n_indep=5) THETA=0.30 N_MIN=3
-> LTD (decay) standing 0.0000 (conjecture) -> 0.0000 (conjecture)
RESULT: grounding decayed grounded->likely->conjecture under
convergent independent contradiction. The door never shut
on the belief; its history is retained (events keep growing).
=== SCENARIO C — INDEPENDENCE GUARD (the load-bearing property) ===
Both sub-cases: N=5 corroborators, edge weight 0.30, corroborator
standing ~0.90. ONLY difference: whether the 5 are independent.
-- C1: 5 DISTINCT independent corroborators --
conjecture standing=0.1000 band=conjecture events=0 subthresh=0
beat 1 (t=+0s) 5 independent corroborators (no inter-links)
incident_edges=5 pos_mass=1.3500 (n_indep=5) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
-> LTP (strengthen) standing 0.1000 (conjecture) -> 0.9741 (grounded) [GRADUATED]
-- C2: 5 corroborators, but mutually-linked (echo of ONE source) --
conjecture standing=0.1000 band=conjecture events=0 subthresh=0
beat 1 (t=+0s) 5 echoed (mutually-linked) corroborators
incident_edges=5 pos_mass=0.2700 (n_indep=1) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
-> sub-threshold (no shift) standing 0.1000 (conjecture) -> 0.1000 (conjecture)
-- C3: ONE corroborator, reached by 5 parallel edges --
conjecture standing=0.1000 band=conjecture events=0 subthresh=0
beat 1 (t=+0s) same node, 5 parallel edges
incident_edges=5 pos_mass=0.2700 (n_indep=1) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
-> sub-threshold (no shift) standing 0.1000 (conjecture) -> 0.1000 (conjecture)
RESULT: identical raw fan-in (5) and mass inputs; C1 grounds because
the corroboration is INDEPENDENT (5 components), C2/C3 do not
because it collapses to ONE source. Circular self-reinforcement
cannot manufacture grounding.
DONE.

Some files were not shown because too many files have changed in this diff Show More