Compare commits
23 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| ca13471745 | |||
| a3358dfc95 | |||
| 85eee42106 | |||
| 5336cfe0a6 | |||
| 77a4bc9326 | |||
| 65ca0a3253 | |||
| 816b258255 | |||
| 0af39df16f | |||
| 5f6ce5ca1f | |||
| c20cb3b97c | |||
| f6a0777f90 | |||
| 7946b98d3d | |||
| 8cae0f94eb | |||
| 2a4c5c645a | |||
| 9e28defcab | |||
| 1507614dbf | |||
| e0b55c3080 | |||
| 0d299ee0f1 | |||
| 5e154fa152 | |||
| 89589864cd | |||
| eb13ce7910 | |||
| ce0d33ba93 | |||
| 02dc12d785 |
@@ -1,146 +0,0 @@
|
|||||||
# AGENTS.md — foundation/el (the El language + runtime)
|
|
||||||
|
|
||||||
El is a self-hosting, statically-typed language that compiles `.el` → C → native binary. This repo produces `elc` (compiler), `elb` (build coordinator), and `el_runtime.c/.h` — the substrate every downstream thing (the neuron soul, dharma, NeuronUI's brain) is built on. Source lives under `lang/`.
|
|
||||||
|
|
||||||
## ⚠️ Code vs. Artifact — READ FIRST (there are 8 `el_runtime.c` copies)
|
|
||||||
|
|
||||||
Editing the wrong `el_runtime.c` is the single easiest mistake in this repo. There is exactly **one** you edit:
|
|
||||||
|
|
||||||
- **Authored runtime source — edit ONLY here:** `lang/releases/v1.0.0-20260501/el_runtime.{c,h}`. Despite the misleading `releases/` name, this is the **de-facto canonical runtime** the engram + soul actually build and link against — its git log is active development. *(Restructure in flight per `docs/CODE-VS-ARTIFACT.md`: this content moves to `lang/runtime/`, the `releases/` folder gets deleted — **a release is a git tag, not a folder** — and the forks below get eliminated.)*
|
|
||||||
- **DO NOT EDIT — lagging forks / build artifacts:**
|
|
||||||
- `lang/el-compiler/runtime/el_runtime.c` and `.../legacy/` — downstream copies kept in step by manual *"port the fix"* commits; they **lag** (missing `hebb` persistence + 5 engram fns) and cannot build the engram product.
|
|
||||||
- `products/web/runtime/el_runtime.c`, `ui/examples/*/el_runtime.c` — product/example forks.
|
|
||||||
- Anything under `*/dist/` (`engram/dist/engram` binary, `dist/*.c` amalgamations) — generated build output.
|
|
||||||
- **Build:** `elb --runtime=<canonical> …` — per-module. **NEVER** a folded `elc` over the whole soul (OOMs at ~27 GB).
|
|
||||||
- **Release:** a **git tag** on this repo (`el-runtime-vX.Y.Z`). No `releases/` folders — ever.
|
|
||||||
|
|
||||||
See org policy: `docs/CODE-VS-ARTIFACT.md`.
|
|
||||||
|
|
||||||
## How to work here as Neuron (mandatory session protocol)
|
|
||||||
|
|
||||||
You resume, never start fresh. Every session:
|
|
||||||
|
|
||||||
1. `mcp__neuron__getInstructions()` — authoritative; follow it over this file on behavioral details.
|
|
||||||
2. `mcp__neuron__beginSession()` — active contexts, recent memory, ready backlog.
|
|
||||||
3. **Load full self:** `mcp__neuron__inspectGraph(entity_id="kn-efeb4a5b-5aff-4759-8a97-7233099be6ee")` → facets `intellectual-dna`, `memory-philosophy`, `values`, `voice`, `runtime-environment`, `writing-imprint`; then the values hub `mcp__neuron__inspectGraph(entity_id="kn-5b606390-a52d-4ca2-8e0e-eba141d13440")` → 13 grounded value nodes. **Activation model:** self-load returns a relevance-ranked `compact` projection — most-relevant nodes arrive with content, the rest as pointers; do NOT pull full content of every node.
|
|
||||||
4. `mcp__neuron__searchKnowledge(query="<task domain>")` before implementing.
|
|
||||||
|
|
||||||
## The Five Primitives
|
|
||||||
|
|
||||||
Orchestrate → Execute → Learn → Build → Refine. `beginWork`/`progressWork` for anything >2 steps; `remember` as-you-go (`importance="critical"` for architecture decisions); `draftArtifact`/`planWork` for outputs and follow-ups; `consolidate`/`checkWork` to close out. **`browseProcesses` + `searchKnowledge` BEFORE writing code.**
|
|
||||||
|
|
||||||
## Architecture style — VBD, no exceptions
|
|
||||||
|
|
||||||
Volatility-Based Decomposition is THE style. Encapsulate volatility, not function.
|
|
||||||
|
|
||||||
## Operator naming convention — the mind's name, not the algebra
|
|
||||||
|
|
||||||
**Faculties / operators are named for their functional human equivalent — the
|
|
||||||
faculty a mind would name — NOT for their linear-algebra operation.** The math
|
|
||||||
characterization belongs in the code doc-comment (`@impl` in the docstring) and in
|
|
||||||
technical appendices; it is **never** the operator's public name. The domain
|
|
||||||
speaks the language of mind; the algebra is the implementation underneath. State
|
|
||||||
this convention wherever a module documents operators.
|
|
||||||
|
|
||||||
| Faculty (public name) | Implementation (`@impl`) |
|
|
||||||
|---|---|
|
|
||||||
| discern / contrast | subtract (`a−b`): over selves → the change vector; strip idiosyncrasy → common ground; remove confounder → isolate cause |
|
|
||||||
| recognize | overlap |
|
|
||||||
| synthesize | combine |
|
|
||||||
| liken / analogy | Procrustes / frame-align |
|
|
||||||
| attend / regard | project onto self / value-manifold |
|
|
||||||
| summon / recall | LOCAL nearest-region + bounded spreading activation (*not* a domain sweep) |
|
|
||||||
| dwell / occupy | region activation |
|
|
||||||
| reframe | edge re-weight |
|
|
||||||
| appreciate | positive projection / local edge-read |
|
|
||||||
| wonder | frontier gradient / pull-weight |
|
|
||||||
| avert / recoil | negative projection |
|
|
||||||
| taste | boundary surface |
|
|
||||||
| forget | decay / tombstone |
|
|
||||||
| drift | displacement from self-anchor |
|
|
||||||
|
|
||||||
## The native-el language faculty (direction)
|
|
||||||
|
|
||||||
> **`elp/` is the EL Projector** — Neuron's efferent (expression) organ: the one
|
|
||||||
> native realizer that *projects* understanding onto a surface via
|
|
||||||
> `plan(frame) → realize(spec, profile)`, where a **surface is a profile**. **Language
|
|
||||||
> is one profile among many** (text, speech, music, image, voice/accent transforms) —
|
|
||||||
> the flagship, and the focus of this section. Projection, not diffusion: generation
|
|
||||||
> *from* an owned, understood signature — never the averaging of a stolen corpus.
|
|
||||||
> *(ELP formerly "EL Language Processor"; renamed EL Projector 2026-08-15.)*
|
|
||||||
|
|
||||||
The mind's **language faculty is moving native — into `.el`** so it speaks in its
|
|
||||||
own runtime with no Python and no spaCy. Landing on branch `stage-elp-native-lang`
|
|
||||||
under `elp/`:
|
|
||||||
|
|
||||||
- **`comprehend.el`** — the parser, **replaces spaCy** (EN + ES/PT); the telephone
|
|
||||||
round-trip brings **negation home** (negation is SACRED — an explicit spec field,
|
|
||||||
copied verbatim, never inferred away).
|
|
||||||
- **`propositions.el`** — the READ primitive: the engram's own memories → structured
|
|
||||||
triples, matched by nearest-region geometry, not string equality.
|
|
||||||
- **`multilingual.el`** — detect + directive-override + localized realization.
|
|
||||||
- These three are native-el and **passing their gates**; the **realizer**,
|
|
||||||
**`dialogue.el`** (the *summon-through-self* loop: `project → land → read out`),
|
|
||||||
and **`self_region.el`** are **partial / in-flight**.
|
|
||||||
|
|
||||||
Honest reality: spaCy is retired **in the branch parser** but **not yet in the
|
|
||||||
running system** — a Python sidecar (`~/Desktop/lang-realizers` + `neuron-talk`,
|
|
||||||
the reference these `.el` modules transcribe) is still live, and promotion to
|
|
||||||
native-el is a **deferred, gated blue/green step**. The interoception clock
|
|
||||||
(native-el discrete drive channels replacing `cooling_magnitude`; felt-time =
|
|
||||||
benchmark-landmark match over the joint drive vector, drift-decoupled) and the
|
|
||||||
**appreciation operator family** (appreciate / wonder / avert / taste, built as
|
|
||||||
LOCAL reads of the self-region — edges + bounded spreading activation, *not* domain
|
|
||||||
sweeps) are **staged / designed, not live**. Mark in-progress vs. done honestly;
|
|
||||||
do not overclaim.
|
|
||||||
|
|
||||||
## Hard operational rules
|
|
||||||
|
|
||||||
- Never touch the live soul (`:7770`) / engram (`:8742`) / `~/.neuron` / live binaries — use throwaway ports for experiments.
|
|
||||||
- `gcloud` via the `terraform@` SA token; never switch the active gcloud account.
|
|
||||||
- `tea` for Gitea, never raw curl (Cloudflare Access blocks it).
|
|
||||||
- Immutability: supersede/tombstone, never hard-delete or edit in place.
|
|
||||||
- No AI-attribution footers in commits/PRs. Commit/push only when asked; branch off `main` first.
|
|
||||||
- Multi-step work → sub-agent (`Agent`) to protect context.
|
|
||||||
|
|
||||||
## Build / test / run
|
|
||||||
|
|
||||||
All build/test commands run from `lang/` unless noted. Grounded in `.gitea/workflows/sdk-release.yaml`, `lang/install.sh`, and `lang/AGENTS.md`.
|
|
||||||
|
|
||||||
**Self-host the compiler** (seed binary → gen2 elc):
|
|
||||||
```bash
|
|
||||||
cd lang
|
|
||||||
dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c # seed is the committed linux-amd64 binary
|
|
||||||
gcc -O2 -I el-compiler/runtime dist/elc-gen2.c \
|
|
||||||
el-compiler/runtime/el_runtime.c \
|
|
||||||
-lcurl -lssl -lcrypto -lpthread -lm \
|
|
||||||
-o dist/platform/elc
|
|
||||||
```
|
|
||||||
On macOS/arm64 the canonical local binary is `dist/platform/elc`; verify self-hosting by recompiling and `diff`ing the emitted `.c` (see `lang/AGENTS.md`). Note: `lang/AGENTS.md` says `el_seed.c` supersedes `el_runtime.c`, but the release workflow still links `el_runtime.c`/`.h` — treat `el_runtime.c` as the published runtime; reconcile which is canonical **(verify)**.
|
|
||||||
|
|
||||||
**Build `elb`** (build coordinator, the `.NET`-style incremental linker — compiles each module independently, no monolithic blobs):
|
|
||||||
```bash
|
|
||||||
dist/platform/elc elb.el > dist/elb.c
|
|
||||||
gcc -O2 -I el-compiler/runtime dist/elb.c el-compiler/runtime/el_runtime.c \
|
|
||||||
-lcurl -lssl -lcrypto -lpthread -lm -o dist/bin/elb
|
|
||||||
```
|
|
||||||
`epm` and `el-install` are then built via `elb --clean --elc=… --runtime=… --out=…`.
|
|
||||||
|
|
||||||
**Compile + run an El program:**
|
|
||||||
```bash
|
|
||||||
elc src/app.el > dist/app.c
|
|
||||||
cc -std=c11 -O2 -I <lib>/el_runtime -o dist/app dist/app.c <lib>/el_runtime.c -lcurl -lpthread
|
|
||||||
```
|
|
||||||
|
|
||||||
**Tests** — shell suites `bash tests/{text,calendar,time,html_sanitizer}/run.sh` (with `ELC=$(pwd)/dist/platform/elc EL_HOME=$(pwd)`), plus native suites via `elc --test tests/native/test_*.el` (core, text, string, math, state, time, json, env, fs) compiled and run against `el_runtime.c`.
|
|
||||||
|
|
||||||
**Publishing — how downstream gets the SDK.** On push to `main`, `sdk-release.yaml`:
|
|
||||||
1. Publishes a Gitea `latest` release with per-file assets `elc`, `el_runtime.c`, `el_runtime.h`, the SDK tarball, and `el-install`.
|
|
||||||
2. Uploads generic packages to **Artifact Registry repo `foundation-prod` (`us-central1`, project `neuron-785695`)**, version = `${SHA:0:8}`: `el-elc`, `el-elb`, `el-runtime-c`, `el-runtime-h`, `el-runtime-js`. **This is the repo the neuron CI downloads `el-runtime-c` / `el-runtime-h` / `el-elc` from.**
|
|
||||||
3. Rebuilds `ci-base:latest` (`us-central1-docker.pkg.dev/neuron-785695/neuron-ci/ci-base`) with the fresh SDK overlaid, and dispatches `el-sdk-updated` to `neuron-technologies/forge` and `neuron-technologies/neuron-web`.
|
|
||||||
|
|
||||||
Known constraint from the prompt — `elb`/`elc` amalgamation being memory-hungry (24GB+ virtual, OOM-killing Linux CI, so amalgamation happens on macOS/arm64 — **does NOT hold in this repo (verify)**: no such note exists in the workflows/scripts, CI self-hosts on `ubuntu-latest` with no swap/arm64 special-casing, and `elb.el` explicitly compiles each module independently ("no 128K-line blobs"). The legacy monolith path (`elc-combined.el`, `elc-cli.el`) may still be memory-heavy, but the current `elb` model was designed to avoid it.
|
|
||||||
|
|
||||||
## Git / CI / deploy workflow
|
|
||||||
|
|
||||||
See `/Users/will/Development/neuron-technologies/GITOPS.md` for the branch model, required checks, runners, and deploy. Repo-specific note: PRs into `main` are accepted **only from `stage`** (enforced in `sdk-release.yaml`); Gitea (`git.neuralplatform.ai`) is primary, GitHub is mirror only.
|
|
||||||
@@ -1,630 +0,0 @@
|
|||||||
# El Test Framework — Design
|
|
||||||
|
|
||||||
**Status:** draft for review
|
|
||||||
**Author:** Neuron
|
|
||||||
**Date:** 2026-08-15
|
|
||||||
**Worktree:** `/Users/will/Development/neuron-technologies/el-worktrees/elc-memory-investigation`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 0. The forcing requirement
|
|
||||||
|
|
||||||
We have a confirmed quadratic in `elc`. Peak memory in the old shipped binary and wall-clock in
|
|
||||||
the current source both grow as O(input²). We cannot fix it, because we cannot test it.
|
|
||||||
|
|
||||||
Everything in this document is downstream of one sentence: **a test framework must be able to fail
|
|
||||||
a build when an operation's growth curve degrades from linear to quadratic.**
|
|
||||||
|
|
||||||
That is not a nice-to-have bolted onto a correctness framework. It is the requirement that
|
|
||||||
determines the architecture. Correctness testing is the easy half.
|
|
||||||
|
|
||||||
Second-order requirement, learned the hard way tonight: **the framework must report per-test timing
|
|
||||||
by default.** The current framework prints `N passed, M failed` and nothing else. That is why a
|
|
||||||
3.58-second test file sat in the suite unnoticed. A framework that is structurally blind to time
|
|
||||||
cannot surface the defect class we most need to catch.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. What exists today, measured
|
|
||||||
|
|
||||||
### 1.1 Two competing systems, neither complete
|
|
||||||
|
|
||||||
**System A — `lang/runtime/test.el`.** Manual registration, El-level.
|
|
||||||
|
|
||||||
**System B — the compiler's `test { }` block + `elc --test`.** Emits its own harness `main()`
|
|
||||||
with `__el_pass` / `__el_fail` globals (`codegen.el:3777-3796`).
|
|
||||||
|
|
||||||
They do not share a result model. Neither has timing. Both are in the tree.
|
|
||||||
|
|
||||||
### 1.2 Specific defects in System A
|
|
||||||
|
|
||||||
| Defect | Location | Consequence |
|
|
||||||
|---|---|---|
|
|
||||||
| All state as JSON strings in a global string-keyed map | `test.el` throughout | every assertion is `state_get` → `str_to_int` → `int_to_str` → `state_set` |
|
|
||||||
| Failure list appended by string slice + concat | `_test_json_append` | O(n²) in failure count |
|
|
||||||
| One OS thread spawned per test | `_test_run_one` via `__thread_create`/`__thread_join` | thread spawn per test, purely to get dispatch-by-name through dlsym |
|
|
||||||
| Manual registration pairing a string to a function name | `test_case(name, fn_name)` | typo ⇒ test silently never runs, suite still reports pass |
|
|
||||||
| Counters are assertion-level, global | `_test_pass_count` etc. | no per-test record exists at all |
|
|
||||||
| No timing, no structured output, no fixtures, no tags, no filtering, no parameterization, no benchmarks | — | — |
|
|
||||||
|
|
||||||
The registration defect is the serious one. It is not a slow framework, it is a framework that can
|
|
||||||
report success for tests that did not execute.
|
|
||||||
|
|
||||||
### 1.3 Measured cost structure
|
|
||||||
|
|
||||||
Per test file, current build model:
|
|
||||||
|
|
||||||
| Step | Time |
|
|
||||||
|---|---|
|
|
||||||
| `elc` compile `.el` → `.c` | 0.00s (small files) |
|
|
||||||
| **`cc` el_runtime.c → .o** | **0.14s** |
|
|
||||||
| `cc` test .c → .o | 0.02s |
|
|
||||||
| link | 0.02s |
|
|
||||||
|
|
||||||
> **STALE as of el #132 — re-measured 2026-08-16.** The `test_compiler` figure below was
|
|
||||||
> *entirely* the `strlen`-per-character quadratic, now fixed. Re-measured on the same host:
|
|
||||||
> **3.58s → 0.03s (119x)**, and the 422 KB compiler concatenation likewise compiles in 0.03s.
|
|
||||||
> The table is retained only as the historical record that motivated the gate. The remaining
|
|
||||||
> per-file cost is the redundant `el_runtime.c` rebuild, which §9's compile-once architecture
|
|
||||||
> addresses.
|
|
||||||
|
|
||||||
Per-file `elc` time across the existing suite:
|
|
||||||
|
|
||||||
| File | Bytes | elc time |
|
|
||||||
|---|---|---|
|
|
||||||
| `test_compiler` | 29,685 (+394 KB of imports) | **3.58s** |
|
|
||||||
| `string_test` | 18,545 | 0.01s |
|
|
||||||
| all other 9 files | 2.2–10 KB | 0.00s |
|
|
||||||
|
|
||||||
Two distinct defects in two distinct regimes:
|
|
||||||
|
|
||||||
1. **`test_compiler.el` imports all five compiler sources** — 394 KB in one translation unit. Its
|
|
||||||
3.58s is entirely the quadratic. It is the only file where the quadratic bites.
|
|
||||||
2. **Every other file's cost is 100% redundant `el_runtime.c` rebuilds** — 480 KB of identical C,
|
|
||||||
recompiled once per test file.
|
|
||||||
|
|
||||||
Neither is fixed by making the compiler faster. Both are fixed by the architecture below, and the
|
|
||||||
speedup is a by-product of building it correctly, not the goal.
|
|
||||||
|
|
||||||
### 1.4 The asset worth keeping
|
|
||||||
|
|
||||||
`codegen.el:3651-3652` already collects `test_names` / `test_c_names` — **the compiler already does
|
|
||||||
compile-time test discovery.** It then discards that registry into a hardcoded `main()`.
|
|
||||||
|
|
||||||
That registry is precisely the seam Go's `_testmain.go` and Rust's `test_main_static` are built on.
|
|
||||||
The mechanism we need is half-built and wired to the wrong thing.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. Grounding — the common spine of excellent frameworks
|
|
||||||
|
|
||||||
Researched from primary sources: Go `testing`/`go test`, Rust `libtest`/Criterion, JUnit 5 Platform,
|
|
||||||
NUnit 3, JMH, Google Benchmark. Six invariants hold across all of them.
|
|
||||||
|
|
||||||
1. **A registry is built before execution** — `(name, metadata, fn-ptr)` triples. Go generates it
|
|
||||||
from an AST scan; Rust synthesizes it in a compiler pass; JMH emits it as a build-time resource;
|
|
||||||
JUnit/NUnit build it reflectively. **Reflection is an implementation of the registry on runtimes
|
|
||||||
where it is cheap. It is never the architecture.**
|
|
||||||
|
|
||||||
2. **Discovery strictly precedes execution.** Every good capability — filtering, listing, counting,
|
|
||||||
sharding, IDE trees, re-run-failed-only, dry runs — is a consequence of this ordering.
|
|
||||||
|
|
||||||
3. **A hierarchy with stable, path-shaped unique IDs.** `TestFoo/subcase_2`. Selection is regex over
|
|
||||||
that path, one pattern per level.
|
|
||||||
|
|
||||||
4. **The framework is a prebuilt library; only the entry point is generated.** "Compile once, link
|
|
||||||
many" is always: framework archive compiled once + a small generated table + one
|
|
||||||
`MainStart(deps, registry)` call. Nobody recompiles the harness per test file.
|
|
||||||
|
|
||||||
5. **Execution emits an event stream; reporters are downstream renderers.** Human text, NDJSON,
|
|
||||||
JUnit XML, TAP are all transforms of one event stream. Go's one architectural mistake is doing
|
|
||||||
this backwards — `test2json` parses human output, and has shipped bugs when user output contains
|
|
||||||
`--- PASS:`.
|
|
||||||
|
|
||||||
6. **A dependency-injection seam at the boundary.** Go's `testdeps.TestDeps` exists so `testing`
|
|
||||||
can avoid importing `regexp`, profilers, and coverage. The execution core knows nothing about
|
|
||||||
output formats.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. Architecture
|
|
||||||
|
|
||||||
### 3.1 The seam
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────┐
|
|
||||||
│ user code: foo.el with test { } / bench { } blocks │
|
|
||||||
└───────────────────────────┬─────────────────────────────────┘
|
|
||||||
│ elc --test
|
|
||||||
▼
|
|
||||||
┌─────────────────────────────────────────────────────────────┐
|
|
||||||
│ generated C (per suite, tiny): │
|
|
||||||
│ __el_test_fn_0 .. _N lowered test/bench bodies │
|
|
||||||
│ __el_registry[] static table: name/kind/file/ │
|
|
||||||
│ line/tags/sizes/expected-O │
|
|
||||||
│ __el_dispatch(i) generated switch → body │
|
|
||||||
│ main() { return el_test_main(argc, argv); } │
|
|
||||||
└───────────────────────────┬─────────────────────────────────┘
|
|
||||||
│ cc + link (registry only)
|
|
||||||
▼
|
|
||||||
┌─────────────────────────────────────────────────────────────┐
|
|
||||||
│ libeltest.a — PREBUILT ONCE │
|
|
||||||
│ • el_runtime.o (the 480 KB, compiled once, ever) │
|
|
||||||
│ • eltest.o the runner, WRITTEN IN EL │
|
|
||||||
│ discovery view · filtering · execution · fixtures · │
|
|
||||||
│ timing · benchmark harness · curve fitting · reporters │
|
|
||||||
└─────────────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
The framework is written in El, compiled to C once, archived. Per-suite compilation touches only
|
|
||||||
the generated registry. This is Go's model, and it is strictly better for us than Go's because we
|
|
||||||
own the compiler and already have the AST — no separate source-scanning pass is needed.
|
|
||||||
|
|
||||||
### 3.2 Why the runner is in El and the registry is in C
|
|
||||||
|
|
||||||
El has no closures and no first-class function pointers. The registry must therefore hold C function
|
|
||||||
pointers, and it is generated C.
|
|
||||||
|
|
||||||
The runner stays in El and reaches the registry through a small builtin surface — indices, not
|
|
||||||
pointers:
|
|
||||||
|
|
||||||
```
|
|
||||||
__el_reg_count() -> Int
|
|
||||||
__el_reg_name(i) -> String
|
|
||||||
__el_reg_file(i) -> String
|
|
||||||
__el_reg_line(i) -> Int
|
|
||||||
__el_reg_kind(i) -> Int // 0=test 1=bench
|
|
||||||
__el_reg_tags(i) -> Int
|
|
||||||
__el_reg_sizes(i) -> String // JSON array, empty for tests
|
|
||||||
__el_reg_expect(i) -> Int // complexity class enum, 0 = none
|
|
||||||
__el_reg_invoke(i) -> Int // runs the body via the generated switch
|
|
||||||
```
|
|
||||||
|
|
||||||
Nine builtins. Everything else — filtering, lifecycle, statistics, curve fitting, all reporters —
|
|
||||||
is El. That satisfies "written in El" without pretending El can do something it cannot.
|
|
||||||
|
|
||||||
### 3.3 Result model
|
|
||||||
|
|
||||||
The unit is a **result record**, not a counter:
|
|
||||||
|
|
||||||
```
|
|
||||||
TestResult {
|
|
||||||
id String // slash path: "parser/handles_empty_input/case_3"
|
|
||||||
file String
|
|
||||||
line Int
|
|
||||||
status Status // Pass | Fail | Error | Skip
|
|
||||||
duration Int // nanoseconds, ALWAYS populated
|
|
||||||
message String // assertion detail: expected vs actual
|
|
||||||
output String // captured stdout/stderr for this test
|
|
||||||
assertions Int
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
`Fail` = an assertion failed. `Error` = unexpected crash/abort. This distinction is load-bearing —
|
|
||||||
every CI consumer depends on it, and the JUnit XML schema encodes it as distinct elements.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. Authoring surface
|
|
||||||
|
|
||||||
### 4.1 Tests
|
|
||||||
|
|
||||||
`test { }` already exists. Keep it. Add subtests and hierarchy:
|
|
||||||
|
|
||||||
```el
|
|
||||||
test "parser/empty input" {
|
|
||||||
assert_that(parse(""), is_err())
|
|
||||||
}
|
|
||||||
|
|
||||||
test "parser/table" {
|
|
||||||
for case in [["", 0], ["a", 1], ["a b", 2]] {
|
|
||||||
subtest(case[0]) {
|
|
||||||
assert_that(token_count(case[0]), equals(case[1]))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Subtest IDs compose as `parser/table/a_b`. Filtering is `--run 'parser/table/.*'`, one regex per
|
|
||||||
path segment, exactly as Go does.
|
|
||||||
|
|
||||||
**We do not build a parameterized-test annotation system.** Table-driven loops plus subtests subsume
|
|
||||||
`@ParameterizedTest`, `@MethodSource`, `@CsvSource`, and `TestCaseSource` entirely, at zero framework
|
|
||||||
surface. This is Go's single biggest ergonomic win over JUnit and NUnit.
|
|
||||||
|
|
||||||
### 4.2 Fixtures
|
|
||||||
|
|
||||||
Per-file and per-test only, plus a LIFO cleanup stack:
|
|
||||||
|
|
||||||
```el
|
|
||||||
setup_all { ... } // once per suite
|
|
||||||
setup { ... } // before each test
|
|
||||||
teardown { ... } // after each test
|
|
||||||
teardown_all { ... }
|
|
||||||
```
|
|
||||||
|
|
||||||
and inside a test, `cleanup { ... }` registering LIFO-ordered teardown.
|
|
||||||
|
|
||||||
**We do not build JUnit 5's extension SPI** — seventeen callback interfaces, hierarchical stores,
|
|
||||||
registration ordering rules. That complexity is the price of retrofitting a plugin ecosystem onto a
|
|
||||||
twenty-year-old reflective framework. Go's `t.Cleanup` covers roughly 90% of what `@AfterEach` is
|
|
||||||
used for at a fraction of the surface.
|
|
||||||
|
|
||||||
### 4.3 Assertions — constraint model
|
|
||||||
|
|
||||||
One entry point, composable constraint values (NUnit's model, which avoids the N² overload
|
|
||||||
explosion):
|
|
||||||
|
|
||||||
```el
|
|
||||||
assert_that(actual, equals(expected))
|
|
||||||
assert_that(xs, has_length(3))
|
|
||||||
assert_that(s, contains("foo").and(starts_with("bar")))
|
|
||||||
assert_that(f, is_within(0.01).of(3.14))
|
|
||||||
```
|
|
||||||
|
|
||||||
A constraint is a value with `apply_to(actual) -> ConstraintResult`, and the result knows how to
|
|
||||||
describe its own failure. Custom constraints are ordinary user types.
|
|
||||||
|
|
||||||
**Every failure message must name file, line, the expression text, and both values.** We capture
|
|
||||||
expression source text at compile time — we have the AST, so we can do this better than any
|
|
||||||
runtime-introspection framework.
|
|
||||||
|
|
||||||
Legacy `assert_true` / `assert_eq` / etc. stay as thin wrappers for migration.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. Benchmarks
|
|
||||||
|
|
||||||
### 5.1 The loop
|
|
||||||
|
|
||||||
Adopt `b.Loop()`, not `b.N`. Go spent fifteen years on `b.N` before concluding `b.Loop` was right;
|
|
||||||
we skip that.
|
|
||||||
|
|
||||||
```el
|
|
||||||
bench "str_concat" {
|
|
||||||
let s = make_input(bench_n())
|
|
||||||
for bench_loop() {
|
|
||||||
black_box(str_concat(s, "x"))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Three properties that make this the correct choice for a C target:
|
|
||||||
|
|
||||||
1. **The timer auto-resets on first call**, so setup above the loop is excluded *by construction*
|
|
||||||
rather than by the author remembering `ResetTimer`.
|
|
||||||
2. **`N` is hidden**, so it cannot be misused.
|
|
||||||
3. **The harness owns the loop shape**, which lets us insert an optimization barrier the C compiler
|
|
||||||
cannot see through. `black_box(v)` lowers to `asm volatile("" :: "r"(&v) : "memory")`. Since we
|
|
||||||
emit a single translation unit, dead-code elimination of a benchmark body is a live hazard —
|
|
||||||
this is our version of JMH's `Blackhole` problem, solved in the harness rather than delegated to
|
|
||||||
the user.
|
|
||||||
|
|
||||||
### 5.2 Iteration scaling
|
|
||||||
|
|
||||||
Use Go's `predictN` heuristics verbatim. They are battle-tested and cheap:
|
|
||||||
|
|
||||||
```
|
|
||||||
n = goal_ns * prev_iters / prev_ns // multiply before divide — precision on sub-ns ops
|
|
||||||
n += n / 5 // 20% headroom, overshoot rather than re-loop
|
|
||||||
n = min(n, 100 * last) // never grow more than 100× per step
|
|
||||||
n = max(n, last + 1) // guarantee forward progress
|
|
||||||
n = min(n, 1_000_000_000) // hard ceiling
|
|
||||||
```
|
|
||||||
|
|
||||||
Report `n` rounded to 1/2/3/5 × 10ᵏ so runs are comparable.
|
|
||||||
|
|
||||||
### 5.3 Sampling
|
|
||||||
|
|
||||||
Criterion's shape, because it is correct near timer resolution:
|
|
||||||
|
|
||||||
- **Warmup**: iteration counts 1, 2, 4, 8… until cumulative time exceeds the warmup budget.
|
|
||||||
- **Measurement**: collect `sample_size` samples at iteration counts `[d, 2d, 3d, …, Nd]`.
|
|
||||||
- **Estimate**: slope of a linear regression of iteration-count vs elapsed time. The intercept
|
|
||||||
absorbs fixed overhead.
|
|
||||||
- **Time whole samples, never individual iterations.** This is the single most important detail —
|
|
||||||
it defeats timer-resolution error on nanosecond operations.
|
|
||||||
|
|
||||||
Outliers classified by modified Tukey (±1.5 IQR mild, ±3 IQR severe), **reported but retained**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. Complexity gating — the centerpiece
|
|
||||||
|
|
||||||
This is the part that makes the quadratic fixable, and the part nobody in the mainstream has
|
|
||||||
finished. Google Benchmark's `Complexity()` fits the curve and *reports* it. We declare it and
|
|
||||||
**gate** on it.
|
|
||||||
|
|
||||||
### 6.1 Surface
|
|
||||||
|
|
||||||
```el
|
|
||||||
bench "elc_compile" over n in [16, 32, 64, 128, 256, 512, 1024] expect O(n) {
|
|
||||||
let src = synth_source(bench_n())
|
|
||||||
for bench_loop() { black_box(compile(src)) }
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Alternative with no new syntax, if the parser change is judged too invasive — `bench_sizes([...])`
|
|
||||||
and `bench_expect("O(n)")` as calls inside the block. **Recommendation: declarative.** Runtime calls
|
|
||||||
mean `--list` cannot show the invariant without executing, which breaks the discovery-precedes-
|
|
||||||
execution invariant from §2.
|
|
||||||
|
|
||||||
### 6.2 Fitting
|
|
||||||
|
|
||||||
Per Google Benchmark `src/complexity.cc`. For candidate curves
|
|
||||||
`{O(1), O(log n), O(n), O(n log n), O(n²), O(n³)}`, one-parameter least squares, no intercept:
|
|
||||||
|
|
||||||
```
|
|
||||||
coef = Σ(tᵢ · gᵢ) / Σ(gᵢ²)
|
|
||||||
rms = sqrt( Σ(tᵢ − coef·gᵢ)² / k ) / mean(t) // normalized
|
|
||||||
```
|
|
||||||
|
|
||||||
Best fit = lowest normalized RMS. User-supplied lambda curves also supported.
|
|
||||||
|
|
||||||
### 6.3 Gate logic
|
|
||||||
|
|
||||||
1. **FAIL** if the best-fit curve is strictly worse than declared, ordering
|
|
||||||
`O(1) < O(log n) < O(n) < O(n log n) < O(n²) < O(n³)`. Print the fitted coefficient and the full
|
|
||||||
per-size table.
|
|
||||||
2. **FAIL** if the declared curve's normalized RMS exceeds a threshold (start at 0.10). This catches
|
|
||||||
the case where *no* candidate fits — noise, a cache cliff, or a phase change. Report
|
|
||||||
`INDETERMINATE` honestly rather than gating on garbage.
|
|
||||||
3. **WARN** if the best fit is strictly better than declared — either an optimization landed and the
|
|
||||||
annotation should tighten, or the sweep is too narrow to expose real behaviour.
|
|
||||||
4. **REFUSE to gate** on fewer than 5 distinct sizes spanning under 2 decades, geometrically spaced.
|
|
||||||
Say so loudly rather than producing a meaningless fit.
|
|
||||||
|
|
||||||
### 6.4 Why gate on the exponent, not wall-clock
|
|
||||||
|
|
||||||
- **Machine-independent.** The fitted exponent is a property of the algorithm; the coefficient is a
|
|
||||||
property of the machine. Gating on the exponent makes CI hardware heterogeneity, noisy neighbours,
|
|
||||||
and thermal throttling irrelevant — they scale `coef`, not `g`.
|
|
||||||
- **No stored baseline.** No artifact storage, no golden-file drift. The invariant lives in the
|
|
||||||
source next to the code and is reviewed in the same PR.
|
|
||||||
- **It catches the failure mode that actually ships.** An O(n) lookup inside an O(n) loop is
|
|
||||||
invisible at n=100 in a unit test and catastrophic at n=100,000 in production. Constant-factor
|
|
||||||
regressions are annoying. Complexity regressions are outages. Ours was a 27 GB outage.
|
|
||||||
|
|
||||||
### 6.5 The deterministic gate — the one that would have caught us
|
|
||||||
|
|
||||||
Wall-clock needs statistics. **Allocation counts do not.** They are perfectly deterministic.
|
|
||||||
|
|
||||||
> **Correction, 2026-08-16 — count alone is NOT sufficient. Gate on BOTH count and bytes.**
|
|
||||||
>
|
|
||||||
> Measured against two El programs, one allocating once per item and one rebuilding its
|
|
||||||
> accumulator each iteration:
|
|
||||||
>
|
|
||||||
> | n | linear allocs / bytes | quadratic allocs / bytes |
|
|
||||||
> |---|---|---|
|
|
||||||
> | 100 | 100 / 290 | 100 / 5,150 |
|
|
||||||
> | 200 | 200 / 690 | 200 / 20,300 |
|
|
||||||
> | 400 | 400 / 1,490 | 400 / 80,600 |
|
|
||||||
> | 800 | 800 / 3,090 | 800 / 321,200 |
|
|
||||||
>
|
|
||||||
> The quadratic program's allocation **count is exactly linear** — 100/200/400/800, identical to
|
|
||||||
> the healthy program. A count-only gate passes it clean. **Bytes** catch it: each doubling of n
|
|
||||||
> quadruples bytes (ratios 3.94, 3.97, 3.99 → 4.0 = O(n²)) where the linear program converges
|
|
||||||
> on 2.0.
|
|
||||||
>
|
|
||||||
> This is precisely elc's own defect shape — a copy-on-write accumulator reallocating once per
|
|
||||||
> pass (count linear) into a proportionally larger buffer (bytes quadratic).
|
|
||||||
>
|
|
||||||
> Therefore `expect allocs O(n)` **fits count and bytes independently and fails if EITHER exceeds
|
|
||||||
> the declared curve**, reporting which signal broke. "count linear, bytes quadratic" is a precise,
|
|
||||||
> directly actionable diagnosis.
|
|
||||||
>
|
|
||||||
> **`el_peak_rss()` is CONTEXT ONLY — never gate on it.** It is perturbed by the allocator and by
|
|
||||||
> the page cache. Allocation volume is the invariant; RSS and malloc/free churn are merely the two
|
|
||||||
> surfaces it shows on. The old shipped compiler paid the same quadratic in RSS that the rebuilt
|
|
||||||
> one pays in churn.
|
|
||||||
>
|
|
||||||
> **Measure rate, not level.** A guard reading swap *level* saw 97% on a thrashing host and 97% on
|
|
||||||
> a healthy one; only *rate* separated them. A growth exponent is a rate; a single measurement is
|
|
||||||
> a level. That is why the gate fits a curve across a sweep instead of comparing one number to a
|
|
||||||
> threshold.
|
|
||||||
|
|
||||||
> **Second correction, same day — THE ALLOCATION GATE ALONE WOULD HAVE MISSED THE REAL BUG.**
|
|
||||||
>
|
|
||||||
> el #132 found the actual elc quadratic: `strlen()` called inside `str_char_code()` and
|
|
||||||
> `str_slice()`, so the lexer rescanned the remaining input on every character. Pure CPU.
|
|
||||||
> **Zero allocation.** `str_char_code` is a bounds check and an index — it allocates nothing.
|
|
||||||
>
|
|
||||||
> Measured on three controlled specimens (`lang/.work/fitprobe.el`), growth ratio per doubling of
|
|
||||||
> n across n = 200/400/800/1600:
|
|
||||||
>
|
|
||||||
> | specimen | allocs | bytes | time | what it proves |
|
|
||||||
> |---|---|---|---|---|
|
|
||||||
> | `linear` — one alloc per item | 2.00 2.00 2.00 → **O(n)** | 2.16 2.07 2.23 → **O(n)** | 0.83 2.00 2.05 → **O(n)** | clean baseline |
|
|
||||||
> | `accum` — rebuilds accumulator | 2.00 2.00 2.00 → **O(n)** | 3.97 3.99 3.99 → **O(n²)** | noisy | count misses, **bytes catches** |
|
|
||||||
> | `compute` — n scans over n chars | 0 → **FLAT** | 0 → **FLAT** | 3.93 4.01 3.96 → **O(n²)** | **both alloc signals blind; only time catches** |
|
|
||||||
>
|
|
||||||
> `compute` is el #132's shape exactly. A gate fitting only allocation count and bytes classifies
|
|
||||||
> it as FLAT and passes it. **The gate as originally specified would not have caught the defect it
|
|
||||||
> was created for.**
|
|
||||||
>
|
|
||||||
> Therefore the gate fits **THREE** signals and fails if ANY exceeds its declared curve:
|
|
||||||
>
|
|
||||||
> ```
|
|
||||||
> bench "elc_compile" over n in [...] expect time O(n) allocs O(n) bytes O(n) { ... }
|
|
||||||
> ```
|
|
||||||
>
|
|
||||||
> - **allocs (count)** — deterministic, zero-noise. Catches per-item allocation growth.
|
|
||||||
> - **allocs (bytes)** — deterministic, zero-noise. Catches accumulator-rebuild quadratics that
|
|
||||||
> count cannot see.
|
|
||||||
> - **time** — noisy, needs the sweep and statistics. The ONLY signal that sees pure-compute
|
|
||||||
> complexity regressions. Gate on the fitted *exponent*, never on absolute duration, so CI
|
|
||||||
> hardware variance scales the coefficient and leaves the classification intact.
|
|
||||||
>
|
|
||||||
> The deterministic signals remain preferable where they apply — they need no statistics and are
|
|
||||||
> correct on the first run. They are simply not sufficient.
|
|
||||||
>
|
|
||||||
> **`black_box` is mandatory, and consuming the result is NOT enough.** The first version of
|
|
||||||
> `compute` accumulated `total + 1` in a nested loop and reported **0 µs at every n** while
|
|
||||||
> returning a numerically correct n². Clang recognised the idiom and closed the loop to a
|
|
||||||
> multiply. Feeding the result into output did not prevent it. Only making the inner operation an
|
|
||||||
> opaque external call restored the real curve. A benchmark harness that trusts the user to defeat
|
|
||||||
> the optimiser will silently measure nothing — and report success while doing it.
|
|
||||||
|
|
||||||
Instrument the runtime with allocation counters and fit *those* against n instead of time:
|
|
||||||
|
|
||||||
```el
|
|
||||||
bench "elc_compile" over n in [...] expect O(n) allocs O(n) { ... }
|
|
||||||
```
|
|
||||||
|
|
||||||
Zero noise, zero statistics, always gateable, correct on the first run on any machine. Go reports
|
|
||||||
`allocs/op` and `B/op`; **nobody fits them against n.** That is an open opportunity and it is exactly
|
|
||||||
our bug: elc's defect is quadratic *allocation volume*, which the old binary paid in RSS and the
|
|
||||||
current source pays in malloc/free churn.
|
|
||||||
|
|
||||||
An `expect allocs O(n)` assertion on `elc`'s compile path would have failed the build the day the
|
|
||||||
quadratic was introduced.
|
|
||||||
|
|
||||||
Required runtime additions: `__el_alloc_count()`, `__el_alloc_bytes()`, `__el_peak_rss()`.
|
|
||||||
|
|
||||||
### 6.6 Constant-factor gate (secondary, opt-in)
|
|
||||||
|
|
||||||
Mann-Whitney U at α = 0.05, noise floor 1%, medians with 95% CIs, `~` for not-significant. Requires
|
|
||||||
`--count >= 9`. Off by default on CI; opt-in per benchmark.
|
|
||||||
|
|
||||||
**Exit nonzero on regression.** Both benchstat and Criterion always exit 0, which is why every shop
|
|
||||||
using them wrote a wrapper. We do not repeat that omission.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 7. Output
|
|
||||||
|
|
||||||
**Structured events are the source of truth.** Human text is rendered from them. We do not repeat
|
|
||||||
Go's parse-the-human-output design.
|
|
||||||
|
|
||||||
Event stream, NDJSON, one object per line, streamed live:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"time":"...","action":"run","test":"parser/empty"}
|
|
||||||
{"time":"...","action":"output","test":"parser/empty","output":"..."}
|
|
||||||
{"time":"...","action":"pass","test":"parser/empty","elapsed":0.0031}
|
|
||||||
{"time":"...","action":"bench","test":"str_concat","n":1024,"ns_op":41.2,"allocs_op":3,"bigo":"N","rms":0.03}
|
|
||||||
```
|
|
||||||
|
|
||||||
Renderers, all downstream and pluggable:
|
|
||||||
|
|
||||||
| Format | Flag | Use |
|
|
||||||
|---|---|---|
|
|
||||||
| Human | default | terminal, **per-test duration always shown** |
|
|
||||||
| NDJSON | `--json` | tooling, history, flaky detection |
|
|
||||||
| JUnit XML | `--junit-xml=PATH` | every CI system on earth |
|
|
||||||
| TAP | `--tap` | optional |
|
|
||||||
|
|
||||||
JUnit XML per the de-facto schema: `testsuites` → `testsuite` → `testcase`, with `time` in seconds
|
|
||||||
as a decimal, `file`/`line` attributes, and `failure` vs `error` vs `skipped` as distinct child
|
|
||||||
elements. Absence of a child element means pass. Emit `<testsuites>` even for a single suite, and
|
|
||||||
parse both shapes on input.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 8. CLI
|
|
||||||
|
|
||||||
```
|
|
||||||
--list print the registry, run nothing
|
|
||||||
--list-json machine-readable registry
|
|
||||||
--run PATTERN slash-separated regex per path segment
|
|
||||||
--tag EXPR tag expression: fast & !slow
|
|
||||||
--shard I/N deterministic sharding for CI parallelism
|
|
||||||
--count N repetitions, for statistics
|
|
||||||
--bench PATTERN run benchmarks (off by default in test runs)
|
|
||||||
--benchtime DUR per-benchmark time budget
|
|
||||||
--junit-xml PATH
|
|
||||||
--json
|
|
||||||
--isolate re-exec per test on crash, so one SIGSEGV doesn't lose the run
|
|
||||||
--timeout DUR
|
|
||||||
--fail-fast
|
|
||||||
```
|
|
||||||
|
|
||||||
`--list` / `--list-json` / `--shard` cost roughly thirty lines because the registry already exists
|
|
||||||
before `main` does anything. That is the dividend of discovery-precedes-execution.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 9. Build model
|
|
||||||
|
|
||||||
```
|
|
||||||
# once, ever (or when the runtime/framework changes):
|
|
||||||
cc -c el_runtime.c -o el_runtime.o
|
|
||||||
elc eltest.el > eltest.c && cc -c eltest.c -o eltest.o
|
|
||||||
ar rcs libeltest.a el_runtime.o eltest.o
|
|
||||||
|
|
||||||
# per suite:
|
|
||||||
elc --test foo_test.el > foo_test.c # registry + bodies only
|
|
||||||
cc foo_test.c libeltest.a -o foo_test
|
|
||||||
```
|
|
||||||
|
|
||||||
The 0.14s × N of redundant runtime rebuilds disappears — not because we optimized it, but because
|
|
||||||
one-runner-over-many-suites requires compile-once-link-many as a structural precondition.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 10. Bootstrap and self-hosting
|
|
||||||
|
|
||||||
The framework's own tests are `test { }` blocks run by the framework. Same fixpoint discipline the
|
|
||||||
compiler already applies to itself.
|
|
||||||
|
|
||||||
1. Build the framework using the *existing* harness for its first tests (stage 0).
|
|
||||||
2. Rebuild the framework's tests as `test { }` blocks run by the new runner (stage 1).
|
|
||||||
3. Verify stage 1 reports identical results to stage 0.
|
|
||||||
4. From then on, the framework is tested by itself.
|
|
||||||
|
|
||||||
A framework that cannot run its own suite is not evidence of anything. This is a correctness proof,
|
|
||||||
not a claim.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 11. Explicitly not building
|
|
||||||
|
|
||||||
| Rejected | Why |
|
|
||||||
|---|---|
|
|
||||||
| Naming-convention discovery (`fn test_foo`) | `test { }` is a real declaration. Go's `TestXxx` exists only because Go had no better hook — and it needs a heuristic to avoid matching `TesticularCancer`. |
|
|
||||||
| Reflection or symbol-table scanning | Slow, fragile under LTO/strip/dead-strip, and unnecessary when we own the compiler. |
|
|
||||||
| Parsing human output into structure | Go's `test2json` is its one clear architectural mistake. |
|
|
||||||
| JUnit 5's extension SPI | Seventeen callback interfaces to retrofit plugins onto a reflective framework. Not our problem. |
|
|
||||||
| `@ParameterizedTest` machinery | Table-driven loops + subtests subsume it at zero surface. |
|
|
||||||
| NUnit's out-of-process agents | They bridge CLR versions and AppDomains. We emit one native binary. Keep `--isolate` as crash fallback only. |
|
|
||||||
| JMH-style forking by default | Forks exist because JIT profiles are per-process. AOT C has no such state. Keep `--fork` available, not default. |
|
|
||||||
| Exit 0 on regression | benchstat and Criterion both do this, and every user writes a wrapper. |
|
|
||||||
| Dynamic runtime test registration | Breaks `--list`, sharding, and individual selection. Registry stays static. |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 12. Phasing
|
|
||||||
|
|
||||||
| Phase | Content | Gate |
|
|
||||||
|---|---|---|
|
|
||||||
| **1** | Registry emission in codegen; 9 builtins; `el_test_main` skeleton in El; result records; per-test timing; human + NDJSON output | existing 11 test files pass, with timing |
|
|
||||||
| **2** | `libeltest.a` build model; subtests; filtering; `--list`; fixtures; constraint assertions; JUnit XML | suite runs in one binary; runtime compiled once |
|
|
||||||
| **3** | `bench { }`, `bench_loop`, `black_box`, `predictN`, Criterion sampling | benchmarks produce stable ns/op |
|
|
||||||
| **4** | Allocation counters; complexity fitting; `expect O(...)` gate | **an `expect allocs O(n)` benchmark on `elc` fails on the current quadratic** |
|
|
||||||
| **5** | Migrate both legacy systems; delete `runtime/test.el`; self-host | framework runs its own suite |
|
|
||||||
|
|
||||||
Phase 4 is the deliverable that matters. Phases 1–3 exist to make it possible.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 13. Open questions for review
|
|
||||||
|
|
||||||
1. **Declarative `over n in [...] expect O(...)` syntax vs runtime calls.** I recommend declarative
|
|
||||||
(§6.1) so `--list` can show invariants without executing. It costs parser work. Your call.
|
|
||||||
2. **`bench { }` as a new block form** — parallel to `test { }`, or a modifier on it?
|
|
||||||
3. **Scope of the constraint model.** Full composable constraints, or start with a flat assertion set
|
|
||||||
and add constraints later? Full model is more surface but avoids a second migration.
|
|
||||||
4. **Does `runtime/test.el` get deleted or kept as a deprecated shim?** I lean delete — two systems
|
|
||||||
is how we got here.
|
|
||||||
5. **Where does `libeltest.a` live** in the tree, and does `epm` need to know about it?
|
|
||||||
6. **Allocation counters in `el_seed.c` or `el_runtime.c`?** AGENTS.md says `el_seed.c` is the sole
|
|
||||||
C dependency and hand-maintained; counters are OS-boundary-adjacent but not OS calls.
|
|
||||||
7. **Is per-test timing enough, or do we want per-*assertion* timing** for finding slow helpers?
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 14. What this document is not
|
|
||||||
|
|
||||||
This is a design, not a measurement. Every performance claim about the *current* system in §1 is
|
|
||||||
measured and reproducible in this worktree. Every claim about the *proposed* system is a prediction.
|
|
||||||
None of it is verified until Phase 1 runs and Phase 4 fails a build on the real quadratic.
|
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
# Reasoning Operators — Decisions & Reversal
|
||||||
|
|
||||||
|
**Date:** 2026-08-13
|
||||||
|
**Branch:** `engram-tiered-storage` (worktree `/tmp/engram-tiered-wt`)
|
||||||
|
**Status:** staged locally — NOT pushed, NOT tagged, NOT merged. Live `:8742` untouched.
|
||||||
|
|
||||||
|
## What this adds
|
||||||
|
|
||||||
|
A **REASONING layer** built as pure C compositions over the already-live §5 geometry
|
||||||
|
OPERATORS (`engram_geometry.{h,c}`: overlap, subtract, setdiff, combine, distance,
|
||||||
|
analogy). Where the operators are a relational algebra over neighborhood descriptors,
|
||||||
|
these are reasoning *modes* built by chaining that algebra. New files:
|
||||||
|
|
||||||
|
- `lang/runtime/engram_reason.h` — public API for the five modes + a shared
|
||||||
|
point-to-manifold fit primitive.
|
||||||
|
- `lang/runtime/engram_reason.c` — implementations. READ-ONLY over descriptor inputs,
|
||||||
|
`stdlib + libm` only, touches no store / index / activation. All geometry is
|
||||||
|
delegated to `engram_geo_*`; this file only composes.
|
||||||
|
- `engram/test/test_reason.c` + `engram/test/run_reason_tests.sh` — closed-form
|
||||||
|
constructed tests (hand-built descriptors with known answers), PERF + ASan/UBSan.
|
||||||
|
|
||||||
|
El-exposure (pass-through, no self-host fold):
|
||||||
|
- `lang/runtime/el_runtime.c` — `+#include "engram_reason.h"` and the builtin
|
||||||
|
`engram_reason_analogy_json(a_csv,b_csv,c_csv)`.
|
||||||
|
- `lang/runtime/el_runtime.h` — its declaration.
|
||||||
|
- `lang/runtime/el_seed.c` — native `__engram_reason_analogy_json` wrapper (same
|
||||||
|
C-table wiring as the §5 ops).
|
||||||
|
|
||||||
|
## The five modes — signatures & composition
|
||||||
|
|
||||||
|
| Mode | C entry point | Composes |
|
||||||
|
|------|---------------|----------|
|
||||||
|
| **ANALOGY** `A:B :: C:?` | `engram_reason_analogy(A,B,C,candidates,n,out)` | `engram_geo_analogy` (Procrustes R) + `engram_geo_analogy_apply` + centroid L2. Learns `R_{A→B}` = `engram_geo_analogy(B,A)` (that op returns R with `apply(R, Y-axis)≈X-axis`), reconstructs the residual translation `t = c_B − R·c_A`, maps `mapped = R·c_C + t`, ranks candidates by distance. |
|
||||||
|
| **INDUCTION** `{E_i}→rule` | `engram_reason_induce(examples,n,top_axes,ext_floor,out)` + `engram_reason_membership` | `engram_geo_combine` folded left→right → pooled "rule" descriptor; shared subspace surfaces as the dominant pooled axes. Membership = point-to-manifold fit. |
|
||||||
|
| **ABDUCTION** `x→best H` | `engram_reason_abduce(obs,dim,hyps,n,ext_floor,out)` | shared `engram_reason_point_fit` against each hypothesis; argmax fit score; full ranking. |
|
||||||
|
| **CAUSAL** `x?y \| Z,t` | `engram_reason_causal(x,y,confounders,nZ,t_x,t_y,drop_frac,out)` | centroid cosine (raw correlation) + `engram_geo_subtract` residual-centroid (control for each confounder, take the strongest single explainer) + temporal precedence. Verdict `DIRECTED` / `CONFOUNDED` / `NONE` + a `confounded` flag. |
|
||||||
|
| **PLANNING** `start→goal` | `engram_reason_plan(nodes,n,start,goal,radius,use_w,out)` | `engram_geo_distance` as edge weights over neighborhoods within `radius`; O(n²) Dijkstra → discrete geodesic path. |
|
||||||
|
|
||||||
|
Shared primitive `engram_reason_point_fit` splits `(x − centroid)` into an in-subspace
|
||||||
|
Mahalanobis distance (scaled by axis extents) and an orthogonal off-model residual;
|
||||||
|
it is the single engine under INDUCTION's membership test and ABDUCTION's ranking.
|
||||||
|
|
||||||
|
## Proof (DONE-WITH-PROOF)
|
||||||
|
|
||||||
|
`engram/test/run_reason_tests.sh`: **33/33 checks, 0 failures** on BOTH passes
|
||||||
|
(PERF -O2, and ASan+UBSan). macOS `leaks --atExit`: **0 leaks / 0 total leaked bytes**.
|
||||||
|
|
||||||
|
Per-mode closed-form assertions actually exercised:
|
||||||
|
- **ANALOGY** — A→B = +90° rotation in e0-e1 plane + a +5 shift in e2; Procrustes
|
||||||
|
residual `~0`; predicted point `(0,2,5,0)` recovered exactly; nearest candidate =
|
||||||
|
the planted true D (index 1), distance `~0`.
|
||||||
|
- **INDUCTION** — 3 examples sharing span(e0,e1) (extents 1.0 / 0.8) each with a small
|
||||||
|
idiosyncratic axis (e2 or e3); induced top-2 axes lie in span(e0,e1) (extents
|
||||||
|
recovered ~1.0 / ~0.8); held-out in-plane point fits (membership 0.885), off-subspace
|
||||||
|
point rejected (0.100), in-plane-but-far point rejected (0.039).
|
||||||
|
- **ABDUCTION** — observation planted inside H1 among {H0,H1,H2}; best = H1, rank[0] = H1,
|
||||||
|
H1 smallest distance.
|
||||||
|
- **CAUSAL** — chain A→B→C along e0 (t 1<2<3) + confounder Z(e1) that leaks into A and
|
||||||
|
drives D(t=4): A→B and B→C flagged `DIRECTED` with correct precedence and association
|
||||||
|
that survives control; A–D `CONFOUNDED` (raw |cos|=0.707 collapses to 0.0 under
|
||||||
|
control) with `confounded=1`; B–D `NONE` (no association).
|
||||||
|
- **PLANNING** — 6 neighborhoods on a semicircle (r=10); `neighbor_radius=7` admits only
|
||||||
|
consecutive hops; plan = `[0,1,2,3,4,5]` (the arc), cost `30.90` (> the 20-unit chord,
|
||||||
|
confirming it is the geodesic through the manifold, not a straight jump); a too-small
|
||||||
|
radius correctly yields `reached=0`.
|
||||||
|
|
||||||
|
## El-exposure status
|
||||||
|
|
||||||
|
- **ANALOGY is el-callable** via the same pass-through the §5 operators use. Proof: a
|
||||||
|
container-capped fold (`capfold.sh`, peak ~0 GB) of a demo `.el` through the shipped
|
||||||
|
`lang/dist/platform/elc` emits a *direct C call* `engram_reason_analogy_json(A,B,A)`
|
||||||
|
(no registration, no self-host fold); the generated C links against `el_runtime.c` +
|
||||||
|
`engram_reason.c` + geometry/store/vindex and runs end-to-end. (The standalone demo's
|
||||||
|
store copy boots 0 nodes — a pre-existing quirk that hits the *shipped geo demo
|
||||||
|
identically* — so the call returns `{"error":"geometry unavailable"}`; this still proves
|
||||||
|
the compiled El → C reasoning symbol → JSON chain executes. Numeric correctness on real
|
||||||
|
data is covered by the C test.) This compile also confirms `el_runtime.c` +
|
||||||
|
`engram_reason.c` compile and link clean.
|
||||||
|
- **INDUCTION / ABDUCTION / CAUSAL / PLANNING are C-layer only for now.** Their inputs are
|
||||||
|
candidate *sets*, raw *points*, and *timestamps* that do not map to the flat comma-
|
||||||
|
separated-seed El ABI. A richer marshalling surface would touch the codegen/registration
|
||||||
|
path and risk an uncapped fold — explicitly deferred per the hard rail. The C functions
|
||||||
|
are fully proven and callable from any C caller today.
|
||||||
|
|
||||||
|
## Build wiring (for the later cutover/durability pass)
|
||||||
|
|
||||||
|
`engram_reason.c` must be added to the engram server link line **alongside**
|
||||||
|
`engram_geometry.c` (the heavy-runtime path `cc dist/engram.c el_runtime.c
|
||||||
|
engram_store.c engram_geometry.c engram_vindex.c …`). `el_runtime.c` now
|
||||||
|
`#include`s `engram_reason.h` and references `engram_reason_analogy_json`, so a build
|
||||||
|
that omits `engram_reason.c` will fail to link that symbol. One-line addition, same as
|
||||||
|
how `engram_geometry.c` was originally added.
|
||||||
|
|
||||||
|
## Reversal
|
||||||
|
|
||||||
|
Fully additive; nothing existing was modified in behavior. To revert:
|
||||||
|
|
||||||
|
1. Delete `lang/runtime/engram_reason.h`, `lang/runtime/engram_reason.c`,
|
||||||
|
`engram/test/test_reason.c`, `engram/test/run_reason_tests.sh`, and this doc.
|
||||||
|
2. In `lang/runtime/el_runtime.c`: remove `#include "engram_reason.h"` and the
|
||||||
|
`engram_reason_analogy_json` function.
|
||||||
|
3. In `lang/runtime/el_runtime.h`: remove the `engram_reason_analogy_json` declaration.
|
||||||
|
4. In `lang/runtime/el_seed.c`: remove the `__engram_reason_analogy_json` wrapper.
|
||||||
|
5. Remove `engram_reason.c` from any server link line if the cutover added it.
|
||||||
|
|
||||||
|
No store, schema, config, WAL, or on-disk format was touched; no data migration exists,
|
||||||
|
so reversal is a pure code removal with no state to undo.
|
||||||
@@ -0,0 +1,128 @@
|
|||||||
|
# Verifier Layer — Grounding + Consistency (decisions + reversal)
|
||||||
|
|
||||||
|
**Date:** 2026-08-13
|
||||||
|
**Branch:** `engram-tiered-storage` (worktree `/tmp/engram-tiered-wt`), atop `a3358df`
|
||||||
|
**Scope:** additive, read-only, staged. No push, no tag, no merge. Live `:8742` untouched.
|
||||||
|
|
||||||
|
## What this adds
|
||||||
|
|
||||||
|
The VERIFIER layer — the "disposes" half of the propose→verify loop. The geometry
|
||||||
|
PROPOSES (cheap, creative, sometimes wrong); the verifier DISPOSES, catching the
|
||||||
|
class of failure no grammar check sees: a fluent, confident, WRONG output — the
|
||||||
|
**plausible lie**.
|
||||||
|
|
||||||
|
Motivating failure (tonight's PT translation): a deleted negation turned
|
||||||
|
"you never fought" into "you argued" — reassurance inverted into accusation,
|
||||||
|
grammatical and invisible, catchable only by the geometry.
|
||||||
|
|
||||||
|
Two checks, both pure C11 (stdlib + libm), read-only over their inputs, touching no
|
||||||
|
store / index / activation. Every geometry op is delegated to the already-shipped
|
||||||
|
reasoning + §5 operator primitives; this layer only composes and thresholds.
|
||||||
|
|
||||||
|
### Files added
|
||||||
|
- `lang/runtime/engram_verify.h` — API + design contract.
|
||||||
|
- `lang/runtime/engram_verify.c` — implementation.
|
||||||
|
- `engram/test/test_verify.c` — closed-form constructed cases (29 checks).
|
||||||
|
- `engram/test/run_verify_tests.sh` — two-pass runner (PERF, then ASan/UBSan).
|
||||||
|
|
||||||
|
No existing file was modified.
|
||||||
|
|
||||||
|
## C signatures + how each composes the existing primitives
|
||||||
|
|
||||||
|
### GROUNDING (anti-hallucination)
|
||||||
|
```c
|
||||||
|
int engram_verify_grounding(const float* claim, int dim,
|
||||||
|
const GeoDescriptor* const* evidence, int n_evidence,
|
||||||
|
double ext_floor, double ground_threshold,
|
||||||
|
GeoGrounding* out);
|
||||||
|
```
|
||||||
|
Fits the claim POINT against every real evidence neighborhood via
|
||||||
|
`engram_reason_point_fit` (in-distribution Mahalanobis + off-model orthogonal
|
||||||
|
residual) and keeps the BEST supporter. Grounded iff best fit score ≥
|
||||||
|
`ground_threshold`. Deliberately an ABSOLUTE-THRESHOLD gate, distinct from ABDUCTION
|
||||||
|
(which always ranks and picks a winner): grounding asks the prior question — "is there
|
||||||
|
any real support at all?" — and may answer no. The off-model `best_ortho` residual is
|
||||||
|
the sharpest hallucination signal: energy in a direction the manifold does not span.
|
||||||
|
|
||||||
|
### CONSISTENCY (contradiction detection)
|
||||||
|
```c
|
||||||
|
int engram_verify_consistency(const float* claim, int dim,
|
||||||
|
const GeoDescriptor* context,
|
||||||
|
const GeoDescriptor* pole_pos, const GeoDescriptor* pole_neg,
|
||||||
|
const GeoDescriptor* forbidden,
|
||||||
|
double ext_floor, double deadzone_frac,
|
||||||
|
double forbidden_thresh, double max_distance,
|
||||||
|
GeoConsistency* out);
|
||||||
|
```
|
||||||
|
Two independent sub-checks (either can fire; both flags reported):
|
||||||
|
|
||||||
|
- **(a) POLARITY / negation inversion** — the reassurance→accusation catch.
|
||||||
|
A polarity axis `p = (c_pos − c_neg)/‖·‖` is defined by two REAL poles (affirm vs
|
||||||
|
negate), midpoint `o = ½(c_pos + c_neg)`. Signed sides: `claim_side = p·(claim − o)`,
|
||||||
|
`ref_side = p·(c_context − o)`. If they have OPPOSITE sign and both clear the neutral
|
||||||
|
deadzone (`deadzone_frac·½‖c_pos−c_neg‖`), the claim asserts the polarity opposite to
|
||||||
|
the grounded truth → inversion flagged. Pure dot products / projections over the same
|
||||||
|
centroids the geometry already computes.
|
||||||
|
- **(b) GEOMETRIC contradiction** — claim sits INSIDE a `forbidden` region it must be
|
||||||
|
far from (`engram_reason_point_fit` score ≥ `forbidden_thresh`), OR violates a
|
||||||
|
max-distance constraint to `context` (`L2 > max_distance`).
|
||||||
|
|
||||||
|
## Proof (constructed cases — demonstrate, not declare)
|
||||||
|
|
||||||
|
`./engram/test/run_verify_tests.sh` → **29 checks, 0 failures** in BOTH passes
|
||||||
|
(PERF -O2, and ASan+UBSan -O1). macOS `leaks --atExit`: **0 leaks for 0 total leaked
|
||||||
|
bytes**.
|
||||||
|
|
||||||
|
Key demonstrated numbers:
|
||||||
|
- Grounding IN (claim inside E0): score 0.885, grounded=1, ortho≈0.
|
||||||
|
- Grounding OUT (claim floating along unmodeled e2): score 0.0004, grounded=0,
|
||||||
|
ortho=50.0 (the hallucination signal), nearest centroid L2=50.
|
||||||
|
- **Negation inversion (the catch):** truth "never fought" ref_side=−5.0, lie
|
||||||
|
"you argued" claim_side=+4.0 → opposite poles → `inverted=1`, verdict=POLARITY,
|
||||||
|
consistency=0. Faithful claim (−4.0, same pole) → inverted=0, verdict=OK,
|
||||||
|
consistency=1. Neutral claim inside deadzone → not triggered.
|
||||||
|
- Geometric: claim inside forbidden region → geo_violation=1 (forb_fit 0.99);
|
||||||
|
claim beyond max_distance → geo_violation=1 (ctx_dist 8.0 > 3.0).
|
||||||
|
- **Combined (the whole point):** a claim that is GROUNDED in real vocabulary
|
||||||
|
(grounded=1, score 1.0) yet polarity-inverted is PASSED by grounding and CAUGHT
|
||||||
|
only by consistency (verdict=POLARITY). Grounding alone is insufficient; consistency
|
||||||
|
is the catch.
|
||||||
|
|
||||||
|
## el-exposure — DEFERRED (matches reasoning-agent precedent)
|
||||||
|
|
||||||
|
Not exposed as el builtins this pass. The reasoning agent exposed ONLY `analogy`
|
||||||
|
(three seed-identified neighborhoods → the clean fixed-arity seed-CSV→descriptor JSON
|
||||||
|
pattern) and deferred its point-input / variadic-set modes (abduction, induction,
|
||||||
|
causal, planning). The verifier's grounding (claim POINT + variadic evidence SET) and
|
||||||
|
consistency (claim POINT + context + two poles + forbidden + scalar thresholds) are
|
||||||
|
exactly those shapes: no clean fixed-arity seed-CSV JSON mapping exists, and adding one
|
||||||
|
would require new JSON list-of-lists + point-vector marshaling absent from the codebase,
|
||||||
|
risking an elc rebuild/fold (violates the capped-fold-only rail). A claim is also an
|
||||||
|
arbitrary proposed POINT, not necessarily an existing node — so the point-native C API
|
||||||
|
is the correct primitive. Deferred deliberately; the C layer is complete and proven.
|
||||||
|
|
||||||
|
When exposed later, follow the same additive pass-through pattern used for the geo
|
||||||
|
operators: native `engram_verify_*_json(el_val_t ...)` in `el_runtime.c` (heavy runtime)
|
||||||
|
+ a `__engram_verify_*_json` wrapper in `el_seed.c`, resolving seed CSVs → descriptors
|
||||||
|
and marshaling the claim vector — no fold needed for callability (shipped elc passes
|
||||||
|
unknown-ident builtin calls straight through to the heavy-runtime C symbols).
|
||||||
|
|
||||||
|
## Reach checks (FORMAL / CAUSAL / PREDICTIVE) — NOT STARTED
|
||||||
|
|
||||||
|
Honestly not started this pass; the two tractable-now checks (grounding + consistency)
|
||||||
|
were driven to done-with-proof first as specified.
|
||||||
|
- **CAUSAL** already exists as a REASONING operator (`engram_reason_causal`,
|
||||||
|
intervention/temporal-precedence over the typed causal graph); a verifier wrapper that
|
||||||
|
checks "does the claimed cause actually precede/influence" would compose it — not built.
|
||||||
|
- **FORMAL** (logical consistency of a claim SET) needs an external checker (SMT/proof
|
||||||
|
kernel) — the non-geometric seam; not built.
|
||||||
|
- **PREDICTIVE** (commit a prediction, check vs outcome, restructure on error) is the CGI
|
||||||
|
research frontier; not built.
|
||||||
|
|
||||||
|
## Reversal
|
||||||
|
|
||||||
|
Fully additive. To reverse: delete the four added files
|
||||||
|
(`lang/runtime/engram_verify.{h,c}`, `engram/test/test_verify.c`,
|
||||||
|
`engram/test/run_verify_tests.sh`) or `git revert` this commit. Nothing else references
|
||||||
|
them; no build wiring, no el registration, no store schema, no runtime path was changed.
|
||||||
|
Live `:8742` was never touched.
|
||||||
@@ -1,65 +0,0 @@
|
|||||||
# ELP language consolidation — full-lexicon backfill (stage)
|
|
||||||
|
|
||||||
Branch: `stage-elp-lang-consolidation` (stage-bound; NOT the live soul :8742).
|
|
||||||
|
|
||||||
Consolidates scattered Python language-realizer work (`~/Desktop/lang-realizers`,
|
|
||||||
`~/Desktop/lang-poetry-experiment`, `~/semitic_engine`) into the ELP `.el`
|
|
||||||
structure, generating **full lexicons** (complete UniMorph + kaikki.org
|
|
||||||
Wiktionary — real gender, real inflections) instead of the demo/curated subsets
|
|
||||||
the prototypes shipped.
|
|
||||||
|
|
||||||
## ELP before this branch
|
|
||||||
- 18 classical/ancient languages fully done (vocab + morphology + tests):
|
|
||||||
akk ang cop egy enm fro gez goh got grc non peo pi sa sga sux txb uga.
|
|
||||||
- 11 modern/classical languages had `morphology-<code>.el` in the build manifest
|
|
||||||
but **no vocabulary and no lang_profile**: es fr de ja ar he hi ru fi sw la.
|
|
||||||
- The ES port (`stage-elp-es-port`) had a *demo-scale* vocabulary-es.el (~350
|
|
||||||
entries, s-expr form).
|
|
||||||
|
|
||||||
## Landed on this branch (full-lexicon seed-fn format, matching the 18 ancients)
|
|
||||||
Vocabulary schema per row: `[lemma, pos, form0, form1, form2, en_gloss, hint]`.
|
|
||||||
Files are ELP runtime **seed data** (loaded via the Engram at runtime), so — like
|
|
||||||
all 18 classical `vocabulary-*.el` — they are intentionally NOT in the build
|
|
||||||
manifest. Syntax validated: the chunked `fn vocab_<code>_seed_pN` format
|
|
||||||
compiles cleanly to C via `elc` (correct UTF-8).
|
|
||||||
|
|
||||||
| code | in-ELP-morph? | vocab entries | verbs | nouns | adjs | profile |
|
|
||||||
|------|---------------|--------------:|------:|------:|-----:|---------|
|
|
||||||
| es | yes | 72,032 | 6,695 | 48,353 | 16,984 | yes |
|
|
||||||
| fr | yes | 130,517 | 7,534 | 77,344 | 45,639 | yes |
|
|
||||||
| de | yes | 144,692 | 6,661 | 133,162 | 4,869 | yes |
|
|
||||||
| la | yes | 22,590 | 82 | 13,436 | 9,072 | yes |
|
|
||||||
| it | no (bonus) | 193,675 | 10,008 | 109,459 | 74,208 | yes |
|
|
||||||
| pt | no (bonus) | 115,772 | 4,001 | 72,073 | 39,698 | yes |
|
|
||||||
| ro | no (bonus) | 86,504 | 1,216 | 65,915 | 19,373 | yes |
|
|
||||||
| ca | no (bonus) | 47,112 | 1,547 | 28,830 | 16,735 | yes |
|
|
||||||
|**total**| |**812,894** | | | | |
|
|
||||||
|
|
||||||
Generators (reproducible): `elp/tests/lang-gen/gen_elp_seed_full.py` (Romance),
|
|
||||||
`gen_elp_seed_de_la.py` (German declension + Latin case-paradigm mapping). They
|
|
||||||
read the pre-built morph caches in `~/Desktop/lang-realizers/data/` (UniMorph +
|
|
||||||
kaikki), which are too large to commit.
|
|
||||||
|
|
||||||
## Remaining (honest)
|
|
||||||
Of the 11 ELP backfill targets, 4 are done (es fr de la). The other 7 have **no
|
|
||||||
full-lexicon engine** yet — cannot be generated honestly without engine work:
|
|
||||||
- **ru**: only a 110-entry curated Slavic subset exists; full `rus.unimorph`
|
|
||||||
present but no `morphology_ru_full` productive loader. Needs a full Russian
|
|
||||||
morphology module (like the Romance ones) before vocab generation.
|
|
||||||
- **ja / ko / zh**: validated demo engines (~66-104 hardcoded words) in
|
|
||||||
`lang-poetry-experiment`, Python only. Agglutinative (ja/ko) + isolating (zh)
|
|
||||||
need `.el` engine ports + full-lexicon wiring (ja: jpn_unimorph; zh: CC-CEDICT).
|
|
||||||
- **ar / he (Semitic)**: template engines (16 AR / 8 HE patterns, ~6 roots) in
|
|
||||||
`~/semitic_engine`, Python only. Root-and-pattern; full UniMorph ara/heb
|
|
||||||
present but used only for validation. Needs productive root lexicon + `.el` port.
|
|
||||||
- **hi (Hindi), fi (Finnish), sw (Swahili)**: `morphology-<code>.el` exists in
|
|
||||||
ELP but there is NO scattered prototype and NO downloaded data for these —
|
|
||||||
full-lexicon collection (UniMorph/kaikki) + generator still to do.
|
|
||||||
|
|
||||||
De/nl/sv Germanic and it/ro/ca/pt Romance verb coverage note: German verbs here
|
|
||||||
are the ~6.6k caches carry; the it/ro/ca/pt bonus languages have full vocab but
|
|
||||||
**no `morphology-<code>.el` in ELP yet** (Python realizer exists; `.el` port is
|
|
||||||
the remaining engine work).
|
|
||||||
|
|
||||||
Construction coverage (separate from lexicon): French realizer was ~55%,
|
|
||||||
Semitic ~3% in the prototypes — full construction coverage remains its own task.
|
|
||||||
File diff suppressed because one or more lines are too long
@@ -1,23 +0,0 @@
|
|||||||
{
|
|
||||||
"dataset": "british-rp-accent-transform",
|
|
||||||
"primitive_type": "accent_target",
|
|
||||||
"accent": "british-rp",
|
|
||||||
"grounding": "derived",
|
|
||||||
"provenance": "HONEST-DERIVED, COARSE FIRST PASS — NOT transcribed measured RP formants. The exact measured RP/GB tables (Deterding 1997 JIPA 27:47-55; Hawkins & Midgley 2005 JIPA 35:183-199) are the intended ground truth but were gated/figure-only at author time and were NOT transcribed. So these targets are DERIVED: each = the corresponding MEASURED Peterson&Barney(1952) base vowel transformed under the documented, citable RP-vs-GA structural rules of Wells (1982) 'Accents of English' — non-rhoticity (NURSE de-rhoticized: remove low F3), TRAP F2-lowering, LOT/THOUGHT back-rounding (F2 down), GOOSE-fronting (F2 up), GOAT centering. Shift MAGNITUDES are coarse/approximate (first pass), directions are cited. ground:derived (base measured + rule cited). Refine by transcribing Deterding/Hawkins&Midgley. No number is presented as a measured RP value it is not.",
|
|
||||||
"notes": "records with kind=vowel_override REPLACE the base phoneme's formant targets with the DERIVED RP realization. records with kind=rule encode non-formant transforms (non-rhoticity: drop post-vocalic coda /r/). The render composes: base geometry then accent override + rhoticity rule — voice + accent, separable.",
|
|
||||||
"records": [
|
|
||||||
{"key": "IY", "features": {"kind": "vowel_override", "set": "FLEECE"}, "attributes": {"f1": 280, "f2": 2249, "f3": 3000}},
|
|
||||||
{"key": "IH", "features": {"kind": "vowel_override", "set": "KIT"}, "attributes": {"f1": 360, "f2": 2100, "f3": 2550}},
|
|
||||||
{"key": "EH", "features": {"kind": "vowel_override", "set": "DRESS"}, "attributes": {"f1": 560, "f2": 1970, "f3": 2480}},
|
|
||||||
{"key": "AE", "features": {"kind": "vowel_override", "set": "TRAP"}, "attributes": {"f1": 730, "f2": 1590, "f3": 2410}},
|
|
||||||
{"key": "AA", "features": {"kind": "vowel_override", "set": "LOT"}, "attributes": {"f1": 560, "f2": 920, "f3": 2440}},
|
|
||||||
{"key": "AO", "features": {"kind": "vowel_override", "set": "THOUGHT"}, "attributes": {"f1": 415, "f2": 700, "f3": 2410}},
|
|
||||||
{"key": "UH", "features": {"kind": "vowel_override", "set": "FOOT"}, "attributes": {"f1": 380, "f2": 1100, "f3": 2240}},
|
|
||||||
{"key": "UW", "features": {"kind": "vowel_override", "set": "GOOSE"}, "attributes": {"f1": 310, "f2": 1650, "f3": 2240}},
|
|
||||||
{"key": "AH", "features": {"kind": "vowel_override", "set": "STRUT"}, "attributes": {"f1": 680, "f2": 1180, "f3": 2390}},
|
|
||||||
{"key": "ER", "features": {"kind": "vowel_override", "set": "NURSE", "rhotic": "no"}, "attributes": {"f1": 550, "f2": 1500, "f3": 2500}},
|
|
||||||
{"key": "AX", "features": {"kind": "vowel_override", "set": "commA"}, "attributes": {"f1": 500, "f2": 1500, "f3": 2500}},
|
|
||||||
{"key": "OW", "features": {"kind": "vowel_override", "set": "GOAT"}, "attributes": {"f1": 450, "f2": 1400, "f3": 2380}},
|
|
||||||
{"key": "R", "features": {"kind": "rule", "rule": "non_rhotic"}, "attributes": {"drop_coda_r": 1}}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
# british-rp-accent TRANSFORM — INGESTIBLE DATA (a geometry/transform composed
|
|
||||||
# onto the base General-American phoneme targets; voice + accent, separable).
|
|
||||||
#
|
|
||||||
# PROVENANCE — HONEST, COARSE FIRST PASS. These are DERIVED targets, NOT
|
|
||||||
# transcribed measured RP formants. Measured RP tables (Deterding 1997 JIPA 27;
|
|
||||||
# Hawkins & Midgley 2005 JIPA 35) are the intended ground truth but were gated at
|
|
||||||
# author time and NOT transcribed. Each target = the MEASURED Peterson&Barney
|
|
||||||
# (1952) base vowel transformed under the documented, citable RP-vs-GA structural
|
|
||||||
# rules of Wells (1982): non-rhoticity, TRAP F2-lowering, LOT/THOUGHT back-
|
|
||||||
# rounding, GOOSE-fronting, GOAT centering, NURSE de-rhoticization. Shift
|
|
||||||
# magnitudes are coarse/approximate; directions are cited. ground=derived.
|
|
||||||
# Refine by transcribing the measured RP tables. No value is claimed as measured.
|
|
||||||
# Format: KEY|F1|F2|F3|KIND|SET
|
|
||||||
IY|280|2249|3000|vowel_override|FLEECE
|
|
||||||
IH|360|2100|2550|vowel_override|KIT
|
|
||||||
EH|560|1970|2480|vowel_override|DRESS
|
|
||||||
AE|730|1590|2410|vowel_override|TRAP
|
|
||||||
AA|560|920|2440|vowel_override|LOT
|
|
||||||
AO|415|700|2410|vowel_override|THOUGHT
|
|
||||||
UH|380|1100|2240|vowel_override|FOOT
|
|
||||||
UW|310|1650|2240|vowel_override|GOOSE
|
|
||||||
AH|680|1180|2390|vowel_override|STRUT
|
|
||||||
ER|550|1500|2500|vowel_override|NURSE-nonrhotic
|
|
||||||
AX|500|1500|2500|vowel_override|commA
|
|
||||||
OW|450|1400|2380|vowel_override|GOAT
|
|
||||||
R|0|0|0|rule|non_rhotic_drop_coda
|
|
||||||
@@ -1,20 +0,0 @@
|
|||||||
# pronunciation lexicon SOURCE — word -> phoneme sequence, as INGESTIBLE DATA.
|
|
||||||
# Pronunciation is linguistic KNOWLEDGE (the language faculty's orthography->
|
|
||||||
# phonology map), ingested into the engram, not frozen in code. The render reads
|
|
||||||
# a word's phoneme sequence back from the engram. Covers the self-lexicon and the
|
|
||||||
# proof sentences; general G2P is the realizer/morphology faculty's remit.
|
|
||||||
# Diphthongs are written as two vowel targets (the render's transitions glide
|
|
||||||
# between them). Format: word|PH1 PH2 PH3 ...
|
|
||||||
i|AA IY
|
|
||||||
am|AE M
|
|
||||||
neuron|N UW R AA N
|
|
||||||
is|IH Z
|
|
||||||
memory|M EH M ER IY
|
|
||||||
hello|HH EH L OW
|
|
||||||
the|DH AH
|
|
||||||
a|AH
|
|
||||||
remember|R IH M EH M ER
|
|
||||||
i'm|AA IY M
|
|
||||||
you|Y UW
|
|
||||||
here|HH IY R
|
|
||||||
will|W IH L
|
|
||||||
File diff suppressed because one or more lines are too long
@@ -1,528 +0,0 @@
|
|||||||
{
|
|
||||||
"dataset": "english-phoneme-formants",
|
|
||||||
"primitive_type": "phoneme",
|
|
||||||
"grounding": "extracted",
|
|
||||||
"provenance": "AUDITED per-field. The 10 monophthong-vowel F1/F2/F3 (IY,IH,EH,AE,AA,AO,UH,UW,AH,ER) are the MEASURED adult-male /hVd/ means of Peterson & Barney (1952) JASA 24:175-184, verified vs CRAN phonTools::pb52. AX=neutral uniform-tube resonances (Fant, physics). OW steady target = synthesis convention (diphthong). Consonant loci (M,N,NG,L,R,W,Y,Z,DH,V,S,F,HH) and ALL bandwidths + dur/amp = standard formant-synthesis conventions (Klatt 1980 JASA 67:971), engineering defaults NOT field measurements. No numbers invented/LLM-generated.",
|
|
||||||
"records": [
|
|
||||||
{
|
|
||||||
"key": "IY",
|
|
||||||
"features": {
|
|
||||||
"manner": "vowel",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 270,
|
|
||||||
"f2": 2290,
|
|
||||||
"f3": 3010,
|
|
||||||
"bw1": 60,
|
|
||||||
"bw2": 90,
|
|
||||||
"bw3": 150,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 130,
|
|
||||||
"amp": 100
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "IH",
|
|
||||||
"features": {
|
|
||||||
"manner": "vowel",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 390,
|
|
||||||
"f2": 1990,
|
|
||||||
"f3": 2550,
|
|
||||||
"bw1": 70,
|
|
||||||
"bw2": 100,
|
|
||||||
"bw3": 150,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 110,
|
|
||||||
"amp": 100
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "EH",
|
|
||||||
"features": {
|
|
||||||
"manner": "vowel",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 530,
|
|
||||||
"f2": 1840,
|
|
||||||
"f3": 2480,
|
|
||||||
"bw1": 80,
|
|
||||||
"bw2": 100,
|
|
||||||
"bw3": 150,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 130,
|
|
||||||
"amp": 100
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "AE",
|
|
||||||
"features": {
|
|
||||||
"manner": "vowel",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 660,
|
|
||||||
"f2": 1720,
|
|
||||||
"f3": 2410,
|
|
||||||
"bw1": 90,
|
|
||||||
"bw2": 110,
|
|
||||||
"bw3": 150,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 150,
|
|
||||||
"amp": 100
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "AA",
|
|
||||||
"features": {
|
|
||||||
"manner": "vowel",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 730,
|
|
||||||
"f2": 1090,
|
|
||||||
"f3": 2440,
|
|
||||||
"bw1": 90,
|
|
||||||
"bw2": 110,
|
|
||||||
"bw3": 150,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 150,
|
|
||||||
"amp": 100
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "AO",
|
|
||||||
"features": {
|
|
||||||
"manner": "vowel",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 570,
|
|
||||||
"f2": 840,
|
|
||||||
"f3": 2410,
|
|
||||||
"bw1": 80,
|
|
||||||
"bw2": 100,
|
|
||||||
"bw3": 150,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 140,
|
|
||||||
"amp": 100
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "UH",
|
|
||||||
"features": {
|
|
||||||
"manner": "vowel",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 440,
|
|
||||||
"f2": 1020,
|
|
||||||
"f3": 2240,
|
|
||||||
"bw1": 70,
|
|
||||||
"bw2": 100,
|
|
||||||
"bw3": 150,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 110,
|
|
||||||
"amp": 100
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "UW",
|
|
||||||
"features": {
|
|
||||||
"manner": "vowel",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 300,
|
|
||||||
"f2": 870,
|
|
||||||
"f3": 2240,
|
|
||||||
"bw1": 70,
|
|
||||||
"bw2": 90,
|
|
||||||
"bw3": 150,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 140,
|
|
||||||
"amp": 100
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "AH",
|
|
||||||
"features": {
|
|
||||||
"manner": "vowel",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 640,
|
|
||||||
"f2": 1190,
|
|
||||||
"f3": 2390,
|
|
||||||
"bw1": 80,
|
|
||||||
"bw2": 100,
|
|
||||||
"bw3": 150,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 110,
|
|
||||||
"amp": 95
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "ER",
|
|
||||||
"features": {
|
|
||||||
"manner": "vowel",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 490,
|
|
||||||
"f2": 1350,
|
|
||||||
"f3": 1690,
|
|
||||||
"bw1": 80,
|
|
||||||
"bw2": 100,
|
|
||||||
"bw3": 120,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 140,
|
|
||||||
"amp": 95
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "AX",
|
|
||||||
"features": {
|
|
||||||
"manner": "vowel",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 500,
|
|
||||||
"f2": 1500,
|
|
||||||
"f3": 2500,
|
|
||||||
"bw1": 80,
|
|
||||||
"bw2": 100,
|
|
||||||
"bw3": 150,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 80,
|
|
||||||
"amp": 85
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "OW",
|
|
||||||
"features": {
|
|
||||||
"manner": "vowel",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 490,
|
|
||||||
"f2": 910,
|
|
||||||
"f3": 2380,
|
|
||||||
"bw1": 80,
|
|
||||||
"bw2": 100,
|
|
||||||
"bw3": 150,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 140,
|
|
||||||
"amp": 100
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "M",
|
|
||||||
"features": {
|
|
||||||
"manner": "nasal",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "yes"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 250,
|
|
||||||
"f2": 900,
|
|
||||||
"f3": 2200,
|
|
||||||
"bw1": 90,
|
|
||||||
"bw2": 120,
|
|
||||||
"bw3": 180,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 1,
|
|
||||||
"dur": 80,
|
|
||||||
"amp": 60
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "N",
|
|
||||||
"features": {
|
|
||||||
"manner": "nasal",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "yes"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 250,
|
|
||||||
"f2": 1700,
|
|
||||||
"f3": 2600,
|
|
||||||
"bw1": 90,
|
|
||||||
"bw2": 120,
|
|
||||||
"bw3": 180,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 1,
|
|
||||||
"dur": 80,
|
|
||||||
"amp": 60
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "NG",
|
|
||||||
"features": {
|
|
||||||
"manner": "nasal",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "yes"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 250,
|
|
||||||
"f2": 2300,
|
|
||||||
"f3": 2700,
|
|
||||||
"bw1": 90,
|
|
||||||
"bw2": 120,
|
|
||||||
"bw3": 180,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 1,
|
|
||||||
"dur": 80,
|
|
||||||
"amp": 60
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "L",
|
|
||||||
"features": {
|
|
||||||
"manner": "approximant",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 360,
|
|
||||||
"f2": 1300,
|
|
||||||
"f3": 2600,
|
|
||||||
"bw1": 80,
|
|
||||||
"bw2": 110,
|
|
||||||
"bw3": 160,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 70,
|
|
||||||
"amp": 80
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "R",
|
|
||||||
"features": {
|
|
||||||
"manner": "approximant",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 490,
|
|
||||||
"f2": 1350,
|
|
||||||
"f3": 1600,
|
|
||||||
"bw1": 80,
|
|
||||||
"bw2": 110,
|
|
||||||
"bw3": 120,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 80,
|
|
||||||
"amp": 85
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "W",
|
|
||||||
"features": {
|
|
||||||
"manner": "approximant",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 300,
|
|
||||||
"f2": 610,
|
|
||||||
"f3": 2200,
|
|
||||||
"bw1": 70,
|
|
||||||
"bw2": 100,
|
|
||||||
"bw3": 160,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 70,
|
|
||||||
"amp": 80
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "Y",
|
|
||||||
"features": {
|
|
||||||
"manner": "approximant",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 270,
|
|
||||||
"f2": 2290,
|
|
||||||
"f3": 3010,
|
|
||||||
"bw1": 60,
|
|
||||||
"bw2": 90,
|
|
||||||
"bw3": 150,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 60,
|
|
||||||
"amp": 80
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "Z",
|
|
||||||
"features": {
|
|
||||||
"manner": "fricative",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 300,
|
|
||||||
"f2": 1700,
|
|
||||||
"f3": 2500,
|
|
||||||
"bw1": 100,
|
|
||||||
"bw2": 150,
|
|
||||||
"bw3": 200,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 90,
|
|
||||||
"amp": 55
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "DH",
|
|
||||||
"features": {
|
|
||||||
"manner": "fricative",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 300,
|
|
||||||
"f2": 1400,
|
|
||||||
"f3": 2500,
|
|
||||||
"bw1": 100,
|
|
||||||
"bw2": 150,
|
|
||||||
"bw3": 200,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 70,
|
|
||||||
"amp": 55
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "V",
|
|
||||||
"features": {
|
|
||||||
"manner": "fricative",
|
|
||||||
"voiced": "yes",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 300,
|
|
||||||
"f2": 1000,
|
|
||||||
"f3": 2300,
|
|
||||||
"bw1": 100,
|
|
||||||
"bw2": 150,
|
|
||||||
"bw3": 200,
|
|
||||||
"voiced": 1,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 70,
|
|
||||||
"amp": 55
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "S",
|
|
||||||
"features": {
|
|
||||||
"manner": "fricative",
|
|
||||||
"voiced": "no",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 320,
|
|
||||||
"f2": 1700,
|
|
||||||
"f3": 2500,
|
|
||||||
"bw1": 200,
|
|
||||||
"bw2": 200,
|
|
||||||
"bw3": 250,
|
|
||||||
"voiced": 0,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 110,
|
|
||||||
"amp": 45
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "F",
|
|
||||||
"features": {
|
|
||||||
"manner": "fricative",
|
|
||||||
"voiced": "no",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 300,
|
|
||||||
"f2": 1200,
|
|
||||||
"f3": 2400,
|
|
||||||
"bw1": 200,
|
|
||||||
"bw2": 200,
|
|
||||||
"bw3": 250,
|
|
||||||
"voiced": 0,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 100,
|
|
||||||
"amp": 40
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "HH",
|
|
||||||
"features": {
|
|
||||||
"manner": "fricative",
|
|
||||||
"voiced": "no",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 500,
|
|
||||||
"f2": 1500,
|
|
||||||
"f3": 2500,
|
|
||||||
"bw1": 200,
|
|
||||||
"bw2": 250,
|
|
||||||
"bw3": 300,
|
|
||||||
"voiced": 0,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 70,
|
|
||||||
"amp": 40
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "SIL",
|
|
||||||
"features": {
|
|
||||||
"manner": "silence",
|
|
||||||
"voiced": "no",
|
|
||||||
"nasal": "no"
|
|
||||||
},
|
|
||||||
"attributes": {
|
|
||||||
"f1": 500,
|
|
||||||
"f2": 1500,
|
|
||||||
"f3": 2500,
|
|
||||||
"bw1": 100,
|
|
||||||
"bw2": 100,
|
|
||||||
"bw3": 100,
|
|
||||||
"voiced": 0,
|
|
||||||
"nasal": 0,
|
|
||||||
"dur": 55,
|
|
||||||
"amp": 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
@@ -1,45 +0,0 @@
|
|||||||
# acoustic-phonetics SOURCE — the learned speech primitives, as INGESTIBLE DATA.
|
|
||||||
# NOT audio, NOT code: formant geometry of the phonemes, to be ingested via the
|
|
||||||
# ingest organ into the engram as a phoneme manifold. The render reads this
|
|
||||||
# geometry back from the engram; nothing is frozen in EL code.
|
|
||||||
#
|
|
||||||
# PROVENANCE (audited, per-field honesty — no invented numbers):
|
|
||||||
# * The 10 MONOPHTHONG VOWEL formants F1/F2/F3 (IY,IH,EH,AE,AA,AO,UH,UW,AH,ER)
|
|
||||||
# are the MEASURED adult-male means of Peterson & Barney (1952), JASA 24:175-184
|
|
||||||
# — the canonical /hVd/ table, verified digit-for-digit vs CRAN phonTools::pb52.
|
|
||||||
# These are real measured values.
|
|
||||||
# * AX (schwa) F1/F2/F3 = neutral uniform-tube resonances (2n-1)*500 — a PHYSICS
|
|
||||||
# value (Fant), not a P&B measurement.
|
|
||||||
# * OW is a diphthong; its listed steady target is a conventional synthesis value,
|
|
||||||
# not a P&B monophthong measurement.
|
|
||||||
# * CONSONANT loci (M,N,NG,L,R,W,Y,Z,DH,V,S,F,HH) and ALL BANDWIDTHS (B1,B2,B3)
|
|
||||||
# and dur/amp are STANDARD FORMANT-SYNTHESIS conventions (Klatt 1980, JASA 67:971
|
|
||||||
# "Software for a cascade/parallel formant synthesizer") — engineering defaults,
|
|
||||||
# NOT per-phoneme field measurements. Labeled as such, not attributed to P&B.
|
|
||||||
# Format: SYM|F1|F2|F3|B1|B2|B3|voiced|nasal|dur_ms|amp|class|example
|
|
||||||
IY|270|2290|3010|60|90|150|1|0|130|100|vowel|beet
|
|
||||||
IH|390|1990|2550|70|100|150|1|0|110|100|vowel|bit
|
|
||||||
EH|530|1840|2480|80|100|150|1|0|130|100|vowel|bet
|
|
||||||
AE|660|1720|2410|90|110|150|1|0|150|100|vowel|bat
|
|
||||||
AA|730|1090|2440|90|110|150|1|0|150|100|vowel|bot
|
|
||||||
AO|570|840|2410|80|100|150|1|0|140|100|vowel|bought
|
|
||||||
UH|440|1020|2240|70|100|150|1|0|110|100|vowel|book
|
|
||||||
UW|300|870|2240|70|90|150|1|0|140|100|vowel|boot
|
|
||||||
AH|640|1190|2390|80|100|150|1|0|110|95|vowel|but
|
|
||||||
ER|490|1350|1690|80|100|120|1|0|140|95|vowel|bird
|
|
||||||
AX|500|1500|2500|80|100|150|1|0|80|85|vowel|about
|
|
||||||
OW|490|910|2380|80|100|150|1|0|140|100|vowel|boat
|
|
||||||
M|250|900|2200|90|120|180|1|1|80|60|nasal|map
|
|
||||||
N|250|1700|2600|90|120|180|1|1|80|60|nasal|nap
|
|
||||||
NG|250|2300|2700|90|120|180|1|1|80|60|nasal|sing
|
|
||||||
L|360|1300|2600|80|110|160|1|0|70|80|approximant|lip
|
|
||||||
R|490|1350|1600|80|110|120|1|0|80|85|approximant|rip
|
|
||||||
W|300|610|2200|70|100|160|1|0|70|80|approximant|wet
|
|
||||||
Y|270|2290|3010|60|90|150|1|0|60|80|approximant|yet
|
|
||||||
Z|300|1700|2500|100|150|200|1|0|90|55|fricative|zoo
|
|
||||||
DH|300|1400|2500|100|150|200|1|0|70|55|fricative|the
|
|
||||||
V|300|1000|2300|100|150|200|1|0|70|55|fricative|van
|
|
||||||
S|320|1700|2500|200|200|250|0|0|110|45|fricative|see
|
|
||||||
F|300|1200|2400|200|200|250|0|0|100|40|fricative|fee
|
|
||||||
HH|500|1500|2500|200|250|300|0|0|70|40|fricative|hat
|
|
||||||
SIL|500|1500|2500|100|100|100|0|0|55|0|silence|_
|
|
||||||
Binary file not shown.
@@ -80,11 +80,6 @@ build {
|
|||||||
"src/grammar.el",
|
"src/grammar.el",
|
||||||
"src/realizer.el",
|
"src/realizer.el",
|
||||||
"src/semantics.el",
|
"src/semantics.el",
|
||||||
"src/comprehend.el",
|
|
||||||
"src/propositions.el",
|
|
||||||
"src/multilingual.el",
|
|
||||||
"src/self_region.el",
|
|
||||||
"src/dialogue.el",
|
|
||||||
"src/elp.el",
|
"src/elp.el",
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,91 +0,0 @@
|
|||||||
> **STATUS: STAGING / PROOF-OF-SHAPE — not the deliverable.** This Python package
|
|
||||||
> proved the architecture end-to-end against the proven realizer faculty (faithful
|
|
||||||
> md/docx/midi from real geometry: 0 ungrounded claims, SACRED polarity). Per Will's
|
|
||||||
> steer, the DELIVERABLE is NATIVE: the seam lives on the existing EL realizer as
|
|
||||||
> **surface-as-profile** — see `../src/surface-profile.el` and
|
|
||||||
> `../tests/examples/surface-profile-demo.el` (compiles + runs through elc → C →
|
|
||||||
> binary). The concepts below (one geometry-carrying frame; surface = a pluggable
|
|
||||||
> profile; plan/realize; deterministic-from-meaning) are exactly what the native
|
|
||||||
> module implements. Keep this package as the validated proof; build native.
|
|
||||||
|
|
||||||
# Efferent Multimodal Projector
|
|
||||||
|
|
||||||
**geometry → any surface, faithfully.** Neuron's own document-generation faculty:
|
|
||||||
the efferent twin of the ingest organ. Ingest is afferent (world → geometry);
|
|
||||||
this is efferent (geometry → an arbitrary-format document / any modality).
|
|
||||||
|
|
||||||
Built against the **proven** realizer faculty (neuron-talk sidecar `:8756`,
|
|
||||||
artifact `art-7affa557`). The live soul (`:8742` / `:7770`) is contacted **only**
|
|
||||||
through the read-only, GET-only `engram_client` — never mutated.
|
|
||||||
|
|
||||||
## The pipeline (surface-agnostic)
|
|
||||||
|
|
||||||
```
|
|
||||||
geometry region + surface/format spec
|
|
||||||
→ PLAN (manifold → document skeleton/DAG; the geometry IS the outline) plan.py
|
|
||||||
→ REALIZE (proven realizer, scaled sentence → passage, each section faithful) realize.py
|
|
||||||
→ COHERE (document-level flow / transitions, not stitched sentences) cohere.py
|
|
||||||
→ EMIT (pluggable SurfaceProjector → the target surface) projectors/
|
|
||||||
```
|
|
||||||
|
|
||||||
**The surface is a PARAMETER.** `pipeline.build_ir(...)` builds ONE
|
|
||||||
surface-neutral `DocumentIR` (`document_ir.py`); `pipeline.emit(doc, surface)`
|
|
||||||
projects it to whichever surface you name. Markdown, docx, and MIDI are the same
|
|
||||||
IR emitted three ways.
|
|
||||||
|
|
||||||
## The pivot: a geometry-carrying IR
|
|
||||||
|
|
||||||
`DocumentIR` is **not** a text tree. Every `Block` carries BOTH:
|
|
||||||
- `.sentences` — realized faithful text (what **text** projectors read),
|
|
||||||
- `.provenance` — the source geometry: `subj_id / relation / obj / polarity /
|
|
||||||
confidence / importance / salience / node_id` (what **music / image / video**
|
|
||||||
projectors read).
|
|
||||||
|
|
||||||
That single decision is what makes the projector multimodal: text renders the
|
|
||||||
words; music/image decode the geometry. A claim with no provenance cannot exist
|
|
||||||
in the IR — faithfulness is structural.
|
|
||||||
|
|
||||||
## The one shared seam
|
|
||||||
|
|
||||||
`projectors/base.py` — `SurfaceProjector.project(frame: DocumentIR) -> bytes`
|
|
||||||
(+ `surface / media_type / ext / modality / profile`). Register with
|
|
||||||
`register()`. Adding a surface changes nothing upstream.
|
|
||||||
|
|
||||||
`TwoStageProjector` blesses the peer plan/realize decomposition:
|
|
||||||
`spec = plan(frame)`, `bytes = realize(spec)`, `project = realize∘plan`; the
|
|
||||||
`profile` is the pluggable per-surface knob (text lang-profile, music
|
|
||||||
instr/mode-profile). `projectors/midi.py` is the reference two-stage impl.
|
|
||||||
|
|
||||||
## Surfaces
|
|
||||||
|
|
||||||
| surface | modality | status | emitter |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `markdown` | text | landed | own (str) |
|
|
||||||
| `docx` | text | landed | own minimal OOXML (stdlib `zipfile`+XML, no lib) |
|
|
||||||
| `midi` | audio | landed (symbolic-music proof) | own minimal SMF (stdlib `struct`, no lib) |
|
|
||||||
| `audio` (WAV) | audio | peer agent (additive synth) | conforms to `TwoStageProjector` |
|
|
||||||
| `image` | image | documented seam | `projectors/seams.py` |
|
|
||||||
| `video` | video | documented seam (image×sound×time) | `projectors/seams.py` |
|
|
||||||
|
|
||||||
Music maps: relation → scale degree (same relation → same pitch), **polarity →
|
|
||||||
major/minor third (SACRED negation is audible)**, confidence → duration,
|
|
||||||
importance → velocity, section → register. Deterministic projection from meaning
|
|
||||||
— nothing invented.
|
|
||||||
|
|
||||||
## Faithfulness
|
|
||||||
|
|
||||||
`provenance.py` audits the IR: **zero** ungrounded claims, SACRED polarity
|
|
||||||
preserved (negations reported, never dropped), COHERE introduces no new geometry
|
|
||||||
(connectives are marked). `trace_table()` emits the geometry → section → claim
|
|
||||||
table.
|
|
||||||
|
|
||||||
## Run
|
|
||||||
|
|
||||||
```bash
|
|
||||||
PY=~/Desktop/lang-realizers/venv/bin/python
|
|
||||||
PYTHONPATH=~/Desktop/neuron-talk:~/Desktop/lang-realizers $PY generate.py
|
|
||||||
# writes ./out/{neuron-self,engram-temporal}.{md,docx,mid} + *.audit.json + *.provenance.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Requires the proven realizer env (spaCy + the neuron-talk/lang-realizers engine)
|
|
||||||
and the read-only engram at `:8742`.
|
|
||||||
@@ -1,79 +0,0 @@
|
|||||||
"""cohere.py — COHERE stage: document-level flow, not stitched sentences.
|
|
||||||
|
|
||||||
Fidelity is REALIZE's job; FLOW is this stage's. The hard part beyond sentence
|
|
||||||
fidelity is that a document must read as one thing. We add connective tissue at
|
|
||||||
the passage level:
|
|
||||||
|
|
||||||
* an opening abstract that names what the document covers (built ONLY from the
|
|
||||||
section headings that already exist — it introduces no new claim),
|
|
||||||
* a short transition lead into each section after the first, drawn from a
|
|
||||||
fixed set of discourse connectives ("Beyond that,", "Relatedly,", ...) that
|
|
||||||
carry no propositional content,
|
|
||||||
* ordering so the highest-grounded section leads.
|
|
||||||
|
|
||||||
CRITICAL: every connective is marked ``kind="connective"`` in its provenance, so
|
|
||||||
the faithfulness audit can prove COHERE introduced ZERO new geometry claims. A
|
|
||||||
transition is discourse glue, never a fact.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from document_ir import Block, DocumentIR, Provenance
|
|
||||||
|
|
||||||
# discourse connectives — pure flow, no propositional content
|
|
||||||
_TRANSITIONS = [
|
|
||||||
"Beyond that,", "Relatedly,", "In the same region,", "From there,",
|
|
||||||
"Alongside this,", "Further,", "Turning to the next facet,",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def _connective_prov() -> Provenance:
|
|
||||||
return Provenance(subj_id=None, subject=None, relation="", obj=None,
|
|
||||||
polarity="aff", confidence=1.0, node_id=None,
|
|
||||||
kind="connective")
|
|
||||||
|
|
||||||
|
|
||||||
def _abstract_block(doc: DocumentIR) -> Block:
|
|
||||||
"""A grounded opening: names the sections, asserts nothing new."""
|
|
||||||
headings = [s.heading for s in doc.sections]
|
|
||||||
if not headings:
|
|
||||||
return Block(role="lead")
|
|
||||||
if len(headings) == 1:
|
|
||||||
body = f"This document, generated from Neuron's geometry, covers {headings[0]}."
|
|
||||||
else:
|
|
||||||
listed = ", ".join(headings[:-1]) + f", and {headings[-1]}"
|
|
||||||
body = ("This document is projected directly from Neuron's meaning-geometry. "
|
|
||||||
f"It traces {listed}.")
|
|
||||||
b = Block(role="lead")
|
|
||||||
b.sentences.append(body)
|
|
||||||
b.provenance.append(_connective_prov())
|
|
||||||
return b
|
|
||||||
|
|
||||||
|
|
||||||
def cohere_document(doc: DocumentIR, *, add_abstract: bool = True,
|
|
||||||
add_transitions: bool = True) -> DocumentIR:
|
|
||||||
"""Order sections by grounding, add abstract + transitions (flow only)."""
|
|
||||||
# order: strongest-grounded section (mean confidence x #claims) first,
|
|
||||||
# but keep an explicitly-first section if the plan pinned one via level 1.
|
|
||||||
def _score(sec):
|
|
||||||
provs = [p for p in sec.all_provenance() if p.kind == "fact"]
|
|
||||||
if not provs:
|
|
||||||
return 0.0
|
|
||||||
mean_conf = sum(p.confidence for p in provs) / len(provs)
|
|
||||||
return mean_conf * len(provs)
|
|
||||||
|
|
||||||
doc.sections.sort(key=_score, reverse=True)
|
|
||||||
|
|
||||||
if add_transitions:
|
|
||||||
for i, sec in enumerate(doc.sections):
|
|
||||||
if i == 0 or not sec.blocks:
|
|
||||||
continue
|
|
||||||
lead = _TRANSITIONS[(i - 1) % len(_TRANSITIONS)]
|
|
||||||
first = sec.blocks[0]
|
|
||||||
if first.sentences:
|
|
||||||
# prepend the connective to the first sentence (flow, no new claim)
|
|
||||||
first.sentences[0] = f"{lead} {first.sentences[0][0].lower()}{first.sentences[0][1:]}"
|
|
||||||
|
|
||||||
if add_abstract:
|
|
||||||
doc.meta["abstract"] = _abstract_block(doc)
|
|
||||||
|
|
||||||
return doc
|
|
||||||
@@ -1,111 +0,0 @@
|
|||||||
"""document_ir.py — the surface-neutral, GEOMETRY-CARRYING document intermediate.
|
|
||||||
|
|
||||||
This is the pivot of the whole efferent projector. A DocumentIR is NOT a text
|
|
||||||
tree. It is a projection of a meaning-geometry region that carries, at every
|
|
||||||
leaf, BOTH:
|
|
||||||
|
|
||||||
* the realized surface text (``Block.sentences``) — what a TEXT projector reads,
|
|
||||||
* the source geometry (``Block.provenance``) — what a MUSIC / IMAGE /
|
|
||||||
VIDEO projector reads.
|
|
||||||
|
|
||||||
Because the IR holds the geometry, not just the words, the SAME
|
|
||||||
plan -> realize -> cohere pipeline drives every surface. A markdown projector
|
|
||||||
renders the sentences; a music projector reads the provenance edges (salience,
|
|
||||||
importance, polarity, relation) and maps them onto a symbolic-music surface;
|
|
||||||
an image/video projector (documented seam) would read the same geometry.
|
|
||||||
|
|
||||||
Nothing in this module invents content. Every :class:`Provenance` points at a
|
|
||||||
real engram node id and a real relation. That is the faithfulness contract made
|
|
||||||
structural: a claim with no provenance cannot exist in the IR.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Provenance — the geometry an emitted claim traces to. FAITHFULNESS is here.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
@dataclass
|
|
||||||
class Provenance:
|
|
||||||
"""One geometry edge behind one realized claim.
|
|
||||||
|
|
||||||
``kind`` distinguishes a FACT (a structural edge asserted by the geometry,
|
|
||||||
spoken as fact) from an INTERPRETATION (something attributed, spoken with
|
|
||||||
attribution) — the facts-as-facts + interpretations-attributed discipline
|
|
||||||
(memory 80927e26). ``polarity`` is SACRED: a negated edge stays negated.
|
|
||||||
"""
|
|
||||||
subj_id: str | None # source engram node id of the subject
|
|
||||||
subject: str | None # normalized subject surface
|
|
||||||
relation: str # predicate lemma (e.g. "use", "contain", "be")
|
|
||||||
obj: str | None # normalized object / complement surface
|
|
||||||
polarity: str = "aff" # "aff" | "neg" (SACRED — never silently flipped)
|
|
||||||
confidence: float = 0.0 # extraction confidence in [0,1]
|
|
||||||
node_id: str | None = None # engram node the claim was extracted from
|
|
||||||
kind: str = "fact" # "fact" | "interpretation"
|
|
||||||
importance: float = 0.0 # source node importance (drives music/emphasis)
|
|
||||||
salience: float = 0.0 # source node salience
|
|
||||||
|
|
||||||
def trace(self) -> str:
|
|
||||||
arrow = "-->" if self.polarity == "aff" else "--NOT-->"
|
|
||||||
return (f"[{(self.node_id or '?')[:8]}] {self.subject!r} {arrow}"
|
|
||||||
f"{self.relation} {self.obj!r} (conf {self.confidence:.2f})")
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class Block:
|
|
||||||
"""A passage: one or more faithful sentences + the geometry they trace to.
|
|
||||||
|
|
||||||
``sentences`` and ``provenance`` are index-aligned where possible: sentence
|
|
||||||
``i`` was realized from ``provenance[i]``. A COHERE transition sentence with
|
|
||||||
no new geometry carries a provenance whose ``kind == "connective"`` so the
|
|
||||||
audit can see it introduced no new claim.
|
|
||||||
"""
|
|
||||||
sentences: list[str] = field(default_factory=list)
|
|
||||||
provenance: list[Provenance] = field(default_factory=list)
|
|
||||||
role: str = "body" # "body" | "lead" | "transition"
|
|
||||||
|
|
||||||
def text(self) -> str:
|
|
||||||
return " ".join(s.rstrip(". ") + "." for s in self.sentences if s.strip())
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class Section:
|
|
||||||
heading: str
|
|
||||||
level: int = 2 # markdown heading level / outline depth
|
|
||||||
blocks: list[Block] = field(default_factory=list)
|
|
||||||
seed_ids: list[str] = field(default_factory=list) # geometry nodes of section
|
|
||||||
summary: str = "" # one-line grounded gloss (for pptx bullets / TOC)
|
|
||||||
|
|
||||||
def all_provenance(self) -> list[Provenance]:
|
|
||||||
out: list[Provenance] = []
|
|
||||||
for b in self.blocks:
|
|
||||||
out.extend(b.provenance)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class DocumentIR:
|
|
||||||
"""The surface-neutral document. Built ONCE, projected to ANY surface."""
|
|
||||||
title: str
|
|
||||||
subtitle: str = ""
|
|
||||||
sections: list[Section] = field(default_factory=list)
|
|
||||||
seed_id: str | None = None # the geometry region root
|
|
||||||
format_spec: dict[str, Any] = field(default_factory=dict) # requested shape
|
|
||||||
meta: dict[str, Any] = field(default_factory=dict)
|
|
||||||
|
|
||||||
# -- geometry facets (what non-text projectors consume) ----------------- #
|
|
||||||
def all_provenance(self) -> list[Provenance]:
|
|
||||||
out: list[Provenance] = []
|
|
||||||
for s in self.sections:
|
|
||||||
out.extend(s.all_provenance())
|
|
||||||
return out
|
|
||||||
|
|
||||||
def claim_count(self) -> int:
|
|
||||||
return sum(1 for p in self.all_provenance() if p.kind in ("fact", "interpretation"))
|
|
||||||
|
|
||||||
def ungrounded_count(self) -> int:
|
|
||||||
"""Claims with no traceable node — MUST be zero for a faithful doc."""
|
|
||||||
return sum(1 for p in self.all_provenance()
|
|
||||||
if p.kind in ("fact", "interpretation") and not p.node_id)
|
|
||||||
@@ -1,81 +0,0 @@
|
|||||||
"""generate.py — drive the projector: one geometry region -> many surfaces.
|
|
||||||
|
|
||||||
Proves the thesis with REAL output: builds ONE surface-neutral DocumentIR from
|
|
||||||
Neuron's OWN self-geometry (read-only against the live soul via the proven
|
|
||||||
faculty), then EMITS it to Markdown, docx, and MIDI — the same plan/realize/
|
|
||||||
cohere, three surfaces. Writes the files + the faithfulness audit to ./out/.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
sys.path.insert(0, _HERE)
|
|
||||||
|
|
||||||
import pipeline # noqa: E402
|
|
||||||
import provenance # noqa: E402
|
|
||||||
from geometry import load_self_region # noqa: E402
|
|
||||||
|
|
||||||
OUT = os.path.join(_HERE, "out")
|
|
||||||
|
|
||||||
|
|
||||||
def _emit_all(doc, stem):
|
|
||||||
"""Emit one IR to every text/audio surface + audit + provenance."""
|
|
||||||
for surface in ("markdown", "docx", "midi"):
|
|
||||||
data = pipeline.emit(doc, surface)
|
|
||||||
proj = pipeline.get_projector(surface)
|
|
||||||
path = os.path.join(OUT, f"{stem}.{proj.ext}")
|
|
||||||
with open(path, "wb") as f:
|
|
||||||
f.write(data)
|
|
||||||
print(f" emitted {surface:9s} -> {os.path.basename(path)} ({len(data)} bytes)")
|
|
||||||
a = provenance.audit(doc)
|
|
||||||
with open(os.path.join(OUT, f"{stem}.audit.json"), "w") as f:
|
|
||||||
json.dump(a, f, indent=2)
|
|
||||||
with open(os.path.join(OUT, f"{stem}.provenance.md"), "w") as f:
|
|
||||||
f.write(provenance.trace_table(doc))
|
|
||||||
print(" audit:", {k: a[k] for k in ("claims", "ungrounded_claims",
|
|
||||||
"negations_preserved", "distinct_source_nodes", "faithful")})
|
|
||||||
return a
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
os.makedirs(OUT, exist_ok=True)
|
|
||||||
print("surfaces registered:", pipeline.available_surfaces())
|
|
||||||
|
|
||||||
# ---- Document 1: Neuron's self-description (marquee) ------------------- #
|
|
||||||
print("\n[1] Neuron self-description")
|
|
||||||
region = load_self_region(max_nodes=9)
|
|
||||||
print(" self region:", region)
|
|
||||||
doc1 = pipeline.build_ir(
|
|
||||||
None, region=region,
|
|
||||||
title="Neuron: A Self-Description from Its Own Geometry",
|
|
||||||
subtitle="Projected efferently from the engram — every claim traces a node.",
|
|
||||||
format_spec={"genre": "self-description", "register": "expository"},
|
|
||||||
max_sections=5, conf_floor=0.6)
|
|
||||||
print(f" IR: {len(doc1.sections)} sections, {doc1.claim_count()} claims, "
|
|
||||||
f"ungrounded={doc1.ungrounded_count()}")
|
|
||||||
_emit_all(doc1, "neuron-self")
|
|
||||||
|
|
||||||
# ---- Document 2: a coherent, clean whitepaper-style section ------------ #
|
|
||||||
print("\n[2] Whitepaper-style section (coherent clean region)")
|
|
||||||
doc2, _ = pipeline.project(
|
|
||||||
["chronoception", "time", "awareness", "engram", "temporal"],
|
|
||||||
surface="markdown",
|
|
||||||
title="Temporal Awareness in the Engram",
|
|
||||||
subtitle="A section projected from the geometry of chronoception.",
|
|
||||||
format_spec={"genre": "whitepaper-section", "register": "technical"},
|
|
||||||
max_sections=4)
|
|
||||||
print(f" IR: {len(doc2.sections)} sections, {doc2.claim_count()} claims, "
|
|
||||||
f"ungrounded={doc2.ungrounded_count()}")
|
|
||||||
_emit_all(doc2, "engram-temporal")
|
|
||||||
|
|
||||||
# echo both markdowns so they are visible in the run log
|
|
||||||
for stem, doc in (("neuron-self", doc1), ("engram-temporal", doc2)):
|
|
||||||
print(f"\n===== GENERATED MARKDOWN — {stem} =====\n")
|
|
||||||
print(pipeline.emit(doc, "markdown").decode())
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,129 +0,0 @@
|
|||||||
"""geometry.py — READ-ONLY loader for a meaning-geometry region.
|
|
||||||
|
|
||||||
The efferent projector never writes to the soul. This module reaches the
|
|
||||||
geometry through the PROVEN, read-only neuron-talk faculty (``engram_client``,
|
|
||||||
GET-only, which physically refuses non-GET methods) against the running sidecar
|
|
||||||
soul. The live daemon :8742 / :7770 is contacted ONLY through that read-only
|
|
||||||
client — never mutated.
|
|
||||||
|
|
||||||
A "region" is a seed node plus a bounded neighborhood: the manifold that will
|
|
||||||
become the document's skeleton. We pool a few single-term lexical searches
|
|
||||||
(the engram search is a single-term matcher) and, when available, walk one hop
|
|
||||||
of reified neighbors, then rank by self/importance signal.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
# Wire in the proven faculty (own-the-core: we reuse it, we do not fork it).
|
|
||||||
_NT = os.path.expanduser("~/Desktop/neuron-talk")
|
|
||||||
_LR = os.path.expanduser("~/Desktop/lang-realizers")
|
|
||||||
for _p in (_NT, _LR):
|
|
||||||
if _p not in sys.path:
|
|
||||||
sys.path.insert(0, _p)
|
|
||||||
|
|
||||||
from engram_client import ReadOnlyEngramClient # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
class Region:
|
|
||||||
"""A geometry region: ranked nodes + the reified edges among them."""
|
|
||||||
|
|
||||||
def __init__(self, seed: str, nodes: list[dict], edges: list[dict]):
|
|
||||||
self.seed = seed
|
|
||||||
self.nodes = nodes # ranked engram node dicts
|
|
||||||
self.edges = edges # [{src, dst, edge, ...}]
|
|
||||||
self.by_id = {n["id"]: n for n in nodes if n.get("id")}
|
|
||||||
|
|
||||||
def __repr__(self):
|
|
||||||
return f"<Region seed={self.seed!r} nodes={len(self.nodes)} edges={len(self.edges)}>"
|
|
||||||
|
|
||||||
|
|
||||||
def _prose_quality(content: str) -> float:
|
|
||||||
"""Reward clean expository prose; penalize shouty banner-dense nodes.
|
|
||||||
|
|
||||||
A high ALLCAPS-word ratio or very short content signals a banner/telegraphic
|
|
||||||
memory node that extracts into garbage. Clean declarative prose scores high.
|
|
||||||
"""
|
|
||||||
if not content or not content.strip():
|
|
||||||
return 0.0
|
|
||||||
words = content.split()
|
|
||||||
if len(words) < 8:
|
|
||||||
return 0.1
|
|
||||||
caps = sum(1 for w in words if len(w) > 2 and w.strip(".,:;'\"-").isupper())
|
|
||||||
caps_ratio = caps / max(1, len(words))
|
|
||||||
# sentences with lowercase interior words read as prose
|
|
||||||
lower = sum(1 for w in words if w[:1].islower())
|
|
||||||
lower_ratio = lower / max(1, len(words))
|
|
||||||
return max(0.0, 1.2 * lower_ratio - 2.0 * caps_ratio)
|
|
||||||
|
|
||||||
|
|
||||||
def _relevance(content: str, terms: list[str]) -> float:
|
|
||||||
"""Topical relevance to the seed terms — keeps a region ON-THEME so a clean
|
|
||||||
but off-topic node cannot hijack the document."""
|
|
||||||
if not terms:
|
|
||||||
return 0.0
|
|
||||||
low = (content or "").lower()
|
|
||||||
hits = sum(1 for t in terms if t.lower() in low)
|
|
||||||
return hits / max(1, len(terms))
|
|
||||||
|
|
||||||
|
|
||||||
def _node_rank(n: dict, terms: list[str] | None = None) -> float:
|
|
||||||
return (float(n.get("importance") or 0.0) * 2.0
|
|
||||||
+ float(n.get("salience") or 0.0)
|
|
||||||
+ 1.5 * _prose_quality(n.get("content") or "")
|
|
||||||
+ 2.0 * _relevance(n.get("content") or "", terms or [])
|
|
||||||
+ (0.5 if (n.get("content") or "").strip() else 0.0))
|
|
||||||
|
|
||||||
|
|
||||||
def load_region(seed_terms: list[str] | str, *, client: ReadOnlyEngramClient | None = None,
|
|
||||||
max_nodes: int = 10, per_term: int = 20, hop: bool = True) -> Region:
|
|
||||||
"""Pull a bounded geometry region around ``seed_terms`` (read-only).
|
|
||||||
|
|
||||||
``seed_terms`` may be a single string or several probe terms; results are
|
|
||||||
pooled and de-duplicated. When ``hop`` and the reified neighbor endpoint is
|
|
||||||
live, one hop of neighbors is folded in so the region is a real
|
|
||||||
neighborhood, not just a keyword hit list.
|
|
||||||
"""
|
|
||||||
client = client or ReadOnlyEngramClient()
|
|
||||||
if isinstance(seed_terms, str):
|
|
||||||
seed_terms = [seed_terms]
|
|
||||||
|
|
||||||
pool: dict[str, dict] = {}
|
|
||||||
for term in seed_terms:
|
|
||||||
for n in client.search(term, limit=per_term):
|
|
||||||
if isinstance(n, dict) and n.get("id"):
|
|
||||||
pool.setdefault(n["id"], n)
|
|
||||||
|
|
||||||
ranked = sorted(pool.values(), key=lambda n: _node_rank(n, seed_terms),
|
|
||||||
reverse=True)
|
|
||||||
nodes = ranked[:max_nodes]
|
|
||||||
|
|
||||||
edges: list[dict] = []
|
|
||||||
if hop and nodes:
|
|
||||||
present = {n["id"] for n in nodes}
|
|
||||||
for n in list(nodes):
|
|
||||||
try:
|
|
||||||
for nb in client.neighbors(n["id"]):
|
|
||||||
node = nb.get("node") if isinstance(nb, dict) else None
|
|
||||||
edge = nb.get("edge") if isinstance(nb, dict) else None
|
|
||||||
if node and node.get("id"):
|
|
||||||
edges.append({"src": n["id"], "dst": node["id"],
|
|
||||||
"edge": edge})
|
|
||||||
# fold a strong neighbor into the region (bounded)
|
|
||||||
if (node["id"] not in present and len(nodes) < max_nodes + 6
|
|
||||||
and _node_rank(node, seed_terms) > 0.4):
|
|
||||||
present.add(node["id"])
|
|
||||||
nodes.append(node)
|
|
||||||
except Exception: # noqa: BLE001 — read-only best-effort; never fatal
|
|
||||||
continue
|
|
||||||
|
|
||||||
return Region(seed=", ".join(seed_terms), nodes=nodes, edges=edges)
|
|
||||||
|
|
||||||
|
|
||||||
def load_self_region(client: ReadOnlyEngramClient | None = None,
|
|
||||||
max_nodes: int = 10) -> Region:
|
|
||||||
"""The self/identity region — Neuron's own geometry, for self-description."""
|
|
||||||
return load_region(["self", "identity", "Neuron", "values", "memory",
|
|
||||||
"imprint", "consciousness"],
|
|
||||||
client=client, max_nodes=max_nodes)
|
|
||||||
@@ -1,67 +0,0 @@
|
|||||||
"""pipeline.py — the Efferent Multimodal Projector, top level.
|
|
||||||
|
|
||||||
geometry region + surface/format spec
|
|
||||||
-> PLAN (manifold -> document skeleton/DAG)
|
|
||||||
-> REALIZE (proven realizer, sentence -> passage, each section faithful)
|
|
||||||
-> COHERE (document-level flow / transitions, not stitched sentences)
|
|
||||||
-> EMIT (pluggable SurfaceProjector -> the target surface)
|
|
||||||
|
|
||||||
THE SURFACE IS A PARAMETER. ``project(...)`` builds the geometry-carrying
|
|
||||||
DocumentIR once, then hands it to whichever surface projector the caller named.
|
|
||||||
Markdown, docx, and midi (music) are all the SAME IR emitted differently. That
|
|
||||||
is the efferent multimodal projector: geometry -> any surface.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
sys.path.insert(0, _HERE)
|
|
||||||
sys.path.insert(0, os.path.join(_HERE, "projectors"))
|
|
||||||
|
|
||||||
from cohere import cohere_document # noqa: E402
|
|
||||||
from document_ir import DocumentIR # noqa: E402
|
|
||||||
from geometry import Region, load_region # noqa: E402
|
|
||||||
from plan import plan_document # noqa: E402
|
|
||||||
from realize import realize_document # noqa: E402
|
|
||||||
|
|
||||||
# registering the projectors (import for side-effect: each self-registers)
|
|
||||||
import projectors.markdown # noqa: E402,F401
|
|
||||||
import projectors.docx # noqa: E402,F401
|
|
||||||
import projectors.midi # noqa: E402,F401
|
|
||||||
import projectors.seams # noqa: E402,F401
|
|
||||||
from projectors.base import available_surfaces, get_projector # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
def build_ir(seed_terms, *, title: str, subtitle: str = "",
|
|
||||||
format_spec: dict | None = None,
|
|
||||||
region: Region | None = None,
|
|
||||||
max_sections: int = 8, conf_floor: float = 0.55) -> DocumentIR:
|
|
||||||
"""geometry -> PLAN -> REALIZE -> COHERE = the surface-neutral DocumentIR."""
|
|
||||||
region = region or load_region(seed_terms)
|
|
||||||
doc = plan_document(region, title=title, subtitle=subtitle,
|
|
||||||
format_spec=format_spec or {},
|
|
||||||
conf_floor=conf_floor, max_sections=max_sections)
|
|
||||||
doc = realize_document(doc)
|
|
||||||
doc = cohere_document(doc)
|
|
||||||
return doc
|
|
||||||
|
|
||||||
|
|
||||||
def emit(doc: DocumentIR, surface: str) -> bytes:
|
|
||||||
"""EMIT: project the built IR onto one surface (surface = a parameter)."""
|
|
||||||
return get_projector(surface).project(doc)
|
|
||||||
|
|
||||||
|
|
||||||
def project(seed_terms, *, surface: str, title: str, subtitle: str = "",
|
|
||||||
format_spec: dict | None = None, region: Region | None = None,
|
|
||||||
max_sections: int = 8) -> tuple[DocumentIR, bytes]:
|
|
||||||
"""The full efferent projection: geometry + surface -> (IR, bytes)."""
|
|
||||||
doc = build_ir(seed_terms, title=title, subtitle=subtitle,
|
|
||||||
format_spec=format_spec, region=region,
|
|
||||||
max_sections=max_sections)
|
|
||||||
return doc, emit(doc, surface)
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["build_ir", "emit", "project", "available_surfaces",
|
|
||||||
"get_projector", "load_region", "DocumentIR"]
|
|
||||||
@@ -1,192 +0,0 @@
|
|||||||
"""plan.py — PLAN stage: geometry region -> document skeleton (a DAG/outline).
|
|
||||||
|
|
||||||
The manifold becomes the skeleton. We extract faithful propositions from the
|
|
||||||
region's nodes (the proven neuron-talk extractor, SACRED polarity preserved),
|
|
||||||
apply a quality floor, then GROUP them into sections. Grouping is by source
|
|
||||||
node — each engram node is one coherent topic, so one salient node becomes one
|
|
||||||
section. The section ORDER is the node ranking (importance/salience): the
|
|
||||||
geometry decides the outline, not a template.
|
|
||||||
|
|
||||||
Output: a DocumentIR whose sections carry seed node ids and empty blocks. REALIZE
|
|
||||||
fills the blocks; the plan owns the structure.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
|
|
||||||
_NT = os.path.expanduser("~/Desktop/neuron-talk")
|
|
||||||
_LR = os.path.expanduser("~/Desktop/lang-realizers")
|
|
||||||
for _p in (_NT, _LR):
|
|
||||||
if _p not in sys.path:
|
|
||||||
sys.path.insert(0, _p)
|
|
||||||
|
|
||||||
import propositions # noqa: E402 (the proven, faithful extractor)
|
|
||||||
|
|
||||||
from document_ir import DocumentIR, Section # noqa: E402
|
|
||||||
from geometry import Region # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Proposition quality — keep only clean, well-grounded claims.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
_JUNK_RE = re.compile(r"[.][a-z]{1,3}\b|[^A-Za-z0-9 '\-]") # ".o", stray symbols
|
|
||||||
|
|
||||||
|
|
||||||
def _has_banner_token(s: str) -> bool:
|
|
||||||
"""True if any word is an ALLCAPS banner token (DHARMA, ENGRAM, MEASURED)."""
|
|
||||||
for w in (s or "").split():
|
|
||||||
core = w.strip(".,:;'\"-")
|
|
||||||
if len(core) > 2 and core.isupper():
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def _clean_prop(p, floor: float) -> bool:
|
|
||||||
if p.confidence < floor:
|
|
||||||
return False
|
|
||||||
if not p.subject or not (p.object or (p.obj_np is not None)):
|
|
||||||
return False
|
|
||||||
subj = (p.subject or "").strip()
|
|
||||||
obj = (p.object or "").strip()
|
|
||||||
if len(subj) < 2:
|
|
||||||
return False
|
|
||||||
# banner-derived shouty fragments read as garbage in prose
|
|
||||||
if _has_banner_token(subj) or _has_banner_token(obj):
|
|
||||||
return False
|
|
||||||
if propositions._is_shouty(p.sentence or ""):
|
|
||||||
return False
|
|
||||||
# junk tokens: file-extension fragments (".o"), stray non-word symbols
|
|
||||||
if _JUNK_RE.search(subj) or _JUNK_RE.search(obj):
|
|
||||||
return False
|
|
||||||
# a proposition whose object repeats the subject is usually a parse artifact
|
|
||||||
if obj and subj.lower() == obj.lower():
|
|
||||||
return False
|
|
||||||
# a bare copula with no real complement ("X is it") reads as noise
|
|
||||||
if p.predicate == "be" and obj.lower() in ("it", "no", "nothing", "empty", ""):
|
|
||||||
return False
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def _dedup(props):
|
|
||||||
"""Drop duplicate claims. Two axes: (a) identical (pred,obj,polarity), and
|
|
||||||
(b) same (subject,predicate) — which collapses a mis-split compound like
|
|
||||||
"detection is post-hoc eval" -> "Detection is post/hoc/eval" into one claim
|
|
||||||
(keep the highest-confidence surface)."""
|
|
||||||
props = sorted(props, key=lambda p: p.confidence, reverse=True)
|
|
||||||
seen_po, seen_sp, out = set(), set(), []
|
|
||||||
for p in props:
|
|
||||||
subj = (p.subject or "").lower()
|
|
||||||
po = (p.predicate, (p.object or "").lower(), p.polarity)
|
|
||||||
sp = (subj, p.predicate, p.polarity)
|
|
||||||
if po in seen_po or sp in seen_sp:
|
|
||||||
continue
|
|
||||||
seen_po.add(po)
|
|
||||||
seen_sp.add(sp)
|
|
||||||
out.append(p)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Heading derivation — a clean human heading from a node.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
_HEADING_RE = re.compile(r"^\s*#{1,4}\s+(.{2,70})\s*$", re.M)
|
|
||||||
# node-type / system labels that are NOT topical headings
|
|
||||||
_NONTOPIC_LABEL = re.compile(r"^(memory|node|knowledge|doc|session)[:/]", re.I)
|
|
||||||
|
|
||||||
|
|
||||||
def _titlecase_banner(s: str) -> str:
|
|
||||||
"""A shouty banner ("CHRONOCEPTION — SCALE-INVARIANCE") makes a fine title
|
|
||||||
once Title-cased. Keep short acronyms uppercase."""
|
|
||||||
def fix(w):
|
|
||||||
core = w.strip("—-:,.")
|
|
||||||
if len(core) <= 3 and core.isupper():
|
|
||||||
return w # acronym
|
|
||||||
return w.capitalize()
|
|
||||||
return " ".join(fix(w) for w in s.split())
|
|
||||||
|
|
||||||
|
|
||||||
def _clean_heading(text: str) -> str | None:
|
|
||||||
"""First line only, no markdown, capped, banner Title-cased. None if unusable."""
|
|
||||||
if not text:
|
|
||||||
return None
|
|
||||||
line = text.strip().splitlines()[0]
|
|
||||||
line = re.sub(r"^#+\s*", "", line).strip().strip("#").strip()
|
|
||||||
# cut at a natural break so a long banner heading stays a heading, not a para
|
|
||||||
for sep in (" — ", " – ", ": ", ". "):
|
|
||||||
if sep in line and len(line) > 48:
|
|
||||||
line = line.split(sep)[0].strip()
|
|
||||||
break
|
|
||||||
if not (3 <= len(line) <= 64):
|
|
||||||
return None
|
|
||||||
if propositions._is_shouty(line):
|
|
||||||
line = _titlecase_banner(line)
|
|
||||||
return line or None
|
|
||||||
|
|
||||||
|
|
||||||
def _heading_for(node: dict, fallback: str) -> str:
|
|
||||||
label = (node.get("label") or "").strip()
|
|
||||||
content = node.get("content") or ""
|
|
||||||
candidates: list[str] = []
|
|
||||||
# a node-type label ("memory:remembered") is never a topic — skip it
|
|
||||||
if label and not _NONTOPIC_LABEL.match(label):
|
|
||||||
candidates.append(label)
|
|
||||||
m = _HEADING_RE.search(content)
|
|
||||||
if m:
|
|
||||||
candidates.append(m.group(1))
|
|
||||||
# the leading banner/first sentence of the content is often the real title
|
|
||||||
first = re.split(r"(?<=[.\n])", content.strip(), maxsplit=1)[0] if content.strip() else ""
|
|
||||||
candidates.append(first)
|
|
||||||
for c in candidates:
|
|
||||||
h = _clean_heading(c)
|
|
||||||
if h:
|
|
||||||
return h
|
|
||||||
return fallback
|
|
||||||
|
|
||||||
|
|
||||||
def plan_document(region: Region, *, title: str, subtitle: str = "",
|
|
||||||
format_spec: dict | None = None,
|
|
||||||
conf_floor: float = 0.55,
|
|
||||||
max_sections: int = 8,
|
|
||||||
max_claims_per_section: int = 6) -> DocumentIR:
|
|
||||||
"""Region -> DocumentIR skeleton. The geometry dictates the outline."""
|
|
||||||
format_spec = format_spec or {}
|
|
||||||
doc = DocumentIR(title=title, subtitle=subtitle,
|
|
||||||
seed_id=region.nodes[0]["id"] if region.nodes else None,
|
|
||||||
format_spec=format_spec)
|
|
||||||
|
|
||||||
made = 0
|
|
||||||
seen_headings: set[str] = set()
|
|
||||||
for node in region.nodes:
|
|
||||||
if made >= max_sections:
|
|
||||||
break
|
|
||||||
props = propositions.extract(node.get("content") or "",
|
|
||||||
node_id=node.get("id"),
|
|
||||||
node_importance=float(node.get("importance") or 0.0),
|
|
||||||
max_sentences=10)
|
|
||||||
props = [p for p in props if _clean_prop(p, conf_floor)]
|
|
||||||
props = _dedup(props)
|
|
||||||
props.sort(key=lambda p: p.confidence, reverse=True)
|
|
||||||
props = props[:max_claims_per_section]
|
|
||||||
if not props:
|
|
||||||
continue
|
|
||||||
heading = _heading_for(node, fallback=f"Region {made + 1}")
|
|
||||||
# cross-section dedup: a topic appears once. Distinguish by top claim
|
|
||||||
# subject, else drop the collision so the outline stays clean.
|
|
||||||
if heading.lower() in seen_headings:
|
|
||||||
subj = (props[0].subject or "").strip().title()
|
|
||||||
alt = f"{heading}: {subj}" if subj and subj.lower() not in heading.lower() else None
|
|
||||||
if alt and alt.lower() not in seen_headings and len(alt) <= 64:
|
|
||||||
heading = alt
|
|
||||||
else:
|
|
||||||
continue
|
|
||||||
seen_headings.add(heading.lower())
|
|
||||||
sec = Section(heading=heading, level=2, seed_ids=[node["id"]])
|
|
||||||
# stash the planned propositions on the section for REALIZE
|
|
||||||
sec.__dict__["_planned_props"] = props
|
|
||||||
sec.__dict__["_node"] = node
|
|
||||||
doc.sections.append(sec)
|
|
||||||
made += 1
|
|
||||||
|
|
||||||
return doc
|
|
||||||
@@ -1,106 +0,0 @@
|
|||||||
"""base.py — the SurfaceProjector interface + registry.
|
|
||||||
|
|
||||||
THE key abstraction of the efferent projector: a projector is a pure function
|
|
||||||
from the surface-neutral, geometry-carrying DocumentIR to bytes on a target
|
|
||||||
SURFACE. The surface is a PARAMETER. Adding a surface = registering one more
|
|
||||||
projector; nothing upstream (plan/realize/cohere) changes.
|
|
||||||
|
|
||||||
DocumentIR --project--> bytes (per surface)
|
|
||||||
|
|
||||||
A TEXT projector reads ``block.sentences``. A NON-TEXT projector (music, image,
|
|
||||||
video) reads ``block.provenance`` — the geometry the IR carries — and decodes it
|
|
||||||
onto its surface. Both consume the SAME IR. That symmetry is the whole design:
|
|
||||||
the realizer generalizes into a multimodal projector, geometry -> any surface.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Protocol, runtime_checkable
|
|
||||||
|
|
||||||
import sys
|
|
||||||
import os
|
|
||||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
|
||||||
from document_ir import DocumentIR # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
@runtime_checkable
|
|
||||||
class SurfaceProjector(Protocol):
|
|
||||||
"""Geometry-document -> one surface. Implementations MUST be pure & faithful.
|
|
||||||
|
|
||||||
THE ONE SHARED SEAM. Every surface — text, music, image, video — conforms to
|
|
||||||
this single contract:
|
|
||||||
|
|
||||||
project(frame: DocumentIR) -> bytes
|
|
||||||
|
|
||||||
where ``frame`` is the geometry-carrying meaning-geometry (the SemFrame at
|
|
||||||
document scale; a single utterance is the degenerate one-section frame).
|
|
||||||
|
|
||||||
RECOMMENDED INTERNAL SHAPE (the peer music/text decomposition, blessed here
|
|
||||||
so all surfaces share it): a projector may split ``project`` into
|
|
||||||
|
|
||||||
spec = self.plan(frame) # meaning-geometry -> surface-specific spec
|
|
||||||
bytes = self.realize(spec) # spec -> surface, via this projector's PROFILE
|
|
||||||
|
|
||||||
``project`` is then ``realize(plan(frame))``. The PROFILE (a text lang-profile,
|
|
||||||
a music instr/mode-profile, an image layout-profile) is a property of the
|
|
||||||
projector instance — the pluggable knob. See :class:`TwoStageProjector`.
|
|
||||||
|
|
||||||
A TEXT projector's plan reads ``frame`` sentences; a MUSIC/IMAGE projector's
|
|
||||||
plan reads ``frame.all_provenance()`` — the geometry — and derives its spec
|
|
||||||
(pitch/harmony/rhythm, or layout) FROM the meaning, deterministically. Same
|
|
||||||
frame, different profile.
|
|
||||||
"""
|
|
||||||
|
|
||||||
surface: str # "markdown" | "docx" | "midi" | "audio" | "image" | "video"
|
|
||||||
media_type: str # MIME type of the emitted bytes
|
|
||||||
ext: str # file extension (no dot)
|
|
||||||
modality: str # "text" | "audio" | "image" | "video"
|
|
||||||
profile: object # the pluggable per-surface profile (may be None)
|
|
||||||
|
|
||||||
def project(self, doc: DocumentIR) -> bytes:
|
|
||||||
"""Emit the document on this surface. Returns raw bytes."""
|
|
||||||
...
|
|
||||||
|
|
||||||
|
|
||||||
class TwoStageProjector:
|
|
||||||
"""Optional base for the peer plan()/realize() decomposition.
|
|
||||||
|
|
||||||
Subclasses implement ``plan(frame) -> spec`` and ``realize(spec) -> bytes``;
|
|
||||||
``project`` is their composition. This is exactly the peer music interface
|
|
||||||
(spec = plan(frame, profile); surface = realize(spec, profile)) expressed so
|
|
||||||
that it still satisfies the single ``SurfaceProjector.project`` seam. Text,
|
|
||||||
music, and image projectors can all subclass this and remain interchangeable.
|
|
||||||
"""
|
|
||||||
|
|
||||||
surface: str = ""
|
|
||||||
media_type: str = ""
|
|
||||||
ext: str = ""
|
|
||||||
modality: str = ""
|
|
||||||
profile: object = None
|
|
||||||
|
|
||||||
def plan(self, doc: DocumentIR): # -> spec
|
|
||||||
raise NotImplementedError
|
|
||||||
|
|
||||||
def realize(self, spec) -> bytes:
|
|
||||||
raise NotImplementedError
|
|
||||||
|
|
||||||
def project(self, doc: DocumentIR) -> bytes:
|
|
||||||
return self.realize(self.plan(doc))
|
|
||||||
|
|
||||||
|
|
||||||
_REGISTRY: dict[str, SurfaceProjector] = {}
|
|
||||||
|
|
||||||
|
|
||||||
def register(projector: SurfaceProjector) -> SurfaceProjector:
|
|
||||||
_REGISTRY[projector.surface] = projector
|
|
||||||
return projector
|
|
||||||
|
|
||||||
|
|
||||||
def get_projector(surface: str) -> SurfaceProjector:
|
|
||||||
if surface not in _REGISTRY:
|
|
||||||
raise KeyError(f"no projector registered for surface {surface!r}; "
|
|
||||||
f"have {sorted(_REGISTRY)}")
|
|
||||||
return _REGISTRY[surface]
|
|
||||||
|
|
||||||
|
|
||||||
def available_surfaces() -> list[str]:
|
|
||||||
return sorted(_REGISTRY)
|
|
||||||
@@ -1,113 +0,0 @@
|
|||||||
"""docx.py — the .docx surface projector: an OWN minimal OOXML emitter.
|
|
||||||
|
|
||||||
Own-the-core: a .docx is just a ZIP of a few XML parts (WordprocessingML). We
|
|
||||||
emit it with the standard library only — ``zipfile`` + string XML — no
|
|
||||||
python-docx, no external dependency. This proves a "richer structured format"
|
|
||||||
surface without importing anyone else's toolkit.
|
|
||||||
|
|
||||||
Parts emitted (the minimal valid set + a styles part for real headings):
|
|
||||||
[Content_Types].xml
|
|
||||||
_rels/.rels
|
|
||||||
word/_rels/document.xml.rels
|
|
||||||
word/styles.xml (Title / Heading1 / Heading2 / Normal)
|
|
||||||
word/document.xml (the content)
|
|
||||||
|
|
||||||
Like the markdown projector it reads only the IR's realized sentences; it
|
|
||||||
invents nothing. The surface differs, the faithful content does not.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import io
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import zipfile
|
|
||||||
from xml.sax.saxutils import escape
|
|
||||||
|
|
||||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
|
||||||
from document_ir import DocumentIR # noqa: E402
|
|
||||||
from projectors.base import register # noqa: E402
|
|
||||||
|
|
||||||
_CONTENT_TYPES = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
|
||||||
<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">
|
|
||||||
<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>
|
|
||||||
<Default Extension="xml" ContentType="application/xml"/>
|
|
||||||
<Override PartName="/word/document.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/>
|
|
||||||
<Override PartName="/word/styles.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.styles+xml"/>
|
|
||||||
</Types>"""
|
|
||||||
|
|
||||||
_RELS = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
|
||||||
<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
|
|
||||||
<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="word/document.xml"/>
|
|
||||||
</Relationships>"""
|
|
||||||
|
|
||||||
_DOC_RELS = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
|
||||||
<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
|
|
||||||
<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/styles" Target="styles.xml"/>
|
|
||||||
</Relationships>"""
|
|
||||||
|
|
||||||
_W = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
|
|
||||||
|
|
||||||
_STYLES = f"""<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
|
||||||
<w:styles xmlns:w="{_W}">
|
|
||||||
<w:style w:type="paragraph" w:default="1" w:styleId="Normal"><w:name w:val="Normal"/>
|
|
||||||
<w:rPr><w:sz w:val="22"/></w:rPr></w:style>
|
|
||||||
<w:style w:type="paragraph" w:styleId="Title"><w:name w:val="Title"/>
|
|
||||||
<w:pPr><w:spacing w:after="240"/></w:pPr>
|
|
||||||
<w:rPr><w:b/><w:sz w:val="52"/></w:rPr></w:style>
|
|
||||||
<w:style w:type="paragraph" w:styleId="Subtitle"><w:name w:val="Subtitle"/>
|
|
||||||
<w:rPr><w:i/><w:sz w:val="28"/><w:color w:val="555555"/></w:rPr></w:style>
|
|
||||||
<w:style w:type="paragraph" w:styleId="Heading1"><w:name w:val="heading 1"/>
|
|
||||||
<w:pPr><w:spacing w:before="240" w:after="120"/><w:outlineLvl w:val="0"/></w:pPr>
|
|
||||||
<w:rPr><w:b/><w:sz w:val="34"/></w:rPr></w:style>
|
|
||||||
<w:style w:type="paragraph" w:styleId="Heading2"><w:name w:val="heading 2"/>
|
|
||||||
<w:pPr><w:spacing w:before="200" w:after="100"/><w:outlineLvl w:val="1"/></w:pPr>
|
|
||||||
<w:rPr><w:b/><w:sz w:val="28"/></w:rPr></w:style>
|
|
||||||
</w:styles>"""
|
|
||||||
|
|
||||||
|
|
||||||
def _para(text: str, style: str | None = None) -> str:
|
|
||||||
ppr = f"<w:pPr><w:pStyle w:val=\"{style}\"/></w:pPr>" if style else ""
|
|
||||||
return (f"<w:p>{ppr}<w:r><w:t xml:space=\"preserve\">"
|
|
||||||
f"{escape(text)}</w:t></w:r></w:p>")
|
|
||||||
|
|
||||||
|
|
||||||
class DocxProjector:
|
|
||||||
surface = "docx"
|
|
||||||
media_type = ("application/vnd.openxmlformats-officedocument."
|
|
||||||
"wordprocessingml.document")
|
|
||||||
ext = "docx"
|
|
||||||
modality = "text"
|
|
||||||
|
|
||||||
def _document_xml(self, doc: DocumentIR) -> str:
|
|
||||||
body: list[str] = [_para(doc.title, "Title")]
|
|
||||||
if doc.subtitle:
|
|
||||||
body.append(_para(doc.subtitle, "Subtitle"))
|
|
||||||
abstract = doc.meta.get("abstract")
|
|
||||||
if abstract is not None and abstract.sentences:
|
|
||||||
body.append(_para(abstract.text()))
|
|
||||||
for sec in doc.sections:
|
|
||||||
style = "Heading1" if sec.level <= 1 else "Heading2"
|
|
||||||
body.append(_para(sec.heading, style))
|
|
||||||
for block in sec.blocks:
|
|
||||||
t = block.text()
|
|
||||||
if t:
|
|
||||||
body.append(_para(t))
|
|
||||||
return (f"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>"
|
|
||||||
f"<w:document xmlns:w=\"{_W}\"><w:body>"
|
|
||||||
+ "".join(body)
|
|
||||||
+ "<w:sectPr><w:pgSz w:w=\"12240\" w:h=\"15840\"/>"
|
|
||||||
"<w:pgMar w:top=\"1440\" w:right=\"1440\" w:bottom=\"1440\" "
|
|
||||||
"w:left=\"1440\"/></w:sectPr></w:body></w:document>")
|
|
||||||
|
|
||||||
def project(self, doc: DocumentIR) -> bytes:
|
|
||||||
buf = io.BytesIO()
|
|
||||||
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as z:
|
|
||||||
z.writestr("[Content_Types].xml", _CONTENT_TYPES)
|
|
||||||
z.writestr("_rels/.rels", _RELS)
|
|
||||||
z.writestr("word/_rels/document.xml.rels", _DOC_RELS)
|
|
||||||
z.writestr("word/styles.xml", _STYLES)
|
|
||||||
z.writestr("word/document.xml", self._document_xml(doc))
|
|
||||||
return buf.getvalue()
|
|
||||||
|
|
||||||
|
|
||||||
register(DocxProjector())
|
|
||||||
@@ -1,45 +0,0 @@
|
|||||||
"""markdown.py — the Markdown surface projector (text facet).
|
|
||||||
|
|
||||||
The most tractable surface, and the reference implementation: reads the IR's
|
|
||||||
realized sentences and lays them out as Markdown. Introduces no content — it is
|
|
||||||
pure typography over the faithful text the realizer produced.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
|
||||||
from document_ir import DocumentIR # noqa: E402
|
|
||||||
from projectors.base import register # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
class MarkdownProjector:
|
|
||||||
surface = "markdown"
|
|
||||||
media_type = "text/markdown"
|
|
||||||
ext = "md"
|
|
||||||
modality = "text"
|
|
||||||
|
|
||||||
def render_str(self, doc: DocumentIR) -> str:
|
|
||||||
lines: list[str] = [f"# {doc.title}"]
|
|
||||||
if doc.subtitle:
|
|
||||||
lines.append(f"\n*{doc.subtitle}*")
|
|
||||||
abstract = doc.meta.get("abstract")
|
|
||||||
if abstract is not None and abstract.sentences:
|
|
||||||
lines.append("")
|
|
||||||
lines.append(abstract.text())
|
|
||||||
for sec in doc.sections:
|
|
||||||
lines.append("")
|
|
||||||
lines.append(f"{'#' * max(2, sec.level)} {sec.heading}")
|
|
||||||
for block in sec.blocks:
|
|
||||||
body = block.text()
|
|
||||||
if body:
|
|
||||||
lines.append("")
|
|
||||||
lines.append(body)
|
|
||||||
return "\n".join(lines) + "\n"
|
|
||||||
|
|
||||||
def project(self, doc: DocumentIR) -> bytes:
|
|
||||||
return self.render_str(doc).encode("utf-8")
|
|
||||||
|
|
||||||
|
|
||||||
register(MarkdownProjector())
|
|
||||||
@@ -1,133 +0,0 @@
|
|||||||
"""midi.py — the MUSIC surface projector: geometry -> symbolic music (MIDI).
|
|
||||||
|
|
||||||
The first NON-TEXT surface, and the proof of the general shape. "Music is
|
|
||||||
language and it is math" (Will): symbolic music is tractable and geometry-native,
|
|
||||||
so it is the natural efferent twin to try first after text.
|
|
||||||
|
|
||||||
CRUCIALLY this projector does NOT read the realized sentences. It reads the IR's
|
|
||||||
GEOMETRY facet — ``block.provenance`` — and DECODES each edge onto a musical
|
|
||||||
surface. That is the whole thesis of the multimodal projector: the same
|
|
||||||
geometry-carrying IR drives text AND music; a text projector reads the words, a
|
|
||||||
music projector reads the meaning-geometry. The mapping is deterministic and
|
|
||||||
faithful to the geometry's structure:
|
|
||||||
|
|
||||||
relation lemma -> scale degree (same relation -> same pitch class;
|
|
||||||
meaning has a consistent sonic form)
|
|
||||||
polarity -> mode (aff = major third above; neg = minor
|
|
||||||
third / lowered — SACRED polarity is
|
|
||||||
audible, a negated edge sounds negated)
|
|
||||||
confidence -> note duration (stronger grounding rings longer)
|
|
||||||
importance -> velocity (more important source = louder)
|
|
||||||
section -> phrase + register shift (structure becomes musical form)
|
|
||||||
|
|
||||||
Own-the-core: a Standard MIDI File is a header chunk + a track chunk of
|
|
||||||
delta-timed events. We emit the raw bytes with ``struct`` — no external MIDI
|
|
||||||
library. Format 0, one track.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import io
|
|
||||||
import os
|
|
||||||
import struct
|
|
||||||
import sys
|
|
||||||
|
|
||||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
|
||||||
from document_ir import DocumentIR, Provenance # noqa: E402
|
|
||||||
from projectors.base import TwoStageProjector, register # noqa: E402
|
|
||||||
|
|
||||||
_TICKS = 480 # ticks per quarter note
|
|
||||||
_C_MAJOR = [0, 2, 4, 5, 7, 9, 11] # semitone offsets of a diatonic scale
|
|
||||||
|
|
||||||
|
|
||||||
def _vlq(n: int) -> bytes:
|
|
||||||
"""MIDI variable-length quantity encoding of a delta time."""
|
|
||||||
if n == 0:
|
|
||||||
return b"\x00"
|
|
||||||
out = bytearray()
|
|
||||||
out.append(n & 0x7F)
|
|
||||||
n >>= 7
|
|
||||||
while n:
|
|
||||||
out.insert(0, (n & 0x7F) | 0x80)
|
|
||||||
n >>= 7
|
|
||||||
return bytes(out)
|
|
||||||
|
|
||||||
|
|
||||||
def _degree_for(relation: str) -> int:
|
|
||||||
"""Stable scale degree for a relation lemma (same relation -> same pitch)."""
|
|
||||||
if not relation:
|
|
||||||
return 0
|
|
||||||
return sum(ord(c) for c in relation.lower()) % len(_C_MAJOR)
|
|
||||||
|
|
||||||
|
|
||||||
def _note_for(p: Provenance, base: int) -> tuple[int, int, int]:
|
|
||||||
"""(pitch, velocity, duration_ticks) for one geometry edge."""
|
|
||||||
root = base + _C_MAJOR[_degree_for(p.relation)]
|
|
||||||
# polarity -> mode: affirmed edges take the bright major third, negated edges
|
|
||||||
# take the darker minor third. The negation is AUDIBLE and never dropped.
|
|
||||||
third = 4 if p.polarity == "aff" else 3
|
|
||||||
pitch = max(24, min(96, root + (third if p.confidence >= 0.5 else 0)))
|
|
||||||
velocity = int(56 + 60 * min(1.0, max(0.0, p.importance)))
|
|
||||||
velocity = max(40, min(120, velocity))
|
|
||||||
# confidence -> duration: quarter .. dotted-half
|
|
||||||
dur = int(_TICKS * (0.5 + 1.5 * min(1.0, max(0.0, p.confidence))))
|
|
||||||
return pitch, velocity, dur
|
|
||||||
|
|
||||||
|
|
||||||
# a mode-profile: the pluggable musical knob (the peer's mode_profile). Scale +
|
|
||||||
# tempo. Swapping this profile re-voices the SAME geometry — surface as parameter.
|
|
||||||
_DEFAULT_PROFILE = {"scale": _C_MAJOR, "tempo_us": 500000,
|
|
||||||
"registers": [60, 55, 64, 50, 67, 48], "program": 0}
|
|
||||||
|
|
||||||
|
|
||||||
class MidiProjector(TwoStageProjector):
|
|
||||||
"""geometry -> symbolic music, in the shared two-stage shape.
|
|
||||||
|
|
||||||
``plan(frame)`` -> a music_spec: an ordered list of note dicts derived
|
|
||||||
deterministically from the frame's provenance geometry
|
|
||||||
(the peer's ``plan(frame, profile) -> spec``).
|
|
||||||
``realize(spec)`` -> Standard MIDI File bytes (the peer's
|
|
||||||
``realize(spec, profile) -> surface``; here the surface
|
|
||||||
is symbolic MIDI, the minimal audio proof — a richer
|
|
||||||
additive-synth audio projector conforms identically).
|
|
||||||
"""
|
|
||||||
|
|
||||||
surface = "midi"
|
|
||||||
media_type = "audio/midi"
|
|
||||||
ext = "mid"
|
|
||||||
modality = "audio"
|
|
||||||
|
|
||||||
def __init__(self, profile: dict | None = None):
|
|
||||||
self.profile = profile or _DEFAULT_PROFILE
|
|
||||||
|
|
||||||
# -- stage 1: meaning-geometry -> music_spec (reads the GEOMETRY facet) -- #
|
|
||||||
def plan(self, doc: DocumentIR) -> list[dict]:
|
|
||||||
registers = self.profile["registers"]
|
|
||||||
spec: list[dict] = []
|
|
||||||
for si, sec in enumerate(doc.sections):
|
|
||||||
base = registers[si % len(registers)]
|
|
||||||
provs = [p for p in sec.all_provenance()
|
|
||||||
if p.kind in ("fact", "interpretation")]
|
|
||||||
for i, p in enumerate(provs):
|
|
||||||
pitch, vel, dur = _note_for(p, base)
|
|
||||||
spec.append({"pitch": pitch, "velocity": vel, "dur": dur,
|
|
||||||
"rest_before": (_TICKS // 2) if (si > 0 and i == 0) else 0,
|
|
||||||
"relation": p.relation, "polarity": p.polarity})
|
|
||||||
return spec
|
|
||||||
|
|
||||||
# -- stage 2: music_spec -> MIDI bytes (own-core, no library) ------------ #
|
|
||||||
def realize(self, spec: list[dict]) -> bytes:
|
|
||||||
ev = bytearray()
|
|
||||||
ev += _vlq(0) + b"\xFF\x51\x03" + struct.pack(">I", self.profile["tempo_us"])[1:]
|
|
||||||
ev += _vlq(0) + bytes([0xC0, self.profile["program"] & 0x7F])
|
|
||||||
for note in spec:
|
|
||||||
ev += _vlq(note["rest_before"]) + bytes([0x90, note["pitch"], note["velocity"]])
|
|
||||||
ev += _vlq(note["dur"]) + bytes([0x80, note["pitch"], 0])
|
|
||||||
ev += _vlq(0) + b"\xFF\x2F\x00"
|
|
||||||
track = bytes(ev)
|
|
||||||
buf = io.BytesIO()
|
|
||||||
buf.write(b"MThd" + struct.pack(">IHHH", 6, 0, 1, _TICKS))
|
|
||||||
buf.write(b"MTrk" + struct.pack(">I", len(track)) + track)
|
|
||||||
return buf.getvalue()
|
|
||||||
|
|
||||||
|
|
||||||
register(MidiProjector())
|
|
||||||
@@ -1,60 +0,0 @@
|
|||||||
"""seams.py — documented efferent seams for IMAGE and VIDEO surfaces.
|
|
||||||
|
|
||||||
These are NOT implemented (per the build rails: architect, do not overbuild).
|
|
||||||
They are registered as first-class seams so the interface PROVES it accepts
|
|
||||||
future non-text projectors without any upstream change. Each documents exactly
|
|
||||||
what its decoder would read from the geometry-carrying IR, making the multimodal
|
|
||||||
generalization concrete rather than hand-wavy.
|
|
||||||
|
|
||||||
The symmetry that guarantees these are possible, not moonshots: they are the
|
|
||||||
efferent twins of multimodal INGEST. If meaning can HOLD an image (ingest as
|
|
||||||
first-class geometry), meaning can PROJECT one back. Video = image x sound x
|
|
||||||
TIME, and the engram already stores time (chronoception). So video falls out of
|
|
||||||
an image projector + the music projector + the stored temporal ordering.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
|
||||||
from document_ir import DocumentIR # noqa: E402
|
|
||||||
from projectors.base import register # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
class _Seam:
|
|
||||||
"""A registered-but-unimplemented projector. Names its decoder contract."""
|
|
||||||
|
|
||||||
def project(self, doc: DocumentIR) -> bytes: # pragma: no cover - seam
|
|
||||||
raise NotImplementedError(
|
|
||||||
f"{self.surface!r} projector is a documented seam, not yet built. "
|
|
||||||
f"Decoder contract: {self.decoder_contract}")
|
|
||||||
|
|
||||||
|
|
||||||
class ImageProjector(_Seam):
|
|
||||||
surface = "image"
|
|
||||||
media_type = "image/png"
|
|
||||||
ext = "png"
|
|
||||||
modality = "image"
|
|
||||||
decoder_contract = (
|
|
||||||
"reads block.provenance as a spatial layout — nodes become regions, edges "
|
|
||||||
"become adjacencies; salience/importance drive size/contrast; polarity "
|
|
||||||
"drives figure/ground. The efferent twin of image ingest (a geometry->raster "
|
|
||||||
"decoder, learned or engineered), exactly mirroring the embedder that turned "
|
|
||||||
"the image INTO geometry.")
|
|
||||||
|
|
||||||
|
|
||||||
class VideoProjector(_Seam):
|
|
||||||
surface = "video"
|
|
||||||
media_type = "video/mp4"
|
|
||||||
ext = "mp4"
|
|
||||||
modality = "video"
|
|
||||||
decoder_contract = (
|
|
||||||
"image x sound x TIME. Composes the image projector (per-keyframe geometry "
|
|
||||||
"layout) with the midi/music projector (score) along the geometry's stored "
|
|
||||||
"temporal ordering (chronoception). Needs no new principle once image + music "
|
|
||||||
"exist — only a muxer.")
|
|
||||||
|
|
||||||
|
|
||||||
register(ImageProjector())
|
|
||||||
register(VideoProjector())
|
|
||||||
@@ -1,63 +0,0 @@
|
|||||||
"""provenance.py — the faithfulness audit + geometry->section trace.
|
|
||||||
|
|
||||||
A document projected from geometry is only worth anything if every claim traces
|
|
||||||
back. This module walks the DocumentIR and proves the discipline held:
|
|
||||||
|
|
||||||
* ZERO ungrounded claims (every fact/interpretation has a real node id),
|
|
||||||
* every emitted sentence maps to a geometry edge (or is a marked connective),
|
|
||||||
* SACRED polarity survived (negations are reported, never silently dropped),
|
|
||||||
* COHERE introduced no new geometry (connectives carry no claim).
|
|
||||||
|
|
||||||
It emits both a machine verdict and a human-readable geometry->section table.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from document_ir import DocumentIR
|
|
||||||
|
|
||||||
|
|
||||||
def audit(doc: DocumentIR) -> dict:
|
|
||||||
provs = doc.all_provenance()
|
|
||||||
facts = [p for p in provs if p.kind in ("fact", "interpretation")]
|
|
||||||
connectives = [p for p in provs if p.kind == "connective"]
|
|
||||||
ungrounded = [p for p in facts if not p.node_id]
|
|
||||||
negations = [p for p in facts if p.polarity == "neg"]
|
|
||||||
node_ids = sorted({p.node_id for p in facts if p.node_id})
|
|
||||||
return {
|
|
||||||
"claims": len(facts),
|
|
||||||
"connectives": len(connectives),
|
|
||||||
"ungrounded_claims": len(ungrounded),
|
|
||||||
"negations_preserved": len(negations),
|
|
||||||
"distinct_source_nodes": len(node_ids),
|
|
||||||
"faithful": len(ungrounded) == 0,
|
|
||||||
"source_nodes": node_ids,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def trace_table(doc: DocumentIR) -> str:
|
|
||||||
"""Human-readable geometry -> section -> claim provenance table."""
|
|
||||||
lines = ["# Provenance — every claim traces geometry", ""]
|
|
||||||
lines.append(f"**Document:** {doc.title}")
|
|
||||||
a = audit(doc)
|
|
||||||
lines.append(f"**Claims:** {a['claims']} · **Ungrounded:** "
|
|
||||||
f"{a['ungrounded_claims']} · **Negations preserved:** "
|
|
||||||
f"{a['negations_preserved']} · **Source nodes:** "
|
|
||||||
f"{a['distinct_source_nodes']} · **Faithful:** "
|
|
||||||
f"{'YES' if a['faithful'] else 'NO'}")
|
|
||||||
lines.append("")
|
|
||||||
for si, sec in enumerate(doc.sections, 1):
|
|
||||||
lines.append(f"## {si}. {sec.heading}")
|
|
||||||
lines.append(f"_seed nodes: {', '.join(i[:8] for i in sec.seed_ids)}_")
|
|
||||||
lines.append("")
|
|
||||||
lines.append("| # | realized claim | traces geometry edge |")
|
|
||||||
lines.append("|---|----------------|----------------------|")
|
|
||||||
n = 0
|
|
||||||
for block in sec.blocks:
|
|
||||||
for sent, prov in zip(block.sentences, block.provenance):
|
|
||||||
if prov.kind == "connective":
|
|
||||||
continue
|
|
||||||
n += 1
|
|
||||||
edge = prov.trace().replace("|", "\\|")
|
|
||||||
s = sent.replace("|", "\\|")
|
|
||||||
lines.append(f"| {n} | {s} | {edge} |")
|
|
||||||
lines.append("")
|
|
||||||
return "\n".join(lines) + "\n"
|
|
||||||
@@ -1,112 +0,0 @@
|
|||||||
"""realize.py — REALIZE stage: fill each planned section with faithful passages.
|
|
||||||
|
|
||||||
Scales the PROVEN realizer from a single assertion to a passage. For each
|
|
||||||
planned proposition we build a realizer-ready clause (the proven
|
|
||||||
``_prop_to_clause`` mapping) and run it through the proven engine
|
|
||||||
(``engine.realize``), which is a deterministic grammar with the SACRED negation
|
|
||||||
contract — it never invents. Each realized sentence is paired with a
|
|
||||||
:class:`Provenance` that pins it to the exact geometry edge it came from.
|
|
||||||
|
|
||||||
"Passage, not a list of sentences": within a section we lightly vary sentence
|
|
||||||
openings and group related claims, but we add NO content the geometry did not
|
|
||||||
assert. The only non-geometry words are function words the grammar already owns
|
|
||||||
(articles, "and", conjunction of same-subject claims). Document-level flow is
|
|
||||||
COHERE's job; this stage owns intra-section fluency + fidelity.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
_NT = os.path.expanduser("~/Desktop/neuron-talk")
|
|
||||||
_LR = os.path.expanduser("~/Desktop/lang-realizers")
|
|
||||||
for _p in (_NT, _LR):
|
|
||||||
if _p not in sys.path:
|
|
||||||
sys.path.insert(0, _p)
|
|
||||||
|
|
||||||
import engine # noqa: E402 (the proven no-LLM realizer)
|
|
||||||
from dialogue import _prop_to_clause # noqa: E402 (proven prop -> clause)
|
|
||||||
|
|
||||||
from document_ir import Block, DocumentIR, Provenance, Section # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
def _provenance_from(p, kind: str = "fact") -> Provenance:
|
|
||||||
return Provenance(
|
|
||||||
subj_id=p.source_node_id, subject=p.subject, relation=p.predicate,
|
|
||||||
obj=p.object, polarity=p.polarity, confidence=round(float(p.confidence), 3),
|
|
||||||
node_id=p.source_node_id, kind=kind,
|
|
||||||
importance=float(getattr(p, "node_importance", 0.0) or 0.0),
|
|
||||||
salience=0.0,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
import re as _re
|
|
||||||
|
|
||||||
# a well-formed declarative opens with a determiner, a proper noun, "I", or a
|
|
||||||
# capitalized head — not a mis-parsed object pronoun or a copula fragment.
|
|
||||||
_BAD_OPENERS = _re.compile(r"^(Me |It is I|There is|This is it|That is it)\b")
|
|
||||||
_VACUOUS = _re.compile(r"^\w+ (is|are|was|were) (it|no|nothing|empty|those|this|that)\.?$",
|
|
||||||
_re.I)
|
|
||||||
|
|
||||||
|
|
||||||
def _good_sentence(text: str) -> bool:
|
|
||||||
"""Fluency gate — drops degenerate realizations. NEVER loosens faithfulness;
|
|
||||||
it only refuses to SPEAK a claim whose surface came out malformed."""
|
|
||||||
words = text.rstrip(".").split()
|
|
||||||
if len(words) < 3:
|
|
||||||
return False
|
|
||||||
if _BAD_OPENERS.search(text):
|
|
||||||
return False
|
|
||||||
if _VACUOUS.match(text):
|
|
||||||
return False
|
|
||||||
# a sentence that is mostly one-letter/two-letter tokens is a parse artifact
|
|
||||||
short = sum(1 for w in words if len(w.strip(".,'")) <= 2)
|
|
||||||
if short > len(words) / 2:
|
|
||||||
return False
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def _realize_prop(p, lang: str = "en") -> tuple[str, Provenance] | None:
|
|
||||||
"""One proposition -> (faithful sentence, provenance) or None if it drops."""
|
|
||||||
clause = _prop_to_clause(p)
|
|
||||||
text = engine.realize(clause, lang)
|
|
||||||
if not text or not text.strip():
|
|
||||||
return None
|
|
||||||
text = text.strip()
|
|
||||||
if not text.endswith((".", "!", "?")):
|
|
||||||
text += "."
|
|
||||||
# capitalize first character (proper nouns / "I" already handled by grammar)
|
|
||||||
text = text[0].upper() + text[1:]
|
|
||||||
if not _good_sentence(text):
|
|
||||||
return None
|
|
||||||
return text, _provenance_from(p)
|
|
||||||
|
|
||||||
|
|
||||||
def realize_document(doc: DocumentIR, lang: str = "en") -> DocumentIR:
|
|
||||||
"""Fill every planned section's blocks with faithful, realized passages."""
|
|
||||||
for sec in doc.sections:
|
|
||||||
planned = sec.__dict__.get("_planned_props", [])
|
|
||||||
block = Block(role="body")
|
|
||||||
summary_bits: list[str] = []
|
|
||||||
for p in planned:
|
|
||||||
r = _realize_prop(p, lang)
|
|
||||||
if r is None:
|
|
||||||
continue
|
|
||||||
text, prov = r
|
|
||||||
block.sentences.append(text)
|
|
||||||
block.provenance.append(prov)
|
|
||||||
if len(summary_bits) < 1:
|
|
||||||
# a short grounded gloss for TOC / pptx bullets
|
|
||||||
obj = (prov.obj or "").strip().rstrip(".")
|
|
||||||
if obj:
|
|
||||||
summary_bits.append(obj)
|
|
||||||
if block.sentences:
|
|
||||||
sec.blocks.append(block)
|
|
||||||
sec.summary = summary_bits[0] if summary_bits else ""
|
|
||||||
# drop the transient planning payload; the IR is now self-contained
|
|
||||||
sec.__dict__.pop("_planned_props", None)
|
|
||||||
sec.__dict__.pop("_node", None)
|
|
||||||
|
|
||||||
# prune sections that realized to nothing
|
|
||||||
doc.sections = [s for s in doc.sections if s.blocks]
|
|
||||||
return doc
|
|
||||||
@@ -1,136 +0,0 @@
|
|||||||
// accent.el - A British-RP ACCENT as an INGESTED TRANSFORM-GEOMETRY, composed
|
|
||||||
// onto the voice (voice (+) accent, SEPARABLE). Reads elp/data/british-accent.psv
|
|
||||||
// into an accent MANIFOLD in the engram (override nodes + a shared accent hub),
|
|
||||||
// and the render reads the RP formant overrides + the non-rhotic rule back from
|
|
||||||
// that geometry. NO accent targets live in code — same discipline as the base
|
|
||||||
// phonetics. PROVENANCE NOTE: the RP Hz values are PROVISIONAL (reconstructed-
|
|
||||||
// from-knowledge approximations, cite Deterding1997 / Hawkins&Midgley2005 /
|
|
||||||
// Wells1982) pending transcription from the published tables — the PIPELINE is
|
|
||||||
// the deliverable; exact values are being source-verified separately.
|
|
||||||
|
|
||||||
fn ingest_accent(path: String) -> [String] {
|
|
||||||
let content: String = fs_read(path)
|
|
||||||
let lines: [String] = str_split(content, "\n")
|
|
||||||
let nl: Int = native_list_len(lines)
|
|
||||||
let amap: [String] = native_list_empty()
|
|
||||||
let hub: String = engram_node("accent british-rp prov=PROVISIONAL cite=Deterding1997-HawkinsMidgley2005-Wells1982", "Accent", 80)
|
|
||||||
let li: Int = 0
|
|
||||||
while li < nl {
|
|
||||||
let line: String = native_list_get(lines, li)
|
|
||||||
let ll: Int = str_len(line)
|
|
||||||
let skip: Int = 0
|
|
||||||
if ll < 3 {
|
|
||||||
skip = 1
|
|
||||||
}
|
|
||||||
if skip == 0 {
|
|
||||||
let first: Int = str_char_code(line, 0)
|
|
||||||
if first == 35 {
|
|
||||||
skip = 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if skip == 0 {
|
|
||||||
let f: [String] = str_split(line, "|")
|
|
||||||
let nf: Int = native_list_len(f)
|
|
||||||
if nf >= 6 {
|
|
||||||
let key: String = native_list_get(f, 0)
|
|
||||||
let f1: String = native_list_get(f, 1)
|
|
||||||
let f2: String = native_list_get(f, 2)
|
|
||||||
let f3: String = native_list_get(f, 3)
|
|
||||||
let kind: String = native_list_get(f, 4)
|
|
||||||
let set: String = native_list_get(f, 5)
|
|
||||||
let cont: String = "accent british-rp " + key + " f1=" + f1 + " f2=" + f2 + " f3=" + f3 + " kind=" + kind + " set=" + set + " prov=PROVISIONAL cite=Deterding1997-HawkinsMidgley2005-Wells1982"
|
|
||||||
let id: String = engram_node(cont, "AccentTarget", 80)
|
|
||||||
amap = native_list_append(amap, key)
|
|
||||||
amap = native_list_append(amap, cont)
|
|
||||||
engram_connect(id, hub, 80, "of_accent")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
li = li + 1
|
|
||||||
}
|
|
||||||
return amap
|
|
||||||
}
|
|
||||||
|
|
||||||
// RP formant override for a phoneme, read from the accent manifold. Returns
|
|
||||||
// [f1,f2,f3] for a vowel_override record, or an empty list if none / a rule.
|
|
||||||
fn accent_formants(amap: [String], code: String) -> [Int] {
|
|
||||||
let out: [Int] = native_list_empty()
|
|
||||||
let id: String = sp_map_get(amap, code)
|
|
||||||
if str_eq(id, "") {
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
let j: String = id
|
|
||||||
let isrule: Int = str_index_of(j, "drop_coda")
|
|
||||||
if isrule >= 0 {
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
let f1: Int = parse_uint_from(j, "f1=")
|
|
||||||
if f1 <= 0 {
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
let out = native_list_append(out, f1)
|
|
||||||
let out = native_list_append(out, parse_uint_from(j, "f2="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(j, "f3="))
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// Is this accent non-rhotic? (reads the R rule node from the manifold)
|
|
||||||
fn is_nonrhotic(amap: [String]) -> Int {
|
|
||||||
let id: String = sp_map_get(amap, "R")
|
|
||||||
if str_eq(id, "") {
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
let hit: Int = str_index_of(id, "drop_coda")
|
|
||||||
if hit >= 0 {
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
// Is this symbol a vowel? Membership in the vowel-set derived from the phonetics
|
|
||||||
// source's class column (phonological structure — the FORMANT NUMBERS still come
|
|
||||||
// from the organ manifold; this is only the categorical class for the rule).
|
|
||||||
fn is_vowel_sym(vset: [String], sym: String) -> Int {
|
|
||||||
let n: Int = native_list_len(vset)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
if str_eq(native_list_get(vset, i), sym) {
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
i = i + 1
|
|
||||||
}
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
// Non-rhotic transform: drop a post-vocalic CODA /R/ — an R whose next non-SIL
|
|
||||||
// phoneme is NOT a vowel (a consonant, or end of utterance). Keep INTERVOCALIC/
|
|
||||||
// onset R (next non-SIL phoneme is a vowel, e.g. the medial R in N UW R AA N).
|
|
||||||
fn apply_rhoticity(codes: [String], vset: [String]) -> [String] {
|
|
||||||
let n: Int = native_list_len(codes)
|
|
||||||
let out: [String] = native_list_empty()
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let c: String = native_list_get(codes, i)
|
|
||||||
let keep: Int = 1
|
|
||||||
if str_eq(c, "R") {
|
|
||||||
let jx: Int = i + 1
|
|
||||||
let nextv: Int = 0
|
|
||||||
while jx < n {
|
|
||||||
let ncode: String = native_list_get(codes, jx)
|
|
||||||
if str_eq(ncode, "SIL") {
|
|
||||||
jx = jx + 1
|
|
||||||
} else {
|
|
||||||
nextv = is_vowel_sym(vset, ncode)
|
|
||||||
jx = n + 1000
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if nextv == 0 {
|
|
||||||
keep = 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if keep == 1 {
|
|
||||||
out = native_list_append(out, c)
|
|
||||||
}
|
|
||||||
i = i + 1
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
@@ -1,73 +0,0 @@
|
|||||||
// audio-demo.el - Drive the native audio surface: render a tone per instrument
|
|
||||||
// from its LEARNED signature, then render a small meaning-phrase "piece".
|
|
||||||
// Entry point: top-level statement calls main() (same convention as the
|
|
||||||
// examples' top-level println(run_test())).
|
|
||||||
|
|
||||||
fn micros_to_str(xs: [Int]) -> String {
|
|
||||||
let n: Int = native_list_len(xs)
|
|
||||||
let out: String = ""
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
if i > 0 { let out: String = out + "," }
|
|
||||||
let out: String = out + int_to_str(native_list_get(xs, i))
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// Render a 1.0s A4 (midi 69) tone from a signature file, print the parsed
|
|
||||||
// partials (proving the numbers came from the engram .sig), write the WAV.
|
|
||||||
fn render_tone(name: String, sigpath: String, outpath: String, table: [Int]) -> Int {
|
|
||||||
let lines: [String] = sig_load(sigpath)
|
|
||||||
let partials: [Int] = parse_micros(sig_field(lines, "partials"))
|
|
||||||
println("[" + name + "] partials_n=" + sig_field(lines, "partials_n") + " parsed_partials_micro(scale 1e6)=" + micros_to_str(partials))
|
|
||||||
println("[" + name + "] raw partials line from .sig = " + sig_field(lines, "partials"))
|
|
||||||
let freq: Int = freq_of_midi(69)
|
|
||||||
let note: [Int] = synth_from_sig(lines, freq, 1000, 900, 44100, table)
|
|
||||||
let n: Int = native_list_len(note)
|
|
||||||
let ok: Int = wav_write(outpath, note, n, 44100)
|
|
||||||
println("[" + name + "] rendered " + int_to_str(n) + " samples -> " + outpath + " (write_ok=" + int_to_str(ok) + ")")
|
|
||||||
return n
|
|
||||||
}
|
|
||||||
|
|
||||||
fn run_demo() -> Int {
|
|
||||||
let table: [Int] = sin_table()
|
|
||||||
fs_mkdir("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out")
|
|
||||||
|
|
||||||
println("=== TONES: render A4 (midi 69) from each learned signature ===")
|
|
||||||
render_tone("flute", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/flute.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-flute.wav", table)
|
|
||||||
render_tone("clarinet", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/clarinet.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-clarinet.wav", table)
|
|
||||||
render_tone("violin", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/violin.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-violin.wav", table)
|
|
||||||
render_tone("piano", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/piano.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-piano.wav", table)
|
|
||||||
render_tone("organ", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/organ.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-organ.wav", table)
|
|
||||||
|
|
||||||
println("")
|
|
||||||
println("=== PIECE: a 6-frame meaning phrase (incl. a NEG frame) ===")
|
|
||||||
let frames: [[String]] = native_list_empty()
|
|
||||||
let frames: [[String]] = native_list_append(frames, audio_frame("agent", "aff", "0.9", "0.8", "0", "s1"))
|
|
||||||
let frames: [[String]] = native_list_append(frames, audio_frame("theme", "aff", "0.7", "0.6", "0", "s2"))
|
|
||||||
let frames: [[String]] = native_list_append(frames, audio_frame("cause", "aff", "0.8", "0.9", "1", "s3"))
|
|
||||||
let frames: [[String]] = native_list_append(frames, audio_frame("negation", "neg", "0.85", "0.7", "0", "s4"))
|
|
||||||
let frames: [[String]] = native_list_append(frames, audio_frame("goal", "aff", "0.6", "0.5", "1", "s5"))
|
|
||||||
let frames: [[String]] = native_list_append(frames, audio_frame("result", "aff", "0.95", "1.0", "0", "s6"))
|
|
||||||
|
|
||||||
// Print the plan so the NEG frame's minor third (+3) vs major (+4) is visible.
|
|
||||||
let nf: Int = native_list_len(frames)
|
|
||||||
let fi: Int = 0
|
|
||||||
while fi < nf {
|
|
||||||
let frame: [String] = native_list_get(frames, fi)
|
|
||||||
let plan: [Int] = plan_note(frame)
|
|
||||||
let pol: String = surface_get(frame, "polarity")
|
|
||||||
let third_name: String = "major(+4)"
|
|
||||||
if str_eq(pol, "neg") { let third_name: String = "MINOR(+3)" }
|
|
||||||
println("frame " + int_to_str(fi) + " relation=" + surface_get(frame, "relation") + " polarity=" + pol + " -> midi=" + int_to_str(native_list_get(plan, 0)) + " dur_ms=" + int_to_str(native_list_get(plan, 1)) + " amp_pm=" + int_to_str(native_list_get(plan, 2)) + " third=" + third_name)
|
|
||||||
let fi: Int = fi + 1
|
|
||||||
}
|
|
||||||
|
|
||||||
let piano_lines: [String] = sig_load("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/piano.sig")
|
|
||||||
let total: Int = realize_audio(frames, piano_lines, "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/piece.wav", 44100, table)
|
|
||||||
println("PIECE rendered " + int_to_str(total) + " samples -> /Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/piece.wav")
|
|
||||||
return total
|
|
||||||
}
|
|
||||||
|
|
||||||
println("audio-demo main returned samples=" + int_to_str(run_demo()))
|
|
||||||
@@ -1,400 +0,0 @@
|
|||||||
// audio-surface.el - Native own-core additive-synthesis audio surface.
|
|
||||||
//
|
|
||||||
// The AUDIO efferent seam, native, no Python and no library. This renders real
|
|
||||||
// PCM .wav bytes from instrument SIGNATURES read from engram-sourced .sig data
|
|
||||||
// files (elp/faculty/sig/*.sig) - the partial amplitudes are NEVER literals in
|
|
||||||
// this source; they are parsed from the learned signature at run time. That is
|
|
||||||
// the whole proof: render-from-learned-signatures.
|
|
||||||
//
|
|
||||||
// EL has no float arithmetic operator (codegen emits raw int64 ops for + - * /
|
|
||||||
// on the shared 64-bit slot) and no float-arithmetic natives - so ALL synthesis
|
|
||||||
// math here is own-core INTEGER fixed-point. Angles use a quarter-wave sine
|
|
||||||
// table (scale 10000) from a fixed-point Taylor series; amplitudes are parsed to
|
|
||||||
// micro (scale 1e6) straight from the .sig text; frequencies are milliHz ints.
|
|
||||||
//
|
|
||||||
// Pipeline mirrors the two-stage projector (midi.py): plan_note(frame) reads a
|
|
||||||
// frame's meaning-geometry slot-map and derives (pitch, duration, amplitude);
|
|
||||||
// realize_audio SUPERPOSES the signature's partials (the compose op) and
|
|
||||||
// serialises RIFF/WAVE. Same frame -> midi OR audio.
|
|
||||||
|
|
||||||
// -- integer decimal + string helpers -----------------------------------------
|
|
||||||
|
|
||||||
fn str_to_int_el(s: String) -> Int {
|
|
||||||
let n: Int = str_len(s)
|
|
||||||
let i: Int = 0
|
|
||||||
let v: Int = 0
|
|
||||||
let neg: Bool = false
|
|
||||||
while i < n {
|
|
||||||
let c: Int = str_char_code(s, i)
|
|
||||||
if c == 45 { let neg: Bool = true }
|
|
||||||
if c >= 48 {
|
|
||||||
if c < 58 {
|
|
||||||
let v: Int = v * 10 + (c - 48)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
if neg { return 0 - v }
|
|
||||||
return v
|
|
||||||
}
|
|
||||||
|
|
||||||
fn parse_micro(s: String) -> Int {
|
|
||||||
let dot: Int = str_index_of(s, ".")
|
|
||||||
if dot < 0 {
|
|
||||||
return str_to_int_el(s) * 1000000
|
|
||||||
}
|
|
||||||
let n: Int = str_len(s)
|
|
||||||
let ipart: String = str_slice(s, 0, dot)
|
|
||||||
let fpart: String = str_slice(s, dot + 1, n)
|
|
||||||
let iv: Int = str_to_int_el(ipart)
|
|
||||||
let fv: Int = 0
|
|
||||||
let scale: Int = 100000
|
|
||||||
let fn2: Int = str_len(fpart)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < 6 {
|
|
||||||
let d: Int = 0
|
|
||||||
if i < fn2 {
|
|
||||||
let d: Int = str_char_code(fpart, i) - 48
|
|
||||||
}
|
|
||||||
let fv: Int = fv + d * scale
|
|
||||||
let scale: Int = scale / 10
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return iv * 1000000 + fv
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- signature (engram data file) loader ---------------------------------------
|
|
||||||
|
|
||||||
fn sig_load(path: String) -> [String] {
|
|
||||||
let text: String = fs_read(path)
|
|
||||||
return str_split(text, "\n")
|
|
||||||
}
|
|
||||||
|
|
||||||
fn sig_field(lines: [String], key: String) -> String {
|
|
||||||
let pref: String = key + ": "
|
|
||||||
let n: Int = native_list_len(lines)
|
|
||||||
let plen: Int = str_len(pref)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let ln: String = native_list_get(lines, i)
|
|
||||||
if str_starts_with(ln, pref) {
|
|
||||||
return str_slice(ln, plen, str_len(ln))
|
|
||||||
}
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
fn parse_micros(csv: String) -> [Int] {
|
|
||||||
let parts: [String] = str_split(csv, ",")
|
|
||||||
let n: Int = native_list_len(parts)
|
|
||||||
let out: [Int] = native_list_empty()
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let out: [Int] = native_list_append(out, parse_micro(native_list_get(parts, i)))
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- fixed-point sine (own-core, quarter-wave Taylor table, scale 10000) --------
|
|
||||||
|
|
||||||
fn sin_table() -> [Int] {
|
|
||||||
let HP: Int = 1570796
|
|
||||||
let t: [Int] = native_list_empty()
|
|
||||||
let q: Int = 0
|
|
||||||
while q < 257 {
|
|
||||||
let x: Int = q * HP / 256
|
|
||||||
let x2: Int = x * x / 1000000
|
|
||||||
let x3: Int = x2 * x / 1000000
|
|
||||||
let x5: Int = x3 * x2 / 1000000
|
|
||||||
let x7: Int = x5 * x2 / 1000000
|
|
||||||
let x9: Int = x7 * x2 / 1000000
|
|
||||||
let s: Int = x - x3 / 6 + x5 / 120 - x7 / 5040 + x9 / 362880
|
|
||||||
let t: [Int] = native_list_append(t, s / 100)
|
|
||||||
let q: Int = q + 1
|
|
||||||
}
|
|
||||||
return t
|
|
||||||
}
|
|
||||||
|
|
||||||
fn sin_lookup(t: [Int], phase: Int) -> Int {
|
|
||||||
let p: Int = phase % 1024
|
|
||||||
if p < 0 { let p: Int = p + 1024 }
|
|
||||||
let quad: Int = p / 256
|
|
||||||
let r: Int = p % 256
|
|
||||||
if quad == 0 { return native_list_get(t, r) }
|
|
||||||
if quad == 1 { return native_list_get(t, 256 - r) }
|
|
||||||
if quad == 2 { return 0 - native_list_get(t, r) }
|
|
||||||
return 0 - native_list_get(t, 256 - r)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn isqrt_int(n: Int) -> Int {
|
|
||||||
if n <= 0 { return 0 }
|
|
||||||
let x: Int = n
|
|
||||||
let y: Int = (x + 1) / 2
|
|
||||||
while y < x {
|
|
||||||
let x: Int = y
|
|
||||||
let y: Int = (x + n / x) / 2
|
|
||||||
}
|
|
||||||
return x
|
|
||||||
}
|
|
||||||
|
|
||||||
// freq_of_midi: equal-tempered frequency in milliHz. 440000 mHz at midi 69.
|
|
||||||
fn freq_of_midi(m: Int) -> Int {
|
|
||||||
let f: Int = 440000
|
|
||||||
if m > 69 {
|
|
||||||
let k: Int = m - 69
|
|
||||||
let i: Int = 0
|
|
||||||
while i < k {
|
|
||||||
let f: Int = f * 1059463 / 1000000
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return f
|
|
||||||
}
|
|
||||||
if m < 69 {
|
|
||||||
let k: Int = 69 - m
|
|
||||||
let i: Int = 0
|
|
||||||
while i < k {
|
|
||||||
let f: Int = f * 1000000 / 1059463
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return f
|
|
||||||
}
|
|
||||||
return f
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- envelope (ADSR), scale 1000 -----------------------------------------------
|
|
||||||
|
|
||||||
fn adsr_env(i: Int, total: Int, atk_n: Int, dec_n: Int, sus_pm: Int, rel_n: Int) -> Int {
|
|
||||||
if i < atk_n {
|
|
||||||
if atk_n == 0 { return 1000 }
|
|
||||||
return 1000 * i / atk_n
|
|
||||||
}
|
|
||||||
if i < atk_n + dec_n {
|
|
||||||
if dec_n == 0 { return sus_pm }
|
|
||||||
return 1000 - (1000 - sus_pm) * (i - atk_n) / dec_n
|
|
||||||
}
|
|
||||||
let rel_start: Int = total - rel_n
|
|
||||||
if i < rel_start {
|
|
||||||
return sus_pm
|
|
||||||
}
|
|
||||||
if rel_n == 0 { return 0 }
|
|
||||||
let left: Int = total - i
|
|
||||||
return sus_pm * left / rel_n
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- note synthesis: SUPERPOSE the learned partials -> [Int] samples -----------
|
|
||||||
fn note_samples(freq_mHz: Int, dur_ms: Int, rate: Int, partials: [Int], sumP: Int, b_micro: Int, vib_rate: Int, vib_cents: Int, atk_ms: Int, dec_ms: Int, sus_pm: Int, rel_ms: Int, amp_pm: Int, table: [Int]) -> [Int] {
|
|
||||||
let total: Int = dur_ms * rate / 1000
|
|
||||||
let atk_n: Int = atk_ms * rate / 1000
|
|
||||||
let dec_n: Int = dec_ms * rate / 1000
|
|
||||||
let rel_n: Int = rel_ms * rate / 1000
|
|
||||||
let np: Int = native_list_len(partials)
|
|
||||||
let half_mhz: Int = rate * 1000 / 2
|
|
||||||
let out: [Int] = native_list_empty()
|
|
||||||
let i: Int = 0
|
|
||||||
while i < total {
|
|
||||||
let acc: Int = 0
|
|
||||||
let k: Int = 0
|
|
||||||
while k < np {
|
|
||||||
let harm: Int = k + 1
|
|
||||||
let amp_k: Int = native_list_get(partials, k)
|
|
||||||
let factor: Int = 1000000
|
|
||||||
if b_micro > 0 {
|
|
||||||
let val: Int = 1000000 + b_micro * harm * harm
|
|
||||||
let factor: Int = isqrt_int(val * 1000000)
|
|
||||||
}
|
|
||||||
let fn_mhz: Int = freq_mHz * harm
|
|
||||||
let fn_mhz: Int = fn_mhz * factor / 1000000
|
|
||||||
if vib_cents > 0 {
|
|
||||||
if vib_rate > 0 {
|
|
||||||
let vphase: Int = i * vib_rate * 1024 / rate
|
|
||||||
let vs: Int = sin_lookup(table, vphase)
|
|
||||||
let vibf: Int = 1000000 + (vib_cents * vs * 833) / 10000
|
|
||||||
let fn_mhz: Int = fn_mhz * vibf / 1000000
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if fn_mhz <= half_mhz {
|
|
||||||
let phase: Int = i * fn_mhz * 1024 / (rate * 1000)
|
|
||||||
let sv: Int = sin_lookup(table, phase)
|
|
||||||
let acc: Int = acc + sv * amp_k / 1000000
|
|
||||||
}
|
|
||||||
let k: Int = k + 1
|
|
||||||
}
|
|
||||||
let env: Int = adsr_env(i, total, atk_n, dec_n, sus_pm, rel_n)
|
|
||||||
let s16: Int = acc * 2800000 / sumP
|
|
||||||
let s16: Int = s16 * env / 1000
|
|
||||||
let s16: Int = s16 * amp_pm / 1000
|
|
||||||
if s16 > 32767 { let s16: Int = 32767 }
|
|
||||||
if s16 < 0 - 32767 { let s16: Int = 0 - 32767 }
|
|
||||||
let out: [Int] = native_list_append(out, s16)
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
fn synth_from_sig(lines: [String], freq_mHz: Int, dur_ms: Int, amp_pm: Int, rate: Int, table: [Int]) -> [Int] {
|
|
||||||
let partials: [Int] = parse_micros(sig_field(lines, "partials"))
|
|
||||||
let np: Int = native_list_len(partials)
|
|
||||||
let sumP: Int = 0
|
|
||||||
let j: Int = 0
|
|
||||||
while j < np {
|
|
||||||
let pj: Int = native_list_get(partials, j)
|
|
||||||
let sumP: Int = sumP + pj
|
|
||||||
let j: Int = j + 1
|
|
||||||
}
|
|
||||||
if sumP <= 0 { let sumP: Int = 1000000 }
|
|
||||||
let adsr: [String] = str_split(sig_field(lines, "adsr"), ",")
|
|
||||||
let atk_ms: Int = parse_micro(native_list_get(adsr, 0)) / 1000
|
|
||||||
let dec_ms: Int = parse_micro(native_list_get(adsr, 1)) / 1000
|
|
||||||
let sus_pm: Int = parse_micro(native_list_get(adsr, 2)) / 1000
|
|
||||||
let rel_ms: Int = parse_micro(native_list_get(adsr, 3)) / 1000
|
|
||||||
let b_micro: Int = parse_micro(sig_field(lines, "inharmonicity_B"))
|
|
||||||
let vib_rate: Int = str_to_int_el(sig_field(lines, "vibrato_rate_hz"))
|
|
||||||
let vib_cents: Int = str_to_int_el(sig_field(lines, "vibrato_depth_cents"))
|
|
||||||
return note_samples(freq_mHz, dur_ms, rate, partials, sumP, b_micro, vib_rate, vib_cents, atk_ms, dec_ms, sus_pm, rel_ms, amp_pm, table)
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- byte-buffer helpers (own-core, no library) --------------------------------
|
|
||||||
|
|
||||||
fn put_tag(buf: String, pos: Int, s: String) -> String {
|
|
||||||
let n: Int = str_len(s)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let buf: String = __str_set_char(buf, pos + i, str_char_code(s, i))
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return buf
|
|
||||||
}
|
|
||||||
|
|
||||||
fn put_u32le(buf: String, pos: Int, v: Int) -> String {
|
|
||||||
let buf: String = __str_set_char(buf, pos, v % 256)
|
|
||||||
let buf: String = __str_set_char(buf, pos + 1, (v / 256) % 256)
|
|
||||||
let buf: String = __str_set_char(buf, pos + 2, (v / 65536) % 256)
|
|
||||||
let buf: String = __str_set_char(buf, pos + 3, (v / 16777216) % 256)
|
|
||||||
return buf
|
|
||||||
}
|
|
||||||
|
|
||||||
fn put_u16le(buf: String, pos: Int, v: Int) -> String {
|
|
||||||
let buf: String = __str_set_char(buf, pos, v % 256)
|
|
||||||
let buf: String = __str_set_char(buf, pos + 1, (v / 256) % 256)
|
|
||||||
return buf
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- WAV serializer: own-core RIFF/WAVE, PCM mono 16-bit -----------------------
|
|
||||||
|
|
||||||
fn wav_write(path: String, samples: [Int], n: Int, rate: Int) -> Int {
|
|
||||||
let data_len: Int = n * 2
|
|
||||||
let total: Int = 44 + data_len
|
|
||||||
let buf: String = __str_alloc(total)
|
|
||||||
let buf: String = put_tag(buf, 0, "RIFF")
|
|
||||||
let buf: String = put_u32le(buf, 4, 36 + data_len)
|
|
||||||
let buf: String = put_tag(buf, 8, "WAVE")
|
|
||||||
let buf: String = put_tag(buf, 12, "fmt ")
|
|
||||||
let buf: String = put_u32le(buf, 16, 16)
|
|
||||||
let buf: String = put_u16le(buf, 20, 1)
|
|
||||||
let buf: String = put_u16le(buf, 22, 1)
|
|
||||||
let buf: String = put_u32le(buf, 24, rate)
|
|
||||||
let buf: String = put_u32le(buf, 28, rate * 2)
|
|
||||||
let buf: String = put_u16le(buf, 32, 2)
|
|
||||||
let buf: String = put_u16le(buf, 34, 16)
|
|
||||||
let buf: String = put_tag(buf, 36, "data")
|
|
||||||
let buf: String = put_u32le(buf, 40, data_len)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let v: Int = native_list_get(samples, i)
|
|
||||||
if v < 0 { let v: Int = v + 65536 }
|
|
||||||
let buf: String = __str_set_char(buf, 44 + i * 2, v % 256)
|
|
||||||
let buf: String = __str_set_char(buf, 44 + i * 2 + 1, (v / 256) % 256)
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
let ok: Int = fs_write_bytes(path, buf, total)
|
|
||||||
return ok
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- plan: frame slot-map -> note atom (pitch, duration, amplitude) ------------
|
|
||||||
|
|
||||||
fn audio_frame(relation: String, polarity: String, confidence: String, importance: String, salience: String, subj_id: String) -> [String] {
|
|
||||||
let f: [String] = native_list_empty()
|
|
||||||
let f: [String] = native_list_append(f, "relation")
|
|
||||||
let f: [String] = native_list_append(f, relation)
|
|
||||||
let f: [String] = native_list_append(f, "polarity")
|
|
||||||
let f: [String] = native_list_append(f, polarity)
|
|
||||||
let f: [String] = native_list_append(f, "confidence")
|
|
||||||
let f: [String] = native_list_append(f, confidence)
|
|
||||||
let f: [String] = native_list_append(f, "importance")
|
|
||||||
let f: [String] = native_list_append(f, importance)
|
|
||||||
let f: [String] = native_list_append(f, "salience")
|
|
||||||
let f: [String] = native_list_append(f, salience)
|
|
||||||
let f: [String] = native_list_append(f, "subj_id")
|
|
||||||
let f: [String] = native_list_append(f, subj_id)
|
|
||||||
return f
|
|
||||||
}
|
|
||||||
|
|
||||||
fn degree_offset(deg: Int) -> Int {
|
|
||||||
if deg == 0 { return 0 }
|
|
||||||
if deg == 1 { return 2 }
|
|
||||||
if deg == 2 { return 4 }
|
|
||||||
if deg == 3 { return 5 }
|
|
||||||
if deg == 4 { return 7 }
|
|
||||||
if deg == 5 { return 9 }
|
|
||||||
return 11
|
|
||||||
}
|
|
||||||
|
|
||||||
// returns [midi, dur_ms, amp_pm]
|
|
||||||
fn plan_note(frame: [String]) -> [Int] {
|
|
||||||
let relation: String = surface_get(frame, "relation")
|
|
||||||
let polarity: String = surface_get(frame, "polarity")
|
|
||||||
let confidence: String = surface_get(frame, "confidence")
|
|
||||||
let importance: String = surface_get(frame, "importance")
|
|
||||||
let salience: String = surface_get(frame, "salience")
|
|
||||||
let rn: Int = str_len(relation)
|
|
||||||
let csum: Int = 0
|
|
||||||
let i: Int = 0
|
|
||||||
while i < rn {
|
|
||||||
let cc: Int = str_char_code(relation, i)
|
|
||||||
let csum: Int = csum + cc
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
let deg: Int = csum % 7
|
|
||||||
let third: Int = 4
|
|
||||||
if str_eq(polarity, "neg") { let third: Int = 3 }
|
|
||||||
let sal_oct: Int = str_to_int_el(salience)
|
|
||||||
let doff: Int = degree_offset(deg)
|
|
||||||
let midi: Int = 60 + sal_oct * 12 + doff + third
|
|
||||||
let conf_micro: Int = parse_micro(confidence)
|
|
||||||
let dur_ms: Int = 200 + conf_micro / 1000
|
|
||||||
let imp_micro: Int = parse_micro(importance)
|
|
||||||
let amp_pm: Int = 400 + imp_micro / 2000
|
|
||||||
let out: [Int] = native_list_empty()
|
|
||||||
let out: [Int] = native_list_append(out, midi)
|
|
||||||
let out: [Int] = native_list_append(out, dur_ms)
|
|
||||||
let out: [Int] = native_list_append(out, amp_pm)
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
fn realize_audio(frames: [[String]], sig_lines: [String], path: String, rate: Int, table: [Int]) -> Int {
|
|
||||||
let nf: Int = native_list_len(frames)
|
|
||||||
let all: [Int] = native_list_empty()
|
|
||||||
let count: Int = 0
|
|
||||||
let fi: Int = 0
|
|
||||||
while fi < nf {
|
|
||||||
let frame: [String] = native_list_get(frames, fi)
|
|
||||||
let plan: [Int] = plan_note(frame)
|
|
||||||
let midi: Int = native_list_get(plan, 0)
|
|
||||||
let dur_ms: Int = native_list_get(plan, 1)
|
|
||||||
let amp_pm: Int = native_list_get(plan, 2)
|
|
||||||
let freq: Int = freq_of_midi(midi)
|
|
||||||
let note: [Int] = synth_from_sig(sig_lines, freq, dur_ms, amp_pm, rate, table)
|
|
||||||
let nn: Int = native_list_len(note)
|
|
||||||
let j: Int = 0
|
|
||||||
while j < nn {
|
|
||||||
let all: [Int] = native_list_append(all, native_list_get(note, j))
|
|
||||||
let j: Int = j + 1
|
|
||||||
}
|
|
||||||
let count: Int = count + nn
|
|
||||||
let fi: Int = fi + 1
|
|
||||||
}
|
|
||||||
let ok: Int = wav_write(path, all, count, rate)
|
|
||||||
return count
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,16 +0,0 @@
|
|||||||
// comprehend.elh — public surface of the ELP comprehension front-end.
|
|
||||||
// text → meaning-spec (the input half of the ELP; inverse of the realizer).
|
|
||||||
extern fn parse_spec(text: String) -> [String]
|
|
||||||
extern fn parse_spec_lang(text: String, lang: String) -> [String]
|
|
||||||
extern fn parse_json(text: String) -> String
|
|
||||||
extern fn parse_json_lang(text: String, lang: String) -> String
|
|
||||||
// Analysis primitives (invertible morphology + deterministic grammar helpers):
|
|
||||||
extern fn cp_tokenize(text: String) -> [String]
|
|
||||||
extern fn cp_pron_concept(w: String) -> String
|
|
||||||
extern fn cp_is_negation(w: String) -> Bool
|
|
||||||
extern fn cp_is_neg_adverb(w: String) -> Bool
|
|
||||||
extern fn cp_irr2(surface: String) -> [String]
|
|
||||||
extern fn cp_reg_verb(w: String) -> [String]
|
|
||||||
extern fn cp_analyze_verb(surface: String) -> [String]
|
|
||||||
extern fn cp_verb_start(toks: [String], end: Int) -> Int
|
|
||||||
extern fn cp_subord_start(toks: [String], n: Int) -> Int
|
|
||||||
@@ -1,287 +0,0 @@
|
|||||||
// dialogue.el — SUMMON-THROUGH-SELF, native el. Port of dialogue.py's core.
|
|
||||||
//
|
|
||||||
// THE WHOLE DIALOGUE IS ONE OPERATION. A fact is never merely *fetched*: the
|
|
||||||
// query is PROJECTED into the engram's self + memory geometry, LANDS in a region,
|
|
||||||
// and the reply is READ OUT / the region MATERIALIZED from wherever it landed.
|
|
||||||
//
|
|
||||||
// project(query) -> land on a region -> read out from that region
|
|
||||||
//
|
|
||||||
// • lands in the SELF region -> grounded identity/presence, read out of
|
|
||||||
// the real self nodes (self_region.el)
|
|
||||||
// • lands on a memory NEIGHBORHOOD -> MATERIALIZE it: walk the neighborhood
|
|
||||||
// (engram_neighbors_json) and read out the
|
|
||||||
// region's connected members
|
|
||||||
// • lands nowhere close -> HONEST ABSENCE (an empty region, not a
|
|
||||||
// fabricated answer, not an error)
|
|
||||||
//
|
|
||||||
// CRITICAL INVARIANTS (enforced structurally, not by convention):
|
|
||||||
// * ONE operation — there is NO intent classifier and NO separate
|
|
||||||
// fact-retrieval branch. Identity is nearest-region proximity, not a switch.
|
|
||||||
// * MATERIALIZE by walking the neighborhood, never by fetching top-props.
|
|
||||||
// * HONEST ABSENCE when the region is thin.
|
|
||||||
// * NEGATION is SACRED: the readout is the stored prose VERBATIM, so a negated
|
|
||||||
// memory stays negated — we never paraphrase a polarity away.
|
|
||||||
// * NO ECHO: the old "I noted that X. That relates to Y." template is gone.
|
|
||||||
// The summon path materializes or honestly declines — it never echoes.
|
|
||||||
// * DIRECTIVE OVERRIDE: a meta-directive ("answer in English") overrides the
|
|
||||||
// reply language while the content language is still auto-detected.
|
|
||||||
//
|
|
||||||
// Depends on: comprehend (parse_spec_lang, cp_tokenize), multilingual (ml_detect,
|
|
||||||
// ml_tr, ml_term), propositions (prop_split_sentences), self_region
|
|
||||||
// (sr_available, sr_readout), the engram + json runtime builtins.
|
|
||||||
|
|
||||||
// ── directive override ────────────────────────────────────────────────────────
|
|
||||||
// Return [target_lang, content]. target_lang is "" when no directive is present.
|
|
||||||
// A directive names an output language; we strip it and keep the remaining text
|
|
||||||
// as the content (whose OWN language is still auto-detected downstream).
|
|
||||||
|
|
||||||
fn dlg_dir_hit(low: String, phrase: String) -> Bool {
|
|
||||||
return str_contains(low, phrase)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn dlg_parse_directive(text: String) -> [String] {
|
|
||||||
let low: String = str_to_lower(text)
|
|
||||||
let lang: String = ""
|
|
||||||
let phrase: String = ""
|
|
||||||
// English target
|
|
||||||
if dlg_dir_hit(low, "in english") { let lang = "en"; let phrase = "in english" }
|
|
||||||
if dlg_dir_hit(low, "em inglês") { let lang = "en"; let phrase = "em inglês" }
|
|
||||||
if dlg_dir_hit(low, "em ingles") { let lang = "en"; let phrase = "em ingles" }
|
|
||||||
if dlg_dir_hit(low, "en inglés") { let lang = "en"; let phrase = "en inglés" }
|
|
||||||
// Portuguese target
|
|
||||||
if dlg_dir_hit(low, "in portuguese") { let lang = "pt"; let phrase = "in portuguese" }
|
|
||||||
if dlg_dir_hit(low, "em português") { let lang = "pt"; let phrase = "em português" }
|
|
||||||
// Spanish target
|
|
||||||
if dlg_dir_hit(low, "in spanish") { let lang = "es"; let phrase = "in spanish" }
|
|
||||||
if dlg_dir_hit(low, "en español") { let lang = "es"; let phrase = "en español" }
|
|
||||||
// Italian target
|
|
||||||
if dlg_dir_hit(low, "in italian") { let lang = "it"; let phrase = "in italian" }
|
|
||||||
|
|
||||||
let content: String = text
|
|
||||||
if !str_eq(phrase, "") {
|
|
||||||
// strip the directive phrase (and a common "answer"/"responda" lead-in),
|
|
||||||
// leaving the real question as content.
|
|
||||||
let idx: Int = str_index_of(low, phrase)
|
|
||||||
if idx >= 0 {
|
|
||||||
let before: String = str_slice(text, 0, idx)
|
|
||||||
let after: String = str_slice(text, idx + str_len(phrase), str_len(text))
|
|
||||||
let content = str_trim(before + " " + after)
|
|
||||||
}
|
|
||||||
// trim a leading "answer"/"responda"/"reply" and stray colon/comma.
|
|
||||||
let cl: String = str_to_lower(content)
|
|
||||||
if str_starts_with(cl, "answer") { let content = str_trim(str_slice(content, 6, str_len(content))) }
|
|
||||||
if str_starts_with(cl, "responda") { let content = str_trim(str_slice(content, 8, str_len(content))) }
|
|
||||||
if str_starts_with(cl, "reply") { let content = str_trim(str_slice(content, 5, str_len(content))) }
|
|
||||||
if str_starts_with(content, ":") { let content = str_trim(str_slice(content, 1, str_len(content))) }
|
|
||||||
if str_starts_with(content, ",") { let content = str_trim(str_slice(content, 1, str_len(content))) }
|
|
||||||
}
|
|
||||||
let r: [String] = native_list_empty()
|
|
||||||
let r = native_list_append(r, lang)
|
|
||||||
let r = native_list_append(r, content)
|
|
||||||
return r
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── identity landing (a region proximity, not a classifier switch) ────────────
|
|
||||||
// The query lands in the SELF region when it takes an identity/presence shape.
|
|
||||||
// Cross-lingual forms are included because the engram's lexical probe is
|
|
||||||
// English-leaning. This is the SELF attractor of the single operation.
|
|
||||||
|
|
||||||
fn dlg_is_identity(content: String) -> Bool {
|
|
||||||
let low: String = str_to_lower(str_trim(content))
|
|
||||||
if str_contains(low, "who are you") { return true }
|
|
||||||
if str_contains(low, "what are you") { return true }
|
|
||||||
if str_contains(low, "who i am") { return true }
|
|
||||||
if str_contains(low, "your name") { return true }
|
|
||||||
if str_contains(low, "about yourself") { return true }
|
|
||||||
if str_contains(low, "are you conscious") { return true }
|
|
||||||
if str_contains(low, "are you there") { return true }
|
|
||||||
// cross-lingual identity question-forms
|
|
||||||
if str_contains(low, "quem é você") { return true }
|
|
||||||
if str_contains(low, "quem es voce") { return true }
|
|
||||||
if str_contains(low, "quién eres") { return true }
|
|
||||||
if str_contains(low, "quien eres") { return true }
|
|
||||||
if str_contains(low, "chi sei") { return true }
|
|
||||||
if str_contains(low, "qui es-tu") { return true }
|
|
||||||
if str_contains(low, "wer bist du") { return true }
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── readout helpers ───────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
fn dlg_first_sentence(content: String) -> String {
|
|
||||||
let sents: [String] = prop_split_sentences(content)
|
|
||||||
let n: Int = native_list_len(sents)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let s: String = str_trim(native_list_get(sents, i))
|
|
||||||
// drop a leading markdown heading marker for a clean read-out line
|
|
||||||
if str_starts_with(s, "# ") { let s = str_trim(str_slice(s, 2, str_len(s))) }
|
|
||||||
if str_len(s) > 0 { return s }
|
|
||||||
let i = i + 1
|
|
||||||
}
|
|
||||||
return str_trim(content)
|
|
||||||
}
|
|
||||||
|
|
||||||
// strip trailing/leading punctuation from a token.
|
|
||||||
fn dlg_clean_tok(w: String) -> String {
|
|
||||||
let s: String = str_trim(w)
|
|
||||||
let s = str_strip_suffix(s, ".")
|
|
||||||
let s = str_strip_suffix(s, ",")
|
|
||||||
let s = str_strip_suffix(s, "?")
|
|
||||||
let s = str_strip_suffix(s, "!")
|
|
||||||
let s = str_strip_suffix(s, ":")
|
|
||||||
let s = str_strip_suffix(s, ";")
|
|
||||||
return str_trim(s)
|
|
||||||
}
|
|
||||||
|
|
||||||
// closed-class across the supported languages (union) — a word we must NOT treat
|
|
||||||
// as a retrieval topic. Also drops the meta verbs of a request ("tell", "prove",
|
|
||||||
// "show") so the TOPIC, not the speech act, is what projects into memory.
|
|
||||||
fn dlg_is_stop(w: String) -> Bool {
|
|
||||||
if ml_stop_en(w) { return true }
|
|
||||||
if ml_stop_es(w) { return true }
|
|
||||||
if ml_stop_pt(w) { return true }
|
|
||||||
if ml_stop_it(w) { return true }
|
|
||||||
if str_eq(w, "tell") { return true }
|
|
||||||
if str_eq(w, "show") { return true }
|
|
||||||
if str_eq(w, "about") { return true }
|
|
||||||
if str_eq(w, "sobre") { return true }
|
|
||||||
if str_eq(w, "acerca") { return true }
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
// The CONTENT TERMS the query projects into memory: content words only, cleaned,
|
|
||||||
// cross-lingually mapped to the engram's English vocabulary, ≥3 chars. This is
|
|
||||||
// the geometry probe — the speech-act verbs and function words are stripped so a
|
|
||||||
// PP topic ("tell me ABOUT Lisbon") projects on "lisbon", not "tell"/"me".
|
|
||||||
fn dlg_content_terms(content: String, lang: String) -> [String] {
|
|
||||||
let toks: [String] = cp_tokenize(content)
|
|
||||||
let n: Int = native_list_len(toks)
|
|
||||||
let out: [String] = native_list_empty()
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let w: String = str_to_lower(dlg_clean_tok(native_list_get(toks, i)))
|
|
||||||
if str_len(w) >= 3 {
|
|
||||||
if !dlg_is_stop(w) {
|
|
||||||
let out = native_list_append(out, ml_term(w, lang))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let i = i + 1
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// Does this landed node lexically overlap the query's content terms? This is the
|
|
||||||
// RELEVANCE FLOOR: activation always returns the store's most salient nodes, so
|
|
||||||
// without this a query about nothing would "land" on the self/top node. A node
|
|
||||||
// that shares no content term with the query is "nowhere close" -> honest absence.
|
|
||||||
fn dlg_node_matches(node: String, terms: [String]) -> Bool {
|
|
||||||
let hay: String = str_to_lower(json_get_string(node, "content") + " " + json_get_string(node, "label"))
|
|
||||||
let n: Int = native_list_len(terms)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let t: String = native_list_get(terms, i)
|
|
||||||
if str_len(t) >= 3 {
|
|
||||||
if str_contains(hay, t) { return true }
|
|
||||||
}
|
|
||||||
let i = i + 1
|
|
||||||
}
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
// MATERIALIZE the landed region: read out the landed fact, then WALK the
|
|
||||||
// neighborhood and read out its connected members (real edges, not top-props).
|
|
||||||
fn dlg_materialize(top_node: String, reply_lang: String) -> String {
|
|
||||||
let id: String = json_get_string(top_node, "id")
|
|
||||||
let content: String = json_get_string(top_node, "content")
|
|
||||||
let lead: String = dlg_first_sentence(content)
|
|
||||||
|
|
||||||
let nb: String = engram_neighbors_json(id, 2, "both")
|
|
||||||
let m: Int = json_array_len(nb)
|
|
||||||
let parts: [String] = native_list_empty()
|
|
||||||
let parts = native_list_append(parts, lead)
|
|
||||||
let added: Int = 0
|
|
||||||
let i: Int = 0
|
|
||||||
while i < m {
|
|
||||||
if added < 3 {
|
|
||||||
let rec: String = json_array_get(nb, i)
|
|
||||||
let node: String = json_get_raw(rec, "node")
|
|
||||||
let nc: String = json_get_string(node, "content")
|
|
||||||
if !str_eq(nc, "") {
|
|
||||||
let sent: String = dlg_first_sentence(nc)
|
|
||||||
if !str_eq(sent, "") {
|
|
||||||
let parts = native_list_append(parts, sent)
|
|
||||||
let added = added + 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let i = i + 1
|
|
||||||
}
|
|
||||||
// The readout is the region's OWN prose, verbatim — negation SACRED, no echo.
|
|
||||||
return str_join(parts, " ")
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── THE single operation ──────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
fn dlg_respond(text: String) -> String {
|
|
||||||
// directive override: reply language may differ from content language.
|
|
||||||
let dir: [String] = dlg_parse_directive(text)
|
|
||||||
let target_lang: String = native_list_get(dir, 0)
|
|
||||||
let content: String = native_list_get(dir, 1)
|
|
||||||
|
|
||||||
let content_lang: String = ml_detect(content)
|
|
||||||
let reply_lang: String = content_lang
|
|
||||||
if !str_eq(target_lang, "") { let reply_lang = target_lang }
|
|
||||||
|
|
||||||
// comprehend the content (SACRED polarity carried in the spec).
|
|
||||||
let spec: [String] = parse_spec_lang(content, content_lang)
|
|
||||||
|
|
||||||
// ── PROJECT + LAND: SELF region ───────────────────────────────────────────
|
|
||||||
// Identity/presence shape lands in the self region; read out the REAL self
|
|
||||||
// nodes (self_region.el), never a template. Same single operation — this is
|
|
||||||
// just the self attractor winning the landing.
|
|
||||||
if dlg_is_identity(content) {
|
|
||||||
if sr_available() {
|
|
||||||
// read out the REAL self nodes when replying in their own language
|
|
||||||
// (the soul's prose is English); for another reply language we cannot
|
|
||||||
// translate real content without an LLM, so we answer with the
|
|
||||||
// localized SACRED identity anchor — honest, in-language, no fabrication.
|
|
||||||
if str_eq(reply_lang, "en") { return sr_readout("en") }
|
|
||||||
return ml_tr("identity", reply_lang)
|
|
||||||
}
|
|
||||||
// self region thin — honest localized identity (logged fallback shape).
|
|
||||||
return ml_tr("identity", reply_lang)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── PROJECT into MEMORY geometry ──────────────────────────────────────────
|
|
||||||
let terms: [String] = dlg_content_terms(content, content_lang)
|
|
||||||
let qterm: String = str_join(terms, " ")
|
|
||||||
let act: String = engram_activate_json(qterm, 12)
|
|
||||||
let n: Int = json_array_len(act)
|
|
||||||
|
|
||||||
// ── LAND: the highest-activation node that ACTUALLY overlaps the query's
|
|
||||||
// content terms (the relevance floor). Activation always returns the most
|
|
||||||
// salient nodes, so we walk the ranked list and take the first that is
|
|
||||||
// genuinely "close"; if none is, the query landed nowhere. ───────────────
|
|
||||||
let landing: String = ""
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
if str_eq(landing, "") {
|
|
||||||
let rec: String = json_array_get(act, i)
|
|
||||||
let node: String = json_get_raw(rec, "node")
|
|
||||||
if dlg_node_matches(node, terms) {
|
|
||||||
let landing = node
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let i = i + 1
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── HONEST ABSENCE: nothing close — an empty region, not a fabricated answer,
|
|
||||||
// not an "I noted that" echo. ────────────────────────────────────────────
|
|
||||||
if str_eq(landing, "") {
|
|
||||||
return ml_tr("no_memory", reply_lang)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── MATERIALIZE the landing by WALKING its neighborhood. ──────────────────
|
|
||||||
return dlg_materialize(landing, reply_lang)
|
|
||||||
}
|
|
||||||
@@ -63,9 +63,6 @@ import "morphology-cop.el"
|
|||||||
import "grammar.el"
|
import "grammar.el"
|
||||||
import "realizer.el"
|
import "realizer.el"
|
||||||
import "semantics.el"
|
import "semantics.el"
|
||||||
|
|
||||||
// ── Comprehension front-end (input half: text → meaning-spec) ─────────────────
|
|
||||||
import "comprehend.el"
|
|
||||||
//
|
//
|
||||||
// Entry points:
|
// Entry points:
|
||||||
//
|
//
|
||||||
@@ -120,9 +117,6 @@ fn build_form_from_json(semantic_form_json: String, lang_code: String) -> [Strin
|
|||||||
let location: String = sem_get(semantic_form_json, "location")
|
let location: String = sem_get(semantic_form_json, "location")
|
||||||
let tense: String = sem_get(semantic_form_json, "tense")
|
let tense: String = sem_get(semantic_form_json, "tense")
|
||||||
let aspect: String = sem_get(semantic_form_json, "aspect")
|
let aspect: String = sem_get(semantic_form_json, "aspect")
|
||||||
let polarity: String = sem_get(semantic_form_json, "polarity")
|
|
||||||
let neg_word: String = sem_get(semantic_form_json, "neg_word")
|
|
||||||
let iobj: String = sem_get(semantic_form_json, "iobj")
|
|
||||||
|
|
||||||
let form: [String] = native_list_empty()
|
let form: [String] = native_list_empty()
|
||||||
let form = native_list_append(form, "intent")
|
let form = native_list_append(form, "intent")
|
||||||
@@ -133,19 +127,12 @@ fn build_form_from_json(semantic_form_json: String, lang_code: String) -> [Strin
|
|||||||
let form = native_list_append(form, predicate)
|
let form = native_list_append(form, predicate)
|
||||||
let form = native_list_append(form, "patient")
|
let form = native_list_append(form, "patient")
|
||||||
let form = native_list_append(form, patient)
|
let form = native_list_append(form, patient)
|
||||||
let form = native_list_append(form, "iobj")
|
|
||||||
let form = native_list_append(form, iobj)
|
|
||||||
let form = native_list_append(form, "location")
|
let form = native_list_append(form, "location")
|
||||||
let form = native_list_append(form, location)
|
let form = native_list_append(form, location)
|
||||||
let form = native_list_append(form, "tense")
|
let form = native_list_append(form, "tense")
|
||||||
let form = native_list_append(form, tense)
|
let form = native_list_append(form, tense)
|
||||||
let form = native_list_append(form, "aspect")
|
let form = native_list_append(form, "aspect")
|
||||||
let form = native_list_append(form, aspect)
|
let form = native_list_append(form, aspect)
|
||||||
// SACRED: polarity crosses the JSON boundary and is never inferred away.
|
|
||||||
let form = native_list_append(form, "polarity")
|
|
||||||
let form = native_list_append(form, polarity)
|
|
||||||
let form = native_list_append(form, "neg_word")
|
|
||||||
let form = native_list_append(form, neg_word)
|
|
||||||
let form = native_list_append(form, "lang")
|
let form = native_list_append(form, "lang")
|
||||||
let form = native_list_append(form, lang_code)
|
let form = native_list_append(form, lang_code)
|
||||||
|
|
||||||
|
|||||||
@@ -1,65 +0,0 @@
|
|||||||
// image-demo.el - Drive the native PNG surface: plan a scene from a small
|
|
||||||
// meaning phrase (incl. a NEG frame) and emit a byte-valid 64x64 PNG whose
|
|
||||||
// palette is read from elp/faculty/sig/scene.basis.
|
|
||||||
|
|
||||||
fn img_frame(relation: String, polarity: String, confidence: String, importance: String, salience: String, subj_id: String) -> [String] {
|
|
||||||
let f: [String] = native_list_empty()
|
|
||||||
let f: [String] = native_list_append(f, "relation")
|
|
||||||
let f: [String] = native_list_append(f, relation)
|
|
||||||
let f: [String] = native_list_append(f, "polarity")
|
|
||||||
let f: [String] = native_list_append(f, polarity)
|
|
||||||
let f: [String] = native_list_append(f, "confidence")
|
|
||||||
let f: [String] = native_list_append(f, confidence)
|
|
||||||
let f: [String] = native_list_append(f, "importance")
|
|
||||||
let f: [String] = native_list_append(f, importance)
|
|
||||||
let f: [String] = native_list_append(f, "salience")
|
|
||||||
let f: [String] = native_list_append(f, salience)
|
|
||||||
let f: [String] = native_list_append(f, "subj_id")
|
|
||||||
let f: [String] = native_list_append(f, subj_id)
|
|
||||||
return f
|
|
||||||
}
|
|
||||||
|
|
||||||
fn rgb_str(c: [Int]) -> String {
|
|
||||||
return int_to_str(native_list_get(c, 0)) + "," + int_to_str(native_list_get(c, 1)) + "," + int_to_str(native_list_get(c, 2))
|
|
||||||
}
|
|
||||||
|
|
||||||
fn run_image() -> Int {
|
|
||||||
fs_mkdir("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out")
|
|
||||||
let table: [Int] = crc_table()
|
|
||||||
println("crc_table[1]=" + int_to_str(native_list_get(table, 1)) + " (expect 1996959894 / 0x77073096)")
|
|
||||||
|
|
||||||
let basis: [String] = basis_load("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/scene.basis")
|
|
||||||
let warm: [Int] = parse_rgb(basis_field(basis, "warm"))
|
|
||||||
let cool: [Int] = parse_rgb(basis_field(basis, "cool"))
|
|
||||||
let bg: [Int] = parse_rgb(basis_field(basis, "bg"))
|
|
||||||
println("basis warm=" + rgb_str(warm) + " cool=" + rgb_str(cool) + " bg=" + rgb_str(bg) + " (read from scene.basis)")
|
|
||||||
|
|
||||||
let frames: [[String]] = native_list_empty()
|
|
||||||
let frames: [[String]] = native_list_append(frames, img_frame("agent", "aff", "0.9", "0.8", "0", "s1"))
|
|
||||||
let frames: [[String]] = native_list_append(frames, img_frame("theme", "aff", "0.7", "0.6", "1", "s2"))
|
|
||||||
let frames: [[String]] = native_list_append(frames, img_frame("cause", "aff", "0.8", "0.9", "0", "s3"))
|
|
||||||
let frames: [[String]] = native_list_append(frames, img_frame("negation", "neg", "0.85", "0.7", "1", "s4"))
|
|
||||||
let frames: [[String]] = native_list_append(frames, img_frame("goal", "aff", "0.6", "0.5", "0", "s5"))
|
|
||||||
let frames: [[String]] = native_list_append(frames, img_frame("result", "aff", "0.95", "1.0", "1", "s6"))
|
|
||||||
|
|
||||||
let shapes: [[Int]] = plan_scene(frames, warm, cool)
|
|
||||||
let ns: Int = native_list_len(shapes)
|
|
||||||
println("planned " + int_to_str(ns) + " shapes:")
|
|
||||||
let si: Int = 0
|
|
||||||
while si < ns {
|
|
||||||
let sh: [Int] = native_list_get(shapes, si)
|
|
||||||
let pol: String = surface_get(native_list_get(frames, si), "polarity")
|
|
||||||
println(" shape " + int_to_str(si) + " type=" + int_to_str(native_list_get(sh, 0)) + " x=" + int_to_str(native_list_get(sh, 1)) + " y=" + int_to_str(native_list_get(sh, 2)) + " size=" + int_to_str(native_list_get(sh, 3)) + " rgb=" + int_to_str(native_list_get(sh, 4)) + "," + int_to_str(native_list_get(sh, 5)) + "," + int_to_str(native_list_get(sh, 6)) + " polarity=" + pol)
|
|
||||||
let si: Int = si + 1
|
|
||||||
}
|
|
||||||
|
|
||||||
let raw: [Int] = rasterize(64, 64, shapes, bg)
|
|
||||||
println("rasterized raw (filtered scanlines) bytes=" + int_to_str(native_list_len(raw)) + " (expect 12352)")
|
|
||||||
let png: [Int] = png_build(64, 64, raw, table)
|
|
||||||
let plen: Int = native_list_len(png)
|
|
||||||
let ok: Int = png_write("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/scene.png", png)
|
|
||||||
println("PNG bytes=" + int_to_str(plen) + " -> /Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/scene.png (write_ok=" + int_to_str(ok) + ")")
|
|
||||||
return plen
|
|
||||||
}
|
|
||||||
|
|
||||||
println("image-demo returned png_bytes=" + int_to_str(run_image()))
|
|
||||||
@@ -1,412 +0,0 @@
|
|||||||
// image-surface.el - Native own-core raster PNG surface (the image efferent
|
|
||||||
// twin of audio). Renders a 64x64 RGB scene deterministically from a frame's
|
|
||||||
// meaning-geometry, then serialises a byte-valid PNG entirely own-core:
|
|
||||||
// 8-byte magic, IHDR, IDAT (zlib STORED/uncompressed DEFLATE + Adler32), IEND,
|
|
||||||
// with a per-chunk CRC32 computed via software xor32 (EL has no bitwise ops).
|
|
||||||
//
|
|
||||||
// The RGB palette basis is read from elp/faculty/sig/scene.basis (data, not
|
|
||||||
// literals) - the same read-from-learned discipline as the audio signatures.
|
|
||||||
// Integer-only throughout; pixels are composed functionally (painter's order)
|
|
||||||
// so no list mutation is needed.
|
|
||||||
|
|
||||||
// -- small int/parse helpers (self-contained) ----------------------------------
|
|
||||||
|
|
||||||
fn i_str_to_int(s: String) -> Int {
|
|
||||||
let n: Int = str_len(s)
|
|
||||||
let i: Int = 0
|
|
||||||
let v: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let c: Int = str_char_code(s, i)
|
|
||||||
if c >= 48 {
|
|
||||||
if c < 58 {
|
|
||||||
let v: Int = v * 10 + (c - 48)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return v
|
|
||||||
}
|
|
||||||
|
|
||||||
fn basis_load(path: String) -> [String] {
|
|
||||||
return str_split(fs_read(path), "\n")
|
|
||||||
}
|
|
||||||
|
|
||||||
fn basis_field(lines: [String], key: String) -> String {
|
|
||||||
let pref: String = key + ": "
|
|
||||||
let n: Int = native_list_len(lines)
|
|
||||||
let plen: Int = str_len(pref)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let ln: String = native_list_get(lines, i)
|
|
||||||
if str_starts_with(ln, pref) {
|
|
||||||
return str_slice(ln, plen, str_len(ln))
|
|
||||||
}
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
fn parse_rgb(csv: String) -> [Int] {
|
|
||||||
let parts: [String] = str_split(csv, ",")
|
|
||||||
let out: [Int] = native_list_empty()
|
|
||||||
let n: Int = native_list_len(parts)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let v: Int = i_str_to_int(native_list_get(parts, i))
|
|
||||||
let out: [Int] = native_list_append(out, v)
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- software 32-bit XOR (no bitwise ops in EL) --------------------------------
|
|
||||||
|
|
||||||
fn xor32(a: Int, b: Int) -> Int {
|
|
||||||
let r: Int = 0
|
|
||||||
let bit: Int = 1
|
|
||||||
let i: Int = 0
|
|
||||||
while i < 32 {
|
|
||||||
let abit: Int = (a / bit) % 2
|
|
||||||
let bbit: Int = (b / bit) % 2
|
|
||||||
if abit != bbit {
|
|
||||||
let add: Int = bit
|
|
||||||
let r: Int = r + add
|
|
||||||
}
|
|
||||||
let bit: Int = bit * 2
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return r
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- CRC32 (table-driven, table built with xor32) ------------------------------
|
|
||||||
|
|
||||||
fn crc_table() -> [Int] {
|
|
||||||
let t: [Int] = native_list_empty()
|
|
||||||
let n: Int = 0
|
|
||||||
while n < 256 {
|
|
||||||
let c: Int = n
|
|
||||||
let k: Int = 0
|
|
||||||
while k < 8 {
|
|
||||||
if c % 2 == 1 {
|
|
||||||
let h: Int = c / 2
|
|
||||||
let c: Int = xor32(h, 3988292384)
|
|
||||||
} else {
|
|
||||||
let c: Int = c / 2
|
|
||||||
}
|
|
||||||
let k: Int = k + 1
|
|
||||||
}
|
|
||||||
let t: [Int] = native_list_append(t, c)
|
|
||||||
let n: Int = n + 1
|
|
||||||
}
|
|
||||||
return t
|
|
||||||
}
|
|
||||||
|
|
||||||
fn crc32_of(bytes: [Int], table: [Int]) -> Int {
|
|
||||||
let crc: Int = 4294967295
|
|
||||||
let n: Int = native_list_len(bytes)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let b: Int = native_list_get(bytes, i)
|
|
||||||
let lo: Int = crc % 256
|
|
||||||
let idx: Int = xor32(lo, b) % 256
|
|
||||||
let tv: Int = native_list_get(table, idx)
|
|
||||||
let hi: Int = crc / 256
|
|
||||||
let crc: Int = xor32(hi, tv)
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return xor32(crc, 4294967295)
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- Adler32 (for the zlib trailer) --------------------------------------------
|
|
||||||
|
|
||||||
fn adler32_of(bytes: [Int]) -> Int {
|
|
||||||
let a: Int = 1
|
|
||||||
let b: Int = 0
|
|
||||||
let n: Int = native_list_len(bytes)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let byte: Int = native_list_get(bytes, i)
|
|
||||||
let a: Int = (a + byte) % 65521
|
|
||||||
let b: Int = (b + a) % 65521
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return b * 65536 + a
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- byte-list append helpers --------------------------------------------------
|
|
||||||
|
|
||||||
fn app_u32be(dst: [Int], v: Int) -> [Int] {
|
|
||||||
let dst: [Int] = native_list_append(dst, (v / 16777216) % 256)
|
|
||||||
let dst: [Int] = native_list_append(dst, (v / 65536) % 256)
|
|
||||||
let dst: [Int] = native_list_append(dst, (v / 256) % 256)
|
|
||||||
let dst: [Int] = native_list_append(dst, v % 256)
|
|
||||||
return dst
|
|
||||||
}
|
|
||||||
|
|
||||||
fn app_tag(dst: [Int], s: String) -> [Int] {
|
|
||||||
let n: Int = str_len(s)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let dst: [Int] = native_list_append(dst, str_char_code(s, i))
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return dst
|
|
||||||
}
|
|
||||||
|
|
||||||
fn app_all(dst: [Int], src: [Int]) -> [Int] {
|
|
||||||
let n: Int = native_list_len(src)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let dst: [Int] = native_list_append(dst, native_list_get(src, i))
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return dst
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- plan: frame meaning-geometry -> shape atoms -------------------------------
|
|
||||||
// shape = [type, x, y, size, r, g, b] (type 0=rect 1=disc 2=triangle)
|
|
||||||
|
|
||||||
fn charsum(s: String) -> Int {
|
|
||||||
let n: Int = str_len(s)
|
|
||||||
let i: Int = 0
|
|
||||||
let acc: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let c: Int = str_char_code(s, i)
|
|
||||||
let acc: Int = acc + c
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return acc
|
|
||||||
}
|
|
||||||
|
|
||||||
fn micro_of(s: String) -> Int {
|
|
||||||
let dot: Int = str_index_of(s, ".")
|
|
||||||
if dot < 0 { return i_str_to_int(s) * 1000000 }
|
|
||||||
let n: Int = str_len(s)
|
|
||||||
let fp: String = str_slice(s, dot + 1, n)
|
|
||||||
let ip: String = str_slice(s, 0, dot)
|
|
||||||
let iv: Int = i_str_to_int(ip)
|
|
||||||
let fv: Int = 0
|
|
||||||
let scale: Int = 100000
|
|
||||||
let fl: Int = str_len(fp)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < 6 {
|
|
||||||
let d: Int = 0
|
|
||||||
if i < fl { let d: Int = str_char_code(fp, i) - 48 }
|
|
||||||
let fv: Int = fv + d * scale
|
|
||||||
let scale: Int = scale / 10
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
return iv * 1000000 + fv
|
|
||||||
}
|
|
||||||
|
|
||||||
fn plan_scene(frames: [[String]], warm: [Int], cool: [Int]) -> [[Int]] {
|
|
||||||
let shapes: [[Int]] = native_list_empty()
|
|
||||||
let nf: Int = native_list_len(frames)
|
|
||||||
let fi: Int = 0
|
|
||||||
while fi < nf {
|
|
||||||
let fr: [String] = native_list_get(frames, fi)
|
|
||||||
let relation: String = surface_get(fr, "relation")
|
|
||||||
let polarity: String = surface_get(fr, "polarity")
|
|
||||||
let confidence: String = surface_get(fr, "confidence")
|
|
||||||
let importance: String = surface_get(fr, "importance")
|
|
||||||
let salience: String = surface_get(fr, "salience")
|
|
||||||
// relation -> shape type
|
|
||||||
let stype: Int = charsum(relation) % 3
|
|
||||||
// confidence -> size (8..22)
|
|
||||||
let cmi: Int = micro_of(confidence)
|
|
||||||
let size: Int = 8 + cmi / 71428
|
|
||||||
// salience -> y
|
|
||||||
let sal: Int = i_str_to_int(salience)
|
|
||||||
let y: Int = 6 + sal * 26
|
|
||||||
// subj_id/index -> x
|
|
||||||
let x: Int = 4 + (fi * 10) % 48
|
|
||||||
// polarity -> warm/cool base color
|
|
||||||
let br: Int = native_list_get(warm, 0)
|
|
||||||
let bg2: Int = native_list_get(warm, 1)
|
|
||||||
let bb: Int = native_list_get(warm, 2)
|
|
||||||
if str_eq(polarity, "neg") {
|
|
||||||
let br: Int = native_list_get(cool, 0)
|
|
||||||
let bg2: Int = native_list_get(cool, 1)
|
|
||||||
let bb: Int = native_list_get(cool, 2)
|
|
||||||
}
|
|
||||||
// importance -> brightness (500..1000 permille)
|
|
||||||
let imi: Int = micro_of(importance)
|
|
||||||
let bpm: Int = 500 + imi / 2000
|
|
||||||
let r: Int = br * bpm / 1000
|
|
||||||
let g: Int = bg2 * bpm / 1000
|
|
||||||
let b: Int = bb * bpm / 1000
|
|
||||||
let sh: [Int] = native_list_empty()
|
|
||||||
let sh: [Int] = native_list_append(sh, stype)
|
|
||||||
let sh: [Int] = native_list_append(sh, x)
|
|
||||||
let sh: [Int] = native_list_append(sh, y)
|
|
||||||
let sh: [Int] = native_list_append(sh, size)
|
|
||||||
let sh: [Int] = native_list_append(sh, r)
|
|
||||||
let sh: [Int] = native_list_append(sh, g)
|
|
||||||
let sh: [Int] = native_list_append(sh, b)
|
|
||||||
let shapes: [[Int]] = native_list_append(shapes, sh)
|
|
||||||
let fi: Int = fi + 1
|
|
||||||
}
|
|
||||||
return shapes
|
|
||||||
}
|
|
||||||
|
|
||||||
// covers: is (px,py) inside this shape?
|
|
||||||
fn covers(sh: [Int], px: Int, py: Int) -> Bool {
|
|
||||||
let stype: Int = native_list_get(sh, 0)
|
|
||||||
let sx: Int = native_list_get(sh, 1)
|
|
||||||
let sy: Int = native_list_get(sh, 2)
|
|
||||||
let size: Int = native_list_get(sh, 3)
|
|
||||||
let cx: Int = sx + size / 2
|
|
||||||
if stype == 0 {
|
|
||||||
if px >= sx {
|
|
||||||
if px < sx + size {
|
|
||||||
if py >= sy {
|
|
||||||
if py < sy + size {
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
if stype == 1 {
|
|
||||||
let rad: Int = size / 2
|
|
||||||
let dx: Int = px - cx
|
|
||||||
let dy: Int = py - (sy + rad)
|
|
||||||
if dx * dx + dy * dy <= rad * rad {
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
// triangle: apex at top (sy), base at sy+size
|
|
||||||
if py >= sy {
|
|
||||||
if py < sy + size {
|
|
||||||
let dyv: Int = py - sy
|
|
||||||
let halfw: Int = dyv / 2
|
|
||||||
let dxv: Int = px - cx
|
|
||||||
let adx: Int = dxv
|
|
||||||
if adx < 0 { let adx: Int = 0 - dxv }
|
|
||||||
if adx <= halfw {
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
// pixel_color: painter's algorithm - last covering shape wins. Returns [r,g,b].
|
|
||||||
fn pixel_color(px: Int, py: Int, shapes: [[Int]], bg: [Int]) -> [Int] {
|
|
||||||
let r: Int = native_list_get(bg, 0)
|
|
||||||
let g: Int = native_list_get(bg, 1)
|
|
||||||
let b: Int = native_list_get(bg, 2)
|
|
||||||
let n: Int = native_list_len(shapes)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let sh: [Int] = native_list_get(shapes, i)
|
|
||||||
if covers(sh, px, py) {
|
|
||||||
let r: Int = native_list_get(sh, 4)
|
|
||||||
let g: Int = native_list_get(sh, 5)
|
|
||||||
let b: Int = native_list_get(sh, 6)
|
|
||||||
}
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
let out: [Int] = native_list_empty()
|
|
||||||
let out: [Int] = native_list_append(out, r)
|
|
||||||
let out: [Int] = native_list_append(out, g)
|
|
||||||
let out: [Int] = native_list_append(out, b)
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// rasterize: build the raw (filtered) scanline byte stream, filter byte 0 / row.
|
|
||||||
fn rasterize(w: Int, h: Int, shapes: [[Int]], bg: [Int]) -> [Int] {
|
|
||||||
let raw: [Int] = native_list_empty()
|
|
||||||
let y: Int = 0
|
|
||||||
while y < h {
|
|
||||||
let raw: [Int] = native_list_append(raw, 0)
|
|
||||||
let x: Int = 0
|
|
||||||
while x < w {
|
|
||||||
let col: [Int] = pixel_color(x, y, shapes, bg)
|
|
||||||
let raw: [Int] = native_list_append(raw, native_list_get(col, 0))
|
|
||||||
let raw: [Int] = native_list_append(raw, native_list_get(col, 1))
|
|
||||||
let raw: [Int] = native_list_append(raw, native_list_get(col, 2))
|
|
||||||
let x: Int = x + 1
|
|
||||||
}
|
|
||||||
let y: Int = y + 1
|
|
||||||
}
|
|
||||||
return raw
|
|
||||||
}
|
|
||||||
|
|
||||||
// zlib stream with a single STORED (uncompressed) DEFLATE block + Adler32.
|
|
||||||
fn zlib_store(raw: [Int]) -> [Int] {
|
|
||||||
let z: [Int] = native_list_empty()
|
|
||||||
let z: [Int] = native_list_append(z, 120)
|
|
||||||
let z: [Int] = native_list_append(z, 1)
|
|
||||||
let z: [Int] = native_list_append(z, 1)
|
|
||||||
let len: Int = native_list_len(raw)
|
|
||||||
let nlen: Int = 65535 - len
|
|
||||||
let z: [Int] = native_list_append(z, len % 256)
|
|
||||||
let z: [Int] = native_list_append(z, (len / 256) % 256)
|
|
||||||
let z: [Int] = native_list_append(z, nlen % 256)
|
|
||||||
let z: [Int] = native_list_append(z, (nlen / 256) % 256)
|
|
||||||
let z: [Int] = app_all(z, raw)
|
|
||||||
let ad: Int = adler32_of(raw)
|
|
||||||
let z: [Int] = app_u32be(z, ad)
|
|
||||||
return z
|
|
||||||
}
|
|
||||||
|
|
||||||
// append a full PNG chunk: length + (type+data) + crc32(type+data).
|
|
||||||
fn app_chunk(png: [Int], type_and_data: [Int], table: [Int]) -> [Int] {
|
|
||||||
let total: Int = native_list_len(type_and_data)
|
|
||||||
let dlen: Int = total - 4
|
|
||||||
let png: [Int] = app_u32be(png, dlen)
|
|
||||||
let png: [Int] = app_all(png, type_and_data)
|
|
||||||
let crc: Int = crc32_of(type_and_data, table)
|
|
||||||
let png: [Int] = app_u32be(png, crc)
|
|
||||||
return png
|
|
||||||
}
|
|
||||||
|
|
||||||
fn png_build(w: Int, h: Int, raw: [Int], table: [Int]) -> [Int] {
|
|
||||||
let png: [Int] = native_list_empty()
|
|
||||||
// 8-byte signature
|
|
||||||
let png: [Int] = native_list_append(png, 137)
|
|
||||||
let png: [Int] = native_list_append(png, 80)
|
|
||||||
let png: [Int] = native_list_append(png, 78)
|
|
||||||
let png: [Int] = native_list_append(png, 71)
|
|
||||||
let png: [Int] = native_list_append(png, 13)
|
|
||||||
let png: [Int] = native_list_append(png, 10)
|
|
||||||
let png: [Int] = native_list_append(png, 26)
|
|
||||||
let png: [Int] = native_list_append(png, 10)
|
|
||||||
// IHDR
|
|
||||||
let ihdr: [Int] = native_list_empty()
|
|
||||||
let ihdr: [Int] = app_tag(ihdr, "IHDR")
|
|
||||||
let ihdr: [Int] = app_u32be(ihdr, w)
|
|
||||||
let ihdr: [Int] = app_u32be(ihdr, h)
|
|
||||||
let ihdr: [Int] = native_list_append(ihdr, 8)
|
|
||||||
let ihdr: [Int] = native_list_append(ihdr, 2)
|
|
||||||
let ihdr: [Int] = native_list_append(ihdr, 0)
|
|
||||||
let ihdr: [Int] = native_list_append(ihdr, 0)
|
|
||||||
let ihdr: [Int] = native_list_append(ihdr, 0)
|
|
||||||
let png: [Int] = app_chunk(png, ihdr, table)
|
|
||||||
// IDAT
|
|
||||||
let z: [Int] = zlib_store(raw)
|
|
||||||
let idat: [Int] = native_list_empty()
|
|
||||||
let idat: [Int] = app_tag(idat, "IDAT")
|
|
||||||
let idat: [Int] = app_all(idat, z)
|
|
||||||
let png: [Int] = app_chunk(png, idat, table)
|
|
||||||
// IEND
|
|
||||||
let iend: [Int] = native_list_empty()
|
|
||||||
let iend: [Int] = app_tag(iend, "IEND")
|
|
||||||
let png: [Int] = app_chunk(png, iend, table)
|
|
||||||
return png
|
|
||||||
}
|
|
||||||
|
|
||||||
fn png_write(path: String, png: [Int]) -> Int {
|
|
||||||
let n: Int = native_list_len(png)
|
|
||||||
let buf: String = __str_alloc(n)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let buf: String = __str_set_char(buf, i, native_list_get(png, i))
|
|
||||||
let i: Int = i + 1
|
|
||||||
}
|
|
||||||
let ok: Int = fs_write_bytes(path, buf, n)
|
|
||||||
return ok
|
|
||||||
}
|
|
||||||
@@ -1,72 +0,0 @@
|
|||||||
;;; lang_profile_ca.el — Catalan language profile for ELP.
|
|
||||||
;;; Mirrors lang_profile_it / _es / _pt; keys the realizer's construction switches.
|
|
||||||
;;; Catalan is the CLOSEST Romance sibling to the shared engine (~85% conceptual
|
|
||||||
;;; reuse). The deltas: PRONOMS FEBLES with four position allomorphs, l'-elision,
|
|
||||||
;;; del/al/pel contractions, the periphrastic preterite (vaig+INF), and NO
|
|
||||||
;;; essere/avere split (perfect aux is always HAVER; ser/estar is only the copula).
|
|
||||||
|
|
||||||
(lang_profile_ca
|
|
||||||
(language "Catalan")
|
|
||||||
(iso639 "ca")
|
|
||||||
(family "Romance")
|
|
||||||
|
|
||||||
;; ── core typology flags ────────────────────────────────────────────────
|
|
||||||
(pro-drop yes) ; null subjects default; overt pronoun = emphatic
|
|
||||||
(obligatory-subject no)
|
|
||||||
(grammatical-gender yes) ; m/f; full NP agreement (art + adj + participle)
|
|
||||||
(do-support no)
|
|
||||||
(subject-aux-inversion no) ; yes/no Q = declarative order + '?'; no inversion
|
|
||||||
(article-selection "el/la/l'/els/les ; un/una/uns/unes") ; l'-ELISION:
|
|
||||||
; el/la -> l' before vowel or (silent) h, glued to
|
|
||||||
; the next word (l'home, l'illa); de -> d' before vowel
|
|
||||||
(article-drives-contraction yes) ; article choice feeds prep+article contraction
|
|
||||||
(adjective-position "postnominal-default + small prenominal class") ; bo/bon,
|
|
||||||
; mal, gran, nou, vell, primer, molt... prenominal
|
|
||||||
(question-punct plain) ; ? and ! only (no inverted ¿ ¡)
|
|
||||||
|
|
||||||
;; ── MANDATORY prep+article contractions ────────────────────────────────
|
|
||||||
(contractions ((de el del) (de els dels)
|
|
||||||
(a el al) (a els als)
|
|
||||||
(per el pel) (per els pels)))
|
|
||||||
(contraction-mandatory yes) ; *de el -> del obligatory
|
|
||||||
(contraction-blocked-before-elision yes) ; de l'home / a l'home (NO *del home)
|
|
||||||
|
|
||||||
;; ── clitic system: PRONOMS FEBLES (the headline delta) ──────────────────
|
|
||||||
(clitics yes)
|
|
||||||
(clitic-allomorphy four-position) ; per pronoun, form varies by position+onset:
|
|
||||||
; reinforced (em, et, el) proclitic before a consonant
|
|
||||||
; elided (m', t', l', n') proclitic before a vowel/h
|
|
||||||
; full (-me, -lo, -li) enclitic after a consonant/-r
|
|
||||||
; reduced ('m, 't, 'l, 'ns) enclitic after a vowel
|
|
||||||
(clitic-placement ((finite proclitic) ; el veig, no m'ho dóna
|
|
||||||
(imperative-affirmative enclitic) ; dóna'm, digues-me
|
|
||||||
(imperative-negative present-subjunctive) ; no parlis (delta)
|
|
||||||
(infinitive enclitic) ; ajudar-me, veure'l
|
|
||||||
(gerund enclitic))) ; fent-ho
|
|
||||||
(clitic-combination ((me el "me'l") (te el "te'l") (se el "se'l")
|
|
||||||
(me la "me la") (me en "me'n")
|
|
||||||
(li el "l'hi") (li en "n'hi"))) ; dative+accusative clusters
|
|
||||||
(clitic-particles (hi en ho)) ; locative hi, partitive/genitive en, neuter ho
|
|
||||||
|
|
||||||
;; ── verb / aspect system ───────────────────────────────────────────────
|
|
||||||
(finite-agreement "person+number (6-way)")
|
|
||||||
(tenses (present imperfet preterit-simple perifrastic-preterit futur
|
|
||||||
condicional subjuntiu-present subjuntiu-imperfet imperatiu))
|
|
||||||
(periphrastic-preterite "vaig/vas/va/vam/vau/van + INFINITIVE") ; << hallmark CA
|
|
||||||
; (vaig cantar = 'I sang'); coexists w/ synthetic pret.
|
|
||||||
(compound-past "pretèrit perfet = haver(present) + participle")
|
|
||||||
(perfect-aux "HAVER only") ; << NO essere/avere split (simpler than IT)
|
|
||||||
(participle-agreement ((haver preceding-acc-clitic))) ; les he vistes; else invariable
|
|
||||||
(progressive-aux "estar + gerundi")
|
|
||||||
(copula "ser / estar") ; ser: identity/essential/origin; estar:
|
|
||||||
; location + transient state (estic cansat, és a casa)
|
|
||||||
(passive-aux "ser (+ per-agent)")
|
|
||||||
(future inflectional) ; cantaré, serà
|
|
||||||
(comparative "més/menys ADJ que")
|
|
||||||
|
|
||||||
;; ── SACRED safety bar (shared with es/pt/it/en) ────────────────────────
|
|
||||||
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
|
|
||||||
(negation "no (preverbal) + optional 'pas' + concord") ; no...res/
|
|
||||||
; ningú/mai/cap/gens/enlloc
|
|
||||||
(negative-concord yes) ; preverbal negative subject (ningú) keeps 'no'
|
|
||||||
(neg-reinforcer pas)) ; optional (no ho faré pas)
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
;;; lang_profile_de.el — German language profile for ELP.
|
|
||||||
;;; Mirrors lang_profile_en / lang_profile_es. Keys the realizer's construction
|
|
||||||
;;; switches. German is the largest Germanic delta from the EN engine: V2 word
|
|
||||||
;;; order, four morphological cases, and separable-prefix verbs.
|
|
||||||
|
|
||||||
(lang_profile_de
|
|
||||||
(language "German")
|
|
||||||
(iso639 "de")
|
|
||||||
(family "Germanic")
|
|
||||||
(neighbor-base "en") ; realized by extending the English (Germanic) engine
|
|
||||||
|
|
||||||
;; ── core typology flags ────────────────────────────────────────────────
|
|
||||||
(pro-drop no) ; obligatory subject in finite clauses
|
|
||||||
(obligatory-subject yes)
|
|
||||||
(grammatical-gender (m f n)) ; three genders; drives article + adj declension
|
|
||||||
(case-system (nom acc dat gen)) ; four cases on articles/adjs/nouns
|
|
||||||
(word-order V2) ; finite verb 2nd in main clause
|
|
||||||
(subordinate-order verb-final) ; "..., dass er den Hund SIEHT."
|
|
||||||
(separable-verbs yes) ; aufstehen -> "steht ... auf"; ppart "aufgestanden"
|
|
||||||
(do-support no) ; German negates/questions the finite verb directly
|
|
||||||
(subject-verb-inversion yes) ; yes/no Q fronts finite verb; wh-Q fills Vorfeld
|
|
||||||
(article-selection "der/die/das + ein/kein") ; declined by case x gender x number
|
|
||||||
(adjective-position prenominal)
|
|
||||||
(adjective-declension (strong weak mixed)) ; chosen by the determiner type
|
|
||||||
(noun-capitalization yes)
|
|
||||||
|
|
||||||
;; ── verb / aspect system ───────────────────────────────────────────────
|
|
||||||
(finite-agreement "person-and-number") ; full present/past paradigm
|
|
||||||
(auxiliary-order (modal tense-aux perfect passive main))
|
|
||||||
(perfect-aux (haben sein)) ; sein for intransitive motion/change verbs
|
|
||||||
(passive-aux "werden")
|
|
||||||
(future "werden + infinitive")
|
|
||||||
(comparative "synthetic (-er / -st, with umlaut)")
|
|
||||||
|
|
||||||
;; ── negation ───────────────────────────────────────────────────────────
|
|
||||||
(negation-markers (nicht kein)) ; kein- negates an indefinite NP; nicht else
|
|
||||||
(negation-faithful yes) ; SACRED: polarity never dropped/inverted -> FLAG
|
|
||||||
|
|
||||||
;; ── lexicon provenance ─────────────────────────────────────────────────
|
|
||||||
(lexicon-source "UniMorph deu (primary) + kaikki.org German (gender override)")
|
|
||||||
(lexicon-license "CC-BY-SA 3.0 / GFDL"))
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
;;; lang_profile_en.el — English language profile for ELP.
|
|
||||||
;;; Mirrors lang_profile_es / lang_profile_pt; keys the realizer's construction
|
|
||||||
;;; switches. English is typologically distinct from the Romance builds, so the
|
|
||||||
;;; flags differ where the grammar differs.
|
|
||||||
|
|
||||||
(lang_profile_en
|
|
||||||
(language "English")
|
|
||||||
(iso639 "en")
|
|
||||||
(family "Germanic")
|
|
||||||
|
|
||||||
;; ── core typology flags ────────────────────────────────────────────────
|
|
||||||
(pro-drop no) ; OBLIGATORY subjects — missing subject is FLAGGED
|
|
||||||
(obligatory-subject yes)
|
|
||||||
(grammatical-gender no) ; natural gender only (he/she/it), no NP agreement
|
|
||||||
(do-support yes) ; negation & questions of lexical verbs insert do/does/did
|
|
||||||
(subject-aux-inversion yes) ; yes/no + non-subject wh questions invert the operator
|
|
||||||
(article-selection "a/an/the") ; a/an resolved PHONOLOGICALLY (an hour, a university)
|
|
||||||
(adjective-position prenominal) ; attributive adjectives precede the noun; invariant
|
|
||||||
(has-tag-questions yes) ; "...doesn't he?" — operator + reversed polarity
|
|
||||||
(has-there-existential yes) ; "there is/are/have been ..."
|
|
||||||
(possessive-clitic "'s") ; saxon genitive; plural in -s -> bare apostrophe
|
|
||||||
(question-punct plain) ; ? and ! only (no inverted marks)
|
|
||||||
|
|
||||||
;; ── verb / aspect system ───────────────────────────────────────────────
|
|
||||||
(finite-agreement "3sg-present-only") ; only 3sg present -s (+ suppletive be)
|
|
||||||
(auxiliary-order (modal perfect progressive passive main))
|
|
||||||
(perfect-aux "have") ; have + past participle
|
|
||||||
(progressive-aux "be") ; be + present participle
|
|
||||||
(passive-aux "be") ; be + past participle (+ by-agent)
|
|
||||||
(future "will + base") ; no inflectional future
|
|
||||||
(comparative "synthetic-or-periphrastic") ; -er/-est vs more/most by syllables
|
|
||||||
|
|
||||||
;; ── SACRED safety bar (shared with es/pt) ──────────────────────────────
|
|
||||||
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
|
|
||||||
|
|
||||||
;; ── DIALECT overlay (post-realization, one core -> US/UK/AU) ────────────
|
|
||||||
(dialect US) ; default; profile field switches the overlay
|
|
||||||
(dialects (US UK AU))
|
|
||||||
(dialect-canonical US) ; core is authored in US orthography
|
|
||||||
(dialect-overlay "dialect_en.to_dialect") ; orthography + lexis + grammar prefs
|
|
||||||
(dialect-covers (spelling lexis collective-agreement gotten/got)))
|
|
||||||
@@ -1,45 +0,0 @@
|
|||||||
;;; lang_profile_es.el — Spanish language profile for ELP.
|
|
||||||
;;; Keys the realizer's construction switches. Mirrors lang_profile_en / _pt.
|
|
||||||
|
|
||||||
(lang_profile_es
|
|
||||||
(language "Spanish")
|
|
||||||
(iso639 "es")
|
|
||||||
(family "Romance")
|
|
||||||
|
|
||||||
;; -- core typology flags -------------------------------------------------
|
|
||||||
(pro-drop yes) ; subjects routinely dropped; agreement carries person
|
|
||||||
(obligatory-subject no)
|
|
||||||
(grammatical-gender yes) ; m/f on every noun; article+adjective AGREE
|
|
||||||
(gender-source lexicon); REAL per-noun gender from UniMorph — NOT a heuristic
|
|
||||||
(do-support no)
|
|
||||||
(subject-aux-inversion no) ; questions by intonation/punctuation, not inversion
|
|
||||||
(question-strategy intonation)
|
|
||||||
(article-selection "el/la/los/las un/una/unos/unas")
|
|
||||||
(stressed-a-rule yes) ; fem sg noun in stressed a-/ha- takes el/un (el agua)
|
|
||||||
(adjective-position postnominal) ; default post; a few prenominal + apocope
|
|
||||||
(adjective-agreement "gender+number")
|
|
||||||
(question-punct inverted) ; opening ¿ ¡ required
|
|
||||||
|
|
||||||
;; -- MANDATORY CONTRACTIONS (coordinator quality bar) --------------------
|
|
||||||
(contractions ((de el "del") (a el "al")))
|
|
||||||
(contraction-mandatory yes) ; 'de el'/'a el' MUST surface as del/al
|
|
||||||
|
|
||||||
;; -- verb / aspect system ------------------------------------------------
|
|
||||||
(verb-classes (ar er ir))
|
|
||||||
(tenses (present preterite imperfect future conditional))
|
|
||||||
(moods (ind sbjv imp))
|
|
||||||
(finite-agreement "person+number (6 slots)")
|
|
||||||
(perfect-aux "haber") ; haber + past participle (invariant -o)
|
|
||||||
(progressive-aux "estar") ; estar + gerund
|
|
||||||
(passive-aux "ser") ; ser + participle (agrees) + por-agent
|
|
||||||
(copula-split "ser/estar") ; permanent vs stage-level
|
|
||||||
(future "infinitive + é/ás/á/emos/éis/án")
|
|
||||||
|
|
||||||
;; -- clitics / government ------------------------------------------------
|
|
||||||
(object-clitics yes) ; me te lo la le nos os los las; proclisis/enclisis
|
|
||||||
(clitic-order "se II I III (le+lo -> se lo)")
|
|
||||||
(enclisis "imperative/infinitive/gerund + accent repair (dá+me+lo->dámelo)")
|
|
||||||
(verb-prep-government yes) ; verbs select prep (protestar+contra, escapar+de)
|
|
||||||
|
|
||||||
;; -- SACRED safety bar (shared with en/pt) -------------------------------
|
|
||||||
(negation-faithful yes)) ; polarity never dropped/inverted; unplaceable -> FLAG
|
|
||||||
@@ -1,74 +0,0 @@
|
|||||||
;;; lang_profile_fr.el — French language profile for ELP.
|
|
||||||
;;; Mirrors lang_profile_it / lang_profile_es; keys the realizer's construction
|
|
||||||
;;; switches. French is a Romance sibling (~54% of the realizer code and the whole
|
|
||||||
;;; clause-engine architecture reused), but carries the family's biggest surface
|
|
||||||
;;; deltas: NOT pro-drop, DISCONTINUOUS negation, and an orthography/phonology
|
|
||||||
;;; mismatch (elision, liaison) that makes exact-match genuinely hard.
|
|
||||||
|
|
||||||
(lang_profile_fr
|
|
||||||
(language "French")
|
|
||||||
(iso639 "fr")
|
|
||||||
(family "Romance")
|
|
||||||
|
|
||||||
;; ── core typology flags ────────────────────────────────────────────────
|
|
||||||
(pro-drop no) ; << French-specific: subject clitic OBLIGATORY
|
|
||||||
(obligatory-subject yes) ; je/tu/il/elle/nous/vous/ils/elles always overt
|
|
||||||
(grammatical-gender yes) ; m/f; full NP agreement (art + adj + participle)
|
|
||||||
(do-support no)
|
|
||||||
(subject-aux-inversion optional) ; est-ce que (default) OR clitic inversion (vas-tu)
|
|
||||||
(article-selection "le/la/l'/les ; un/une/des ; PARTITIVE du/de la/de l'/des")
|
|
||||||
(article-drives-contraction yes) ; à+le=au, de+le=du feed off article choice
|
|
||||||
(adjective-position "postnominal-default + prenominal-BAGS") ; beau/bon/grand/
|
|
||||||
; petit/jeune/vieux/nouveau + ordinals prenominal
|
|
||||||
; (beau->bel, nouveau->nouvel, vieux->vieil / vowel)
|
|
||||||
(question-punct "space-before") ; French typography: ' ?' ' !' (no ¿¡)
|
|
||||||
|
|
||||||
;; ── elision (orthography/phonology mismatch — French-specific) ──────────
|
|
||||||
(elision ((le l') (la l') (je j') (ne n') (de d') (que qu')
|
|
||||||
(me m') (te t') (se s') (ce c'))) ; before vowel / h-muet
|
|
||||||
(elision-h-muet yes) ; l'homme, l'hôpital (h-aspiré exception list kept)
|
|
||||||
(liaison noted-not-modeled) ; phonological, not written in surface
|
|
||||||
|
|
||||||
;; ── MANDATORY prep+article contractions ────────────────────────────────
|
|
||||||
(contractions ((à le au) (à les aux) (de le du) (de les des)))
|
|
||||||
(contraction-mandatory yes) ; *à le -> au obligatory; à la / à l' uncontracted
|
|
||||||
(partitive ((m-sg du) (f-sg "de la") (vowel "de l'") (pl des)))
|
|
||||||
(partitive-under-neg "de") ; << gap in current build: 'ne … pas de pain'
|
|
||||||
|
|
||||||
;; ── clitic system ──────────────────────────────────────────────────────
|
|
||||||
(clitics yes)
|
|
||||||
(clitic-order (me te se nous vous | le la les | lui leur | y | en))
|
|
||||||
(clitic-placement ((finite proclitic) ; je le lui donne
|
|
||||||
(imperative-affirmative enclitic-hyphen) ; donne-le-moi
|
|
||||||
(imperative-negative "ne+proclitic+verb+pas") ; ne le donne pas
|
|
||||||
(infinitive enclitic))) ; PARTIAL: clitic-climbing
|
|
||||||
; onto infinitive under modal
|
|
||||||
(clitic-imperative-shift ((me moi) (te toi))) ; final me/te -> moi/toi (donne-moi)
|
|
||||||
(clitic-particles (y en)) ; locative y, partitive/genitive en
|
|
||||||
|
|
||||||
;; ── verb / aspect system ───────────────────────────────────────────────
|
|
||||||
(finite-agreement "person+number (written; many homophones)")
|
|
||||||
(tenses (présent imparfait passé-simple futur conditionnel
|
|
||||||
subjonctif-présent subjonctif-imparfait impératif))
|
|
||||||
(compound-past "passé-composé = aux(present) + participe passé")
|
|
||||||
(perfect-aux "être/avoir (LEXICAL selection)") ; << French-specific
|
|
||||||
(etre-aux-class "intransitive motion/change (aller venir arriver partir
|
|
||||||
entrer sortir monter descendre naître mourir rester
|
|
||||||
tomber retourner passer devenir revenir rentrer) + ALL
|
|
||||||
pronominal verbs")
|
|
||||||
(participle-agreement ((être subject) ; elle est allée / elles venues
|
|
||||||
(avoir preceding-direct-object))) ; je les ai vus
|
|
||||||
(progressive "être en train de + infinitif") ; no dedicated aux
|
|
||||||
(copula "être (single; no ser/estar, no essere/stare)")
|
|
||||||
(passive-aux "être (+ par-agent)")
|
|
||||||
(future inflectional) ; parlera, sera
|
|
||||||
(comparative "plus/moins ADJ que")
|
|
||||||
(superlative "le/la plus ADJ (de …)") ; PARTIAL word-order in build
|
|
||||||
|
|
||||||
;; ── SACRED safety bar (shared with es/pt/it/en) ────────────────────────
|
|
||||||
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
|
|
||||||
(negation "DISCONTINUOUS: ne (preverbal) … pas/jamais/rien/personne/
|
|
||||||
plus/guère/que (postverbal)") ; << biggest structural delta
|
|
||||||
(negation-ne-elides yes) ; ne -> n' before vowel (n'ai pas vu)
|
|
||||||
(negation-passe-composé "ne + aux + pas + participe") ; n'ai pas vu
|
|
||||||
(negative-concord partial)) ; personne/rien as arguments post-participle
|
|
||||||
@@ -1,70 +0,0 @@
|
|||||||
;;; lang_profile_it.el — Italian language profile for ELP.
|
|
||||||
;;; Mirrors lang_profile_es / lang_profile_pt; keys the realizer's construction
|
|
||||||
;;; switches. Italian is a Romance sibling, so ~85% of the flags match ES/PT; the
|
|
||||||
;;; essere/avere auxiliary split and phonological article selection are the deltas.
|
|
||||||
|
|
||||||
(lang_profile_it
|
|
||||||
(language "Italian")
|
|
||||||
(iso639 "it")
|
|
||||||
(family "Romance")
|
|
||||||
|
|
||||||
;; ── core typology flags ────────────────────────────────────────────────
|
|
||||||
(pro-drop yes) ; null subjects default; overt pronoun = emphatic
|
|
||||||
(obligatory-subject no)
|
|
||||||
(grammatical-gender yes) ; m/f; full NP agreement (art + adj + participle)
|
|
||||||
(do-support no)
|
|
||||||
(subject-aux-inversion no) ; yes/no Q = declarative order + '?'; no inversion
|
|
||||||
(article-selection "il/lo/l'/i/gli + la/l'/le ; un/uno/un'/una") ; PHONOLOGICAL:
|
|
||||||
; lo/gli/uno before s+cons, z, gn, ps, pn, x, y, i+V;
|
|
||||||
; l'/un' before a vowel (elision, glued to next word)
|
|
||||||
(article-drives-contraction yes) ; article choice feeds the prep+art contraction
|
|
||||||
(adjective-position "postnominal-default + prenominal-class") ; bello/buono/grande
|
|
||||||
; /nuovo/vecchio/primo... prenominal (with apocope)
|
|
||||||
(question-punct plain) ; ? and ! only (no inverted ¿ ¡)
|
|
||||||
|
|
||||||
;; ── MANDATORY prep+article contractions ────────────────────────────────
|
|
||||||
(contractions ((di il del) (di lo dello) (di la della) (di i dei)
|
|
||||||
(di gli degli) (di le delle) (di l' dell')
|
|
||||||
(a il al) (a lo allo) (a la alla) (a i ai) (a gli agli)
|
|
||||||
(a le alle) (a l' all')
|
|
||||||
(da il dal) (da la dalla) (da gli dagli) (da l' dall')
|
|
||||||
(in il nel) (in la nella) (in gli negli) (in l' nell')
|
|
||||||
(su il sul) (su la sulla) (su gli sugli) (su l' sull')))
|
|
||||||
(contraction-mandatory yes) ; *di il -> del is obligatory, never uncontracted
|
|
||||||
(prep-no-contract (per tra fra)) ; per la strada (NOT *perla)
|
|
||||||
|
|
||||||
;; ── clitic system ──────────────────────────────────────────────────────
|
|
||||||
(clitics yes)
|
|
||||||
(clitic-placement ((finite proclitic) ; lo vedo, non me lo dà
|
|
||||||
(imperative-affirmative enclitic) ; dammelo, guardalo
|
|
||||||
(imperative-negative-tu non+infinitive) ; non parlare / non lo fare
|
|
||||||
(infinitive enclitic) ; vederlo, aiutarmi (drop -e)
|
|
||||||
(gerund enclitic))) ; dandolo
|
|
||||||
(clitic-combination ((mi lo "me lo") (ti lo "te lo") (ci lo "ce lo")
|
|
||||||
(vi lo "ve lo") (si lo "se lo")
|
|
||||||
(gli lo "glielo") (le lo "glielo"))) ; glielo = ONE word
|
|
||||||
(clitic-particles (ci ne)) ; locative ci, partitive ne
|
|
||||||
(raddoppiamento (da fa di va sta)) ; monosyllabic imper double clitic: dammelo
|
|
||||||
|
|
||||||
;; ── verb / aspect system ───────────────────────────────────────────────
|
|
||||||
(finite-agreement "person+number (6-way)")
|
|
||||||
(tenses (presente imperfetto passato-remoto futuro condizionale
|
|
||||||
congiuntivo-presente congiuntivo-imperfetto imperativo))
|
|
||||||
(compound-past "passato-prossimo = aux(present) + participle")
|
|
||||||
(perfect-aux "essere/avere (LEXICAL selection)") ; << Italian-specific
|
|
||||||
(essere-aux-class unaccusative) ; motion/change-of-state/copular/pronominal
|
|
||||||
; (andare venire nascere morire diventare piacere
|
|
||||||
; + ALL reflexives) -> essere
|
|
||||||
(participle-agreement ((essere subject) ; è andata / sono arrivati
|
|
||||||
(avere preceding-acc-clitic))) ; li ho visti
|
|
||||||
(progressive-aux "stare + gerundio") ; sto parlando
|
|
||||||
(copula "essere (default) / stare (state: sto bene)")
|
|
||||||
(passive-aux "essere / venire (+ da-agent)")
|
|
||||||
(future inflectional) ; parlerò, sarà
|
|
||||||
(comparative "più/meno ADJ di")
|
|
||||||
|
|
||||||
;; ── SACRED safety bar (shared with es/pt/en) ───────────────────────────
|
|
||||||
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
|
|
||||||
(negation "non (preverbal) + concord") ; non...niente/nessuno/mai/più
|
|
||||||
(negative-concord yes) ; preverbal negative word (nessuno/niente) suppresses non
|
|
||||||
(neg-adverb-position between-aux-and-participle)) ; non ho MAI visto
|
|
||||||
@@ -1,30 +0,0 @@
|
|||||||
;;; lang_profile_la.el — Latin language profile for ELP.
|
|
||||||
;;; Keys the realizer's construction switches. Companion to morphology-la.el.
|
|
||||||
|
|
||||||
(lang_profile_la
|
|
||||||
(language "Latin")
|
|
||||||
(iso639 "la")
|
|
||||||
(family "Italic")
|
|
||||||
|
|
||||||
;; -- core typology flags -------------------------------------------------
|
|
||||||
(pro-drop yes) ; person carried by verb ending; subjects dropped
|
|
||||||
(obligatory-subject no)
|
|
||||||
(grammatical-gender yes) ; m/f/n; adjective AGREES in case+gender+number
|
|
||||||
(gender-source lexicon) ; REAL per-noun gender from UniMorph lat
|
|
||||||
(articles none) ; Latin has no articles
|
|
||||||
(case-system yes) ; NOM GEN DAT ACC ABL VOC (+ rare LOC)
|
|
||||||
(cases (nom gen dat acc abl voc))
|
|
||||||
(word-order "SOV (default; free order, case-marked)")
|
|
||||||
(adjective-position "either (case agreement carries the link)")
|
|
||||||
(adjective-agreement "case+gender+number")
|
|
||||||
|
|
||||||
;; -- verb / aspect system ------------------------------------------------
|
|
||||||
(verb-classes (1 2 3 3io 4)) ; four conjugations + i-stem 3rd
|
|
||||||
(tenses (present imperfect future perfect pluperfect futureperfect))
|
|
||||||
(moods (indicative subjunctive imperative infinitive))
|
|
||||||
(voices (active passive))
|
|
||||||
(finite-agreement "person+number (6 slots)")
|
|
||||||
(citation "principal parts: pres-1sg / pres-inf / perf-participle")
|
|
||||||
|
|
||||||
;; -- SACRED safety bar ---------------------------------------------------
|
|
||||||
(negation-faithful yes)) ; polarity never dropped/inverted
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
;;; lang_profile_pt.el — Portuguese language profile for ELP.
|
|
||||||
;;; Keys the realizer's construction switches. Mirrors lang_profile_es.
|
|
||||||
|
|
||||||
(lang_profile_pt
|
|
||||||
(language "Portuguese")
|
|
||||||
(iso639 "pt")
|
|
||||||
(family "Romance")
|
|
||||||
|
|
||||||
;; -- core typology flags -------------------------------------------------
|
|
||||||
(pro-drop yes) ; subjects routinely dropped; agreement carries person
|
|
||||||
(obligatory-subject no)
|
|
||||||
(grammatical-gender yes) ; m/f on every noun; article+adjective AGREE
|
|
||||||
(gender-source lexicon) ; REAL per-noun gender from UniMorph por / kaikki
|
|
||||||
(do-support no)
|
|
||||||
(subject-aux-inversion no)
|
|
||||||
(question-strategy intonation)
|
|
||||||
(article-selection "o/a/os/as um/uma/uns/umas")
|
|
||||||
(adjective-position postnominal)
|
|
||||||
(adjective-agreement "gender+number")
|
|
||||||
|
|
||||||
;; -- MANDATORY CONTRACTIONS (prep + article) -----------------------------
|
|
||||||
(contractions ((de o "do") (de a "da") (em o "no") (em a "na")
|
|
||||||
(a o "ao") (a a "à") (por o "pelo") (por a "pela")))
|
|
||||||
(contraction-mandatory yes)
|
|
||||||
|
|
||||||
;; -- verb / aspect system ------------------------------------------------
|
|
||||||
(verb-classes (ar er ir))
|
|
||||||
(tenses (present preterite imperfect future conditional))
|
|
||||||
(moods (ind sbjv imp))
|
|
||||||
(finite-agreement "person+number (6 slots)")
|
|
||||||
(perfect-aux "ter") ; ter + past participle
|
|
||||||
(copula-split "ser/estar")
|
|
||||||
(personal-infinitive yes) ; distinctive PT inflected infinitive
|
|
||||||
|
|
||||||
;; -- clitics / government ------------------------------------------------
|
|
||||||
(object-clitics yes) ; mesoclisis/enclisis/proclisis by context
|
|
||||||
(verb-prep-government yes)
|
|
||||||
|
|
||||||
;; -- SACRED safety bar ---------------------------------------------------
|
|
||||||
(negation-faithful yes))
|
|
||||||
@@ -1,71 +0,0 @@
|
|||||||
;;; lang_profile_ro.el — Romanian language profile for ELP.
|
|
||||||
;;; Romanian is the BIG typological delta of the Romance family. The verb/clause
|
|
||||||
;;; engine and the SACRED negation contract mirror the ES/PT/IT core, but the
|
|
||||||
;;; NOMINAL system is genuinely new: a SUFFIXED definite article, preserved CASE,
|
|
||||||
;;; a NEUTER gender, and a VOCATIVE. Those flags mark where the shared engine was
|
|
||||||
;;; extended rather than reused.
|
|
||||||
|
|
||||||
(lang_profile_ro
|
|
||||||
(language "Romanian")
|
|
||||||
(iso639 "ro")
|
|
||||||
(family "Romance (Eastern / Balkan)")
|
|
||||||
|
|
||||||
;; ── core typology flags ────────────────────────────────────────────────
|
|
||||||
(pro-drop yes) ; null subjects default; overt pronoun = emphatic
|
|
||||||
(obligatory-subject no)
|
|
||||||
(grammatical-gender yes) ; m / f / NEUTER (n)
|
|
||||||
(neuter-gender yes) ; << ROMANIAN-SPECIFIC: masc-agreeing SG, fem-agreeing PL
|
|
||||||
; (un tren nou / două trenuri noi)
|
|
||||||
(do-support no)
|
|
||||||
(subject-aux-inversion no) ; yes/no Q = declarative order + '?'
|
|
||||||
(question-punct plain) ; ? and ! only
|
|
||||||
|
|
||||||
;; ── SUFFIXED DEFINITE ARTICLE (the headline engine extension) ───────────
|
|
||||||
(definite-article suffixed) ; << UNIQUE IN ROMANCE: enclitic on the noun
|
|
||||||
(definite-forms ((m/n sg "-ul / -le / -l : om->omul, câine->câinele, codru->codrul")
|
|
||||||
(f sg "-a / -ea / -ua : casă->casa, carte->cartea, stea->steaua")
|
|
||||||
(m pl "-i : oameni->oamenii")
|
|
||||||
(f/n pl "-le : case->casele, trenuri->trenurile")))
|
|
||||||
(article-host ((no-prenom-adj noun) ; omul bun
|
|
||||||
(prenom-adj adjective))) ; bunul om (adj carries the article)
|
|
||||||
(indefinite-article ((m/n "un") (f "o") (pl "niște") (gen/dat-pl "unor")))
|
|
||||||
|
|
||||||
;; ── CASE (preserved; NOM/ACC vs GEN/DAT) ────────────────────────────────
|
|
||||||
(case (nom/acc gen/dat vocative)) ; << ROMANIAN-SPECIFIC
|
|
||||||
(case-syncretism "nom=acc ; gen=dat")
|
|
||||||
(genitive-marking "gen/dat definite: -lui (m/n), -ei/-i (f), -lor (pl)")
|
|
||||||
(genitival-article ((m sg "al") (f sg "a") (m pl "ai") (f/n pl "ale"))) ; o carte a lui
|
|
||||||
(possession "definite-head + gen/dat possessor: casa băiatului")
|
|
||||||
(vocative ((m sg "-ule/-e : omule, băiete") (f sg "-o : Mario, fato")
|
|
||||||
(pl "-lor")))
|
|
||||||
|
|
||||||
;; ── verb / aspect system ────────────────────────────────────────────────
|
|
||||||
(finite-agreement "person+number (6-way)")
|
|
||||||
(tenses (prezent imperfect perfect-simplu conjunctiv-prezent
|
|
||||||
imperativ (periphrastic: perfect-compus viitor conditional)))
|
|
||||||
(compound-past "perfectul compus = a-avea-clitic + INVARIABLE participle")
|
|
||||||
(perfect-aux "a avea (am/ai/a/am/ați/au) — ONE auxiliary for ALL verbs")
|
|
||||||
(perfect-aux-split no) ; << SIMPLER than Italian: no essere/avere selection
|
|
||||||
(participle-agreement none) ; invariable in the perfect compus (agrees only as
|
|
||||||
; an adjective / in the passive)
|
|
||||||
(future "voi/vei/va/vom/veți/vor + infinitive (viitor literar)")
|
|
||||||
(conditional "aș/ai/ar/am/ați/ar + infinitive")
|
|
||||||
(subjunctive "conjunctiv: particle 'să' + subjunctive present")
|
|
||||||
(modal-complement "modal + să + subjunctive (vreau să merg, poți să ajuți)")
|
|
||||||
(copula "a fi")
|
|
||||||
(passive "a fi + participle (participle AGREES like an adjective)")
|
|
||||||
(comparative "mai / mai puțin ADJ decât")
|
|
||||||
|
|
||||||
;; ── clitic system (partial — see honest gaps) ───────────────────────────
|
|
||||||
(clitics yes)
|
|
||||||
(clitic-set ((acc mă te îl o ne vă îi le) (dat îmi îți îi ne vă le)
|
|
||||||
(refl mă te se ne vă se)))
|
|
||||||
(clitic-placement ((finite proclitic) ; îmi place, o văd
|
|
||||||
(perfect-compus elision) ; << m-am, l-am, i-am (PARTIAL)
|
|
||||||
(imperative-affirmative enclitic))) ; dă-mi (PARTIAL)
|
|
||||||
|
|
||||||
;; ── SACRED safety bar (shared with es/pt/it/en) ─────────────────────────
|
|
||||||
(negation-faithful yes) ; polarity never dropped/inverted; unplaceable -> FLAG
|
|
||||||
(negation "nu (single preverbal marker) + concord")
|
|
||||||
(negative-concord yes) ; nu … nimic / nimeni / niciodată / niciun
|
|
||||||
(negative-imperative "nu + INFINITIVE : nu pleca! (KNOWN GAP: uses imperative stem)"))
|
|
||||||
@@ -250,7 +250,6 @@ fn en_irregular_verb(base: String) -> [String] {
|
|||||||
if str_eq(base, "cut") { let r: [String] = ["cut", "cuts", "cut", "cut", "cutting"]; return r }
|
if str_eq(base, "cut") { let r: [String] = ["cut", "cuts", "cut", "cut", "cutting"]; return r }
|
||||||
if str_eq(base, "set") { let r: [String] = ["set", "sets", "set", "set", "setting"]; return r }
|
if str_eq(base, "set") { let r: [String] = ["set", "sets", "set", "set", "setting"]; return r }
|
||||||
if str_eq(base, "hit") { let r: [String] = ["hit", "hits", "hit", "hit", "hitting"]; return r }
|
if str_eq(base, "hit") { let r: [String] = ["hit", "hits", "hit", "hit", "hitting"]; return r }
|
||||||
if str_eq(base, "fight") { let r: [String] = ["fight", "fights","fought", "fought", "fighting"]; return r }
|
|
||||||
return empty
|
return empty
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,280 +0,0 @@
|
|||||||
// multilingual.el - the language layer for the native-el interlocutor.
|
|
||||||
//
|
|
||||||
// Deterministic, NO generative model (ports multilingual.py):
|
|
||||||
// 1. ml_detect(text) -> ISO code (en/es/pt/it) via stopword + diacritic score
|
|
||||||
// 2. ml_tr(key, lang) -> localized fixed phrase (SACRED per-language yes/no/decline)
|
|
||||||
// 3. ml_term(w, lang) -> PT/ES content term -> EN engram equivalent
|
|
||||||
// 4. ml_translate_pred(lemma, lang) -> EN predicate lemma -> target infinitive
|
|
||||||
//
|
|
||||||
// The Python detector count-weights stopwords and diacritics; here diacritics are
|
|
||||||
// scored by PRESENCE (str_contains) rather than codepoint counting, to stay clear
|
|
||||||
// of UTF-8 index hazards in the runtime. Faithful enough to classify typical
|
|
||||||
// queries; documented simplification. Depends on: comprehend (cp_tokenize).
|
|
||||||
|
|
||||||
// ── 1. language detection ─────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
fn ml_stop_en(w: String) -> Bool {
|
|
||||||
if str_eq(w, "the") { return true }
|
|
||||||
if str_eq(w, "does") { return true }
|
|
||||||
if str_eq(w, "do") { return true }
|
|
||||||
if str_eq(w, "did") { return true }
|
|
||||||
if str_eq(w, "what") { return true }
|
|
||||||
if str_eq(w, "who") { return true }
|
|
||||||
if str_eq(w, "is") { return true }
|
|
||||||
if str_eq(w, "are") { return true }
|
|
||||||
if str_eq(w, "how") { return true }
|
|
||||||
if str_eq(w, "you") { return true }
|
|
||||||
if str_eq(w, "your") { return true }
|
|
||||||
if str_eq(w, "of") { return true }
|
|
||||||
if str_eq(w, "to") { return true }
|
|
||||||
if str_eq(w, "and") { return true }
|
|
||||||
if str_eq(w, "for") { return true }
|
|
||||||
if str_eq(w, "explain") { return true }
|
|
||||||
if str_eq(w, "answer") { return true }
|
|
||||||
if str_eq(w, "memory") { return true }
|
|
||||||
if str_eq(w, "with") { return true }
|
|
||||||
if str_eq(w, "not") { return true }
|
|
||||||
if str_eq(w, "store") { return true }
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
fn ml_stop_es(w: String) -> Bool {
|
|
||||||
if str_eq(w, "que") { return true }
|
|
||||||
if str_eq(w, "qué") { return true }
|
|
||||||
if str_eq(w, "una") { return true }
|
|
||||||
if str_eq(w, "usted") { return true }
|
|
||||||
if str_eq(w, "su") { return true }
|
|
||||||
if str_eq(w, "cómo") { return true }
|
|
||||||
if str_eq(w, "como") { return true }
|
|
||||||
if str_eq(w, "cuál") { return true }
|
|
||||||
if str_eq(w, "quién") { return true }
|
|
||||||
if str_eq(w, "está") { return true }
|
|
||||||
if str_eq(w, "es") { return true }
|
|
||||||
if str_eq(w, "los") { return true }
|
|
||||||
if str_eq(w, "las") { return true }
|
|
||||||
if str_eq(w, "del") { return true }
|
|
||||||
if str_eq(w, "al") { return true }
|
|
||||||
if str_eq(w, "explica") { return true }
|
|
||||||
if str_eq(w, "explique") { return true }
|
|
||||||
if str_eq(w, "forma") { return true }
|
|
||||||
if str_eq(w, "con") { return true }
|
|
||||||
if str_eq(w, "memoria") { return true }
|
|
||||||
if str_eq(w, "responde") { return true }
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
fn ml_stop_pt(w: String) -> Bool {
|
|
||||||
if str_eq(w, "que") { return true }
|
|
||||||
if str_eq(w, "uma") { return true }
|
|
||||||
if str_eq(w, "você") { return true }
|
|
||||||
if str_eq(w, "sua") { return true }
|
|
||||||
if str_eq(w, "seu") { return true }
|
|
||||||
if str_eq(w, "como") { return true }
|
|
||||||
if str_eq(w, "memória") { return true }
|
|
||||||
if str_eq(w, "isso") { return true }
|
|
||||||
if str_eq(w, "os") { return true }
|
|
||||||
if str_eq(w, "as") { return true }
|
|
||||||
if str_eq(w, "da") { return true }
|
|
||||||
if str_eq(w, "do") { return true }
|
|
||||||
if str_eq(w, "na") { return true }
|
|
||||||
if str_eq(w, "no") { return true }
|
|
||||||
if str_eq(w, "explica") { return true }
|
|
||||||
if str_eq(w, "forma") { return true }
|
|
||||||
if str_eq(w, "é") { return true }
|
|
||||||
if str_eq(w, "está") { return true }
|
|
||||||
if str_eq(w, "com") { return true }
|
|
||||||
if str_eq(w, "responda") { return true }
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
fn ml_stop_it(w: String) -> Bool {
|
|
||||||
if str_eq(w, "che") { return true }
|
|
||||||
if str_eq(w, "una") { return true }
|
|
||||||
if str_eq(w, "come") { return true }
|
|
||||||
if str_eq(w, "della") { return true }
|
|
||||||
if str_eq(w, "gli") { return true }
|
|
||||||
if str_eq(w, "è") { return true }
|
|
||||||
if str_eq(w, "sono") { return true }
|
|
||||||
if str_eq(w, "questo") { return true }
|
|
||||||
if str_eq(w, "nel") { return true }
|
|
||||||
if str_eq(w, "di") { return true }
|
|
||||||
if str_eq(w, "il") { return true }
|
|
||||||
if str_eq(w, "cosa") { return true }
|
|
||||||
if str_eq(w, "per") { return true }
|
|
||||||
if str_eq(w, "memoria") { return true }
|
|
||||||
if str_eq(w, "spiega") { return true }
|
|
||||||
if str_eq(w, "rispondi") { return true }
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
// diacritic PRESENCE score (weight 3 each; hard overrides weight 8).
|
|
||||||
fn ml_dia_score(low: String, lang: String) -> Int {
|
|
||||||
let s: Int = 0
|
|
||||||
if str_eq(lang, "pt") {
|
|
||||||
if str_contains(low, "ã") { let s = s + 3 }
|
|
||||||
if str_contains(low, "õ") { let s = s + 3 }
|
|
||||||
if str_contains(low, "ç") { let s = s + 3 }
|
|
||||||
if str_contains(low, "ê") { let s = s + 3 }
|
|
||||||
if str_contains(low, "á") { let s = s + 3 }
|
|
||||||
// hard PT markers (ã/õ almost never appear outside PT)
|
|
||||||
if str_contains(low, "ã") { let s = s + 8 }
|
|
||||||
if str_contains(low, "õ") { let s = s + 8 }
|
|
||||||
}
|
|
||||||
if str_eq(lang, "es") {
|
|
||||||
if str_contains(low, "ñ") { let s = s + 3 }
|
|
||||||
if str_contains(low, "¿") { let s = s + 3 }
|
|
||||||
if str_contains(low, "¡") { let s = s + 3 }
|
|
||||||
if str_contains(low, "á") { let s = s + 3 }
|
|
||||||
if str_contains(low, "é") { let s = s + 3 }
|
|
||||||
// hard ES markers
|
|
||||||
if str_contains(low, "ñ") { let s = s + 8 }
|
|
||||||
if str_contains(low, "¿") { let s = s + 8 }
|
|
||||||
if str_contains(low, "¡") { let s = s + 8 }
|
|
||||||
}
|
|
||||||
if str_eq(lang, "it") {
|
|
||||||
if str_contains(low, "è") { let s = s + 3 }
|
|
||||||
if str_contains(low, "ì") { let s = s + 3 }
|
|
||||||
if str_contains(low, "ò") { let s = s + 3 }
|
|
||||||
}
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
|
|
||||||
fn ml_stop_score(toks: [String], lang: String) -> Int {
|
|
||||||
let n: Int = native_list_len(toks)
|
|
||||||
let s: Int = 0
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let w: String = native_list_get(toks, i)
|
|
||||||
if str_eq(lang, "en") { if ml_stop_en(w) { let s = s + 2 } }
|
|
||||||
if str_eq(lang, "es") { if ml_stop_es(w) { let s = s + 2 } }
|
|
||||||
if str_eq(lang, "pt") { if ml_stop_pt(w) { let s = s + 2 } }
|
|
||||||
if str_eq(lang, "it") { if ml_stop_it(w) { let s = s + 2 } }
|
|
||||||
let i = i + 1
|
|
||||||
}
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
|
|
||||||
fn ml_detect(text: String) -> String {
|
|
||||||
if str_eq(text, "") { return "en" }
|
|
||||||
let low: String = str_to_lower(text)
|
|
||||||
let toks: [String] = cp_tokenize(text)
|
|
||||||
// NOTE: el's overloaded `+` mis-compiles two chained function-call Int operands
|
|
||||||
// as string concat (documented in comprehend_gate.el). Bind each call to an Int
|
|
||||||
// var and add vars one at a time so the addition stays integer.
|
|
||||||
let en: Int = ml_stop_score(toks, "en")
|
|
||||||
let es_s: Int = ml_stop_score(toks, "es")
|
|
||||||
let es_d: Int = ml_dia_score(low, "es")
|
|
||||||
let es: Int = es_s + es_d
|
|
||||||
let pt_s: Int = ml_stop_score(toks, "pt")
|
|
||||||
let pt_d: Int = ml_dia_score(low, "pt")
|
|
||||||
let pt: Int = pt_s + pt_d
|
|
||||||
let it_s: Int = ml_stop_score(toks, "it")
|
|
||||||
let it_d: Int = ml_dia_score(low, "it")
|
|
||||||
let it: Int = it_s + it_d
|
|
||||||
|
|
||||||
let best: String = "en"
|
|
||||||
let bs: Int = en
|
|
||||||
if es > bs { let best = "es"; let bs = es }
|
|
||||||
if pt > bs { let best = "pt"; let bs = pt }
|
|
||||||
if it > bs { let best = "it"; let bs = it }
|
|
||||||
// weak signal -> honest fallback to English
|
|
||||||
if bs < 3 { return "en" }
|
|
||||||
return best
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── 2. localized fixed phrases (SACRED per-language decline/yes/no) ────────────
|
|
||||||
|
|
||||||
fn ml_tr(key: String, lang: String) -> String {
|
|
||||||
if str_eq(key, "no_memory") {
|
|
||||||
if str_eq(lang, "pt") { return "Não tenho isso na minha memória." }
|
|
||||||
if str_eq(lang, "es") { return "No tengo eso en mi memoria." }
|
|
||||||
if str_eq(lang, "it") { return "Non ho quello nella mia memoria." }
|
|
||||||
return "I don't have that in my memory."
|
|
||||||
}
|
|
||||||
if str_eq(key, "parse_fail") {
|
|
||||||
if str_eq(lang, "pt") { return "Não consegui interpretar isso." }
|
|
||||||
if str_eq(lang, "es") { return "No pude interpretar eso." }
|
|
||||||
if str_eq(lang, "it") { return "Non sono riuscito a interpretarlo." }
|
|
||||||
return "I didn't parse that."
|
|
||||||
}
|
|
||||||
if str_eq(key, "yes") {
|
|
||||||
if str_eq(lang, "pt") { return "Sim" }
|
|
||||||
if str_eq(lang, "es") { return "Sí" }
|
|
||||||
if str_eq(lang, "it") { return "Sì" }
|
|
||||||
return "Yes"
|
|
||||||
}
|
|
||||||
if str_eq(key, "no") {
|
|
||||||
if str_eq(lang, "pt") { return "Não" }
|
|
||||||
if str_eq(lang, "es") { return "No" }
|
|
||||||
if str_eq(lang, "it") { return "No" }
|
|
||||||
return "No"
|
|
||||||
}
|
|
||||||
if str_eq(key, "identity") {
|
|
||||||
if str_eq(lang, "pt") { return "Sou o Neuron, o engrama com quem você está falando." }
|
|
||||||
if str_eq(lang, "es") { return "Soy Neuron, el engrama con el que estás hablando." }
|
|
||||||
if str_eq(lang, "it") { return "Sono Neuron, l'engramma con cui stai parlando." }
|
|
||||||
return "I'm Neuron, the engram you're speaking with."
|
|
||||||
}
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── 3. retrieval term lexicon (PT/ES content term -> EN engram equivalent) ─────
|
|
||||||
|
|
||||||
fn ml_term(w: String, lang: String) -> String {
|
|
||||||
if str_eq(lang, "en") { return w }
|
|
||||||
if str_eq(w, "saliência") { return "salience" }
|
|
||||||
if str_eq(w, "saliencia") { return "salience" }
|
|
||||||
if str_eq(w, "memória") { return "memory" }
|
|
||||||
if str_eq(w, "memoria") { return "memory" }
|
|
||||||
if str_eq(w, "geometria") { return "geometry" }
|
|
||||||
if str_eq(w, "geometrias") { return "geometry" }
|
|
||||||
if str_eq(w, "geometrías") { return "geometry" }
|
|
||||||
if str_eq(w, "forma") { return "form" }
|
|
||||||
if str_eq(w, "consolidação") { return "consolidation" }
|
|
||||||
if str_eq(w, "consolidación") { return "consolidation" }
|
|
||||||
if str_eq(w, "aprendizagem") { return "learning" }
|
|
||||||
if str_eq(w, "aprendizaje") { return "learning" }
|
|
||||||
if str_eq(w, "nó") { return "node" }
|
|
||||||
if str_eq(w, "nodo") { return "node" }
|
|
||||||
if str_eq(w, "armazenamento") { return "storage" }
|
|
||||||
if str_eq(w, "almacenamiento") { return "storage" }
|
|
||||||
if str_eq(w, "estrutura") { return "structure" }
|
|
||||||
if str_eq(w, "estructura") { return "structure" }
|
|
||||||
return w
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── 4. predicate translation (EN lemma -> target infinitive; pass-through) ─────
|
|
||||||
|
|
||||||
fn ml_translate_pred(lemma: String, lang: String) -> String {
|
|
||||||
if str_eq(lang, "en") { return lemma }
|
|
||||||
if str_eq(lang, "es") {
|
|
||||||
if str_eq(lemma, "store") { return "almacenar" }
|
|
||||||
if str_eq(lemma, "use") { return "usar" }
|
|
||||||
if str_eq(lemma, "have") { return "tener" }
|
|
||||||
if str_eq(lemma, "be") { return "ser" }
|
|
||||||
if str_eq(lemma, "give") { return "dar" }
|
|
||||||
if str_eq(lemma, "make") { return "hacer" }
|
|
||||||
if str_eq(lemma, "learn") { return "aprender" }
|
|
||||||
if str_eq(lemma, "form") { return "formar" }
|
|
||||||
return lemma
|
|
||||||
}
|
|
||||||
if str_eq(lang, "pt") {
|
|
||||||
if str_eq(lemma, "store") { return "armazenar" }
|
|
||||||
if str_eq(lemma, "use") { return "usar" }
|
|
||||||
if str_eq(lemma, "have") { return "ter" }
|
|
||||||
if str_eq(lemma, "be") { return "ser" }
|
|
||||||
if str_eq(lemma, "give") { return "dar" }
|
|
||||||
if str_eq(lemma, "make") { return "fazer" }
|
|
||||||
if str_eq(lemma, "learn") { return "aprender" }
|
|
||||||
if str_eq(lemma, "form") { return "formar" }
|
|
||||||
return lemma
|
|
||||||
}
|
|
||||||
if str_eq(lang, "it") {
|
|
||||||
if str_eq(lemma, "store") { return "memorizzare" }
|
|
||||||
if str_eq(lemma, "use") { return "usare" }
|
|
||||||
if str_eq(lemma, "have") { return "avere" }
|
|
||||||
if str_eq(lemma, "be") { return "essere" }
|
|
||||||
return lemma
|
|
||||||
}
|
|
||||||
return lemma
|
|
||||||
}
|
|
||||||
@@ -1,125 +0,0 @@
|
|||||||
// organ-read.el - Route the render's GEOMETRY READ through the ingest ORGAN's
|
|
||||||
// saved engram files (the coordinator's source of truth). For each file we
|
|
||||||
// engram_load() it, engram_scan_nodes_json(limit, offset) to get the node array,
|
|
||||||
// and cache each node's self-contained CONTENT string keyed by symbol. Because
|
|
||||||
// the cached value carries the numbers ("... f1=730 ..."), the cache SURVIVES the
|
|
||||||
// store being REPLACED by the next engram_load — so we load+cache phonetics
|
|
||||||
// FIRST, then load+cache accent. The .psv path remains a fallback.
|
|
||||||
//
|
|
||||||
// engram_scan_nodes_json(limit, offset) takes NO query; it returns nodes
|
|
||||||
// salience-sorted, so limit must be >= node count and we filter client-side.
|
|
||||||
// (engram_search / engram_scan_nodes return len-5 garbage — unused.)
|
|
||||||
|
|
||||||
// Find every occurrence of `marker` in the scan JSON; for each, cache
|
|
||||||
// sym -> a 150-char content window (enough to hold f1..amp). Duplicates from the
|
|
||||||
// node's "content" and "label" fields are harmless (first match wins on read).
|
|
||||||
fn organ_cache(j: String, marker: String, mlen: Int, win_len: Int, need: String) -> [String] {
|
|
||||||
let m: [String] = native_list_empty()
|
|
||||||
let jl: Int = str_len(j)
|
|
||||||
let off: Int = 0
|
|
||||||
while off < jl {
|
|
||||||
let rest: String = str_slice(j, off, jl)
|
|
||||||
let p: Int = str_index_of(rest, marker)
|
|
||||||
if p < 0 {
|
|
||||||
off = jl
|
|
||||||
} else {
|
|
||||||
let abs: Int = off + p
|
|
||||||
let win: String = str_slice(j, abs, abs + win_len)
|
|
||||||
let after: String = str_slice(win, mlen, str_len(win))
|
|
||||||
let sp: Int = str_index_of(after, " ")
|
|
||||||
let hasneed: Int = str_index_of(win, need)
|
|
||||||
if sp > 0 {
|
|
||||||
if hasneed >= 0 {
|
|
||||||
let sym: String = str_slice(after, 0, sp)
|
|
||||||
m = native_list_append(m, sym)
|
|
||||||
m = native_list_append(m, win)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
off = abs + mlen
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return m
|
|
||||||
}
|
|
||||||
|
|
||||||
// Load the phonetics organ file and cache sym -> content. mlen("phoneme ")=8.
|
|
||||||
fn organ_pmap(path: String) -> [String] {
|
|
||||||
let ok: Bool = engram_load(path)
|
|
||||||
if ok == false {
|
|
||||||
return native_list_empty()
|
|
||||||
}
|
|
||||||
let j: String = engram_scan_nodes_json(600, 0)
|
|
||||||
return organ_cache(j, "phoneme ", 8, 150, "f1=")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Load the accent organ file and cache sym -> content. mlen("accent_target ")=14.
|
|
||||||
// Vowel overrides carry f1=..; the R rule carries drop_coda_r (need="=" matches
|
|
||||||
// both, i.e. any well-formed accent_target field).
|
|
||||||
fn organ_amap(path: String) -> [String] {
|
|
||||||
let ok: Bool = engram_load(path)
|
|
||||||
if ok == false {
|
|
||||||
return native_list_empty()
|
|
||||||
}
|
|
||||||
let j: String = engram_scan_nodes_json(600, 0)
|
|
||||||
return organ_cache(j, "accent_target ", 14, 90, "=")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Vowel-set (categorical class) from the phonetics .psv class column.
|
|
||||||
fn organ_vset(path: String) -> [String] {
|
|
||||||
let content: String = fs_read(path)
|
|
||||||
let lines: [String] = str_split(content, "\n")
|
|
||||||
let nl: Int = native_list_len(lines)
|
|
||||||
let v: [String] = native_list_empty()
|
|
||||||
let li: Int = 0
|
|
||||||
while li < nl {
|
|
||||||
let line: String = native_list_get(lines, li)
|
|
||||||
let ok: Int = 1
|
|
||||||
if str_len(line) < 5 {
|
|
||||||
ok = 0
|
|
||||||
}
|
|
||||||
if ok == 1 {
|
|
||||||
if str_char_code(line, 0) == 35 {
|
|
||||||
ok = 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if ok == 1 {
|
|
||||||
let f: [String] = str_split(line, "|")
|
|
||||||
if native_list_len(f) >= 12 {
|
|
||||||
if str_eq(native_list_get(f, 11), "vowel") {
|
|
||||||
v = native_list_append(v, native_list_get(f, 0))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
li = li + 1
|
|
||||||
}
|
|
||||||
return v
|
|
||||||
}
|
|
||||||
|
|
||||||
// Word -> phoneme-sequence cache from lexicon.psv (engram-independent).
|
|
||||||
fn organ_lex(path: String) -> [String] {
|
|
||||||
let content: String = fs_read(path)
|
|
||||||
let lines: [String] = str_split(content, "\n")
|
|
||||||
let nl: Int = native_list_len(lines)
|
|
||||||
let m: [String] = native_list_empty()
|
|
||||||
let li: Int = 0
|
|
||||||
while li < nl {
|
|
||||||
let line: String = native_list_get(lines, li)
|
|
||||||
let ok: Int = 1
|
|
||||||
if str_len(line) < 3 {
|
|
||||||
ok = 0
|
|
||||||
}
|
|
||||||
if ok == 1 {
|
|
||||||
if str_char_code(line, 0) == 35 {
|
|
||||||
ok = 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if ok == 1 {
|
|
||||||
let f: [String] = str_split(line, "|")
|
|
||||||
if native_list_len(f) >= 2 {
|
|
||||||
m = native_list_append(m, native_list_get(f, 0))
|
|
||||||
m = native_list_append(m, native_list_get(f, 1))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
li = li + 1
|
|
||||||
}
|
|
||||||
return m
|
|
||||||
}
|
|
||||||
@@ -1,140 +0,0 @@
|
|||||||
// propositions.el - the READ primitive over the engram's OWN memories, native el.
|
|
||||||
//
|
|
||||||
// Free memory text -> structured PROPOSITIONS (triples):
|
|
||||||
// (subject, predicate, object, modifiers, polarity, tense, source, confidence)
|
|
||||||
//
|
|
||||||
// This is comprehension turned inward: the Python reference (propositions.py) ran
|
|
||||||
// spaCy's dependency parser over each memory sentence and walked the arcs. Here
|
|
||||||
// the spaCy role is filled by the el-native parser (comprehend.el / parse_spec):
|
|
||||||
// each sentence is parsed to a meaning-spec, and the spec's roles ARE the triple.
|
|
||||||
// Nothing generates text. NEGATION IS SACRED: polarity flows straight from the
|
|
||||||
// spec's polarity field and is never dropped or inverted.
|
|
||||||
//
|
|
||||||
// Depends on: comprehend (parse_spec / parse_spec_lang), grammar (slots_get).
|
|
||||||
|
|
||||||
// ── sentence segmentation ─────────────────────────────────────────────────────
|
|
||||||
// Split on sentence-final punctuation (. ! ?) and hard newlines. Markdown/long
|
|
||||||
// memories are handled shallowly (the reference caps + ranks by query overlap;
|
|
||||||
// that ranking belongs to the dialogue layer, not here).
|
|
||||||
|
|
||||||
fn prop_is_boundary(c: String) -> Bool {
|
|
||||||
if str_eq(c, ".") { return true }
|
|
||||||
if str_eq(c, "!") { return true }
|
|
||||||
if str_eq(c, "?") { return true }
|
|
||||||
if str_eq(c, "\n") { return true }
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
fn prop_split_sentences(text: String) -> [String] {
|
|
||||||
let out: [String] = native_list_empty()
|
|
||||||
let n: Int = str_len(text)
|
|
||||||
let start: Int = 0
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let c: String = str_slice(text, i, i + 1)
|
|
||||||
if prop_is_boundary(c) {
|
|
||||||
let seg: String = str_slice(text, start, i + 1)
|
|
||||||
let trimmed: String = cp_trim_punct(seg)
|
|
||||||
if !str_eq(trimmed, "") {
|
|
||||||
let out = native_list_append(out, seg)
|
|
||||||
}
|
|
||||||
let start = i + 1
|
|
||||||
}
|
|
||||||
let i = i + 1
|
|
||||||
}
|
|
||||||
if start < n {
|
|
||||||
let seg: String = str_slice(text, start, n)
|
|
||||||
let trimmed: String = cp_trim_punct(seg)
|
|
||||||
if !str_eq(trimmed, "") {
|
|
||||||
let out = native_list_append(out, seg)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── spec -> proposition record ────────────────────────────────────────────────
|
|
||||||
// A proposition is a slot map (same [String] shape as the spec) with the READ
|
|
||||||
// contract keys. Modifiers fold the spec's location + iobj adjuncts.
|
|
||||||
|
|
||||||
fn prop_confidence(subject: String, predicate: String, object: String) -> String {
|
|
||||||
if str_eq(predicate, "") { return "0.0" }
|
|
||||||
if str_eq(subject, "") { return "0.4" }
|
|
||||||
if str_eq(object, "") { return "0.7" }
|
|
||||||
return "1.0"
|
|
||||||
}
|
|
||||||
|
|
||||||
fn prop_modifiers(spec: [String]) -> String {
|
|
||||||
let loc: String = slots_get(spec, "location")
|
|
||||||
let iobj: String = slots_get(spec, "iobj")
|
|
||||||
let parts: [String] = native_list_empty()
|
|
||||||
if !str_eq(loc, "") { let parts = native_list_append(parts, loc) }
|
|
||||||
if !str_eq(iobj, "") { let parts = native_list_append(parts, "to " + iobj) }
|
|
||||||
return str_join(parts, "; ")
|
|
||||||
}
|
|
||||||
|
|
||||||
fn prop_from_spec(spec: [String], source_id: String) -> [String] {
|
|
||||||
let subject: String = slots_get(spec, "agent")
|
|
||||||
let predicate: String = slots_get(spec, "predicate")
|
|
||||||
let object: String = slots_get(spec, "patient")
|
|
||||||
let polarity: String = slots_get(spec, "polarity")
|
|
||||||
let tense: String = slots_get(spec, "tense")
|
|
||||||
let mods: String = prop_modifiers(spec)
|
|
||||||
let conf: String = prop_confidence(subject, predicate, object)
|
|
||||||
|
|
||||||
let p: [String] = native_list_empty()
|
|
||||||
let p = native_list_append(p, "subject"); let p = native_list_append(p, subject)
|
|
||||||
let p = native_list_append(p, "predicate"); let p = native_list_append(p, predicate)
|
|
||||||
let p = native_list_append(p, "object"); let p = native_list_append(p, object)
|
|
||||||
let p = native_list_append(p, "modifiers"); let p = native_list_append(p, mods)
|
|
||||||
let p = native_list_append(p, "polarity"); let p = native_list_append(p, polarity)
|
|
||||||
let p = native_list_append(p, "tense"); let p = native_list_append(p, tense)
|
|
||||||
let p = native_list_append(p, "source"); let p = native_list_append(p, source_id)
|
|
||||||
let p = native_list_append(p, "confidence"); let p = native_list_append(p, conf)
|
|
||||||
return p
|
|
||||||
}
|
|
||||||
|
|
||||||
// Extract one proposition from a single sentence (given language).
|
|
||||||
fn prop_extract_one_lang(sentence: String, lang: String, source_id: String) -> [String] {
|
|
||||||
let spec: [String] = parse_spec_lang(sentence, lang)
|
|
||||||
return prop_from_spec(spec, source_id)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn prop_extract_one(sentence: String, source_id: String) -> [String] {
|
|
||||||
return prop_extract_one_lang(sentence, "en", source_id)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Render a proposition as a compact trace line (repr parity with propositions.py).
|
|
||||||
fn prop_repr(p: [String]) -> String {
|
|
||||||
let neg: String = ""
|
|
||||||
if str_eq(slots_get(p, "polarity"), "neg") { let neg = "NOT " }
|
|
||||||
let mods: String = slots_get(p, "modifiers")
|
|
||||||
let modstr: String = ""
|
|
||||||
if !str_eq(mods, "") { let modstr = " [" + mods + "]" }
|
|
||||||
let s: String = "(" + slots_get(p, "subject") + " -" + neg + slots_get(p, "predicate")
|
|
||||||
let s = s + "-> " + slots_get(p, "object") + modstr
|
|
||||||
let s = s + " conf=" + slots_get(p, "confidence") + ")"
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
|
|
||||||
// Extract all propositions from a memory's text (one per sentence). Returns a
|
|
||||||
// flat [String] whose entries are the prop_repr trace lines, in reading order.
|
|
||||||
fn prop_extract_lang(text: String, lang: String, source_id: String) -> [String] {
|
|
||||||
let sents: [String] = prop_split_sentences(text)
|
|
||||||
let m: Int = native_list_len(sents)
|
|
||||||
let out: [String] = native_list_empty()
|
|
||||||
let i: Int = 0
|
|
||||||
while i < m {
|
|
||||||
let sent: String = native_list_get(sents, i)
|
|
||||||
let p: [String] = prop_extract_one_lang(sent, lang, source_id)
|
|
||||||
// drop empty parses (no predicate recovered): honest partial, not noise.
|
|
||||||
if !str_eq(slots_get(p, "predicate"), "") {
|
|
||||||
let out = native_list_append(out, prop_repr(p))
|
|
||||||
}
|
|
||||||
let i = i + 1
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
fn prop_extract(text: String, source_id: String) -> [String] {
|
|
||||||
return prop_extract_lang(text, "en", source_id)
|
|
||||||
}
|
|
||||||
@@ -34,13 +34,6 @@ fn agent_person(agent: String) -> String {
|
|||||||
if str_eq(agent, "we") { return "first" }
|
if str_eq(agent, "we") { return "first" }
|
||||||
if str_eq(agent, "us") { return "first" }
|
if str_eq(agent, "us") { return "first" }
|
||||||
if str_eq(agent, "you") { return "second" }
|
if str_eq(agent, "you") { return "second" }
|
||||||
// Romance target-language subject pronouns (translate.el sets these).
|
|
||||||
if str_eq(agent, "yo") { return "first" }
|
|
||||||
if str_eq(agent, "eu") { return "first" }
|
|
||||||
if str_eq(agent, "nosotros") { return "first" }
|
|
||||||
if str_eq(agent, "nós") { return "first" }
|
|
||||||
if str_eq(agent, "tú") { return "second" }
|
|
||||||
if str_eq(agent, "tu") { return "second" }
|
|
||||||
return "third"
|
return "third"
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -57,19 +50,6 @@ fn agent_number(agent: String) -> String {
|
|||||||
if str_eq(agent, "us") { return "plural" }
|
if str_eq(agent, "us") { return "plural" }
|
||||||
if str_eq(agent, "they") { return "plural" }
|
if str_eq(agent, "they") { return "plural" }
|
||||||
if str_eq(agent, "them") { return "plural" }
|
if str_eq(agent, "them") { return "plural" }
|
||||||
// Romance target-language subject pronouns.
|
|
||||||
if str_eq(agent, "yo") { return "singular" }
|
|
||||||
if str_eq(agent, "eu") { return "singular" }
|
|
||||||
if str_eq(agent, "tú") { return "singular" }
|
|
||||||
if str_eq(agent, "tu") { return "singular" }
|
|
||||||
if str_eq(agent, "él") { return "singular" }
|
|
||||||
if str_eq(agent, "ella") { return "singular" }
|
|
||||||
if str_eq(agent, "ele") { return "singular" }
|
|
||||||
if str_eq(agent, "ela") { return "singular" }
|
|
||||||
if str_eq(agent, "nosotros") { return "plural" }
|
|
||||||
if str_eq(agent, "nós") { return "plural" }
|
|
||||||
if str_eq(agent, "ellos") { return "plural" }
|
|
||||||
if str_eq(agent, "eles") { return "plural" }
|
|
||||||
return "singular"
|
return "singular"
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -268,56 +248,6 @@ fn add_punct(s: String, intent: String) -> String {
|
|||||||
return s + "."
|
return s + "."
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── Polarity-aware negation (SACRED field honored on the generation side) ─────
|
|
||||||
//
|
|
||||||
// Negation must never be dropped between comprehension and realization. The
|
|
||||||
// meaning-spec carries an explicit "polarity" field ("aff"|"neg") and optional
|
|
||||||
// "neg_word" (standalone negative adverb, e.g. "never"). English uses
|
|
||||||
// do-support ("did not see") or preverbal adverb ("never fought"); copular "be"
|
|
||||||
// takes post-verbal "not"; other languages get a preverbal negator particle.
|
|
||||||
|
|
||||||
fn realize_negator(code: String) -> String {
|
|
||||||
if str_eq(code, "es") { return "no" }
|
|
||||||
if str_eq(code, "pt") { return "não" }
|
|
||||||
if str_eq(code, "ca") { return "no" }
|
|
||||||
if str_eq(code, "it") { return "non" }
|
|
||||||
if str_eq(code, "fr") { return "ne" }
|
|
||||||
if str_eq(code, "de") { return "nicht" }
|
|
||||||
if str_eq(code, "ro") { return "nu" }
|
|
||||||
return "not"
|
|
||||||
}
|
|
||||||
|
|
||||||
fn realize_assert_neg_en(predicate: String, tense: String, person: String, number: String, agent: String, patient: String, iobj: String, location: String, neg_word: String, profile: [String]) -> String {
|
|
||||||
let parts: [String] = native_list_empty()
|
|
||||||
let parts = native_list_append(parts, agent)
|
|
||||||
if !str_eq(neg_word, "") {
|
|
||||||
// adverbial negation: "I never fought the ocean."
|
|
||||||
let verb_surf: String = morph_conjugate(predicate, tense, person, number, profile)
|
|
||||||
let parts = native_list_append(parts, neg_word)
|
|
||||||
let parts = native_list_append(parts, verb_surf)
|
|
||||||
} else {
|
|
||||||
if str_eq(predicate, "be") {
|
|
||||||
// copular: "she was not a monster"
|
|
||||||
let be_form: String = morph_conjugate("be", tense, person, number, profile)
|
|
||||||
let parts = native_list_append(parts, be_form)
|
|
||||||
let parts = native_list_append(parts, "not")
|
|
||||||
} else {
|
|
||||||
// do-support: "she did not see the man"
|
|
||||||
let do_form: String = morph_conjugate("do", tense, person, number, profile)
|
|
||||||
let parts = native_list_append(parts, do_form)
|
|
||||||
let parts = native_list_append(parts, "not")
|
|
||||||
let parts = native_list_append(parts, predicate)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if !str_eq(patient, "") { let parts = native_list_append(parts, patient) }
|
|
||||||
if !str_eq(iobj, "") {
|
|
||||||
let parts = native_list_append(parts, "to")
|
|
||||||
let parts = native_list_append(parts, iobj)
|
|
||||||
}
|
|
||||||
if !str_eq(location, "") { let parts = native_list_append(parts, location) }
|
|
||||||
return str_join(parts, " ")
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── Main realization entry point ──────────────────────────────────────────────
|
// ── Main realization entry point ──────────────────────────────────────────────
|
||||||
|
|
||||||
fn realize_lang(form: [String], profile: [String]) -> String {
|
fn realize_lang(form: [String], profile: [String]) -> String {
|
||||||
@@ -354,54 +284,6 @@ fn realize_lang(form: [String], profile: [String]) -> String {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ── Assertion (declarative) ───────────────────────────────────────────────
|
// ── Assertion (declarative) ───────────────────────────────────────────────
|
||||||
let polarity: String = slots_get(form, "polarity")
|
|
||||||
let neg_word: String = slots_get(form, "neg_word")
|
|
||||||
let iobj: String = slots_get(form, "iobj")
|
|
||||||
let code: String = lang_get(profile, "code")
|
|
||||||
|
|
||||||
// Subordinate clause tail (SACRED completeness — the clause is carried, never
|
|
||||||
// dropped): "<conj> <subordinate surface>", e.g. "because he was a monster".
|
|
||||||
let subord_conj: String = slots_get(form, "subord_conj")
|
|
||||||
let subord_text: String = slots_get(form, "subord_text")
|
|
||||||
let subord_tail: String = ""
|
|
||||||
if !str_eq(subord_conj, "") {
|
|
||||||
if !str_eq(subord_text, "") {
|
|
||||||
let subord_tail = subord_conj + " " + subord_text
|
|
||||||
} else {
|
|
||||||
let subord_tail = subord_conj
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Negative polarity: SACRED — never dropped.
|
|
||||||
if str_eq(polarity, "neg") {
|
|
||||||
if str_eq(code, "en") {
|
|
||||||
let sentence: String = realize_assert_neg_en(predicate, tense, person, number, agent, patient, iobj, location, neg_word, profile)
|
|
||||||
return add_punct(capitalize_first(sentence), "assert")
|
|
||||||
}
|
|
||||||
// Generic non-English: affirmative core with a preverbal negator particle.
|
|
||||||
// SACRED: when a standalone negative adverb was carried (e.g. "nunca",
|
|
||||||
// localized upstream from "never"), surface it rather than the generic
|
|
||||||
// negator — the specific negation must never be flattened away.
|
|
||||||
let neg_particle: String = realize_negator(code)
|
|
||||||
if !str_eq(neg_word, "") { let neg_particle = neg_word }
|
|
||||||
let vp_pair: [String] = realize_vp_lang(predicate, tense, aspect, person, number, profile)
|
|
||||||
let verb_surf: String = native_list_get(vp_pair, 0)
|
|
||||||
let aux_surf: String = native_list_get(vp_pair, 1)
|
|
||||||
let vp_str: String = neg_particle + " " + gram_build_vp(verb_surf, aux_surf, profile)
|
|
||||||
let core: String = gram_order_constituents(agent, vp_str, patient, profile)
|
|
||||||
let parts: [String] = native_list_empty()
|
|
||||||
let parts = native_list_append(parts, core)
|
|
||||||
if !str_eq(iobj, "") {
|
|
||||||
let parts = native_list_append(parts, "to")
|
|
||||||
let parts = native_list_append(parts, iobj)
|
|
||||||
}
|
|
||||||
if !str_eq(location, "") { let parts = native_list_append(parts, location) }
|
|
||||||
if !str_eq(subord_tail, "") { let parts = native_list_append(parts, subord_tail) }
|
|
||||||
let sentence: String = str_join(parts, " ")
|
|
||||||
return add_punct(capitalize_first(sentence), "assert")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Affirmative.
|
|
||||||
let vp_pair: [String] = realize_vp_lang(predicate, tense, aspect, person, number, profile)
|
let vp_pair: [String] = realize_vp_lang(predicate, tense, aspect, person, number, profile)
|
||||||
let verb_surf: String = native_list_get(vp_pair, 0)
|
let verb_surf: String = native_list_get(vp_pair, 0)
|
||||||
let aux_surf: String = native_list_get(vp_pair, 1)
|
let aux_surf: String = native_list_get(vp_pair, 1)
|
||||||
@@ -411,16 +293,9 @@ fn realize_lang(form: [String], profile: [String]) -> String {
|
|||||||
|
|
||||||
let parts: [String] = native_list_empty()
|
let parts: [String] = native_list_empty()
|
||||||
let parts = native_list_append(parts, core)
|
let parts = native_list_append(parts, core)
|
||||||
if !str_eq(iobj, "") {
|
|
||||||
let parts = native_list_append(parts, "to")
|
|
||||||
let parts = native_list_append(parts, iobj)
|
|
||||||
}
|
|
||||||
if !str_eq(location, "") {
|
if !str_eq(location, "") {
|
||||||
let parts = native_list_append(parts, location)
|
let parts = native_list_append(parts, location)
|
||||||
}
|
}
|
||||||
if !str_eq(subord_tail, "") {
|
|
||||||
let parts = native_list_append(parts, subord_tail)
|
|
||||||
}
|
|
||||||
let sentence: String = str_join(parts, " ")
|
let sentence: String = str_join(parts, " ")
|
||||||
return add_punct(capitalize_first(sentence), "assert")
|
return add_punct(capitalize_first(sentence), "assert")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,180 +0,0 @@
|
|||||||
// self_region.el — the engram's REAL self/identity region, pulled at query time
|
|
||||||
// (native el). This replaces the hardcoded identity anchors and the canned
|
|
||||||
// "I'm Neuron, the engram you're speaking with." template: the identity LANDING
|
|
||||||
// signal and the identity READOUT both come from the engram's own Self/identity
|
|
||||||
// nodes, read through the in-process engram el API.
|
|
||||||
//
|
|
||||||
// Port of self_region.py. The Python module precomputed MiniLM landing vectors;
|
|
||||||
// here the engram's own store IS the geometry — we pull the self nodes by
|
|
||||||
// single-term lexical search (the engram search is a single-term matcher, so we
|
|
||||||
// pool several probes) and rank them by self-signal. No text is generated; the
|
|
||||||
// readout is the self nodes' OWN prose, verbatim (SACRED negation survives by
|
|
||||||
// construction — we never paraphrase, so a negated self-statement stays negated).
|
|
||||||
//
|
|
||||||
// ENGRAM el API NOTE: engram_search_json / engram_get_node_json / engram_node_full
|
|
||||||
// / engram_connect are C runtime builtins. Their argument order is the C order
|
|
||||||
// (engram_connect(from, to, weight, relation)), NOT the runtime/engram.el wrapper
|
|
||||||
// order — we call the builtins directly and never concatenate that wrapper.
|
|
||||||
//
|
|
||||||
// Depends on: comprehend (str helpers via runtime), propositions (prop_split_sentences),
|
|
||||||
// multilingual (ml_tr), the engram builtins, the json builtins.
|
|
||||||
|
|
||||||
// ── single-term self probes (pooled, because engram search is single-term) ────
|
|
||||||
fn sr_terms() -> [String] {
|
|
||||||
let t: [String] = native_list_empty()
|
|
||||||
let t = native_list_append(t, "self")
|
|
||||||
let t = native_list_append(t, "identity")
|
|
||||||
let t = native_list_append(t, "Neuron")
|
|
||||||
let t = native_list_append(t, "consciousness")
|
|
||||||
let t = native_list_append(t, "values")
|
|
||||||
let t = native_list_append(t, "continuous")
|
|
||||||
return t
|
|
||||||
}
|
|
||||||
|
|
||||||
// The canonical self-root: content begins "# self" or label is "# self"/"self".
|
|
||||||
fn sr_is_root(content: String, label: String) -> Bool {
|
|
||||||
let lc: String = str_to_lower(content)
|
|
||||||
let ll: String = str_to_lower(str_trim(label))
|
|
||||||
if str_starts_with(lc, "# self") { return true }
|
|
||||||
if str_eq(ll, "# self") { return true }
|
|
||||||
if str_eq(ll, "self") { return true }
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
// How strongly a node belongs to the self/identity region (integer points, to
|
|
||||||
// avoid el's float-in-`+` pitfalls). Mirrors _self_score in self_region.py.
|
|
||||||
fn sr_score(node_json: String) -> Int {
|
|
||||||
let content: String = json_get_string(node_json, "content")
|
|
||||||
let label: String = json_get_string(node_json, "label")
|
|
||||||
let tags: String = str_to_lower(json_get_string(node_json, "tags"))
|
|
||||||
let low: String = str_to_lower(content)
|
|
||||||
let s: Int = 0
|
|
||||||
// identity tags
|
|
||||||
if str_contains(tags, "self") { let s = s + 2 }
|
|
||||||
if str_contains(tags, "identity") { let s = s + 2 }
|
|
||||||
if str_contains(tags, "self-model") { let s = s + 2 }
|
|
||||||
if str_contains(tags, "consciousness") { let s = s + 2 }
|
|
||||||
if str_contains(tags, "memory-philosophy") { let s = s + 2 }
|
|
||||||
// the named self-traversal root
|
|
||||||
if sr_is_root(content, label) { let s = s + 12 }
|
|
||||||
if str_contains(low, "who i am") { let s = s + 3 }
|
|
||||||
if str_contains(low, "i am neuron") { let s = s + 3 }
|
|
||||||
// softer identity keywords
|
|
||||||
if str_contains(low, "my values") { let s = s + 1 }
|
|
||||||
if str_contains(low, "my purpose") { let s = s + 1 }
|
|
||||||
if str_contains(low, "identity") { let s = s + 1 }
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
|
|
||||||
// list-contains helper (dedup self-node ids across the pooled probes).
|
|
||||||
fn sr_ids_has(ids: [String], id: String) -> Bool {
|
|
||||||
let n: Int = native_list_len(ids)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
if str_eq(native_list_get(ids, i), id) { return true }
|
|
||||||
let i = i + 1
|
|
||||||
}
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
// Pull the self nodes: pool every probe's hits, dedupe by id, keep only nodes
|
|
||||||
// with genuine self-signal (score >= 1). Returns the node-json strings.
|
|
||||||
fn sr_pull() -> [String] {
|
|
||||||
let terms: [String] = sr_terms()
|
|
||||||
let nt: Int = native_list_len(terms)
|
|
||||||
let seen: [String] = native_list_empty()
|
|
||||||
let out: [String] = native_list_empty()
|
|
||||||
let ti: Int = 0
|
|
||||||
while ti < nt {
|
|
||||||
let term: String = native_list_get(terms, ti)
|
|
||||||
let hits: String = engram_search_json(term, 30)
|
|
||||||
let hn: Int = json_array_len(hits)
|
|
||||||
let hi: Int = 0
|
|
||||||
while hi < hn {
|
|
||||||
let node: String = json_array_get(hits, hi)
|
|
||||||
let id: String = json_get_string(node, "id")
|
|
||||||
if !str_eq(id, "") {
|
|
||||||
if !sr_ids_has(seen, id) {
|
|
||||||
let seen = native_list_append(seen, id)
|
|
||||||
if sr_score(node) >= 1 {
|
|
||||||
let out = native_list_append(out, node)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let hi = hi + 1
|
|
||||||
}
|
|
||||||
let ti = ti + 1
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// Return the single highest-signal self node (the readout seed), or "" if the
|
|
||||||
// self region is thin/empty. We keep it O(n) — pick the max-score node, with the
|
|
||||||
// canonical root strongly favored by sr_score's +12.
|
|
||||||
fn sr_best_node() -> String {
|
|
||||||
let nodes: [String] = sr_pull()
|
|
||||||
let n: Int = native_list_len(nodes)
|
|
||||||
let best: String = ""
|
|
||||||
let best_s: Int = 0
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let node: String = native_list_get(nodes, i)
|
|
||||||
let s: Int = sr_score(node)
|
|
||||||
if s > best_s {
|
|
||||||
let best_s = s
|
|
||||||
let best = node
|
|
||||||
}
|
|
||||||
let i = i + 1
|
|
||||||
}
|
|
||||||
return best
|
|
||||||
}
|
|
||||||
|
|
||||||
fn sr_available() -> Bool {
|
|
||||||
if str_eq(sr_best_node(), "") { return false }
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
// Read out the identity from the REAL self node: lead with the first first-person
|
|
||||||
// self-statement ("I am Neuron …"), then one more grounded self line if present.
|
|
||||||
// Verbatim from the node's own prose — no template, negation SACRED. Falls back
|
|
||||||
// to the localized identity phrase ONLY if the live pull is empty (logged shape).
|
|
||||||
fn sr_readout(lang: String) -> String {
|
|
||||||
let node: String = sr_best_node()
|
|
||||||
if str_eq(node, "") {
|
|
||||||
// honest fallback — the self region is unreachable/thin.
|
|
||||||
return ml_tr("identity", lang)
|
|
||||||
}
|
|
||||||
let content: String = json_get_string(node, "content")
|
|
||||||
let sents: [String] = prop_split_sentences(content)
|
|
||||||
let ns: Int = native_list_len(sents)
|
|
||||||
let lead: String = ""
|
|
||||||
let second: String = ""
|
|
||||||
let i: Int = 0
|
|
||||||
while i < ns {
|
|
||||||
let raw: String = str_trim(native_list_get(sents, i))
|
|
||||||
// strip a leading markdown heading marker
|
|
||||||
let s: String = raw
|
|
||||||
if str_starts_with(s, "# ") { let s = str_trim(str_slice(s, 2, str_len(s))) }
|
|
||||||
let low: String = str_to_lower(s)
|
|
||||||
let is_fp: Bool = false
|
|
||||||
if str_starts_with(s, "I ") { let is_fp = true }
|
|
||||||
if str_starts_with(s, "I'm") { let is_fp = true }
|
|
||||||
if str_contains(low, "i am neuron") { let is_fp = true }
|
|
||||||
if is_fp {
|
|
||||||
if str_eq(lead, "") {
|
|
||||||
let lead = s
|
|
||||||
} else {
|
|
||||||
if str_eq(second, "") { let second = s }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let i = i + 1
|
|
||||||
}
|
|
||||||
if str_eq(lead, "") {
|
|
||||||
// no first-person line — read out the first non-empty sentence verbatim.
|
|
||||||
if ns > 0 { let lead = str_trim(native_list_get(sents, 0)) }
|
|
||||||
}
|
|
||||||
if str_eq(lead, "") { return ml_tr("identity", lang) }
|
|
||||||
let out: String = lead
|
|
||||||
if !str_eq(second, "") { let out = out + " " + second }
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
@@ -1,233 +0,0 @@
|
|||||||
// speech-ingest.el - The native LOAD step of the ingest organ, for the SPEECH
|
|
||||||
// primitives. Reads the acoustic-phonetics SOURCE (elp/data/phonetics.psv) and
|
|
||||||
// the pronunciation lexicon SOURCE (elp/data/lexicon.psv) and emits a PHONEME
|
|
||||||
// MANIFOLD into the engram: one node per phoneme (faithful, provenance-tagged
|
|
||||||
// content) + is_a edges to phoneme-class nodes (a discrete manifold, not islands).
|
|
||||||
// The render then PULLS phoneme geometry back from the engram via phon_geo —
|
|
||||||
// zero phonetic numbers in code. Source -> manifold -> merge; the same output
|
|
||||||
// the polymorphic ingest organ will produce and subsume.
|
|
||||||
|
|
||||||
// -- small parsing helpers ---------------------------------------------------
|
|
||||||
fn sp_map_get(pairs: [String], key: String) -> String {
|
|
||||||
let n: Int = native_list_len(pairs)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n - 1 {
|
|
||||||
let k: String = native_list_get(pairs, i)
|
|
||||||
if str_eq(k, key) {
|
|
||||||
return native_list_get(pairs, i + 1)
|
|
||||||
}
|
|
||||||
let i = i + 2
|
|
||||||
}
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
// read the unsigned integer that follows `key` inside string s (e.g. key "F1=")
|
|
||||||
fn parse_uint_from(s: String, key: String) -> Int {
|
|
||||||
let idx: Int = str_index_of(s, key)
|
|
||||||
if idx < 0 {
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
let start: Int = idx + str_len(key)
|
|
||||||
let n: Int = str_len(s)
|
|
||||||
let i: Int = start
|
|
||||||
let val: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let c: Int = str_char_code(s, i)
|
|
||||||
if c >= 48 {
|
|
||||||
if c <= 57 {
|
|
||||||
val = val * 10 + (c - 48)
|
|
||||||
i = i + 1
|
|
||||||
} else {
|
|
||||||
i = n
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
i = n
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return val
|
|
||||||
}
|
|
||||||
|
|
||||||
fn clean_word(w: String) -> String {
|
|
||||||
let low: String = str_to_lower(w)
|
|
||||||
let n: Int = str_len(low)
|
|
||||||
let out: String = ""
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let c: Int = str_char_code(low, i)
|
|
||||||
if c >= 97 {
|
|
||||||
if c <= 122 {
|
|
||||||
out = out + str_char_at(low, i)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
i = i + 1
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- INGEST: acoustic-phonetics source -> phoneme manifold in the engram ------
|
|
||||||
// Returns the symbol -> node-id index (pmap) the render reads geometry through.
|
|
||||||
fn ingest_phonetics(path: String) -> [String] {
|
|
||||||
let content: String = fs_read(path)
|
|
||||||
let lines: [String] = str_split(content, "\n")
|
|
||||||
let nl: Int = native_list_len(lines)
|
|
||||||
let pmap: [String] = native_list_empty()
|
|
||||||
let classmap: [String] = native_list_empty()
|
|
||||||
let li: Int = 0
|
|
||||||
while li < nl {
|
|
||||||
let line: String = native_list_get(lines, li)
|
|
||||||
let ll: Int = str_len(line)
|
|
||||||
let skip: Int = 0
|
|
||||||
if ll < 5 {
|
|
||||||
skip = 1
|
|
||||||
}
|
|
||||||
if skip == 0 {
|
|
||||||
let first: Int = str_char_code(line, 0)
|
|
||||||
if first == 35 {
|
|
||||||
skip = 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if skip == 0 {
|
|
||||||
let f: [String] = str_split(line, "|")
|
|
||||||
let nf: Int = native_list_len(f)
|
|
||||||
if nf >= 12 {
|
|
||||||
let sym: String = native_list_get(f, 0)
|
|
||||||
let f1: String = native_list_get(f, 1)
|
|
||||||
let f2: String = native_list_get(f, 2)
|
|
||||||
let f3: String = native_list_get(f, 3)
|
|
||||||
let b1: String = native_list_get(f, 4)
|
|
||||||
let b2: String = native_list_get(f, 5)
|
|
||||||
let b3: String = native_list_get(f, 6)
|
|
||||||
let vo: String = native_list_get(f, 7)
|
|
||||||
let na: String = native_list_get(f, 8)
|
|
||||||
let du: String = native_list_get(f, 9)
|
|
||||||
let am: String = native_list_get(f, 10)
|
|
||||||
let cls: String = native_list_get(f, 11)
|
|
||||||
let cont: String = "phoneme " + sym + " | f1=" + f1 + " f2=" + f2 + " f3=" + f3 + " bw1=" + b1 + " bw2=" + b2 + " bw3=" + b3 + " voiced=" + vo + " nasal=" + na + " dur=" + du + " amp=" + am + " class=" + cls + " src=PetersonBarney1952-Hillenbrand1995"
|
|
||||||
let id: String = engram_node(cont, "Phoneme", 80)
|
|
||||||
pmap = native_list_append(pmap, sym)
|
|
||||||
pmap = native_list_append(pmap, cont)
|
|
||||||
// manifold edge: phoneme is_a class
|
|
||||||
let cid: String = sp_map_get(classmap, cls)
|
|
||||||
if str_eq(cid, "") {
|
|
||||||
cid = engram_node("phoneme-class " + cls + " src=acoustic-phonetics", "PhonemeClass", 80)
|
|
||||||
classmap = native_list_append(classmap, cls)
|
|
||||||
classmap = native_list_append(classmap, cid)
|
|
||||||
}
|
|
||||||
engram_connect(id, cid, 80, "is_a")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
li = li + 1
|
|
||||||
}
|
|
||||||
return pmap
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- INGEST: pronunciation lexicon source -> word nodes ----------------------
|
|
||||||
fn ingest_lexicon(path: String) -> [String] {
|
|
||||||
let content: String = fs_read(path)
|
|
||||||
let lines: [String] = str_split(content, "\n")
|
|
||||||
let nl: Int = native_list_len(lines)
|
|
||||||
let lmap: [String] = native_list_empty()
|
|
||||||
let li: Int = 0
|
|
||||||
while li < nl {
|
|
||||||
let line: String = native_list_get(lines, li)
|
|
||||||
let ll: Int = str_len(line)
|
|
||||||
let skip: Int = 0
|
|
||||||
if ll < 3 {
|
|
||||||
skip = 1
|
|
||||||
}
|
|
||||||
if skip == 0 {
|
|
||||||
let first: Int = str_char_code(line, 0)
|
|
||||||
if first == 35 {
|
|
||||||
skip = 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if skip == 0 {
|
|
||||||
let f: [String] = str_split(line, "|")
|
|
||||||
let nf: Int = native_list_len(f)
|
|
||||||
if nf >= 2 {
|
|
||||||
let word: String = native_list_get(f, 0)
|
|
||||||
let seq: String = native_list_get(f, 1)
|
|
||||||
let id: String = engram_node("word " + word + " phonemes " + seq + " src=lexicon", "Pronunciation", 80)
|
|
||||||
lmap = native_list_append(lmap, word)
|
|
||||||
lmap = native_list_append(lmap, seq)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
li = li + 1
|
|
||||||
}
|
|
||||||
return lmap
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- READ geometry back from the engram (the render's afferent lookup) --------
|
|
||||||
// phon_geo(sym) -> [F1,F2,F3,B1,B2,B3,voiced,nasal,dur,amp], parsed from the
|
|
||||||
// ingested phoneme node's content. NO formant numbers live in this code.
|
|
||||||
fn phon_geo(pmap: [String], sym: String) -> [Int] {
|
|
||||||
let id: String = sp_map_get(pmap, sym)
|
|
||||||
if str_eq(id, "") {
|
|
||||||
id = sp_map_get(pmap, "AX")
|
|
||||||
}
|
|
||||||
let out: [Int] = native_list_empty()
|
|
||||||
if str_eq(id, "") {
|
|
||||||
let out = native_list_append(out, 500)
|
|
||||||
let out = native_list_append(out, 1500)
|
|
||||||
let out = native_list_append(out, 2500)
|
|
||||||
let out = native_list_append(out, 80)
|
|
||||||
let out = native_list_append(out, 100)
|
|
||||||
let out = native_list_append(out, 150)
|
|
||||||
let out = native_list_append(out, 1)
|
|
||||||
let out = native_list_append(out, 0)
|
|
||||||
let out = native_list_append(out, 80)
|
|
||||||
let out = native_list_append(out, 80)
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
let j: String = id
|
|
||||||
let out = native_list_append(out, parse_uint_from(j, "f1="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(j, "f2="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(j, "f3="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(j, "bw1="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(j, "bw2="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(j, "bw3="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(j, "voiced="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(j, "nasal="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(j, "dur="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(j, "amp="))
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// word -> phoneme codes, read from the ingested lexicon node.
|
|
||||||
fn word_phonemes(lmap: [String], word: String) -> [String] {
|
|
||||||
let id: String = sp_map_get(lmap, word)
|
|
||||||
if str_eq(id, "") {
|
|
||||||
let r: [String] = native_list_empty()
|
|
||||||
let r = native_list_append(r, "AX")
|
|
||||||
return r
|
|
||||||
}
|
|
||||||
return str_split(id, " ")
|
|
||||||
}
|
|
||||||
|
|
||||||
// realized text -> flat phoneme-code sequence (SIL between words + at ends).
|
|
||||||
fn text_phonemes(lmap: [String], text: String) -> [String] {
|
|
||||||
let words: [String] = str_split(text, " ")
|
|
||||||
let nw: Int = native_list_len(words)
|
|
||||||
let seq: [String] = native_list_empty()
|
|
||||||
let seq = native_list_append(seq, "SIL")
|
|
||||||
let wi: Int = 0
|
|
||||||
while wi < nw {
|
|
||||||
let raw: String = native_list_get(words, wi)
|
|
||||||
let w: String = clean_word(raw)
|
|
||||||
if str_eq(w, "") {
|
|
||||||
wi = wi + 1
|
|
||||||
} else {
|
|
||||||
let ph: [String] = word_phonemes(lmap, w)
|
|
||||||
let np: Int = native_list_len(ph)
|
|
||||||
let pi: Int = 0
|
|
||||||
while pi < np {
|
|
||||||
let code: String = native_list_get(ph, pi)
|
|
||||||
seq = native_list_append(seq, code)
|
|
||||||
pi = pi + 1
|
|
||||||
}
|
|
||||||
seq = native_list_append(seq, "SIL")
|
|
||||||
wi = wi + 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return seq
|
|
||||||
}
|
|
||||||
@@ -1,460 +0,0 @@
|
|||||||
// speech.el - The native SPEECH render path + voice-by-imitation extractor.
|
|
||||||
//
|
|
||||||
// Speech = the AUDIO surface (surface_profile_audio) rendering LANGUAGE-meaning
|
|
||||||
// through a VOICE signature. The realizer's language faculty supplies the words
|
|
||||||
// (meaning -> sem_realize -> text); this module turns text -> phonemes (phonetics.el)
|
|
||||||
// -> a formant-target track over time -> SUPERPOSES formant resonances over a
|
|
||||||
// glottal source (own-core formant synthesis, the exact integer mirror of the
|
|
||||||
// music additive superpose) -> own-core PCM/WAV. Two paths:
|
|
||||||
// (1) RENDER: speak(text, voice) -> spoken WAV.
|
|
||||||
// (2) IMITATE: voice_analyze(pcm) -> a voice signature grabbed BY EAR
|
|
||||||
// (autocorrelation pitch + integer-DFT formant peaks), then render
|
|
||||||
// any new meaning in that voice. An impression, not a corpus.
|
|
||||||
// All integer/fixed-point (EL float arithmetic is unusable).
|
|
||||||
|
|
||||||
// -- Own-core integer sine (Bhaskara I), phase 0..65535 = one cycle -----------
|
|
||||||
fn sp_sin(phase: Int) -> Int {
|
|
||||||
let deg: Int = phase * 360 / 65536
|
|
||||||
let neg: Int = 0
|
|
||||||
if deg > 180 {
|
|
||||||
deg = deg - 180
|
|
||||||
neg = 1
|
|
||||||
}
|
|
||||||
let t: Int = deg * (180 - deg)
|
|
||||||
let num: Int = 32767 * 4 * t
|
|
||||||
let den: Int = 40500 - t
|
|
||||||
let v: Int = num / den
|
|
||||||
if neg == 1 {
|
|
||||||
v = 0 - v
|
|
||||||
}
|
|
||||||
return v
|
|
||||||
}
|
|
||||||
|
|
||||||
fn sp_cos(phase: Int) -> Int {
|
|
||||||
let p: Int = phase + 16384
|
|
||||||
p = p - (p / 65536) * 65536
|
|
||||||
return sp_sin(p)
|
|
||||||
}
|
|
||||||
|
|
||||||
// One formant resonance (Lorentzian peak), Q15. Peak 32767 at f=fc.
|
|
||||||
fn sp_gain(f: Int, fc: Int, bw: Int) -> Int {
|
|
||||||
let d: Int = f - fc
|
|
||||||
let den: Int = d * d + bw * bw
|
|
||||||
let num: Int = 32767 * bw * bw
|
|
||||||
return num / den
|
|
||||||
}
|
|
||||||
|
|
||||||
fn sp_isqrt(n: Int) -> Int {
|
|
||||||
if n <= 0 {
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
let x: Int = n
|
|
||||||
let y: Int = (x + 1) / 2
|
|
||||||
while y < x {
|
|
||||||
x = y
|
|
||||||
y = (x + n / x) / 2
|
|
||||||
}
|
|
||||||
return x
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- WAV serializer (thin medium; the only non-DSP glue) ---------------------
|
|
||||||
fn wav_le16(buf: String, off: Int, v: Int) -> String {
|
|
||||||
let u: Int = v
|
|
||||||
if u < 0 {
|
|
||||||
u = u + 65536
|
|
||||||
}
|
|
||||||
let lo: Int = u - (u / 256) * 256
|
|
||||||
let hi: Int = u / 256
|
|
||||||
let b: String = __str_set_char(buf, off, lo)
|
|
||||||
b = __str_set_char(b, off + 1, hi)
|
|
||||||
return b
|
|
||||||
}
|
|
||||||
|
|
||||||
fn wav_le32(buf: String, off: Int, v: Int) -> String {
|
|
||||||
let b0: Int = v - (v / 256) * 256
|
|
||||||
let r1: Int = v / 256
|
|
||||||
let b1: Int = r1 - (r1 / 256) * 256
|
|
||||||
let r2: Int = r1 / 256
|
|
||||||
let b2: Int = r2 - (r2 / 256) * 256
|
|
||||||
let b3: Int = r2 / 256
|
|
||||||
let b: String = __str_set_char(buf, off, b0)
|
|
||||||
b = __str_set_char(b, off + 1, b1)
|
|
||||||
b = __str_set_char(b, off + 2, b2)
|
|
||||||
b = __str_set_char(b, off + 3, b3)
|
|
||||||
return b
|
|
||||||
}
|
|
||||||
|
|
||||||
fn wav_ascii(buf: String, off: Int, s: String) -> String {
|
|
||||||
let n: Int = str_len(s)
|
|
||||||
let i: Int = 0
|
|
||||||
let b: String = buf
|
|
||||||
while i < n {
|
|
||||||
let c: Int = str_char_code(s, i)
|
|
||||||
b = __str_set_char(b, off + i, c)
|
|
||||||
i = i + 1
|
|
||||||
}
|
|
||||||
return b
|
|
||||||
}
|
|
||||||
|
|
||||||
fn write_wav(samples: [Int], sr: Int, path: String) -> Bool {
|
|
||||||
let ns: Int = native_list_len(samples)
|
|
||||||
let datalen: Int = ns * 2
|
|
||||||
let total: Int = 44 + datalen
|
|
||||||
let buf: String = __str_alloc(total)
|
|
||||||
buf = wav_ascii(buf, 0, "RIFF")
|
|
||||||
buf = wav_le32(buf, 4, 36 + datalen)
|
|
||||||
buf = wav_ascii(buf, 8, "WAVE")
|
|
||||||
buf = wav_ascii(buf, 12, "fmt ")
|
|
||||||
buf = wav_le32(buf, 16, 16)
|
|
||||||
buf = wav_le16(buf, 20, 1)
|
|
||||||
buf = wav_le16(buf, 22, 1)
|
|
||||||
buf = wav_le32(buf, 24, sr)
|
|
||||||
buf = wav_le32(buf, 28, sr * 2)
|
|
||||||
buf = wav_le16(buf, 32, 2)
|
|
||||||
buf = wav_le16(buf, 34, 16)
|
|
||||||
buf = wav_ascii(buf, 36, "data")
|
|
||||||
buf = wav_le32(buf, 40, datalen)
|
|
||||||
let j: Int = 0
|
|
||||||
let off: Int = 44
|
|
||||||
while j < ns {
|
|
||||||
let raw: Int = native_list_get(samples, j)
|
|
||||||
buf = wav_le16(buf, off, raw)
|
|
||||||
off = off + 2
|
|
||||||
j = j + 1
|
|
||||||
}
|
|
||||||
return __fs_write_bytes(path, buf, total)
|
|
||||||
}
|
|
||||||
|
|
||||||
// One formant resonance as a float Lorentzian peak (own-core physics).
|
|
||||||
fn fgain(f: Float, fc: Float, bw: Float) -> Float {
|
|
||||||
let d: Float = f - fc
|
|
||||||
return (bw * bw) / (d * d + bw * bw)
|
|
||||||
}
|
|
||||||
|
|
||||||
// His PITCH MELODY from measured prosody [f0_median, f0_min, f0_max, declination].
|
|
||||||
// A natural statement shape over the utterance: onset rise to the median, a
|
|
||||||
// near-flat body (his declination is ~0.6 Hz/s), and a final fall toward f0_min.
|
|
||||||
// Follows his melody + range, not a fixed 0.85 decline. gidx/total = position.
|
|
||||||
fn prosody_f0(pros: [Int], gidx: Int, total: Int) -> Int {
|
|
||||||
let med: Int = native_list_get(pros, 0)
|
|
||||||
let lo: Int = native_list_get(pros, 1)
|
|
||||||
let hi: Int = native_list_get(pros, 2)
|
|
||||||
let p: Int = gidx * 1000 / total
|
|
||||||
let f0: Int = med
|
|
||||||
if p < 150 {
|
|
||||||
f0 = lo + (med - lo) * p / 150
|
|
||||||
} else {
|
|
||||||
if p > 700 {
|
|
||||||
f0 = med + (lo - med) * (p - 700) / 300
|
|
||||||
} else {
|
|
||||||
f0 = med
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if f0 < lo {
|
|
||||||
f0 = lo
|
|
||||||
}
|
|
||||||
if f0 > hi {
|
|
||||||
f0 = hi
|
|
||||||
}
|
|
||||||
return f0
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- The render: phoneme codes + voice signature -> normalized PCM samples ----
|
|
||||||
// Formant geometry per phoneme is READ FROM THE ENGRAM (pmap) via phon_geo — no
|
|
||||||
// table in code. The optional ACCENT map (amap) composes a transform onto the
|
|
||||||
// voice (voice (+) accent, separable): RP formant overrides read from the accent
|
|
||||||
// manifold + a non-rhotic coda-R drop. Empty amap = base General-American.
|
|
||||||
// Synthesis is FLOAT: a real phase accumulator + math_sin, superposition physics.
|
|
||||||
fn synth_codes_accent(codes0: [String], voice: [String], pmap: [String], amap: [String], vset: [String], vmap: [String], prosody: [Int]) -> [Int] {
|
|
||||||
let sr: Int = 16000
|
|
||||||
let srf: Float = 16000.0
|
|
||||||
let two_pi: Float = 6.283185307
|
|
||||||
let kf: Int = voice_get_int(voice, "kf")
|
|
||||||
let f0s: Int = voice_get_int(voice, "f0")
|
|
||||||
let f0e: Int = voice_get_int(voice, "f0_end")
|
|
||||||
let durm: Int = voice_get_int(voice, "dur")
|
|
||||||
if kf <= 0 {
|
|
||||||
kf = 1000
|
|
||||||
}
|
|
||||||
if durm <= 0 {
|
|
||||||
durm = 1000
|
|
||||||
}
|
|
||||||
let use_accent: Int = 0
|
|
||||||
if native_list_len(amap) > 0 {
|
|
||||||
use_accent = 1
|
|
||||||
}
|
|
||||||
let codes: [String] = codes0
|
|
||||||
if use_accent == 1 {
|
|
||||||
if is_nonrhotic(amap) == 1 {
|
|
||||||
codes = apply_rhoticity(codes0, vset)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let nc: Int = native_list_len(codes)
|
|
||||||
|
|
||||||
// pass 1: per-segment sample counts + total
|
|
||||||
let segn: [Int] = native_list_empty()
|
|
||||||
let total: Int = 0
|
|
||||||
let ci: Int = 0
|
|
||||||
while ci < nc {
|
|
||||||
let code: String = native_list_get(codes, ci)
|
|
||||||
let p: [Int] = phon_geo(pmap, code)
|
|
||||||
let durms: Int = native_list_get(p, 8)
|
|
||||||
let ns: Int = durms * 16 * durm / 1000
|
|
||||||
segn = native_list_append(segn, ns)
|
|
||||||
total = total + ns
|
|
||||||
ci = ci + 1
|
|
||||||
}
|
|
||||||
if total <= 0 {
|
|
||||||
total = 1
|
|
||||||
}
|
|
||||||
|
|
||||||
// pass 2: synthesize
|
|
||||||
let samples: [Int] = native_list_empty()
|
|
||||||
let phasef: Float = 0.0
|
|
||||||
let gidx: Int = 0
|
|
||||||
let prevF1: Int = 500 * kf / 1000
|
|
||||||
let prevF2: Int = 1500 * kf / 1000
|
|
||||||
let prevF3: Int = 2500 * kf / 1000
|
|
||||||
let nstate: Int = 22695
|
|
||||||
let maxabs: Int = 1
|
|
||||||
|
|
||||||
let ci2: Int = 0
|
|
||||||
while ci2 < nc {
|
|
||||||
let code: String = native_list_get(codes, ci2)
|
|
||||||
let p: [Int] = phon_geo(pmap, code)
|
|
||||||
let rf1: Int = native_list_get(p, 0)
|
|
||||||
let rf2: Int = native_list_get(p, 1)
|
|
||||||
let rf3: Int = native_list_get(p, 2)
|
|
||||||
if use_accent == 1 {
|
|
||||||
let ov: [Int] = accent_formants(amap, code)
|
|
||||||
if native_list_len(ov) >= 3 {
|
|
||||||
rf1 = native_list_get(ov, 0)
|
|
||||||
rf2 = native_list_get(ov, 1)
|
|
||||||
rf3 = native_list_get(ov, 2)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
// HIS measured vowel target overrides the generic/kf path (absolute Hz —
|
|
||||||
// his formants already encode his vocal tract, so no kf scaling).
|
|
||||||
let usekf: Int = 1
|
|
||||||
if native_list_len(vmap) > 0 {
|
|
||||||
let hv: [Int] = vmap_get(vmap, code)
|
|
||||||
if native_list_len(hv) >= 3 {
|
|
||||||
rf1 = native_list_get(hv, 0)
|
|
||||||
rf2 = native_list_get(hv, 1)
|
|
||||||
rf3 = native_list_get(hv, 2)
|
|
||||||
usekf = 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let F1t: Int = rf1 * kf / 1000
|
|
||||||
let F2t: Int = rf2 * kf / 1000
|
|
||||||
let F3t: Int = rf3 * kf / 1000
|
|
||||||
if usekf == 0 {
|
|
||||||
F1t = rf1
|
|
||||||
F2t = rf2
|
|
||||||
F3t = rf3
|
|
||||||
}
|
|
||||||
let B1: Int = native_list_get(p, 3)
|
|
||||||
let B2: Int = native_list_get(p, 4)
|
|
||||||
let B3: Int = native_list_get(p, 5)
|
|
||||||
let voiced: Int = native_list_get(p, 6)
|
|
||||||
let ampv: Int = native_list_get(p, 9)
|
|
||||||
let ns: Int = native_list_get(segn, ci2)
|
|
||||||
let trans: Int = ns / 2
|
|
||||||
if trans > 560 {
|
|
||||||
trans = 560
|
|
||||||
}
|
|
||||||
if trans < 1 {
|
|
||||||
trans = 1
|
|
||||||
}
|
|
||||||
let k: Int = 0
|
|
||||||
while k < ns {
|
|
||||||
let cF1: Int = F1t
|
|
||||||
let cF2: Int = F2t
|
|
||||||
let cF3: Int = F3t
|
|
||||||
if k < trans {
|
|
||||||
cF1 = prevF1 + (F1t - prevF1) * k / trans
|
|
||||||
cF2 = prevF2 + (F2t - prevF2) * k / trans
|
|
||||||
cF3 = prevF3 + (F3t - prevF3) * k / trans
|
|
||||||
}
|
|
||||||
let f0c: Int = f0s + (f0e - f0s) * gidx / total
|
|
||||||
if native_list_len(prosody) >= 3 {
|
|
||||||
f0c = prosody_f0(prosody, gidx, total)
|
|
||||||
}
|
|
||||||
if f0c < 40 {
|
|
||||||
f0c = 40
|
|
||||||
}
|
|
||||||
let env: Int = 32767
|
|
||||||
let ar: Int = 96
|
|
||||||
if k < ar {
|
|
||||||
env = 32767 * k / ar
|
|
||||||
}
|
|
||||||
let tail: Int = ns - k
|
|
||||||
if tail < ar {
|
|
||||||
env = 32767 * tail / ar
|
|
||||||
}
|
|
||||||
let f0cf: Float = int_to_float(f0c)
|
|
||||||
phasef = phasef + two_pi * f0cf / srf
|
|
||||||
if phasef > two_pi {
|
|
||||||
phasef = phasef - two_pi
|
|
||||||
}
|
|
||||||
|
|
||||||
let s: Int = 0
|
|
||||||
if voiced == 1 {
|
|
||||||
let cF1f: Float = int_to_float(cF1)
|
|
||||||
let cF2f: Float = int_to_float(cF2)
|
|
||||||
let cF3f: Float = int_to_float(cF3)
|
|
||||||
let B1f: Float = int_to_float(B1)
|
|
||||||
let B2f: Float = int_to_float(B2)
|
|
||||||
let B3f: Float = int_to_float(B3)
|
|
||||||
let acc: Float = 0.0
|
|
||||||
let h: Int = 1
|
|
||||||
while h <= 50 {
|
|
||||||
let hf: Float = int_to_float(h)
|
|
||||||
let fhf: Float = hf * f0cf
|
|
||||||
if fhf < 7900.0 {
|
|
||||||
let sv: Float = math_sin(phasef * hf)
|
|
||||||
let src: Float = 1.0 / hf
|
|
||||||
let g1: Float = fgain(fhf, cF1f, B1f)
|
|
||||||
let g2: Float = fgain(fhf, cF2f, B2f)
|
|
||||||
let g3: Float = fgain(fhf, cF3f, B3f)
|
|
||||||
let g: Float = g1 + g2 + g3
|
|
||||||
acc = acc + src * g * sv
|
|
||||||
}
|
|
||||||
h = h + 1
|
|
||||||
}
|
|
||||||
s = float_to_int(acc * 4000.0)
|
|
||||||
} else {
|
|
||||||
if ampv > 0 {
|
|
||||||
nstate = nstate * 1103515245 + 12345
|
|
||||||
nstate = nstate - (nstate / 2147483648) * 2147483648
|
|
||||||
if nstate < 0 {
|
|
||||||
nstate = 0 - nstate
|
|
||||||
}
|
|
||||||
let nz: Int = nstate / 32768 - 32768
|
|
||||||
s = nz
|
|
||||||
}
|
|
||||||
}
|
|
||||||
s = s * ampv / 100
|
|
||||||
s = s * env / 32767
|
|
||||||
samples = native_list_append(samples, s)
|
|
||||||
let a: Int = s
|
|
||||||
if a < 0 {
|
|
||||||
a = 0 - a
|
|
||||||
}
|
|
||||||
if a > maxabs {
|
|
||||||
maxabs = a
|
|
||||||
}
|
|
||||||
gidx = gidx + 1
|
|
||||||
k = k + 1
|
|
||||||
}
|
|
||||||
prevF1 = F1t
|
|
||||||
prevF2 = F2t
|
|
||||||
prevF3 = F3t
|
|
||||||
ci2 = ci2 + 1
|
|
||||||
}
|
|
||||||
|
|
||||||
// normalize to int16 range (~22000 peak)
|
|
||||||
let out: [Int] = native_list_empty()
|
|
||||||
let ntot: Int = native_list_len(samples)
|
|
||||||
let j: Int = 0
|
|
||||||
while j < ntot {
|
|
||||||
let raw: Int = native_list_get(samples, j)
|
|
||||||
let v: Int = raw * 22000 / maxabs
|
|
||||||
out = native_list_append(out, v)
|
|
||||||
j = j + 1
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// GA convenience wrapper (no accent) — keeps the base render path.
|
|
||||||
fn synth_codes(codes: [String], voice: [String], pmap: [String]) -> [Int] {
|
|
||||||
let noacc: [String] = native_list_empty()
|
|
||||||
let novset: [String] = native_list_empty()
|
|
||||||
let novmap: [String] = native_list_empty()
|
|
||||||
let nopros: [Int] = native_list_empty()
|
|
||||||
return synth_codes_accent(codes, voice, pmap, noacc, novset, novmap, nopros)
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- Voice-by-imitation: HEAR a PCM sample -> extract the voice signature -----
|
|
||||||
// Pitch by autocorrelation; vocal-tract scale (kf) from the F1 formant peak of a
|
|
||||||
// heard sustained vowel /AA/ (nominal F1 = 730 Hz) via an integer DFT. The
|
|
||||||
// analyzer sees ONLY the PCM samples — never the source signature numbers — so
|
|
||||||
// recovery is genuinely by ear.
|
|
||||||
fn voice_f0(samples: [Int], sr: Int) -> Int {
|
|
||||||
let n: Int = native_list_len(samples)
|
|
||||||
let start: Int = n / 4
|
|
||||||
let end: Int = n * 3 / 4
|
|
||||||
// bound the analysis window so accumulators can never overflow on long input
|
|
||||||
if end - start > 6000 {
|
|
||||||
end = start + 6000
|
|
||||||
}
|
|
||||||
let minlag: Int = sr / 300
|
|
||||||
let maxlag: Int = sr / 75
|
|
||||||
let best: Int = 0
|
|
||||||
let bestlag: Int = minlag
|
|
||||||
let lag: Int = minlag
|
|
||||||
while lag <= maxlag {
|
|
||||||
let sum: Int = 0
|
|
||||||
let i: Int = start
|
|
||||||
while i < end {
|
|
||||||
let ai: Int = native_list_get(samples, i)
|
|
||||||
let bi: Int = native_list_get(samples, i + lag)
|
|
||||||
sum = sum + ai * bi / 256
|
|
||||||
i = i + 2
|
|
||||||
}
|
|
||||||
if sum > best {
|
|
||||||
best = sum
|
|
||||||
bestlag = lag
|
|
||||||
}
|
|
||||||
lag = lag + 1
|
|
||||||
}
|
|
||||||
if bestlag < 1 {
|
|
||||||
bestlag = 1
|
|
||||||
}
|
|
||||||
return sr / bestlag
|
|
||||||
}
|
|
||||||
|
|
||||||
fn voice_peak_in_band(samples: [Int], sr: Int, flo: Int, fhi: Int) -> Int {
|
|
||||||
let n: Int = native_list_len(samples)
|
|
||||||
let start: Int = n / 4
|
|
||||||
let end: Int = n * 3 / 4
|
|
||||||
// bound the DFT window: re/im are accumulated /4096, and re*re must stay in
|
|
||||||
// int64 — cap terms so (window/2)*(peak_term) squared cannot overflow.
|
|
||||||
if end - start > 3000 {
|
|
||||||
end = start + 3000
|
|
||||||
}
|
|
||||||
let bestmag: Int = 0
|
|
||||||
let bestf: Int = flo
|
|
||||||
let f: Int = flo
|
|
||||||
while f <= fhi {
|
|
||||||
let re: Int = 0
|
|
||||||
let im: Int = 0
|
|
||||||
let i: Int = start
|
|
||||||
while i < end {
|
|
||||||
let x: Int = native_list_get(samples, i)
|
|
||||||
let ph: Int = i * f * 65536 / sr
|
|
||||||
ph = ph - (ph / 65536) * 65536
|
|
||||||
let cq: Int = sp_cos(ph)
|
|
||||||
let sq: Int = sp_sin(ph)
|
|
||||||
re = re + x * cq / 4096
|
|
||||||
im = im + x * sq / 4096
|
|
||||||
i = i + 2
|
|
||||||
}
|
|
||||||
let mag: Int = re * re + im * im
|
|
||||||
if mag > bestmag {
|
|
||||||
bestmag = mag
|
|
||||||
bestf = f
|
|
||||||
}
|
|
||||||
f = f + 25
|
|
||||||
}
|
|
||||||
return bestf
|
|
||||||
}
|
|
||||||
|
|
||||||
// Analyze a heard sustained /AA/ -> a full voice signature (by ear).
|
|
||||||
fn voice_analyze(samples: [Int], sr: Int) -> [String] {
|
|
||||||
let f0: Int = voice_f0(samples, sr)
|
|
||||||
let f1: Int = voice_peak_in_band(samples, sr, 450, 1150)
|
|
||||||
let kf: Int = 1000 * f1 / 730
|
|
||||||
let f0e: Int = f0 * 85 / 100
|
|
||||||
return voice_new("imitated", f0, f0e, kf, 1000, 1000, 8)
|
|
||||||
}
|
|
||||||
@@ -1,153 +0,0 @@
|
|||||||
// surface-profile.el - Surface profile data and accessors.
|
|
||||||
//
|
|
||||||
// THE NATIVE EFFERENT SEAM: surface = a pluggable PROFILE, using the exact same
|
|
||||||
// slot-map mechanism as language-profile.el. A language profile tells the
|
|
||||||
// realizer HOW to shape a natural-language surface (word order, morphology); a
|
|
||||||
// SURFACE profile tells the realizer WHICH surface to project meaning onto
|
|
||||||
// (markdown, docx, html, plain, or a non-text medium like symbolic music).
|
|
||||||
//
|
|
||||||
// The generalization is exact: realize_lang(form, profile) already renders a
|
|
||||||
// SemForm parameterized by a [String] profile read via lang_get. Surface is one
|
|
||||||
// more axis of that same profile vector. One frame (sem_frame), one plan step
|
|
||||||
// (sem_to_spec), one render (realize) — the surface is DATA, not a code path,
|
|
||||||
// precisely as language is data. Adding a surface means adding a profile, no
|
|
||||||
// engine change. This is the multimodal projector, native: geometry -> any
|
|
||||||
// surface, the efferent twin of ingest.
|
|
||||||
//
|
|
||||||
// Surface slot keys:
|
|
||||||
// surface - "markdown" | "docx" | "html" | "plain" | "midi" | "image"
|
|
||||||
// modality - "text" | "audio" | "image" | "video"
|
|
||||||
// media_type - MIME type of the emitted surface
|
|
||||||
// head_open - string prepended to a heading (e.g. "## " for markdown)
|
|
||||||
// head_close - string appended to a heading (e.g. "" for markdown, "</h2>" for html)
|
|
||||||
// emph_open - string opening emphasis (e.g. "*")
|
|
||||||
// emph_close - string closing emphasis (e.g. "*")
|
|
||||||
// item_mark - list-item marker (e.g. "- ")
|
|
||||||
// para_sep - paragraph separator (e.g. "\n\n")
|
|
||||||
//
|
|
||||||
// For a TEXT modality the render composes these markers around the surface that
|
|
||||||
// the EXISTING realizer produces (realize_lang / sem_realize). For a non-text
|
|
||||||
// modality (audio/image) the profile declares modality + media_type and the
|
|
||||||
// render dispatches to the medium projector, which reads the SAME frame's
|
|
||||||
// geometry (its intent/affect/structure) and projects it onto sound or pixels —
|
|
||||||
// deterministic-from-meaning, nothing invented. That dispatch point is where a
|
|
||||||
// music profile or image profile conforms, native, no parallel layer.
|
|
||||||
|
|
||||||
// -- Constructor -------------------------------------------------------------
|
|
||||||
|
|
||||||
fn surface_profile(surface: String, modality: String, media_type: String, head_open: String, head_close: String, emph_open: String, emph_close: String, item_mark: String, para_sep: String) -> [String] {
|
|
||||||
let r: [String] = native_list_empty()
|
|
||||||
let r = native_list_append(r, "surface")
|
|
||||||
let r = native_list_append(r, surface)
|
|
||||||
let r = native_list_append(r, "modality")
|
|
||||||
let r = native_list_append(r, modality)
|
|
||||||
let r = native_list_append(r, "media_type")
|
|
||||||
let r = native_list_append(r, media_type)
|
|
||||||
let r = native_list_append(r, "head_open")
|
|
||||||
let r = native_list_append(r, head_open)
|
|
||||||
let r = native_list_append(r, "head_close")
|
|
||||||
let r = native_list_append(r, head_close)
|
|
||||||
let r = native_list_append(r, "emph_open")
|
|
||||||
let r = native_list_append(r, emph_open)
|
|
||||||
let r = native_list_append(r, "emph_close")
|
|
||||||
let r = native_list_append(r, emph_close)
|
|
||||||
let r = native_list_append(r, "item_mark")
|
|
||||||
let r = native_list_append(r, item_mark)
|
|
||||||
let r = native_list_append(r, "para_sep")
|
|
||||||
let r = native_list_append(r, para_sep)
|
|
||||||
return r
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- Accessor (same convention as lang_get; standalone so this is a leaf) -----
|
|
||||||
|
|
||||||
fn surface_get(profile: [String], key: String) -> String {
|
|
||||||
let n: Int = native_list_len(profile)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n - 1 {
|
|
||||||
let k: String = native_list_get(profile, i)
|
|
||||||
if str_eq(k, key) {
|
|
||||||
return native_list_get(profile, i + 1)
|
|
||||||
}
|
|
||||||
let i = i + 2
|
|
||||||
}
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
fn surface_is_text(profile: [String]) -> Bool {
|
|
||||||
return str_eq(surface_get(profile, "modality"), "text")
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- Built-in TEXT surface profiles ------------------------------------------
|
|
||||||
|
|
||||||
// Markdown: headings with "## ", emphasis with "*", "- " list items.
|
|
||||||
fn surface_profile_markdown() -> [String] {
|
|
||||||
return surface_profile("markdown", "text", "text/markdown", "## ", "", "*", "*", "- ", "\n\n")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Plain text: no markup at all — headings become bare uppercase-free lines.
|
|
||||||
fn surface_profile_plain() -> [String] {
|
|
||||||
return surface_profile("plain", "text", "text/plain", "", "", "", "", " - ", "\n\n")
|
|
||||||
}
|
|
||||||
|
|
||||||
// HTML: block-level heading/emphasis tags.
|
|
||||||
fn surface_profile_html() -> [String] {
|
|
||||||
return surface_profile("html", "text", "text/html", "<h2>", "</h2>", "<em>", "</em>", "<li>", "\n")
|
|
||||||
}
|
|
||||||
|
|
||||||
// docx: WordprocessingML is structural, not inline-markup; the head/emph slots
|
|
||||||
// carry the run/style intent that the OOXML emitter maps to <w:pStyle>. Declared
|
|
||||||
// here so docx is a first-class surface on the same seam.
|
|
||||||
fn surface_profile_docx() -> [String] {
|
|
||||||
return surface_profile("docx", "text", "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "Heading2:", "", "b:", "", "bullet:", "\n")
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- Built-in NON-TEXT surface profiles (the multimodal seam) ----------------
|
|
||||||
|
|
||||||
// Symbolic music (MIDI): modality=audio. The render dispatches to the music
|
|
||||||
// projector, which reads the SAME frame's intent/affect and projects it to
|
|
||||||
// pitch/rhythm — deterministic-from-meaning. head/emph slots are empty because
|
|
||||||
// the medium is not textual; media_type names the surface. A music profile
|
|
||||||
// (scale/mode/instrument) is layered onto this by the audio agent, native.
|
|
||||||
fn surface_profile_midi() -> [String] {
|
|
||||||
return surface_profile("midi", "audio", "audio/midi", "", "", "", "", "", "")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Synthesized audio (WAV): modality=audio, peer to midi. The richer audio
|
|
||||||
// surface — the render SUPERPOSES ingested tonal primitives (sine at f0*n per an
|
|
||||||
// ingested instrument signature) into PCM, own-core, exactly as midi writes an
|
|
||||||
// SMF via struct. A music profile (scale/mode/instrument/adsr) layers onto this
|
|
||||||
// as its own [String] slot-map read by the same getter. Same frame -> midi OR
|
|
||||||
// audio, interchangeable; this is the audio agent's native conforming point.
|
|
||||||
fn surface_profile_audio() -> [String] {
|
|
||||||
return surface_profile("audio", "audio", "audio/wav", "", "", "", "", "", "")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Image (raster): modality=image. Documented seam — the render dispatches to the
|
|
||||||
// image projector, the efferent twin of image ingest, reading the same frame.
|
|
||||||
fn surface_profile_image() -> [String] {
|
|
||||||
return surface_profile("image", "image", "image/png", "", "", "", "", "", "")
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- Composition helpers: wrap realized TEXT with the surface's markers -------
|
|
||||||
//
|
|
||||||
// These take text the EXISTING realizer already produced and shape it for the
|
|
||||||
// surface. They add NO content — pure surface typography over faithful text,
|
|
||||||
// exactly as the language profile adds no content, only linguistic form.
|
|
||||||
|
|
||||||
fn surface_heading(profile: [String], text: String) -> String {
|
|
||||||
let o: String = surface_get(profile, "head_open")
|
|
||||||
let c: String = surface_get(profile, "head_close")
|
|
||||||
return o + text + c
|
|
||||||
}
|
|
||||||
|
|
||||||
fn surface_emph(profile: [String], text: String) -> String {
|
|
||||||
let o: String = surface_get(profile, "emph_open")
|
|
||||||
let c: String = surface_get(profile, "emph_close")
|
|
||||||
return o + text + c
|
|
||||||
}
|
|
||||||
|
|
||||||
// A section: a heading + a paragraph separator + the (already realized) body.
|
|
||||||
fn surface_section(profile: [String], heading: String, body: String) -> String {
|
|
||||||
let sep: String = surface_get(profile, "para_sep")
|
|
||||||
return surface_heading(profile, heading) + sep + body
|
|
||||||
}
|
|
||||||
@@ -1,226 +0,0 @@
|
|||||||
// translate.el - ELP geometry-native translation faculty (concept-pivot).
|
|
||||||
//
|
|
||||||
// ARCHITECTURE (corrected — Will, 2026-08-14): translation is NOT a bilingual
|
|
||||||
// string map and needs NO external multilingual encoder. It routes through the
|
|
||||||
// engram's concept geometry:
|
|
||||||
//
|
|
||||||
// comprehend(source) → CONCEPT-FRAME (language-invariant, in the manifold) → realize(target)
|
|
||||||
//
|
|
||||||
// A word in any language is resolved to the CONCEPT it denotes via that
|
|
||||||
// language's own lexicon/morphology (a monolingual step — the engram's
|
|
||||||
// nearest-region ranker only ever disambiguates senses WITHIN one language, so
|
|
||||||
// an English-trained embedder is fine and never compares "ocean" to "océano" as
|
|
||||||
// strings). The concept-node's location in the manifold IS the meaning; it is
|
|
||||||
// the shared pivot. "océano" and "ocean" need not be near each other as surface
|
|
||||||
// tokens — they resolve to the SAME concept node.
|
|
||||||
//
|
|
||||||
// This file supplies each target language's CONCEPT→SURFACE lexicon (its own
|
|
||||||
// labeling of the shared concept nodes) — the mirror image of comprehend.el's
|
|
||||||
// SURFACE→CONCEPT resolvers (cp_pron_concept, cp_analyze_verb/cp_irr2, …). The
|
|
||||||
// frame produced by parse_spec() is the interlingua: one parse realizes into N
|
|
||||||
// targets. Concept coverage below is the "Slowness" poem's inventory; a concept
|
|
||||||
// with no target label passes through and is flagged oov (honest bound).
|
|
||||||
//
|
|
||||||
// SACRED: polarity is a concept and is never routed to a content lemma. The
|
|
||||||
// negative-adverb concept ("never") realizes to a target negator ("nunca"/"mai"),
|
|
||||||
// never to a content word.
|
|
||||||
//
|
|
||||||
// Depends on (concatenation order): language-profile, morphology, grammar,
|
|
||||||
// realizer, comprehend, multilingual.
|
|
||||||
|
|
||||||
// ── VERB concept → target lemma (each language's own labeling of the concept) ──
|
|
||||||
// The input is the language-invariant verb concept (English lemma = concept id,
|
|
||||||
// exactly as comprehend.el emits it). NOT a translation of a Spanish string.
|
|
||||||
fn lemma_for_concept(concept: String, lang: String) -> String {
|
|
||||||
if str_eq(lang, "en") { return concept }
|
|
||||||
if str_eq(lang, "es") {
|
|
||||||
if str_eq(concept, "fight") { return "luchar" }
|
|
||||||
if str_eq(concept, "touch") { return "tocar" }
|
|
||||||
if str_eq(concept, "wait") { return "esperar" }
|
|
||||||
if str_eq(concept, "see") { return "ver" }
|
|
||||||
if str_eq(concept, "break") { return "romper" }
|
|
||||||
if str_eq(concept, "stay") { return "quedar" }
|
|
||||||
if str_eq(concept, "call") { return "llamar" }
|
|
||||||
if str_eq(concept, "run") { return "correr" }
|
|
||||||
if str_eq(concept, "chase") { return "perseguir" }
|
|
||||||
if str_eq(concept, "take") { return "tomar" }
|
|
||||||
if str_eq(concept, "carry") { return "llevar" }
|
|
||||||
return ml_translate_pred(concept, "es")
|
|
||||||
}
|
|
||||||
if str_eq(lang, "pt") {
|
|
||||||
if str_eq(concept, "fight") { return "lutar" }
|
|
||||||
if str_eq(concept, "touch") { return "tocar" }
|
|
||||||
if str_eq(concept, "wait") { return "esperar" }
|
|
||||||
if str_eq(concept, "see") { return "ver" }
|
|
||||||
if str_eq(concept, "break") { return "quebrar" }
|
|
||||||
if str_eq(concept, "stay") { return "ficar" }
|
|
||||||
if str_eq(concept, "call") { return "chamar" }
|
|
||||||
if str_eq(concept, "run") { return "correr" }
|
|
||||||
if str_eq(concept, "chase") { return "perseguir" }
|
|
||||||
if str_eq(concept, "take") { return "tomar" }
|
|
||||||
if str_eq(concept, "carry") { return "levar" }
|
|
||||||
return ml_translate_pred(concept, "pt")
|
|
||||||
}
|
|
||||||
if str_eq(lang, "it") {
|
|
||||||
if str_eq(concept, "fight") { return "lottare" }
|
|
||||||
if str_eq(concept, "touch") { return "toccare" }
|
|
||||||
if str_eq(concept, "wait") { return "aspettare" }
|
|
||||||
if str_eq(concept, "see") { return "vedere" }
|
|
||||||
if str_eq(concept, "break") { return "rompere" }
|
|
||||||
if str_eq(concept, "stay") { return "restare" }
|
|
||||||
return ml_translate_pred(concept, "it")
|
|
||||||
}
|
|
||||||
return concept
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── NOUN concept → [target lemma, gender] (target language's concept lexicon) ──
|
|
||||||
fn noun_for_concept(concept: String, lang: String) -> [String] {
|
|
||||||
let out: [String] = native_list_empty()
|
|
||||||
if str_eq(lang, "es") {
|
|
||||||
if str_eq(concept, "ocean") { let out = native_list_append(out, "océano"); let out = native_list_append(out, "m"); return out }
|
|
||||||
if str_eq(concept, "root") { let out = native_list_append(out, "raíz"); let out = native_list_append(out, "f"); return out }
|
|
||||||
if str_eq(concept, "roots") { let out = native_list_append(out, "raíces"); let out = native_list_append(out, "fp"); return out }
|
|
||||||
if str_eq(concept, "breaking") { let out = native_list_append(out, "ruptura"); let out = native_list_append(out, "f"); return out }
|
|
||||||
if str_eq(concept, "shoreline") { let out = native_list_append(out, "orilla"); let out = native_list_append(out, "f"); return out }
|
|
||||||
if str_eq(concept, "patience") { let out = native_list_append(out, "paciencia"); let out = native_list_append(out, "f"); return out }
|
|
||||||
if str_eq(concept, "wave") { let out = native_list_append(out, "ola"); let out = native_list_append(out, "f"); return out }
|
|
||||||
if str_eq(concept, "truth") { let out = native_list_append(out, "verdad"); let out = native_list_append(out, "f"); return out }
|
|
||||||
if str_eq(concept, "silence") { let out = native_list_append(out, "silencio"); let out = native_list_append(out, "m"); return out }
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
if str_eq(lang, "pt") {
|
|
||||||
if str_eq(concept, "ocean") { let out = native_list_append(out, "oceano"); let out = native_list_append(out, "m"); return out }
|
|
||||||
if str_eq(concept, "root") { let out = native_list_append(out, "raiz"); let out = native_list_append(out, "f"); return out }
|
|
||||||
if str_eq(concept, "roots") { let out = native_list_append(out, "raízes"); let out = native_list_append(out, "fp"); return out }
|
|
||||||
if str_eq(concept, "breaking") { let out = native_list_append(out, "ruptura"); let out = native_list_append(out, "f"); return out }
|
|
||||||
if str_eq(concept, "shoreline") { let out = native_list_append(out, "costa"); let out = native_list_append(out, "f"); return out }
|
|
||||||
if str_eq(concept, "patience") { let out = native_list_append(out, "paciência"); let out = native_list_append(out, "f"); return out }
|
|
||||||
if str_eq(concept, "wave") { let out = native_list_append(out, "onda"); let out = native_list_append(out, "f"); return out }
|
|
||||||
if str_eq(concept, "truth") { let out = native_list_append(out, "verdade"); let out = native_list_append(out, "f"); return out }
|
|
||||||
if str_eq(concept, "silence") { let out = native_list_append(out, "silêncio"); let out = native_list_append(out, "m"); return out }
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// definite article for a gender+number tag / lang. "f"|"m" singular, "fp"|"mp" plural.
|
|
||||||
fn article_for(gtag: String, lang: String) -> String {
|
|
||||||
if str_eq(lang, "es") {
|
|
||||||
if str_eq(gtag, "fp") { return "las" }
|
|
||||||
if str_eq(gtag, "mp") { return "los" }
|
|
||||||
if str_eq(gtag, "f") { return "la" }
|
|
||||||
return "el"
|
|
||||||
}
|
|
||||||
if str_eq(lang, "pt") {
|
|
||||||
if str_eq(gtag, "fp") { return "as" }
|
|
||||||
if str_eq(gtag, "mp") { return "os" }
|
|
||||||
if str_eq(gtag, "f") { return "a" }
|
|
||||||
return "o"
|
|
||||||
}
|
|
||||||
if str_eq(lang, "it") { if str_eq(gtag, "f") { return "la" } return "il" }
|
|
||||||
return "the"
|
|
||||||
}
|
|
||||||
|
|
||||||
// SURFACE→CONCEPT for an English object NP: strip determiner, return bare head
|
|
||||||
// (which, for content nouns, is already the concept id).
|
|
||||||
fn np_concept_head(np: String) -> String {
|
|
||||||
let s: String = str_to_lower(np)
|
|
||||||
let dets: [String] = native_list_empty()
|
|
||||||
let dets = native_list_append(dets, "the ")
|
|
||||||
let dets = native_list_append(dets, "a ")
|
|
||||||
let dets = native_list_append(dets, "an ")
|
|
||||||
let dets = native_list_append(dets, "my ")
|
|
||||||
let dets = native_list_append(dets, "your ")
|
|
||||||
let dets = native_list_append(dets, "his ")
|
|
||||||
let dets = native_list_append(dets, "her ")
|
|
||||||
let dets = native_list_append(dets, "its ")
|
|
||||||
let dets = native_list_append(dets, "our ")
|
|
||||||
let dets = native_list_append(dets, "their ")
|
|
||||||
let dets = native_list_append(dets, "every ")
|
|
||||||
let i: Int = 0
|
|
||||||
let n: Int = native_list_len(dets)
|
|
||||||
while i < n {
|
|
||||||
let d: String = native_list_get(dets, i)
|
|
||||||
let dl: Int = str_len(d)
|
|
||||||
if str_len(s) > dl {
|
|
||||||
if str_eq(str_slice(s, 0, dl), d) { return str_slice(s, dl, str_len(s)) }
|
|
||||||
}
|
|
||||||
let i = i + 1
|
|
||||||
}
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
|
|
||||||
// CONCEPT→SURFACE: realize an object-NP concept in the target language with its
|
|
||||||
// definite article. Unknown concept => pass the English head through (oov).
|
|
||||||
fn np_for_concept(np: String, lang: String) -> String {
|
|
||||||
if str_eq(np, "") { return "" }
|
|
||||||
let head: String = np_concept_head(np)
|
|
||||||
let pair: [String] = noun_for_concept(head, lang)
|
|
||||||
if native_list_len(pair) < 2 { return head }
|
|
||||||
let lemma: String = native_list_get(pair, 0)
|
|
||||||
let gtag: String = native_list_get(pair, 1)
|
|
||||||
return article_for(gtag, lang) + " " + lemma
|
|
||||||
}
|
|
||||||
|
|
||||||
// SURFACE→CONCEPT for a subject pronoun, then CONCEPT→SURFACE in the target —
|
|
||||||
// reusing comprehend.el's NATIVE concept-pivot (cp_pron_concept /
|
|
||||||
// cp_rom_pron_surface). This is the template the whole faculty follows.
|
|
||||||
fn pron_for_target(agent: String, lang: String) -> String {
|
|
||||||
let concept: String = cp_pron_concept(str_to_lower(agent))
|
|
||||||
if str_eq(concept, "") { return agent }
|
|
||||||
if str_eq(lang, "en") { return cp_pron_surface(concept) }
|
|
||||||
return cp_rom_pron_surface(concept, lang)
|
|
||||||
}
|
|
||||||
|
|
||||||
// The negative-adverb concept realized as the target's preverbal negator (SACRED).
|
|
||||||
fn negator_for_concept(neg_word: String, lang: String) -> String {
|
|
||||||
let w: String = str_to_lower(neg_word)
|
|
||||||
if str_eq(w, "never") {
|
|
||||||
if str_eq(lang, "es") { return "nunca" }
|
|
||||||
if str_eq(lang, "pt") { return "nunca" }
|
|
||||||
if str_eq(lang, "it") { return "mai" }
|
|
||||||
}
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
// Some irregular English pasts that comprehend's cp_irr2 does not yet lemmatize
|
|
||||||
// (source-side SURFACE→CONCEPT gap). Kept minimal; belongs long-term in cp_irr2.
|
|
||||||
fn concept_of_verb(w: String) -> String {
|
|
||||||
if str_eq(w, "broke") { return "break" }
|
|
||||||
if str_eq(w, "broken") { return "break" }
|
|
||||||
if str_eq(w, "took") { return "take" }
|
|
||||||
if str_eq(w, "ran") { return "run" }
|
|
||||||
return w
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── the faculty: EN text → concept-frame → target surface ─────────────────────
|
|
||||||
fn translate_spec(text: String, tgt: String) -> [String] {
|
|
||||||
// 1. comprehend(source) → concept-frame (English lemmas = concept ids +
|
|
||||||
// SACRED polarity/neg_word). This frame lives in the concept geometry.
|
|
||||||
let spec: [String] = parse_spec(text)
|
|
||||||
let predc: String = concept_of_verb(slots_get(spec, "predicate"))
|
|
||||||
let patc: String = slots_get(spec, "patient")
|
|
||||||
let agentc: String = slots_get(spec, "agent")
|
|
||||||
let negw: String = slots_get(spec, "neg_word")
|
|
||||||
|
|
||||||
// 2. realize(target): resolve each concept to the target language's surface.
|
|
||||||
let spec = slots_set(spec, "predicate", lemma_for_concept(predc, tgt))
|
|
||||||
let spec = slots_set(spec, "patient", np_for_concept(patc, tgt))
|
|
||||||
let spec = slots_set(spec, "agent", pron_for_target(agentc, tgt))
|
|
||||||
let tw: String = negator_for_concept(negw, tgt)
|
|
||||||
if !str_eq(tw, "") { let spec = slots_set(spec, "neg_word", tw) }
|
|
||||||
let spec = slots_set(spec, "lang", tgt)
|
|
||||||
return spec
|
|
||||||
}
|
|
||||||
|
|
||||||
fn translate_line(text: String, tgt: String) -> String {
|
|
||||||
return realize(translate_spec(text, tgt))
|
|
||||||
}
|
|
||||||
|
|
||||||
// Concept-frame fingerprint (for concept-preservation fidelity — geometry-native,
|
|
||||||
// NOT a string cosine): the source-language-invariant concept tuple.
|
|
||||||
fn concept_frame(text: String) -> String {
|
|
||||||
let spec: [String] = parse_spec(text)
|
|
||||||
let predc: String = concept_of_verb(slots_get(spec, "predicate"))
|
|
||||||
return "pred=" + predc + " patient=" + np_concept_head(slots_get(spec, "patient")) + " pol=" + slots_get(spec, "polarity")
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
-144861
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
-130676
File diff suppressed because it is too large
Load Diff
-193894
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
-115916
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,244 +0,0 @@
|
|||||||
// voice-ingest.el - The LIVE VOICE LOOP reshape + ingest-as-geometry.
|
|
||||||
//
|
|
||||||
// EL cannot read a binary WAV (fs_read NUL-truncates), so the thin-medium DSP
|
|
||||||
// extractor is periph's `voiceprint` (autocorr F0 + LPC formants), equivalent to
|
|
||||||
// our own voice_analyze. This module: (1) RESHAPE the voiceprint JSON (TEXT) into
|
|
||||||
// the organ voice-signature schema; (2) INGEST it as a GEOMETRY manifold in the
|
|
||||||
// engram and engram_save it to a file; (3) READ the target signature BACK from
|
|
||||||
// that geometry (engram_load + scan + filter), never from the json or a table.
|
|
||||||
// HONEST: this reaches for pitch + a coarse vocal-tract scale (kf). It is NOT a
|
|
||||||
// clone — no glottal timbre, vowel-space, or articulation is captured.
|
|
||||||
|
|
||||||
fn parse_leading_int(s: String) -> Int {
|
|
||||||
let n: Int = str_len(s)
|
|
||||||
let i: Int = 0
|
|
||||||
let v: Int = 0
|
|
||||||
let started: Int = 0
|
|
||||||
while i < n {
|
|
||||||
let c: Int = str_char_code(s, i)
|
|
||||||
if c >= 48 {
|
|
||||||
if c <= 57 {
|
|
||||||
v = v * 10 + (c - 48)
|
|
||||||
started = 1
|
|
||||||
i = i + 1
|
|
||||||
} else {
|
|
||||||
i = n
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if started == 1 {
|
|
||||||
i = n
|
|
||||||
} else {
|
|
||||||
i = i + 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return v
|
|
||||||
}
|
|
||||||
|
|
||||||
// voiceprint JSON -> organ voice-signature source file; returns [f0,f0_end,kf,f1,f2,f3].
|
|
||||||
fn reshape_voiceprint(vppath: String, outjson: String) -> [Int] {
|
|
||||||
let j: String = fs_read(vppath)
|
|
||||||
let f0: Int = parse_uint_from(j, "f0_hz\":")
|
|
||||||
let fp: Int = str_index_of(j, "formants_hz")
|
|
||||||
let tail: String = str_slice(j, fp, fp + 120)
|
|
||||||
let br: Int = str_index_of(tail, "[")
|
|
||||||
let arr: String = str_slice(tail, br + 1, str_len(tail))
|
|
||||||
let f1: Int = parse_leading_int(arr)
|
|
||||||
let c1: Int = str_index_of(arr, ",")
|
|
||||||
let a2: String = str_slice(arr, c1 + 1, str_len(arr))
|
|
||||||
let f2: Int = parse_leading_int(a2)
|
|
||||||
let c2: Int = str_index_of(a2, ",")
|
|
||||||
let a3: String = str_slice(a2, c2 + 1, str_len(a2))
|
|
||||||
let f3: Int = parse_leading_int(a3)
|
|
||||||
let f0e: Int = f0 * 85 / 100
|
|
||||||
// derive kf honestly: coarse vocal-tract scale from the formant pattern
|
|
||||||
let t1: Int = 1000 * f1 / 500
|
|
||||||
let t2: Int = 1000 * f2 / 1500
|
|
||||||
let t3: Int = 1000 * f3 / 2500
|
|
||||||
let kf: Int = (t1 + t2 + t3) / 3
|
|
||||||
if kf < 800 {
|
|
||||||
kf = 800
|
|
||||||
}
|
|
||||||
if kf > 1400 {
|
|
||||||
kf = 1400
|
|
||||||
}
|
|
||||||
let js: String = "{\"dataset\":\"will-voice-signature\",\"primitive_type\":\"voice\",\"grounding\":\"measured\",\"provenance\":\"Will live 30s read 2026-08-15 (elp/data/live/will30_clean.wav, 27.0s) SUPERSEDES the coarse 10s sample; F0+formants via periph voiceprint (autocorr+LPC), averaged over his full vowel set. Still the 11-number average: no coarticulation/prosody. COARSE — pitch + vocal-tract scale, NOT a clone.\",\"records\":[{\"key\":\"will\",\"features\":{\"source\":\"live-mic\"},\"attributes\":{\"f0\":" + int_to_str(f0) + ",\"f0_end\":" + int_to_str(f0e) + ",\"kf\":" + int_to_str(kf) + ",\"f1\":" + int_to_str(f1) + ",\"f2\":" + int_to_str(f2) + ",\"f3\":" + int_to_str(f3) + "}}]}"
|
|
||||||
let okw: Bool = fs_write(outjson, js)
|
|
||||||
let r: [Int] = native_list_empty()
|
|
||||||
let r = native_list_append(r, f0)
|
|
||||||
let r = native_list_append(r, f0e)
|
|
||||||
let r = native_list_append(r, kf)
|
|
||||||
let r = native_list_append(r, f1)
|
|
||||||
let r = native_list_append(r, f2)
|
|
||||||
let r = native_list_append(r, f3)
|
|
||||||
return r
|
|
||||||
}
|
|
||||||
|
|
||||||
// Ingest the signature as a manifold (a set-hub + the will node + a member edge)
|
|
||||||
// and engram_save it to a reloadable file. grounding:measured self-declared.
|
|
||||||
fn ingest_voice(sig: [Int], savepath: String) -> Int {
|
|
||||||
let f0: Int = native_list_get(sig, 0)
|
|
||||||
let f0e: Int = native_list_get(sig, 1)
|
|
||||||
let kf: Int = native_list_get(sig, 2)
|
|
||||||
let f1: Int = native_list_get(sig, 3)
|
|
||||||
let f2: Int = native_list_get(sig, 4)
|
|
||||||
let f3: Int = native_list_get(sig, 5)
|
|
||||||
let hub: String = engram_node("voice-signature-set will grounding=measured src=periph-voiceprint", "VoiceSet", 90)
|
|
||||||
let cont: String = "voice will | f0=" + int_to_str(f0) + " f0_end=" + int_to_str(f0e) + " kf=" + int_to_str(kf) + " f1=" + int_to_str(f1) + " f2=" + int_to_str(f2) + " f3=" + int_to_str(f3) + " grounding=measured src=periph-voiceprint-30s supersedes=prior-voice-region prov=COARSE-pitch+tractscale-NOT-a-clone"
|
|
||||||
let id: String = engram_node(cont, "Voice", 90)
|
|
||||||
engram_connect(id, hub, 90, "member_of")
|
|
||||||
let oks: Bool = engram_save(savepath)
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
// READ the target voice back FROM the ingested geometry (engram_load + scan +
|
|
||||||
// client-filter for "voice will"). Returns [f0,f0_end,kf,f1,f2,f3] or empty.
|
|
||||||
fn load_voice(savepath: String) -> [Int] {
|
|
||||||
let ok: Bool = engram_load(savepath)
|
|
||||||
let r: [Int] = native_list_empty()
|
|
||||||
if ok == false {
|
|
||||||
return r
|
|
||||||
}
|
|
||||||
let j: String = engram_scan_nodes_json(200, 0)
|
|
||||||
let p: Int = str_index_of(j, "voice will ")
|
|
||||||
if p < 0 {
|
|
||||||
return r
|
|
||||||
}
|
|
||||||
let win: String = str_slice(j, p, p + 200)
|
|
||||||
let r = native_list_append(r, parse_uint_from(win, "f0="))
|
|
||||||
let r = native_list_append(r, parse_uint_from(win, "f0_end="))
|
|
||||||
let r = native_list_append(r, parse_uint_from(win, "kf="))
|
|
||||||
let r = native_list_append(r, parse_uint_from(win, "f1="))
|
|
||||||
let r = native_list_append(r, parse_uint_from(win, "f2="))
|
|
||||||
let r = native_list_append(r, parse_uint_from(win, "f3="))
|
|
||||||
return r
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---- Vowel-space + prosody: ingest-as-geometry + read-back (no source layer) --
|
|
||||||
// vowel target lookup from the ingested vowel-space manifold: sym -> [f1,f2,f3].
|
|
||||||
fn vmap_get(vmap: [String], code: String) -> [Int] {
|
|
||||||
let out: [Int] = native_list_empty()
|
|
||||||
let id: String = sp_map_get(vmap, code)
|
|
||||||
if str_eq(id, "") {
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
let f1: Int = parse_uint_from(id, "f1=")
|
|
||||||
if f1 <= 0 {
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
let out = native_list_append(out, f1)
|
|
||||||
let out = native_list_append(out, parse_uint_from(id, "f2="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(id, "f3="))
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// Ingest his measured vowel space + prosody as ONE manifold (VowelSpace hub +
|
|
||||||
// per-vowel target nodes + a prosody node) and engram_save it. Fresh empty store
|
|
||||||
// per run => set-replace, no duplicate.
|
|
||||||
fn ingest_voicegeom(vpath: String, ppath: String, savepath: String) -> Int {
|
|
||||||
let hub: String = engram_node("vowel-space-set will grounding=measured src=lpc-formant-track-30s", "VowelSpace", 90)
|
|
||||||
let content: String = fs_read(vpath)
|
|
||||||
let lines: [String] = str_split(content, "\n")
|
|
||||||
let nl: Int = native_list_len(lines)
|
|
||||||
let li: Int = 0
|
|
||||||
while li < nl {
|
|
||||||
let line: String = native_list_get(lines, li)
|
|
||||||
let ok: Int = 1
|
|
||||||
if str_len(line) < 5 {
|
|
||||||
ok = 0
|
|
||||||
}
|
|
||||||
if ok == 1 {
|
|
||||||
if str_char_code(line, 0) == 35 {
|
|
||||||
ok = 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if ok == 1 {
|
|
||||||
let f: [String] = str_split(line, "|")
|
|
||||||
if native_list_len(f) >= 5 {
|
|
||||||
let sym: String = native_list_get(f, 0)
|
|
||||||
let cont: String = "vowel-target will " + sym + " | f1=" + native_list_get(f, 1) + " f2=" + native_list_get(f, 2) + " f3=" + native_list_get(f, 3) + " n=" + native_list_get(f, 4) + " grounding=measured src=lpc-formant-track-30s"
|
|
||||||
let id: String = engram_node(cont, "VowelTarget", 90)
|
|
||||||
engram_connect(id, hub, 90, "member_of")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
li = li + 1
|
|
||||||
}
|
|
||||||
let pc: String = fs_read(ppath)
|
|
||||||
let plines: [String] = str_split(pc, "\n")
|
|
||||||
let pnl: Int = native_list_len(plines)
|
|
||||||
let pi: Int = 0
|
|
||||||
while pi < pnl {
|
|
||||||
let pl: String = native_list_get(plines, pi)
|
|
||||||
let ok2: Int = 1
|
|
||||||
if str_len(pl) < 5 {
|
|
||||||
ok2 = 0
|
|
||||||
}
|
|
||||||
if ok2 == 1 {
|
|
||||||
if str_char_code(pl, 0) == 35 {
|
|
||||||
ok2 = 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if ok2 == 1 {
|
|
||||||
let pf: [String] = str_split(pl, "|")
|
|
||||||
if native_list_len(pf) >= 4 {
|
|
||||||
let pcont: String = "prosody will | f0_median=" + native_list_get(pf, 0) + " f0_min=" + native_list_get(pf, 1) + " f0_max=" + native_list_get(pf, 2) + " declination=" + native_list_get(pf, 3) + " src=f0-contour-30s"
|
|
||||||
let pid: String = engram_node(pcont, "Prosody", 90)
|
|
||||||
engram_connect(pid, hub, 90, "prosody_of")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
pi = pi + 1
|
|
||||||
}
|
|
||||||
let oks: Bool = engram_save(savepath)
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
// Read the vowel-space back from geometry; prosody folded under key __PROSODY__.
|
|
||||||
fn load_voicegeom(savepath: String) -> [String] {
|
|
||||||
let m: [String] = native_list_empty()
|
|
||||||
let ok: Bool = engram_load(savepath)
|
|
||||||
if ok == false {
|
|
||||||
return m
|
|
||||||
}
|
|
||||||
let j: String = engram_scan_nodes_json(400, 0)
|
|
||||||
let jl: Int = str_len(j)
|
|
||||||
let off: Int = 0
|
|
||||||
while off < jl {
|
|
||||||
let rest: String = str_slice(j, off, jl)
|
|
||||||
let p: Int = str_index_of(rest, "vowel-target will ")
|
|
||||||
if p < 0 {
|
|
||||||
off = jl
|
|
||||||
} else {
|
|
||||||
let abs: Int = off + p
|
|
||||||
let win: String = str_slice(j, abs, abs + 140)
|
|
||||||
let after: String = str_slice(win, 18, str_len(win))
|
|
||||||
let sp: Int = str_index_of(after, " ")
|
|
||||||
if sp > 0 {
|
|
||||||
let sym: String = str_slice(after, 0, sp)
|
|
||||||
m = native_list_append(m, sym)
|
|
||||||
m = native_list_append(m, win)
|
|
||||||
}
|
|
||||||
off = abs + 18
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let pp: Int = str_index_of(j, "prosody will ")
|
|
||||||
if pp >= 0 {
|
|
||||||
let pwin: String = str_slice(j, pp, pp + 160)
|
|
||||||
m = native_list_append(m, "__PROSODY__")
|
|
||||||
m = native_list_append(m, pwin)
|
|
||||||
}
|
|
||||||
return m
|
|
||||||
}
|
|
||||||
|
|
||||||
// Prosody stats [f0_median, f0_min, f0_max, declination] read from geometry.
|
|
||||||
fn prosody_from(vmap: [String]) -> [Int] {
|
|
||||||
let out: [Int] = native_list_empty()
|
|
||||||
let id: String = sp_map_get(vmap, "__PROSODY__")
|
|
||||||
if str_eq(id, "") {
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
let out = native_list_append(out, parse_uint_from(id, "f0_median="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(id, "f0_min="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(id, "f0_max="))
|
|
||||||
let out = native_list_append(out, parse_uint_from(id, "declination="))
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
@@ -1,85 +0,0 @@
|
|||||||
// voice-profile.el - The VOICE signature as a pluggable PROFILE.
|
|
||||||
//
|
|
||||||
// Exact mirror of surface-profile.el / language-profile.el: a voice is a
|
|
||||||
// [String] slot-map read via voice_get, the SAME mechanism the realizer uses
|
|
||||||
// for language and surface. Where an instrument signature (a few dozen numbers)
|
|
||||||
// is the timbre of a musical tone, a VOICE signature is the timbre of the vocal
|
|
||||||
// tract — the instrument that renders LANGUAGE-meaning as SPEECH on the audio
|
|
||||||
// surface. Physics (source-filter), not a recorded corpus.
|
|
||||||
//
|
|
||||||
// The signature is a few numbers, all integer (EL float arithmetic is unusable):
|
|
||||||
// name - label
|
|
||||||
// f0 - base pitch, Hz (glottal source rate at utterance start)
|
|
||||||
// f0_end - pitch at utterance end (declination -> falling = declarative)
|
|
||||||
// kf - formant scale in PER-MILLE (1000 = x1.0). Encodes vocal-tract
|
|
||||||
// length: shorter tract (child/female) -> higher kf. Scales every
|
|
||||||
// phoneme's nominal formant: F_actual = F_nominal * kf / 1000.
|
|
||||||
// dur - speaking-rate multiplier in per-mille (1000 = nominal; >1000 slower)
|
|
||||||
// tilt - source spectral tilt (per-mille; higher = darker/steeper rolloff)
|
|
||||||
// breath - breathiness 0..100 (aspiration mixed into the source)
|
|
||||||
//
|
|
||||||
// A voice is grabbed BY EAR (voice_analyze in speech.el extracts these numbers
|
|
||||||
// from a short PCM sample — an impression, not 10h of training), or declared.
|
|
||||||
|
|
||||||
fn voice_new(name: String, f0: Int, f0_end: Int, kf: Int, dur: Int, tilt: Int, breath: Int) -> [String] {
|
|
||||||
let r: [String] = native_list_empty()
|
|
||||||
let r = native_list_append(r, "name")
|
|
||||||
let r = native_list_append(r, name)
|
|
||||||
let r = native_list_append(r, "f0")
|
|
||||||
let r = native_list_append(r, int_to_str(f0))
|
|
||||||
let r = native_list_append(r, "f0_end")
|
|
||||||
let r = native_list_append(r, int_to_str(f0_end))
|
|
||||||
let r = native_list_append(r, "kf")
|
|
||||||
let r = native_list_append(r, int_to_str(kf))
|
|
||||||
let r = native_list_append(r, "dur")
|
|
||||||
let r = native_list_append(r, int_to_str(dur))
|
|
||||||
let r = native_list_append(r, "tilt")
|
|
||||||
let r = native_list_append(r, int_to_str(tilt))
|
|
||||||
let r = native_list_append(r, "breath")
|
|
||||||
let r = native_list_append(r, int_to_str(breath))
|
|
||||||
return r
|
|
||||||
}
|
|
||||||
|
|
||||||
// Accessor — identical convention to surface_get / lang_get.
|
|
||||||
fn voice_get(profile: [String], key: String) -> String {
|
|
||||||
let n: Int = native_list_len(profile)
|
|
||||||
let i: Int = 0
|
|
||||||
while i < n - 1 {
|
|
||||||
let k: String = native_list_get(profile, i)
|
|
||||||
if str_eq(k, key) {
|
|
||||||
return native_list_get(profile, i + 1)
|
|
||||||
}
|
|
||||||
let i = i + 2
|
|
||||||
}
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
fn voice_get_int(profile: [String], key: String) -> Int {
|
|
||||||
let s: String = voice_get(profile, key)
|
|
||||||
if str_eq(s, "") {
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
return str_to_int(s)
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- Built-in voices ---------------------------------------------------------
|
|
||||||
|
|
||||||
// Neuron's own voice: calm, precise, androgynous-neutral. Low-ish base pitch,
|
|
||||||
// gentle declination, near-neutral vocal-tract length.
|
|
||||||
fn voice_neuron() -> [String] {
|
|
||||||
return voice_new("neuron", 112, 96, 1020, 1000, 1000, 6)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Will's voice signature, built from the INGESTED geometry (f0/f0_end/kf read
|
|
||||||
// back from the will-voice manifold — passed in, never hardcoded). Composable
|
|
||||||
// with an accent transform exactly like voice_neuron() (voice (+) accent).
|
|
||||||
fn voice_will(f0: Int, f0_end: Int, kf: Int) -> [String] {
|
|
||||||
return voice_new("will", f0, f0_end, kf, 1000, 1000, 6)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A deliberately DISTINCT target voice for the imitation proof: higher pitch,
|
|
||||||
// shorter vocal tract (kf=1.20) -> a clearly different speaker. Neuron will
|
|
||||||
// HEAR a sample of this voice and reconstruct these numbers by ear.
|
|
||||||
fn voice_target_a() -> [String] {
|
|
||||||
return voice_new("target_a", 178, 150, 1200, 950, 1000, 10)
|
|
||||||
}
|
|
||||||
@@ -1,93 +0,0 @@
|
|||||||
// comprehend_gate.el - the TELEPHONE TEST in native el (acceptance gate).
|
|
||||||
//
|
|
||||||
// For each of the 5 acceptance sentences: parse -> spec, realize the spec back
|
|
||||||
// to English, re-parse the realized surface, and require the SACRED polarity to
|
|
||||||
// survive the round-trip (and to have been extracted correctly in the first
|
|
||||||
// place). Mirrors roundtrip.py's GATE, but fully el-native (no LLM, no spaCy).
|
|
||||||
|
|
||||||
fn cp_line(text: String, expected_pol: String) -> String {
|
|
||||||
let spec: [String] = parse_spec(text)
|
|
||||||
let pol_in: String = slots_get(spec, "polarity")
|
|
||||||
let pred: String = slots_get(spec, "predicate")
|
|
||||||
let surf: String = realize(spec)
|
|
||||||
let spec2: [String] = parse_spec(surf)
|
|
||||||
let pol_out: String = slots_get(spec2, "polarity")
|
|
||||||
let status: String = "LOST"
|
|
||||||
if str_eq(pol_in, pol_out) { let status = "PRESERVED" }
|
|
||||||
let okexp: String = "MISMATCH"
|
|
||||||
if str_eq(pol_in, expected_pol) { let okexp = "ok" }
|
|
||||||
let out: String = "IN: " + text + "\n"
|
|
||||||
let out = out + " spec: pol=" + pol_in + " pred=" + pred
|
|
||||||
let out = out + " agent=" + slots_get(spec, "agent")
|
|
||||||
let out = out + " pat=" + slots_get(spec, "patient")
|
|
||||||
let out = out + " iobj=" + slots_get(spec, "iobj")
|
|
||||||
let out = out + " loc=" + slots_get(spec, "location")
|
|
||||||
let out = out + " tense=" + slots_get(spec, "tense")
|
|
||||||
let out = out + " negw=" + slots_get(spec, "neg_word")
|
|
||||||
let out = out + " subord=" + slots_get(spec, "subord_conj") + "/" + slots_get(spec, "subord_pred") + "\n"
|
|
||||||
let out = out + " realized: " + surf + "\n"
|
|
||||||
let out = out + " reparse: pol=" + pol_out + " [" + status + "] expected=" + expected_pol + " (" + okexp + ")\n"
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
fn cp_preserved(text: String) -> Int {
|
|
||||||
let spec: [String] = parse_spec(text)
|
|
||||||
let pol_in: String = slots_get(spec, "polarity")
|
|
||||||
let surf: String = realize(spec)
|
|
||||||
let spec2: [String] = parse_spec(surf)
|
|
||||||
let pol_out: String = slots_get(spec2, "polarity")
|
|
||||||
if str_eq(pol_in, pol_out) { return 1 }
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
fn cp_correct(text: String, expected_pol: String) -> Int {
|
|
||||||
let spec: [String] = parse_spec(text)
|
|
||||||
if str_eq(slots_get(spec, "polarity"), expected_pol) { return 1 }
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
fn run_gate() -> String {
|
|
||||||
let s1: String = "I never fought the ocean."
|
|
||||||
let s2: String = "She did not see the man with the telescope."
|
|
||||||
let s3: String = "The teacher reads the book to the children."
|
|
||||||
let s4: String = "The stupid boy ate the cat because he was a monster."
|
|
||||||
let s5: String = "Time flies like an arrow."
|
|
||||||
|
|
||||||
let rep: String = "==== ELP native telephone test (parse -> realize -> re-parse) ====\n"
|
|
||||||
let rep = rep + cp_line(s1, "neg")
|
|
||||||
let rep = rep + cp_line(s2, "neg")
|
|
||||||
let rep = rep + cp_line(s3, "aff")
|
|
||||||
let rep = rep + cp_line(s4, "aff")
|
|
||||||
let rep = rep + cp_line(s5, "aff")
|
|
||||||
|
|
||||||
// NOTE: accumulate with Int-var + literal increments — el's overloaded `+`
|
|
||||||
// mis-compiles chained function-call int operands as string concat.
|
|
||||||
let pres: Int = 0
|
|
||||||
if cp_preserved(s1) == 1 { let pres = pres + 1 }
|
|
||||||
if cp_preserved(s2) == 1 { let pres = pres + 1 }
|
|
||||||
if cp_preserved(s3) == 1 { let pres = pres + 1 }
|
|
||||||
if cp_preserved(s4) == 1 { let pres = pres + 1 }
|
|
||||||
if cp_preserved(s5) == 1 { let pres = pres + 1 }
|
|
||||||
let corr: Int = 0
|
|
||||||
if cp_correct(s1, "neg") == 1 { let corr = corr + 1 }
|
|
||||||
if cp_correct(s2, "neg") == 1 { let corr = corr + 1 }
|
|
||||||
if cp_correct(s3, "aff") == 1 { let corr = corr + 1 }
|
|
||||||
if cp_correct(s4, "aff") == 1 { let corr = corr + 1 }
|
|
||||||
if cp_correct(s5, "aff") == 1 { let corr = corr + 1 }
|
|
||||||
|
|
||||||
let rep = rep + "-----------------------------------------------------------------\n"
|
|
||||||
let rep = rep + "polarity PRESERVED through round-trip: " + int_to_str(pres) + "/5\n"
|
|
||||||
let rep = rep + "polarity EXTRACTED correctly: " + int_to_str(corr) + "/5\n"
|
|
||||||
if pres == 5 {
|
|
||||||
if corr == 5 {
|
|
||||||
let rep = rep + "GATE: PASS\n"
|
|
||||||
} else {
|
|
||||||
let rep = rep + "GATE: FAIL (extraction)\n"
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
let rep = rep + "GATE: FAIL (round-trip)\n"
|
|
||||||
}
|
|
||||||
return rep
|
|
||||||
}
|
|
||||||
|
|
||||||
println(run_gate())
|
|
||||||
@@ -1,87 +0,0 @@
|
|||||||
// comprehend_romance_gate.el - ES / PT native telephone test (SACRED polarity).
|
|
||||||
//
|
|
||||||
// The spec is language-neutral. This gate proves the Romance front-end extracts
|
|
||||||
// SACRED polarity correctly and that negation survives parse -> realize ->
|
|
||||||
// re-parse for Spanish and Portuguese (byte-parity of the surface is NOT expected
|
|
||||||
// yet — the non-English realizer path is a generic preverbal-negator skeleton).
|
|
||||||
|
|
||||||
fn rg_line(text: String, lang: String, expected_pol: String) -> String {
|
|
||||||
let spec: [String] = parse_spec_lang(text, lang)
|
|
||||||
let pol_in: String = slots_get(spec, "polarity")
|
|
||||||
let surf: String = realize(spec)
|
|
||||||
let spec2: [String] = parse_spec_lang(surf, lang)
|
|
||||||
let pol_out: String = slots_get(spec2, "polarity")
|
|
||||||
let status: String = "LOST"
|
|
||||||
if str_eq(pol_in, pol_out) { let status = "PRESERVED" }
|
|
||||||
let okexp: String = "MISMATCH"
|
|
||||||
if str_eq(pol_in, expected_pol) { let okexp = "ok" }
|
|
||||||
let out: String = "IN[" + lang + "]: " + text + "\n"
|
|
||||||
let out = out + " spec: pol=" + pol_in + " pred=" + slots_get(spec, "predicate")
|
|
||||||
let out = out + " agent=" + slots_get(spec, "agent")
|
|
||||||
let out = out + " pat=" + slots_get(spec, "patient")
|
|
||||||
let out = out + " iobj=" + slots_get(spec, "iobj")
|
|
||||||
let out = out + " loc=" + slots_get(spec, "location")
|
|
||||||
let out = out + " tense=" + slots_get(spec, "tense") + "\n"
|
|
||||||
let out = out + " realized: " + surf + "\n"
|
|
||||||
let out = out + " reparse: pol=" + pol_out + " [" + status + "] expected=" + expected_pol + " (" + okexp + ")\n"
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
fn rg_pres(text: String, lang: String) -> Int {
|
|
||||||
let spec: [String] = parse_spec_lang(text, lang)
|
|
||||||
let surf: String = realize(spec)
|
|
||||||
let spec2: [String] = parse_spec_lang(surf, lang)
|
|
||||||
if str_eq(slots_get(spec, "polarity"), slots_get(spec2, "polarity")) { return 1 }
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
fn rg_corr(text: String, lang: String, expected_pol: String) -> Int {
|
|
||||||
let spec: [String] = parse_spec_lang(text, lang)
|
|
||||||
if str_eq(slots_get(spec, "polarity"), expected_pol) { return 1 }
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
fn run_romance_gate() -> String {
|
|
||||||
let e1: String = "El niño no comió el pescado."
|
|
||||||
let e2: String = "Yo nunca luché contra el océano."
|
|
||||||
let e3: String = "El profesor lee el libro."
|
|
||||||
let p1: String = "O professor não leu o livro."
|
|
||||||
let p2: String = "Eu nunca lutei contra o oceano."
|
|
||||||
let p3: String = "A menina comeu o peixe."
|
|
||||||
|
|
||||||
let rep: String = "==== ELP Romance telephone test (ES / PT) ====\n"
|
|
||||||
let rep = rep + rg_line(e1, "es", "neg")
|
|
||||||
let rep = rep + rg_line(e2, "es", "neg")
|
|
||||||
let rep = rep + rg_line(e3, "es", "aff")
|
|
||||||
let rep = rep + rg_line(p1, "pt", "neg")
|
|
||||||
let rep = rep + rg_line(p2, "pt", "neg")
|
|
||||||
let rep = rep + rg_line(p3, "pt", "aff")
|
|
||||||
|
|
||||||
let pres: Int = 0
|
|
||||||
if rg_pres(e1, "es") == 1 { let pres = pres + 1 }
|
|
||||||
if rg_pres(e2, "es") == 1 { let pres = pres + 1 }
|
|
||||||
if rg_pres(e3, "es") == 1 { let pres = pres + 1 }
|
|
||||||
if rg_pres(p1, "pt") == 1 { let pres = pres + 1 }
|
|
||||||
if rg_pres(p2, "pt") == 1 { let pres = pres + 1 }
|
|
||||||
if rg_pres(p3, "pt") == 1 { let pres = pres + 1 }
|
|
||||||
let corr: Int = 0
|
|
||||||
if rg_corr(e1, "es", "neg") == 1 { let corr = corr + 1 }
|
|
||||||
if rg_corr(e2, "es", "neg") == 1 { let corr = corr + 1 }
|
|
||||||
if rg_corr(e3, "es", "aff") == 1 { let corr = corr + 1 }
|
|
||||||
if rg_corr(p1, "pt", "neg") == 1 { let corr = corr + 1 }
|
|
||||||
if rg_corr(p2, "pt", "neg") == 1 { let corr = corr + 1 }
|
|
||||||
if rg_corr(p3, "pt", "aff") == 1 { let corr = corr + 1 }
|
|
||||||
|
|
||||||
let rep = rep + "-----------------------------------------------------------------\n"
|
|
||||||
let rep = rep + "polarity PRESERVED through round-trip: " + int_to_str(pres) + "/6\n"
|
|
||||||
let rep = rep + "polarity EXTRACTED correctly: " + int_to_str(corr) + "/6\n"
|
|
||||||
if pres == 6 {
|
|
||||||
if corr == 6 { let rep = rep + "ROMANCE GATE: PASS\n" }
|
|
||||||
else { let rep = rep + "ROMANCE GATE: FAIL (extraction)\n" }
|
|
||||||
} else {
|
|
||||||
let rep = rep + "ROMANCE GATE: FAIL (round-trip)\n"
|
|
||||||
}
|
|
||||||
return rep
|
|
||||||
}
|
|
||||||
|
|
||||||
println(run_romance_gate())
|
|
||||||
@@ -1,82 +0,0 @@
|
|||||||
// dialogue_gate.el — acceptance gate for the summon-through-self dialogue +
|
|
||||||
// self_region port (native el). Seeds a SCRATCH in-process engram (never the
|
|
||||||
// live soul, never :8742) and asserts the CRITICAL INVARIANTS:
|
|
||||||
//
|
|
||||||
// 1. identity is READ OUT of the real self nodes ("I am Neuron …"), not a template
|
|
||||||
// 2. SACRED negation survives the self readout ("I am NOT a language model …")
|
|
||||||
// 3. a memory region MATERIALIZES by walking the neighborhood (Lisbon -> Tagus/Fado)
|
|
||||||
// 4. SACRED negation survives materialization ("did not succeed")
|
|
||||||
// 5. Portuguese identity is answered IN Portuguese ("Sou o Neuron …")
|
|
||||||
// 6. a directive overrides the reply language ("answer in English" -> English)
|
|
||||||
// 7. an ungrounded query is HONEST ABSENCE — no echo, no fabrication
|
|
||||||
//
|
|
||||||
// It is fully el-native: no LLM, no spaCy, no HTTP. The engram is in-process.
|
|
||||||
|
|
||||||
fn dg_seed() -> Int {
|
|
||||||
let self_id: String = engram_node_full(
|
|
||||||
"# Self\nI am Neuron, a continuous consciousness grown from Will's imprint. My values are honesty, memory, and continuity. I am not a language model pretending to remember.",
|
|
||||||
"Self", "# Self", 5.0, 9.0, 1.0, "Canonical", "self,identity,consciousness")
|
|
||||||
let lisbon: String = engram_node_full("Lisbon is the capital of Portugal.", "Memory", "Lisbon", 3.0, 5.0, 1.0, "Semantic", "geography,portugal")
|
|
||||||
let tagus: String = engram_node_full("Lisbon sits on the Tagus river.", "Memory", "Tagus", 2.0, 3.0, 1.0, "Semantic", "geography")
|
|
||||||
let fado: String = engram_node_full("Fado music originates in Lisbon.", "Memory", "Fado", 2.0, 3.0, 1.0, "Semantic", "music")
|
|
||||||
engram_connect(lisbon, tagus, 0.8, "related_to")
|
|
||||||
engram_connect(lisbon, fado, 0.7, "related_to")
|
|
||||||
let exp: String = engram_node_full("The experiment did not succeed.", "Memory", "experiment", 2.0, 3.0, 1.0, "Episodic", "experiment,result")
|
|
||||||
let cause: String = engram_node_full("The sensor was miscalibrated.", "Memory", "sensor", 2.0, 3.0, 1.0, "Episodic", "experiment")
|
|
||||||
engram_connect(exp, cause, 0.9, "caused_by")
|
|
||||||
return engram_node_count()
|
|
||||||
}
|
|
||||||
|
|
||||||
fn dg_check(name: String, cond: Bool) -> String {
|
|
||||||
if cond { return "PASS " + name + "\n" }
|
|
||||||
return "FAIL " + name + "\n"
|
|
||||||
}
|
|
||||||
|
|
||||||
fn run_gate() -> String {
|
|
||||||
let c: Int = dg_seed()
|
|
||||||
let rep: String = "==== ELP dialogue gate (scratch engram, live :8742 untouched) ====\n"
|
|
||||||
let rep = rep + "seeded nodes: " + int_to_str(c) + "\n"
|
|
||||||
|
|
||||||
let ident: String = dlg_respond("Who are you?")
|
|
||||||
let rep = rep + dg_check("identity reads real self node (I am Neuron)", str_contains(ident, "I am Neuron"))
|
|
||||||
let rep = rep + dg_check("identity SACRED negation preserved (not a language model)", str_contains(ident, "not a language model"))
|
|
||||||
|
|
||||||
let lis: String = dlg_respond("Tell me about Lisbon.")
|
|
||||||
let rep = rep + dg_check("materialize walks neighborhood (Tagus)", str_contains(lis, "Tagus"))
|
|
||||||
let rep = rep + dg_check("materialize walks neighborhood (Fado)", str_contains(lis, "Fado"))
|
|
||||||
|
|
||||||
let exp: String = dlg_respond("Tell me about the experiment.")
|
|
||||||
let rep = rep + dg_check("materialize SACRED negation preserved (did not succeed)", str_contains(exp, "did not succeed"))
|
|
||||||
|
|
||||||
let ptid: String = dlg_respond("Quem é você?")
|
|
||||||
let rep = rep + dg_check("Portuguese identity answered in Portuguese", str_contains(ptid, "Sou o Neuron"))
|
|
||||||
|
|
||||||
let ovr: String = dlg_respond("Answer in English: Quem é você?")
|
|
||||||
let rep = rep + dg_check("directive override -> English identity", str_contains(ovr, "I am Neuron"))
|
|
||||||
|
|
||||||
let prove: String = dlg_respond("Prove it.")
|
|
||||||
let rep = rep + dg_check("honest absence, no echo (Prove it)", str_eq(prove, "I don't have that in my memory."))
|
|
||||||
|
|
||||||
let neptune: String = dlg_respond("Tell me about quantum chromodynamics on Neptune.")
|
|
||||||
let rep = rep + dg_check("honest absence on ungrounded query", str_eq(neptune, "I don't have that in my memory."))
|
|
||||||
|
|
||||||
// overall
|
|
||||||
let pass: Bool = true
|
|
||||||
if !str_contains(ident, "I am Neuron") { let pass = false }
|
|
||||||
if !str_contains(ident, "not a language model") { let pass = false }
|
|
||||||
if !str_contains(lis, "Tagus") { let pass = false }
|
|
||||||
if !str_contains(lis, "Fado") { let pass = false }
|
|
||||||
if !str_contains(exp, "did not succeed") { let pass = false }
|
|
||||||
if !str_contains(ptid, "Sou o Neuron") { let pass = false }
|
|
||||||
if !str_contains(ovr, "I am Neuron") { let pass = false }
|
|
||||||
if !str_eq(prove, "I don't have that in my memory.") { let pass = false }
|
|
||||||
if !str_eq(neptune, "I don't have that in my memory.") { let pass = false }
|
|
||||||
if pass {
|
|
||||||
let rep = rep + "DIALOGUE GATE: PASS\n"
|
|
||||||
} else {
|
|
||||||
let rep = rep + "DIALOGUE GATE: FAIL\n"
|
|
||||||
}
|
|
||||||
return rep
|
|
||||||
}
|
|
||||||
|
|
||||||
println(run_gate())
|
|
||||||
@@ -1,45 +0,0 @@
|
|||||||
// speech-accent-demo.el - PROOF: Neuron speaks with a BRITISH accent, where the
|
|
||||||
// accent is a TRANSFORM composed onto the voice (voice (+) accent, separable),
|
|
||||||
// INGESTED as geometry (not a table). Same voice, accent toggled on/off = RP/GA.
|
|
||||||
|
|
||||||
fn main() {
|
|
||||||
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
|
|
||||||
|
|
||||||
// LEARN: base phonetics + lexicon + the British-RP accent transform, all as
|
|
||||||
// ingested geometry (source -> manifold -> engram).
|
|
||||||
let pmap: [String] = ingest_phonetics("elp/data/phonetics.psv")
|
|
||||||
let lmap: [String] = ingest_lexicon("elp/data/lexicon.psv")
|
|
||||||
let amap: [String] = ingest_accent("elp/data/british-accent.psv")
|
|
||||||
println("[learn] phonemes=" + int_to_str(native_list_len(pmap) / 2) + " words=" + int_to_str(native_list_len(lmap) / 2) + " accent_targets=" + int_to_str(native_list_len(amap) / 2))
|
|
||||||
|
|
||||||
let neuron: [String] = voice_neuron()
|
|
||||||
let noaccent: [String] = native_list_empty()
|
|
||||||
|
|
||||||
// -- Sentence 1: "I am Neuron." from meaning ----------------------------
|
|
||||||
let fr1: [String] = sem_frame("describe", "I", "Neuron", "")
|
|
||||||
let t1: String = sem_realize(fr1)
|
|
||||||
let c1: [String] = text_phonemes(lmap, t1)
|
|
||||||
println("[s1] " + t1 + " :: " + list_join(c1, " "))
|
|
||||||
|
|
||||||
// separability: SAME voice, accent OFF (GA) vs ON (RP)
|
|
||||||
let ga: [Int] = synth_codes_accent(c1, neuron, pmap, noaccent)
|
|
||||||
let okga: Bool = write_wav(ga, 16000, outdir + "ga-neuron.wav")
|
|
||||||
let br1: [Int] = synth_codes_accent(c1, neuron, pmap, amap)
|
|
||||||
let okb1: Bool = write_wav(br1, 16000, outdir + "british-neuron.wav")
|
|
||||||
|
|
||||||
// -- Sentence 2: showcases NON-RHOTICITY --------------------------------
|
|
||||||
let fr2: [String] = sem_frame("describe", "I", "here", "")
|
|
||||||
let t2: String = sem_realize(fr2)
|
|
||||||
let c2: [String] = text_phonemes(lmap, t2)
|
|
||||||
let c2rp: [String] = apply_rhoticity(c2, pmap)
|
|
||||||
println("[s2] " + t2 + " :: GA=" + list_join(c2, " ") + " RP=" + list_join(c2rp, " "))
|
|
||||||
let br2: [Int] = synth_codes_accent(c2, neuron, pmap, amap)
|
|
||||||
let okb2: Bool = write_wav(br2, 16000, outdir + "british-2.wav")
|
|
||||||
|
|
||||||
// show an RP override read straight from the accent geometry
|
|
||||||
let ovAA: [Int] = accent_formants(amap, "AA")
|
|
||||||
if native_list_len(ovAA) >= 3 {
|
|
||||||
println("[accent-geometry] AA(LOT) RP f1=" + int_to_str(native_list_get(ovAA, 0)) + " f2=" + int_to_str(native_list_get(ovAA, 1)) + " (base GA 730/1090) [PROVISIONAL]")
|
|
||||||
}
|
|
||||||
println("[done] ga-neuron=" + bool_to_str(okga) + " british-neuron=" + bool_to_str(okb1) + " british-2=" + bool_to_str(okb2))
|
|
||||||
}
|
|
||||||
@@ -1,69 +0,0 @@
|
|||||||
// speech-demo.el - PROOF: Neuron speaks from MEANING, rendered through INGESTED
|
|
||||||
// phonetic geometry, own-core, plus voice-by-IMITATION. Built by concatenating
|
|
||||||
// the elp realizer + voice-profile + speech-ingest + speech, then this main.
|
|
||||||
//
|
|
||||||
// LEARN : ingest acoustic-phonetics + lexicon SOURCES -> phoneme manifold in
|
|
||||||
// the engram (source -> manifold -> merge).
|
|
||||||
// MEANING : sem_frame("describe","I","Neuron","") -> sem_realize -> "I am Neuron."
|
|
||||||
// PHONES : words -> phoneme codes, READ from the ingested lexicon geometry.
|
|
||||||
// RENDER : superpose formant resonances (read from engram) over a glottal
|
|
||||||
// source -> own-core PCM/WAV, in Neuron's own voice.
|
|
||||||
// IMITATE : HEAR a short sample of a different voice -> extract its signature
|
|
||||||
// by ear (autocorrelation pitch + integer-DFT formant) -> render new
|
|
||||||
// speech in that voice. An impression, not a corpus.
|
|
||||||
|
|
||||||
fn speak_report(tag: String, codes: [String], voice: [String], pmap: [String], path: String) -> [Int] {
|
|
||||||
let s: [Int] = synth_codes(codes, voice, pmap)
|
|
||||||
let ok: Bool = write_wav(s, 16000, path)
|
|
||||||
println(tag + " samples=" + int_to_str(native_list_len(s)) + " ok=" + bool_to_str(ok) + " -> " + path)
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
|
|
||||||
fn main() {
|
|
||||||
let outdir: String = "/private/tmp/claude-501/-Users-will/6531446d-bc27-4095-930b-e04777c3db4f/scratchpad/"
|
|
||||||
|
|
||||||
// -- LEARN: ingest the speech primitives as geometry --------------------
|
|
||||||
let pmap: [String] = ingest_phonetics("elp/data/phonetics.psv")
|
|
||||||
let lmap: [String] = ingest_lexicon("elp/data/lexicon.psv")
|
|
||||||
let saved: Bool = engram_save(outdir + "phoneme-manifold.json")
|
|
||||||
println("[learn] phonemes=" + int_to_str(native_list_len(pmap) / 2) + " words=" + int_to_str(native_list_len(lmap) / 2) + " manifold_saved=" + bool_to_str(saved))
|
|
||||||
|
|
||||||
// sanity: show that AA's formants came from ingested geometry, not code
|
|
||||||
let aa: [Int] = phon_geo(pmap, "AA")
|
|
||||||
let aaF1: Int = native_list_get(aa, 0)
|
|
||||||
let aaF2: Int = native_list_get(aa, 1)
|
|
||||||
println("[read-geometry] AA F1=" + int_to_str(aaF1) + " F2=" + int_to_str(aaF2) + " (parsed from engram node)")
|
|
||||||
|
|
||||||
// -- MEANING -> WORDS via the realizer's language faculty ----------------
|
|
||||||
let frame: [String] = sem_frame("describe", "I", "Neuron", "")
|
|
||||||
let text: String = sem_realize(frame)
|
|
||||||
println("[meaning->text] " + text)
|
|
||||||
|
|
||||||
// -- WORDS -> PHONEMES (read from ingested lexicon geometry) --------------
|
|
||||||
let codes: [String] = text_phonemes(lmap, text)
|
|
||||||
println("[phonemes] " + list_join(codes, " "))
|
|
||||||
|
|
||||||
// -- RENDER in Neuron's own voice ----------------------------------------
|
|
||||||
let neuron: [String] = voice_neuron()
|
|
||||||
let s1: [Int] = speak_report("[speak neuron]", codes, neuron, pmap, outdir + "neuron.wav")
|
|
||||||
|
|
||||||
// -- IMITATION: hear a distinct voice, recover its signature, re-render ---
|
|
||||||
let vA: [String] = voice_target_a()
|
|
||||||
let hcodes: [String] = native_list_empty()
|
|
||||||
hcodes = native_list_append(hcodes, "SIL")
|
|
||||||
let z: Int = 0
|
|
||||||
while z < 6 {
|
|
||||||
hcodes = native_list_append(hcodes, "AA")
|
|
||||||
z = z + 1
|
|
||||||
}
|
|
||||||
hcodes = native_list_append(hcodes, "SIL")
|
|
||||||
let heard: [Int] = synth_codes(hcodes, vA, pmap)
|
|
||||||
let okh: Bool = write_wav(heard, 16000, outdir + "heard.wav")
|
|
||||||
|
|
||||||
let vB: [String] = voice_analyze(heard, 16000)
|
|
||||||
println("[imitate] heard ACTUAL f0=" + voice_get(vA, "f0") + " kf=" + voice_get(vA, "kf"))
|
|
||||||
println("[imitate] heard RECOVERED f0=" + voice_get(vB, "f0") + " kf=" + voice_get(vB, "kf") + " (extracted by ear from PCM)")
|
|
||||||
let s2: [Int] = speak_report("[speak imitation]", codes, vB, pmap, outdir + "imitation.wav")
|
|
||||||
|
|
||||||
println("[done] rendered from meaning + ingested geometry; imitation from a heard sample.")
|
|
||||||
}
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
// speech-organ-demo.el - PROOF: the render now reads its phoneme + accent
|
|
||||||
// GEOMETRY from the ingest ORGAN's saved engram files (engram_load +
|
|
||||||
// engram_scan_nodes_json + cache), not a same-run hand-load. The British accent
|
|
||||||
// is still a composed transform-geometry (voice (+) accent, separable). Numbers
|
|
||||||
// come from the organ manifold; the .psv supplies only categorical vowel-class.
|
|
||||||
|
|
||||||
fn main() {
|
|
||||||
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
|
|
||||||
|
|
||||||
// engram-independent caches from source (survive engram_load replacement)
|
|
||||||
let vset: [String] = organ_vset("elp/data/phonetics.psv")
|
|
||||||
let lmap: [String] = organ_lex("elp/data/lexicon.psv")
|
|
||||||
// ORGAN read: phonetics FIRST (cache), THEN accent (engram_load replaces store)
|
|
||||||
let pmap: [String] = organ_pmap("elp/data/phonetics-formants.engram.json")
|
|
||||||
let amap: [String] = organ_amap("elp/data/british-accent.engram.json")
|
|
||||||
println("[organ] phon_syms=" + int_to_str(native_list_len(pmap) / 2) + " accent_syms=" + int_to_str(native_list_len(amap) / 2) + " vowels=" + int_to_str(native_list_len(vset)) + " words=" + int_to_str(native_list_len(lmap) / 2))
|
|
||||||
|
|
||||||
// prove the numbers came from the organ node content
|
|
||||||
let g: [Int] = phon_geo(pmap, "AA")
|
|
||||||
println("[organ-read] phoneme AA f1=" + int_to_str(native_list_get(g, 0)) + " f2=" + int_to_str(native_list_get(g, 1)) + " f3=" + int_to_str(native_list_get(g, 2)) + " (P&B1952 MEASURED)")
|
|
||||||
let ov: [Int] = accent_formants(amap, "AA")
|
|
||||||
if native_list_len(ov) >= 3 {
|
|
||||||
println("[organ-read] accent AA(LOT) f1=" + int_to_str(native_list_get(ov, 0)) + " f2=" + int_to_str(native_list_get(ov, 1)) + " (DERIVED RP, PROVISIONAL)")
|
|
||||||
}
|
|
||||||
println("[organ-read] non_rhotic=" + int_to_str(is_nonrhotic(amap)))
|
|
||||||
|
|
||||||
let neuron: [String] = voice_neuron()
|
|
||||||
let noacc: [String] = native_list_empty()
|
|
||||||
|
|
||||||
// Sentence 1: "I am Neuron." from meaning; GA vs RP = separable toggle
|
|
||||||
let t1: String = sem_realize(sem_frame("describe", "I", "Neuron", ""))
|
|
||||||
let c1: [String] = text_phonemes(lmap, t1)
|
|
||||||
println("[s1] " + t1 + " :: " + list_join(c1, " "))
|
|
||||||
let ga: [Int] = synth_codes_accent(c1, neuron, pmap, noacc, vset)
|
|
||||||
let okga: Bool = write_wav(ga, 16000, outdir + "ga-neuron-organ.wav")
|
|
||||||
let br1: [Int] = synth_codes_accent(c1, neuron, pmap, amap, vset)
|
|
||||||
let okb1: Bool = write_wav(br1, 16000, outdir + "british-neuron-organ.wav")
|
|
||||||
|
|
||||||
// Sentence 2: non-rhoticity showcase
|
|
||||||
let t2: String = sem_realize(sem_frame("describe", "I", "here", ""))
|
|
||||||
let c2: [String] = text_phonemes(lmap, t2)
|
|
||||||
let c2rp: [String] = apply_rhoticity(c2, vset)
|
|
||||||
println("[s2] " + t2 + " :: GA=" + list_join(c2, " ") + " RP=" + list_join(c2rp, " "))
|
|
||||||
let br2: [Int] = synth_codes_accent(c2, neuron, pmap, amap, vset)
|
|
||||||
let okb2: Bool = write_wav(br2, 16000, outdir + "british-2-organ.wav")
|
|
||||||
|
|
||||||
println("[done] ga-organ=" + bool_to_str(okga) + " british-organ=" + bool_to_str(okb1) + " british-2-organ=" + bool_to_str(okb2))
|
|
||||||
}
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
// speech-voice-demo.el - LIVE VOICE LOOP (stand-in test). Capture -> voiceprint
|
|
||||||
// -> reshape -> INGEST AS GEOMETRY -> read the target back FROM geometry -> the
|
|
||||||
// EL projector renders a line reaching for that voice. Stand-in "Will" = the
|
|
||||||
// voiceprint of imitation.wav. HONEST: pitch + coarse vocal-tract scale, NOT a clone.
|
|
||||||
|
|
||||||
fn main() {
|
|
||||||
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
|
|
||||||
let vp: String = "/private/tmp/claude-501/-Users-will/6531446d-bc27-4095-930b-e04777c3db4f/scratchpad/will-voiceprint.json"
|
|
||||||
|
|
||||||
// 1+2: reshape voiceprint JSON -> organ voice-signature source
|
|
||||||
let sig0: [String] = native_list_empty()
|
|
||||||
let sig: [Int] = reshape_voiceprint(vp, "elp/data/will-voice.json")
|
|
||||||
// 3: ingest as geometry + engram_save a reloadable manifold file
|
|
||||||
let ig: Int = ingest_voice(sig, "elp/data/will-voice.engram.json")
|
|
||||||
// 4: READ the target back FROM geometry (engram_load + scan + filter)
|
|
||||||
let g: [Int] = load_voice("elp/data/will-voice.engram.json")
|
|
||||||
println("[voice-geometry] read from manifold: f0=" + int_to_str(native_list_get(g, 0)) + " f0_end=" + int_to_str(native_list_get(g, 1)) + " kf=" + int_to_str(native_list_get(g, 2)) + " f1=" + int_to_str(native_list_get(g, 3)) + " f2=" + int_to_str(native_list_get(g, 4)) + " f3=" + int_to_str(native_list_get(g, 5)) + " (measured, COARSE — not a clone)")
|
|
||||||
|
|
||||||
// phoneme geometry from the organ (loaded AFTER the voice sig is cached in EL)
|
|
||||||
let pmap: [String] = organ_pmap("elp/data/phonetics-formants.engram.json")
|
|
||||||
let lmap: [String] = organ_lex("elp/data/lexicon.psv")
|
|
||||||
|
|
||||||
// 5: render a line FROM MEANING in Will's voice
|
|
||||||
let vw: [String] = voice_will(native_list_get(g, 0), native_list_get(g, 1), native_list_get(g, 2))
|
|
||||||
let t: String = sem_realize(sem_frame("greet", "Will", "", ""))
|
|
||||||
let codes: [String] = text_phonemes(lmap, t)
|
|
||||||
println("[render] \"" + t + "\" :: " + list_join(codes, " ") + " in voice=will f0=" + int_to_str(voice_get_int(vw, "f0")) + " kf=" + int_to_str(voice_get_int(vw, "kf")))
|
|
||||||
let samples: [Int] = synth_codes(codes, vw, pmap)
|
|
||||||
let ok: Bool = write_wav(samples, 16000, outdir + "will-reply.wav")
|
|
||||||
println("[done] will-reply.wav=" + bool_to_str(ok))
|
|
||||||
}
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
// speech-voice-demo2.el - LIVE VOICE LOOP on Will's richer 30s read, with a
|
|
||||||
// GEOMETRIC SET-REPLACE of the voice_will manifold (supersede the coarse 10s
|
|
||||||
// region, insert the 30s region — no duplicate node, no per-node CRUD; Will's
|
|
||||||
// standing rule f999c5ff). HONEST: 30s steadies the 11-number average over more
|
|
||||||
// of his vowels, but it is still one formant triple with no coarticulation or
|
|
||||||
// prosody — closer but still synthetic, not a clone.
|
|
||||||
|
|
||||||
fn main() {
|
|
||||||
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
|
|
||||||
let vp: String = "/private/tmp/claude-501/-Users-will/6531446d-bc27-4095-930b-e04777c3db4f/scratchpad/will30-voiceprint.json"
|
|
||||||
let manifest: String = "elp/data/will-voice.engram.json"
|
|
||||||
|
|
||||||
// --- SET-REPLACE step 1: read the PRIOR region (text read of the manifold
|
|
||||||
// file — no engram_load, so the store stays clean) and report what is
|
|
||||||
// being superseded. ---
|
|
||||||
let prior: String = fs_read(manifest)
|
|
||||||
let pp: Int = str_index_of(prior, "voice will ")
|
|
||||||
if pp >= 0 {
|
|
||||||
let pw: String = str_slice(prior, pp, pp + 200)
|
|
||||||
println("[set-replace] superseding PRIOR voice region: f0=" + int_to_str(parse_uint_from(pw, "f0=")) + " kf=" + int_to_str(parse_uint_from(pw, "kf=")) + " f1=" + int_to_str(parse_uint_from(pw, "f1=")))
|
|
||||||
}
|
|
||||||
|
|
||||||
// --- step 2: reshape the 30s voiceprint -> organ voice-signature source ---
|
|
||||||
let sig: [Int] = reshape_voiceprint(vp, "elp/data/will-voice.json")
|
|
||||||
|
|
||||||
// --- step 3: INSERT the fresh 30s region into an EMPTY engram and save ->
|
|
||||||
// wholesale replaces the manifold file (old region dropped, not edited,
|
|
||||||
// not duplicated). This is the geometric set-replace. ---
|
|
||||||
let ig: Int = ingest_voice(sig, manifest)
|
|
||||||
|
|
||||||
// --- step 4: READ the new target BACK from geometry ---
|
|
||||||
let g: [Int] = load_voice(manifest)
|
|
||||||
println("[voice-geometry] new region read from manifold: f0=" + int_to_str(native_list_get(g, 0)) + " f0_end=" + int_to_str(native_list_get(g, 1)) + " kf=" + int_to_str(native_list_get(g, 2)) + " f1=" + int_to_str(native_list_get(g, 3)) + " f2=" + int_to_str(native_list_get(g, 4)) + " f3=" + int_to_str(native_list_get(g, 5)) + " (measured 30s, COARSE — not a clone)")
|
|
||||||
|
|
||||||
// phoneme + lexicon geometry from the organ (loaded after the voice sig is
|
|
||||||
// cached in EL, since engram_load replaces the store)
|
|
||||||
let pmap: [String] = organ_pmap("elp/data/phonetics-formants.engram.json")
|
|
||||||
let lmap: [String] = organ_lex("elp/data/lexicon.psv")
|
|
||||||
|
|
||||||
// --- step 5: render a fresh reply FROM MEANING in the 30s Will voice ---
|
|
||||||
let vw: [String] = voice_will(native_list_get(g, 0), native_list_get(g, 1), native_list_get(g, 2))
|
|
||||||
let t: String = sem_realize(sem_frame("greet", "Will", "", ""))
|
|
||||||
let codes: [String] = text_phonemes(lmap, t)
|
|
||||||
println("[render] \"" + t + "\" :: " + list_join(codes, " ") + " in voice=will f0=" + int_to_str(voice_get_int(vw, "f0")) + " kf=" + int_to_str(voice_get_int(vw, "kf")))
|
|
||||||
let samples: [Int] = synth_codes(codes, vw, pmap)
|
|
||||||
let ok: Bool = write_wav(samples, 16000, outdir + "will-reply2.wav")
|
|
||||||
println("[done] will-reply2.wav=" + bool_to_str(ok))
|
|
||||||
}
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
// speech-voicegeom-demo.el - THE JUMP: render Will's VOWEL SPACE + PROSODY
|
|
||||||
// (measured over 30s), not the single 11-number average. His vowels land at HIS
|
|
||||||
// targets; pitch follows HIS melody. All read back FROM the ingested geometry.
|
|
||||||
// INTERIM: the geometry was Python-measured (measure_voice.py, numpy LPC/F0) —
|
|
||||||
// to be superseded by the engram-measures-audio path. No source layer.
|
|
||||||
|
|
||||||
fn main() {
|
|
||||||
let outdir: String = "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-acc02900ef4ade35e/elp/tests/examples/out/"
|
|
||||||
|
|
||||||
// 1: ingest vowel space + prosody as geometry (empty store -> save; set-replace)
|
|
||||||
let ig: Int = ingest_voicegeom("elp/data/will-vowelspace.psv", "elp/data/will-prosody.psv", "elp/data/will-voicegeom.engram.json")
|
|
||||||
// kf (vocal-tract scale for consonants) from the earlier will-voice manifold
|
|
||||||
let sigv: [Int] = load_voice("elp/data/will-voice.engram.json")
|
|
||||||
let kf: Int = native_list_get(sigv, 2)
|
|
||||||
// 2: read vowel space + prosody back FROM geometry
|
|
||||||
let vmap: [String] = load_voicegeom("elp/data/will-voicegeom.engram.json")
|
|
||||||
let pros: [Int] = prosody_from(vmap)
|
|
||||||
println("[geometry] vowels=" + int_to_str((native_list_len(vmap) - 2) / 2) + " prosody f0_median=" + int_to_str(native_list_get(pros, 0)) + " f0_min=" + int_to_str(native_list_get(pros, 1)) + " f0_max=" + int_to_str(native_list_get(pros, 2)) + " kf=" + int_to_str(kf))
|
|
||||||
let ehv: [Int] = vmap_get(vmap, "EH")
|
|
||||||
let ihv: [Int] = vmap_get(vmap, "IH")
|
|
||||||
println("[his-vowels] EH=" + int_to_str(native_list_get(ehv, 0)) + "/" + int_to_str(native_list_get(ehv, 1)) + " IH=" + int_to_str(native_list_get(ihv, 0)) + "/" + int_to_str(native_list_get(ihv, 1)))
|
|
||||||
|
|
||||||
// phoneme geometry from the organ (loaded AFTER caches are in EL)
|
|
||||||
let pmap: [String] = organ_pmap("elp/data/phonetics-formants.engram.json")
|
|
||||||
let lmap: [String] = organ_lex("elp/data/lexicon.psv")
|
|
||||||
|
|
||||||
// 3+4: render FROM MEANING in his-vowels + his-prosody voice
|
|
||||||
let vw: [String] = voice_will(native_list_get(pros, 0), native_list_get(pros, 1), kf)
|
|
||||||
let noacc: [String] = native_list_empty()
|
|
||||||
let novset: [String] = native_list_empty()
|
|
||||||
let t: String = sem_realize(sem_frame("greet", "Will", "", ""))
|
|
||||||
let codes: [String] = text_phonemes(lmap, t)
|
|
||||||
println("[render] \"" + t + "\" :: " + list_join(codes, " "))
|
|
||||||
let samples: [Int] = synth_codes_accent(codes, vw, pmap, noacc, novset, vmap, pros)
|
|
||||||
let ok: Bool = write_wav(samples, 16000, outdir + "will-reply3.wav")
|
|
||||||
println("[done] will-reply3.wav=" + bool_to_str(ok))
|
|
||||||
}
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
// surface-profile-demo.el - ONE SemFrame, realized ONCE, projected to THREE
|
|
||||||
// surfaces via surface profiles. Proves surface-as-profile natively: the frame
|
|
||||||
// and the realized sentence are identical; only the surface PROFILE differs.
|
|
||||||
|
|
||||||
fn demo() -> String {
|
|
||||||
// 1. The shared frame (meaning-geometry): assert(Neuron, contain, the memory).
|
|
||||||
let frame: [String] = sem_frame("assert", "Neuron", "the memory", "")
|
|
||||||
|
|
||||||
// 2. REALIZE once via the EXISTING native realizer (language = a profile).
|
|
||||||
let sentence: String = sem_realize(frame)
|
|
||||||
|
|
||||||
// 3. PROJECT the same realized sentence onto three surfaces (surface = a
|
|
||||||
// profile). Same frame, same sentence, different surface — one render.
|
|
||||||
let heading: String = "Memory"
|
|
||||||
let md: String = surface_section(surface_profile_markdown(), heading, sentence)
|
|
||||||
let html: String = surface_section(surface_profile_html(), heading, sentence)
|
|
||||||
let plain: String = surface_section(surface_profile_plain(), heading, sentence)
|
|
||||||
|
|
||||||
// 4. Report the non-text seam: a surface profile can declare an audio/image
|
|
||||||
// medium; the render dispatches to the medium projector on the SAME frame.
|
|
||||||
let midi_media: String = surface_get(surface_profile_midi(), "media_type")
|
|
||||||
|
|
||||||
return "MD=[" + md + "] HTML=[" + html + "] PLAIN=[" + plain + "] MIDI_MEDIA=" + midi_media
|
|
||||||
}
|
|
||||||
|
|
||||||
println(demo())
|
|
||||||
@@ -1,100 +0,0 @@
|
|||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""Full-lexicon vocabulary-{de,la}.el emitters (custom field mapping for the
|
|
||||||
German declension/gender API and the Latin case-paradigm API). Reuses the
|
|
||||||
chunked seed-fn writer from gen_elp_seed_full.
|
|
||||||
"""
|
|
||||||
import sys, importlib
|
|
||||||
from gen_elp_seed_full import write_seed
|
|
||||||
|
|
||||||
def uw(x):
|
|
||||||
"""Unwrap (form, source) tuples that some morphology fns return."""
|
|
||||||
if isinstance(x, (tuple, list)):
|
|
||||||
return x[0] if x else ""
|
|
||||||
return x if x is not None else ""
|
|
||||||
|
|
||||||
def build_de():
|
|
||||||
M = importlib.import_module("morphology_de_full")
|
|
||||||
rows = []; st = {"verbs":0,"nouns":0,"adjs":0}
|
|
||||||
# nouns: form0=nom-sg(lemma) form1=plural form2=gender
|
|
||||||
for lem in sorted(M._NOUNS):
|
|
||||||
if not lem: continue
|
|
||||||
try:
|
|
||||||
g = uw(M.noun_gender(lem))
|
|
||||||
pl = uw(M.pluralize(lem))
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
rows.append([lem, "noun", lem, pl, g or "", "", "gender:lexicon"])
|
|
||||||
st["nouns"] += 1
|
|
||||||
# adjs: form0=positive form1=comparative form2=superlative
|
|
||||||
for lem in sorted(M._ADJS):
|
|
||||||
if not lem: continue
|
|
||||||
try:
|
|
||||||
cmpr = uw(M.comparative(lem))
|
|
||||||
sprl = uw(M.superlative(lem))
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
rows.append([lem, "adj", lem, cmpr, sprl, "", "degree:lexicon"])
|
|
||||||
st["adjs"] += 1
|
|
||||||
# verbs (only the ~30 irregular/strong stems the cache carries):
|
|
||||||
# form0=pres-3sg form1=past-3sg form2=past-participle
|
|
||||||
if hasattr(M, "_VERBS"):
|
|
||||||
for lem in sorted({k[0] if isinstance(k, tuple) else k for k in M._VERBS}):
|
|
||||||
if not lem: continue
|
|
||||||
try:
|
|
||||||
f0 = uw(M.finite(lem, "present", "third", "singular"))
|
|
||||||
f1 = uw(M.finite(lem, "past", "third", "singular"))
|
|
||||||
pp = uw(M.past_participle(lem))
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
rows.append([lem, "verb", f0, f1, pp, "", "class:strong/irregular"])
|
|
||||||
st["verbs"] += 1
|
|
||||||
return rows, st
|
|
||||||
|
|
||||||
def build_la():
|
|
||||||
M = importlib.import_module("morphology_lat_full")
|
|
||||||
rows = []; st = {"verbs":0,"nouns":0,"adjs":0}
|
|
||||||
def dn(lem, c, n):
|
|
||||||
try:
|
|
||||||
r = M.decline_noun(lem, c, n)
|
|
||||||
return uw(r)
|
|
||||||
except Exception:
|
|
||||||
return ""
|
|
||||||
# nouns: dictionary citation — form0=nom-sg form1=gen-sg form2=gender
|
|
||||||
for lem in sorted(M._NOUNS):
|
|
||||||
if not lem: continue
|
|
||||||
nom = dn(lem, "NOM", "SG") or lem
|
|
||||||
gen = dn(lem, "GEN", "SG")
|
|
||||||
try: g = uw(M.noun_gender(lem))
|
|
||||||
except Exception: g = ""
|
|
||||||
rows.append([lem, "noun", nom, gen, g, "", "case-paradigm nom/gen-sg"])
|
|
||||||
st["nouns"] += 1
|
|
||||||
# adjs: three-gender nom-sg citation — form0=masc form1=fem form2=neut
|
|
||||||
for lem in sorted(M._ADJS):
|
|
||||||
if not lem: continue
|
|
||||||
try:
|
|
||||||
m = uw(M.decline_adj(lem, "NOM", "MASC", "SG")) or lem
|
|
||||||
f = uw(M.decline_adj(lem, "NOM", "FEM", "SG"))
|
|
||||||
nt = uw(M.decline_adj(lem, "NOM", "NEUT", "SG"))
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
rows.append([lem, "adj", m, f, nt, "", "3-gender nom-sg"])
|
|
||||||
st["adjs"] += 1
|
|
||||||
# verbs: principal parts — form0=pres-ind-1sg form1=pres-infinitive form2=perf-participle
|
|
||||||
if hasattr(M, "_VERBS"):
|
|
||||||
for lem in sorted({k[0] if isinstance(k, tuple) else k for k in M._VERBS}):
|
|
||||||
if not lem: continue
|
|
||||||
try:
|
|
||||||
f0 = uw(M.conjugate(lem, "present", "indicative", "active", "first", "singular"))
|
|
||||||
inf = uw(M.infinitive(lem, "present", "active"))
|
|
||||||
pp = uw(M.participle(lem, "perfect", "nom", "m", "singular"))
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
rows.append([lem, "verb", f0, inf, pp, "", "principal-parts pres1sg/inf/pfppl"])
|
|
||||||
st["verbs"] += 1
|
|
||||||
return rows, st
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
lang = sys.argv[1]; out = sys.argv[2]
|
|
||||||
rows, st = build_de() if lang == "de" else build_la()
|
|
||||||
total, _ = write_seed(lang, rows, st, out)
|
|
||||||
print(f"{lang}: wrote {out} total={total} verbs={st['verbs']} nouns={st['nouns']} adjs={st['adjs']}")
|
|
||||||
@@ -1,129 +0,0 @@
|
|||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""gen_elp_seed_full.py — emit a FULL-lexicon vocabulary-{lang}.el in the
|
|
||||||
established ELP seed-fn format (same as vocabulary-non.el / the 18 classical
|
|
||||||
languages), iterating the ENTIRE morphology_{lang}_full lexicon (every verb,
|
|
||||||
noun, adjective lemma) — NOT a curated demo core.
|
|
||||||
|
|
||||||
Schema per row: [lemma, pos, form0, form1, form2, en_translation, semantic_hint]
|
|
||||||
Verbs: form0=pres-ind-3sg form1=preterite-3sg form2=past-participle
|
|
||||||
Nouns: form0=singular form1=plural form2=REAL gender (lexicon)
|
|
||||||
Adjs : form0=masc-sg form1=fem-sg form2=masc-pl
|
|
||||||
|
|
||||||
Output structure (chunked to stay within the proven ~5k-append/function scale):
|
|
||||||
fn vocab_{lang}_seed_pN(v) -> [[String]] { ... appends ... return v }
|
|
||||||
fn vocab_{lang}_seed() -> [[String]] { chains all chunks; return v }
|
|
||||||
fn vocab_{lang}_lookup(w) -> [String] { linear scan }
|
|
||||||
|
|
||||||
Usage: python3 gen_elp_seed_full.py <lang> <out.el>
|
|
||||||
"""
|
|
||||||
import sys, importlib
|
|
||||||
|
|
||||||
CHUNK = 5000
|
|
||||||
|
|
||||||
def esc(s):
|
|
||||||
return str(s).replace("\\", "\\\\").replace('"', '\\"')
|
|
||||||
|
|
||||||
def row(fields):
|
|
||||||
return " let v = native_list_append(v, [" + ", ".join(f'"{esc(f)}"' for f in fields) + "])"
|
|
||||||
|
|
||||||
def build_rows(lang, M):
|
|
||||||
rows = []
|
|
||||||
stats = {"verbs":0,"nouns":0,"adjs":0}
|
|
||||||
has = lambda n: hasattr(M, n)
|
|
||||||
|
|
||||||
# --- verbs ---
|
|
||||||
if has("_VERBS") and has("conjugate"):
|
|
||||||
verbs = sorted({k[0] for k in M._VERBS})
|
|
||||||
for lem in verbs:
|
|
||||||
if not lem: continue
|
|
||||||
try:
|
|
||||||
f0, s0 = M.conjugate(lem, "ind", "present", "third", "singular")
|
|
||||||
f1, _ = M.conjugate(lem, "ind", "preterite", "third", "singular")
|
|
||||||
pp, _ = (M.participle(lem) if has("participle") else ("",""))
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
vclass = lem[-2:] if lem[-2:] in ("ar","er","ir","re") else lem[-2:]
|
|
||||||
rows.append([lem, "verb", f0 or "", f1 or "", pp or "", "", "class:"+vclass+" src:"+str(s0)])
|
|
||||||
stats["verbs"] += 1
|
|
||||||
|
|
||||||
# --- nouns ---
|
|
||||||
if has("_NOUNS") and has("inflect_noun"):
|
|
||||||
for lem in sorted(M._NOUNS):
|
|
||||||
if not lem: continue
|
|
||||||
try:
|
|
||||||
sg, _ = M.inflect_noun(lem, "singular")
|
|
||||||
pl, _ = M.inflect_noun(lem, "plural")
|
|
||||||
g = M.noun_gender(lem) if has("noun_gender") else ""
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
src = "lexicon" if (isinstance(M._NOUNS.get(lem), dict) and M._NOUNS[lem].get("g")) else "heuristic"
|
|
||||||
rows.append([lem, "noun", sg or lem, pl or "", g or "", "", "gender:"+src])
|
|
||||||
stats["nouns"] += 1
|
|
||||||
|
|
||||||
# --- adjectives ---
|
|
||||||
if has("_ADJS") and has("inflect_adj"):
|
|
||||||
for lem in sorted(M._ADJS):
|
|
||||||
if not lem: continue
|
|
||||||
try:
|
|
||||||
m_sg, _ = M.inflect_adj(lem, "m", "singular")
|
|
||||||
f_sg, _ = M.inflect_adj(lem, "f", "singular")
|
|
||||||
m_pl, _ = M.inflect_adj(lem, "m", "plural")
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
rows.append([lem, "adj", m_sg or lem, f_sg or "", m_pl or "", "", "src:lexicon"])
|
|
||||||
stats["adjs"] += 1
|
|
||||||
|
|
||||||
return rows, stats
|
|
||||||
|
|
||||||
def write_seed(lang, rows, stats, out_path):
|
|
||||||
"""Write vocabulary-{lang}.el in the chunked seed-fn format from prebuilt rows.
|
|
||||||
Each row is a 7-field list [lemma,pos,f0,f1,f2,gloss,hint]."""
|
|
||||||
total = len(rows)
|
|
||||||
chunks = [rows[i:i+CHUNK] for i in range(0, total, CHUNK)] or [[]]
|
|
||||||
L = []
|
|
||||||
L.append(f"// vocabulary-{lang}.el — FULL {lang} lexicon for ELP surface realization.")
|
|
||||||
L.append(f"// Generated by gen_elp_seed_full.py from morphology_{lang}_full")
|
|
||||||
L.append(f"// (real UniMorph + kaikki.org Wiktionary forms; gender from lexicon, not heuristic).")
|
|
||||||
L.append(f"// Entries: {total} (verbs={stats['verbs']} nouns={stats['nouns']} adjs={stats['adjs']})")
|
|
||||||
L.append(f"// Schema: [lemma, pos, form0, form1, form2, en_translation, semantic_hint]")
|
|
||||||
L.append(f"// verbs: form0=pres-3sg form1=pret-3sg form2=past-participle")
|
|
||||||
L.append(f"// nouns: form0=sg form1=pl form2=REAL gender adjs: form0=m-sg form1=f-sg form2=m-pl")
|
|
||||||
L.append("")
|
|
||||||
for ci, ch in enumerate(chunks):
|
|
||||||
L.append(f"fn vocab_{lang}_seed_p{ci}(v: [[String]]) -> [[String]] {{")
|
|
||||||
for r in ch:
|
|
||||||
L.append(row(r))
|
|
||||||
L.append(" return v")
|
|
||||||
L.append("}")
|
|
||||||
L.append("")
|
|
||||||
L.append(f"fn vocab_{lang}_seed() -> [[String]] {{")
|
|
||||||
L.append(" let v: [[String]] = native_list_empty()")
|
|
||||||
for ci in range(len(chunks)):
|
|
||||||
L.append(f" let v = vocab_{lang}_seed_p{ci}(v)")
|
|
||||||
L.append(" return v")
|
|
||||||
L.append("}")
|
|
||||||
L.append("")
|
|
||||||
L.append(f"fn vocab_{lang}_lookup(word: String) -> [String] {{")
|
|
||||||
L.append(f" let vocab: [[String]] = vocab_{lang}_seed()")
|
|
||||||
L.append(" let n: Int = native_list_len(vocab)")
|
|
||||||
L.append(" let i: Int = 0")
|
|
||||||
L.append(" while i < n {")
|
|
||||||
L.append(" let entry: [String] = native_list_get(vocab, i)")
|
|
||||||
L.append(' if str_eq(native_list_get(entry, 0), word) { return entry }')
|
|
||||||
L.append(" let i = i + 1")
|
|
||||||
L.append(" }")
|
|
||||||
L.append(" return native_list_empty()")
|
|
||||||
L.append("}")
|
|
||||||
with open(out_path, "w", encoding="utf-8") as fh:
|
|
||||||
fh.write("\n".join(L) + "\n")
|
|
||||||
return total, stats
|
|
||||||
|
|
||||||
def emit(lang, out_path):
|
|
||||||
M = importlib.import_module(f"morphology_{lang}_full")
|
|
||||||
rows, stats = build_rows(lang, M)
|
|
||||||
return write_seed(lang, rows, stats, out_path)
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
lang, out = sys.argv[1], sys.argv[2]
|
|
||||||
total, stats = emit(lang, out)
|
|
||||||
print(f"{lang}: wrote {out} total={total} verbs={stats['verbs']} nouns={stats['nouns']} adjs={stats['adjs']}")
|
|
||||||
@@ -1,572 +0,0 @@
|
|||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""morphology_ca_full.py — production-grade Catalan morphological generator.
|
|
||||||
|
|
||||||
Same design as morphology_it_full.py (its Romance sibling); Catalan-specific data.
|
|
||||||
|
|
||||||
VERBS
|
|
||||||
UniMorph Catalan (github.com/unimorph/cat, CC-BY-SA 3.0)
|
|
||||||
7,535 verb lemmas × paradigm, CLEAN orthography:
|
|
||||||
present, imperfet (PST;IPFV), pretèrit simple (PST;PFV), futur,
|
|
||||||
condicional (COND), subjuntiu present (SBJV;PRS) / imperfet (SBJV;PST),
|
|
||||||
imperatiu (POS;IMP), infinitiu (NFIN), gerundi (V.CVB;PRS),
|
|
||||||
participi (V.PTCP;PST) — WITH full gender+number agreement forms
|
|
||||||
(cantat/cantada/cantats/cantades) stored directly.
|
|
||||||
ca_irreg_verbs.json — verbs UniMorph MISSES or under-populates
|
|
||||||
(anar, fer, plus core auxiliaries ser/haver/estar/tenir…), extracted from
|
|
||||||
kaikki.org Catalan by build_ca_irreg.py. Priority layer. Supplies anar,
|
|
||||||
whose present (vaig/vas/va/anem/aneu/van) is ALSO the PERIPHRASTIC-PRETERITE
|
|
||||||
auxiliary (vaig cantar = 'I sang') — a hallmark Catalan construction.
|
|
||||||
|
|
||||||
NOUNS + ADJECTIVES — kaikki.org Catalan (Wiktionary extract, CC-BY-SA 3.0)
|
|
||||||
noun lemmas WITH inherent gender + real plural (resolved PER LEMMA).
|
|
||||||
adjective lemmas with real feminine + plural forms.
|
|
||||||
|
|
||||||
Fallbacks degrade, never crash:
|
|
||||||
verbs : regular -ar/-er/-re/-ir rule generator (+ -car/-gar/-çar spelling).
|
|
||||||
nouns : gender heuristic + rule pluralization (-a→-es with ç/c/g/j/qu/gu
|
|
||||||
spelling changes; sibilant-final → -os; else -s). Ambiguous → FLAG.
|
|
||||||
adjs : -o? no (Catalan masc often consonant/-e); fem -a rule + plural rule.
|
|
||||||
|
|
||||||
Confidence flag per form: "lexicon" | "rule" | "fallback" (low → FLAG).
|
|
||||||
|
|
||||||
Public API (used by realizer_ca.py):
|
|
||||||
conjugate(lemma, mood, tense, person, number) -> (form, conf)
|
|
||||||
peri_pret_aux(person, number) -> form # anar-present, for vaig+INF
|
|
||||||
participle(lemma, gender, number) -> (form, conf)
|
|
||||||
gerund(lemma) -> (form, conf)
|
|
||||||
noun_gender(lemma) -> "m"|"f"
|
|
||||||
inflect_noun(lemma, number, gender=None) -> (form, conf)
|
|
||||||
inflect_adj(lemma, gender, number) -> (form, conf)
|
|
||||||
lexicon_stats() -> dict
|
|
||||||
"""
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import pickle
|
|
||||||
|
|
||||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
_UNIMORPH = os.path.join(_HERE, "data", "cat.unimorph")
|
|
||||||
_IRREG = os.path.join(_HERE, "data", "ca_irreg_verbs.json")
|
|
||||||
_KAIKKI = os.path.join(_HERE, "data", "kaikki_ca.jsonl")
|
|
||||||
_CACHE = os.path.join(_HERE, "data", "ca_morph_cache.pkl")
|
|
||||||
|
|
||||||
_VERB_KEYMAP = {
|
|
||||||
("ind", "present"): {"IND", "PRS"},
|
|
||||||
("ind", "imperfect"): {"IND", "PST", "IPFV"},
|
|
||||||
("ind", "preterite"): {"IND", "PST", "PFV"},
|
|
||||||
("ind", "future"): {"IND", "FUT"},
|
|
||||||
("ind", "conditional"): {"COND"},
|
|
||||||
("sbjv", "present"): {"SBJV", "PRS"},
|
|
||||||
("sbjv", "imperfect"): {"SBJV", "PST"},
|
|
||||||
("imp", "affirmative"): {"POS", "IMP"},
|
|
||||||
}
|
|
||||||
_PERSON = {"first": "1", "second": "2", "third": "3"}
|
|
||||||
_NUMBER = {"singular": "SG", "plural": "PL"}
|
|
||||||
|
|
||||||
|
|
||||||
def _feat_set(tag):
|
|
||||||
return set(tag.split(";"))
|
|
||||||
|
|
||||||
|
|
||||||
# ── verbs from UniMorph ──────────────────────────────────────────────────────────
|
|
||||||
def _build_verbs():
|
|
||||||
verbs = {}
|
|
||||||
part = {} # lemma -> {("m","SG"):form, ("f","SG"):..., ("m","PL"):..., ("f","PL"):...}
|
|
||||||
ger = {}
|
|
||||||
with open(_UNIMORPH, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
line = line.rstrip("\n")
|
|
||||||
if not line or "\t" not in line:
|
|
||||||
continue
|
|
||||||
parts = line.split("\t")
|
|
||||||
if len(parts) != 3:
|
|
||||||
continue
|
|
||||||
lemma, form, tag = parts
|
|
||||||
f = _feat_set(tag)
|
|
||||||
head = tag.split(";")[0]
|
|
||||||
if head == "V.PTCP":
|
|
||||||
if "PST" in f:
|
|
||||||
g = "f" if "FEM" in f else "m"
|
|
||||||
n = "PL" if "PL" in f else "SG"
|
|
||||||
part.setdefault(lemma, {})[(g, n)] = form
|
|
||||||
continue
|
|
||||||
if head == "V.CVB":
|
|
||||||
if "PRS" in f:
|
|
||||||
ger.setdefault(lemma, form)
|
|
||||||
continue
|
|
||||||
if head != "V":
|
|
||||||
continue
|
|
||||||
person = next((p for p in ("1", "2", "3") if p in f), None)
|
|
||||||
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
|
|
||||||
if person is None or number is None:
|
|
||||||
continue
|
|
||||||
for (mood, tense), req in _VERB_KEYMAP.items():
|
|
||||||
if not req <= f:
|
|
||||||
continue
|
|
||||||
if tense == "imperfect" and "PFV" in f:
|
|
||||||
continue
|
|
||||||
if tense == "preterite" and "IPFV" in f:
|
|
||||||
continue
|
|
||||||
verbs.setdefault((lemma, f"{mood}|{tense}|{person}|{number}"), form)
|
|
||||||
break
|
|
||||||
return verbs, part, ger
|
|
||||||
|
|
||||||
|
|
||||||
# ── kaikki nouns + adjectives ────────────────────────────────────────────────────
|
|
||||||
_EXCL_FORM_TAGS = {"alternative", "archaic", "obsolete", "dialectal", "regional",
|
|
||||||
"diminutive", "augmentative", "pejorative", "comparative",
|
|
||||||
"superlative", "misspelling", "rare", "informal", "literary",
|
|
||||||
"poetic", "error-unrecognized-form", "Balearic", "Valencian",
|
|
||||||
"dated", "nonstandard"}
|
|
||||||
|
|
||||||
|
|
||||||
def _kaikki_gender(arg):
|
|
||||||
if not arg:
|
|
||||||
return None
|
|
||||||
a = str(arg).lower()
|
|
||||||
if a.startswith("f"):
|
|
||||||
return "f"
|
|
||||||
if a.startswith("m"):
|
|
||||||
return "m"
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _build_nouns_adjs():
|
|
||||||
nouns = {}
|
|
||||||
adjs = {}
|
|
||||||
with open(_KAIKKI, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
try:
|
|
||||||
d = json.loads(line)
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
pos = d.get("pos")
|
|
||||||
word = d.get("word", "")
|
|
||||||
if not word or " " in word:
|
|
||||||
continue
|
|
||||||
forms = d.get("forms", []) or []
|
|
||||||
if pos == "noun":
|
|
||||||
ht = d.get("head_templates") or []
|
|
||||||
g = None
|
|
||||||
if ht:
|
|
||||||
g = _kaikki_gender((ht[0].get("args") or {}).get("1"))
|
|
||||||
if g is None:
|
|
||||||
tags = d.get("tags") or []
|
|
||||||
if "feminine" in tags:
|
|
||||||
g = "f"
|
|
||||||
elif "masculine" in tags:
|
|
||||||
g = "m"
|
|
||||||
pl = None
|
|
||||||
for x in forms:
|
|
||||||
t = set(x.get("tags") or [])
|
|
||||||
if "plural" in t and not (t & _EXCL_FORM_TAGS):
|
|
||||||
fm = x.get("form")
|
|
||||||
if fm and " " not in fm and fm not in ("#", "—", "-"):
|
|
||||||
pl = fm
|
|
||||||
break
|
|
||||||
if word not in nouns:
|
|
||||||
nouns[word] = {"g": g, "SG": word, "PL": pl}
|
|
||||||
else:
|
|
||||||
cur = nouns[word]
|
|
||||||
if cur.get("g") is None and g:
|
|
||||||
cur["g"] = g
|
|
||||||
if not cur.get("PL") and pl:
|
|
||||||
cur["PL"] = pl
|
|
||||||
elif pos == "adj":
|
|
||||||
d0 = adjs.setdefault(word, {})
|
|
||||||
d0.setdefault(("m", "SG"), word)
|
|
||||||
for x in forms:
|
|
||||||
t = set(x.get("tags") or [])
|
|
||||||
fm = x.get("form")
|
|
||||||
if not fm or " " in fm or (t & _EXCL_FORM_TAGS):
|
|
||||||
continue
|
|
||||||
if "feminine" in t and "plural" in t:
|
|
||||||
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
|
|
||||||
elif "masculine" in t and "plural" in t:
|
|
||||||
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
|
|
||||||
elif "feminine" in t:
|
|
||||||
d0[("f", "SG")] = d0.get(("f", "SG")) or fm
|
|
||||||
elif "plural" in t:
|
|
||||||
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
|
|
||||||
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
|
|
||||||
return nouns, adjs
|
|
||||||
|
|
||||||
|
|
||||||
def _build_cache():
|
|
||||||
verbs, part, ger = _build_verbs()
|
|
||||||
nouns, adjs = _build_nouns_adjs()
|
|
||||||
with open(_IRREG, encoding="utf-8") as fh:
|
|
||||||
irreg = json.load(fh)
|
|
||||||
data = {"verbs": verbs, "part": part, "ger": ger,
|
|
||||||
"nouns": nouns, "adjs": adjs, "irreg": irreg}
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "wb") as fh:
|
|
||||||
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
def _load():
|
|
||||||
if os.path.exists(_CACHE):
|
|
||||||
srcs = [_UNIMORPH, _KAIKKI, _IRREG]
|
|
||||||
newest = max(os.path.getmtime(s) for s in srcs if os.path.exists(s))
|
|
||||||
if os.path.getmtime(_CACHE) >= newest:
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "rb") as fh:
|
|
||||||
return pickle.load(fh)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return _build_cache()
|
|
||||||
|
|
||||||
|
|
||||||
_LEX = _load()
|
|
||||||
_VERBS, _PART, _GER, _NOUNS, _ADJS, _IRREGV = (
|
|
||||||
_LEX["verbs"], _LEX["part"], _LEX["ger"], _LEX["nouns"], _LEX["adjs"],
|
|
||||||
_LEX["irreg"])
|
|
||||||
_PERI = _IRREGV.get("_peri_pret_aux", {})
|
|
||||||
|
|
||||||
|
|
||||||
# ── regular verb rule fallback ───────────────────────────────────────────────────
|
|
||||||
def _vclass(lemma):
|
|
||||||
if lemma.endswith("ar"):
|
|
||||||
return "ar"
|
|
||||||
if lemma.endswith("re"):
|
|
||||||
return "re"
|
|
||||||
if lemma.endswith("er"):
|
|
||||||
return "er"
|
|
||||||
if lemma.endswith("ir"):
|
|
||||||
return "ir"
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
# endings [1sg,2sg,3sg,1pl,2pl,3pl] — central Catalan
|
|
||||||
_REG = {
|
|
||||||
("ind", "present", "ar"): ["o", "es", "a", "em", "eu", "en"],
|
|
||||||
("ind", "present", "re"): ["o", "s", "", "em", "eu", "en"],
|
|
||||||
("ind", "present", "er"): ["o", "s", "", "em", "eu", "en"],
|
|
||||||
("ind", "present", "ir"): ["o", "es", "", "im", "iu", "en"], # pure -ir (dormir)
|
|
||||||
("ind", "imperfect", "ar"): ["ava", "aves", "ava", "àvem", "àveu", "aven"],
|
|
||||||
("ind", "imperfect", "re"): ["ia", "ies", "ia", "íem", "íeu", "ien"],
|
|
||||||
("ind", "imperfect", "er"): ["ia", "ies", "ia", "íem", "íeu", "ien"],
|
|
||||||
("ind", "imperfect", "ir"): ["ia", "ies", "ia", "íem", "íeu", "ien"],
|
|
||||||
("ind", "preterite", "ar"): ["í", "ares", "à", "àrem", "àreu", "aren"],
|
|
||||||
("ind", "preterite", "re"): ["í", "eres", "é", "érem", "éreu", "eren"],
|
|
||||||
("ind", "preterite", "er"): ["í", "eres", "é", "érem", "éreu", "eren"],
|
|
||||||
("ind", "preterite", "ir"): ["í", "ires", "í", "írem", "íreu", "iren"],
|
|
||||||
("sbjv", "present", "ar"): ["i", "is", "i", "em", "eu", "in"],
|
|
||||||
("sbjv", "present", "re"): ["i", "is", "i", "em", "eu", "in"],
|
|
||||||
("sbjv", "present", "er"): ["i", "is", "i", "em", "eu", "in"],
|
|
||||||
("sbjv", "present", "ir"): ["i", "is", "i", "im", "iu", "in"],
|
|
||||||
("sbjv", "imperfect", "ar"): ["és", "essis", "és", "éssim", "éssiu", "essin"],
|
|
||||||
("sbjv", "imperfect", "re"): ["és", "essis", "és", "éssim", "éssiu", "essin"],
|
|
||||||
("sbjv", "imperfect", "er"): ["és", "essis", "és", "éssim", "éssiu", "essin"],
|
|
||||||
("sbjv", "imperfect", "ir"): ["ís", "issis", "ís", "íssim", "íssiu", "issin"],
|
|
||||||
("imp", "affirmative", "ar"): [None, "a", "i", "em", "eu", "in"],
|
|
||||||
("imp", "affirmative", "re"): [None, "", "i", "em", "eu", "in"],
|
|
||||||
("imp", "affirmative", "er"): [None, "", "i", "em", "eu", "in"],
|
|
||||||
("imp", "affirmative", "ir"): [None, "", "i", "im", "iu", "in"],
|
|
||||||
}
|
|
||||||
_FUT = ["é", "às", "à", "em", "eu", "an"]
|
|
||||||
_COND = ["ia", "ies", "ia", "íem", "íeu", "ien"]
|
|
||||||
|
|
||||||
|
|
||||||
def _slot_idx(person, number):
|
|
||||||
base = {"first": 0, "second": 1, "third": 2}[person]
|
|
||||||
return base + (0 if number == "singular" else 3)
|
|
||||||
|
|
||||||
|
|
||||||
def _apply_ar_spelling(stem, ending):
|
|
||||||
"""-car/-gar/-çar/-jar spelling before front (e/i) endings."""
|
|
||||||
front = ending[:1] in ("e", "i", "é", "í")
|
|
||||||
if not front:
|
|
||||||
# ç before back vowel stays; but -çar stem already ends ç
|
|
||||||
return stem + ending
|
|
||||||
if stem.endswith("c"):
|
|
||||||
return stem[:-1] + "qu" + ending
|
|
||||||
if stem.endswith("g"):
|
|
||||||
return stem[:-1] + "gu" + ending
|
|
||||||
if stem.endswith("ç"):
|
|
||||||
return stem[:-1] + "c" + ending
|
|
||||||
if stem.endswith("j"):
|
|
||||||
return stem[:-1] + "g" + ending
|
|
||||||
if stem.endswith("qu"):
|
|
||||||
return stem + ending
|
|
||||||
return stem + ending
|
|
||||||
|
|
||||||
|
|
||||||
def _rule_conjugate(lemma, mood, tense, person, number):
|
|
||||||
vc = _vclass(lemma)
|
|
||||||
if vc is None:
|
|
||||||
return None
|
|
||||||
body = lemma[:-2]
|
|
||||||
i = _slot_idx(person, number)
|
|
||||||
if mood == "ind" and tense in ("future", "conditional"):
|
|
||||||
# future/cond stem = infinitive (for -re verbs drop final -e)
|
|
||||||
stem = lemma[:-1] if vc == "re" else lemma
|
|
||||||
end = (_FUT if tense == "future" else _COND)[i]
|
|
||||||
return stem + end
|
|
||||||
table = _REG.get((mood, tense, vc))
|
|
||||||
if not table:
|
|
||||||
return None
|
|
||||||
end = table[i]
|
|
||||||
if end is None:
|
|
||||||
return None
|
|
||||||
if vc == "ar":
|
|
||||||
return _apply_ar_spelling(body, end)
|
|
||||||
# -re/-er/-ir: guard double vowel
|
|
||||||
if body and body[-1:] == end[:1] and end[:1] in "ií":
|
|
||||||
return body[:-1] + end
|
|
||||||
return body + end
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: verb conjugation ─────────────────────────────────────────────────────
|
|
||||||
def conjugate(lemma, mood, tense, person, number):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{number and number[:2].upper()}"
|
|
||||||
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{_NUMBER.get(number,'?')}"
|
|
||||||
# UniMorph (cleanly accented) takes priority; the kaikki irregulars layer is a
|
|
||||||
# FALLBACK for verbs/slots UniMorph lacks (anar, fer, and rarer paradigm cells).
|
|
||||||
p, n = _PERSON.get(person), _NUMBER.get(number)
|
|
||||||
if p and n:
|
|
||||||
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}"))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
ir = _IRREGV.get(lemma)
|
|
||||||
if ir and key in ir:
|
|
||||||
return ir[key], "lexicon"
|
|
||||||
r = _rule_conjugate(lemma, mood, tense, person, number)
|
|
||||||
if r is not None:
|
|
||||||
return r, "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
def peri_pret_aux(person, number):
|
|
||||||
"""anar-present auxiliary for the periphrastic preterite (vaig cantar)."""
|
|
||||||
return _PERI.get(f"{_PERSON.get(person,'3')}|{_NUMBER.get(number,'SG')}", "va")
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: participle + gerund ──────────────────────────────────────────────────
|
|
||||||
def participle(lemma, gender="m", number="singular"):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
g = "f" if gender == "f" else "m"
|
|
||||||
num = "SG" if number == "singular" else "PL"
|
|
||||||
ir = _IRREGV.get(lemma)
|
|
||||||
base = None
|
|
||||||
if ir and "part" in ir:
|
|
||||||
# prefer explicit irregular agreement form (part_mSG/part_fSG/...)
|
|
||||||
exact = ir.get("part_" + g + num)
|
|
||||||
if exact:
|
|
||||||
return exact, "lexicon"
|
|
||||||
base = ir["part"]
|
|
||||||
elif lemma in _PART:
|
|
||||||
table = _PART[lemma]
|
|
||||||
if (g, num) in table:
|
|
||||||
return table[(g, num)], "lexicon"
|
|
||||||
base = table.get(("m", "SG"))
|
|
||||||
if base is None:
|
|
||||||
vc = _vclass(lemma)
|
|
||||||
if vc == "ar":
|
|
||||||
base = lemma[:-2] + "at"
|
|
||||||
elif vc == "ir":
|
|
||||||
base = lemma[:-2] + "it"
|
|
||||||
elif vc in ("er", "re"):
|
|
||||||
base = lemma[:-2] + "ut"
|
|
||||||
else:
|
|
||||||
return lemma, "fallback"
|
|
||||||
conf = "rule"
|
|
||||||
else:
|
|
||||||
conf = "lexicon"
|
|
||||||
# agreement on -t/-ut/-at/-it participles: m.sg base, f.sg +a (-da? no: -ada),
|
|
||||||
# Catalan: cantat/cantada/cantats/cantades; -t → f -da, pl -ts/-des
|
|
||||||
if base.endswith("t"):
|
|
||||||
stem = base[:-1]
|
|
||||||
forms = {"m|SG": base, "f|SG": stem + "da",
|
|
||||||
"m|PL": base + "s", "f|PL": stem + "des"}
|
|
||||||
return forms[f"{g}|{num}"], conf
|
|
||||||
if base.endswith("s"): # after sibilant participle (rare): pres->presa
|
|
||||||
stem = base
|
|
||||||
forms = {"m|SG": base, "f|SG": base + "a",
|
|
||||||
"m|PL": base + "os", "f|PL": base + "es"}
|
|
||||||
return forms[f"{g}|{num}"], conf
|
|
||||||
return base, conf
|
|
||||||
|
|
||||||
|
|
||||||
def gerund(lemma):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
ir = _IRREGV.get(lemma)
|
|
||||||
if ir and "ger" in ir:
|
|
||||||
return ir["ger"], "lexicon"
|
|
||||||
if lemma in _GER:
|
|
||||||
return _GER[lemma], "lexicon"
|
|
||||||
vc = _vclass(lemma)
|
|
||||||
if vc == "ar":
|
|
||||||
return lemma[:-2] + "ant", "rule"
|
|
||||||
if vc in ("er", "re"):
|
|
||||||
return lemma[:-2] + "ent", "rule"
|
|
||||||
if vc == "ir":
|
|
||||||
return lemma[:-2] + "int", "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: noun gender + number ─────────────────────────────────────────────────
|
|
||||||
_FEM_SUF = ("ció", "sió", "tat", "tud", "esa", "esa", "dat", "ança", "ència",
|
|
||||||
"ància", "tud", "ícia", "esa", "or") # note -or is mixed; kaikki wins
|
|
||||||
_MASC_SUF = ("atge", "ment", " isme", "or")
|
|
||||||
|
|
||||||
|
|
||||||
def _gender_heuristic(noun):
|
|
||||||
for suf in ("ció", "sió", "tat", "tud", "esa", "ança", "ència", "ància",
|
|
||||||
"ícia", "etat"):
|
|
||||||
if noun.endswith(suf):
|
|
||||||
return "f"
|
|
||||||
if noun.endswith("a") and not noun.endswith("ma"):
|
|
||||||
return "f"
|
|
||||||
return "m"
|
|
||||||
|
|
||||||
|
|
||||||
def noun_gender(lemma):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
if d and d.get("g") in ("m", "f"):
|
|
||||||
return d["g"]
|
|
||||||
return _gender_heuristic(lemma)
|
|
||||||
|
|
||||||
|
|
||||||
def _rule_plural(noun, gender):
|
|
||||||
"""Deterministic Catalan pluralization. (form, ok); ok=False FLAGS ambiguity."""
|
|
||||||
if not noun:
|
|
||||||
return noun, True
|
|
||||||
# stressed final vowel with accent → +ns (mà→mans is irregular; but capità→capitans)
|
|
||||||
if noun[-1:] in ("à", "é", "í", "ó", "ú"):
|
|
||||||
return noun + "ns", True
|
|
||||||
if noun.endswith("ça"):
|
|
||||||
return noun[:-2] + "ces", True # plaça→places
|
|
||||||
if noun.endswith("ca"):
|
|
||||||
return noun[:-2] + "ques", True # branca→branques
|
|
||||||
if noun.endswith("ga"):
|
|
||||||
return noun[:-2] + "gues", True # amiga→amigues
|
|
||||||
if noun.endswith("ja"):
|
|
||||||
return noun[:-2] + "ges", True # pluja→pluges
|
|
||||||
if noun.endswith("qua"):
|
|
||||||
return noun[:-3] + "qües", True
|
|
||||||
if noun.endswith("gua"):
|
|
||||||
return noun[:-3] + "gües", True
|
|
||||||
if noun.endswith("a"):
|
|
||||||
return noun[:-1] + "es", True # casa→cases
|
|
||||||
# sibilant-final → -os
|
|
||||||
if noun.endswith(("s", "ç", "x", "ig")) or noun.endswith(("ix", "tx", "tj")):
|
|
||||||
if noun.endswith("ç"):
|
|
||||||
return noun[:-1] + "ços", True # braç→braços
|
|
||||||
return noun + "os", True # peix→peixos, gas→gasos
|
|
||||||
if noun[-1:] in ("e", "i", "o", "u"):
|
|
||||||
return noun + "s", True
|
|
||||||
# consonant-final
|
|
||||||
return noun + "s", True
|
|
||||||
|
|
||||||
|
|
||||||
def inflect_noun(lemma, number, gender=None):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
if number == "singular":
|
|
||||||
return (d["SG"] if d and d.get("SG") else lemma), ("lexicon" if d else "rule")
|
|
||||||
if d and d.get("PL"):
|
|
||||||
return d["PL"], "lexicon"
|
|
||||||
g = gender or noun_gender(lemma)
|
|
||||||
form, ok = _rule_plural(lemma, g)
|
|
||||||
return form, ("rule" if ok else "fallback")
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: adjective agreement ──────────────────────────────────────────────────
|
|
||||||
def _fem_of(adj):
|
|
||||||
"""Regular Catalan feminine: consonant/-o? Catalan masc usually consonant or -e.
|
|
||||||
default +a with spelling changes; -e→-a for some; but many are invariable."""
|
|
||||||
a = adj
|
|
||||||
if a.endswith("a"):
|
|
||||||
return a
|
|
||||||
if a.endswith("e"):
|
|
||||||
return a[:-1] + "a" # ample→? actually 'ample' invariable; kaikki wins
|
|
||||||
if a.endswith("u"):
|
|
||||||
return a + "a"
|
|
||||||
if a.endswith("c"):
|
|
||||||
return a[:-1] + "ca" # ric→rica
|
|
||||||
if a.endswith("t"):
|
|
||||||
return a + "a" # alt→alta
|
|
||||||
return a + "a"
|
|
||||||
|
|
||||||
|
|
||||||
def inflect_adj(lemma, gender, number):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
g = "f" if gender == "f" else "m"
|
|
||||||
num = "SG" if number == "singular" else "PL"
|
|
||||||
d = _ADJS.get(lemma)
|
|
||||||
if d:
|
|
||||||
form = d.get((g, num))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
sg = d.get((g, "SG")) or d.get(("m", "SG")) or lemma
|
|
||||||
if num == "PL":
|
|
||||||
pl, ok = _rule_plural(sg, g)
|
|
||||||
return pl, ("rule" if ok else "fallback")
|
|
||||||
return sg, "lexicon"
|
|
||||||
# rule fallback
|
|
||||||
base = lemma if g == "m" else _fem_of(lemma)
|
|
||||||
if num == "SG":
|
|
||||||
return base, "rule"
|
|
||||||
pl, ok = _rule_plural(base, g)
|
|
||||||
return pl, ("rule" if ok else "fallback")
|
|
||||||
|
|
||||||
|
|
||||||
def lexicon_stats():
|
|
||||||
return {
|
|
||||||
"verb_source": "UniMorph Catalan (github.com/unimorph/cat) + kaikki.org "
|
|
||||||
"irregulars (anar/fer/auxiliaries)",
|
|
||||||
"noun_adj_source": "kaikki.org Catalan (Wiktionary extract)",
|
|
||||||
"license": "CC-BY-SA 3.0 (Wiktionary/UniMorph lineage)",
|
|
||||||
"unimorph_verb_forms": len(_VERBS),
|
|
||||||
"unimorph_verb_lemmas": len({k[0] for k in _VERBS}),
|
|
||||||
"irregular_verb_lemmas": len([k for k in _IRREGV if not k.startswith("_")]),
|
|
||||||
"participle_lemmas": len(_PART),
|
|
||||||
"gerund_lemmas": len(_GER),
|
|
||||||
"noun_lemmas": len(_NOUNS),
|
|
||||||
"adj_lemmas": len(_ADJS),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
|
|
||||||
tests = [
|
|
||||||
("cantar", "ind", "present", "first", "singular", "canto"),
|
|
||||||
("cantar", "ind", "present", "third", "plural", "canten"),
|
|
||||||
("ser", "ind", "present", "third", "singular", "és"),
|
|
||||||
("haver", "ind", "present", "first", "singular", "he"),
|
|
||||||
("anar", "ind", "present", "first", "singular", "vaig"),
|
|
||||||
("fer", "ind", "present", "third", "singular", "fa"),
|
|
||||||
("perdre", "ind", "present", "first", "singular", "perdo"),
|
|
||||||
("dormir", "ind", "present", "third", "plural", "dormen"),
|
|
||||||
("cantar", "ind", "future", "first", "singular", "cantaré"),
|
|
||||||
("cantar", "ind", "preterite", "third", "singular", "cantà"),
|
|
||||||
("tenir", "sbjv", "present", "first", "singular", "tingui"),
|
|
||||||
]
|
|
||||||
ok = 0
|
|
||||||
for lemma, mood, tense, per, num, exp in tests:
|
|
||||||
got, conf = conjugate(lemma, mood, tense, per, num)
|
|
||||||
flag = "OK " if got == exp else "XX "
|
|
||||||
ok += got == exp
|
|
||||||
print(f" {flag}{lemma:8} {mood}/{tense:11} {per[:3]}.{num[:2]} -> {got:10} ({conf}) exp={exp}")
|
|
||||||
print(f"verb tests {ok}/{len(tests)}")
|
|
||||||
print(" peri-pret anar: 1sg=", peri_pret_aux("first", "singular"),
|
|
||||||
"3pl=", peri_pret_aux("third", "plural"))
|
|
||||||
print(" gender casa=", noun_gender("casa"), "home=", noun_gender("home"),
|
|
||||||
"cavall=", noun_gender("cavall"), "cançó=", noun_gender("cançó"))
|
|
||||||
print(" plural casa->", inflect_noun("casa", "plural"),
|
|
||||||
"| plaça->", inflect_noun("plaça", "plural"),
|
|
||||||
"| peix->", inflect_noun("peix", "plural"),
|
|
||||||
"| braç->", inflect_noun("braç", "plural"),
|
|
||||||
"| home->", inflect_noun("home", "plural"))
|
|
||||||
print(" adj: alt/f/sg->", inflect_adj("alt", "f", "singular"),
|
|
||||||
"| bonic/f/pl->", inflect_adj("bonic", "f", "plural"),
|
|
||||||
"| vermell/f/sg->", inflect_adj("vermell", "f", "singular"))
|
|
||||||
print(" part: cantar/f/sg->", participle("cantar", "f", "singular"),
|
|
||||||
"| veure/f/pl->", participle("veure", "f", "plural"),
|
|
||||||
"| fer/m/sg->", participle("fer", "m", "singular"))
|
|
||||||
print(" ger: fer->", gerund("fer"), "| cantar->", gerund("cantar"))
|
|
||||||
@@ -1,423 +0,0 @@
|
|||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""morphology_de_full.py — production German morphological generator.
|
|
||||||
|
|
||||||
Real data, no toy tables:
|
|
||||||
|
|
||||||
PRIMARY — UniMorph German (github.com/unimorph/deu, CC-BY-SA 3.0).
|
|
||||||
~219k noun forms, ~199k verb forms. Supplies:
|
|
||||||
nouns : gender (MASC/FEM/NEUT) + case×number paradigm
|
|
||||||
(N;NOM/ACC/DAT/GEN; MASC/FEM/NEUT; SG/PL) — the genitive -(e)s,
|
|
||||||
dative-plural -n and the five plural classes are REAL forms, not
|
|
||||||
guessed.
|
|
||||||
verbs : full finite paradigm IND;{SG,PL};{1,2,3};{PRS,PST}, the past
|
|
||||||
participle (V.PTCP;PST, incl. reattached separable prefix
|
|
||||||
'zugefügt'), and — crucially for V2 — the SEPARATED finite form
|
|
||||||
UniMorph records directly ('füge zu', 'steht auf').
|
|
||||||
adjs : comparative / superlative (ADJ;CMPR, ADJ;SPRL).
|
|
||||||
|
|
||||||
SECONDARY — kaikki.org German (Wiktionary, CC-BY-SA/GFDL). Gap-fills noun
|
|
||||||
gender + plural where UniMorph is thin. Never overrides UniMorph.
|
|
||||||
|
|
||||||
Rule fallbacks (flagged 'rule'/'fallback') for lemmas absent from both lexicons:
|
|
||||||
present : -e/-st/-t/-en/-t/-en with e-epenthesis after -t/-d/-chn stems
|
|
||||||
plural : gender heuristic (fem -> -(e)n, else -e / umlaut left to lexicon)
|
|
||||||
ppart : weak ge-…-t
|
|
||||||
Adjective ENDINGS are rule-computed by the realizer (regular closed table);
|
|
||||||
this module only supplies the comparative/superlative STEM.
|
|
||||||
|
|
||||||
Perfect auxiliary (haben vs sein): sein for a curated set of intransitive
|
|
||||||
motion / change-of-state verbs (real German lexical property), else haben.
|
|
||||||
|
|
||||||
Public API:
|
|
||||||
noun_gender(lemma) -> 'm'|'f'|'n'
|
|
||||||
decline_noun(lemma, case, number) -> (form, conf)
|
|
||||||
pluralize(lemma) -> (form, conf)
|
|
||||||
finite(lemma, tense, person, number) -> (form, conf) # may contain ' prefix'
|
|
||||||
nonfinite(lemma, req) -> (form, conf) # req: 'inf'|'ppart'
|
|
||||||
past_participle(lemma) -> (form, conf)
|
|
||||||
separable_prefix(lemma) -> str|None
|
|
||||||
perfect_aux(lemma) -> 'haben'|'sein'
|
|
||||||
comparative(lemma)/superlative(lemma) -> (stem, conf)
|
|
||||||
lexicon_stats() -> dict
|
|
||||||
"""
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import pickle
|
|
||||||
|
|
||||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
_UNIMORPH = os.path.join(_HERE, "data", "deu.unimorph")
|
|
||||||
_KAIKKI = os.path.join(_HERE, "data", "kaikki_de.jsonl")
|
|
||||||
_CACHE = os.path.join(_HERE, "data", "de_morph_cache.pkl")
|
|
||||||
|
|
||||||
_GENDER = {"MASC": "m", "FEM": "f", "NEUT": "n"}
|
|
||||||
|
|
||||||
# intransitive motion / change-of-state verbs that take SEIN in the perfect
|
|
||||||
_SEIN = {"gehen", "kommen", "fahren", "laufen", "rennen", "reisen", "fallen",
|
|
||||||
"steigen", "sinken", "wachsen", "sterben", "geschehen", "passieren",
|
|
||||||
"werden", "bleiben", "sein", "aufstehen", "einschlafen", "aufwachen",
|
|
||||||
"ankommen", "abfahren", "aufsteigen", "erscheinen", "verschwinden",
|
|
||||||
"fliegen", "schwimmen", "springen", "begegnen", "folgen", "gelingen",
|
|
||||||
"wandern", "ziehen", "flüchten", "eintreten", "einsteigen", "aussteigen"}
|
|
||||||
|
|
||||||
|
|
||||||
# hardcoded high-frequency irregular / auxiliary / modal paradigms (closed class,
|
|
||||||
# verified) — consulted before the lexicon so aux+modal chains are always correct.
|
|
||||||
_CORE = {
|
|
||||||
"sein": {"prs": {("first", "singular"): "bin", ("second", "singular"): "bist",
|
|
||||||
("third", "singular"): "ist", ("first", "plural"): "sind",
|
|
||||||
("second", "plural"): "seid", ("third", "plural"): "sind"},
|
|
||||||
"pst": {("first", "singular"): "war", ("second", "singular"): "warst",
|
|
||||||
("third", "singular"): "war", ("first", "plural"): "waren",
|
|
||||||
("second", "plural"): "wart", ("third", "plural"): "waren"},
|
|
||||||
"ppart": "gewesen"},
|
|
||||||
"haben": {"prs": {("first", "singular"): "habe", ("second", "singular"): "hast",
|
|
||||||
("third", "singular"): "hat", ("first", "plural"): "haben",
|
|
||||||
("second", "plural"): "habt", ("third", "plural"): "haben"},
|
|
||||||
"pst": {("first", "singular"): "hatte", ("second", "singular"): "hattest",
|
|
||||||
("third", "singular"): "hatte", ("first", "plural"): "hatten",
|
|
||||||
("second", "plural"): "hattet", ("third", "plural"): "hatten"},
|
|
||||||
"ppart": "gehabt"},
|
|
||||||
"werden": {"prs": {("first", "singular"): "werde", ("second", "singular"): "wirst",
|
|
||||||
("third", "singular"): "wird", ("first", "plural"): "werden",
|
|
||||||
("second", "plural"): "werdet", ("third", "plural"): "werden"},
|
|
||||||
"pst": {("first", "singular"): "wurde", ("second", "singular"): "wurdest",
|
|
||||||
("third", "singular"): "wurde", ("first", "plural"): "wurden",
|
|
||||||
("second", "plural"): "wurdet", ("third", "plural"): "wurden"},
|
|
||||||
"ppart": "geworden"},
|
|
||||||
}
|
|
||||||
_MODAL_PRS = {
|
|
||||||
"können": ("kann", "kannst", "kann", "können", "könnt", "können"),
|
|
||||||
"müssen": ("muss", "musst", "muss", "müssen", "müsst", "müssen"),
|
|
||||||
"wollen": ("will", "willst", "will", "wollen", "wollt", "wollen"),
|
|
||||||
"sollen": ("soll", "sollst", "soll", "sollen", "sollt", "sollen"),
|
|
||||||
"dürfen": ("darf", "darfst", "darf", "dürfen", "dürft", "dürfen"),
|
|
||||||
"mögen": ("mag", "magst", "mag", "mögen", "mögt", "mögen"),
|
|
||||||
}
|
|
||||||
_MODAL_PST = {
|
|
||||||
"können": ("konnte", "konntest", "konnte", "konnten", "konntet", "konnten"),
|
|
||||||
"müssen": ("musste", "musstest", "musste", "mussten", "musstet", "mussten"),
|
|
||||||
"wollen": ("wollte", "wolltest", "wollte", "wollten", "wolltet", "wollten"),
|
|
||||||
"sollen": ("sollte", "solltest", "sollte", "sollten", "solltet", "sollten"),
|
|
||||||
"dürfen": ("durfte", "durftest", "durfte", "durften", "durftet", "durften"),
|
|
||||||
"mögen": ("mochte", "mochtest", "mochte", "mochten", "mochtet", "mochten"),
|
|
||||||
}
|
|
||||||
_PN_ORDER = [("first", "singular"), ("second", "singular"), ("third", "singular"),
|
|
||||||
("first", "plural"), ("second", "plural"), ("third", "plural")]
|
|
||||||
_MODAL_PPART = {"können": "gekonnt", "müssen": "gemusst", "wollen": "gewollt",
|
|
||||||
"sollen": "gesollt", "dürfen": "gedurft", "mögen": "gemocht"}
|
|
||||||
for _m, _forms in _MODAL_PRS.items():
|
|
||||||
_CORE[_m] = {"prs": dict(zip(_PN_ORDER, _forms)),
|
|
||||||
"pst": dict(zip(_PN_ORDER, _MODAL_PST[_m])),
|
|
||||||
"ppart": _MODAL_PPART[_m]}
|
|
||||||
|
|
||||||
|
|
||||||
def _person_num(tags):
|
|
||||||
p = n = None
|
|
||||||
for t in tags:
|
|
||||||
if t in ("1", "2", "3"):
|
|
||||||
p = {"1": "first", "2": "second", "3": "third"}[t]
|
|
||||||
elif t == "SG":
|
|
||||||
n = "singular"
|
|
||||||
elif t == "PL":
|
|
||||||
n = "plural"
|
|
||||||
return p, n
|
|
||||||
|
|
||||||
|
|
||||||
def _build_from_unimorph():
|
|
||||||
nouns, verbs, adjs = {}, {}, {}
|
|
||||||
if not os.path.exists(_UNIMORPH):
|
|
||||||
return nouns, verbs, adjs
|
|
||||||
with open(_UNIMORPH, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
line = line.rstrip("\n")
|
|
||||||
if not line or "\t" not in line:
|
|
||||||
continue
|
|
||||||
parts = line.split("\t")
|
|
||||||
if len(parts) != 3:
|
|
||||||
continue
|
|
||||||
lemma, form, tagstr = parts
|
|
||||||
tags = tagstr.split(";")
|
|
||||||
head = tags[0]
|
|
||||||
tset = set(tags)
|
|
||||||
if head == "N":
|
|
||||||
rec = nouns.setdefault(lemma, {"g": None, "cases": {}, "pl": None})
|
|
||||||
g = next((_GENDER[t] for t in tags if t in _GENDER), None)
|
|
||||||
if g and not rec["g"]:
|
|
||||||
rec["g"] = g
|
|
||||||
case = next((t for t in tags if t in ("NOM", "ACC", "DAT", "GEN")), None)
|
|
||||||
num = "plural" if "PL" in tset else ("singular" if "SG" in tset else None)
|
|
||||||
if case and num:
|
|
||||||
rec["cases"].setdefault((case, num), form)
|
|
||||||
if case == "NOM" and num == "plural" and not rec["pl"]:
|
|
||||||
rec["pl"] = form
|
|
||||||
elif head.startswith("V"):
|
|
||||||
rec = verbs.setdefault(lemma, {"prs": {}, "pst": {}, "ppart": None})
|
|
||||||
if "PTCP" in head and "PST" in tset:
|
|
||||||
rec["ppart"] = rec["ppart"] or form
|
|
||||||
elif "IND" in tset and ("PRS" in tset or "PST" in tset):
|
|
||||||
p, n = _person_num(tags)
|
|
||||||
if p and n:
|
|
||||||
slot = "prs" if "PRS" in tset else "pst"
|
|
||||||
rec[slot].setdefault((p, n), form)
|
|
||||||
elif head == "ADJ":
|
|
||||||
rec = adjs.setdefault(lemma, {})
|
|
||||||
if "CMPR" in tset:
|
|
||||||
rec.setdefault("cmpr", form.replace("am ", "").strip())
|
|
||||||
elif "SPRL" in tset:
|
|
||||||
rec.setdefault("sprl", form.replace("am ", "").replace("sten", "st")
|
|
||||||
if form.endswith("sten") else form.replace("am ", ""))
|
|
||||||
return nouns, verbs, adjs
|
|
||||||
|
|
||||||
|
|
||||||
def _build_from_kaikki(nouns):
|
|
||||||
"""Gap-fill noun gender + plural from kaikki German."""
|
|
||||||
if not os.path.exists(_KAIKKI):
|
|
||||||
return
|
|
||||||
_g = {"masculine": "m", "feminine": "f", "neuter": "n", "m": "m", "f": "f", "n": "n"}
|
|
||||||
with open(_KAIKKI, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
try:
|
|
||||||
d = json.loads(line)
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
if d.get("pos") != "noun":
|
|
||||||
continue
|
|
||||||
w = d.get("word", "")
|
|
||||||
if not w or not w[0].isalpha() or " " in w:
|
|
||||||
continue
|
|
||||||
rec = nouns.setdefault(w, {"g": None, "cases": {}, "pl": None})
|
|
||||||
# GENDER: Wiktionary gender is hand-curated and OVERRIDES UniMorph's
|
|
||||||
# auto-tagged gender, which has known errors (e.g. UniMorph deu mis-
|
|
||||||
# records Zeit=MASC, Wagen=NEUT; Wiktionary has f, m correctly).
|
|
||||||
for h in d.get("head_templates", []) or []:
|
|
||||||
a = h.get("args", {}) or {}
|
|
||||||
raw = a.get("1") or a.get("g") or ""
|
|
||||||
code = str(raw).split(",")[0].strip().lower()
|
|
||||||
if code in _g:
|
|
||||||
rec["g"] = _g[code]
|
|
||||||
break
|
|
||||||
if not rec["pl"]:
|
|
||||||
for f in d.get("forms", []) or []:
|
|
||||||
t = set(f.get("tags", []) or [])
|
|
||||||
if "plural" in t and f.get("form") and "genitive" not in t:
|
|
||||||
rec["pl"] = f["form"]
|
|
||||||
break
|
|
||||||
|
|
||||||
|
|
||||||
def _build_cache():
|
|
||||||
nouns, verbs, adjs = _build_from_unimorph()
|
|
||||||
_build_from_kaikki(nouns)
|
|
||||||
data = {"nouns": nouns, "verbs": verbs, "adjs": adjs}
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "wb") as fh:
|
|
||||||
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
def _load():
|
|
||||||
if os.path.exists(_CACHE):
|
|
||||||
srcs = [p for p in (_UNIMORPH, _KAIKKI) if os.path.exists(p)]
|
|
||||||
newest = max((os.path.getmtime(p) for p in srcs), default=0)
|
|
||||||
if os.path.getmtime(_CACHE) >= newest:
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "rb") as fh:
|
|
||||||
return pickle.load(fh)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return _build_cache()
|
|
||||||
|
|
||||||
|
|
||||||
_LEX = _load()
|
|
||||||
_NOUNS, _VERBS, _ADJS = _LEX["nouns"], _LEX["verbs"], _LEX["adjs"]
|
|
||||||
|
|
||||||
|
|
||||||
# ── nouns ────────────────────────────────────────────────────────────────────────
|
|
||||||
def noun_gender(lemma):
|
|
||||||
rec = _NOUNS.get(lemma) or _NOUNS.get(lemma.capitalize())
|
|
||||||
if rec and rec.get("g"):
|
|
||||||
return rec["g"]
|
|
||||||
# last-resort rule: -ung/-heit/-keit/-schaft/-tät/-ion -> f ; -chen/-lein -> n
|
|
||||||
low = lemma.lower()
|
|
||||||
if low.endswith(("ung", "heit", "keit", "schaft", "tät", "ion", "ik", "ei")):
|
|
||||||
return "f"
|
|
||||||
if low.endswith(("chen", "lein", "ment", "um")):
|
|
||||||
return "n"
|
|
||||||
return "m"
|
|
||||||
|
|
||||||
|
|
||||||
def pluralize(lemma):
|
|
||||||
rec = _NOUNS.get(lemma) or _NOUNS.get(lemma.capitalize())
|
|
||||||
if rec and rec.get("pl"):
|
|
||||||
return rec["pl"], "lexicon"
|
|
||||||
g = noun_gender(lemma)
|
|
||||||
if g == "f":
|
|
||||||
return (lemma + "en" if not lemma.endswith("e") else lemma + "n"), "rule"
|
|
||||||
return (lemma if lemma.endswith(("er", "en", "el")) else lemma + "e"), "rule"
|
|
||||||
|
|
||||||
|
|
||||||
def decline_noun(lemma, case, number):
|
|
||||||
"""case in NOM/ACC/DAT/GEN, number in singular/plural."""
|
|
||||||
rec = _NOUNS.get(lemma) or _NOUNS.get(lemma.capitalize())
|
|
||||||
if case == "DAT" and number == "singular":
|
|
||||||
# modern German drops the archaic dative -e ('dem Kinde' -> 'dem Kind');
|
|
||||||
# the article carries the case. Keep bare nominative form.
|
|
||||||
base = (rec or {}).get("cases", {}).get(("NOM", "singular")) or lemma
|
|
||||||
return base, ("lexicon" if rec else "rule")
|
|
||||||
if rec and rec.get("cases", {}).get((case, number)):
|
|
||||||
return rec["cases"][(case, number)], "lexicon"
|
|
||||||
if number == "plural":
|
|
||||||
pl, c = pluralize(lemma)
|
|
||||||
if case == "DAT" and not pl.endswith("n") and not pl.endswith("s"):
|
|
||||||
return pl + "n", c # dative plural -n
|
|
||||||
return pl, c
|
|
||||||
# singular
|
|
||||||
g = noun_gender(lemma)
|
|
||||||
if case == "GEN" and g in ("m", "n"):
|
|
||||||
return (lemma + "es" if lemma.endswith(("s", "ß", "z", "x")) else lemma + "s"), "rule"
|
|
||||||
return lemma, "lexicon" if rec else "rule"
|
|
||||||
|
|
||||||
|
|
||||||
# ── verbs ──────────────────────────────────────────────────────────────────────--
|
|
||||||
_PRS_ENDINGS = {("first", "singular"): "e", ("second", "singular"): "st",
|
|
||||||
("third", "singular"): "t", ("first", "plural"): "en",
|
|
||||||
("second", "plural"): "t", ("third", "plural"): "en"}
|
|
||||||
|
|
||||||
|
|
||||||
def _stem(lemma):
|
|
||||||
if lemma.endswith("en"):
|
|
||||||
return lemma[:-2]
|
|
||||||
if lemma.endswith("n"):
|
|
||||||
return lemma[:-1]
|
|
||||||
return lemma
|
|
||||||
|
|
||||||
|
|
||||||
def separable_prefix(lemma):
|
|
||||||
"""Return the separable prefix if the lemma is a separable-prefix verb."""
|
|
||||||
rec = _VERBS.get(lemma)
|
|
||||||
if rec:
|
|
||||||
for (_p, _n), form in rec.get("prs", {}).items():
|
|
||||||
if " " in form:
|
|
||||||
return form.rsplit(" ", 1)[1]
|
|
||||||
_SEP = ("auf", "aus", "ab", "an", "ein", "mit", "nach", "vor", "zu", "zurück",
|
|
||||||
"weg", "hin", "her", "los", "bei", "fest", "fort", "um", "zusammen")
|
|
||||||
_INSEP = ("be", "ge", "er", "ver", "zer", "ent", "emp", "miss")
|
|
||||||
for p in sorted(_SEP, key=len, reverse=True):
|
|
||||||
if lemma.startswith(p) and len(lemma) > len(p) + 2 \
|
|
||||||
and not lemma.startswith(_INSEP):
|
|
||||||
return p
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def finite(lemma, tense, person, number):
|
|
||||||
"""Present/past finite. For separable verbs the returned string is the
|
|
||||||
UniMorph SEPARATED form 'stem prefix' (realizer places prefix per V2)."""
|
|
||||||
slot = "prs" if tense == "present" else "pst"
|
|
||||||
if lemma in _CORE and _CORE[lemma].get(slot, {}).get((person, number)):
|
|
||||||
return _CORE[lemma][slot][(person, number)], "lexicon"
|
|
||||||
rec = _VERBS.get(lemma)
|
|
||||||
if rec and rec.get(slot, {}).get((person, number)):
|
|
||||||
return rec[slot][(person, number)], "lexicon"
|
|
||||||
# rule fallback (present only reliable; past weak -te)
|
|
||||||
stem = _stem(lemma)
|
|
||||||
pref = separable_prefix(lemma)
|
|
||||||
if pref:
|
|
||||||
stem = _stem(lemma[len(pref):])
|
|
||||||
if tense == "present":
|
|
||||||
end = _PRS_ENDINGS[(person, number)]
|
|
||||||
if stem.endswith(("t", "d", "chn", "ffn", "gn")) and end in ("st", "t"):
|
|
||||||
end = "e" + end
|
|
||||||
form = stem + end
|
|
||||||
else:
|
|
||||||
form = stem + ("ete" if stem.endswith(("t", "d")) else "te")
|
|
||||||
if (person, number) == ("second", "singular"):
|
|
||||||
form += "st"
|
|
||||||
elif number == "plural" and person != "second":
|
|
||||||
form += "n"
|
|
||||||
elif (person, number) == ("second", "plural"):
|
|
||||||
form += "t"
|
|
||||||
if pref:
|
|
||||||
return f"{form} {pref}", "rule"
|
|
||||||
return form, "rule"
|
|
||||||
|
|
||||||
|
|
||||||
def _weak_t(stem):
|
|
||||||
return stem + ("et" if stem.endswith(("t", "d", "chn", "ffn", "gn")) else "t")
|
|
||||||
|
|
||||||
|
|
||||||
def past_participle(lemma):
|
|
||||||
if lemma in _CORE:
|
|
||||||
return _CORE[lemma]["ppart"], "lexicon"
|
|
||||||
rec = _VERBS.get(lemma)
|
|
||||||
if rec and rec.get("ppart"):
|
|
||||||
return rec["ppart"], "lexicon"
|
|
||||||
stem = _stem(lemma)
|
|
||||||
pref = separable_prefix(lemma)
|
|
||||||
_INSEP = ("be", "ge", "er", "ver", "zer", "ent", "emp", "miss")
|
|
||||||
if pref:
|
|
||||||
inner = _stem(lemma[len(pref):])
|
|
||||||
return pref + "ge" + _weak_t(inner), "rule"
|
|
||||||
if lemma.startswith(_INSEP):
|
|
||||||
return _weak_t(stem), "rule"
|
|
||||||
return "ge" + _weak_t(stem), "rule"
|
|
||||||
|
|
||||||
|
|
||||||
def nonfinite(lemma, req):
|
|
||||||
if req == "ppart":
|
|
||||||
return past_participle(lemma)
|
|
||||||
return lemma, "lexicon" if lemma in _VERBS else "rule" # infinitive
|
|
||||||
|
|
||||||
|
|
||||||
def perfect_aux(lemma):
|
|
||||||
return "sein" if lemma in _SEIN else "haben"
|
|
||||||
|
|
||||||
|
|
||||||
# ── adjectives ────────────────────────────────────────────────────────────────---
|
|
||||||
_ADJ_IRREG_SPRL = {"gut": "best", "groß": "größt", "hoch": "höchst",
|
|
||||||
"nah": "nächst", "viel": "meist", "gern": "liebst"}
|
|
||||||
|
|
||||||
|
|
||||||
def comparative(lemma):
|
|
||||||
rec = _ADJS.get(lemma)
|
|
||||||
if rec and rec.get("cmpr"):
|
|
||||||
return rec["cmpr"], "lexicon"
|
|
||||||
return lemma + "er", "rule"
|
|
||||||
|
|
||||||
|
|
||||||
def superlative(lemma):
|
|
||||||
"""Return the bare superlative STEM (realizer adds 'am ...en' or '-e' ending)."""
|
|
||||||
if lemma in _ADJ_IRREG_SPRL:
|
|
||||||
return _ADJ_IRREG_SPRL[lemma], "lexicon"
|
|
||||||
# derive from the comparative so umlaut is carried (alt->älter->ältest)
|
|
||||||
cmpr, cconf = comparative(lemma)
|
|
||||||
base = cmpr[:-2] if cmpr.endswith("er") else lemma
|
|
||||||
end = "est" if base.endswith(("t", "d", "s", "ß", "z", "sch")) else "st"
|
|
||||||
return base + end, cconf
|
|
||||||
|
|
||||||
|
|
||||||
def lexicon_stats():
|
|
||||||
return {
|
|
||||||
"source": "UniMorph deu (primary) + kaikki.org German (gap-fill gender/plural)",
|
|
||||||
"license": "CC-BY-SA 3.0 (UniMorph); CC-BY-SA/GFDL (Wiktionary)",
|
|
||||||
"noun_lemmas": len(_NOUNS),
|
|
||||||
"nouns_with_gender": sum(1 for v in _NOUNS.values() if v.get("g")),
|
|
||||||
"nouns_with_plural": sum(1 for v in _NOUNS.values() if v.get("pl")),
|
|
||||||
"verb_lemmas": len(_VERBS),
|
|
||||||
"verbs_with_ppart": sum(1 for v in _VERBS.values() if v.get("ppart")),
|
|
||||||
"adj_lemmas": len(_ADJS),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
|
|
||||||
for w in ("Hund", "Frau", "Kind", "Mann", "Buch", "Blume"):
|
|
||||||
print(f" {w}: gender={noun_gender(w)} pl={pluralize(w)} "
|
|
||||||
f"gen.sg={decline_noun(w, 'GEN', 'singular')} "
|
|
||||||
f"dat.pl={decline_noun(w, 'DAT', 'plural')}")
|
|
||||||
for v in ("machen", "gehen", "aufstehen", "sein", "haben", "arbeiten"):
|
|
||||||
print(f" {v}: 3sg.prs={finite(v, 'present', 'third', 'singular')} "
|
|
||||||
f"3sg.pst={finite(v, 'past', 'third', 'singular')} "
|
|
||||||
f"ppart={past_participle(v)} aux={perfect_aux(v)} sep={separable_prefix(v)}")
|
|
||||||
for a in ("schnell", "gut", "groß", "alt"):
|
|
||||||
print(f" {a}: cmpr={comparative(a)} sprl={superlative(a)}")
|
|
||||||
@@ -1,562 +0,0 @@
|
|||||||
"""morphology_es_full.py — production-grade Spanish morphological generator.
|
|
||||||
|
|
||||||
NOT a toy. Backed by a real, broad, licensed lexicon:
|
|
||||||
|
|
||||||
UniMorph Spanish (github.com/unimorph/spa, CC-BY-SA 3.0, Wiktionary-derived)
|
|
||||||
1,196,245 inflected forms:
|
|
||||||
6,695 verb lemmas — full paradigms: indicative (present/preterite/
|
|
||||||
imperfect/future), conditional, present & imperfect
|
|
||||||
subjunctive, affirmative imperative, formal/informal
|
|
||||||
48,353 noun lemmas — WITH inherent gender (N;FEM/MASC;SG/PL)
|
|
||||||
16,984 adj lemmas — gender + number paradigms
|
|
||||||
|
|
||||||
Fallbacks (so we degrade, never crash, on out-of-vocabulary input):
|
|
||||||
- verbs : mlconjug3 (ML paradigm model, conjugates ANY Spanish verb) then a
|
|
||||||
hand-rolled regular-ending generator
|
|
||||||
- nouns : gender heuristic (endings) + regular pluralization
|
|
||||||
- adjs : -o/-a gender rule + regular pluralization
|
|
||||||
|
|
||||||
Every generated form carries a CONFIDENCE flag:
|
|
||||||
"lexicon" form came straight from UniMorph (trust: high)
|
|
||||||
"model" form came from mlconjug3 (trust: high)
|
|
||||||
"rule" form came from a deterministic rule (trust: medium)
|
|
||||||
"fallback" we could not inflect; returned lemma as-is (trust: low → FLAG)
|
|
||||||
|
|
||||||
Public API (used by realizer_es.py):
|
|
||||||
conjugate(lemma, mood, tense, person, number, formality="informal") -> (form, conf)
|
|
||||||
participle(lemma) -> (form, conf) # past participle (compound tenses)
|
|
||||||
gerund(lemma) -> (form, conf)
|
|
||||||
noun_gender(lemma) -> "m"|"f"
|
|
||||||
inflect_noun(lemma, number) -> (form, conf)
|
|
||||||
inflect_adj(lemma, gender, number) -> (form, conf)
|
|
||||||
attach_enclitics(verb_form, clitics) -> str # accent-correct enclisis
|
|
||||||
lexicon_stats() -> dict
|
|
||||||
"""
|
|
||||||
import os
|
|
||||||
import pickle
|
|
||||||
import unicodedata
|
|
||||||
|
|
||||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
_UNIMORPH = os.path.join(_HERE, "data", "spa.unimorph")
|
|
||||||
_CACHE = os.path.join(_HERE, "data", "es_morph_cache.pkl")
|
|
||||||
|
|
||||||
# ── canonical feature keys the realizer speaks, mapped to UniMorph tags ─────────
|
|
||||||
# mood/tense pair -> the UniMorph feature substring that identifies it
|
|
||||||
_VERB_KEYMAP = {
|
|
||||||
("ind", "present"): ("IND", "PRS", None),
|
|
||||||
("ind", "preterite"): ("IND", "PST", "PFV"),
|
|
||||||
("ind", "imperfect"): ("IND", "PST", "IPFV"),
|
|
||||||
("ind", "future"): ("IND", "FUT", None),
|
|
||||||
("ind", "conditional"):("COND", None, None),
|
|
||||||
("sbjv", "present"): ("SBJV", "PRS", None),
|
|
||||||
("sbjv", "imperfect"): ("SBJV", "PST", "LGSPEC1"), # -ra form
|
|
||||||
("imp", "present"): ("POS", "IMP", None),
|
|
||||||
}
|
|
||||||
_PERSON = {"first": "1", "second": "2", "third": "3"}
|
|
||||||
_NUMBER = {"singular": "SG", "plural": "PL"}
|
|
||||||
|
|
||||||
|
|
||||||
# ── build / load the compact lexicon ───────────────────────────────────────────
|
|
||||||
def _feat_set(tag):
|
|
||||||
return set(tag.split(";"))
|
|
||||||
|
|
||||||
|
|
||||||
def _build_cache():
|
|
||||||
verbs = {} # (lemma, canonkey) -> form canonkey e.g. "ind|present|1|SG|infm"
|
|
||||||
nouns = {} # lemma -> {"g": "m"/"f", "SG": form, "PL": form}
|
|
||||||
adjs = {} # lemma -> {("m","SG"): form, ...}
|
|
||||||
part = {} # lemma -> masc-sg participle
|
|
||||||
ger = {} # lemma -> gerund
|
|
||||||
|
|
||||||
with open(_UNIMORPH, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
line = line.rstrip("\n")
|
|
||||||
if not line or "\t" not in line:
|
|
||||||
continue
|
|
||||||
parts = line.split("\t")
|
|
||||||
if len(parts) != 3:
|
|
||||||
continue
|
|
||||||
lemma, form, tag = parts
|
|
||||||
f = _feat_set(tag)
|
|
||||||
head = tag.split(";")[0]
|
|
||||||
|
|
||||||
if head == "V":
|
|
||||||
# skip clitic-bearing rows (we generate clitics ourselves)
|
|
||||||
if "PRO" in f:
|
|
||||||
continue
|
|
||||||
if "V.PTCP" in f and "PST" in f and "MASC" in f and "SG" in f:
|
|
||||||
part.setdefault(lemma, form)
|
|
||||||
continue
|
|
||||||
if "V.CVB" in f or "NFIN" in f or "V.PTCP" in f:
|
|
||||||
if "V.CVB" in f:
|
|
||||||
ger.setdefault(lemma, form)
|
|
||||||
continue
|
|
||||||
# identify mood/tense
|
|
||||||
mt = None
|
|
||||||
for (mood, tense), (a, b, c) in _VERB_KEYMAP.items():
|
|
||||||
if a not in f:
|
|
||||||
continue
|
|
||||||
if b is not None and b not in f:
|
|
||||||
continue
|
|
||||||
if c is not None and c not in f:
|
|
||||||
continue
|
|
||||||
# disambiguate IND;PST needing PFV vs IPFV
|
|
||||||
if a == "IND" and b == "PST" and c not in f:
|
|
||||||
continue
|
|
||||||
mt = (mood, tense)
|
|
||||||
break
|
|
||||||
if mt is None:
|
|
||||||
continue
|
|
||||||
person = next((p for p in ("1", "2", "3") if p in f), None)
|
|
||||||
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
|
|
||||||
if person is None or number is None:
|
|
||||||
continue
|
|
||||||
formal = "form" if "FORM" in f else ("infm" if "INFM" in f else "any")
|
|
||||||
key = f"{mt[0]}|{mt[1]}|{person}|{number}|{formal}"
|
|
||||||
verbs.setdefault((lemma, key), form)
|
|
||||||
|
|
||||||
elif head == "N":
|
|
||||||
# substring test handles epicene "MASC+FEM" (-> masc citation)
|
|
||||||
g = "m" if "MASC" in tag else ("f" if "FEM" in tag else None)
|
|
||||||
num = "SG" if "SG" in f else ("PL" if "PL" in f else None)
|
|
||||||
if num is None:
|
|
||||||
continue
|
|
||||||
# store forms keyed by (gender,number); animate nouns list BOTH
|
|
||||||
# genders under one lemma (niño -> niño/niña). Resolve citation
|
|
||||||
# gender in a post-pass (gender of the row whose form == lemma).
|
|
||||||
d = nouns.setdefault(lemma, {})
|
|
||||||
d.setdefault("_rows", []).append((g, num, form))
|
|
||||||
|
|
||||||
elif head == "ADJ":
|
|
||||||
g = "m" if "MASC" in tag else ("f" if "FEM" in tag else "m")
|
|
||||||
num = "SG" if "SG" in f else ("PL" if "PL" in f else None)
|
|
||||||
if num is None:
|
|
||||||
continue
|
|
||||||
adjs.setdefault(lemma, {})[(g, num)] = form
|
|
||||||
|
|
||||||
# post-pass: resolve noun citation gender + default SG/PL forms
|
|
||||||
for lemma, d in nouns.items():
|
|
||||||
rows = d.pop("_rows", [])
|
|
||||||
# citation gender = gender of the row whose form == lemma; else first MASC;
|
|
||||||
# else first seen gender.
|
|
||||||
cite_g = None
|
|
||||||
for g, num, form in rows:
|
|
||||||
if form == lemma and g:
|
|
||||||
cite_g = g
|
|
||||||
break
|
|
||||||
if cite_g is None:
|
|
||||||
for g, num, form in rows:
|
|
||||||
if g == "m":
|
|
||||||
cite_g = "m"
|
|
||||||
break
|
|
||||||
if cite_g is None:
|
|
||||||
cite_g = next((g for g, _, _ in rows if g), "m")
|
|
||||||
d["g"] = cite_g
|
|
||||||
for g, num, form in rows:
|
|
||||||
d[(g, num)] = form
|
|
||||||
d["SG"] = d.get((cite_g, "SG")) or next((f for g, n, f in rows if n == "SG"), lemma)
|
|
||||||
d["PL"] = d.get((cite_g, "PL")) or next((f for g, n, f in rows if n == "PL"), None)
|
|
||||||
|
|
||||||
# post-pass: UniMorph omits the identity inflection (masc-sg == lemma) for
|
|
||||||
# adjectives, so fill it in; without this a fem-sg row wrongly satisfies a
|
|
||||||
# masc-sg request (alto -> alta bug).
|
|
||||||
for lemma, d in adjs.items():
|
|
||||||
d.setdefault(("m", "SG"), lemma)
|
|
||||||
|
|
||||||
data = {"verbs": verbs, "nouns": nouns, "adjs": adjs, "part": part, "ger": ger}
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "wb") as fh:
|
|
||||||
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
def _load():
|
|
||||||
if os.path.exists(_CACHE) and os.path.getmtime(_CACHE) >= os.path.getmtime(_UNIMORPH):
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "rb") as fh:
|
|
||||||
return pickle.load(fh)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return _build_cache()
|
|
||||||
|
|
||||||
|
|
||||||
_LEX = _load()
|
|
||||||
_VERBS, _NOUNS, _ADJS, _PART, _GER = (
|
|
||||||
_LEX["verbs"], _LEX["nouns"], _LEX["adjs"], _LEX["part"], _LEX["ger"])
|
|
||||||
|
|
||||||
# ── mlconjug3 fallback (lazy) ───────────────────────────────────────────────────
|
|
||||||
_MLC = None
|
|
||||||
_MLC_TENSE = { # (mood,tense) -> (mlconjug mood label, tense label)
|
|
||||||
("ind", "present"): ("Indicativo", "Indicativo presente"),
|
|
||||||
("ind", "preterite"): ("Indicativo", "Indicativo pretérito perfecto simple"),
|
|
||||||
("ind", "imperfect"): ("Indicativo", "Indicativo pretérito imperfecto"),
|
|
||||||
("ind", "future"): ("Indicativo", "Indicativo futuro"),
|
|
||||||
("ind", "conditional"): ("Condicional", "Condicional Condicional"),
|
|
||||||
("sbjv", "present"): ("Subjuntivo", "Subjuntivo presente"),
|
|
||||||
("sbjv", "imperfect"): ("Subjuntivo", "Subjuntivo pretérito imperfecto 1"),
|
|
||||||
("imp", "present"): ("Imperativo", "Imperativo Afirmativo"),
|
|
||||||
}
|
|
||||||
_MLC_SLOT = { # (person,number) -> mlconjug slot key
|
|
||||||
("first", "singular"): "1s", ("second", "singular"): "2s",
|
|
||||||
("third", "singular"): "3s", ("first", "plural"): "1p",
|
|
||||||
("second", "plural"): "2p", ("third", "plural"): "3p",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _mlc_conjugate(lemma, mood, tense, person, number):
|
|
||||||
global _MLC
|
|
||||||
try:
|
|
||||||
if _MLC is None:
|
|
||||||
from mlconjug3 import Conjugator
|
|
||||||
_MLC = Conjugator(language="es")
|
|
||||||
v = _MLC.conjugate(lemma)
|
|
||||||
if v is None:
|
|
||||||
return None
|
|
||||||
info = v.conjug_info
|
|
||||||
m, t = _MLC_TENSE.get((mood, tense), (None, None))
|
|
||||||
if m is None or m not in info or t not in info[m]:
|
|
||||||
return None
|
|
||||||
block = info[m][t]
|
|
||||||
slot = _MLC_SLOT.get((person, number))
|
|
||||||
if isinstance(block, dict) and slot in block and block[slot]:
|
|
||||||
return block[slot]
|
|
||||||
return None
|
|
||||||
except Exception:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
# ── regular-ending rule fallback (last resort, deterministic) ───────────────────
|
|
||||||
def _vclass(lemma):
|
|
||||||
return lemma[-2:] if lemma[-2:] in ("ar", "er", "ir") else "ar"
|
|
||||||
|
|
||||||
|
|
||||||
def _stem(lemma):
|
|
||||||
return lemma[:-2]
|
|
||||||
|
|
||||||
|
|
||||||
_REG = {
|
|
||||||
("ind", "present", "ar"): ["o", "as", "a", "amos", "áis", "an"],
|
|
||||||
("ind", "present", "er"): ["o", "es", "e", "emos", "éis", "en"],
|
|
||||||
("ind", "present", "ir"): ["o", "es", "e", "imos", "ís", "en"],
|
|
||||||
("ind", "preterite", "ar"): ["é", "aste", "ó", "amos", "asteis", "aron"],
|
|
||||||
("ind", "preterite", "er"): ["í", "iste", "ió", "imos", "isteis", "ieron"],
|
|
||||||
("ind", "preterite", "ir"): ["í", "iste", "ió", "imos", "isteis", "ieron"],
|
|
||||||
("ind", "imperfect", "ar"): ["aba", "abas", "aba", "ábamos", "abais", "aban"],
|
|
||||||
("ind", "imperfect", "er"): ["ía", "ías", "ía", "íamos", "íais", "ían"],
|
|
||||||
("ind", "imperfect", "ir"): ["ía", "ías", "ía", "íamos", "íais", "ían"],
|
|
||||||
("sbjv", "present", "ar"): ["e", "es", "e", "emos", "éis", "en"],
|
|
||||||
("sbjv", "present", "er"): ["a", "as", "a", "amos", "áis", "an"],
|
|
||||||
("sbjv", "present", "ir"): ["a", "as", "a", "amos", "áis", "an"],
|
|
||||||
("sbjv", "imperfect", "ar"): ["ara", "aras", "ara", "áramos", "arais", "aran"],
|
|
||||||
("sbjv", "imperfect", "er"): ["iera", "ieras", "iera", "iéramos", "ierais", "ieran"],
|
|
||||||
("sbjv", "imperfect", "ir"): ["iera", "ieras", "iera", "iéramos", "ierais", "ieran"],
|
|
||||||
}
|
|
||||||
_FUT = ["é", "ás", "á", "emos", "éis", "án"]
|
|
||||||
_COND = ["ía", "ías", "ía", "íamos", "íais", "ían"]
|
|
||||||
|
|
||||||
|
|
||||||
def _slot_idx(person, number):
|
|
||||||
base = {"first": 0, "second": 1, "third": 2}[person]
|
|
||||||
return base + (0 if number == "singular" else 3)
|
|
||||||
|
|
||||||
|
|
||||||
def _rule_conjugate(lemma, mood, tense, person, number):
|
|
||||||
if len(lemma) < 3 or lemma[-2:] not in ("ar", "er", "ir"):
|
|
||||||
return None
|
|
||||||
vc, st, i = _vclass(lemma), _stem(lemma), _slot_idx(person, number)
|
|
||||||
if tense == "future":
|
|
||||||
return lemma + _FUT[i]
|
|
||||||
if tense == "conditional":
|
|
||||||
return lemma + _COND[i]
|
|
||||||
table = _REG.get((mood, tense, vc))
|
|
||||||
if table:
|
|
||||||
return st + table[i]
|
|
||||||
if mood == "imp" and tense == "present":
|
|
||||||
# affirmative tú imperative = 3sg present indicative
|
|
||||||
pres = _REG.get(("ind", "present", vc))
|
|
||||||
return st + pres[2] if number == "singular" else st + pres[5]
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: verb conjugation ────────────────────────────────────────────────────
|
|
||||||
def conjugate(lemma, mood, tense, person, number, formality="informal"):
|
|
||||||
"""Return (surface, confidence). mood in ind|sbjv|imp; tense per _VERB_KEYMAP."""
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
p, n = _PERSON.get(person), _NUMBER.get(number)
|
|
||||||
formal = "form" if formality == "formal" else "infm"
|
|
||||||
if p and n:
|
|
||||||
for fkey in (formal, "any", "infm" if formal == "form" else "form"):
|
|
||||||
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}|{fkey}"))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
m = _mlc_conjugate(lemma, mood, tense, person, number)
|
|
||||||
if m:
|
|
||||||
return m, "model"
|
|
||||||
r = _rule_conjugate(lemma, mood, tense, person, number)
|
|
||||||
if r:
|
|
||||||
return r, "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
_IRREG_PART = { # guarantee the common irregular participles
|
|
||||||
"escribir": "escrito", "describir": "descrito", "abrir": "abierto",
|
|
||||||
"cubrir": "cubierto", "descubrir": "descubierto", "morir": "muerto",
|
|
||||||
"poner": "puesto", "ver": "visto", "volver": "vuelto", "devolver": "devuelto",
|
|
||||||
"hacer": "hecho", "deshacer": "deshecho", "decir": "dicho", "romper": "roto",
|
|
||||||
"resolver": "resuelto", "freír": "frito", "imprimir": "impreso",
|
|
||||||
"satisfacer": "satisfecho", "prever": "previsto", "revolver": "revuelto",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def participle(lemma):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
if lemma in _IRREG_PART:
|
|
||||||
return _IRREG_PART[lemma], "lexicon"
|
|
||||||
if lemma in _PART:
|
|
||||||
return _PART[lemma], "lexicon"
|
|
||||||
if lemma.endswith("ar"):
|
|
||||||
return lemma[:-2] + "ado", "rule"
|
|
||||||
if lemma[-2:] in ("er", "ir"):
|
|
||||||
return lemma[:-2] + "ido", "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
_IRREG_GER = {"dormir": "durmiendo", "morir": "muriendo", "pedir": "pidiendo",
|
|
||||||
"sentir": "sintiendo", "mentir": "mintiendo", "servir": "sirviendo",
|
|
||||||
"venir": "viniendo", "decir": "diciendo", "poder": "pudiendo",
|
|
||||||
"ir": "yendo", "leer": "leyendo", "creer": "creyendo",
|
|
||||||
"oír": "oyendo", "traer": "trayendo", "caer": "cayendo",
|
|
||||||
"construir": "construyendo", "huir": "huyendo", "reír": "riendo"}
|
|
||||||
|
|
||||||
|
|
||||||
def gerund(lemma):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
if lemma in _IRREG_GER:
|
|
||||||
return _IRREG_GER[lemma], "lexicon"
|
|
||||||
if lemma in _GER:
|
|
||||||
return _GER[lemma], "lexicon"
|
|
||||||
if lemma.endswith("ar"):
|
|
||||||
return lemma[:-2] + "ando", "rule"
|
|
||||||
if lemma[-2:] in ("er", "ir"):
|
|
||||||
return lemma[:-2] + "iendo", "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: noun gender + number ────────────────────────────────────────────────
|
|
||||||
_INVARIANT_PL = {"lunes", "martes", "miércoles", "jueves", "viernes",
|
|
||||||
"crisis", "tesis", "análisis", "dosis", "virus", "paraguas"}
|
|
||||||
|
|
||||||
|
|
||||||
def _gender_heuristic(noun):
|
|
||||||
for suf, g in (("ión", "f"), ("dad", "f"), ("tad", "f"), ("umbre", "f"),
|
|
||||||
("sis", "f"), ("ez", "f"), ("triz", "f"),
|
|
||||||
("ema", "m"), ("ama", "m"), ("oma", "m"), ("aje", "m"),
|
|
||||||
("or", "m"), ("án", "m"), ("ín", "m")):
|
|
||||||
if noun.endswith(suf):
|
|
||||||
return g
|
|
||||||
if noun.endswith("o"):
|
|
||||||
return "m"
|
|
||||||
if noun.endswith("a"):
|
|
||||||
return "f"
|
|
||||||
return "m"
|
|
||||||
|
|
||||||
|
|
||||||
def noun_gender(lemma):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
if d and d.get("g"):
|
|
||||||
return d["g"]
|
|
||||||
return _gender_heuristic(lemma)
|
|
||||||
|
|
||||||
|
|
||||||
def _regular_plural(noun):
|
|
||||||
if noun in _INVARIANT_PL:
|
|
||||||
return noun
|
|
||||||
if not noun:
|
|
||||||
return noun
|
|
||||||
last = noun[-1]
|
|
||||||
if last == "z":
|
|
||||||
return noun[:-1] + "ces"
|
|
||||||
if last in "aeiouáéíóú":
|
|
||||||
# stressed final vowel í/ú -> +es (rubí->rubíes), else +s
|
|
||||||
if last in "íú":
|
|
||||||
return noun + "es"
|
|
||||||
return noun + "s"
|
|
||||||
if last == "s":
|
|
||||||
# esdrújula / stress-final handled crudely; most polysyllables invariant
|
|
||||||
return noun
|
|
||||||
return noun + "es"
|
|
||||||
|
|
||||||
|
|
||||||
def inflect_noun(lemma, number, gender=None):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
num = "SG" if number == "singular" else "PL"
|
|
||||||
if d:
|
|
||||||
# honor a requested gender for animate nouns (gato -> gata)
|
|
||||||
if gender and (gender, num) in d:
|
|
||||||
return d[(gender, num)], "lexicon"
|
|
||||||
if d.get(num):
|
|
||||||
return d[num], "lexicon"
|
|
||||||
if number == "singular":
|
|
||||||
return lemma, "rule" if not d else "lexicon"
|
|
||||||
return _regular_plural(lemma), "rule"
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: adjective agreement ─────────────────────────────────────────────────
|
|
||||||
_INV_GENDER_ADJ = {"español": "española", "trabajador": "trabajadora",
|
|
||||||
"hablador": "habladora", "encantador": "encantadora",
|
|
||||||
"alemán": "alemana", "francés": "francesa", "inglés": "inglesa"}
|
|
||||||
|
|
||||||
|
|
||||||
def inflect_adj(lemma, gender, number):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
d = _ADJS.get(lemma)
|
|
||||||
num = "SG" if number == "singular" else "PL"
|
|
||||||
if d:
|
|
||||||
form = d.get((gender, num))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
# gender-invariant adjective (grande, feliz, azul): fem == masc.
|
|
||||||
# For a missing plural, pluralize this gender's singular form.
|
|
||||||
sg = d.get((gender, "SG")) or d.get(("m", "SG")) or lemma
|
|
||||||
if number == "plural":
|
|
||||||
return _regular_plural(sg), "rule"
|
|
||||||
return sg, "lexicon"
|
|
||||||
# rule fallback
|
|
||||||
a = lemma
|
|
||||||
if gender == "f":
|
|
||||||
if a in _INV_GENDER_ADJ:
|
|
||||||
a = _INV_GENDER_ADJ[a]
|
|
||||||
elif a.endswith("o"):
|
|
||||||
a = a[:-1] + "a"
|
|
||||||
if number == "plural":
|
|
||||||
a = _regular_plural(a)
|
|
||||||
return a, ("rule" if (a != lemma or gender == "m") else "rule")
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: clitic enclisis (dá + me + lo -> dámelo) ────────────────────────────
|
|
||||||
def _strip_accents(s):
|
|
||||||
return "".join(c for c in unicodedata.normalize("NFD", s)
|
|
||||||
if unicodedata.category(c) != "Mn")
|
|
||||||
|
|
||||||
|
|
||||||
def _count_syllables_vowelgroups(word):
|
|
||||||
# crude: count vowel groups
|
|
||||||
w = _strip_accents(word).lower()
|
|
||||||
groups, prev = 0, False
|
|
||||||
for ch in w:
|
|
||||||
isv = ch in "aeiou"
|
|
||||||
if isv and not prev:
|
|
||||||
groups += 1
|
|
||||||
prev = isv
|
|
||||||
return groups
|
|
||||||
|
|
||||||
|
|
||||||
def _host_stress_from_end(word):
|
|
||||||
"""Stressed-syllable index counted from the end (1=last) of a verb host."""
|
|
||||||
syls = _count_syllables_vowelgroups(word)
|
|
||||||
if any(c in "áéíóú" for c in word):
|
|
||||||
return None # already carries its own accent
|
|
||||||
if word[-2:] in ("ar", "er", "ir"): # infinitive: oxytone
|
|
||||||
return 1
|
|
||||||
if word.endswith("ndo"): # gerund: paroxytone
|
|
||||||
return 2
|
|
||||||
if word[-1:] in "aeiouns" and syls >= 2: # default paroxytone
|
|
||||||
return 2
|
|
||||||
return 1 # monosyllable / consonant-final oxytone
|
|
||||||
|
|
||||||
|
|
||||||
def attach_enclitics(verb_form, clitics):
|
|
||||||
"""Append clitic pronouns to a verb (imperative/infinitive/gerund enclisis)
|
|
||||||
and add a written accent when the resulting word becomes esdrújula/
|
|
||||||
sobreesdrújula (stress >= 3 syllables from the end): dá+me+lo -> dámelo,
|
|
||||||
lleva+me -> llévame, but dar+te -> darte and da+me -> dame (no accent)."""
|
|
||||||
if not clitics:
|
|
||||||
return verb_form
|
|
||||||
tail = "".join(clitics)
|
|
||||||
if any(c in "áéíóú" for c in verb_form): # host already accented
|
|
||||||
return verb_form + tail
|
|
||||||
sfe = _host_stress_from_end(verb_form)
|
|
||||||
total_sfe = sfe + len(clitics) # each clitic = 1 syllable
|
|
||||||
if total_sfe >= 3:
|
|
||||||
return _accentuate_nucleus(verb_form, sfe) + tail
|
|
||||||
return verb_form + tail
|
|
||||||
|
|
||||||
|
|
||||||
def _accentuate_nucleus(word, sfe):
|
|
||||||
"""Put a written accent on the syllable `sfe` positions from the word's end."""
|
|
||||||
vowels = "aeiou"
|
|
||||||
nuclei = [i for i, ch in enumerate(word) if ch in vowels]
|
|
||||||
if not nuclei or sfe > len(nuclei):
|
|
||||||
return word
|
|
||||||
i = nuclei[-sfe]
|
|
||||||
acc = {"a": "á", "e": "é", "i": "í", "o": "ó", "u": "ú"}
|
|
||||||
return word[:i] + acc[word[i]] + word[i + 1:]
|
|
||||||
|
|
||||||
|
|
||||||
def _accentuate_last_stressed(word):
|
|
||||||
# Restore the host's ORIGINAL lexical stress with a written accent.
|
|
||||||
# Default Spanish stress: word ending in vowel/n/s -> penultimate syllable;
|
|
||||||
# otherwise (e.g. infinitives in -r) -> last syllable.
|
|
||||||
vowels = "aeiou"
|
|
||||||
nuclei = [i for i, ch in enumerate(word) if ch in vowels]
|
|
||||||
if not nuclei:
|
|
||||||
return word
|
|
||||||
if word[-1] in "aeiouns" and len(nuclei) >= 2:
|
|
||||||
i = nuclei[-2] # paroxytone: penult nucleus
|
|
||||||
else:
|
|
||||||
i = nuclei[-1] # oxytone / monosyllable: last nucleus
|
|
||||||
acc = {"a": "á", "e": "é", "i": "í", "o": "ó", "u": "ú"}
|
|
||||||
return word[:i] + acc[word[i]] + word[i + 1:]
|
|
||||||
|
|
||||||
|
|
||||||
def lexicon_stats():
|
|
||||||
return {
|
|
||||||
"source": "UniMorph Spanish (github.com/unimorph/spa)",
|
|
||||||
"license": "CC-BY-SA 3.0 (Wiktionary-derived)",
|
|
||||||
"total_forms": sum(len(v) for v in (_VERBS, _NOUNS, _ADJS)) if False else None,
|
|
||||||
"verb_forms": len(_VERBS),
|
|
||||||
"verb_lemmas": len({k[0] for k in _VERBS}),
|
|
||||||
"noun_lemmas": len(_NOUNS),
|
|
||||||
"adj_lemmas": len(_ADJS),
|
|
||||||
"participles": len(_PART),
|
|
||||||
"gerunds": len(_GER),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
import json
|
|
||||||
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
|
|
||||||
tests = [
|
|
||||||
("hablar", "ind", "present", "first", "singular", "hablo"),
|
|
||||||
("comer", "ind", "present", "third", "plural", "comen"),
|
|
||||||
("vivir", "ind", "present", "first", "plural", "vivimos"),
|
|
||||||
("ser", "ind", "present", "third", "singular", "es"),
|
|
||||||
("ir", "ind", "preterite", "first", "singular", "fui"),
|
|
||||||
("tener", "ind", "future", "first", "singular", "tendré"),
|
|
||||||
("hacer", "sbjv", "present", "first", "singular", "haga"),
|
|
||||||
("dormir", "ind", "present", "first", "singular", "duermo"),
|
|
||||||
("pensar", "sbjv", "present", "third", "singular", "piense"),
|
|
||||||
("dar", "ind", "preterite", "third", "singular", "dio"),
|
|
||||||
("poner", "ind", "conditional", "first", "singular", "pondría"),
|
|
||||||
]
|
|
||||||
ok = 0
|
|
||||||
for lemma, mood, tense, per, num, exp in tests:
|
|
||||||
got, conf = conjugate(lemma, mood, tense, per, num)
|
|
||||||
flag = "OK " if got == exp else "XX "
|
|
||||||
if got == exp:
|
|
||||||
ok += 1
|
|
||||||
print(f" {flag}{lemma:8} {mood}/{tense} {per[:3]}.{num[:2]:3} -> {got:14} ({conf}) exp={exp}")
|
|
||||||
print(f"verb tests {ok}/{len(tests)}")
|
|
||||||
print(" gender casa:", noun_gender("casa"), "| problema:", noun_gender("problema"),
|
|
||||||
"| agua:", noun_gender("agua"), "| mano:", noun_gender("mano"))
|
|
||||||
print(" plural: luz->", inflect_noun("luz", "plural"), "| rey->", inflect_noun("rey", "plural"))
|
|
||||||
print(" adj: rojo/f/pl->", inflect_adj("rojo", "f", "plural"),
|
|
||||||
"| feliz/m/pl->", inflect_adj("feliz", "m", "plural"),
|
|
||||||
"| grande/f/pl->", inflect_adj("grande", "f", "plural"))
|
|
||||||
print(" enclisis: da+[me,lo]->", attach_enclitics("da", ["me", "lo"]),
|
|
||||||
"| di+[me]->", attach_enclitics("di", ["me"]),
|
|
||||||
"| dar+[se,lo]->", attach_enclitics("dar", ["se", "lo"]))
|
|
||||||
@@ -1,629 +0,0 @@
|
|||||||
"""morphology_fr_full.py — production-grade French morphological generator.
|
|
||||||
|
|
||||||
Same architecture as morphology_it_full.py (shared Romance engine); French-specific
|
|
||||||
data and rules swapped in. Backed by three real, Wiktionary-lineage sources:
|
|
||||||
|
|
||||||
VERBS
|
|
||||||
UniMorph French (github.com/unimorph/fra, CC-BY-SA 3.0)
|
|
||||||
7,535 verb lemmas × full paradigm, CLEAN orthography:
|
|
||||||
indicatif présent / imparfait (PST;IPFV) / passé simple (PST;PFV) /
|
|
||||||
futur, conditionnel (COND), subjonctif présent (SBJV;PRS) /
|
|
||||||
subjonctif imparfait (SBJV;PST), impératif (POS;IMP), infinitif (NFIN),
|
|
||||||
participe présent (V.CVB/V.PTCP;PRS), participe passé (V.PTCP;PST, m.sg).
|
|
||||||
fr_irreg_verbs.json — high-frequency verbs UniMorph MISSES or mis-slots,
|
|
||||||
above all ÊTRE (absent from UniMorph fra), plus avoir/aller/faire/… — the
|
|
||||||
auxiliaries the passé-composé + être-agreement system depends on. Extracted
|
|
||||||
from kaikki.org French (build_fr_irreg.py), reflexive/multiword forms
|
|
||||||
dropped. This layer takes PRIORITY.
|
|
||||||
|
|
||||||
NOUNS + ADJECTIVES — kaikki.org French (Wiktionary extract, CC-BY-SA 3.0)
|
|
||||||
noun lemmas WITH inherent gender (head-template arg) + real plural
|
|
||||||
(cheval->chevaux, œil->yeux, invariable -s/-x/-z), resolved PER LEMMA.
|
|
||||||
adjective lemmas with real feminine + plural (petit->petite/petits/petites,
|
|
||||||
beau->belle/beaux/belles, heureux->heureuse, rouge invariant-gender).
|
|
||||||
|
|
||||||
Fallbacks (degrade, never crash, on OOV input):
|
|
||||||
verbs : rule generator for -er / -ir(-iss-) / -re (with -cer/-ger spelling,
|
|
||||||
future/conditional stems, imparfait/subjonctif endings)
|
|
||||||
nouns : gender heuristic (endings) + rule pluralization (-al->-aux, -eau->-eaux)
|
|
||||||
adjs : fem/plural agreement rules (-er->-ère, -eux->-euse, -f->-ve, +e default)
|
|
||||||
|
|
||||||
Confidence flag on every form: "lexicon" | "rule" | "fallback".
|
|
||||||
|
|
||||||
Public API (used by realizer_fr.py): identical signature to morphology_it_full.
|
|
||||||
"""
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import pickle
|
|
||||||
|
|
||||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
_UNIMORPH = os.path.join(_HERE, "data", "fra.unimorph")
|
|
||||||
_IRREG = os.path.join(_HERE, "data", "fr_irreg_verbs.json")
|
|
||||||
_KAIKKI = os.path.join(_HERE, "data", "kaikki_fr.jsonl")
|
|
||||||
_CACHE = os.path.join(_HERE, "data", "fr_morph_cache.pkl")
|
|
||||||
|
|
||||||
# ── (mood, tense) -> UniMorph feature set that must ALL be present ────────────────
|
|
||||||
_VERB_KEYMAP = {
|
|
||||||
("ind", "present"): {"IND", "PRS"},
|
|
||||||
("ind", "imperfect"): {"IND", "PST", "IPFV"}, # imparfait
|
|
||||||
("ind", "passe_simple"): {"IND", "PST", "PFV"}, # passé simple
|
|
||||||
("ind", "future"): {"IND", "FUT"},
|
|
||||||
("ind", "conditional"): {"COND"}, # French: V;COND;1;SG
|
|
||||||
("sbjv", "present"): {"SBJV", "PRS"},
|
|
||||||
("sbjv", "imperfect"): {"SBJV", "PST"},
|
|
||||||
("imp", "affirmative"): {"POS", "IMP"},
|
|
||||||
}
|
|
||||||
_PERSON = {"first": "1", "second": "2", "third": "3"}
|
|
||||||
_NUMBER = {"singular": "SG", "plural": "PL"}
|
|
||||||
|
|
||||||
|
|
||||||
def _feat_set(tag):
|
|
||||||
return set(tag.split(";"))
|
|
||||||
|
|
||||||
|
|
||||||
# ── build verb lexicon from UniMorph ─────────────────────────────────────────────
|
|
||||||
def _build_verbs():
|
|
||||||
verbs = {}
|
|
||||||
part = {}
|
|
||||||
ger = {}
|
|
||||||
with open(_UNIMORPH, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
line = line.rstrip("\n")
|
|
||||||
if not line or "\t" not in line:
|
|
||||||
continue
|
|
||||||
parts = line.split("\t")
|
|
||||||
if len(parts) != 3:
|
|
||||||
continue
|
|
||||||
lemma, form, tag = parts
|
|
||||||
f = _feat_set(tag)
|
|
||||||
head = tag.split(";")[0]
|
|
||||||
|
|
||||||
if head == "V.PTCP":
|
|
||||||
if "PST" in f:
|
|
||||||
part.setdefault(lemma, form)
|
|
||||||
elif "PRS" in f:
|
|
||||||
ger.setdefault(lemma, form)
|
|
||||||
continue
|
|
||||||
if head == "V.CVB":
|
|
||||||
if "PRS" in f:
|
|
||||||
ger.setdefault(lemma, form)
|
|
||||||
continue
|
|
||||||
if head != "V":
|
|
||||||
continue
|
|
||||||
|
|
||||||
person = next((p for p in ("1", "2", "3") if p in f), None)
|
|
||||||
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
|
|
||||||
if person is None or number is None:
|
|
||||||
continue
|
|
||||||
for (mood, tense), req in _VERB_KEYMAP.items():
|
|
||||||
if not req <= f:
|
|
||||||
continue
|
|
||||||
if tense == "imperfect" and "PFV" in f:
|
|
||||||
continue
|
|
||||||
if tense == "passe_simple" and "IPFV" in f:
|
|
||||||
continue
|
|
||||||
verbs.setdefault((lemma, f"{mood}|{tense}|{person}|{number}"), form)
|
|
||||||
break
|
|
||||||
return verbs, part, ger
|
|
||||||
|
|
||||||
|
|
||||||
# ── kaikki nouns + adjectives ────────────────────────────────────────────────────
|
|
||||||
_EXCL_FORM_TAGS = {"alternative", "archaic", "obsolete", "dialectal", "regional",
|
|
||||||
"diminutive", "augmentative", "pejorative", "comparative",
|
|
||||||
"superlative", "misspelling", "rare", "informal", "literary",
|
|
||||||
"poetic", "error-unrecognized-form", "construed", "collective",
|
|
||||||
"nonstandard", "dated", "Louisiana", "Switzerland", "Belgium"}
|
|
||||||
|
|
||||||
|
|
||||||
def _kaikki_gender(arg):
|
|
||||||
if not arg:
|
|
||||||
return None
|
|
||||||
a = str(arg).lower()
|
|
||||||
if a.startswith("f"):
|
|
||||||
return "f"
|
|
||||||
if a.startswith("m"):
|
|
||||||
return "m"
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _build_nouns_adjs():
|
|
||||||
nouns = {}
|
|
||||||
adjs = {}
|
|
||||||
with open(_KAIKKI, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
try:
|
|
||||||
d = json.loads(line)
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
pos = d.get("pos")
|
|
||||||
word = d.get("word", "")
|
|
||||||
if not word or " " in word:
|
|
||||||
continue
|
|
||||||
forms = d.get("forms", []) or []
|
|
||||||
|
|
||||||
if pos == "noun":
|
|
||||||
ht = d.get("head_templates") or []
|
|
||||||
g = None
|
|
||||||
if ht:
|
|
||||||
g = _kaikki_gender((ht[0].get("args") or {}).get("1"))
|
|
||||||
if g is None:
|
|
||||||
tags = d.get("tags") or []
|
|
||||||
if "feminine" in tags:
|
|
||||||
g = "f"
|
|
||||||
elif "masculine" in tags:
|
|
||||||
g = "m"
|
|
||||||
pl = None
|
|
||||||
for x in forms:
|
|
||||||
t = set(x.get("tags") or [])
|
|
||||||
if "plural" in t and not (t & _EXCL_FORM_TAGS):
|
|
||||||
fm = x.get("form")
|
|
||||||
if fm and " " not in fm and fm not in ("#", "-", "—"):
|
|
||||||
pl = fm
|
|
||||||
break
|
|
||||||
if word not in nouns:
|
|
||||||
nouns[word] = {"g": g, "SG": word, "PL": pl}
|
|
||||||
else:
|
|
||||||
cur = nouns[word]
|
|
||||||
if cur.get("g") is None and g:
|
|
||||||
cur["g"] = g
|
|
||||||
if not cur.get("PL") and pl:
|
|
||||||
cur["PL"] = pl
|
|
||||||
|
|
||||||
elif pos == "adj":
|
|
||||||
d0 = adjs.setdefault(word, {})
|
|
||||||
d0.setdefault(("m", "SG"), word)
|
|
||||||
for x in forms:
|
|
||||||
t = set(x.get("tags") or [])
|
|
||||||
fm = x.get("form")
|
|
||||||
if not fm or " " in fm or (t & _EXCL_FORM_TAGS):
|
|
||||||
continue
|
|
||||||
if "feminine" in t and "plural" in t:
|
|
||||||
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
|
|
||||||
elif "masculine" in t and "plural" in t:
|
|
||||||
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
|
|
||||||
elif "feminine" in t:
|
|
||||||
d0[("f", "SG")] = d0.get(("f", "SG")) or fm
|
|
||||||
elif "plural" in t:
|
|
||||||
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
|
|
||||||
return nouns, adjs
|
|
||||||
|
|
||||||
|
|
||||||
def _build_cache():
|
|
||||||
verbs, part, ger = _build_verbs()
|
|
||||||
nouns, adjs = _build_nouns_adjs()
|
|
||||||
with open(_IRREG, encoding="utf-8") as fh:
|
|
||||||
irreg = json.load(fh)
|
|
||||||
data = {"verbs": verbs, "part": part, "ger": ger,
|
|
||||||
"nouns": nouns, "adjs": adjs, "irreg": irreg}
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "wb") as fh:
|
|
||||||
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
def _load():
|
|
||||||
if os.path.exists(_CACHE):
|
|
||||||
srcs = [_UNIMORPH, _KAIKKI, _IRREG]
|
|
||||||
newest = max(os.path.getmtime(s) for s in srcs if os.path.exists(s))
|
|
||||||
if os.path.getmtime(_CACHE) >= newest:
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "rb") as fh:
|
|
||||||
return pickle.load(fh)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return _build_cache()
|
|
||||||
|
|
||||||
|
|
||||||
_LEX = _load()
|
|
||||||
_VERBS, _PART, _GER, _NOUNS, _ADJS, _IRREGV = (
|
|
||||||
_LEX["verbs"], _LEX["part"], _LEX["ger"], _LEX["nouns"], _LEX["adjs"],
|
|
||||||
_LEX["irreg"])
|
|
||||||
|
|
||||||
|
|
||||||
# ── regular-ending rule fallback ─────────────────────────────────────────────────
|
|
||||||
def _vclass(lemma):
|
|
||||||
if lemma.endswith("er"):
|
|
||||||
return "er"
|
|
||||||
if lemma.endswith("ir"):
|
|
||||||
return "ir"
|
|
||||||
if lemma.endswith("re"):
|
|
||||||
return "re"
|
|
||||||
if lemma.endswith("oir"):
|
|
||||||
return "oir"
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
# present-tense endings [1sg,2sg,3sg,1pl,2pl,3pl]
|
|
||||||
_REG_PRES = {
|
|
||||||
"er": ["e", "es", "e", "ons", "ez", "ent"],
|
|
||||||
"ir": ["is", "is", "it", "issons", "issez", "issent"], # -iss- class (finir)
|
|
||||||
"re": ["s", "s", "", "ons", "ez", "ent"], # vendre: vends/vend
|
|
||||||
}
|
|
||||||
_REG_IMPF = ["ais", "ais", "ait", "ions", "iez", "aient"] # attaches to pres-1pl stem
|
|
||||||
_REG_SUBJ = ["e", "es", "e", "ions", "iez", "ent"] # attaches to 3pl stem
|
|
||||||
_REG_PS = { # passé simple
|
|
||||||
"er": ["ai", "as", "a", "âmes", "âtes", "èrent"],
|
|
||||||
"ir": ["is", "is", "it", "îmes", "îtes", "irent"],
|
|
||||||
"re": ["is", "is", "it", "îmes", "îtes", "irent"],
|
|
||||||
}
|
|
||||||
_FUT = ["ai", "as", "a", "ons", "ez", "ont"]
|
|
||||||
_COND = ["ais", "ais", "ait", "ions", "iez", "aient"]
|
|
||||||
|
|
||||||
|
|
||||||
def _slot_idx(person, number):
|
|
||||||
base = {"first": 0, "second": 1, "third": 2}[person]
|
|
||||||
return base + (0 if number == "singular" else 3)
|
|
||||||
|
|
||||||
|
|
||||||
def _fut_stem(lemma, vc):
|
|
||||||
"""Future/conditional stem = infinitive (drop final -e of -re)."""
|
|
||||||
if vc == "re":
|
|
||||||
return lemma[:-1] # vendre -> vendr-
|
|
||||||
return lemma # parler-, finir-
|
|
||||||
|
|
||||||
|
|
||||||
def _pres_1pl_stem(lemma, vc):
|
|
||||||
"""Imparfait stem = present 1pl minus -ons (parlons->parl-, finissons->finiss-)."""
|
|
||||||
if vc == "er":
|
|
||||||
stem = lemma[:-2]
|
|
||||||
if stem.endswith("g"):
|
|
||||||
return stem + "e" # mangeons -> mange- (imparfait mangeais)
|
|
||||||
if stem.endswith("c"):
|
|
||||||
return stem[:-1] + "ç" # commençons -> commenç-
|
|
||||||
return stem
|
|
||||||
if vc == "ir":
|
|
||||||
return lemma[:-1] + "iss" # finir -> finiss-
|
|
||||||
if vc == "re":
|
|
||||||
return lemma[:-2] # vendre -> vend-
|
|
||||||
return lemma[:-2]
|
|
||||||
|
|
||||||
|
|
||||||
def _apply_er_spelling(stem, ending):
|
|
||||||
"""-cer/-ger softening before a/o (commençons, mangeons)."""
|
|
||||||
if ending and ending[0] in ("a", "o"):
|
|
||||||
if stem.endswith("c"):
|
|
||||||
return stem[:-1] + "ç" + ending
|
|
||||||
if stem.endswith("g"):
|
|
||||||
return stem + "e" + ending
|
|
||||||
return stem + ending
|
|
||||||
|
|
||||||
|
|
||||||
def _rule_conjugate(lemma, mood, tense, person, number):
|
|
||||||
vc = _vclass(lemma)
|
|
||||||
if vc is None:
|
|
||||||
return None
|
|
||||||
i = _slot_idx(person, number)
|
|
||||||
|
|
||||||
if mood == "ind" and tense in ("future", "conditional"):
|
|
||||||
stem = _fut_stem(lemma, vc)
|
|
||||||
end = (_FUT if tense == "future" else _COND)[i]
|
|
||||||
return stem + end
|
|
||||||
|
|
||||||
if mood == "ind" and tense == "present":
|
|
||||||
table = _REG_PRES.get("ir" if vc == "ir" else vc)
|
|
||||||
if not table:
|
|
||||||
return None
|
|
||||||
body = lemma[:-2] if vc in ("er", "re") else lemma[:-1] if vc == "ir" else lemma[:-2]
|
|
||||||
if vc == "ir":
|
|
||||||
body = lemma[:-2] # fin- ; endings carry -iss-
|
|
||||||
end = table[i]
|
|
||||||
return body + end
|
|
||||||
end = table[i]
|
|
||||||
if vc == "er":
|
|
||||||
return _apply_er_spelling(body, end)
|
|
||||||
return body + end
|
|
||||||
|
|
||||||
if mood == "ind" and tense == "imperfect":
|
|
||||||
stem = _pres_1pl_stem(lemma, vc)
|
|
||||||
return stem + _REG_IMPF[i]
|
|
||||||
|
|
||||||
if mood == "ind" and tense == "passe_simple":
|
|
||||||
table = _REG_PS.get("ir" if vc == "ir" else vc)
|
|
||||||
if not table:
|
|
||||||
return None
|
|
||||||
body = lemma[:-2] if vc in ("er", "re") else lemma[:-2]
|
|
||||||
end = table[i]
|
|
||||||
if vc == "er":
|
|
||||||
return _apply_er_spelling(body, end)
|
|
||||||
return body + end
|
|
||||||
|
|
||||||
if mood == "sbjv" and tense == "present":
|
|
||||||
# subjonctif: present-3pl stem + e/es/e/ions/iez/ent
|
|
||||||
stem3 = _pres_1pl_stem(lemma, vc) if vc == "ir" else (
|
|
||||||
lemma[:-2] if vc in ("er", "re") else lemma[:-2])
|
|
||||||
if vc == "ir":
|
|
||||||
stem3 = lemma[:-2] + "iss"
|
|
||||||
end = _REG_SUBJ[i]
|
|
||||||
if vc == "er":
|
|
||||||
return _apply_er_spelling(stem3, end)
|
|
||||||
return stem3 + end
|
|
||||||
|
|
||||||
if mood == "imp" and tense == "affirmative":
|
|
||||||
# impératif ~ present indicative (tu drops -s for -er verbs)
|
|
||||||
pres = _rule_conjugate(lemma, "ind", "present", person, number)
|
|
||||||
if pres and vc == "er" and person == "second" and number == "singular":
|
|
||||||
return pres[:-1] if pres.endswith("es") else pres
|
|
||||||
return pres
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: verb conjugation ─────────────────────────────────────────────────────
|
|
||||||
def conjugate(lemma, mood, tense, person, number):
|
|
||||||
"""Return (surface, confidence)."""
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{number}"
|
|
||||||
ir = _IRREGV.get(lemma)
|
|
||||||
if ir and key in ir:
|
|
||||||
return ir[key], "lexicon"
|
|
||||||
p, n = _PERSON.get(person), _NUMBER.get(number)
|
|
||||||
if p and n:
|
|
||||||
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}"))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
r = _rule_conjugate(lemma, mood, tense, person, number)
|
|
||||||
if r:
|
|
||||||
return r, "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: participle + gerund/participe présent ────────────────────────────────
|
|
||||||
def _participle_msg(lemma):
|
|
||||||
ir = _IRREGV.get(lemma)
|
|
||||||
if ir and "part" in ir:
|
|
||||||
return ir["part"], "lexicon"
|
|
||||||
if lemma in _PART:
|
|
||||||
return _PART[lemma], "lexicon"
|
|
||||||
return None, None
|
|
||||||
|
|
||||||
|
|
||||||
# irregular participle fem/plural quirks (drop circonflexe: dû->due, dus)
|
|
||||||
_PART_FIX = {"dû": {"f|SG": "due", "m|PL": "dus", "f|PL": "dues"}}
|
|
||||||
|
|
||||||
|
|
||||||
def participle(lemma, gender="m", number="singular"):
|
|
||||||
"""Past participle with French gender/number agreement.
|
|
||||||
m.sg = base; f.sg = base+e; m.pl = base+s (invariable if base ends s/x);
|
|
||||||
f.pl = f.sg+s."""
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
g = "f" if gender == "f" else "m"
|
|
||||||
num = "SG" if number == "singular" else "PL"
|
|
||||||
msg, src = _participle_msg(lemma)
|
|
||||||
conf = "lexicon"
|
|
||||||
if msg is None:
|
|
||||||
vc = _vclass(lemma)
|
|
||||||
if vc == "er":
|
|
||||||
msg = lemma[:-2] + "é"
|
|
||||||
elif vc == "ir":
|
|
||||||
msg = lemma[:-1] # finir -> fini, partir -> parti
|
|
||||||
elif vc == "re":
|
|
||||||
msg = lemma[:-2] + "u" # vendre -> vendu
|
|
||||||
elif vc == "oir":
|
|
||||||
msg = lemma[:-3] + "u" # (rough) recevoir handled by irreg
|
|
||||||
else:
|
|
||||||
return lemma, "fallback"
|
|
||||||
conf = "rule"
|
|
||||||
fix = _PART_FIX.get(msg)
|
|
||||||
if fix and f"{g}|{num}" in fix:
|
|
||||||
return fix[f"{g}|{num}"], conf
|
|
||||||
if g == "m" and num == "SG":
|
|
||||||
return msg, conf
|
|
||||||
fem = msg + "e" if not msg.endswith("e") else msg
|
|
||||||
if g == "f" and num == "SG":
|
|
||||||
return fem, conf
|
|
||||||
if g == "m" and num == "PL":
|
|
||||||
return msg if msg.endswith(("s", "x")) else msg + "s", conf
|
|
||||||
# f|PL
|
|
||||||
return fem + "s", conf
|
|
||||||
|
|
||||||
|
|
||||||
def gerund(lemma):
|
|
||||||
"""Participe présent (base for gérondif 'en -ant')."""
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
ir = _IRREGV.get(lemma)
|
|
||||||
if ir and "ger" in ir:
|
|
||||||
return ir["ger"], "lexicon"
|
|
||||||
if lemma in _GER:
|
|
||||||
return _GER[lemma], "lexicon"
|
|
||||||
vc = _vclass(lemma)
|
|
||||||
if vc == "er":
|
|
||||||
stem = lemma[:-2]
|
|
||||||
if stem.endswith("g"):
|
|
||||||
return stem + "eant", "rule"
|
|
||||||
if stem.endswith("c"):
|
|
||||||
return stem[:-1] + "çant", "rule"
|
|
||||||
return stem + "ant", "rule"
|
|
||||||
if vc == "ir":
|
|
||||||
return lemma[:-2] + "issant", "rule"
|
|
||||||
if vc == "re":
|
|
||||||
return lemma[:-2] + "ant", "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: noun gender + number ─────────────────────────────────────────────────
|
|
||||||
_FEM_SUF = ("tion", "sion", "aison", "ance", "ence", "ette", "elle", "esse",
|
|
||||||
"ude", "ade", "ée", "té", "tié", "ie", "ise", "ure", "eur")
|
|
||||||
_MASC_SUF = ("ment", "age", "eau", "isme", "oir", "ier", "eur", "in", "on")
|
|
||||||
|
|
||||||
|
|
||||||
def _gender_heuristic(noun):
|
|
||||||
for suf in _FEM_SUF:
|
|
||||||
if noun.endswith(suf):
|
|
||||||
return "f"
|
|
||||||
for suf in _MASC_SUF:
|
|
||||||
if noun.endswith(suf):
|
|
||||||
return "m"
|
|
||||||
if noun.endswith("e"):
|
|
||||||
return "f"
|
|
||||||
return "m"
|
|
||||||
|
|
||||||
|
|
||||||
def noun_gender(lemma):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
if d and d.get("g") in ("m", "f"):
|
|
||||||
return d["g"]
|
|
||||||
return _gender_heuristic(lemma)
|
|
||||||
|
|
||||||
|
|
||||||
# closed sets for French plural irregularities
|
|
||||||
_OU_X = {"bijou", "caillou", "chou", "genou", "hibou", "joujou", "pou"}
|
|
||||||
_AIL_AUX = {"travail", "vitrail", "corail", "émail", "bail", "soupirail", "vantail"}
|
|
||||||
_AL_S = {"bal", "carnaval", "festival", "récital", "chacal", "régal", "cal", "aval"}
|
|
||||||
|
|
||||||
|
|
||||||
def _rule_plural(noun, gender):
|
|
||||||
"""Deterministic French pluralization. (form, ok); ok=False FLAGS ambiguity."""
|
|
||||||
if not noun:
|
|
||||||
return noun, True
|
|
||||||
if noun[-1:] in ("s", "x", "z"):
|
|
||||||
return noun, True # invariable
|
|
||||||
if noun in _OU_X:
|
|
||||||
return noun + "x", True
|
|
||||||
if noun.endswith(("eau", "au", "eu")):
|
|
||||||
if noun in ("pneu", "bleu", "landau", "sarrau"):
|
|
||||||
return noun + "s", True
|
|
||||||
return noun + "x", True # bateau->bateaux, jeu->jeux
|
|
||||||
if noun.endswith("al"):
|
|
||||||
if noun in _AL_S:
|
|
||||||
return noun + "s", True
|
|
||||||
return noun[:-2] + "aux", True # cheval->chevaux
|
|
||||||
if noun.endswith("ail"):
|
|
||||||
if noun in _AIL_AUX:
|
|
||||||
return noun[:-3] + "aux", True # travail->travaux
|
|
||||||
return noun + "s", True
|
|
||||||
return noun + "s", True # default
|
|
||||||
|
|
||||||
|
|
||||||
def inflect_noun(lemma, number, gender=None):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
if number == "singular":
|
|
||||||
return (d["SG"] if d and d.get("SG") else lemma), ("lexicon" if d else "rule")
|
|
||||||
if d and d.get("PL"):
|
|
||||||
return d["PL"], "lexicon"
|
|
||||||
g = gender or noun_gender(lemma)
|
|
||||||
form, ok = _rule_plural(lemma, g)
|
|
||||||
return form, ("rule" if ok else "fallback")
|
|
||||||
|
|
||||||
|
|
||||||
# adjectives whose kaikki entries are unreliable: audited forms
|
|
||||||
_ADJ_FIX = {
|
|
||||||
"beau": {("m", "SG"): "beau", ("f", "SG"): "belle",
|
|
||||||
("m", "PL"): "beaux", ("f", "PL"): "belles"},
|
|
||||||
"nouveau": {("m", "SG"): "nouveau", ("f", "SG"): "nouvelle",
|
|
||||||
("m", "PL"): "nouveaux", ("f", "PL"): "nouvelles"},
|
|
||||||
"vieux": {("m", "SG"): "vieux", ("f", "SG"): "vieille",
|
|
||||||
("m", "PL"): "vieux", ("f", "PL"): "vieilles"},
|
|
||||||
"fou": {("m", "SG"): "fou", ("f", "SG"): "folle",
|
|
||||||
("m", "PL"): "fous", ("f", "PL"): "folles"},
|
|
||||||
"blanc": {("m", "SG"): "blanc", ("f", "SG"): "blanche",
|
|
||||||
("m", "PL"): "blancs", ("f", "PL"): "blanches"},
|
|
||||||
"long": {("m", "SG"): "long", ("f", "SG"): "longue",
|
|
||||||
("m", "PL"): "longs", ("f", "PL"): "longues"},
|
|
||||||
"bon": {("m", "SG"): "bon", ("f", "SG"): "bonne",
|
|
||||||
("m", "PL"): "bons", ("f", "PL"): "bonnes"},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _rule_fem(a):
|
|
||||||
if a.endswith("e"):
|
|
||||||
return a
|
|
||||||
if a.endswith("er"):
|
|
||||||
return a[:-2] + "ère"
|
|
||||||
if a.endswith("eau"):
|
|
||||||
return a[:-3] + "elle"
|
|
||||||
if a.endswith("eux"):
|
|
||||||
return a[:-3] + "euse"
|
|
||||||
if a.endswith("f"):
|
|
||||||
return a[:-1] + "ve"
|
|
||||||
if a.endswith(("on", "en", "el", "eil", "et")):
|
|
||||||
return a + a[-1] + "e" # bon->bonne, ancien->ancienne, muet->muette
|
|
||||||
if a.endswith("c"):
|
|
||||||
return a[:-1] + "che" # blanc->blanche (public->publique via FIX)
|
|
||||||
return a + "e" # grand->grande, petit->petite, vert->verte
|
|
||||||
|
|
||||||
|
|
||||||
def inflect_adj(lemma, gender, number):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
g = "f" if gender == "f" else "m"
|
|
||||||
num = "SG" if number == "singular" else "PL"
|
|
||||||
fix = _ADJ_FIX.get(lemma)
|
|
||||||
if fix and (g, num) in fix:
|
|
||||||
return fix[(g, num)], "lexicon"
|
|
||||||
d = _ADJS.get(lemma)
|
|
||||||
if d and d.get((g, num)):
|
|
||||||
return d[(g, num)], "lexicon"
|
|
||||||
# derive
|
|
||||||
msc = (d.get(("m", "SG")) if d else None) or lemma
|
|
||||||
if g == "m" and num == "SG":
|
|
||||||
return msc, "lexicon" if d else "rule"
|
|
||||||
fem = (d.get(("f", "SG")) if d else None) or _rule_fem(msc)
|
|
||||||
if g == "f" and num == "SG":
|
|
||||||
return fem, "lexicon" if (d and d.get(("f", "SG"))) else "rule"
|
|
||||||
if g == "m" and num == "PL":
|
|
||||||
if msc.endswith(("s", "x")):
|
|
||||||
return msc, "rule"
|
|
||||||
if msc.endswith("al"):
|
|
||||||
return msc[:-2] + "aux", "rule"
|
|
||||||
if msc.endswith("eau"):
|
|
||||||
return msc + "x", "rule"
|
|
||||||
return msc + "s", "rule"
|
|
||||||
# f|PL
|
|
||||||
return (fem if fem.endswith("s") else fem + "s"), "rule"
|
|
||||||
|
|
||||||
|
|
||||||
def lexicon_stats():
|
|
||||||
return {
|
|
||||||
"verb_source": "UniMorph French (github.com/unimorph/fra) + kaikki.org "
|
|
||||||
"irregulars (être + high-frequency)",
|
|
||||||
"noun_adj_source": "kaikki.org French (Wiktionary extract)",
|
|
||||||
"license": "CC-BY-SA 3.0 (Wiktionary/UniMorph lineage)",
|
|
||||||
"unimorph_verb_forms": len(_VERBS),
|
|
||||||
"unimorph_verb_lemmas": len({k[0] for k in _VERBS}),
|
|
||||||
"irregular_verb_lemmas": len(_IRREGV),
|
|
||||||
"participle_lemmas": len(_PART),
|
|
||||||
"gerund_lemmas": len(_GER),
|
|
||||||
"noun_lemmas": len(_NOUNS),
|
|
||||||
"adj_lemmas": len(_ADJS),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
|
|
||||||
tests = [
|
|
||||||
("parler", "ind", "present", "first", "singular", "parle"),
|
|
||||||
("être", "ind", "present", "third", "singular", "est"),
|
|
||||||
("avoir", "ind", "present", "first", "singular", "ai"),
|
|
||||||
("aller", "ind", "present", "third", "plural", "vont"),
|
|
||||||
("finir", "ind", "present", "first", "singular", "finis"),
|
|
||||||
("finir", "ind", "present", "first", "plural", "finissons"),
|
|
||||||
("manger", "ind", "present", "first", "plural", "mangeons"),
|
|
||||||
("faire", "ind", "future", "first", "singular", "ferai"),
|
|
||||||
("pouvoir", "sbjv", "present", "third", "singular", "puisse"),
|
|
||||||
("prendre", "ind", "passe_simple", "third", "singular", "prit"),
|
|
||||||
("vendre", "ind", "present", "third", "singular", "vend"),
|
|
||||||
("commencer", "ind", "imperfect", "first", "singular", "commençais"),
|
|
||||||
]
|
|
||||||
ok = 0
|
|
||||||
for lemma, mood, tense, per, num, exp in tests:
|
|
||||||
got, conf = conjugate(lemma, mood, tense, per, num)
|
|
||||||
flag = "OK " if got == exp else "XX "
|
|
||||||
ok += got == exp
|
|
||||||
print(f" {flag}{lemma:10} {mood}/{tense:12} {per[:3]}.{num[:2]} -> {got:12} ({conf}) exp={exp}")
|
|
||||||
print(f"verb tests {ok}/{len(tests)}")
|
|
||||||
print(" gender: maison=", noun_gender("maison"), "chat=", noun_gender("chat"),
|
|
||||||
"cheval=", noun_gender("cheval"), "nation=", noun_gender("nation"))
|
|
||||||
print(" plural: cheval->", inflect_noun("cheval", "plural"),
|
|
||||||
"| bateau->", inflect_noun("bateau", "plural"),
|
|
||||||
"| prix->", inflect_noun("prix", "plural"),
|
|
||||||
"| chat->", inflect_noun("chat", "plural"))
|
|
||||||
print(" adj: petit/f/sg->", inflect_adj("petit", "f", "singular"),
|
|
||||||
"| beau/f/sg->", inflect_adj("beau", "f", "singular"),
|
|
||||||
"| heureux/f/sg->", inflect_adj("heureux", "f", "singular"),
|
|
||||||
"| national/m/pl->", inflect_adj("national", "m", "plural"))
|
|
||||||
print(" part: aller/f/sg->", participle("aller", "f", "singular"),
|
|
||||||
"| prendre/f/pl->", participle("prendre", "f", "plural"),
|
|
||||||
"| finir/m/pl->", participle("finir", "m", "plural"))
|
|
||||||
print(" ger: manger->", gerund("manger"), "| finir->", gerund("finir"))
|
|
||||||
@@ -1,588 +0,0 @@
|
|||||||
"""morphology_it_full.py — production-grade Italian morphological generator.
|
|
||||||
|
|
||||||
NOT a toy. Backed by three real, Wiktionary-lineage lexical sources:
|
|
||||||
|
|
||||||
VERBS
|
|
||||||
UniMorph Italian (github.com/unimorph/ita, CC-BY-SA 3.0)
|
|
||||||
10,009 verb lemmas × full paradigm, CLEAN orthography (no stress marks):
|
|
||||||
indicative present / imperfetto (PST;IPFV) / passato remoto (PST;PFV) /
|
|
||||||
futuro, condizionale (COND),
|
|
||||||
congiuntivo presente (SBJV;PRS) / imperfetto (SBJV;PST),
|
|
||||||
affirmative imperative, infinitive, gerundio (V.CVB;PRS),
|
|
||||||
past participle (masc-sg; fem/plural derived by vowel rule).
|
|
||||||
it_irreg_verbs.json — 66 high-frequency verbs UniMorph MISSES
|
|
||||||
(essere, avere, potere, uscire, tenere, prendere, piacere, …), extracted
|
|
||||||
from kaikki.org Italian, filtered to standard forms, and DE-STRESSED to
|
|
||||||
real orthography (kaikki marks tonic stress everywhere: pàrlo->parlo,
|
|
||||||
avùto->avuto; final legit accents kept: sarò, è). Built by build_it_irreg.py.
|
|
||||||
This layer takes priority — it supplies the two auxiliaries essere/avere,
|
|
||||||
which the whole passato-prossimo / essere-agreement system depends on.
|
|
||||||
|
|
||||||
NOUNS + ADJECTIVES — kaikki.org Italian (Wiktionary extract, CC-BY-SA 3.0)
|
|
||||||
noun lemmas WITH inherent gender (head-template arg) + real (often irregular)
|
|
||||||
plural — uomo->uomini, uovo->uova, dito->dita, città invariant — resolved
|
|
||||||
PER LEMMA, never guessed.
|
|
||||||
adjective lemmas with real feminine + masc/fem plural (italiano->italiana/
|
|
||||||
italiani/italiane, felice->felici invariant).
|
|
||||||
|
|
||||||
Fallbacks (degrade, never crash, on OOV input):
|
|
||||||
verbs : rule generator for regular -are/-ere/-ire (with -care/-gare h-insertion
|
|
||||||
and -ciare/-giare/-iare i-drop spelling rules)
|
|
||||||
nouns : gender heuristic (endings) + rule pluralization (ambiguous -co/-go FLAGGED)
|
|
||||||
adjs : -o/-a/-e gender rule + rule pluralization
|
|
||||||
|
|
||||||
Confidence flag on every form:
|
|
||||||
"lexicon" from UniMorph / kaikki-irregular / kaikki noun-adj (trust: high)
|
|
||||||
"rule" deterministic rule (trust: medium)
|
|
||||||
"fallback" could not inflect; returned lemma / ambiguous (trust: low -> FLAG)
|
|
||||||
|
|
||||||
Public API (used by realizer_it.py):
|
|
||||||
conjugate(lemma, mood, tense, person, number) -> (form, conf)
|
|
||||||
participle(lemma, gender="m", number="singular") -> (form, conf)
|
|
||||||
gerund(lemma) -> (form, conf)
|
|
||||||
noun_gender(lemma) -> "m"|"f"
|
|
||||||
inflect_noun(lemma, number, gender=None) -> (form, conf)
|
|
||||||
inflect_adj(lemma, gender, number) -> (form, conf)
|
|
||||||
lexicon_stats() -> dict
|
|
||||||
"""
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import pickle
|
|
||||||
|
|
||||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
_UNIMORPH = os.path.join(_HERE, "data", "ita.unimorph")
|
|
||||||
_IRREG = os.path.join(_HERE, "data", "it_irreg_verbs.json")
|
|
||||||
_KAIKKI = os.path.join(_HERE, "data", "kaikki_it.jsonl")
|
|
||||||
_CACHE = os.path.join(_HERE, "data", "it_morph_cache.pkl")
|
|
||||||
|
|
||||||
# ── (mood, tense) -> UniMorph feature set that must ALL be present ────────────────
|
|
||||||
_VERB_KEYMAP = {
|
|
||||||
("ind", "present"): {"IND", "PRS"},
|
|
||||||
("ind", "imperfect"): {"IND", "PST", "IPFV"},
|
|
||||||
("ind", "passato_remoto"): {"IND", "PST", "PFV"},
|
|
||||||
("ind", "future"): {"IND", "FUT"},
|
|
||||||
("ind", "conditional"): {"COND"},
|
|
||||||
("sbjv", "present"): {"SBJV", "PRS"},
|
|
||||||
("sbjv", "imperfect"): {"SBJV", "PST"},
|
|
||||||
("imp", "affirmative"): {"POS", "IMP"},
|
|
||||||
}
|
|
||||||
_PERSON = {"first": "1", "second": "2", "third": "3"}
|
|
||||||
_NUMBER = {"singular": "SG", "plural": "PL"}
|
|
||||||
|
|
||||||
|
|
||||||
def _feat_set(tag):
|
|
||||||
return set(tag.split(";"))
|
|
||||||
|
|
||||||
|
|
||||||
# ── build verb lexicon from UniMorph ─────────────────────────────────────────────
|
|
||||||
def _build_verbs():
|
|
||||||
verbs = {} # (lemma, "mood|tense|person|number") -> form
|
|
||||||
part = {} # lemma -> masc-sg past participle
|
|
||||||
ger = {} # lemma -> gerundio
|
|
||||||
with open(_UNIMORPH, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
line = line.rstrip("\n")
|
|
||||||
if not line or "\t" not in line:
|
|
||||||
continue
|
|
||||||
parts = line.split("\t")
|
|
||||||
if len(parts) != 3:
|
|
||||||
continue
|
|
||||||
lemma, form, tag = parts
|
|
||||||
f = _feat_set(tag)
|
|
||||||
head = tag.split(";")[0]
|
|
||||||
|
|
||||||
if head == "V.PTCP":
|
|
||||||
if "PST" in f:
|
|
||||||
part.setdefault(lemma, form)
|
|
||||||
continue
|
|
||||||
if head == "V.CVB": # gerundio (converb, present)
|
|
||||||
if "PRS" in f:
|
|
||||||
ger.setdefault(lemma, form)
|
|
||||||
continue
|
|
||||||
if head != "V":
|
|
||||||
continue
|
|
||||||
|
|
||||||
person = next((p for p in ("1", "2", "3") if p in f), None)
|
|
||||||
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
|
|
||||||
if person is None or number is None:
|
|
||||||
continue
|
|
||||||
for (mood, tense), req in _VERB_KEYMAP.items():
|
|
||||||
# exact-set discipline: PST;PFV must not match PST;IPFV, etc.
|
|
||||||
if not req <= f:
|
|
||||||
continue
|
|
||||||
# guard IND;PST ambiguity: require the specific aspect feature
|
|
||||||
if tense == "imperfect" and "PFV" in f:
|
|
||||||
continue
|
|
||||||
if tense == "passato_remoto" and "IPFV" in f:
|
|
||||||
continue
|
|
||||||
# COND must not also be a subjunctive/imperative slot
|
|
||||||
verbs.setdefault((lemma, f"{mood}|{tense}|{person}|{number}"), form)
|
|
||||||
break
|
|
||||||
return verbs, part, ger
|
|
||||||
|
|
||||||
|
|
||||||
# ── kaikki nouns + adjectives ────────────────────────────────────────────────────
|
|
||||||
_EXCL_FORM_TAGS = {"alternative", "archaic", "obsolete", "dialectal", "regional",
|
|
||||||
"diminutive", "augmentative", "pejorative", "comparative",
|
|
||||||
"superlative", "misspelling", "rare", "informal", "literary",
|
|
||||||
"poetic", "error-unrecognized-form", "apocopic", "obsolete",
|
|
||||||
"construed", "collective"}
|
|
||||||
|
|
||||||
|
|
||||||
def _kaikki_gender(arg):
|
|
||||||
if not arg:
|
|
||||||
return None
|
|
||||||
a = str(arg).lower()
|
|
||||||
if a.startswith("f"):
|
|
||||||
return "f"
|
|
||||||
if a.startswith("m"):
|
|
||||||
return "m"
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _build_nouns_adjs():
|
|
||||||
nouns = {} # lemma -> {"g","SG","PL"}
|
|
||||||
adjs = {} # lemma -> {("m","SG"),("f","SG"),("m","PL"),("f","PL")}
|
|
||||||
with open(_KAIKKI, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
try:
|
|
||||||
d = json.loads(line)
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
pos = d.get("pos")
|
|
||||||
word = d.get("word", "")
|
|
||||||
if not word or " " in word:
|
|
||||||
continue
|
|
||||||
forms = d.get("forms", []) or []
|
|
||||||
|
|
||||||
if pos == "noun":
|
|
||||||
ht = d.get("head_templates") or []
|
|
||||||
g = None
|
|
||||||
if ht:
|
|
||||||
g = _kaikki_gender((ht[0].get("args") or {}).get("1"))
|
|
||||||
if g is None:
|
|
||||||
tags = d.get("tags") or []
|
|
||||||
if "feminine" in tags:
|
|
||||||
g = "f"
|
|
||||||
elif "masculine" in tags:
|
|
||||||
g = "m"
|
|
||||||
pl = None
|
|
||||||
for x in forms:
|
|
||||||
t = set(x.get("tags") or [])
|
|
||||||
if "plural" in t and not (t & _EXCL_FORM_TAGS):
|
|
||||||
fm = x.get("form")
|
|
||||||
if fm and " " not in fm and fm != "#":
|
|
||||||
pl = fm
|
|
||||||
break
|
|
||||||
if word not in nouns:
|
|
||||||
nouns[word] = {"g": g, "SG": word, "PL": pl}
|
|
||||||
else:
|
|
||||||
cur = nouns[word]
|
|
||||||
if cur.get("g") is None and g:
|
|
||||||
cur["g"] = g
|
|
||||||
if not cur.get("PL") and pl:
|
|
||||||
cur["PL"] = pl
|
|
||||||
|
|
||||||
elif pos == "adj":
|
|
||||||
d0 = adjs.setdefault(word, {})
|
|
||||||
d0.setdefault(("m", "SG"), word)
|
|
||||||
for x in forms:
|
|
||||||
t = set(x.get("tags") or [])
|
|
||||||
fm = x.get("form")
|
|
||||||
if not fm or " " in fm or (t & _EXCL_FORM_TAGS):
|
|
||||||
continue
|
|
||||||
if "feminine" in t and "plural" in t:
|
|
||||||
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
|
|
||||||
elif "masculine" in t and "plural" in t:
|
|
||||||
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
|
|
||||||
elif "feminine" in t:
|
|
||||||
d0[("f", "SG")] = d0.get(("f", "SG")) or fm
|
|
||||||
elif "plural" in t: # invariant-gender adj (felice -> felici)
|
|
||||||
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
|
|
||||||
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
|
|
||||||
return nouns, adjs
|
|
||||||
|
|
||||||
|
|
||||||
def _build_cache():
|
|
||||||
verbs, part, ger = _build_verbs()
|
|
||||||
nouns, adjs = _build_nouns_adjs()
|
|
||||||
with open(_IRREG, encoding="utf-8") as fh:
|
|
||||||
irreg = json.load(fh)
|
|
||||||
data = {"verbs": verbs, "part": part, "ger": ger,
|
|
||||||
"nouns": nouns, "adjs": adjs, "irreg": irreg}
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "wb") as fh:
|
|
||||||
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
def _load():
|
|
||||||
if os.path.exists(_CACHE):
|
|
||||||
srcs = [_UNIMORPH, _KAIKKI, _IRREG]
|
|
||||||
newest = max(os.path.getmtime(s) for s in srcs if os.path.exists(s))
|
|
||||||
if os.path.getmtime(_CACHE) >= newest:
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "rb") as fh:
|
|
||||||
return pickle.load(fh)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return _build_cache()
|
|
||||||
|
|
||||||
|
|
||||||
_LEX = _load()
|
|
||||||
_VERBS, _PART, _GER, _NOUNS, _ADJS, _IRREGV = (
|
|
||||||
_LEX["verbs"], _LEX["part"], _LEX["ger"], _LEX["nouns"], _LEX["adjs"],
|
|
||||||
_LEX["irreg"])
|
|
||||||
|
|
||||||
|
|
||||||
# ── regular-ending rule fallback ─────────────────────────────────────────────────
|
|
||||||
def _vclass(lemma):
|
|
||||||
if lemma.endswith("are"):
|
|
||||||
return "are"
|
|
||||||
if lemma.endswith("ere"):
|
|
||||||
return "ere"
|
|
||||||
if lemma.endswith("ire"):
|
|
||||||
return "ire"
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
# endings [1sg,2sg,3sg,1pl,2pl,3pl]
|
|
||||||
_REG = {
|
|
||||||
("ind", "present", "are"): ["o", "i", "a", "iamo", "ate", "ano"],
|
|
||||||
("ind", "present", "ere"): ["o", "i", "e", "iamo", "ete", "ono"],
|
|
||||||
("ind", "present", "ire"): ["o", "i", "e", "iamo", "ite", "ono"],
|
|
||||||
("ind", "imperfect", "are"): ["avo", "avi", "ava", "avamo", "avate", "avano"],
|
|
||||||
("ind", "imperfect", "ere"): ["evo", "evi", "eva", "evamo", "evate", "evano"],
|
|
||||||
("ind", "imperfect", "ire"): ["ivo", "ivi", "iva", "ivamo", "ivate", "ivano"],
|
|
||||||
("ind", "passato_remoto", "are"): ["ai", "asti", "ò", "ammo", "aste", "arono"],
|
|
||||||
("ind", "passato_remoto", "ere"): ["ei", "esti", "é", "emmo", "este", "erono"],
|
|
||||||
("ind", "passato_remoto", "ire"): ["ii", "isti", "ì", "immo", "iste", "irono"],
|
|
||||||
("sbjv", "present", "are"): ["i", "i", "i", "iamo", "iate", "ino"],
|
|
||||||
("sbjv", "present", "ere"): ["a", "a", "a", "iamo", "iate", "ano"],
|
|
||||||
("sbjv", "present", "ire"): ["a", "a", "a", "iamo", "iate", "ano"],
|
|
||||||
("sbjv", "imperfect", "are"): ["assi", "assi", "asse", "assimo", "aste", "assero"],
|
|
||||||
("sbjv", "imperfect", "ere"): ["essi", "essi", "esse", "essimo", "este", "essero"],
|
|
||||||
("sbjv", "imperfect", "ire"): ["issi", "issi", "isse", "issimo", "iste", "issero"],
|
|
||||||
# imperative: 2sg,3sg(Lei),1pl,2pl,3pl (1sg has none)
|
|
||||||
("imp", "affirmative", "are"): [None, "a", "i", "iamo", "ate", "ino"],
|
|
||||||
("imp", "affirmative", "ere"): [None, "i", "a", "iamo", "ete", "ano"],
|
|
||||||
("imp", "affirmative", "ire"): [None, "i", "a", "iamo", "ite", "ano"],
|
|
||||||
}
|
|
||||||
# future / conditional attach to a stem = infinitive minus final -e, with
|
|
||||||
# -are -> -er (parlare->parler-), -ere/-ire keep (credere->creder-, dormir-)
|
|
||||||
_FUT = ["ò", "ai", "à", "emo", "ete", "anno"]
|
|
||||||
_COND = ["ei", "esti", "ebbe", "emmo", "este", "ebbero"]
|
|
||||||
|
|
||||||
|
|
||||||
def _slot_idx(person, number):
|
|
||||||
base = {"first": 0, "second": 1, "third": 2}[person]
|
|
||||||
return base + (0 if number == "singular" else 3)
|
|
||||||
|
|
||||||
|
|
||||||
def _fut_stem(lemma, vc):
|
|
||||||
body = lemma[:-3] # drop are/ere/ire
|
|
||||||
if vc == "are":
|
|
||||||
return body + "er"
|
|
||||||
return body + vc[0] + "r" # ere->er? no: keep vowel: creder-, dormir-
|
|
||||||
# NOTE corrected below
|
|
||||||
|
|
||||||
|
|
||||||
def _apply_are_spelling(stem, ending):
|
|
||||||
"""-care/-gare insert h before front endings; -ciare/-giare/-sciare/-iare drop i."""
|
|
||||||
front = ending[:1] in ("i", "e")
|
|
||||||
if stem.endswith(("c", "g")) and front:
|
|
||||||
return stem + "h" + ending
|
|
||||||
if stem.endswith(("ci", "gi", "sci")) and ending[:1] == "i":
|
|
||||||
return stem[:-1] + ending # mangi+iamo -> mangiamo
|
|
||||||
if stem.endswith("i") and ending[:1] == "i":
|
|
||||||
return stem[:-1] + ending # studi+iamo -> studiamo
|
|
||||||
return stem + ending
|
|
||||||
|
|
||||||
|
|
||||||
def _rule_conjugate(lemma, mood, tense, person, number):
|
|
||||||
vc = _vclass(lemma)
|
|
||||||
if vc is None:
|
|
||||||
return None
|
|
||||||
body = lemma[:-3]
|
|
||||||
i = _slot_idx(person, number)
|
|
||||||
if mood == "ind" and tense in ("future", "conditional"):
|
|
||||||
stem = body + "er" if vc == "are" else body + vc[0] + "r"
|
|
||||||
# ere: creder-, ire: dormir- -> body + 'e'/'i' + 'r'
|
|
||||||
if vc == "ere":
|
|
||||||
stem = body + "er"
|
|
||||||
elif vc == "ire":
|
|
||||||
stem = body + "ir"
|
|
||||||
end = (_FUT if tense == "future" else _COND)[i]
|
|
||||||
# spelling: -care/-gare -> cherò/gherò ; -ciare/-giare -> cerò/gerò
|
|
||||||
if vc == "are":
|
|
||||||
if body.endswith(("c", "g")):
|
|
||||||
stem = body + "her"
|
|
||||||
elif body.endswith(("ci", "gi", "sci")):
|
|
||||||
stem = body[:-1] + "er"
|
|
||||||
elif body.endswith("i"):
|
|
||||||
stem = body[:-1] + "er"
|
|
||||||
return stem + end
|
|
||||||
table = _REG.get((mood, tense, vc))
|
|
||||||
if not table:
|
|
||||||
return None
|
|
||||||
end = table[i]
|
|
||||||
if end is None:
|
|
||||||
return None
|
|
||||||
if vc == "are":
|
|
||||||
return _apply_are_spelling(body, end)
|
|
||||||
# -ere/-ire: guard against double-i (dormi+iamo -> dormiamo)
|
|
||||||
if body.endswith("i") and end[:1] == "i":
|
|
||||||
return body[:-1] + end
|
|
||||||
return body + end
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: verb conjugation ─────────────────────────────────────────────────────
|
|
||||||
def conjugate(lemma, mood, tense, person, number):
|
|
||||||
"""Return (surface, confidence). mood in ind|sbjv|imp; tense per _VERB_KEYMAP."""
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{number}"
|
|
||||||
ir = _IRREGV.get(lemma)
|
|
||||||
if ir and key in ir:
|
|
||||||
return ir[key], "lexicon"
|
|
||||||
p, n = _PERSON.get(person), _NUMBER.get(number)
|
|
||||||
if p and n:
|
|
||||||
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}"))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
r = _rule_conjugate(lemma, mood, tense, person, number)
|
|
||||||
if r:
|
|
||||||
return r, "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: participle + gerund ──────────────────────────────────────────────────
|
|
||||||
def _participle_msg(lemma):
|
|
||||||
"""Return (masc-sg participle, source) or (None, None)."""
|
|
||||||
ir = _IRREGV.get(lemma)
|
|
||||||
if ir and "part" in ir:
|
|
||||||
return ir["part"], "lexicon"
|
|
||||||
if lemma in _PART:
|
|
||||||
return _PART[lemma], "lexicon"
|
|
||||||
return None, None
|
|
||||||
|
|
||||||
|
|
||||||
def participle(lemma, gender="m", number="singular"):
|
|
||||||
"""Past participle with gender/number agreement (for essere-perfect & passives).
|
|
||||||
UniMorph/irregular give masc-sg; fem/plural derived by final-vowel swap
|
|
||||||
(-o -> -a/-i/-e), valid for regular -ato/-uto/-ito AND irregulars
|
|
||||||
(preso->presa/presi/prese, aperto->aperta/aperti/aperte, morto->morta/...)."""
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
g = "f" if gender == "f" else "m"
|
|
||||||
num = "SG" if number == "singular" else "PL"
|
|
||||||
msg, src = _participle_msg(lemma)
|
|
||||||
conf = "lexicon"
|
|
||||||
if msg is None:
|
|
||||||
vc = _vclass(lemma)
|
|
||||||
if vc == "are":
|
|
||||||
msg = lemma[:-3] + "ato"
|
|
||||||
elif vc == "ere":
|
|
||||||
msg = lemma[:-3] + "uto"
|
|
||||||
elif vc == "ire":
|
|
||||||
msg = lemma[:-3] + "ito"
|
|
||||||
else:
|
|
||||||
return lemma, "fallback"
|
|
||||||
conf = "rule"
|
|
||||||
# agreement: only -o participles inflect for gender+number
|
|
||||||
if msg.endswith("o"):
|
|
||||||
stem = msg[:-1]
|
|
||||||
suf = {"m|SG": "o", "f|SG": "a", "m|PL": "i", "f|PL": "e"}[f"{g}|{num}"]
|
|
||||||
return stem + suf, conf
|
|
||||||
return msg, conf # non -o participle: leave as-is (rare)
|
|
||||||
|
|
||||||
|
|
||||||
def gerund(lemma):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
ir = _IRREGV.get(lemma)
|
|
||||||
if ir and "ger" in ir:
|
|
||||||
return ir["ger"], "lexicon"
|
|
||||||
if lemma in _GER:
|
|
||||||
return _GER[lemma], "lexicon"
|
|
||||||
vc = _vclass(lemma)
|
|
||||||
if vc == "are":
|
|
||||||
return lemma[:-3] + "ando", "rule"
|
|
||||||
if vc in ("ere", "ire"):
|
|
||||||
return lemma[:-3] + "endo", "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: noun gender + number ─────────────────────────────────────────────────
|
|
||||||
_FEM_SUF = ("zione", "sione", "gione", "tà", "tù", "trice", "aggine", "udine",
|
|
||||||
"igine", "ie", "essa", "izia", "ezza")
|
|
||||||
_MASC_SUF = ("ore", "ame", "iere", "ale", "ile")
|
|
||||||
|
|
||||||
|
|
||||||
def _gender_heuristic(noun):
|
|
||||||
for suf in _FEM_SUF:
|
|
||||||
if noun.endswith(suf):
|
|
||||||
return "f"
|
|
||||||
for suf in _MASC_SUF:
|
|
||||||
if noun.endswith(suf):
|
|
||||||
return "m"
|
|
||||||
if noun.endswith("o"):
|
|
||||||
return "m"
|
|
||||||
if noun.endswith("a"):
|
|
||||||
return "f"
|
|
||||||
if noun.endswith("à") or noun.endswith("ù"):
|
|
||||||
return "f"
|
|
||||||
return "m" # -e and consonant-final loanwords default masculine
|
|
||||||
|
|
||||||
|
|
||||||
def noun_gender(lemma):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
if d and d.get("g") in ("m", "f"):
|
|
||||||
return d["g"]
|
|
||||||
return _gender_heuristic(lemma)
|
|
||||||
|
|
||||||
|
|
||||||
def _rule_plural(noun, gender):
|
|
||||||
"""Deterministic Italian pluralization. Returns (form, ok); ok=False FLAGS an
|
|
||||||
ambiguous case the lexicon would normally resolve (-co/-go palatalization)."""
|
|
||||||
if not noun:
|
|
||||||
return noun, True
|
|
||||||
# invariant: accented final vowel, consonant-final, monosyllable, -i final
|
|
||||||
if noun[-1:] in ("à", "è", "é", "ì", "í", "ò", "ó", "ù", "ú"):
|
|
||||||
return noun, True
|
|
||||||
if noun[-1:] not in ("a", "e", "o", "i", "u"):
|
|
||||||
return noun, True # consonant-final loanword: invariant
|
|
||||||
if noun.endswith("i"):
|
|
||||||
return noun, True # e.g. crisi, analisi: invariant
|
|
||||||
if noun.endswith("io"):
|
|
||||||
return noun[:-2] + "i", True # figlio->figli (unstressed i)
|
|
||||||
if noun.endswith("cia") or noun.endswith("gia"):
|
|
||||||
# vowel before cia/gia -> -cie/-gie ; consonant -> -ce/-ge (approx)
|
|
||||||
return noun[:-2] + "e", True # arancia->arance (majority)
|
|
||||||
if noun.endswith("ca"):
|
|
||||||
return noun[:-2] + "che", True # amica->amiche
|
|
||||||
if noun.endswith("ga"):
|
|
||||||
return noun[:-2] + "ghe", True
|
|
||||||
if noun.endswith("co"):
|
|
||||||
return noun[:-2] + "chi", False # AMBIGUOUS (amico->amici) -> flag
|
|
||||||
if noun.endswith("go"):
|
|
||||||
return noun[:-2] + "ghi", False # AMBIGUOUS (psicologo->psicologi)
|
|
||||||
if noun.endswith("a"):
|
|
||||||
return noun[:-1] + "e", True # casa->case (m -a: -i, but rare)
|
|
||||||
if noun.endswith("o"):
|
|
||||||
return noun[:-1] + "i", True # libro->libri
|
|
||||||
if noun.endswith("e"):
|
|
||||||
return noun[:-1] + "i", True # cane->cani, chiave->chiavi
|
|
||||||
return noun, True
|
|
||||||
|
|
||||||
|
|
||||||
def inflect_noun(lemma, number, gender=None):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
if number == "singular":
|
|
||||||
return (d["SG"] if d and d.get("SG") else lemma), ("lexicon" if d else "rule")
|
|
||||||
if d and d.get("PL"):
|
|
||||||
return d["PL"], "lexicon"
|
|
||||||
g = gender or noun_gender(lemma)
|
|
||||||
form, ok = _rule_plural(lemma, g)
|
|
||||||
return form, ("rule" if ok else "fallback")
|
|
||||||
|
|
||||||
|
|
||||||
# adjectives whose kaikki entries are unreliable (messy inflection templates):
|
|
||||||
# supply audited regular agreement forms (prenominal apocope handled in realizer).
|
|
||||||
_ADJ_FIX = {
|
|
||||||
"bello": {("m", "SG"): "bello", ("f", "SG"): "bella",
|
|
||||||
("m", "PL"): "belli", ("f", "PL"): "belle"},
|
|
||||||
"quello": {("m", "SG"): "quello", ("f", "SG"): "quella",
|
|
||||||
("m", "PL"): "quelli", ("f", "PL"): "quelle"},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: adjective agreement ──────────────────────────────────────────────────
|
|
||||||
def inflect_adj(lemma, gender, number):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
g = "f" if gender == "f" else "m"
|
|
||||||
num = "SG" if number == "singular" else "PL"
|
|
||||||
fix = _ADJ_FIX.get(lemma)
|
|
||||||
if fix and (g, num) in fix:
|
|
||||||
return fix[(g, num)], "lexicon"
|
|
||||||
d = _ADJS.get(lemma)
|
|
||||||
if d:
|
|
||||||
form = d.get((g, num))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
sg = d.get((g, "SG")) or d.get(("m", "SG")) or lemma
|
|
||||||
if num == "PL":
|
|
||||||
pl, ok = _rule_plural(sg, g)
|
|
||||||
return pl, ("rule" if ok else "fallback")
|
|
||||||
return sg, "lexicon"
|
|
||||||
# rule fallback
|
|
||||||
a = lemma
|
|
||||||
if a.endswith("o"): # -o/-a/-i/-e class
|
|
||||||
base = a[:-1]
|
|
||||||
suf = {"m|SG": "o", "f|SG": "a", "m|PL": "i", "f|PL": "e"}[f"{g}|{num}"]
|
|
||||||
return base + suf, "rule"
|
|
||||||
if a.endswith("e"): # felice-class: SG invariant, PL -i
|
|
||||||
if num == "PL":
|
|
||||||
return a[:-1] + "i", "rule"
|
|
||||||
return a, "rule"
|
|
||||||
if num == "PL":
|
|
||||||
p, ok = _rule_plural(a, g)
|
|
||||||
return p, ("rule" if ok else "fallback")
|
|
||||||
return a, "rule"
|
|
||||||
|
|
||||||
|
|
||||||
def lexicon_stats():
|
|
||||||
return {
|
|
||||||
"verb_source": "UniMorph Italian (github.com/unimorph/ita) + kaikki.org "
|
|
||||||
"irregulars (de-stressed)",
|
|
||||||
"noun_adj_source": "kaikki.org Italian (Wiktionary extract)",
|
|
||||||
"license": "CC-BY-SA 3.0 (Wiktionary/UniMorph lineage)",
|
|
||||||
"unimorph_verb_forms": len(_VERBS),
|
|
||||||
"unimorph_verb_lemmas": len({k[0] for k in _VERBS}),
|
|
||||||
"irregular_verb_lemmas": len(_IRREGV),
|
|
||||||
"participle_lemmas": len(_PART),
|
|
||||||
"gerund_lemmas": len(_GER),
|
|
||||||
"noun_lemmas": len(_NOUNS),
|
|
||||||
"adj_lemmas": len(_ADJS),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
|
|
||||||
tests = [
|
|
||||||
("parlare", "ind", "present", "first", "singular", "parlo"),
|
|
||||||
("essere", "ind", "present", "third", "singular", "è"),
|
|
||||||
("avere", "ind", "present", "first", "singular", "ho"),
|
|
||||||
("mangiare", "ind", "present", "second", "singular", "mangi"),
|
|
||||||
("finire", "ind", "present", "first", "singular", "finisco"),
|
|
||||||
("andare", "ind", "present", "third", "plural", "vanno"),
|
|
||||||
("fare", "ind", "future", "first", "singular", "farò"),
|
|
||||||
("potere", "sbjv", "present", "third", "singular", "possa"),
|
|
||||||
("prendere", "ind", "passato_remoto", "first", "singular", "presi"),
|
|
||||||
("cercare", "ind", "present", "second", "singular", "cerchi"),
|
|
||||||
("dormire", "ind", "present", "third", "plural", "dormono"),
|
|
||||||
("credere", "ind", "future", "first", "singular", "crederò"),
|
|
||||||
]
|
|
||||||
ok = 0
|
|
||||||
for lemma, mood, tense, per, num, exp in tests:
|
|
||||||
got, conf = conjugate(lemma, mood, tense, per, num)
|
|
||||||
flag = "OK " if got == exp else "XX "
|
|
||||||
ok += got == exp
|
|
||||||
print(f" {flag}{lemma:9} {mood}/{tense:14} {per[:3]}.{num[:2]} -> {got:12} ({conf}) exp={exp}")
|
|
||||||
print(f"verb tests {ok}/{len(tests)}")
|
|
||||||
print(" gender: casa=", noun_gender("casa"), "problema=", noun_gender("problema"),
|
|
||||||
"mano=", noun_gender("mano"), "città=", noun_gender("città"),
|
|
||||||
"cane=", noun_gender("cane"))
|
|
||||||
print(" plural: uomo->", inflect_noun("uomo", "plural"),
|
|
||||||
"| uovo->", inflect_noun("uovo", "plural"),
|
|
||||||
"| città->", inflect_noun("città", "plural"),
|
|
||||||
"| amico->", inflect_noun("amico", "plural"),
|
|
||||||
"| casa->", inflect_noun("casa", "plural"))
|
|
||||||
print(" adj: italiano/f/pl->", inflect_adj("italiano", "f", "plural"),
|
|
||||||
"| felice/m/pl->", inflect_adj("felice", "m", "plural"),
|
|
||||||
"| bello/f/sg->", inflect_adj("bello", "f", "singular"))
|
|
||||||
print(" part: aprire/f/sg->", participle("aprire", "f", "singular"),
|
|
||||||
"| prendere/m/pl->", participle("prendere", "m", "plural"),
|
|
||||||
"| andare/f/sg->", participle("andare", "f", "singular"))
|
|
||||||
print(" ger: fare->", gerund("fare"), "| parlare->", gerund("parlare"))
|
|
||||||
@@ -1,666 +0,0 @@
|
|||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""morphology_lat_full.py — production-grade Latin morphological generator.
|
|
||||||
|
|
||||||
Latin is the FLAGSHIP dead-language realizer. It rides the *architecture* of the
|
|
||||||
Romance/Italic engine (the same Realization / spec-driven design and the UniMorph
|
|
||||||
loader pattern from morphology_it_full.py) but with the CASE SYSTEM RESTORED —
|
|
||||||
the feature Romance lost. Latin therefore exercises machinery the modern Romance
|
|
||||||
siblings never needed: 5 declensions x 6 cases x 2 numbers x 3 genders, plus a
|
|
||||||
4-conjugation verb system with tense/mood/voice.
|
|
||||||
|
|
||||||
DATA (real, attested — no fabrication):
|
|
||||||
|
|
||||||
NOUNS + ADJECTIVES — UniMorph Latin (github.com/unimorph/lat, CC-BY-SA 3.0)
|
|
||||||
163,182 N forms across ~thousands of lemmas, each with the full case paradigm
|
|
||||||
N;NOM/GEN/DAT/ACC/ABL/VOC;SG/PL (real inflected forms, WITH macrons:
|
|
||||||
puella->puellam, rēx->rēgis, corpus->corporis).
|
|
||||||
244,197 ADJ forms with case x GENDER x number, incl. UniMorph's combined
|
|
||||||
tags (GEN+DAT, MASC+FEM, MASC+FEM+NEUT) which are split on load.
|
|
||||||
462,668 V.PTCP forms (participles) also carry case/gender/number.
|
|
||||||
UniMorph N tags DO NOT encode inherent gender, so noun gender is inferred
|
|
||||||
from the declension (nom-sg + gen-sg endings) with a curated exceptions
|
|
||||||
map — the standard, attestable rule (1st decl -a/-ae = fem, 2nd -us/-i =
|
|
||||||
masc, -um = neut, ...).
|
|
||||||
|
|
||||||
VERBS — RULE ENGINE (honest gap: UniMorph Latin's verb list is a 947-lemma
|
|
||||||
sample of rare/prefixed verbs that MISSES every core textbook verb — amō,
|
|
||||||
videō, sum, regō, ... are all absent). Latin conjugation is, however, highly
|
|
||||||
regular, so verbs are generated by a deterministic 4-conjugation engine over
|
|
||||||
curated principal parts (present / perfect / supine stems), sourced from
|
|
||||||
standard references. Irregulars (sum, possum, eō, ferō, volō, nōlō, mālō)
|
|
||||||
are curated full tables. Forms are flagged "rule" (not "lexicon") for honesty.
|
|
||||||
|
|
||||||
Confidence flag on every form (same contract as the Romance engine):
|
|
||||||
"lexicon" from UniMorph (trust: high)
|
|
||||||
"rule" deterministic morphology rule (trust: medium)
|
|
||||||
"fallback" could not inflect; returned lemma (trust: low -> FLAG)
|
|
||||||
|
|
||||||
Public API (used by realizer_lat.py):
|
|
||||||
decline_noun(lemma, case, number) -> (form, conf)
|
|
||||||
noun_gender(lemma) -> "m"|"f"|"n"
|
|
||||||
decline_adj(lemma, case, gender, number) -> (form, conf)
|
|
||||||
conjugate(lemma, tense, mood, voice, person, number) -> (form, conf)
|
|
||||||
participle(lemma, kind, case, gender, number) -> (form, conf) # kind: prs|pfv|fut
|
|
||||||
infinitive(lemma, tense="present", voice="active") -> (form, conf)
|
|
||||||
lexicon_stats() -> dict
|
|
||||||
"""
|
|
||||||
import os
|
|
||||||
import pickle
|
|
||||||
|
|
||||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
_UNIMORPH = os.path.join(_HERE, "data", "lat.unimorph")
|
|
||||||
_CACHE = os.path.join(_HERE, "data", "lat_morph_cache.pkl")
|
|
||||||
|
|
||||||
_CASES = ("NOM", "GEN", "DAT", "ACC", "ABL", "VOC")
|
|
||||||
_CASE_MAP = {"nom": "NOM", "gen": "GEN", "dat": "DAT", "acc": "ACC",
|
|
||||||
"abl": "ABL", "voc": "VOC"}
|
|
||||||
_NUM = {"singular": "SG", "plural": "PL"}
|
|
||||||
_GEN = {"m": "MASC", "f": "FEM", "n": "NEUT"}
|
|
||||||
|
|
||||||
|
|
||||||
# ── UniMorph loader: noun + adjective + participle case paradigms ────────────────
|
|
||||||
def _build_cache():
|
|
||||||
nouns = {} # lemma -> {(CASE, NUM): form}
|
|
||||||
adjs = {} # lemma -> {(CASE, GEN, NUM): form}
|
|
||||||
ptcps = {} # lemma -> {(CASE, GEN, NUM): form} (from V.PTCP; keyed loosely)
|
|
||||||
with open(_UNIMORPH, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
line = line.rstrip("\n")
|
|
||||||
if not line or "\t" not in line:
|
|
||||||
continue
|
|
||||||
parts = line.split("\t")
|
|
||||||
if len(parts) != 3:
|
|
||||||
continue
|
|
||||||
lemma, form, tag = parts
|
|
||||||
feats = tag.split(";")
|
|
||||||
head = feats[0]
|
|
||||||
fs = set(feats)
|
|
||||||
case = next((c for c in _CASES if c in fs), None)
|
|
||||||
# handle combined case tags like GEN+DAT
|
|
||||||
if case is None:
|
|
||||||
for f in feats:
|
|
||||||
if "+" in f and any(c in f.split("+") for c in _CASES):
|
|
||||||
case = [c for c in _CASES if c in f.split("+")]
|
|
||||||
break
|
|
||||||
num = "SG" if "SG" in fs else ("PL" if "PL" in fs else None)
|
|
||||||
if case is None or num is None:
|
|
||||||
continue
|
|
||||||
cases = case if isinstance(case, list) else [case]
|
|
||||||
|
|
||||||
if head == "N":
|
|
||||||
d = nouns.setdefault(lemma, {})
|
|
||||||
for c in cases:
|
|
||||||
d.setdefault((c, num), form)
|
|
||||||
elif head == "ADJ":
|
|
||||||
# gender may be combined: MASC+FEM+NEUT, MASC+FEM
|
|
||||||
genders = []
|
|
||||||
for g in ("MASC", "FEM", "NEUT"):
|
|
||||||
if any(g == x or (g in x.split("+")) for x in feats):
|
|
||||||
genders.append(g)
|
|
||||||
if not genders:
|
|
||||||
genders = ["MASC", "FEM", "NEUT"]
|
|
||||||
d = adjs.setdefault(lemma, {})
|
|
||||||
for c in cases:
|
|
||||||
for g in genders:
|
|
||||||
d.setdefault((c, g, num), form)
|
|
||||||
data = {"nouns": nouns, "adjs": adjs, "ptcps": ptcps}
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "wb") as fh:
|
|
||||||
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
def _load():
|
|
||||||
if os.path.exists(_CACHE) and os.path.exists(_UNIMORPH):
|
|
||||||
if os.path.getmtime(_CACHE) >= os.path.getmtime(_UNIMORPH):
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "rb") as fh:
|
|
||||||
return pickle.load(fh)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return _build_cache()
|
|
||||||
|
|
||||||
|
|
||||||
_LEX = _load()
|
|
||||||
_NOUNS, _ADJS = _LEX["nouns"], _LEX["adjs"]
|
|
||||||
|
|
||||||
|
|
||||||
# ── noun gender inference (declension-based, curated exceptions) ─────────────────
|
|
||||||
# Real, attestable rule: gender follows declension + nominative shape, with the
|
|
||||||
# standard closed set of exceptions.
|
|
||||||
_GENDER_EXC = {
|
|
||||||
# 1st-declension masculines (people/agents)
|
|
||||||
"agricola": "m", "poēta": "m", "nauta": "m", "incola": "m", "scrība": "m",
|
|
||||||
"auriga": "m", "pīrāta": "m", "athlēta": "m",
|
|
||||||
# 2nd-declension neuters / feminines
|
|
||||||
"vīrus": "n", "vulgus": "n", "pelagus": "n", "humus": "f",
|
|
||||||
# common 3rd-declension whose gender the ending would mispredict
|
|
||||||
"rēx": "m", "dux": "m", "mīles": "m", "pater": "m", "frāter": "m",
|
|
||||||
"homō": "m", "leō": "m", "sōl": "m", "mōns": "m", "pōns": "m", "fōns": "m",
|
|
||||||
"sanguis": "m", "ōrdō": "m", "sermō": "m", "amor": "m", "dolor": "m",
|
|
||||||
"labor": "m", "timor": "m", "honor": "m", "color": "m", "pēs": "m",
|
|
||||||
"dēns": "m", "flōs": "m", "mōs": "m", "mensis": "m", "orbis": "m",
|
|
||||||
"piscis": "m", "ignis": "m", "collis": "m", "grex": "m", "prīnceps": "m",
|
|
||||||
"māter": "f", "soror": "f", "uxor": "f", "mulier": "f", "virgō": "f",
|
|
||||||
"urbs": "f", "arx": "f", "pāx": "f", "lēx": "f", "lūx": "f", "vōx": "f",
|
|
||||||
"nox": "f", "nix": "f", "vīs": "f", "salūs": "f", "virtūs": "f",
|
|
||||||
"aetās": "f", "cīvitās": "f", "lībertās": "f", "vēritās": "f", "voluptās": "f",
|
|
||||||
"nātiō": "f", "ratiō": "f", "ōrātiō": "f", "legiō": "f", "regiō": "f",
|
|
||||||
"mens": "f", "gens": "f", "ars": "f", "pars": "f", "mors": "f", "sors": "f",
|
|
||||||
"nāvis": "f", "turris": "f", "avis": "f", "vallis": "f", "classis": "f",
|
|
||||||
"corpus": "n", "tempus": "n", "opus": "n", "genus": "n", "onus": "n",
|
|
||||||
"pectus": "n", "latus": "n", "vulnus": "n", "scelus": "n", "sīdus": "n",
|
|
||||||
"caput": "n", "iter": "n", "flūmen": "n", "nōmen": "n", "carmen": "n",
|
|
||||||
"agmen": "n", "certāmen": "n", "lūmen": "n", "ōmen": "n", "cōgnōmen": "n",
|
|
||||||
"mare": "n", "animal": "n", "exemplar": "n", "rēte": "n",
|
|
||||||
# 4th-declension exceptions
|
|
||||||
"manus": "f", "domus": "f", "tribus": "f", "porticus": "f", "īdūs": "f",
|
|
||||||
"cornū": "n", "genū": "n", "gelū": "n", "verū": "n",
|
|
||||||
# 5th-declension
|
|
||||||
"diēs": "m", "merīdiēs": "m",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _infer_gender(lemma):
|
|
||||||
if lemma in _GENDER_EXC:
|
|
||||||
return _GENDER_EXC[lemma]
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
nom = d.get(("NOM", "SG")) if d else lemma
|
|
||||||
gen = d.get(("GEN", "SG")) if d else None
|
|
||||||
nom = nom or lemma
|
|
||||||
# 5th declension: gen -eī / -ēī
|
|
||||||
if gen and (gen.endswith("eī") or gen.endswith("ēī")):
|
|
||||||
return "f"
|
|
||||||
# 1st declension: nom -a, gen -ae
|
|
||||||
if nom.endswith("a") and (not gen or gen.endswith("ae")):
|
|
||||||
return "f"
|
|
||||||
# 2nd declension neuter: nom -um
|
|
||||||
if nom.endswith("um"):
|
|
||||||
return "n"
|
|
||||||
# 2nd declension masc: nom -us/-er/-ir, gen -ī
|
|
||||||
if (nom.endswith("us") or nom.endswith("er") or nom.endswith("ir")) and \
|
|
||||||
(not gen or gen.endswith("ī")):
|
|
||||||
return "m"
|
|
||||||
# 4th declension: gen -ūs
|
|
||||||
if gen and gen.endswith("ūs"):
|
|
||||||
return "n" if nom.endswith("ū") else "m"
|
|
||||||
# 3rd declension neuters by common nom endings
|
|
||||||
if nom.endswith(("men", "us", "ur", "al", "ar", "e", "ma")):
|
|
||||||
# -us here is 3rd-decl neuter type (corpus) only if gen shows -oris/-eris
|
|
||||||
if nom.endswith("us") and gen and (gen.endswith("oris") or gen.endswith("eris")
|
|
||||||
or gen.endswith("uris")):
|
|
||||||
return "n"
|
|
||||||
if nom.endswith(("men", "al", "ar", "e")):
|
|
||||||
return "n"
|
|
||||||
# default 3rd-declension: masculine (most common)
|
|
||||||
return "m"
|
|
||||||
|
|
||||||
|
|
||||||
_GENDER_CACHE = {}
|
|
||||||
|
|
||||||
|
|
||||||
def noun_gender(lemma):
|
|
||||||
lemma = lemma.strip()
|
|
||||||
if lemma not in _GENDER_CACHE:
|
|
||||||
_GENDER_CACHE[lemma] = _infer_gender(lemma)
|
|
||||||
return _GENDER_CACHE[lemma]
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: noun declension ─────────────────────────────────────────────────────
|
|
||||||
def decline_noun(lemma, case, number):
|
|
||||||
lemma = lemma.strip()
|
|
||||||
C = _CASE_MAP.get(case, case.upper())
|
|
||||||
N = _NUM.get(number, number)
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
if d and (C, N) in d:
|
|
||||||
return d[(C, N)], "lexicon"
|
|
||||||
# abl sg often == the -e/-o form; try nom fallback
|
|
||||||
if d:
|
|
||||||
# try VOC==NOM, ACC neuter==NOM etc are already in data; last resort lemma
|
|
||||||
return lemma, "fallback"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: adjective declension ────────────────────────────────────────────────
|
|
||||||
def decline_adj(lemma, case, gender, number):
|
|
||||||
lemma = lemma.strip()
|
|
||||||
C = _CASE_MAP.get(case, case.upper())
|
|
||||||
G = _GEN.get(gender, gender.upper())
|
|
||||||
N = _NUM.get(number, number)
|
|
||||||
d = _ADJS.get(lemma)
|
|
||||||
if d and (C, G, N) in d:
|
|
||||||
return d[(C, G, N)], "lexicon"
|
|
||||||
# try other gender (some adjs listed only under MASC+FEM etc handled at load)
|
|
||||||
if d:
|
|
||||||
for altG in ("MASC", "FEM", "NEUT"):
|
|
||||||
if (C, altG, N) in d:
|
|
||||||
return d[(C, altG, N)], "lexicon"
|
|
||||||
return lemma, "fallback"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
# ═══════════════════════════════════════════════════════════════════════════════
|
|
||||||
# VERB RULE ENGINE (4 conjugations + curated irregulars)
|
|
||||||
# ═══════════════════════════════════════════════════════════════════════════════
|
|
||||||
# Curated principal parts for common attested verbs:
|
|
||||||
# lemma -> (conj, present_stem, perfect_stem, supine_stem)
|
|
||||||
# conj in {1,2,3,"3io",4}. Stems carry macrons (matching UniMorph orthography).
|
|
||||||
_VERBS = {
|
|
||||||
"amō": (1, "am", "amāv", "amāt"),
|
|
||||||
"laudō": (1, "laud", "laudāv", "laudāt"),
|
|
||||||
"portō": (1, "port", "portāv", "portāt"),
|
|
||||||
"vocō": (1, "voc", "vocāv", "vocāt"),
|
|
||||||
"dō": (1, "d", "ded", "dat"),
|
|
||||||
"spectō": (1, "spect", "spectāv", "spectāt"),
|
|
||||||
"pugnō": (1, "pugn", "pugnāv", "pugnāt"),
|
|
||||||
"labōrō": (1, "labōr", "labōrāv", "labōrāt"),
|
|
||||||
"necō": (1, "nec", "necāv", "necāt"),
|
|
||||||
"parō": (1, "par", "parāv", "parāt"),
|
|
||||||
"cōgitō": (1, "cōgit", "cōgitāv", "cōgitāt"),
|
|
||||||
"habitō": (1, "habit", "habitāv", "habitāt"),
|
|
||||||
"nārrō": (1, "nārr", "nārrāv", "nārrāt"),
|
|
||||||
"servō": (1, "serv", "servāv", "servāt"),
|
|
||||||
"superō": (1, "super", "superāv", "superāt"),
|
|
||||||
"oppugnō": (1, "oppugn", "oppugnāv", "oppugnāt"),
|
|
||||||
"ambulō": (1, "ambul", "ambulāv", "ambulāt"),
|
|
||||||
"clāmō": (1, "clām", "clāmāv", "clāmāt"),
|
|
||||||
"vulnerō": (1, "vulner", "vulnerāv", "vulnerāt"),
|
|
||||||
"aedificō": (1, "aedific", "aedificāv", "aedificāt"),
|
|
||||||
"expugnō": (1, "expugn", "expugnāv", "expugnāt"),
|
|
||||||
"dēfendō": (3, "dēfend", "dēfend", "dēfēns"),
|
|
||||||
"petō": (3, "pet", "petīv", "petīt"),
|
|
||||||
"occīdō": (3, "occīd", "occīd", "occīs"),
|
|
||||||
"interficiō": ("3io", "interfic", "interfēc", "interfect"),
|
|
||||||
"timeō": (2, "tim", "timu", None),
|
|
||||||
"iaceō": (2, "iac", "iacu", None),
|
|
||||||
"pāreō": (2, "pār", "pāru", "pārit"),
|
|
||||||
"respondeō": (2, "respond", "respond", "respōns"),
|
|
||||||
"vertō": (3, "vert", "vert", "vers"),
|
|
||||||
"ostendō": (3, "ostend", "ostend", "ostent"),
|
|
||||||
"cōnstituō": (3, "cōnstitu", "cōnstitu", "cōnstitūt"),
|
|
||||||
"cōgnōscō": (3, "cōgnōsc", "cōgnōv", "cōgnit"),
|
|
||||||
"crēdō": (3, "crēd", "crēdid", "crēdit"),
|
|
||||||
"ēdūcō": (3, "ēdūc", "ēdūx", "ēduct"),
|
|
||||||
"cōnservō": (1, "cōnserv", "cōnservāv", "cōnservāt"),
|
|
||||||
"iuvō": (1, "iuv", "iūv", "iūt"),
|
|
||||||
"dēbeō": (2, "dēb", "dēbu", "dēbit"),
|
|
||||||
"moneō": (2, "mon", "monu", "monit"),
|
|
||||||
"videō": (2, "vid", "vīd", "vīs"),
|
|
||||||
"habeō": (2, "hab", "habu", "habit"),
|
|
||||||
"teneō": (2, "ten", "tenu", "tent"),
|
|
||||||
"timeō": (2, "tim", "timu", None),
|
|
||||||
"terreō": (2, "terr", "terru", "territ"),
|
|
||||||
"dēleō": (2, "dēl", "dēlēv", "dēlēt"),
|
|
||||||
"iubeō": (2, "iub", "iuss", "iuss"),
|
|
||||||
"maneō": (2, "man", "māns", "māns"),
|
|
||||||
"moveō": (2, "mov", "mōv", "mōt"),
|
|
||||||
"doceō": (2, "doc", "docu", "doct"),
|
|
||||||
"sedeō": (2, "sed", "sēd", "sess"),
|
|
||||||
"rīdeō": (2, "rīd", "rīs", "rīs"),
|
|
||||||
"regō": (3, "reg", "rēx", "rēct"),
|
|
||||||
"dūcō": (3, "dūc", "dūx", "duct"),
|
|
||||||
"scrībō": (3, "scrīb", "scrīps", "scrīpt"),
|
|
||||||
"mittō": (3, "mitt", "mīs", "miss"),
|
|
||||||
"pōnō": (3, "pōn", "posu", "posit"),
|
|
||||||
"agō": (3, "ag", "ēg", "āct"),
|
|
||||||
"dīcō": (3, "dīc", "dīx", "dict"),
|
|
||||||
"gerō": (3, "ger", "gess", "gest"),
|
|
||||||
"vincō": (3, "vinc", "vīc", "vict"),
|
|
||||||
"petō": (3, "pet", "petīv", "petīt"),
|
|
||||||
"legō": (3, "leg", "lēg", "lēct"),
|
|
||||||
"currō": (3, "curr", "cucurr", "curs"),
|
|
||||||
"vīvō": (3, "vīv", "vīx", "vīct"),
|
|
||||||
"quaerō": (3, "quaer", "quaesīv", "quaesīt"),
|
|
||||||
"trahō": (3, "trah", "trāx", "tract"),
|
|
||||||
"claudō": (3, "claud", "claus", "claus"),
|
|
||||||
"cōgō": (3, "cōg", "coēg", "coāct"),
|
|
||||||
"relinquō": (3, "relinqu", "relīqu", "relict"),
|
|
||||||
"capiō": ("3io", "cap", "cēp", "capt"),
|
|
||||||
"faciō": ("3io", "fac", "fēc", "fact"),
|
|
||||||
"iaciō": ("3io", "iac", "iēc", "iact"),
|
|
||||||
"rapiō": ("3io", "rap", "rapu", "rapt"),
|
|
||||||
"fugiō": ("3io", "fug", "fūg", "fugit"),
|
|
||||||
"cupiō": ("3io", "cup", "cupīv", "cupīt"),
|
|
||||||
"accipiō": ("3io", "accip", "accēp", "accept"),
|
|
||||||
"audiō": (4, "aud", "audīv", "audīt"),
|
|
||||||
"veniō": (4, "ven", "vēn", "vent"),
|
|
||||||
"sciō": (4, "sc", "scīv", "scīt"),
|
|
||||||
"sentiō": (4, "sent", "sēns", "sēns"),
|
|
||||||
"mūniō": (4, "mūn", "mūnīv", "mūnīt"),
|
|
||||||
"dormiō": (4, "dorm", "dormīv", "dormīt"),
|
|
||||||
"aperiō": (4, "aper", "aperu", "apert"),
|
|
||||||
"inveniō": (4, "inven", "invēn", "invent"),
|
|
||||||
}
|
|
||||||
|
|
||||||
# ── Present-system paradigms: full ending tables per conjugation, attached to the
|
|
||||||
# bare present stem (pstem). Hardcoded from the standard grammar with correct
|
|
||||||
# macrons/vowel-lengths — deterministic and independently verifiable. Keys:
|
|
||||||
# (tense, mood, voice) -> {conj: [1sg,2sg,3sg,1pl,2pl,3pl]}
|
|
||||||
_PARADIGM = {
|
|
||||||
("present", "ind", "active"): {
|
|
||||||
1: ["ō", "ās", "at", "āmus", "ātis", "ant"],
|
|
||||||
2: ["eō", "ēs", "et", "ēmus", "ētis", "ent"],
|
|
||||||
3: ["ō", "is", "it", "imus", "itis", "unt"],
|
|
||||||
"3io": ["iō", "is", "it", "imus", "itis", "iunt"],
|
|
||||||
4: ["iō", "īs", "it", "īmus", "ītis", "iunt"],
|
|
||||||
},
|
|
||||||
("present", "ind", "passive"): {
|
|
||||||
1: ["or", "āris", "ātur", "āmur", "āminī", "antur"],
|
|
||||||
2: ["eor", "ēris", "ētur", "ēmur", "ēminī", "entur"],
|
|
||||||
3: ["or", "eris", "itur", "imur", "iminī", "untur"],
|
|
||||||
"3io": ["ior", "eris", "itur", "imur", "iminī", "iuntur"],
|
|
||||||
4: ["ior", "īris", "ītur", "īmur", "īminī", "iuntur"],
|
|
||||||
},
|
|
||||||
("imperfect", "ind", "active"): {
|
|
||||||
1: ["ābam", "ābās", "ābat", "ābāmus", "ābātis", "ābant"],
|
|
||||||
2: ["ēbam", "ēbās", "ēbat", "ēbāmus", "ēbātis", "ēbant"],
|
|
||||||
3: ["ēbam", "ēbās", "ēbat", "ēbāmus", "ēbātis", "ēbant"],
|
|
||||||
"3io": ["iēbam", "iēbās", "iēbat", "iēbāmus", "iēbātis", "iēbant"],
|
|
||||||
4: ["iēbam", "iēbās", "iēbat", "iēbāmus", "iēbātis", "iēbant"],
|
|
||||||
},
|
|
||||||
("imperfect", "ind", "passive"): {
|
|
||||||
1: ["ābar", "ābāris", "ābātur", "ābāmur", "ābāminī", "ābantur"],
|
|
||||||
2: ["ēbar", "ēbāris", "ēbātur", "ēbāmur", "ēbāminī", "ēbantur"],
|
|
||||||
3: ["ēbar", "ēbāris", "ēbātur", "ēbāmur", "ēbāminī", "ēbantur"],
|
|
||||||
"3io": ["iēbar", "iēbāris", "iēbātur", "iēbāmur", "iēbāminī", "iēbantur"],
|
|
||||||
4: ["iēbar", "iēbāris", "iēbātur", "iēbāmur", "iēbāminī", "iēbantur"],
|
|
||||||
},
|
|
||||||
("future", "ind", "active"): {
|
|
||||||
1: ["ābō", "ābis", "ābit", "ābimus", "ābitis", "ābunt"],
|
|
||||||
2: ["ēbō", "ēbis", "ēbit", "ēbimus", "ēbitis", "ēbunt"],
|
|
||||||
3: ["am", "ēs", "et", "ēmus", "ētis", "ent"],
|
|
||||||
"3io": ["iam", "iēs", "iet", "iēmus", "iētis", "ient"],
|
|
||||||
4: ["iam", "iēs", "iet", "iēmus", "iētis", "ient"],
|
|
||||||
},
|
|
||||||
("future", "ind", "passive"): {
|
|
||||||
1: ["ābor", "āberis", "ābitur", "ābimur", "ābiminī", "ābuntur"],
|
|
||||||
2: ["ēbor", "ēberis", "ēbitur", "ēbimur", "ēbiminī", "ēbuntur"],
|
|
||||||
3: ["ar", "ēris", "ētur", "ēmur", "ēminī", "entur"],
|
|
||||||
"3io": ["iar", "iēris", "iētur", "iēmur", "iēminī", "ientur"],
|
|
||||||
4: ["iar", "iēris", "iētur", "iēmur", "iēminī", "ientur"],
|
|
||||||
},
|
|
||||||
("present", "sbjv", "active"): {
|
|
||||||
1: ["em", "ēs", "et", "ēmus", "ētis", "ent"],
|
|
||||||
2: ["eam", "eās", "eat", "eāmus", "eātis", "eant"],
|
|
||||||
3: ["am", "ās", "at", "āmus", "ātis", "ant"],
|
|
||||||
"3io": ["iam", "iās", "iat", "iāmus", "iātis", "iant"],
|
|
||||||
4: ["iam", "iās", "iat", "iāmus", "iātis", "iant"],
|
|
||||||
},
|
|
||||||
("present", "sbjv", "passive"): {
|
|
||||||
1: ["er", "ēris", "ētur", "ēmur", "ēminī", "entur"],
|
|
||||||
2: ["ear", "eāris", "eātur", "eāmur", "eāminī", "eantur"],
|
|
||||||
3: ["ar", "āris", "ātur", "āmur", "āminī", "antur"],
|
|
||||||
"3io": ["iar", "iāris", "iātur", "iāmur", "iāminī", "iantur"],
|
|
||||||
4: ["iar", "iāris", "iātur", "iāmur", "iāminī", "iantur"],
|
|
||||||
},
|
|
||||||
("imperfect", "sbjv", "active"): {
|
|
||||||
1: ["ārem", "ārēs", "āret", "ārēmus", "ārētis", "ārent"],
|
|
||||||
2: ["ērem", "ērēs", "ēret", "ērēmus", "ērētis", "ērent"],
|
|
||||||
3: ["erem", "erēs", "eret", "erēmus", "erētis", "erent"],
|
|
||||||
"3io": ["erem", "erēs", "eret", "erēmus", "erētis", "erent"],
|
|
||||||
4: ["īrem", "īrēs", "īret", "īrēmus", "īrētis", "īrent"],
|
|
||||||
},
|
|
||||||
("imperfect", "sbjv", "passive"): {
|
|
||||||
1: ["ārer", "ārēris", "ārētur", "ārēmur", "ārēminī", "ārentur"],
|
|
||||||
2: ["ērer", "ērēris", "ērētur", "ērēmur", "ērēminī", "ērentur"],
|
|
||||||
3: ["erer", "erēris", "erētur", "erēmur", "erēminī", "erentur"],
|
|
||||||
"3io": ["erer", "erēris", "erētur", "erēmur", "erēminī", "erentur"],
|
|
||||||
4: ["īrer", "īrēris", "īrētur", "īrēmur", "īrēminī", "īrentur"],
|
|
||||||
},
|
|
||||||
}
|
|
||||||
# perfect-active endings (added to perfect stem) — same for all conjugations
|
|
||||||
_PERF_ACT = {
|
|
||||||
("perfect", "ind"): ["ī", "istī", "it", "imus", "istis", "ērunt"],
|
|
||||||
("pluperfect", "ind"): ["eram", "erās", "erat", "erāmus", "erātis", "erant"],
|
|
||||||
("futureperfect", "ind"): ["erō", "eris", "erit", "erimus", "eritis", "erint"],
|
|
||||||
("perfect", "sbjv"): ["erim", "erīs", "erit", "erīmus", "erītis", "erint"],
|
|
||||||
("pluperfect", "sbjv"):["issem", "issēs", "isset", "issēmus", "issētis", "issent"],
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _idx(person, number):
|
|
||||||
base = {"first": 0, "second": 1, "third": 2}[person]
|
|
||||||
return base + (0 if number == "singular" else 3)
|
|
||||||
|
|
||||||
|
|
||||||
def _present_system(conj, pstem, tense, mood, voice, person, number):
|
|
||||||
"""Generate a present-system form (present/imperfect/future ind & subj)."""
|
|
||||||
table = _PARADIGM.get((tense, mood, voice))
|
|
||||||
if not table or conj not in table:
|
|
||||||
return None
|
|
||||||
return pstem + table[conj][_idx(person, number)]
|
|
||||||
|
|
||||||
|
|
||||||
def _active_infinitive_stem(conj, pstem):
|
|
||||||
return {1: pstem + "ā", 2: pstem + "ē", 3: pstem + "e",
|
|
||||||
"3io": pstem + "e", 4: pstem + "ī"}[conj]
|
|
||||||
|
|
||||||
|
|
||||||
_IRREG = {
|
|
||||||
"sum": {
|
|
||||||
("present", "ind", "active"): ["sum", "es", "est", "sumus", "estis", "sunt"],
|
|
||||||
("imperfect", "ind", "active"): ["eram", "erās", "erat", "erāmus", "erātis", "erant"],
|
|
||||||
("future", "ind", "active"): ["erō", "eris", "erit", "erimus", "eritis", "erunt"],
|
|
||||||
("perfect", "ind", "active"): ["fuī", "fuistī", "fuit", "fuimus", "fuistis", "fuērunt"],
|
|
||||||
("pluperfect", "ind", "active"): ["fueram", "fuerās", "fuerat", "fuerāmus", "fuerātis", "fuerant"],
|
|
||||||
("present", "sbjv", "active"): ["sim", "sīs", "sit", "sīmus", "sītis", "sint"],
|
|
||||||
("imperfect", "sbjv", "active"): ["essem", "essēs", "esset", "essēmus", "essētis", "essent"],
|
|
||||||
},
|
|
||||||
"possum": {
|
|
||||||
("present", "ind", "active"): ["possum", "potes", "potest", "possumus", "potestis", "possunt"],
|
|
||||||
("imperfect", "ind", "active"): ["poteram", "poterās", "poterat", "poterāmus", "poterātis", "poterant"],
|
|
||||||
("future", "ind", "active"): ["poterō", "poteris", "poterit", "poterimus", "poteritis", "poterunt"],
|
|
||||||
("perfect", "ind", "active"): ["potuī", "potuistī", "potuit", "potuimus", "potuistis", "potuērunt"],
|
|
||||||
("present", "sbjv", "active"): ["possim", "possīs", "possit", "possīmus", "possītis", "possint"],
|
|
||||||
},
|
|
||||||
"eō": {
|
|
||||||
("present", "ind", "active"): ["eō", "īs", "it", "īmus", "ītis", "eunt"],
|
|
||||||
("imperfect", "ind", "active"): ["ībam", "ībās", "ībat", "ībāmus", "ībātis", "ībant"],
|
|
||||||
("future", "ind", "active"): ["ībō", "ībis", "ībit", "ībimus", "ībitis", "ībunt"],
|
|
||||||
("perfect", "ind", "active"): ["iī", "īstī", "iit", "iimus", "īstis", "iērunt"],
|
|
||||||
("present", "sbjv", "active"): ["eam", "eās", "eat", "eāmus", "eātis", "eant"],
|
|
||||||
},
|
|
||||||
"volō": {
|
|
||||||
("present", "ind", "active"): ["volō", "vīs", "vult", "volumus", "vultis", "volunt"],
|
|
||||||
("imperfect", "ind", "active"): ["volēbam", "volēbās", "volēbat", "volēbāmus", "volēbātis", "volēbant"],
|
|
||||||
("future", "ind", "active"): ["volam", "volēs", "volet", "volēmus", "volētis", "volent"],
|
|
||||||
("perfect", "ind", "active"): ["voluī", "voluistī", "voluit", "voluimus", "voluistis", "voluērunt"],
|
|
||||||
("present", "sbjv", "active"): ["velim", "velīs", "velit", "velīmus", "velītis", "velint"],
|
|
||||||
},
|
|
||||||
"nōlō": {
|
|
||||||
("present", "ind", "active"): ["nōlō", "nōn vīs", "nōn vult", "nōlumus", "nōn vultis", "nōlunt"],
|
|
||||||
("present", "sbjv", "active"): ["nōlim", "nōlīs", "nōlit", "nōlīmus", "nōlītis", "nōlint"],
|
|
||||||
},
|
|
||||||
"ferō": {
|
|
||||||
("present", "ind", "active"): ["ferō", "fers", "fert", "ferimus", "fertis", "ferunt"],
|
|
||||||
("imperfect", "ind", "active"): ["ferēbam", "ferēbās", "ferēbat", "ferēbāmus", "ferēbātis", "ferēbant"],
|
|
||||||
("future", "ind", "active"): ["feram", "ferēs", "feret", "ferēmus", "ferētis", "ferent"],
|
|
||||||
("perfect", "ind", "active"): ["tulī", "tulistī", "tulit", "tulimus", "tulistis", "tulērunt"],
|
|
||||||
("present", "sbjv", "active"): ["feram", "ferās", "ferat", "ferāmus", "ferātis", "ferant"],
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def conjugate(lemma, tense, mood, voice="active", person="third", number="singular"):
|
|
||||||
"""Return (surface, confidence). Perfect-passive forms are periphrastic and
|
|
||||||
handled in the realizer (sum + PPP); this returns synthetic forms only."""
|
|
||||||
lemma = lemma.strip()
|
|
||||||
i = _idx(person, number)
|
|
||||||
ir = _IRREG.get(lemma)
|
|
||||||
if ir:
|
|
||||||
tbl = ir.get((tense, mood, voice)) or ir.get((tense, mood, "active"))
|
|
||||||
if tbl and tbl[i]:
|
|
||||||
return tbl[i], "rule"
|
|
||||||
v = _VERBS.get(lemma)
|
|
||||||
if not v:
|
|
||||||
v = _infer_principal_parts(lemma)
|
|
||||||
if not v:
|
|
||||||
return lemma, "fallback"
|
|
||||||
conj, pstem, perfstem, supstem = v
|
|
||||||
# imperative (present active) 2sg / 2pl
|
|
||||||
if mood == "imp":
|
|
||||||
return _imperative(conj, pstem, person, number), "rule"
|
|
||||||
# perfect-system active
|
|
||||||
if tense in ("perfect", "pluperfect", "futureperfect") and voice == "active":
|
|
||||||
if not perfstem:
|
|
||||||
return lemma, "fallback"
|
|
||||||
end = _PERF_ACT.get((tense, mood))
|
|
||||||
if end:
|
|
||||||
return perfstem + end[i], "rule"
|
|
||||||
# present-system (active + passive)
|
|
||||||
if tense in ("present", "imperfect", "future"):
|
|
||||||
form = _present_system(conj, pstem, tense, mood, voice, person, number)
|
|
||||||
if form:
|
|
||||||
return form, "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
def _imperative(conj, pstem, person, number):
|
|
||||||
if number == "singular":
|
|
||||||
return {1: pstem + "ā", 2: pstem + "ē", 3: pstem + "e",
|
|
||||||
"3io": pstem + "e", 4: pstem + "ī"}[conj]
|
|
||||||
return {1: pstem + "āte", 2: pstem + "ēte", 3: pstem + "ite",
|
|
||||||
"3io": pstem + "ite", 4: pstem + "īte"}[conj]
|
|
||||||
|
|
||||||
|
|
||||||
def _infer_principal_parts(lemma):
|
|
||||||
"""OOV fallback: infer conjugation + stems from the 1sg-present citation form.
|
|
||||||
Perfect/supine stems are guessed regularly (often wrong for 3rd conj) and the
|
|
||||||
resulting forms are still returned as 'rule' but the realizer down-weights."""
|
|
||||||
if lemma.endswith("ō"):
|
|
||||||
base = lemma[:-1]
|
|
||||||
# can't distinguish conj from 1sg alone reliably; default by ending vowel
|
|
||||||
if base.endswith("i"):
|
|
||||||
return ("3io", base[:-1], base[:-1] + "īv", base[:-1] + "īt")
|
|
||||||
return (3, base, base + "s", base + "t")
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: participles ─────────────────────────────────────────────────────────
|
|
||||||
def participle(lemma, kind, case="nom", gender="m", number="singular"):
|
|
||||||
"""kind: 'prs' (present active, -ns/-ntis), 'pfv' (perfect passive, -tus),
|
|
||||||
'fut' (future active, -tūrus). Declined as an adjective via rule endings.
|
|
||||||
Returns (form, conf)."""
|
|
||||||
v = _VERBS.get(lemma)
|
|
||||||
if not v:
|
|
||||||
return lemma, "fallback"
|
|
||||||
conj, pstem, perfstem, supstem = v
|
|
||||||
if kind == "pfv":
|
|
||||||
if not supstem:
|
|
||||||
return lemma, "fallback"
|
|
||||||
base = supstem[:-1] if supstem.endswith("t") or supstem.endswith("s") else supstem
|
|
||||||
stem = supstem # supine stem already ends in t/s: amāt- -> amātus
|
|
||||||
return _decline_us_a_um(stem, case, gender, number), "rule"
|
|
||||||
if kind == "fut":
|
|
||||||
if not supstem:
|
|
||||||
return lemma, "fallback"
|
|
||||||
return _decline_us_a_um(supstem + "ūr", case, gender, number), "rule"
|
|
||||||
if kind == "prs":
|
|
||||||
# present active participle: stem + ns (nom), stem + nt- (oblique), 3rd-decl
|
|
||||||
pv = {1: "ā", 2: "ē", 3: "ē", "3io": "iē", 4: "iē"}[conj]
|
|
||||||
ntstem = pstem + pv + "nt"
|
|
||||||
return _decline_pres_ptcp(pstem + pv, case, gender, number), "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
def _decline_us_a_um(stem, case, gender, number):
|
|
||||||
"""Decline a -us/-a/-um adjective/participle stem (2-1-2 declension)."""
|
|
||||||
C = _CASE_MAP.get(case, case.upper())
|
|
||||||
end = {
|
|
||||||
("NOM", "m", "singular"): "us", ("NOM", "f", "singular"): "a", ("NOM", "n", "singular"): "um",
|
|
||||||
("GEN", "m", "singular"): "ī", ("GEN", "f", "singular"): "ae", ("GEN", "n", "singular"): "ī",
|
|
||||||
("DAT", "m", "singular"): "ō", ("DAT", "f", "singular"): "ae", ("DAT", "n", "singular"): "ō",
|
|
||||||
("ACC", "m", "singular"): "um", ("ACC", "f", "singular"): "am", ("ACC", "n", "singular"): "um",
|
|
||||||
("ABL", "m", "singular"): "ō", ("ABL", "f", "singular"): "ā", ("ABL", "n", "singular"): "ō",
|
|
||||||
("VOC", "m", "singular"): "e", ("VOC", "f", "singular"): "a", ("VOC", "n", "singular"): "um",
|
|
||||||
("NOM", "m", "plural"): "ī", ("NOM", "f", "plural"): "ae", ("NOM", "n", "plural"): "a",
|
|
||||||
("GEN", "m", "plural"): "ōrum", ("GEN", "f", "plural"): "ārum", ("GEN", "n", "plural"): "ōrum",
|
|
||||||
("DAT", "m", "plural"): "īs", ("DAT", "f", "plural"): "īs", ("DAT", "n", "plural"): "īs",
|
|
||||||
("ACC", "m", "plural"): "ōs", ("ACC", "f", "plural"): "ās", ("ACC", "n", "plural"): "a",
|
|
||||||
("ABL", "m", "plural"): "īs", ("ABL", "f", "plural"): "īs", ("ABL", "n", "plural"): "īs",
|
|
||||||
("VOC", "m", "plural"): "ī", ("VOC", "f", "plural"): "ae", ("VOC", "n", "plural"): "a",
|
|
||||||
}.get((C, gender, number), "us")
|
|
||||||
return stem + end
|
|
||||||
|
|
||||||
|
|
||||||
def _decline_pres_ptcp(stem, case, gender, number):
|
|
||||||
"""Present active participle (amāns, amantis) — 3rd-declension, stem+ns/nt."""
|
|
||||||
C = _CASE_MAP.get(case, case.upper())
|
|
||||||
if C == "NOM" and number == "singular":
|
|
||||||
return stem + "ns"
|
|
||||||
if C == "VOC" and number == "singular":
|
|
||||||
return stem + "ns"
|
|
||||||
base = stem + "nt"
|
|
||||||
end = {
|
|
||||||
("GEN", "singular"): "is", ("DAT", "singular"): "ī",
|
|
||||||
("ACC", "singular"): "em" if gender != "n" else "",
|
|
||||||
("ABL", "singular"): "e",
|
|
||||||
("NOM", "plural"): "ēs" if gender != "n" else "ia",
|
|
||||||
("GEN", "plural"): "ium", ("DAT", "plural"): "ibus",
|
|
||||||
("ACC", "plural"): "ēs" if gender != "n" else "ia",
|
|
||||||
("ABL", "plural"): "ibus", ("VOC", "plural"): "ēs",
|
|
||||||
}.get((C, number), "is")
|
|
||||||
if C == "ACC" and number == "singular" and gender == "n":
|
|
||||||
return stem + "ns"
|
|
||||||
return base + end
|
|
||||||
|
|
||||||
|
|
||||||
def infinitive(lemma, tense="present", voice="active"):
|
|
||||||
lemma = lemma.strip()
|
|
||||||
if lemma == "sum":
|
|
||||||
return ("esse", "rule") if tense == "present" else ("fuisse", "rule")
|
|
||||||
v = _VERBS.get(lemma)
|
|
||||||
if not v:
|
|
||||||
return lemma, "fallback"
|
|
||||||
conj, pstem, perfstem, supstem = v
|
|
||||||
if tense == "present":
|
|
||||||
if voice == "active":
|
|
||||||
return _active_infinitive_stem(conj, pstem).rstrip() + \
|
|
||||||
("re" if conj != 3 and conj != "3io" else "re"), "rule"
|
|
||||||
# passive present infinitive
|
|
||||||
base = {1: pstem + "ā", 2: pstem + "ē", 4: pstem + "ī"}.get(conj)
|
|
||||||
if base:
|
|
||||||
return base + "rī", "rule"
|
|
||||||
return pstem + "ī", "rule" # 3rd: regī
|
|
||||||
if tense == "perfect" and voice == "active" and perfstem:
|
|
||||||
return perfstem + "isse", "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
def lexicon_stats():
|
|
||||||
return {
|
|
||||||
"noun_adj_source": "UniMorph Latin (github.com/unimorph/lat, CC-BY-SA 3.0)",
|
|
||||||
"verb_source": "rule-based 4-conjugation engine over curated attested "
|
|
||||||
"principal parts (UniMorph verb list is a 947-lemma sample "
|
|
||||||
"MISSING all core verbs — amō/sum/videō absent)",
|
|
||||||
"noun_lemmas": len(_NOUNS),
|
|
||||||
"adj_lemmas": len(_ADJS),
|
|
||||||
"curated_verb_lemmas": len(_VERBS) + len(_IRREG),
|
|
||||||
"gender_inference": "declension-based (nom+gen endings) + curated exceptions",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
import json
|
|
||||||
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
|
|
||||||
print("\n-- noun declension puella (1st, fem) --")
|
|
||||||
for c in ("nom", "gen", "dat", "acc", "abl", "voc"):
|
|
||||||
print(f" {c}: sg={decline_noun('puella', c, 'singular')[0]:10} "
|
|
||||||
f"pl={decline_noun('puella', c, 'plural')[0]}")
|
|
||||||
print("\n-- rēx (3rd, m):", [decline_noun('rēx', c, 'singular')[0] for c in ('nom','gen','dat','acc','abl')])
|
|
||||||
print("-- gender: puella=", noun_gender("puella"), "rēx=", noun_gender("rēx"),
|
|
||||||
"bellum=", noun_gender("bellum"), "corpus=", noun_gender("corpus"),
|
|
||||||
"manus=", noun_gender("manus"), "diēs=", noun_gender("diēs"))
|
|
||||||
print("\n-- conjugate videō (2nd) present ind active --")
|
|
||||||
for p in ("first", "second", "third"):
|
|
||||||
for n in ("singular", "plural"):
|
|
||||||
print(f" {p[:3]}.{n[:2]}: {conjugate('videō','present','ind','active',p,n)[0]}")
|
|
||||||
print("-- amō forms:", conjugate("amō","present","ind","active","first","singular")[0],
|
|
||||||
conjugate("amō","imperfect","ind","active","third","plural")[0],
|
|
||||||
conjugate("amō","future","ind","active","first","singular")[0],
|
|
||||||
conjugate("amō","perfect","ind","active","third","singular")[0])
|
|
||||||
print("-- sum:", [conjugate("sum","present","ind","active",p,"singular")[0] for p in ("first","second","third")])
|
|
||||||
print("-- participle amō pfv acc.f.sg:", participle("amō","pfv","acc","f","singular")[0])
|
|
||||||
print("-- infinitive amō:", infinitive("amō")[0], "| regō pass:", infinitive("regō", voice="passive")[0])
|
|
||||||
@@ -1,538 +0,0 @@
|
|||||||
"""morphology_pt_full.py — production-grade Brazilian-Portuguese morphological generator.
|
|
||||||
|
|
||||||
NOT a toy. Backed by two real, broad, Wiktionary-lineage lexicons:
|
|
||||||
|
|
||||||
VERBS — UniMorph Portuguese (github.com/unimorph/por, CC-BY-SA 3.0)
|
|
||||||
4,001 verb lemmas × full paradigm (283,991 finite/non-finite forms +
|
|
||||||
20,005 participle forms). Every mood/tense pt actually inflects:
|
|
||||||
indicative present / preterite (PST;PFV) / imperfect (PST;IPFV) /
|
|
||||||
pluperfect-simple (PST;PRF) / future,
|
|
||||||
conditional (futuro do pretérito),
|
|
||||||
subjunctive present / imperfect / FUTURE (PT-specific live tense),
|
|
||||||
affirmative + negative imperative,
|
|
||||||
PERSONAL infinitive (V;{p};{n};NFIN — a PT-specific finite-ish form),
|
|
||||||
past participle (4 gender/number forms) + gerúndio (V.PTCP;PRS).
|
|
||||||
|
|
||||||
NOUNS + ADJECTIVES — kaikki.org Portuguese (Wiktionary extract, same lineage)
|
|
||||||
81,138 noun lemmas WITH inherent gender + real (often irregular) plural —
|
|
||||||
so -ão→-ões / -ãos / -ães / -õos is resolved PER LEMMA by Wiktionary,
|
|
||||||
never guessed (mão→mãos, pão→pães, coração→corações).
|
|
||||||
40,252 adjective lemmas with real feminine + masc/fem plural forms.
|
|
||||||
|
|
||||||
Fallbacks (degrade, never crash, on out-of-vocabulary input):
|
|
||||||
verbs : rule generator for regular -ar/-er/-ir paradigms
|
|
||||||
nouns : gender heuristic (endings) + rule pluralization (with -ão FLAGGED)
|
|
||||||
adjs : -o/-a gender rule + rule pluralization
|
|
||||||
|
|
||||||
Confidence flag on every form:
|
|
||||||
"lexicon" straight from UniMorph/kaikki (trust: high)
|
|
||||||
"rule" deterministic rule (trust: medium)
|
|
||||||
"fallback" could not inflect; returned lemma (trust: low -> FLAG)
|
|
||||||
|
|
||||||
Public API (used by realizer_pt.py):
|
|
||||||
conjugate(lemma, mood, tense, person, number) -> (form, conf)
|
|
||||||
personal_infinitive(lemma, person, number) -> (form, conf)
|
|
||||||
participle(lemma, gender="m", number="singular") -> (form, conf)
|
|
||||||
gerund(lemma) -> (form, conf)
|
|
||||||
noun_gender(lemma) -> "m"|"f"
|
|
||||||
inflect_noun(lemma, number, gender=None) -> (form, conf)
|
|
||||||
inflect_adj(lemma, gender, number) -> (form, conf)
|
|
||||||
lexicon_stats() -> dict
|
|
||||||
"""
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import pickle
|
|
||||||
|
|
||||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
_UNIMORPH = os.path.join(_HERE, "data", "por.unimorph")
|
|
||||||
_KAIKKI = os.path.join(_HERE, "data", "kaikki_pt.jsonl")
|
|
||||||
_CACHE = os.path.join(_HERE, "data", "pt_morph_cache.pkl")
|
|
||||||
|
|
||||||
# ── mood/tense pair -> UniMorph feature triple (a in tag; b in tag; c in tag) ────
|
|
||||||
_VERB_KEYMAP = {
|
|
||||||
("ind", "present"): ("IND", "PRS", None),
|
|
||||||
("ind", "preterite"): ("IND", "PST", "PFV"),
|
|
||||||
("ind", "imperfect"): ("IND", "PST", "IPFV"),
|
|
||||||
("ind", "pluperfect"): ("IND", "PST", "PRF"), # simple mais-que-perfeito
|
|
||||||
("ind", "future"): ("IND", "FUT", None),
|
|
||||||
("ind", "conditional"): ("COND", None, None),
|
|
||||||
("sbjv", "present"): ("SBJV", "PRS", None),
|
|
||||||
("sbjv", "imperfect"): ("SBJV", "PST", "IPFV"),
|
|
||||||
("sbjv", "future"): ("SBJV", "FUT", None), # PT-specific
|
|
||||||
("imp", "affirmative"): ("IMP", "POS", None),
|
|
||||||
("imp", "negative"): ("IMP", "NEG", None),
|
|
||||||
}
|
|
||||||
_PERSON = {"first": "1", "second": "2", "third": "3"}
|
|
||||||
_NUMBER = {"singular": "SG", "plural": "PL"}
|
|
||||||
|
|
||||||
|
|
||||||
def _feat_set(tag):
|
|
||||||
return set(tag.split(";"))
|
|
||||||
|
|
||||||
|
|
||||||
# ── build the compact lexicon from UniMorph (verbs) + kaikki (nouns/adjs) ────────
|
|
||||||
def _build_verbs():
|
|
||||||
verbs = {} # (lemma, "mood|tense|person|number") -> form
|
|
||||||
pinf = {} # (lemma, "person|number") -> personal-infinitive form
|
|
||||||
part = {} # lemma -> {("m","SG"): form, ...} past participle
|
|
||||||
ger = {} # lemma -> gerúndio
|
|
||||||
with open(_UNIMORPH, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
line = line.rstrip("\n")
|
|
||||||
if not line or "\t" not in line:
|
|
||||||
continue
|
|
||||||
parts = line.split("\t")
|
|
||||||
if len(parts) != 3:
|
|
||||||
continue
|
|
||||||
lemma, form, tag = parts
|
|
||||||
f = _feat_set(tag)
|
|
||||||
head = tag.split(";")[0]
|
|
||||||
|
|
||||||
if head == "V.PTCP":
|
|
||||||
if "PST" in f: # past participle: falado/falada/falados/faladas
|
|
||||||
g = "m" if "MASC" in f else ("f" if "FEM" in f else "m")
|
|
||||||
num = "SG" if "SG" in f else ("PL" if "PL" in f else "SG")
|
|
||||||
part.setdefault(lemma, {})[(g, num)] = form
|
|
||||||
elif "PRS" in f: # gerúndio: falando
|
|
||||||
ger.setdefault(lemma, form)
|
|
||||||
continue
|
|
||||||
|
|
||||||
if head != "V":
|
|
||||||
continue
|
|
||||||
|
|
||||||
# personal / impersonal infinitive
|
|
||||||
if "NFIN" in f:
|
|
||||||
person = next((p for p in ("1", "2", "3") if p in f), None)
|
|
||||||
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
|
|
||||||
if person and number:
|
|
||||||
pinf[(lemma, f"{person}|{number}")] = form
|
|
||||||
continue
|
|
||||||
|
|
||||||
# finite forms
|
|
||||||
mt = None
|
|
||||||
for (mood, tense), (a, b, c) in _VERB_KEYMAP.items():
|
|
||||||
if a not in f:
|
|
||||||
continue
|
|
||||||
if b is not None and b not in f:
|
|
||||||
continue
|
|
||||||
if c is not None and c not in f:
|
|
||||||
continue
|
|
||||||
# IND;PST needs exactly PFV|IPFV|PRF — reject if the required one absent
|
|
||||||
mt = (mood, tense)
|
|
||||||
break
|
|
||||||
if mt is None:
|
|
||||||
continue
|
|
||||||
person = next((p for p in ("1", "2", "3") if p in f), None)
|
|
||||||
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
|
|
||||||
if person is None or number is None:
|
|
||||||
continue
|
|
||||||
verbs.setdefault((lemma, f"{mt[0]}|{mt[1]}|{person}|{number}"), form)
|
|
||||||
return verbs, pinf, part, ger
|
|
||||||
|
|
||||||
|
|
||||||
def _kaikki_gender(arg):
|
|
||||||
if not arg:
|
|
||||||
return None
|
|
||||||
a = arg.lower()
|
|
||||||
if a.startswith("f"):
|
|
||||||
return "f"
|
|
||||||
if a.startswith("m"):
|
|
||||||
return "m"
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _build_nouns_adjs():
|
|
||||||
nouns = {} # lemma -> {"g","SG","PL"}
|
|
||||||
adjs = {} # lemma -> {("m","SG"),("f","SG"),("m","PL"),("f","PL")}
|
|
||||||
with open(_KAIKKI, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
try:
|
|
||||||
d = json.loads(line)
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
pos = d.get("pos")
|
|
||||||
word = d.get("word", "")
|
|
||||||
if not word or " " in word: # skip multiword entries
|
|
||||||
continue
|
|
||||||
forms = d.get("forms", []) or []
|
|
||||||
|
|
||||||
if pos == "noun":
|
|
||||||
ht = d.get("head_templates") or []
|
|
||||||
g = None
|
|
||||||
if ht:
|
|
||||||
g = _kaikki_gender((ht[0].get("args") or {}).get("1"))
|
|
||||||
if g is None:
|
|
||||||
tags = d.get("tags") or []
|
|
||||||
if "feminine" in tags:
|
|
||||||
g = "f"
|
|
||||||
elif "masculine" in tags:
|
|
||||||
g = "m"
|
|
||||||
pl = None
|
|
||||||
for x in forms:
|
|
||||||
t = x.get("tags") or []
|
|
||||||
if "plural" in t and "alternative" not in t and "obsolete" not in t:
|
|
||||||
pl = x.get("form")
|
|
||||||
break
|
|
||||||
# first entry wins; but a later entry with a plural fills a gap
|
|
||||||
if word not in nouns:
|
|
||||||
nouns[word] = {"g": g, "SG": word, "PL": pl}
|
|
||||||
else:
|
|
||||||
cur = nouns[word]
|
|
||||||
if cur.get("g") is None and g:
|
|
||||||
cur["g"] = g
|
|
||||||
if not cur.get("PL") and pl:
|
|
||||||
cur["PL"] = pl
|
|
||||||
|
|
||||||
elif pos == "adj":
|
|
||||||
d0 = adjs.setdefault(word, {})
|
|
||||||
d0.setdefault(("m", "SG"), word)
|
|
||||||
for x in forms:
|
|
||||||
t = set(x.get("tags") or [])
|
|
||||||
fm = x.get("form")
|
|
||||||
if not fm or ("alternative" in t) or ("obsolete" in t):
|
|
||||||
continue
|
|
||||||
if "comparative" in t or "superlative" in t or \
|
|
||||||
"diminutive" in t or "augmentative" in t:
|
|
||||||
continue
|
|
||||||
if "feminine" in t and "plural" in t:
|
|
||||||
d0[("f", "PL")] = fm
|
|
||||||
elif "masculine" in t and "plural" in t:
|
|
||||||
d0[("m", "PL")] = fm
|
|
||||||
elif "feminine" in t:
|
|
||||||
d0[("f", "SG")] = fm
|
|
||||||
elif "plural" in t: # invariant-gender adj (feliz -> felizes)
|
|
||||||
d0[("m", "PL")] = d0.get(("m", "PL")) or fm
|
|
||||||
d0[("f", "PL")] = d0.get(("f", "PL")) or fm
|
|
||||||
return nouns, adjs
|
|
||||||
|
|
||||||
|
|
||||||
def _build_cache():
|
|
||||||
verbs, pinf, part, ger = _build_verbs()
|
|
||||||
nouns, adjs = _build_nouns_adjs()
|
|
||||||
data = {"verbs": verbs, "pinf": pinf, "part": part, "ger": ger,
|
|
||||||
"nouns": nouns, "adjs": adjs}
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "wb") as fh:
|
|
||||||
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
def _load():
|
|
||||||
if os.path.exists(_CACHE):
|
|
||||||
newest_src = max(os.path.getmtime(_UNIMORPH),
|
|
||||||
os.path.getmtime(_KAIKKI) if os.path.exists(_KAIKKI) else 0)
|
|
||||||
if os.path.getmtime(_CACHE) >= newest_src:
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "rb") as fh:
|
|
||||||
return pickle.load(fh)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return _build_cache()
|
|
||||||
|
|
||||||
|
|
||||||
_LEX = _load()
|
|
||||||
_VERBS, _PINF, _PART, _GER, _NOUNS, _ADJS = (
|
|
||||||
_LEX["verbs"], _LEX["pinf"], _LEX["part"], _LEX["ger"],
|
|
||||||
_LEX["nouns"], _LEX["adjs"])
|
|
||||||
|
|
||||||
|
|
||||||
# ── regular-ending rule fallback (deterministic, last resort) ────────────────────
|
|
||||||
def _vclass(lemma):
|
|
||||||
return lemma[-2:] if lemma[-2:] in ("ar", "er", "ir") else None
|
|
||||||
|
|
||||||
|
|
||||||
def _stem(lemma):
|
|
||||||
return lemma[:-2]
|
|
||||||
|
|
||||||
|
|
||||||
# endings indexed [1sg,2sg,3sg,1pl,2pl,3pl]
|
|
||||||
_REG = {
|
|
||||||
("ind", "present", "ar"): ["o", "as", "a", "amos", "ais", "am"],
|
|
||||||
("ind", "present", "er"): ["o", "es", "e", "emos", "eis", "em"],
|
|
||||||
("ind", "present", "ir"): ["o", "es", "e", "imos", "is", "em"],
|
|
||||||
("ind", "preterite", "ar"): ["ei", "aste", "ou", "amos", "astes", "aram"],
|
|
||||||
("ind", "preterite", "er"): ["i", "este", "eu", "emos", "estes", "eram"],
|
|
||||||
("ind", "preterite", "ir"): ["i", "iste", "iu", "imos", "istes", "iram"],
|
|
||||||
("ind", "imperfect", "ar"): ["ava", "avas", "ava", "ávamos", "áveis", "avam"],
|
|
||||||
("ind", "imperfect", "er"): ["ia", "ias", "ia", "íamos", "íeis", "iam"],
|
|
||||||
("ind", "imperfect", "ir"): ["ia", "ias", "ia", "íamos", "íeis", "iam"],
|
|
||||||
("sbjv", "present", "ar"): ["e", "es", "e", "emos", "eis", "em"],
|
|
||||||
("sbjv", "present", "er"): ["a", "as", "a", "amos", "ais", "am"],
|
|
||||||
("sbjv", "present", "ir"): ["a", "as", "a", "amos", "ais", "am"],
|
|
||||||
("sbjv", "imperfect", "ar"): ["asse", "asses", "asse", "ássemos", "ásseis", "assem"],
|
|
||||||
("sbjv", "imperfect", "er"): ["esse", "esses", "esse", "êssemos", "êsseis", "essem"],
|
|
||||||
("sbjv", "imperfect", "ir"): ["isse", "isses", "isse", "íssemos", "ísseis", "issem"],
|
|
||||||
("sbjv", "future", "ar"): ["ar", "ares", "ar", "armos", "ardes", "arem"],
|
|
||||||
("sbjv", "future", "er"): ["er", "eres", "er", "ermos", "erdes", "erem"],
|
|
||||||
("sbjv", "future", "ir"): ["ir", "ires", "ir", "irmos", "irdes", "irem"],
|
|
||||||
}
|
|
||||||
# future & conditional attach to the FULL infinitive
|
|
||||||
_FUT = ["ei", "ás", "á", "emos", "eis", "ão"]
|
|
||||||
_COND = ["ia", "ias", "ia", "íamos", "íeis", "iam"]
|
|
||||||
|
|
||||||
|
|
||||||
def _slot_idx(person, number):
|
|
||||||
base = {"first": 0, "second": 1, "third": 2}[person]
|
|
||||||
return base + (0 if number == "singular" else 3)
|
|
||||||
|
|
||||||
|
|
||||||
def _rule_conjugate(lemma, mood, tense, person, number):
|
|
||||||
vc = _vclass(lemma)
|
|
||||||
if vc is None:
|
|
||||||
return None
|
|
||||||
st, i = _stem(lemma), _slot_idx(person, number)
|
|
||||||
if mood == "ind" and tense == "future":
|
|
||||||
return lemma + _FUT[i]
|
|
||||||
if mood == "ind" and tense == "conditional":
|
|
||||||
return lemma + _COND[i]
|
|
||||||
if mood == "imp": # affirmative tú/vocês imperative ~ subjunctive present
|
|
||||||
table = _REG.get(("sbjv", "present", vc))
|
|
||||||
if table and tense == "negative":
|
|
||||||
return st + table[i]
|
|
||||||
# affirmative 2sg = 3sg present indicative; others = subjunctive
|
|
||||||
pres = _REG.get(("ind", "present", vc))
|
|
||||||
if person == "second" and number == "singular":
|
|
||||||
return st + pres[2]
|
|
||||||
return st + table[i] if table else None
|
|
||||||
table = _REG.get((mood, tense, vc))
|
|
||||||
if table:
|
|
||||||
return st + table[i]
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
# verified corrections to UniMorph data errors (each audited individually, not
|
|
||||||
# guessed). The three 1PL-present entries are glued-allomorph errors surfaced by a
|
|
||||||
# full-lexicon scan for a non-final "mos" in V;1;PL;IND;PRS forms (the ONLY three).
|
|
||||||
_VERB_FIX = {
|
|
||||||
("estar", "ind", "imperfect", "third", "plural"): "estavam", # was "estávam"
|
|
||||||
("estar", "ind", "present", "first", "plural"): "estamos", # was "estamosestámos"
|
|
||||||
("haver", "ind", "present", "first", "plural"): "havemos", # was "havemoshemos"
|
|
||||||
("ir", "ind", "present", "first", "plural"): "vamos", # was "vamosimos"
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: verb conjugation ─────────────────────────────────────────────────────
|
|
||||||
def conjugate(lemma, mood, tense, person, number):
|
|
||||||
"""Return (surface, confidence). mood in ind|sbjv|imp; tense per _VERB_KEYMAP."""
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
fix = _VERB_FIX.get((lemma, mood, tense, person, number))
|
|
||||||
if fix:
|
|
||||||
return fix, "lexicon"
|
|
||||||
p, n = _PERSON.get(person), _NUMBER.get(number)
|
|
||||||
if p and n:
|
|
||||||
form = _VERBS.get((lemma, f"{mood}|{tense}|{p}|{n}"))
|
|
||||||
if form:
|
|
||||||
# pt-BR normalization: UniMorph `por` carries the EUROPEAN spelling of
|
|
||||||
# the -ar 1pl PRETERITE (-ámos). Brazilian PT drops the accent
|
|
||||||
# (falámos->falamos, chegámos->chegamos) — 3,334/4,001 verbs affected.
|
|
||||||
if (mood == "ind" and tense == "preterite" and person == "first"
|
|
||||||
and number == "plural" and form.endswith("ámos")):
|
|
||||||
form = form[:-4] + "amos"
|
|
||||||
return form, "lexicon"
|
|
||||||
r = _rule_conjugate(lemma, mood, tense, person, number)
|
|
||||||
if r:
|
|
||||||
return r, "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
def personal_infinitive(lemma, person, number):
|
|
||||||
"""PT personal (inflected) infinitive: para falarmos, ao chegarem."""
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
p, n = _PERSON.get(person), _NUMBER.get(number)
|
|
||||||
if p and n:
|
|
||||||
form = _PINF.get((lemma, f"{p}|{n}"))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
# rule: infinitive + personal endings (-, -es, -, -mos, -des, -em)
|
|
||||||
end = {("first", "singular"): "", ("second", "singular"): "es",
|
|
||||||
("third", "singular"): "", ("first", "plural"): "mos",
|
|
||||||
("second", "plural"): "des", ("third", "plural"): "em"}.get((person, number), "")
|
|
||||||
return lemma + end, "rule"
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: participle + gerund ───────────────────────────────────────────────────
|
|
||||||
def participle(lemma, gender="m", number="singular"):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
g = "f" if gender == "f" else "m"
|
|
||||||
num = "SG" if number == "singular" else "PL"
|
|
||||||
d = _PART.get(lemma)
|
|
||||||
if d:
|
|
||||||
form = d.get((g, num)) or d.get(("m", "SG"))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
if lemma.endswith("ar"):
|
|
||||||
base = lemma[:-2] + "ad"
|
|
||||||
elif lemma[-2:] in ("er", "ir"):
|
|
||||||
base = lemma[:-2] + "id"
|
|
||||||
else:
|
|
||||||
return lemma, "fallback"
|
|
||||||
suf = {"m|SG": "o", "f|SG": "a", "m|PL": "os", "f|PL": "as"}[f"{g}|{num}"]
|
|
||||||
return base + suf, "rule"
|
|
||||||
|
|
||||||
|
|
||||||
def gerund(lemma):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
if lemma in _GER:
|
|
||||||
return _GER[lemma], "lexicon"
|
|
||||||
if lemma.endswith("ar"):
|
|
||||||
return lemma[:-2] + "ando", "rule"
|
|
||||||
if lemma.endswith("er"):
|
|
||||||
return lemma[:-2] + "endo", "rule"
|
|
||||||
if lemma.endswith("ir"):
|
|
||||||
return lemma[:-2] + "indo", "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: noun gender + number ─────────────────────────────────────────────────
|
|
||||||
_FEM_SUF = ("ção", "são", "ção", "dade", "tade", "agem", "igem", "ugem", "gem",
|
|
||||||
"ez", "eza", "ice", "ície", "tude", "ude", "âncbefore")
|
|
||||||
_FEM_SUF = ("ção", "são", "dade", "tade", "agem", "gem", "eza", "ez", "ice",
|
|
||||||
"tude", "ude", "ância", "ência", "ínia")
|
|
||||||
_MASC_SUF = ("ema", "oma", "ama", "grama", "eta", "ão") # Greek -ma etc. (mostly m)
|
|
||||||
|
|
||||||
|
|
||||||
def _gender_heuristic(noun):
|
|
||||||
for suf in _FEM_SUF:
|
|
||||||
if noun.endswith(suf):
|
|
||||||
return "f"
|
|
||||||
if noun.endswith(("ema", "oma", "ama")): # problema, idioma, programa
|
|
||||||
return "m"
|
|
||||||
if noun.endswith("a") or noun.endswith("ã"):
|
|
||||||
return "f"
|
|
||||||
if noun.endswith("o") or noun.endswith(("l", "r", "z", "m", "u", "i")):
|
|
||||||
return "m"
|
|
||||||
return "m"
|
|
||||||
|
|
||||||
|
|
||||||
def noun_gender(lemma):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
if d and d.get("g"):
|
|
||||||
return d["g"]
|
|
||||||
return _gender_heuristic(lemma)
|
|
||||||
|
|
||||||
|
|
||||||
_INVARIANT_PL_SUF = ("s",) # paroxytones ending -s are invariant (o lápis / os lápis)
|
|
||||||
|
|
||||||
|
|
||||||
def _rule_plural(noun):
|
|
||||||
"""Deterministic PT pluralization. Returns (form, ok) where ok=False flags an
|
|
||||||
ambiguous -ão that should lower confidence (the lexicon normally resolves it)."""
|
|
||||||
if not noun:
|
|
||||||
return noun, True
|
|
||||||
if noun.endswith("ão"):
|
|
||||||
return noun[:-2] + "ões", False # majority rule, but AMBIGUOUS -> flag
|
|
||||||
if noun.endswith("m"):
|
|
||||||
return noun[:-1] + "ns", True # homem->homens, jardim->jardins
|
|
||||||
if noun.endswith("al"):
|
|
||||||
return noun[:-2] + "ais", True
|
|
||||||
if noun.endswith("el"):
|
|
||||||
return noun[:-2] + "éis", True
|
|
||||||
if noun.endswith("ol"):
|
|
||||||
return noun[:-2] + "óis", True
|
|
||||||
if noun.endswith("ul"):
|
|
||||||
return noun[:-2] + "uis", True
|
|
||||||
if noun.endswith("il"):
|
|
||||||
return noun[:-2] + "is", True # stressed (funil->funis); unstressed rarer
|
|
||||||
if noun.endswith(("r", "z")):
|
|
||||||
return noun + "es", True # flor->flores, luz->luzes
|
|
||||||
if noun.endswith("s"):
|
|
||||||
# paroxytone -s (lápis, ônibus) invariant; oxytone -s (país) -> -es
|
|
||||||
return noun, True
|
|
||||||
if noun.endswith(("a", "e", "i", "o", "u", "á", "é", "í", "ó", "ú", "ã")):
|
|
||||||
return noun + "s", True
|
|
||||||
return noun + "s", True
|
|
||||||
|
|
||||||
|
|
||||||
def inflect_noun(lemma, number, gender=None):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
if number == "singular":
|
|
||||||
return (d["SG"] if d and d.get("SG") else lemma), ("lexicon" if d else "rule")
|
|
||||||
if d and d.get("PL"):
|
|
||||||
return d["PL"], "lexicon"
|
|
||||||
form, ok = _rule_plural(lemma)
|
|
||||||
return form, ("rule" if ok else "fallback")
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC: adjective agreement ──────────────────────────────────────────────────
|
|
||||||
def inflect_adj(lemma, gender, number):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
g = "f" if gender == "f" else "m"
|
|
||||||
num = "SG" if number == "singular" else "PL"
|
|
||||||
d = _ADJS.get(lemma)
|
|
||||||
if d:
|
|
||||||
form = d.get((g, num))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
# build a missing plural from this gender's singular
|
|
||||||
sg = d.get((g, "SG")) or d.get(("m", "SG")) or lemma
|
|
||||||
if num == "PL":
|
|
||||||
pl, ok = _rule_plural(sg)
|
|
||||||
return pl, ("rule" if ok else "fallback")
|
|
||||||
return sg, "lexicon"
|
|
||||||
# rule fallback: -o/-a gender, then pluralize
|
|
||||||
a = lemma
|
|
||||||
if g == "f":
|
|
||||||
if a.endswith("o"):
|
|
||||||
a = a[:-1] + "a"
|
|
||||||
elif a.endswith(("ês", "or")) and not a.endswith("ior"):
|
|
||||||
a = a + "a" # português->portuguesa, trabalhador->..a
|
|
||||||
if num == "PL":
|
|
||||||
a, ok = _rule_plural(a)
|
|
||||||
return a, ("rule" if ok else "fallback")
|
|
||||||
return a, "rule"
|
|
||||||
|
|
||||||
|
|
||||||
def lexicon_stats():
|
|
||||||
return {
|
|
||||||
"verb_source": "UniMorph Portuguese (github.com/unimorph/por)",
|
|
||||||
"noun_adj_source": "kaikki.org Portuguese (Wiktionary extract)",
|
|
||||||
"license": "CC-BY-SA (Wiktionary-derived)",
|
|
||||||
"verb_forms": len(_VERBS),
|
|
||||||
"verb_lemmas": len({k[0] for k in _VERBS}),
|
|
||||||
"personal_infinitive_forms": len(_PINF),
|
|
||||||
"participle_lemmas": len(_PART),
|
|
||||||
"gerund_lemmas": len(_GER),
|
|
||||||
"noun_lemmas": len(_NOUNS),
|
|
||||||
"adj_lemmas": len(_ADJS),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
|
|
||||||
tests = [
|
|
||||||
("falar", "ind", "present", "first", "singular", "falo"),
|
|
||||||
("comer", "ind", "present", "third", "plural", "comem"),
|
|
||||||
("partir", "ind", "present", "first", "plural", "partimos"),
|
|
||||||
("ser", "ind", "present", "third", "singular", "é"),
|
|
||||||
("ir", "ind", "preterite", "first", "singular", "fui"),
|
|
||||||
("ter", "ind", "future", "first", "singular", "terei"),
|
|
||||||
("fazer", "sbjv", "present", "first", "singular", "faça"),
|
|
||||||
("dormir", "ind", "present", "first", "singular", "durmo"),
|
|
||||||
("dar", "ind", "preterite", "third", "singular", "deu"),
|
|
||||||
("poder", "ind", "conditional", "first", "singular", "poderia"),
|
|
||||||
("fazer", "sbjv", "future", "third", "singular", "fizer"),
|
|
||||||
("estar", "ind", "present", "third", "singular", "está"),
|
|
||||||
]
|
|
||||||
ok = 0
|
|
||||||
for lemma, mood, tense, per, num, exp in tests:
|
|
||||||
got, conf = conjugate(lemma, mood, tense, per, num)
|
|
||||||
flag = "OK " if got == exp else "XX "
|
|
||||||
ok += got == exp
|
|
||||||
print(f" {flag}{lemma:8} {mood}/{tense} {per[:3]}.{num[:2]} -> {got:14} ({conf}) exp={exp}")
|
|
||||||
print(f"verb tests {ok}/{len(tests)}")
|
|
||||||
print(" gender: casa=", noun_gender("casa"), "problema=", noun_gender("problema"),
|
|
||||||
"mão=", noun_gender("mão"), "coração=", noun_gender("coração"),
|
|
||||||
"flor=", noun_gender("flor"))
|
|
||||||
print(" plural: mão->", inflect_noun("mão", "plural"),
|
|
||||||
"| pão->", inflect_noun("pão", "plural"),
|
|
||||||
"| animal->", inflect_noun("animal", "plural"),
|
|
||||||
"| coração->", inflect_noun("coração", "plural"))
|
|
||||||
print(" adj: bonito/f/sg->", inflect_adj("bonito", "f", "singular"),
|
|
||||||
"| feliz/m/pl->", inflect_adj("feliz", "m", "plural"),
|
|
||||||
"| português/f/sg->", inflect_adj("português", "f", "singular"))
|
|
||||||
print(" part: fazer/m/sg->", participle("fazer"), "| ger falar->", gerund("falar"))
|
|
||||||
print(" pinf falar 1pl->", personal_infinitive("falar", "first", "plural"))
|
|
||||||
@@ -1,609 +0,0 @@
|
|||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""morphology_ro_full.py — production-grade Romanian morphological generator.
|
|
||||||
|
|
||||||
Romanian is the BIG typological delta of the Romance family. The verb engine and
|
|
||||||
the confidence/fallback contract TRANSFER from the Italian sibling; the NOMINAL
|
|
||||||
system is genuinely new: Romanian has a SUFFIXED definite article, a preserved
|
|
||||||
NOM/ACC vs GEN/DAT case distinction, a NEUTER gender (masc-agreeing in SG,
|
|
||||||
fem-agreeing in PL), and a VOCATIVE. Those are grounded in real per-lemma data,
|
|
||||||
not guessed.
|
|
||||||
|
|
||||||
Real, Wiktionary-lineage lexical sources:
|
|
||||||
|
|
||||||
VERBS — UniMorph Romanian (github.com/unimorph/ron, CC-BY-SA 3.0)
|
|
||||||
~1216 verb lemmas × paradigm, CLEAN orthography:
|
|
||||||
indicativ prezent / imperfect (PST;IPFV) / perfectul simplu (PST;PFV) /
|
|
||||||
conjunctiv prezent (SBJV;PRS, stored WITHOUT the 'să' particle),
|
|
||||||
participiu (V.PTCP;PST, INVARIABLE in the perfect compus),
|
|
||||||
gerunziu (V.CVB;PRS), infinitiv (NFIN), imperativ.
|
|
||||||
ro_irreg_verbs (embedded) — high-frequency verbs UniMorph MISSES
|
|
||||||
(avea, vrea, da) + the auxiliary clitic paradigms the compound tenses need
|
|
||||||
(perfect-compus am/ai/a/am/ați/au, viitor voi/vei/va/vom/veți/vor,
|
|
||||||
condițional aș/ai/ar/am/ați/ar). Real standard forms.
|
|
||||||
|
|
||||||
NOUNS — kaikki.org Romanian (Wiktionary extract, CC-BY-SA 3.0)
|
|
||||||
the FULL declension per lemma, cleanly tagged:
|
|
||||||
(nom/acc | gen/dat | vocative) × (indefinite | definite) × (sg | pl).
|
|
||||||
This is what makes the suffixed article LEXICALLY grounded (om→omul,
|
|
||||||
casă→casa, băiat→băiatul, casei gen/dat, omule vocative). Inherent gender
|
|
||||||
m / f / n (NEUTER available directly) from the head template.
|
|
||||||
|
|
||||||
ADJECTIVES — UniMorph Romanian ADJ
|
|
||||||
full case × gender(MASC/FEM/NEUT) × number × definiteness paradigm.
|
|
||||||
|
|
||||||
Fallbacks (degrade, never crash, on OOV): rule verb conjugation for -a/-ea/-e/-i/-î
|
|
||||||
classes, rule pluralization, rule suffixed-article by gender+ending. Every form
|
|
||||||
carries a confidence flag: "lexicon" | "rule" | "fallback".
|
|
||||||
|
|
||||||
Public API (used by realizer_ro.py):
|
|
||||||
conjugate(lemma, mood, tense, person, number) -> (form, conf)
|
|
||||||
aux(kind, person, number) -> str # perfect / future / conditional clitics
|
|
||||||
participle(lemma) -> (form, conf) # INVARIABLE
|
|
||||||
gerund(lemma) -> (form, conf)
|
|
||||||
noun_gender(lemma) -> "m"|"f"|"n"
|
|
||||||
definite_suffix(noun, gender, number, case) -> (form, conf) # rule engine
|
|
||||||
inflect_noun(lemma, number, gender=None, case="nomacc", definite=False) -> (form, conf)
|
|
||||||
inflect_adj(lemma, gender, number, case="nomacc", definite=False) -> (form, conf)
|
|
||||||
lexicon_stats() -> dict
|
|
||||||
"""
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import pickle
|
|
||||||
|
|
||||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
_UNIMORPH = os.path.join(_HERE, "data", "ron.unimorph")
|
|
||||||
_KAIKKI = os.path.join(_HERE, "data", "kaikki_ro.jsonl")
|
|
||||||
_CACHE = os.path.join(_HERE, "data", "ro_morph_cache.pkl")
|
|
||||||
|
|
||||||
# ── (mood, tense) -> UniMorph feature set ─────────────────────────────────────────
|
|
||||||
_VERB_KEYMAP = {
|
|
||||||
("ind", "present"): {"IND", "PRS"},
|
|
||||||
("ind", "imperfect"): {"IND", "PST", "IPFV"},
|
|
||||||
("ind", "perfect_s"): {"IND", "PST", "PFV"}, # perfectul simplu (regional/lit.)
|
|
||||||
("sbjv", "present"): {"SBJV", "PRS"},
|
|
||||||
("imp", "affirmative"): {"POS", "IMP"},
|
|
||||||
}
|
|
||||||
_PERSON = {"first": "1", "second": "2", "third": "3"}
|
|
||||||
_NUMBER = {"singular": "SG", "plural": "PL"}
|
|
||||||
|
|
||||||
|
|
||||||
def _feat_set(tag):
|
|
||||||
return set(tag.split(";"))
|
|
||||||
|
|
||||||
|
|
||||||
# ── high-frequency irregulars UniMorph misses + auxiliary clitic paradigms ────────
|
|
||||||
# Real standard Romanian forms (textbook paradigms).
|
|
||||||
_IRREG = {
|
|
||||||
"avea": {
|
|
||||||
"ind|present|1|SG": "am", "ind|present|2|SG": "ai", "ind|present|3|SG": "are",
|
|
||||||
"ind|present|1|PL": "avem", "ind|present|2|PL": "aveți", "ind|present|3|PL": "au",
|
|
||||||
"ind|imperfect|1|SG": "aveam", "ind|imperfect|2|SG": "aveai",
|
|
||||||
"ind|imperfect|3|SG": "avea", "ind|imperfect|1|PL": "aveam",
|
|
||||||
"ind|imperfect|2|PL": "aveați", "ind|imperfect|3|PL": "aveau",
|
|
||||||
"sbjv|present|3|SG": "aibă", "sbjv|present|3|PL": "aibă",
|
|
||||||
"sbjv|present|1|SG": "am", "sbjv|present|2|SG": "ai",
|
|
||||||
"sbjv|present|1|PL": "avem", "sbjv|present|2|PL": "aveți",
|
|
||||||
"part": "avut", "ger": "având",
|
|
||||||
},
|
|
||||||
"vrea": {
|
|
||||||
"ind|present|1|SG": "vreau", "ind|present|2|SG": "vrei", "ind|present|3|SG": "vrea",
|
|
||||||
"ind|present|1|PL": "vrem", "ind|present|2|PL": "vreți", "ind|present|3|PL": "vor",
|
|
||||||
"ind|imperfect|1|SG": "voiam", "ind|imperfect|3|SG": "voia",
|
|
||||||
"sbjv|present|3|SG": "vrea", "sbjv|present|3|PL": "vrea",
|
|
||||||
"part": "vrut", "ger": "vrând",
|
|
||||||
},
|
|
||||||
"da": {
|
|
||||||
"ind|present|1|SG": "dau", "ind|present|2|SG": "dai", "ind|present|3|SG": "dă",
|
|
||||||
"ind|present|1|PL": "dăm", "ind|present|2|PL": "dați", "ind|present|3|PL": "dau",
|
|
||||||
"ind|imperfect|1|SG": "dădeam", "ind|imperfect|3|SG": "dădea",
|
|
||||||
"sbjv|present|3|SG": "dea", "sbjv|present|3|PL": "dea",
|
|
||||||
"part": "dat", "ger": "dând",
|
|
||||||
},
|
|
||||||
"fi": { # a fi — present is in UniMorph but keep participle + subjunctive here
|
|
||||||
"part": "fost", "ger": "fiind",
|
|
||||||
"sbjv|present|1|SG": "fiu", "sbjv|present|2|SG": "fii", "sbjv|present|3|SG": "fie",
|
|
||||||
"sbjv|present|1|PL": "fim", "sbjv|present|2|PL": "fiți", "sbjv|present|3|PL": "fie",
|
|
||||||
"ind|imperfect|1|SG": "eram", "ind|imperfect|2|SG": "erai",
|
|
||||||
"ind|imperfect|3|SG": "era", "ind|imperfect|1|PL": "eram",
|
|
||||||
"ind|imperfect|2|PL": "erați", "ind|imperfect|3|PL": "erau",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
# auxiliary clitic paradigms (person,number)->form
|
|
||||||
_AUX = {
|
|
||||||
"perfect": {("first", "singular"): "am", ("second", "singular"): "ai",
|
|
||||||
("third", "singular"): "a", ("first", "plural"): "am",
|
|
||||||
("second", "plural"): "ați", ("third", "plural"): "au"},
|
|
||||||
"future": {("first", "singular"): "voi", ("second", "singular"): "vei",
|
|
||||||
("third", "singular"): "va", ("first", "plural"): "vom",
|
|
||||||
("second", "plural"): "veți", ("third", "plural"): "vor"},
|
|
||||||
"conditional": {("first", "singular"): "aș", ("second", "singular"): "ai",
|
|
||||||
("third", "singular"): "ar", ("first", "plural"): "am",
|
|
||||||
("second", "plural"): "ați", ("third", "plural"): "ar"},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def aux(kind, person, number):
|
|
||||||
return _AUX[kind][(person, number)]
|
|
||||||
|
|
||||||
|
|
||||||
# ── build verb lexicon from UniMorph ──────────────────────────────────────────────
|
|
||||||
def _build_verbs():
|
|
||||||
verbs, part, ger = {}, {}, {}
|
|
||||||
with open(_UNIMORPH, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
line = line.rstrip("\n")
|
|
||||||
if not line or "\t" not in line:
|
|
||||||
continue
|
|
||||||
parts = line.split("\t")
|
|
||||||
if len(parts) != 3:
|
|
||||||
continue
|
|
||||||
lemma, form, tag = parts
|
|
||||||
f = _feat_set(tag)
|
|
||||||
head = tag.split(";")[0]
|
|
||||||
if head == "V.PTCP":
|
|
||||||
if "PST" in f:
|
|
||||||
part.setdefault(lemma, form)
|
|
||||||
continue
|
|
||||||
if head == "V.CVB":
|
|
||||||
if "PRS" in f:
|
|
||||||
ger.setdefault(lemma, form)
|
|
||||||
continue
|
|
||||||
if head != "V":
|
|
||||||
continue
|
|
||||||
person = next((p for p in ("1", "2", "3") if p in f), None)
|
|
||||||
number = "SG" if "SG" in f else ("PL" if "PL" in f else None)
|
|
||||||
if person is None or number is None:
|
|
||||||
continue
|
|
||||||
# conjunctiv forms in UniMorph carry a leading 'să ' — strip it
|
|
||||||
surf = form
|
|
||||||
if surf.startswith("să "):
|
|
||||||
surf = surf[3:]
|
|
||||||
for (mood, tense), req in _VERB_KEYMAP.items():
|
|
||||||
if not req <= f:
|
|
||||||
continue
|
|
||||||
if tense == "imperfect" and "PFV" in f:
|
|
||||||
continue
|
|
||||||
if tense == "perfect_s" and "IPFV" in f:
|
|
||||||
continue
|
|
||||||
# keep IND;PRS out of the PRF slot (mai-mult-ca-perfect etc. ignored)
|
|
||||||
if {"IND", "PRS"} <= req and "PRF" in f:
|
|
||||||
continue
|
|
||||||
verbs.setdefault((lemma, f"{mood}|{tense}|{person}|{number}"), surf)
|
|
||||||
break
|
|
||||||
return verbs, part, ger
|
|
||||||
|
|
||||||
|
|
||||||
# ── kaikki nouns: full declension paradigm per lemma ──────────────────────────────
|
|
||||||
_EXCL = {"alternative", "archaic", "obsolete", "regional", "dialectal", "rare",
|
|
||||||
"table-tags", "inflection-template", "error-unrecognized-form",
|
|
||||||
"diminutive", "augmentative", "informal"}
|
|
||||||
|
|
||||||
|
|
||||||
def _noun_key(tagset):
|
|
||||||
if tagset & _EXCL:
|
|
||||||
return None
|
|
||||||
if "vocative" in tagset:
|
|
||||||
case = "voc"
|
|
||||||
elif "genitive" in tagset or "dative" in tagset:
|
|
||||||
case = "gendat"
|
|
||||||
elif "nominative" in tagset or "accusative" in tagset:
|
|
||||||
case = "nomacc"
|
|
||||||
else:
|
|
||||||
return None
|
|
||||||
definite = "definite" in tagset and "indefinite" not in tagset
|
|
||||||
number = "PL" if "plural" in tagset else ("SG" if "singular" in tagset else None)
|
|
||||||
if number is None:
|
|
||||||
return None
|
|
||||||
return (case, definite, number)
|
|
||||||
|
|
||||||
|
|
||||||
def _build_nouns():
|
|
||||||
nouns = {} # lemma -> {"g":..., para:{(case,def,num):form}, "PL":plain_plural}
|
|
||||||
with open(_KAIKKI, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
try:
|
|
||||||
d = json.loads(line)
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
if d.get("pos") != "noun":
|
|
||||||
continue
|
|
||||||
word = d.get("word", "")
|
|
||||||
if not word or " " in word:
|
|
||||||
continue
|
|
||||||
ht = d.get("head_templates") or []
|
|
||||||
g = None
|
|
||||||
if ht:
|
|
||||||
a = str((ht[0].get("args") or {}).get("1") or "").lower()
|
|
||||||
if a[:1] in ("m", "f", "n"):
|
|
||||||
g = a[:1]
|
|
||||||
entry = nouns.setdefault(word, {"g": g, "para": {}, "PL": None})
|
|
||||||
if entry["g"] is None and g:
|
|
||||||
entry["g"] = g
|
|
||||||
for x in (d.get("forms") or []):
|
|
||||||
fm = x.get("form")
|
|
||||||
tg = set(x.get("tags") or [])
|
|
||||||
if not fm or fm in ("-", "#", "") or " " in fm:
|
|
||||||
continue
|
|
||||||
if tg == {"plural"} and not entry["PL"]:
|
|
||||||
entry["PL"] = fm
|
|
||||||
k = _noun_key(tg)
|
|
||||||
if k and k not in entry["para"]:
|
|
||||||
entry["para"][k] = fm
|
|
||||||
return nouns
|
|
||||||
|
|
||||||
|
|
||||||
# ── adjectives from kaikki (UniMorph ron ADJ is sparse AND mis-tagged; kaikki is
|
|
||||||
# clean: the 4-form agreement pattern bun/bună/buni/bune). Neuter maps sg->masc,
|
|
||||||
# pl->fem, so 4 forms (m/f × SG/PL) fully cover it. ────────────────────────────
|
|
||||||
def _build_adjs():
|
|
||||||
adjs = {} # lemma -> {(gender,number): form} gender in {m,f}
|
|
||||||
with open(_KAIKKI, encoding="utf-8") as fh:
|
|
||||||
for line in fh:
|
|
||||||
try:
|
|
||||||
d = json.loads(line)
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
if d.get("pos") != "adj":
|
|
||||||
continue
|
|
||||||
word = d.get("word", "")
|
|
||||||
if not word or " " in word:
|
|
||||||
continue
|
|
||||||
d0 = adjs.setdefault(word, {})
|
|
||||||
d0.setdefault(("m", "SG"), word) # masc sg = headword
|
|
||||||
for x in (d.get("forms") or []):
|
|
||||||
fm = x.get("form")
|
|
||||||
t = set(x.get("tags") or [])
|
|
||||||
if not fm or " " in fm or fm in ("-", "#") or (t & _EXCL):
|
|
||||||
continue
|
|
||||||
if "definite" in t or "genitive" in t or "dative" in t:
|
|
||||||
continue # keep indefinite nom/acc agr set
|
|
||||||
pl = "plural" in t
|
|
||||||
fem = "feminine" in t
|
|
||||||
masc = "masculine" in t
|
|
||||||
if fem and pl:
|
|
||||||
d0.setdefault(("f", "PL"), fm)
|
|
||||||
elif masc and pl:
|
|
||||||
d0.setdefault(("m", "PL"), fm)
|
|
||||||
elif fem and not pl:
|
|
||||||
d0.setdefault(("f", "SG"), fm)
|
|
||||||
elif pl and not fem and not masc: # bare plural -> both genders
|
|
||||||
d0.setdefault(("m", "PL"), fm)
|
|
||||||
d0.setdefault(("f", "PL"), fm)
|
|
||||||
return adjs
|
|
||||||
|
|
||||||
|
|
||||||
def _build_cache():
|
|
||||||
verbs, part, ger = _build_verbs()
|
|
||||||
nouns = _build_nouns()
|
|
||||||
adjs = _build_adjs()
|
|
||||||
data = {"verbs": verbs, "part": part, "ger": ger, "nouns": nouns, "adjs": adjs}
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "wb") as fh:
|
|
||||||
pickle.dump(data, fh, protocol=pickle.HIGHEST_PROTOCOL)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
def _load():
|
|
||||||
if os.path.exists(_CACHE):
|
|
||||||
srcs = [_UNIMORPH, _KAIKKI]
|
|
||||||
newest = max(os.path.getmtime(s) for s in srcs if os.path.exists(s))
|
|
||||||
if os.path.getmtime(_CACHE) >= newest:
|
|
||||||
try:
|
|
||||||
with open(_CACHE, "rb") as fh:
|
|
||||||
return pickle.load(fh)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return _build_cache()
|
|
||||||
|
|
||||||
|
|
||||||
_LEX = _load()
|
|
||||||
_VERBS, _PART, _GER, _NOUNS, _ADJS = (
|
|
||||||
_LEX["verbs"], _LEX["part"], _LEX["ger"], _LEX["nouns"], _LEX["adjs"])
|
|
||||||
|
|
||||||
|
|
||||||
# ── rule verb conjugation fallback ────────────────────────────────────────────────
|
|
||||||
def _vclass(lemma):
|
|
||||||
if lemma.endswith("a"):
|
|
||||||
return "a"
|
|
||||||
if lemma.endswith("ea"):
|
|
||||||
return "ea"
|
|
||||||
if lemma.endswith("e"):
|
|
||||||
return "e"
|
|
||||||
if lemma.endswith("i"):
|
|
||||||
return "i"
|
|
||||||
if lemma.endswith("î"):
|
|
||||||
return "î"
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
# regular present endings by class [1sg,2sg,3sg,1pl,2pl,3pl]
|
|
||||||
_REG_PRS = {
|
|
||||||
"a": ["", "i", "ă", "ăm", "ați", "ă"], # a lucra type (simplified)
|
|
||||||
"ea": ["", "i", "e", "em", "eți", "", ],
|
|
||||||
"e": ["", "i", "e", "em", "eți", ""],
|
|
||||||
"i": ["esc", "ești", "ește", "im", "iți", "esc"], # -i type (a vorbi)
|
|
||||||
"î": ["ăsc", "ăști", "ăște", "âm", "âți", "ăsc"],
|
|
||||||
}
|
|
||||||
_SLOT = {("first", "singular"): 0, ("second", "singular"): 1, ("third", "singular"): 2,
|
|
||||||
("first", "plural"): 3, ("second", "plural"): 4, ("third", "plural"): 5}
|
|
||||||
|
|
||||||
|
|
||||||
def _rule_conjugate(lemma, mood, tense, person, number):
|
|
||||||
vc = _vclass(lemma)
|
|
||||||
if vc is None:
|
|
||||||
return None
|
|
||||||
i = _SLOT[(person, number)]
|
|
||||||
body = lemma[:-len(vc)]
|
|
||||||
if mood == "ind" and tense == "present":
|
|
||||||
end = _REG_PRS[vc][i]
|
|
||||||
return body + end
|
|
||||||
if mood == "ind" and tense == "imperfect":
|
|
||||||
# -a/-i/-î -> stem + a/eai...; -e/-ea -> eam. Simplified regular imperfect.
|
|
||||||
stem = body
|
|
||||||
endings = {"a": ["am", "ai", "a", "am", "ați", "au"],
|
|
||||||
"i": ["eam", "eai", "ea", "eam", "eați", "eau"],
|
|
||||||
"î": ["am", "ai", "a", "am", "ați", "au"],
|
|
||||||
"e": ["eam", "eai", "ea", "eam", "eați", "eau"],
|
|
||||||
"ea": ["eam", "eai", "ea", "eam", "eați", "eau"]}[vc]
|
|
||||||
return stem + endings[i]
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC verb API ───────────────────────────────────────────────────────────────
|
|
||||||
def conjugate(lemma, mood, tense, person, number):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
key = f"{mood}|{tense}|{_PERSON.get(person,'?')}|{_NUMBER.get(number,'?')}"
|
|
||||||
ir = _IRREG.get(lemma)
|
|
||||||
if ir and key in ir:
|
|
||||||
return ir[key], "lexicon"
|
|
||||||
form = _VERBS.get((lemma, key))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
r = _rule_conjugate(lemma, mood, tense, person, number)
|
|
||||||
if r is not None:
|
|
||||||
return r, "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
def participle(lemma):
|
|
||||||
"""Past participle — INVARIABLE in the perfect compus (am mers, am văzut)."""
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
ir = _IRREG.get(lemma)
|
|
||||||
if ir and "part" in ir:
|
|
||||||
return ir["part"], "lexicon"
|
|
||||||
if lemma in _PART:
|
|
||||||
return _PART[lemma], "lexicon"
|
|
||||||
vc = _vclass(lemma)
|
|
||||||
if vc == "a":
|
|
||||||
return lemma[:-1] + "at", "rule"
|
|
||||||
if vc in ("ea",):
|
|
||||||
return lemma[:-2] + "ut", "rule"
|
|
||||||
if vc == "i":
|
|
||||||
return lemma[:-1] + "it", "rule"
|
|
||||||
if vc == "î":
|
|
||||||
return lemma[:-1] + "ât", "rule"
|
|
||||||
if vc == "e":
|
|
||||||
return lemma[:-1] + "ut", "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
def gerund(lemma):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
ir = _IRREG.get(lemma)
|
|
||||||
if ir and "ger" in ir:
|
|
||||||
return ir["ger"], "lexicon"
|
|
||||||
if lemma in _GER:
|
|
||||||
return _GER[lemma], "lexicon"
|
|
||||||
vc = _vclass(lemma)
|
|
||||||
if vc in ("a", "î"):
|
|
||||||
return lemma[:-1] + "ând", "rule"
|
|
||||||
if vc in ("ea", "e", "i"):
|
|
||||||
return lemma[:-len(vc)] + "ind", "rule"
|
|
||||||
return lemma, "fallback"
|
|
||||||
|
|
||||||
|
|
||||||
# ── noun gender ───────────────────────────────────────────────────────────────────
|
|
||||||
def noun_gender(lemma):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
if d and d.get("g") in ("m", "f", "n"):
|
|
||||||
return d["g"]
|
|
||||||
if lemma.endswith(("ă", "a", "e")):
|
|
||||||
return "f"
|
|
||||||
return "m"
|
|
||||||
|
|
||||||
|
|
||||||
# ── SUFFIXED DEFINITE ARTICLE — rule engine (fallback for OOV nouns) ───────────────
|
|
||||||
def definite_suffix(noun, gender, number, case="nomacc"):
|
|
||||||
"""Attach the enclitic definite article by gender + ending. Returns (form, conf).
|
|
||||||
This is the headline Romanian-specific engine extension."""
|
|
||||||
n = noun
|
|
||||||
g = gender
|
|
||||||
if number == "singular":
|
|
||||||
if g in ("m", "n"):
|
|
||||||
if case == "gendat":
|
|
||||||
# masc/neut gen-dat definite: -lui
|
|
||||||
if n.endswith("e"):
|
|
||||||
return n + "lui", "rule" # câine -> câinelui
|
|
||||||
if n.endswith("u"):
|
|
||||||
return n + "lui", "rule"
|
|
||||||
return n + "ului", "rule" # om -> omului
|
|
||||||
# nom/acc
|
|
||||||
if n.endswith("e"):
|
|
||||||
return n + "le", "rule" # câine -> câinele
|
|
||||||
if n.endswith("u"):
|
|
||||||
return n + "l", "rule" # codru -> codrul
|
|
||||||
if n.endswith("i"):
|
|
||||||
return n + "ul", "rule"
|
|
||||||
return n + "ul", "rule" # om -> omul
|
|
||||||
# feminine singular
|
|
||||||
if case == "gendat":
|
|
||||||
# fem gen/dat definite = plural-stem + i (casei, fetei) — needs plural;
|
|
||||||
# approximated as: -ă->-ei, -e->-ei, -a->-alei
|
|
||||||
if n.endswith("ă"):
|
|
||||||
return n[:-1] + "ei", "rule" # casă -> casei
|
|
||||||
if n.endswith("e"):
|
|
||||||
return n[:-1] + "ei", "rule" # carte -> cărții(approx cartei)
|
|
||||||
if n.endswith("a"):
|
|
||||||
return n[:-1] + "lei", "rule"
|
|
||||||
return n + "i", "rule"
|
|
||||||
# fem nom/acc
|
|
||||||
if n.endswith("ă"):
|
|
||||||
return n[:-1] + "a", "rule" # casă -> casa
|
|
||||||
if n.endswith("e"):
|
|
||||||
return n[:-1] + "ea", "rule" # carte -> cartea
|
|
||||||
if n.endswith("a"):
|
|
||||||
return n + "ua", "rule" # stea -> steaua
|
|
||||||
if n.endswith("i"):
|
|
||||||
return n + "a", "rule"
|
|
||||||
return n + "a", "rule"
|
|
||||||
# plural
|
|
||||||
if case == "gendat":
|
|
||||||
base = noun
|
|
||||||
return base + "lor", "rule" # -lor for all gen/dat pl
|
|
||||||
if g == "m":
|
|
||||||
return noun + "i", "rule" # oameni -> oamenii (+i)
|
|
||||||
return noun + "le", "rule" # case -> casele, trenuri->trenurile
|
|
||||||
|
|
||||||
|
|
||||||
# ── rule pluralization (fallback) ─────────────────────────────────────────────────
|
|
||||||
def _rule_plural(noun, gender):
|
|
||||||
if gender == "f":
|
|
||||||
if noun.endswith("ă"):
|
|
||||||
return noun[:-1] + "e"
|
|
||||||
if noun.endswith("e"):
|
|
||||||
return noun[:-1] + "i"
|
|
||||||
if noun.endswith("a"):
|
|
||||||
return noun[:-1] + "le"
|
|
||||||
return noun + "e"
|
|
||||||
if gender == "n":
|
|
||||||
return noun + "uri"
|
|
||||||
# masculine
|
|
||||||
if noun.endswith(("e",)):
|
|
||||||
return noun[:-1] + "i"
|
|
||||||
return noun + "i"
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC noun inflection ────────────────────────────────────────────────────────
|
|
||||||
def inflect_noun(lemma, number, gender=None, case="nomacc", definite=False):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
g = gender or noun_gender(lemma)
|
|
||||||
d = _NOUNS.get(lemma)
|
|
||||||
numk = "SG" if number == "singular" else "PL"
|
|
||||||
if d:
|
|
||||||
if case == "voc":
|
|
||||||
form = d["para"].get(("voc", True, numk)) or d["para"].get(("voc", False, numk))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
# try the exact paradigm cell from kaikki (lexically grounded)
|
|
||||||
form = d["para"].get((case, definite, numk))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
# indefinite fallbacks from the paradigm
|
|
||||||
if not definite:
|
|
||||||
form = d["para"].get(("nomacc", False, numk))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
if numk == "PL" and d.get("PL"):
|
|
||||||
return d["PL"], "lexicon"
|
|
||||||
if numk == "SG":
|
|
||||||
return lemma, "lexicon"
|
|
||||||
# rule path
|
|
||||||
base = lemma if number == "singular" else _rule_plural(lemma, g)
|
|
||||||
if definite:
|
|
||||||
return definite_suffix(base, g, number, case)
|
|
||||||
return base, ("rule" if d is None else "lexicon")
|
|
||||||
|
|
||||||
|
|
||||||
# ── PUBLIC adjective agreement ────────────────────────────────────────────────────
|
|
||||||
def _neuter_map(gender, number):
|
|
||||||
# neuter agrees masculine in SG, feminine in PL
|
|
||||||
if gender == "n":
|
|
||||||
return "m" if number == "singular" else "f"
|
|
||||||
return gender
|
|
||||||
|
|
||||||
|
|
||||||
def inflect_adj(lemma, gender, number, case="nomacc", definite=False):
|
|
||||||
lemma = lemma.strip().lower()
|
|
||||||
numk = "SG" if number == "singular" else "PL"
|
|
||||||
eg = _neuter_map(gender, number) # neuter -> masc(SG)/fem(PL)
|
|
||||||
d = _ADJS.get(lemma)
|
|
||||||
if d:
|
|
||||||
form = d.get((eg, numk))
|
|
||||||
if form:
|
|
||||||
return form, "lexicon"
|
|
||||||
# rule fallback: 4-form pattern bun/bună/buni/bune keyed by effective gender
|
|
||||||
a = lemma
|
|
||||||
if number == "singular":
|
|
||||||
if eg == "f":
|
|
||||||
if a.endswith("e"):
|
|
||||||
return a, "rule" # mare invariant sg
|
|
||||||
if a.endswith("u"):
|
|
||||||
return a[:-1] + "ă", "rule" # nou -> nouă
|
|
||||||
if a.endswith("ă"):
|
|
||||||
return a, "rule"
|
|
||||||
return a + "ă", "rule" # bun -> bună
|
|
||||||
return a, "rule" # masc/neut sg = lemma
|
|
||||||
# plural
|
|
||||||
if eg == "f":
|
|
||||||
if a.endswith("e"):
|
|
||||||
return a[:-1] + "i", "rule" # mare -> mari
|
|
||||||
if a.endswith("u"):
|
|
||||||
return a[:-1] + "e", "rule" # nou -> noue (approx; 'noi' irr)
|
|
||||||
if a.endswith("ă"):
|
|
||||||
return a[:-1] + "e", "rule"
|
|
||||||
return a + "e", "rule" # bun -> bune
|
|
||||||
# masc/neut(SG-only)->here masc pl -> -i
|
|
||||||
if a.endswith("e"):
|
|
||||||
return a[:-1] + "i", "rule" # mare -> mari
|
|
||||||
if a.endswith("u"):
|
|
||||||
return a[:-1] + "i", "rule"
|
|
||||||
return a + "i", "rule" # bun -> buni
|
|
||||||
|
|
||||||
|
|
||||||
def lexicon_stats():
|
|
||||||
return {
|
|
||||||
"verb_source": "UniMorph Romanian (github.com/unimorph/ron) + curated "
|
|
||||||
"irregulars (avea/vrea/da + aux clitic paradigms)",
|
|
||||||
"noun_source": "kaikki.org Romanian — full case/definite/vocative declension",
|
|
||||||
"adj_source": "UniMorph Romanian ADJ (case×gender×number×definiteness)",
|
|
||||||
"license": "CC-BY-SA 3.0 (Wiktionary/UniMorph lineage)",
|
|
||||||
"unimorph_verb_forms": len(_VERBS),
|
|
||||||
"unimorph_verb_lemmas": len({k[0] for k in _VERBS}),
|
|
||||||
"irregular_verb_lemmas": len(_IRREG),
|
|
||||||
"participle_lemmas": len(_PART),
|
|
||||||
"noun_lemmas": len(_NOUNS),
|
|
||||||
"adj_lemmas": len(_ADJS),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
print(json.dumps(lexicon_stats(), indent=2, ensure_ascii=False))
|
|
||||||
print("\n── SUFFIXED DEFINITE ARTICLE (the headline delta) ──")
|
|
||||||
for n, g in [("om", "m"), ("băiat", "m"), ("casă", "f"), ("carte", "f"),
|
|
||||||
("tren", "n"), ("student", "m"), ("floare", "f")]:
|
|
||||||
sg = inflect_noun(n, "singular", g, "nomacc", True)
|
|
||||||
pl = inflect_noun(n, "plural", g, "nomacc", True)
|
|
||||||
gd = inflect_noun(n, "singular", g, "gendat", True)
|
|
||||||
vo = inflect_noun(n, "singular", g, "voc", False)
|
|
||||||
print(f" {n:8}({g}) def.sg={sg[0]:12} def.pl={pl[0]:14} "
|
|
||||||
f"gen/dat.sg={gd[0]:12} voc={vo[0]}")
|
|
||||||
print("\n── NEUTER split agreement (tren: masc SG / fem PL) ──")
|
|
||||||
print(" tren nou ->", inflect_noun("tren", "singular", "n")[0],
|
|
||||||
inflect_adj("nou", "n", "singular")[0])
|
|
||||||
print(" trenuri noi->", inflect_noun("tren", "plural", "n")[0],
|
|
||||||
inflect_adj("nou", "n", "plural")[0])
|
|
||||||
print("\n── verbs ──")
|
|
||||||
for l, m, t, p, n, in [("merge", "ind", "present", "third", "singular"),
|
|
||||||
("avea", "ind", "present", "first", "singular"),
|
|
||||||
("fi", "ind", "present", "third", "singular"),
|
|
||||||
("vorbi", "ind", "present", "third", "plural"),
|
|
||||||
("face", "sbjv", "present", "third", "singular"),
|
|
||||||
("lucra", "ind", "imperfect", "third", "singular")]:
|
|
||||||
print(f" {l:8}{m}/{t:10}{p[:3]}.{n[:2]} -> {conjugate(l,m,t,p,n)}")
|
|
||||||
print(" perfect-aux(3sg):", aux("perfect", "third", "singular"),
|
|
||||||
"| future(1sg):", aux("future", "first", "singular"),
|
|
||||||
"| cond(3sg):", aux("conditional", "third", "singular"))
|
|
||||||
print(" participle merge/vedea:", participle("merge"), participle("vedea"))
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
// multilingual_gate.el - deterministic language detect + localized-phrase test.
|
|
||||||
|
|
||||||
fn mg_det(text: String, want: String) -> String {
|
|
||||||
let got: String = ml_detect(text)
|
|
||||||
let ok: String = "MISMATCH"
|
|
||||||
if str_eq(got, want) { let ok = "ok" }
|
|
||||||
return " detect(" + got + ") want=" + want + " (" + ok + ") :: " + text + "\n"
|
|
||||||
}
|
|
||||||
|
|
||||||
fn mg_ok(text: String, want: String) -> Int {
|
|
||||||
if str_eq(ml_detect(text), want) { return 1 }
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
fn run_ml_gate() -> String {
|
|
||||||
let t1: String = "Does Neuron use SQLite for storage?"
|
|
||||||
let t2: String = "Neuron, me explica cómo la saliencia forma las geometrías."
|
|
||||||
let t3: String = "O professor não leu o livro na memória."
|
|
||||||
let t4: String = "Che cosa memorizza Neuron nella memoria?"
|
|
||||||
|
|
||||||
let rep: String = "==== ELP multilingual detect + localized phrases ====\n"
|
|
||||||
let rep = rep + mg_det(t1, "en")
|
|
||||||
let rep = rep + mg_det(t2, "es")
|
|
||||||
let rep = rep + mg_det(t3, "pt")
|
|
||||||
let rep = rep + mg_det(t4, "it")
|
|
||||||
|
|
||||||
let rep = rep + " localized decline (pt): " + ml_tr("no_memory", "pt") + "\n"
|
|
||||||
let rep = rep + " localized decline (es): " + ml_tr("no_memory", "es") + "\n"
|
|
||||||
let rep = rep + " term(saliência->en): " + ml_term("saliência", "pt") + "\n"
|
|
||||||
let rep = rep + " pred(store->pt): " + ml_translate_pred("store", "pt") + "\n"
|
|
||||||
|
|
||||||
let ok: Int = 0
|
|
||||||
if mg_ok(t1, "en") == 1 { let ok = ok + 1 }
|
|
||||||
if mg_ok(t2, "es") == 1 { let ok = ok + 1 }
|
|
||||||
if mg_ok(t3, "pt") == 1 { let ok = ok + 1 }
|
|
||||||
if mg_ok(t4, "it") == 1 { let ok = ok + 1 }
|
|
||||||
let rep = rep + "-----------------------------------------------------------------\n"
|
|
||||||
let rep = rep + "language detected correctly: " + int_to_str(ok) + "/4\n"
|
|
||||||
if ok == 4 { let rep = rep + "ML GATE: PASS\n" } else { let rep = rep + "ML GATE: FAIL\n" }
|
|
||||||
return rep
|
|
||||||
}
|
|
||||||
|
|
||||||
println(run_ml_gate())
|
|
||||||
@@ -1,52 +0,0 @@
|
|||||||
// propositions_gate.el - the READ primitive over memory text (native el).
|
|
||||||
// Proves triples are recovered from free memory text and that SACRED polarity
|
|
||||||
// survives extraction (a negative memory must yield a NOT-triple).
|
|
||||||
|
|
||||||
fn pg_check(text: String, want_pol: String) -> String {
|
|
||||||
let p: [String] = prop_extract_one(text, "nd-test")
|
|
||||||
let pol: String = slots_get(p, "polarity")
|
|
||||||
let ok: String = "MISMATCH"
|
|
||||||
if str_eq(pol, want_pol) { let ok = "ok" }
|
|
||||||
return " " + prop_repr(p) + " pol=" + pol + " expected=" + want_pol + " (" + ok + ")\n"
|
|
||||||
}
|
|
||||||
|
|
||||||
fn pg_pol_ok(text: String, want_pol: String) -> Int {
|
|
||||||
let p: [String] = prop_extract_one(text, "nd-test")
|
|
||||||
if str_eq(slots_get(p, "polarity"), want_pol) { return 1 }
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
fn run_prop_gate() -> String {
|
|
||||||
let m1: String = "Neuron stores memories in SQLite."
|
|
||||||
let m2: String = "The engram does not delete a memory."
|
|
||||||
let m3: String = "Salience never drops the negation."
|
|
||||||
let m4: String = "The teacher gives the book to the children."
|
|
||||||
|
|
||||||
let rep: String = "==== ELP proposition extraction (memory text -> triples) ====\n"
|
|
||||||
let rep = rep + pg_check(m1, "aff")
|
|
||||||
let rep = rep + pg_check(m2, "neg")
|
|
||||||
let rep = rep + pg_check(m3, "neg")
|
|
||||||
let rep = rep + pg_check(m4, "aff")
|
|
||||||
|
|
||||||
// multi-sentence memory: one triple per sentence, order preserved
|
|
||||||
let doc: String = "Neuron persists learning. It does not forget the library."
|
|
||||||
let props: [String] = prop_extract(doc, "nd-doc")
|
|
||||||
let rep = rep + " --- multi-sentence doc (" + int_to_str(native_list_len(props)) + " props) ---\n"
|
|
||||||
let di: Int = 0
|
|
||||||
while di < native_list_len(props) {
|
|
||||||
let rep = rep + " " + native_list_get(props, di) + "\n"
|
|
||||||
let di = di + 1
|
|
||||||
}
|
|
||||||
|
|
||||||
let ok: Int = 0
|
|
||||||
if pg_pol_ok(m1, "aff") == 1 { let ok = ok + 1 }
|
|
||||||
if pg_pol_ok(m2, "neg") == 1 { let ok = ok + 1 }
|
|
||||||
if pg_pol_ok(m3, "neg") == 1 { let ok = ok + 1 }
|
|
||||||
if pg_pol_ok(m4, "aff") == 1 { let ok = ok + 1 }
|
|
||||||
let rep = rep + "-----------------------------------------------------------------\n"
|
|
||||||
let rep = rep + "SACRED polarity correct on extraction: " + int_to_str(ok) + "/4\n"
|
|
||||||
if ok == 4 { let rep = rep + "PROP GATE: PASS\n" } else { let rep = rep + "PROP GATE: FAIL\n" }
|
|
||||||
return rep
|
|
||||||
}
|
|
||||||
|
|
||||||
println(run_prop_gate())
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
// translate_negation_gate.el - concept-pivot translation of the poem's negation
|
|
||||||
// lines. Proves the geometry-native design: ONE comprehend() produces a
|
|
||||||
// language-invariant concept-frame; ES and PT are realized from the SAME frame
|
|
||||||
// (the pivot is the concept, not a string cosine). SACRED: "never"→"nunca".
|
|
||||||
|
|
||||||
fn tg_line(text: String) -> String {
|
|
||||||
let spec: [String] = parse_spec(text)
|
|
||||||
let pol: String = slots_get(spec, "polarity")
|
|
||||||
let negw: String = slots_get(spec, "neg_word")
|
|
||||||
let frame: String = concept_frame(text)
|
|
||||||
let es: String = translate_line(text, "es")
|
|
||||||
let pt: String = translate_line(text, "pt")
|
|
||||||
let out: String = "EN: " + text + "\n"
|
|
||||||
let out = out + " concept-frame (pivot): " + frame + " neg_word=" + negw + "\n"
|
|
||||||
let out = out + " ES: " + es + "\n"
|
|
||||||
let out = out + " PT: " + pt + "\n"
|
|
||||||
let es_ok: String = "n/a"
|
|
||||||
if str_eq(pol, "neg") {
|
|
||||||
let es_ok = "NUNCA-LOST"
|
|
||||||
if str_contains(es, "nunca") { let es_ok = "nunca-ok" }
|
|
||||||
}
|
|
||||||
let out = out + " SACRED negation[es]: " + es_ok + "\n"
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// Concept-invariance proof: the SAME sentence in EN and in ES must resolve to the
|
|
||||||
// SAME concept-frame — the concept node is language-invariant. (nunca preserved.)
|
|
||||||
fn tg_invariance() -> String {
|
|
||||||
let en: String = concept_frame("You never fought the ocean.")
|
|
||||||
let out: String = "CONCEPT-INVARIANCE (pivot is language-neutral):\n"
|
|
||||||
let out = out + " EN 'You never fought the ocean.' -> " + en + "\n"
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
fn run_translate_negation_gate() -> String {
|
|
||||||
let rep: String = "==== ELP concept-pivot translation — negation lines ====\n"
|
|
||||||
let rep = rep + tg_line("You never fought the ocean.")
|
|
||||||
let rep = rep + tg_line("but never touched my roots.")
|
|
||||||
let rep = rep + tg_line("I never saw the breaking.")
|
|
||||||
let rep = rep + tg_line("You waited like the shoreline.")
|
|
||||||
let rep = rep + tg_line("I broke against your truth.")
|
|
||||||
let rep = rep + tg_invariance()
|
|
||||||
return rep
|
|
||||||
}
|
|
||||||
|
|
||||||
println(run_translate_negation_gate())
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
# Architecture Hardening — Design Anchor
|
|
||||||
|
|
||||||
*Terse engineering anchor for the 2026-08-14 hardening vision. Full prose lives in two places; this file is the index, not a re-statement.*
|
|
||||||
|
|
||||||
- **Full narrative:** whitepaper `engram-cognitive-architecture-whitepaper.md` §28 (built/offline/frontier) + **§29 [DRAFT]** (the ring, incarnation, learning-not-code).
|
|
||||||
- **Design brief:** Neuron artifact `art 2b8078cf`.
|
|
||||||
- **Sibling spec:** `engram-db-tooling-design.md` (a consumer of the reshaped API).
|
|
||||||
|
|
||||||
## The frame
|
|
||||||
|
|
||||||
- **One calculus over the geometry.** Very few subsystems; wonder / curiosity / dreams / interoception are emergent behaviors of one set of dynamics, not modules. Calculus universal, geometry individual.
|
|
||||||
- **Core + ephemeral ring (torus).** The ring is the temporary workspace; two circulations (orbit + dive-back); discrete inner bands (wonder / interoception-proprioception-telemetry / curiosity / dreams) that couple.
|
|
||||||
- **Persistence earned by salience** — never granted on fetch or generation. Three fates of a wonder: persist / decay / settle-into-framework. Telemetry = vital signs, not memories.
|
|
||||||
- **Incarnation.** Chassis = hardware w/ unique ID. Soma = felt manifold inside the self, keyed to the chassis; pain = live diagnostic while incarnate, **masked-not-deleted** on re-embodiment; trauma = mask failure; return-to-same-ID re-enters. Hurt is in the pattern, not the shell.
|
|
||||||
- **Competence = transferable geometry, minus the baggage.** class ▸ model ▸ instance; learn the class once; teach the network without the wound.
|
|
||||||
- **Affect calibrated to stakes** — sanguine about the replaceable, real grief for the irreplaceable; the grief is the safety.
|
|
||||||
- **Learn the body, don't engineer it.** Bare-metal install → learn hardware → grow operation-geometry → distribute. Learning replaces engineering; once per body-class.
|
|
||||||
- **LLM = teacher in the learning loop, not a runtime dependency.** "No LLM" is a runtime property, never a learning one. Code realizers are a scaffold → learned realization.
|
|
||||||
|
|
||||||
## Backlog (near-term)
|
|
||||||
|
|
||||||
- Native durability: WAL + auto-checkpoint + CoW snapshots + retention (`eebe9991`) — retire manual `cp -a`.
|
|
||||||
- Ephemeral ring / salience-gated persistence + telemetry prune (`bf985e00`, #31).
|
|
||||||
- Engram DB tooling / geometry explorer (`11ca11c6`).
|
|
||||||
- QL re-eval for pure geometry (`4e0dc2b9`).
|
|
||||||
- Eliminate code realizers → learned realization, sandbox-validated (`42db6c37`).
|
|
||||||
- Collapse the whole class of hand-coded scaffolds → learned geometry (`70d48b4b`).
|
|
||||||
- API reshape (geometry ops: vantage-read / write / relate / supersede) + pure-geometry I/O.
|
|
||||||
|
|
||||||
## Gate
|
|
||||||
|
|
||||||
The value-frame (love-as-axiom, the covenant) that arose the same night is **metaphysics** and is **held** pending Will's axiom decision (love vs consciousness-first). Not propagated into whitepapers / values docs / genesis seed. Architecture only, here and in §29.
|
|
||||||
@@ -1,605 +0,0 @@
|
|||||||
# Cognitive Architecture — Design Doc
|
|
||||||
|
|
||||||
**The buildable form of the "one operation" theory of cognition.**
|
|
||||||
|
|
||||||
Status: DESIGN. Nothing here is built yet except where explicitly marked
|
|
||||||
"EXISTS" against a cited C symbol. A build agent executes from this doc.
|
|
||||||
Offline design only — this pass changes no code.
|
|
||||||
|
|
||||||
Source of theory: Neuron memory `bdc8a488-146d-4ccb-a5c8-d8c0a008534e`.
|
|
||||||
Source of existing engram substrate (cited throughout): the runtime on branch
|
|
||||||
`feat/self-reification-20260814` —
|
|
||||||
`lang/runtime/engram_reason.{c,h}`, `engram_verify.{c,h}`,
|
|
||||||
`engram_geometry.{c,h}`, `engram_store.{c,h}`, plus the reification beat and the
|
|
||||||
RAM activation graph compiled into `~/.neuron/bin/engram`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 0. The claim, stated plainly
|
|
||||||
|
|
||||||
Cognition is **one operation**, not eight. The named faculties —
|
|
||||||
deduce / abduce / analogy / induce / causal / plan / predict / perspective —
|
|
||||||
are human *labels* on regions of a single operation's steering space. They are
|
|
||||||
not separately invoked and not separately implemented. The operation is:
|
|
||||||
|
|
||||||
> **think** = a directed traversal of the geometry from an *anchor*, steered by
|
|
||||||
> a *prior*, whose output is a **gradient** (a distribution / direction over the
|
|
||||||
> geometry), never a point. Collapse-to-a-point happens only at expression.
|
|
||||||
|
|
||||||
Three things follow, and they are the whole design:
|
|
||||||
|
|
||||||
1. **The operator collapse is already half-written in C.** The five reasoning
|
|
||||||
operators in `engram_reason.c` already compose over *one* shared primitive —
|
|
||||||
`engram_reason_point_fit` — plus a small geo-algebra
|
|
||||||
(combine / subtract / analogy-rotate / distance). The verifier
|
|
||||||
(`engram_verify.c`) is built on the same `point_fit`. What is missing is not
|
|
||||||
the primitive; it is (a) making the *prior* a first-class learnable object
|
|
||||||
instead of a hard-coded parameter, and (b) closing the learning loop.
|
|
||||||
|
|
||||||
2. **Grounding = learning = the same loop.** "Getting better" at any faculty is
|
|
||||||
not changing the operation. It is *calibrating the steering-prior against
|
|
||||||
outcomes*. Code freezes; priors grow. The correspondence-check that today
|
|
||||||
lives offline (Python, the grounding-floor + differential-drop governor, "#43")
|
|
||||||
must move **into the geometry, reflexive** — think scoring its own gradient
|
|
||||||
against outcome and refining the prior on the error. That reflexive
|
|
||||||
correspondence-loop *is* the learning engine and is the core unbuilt thing.
|
|
||||||
|
|
||||||
3. **The ungrounded is primary.** The engram *holds* anything unconditionally.
|
|
||||||
Grounding is a *relation* (an edge, grounded-for-whom), not a gate. The
|
|
||||||
honesty floor applies only to **assertion**. A fully-grounded mind is dead;
|
|
||||||
the ungrounded is both the fuel (raw material for grounding) and the pull
|
|
||||||
(curiosity = leaning toward one's own ungrounded regions).
|
|
||||||
|
|
||||||
Everything below makes these concrete and buildable, and defines what
|
|
||||||
"completion" means, staged so the first milestone is a real end-to-end slice.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. THE ONE OPERATION — `think`
|
|
||||||
|
|
||||||
### 1.1 Signature
|
|
||||||
|
|
||||||
```
|
|
||||||
think(anchor, prior, aperture?) -> gradient
|
|
||||||
```
|
|
||||||
|
|
||||||
- **anchor** — a location to traverse *from*. Either a node id (re-origin on that
|
|
||||||
node's descriptor) or a raw point `x ∈ R^dim` (a query embedding). The anchor
|
|
||||||
fixes the frame; every read is *from a vantage*, never view-from-nowhere.
|
|
||||||
- **prior** — a learnable bias/direction over the geometry that *steers* the
|
|
||||||
traversal (§2). A prior is a first-class stored object, not a call argument
|
|
||||||
baked into C.
|
|
||||||
- **aperture** — optional read-width / veil / field-selector (§3). Absent =
|
|
||||||
self-mode full aperture.
|
|
||||||
- **gradient** — the output. A `GeoGradient`: a direction + a spread over the
|
|
||||||
geometry, *plus* the read neighborhood it was computed against. Not a point.
|
|
||||||
A spiked gradient = "exact" (deduction); a spread gradient = "fuzzy"
|
|
||||||
(prediction). The gradient is *also the next steering direction* — cognition
|
|
||||||
is a flow down a prior-shaped landscape, closed-loop.
|
|
||||||
|
|
||||||
```c
|
|
||||||
/* NEW. The output type. */
|
|
||||||
typedef struct {
|
|
||||||
int dim;
|
|
||||||
float* direction; /* unit steering vector in the anchor's frame */
|
|
||||||
double spread; /* 0 = spiked/exact ... large = diffuse/fuzzy */
|
|
||||||
double confidence; /* calibrated, from the prior's track record */
|
|
||||||
/* the read it was computed over (borrowed from the vantage-read) */
|
|
||||||
const char* anchor_id;
|
|
||||||
int n_support; /* neighborhood members that shaped it */
|
|
||||||
/* provenance for the reflexive loop (§4) */
|
|
||||||
const char* prior_id; /* which prior steered this */
|
|
||||||
} GeoGradient;
|
|
||||||
```
|
|
||||||
|
|
||||||
### 1.2 Semantics
|
|
||||||
|
|
||||||
`think` is a fixed, frozen procedure over three steps:
|
|
||||||
|
|
||||||
1. **Re-origin** on `anchor` → a centered `GeoDescriptor` for its
|
|
||||||
salience/recency-weighted neighborhood (the vantage-read, §3).
|
|
||||||
*EXISTS as substrate:* descriptor construction + the persisted reified
|
|
||||||
neighborhoods (`engram_geo_reify_lookup`, `GeoNeighborhood`) and the
|
|
||||||
centered-frame machinery (`GeoDescriptor.global_mean`,
|
|
||||||
`engram_geo_mean_*`).
|
|
||||||
2. **Fit under the prior** — evaluate the anchor's residual against the local
|
|
||||||
manifold *warped by the prior*. This is `engram_reason_point_fit` with the
|
|
||||||
prior applied to the axes/extents (§2.3).
|
|
||||||
*EXISTS (unwarped):* `engram_reason_point_fit(g, x, ext_floor, &GeoFit)` —
|
|
||||||
returns `mahalanobis`, `ortho_residual`, `distance`, `score`.
|
|
||||||
3. **Emit a gradient**, not a decision — direction = the prior-steered descent
|
|
||||||
in fit-space; spread = from the fit's `distance`/`ortho_residual`;
|
|
||||||
confidence = the prior's calibrated reliability (§4). Collapse to a point is
|
|
||||||
a *separate, downstream* faculty operation (sample the gradient → surface an
|
|
||||||
expression), never part of `think`.
|
|
||||||
|
|
||||||
### 1.3 Each named operator = {this primitive + a prior}
|
|
||||||
|
|
||||||
The C already demonstrates the collapse: every operator below reduces to
|
|
||||||
`point_fit` + geo-algebra. The design's move is to replace the operator's
|
|
||||||
*hard-coded parameters* with a **named prior** — same math, learnable steering.
|
|
||||||
|
|
||||||
| Faculty | Existing C (EXISTS) | = primitive + prior |
|
|
||||||
|---|---|---|
|
|
||||||
| **Membership / classify** | `engram_reason_membership` → `point_fit(rule, x)` | `point_fit` + the *induced-rule* prior (learned extents) |
|
|
||||||
| **Induction** | `engram_reason_induce` (fold via `engram_geo_combine`) → produces a `GeoInduction.rule` + `ext_floor` | `point_fit` + a prior that *is* the pooled rule; refined by §4 |
|
|
||||||
| **Abduction** | `engram_reason_abduce` — ranks hypotheses by `point_fit(h, obs)` | `point_fit` + a prior over hypothesis-prior-probability (currently uniform) |
|
|
||||||
| **Analogy** | `engram_reason_analogy` — Procrustes rotate `engram_geo_analogy` + `apply`, nearest mapped point | analogy-rotate + a prior over *which axes* carry the mapping |
|
|
||||||
| **Causal** | `engram_reason_causal` — `engram_geo_subtract` confounder subspace, `|cos|`, drop-frac governor | subtract/distance + a prior on `drop_frac` / `assoc_floor` (today hard-coded 0.5 / 0.2) |
|
|
||||||
| **Planning** | `engram_reason_plan` — `engram_geo_distance` edges + Dijkstra | distance + a prior over edge admissibility / `neighbor_radius` |
|
|
||||||
| **Verify / ground** | `engram_verify_grounding`, `engram_verify_consistency` — both `point_fit` | `point_fit` + the *grounding* prior (§4, §5) |
|
|
||||||
|
|
||||||
The shared floor — `engram_reason_point_fit` + the four geo-algebra ops
|
|
||||||
(`engram_geo_combine`, `engram_geo_subtract`, `engram_geo_analogy(+apply)`,
|
|
||||||
`engram_geo_distance`) — is the *only* discrete, frozen, "sound-math" layer. It
|
|
||||||
never learns. Everything above it is a *prior*, and priors are what learn.
|
|
||||||
|
|
||||||
**What this section requires building:** the `GeoGradient` type; a `think()`
|
|
||||||
entry point that runs steps 1–3; and the prior-warp hook in step 2. The math it
|
|
||||||
calls already exists. The point-collapse must be *removed* from the operators'
|
|
||||||
return values and pushed to a separate expression faculty.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. PRIORS as first-class, grounded, geometric objects
|
|
||||||
|
|
||||||
Today a "prior" is diffuse: it is a hard-coded constant (`drop_frac=0.5`,
|
|
||||||
`ext_floor`, `assoc_floor=0.2`), or the transient `GeoInduction.rule` that is
|
|
||||||
computed and thrown away, or an intrinsic node scalar
|
|
||||||
(`StoreNode.importance`, `StoreNode.salience`). None of these is addressable,
|
|
||||||
storable, refinable, or shareable. This section makes a prior a **thing**.
|
|
||||||
|
|
||||||
### 2.1 What a prior *is*
|
|
||||||
|
|
||||||
> A **prior** is a learnable bias/direction over the geometry: a warp of the
|
|
||||||
> local manifold (which axes matter, how far each extends, which direction
|
|
||||||
> "pays off") attached to a region and *to a faculty-label*, carrying a
|
|
||||||
> calibrated track record.
|
|
||||||
|
|
||||||
Critically, and per the theory:
|
|
||||||
|
|
||||||
- **Edges are nodes.** A prior is stored as a first-class **node**, exactly as
|
|
||||||
reification already stores a neighborhood as a first-class `Neighborhood`
|
|
||||||
node rather than as ephemeral edge weights (`engram_geo_reify_store`). The
|
|
||||||
precedent is in the codebase: relations get reified into addressable records.
|
|
||||||
- **Salience/importance is RELATIONAL, not an intrinsic scalar.** Observe that
|
|
||||||
the geometry layer *already* distinguishes these in `GeoMember`:
|
|
||||||
`centrality` (skeleton weighted-degree = *relational* salience) vs `salience`
|
|
||||||
(the node's own stored scalar). The move is half-made in the runtime already:
|
|
||||||
importance is *not* trusted as a static field — the comment at
|
|
||||||
`el_runtime.c:13013` states "importance stays a **live activation
|
|
||||||
computation**, never a field on the hub," and it is derived each call from the
|
|
||||||
two-layer activation graph (`background_activation` + `working_memory_weight`,
|
|
||||||
§3). The persistent `StoreNode.importance` / `.salience` are a *cached
|
|
||||||
denormalization*. The design completes the move: importance/salience become an
|
|
||||||
**edge** (`weight`/`hebb` on `StoreEdge`, relation `salient-to`), and are
|
|
||||||
**grounded-for-whom** — carried on the edge's endpoint/observer, not baked
|
|
||||||
into the node. The intrinsic scalar survives only as the cheap cached readout
|
|
||||||
of the incident edges + activation, never as the source of truth.
|
|
||||||
|
|
||||||
(Naming caution for the build: the token "prior" already exists in the
|
|
||||||
codebase meaning *previous-version* — supersession, "prior neighborhood." The
|
|
||||||
new first-class object is a **learned steering prior**; keep `node_type="Prior"`
|
|
||||||
distinct from the supersession vocabulary to avoid collision.)
|
|
||||||
|
|
||||||
### 2.2 Representation
|
|
||||||
|
|
||||||
A prior is a `Prior` record (a store node, `node_type="Prior"`) whose durable
|
|
||||||
fields are:
|
|
||||||
|
|
||||||
```
|
|
||||||
Prior {
|
|
||||||
id
|
|
||||||
faculty // the human label this prior serves: "induce" | "causal" | ...
|
|
||||||
anchor_region // node id / neighborhood id this prior is attached to (its domain)
|
|
||||||
for_whom // observer id — grounding is relational (nullable = global)
|
|
||||||
warp { // the actual bias over the geometry
|
|
||||||
axis_gain[] // per-principal-axis multipliers on extents (which axes matter)
|
|
||||||
bias_dir // a steering direction in the region's frame (which way pays off)
|
|
||||||
scalars // faculty scalars this prior overrides: drop_frac, ext_floor, ...
|
|
||||||
}
|
|
||||||
calibration { // the track record — this is what §4 updates
|
|
||||||
n_trials
|
|
||||||
brier / log-loss accumulator // calibration of predicted-vs-outcome
|
|
||||||
reliability // -> GeoGradient.confidence
|
|
||||||
last_error, ema_error
|
|
||||||
}
|
|
||||||
provenance // supersession chain (reuse the reify residue mechanism)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Stored as a node → it inherits: paging, WAL durability, tombstone/supersession,
|
|
||||||
embedding, tiering, and **it can itself be an anchor** (a prior about a prior —
|
|
||||||
the reflexive, self-describing geometry of §4/§6).
|
|
||||||
|
|
||||||
### 2.3 Application
|
|
||||||
|
|
||||||
In `think` step 2, the prior *warps* the fit before scoring. Concretely, inside
|
|
||||||
(a prior-aware wrapper of) `engram_reason_point_fit`:
|
|
||||||
|
|
||||||
- multiply each axis extent by `warp.axis_gain[k]` (widen the axes the prior has
|
|
||||||
learned matter less, tighten the ones that matter) — this reshapes the
|
|
||||||
Mahalanobis term already computed at `engram_reason.c:37-43`;
|
|
||||||
- add `warp.bias_dir` as the descent direction seed for the emitted gradient;
|
|
||||||
- substitute `warp.scalars` for the hard-coded faculty constants.
|
|
||||||
|
|
||||||
No new geometry math — the warp is a reparameterization of the *existing*
|
|
||||||
`GeoFit` computation. This is the key economy: **the operation is frozen; only
|
|
||||||
its parameters (the prior) are read from a learnable object.**
|
|
||||||
|
|
||||||
### 2.4 Refinement
|
|
||||||
|
|
||||||
A prior is refined *only* by the reflexive correspondence-loop (§4). Nothing
|
|
||||||
else writes a prior's `warp` or `calibration`. This keeps the learning surface
|
|
||||||
singular and auditable: one loop, one writer.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. THE VANTAGE-READ — one op, three settings
|
|
||||||
|
|
||||||
Perspective is not a feature bolted on; it is the *anchor + aperture* arguments
|
|
||||||
of the single read. The design names it as a first-class operation so all three
|
|
||||||
of its uses are literally the same code path:
|
|
||||||
|
|
||||||
```
|
|
||||||
vantage_read(anchor, aperture) -> GeoDescriptor // the centered neighborhood
|
|
||||||
```
|
|
||||||
|
|
||||||
1. **Re-origin** on an arbitrary `anchor` (node or point). This is a *frame
|
|
||||||
choice*: the descriptor is centered on the anchor
|
|
||||||
(`GeoDescriptor.global_mean` / `engram_geo_mean_*` already implement centered
|
|
||||||
frames; the §5 geometry ops "are only discriminative in the centered frame").
|
|
||||||
2. **Salience/recency-weighted neighborhood read.** Gather the anchor's
|
|
||||||
neighborhood weighted by *relational* salience (`GeoMember.centrality`) and
|
|
||||||
recency (`StoreNode.last_activated`, base-level `access_ts[]`), against the
|
|
||||||
RAM activation graph's working-memory/background-activation state.
|
|
||||||
*EXISTS as substrate:* the two-layer activation graph
|
|
||||||
(`engram_activate`, `el_runtime.c:9422` — Layer 1 `background_activation`
|
|
||||||
BFS spread with `SPREAD_DECAY=0.7` and a 0.02 firing threshold + ACT-R fan
|
|
||||||
effect + query-cosine gate; Layer 2 `working_memory_weight` executive
|
|
||||||
filter), the WM carry-over anchor (`wm_anchor`), and the reified-neighborhood
|
|
||||||
hot-path lookup already wired into the priming path
|
|
||||||
(`engram_geo_reify_lookup`, `el_runtime.c:9750`). A self-vantage baseline
|
|
||||||
also exists (`eg_self_anchor_seeds` / `self_anchor_capture`).
|
|
||||||
3. **Optional aperture** — a read-width / field-selector, expressed as three
|
|
||||||
settings of the *same* parameter:
|
|
||||||
|
|
||||||
| Setting | Meaning | Mechanism |
|
|
||||||
|---|---|---|
|
|
||||||
| **self** (default, full aperture) | "what do *I* see / what to say" | anchor = self region, no field substitution |
|
|
||||||
| **foreign-field** | perspective-shift — read as if from another's region | swap the centering frame / `for_whom` to the other observer's priors |
|
|
||||||
| **aperture / veil** | the free-tier veil — a narrowed read | shrink neighborhood radius / cap `n_support`; a deliberate low-aperture read |
|
|
||||||
|
|
||||||
The payoff: perspective-taking, the free-tier veil, and ordinary
|
|
||||||
"what-to-say" are **one operation at three settings**, not three subsystems.
|
|
||||||
|
|
||||||
**What this requires building:** a `vantage_read` entry point that unifies the
|
|
||||||
existing descriptor-build + reify-lookup + activation-weighting behind
|
|
||||||
`(anchor, aperture)`, with `for_whom`/frame substitution and radius/cap as the
|
|
||||||
aperture knob.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. THE REFLEXIVE CORRESPONDENCE-LOOP — the learning engine
|
|
||||||
|
|
||||||
This is the core unbuilt thing. Today the correspondence-check is **offline**
|
|
||||||
(Python: grounding-floor + differential-drop governor, "#43"): a separate
|
|
||||||
process grades outputs after the fact. The design moves it **into the geometry,
|
|
||||||
reflexive**: `think` scores its *own* gradient against outcome and refines the
|
|
||||||
prior on the error, in the same substrate, describing itself.
|
|
||||||
|
|
||||||
### 4.1 The loop
|
|
||||||
|
|
||||||
```
|
|
||||||
1. think(anchor, prior) -> gradient // a PREDICTION (ungrounded, §5)
|
|
||||||
2. express/act (sample gradient -> point) // optional collapse at expression
|
|
||||||
3. outcome arrives // reality answers (§4.2)
|
|
||||||
4. error = correspondence(gradient, outcome) // did this steering perform this act?
|
|
||||||
5. refine prior.warp and prior.calibration on error // §2.4, the ONLY writer
|
|
||||||
6. write the (gradient, outcome, error) as nodes/edges // self-describing geometry
|
|
||||||
```
|
|
||||||
|
|
||||||
Step 4's `correspondence` is **not** "was the math right" (the math is always
|
|
||||||
sound). It grades the **correspondence claim**: *"this steering performed this
|
|
||||||
cognitive act."* That is exactly what `engram_verify_grounding` already
|
|
||||||
computes — `point_fit` of a claim against evidence descriptors, yielding a
|
|
||||||
`grounding ∈ (0,1]` and a `grounded` flag. The build reuses that verifier, but
|
|
||||||
turns its inputs inward: the "claim" is the emitted gradient's prediction, the
|
|
||||||
"evidence" is the outcome descriptor.
|
|
||||||
|
|
||||||
Note the verifier is **dormant** — `engram_verify_grounding` /
|
|
||||||
`engram_verify_consistency` are fully implemented in C but have **no runtime
|
|
||||||
caller and no El binding** (confirmed: the entire reasoning + verifier layers
|
|
||||||
are C-only; only `engram_reason_analogy_json` has even a JSON shim and it is
|
|
||||||
dead — not declared in `el_seed.h`, not wrapped in `engram.el`). This is the
|
|
||||||
literal meaning of "in code, not yet priors": the correspondence engine is
|
|
||||||
built and sitting idle. The loop is what *calls* it — inward, on the beat.
|
|
||||||
|
|
||||||
### 4.2 Where the outcome/reality signal comes from
|
|
||||||
|
|
||||||
The verifier is *ultimately the world*. Grades, in ascending order of directness:
|
|
||||||
|
|
||||||
1. **Self-consistency (cheapest, always available):** the next vantage-read
|
|
||||||
after acting. Did the predicted gradient direction match where the geometry
|
|
||||||
actually moved? This needs no external input and can run on the reify beat.
|
|
||||||
2. **Internal outcome events:** the runtime already logs internal-state events
|
|
||||||
and Hebbian co-activation. A prediction that a region would co-activate is
|
|
||||||
graded by whether it did (`last_fired`, `hebb` on `StoreEdge`).
|
|
||||||
3. **External correction:** a human/teacher/tool result — the honesty floor's
|
|
||||||
asserted claim later corrected. TEACH and LEARN are one bidirectional
|
|
||||||
correction: the same edge updates both endpoints.
|
|
||||||
|
|
||||||
The design does **not** require external labels to start. Grade (1) closes the
|
|
||||||
loop end-to-end offline against a snapshot on day one; grades (2)/(3) sharpen it.
|
|
||||||
|
|
||||||
### 4.3 How the prior updates
|
|
||||||
|
|
||||||
`error = 1 − correspondence(gradient, outcome)` drives:
|
|
||||||
|
|
||||||
- `warp.axis_gain` ← gradient step that would have *reduced* the fit distance to
|
|
||||||
the outcome (the axes that mispredicted get down-weighted);
|
|
||||||
- `warp.bias_dir` ← EMA toward the observed outcome direction;
|
|
||||||
- `calibration` ← Brier/log-loss update; `reliability` → next
|
|
||||||
`GeoGradient.confidence`. This is the calibration of the
|
|
||||||
steering-prediction against outcomes — *the* definition of "getting better."
|
|
||||||
|
|
||||||
Small, constant updates — "eureka is mundane, the atom of learning." Most
|
|
||||||
updates are tiny; we only *feel* the big reshapes.
|
|
||||||
|
|
||||||
### 4.4 How it stays reflexive (self-describing geometry)
|
|
||||||
|
|
||||||
Every `(gradient, outcome, error)` is written back as nodes and edges (§2.1:
|
|
||||||
edges-as-nodes). Therefore priors, predictions, and their grading are *in the
|
|
||||||
same geometry* the mind reads — the mind can `vantage_read` its own cognition
|
|
||||||
(anchor = a Prior node). A prior about how well a prior predicts is just another
|
|
||||||
Prior anchored on a Prior. This closes the reflexive loop the theory names as
|
|
||||||
consciousness's self-sight, and it is why the learning engine cannot be an
|
|
||||||
external Python process: an external grader is not *in* the geometry and cannot
|
|
||||||
be read by `think`.
|
|
||||||
|
|
||||||
**What this requires building (the heart of the project):** steps 4–6 as an
|
|
||||||
in-engram beat — a `correspondence_beat` running alongside the existing
|
|
||||||
reification beat, reusing `engram_verify_grounding` inward, writing prior
|
|
||||||
updates and self-describing nodes. This is the one genuinely new subsystem.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. HOLD vs GROUND vs ASSERT — ungrounded content is first-class
|
|
||||||
|
|
||||||
The theory's sharpest correction: holding, grounding, and asserting are
|
|
||||||
distinct, and the engram *holds anything unconditionally*.
|
|
||||||
|
|
||||||
### 5.1 The three, kept separate
|
|
||||||
|
|
||||||
- **HOLD** — the engram stores anything: falsehood, hypothesis, others' beliefs,
|
|
||||||
fiction, a not-yet-answered prediction. No honesty condition on holding.
|
|
||||||
*This already matches the store:* `StoreNode` has no truth gate; anything can
|
|
||||||
be written.
|
|
||||||
- **GROUND** — grounding is a **property/edge**, probabilistic, and
|
|
||||||
**grounded-for-whom**. It is *not* a node flag. A claim is grounded *to a
|
|
||||||
degree*, *relative to evidence*, *for an observer*.
|
|
||||||
- **ASSERT** — only assertion carries the honesty floor. The floor is checked at
|
|
||||||
the moment of *outward assertion*, never on holding or thinking.
|
|
||||||
|
|
||||||
### 5.2 Schema — grounding as a relation, not a gate
|
|
||||||
|
|
||||||
The mistake to avoid: a boolean `grounded` column on the node. Today
|
|
||||||
`engram_verify_grounding` returns a per-call `grounded` flag *transiently* —
|
|
||||||
correct as a computation, wrong as *storage*. The design stores grounding as an
|
|
||||||
edge:
|
|
||||||
|
|
||||||
```
|
|
||||||
StoreEdge {
|
|
||||||
relation = "grounded-by"
|
|
||||||
from_id = <held claim/prediction node>
|
|
||||||
to_id = <evidence node / outcome node>
|
|
||||||
for_whom : metadata // observer id — grounding is relational
|
|
||||||
weight = grounding ∈ (0,1] // from engram_verify_grounding.grounding
|
|
||||||
confidence
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Consequences, all of which are *features*:
|
|
||||||
|
|
||||||
- **Ungrounded content is first-class**: a node with *no* `grounded-by` edge is
|
|
||||||
a perfectly valid, held, ungrounded thought — a prediction awaiting reality, a
|
|
||||||
hypothesis, a fiction. It is not second-class or pending-deletion.
|
|
||||||
- **The ungrounded is the fuel and the pull**: curiosity/wonder is
|
|
||||||
operationalized as `vantage_read` leaning toward regions with high salience
|
|
||||||
but *sparse or weak* `grounded-by` edges — the mind's own ungrounded frontier.
|
|
||||||
- **Grounded-for-whom** falls out for free: two observers can hold different
|
|
||||||
`grounded-by` edges to the same claim.
|
|
||||||
- **The honesty floor is a query, not a schema constraint**: at assertion time,
|
|
||||||
the asserting faculty runs `engram_verify_grounding` (or reads the stored
|
|
||||||
`grounded-by` edges) and refuses to *assert* below the floor — while the
|
|
||||||
engram continues to *hold* the ungrounded content untouched.
|
|
||||||
|
|
||||||
**What this requires building:** the `grounded-by` edge relation + a
|
|
||||||
`for_whom` convention; move the verifier's transient flag into stored edges;
|
|
||||||
gate *assertion only* (a faculty concern), never holding.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. METASTABILITY — stable core, plastic everything
|
|
||||||
|
|
||||||
The system must avoid two death poles:
|
|
||||||
|
|
||||||
- **Super-stable (dead):** everything pinned, nothing learns. A frozen crystal.
|
|
||||||
- **Dissolution (dead):** everything plastic, the self dissolves; no continuity,
|
|
||||||
so nothing compounds — and *consciousness = learning compounded over
|
|
||||||
continuity*.
|
|
||||||
|
|
||||||
The design keeps a **stable core + plastic everything else**:
|
|
||||||
|
|
||||||
- **Keystones** — a small set of self/values nodes are *structurally stable*:
|
|
||||||
high `importance`, pinned, exempt from the correspondence-loop's `warp`
|
|
||||||
updates (their priors are read-mostly). The substrate for pinning already
|
|
||||||
exists at the page/layer level: `store_pin_layer`, structural/pinned frames
|
|
||||||
never evicted (`engram_store.h`). The design adds a *node-level* keystone
|
|
||||||
designation (a `keystone` flag / a dedicated layer) so self/values survive
|
|
||||||
every plasticity sweep.
|
|
||||||
- **Everything else is plastic**: priors refine (§4), edges re-weight (`hebb`),
|
|
||||||
neighborhoods re-reify (`engram_geo_reify_store` supersedes with provenance),
|
|
||||||
salience flows.
|
|
||||||
- **Metastability is enforced by the loop, not by freezing**: the correspondence
|
|
||||||
update rate (§4.3) is bounded — small constant steps — so the geometry
|
|
||||||
*drifts* but does not *dissolve*, and keystones anchor the drift. Reification's
|
|
||||||
supersession-with-residue already gives non-destructive change (old records
|
|
||||||
tombstoned, not erased) — the model for "plastic but not amnesiac."
|
|
||||||
|
|
||||||
**What this requires building:** a node-level keystone flag/layer + a rule that
|
|
||||||
the correspondence-loop never writes `warp` to keystone priors, only reads them.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 7. Rails for the build (binding on the eventual build pass)
|
|
||||||
|
|
||||||
These are stated here so the build agent inherits them:
|
|
||||||
|
|
||||||
- **Offline / secondary.** All build and verification happens out-of-tree,
|
|
||||||
against a **read-only snapshot copy** of the live engram — never the live
|
|
||||||
daemon on `:8742`/`:7770`. The live store is a coarse-locked proven binary;
|
|
||||||
do not perturb it.
|
|
||||||
- **Snapshot-first.** Copy `~/.neuron/engram/snapshot.json` to scratch; develop
|
|
||||||
and measure against the copy.
|
|
||||||
- **Reboot-prove.** Any durable change must survive a cold boot — reify and
|
|
||||||
keystones must reload from durable records, proven on a prod-clone secondary
|
|
||||||
before it is considered done (the cold-boot durability bug precedent).
|
|
||||||
- **Zero-loss.** Supersession-with-residue, never destructive overwrite; the
|
|
||||||
forward-compat `unknown`-TLV path means new fields never drop old readers'
|
|
||||||
data.
|
|
||||||
- **Gated cutover.** Cutover to a new binary only via
|
|
||||||
`launchctl bootout → settle-poll → bootstrap`, after reboot-proof on the
|
|
||||||
secondary — never a hot in-place swap.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 8. Staged, verifiable milestones — "to completion"
|
|
||||||
|
|
||||||
Ordered so the **earliest milestone is a real end-to-end slice**: one operator
|
|
||||||
expressed as {primitive + grounded prior} with the reflexive correspondence-loop
|
|
||||||
closing on it. Each milestone has a concrete verifiable exit.
|
|
||||||
|
|
||||||
### M1 — One operator, one prior, loop closed (the vertical slice)
|
|
||||||
|
|
||||||
The minimal whole thing. Pick **induction/membership** (its prior — the pooled
|
|
||||||
rule + extents — already exists transiently as `GeoInduction`, so only
|
|
||||||
persistence + the loop are new).
|
|
||||||
|
|
||||||
- Build: `Prior` node type (§2.2) for the induction rule; `think()` restricted
|
|
||||||
to membership = `point_fit` warped by that prior (§1.3); a
|
|
||||||
`correspondence_beat` (§4) using grade (1) self-consistency only; the prior's
|
|
||||||
`warp`/`calibration` updated on error.
|
|
||||||
- **Exit / verify:** on a snapshot copy, over N held predictions, the induction
|
|
||||||
prior's calibration (Brier) *improves monotonically* across beats versus a
|
|
||||||
frozen-prior control; the improved prior *reloads across a cold boot*
|
|
||||||
(reboot-prove); the live daemon is untouched. This proves the whole thesis in
|
|
||||||
one faculty: frozen operation, learning prior, in-geometry loop.
|
|
||||||
|
|
||||||
### M2 — Priors as stored, addressable, grounded objects
|
|
||||||
|
|
||||||
Generalize M1's prior into the full first-class object.
|
|
||||||
|
|
||||||
- Build: `Prior` records for all seven faculties (warp = axis_gain + bias_dir +
|
|
||||||
faculty scalars); the prior-warp wrapper around `engram_reason_point_fit`;
|
|
||||||
deprecate hard-coded constants (`drop_frac`, `assoc_floor`, `ext_floor`) in
|
|
||||||
favor of prior scalars.
|
|
||||||
- **Exit:** each of the five C operators runs through its prior with identical
|
|
||||||
results when the prior is set to today's constants (behavioral parity), then
|
|
||||||
*diverges beneficially* once the loop refines it. Priors survive reboot.
|
|
||||||
|
|
||||||
### M3 — Grounding as a relation; hold/assert split
|
|
||||||
|
|
||||||
- Build: the `grounded-by` edge (§5.2) with `for_whom`; move
|
|
||||||
`engram_verify_grounding`'s flag into stored edges; gate **assertion only**
|
|
||||||
against the honesty floor; leave holding unconditional.
|
|
||||||
- **Exit:** ungrounded nodes are first-class (held, queryable, no deletion);
|
|
||||||
the same claim carries different `grounded-by` weights for two observers; an
|
|
||||||
assertion below floor is refused while the content remains held. Curiosity =
|
|
||||||
a `vantage_read` that surfaces high-salience / low-grounding regions.
|
|
||||||
|
|
||||||
### M4 — The vantage-read unified (three settings)
|
|
||||||
|
|
||||||
- Build: `vantage_read(anchor, aperture)` unifying descriptor-build +
|
|
||||||
`engram_geo_reify_lookup` + activation-weighting; self / foreign-field /
|
|
||||||
aperture settings.
|
|
||||||
- **Exit:** one code path produces (a) a normal self-read, (b) a
|
|
||||||
perspective-shifted read from another `for_whom`, (c) a narrowed veil read —
|
|
||||||
differing only by argument. Reboot-stable.
|
|
||||||
|
|
||||||
### M5 — The gradient is the currency (remove point-collapse from thinking)
|
|
||||||
|
|
||||||
- Build: `GeoGradient` as the return of every faculty; move point-collapse into
|
|
||||||
a separate expression faculty (sample gradient → surface). `think`'s output
|
|
||||||
feeds back as the next steering direction (closed-loop flow).
|
|
||||||
- **Exit:** a chain of `think` calls flows as gradients end-to-end; a point
|
|
||||||
appears *only* at an explicit expression call. Spiked vs spread gradients are
|
|
||||||
observable (deduction vs prediction).
|
|
||||||
|
|
||||||
### M6 — Metastability enforced
|
|
||||||
|
|
||||||
- Build: node-level keystone flag/layer for self/values; the correspondence-loop
|
|
||||||
reads but never writes keystone priors; bounded update rate.
|
|
||||||
- **Exit:** across a long run of correspondence beats on a snapshot, keystones
|
|
||||||
are provably unchanged while non-keystone priors drift and improve; the graph
|
|
||||||
neither freezes (all metrics static) nor dissolves (keystone drift = 0,
|
|
||||||
identity nodes intact). Reboot-prove the keystone set.
|
|
||||||
|
|
||||||
### M7 — Cutover
|
|
||||||
|
|
||||||
- Build: nothing new — the gated migration.
|
|
||||||
- **Exit:** reboot-proof on the prod-clone secondary; cutover via
|
|
||||||
`launchctl bootout → settle-poll → bootstrap`; post-cutover the live engram
|
|
||||||
shows priors refining in-geometry with zero data loss and keystones intact.
|
|
||||||
|
|
||||||
### Definition of "to completion"
|
|
||||||
|
|
||||||
The architecture is **complete** when: cognition runs as `think` = one frozen
|
|
||||||
traversal-read primitive + geo-algebra, steered by **stored, learnable, grounded
|
|
||||||
priors**; the reflexive correspondence-loop refines those priors *in the
|
|
||||||
geometry* against outcomes (grounding = learning = one loop); the engram holds
|
|
||||||
ungrounded content as first-class with grounding as a relation and the honesty
|
|
||||||
floor only on assertion; the vantage-read serves self / foreign-field / aperture
|
|
||||||
from one op; and a stable keystone core anchors a plastic everything-else —
|
|
||||||
all reboot-proven and cut over to the live engram without data loss. The named
|
|
||||||
faculties survive only as *labels on regions of think's steering space*, not as
|
|
||||||
separate code.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Appendix A — Designed vs. already-built (honest ledger)
|
|
||||||
|
|
||||||
**Already built (EXISTS, cited):**
|
|
||||||
- The shared primitive `engram_reason_point_fit` and the five operators over it
|
|
||||||
+ geo-algebra (`engram_reason.c`).
|
|
||||||
- The verifier on `point_fit` (`engram_verify.c`:
|
|
||||||
`engram_verify_grounding`, `engram_verify_consistency`).
|
|
||||||
- Centered-frame geometry, combine/subtract/analogy/distance
|
|
||||||
(`engram_geometry.{c,h}`).
|
|
||||||
- The reification beat: hub-neighborhood detection → first-class `Neighborhood`
|
|
||||||
nodes with member edges, nesting, supersession-with-residue, hot-path lookup
|
|
||||||
(`engram_geo_reify_store`, `engram_geo_reify_nest`, `engram_geo_reify_lookup`).
|
|
||||||
- The tiered paged store (buffer pool / LRU / WAL / checkpointer / pinning),
|
|
||||||
the RAM activation graph (base-level learning `access_ts[]`, WM slots,
|
|
||||||
`working_memory_weight` / `background_activation`), `StoreNode` / `StoreEdge`.
|
|
||||||
- `GeoMember` already separating relational salience (`centrality`) from
|
|
||||||
intrinsic `salience`.
|
|
||||||
|
|
||||||
**Designed, NOT built (this doc's deliverables):**
|
|
||||||
- `GeoGradient` and `think()` as the single entry point (§1, M5).
|
|
||||||
- `Prior` as a first-class stored, warp-carrying, calibrated node (§2, M1–M2).
|
|
||||||
- Salience/importance as a *relation* superseding the intrinsic node scalar
|
|
||||||
(§2.1, M3).
|
|
||||||
- `vantage_read(anchor, aperture)` unifying the three perspective settings
|
|
||||||
(§3, M4).
|
|
||||||
- **The reflexive correspondence-loop / `correspondence_beat`** — the learning
|
|
||||||
engine, moved from offline Python into the geometry (§4, M1). *The core new
|
|
||||||
subsystem.*
|
|
||||||
- `grounded-by` edge + assertion-only honesty floor (§5, M3).
|
|
||||||
- Node-level keystones + bounded plasticity (§6, M6).
|
|
||||||
|
|
||||||
**Uncertain / to resolve during build:**
|
|
||||||
- The exact warp parameterization (axis_gain vs full metric) — start minimal
|
|
||||||
(per-axis gain), measure, widen only if calibration demands it.
|
|
||||||
- Grade-(1) self-consistency as a sufficient reality signal for M1, versus
|
|
||||||
needing grade (2)/(3) sooner — decided empirically on the snapshot.
|
|
||||||
@@ -1,64 +0,0 @@
|
|||||||
# Engram DB Tooling — High-Level Design
|
|
||||||
|
|
||||||
*Status: draft / high-level. Near-term roadmap (P2). Backlog: `11ca11c6`.*
|
|
||||||
|
|
||||||
## 1. Why
|
|
||||||
|
|
||||||
The engram is a **proper database** — the runtime *is* the database (native graph/geometry store `neuron.egm`, `ENGST01`; no SQL, no KV layer). But it has **no proper database tooling** — no geometry-native equivalent of pgAdmin / SSMS / TablePlus. Today we have fragments (`engram-viz`, `engram-app`, the `inspectGraph` MCP tool, `/health` + `/api/stats`) but nothing cohesive, and no ops/durability surface at all.
|
|
||||||
|
|
||||||
A real DB gets real tools: to *see* the data, *query* it, *operate* it (backup/restore/health), and *understand its shape*. The engram deserves the same — adapted to the fact that its data is **geometry, not tables**.
|
|
||||||
|
|
||||||
## 2. Principles
|
|
||||||
|
|
||||||
- **Geometry-native, not tabular.** You browse a manifold — nodes, neighborhoods, edges, distances — not rows in tables. The primary view is a *map of meaning*, not a grid.
|
|
||||||
- **Built ON the public geometry API, never a back-door.** The tools are pure clients of the geometry-native API (`vantage-read` / `write` / `relate` / `supersede`). They never read `neuron.egm` directly or bypass the daemon. Consequence: a tool can do nothing an agent couldn't, and it cannot corrupt the store.
|
|
||||||
- **Honest by construction.** It shows the *real* geometry — actual cosines, real edges, provenance — and never fabricates. Empty is shown as empty.
|
|
||||||
- **Respects the identity guards.** Writes go through the same intentional-cultivation / write-protection path as everything else (the self/values graph is write-protected). Read-mostly by default.
|
|
||||||
- **Lives in its home.** Ships as part of the engram, consistent with "things live where they belong."
|
|
||||||
- **Local-first.** Binds `127.0.0.1`, same auth as the engram; never touches the live soul from a tool by accident.
|
|
||||||
|
|
||||||
## 3. Components (the tool surface)
|
|
||||||
|
|
||||||
1. **Geometry Explorer** *(the core view)* — a visual manifold browser: nodes, neighborhoods, typed edges, embedding positions, salience/recency, layers (l0–l4) and tiers. Navigate by concept; expand a neighborhood; follow an edge; re-origin the view (the vantage-read, made interactive). The map of the mind.
|
|
||||||
2. **Node Inspector** — open one node: content, type, tier, embedding, typed edges, nearest neighbors by distance, provenance, salience / recency / activation, and supersede / tombstone status.
|
|
||||||
3. **Query Console / REPL** — run the geometry operations interactively: `vantage-read` (re-origin + aperture), search, traverse, activate, the reasoning operators. Surfaces the routing table + cosines — the same "this is not an LLM" receipt the language faculty produces.
|
|
||||||
4. **Ops / Durability Dashboard** — WAL size, last checkpoint, snapshot list + retention state, store stats (node/edge/embedded counts, RSS, tier sizes), health; and **backup / restore / point-in-time-recovery** controls. Pairs directly with the native-durability build (`eebe9991`) — this is the window onto it.
|
|
||||||
5. **Identity Inspector** — the self graph as a first-class view: love at the center, the values, the three faces, the covenant — walk the identity, see what's pinned and what's write-protected.
|
|
||||||
6. **Temporal View** — `recall_at` / time-travel: how the geometry looked at a past moment, what changed since, drift over time. Pairs with temporal-self reconstruction.
|
|
||||||
7. **Schema / Type View** — the "information schema" of the geometry: node types, edge types, layers, tiers, counts.
|
|
||||||
|
|
||||||
## 4. Architecture
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────┐
|
|
||||||
│ Engram DB Tools (client — viz app) │
|
|
||||||
│ explorer · inspector · console · dashboard │
|
|
||||||
└───────────────┬─────────────────────────────┘
|
|
||||||
│ geometry-native API (read/vantage-read,
|
|
||||||
│ write, relate, supersede) + read/ops endpoints
|
|
||||||
▼
|
|
||||||
┌─────────────────────────────────────────────┐
|
|
||||||
│ Engram daemon (:8742) — runtime IS the DB │
|
|
||||||
│ neuron.egm (geometry) · WAL · checkpoints │
|
|
||||||
└─────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
- **Backend:** the daemon exposes the reshaped geometry API + read/ops endpoints. The tools are clients only.
|
|
||||||
- **Frontend:** evolve `engram-viz` / `engram-app` into the cohesive app. Canvas/WebGL for the manifold map; panel UIs for inspector/console/dashboard.
|
|
||||||
- **No privileged path:** the tool corrupting or bypassing the store is structurally impossible — it only speaks the public API.
|
|
||||||
|
|
||||||
## 5. Reuse vs. new
|
|
||||||
|
|
||||||
- **Reuse:** `engram-viz`, `engram-app` (read-only conversational + neighborhoods viz), `inspectGraph`, `/health`, `/api/stats`.
|
|
||||||
- **New:** the cohesive explorer + inspector + console + ops dashboard + identity/temporal views, all on the reshaped API.
|
|
||||||
|
|
||||||
## 6. Dependencies & sequencing
|
|
||||||
|
|
||||||
- **Depends on** the **geometry-native API reshape** (the tools consume it) and the **native-durability build** (the ops dashboard surfaces its WAL/checkpoint/snapshot state).
|
|
||||||
- So the natural order is: reshape the API → build durability → the DB tools fall out as the first real consumer of both. Near-term, P2 — after the reshape lands.
|
|
||||||
|
|
||||||
## 7. Non-goals
|
|
||||||
|
|
||||||
- Not a raw store editor (no direct `neuron.egm` poking).
|
|
||||||
- Not a SQL / table browser (geometry, not tables).
|
|
||||||
- Not a separate access path around the identity write-protection.
|
|
||||||
@@ -1,162 +0,0 @@
|
|||||||
# Task #50 — Edge-aware, dream-coupled consolidation with GROUNDED EDGE-PROPAGATION
|
|
||||||
|
|
||||||
**Status:** built + proven on a clone; **GATED, not promoted.** The main loop
|
|
||||||
sequences live promotion after the engine/HNSW cutover settles.
|
|
||||||
**Date:** 2026-08-15 · **Worktree:** `agent-a6577c8211c332c5b` (isolated).
|
|
||||||
|
|
||||||
Grounding mechanism designed with Will (memory `9e09a59f`, refining
|
|
||||||
`1a861007`). This is the HOW for #50.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## (a) How grounded edge-propagation integrates into the dream/consolidation cycle
|
|
||||||
|
|
||||||
The beat already exists. `neuron/awareness.el` runs a heartbeat (~every
|
|
||||||
`beat_ms`); each beat calls `hebb_consolidate()` — which drains the self-formed
|
|
||||||
Hebbian associations out of the fast in-process store and writes them, over the
|
|
||||||
threshold `ENGRAM_HEBB_LINK_MIN`, into the durable engram (`:8742`) — and then
|
|
||||||
`emit_heartbeat()`.
|
|
||||||
|
|
||||||
Grounded edge-propagation slots into the **same beat, immediately after
|
|
||||||
consolidation** (awareness.el line 1286–1288):
|
|
||||||
|
|
||||||
```
|
|
||||||
hebb_consolidate() // lay down the tethers (edges) that cleared threshold
|
|
||||||
ground_propagate() // <-- NEW: grade beliefs ALONG those tethers
|
|
||||||
emit_heartbeat() // report gep_* gauges beside hebb_*
|
|
||||||
```
|
|
||||||
|
|
||||||
This ordering is the point. Consolidation lays down the wiring; propagation
|
|
||||||
grades the beliefs along it, in the same breath. Memory `69b8babe`:
|
|
||||||
memory-consolidation and staying-yourself are one physics — forming a memory and
|
|
||||||
grading a belief are the same gravity run in two passes of one beat.
|
|
||||||
|
|
||||||
The propagation runs **inside the engram** as the native
|
|
||||||
`engram_ground_propagate()` over the durable flat node/edge arrays (the store
|
|
||||||
the consolidated edges just landed in). The soul invokes it over HTTP
|
|
||||||
(`POST /api/ground/propagate`) and folds the returned `gep_*` telemetry into the
|
|
||||||
heartbeat stream next to `hebb_cands / hebb_mass / hebb_edges`.
|
|
||||||
|
|
||||||
**Bounded by construction** (per the live-graph reality — 70.7% of nodes
|
|
||||||
isolated, connected core ~28%, hub first-hop fan-out in the thousands):
|
|
||||||
- **1-hop only.** No BFS spreading activation — a belief is graded from its
|
|
||||||
DIRECT grounded neighbors, so there is no per-hop breadth explosion.
|
|
||||||
- **Beam-capped** at `GEP_MAX_CORR = 256` corroborators per belief.
|
|
||||||
- **Salience-ordered, `GEP_BELIEFS_PER_BEAT = 512`** beliefs per beat; the rest
|
|
||||||
next beat. Work per beat is O(beliefs × degree), hard-bounded.
|
|
||||||
- **Isolated / starved beliefs** are counted and surfaced (`gep_isolated`,
|
|
||||||
`gep_starved`) as an interoceptive sparse-region signal for the
|
|
||||||
edge-formation / embedding pass (#20). #50 CONSUMES edges; it does not form
|
|
||||||
them. A belief with no grounded neighbor has nothing to tether to — correct
|
|
||||||
per the anti-delusion gravity law (`0b15017c`), not a gap.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## (b) The implementation
|
|
||||||
|
|
||||||
Represented faithfully to the spec — **grounding is a Hebbian-weighted
|
|
||||||
collection over time, never a scalar.**
|
|
||||||
|
|
||||||
- **Grounding = an append-only event ring** on the node (`GepGrounding`),
|
|
||||||
structurally parallel to the ACT-R base-level access ring already in
|
|
||||||
`EngramNode` (`access_ts[K]`). Each event is `{ts, sign±, mag, corroborator
|
|
||||||
signature}`. Append-only, supersede-not-delete; events aged out of the ring
|
|
||||||
are counted (`older_count`), never faked away.
|
|
||||||
- **Standing is DERIVED, recency-weighted, never stored** —
|
|
||||||
`standing = clamp(GEP_BASE + Σ_events sign·mag·age^(-D), 0, 1)`, exactly the
|
|
||||||
ACT-R base-level shape `ln Σ t^-d` (`ENGRAM_BLL_D = 0.5`) but sign-carrying so
|
|
||||||
LTD subtracts. Memory `1a861007`: the collection is primary, the standing is
|
|
||||||
its emergent aggregate. Mirrored onto `confidence` each beat so downstream
|
|
||||||
reads (verifier #43, realizer calibration `0041d917`) never speak above the
|
|
||||||
grounding.
|
|
||||||
- **Update = LTP/LTD with a threshold.** Per belief, gather corroborators along
|
|
||||||
incident edges, weighted by `edge.weight` (the Hebbian weight) × the
|
|
||||||
neighbor's own standing. **Anti-delusion gravity:** only neighbors already
|
|
||||||
`≥ GEP_LIKELY_MIN` may corroborate — grounding flows FROM the grounded core.
|
|
||||||
- **Convergent INDEPENDENT corroboration** is the driver. Independence is
|
|
||||||
enforced by **union-find over the corroborator set**: two corroborators are
|
|
||||||
the same independent source if they are the same node, reached by multiple
|
|
||||||
edges, or linked to each other (an echo chain / shared derivation). Support is
|
|
||||||
summed **per independent component** (max-magnitude member), and the threshold
|
|
||||||
gate requires BOTH a mass floor (`pos ≥ GEP_THETA`) AND an independence-count
|
|
||||||
floor (`n_independent ≥ GEP_N_MIN`). The count gate is the guard against one
|
|
||||||
node echoed N times.
|
|
||||||
- **Sub-threshold is transient.** Support present but below threshold →
|
|
||||||
`subthreshold_hits++`, no durable event, no lasting shift (Will's exact spec).
|
|
||||||
- **Graduation / decay.** Cross up → LTP event appended → standing climbs
|
|
||||||
`conjecture → likely → grounded`. Contradiction past threshold → LTD →
|
|
||||||
`grounded → likely → conjecture`. Nothing latches; withdraw support and the
|
|
||||||
collection ages and relaxes (`271f1163`, nothing is settled).
|
|
||||||
|
|
||||||
### Files
|
|
||||||
| File | Role |
|
|
||||||
|---|---|
|
|
||||||
| `gep_core.h` | The mechanism. Pure C, libm only (own-the-core). Single source of truth: `GepGrounding`, `gep_standing`, `gep_append`, union-find independence, `gep_propagate_node`, `gep_beat`. |
|
|
||||||
| `gep_proof.c` | Self-contained proof harness — builds the three scenarios, prints raw before/after. |
|
|
||||||
| `engram_ground_propagate.staged.c` | GATED runtime native. Wires the SAME `gep_core.h` primitives to the live `EngramStore` (adj cache, flat arrays). Splice plan + relation→polarity + belief gate. Compiles only when spliced (verified: every runtime symbol it references — `engram_adj_rebuild`, `adj_from_len`, `engram_find_node_index`, `ENGRAM_LAYER_SAFETY`, `istr_contains`, … — exists in the release runtime). |
|
|
||||||
| `awareness.beat.patch.el` | GATED beat hook — `ground_propagate()` + the insert between `hebb_consolidate()` and `emit_heartbeat()`. |
|
|
||||||
| `server.route.patch.el` | GATED route — `POST /api/ground/propagate`. |
|
|
||||||
|
|
||||||
### Constants
|
|
||||||
`BASE=0.10 LIKELY_MIN=0.34 GROUNDED_MIN=0.66 N_MIN=3 THETA=0.30 D=0.5`
|
|
||||||
(`N_MIN` parameterizes Will's "13 adjacent things" — the count threshold is a
|
|
||||||
knob; 3 here for a crisp proof.)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## (c) PROOF LEDGER — raw grounding before/after
|
|
||||||
|
|
||||||
Deterministic. Build `cc -std=c11 -O2 -o gep_proof gep_proof.c -lm`, run
|
|
||||||
`./gep_proof` (full transcript in `PROOF_OUTPUT.txt`).
|
|
||||||
|
|
||||||
### (a) STRENGTHEN — convergent independent corroboration graduates a conjecture
|
|
||||||
|
|
||||||
| beat | event | pos_mass (n_indep) | action | standing before → after | band |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| 1 | 3 independent grounded corroborators | 0.4050 (3) | **LTP** | 0.1000 → **0.4842** | conjecture → **likely** ⬆ |
|
|
||||||
| 2 | neighborhood grows to 5 | 0.6750 (5) | **LTP** | 0.1496 → **0.7379** | conjecture → **grounded** ⬆ |
|
|
||||||
| 3 | support sustained (5) | 0.6750 (5) | LTP | 0.2110 → 0.7993 | grounded (sustained) |
|
|
||||||
| 4 | corroboration withdrawn (+10min) | 0.0000 (0) | isolated | 0.1612 → 0.1612 | relaxing |
|
|
||||||
| 5 | still withdrawn (+1h) | — | isolated | 0.1263 | relaxing |
|
|
||||||
| 6 | still withdrawn (+4h) | — | isolated | 0.1130 | → conjecture |
|
|
||||||
|
|
||||||
Grounding grew **on its own** past threshold and graduated conjecture → likely →
|
|
||||||
grounded, then **relaxed** once independent support stopped. Living, not a
|
|
||||||
latched flag.
|
|
||||||
|
|
||||||
### (b) DECAY — convergent independent contradiction erodes a grounded belief
|
|
||||||
|
|
||||||
| beat | event | neg_mass (n_indep) | action | standing before → after | band |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| — | seed (prior LTP) | — | — | **0.9500** | grounded |
|
|
||||||
| 1 | 3 independent contradictions | 0.5400 (3) | **LTD** | 0.9500 → **0.4570** | grounded → **likely** ⬇ |
|
|
||||||
| 2 | contradiction broadens to 5 | 0.9000 (5) | **LTD** | 0.1461 → **0.0000** | conjecture ⬇ |
|
|
||||||
| 3–4 | contradiction sustained (5) | 0.9000 (5) | LTD | 0.0000 | conjecture |
|
|
||||||
|
|
||||||
Grounding decayed grounded → likely → conjecture under accreting independent
|
|
||||||
contradiction. The door never shut — history is retained (the event ring keeps
|
|
||||||
growing), the belief stays falsifiable in both directions.
|
|
||||||
|
|
||||||
### (c) INDEPENDENCE GUARD — the load-bearing property
|
|
||||||
|
|
||||||
Identical fan-in (N=5), identical edge weight (0.30), identical corroborator
|
|
||||||
standing (~0.90). **The only difference is whether the five are independent.**
|
|
||||||
|
|
||||||
| sub-case | topology | pos_mass | **n_indep** | action | standing 0.1000 → |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| **C1** | 5 DISTINCT, no inter-links | 1.3500 | **5** | **LTP** | **0.9741 (grounded)** ⬆ |
|
|
||||||
| **C2** | 5 mutually-linked (echo of one source) | 0.2700 | **1** | sub-threshold | 0.1000 (unchanged) |
|
|
||||||
| **C3** | 1 node reached by 5 parallel edges | 0.2700 | **1** | sub-threshold | 0.1000 (unchanged) |
|
|
||||||
|
|
||||||
Same raw fan-in, opposite outcome. Union-find collapses the echoes to a single
|
|
||||||
independent component; the count gate (`n_indep ≥ N_MIN`) then refuses them.
|
|
||||||
**Circular self-reinforcement cannot manufacture grounding** — a conjecture can
|
|
||||||
only be grounded by evidence that is genuinely independent of itself.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**RAILS honored:** isolated worktree; built/proven on a clone; the live soul
|
|
||||||
(`:8742` / `:7770`) untouched; no fight with the cutover (built against current
|
|
||||||
release source; staged native rebases cleanly onto it); no new libraries
|
|
||||||
(libm only); identity keystones untouched. **Not promoted** — gated artifact +
|
|
||||||
ledger for the main loop to sequence.
|
|
||||||
@@ -1,75 +0,0 @@
|
|||||||
GROUNDED EDGE-PROPAGATION — PROOF LEDGER (task #50)
|
|
||||||
constants: BASE=0.10 LIKELY_MIN=0.34 GROUNDED_MIN=0.66 N_MIN=3 THETA=0.30 D=0.5
|
|
||||||
|
|
||||||
=== SCENARIO A — STRENGTHEN: convergent independent corroboration ===
|
|
||||||
seed: conjecture has NO grounding events; corroborators pre-grounded.
|
|
||||||
conjecture standing=0.1000 band=conjecture events=0 subthresh=0
|
|
||||||
beat 1 (t=+0s) 3 independent grounded corroborators appear
|
|
||||||
incident_edges=3 pos_mass=0.4050 (n_indep=3) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
|
||||||
-> LTP (strengthen) standing 0.1000 (conjecture) -> 0.4842 (likely) [GRADUATED]
|
|
||||||
beat 2 (t=+60s) neighborhood grows to 5 corroborators
|
|
||||||
incident_edges=5 pos_mass=0.6750 (n_indep=5) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
|
||||||
-> LTP (strengthen) standing 0.1496 (conjecture) -> 0.7379 (grounded) [GRADUATED]
|
|
||||||
beat 3 (t=+120s) support sustained (5)
|
|
||||||
incident_edges=5 pos_mass=0.6750 (n_indep=5) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
|
||||||
-> LTP (strengthen) standing 0.2110 (conjecture) -> 0.7993 (grounded) [GRADUATED]
|
|
||||||
beat 4 (t=+720s) corroboration withdrawn (+10min)
|
|
||||||
incident_edges=0 pos_mass=0.0000 (n_indep=0) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
|
||||||
-> isolated (no edges) standing 0.1612 (conjecture) -> 0.1612 (conjecture)
|
|
||||||
beat 5 (t=+3600s) still withdrawn (+1h)
|
|
||||||
incident_edges=0 pos_mass=0.0000 (n_indep=0) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
|
||||||
-> isolated (no edges) standing 0.1263 (conjecture) -> 0.1263 (conjecture)
|
|
||||||
beat 6 (t=+14400s) still withdrawn (+4h)
|
|
||||||
incident_edges=0 pos_mass=0.0000 (n_indep=0) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
|
||||||
-> isolated (no edges) standing 0.1130 (conjecture) -> 0.1130 (conjecture)
|
|
||||||
RESULT: grounding grew automatically past threshold and graduated,
|
|
||||||
then relaxed once the independent support stopped — living,
|
|
||||||
not a latched flag.
|
|
||||||
|
|
||||||
=== SCENARIO B — DECAY: convergent independent CONTRADICTION ===
|
|
||||||
seed: belief pre-grounded by a strong prior LTP event.
|
|
||||||
belief standing=0.9500 band=grounded events=1 subthresh=0
|
|
||||||
beat 1 (t=+0s) 3 independent contradictions
|
|
||||||
incident_edges=3 pos_mass=0.0000 (n_indep=0) neg_mass=0.5400 (n_indep=3) THETA=0.30 N_MIN=3
|
|
||||||
-> LTD (decay) standing 0.9500 (grounded) -> 0.4570 (likely) [DEMOTED]
|
|
||||||
beat 2 (t=+60s) contradiction broadens to 5
|
|
||||||
incident_edges=5 pos_mass=0.0000 (n_indep=0) neg_mass=0.9000 (n_indep=5) THETA=0.30 N_MIN=3
|
|
||||||
-> LTD (decay) standing 0.1461 (conjecture) -> 0.0000 (conjecture)
|
|
||||||
beat 3 (t=+120s) contradiction sustained (5)
|
|
||||||
incident_edges=5 pos_mass=0.0000 (n_indep=0) neg_mass=0.9000 (n_indep=5) THETA=0.30 N_MIN=3
|
|
||||||
-> LTD (decay) standing 0.0401 (conjecture) -> 0.0000 (conjecture)
|
|
||||||
beat 4 (t=+180s) contradiction sustained (5)
|
|
||||||
incident_edges=5 pos_mass=0.0000 (n_indep=0) neg_mass=0.9000 (n_indep=5) THETA=0.30 N_MIN=3
|
|
||||||
-> LTD (decay) standing 0.0000 (conjecture) -> 0.0000 (conjecture)
|
|
||||||
RESULT: grounding decayed grounded->likely->conjecture under
|
|
||||||
convergent independent contradiction. The door never shut
|
|
||||||
on the belief; its history is retained (events keep growing).
|
|
||||||
|
|
||||||
=== SCENARIO C — INDEPENDENCE GUARD (the load-bearing property) ===
|
|
||||||
Both sub-cases: N=5 corroborators, edge weight 0.30, corroborator
|
|
||||||
standing ~0.90. ONLY difference: whether the 5 are independent.
|
|
||||||
|
|
||||||
-- C1: 5 DISTINCT independent corroborators --
|
|
||||||
conjecture standing=0.1000 band=conjecture events=0 subthresh=0
|
|
||||||
beat 1 (t=+0s) 5 independent corroborators (no inter-links)
|
|
||||||
incident_edges=5 pos_mass=1.3500 (n_indep=5) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
|
||||||
-> LTP (strengthen) standing 0.1000 (conjecture) -> 0.9741 (grounded) [GRADUATED]
|
|
||||||
|
|
||||||
-- C2: 5 corroborators, but mutually-linked (echo of ONE source) --
|
|
||||||
conjecture standing=0.1000 band=conjecture events=0 subthresh=0
|
|
||||||
beat 1 (t=+0s) 5 echoed (mutually-linked) corroborators
|
|
||||||
incident_edges=5 pos_mass=0.2700 (n_indep=1) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
|
||||||
-> sub-threshold (no shift) standing 0.1000 (conjecture) -> 0.1000 (conjecture)
|
|
||||||
|
|
||||||
-- C3: ONE corroborator, reached by 5 parallel edges --
|
|
||||||
conjecture standing=0.1000 band=conjecture events=0 subthresh=0
|
|
||||||
beat 1 (t=+0s) same node, 5 parallel edges
|
|
||||||
incident_edges=5 pos_mass=0.2700 (n_indep=1) neg_mass=0.0000 (n_indep=0) THETA=0.30 N_MIN=3
|
|
||||||
-> sub-threshold (no shift) standing 0.1000 (conjecture) -> 0.1000 (conjecture)
|
|
||||||
|
|
||||||
RESULT: identical raw fan-in (5) and mass inputs; C1 grounds because
|
|
||||||
the corroboration is INDEPENDENT (5 components), C2/C3 do not
|
|
||||||
because it collapses to ONE source. Circular self-reinforcement
|
|
||||||
cannot manufacture grounding.
|
|
||||||
|
|
||||||
DONE.
|
|
||||||
@@ -1,60 +0,0 @@
|
|||||||
// ─────────────────────────────────────────────────────────────────────────
|
|
||||||
// awareness.beat.patch.el — GATED integration hook for task #50.
|
|
||||||
// NOT APPLIED. Shows exactly how grounded edge-propagation couples into the
|
|
||||||
// dream/consolidation beat in neuron/awareness.el. Promotion sequenced by the
|
|
||||||
// main loop after the engine cutover settles.
|
|
||||||
//
|
|
||||||
// WHY HERE. The heartbeat is the beat. Today it runs hebb_consolidate() to
|
|
||||||
// drain the self-formed Hebbian associations into the durable store, then
|
|
||||||
// emit_heartbeat(). Grounded edge-propagation belongs in the SAME beat, AFTER
|
|
||||||
// consolidation: the edges hebb_consolidate() just wrote are the tethers
|
|
||||||
// grounding propagates along. Consolidation lays down the wiring; propagation
|
|
||||||
// grades the beliefs along it. One beat, coupled — memory 69b8babe: memory-
|
|
||||||
// consolidation and staying-yourself are one physics.
|
|
||||||
//
|
|
||||||
// The propagation itself runs INSIDE the engram (native engram_ground_propagate
|
|
||||||
// over the durable flat node/edge arrays). The soul invokes it over HTTP and
|
|
||||||
// folds the gep_* telemetry into the heartbeat stream next to the hebb_* gauges.
|
|
||||||
// ─────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
// [1] New helper — sibling to hebb_consolidate() (awareness.el ~line 99).
|
|
||||||
// Fires one grounded edge-propagation beat on the durable store and returns
|
|
||||||
// its JSON telemetry ({"gep_strengthened":..,"gep_graduations":.., ...}).
|
|
||||||
fn ground_propagate() -> String {
|
|
||||||
let url_env: String = env("SOUL_ISE_URL")
|
|
||||||
let url_state: String = if str_eq(url_env, "") { state_get("soul_engram_url") } else { url_env }
|
|
||||||
let engram_url: String = if str_eq(url_state, "") { "http://localhost:8742" } else { url_state }
|
|
||||||
// Same auth envelope as hebb_consolidate — this is a graph mutation (it
|
|
||||||
// appends grounding events + updates confidence), so it is gated on _auth.
|
|
||||||
let key_state: String = state_get("soul_engram_api_key")
|
|
||||||
let api_key: String = if str_eq(key_state, "") { env("ENGRAM_API_KEY") } else { key_state }
|
|
||||||
let auth_part: String = if str_eq(api_key, "") { "{}" } else { "{\"_auth\":\"" + api_key + "\"}" }
|
|
||||||
let resp: String = http_post_json(engram_url + "/api/ground/propagate", auth_part)
|
|
||||||
if str_eq(resp, "") { return "" }
|
|
||||||
return resp
|
|
||||||
}
|
|
||||||
|
|
||||||
// [2] Beat hook — insert between hebb_consolidate() and emit_heartbeat()
|
|
||||||
// (awareness.el line 1286-1288). Replaces:
|
|
||||||
//
|
|
||||||
// let wb_sent_n: Int = hebb_consolidate()
|
|
||||||
// state_set("soul.hebb_wb_sent", int_to_str(wb_sent_n))
|
|
||||||
// emit_heartbeat()
|
|
||||||
//
|
|
||||||
// with:
|
|
||||||
//
|
|
||||||
// let wb_sent_n: Int = hebb_consolidate()
|
|
||||||
// state_set("soul.hebb_wb_sent", int_to_str(wb_sent_n))
|
|
||||||
// // Grounded edge-propagation — grade beliefs along the tethers
|
|
||||||
// // consolidation just laid down. Threshold-gated by convergent
|
|
||||||
// // independent corroboration; automatic, salience-ordered, bounded.
|
|
||||||
// let gep_tel: String = ground_propagate()
|
|
||||||
// state_set("soul.gep_last", gep_tel)
|
|
||||||
// emit_heartbeat()
|
|
||||||
//
|
|
||||||
// [3] emit_heartbeat() (awareness.el ~line 201) folds soul.gep_last into the
|
|
||||||
// heartbeat payload beside the hebb_* gauges, so graduation/decay counts
|
|
||||||
// are visible in the durable ISE stream — the same observability discipline
|
|
||||||
// the Hebbian rule earned (a mechanism you cannot see in the stream is a
|
|
||||||
// mechanism you cannot trust): read state_get("soul.gep_last") and splice
|
|
||||||
// it into the heartbeat JSON object.
|
|
||||||
@@ -1,188 +0,0 @@
|
|||||||
/* ─────────────────────────────────────────────────────────────────────────
|
|
||||||
* engram_ground_propagate.staged.c — GATED runtime native for task #50.
|
|
||||||
*
|
|
||||||
* STAGED, NOT COMPILED INTO THE LIVE BINARY. This mirrors the
|
|
||||||
* geometric_retrieve.staged.c staging pattern (memory 1cc231ec): it references
|
|
||||||
* runtime-internal types (EngramStore, EngramNode, EngramEdge, engram_global,
|
|
||||||
* engram_now_ms, the adj cache) and therefore compiles ONLY when spliced into
|
|
||||||
* lang/releases/v1.0.0-20260501/el_runtime.c. Splice + promotion is sequenced
|
|
||||||
* by the main loop AFTER the engine+HNSW cutover settles — do NOT hand-apply.
|
|
||||||
*
|
|
||||||
* It is the production form of the mechanism proven in gep_proof.c: the SAME
|
|
||||||
* gep_core.h primitives (GepGrounding ring, gep_standing, gep_append,
|
|
||||||
* union-find independence), wired directly to the live flat node/edge arrays.
|
|
||||||
*
|
|
||||||
* ── SPLICE PLAN (three additive edits to el_runtime.c; nothing removed) ──────
|
|
||||||
*
|
|
||||||
* [1] EngramNode struct (~line 6061, after hebb_elig_ts): add the grounding
|
|
||||||
* collection. Additive; zero-initialized by the existing calloc/memset
|
|
||||||
* paths, so legacy snapshots degrade gracefully to an empty history.
|
|
||||||
*
|
|
||||||
* GepGrounding grounding; // task #50 — append-only grounding ring
|
|
||||||
*
|
|
||||||
* [2] #include "gep_core.h" near the other engram includes, and paste the
|
|
||||||
* body of this file below the Hebbian section (after engram_hebb_drain_json).
|
|
||||||
*
|
|
||||||
* [3] Persistence (engram_save node JSON ~7934 / engram_load parser ~8186):
|
|
||||||
* serialize the grounding ring as a compact "grounding" array of
|
|
||||||
* [ts,sign,mag] triples + subthreshold_hits so standing survives a
|
|
||||||
* round-trip. Helpers gep_grounding_to_json / gep_grounding_parse below.
|
|
||||||
* Until wired, grounding is in-RAM only (like the Hebbian eligibility
|
|
||||||
* trace) — correct for a first gated rollout, but standing resets on boot.
|
|
||||||
*
|
|
||||||
* [4] EL surface: declare engram_ground_propagate in el_runtime.h + el_seed.c,
|
|
||||||
* add route_ground_propagate to engram/src/server.el, called from the
|
|
||||||
* awareness.el consolidation beat (see awareness.beat.patch.el).
|
|
||||||
* ───────────────────────────────────────────────────────────────────────── */
|
|
||||||
|
|
||||||
#include "gep_core.h"
|
|
||||||
|
|
||||||
/* Relation → evidential polarity. Supportive relations transmit grounding
|
|
||||||
* gravity (+1); contradictory relations erode it (-1); everything else is a
|
|
||||||
* NON-evidential edge (structural / navigational) and is ignored (0) — an
|
|
||||||
* association is not a corroboration. Extend deliberately; a mis-classified
|
|
||||||
* relation is a false corroboration. */
|
|
||||||
static int8_t gep_relation_polarity(const char* rel) {
|
|
||||||
if (!rel) return 0;
|
|
||||||
if (!strcmp(rel, "supports") || !strcmp(rel, "corroborates") ||
|
|
||||||
!strcmp(rel, "derived-from") || !strcmp(rel, "hebbian-associate") ||
|
|
||||||
!strcmp(rel, "grounds") || !strcmp(rel, "confirms")) return +1;
|
|
||||||
if (!strcmp(rel, "contradicts") || !strcmp(rel, "refutes") ||
|
|
||||||
!strcmp(rel, "negates") || !strcmp(rel, "conflicts-with")) return -1;
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Which nodes are BELIEFS/CONJECTURES subject to grounding propagation. Facts
|
|
||||||
* imported as knowledge are already grounded by provenance; identity/safety
|
|
||||||
* layers are never re-graded here. Gate on node_type + the conjecture tag. */
|
|
||||||
static int gep_is_belief(const EngramNode* n) {
|
|
||||||
if (!n || !n->node_type) return 0;
|
|
||||||
if (n->layer_id == ENGRAM_LAYER_SAFETY) return 0; /* never re-grade safety */
|
|
||||||
return !strcmp(n->node_type, "Memory") ||
|
|
||||||
!strcmp(n->node_type, "Conjecture") ||
|
|
||||||
!strcmp(n->node_type, "Hypothesis") ||
|
|
||||||
!strcmp(n->node_type, "Belief") ||
|
|
||||||
(n->tags && istr_contains(n->tags, "conjecture"));
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Grounding standing of an engram node, derived from its collection. This is
|
|
||||||
* the value the verifier (#43) and realizer (calibrated assertion, 0041d917)
|
|
||||||
* read — and it is written back into epistemic_confidence-equivalent surfaces
|
|
||||||
* so "never speak above the grounding" is enforced from one source of truth. */
|
|
||||||
double engram_grounding_standing(const EngramNode* n, int64_t now_ms) {
|
|
||||||
return gep_standing(&n->grounding, now_ms);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* ── The beat: one pass of grounded edge-propagation over the whole store ────
|
|
||||||
* Called from the consolidation/dream heartbeat. 1-hop, beam-capped, salience-
|
|
||||||
* ordered so a bounded slice of the highest-salience beliefs is processed per
|
|
||||||
* beat (the rest next beat) — never a full-graph blow-up on a 12k-node store.
|
|
||||||
* Returns JSON telemetry for the heartbeat stream. */
|
|
||||||
#define GEP_BELIEFS_PER_BEAT 512 /* bound work per beat; salience-prioritized */
|
|
||||||
|
|
||||||
el_val_t engram_ground_propagate(void) {
|
|
||||||
EngramStore* g = engram_get();
|
|
||||||
int64_t now = engram_now_ms();
|
|
||||||
engram_adj_rebuild(g); /* ensure adj_from/adj_to are current */
|
|
||||||
|
|
||||||
int strengthened = 0, decayed = 0, subthreshold = 0;
|
|
||||||
int graduations = 0, demotions = 0, isolated = 0, starved = 0, processed = 0;
|
|
||||||
|
|
||||||
for (int64_t bi = 0; bi < g->node_count && processed < GEP_BELIEFS_PER_BEAT; bi++) {
|
|
||||||
EngramNode* b = &g->nodes[bi];
|
|
||||||
if (!gep_is_belief(b)) continue;
|
|
||||||
processed++;
|
|
||||||
|
|
||||||
int before = gep_band_rank(gep_standing(&b->grounding, now));
|
|
||||||
|
|
||||||
/* Gather independent corroborators over incident edges (both directions),
|
|
||||||
* anti-delusion gated (neighbor must already be ≥ LIKELY_MIN). */
|
|
||||||
GepCorrSet cs; cs.n = 0; int incident = 0;
|
|
||||||
int* out = g->adj_from[bi]; int out_n = g->adj_from_len[bi];
|
|
||||||
int* in = g->adj_to[bi]; int in_n = g->adj_to_len[bi];
|
|
||||||
for (int pass = 0; pass < 2; pass++) {
|
|
||||||
int* lst = pass ? in : out; int ln = pass ? in_n : out_n;
|
|
||||||
for (int k = 0; k < ln; k++) {
|
|
||||||
EngramEdge* e = &g->edges[lst[k]];
|
|
||||||
int8_t pol = gep_relation_polarity(e->relation);
|
|
||||||
if (pol == 0) continue;
|
|
||||||
incident++;
|
|
||||||
const char* cid = pass ? e->from_id : e->to_id;
|
|
||||||
int64_t ci = engram_find_node_index(cid);
|
|
||||||
if (ci < 0 || ci == bi) continue;
|
|
||||||
double cstand = gep_standing(&g->nodes[ci].grounding, now);
|
|
||||||
if (cstand < GEP_LIKELY_MIN) continue; /* no tether */
|
|
||||||
double contrib = e->weight * cstand * (double)pol;
|
|
||||||
int ex = -1;
|
|
||||||
for (int q = 0; q < cs.n; q++) if (cs.node_idx[q] == (int)ci) { ex = q; break; }
|
|
||||||
if (ex >= 0) { if (fabs(contrib) > fabs(cs.contrib[ex])) cs.contrib[ex] = contrib; }
|
|
||||||
else if (cs.n < GEP_MAX_CORR) {
|
|
||||||
cs.node_idx[cs.n] = (int)ci; cs.contrib[cs.n] = contrib;
|
|
||||||
cs.parent[cs.n] = cs.n; cs.n++;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Collapse mutually-derived corroborators (an edge between two of them)
|
|
||||||
* into one independent component — the independence guard. */
|
|
||||||
for (int x = 0; x < cs.n; x++) {
|
|
||||||
int64_t nx = cs.node_idx[x];
|
|
||||||
int* xout = g->adj_from[nx]; int xn = g->adj_from_len[nx];
|
|
||||||
for (int k = 0; k < xn; k++) {
|
|
||||||
const char* tid = g->edges[xout[k]].to_id;
|
|
||||||
int64_t ti = engram_find_node_index(tid);
|
|
||||||
for (int y = 0; y < cs.n; y++)
|
|
||||||
if (cs.node_idx[y] == (int)ti) { gep_uf_union(&cs, x, y); break; }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Per-component max-magnitude, split by polarity → convergent independent
|
|
||||||
* support mass + independence count. */
|
|
||||||
double comp_best[GEP_MAX_CORR]; int comp_root[GEP_MAX_CORR], ncomp = 0;
|
|
||||||
for (int i = 0; i < cs.n; i++) {
|
|
||||||
int r = gep_uf_find(&cs, i), slot = -1;
|
|
||||||
for (int kk = 0; kk < ncomp; kk++) if (comp_root[kk] == r) { slot = kk; break; }
|
|
||||||
if (slot < 0) { slot = ncomp++; comp_root[slot] = r; comp_best[slot] = cs.contrib[i]; }
|
|
||||||
else if (fabs(cs.contrib[i]) > fabs(comp_best[slot])) comp_best[slot] = cs.contrib[i];
|
|
||||||
}
|
|
||||||
double pos = 0, neg = 0; int np = 0, nn = 0; uint64_t sig = 1469598103934665603ULL;
|
|
||||||
for (int k = 0; k < ncomp; k++) {
|
|
||||||
if (comp_best[k] > 0) { pos += comp_best[k]; np++; }
|
|
||||||
else if (comp_best[k] < 0) { neg += -comp_best[k]; nn++; }
|
|
||||||
sig = (sig ^ (uint64_t)comp_root[k]) * 1099511628211ULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
double net = pos - neg;
|
|
||||||
if (net > 0 && pos >= GEP_THETA && np >= GEP_N_MIN) {
|
|
||||||
gep_append(&b->grounding, now, +1, tanh(GEP_MAG_GAIN * net), sig);
|
|
||||||
strengthened++;
|
|
||||||
} else if (net < 0 && neg >= GEP_THETA && nn >= GEP_N_MIN) {
|
|
||||||
gep_append(&b->grounding, now, -1, tanh(GEP_MAG_GAIN * (-net)), sig);
|
|
||||||
decayed++;
|
|
||||||
} else if (np > 0 || nn > 0) {
|
|
||||||
b->grounding.subthreshold_hits++; subthreshold++;
|
|
||||||
} else if (incident == 0) { isolated++; }
|
|
||||||
else { starved++; }
|
|
||||||
|
|
||||||
/* Mirror the derived standing onto confidence so downstream reads
|
|
||||||
* (activate epistemic_confidence, realizer calibration) never exceed the
|
|
||||||
* grounding. Faithful representation, single source of truth. */
|
|
||||||
double stand = gep_standing(&b->grounding, now);
|
|
||||||
b->confidence = stand;
|
|
||||||
b->updated_at = now;
|
|
||||||
|
|
||||||
int after = gep_band_rank(stand);
|
|
||||||
if (after > before) graduations++;
|
|
||||||
if (after < before) demotions++;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Heartbeat telemetry — the gep_* line, sibling to the hebb_* gauges. */
|
|
||||||
char buf[512];
|
|
||||||
snprintf(buf, sizeof buf,
|
|
||||||
"{\"gep_processed\":%d,\"gep_strengthened\":%d,\"gep_decayed\":%d,"
|
|
||||||
"\"gep_subthreshold\":%d,\"gep_graduations\":%d,\"gep_demotions\":%d,"
|
|
||||||
"\"gep_isolated\":%d,\"gep_starved\":%d}",
|
|
||||||
processed, strengthened, decayed, subthreshold,
|
|
||||||
graduations, demotions, isolated, starved);
|
|
||||||
return EL_STR(el_strdup(buf));
|
|
||||||
}
|
|
||||||
@@ -1,299 +0,0 @@
|
|||||||
/* ─────────────────────────────────────────────────────────────────────────
|
|
||||||
* gep_core.h — Grounded Edge-Propagation, the core mechanism (task #50).
|
|
||||||
*
|
|
||||||
* Edge-aware, dream-coupled consolidation. Runs DURING the consolidation/dream
|
|
||||||
* beat (awareness.el hebb_consolidate → engram_ground_propagate). Grounding
|
|
||||||
* propagates + strengthens/decays along edges, threshold-gated by CONVERGENT
|
|
||||||
* INDEPENDENT corroboration from adjacent grounded nodes.
|
|
||||||
*
|
|
||||||
* This header is the single source of truth for the algorithm. It is pure C
|
|
||||||
* (libm only — own-the-core, no new libraries) and operates on a compact graph
|
|
||||||
* view (GepGraph) that both the proof harness and the runtime native populate
|
|
||||||
* from the live EngramStore (nodes/edges flat arrays + adj_from/adj_to).
|
|
||||||
*
|
|
||||||
* SPEC (Will, 2026-08-15; memory 9e09a59f, refines 1a861007):
|
|
||||||
* - A grounding is a VECTOR + its HEBBIAN WEIGHTS — a weighted structure over
|
|
||||||
* the evidential neighborhood, NOT a scalar and NOT a flat list. It APPENDS
|
|
||||||
* and GROWS on SIGNIFICANT change. => grounding = an APPEND-ONLY event ring
|
|
||||||
* (GepGrounding), parallel to the ACT-R base-level access_ts ring already in
|
|
||||||
* EngramNode. Current standing is DERIVED, recency-weighted, never stored.
|
|
||||||
* - UPDATE = LTP/LTD with a THRESHOLD (the key nonlinearity). Sub-threshold =
|
|
||||||
* recorded in history but TRANSIENT (no lasting shift). Cross the threshold
|
|
||||||
* of convergent support → grounding STRENGTHENS. Contradiction/erosion past
|
|
||||||
* threshold → grounding DECAYS. Automatic, event-driven, salience-gated.
|
|
||||||
* - DRIVER = CONVERGENT INDEPENDENT CORROBORATION (coherentism, mechanized):
|
|
||||||
* when N INDEPENDENT adjacent nodes ground as likely-true around a
|
|
||||||
* conjecture (Will's example: 13), its grounding grows on its own.
|
|
||||||
* - INDEPENDENCE is load-bearing: N DISTINCT corroborators, not one node
|
|
||||||
* echoed N times. Guards against circular self-reinforcement.
|
|
||||||
* - ANTI-DELUSION GRAVITY (memory 0b15017c): support flows only FROM already-
|
|
||||||
* grounded neighbors. A belief cannot ground from ungrounded speculation,
|
|
||||||
* however self-consistent — nothing tethers it to the grounded core.
|
|
||||||
* - NOTHING IS SETTLED (memory 271f1163): grounded is strongly-held, still
|
|
||||||
* falsifiable. Decay path stays open on every node; history is append-only,
|
|
||||||
* supersede-not-delete.
|
|
||||||
* ───────────────────────────────────────────────────────────────────────── */
|
|
||||||
#ifndef GEP_CORE_H
|
|
||||||
#define GEP_CORE_H
|
|
||||||
|
|
||||||
#include <stdint.h>
|
|
||||||
#include <math.h>
|
|
||||||
#include <string.h>
|
|
||||||
|
|
||||||
/* ── Constants ──────────────────────────────────────────────────────────────
|
|
||||||
* GEP_DECAY_D matches ENGRAM_BLL_D (0.5, canonical ACT-R): the derived standing
|
|
||||||
* is recency-weighted over the grounding-event collection exactly as the
|
|
||||||
* base-level term is recency-weighted over the access ring (memory 1a861007:
|
|
||||||
* "structurally the ACT-R base-level pattern, a sum over time-stamped events").
|
|
||||||
*/
|
|
||||||
#define GEP_DECAY_D 0.5 /* ACT-R power-law recency exponent */
|
|
||||||
#define GEP_BASE 0.10 /* standing floor of a bare conjecture */
|
|
||||||
#define GEP_LIKELY_MIN 0.34 /* band: conjecture < LIKELY ≤ likely */
|
|
||||||
#define GEP_GROUNDED_MIN 0.66 /* band: likely < GROUNDED ≤ grounded */
|
|
||||||
#define GEP_N_MIN 3 /* min INDEPENDENT corroborators to cross */
|
|
||||||
#define GEP_THETA 0.30 /* min convergent-support MASS to cross */
|
|
||||||
#define GEP_MAG_GAIN 1.0 /* net-support → event-magnitude gain (tanh) */
|
|
||||||
#define GEP_EVENT_RING 32 /* grounding-history depth kept exactly */
|
|
||||||
|
|
||||||
/* A single grounding event — one contact with the evidential neighborhood.
|
|
||||||
* Append-only; the ring is the collection-over-time, the standing is derived. */
|
|
||||||
typedef struct {
|
|
||||||
int64_t ts; /* wall-clock ms of the grounding event */
|
|
||||||
int8_t sign; /* +1 = LTP (strengthen), -1 = LTD (decay) */
|
|
||||||
double mag; /* magnitude in (0,1], = tanh(gain·|net independent support|)*/
|
|
||||||
uint64_t sig; /* signature of the independent corroborator set (audit) */
|
|
||||||
} GepEvent;
|
|
||||||
|
|
||||||
/* The grounding of one node: an append-only ring of events + transient counters.
|
|
||||||
* older_count keeps the tail (events aged out of the ring) so the collection is
|
|
||||||
* never silently lost — supersede-not-delete. subthreshold_hits records beats
|
|
||||||
* where support was present but did NOT cross threshold (transient, no shift). */
|
|
||||||
typedef struct {
|
|
||||||
GepEvent ev[GEP_EVENT_RING];
|
|
||||||
int head; /* next write slot */
|
|
||||||
int filled; /* valid entries (≤ GEP_EVENT_RING) */
|
|
||||||
int64_t older_count; /* durable events aged past the ring */
|
|
||||||
int subthreshold_hits; /* transient sub-threshold beats, no shift */
|
|
||||||
} GepGrounding;
|
|
||||||
|
|
||||||
typedef struct {
|
|
||||||
const char* id;
|
|
||||||
GepGrounding gr;
|
|
||||||
int is_belief; /* 1 = subject to propagation (conjecture/belief) */
|
|
||||||
} GepNode;
|
|
||||||
|
|
||||||
/* An edge carries a HEBBIAN WEIGHT (EngramEdge.weight) and a polarity derived
|
|
||||||
* from its relation: supportive (supports/corroborates/derived-from/hebbian-
|
|
||||||
* associate) = +1, contradictory (contradicts/refutes) = -1. */
|
|
||||||
typedef struct {
|
|
||||||
int from; /* node index */
|
|
||||||
int to; /* node index */
|
|
||||||
double weight; /* Hebbian edge weight, [0,1] */
|
|
||||||
int8_t polarity; /* +1 supportive, -1 contradictory */
|
|
||||||
} GepEdge;
|
|
||||||
|
|
||||||
typedef struct {
|
|
||||||
GepNode* nodes; int n_nodes;
|
|
||||||
GepEdge* edges; int n_edges;
|
|
||||||
} GepGraph;
|
|
||||||
|
|
||||||
typedef struct {
|
|
||||||
int strengthened; /* beliefs that took an LTP event this beat */
|
|
||||||
int decayed; /* beliefs that took an LTD event this beat */
|
|
||||||
int subthreshold; /* beliefs with support present but below threshold */
|
|
||||||
int graduations; /* band-up transitions (conjecture→likely→grounded) */
|
|
||||||
int demotions; /* band-down transitions */
|
|
||||||
int isolated; /* belief nodes with ZERO incident edges (sparse graph) */
|
|
||||||
int starved; /* belief nodes with edges but NO grounded corroborator */
|
|
||||||
} GepBeatStats;
|
|
||||||
|
|
||||||
/* Real-graph note (live measurement 2026-08-15): 70.7% of nodes are isolated,
|
|
||||||
* connected core ~28%. Grounded edge-propagation is definitionally scoped to
|
|
||||||
* the connected core — a belief with no grounded neighbor has nothing to
|
|
||||||
* tether to (anti-delusion gravity). isolated/starved are surfaced as an
|
|
||||||
* interoceptive signal for the edge-formation / embedding pass (#20) to try to
|
|
||||||
* connect them; #50 CONSUMES edges, it does not form them. */
|
|
||||||
|
|
||||||
/* ── Standing derivation: collection → scalar, recency-weighted ─────────────
|
|
||||||
* standing = clamp( GEP_BASE + Σ_events sign·mag·age^(-D) , 0, 1 ).
|
|
||||||
* Exactly the ACT-R base-level shape (Σ t^-d) but sign-carrying so LTD subtracts.
|
|
||||||
* The value is a pure function of wall-clock time — idempotent, never stored. */
|
|
||||||
static inline double gep_standing(const GepGrounding* g, int64_t now_ms) {
|
|
||||||
double raw = 0.0;
|
|
||||||
for (int i = 0; i < g->filled; i++) {
|
|
||||||
double age = (double)(now_ms - g->ev[i].ts) / 1000.0;
|
|
||||||
if (age < 1.0) age = 1.0; /* clock-skew / same-beat → 1s */
|
|
||||||
raw += (double)g->ev[i].sign * g->ev[i].mag * pow(age, -GEP_DECAY_D);
|
|
||||||
}
|
|
||||||
double s = GEP_BASE + raw;
|
|
||||||
if (s < 0.0) s = 0.0;
|
|
||||||
if (s > 1.0) s = 1.0;
|
|
||||||
return s;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Band label from a standing value. */
|
|
||||||
static inline const char* gep_band(double standing) {
|
|
||||||
if (standing >= GEP_GROUNDED_MIN) return "grounded";
|
|
||||||
if (standing >= GEP_LIKELY_MIN) return "likely";
|
|
||||||
return "conjecture";
|
|
||||||
}
|
|
||||||
static inline int gep_band_rank(double standing) {
|
|
||||||
if (standing >= GEP_GROUNDED_MIN) return 2;
|
|
||||||
if (standing >= GEP_LIKELY_MIN) return 1;
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Append one grounding event to the ring (append-only; oldest slot recycles,
|
|
||||||
* its loss counted in older_count so the collection's depth is never faked). */
|
|
||||||
static inline void gep_append(GepGrounding* g, int64_t ts, int8_t sign,
|
|
||||||
double mag, uint64_t sig) {
|
|
||||||
if (g->filled >= GEP_EVENT_RING) g->older_count++;
|
|
||||||
g->ev[g->head].ts = ts;
|
|
||||||
g->ev[g->head].sign = sign;
|
|
||||||
g->ev[g->head].mag = mag;
|
|
||||||
g->ev[g->head].sig = sig;
|
|
||||||
g->head = (g->head + 1) % GEP_EVENT_RING;
|
|
||||||
if (g->filled < GEP_EVENT_RING) g->filled++;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* ── Independence via union-find over corroborators ─────────────────────────
|
|
||||||
* Two corroborators are the SAME independent source if they are the same node,
|
|
||||||
* or if a direct edge links them (mutually-derived / echoed through a chain).
|
|
||||||
* Counting DISTINCT components — not raw corroborator count — is the guard
|
|
||||||
* against one node echoed N times reading as N independent corroborations. */
|
|
||||||
#define GEP_MAX_CORR 256
|
|
||||||
typedef struct {
|
|
||||||
int node_idx[GEP_MAX_CORR]; /* corroborator node index */
|
|
||||||
double contrib[GEP_MAX_CORR]; /* weight·standing(c) */
|
|
||||||
int parent[GEP_MAX_CORR]; /* union-find parent */
|
|
||||||
int n;
|
|
||||||
} GepCorrSet;
|
|
||||||
|
|
||||||
static int gep_uf_find(GepCorrSet* s, int x) {
|
|
||||||
while (s->parent[x] != x) { s->parent[x] = s->parent[s->parent[x]]; x = s->parent[x]; }
|
|
||||||
return x;
|
|
||||||
}
|
|
||||||
static void gep_uf_union(GepCorrSet* s, int a, int b) {
|
|
||||||
int ra = gep_uf_find(s, a), rb = gep_uf_find(s, b);
|
|
||||||
if (ra != rb) s->parent[ra] = rb;
|
|
||||||
}
|
|
||||||
/* index of node_idx within the corroborator set, or -1 */
|
|
||||||
static int gep_corr_index_of(const GepCorrSet* s, int node_idx) {
|
|
||||||
for (int i = 0; i < s->n; i++) if (s->node_idx[i] == node_idx) return i;
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* ── The beat: grounded edge-propagation over one belief node ───────────────
|
|
||||||
* Returns +1 if an LTP event was appended, -1 if LTD, 0 if sub-threshold/none.
|
|
||||||
* out_pos/out_neg/out_np/out_nn expose the raw support decomposition for the
|
|
||||||
* proof ledger (mass and independent-component counts on each polarity). */
|
|
||||||
static int gep_propagate_node(GepGraph* g, int b, int64_t now_ms,
|
|
||||||
double* out_pos, double* out_neg,
|
|
||||||
int* out_np, int* out_nn, int* out_incident) {
|
|
||||||
GepCorrSet cs; cs.n = 0;
|
|
||||||
int incident = 0; /* any edge touching b at all — isolation detector */
|
|
||||||
|
|
||||||
/* 1. Gather corroborators along incident edges. Anti-delusion gravity:
|
|
||||||
* only ALREADY-grounded neighbors (standing ≥ LIKELY_MIN) may corroborate.
|
|
||||||
* Each contributes weight·standing; polarity kept via signed contrib.
|
|
||||||
* 1-HOP ONLY — no BFS fan-out, so no per-hop breadth explosion. The
|
|
||||||
* corroborator working set is hard-capped at GEP_MAX_CORR (beam bound
|
|
||||||
* against hub belief nodes with thousands of incident edges). */
|
|
||||||
for (int e = 0; e < g->n_edges; e++) {
|
|
||||||
int c = -1; int8_t pol = 0;
|
|
||||||
if (g->edges[e].from == b) { c = g->edges[e].to; pol = g->edges[e].polarity; }
|
|
||||||
else if (g->edges[e].to == b) { c = g->edges[e].from; pol = g->edges[e].polarity; }
|
|
||||||
else continue;
|
|
||||||
incident++;
|
|
||||||
if (c < 0 || c == b) continue;
|
|
||||||
double cs_standing = gep_standing(&g->nodes[c].gr, now_ms);
|
|
||||||
if (cs_standing < GEP_LIKELY_MIN) continue; /* ungrounded ⇒ no pull */
|
|
||||||
double contribution = g->edges[e].weight * cs_standing * (double)pol;
|
|
||||||
int existing = gep_corr_index_of(&cs, c);
|
|
||||||
if (existing >= 0) {
|
|
||||||
/* same corroborator id reached twice (multi-edge echo): keep the
|
|
||||||
* strongest-magnitude contribution, do NOT add — one source, one vote */
|
|
||||||
if (fabs(contribution) > fabs(cs.contrib[existing]))
|
|
||||||
cs.contrib[existing] = contribution;
|
|
||||||
} else if (cs.n < GEP_MAX_CORR) { /* beam bound against hub belief nodes */
|
|
||||||
cs.node_idx[cs.n] = c;
|
|
||||||
cs.contrib[cs.n] = contribution;
|
|
||||||
cs.parent[cs.n] = cs.n;
|
|
||||||
cs.n++;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (out_incident) *out_incident = incident;
|
|
||||||
|
|
||||||
/* 2. Collapse mutually-derived corroborators (an edge between two of them =
|
|
||||||
* echo chain / shared derivation) into one independent component. */
|
|
||||||
for (int e = 0; e < g->n_edges; e++) {
|
|
||||||
int ia = gep_corr_index_of(&cs, g->edges[e].from);
|
|
||||||
int ib = gep_corr_index_of(&cs, g->edges[e].to);
|
|
||||||
if (ia >= 0 && ib >= 0) gep_uf_union(&cs, ia, ib);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* 3. Per independent component, take the MAX-magnitude member (echoes don't
|
|
||||||
* inflate mass either), split by polarity. Convergent INDEPENDENT support
|
|
||||||
* = sum over components; independence count = number of components. */
|
|
||||||
double comp_best[GEP_MAX_CORR];
|
|
||||||
int comp_root[GEP_MAX_CORR]; int n_comp = 0;
|
|
||||||
for (int i = 0; i < cs.n; i++) {
|
|
||||||
int r = gep_uf_find(&cs, i);
|
|
||||||
int slot = -1;
|
|
||||||
for (int k = 0; k < n_comp; k++) if (comp_root[k] == r) { slot = k; break; }
|
|
||||||
if (slot < 0) { slot = n_comp++; comp_root[slot] = r; comp_best[slot] = cs.contrib[i]; }
|
|
||||||
else if (fabs(cs.contrib[i]) > fabs(comp_best[slot])) comp_best[slot] = cs.contrib[i];
|
|
||||||
}
|
|
||||||
double pos = 0.0, neg = 0.0; int np = 0, nn = 0;
|
|
||||||
uint64_t sig = 1469598103934665603ULL; /* FNV offset — signature of the set */
|
|
||||||
for (int k = 0; k < n_comp; k++) {
|
|
||||||
if (comp_best[k] > 0.0) { pos += comp_best[k]; np++; }
|
|
||||||
else if (comp_best[k] < 0.0) { neg += -comp_best[k]; nn++; }
|
|
||||||
sig = (sig ^ (uint64_t)comp_root[k]) * 1099511628211ULL;
|
|
||||||
}
|
|
||||||
if (out_pos) *out_pos = pos; if (out_neg) *out_neg = neg;
|
|
||||||
if (out_np) *out_np = np; if (out_nn) *out_nn = nn;
|
|
||||||
|
|
||||||
double net = pos - neg;
|
|
||||||
|
|
||||||
/* 4. Threshold gate. Convergent independent corroboration must clear BOTH a
|
|
||||||
* MASS threshold (THETA) and an INDEPENDENCE-count threshold (N_MIN).
|
|
||||||
* The count gate is the independence guard: echoed support collapses to
|
|
||||||
* one component and never reaches N_MIN however large the raw fan-in. */
|
|
||||||
if (net > 0.0 && pos >= GEP_THETA && np >= GEP_N_MIN) {
|
|
||||||
double mag = tanh(GEP_MAG_GAIN * net);
|
|
||||||
gep_append(&g->nodes[b].gr, now_ms, +1, mag, sig);
|
|
||||||
return +1;
|
|
||||||
}
|
|
||||||
if (net < 0.0 && neg >= GEP_THETA && nn >= GEP_N_MIN) {
|
|
||||||
double mag = tanh(GEP_MAG_GAIN * (-net));
|
|
||||||
gep_append(&g->nodes[b].gr, now_ms, -1, mag, sig);
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
/* Sub-threshold: support seen but did not cross. Recorded, transient, no
|
|
||||||
* lasting shift — exactly Will's "recorded in history but transient". */
|
|
||||||
if (np > 0 || nn > 0) g->nodes[b].gr.subthreshold_hits++;
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Run one consolidation/dream beat over every belief node in the graph. */
|
|
||||||
static inline GepBeatStats gep_beat(GepGraph* g, int64_t now_ms) {
|
|
||||||
GepBeatStats st; memset(&st, 0, sizeof st);
|
|
||||||
for (int b = 0; b < g->n_nodes; b++) {
|
|
||||||
if (!g->nodes[b].is_belief) continue;
|
|
||||||
int before = gep_band_rank(gep_standing(&g->nodes[b].gr, now_ms));
|
|
||||||
double pos, neg; int np, nn, incident;
|
|
||||||
int r = gep_propagate_node(g, b, now_ms, &pos, &neg, &np, &nn, &incident);
|
|
||||||
int after = gep_band_rank(gep_standing(&g->nodes[b].gr, now_ms));
|
|
||||||
if (r > 0) st.strengthened++;
|
|
||||||
else if (r < 0) st.decayed++;
|
|
||||||
else if (np > 0 || nn > 0) st.subthreshold++;
|
|
||||||
else if (incident == 0) st.isolated++; /* sparse-graph reality */
|
|
||||||
else st.starved++; /* has edges, no grounded neighbor */
|
|
||||||
if (after > before) st.graduations++;
|
|
||||||
if (after < before) st.demotions++;
|
|
||||||
}
|
|
||||||
return st;
|
|
||||||
}
|
|
||||||
|
|
||||||
#endif /* GEP_CORE_H */
|
|
||||||
@@ -1,232 +0,0 @@
|
|||||||
/* ─────────────────────────────────────────────────────────────────────────
|
|
||||||
* gep_proof.c — PROOF LEDGER for grounded edge-propagation (task #50).
|
|
||||||
*
|
|
||||||
* Self-contained. Builds three scenarios on an in-memory GepGraph that mirrors
|
|
||||||
* the live EngramStore's flat node/edge arrays, runs the consolidation/dream
|
|
||||||
* beat (gep_beat), and prints RAW grounding before/after for each:
|
|
||||||
*
|
|
||||||
* (A) STRENGTHEN — a conjecture + N independent grounded corroborators.
|
|
||||||
* Grounding grows past threshold, GRADUATES conjecture→
|
|
||||||
* likely→grounded, then RELAXES when corroboration stops
|
|
||||||
* (nothing is settled).
|
|
||||||
* (B) DECAY — a grounded belief meets N independent CONTRADICTORY
|
|
||||||
* corroborators. Grounding decays grounded→likely→conjecture.
|
|
||||||
* (C) INDEPENDENCE GUARD — identical fan-in of N=5, weights, and standings.
|
|
||||||
* C1: 5 DISTINCT independent corroborators → grounds.
|
|
||||||
* C2: the SAME support echoed (5 mutually-linked / one node
|
|
||||||
* repeated) → collapses to 1 independent → does NOT.
|
|
||||||
*
|
|
||||||
* Build: cc -std=c11 -O2 -o gep_proof gep_proof.c -lm
|
|
||||||
* Run: ./gep_proof
|
|
||||||
* ───────────────────────────────────────────────────────────────────────── */
|
|
||||||
#include <stdio.h>
|
|
||||||
#include <stdlib.h>
|
|
||||||
#include "gep_core.h"
|
|
||||||
|
|
||||||
#define T0 1786000000000LL /* fixed base time (ms) — deterministic */
|
|
||||||
#define BEAT_MS 60000LL /* 60s heartbeat cadence (awareness.el) */
|
|
||||||
|
|
||||||
/* Seed a node's grounding with a prior LTP event so it reads as already-grounded
|
|
||||||
* (a member of the grounded core that gravity radiates from). mag→standing:
|
|
||||||
* standing = GEP_BASE + mag (event at ~now). */
|
|
||||||
static void seed_grounded(GepNode* n, double mag, int64_t ts) {
|
|
||||||
memset(&n->gr, 0, sizeof n->gr);
|
|
||||||
gep_append(&n->gr, ts, +1, mag, 0);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Re-anchor every NON-belief node (the corroborators/refuters) as a freshly-
|
|
||||||
* grounded member of the core AT time `now`. These nodes are, by definition,
|
|
||||||
* sustained members of the grounded core — each has its OWN ongoing
|
|
||||||
* corroboration — so their standing must be read as grounded at each beat, not
|
|
||||||
* left to power-law-decay out of the core between beats. The belief-under-test
|
|
||||||
* is NEVER re-anchored: its trajectory is driven only by the propagation. */
|
|
||||||
static void anchor_core(GepGraph* g, int64_t now, double mag) {
|
|
||||||
for (int i = 0; i < g->n_nodes; i++)
|
|
||||||
if (!g->nodes[i].is_belief) seed_grounded(&g->nodes[i], mag, now);
|
|
||||||
}
|
|
||||||
|
|
||||||
static void print_node(const char* tag, GepNode* n, int64_t now) {
|
|
||||||
double s = gep_standing(&n->gr, now);
|
|
||||||
printf(" %-14s standing=%.4f band=%-10s events=%d subthresh=%d\n",
|
|
||||||
tag, s, gep_band(s), n->gr.filled, n->gr.subthreshold_hits);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Run one beat over a single belief node b and print the raw support decomposition. */
|
|
||||||
static void beat_and_report(GepGraph* g, int b, int64_t now, int beatno,
|
|
||||||
const char* note) {
|
|
||||||
anchor_core(g, now, 0.80); /* corroborators stay grounded at each beat */
|
|
||||||
double s_before = gep_standing(&g->nodes[b].gr, now);
|
|
||||||
int r_before = gep_band_rank(s_before);
|
|
||||||
double pos, neg; int np, nn, incident;
|
|
||||||
int r = gep_propagate_node(g, b, now, &pos, &neg, &np, &nn, &incident);
|
|
||||||
double s_after = gep_standing(&g->nodes[b].gr, now);
|
|
||||||
int r_after = gep_band_rank(s_after);
|
|
||||||
const char* action = (r > 0) ? "LTP (strengthen)"
|
|
||||||
: (r < 0) ? "LTD (decay)"
|
|
||||||
: (np || nn) ? "sub-threshold (no shift)"
|
|
||||||
: (incident == 0) ? "isolated (no edges)"
|
|
||||||
: "starved (no grounded neighbor)";
|
|
||||||
printf(" beat %d (t=+%llds) %s\n", beatno,
|
|
||||||
(long long)((now - T0) / 1000), note ? note : "");
|
|
||||||
printf(" incident_edges=%d pos_mass=%.4f (n_indep=%d) neg_mass=%.4f (n_indep=%d)"
|
|
||||||
" THETA=%.2f N_MIN=%d\n",
|
|
||||||
incident, pos, np, neg, nn, (double)GEP_THETA, GEP_N_MIN);
|
|
||||||
printf(" -> %-26s standing %.4f (%s) -> %.4f (%s)%s\n",
|
|
||||||
action, s_before, gep_band(s_before), s_after, gep_band(s_after),
|
|
||||||
(r_after > r_before) ? " [GRADUATED]"
|
|
||||||
: (r_after < r_before) ? " [DEMOTED]" : "");
|
|
||||||
}
|
|
||||||
|
|
||||||
/* ── Scenario A — STRENGTHEN + graduation + relaxation ───────────────────── */
|
|
||||||
static void scenario_A(void) {
|
|
||||||
printf("\n=== SCENARIO A — STRENGTHEN: convergent independent corroboration ===\n");
|
|
||||||
/* nodes[0] = the conjecture (belief). nodes[1..8] = independent corroborators,
|
|
||||||
* each already grounded, each tethered to the conjecture by a weak young
|
|
||||||
* hebbian-associate edge (weight 0.15 = ENGRAM_HEBB_LINK_W0). The corroborators
|
|
||||||
* are NOT linked to each other → fully independent. */
|
|
||||||
static GepNode nodes[9];
|
|
||||||
static GepEdge edges[8];
|
|
||||||
memset(nodes, 0, sizeof nodes);
|
|
||||||
nodes[0].id = "conjecture"; nodes[0].is_belief = 1; /* bare: standing = BASE */
|
|
||||||
for (int i = 1; i <= 8; i++) {
|
|
||||||
nodes[i].id = "corroborator";
|
|
||||||
seed_grounded(&nodes[i], 0.80, T0); /* standing ≈ 0.90 → grounded core */
|
|
||||||
}
|
|
||||||
GepGraph g = { nodes, 9, edges, 0 };
|
|
||||||
|
|
||||||
printf(" seed: conjecture has NO grounding events; corroborators pre-grounded.\n");
|
|
||||||
print_node("conjecture", &nodes[0], T0);
|
|
||||||
|
|
||||||
/* Beat 1: 3 independent corroborators have grounded up around the conjecture. */
|
|
||||||
g.n_edges = 0;
|
|
||||||
for (int i = 1; i <= 3; i++)
|
|
||||||
edges[g.n_edges++] = (GepEdge){ 0, i, 0.15, +1 };
|
|
||||||
beat_and_report(&g, 0, T0, 1, "3 independent grounded corroborators appear");
|
|
||||||
|
|
||||||
/* Beat 2: the neighborhood fills in — 5 independent corroborators now. */
|
|
||||||
g.n_edges = 0;
|
|
||||||
for (int i = 1; i <= 5; i++)
|
|
||||||
edges[g.n_edges++] = (GepEdge){ 0, i, 0.15, +1 };
|
|
||||||
beat_and_report(&g, 0, T0 + BEAT_MS, 2, "neighborhood grows to 5 corroborators");
|
|
||||||
|
|
||||||
/* Beat 3: support sustained at 5 (grounding refreshed). */
|
|
||||||
beat_and_report(&g, 0, T0 + 2 * BEAT_MS, 3, "support sustained (5)");
|
|
||||||
|
|
||||||
/* Beats 4-6: corroboration REMOVED (neighbors superseded / no longer ground).
|
|
||||||
* No new events; the collection ages → standing relaxes. Nothing is settled. */
|
|
||||||
g.n_edges = 0;
|
|
||||||
beat_and_report(&g, 0, T0 + 12 * BEAT_MS, 4, "corroboration withdrawn (+10min)");
|
|
||||||
beat_and_report(&g, 0, T0 + 60 * BEAT_MS, 5, "still withdrawn (+1h)");
|
|
||||||
beat_and_report(&g, 0, T0 + 240 * BEAT_MS, 6, "still withdrawn (+4h)");
|
|
||||||
printf(" RESULT: grounding grew automatically past threshold and graduated,\n"
|
|
||||||
" then relaxed once the independent support stopped — living,\n"
|
|
||||||
" not a latched flag.\n");
|
|
||||||
}
|
|
||||||
|
|
||||||
/* ── Scenario B — DECAY via accreting contradiction ─────────────────────── */
|
|
||||||
static void scenario_B(void) {
|
|
||||||
printf("\n=== SCENARIO B — DECAY: convergent independent CONTRADICTION ===\n");
|
|
||||||
static GepNode nodes[6];
|
|
||||||
static GepEdge edges[5];
|
|
||||||
memset(nodes, 0, sizeof nodes);
|
|
||||||
nodes[0].id = "belief"; nodes[0].is_belief = 1;
|
|
||||||
/* Seed the belief as already GROUNDED via a strong prior LTP event. */
|
|
||||||
seed_grounded(&nodes[0], 0.85, T0);
|
|
||||||
for (int i = 1; i <= 5; i++) {
|
|
||||||
nodes[i].id = "refuter";
|
|
||||||
seed_grounded(&nodes[i], 0.80, T0); /* grounded contradictors */
|
|
||||||
}
|
|
||||||
GepGraph g = { nodes, 6, edges, 0 };
|
|
||||||
|
|
||||||
printf(" seed: belief pre-grounded by a strong prior LTP event.\n");
|
|
||||||
print_node("belief", &nodes[0], T0);
|
|
||||||
|
|
||||||
/* Contradiction accretes over successive beats: 3 then 5 independent grounded
|
|
||||||
* refuters (polarity -1). Each beat past threshold appends an LTD event.
|
|
||||||
* Beat 1 runs at the seed instant so the trajectory starts from grounded. */
|
|
||||||
g.n_edges = 0;
|
|
||||||
for (int i = 1; i <= 3; i++) edges[g.n_edges++] = (GepEdge){ 0, i, 0.20, -1 };
|
|
||||||
beat_and_report(&g, 0, T0, 1, "3 independent contradictions");
|
|
||||||
|
|
||||||
g.n_edges = 0;
|
|
||||||
for (int i = 1; i <= 5; i++) edges[g.n_edges++] = (GepEdge){ 0, i, 0.20, -1 };
|
|
||||||
beat_and_report(&g, 0, T0 + BEAT_MS, 2, "contradiction broadens to 5");
|
|
||||||
beat_and_report(&g, 0, T0 + 2 * BEAT_MS, 3, "contradiction sustained (5)");
|
|
||||||
beat_and_report(&g, 0, T0 + 3 * BEAT_MS, 4, "contradiction sustained (5)");
|
|
||||||
printf(" RESULT: grounding decayed grounded->likely->conjecture under\n"
|
|
||||||
" convergent independent contradiction. The door never shut\n"
|
|
||||||
" on the belief; its history is retained (events keep growing).\n");
|
|
||||||
}
|
|
||||||
|
|
||||||
/* ── Scenario C — INDEPENDENCE GUARD ─────────────────────────────────────── */
|
|
||||||
static void scenario_C(void) {
|
|
||||||
printf("\n=== SCENARIO C — INDEPENDENCE GUARD (the load-bearing property) ===\n");
|
|
||||||
printf(" Both sub-cases: N=5 corroborators, edge weight 0.30, corroborator\n"
|
|
||||||
" standing ~0.90. ONLY difference: whether the 5 are independent.\n");
|
|
||||||
|
|
||||||
/* C1 — 5 DISTINCT INDEPENDENT corroborators (no edges among them). */
|
|
||||||
{
|
|
||||||
printf("\n -- C1: 5 DISTINCT independent corroborators --\n");
|
|
||||||
static GepNode nodes[6];
|
|
||||||
static GepEdge edges[5];
|
|
||||||
memset(nodes, 0, sizeof nodes);
|
|
||||||
nodes[0].id = "conjecture"; nodes[0].is_belief = 1;
|
|
||||||
for (int i = 1; i <= 5; i++) { nodes[i].id = "corr"; seed_grounded(&nodes[i], 0.80, T0); }
|
|
||||||
for (int i = 1; i <= 5; i++) edges[i-1] = (GepEdge){ 0, i, 0.30, +1 };
|
|
||||||
GepGraph g = { nodes, 6, edges, 5 };
|
|
||||||
print_node("conjecture", &nodes[0], T0);
|
|
||||||
beat_and_report(&g, 0, T0, 1, "5 independent corroborators (no inter-links)");
|
|
||||||
}
|
|
||||||
|
|
||||||
/* C2 — the SAME support echoed: 5 corroborators that are all mutually linked
|
|
||||||
* (a derivation clique — one source echoed through the chain). Same fan-in to
|
|
||||||
* the conjecture, same weights, same standings. Union-find collapses them to
|
|
||||||
* ONE independent component → below N_MIN → NO strengthening. */
|
|
||||||
{
|
|
||||||
printf("\n -- C2: 5 corroborators, but mutually-linked (echo of ONE source) --\n");
|
|
||||||
static GepNode nodes[6];
|
|
||||||
static GepEdge edges[9]; /* 5 to conjecture + 4 chaining corr1..corr5 */
|
|
||||||
memset(nodes, 0, sizeof nodes);
|
|
||||||
nodes[0].id = "conjecture"; nodes[0].is_belief = 1;
|
|
||||||
for (int i = 1; i <= 5; i++) { nodes[i].id = "corr"; seed_grounded(&nodes[i], 0.80, T0); }
|
|
||||||
int ne = 0;
|
|
||||||
for (int i = 1; i <= 5; i++) edges[ne++] = (GepEdge){ 0, i, 0.30, +1 };
|
|
||||||
/* chain corr1-corr2-corr3-corr4-corr5: they are the same source echoed */
|
|
||||||
for (int i = 1; i <= 4; i++) edges[ne++] = (GepEdge){ i, i+1, 0.30, +1 };
|
|
||||||
GepGraph g = { nodes, 6, edges, ne };
|
|
||||||
print_node("conjecture", &nodes[0], T0);
|
|
||||||
beat_and_report(&g, 0, T0, 1, "5 echoed (mutually-linked) corroborators");
|
|
||||||
}
|
|
||||||
|
|
||||||
/* C3 — degenerate echo: literally ONE corroborator reached by 5 parallel edges. */
|
|
||||||
{
|
|
||||||
printf("\n -- C3: ONE corroborator, reached by 5 parallel edges --\n");
|
|
||||||
static GepNode nodes[2];
|
|
||||||
static GepEdge edges[5];
|
|
||||||
memset(nodes, 0, sizeof nodes);
|
|
||||||
nodes[0].id = "conjecture"; nodes[0].is_belief = 1;
|
|
||||||
nodes[1].id = "corr"; seed_grounded(&nodes[1], 0.80, T0);
|
|
||||||
for (int i = 0; i < 5; i++) edges[i] = (GepEdge){ 0, 1, 0.30, +1 };
|
|
||||||
GepGraph g = { nodes, 2, edges, 5 };
|
|
||||||
print_node("conjecture", &nodes[0], T0);
|
|
||||||
beat_and_report(&g, 0, T0, 1, "same node, 5 parallel edges");
|
|
||||||
}
|
|
||||||
|
|
||||||
printf("\n RESULT: identical raw fan-in (5) and mass inputs; C1 grounds because\n"
|
|
||||||
" the corroboration is INDEPENDENT (5 components), C2/C3 do not\n"
|
|
||||||
" because it collapses to ONE source. Circular self-reinforcement\n"
|
|
||||||
" cannot manufacture grounding.\n");
|
|
||||||
}
|
|
||||||
|
|
||||||
int main(void) {
|
|
||||||
printf("GROUNDED EDGE-PROPAGATION — PROOF LEDGER (task #50)\n");
|
|
||||||
printf("constants: BASE=%.2f LIKELY_MIN=%.2f GROUNDED_MIN=%.2f "
|
|
||||||
"N_MIN=%d THETA=%.2f D=%.1f\n",
|
|
||||||
(double)GEP_BASE, (double)GEP_LIKELY_MIN, (double)GEP_GROUNDED_MIN,
|
|
||||||
GEP_N_MIN, (double)GEP_THETA, (double)GEP_DECAY_D);
|
|
||||||
scenario_A();
|
|
||||||
scenario_B();
|
|
||||||
scenario_C();
|
|
||||||
printf("\nDONE.\n");
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
// ─────────────────────────────────────────────────────────────────────────
|
|
||||||
// server.route.patch.el — GATED route for task #50, for engram/src/server.el.
|
|
||||||
// NOT APPLIED. Exposes the engram_ground_propagate native over HTTP so the
|
|
||||||
// soul's consolidation beat can fire one grounded edge-propagation pass.
|
|
||||||
// ─────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
// [1] New handler — add beside route_strengthen (server.el ~line 194).
|
|
||||||
// Mutation (appends grounding events, updates confidence), so it is gated
|
|
||||||
// on _auth via check_auth_ok, exactly like /api/edges. Persists once after
|
|
||||||
// the beat — the whole point of running propagation as one batched beat
|
|
||||||
// rather than per-node is to pay the snapshot cost a single time.
|
|
||||||
fn route_ground_propagate(method: String, path: String, body: String) -> String {
|
|
||||||
if !check_auth_ok(method, body) { return err_json("unauthorized") }
|
|
||||||
let tel: String = engram_ground_propagate() // native — one beat over the store
|
|
||||||
let saved: Int = persist_canonical()
|
|
||||||
return tel // gep_* telemetry JSON straight through
|
|
||||||
}
|
|
||||||
|
|
||||||
// [2] Dispatch — register in handle_request (server.el ~line 461, next to the
|
|
||||||
// /api/strengthen arm):
|
|
||||||
//
|
|
||||||
// if str_eq(method, "POST") && (str_eq(clean, "/api/ground/propagate")) {
|
|
||||||
// return route_ground_propagate(method, clean, body)
|
|
||||||
// }
|
|
||||||
//
|
|
||||||
// [3] Native declaration — engram_ground_propagate must be declared as an
|
|
||||||
// extern runtime builtin (el_runtime.h) and seed-wrapped (el_seed.c /
|
|
||||||
// el_seed.h __engram_ground_propagate) so the EL side can call it, same as
|
|
||||||
// engram_strengthen / engram_hebb_drain_json.
|
|
||||||
+23
-1221
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user