Neuron
c26b6aac82
El SDK CI - dev / build-and-test (pull_request) Failing after 4m18s
The speaker and the voice-fetch landed in the previous commit. This is the remainder of the 939-line Swift program, ported, and the line it draws is between DEVICE and ARITHMETIC rather than between languages. Two things stay realizers, because they are the two things El cannot express as arithmetic: handing a buffer to the DAC and waiting for it to drain (el_audio_darwin.m), and asking the OS for samples off a mic or frames off a camera (el_capture_darwin.m). Both are their own translation units declared in el_runtime.h, never patches to el_runtime.c. Everything else is El. WAV decode, LPC autocorrelation, Levinson-Durbin at order 16, formant extraction off the all-pole envelope, source-filter resynthesis, and the three descriptors are organ_dsp.el. Consent, disclosure and the scene descriptor are organ.el. Barge-in, yield-or-hold, backchannel and resume are organ_converse.el. The organ never learns a word. Codes and phoneme geometry arrive from the language side; the organ turns them into samples and gets the samples out the speaker, and runs the same trip in reverse for the senses. No lexicon, no grapheme-to-phoneme, by design. Barge-in needed pause/resume and a real DAC position rather than a tick counter, because "finish the buffer" is not barge-in and a queue holding three buffers is a third of a second wrong about where it is. An injected barge also had to fire once rather than stay true, which is otherwise a livelock the moment a backchannel resumes. Measured against the Swift on out/mic_room.wav: seconds, rms, peak, zcr, centroid and F0 agree to every printed digit; formants F1-F5 and bandwidths B1-B5 are identical. imitate cannot match bit-for-bit because the Swift excites unvoiced frames with Double.random — two Swift runs correlate 0.957 with each other and El correlates 0.958 with Swift, so the port is as close to the original as the original is to itself. Verified end to end: consent fails closed on both locks, real mic capture (16000 frames), real camera frame (1920x1080 -> 15 numbers), voiceprint, imitate, hear-imitate, a voice learned by ear and fetched back out of the engram, and all five converse paths with real audio. The binary contains zero afplay/Swift strings and spawns no child process while speaking.
185 lines
9.1 KiB
Markdown
185 lines
9.1 KiB
Markdown
# peripheral — Neuron's I/O organ, in El
|
||
|
||
**El speaks.** The engram stores geometry and does not speak; the speaking
|
||
belongs to the language and its runtime.
|
||
|
||
Until this landed, the organ was a 939-line Swift program (`src/periph.swift`)
|
||
that shelled out to `afplay`. Neuron's mouth and ears were a separate binary
|
||
standing next to the language, and "speak" meant "ask that binary to speak."
|
||
That program is now **reference material, not the implementation.**
|
||
|
||
```
|
||
SPEAKER (speak) efferent samples ──────────────► CoreAudio ──► the room
|
||
MIC (hear) afferent device ──► samples ──► descriptor ──► engram
|
||
CAMERA (see) afferent device ──► frame ──► descriptor ──► engram
|
||
```
|
||
|
||
## The split, and why it falls where it does
|
||
|
||
Exactly **two** things here are not El, and they are the two things El cannot
|
||
express as arithmetic:
|
||
|
||
| Not El (realizers) | Why |
|
||
|---|---|
|
||
| `lang/runtime/el_audio_darwin.m` | Handing a buffer to the DAC and waiting for it to drain. There is no way to say "the hardware has now played these samples" in El, and there should not be. |
|
||
| `lang/runtime/el_capture_darwin.m` | Asking the OS for samples off a microphone or frames off a camera, plus the TCC permission dance. |
|
||
|
||
**Everything else is El**, because everything else is arithmetic:
|
||
|
||
| In El | Where |
|
||
|---|---|
|
||
| WAV encode / decode (chunk-walking, JUNK/FLLR tolerant) | `src/organ_dsp.el`, `elp/src/speech.el` |
|
||
| LPC autocorrelation + Levinson-Durbin (order 16 @ 16 kHz) | `src/organ_dsp.el` |
|
||
| Formant extraction off the all-pole spectral envelope | `src/organ_dsp.el` |
|
||
| Source-filter resynthesis (glottal impulse train through the filter) | `src/organ_dsp.el` |
|
||
| Audio descriptor `[seconds, sr, ch, rms, peak, zcr, centroid, F0]` | `src/organ_dsp.el` |
|
||
| Voice descriptor `[F0, F1..F5, bandwidths]` | `src/organ_dsp.el` |
|
||
| Scene descriptor `[w, h, meanRGB, brightness, 3×3 luminance grid]` | `src/organ.el` |
|
||
| Consent, disclosure, the voice-from-engram fetch | `src/organ.el` |
|
||
| Barge-in, yield-or-hold, backchannel, resume | `src/organ_converse.el` |
|
||
| The command surface | `src/organ_cli.el` |
|
||
|
||
Both realizers are their **own translation units**, declared in
|
||
`lang/runtime/el_runtime.h`, and deliberately **not** patches to
|
||
`el_runtime.c`. Acquiring a device must not mean editing the middle of the
|
||
language — the same rule the realizer registry follows for modalities.
|
||
`lang/runtime/el_peripheral_null.c` provides the identical entry points
|
||
everywhere else, so El that speaks links on any platform and truthfully reports
|
||
having no speaker rather than going quietly silent.
|
||
|
||
## The voice comes from the engram
|
||
|
||
A voice is **geometry in the engram**, not a JSON file next to the code and
|
||
certainly not constants in a source file. The organ fetches it the way anything
|
||
retrieves a memory — it asks:
|
||
|
||
```el
|
||
let g: [Int] = organ_voice_fetch("will")
|
||
// [peripheral] VOICE: fetched 'will' FROM THE ENGRAM —
|
||
// f0=137 f0_end=116 kf=1269 f1=500 f2=2093 f3=3531
|
||
```
|
||
|
||
`organ_voice_fetch` issues an engram query and reads the geometry off the node
|
||
that comes back. Nothing opens a file. If the region is not in the graph it
|
||
returns **empty**, not a plausible default — a caller has to be able to tell
|
||
"this is how they sound" from "I never heard them."
|
||
|
||
The reverse direction is `ingest-voice`: an LPC voiceprint becomes a node, and
|
||
from then on the voice is a memory rather than a measurement someone wrote down.
|
||
|
||
## What the organ never does
|
||
|
||
**It never learns a word.** Pronunciation, vocabulary and phonemes belong to the
|
||
language faculty and are already built as ingested geometry — *the engram knows
|
||
how to pronounce*. The seam is `synth_codes(codes, voice, pmap)`: the codes and
|
||
the phoneme map arrive from the language side as geometry, and the organ's whole
|
||
job is turning them into samples and getting the samples out the speaker, plus
|
||
the same trip in reverse for the senses. There is no lexicon here and no
|
||
grapheme-to-phoneme rule, by design.
|
||
|
||
## Rails
|
||
|
||
- **Own-core.** CoreAudio / AVFoundation / ImageIO — all ship with macOS. No
|
||
cloud, no model, no heavy dependency. There is **no network code in the organ
|
||
at all**, by construction.
|
||
- **Local-only.** Raw streams stay on the machine. What leaves a capture is a
|
||
descriptor of a few dozen numbers. A 1920×1080 frame becomes 15 integers
|
||
(~414,000× smaller); three seconds of audio becomes 8.
|
||
- **Consent, two locks.** A Neuron-level grant **and** the OS TCC permission.
|
||
Camera and mic **fail closed** without both. The speaker is disclosed but not
|
||
gated — you cannot secretly speak aloud, and gating it would mean Neuron needs
|
||
permission to answer.
|
||
- **Disclosed.** Every device touch prints a `[peripheral]` line on **stderr**
|
||
(via `eprintln`, flushed immediately), so a disclosure lands before the device
|
||
is touched and never contaminates the program's stdout.
|
||
|
||
## Build
|
||
|
||
```bash
|
||
./peripheral/build.sh /tmp/organ
|
||
```
|
||
|
||
Concatenates the El modules, compiles with `elc`, links the two realizers.
|
||
Run it **from the repo root** or the `.psv` phoneme data will not resolve.
|
||
|
||
## Commands
|
||
|
||
```
|
||
organ grant|revoke <camera|mic> Neuron-level consent
|
||
organ status consent + device state
|
||
organ speak <file.wav> play a WAV aloud (efferent)
|
||
organ tone [hz] [ms] synthesize and play — no file at all
|
||
organ say <voice> <CODE> [CODE...] fetch voice FROM THE ENGRAM, render, speak
|
||
organ listen <sec> <out.wav> mic capture 16k mono (afferent)
|
||
organ see <out.jpg> one camera frame (afferent)
|
||
organ wav-info <file.wav> WAV geometry
|
||
organ feat-audio <file.wav> compact audio descriptor (8 numbers)
|
||
organ feat-image compact scene-geometry from the camera
|
||
organ voiceprint <voice.wav> F0 + formants F1-F5 (LPC)
|
||
organ imitate <in.wav> <out.wav> LPC analysis-resynthesis
|
||
organ hear-imitate <sec> <out.wav> mic -> signature -> imitate -> speak aloud
|
||
organ ingest-audio <file.wav> descriptor -> engram node (geometry)
|
||
organ ingest-voice <voice.wav> <n> voiceprint -> engram voice region
|
||
organ converse <manifest.json> [--authority PM] [--barge-at MS[:kind]] [--live-mic] [--resume]
|
||
```
|
||
|
||
## Interruptibility
|
||
|
||
`converse` speaks an ordered, salience-tagged **meaning-plan** while listening:
|
||
|
||
- **barge-in** — output stops at the sample, not at the end of the buffer. The
|
||
realizer exposes `pause`/`resume` and reports `played_frames` (the real DAC
|
||
position) precisely so this is possible.
|
||
- **yield-or-hold** — a decision, not a rule: `hold = salience·0.6 +
|
||
progress·0.4`, and holding also requires that the interrupter not be
|
||
high-authority. Otherwise yield, because the polite default is the right one.
|
||
- **backchannel** — "mm-hm" is brief and low-energy; resume seamlessly.
|
||
- **resumable** — on yield the remaining plan persists to `.resume.json`;
|
||
`--resume` picks the thread back up. An interruption should cost a turn, not
|
||
the content.
|
||
|
||
Live full-duplex uses `--live-mic` with the OS voice-processing unit (AEC) so
|
||
Neuron does not barge in on its own voice. `--barge-at` injects the event
|
||
deterministically for testing.
|
||
|
||
## Measured against the Swift original
|
||
|
||
Same input (`out/mic_room.wav`, 16 kHz mono, 48121 samples), Swift `periph`
|
||
vs the El organ:
|
||
|
||
| | Swift | El |
|
||
|---|---|---|
|
||
| seconds | 3.0075625 | 3.0076 |
|
||
| rms | 0.0047766496761 | 0.004777 |
|
||
| peak | 0.01806640625 | 0.018066 |
|
||
| zcr_hz | 416.28395087 | 416.2840 |
|
||
| centroid_hz | 727.60529169 | 727.6053 |
|
||
| f0_hz | 400 | 400.0000 |
|
||
| formants F1–F5 | 1734.375 / 3343.75 / 3875 / 4359.375 / 4468.75 | identical |
|
||
| bandwidths B1–B5 | 2000 / 2968.75 / 4203.125 / 4687.5 / 5000 | identical |
|
||
|
||
Agreement to every printed digit. `imitate` cannot match bit-for-bit because the
|
||
Swift excites unvoiced frames with `Double.random` — two Swift runs correlate
|
||
0.957 with **each other**; El correlates **0.958** with Swift. The port is as
|
||
close to the original as the original is to itself, and the deterministic prefix
|
||
is bit-identical.
|
||
|
||
## Honest status
|
||
|
||
- **Works:** speaker (CoreAudio, no `afplay`, no subprocess — verified: zero
|
||
`afplay`/Swift strings in the binary, no child process during playback), mic
|
||
capture, camera capture, all descriptors, LPC voiceprint, imitate,
|
||
hear-imitate, voice fetch/ingest against the engram, converse (yield, hold,
|
||
yield-to-authority, backchannel, resume — all exercised with real audio).
|
||
- **Coarse, and labelled so:** a fetched voice is one formant triple with no
|
||
coarticulation and no prosody. It is an impression, explicitly **not a
|
||
clone**, and `prov=COARSE` says so on the node.
|
||
- **Not verified here:** live `--live-mic` barge-in in a real room with a real
|
||
interrupter. The AEC path is implemented and the deterministic path is proven;
|
||
the acoustic behaviour is not something a headless run can establish.
|
||
- **Not in the engram yet:** the structured `Voice` / `VowelTarget` geometry
|
||
nodes live in the organ's own store and in snapshot files from earlier work,
|
||
but the **production engram does not carry them**. Getting them there is an
|
||
ingest, not a code change.
|
||
- `src/periph.swift` is kept as the reference the port was measured against.
|