neuron-technologies/el
3
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases 5 Wiki Activity

organ: the rest of the peripheral moves into El
El SDK CI - dev / build-and-test (pull_request) Failing after 4m18s
Details

Browse Source 
The speaker and the voice-fetch landed in the previous commit. This is the
remainder of the 939-line Swift program, ported, and the line it draws is
between DEVICE and ARITHMETIC rather than between languages.

Two things stay realizers, because they are the two things El cannot express
as arithmetic: handing a buffer to the DAC and waiting for it to drain
(el_audio_darwin.m), and asking the OS for samples off a mic or frames off a
camera (el_capture_darwin.m). Both are their own translation units declared
in el_runtime.h, never patches to el_runtime.c.

Everything else is El. WAV decode, LPC autocorrelation, Levinson-Durbin at
order 16, formant extraction off the all-pole envelope, source-filter
resynthesis, and the three descriptors are organ_dsp.el. Consent, disclosure
and the scene descriptor are organ.el. Barge-in, yield-or-hold, backchannel
and resume are organ_converse.el.

The organ never learns a word. Codes and phoneme geometry arrive from the
language side; the organ turns them into samples and gets the samples out the
speaker, and runs the same trip in reverse for the senses. No lexicon, no
grapheme-to-phoneme, by design.

Barge-in needed pause/resume and a real DAC position rather than a tick
counter, because "finish the buffer" is not barge-in and a queue holding
three buffers is a third of a second wrong about where it is. An injected
barge also had to fire once rather than stay true, which is otherwise a
livelock the moment a backchannel resumes.

Measured against the Swift on out/mic_room.wav: seconds, rms, peak, zcr,
centroid and F0 agree to every printed digit; formants F1-F5 and bandwidths
B1-B5 are identical. imitate cannot match bit-for-bit because the Swift
excites unvoiced frames with Double.random — two Swift runs correlate 0.957
with each other and El correlates 0.958 with Swift, so the port is as close
to the original as the original is to itself.

Verified end to end: consent fails closed on both locks, real mic capture
(16000 frames), real camera frame (1920x1080 -> 15 numbers), voiceprint,
imitate, hear-imitate, a voice learned by ear and fetched back out of the
engram, and all five converse paths with real audio. The binary contains
zero afplay/Swift strings and spawns no child process while speaking.
This commit is contained in:
Neuron
2026-08-16 16:42:41 -05:00
parent 5503e1d9a4
commit c26b6aac82
10 changed files with 2530 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
peripheral/README.md
+165 -61
View File
@@ -1,80 +1,184 @@
# peripheral — Neuron's I/O organ (own-core, local, consent-gated)
# peripheral — Neuron's I/O organ, in El
The interface made physical. Two afferent senses in, one efferent voice out —
all reached the way the agentic surface reaches any tool.
**El speaks.** The engram stores geometry and does not speak; the speaking
belongs to the language and its runtime.
Until this landed, the organ was a 939-line Swift program (`src/periph.swift`)
that shelled out to `afplay`. Neuron's mouth and ears were a separate binary
standing next to the language, and "speak" meant "ask that binary to speak."
That program is now **reference material, not the implementation.**
```
MIC (hear) afferent device -> capture -> descriptor -> ingest -> geometry
CAMERA (see) afferent device -> capture -> descriptor -> ingest -> scene-geometry
SPEAKER(speak) efferent render WAV -> PLAY ALOUD out the speaker
SPEAKER (speak) efferent samples ──────────────► CoreAudio ──► the room
MIC (hear) afferent device ──► samples ──► descriptor ──► engram
CAMERA (see) afferent device ──► frame ──► descriptor ──► engram
```
Closes the conversational loop: **hear (mic) -> understand (engram) -> speak (speaker)**.
## The split, and why it falls where it does
Exactly **two** things here are not El, and they are the two things El cannot
express as arithmetic:
| Not El (realizers) | Why |
|---|---|
| `lang/runtime/el_audio_darwin.m` | Handing a buffer to the DAC and waiting for it to drain. There is no way to say "the hardware has now played these samples" in El, and there should not be. |
| `lang/runtime/el_capture_darwin.m` | Asking the OS for samples off a microphone or frames off a camera, plus the TCC permission dance. |
**Everything else is El**, because everything else is arithmetic:
| In El | Where |
|---|---|
| WAV encode / decode (chunk-walking, JUNK/FLLR tolerant) | `src/organ_dsp.el`, `elp/src/speech.el` |
| LPC autocorrelation + Levinson-Durbin (order 16 @ 16 kHz) | `src/organ_dsp.el` |
| Formant extraction off the all-pole spectral envelope | `src/organ_dsp.el` |
| Source-filter resynthesis (glottal impulse train through the filter) | `src/organ_dsp.el` |
| Audio descriptor `[seconds, sr, ch, rms, peak, zcr, centroid, F0]` | `src/organ_dsp.el` |
| Voice descriptor `[F0, F1..F5, bandwidths]` | `src/organ_dsp.el` |
| Scene descriptor `[w, h, meanRGB, brightness, 3×3 luminance grid]` | `src/organ.el` |
| Consent, disclosure, the voice-from-engram fetch | `src/organ.el` |
| Barge-in, yield-or-hold, backchannel, resume | `src/organ_converse.el` |
| The command surface | `src/organ_cli.el` |
Both realizers are their **own translation units**, declared in
`lang/runtime/el_runtime.h`, and deliberately **not** patches to
`el_runtime.c`. Acquiring a device must not mean editing the middle of the
language — the same rule the realizer registry follows for modalities.
`lang/runtime/el_peripheral_null.c` provides the identical entry points
everywhere else, so El that speaks links on any platform and truthfully reports
having no speaker rather than going quietly silent.
## The voice comes from the engram
A voice is **geometry in the engram**, not a JSON file next to the code and
certainly not constants in a source file. The organ fetches it the way anything
retrieves a memory — it asks:
```el
let g: [Int] = organ_voice_fetch("will")
// [peripheral] VOICE: fetched 'will' FROM THE ENGRAM —
// f0=137 f0_end=116 kf=1269 f1=500 f2=2093 f3=3531
```
`organ_voice_fetch` issues an engram query and reads the geometry off the node
that comes back. Nothing opens a file. If the region is not in the graph it
returns **empty**, not a plausible default — a caller has to be able to tell
"this is how they sound" from "I never heard them."
The reverse direction is `ingest-voice`: an LPC voiceprint becomes a node, and
from then on the voice is a memory rather than a measurement someone wrote down.
## What the organ never does
**It never learns a word.** Pronunciation, vocabulary and phonemes belong to the
language faculty and are already built as ingested geometry — *the engram knows
how to pronounce*. The seam is `synth_codes(codes, voice, pmap)`: the codes and
the phoneme map arrive from the language side as geometry, and the organ's whole
job is turning them into samples and getting the samples out the speaker, plus
the same trip in reverse for the senses. There is no lexicon here and no
grapheme-to-phoneme rule, by design.
## Rails
- **Own-core.** macOS-native only: AVFoundation (camera/mic), CoreAudio voice-
processing (AEC), afplay (speaker), ImageIO/CoreGraphics (frames), hand-rolled
DSP (WAV, LPC, formant synthesis). No cloud, no heavy deps.
- **Local-only.** Raw streams are written to `out/` and never egress. `.gitignore`
keeps captured media out of git.
- **Consent-gated (two locks).** A Neuron-level grant (`grant`/`revoke`) *and* the
OS TCC permission. Sensitive senses (camera/mic) fail closed without both.
- **Disclosed.** Every device touch prints a `[peripheral]` line on stderr.
- **Own-core.** CoreAudio / AVFoundation / ImageIO — all ship with macOS. No
cloud, no model, no heavy dependency. There is **no network code in the organ
at all**, by construction.
- **Local-only.** Raw streams stay on the machine. What leaves a capture is a
descriptor of a few dozen numbers. A 1920×1080 frame becomes 15 integers
(~414,000× smaller); three seconds of audio becomes 8.
- **Consent, two locks.** A Neuron-level grant **and** the OS TCC permission.
Camera and mic **fail closed** without both. The speaker is disclosed but not
gated — you cannot secretly speak aloud, and gating it would mean Neuron needs
permission to answer.
- **Disclosed.** Every device touch prints a `[peripheral]` line on **stderr**
(via `eprintln`, flushed immediately), so a disclosure lands before the device
is touched and never contaminates the program's stdout.
## Build
```bash
./peripheral/build.sh /tmp/organ
```
swiftc -O -o bin/periph src/periph.swift \
-framework AVFoundation -framework CoreMedia -framework Foundation \
-framework CoreGraphics -framework ImageIO -framework CoreImage
```
Concatenates the El modules, compiles with `elc`, links the two realizers.
Run it **from the repo root** or the `.psv` phoneme data will not resolve.
## Commands
```
periph grant|revoke <camera|mic> # Neuron-level consent
periph status
periph speak <file.wav> # SPEAK ALOUD (efferent)
periph tone <out.wav> [hz] [sec] # own-core WAV synth
periph listen <sec> <out.wav> # MIC capture (afferent), 16k mono
periph see <out.jpg> # CAMERA one frame (afferent)
periph feat-audio <wav> | feat-image <jpg> # capture -> compact descriptor
periph ingest-audio|ingest-image <file> <engramURL> # descriptor -> engram node (geometry)
periph voiceprint <voice.wav> # extract F0 + formants F1-F5
periph imitate <voice.wav> <out.wav> # speak back in that voice (LPC resynthesis)
periph hear-imitate <sec> <out.wav> # MIC -> signature -> imitate -> SPEAK ALOUD
periph converse <manifest.json> [--authority F] [--barge-at S[:backchannel|:bargein]] [--resume] [--live-mic]
organ grant|revoke <camera|mic> Neuron-level consent
organ status consent + device state
organ speak <file.wav> play a WAV aloud (efferent)
organ tone [hz] [ms] synthesize and play — no file at all
organ say <voice> <CODE> [CODE...] fetch voice FROM THE ENGRAM, render, speak
organ listen <sec> <out.wav> mic capture 16k mono (afferent)
organ see <out.jpg> one camera frame (afferent)
organ wav-info <file.wav> WAV geometry
organ feat-audio <file.wav> compact audio descriptor (8 numbers)
organ feat-image compact scene-geometry from the camera
organ voiceprint <voice.wav> F0 + formants F1-F5 (LPC)
organ imitate <in.wav> <out.wav> LPC analysis-resynthesis
organ hear-imitate <sec> <out.wav> mic -> signature -> imitate -> speak aloud
organ ingest-audio <file.wav> descriptor -> engram node (geometry)
organ ingest-voice <voice.wav> <n> voiceprint -> engram voice region
organ converse <manifest.json> [--authority PM] [--barge-at MS[:kind]] [--live-mic] [--resume]
```
## The afferent metabolism
A capture is never shipped raw. It becomes a **compact descriptor** — the afferent
twin of the music instrument-signature:
- audio -> `[seconds, sr, ch, rms, peak, zcr, centroid, F0]` (~2400-6000x smaller)
- image -> `[w, h, meanRGB, brightness, 3x3 luminance grid]` (~400000x smaller)
- voice -> `[F0, F1..F5, bandwidths]` (11 numbers)
## Interruptibility
That descriptor is what the ingest organ (engram `POST /api/nodes`) turns into an
embedded node = geometry.
`converse` speaks an ordered, salience-tagged **meaning-plan** while listening:
## Voice by imitation
`voiceprint`/`imitate` are own-core LPC (autocorrelation + Levinson-Durbin, order
16 @ 16 kHz), formant extraction from the LPC spectral envelope, and source-filter
resynthesis (glottal impulse train at F0 through the all-pole formant filter). A
voice is grabbed by ear as ~a dozen numbers and spoken back — **no training, no
stolen voice.** Measured fidelity on real speech: resynthesized formants match the
source within 2-3%. The full phoneme->formant path for *novel* sentences is the
speech faculty's seam (`elp` audio surface profile); this engine provides the
formant synthesis primitive it renders through.
- **barge-in** — output stops at the sample, not at the end of the buffer. The
realizer exposes `pause`/`resume` and reports `played_frames` (the real DAC
position) precisely so this is possible.
- **yield-or-hold** — a decision, not a rule: `hold = salience·0.6 +
progress·0.4`, and holding also requires that the interrupter not be
high-authority. Otherwise yield, because the polite default is the right one.
- **backchannel** — "mm-hm" is brief and low-energy; resume seamlessly.
- **resumable** — on yield the remaining plan persists to `.resume.json`;
`--resume` picks the thread back up. An interruption should cost a turn, not
the content.
## Interruptibility (native turn-taking)
`converse` plays the utterance as an ordered, salience-tagged **meaning-plan**
while the mic listens (full-duplex, AEC on so it never barges in on its own voice):
- **barge-in**: user speech -> pause on the spot (sample-accurate), not "finish the buffer."
- **yield-or-hold**: a decision grounded in the current segment's salience + progress
+ the interrupter's authority — YIELD (stop) or HOLD ("hang on, let me finish").
- **backchannel** ("mm-hm"): brief/low -> keep going, resume seamlessly.
- **resumable**: on yield the remaining plan persists (`.resume.json`); `--resume`
picks the thread back up ("as I was saying").
Live full-duplex uses `--live-mic` with the OS voice-processing unit (AEC) so
Neuron does not barge in on its own voice. `--barge-at` injects the event
deterministically for testing.
Live full-duplex uses `--live-mic` (OS AEC). Injected `--barge-at` drives the
decision loop deterministically for testing.
```
```
## Measured against the Swift original
Same input (`out/mic_room.wav`, 16 kHz mono, 48121 samples), Swift `periph`
vs the El organ:
| | Swift | El |
|---|---|---|
| seconds | 3.0075625 | 3.0076 |
| rms | 0.0047766496761 | 0.004777 |
| peak | 0.01806640625 | 0.018066 |
| zcr_hz | 416.28395087 | 416.2840 |
| centroid_hz | 727.60529169 | 727.6053 |
| f0_hz | 400 | 400.0000 |
| formants F1–F5 | 1734.375 / 3343.75 / 3875 / 4359.375 / 4468.75 | identical |
| bandwidths B1–B5 | 2000 / 2968.75 / 4203.125 / 4687.5 / 5000 | identical |
Agreement to every printed digit. `imitate` cannot match bit-for-bit because the
Swift excites unvoiced frames with `Double.random` — two Swift runs correlate
0.957 with **each other**; El correlates **0.958** with Swift. The port is as
close to the original as the original is to itself, and the deterministic prefix
is bit-identical.
## Honest status
- **Works:** speaker (CoreAudio, no `afplay`, no subprocess — verified: zero
`afplay`/Swift strings in the binary, no child process during playback), mic
capture, camera capture, all descriptors, LPC voiceprint, imitate,
hear-imitate, voice fetch/ingest against the engram, converse (yield, hold,
yield-to-authority, backchannel, resume — all exercised with real audio).
- **Coarse, and labelled so:** a fetched voice is one formant triple with no
coarticulation and no prosody. It is an impression, explicitly **not a
clone**, and `prov=COARSE` says so on the node.
- **Not verified here:** live `--live-mic` barge-in in a real room with a real
interrupter. The AEC path is implemented and the deterministic path is proven;
the acoustic behaviour is not something a headless run can establish.
- **Not in the engram yet:** the structured `Voice` / `VowelTarget` geometry
nodes live in the organ's own store and in snapshot files from earlier work,
but the **production engram does not carry them**. Getting them there is an
ingest, not a code change.
- `src/periph.swift` is kept as the reference the port was measured against.