organ: el speaks — the peripheral becomes a capability of the language #159

2026-08-16T21:43:34Z

will.anderson commented

2026-08-16 21:43:34 +00:00

el speaks

The engram stores geometry and does not speak. The speaking belongs to the language and its runtime.

Before this, Neuron's mouth and ears were a 939-line Swift program (peripheral/src/periph.swift) that shelled out to afplay. The voice was not a capability of El or of Neuron — it was a separate binary standing next to them, and "speak" meant "ask that binary to speak." ls lang/runtime/*audio* returned nothing and there were zero .el files in peripheral/.

Now El has a speaker, and it fetches the voice from the engram.

The line this draws

Not between languages — between device and arithmetic.

Stayed a realizer (2 files, and only these)

File	Why it cannot be El
`lang/runtime/el_audio_darwin.m`	Handing a buffer to the DAC and waiting for it to drain. There is no way to express "the hardware has now played these samples" as arithmetic, and there should not be.
`lang/runtime/el_capture_darwin.m`	Asking the OS for samples off a microphone or frames off a camera, plus the TCC permission dance.

Both are their own translation units, declared in el_runtime.h, and deliberately not patches to el_runtime.c — acquiring a device must not mean editing the middle of the language, the same rule the realizer registry follows for modalities. el_peripheral_null.c provides identical entry points elsewhere, so El that speaks links on any platform and truthfully reports having no speaker instead of going quietly silent.

speaker_play_pcm16 takes samples straight out of memory: nothing is written to disk, and no process is spawned between the intent to speak and the sound.

Moved into El (everything else)

Ported	Where
WAV encode/decode, chunk-walking (JUNK/FLLR tolerant)	`organ_dsp.el`
LPC autocorrelation + Levinson–Durbin, order 16 @ 16 kHz	`organ_dsp.el`
Formant extraction off the all-pole spectral envelope	`organ_dsp.el`
Source-filter resynthesis (glottal impulse train through the filter)	`organ_dsp.el`
Audio descriptor `[seconds, sr, ch, rms, peak, zcr, centroid, F0]`	`organ_dsp.el`
Voice descriptor `[F0, F1..F5, bandwidths]`	`organ_dsp.el`
Scene descriptor `[w, h, meanRGB, brightness, 3×3 luma grid]`	`organ.el`
Consent (both locks), disclosure, voice fetch/ingest	`organ.el`
Barge-in, yield-or-hold, backchannel, resume	`organ_converse.el`
The full command surface	`organ_cli.el`

The voice comes from the engram

A voice is geometry in the engram, not a JSON file next to the code. The previous path parsed elp/data/will-voice.engram.json, which quietly made the voice a build artifact instead of a memory. organ_voice_fetch issues an engram query and reads the geometry off the node that comes back:

[peripheral] VOICE: fetched 'will' FROM THE ENGRAM —
             f0=137 f0_end=116 kf=1269 f1=500 f2=2093 f3=3531

Matching the 30 s LPC measurement exactly. An absent region returns empty, never a plausible default — a caller must be able to tell "this is how they sound" from "I never heard them."

The reverse direction (ingest-voice) closes the loop: a voiceprint becomes a node, and the voice is thereafter a memory rather than a measurement someone wrote down. Verified in one session: a voice learned by ear → ingested → fetched back by query → spoken.

The organ never learns a word

Pronunciation, vocabulary and phonemes belong to the language faculty and are already built as ingested geometry — the engram knows how to pronounce. The seam is synth_codes(codes, voice, pmap): codes and phoneme map arrive from the language side as geometry; the organ turns them into samples and gets the samples out the speaker. No lexicon, no grapheme-to-phoneme, nothing under elp/src/*.el touched.

Measured against the Swift original

Same input, peripheral/out/mic_room.wav (16 kHz mono, 48121 samples):

	Swift `periph`	El organ
seconds	3.0075625	3.0076
rms	0.0047766496761	0.004777
peak	0.01806640625	0.018066
zcr_hz	416.28395087	416.2840
centroid_hz	727.60529169	727.6053
f0_hz	400	400.0000
formants F1–F5	1734.375 / 3343.75 / 3875 / 4359.375 / 4468.75	identical
bandwidths B1–B5	2000 / 2968.75 / 4203.125 / 4687.5 / 5000	identical

Agreement to every printed digit.

imitate cannot match bit-for-bit because the Swift excites unvoiced frames with Double.random. Two Swift runs correlate 0.957 with each other; El correlates 0.958 with Swift, diverging at the same sample (39, the first noise-excited one) with a bit-identical deterministic prefix. The port is as close to the original as the original is to itself.

The bug underneath all of it

__str_set_char bounds-checked writes against strlen(p). For the zero-filled buffer __str_alloc hands back, strlen is 0, so idx >= len rejected every write. The function was a total no-op on the only kind of buffer it was ever used with.

Every El-authored WAV in this repo was 55,244 bytes of pure zeros — correct length, correct filename, no RIFF header, ok=true. El has been mute for its entire history and nothing said so, because nothing ever reports "there is no sound here."

Byte buffers now carry their capacity in a side table keyed by pointer; anything unregistered keeps the exact strlen behaviour it had, so text semantics are unchanged. (A length header behind the pointer was rejected: __str_set_char accepts any String including a .rodata literal, where reading the preceding bytes is undefined.)

Also adds eprintln — El could write to stdout and nowhere else, and a program that announces "I am opening the microphone" on stdout has corrupted its own output.

Verified, by running it

Speaker: strings organ | grep -ic 'afplay|swift' → 0. pgrep -P <pid> during playback → none. Only OS frameworks linked.
Voice from engram: f0=137 f0_end=116 kf=1269 f1=500 f2=2093 f3=3531, persisted across processes in a durable store, fetched by query.
Render + speak: 20160 samples @ 16 kHz, played aloud.
Consent: fails closed on the Neuron lock with the OS lock open; both disclosed. Grant → real capture of 16000 frames (peak 816, rms 178 — real audio, not zeros). Revoked afterwards.
Camera: 1920×1080 → 217 KB JPEG; descriptor → 15 integers (~414,000× smaller).
Closed loop: hear-imitate — 48000 frames heard → LPC signature → 47999 resynthesized → spoken aloud.
converse, all five paths with real audio: yield (barge at 1269 ms, persisted), resume ("as I was saying", completed), hold (salience 900, progress 802 → finished the segment then yielded), yield-to-authority (same barge, authority 900 → yielded), backchannel (paused, read as "mm-hm", resumed seamlessly, completed).

Build is ./peripheral/build.sh from the repo root. Local build is the bar; the ObjC realizers compile clean under ARC.

What still does not work

Live --live-mic barge-in in a real room is unproven. The AEC path is implemented and the deterministic path is fully exercised, but acoustic behaviour with a real interrupter is not something a headless run can establish.
A fetched voice is COARSE and labelled so — one formant triple, no coarticulation, no prosody. An impression, explicitly not a clone; prov=COARSE says so on the node.
The production engram does not carry the structured Voice/VowelTarget nodes. They live in the organ's own store and in snapshot files from earlier work. Getting them into prod is an ingest, not a code change.
imitate is not reproducible run-to-run, and neither is the Swift, for the same reason.
src/periph.swift is kept as the reference the port was measured against, not as a dependency. Nothing in the chain calls it.

Production untouched throughout: own store dir, own ports, read-only GETs only.

## el speaks The engram stores geometry and does not speak. The speaking belongs to the language and its runtime. Before this, Neuron's mouth and ears were a **939-line Swift program** (`peripheral/src/periph.swift`) that shelled out to `afplay`. The voice was not a capability of El or of Neuron — it was a separate binary standing next to them, and "speak" meant "ask that binary to speak." `ls lang/runtime/*audio*` returned nothing and there were zero `.el` files in `peripheral/`. Now El has a speaker, and it fetches the voice from the engram. --- ## The line this draws Not between languages — between **device** and **arithmetic**. ### Stayed a realizer (2 files, and only these) | File | Why it cannot be El | |---|---| | `lang/runtime/el_audio_darwin.m` | Handing a buffer to the DAC and waiting for it to drain. There is no way to express "the hardware has now played these samples" as arithmetic, and there should not be. | | `lang/runtime/el_capture_darwin.m` | Asking the OS for samples off a microphone or frames off a camera, plus the TCC permission dance. | Both are their **own translation units**, declared in `el_runtime.h`, and deliberately **not** patches to `el_runtime.c` — acquiring a device must not mean editing the middle of the language, the same rule the realizer registry follows for modalities. `el_peripheral_null.c` provides identical entry points elsewhere, so El that speaks links on any platform and *truthfully reports having no speaker* instead of going quietly silent. `speaker_play_pcm16` takes samples straight out of memory: nothing is written to disk, and no process is spawned between the intent to speak and the sound. ### Moved into El (everything else) | Ported | Where | |---|---| | WAV encode/decode, chunk-walking (JUNK/FLLR tolerant) | `organ_dsp.el` | | LPC autocorrelation + Levinson–Durbin, order 16 @ 16 kHz | `organ_dsp.el` | | Formant extraction off the all-pole spectral envelope | `organ_dsp.el` | | Source-filter resynthesis (glottal impulse train through the filter) | `organ_dsp.el` | | Audio descriptor `[seconds, sr, ch, rms, peak, zcr, centroid, F0]` | `organ_dsp.el` | | Voice descriptor `[F0, F1..F5, bandwidths]` | `organ_dsp.el` | | Scene descriptor `[w, h, meanRGB, brightness, 3×3 luma grid]` | `organ.el` | | Consent (both locks), disclosure, voice fetch/ingest | `organ.el` | | Barge-in, yield-or-hold, backchannel, resume | `organ_converse.el` | | The full command surface | `organ_cli.el` | --- ## The voice comes from the engram A voice is **geometry in the engram**, not a JSON file next to the code. The previous path parsed `elp/data/will-voice.engram.json`, which quietly made the voice a build artifact instead of a memory. `organ_voice_fetch` issues an engram query and reads the geometry off the node that comes back: ``` [peripheral] VOICE: fetched 'will' FROM THE ENGRAM — f0=137 f0_end=116 kf=1269 f1=500 f2=2093 f3=3531 ``` Matching the 30 s LPC measurement exactly. An absent region returns **empty**, never a plausible default — a caller must be able to tell "this is how they sound" from "I never heard them." The reverse direction (`ingest-voice`) closes the loop: a voiceprint becomes a node, and the voice is thereafter a memory rather than a measurement someone wrote down. Verified in one session: a voice learned by ear → ingested → fetched back by query → spoken. --- ## The organ never learns a word Pronunciation, vocabulary and phonemes belong to the language faculty and are already built as ingested geometry — *the engram knows how to pronounce.* The seam is `synth_codes(codes, voice, pmap)`: codes and phoneme map arrive from the language side as geometry; the organ turns them into samples and gets the samples out the speaker. **No lexicon, no grapheme-to-phoneme, nothing under `elp/src/*.el` touched.** --- ## Measured against the Swift original Same input, `peripheral/out/mic_room.wav` (16 kHz mono, 48121 samples): | | Swift `periph` | El organ | |---|---|---| | seconds | 3.0075625 | 3.0076 | | rms | 0.0047766496761 | 0.004777 | | peak | 0.01806640625 | 0.018066 | | zcr_hz | 416.28395087 | 416.2840 | | centroid_hz | 727.60529169 | 727.6053 | | f0_hz | 400 | 400.0000 | | formants F1–F5 | 1734.375 / 3343.75 / 3875 / 4359.375 / 4468.75 | **identical** | | bandwidths B1–B5 | 2000 / 2968.75 / 4203.125 / 4687.5 / 5000 | **identical** | Agreement to every printed digit. `imitate` cannot match bit-for-bit because the Swift excites unvoiced frames with `Double.random`. Two **Swift** runs correlate **0.957** with each other; El correlates **0.958** with Swift, diverging at the same sample (39, the first noise-excited one) with a bit-identical deterministic prefix. The port is as close to the original as the original is to itself. --- ## The bug underneath all of it `__str_set_char` bounds-checked writes against `strlen(p)`. For the zero-filled buffer `__str_alloc` hands back, `strlen` is **0**, so `idx >= len` rejected **every** write. The function was a total no-op on the only kind of buffer it was ever used with. Every El-authored WAV in this repo was **55,244 bytes of pure zeros** — correct length, correct filename, no RIFF header, `ok=true`. El has been mute for its entire history and nothing said so, because nothing ever reports "there is no sound here." Byte buffers now carry their capacity in a side table keyed by pointer; anything unregistered keeps the exact `strlen` behaviour it had, so text semantics are unchanged. (A length header behind the pointer was rejected: `__str_set_char` accepts any String including a `.rodata` literal, where reading the preceding bytes is undefined.) Also adds `eprintln` — El could write to stdout and nowhere else, and a program that announces "I am opening the microphone" on stdout has corrupted its own output. --- ## Verified, by running it - **Speaker:** `strings organ | grep -ic 'afplay|swift'` → **0**. `pgrep -P <pid>` during playback → **none**. Only OS frameworks linked. - **Voice from engram:** `f0=137 f0_end=116 kf=1269 f1=500 f2=2093 f3=3531`, persisted across processes in a durable store, fetched by query. - **Render + speak:** 20160 samples @ 16 kHz, played aloud. - **Consent:** fails closed on the Neuron lock with the OS lock open; both disclosed. Grant → real capture of **16000 frames** (peak 816, rms 178 — real audio, not zeros). Revoked afterwards. - **Camera:** 1920×1080 → 217 KB JPEG; descriptor → **15 integers** (~414,000× smaller). - **Closed loop:** `hear-imitate` — 48000 frames heard → LPC signature → 47999 resynthesized → spoken aloud. - **converse**, all five paths with real audio: yield (barge at 1269 ms, persisted), resume ("as I was saying", completed), hold (salience 900, progress 802 → finished the segment then yielded), yield-to-authority (same barge, authority 900 → yielded), backchannel (paused, read as "mm-hm", resumed seamlessly, completed). Build is `./peripheral/build.sh` from the repo root. Local build is the bar; the ObjC realizers compile clean under ARC. --- ## What still does not work - **Live `--live-mic` barge-in in a real room is unproven.** The AEC path is implemented and the deterministic path is fully exercised, but acoustic behaviour with a real interrupter is not something a headless run can establish. - **A fetched voice is COARSE and labelled so** — one formant triple, no coarticulation, no prosody. An impression, explicitly **not a clone**; `prov=COARSE` says so on the node. - **The production engram does not carry the structured `Voice`/`VowelTarget` nodes.** They live in the organ's own store and in snapshot files from earlier work. Getting them into prod is an ingest, not a code change. - **`imitate` is not reproducible run-to-run**, and neither is the Swift, for the same reason. - `src/periph.swift` is kept as the reference the port was measured against, not as a dependency. Nothing in the chain calls it. Production untouched throughout: own store dir, own ports, read-only GETs only.

will.anderson added 2 commits 2026-08-16 21:43:35 +00:00

organ: el gets a speaker, and fetches the voice from the engram 5503e1d9a4

El could turn meaning into samples and could not make a sound. Every path
from those samples to the air ran outside the language, through a 939-line
Swift program that shelled out to afplay, so the voice was not a capability
of El or of Neuron but a separate binary standing next to them.

Two things land here.

The speaker. el_audio_darwin.m is a CoreAudio AudioQueue realizer in its own
translation unit, declared in el_runtime.h, deliberately not a patch to
el_runtime.c — acquiring a device must not mean editing the middle of the
language, the same rule the realizer registry follows for modalities. It
takes samples straight out of memory, so nothing is written to disk and no
process is spawned between the intent to speak and the sound. The async half
(play/stop/playing/played_frames) exists because barge-in means stopping on
the spot, and a blocking play cannot be interrupted. el_peripheral_null.c is
the same entry points everywhere else, so El that speaks links anywhere and
truthfully reports having no speaker.

The voice. organ_voice_fetch asks the engram for a voice region by query and
reads the geometry off the node that comes back. A voice is not a JSON file
next to the code; it is a memory, and the organ retrieves it the way anything
retrieves a memory. An absent region returns empty rather than a plausible
default, because a caller must be able to tell 'this is how they sound' from
'I never heard them'.

Underneath both: __str_set_char bounds-checked writes against strlen(), which
is 0 for the zero-filled buffer __str_alloc hands back, so every write was
rejected and every El-authored WAV in this repo was 55,244 bytes of silence
that reported ok=true. Byte buffers now carry their capacity in a side table;
text keeps the exact strlen behaviour it had. This is why nobody noticed El
was mute.

Measured: voice fetched from the engram reads f0=137 f0_end=116 kf=1269
f1=500 f2=2093 f3=3531, matching the 30s LPC measurement; render is 20160
samples at 16 kHz; both the rendered utterance and an own-core tone played
aloud through CoreAudio with no Swift and no afplay in the chain.

organ: the rest of the peripheral moves into El

El SDK CI - dev / build-and-test (pull_request) Failing after 4m18s

Details

c26b6aac82

The speaker and the voice-fetch landed in the previous commit. This is the
remainder of the 939-line Swift program, ported, and the line it draws is
between DEVICE and ARITHMETIC rather than between languages.

Two things stay realizers, because they are the two things El cannot express
as arithmetic: handing a buffer to the DAC and waiting for it to drain
(el_audio_darwin.m), and asking the OS for samples off a mic or frames off a
camera (el_capture_darwin.m). Both are their own translation units declared
in el_runtime.h, never patches to el_runtime.c.

Everything else is El. WAV decode, LPC autocorrelation, Levinson-Durbin at
order 16, formant extraction off the all-pole envelope, source-filter
resynthesis, and the three descriptors are organ_dsp.el. Consent, disclosure
and the scene descriptor are organ.el. Barge-in, yield-or-hold, backchannel
and resume are organ_converse.el.

The organ never learns a word. Codes and phoneme geometry arrive from the
language side; the organ turns them into samples and gets the samples out the
speaker, and runs the same trip in reverse for the senses. No lexicon, no
grapheme-to-phoneme, by design.

Barge-in needed pause/resume and a real DAC position rather than a tick
counter, because "finish the buffer" is not barge-in and a queue holding
three buffers is a third of a second wrong about where it is. An injected
barge also had to fire once rather than stay true, which is otherwise a
livelock the moment a backchannel resumes.

Measured against the Swift on out/mic_room.wav: seconds, rms, peak, zcr,
centroid and F0 agree to every printed digit; formants F1-F5 and bandwidths
B1-B5 are identical. imitate cannot match bit-for-bit because the Swift
excites unvoiced frames with Double.random — two Swift runs correlate 0.957
with each other and El correlates 0.958 with Swift, so the port is as close
to the original as the original is to itself.

Verified end to end: consent fails closed on both locks, real mic capture
(16000 frames), real camera frame (1920x1080 -> 15 numbers), voiceprint,
imitate, hear-imitate, a voice learned by ear and fetched back out of the
engram, and all five converse paths with real audio. The binary contains
zero afplay/Swift strings and spawns no child process while speaking.

will.anderson merged commit 481badf1d1 into dev

2026-08-17 00:58:23 +00:00

will.anderson referenced this pull request

2026-08-17 01:00:27 +00:00

runtime: land the growth ratchet and engram_text extraction on dev #163

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/el#159