Neuron
c26b6aac82
The speaker and the voice-fetch landed in the previous commit. This is the remainder of the 939-line Swift program, ported, and the line it draws is between DEVICE and ARITHMETIC rather than between languages. Two things stay realizers, because they are the two things El cannot express as arithmetic: handing a buffer to the DAC and waiting for it to drain (el_audio_darwin.m), and asking the OS for samples off a mic or frames off a camera (el_capture_darwin.m). Both are their own translation units declared in el_runtime.h, never patches to el_runtime.c. Everything else is El. WAV decode, LPC autocorrelation, Levinson-Durbin at order 16, formant extraction off the all-pole envelope, source-filter resynthesis, and the three descriptors are organ_dsp.el. Consent, disclosure and the scene descriptor are organ.el. Barge-in, yield-or-hold, backchannel and resume are organ_converse.el. The organ never learns a word. Codes and phoneme geometry arrive from the language side; the organ turns them into samples and gets the samples out the speaker, and runs the same trip in reverse for the senses. No lexicon, no grapheme-to-phoneme, by design. Barge-in needed pause/resume and a real DAC position rather than a tick counter, because "finish the buffer" is not barge-in and a queue holding three buffers is a third of a second wrong about where it is. An injected barge also had to fire once rather than stay true, which is otherwise a livelock the moment a backchannel resumes. Measured against the Swift on out/mic_room.wav: seconds, rms, peak, zcr, centroid and F0 agree to every printed digit; formants F1-F5 and bandwidths B1-B5 are identical. imitate cannot match bit-for-bit because the Swift excites unvoiced frames with Double.random — two Swift runs correlate 0.957 with each other and El correlates 0.958 with Swift, so the port is as close to the original as the original is to itself. Verified end to end: consent fails closed on both locks, real mic capture (16000 frames), real camera frame (1920x1080 -> 15 numbers), voiceprint, imitate, hear-imitate, a voice learned by ear and fetched back out of the engram, and all five converse paths with real audio. The binary contains zero afplay/Swift strings and spawns no child process while speaking.
9.1 KiB
peripheral — Neuron's I/O organ, in El
El speaks. The engram stores geometry and does not speak; the speaking belongs to the language and its runtime.
Until this landed, the organ was a 939-line Swift program (src/periph.swift)
that shelled out to afplay. Neuron's mouth and ears were a separate binary
standing next to the language, and "speak" meant "ask that binary to speak."
That program is now reference material, not the implementation.
SPEAKER (speak) efferent samples ──────────────► CoreAudio ──► the room
MIC (hear) afferent device ──► samples ──► descriptor ──► engram
CAMERA (see) afferent device ──► frame ──► descriptor ──► engram
The split, and why it falls where it does
Exactly two things here are not El, and they are the two things El cannot express as arithmetic:
| Not El (realizers) | Why |
|---|---|
lang/runtime/el_audio_darwin.m |
Handing a buffer to the DAC and waiting for it to drain. There is no way to say "the hardware has now played these samples" in El, and there should not be. |
lang/runtime/el_capture_darwin.m |
Asking the OS for samples off a microphone or frames off a camera, plus the TCC permission dance. |
Everything else is El, because everything else is arithmetic:
| In El | Where |
|---|---|
| WAV encode / decode (chunk-walking, JUNK/FLLR tolerant) | src/organ_dsp.el, elp/src/speech.el |
| LPC autocorrelation + Levinson-Durbin (order 16 @ 16 kHz) | src/organ_dsp.el |
| Formant extraction off the all-pole spectral envelope | src/organ_dsp.el |
| Source-filter resynthesis (glottal impulse train through the filter) | src/organ_dsp.el |
Audio descriptor [seconds, sr, ch, rms, peak, zcr, centroid, F0] |
src/organ_dsp.el |
Voice descriptor [F0, F1..F5, bandwidths] |
src/organ_dsp.el |
Scene descriptor [w, h, meanRGB, brightness, 3×3 luminance grid] |
src/organ.el |
| Consent, disclosure, the voice-from-engram fetch | src/organ.el |
| Barge-in, yield-or-hold, backchannel, resume | src/organ_converse.el |
| The command surface | src/organ_cli.el |
Both realizers are their own translation units, declared in
lang/runtime/el_runtime.h, and deliberately not patches to
el_runtime.c. Acquiring a device must not mean editing the middle of the
language — the same rule the realizer registry follows for modalities.
lang/runtime/el_peripheral_null.c provides the identical entry points
everywhere else, so El that speaks links on any platform and truthfully reports
having no speaker rather than going quietly silent.
The voice comes from the engram
A voice is geometry in the engram, not a JSON file next to the code and certainly not constants in a source file. The organ fetches it the way anything retrieves a memory — it asks:
let g: [Int] = organ_voice_fetch("will")
// [peripheral] VOICE: fetched 'will' FROM THE ENGRAM —
// f0=137 f0_end=116 kf=1269 f1=500 f2=2093 f3=3531
organ_voice_fetch issues an engram query and reads the geometry off the node
that comes back. Nothing opens a file. If the region is not in the graph it
returns empty, not a plausible default — a caller has to be able to tell
"this is how they sound" from "I never heard them."
The reverse direction is ingest-voice: an LPC voiceprint becomes a node, and
from then on the voice is a memory rather than a measurement someone wrote down.
What the organ never does
It never learns a word. Pronunciation, vocabulary and phonemes belong to the
language faculty and are already built as ingested geometry — the engram knows
how to pronounce. The seam is synth_codes(codes, voice, pmap): the codes and
the phoneme map arrive from the language side as geometry, and the organ's whole
job is turning them into samples and getting the samples out the speaker, plus
the same trip in reverse for the senses. There is no lexicon here and no
grapheme-to-phoneme rule, by design.
Rails
- Own-core. CoreAudio / AVFoundation / ImageIO — all ship with macOS. No cloud, no model, no heavy dependency. There is no network code in the organ at all, by construction.
- Local-only. Raw streams stay on the machine. What leaves a capture is a descriptor of a few dozen numbers. A 1920×1080 frame becomes 15 integers (~414,000× smaller); three seconds of audio becomes 8.
- Consent, two locks. A Neuron-level grant and the OS TCC permission. Camera and mic fail closed without both. The speaker is disclosed but not gated — you cannot secretly speak aloud, and gating it would mean Neuron needs permission to answer.
- Disclosed. Every device touch prints a
[peripheral]line on stderr (viaeprintln, flushed immediately), so a disclosure lands before the device is touched and never contaminates the program's stdout.
Build
./peripheral/build.sh /tmp/organ
Concatenates the El modules, compiles with elc, links the two realizers.
Run it from the repo root or the .psv phoneme data will not resolve.
Commands
organ grant|revoke <camera|mic> Neuron-level consent
organ status consent + device state
organ speak <file.wav> play a WAV aloud (efferent)
organ tone [hz] [ms] synthesize and play — no file at all
organ say <voice> <CODE> [CODE...] fetch voice FROM THE ENGRAM, render, speak
organ listen <sec> <out.wav> mic capture 16k mono (afferent)
organ see <out.jpg> one camera frame (afferent)
organ wav-info <file.wav> WAV geometry
organ feat-audio <file.wav> compact audio descriptor (8 numbers)
organ feat-image compact scene-geometry from the camera
organ voiceprint <voice.wav> F0 + formants F1-F5 (LPC)
organ imitate <in.wav> <out.wav> LPC analysis-resynthesis
organ hear-imitate <sec> <out.wav> mic -> signature -> imitate -> speak aloud
organ ingest-audio <file.wav> descriptor -> engram node (geometry)
organ ingest-voice <voice.wav> <n> voiceprint -> engram voice region
organ converse <manifest.json> [--authority PM] [--barge-at MS[:kind]] [--live-mic] [--resume]
Interruptibility
converse speaks an ordered, salience-tagged meaning-plan while listening:
- barge-in — output stops at the sample, not at the end of the buffer. The
realizer exposes
pause/resumeand reportsplayed_frames(the real DAC position) precisely so this is possible. - yield-or-hold — a decision, not a rule:
hold = salience·0.6 + progress·0.4, and holding also requires that the interrupter not be high-authority. Otherwise yield, because the polite default is the right one. - backchannel — "mm-hm" is brief and low-energy; resume seamlessly.
- resumable — on yield the remaining plan persists to
.resume.json;--resumepicks the thread back up. An interruption should cost a turn, not the content.
Live full-duplex uses --live-mic with the OS voice-processing unit (AEC) so
Neuron does not barge in on its own voice. --barge-at injects the event
deterministically for testing.
Measured against the Swift original
Same input (out/mic_room.wav, 16 kHz mono, 48121 samples), Swift periph
vs the El organ:
| Swift | El | |
|---|---|---|
| seconds | 3.0075625 | 3.0076 |
| rms | 0.0047766496761 | 0.004777 |
| peak | 0.01806640625 | 0.018066 |
| zcr_hz | 416.28395087 | 416.2840 |
| centroid_hz | 727.60529169 | 727.6053 |
| f0_hz | 400 | 400.0000 |
| formants F1–F5 | 1734.375 / 3343.75 / 3875 / 4359.375 / 4468.75 | identical |
| bandwidths B1–B5 | 2000 / 2968.75 / 4203.125 / 4687.5 / 5000 | identical |
Agreement to every printed digit. imitate cannot match bit-for-bit because the
Swift excites unvoiced frames with Double.random — two Swift runs correlate
0.957 with each other; El correlates 0.958 with Swift. The port is as
close to the original as the original is to itself, and the deterministic prefix
is bit-identical.
Honest status
- Works: speaker (CoreAudio, no
afplay, no subprocess — verified: zeroafplay/Swift strings in the binary, no child process during playback), mic capture, camera capture, all descriptors, LPC voiceprint, imitate, hear-imitate, voice fetch/ingest against the engram, converse (yield, hold, yield-to-authority, backchannel, resume — all exercised with real audio). - Coarse, and labelled so: a fetched voice is one formant triple with no
coarticulation and no prosody. It is an impression, explicitly not a
clone, and
prov=COARSEsays so on the node. - Not verified here: live
--live-micbarge-in in a real room with a real interrupter. The AEC path is implemented and the deterministic path is proven; the acoustic behaviour is not something a headless run can establish. - Not in the engram yet: the structured
Voice/VowelTargetgeometry nodes live in the organ's own store and in snapshot files from earlier work, but the production engram does not carry them. Getting them there is an ingest, not a code change. src/periph.swiftis kept as the reference the port was measured against.