Neuron
c26b6aac82
El SDK CI - dev / build-and-test (pull_request) Failing after 4m18s
The speaker and the voice-fetch landed in the previous commit. This is the remainder of the 939-line Swift program, ported, and the line it draws is between DEVICE and ARITHMETIC rather than between languages. Two things stay realizers, because they are the two things El cannot express as arithmetic: handing a buffer to the DAC and waiting for it to drain (el_audio_darwin.m), and asking the OS for samples off a mic or frames off a camera (el_capture_darwin.m). Both are their own translation units declared in el_runtime.h, never patches to el_runtime.c. Everything else is El. WAV decode, LPC autocorrelation, Levinson-Durbin at order 16, formant extraction off the all-pole envelope, source-filter resynthesis, and the three descriptors are organ_dsp.el. Consent, disclosure and the scene descriptor are organ.el. Barge-in, yield-or-hold, backchannel and resume are organ_converse.el. The organ never learns a word. Codes and phoneme geometry arrive from the language side; the organ turns them into samples and gets the samples out the speaker, and runs the same trip in reverse for the senses. No lexicon, no grapheme-to-phoneme, by design. Barge-in needed pause/resume and a real DAC position rather than a tick counter, because "finish the buffer" is not barge-in and a queue holding three buffers is a third of a second wrong about where it is. An injected barge also had to fire once rather than stay true, which is otherwise a livelock the moment a backchannel resumes. Measured against the Swift on out/mic_room.wav: seconds, rms, peak, zcr, centroid and F0 agree to every printed digit; formants F1-F5 and bandwidths B1-B5 are identical. imitate cannot match bit-for-bit because the Swift excites unvoiced frames with Double.random — two Swift runs correlate 0.957 with each other and El correlates 0.958 with Swift, so the port is as close to the original as the original is to itself. Verified end to end: consent fails closed on both locks, real mic capture (16000 frames), real camera frame (1920x1080 -> 15 numbers), voiceprint, imitate, hear-imitate, a voice learned by ear and fetched back out of the engram, and all five converse paths with real audio. The binary contains zero afplay/Swift strings and spawns no child process while speaking.
455 lines
19 KiB
EmacsLisp
455 lines
19 KiB
EmacsLisp
// organ_converse.el — full-duplex, interruptible speech. The turn-taking organ.
|
|
//
|
|
// WHAT THIS IS FOR. A system that plays an utterance to completion and only
|
|
// then listens is not conversational, it is a loudspeaker with a queue. Being
|
|
// interruptible is not a feature bolted onto speech; it is most of what makes
|
|
// speech social. So the utterance is not a blob of audio — it is an ordered,
|
|
// SALIENCE-TAGGED MEANING-PLAN, and the organ speaks it while listening, decides
|
|
// what to do when interrupted, and can pick the thread back up afterwards.
|
|
//
|
|
// THREE THINGS HAVE TO BE TRUE, and each one is a place naive implementations
|
|
// go wrong:
|
|
//
|
|
// Barge-in is AT THE SAMPLE. When the mic hears speech, output stops on the
|
|
// spot — not at the end of the current buffer, not at the end of the segment.
|
|
// A listener experiences even a fifth of a second of continued talking as
|
|
// being talked over. This is why the speaker realizer has pause/resume and
|
|
// reports played_frames: "finish the buffer" is not barge-in.
|
|
//
|
|
// Yield-or-hold is a DECISION, not a rule. Stopping every time anyone makes a
|
|
// noise is its own failure — it means Neuron can never finish a sentence that
|
|
// matters. So the choice is grounded: how salient is what I am mid-saying,
|
|
// how close am I to done, and how much authority does the interrupter have.
|
|
// Holding the floor is justified when what I am saying matters AND finishing
|
|
// is cheap AND the interrupter is not high-priority. Otherwise yield, because
|
|
// the polite default is the right default.
|
|
//
|
|
// A backchannel is NOT an interruption. "mm-hm" means keep going. Treating it
|
|
// as a barge-in makes the system stop every three seconds during ordinary
|
|
// listening behaviour, which is worse than not listening at all. It is
|
|
// distinguished by being brief and low-energy: sample again shortly after
|
|
// onset, and if the speech already died away it was a backchannel.
|
|
//
|
|
// AND THE UTTERANCE SURVIVES. On yield, the remaining plan is persisted, so
|
|
// Neuron can resume — "as I was saying" — instead of losing the thought. An
|
|
// interruption should cost a turn, not the content.
|
|
//
|
|
// The AEC rail: the microphone runs with the OS voice-processing unit enabled
|
|
// so it does not hear our own speaker. Without it Neuron barges in on its own
|
|
// voice on the first syllable and the whole loop is unusable in a real room.
|
|
//
|
|
// Note what is NOT here: nothing about words. A segment carries a `text` field
|
|
// purely as a label for disclosure. The organ speaks pre-rendered audio and
|
|
// never inspects language — that is the language faculty's, and the seam holds.
|
|
|
|
// ── The meaning-plan ─────────────────────────────────────────────────────────
|
|
//
|
|
// Stored as a flat [String] with stride 3 — file, salience-per-mille, text —
|
|
// because El has no record type and parallel lists drift out of step under
|
|
// editing. Salience is an integer per-mille rather than a Float so the decision
|
|
// arithmetic stays exact and reproducible; a turn-taking decision that varies
|
|
// with floating-point rounding is not one you can debug.
|
|
|
|
fn plan_new() -> [String] {
|
|
return native_list_empty()
|
|
}
|
|
|
|
fn plan_add(plan: [String], file: String, salience_pm: Int, text: String) -> [String] {
|
|
let p: [String] = plan
|
|
p = native_list_append(p, file)
|
|
p = native_list_append(p, int_to_str(salience_pm))
|
|
p = native_list_append(p, text)
|
|
return p
|
|
}
|
|
|
|
fn plan_count(plan: [String]) -> Int {
|
|
return native_list_len(plan) / 3
|
|
}
|
|
|
|
fn plan_file(plan: [String], i: Int) -> String {
|
|
return native_list_get(plan, i * 3)
|
|
}
|
|
|
|
fn plan_salience(plan: [String], i: Int) -> Int {
|
|
return str_to_int(native_list_get(plan, i * 3 + 1))
|
|
}
|
|
|
|
fn plan_text(plan: [String], i: Int) -> String {
|
|
return native_list_get(plan, i * 3 + 2)
|
|
}
|
|
|
|
// ── Manifest ─────────────────────────────────────────────────────────────────
|
|
//
|
|
// {"utterance": "...", "segments": [{"file":..., "salience":0.9, "text":"..."}]}
|
|
// Salience arrives as a 0..1 float in the manifest and is converted once, here,
|
|
// at the edge — the same discipline the runtime uses for wire encodings.
|
|
|
|
fn conv_salience_pm(raw: String) -> Int {
|
|
// "0.85" -> 850. Parsed by hand rather than through a float so a manifest
|
|
// typo degrades to a visible number instead of a silent 0.0.
|
|
let dot: Int = str_index_of(raw, ".")
|
|
if dot < 0 {
|
|
let whole: Int = str_to_int(raw)
|
|
return whole * 1000
|
|
}
|
|
let ip: Int = str_to_int(str_slice(raw, 0, dot))
|
|
let frac: String = str_slice(raw, dot + 1, str_len(raw))
|
|
let pm: Int = 0
|
|
let scale: Int = 100
|
|
let i: Int = 0
|
|
while i < 3 {
|
|
let d: Int = 0
|
|
if i < str_len(frac) {
|
|
let c: Int = str_char_code(frac, i)
|
|
if c >= 48 {
|
|
if c <= 57 {
|
|
d = c - 48
|
|
}
|
|
}
|
|
}
|
|
pm = pm + d * scale
|
|
scale = scale / 10
|
|
i = i + 1
|
|
}
|
|
return ip * 1000 + pm
|
|
}
|
|
|
|
fn conv_load_manifest(path: String) -> [String] {
|
|
let plan: [String] = plan_new()
|
|
let raw: String = fs_read(path)
|
|
if str_eq(raw, "") {
|
|
organ_disclose("CONVERSE: cannot read manifest " + path)
|
|
return plan
|
|
}
|
|
let segs: String = json_get_raw(raw, "segments")
|
|
let n: Int = json_array_len(segs)
|
|
let i: Int = 0
|
|
while i < n {
|
|
let seg: String = json_array_get(segs, i)
|
|
let file: String = json_get_string(seg, "file")
|
|
let text: String = json_get_string(seg, "text")
|
|
let sal: String = json_get_raw(seg, "salience")
|
|
let pm: Int = conv_salience_pm(sal)
|
|
if pm <= 0 {
|
|
pm = 500
|
|
}
|
|
plan = plan_add(plan, file, pm, text)
|
|
i = i + 1
|
|
}
|
|
return plan
|
|
}
|
|
|
|
fn conv_utterance(path: String) -> String {
|
|
let raw: String = fs_read(path)
|
|
return json_get_string(raw, "utterance")
|
|
}
|
|
|
|
// ── The decision ─────────────────────────────────────────────────────────────
|
|
//
|
|
// Returns: 0 = backchannel, carry on seamlessly
|
|
// 1 = hold the floor ("hang on, let me finish this thought")
|
|
// 2 = yield (stop, let them in)
|
|
//
|
|
// All arguments are per-mille integers. Holding requires BOTH that the material
|
|
// is worth finishing AND that the interrupter is not high-authority — either
|
|
// condition alone is not enough, because "what I'm saying is important" is
|
|
// exactly the reasoning that produces a system nobody can get a word in against.
|
|
fn conv_decide(salience_pm: Int, progress_pm: Int, authority_pm: Int, is_backchannel: Bool) -> Int {
|
|
if is_backchannel {
|
|
return 0
|
|
}
|
|
let hold_score: Int = (salience_pm * 6 + progress_pm * 4) / 10
|
|
if hold_score >= 600 {
|
|
if authority_pm < 800 {
|
|
return 1
|
|
}
|
|
}
|
|
return 2
|
|
}
|
|
|
|
// ── Resume ───────────────────────────────────────────────────────────────────
|
|
//
|
|
// The remaining plan, written where a later run can find it. This is what turns
|
|
// an interruption into a pause rather than a loss.
|
|
|
|
fn conv_resume_path() -> String {
|
|
let home: String = env("PERIPH_HOME")
|
|
if str_eq(home, "") {
|
|
return "peripheral/.resume.json"
|
|
}
|
|
return home + "/.resume.json"
|
|
}
|
|
|
|
// Minimal JSON string escaping. Written here rather than reached for from the
|
|
// runtime because the organ needs exactly two escapes and no dependency: a
|
|
// segment label containing a quote or a backslash must not be able to produce a
|
|
// resume file that fails to parse and silently loses the thread.
|
|
fn conv_escape(s: String) -> String {
|
|
let n: Int = str_len(s)
|
|
let out: String = ""
|
|
let i: Int = 0
|
|
while i < n {
|
|
let c: Int = str_char_code(s, i)
|
|
if c == 34 {
|
|
out = out + "\\\""
|
|
} else {
|
|
if c == 92 {
|
|
out = out + "\\\\"
|
|
} else {
|
|
if c >= 32 {
|
|
out = out + str_slice(s, i, i + 1)
|
|
}
|
|
}
|
|
}
|
|
i = i + 1
|
|
}
|
|
return out
|
|
}
|
|
|
|
fn conv_persist_resume(plan: [String], start_at: Int, reason: String) -> Bool {
|
|
let n: Int = plan_count(plan)
|
|
let body: String = "{\"resume_from\": " + int_to_str(start_at) + ", \"reason\": \"" + reason + "\", \"segments\": ["
|
|
let i: Int = start_at
|
|
let first: Bool = true
|
|
while i < n {
|
|
if first == false {
|
|
body = body + ", "
|
|
}
|
|
body = body + "{\"file\": \"" + plan_file(plan, i) + "\", \"salience\": " + int_to_str(plan_salience(plan, i)) + ", \"text\": \"" + conv_escape(plan_text(plan, i)) + "\"}"
|
|
first = false
|
|
i = i + 1
|
|
}
|
|
body = body + "]}\n"
|
|
let ok: Bool = fs_write(conv_resume_path(), body)
|
|
organ_disclose("CONVERSE: meaning-plan persisted (" + int_to_str(n - start_at) + " segments remain) — Neuron can resume the thread.")
|
|
return ok
|
|
}
|
|
|
|
fn conv_clear_resume() -> Bool {
|
|
return fs_write(conv_resume_path(), "")
|
|
}
|
|
|
|
// Read a persisted plan back. Salience is already per-mille here (we wrote it),
|
|
// so it is NOT re-scaled — the manifest and the resume file are different
|
|
// formats on purpose, and conflating them silently divides every salience by a
|
|
// thousand.
|
|
fn conv_load_resume() -> [String] {
|
|
let plan: [String] = plan_new()
|
|
let raw: String = fs_read(conv_resume_path())
|
|
if str_eq(raw, "") {
|
|
return plan
|
|
}
|
|
let segs: String = json_get_raw(raw, "segments")
|
|
let n: Int = json_array_len(segs)
|
|
let i: Int = 0
|
|
while i < n {
|
|
let seg: String = json_array_get(segs, i)
|
|
plan = plan_add(plan, json_get_string(seg, "file"), json_get_int(seg, "salience"), json_get_string(seg, "text"))
|
|
i = i + 1
|
|
}
|
|
return plan
|
|
}
|
|
|
|
// ── The loop ─────────────────────────────────────────────────────────────────
|
|
//
|
|
// live_mic : open the microphone with AEC and let real speech drive barge-in.
|
|
// barge_ms : if >= 0, inject a barge event at that offset into the utterance
|
|
// instead. Deterministic, so the decision paths can be exercised
|
|
// without a room and a person — the same reason periph.swift has it.
|
|
// kind : "backchannel" or "bargein", for the injected case.
|
|
// authority : interrupter authority, per-mille.
|
|
//
|
|
// Returns the index the utterance stopped at, or -1 if it completed.
|
|
|
|
fn conv_run(plan: [String], authority_pm: Int, barge_ms: Int, kind: String, live_mic: Bool) -> Int {
|
|
let n: Int = plan_count(plan)
|
|
if n <= 0 {
|
|
organ_disclose("CONVERSE: nothing to say.")
|
|
return 0 - 1
|
|
}
|
|
if speaker_available() == 0 {
|
|
organ_disclose("CONVERSE: no speaker on this build — cannot hold a conversation.")
|
|
return 0 - 1
|
|
}
|
|
|
|
let mic_live: Bool = false
|
|
if live_mic {
|
|
if organ_may_listen() {
|
|
let m: Int = mic_monitor_start()
|
|
if m == 1 {
|
|
organ_disclose("CONVERSE: full-duplex — mic listening WHILE speaking, AEC on (won't self-interrupt).")
|
|
mic_live = true
|
|
}
|
|
if m == 2 {
|
|
organ_disclose("CONVERSE: full-duplex — mic listening, but AEC UNAVAILABLE; raising the VAD floor so we do not barge in on ourselves.")
|
|
mic_live = true
|
|
}
|
|
if m == 0 {
|
|
organ_disclose("CONVERSE: could not open the mic monitor — falling back to injected events.")
|
|
}
|
|
}
|
|
}
|
|
if mic_live == false {
|
|
organ_disclose("CONVERSE: deterministic mode (live mic off).")
|
|
}
|
|
|
|
// Without AEC the mic hears the speaker, so the threshold has to sit above
|
|
// our own output. This is a mitigation and not a fix: the honest note is
|
|
// that barge-in is markedly less sensitive in this mode.
|
|
let vad_pm: Int = 20
|
|
if mic_live {
|
|
if mic_monitor_start() == 2 {
|
|
vad_pm = 60
|
|
}
|
|
}
|
|
|
|
let elapsed_ms: Int = 0
|
|
let prior_ms: Int = 0
|
|
let handled: Bool = false
|
|
// An injected barge is ONE event, not a condition that stays true. Without
|
|
// this the deadline re-fires on every poll after a backchannel resume, and
|
|
// the utterance live-locks: paused, resumed, paused again, forever.
|
|
let injected_fired: Bool = false
|
|
let i: Int = 0
|
|
|
|
while i < n {
|
|
let file: String = plan_file(plan, i)
|
|
let sal: Int = plan_salience(plan, i)
|
|
let frames: Int = wav_frames(file)
|
|
let rate: Int = wav_rate(file)
|
|
if frames <= 0 {
|
|
organ_disclose("CONVERSE: missing or unreadable segment '" + file + "', skipping.")
|
|
i = i + 1
|
|
} else {
|
|
let dur_ms: Int = frames * 1000 / rate
|
|
organ_disclose("CONVERSE: speaking segment " + int_to_str(i + 1) + "/" + int_to_str(n) + " (salience " + int_to_str(sal) + "/1000) — \"" + plan_text(plan, i) + "\"")
|
|
let started: Int = speaker_play_wav_async(file)
|
|
if started == 0 {
|
|
organ_disclose("CONVERSE: could not start playback for '" + file + "'.")
|
|
i = i + 1
|
|
} else {
|
|
let seg_ms: Int = 0
|
|
let done: Bool = false
|
|
let interrupted: Bool = false
|
|
let speech_ticks: Int = 0
|
|
|
|
while done == false {
|
|
sleep_ms(10)
|
|
seg_ms = seg_ms + 10
|
|
|
|
if speaker_playing() == 0 {
|
|
done = true
|
|
} else {
|
|
// The tick counter is an approximation — each pass costs
|
|
// more than the sleep it asked for. The DAC position is
|
|
// the truth, so drive the injected deadline off THAT and
|
|
// an injected barge lands where it was asked to land.
|
|
let pos_ms: Int = speaker_played_frames() * 1000 / rate
|
|
elapsed_ms = prior_ms + pos_ms
|
|
// --- onset detection: real speech, or an injected event ---
|
|
let onset: Bool = false
|
|
if mic_live {
|
|
let rms: Float = mic_monitor_rms()
|
|
let rms_pm: Int = float_to_int(rms * 1000.0)
|
|
if rms_pm > vad_pm {
|
|
speech_ticks = speech_ticks + 1
|
|
} else {
|
|
speech_ticks = 0
|
|
}
|
|
// ~60ms of continuous voice: short enough to feel
|
|
// instant, long enough that a door closing is not a turn.
|
|
if speech_ticks >= 3 {
|
|
if handled == false {
|
|
onset = true
|
|
}
|
|
}
|
|
}
|
|
if barge_ms >= 0 {
|
|
if injected_fired == false {
|
|
if elapsed_ms >= barge_ms {
|
|
onset = true
|
|
injected_fired = true
|
|
}
|
|
}
|
|
}
|
|
|
|
if onset {
|
|
handled = true
|
|
// (1) BARGE-IN — pause on the spot.
|
|
speaker_pause()
|
|
let played: Int = speaker_played_frames()
|
|
let at_ms: Int = played * 1000 / rate
|
|
let progress_pm: Int = at_ms * 1000 / dur_ms
|
|
if progress_pm > 1000 {
|
|
progress_pm = 1000
|
|
}
|
|
organ_disclose("CONVERSE: << user speech at " + int_to_str(at_ms) + "ms into segment " + int_to_str(i + 1) + " — PAUSED instantly >>")
|
|
|
|
// (2) backchannel or real barge-in?
|
|
let is_bc: Bool = false
|
|
if barge_ms >= 0 {
|
|
if str_eq(kind, "backchannel") {
|
|
is_bc = true
|
|
}
|
|
} else {
|
|
// Live: look again ~250ms after onset. If the
|
|
// energy has already collapsed it was "mm-hm".
|
|
sleep_ms(250)
|
|
let r2: Float = mic_monitor_rms()
|
|
if float_to_int(r2 * 1000.0) < 15 {
|
|
is_bc = true
|
|
}
|
|
}
|
|
|
|
// (3) yield, hold, or carry on
|
|
let d: Int = conv_decide(sal, progress_pm, authority_pm, is_bc)
|
|
if d == 0 {
|
|
organ_disclose("CONVERSE: read as BACKCHANNEL (\"mm-hm\") — keep going, resume seamlessly.")
|
|
handled = false
|
|
speech_ticks = 0
|
|
speaker_resume()
|
|
}
|
|
if d == 1 {
|
|
organ_disclose("CONVERSE: HOLD the floor — \"hang on, let me finish this thought.\" (salience " + int_to_str(sal) + ", progress " + int_to_str(progress_pm) + ")")
|
|
speaker_resume()
|
|
// Finish THIS segment, then yield the remainder:
|
|
// holding is a request for a moment, not a claim
|
|
// on the rest of the conversation.
|
|
while speaker_playing() == 1 {
|
|
sleep_ms(20)
|
|
}
|
|
speaker_stop()
|
|
conv_persist_resume(plan, i + 1, "held-then-yield")
|
|
if mic_live {
|
|
mic_monitor_stop()
|
|
}
|
|
return i + 1
|
|
}
|
|
if d == 2 {
|
|
organ_disclose("CONVERSE: YIELD — stop, let them in. Remembering where I was (resumable).")
|
|
speaker_stop()
|
|
conv_persist_resume(plan, i, "yield")
|
|
if mic_live {
|
|
mic_monitor_stop()
|
|
}
|
|
return i
|
|
}
|
|
}
|
|
}
|
|
}
|
|
if interrupted == false {
|
|
prior_ms = prior_ms + dur_ms
|
|
i = i + 1
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
conv_clear_resume()
|
|
organ_disclose("CONVERSE: utterance complete (uninterrupted).")
|
|
if mic_live {
|
|
mic_monitor_stop()
|
|
}
|
|
return 0 - 1
|
|
}
|