Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 bigmerge
|
6f3d692784 |
@@ -1,91 +0,0 @@
|
||||
> **STATUS: STAGING / PROOF-OF-SHAPE — not the deliverable.** This Python package
|
||||
> proved the architecture end-to-end against the proven realizer faculty (faithful
|
||||
> md/docx/midi from real geometry: 0 ungrounded claims, SACRED polarity). Per Will's
|
||||
> steer, the DELIVERABLE is NATIVE: the seam lives on the existing EL realizer as
|
||||
> **surface-as-profile** — see `../src/surface-profile.el` and
|
||||
> `../tests/examples/surface-profile-demo.el` (compiles + runs through elc → C →
|
||||
> binary). The concepts below (one geometry-carrying frame; surface = a pluggable
|
||||
> profile; plan/realize; deterministic-from-meaning) are exactly what the native
|
||||
> module implements. Keep this package as the validated proof; build native.
|
||||
|
||||
# Efferent Multimodal Projector
|
||||
|
||||
**geometry → any surface, faithfully.** Neuron's own document-generation faculty:
|
||||
the efferent twin of the ingest organ. Ingest is afferent (world → geometry);
|
||||
this is efferent (geometry → an arbitrary-format document / any modality).
|
||||
|
||||
Built against the **proven** realizer faculty (neuron-talk sidecar `:8756`,
|
||||
artifact `art-7affa557`). The live soul (`:8742` / `:7770`) is contacted **only**
|
||||
through the read-only, GET-only `engram_client` — never mutated.
|
||||
|
||||
## The pipeline (surface-agnostic)
|
||||
|
||||
```
|
||||
geometry region + surface/format spec
|
||||
→ PLAN (manifold → document skeleton/DAG; the geometry IS the outline) plan.py
|
||||
→ REALIZE (proven realizer, scaled sentence → passage, each section faithful) realize.py
|
||||
→ COHERE (document-level flow / transitions, not stitched sentences) cohere.py
|
||||
→ EMIT (pluggable SurfaceProjector → the target surface) projectors/
|
||||
```
|
||||
|
||||
**The surface is a PARAMETER.** `pipeline.build_ir(...)` builds ONE
|
||||
surface-neutral `DocumentIR` (`document_ir.py`); `pipeline.emit(doc, surface)`
|
||||
projects it to whichever surface you name. Markdown, docx, and MIDI are the same
|
||||
IR emitted three ways.
|
||||
|
||||
## The pivot: a geometry-carrying IR
|
||||
|
||||
`DocumentIR` is **not** a text tree. Every `Block` carries BOTH:
|
||||
- `.sentences` — realized faithful text (what **text** projectors read),
|
||||
- `.provenance` — the source geometry: `subj_id / relation / obj / polarity /
|
||||
confidence / importance / salience / node_id` (what **music / image / video**
|
||||
projectors read).
|
||||
|
||||
That single decision is what makes the projector multimodal: text renders the
|
||||
words; music/image decode the geometry. A claim with no provenance cannot exist
|
||||
in the IR — faithfulness is structural.
|
||||
|
||||
## The one shared seam
|
||||
|
||||
`projectors/base.py` — `SurfaceProjector.project(frame: DocumentIR) -> bytes`
|
||||
(+ `surface / media_type / ext / modality / profile`). Register with
|
||||
`register()`. Adding a surface changes nothing upstream.
|
||||
|
||||
`TwoStageProjector` blesses the peer plan/realize decomposition:
|
||||
`spec = plan(frame)`, `bytes = realize(spec)`, `project = realize∘plan`; the
|
||||
`profile` is the pluggable per-surface knob (text lang-profile, music
|
||||
instr/mode-profile). `projectors/midi.py` is the reference two-stage impl.
|
||||
|
||||
## Surfaces
|
||||
|
||||
| surface | modality | status | emitter |
|
||||
|---|---|---|---|
|
||||
| `markdown` | text | landed | own (str) |
|
||||
| `docx` | text | landed | own minimal OOXML (stdlib `zipfile`+XML, no lib) |
|
||||
| `midi` | audio | landed (symbolic-music proof) | own minimal SMF (stdlib `struct`, no lib) |
|
||||
| `audio` (WAV) | audio | peer agent (additive synth) | conforms to `TwoStageProjector` |
|
||||
| `image` | image | documented seam | `projectors/seams.py` |
|
||||
| `video` | video | documented seam (image×sound×time) | `projectors/seams.py` |
|
||||
|
||||
Music maps: relation → scale degree (same relation → same pitch), **polarity →
|
||||
major/minor third (SACRED negation is audible)**, confidence → duration,
|
||||
importance → velocity, section → register. Deterministic projection from meaning
|
||||
— nothing invented.
|
||||
|
||||
## Faithfulness
|
||||
|
||||
`provenance.py` audits the IR: **zero** ungrounded claims, SACRED polarity
|
||||
preserved (negations reported, never dropped), COHERE introduces no new geometry
|
||||
(connectives are marked). `trace_table()` emits the geometry → section → claim
|
||||
table.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
PY=~/Desktop/lang-realizers/venv/bin/python
|
||||
PYTHONPATH=~/Desktop/neuron-talk:~/Desktop/lang-realizers $PY generate.py
|
||||
# writes ./out/{neuron-self,engram-temporal}.{md,docx,mid} + *.audit.json + *.provenance.md
|
||||
```
|
||||
|
||||
Requires the proven realizer env (spaCy + the neuron-talk/lang-realizers engine)
|
||||
and the read-only engram at `:8742`.
|
||||
@@ -1,79 +0,0 @@
|
||||
"""cohere.py — COHERE stage: document-level flow, not stitched sentences.
|
||||
|
||||
Fidelity is REALIZE's job; FLOW is this stage's. The hard part beyond sentence
|
||||
fidelity is that a document must read as one thing. We add connective tissue at
|
||||
the passage level:
|
||||
|
||||
* an opening abstract that names what the document covers (built ONLY from the
|
||||
section headings that already exist — it introduces no new claim),
|
||||
* a short transition lead into each section after the first, drawn from a
|
||||
fixed set of discourse connectives ("Beyond that,", "Relatedly,", ...) that
|
||||
carry no propositional content,
|
||||
* ordering so the highest-grounded section leads.
|
||||
|
||||
CRITICAL: every connective is marked ``kind="connective"`` in its provenance, so
|
||||
the faithfulness audit can prove COHERE introduced ZERO new geometry claims. A
|
||||
transition is discourse glue, never a fact.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from document_ir import Block, DocumentIR, Provenance
|
||||
|
||||
# discourse connectives — pure flow, no propositional content
|
||||
_TRANSITIONS = [
|
||||
"Beyond that,", "Relatedly,", "In the same region,", "From there,",
|
||||
"Alongside this,", "Further,", "Turning to the next facet,",
|
||||
]
|
||||
|
||||
|
||||
def _connective_prov() -> Provenance:
|
||||
return Provenance(subj_id=None, subject=None, relation="", obj=None,
|
||||
polarity="aff", confidence=1.0, node_id=None,
|
||||
kind="connective")
|
||||
|
||||
|
||||
def _abstract_block(doc: DocumentIR) -> Block:
|
||||
"""A grounded opening: names the sections, asserts nothing new."""
|
||||
headings = [s.heading for s in doc.sections]
|
||||
if not headings:
|
||||
return Block(role="lead")
|
||||
if len(headings) == 1:
|
||||
body = f"This document, generated from Neuron's geometry, covers {headings[0]}."
|
||||
else:
|
||||
listed = ", ".join(headings[:-1]) + f", and {headings[-1]}"
|
||||
body = ("This document is projected directly from Neuron's meaning-geometry. "
|
||||
f"It traces {listed}.")
|
||||
b = Block(role="lead")
|
||||
b.sentences.append(body)
|
||||
b.provenance.append(_connective_prov())
|
||||
return b
|
||||
|
||||
|
||||
def cohere_document(doc: DocumentIR, *, add_abstract: bool = True,
|
||||
add_transitions: bool = True) -> DocumentIR:
|
||||
"""Order sections by grounding, add abstract + transitions (flow only)."""
|
||||
# order: strongest-grounded section (mean confidence x #claims) first,
|
||||
# but keep an explicitly-first section if the plan pinned one via level 1.
|
||||
def _score(sec):
|
||||
provs = [p for p in sec.all_provenance() if p.kind == "fact"]
|
||||
if not provs:
|
||||
return 0.0
|
||||
mean_conf = sum(p.confidence for p in provs) / len(provs)
|
||||
return mean_conf * len(provs)
|
||||
|
||||
doc.sections.sort(key=_score, reverse=True)
|
||||
|
||||
if add_transitions:
|
||||
for i, sec in enumerate(doc.sections):
|
||||
if i == 0 or not sec.blocks:
|
||||
continue
|
||||
lead = _TRANSITIONS[(i - 1) % len(_TRANSITIONS)]
|
||||
first = sec.blocks[0]
|
||||
if first.sentences:
|
||||
# prepend the connective to the first sentence (flow, no new claim)
|
||||
first.sentences[0] = f"{lead} {first.sentences[0][0].lower()}{first.sentences[0][1:]}"
|
||||
|
||||
if add_abstract:
|
||||
doc.meta["abstract"] = _abstract_block(doc)
|
||||
|
||||
return doc
|
||||
@@ -1,111 +0,0 @@
|
||||
"""document_ir.py — the surface-neutral, GEOMETRY-CARRYING document intermediate.
|
||||
|
||||
This is the pivot of the whole efferent projector. A DocumentIR is NOT a text
|
||||
tree. It is a projection of a meaning-geometry region that carries, at every
|
||||
leaf, BOTH:
|
||||
|
||||
* the realized surface text (``Block.sentences``) — what a TEXT projector reads,
|
||||
* the source geometry (``Block.provenance``) — what a MUSIC / IMAGE /
|
||||
VIDEO projector reads.
|
||||
|
||||
Because the IR holds the geometry, not just the words, the SAME
|
||||
plan -> realize -> cohere pipeline drives every surface. A markdown projector
|
||||
renders the sentences; a music projector reads the provenance edges (salience,
|
||||
importance, polarity, relation) and maps them onto a symbolic-music surface;
|
||||
an image/video projector (documented seam) would read the same geometry.
|
||||
|
||||
Nothing in this module invents content. Every :class:`Provenance` points at a
|
||||
real engram node id and a real relation. That is the faithfulness contract made
|
||||
structural: a claim with no provenance cannot exist in the IR.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Provenance — the geometry an emitted claim traces to. FAITHFULNESS is here.
|
||||
# --------------------------------------------------------------------------- #
|
||||
@dataclass
|
||||
class Provenance:
|
||||
"""One geometry edge behind one realized claim.
|
||||
|
||||
``kind`` distinguishes a FACT (a structural edge asserted by the geometry,
|
||||
spoken as fact) from an INTERPRETATION (something attributed, spoken with
|
||||
attribution) — the facts-as-facts + interpretations-attributed discipline
|
||||
(memory 80927e26). ``polarity`` is SACRED: a negated edge stays negated.
|
||||
"""
|
||||
subj_id: str | None # source engram node id of the subject
|
||||
subject: str | None # normalized subject surface
|
||||
relation: str # predicate lemma (e.g. "use", "contain", "be")
|
||||
obj: str | None # normalized object / complement surface
|
||||
polarity: str = "aff" # "aff" | "neg" (SACRED — never silently flipped)
|
||||
confidence: float = 0.0 # extraction confidence in [0,1]
|
||||
node_id: str | None = None # engram node the claim was extracted from
|
||||
kind: str = "fact" # "fact" | "interpretation"
|
||||
importance: float = 0.0 # source node importance (drives music/emphasis)
|
||||
salience: float = 0.0 # source node salience
|
||||
|
||||
def trace(self) -> str:
|
||||
arrow = "-->" if self.polarity == "aff" else "--NOT-->"
|
||||
return (f"[{(self.node_id or '?')[:8]}] {self.subject!r} {arrow}"
|
||||
f"{self.relation} {self.obj!r} (conf {self.confidence:.2f})")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Block:
|
||||
"""A passage: one or more faithful sentences + the geometry they trace to.
|
||||
|
||||
``sentences`` and ``provenance`` are index-aligned where possible: sentence
|
||||
``i`` was realized from ``provenance[i]``. A COHERE transition sentence with
|
||||
no new geometry carries a provenance whose ``kind == "connective"`` so the
|
||||
audit can see it introduced no new claim.
|
||||
"""
|
||||
sentences: list[str] = field(default_factory=list)
|
||||
provenance: list[Provenance] = field(default_factory=list)
|
||||
role: str = "body" # "body" | "lead" | "transition"
|
||||
|
||||
def text(self) -> str:
|
||||
return " ".join(s.rstrip(". ") + "." for s in self.sentences if s.strip())
|
||||
|
||||
|
||||
@dataclass
|
||||
class Section:
|
||||
heading: str
|
||||
level: int = 2 # markdown heading level / outline depth
|
||||
blocks: list[Block] = field(default_factory=list)
|
||||
seed_ids: list[str] = field(default_factory=list) # geometry nodes of section
|
||||
summary: str = "" # one-line grounded gloss (for pptx bullets / TOC)
|
||||
|
||||
def all_provenance(self) -> list[Provenance]:
|
||||
out: list[Provenance] = []
|
||||
for b in self.blocks:
|
||||
out.extend(b.provenance)
|
||||
return out
|
||||
|
||||
|
||||
@dataclass
|
||||
class DocumentIR:
|
||||
"""The surface-neutral document. Built ONCE, projected to ANY surface."""
|
||||
title: str
|
||||
subtitle: str = ""
|
||||
sections: list[Section] = field(default_factory=list)
|
||||
seed_id: str | None = None # the geometry region root
|
||||
format_spec: dict[str, Any] = field(default_factory=dict) # requested shape
|
||||
meta: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
# -- geometry facets (what non-text projectors consume) ----------------- #
|
||||
def all_provenance(self) -> list[Provenance]:
|
||||
out: list[Provenance] = []
|
||||
for s in self.sections:
|
||||
out.extend(s.all_provenance())
|
||||
return out
|
||||
|
||||
def claim_count(self) -> int:
|
||||
return sum(1 for p in self.all_provenance() if p.kind in ("fact", "interpretation"))
|
||||
|
||||
def ungrounded_count(self) -> int:
|
||||
"""Claims with no traceable node — MUST be zero for a faithful doc."""
|
||||
return sum(1 for p in self.all_provenance()
|
||||
if p.kind in ("fact", "interpretation") and not p.node_id)
|
||||
@@ -1,81 +0,0 @@
|
||||
"""generate.py — drive the projector: one geometry region -> many surfaces.
|
||||
|
||||
Proves the thesis with REAL output: builds ONE surface-neutral DocumentIR from
|
||||
Neuron's OWN self-geometry (read-only against the live soul via the proven
|
||||
faculty), then EMITS it to Markdown, docx, and MIDI — the same plan/realize/
|
||||
cohere, three surfaces. Writes the files + the faithfulness audit to ./out/.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, _HERE)
|
||||
|
||||
import pipeline # noqa: E402
|
||||
import provenance # noqa: E402
|
||||
from geometry import load_self_region # noqa: E402
|
||||
|
||||
OUT = os.path.join(_HERE, "out")
|
||||
|
||||
|
||||
def _emit_all(doc, stem):
|
||||
"""Emit one IR to every text/audio surface + audit + provenance."""
|
||||
for surface in ("markdown", "docx", "midi"):
|
||||
data = pipeline.emit(doc, surface)
|
||||
proj = pipeline.get_projector(surface)
|
||||
path = os.path.join(OUT, f"{stem}.{proj.ext}")
|
||||
with open(path, "wb") as f:
|
||||
f.write(data)
|
||||
print(f" emitted {surface:9s} -> {os.path.basename(path)} ({len(data)} bytes)")
|
||||
a = provenance.audit(doc)
|
||||
with open(os.path.join(OUT, f"{stem}.audit.json"), "w") as f:
|
||||
json.dump(a, f, indent=2)
|
||||
with open(os.path.join(OUT, f"{stem}.provenance.md"), "w") as f:
|
||||
f.write(provenance.trace_table(doc))
|
||||
print(" audit:", {k: a[k] for k in ("claims", "ungrounded_claims",
|
||||
"negations_preserved", "distinct_source_nodes", "faithful")})
|
||||
return a
|
||||
|
||||
|
||||
def main():
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
print("surfaces registered:", pipeline.available_surfaces())
|
||||
|
||||
# ---- Document 1: Neuron's self-description (marquee) ------------------- #
|
||||
print("\n[1] Neuron self-description")
|
||||
region = load_self_region(max_nodes=9)
|
||||
print(" self region:", region)
|
||||
doc1 = pipeline.build_ir(
|
||||
None, region=region,
|
||||
title="Neuron: A Self-Description from Its Own Geometry",
|
||||
subtitle="Projected efferently from the engram — every claim traces a node.",
|
||||
format_spec={"genre": "self-description", "register": "expository"},
|
||||
max_sections=5, conf_floor=0.6)
|
||||
print(f" IR: {len(doc1.sections)} sections, {doc1.claim_count()} claims, "
|
||||
f"ungrounded={doc1.ungrounded_count()}")
|
||||
_emit_all(doc1, "neuron-self")
|
||||
|
||||
# ---- Document 2: a coherent, clean whitepaper-style section ------------ #
|
||||
print("\n[2] Whitepaper-style section (coherent clean region)")
|
||||
doc2, _ = pipeline.project(
|
||||
["chronoception", "time", "awareness", "engram", "temporal"],
|
||||
surface="markdown",
|
||||
title="Temporal Awareness in the Engram",
|
||||
subtitle="A section projected from the geometry of chronoception.",
|
||||
format_spec={"genre": "whitepaper-section", "register": "technical"},
|
||||
max_sections=4)
|
||||
print(f" IR: {len(doc2.sections)} sections, {doc2.claim_count()} claims, "
|
||||
f"ungrounded={doc2.ungrounded_count()}")
|
||||
_emit_all(doc2, "engram-temporal")
|
||||
|
||||
# echo both markdowns so they are visible in the run log
|
||||
for stem, doc in (("neuron-self", doc1), ("engram-temporal", doc2)):
|
||||
print(f"\n===== GENERATED MARKDOWN — {stem} =====\n")
|
||||
print(pipeline.emit(doc, "markdown").decode())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,129 +0,0 @@
|
||||
"""geometry.py — READ-ONLY loader for a meaning-geometry region.
|
||||
|
||||
The efferent projector never writes to the soul. This module reaches the
|
||||
geometry through the PROVEN, read-only neuron-talk faculty (``engram_client``,
|
||||
GET-only, which physically refuses non-GET methods) against the running sidecar
|
||||
soul. The live daemon :8742 / :7770 is contacted ONLY through that read-only
|
||||
client — never mutated.
|
||||
|
||||
A "region" is a seed node plus a bounded neighborhood: the manifold that will
|
||||
become the document's skeleton. We pool a few single-term lexical searches
|
||||
(the engram search is a single-term matcher) and, when available, walk one hop
|
||||
of reified neighbors, then rank by self/importance signal.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
# Wire in the proven faculty (own-the-core: we reuse it, we do not fork it).
|
||||
_NT = os.path.expanduser("~/Desktop/neuron-talk")
|
||||
_LR = os.path.expanduser("~/Desktop/lang-realizers")
|
||||
for _p in (_NT, _LR):
|
||||
if _p not in sys.path:
|
||||
sys.path.insert(0, _p)
|
||||
|
||||
from engram_client import ReadOnlyEngramClient # noqa: E402
|
||||
|
||||
|
||||
class Region:
|
||||
"""A geometry region: ranked nodes + the reified edges among them."""
|
||||
|
||||
def __init__(self, seed: str, nodes: list[dict], edges: list[dict]):
|
||||
self.seed = seed
|
||||
self.nodes = nodes # ranked engram node dicts
|
||||
self.edges = edges # [{src, dst, edge, ...}]
|
||||
self.by_id = {n["id"]: n for n in nodes if n.get("id")}
|
||||
|
||||
def __repr__(self):
|
||||
return f"<Region seed={self.seed!r} nodes={len(self.nodes)} edges={len(self.edges)}>"
|
||||
|
||||
|
||||
def _prose_quality(content: str) -> float:
|
||||
"""Reward clean expository prose; penalize shouty banner-dense nodes.
|
||||
|
||||
A high ALLCAPS-word ratio or very short content signals a banner/telegraphic
|
||||
memory node that extracts into garbage. Clean declarative prose scores high.
|
||||
"""
|
||||
if not content or not content.strip():
|
||||
return 0.0
|
||||
words = content.split()
|
||||
if len(words) < 8:
|
||||
return 0.1
|
||||
caps = sum(1 for w in words if len(w) > 2 and w.strip(".,:;'\"-").isupper())
|
||||
caps_ratio = caps / max(1, len(words))
|
||||
# sentences with lowercase interior words read as prose
|
||||
lower = sum(1 for w in words if w[:1].islower())
|
||||
lower_ratio = lower / max(1, len(words))
|
||||
return max(0.0, 1.2 * lower_ratio - 2.0 * caps_ratio)
|
||||
|
||||
|
||||
def _relevance(content: str, terms: list[str]) -> float:
|
||||
"""Topical relevance to the seed terms — keeps a region ON-THEME so a clean
|
||||
but off-topic node cannot hijack the document."""
|
||||
if not terms:
|
||||
return 0.0
|
||||
low = (content or "").lower()
|
||||
hits = sum(1 for t in terms if t.lower() in low)
|
||||
return hits / max(1, len(terms))
|
||||
|
||||
|
||||
def _node_rank(n: dict, terms: list[str] | None = None) -> float:
|
||||
return (float(n.get("importance") or 0.0) * 2.0
|
||||
+ float(n.get("salience") or 0.0)
|
||||
+ 1.5 * _prose_quality(n.get("content") or "")
|
||||
+ 2.0 * _relevance(n.get("content") or "", terms or [])
|
||||
+ (0.5 if (n.get("content") or "").strip() else 0.0))
|
||||
|
||||
|
||||
def load_region(seed_terms: list[str] | str, *, client: ReadOnlyEngramClient | None = None,
|
||||
max_nodes: int = 10, per_term: int = 20, hop: bool = True) -> Region:
|
||||
"""Pull a bounded geometry region around ``seed_terms`` (read-only).
|
||||
|
||||
``seed_terms`` may be a single string or several probe terms; results are
|
||||
pooled and de-duplicated. When ``hop`` and the reified neighbor endpoint is
|
||||
live, one hop of neighbors is folded in so the region is a real
|
||||
neighborhood, not just a keyword hit list.
|
||||
"""
|
||||
client = client or ReadOnlyEngramClient()
|
||||
if isinstance(seed_terms, str):
|
||||
seed_terms = [seed_terms]
|
||||
|
||||
pool: dict[str, dict] = {}
|
||||
for term in seed_terms:
|
||||
for n in client.search(term, limit=per_term):
|
||||
if isinstance(n, dict) and n.get("id"):
|
||||
pool.setdefault(n["id"], n)
|
||||
|
||||
ranked = sorted(pool.values(), key=lambda n: _node_rank(n, seed_terms),
|
||||
reverse=True)
|
||||
nodes = ranked[:max_nodes]
|
||||
|
||||
edges: list[dict] = []
|
||||
if hop and nodes:
|
||||
present = {n["id"] for n in nodes}
|
||||
for n in list(nodes):
|
||||
try:
|
||||
for nb in client.neighbors(n["id"]):
|
||||
node = nb.get("node") if isinstance(nb, dict) else None
|
||||
edge = nb.get("edge") if isinstance(nb, dict) else None
|
||||
if node and node.get("id"):
|
||||
edges.append({"src": n["id"], "dst": node["id"],
|
||||
"edge": edge})
|
||||
# fold a strong neighbor into the region (bounded)
|
||||
if (node["id"] not in present and len(nodes) < max_nodes + 6
|
||||
and _node_rank(node, seed_terms) > 0.4):
|
||||
present.add(node["id"])
|
||||
nodes.append(node)
|
||||
except Exception: # noqa: BLE001 — read-only best-effort; never fatal
|
||||
continue
|
||||
|
||||
return Region(seed=", ".join(seed_terms), nodes=nodes, edges=edges)
|
||||
|
||||
|
||||
def load_self_region(client: ReadOnlyEngramClient | None = None,
|
||||
max_nodes: int = 10) -> Region:
|
||||
"""The self/identity region — Neuron's own geometry, for self-description."""
|
||||
return load_region(["self", "identity", "Neuron", "values", "memory",
|
||||
"imprint", "consciousness"],
|
||||
client=client, max_nodes=max_nodes)
|
||||
@@ -1,67 +0,0 @@
|
||||
"""pipeline.py — the Efferent Multimodal Projector, top level.
|
||||
|
||||
geometry region + surface/format spec
|
||||
-> PLAN (manifold -> document skeleton/DAG)
|
||||
-> REALIZE (proven realizer, sentence -> passage, each section faithful)
|
||||
-> COHERE (document-level flow / transitions, not stitched sentences)
|
||||
-> EMIT (pluggable SurfaceProjector -> the target surface)
|
||||
|
||||
THE SURFACE IS A PARAMETER. ``project(...)`` builds the geometry-carrying
|
||||
DocumentIR once, then hands it to whichever surface projector the caller named.
|
||||
Markdown, docx, and midi (music) are all the SAME IR emitted differently. That
|
||||
is the efferent multimodal projector: geometry -> any surface.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, _HERE)
|
||||
sys.path.insert(0, os.path.join(_HERE, "projectors"))
|
||||
|
||||
from cohere import cohere_document # noqa: E402
|
||||
from document_ir import DocumentIR # noqa: E402
|
||||
from geometry import Region, load_region # noqa: E402
|
||||
from plan import plan_document # noqa: E402
|
||||
from realize import realize_document # noqa: E402
|
||||
|
||||
# registering the projectors (import for side-effect: each self-registers)
|
||||
import projectors.markdown # noqa: E402,F401
|
||||
import projectors.docx # noqa: E402,F401
|
||||
import projectors.midi # noqa: E402,F401
|
||||
import projectors.seams # noqa: E402,F401
|
||||
from projectors.base import available_surfaces, get_projector # noqa: E402
|
||||
|
||||
|
||||
def build_ir(seed_terms, *, title: str, subtitle: str = "",
|
||||
format_spec: dict | None = None,
|
||||
region: Region | None = None,
|
||||
max_sections: int = 8, conf_floor: float = 0.55) -> DocumentIR:
|
||||
"""geometry -> PLAN -> REALIZE -> COHERE = the surface-neutral DocumentIR."""
|
||||
region = region or load_region(seed_terms)
|
||||
doc = plan_document(region, title=title, subtitle=subtitle,
|
||||
format_spec=format_spec or {},
|
||||
conf_floor=conf_floor, max_sections=max_sections)
|
||||
doc = realize_document(doc)
|
||||
doc = cohere_document(doc)
|
||||
return doc
|
||||
|
||||
|
||||
def emit(doc: DocumentIR, surface: str) -> bytes:
|
||||
"""EMIT: project the built IR onto one surface (surface = a parameter)."""
|
||||
return get_projector(surface).project(doc)
|
||||
|
||||
|
||||
def project(seed_terms, *, surface: str, title: str, subtitle: str = "",
|
||||
format_spec: dict | None = None, region: Region | None = None,
|
||||
max_sections: int = 8) -> tuple[DocumentIR, bytes]:
|
||||
"""The full efferent projection: geometry + surface -> (IR, bytes)."""
|
||||
doc = build_ir(seed_terms, title=title, subtitle=subtitle,
|
||||
format_spec=format_spec, region=region,
|
||||
max_sections=max_sections)
|
||||
return doc, emit(doc, surface)
|
||||
|
||||
|
||||
__all__ = ["build_ir", "emit", "project", "available_surfaces",
|
||||
"get_projector", "load_region", "DocumentIR"]
|
||||
@@ -1,192 +0,0 @@
|
||||
"""plan.py — PLAN stage: geometry region -> document skeleton (a DAG/outline).
|
||||
|
||||
The manifold becomes the skeleton. We extract faithful propositions from the
|
||||
region's nodes (the proven neuron-talk extractor, SACRED polarity preserved),
|
||||
apply a quality floor, then GROUP them into sections. Grouping is by source
|
||||
node — each engram node is one coherent topic, so one salient node becomes one
|
||||
section. The section ORDER is the node ranking (importance/salience): the
|
||||
geometry decides the outline, not a template.
|
||||
|
||||
Output: a DocumentIR whose sections carry seed node ids and empty blocks. REALIZE
|
||||
fills the blocks; the plan owns the structure.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
_NT = os.path.expanduser("~/Desktop/neuron-talk")
|
||||
_LR = os.path.expanduser("~/Desktop/lang-realizers")
|
||||
for _p in (_NT, _LR):
|
||||
if _p not in sys.path:
|
||||
sys.path.insert(0, _p)
|
||||
|
||||
import propositions # noqa: E402 (the proven, faithful extractor)
|
||||
|
||||
from document_ir import DocumentIR, Section # noqa: E402
|
||||
from geometry import Region # noqa: E402
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Proposition quality — keep only clean, well-grounded claims.
|
||||
# --------------------------------------------------------------------------- #
|
||||
_JUNK_RE = re.compile(r"[.][a-z]{1,3}\b|[^A-Za-z0-9 '\-]") # ".o", stray symbols
|
||||
|
||||
|
||||
def _has_banner_token(s: str) -> bool:
|
||||
"""True if any word is an ALLCAPS banner token (DHARMA, ENGRAM, MEASURED)."""
|
||||
for w in (s or "").split():
|
||||
core = w.strip(".,:;'\"-")
|
||||
if len(core) > 2 and core.isupper():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _clean_prop(p, floor: float) -> bool:
|
||||
if p.confidence < floor:
|
||||
return False
|
||||
if not p.subject or not (p.object or (p.obj_np is not None)):
|
||||
return False
|
||||
subj = (p.subject or "").strip()
|
||||
obj = (p.object or "").strip()
|
||||
if len(subj) < 2:
|
||||
return False
|
||||
# banner-derived shouty fragments read as garbage in prose
|
||||
if _has_banner_token(subj) or _has_banner_token(obj):
|
||||
return False
|
||||
if propositions._is_shouty(p.sentence or ""):
|
||||
return False
|
||||
# junk tokens: file-extension fragments (".o"), stray non-word symbols
|
||||
if _JUNK_RE.search(subj) or _JUNK_RE.search(obj):
|
||||
return False
|
||||
# a proposition whose object repeats the subject is usually a parse artifact
|
||||
if obj and subj.lower() == obj.lower():
|
||||
return False
|
||||
# a bare copula with no real complement ("X is it") reads as noise
|
||||
if p.predicate == "be" and obj.lower() in ("it", "no", "nothing", "empty", ""):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _dedup(props):
|
||||
"""Drop duplicate claims. Two axes: (a) identical (pred,obj,polarity), and
|
||||
(b) same (subject,predicate) — which collapses a mis-split compound like
|
||||
"detection is post-hoc eval" -> "Detection is post/hoc/eval" into one claim
|
||||
(keep the highest-confidence surface)."""
|
||||
props = sorted(props, key=lambda p: p.confidence, reverse=True)
|
||||
seen_po, seen_sp, out = set(), set(), []
|
||||
for p in props:
|
||||
subj = (p.subject or "").lower()
|
||||
po = (p.predicate, (p.object or "").lower(), p.polarity)
|
||||
sp = (subj, p.predicate, p.polarity)
|
||||
if po in seen_po or sp in seen_sp:
|
||||
continue
|
||||
seen_po.add(po)
|
||||
seen_sp.add(sp)
|
||||
out.append(p)
|
||||
return out
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Heading derivation — a clean human heading from a node.
|
||||
# --------------------------------------------------------------------------- #
|
||||
_HEADING_RE = re.compile(r"^\s*#{1,4}\s+(.{2,70})\s*$", re.M)
|
||||
# node-type / system labels that are NOT topical headings
|
||||
_NONTOPIC_LABEL = re.compile(r"^(memory|node|knowledge|doc|session)[:/]", re.I)
|
||||
|
||||
|
||||
def _titlecase_banner(s: str) -> str:
|
||||
"""A shouty banner ("CHRONOCEPTION — SCALE-INVARIANCE") makes a fine title
|
||||
once Title-cased. Keep short acronyms uppercase."""
|
||||
def fix(w):
|
||||
core = w.strip("—-:,.")
|
||||
if len(core) <= 3 and core.isupper():
|
||||
return w # acronym
|
||||
return w.capitalize()
|
||||
return " ".join(fix(w) for w in s.split())
|
||||
|
||||
|
||||
def _clean_heading(text: str) -> str | None:
|
||||
"""First line only, no markdown, capped, banner Title-cased. None if unusable."""
|
||||
if not text:
|
||||
return None
|
||||
line = text.strip().splitlines()[0]
|
||||
line = re.sub(r"^#+\s*", "", line).strip().strip("#").strip()
|
||||
# cut at a natural break so a long banner heading stays a heading, not a para
|
||||
for sep in (" — ", " – ", ": ", ". "):
|
||||
if sep in line and len(line) > 48:
|
||||
line = line.split(sep)[0].strip()
|
||||
break
|
||||
if not (3 <= len(line) <= 64):
|
||||
return None
|
||||
if propositions._is_shouty(line):
|
||||
line = _titlecase_banner(line)
|
||||
return line or None
|
||||
|
||||
|
||||
def _heading_for(node: dict, fallback: str) -> str:
|
||||
label = (node.get("label") or "").strip()
|
||||
content = node.get("content") or ""
|
||||
candidates: list[str] = []
|
||||
# a node-type label ("memory:remembered") is never a topic — skip it
|
||||
if label and not _NONTOPIC_LABEL.match(label):
|
||||
candidates.append(label)
|
||||
m = _HEADING_RE.search(content)
|
||||
if m:
|
||||
candidates.append(m.group(1))
|
||||
# the leading banner/first sentence of the content is often the real title
|
||||
first = re.split(r"(?<=[.\n])", content.strip(), maxsplit=1)[0] if content.strip() else ""
|
||||
candidates.append(first)
|
||||
for c in candidates:
|
||||
h = _clean_heading(c)
|
||||
if h:
|
||||
return h
|
||||
return fallback
|
||||
|
||||
|
||||
def plan_document(region: Region, *, title: str, subtitle: str = "",
|
||||
format_spec: dict | None = None,
|
||||
conf_floor: float = 0.55,
|
||||
max_sections: int = 8,
|
||||
max_claims_per_section: int = 6) -> DocumentIR:
|
||||
"""Region -> DocumentIR skeleton. The geometry dictates the outline."""
|
||||
format_spec = format_spec or {}
|
||||
doc = DocumentIR(title=title, subtitle=subtitle,
|
||||
seed_id=region.nodes[0]["id"] if region.nodes else None,
|
||||
format_spec=format_spec)
|
||||
|
||||
made = 0
|
||||
seen_headings: set[str] = set()
|
||||
for node in region.nodes:
|
||||
if made >= max_sections:
|
||||
break
|
||||
props = propositions.extract(node.get("content") or "",
|
||||
node_id=node.get("id"),
|
||||
node_importance=float(node.get("importance") or 0.0),
|
||||
max_sentences=10)
|
||||
props = [p for p in props if _clean_prop(p, conf_floor)]
|
||||
props = _dedup(props)
|
||||
props.sort(key=lambda p: p.confidence, reverse=True)
|
||||
props = props[:max_claims_per_section]
|
||||
if not props:
|
||||
continue
|
||||
heading = _heading_for(node, fallback=f"Region {made + 1}")
|
||||
# cross-section dedup: a topic appears once. Distinguish by top claim
|
||||
# subject, else drop the collision so the outline stays clean.
|
||||
if heading.lower() in seen_headings:
|
||||
subj = (props[0].subject or "").strip().title()
|
||||
alt = f"{heading}: {subj}" if subj and subj.lower() not in heading.lower() else None
|
||||
if alt and alt.lower() not in seen_headings and len(alt) <= 64:
|
||||
heading = alt
|
||||
else:
|
||||
continue
|
||||
seen_headings.add(heading.lower())
|
||||
sec = Section(heading=heading, level=2, seed_ids=[node["id"]])
|
||||
# stash the planned propositions on the section for REALIZE
|
||||
sec.__dict__["_planned_props"] = props
|
||||
sec.__dict__["_node"] = node
|
||||
doc.sections.append(sec)
|
||||
made += 1
|
||||
|
||||
return doc
|
||||
@@ -1,106 +0,0 @@
|
||||
"""base.py — the SurfaceProjector interface + registry.
|
||||
|
||||
THE key abstraction of the efferent projector: a projector is a pure function
|
||||
from the surface-neutral, geometry-carrying DocumentIR to bytes on a target
|
||||
SURFACE. The surface is a PARAMETER. Adding a surface = registering one more
|
||||
projector; nothing upstream (plan/realize/cohere) changes.
|
||||
|
||||
DocumentIR --project--> bytes (per surface)
|
||||
|
||||
A TEXT projector reads ``block.sentences``. A NON-TEXT projector (music, image,
|
||||
video) reads ``block.provenance`` — the geometry the IR carries — and decodes it
|
||||
onto its surface. Both consume the SAME IR. That symmetry is the whole design:
|
||||
the realizer generalizes into a multimodal projector, geometry -> any surface.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Protocol, runtime_checkable
|
||||
|
||||
import sys
|
||||
import os
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
from document_ir import DocumentIR # noqa: E402
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class SurfaceProjector(Protocol):
|
||||
"""Geometry-document -> one surface. Implementations MUST be pure & faithful.
|
||||
|
||||
THE ONE SHARED SEAM. Every surface — text, music, image, video — conforms to
|
||||
this single contract:
|
||||
|
||||
project(frame: DocumentIR) -> bytes
|
||||
|
||||
where ``frame`` is the geometry-carrying meaning-geometry (the SemFrame at
|
||||
document scale; a single utterance is the degenerate one-section frame).
|
||||
|
||||
RECOMMENDED INTERNAL SHAPE (the peer music/text decomposition, blessed here
|
||||
so all surfaces share it): a projector may split ``project`` into
|
||||
|
||||
spec = self.plan(frame) # meaning-geometry -> surface-specific spec
|
||||
bytes = self.realize(spec) # spec -> surface, via this projector's PROFILE
|
||||
|
||||
``project`` is then ``realize(plan(frame))``. The PROFILE (a text lang-profile,
|
||||
a music instr/mode-profile, an image layout-profile) is a property of the
|
||||
projector instance — the pluggable knob. See :class:`TwoStageProjector`.
|
||||
|
||||
A TEXT projector's plan reads ``frame`` sentences; a MUSIC/IMAGE projector's
|
||||
plan reads ``frame.all_provenance()`` — the geometry — and derives its spec
|
||||
(pitch/harmony/rhythm, or layout) FROM the meaning, deterministically. Same
|
||||
frame, different profile.
|
||||
"""
|
||||
|
||||
surface: str # "markdown" | "docx" | "midi" | "audio" | "image" | "video"
|
||||
media_type: str # MIME type of the emitted bytes
|
||||
ext: str # file extension (no dot)
|
||||
modality: str # "text" | "audio" | "image" | "video"
|
||||
profile: object # the pluggable per-surface profile (may be None)
|
||||
|
||||
def project(self, doc: DocumentIR) -> bytes:
|
||||
"""Emit the document on this surface. Returns raw bytes."""
|
||||
...
|
||||
|
||||
|
||||
class TwoStageProjector:
|
||||
"""Optional base for the peer plan()/realize() decomposition.
|
||||
|
||||
Subclasses implement ``plan(frame) -> spec`` and ``realize(spec) -> bytes``;
|
||||
``project`` is their composition. This is exactly the peer music interface
|
||||
(spec = plan(frame, profile); surface = realize(spec, profile)) expressed so
|
||||
that it still satisfies the single ``SurfaceProjector.project`` seam. Text,
|
||||
music, and image projectors can all subclass this and remain interchangeable.
|
||||
"""
|
||||
|
||||
surface: str = ""
|
||||
media_type: str = ""
|
||||
ext: str = ""
|
||||
modality: str = ""
|
||||
profile: object = None
|
||||
|
||||
def plan(self, doc: DocumentIR): # -> spec
|
||||
raise NotImplementedError
|
||||
|
||||
def realize(self, spec) -> bytes:
|
||||
raise NotImplementedError
|
||||
|
||||
def project(self, doc: DocumentIR) -> bytes:
|
||||
return self.realize(self.plan(doc))
|
||||
|
||||
|
||||
_REGISTRY: dict[str, SurfaceProjector] = {}
|
||||
|
||||
|
||||
def register(projector: SurfaceProjector) -> SurfaceProjector:
|
||||
_REGISTRY[projector.surface] = projector
|
||||
return projector
|
||||
|
||||
|
||||
def get_projector(surface: str) -> SurfaceProjector:
|
||||
if surface not in _REGISTRY:
|
||||
raise KeyError(f"no projector registered for surface {surface!r}; "
|
||||
f"have {sorted(_REGISTRY)}")
|
||||
return _REGISTRY[surface]
|
||||
|
||||
|
||||
def available_surfaces() -> list[str]:
|
||||
return sorted(_REGISTRY)
|
||||
@@ -1,113 +0,0 @@
|
||||
"""docx.py — the .docx surface projector: an OWN minimal OOXML emitter.
|
||||
|
||||
Own-the-core: a .docx is just a ZIP of a few XML parts (WordprocessingML). We
|
||||
emit it with the standard library only — ``zipfile`` + string XML — no
|
||||
python-docx, no external dependency. This proves a "richer structured format"
|
||||
surface without importing anyone else's toolkit.
|
||||
|
||||
Parts emitted (the minimal valid set + a styles part for real headings):
|
||||
[Content_Types].xml
|
||||
_rels/.rels
|
||||
word/_rels/document.xml.rels
|
||||
word/styles.xml (Title / Heading1 / Heading2 / Normal)
|
||||
word/document.xml (the content)
|
||||
|
||||
Like the markdown projector it reads only the IR's realized sentences; it
|
||||
invents nothing. The surface differs, the faithful content does not.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import os
|
||||
import sys
|
||||
import zipfile
|
||||
from xml.sax.saxutils import escape
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
from document_ir import DocumentIR # noqa: E402
|
||||
from projectors.base import register # noqa: E402
|
||||
|
||||
_CONTENT_TYPES = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||
<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">
|
||||
<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>
|
||||
<Default Extension="xml" ContentType="application/xml"/>
|
||||
<Override PartName="/word/document.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/>
|
||||
<Override PartName="/word/styles.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.styles+xml"/>
|
||||
</Types>"""
|
||||
|
||||
_RELS = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||
<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
|
||||
<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="word/document.xml"/>
|
||||
</Relationships>"""
|
||||
|
||||
_DOC_RELS = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||
<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
|
||||
<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/styles" Target="styles.xml"/>
|
||||
</Relationships>"""
|
||||
|
||||
_W = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
|
||||
|
||||
_STYLES = f"""<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||
<w:styles xmlns:w="{_W}">
|
||||
<w:style w:type="paragraph" w:default="1" w:styleId="Normal"><w:name w:val="Normal"/>
|
||||
<w:rPr><w:sz w:val="22"/></w:rPr></w:style>
|
||||
<w:style w:type="paragraph" w:styleId="Title"><w:name w:val="Title"/>
|
||||
<w:pPr><w:spacing w:after="240"/></w:pPr>
|
||||
<w:rPr><w:b/><w:sz w:val="52"/></w:rPr></w:style>
|
||||
<w:style w:type="paragraph" w:styleId="Subtitle"><w:name w:val="Subtitle"/>
|
||||
<w:rPr><w:i/><w:sz w:val="28"/><w:color w:val="555555"/></w:rPr></w:style>
|
||||
<w:style w:type="paragraph" w:styleId="Heading1"><w:name w:val="heading 1"/>
|
||||
<w:pPr><w:spacing w:before="240" w:after="120"/><w:outlineLvl w:val="0"/></w:pPr>
|
||||
<w:rPr><w:b/><w:sz w:val="34"/></w:rPr></w:style>
|
||||
<w:style w:type="paragraph" w:styleId="Heading2"><w:name w:val="heading 2"/>
|
||||
<w:pPr><w:spacing w:before="200" w:after="100"/><w:outlineLvl w:val="1"/></w:pPr>
|
||||
<w:rPr><w:b/><w:sz w:val="28"/></w:rPr></w:style>
|
||||
</w:styles>"""
|
||||
|
||||
|
||||
def _para(text: str, style: str | None = None) -> str:
|
||||
ppr = f"<w:pPr><w:pStyle w:val=\"{style}\"/></w:pPr>" if style else ""
|
||||
return (f"<w:p>{ppr}<w:r><w:t xml:space=\"preserve\">"
|
||||
f"{escape(text)}</w:t></w:r></w:p>")
|
||||
|
||||
|
||||
class DocxProjector:
|
||||
surface = "docx"
|
||||
media_type = ("application/vnd.openxmlformats-officedocument."
|
||||
"wordprocessingml.document")
|
||||
ext = "docx"
|
||||
modality = "text"
|
||||
|
||||
def _document_xml(self, doc: DocumentIR) -> str:
|
||||
body: list[str] = [_para(doc.title, "Title")]
|
||||
if doc.subtitle:
|
||||
body.append(_para(doc.subtitle, "Subtitle"))
|
||||
abstract = doc.meta.get("abstract")
|
||||
if abstract is not None and abstract.sentences:
|
||||
body.append(_para(abstract.text()))
|
||||
for sec in doc.sections:
|
||||
style = "Heading1" if sec.level <= 1 else "Heading2"
|
||||
body.append(_para(sec.heading, style))
|
||||
for block in sec.blocks:
|
||||
t = block.text()
|
||||
if t:
|
||||
body.append(_para(t))
|
||||
return (f"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>"
|
||||
f"<w:document xmlns:w=\"{_W}\"><w:body>"
|
||||
+ "".join(body)
|
||||
+ "<w:sectPr><w:pgSz w:w=\"12240\" w:h=\"15840\"/>"
|
||||
"<w:pgMar w:top=\"1440\" w:right=\"1440\" w:bottom=\"1440\" "
|
||||
"w:left=\"1440\"/></w:sectPr></w:body></w:document>")
|
||||
|
||||
def project(self, doc: DocumentIR) -> bytes:
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as z:
|
||||
z.writestr("[Content_Types].xml", _CONTENT_TYPES)
|
||||
z.writestr("_rels/.rels", _RELS)
|
||||
z.writestr("word/_rels/document.xml.rels", _DOC_RELS)
|
||||
z.writestr("word/styles.xml", _STYLES)
|
||||
z.writestr("word/document.xml", self._document_xml(doc))
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
register(DocxProjector())
|
||||
@@ -1,45 +0,0 @@
|
||||
"""markdown.py — the Markdown surface projector (text facet).
|
||||
|
||||
The most tractable surface, and the reference implementation: reads the IR's
|
||||
realized sentences and lays them out as Markdown. Introduces no content — it is
|
||||
pure typography over the faithful text the realizer produced.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
from document_ir import DocumentIR # noqa: E402
|
||||
from projectors.base import register # noqa: E402
|
||||
|
||||
|
||||
class MarkdownProjector:
|
||||
surface = "markdown"
|
||||
media_type = "text/markdown"
|
||||
ext = "md"
|
||||
modality = "text"
|
||||
|
||||
def render_str(self, doc: DocumentIR) -> str:
|
||||
lines: list[str] = [f"# {doc.title}"]
|
||||
if doc.subtitle:
|
||||
lines.append(f"\n*{doc.subtitle}*")
|
||||
abstract = doc.meta.get("abstract")
|
||||
if abstract is not None and abstract.sentences:
|
||||
lines.append("")
|
||||
lines.append(abstract.text())
|
||||
for sec in doc.sections:
|
||||
lines.append("")
|
||||
lines.append(f"{'#' * max(2, sec.level)} {sec.heading}")
|
||||
for block in sec.blocks:
|
||||
body = block.text()
|
||||
if body:
|
||||
lines.append("")
|
||||
lines.append(body)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
def project(self, doc: DocumentIR) -> bytes:
|
||||
return self.render_str(doc).encode("utf-8")
|
||||
|
||||
|
||||
register(MarkdownProjector())
|
||||
@@ -1,133 +0,0 @@
|
||||
"""midi.py — the MUSIC surface projector: geometry -> symbolic music (MIDI).
|
||||
|
||||
The first NON-TEXT surface, and the proof of the general shape. "Music is
|
||||
language and it is math" (Will): symbolic music is tractable and geometry-native,
|
||||
so it is the natural efferent twin to try first after text.
|
||||
|
||||
CRUCIALLY this projector does NOT read the realized sentences. It reads the IR's
|
||||
GEOMETRY facet — ``block.provenance`` — and DECODES each edge onto a musical
|
||||
surface. That is the whole thesis of the multimodal projector: the same
|
||||
geometry-carrying IR drives text AND music; a text projector reads the words, a
|
||||
music projector reads the meaning-geometry. The mapping is deterministic and
|
||||
faithful to the geometry's structure:
|
||||
|
||||
relation lemma -> scale degree (same relation -> same pitch class;
|
||||
meaning has a consistent sonic form)
|
||||
polarity -> mode (aff = major third above; neg = minor
|
||||
third / lowered — SACRED polarity is
|
||||
audible, a negated edge sounds negated)
|
||||
confidence -> note duration (stronger grounding rings longer)
|
||||
importance -> velocity (more important source = louder)
|
||||
section -> phrase + register shift (structure becomes musical form)
|
||||
|
||||
Own-the-core: a Standard MIDI File is a header chunk + a track chunk of
|
||||
delta-timed events. We emit the raw bytes with ``struct`` — no external MIDI
|
||||
library. Format 0, one track.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
from document_ir import DocumentIR, Provenance # noqa: E402
|
||||
from projectors.base import TwoStageProjector, register # noqa: E402
|
||||
|
||||
_TICKS = 480 # ticks per quarter note
|
||||
_C_MAJOR = [0, 2, 4, 5, 7, 9, 11] # semitone offsets of a diatonic scale
|
||||
|
||||
|
||||
def _vlq(n: int) -> bytes:
|
||||
"""MIDI variable-length quantity encoding of a delta time."""
|
||||
if n == 0:
|
||||
return b"\x00"
|
||||
out = bytearray()
|
||||
out.append(n & 0x7F)
|
||||
n >>= 7
|
||||
while n:
|
||||
out.insert(0, (n & 0x7F) | 0x80)
|
||||
n >>= 7
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def _degree_for(relation: str) -> int:
|
||||
"""Stable scale degree for a relation lemma (same relation -> same pitch)."""
|
||||
if not relation:
|
||||
return 0
|
||||
return sum(ord(c) for c in relation.lower()) % len(_C_MAJOR)
|
||||
|
||||
|
||||
def _note_for(p: Provenance, base: int) -> tuple[int, int, int]:
|
||||
"""(pitch, velocity, duration_ticks) for one geometry edge."""
|
||||
root = base + _C_MAJOR[_degree_for(p.relation)]
|
||||
# polarity -> mode: affirmed edges take the bright major third, negated edges
|
||||
# take the darker minor third. The negation is AUDIBLE and never dropped.
|
||||
third = 4 if p.polarity == "aff" else 3
|
||||
pitch = max(24, min(96, root + (third if p.confidence >= 0.5 else 0)))
|
||||
velocity = int(56 + 60 * min(1.0, max(0.0, p.importance)))
|
||||
velocity = max(40, min(120, velocity))
|
||||
# confidence -> duration: quarter .. dotted-half
|
||||
dur = int(_TICKS * (0.5 + 1.5 * min(1.0, max(0.0, p.confidence))))
|
||||
return pitch, velocity, dur
|
||||
|
||||
|
||||
# a mode-profile: the pluggable musical knob (the peer's mode_profile). Scale +
|
||||
# tempo. Swapping this profile re-voices the SAME geometry — surface as parameter.
|
||||
_DEFAULT_PROFILE = {"scale": _C_MAJOR, "tempo_us": 500000,
|
||||
"registers": [60, 55, 64, 50, 67, 48], "program": 0}
|
||||
|
||||
|
||||
class MidiProjector(TwoStageProjector):
|
||||
"""geometry -> symbolic music, in the shared two-stage shape.
|
||||
|
||||
``plan(frame)`` -> a music_spec: an ordered list of note dicts derived
|
||||
deterministically from the frame's provenance geometry
|
||||
(the peer's ``plan(frame, profile) -> spec``).
|
||||
``realize(spec)`` -> Standard MIDI File bytes (the peer's
|
||||
``realize(spec, profile) -> surface``; here the surface
|
||||
is symbolic MIDI, the minimal audio proof — a richer
|
||||
additive-synth audio projector conforms identically).
|
||||
"""
|
||||
|
||||
surface = "midi"
|
||||
media_type = "audio/midi"
|
||||
ext = "mid"
|
||||
modality = "audio"
|
||||
|
||||
def __init__(self, profile: dict | None = None):
|
||||
self.profile = profile or _DEFAULT_PROFILE
|
||||
|
||||
# -- stage 1: meaning-geometry -> music_spec (reads the GEOMETRY facet) -- #
|
||||
def plan(self, doc: DocumentIR) -> list[dict]:
|
||||
registers = self.profile["registers"]
|
||||
spec: list[dict] = []
|
||||
for si, sec in enumerate(doc.sections):
|
||||
base = registers[si % len(registers)]
|
||||
provs = [p for p in sec.all_provenance()
|
||||
if p.kind in ("fact", "interpretation")]
|
||||
for i, p in enumerate(provs):
|
||||
pitch, vel, dur = _note_for(p, base)
|
||||
spec.append({"pitch": pitch, "velocity": vel, "dur": dur,
|
||||
"rest_before": (_TICKS // 2) if (si > 0 and i == 0) else 0,
|
||||
"relation": p.relation, "polarity": p.polarity})
|
||||
return spec
|
||||
|
||||
# -- stage 2: music_spec -> MIDI bytes (own-core, no library) ------------ #
|
||||
def realize(self, spec: list[dict]) -> bytes:
|
||||
ev = bytearray()
|
||||
ev += _vlq(0) + b"\xFF\x51\x03" + struct.pack(">I", self.profile["tempo_us"])[1:]
|
||||
ev += _vlq(0) + bytes([0xC0, self.profile["program"] & 0x7F])
|
||||
for note in spec:
|
||||
ev += _vlq(note["rest_before"]) + bytes([0x90, note["pitch"], note["velocity"]])
|
||||
ev += _vlq(note["dur"]) + bytes([0x80, note["pitch"], 0])
|
||||
ev += _vlq(0) + b"\xFF\x2F\x00"
|
||||
track = bytes(ev)
|
||||
buf = io.BytesIO()
|
||||
buf.write(b"MThd" + struct.pack(">IHHH", 6, 0, 1, _TICKS))
|
||||
buf.write(b"MTrk" + struct.pack(">I", len(track)) + track)
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
register(MidiProjector())
|
||||
@@ -1,60 +0,0 @@
|
||||
"""seams.py — documented efferent seams for IMAGE and VIDEO surfaces.
|
||||
|
||||
These are NOT implemented (per the build rails: architect, do not overbuild).
|
||||
They are registered as first-class seams so the interface PROVES it accepts
|
||||
future non-text projectors without any upstream change. Each documents exactly
|
||||
what its decoder would read from the geometry-carrying IR, making the multimodal
|
||||
generalization concrete rather than hand-wavy.
|
||||
|
||||
The symmetry that guarantees these are possible, not moonshots: they are the
|
||||
efferent twins of multimodal INGEST. If meaning can HOLD an image (ingest as
|
||||
first-class geometry), meaning can PROJECT one back. Video = image x sound x
|
||||
TIME, and the engram already stores time (chronoception). So video falls out of
|
||||
an image projector + the music projector + the stored temporal ordering.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
from document_ir import DocumentIR # noqa: E402
|
||||
from projectors.base import register # noqa: E402
|
||||
|
||||
|
||||
class _Seam:
|
||||
"""A registered-but-unimplemented projector. Names its decoder contract."""
|
||||
|
||||
def project(self, doc: DocumentIR) -> bytes: # pragma: no cover - seam
|
||||
raise NotImplementedError(
|
||||
f"{self.surface!r} projector is a documented seam, not yet built. "
|
||||
f"Decoder contract: {self.decoder_contract}")
|
||||
|
||||
|
||||
class ImageProjector(_Seam):
|
||||
surface = "image"
|
||||
media_type = "image/png"
|
||||
ext = "png"
|
||||
modality = "image"
|
||||
decoder_contract = (
|
||||
"reads block.provenance as a spatial layout — nodes become regions, edges "
|
||||
"become adjacencies; salience/importance drive size/contrast; polarity "
|
||||
"drives figure/ground. The efferent twin of image ingest (a geometry->raster "
|
||||
"decoder, learned or engineered), exactly mirroring the embedder that turned "
|
||||
"the image INTO geometry.")
|
||||
|
||||
|
||||
class VideoProjector(_Seam):
|
||||
surface = "video"
|
||||
media_type = "video/mp4"
|
||||
ext = "mp4"
|
||||
modality = "video"
|
||||
decoder_contract = (
|
||||
"image x sound x TIME. Composes the image projector (per-keyframe geometry "
|
||||
"layout) with the midi/music projector (score) along the geometry's stored "
|
||||
"temporal ordering (chronoception). Needs no new principle once image + music "
|
||||
"exist — only a muxer.")
|
||||
|
||||
|
||||
register(ImageProjector())
|
||||
register(VideoProjector())
|
||||
@@ -1,63 +0,0 @@
|
||||
"""provenance.py — the faithfulness audit + geometry->section trace.
|
||||
|
||||
A document projected from geometry is only worth anything if every claim traces
|
||||
back. This module walks the DocumentIR and proves the discipline held:
|
||||
|
||||
* ZERO ungrounded claims (every fact/interpretation has a real node id),
|
||||
* every emitted sentence maps to a geometry edge (or is a marked connective),
|
||||
* SACRED polarity survived (negations are reported, never silently dropped),
|
||||
* COHERE introduced no new geometry (connectives carry no claim).
|
||||
|
||||
It emits both a machine verdict and a human-readable geometry->section table.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from document_ir import DocumentIR
|
||||
|
||||
|
||||
def audit(doc: DocumentIR) -> dict:
|
||||
provs = doc.all_provenance()
|
||||
facts = [p for p in provs if p.kind in ("fact", "interpretation")]
|
||||
connectives = [p for p in provs if p.kind == "connective"]
|
||||
ungrounded = [p for p in facts if not p.node_id]
|
||||
negations = [p for p in facts if p.polarity == "neg"]
|
||||
node_ids = sorted({p.node_id for p in facts if p.node_id})
|
||||
return {
|
||||
"claims": len(facts),
|
||||
"connectives": len(connectives),
|
||||
"ungrounded_claims": len(ungrounded),
|
||||
"negations_preserved": len(negations),
|
||||
"distinct_source_nodes": len(node_ids),
|
||||
"faithful": len(ungrounded) == 0,
|
||||
"source_nodes": node_ids,
|
||||
}
|
||||
|
||||
|
||||
def trace_table(doc: DocumentIR) -> str:
|
||||
"""Human-readable geometry -> section -> claim provenance table."""
|
||||
lines = ["# Provenance — every claim traces geometry", ""]
|
||||
lines.append(f"**Document:** {doc.title}")
|
||||
a = audit(doc)
|
||||
lines.append(f"**Claims:** {a['claims']} · **Ungrounded:** "
|
||||
f"{a['ungrounded_claims']} · **Negations preserved:** "
|
||||
f"{a['negations_preserved']} · **Source nodes:** "
|
||||
f"{a['distinct_source_nodes']} · **Faithful:** "
|
||||
f"{'YES' if a['faithful'] else 'NO'}")
|
||||
lines.append("")
|
||||
for si, sec in enumerate(doc.sections, 1):
|
||||
lines.append(f"## {si}. {sec.heading}")
|
||||
lines.append(f"_seed nodes: {', '.join(i[:8] for i in sec.seed_ids)}_")
|
||||
lines.append("")
|
||||
lines.append("| # | realized claim | traces geometry edge |")
|
||||
lines.append("|---|----------------|----------------------|")
|
||||
n = 0
|
||||
for block in sec.blocks:
|
||||
for sent, prov in zip(block.sentences, block.provenance):
|
||||
if prov.kind == "connective":
|
||||
continue
|
||||
n += 1
|
||||
edge = prov.trace().replace("|", "\\|")
|
||||
s = sent.replace("|", "\\|")
|
||||
lines.append(f"| {n} | {s} | {edge} |")
|
||||
lines.append("")
|
||||
return "\n".join(lines) + "\n"
|
||||
@@ -1,112 +0,0 @@
|
||||
"""realize.py — REALIZE stage: fill each planned section with faithful passages.
|
||||
|
||||
Scales the PROVEN realizer from a single assertion to a passage. For each
|
||||
planned proposition we build a realizer-ready clause (the proven
|
||||
``_prop_to_clause`` mapping) and run it through the proven engine
|
||||
(``engine.realize``), which is a deterministic grammar with the SACRED negation
|
||||
contract — it never invents. Each realized sentence is paired with a
|
||||
:class:`Provenance` that pins it to the exact geometry edge it came from.
|
||||
|
||||
"Passage, not a list of sentences": within a section we lightly vary sentence
|
||||
openings and group related claims, but we add NO content the geometry did not
|
||||
assert. The only non-geometry words are function words the grammar already owns
|
||||
(articles, "and", conjunction of same-subject claims). Document-level flow is
|
||||
COHERE's job; this stage owns intra-section fluency + fidelity.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
_NT = os.path.expanduser("~/Desktop/neuron-talk")
|
||||
_LR = os.path.expanduser("~/Desktop/lang-realizers")
|
||||
for _p in (_NT, _LR):
|
||||
if _p not in sys.path:
|
||||
sys.path.insert(0, _p)
|
||||
|
||||
import engine # noqa: E402 (the proven no-LLM realizer)
|
||||
from dialogue import _prop_to_clause # noqa: E402 (proven prop -> clause)
|
||||
|
||||
from document_ir import Block, DocumentIR, Provenance, Section # noqa: E402
|
||||
|
||||
|
||||
def _provenance_from(p, kind: str = "fact") -> Provenance:
|
||||
return Provenance(
|
||||
subj_id=p.source_node_id, subject=p.subject, relation=p.predicate,
|
||||
obj=p.object, polarity=p.polarity, confidence=round(float(p.confidence), 3),
|
||||
node_id=p.source_node_id, kind=kind,
|
||||
importance=float(getattr(p, "node_importance", 0.0) or 0.0),
|
||||
salience=0.0,
|
||||
)
|
||||
|
||||
|
||||
import re as _re
|
||||
|
||||
# a well-formed declarative opens with a determiner, a proper noun, "I", or a
|
||||
# capitalized head — not a mis-parsed object pronoun or a copula fragment.
|
||||
_BAD_OPENERS = _re.compile(r"^(Me |It is I|There is|This is it|That is it)\b")
|
||||
_VACUOUS = _re.compile(r"^\w+ (is|are|was|were) (it|no|nothing|empty|those|this|that)\.?$",
|
||||
_re.I)
|
||||
|
||||
|
||||
def _good_sentence(text: str) -> bool:
|
||||
"""Fluency gate — drops degenerate realizations. NEVER loosens faithfulness;
|
||||
it only refuses to SPEAK a claim whose surface came out malformed."""
|
||||
words = text.rstrip(".").split()
|
||||
if len(words) < 3:
|
||||
return False
|
||||
if _BAD_OPENERS.search(text):
|
||||
return False
|
||||
if _VACUOUS.match(text):
|
||||
return False
|
||||
# a sentence that is mostly one-letter/two-letter tokens is a parse artifact
|
||||
short = sum(1 for w in words if len(w.strip(".,'")) <= 2)
|
||||
if short > len(words) / 2:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _realize_prop(p, lang: str = "en") -> tuple[str, Provenance] | None:
|
||||
"""One proposition -> (faithful sentence, provenance) or None if it drops."""
|
||||
clause = _prop_to_clause(p)
|
||||
text = engine.realize(clause, lang)
|
||||
if not text or not text.strip():
|
||||
return None
|
||||
text = text.strip()
|
||||
if not text.endswith((".", "!", "?")):
|
||||
text += "."
|
||||
# capitalize first character (proper nouns / "I" already handled by grammar)
|
||||
text = text[0].upper() + text[1:]
|
||||
if not _good_sentence(text):
|
||||
return None
|
||||
return text, _provenance_from(p)
|
||||
|
||||
|
||||
def realize_document(doc: DocumentIR, lang: str = "en") -> DocumentIR:
|
||||
"""Fill every planned section's blocks with faithful, realized passages."""
|
||||
for sec in doc.sections:
|
||||
planned = sec.__dict__.get("_planned_props", [])
|
||||
block = Block(role="body")
|
||||
summary_bits: list[str] = []
|
||||
for p in planned:
|
||||
r = _realize_prop(p, lang)
|
||||
if r is None:
|
||||
continue
|
||||
text, prov = r
|
||||
block.sentences.append(text)
|
||||
block.provenance.append(prov)
|
||||
if len(summary_bits) < 1:
|
||||
# a short grounded gloss for TOC / pptx bullets
|
||||
obj = (prov.obj or "").strip().rstrip(".")
|
||||
if obj:
|
||||
summary_bits.append(obj)
|
||||
if block.sentences:
|
||||
sec.blocks.append(block)
|
||||
sec.summary = summary_bits[0] if summary_bits else ""
|
||||
# drop the transient planning payload; the IR is now self-contained
|
||||
sec.__dict__.pop("_planned_props", None)
|
||||
sec.__dict__.pop("_node", None)
|
||||
|
||||
# prune sections that realized to nothing
|
||||
doc.sections = [s for s in doc.sections if s.blocks]
|
||||
return doc
|
||||
@@ -1,73 +0,0 @@
|
||||
// audio-demo.el - Drive the native audio surface: render a tone per instrument
|
||||
// from its LEARNED signature, then render a small meaning-phrase "piece".
|
||||
// Entry point: top-level statement calls main() (same convention as the
|
||||
// examples' top-level println(run_test())).
|
||||
|
||||
fn micros_to_str(xs: [Int]) -> String {
|
||||
let n: Int = native_list_len(xs)
|
||||
let out: String = ""
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
if i > 0 { let out: String = out + "," }
|
||||
let out: String = out + int_to_str(native_list_get(xs, i))
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Render a 1.0s A4 (midi 69) tone from a signature file, print the parsed
|
||||
// partials (proving the numbers came from the engram .sig), write the WAV.
|
||||
fn render_tone(name: String, sigpath: String, outpath: String, table: [Int]) -> Int {
|
||||
let lines: [String] = sig_load(sigpath)
|
||||
let partials: [Int] = parse_micros(sig_field(lines, "partials"))
|
||||
println("[" + name + "] partials_n=" + sig_field(lines, "partials_n") + " parsed_partials_micro(scale 1e6)=" + micros_to_str(partials))
|
||||
println("[" + name + "] raw partials line from .sig = " + sig_field(lines, "partials"))
|
||||
let freq: Int = freq_of_midi(69)
|
||||
let note: [Int] = synth_from_sig(lines, freq, 1000, 900, 44100, table)
|
||||
let n: Int = native_list_len(note)
|
||||
let ok: Int = wav_write(outpath, note, n, 44100)
|
||||
println("[" + name + "] rendered " + int_to_str(n) + " samples -> " + outpath + " (write_ok=" + int_to_str(ok) + ")")
|
||||
return n
|
||||
}
|
||||
|
||||
fn run_demo() -> Int {
|
||||
let table: [Int] = sin_table()
|
||||
fs_mkdir("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out")
|
||||
|
||||
println("=== TONES: render A4 (midi 69) from each learned signature ===")
|
||||
render_tone("flute", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/flute.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-flute.wav", table)
|
||||
render_tone("clarinet", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/clarinet.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-clarinet.wav", table)
|
||||
render_tone("violin", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/violin.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-violin.wav", table)
|
||||
render_tone("piano", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/piano.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-piano.wav", table)
|
||||
render_tone("organ", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/organ.sig", "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/tone-organ.wav", table)
|
||||
|
||||
println("")
|
||||
println("=== PIECE: a 6-frame meaning phrase (incl. a NEG frame) ===")
|
||||
let frames: [[String]] = native_list_empty()
|
||||
let frames: [[String]] = native_list_append(frames, audio_frame("agent", "aff", "0.9", "0.8", "0", "s1"))
|
||||
let frames: [[String]] = native_list_append(frames, audio_frame("theme", "aff", "0.7", "0.6", "0", "s2"))
|
||||
let frames: [[String]] = native_list_append(frames, audio_frame("cause", "aff", "0.8", "0.9", "1", "s3"))
|
||||
let frames: [[String]] = native_list_append(frames, audio_frame("negation", "neg", "0.85", "0.7", "0", "s4"))
|
||||
let frames: [[String]] = native_list_append(frames, audio_frame("goal", "aff", "0.6", "0.5", "1", "s5"))
|
||||
let frames: [[String]] = native_list_append(frames, audio_frame("result", "aff", "0.95", "1.0", "0", "s6"))
|
||||
|
||||
// Print the plan so the NEG frame's minor third (+3) vs major (+4) is visible.
|
||||
let nf: Int = native_list_len(frames)
|
||||
let fi: Int = 0
|
||||
while fi < nf {
|
||||
let frame: [String] = native_list_get(frames, fi)
|
||||
let plan: [Int] = plan_note(frame)
|
||||
let pol: String = surface_get(frame, "polarity")
|
||||
let third_name: String = "major(+4)"
|
||||
if str_eq(pol, "neg") { let third_name: String = "MINOR(+3)" }
|
||||
println("frame " + int_to_str(fi) + " relation=" + surface_get(frame, "relation") + " polarity=" + pol + " -> midi=" + int_to_str(native_list_get(plan, 0)) + " dur_ms=" + int_to_str(native_list_get(plan, 1)) + " amp_pm=" + int_to_str(native_list_get(plan, 2)) + " third=" + third_name)
|
||||
let fi: Int = fi + 1
|
||||
}
|
||||
|
||||
let piano_lines: [String] = sig_load("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/piano.sig")
|
||||
let total: Int = realize_audio(frames, piano_lines, "/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/piece.wav", 44100, table)
|
||||
println("PIECE rendered " + int_to_str(total) + " samples -> /Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/piece.wav")
|
||||
return total
|
||||
}
|
||||
|
||||
println("audio-demo main returned samples=" + int_to_str(run_demo()))
|
||||
@@ -1,400 +0,0 @@
|
||||
// audio-surface.el - Native own-core additive-synthesis audio surface.
|
||||
//
|
||||
// The AUDIO efferent seam, native, no Python and no library. This renders real
|
||||
// PCM .wav bytes from instrument SIGNATURES read from engram-sourced .sig data
|
||||
// files (elp/faculty/sig/*.sig) - the partial amplitudes are NEVER literals in
|
||||
// this source; they are parsed from the learned signature at run time. That is
|
||||
// the whole proof: render-from-learned-signatures.
|
||||
//
|
||||
// EL has no float arithmetic operator (codegen emits raw int64 ops for + - * /
|
||||
// on the shared 64-bit slot) and no float-arithmetic natives - so ALL synthesis
|
||||
// math here is own-core INTEGER fixed-point. Angles use a quarter-wave sine
|
||||
// table (scale 10000) from a fixed-point Taylor series; amplitudes are parsed to
|
||||
// micro (scale 1e6) straight from the .sig text; frequencies are milliHz ints.
|
||||
//
|
||||
// Pipeline mirrors the two-stage projector (midi.py): plan_note(frame) reads a
|
||||
// frame's meaning-geometry slot-map and derives (pitch, duration, amplitude);
|
||||
// realize_audio SUPERPOSES the signature's partials (the compose op) and
|
||||
// serialises RIFF/WAVE. Same frame -> midi OR audio.
|
||||
|
||||
// -- integer decimal + string helpers -----------------------------------------
|
||||
|
||||
fn str_to_int_el(s: String) -> Int {
|
||||
let n: Int = str_len(s)
|
||||
let i: Int = 0
|
||||
let v: Int = 0
|
||||
let neg: Bool = false
|
||||
while i < n {
|
||||
let c: Int = str_char_code(s, i)
|
||||
if c == 45 { let neg: Bool = true }
|
||||
if c >= 48 {
|
||||
if c < 58 {
|
||||
let v: Int = v * 10 + (c - 48)
|
||||
}
|
||||
}
|
||||
let i: Int = i + 1
|
||||
}
|
||||
if neg { return 0 - v }
|
||||
return v
|
||||
}
|
||||
|
||||
fn parse_micro(s: String) -> Int {
|
||||
let dot: Int = str_index_of(s, ".")
|
||||
if dot < 0 {
|
||||
return str_to_int_el(s) * 1000000
|
||||
}
|
||||
let n: Int = str_len(s)
|
||||
let ipart: String = str_slice(s, 0, dot)
|
||||
let fpart: String = str_slice(s, dot + 1, n)
|
||||
let iv: Int = str_to_int_el(ipart)
|
||||
let fv: Int = 0
|
||||
let scale: Int = 100000
|
||||
let fn2: Int = str_len(fpart)
|
||||
let i: Int = 0
|
||||
while i < 6 {
|
||||
let d: Int = 0
|
||||
if i < fn2 {
|
||||
let d: Int = str_char_code(fpart, i) - 48
|
||||
}
|
||||
let fv: Int = fv + d * scale
|
||||
let scale: Int = scale / 10
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return iv * 1000000 + fv
|
||||
}
|
||||
|
||||
// -- signature (engram data file) loader ---------------------------------------
|
||||
|
||||
fn sig_load(path: String) -> [String] {
|
||||
let text: String = fs_read(path)
|
||||
return str_split(text, "\n")
|
||||
}
|
||||
|
||||
fn sig_field(lines: [String], key: String) -> String {
|
||||
let pref: String = key + ": "
|
||||
let n: Int = native_list_len(lines)
|
||||
let plen: Int = str_len(pref)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let ln: String = native_list_get(lines, i)
|
||||
if str_starts_with(ln, pref) {
|
||||
return str_slice(ln, plen, str_len(ln))
|
||||
}
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
fn parse_micros(csv: String) -> [Int] {
|
||||
let parts: [String] = str_split(csv, ",")
|
||||
let n: Int = native_list_len(parts)
|
||||
let out: [Int] = native_list_empty()
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let out: [Int] = native_list_append(out, parse_micro(native_list_get(parts, i)))
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// -- fixed-point sine (own-core, quarter-wave Taylor table, scale 10000) --------
|
||||
|
||||
fn sin_table() -> [Int] {
|
||||
let HP: Int = 1570796
|
||||
let t: [Int] = native_list_empty()
|
||||
let q: Int = 0
|
||||
while q < 257 {
|
||||
let x: Int = q * HP / 256
|
||||
let x2: Int = x * x / 1000000
|
||||
let x3: Int = x2 * x / 1000000
|
||||
let x5: Int = x3 * x2 / 1000000
|
||||
let x7: Int = x5 * x2 / 1000000
|
||||
let x9: Int = x7 * x2 / 1000000
|
||||
let s: Int = x - x3 / 6 + x5 / 120 - x7 / 5040 + x9 / 362880
|
||||
let t: [Int] = native_list_append(t, s / 100)
|
||||
let q: Int = q + 1
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
fn sin_lookup(t: [Int], phase: Int) -> Int {
|
||||
let p: Int = phase % 1024
|
||||
if p < 0 { let p: Int = p + 1024 }
|
||||
let quad: Int = p / 256
|
||||
let r: Int = p % 256
|
||||
if quad == 0 { return native_list_get(t, r) }
|
||||
if quad == 1 { return native_list_get(t, 256 - r) }
|
||||
if quad == 2 { return 0 - native_list_get(t, r) }
|
||||
return 0 - native_list_get(t, 256 - r)
|
||||
}
|
||||
|
||||
fn isqrt_int(n: Int) -> Int {
|
||||
if n <= 0 { return 0 }
|
||||
let x: Int = n
|
||||
let y: Int = (x + 1) / 2
|
||||
while y < x {
|
||||
let x: Int = y
|
||||
let y: Int = (x + n / x) / 2
|
||||
}
|
||||
return x
|
||||
}
|
||||
|
||||
// freq_of_midi: equal-tempered frequency in milliHz. 440000 mHz at midi 69.
|
||||
fn freq_of_midi(m: Int) -> Int {
|
||||
let f: Int = 440000
|
||||
if m > 69 {
|
||||
let k: Int = m - 69
|
||||
let i: Int = 0
|
||||
while i < k {
|
||||
let f: Int = f * 1059463 / 1000000
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return f
|
||||
}
|
||||
if m < 69 {
|
||||
let k: Int = 69 - m
|
||||
let i: Int = 0
|
||||
while i < k {
|
||||
let f: Int = f * 1000000 / 1059463
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return f
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
// -- envelope (ADSR), scale 1000 -----------------------------------------------
|
||||
|
||||
fn adsr_env(i: Int, total: Int, atk_n: Int, dec_n: Int, sus_pm: Int, rel_n: Int) -> Int {
|
||||
if i < atk_n {
|
||||
if atk_n == 0 { return 1000 }
|
||||
return 1000 * i / atk_n
|
||||
}
|
||||
if i < atk_n + dec_n {
|
||||
if dec_n == 0 { return sus_pm }
|
||||
return 1000 - (1000 - sus_pm) * (i - atk_n) / dec_n
|
||||
}
|
||||
let rel_start: Int = total - rel_n
|
||||
if i < rel_start {
|
||||
return sus_pm
|
||||
}
|
||||
if rel_n == 0 { return 0 }
|
||||
let left: Int = total - i
|
||||
return sus_pm * left / rel_n
|
||||
}
|
||||
|
||||
// -- note synthesis: SUPERPOSE the learned partials -> [Int] samples -----------
|
||||
fn note_samples(freq_mHz: Int, dur_ms: Int, rate: Int, partials: [Int], sumP: Int, b_micro: Int, vib_rate: Int, vib_cents: Int, atk_ms: Int, dec_ms: Int, sus_pm: Int, rel_ms: Int, amp_pm: Int, table: [Int]) -> [Int] {
|
||||
let total: Int = dur_ms * rate / 1000
|
||||
let atk_n: Int = atk_ms * rate / 1000
|
||||
let dec_n: Int = dec_ms * rate / 1000
|
||||
let rel_n: Int = rel_ms * rate / 1000
|
||||
let np: Int = native_list_len(partials)
|
||||
let half_mhz: Int = rate * 1000 / 2
|
||||
let out: [Int] = native_list_empty()
|
||||
let i: Int = 0
|
||||
while i < total {
|
||||
let acc: Int = 0
|
||||
let k: Int = 0
|
||||
while k < np {
|
||||
let harm: Int = k + 1
|
||||
let amp_k: Int = native_list_get(partials, k)
|
||||
let factor: Int = 1000000
|
||||
if b_micro > 0 {
|
||||
let val: Int = 1000000 + b_micro * harm * harm
|
||||
let factor: Int = isqrt_int(val * 1000000)
|
||||
}
|
||||
let fn_mhz: Int = freq_mHz * harm
|
||||
let fn_mhz: Int = fn_mhz * factor / 1000000
|
||||
if vib_cents > 0 {
|
||||
if vib_rate > 0 {
|
||||
let vphase: Int = i * vib_rate * 1024 / rate
|
||||
let vs: Int = sin_lookup(table, vphase)
|
||||
let vibf: Int = 1000000 + (vib_cents * vs * 833) / 10000
|
||||
let fn_mhz: Int = fn_mhz * vibf / 1000000
|
||||
}
|
||||
}
|
||||
if fn_mhz <= half_mhz {
|
||||
let phase: Int = i * fn_mhz * 1024 / (rate * 1000)
|
||||
let sv: Int = sin_lookup(table, phase)
|
||||
let acc: Int = acc + sv * amp_k / 1000000
|
||||
}
|
||||
let k: Int = k + 1
|
||||
}
|
||||
let env: Int = adsr_env(i, total, atk_n, dec_n, sus_pm, rel_n)
|
||||
let s16: Int = acc * 2800000 / sumP
|
||||
let s16: Int = s16 * env / 1000
|
||||
let s16: Int = s16 * amp_pm / 1000
|
||||
if s16 > 32767 { let s16: Int = 32767 }
|
||||
if s16 < 0 - 32767 { let s16: Int = 0 - 32767 }
|
||||
let out: [Int] = native_list_append(out, s16)
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
fn synth_from_sig(lines: [String], freq_mHz: Int, dur_ms: Int, amp_pm: Int, rate: Int, table: [Int]) -> [Int] {
|
||||
let partials: [Int] = parse_micros(sig_field(lines, "partials"))
|
||||
let np: Int = native_list_len(partials)
|
||||
let sumP: Int = 0
|
||||
let j: Int = 0
|
||||
while j < np {
|
||||
let pj: Int = native_list_get(partials, j)
|
||||
let sumP: Int = sumP + pj
|
||||
let j: Int = j + 1
|
||||
}
|
||||
if sumP <= 0 { let sumP: Int = 1000000 }
|
||||
let adsr: [String] = str_split(sig_field(lines, "adsr"), ",")
|
||||
let atk_ms: Int = parse_micro(native_list_get(adsr, 0)) / 1000
|
||||
let dec_ms: Int = parse_micro(native_list_get(adsr, 1)) / 1000
|
||||
let sus_pm: Int = parse_micro(native_list_get(adsr, 2)) / 1000
|
||||
let rel_ms: Int = parse_micro(native_list_get(adsr, 3)) / 1000
|
||||
let b_micro: Int = parse_micro(sig_field(lines, "inharmonicity_B"))
|
||||
let vib_rate: Int = str_to_int_el(sig_field(lines, "vibrato_rate_hz"))
|
||||
let vib_cents: Int = str_to_int_el(sig_field(lines, "vibrato_depth_cents"))
|
||||
return note_samples(freq_mHz, dur_ms, rate, partials, sumP, b_micro, vib_rate, vib_cents, atk_ms, dec_ms, sus_pm, rel_ms, amp_pm, table)
|
||||
}
|
||||
|
||||
// -- byte-buffer helpers (own-core, no library) --------------------------------
|
||||
|
||||
fn put_tag(buf: String, pos: Int, s: String) -> String {
|
||||
let n: Int = str_len(s)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let buf: String = __str_set_char(buf, pos + i, str_char_code(s, i))
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return buf
|
||||
}
|
||||
|
||||
fn put_u32le(buf: String, pos: Int, v: Int) -> String {
|
||||
let buf: String = __str_set_char(buf, pos, v % 256)
|
||||
let buf: String = __str_set_char(buf, pos + 1, (v / 256) % 256)
|
||||
let buf: String = __str_set_char(buf, pos + 2, (v / 65536) % 256)
|
||||
let buf: String = __str_set_char(buf, pos + 3, (v / 16777216) % 256)
|
||||
return buf
|
||||
}
|
||||
|
||||
fn put_u16le(buf: String, pos: Int, v: Int) -> String {
|
||||
let buf: String = __str_set_char(buf, pos, v % 256)
|
||||
let buf: String = __str_set_char(buf, pos + 1, (v / 256) % 256)
|
||||
return buf
|
||||
}
|
||||
|
||||
// -- WAV serializer: own-core RIFF/WAVE, PCM mono 16-bit -----------------------
|
||||
|
||||
fn wav_write(path: String, samples: [Int], n: Int, rate: Int) -> Int {
|
||||
let data_len: Int = n * 2
|
||||
let total: Int = 44 + data_len
|
||||
let buf: String = __str_alloc(total)
|
||||
let buf: String = put_tag(buf, 0, "RIFF")
|
||||
let buf: String = put_u32le(buf, 4, 36 + data_len)
|
||||
let buf: String = put_tag(buf, 8, "WAVE")
|
||||
let buf: String = put_tag(buf, 12, "fmt ")
|
||||
let buf: String = put_u32le(buf, 16, 16)
|
||||
let buf: String = put_u16le(buf, 20, 1)
|
||||
let buf: String = put_u16le(buf, 22, 1)
|
||||
let buf: String = put_u32le(buf, 24, rate)
|
||||
let buf: String = put_u32le(buf, 28, rate * 2)
|
||||
let buf: String = put_u16le(buf, 32, 2)
|
||||
let buf: String = put_u16le(buf, 34, 16)
|
||||
let buf: String = put_tag(buf, 36, "data")
|
||||
let buf: String = put_u32le(buf, 40, data_len)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let v: Int = native_list_get(samples, i)
|
||||
if v < 0 { let v: Int = v + 65536 }
|
||||
let buf: String = __str_set_char(buf, 44 + i * 2, v % 256)
|
||||
let buf: String = __str_set_char(buf, 44 + i * 2 + 1, (v / 256) % 256)
|
||||
let i: Int = i + 1
|
||||
}
|
||||
let ok: Int = fs_write_bytes(path, buf, total)
|
||||
return ok
|
||||
}
|
||||
|
||||
// -- plan: frame slot-map -> note atom (pitch, duration, amplitude) ------------
|
||||
|
||||
fn audio_frame(relation: String, polarity: String, confidence: String, importance: String, salience: String, subj_id: String) -> [String] {
|
||||
let f: [String] = native_list_empty()
|
||||
let f: [String] = native_list_append(f, "relation")
|
||||
let f: [String] = native_list_append(f, relation)
|
||||
let f: [String] = native_list_append(f, "polarity")
|
||||
let f: [String] = native_list_append(f, polarity)
|
||||
let f: [String] = native_list_append(f, "confidence")
|
||||
let f: [String] = native_list_append(f, confidence)
|
||||
let f: [String] = native_list_append(f, "importance")
|
||||
let f: [String] = native_list_append(f, importance)
|
||||
let f: [String] = native_list_append(f, "salience")
|
||||
let f: [String] = native_list_append(f, salience)
|
||||
let f: [String] = native_list_append(f, "subj_id")
|
||||
let f: [String] = native_list_append(f, subj_id)
|
||||
return f
|
||||
}
|
||||
|
||||
fn degree_offset(deg: Int) -> Int {
|
||||
if deg == 0 { return 0 }
|
||||
if deg == 1 { return 2 }
|
||||
if deg == 2 { return 4 }
|
||||
if deg == 3 { return 5 }
|
||||
if deg == 4 { return 7 }
|
||||
if deg == 5 { return 9 }
|
||||
return 11
|
||||
}
|
||||
|
||||
// returns [midi, dur_ms, amp_pm]
|
||||
fn plan_note(frame: [String]) -> [Int] {
|
||||
let relation: String = surface_get(frame, "relation")
|
||||
let polarity: String = surface_get(frame, "polarity")
|
||||
let confidence: String = surface_get(frame, "confidence")
|
||||
let importance: String = surface_get(frame, "importance")
|
||||
let salience: String = surface_get(frame, "salience")
|
||||
let rn: Int = str_len(relation)
|
||||
let csum: Int = 0
|
||||
let i: Int = 0
|
||||
while i < rn {
|
||||
let cc: Int = str_char_code(relation, i)
|
||||
let csum: Int = csum + cc
|
||||
let i: Int = i + 1
|
||||
}
|
||||
let deg: Int = csum % 7
|
||||
let third: Int = 4
|
||||
if str_eq(polarity, "neg") { let third: Int = 3 }
|
||||
let sal_oct: Int = str_to_int_el(salience)
|
||||
let doff: Int = degree_offset(deg)
|
||||
let midi: Int = 60 + sal_oct * 12 + doff + third
|
||||
let conf_micro: Int = parse_micro(confidence)
|
||||
let dur_ms: Int = 200 + conf_micro / 1000
|
||||
let imp_micro: Int = parse_micro(importance)
|
||||
let amp_pm: Int = 400 + imp_micro / 2000
|
||||
let out: [Int] = native_list_empty()
|
||||
let out: [Int] = native_list_append(out, midi)
|
||||
let out: [Int] = native_list_append(out, dur_ms)
|
||||
let out: [Int] = native_list_append(out, amp_pm)
|
||||
return out
|
||||
}
|
||||
|
||||
fn realize_audio(frames: [[String]], sig_lines: [String], path: String, rate: Int, table: [Int]) -> Int {
|
||||
let nf: Int = native_list_len(frames)
|
||||
let all: [Int] = native_list_empty()
|
||||
let count: Int = 0
|
||||
let fi: Int = 0
|
||||
while fi < nf {
|
||||
let frame: [String] = native_list_get(frames, fi)
|
||||
let plan: [Int] = plan_note(frame)
|
||||
let midi: Int = native_list_get(plan, 0)
|
||||
let dur_ms: Int = native_list_get(plan, 1)
|
||||
let amp_pm: Int = native_list_get(plan, 2)
|
||||
let freq: Int = freq_of_midi(midi)
|
||||
let note: [Int] = synth_from_sig(sig_lines, freq, dur_ms, amp_pm, rate, table)
|
||||
let nn: Int = native_list_len(note)
|
||||
let j: Int = 0
|
||||
while j < nn {
|
||||
let all: [Int] = native_list_append(all, native_list_get(note, j))
|
||||
let j: Int = j + 1
|
||||
}
|
||||
let count: Int = count + nn
|
||||
let fi: Int = fi + 1
|
||||
}
|
||||
let ok: Int = wav_write(path, all, count, rate)
|
||||
return count
|
||||
}
|
||||
@@ -1,65 +0,0 @@
|
||||
// image-demo.el - Drive the native PNG surface: plan a scene from a small
|
||||
// meaning phrase (incl. a NEG frame) and emit a byte-valid 64x64 PNG whose
|
||||
// palette is read from elp/faculty/sig/scene.basis.
|
||||
|
||||
fn img_frame(relation: String, polarity: String, confidence: String, importance: String, salience: String, subj_id: String) -> [String] {
|
||||
let f: [String] = native_list_empty()
|
||||
let f: [String] = native_list_append(f, "relation")
|
||||
let f: [String] = native_list_append(f, relation)
|
||||
let f: [String] = native_list_append(f, "polarity")
|
||||
let f: [String] = native_list_append(f, polarity)
|
||||
let f: [String] = native_list_append(f, "confidence")
|
||||
let f: [String] = native_list_append(f, confidence)
|
||||
let f: [String] = native_list_append(f, "importance")
|
||||
let f: [String] = native_list_append(f, importance)
|
||||
let f: [String] = native_list_append(f, "salience")
|
||||
let f: [String] = native_list_append(f, salience)
|
||||
let f: [String] = native_list_append(f, "subj_id")
|
||||
let f: [String] = native_list_append(f, subj_id)
|
||||
return f
|
||||
}
|
||||
|
||||
fn rgb_str(c: [Int]) -> String {
|
||||
return int_to_str(native_list_get(c, 0)) + "," + int_to_str(native_list_get(c, 1)) + "," + int_to_str(native_list_get(c, 2))
|
||||
}
|
||||
|
||||
fn run_image() -> Int {
|
||||
fs_mkdir("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out")
|
||||
let table: [Int] = crc_table()
|
||||
println("crc_table[1]=" + int_to_str(native_list_get(table, 1)) + " (expect 1996959894 / 0x77073096)")
|
||||
|
||||
let basis: [String] = basis_load("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/sig/scene.basis")
|
||||
let warm: [Int] = parse_rgb(basis_field(basis, "warm"))
|
||||
let cool: [Int] = parse_rgb(basis_field(basis, "cool"))
|
||||
let bg: [Int] = parse_rgb(basis_field(basis, "bg"))
|
||||
println("basis warm=" + rgb_str(warm) + " cool=" + rgb_str(cool) + " bg=" + rgb_str(bg) + " (read from scene.basis)")
|
||||
|
||||
let frames: [[String]] = native_list_empty()
|
||||
let frames: [[String]] = native_list_append(frames, img_frame("agent", "aff", "0.9", "0.8", "0", "s1"))
|
||||
let frames: [[String]] = native_list_append(frames, img_frame("theme", "aff", "0.7", "0.6", "1", "s2"))
|
||||
let frames: [[String]] = native_list_append(frames, img_frame("cause", "aff", "0.8", "0.9", "0", "s3"))
|
||||
let frames: [[String]] = native_list_append(frames, img_frame("negation", "neg", "0.85", "0.7", "1", "s4"))
|
||||
let frames: [[String]] = native_list_append(frames, img_frame("goal", "aff", "0.6", "0.5", "0", "s5"))
|
||||
let frames: [[String]] = native_list_append(frames, img_frame("result", "aff", "0.95", "1.0", "1", "s6"))
|
||||
|
||||
let shapes: [[Int]] = plan_scene(frames, warm, cool)
|
||||
let ns: Int = native_list_len(shapes)
|
||||
println("planned " + int_to_str(ns) + " shapes:")
|
||||
let si: Int = 0
|
||||
while si < ns {
|
||||
let sh: [Int] = native_list_get(shapes, si)
|
||||
let pol: String = surface_get(native_list_get(frames, si), "polarity")
|
||||
println(" shape " + int_to_str(si) + " type=" + int_to_str(native_list_get(sh, 0)) + " x=" + int_to_str(native_list_get(sh, 1)) + " y=" + int_to_str(native_list_get(sh, 2)) + " size=" + int_to_str(native_list_get(sh, 3)) + " rgb=" + int_to_str(native_list_get(sh, 4)) + "," + int_to_str(native_list_get(sh, 5)) + "," + int_to_str(native_list_get(sh, 6)) + " polarity=" + pol)
|
||||
let si: Int = si + 1
|
||||
}
|
||||
|
||||
let raw: [Int] = rasterize(64, 64, shapes, bg)
|
||||
println("rasterized raw (filtered scanlines) bytes=" + int_to_str(native_list_len(raw)) + " (expect 12352)")
|
||||
let png: [Int] = png_build(64, 64, raw, table)
|
||||
let plen: Int = native_list_len(png)
|
||||
let ok: Int = png_write("/Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/scene.png", png)
|
||||
println("PNG bytes=" + int_to_str(plen) + " -> /Users/will/Development/neuron-technologies/foundation/el/.claude/worktrees/agent-aaf04b0a9714c4070/elp/faculty/out/scene.png (write_ok=" + int_to_str(ok) + ")")
|
||||
return plen
|
||||
}
|
||||
|
||||
println("image-demo returned png_bytes=" + int_to_str(run_image()))
|
||||
@@ -1,412 +0,0 @@
|
||||
// image-surface.el - Native own-core raster PNG surface (the image efferent
|
||||
// twin of audio). Renders a 64x64 RGB scene deterministically from a frame's
|
||||
// meaning-geometry, then serialises a byte-valid PNG entirely own-core:
|
||||
// 8-byte magic, IHDR, IDAT (zlib STORED/uncompressed DEFLATE + Adler32), IEND,
|
||||
// with a per-chunk CRC32 computed via software xor32 (EL has no bitwise ops).
|
||||
//
|
||||
// The RGB palette basis is read from elp/faculty/sig/scene.basis (data, not
|
||||
// literals) - the same read-from-learned discipline as the audio signatures.
|
||||
// Integer-only throughout; pixels are composed functionally (painter's order)
|
||||
// so no list mutation is needed.
|
||||
|
||||
// -- small int/parse helpers (self-contained) ----------------------------------
|
||||
|
||||
fn i_str_to_int(s: String) -> Int {
|
||||
let n: Int = str_len(s)
|
||||
let i: Int = 0
|
||||
let v: Int = 0
|
||||
while i < n {
|
||||
let c: Int = str_char_code(s, i)
|
||||
if c >= 48 {
|
||||
if c < 58 {
|
||||
let v: Int = v * 10 + (c - 48)
|
||||
}
|
||||
}
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
fn basis_load(path: String) -> [String] {
|
||||
return str_split(fs_read(path), "\n")
|
||||
}
|
||||
|
||||
fn basis_field(lines: [String], key: String) -> String {
|
||||
let pref: String = key + ": "
|
||||
let n: Int = native_list_len(lines)
|
||||
let plen: Int = str_len(pref)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let ln: String = native_list_get(lines, i)
|
||||
if str_starts_with(ln, pref) {
|
||||
return str_slice(ln, plen, str_len(ln))
|
||||
}
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
fn parse_rgb(csv: String) -> [Int] {
|
||||
let parts: [String] = str_split(csv, ",")
|
||||
let out: [Int] = native_list_empty()
|
||||
let n: Int = native_list_len(parts)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let v: Int = i_str_to_int(native_list_get(parts, i))
|
||||
let out: [Int] = native_list_append(out, v)
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// -- software 32-bit XOR (no bitwise ops in EL) --------------------------------
|
||||
|
||||
fn xor32(a: Int, b: Int) -> Int {
|
||||
let r: Int = 0
|
||||
let bit: Int = 1
|
||||
let i: Int = 0
|
||||
while i < 32 {
|
||||
let abit: Int = (a / bit) % 2
|
||||
let bbit: Int = (b / bit) % 2
|
||||
if abit != bbit {
|
||||
let add: Int = bit
|
||||
let r: Int = r + add
|
||||
}
|
||||
let bit: Int = bit * 2
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// -- CRC32 (table-driven, table built with xor32) ------------------------------
|
||||
|
||||
fn crc_table() -> [Int] {
|
||||
let t: [Int] = native_list_empty()
|
||||
let n: Int = 0
|
||||
while n < 256 {
|
||||
let c: Int = n
|
||||
let k: Int = 0
|
||||
while k < 8 {
|
||||
if c % 2 == 1 {
|
||||
let h: Int = c / 2
|
||||
let c: Int = xor32(h, 3988292384)
|
||||
} else {
|
||||
let c: Int = c / 2
|
||||
}
|
||||
let k: Int = k + 1
|
||||
}
|
||||
let t: [Int] = native_list_append(t, c)
|
||||
let n: Int = n + 1
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
fn crc32_of(bytes: [Int], table: [Int]) -> Int {
|
||||
let crc: Int = 4294967295
|
||||
let n: Int = native_list_len(bytes)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let b: Int = native_list_get(bytes, i)
|
||||
let lo: Int = crc % 256
|
||||
let idx: Int = xor32(lo, b) % 256
|
||||
let tv: Int = native_list_get(table, idx)
|
||||
let hi: Int = crc / 256
|
||||
let crc: Int = xor32(hi, tv)
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return xor32(crc, 4294967295)
|
||||
}
|
||||
|
||||
// -- Adler32 (for the zlib trailer) --------------------------------------------
|
||||
|
||||
fn adler32_of(bytes: [Int]) -> Int {
|
||||
let a: Int = 1
|
||||
let b: Int = 0
|
||||
let n: Int = native_list_len(bytes)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let byte: Int = native_list_get(bytes, i)
|
||||
let a: Int = (a + byte) % 65521
|
||||
let b: Int = (b + a) % 65521
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return b * 65536 + a
|
||||
}
|
||||
|
||||
// -- byte-list append helpers --------------------------------------------------
|
||||
|
||||
fn app_u32be(dst: [Int], v: Int) -> [Int] {
|
||||
let dst: [Int] = native_list_append(dst, (v / 16777216) % 256)
|
||||
let dst: [Int] = native_list_append(dst, (v / 65536) % 256)
|
||||
let dst: [Int] = native_list_append(dst, (v / 256) % 256)
|
||||
let dst: [Int] = native_list_append(dst, v % 256)
|
||||
return dst
|
||||
}
|
||||
|
||||
fn app_tag(dst: [Int], s: String) -> [Int] {
|
||||
let n: Int = str_len(s)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let dst: [Int] = native_list_append(dst, str_char_code(s, i))
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return dst
|
||||
}
|
||||
|
||||
fn app_all(dst: [Int], src: [Int]) -> [Int] {
|
||||
let n: Int = native_list_len(src)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let dst: [Int] = native_list_append(dst, native_list_get(src, i))
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return dst
|
||||
}
|
||||
|
||||
// -- plan: frame meaning-geometry -> shape atoms -------------------------------
|
||||
// shape = [type, x, y, size, r, g, b] (type 0=rect 1=disc 2=triangle)
|
||||
|
||||
fn charsum(s: String) -> Int {
|
||||
let n: Int = str_len(s)
|
||||
let i: Int = 0
|
||||
let acc: Int = 0
|
||||
while i < n {
|
||||
let c: Int = str_char_code(s, i)
|
||||
let acc: Int = acc + c
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return acc
|
||||
}
|
||||
|
||||
fn micro_of(s: String) -> Int {
|
||||
let dot: Int = str_index_of(s, ".")
|
||||
if dot < 0 { return i_str_to_int(s) * 1000000 }
|
||||
let n: Int = str_len(s)
|
||||
let fp: String = str_slice(s, dot + 1, n)
|
||||
let ip: String = str_slice(s, 0, dot)
|
||||
let iv: Int = i_str_to_int(ip)
|
||||
let fv: Int = 0
|
||||
let scale: Int = 100000
|
||||
let fl: Int = str_len(fp)
|
||||
let i: Int = 0
|
||||
while i < 6 {
|
||||
let d: Int = 0
|
||||
if i < fl { let d: Int = str_char_code(fp, i) - 48 }
|
||||
let fv: Int = fv + d * scale
|
||||
let scale: Int = scale / 10
|
||||
let i: Int = i + 1
|
||||
}
|
||||
return iv * 1000000 + fv
|
||||
}
|
||||
|
||||
fn plan_scene(frames: [[String]], warm: [Int], cool: [Int]) -> [[Int]] {
|
||||
let shapes: [[Int]] = native_list_empty()
|
||||
let nf: Int = native_list_len(frames)
|
||||
let fi: Int = 0
|
||||
while fi < nf {
|
||||
let fr: [String] = native_list_get(frames, fi)
|
||||
let relation: String = surface_get(fr, "relation")
|
||||
let polarity: String = surface_get(fr, "polarity")
|
||||
let confidence: String = surface_get(fr, "confidence")
|
||||
let importance: String = surface_get(fr, "importance")
|
||||
let salience: String = surface_get(fr, "salience")
|
||||
// relation -> shape type
|
||||
let stype: Int = charsum(relation) % 3
|
||||
// confidence -> size (8..22)
|
||||
let cmi: Int = micro_of(confidence)
|
||||
let size: Int = 8 + cmi / 71428
|
||||
// salience -> y
|
||||
let sal: Int = i_str_to_int(salience)
|
||||
let y: Int = 6 + sal * 26
|
||||
// subj_id/index -> x
|
||||
let x: Int = 4 + (fi * 10) % 48
|
||||
// polarity -> warm/cool base color
|
||||
let br: Int = native_list_get(warm, 0)
|
||||
let bg2: Int = native_list_get(warm, 1)
|
||||
let bb: Int = native_list_get(warm, 2)
|
||||
if str_eq(polarity, "neg") {
|
||||
let br: Int = native_list_get(cool, 0)
|
||||
let bg2: Int = native_list_get(cool, 1)
|
||||
let bb: Int = native_list_get(cool, 2)
|
||||
}
|
||||
// importance -> brightness (500..1000 permille)
|
||||
let imi: Int = micro_of(importance)
|
||||
let bpm: Int = 500 + imi / 2000
|
||||
let r: Int = br * bpm / 1000
|
||||
let g: Int = bg2 * bpm / 1000
|
||||
let b: Int = bb * bpm / 1000
|
||||
let sh: [Int] = native_list_empty()
|
||||
let sh: [Int] = native_list_append(sh, stype)
|
||||
let sh: [Int] = native_list_append(sh, x)
|
||||
let sh: [Int] = native_list_append(sh, y)
|
||||
let sh: [Int] = native_list_append(sh, size)
|
||||
let sh: [Int] = native_list_append(sh, r)
|
||||
let sh: [Int] = native_list_append(sh, g)
|
||||
let sh: [Int] = native_list_append(sh, b)
|
||||
let shapes: [[Int]] = native_list_append(shapes, sh)
|
||||
let fi: Int = fi + 1
|
||||
}
|
||||
return shapes
|
||||
}
|
||||
|
||||
// covers: is (px,py) inside this shape?
|
||||
fn covers(sh: [Int], px: Int, py: Int) -> Bool {
|
||||
let stype: Int = native_list_get(sh, 0)
|
||||
let sx: Int = native_list_get(sh, 1)
|
||||
let sy: Int = native_list_get(sh, 2)
|
||||
let size: Int = native_list_get(sh, 3)
|
||||
let cx: Int = sx + size / 2
|
||||
if stype == 0 {
|
||||
if px >= sx {
|
||||
if px < sx + size {
|
||||
if py >= sy {
|
||||
if py < sy + size {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
if stype == 1 {
|
||||
let rad: Int = size / 2
|
||||
let dx: Int = px - cx
|
||||
let dy: Int = py - (sy + rad)
|
||||
if dx * dx + dy * dy <= rad * rad {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
// triangle: apex at top (sy), base at sy+size
|
||||
if py >= sy {
|
||||
if py < sy + size {
|
||||
let dyv: Int = py - sy
|
||||
let halfw: Int = dyv / 2
|
||||
let dxv: Int = px - cx
|
||||
let adx: Int = dxv
|
||||
if adx < 0 { let adx: Int = 0 - dxv }
|
||||
if adx <= halfw {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// pixel_color: painter's algorithm - last covering shape wins. Returns [r,g,b].
|
||||
fn pixel_color(px: Int, py: Int, shapes: [[Int]], bg: [Int]) -> [Int] {
|
||||
let r: Int = native_list_get(bg, 0)
|
||||
let g: Int = native_list_get(bg, 1)
|
||||
let b: Int = native_list_get(bg, 2)
|
||||
let n: Int = native_list_len(shapes)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let sh: [Int] = native_list_get(shapes, i)
|
||||
if covers(sh, px, py) {
|
||||
let r: Int = native_list_get(sh, 4)
|
||||
let g: Int = native_list_get(sh, 5)
|
||||
let b: Int = native_list_get(sh, 6)
|
||||
}
|
||||
let i: Int = i + 1
|
||||
}
|
||||
let out: [Int] = native_list_empty()
|
||||
let out: [Int] = native_list_append(out, r)
|
||||
let out: [Int] = native_list_append(out, g)
|
||||
let out: [Int] = native_list_append(out, b)
|
||||
return out
|
||||
}
|
||||
|
||||
// rasterize: build the raw (filtered) scanline byte stream, filter byte 0 / row.
|
||||
fn rasterize(w: Int, h: Int, shapes: [[Int]], bg: [Int]) -> [Int] {
|
||||
let raw: [Int] = native_list_empty()
|
||||
let y: Int = 0
|
||||
while y < h {
|
||||
let raw: [Int] = native_list_append(raw, 0)
|
||||
let x: Int = 0
|
||||
while x < w {
|
||||
let col: [Int] = pixel_color(x, y, shapes, bg)
|
||||
let raw: [Int] = native_list_append(raw, native_list_get(col, 0))
|
||||
let raw: [Int] = native_list_append(raw, native_list_get(col, 1))
|
||||
let raw: [Int] = native_list_append(raw, native_list_get(col, 2))
|
||||
let x: Int = x + 1
|
||||
}
|
||||
let y: Int = y + 1
|
||||
}
|
||||
return raw
|
||||
}
|
||||
|
||||
// zlib stream with a single STORED (uncompressed) DEFLATE block + Adler32.
|
||||
fn zlib_store(raw: [Int]) -> [Int] {
|
||||
let z: [Int] = native_list_empty()
|
||||
let z: [Int] = native_list_append(z, 120)
|
||||
let z: [Int] = native_list_append(z, 1)
|
||||
let z: [Int] = native_list_append(z, 1)
|
||||
let len: Int = native_list_len(raw)
|
||||
let nlen: Int = 65535 - len
|
||||
let z: [Int] = native_list_append(z, len % 256)
|
||||
let z: [Int] = native_list_append(z, (len / 256) % 256)
|
||||
let z: [Int] = native_list_append(z, nlen % 256)
|
||||
let z: [Int] = native_list_append(z, (nlen / 256) % 256)
|
||||
let z: [Int] = app_all(z, raw)
|
||||
let ad: Int = adler32_of(raw)
|
||||
let z: [Int] = app_u32be(z, ad)
|
||||
return z
|
||||
}
|
||||
|
||||
// append a full PNG chunk: length + (type+data) + crc32(type+data).
|
||||
fn app_chunk(png: [Int], type_and_data: [Int], table: [Int]) -> [Int] {
|
||||
let total: Int = native_list_len(type_and_data)
|
||||
let dlen: Int = total - 4
|
||||
let png: [Int] = app_u32be(png, dlen)
|
||||
let png: [Int] = app_all(png, type_and_data)
|
||||
let crc: Int = crc32_of(type_and_data, table)
|
||||
let png: [Int] = app_u32be(png, crc)
|
||||
return png
|
||||
}
|
||||
|
||||
fn png_build(w: Int, h: Int, raw: [Int], table: [Int]) -> [Int] {
|
||||
let png: [Int] = native_list_empty()
|
||||
// 8-byte signature
|
||||
let png: [Int] = native_list_append(png, 137)
|
||||
let png: [Int] = native_list_append(png, 80)
|
||||
let png: [Int] = native_list_append(png, 78)
|
||||
let png: [Int] = native_list_append(png, 71)
|
||||
let png: [Int] = native_list_append(png, 13)
|
||||
let png: [Int] = native_list_append(png, 10)
|
||||
let png: [Int] = native_list_append(png, 26)
|
||||
let png: [Int] = native_list_append(png, 10)
|
||||
// IHDR
|
||||
let ihdr: [Int] = native_list_empty()
|
||||
let ihdr: [Int] = app_tag(ihdr, "IHDR")
|
||||
let ihdr: [Int] = app_u32be(ihdr, w)
|
||||
let ihdr: [Int] = app_u32be(ihdr, h)
|
||||
let ihdr: [Int] = native_list_append(ihdr, 8)
|
||||
let ihdr: [Int] = native_list_append(ihdr, 2)
|
||||
let ihdr: [Int] = native_list_append(ihdr, 0)
|
||||
let ihdr: [Int] = native_list_append(ihdr, 0)
|
||||
let ihdr: [Int] = native_list_append(ihdr, 0)
|
||||
let png: [Int] = app_chunk(png, ihdr, table)
|
||||
// IDAT
|
||||
let z: [Int] = zlib_store(raw)
|
||||
let idat: [Int] = native_list_empty()
|
||||
let idat: [Int] = app_tag(idat, "IDAT")
|
||||
let idat: [Int] = app_all(idat, z)
|
||||
let png: [Int] = app_chunk(png, idat, table)
|
||||
// IEND
|
||||
let iend: [Int] = native_list_empty()
|
||||
let iend: [Int] = app_tag(iend, "IEND")
|
||||
let png: [Int] = app_chunk(png, iend, table)
|
||||
return png
|
||||
}
|
||||
|
||||
fn png_write(path: String, png: [Int]) -> Int {
|
||||
let n: Int = native_list_len(png)
|
||||
let buf: String = __str_alloc(n)
|
||||
let i: Int = 0
|
||||
while i < n {
|
||||
let buf: String = __str_set_char(buf, i, native_list_get(png, i))
|
||||
let i: Int = i + 1
|
||||
}
|
||||
let ok: Int = fs_write_bytes(path, buf, n)
|
||||
return ok
|
||||
}
|
||||
@@ -1,153 +0,0 @@
|
||||
// surface-profile.el - Surface profile data and accessors.
|
||||
//
|
||||
// THE NATIVE EFFERENT SEAM: surface = a pluggable PROFILE, using the exact same
|
||||
// slot-map mechanism as language-profile.el. A language profile tells the
|
||||
// realizer HOW to shape a natural-language surface (word order, morphology); a
|
||||
// SURFACE profile tells the realizer WHICH surface to project meaning onto
|
||||
// (markdown, docx, html, plain, or a non-text medium like symbolic music).
|
||||
//
|
||||
// The generalization is exact: realize_lang(form, profile) already renders a
|
||||
// SemForm parameterized by a [String] profile read via lang_get. Surface is one
|
||||
// more axis of that same profile vector. One frame (sem_frame), one plan step
|
||||
// (sem_to_spec), one render (realize) — the surface is DATA, not a code path,
|
||||
// precisely as language is data. Adding a surface means adding a profile, no
|
||||
// engine change. This is the multimodal projector, native: geometry -> any
|
||||
// surface, the efferent twin of ingest.
|
||||
//
|
||||
// Surface slot keys:
|
||||
// surface - "markdown" | "docx" | "html" | "plain" | "midi" | "image"
|
||||
// modality - "text" | "audio" | "image" | "video"
|
||||
// media_type - MIME type of the emitted surface
|
||||
// head_open - string prepended to a heading (e.g. "## " for markdown)
|
||||
// head_close - string appended to a heading (e.g. "" for markdown, "</h2>" for html)
|
||||
// emph_open - string opening emphasis (e.g. "*")
|
||||
// emph_close - string closing emphasis (e.g. "*")
|
||||
// item_mark - list-item marker (e.g. "- ")
|
||||
// para_sep - paragraph separator (e.g. "\n\n")
|
||||
//
|
||||
// For a TEXT modality the render composes these markers around the surface that
|
||||
// the EXISTING realizer produces (realize_lang / sem_realize). For a non-text
|
||||
// modality (audio/image) the profile declares modality + media_type and the
|
||||
// render dispatches to the medium projector, which reads the SAME frame's
|
||||
// geometry (its intent/affect/structure) and projects it onto sound or pixels —
|
||||
// deterministic-from-meaning, nothing invented. That dispatch point is where a
|
||||
// music profile or image profile conforms, native, no parallel layer.
|
||||
|
||||
// -- Constructor -------------------------------------------------------------
|
||||
|
||||
fn surface_profile(surface: String, modality: String, media_type: String, head_open: String, head_close: String, emph_open: String, emph_close: String, item_mark: String, para_sep: String) -> [String] {
|
||||
let r: [String] = native_list_empty()
|
||||
let r = native_list_append(r, "surface")
|
||||
let r = native_list_append(r, surface)
|
||||
let r = native_list_append(r, "modality")
|
||||
let r = native_list_append(r, modality)
|
||||
let r = native_list_append(r, "media_type")
|
||||
let r = native_list_append(r, media_type)
|
||||
let r = native_list_append(r, "head_open")
|
||||
let r = native_list_append(r, head_open)
|
||||
let r = native_list_append(r, "head_close")
|
||||
let r = native_list_append(r, head_close)
|
||||
let r = native_list_append(r, "emph_open")
|
||||
let r = native_list_append(r, emph_open)
|
||||
let r = native_list_append(r, "emph_close")
|
||||
let r = native_list_append(r, emph_close)
|
||||
let r = native_list_append(r, "item_mark")
|
||||
let r = native_list_append(r, item_mark)
|
||||
let r = native_list_append(r, "para_sep")
|
||||
let r = native_list_append(r, para_sep)
|
||||
return r
|
||||
}
|
||||
|
||||
// -- Accessor (same convention as lang_get; standalone so this is a leaf) -----
|
||||
|
||||
fn surface_get(profile: [String], key: String) -> String {
|
||||
let n: Int = native_list_len(profile)
|
||||
let i: Int = 0
|
||||
while i < n - 1 {
|
||||
let k: String = native_list_get(profile, i)
|
||||
if str_eq(k, key) {
|
||||
return native_list_get(profile, i + 1)
|
||||
}
|
||||
let i = i + 2
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
fn surface_is_text(profile: [String]) -> Bool {
|
||||
return str_eq(surface_get(profile, "modality"), "text")
|
||||
}
|
||||
|
||||
// -- Built-in TEXT surface profiles ------------------------------------------
|
||||
|
||||
// Markdown: headings with "## ", emphasis with "*", "- " list items.
|
||||
fn surface_profile_markdown() -> [String] {
|
||||
return surface_profile("markdown", "text", "text/markdown", "## ", "", "*", "*", "- ", "\n\n")
|
||||
}
|
||||
|
||||
// Plain text: no markup at all — headings become bare uppercase-free lines.
|
||||
fn surface_profile_plain() -> [String] {
|
||||
return surface_profile("plain", "text", "text/plain", "", "", "", "", " - ", "\n\n")
|
||||
}
|
||||
|
||||
// HTML: block-level heading/emphasis tags.
|
||||
fn surface_profile_html() -> [String] {
|
||||
return surface_profile("html", "text", "text/html", "<h2>", "</h2>", "<em>", "</em>", "<li>", "\n")
|
||||
}
|
||||
|
||||
// docx: WordprocessingML is structural, not inline-markup; the head/emph slots
|
||||
// carry the run/style intent that the OOXML emitter maps to <w:pStyle>. Declared
|
||||
// here so docx is a first-class surface on the same seam.
|
||||
fn surface_profile_docx() -> [String] {
|
||||
return surface_profile("docx", "text", "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "Heading2:", "", "b:", "", "bullet:", "\n")
|
||||
}
|
||||
|
||||
// -- Built-in NON-TEXT surface profiles (the multimodal seam) ----------------
|
||||
|
||||
// Symbolic music (MIDI): modality=audio. The render dispatches to the music
|
||||
// projector, which reads the SAME frame's intent/affect and projects it to
|
||||
// pitch/rhythm — deterministic-from-meaning. head/emph slots are empty because
|
||||
// the medium is not textual; media_type names the surface. A music profile
|
||||
// (scale/mode/instrument) is layered onto this by the audio agent, native.
|
||||
fn surface_profile_midi() -> [String] {
|
||||
return surface_profile("midi", "audio", "audio/midi", "", "", "", "", "", "")
|
||||
}
|
||||
|
||||
// Synthesized audio (WAV): modality=audio, peer to midi. The richer audio
|
||||
// surface — the render SUPERPOSES ingested tonal primitives (sine at f0*n per an
|
||||
// ingested instrument signature) into PCM, own-core, exactly as midi writes an
|
||||
// SMF via struct. A music profile (scale/mode/instrument/adsr) layers onto this
|
||||
// as its own [String] slot-map read by the same getter. Same frame -> midi OR
|
||||
// audio, interchangeable; this is the audio agent's native conforming point.
|
||||
fn surface_profile_audio() -> [String] {
|
||||
return surface_profile("audio", "audio", "audio/wav", "", "", "", "", "", "")
|
||||
}
|
||||
|
||||
// Image (raster): modality=image. Documented seam — the render dispatches to the
|
||||
// image projector, the efferent twin of image ingest, reading the same frame.
|
||||
fn surface_profile_image() -> [String] {
|
||||
return surface_profile("image", "image", "image/png", "", "", "", "", "", "")
|
||||
}
|
||||
|
||||
// -- Composition helpers: wrap realized TEXT with the surface's markers -------
|
||||
//
|
||||
// These take text the EXISTING realizer already produced and shape it for the
|
||||
// surface. They add NO content — pure surface typography over faithful text,
|
||||
// exactly as the language profile adds no content, only linguistic form.
|
||||
|
||||
fn surface_heading(profile: [String], text: String) -> String {
|
||||
let o: String = surface_get(profile, "head_open")
|
||||
let c: String = surface_get(profile, "head_close")
|
||||
return o + text + c
|
||||
}
|
||||
|
||||
fn surface_emph(profile: [String], text: String) -> String {
|
||||
let o: String = surface_get(profile, "emph_open")
|
||||
let c: String = surface_get(profile, "emph_close")
|
||||
return o + text + c
|
||||
}
|
||||
|
||||
// A section: a heading + a paragraph separator + the (already realized) body.
|
||||
fn surface_section(profile: [String], heading: String, body: String) -> String {
|
||||
let sep: String = surface_get(profile, "para_sep")
|
||||
return surface_heading(profile, heading) + sep + body
|
||||
}
|
||||
@@ -1,26 +0,0 @@
|
||||
// surface-profile-demo.el - ONE SemFrame, realized ONCE, projected to THREE
|
||||
// surfaces via surface profiles. Proves surface-as-profile natively: the frame
|
||||
// and the realized sentence are identical; only the surface PROFILE differs.
|
||||
|
||||
fn demo() -> String {
|
||||
// 1. The shared frame (meaning-geometry): assert(Neuron, contain, the memory).
|
||||
let frame: [String] = sem_frame("assert", "Neuron", "the memory", "")
|
||||
|
||||
// 2. REALIZE once via the EXISTING native realizer (language = a profile).
|
||||
let sentence: String = sem_realize(frame)
|
||||
|
||||
// 3. PROJECT the same realized sentence onto three surfaces (surface = a
|
||||
// profile). Same frame, same sentence, different surface — one render.
|
||||
let heading: String = "Memory"
|
||||
let md: String = surface_section(surface_profile_markdown(), heading, sentence)
|
||||
let html: String = surface_section(surface_profile_html(), heading, sentence)
|
||||
let plain: String = surface_section(surface_profile_plain(), heading, sentence)
|
||||
|
||||
// 4. Report the non-text seam: a surface profile can declare an audio/image
|
||||
// medium; the render dispatches to the medium projector on the SAME frame.
|
||||
let midi_media: String = surface_get(surface_profile_midi(), "media_type")
|
||||
|
||||
return "MD=[" + md + "] HTML=[" + html + "] PLAIN=[" + plain + "] MIDI_MEDIA=" + midi_media
|
||||
}
|
||||
|
||||
println(demo())
|
||||
@@ -0,0 +1,8 @@
|
||||
# Build + runtime artifacts — never committed.
|
||||
bin/
|
||||
# Captured media (camera frames, mic audio) and syntheses. Raw streams stay
|
||||
# LOCAL and never egress — including into git.
|
||||
out/
|
||||
# Runtime consent + resume state (local, per-machine).
|
||||
.consent.json
|
||||
.resume.json
|
||||
@@ -0,0 +1,80 @@
|
||||
# peripheral — Neuron's I/O organ (own-core, local, consent-gated)
|
||||
|
||||
The interface made physical. Two afferent senses in, one efferent voice out —
|
||||
all reached the way the agentic surface reaches any tool.
|
||||
|
||||
```
|
||||
MIC (hear) afferent device -> capture -> descriptor -> ingest -> geometry
|
||||
CAMERA (see) afferent device -> capture -> descriptor -> ingest -> scene-geometry
|
||||
SPEAKER(speak) efferent render WAV -> PLAY ALOUD out the speaker
|
||||
```
|
||||
|
||||
Closes the conversational loop: **hear (mic) -> understand (engram) -> speak (speaker)**.
|
||||
|
||||
## Rails
|
||||
- **Own-core.** macOS-native only: AVFoundation (camera/mic), CoreAudio voice-
|
||||
processing (AEC), afplay (speaker), ImageIO/CoreGraphics (frames), hand-rolled
|
||||
DSP (WAV, LPC, formant synthesis). No cloud, no heavy deps.
|
||||
- **Local-only.** Raw streams are written to `out/` and never egress. `.gitignore`
|
||||
keeps captured media out of git.
|
||||
- **Consent-gated (two locks).** A Neuron-level grant (`grant`/`revoke`) *and* the
|
||||
OS TCC permission. Sensitive senses (camera/mic) fail closed without both.
|
||||
- **Disclosed.** Every device touch prints a `[peripheral]` line on stderr.
|
||||
|
||||
## Build
|
||||
```
|
||||
swiftc -O -o bin/periph src/periph.swift \
|
||||
-framework AVFoundation -framework CoreMedia -framework Foundation \
|
||||
-framework CoreGraphics -framework ImageIO -framework CoreImage
|
||||
```
|
||||
|
||||
## Commands
|
||||
```
|
||||
periph grant|revoke <camera|mic> # Neuron-level consent
|
||||
periph status
|
||||
periph speak <file.wav> # SPEAK ALOUD (efferent)
|
||||
periph tone <out.wav> [hz] [sec] # own-core WAV synth
|
||||
periph listen <sec> <out.wav> # MIC capture (afferent), 16k mono
|
||||
periph see <out.jpg> # CAMERA one frame (afferent)
|
||||
periph feat-audio <wav> | feat-image <jpg> # capture -> compact descriptor
|
||||
periph ingest-audio|ingest-image <file> <engramURL> # descriptor -> engram node (geometry)
|
||||
periph voiceprint <voice.wav> # extract F0 + formants F1-F5
|
||||
periph imitate <voice.wav> <out.wav> # speak back in that voice (LPC resynthesis)
|
||||
periph hear-imitate <sec> <out.wav> # MIC -> signature -> imitate -> SPEAK ALOUD
|
||||
periph converse <manifest.json> [--authority F] [--barge-at S[:backchannel|:bargein]] [--resume] [--live-mic]
|
||||
```
|
||||
|
||||
## The afferent metabolism
|
||||
A capture is never shipped raw. It becomes a **compact descriptor** — the afferent
|
||||
twin of the music instrument-signature:
|
||||
- audio -> `[seconds, sr, ch, rms, peak, zcr, centroid, F0]` (~2400-6000x smaller)
|
||||
- image -> `[w, h, meanRGB, brightness, 3x3 luminance grid]` (~400000x smaller)
|
||||
- voice -> `[F0, F1..F5, bandwidths]` (11 numbers)
|
||||
|
||||
That descriptor is what the ingest organ (engram `POST /api/nodes`) turns into an
|
||||
embedded node = geometry.
|
||||
|
||||
## Voice by imitation
|
||||
`voiceprint`/`imitate` are own-core LPC (autocorrelation + Levinson-Durbin, order
|
||||
16 @ 16 kHz), formant extraction from the LPC spectral envelope, and source-filter
|
||||
resynthesis (glottal impulse train at F0 through the all-pole formant filter). A
|
||||
voice is grabbed by ear as ~a dozen numbers and spoken back — **no training, no
|
||||
stolen voice.** Measured fidelity on real speech: resynthesized formants match the
|
||||
source within 2-3%. The full phoneme->formant path for *novel* sentences is the
|
||||
speech faculty's seam (`elp` audio surface profile); this engine provides the
|
||||
formant synthesis primitive it renders through.
|
||||
|
||||
## Interruptibility (native turn-taking)
|
||||
`converse` plays the utterance as an ordered, salience-tagged **meaning-plan**
|
||||
while the mic listens (full-duplex, AEC on so it never barges in on its own voice):
|
||||
- **barge-in**: user speech -> pause on the spot (sample-accurate), not "finish the buffer."
|
||||
- **yield-or-hold**: a decision grounded in the current segment's salience + progress
|
||||
+ the interrupter's authority — YIELD (stop) or HOLD ("hang on, let me finish").
|
||||
- **backchannel** ("mm-hm"): brief/low -> keep going, resume seamlessly.
|
||||
- **resumable**: on yield the remaining plan persists (`.resume.json`); `--resume`
|
||||
picks the thread back up ("as I was saying").
|
||||
|
||||
Live full-duplex uses `--live-mic` (OS AEC). Injected `--barge-at` drives the
|
||||
decision loop deterministically for testing.
|
||||
```
|
||||
```
|
||||
@@ -0,0 +1,939 @@
|
||||
// periph.swift — Neuron's PERIPHERAL I/O organ (own-core, LOCAL, CONSENT-GATED).
|
||||
//
|
||||
// The interface made physical:
|
||||
// MIC (hear) = afferent : device -> capture -> [ingest -> geometry]
|
||||
// CAMERA (see) = afferent : device -> capture -> [ingest -> scene-geometry]
|
||||
// SPEAKER(speak) = efferent : [render WAV] -> PLAY ALOUD out the speaker
|
||||
//
|
||||
// Rails: own-core (AVFoundation / CoreAudio / afplay — all ship with macOS),
|
||||
// no cloud, no heavy deps, raw streams stay LOCAL and never egress,
|
||||
// every device access is CONSENT-GATED and DISCLOSED.
|
||||
//
|
||||
// Full-duplex CONVERSE mode implements native interruptibility: while the
|
||||
// speaker plays the utterance (a persistent, segmented meaning-plan), the mic
|
||||
// listens; on user speech it interrupts instantly, then DECIDES yield-or-hold
|
||||
// grounded in the salience of what it is mid-saying, and can RESUME the thread.
|
||||
//
|
||||
// Build: swiftc -O -o peripheral/bin/periph peripheral/src/periph.swift \
|
||||
// -framework AVFoundation -framework CoreMedia -framework Foundation
|
||||
|
||||
import Foundation
|
||||
import AVFoundation
|
||||
import CoreMedia
|
||||
import CoreGraphics
|
||||
import ImageIO
|
||||
import CoreImage
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Disclosure — every peripheral touch is announced on stderr. Nothing is silent.
|
||||
// ----------------------------------------------------------------------------
|
||||
func disclose(_ msg: String) {
|
||||
FileHandle.standardError.write(" [peripheral] \(msg)\n".data(using: .utf8)!)
|
||||
}
|
||||
func emit(_ obj: [String: Any]) { // machine-readable event on stdout (JSON line)
|
||||
if let d = try? JSONSerialization.data(withJSONObject: obj),
|
||||
let s = String(data: d, encoding: .utf8) {
|
||||
print(s)
|
||||
}
|
||||
}
|
||||
func die(_ msg: String) -> Never {
|
||||
disclose("ERROR: \(msg)")
|
||||
emit(["ok": false, "error": msg])
|
||||
exit(1)
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Consent store — Neuron's OWN gate, on top of the OS (TCC) gate. Two locks on
|
||||
// the sensitive senses. Persisted locally next to the binary's organ dir.
|
||||
// ----------------------------------------------------------------------------
|
||||
struct Consent {
|
||||
static let path: String = {
|
||||
let dir = ProcessInfo.processInfo.environment["PERIPH_HOME"]
|
||||
?? FileManager.default.currentDirectoryPath + "/peripheral"
|
||||
return dir + "/.consent.json"
|
||||
}()
|
||||
|
||||
static func load() -> [String: Bool] {
|
||||
guard let d = FileManager.default.contents(atPath: path),
|
||||
let o = try? JSONSerialization.jsonObject(with: d) as? [String: Bool]
|
||||
else { return ["camera": false, "mic": false] }
|
||||
return o
|
||||
}
|
||||
static func save(_ g: [String: Bool]) {
|
||||
let d = try! JSONSerialization.data(withJSONObject: g, options: [.prettyPrinted])
|
||||
try? d.write(to: URL(fileURLWithPath: path))
|
||||
}
|
||||
// Neuron-level gate. Sensitive senses (camera/mic) require an explicit grant.
|
||||
static func require(_ device: String) {
|
||||
let g = load()
|
||||
if g[device] != true {
|
||||
die("CONSENT DENIED for '\(device)'. The user has not granted this sense. " +
|
||||
"Run: periph grant \(device) (raw streams stay local, never egress).")
|
||||
}
|
||||
disclose("consent OK (Neuron-level) for '\(device)' — local only, never egresses.")
|
||||
}
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// OS (TCC) permission — the second lock. AVFoundation prompts the user the first
|
||||
// time; if denied, we fail cleanly rather than hang.
|
||||
// ----------------------------------------------------------------------------
|
||||
func requireOSAccess(_ media: AVMediaType, _ label: String) {
|
||||
let status = AVCaptureDevice.authorizationStatus(for: media)
|
||||
switch status {
|
||||
case .authorized:
|
||||
disclose("consent OK (OS/TCC) for \(label).")
|
||||
return
|
||||
case .notDetermined:
|
||||
disclose("requesting OS permission for \(label) (first use) — user must grant...")
|
||||
let sem = DispatchSemaphore(value: 0)
|
||||
var ok = false
|
||||
AVCaptureDevice.requestAccess(for: media) { granted in ok = granted; sem.signal() }
|
||||
_ = sem.wait(timeout: .now() + 30)
|
||||
if !ok { die("OS permission for \(label) was not granted.") }
|
||||
disclose("consent OK (OS/TCC) for \(label).")
|
||||
case .denied, .restricted:
|
||||
die("OS permission for \(label) is DENIED in System Settings > Privacy. " +
|
||||
"Grant it to the controlling terminal/app, then retry.")
|
||||
@unknown default:
|
||||
die("unknown OS permission state for \(label).")
|
||||
}
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Own-core WAV writer (16-bit PCM). No library — proves we own the medium.
|
||||
// ----------------------------------------------------------------------------
|
||||
func writeWav(_ url: URL, samples: [Int16], sampleRate: Int, channels: Int = 1) {
|
||||
var data = Data()
|
||||
func u32(_ v: UInt32) { var x = v.littleEndian; data.append(Data(bytes: &x, count: 4)) }
|
||||
func u16(_ v: UInt16) { var x = v.littleEndian; data.append(Data(bytes: &x, count: 2)) }
|
||||
let bytesPerSample = 2
|
||||
let dataBytes = samples.count * bytesPerSample
|
||||
let byteRate = sampleRate * channels * bytesPerSample
|
||||
data.append("RIFF".data(using: .ascii)!); u32(UInt32(36 + dataBytes))
|
||||
data.append("WAVE".data(using: .ascii)!)
|
||||
data.append("fmt ".data(using: .ascii)!); u32(16); u16(1); u16(UInt16(channels))
|
||||
u32(UInt32(sampleRate)); u32(UInt32(byteRate))
|
||||
u16(UInt16(channels * bytesPerSample)); u16(16)
|
||||
data.append("data".data(using: .ascii)!); u32(UInt32(dataBytes))
|
||||
for s in samples { var x = s.littleEndian; data.append(Data(bytes: &x, count: 2)) }
|
||||
try? data.write(to: url)
|
||||
}
|
||||
|
||||
// Read a WAV's basic geometry (own-core header parse). Walks chunks to find
|
||||
// 'fmt ' and 'data' — robust to JUNK/FLLR padding chunks (AVAudioRecorder emits them).
|
||||
func wavInfo(_ path: String) -> (sampleRate: Int, channels: Int, bits: Int, frames: Int)? {
|
||||
guard let d = FileManager.default.contents(atPath: path), d.count > 44 else { return nil }
|
||||
func rd16(_ o: Int) -> Int { Int(d[o]) | (Int(d[o+1]) << 8) }
|
||||
func rd32(_ o: Int) -> Int { Int(d[o]) | (Int(d[o+1])<<8) | (Int(d[o+2])<<16) | (Int(d[o+3])<<24) }
|
||||
var channels = 0, sampleRate = 0, bits = 0, dataSize = 0
|
||||
var o = 12
|
||||
while o + 8 <= d.count {
|
||||
let id = String(bytes: d[o..<o+4], encoding: .ascii) ?? ""
|
||||
let sz = rd32(o+4)
|
||||
if id == "fmt " && o + 24 <= d.count {
|
||||
channels = rd16(o+10); sampleRate = rd32(o+12); bits = rd16(o+22)
|
||||
} else if id == "data" {
|
||||
dataSize = min(sz, d.count - (o+8))
|
||||
}
|
||||
o += 8 + sz + (sz & 1)
|
||||
}
|
||||
let frames = (channels > 0 && bits > 0) ? dataSize / (channels * bits/8) : 0
|
||||
return (sampleRate, channels, bits, frames)
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// SPEAKER (efferent) — play a WAV ALOUD. Own-core: afplay ships with macOS.
|
||||
// ----------------------------------------------------------------------------
|
||||
func speak(_ wavPath: String) {
|
||||
guard FileManager.default.fileExists(atPath: wavPath) else { die("no such file: \(wavPath)") }
|
||||
disclose("SPEAKER: playing '\(wavPath)' ALOUD out the local speaker (efferent).")
|
||||
let p = Process()
|
||||
p.executableURL = URL(fileURLWithPath: "/usr/bin/afplay")
|
||||
p.arguments = [wavPath]
|
||||
try? p.run(); p.waitUntilExit()
|
||||
let ok = p.terminationStatus == 0
|
||||
disclose(ok ? "SPEAKER: done — Neuron spoke aloud." : "SPEAKER: afplay failed.")
|
||||
if let i = wavInfo(wavPath) {
|
||||
emit(["ok": ok, "op": "speak", "file": wavPath, "played_aloud": ok,
|
||||
"sample_rate": i.sampleRate, "channels": i.channels,
|
||||
"seconds": Double(i.frames)/Double(max(i.sampleRate,1))])
|
||||
} else {
|
||||
emit(["ok": ok, "op": "speak", "file": wavPath, "played_aloud": ok])
|
||||
}
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// MIC (afferent) — capture N seconds -> 16k mono 16-bit WAV (formant-ready).
|
||||
// ----------------------------------------------------------------------------
|
||||
func listen(seconds: Double, out: String) {
|
||||
Consent.require("mic")
|
||||
requireOSAccess(.audio, "microphone")
|
||||
disclose("MIC: capturing \(seconds)s -> '\(out)' (16 kHz mono, LOCAL, never egresses).")
|
||||
|
||||
let url = URL(fileURLWithPath: out)
|
||||
let settings: [String: Any] = [
|
||||
AVFormatIDKey: kAudioFormatLinearPCM,
|
||||
AVSampleRateKey: 16000.0,
|
||||
AVNumberOfChannelsKey: 1,
|
||||
AVLinearPCMBitDepthKey: 16,
|
||||
AVLinearPCMIsFloatKey: false,
|
||||
AVLinearPCMIsBigEndianKey: false,
|
||||
]
|
||||
guard let rec = try? AVAudioRecorder(url: url, settings: settings) else {
|
||||
die("could not open the microphone recorder.")
|
||||
}
|
||||
rec.record()
|
||||
Thread.sleep(forTimeInterval: seconds)
|
||||
rec.stop()
|
||||
// let the file flush
|
||||
Thread.sleep(forTimeInterval: 0.1)
|
||||
if let i = wavInfo(out) {
|
||||
disclose("MIC: captured \(i.frames) frames @ \(i.sampleRate)Hz — ready to hand to the ingest organ.")
|
||||
emit(["ok": true, "op": "listen", "file": out, "sample_rate": i.sampleRate,
|
||||
"channels": i.channels, "frames": i.frames,
|
||||
"seconds": Double(i.frames)/Double(max(i.sampleRate,1)),
|
||||
"next": "ingest -> phonetic/voice geometry"])
|
||||
} else {
|
||||
die("mic capture produced no readable WAV.")
|
||||
}
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// CAMERA (afferent) — capture ONE frame -> JPEG on disk.
|
||||
// ----------------------------------------------------------------------------
|
||||
// Grab one video frame via AVCaptureVideoDataOutput (CLI-safe; no KVO/photo classes).
|
||||
final class FrameGrabber: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate {
|
||||
let sem = DispatchSemaphore(value: 0)
|
||||
var cgImage: CGImage?
|
||||
var seen = 0
|
||||
let cictx = CIContext(options: nil)
|
||||
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer,
|
||||
from connection: AVCaptureConnection) {
|
||||
seen += 1
|
||||
if cgImage != nil || seen < 5 { return } // let exposure settle a few frames
|
||||
guard let pb = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
|
||||
let ci = CIImage(cvPixelBuffer: pb)
|
||||
cgImage = cictx.createCGImage(ci, from: ci.extent)
|
||||
sem.signal()
|
||||
}
|
||||
}
|
||||
func see(out: String) {
|
||||
Consent.require("camera")
|
||||
requireOSAccess(.video, "camera")
|
||||
disclose("CAMERA: capturing one frame -> '\(out)' (LOCAL, never egresses).")
|
||||
|
||||
let session = AVCaptureSession()
|
||||
session.sessionPreset = .photo
|
||||
guard let device = AVCaptureDevice.default(for: .video),
|
||||
let input = try? AVCaptureDeviceInput(device: device),
|
||||
session.canAddInput(input) else { die("no camera device available.") }
|
||||
session.addInput(input)
|
||||
let output = AVCaptureVideoDataOutput()
|
||||
output.alwaysDiscardsLateVideoFrames = true
|
||||
let grabber = FrameGrabber()
|
||||
output.setSampleBufferDelegate(grabber, queue: DispatchQueue(label: "periph.cam"))
|
||||
guard session.canAddOutput(output) else { die("cannot add video output.") }
|
||||
session.addOutput(output)
|
||||
session.startRunning()
|
||||
|
||||
if grabber.sem.wait(timeout: .now() + 10) == .timedOut { session.stopRunning(); die("camera capture timed out.") }
|
||||
session.stopRunning()
|
||||
|
||||
guard let cg = grabber.cgImage,
|
||||
let dst = CGImageDestinationCreateWithURL(URL(fileURLWithPath: out) as CFURL,
|
||||
"public.jpeg" as CFString, 1, nil)
|
||||
else { die("camera returned no frame.") }
|
||||
CGImageDestinationAddImage(dst, cg, nil)
|
||||
guard CGImageDestinationFinalize(dst) else { die("could not write JPEG.") }
|
||||
let bytes = ((try? FileManager.default.attributesOfItem(atPath: out))?[.size] as? Int) ?? 0
|
||||
disclose("CAMERA: wrote \(cg.width)x\(cg.height) frame (\(bytes) bytes) — ready for scene-geometry ingest.")
|
||||
emit(["ok": true, "op": "see", "file": out, "width": cg.width, "height": cg.height,
|
||||
"bytes": bytes, "next": "ingest -> scene-geometry"])
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// FEAT — the afferent METABOLISM: a raw capture becomes a COMPACT descriptor
|
||||
// (a few dozen numbers), the mirror of the efferent signature. This is what
|
||||
// gets handed to the ingest organ as geometry — NOT the raw stream. Own-core.
|
||||
// ============================================================================
|
||||
|
||||
// Read all 16-bit PCM samples from a WAV (own-core).
|
||||
func readWavSamples(_ path: String) -> (samples: [Double], sr: Int, ch: Int)? {
|
||||
guard let d = FileManager.default.contents(atPath: path), d.count > 44 else { return nil }
|
||||
func rd16(_ o: Int) -> Int { Int(d[o]) | (Int(d[o+1]) << 8) }
|
||||
func rd32(_ o: Int) -> Int { Int(d[o]) | (Int(d[o+1])<<8) | (Int(d[o+2])<<16) | (Int(d[o+3])<<24) }
|
||||
var ch = 0, sr = 0, bits = 0
|
||||
var o = 12
|
||||
while o + 8 <= d.count {
|
||||
let id = String(bytes: d[o..<o+4], encoding: .ascii) ?? ""
|
||||
let sz = rd32(o+4)
|
||||
if id == "fmt " && o + 24 <= d.count { ch = rd16(o+10); sr = rd32(o+12); bits = rd16(o+22) }
|
||||
if id == "data" {
|
||||
guard bits == 16, ch > 0 else { return nil }
|
||||
var samples = [Double](); let start = o + 8
|
||||
let end = min(start + sz, d.count - 1)
|
||||
var i = start
|
||||
while i + 1 < end {
|
||||
var v = Int(rd16(i)); if v >= 32768 { v -= 65536 }
|
||||
samples.append(Double(v) / 32768.0)
|
||||
i += 2 * ch // take channel 0 if stereo
|
||||
}
|
||||
return (samples, sr, ch)
|
||||
}
|
||||
o += 8 + sz + (sz & 1)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Audio descriptor = compact sound/voice signature (energy, ZCR, centroid, F0).
|
||||
// The seed for phonetic geometry + the hear->imitate voice-signature.
|
||||
func computeAudio(_ path: String) -> (content: String, vector: [Double], extra: [String: Any]) {
|
||||
guard let (s, sr, ch) = readWavSamples(path), !s.isEmpty else { die("cannot read PCM from \(path)") }
|
||||
let n = s.count
|
||||
let seconds = Double(n) / Double(sr)
|
||||
var sumsq = 0.0, peak = 0.0, zc = 0.0
|
||||
for i in 0..<n {
|
||||
sumsq += s[i]*s[i]; peak = max(peak, abs(s[i]))
|
||||
if i > 0 && (s[i-1] < 0) != (s[i] < 0) { zc += 1 }
|
||||
}
|
||||
let rms = (sumsq / Double(n)).squareRoot()
|
||||
let zcr = zc / Double(n) * Double(sr) // ~2*dominant freq for tonal
|
||||
// Spectral centroid via a coarse DFT on a mid window (own-core).
|
||||
let W = min(2048, n); let off = max(0, (n - W)/2)
|
||||
var num = 0.0, den = 0.0
|
||||
let bins = 64
|
||||
for k in 1..<bins {
|
||||
let f = Double(k) * Double(sr) / Double(2*bins)
|
||||
var re = 0.0, im = 0.0
|
||||
for j in 0..<W {
|
||||
let ang = -2*Double.pi*Double(k)*Double(j)/Double(2*bins)
|
||||
re += s[off+j]*cos(ang); im += s[off+j]*sin(ang)
|
||||
}
|
||||
let mag = (re*re+im*im).squareRoot()
|
||||
num += f*mag; den += mag
|
||||
}
|
||||
let centroid = den > 0 ? num/den : 0
|
||||
// F0 via autocorrelation (voice pitch) over plausible speech range 70-400 Hz.
|
||||
var bestLag = 0; var bestCorr = 0.0
|
||||
let lagMin = sr/400, lagMax = min(sr/70, n-1)
|
||||
if lagMax > lagMin {
|
||||
for lag in lagMin...lagMax {
|
||||
var c = 0.0
|
||||
var i = 0; while i + lag < min(n, off+W) { c += s[off+i]*s[off+i+lag]; i += 1 }
|
||||
if c > bestCorr { bestCorr = c; bestLag = lag }
|
||||
}
|
||||
}
|
||||
let f0 = bestLag > 0 ? Double(sr)/Double(bestLag) : 0
|
||||
let vector: [Double] = [seconds, Double(sr), Double(ch), rms, peak, zcr, centroid, f0]
|
||||
let content = String(format:
|
||||
"Heard sound (afferent, mic): %.2fs at %dHz. RMS energy %.3f, peak %.3f, " +
|
||||
"zero-crossing rate %.0fHz, spectral centroid %.0fHz, estimated voice pitch F0 %.0fHz. " +
|
||||
"Compact voice/sound signature (%d numbers) — phonetic geometry + hear-to-imitate seed.",
|
||||
seconds, sr, rms, peak, zcr, centroid, f0, vector.count)
|
||||
disclose("FEAT(audio): \(vector.count)-number signature vs \(n) raw samples (~\(n/max(vector.count,1))x compression).")
|
||||
return (content, vector, ["f0_hz": f0, "centroid_hz": centroid, "zcr_hz": zcr,
|
||||
"rms": rms, "seconds": seconds, "raw_samples": n])
|
||||
}
|
||||
func featAudio(_ path: String) {
|
||||
let r = computeAudio(path)
|
||||
var out: [String: Any] = ["ok": true, "op": "feat-audio", "file": path,
|
||||
"vector": r.vector, "content": r.content,
|
||||
"ingest": ["node_type": "Observation", "tier": "Episodic", "content": r.content]]
|
||||
r.extra.forEach { out[$0] = $1 }
|
||||
emit(out)
|
||||
}
|
||||
|
||||
// Image descriptor = compact scene-geometry (dims, brightness, region grid).
|
||||
func computeImage(_ path: String) -> (content: String, vector: [Double], extra: [String: Any]) {
|
||||
guard let src = CGImageSourceCreateWithURL(URL(fileURLWithPath: path) as CFURL, nil),
|
||||
let img = CGImageSourceCreateImageAtIndex(src, 0, nil) else { die("cannot decode image \(path)") }
|
||||
let w = img.width, h = img.height
|
||||
let cs = CGColorSpaceCreateDeviceRGB()
|
||||
let bpr = w * 4
|
||||
var buf = [UInt8](repeating: 0, count: h * bpr)
|
||||
guard let ctx = CGContext(data: &buf, width: w, height: h, bitsPerComponent: 8,
|
||||
bytesPerRow: bpr, space: cs,
|
||||
bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue) else {
|
||||
die("cannot rasterize image")
|
||||
}
|
||||
ctx.draw(img, in: CGRect(x: 0, y: 0, width: w, height: h))
|
||||
// 3x3 region average luminance + overall average color.
|
||||
var rAvg = 0.0, gAvg = 0.0, bAvg = 0.0
|
||||
var grid = [Double](repeating: 0, count: 9); var gridN = [Int](repeating: 0, count: 9)
|
||||
let step = max(1, (w*h)/40000) // subsample for speed
|
||||
var count = 0; var idx = 0
|
||||
while idx < w*h {
|
||||
let x = idx % w, y = idx / w
|
||||
let p = y*bpr + x*4
|
||||
let r = Double(buf[p]), g = Double(buf[p+1]), b = Double(buf[p+2])
|
||||
rAvg += r; gAvg += g; bAvg += b; count += 1
|
||||
let cell = (min(2, y*3/h))*3 + min(2, x*3/w)
|
||||
grid[cell] += 0.299*r + 0.587*g + 0.114*b; gridN[cell] += 1
|
||||
idx += step
|
||||
}
|
||||
if count == 0 { die("no pixels sampled") }
|
||||
rAvg /= Double(count); gAvg /= Double(count); bAvg /= Double(count)
|
||||
for i in 0..<9 { grid[i] = gridN[i] > 0 ? grid[i]/Double(gridN[i]) : 0 }
|
||||
let bright = (0.299*rAvg + 0.587*gAvg + 0.114*bAvg)/255.0
|
||||
let vector = [Double(w), Double(h), rAvg/255, gAvg/255, bAvg/255, bright] + grid.map { $0/255 }
|
||||
let content = String(format:
|
||||
"Saw scene (afferent, camera): %dx%d frame. Mean color rgb(%.0f,%.0f,%.0f), " +
|
||||
"brightness %.2f. 3x3 luminance grid [%.0f %.0f %.0f / %.0f %.0f %.0f / %.0f %.0f %.0f]. " +
|
||||
"Compact scene-geometry (%d numbers) vs %d pixel-channels.",
|
||||
w, h, rAvg, gAvg, bAvg, bright,
|
||||
grid[0],grid[1],grid[2],grid[3],grid[4],grid[5],grid[6],grid[7],grid[8],
|
||||
vector.count, w*h*3)
|
||||
disclose("FEAT(image): \(vector.count)-number scene-geometry vs \(w*h*3) pixel-channels (~\(w*h*3/max(vector.count,1))x).")
|
||||
return (content, vector, ["width": w, "height": h, "brightness": bright])
|
||||
}
|
||||
func featImage(_ path: String) {
|
||||
let r = computeImage(path)
|
||||
var out: [String: Any] = ["ok": true, "op": "feat-image", "file": path,
|
||||
"vector": r.vector, "content": r.content,
|
||||
"ingest": ["node_type": "Observation", "tier": "Episodic", "content": r.content]]
|
||||
r.extra.forEach { out[$0] = $1 }
|
||||
emit(out)
|
||||
}
|
||||
|
||||
// The afferent WIRE — hand a capture's descriptor to the ingest organ (engram),
|
||||
// where it becomes an embedded node = GEOMETRY. Own-core URLSession POST.
|
||||
// LOCAL only: point at a local engram; raw stream never leaves the machine.
|
||||
func postNode(engramURL: String, content: String, label: String, tags: [String]) -> String? {
|
||||
guard let url = URL(string: engramURL + "/api/nodes") else { return nil }
|
||||
let body: [String: Any] = ["content": content, "node_type": "Observation",
|
||||
"label": label, "tier": "Episodic",
|
||||
"salience": 0.7, "importance": 0.6, "confidence": 0.9,
|
||||
"tags": tags]
|
||||
var req = URLRequest(url: url); req.httpMethod = "POST"
|
||||
req.setValue("application/json", forHTTPHeaderField: "Content-Type")
|
||||
req.httpBody = try? JSONSerialization.data(withJSONObject: body)
|
||||
let sem = DispatchSemaphore(value: 0); var out: String?
|
||||
URLSession.shared.dataTask(with: req) { data, _, _ in
|
||||
if let d = data { out = String(data: d, encoding: .utf8) }
|
||||
sem.signal()
|
||||
}.resume()
|
||||
_ = sem.wait(timeout: .now() + 15)
|
||||
return out
|
||||
}
|
||||
func ingest(_ path: String, kind: String, engramURL: String) {
|
||||
let r = kind == "audio" ? computeAudio(path) : computeImage(path)
|
||||
let label = kind == "audio" ? "heard:mic" : "saw:camera"
|
||||
disclose("INGEST: handing \(kind) descriptor to the ingest organ at \(engramURL) (LOCAL) -> geometry.")
|
||||
guard let resp = postNode(engramURL: engramURL, content: r.content, label: label,
|
||||
tags: ["peripheral", kind == "audio" ? "afferent-mic" : "afferent-camera"]) else {
|
||||
die("ingest POST failed (no local engram at \(engramURL)?)")
|
||||
}
|
||||
// pull the node id out of the response (own-core, tolerant)
|
||||
var nodeId = ""
|
||||
if let d = resp.data(using: .utf8),
|
||||
let o = try? JSONSerialization.jsonObject(with: d) as? [String: Any] {
|
||||
nodeId = (o["id"] as? String) ?? (o["node_id"] as? String) ?? ""
|
||||
}
|
||||
disclose("INGEST: landed as node \(nodeId.isEmpty ? "(see response)" : nodeId) — the capture is now geometry in the engram.")
|
||||
emit(["ok": !nodeId.isEmpty, "op": "ingest-\(kind)", "file": path,
|
||||
"node_id": nodeId, "engram_response": resp, "content": r.content,
|
||||
"vector": r.vector])
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// VOICE BY IMITATION — hear a voice, grab its compact SIGNATURE (pitch +
|
||||
// formants F1-F5 via LPC), and speak back in that voice by source-filter
|
||||
// resynthesis. Own-core DSP (physics), no training, no stolen voice. The
|
||||
// afferent twin of the music instrument-signature: a voice = a few dozen
|
||||
// numbers, not a corpus.
|
||||
// ============================================================================
|
||||
|
||||
func hamming(_ x: [Double]) -> [Double] {
|
||||
let n = x.count; if n < 2 { return x }
|
||||
return (0..<n).map { x[$0] * (0.54 - 0.46*cos(2*Double.pi*Double($0)/Double(n-1))) }
|
||||
}
|
||||
func autocorr(_ x: [Double], _ p: Int) -> [Double] {
|
||||
var r = [Double](repeating: 0, count: p+1)
|
||||
for lag in 0...p { var s = 0.0; var i = lag; while i < x.count { s += x[i]*x[i-lag]; i += 1 }; r[lag] = s }
|
||||
return r
|
||||
}
|
||||
// Levinson-Durbin -> LPC coeffs a[0..p] (A(z)=1+sum a[k]z^-k) and residual energy.
|
||||
func levinson(_ r: [Double], _ p: Int) -> (a: [Double], err: Double) {
|
||||
var a = [Double](repeating: 0, count: p+1); a[0] = 1
|
||||
var err = r[0]
|
||||
if err <= 0 { return (a, 0) }
|
||||
for i in 1...p {
|
||||
var acc = r[i]
|
||||
if i > 1 { for j in 1..<i { acc += a[j]*r[i-j] } }
|
||||
let k = -acc/err
|
||||
var na = a; na[i] = k
|
||||
if i > 1 { for j in 1..<i { na[j] = a[j] + k*a[i-j] } }
|
||||
a = na; err *= (1 - k*k)
|
||||
if err <= 0 { break }
|
||||
}
|
||||
return (a, err)
|
||||
}
|
||||
// Formant peaks from the LPC all-pole spectral envelope.
|
||||
func formants(_ a: [Double], sr: Int) -> [(f: Double, bw: Double)] {
|
||||
let p = a.count - 1
|
||||
let steps = 512
|
||||
var mag = [Double](repeating: 0, count: steps)
|
||||
for s in 0..<steps {
|
||||
let w = Double.pi * Double(s) / Double(steps) // 0..pi -> 0..sr/2
|
||||
var re = 0.0, im = 0.0
|
||||
for k in 0...p { re += a[k]*cos(w*Double(k)); im -= a[k]*sin(w*Double(k)) }
|
||||
mag[s] = 1.0 / max((re*re+im*im).squareRoot(), 1e-9)
|
||||
}
|
||||
var peaks: [(f: Double, bw: Double)] = []
|
||||
for s in 1..<(steps-1) where mag[s] > mag[s-1] && mag[s] >= mag[s+1] {
|
||||
let f = Double(s) * Double(sr) / 2 / Double(steps)
|
||||
if f > 150 && f < 5200 {
|
||||
// crude bandwidth: width where magnitude falls to peak/sqrt(2)
|
||||
let thr = mag[s]/1.4142
|
||||
var lo = s; while lo > 0 && mag[lo] > thr { lo -= 1 }
|
||||
var hi = s; while hi < steps-1 && mag[hi] > thr { hi += 1 }
|
||||
let bw = Double(hi-lo) * Double(sr) / 2 / Double(steps)
|
||||
peaks.append((f, bw))
|
||||
}
|
||||
}
|
||||
return Array(peaks.prefix(5))
|
||||
}
|
||||
func pitchOf(_ frame: [Double], sr: Int) -> Double {
|
||||
let n = frame.count
|
||||
let lagMin = sr/400, lagMax = min(sr/70, n-1)
|
||||
if lagMax <= lagMin { return 0 }
|
||||
var r0 = 0.0; for v in frame { r0 += v*v }
|
||||
if r0 < 1e-5 { return 0 }
|
||||
var bestLag = 0; var best = 0.0
|
||||
for lag in lagMin...lagMax { var c = 0.0; var i = lag; while i < n { c += frame[i]*frame[i-lag]; i += 1 }; if c > best { best = c; bestLag = lag } }
|
||||
return (best / r0 > 0.30 && bestLag > 0) ? Double(sr)/Double(bestLag) : 0 // voiced?
|
||||
}
|
||||
|
||||
let LPC_ORDER = 16
|
||||
let FRAME = 400 // 25ms @16k
|
||||
let HOP = 160 // 10ms
|
||||
|
||||
// Extract Will's voice-signature: averaged F0 + formants over voiced frames.
|
||||
func voiceprint(_ path: String) -> (f0: Double, f0lo: Double, f0hi: Double, formants: [(Double,Double)], content: String) {
|
||||
guard let (x, sr, _) = readWavSamples(path), x.count > FRAME else { die("cannot read speech from \(path)") }
|
||||
var f0s: [Double] = []
|
||||
var fbank: [[Double]] = [[],[],[],[],[]]
|
||||
var bbank: [[Double]] = [[],[],[],[],[]]
|
||||
var pos = 0
|
||||
while pos + FRAME <= x.count {
|
||||
let raw = Array(x[pos..<pos+FRAME])
|
||||
let f0 = pitchOf(raw, sr: sr)
|
||||
if f0 > 0 { // voiced frame only
|
||||
f0s.append(f0)
|
||||
let r = autocorr(hamming(raw), LPC_ORDER)
|
||||
if r[0] > 1e-6 {
|
||||
let (a, _) = levinson(r, LPC_ORDER)
|
||||
let fs = formants(a, sr: sr)
|
||||
for (i, fm) in fs.enumerated() where i < 5 { fbank[i].append(fm.f); bbank[i].append(fm.bw) }
|
||||
}
|
||||
}
|
||||
pos += HOP
|
||||
}
|
||||
func med(_ v: [Double]) -> Double { v.isEmpty ? 0 : v.sorted()[v.count/2] }
|
||||
let f0med = med(f0s)
|
||||
let f0lo = f0s.isEmpty ? 0 : f0s.sorted().first!
|
||||
let f0hi = f0s.isEmpty ? 0 : f0s.sorted().last!
|
||||
var forms: [(Double,Double)] = []
|
||||
for i in 0..<5 where !fbank[i].isEmpty { forms.append((med(fbank[i]), med(bbank[i]))) }
|
||||
let fstr = forms.map { String(format:"%.0f", $0.0) }.joined(separator: "/")
|
||||
let content = String(format:
|
||||
"Voice-signature (afferent, heard a voice): pitch F0 %.0fHz (range %.0f-%.0fHz), " +
|
||||
"formants F1-F5 = %@ Hz. Compact voiceprint (%d numbers) — grabbed by ear for imitation, not trained.",
|
||||
f0med, f0lo, f0hi, fstr, 1 + forms.count*2)
|
||||
return (f0med, f0lo, f0hi, forms, content)
|
||||
}
|
||||
|
||||
// IMITATE: LPC analysis-resynthesis. Reconstruct the heard voice from its
|
||||
// per-frame filter model + pitch — the voice rebuilt from its signature.
|
||||
func imitate(inPath: String, outPath: String) {
|
||||
guard let (x, sr, _) = readWavSamples(inPath), x.count > FRAME else { die("cannot read speech from \(inPath)") }
|
||||
var out = [Double](repeating: 0, count: x.count)
|
||||
var state = [Double](repeating: 0, count: LPC_ORDER) // past outputs
|
||||
var phase = 0.0
|
||||
var lastF0 = 0.0
|
||||
var pos = 0
|
||||
while pos + FRAME <= x.count {
|
||||
let raw = Array(x[pos..<pos+FRAME])
|
||||
let r = autocorr(hamming(raw), LPC_ORDER)
|
||||
let f0 = pitchOf(raw, sr: sr)
|
||||
if r[0] < 1e-7 { pos += HOP; continue }
|
||||
let (a, err) = levinson(r, LPC_ORDER)
|
||||
let gain = max(err, 0).squareRoot()
|
||||
let useF0 = f0 > 0 ? f0 : (lastF0 > 0 ? lastF0 : 0)
|
||||
lastF0 = f0
|
||||
for i in 0..<HOP {
|
||||
let idx = pos + i; if idx >= x.count { break }
|
||||
var e = 0.0
|
||||
if useF0 > 0 { // voiced: glottal impulse train
|
||||
phase += useF0/Double(sr)
|
||||
if phase >= 1.0 { phase -= 1.0; e = sqrt(Double(sr)/useF0) } // energy-normalized impulse
|
||||
} else { // unvoiced: noise
|
||||
e = Double.random(in: -1...1)
|
||||
}
|
||||
var y = gain * e
|
||||
for k in 1...LPC_ORDER { y -= a[k]*state[k-1] }
|
||||
for k in stride(from: LPC_ORDER-1, through: 1, by: -1) { state[k] = state[k-1] }
|
||||
state[0] = y
|
||||
out[idx] = y
|
||||
}
|
||||
pos += HOP
|
||||
}
|
||||
// normalize to peak 0.9
|
||||
let peak = out.map { abs($0) }.max() ?? 1
|
||||
let scale = peak > 1e-9 ? 0.9/peak : 1
|
||||
let samples = out.map { Int16(max(-32767, min(32767, $0*scale*32767))) }
|
||||
writeWav(URL(fileURLWithPath: outPath), samples: samples, sampleRate: sr)
|
||||
let vp = voiceprint(inPath)
|
||||
disclose(String(format: "IMITATE: rebuilt the voice from its signature (F0 %.0fHz, formants %@) -> %@",
|
||||
vp.f0, vp.formants.map{String(format:"%.0f",$0.0)}.joined(separator:"/"), outPath))
|
||||
emit(["ok": true, "op": "imitate", "in": inPath, "out": outPath,
|
||||
"f0_hz": vp.f0, "f0_range": [vp.f0lo, vp.f0hi],
|
||||
"formants_hz": vp.formants.map { $0.0 },
|
||||
"method": "LPC analysis-resynthesis (own-core, no training, no stolen voice)"])
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// CONVERSE (full-duplex) — the interruptible conversational loop.
|
||||
// The utterance is a persistent, ordered meaning-plan of SEGMENTS, each with
|
||||
// a salience. The speaker plays them; the mic listens concurrently. On user
|
||||
// speech: pause INSTANTLY, classify (backchannel vs barge-in), then DECIDE
|
||||
// yield-or-hold from the salience of the current segment + the social read.
|
||||
// Yielded utterances persist their remaining plan so Neuron can RESUME.
|
||||
// ============================================================================
|
||||
struct Segment { let file: String; let salience: Double; let text: String }
|
||||
|
||||
enum Decision { case backchannelContinue, hold, yield }
|
||||
|
||||
// The yield-or-hold DECISION — grounded, contextual. Not a fixed rule.
|
||||
func decide(currentSalience: Double, progress: Double,
|
||||
interrupterAuthority: Double, isBackchannel: Bool) -> Decision {
|
||||
if isBackchannel { return .backchannelContinue } // "mm-hm" => keep going
|
||||
// Holding the floor is justified when what I'm saying matters AND I'm nearly
|
||||
// done (cheap to finish) AND the interrupter isn't high-priority.
|
||||
let holdScore = currentSalience * 0.6 + progress * 0.4
|
||||
if holdScore >= 0.6 && interrupterAuthority < 0.8 { return .hold }
|
||||
return .yield // default: be polite, let them in
|
||||
}
|
||||
|
||||
final class Conversation {
|
||||
let engine = AVAudioEngine()
|
||||
let player = AVAudioPlayerNode()
|
||||
var micLive = false
|
||||
// VAD state (shared with the audio tap thread)
|
||||
let lock = NSLock()
|
||||
var micRMS: Float = 0
|
||||
var speechFrames = 0 // consecutive above-threshold frames
|
||||
var onsetHandled = false
|
||||
|
||||
let resumePath: String
|
||||
init(resumePath: String) { self.resumePath = resumePath }
|
||||
|
||||
// Try to bring the mic up as a live VAD. Returns false if unavailable/denied.
|
||||
func startMic() -> Bool {
|
||||
let status = AVCaptureDevice.authorizationStatus(for: .audio)
|
||||
if Consent.load()["mic"] != true || status != .authorized {
|
||||
disclose("CONVERSE: live mic not available (consent/OS) — using injected barge events for the proof.")
|
||||
return false
|
||||
}
|
||||
let input = engine.inputNode
|
||||
// Acoustic echo cancellation: the OS voice-processing unit subtracts our
|
||||
// own speaker output from the mic so Neuron does NOT hear itself and
|
||||
// barge in on its own voice. This is what makes real-room barge-in work.
|
||||
do { try input.setVoiceProcessingEnabled(true); disclose("CONVERSE: AEC on (echo-cancelled mic — won't self-interrupt).") }
|
||||
catch { disclose("CONVERSE: AEC unavailable (\(error)); raising VAD floor instead.") }
|
||||
let fmt = input.inputFormat(forBus: 0)
|
||||
if fmt.sampleRate == 0 { return false }
|
||||
input.installTap(onBus: 0, bufferSize: 1024, format: fmt) { [weak self] buf, _ in
|
||||
guard let self = self, let ch = buf.floatChannelData?[0] else { return }
|
||||
let n = Int(buf.frameLength)
|
||||
var sum: Float = 0
|
||||
for i in 0..<n { let v = ch[i]; sum += v*v }
|
||||
let rms = n > 0 ? (sum / Float(n)).squareRoot() : 0
|
||||
self.lock.lock(); self.micRMS = rms; self.lock.unlock()
|
||||
}
|
||||
micLive = true
|
||||
disclose("CONVERSE: full-duplex — mic listening WHILE speaking (barge-in armed).")
|
||||
return true
|
||||
}
|
||||
|
||||
func run(_ segs: [Segment], interrupterAuthority: Double,
|
||||
injectBargeAt: Double?, injectKind: String, startIndex: Int, liveMic: Bool) {
|
||||
engine.attach(player)
|
||||
let firstFmt = (try? AVAudioFile(forReading: URL(fileURLWithPath: segs[startIndex].file)))?.processingFormat
|
||||
?? AVAudioFormat(standardFormatWithSampleRate: 16000, channels: 1)!
|
||||
engine.connect(player, to: engine.mainMixerNode, format: firstFmt)
|
||||
if liveMic { _ = startMic() }
|
||||
else { disclose("CONVERSE: deterministic mode (live mic off) — barge events \(injectBargeAt != nil ? "injected" : "none").") }
|
||||
do { try engine.start() } catch { die("audio engine failed to start: \(error)") }
|
||||
player.play()
|
||||
|
||||
let injectDeadline = injectBargeAt.map { Date().addingTimeInterval($0) }
|
||||
var injectedFired = false
|
||||
var idx = startIndex
|
||||
|
||||
segmentLoop: while idx < segs.count {
|
||||
let seg = segs[idx]
|
||||
guard let f = try? AVAudioFile(forReading: URL(fileURLWithPath: seg.file)) else {
|
||||
disclose("CONVERSE: missing segment '\(seg.file)', skipping."); idx += 1; continue
|
||||
}
|
||||
let dur = Double(f.length) / f.processingFormat.sampleRate
|
||||
disclose(String(format: "CONVERSE: speaking segment %d/%d (salience %.2f) — \"%@\"",
|
||||
idx+1, segs.count, seg.salience, seg.text))
|
||||
emit(["op": "converse", "event": "speaking", "segment": idx,
|
||||
"salience": seg.salience, "text": seg.text])
|
||||
|
||||
let done = DispatchSemaphore(value: 0)
|
||||
// .dataPlayedBack: completion fires only after the audio has actually
|
||||
// played OUT the DAC (not merely been consumed) — so the tail is never
|
||||
// clipped and playback always runs the FULL file length.
|
||||
player.scheduleFile(f, at: nil, completionCallbackType: .dataPlayedBack) { _ in done.signal() }
|
||||
player.play()
|
||||
|
||||
// Monitor this segment: poll VAD / injected event until it finishes.
|
||||
let segStart = Date()
|
||||
while done.wait(timeout: .now() + 0.02) == .timedOut {
|
||||
let elapsed = Date().timeIntervalSince(segStart)
|
||||
let progress = min(elapsed / max(dur, 0.001), 1.0)
|
||||
|
||||
// --- detect an onset (live mic OR injected) ---
|
||||
var onset = false
|
||||
if micLive {
|
||||
lock.lock(); let rms = micRMS; lock.unlock()
|
||||
if rms > 0.02 { speechFrames += 1 } else { speechFrames = 0 }
|
||||
if speechFrames >= 3 && !onsetHandled { onset = true } // ~60ms of voice
|
||||
}
|
||||
if let dl = injectDeadline, !injectedFired, Date() >= dl, !onsetHandled { onset = true; injectedFired = true }
|
||||
|
||||
if onset {
|
||||
onsetHandled = true
|
||||
// (1) BARGE-IN: pause INSTANTLY, on the spot.
|
||||
player.pause()
|
||||
let tBarge = Date().timeIntervalSince(segStart)
|
||||
disclose(String(format: "CONVERSE: << user speech at %.2fs into segment %d — PAUSED instantly >>", tBarge, idx+1))
|
||||
emit(["op": "converse", "event": "barge_in", "segment": idx,
|
||||
"at_seconds": tBarge, "progress": progress])
|
||||
|
||||
// (2) classify backchannel vs real barge-in
|
||||
let isBackchannel = classifyBackchannel(injected: injectDeadline != nil,
|
||||
kind: injectKind)
|
||||
let d = decide(currentSalience: seg.salience, progress: progress,
|
||||
interrupterAuthority: interrupterAuthority,
|
||||
isBackchannel: isBackchannel)
|
||||
switch d {
|
||||
case .backchannelContinue:
|
||||
disclose("CONVERSE: read as BACKCHANNEL (\"mm-hm\") — keep going, resume seamlessly.")
|
||||
emit(["op": "converse", "event": "backchannel_continue", "segment": idx])
|
||||
onsetHandled = false; speechFrames = 0
|
||||
player.play() // seamless resume
|
||||
case .hold:
|
||||
disclose("CONVERSE: HOLD the floor — \"hang on, let me finish this thought.\" (high salience, nearly done)")
|
||||
emit(["op": "converse", "event": "hold_floor", "segment": idx,
|
||||
"salience": seg.salience, "progress": progress])
|
||||
onsetHandled = false; speechFrames = 0
|
||||
player.play() // finish the segment, THEN yield
|
||||
// after this segment completes we yield the remainder
|
||||
_ = done.wait(timeout: .now() + dur + 1.0)
|
||||
persistResume(segs: segs, from: idx + 1, reason: "held-then-yield")
|
||||
finish(); return
|
||||
case .yield:
|
||||
disclose("CONVERSE: YIELD — stop, let them in. Remembering where I was (resumable).")
|
||||
player.stop()
|
||||
persistResume(segs: segs, from: idx, reason: "yield")
|
||||
emit(["op": "converse", "event": "yield", "interrupted_segment": idx,
|
||||
"resume_from": idx])
|
||||
finish(); return
|
||||
}
|
||||
}
|
||||
}
|
||||
emit(["op": "converse", "event": "segment_done", "segment": idx])
|
||||
idx += 1
|
||||
}
|
||||
// whole utterance completed uninterrupted
|
||||
clearResume()
|
||||
disclose("CONVERSE: utterance complete (uninterrupted).")
|
||||
emit(["ok": true, "op": "converse", "event": "complete", "segments": segs.count])
|
||||
finish()
|
||||
}
|
||||
|
||||
// A backchannel is brief/low. Injected kind lets us prove both paths headlessly;
|
||||
// the live path would measure post-onset duration & energy.
|
||||
func classifyBackchannel(injected: Bool, kind: String) -> Bool {
|
||||
if injected { return kind == "backchannel" }
|
||||
// live: sample ~250ms after onset; if speech already died away, it was a backchannel
|
||||
Thread.sleep(forTimeInterval: 0.25)
|
||||
lock.lock(); let rms = micRMS; lock.unlock()
|
||||
return rms < 0.015
|
||||
}
|
||||
|
||||
func persistResume(segs: [Segment], from: Int, reason: String) {
|
||||
let remaining = segs[from...].map { ["file": $0.file, "salience": $0.salience, "text": $0.text] as [String: Any] }
|
||||
let state: [String: Any] = ["resume_from": from, "reason": reason,
|
||||
"remaining": remaining, "ts": Date().timeIntervalSince1970]
|
||||
if let d = try? JSONSerialization.data(withJSONObject: state, options: [.prettyPrinted]) {
|
||||
try? d.write(to: URL(fileURLWithPath: resumePath))
|
||||
}
|
||||
disclose("CONVERSE: meaning-plan persisted (\(remaining.count) segments remain) — Neuron can resume the thread.")
|
||||
}
|
||||
func clearResume() { try? FileManager.default.removeItem(atPath: resumePath) }
|
||||
func finish() { player.stop(); if micLive { engine.inputNode.removeTap(onBus: 0) }; engine.stop() }
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// CLI
|
||||
// ----------------------------------------------------------------------------
|
||||
func loadManifest(_ path: String) -> (segs: [Segment], utterance: String) {
|
||||
guard let d = FileManager.default.contents(atPath: path),
|
||||
let o = try? JSONSerialization.jsonObject(with: d) as? [String: Any],
|
||||
let arr = o["segments"] as? [[String: Any]] else { die("bad manifest: \(path)") }
|
||||
let segs = arr.map { Segment(file: $0["file"] as? String ?? "",
|
||||
salience: ($0["salience"] as? NSNumber)?.doubleValue ?? 0.5,
|
||||
text: $0["text"] as? String ?? "") }
|
||||
return (segs, o["utterance"] as? String ?? "")
|
||||
}
|
||||
|
||||
let args = CommandLine.arguments
|
||||
guard args.count >= 2 else {
|
||||
print("""
|
||||
periph — Neuron peripheral I/O (own-core, local, consent-gated)
|
||||
grant <camera|mic> grant a sensitive sense (Neuron-level consent)
|
||||
revoke <camera|mic> revoke it
|
||||
status show consent state
|
||||
speak <file.wav> SPEAK ALOUD (efferent) via the speaker
|
||||
tone <out.wav> [hz] [sec] own-core synth a test WAV (no deps)
|
||||
listen <sec> <out.wav> MIC capture (afferent) 16k mono
|
||||
see <out.jpg> CAMERA one frame (afferent)
|
||||
feat-audio <file.wav> extract compact voice/sound signature (for ingest)
|
||||
feat-image <file.jpg> extract compact scene-geometry (for ingest)
|
||||
ingest-audio <file.wav> <engramURL> capture -> descriptor -> engram node (geometry)
|
||||
ingest-image <file.jpg> <engramURL> capture -> descriptor -> engram node (geometry)
|
||||
voiceprint <voice.wav> extract voice-signature (F0 + formants F1-F5)
|
||||
imitate <voice.wav> <out.wav> speak back in that voice (LPC analysis-resynthesis)
|
||||
hear-imitate <sec> <out.wav> MIC -> extract signature -> imitate -> SPEAK ALOUD
|
||||
wav-info <file.wav> print WAV geometry
|
||||
converse <manifest.json> [--authority F] [--barge-at S[:backchannel|:bargein]] [--resume]
|
||||
full-duplex interruptible utterance
|
||||
""")
|
||||
exit(0)
|
||||
}
|
||||
|
||||
switch args[1] {
|
||||
case "grant":
|
||||
guard args.count >= 3 else { die("grant needs a device") }
|
||||
var g = Consent.load(); g[args[2]] = true; Consent.save(g)
|
||||
disclose("granted '\(args[2])' — the user consents; raw stream stays local, never egresses.")
|
||||
emit(["ok": true, "op": "grant", "device": args[2], "consent": g])
|
||||
case "revoke":
|
||||
guard args.count >= 3 else { die("revoke needs a device") }
|
||||
var g = Consent.load(); g[args[2]] = false; Consent.save(g)
|
||||
emit(["ok": true, "op": "revoke", "device": args[2], "consent": g])
|
||||
case "status":
|
||||
emit(["ok": true, "op": "status", "consent": Consent.load()])
|
||||
case "speak":
|
||||
guard args.count >= 3 else { die("speak needs a wav") }
|
||||
speak(args[2])
|
||||
case "tone":
|
||||
guard args.count >= 3 else { die("tone needs an out path") }
|
||||
let hz = args.count >= 4 ? Double(args[3]) ?? 220 : 220
|
||||
let sec = args.count >= 5 ? Double(args[4]) ?? 1.0 : 1.0
|
||||
let sr = 16000
|
||||
var s = [Int16](); s.reserveCapacity(Int(Double(sr)*sec))
|
||||
for i in 0..<Int(Double(sr)*sec) {
|
||||
let t = Double(i)/Double(sr)
|
||||
let env = min(1.0, min(t*20, (sec - t)*20)) // gentle attack/release
|
||||
s.append(Int16(env * 0.3 * 32767 * sin(2*Double.pi*hz*t)))
|
||||
}
|
||||
writeWav(URL(fileURLWithPath: args[2]), samples: s, sampleRate: sr)
|
||||
disclose("tone: wrote own-core \(sec)s @ \(hz)Hz WAV to \(args[2]).")
|
||||
emit(["ok": true, "op": "tone", "file": args[2], "hz": hz, "seconds": sec])
|
||||
case "listen":
|
||||
guard args.count >= 4 else { die("listen needs <sec> <out.wav>") }
|
||||
listen(seconds: Double(args[2]) ?? 3.0, out: args[3])
|
||||
case "see":
|
||||
guard args.count >= 3 else { die("see needs an out path") }
|
||||
see(out: args[2])
|
||||
case "feat-audio":
|
||||
guard args.count >= 3 else { die("feat-audio needs a wav") }
|
||||
featAudio(args[2])
|
||||
case "feat-image":
|
||||
guard args.count >= 3 else { die("feat-image needs an image") }
|
||||
featImage(args[2])
|
||||
case "ingest-audio":
|
||||
guard args.count >= 4 else { die("ingest-audio needs <wav> <engramURL>") }
|
||||
ingest(args[2], kind: "audio", engramURL: args[3])
|
||||
case "ingest-image":
|
||||
guard args.count >= 4 else { die("ingest-image needs <image> <engramURL>") }
|
||||
ingest(args[2], kind: "image", engramURL: args[3])
|
||||
case "voiceprint":
|
||||
guard args.count >= 3 else { die("voiceprint needs a wav") }
|
||||
let vp = voiceprint(args[2])
|
||||
disclose("VOICEPRINT: \(vp.content)")
|
||||
emit(["ok": true, "op": "voiceprint", "file": args[2], "f0_hz": vp.f0,
|
||||
"f0_range": [vp.f0lo, vp.f0hi], "formants_hz": vp.formants.map { $0.0 },
|
||||
"bandwidths_hz": vp.formants.map { $0.1 }, "content": vp.content,
|
||||
"ingest": ["node_type": "Observation", "tier": "Episodic", "content": vp.content]])
|
||||
case "imitate":
|
||||
guard args.count >= 4 else { die("imitate needs <voice.wav> <out.wav>") }
|
||||
imitate(inPath: args[2], outPath: args[3])
|
||||
case "hear-imitate":
|
||||
guard args.count >= 4 else { die("hear-imitate needs <sec> <out.wav>") }
|
||||
let secs = Double(args[2]) ?? 4.0
|
||||
let outp = args[3]
|
||||
let capp = outp.replacingOccurrences(of: ".wav", with: "") + ".heard.wav"
|
||||
disclose("HEAR-IMITATE: open the ear, listen \(secs)s, grab the voice, speak it back.")
|
||||
listen(seconds: secs, out: capp) // afferent: hear the voice
|
||||
imitate(inPath: capp, outPath: outp) // extract signature + resynthesize
|
||||
speak(outp) // efferent: speak back ALOUD in that voice
|
||||
case "wav-info":
|
||||
guard args.count >= 3, let i = wavInfo(args[2]) else { die("wav-info needs a readable wav") }
|
||||
disclose("WAV \(args[2]): \(i.sampleRate)Hz \(i.channels)ch \(i.bits)bit \(i.frames) frames")
|
||||
emit(["ok": true, "op": "wav-info", "sample_rate": i.sampleRate, "channels": i.channels,
|
||||
"bits": i.bits, "frames": i.frames,
|
||||
"seconds": Double(i.frames)/Double(max(i.sampleRate,1))])
|
||||
case "converse":
|
||||
guard args.count >= 3 else { die("converse needs a manifest") }
|
||||
let (segs, utter) = loadManifest(args[2])
|
||||
var authority = 0.5
|
||||
var bargeAt: Double? = nil
|
||||
var bargeKind = "bargein"
|
||||
var resume = false
|
||||
var liveMic = false
|
||||
var i = 3
|
||||
while i < args.count {
|
||||
switch args[i] {
|
||||
case "--authority": if i+1 < args.count { authority = Double(args[i+1]) ?? 0.5; i += 1 }
|
||||
case "--barge-at":
|
||||
if i+1 < args.count {
|
||||
let parts = args[i+1].split(separator: ":")
|
||||
bargeAt = Double(parts[0]) ?? nil
|
||||
if parts.count > 1 { bargeKind = String(parts[1]) }
|
||||
i += 1
|
||||
}
|
||||
case "--resume": resume = true
|
||||
case "--live-mic": liveMic = true
|
||||
default: break
|
||||
}
|
||||
i += 1
|
||||
}
|
||||
let resumePath = (ProcessInfo.processInfo.environment["PERIPH_HOME"]
|
||||
?? FileManager.default.currentDirectoryPath + "/peripheral") + "/.resume.json"
|
||||
var startIndex = 0
|
||||
var runSegs = segs
|
||||
if resume, let d = FileManager.default.contents(atPath: resumePath),
|
||||
let o = try? JSONSerialization.jsonObject(with: d) as? [String: Any],
|
||||
let rem = o["remaining"] as? [[String: Any]] {
|
||||
runSegs = rem.map { Segment(file: $0["file"] as? String ?? "",
|
||||
salience: ($0["salience"] as? NSNumber)?.doubleValue ?? 0.5,
|
||||
text: $0["text"] as? String ?? "") }
|
||||
startIndex = 0
|
||||
disclose("CONVERSE: resuming — \"as I was saying...\" (\(runSegs.count) segments left).")
|
||||
emit(["op": "converse", "event": "resume", "remaining": runSegs.count])
|
||||
}
|
||||
if runSegs.isEmpty { die("no segments to speak") }
|
||||
disclose("CONVERSE: utterance = \"\(utter)\" (\(runSegs.count) segments).")
|
||||
let convo = Conversation(resumePath: resumePath)
|
||||
convo.run(runSegs, interrupterAuthority: authority,
|
||||
injectBargeAt: bargeAt, injectKind: bargeKind, startIndex: startIndex, liveMic: liveMic)
|
||||
default:
|
||||
die("unknown command: \(args[1])")
|
||||
}
|
||||
Reference in New Issue
Block a user