37. Audio

Status

Audio admission is Partial. The current v1 compatibility spine can carry immutable audio ArtifactRef metadata, express audio input through modalities_in, and declare stt and tts capability families. Media-type projection to input modality remains Designed. Those labels prove schema only, not a working transcription or synthesis call.

A pure Echo inspector also checks supplied bytes against an exact artifact reference and the bounded PCM subset below. The separate explicit local Reliquary adapter can retain supplied WAV bytes in quarantine. A trusted Echo consumer reruns the inspector, requires separate admission, and reads under current authority. These unregistered helpers supply no capture or speech path.

LychD cannot capture, upload, transcode, transport, or play audio bytes. Bridge does not forward audio modality into dispatch. No credential-backed audio admission, speech timeline, streaming transport, working STT/TTS adapter, selectable Echo extension, or Audio Coven ships. State of Work owns this boundary and the local custody limits.

Decision

Echo is the speech-lifecycle Domain. Its first profile is record and send: one explicit, bounded utterance becomes an immutable artifact, then an admitted provider returns text or another audio artifact.

There is no audio capability family. Audio is material; current stt, tts, and chat are v1 compatibility families, while the Designed general-service path uses exact interfaces, profiles, operations, and call/session grants.

Engine, model, protocol, and language

Speech integration keeps four axes separate:

A shared file format or OpenAI-compatible route does not make an unknown model architecture loadable. Engine support means that the exact family has a loader, preprocessing, decoding, and tested execution path. Likewise, two servers exposing /v1/audio/transcriptions or /v1/audio/speech may accept different fields and provide different streaming or cancellation semantics; each admitted endpoint declares and proves its dialect.

The first planned local engine profile is audio.cpp behind the bounded OpenAI audio batch subset. It is one initial runtime choice, not a new universal audio abstraction and not evidence that every audio.cpp family works. Its Rune must pin the runtime revision, model family, checkpoint and digest, backend and device, declared and verified languages, license, and the exact stt or tts behavior admitted by a bake. The first record-and-send slice admits no second local speech runtime and no automatic runtime fallback. Echo owns the current candidate models and promotion criteria; delivery remains Not yet.

Language is neither a host-global switch nor a property inferred from the engine. An explicit request language wins, then the admitted Principal or Persona preference, then model-supported detection when policy allows it. A model may route a language only after declaring it; a verified language additionally passed the relevant LychD bake. STT detection never silently authorizes translation, and TTS may not substitute another language or voice when the selected output locale is unavailable. Translation is a separate semantic act.

Altar interface locale is independent of this speech policy: a Slovak interface may capture an English utterance and request a Japanese Voice, provided each exact step is admitted. Musical lyrics, singing, arrangement, and musical acceptance belong to Riffmaw rather than Echo; they reuse the same no-global-language and no-silent-translation law without turning a song generator into TTS.

Language Edition owns an attributed timed-media language edition: aligned source and target words, casting, performance direction, dialogue timing, caption fit, and acceptance of a constrained language-track replacement. It may place STT, Translation, and TTS or human-performance steps in one Pattern, but that orchestration does not make Echo the application or Translation an ambient feature. Language Edition may alter only declared language tracks against a locked master; Broadcast retains editorial recutting, final assembly, and publication.

Capture and custody

Capture authority is explicit, visible, time-bounded, and revocable. Consent to one utterance is not permanent microphone access; device indicators and server state must agree whether capture is armed, active, stopped, or failed.

Voice cloning is not implied by TTS support. A cloning request separately binds the authorized speaker or source, reference-audio custody, permitted words and use, model and license, and the derived voice artifact. A convincing result grants no Persona, Principal, or attribution authority.

Reliquary custody binds recording or synthesis identity, digest, media type, byte size, validated duration/codec facts, classification, Principal, retention, and derivation from recording, transcript, or request. Bytes never enter a Graph checkpoint, queue payload, event envelope, or log—only references and bounded timeline state do. Decoders and transcoders treat them as hostile: format, duration, channels, sample rate, decompression, parser resources, and metadata are bounded before provider use.

Bounded PCM inspection

Echo's first local material-inspection contract accepts supplied immutable bytes and an existing ArtifactRef. It checks the exact byte length and SHA-256 digest against that reference; it neither retrieves the artifact nor establishes custody or permission to use it.

This first subset is deliberately exact: audio/wav, a 44-byte RIFF/WAVE header with a 16-byte PCM fmt chunk immediately followed by one data chunk, signed 16-bit little-endian samples, one channel, and 16,000 frames per second. The RIFF length, data length, byte rate, block alignment, and sample width must agree with the actual bytes. Accept 1 through 480,000 complete frames (up to 30 seconds) and no more than 1 MiB in the supplied artifact. Refuse ancillary chunks, extended headers, compression, trailing bytes, truncation, empty data, and other WAV variants; there is no implicit conversion or resampling.

A successful inspection returns immutable source and PCM facts. It establishes only that these bytes match this reference and this structural subset. Silence is valid PCM; inspection proves neither speech, origin, consent, semantic eligibility, model compatibility, nor an executed transcription. Construction of a facts record alone does not prove inspection. This boundary starts after a caller has obtained bytes; it does not bound an upstream upload or retrieval.

Record, send, account

explicit capture → immutable source artifact → eligible Ear or audio-capable Mind
→ attributed transcript or native observation → ordinary Agent step → optional Voice
→ delivery receipt

A transcript retains source, available time regions, provider and revision, language assumptions, and uncertainty; it interprets a recording rather than replacing it. Synthesized audio is a new artifact. “Generated” says bytes were produced, not that a client received, buffered, played, or completed them.

When speech is required, the admitted Pattern's finish policy declares the required result, the evidence needed to settle it, and any explicitly configured accepted fallback. Failure or missing evidence at that required boundary produces partial or non-completion under the pinned score unless an explicitly admitted fallback is fulfilled. Preserved text does not fulfill a required speech obligation; text-only completion is permitted only when that fallback was explicitly admitted. An observed playback completion establishes the reported playback boundary, not that the person heard or understood the response.

The Pattern supplies exact interface/operation, material facts, classification, priority, and authority. The Dispatcher chooses an exact Ear, Voice, or Mind profile; current v1 can express only the stt, tts, or chat compatibility projection. For an otherwise eligible managed binding that is not WARM, Dispatcher returns HardwareTransitionRequired; the requesting Run enters Graph Stasis while Orchestrator converges readiness, then re-dispatches.

In current source, even WARM v1 stt and tts declarations fail closed at grant issue because neither has a typed call surface. Readiness convergence cannot turn those labels into Ear or Voice.

Echo may not preempt, revoke another grant, make its own continuing session, or silently select a remote provider. A Portal needs egress eligibility, consent where required, and a cost bound. Local execution still requires capture, retention, and tool authority; source influence persists under Context privatization.

Streaming without pretending a socket is a protocol

Record-and-send is the minimum. Half-duplex may stream one response after capture closes. Full-duplex additionally owes simultaneous capture/playback, VAD, barge-in, echo cancellation, and contested turn ownership.

Voice-activity detection answers whether speech is present; it does not prove that an activation phrase was spoken. The first profile therefore uses visible push-to-talk. A future wake-word backend belongs to the Listener and emits a bounded activation event; it is neither an stt Animator nor ambient permission to retain or transcribe surrounding audio.

A Resonance Session keeps one monotonic timeline for segments and attempts: capture armed/active/stopped/failed; transcription and synthesis in progress/settled/generated; delivery offered/accepted/failed; playback started/completed/interrupted/unknown; and cancellation, expiry, or retry. WebSocket, WebRTC, or another transport can carry these records but supplies none of their semantics. Reconnect resumes by acknowledged sequence and artifact identity, never by guessing from a live socket.

Cancellation names its scope: stopping playback need not erase an artifact, revoking capture stops new input, and cancelling an Invocation settles every admitted provider and transport attempt. Closing local result adoption or a transport connection does not prove that provider execution stopped. After submission, retain execution ownership until completion or an owned stop and reconciliation proves settlement; uncertain execution cannot authorize a retry or release a compute reservation. Late output may settle execution while remaining ineligible for adoption. A future Companion client can own physical capture, playback, route selection, and low-level device state; a private Tether still supplies transport, not application authority, classification, or consent.

Application-owned live audio

Echo's Resonance Session owns speech chronology; it does not absorb musical performance or every application that carries sound. Riffmaw may own a musical performance session and clock overlay; Language Edition may own a language-edition timeline over exact speech attempts. Each remains an application record referencing Echo chronology rather than a duplicate capture, synthesis, delivery, or playback ledger, and every live route still obeys Audio custody, transport, interruption, and uncertain-playback law.

Avatar may use an authorized exact voice profile or derived artifact inside one profile or Morphe. Avatar can narrow purpose, audience, words, disclosure, retention, and revocation, but cannot create speaker authority, copy a Persona, widen the cloning grant, or claim that synthesized speech was played. Echo retains those lifecycle facts. Language Edition may cast the same eligible reference for one exact language edition, but owns only role fit, pronunciation, timing, disclosure, and performance acceptance; it cannot widen either Avatar's presentation envelope or the underlying performer and voice-use authority.

Monitoring, recording, retention, analysis, transformation, remote transmission, and machine response are independently admitted scopes. A visible armed input proves none of the others. A live route binds an exact route and clock epoch, monotonic and sample-frame mapping, sequence, scheduled window, absolute deadline, cancellation fence, and output acknowledgement. Late or stale frames are rejected locally after expiry, disconnect, cancellation, or epoch change; a restarted device, peer, model, or plug-in requires reconciliation and a newly admitted live Invocation.

The realtime callback never waits on a model, disk, network, control-plane action, or unsafe plug-in operation. Declared direct or otherwise safe monitoring, output limiting, feedback protection, safe bypass, and immediate local stop remain available without model cooperation. Model and planning work occurs outside that callback and may emit only through the deadline fence. PCM, MIDI streams, device handles, and plug-in state never enter Graph checkpoints or events; only artifact references, bounded sequence state, epochs, and receipts cross the durable seam.

A tempo or phase protocol does not establish audio transport or a shared sample clock. A remote audio link does not establish recording consent. Each application records mappings, clock-rate divergence, resampling, loss, jitter, latency, discontinuity, and uncertainty rather than naming two devices “synchronized” by assertion.

Consequences and acceptance

Echo distinguishes generation, delivery, and playback and lets independent Ears, Voices, Minds, and Listeners compose. The cost is sensitive custody, storage/retention pressure, and timing, cancellation, and egress races.

Bounded material inspection can be accepted through actual byte-parsing and refusal tests without claiming a speech runtime. Usable record-and-send execution additionally requires evidence for visible capture consent, custody, hostile-audio limits at acquisition and decoding, modality forwarding, local and Portal policy, STT/TTS conversion, transcript provenance, delivery/playback receipts, interruption, retention/deletion, and disconnect recovery.