# Voice Assistant
**Realtime speech processing over WebSocket — NVIDIA ASR · Agent orchestration · VoxCPM TTS**
  
A modular voice pipeline for streaming audio input, speech transcription, agent responses, and synthesized speech output. Built with FastAPI and asynchronous Python, with explicit session lifecycle, turn ownership, interruption, and provider cleanup.
> **V1 scope:** NVIDIA STT and VoxCPM TTS are real providers; the Agent is a mock implementation. Production readiness and container deployment have not yet been validated.
## Contents
- [Architecture](#architecture)
- [Quickstart](#quickstart)
- [Configuration](#configuration)
- [WebSocket protocol](#websocket-protocol)
- [Lifecycle and interruption](#lifecycle-and-interruption)
- [Testing](#testing)
- [Project structure](#project-structure)
- [Known limitations](#known-limitations)
- [Architecture Backbone & Abstraction Boundaries](#architecture-backbone--abstraction-boundaries)
## Architecture
```mermaid
flowchart TB
C[Audio Client] <-->|WebSocket /ws/audio · JSON + binary PCM| G[Audio Gateway · FastAPI]
G --> U[VoiceSession · conversation lifecycle]
U --> P[VoicePipeline]
P --- S[SessionState · session and audio sequence]
P --- T[TurnManager · semantic turn state]
U --> STT[NVIDIA ASR · STT per turn]
U --> VAD[Silero VAD · local CPU · per continuous session]
STT -->|Transcript events| P
P --> A[Mock Agent · per session]
A -->|Response text| P
P --> V[VoxCPM TTS · per session]
V -->|PCM chunks| G
G -->|Streamed audio| C
```
The **Audio Gateway** handles transport, protocol validation, and application composition. **VoiceSession** coordinates STT, finalization, interruption, rollback, and provider cleanup. **VoicePipeline** retains processing, turn-state operations, and execution-task ownership. **SessionState** tracks session and audio sequence; **TurnManager** enforces turn transitions. A new STT adapter is created for each turn, while Agent and TTS adapters are created per session.
## Architecture Backbone & Abstraction Boundaries
AgentVoice follows a **provider-independent architecture** built around explicit contracts, lifecycle ownership, and asynchronous streaming.
The architecture separates five concerns:
- **Transport:** WebSocket communication and protocol serialization.
- **Orchestration:** Coordination of STT, Agent, and TTS execution.
- **Lifecycle:** Session state, turn transitions, interruption, and task ownership.
- **Provider contracts:** Abstract interfaces defining component behavior.
- **Provider implementations:** Concrete integrations with NVIDIA, OCI Generative AI, and VoxCPM.
### Architectural Backbone
```mermaid
flowchart TB
Client["Web Client / Audio Client"]
Gateway["Audio Gateway
FastAPI · WebSocket"]
UseCase["VoiceSession
Conversation Lifecycle"]
Pipeline["VoicePipeline
Orchestration & Streaming"]
Session["SessionState
Session Lifecycle"]
Turns["TurnManager
Semantic Turn Lifecycle"]
subgraph Contracts["Provider Abstraction Layer"]
direction LR
STTA["STTAdapter
Abstract Contract"]
AgentA["AgentAdapter
Abstract Contract"]
TTSA["TTSAdapter
Abstract Contract"]
end
subgraph Implementations["Provider Implementation Layer"]
direction LR
Nvidia["NvidiaSTTAdapter"]
OCI["OCIAgentAdapter"]
Mock["MockAgentAdapter"]
Vox["VoxCPMTTSAdapter"]
end
subgraph External["External AI Services"]
ASR["NVIDIA ASR"]
LLM["OCI Generative AI"]
TTS["VoxCPM"]
end
Client <-->|"JSON Events + Binary PCM"| Gateway
Gateway --> UseCase
UseCase --> Pipeline
Pipeline --- Session
Pipeline --- Turns
Pipeline --> STTA
Pipeline --> AgentA
Pipeline --> TTSA
STTA -.->|implements| Nvidia
AgentA -.->|implements| OCI
AgentA -.->|implements| Mock
TTSA -.->|implements| Vox
Nvidia --> ASR
OCI --> LLM
Vox --> TTS
```
The `VoicePipeline` depends on provider contracts rather than concrete implementations. Provider selection is performed through configuration and dynamic factories.
This allows implementations to evolve without requiring the Audio Gateway to understand provider-specific APIs.
### Provider Contracts
| Abstraction | Contract | Implementations | Responsibility |
|---|---|---|---|
| Speech-to-text | `STTAdapter` | `NvidiaSTTAdapter`, `MockSTTAdapter` | Audio ingestion and transcript events |
| Agent | `AgentAdapter` | `OCIAgentAdapter`, `MockAgentAdapter` | Text processing and conversational responses |
| Text-to-speech | `TTSAdapter` | `VoxCPMTTSAdapter`, `MockTTSAdapter` | Speech synthesis and audio streaming |
The `AgentAdapter` defines complete-response processing and session cleanup. `StreamingAgentAdapter` explicitly adds incremental text generation; OCI and DeepAgents implement this contract. Streaming returns an asynchronous generator that the consumer closes on early exit. `TTSAdapter` defines the same ownership rule for synthesized audio. Both contracts expose `aclose()`; providers retaining session resources override its default no-op implementation.
### Runtime Composition
Provider implementations are selected through environment configuration and resolved by the provider factory.
```mermaid
flowchart LR
Config["Environment YAML"]
Catalog["Provider Catalog"]
Factory["Provider Factories"]
Contracts["Adapter Contracts"]
Runtime["VoicePipeline"]
Config --> Factory
Catalog --> Factory
Factory -->|Validate implementation| Contracts
Factory -->|Instantiate providers| Runtime
```
The primary configuration components are:
| Component | Location |
|---|---|
| Environment selection | `config/loader.py` |
| Typed configuration | `config/models.py` |
| Provider catalog | `config/providers.yaml` |
| Dynamic provider loading | `providers.py` |
| Session lifecycle | `domain/session.py` |
| Semantic turn lifecycle | `turn_manager.py` |
| Conversation lifecycle | `application/voice_session.py` |
| Application output events | `application/events.py` |
| Processing and execution tasks | `voice_pipeline.py` |
### Streaming Execution Backbone
The streaming pipeline allows OCI Generative AI to continue producing text while VoxCPM synthesizes previously completed text segments.
```mermaid
flowchart TB
Input["Microphone / PCM Input"]
Gateway["Audio Gateway"]
STT["NVIDIA STT"]
Transcript["Final Transcript"]
subgraph Pipeline["VoicePipeline · Concurrent Execution"]
direction TB
Agent["OCIAgentAdapter.stream()"]
Bridge["OCI SSE Async Bridge"]
Chunker["TextChunker"]
Queue["Bounded Segment Queue"]
Synth["VoxCPM TTS Streaming"]
Agent --> Bridge
Bridge --> Chunker
Chunker --> Queue
Queue --> Synth
end
OCI["OCI Generative AI"]
Vox["VoxCPM Service"]
Output["WebSocket PCM Output"]
Input --> Gateway
Gateway --> UseCase["VoiceSession"]
UseCase --> STT
STT --> Transcript
Transcript --> Agent
Agent <--> OCI
Synth <--> Vox
Synth --> Gateway
Gateway --> Output
```
The pipeline uses bounded queues and asynchronous task coordination to support incremental generation, ordered synthesis, and cooperative cancellation.
### Lifecycle Ownership
AgentVoice separates semantic state from asynchronous execution.
```mermaid
flowchart LR
Listening["LISTENING"]
Processing["PROCESSING"]
Speaking["SPEAKING"]
Completed["COMPLETED"]
Interrupted["INTERRUPTED"]
Listening -->|"STT final"| Processing
Processing -->|"First TTS segment"| Speaking
Speaking -->|"Synthesis complete"| Completed
Listening --> Interrupted
Processing --> Interrupted
Speaking --> Interrupted
```
**Ownership responsibilities:**
| Component | Owns |
|---|---|
| Audio Gateway | WebSocket connection, protocol validation, session composition, event delivery, and shielded application cleanup |
| SessionState | Session lifecycle and audio sequence validation |
| TurnManager | Semantic turn state transitions |
| VoiceSession | Input history, VAD boundaries, STT lifecycle, finalization, interruption, and provider cleanup |
| VoicePipeline | Turn execution task and Agent/TTS orchestration |
| OCI streaming bridge | SSE reader, event queue, and stream closure |
| Provider adapters | Provider-specific requests, connections, and data formats |
Cancellation is cooperative. The system requests cancellation and waits for task termination, but termination of blocking external operations depends on the underlying provider and transport behavior.
### Architectural Extension Points
The abstraction layer allows additional implementations to be introduced through the provider catalog.
For example, a future LiteLLM integration could implement the Agent contract without requiring a redesign of the Gateway or the semantic turn lifecycle.
```mermaid
flowchart TB
Pipeline["VoicePipeline"]
Contract["AgentAdapter"]
OCI["OCIAgentAdapter
Current"]
Mock["MockAgentAdapter
Testing"]
Lite["LiteLLMAgentAdapter
Future"]
Pipeline --> Contract
Contract -.-> OCI
Contract -.-> Mock
Contract -.-> Lite
```
**Design principle:** transport, orchestration, lifecycle, and provider-specific integrations remain separate concerns. Provider independence is established through contracts and factories, while streaming and cancellation behavior are validated at their respective integration boundaries.
### DeepAgents: autenticação e streaming
No Mac, `API_KEY` lê o perfil OCI e a chave indicados pelo arquivo de configuração.
Para uma sessão OCI CLI, use `SECURITY_TOKEN` e um perfil com token válido.
O DeepAgents também aceita `INSTANCE_PRINCIPAL` e `RESOURCE_PRINCIPAL` em
ambientes OCI compatíveis. O padrão do construtor permanece `INSTANCE_PRINCIPAL`
quando `auth_type` é omitido. Credenciais e permissões precisam ser válidas.
Dentro do container, monte o arquivo OCI e as chaves/tokens necessários somente
para leitura, com caminhos acessíveis ao usuário do processo, e ajuste
`auth_file_location`; `~` refere-se ao usuário do container, não ao do Mac.
Não há integração MCP/RAG automática nesse bloco; a implementação usa `tools=[]`.
`agent.options.timeout_seconds` limita a espera por cada próximo evento do
DeepAgents (inclusive eventos internos), sem incluir o tempo em que o consumidor
está processando um fragmento já entregue. Não é prazo total da resposta nem
limite de inicialização/autenticação. A expiração fecha o stream e propaga erro.
## Quickstart
### Requirements
- Python 3.12 or newer and [uv](https://docs.astral.sh/uv/)
- Accessible NVIDIA ASR and VoxCPM services
- Run commands from the repository root (`config/` is resolved relative to the working directory)
### Install
```bash
uv sync
```
### Start the gateway
With the configured provider services running:
```bash
APP_ENV=development uv run uvicorn \
voice_assistant.audio_gateway.app.main:app \
--host 127.0.0.1 --port 8001
```
Open `http://127.0.0.1:8001/ui`, allow microphone access, and click **Iniciar conversa**.
The client sends continuous audio; backend Silero detects turns and new speech
interrupts an active response. Audio capture continues during playback.
Transcripts currently appear in the browser developer console as `[STT PARTIAL]`
and `[STT FINAL]`; Agent text appears as `[AGENT RESPONSE]` or `[AGENT DELTA]`.
On Firefox for macOS, open the console with **Command + Option + K**.
The page displays conversation status and plays synthesized audio.
The Gateway and VAD run locally without LiveKit or a LiveKit Cloud account.
STT and TTS are separate services and may run remotely; the loopback addresses
in development must point to local services or active port forwards. Audio is
sent to those services, so the full processing path is not offline. Model and
Python dependencies must be installed before offline VAD execution.
If port 8001 is occupied, choose another port (for example, 8002) and use that
same port in the browser, health checks, and WebSocket tools.
Check availability:
```bash
curl http://127.0.0.1:8001/ready
# {"status":"ready"}
```
> `/ready` reports Gateway availability; it does **not** verify NVIDIA ASR or VoxCPM health.
## Configuration
`APP_ENV` selects `config/.yaml`; the default is `development`. `config/providers.yaml` maps provider names to Python implementations and capability metadata. YAML data is validated using Pydantic models.
| Component | Development provider | Configuration |
| --- | --- | --- |
| Speech-to-text | NVIDIA ASR | `127.0.0.1:9000`, `pt-BR`, punctuation enabled |
| Agent | Mock Agent | In-process; simulated text response |
| Text-to-speech | VoxCPM | `http://127.0.0.1:8000`, `openbmb/VoxCPM2`, `default` voice |
| Voice activity | Silero ONNX | Local CPU; threshold 0.6 and 800 ms silence |
To select another environment:
```bash
APP_ENV=staging uv run uvicorn \
voice_assistant.audio_gateway.app.main:app \
--host 127.0.0.1 --port 8001
```
The `staging` and `production` files are currently empty placeholders. Populate
the selected file before using it; the loader rejects an empty YAML. Start from
`development.yaml`, then configure reachable provider endpoints and VAD/input
settings for that environment. Deployment readiness has not been established.
TTS voice selection is `tts.options.voice` in the selected environment YAML.
The service determines which voice identifiers are supported. Configuration is
loaded at Gateway startup: restart the process and open a new conversation after
changing YAML; refreshing the browser alone does not reload server configuration.
## WebSocket protocol
| Property | Value |
|---|---|
| Endpoint | `ws://127.0.0.1:8001/ws/audio` |
| Protocol version | `1.0` |
| Input audio | PCM signed 16-bit little-endian, 16 kHz, mono |
| Output audio | Declared by `session.ready.output_audio`; current VoxCPM: PCM S16LE, 48 kHz, mono |
| Transport: | JSON control messages and binary PCM frames. |
### Typical exchange
```mermaid
sequenceDiagram
participant C as Client
participant G as Audio Gateway
participant P as VoicePipeline / Providers
C->>G: session.start (turn_detection=server)
G-->>C: session.ready
loop Continuous audio input, including silence and playback
C->>G: audio.input (sequence_number)
C->>G: Binary PCM frame
G-->>C: turn.started (when backend VAD detects speech)
P-->>G: STT partial events
G-->>C: stt.partial
end
Note over G,P: Backend VAD ends input after silence
P-->>G: STT final transcript
G-->>C: stt.final
P-->>G: Agent response
G-->>C: agent.response
P-->>G: TTS PCM chunks
G-->>C: Binary audio frames
G-->>C: turn.ended
```
STT events may arrive while input audio is still streaming. In server detection mode, each binary PCM frame is preceded by `audio.input` with a session-wide `sequence_number`, starting at zero and continuing across turns. `session.ready.max_audio_chunk_bytes` declares the accepted limit, capped at 64 KiB. The browser sizes worklet chunks to respect it. Manual mode uses `audio.chunk` with `turn_id` and a per-turn sequence. Input frames must contain complete PCM samples; rejected frames do not advance the audio sequence. `session.ready.audio` describes input and `session.ready.output_audio` describes output. The browser configures playback from the declared output format. VoxCPM reassembles HTTP fragments into complete PCM frames and rejects a truncated final sample. Its `sample_rate` option declares the expected native output rate; it does not request resampling from the external service.
### Interruption and session closure
- In server detection mode, `SpeechStarted` makes `VoiceSession` interrupt an active turn before opening a new one. The Gateway maps application events to `interrupted` and `turn.started`; the client stops obsolete playback.
- Explicit `interrupt` targets a `turn_id`. For a matching active turn, `VoiceSession` cancels and awaits execution, aborts STT, and releases turn state. The Gateway then delivers the acknowledgement.
- `session.end` requests cleanup and session closure; successful completion is acknowledged with `session.ended`.
- Errors are returned as structured `error` events (for example, `STT_START_FAILED` or `TURN_CANCEL_FAILED`).
## Lifecycle and interruption
The system separates **semantic state** from **asynchronous execution**. `TurnManager` controls listening, processing, speaking, and completion; `VoicePipeline` owns the registered turn task. VoiceSession coordinates provider shutdown; the Gateway owns WebSocket delivery and shields cleanup from transport cancellation.
Cancellation is cooperative: `task.cancel()` requests termination, and `VoiceSession` awaits task completion before reporting a successful interruption. Startup rollback attempts to cancel any created task and abort a partially initialized STT provider. Failed cleanup is reported rather than represented as a successful retry.
## Testing
Run the automated suite:
```bash
uv run pytest -q
```
Run Gateway tests or targeted static checks:
```bash
uv run pytest -q tests/test_gateway.py
uv run ruff check src tests --select F401,F811,F821
```
Run browser controller, worklet, and playback tests:
```bash
node --test tests/js/*.test.mjs
```
### End-to-end smoke test
With the Gateway, NVIDIA ASR, and VoxCPM services available:
```bash
uv run python tools/voice_pipeline_smoke_test.py
```
This existing script uses manual turn control, not continuous server VAD.
Its `AUDIO_FILE` currently points to the absolute path
`/home/ubuntu/voice-assistant/audios/chafik_normalize.wav`; provide a real
16 kHz, mono, 16-bit PCM WAV and update that constant before running it elsewhere.
`GATEWAY_URL` must match the running port. It checks for a final transcript, an
Agent response, synthesized audio, and `turn.ended`, and writes raw output to
`/tmp/voice_pipeline_response.pcm`. It does not assess recognition accuracy.
### Continuous conversation checks and tuning
With the browser console open, test initial silence, background noise, a short
phrase, a longer phrase with natural pauses, interruption during playback, and
session closure/reconnection. Compare `[STT FINAL]` with the words actually spoken;
hearing synthesized speech alone does not confirm recognition accuracy.
During development, the phrase "fala um oi pra Lara" was transcribed with a
missing ending at 300 ms silence and recognized correctly after increasing it to
800 ms. Threshold was then increased from 0.5 to 0.6 for a background-noise trial.
These are environment-specific adjustments, not universal recommended values.
The current application interrupts on `SpeechStarted`, without an additional
minimum sustained-speech confirmation. False positives can therefore interrupt
playback; background speech and acoustic echo need separate evaluation.
Functional conversation checks do not establish a WER benchmark or resource
budget. Systematic WER, end-to-end latency distributions, and CPU/memory under
concurrent sessions have not been measured. A repeatable quality evaluation needs
recorded audio plus reference transcripts, with consistent normalization and
identical input for each configuration. Measure response latency to audible
playback separately from server synthesis completion. Results from the Mock
Agent do not characterize latency with a production LLM.
## Project structure
```text
config/ Environment YAML and provider catalog
src/voice_assistant/
audio_gateway/app/ WebSocket endpoint and protocol
application/ VoiceSession, continuous PCM history, and application events
domain/ Session state and domain rules
vad/ VAD port, speech boundaries, and local Silero adapter
audio/ Canonical audio models
stt/ STT contract, mock and NVIDIA adapter
agent/ Agent contract and mock adapter
tts/ TTS contract, mock and VoxCPM adapter
config/ Typed configuration and loading
voice_pipeline.py Turn execution and orchestration
turn_manager.py Semantic turn state machine
providers.py Provider factories
tests/ Automated tests
tools/ Integration and smoke scripts
```
## Known limitations
- **Mock Agent:** responses are simulated; the V1 does not yet include a production conversational agent.
- **Cooperative cancellation:** a coroutine that ignores cancellation may remain active after a timeout. Logical session closure does not guarantee that all external resources have terminated.
- **Provider availability:** readiness does not constitute an end-to-end health check.
- **Development-oriented deployment:** loopback endpoints and a fixed smoke-test WAV are configured for local execution.
- **Containers:** the repository's Dockerfile has not been implemented or validated for deployment.
---
**Status:** V1 development baseline · Python 3.12+ · WebSocket protocol 1.0
## Backend VAD contract
`vad/adapter.py` defines one stateful detector per continuous input stream.
Its input is the canonical PCM S16LE, mono, 16 kHz audio format. Arbitrary
complete PCM frames are buffered into model-specific windows by the adapter.
`vad/models.py` defines `SpeechStarted` and `SpeechEnded`; their sample offsets
refer to input audio since initialization or reset, with inclusive starts
and exclusive ends. Detection confirmation may arrive later than its audio
boundary. Model padding never extends the input timeline.
`process()` returns ordered boundaries, `flush()` handles normal end of input,
`reset()` discards detection state and restarts the timeline, and `aclose()`
releases resources without emitting new speech events. Calls are serialized.
These contracts do not start turns, select providers, or carry transport IDs.
The Silero adapter can control turns through the continuous VoiceSession API.
The main web client requests server detection and streams continuous PCM through
the WebSocket route. Speech detection and turn decisions run in the backend.
### Local Silero VAD adapter
`vad/silero.py` implements `VADAdapter` with the official Silero model and
`VADIterator`. `silero-vad[onnx-cpu]==6.2.3` bundles the model files; model loading
and inference use CPU locally, without runtime downloads or `torch.hub`.
The upstream package also requires PyTorch. The uv source configuration selects
CPU builds on Linux/Windows and the standard PyPI build on macOS.
Each adapter owns its model state, PCM buffer, and input timeline. It converts
PCM S16LE to normalized float samples, buffers 512-sample windows, and maps
Silero boundaries to the application events. Defaults are threshold 0.5,
300 ms of silence, 30 ms speech padding, and a 10-second worker wait limit.
These constructor defaults apply when options are omitted. The development YAML
overrides threshold to 0.6 and silence to 800 ms; tune them with representative
audio rather than changing constructor defaults.
`process()` may return several events for a large chunk. `flush()` pads a final
partial window internally while keeping event positions within real audio.
`reset()` discards detection state and restarts the timeline. `aclose()` discards
pending input and releases model references without emitting speech events.
Model loading and inference run outside the event loop. Cancellation or a worker
timeout requires reset before accepting more audio. Outstanding workers retain
ownership: reset/close wait for them, with bounded waits and retry on timeout.
Running native inference is not forcibly terminated by coroutine cancellation.
Tests with a controlled iterator cover framing, positions, isolation, and worker
ownership. `tests/test_silero_vad_integration.py` exercises the installed ONNX
model with network access blocked; it skips when the Silero dependency is absent.
It can be run with:
```bash
uv run pytest -q tests/test_silero_vad.py tests/test_silero_vad_integration.py
```
Upstream source and license: [Silero VAD](https://github.com/snakers4/silero-vad)
([MIT](https://github.com/snakers4/silero-vad/blob/master/LICENSE)).
### VAD configuration by environment
The active environment YAML configures VAD using the same `provider`/`options`
structure as STT, Agent, and TTS. `config/development.yaml` contains:
```yaml
vad:
provider: silero
options:
threshold: 0.6
min_silence_duration_ms: 800
speech_pad_ms: 30
worker_timeout_seconds: 10.0
```
`APP_ENV` selects the environment through the existing configuration loader.
`create_vad_adapter(config, catalog)` resolves the VAD from `config/providers.yaml`
and passes the environment options to its constructor. The Silero constructor
retains defaults for direct use and missing options. A missing `vad` section
remains compatible with existing configurations; explicitly creating VAD in
such an environment raises a configuration error. Changing YAML values affects
new adapter instances after configuration is reloaded; there is no live reload.
The main web client uses the VAD factory through continuous voice sessions.
Its detection settings come from the active environment YAML.
| Option | Development value | Effect |
| --- | --- | --- |
| `vad.options.threshold` | `0.6` | Speech probability activation threshold. Raising it can reduce false starts but miss quiet speech. |
| `vad.options.min_silence_duration_ms` | `800` | Silence required to end detected speech. Longer waits preserve pauses but delay finalization. |
| `vad.options.speech_pad_ms` | `30` | Padding around detected boundaries; does not confirm sustained speech. |
| `vad.options.worker_timeout_seconds` | `10.0` | Maximum wait for a model worker operation; not an endpointing delay or forced thread termination. |
| `voice_input.pre_roll_ms` | `200` | Audio retained before the detected start and forwarded to STT, bounded by history and the previous utterance. |
| `voice_input.history_ms` | `2000` | Retained input history for delayed boundary events; must cover pre-roll. |
| `voice_input.max_input_chunk_bytes` | `65536` | Maximum continuous PCM input chunk, also capped by the Gateway at 64 KiB. |
Tune one parameter at a time using the same audio. End-of-speech silence does
not filter start-of-speech noise triggers. No additional interruption-confirmation
parameter is implemented; that would require an application policy change.
### Continuous audio and backend turn control
A `VoiceSession` constructed with a VAD receives continuous canonical PCM through
`await session.receive_audio(pcm)`. The caller supplies audio without a turn ID;
the channel owns transport sequencing. `SpeechStarted` opens an STT turn and
`SpeechEnded` schedules normal STT finalization in the background. New speech
interrupts an active response/finalization before opening a fresh turn. Empty
STT results release the turn through an interruption event rather than leaving
an active turn behind. The VAD is not reset between conversation turns.
The application's sample-indexed audio history preserves the beginning of speech
when VAD confirmation arrives later. PCM already sent to a turn is not replayed
within that turn; pre-roll never includes the previous utterance. History and
input size are bounded by environment configuration:
```yaml
voice_input:
pre_roll_ms: 200
history_ms: 2000
max_input_chunk_bytes: 65536
```
History is trimmed after processing each input chunk; transient storage is at most
retained history plus one accepted chunk. A speech start older than retained
history is reported as an input error instead of silently dropping speech.
Invalid PCM is rejected before detection or timeline advancement. A processing
failure requires a new session; cleanup still releases VAD and provider resources.
Use `finish_audio_input()` for normal end of a continuous stream. Session end or
disconnect aborts through `end()`/`aclose()`, without forcing a final speech event.
The caller shields cleanup from transport cancellation, as for manual sessions.
Backend-controlled sessions emit `TurnOpened` and `TurnInterrupted`; the WebSocket
mapper translates them to the existing `turn.started` and `interrupted` events.
Adapters should discard outdated queued playback when a new turn opens: backend
turn completion describes produced output, not physical speaker completion.
Manual `start_turn`/`send_audio`/`stop_turn` commands remain available on sessions
without VAD. Mixing those commands with a VAD-controlled session is rejected.
`create_voice_session(..., server_vad=True)` composes the continuous backend API;
`/ws/audio` selects this mode when `session.start.turn_detection` is `server`.
Omitting that option selects the existing manual input mode for compatible clients.
### Browser continuous capture
The main `/ui` client requests `turn_detection: server` and waits for the matching
confirmation in `session.ready` before starting microphone capture. The worklet
then stays active throughout listening, processing, response playback, and
interruption. Input sequence numbers do not reset between turns. Microphone
permission, echo cancellation, and audio device ownership remain in the page.
The browser's custom `vad.js` and its detection tests have been removed. The voice
controller now handles connection, continuous PCM transport, protocol events,
and playback. Existing visual amplitude analysis is used only for animation.
It sends no automatic `audio.start`, `audio.stop`, or `interrupt` commands.
A new `turn.started` stops previous playback and discards obsolete queued audio.
`turn.ended` waits for scheduled speaker playback before updating the display to
listening; microphone streaming continues during that wait. Stale turn events
and completion callbacks cannot overwrite a newer turn. Outbound WebSocket
buffering and handshake waiting are bounded; failures stop local capture and
close the session.
`audio.input.end` finishes a finite continuous stream normally and requires no
pending binary frame. It is intended for file/terminal clients; the live browser
uses `session.end` or disconnect to abort. Manual input remains available with
`turn_detection: client`, including the standalone manual `app.js` and existing
smoke scripts. The main page uses `voice-session.js`.