voiceagent/README.md

Voice Assistant

Realtime speech processing over WebSocket — NVIDIA ASR · Agent orchestration · VoxCPM TTS

A modular voice pipeline for streaming audio input, speech transcription, agent responses, and synthesized speech output. Built with FastAPI and asynchronous Python, with explicit session lifecycle, turn ownership, interruption, and provider cleanup.

V1 scope: NVIDIA STT and VoxCPM TTS are real providers; the Agent is a mock implementation. Production readiness and container deployment have not yet been validated.

Contents

• Architecture
• Quickstart
• Configuration
• WebSocket protocol
• Lifecycle and interruption
• Testing
• Project structure
• Known limitations
• Architecture Backbone & Abstraction Boundaries

Architecture

flowchart TB
    C[Audio Client] <-->|WebSocket /ws/audio · JSON + binary PCM| G[Audio Gateway · FastAPI]
    G --> U[VoiceSession · conversation lifecycle]
    U --> P[VoicePipeline]
    P --- S[SessionState · session and audio sequence]
    P --- T[TurnManager · semantic turn state]
    U --> STT[NVIDIA ASR · STT per turn]
    U --> VAD[Silero VAD · local CPU · per continuous session]
    STT -->|Transcript events| P
    P --> A[Mock Agent · per session]
    A -->|Response text| P
    P --> V[VoxCPM TTS · per session]
    V -->|PCM chunks| G
    G -->|Streamed audio| C

The Audio Gateway handles transport, protocol validation, and application composition. VoiceSession coordinates STT, finalization, interruption, rollback, and provider cleanup. VoicePipeline retains processing, turn-state operations, and execution-task ownership. SessionState tracks session and audio sequence; TurnManager enforces turn transitions. A new STT adapter is created for each turn, while Agent and TTS adapters are created per session.

Architecture Backbone & Abstraction Boundaries

AgentVoice follows a provider-independent architecture built around explicit contracts, lifecycle ownership, and asynchronous streaming.

The architecture separates five concerns:

• Transport: WebSocket communication and protocol serialization.
• Orchestration: Coordination of STT, Agent, and TTS execution.
• Lifecycle: Session state, turn transitions, interruption, and task ownership.
• Provider contracts: Abstract interfaces defining component behavior.
• Provider implementations: Concrete integrations with NVIDIA, OCI Generative AI, and VoxCPM.

Architectural Backbone

flowchart TB
    Client["Web Client / Audio Client"]
    Gateway["Audio Gateway<br/>FastAPI · WebSocket"]
    UseCase["VoiceSession<br/>Conversation Lifecycle"]
    Pipeline["VoicePipeline<br/>Orchestration & Streaming"]
    Session["SessionState<br/>Session Lifecycle"]
    Turns["TurnManager<br/>Semantic Turn Lifecycle"]

    subgraph Contracts["Provider Abstraction Layer"]
        direction LR
        STTA["STTAdapter<br/>Abstract Contract"]
        AgentA["AgentAdapter<br/>Abstract Contract"]
        TTSA["TTSAdapter<br/>Abstract Contract"]
    end

    subgraph Implementations["Provider Implementation Layer"]
        direction LR
        Nvidia["NvidiaSTTAdapter"]
        OCI["OCIAgentAdapter"]
        Mock["MockAgentAdapter"]
        Vox["VoxCPMTTSAdapter"]
    end

    subgraph External["External AI Services"]
        ASR["NVIDIA ASR"]
        LLM["OCI Generative AI"]
        TTS["VoxCPM"]
    end

    Client <-->|"JSON Events + Binary PCM"| Gateway
    Gateway --> UseCase
    UseCase --> Pipeline

    Pipeline --- Session
    Pipeline --- Turns

    Pipeline --> STTA
    Pipeline --> AgentA
    Pipeline --> TTSA

    STTA -.->|implements| Nvidia
    AgentA -.->|implements| OCI
    AgentA -.->|implements| Mock
    TTSA -.->|implements| Vox

    Nvidia --> ASR
    OCI --> LLM
    Vox --> TTS

The VoicePipeline depends on provider contracts rather than concrete implementations. Provider selection is performed through configuration and dynamic factories.

This allows implementations to evolve without requiring the Audio Gateway to understand provider-specific APIs.

Provider Contracts

Abstraction	Contract	Implementations	Responsibility
Speech-to-text	STTAdapter	NvidiaSTTAdapter, MockSTTAdapter	Audio ingestion and transcript events
Agent	AgentAdapter	OCIAgentAdapter, MockAgentAdapter	Text processing and conversational responses
Text-to-speech	TTSAdapter	VoxCPMTTSAdapter, MockTTSAdapter	Speech synthesis and audio streaming

The AgentAdapter defines complete-response processing and session cleanup. StreamingAgentAdapter explicitly adds incremental text generation; OCI and DeepAgents implement this contract. Streaming returns an asynchronous generator that the consumer closes on early exit. TTSAdapter defines the same ownership rule for synthesized audio. Both contracts expose aclose(); providers retaining session resources override its default no-op implementation.

Runtime Composition

Provider implementations are selected through environment configuration and resolved by the provider factory.

flowchart LR
    Config["Environment YAML"]
    Catalog["Provider Catalog"]
    Factory["Provider Factories"]
    Contracts["Adapter Contracts"]
    Runtime["VoicePipeline"]

    Config --> Factory
    Catalog --> Factory
    Factory -->|Validate implementation| Contracts
    Factory -->|Instantiate providers| Runtime

The primary configuration components are:

Component	Location
Environment selection	config/loader.py
Typed configuration	config/models.py
Provider catalog	config/providers.yaml
Dynamic provider loading	providers.py
Session lifecycle	domain/session.py
Semantic turn lifecycle	turn_manager.py
Conversation lifecycle	application/voice_session.py
Application output events	application/events.py
Processing and execution tasks	voice_pipeline.py

Streaming Execution Backbone

The streaming pipeline allows OCI Generative AI to continue producing text while VoxCPM synthesizes previously completed text segments.

flowchart TB
    Input["Microphone / PCM Input"]
    Gateway["Audio Gateway"]
    STT["NVIDIA STT"]
    Transcript["Final Transcript"]

    subgraph Pipeline["VoicePipeline · Concurrent Execution"]
        direction TB
        Agent["OCIAgentAdapter.stream()"]
        Bridge["OCI SSE Async Bridge"]
        Chunker["TextChunker"]
        Queue["Bounded Segment Queue"]
        Synth["VoxCPM TTS Streaming"]

        Agent --> Bridge
        Bridge --> Chunker
        Chunker --> Queue
        Queue --> Synth
    end

    OCI["OCI Generative AI"]
    Vox["VoxCPM Service"]
    Output["WebSocket PCM Output"]

    Input --> Gateway
    Gateway --> UseCase["VoiceSession"]
    UseCase --> STT
    STT --> Transcript
    Transcript --> Agent

    Agent <--> OCI
    Synth <--> Vox
    Synth --> Gateway
    Gateway --> Output

The pipeline uses bounded queues and asynchronous task coordination to support incremental generation, ordered synthesis, and cooperative cancellation.

Lifecycle Ownership

AgentVoice separates semantic state from asynchronous execution.

flowchart LR
    Listening["LISTENING"]
    Processing["PROCESSING"]
    Speaking["SPEAKING"]
    Completed["COMPLETED"]
    Interrupted["INTERRUPTED"]

    Listening -->|"STT final"| Processing
    Processing -->|"First TTS segment"| Speaking
    Speaking -->|"Synthesis complete"| Completed

    Listening --> Interrupted
    Processing --> Interrupted
    Speaking --> Interrupted

Ownership responsibilities:

Component	Owns
Audio Gateway	WebSocket connection, protocol validation, session composition, event delivery, and shielded application cleanup
SessionState	Session lifecycle and audio sequence validation
TurnManager	Semantic turn state transitions
VoiceSession	Input history, VAD boundaries, STT lifecycle, finalization, interruption, and provider cleanup
VoicePipeline	Turn execution task and Agent/TTS orchestration
OCI streaming bridge	SSE reader, event queue, and stream closure
Provider adapters	Provider-specific requests, connections, and data formats

Cancellation is cooperative. The system requests cancellation and waits for task termination, but termination of blocking external operations depends on the underlying provider and transport behavior.

Architectural Extension Points

The abstraction layer allows additional implementations to be introduced through the provider catalog.

For example, a future LiteLLM integration could implement the Agent contract without requiring a redesign of the Gateway or the semantic turn lifecycle.

flowchart TB
    Pipeline["VoicePipeline"]
    Contract["AgentAdapter"]

    OCI["OCIAgentAdapter<br/>Current"]
    Mock["MockAgentAdapter<br/>Testing"]
    Lite["LiteLLMAgentAdapter<br/>Future"]

    Pipeline --> Contract
    Contract -.-> OCI
    Contract -.-> Mock
    Contract -.-> Lite

Design principle: transport, orchestration, lifecycle, and provider-specific integrations remain separate concerns. Provider independence is established through contracts and factories, while streaming and cancellation behavior are validated at their respective integration boundaries.

DeepAgents: autenticação e streaming

No Mac, API_KEY lê o perfil OCI e a chave indicados pelo arquivo de configuração. Para uma sessão OCI CLI, use SECURITY_TOKEN e um perfil com token válido. O DeepAgents também aceita INSTANCE_PRINCIPAL e RESOURCE_PRINCIPAL em ambientes OCI compatíveis. O padrão do construtor permanece INSTANCE_PRINCIPAL quando auth_type é omitido. Credenciais e permissões precisam ser válidas. Dentro do container, monte o arquivo OCI e as chaves/tokens necessários somente para leitura, com caminhos acessíveis ao usuário do processo, e ajuste auth_file_location; ~ refere-se ao usuário do container, não ao do Mac. Não há integração MCP/RAG automática nesse bloco; a implementação usa tools=[]. A entrada deepagents do catálogo seleciona OCIDeepAgentsAdapter, que compõe modelo OCI, grafo e memória. DeepAgentsAdapter recebe um grafo pronto e não importa o SDK OCI. A configuração existente continua válida; chamadas Python diretas com opções OCI devem usar a nova classe de composição.

initialize() prepara a sessão antes de session.ready. A criação/autenticação do modelo ocorre em worker fora do event loop. initialization_timeout_seconds (padrão 5 s) limita essa espera; cleanup_timeout_seconds (padrão 1 s) limita a espera pelo worker durante o fechamento. Um timeout não mata a thread: a tarefa retém a posse do cliente e o fecha quando o worker terminar. Se ainda houver trabalho pendente, o fechamento reporta timeout, em vez de confirmar liberação. A limpeza fecha tanto o HTTP assíncrono do modelo quanto a sessão HTTP do SDK. Prazos maiores devem considerar o limite atual de 10 s do handshake da UI.

tools=[] fornece zero ferramentas adicionais; as ferramentas nativas do DeepAgents continuam habilitadas. Essa entrega preserva essa política. O backend de arquivos padrão do framework usa estado virtual; não implica acesso automático aos arquivos do host. Ferramentas futuras devem ser selecionadas na composição.

agent.options.timeout_seconds limita a espera por cada próximo evento do DeepAgents (inclusive eventos internos), sem incluir o tempo em que o consumidor está processando um fragmento já entregue. Não é prazo total da resposta; a inicialização tem um limite separado. A expiração fecha o stream e propaga erro.

Quickstart

Requirements

• Python 3.12 or newer and uv
• Accessible NVIDIA ASR and VoxCPM services
• Run commands from the repository root (config/ is resolved relative to the working directory)

Install

uv sync

Start the gateway

With the configured provider services running:

APP_ENV=development uv run uvicorn \
  voice_assistant.audio_gateway.app.main:app \
  --host 127.0.0.1 --port 8001

Open http://127.0.0.1:8001/ui, allow microphone access, and click Iniciar conversa. The client sends continuous audio; backend Silero detects turns and new speech interrupts an active response. Audio capture continues during playback.

Transcripts currently appear in the browser developer console as [STT PARTIAL] and [STT FINAL]; Agent text appears as [AGENT RESPONSE] or [AGENT DELTA]. On Firefox for macOS, open the console with Command + Option + K. The page displays conversation status and plays synthesized audio.

The Gateway and VAD run locally without LiveKit or a LiveKit Cloud account. STT and TTS are separate services and may run remotely; the loopback addresses in development must point to local services or active port forwards. Audio is sent to those services, so the full processing path is not offline. Model and Python dependencies must be installed before offline VAD execution.

If port 8001 is occupied, choose another port (for example, 8002) and use that same port in the browser, health checks, and WebSocket tools.

Check availability:

curl http://127.0.0.1:8001/ready
# {"status":"ready"}

/ready reports Gateway availability; it does not verify NVIDIA ASR or VoxCPM health.

Configuration

APP_ENV selects config/<environment>.yaml; the default is development. config/providers.yaml maps provider names to Python implementations and capability metadata. YAML data is validated using Pydantic models.

Component	Development provider	Configuration
Speech-to-text	NVIDIA ASR	127.0.0.1:9000, pt-BR, punctuation enabled
Agent	Mock Agent	In-process; simulated text response
Text-to-speech	VoxCPM	http://127.0.0.1:8000, openbmb/VoxCPM2, default voice
Voice activity	Silero ONNX	Local CPU; threshold 0.6 and 800 ms silence

To select another environment:

APP_ENV=staging uv run uvicorn \
  voice_assistant.audio_gateway.app.main:app \
  --host 127.0.0.1 --port 8001

The staging and production files are currently empty placeholders. Populate the selected file before using it; the loader rejects an empty YAML. Start from development.yaml, then configure reachable provider endpoints and VAD/input settings for that environment. Deployment readiness has not been established.

TTS voice selection is tts.options.voice in the selected environment YAML. The service determines which voice identifiers are supported. Configuration is loaded at Gateway startup: restart the process and open a new conversation after changing YAML; refreshing the browser alone does not reload server configuration.

WebSocket protocol

Property	Value
Endpoint	ws://127.0.0.1:8001/ws/audio
Protocol version	1.0
Input audio	PCM signed 16-bit little-endian, 16 kHz, mono
Output audio	Declared by session.ready.output_audio; current VoxCPM: PCM S16LE, 48 kHz, mono
Transport:	JSON control messages and binary PCM frames.

Typical exchange

sequenceDiagram
    participant C as Client
    participant G as Audio Gateway
    participant P as VoicePipeline / Providers
    C->>G: session.start (turn_detection=server)
    G-->>C: session.ready
    loop Continuous audio input, including silence and playback
        C->>G: audio.input (sequence_number)
        C->>G: Binary PCM frame
        G-->>C: turn.started (when backend VAD detects speech)
        P-->>G: STT partial events
        G-->>C: stt.partial
    end
    Note over G,P: Backend VAD ends input after silence
    P-->>G: STT final transcript
    G-->>C: stt.final
    P-->>G: Agent response
    G-->>C: agent.response
    P-->>G: TTS PCM chunks
    G-->>C: Binary audio frames
    G-->>C: turn.ended

STT events may arrive while input audio is still streaming. In server detection mode, each binary PCM frame is preceded by audio.input with a session-wide sequence_number, starting at zero and continuing across turns. session.ready.max_audio_chunk_bytes declares the accepted limit, capped at 64 KiB. The browser sizes worklet chunks to respect it. Manual mode uses audio.chunk with turn_id and a per-turn sequence. Input frames must contain complete PCM samples; rejected frames do not advance the audio sequence. session.ready.audio describes input and session.ready.output_audio describes output. The browser configures playback from the declared output format. VoxCPM reassembles HTTP fragments into complete PCM frames and rejects a truncated final sample. Its sample_rate option declares the expected native output rate; it does not request resampling from the external service.

Interruption and session closure

• In server detection mode, SpeechStarted makes VoiceSession interrupt an active turn before opening a new one. The Gateway maps application events to interrupted and turn.started; the client stops obsolete playback.
• Explicit interrupt targets a turn_id. For a matching active turn, VoiceSession cancels and awaits execution, aborts STT, and releases turn state. The Gateway then delivers the acknowledgement.
• session.end requests cleanup and session closure; successful completion is acknowledged with session.ended.
• Errors are returned as structured error events (for example, STT_START_FAILED or TURN_CANCEL_FAILED).

Lifecycle and interruption

The system separates semantic state from asynchronous execution. TurnManager controls listening, processing, speaking, and completion; VoicePipeline owns the registered turn task. VoiceSession coordinates provider shutdown; the Gateway owns WebSocket delivery and shields cleanup from transport cancellation.

Cancellation is cooperative: task.cancel() requests termination, and VoiceSession awaits task completion before reporting a successful interruption. Startup rollback attempts to cancel any created task and abort a partially initialized STT provider. Failed cleanup is reported rather than represented as a successful retry.

Testing

Run the automated suite:

uv run pytest -q

Run Gateway tests or targeted static checks:

uv run pytest -q tests/test_gateway.py
uv run ruff check src tests --select F401,F811,F821

Run browser controller, worklet, and playback tests:

node --test tests/js/*.test.mjs

End-to-end smoke test

With the Gateway, NVIDIA ASR, and VoxCPM services available:

uv run python tools/voice_pipeline_smoke_test.py

This existing script uses manual turn control, not continuous server VAD. Its AUDIO_FILE currently points to the absolute path /home/ubuntu/voice-assistant/audios/chafik_normalize.wav; provide a real 16 kHz, mono, 16-bit PCM WAV and update that constant before running it elsewhere. GATEWAY_URL must match the running port. It checks for a final transcript, an Agent response, synthesized audio, and turn.ended, and writes raw output to /tmp/voice_pipeline_response.pcm. It does not assess recognition accuracy.

Continuous conversation checks and tuning

With the browser console open, test initial silence, background noise, a short phrase, a longer phrase with natural pauses, interruption during playback, and session closure/reconnection. Compare [STT FINAL] with the words actually spoken; hearing synthesized speech alone does not confirm recognition accuracy.

During development, the phrase "fala um oi pra Lara" was transcribed with a missing ending at 300 ms silence and recognized correctly after increasing it to 800 ms. Threshold was then increased from 0.5 to 0.6 for a background-noise trial. These are environment-specific adjustments, not universal recommended values. The current application interrupts on SpeechStarted, without an additional minimum sustained-speech confirmation. False positives can therefore interrupt playback; background speech and acoustic echo need separate evaluation.

Functional conversation checks do not establish a WER benchmark or resource budget. Systematic WER, end-to-end latency distributions, and CPU/memory under concurrent sessions have not been measured. A repeatable quality evaluation needs recorded audio plus reference transcripts, with consistent normalization and identical input for each configuration. Measure response latency to audible playback separately from server synthesis completion. Results from the Mock Agent do not characterize latency with a production LLM.

Project structure

config/                         Environment YAML and provider catalog
src/voice_assistant/
  audio_gateway/app/            WebSocket endpoint and protocol
  application/                  VoiceSession, continuous PCM history, and application events
  domain/                       Session state and domain rules
  vad/                          VAD port, speech boundaries, and local Silero adapter
  audio/                        Canonical audio models
  stt/                          STT contract, mock and NVIDIA adapter
  agent/                        Agent contract and mock adapter
  tts/                          TTS contract, mock and VoxCPM adapter
  config/                       Typed configuration and loading
  voice_pipeline.py             Turn execution and orchestration
  turn_manager.py               Semantic turn state machine
  providers.py                  Provider factories
tests/                          Automated tests
tools/                          Integration and smoke scripts

Known limitations

• Mock Agent: responses are simulated; the V1 does not yet include a production conversational agent.
• Cooperative cancellation: a coroutine that ignores cancellation may remain active after a timeout. Logical session closure does not guarantee that all external resources have terminated.
• Provider availability: readiness does not constitute an end-to-end health check.
• Development-oriented deployment: loopback endpoints and a fixed smoke-test WAV are configured for local execution.
• Containers: the repository's Dockerfile has not been implemented or validated for deployment.

---

Status: V1 development baseline · Python 3.12+ · WebSocket protocol 1.0

Backend VAD contract

vad/adapter.py defines one stateful detector per continuous input stream. Its input is the canonical PCM S16LE, mono, 16 kHz audio format. Arbitrary complete PCM frames are buffered into model-specific windows by the adapter. vad/models.py defines SpeechStarted and SpeechEnded; their sample offsets refer to input audio since initialization or reset, with inclusive starts and exclusive ends. Detection confirmation may arrive later than its audio boundary. Model padding never extends the input timeline.

process() returns ordered boundaries, flush() handles normal end of input, reset() discards detection state and restarts the timeline, and aclose() releases resources without emitting new speech events. Calls are serialized. These contracts do not start turns, select providers, or carry transport IDs. The Silero adapter can control turns through the continuous VoiceSession API. The main web client requests server detection and streams continuous PCM through the WebSocket route. Speech detection and turn decisions run in the backend.

Local Silero VAD adapter

vad/silero.py implements VADAdapter with the official Silero model and VADIterator. silero-vad[onnx-cpu]==6.2.3 bundles the model files; model loading and inference use CPU locally, without runtime downloads or torch.hub. The upstream package also requires PyTorch. The uv source configuration selects CPU builds on Linux/Windows and the standard PyPI build on macOS.

Each adapter owns its model state, PCM buffer, and input timeline. It converts PCM S16LE to normalized float samples, buffers 512-sample windows, and maps Silero boundaries to the application events. Defaults are threshold 0.5, 300 ms of silence, 30 ms speech padding, and a 10-second worker wait limit. These constructor defaults apply when options are omitted. The development YAML overrides threshold to 0.6 and silence to 800 ms; tune them with representative audio rather than changing constructor defaults.

process() may return several events for a large chunk. flush() pads a final partial window internally while keeping event positions within real audio. reset() discards detection state and restarts the timeline. aclose() discards pending input and releases model references without emitting speech events.

Model loading and inference run outside the event loop. Cancellation or a worker timeout requires reset before accepting more audio. Outstanding workers retain ownership: reset/close wait for them, with bounded waits and retry on timeout. Running native inference is not forcibly terminated by coroutine cancellation.

Tests with a controlled iterator cover framing, positions, isolation, and worker ownership. tests/test_silero_vad_integration.py exercises the installed ONNX model with network access blocked; it skips when the Silero dependency is absent. It can be run with:

uv run pytest -q tests/test_silero_vad.py tests/test_silero_vad_integration.py

Upstream source and license: Silero VAD (MIT).

VAD configuration by environment

The active environment YAML configures VAD using the same provider/options structure as STT, Agent, and TTS. config/development.yaml contains:

vad:
  provider: silero
  options:
    threshold: 0.6
    min_silence_duration_ms: 800
    speech_pad_ms: 30
    worker_timeout_seconds: 10.0

APP_ENV selects the environment through the existing configuration loader. create_vad_adapter(config, catalog) resolves the VAD from config/providers.yaml and passes the environment options to its constructor. The Silero constructor retains defaults for direct use and missing options. A missing vad section remains compatible with existing configurations; explicitly creating VAD in such an environment raises a configuration error. Changing YAML values affects new adapter instances after configuration is reloaded; there is no live reload.

The main web client uses the VAD factory through continuous voice sessions. Its detection settings come from the active environment YAML.

Option	Development value	Effect
vad.options.threshold	0.6	Speech probability activation threshold. Raising it can reduce false starts but miss quiet speech.
vad.options.min_silence_duration_ms	800	Silence required to end detected speech. Longer waits preserve pauses but delay finalization.
vad.options.speech_pad_ms	30	Padding around detected boundaries; does not confirm sustained speech.
vad.options.worker_timeout_seconds	10.0	Maximum wait for a model worker operation; not an endpointing delay or forced thread termination.
voice_input.pre_roll_ms	200	Audio retained before the detected start and forwarded to STT, bounded by history and the previous utterance.
voice_input.history_ms	2000	Retained input history for delayed boundary events; must cover pre-roll.
voice_input.max_input_chunk_bytes	65536	Maximum continuous PCM input chunk, also capped by the Gateway at 64 KiB.

Tune one parameter at a time using the same audio. End-of-speech silence does not filter start-of-speech noise triggers. No additional interruption-confirmation parameter is implemented; that would require an application policy change.

Continuous audio and backend turn control

A VoiceSession constructed with a VAD receives continuous canonical PCM through await session.receive_audio(pcm). The caller supplies audio without a turn ID; the channel owns transport sequencing. SpeechStarted opens an STT turn and SpeechEnded schedules normal STT finalization in the background. New speech interrupts an active response/finalization before opening a fresh turn. Empty STT results release the turn through an interruption event rather than leaving an active turn behind. The VAD is not reset between conversation turns.

The application's sample-indexed audio history preserves the beginning of speech when VAD confirmation arrives later. PCM already sent to a turn is not replayed within that turn; pre-roll never includes the previous utterance. History and input size are bounded by environment configuration:

voice_input:
  pre_roll_ms: 200
  history_ms: 2000
  max_input_chunk_bytes: 65536

History is trimmed after processing each input chunk; transient storage is at most retained history plus one accepted chunk. A speech start older than retained history is reported as an input error instead of silently dropping speech. Invalid PCM is rejected before detection or timeline advancement. A processing failure requires a new session; cleanup still releases VAD and provider resources.

Use finish_audio_input() for normal end of a continuous stream. Session end or disconnect aborts through end()/aclose(), without forcing a final speech event. The caller shields cleanup from transport cancellation, as for manual sessions.

Backend-controlled sessions emit TurnOpened and TurnInterrupted; the WebSocket mapper translates them to the existing turn.started and interrupted events. Adapters should discard outdated queued playback when a new turn opens: backend turn completion describes produced output, not physical speaker completion.

Manual start_turn/send_audio/stop_turn commands remain available on sessions without VAD. Mixing those commands with a VAD-controlled session is rejected. create_voice_session(..., server_vad=True) composes the continuous backend API; /ws/audio selects this mode when session.start.turn_detection is server. Omitting that option selects the existing manual input mode for compatible clients.

Browser continuous capture

The main /ui client requests turn_detection: server and waits for the matching confirmation in session.ready before starting microphone capture. The worklet then stays active throughout listening, processing, response playback, and interruption. Input sequence numbers do not reset between turns. Microphone permission, echo cancellation, and audio device ownership remain in the page.

The browser's custom vad.js and its detection tests have been removed. The voice controller now handles connection, continuous PCM transport, protocol events, and playback. Existing visual amplitude analysis is used only for animation. It sends no automatic audio.start, audio.stop, or interrupt commands.

A new turn.started stops previous playback and discards obsolete queued audio. turn.ended waits for scheduled speaker playback before updating the display to listening; microphone streaming continues during that wait. Stale turn events and completion callbacks cannot overwrite a newer turn. Outbound WebSocket buffering and handshake waiting are bounded; failures stop local capture and close the session.

audio.input.end finishes a finite continuous stream normally and requires no pending binary frame. It is intended for file/terminal clients; the live browser uses session.end or disconnect to abort. Manual input remains available with turn_detection: client, including the standalone manual app.js and existing smoke scripts. The main page uses voice-session.js.