502 lines
20 KiB
Markdown
502 lines
20 KiB
Markdown
# Voice Assistant
|
|
|
|
**Realtime speech processing over WebSocket — NVIDIA ASR · Agent orchestration · VoxCPM TTS**
|
|
|
|
  
|
|
|
|
A modular voice pipeline for streaming audio input, speech transcription, agent responses, and synthesized speech output. Built with FastAPI and asynchronous Python, with explicit session lifecycle, turn ownership, interruption, and provider cleanup.
|
|
|
|
> **V1 scope:** NVIDIA STT and VoxCPM TTS are real providers; the Agent is a mock implementation. Production readiness and container deployment have not yet been validated.
|
|
|
|
## Contents
|
|
|
|
- [Architecture](#architecture)
|
|
- [Quickstart](#quickstart)
|
|
- [Configuration](#configuration)
|
|
- [WebSocket protocol](#websocket-protocol)
|
|
- [Lifecycle and interruption](#lifecycle-and-interruption)
|
|
- [Testing](#testing)
|
|
- [Project structure](#project-structure)
|
|
- [Known limitations](#known-limitations)
|
|
- [Architecture Backbone & Abstraction Boundaries](#architecture-backbone--abstraction-boundaries)
|
|
|
|
## Architecture
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
C[Audio Client] <-->|WebSocket /ws/audio · JSON + binary PCM| G[Audio Gateway · FastAPI]
|
|
G --> U[VoiceSession · conversation lifecycle]
|
|
U --> P[VoicePipeline]
|
|
P --- S[SessionState · session and audio sequence]
|
|
P --- T[TurnManager · semantic turn state]
|
|
G --> STT[NVIDIA ASR · STT per turn]
|
|
STT -->|Transcript events| P
|
|
P --> A[Mock Agent · per session]
|
|
A -->|Response text| P
|
|
P --> V[VoxCPM TTS · per session]
|
|
V -->|PCM chunks| G
|
|
G -->|Streamed audio| C
|
|
```
|
|
|
|
The **Audio Gateway** handles transport, protocol validation, and application composition. **VoiceSession** coordinates STT, finalization, interruption, rollback, and provider cleanup. **VoicePipeline** retains processing, turn-state operations, and execution-task ownership. **SessionState** tracks session and audio sequence; **TurnManager** enforces turn transitions. A new STT adapter is created for each turn, while Agent and TTS adapters are created per session.
|
|
|
|
## Architecture Backbone & Abstraction Boundaries
|
|
|
|
AgentVoice follows a **provider-independent architecture** built around explicit contracts, lifecycle ownership, and asynchronous streaming.
|
|
|
|
The architecture separates five concerns:
|
|
|
|
- **Transport:** WebSocket communication and protocol serialization.
|
|
- **Orchestration:** Coordination of STT, Agent, and TTS execution.
|
|
- **Lifecycle:** Session state, turn transitions, interruption, and task ownership.
|
|
- **Provider contracts:** Abstract interfaces defining component behavior.
|
|
- **Provider implementations:** Concrete integrations with NVIDIA, OCI Generative AI, and VoxCPM.
|
|
|
|
### Architectural Backbone
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
Client["Web Client / Audio Client"]
|
|
Gateway["Audio Gateway<br/>FastAPI · WebSocket"]
|
|
UseCase["VoiceSession<br/>Conversation Lifecycle"]
|
|
Pipeline["VoicePipeline<br/>Orchestration & Streaming"]
|
|
Session["SessionState<br/>Session Lifecycle"]
|
|
Turns["TurnManager<br/>Semantic Turn Lifecycle"]
|
|
|
|
subgraph Contracts["Provider Abstraction Layer"]
|
|
direction LR
|
|
STTA["STTAdapter<br/>Abstract Contract"]
|
|
AgentA["AgentAdapter<br/>Abstract Contract"]
|
|
TTSA["TTSAdapter<br/>Abstract Contract"]
|
|
end
|
|
|
|
subgraph Implementations["Provider Implementation Layer"]
|
|
direction LR
|
|
Nvidia["NvidiaSTTAdapter"]
|
|
OCI["OCIAgentAdapter"]
|
|
Mock["MockAgentAdapter"]
|
|
Vox["VoxCPMTTSAdapter"]
|
|
end
|
|
|
|
subgraph External["External AI Services"]
|
|
ASR["NVIDIA ASR"]
|
|
LLM["OCI Generative AI"]
|
|
TTS["VoxCPM"]
|
|
end
|
|
|
|
Client <-->|"JSON Events + Binary PCM"| Gateway
|
|
Gateway --> UseCase
|
|
UseCase --> Pipeline
|
|
|
|
Pipeline --- Session
|
|
Pipeline --- Turns
|
|
|
|
Pipeline --> STTA
|
|
Pipeline --> AgentA
|
|
Pipeline --> TTSA
|
|
|
|
STTA -.->|implements| Nvidia
|
|
AgentA -.->|implements| OCI
|
|
AgentA -.->|implements| Mock
|
|
TTSA -.->|implements| Vox
|
|
|
|
Nvidia --> ASR
|
|
OCI --> LLM
|
|
Vox --> TTS
|
|
```
|
|
|
|
The `VoicePipeline` depends on provider contracts rather than concrete implementations. Provider selection is performed through configuration and dynamic factories.
|
|
|
|
This allows implementations to evolve without requiring the Audio Gateway to understand provider-specific APIs.
|
|
|
|
### Provider Contracts
|
|
|
|
| Abstraction | Contract | Implementations | Responsibility |
|
|
|---|---|---|---|
|
|
| Speech-to-text | `STTAdapter` | `NvidiaSTTAdapter`, `MockSTTAdapter` | Audio ingestion and transcript events |
|
|
| Agent | `AgentAdapter` | `OCIAgentAdapter`, `MockAgentAdapter` | Text processing and conversational responses |
|
|
| Text-to-speech | `TTSAdapter` | `VoxCPMTTSAdapter`, `MockTTSAdapter` | Speech synthesis and audio streaming |
|
|
|
|
The `AgentAdapter` defines complete-response processing and session cleanup. `StreamingAgentAdapter` explicitly adds incremental text generation; OCI and DeepAgents implement this contract. Streaming returns an asynchronous generator that the consumer closes on early exit. `TTSAdapter` defines the same ownership rule for synthesized audio. Both contracts expose `aclose()`; providers retaining session resources override its default no-op implementation.
|
|
|
|
### Runtime Composition
|
|
|
|
Provider implementations are selected through environment configuration and resolved by the provider factory.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
Config["Environment YAML"]
|
|
Catalog["Provider Catalog"]
|
|
Factory["Provider Factories"]
|
|
Contracts["Adapter Contracts"]
|
|
Runtime["VoicePipeline"]
|
|
|
|
Config --> Factory
|
|
Catalog --> Factory
|
|
Factory -->|Validate implementation| Contracts
|
|
Factory -->|Instantiate providers| Runtime
|
|
```
|
|
|
|
The primary configuration components are:
|
|
|
|
| Component | Location |
|
|
|---|---|
|
|
| Environment selection | `config/loader.py` |
|
|
| Typed configuration | `config/models.py` |
|
|
| Provider catalog | `config/providers.yaml` |
|
|
| Dynamic provider loading | `providers.py` |
|
|
| Session lifecycle | `domain/session.py` |
|
|
| Semantic turn lifecycle | `turn_manager.py` |
|
|
| Conversation lifecycle | `application/voice_session.py` |
|
|
| Application output events | `application/events.py` |
|
|
| Processing and execution tasks | `voice_pipeline.py` |
|
|
|
|
### Streaming Execution Backbone
|
|
|
|
The streaming pipeline allows OCI Generative AI to continue producing text while VoxCPM synthesizes previously completed text segments.
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
Input["Microphone / PCM Input"]
|
|
Gateway["Audio Gateway"]
|
|
STT["NVIDIA STT"]
|
|
Transcript["Final Transcript"]
|
|
|
|
subgraph Pipeline["VoicePipeline · Concurrent Execution"]
|
|
direction TB
|
|
Agent["OCIAgentAdapter.stream()"]
|
|
Bridge["OCI SSE Async Bridge"]
|
|
Chunker["TextChunker"]
|
|
Queue["Bounded Segment Queue"]
|
|
Synth["VoxCPM TTS Streaming"]
|
|
|
|
Agent --> Bridge
|
|
Bridge --> Chunker
|
|
Chunker --> Queue
|
|
Queue --> Synth
|
|
end
|
|
|
|
OCI["OCI Generative AI"]
|
|
Vox["VoxCPM Service"]
|
|
Output["WebSocket PCM Output"]
|
|
|
|
Input --> Gateway
|
|
Gateway --> UseCase["VoiceSession"]
|
|
UseCase --> STT
|
|
STT --> Transcript
|
|
Transcript --> Agent
|
|
|
|
Agent <--> OCI
|
|
Synth <--> Vox
|
|
Synth --> Gateway
|
|
Gateway --> Output
|
|
```
|
|
|
|
The pipeline uses bounded queues and asynchronous task coordination to support incremental generation, ordered synthesis, and cooperative cancellation.
|
|
|
|
### Lifecycle Ownership
|
|
|
|
AgentVoice separates semantic state from asynchronous execution.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
Listening["LISTENING"]
|
|
Processing["PROCESSING"]
|
|
Speaking["SPEAKING"]
|
|
Completed["COMPLETED"]
|
|
Interrupted["INTERRUPTED"]
|
|
|
|
Listening -->|"STT final"| Processing
|
|
Processing -->|"First TTS segment"| Speaking
|
|
Speaking -->|"Synthesis complete"| Completed
|
|
|
|
Listening --> Interrupted
|
|
Processing --> Interrupted
|
|
Speaking --> Interrupted
|
|
```
|
|
|
|
**Ownership responsibilities:**
|
|
|
|
| Component | Owns |
|
|
|---|---|
|
|
| Audio Gateway | WebSocket connection, protocol messages, provider cleanup coordination |
|
|
| SessionState | Session lifecycle and audio sequence validation |
|
|
| TurnManager | Semantic turn state transitions |
|
|
| VoicePipeline | Turn execution task and Agent/TTS orchestration |
|
|
| OCI streaming bridge | SSE reader, event queue, and stream closure |
|
|
| Provider adapters | Provider-specific requests, connections, and data formats |
|
|
|
|
Cancellation is cooperative. The system requests cancellation and waits for task termination, but termination of blocking external operations depends on the underlying provider and transport behavior.
|
|
|
|
### Architectural Extension Points
|
|
|
|
The abstraction layer allows additional implementations to be introduced through the provider catalog.
|
|
|
|
For example, a future LiteLLM integration could implement the Agent contract without requiring a redesign of the Gateway or the semantic turn lifecycle.
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
Pipeline["VoicePipeline"]
|
|
Contract["AgentAdapter"]
|
|
|
|
OCI["OCIAgentAdapter<br/>Current"]
|
|
Mock["MockAgentAdapter<br/>Testing"]
|
|
Lite["LiteLLMAgentAdapter<br/>Future"]
|
|
|
|
Pipeline --> Contract
|
|
Contract -.-> OCI
|
|
Contract -.-> Mock
|
|
Contract -.-> Lite
|
|
```
|
|
|
|
**Design principle:** transport, orchestration, lifecycle, and provider-specific integrations remain separate concerns. Provider independence is established through contracts and factories, while streaming and cancellation behavior are validated at their respective integration boundaries.
|
|
|
|
## Quickstart
|
|
|
|
### Requirements
|
|
|
|
- Python 3.12 or newer and [uv](https://docs.astral.sh/uv/)
|
|
- Accessible NVIDIA ASR and VoxCPM services
|
|
- Run commands from the repository root (`config/` is resolved relative to the working directory)
|
|
|
|
### Install
|
|
|
|
```bash
|
|
uv sync
|
|
```
|
|
|
|
### Start the gateway
|
|
|
|
With the configured provider services running:
|
|
|
|
```bash
|
|
APP_ENV=development uv run uvicorn \
|
|
voice_assistant.audio_gateway.app.main:app \
|
|
--host 127.0.0.1 --port 8001
|
|
```
|
|
|
|
Check availability:
|
|
|
|
```bash
|
|
curl http://127.0.0.1:8001/ready
|
|
# {"status":"ready"}
|
|
```
|
|
|
|
> `/ready` reports Gateway availability; it does **not** verify NVIDIA ASR or VoxCPM health.
|
|
|
|
## Configuration
|
|
|
|
`APP_ENV` selects `config/<environment>.yaml`; the default is `development`. `config/providers.yaml` maps provider names to Python implementations and capability metadata. YAML data is validated using Pydantic models.
|
|
|
|
| Component | Development provider | Configuration |
|
|
| --- | --- | --- |
|
|
| Speech-to-text | NVIDIA ASR | `127.0.0.1:9000`, `pt-BR`, punctuation enabled |
|
|
| Agent | Mock Agent | In-process; simulated text response |
|
|
| Text-to-speech | VoxCPM | `http://127.0.0.1:8000`, `openbmb/VoxCPM2`, `default` voice |
|
|
|
|
To select another environment:
|
|
|
|
```bash
|
|
APP_ENV=staging uv run uvicorn \
|
|
voice_assistant.audio_gateway.app.main:app \
|
|
--host 127.0.0.1 --port 8001
|
|
```
|
|
|
|
The `staging` and `production` files are present, but deployment readiness for those environments has not been established. When services run in different containers or machines, replace loopback addresses with reachable service endpoints.
|
|
|
|
## WebSocket protocol
|
|
|
|
| Property | Value |
|
|
|---|---|
|
|
| Endpoint | `ws://127.0.0.1:8001/ws/audio` |
|
|
| Protocol version | `1.0` |
|
|
| Input audio | PCM signed 16-bit little-endian, 16 kHz, mono |
|
|
| Output audio | Declared by `session.ready.output_audio`; current VoxCPM: PCM S16LE, 48 kHz, mono |
|
|
| Transport: | JSON control messages and binary PCM frames. |
|
|
|
|
### Typical exchange
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant C as Client
|
|
participant G as Audio Gateway
|
|
participant P as VoicePipeline / Providers
|
|
C->>G: session.start
|
|
G-->>C: session.ready
|
|
C->>G: audio.start
|
|
G-->>C: turn.started
|
|
loop Audio input
|
|
C->>G: audio.chunk (turn_id, sequence_number)
|
|
C->>G: Binary PCM frame
|
|
P-->>G: STT partial events
|
|
G-->>C: stt.partial
|
|
end
|
|
C->>G: audio.stop (turn_id)
|
|
P-->>G: STT final transcript
|
|
G-->>C: stt.final
|
|
P-->>G: Agent response
|
|
G-->>C: agent.response
|
|
P-->>G: TTS PCM chunks
|
|
G-->>C: Binary audio frames
|
|
G-->>C: turn.ended
|
|
```
|
|
|
|
STT events may arrive while input audio is still streaming. Binary audio input is preceded by an `audio.chunk` control message containing `turn_id` and `sequence_number`. The Gateway limits each binary chunk to 64 KiB. Input frames must contain complete PCM samples; rejected frames do not advance the audio sequence. `session.ready.audio` describes input and `session.ready.output_audio` describes output. The browser configures playback from the declared output format. VoxCPM reassembles HTTP fragments into complete PCM frames and rejects a truncated final sample. Its `sample_rate` option declares the expected native output rate; it does not request resampling from the external service.
|
|
|
|
### Interruption and session closure
|
|
|
|
- `interrupt` targets a `turn_id`. For a matching active turn, the Gateway cancels and awaits execution, aborts STT, releases turn state, and then acknowledges with `interrupted`.
|
|
- `session.end` requests cleanup and session closure; successful completion is acknowledged with `session.ended`.
|
|
- Errors are returned as structured `error` events (for example, `STT_START_FAILED` or `TURN_CANCEL_FAILED`).
|
|
|
|
## Lifecycle and interruption
|
|
|
|
The system separates **semantic state** from **asynchronous execution**. `TurnManager` controls listening, processing, speaking, and completion; `VoicePipeline` owns the registered turn task. The Gateway coordinates provider shutdown and WebSocket delivery.
|
|
|
|
Cancellation is cooperative: `task.cancel()` requests termination, and the Gateway awaits task completion before confirming a successful interruption. Startup rollback attempts to cancel any created task and abort a partially initialized STT provider. Failed cleanup is reported rather than represented as a successful retry.
|
|
|
|
## Testing
|
|
|
|
Run the automated suite:
|
|
|
|
```bash
|
|
uv run pytest -q
|
|
```
|
|
|
|
Run Gateway tests or targeted static checks:
|
|
|
|
```bash
|
|
uv run pytest -q tests/test_gateway.py
|
|
uv run ruff check src tests --select F401,F811,F821
|
|
```
|
|
|
|
### End-to-end smoke test
|
|
|
|
With the Gateway, NVIDIA ASR, and VoxCPM services available:
|
|
|
|
```bash
|
|
uv run python tools/voice_pipeline_smoke_test.py
|
|
```
|
|
|
|
The script reads `audios/chafik_normalize.wav` (16 kHz, mono, 16-bit PCM), sends audio over WebSocket, and checks for a final transcript, an Agent response, synthesized audio, and `turn.ended`. It writes raw output to `/tmp/voice_pipeline_response.pcm`.
|
|
|
|
**Last verified checkpoint:** 142 automated tests passing; the full NVIDIA STT → Mock Agent → VoxCPM TTS smoke test completed successfully. These results describe the development environment, not a production SLA.
|
|
|
|
## Project structure
|
|
|
|
```text
|
|
config/ Environment YAML and provider catalog
|
|
src/voice_assistant/
|
|
audio_gateway/app/ WebSocket endpoint and protocol
|
|
application/ VoiceSession and transport-independent events
|
|
domain/ Session state and domain rules
|
|
vad/ VAD port, speech boundaries, and local Silero adapter
|
|
audio/ Canonical audio models
|
|
stt/ STT contract, mock and NVIDIA adapter
|
|
agent/ Agent contract and mock adapter
|
|
tts/ TTS contract, mock and VoxCPM adapter
|
|
config/ Typed configuration and loading
|
|
voice_pipeline.py Turn execution and orchestration
|
|
turn_manager.py Semantic turn state machine
|
|
providers.py Provider factories
|
|
tests/ Automated tests
|
|
tools/ Integration and smoke scripts
|
|
```
|
|
|
|
## Known limitations
|
|
|
|
- **Mock Agent:** responses are simulated; the V1 does not yet include a production conversational agent.
|
|
- **Cooperative cancellation:** a coroutine that ignores cancellation may remain active after a timeout. Logical session closure does not guarantee that all external resources have terminated.
|
|
- **Provider availability:** readiness does not constitute an end-to-end health check.
|
|
- **Development-oriented deployment:** loopback endpoints and a fixed smoke-test WAV are configured for local execution.
|
|
- **Containers:** the repository's Dockerfile has not been implemented or validated for deployment.
|
|
|
|
---
|
|
|
|
**Status:** V1 development baseline · Python 3.12+ · WebSocket protocol 1.0
|
|
|
|
|
|
|
|
|
|
## Backend VAD contract (integration pending)
|
|
|
|
`vad/adapter.py` defines one stateful detector per continuous input stream.
|
|
Its input is the canonical PCM S16LE, mono, 16 kHz audio format. Arbitrary
|
|
complete PCM frames are buffered into model-specific windows by the adapter.
|
|
`vad/models.py` defines `SpeechStarted` and `SpeechEnded`; their sample offsets
|
|
refer to input audio since initialization or reset, with inclusive starts
|
|
and exclusive ends. Detection confirmation may arrive later than its audio
|
|
boundary. Model padding never extends the input timeline.
|
|
|
|
`process()` returns ordered boundaries, `flush()` handles normal end of input,
|
|
`reset()` discards detection state and restarts the timeline, and `aclose()`
|
|
releases resources without emitting new speech events. Calls are serialized.
|
|
These contracts do not start turns, select providers, or carry transport IDs.
|
|
The local Silero implementation is available independently of the running
|
|
conversation. Application integration and browser VAD replacement are subsequent
|
|
tasks; the active conversation flow still uses its existing browser VAD.
|
|
|
|
|
|
### Local Silero VAD adapter
|
|
|
|
`vad/silero.py` implements `VADAdapter` with the official Silero model and
|
|
`VADIterator`. `silero-vad[onnx-cpu]==6.2.3` bundles the model files; model loading
|
|
and inference use CPU locally, without runtime downloads or `torch.hub`.
|
|
The upstream package also requires PyTorch. The uv source configuration selects
|
|
CPU builds on Linux/Windows and the standard PyPI build on macOS.
|
|
|
|
Each adapter owns its model state, PCM buffer, and input timeline. It converts
|
|
PCM S16LE to normalized float samples, buffers 512-sample windows, and maps
|
|
Silero boundaries to the application events. Defaults are threshold 0.5,
|
|
300 ms of silence, 30 ms speech padding, and a 10-second worker wait limit.
|
|
These values are initial settings, to be tuned with real conversational audio.
|
|
|
|
`process()` may return several events for a large chunk. `flush()` pads a final
|
|
partial window internally while keeping event positions within real audio.
|
|
`reset()` discards detection state and restarts the timeline. `aclose()` discards
|
|
pending input and releases model references without emitting speech events.
|
|
|
|
Model loading and inference run outside the event loop. Cancellation or a worker
|
|
timeout requires reset before accepting more audio. Outstanding workers retain
|
|
ownership: reset/close wait for them, with bounded waits and retry on timeout.
|
|
Running native inference is not forcibly terminated by coroutine cancellation.
|
|
|
|
Tests with a controlled iterator cover framing, positions, isolation, and worker
|
|
ownership. `tests/test_silero_vad_integration.py` exercises the installed ONNX
|
|
model with network access blocked; it skips when the Silero dependency is absent.
|
|
It can be run with:
|
|
|
|
```bash
|
|
uv run pytest -q tests/test_silero_vad.py tests/test_silero_vad_integration.py
|
|
```
|
|
|
|
Upstream source and license: [Silero VAD](https://github.com/snakers4/silero-vad)
|
|
([MIT](https://github.com/snakers4/silero-vad/blob/master/LICENSE)).
|
|
|
|
|
|
### VAD configuration by environment
|
|
|
|
The active environment YAML configures VAD using the same `provider`/`options`
|
|
structure as STT, Agent, and TTS. `config/development.yaml` contains:
|
|
|
|
```yaml
|
|
vad:
|
|
provider: silero
|
|
options:
|
|
threshold: 0.5
|
|
min_silence_duration_ms: 300
|
|
speech_pad_ms: 30
|
|
worker_timeout_seconds: 10.0
|
|
```
|
|
|
|
`APP_ENV` selects the environment through the existing configuration loader.
|
|
`create_vad_adapter(config, catalog)` resolves the VAD from `config/providers.yaml`
|
|
and passes the environment options to its constructor. The Silero constructor
|
|
retains defaults for direct use and missing options. A missing `vad` section
|
|
remains compatible with existing configurations; explicitly creating VAD in
|
|
such an environment raises a configuration error. Changing YAML values affects
|
|
new adapter instances after configuration is reloaded; there is no live reload.
|
|
|
|
This factory makes the configuration usable independently. Connecting it to the
|
|
conversation lifecycle remains the next task.
|