200 lines
7.9 KiB
Markdown
200 lines
7.9 KiB
Markdown
# Voice Assistant
|
|
|
|
**Realtime speech processing over WebSocket — NVIDIA ASR · Agent orchestration · VoxCPM TTS**
|
|
|
|
  
|
|
|
|
A modular voice pipeline for streaming audio input, speech transcription, agent responses, and synthesized speech output. Built with FastAPI and asynchronous Python, with explicit session lifecycle, turn ownership, interruption, and provider cleanup.
|
|
|
|
> **V1 scope:** NVIDIA STT and VoxCPM TTS are real providers; the Agent is a mock implementation. Production readiness and container deployment have not yet been validated.
|
|
|
|
## Contents
|
|
|
|
- [Architecture](#architecture)
|
|
- [Quickstart](#quickstart)
|
|
- [Configuration](#configuration)
|
|
- [WebSocket protocol](#websocket-protocol)
|
|
- [Lifecycle and interruption](#lifecycle-and-interruption)
|
|
- [Testing](#testing)
|
|
- [Project structure](#project-structure)
|
|
- [Known limitations](#known-limitations)
|
|
|
|
## Architecture
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
C[Audio Client] <-->|WebSocket /ws/audio · JSON + binary PCM| G[Audio Gateway · FastAPI]
|
|
G --> P[VoicePipeline]
|
|
P --- S[SessionState · session and audio sequence]
|
|
P --- T[TurnManager · semantic turn state]
|
|
G --> STT[NVIDIA ASR · STT per turn]
|
|
STT -->|Transcript events| P
|
|
P --> A[Mock Agent · per session]
|
|
A -->|Response text| P
|
|
P --> V[VoxCPM TTS · per session]
|
|
V -->|PCM chunks| G
|
|
G -->|Streamed audio| C
|
|
```
|
|
|
|
The **Audio Gateway** handles transport and protocol messages. **VoicePipeline** coordinates processing and task ownership. **SessionState** tracks session and audio sequence; **TurnManager** enforces turn transitions. A new STT adapter is created for each turn, while Agent and TTS adapters are created per session.
|
|
|
|
## Quickstart
|
|
|
|
### Requirements
|
|
|
|
- Python 3.12 or newer and [uv](https://docs.astral.sh/uv/)
|
|
- Accessible NVIDIA ASR and VoxCPM services
|
|
- Run commands from the repository root (`config/` is resolved relative to the working directory)
|
|
|
|
### Install
|
|
|
|
```bash
|
|
uv sync
|
|
```
|
|
|
|
### Start the gateway
|
|
|
|
With the configured provider services running:
|
|
|
|
```bash
|
|
APP_ENV=development uv run uvicorn \
|
|
voice_assistant.audio_gateway.app.main:app \
|
|
--host 127.0.0.1 --port 8001
|
|
```
|
|
|
|
Check availability:
|
|
|
|
```bash
|
|
curl http://127.0.0.1:8001/ready
|
|
# {"status":"ready"}
|
|
```
|
|
|
|
> `/ready` reports Gateway availability; it does **not** verify NVIDIA ASR or VoxCPM health.
|
|
|
|
## Configuration
|
|
|
|
`APP_ENV` selects `config/<environment>.yaml`; the default is `development`. `config/providers.yaml` maps provider names to Python implementations and capability metadata. YAML data is validated using Pydantic models.
|
|
|
|
| Component | Development provider | Configuration |
|
|
| --- | --- | --- |
|
|
| Speech-to-text | NVIDIA ASR | `127.0.0.1:9000`, `pt-BR`, punctuation enabled |
|
|
| Agent | Mock Agent | In-process; simulated text response |
|
|
| Text-to-speech | VoxCPM | `http://127.0.0.1:8000`, `openbmb/VoxCPM2`, `default` voice |
|
|
|
|
To select another environment:
|
|
|
|
```bash
|
|
APP_ENV=staging uv run uvicorn \
|
|
voice_assistant.audio_gateway.app.main:app \
|
|
--host 127.0.0.1 --port 8001
|
|
```
|
|
|
|
The `staging` and `production` files are present, but deployment readiness for those environments has not been established. When services run in different containers or machines, replace loopback addresses with reachable service endpoints.
|
|
|
|
## WebSocket protocol
|
|
|
|
| Property | Value |
|
|
|---|---|
|
|
| Endpoint | `ws://127.0.0.1:8001/ws/audio` |
|
|
| Protocol version | `1.0` |
|
|
| Input audio | PCM signed 16-bit little-endian, 16 kHz, mono |
|
|
| Transport: | JSON control messages and binary PCM frames. |
|
|
|
|
### Typical exchange
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant C as Client
|
|
participant G as Audio Gateway
|
|
participant P as VoicePipeline / Providers
|
|
C->>G: session.start
|
|
G-->>C: session.ready
|
|
C->>G: audio.start
|
|
G-->>C: turn.started
|
|
loop Audio input
|
|
C->>G: audio.chunk (turn_id, sequence_number)
|
|
C->>G: Binary PCM frame
|
|
P-->>G: STT partial events
|
|
G-->>C: stt.partial
|
|
end
|
|
C->>G: audio.stop (turn_id)
|
|
P-->>G: STT final transcript
|
|
G-->>C: stt.final
|
|
P-->>G: Agent response
|
|
G-->>C: agent.response
|
|
P-->>G: TTS PCM chunks
|
|
G-->>C: Binary audio frames
|
|
G-->>C: turn.ended
|
|
```
|
|
|
|
STT events may arrive while input audio is still streaming. Binary audio input is preceded by an `audio.chunk` control message containing `turn_id` and `sequence_number`. The Gateway limits each binary chunk to 64 KiB.
|
|
|
|
### Interruption and session closure
|
|
|
|
- `interrupt` targets a `turn_id`. For a matching active turn, the Gateway cancels and awaits execution, aborts STT, releases turn state, and then acknowledges with `interrupted`.
|
|
- `session.end` requests cleanup and session closure; successful completion is acknowledged with `session.ended`.
|
|
- Errors are returned as structured `error` events (for example, `STT_START_FAILED` or `TURN_CANCEL_FAILED`).
|
|
|
|
## Lifecycle and interruption
|
|
|
|
The system separates **semantic state** from **asynchronous execution**. `TurnManager` controls listening, processing, speaking, and completion; `VoicePipeline` owns the registered turn task. The Gateway coordinates provider shutdown and WebSocket delivery.
|
|
|
|
Cancellation is cooperative: `task.cancel()` requests termination, and the Gateway awaits task completion before confirming a successful interruption. Startup rollback attempts to cancel any created task and abort a partially initialized STT provider. Failed cleanup is reported rather than represented as a successful retry.
|
|
|
|
## Testing
|
|
|
|
Run the automated suite:
|
|
|
|
```bash
|
|
uv run pytest -q
|
|
```
|
|
|
|
Run Gateway tests or targeted static checks:
|
|
|
|
```bash
|
|
uv run pytest -q tests/test_gateway.py
|
|
uv run ruff check src tests --select F401,F811,F821
|
|
```
|
|
|
|
### End-to-end smoke test
|
|
|
|
With the Gateway, NVIDIA ASR, and VoxCPM services available:
|
|
|
|
```bash
|
|
uv run python tools/voice_pipeline_smoke_test.py
|
|
```
|
|
|
|
The script reads `audios/chafik_normalize.wav` (16 kHz, mono, 16-bit PCM), sends audio over WebSocket, and checks for a final transcript, an Agent response, synthesized audio, and `turn.ended`. It writes raw output to `/tmp/voice_pipeline_response.pcm`.
|
|
|
|
**Last verified checkpoint:** 142 automated tests passing; the full NVIDIA STT → Mock Agent → VoxCPM TTS smoke test completed successfully. These results describe the development environment, not a production SLA.
|
|
|
|
## Project structure
|
|
|
|
```text
|
|
config/ Environment YAML and provider catalog
|
|
src/voice_assistant/
|
|
audio_gateway/app/ WebSocket endpoint, protocol, session state
|
|
audio/ Canonical audio models
|
|
stt/ STT contract, mock and NVIDIA adapter
|
|
agent/ Agent contract and mock adapter
|
|
tts/ TTS contract, mock and VoxCPM adapter
|
|
config/ Typed configuration and loading
|
|
voice_pipeline.py Turn execution and orchestration
|
|
turn_manager.py Semantic turn state machine
|
|
providers.py Provider factories
|
|
tests/ Automated tests
|
|
tools/ Integration and smoke scripts
|
|
```
|
|
|
|
## Known limitations
|
|
|
|
- **Mock Agent:** responses are simulated; the V1 does not yet include a production conversational agent.
|
|
- **Cooperative cancellation:** a coroutine that ignores cancellation may remain active after a timeout. Logical session closure does not guarantee that all external resources have terminated.
|
|
- **Provider availability:** readiness does not constitute an end-to-end health check.
|
|
- **Development-oriented deployment:** loopback endpoints and a fixed smoke-test WAV are configured for local execution.
|
|
- **Containers:** the repository's Dockerfile has not been implemented or validated for deployment.
|
|
|
|
---
|
|
|
|
**Status:** V1 development baseline · Python 3.12+ · WebSocket protocol 1.0
|