# Voice Assistant

**Realtime speech processing over WebSocket — NVIDIA ASR · Agent orchestration · VoxCPM TTS**

![Python](https://img.shields.io/badge/Python-3.12%2B-3776AB?logo=python&logoColor=white) ![Protocol](https://img.shields.io/badge/WebSocket-Protocol%201.0-39485C) ![Release](https://img.shields.io/badge/Release-V1%20development-566575)

A modular voice pipeline for streaming audio input, speech transcription, agent responses, and synthesized speech output. Built with FastAPI and asynchronous Python, with explicit session lifecycle, turn ownership, interruption, and provider cleanup.

> **V1 scope:** NVIDIA STT and VoxCPM TTS are real providers; the Agent is a mock implementation. Production readiness and container deployment have not yet been validated.

## Contents

- [Architecture](#architecture)
- [Quickstart](#quickstart)
- [Configuration](#configuration)
- [WebSocket protocol](#websocket-protocol)
- [Lifecycle and interruption](#lifecycle-and-interruption)
- [Testing](#testing)
- [Project structure](#project-structure)
- [Known limitations](#known-limitations)
- [Architecture Backbone & Abstraction Boundaries](#architecture-backbone--abstraction-boundaries)

## Architecture

```mermaid
flowchart TB
    C[Audio Client] <-->|WebSocket /ws/audio · JSON + binary PCM| G[Audio Gateway · FastAPI]
    G --> P[VoicePipeline]
    P --- S[SessionState · session and audio sequence]
    P --- T[TurnManager · semantic turn state]
    G --> STT[NVIDIA ASR · STT per turn]
    STT -->|Transcript events| P
    P --> A[Mock Agent · per session]
    A -->|Response text| P
    P --> V[VoxCPM TTS · per session]
    V -->|PCM chunks| G
    G -->|Streamed audio| C
```

The **Audio Gateway** handles transport and protocol messages. **VoicePipeline** coordinates processing and task ownership. **SessionState** tracks session and audio sequence; **TurnManager** enforces turn transitions. A new STT adapter is created for each turn, while Agent and TTS adapters are created per session.

## Architecture Backbone & Abstraction Boundaries

AgentVoice follows a **provider-independent architecture** built around explicit contracts, lifecycle ownership, and asynchronous streaming.

The architecture separates five concerns:

- **Transport:** WebSocket communication and protocol serialization.
- **Orchestration:** Coordination of STT, Agent, and TTS execution.
- **Lifecycle:** Session state, turn transitions, interruption, and task ownership.
- **Provider contracts:** Abstract interfaces defining component behavior.
- **Provider implementations:** Concrete integrations with NVIDIA, OCI Generative AI, and VoxCPM.

### Architectural Backbone

```mermaid
flowchart TB
    Client["Web Client / Audio Client"]
    Gateway["Audio Gateway<br/>FastAPI · WebSocket"]
    Pipeline["VoicePipeline<br/>Orchestration & Streaming"]
    Session["SessionState<br/>Session Lifecycle"]
    Turns["TurnManager<br/>Semantic Turn Lifecycle"]

    subgraph Contracts["Provider Abstraction Layer"]
        direction LR
        STTA["STTAdapter<br/>Abstract Contract"]
        AgentA["AgentAdapter<br/>Abstract Contract"]
        TTSA["TTSAdapter<br/>Abstract Contract"]
    end

    subgraph Implementations["Provider Implementation Layer"]
        direction LR
        Nvidia["NvidiaSTTAdapter"]
        OCI["OCIAgentAdapter"]
        Mock["MockAgentAdapter"]
        Vox["VoxCPMTTSAdapter"]
    end

    subgraph External["External AI Services"]
        ASR["NVIDIA ASR"]
        LLM["OCI Generative AI"]
        TTS["VoxCPM"]
    end

    Client <-->|"JSON Events + Binary PCM"| Gateway
    Gateway --> Pipeline

    Pipeline --- Session
    Pipeline --- Turns

    Pipeline --> STTA
    Pipeline --> AgentA
    Pipeline --> TTSA

    STTA -.->|implements| Nvidia
    AgentA -.->|implements| OCI
    AgentA -.->|implements| Mock
    TTSA -.->|implements| Vox

    Nvidia --> ASR
    OCI --> LLM
    Vox --> TTS
```

The `VoicePipeline` depends on provider contracts rather than concrete implementations. Provider selection is performed through configuration and dynamic factories.

This allows implementations to evolve without requiring the Audio Gateway to understand provider-specific APIs.

### Provider Contracts

| Abstraction | Contract | Implementations | Responsibility |
|---|---|---|---|
| Speech-to-text | `STTAdapter` | `NvidiaSTTAdapter`, `MockSTTAdapter` | Audio ingestion and transcript events |
| Agent | `AgentAdapter` | `OCIAgentAdapter`, `MockAgentAdapter` | Text processing and conversational responses |
| Text-to-speech | `TTSAdapter` | `VoxCPMTTSAdapter`, `MockTTSAdapter` | Speech synthesis and audio streaming |

The `AgentAdapter` defines the common text-processing interface. OCI-specific incremental generation is currently exposed through an additional `stream()` method, with streaming capability detected by the orchestration layer.

### Runtime Composition

Provider implementations are selected through environment configuration and resolved by the provider factory.

```mermaid
flowchart LR
    Config["Environment YAML"]
    Catalog["Provider Catalog"]
    Factory["Provider Factories"]
    Contracts["Adapter Contracts"]
    Runtime["VoicePipeline"]

    Config --> Factory
    Catalog --> Factory
    Factory -->|Validate implementation| Contracts
    Factory -->|Instantiate providers| Runtime
```

The primary configuration components are:

| Component | Location |
|---|---|
| Environment selection | `config/loader.py` |
| Typed configuration | `config/models.py` |
| Provider catalog | `config/providers.yaml` |
| Dynamic provider loading | `providers.py` |
| Session lifecycle | `domain/session.py` |
| Semantic turn lifecycle | `turn_manager.py` |
| Execution orchestration | `voice_pipeline.py` |

### Streaming Execution Backbone

The streaming pipeline allows OCI Generative AI to continue producing text while VoxCPM synthesizes previously completed text segments.

```mermaid
flowchart TB
    Input["Microphone / PCM Input"]
    Gateway["Audio Gateway"]
    STT["NVIDIA STT"]
    Transcript["Final Transcript"]

    subgraph Pipeline["VoicePipeline · Concurrent Execution"]
        direction TB
        Agent["OCIAgentAdapter.stream()"]
        Bridge["OCI SSE Async Bridge"]
        Chunker["TextChunker"]
        Queue["Bounded Segment Queue"]
        Synth["VoxCPM TTS Streaming"]

        Agent --> Bridge
        Bridge --> Chunker
        Chunker --> Queue
        Queue --> Synth
    end

    OCI["OCI Generative AI"]
    Vox["VoxCPM Service"]
    Output["WebSocket PCM Output"]

    Input --> Gateway
    Gateway --> STT
    STT --> Transcript
    Transcript --> Agent

    Agent <--> OCI
    Synth <--> Vox
    Synth --> Gateway
    Gateway --> Output
```

The pipeline uses bounded queues and asynchronous task coordination to support incremental generation, ordered synthesis, and cooperative cancellation.

### Lifecycle Ownership

AgentVoice separates semantic state from asynchronous execution.

```mermaid
flowchart LR
    Listening["LISTENING"]
    Processing["PROCESSING"]
    Speaking["SPEAKING"]
    Completed["COMPLETED"]
    Interrupted["INTERRUPTED"]

    Listening -->|"STT final"| Processing
    Processing -->|"First TTS segment"| Speaking
    Speaking -->|"Synthesis complete"| Completed

    Listening --> Interrupted
    Processing --> Interrupted
    Speaking --> Interrupted
```

**Ownership responsibilities:**

| Component | Owns |
|---|---|
| Audio Gateway | WebSocket connection, protocol messages, provider cleanup coordination |
| SessionState | Session lifecycle and audio sequence validation |
| TurnManager | Semantic turn state transitions |
| VoicePipeline | Turn execution task and Agent/TTS orchestration |
| OCI streaming bridge | SSE reader, event queue, and stream closure |
| Provider adapters | Provider-specific requests, connections, and data formats |

Cancellation is cooperative. The system requests cancellation and waits for task termination, but termination of blocking external operations depends on the underlying provider and transport behavior.

### Architectural Extension Points

The abstraction layer allows additional implementations to be introduced through the provider catalog.

For example, a future LiteLLM integration could implement the Agent contract without requiring a redesign of the Gateway or the semantic turn lifecycle.

```mermaid
flowchart TB
    Pipeline["VoicePipeline"]
    Contract["AgentAdapter"]

    OCI["OCIAgentAdapter<br/>Current"]
    Mock["MockAgentAdapter<br/>Testing"]
    Lite["LiteLLMAgentAdapter<br/>Future"]

    Pipeline --> Contract
    Contract -.-> OCI
    Contract -.-> Mock
    Contract -.-> Lite
```

**Design principle:** transport, orchestration, lifecycle, and provider-specific integrations remain separate concerns. Provider independence is established through contracts and factories, while streaming and cancellation behavior are validated at their respective integration boundaries.

## Quickstart

### Requirements

- Python 3.12 or newer and [uv](https://docs.astral.sh/uv/)
- Accessible NVIDIA ASR and VoxCPM services
- Run commands from the repository root (`config/` is resolved relative to the working directory)

### Install

```bash
uv sync
```

### Start the gateway

With the configured provider services running:

```bash
APP_ENV=development uv run uvicorn \
  voice_assistant.audio_gateway.app.main:app \
  --host 127.0.0.1 --port 8001
```

Check availability:

```bash
curl http://127.0.0.1:8001/ready
# {"status":"ready"}
```

> `/ready` reports Gateway availability; it does **not** verify NVIDIA ASR or VoxCPM health.

## Configuration

`APP_ENV` selects `config/<environment>.yaml`; the default is `development`. `config/providers.yaml` maps provider names to Python implementations and capability metadata. YAML data is validated using Pydantic models.

| Component | Development provider | Configuration |
| --- | --- | --- |
| Speech-to-text | NVIDIA ASR | `127.0.0.1:9000`, `pt-BR`, punctuation enabled |
| Agent | Mock Agent | In-process; simulated text response |
| Text-to-speech | VoxCPM | `http://127.0.0.1:8000`, `openbmb/VoxCPM2`, `default` voice |

To select another environment:

```bash
APP_ENV=staging uv run uvicorn \
  voice_assistant.audio_gateway.app.main:app \
  --host 127.0.0.1 --port 8001
```

The `staging` and `production` files are present, but deployment readiness for those environments has not been established. When services run in different containers or machines, replace loopback addresses with reachable service endpoints.

## WebSocket protocol

| Property | Value |
|---|---|
| Endpoint | `ws://127.0.0.1:8001/ws/audio` |
| Protocol version | `1.0` |
| Input audio | PCM signed 16-bit little-endian, 16 kHz, mono |
| Transport:  | JSON control messages and binary PCM frames. |

### Typical exchange

```mermaid
sequenceDiagram
    participant C as Client
    participant G as Audio Gateway
    participant P as VoicePipeline / Providers
    C->>G: session.start
    G-->>C: session.ready
    C->>G: audio.start
    G-->>C: turn.started
    loop Audio input
        C->>G: audio.chunk (turn_id, sequence_number)
        C->>G: Binary PCM frame
        P-->>G: STT partial events
        G-->>C: stt.partial
    end
    C->>G: audio.stop (turn_id)
    P-->>G: STT final transcript
    G-->>C: stt.final
    P-->>G: Agent response
    G-->>C: agent.response
    P-->>G: TTS PCM chunks
    G-->>C: Binary audio frames
    G-->>C: turn.ended
```

STT events may arrive while input audio is still streaming. Binary audio input is preceded by an `audio.chunk` control message containing `turn_id` and `sequence_number`. The Gateway limits each binary chunk to 64 KiB.

### Interruption and session closure

- `interrupt` targets a `turn_id`. For a matching active turn, the Gateway cancels and awaits execution, aborts STT, releases turn state, and then acknowledges with `interrupted`.
- `session.end` requests cleanup and session closure; successful completion is acknowledged with `session.ended`.
- Errors are returned as structured `error` events (for example, `STT_START_FAILED` or `TURN_CANCEL_FAILED`).

## Lifecycle and interruption

The system separates **semantic state** from **asynchronous execution**. `TurnManager` controls listening, processing, speaking, and completion; `VoicePipeline` owns the registered turn task. The Gateway coordinates provider shutdown and WebSocket delivery.

Cancellation is cooperative: `task.cancel()` requests termination, and the Gateway awaits task completion before confirming a successful interruption. Startup rollback attempts to cancel any created task and abort a partially initialized STT provider. Failed cleanup is reported rather than represented as a successful retry.

## Testing

Run the automated suite:

```bash
uv run pytest -q
```

Run Gateway tests or targeted static checks:

```bash
uv run pytest -q tests/test_gateway.py
uv run ruff check src tests --select F401,F811,F821
```

### End-to-end smoke test

With the Gateway, NVIDIA ASR, and VoxCPM services available:

```bash
uv run python tools/voice_pipeline_smoke_test.py
```

The script reads `audios/chafik_normalize.wav` (16 kHz, mono, 16-bit PCM), sends audio over WebSocket, and checks for a final transcript, an Agent response, synthesized audio, and `turn.ended`. It writes raw output to `/tmp/voice_pipeline_response.pcm`.

**Last verified checkpoint:** 142 automated tests passing; the full NVIDIA STT → Mock Agent → VoxCPM TTS smoke test completed successfully. These results describe the development environment, not a production SLA.

## Project structure

```text
config/                         Environment YAML and provider catalog
src/voice_assistant/
  audio_gateway/app/            WebSocket endpoint and protocol
  domain/                       Session state and domain rules
  audio/                        Canonical audio models
  stt/                          STT contract, mock and NVIDIA adapter
  agent/                        Agent contract and mock adapter
  tts/                          TTS contract, mock and VoxCPM adapter
  config/                       Typed configuration and loading
  voice_pipeline.py             Turn execution and orchestration
  turn_manager.py               Semantic turn state machine
  providers.py                  Provider factories
tests/                          Automated tests
tools/                          Integration and smoke scripts
```

## Known limitations

- **Mock Agent:** responses are simulated; the V1 does not yet include a production conversational agent.
- **Cooperative cancellation:** a coroutine that ignores cancellation may remain active after a timeout. Logical session closure does not guarantee that all external resources have terminated.
- **Provider availability:** readiness does not constitute an end-to-end health check.
- **Development-oriented deployment:** loopback endpoints and a fixed smoke-test WAV are configured for local execution.
- **Containers:** the repository's Dockerfile has not been implemented or validated for deployment.

---

**Status:** V1 development baseline · Python 3.12+ · WebSocket protocol 1.0