feat: integrate OCI agent streaming and web voice client

2026-10-13 19:49:27 +00:00
parent 088473a0c8
commit 19a415daae
19 changed files with 2034 additions and 23 deletions
--- a/README.md
+++ b/README.md
@@ -18,6 +18,7 @@ A modular voice pipeline for streaming audio input, speech transcription, agent
 - [Testing](#testing)
 - [Project structure](#project-structure)
 - [Known limitations](#known-limitations)
+- [Architecture Backbone & Abstraction Boundaries](#architecture-backbone--abstraction-boundaries)

 ## Architecture

@@ -38,6 +39,212 @@ flowchart TB

 The **Audio Gateway** handles transport and protocol messages. **VoicePipeline** coordinates processing and task ownership. **SessionState** tracks session and audio sequence; **TurnManager** enforces turn transitions. A new STT adapter is created for each turn, while Agent and TTS adapters are created per session.

+## Architecture Backbone & Abstraction Boundaries
+
+AgentVoice follows a **provider-independent architecture** built around explicit contracts, lifecycle ownership, and asynchronous streaming.
+
+The architecture separates five concerns:
+
+- **Transport:** WebSocket communication and protocol serialization.
+- **Orchestration:** Coordination of STT, Agent, and TTS execution.
+- **Lifecycle:** Session state, turn transitions, interruption, and task ownership.
+- **Provider contracts:** Abstract interfaces defining component behavior.
+- **Provider implementations:** Concrete integrations with NVIDIA, OCI Generative AI, and VoxCPM.
+
+### Architectural Backbone
+
+```mermaid
+flowchart TB
+    Client["Web Client / Audio Client"]
+    Gateway["Audio Gateway<br/>FastAPI · WebSocket"]
+    Pipeline["VoicePipeline<br/>Orchestration & Streaming"]
+    Session["SessionState<br/>Session Lifecycle"]
+    Turns["TurnManager<br/>Semantic Turn Lifecycle"]
+
+    subgraph Contracts["Provider Abstraction Layer"]
+        direction LR
+        STTA["STTAdapter<br/>Abstract Contract"]
+        AgentA["AgentAdapter<br/>Abstract Contract"]
+        TTSA["TTSAdapter<br/>Abstract Contract"]
+    end
+
+    subgraph Implementations["Provider Implementation Layer"]
+        direction LR
+        Nvidia["NvidiaSTTAdapter"]
+        OCI["OCIAgentAdapter"]
+        Mock["MockAgentAdapter"]
+        Vox["VoxCPMTTSAdapter"]
+    end
+
+    subgraph External["External AI Services"]
+        ASR["NVIDIA ASR"]
+        LLM["OCI Generative AI"]
+        TTS["VoxCPM"]
+    end
+
+    Client <-->|"JSON Events + Binary PCM"| Gateway
+    Gateway --> Pipeline
+
+    Pipeline --- Session
+    Pipeline --- Turns
+
+    Pipeline --> STTA
+    Pipeline --> AgentA
+    Pipeline --> TTSA
+
+    STTA -.->|implements| Nvidia
+    AgentA -.->|implements| OCI
+    AgentA -.->|implements| Mock
+    TTSA -.->|implements| Vox
+
+    Nvidia --> ASR
+    OCI --> LLM
+    Vox --> TTS
+```
+
+The `VoicePipeline` depends on provider contracts rather than concrete implementations. Provider selection is performed through configuration and dynamic factories.
+
+This allows implementations to evolve without requiring the Audio Gateway to understand provider-specific APIs.
+
+### Provider Contracts
+
+| Abstraction | Contract | Implementations | Responsibility |
+|---|---|---|---|
+| Speech-to-text | `STTAdapter` | `NvidiaSTTAdapter`, `MockSTTAdapter` | Audio ingestion and transcript events |
+| Agent | `AgentAdapter` | `OCIAgentAdapter`, `MockAgentAdapter` | Text processing and conversational responses |
+| Text-to-speech | `TTSAdapter` | `VoxCPMTTSAdapter`, `MockTTSAdapter` | Speech synthesis and audio streaming |
+
+The `AgentAdapter` defines the common text-processing interface. OCI-specific incremental generation is currently exposed through an additional `stream()` method, with streaming capability detected by the orchestration layer.
+
+### Runtime Composition
+
+Provider implementations are selected through environment configuration and resolved by the provider factory.
+
+```mermaid
+flowchart LR
+    Config["Environment YAML"]
+    Catalog["Provider Catalog"]
+    Factory["Provider Factories"]
+    Contracts["Adapter Contracts"]
+    Runtime["VoicePipeline"]
+
+    Config --> Factory
+    Catalog --> Factory
+    Factory -->|Validate implementation| Contracts
+    Factory -->|Instantiate providers| Runtime
+```
+
+The primary configuration components are:
+
+| Component | Location |
+|---|---|
+| Environment selection | `config/loader.py` |
+| Typed configuration | `config/models.py` |
+| Provider catalog | `config/providers.yaml` |
+| Dynamic provider loading | `providers.py` |
+| Session lifecycle | `audio_gateway/app/session.py` |
+| Semantic turn lifecycle | `turn_manager.py` |
+| Execution orchestration | `voice_pipeline.py` |
+
+### Streaming Execution Backbone
+
+The streaming pipeline allows OCI Generative AI to continue producing text while VoxCPM synthesizes previously completed text segments.
+
+```mermaid
+flowchart TB
+    Input["Microphone / PCM Input"]
+    Gateway["Audio Gateway"]
+    STT["NVIDIA STT"]
+    Transcript["Final Transcript"]
+
+    subgraph Pipeline["VoicePipeline · Concurrent Execution"]
+        direction TB
+        Agent["OCIAgentAdapter.stream()"]
+        Bridge["OCI SSE Async Bridge"]
+        Chunker["TextChunker"]
+        Queue["Bounded Segment Queue"]
+        Synth["VoxCPM TTS Streaming"]
+
+        Agent --> Bridge
+        Bridge --> Chunker
+        Chunker --> Queue
+        Queue --> Synth
+    end
+
+    OCI["OCI Generative AI"]
+    Vox["VoxCPM Service"]
+    Output["WebSocket PCM Output"]
+
+    Input --> Gateway
+    Gateway --> STT
+    STT --> Transcript
+    Transcript --> Agent
+
+    Agent <--> OCI
+    Synth <--> Vox
+    Synth --> Gateway
+    Gateway --> Output
+```
+
+The pipeline uses bounded queues and asynchronous task coordination to support incremental generation, ordered synthesis, and cooperative cancellation.
+
+### Lifecycle Ownership
+
+AgentVoice separates semantic state from asynchronous execution.
+
+```mermaid
+flowchart LR
+    Listening["LISTENING"]
+    Processing["PROCESSING"]
+    Speaking["SPEAKING"]
+    Completed["COMPLETED"]
+    Interrupted["INTERRUPTED"]
+
+    Listening -->|"STT final"| Processing
+    Processing -->|"First TTS segment"| Speaking
+    Speaking -->|"Synthesis complete"| Completed
+
+    Listening --> Interrupted
+    Processing --> Interrupted
+    Speaking --> Interrupted
+```
+
+**Ownership responsibilities:**
+
+| Component | Owns |
+|---|---|
+| Audio Gateway | WebSocket connection, protocol messages, provider cleanup coordination |
+| SessionState | Session lifecycle and audio sequence validation |
+| TurnManager | Semantic turn state transitions |
+| VoicePipeline | Turn execution task and Agent/TTS orchestration |
+| OCI streaming bridge | SSE reader, event queue, and stream closure |
+| Provider adapters | Provider-specific requests, connections, and data formats |
+
+Cancellation is cooperative. The system requests cancellation and waits for task termination, but termination of blocking external operations depends on the underlying provider and transport behavior.
+
+### Architectural Extension Points
+
+The abstraction layer allows additional implementations to be introduced through the provider catalog.
+
+For example, a future LiteLLM integration could implement the Agent contract without requiring a redesign of the Gateway or the semantic turn lifecycle.
+
+```mermaid
+flowchart TB
+    Pipeline["VoicePipeline"]
+    Contract["AgentAdapter"]
+
+    OCI["OCIAgentAdapter<br/>Current"]
+    Mock["MockAgentAdapter<br/>Testing"]
+    Lite["LiteLLMAgentAdapter<br/>Future"]
+
+    Pipeline --> Contract
+    Contract -.-> OCI
+    Contract -.-> Mock
+    Contract -.-> Lite
+```
+
+**Design principle:** transport, orchestration, lifecycle, and provider-specific integrations remain separate concerns. Provider independence is established through contracts and factories, while streaming and cancellation behavior are validated at their respective integration boundaries.
+
 ## Quickstart

 ### Requirements
@@ -197,3 +404,5 @@ tools/                          Integration and smoke scripts
 ---

 **Status:** V1 development baseline · Python 3.12+ · WebSocket protocol 1.0
+
+