From 6f24a02ccce317640c46e5c9ccdba9ce29712f65 Mon Sep 17 00:00:00 2001 From: alex alves Date: Tue, 3 Nov 2026 19:38:46 -0300 Subject: [PATCH] =?UTF-8?q?docs:=20atualiza=20execu=C3=A7=C3=A3o=20local,?= =?UTF-8?q?=20configura=C3=A7=C3=A3o=20e=20arquitetura=20de=20voz?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 99 +++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 89 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 80ab69a..7ca06e3 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,8 @@ flowchart TB U --> P[VoicePipeline] P --- S[SessionState · session and audio sequence] P --- T[TurnManager · semantic turn state] - G --> STT[NVIDIA ASR · STT per turn] + U --> STT[NVIDIA ASR · STT per turn] + U --> VAD[Silero VAD · local CPU · per continuous session] STT -->|Transcript events| P P --> A[Mock Agent · per session] A -->|Response text| P @@ -219,9 +220,10 @@ flowchart LR | Component | Owns | |---|---| -| Audio Gateway | WebSocket connection, protocol messages, provider cleanup coordination | +| Audio Gateway | WebSocket connection, protocol validation, session composition, event delivery, and shielded application cleanup | | SessionState | Session lifecycle and audio sequence validation | | TurnManager | Semantic turn state transitions | +| VoiceSession | Input history, VAD boundaries, STT lifecycle, finalization, interruption, and provider cleanup | | VoicePipeline | Turn execution task and Agent/TTS orchestration | | OCI streaming bridge | SSE reader, event queue, and stream closure | | Provider adapters | Provider-specific requests, connections, and data formats | @@ -275,6 +277,24 @@ APP_ENV=development uv run uvicorn \ --host 127.0.0.1 --port 8001 ``` +Open `http://127.0.0.1:8001/ui`, allow microphone access, and click **Iniciar conversa**. +The client sends continuous audio; backend Silero detects turns and new speech +interrupts an active response. Audio capture continues during playback. + +Transcripts currently appear in the browser developer console as `[STT PARTIAL]` +and `[STT FINAL]`; Agent text appears as `[AGENT RESPONSE]` or `[AGENT DELTA]`. +On Firefox for macOS, open the console with **Command + Option + K**. +The page displays conversation status and plays synthesized audio. + +The Gateway and VAD run locally without LiveKit or a LiveKit Cloud account. +STT and TTS are separate services and may run remotely; the loopback addresses +in development must point to local services or active port forwards. Audio is +sent to those services, so the full processing path is not offline. Model and +Python dependencies must be installed before offline VAD execution. + +If port 8001 is occupied, choose another port (for example, 8002) and use that +same port in the browser, health checks, and WebSocket tools. + Check availability: ```bash @@ -293,6 +313,7 @@ curl http://127.0.0.1:8001/ready | Speech-to-text | NVIDIA ASR | `127.0.0.1:9000`, `pt-BR`, punctuation enabled | | Agent | Mock Agent | In-process; simulated text response | | Text-to-speech | VoxCPM | `http://127.0.0.1:8000`, `openbmb/VoxCPM2`, `default` voice | +| Voice activity | Silero ONNX | Local CPU; threshold 0.6 and 800 ms silence | To select another environment: @@ -302,7 +323,15 @@ APP_ENV=staging uv run uvicorn \ --host 127.0.0.1 --port 8001 ``` -The `staging` and `production` files are present, but deployment readiness for those environments has not been established. When services run in different containers or machines, replace loopback addresses with reachable service endpoints. +The `staging` and `production` files are currently empty placeholders. Populate +the selected file before using it; the loader rejects an empty YAML. Start from +`development.yaml`, then configure reachable provider endpoints and VAD/input +settings for that environment. Deployment readiness has not been established. + +TTS voice selection is `tts.options.voice` in the selected environment YAML. +The service determines which voice identifiers are supported. Configuration is +loaded at Gateway startup: restart the process and open a new conversation after +changing YAML; refreshing the browser alone does not reload server configuration. ## WebSocket protocol @@ -344,7 +373,8 @@ STT events may arrive while input audio is still streaming. In server detection ### Interruption and session closure -- `interrupt` targets a `turn_id`. For a matching active turn, the Gateway cancels and awaits execution, aborts STT, releases turn state, and then acknowledges with `interrupted`. +- In server detection mode, `SpeechStarted` makes `VoiceSession` interrupt an active turn before opening a new one. The Gateway maps application events to `interrupted` and `turn.started`; the client stops obsolete playback. +- Explicit `interrupt` targets a `turn_id`. For a matching active turn, `VoiceSession` cancels and awaits execution, aborts STT, and releases turn state. The Gateway then delivers the acknowledgement. - `session.end` requests cleanup and session closure; successful completion is acknowledged with `session.ended`. - Errors are returned as structured `error` events (for example, `STT_START_FAILED` or `TURN_CANCEL_FAILED`). @@ -352,7 +382,7 @@ STT events may arrive while input audio is still streaming. In server detection The system separates **semantic state** from **asynchronous execution**. `TurnManager` controls listening, processing, speaking, and completion; `VoicePipeline` owns the registered turn task. VoiceSession coordinates provider shutdown; the Gateway owns WebSocket delivery and shields cleanup from transport cancellation. -Cancellation is cooperative: `task.cancel()` requests termination, and the Gateway awaits task completion before confirming a successful interruption. Startup rollback attempts to cancel any created task and abort a partially initialized STT provider. Failed cleanup is reported rather than represented as a successful retry. +Cancellation is cooperative: `task.cancel()` requests termination, and `VoiceSession` awaits task completion before reporting a successful interruption. Startup rollback attempts to cancel any created task and abort a partially initialized STT provider. Failed cleanup is reported rather than represented as a successful retry. ## Testing @@ -369,6 +399,12 @@ uv run pytest -q tests/test_gateway.py uv run ruff check src tests --select F401,F811,F821 ``` +Run browser controller, worklet, and playback tests: + +```bash +node --test tests/js/*.test.mjs +``` + ### End-to-end smoke test With the Gateway, NVIDIA ASR, and VoxCPM services available: @@ -377,9 +413,36 @@ With the Gateway, NVIDIA ASR, and VoxCPM services available: uv run python tools/voice_pipeline_smoke_test.py ``` -The script reads `audios/chafik_normalize.wav` (16 kHz, mono, 16-bit PCM), sends audio over WebSocket, and checks for a final transcript, an Agent response, synthesized audio, and `turn.ended`. It writes raw output to `/tmp/voice_pipeline_response.pcm`. +This existing script uses manual turn control, not continuous server VAD. +Its `AUDIO_FILE` currently points to the absolute path +`/home/ubuntu/voice-assistant/audios/chafik_normalize.wav`; provide a real +16 kHz, mono, 16-bit PCM WAV and update that constant before running it elsewhere. +`GATEWAY_URL` must match the running port. It checks for a final transcript, an +Agent response, synthesized audio, and `turn.ended`, and writes raw output to +`/tmp/voice_pipeline_response.pcm`. It does not assess recognition accuracy. -**Last verified checkpoint:** 142 automated tests passing; the full NVIDIA STT → Mock Agent → VoxCPM TTS smoke test completed successfully. These results describe the development environment, not a production SLA. +### Continuous conversation checks and tuning + +With the browser console open, test initial silence, background noise, a short +phrase, a longer phrase with natural pauses, interruption during playback, and +session closure/reconnection. Compare `[STT FINAL]` with the words actually spoken; +hearing synthesized speech alone does not confirm recognition accuracy. + +During development, the phrase "fala um oi pra Lara" was transcribed with a +missing ending at 300 ms silence and recognized correctly after increasing it to +800 ms. Threshold was then increased from 0.5 to 0.6 for a background-noise trial. +These are environment-specific adjustments, not universal recommended values. +The current application interrupts on `SpeechStarted`, without an additional +minimum sustained-speech confirmation. False positives can therefore interrupt +playback; background speech and acoustic echo need separate evaluation. + +Functional conversation checks do not establish a WER benchmark or resource +budget. Systematic WER, end-to-end latency distributions, and CPU/memory under +concurrent sessions have not been measured. A repeatable quality evaluation needs +recorded audio plus reference transcripts, with consistent normalization and +identical input for each configuration. Measure response latency to audible +playback separately from server synthesis completion. Results from the Mock +Agent do not characterize latency with a production LLM. ## Project structure @@ -448,7 +511,9 @@ Each adapter owns its model state, PCM buffer, and input timeline. It converts PCM S16LE to normalized float samples, buffers 512-sample windows, and maps Silero boundaries to the application events. Defaults are threshold 0.5, 300 ms of silence, 30 ms speech padding, and a 10-second worker wait limit. -These values are initial settings, to be tuned with real conversational audio. +These constructor defaults apply when options are omitted. The development YAML +overrides threshold to 0.6 and silence to 800 ms; tune them with representative +audio rather than changing constructor defaults. `process()` may return several events for a large chunk. `flush()` pads a final partial window internally while keeping event positions within real audio. @@ -482,8 +547,8 @@ structure as STT, Agent, and TTS. `config/development.yaml` contains: vad: provider: silero options: - threshold: 0.5 - min_silence_duration_ms: 300 + threshold: 0.6 + min_silence_duration_ms: 800 speech_pad_ms: 30 worker_timeout_seconds: 10.0 ``` @@ -499,6 +564,20 @@ new adapter instances after configuration is reloaded; there is no live reload. The main web client uses the VAD factory through continuous voice sessions. Its detection settings come from the active environment YAML. +| Option | Development value | Effect | +| --- | --- | --- | +| `vad.options.threshold` | `0.6` | Speech probability activation threshold. Raising it can reduce false starts but miss quiet speech. | +| `vad.options.min_silence_duration_ms` | `800` | Silence required to end detected speech. Longer waits preserve pauses but delay finalization. | +| `vad.options.speech_pad_ms` | `30` | Padding around detected boundaries; does not confirm sustained speech. | +| `vad.options.worker_timeout_seconds` | `10.0` | Maximum wait for a model worker operation; not an endpointing delay or forced thread termination. | +| `voice_input.pre_roll_ms` | `200` | Audio retained before the detected start and forwarded to STT, bounded by history and the previous utterance. | +| `voice_input.history_ms` | `2000` | Retained input history for delayed boundary events; must cover pre-roll. | +| `voice_input.max_input_chunk_bytes` | `65536` | Maximum continuous PCM input chunk, also capped by the Gateway at 64 KiB. | + +Tune one parameter at a time using the same audio. End-of-speech silence does +not filter start-of-speech noise triggers. No additional interruption-confirmation +parameter is implemented; that would require an application policy change. + ### Continuous audio and backend turn control