feat: integra VAD ao controle de turnos da conversa

This commit is contained in:
alex alves
2026-11-03 17:44:30 -03:00
parent d27c26c166
commit bdd2f6f67d
9 changed files with 770 additions and 24 deletions

View File

@@ -388,7 +388,7 @@ The script reads `audios/chafik_normalize.wav` (16 kHz, mono, 16-bit PCM), sends
config/ Environment YAML and provider catalog
src/voice_assistant/
audio_gateway/app/ WebSocket endpoint and protocol
application/ VoiceSession and transport-independent events
application/ VoiceSession, continuous PCM history, and application events
domain/ Session state and domain rules
vad/ VAD port, speech boundaries, and local Silero adapter
audio/ Canonical audio models
@@ -418,7 +418,7 @@ tools/ Integration and smoke scripts
## Backend VAD contract (integration pending)
## Backend VAD contract
`vad/adapter.py` defines one stateful detector per continuous input stream.
Its input is the canonical PCM S16LE, mono, 16 kHz audio format. Arbitrary
@@ -432,9 +432,9 @@ boundary. Model padding never extends the input timeline.
`reset()` discards detection state and restarts the timeline, and `aclose()`
releases resources without emitting new speech events. Calls are serialized.
These contracts do not start turns, select providers, or carry transport IDs.
The local Silero implementation is available independently of the running
conversation. Application integration and browser VAD replacement are subsequent
tasks; the active conversation flow still uses its existing browser VAD.
The Silero adapter can control turns through the continuous VoiceSession API.
Browser/WebSocket migration is the next task; the existing web client continues
to use its current turn commands until that migration.
### Local Silero VAD adapter
@@ -497,5 +497,49 @@ remains compatible with existing configurations; explicitly creating VAD in
such an environment raises a configuration error. Changing YAML values affects
new adapter instances after configuration is reloaded; there is no live reload.
This factory makes the configuration usable independently. Connecting it to the
conversation lifecycle remains the next task.
The VAD factory can be supplied to continuous voice sessions. The existing web
client remains on manual turn commands until the browser migration.
### Continuous audio and backend turn control
A `VoiceSession` constructed with a VAD receives continuous canonical PCM through
`await session.receive_audio(pcm)`. The caller supplies audio without a turn ID;
the channel owns transport sequencing. `SpeechStarted` opens an STT turn and
`SpeechEnded` schedules normal STT finalization in the background. New speech
interrupts an active response/finalization before opening a fresh turn. Empty
STT results release the turn through an interruption event rather than leaving
an active turn behind. The VAD is not reset between conversation turns.
The application's sample-indexed audio history preserves the beginning of speech
when VAD confirmation arrives later. PCM already sent to a turn is not replayed
within that turn; pre-roll never includes the previous utterance. History and
input size are bounded by environment configuration:
```yaml
voice_input:
pre_roll_ms: 200
history_ms: 2000
max_input_chunk_bytes: 65536
```
History is trimmed after processing each input chunk; transient storage is at most
retained history plus one accepted chunk. A speech start older than retained
history is reported as an input error instead of silently dropping speech.
Invalid PCM is rejected before detection or timeline advancement. A processing
failure requires a new session; cleanup still releases VAD and provider resources.
Use `finish_audio_input()` for normal end of a continuous stream. Session end or
disconnect aborts through `end()`/`aclose()`, without forcing a final speech event.
The caller shields cleanup from transport cancellation, as for manual sessions.
Backend-controlled sessions emit `TurnOpened` and `TurnInterrupted`; the WebSocket
mapper translates them to the existing `turn.started` and `interrupted` events.
Adapters should discard outdated queued playback when a new turn opens: backend
turn completion describes produced output, not physical speaker completion.
Manual `start_turn`/`send_audio`/`stop_turn` commands remain available on sessions
without VAD. Mixing those commands with a VAD-controlled session is rejected.
`create_voice_session(..., server_vad=True)` composes the continuous backend API;
the current `/ws/audio` receive loop still uses the manual mode. The next task
connects continuous input to that route and simplifies the browser accordingly.