alex.a.alves/voiceagent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: migra cliente web para áudio contínuo com VAD no backend

Browse Source
This commit is contained in:
alex alves
2026-11-03 18:34:45 -03:00
parent bdd2f6f67d
commit 7cbe86ed39
14 changed files with 834 additions and 1574 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
README.md
View File

@@ -321,17 +321,16 @@ sequenceDiagram
participant C as Client
participant G as Audio Gateway
participant P as VoicePipeline / Providers
C->>G: session.start
C->>G: session.start (turn_detection=server)
G-->>C: session.ready
C->>G: audio.start
G-->>C: turn.started
loop Audio input
C->>G: audio.chunk (turn_id, sequence_number)
loop Continuous audio input, including silence and playback
C->>G: audio.input (sequence_number)
C->>G: Binary PCM frame
G-->>C: turn.started (when backend VAD detects speech)
P-->>G: STT partial events
G-->>C: stt.partial
end
C->>G: audio.stop (turn_id)
Note over G,P: Backend VAD ends input after silence
P-->>G: STT final transcript
G-->>C: stt.final
P-->>G: Agent response
@@ -341,7 +340,7 @@ sequenceDiagram
G-->>C: turn.ended
```
STT events may arrive while input audio is still streaming. Binary audio input is preceded by an `audio.chunk` control message containing `turn_id` and `sequence_number`. The Gateway limits each binary chunk to 64 KiB. Input frames must contain complete PCM samples; rejected frames do not advance the audio sequence. `session.ready.audio` describes input and `session.ready.output_audio` describes output. The browser configures playback from the declared output format. VoxCPM reassembles HTTP fragments into complete PCM frames and rejects a truncated final sample. Its `sample_rate` option declares the expected native output rate; it does not request resampling from the external service.
STT events may arrive while input audio is still streaming. In server detection mode, each binary PCM frame is preceded by `audio.input` with a session-wide `sequence_number`, starting at zero and continuing across turns. `session.ready.max_audio_chunk_bytes` declares the accepted limit, capped at 64 KiB. The browser sizes worklet chunks to respect it. Manual mode uses `audio.chunk` with `turn_id` and a per-turn sequence. Input frames must contain complete PCM samples; rejected frames do not advance the audio sequence. `session.ready.audio` describes input and `session.ready.output_audio` describes output. The browser configures playback from the declared output format. VoxCPM reassembles HTTP fragments into complete PCM frames and rejects a truncated final sample. Its `sample_rate` option declares the expected native output rate; it does not request resampling from the external service.
### Interruption and session closure
@@ -351,7 +350,7 @@ STT events may arrive while input audio is still streaming. Binary audio input i
## Lifecycle and interruption
The system separates **semantic state** from **asynchronous execution**. `TurnManager` controls listening, processing, speaking, and completion; `VoicePipeline` owns the registered turn task. The Gateway coordinates provider shutdown and WebSocket delivery.
The system separates **semantic state** from **asynchronous execution**. `TurnManager` controls listening, processing, speaking, and completion; `VoicePipeline` owns the registered turn task. VoiceSession coordinates provider shutdown; the Gateway owns WebSocket delivery and shields cleanup from transport cancellation.
Cancellation is cooperative: `task.cancel()` requests termination, and the Gateway awaits task completion before confirming a successful interruption. Startup rollback attempts to cancel any created task and abort a partially initialized STT provider. Failed cleanup is reported rather than represented as a successful retry.
@@ -433,8 +432,8 @@ boundary. Model padding never extends the input timeline.
releases resources without emitting new speech events. Calls are serialized.
These contracts do not start turns, select providers, or carry transport IDs.
The Silero adapter can control turns through the continuous VoiceSession API.
Browser/WebSocket migration is the next task; the existing web client continues
to use its current turn commands until that migration.
The main web client requests server detection and streams continuous PCM through
the WebSocket route. Speech detection and turn decisions run in the backend.
### Local Silero VAD adapter
@@ -497,8 +496,8 @@ remains compatible with existing configurations; explicitly creating VAD in
such an environment raises a configuration error. Changing YAML values affects
new adapter instances after configuration is reloaded; there is no live reload.
The VAD factory can be supplied to continuous voice sessions. The existing web
client remains on manual turn commands until the browser migration.
The main web client uses the VAD factory through continuous voice sessions.
Its detection settings come from the active environment YAML.
### Continuous audio and backend turn control
@@ -541,5 +540,32 @@ turn completion describes produced output, not physical speaker completion.
Manual `start_turn`/`send_audio`/`stop_turn` commands remain available on sessions
without VAD. Mixing those commands with a VAD-controlled session is rejected.
`create_voice_session(..., server_vad=True)` composes the continuous backend API;
the current `/ws/audio` receive loop still uses the manual mode. The next task
connects continuous input to that route and simplifies the browser accordingly.
`/ws/audio` selects this mode when `session.start.turn_detection` is `server`.
Omitting that option selects the existing manual input mode for compatible clients.
### Browser continuous capture
The main `/ui` client requests `turn_detection: server` and waits for the matching
confirmation in `session.ready` before starting microphone capture. The worklet
then stays active throughout listening, processing, response playback, and
interruption. Input sequence numbers do not reset between turns. Microphone
permission, echo cancellation, and audio device ownership remain in the page.
The browser's custom `vad.js` and its detection tests have been removed. The voice
controller now handles connection, continuous PCM transport, protocol events,
and playback. Existing visual amplitude analysis is used only for animation.
It sends no automatic `audio.start`, `audio.stop`, or `interrupt` commands.
A new `turn.started` stops previous playback and discards obsolete queued audio.
`turn.ended` waits for scheduled speaker playback before updating the display to
listening; microphone streaming continues during that wait. Stale turn events
and completion callbacks cannot overwrite a newer turn. Outbound WebSocket
buffering and handshake waiting are bounded; failures stop local capture and
close the session.
`audio.input.end` finishes a finite continuous stream normally and requires no
pending binary frame. It is intended for file/terminal clients; the live browser
uses `session.end` or disconnect to abort. Manual input remains available with
`turn_detection: client`, including the standalone manual `app.js` and existing
smoke scripts. The main page uses `voice-session.js`.