feat(stt): pre-upload silence trim for cloud providers
Local faster-whisper gets Silero VAD (bf8004e3a) so silence never
reaches the model. Cloud providers got no such protection: the raw
file uploads untouched, so every second of silence in a voice note is
paid for twice — upload time and per-audio-minute billing — and cloud
Whisper hallucinates junk tokens on silent stretches exactly like
local Whisper did before the VAD hardening. A 13s voice note with two
long pauses is billed as 13s of audio to transcribe ~6s of speech.
Close the gap client-side: before uploading to a built-in cloud
provider (groq/openai/mistral/xai/elevenlabs/deepinfra), collapse long
pauses with ffmpeg's silenceremove filter, keeping
stt.cloud_trim_keep_ms (default 300) of every pause so word boundaries
and natural pacing survive. Uses ffmpeg, already a dependency of this
exact path via _transcode_audio_for_stt — no new dependency.
The trim is strictly best-effort — ALL of these upload the original
untouched, transcription never fails because of the trim:
- stt.cloud_trim_silence: false
- ffmpeg/ffprobe missing, trim failure, or timeout
- trimmed result ~empty (mostly-silence clip: the provider, not a
client-side dB heuristic, decides whether it contains speech)
- trim saves <10% (re-encoding for nothing)
Command-type and plugin providers are deliberately NOT trimmed: they
may wrap local CLIs that want the original bytes or run their own VAD.
E2E (real ffmpeg + faster-whisper): 13.2s voice note with 7s pause ->
6.2s upload (-53%); transcript of trimmed audio matches the original
on both utterances. Dense-speech and all-silence WAVs correctly fall
back to the original. 22 unit+E2E tests; STT/voice suite failures
identical to upstream/main baseline (all pre-existing).
This commit is contained in:
@@ -1193,6 +1193,13 @@ platform_toolsets:
|
||||
stt:
|
||||
enabled: true
|
||||
# provider: "local" # auto-detected if omitted
|
||||
# --- Cloud pre-upload silence trim (groq/openai/mistral/xai/elevenlabs/deepinfra) ---
|
||||
# Local whisper gets Silero VAD; cloud endpoints otherwise receive raw audio —
|
||||
# silence inflates upload time, per-audio-minute billing, and hallucination risk.
|
||||
# Collapses pauses with ffmpeg client-side; on any failure the original uploads untouched.
|
||||
# cloud_trim_silence: true # set false to always upload the original audio
|
||||
# cloud_trim_threshold_db: -40 # audio quieter than this counts as silence
|
||||
# cloud_trim_keep_ms: 300 # how much of each pause survives (keeps natural pacing)
|
||||
local:
|
||||
model: "base" # tiny | base | small | medium | large-v3 | turbo
|
||||
# language: "" # auto-detect; set to "en", "es", "fr", etc. to force
|
||||
|
||||
Reference in New Issue
Block a user