feat(tts): speech toggles warm up and unload local TTS engines (#100881)
Desktop "Read replies aloud" / voice conversation, TUI and CLI /voice tts now hold a lease on the TTS engine. Acquiring pre-loads the configured provider (piper/kittentts model into the same LRU slot synthesis reads; lazily-installed cloud SDKs), so the first spoken reply no longer pays the model load as dead air. Releasing the last lease across surfaces unloads resident local models. - tools/tts_tool.py: warm_tts_provider / release_tts_provider / acquire_tts_lease / release_tts_lease over a _LOCAL_TTS_MODEL_CACHES registry; piper/kittentts loaders extracted so warm-up and synthesis share one resolution path. - web_server: POST /api/audio/tts-lease (profile-scoped, off-loop, failures reported in body never as HTTP errors). - tui_gateway voice.toggle + cli.py /voice tts|on|off wire the lease. - desktop: lib/tts-lease.ts (dedupe, per-lease serialization, latest intent wins) driven from useComposerVoice; setTtsLease API client. - docs: features/tts.md section. Live (real piper, isolated HERMES_HOME): first synthesis 988ms cold → 92ms after the toggle warmed the engine; release drops the model.
This commit is contained in:
Teknium
@@ -3,6 +3,7 @@ import type {
|
||||
ActionStatusResponse,
|
||||
AudioSpeakResponse,
|
||||
AudioTranscriptionResponse,
|
||||
AudioTtsLeaseResponse,
|
||||
BackendUpdateCheckResponse,
|
||||
CuratorStatusResponse,
|
||||
DebugShareResponse,
|
||||
@@ -196,6 +197,26 @@ export function speakText(text: string): Promise<AudioSpeakResponse> {
|
||||
})
|
||||
}
|
||||
|
||||
// Acquiring a lease pre-loads the configured TTS engine. For local engines
|
||||
// that is a model load and, on a fresh install, a voice download — well past
|
||||
// the default 15s Electron backend timeout.
|
||||
export const AUDIO_TTS_LEASE_REQUEST_TIMEOUT_MS = 180_000
|
||||
|
||||
/**
|
||||
* Tell the backend a speech-output toggle flipped so it can warm the TTS engine
|
||||
* (`active: true`) or release it once no surface needs it (`active: false`).
|
||||
* `lease` names the toggle — `desktop:read-aloud`, `desktop:conversation`.
|
||||
*/
|
||||
export function setTtsLease(lease: string, active: boolean): Promise<AudioTtsLeaseResponse> {
|
||||
return hermesApi<AudioTtsLeaseResponse>({
|
||||
...profileScoped(),
|
||||
path: '/api/audio/tts-lease',
|
||||
method: 'POST',
|
||||
body: { active, lease },
|
||||
timeoutMs: AUDIO_TTS_LEASE_REQUEST_TIMEOUT_MS
|
||||
})
|
||||
}
|
||||
|
||||
export function getElevenLabsVoices(profile?: null | string): Promise<ElevenLabsVoicesResponse> {
|
||||
return hermesApi<ElevenLabsVoicesResponse>({
|
||||
path: '/api/audio/elevenlabs/voices',
|
||||
|
||||
@@ -5,6 +5,7 @@ import { useI18n } from '@/i18n'
|
||||
import { chatMessageText, collectUnspokenTurnSpeech } from '@/lib/chat-messages'
|
||||
import { triggerHaptic } from '@/lib/haptics'
|
||||
import { markAssistantIdSpoken, resolveSpokenReply } from '@/lib/spoken-reply'
|
||||
import { CONVERSATION_LEASE, READ_ALOUD_LEASE, syncTtsLease } from '@/lib/tts-lease'
|
||||
import { clearWakeIndicator, syncWakeIndicatorWithVoice } from '@/lib/wake-indicator'
|
||||
import { $voiceConversationStartRequest, takeVoiceConversationStart } from '@/store/composer'
|
||||
import { resetBrowseState } from '@/store/composer-input-history'
|
||||
@@ -265,6 +266,26 @@ export function useComposerVoice({
|
||||
|
||||
useEffect(() => resumeWakeIfPaused, [resumeWakeIfPaused])
|
||||
|
||||
// Speech-output toggles are TTS warm-up / release signals. Entering a voice
|
||||
// conversation acquires this window's lease (pre-loads the engine so the
|
||||
// first spoken reply doesn't start with dead air); ending it releases the
|
||||
// lease, and the backend unloads resident local models once no surface holds
|
||||
// one. Fire-and-forget — the toggle never waits on or fails from this.
|
||||
useEffect(() => {
|
||||
void syncTtsLease(CONVERSATION_LEASE, voiceConversationActive)
|
||||
}, [voiceConversationActive])
|
||||
|
||||
useEffect(() => () => void syncTtsLease(CONVERSATION_LEASE, false), [])
|
||||
|
||||
// "Read replies aloud" is the same signal, held for as long as the toggle is
|
||||
// on (it mirrors voice.auto_tts, so this also warms at startup when the
|
||||
// preference is already set).
|
||||
const autoSpeakReplies = useStore($autoSpeakReplies)
|
||||
|
||||
useEffect(() => {
|
||||
void syncTtsLease(READ_ALOUD_LEASE, autoSpeakReplies)
|
||||
}, [autoSpeakReplies])
|
||||
|
||||
// Explicit start/end for the on-screen conversation controls (the hotkey uses
|
||||
// the gated toggle above).
|
||||
const startConversation = useCallback(() => setVoiceConversationActive(true), [])
|
||||
|
||||
@@ -40,6 +40,7 @@ export type {
|
||||
AnalyticsTotals,
|
||||
AudioSpeakResponse,
|
||||
AudioTranscriptionResponse,
|
||||
AudioTtsLeaseResponse,
|
||||
AutomationBlueprint,
|
||||
AutomationBlueprintField,
|
||||
AuxiliaryModelsResponse,
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const setTtsLease = vi.fn(async (_lease: string, _active: boolean) => ({ ok: true }))
|
||||
|
||||
vi.mock('@/hermes', () => ({
|
||||
setTtsLease: (lease: string, active: boolean) => setTtsLease(lease, active)
|
||||
}))
|
||||
|
||||
import { CONVERSATION_LEASE, READ_ALOUD_LEASE, resetTtsLeasesForTests, syncTtsLease } from './tts-lease'
|
||||
|
||||
describe('syncTtsLease', () => {
|
||||
beforeEach(() => {
|
||||
resetTtsLeasesForTests()
|
||||
setTtsLease.mockReset()
|
||||
setTtsLease.mockImplementation(async () => ({ ok: true }))
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
resetTtsLeasesForTests()
|
||||
})
|
||||
|
||||
it('acquires on the first on and releases on off', async () => {
|
||||
await syncTtsLease(READ_ALOUD_LEASE, true)
|
||||
await syncTtsLease(READ_ALOUD_LEASE, false)
|
||||
|
||||
expect(setTtsLease.mock.calls).toEqual([
|
||||
[READ_ALOUD_LEASE, true],
|
||||
[READ_ALOUD_LEASE, false]
|
||||
])
|
||||
})
|
||||
|
||||
it('skips an initial off — never releases a lease it did not hold', async () => {
|
||||
await syncTtsLease(CONVERSATION_LEASE, false)
|
||||
|
||||
expect(setTtsLease).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('dedupes a repeat of the last sent state', async () => {
|
||||
await syncTtsLease(READ_ALOUD_LEASE, true)
|
||||
await syncTtsLease(READ_ALOUD_LEASE, true)
|
||||
await syncTtsLease(READ_ALOUD_LEASE, true)
|
||||
|
||||
expect(setTtsLease).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('queues an off behind an in-flight on so the wire never sees them reordered', async () => {
|
||||
let finishAcquire: () => void = () => undefined
|
||||
setTtsLease.mockImplementationOnce(
|
||||
() =>
|
||||
new Promise(resolve => {
|
||||
finishAcquire = () => resolve({ ok: true })
|
||||
})
|
||||
)
|
||||
|
||||
const on = syncTtsLease(CONVERSATION_LEASE, true)
|
||||
// Let the acquire actually go out (it runs on a microtask).
|
||||
await Promise.resolve()
|
||||
expect(setTtsLease.mock.calls).toEqual([[CONVERSATION_LEASE, true]])
|
||||
|
||||
const off = syncTtsLease(CONVERSATION_LEASE, false)
|
||||
await Promise.resolve()
|
||||
// Still only the acquire — the release waits for it to finish.
|
||||
expect(setTtsLease).toHaveBeenCalledTimes(1)
|
||||
|
||||
finishAcquire()
|
||||
await Promise.all([on, off])
|
||||
|
||||
expect(setTtsLease.mock.calls).toEqual([
|
||||
[CONVERSATION_LEASE, true],
|
||||
[CONVERSATION_LEASE, false]
|
||||
])
|
||||
})
|
||||
|
||||
it('coalesces a flip that reverses before its call went out — latest intent wins', async () => {
|
||||
const on = syncTtsLease(CONVERSATION_LEASE, true)
|
||||
const off = syncTtsLease(CONVERSATION_LEASE, false)
|
||||
await Promise.all([on, off])
|
||||
|
||||
// The acquire never had a chance to go out; only the terminal state is sent
|
||||
// (a release of a never-held lease is a backend no-op).
|
||||
expect(setTtsLease.mock.calls).toEqual([[CONVERSATION_LEASE, false]])
|
||||
})
|
||||
|
||||
it('forgets the sent state on failure so the next flip retries', async () => {
|
||||
setTtsLease.mockImplementationOnce(async () => {
|
||||
throw new Error('backend not ready')
|
||||
})
|
||||
|
||||
await expect(syncTtsLease(READ_ALOUD_LEASE, true)).resolves.toBeUndefined()
|
||||
await syncTtsLease(READ_ALOUD_LEASE, true)
|
||||
|
||||
expect(setTtsLease).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('conversation lease is per renderer, read-aloud lease is shared', () => {
|
||||
expect(CONVERSATION_LEASE).toMatch(/^desktop:conversation:[a-z0-9]+$/)
|
||||
expect(READ_ALOUD_LEASE).toBe('desktop:read-aloud')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,76 @@
|
||||
import { setTtsLease } from '@/hermes'
|
||||
|
||||
// The desktop's speech-output toggles — "Read replies aloud" and voice
|
||||
// conversation mode — are the user telling us TTS is about to be needed (or no
|
||||
// longer is). The backend turns that into engine lifecycle: acquiring a lease
|
||||
// pre-loads the configured provider (a local piper/kittentts model, a lazily
|
||||
// installed SDK) so the first spoken reply starts hot instead of paying the load
|
||||
// as dead air; releasing the last lease unloads resident local models.
|
||||
//
|
||||
// This module is the renderer's single choke point for that signal. It dedupes
|
||||
// (several composers/tiles observe the same toggle), serializes per lease so a
|
||||
// fast on→off→on can't be reordered on the wire, and never surfaces failures —
|
||||
// warm-up is an optimization; the toggle itself must not depend on it.
|
||||
|
||||
// Per-renderer id so two windows in conversation mode hold DISTINCT leases —
|
||||
// window A ending its conversation must not release the engine window B is
|
||||
// still speaking through. Read-aloud mirrors one config key shared by every
|
||||
// window, so it deliberately uses one shared lease name.
|
||||
const RENDERER_ID = Math.random().toString(36).slice(2, 10)
|
||||
|
||||
export const READ_ALOUD_LEASE = 'desktop:read-aloud'
|
||||
export const CONVERSATION_LEASE = `desktop:conversation:${RENDERER_ID}`
|
||||
|
||||
const sent = new Map<string, boolean>()
|
||||
const inFlight = new Map<string, Promise<void>>()
|
||||
|
||||
/**
|
||||
* Bring the backend's view of `lease` in line with `active`. Idempotent: a
|
||||
* repeat of the last sent state is a no-op. The initial `false` (nothing was
|
||||
* ever acquired) is also skipped — releasing a lease we never held would only
|
||||
* churn the backend on app start.
|
||||
*/
|
||||
export function syncTtsLease(lease: string, active: boolean): Promise<void> {
|
||||
const last = sent.get(lease)
|
||||
|
||||
if (last === active || (last === undefined && !active)) {
|
||||
return inFlight.get(lease) ?? Promise.resolve()
|
||||
}
|
||||
|
||||
sent.set(lease, active)
|
||||
|
||||
const previous = inFlight.get(lease) ?? Promise.resolve()
|
||||
|
||||
const next = previous
|
||||
.then(async () => {
|
||||
// Latest intent wins: if the toggle flipped again while we were queued,
|
||||
// the newer call sends its own state and this one has nothing to say.
|
||||
if (sent.get(lease) !== active) {
|
||||
return
|
||||
}
|
||||
|
||||
await setTtsLease(lease, active)
|
||||
})
|
||||
.catch(() => {
|
||||
// Backend not up yet / older backend without the endpoint / warm-up
|
||||
// failure: forget what we "sent" so the next flip retries honestly.
|
||||
if (sent.get(lease) === active) {
|
||||
sent.delete(lease)
|
||||
}
|
||||
})
|
||||
.finally(() => {
|
||||
if (inFlight.get(lease) === next) {
|
||||
inFlight.delete(lease)
|
||||
}
|
||||
})
|
||||
|
||||
inFlight.set(lease, next)
|
||||
|
||||
return next
|
||||
}
|
||||
|
||||
/** Test seam — forget every sent state. */
|
||||
export function resetTtsLeasesForTests() {
|
||||
sent.clear()
|
||||
inFlight.clear()
|
||||
}
|
||||
@@ -29,6 +29,21 @@ export interface AudioSpeakResponse {
|
||||
provider?: string
|
||||
}
|
||||
|
||||
/** `POST /api/audio/tts-lease` — TTS engine warm-up / release driven by speech toggles. */
|
||||
export interface AudioTtsLeaseResponse {
|
||||
ok: boolean
|
||||
lease: string
|
||||
active: boolean
|
||||
/** Live lease holders after this call (null when the backend call itself failed). */
|
||||
leases: null | number
|
||||
/** Warm-up outcome: `loaded` | `cached` | `installed` | `noop` | `error`. */
|
||||
action?: string
|
||||
provider?: string
|
||||
/** Resident local models dropped (release path). */
|
||||
released?: number
|
||||
error?: string
|
||||
}
|
||||
|
||||
export interface ElevenLabsVoice {
|
||||
label: string
|
||||
name: string
|
||||
|
||||
@@ -15765,6 +15765,10 @@ class HermesCLI(CLIAgentSetupMixin, CLICommandsMixin, CLIBillingMixin):
|
||||
# _voice_message_prefix property and its usage in _process_message().
|
||||
|
||||
tts_status = " (TTS enabled)" if self._voice_tts else ""
|
||||
if self._voice_tts:
|
||||
# Speech output is on from the start — warm the engine now so the
|
||||
# first spoken reply doesn't pay the model load as dead air.
|
||||
self._tts_lease_async(True)
|
||||
# Use the startup-pinned cache so the advertised shortcut always
|
||||
# matches the live prompt_toolkit binding — reading live config
|
||||
# here would drift after a mid-session config edit (Copilot
|
||||
@@ -15823,6 +15827,11 @@ class HermesCLI(CLIAgentSetupMixin, CLICommandsMixin, CLIBillingMixin):
|
||||
self._voice_tts = False
|
||||
self._voice_continuous = False
|
||||
|
||||
# Speech output is off with the mode — release the TTS engine lease so
|
||||
# a resident local model (piper/kittentts) is freed once nothing else
|
||||
# in this process still needs it.
|
||||
self._tts_lease_async(False)
|
||||
|
||||
# Shut down the persistent audio stream in background
|
||||
if recorder is not None:
|
||||
def _bg_shutdown(rec=recorder):
|
||||
@@ -16070,6 +16079,29 @@ class HermesCLI(CLIAgentSetupMixin, CLICommandsMixin, CLIBillingMixin):
|
||||
if not owned:
|
||||
_cprint(f" {_DIM}Enable with /wake on{_RST}")
|
||||
|
||||
def _tts_lease_async(self, active: bool) -> None:
|
||||
"""Acquire/release this CLI's TTS engine lease in the background.
|
||||
|
||||
The /voice tts toggle (and voice-mode on/off with speech output set)
|
||||
is the "TTS is about to be needed / no longer needed" signal:
|
||||
acquiring pre-loads the configured provider so the first reply starts
|
||||
hot; releasing lets the last-holder path unload resident local models.
|
||||
Never blocks the toggle and never fails it.
|
||||
"""
|
||||
|
||||
def _run():
|
||||
try:
|
||||
from tools.tts_tool import acquire_tts_lease, release_tts_lease
|
||||
|
||||
if active:
|
||||
acquire_tts_lease("cli:voice-tts")
|
||||
else:
|
||||
release_tts_lease("cli:voice-tts")
|
||||
except Exception as e:
|
||||
logger.debug("voice: tts lease active=%s failed: %s", active, e)
|
||||
|
||||
threading.Thread(target=_run, name="tts-lease-cli", daemon=True).start()
|
||||
|
||||
def _toggle_voice_tts(self):
|
||||
"""Toggle TTS output for voice mode."""
|
||||
if not self._voice_mode:
|
||||
@@ -16085,6 +16117,10 @@ class HermesCLI(CLIAgentSetupMixin, CLICommandsMixin, CLIBillingMixin):
|
||||
if not check_tts_requirements():
|
||||
_cprint(f"{_DIM}Warning: No TTS provider available. Install edge-tts or set API keys.{_RST}")
|
||||
|
||||
# Toggle = warm-up / release signal for the TTS engine (see
|
||||
# tools.tts_tool.acquire_tts_lease).
|
||||
self._tts_lease_async(self._voice_tts)
|
||||
|
||||
_cprint(f"{_ACCENT}Voice TTS {status}.{_RST}")
|
||||
|
||||
def _show_voice_status(self):
|
||||
|
||||
@@ -306,6 +306,17 @@ class TTSSpeakRequest(BaseModel):
|
||||
text: str
|
||||
|
||||
|
||||
class TTSLeaseRequest(BaseModel):
|
||||
"""Body for ``POST /api/audio/tts-lease``.
|
||||
|
||||
``lease`` names the toggle/surface holding the lease (``desktop:read-aloud``,
|
||||
``desktop:conversation``); ``active`` True acquires + warms, False releases.
|
||||
"""
|
||||
|
||||
lease: str
|
||||
active: bool = True
|
||||
|
||||
|
||||
# --- from web_server.py (originally lines 11549-11551) ---
|
||||
|
||||
class OAuthSubmitBody(BaseModel):
|
||||
|
||||
@@ -1836,6 +1836,7 @@ from hermes_cli.web_models import ( # noqa: F401
|
||||
LearningNodeEdit,
|
||||
DebugShareRequest,
|
||||
TTSSpeakRequest,
|
||||
TTSLeaseRequest,
|
||||
OAuthSubmitBody,
|
||||
BulkDeleteSessions,
|
||||
SessionImport,
|
||||
@@ -5665,6 +5666,43 @@ async def speak_text(payload: TTSSpeakRequest, profile: Optional[str] = None):
|
||||
}
|
||||
|
||||
|
||||
@app.post("/api/audio/tts-lease")
|
||||
async def tts_lease(payload: TTSLeaseRequest, profile: Optional[str] = None):
|
||||
"""Desktop TTS-output toggles as warm-up / release signals.
|
||||
|
||||
"Read replies aloud" and voice-conversation mode are explicit "speech is
|
||||
about to be needed" gestures. ``active: true`` registers the toggle as a
|
||||
lease on the TTS engine and pre-loads the configured provider (local
|
||||
piper/kittentts model, lazily-installed SDK) so the first spoken reply
|
||||
doesn't pay the load as dead air; ``active: false`` drops the lease and,
|
||||
once no surface holds one, unloads resident local models.
|
||||
|
||||
Blocking work (model load, voice download) runs off the event loop.
|
||||
Warm-up failures are reported in the body, never as an HTTP error — the
|
||||
toggle must succeed even when the engine can't preload.
|
||||
"""
|
||||
lease = (payload.lease or "").strip()
|
||||
if not lease:
|
||||
raise HTTPException(status_code=400, detail="lease is required")
|
||||
|
||||
def _apply():
|
||||
from tools.tts_tool import acquire_tts_lease, release_tts_lease
|
||||
|
||||
if payload.active:
|
||||
with _config_profile_scope(profile):
|
||||
return acquire_tts_lease(lease)
|
||||
return release_tts_lease(lease)
|
||||
|
||||
try:
|
||||
result = await asyncio.get_running_loop().run_in_executor(None, _apply)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as exc:
|
||||
_log.warning("TTS lease %s (%s) failed: %s", lease, payload.active, exc)
|
||||
result = {"leases": None, "action": "error", "error": str(exc)}
|
||||
return {"ok": True, "lease": lease, "active": payload.active, **result}
|
||||
|
||||
|
||||
def _split_text_for_speak_stream(text: str, cap: int) -> list:
|
||||
"""Split *text* into provider-cap-sized pieces on sentence boundaries.
|
||||
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
"""``POST /api/audio/tts-lease`` — desktop speech toggles as TTS warm-up/release.
|
||||
|
||||
The desktop's "Read replies aloud" and voice-conversation toggles call this so
|
||||
the backend can pre-load the configured TTS engine when speech is about to be
|
||||
needed and unload resident local models once no surface holds a lease.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def isolated_profiles(tmp_path, monkeypatch, _isolate_hermes_home):
|
||||
from hermes_constants import get_hermes_home
|
||||
from hermes_cli import profiles
|
||||
|
||||
default_home = get_hermes_home()
|
||||
profiles_root = default_home / "profiles"
|
||||
worker_home = profiles_root / "worker_beta"
|
||||
for home in (default_home, worker_home):
|
||||
home.mkdir(parents=True, exist_ok=True)
|
||||
(home / "config.yaml").write_text("{}\n", encoding="utf-8")
|
||||
(worker_home / ".env").write_text("", encoding="utf-8")
|
||||
|
||||
monkeypatch.setattr(profiles, "_get_default_hermes_home", lambda: default_home)
|
||||
monkeypatch.setattr(profiles, "_get_profiles_root", lambda: profiles_root)
|
||||
return {"default": default_home, "worker_beta": worker_home}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(monkeypatch, isolated_profiles):
|
||||
try:
|
||||
from starlette.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("fastapi/starlette not installed")
|
||||
|
||||
import hermes_state
|
||||
from hermes_constants import get_hermes_home
|
||||
from hermes_cli.web_server import app, _SESSION_HEADER_NAME, _SESSION_TOKEN
|
||||
|
||||
monkeypatch.setattr(hermes_state, "DEFAULT_DB_PATH", get_hermes_home() / "state.db")
|
||||
c = TestClient(app)
|
||||
c.headers[_SESSION_HEADER_NAME] = _SESSION_TOKEN
|
||||
return c
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_leases():
|
||||
from tools import tts_tool
|
||||
|
||||
tts_tool._reset_tts_leases_for_tests()
|
||||
for cache in tts_tool._LOCAL_TTS_MODEL_CACHES.values():
|
||||
cache.clear()
|
||||
yield
|
||||
tts_tool._reset_tts_leases_for_tests()
|
||||
for cache in tts_tool._LOCAL_TTS_MODEL_CACHES.values():
|
||||
cache.clear()
|
||||
|
||||
|
||||
def test_active_acquires_and_warms(client, monkeypatch):
|
||||
from tools import tts_tool
|
||||
|
||||
warmed = []
|
||||
monkeypatch.setattr(
|
||||
tts_tool,
|
||||
"warm_tts_provider",
|
||||
lambda cfg=None, provider=None: warmed.append(1) or {"provider": "piper", "warmed": True, "action": "loaded"},
|
||||
)
|
||||
|
||||
resp = client.post("/api/audio/tts-lease", json={"lease": "desktop:read-aloud", "active": True})
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["ok"] is True
|
||||
assert body["lease"] == "desktop:read-aloud"
|
||||
assert body["active"] is True
|
||||
assert body["leases"] == 1
|
||||
assert body["action"] == "loaded"
|
||||
assert warmed == [1]
|
||||
assert tts_tool.tts_lease_holders() == ["desktop:read-aloud"]
|
||||
|
||||
|
||||
def test_inactive_releases_and_unloads_when_last(client, monkeypatch):
|
||||
from tools import tts_tool
|
||||
|
||||
monkeypatch.setattr(tts_tool, "warm_tts_provider", lambda cfg=None, provider=None: {"action": "noop", "warmed": False, "provider": "piper"})
|
||||
client.post("/api/audio/tts-lease", json={"lease": "desktop:read-aloud", "active": True})
|
||||
client.post("/api/audio/tts-lease", json={"lease": "desktop:conversation:abc", "active": True})
|
||||
tts_tool._piper_voice_cache["voice"] = object()
|
||||
|
||||
first = client.post("/api/audio/tts-lease", json={"lease": "desktop:read-aloud", "active": False}).json()
|
||||
assert first["leases"] == 1
|
||||
assert first["released"] == 0
|
||||
assert len(tts_tool._piper_voice_cache) == 1
|
||||
|
||||
last = client.post("/api/audio/tts-lease", json={"lease": "desktop:conversation:abc", "active": False}).json()
|
||||
assert last["leases"] == 0
|
||||
assert last["released"] == 1
|
||||
assert tts_tool._piper_voice_cache == {}
|
||||
|
||||
|
||||
def test_warm_failure_is_reported_not_an_http_error(client, monkeypatch):
|
||||
from tools import tts_tool
|
||||
|
||||
def _boom(cfg=None, provider=None):
|
||||
raise RuntimeError("engine exploded")
|
||||
|
||||
monkeypatch.setattr(tts_tool, "warm_tts_provider", _boom)
|
||||
resp = client.post("/api/audio/tts-lease", json={"lease": "desktop:read-aloud", "active": True})
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["ok"] is True
|
||||
assert body["action"] == "error"
|
||||
assert "engine exploded" in body["error"]
|
||||
|
||||
|
||||
def test_blank_lease_rejected(client):
|
||||
resp = client.post("/api/audio/tts-lease", json={"lease": " ", "active": True})
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
def test_active_default_true(client, monkeypatch):
|
||||
from tools import tts_tool
|
||||
|
||||
monkeypatch.setattr(tts_tool, "warm_tts_provider", lambda cfg=None, provider=None: {"action": "noop", "warmed": False, "provider": "x"})
|
||||
resp = client.post("/api/audio/tts-lease", json={"lease": "tui:x"})
|
||||
assert resp.json()["active"] is True
|
||||
assert tts_tool.tts_lease_holders() == ["tui:x"]
|
||||
|
||||
|
||||
def test_acquire_resolves_provider_inside_target_profile(client, isolated_profiles, monkeypatch):
|
||||
"""Warm-up must read the REQUESTING profile's tts config, like /api/audio/speak."""
|
||||
import yaml
|
||||
from tools import tts_tool
|
||||
|
||||
(isolated_profiles["worker_beta"] / "config.yaml").write_text(
|
||||
yaml.safe_dump({"tts": {"provider": "kittentts"}}), encoding="utf-8"
|
||||
)
|
||||
seen = {}
|
||||
|
||||
def _fake_warm(cfg=None, provider=None):
|
||||
from hermes_constants import get_hermes_home
|
||||
|
||||
seen["home"] = str(get_hermes_home())
|
||||
seen["provider"] = tts_tool._get_provider(tts_tool._load_tts_config())
|
||||
return {"action": "noop", "warmed": False, "provider": seen["provider"]}
|
||||
|
||||
monkeypatch.setattr(tts_tool, "warm_tts_provider", _fake_warm)
|
||||
resp = client.post("/api/audio/tts-lease?profile=worker_beta", json={"lease": "desktop:read-aloud", "active": True})
|
||||
assert resp.status_code == 200
|
||||
assert seen["home"] == str(isolated_profiles["worker_beta"])
|
||||
assert seen["provider"] == "kittentts"
|
||||
|
||||
|
||||
def test_unknown_profile_404(client):
|
||||
resp = client.post("/api/audio/tts-lease?profile=ghost", json={"lease": "desktop:read-aloud", "active": True})
|
||||
assert resp.status_code == 404
|
||||
@@ -0,0 +1,233 @@
|
||||
"""TTS engine lifecycle driven by speech-output toggles (issue #100881).
|
||||
|
||||
Local engines load lazily on first synthesis, so the first spoken reply after
|
||||
"read replies aloud" / voice conversation turns on pays the model load as dead
|
||||
air. The toggles now hold *leases*: acquiring warms the configured provider
|
||||
into the SAME cache slot synthesis reads; releasing the last lease unloads
|
||||
resident local models.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from tools import tts_tool
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_lifecycle(monkeypatch):
|
||||
tts_tool._reset_tts_leases_for_tests()
|
||||
for cache in tts_tool._LOCAL_TTS_MODEL_CACHES.values():
|
||||
cache.clear()
|
||||
yield
|
||||
tts_tool._reset_tts_leases_for_tests()
|
||||
for cache in tts_tool._LOCAL_TTS_MODEL_CACHES.values():
|
||||
cache.clear()
|
||||
|
||||
|
||||
class _FakePiperVoice:
|
||||
loads = 0
|
||||
synthesized: list = []
|
||||
|
||||
@classmethod
|
||||
def load(cls, model_path, use_cuda=False):
|
||||
cls.loads += 1
|
||||
inst = cls()
|
||||
inst.model_path = model_path
|
||||
return inst
|
||||
|
||||
def synthesize_wav(self, text, wav_file, syn_config=None):
|
||||
type(self).synthesized.append(text)
|
||||
wav_file.setnchannels(1)
|
||||
wav_file.setsampwidth(2)
|
||||
wav_file.setframerate(16000)
|
||||
wav_file.writeframes(b"\x00\x00" * 160)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fake_piper(monkeypatch, tmp_path):
|
||||
_FakePiperVoice.loads = 0
|
||||
_FakePiperVoice.synthesized = []
|
||||
monkeypatch.setattr(tts_tool, "_import_piper", lambda: _FakePiperVoice)
|
||||
# Pretend the voice is already on disk so no download subprocess runs.
|
||||
voices_dir = tmp_path / "voices"
|
||||
voices_dir.mkdir()
|
||||
(voices_dir / "en_US-test-medium.onnx").write_bytes(b"onnx")
|
||||
(voices_dir / "en_US-test-medium.onnx.json").write_text("{}")
|
||||
cfg = {"provider": "piper", "piper": {"voice": "en_US-test-medium", "voices_dir": str(voices_dir)}}
|
||||
monkeypatch.setattr(tts_tool, "_load_tts_config", lambda: cfg)
|
||||
return cfg
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# warm_tts_provider: warm-up populates the exact slot synthesis reads
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_warm_loads_piper_into_synthesis_cache(fake_piper, tmp_path):
|
||||
result = tts_tool.warm_tts_provider(fake_piper)
|
||||
|
||||
assert result["warmed"] is True
|
||||
assert result["action"] == "loaded"
|
||||
assert result["provider"] == "piper"
|
||||
assert _FakePiperVoice.loads == 1
|
||||
assert len(tts_tool._piper_voice_cache) == 1
|
||||
|
||||
# The load that would have happened on the first reply is already done:
|
||||
# synthesis reuses the warmed instance without loading again.
|
||||
out = tts_tool._generate_piper_tts("hello", str(tmp_path / "out.wav"), fake_piper)
|
||||
assert out.endswith(".wav")
|
||||
assert _FakePiperVoice.loads == 1
|
||||
assert _FakePiperVoice.synthesized == ["hello"]
|
||||
|
||||
|
||||
def test_warm_twice_is_a_cache_hit(fake_piper):
|
||||
tts_tool.warm_tts_provider(fake_piper)
|
||||
second = tts_tool.warm_tts_provider(fake_piper)
|
||||
|
||||
assert second["action"] == "cached"
|
||||
assert _FakePiperVoice.loads == 1
|
||||
|
||||
|
||||
def test_warm_reads_configured_provider_when_none_given(fake_piper):
|
||||
result = tts_tool.warm_tts_provider()
|
||||
assert result["provider"] == "piper"
|
||||
assert result["action"] == "loaded"
|
||||
|
||||
|
||||
def test_warm_never_raises_on_engine_failure(monkeypatch):
|
||||
def _boom():
|
||||
raise ImportError("No module named 'piper'")
|
||||
|
||||
monkeypatch.setattr(tts_tool, "_import_piper", _boom)
|
||||
result = tts_tool.warm_tts_provider({"provider": "piper"})
|
||||
|
||||
assert result["warmed"] is False
|
||||
assert result["action"] == "error"
|
||||
assert "piper" in result["error"]
|
||||
assert tts_tool._piper_voice_cache == {}
|
||||
|
||||
|
||||
def test_warm_is_noop_for_cloud_provider_without_lazy_sdk(monkeypatch):
|
||||
result = tts_tool.warm_tts_provider({"provider": "openai"})
|
||||
assert result == {"provider": "openai", "warmed": False, "action": "noop"}
|
||||
|
||||
|
||||
def test_warm_lazy_sdk_provider_reports_cached_when_installed(monkeypatch):
|
||||
import types
|
||||
|
||||
fake = types.SimpleNamespace(
|
||||
is_available=lambda feature: feature == "tts.edge",
|
||||
ensure=lambda *a, **k: pytest.fail("ensure must not run when the SDK is present"),
|
||||
)
|
||||
monkeypatch.setitem(__import__("sys").modules, "tools.lazy_deps", fake)
|
||||
result = tts_tool.warm_tts_provider({"provider": "edge"})
|
||||
assert result["warmed"] is True
|
||||
assert result["action"] == "cached"
|
||||
|
||||
|
||||
def test_warm_lazy_sdk_provider_installs_when_missing(monkeypatch):
|
||||
import types
|
||||
|
||||
calls = []
|
||||
fake = types.SimpleNamespace(
|
||||
is_available=lambda feature: False,
|
||||
ensure=lambda feature, prompt: calls.append((feature, prompt)),
|
||||
)
|
||||
monkeypatch.setitem(__import__("sys").modules, "tools.lazy_deps", fake)
|
||||
result = tts_tool.warm_tts_provider({"provider": "edge"})
|
||||
assert result["action"] == "installed"
|
||||
assert calls == [("tts.edge", False)]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# release_tts_provider
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_release_drops_every_local_cache(fake_piper):
|
||||
tts_tool.warm_tts_provider(fake_piper)
|
||||
tts_tool._kittentts_model_cache["m"] = object()
|
||||
|
||||
assert tts_tool.release_tts_provider() == {"released": 2}
|
||||
assert tts_tool._piper_voice_cache == {}
|
||||
assert tts_tool._kittentts_model_cache == {}
|
||||
|
||||
|
||||
def test_release_scoped_to_one_provider(fake_piper):
|
||||
tts_tool.warm_tts_provider(fake_piper)
|
||||
tts_tool._kittentts_model_cache["m"] = object()
|
||||
|
||||
assert tts_tool.release_tts_provider("kittentts") == {"released": 1}
|
||||
assert len(tts_tool._piper_voice_cache) == 1
|
||||
|
||||
|
||||
def test_release_with_nothing_resident_is_zero():
|
||||
assert tts_tool.release_tts_provider() == {"released": 0}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Leases: warm on acquire, unload only when the LAST holder releases
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_acquire_warms_and_counts(fake_piper):
|
||||
result = tts_tool.acquire_tts_lease("desktop:read-aloud")
|
||||
assert result["leases"] == 1
|
||||
assert result["action"] == "loaded"
|
||||
assert tts_tool.tts_lease_holders() == ["desktop:read-aloud"]
|
||||
|
||||
|
||||
def test_last_release_unloads_but_earlier_release_does_not(fake_piper):
|
||||
tts_tool.acquire_tts_lease("desktop:read-aloud")
|
||||
tts_tool.acquire_tts_lease("tui:voice-tts")
|
||||
assert len(tts_tool._piper_voice_cache) == 1
|
||||
|
||||
# One surface turning speech off must not pull the model from under the
|
||||
# other surface that still speaks through this process.
|
||||
first = tts_tool.release_tts_lease("desktop:read-aloud")
|
||||
assert first == {"leases": 1, "released": 0}
|
||||
assert len(tts_tool._piper_voice_cache) == 1
|
||||
|
||||
last = tts_tool.release_tts_lease("tui:voice-tts")
|
||||
assert last == {"leases": 0, "released": 1}
|
||||
assert tts_tool._piper_voice_cache == {}
|
||||
|
||||
|
||||
def test_reacquire_is_idempotent_and_reheals_cache(fake_piper):
|
||||
tts_tool.acquire_tts_lease("cli:voice-tts")
|
||||
tts_tool.release_tts_provider() # something else dropped the model
|
||||
result = tts_tool.acquire_tts_lease("cli:voice-tts")
|
||||
|
||||
assert result["leases"] == 1
|
||||
assert result["action"] == "loaded"
|
||||
assert _FakePiperVoice.loads == 2
|
||||
|
||||
|
||||
def test_release_unknown_lease_is_noop(fake_piper):
|
||||
tts_tool.acquire_tts_lease("a")
|
||||
assert tts_tool.release_tts_lease("never-acquired") == {"leases": 1, "released": 0}
|
||||
assert len(tts_tool._piper_voice_cache) == 1
|
||||
|
||||
|
||||
def test_acquire_failure_still_registers_lease(monkeypatch):
|
||||
def _boom():
|
||||
raise RuntimeError("engine missing")
|
||||
|
||||
monkeypatch.setattr(tts_tool, "_import_piper", _boom)
|
||||
result = tts_tool.acquire_tts_lease("desktop:conversation", {"provider": "piper"})
|
||||
assert result["action"] == "error"
|
||||
assert result["leases"] == 1
|
||||
assert tts_tool.tts_lease_holders() == ["desktop:conversation"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Registry invariant: every local engine cache is release-able
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_every_local_warmer_has_a_registered_cache():
|
||||
warmers = tts_tool._local_tts_warmers()
|
||||
assert set(warmers) == set(tts_tool._LOCAL_TTS_MODEL_CACHES)
|
||||
assert tts_tool._LOCAL_TTS_MODEL_CACHES["piper"] is tts_tool._piper_voice_cache
|
||||
assert tts_tool._LOCAL_TTS_MODEL_CACHES["kittentts"] is tts_tool._kittentts_model_cache
|
||||
+224
-27
@@ -2893,10 +2893,182 @@ def _tts_cache_get_or_load(cache: Dict[str, Any], key: str, load: Callable[[], A
|
||||
return value
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Local-engine lifecycle: warm-up / release driven by TTS-output toggles
|
||||
# ===========================================================================
|
||||
#
|
||||
# Local engines (Piper, KittenTTS) load their model lazily on the first
|
||||
# synthesis call, so the first spoken reply after a user turns on "read
|
||||
# replies aloud" / a voice conversation pays the whole load (plus a voice
|
||||
# download on a fresh install) as dead air before the first word. And once
|
||||
# loaded, the model stays resident for the process lifetime even after every
|
||||
# TTS-output toggle is off again.
|
||||
#
|
||||
# The toggles ARE the intent signal. Every surface that flips speech output
|
||||
# on holds a *lease* here (warming the configured engine as a side effect);
|
||||
# flipping it off releases the lease, and when the last lease is gone the
|
||||
# local model caches are dropped. Lease-counting instead of a bare
|
||||
# on/off keeps one surface's "off" from unloading a model another surface
|
||||
# (TUI /voice tts, desktop read-aloud, desktop conversation) still needs —
|
||||
# they share this process's caches.
|
||||
#
|
||||
# Cloud providers have no resident model; warming them is a no-op beyond
|
||||
# making sure the lazily-installed SDK is importable (edge-tts), which is
|
||||
# also first-use latency users see as silence.
|
||||
|
||||
# Provider name → local model cache it populates. The single registry both
|
||||
# warm_tts_provider() and the release path consult — a new local engine adds
|
||||
# one row here (at its cache declaration) plus a loader in
|
||||
# _local_tts_warmers() and gets warm/release for free.
|
||||
_LOCAL_TTS_MODEL_CACHES: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
|
||||
def _local_tts_warmers() -> Dict[str, Callable[[Dict[str, Any]], Any]]:
|
||||
# Resolved lazily: the loader functions are defined later in this module.
|
||||
return {
|
||||
"piper": lambda cfg: _load_piper_voice_for_config(cfg)[0],
|
||||
"kittentts": lambda cfg: _load_kittentts_model_for_config(cfg)[0],
|
||||
}
|
||||
|
||||
|
||||
def _lazy_sdk_feature_for_provider(provider: str) -> Optional[str]:
|
||||
"""tools.lazy_deps feature key for providers whose SDK installs on first use."""
|
||||
return {
|
||||
"edge": "tts.edge",
|
||||
"elevenlabs": "tts.elevenlabs",
|
||||
"mistral": "tts.mistral",
|
||||
}.get(provider)
|
||||
|
||||
|
||||
_tts_lease_lock = threading.Lock()
|
||||
_tts_leases: set = set()
|
||||
|
||||
|
||||
def warm_tts_provider(
|
||||
tts_config: Optional[Dict[str, Any]] = None,
|
||||
provider: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Pre-load the configured TTS provider so the next synthesis starts hot.
|
||||
|
||||
* Local engines (Piper, KittenTTS): resolve the configured voice/model
|
||||
exactly as synthesis would (including first-use voice download) and
|
||||
load it into the same LRU cache slot synthesis reads.
|
||||
* Lazily-installed cloud SDKs (edge-tts, ElevenLabs, Mistral): make sure
|
||||
the SDK is importable, installing it if lazy installs are allowed.
|
||||
* Everything else: nothing to warm — reported as ``action: "noop"``.
|
||||
|
||||
Never raises; the result dict carries ``warmed`` / ``action`` / ``error``
|
||||
so callers on a toggle path can log and move on. Blocking — callers on a
|
||||
UI thread should run it in the background.
|
||||
"""
|
||||
if tts_config is None:
|
||||
tts_config = _load_tts_config()
|
||||
name = (provider or _get_provider(tts_config) or "").lower().strip()
|
||||
result: Dict[str, Any] = {"provider": name, "warmed": False, "action": "noop"}
|
||||
|
||||
warmer = _local_tts_warmers().get(name)
|
||||
if warmer is not None:
|
||||
cache = _LOCAL_TTS_MODEL_CACHES.get(name)
|
||||
before = len(cache) if cache is not None else 0
|
||||
started = time.monotonic()
|
||||
try:
|
||||
warmer(tts_config)
|
||||
except Exception as exc: # engine missing, download failed, bad voice…
|
||||
logger.warning("[TTS] warm-up for %s failed: %s", name, exc)
|
||||
result.update(action="error", error=str(exc))
|
||||
return result
|
||||
after = len(cache) if cache is not None else 0
|
||||
result.update(
|
||||
warmed=True,
|
||||
action="loaded" if after > before else "cached",
|
||||
elapsed_ms=int((time.monotonic() - started) * 1000),
|
||||
)
|
||||
logger.info("[TTS] warm-up %s: %s in %dms", name, result["action"], result["elapsed_ms"])
|
||||
return result
|
||||
|
||||
feature = _lazy_sdk_feature_for_provider(name)
|
||||
if feature is not None:
|
||||
try:
|
||||
from tools.lazy_deps import ensure, is_available
|
||||
|
||||
if is_available(feature):
|
||||
result.update(warmed=True, action="cached")
|
||||
else:
|
||||
ensure(feature, prompt=False)
|
||||
result.update(warmed=True, action="installed")
|
||||
except Exception as exc:
|
||||
logger.debug("[TTS] SDK warm-up for %s skipped: %s", name, exc)
|
||||
result.update(action="error", error=str(exc))
|
||||
return result
|
||||
|
||||
|
||||
def release_tts_provider(provider: Optional[str] = None) -> Dict[str, Any]:
|
||||
"""Drop resident local TTS models so their memory is returned.
|
||||
|
||||
With ``provider`` given, only that engine's cache is cleared; otherwise
|
||||
every local engine cache is. Cloud providers hold nothing to release.
|
||||
Returns ``{"released": <number of model instances dropped>}``. The next
|
||||
synthesis simply reloads (or a warm-up does it ahead of time).
|
||||
"""
|
||||
name = (provider or "").lower().strip()
|
||||
released = 0
|
||||
for cache_name, cache in _LOCAL_TTS_MODEL_CACHES.items():
|
||||
if name and cache_name != name:
|
||||
continue
|
||||
released += len(cache)
|
||||
cache.clear()
|
||||
if released:
|
||||
logger.info("[TTS] released %d resident local model(s)", released)
|
||||
return {"released": released}
|
||||
|
||||
|
||||
def acquire_tts_lease(lease: str, tts_config: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
|
||||
"""Register ``lease`` as a live TTS-output consumer and warm the provider.
|
||||
|
||||
``lease`` names the surface/toggle (e.g. ``"desktop:read-aloud"``,
|
||||
``"tui:voice-tts"``). Re-acquiring an existing lease is idempotent (still
|
||||
re-warms — cheap on a cache hit, and heals a cache cleared elsewhere).
|
||||
"""
|
||||
with _tts_lease_lock:
|
||||
_tts_leases.add(lease)
|
||||
holders = len(_tts_leases)
|
||||
result = warm_tts_provider(tts_config)
|
||||
result["leases"] = holders
|
||||
return result
|
||||
|
||||
|
||||
def release_tts_lease(lease: str) -> Dict[str, Any]:
|
||||
"""Drop ``lease``; when it was the last one, unload resident local models.
|
||||
|
||||
Releasing a lease that was never acquired is a no-op (still reports the
|
||||
live holder count) so surfaces can call it unconditionally on their
|
||||
"off" path.
|
||||
"""
|
||||
with _tts_lease_lock:
|
||||
_tts_leases.discard(lease)
|
||||
holders = len(_tts_leases)
|
||||
result: Dict[str, Any] = {"leases": holders, "released": 0}
|
||||
if holders == 0:
|
||||
result["released"] = release_tts_provider()["released"]
|
||||
return result
|
||||
|
||||
|
||||
def tts_lease_holders() -> List[str]:
|
||||
"""Snapshot of live lease names (diagnostics / tests)."""
|
||||
with _tts_lease_lock:
|
||||
return sorted(_tts_leases)
|
||||
|
||||
|
||||
def _reset_tts_leases_for_tests() -> None:
|
||||
with _tts_lease_lock:
|
||||
_tts_leases.clear()
|
||||
|
||||
|
||||
# Module-level cache for Piper voice instances. Voices are keyed on their
|
||||
# absolute .onnx model path so switching voices doesn't invalidate older
|
||||
# cached voices.
|
||||
_piper_voice_cache: Dict[str, Any] = {}
|
||||
_LOCAL_TTS_MODEL_CACHES["piper"] = _piper_voice_cache
|
||||
|
||||
|
||||
def _check_piper_available() -> bool:
|
||||
@@ -2973,15 +3145,16 @@ def _resolve_piper_voice_path(voice: str, download_dir: Path) -> str:
|
||||
return str(cached)
|
||||
|
||||
|
||||
def _generate_piper_tts(text: str, output_path: str, tts_config: Dict[str, Any]) -> str:
|
||||
"""Generate speech using the local Piper engine.
|
||||
def _load_piper_voice_for_config(tts_config: Dict[str, Any]) -> Tuple[Any, Dict[str, Any]]:
|
||||
"""Resolve + load (or fetch from cache) the Piper voice ``tts_config`` selects.
|
||||
|
||||
Loads the voice model once per process (cached by absolute path) and
|
||||
writes a WAV file. Caller is responsible for converting to MP3/Opus
|
||||
via ffmpeg when a different output format is required.
|
||||
Shared by synthesis and :func:`warm_tts_provider` so a warm-up populates
|
||||
exactly the cache slot the next synthesis call will hit — same voice
|
||||
resolution, same download-on-first-use, same cache key.
|
||||
|
||||
Returns ``(voice, piper_config)``.
|
||||
"""
|
||||
PiperVoice = _import_piper()
|
||||
import wave
|
||||
|
||||
piper_config = tts_config.get("piper") or {} if isinstance(tts_config, dict) else {}
|
||||
voice_name = piper_config.get("voice") or DEFAULT_PIPER_VOICE
|
||||
@@ -2991,15 +3164,6 @@ def _generate_piper_tts(text: str, output_path: str, tts_config: Dict[str, Any])
|
||||
|
||||
model_path = _resolve_piper_voice_path(voice_name, download_dir)
|
||||
|
||||
# Tolerant speaker_id parse: drop bad input (non-int strings, lists, dicts)
|
||||
# to 0 (Piper's own default). Booleans are rejected outright — True/False
|
||||
# would silently coerce to 1/0 and hide a config mistake.
|
||||
_raw_speaker = piper_config.get("speaker_id", 0)
|
||||
if isinstance(_raw_speaker, bool) or not isinstance(_raw_speaker, int):
|
||||
speaker_id = 0
|
||||
else:
|
||||
speaker_id = _raw_speaker
|
||||
|
||||
# speaker_id is applied per-call via syn_config.speaker_id — the same
|
||||
# PiperVoice instance serves all speakers, so it stays out of the cache
|
||||
# key. Multi-speaker workflows share one model load.
|
||||
@@ -3012,6 +3176,28 @@ def _generate_piper_tts(text: str, output_path: str, tts_config: Dict[str, Any])
|
||||
return v
|
||||
|
||||
voice = _tts_cache_get_or_load(_piper_voice_cache, cache_key, _load_piper_voice)
|
||||
return voice, piper_config
|
||||
|
||||
|
||||
def _generate_piper_tts(text: str, output_path: str, tts_config: Dict[str, Any]) -> str:
|
||||
"""Generate speech using the local Piper engine.
|
||||
|
||||
Loads the voice model once per process (cached by absolute path) and
|
||||
writes a WAV file. Caller is responsible for converting to MP3/Opus
|
||||
via ffmpeg when a different output format is required.
|
||||
"""
|
||||
import wave
|
||||
|
||||
voice, piper_config = _load_piper_voice_for_config(tts_config)
|
||||
|
||||
# Tolerant speaker_id parse: drop bad input (non-int strings, lists, dicts)
|
||||
# to 0 (Piper's own default). Booleans are rejected outright — True/False
|
||||
# would silently coerce to 1/0 and hide a config mistake.
|
||||
_raw_speaker = piper_config.get("speaker_id", 0)
|
||||
if isinstance(_raw_speaker, bool) or not isinstance(_raw_speaker, int):
|
||||
speaker_id = 0
|
||||
else:
|
||||
speaker_id = _raw_speaker
|
||||
|
||||
# Optional synthesis knobs — only pass a SynthesisConfig when at least
|
||||
# one advanced knob is configured, so we don't depend on a newer Piper
|
||||
@@ -3079,6 +3265,28 @@ def _generate_piper_tts(text: str, output_path: str, tts_config: Dict[str, Any])
|
||||
|
||||
# Module-level cache for KittenTTS model instance
|
||||
_kittentts_model_cache: Dict[str, Any] = {}
|
||||
_LOCAL_TTS_MODEL_CACHES["kittentts"] = _kittentts_model_cache
|
||||
|
||||
|
||||
def _load_kittentts_model_for_config(tts_config: Dict[str, Any]) -> Tuple[Any, Dict[str, Any]]:
|
||||
"""Load (or fetch from cache) the KittenTTS model ``tts_config`` selects.
|
||||
|
||||
Shared by synthesis and :func:`warm_tts_provider` — same model name,
|
||||
same cache key. Returns ``(model, kittentts_config)``.
|
||||
"""
|
||||
KittenTTS = _import_kittentts()
|
||||
kt_config = tts_config.get("kittentts", {}) if isinstance(tts_config, dict) else {}
|
||||
kt_config = kt_config or {}
|
||||
model_name = kt_config.get("model", DEFAULT_KITTENTTS_MODEL)
|
||||
|
||||
def _load_kittentts_model():
|
||||
logger.info("[KittenTTS] Loading model: %s", model_name)
|
||||
m = KittenTTS(model_name)
|
||||
logger.info("[KittenTTS] Model loaded successfully")
|
||||
return m
|
||||
|
||||
model = _tts_cache_get_or_load(_kittentts_model_cache, model_name, _load_kittentts_model)
|
||||
return model, kt_config
|
||||
|
||||
|
||||
def _generate_kittentts(text: str, output_path: str, tts_config: Dict[str, Any]) -> str:
|
||||
@@ -3095,22 +3303,11 @@ def _generate_kittentts(text: str, output_path: str, tts_config: Dict[str, Any])
|
||||
Returns:
|
||||
Path to the saved audio file.
|
||||
"""
|
||||
KittenTTS = _import_kittentts()
|
||||
kt_config = tts_config.get("kittentts", {})
|
||||
model_name = kt_config.get("model", DEFAULT_KITTENTTS_MODEL)
|
||||
model, kt_config = _load_kittentts_model_for_config(tts_config)
|
||||
voice = kt_config.get("voice", DEFAULT_KITTENTTS_VOICE)
|
||||
speed = kt_config.get("speed", 1.0)
|
||||
clean_text = kt_config.get("clean_text", True)
|
||||
|
||||
# Use cached model instance if available
|
||||
def _load_kittentts_model():
|
||||
logger.info("[KittenTTS] Loading model: %s", model_name)
|
||||
m = KittenTTS(model_name)
|
||||
logger.info("[KittenTTS] Model loaded successfully")
|
||||
return m
|
||||
|
||||
model = _tts_cache_get_or_load(_kittentts_model_cache, model_name, _load_kittentts_model)
|
||||
|
||||
# Generate audio (returns numpy array at 24kHz)
|
||||
audio = model.generate(text, voice=voice, speed=speed, clean_text=clean_text)
|
||||
|
||||
|
||||
@@ -16702,6 +16702,30 @@ def _voice_tts_enabled() -> bool:
|
||||
return os.environ.get("HERMES_VOICE_TTS", "").strip() == "1"
|
||||
|
||||
|
||||
def _tts_lease_async(lease: str, active: bool) -> None:
|
||||
"""Acquire/release a TTS engine lease off the RPC thread.
|
||||
|
||||
Speech-output toggles are the signal that TTS is about to be needed (or
|
||||
no longer is). Acquiring warms the configured provider — for local
|
||||
engines that is a model load, possibly a voice download — so it must not
|
||||
block the toggle's RPC reply. Release is cheap but rides the same thread
|
||||
for symmetry. Best-effort: a failure here never affects the toggle.
|
||||
"""
|
||||
|
||||
def _run():
|
||||
try:
|
||||
from tools.tts_tool import acquire_tts_lease, release_tts_lease
|
||||
|
||||
if active:
|
||||
acquire_tts_lease(lease)
|
||||
else:
|
||||
release_tts_lease(lease)
|
||||
except Exception as e:
|
||||
logger.debug("voice: tts lease %s active=%s failed: %s", lease, active, e)
|
||||
|
||||
threading.Thread(target=_run, name=f"tts-lease-{lease}", daemon=True).start()
|
||||
|
||||
|
||||
def _any_session_running() -> bool:
|
||||
"""True while any session's agent turn is in flight.
|
||||
|
||||
@@ -17551,6 +17575,12 @@ def _(rid, params: dict) -> dict:
|
||||
except Exception:
|
||||
stop_hint = ""
|
||||
|
||||
# Voice mode with speech output already on (voice.auto_tts /
|
||||
# prior /voice tts) means replies will be spoken — warm the
|
||||
# engine now rather than on the first reply.
|
||||
if _voice_tts_enabled():
|
||||
_tts_lease_async("tui:voice-tts", True)
|
||||
|
||||
if not enabled:
|
||||
# Disabling the mode must tear the continuous loop down; the
|
||||
# loop holds the microphone and would otherwise keep running.
|
||||
@@ -17567,6 +17597,7 @@ def _(rid, params: dict) -> dict:
|
||||
# and silence any in-flight streaming speech.
|
||||
os.environ["HERMES_VOICE_TTS"] = "0"
|
||||
_tts_stream_stop(user_barge=False)
|
||||
_tts_lease_async("tui:voice-tts", False)
|
||||
|
||||
return _ok(
|
||||
rid,
|
||||
@@ -17586,6 +17617,10 @@ def _(rid, params: dict) -> dict:
|
||||
os.environ["HERMES_VOICE_TTS"] = "1" if new_value else "0"
|
||||
if not new_value:
|
||||
_tts_stream_stop(user_barge=False)
|
||||
# The TTS toggle is the "speech is about to be needed" signal: on →
|
||||
# pre-load the configured engine so the first reply starts hot; off →
|
||||
# release the lease (last holder gone = resident local model freed).
|
||||
_tts_lease_async("tui:voice-tts", new_value)
|
||||
# Include ``record_key`` on every branch so a /voice tts toggle
|
||||
# doesn't reset the TUI's cached shortcut to the default when a
|
||||
# user has a custom binding configured (Copilot review, round 2
|
||||
|
||||
@@ -256,6 +256,17 @@ tts:
|
||||
|
||||
**Advanced knobs** (`tts.piper.length_scale` / `noise_scale` / `noise_w_scale` / `volume` / `normalize_audio`, `use_cuda`) correspond 1:1 to Piper's `SynthesisConfig`. They're ignored on older `piper-tts` versions.
|
||||
|
||||
### Warm-up and unload via speech toggles (local engines)
|
||||
|
||||
Local engines (Piper, KittenTTS) load their model lazily, so without help the *first* spoken reply after you turn speech on pays the whole model load — and on a fresh install the voice download — as silence before the first word. Hermes treats the speech-output toggles as the signal that TTS is about to be needed:
|
||||
|
||||
- **Desktop** — turning on **Read replies aloud**, or starting a **voice conversation**, pre-loads the configured engine in the background right away. Turning both off again unloads the resident model (a Piper voice is tens of MB; KittenTTS up to ~80MB) so it isn't parked in RAM for nothing.
|
||||
- **CLI / TUI** — `/voice tts` (and `/voice on` when `voice.auto_tts` is set) do the same; `/voice off` releases.
|
||||
|
||||
Each toggle holds a *lease* on the engine; the model is only unloaded when the last lease across surfaces is released, so switching off read-aloud in one Desktop window never pulls the voice out from under a conversation running in another. For cloud providers there is no model to hold — the toggle only makes sure a lazily-installed SDK (edge-tts, ElevenLabs, Mistral) is present. Warm-up is best-effort: if the engine can't load, the toggle still succeeds and the first reply falls back to loading on demand as before.
|
||||
|
||||
The Desktop calls `POST /api/audio/tts-lease` with `{"lease": "<name>", "active": true|false}`; other frontends can use the same endpoint.
|
||||
|
||||
### Custom command providers
|
||||
|
||||
If a TTS engine you want isn't natively supported (VoxCPM, MLX-Kokoro, XTTS CLI, a voice-cloning script, anything else that exposes a CLI), you can wire it in as a **command-type provider** without writing any Python. Hermes writes the input text to a temp UTF-8 file, runs your shell command, and reads the audio file the command produced.
|
||||
|
||||
Reference in New Issue
Block a user