Commit Graph

2089 Commits

Author SHA1 Message Date
BenSheridanEdwards 36de3c5c3e fix(stt): retry xAI OAuth after auth rejection 2026-07-28 11:54:01 -07:00
Tobias Safaie geb. Schmidt-Philipp d889c980f5 fix(stt): prefer explicit xAI API key 2026-07-28 11:54:01 -07:00
Teknium 4eadabb8ea test: add BadRequestError to the fake openai module fixture
_transcribe_openai now imports BadRequestError for the container-retry
path; the managed-gateway fake module needs to provide it.
2026-07-28 11:53:36 -07:00
Teknium 0897e0adb8 test: align dispatch tests with provider-scoped validation and named registration errors
Follow-ups for the salvaged wave: the auto-detect legacy-error test now
stubs the split validators, the unknown-command-provider test expects
the new provider_not_registered error, and _transcribe_local tolerates
a null stt.local config section again.
2026-07-28 11:53:36 -07:00
Teknium 9467dc135f fix(stt): lock local model load; allow keyless local OpenAI-compatible STT
Two small fresh fixes on top of the salvage wave:

- Wrap the check-then-load of the module-global faster-whisper model in a
  double-checked threading.Lock so concurrent voice messages can't both
  download/load the model (#24767).
- Treat an empty stt.openai.api_key as no-auth when stt.openai.base_url
  points at a loopback/RFC-1918/.local host, so local OpenAI-compatible
  STT servers (faster-whisper-server, speaches, vLLM whisper) work
  without a sham api_key value. Reimplements the idea from PR #25193 —
  credit @nnnet.

Co-authored-by: nnnet <nnnet@users.noreply.github.com>
2026-07-28 11:53:36 -07:00
Dennis f50a7c307a fix(stt): preprocess .silk voice notes before transcription
Decode WeChat/QQ SILK v3 voice notes to WAV inside transcribe_audio so
any platform that caches a .silk file gets STT for free (same central-
normalization philosophy as the outbound container repair). pilk is
lazy-installed on first use (stt.silk in tools/lazy_deps.py) instead of
being added to the voice extra.

Fixes the inbound half of #32196.

(cherry picked from commit e5db79369d; reworked to compose with the
provider-scoped upload size cap and to lazy-dep pilk)
2026-07-28 11:53:36 -07:00
Zehua Wang 7a56ab2aa2 fix(stt): validate selected voice provider availability 2026-07-28 11:53:36 -07:00
Zehua Wang eaa2dd6d09 fix(stt): check selected provider (not any) + plugin support
PR review feedback:
- Replace _has_any_command_stt_provider() with selected-provider
  check via _resolve_command_stt_provider_config()
- Add _check_plugin_stt_provider() for plugin-registered backends
- Add tests: selected command, unrelated command (should NOT pass),
  and plugin provider path
2026-07-28 11:53:36 -07:00
LauraGPT 7d2b8a3cad fix(stt): anchor qwen asr envelope stripping 2026-07-28 11:53:36 -07:00
LauraGPT d219392e5b fix(stt): anchor Qwen3-ASR envelope stripping 2026-07-28 11:53:36 -07:00
LauraGPT 517b8debbd fix(stt): strip Qwen3-ASR response prefix
Normalize the structured <asr_text> marker after extracting text from string, SDK object, and dictionary transcription responses. Preserve the current provider-aware STT configuration architecture.

Refreshes #8773 on current main.

Co-authored-by: angelos <angelos@oikos.lan.home.malaiwah.com>

Assisted-by: Codex:gpt-5.6
2026-07-28 11:53:36 -07:00
luyifan a5074c0ca8 fix(stt): report unregistered configured providers 2026-07-28 11:53:36 -07:00
Tushar 9e114c5e97 test(tts): add STT fallback regression for _transcribe_mistral (#53259)
Add isolated test where ensure('stt.mistral') raises FeatureUnavailable
but the raw mistralai.client.Mistral import succeeds, verifying the
transcription_tools.py fallthrough path introduced in the same PR.
2026-07-28 11:53:36 -07:00
Tushar a6a439338e fix(tts): fall through to raw import when lazy_deps fails (#53259)
Replace 
aise ImportError(str(e)) with pass in the except Exception
handler of _import_edge_tts(), _import_elevenlabs(), and
_import_mistral_client() so packages installed via PYTHONPATH or Docker
layered filesystems still work when lazy_deps.ensure() raises.

Also fix the Mistral STT path in transcription_tools.py which only
caught ImportError, not FeatureUnavailable.

Adds 6 regression tests using sys.modules fixtures (no
builtins.__import__ patching).
2026-07-28 11:53:36 -07:00
Charles Cha ffc3ce27d5 fix(stt): scope upload size limits to remote providers 2026-07-28 11:53:36 -07:00
AnthonyAssistantAi 884900ffd6 fix: avoid local STT crash on Apple Silicon
Force CPU (int8) for faster-whisper on Apple Silicon / Rosetta, where
ctranslate2's device=auto path can hard-abort in native code. Salvaged
from PR #28624 without the numpy pin change (main already moved on).

(cherry picked from commit 7edf2d5196, pyproject.toml hunk dropped)
2026-07-28 11:53:36 -07:00
liuhao1024 766e856118 fix(stt): treat CUBLAS_STATUS_NOT_SUPPORTED as CUDA lib error
- Add Blackwell-specific cuBLAS error marker to _CUDA_LIB_ERROR_MARKERS
- Allows CPU fallback on RTX 5090 (sm_120) when faster-whisper
  reports CUBLAS_STATUS_NOT_SUPPORTED instead of loading successfully
- Add regression test for CUBLAS_STATUS_NOT_SUPPORTED path

Closes #17526
2026-07-28 11:53:36 -07:00
Tranquil-Flow 06fc6e0c29 fix(stt): respect device and compute_type from config.yaml
The local STT transcription function hardcoded device="auto" and
compute_type="auto" when instantiating WhisperModel, ignoring the
user's stt.local.device and stt.local.compute_type config values.

Closes #8319
2026-07-28 11:53:36 -07:00
Teknium c136400c9e fix(voice): single scoped resolver — STT/TTS keys fall back to the credential pool
Rework of #68509 per triage: hoist the duplicated per-tool
_resolve_provider_key helpers into one owner,
tools.tool_backend_helpers.resolve_provider_secret(), and migrate every
STT/TTS key lookup site to it.

Resolution order: explicit config.yaml value > profile secret scope /
env / ~/.hermes/.env > credential pool (checks both '<provider>' and
'custom:<provider>' pool keys, so keys added via 'hermes auth add
mistral' or declared under providers.<name> both resolve). Under an
active multiplex turn the profile scope stays authoritative — no pool
or .env fallback that could borrow another profile's key (composes with
the #69469 scope fix).

Coverage now includes GROQ_API_KEY, MISTRAL_API_KEY, ELEVENLABS_API_KEY,
DEEPINFRA_API_KEY, MINIMAX_API_KEY, GEMINI_API_KEY/GOOGLE_API_KEY, the
XAI_API_KEY fallback in resolve_xai_http_credentials, and the OpenAI
audio key (resolve_openai_audio_api_key now pool-aware for
OPENAI_API_KEY via 'hermes auth add openai-api').

Unit tests: fake pool entry proves each provider resolves from the pool
when env is empty; env still wins when set; config wins over both; a
multiplex scope miss never borrows the pool; pool read failures never
raise; tool-level wiring for STT, TTS, xAI, and OpenAI audio.

Fixes #68003
2026-07-28 11:53:09 -07:00
Drexuxux ed591a5664 fix(voice): resolve the TTS/STT OpenAI key under the profile secret scope
`resolve_openai_audio_api_key()` reads the key that authenticates the audio
client straight from the process environment:

    return (
        os.getenv("VOICE_TOOLS_OPENAI_KEY", "")
        or os.getenv("OPENAI_API_KEY", "")
    ).strip()

That value is not advisory. It flows through
`_resolve_openai_audio_client_config()` into `OpenAIClient(api_key=...)` for
TTS, and through `transcription_tools` for voice-note STT — both on the
per-turn tool path, inside the profile secret scope the gateway installs.

`agent/vertex_adapter` states the contract this breaks:

    in a multiplex gateway serving several profiles from one process,
    os.environ reflects whichever profile's .env happened to be loaded at
    boot, not the profile the current turn belongs to. Reading it directly
    here would let one profile mint tokens from — and get billed against —
    a different profile's service-account file.

Reproduced with the real resolver, multiplexing on and profile A's scope
installed:

    scope-aware get_secret  -> sk-PROFILE-A-key
    voice/STT resolver      -> sk-PROFILE-B-key

So profile A's spoken reply and its users' voice notes are sent to OpenAI on
profile B's account, and billed there.

Route both reads through `agent.secret_scope.get_secret`, the same fix already
merged for the WeChat send path (#59662) and pending for QQ (#60420) — neither
covers the audio credential family. Under multiplexing the scope stays
authoritative, so a scope miss now yields no key instead of borrowing another
profile's; with multiplexing off `get_secret` falls through to `os.environ`
exactly as before, so single-profile deployments are untouched. The
VOICE_TOOLS_OPENAI_KEY > OPENAI_API_KEY precedence is unchanged.

Deliberately narrow: `fal_key_is_configured()` and
`has_direct_modal_credentials()` in this file are presence checks, not
authentication, and the former is already being reworked in open PR #20929.

tests/tools/test_tool_backend_helpers.py: the scope wins over another
profile's `os.environ`; a scope miss does not borrow another profile's key;
voice-key precedence holds inside a scope; and a control proves the
single-profile path still reads `os.environ`. The three isolation tests fail
on main; the control passes there. 320 passed across the helper, secret-scope,
and consumer suites (the fluctuating voice_mode/voice_cli failures are
pre-existing PulseAudio/ordering artifacts — the differing test passes 3/3 in
isolation on both main and this branch).
2026-07-28 11:53:09 -07:00
Teknium d464ae3652 feat(cron): user-owned model pins + cron.model fleet default
Per-job cron inference pins are now user-owned: the agent-facing cronjob
tool schema no longer exposes model/provider/base_url, and the registered
handler ignores them even if a model hallucinates the old parameters.
Users set pins via the dashboard, hermes cron create/edit --model/--provider,
or jobs.json directly — and once set, a pin sticks until the user changes it.
Existing agent-era pins are grandfathered untouched.

New cron.model / cron.model_provider config keys give the cron fleet its
own default model, independent of the chat model. Fire-time resolution:
per-job pin > cron.model > HERMES_MODEL > model.default. An axis covered
by the explicit cron-fleet default is deliberate routing, not drift, so
the #44585 fail-closed guard skips it — switching your chat model with
/model or hermes model no longer breaks unpinned cron fleets.

- tools/cronjob_tools.py: drop model param from agent schema + handler;
  remove now-dead _resolve_model_override
- cron/scheduler.py: cron.model/model_provider resolution + per-axis
  drift-guard skip
- cron/jobs.py: snapshot resolution mirrors the new precedence
- hermes_cli/subcommands/cron.py + hermes_cli/cron.py: --model/--provider
  on hermes cron create/edit
- hermes_cli/config.py: cron.model / cron.model_provider defaults
- docs: cron.md model-resolution tip rewritten
2026-07-28 11:52:47 -07:00
Teknium 156edc5ded refactor: extract shared audio container sniffer to tools/audio_container.py
One sniffer owns magic-byte container detection (Teknium's one-concept-
one-owner rule): the new tools/audio_container.py is used by

- gateway/platforms/base.py _sniff_audio_ext (inbound cache — PR #36166's
  central sniffer, now covering AAC/ADTS, MP4-brand disambiguation, webm)
- gateway/platforms/signal.py _guess_extension (audio/AV branches
  delegated; RIFF/WAVE fix from PR #50690 and M4A-brand fix from
  PR #72490 now live centrally)
- tools/tts_tool.py _sniff_audio_container (outbound repair, PR #73072)

cache_audio_from_url inherits the sniff via cache_audio_from_bytes.
Adds tests/tools/test_audio_container.py covering every magic-byte type,
wrong-extension repair on the inbound cache, unknown passthrough, the
URL path, and Signal's delegation.
2026-07-28 11:52:44 -07:00
luyifan 4ae27548d6 fix(media): recognize m2a audio attachments 2026-07-28 11:52:44 -07:00
Elie BRUNO 971cb9b886 fix(stt): accept .oga and .opus voice notes for transcription
Telegram sends voice notes as .oga (OGG/Opus). SUPPORTED_FORMATS listed
.ogg but not .oga, so transcribe_audio rejected every Telegram voice note
with "Unsupported format: .oga" before reaching any STT backend. Add .oga
and .opus to the allowlist, with a regression test.
2026-07-28 11:52:44 -07:00
Teknium f106e0ebc2 fix(wake): raise default sensitivity to 0.6 and fix inverted Porcupine direction
'hey hor' triggered the wake word: the default sensitivity was 0.5, which
for openWakeWord IS the raw per-frame score threshold — openWakeWord's own
permissive baseline that near-misses clear. Raised the default to 0.6 so
phonetic near-misses fall short while real 'hey hermes' (typically 0.9+)
still fires easily.

Also fixed a real cross-engine inconsistency found while checking: the
sensitivity knob is documented 'higher = stricter' and behaves that way
for openWakeWord (threshold = sensitivity) and sherpa (0.05 + 0.4*s), but
Porcupine's own 'sensitivities' param runs the opposite way (higher = MORE
false alarms, per Picovoice). Turning sensitivity up made Porcupine looser
— backwards. Now inverted (1 - sensitivity) so 'higher = stricter' holds
for every engine.

- tools/wake_word.py: default 0.6; _sensitivity fallback uses _DEFAULTS;
  Porcupine sensitivity inverted with rationale
- hermes_cli/config.py + docs: default + consistent-direction note
- tests: Porcupine inversion, default>=0.6 regression, fallback-to-default
2026-07-28 09:43:17 -07:00
Alex Fournier b1a5d67e71 Merge upstream main into fix/hermes-relay-anthropic-context
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-28 08:10:58 -07:00
Teknium 30ed3f82bd fix(wake): reject ambient-speech false triggers with consecutive-frame confirmation
openWakeWord scores one ~80ms frame at a time and the detector fired the
instant a SINGLE frame crossed threshold — so a stray phoneme in background
conversation could trigger the wake word unintentionally (reported in
testing). A real utterance of the phrase holds a high score across several
consecutive frames; an ambient blip spikes just one.

_OpenWakeWordEngine now requires N consecutive over-threshold frames
(wake_word.confirmation_frames, default 3) before firing. The streak resets
on any sub-threshold frame and on engine reset() (pause/resume), so a
pre-pause frame can't count toward a post-resume fire. confirmation_frames=1
restores the old single-frame behavior; clamped 1..10.

Only openWakeWord is affected — sherpa (streaming transducer) and porcupine
decode the whole phrase internally and already reject single-frame spikes.

- tools/wake_word.py: _confirmation_frames() accessor, streak logic in
  process()/reset(), config default
- hermes_cli/config.py: wake_word.confirmation_frames documented default
- tests: 5 new (spike rejected, sustained fires once, =1 legacy behavior,
  reset clears streak, config clamp) — 58 wake tests green
- docs: 'Reducing false triggers on ambient speech' section
2026-07-28 07:59:35 -07:00
Teknium 913aa7709b test(wake): pin tflite runtime in artifact-selection test; merge tflite_ok into available gate
Follow-up to benbarclay's macOS ARM64 tflite fix:
- test_openwakeword_bundled_model_matches_framework stubs
  ensure_tflite_runtime()=True so it exercises artifact selection, not
  runtime availability — off-Darwin the bridge legitimately returns
  False and the engine falls back to onnx (that path is covered by its
  own tests). Was failing on Linux CI otherwise.
- check_wake_word_requirements now ANDs both this branch's STT/TTS gate
  and Ben's tflite_ok into 'available' (cherry-pick conflict resolution).
2026-07-28 07:59:34 -07:00
Ben Barclay 7d19033d2e fix(wake): run openWakeWord on tflite on macOS ARM64
openWakeWord's ONNX backend returns near-zero scores on Apple Silicon
(dscripka/openWakeWord#336), so "Hey Hermes" never crossed the 0.5
threshold: the listener armed, the microphone worked, and nothing fired.

Bisecting the pipeline puts the fault in exactly one stage — feeding the
same audio through both backends, the melspectrogram front-end is
bit-identical (maxdiff 0.00000) and the wake classifier agrees on
identical features, while the shared embedding model diverges by 45.44.
Cross-feeding confirms it: tflite features scored through the *onnx*
classifier give 0.9948 vs 0.000009 for onnx features. A telling
secondary symptom is that scores fall as input gets louder (0.5x ->
0.00031, 8x -> 0.000066), which is garbage inference rather than a weak
detection.

Selecting tflite in config alone does not fix it. openWakeWord hardcodes
`import tflite_runtime.interpreter` but declares tflite-runtime for
`platform_system == "Linux"` only; on macOS the equivalent wheel is
ai-edge-litert, so that import always fails and model.py silently
downgrades back to onnx. The result is a detector that reports itself
listening and can never fire.

- default the backend per platform (tflite on macOS ARM64, onnx
  elsewhere) instead of hardcoding onnx, and pick the matching bundled
  model artifact
- bridge tflite_runtime -> ai_edge_litert through sys.modules, in-process,
  with no writes to site-packages
- refuse the silent onnx downgrade on macOS ARM64 and report the missing
  runtime through check_wake_word_requirements() so the GUI surfaces an
  actionable hint rather than arming a dead ear
- lazy-install ai-edge-litert via its own feature key, because lazy-dep
  specs cannot carry PEP 508 markers (_spec_is_safe rejects ";")

An explicit `inference_framework` in config still wins, so anyone pinning
a backend keeps it.

Verified on macOS 26.5.2 / M-series: "hey hermes" scores 0.0005 on onnx
and 0.9423 on tflite from the same clip, with cross-phrase controls at
0.0003. Live over-the-air through the real microphone fires 4/4
utterances (peak 0.9532).
2026-07-28 07:59:34 -07:00
Teknium a832139ba3 feat(wake): eager-install voice deps with the desktop; wake probes never run pip
Two fixes from live testing (Teknium):

1. Desktop installs now ship the wake/voice stacks up front.
   install.sh + install.ps1 desktop stages run 'uv pip install
   -e .[wake,voice]' (best-effort, lazy-install remains the fallback)
   before building the app, so the first ear-click arms instantly
   instead of sitting through a multi-minute onnxruntime download.
   CLI-only installs keep the lazy path — [all] curation unchanged.

2. The vanished ear: the STT/TTS gate made wake.status call
   check_tts_requirements(), whose edge path runs _import_edge_tts →
   lazy_deps.ensure — a synchronous PIP INSTALL inside a status poll.
   On a venv without edge-tts that blew the desktop's 30s RPC timeout,
   armWakeWord caught the error, the atom never learned enabled=true,
   and the ear unmounted. _tts_ready is now a pure probe: deps missing
   + lazy installs allowed counts as ready (installs at first speak)
   WITHOUT touching pip; check_tts_requirements only runs once deps
   are present. Regression test asserts the probe never calls it while
   deps are missing.
2026-07-28 07:59:34 -07:00
Teknium f03bb2b4ef feat(wake): gate arming on STT + TTS readiness
The wake loop is wake → record → STT → agent → TTS. Arming without
either end configured gives a mic that hears you and then does nothing
perceivable — a useless experience. check_wake_word_requirements now
probes both (same probes /voice uses: stt.enabled + provider != none;
check_tts_requirements) and refuses with a pointer to `hermes tools`
naming exactly which half is missing. The desktop ear hides (available:
false already hides the button), /wake on prints the hint on CLI/TUI.

wake.start also validates requirements BEFORE persisting
wake_word.enabled, so a refused gesture can't leave config claiming on
while nothing can ever arm.

Tests: per-half and both-missing hint assertions; existing requirements
tests pinned via _voice_loop_ready so they don't depend on the test
venv's installed voice stack. E2E: stt.enabled=false in a real config
-> unavailable with the speech-to-text hint.
2026-07-28 07:59:34 -07:00
Teknium 514dd59cad fix(wake): detect dead-mic streams, stop the ear freezing during first-use install
Internal testing (macOS): clicking the ear froze the button for ~30s,
then it went blue but 'hey hermes' never fired even though STT worked.

Two distinct bugs:

1. Frozen-then-timeout button: first-use wake.start lazy-installs the
   detection engine (onnxruntime is a large wheel), which blows the
   desktop's default 30s WS request timeout — the RPC 'failed' client-
   side while the backend kept installing and armed later on its own.
   wake.start now gets a 180s budget and the pending state says
   'arming — first use may take a minute while the engine installs'
   instead of a silent disabled button.

2. Armed but deaf: macOS grants mic permission per PROCESS. The
   renderer having mic access (STT working) does not grant the Python
   backend anything — CoreAudio hands an unentitled process a 'working'
   stream that delivers zeros forever, so the listener looks healthy
   and can never hear the phrase. The detector now tracks frame peaks:
   10s of consecutive near-zero frames sets audio_silent, logged with
   the exact macOS Settings path, cleared automatically when audio
   appears. Surfaced everywhere: wake.status (audio_silent + hint),
   desktop ear tooltip (kept visible while listening), /wake status on
   TUI (⚠ line) and classic CLI, plus a docs troubleshooting section.

Tests: detector silent-flag set/recover cycle (fake silent/loud
streams), desktop tooltip keeps the dead-mic hint while listening.
44 wake Python tests, 22 desktop + 14 TUI vitest green.
2026-07-28 07:59:34 -07:00
Teknium 7a87c6ffd6 fix(wake): make the lazy-install path reachable on fresh installs
check_wake_word_requirements() gated 'available' on the audio probe,
but the probe imports sounddevice + numpy — two of the packages the
lazy installer would install. On a fresh machine deps_ok was False, so
audio_ok was always False and /wake on printed the manual pip hint and
bailed before the engine constructors' lazy_deps.ensure() could run.

Now the audio probe only runs once deps are installed; with deps
missing and lazy installs allowed (the default), /wake on proceeds and
ensure() installs the pinned engine deps in-process — no restart. The
manual pip hint remains for security.allow_lazy_installs=false, and a
mic hint still blocks when deps are present but no audio device works.
The CLI announces the one-time engine install so the pause is explained.
2026-07-28 07:59:34 -07:00
Hermes Agent 625d39632a test(wake): stub numpy in the fake-sherpa fixture for hermetic CI
numpy is a lazy voice-extra dep absent from CI's hermetic env; the two
engine process() tests imported it for real. Stub asarray/float32 in
the fixture (verified against a blocked-numpy import, matching CI).
2026-07-28 07:59:34 -07:00
Hermes Agent 2a35c8f0b8 feat(voice): route wake phrases to their profile — "hey <profile>" wakes that profile
One sherpa listener now enrolls every wake-enabled profile's phrase
(defaulting to "hey <profile name>") and reports WHICH phrase fired.
wake.detected gains a profile field; the desktop live-switches to the
matching profile (same path as the profile rail), opens a fresh session
there, and starts hands-free voice. The single-profile CLI/TUI print the
hermes -p switch command for foreign-profile phrases instead of
answering as the wrong profile. Opt out per listener with
wake_word.profile_routing: false.
2026-07-28 07:59:34 -07:00
Hermes Agent 0ae305ed4e feat(voice): open-vocabulary wake phrases via sherpa-onnx KWS
New "sherpa" wake_word provider: the configured phrase is BPE-tokenized
at runtime against a small streaming zipformer KWS model (~13 MB English,
one-time download cached under HERMES_HOME), so ANY typed phrase works
with zero training — including per-profile phrases like "hey coder".

wake.sherpa lazy-dep group + [wake] extra grow sherpa-onnx/sentencepiece;
requirements probe routes per provider; sensitivity maps onto sherpa
keywords_threshold. E2E-verified on real audio (target phrase fires,
foreign phrase stays silent, reset drops buffered state).
2026-07-28 07:58:16 -07:00
Brooklyn Nicholson 9f84bc30bd feat(voice): bundle the trained "hey hermes" model as the out-of-the-box default
From #53378: ships hey_hermes.onnx/.tflite (openWakeWord pipeline,
Apache-2.0) under tools/wakewords/, resolves the default (and hey_hermes
aliases) to the bundled file, ensures openWakeWord base feature models
are fetched for custom paths too, and updates config defaults + docs
from hey_jarvis to hey hermes.
2026-07-28 07:58:16 -07:00
Omid Saadat 5839aad13d fix(wake-word): enforce single-owner lifecycle 2026-07-28 07:58:01 -07:00
Brooklyn Nicholson edb12bf423 fix(voice): stop wake re-fire loop and empty-transcript error spam
Two bugs surfaced by the desktop wake conversation:

1. Runaway loop: wake -> voice -> resume -> wake fired again within
   ~200ms. openWakeWord keeps its rolling feature buffer across
   pause/resume, so on resume it immediately re-scored the "hey jarvis"
   captured before the pause and re-fired, reopening a session and
   restarting voice in a tight cycle. Reset the engine buffer on every
   detector (re)start so resume begins from clean audio.

2. Empty-transcript toast: a silent re-listen returns
   success:false / "… STT returned empty transcript", which the desktop
   transcribe endpoint turned into a 400 -> thrown error -> "Voice
   transcription failed" notification on every silent gap. Treat an empty
   transcript as no-speech: return {ok, transcript: ""} so the voice loop
   quietly re-listens. Real failures still 4xx/5xx.
2026-07-28 07:58:01 -07:00
Brooklyn Nicholson 86d5b8b90f feat(voice): extend "Hey Hermes" wake word to TUI + desktop GUI
Makes the wake word a tri-surface feature with one configurable owner.

- wake_word.surface ("auto" | "cli" | "tui" | "gui") + shared
  wake_surface_enabled() gate consulted by every surface, so exactly one
  place owns the listener and the new session it opens.
- tui_gateway: wake.start/stop/pause/resume/status RPCs + a wake.detected
  event, sharing one server-side detector for both TUI and desktop. The
  detector yields the mic to voice.record (pause on capture start, resume
  on terminal) and to the desktop's browser mic (wake.pause/resume).
- TUI (Ink): arm wake.start on gateway.ready; on wake.detected open a
  fresh session and start voice capture.
- Desktop (Electron): arm wake.start on connect; on wake.detected open a
  fresh session.
- CLI now gates on wake_surface_enabled("cli"); /wake status shows surface.
- Tests for the surface gate; docs cover the surface knob + cross-surface.
2026-07-28 07:56:45 -07:00
Brooklyn Nicholson 5f43452e91 feat(voice): add "Hey Hermes" wake word to start a hands-free session
Adds an opt-in, on-device hotword listener for the CLI. With
wake_word.enabled (or /wake on), Hermes listens in the background for a
wake phrase; on detection it starts a fresh session, captures one
utterance through the existing voice pipeline, and answers — the
"Hey Siri" pattern.

- tools/wake_word.py: provider-pluggable detector (openWakeWord, free
  local default; Porcupine, premium) over the shared 16 kHz sounddevice
  capture path. Background daemon thread with pause/resume so it yields
  the mic during a voice turn.
- CLI wiring: startup listener (off-thread), on-wake flow, an idle
  watchdog that resumes the detector after each turn, cleanup hook, and
  a /wake [on|off|status] command.
- config.yaml wake_word section; PORCUPINE_ACCESS_KEY as an optional
  secret. Engines lazy-install via the [wake] extra.
- Hands a transcript to the input queue exactly like voice mode, so no
  system-prompt/cache mutation. No new core model tool.
- Tests (mocked, no live audio/network) + feature docs.
2026-07-28 07:56:45 -07:00
Victor Kyriazakos 6a174e9967 Merge origin/main into feat/gateway-health-diagnostics
# Conflicts:
#	cron/executions.py
#	cron/jobs.py
2026-07-28 13:09:17 +00:00
Booker 1dfe781edd fix(skills): avoid redundant bind-mount scans (#72622)
Skip hashing active copies when the bundled origin is unchanged, build rename and optional migration indexes lazily, and reuse one optional-skill directory index per sync.

Add regression coverage for unchanged copies, deferred modification detection, lazy rename recovery, and optional provenance scans.
2026-07-28 04:24:50 -05:00
Brooklyn Nicholson 1eb5ee1eaa fix(mcp): make Figma remote OAuth work via DCR allowlist defaults
Figma's mcp.figma.com register endpoint is a client_name allowlist
(Claude Code / Codex succeed; Hermes Agent 403s) and returns a client
secret while advertising auth_method=none, then requires the secret on
token exchange. Auto-set client_name + client_secret_post for Figma
hosts, pass oauth cfg through login/add paths, force interactive OAuth
for hermes mcp login from non-TTY desktop shells, and ship a catalog
entry. Proven: hermes mcp login figma → 26 tools.
2026-07-28 00:53:16 -05:00
Teknium fab4c888ae feat(voice): say 'stop' to end a voice chat hands-free
Saying EXACTLY a configured stop phrase (default: 'stop') and nothing
else now ends the voice conversation instead of being sent to the agent
as a prompt. Match is deliberately strict — whole utterance,
case-insensitive, surrounding punctuation stripped — so 'stop doing
that and try X' still reaches the agent.

- tools/voice_mode.py: is_voice_stop_phrase() + voice.stop_phrases
  config loader (default ['stop'], [] disables, malformed config falls
  back safely).
- hermes_cli/voice.py: shared continuous loop (TUI + desktop) halts on
  a stop phrase exactly like the silent-cycle limit (fires
  on_silent_limit so every UI turns voice off); stop_continuous
  force-transcribe path swallows the phrase without counting a silent
  cycle.
- cli.py: classic CLI push-to-talk/continuous path and barge-in
  utterance path disable voice mode on a stop phrase.
- Config default voice.stop_phrases: ['stop']; docs updated.

Tests: tests/tools/test_voice_stop_phrase.py (27 tests,
sabotage-verified: loop test fails when detection is disabled);
voice suites green (110 + 44 passed).
2026-07-27 21:26:23 -07:00
Teknium bc997a36a8 feat(stt): default global stt.language to 'en'
Whisper auto-detection frequently misidentifies short/accented clips,
which users experience as voice notes transcribed in the wrong language
(Teknium + CTO both hit this). The unified resolver from #73067 made a
global hint possible; this makes it the DEFAULT so stock installs stop
guessing. Non-English users set stt.language once; '' restores
auto-detect for multilingual use.

Deep-merge gives existing configs the new default automatically (no
_config_version bump needed); any explicit per-provider or global
language setting still wins.
2026-07-27 21:26:20 -07:00
Alex Fournier 14bed44c8c Reapply "feat(observability): integrate NeMo Relay runtime and shared metrics"
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 21:10:51 -07:00
Teknium fae29c8410 fix(tts): class-level .ogg container repair + multi-platform opus voice detection
Root-cause fix for the 'TTS voice bubble broken' issue family (#57048,
#54589, #57213, #58845, #14841, #45557, #57049). Two class-level defects:

1. Several backends silently write MP3/WAV bytes into a .ogg output path
   (Edge only emits MP3, Piper writes WAV, xAI writes MP3, some
   OpenAI-compatible servers ignore response_format=opus). Platforms that
   need real Ogg/Opus render 0-second/broken voice bubbles. Instead of
   per-provider patches, text_to_speech_tool now sniffs magic bytes once
   after synthesis (_sniff_audio_container) and repairs the container
   centrally (_repair_ogg_container): ffmpeg transcode in place, or rename
   to the honest extension when ffmpeg is unavailable. Covers every
   current and future provider, including command providers and plugins.

2. want_opus only recognized Telegram, so Matrix/Feishu/WhatsApp/Signal
   auto-TTS voice replies were synthesized as MP3 and delivered as broken
   attachments. New OPUS_VOICE_PLATFORMS set covers all voice-bubble
   platforms.

_convert_to_opus refactored onto a shared _ffmpeg_transcode_to_opus that
supports safe in-place transcodes (-f ogg forced muxer, temp file +
os.replace).

Tests: tests/tools/test_tts_container_repair.py (13 tests incl. a live
ffmpeg round-trip); full TTS suite green (153 + 187 passed).
2026-07-27 20:28:35 -07:00
Teknium a10bd49ddd feat(stt): unify language resolution across all STT providers
Class-level fix for the 'STT transcribes the wrong language' issue family
(#55551, #50181 and siblings). Previously language handling was per-provider
chaos: local honoured stt.local.language, Groq/OpenAI/Mistral/DeepInfra sent
no language hint at all, xAI silently forced 'en', ElevenLabs used its own
language_code key, and there was no global setting.

- New _resolve_stt_language() helper: stt.<provider>.language >
  stt.language (new global key) > HERMES_LOCAL_STT_LANGUAGE > auto-detect.
- Threaded through ALL providers: local, local_command, groq, openai,
  mistral, xai, elevenlabs, deepinfra (shared OpenAI handler), command
  providers, and plugin dispatch.
- xAI no longer forces English when nothing is configured (auto-detect).
- Mistral Voxtral now receives a language hint when configured.
- stt.groq.model is now honoured from config (previously env-only).
- DEFAULT_CONFIG gains stt.language, stt.groq, stt.xai, stt.mistral.language.
- Tests: tests/tools/test_stt_language_resolution.py (11 tests, sabotage-
  verified) + full transcription suite green (236 passed).

Builds on cherry-picked contributor work from #19786 (@zombopanda),
#23161 (@materemias), #50684 (@BlackishGreen33).
2026-07-27 20:28:31 -07:00
墨綠BG 7e75752516 ✅ test(stt): isolate null config from language env 2026-07-27 20:28:31 -07:00