* feat(wake): client-capture wake word for remote desktop
Remote headless backends have no PortAudio mic, so "hey hermes" fails even
when openWakeWord is installed. Let the desktop stream 16 kHz int16 PCM via
wake.feed while detection stays server-side.
- wake_word.capture: auto|local|client (+ GUI client_capture prefer)
- WakeWordDetector external_audio queue + feed_audio API
- wake.feed RPC; wake.start/status report capture + frame_length
- Desktop getUserMedia feeder; stop on wake.detected, re-arm after voice
- Docs + unit tests (26 pass in tests/tools/test_wake_word.py)
* fix(wake): address review on client-capture re-arm and feed queue
- wake.status reports effective capture from the armed detector (client vs
local), plus frame_length/sample_rate; GUI status probes prefer client
- Gateway test doubles accept external_audio on start_listening
- Desktop PCM feeder uses a bounded ordered queue instead of dropping frames
while a wake.feed RPC is in flight
- /wake on and status/re-arm paths pass client_capture so remote reattach works
* fix(wake): auto capture keeps the backend mic when one exists
With capture:auto the desktop always preferred client streaming, so a local
desktop with a working backend mic silently switched from PortAudio to
getUserMedia default-device — dropping wake_word.input_device selection
(#74363). A ready backend input now wins under auto; client capture is the
fallback for a preferring surface on a mic-less backend, and capture:client
still forces streaming.
Also removes the dead auto branch (both arms returned local) and lets the
client-feed test skip cleanly when numpy is absent.
* perf(desktop): coalesce wake.feed frames
Sending one 80 ms frame per RPC is ~12.5 gateway calls/s for as long as the
ear is armed. Drain up to 4 queued frames into a single wake.feed payload
(backend feed() already splits long buffers into engine frames) — ~3 RPCs/s
steady-state. Fix the wake.feed size-cap comment (64000 bytes = 2 s, not
0.5 s).
* docs(config): document wake_word.capture in cli-config.yaml.example
---------
Co-authored-by: Andrew <drew@kainotomic.com>
PDFs were classified as generic binary/text previews, rendering raw %PDF
bytes locally and failing entirely for remote-only files. Classify PDFs as
their own preview kind, load bytes through the existing local/remote
filesystem bridge, convert them to revocable Blob URLs for Chromium's
embedded viewer, migrate persisted pre-PDF tabs at restore, and retry
restored previews when the active filesystem connection changes.
Salvaged from #76008-era base onto current main: PDF classification now
composes with the remote-HTML enrichment branch, and the persisted-tab
migration runs before the One-Browser URL rekey in decodePreviewTabs.
Supersedes #76565.
Co-authored-by: Brooklyn Nicholson <brooklyn@brooklyn.sh>
Sending one 80 ms frame per RPC is ~12.5 gateway calls/s for as long as the
ear is armed. Drain up to 4 queued frames into a single wake.feed payload
(backend feed() already splits long buffers into engine frames) — ~3 RPCs/s
steady-state. Fix the wake.feed size-cap comment (64000 bytes = 2 s, not
0.5 s).
- wake.status reports effective capture from the armed detector (client vs
local), plus frame_length/sample_rate; GUI status probes prefer client
- Gateway test doubles accept external_audio on start_listening
- Desktop PCM feeder uses a bounded ordered queue instead of dropping frames
while a wake.feed RPC is in flight
- /wake on and status/re-arm paths pass client_capture so remote reattach works
Remote headless backends have no PortAudio mic, so "hey hermes" fails even
when openWakeWord is installed. Let the desktop stream 16 kHz int16 PCM via
wake.feed while detection stays server-side.
- wake_word.capture: auto|local|client (+ GUI client_capture prefer)
- WakeWordDetector external_audio queue + feed_audio API
- wake.feed RPC; wake.start/status report capture + frame_length
- Desktop getUserMedia feeder; stop on wake.detected, re-arm after voice
- Docs + unit tests (26 pass in tests/tools/test_wake_word.py)
messageRenderWeight moves out of thread/list.tsx into lib/render-weight.ts.
The DOM page budget already spends render cost rather than message count —
the store window added next needs the same currency, and one weight function
keeps the two layers from drifting apart.
No behavior change.
Only skip/graft structure for the current live assistant (stream id,
pending, or after the latest user), not completed historical tool rows.
Require live-tail identity for same-turn structure carry. Align journal
overlay with strict answer-text extension.
Addresses review + CI on #76744.
Skip pure-text inflight.assistant projections when the transcript already
has reasoning/tool-call structure, and only overlay journal answer text
on strict extension.
Fixes#76444
Follow-up to #77653: textWithoutReferenceLines built a fresh /g RegExp per call and hand-managed lastIndex — but it runs on both sides of every message comparison in the reconcile loops. An anchored non-global regex has no shared-lastIndex hazard and can be hoisted to module scope. Also removes textWithoutImageRefs, whose last production consumer #77653 replaced (kept IMAGE_REF_LINE_RE for extractImageRefs), and retargets its now-stale comment.
Re-derive of PR #38470 on today's main (its target file desktop-controller.tsx no longer exists after the contrib/ refactor; the three surviving ideas are applied at their new homes):
- incremental-external-store-runtime: the dep-less setAdapter effect ran every render; gate on [runtime, store] — behavior-preserving because __internal_setAdapter early-exits on identical store.
- ChatView is now memo()d, and session-tile hoists its inline arrow props to useCallbacks/module constants so the memo actually holds.
- Render-count regression test (mocked Thread) proves an unrelated parent re-render no longer re-renders the chat shell.
Credit: idea and original implementation by @hdd69 in #38470.
All three desktop reconnect loops (primary gateway boot, secondary
multi-profile gateway pool, plugin event socket) used bare exponential
backoff with no jitter. After a gateway restart every disconnected
client redials on the exact same schedule, so the reconnect attempts
land in lockstep instead of spreading out -- a burst that can starve
the gateway's file descriptors while it's still coming back up.
Add reconnect-backoff.ts implementing AWS-style full-jitter backoff
(random delay in [0, min(cap, base * 2^attempt))) and wire it into all
three call sites in place of their local Math.min/2**attempt math.
Manual reconnect paths already reset the attempt counter and bypass
the timer entirely -- unchanged.
Same gesture the pane tabs already carry — isMetaClose hoists from
pane-tab.tsx into lib/middle-click.ts beside its sibling middle-click
gesture, so the two surfaces share one predicate instead of drifting.
Users pasting a Tailscale IP or LAN host as 'host:port' (no http://) hit
either a hard 'URL is not valid' error in the main process or, worse, a
silent dead probe in the renderer: the ^https?:// gates in the settings
and first-run forms never fired, so the field sat idle with no feedback.
- normalizeRemoteBaseUrl() (electron/connection-config.ts) now prepends
http:// when the input has no scheme:// prefix; explicit non-http
schemes (ws://, ftp://) still reach the protocol check and get a clear
rejection.
- New renderer twin coerceRemoteUrlScheme() (src/lib/remote-url.ts),
wired into both probe gates (gateway-settings.tsx and
first-run-remote-form.tsx) so the debounced /api/status probe, sign-in,
test, and save all see the coerced URL.
- Tests for both sides (electron/connection-config.test.ts,
src/lib/remote-url.test.ts).
Chromium on Windows and Linux answers a middle press inside a scroller by
starting the autoscroll pan, and the mouseup that ends the pan never becomes
an auxclick. Every surface carrying the gesture — tab strips, the session
list, the terminal rail — is a scroller, so middle-click only ever worked on
macOS, where autoscroll doesn't exist.
Arm on pointerdown, spend on the pointerup over the same element (press one
tab, release on another and nothing happens), and cancel the middle mousedown
on every press so the pan widget can't appear on a surface that owns the
button. One helper, four call sites.
main landed a lint rule banning refs mirrored from reactive values in an
effect — they lag a render and cause stale reads. The lane-collapse
override tracker did exactly that with a counts ref.
Hold the empty/non-empty signature in state instead. React bails out
when it's unchanged, so a poll where no lane's emptiness moved costs no
extra render, and the comparison always sees the current value.
$bindings is seeded at module init from the actions known then, so an
action a plugin contributes later is absent from it. Both hotkey-hint
call sites did a raw bindings[id] lookup, so a plugin command rendered
with no combo in the palette and no hint on its tooltip even though the
dispatcher (which goes through $comboIndex → bindingsFor) fired it fine.
Route both through bindingsFor, the resolver that already falls back to
the stored override and the action's shipped defaults, and subscribe the
hint hook to the registry version so a late registration repaints.
Covered by behavior tests over the contributed-action contract: dispatch,
combo resolution, panel row, teardown, and the no-shadowing guard.
The gateway now queues a model switch made during a turn and applies it
at the next turn start (#74756), but the desktop still bounced the pill
back to the old model: the post-switch refetch answered with the model
still running and repainted over the pick.
Skip that refetch when the switch was deferred — the apply publishes
session.info when it lands, and that is what re-syncs every surface.
An older gateway that still refuses with 4009 keeps the pick too rather
than rolling back and toasting at a user who did nothing wrong; it is
what the next turn runs anyway. Real failures still roll back and report.
The 4009 predicate lives beside the other gateway-compat probes in
lib/gateway-rpc.
A reference had two styling systems: a Tailwind class string assembled in
TypeScript (`directiveChipClass`) and a separate `link-chip` for prose links,
each carrying its own color-mix(). Same concept, three appearances.
Now every inline reference — a composer chip, a sent message's mention, a
markdown link, a completion row's glyph — is `class="ref"` plus
`data-ref="<kind>"`, and styles.css owns the accent. No hex or color-mix()
ships from a component, so a skin restyles all of them at once.
Keying the accent on `[data-ref]` alone rather than `.ref[data-ref]` also lets
the popover's icon column take a kind's hue without inheriting its inline-text
layout.
The `/` path has had a completion cache since the skills-scan work; the `@`
path never got one. Every keystroke was an uncached round trip behind the 60ms
debounce, so walking a tree — Tab in, Backspace out, retype a segment — paid
full price for paths it had just listed.
Measured in-process against this repo (8,036 files): `git ls-files` ~38ms,
ranking ~12ms. The backend already caches the file list for 5s, so the fix
belongs in the renderer: reuse the existing cache module with a short 15s TTL
(a directory listing, unlike the command catalog, can change under the user)
keyed on cwd + session + query, and wire `isCached` so a warm query skips BOTH
the debounce and the loading state.
That last part is what makes it feel instant rather than merely fast — a
spinner over an answer already in hand reads as latency the user isn't paying.
YOLO had a status-bar zap and a slash command but was never registered as a
palette contribution, so ⌘K couldn't reach it.
Adding it exposed the wider gap: "Toggle status bar" doesn't say which way it
will go. Rows already carry a muted `detail` slot, so paletteToggle fills it
with on/off — no new chrome, and the verb stays. Status bar, logs, and layout
edit mode go through the same helper.
workspace.openFolder (default mod+o, the editor-standard open-folder chord)
runs openFolderAsProject: pick a folder, enter the project that already owns
it or create one named after the folder, scope the sidebar, and land on a
fresh session draft anchored there. A stale backend without the projects.*
RPC still gets the workspace session, with a warning.
StartWorkSessionRequest grows an openTab flag so these opens-from-nowhere
stack a tab instead of spending an occupied main, and goToProject/
resolveNewSessionCwd share one projectRootCwd resolver.
One slot, Slack-style: on assistant rows the picker trigger and the landed
reaction are the same far-right element, so reacting never shifts layout
(empty → ☺ following the action bar's hover fade; reacted → the emoji, always
visible, always full-strength). User bubbles react via right-click and show
the badge beneath, in the checkpoint row's register.
- Clicks paint instantly from a local nanostores overlay — no round-trip in
the loop; the RPC persists behind it and rolls back visibly on rejection
- Agent reactions land via the message.reaction event into an overlay keyed
by DURABLE row id, so the end-of-turn resume (which regenerates renderer
ids and rebuilds from in-memory history) can't clobber the paint
- Full picker is frimousse behind the six-emoji quick row, fed from bundled
emojibase-data served at ./emojibase by a small vite plugin (offline, no
CDN), with Slack-style alternating cell tints keyed off the codepoint
- Reaction picker opts out of the shared popover glass: solid surface so
15%-alpha hover tints stay readable
- react_to_message tool blocks are suppressed in the transcript (like todo):
the reaction appearing IS the UI; failures still render
- rowId reaches rehydrated messages from both transcript shapes (gateway
row_id, REST numeric id)
composer.modelPicker shipped unbound and opened the full-screen picker
dialog. It now defaults to ⌘⇧M — the chord LibreChat, Open WebUI, and
Cherry Studio independently converged on — and toggles the composer
pill's live dropdown instead, search field ready: ⌘⇧M → 'gr' → Enter
switches model in two keypresses.
Routing follows the tab-verb convention (#74447): the request lands on
the chat surface in the hovered zone first, then the active composer,
skipping hidden keep-alive tabs. With no chat surface on screen
(settings, profiles) the keybind falls back to the full dialog; with
the gateway closed the pill opens the dialog like a click would.
The voice-interruption fix (5081551f0) covered the Python surfaces (CLI +
TUI gateway) but the desktop app has its own mic path in voice-barge-in.ts,
which still had both bugs on Windows:
- HALF-DUPLEX GAP: the barge monitor only opened when TTS playback started;
during LLM generation no mic was listening at all.
- PLAYBACK DEAFNESS: the monitor calibrated its noise floor while the
speakers were already playing TTS, baking bleed into the floor. On
Windows, Chromium echoCancellation does not reliably cancel same-app
output (measured live: quiet floor ~35-50 RMS vs playback bleed
600-1700 RMS), making the trigger unreachable.
Changes mirror tools/voice_mode.full_duplex_listen:
- voice-barge-in.ts: phase-aware full-duplex monitor — quiet-only
calibration (floor held through playback, never recalibrated against
bleed), playback min-trigger clamp + ceiling, 500ms grace on playback
onset only, windowed-majority detection.
- use-voice-conversation.ts: monitor arms at turn submit and spans
generation + playback (ensureBargeMonitor, idempotent). Mid-generation
speech fires the new onInterrupt callback; spoken stop-word during a
barge ends the conversation; submit waits for the interrupt to settle.
- use-composer-voice.ts + composer/index.tsx: plumb onInterrupt: haltRun
(same seam as the Stop button).
Tests: use-voice-conversation.test.tsx (6 tests) covering monitor lifecycle
across generation + playback, mid-generation interrupt, stop-word handling.
npm run typecheck + eslint clean; remaining desktop vitest failures
reproduce on a clean tree (pre-existing Windows env failures).
Long thinking/tool stretches in a voice conversation are dead air — the
user cannot tell whether the agent is alive. New: quiet, repeating soft
bubble blips while the agent works and no speech audio is flowing.
- tools/voice_mode.py: numpy-synthesized blips (no binary assets) — two
alternating low pitches (G4/E4) with pitch glide + smooth attack/decay
envelopes, ~0.8-1.2s randomized spacing, volume = voice.beep_volume * 0.5.
start_thinking_sound(should_play=...) / stop_thinking_sound() daemon-loop
lifecycle; macOS-TCC-safe (sounddevice output gated there → silent skip,
no per-second afplay churn). New mark_audio_output_active()/
is_audio_output_active() ref-count wraps play_audio_file and the
streaming OutputStream sentence writes so "audio is flowing" is accurate.
- Config: voice.thinking_sound (default true) off-switch.
- cli.py: starts when a voice-mode turn begins, per-blip gate skips while
TTS speaks / mic records / barge capture owns the mic; stopped in the
chat() finally on every exit path.
- tui_gateway/server.py: same lifecycle around _run_prompt_submit turns
(voice mode on), gated on is_audio_output_active + continuous capture.
- Desktop: renderer owns voice-conversation audio, so a matching WebAudio
implementation (src/lib/thinking-sound.ts, same envelope/pitches) runs
while conversation status === "thinking"; honors voice.thinking_sound
(via config store) and the shared sound-mute toggle; stops instantly on
speaking/listening/end.
Saying OR typing a configured stop phrase (voice.stop_phrases, default
"stop") now ends the voice chat everywhere, not just classic CLI PTT:
- hermes_cli/voice.py: new explicit on_stop_phrase callback through
start_continuous/stop_continuous. The force-transcribe path previously
DISCARDED the stop phrase silently — with auto_restart=False the client
re-arms the next capture, so the conversation never ended. Both halt
paths now fire on_stop_phrase (fallback: on_silent_limit for legacy
callers) as user intent, distinct from the no-speech timeout.
- tui_gateway/server.py: voice.record wires on_stop_phrase and emits
voice.transcript {stop_phrase: true} after flipping HERMES_VOICE(_TTS)
off and stopping streaming TTS — same teardown as /voice off. The TTS
barge-in monitor stop-checks its transcript too. prompt.submit consumes
a TYPED bare stop phrase at the server-side choke point when voice mode
is on (returns {voice_stopped: true}, no turn starts).
- ui-tui: voice.transcript {stop_phrase} ends voice mode with a clear
'voice chat ended' notice (distinct from the no-speech-limit message);
submitPrompt releases the busy latch on a consumed voice_stopped reply.
- cli.py: _typed_voice_stop in process_loop — typing a bare stop phrase
while voice mode/continuous is active ends voice mode instead of
sending 'stop' to the agent; typed 'stop' outside voice mode is
unchanged. Voice transcripts skip the check (already stop-checked).
- desktop: interceptsTypedVoiceStop — the composer's onSubmit ends the
live voice conversation (same path as clicking end on the pill) when a
bare stop command is typed with no attachments; renderer-owned loop, so
handled client-side like the existing spoken isVoiceStopCommand.
- tools/voice_mode.py: transcribe_recording never lets the Whisper
hallucination filter swallow a configured stop phrase (e.g. 'bye'
configured as a stop phrase is both a hallucination-blocklist entry and
a stop phrase — stop-phrase check now wins).
Tests: continuous-loop signal (sabotage-verified), force-transcribe stop
signal + legacy fallback, hallucination-filter ordering, typed-stop CLI
unit tests (voice on/off/longer text), prompt.submit typed-stop gateway
tests, TUI vitest for stop_phrase event handling, desktop vitest for the
typed-stop interceptor.
The statusbar derived its client and backend version labels inline, in two
near-identical blocks. Move the wording into a pure resolver so every surface
that names an install agrees on the label, the commit diff, and the tooltip.
Assert isMessagingSource('photon') is true and that Photon/iMessage search aliases resolve, so a silent removal of the photon entry from MESSAGING_SESSION_SOURCE_IDS fails CI instead of regressing the sidebar section added in #47395.
Register Photon as a messaging platform in the Desktop sidebar:
- Add Photon brand icon (three-bar mark) to PlatformAvatar catalog
- Add photon to SOURCE_LABELS, SOURCE_ALIASES, and MESSAGING_SESSION_SOURCE_IDS
Closes#46761
xterm paints to a canvas, so its selection is not a DOM selection: the Edit
menu's Copy and the right-click Copy both call webContents.copy(), find
nothing, and copy nothing. On macOS the menu also swallows the Cmd+C
accelerator before the renderer sees it. That left Ctrl+C as the only key that
did anything, and xterm correctly forwards it to the PTY as SIGINT.
Mirror the selection into xterm's hidden helper textarea (the mechanism xterm
already uses for Linux middle-click paste) so the OS sees a real selection and
every platform copy path works, and add explicit chords on top: Cmd+C/Cmd+V on
macOS, Ctrl+Shift+C/V elsewhere, matching VS Code. Plain Ctrl+C copies only
when text is selected -- the behavior Windows Terminal and Tabby ship -- and
stays SIGINT otherwise, so interrupting a process never breaks.
Reads go through a new hermes:readClipboard IPC handler for the same reason
writes already do: the renderer's clipboard API throws whenever the document
isn't focused.
The `/` popover listed skills alphabetically, so on a 200-skill install
the ones invoked daily sat below a wall of skills that shipped with
Hermes and were never opened — /research-paper-writing (never used)
outranked /research (60 invocations).
Sort the Skills section most-used first, A-Z within a tie, and on a bare
`/` drop bundled skills with no recorded activity. Typing a query keeps
every match: a search that hides a result is broken, so a typed `/re`
only reorders.
brooklyn!