Follow-ups for the salvaged #93098:
- Move the tri-state liveness heuristic into hermes_cli.cron
(_builtin_gateway_liveness) so the CLI warning and the cronjob tool
share one implementation instead of two drifting copies.
- Surface gateway_running/warning on the list action too — an agent
inspecting jobs in a gateway-less environment has the same silent-
inert-job failure mode (#87033) as create. Empty lists stay quiet.
The builtin cron ticker only runs inside the gateway process. The CLI
surfaces this ('hermes cron list' / 'hermes cron status' both warn when
no gateway is running), but the model-facing cronjob tool returned a
clean success on create even with no gateway running - so the agent
confidently told the user a recurring task was scheduled while the job
could never fire.
Mirror the CLI's liveness heuristic in the tool's create path and attach
a tri-state gateway_running field to the result:
- true -> gateway running (or a non-builtin scheduler provider owns
firing, e.g. Chronos, which is exempt by design)
- false -> explicit warning telling the model the job is saved but will
NOT fire until the gateway starts, so it can relay that to
the user instead of reporting unqualified success
- null -> probe failed; claim neither way
Fixes#87033
the client half of the gateway.ping heartbeat contract (#89958); detects a silently-dropped socket via missed ping-acks and reconnects with bounded backoff; part of the #83166 recovery series.
The shared-client half of the gateway.ping heartbeat contract (#89958);
tracks lastInboundAt, sends pings, invalidates a silently-dead socket.
Part of #83166.
Additive WebSocket wire contract for a client-driven heartbeat.
The gateway.ready payload now advertises "heartbeat": True so clients can
discover the capability, and the WS read loop answers a gateway.ping request
with a {"ok": True} pong short-circuited BEFORE method dispatch (no method is
invoked). The WSTransport gains closed / last_inbound_at properties and a
mark_inbound() hook, updated on every inbound frame, for later liveness checks.
Backward-compatible in both directions: old clients never send gateway.ping,
and old servers simply never advertise the heartbeat flag. This is the first
slice of a WebSocket-recovery series; the follow-on slices consume this
contract (server-side transport rebind, TUI/desktop clients).
Receipts:
bash scripts/run_tests.sh tests/test_tui_gateway_ws.py -q
=> 1 file, 7 tests passed, 0 failed (100%) in 0.8s; exit 0
The Windows update hand-off shim was hardcoded to Microsoft Edge
(Find-EdgeExe), so machines whose default browser is Chrome still got
an Edge --app progress window, and every run leaked a throwaway
browser profile (hermes-update-ui-<pid>) under %TEMP% that was never
removed.
- Get-DefaultBrowserExe replaces Find-EdgeExe: resolves the OS default
browser from the UserChoice ProgId (https first, http fallback).
ChromeHTML -> Chrome, MSEdgeHTM -> Edge; any other ProgId returns
$null and degrades to the existing WinForms card.
- The dedicated --user-data-dir profile is now removed when the shim
closes, and stale hermes-update-ui-* leftovers from interrupted
runs are swept from %TEMP% in the same pass.
- --app + --user-data-dir is Chromium-only, so the whitelist is
intentionally limited to chrome/msedge; Edge keeps
--disable-features=msImplicitSignin to suppress the implicit MSA
sign-in that leaks into shim windows (#88410).
Review round (#86412):
- the except-import fallback returned the RAW oversized value, re-opening
the exact macOS time_t overflow this fix prevents; it now fails closed
to a finite ~1-year cap matching agent.deadline.MAX_SAFE_TIMEOUT_S
- clamp engagement logs a WARNING so operators see the semantic change
- new tests: float-form oversized value (YAML 1e18), warning emission,
and the import-failure fail-closed path (blocked-import probe proving
the result stays Lock.acquire-safe)
A group member turn is a session-scoped RPC sequence (resume → attach →
prompt.submit → poll) issued with the runtime id its first RPC minted, but
requestForBot routes every RPC through its own request-scoped socket lease
(retained:false secondaries in store/gateway). Between two RPCs the refcount
hits 0, the leased socket closes, the gateway detaches the runtime session on
WS disconnect, the orphan reaper frees it after grace, and the next RPC —
prompt.submit, unwrapped — dies 4001 'not in memory'. The member turn aborts
and the sub-profile bot goes silent in the room.
- store/gateway: retainGatewayForAgent(connectionId, profile) — refcounted
hold on the pooled socket with an idempotent release, mirroring the
existing request-lease machinery.
- sdk: host.retainProfile(route) exposes the retain to plugins
(feature-detected by consumers; older hosts keep working).
- hermes-bots plugin: runGroupChatMemberTurn acquires the lease before
ensureGroupChatSession's first RPC and releases in finally, so the socket
that minted the runtime id stays open across attach+submit+poll; and
prompt.submit gets a one-shot catch-and-retry that re-resumes via the
STORED session id on 4001-class failures (belt-and-braces for routes the
lease can't cover). 4007 'never existed' keeps flowing to session.create.
Tests: simulated 4001 on first submit recovers via re-resume and delivers;
lease held across attach+submit (mock refcount never hits 0 mid-turn); lease
released after success AND failure; no-retainProfile host feature detection;
store-level retain/release + idempotent double-release + the unretained
disposal race.
Widen the salvaged gate-site clamp to the bug class. The gate min() from
the contributor PR capped the gate bound at 360s, which would break the
#79719 contract (gate must extend while a legitimate >360s approval
prompt is answerable) and left the sibling overflow sites live: the CLI
prompt thread.join, the gateway poll deadline, and human_wait_ceiling
all consume the same config value.
Clamp once in _get_approval_timeout() via agent.deadline.MAX_SAFE_TIMEOUT_S
(1 year - semantically unbounded, platform-safe). The gate keeps its
approvals.timeout-tracking behavior above 360s; 7 regression tests pin
lock-acquire/thread-join safety and the gate-extension contract.
Bilateral E2E: with approvals.timeout=1e20 in a real config.yaml, main
crashes every consumer with OverflowError; this branch survives all 5
probes.
The _ConcurrentToolAuthorizationGate uses threading.Lock.acquire(timeout=...)
where timeout comes from human_wait_ceiling() (approvals.timeout + 60s).
When approvals.timeout is set very large (e.g. 999999999999 to effectively
disable timeouts), this overflows macOS timespec and raises:
OverflowError: timestamp out of range for platform time_t
The gate only needs to serialize parallel dispatch — it should never wait
longer than the wedged-holder bound (_AUTHORIZATION_GATE_LOCK_TIMEOUT_S =
360s). Clamp the return value with min() so unbounded approvals.timeout
cannot overflow the lock acquire.
Fixes#83220
#93515 reports auto-speak reading each reply twice when the Edge TTS
streaming attempt falls back to the POST endpoint and the reply's
renderer id gets rewritten to its durable id mid-flight. That was true
before 63565fa26b, but resolveSpokenReply()'s ordinal-anchored dedupe
(landed 2026-08-19, five days before this issue was filed) already
follows the rewrite. No source change — this pins the behavior with a
regression test at the hook/store integration level, one layer above
the existing spoken-reply.ts unit tests.
forkBranch was unconditionally routing every branched session into the
main pane via resumeSession, including sidebar/background branches of
a session the user isn't currently viewing. That reintroduces the
#69750 focus-stealing bug for that path: branching a different session
from the sidebar yanked the active view away from whatever was open.
Only take over the main pane when the branch's parent is the session
already selected; otherwise keep opening it as its own tile.
forkBranch ended by opening the branch as a session-tile and leaving the
primary selection on the parent (#69750). In the default layout there is
no visible tile pane, so branching only added a sidebar row with no
feedback in the main area — and openSessionTile no-ops when the target
is already the selected session, the common case of branching the chat
you're viewing.
Load the branch as the primary session via resumeSession instead, which
reuses the runtime already warm-cached by forkBranch's
ensureSessionState/updateSessionState calls, so it doesn't cost an extra
resume RPC.
Fixes#93444
Salvage round on the Phase 1 skeleton (three findings from review):
1. The DELETE leg was silently upgraded to WAL on healthy SQLite:
pre-seeding the file via PRAGMA was undone by SessionDB.__init__'s
apply_wal_with_fallback(), which upgrades any non-WAL file whenever
the configured mode (default wal) says so — only WAL-reset-vulnerable
interpreters preserved DELETE, i.e. the leg tested the advertised
mode only where CI wasn't running. Each matrix leg now pins
database.journal_mode in an isolated HERMES_HOME for the child and
audits the ON-DISK mode after the run (effective_mode_or_skip):
a leg that ran in a different mode skips instead of double-counting.
2. Cell 2 exit-code conflation: a claimant crashing with an unhandled
exception exits 1 — indistinguishable from the clean "lost the
claim" exit(1), so one winner + seven crashes passed as consume-once
proof. Codes are now disjoint (0=won, 10=lost, anything else=crash).
3. Crashed-writer diagnostics: wait_for now fails immediately with the
child's stderr when the writer dies before reaching the kill window
(was: 60s opaque deadline, stderr discarded). spawn_child prepends
to an inherited PYTHONPATH instead of clobbering it.
Plus: dead `if False` scaffolding removed from cell 1's writer; README
matrix section corrected (cell 2 is default-mode-only by design — the
consume-once property rests on a single predicated UPDATE).
The unlayered *:focus-visible reset in styles.css intentionally zeroes
--tw-ring-shadow ('No focus rings, anywhere'), so any control that relied
solely on focus-visible:ring-* had no visible keyboard focus state at all.
Mirror each control's hover treatment as a focus-visible background/text
affordance instead, keeping the global reset intact:
- ui/sidebar.tsx: group label, group action, menu button, menu action,
menu sub-button get focus-visible:bg-sidebar-accent + accent foreground
- ui/tabs.tsx: TabsTrigger gets focus-visible:bg-background + text-foreground
- ui/text-tab.tsx: focus-visible:text-foreground (matches its hover)
- chat/composer/micro-actions.tsx: pill gets focus-visible chrome-action-hover
- right-sidebar/index.tsx HEADER_ACTION_CLASS: focus-visible sidebar-accent
- right-sidebar/terminal/rail.tsx RAIL_ACTION: focus-visible chrome-action-hover
- chat/sidebar/cron-jobs-section.tsx (row body + run rows): focus-visible
chrome-action-hover
- chat/sidebar/session-row.tsx <time>: focus-visible:text-foreground
Sweep verified: remaining focus-visible:ring-* usages under apps/desktop/src
already pair with a border/bg/text companion (button/checkbox/switch/input,
starmap share-controls) or are covered by PR #93460's row-hover work
(cron/index.tsx run rows).
Fixes#93462. Reported by @fred0m.
Adds an optional occurred_at (ISO-8601 date/datetime) parameter to the
hindsight_retain tool schema, threaded into the retain item's timestamp
field. When absent, the item timestamp defaults to the configured event
clock (base from PR #82928 by @ragingbulld, authorship preserved) so the
Hindsight server can resolve relative time phrases; previously no item
timestamp was ever sent and temporal memories landed with null
occurred_start/occurred_end.
Fixes#93568. Salvages #82928.
Use Hermes timezone-aware timestamps for retained events and turn messages. Pass the public timestamp field supported by hindsight-client 0.6.1 and cover the final serialized request field.
Every opened file registered its preview pane with dock dir 'right', so
each open split a new zone off the right edge — three file opens made
three ever-narrower columns (#93610). The first preview still opens its
own zone docked beside main; every subsequent preview now anchors to an
existing preview-tile pane with dir 'center', so it stacks as a tab in
the same preview zone. Covers files, artifacts, and the Browser tab
alike (all flow through openPreview/$previewTabs); session tiles are
untouched.
Fixes#93610
Cron runs finish unwatched by design, so counting them in
$unreadSessionCount turned the titlebar badge into a permanently-lit
cron run counter (#93552). The badge now counts regular + messaging
sessions only; cron unread state stays visible on the sidebar cron
section rows, and 'Mark all as read' (markAllSessionsRead +
ackAllSessionsRead, which iterates cron rows) still clears them.
Fixes#93552
The bot relay's drain loop RPCs every registered connection through
requestGatewayForAgent's per-request lease. With no other consumer
holding the route, the refcount hit 0 after every tick and the pooled
secondary was disposed — a fresh WebSocket dial + teardown per
connection every 4s, flooding the gateway logs with connect/disconnect
pairs (#93594).
Two changes, both directions from the issue:
- Retained relay-route secondaries: retainGatewayForRelay pins a
route's pooled socket with a counted retention (never clobbering the
foreground 'retained' flag) for the relay's active lifetime, reusing
the existing scheduleReconnect/full-jitter machinery on drops. The
plugin pins each registered connection once via the new feature-
detected host.retainProfileSocket door, reconciles pins with the
current connection set on every drain, and releases everything in
stopBotRelay/dispose. Local routes (null/'local') are exempt so the
idle reaper can still reclaim spawned local backends. The live-work
pruner also respects the pin.
- RELAY_DRAIN_INTERVAL_MS 4s -> 30s: the push path (#93091,
bot_relay.outbox.pending) carries envelope latency, so the poll is
purely a backstop — 30s matches LIVE_SESSION_STATUS_BACKSTOP_INTERVAL_MS.
Tests: relay-push-drain updated to the new backstop semantics; new
gateway-relay-retention.test.ts proves one socket construction across
5 drain ticks (vs 3 constructions for 3 unretained ticks) and that
release/prune/local-exemption behave; new relay-socket-retention
plugin test pins the pin-once / release-on-departure / stop-releases
contracts.
The pricing snapshot could only express flat per-million rates, so
gemini-3.1-pro sessions with prompts over 200k tokens under-counted
input 2x ($2 vs $4/M) and output 1.5x ($12 vs $18/M).
- Add optional tier fields to PricingEntry: tier_threshold_tokens,
input/output/cache_read_cost_per_million_above (None = flat, falls
back to base rate per-field).
- estimate_usage_cost selects the above-threshold rates for the WHOLE
request once usage.prompt_tokens (input + cache read + cache write)
exceeds the threshold, matching Google's billing semantics.
- Populate gemini-3.1-pro (4.00/18.00/0.40 above 200k; alias
gemini-3.1-pro-preview inherits) and gemini-2.5-pro (2.50/15.00
above 200k).
- Flat entries are untouched: no threshold means no behavior change.
Reported and tier-field shape designed by @tornike14 (#93469).
Tests: below/at threshold unchanged, above-threshold tiered whole-request
pricing, cache-read tier rate and base-rate fallback, preview alias,
flat entries unaffected.
A malformed OPENROUTER_API_KEY in ~/.hermes/.env (truncated paste, wrong
provider's key) passed has_usable_secret's length/placeholder check and was
returned by _resolve_api_key_provider_secret before the credential-pool
fallback was ever reached, producing opaque '401 Missing Authentication
header' errors even when a valid pool entry existed (#93593).
- Add KNOWN_PROVIDER_KEY_PREFIXES (openrouter: sk-or-) and skip env values
that mismatch a declared prefix, logging a WARNING naming the env var and
expected prefix, then continuing to the next env var / pool fallback.
- Iterate credential-pool entries (peek first, then entries()) instead of
only peek(), so one malformed pool entry doesn't block a valid one.
- Providers without a declared prefix are fail-open: unknown key formats
are never rejected. Valid env keys still win over the pool (precedence
unchanged).
Fixes#93593
Follow-ups on top of the cherry-picked #93532 cap:
- Regression tests: suppressed (in-cooldown) matches must NOT consume the
lifetime budget; the cap trips exactly at the Nth DELIVERED match and
promotes to notify_on_complete with the watch_disabled summary queued
right after the final match.
- Extract _emit_lifetime_watch_disabled() and emit the summary even when
the global breaker drops the final match, so the user always learns why
watching went quiet (parity with the strike-limit path).
- Mention the lifetime cap in the terminal tool docstring (the schema text
was already updated by #93532).
Refs #93513
Per-session rate limiting only counts consecutive strike windows, so a
pattern that recurs at a cadence just above WATCH_MIN_INTERVAL_SECONDS
(e.g. a service restarted repeatedly over a day) never trips the
existing strike-limit disable — each match lands in its own clean
cooldown window. Every one of those matches still forces a full-context
agent turn, which stalls the event loop on large sessions (#93513).
Add WATCH_LIFETIME_MAX_HITS: once a session has delivered this many
watch_match notifications over its whole life, disable watch_patterns
and fall back to notify_on_complete, reusing the existing disable path.
missingRendererAssets only checked the module refs index.html itself names
(<script type=module> + modulepreload), so a torn install whose boot-critical
files were intact but whose lazy chunks were gone passed the generation check
and died minutes later on the first React.lazy() route with 'Failed to fetch
dynamically imported module' (#93479: syntax-diff-*, shiki-*, mermaid-embed-*).
Walk the generation's module graph: for every present JS chunk, parse its
inline __vite__mapDeps filename table (the lazy-import manifest Vite bakes
into each chunk) and check those files too, transitively and cycle-safe.
resolveRendererIndex now skips a lazy-chunk-torn candidate in favor of the
intact copy instead of shipping a delayed crash.
Tests cover the mapDeps parser (definition table vs index-only call sites,
CDN refs), the exact #93479 tear shape, transitive/cyclic walks, and the
torn-vs-intact preference end to end.
The renderer index resolver tried APP_ROOT/dist/index.html — inside app.asar
when packaged — before the app.asar.unpacked copy that asarUnpack (dist/**)
ships and that resolveWebDist() already prefers for the embedded dashboard.
Loading the asar-internal index is how lazily imported chunks (syntax-diff-*,
shiki-*, mermaid-embed-*) end up fetched from a path that cannot serve them,
killing the workspace pane (#93479).
Reorder the candidate ladder to prefer the unpacked web dist when packaged,
following the unpackedPathFor/resolveWebDist precedent. All window loaders
(main, overlay, quick) share resolveRendererIndex, so one reorder covers
every surface. Dev behavior is unchanged: outside an asar both candidates
collapse to APP_ROOT/dist and keep the original order.
React.lazy(() => import('./syntax-diff')) only has its pending state
covered by Suspense. When the dynamic import rejects (e.g. a packaged
app whose renderer window resolves to the app.asar copy of dist/ while
the chunk exists only in app.asar.unpacked, #93479), the rejection
throws past Suspense to the nearest error boundary, which is the whole
workspace ContribBoundary. One missing highlighter chunk then blanks
the entire chat transcript instead of just the diff falling back to
the plain colored DiffBody, the way markdown-text.tsx already isolates
this failure class for markdown.
Wraps LazySyntaxDiff in a local ErrorBoundary that renders DiffBody on
catch, so a failed highlight chunk degrades in place.
Audit of the remaining unguarded botConnectionRoute() callers a pane
render can reach (#93492 follow-up to the botRosterMeta split). Each now
uses the non-throwing resolveBotConnectionRoute() and degrades on an
owner_removed row instead of throwing into the pane's error boundary:
- botWorkspaceOwnerKey / setBotsWorkspaceOwner: sidebar visibility
listener, Bots home open, and roster context menus recompute these on
passive UI edges; an orphaned selection now yields the name-keyed owner
and the blocked workspace target.
- durableGroupChatMembers: rebuilt on every group send over the whole
seated roster; one orphaned member no longer aborts the room update,
and a swept member's degraded mark now survives the rebuild.
- useModelOptions: hook body runs during render; the query is disabled
for an orphaned row and the picker paints its error/disabled state.
- AdvancedProfileConfig: dialog falls back to the bot's own name scope.
Strict dispatch callers (requestForBot, session creation, deleteBot,
duplicateBot, ensureBotMetadata, routines) intentionally keep the
fail-closed throw — remote-routing-races.test.mjs still asserts it.
Adds orphaned-connection-members.test.mjs covering the removed-connection
sweep, the hydrate annotate (with/without a readable registry), the
degraded 'Gateway removed' rendering of swept rows, and every guarded
caller.
Rows poisoned before the removed-connection sweep existed (their
connection was deleted while an older Desktop ran, so no lifecycle push
ever swept them) are what made #93492 survive app restarts. After the
persisted 'group-chats' hydrate, run a pure annotate pass over the rooms:
- a descriptor that lost its connectionId (route unresolvable — the exact
shape that threw on render) is always marked;
- a descriptor whose connectionId is absent from the live connection
registry is marked only when the registry could actually be read —
an unavailable registry must not read as 'everything is orphaned'.
Marked rows keep their identity and degrade to the existing 'Gateway
removed' state; nothing is deleted.
Root cause of #93492: deleting a cloud/remote connection disposed its
gateways (store/gateway.ts) but never touched the persisted 'group-chats'
storage, so every member descriptor referencing the deleted connection
stayed behind as a poisoned row (remoteSource: true, connection gone) that
render-path route lookups tripped over forever.
Subscribe to the connection registry's 'removed' lifecycle push
(window.hermesDesktop.connections.onChanged, feature-detected — older
Electron mains don't emit it) and annotate every persisted group-chat
member owned by the deleted connection. Rows are marked
(sourceMissing/sourceReachable), never silently deleted: the member keeps
its identity and panes render the existing degraded 'Gateway removed'
botSourceStatus state. Writes ride updateGroupChat so the durable record
keeps its full shape, and the listener unbinds on plugin dispose.
botConnectionRoute() stays the strict, throwing dispatch path for real
routing (requestForBot, session creation). botRosterMeta() is passive
display code and previously reached that throw through a bare catch,
which would have swallowed any unrelated failure the same way. It now
calls a new non-throwing resolveBotConnectionRoute() and branches on a
typed resolved | owner_removed | not_scoped status instead.
Adds witnesses for the split: the typed statuses themselves, that
strict dispatch still fails closed on an orphaned row, and that an
unrelated failure while resolving meta for a live route still
propagates instead of being swallowed.
botRosterMeta() calls botConnectionRoute() for every sourceScoped/remoteSource
row to look up its metadata. That's a passive display lookup, but
botConnectionRoute() throws whenever connectionId can't be resolved -- which
is exactly what a stale group-chat roster row looks like once its connection
is deleted (its persisted descriptor keeps remoteSource: true but loses
connectionId). Since botRosterMeta() is called for every member on every
group-chat render, opening a group that still references a deleted
connection threw on render and crashed the pane's error boundary in a loop
that survived app restarts (the poisoned row is in Local Storage).
botConnectionRoute()'s fail-closed throw is correct and stays for its actual
callers -- routing a real request to a bot (requestForBot, session
creation, etc., covered by remote-routing-races.test.mjs). botRosterMeta()
now catches that throw and treats the row as having no resolvable route,
same as a bot with no meta at all, instead of letting it blow up rendering.
Fixes#93492
Follow-up to the reconnect-loop fix: the same unbounded ticket-mint await
exists on the soft gateway-switch path and the initial boot() path. Bound
both with the same withTimeout/RECONNECT_ATTEMPT_TIMEOUT_MS so a wedged
IPC round-trip fails into the existing retry paths instead of hanging the
switch or the 'Starting Hermes…' screen forever.
attemptReconnect() awaited desktop.revalidateConnection?.() unbounded,
immediately before the two IPC calls the previous commit wrapped in
withTimeout(). A wedged revalidation after a liveness-probe trip -
the exact trigger #93454 and this file's own comment describe - hung
that await forever, so the reconnecting guard never cleared and the
prior fix never got reached.
Wrap it in the same 20s withTimeout() (still swallowing the result via
.catch, matching its existing best-effort semantics) and extend the
regression test to hang revalidateConnection() specifically, proving
getConnection() and the socket still proceed once the stall times out.
After a liveness-probe-triggered reconnect on a remote gateway,
attemptReconnect() awaits desktop.getConnection() and resolveGatewayWsUrl()
with no timeout. If either stalls (e.g. main process wedged mid-revalidation
even though the backend itself is reachable), the `reconnecting` guard never
clears, so every later scheduleReconnect()/attemptReconnect() early-returns
forever and the UI stays stuck in "reconnecting" until the app is restarted.
Bound both awaits with a 20s timeout so a stall rejects instead of hanging;
the existing catch/finally already clears the guard and resumes backoff on
rejection. gateway.connect() keeps its own separate connect timeout.
Fixes#93454
pty_ws already fell back to the per-channel active-session file when a
/chat WS connects with no ?resume= param, replaying the whole session
into the PTY, but the frontend only pinned xterm's viewport to the
bottom when resumeParam came from the URL (#59591). The implicit path
had no way to learn a replay was happening, so the viewport stayed at
the top of the scrollback.
pty_ws now sends a one-off JSON control frame naming the session id it
resolved from the active-session file, before any PTY bytes; PTY
output itself always arrives as binary frames, so this is unambiguous
on the wire. ChatPage tracks an `effectiveResume` value seeded from
resumeParam and updated when this control frame arrives, and the
existing follow-scroll/sanitizer/hydration logic keys off it instead
of the URL param alone.
Fixes#93518.
Salvage hardening on top of #93601 (with #93597 covering the same core
mechanisms) for #93590:
- _hermes_cli(): after the venv-sibling check (hermes.exe on win32),
try shutil.which('hermes') before the bare-name fallback, so
environments with a PATH but no venv sibling resolve exactly what an
interactive shell would. Platform test switched os.name -> sys.platform
('win32') per repo convention.
- tui_gateway/methods_bot_relay.py deliver: pin encoding='utf-8',
errors='replace' on both subprocess.run sites — without them the
child's UTF-8 output is decoded with the locale codec (cp1252/GBK on
Windows), mangling non-ASCII replies or raising on undecodable bytes.
- Regression tests: shutil.which resolution step, bare-name fallback
with which=None, and encoding-pin assertions in the deliver transport
test.
Refs #93590, #93597, #93601
CI runners have a real hermes sibling next to the venv python, so
local_delivery_command now resolves an absolute path there — the exact
argv filters in the retry-policy fakes and the relay-methods pins must
match by basename instead of the literal "hermes", mirroring the
_delivery_lock matcher.
Two failures on a Windows desktop install relaying to a remote gateway
(#93590):
1. waiter_command embeds the reply path in generated python -c source
with !r. repr escapes each backslash, but the Windows execution layer
folds \\ back to \, so \U in C:\Users\... parses as a unicode escape
and SyntaxErrors the whole waiter script. Raw-string literals keep
the folded single backslash a literal; POSIX paths have no
backslashes so the prefix is a no-op there, and \' inside a raw
literal still cannot terminate the string, keeping the #93091
injection defense intact.
2. local_delivery_command hardcoded "hermes", relying on PATH — absent
in service contexts (systemd units, desktop launchers, non-login SSH
shells), so delivery died with ENOENT. It now resolves the CLI next
to this gateway's own interpreter (venv bin/Scripts sibling,
hermes.exe on Windows) with a bare-name fallback. The #93091
per-profile turn-lock recognition in bot_mode_dm now matches the CLI
element by basename (split on both separators) so resolved absolute
paths still take the lock instead of silently bypassing it.
Fixes#93590
The re-exec'd venv child spawned by
_reexec_dependency_sync_off_windows_shim completes every update step —
the receipt records success / "completed at command boundary" — but then
hangs in interpreter shutdown on a leftover non-daemon thread, freezing
the PowerShell window for minutes after "Update complete!". On the
hand-off path only (HERMES_UPDATE_REEXEC=1), after the receipt is
finalized, the update lock released, and stdio restored, flush and
os._exit(code) instead of unwinding — the same treatment #79040's cron
workaround applies. SystemExit codes (including early refusals)
propagate to the hard exit; real exceptions keep the normal raise path
so tracebacks still print. Non-hand-off invocations are untouched: the
marker env is set solely when the shim spawns the child.
Fixes#93581
The #93410 guard keyed on (restarted_services or killed_pids), which never
fires on Windows: _pause_windows_gateways_for_update /
_resume_windows_gateways_after_update populate neither list, so a healthy
resumed Windows gateway still yielded zero fleet rows and exit 0.
Hoist the decision into _fleet_probe_expected_runtimes(), keyed on every
pre-update liveness signal:
- restarted_services / killed_pids (POSIX restart bookkeeping)
- _pre_restart_gateway_pids non-empty or None (unreadable pre-state,
same fail-closed contract as _restart_phase_failure_is_incomplete, #78574)
- pre-update plan inventoried >=1 runtime
- Windows pause/resume token carries profiles or unmapped entries
Gate the 2.0s settle sleep on the same condition so a resumed Windows
gateway gets its settle window before the probe. The guard keys only on
zero-rows-despite-expected-runtimes; non-empty snapshots (including
'unknown'-state rows) are still judged solely by print_fleet_version_matrix.
Regression tests cover: empty snapshot + plan runtimes -> incomplete;
empty snapshot + genuinely idle -> success; Windows-resume token path ->
fail-closed + settle sleep wiring.
Builds on RelaxJonh's #93410. Fixes#93406
collect_fleet_versions() swallows every probe exception via
logger.debug() and returns whatever accumulated — which can be an
empty list. print_fleet_version_matrix([]) returns False (no rows
to report), so the update exits 0 with "success" even though no
gateway was actually verified.
After the restart phase touches live gateways (restarted_services or
killed_pids is truthy), an empty fleet snapshot means verification
failed, not that everything is healthy. Treat it as incomplete so
the receipt records "partial" and the exit code is 1.
Fixes#93406
Complete the #93527 fix: the tombstone now carries the human prompt
text via _entry_prompt_text (handling flat prompt, ShareGPT, and
chat-style shapes), _scan_completed_prompts_by_content counts
discarded rows as completed instead of only reading ShareGPT
conversations, and the merge step reports excluded tombstones in the
combined-count summary. Adds a dedicated regression suite covering the
tombstone round-trip, the all-discarded-batch resume path, and merge
exclusion.
Salvaged from #93579 (issue reporter's PR), building on #93542.
Fixes#93527
The no-reasoning discard branch in _process_batch_worker continued
before writing any JSONL row, so run(resume=True) — which filters
solely via _scan_completed_prompts_by_content over batch_*.jsonl —
never saw discarded prompts and re-ran them at full cost on every
resume. Write a tombstone row on discard, exclude tombstones from the
trajectories.jsonl merge, and report discarded_no_reasoning in
final statistics.
Salvaged from #93542.
Fixes#93527
Adjust the #93423 max_tokens-only regression test to the merged policy:
max_tokens stays as an explicit LAST-RESORT fallback (some local servers
report nothing else) instead of being dropped entirely, and add coverage
that _reconcile_local_cached_context_length rewrites a cache entry
poisoned by the old probe (393216) upward to the real window (1048576)
once the probe is fixed.
Co-authored-by: pju-hoge <grkt@ppmz.com>
Co-authored-by: re-ITRT <1940428933@qq.com>
kshitijk4poor