run_job opened state.db (SessionDB) at the top of the function, before the
wake-gate (wakeAgent: false), prompt-injection block, and drift-skip early
returns. Every gated run therefore opened a full SessionDB — read pool,
token-writer machinery, .db/-wal/-shm handles — and returned without
reaching the finally that closes it, relying on GC/__del__ to release the
descriptors. On a gateway whose monitor-gated jobs tick every few minutes,
that is constant wasted open/migrate work and GC-dependent fd lifetime.
Move the init inside the main try, immediately before AIAgent construction,
after every early-return path. The timeout resolution now reuses the _cfg
already loaded for model routing instead of a second load_config() call.
Behavior on the normal (non-gated) path is unchanged: same env/config/default
timeout resolution, same abandoned-worker done-callback close (#72782), and
the existing finally still closes the store after the agent turn.
Salvaged from PR #96290 (cron slice) with a mutation-checked regression test
(fails on main: gated run opens SessionDB; passes with the reorder).
_sqlite_connect opened a connection via connect_tracked and then ran the
busy_timeout PRAGMA; if that raised, the half-open connection was abandoned
— leaking its fd AND leaving a stale entry in the sqlite_safe_read
live-connection registry (which only clears on close), permanently blocking
byte-level probes of the kanban database. Close before re-raising.
Salvaged from PR #96290 (kanban slice) with regression test.
agent/deadline.py defined SuspectableBackend twice: the Phase 3a Protocol
(sync ensure_healthy(self) -> bool) and, further down the same module, an
unrelated concrete class with the same name (async
ensure_healthy(self, timeout=5.0)) added later by the MCP Phase 3b adopter.
Since Python executes class statements top-to-bottom, the second definition
silently shadowed the first at module scope.
Nothing in the tree imports or subclasses either by name today — the MCP
adopter duck-types the same-shaped contract directly on its own connection
class rather than referencing agent.deadline.SuspectableBackend — so this
caused no live behavior change. But it left the wrong (and differently
shaped) class resolvable under that name for the next Phase 3b adopter that
does import it for a type hint.
httpx timeout exceptions (ReadTimeout, WriteTimeout) stringify to "",
which defeats _is_timeout_error's first-line guard (if not error: return
False). The base-layer plain-text fallback then re-sends an already-
delivered message — the user receives it twice.
Replace error=str(exc) with error=str(exc) or type(exc).__name__ at every
httpx-based adapter boundary so the existing matcher ("readtimeout",
"writetimeout") still fires. ConnectTimeout intentionally stays
unmatched: if the connection never opened the message was not delivered,
so retry/fallback remains correct.
Applies to BlueBubbles (send + _create_chat_for_handle), WhatsApp Cloud
(text + interactive + media), QQ Bot (send chunk + keyboard + media), and
Yuanbao media handler — the same latent bug exists in every adapter that
stores error=str(exc) from an httpx call.
- _write_machine_sentinel_line: wrap the print() fallback so a closed
redirected stream (ValueError, not OSError) can't propagate out of the
ready path and kill a healthy serve; document that pythonw port
discovery relies on the HERMES_DESKTOP_READY_FILE channel, not stdout
- regression test: stderr=DEVNULL instead of PIPE — with the stdout
redirect active all server logging lands on stderr, and an unread
stderr pipe can fill and block the child before the sentinel, flaking
the test at the 120s timeout
The same stdout redirect that rerouted the READY sentinel (#96282) also
reroutes the machine-parsed BACKEND_PORT_IN_USE sentinel printed by
_report_port_in_use() — both preflight and probe-to-bind-race callers run
after tui_gateway.server's sys.stdout=sys.stderr swap. Extract the fd-1
write into _write_machine_sentinel_line() and use it at both sentinel
sites; human-facing hint lines stay on print().
Since 6d4e851d8 the serve startup path imports tui_gateway.server (for the
flush-on-SIGTERM handlers) before the READY sentinel is printed. That module
redirects sys.stdout to sys.stderr at import time, so the
HERMES_(BACKEND|DASHBOARD)_READY port=<n> sentinel landed on stderr while the
Electron desktop spawn watches child.stdout only — the desktop timed out
after 90s and killed a perfectly healthy backend (issue #96282).
Write the sentinel to the real stdout file descriptor (fd 1 is untouched by
the Python-level redirect), with a print() fallback.
Adds a regression test that captures stdout/stderr separately — the existing
E2E suite merges them, which is exactly how this slipped past CI.
Live /api/v1/models probe (2026-08-27) confirms the id is gone from the
catalog, so the curated picker entry was a dead pick. Manifest
regenerated. No provider-agnostic metadata existed for the slug.
Delist credit: @orouge97 flagged this in PR #80036.
OpenRouter and Nous already list z-ai/glm-5.3-flash (#95621). The
native z.ai picker, OpenCode Go/Zen fallbacks, setup wizard, and
Coding Plan probes did not. Context still resolves through the
existing glm-5.3 1M key.
The agent loop writes an internal "(empty)" sentinel when the
nudge/prefill/empties/fallback ladder all fail. The gateway converts it
into a user-friendly notice at delivery, but the desktop group-chat
bridge appended the raw sentinel into the room log (seen posting
"(empty)" in a Bot Mode group room), and it synced to the shared
ui_meta for mobile.
Normalize at the single choke point, appendGroupChatEntry, mirroring
gateway/run.py substitution so group chat and gateway surfaces show the
same text. (pass)/empty silence semantics unchanged. Includes a
regression test proven to fail on the pre-fix code.
Fixes#94308
Address review feedback on #94386: the new tests only exercised
runGroupChatMemberTurn's use of pickGroupTurnReply. Add the analogous
case for harvestStrandedGroupReply (substantive answer -> synthetic
continuation nudge -> (pass) tail) and document the pass-only tie-break
(newest wins) in pickGroupTurnReply's docstring.
runGroupChatMemberTurn (and harvestStrandedGroupReply) selected only the
last assistant message in a finished turn. A Codex intent-ack continuation
nudge can land a complete, substantive room answer and then get a
synthetic "(pass)" reply to the nudge itself — the terminal message picked
by the old scan, which silently discarded the real answer (#94376).
Both call sites now scan the messages appended this turn for the last
substantive (non-pass) assistant reply, falling back to a pass only when
no substantive answer exists in that window.
Source-contract tests (the group-room-ux pattern): the workspace renders
the Stop button only while room.running, wires it to stopGroupThread
(not the #94570 per-member interrupt spray), and carries no hardcoded
CJK label.
Salvaged from #94570 (@ShonnQ): the Activity bar gains a Stop button
while a round is running (room.running), plus an inline Stop on the
expanded 'working' activity row. Rewired from the original per-member
session.interrupt spray onto the stopGroupThread primitive so the round
loop actually stops (epoch bump + holds + on-turn interrupt) instead of
marching to the next member; labels are plain English like the rest of
the plugin's UI strings.
Co-authored-by: Hermes Agent <agent@nousresearch.com>
stopGroupThread(group, thread, members?) is the room's first true
cancellation primitive: it bumps the room epoch (the driving loop bails
at its next member boundary), sets #93129 holds for every member (no
future turns until an explicit release), records a 'stopped' activity
event on the new epoch, and sends session.interrupt to the member
currently on turn via its own route — previously the plugin issued zero
interrupt RPCs, so 'stop' meant waiting out the in-flight model call.
The runGroupChatMemberTurnLeased poll loop now abandons a turn whose
dispatch epoch went stale WHILE its member is held — the stop signature.
An ordinary newer-send epoch bump without a hold still polls to
completion so late work keeps landing (#93127 commit check unchanged).
Allowlist hot-path hooks for abandon-on-timeout, keep subagent_stop on the caller thread, suppress re-fires of hung callbacks, and block tools when pre_tool_call times out.
Builds on e11187208f (salvage of #72206 by @luyifan, authorship
preserved) which added the post-timeout session reset. This commit
completes the Phase 3b contract:
- suspect flag: a command timeout marks the session key suspect;
the NEXT _get_session_info for that key health-checks and recycles
the session instead of handing back the poisoned handle (#72205)
- flag cleared on fresh-session store so a healthy new session is
never spuriously recycled; cross-key leakage fixed
- wedged-vs-alive rule (#68139): after a timeout, if the daemon's
socket still answers, recycle the session only; if it is
unresponsive (or no socket exists to probe — conservative), tree-
kill via agent.deadline.kill_process_tree and evict
- negative probe: successful commands never mark or recycle
Tests: tests/tools/test_browser_suspect_recycle.py (20 tests:
mark-once, recycle-then-succeed, success-never-recycles, tree-kill
invoked on the wedged path with pid assertion, flag lifecycle).
Co-authored-by: luyifan <al3060388206@gmail.com>
Sibling site of the salvaged #95947 fix (same file): cron_status
declared 'Gateway is not running — cron jobs will NOT fire' from a bare
find_gateway_pids() miss even while the runtime lock proved the gateway
alive. Now the not-running verdict requires both the scan AND the lock
to read dead; when only the lock answers, the pid line falls back to
the recorded gateway pid (or is omitted).
Two regression tests pin the false-alarm suppression and the genuine
not-running warning.
Follow-ups to the salvaged #95947 cron commit:
- Wrap the lock probe in its own try/except: a crashing probe is
'unknown', not 'dead' — the pid scan still decides instead of the
whole tri-state collapsing to None.
- Regression tests (shape adapted from #94155 by @liuhao1024): lock
held + empty pid scan -> alive (the reported false alarm); lock
inactive -> pid-scan fallback both ways; crashing lock probe still
falls back.
- patch_liveness now pins the lock probe inactive by default so the
pre-existing pid-scan tests stay deterministic on machines where a
real gateway holds the real lock.
- contributors mapping for magnus.lundstedt@infidyne.com.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
_builtin_gateway_liveness() decides whether the builtin cron ticker can fire by
PID-scanning via find_gateway_pids(). That scan can transiently return empty even
while the gateway is up (e.g. just after a restart), so the in-gateway cronjob tool
emits a false 'Gateway is not running — jobs won't fire' while jobs are firing on time.
Prefer the gateway runtime lock: it is held for exactly the gateway's lifetime (a
reliable liveness signal), and inside the gateway process it short-circuits to True,
so the in-gateway check can never false-alarm. Fall back to the PID scan only when the
lock reads inactive (the external-CLI path).
- move minimax/minimax-m3:free into the Free tier section (house
convention: :free SKUs group together, matching glm-5.2:free and the
nemotron :free entries) and regenerate model-catalog.json
- add Inkling family context length (1,048,576 — OpenRouter live
metadata, 2026-08-27) to DEFAULT_CONTEXT_LENGTHS; new family slug
otherwise fell through to no entry
- add Inkling to the reasoning stale-timeout floor table (300s tier,
same as Grok reasoning / Ox Alpha; OpenRouter marks the family as
reasoning-capable)
- widen the floor matcher's right-anchor separator class to include
':' so OpenRouter SKU suffixes (:free/:batch/:nitro) inherit the
family floor — inkling:free previously missed the inkling entry
- regression tests for the inkling floor + ':' separator
Follow-up to the #94755 salvage: every runGroupChatRounds exit recorded
'settled', so a room that died at the round/message/continuation cap looked
identical to genuine consensus. Track the exit kind and record 'capped'
(with label + glyph) when a cap ended the drive, and pin the exit-path
wiring with source-contract tests.
- GROUP_CHAT_MAX_CONTINUATIONS=2 caps continuation rounds independently of
the message cap, so pathological @mention chains can't consume the room's
whole budget on handoffs.
- unaddressedGroupMentions now orders by log INDEX instead of entry id:
ids are UUIDs (groupChatEntryId), not monotonic — string comparison could
both re-drive answered members and miss stranded ones.
- New unaddressed-mentions.test.mjs exercises the REAL function via the
vm-slice pattern (replacing concept-only helpers) and pins the ordering
fix with a UUID-vs-log-order case.
In Bot Mode group chats, a member reply that @mentions a teammate never
drove the cited bot when the current round went quiet: the
'spokeThisRound === 0' early exit treated a zero-reply round as 'everyone
passed' and settled the room, even though the reply's @mention was
pending. The same silent settle happened whenever GROUP_CHAT_MAX_ROUNDS
or GROUP_CHAT_MAX_MESSAGES landed between the mention and the next
round (#94478).
Fix:
- unaddressedGroupMentions() detects member-to-member citations in the
thread whose cited member has not posted anything after the citing
entry (self-mentions excluded; user sends re-drive everyone anyway).
- The quiet-round exit now checks for such pending handoffs and runs one
bounded continuation round driving exactly those cited members — same
holds/stranded/epoch/cap machinery as ordinary rounds, so nothing new
is trusted.
- If the continuation also produces nothing (pass, failure, or cap), the
room settles as before; behavior only changes where a bot was actually
called and never answered.
Regression tests in plugins tests family (2 new); full hermes-bots
plugin suite stays green (558/558).
Fixes#94478
1. Quoting: the spawn payload wrapped expandRemotePath() output -- already
a shell-quoted fragment like "$HOME"'/...' -- in shq() again, so the
reservation/lock/owner_file variables hold the quote characters
literally and every mkdir "$reservation" fails forever (~5 min per
attempt spinning in the reservation loop while holding the box-global
update mutex; queued spawns starve behind it). The same double quoting
sits in the stale-reaper identity guards, making every reap REFUSE.
The lockfile-reuse path masks the bug for existing backends, so it
only bites on fresh spawns.
2. Bashism: lockfile publication used ${var//__PID__/$child} -- bash-only
substitution in a payload run under plain sh (dash on Ubuntu), which
aborts the script AFTER the serve was spawned. The client then saw an
unknown failure, ran its error cleanup (deleting the token file), and
the just-booted serve died on the missing token -- orphaning one serve
per attempt. Replaced with a POSIX sed substitution.
Adds two regression tests: payload variables must keep $HOME expandable
(no re-quoting), and the pid substitution must be POSIX sh. Both fail
against the previous code; all 89 remote-lifecycle tests pass with the
fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The OpenRouter model picker builds its list from a curated set of model
IDs, then filters against OpenRouter's live catalog. minimax/minimax-m3:free
exists on OpenRouter (free tier, 1M context, tool-calling) but was missing
from both the in-repo fallback list and the remote catalog manifest.
Add it to OPENROUTER_MODELS and website/static/api/model-catalog.json so
the free variant surfaces in the picker alongside the paid one.
Independent review finding on the merged #95980:
_apply_live_compression_config only acted on PRESENT keys, so removing
tail_mode / model.context_length / target_ratio / model_thresholds /
proactive_prune_* / protect_last_n / min_tail_user_messages / threshold /
idle_compact_after_seconds from config.yaml left stale values active in
live sessions forever (probe-verified: all six stale after applying
empty mappings).
Absence now restores the normalized default — or the model-derived
value — through the SAME derivation the construction path uses:
- ContextCompressor ctor defaults read off its real __init__ signature
(no hardcoded copies to drift)
- compression.threshold removal re-derives via agent_init's
_resolve_compression_threshold (Codex gpt-5.4/5.5 + spark autoraise
included)
- model.context_length removal drops the config override and forces
re-inference through the deferred get_model_context_length resolution,
which also re-applies the small-context threshold floor
- model_thresholds removal clears stale per-model overrides from the
live threshold; tail_mode falls back to the ctor's 'lean' (the old
present-key path normalized invalid values to 'legacy', diverging
from the compressor's own fallback)
Also fixes proactive_prune_min_reclaim_tokens's present-but-null default
(was 0; the real default is 4096).
Refs #94724
A hermes serve killed mid-update lost every un-flushed in-memory session
(#94724 item 2, reported by @ruangraung): the next RPC failed with
'session-scoped RPC rejected: not in memory (detached/reaped runtime)'
and no store held the transcript. #95576 made serves survive future
updates; this closes the kill path itself:
- install chaining SIGTERM/SIGINT handlers (hermes serve / dashboard
startup, before uvicorn's capture_signals) that first persist
in-memory session transcripts to state.db — bounded by
HERMES_TUI_EXIT_FLUSH_BUDGET_S (default 5s, daemon worker + join) so
a hung SQLite write can never block exit
- _shutdown_sessions (atexit) runs the same bounded flush FIRST, before
the slow per-session teardown a supervisor may SIGKILL mid-way
- the idle-reaper scan piggybacks a periodic incremental flush
(marker-deduped agent._persist_session, running sessions skipped) so
even a SIGKILL loses at most one flush interval — no new timer
subsystem
Refs #94724
The fail-closed owner ladder (#95407) is correct for new sessions, but
legacy unowned rows on registry-topology installs dead-ended in
SessionOwnerResolutionError (reporter's Error B) with their transcripts
fully intact in state.db.
- resolveLegacyOwnerBackfillScope: pick the single-match store for the
server-side owner backfill at enumeration time (serving registered
connection / primary pool); fail closed on multi-candidate topologies.
- maybeBackfillLegacySessionOwners: one-shot per scope per renderer,
fire-and-forget from the #95407 stamp path, logs the stamped count.
- Read-only stored-transcript resume: when session.resume fails closed,
fetch the transcript over id-only REST (ambient first, then registered
backends, read-only probes only) and open the session as a read-only
transcript instead of dead-ending; sends are refused with a notice and
a later successful live resume clears the latch. Wired into the main
pane resume recovery and the session-tile delegate (which now runs the
same fail-closed owner gate as the RPC dispatcher).
Refs #94724
POST /api/sessions/owner-backfill stamps a store's own serving-profile
identity onto its pre-#95407 'profile_name = NULL' session rows. Single
match by construction (each profile's state.db belongs to exactly one
profile), idempotent, one-shot-per-row, never overwrites a non-NULL
owner, and reports the stamped count for logging.
Refs #94724
The smart-approval guardian (`_smart_approve`) gates every flagged
terminal command with a synchronous auxiliary LLM call, but it never
passes `timeout=` and logs nothing on the normal path. In production a
stalled provider response silently froze the agent turn for 62 minutes
with zero log output; the gateway kill-switch eventually fired, and only
an unrelated error surfaced afterwards (#82846; watchdog-style fix in
#72500). The call was invisible by design — nothing logs at the hang
point.
Changes in tools/approval.py:
- Resolve the same configured timeout the client would use internally
(`auxiliary.approval.timeout` via `_get_task_timeout("approval")`) and
pass it explicitly to `call_llm`, so the deadline cannot be lost if the
internal default resolution changes or is misconfigured.
- Log the assessment call and its duration (DEBUG), and promote the
failure branch from DEBUG to WARNING with elapsed time + exception
class, so a wedged guardian call is visible in the logs instead of
silent.
- Failure still returns "escalate" (fail open to the human/pattern
gate) — behavior unchanged, observability only.
Complements #72500 (watchdog hard ceiling) rather than duplicating it:
explicit timeout is the root-cause hardening, logging closes the
silence gap; the watchdog remains the safety net if the SDK-level
timeout itself is defeated.
Tests: explicit timeout forwarded to call_llm (revert-fails), failure
logs WARNING + escalates. 49 approval-adjacent tests pass; one unrelated
test_approval.py failure is pre-existing (fails on clean main too).
The script-timeout path used a site-local process-group kill, which
cannot reach a grandchild that created its OWN session (start_new_session
background jobs, watchdogs). Such descendants kept running after the job
reported failure (#71148, #59549). Migrate the timeout handler to the
unified deadline layer's kill_process_tree (#85147, d6a5cb9725): psutil
snapshots the descendant set before signalling, so own-session
grandchildren are reached too. Fallback to the site-local group kill if
the import ever fails, so the path cannot re-wedge.
The explicit script-timeout message stays the classification anchor
(#85536's contract), keeping cron timeouts distinct from provider
timeouts.
Salvage additions on review (#85125 Phase 4a):
- migrate the sibling kill site too — the cancel_event/"ownership was
lost" path orphaned setsid grandchildren the same way (whole-bug-class
rule); pinned by test_cancel_path_also_tree_kills
- proc.poll() early-return in _terminate_cron_script_tree so a script
that exits right at the deadline doesn't log a spurious "no signal"
warning (mirrors _terminate_cron_script_process); pinned by
test_already_exited_proc_is_left_alone
- acceptance test's script timeout 1s -> 2s: interpreter startup under
CI load could eat the whole 1s window before the spawner wrote its
pid file
- note: kill_process_tree hard-kills (SIGKILL) immediately, whereas the
old path gave a 1s SIGTERM grace window; intended for a deadline-
expiry hard stop (both docstrings say "hard stop")
Based on #86791 by @ayushnangia; cherry-picked to preserve authorship.
Co-authored-by: dante32683 <dante32683@users.noreply.github.com>
Co-authored-by: supotato-ipj <supotato-ipj@users.noreply.github.com>
Apply-ready delta distilled by @andrexibiza: deterministic watcher-consumer
tests (watcher times out while a child is alive, resolves when all are dead),
psutil-unavailable fail-open pin, and probe-failure fail-open handling in
_stdio_children_dead (unknown is never proof that every child exited).
Local: 8 passed on tests/tools/test_mcp_stdio_children_dead.py
_stdio_children_dead returned True ('all children dead') on the first LIVE
pid — the intended False was dead code right below it. Every spawn path
that captures child PIDs (observed in hermes -z oneshots) then failed the
#81995 pre-call fast-fail with 'TimeoutError: MCP stdio subprocess ... has
exited' on every tools/call while the subprocess was demonstrably alive.
Long-lived gateway/dashboard sessions were unaffected only when
_stdio_child_pids was empty (the not-pids short-circuit).
Return False on the first live pid and drop the unreachable line.
Heath Harris