Branch B of _GATEWAY_LIFECYCLE_PATTERN enumerated launchd verbs but omitted
`bootout` - the modern replacement for the `unload` it already listed, and
the paired inverse of the `bootstrap` it already listed. `remove` (legacy
sibling of bootout) and `disable` (what makes an unload durable) were
missing for the same reason.
This matters because the two enforcement layers are not interchangeable. In
tools/terminal_tool.py under _HERMES_GATEWAY == "1":
- the cron.lifecycle_guard hard block is documented as applying
unconditionally ("force=True cannot help here")
- detect_dangerous_command below it is explicitly skipped when force=True
detect_dangerous_command already flags all three verbs, so the default path
was covered - but with force=True inside the gateway they reached execution
while stop/unload/kickstart did not. SIGTERM then propagates to the child
before the command completes and the service may never come back, which is
the state described in #74973.
The label anchor (\bhermes[.\-]?gateway) is unchanged, so unrelated services
such as `launchctl bootout gui/501/ai.hermes.update-checker` stay runnable.
Adds TestLifecycleGuardLaunchctlParity, which pins the one-directional
invariant: anything the bypassable approval layer flags, the unbypassable
hard block must also catch. Deliberately not equality - the hard block is
legitimately stricter (it also covers load/restart, which the approval layer
leaves alone). Verified failing on the parent commit for exactly bootout,
remove and disable.
Closes#80260
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Bot Mode 'session not found' / bot-runs-on-wrong-backend bug. wiring's
requestGateway is ONE shared closure for every session-scoped RPC in the
window, but it derived the owning profile from the globally-FOCUSED tile
($focusedStoredSessionId). A bot chat is a background tile while another pane
is active, so its prompt.submit carried the bot's own session_id yet was
dispatched on the FOCUSED tile's backend — the default backend served the bot
via ?profile= from the default's state.db, or answered 4001 'session not
found' when it didn't hold the runtime session.
Route by the session the RPC TARGETS (params.session_id) instead. session_id
is a RUNTIME id while tiles/rows key on the STORED id, so translate via the
state cache then a reverse scan of the stored->runtime map (the same ladder
use-session-tile-delegate's storedSessionIdForRuntime uses); an unresolved id
is already a stored id (several RPCs pass stored ids directly). RPCs with no
session_id (ambient/config) keep the focused->selected fallback.
Pure helpers extracted to wiring-routing.ts so they're unit-testable without
importing the React controller; 6 tests cover the target-vs-focused routing,
the stored-id passthrough, and the no-session fallback. tsc 0 errors.
Diagnosis verified on a live install: the fix was present in source but the
running build still misrouted, and logs showed the bot's turn executing on the
default backend while its own per-profile backend sat idle.
Prose false positives (Branch A trailing boundary, Branch D leading
boundary), quoted-multiline data payloads, and the fail-closed
unbalanced-quote fallback.
- Add _split_logical_lines() to split on newlines outside quotes, fixing
false positives from quoted multi-line payloads (e.g. python -c "...")
being torn into fragments and scanned as referenced scripts.
- Make _iter_command_segments() use logical line splitting with fallback
to per-physical-line tokenization for unbalanced quotes.
- Add missing word boundaries to _GATEWAY_LIFECYCLE_PATTERN:
* Branch A: trailing \b after restart|stop
* Branch D: leading \b before p?kill to prevent matching "skill"
(and similar words ending in "kill")
Fixes#92372: gateway lifecycle guard false-blocks on prose inside
a referenced data file.
A legacy remote primary carries no registry connectionId, so the scoped
reconnect reset could not name the restarted owner and fell back to
preserving every owner-routed Bot tile -- leaving the restarted backend's
own Bot Chat bound to its dead runtime (the original bug, persisting for
that one connection shape).
Unknown identity now fails toward recovery instead: preserve only Bot
runtimes owned by provably-live secondary connections
(liveSecondaryConnectionIds()); everything else drops its binding and
re-resumes. A reset only costs a re-resume, so this is safe for the
preserved-set survivors and correct for the dead one.
Follow-up to @BrunoBza's #93062:
1. Set failed=True only for the new repeated_outer_errors exit reason.
Previously the error exit left failed=False, so finalize_turn reported
completed=True for a turn that actually failed — incorrect.
2. Don't append_message the assistant response at the break. A thinking-
prefill or interim assistant may already be the tail, and appending
would create assistant→assistant role-alternation violation.
finalize_turn (lines 341-353) handles this safely by checking
_tail_role != 'assistant' before appending.
3. Update test to assert failed=True and completed=False for the
repeated_outer_errors exit.
The outer conversation-loop except handler only left the loop on a
local-processing error or when api_call_count >= max_iterations - 1.
With the turn budget now unlimited by default (sys.maxsize), a
permanent failure that escaped the inner retry/fallback machinery
retried forever: ~64 retries/s, one core pegged, and the rotated
agent.log history overwritten within minutes.
Bound the loop with a small per-turn cap on total escaping exceptions
(_MAX_OUTER_LOOP_ERRORS = 8, scaled down by a tiny explicit
max_iterations so a manually bounded budget still governs). The legacy
local-processing and near-limit exits are byte-identical; a new
'repeated_outer_errors' exit reason gets a user-facing explanation.
The inner retry/fallback layer owns transient API recovery and
terminates on its own, so only exceptions that escape it reach this
cap - a successful turn is unaffected.
Fixes#92450
Brief transport blips often self-heal in 1–3 minutes. Raise the
non-blocking escalate toast from 45s to 5m so those windows stay quiet
while chat remains readable/draftable. Confirmed reauth still takes the
full-screen recovery overlay immediately.
Post-boot WebSocket ticket mint failures and prolonged reconnects were
promoting into the full-screen "Hermes couldn't start" overlay, locking
users out of reading/drafting during brief 1–3 minute remote flaps.
- Ignore non-reauth boot-progress errors after a healthy cold boot
- Escalate prolonged transport reconnects with a non-blocking toast
- Soft-reset remote liveness rebuilds (no boot UI reset)
- Retry transient ws-ticket mints; auth rejections still fail fast
requestGateway (contrib/wiring) resolved the owning backend from
$focusedStoredSessionId — the WINDOW's focused tile — for every RPC it
dispatched. A session-scoped RPC names its real target in
params.session_id; whenever that session's chat was NOT the focused
pane (any background bot chat — the normal Bot Mode case), the RPC was
dispatched on whichever backend the focused tile happened to own. A
profile bot's prompt.submit then executed on the DEFAULT backend
(sessions created in the default store, profile logs empty), or failed
with 4001 'session not found' when default didn't hold the session.
Root cause isolated by Teknium: submit for a Developer-profile bot ran
in root logs/agent.log while profiles/developer/logs sat empty, and
completed only when default happened to hold the session — proving the
misroute sits downstream of the tile-owner-route lookup, in the
routing-key choice itself.
requestGateway now routes by the RPC's own target first:
params.session_id (a RUNTIME id) is translated to the stored id via the
tile map — new storedSessionIdForRuntimeId() in session-states, where
tiles already carry both identities — and only session-less RPCs
(config reads, list refreshes, cron) fall back to the focused-tile key,
which is genuinely window-ambient. Stored-id claims win over runtime
bindings in the lookup so a stale tile's dead runtimeId can never
hijack a live tile's identity.
PR #93269 (kshitijk4poor) landed the outer-handler break for the same
symptom while this branch was in flight. Keep his guard (it covers
shutdown errors from local post-processing and does the resume-hint +
best-effort persist) and keep this branch's inner-retry-handler return
(it fires BEFORE the ⚠️ retry trace, credential rotation, and fallback
attempts that the outer handler never sees). Point his
_is_interpreter_shutdown_error at tools/interpreter_shutdown.py so the
class has exactly one text-matching site, preserving his RuntimeError
type gate and all 7 of his tests.
When the TUI exits while the post-turn background review fork is still
mid-request, every further API attempt raises 'cannot schedule new
futures after interpreter shutdown'. The conversation loop treated this
as a retryable API error: un-gated ❌ prints leaked onto the user's
shell AFTER the TUI exited (call #4, #5, #6...) and the loop retried a
doomed request until the interpreter froze the thread.
Fix the class, not the site:
- tools/interpreter_shutdown.py: single shared shutdown predicate
(matches both CPython message variants + sys.is_finalizing()).
- cron/scheduler.py, agent/tool_executor.py: existing per-site
predicates now delegate to the shared home (tool_executor previously
matched only the fuller variant).
- agent/conversation_loop.py: inner retry handler recognizes the
shutdown signal and abandons the turn — one log warning, no print,
no traceback, no debug dump, no retry; outer handler gets the same
guard for shutdown errors raised outside the API call.
- The outer handler's bare print() now honors suppress_status_output
(set by the background-review fork) instead of bypassing it.
Refs #55924#58720 (same class in cron delivery), adjacent to #90683.
#90587 replaced the bundled Nous with a GitHub-theme fork. The earlier
glass-and-cream palette stays available under its own name; the default
does not change.
The status stack hydrates the goal indicator on every session open via
/goal status. A finished goal stays status=done in the DB permanently,
so hydration re-created the '✓ Goal done' chip on every mount — the 8s
linger only applies to the live completion event, not re-hydration. In
Bot Mode (one endless session) the completed overlay never went away.
applyGoalStatusText now takes a hydrate flag: terminal (done) goals are
treated as no-goal during hydration and any lingering done chip is
dropped, while live events keep the 8s linger behavior.
Test-helper + regression test from PR #92633 by @DavidMetcalfe:
mock _is_supervised_gateway_process instead of setting the raw env
var, and add a test proving a CLI agent session with inherited
_HERMES_GATEWAY=1 but no PID ownership is no longer blocked.
The terminal tool lifecycle guard and the gateway stop/restart CLI
guards keyed on the raw _HERMES_GATEWAY=1 env marker, which every
gateway descendant inherits (and importing gateway.run sets it too).
CLI/TUI agent sessions were falsely blocked from documented gateway
management commands. Gate on _is_supervised_gateway_process() instead,
which requires owning the live gateway PID file.
Salvaged from PR #92196 (guard half) by @nbxuhk. Fixes#92560.
Follow-up to the #91701 salvage: persistent registrations survive a routine
unload-all, but must not outlive their plugin.
- Targeted unload (plugin disable/uninstall) now gathers persistent rows
from the ownership ledger and disposes them.
- Unload-all parks live persistent handles in _persistent_carryover;
discover_and_load(force=True) evicts the ones whose plugin did not
re-register the same (kind, key) — superseded handles are dropped
without disposal so a same-object re-registration stays live.
The dashboard auth registry is process-global, but a bundled auth provider
was registered under the per-home plugin manager's scope and enrolled in that
manager's reverse-order teardown. A per-home manager is unloaded routinely
(profile-scoped dashboard activity, forced re-discovery), and that teardown
disposed the registration — emptying the auth registry for the whole process
and permanently disabling sign-in until restart.
Register dashboard-auth providers in the process-global slot as persistent
host-owned registrations kept out of per-home manager teardown, so a routine
unload can no longer disable authentication process-wide. Registration upserts,
so a forced re-discovery (e.g. a password change) still rotates the provider in
place. The test-only manager reset now clears the auth registry too, since
persistent registrations deliberately survive unload.
Fixes#91701
The protected agent-instruction gate grants one operation and persists
nothing, but only the TUI/desktop and Runs transports were taught that.
The prompt_toolkit panel, the input() fallback, and the ACP editor menu
still rendered "Allow for session", so a user editing SOUL.md tapped it,
got re-prompted on the next write, and read the gate as broken.
Thread allow_session through prompt_dangerous_approval so a caller that
re-asks every time collapses every surface to once/deny, and cover the
producer-to-transport contract end to end.
When the Python interpreter begins teardown (user closes hermes, SIGTERM,
OOM-kill), every executor-backed operation raises 'cannot schedule new
futures after interpreter shutdown'. The outer except handler in
run_conversation caught this error but did not recognize it as fatal —
it kept retrying (API calls #4, #5, #6) until max_iterations, each time
hitting the same dead executor and printing another traceback.
The fix adds an early check: if sys.is_finalizing() or the error matches
the 'cannot schedule new futures' pattern, break immediately with a clean
interpreter_shutdown exit reason instead of retrying. The codebase already
had this pattern in cron/scheduler.py and agent/tool_executor.py — the
conversation loop just wasn't using it.
Drives the real `syncBotsHomeWorkspace` against a shell whose reveal
does not hand the tab its zone's active slot — modelled on
`revealTreePane`'s hidden-pane early return — and asserts the passive
reconcile re-fronts once rather than on every pass. Against the
unfixed code the first case remounts the view 21 times for 21 passes.
The other three keep the bound from becoming a regression of its own:
giving up must leave the tab OPEN (a closed home drops the Bots tab
through to the ownerless Sessions composer); a cooperative shell still
gets its legitimate re-front, and gets another one the next time the
tab is genuinely backgrounded, so the budget is per-attempt rather than
one-shot for the life of the process; and an explicit gesture re-fronts
even on a shell where passive reconciles have already given up.
Re-fronting the Bots home tab is a close followed by a re-open, which
tears down and rebuilds the entire Bots view. `openBotsHomeWorkspace`
took that path on EVERY passive reconcile that found the tab open but
not holding its zone's active slot, with nothing bounding the retries.
That condition is not always transient. `revealTreePane` returns early
for a pane in `$hiddenTreePanes` without ever activating it,
`isPaneVisible` is false for a minimized zone, and a pane the tree
never adopted has no group to be active in. Pinned in any of those
states, every signal that reaches a surface sync — sidebar visibility
flips, focus churn, group changes — bought one more full remount, and
the view visibly strobed.
A passive reconcile now gets one attempt. The reveal has already
granted or refused the active slot by the time `openWorkspace` returns,
so the budget settles on that answer directly instead of waiting for a
visibility notification that is not coming: a computed store stays
silent when the value does not change. Retiring the tab starts a fresh
budget, and an explicit gesture is never blocked.
Giving up keeps the surface rather than closing it — a closed home
drops the Bots tab through to the ownerless Sessions composer, which is
the hole the home exists to plug.
Covers the behavior, not the markup: a row exposes an activation target
that opens THAT job; the opener never contains the switch or the delete
control (a nested interactive element would swallow the toggle and is
invalid markup anyway); detail rows carry only fields the gateway
actually sent, so a job that has never run drops those rows instead of
rendering "undefined"; a paused job reports Paused and promises no next
run; the raw schedule appears only when the humanized label dropped
something; and a failing job explains itself in failure order — the run
that never happened outranks the delivery of a run that did.
`routine-owner.test.mjs` asserted the row's owner routing by matching
`function RoutineRow({ job, owner })` against the plugin source, so it
broke on a parameter addition that changed no behavior. Replaced with
the real invariant it was reaching for: toggling the switch sends
`cron.manage` for the owner that rendered the row and evicts that
owner's cache key — which a signature change cannot fake.
In the Bots pane the Cronjobs rows were inert. The only interactive
controls were the enable switch and the hover-only delete button, so
clicking a cronjob to see what it runs, when it runs next, or why it
stopped did nothing at all — while the same job on the main Cron page
opens a full detail panel.
The gateway already ships every one of those facts with
`cron.manage list` (schedule, repeat, next/last run, last status,
delivery target, model, workdir, prompt preview, and the
fire/delivery/pause failures). None of it had a surface in Bot Mode: a
job failing every run reads exactly like a healthy paused one.
The row title becomes a real button that opens a read-only inspector
rendered from the record the pane is already holding — no extra RPC,
and no second mutation path beside the row's own switch and delete. The
switch and delete button stay siblings of the opener, so a toggle can
never be swallowed by the open. The inspector tracks the job by id
rather than by object, so the 20s poll keeps an open panel live instead
of freezing the snapshot it opened with.
- Clear the accidental end stamp on resurrection (at the lineage tip):
a surviving ws_orphan_reap/agent_close reason made a LATER deliberate
archive auto-resurrect on the next lookup — the user could never retire
the canonical chat. Test pins the resurrect -> deliberate-archive ->
stays-archived cycle.
- Judge recoverability at the compression TIP: the registry row of a
compressed lineage carries end_reason='compression', so tip-stamped
accidents were unrecoverable through the registry row. Lineage test.
- Heal the third lookup: the api_server exact-title listing (hermes peer
dm resolution) filtered archived rows out via list_sessions_rich and
still failed for reap-archived canonical chats.
- Single source of truth for the recoverable set: tuple moved to
hermes_state_common (mirroring _RESET_END_REASONS_SQL) and interpolated
into all three recovery SQL sites — literals cannot drift.
- methods_session gate uses BOT_CHAT_TITLE (not a literal) and re-fetches
by id after resurrection (title has no DB-level UNIQUE).
- Idempotence pinned: two consecutive profiles.list calls both resolve.
Final-diff pass: trimGroupChatLog drops entries from the FRONT once a room
crosses the history cap, so slicing the post-turn log at the pre-turn
LENGTH could overshoot after a mid-turn trim, read an empty tail, and
silently commit a stale turn — re-opening #93127's double delivery in
long-history rooms exactly. Anchor on the last pre-turn entry's id; if the
anchor itself was trimmed, every surviving entry is newer, so scanning the
whole log stays exact.
- Cross-thread supersession no longer discards finished work: an epoch bump
from a send in ANOTHER thread doesn't re-drive this thread's members (delta
filters are thread-scoped), so dropping the finished reply lost completed
work until someone revisited the old thread. shouldCommitMemberTurn now
drops only when a newer USER entry landed in the same thread; the caller
computes that from the log tail past the pre-turn length.
- '@all stop' now holds every member — it parsed to everyone:true with no
mentions and silently held nobody, the asymmetric twin of the tested
'@all resume'. classifyGroupHoldDirective gains holdAll; the send path
passes the room's member keys for expansion.
- Tests pin both: cross-thread commit preserved, @all-stop holds all
(mutation-checked: reverting either guard fails its test).
- Drop the false fairness claim from acquire_turn_lock's docstring (LOCK_NB
probe + sleep retry gives no arrival-order guarantee; only the budget is).
- logger.debug once when the lock degrades to a no-op on fcntl-less
platforms so silent serialization loss stays diagnosable.
- Document the real worst-case deliver handler hold (120s lock wait + 600s
turn = ~720s) where clients tune their timeouts against it.
- Pin non-reentry: local_delivery_command must stay a raw 'hermes -p' argv —
wrapping it in --run-delivery would make the child contend with its
parent's own flock and fail every relay delivery with target_busy.
- De-flake: the cross-profile test's upper-bound wall-time assert tolerates
loaded CI runners; the wait-duration message assert matches ~Ns generally.