Inference is now any args_hint without subcommands → text. Mixed is the
only remaining hint-token path. desktop= and the few argument_mode
overrides live on the registry entry; the side tables are gone. Catalog
aliases get their own dict copy. Composer tests seed the catalog so
/goal stays mixed without an overlay row.
New commands and plugins declare argument_mode and desktop availability
on CommandDef / register_command. commands.catalog ships that map so
desktop does not need a second command list.
Automatic Desktop ends (ws_orphan_reap, disconnect, idle, LRU, shutdown)
now drop the local runtime but keep the state.db row and durable-key
delegations when another live lease still owns the session.
Co-authored-by: metamindedu <metamind@kakao.com>
A live sibling serve sharing state.db is no longer treated as a dead process by the startup orphan sweep.
Covers the sweep half of #94895. The launchd Errno 48 KeepAlive loop is not addressed here.
Credit: @Finn763
Independent review finding on the merged #95980:
_apply_live_compression_config only acted on PRESENT keys, so removing
tail_mode / model.context_length / target_ratio / model_thresholds /
proactive_prune_* / protect_last_n / min_tail_user_messages / threshold /
idle_compact_after_seconds from config.yaml left stale values active in
live sessions forever (probe-verified: all six stale after applying
empty mappings).
Absence now restores the normalized default — or the model-derived
value — through the SAME derivation the construction path uses:
- ContextCompressor ctor defaults read off its real __init__ signature
(no hardcoded copies to drift)
- compression.threshold removal re-derives via agent_init's
_resolve_compression_threshold (Codex gpt-5.4/5.5 + spark autoraise
included)
- model.context_length removal drops the config override and forces
re-inference through the deferred get_model_context_length resolution,
which also re-applies the small-context threshold floor
- model_thresholds removal clears stale per-model overrides from the
live threshold; tail_mode falls back to the ctor's 'lean' (the old
present-key path normalized invalid values to 'legacy', diverging
from the compressor's own fallback)
Also fixes proactive_prune_min_reclaim_tokens's present-but-null default
(was 0; the real default is 4096).
Refs #94724
A hermes serve killed mid-update lost every un-flushed in-memory session
(#94724 item 2, reported by @ruangraung): the next RPC failed with
'session-scoped RPC rejected: not in memory (detached/reaped runtime)'
and no store held the transcript. #95576 made serves survive future
updates; this closes the kill path itself:
- install chaining SIGTERM/SIGINT handlers (hermes serve / dashboard
startup, before uvicorn's capture_signals) that first persist
in-memory session transcripts to state.db — bounded by
HERMES_TUI_EXIT_FLUSH_BUDGET_S (default 5s, daemon worker + join) so
a hung SQLite write can never block exit
- _shutdown_sessions (atexit) runs the same bounded flush FIRST, before
the slow per-session teardown a supervisor may SIGKILL mid-way
- the idle-reaper scan piggybacks a periodic incremental flush
(marker-deduped agent._persist_session, running sessions skipped) so
even a SIGKILL loses at most one flush interval — no new timer
subsystem
Refs #94724
The Bots editor's model write (profiles.configure) was the one switch
surface that bypassed the data-policy / expensive-model selection guard:
a guarded pick (e.g. muse-spark contributor tier) was applied silently,
with no confirm flow anywhere — the #95293 remainder after the core
picker's confirm handshake landed in use-model-controls.
Gateway: profiles.configure now answers confirm_required +
confirm_message for a guarded model (same handshake as config.set
model) and writes NOTHING until the client resends with
confirm_expensive_model: true. Other sections still apply; the pending
model section is not reported as failed.
Desktop: the confirm flow is extracted out of use-model-controls into
one shared applier (lib/guarded-model-switch.ts, exported through the
plugin SDK) — warning toast, staleness-guarded Confirm, single
confirmed resend, never a retry loop. The core picker and the Bots
editor now consume the SAME handler; the Bots editor's Confirm resends
only the model section with confirm_expensive_model: true.
Fixes#95293 (Bots surface remainder).
Sibling site of the same class fixed in the previous commit: a failed
profile-store open during _init_session fell back to _get_db(), hydrating
and persisting a named-profile session's cwd row against the launch
state.db. Fail closed instead — skip the hydration (log a warning) so
nothing ever reads or writes the wrong profile's store. The other
profile-store open sites (_db_for_profile, _ensure_session_db_row,
_session_db) already degrade to None/skip and were left as-is.
A deferred agent build for a named-profile session swallowed a failed
profile-store open (except Exception -> session_db = None), so _make_agent
silently bound the launch _get_db() handle and every turn bled into the
wrong profile's state.db exactly when the profile store was briefly
unopenable. Opening the named profile then looked blank.
Route the open through _open_profile_session_db, which raises a clear
'profile session store unavailable' error instead; the deferred build's
existing except path turns that into agent_error + an error event, so the
user gets a clear failure and no agent turn against the wrong store.
Salvaged from #90219 (hardening half), adapted to main's current
deferred-build/_transfer_db_to_agent structure.
Related to #87723 and #89789. #88532 covered SessionStore only.
* fix(tui_gateway): ask before queueing a guarded model picked mid-turn
config.set model on a running session cannot swap the agent in place, so it
stashes the pick in session["pending_model_switch"] and applies it at the
next turn start. That branch answered confirm_required=False without ever
running the selection guards.
A client that implements the confirm round-trip was therefore told no
consent was needed and never prompted. One turn later
_apply_pending_model_switch ran the guards with the stashed (unconfirmed)
flag, saw the warning, and dropped the switch by design. The model reverted
with no confirm ever offered, because the only moment a round-trip was
possible had already passed.
Evaluate the guards before stashing, where the client still has a live
response to turn into a prompt. Nothing is queued for an unconfirmed
guarded pick, so the session is left exactly as it was and the re-send
carrying confirm_expensive_model queues it for real. The apply-time check
stays as the backstop for guards that can only decide after resolution.
The data-policy guard keys on the model id alone, which is all this branch
can see before resolution. The cost guard returns None when pricing is
unknown and its models.dev lookup is allow_network=False, so calling it
early can only under-fire and never blocks the RPC thread.
* test(tui_gateway): pin provider forwarding, name the canonical confirm field
Two review follow-ups, no behavior change.
_pending_switch_selection_warning forwards `provider=provider or None`, but
nothing asserted it: a guarded model id fires the data-policy guard on the
model alone, so the existing tests passed with `provider` dropped entirely.
Record the kwargs instead. Dropping the argument fails the first test;
removing the `or None` normalization fails the second.
The confirm responses carry `warning` and `confirm_message` with identical
text, which reads like an accident. Name which one clients should read
(`confirm_message`; `warning` is the pre-confirm-era alias that
_apply_pending_model_switch already treats as a fallback) so the two do not
drift apart later.
Both raised by @Enough1122 in review.
The composer status stack polls `process.list` every 5s while a background
process row is on screen. `process.list` is session-scoped, so against a
runtime id the gateway no longer holds it returns 4001 "session not found".
`refreshBackgroundProcesses` swallowed *every* failure with a bare `catch {}`
commented "transient socket loss". A gone session is not transient: the poll
re-sent the same dead runtime id every 5 seconds for the lifetime of the
window. On one machine this produced 31,518 gateway rejections in a day
(vs 663 the day before), 18,614 of them against a single runtime id, and it
is what users see reported as "sessions stopped with a session not found
error" after an update.
The trigger is a reconnect, not the poll itself: anything that mints a fresh
runtime (gateway restart, the #94219 reconnect/replay work, an idle-reaped
pooled backend) strands the id the status stack is still holding, and nothing
in this path ever re-checked it.
Distinguish the two failure classes:
- 4001 / "session not found" is TERMINAL for that runtime id — latch the id
and stop polling it.
- A timeout or transport error is transient — keep retrying, since the
session may well still be alive. Misclassifying that direction would
silently freeze the status stack on a healthy session.
The latch is cleared when the status stack (re)binds a session id, so a
session that comes back under a fresh runtime resumes polling normally
rather than staying dark for the life of the app.
Also name the method in the gateway's 4001 warning. That line was added in
c305839442 "for diagnosability", but without the RPC name it cannot say WHICH
client call is looping — the reason this storm could not be attributed from
the logs alone. A ContextVar set in `handle_request` carries it; it is
diagnostic only and never used for authorization.
Tests:
- composer-status: 4001 stops the poll, a timeout does not, one gone session
never suppresses a healthy sibling, and a rebind resumes polling.
- tui_gateway: the rejection warning names the method.
profiles.list opened every profile state.db as a writable SessionDB,
which waits out write-lock patience while that profile's backend is
mid-turn. The desktop RPC timed out and Bot Mode's infinite React
Query retry kept the sidebar on a spinner.
Inspect those DBs read-only and bound roster retries so names still
paint.
Chat/event-plane quality work for the amended Phase 1 scope of #94484
(maintainer restructure: lean chat/event plane, no control-plane
changes). Three fixes came out of a source-level comparison against
OpenHands, Chainlit, VS Code, Zed, LangGraph, and Goose.
1. Per-turn trace_id + active-turn telemetry: _start_inflight_turn mints
a 12-hex trace_id; _event_frame stamps it on every event frame in the
turn, so a client can correlate the full lifecycle (dispatch -> first
token -> tool calls -> complete) from one identifier — none of the six
surveyed projects has frame-level turn correlation.
session.events.stats now reports active_turns (session_id, trace_id,
elapsed_s, streaming).
2. Transient vs durable events (OpenHands StreamingDeltaEvent pattern):
message.delta / thinking.delta are stamped with seqs (live ordering
holds) but never buffered — one streaming turn emitted hundreds of
delta frames and evicted every durable control event from the
512-slot ring, defeating replay for the exact reconnect window it
exists to cover. The ring now evicts manually and records the highest
DURABLE seq dropped, so truncated means real data loss and delta-only
gaps no longer false-positive.
3. Seq-namespace epoch (Goose stale-cursor recovery): event_replay.EPOCH
(8-hex per boot) is announced in gateway.ready and echoed by
session.events.since; the client drops its seq watermarks when the
epoch changes, so a stale HIGH watermark from a previous gateway
process can no longer suppress replay/gap-detection forever. Legacy
backends without an epoch are unaffected (client keeps watermarks).
Validation: Python 85/85 across replay/entry_ws/keepalive/protocol
(12 replay tests, 3 new); vitest 9/9 shared (2 new epoch tests), 66/66
desktop; full tui_gateway sweep 614/615 (1 known ordering flake, passes
in isolation). Live e2e on this tree: 12-event turn -> seqs contiguous
1..12, 8 durable frames buffered + 4 deltas live-only, single trace_id
on all frames, active_turns elapsed_s matches the real turn duration.
Research provenance: NousResearch/hermes-agent#94484 (comparative-scan
comment); techniques credited to OpenHands (transient split), Goose
(epoch/stale-cursor), per maintainer-restructured plan.
The function _persist_live_session_system_prompt runs on the RPC
dispatcher thread. A model switch calls it. On that thread the
_SESSION_CWD contextvar is not set. Because of this,
resolve_agent_cwd() falls back to the process TERMINAL_CWD value.
The desktop pins TERMINAL_CWD to the home directory. The rebuilt
prompt then records the home directory as the working directory.
The wrong prompt persists to the session database. Later turns
restore the stored bytes without change, because the turn prologue
rebuilds the prompt only when the cache is empty. The wrong line
never heals. The terminal tool is not affected, because it reads
the per-session cwd record.
The desktop composer sends a model switch before the first turn
when its model differs from the config default. Because of this,
a new project session can show the wrong working directory for
its full life.
Fix: bind the session context around the rebuild, in the same way
the function already binds the profile home for issue #50233. The
test drives the real function from a bare thread and asserts the
persisted prompt carries the session cwd.
Review F1: --ws-only is now OPT-IN (HERMES_DESKTOP_WS_ONLY=1). The slim
server has no HTTP routes while the desktop's hermes:api REST plane only
speaks http — shipping it on-by-default made every REST call fail with
'Unsupported Hermes backend URL protocol: ws:'. The descriptor keeps
baseUrl http-only and flags the transport via wsOnlyTransport instead of
overloading baseUrl. Default flips after the REST consumers migrate to
JSON-RPC (#94484 phase 3).
Review F3: entry_ws.run() fails closed on non-loopback hosts — the static
query-token auth is the desktop loopback shortcut, not a posture fit for
network exposure (no gated one-time-ticket auth).
Review F2 (test half): tests/test_tui_gateway_entry_ws.py drives a REAL
websockets-15.0.1 server + sync client through the actual handshake —
valid token accepted (path read from the v15 ServerConnection), missing/
wrong token rejected with 4401, plus the loopback guard both ways.
The desktop app spawns `hermes serve --ws-only` instead of `hermes serve`.
The slim server (tui_gateway/entry_ws.py) uses the bare websockets library
to call handle_ws directly — same dispatch surface, same 158 RPCs, same
events, zero HTTP framework. tui_gateway.server has zero FastAPI imports;
the slim server never touches web_server.py (19.8K lines) at all.
Full backward compat: older runtimes strip --ws-only and fall back to
regular serve (still headless, just through uvicorn).
The #94219 replay was a production no-op: the server returned full
JSON-RPC envelopes from session.events.since while the client's replay
loop dispatches only elements with a top-level 'type' — every replayed
event was silently skipped. Each side's tests validated its own
assumption, so both suites stayed green.
- server: events_since() now returns bare event objects (the frame's
params), the exact shape the live dispatch path consumes; ring stores
params directly; cross-language contract test added on both sides.
- client: live frames racing an in-flight replay are parked and flushed
seq-gated afterward — no double dispatch of deltas, no gap-skip from
a watermark advanced past the replay window.
- restart poisoning: seq counters are in-process, so a backend restart
reset them while clients kept high watermarks (replay forever empty,
truncated=false). New replay_epoch advertised in gateway.ready and
echoed by session.events.since; the client clears watermarks on epoch
change.
- methods_session no longer reaches into event_replay privates
(is_truncated() accessor).
Live repro: pre-fix, 3 stamped frames -> 0 dispatchable by the client
gate; post-fix 3/3. Tests: 16 py (replay+ws), 8 vitest, tsc clean, ruff
clean.
Server: per-session monotonic seq on every routed event frame, bounded
512-frame replay ring (64 sessions, FIFO eviction), plus two new RPCs —
session.events.since (replay newer-than-watermark, reports latest_seq +
truncated so clients detect gaps) and session.events.stats (telemetry).
Client: per-session seq watermarks recorded from live frames; after any
successful reconnect a fire-and-forget fetchReplay() drains missed events
through the normal dispatch path (recordSeq ignores non-increasing seqs,
so stale replay can never regress a watermark); focus-triggered reconnect
nudge in use-gateway-boot for the Electron unfocused case where macOS wake
skips visibilitychange.
Replay failures are swallowed by design: lossless resume is an upgrade
over the previous lossy reconnect, never a new failure mode.
Additive WebSocket wire contract for a client-driven heartbeat.
The gateway.ready payload now advertises "heartbeat": True so clients can
discover the capability, and the WS read loop answers a gateway.ping request
with a {"ok": True} pong short-circuited BEFORE method dispatch (no method is
invoked). The WSTransport gains closed / last_inbound_at properties and a
mark_inbound() hook, updated on every inbound frame, for later liveness checks.
Backward-compatible in both directions: old clients never send gateway.ping,
and old servers simply never advertise the heartbeat flag. This is the first
slice of a WebSocket-recovery series; the follow-on slices consume this
contract (server-side transport rebind, TUI/desktop clients).
Receipts:
bash scripts/run_tests.sh tests/test_tui_gateway_ws.py -q
=> 1 file, 7 tests passed, 0 failed (100%) in 0.8s; exit 0
Salvage hardening on top of #93601 (with #93597 covering the same core
mechanisms) for #93590:
- _hermes_cli(): after the venv-sibling check (hermes.exe on win32),
try shutil.which('hermes') before the bare-name fallback, so
environments with a PATH but no venv sibling resolve exactly what an
interactive shell would. Platform test switched os.name -> sys.platform
('win32') per repo convention.
- tui_gateway/methods_bot_relay.py deliver: pin encoding='utf-8',
errors='replace' on both subprocess.run sites — without them the
child's UTF-8 output is decoded with the locale codec (cp1252/GBK on
Windows), mangling non-ASCII replies or raising on undecodable bytes.
- Regression tests: shutil.which resolution step, bare-name fallback
with which=None, and encoding-pin assertions in the deliver transport
test.
Refs #93590, #93597, #93601
macOS sleep/wake (or a silent network drop) can leave the renderer's
WebSocket half-open: no close event fires, so connectionState stays
'open' while every RPC hangs until its per-call timeout. prompt.submit's
timeout is 30 minutes, so the user's next message reads as "enter does
nothing until I restart the app".
- Add a minimal ping RPC (tui_gateway/server.py) answered synchronously
on the WS reader thread.
- On wake signals, reconnectNow now probes the open-looking socket with a
5s-bounded ping and force-closes it on failure, letting the existing
reconnect machinery (backoff, tile rebinding, session refresh) take
over. A pre-ping backend answering -32601 is treated as healthy.
- Tests: half-open socket force-reconnects; healthy socket untouched;
method-not-found backend untouched; backend ping envelope contract.
Maintainer ruling (2026-08-23): a retried bot turn never mints a fresh
session. retry_action() maps the #93091 item-1 reason enum to one of
resume / compress_then_resume / none:
- transient classes (runtime_offline, delivery_timeout, rate limit,
server error) re-run the same Bot Chat session once;
- context_overflow also re-runs the same session — the retried turn
goes through the pre-API compaction pass in conversation_loop.py,
which compacts the over-threshold transcript first (the one
sanctioned context mutation); no fresh-session escape hatch exists;
- auth/quota/config/model classes never auto-retry.
Wired at both delivery surfaces (fix the class, not one site):
bot_relay.deliver (relay handler) and _run_delivery (local
message_agent runner). Failed deliveries now carry the classified
reason in the structured error payload (error.data.reason).
Sabotage-verified: with the retry blocks removed, 3 consumer tests
fail; with them present, 22/22 pass.
Without SO_KEEPALIVE a silently-dropped client (SSH tunnel reset, laptop
sleep, NAT timeout) leaves the TCP leg half-open forever: receive_text()
blocks indefinitely and the disconnect teardown (detach, orphan reap,
resume replay) never runs. The server then leaks the session and never
reclaims its orphans.
_disable_nagle already reaches the raw socket, so enable keepalive there:
SO_KEEPALIVE on, plus TCP_KEEPIDLE=30s / TCP_KEEPINTVL=10s /
TCP_KEEPCNT=3 on Linux and TCP_KEEPALIVE=30s on macOS. A dead peer is now
detected in ~60s instead of never. Best-effort like the Nagle tuning —
any failure to reach the socket is logged at debug and skipped.
Tests: new tests/tui_gateway/test_ws_keepalive.py fakes the socket and
pins SO_KEEPALIVE + the platform-specific idle tuning, plus the
no-transport no-raise path. tests/tui_gateway: 336 passed.
Reimplements the concept from #77129 on the current structure (viewer
rebinding from #83716 and _client_gone_interrupt_requested clearing are
preserved).
_close_sessions_for_transport snapshots owned sessions under
_sessions_lock, then wrote session['transport'] = _detached_ws_transport
WITHOUT re-checking that the session still pointed at the disconnecting
transport. A session.resume that rebinds the session to a new live
transport between the snapshot and the stomp got knocked back onto the
drop sentinel with an orphan-reap Timer armed against a client that is
attached right now.
The park now happens under _sessions_lock and first revalidates
ownership: if the session already moved to a different live transport,
the disconnect has nothing to tear down — skip the sentinel park AND the
reap scheduling. Regression test simulates the rebind landing between
snapshot and stomp.
Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
Follow-up to the #65422 salvage:
- startup_orphan_reap joins _RECOVERABLE_END_REASONS (kept distinct from
ws_orphan_reap for forensics): every recovery fence
(find_latest_gateway_session_for_peer, unarchive_recoverable_session,
promote_to_session_reset) now treats a startup-swept row as an
accidental end, so a sweep never makes a session unresumable.
- Config key moves from sessions.orphan_reaper to
dashboard.startup_orphan_sweep in DEFAULT_CONFIG, next to its siblings
ws_ping_interval / ws_ping_timeout / ws_orphan_reap_grace_s; the raw
loader in tui_gateway.server reads the new key (fail-open on missing).
cli-config.yaml.example and website/docs/user-guide/configuration.md
follow the dashboard.* documentation pattern.
- New regression test: a stranded 'active' row (ended_at NULL, no live
runtime) is swept AND still recoverable via peer-keyed lookup and fully
revivable via reopen_session afterward.
Close session rows left ended_at IS NULL when the in-process websocket
orphan timer dies with the process (#65194). Dual-clock staleness
(started_at AND newest message), desktop included, live in-memory
sessions excluded, scheduled once from both entry.main and the WS
sidecar so desktop/dashboard boots also run the sweep.
Messages sent into a session whose in-memory runtime was detached on WS
disconnect and orphan-reaped vanished silently: _sess_nowait returned
4001 with no log line, so 'request arrived and was rejected' was
indistinguishable from 'request never arrived' in a 'message vanished'
report. Log a WARNING with the session id and request id on every
session-scoped RPC rejected against an unknown runtime id.
Adds a regression test asserting the 4001 response and the warning.
Closes#90428
Review batch (3 reviewers) on the final diff surfaced:
- H1: title-based donor matching could adopt AND non-recoverably retire
an UNRELATED default-store conversation (bot titles collide by design;
get_session_by_title has no archived filter/ordering). Donor probe is
now exact-id only — the stranded repro always has the id.
- H2: re-adoption after a partial run could retire a donor that had
accumulated NEWER messages than the profile copy (skip-based
idempotency never merges). New divergence guard compares message
counts and refuses retirement when the donor is ahead (still adopts).
- M1: donor_retired reported True even when every retirement step
failed under suppress. Now per-segment tracked + warn-logged;
True only when all applied.
- M3: adopted=False (e.g. import validation limits) was silent — now
warn-logged with import errors.
- M4: archived donors are never re-adopted (no cross-profile cloning).
- Dead 'from pathlib import Path' dropped; contextlib no longer needed.
5 new red-first-verified regressions (title-collision immunity,
archived-donor immunity, non-vacuous owns_db gating with a real donor
seeded, divergent-donor retirement refusal, donor_retired truthfulness).
tests/tui_gateway: 578 passed. ruff clean.
Pre-#93296, the desktop routed session RPCs by the focused tile, so a
profile bot's turns executed on the default backend and its canonical
session accumulated in the DEFAULT profile's state.db. Post-fix, the
profile backend correctly receives the resume — but its store has never
seen the session, so the same chat 4001s forever (unreachable instead
of misrouted). Live repro: Teknium's Developer bot, session c93770.
- hermes_state_portability: SessionDB.adopt_session_lineage_from() —
composes the existing export_session_lineage()/import_sessions()
primitives; donor rows are archived (never deleted) with
end_reason=adopted_by_profile, which is deliberately NOT in
RECOVERABLE_END_REASONS so canonical-lookup resurrection cannot undo
an adoption. Idempotent (already-present ids skip).
- tui_gateway/methods_session: profile-scoped session.resume falls back
to adoption from the default store right before the 4007; ids unknown
to BOTH stores still 4007 exactly as before, and launch-profile
resumes never consult the fallback.
- tests: 10 new (7 unit on the primitive incl. compression-lineage
unit adoption + non-resurrectable archive; 3 handler-level through
server.handle_request incl. the live repro shape); db-ownership
leak test taught that the shared launch handle probe is by design.
Follow-up to #93296/#93311; part of #93091.
Live WS E2E after the #93361 merge (real web_server + tui_gateway, isolated
HERMES_HOME, 2s grace): drop socket -> re-resume stored id on a new socket
still produced a ws_orphan_reap reclaim. The lazy/unpersisted resume branch
(no state.db row yet -- every fresh Bot Chat) returned the sentinel-parked
live record without rebinding its transport or cancelling the armed reap
Timer, so the storm survived for exactly the Bot Mode sessions the cluster
targeted. The unit-covered paths (_live_session_payload, _reuse_live_response,
_claim_or_reuse_live) were all correct; this branch bypassed them.
Regression test drives the real session.resume RPC against a sentinel-parked
unpersisted record (sabotage-verified: fails without the fix). After the fix
the full live E2E passes 10/10 scenarios including a 4-cycle drop/resume storm
loop with zero reclaim broadcasts.
Server half of #91684: the desktop can answer an approval prompt with a
stale live sid — its runtime record was re-minted after a reconnect while
the prompt stayed on screen. approval.respond now falls back, on 4001
only, to resolving the target session (1) by the unique approval
request_id across every live session's pending gateway approvals, then
(2) by treating session_id as a STORED session id mapped to its live
runtime record. Only when neither resolves does it return 4001.
Tests: request_id fallback, stored-id fallback, and 4001 when nothing
resolves.
- New dashboard.ws_ping_interval / dashboard.ws_ping_timeout defaults
(20.0/20.0) in DEFAULT_CONFIG; hermes_cli/web_server.py reads them for
non-loopback binds. Loopback keeps ws_ping=None (event-loop stalls must
never kill a healthy local connection).
- New dashboard.ws_orphan_reap_grace_s (20.0): tui_gateway/server.py's
_WS_ORPHAN_REAP_GRACE_S now resolves from config via
_resolve_ws_orphan_reap_grace(); the HERMES_TUI_WS_ORPHAN_REAP_GRACE_S
env var is kept as an internal override for backward compat and wins
when set.
- tests/test_ws_keepalive_config.py: real load_config against a temp
HERMES_HOME yaml — defaults, propagation, deep-merge, env override,
invalid-value fallback.
Storm killer for the reap->broadcast->auto-re-resume feedback loop:
- New _pending_ws_reaps registry (sid -> Timer): _schedule_ws_orphan_reap
registers, _reap pops, and _cancel_ws_orphan_reap(sid) is called from
every resume/reuse/rebind path — the session.resume fast-path reuse
(methods_session.py), _claim_or_reuse_live winners, and the
_live_session_payload live-transport rebind.
- When a resume mints a fresh runtime for stored session id S, any prior
runtime for S still parked on the detached-WS sentinel is claimed under
the resume lock, its reap Timer cancelled, and the record finalized
quietly with end_reason superseded_by_resume — NOT in
_RECLAIM_END_REASONS, so no session.reclaimed broadcast fires and the
client's auto-re-resume can't storm.
- superseded_by_resume added to _RECOVERABLE_END_REASONS in
hermes_state_common.py so canonical Bot Chat resurrection still applies.
Unit tests: resume cancels the reap timer, superseded runtimes finalize
without a reclaimed broadcast, and the normal orphan reap still fires
when nobody re-resumes.
Live sessions hold one transport; a pop-out window's session.resume
rebinds it, and on pop-out close the disconnect path parked the session
on the drop sentinel — the original window never received stream events
again until a manual re-resume (#83716).
Sessions now track every transport that has shown them (viewers, stamped
in _live_session_payload). _close_sessions_for_transport re-binds to the
most recent surviving viewer instead of detaching when one exists; dead
viewers are filtered; the drop sentinel + grace reap remain the path for
the last viewer. Root cause and repro by CharlesR-sudo on #83716.
If an interrupted turn never settles (agent thread hung in a syscall,
supervisor lost), the 1s poll chain rescheduled forever — trading the old
leak-one-worker bug for leak-one-session-plus-timer-chain. After
_WS_ORPHAN_INTERRUPT_REAP_MAX_POLLS (60 = ~60s, 3x the default grace) the
reaper logs loudly and force-reaps, mirroring the pre-existing stuck-running
safety net's deadlock-breaking role. Focused suite 4 passed, 1 skipped.
After the existing reconnect grace, route a still-detached running session through the same interrupt mechanism as session.interrupt. Preserve delegation deferral, sidecar teardown, partial history, and single-owner reap semantics.
Verified on upstream main: RED 4 failed/2 passed without production changes; GREEN 603 related gateway/compute-host tests. Ruff and py_compile passed. Momus pass 2: APPROVE.
The grace timer treated mid-turn detached sessions as not-orphaned and
returned without arming another timer. Eviction paths also skip
running sessions, so the in-memory agent leaked until process restart.
Fixes#85578
/review takes the last 10 chat messages plus optional instructions,
spawns a full-privilege background subagent (the async delegation
rail) that investigates the referenced work (PR, code, docs), and its
complete review re-enters the spawning session as a normal
async-delegation completion the primary agent can act on.
- agent/review_engine.py: shared engine (snapshot, briefing,
auxiliary.review credential resolution, dispatch, note formatting)
- tools/delegate_tool.py: internal credentials_cfg per-call override
(never model-facing) resolved through the same credential system as
delegation.provider pins
- auxiliary.review config block (provider/model/base_url/api_key/
api_mode); provider auto + empty model = inherit the main model
- Surfaces: CLI process_command, gateway run.py dispatch +
slash_commands handler (binds the approval session key so the
completion routes back), TUI/Desktop live dispatch in
tui_gateway/server.py, CommandDef registry (+Slack /hermes-only cap)
- Docs: delegation.md section + slash-commands.md (both tables)
- Tests: 15 engine tests (sabotage-verified: credentials_cfg and
dispatch tests fail without the fix), 4 gateway handler tests
through the real async rail
- Clear the accidental end stamp on resurrection (at the lineage tip):
a surviving ws_orphan_reap/agent_close reason made a LATER deliberate
archive auto-resurrect on the next lookup — the user could never retire
the canonical chat. Test pins the resurrect -> deliberate-archive ->
stays-archived cycle.
- Judge recoverability at the compression TIP: the registry row of a
compressed lineage carries end_reason='compression', so tip-stamped
accidents were unrecoverable through the registry row. Lineage test.
- Heal the third lookup: the api_server exact-title listing (hermes peer
dm resolution) filtered archived rows out via list_sessions_rich and
still failed for reap-archived canonical chats.
- Single source of truth for the recoverable set: tuple moved to
hermes_state_common (mirroring _RESET_END_REASONS_SQL) and interpolated
into all three recovery SQL sites — literals cannot drift.
- methods_session gate uses BOT_CHAT_TITLE (not a literal) and re-fetches
by id after resurrection (title has no DB-level UNIQUE).
- Idempotence pinned: two consecutive profiles.list calls both resolve.
- Drop the false fairness claim from acquire_turn_lock's docstring (LOCK_NB
probe + sleep retry gives no arrival-order guarantee; only the budget is).
- logger.debug once when the lock degrades to a no-op on fcntl-less
platforms so silent serialization loss stays diagnosable.
- Document the real worst-case deliver handler hold (120s lock wait + 600s
turn = ~720s) where clients tune their timeouts against it.
- Pin non-reentry: local_delivery_command must stay a raw 'hermes -p' argv —
wrapping it in --run-delivery would make the child contend with its
parent's own flock and fail every relay delivery with target_busy.
- De-flake: the cross-profile test's upper-bound wall-time assert tolerates
loaded CI runners; the wait-duration message assert matches ~Ns generally.