_read_loop opened with 'assert self._ws is not None' — an exit that
escaped BEFORE the finally that fails pending futures, contradicting
the 'fails pending on ANY exit path' invariant the hardening commit
established. Production currently assigns _ws before scheduling the
reader, so this was latent, but any future lifecycle change hitting it
would strand every in-flight waiter for the full outbound timeout with
only an AssertionError in the logs.
Turn it into a guarded early-return INSIDE the try: the reader logs
the lifecycle bug and unwinds through the same finally as every other
exit, settling all waiters. Regression test drives _read_loop with
_ws=None against a registered pending future.
The mid-redial fail-fast guard used 'supervisor task not done' as the
definition of the redial window, and that signal is wrong from both
directions:
- Too narrow: the wedge it fixed also occurs on reader exits that arm
NO supervisor (terminal 4401 revocation, reconnect=False) — those
stayed wedged.
- Too broad: _reconnect_loop -> _dial_and_start installs the fresh
socket and starts its reader, THEN awaits one hello send per fronted
identity before the supervisor unwinds. Through those awaits the
transport is fully live, yet the guard rejected every send as
'reconnecting' — refusing real traffic on a healthy socket.
With the previous commit the reader clears _ws on unexpected exit, so
the existing 'is None' check now covers the entire outage window
honestly: _ws is the single liveness signal. Drop the supervisor-state
guard.
The redial-window test now drives the real sequence (reader exit arms
the supervisor and clears _ws) instead of hand-crafting a stale-_ws
state the transport can no longer reach, and a new test pins the
post-dial window: a send issued while the supervisor is still
unwinding past a live socket must reach that socket (fails with the
guard reinstated).
Two reader-exit paths arm NO reconnect supervisor: a terminal 4401
revocation (deliberately never re-dials) and reconnect=False
transports. On both, _ws kept pointing at the dead socket after the
reader unwound, so the 'is None' liveness guard reported connected and
a send registered a future nothing could ever resolve — the full
_outbound_timeout_s (~30s) wedge this PR exists to eliminate. The
revocation case is the sharpest: the fatal-error notification that
path emits is itself an outbound send, so it ate the stall.
Null the handle in the reader's finally, identity-guarded (only if _ws
still points at the socket THIS reader served) so a supervisor re-dial
that already installed a fresh socket is never clobbered, and gated on
not _closing so disconnect() keeps sole ownership of teardown.
This also makes _ws the single honest liveness signal for the redial
window itself — groundwork for retiring the supervisor-state send
guard, which misreads that window from both directions.
Regression tests cover both uncovered paths, asserting _ws is cleared
and a post-drop send fails fast; both wedge (fail) with the finally
reverted.
The dedupe tests validated hand-built events only, so a key that read a
field the production event type doesn't have still went green — and the
'all 7 new tests fail against base' mutation claim didn't hold either
(the fail-open and distinct-message tests pass against base because a
no-op dedupe trivially satisfies both).
Add a wire-level class that decodes a connector frame with
_event_from_wire and dispatches it through _on_inbound — the exact
production path — asserting: a decoded event yields a dedupe key at
all, a re-delivered frame is dropped (fresh decode each time, so
identity must come from the key, not the object), and identical
chat/message ids on two different platforms are NOT conflated
(Phase 1.5 multiplex).
Mutation-checked: with the previous event-shape key reinstated, the
wire tests fail (3 failed); with the fix, all 7 pass.
The dedupe key read event.chat_id — a field MessageEvent does not have
(chat identity lives on event.source.chat_id; see how every other read
in this adapter resolves it). getattr defaulted to None, so
_inbound_dedupe_key returned None for EVERY production event: the
fail-open branch always taken, the seen-set permanently empty, and a
replayed inbound still re-ran the whole turn. The tests passed because
their SimpleNamespace events carried a top-level chat_id no production
code path produces.
Read the chat id from event.source, and join the underlying platform
into the key: one relay adapter fronts several platforms (Phase 1.5
multiplex), and two platforms' numeric chat/message ids must not
collide into one replay identity.
The test event factory now builds real MessageEvent/SessionSource
objects in the wire decoder's shape, so the replay and bounded-set
tests fail against the broken key instead of green-lighting it.
Live-canary finding #3 (Alice, staging): the relay inbound leg is
at-least-once. On WS re-handshake the connector replays its durable
per-instance buffer; a long multi-tool turn (60-100s) straddling a quiet
socket drop got its ORIGINAL inbound replayed after the turn finished,
re-running the entire turn — the user saw the final answer posted 2-5x
(each a separate execution, hence slightly different texts). Receipts:
same msg text at history=0 in back-to-back sessions 121647/121840, no
Slack-side retry on the connector (envelope dedupe never fired).
Consumer-side idempotency: bounded FIFO seen-set (512) keyed by platform
message identity; events without a message_id never dedupe (fail-open —
dropping a real message is worse than rerunning one). No wire change;
contract v1 untouched.
Transplanted-from: victor-fork/feat/relay-slack-live-cards@73ce04ae75 (extracted for the rc.4 relay-fixes train; tests moved to a standalone file with no live-cards dependencies)
Customer gateways cross WAN paths to the connector; the websockets
library defaults (ping 20s, pong deadline 20s) close the socket with
1011 keepalive ping timeout under transient latency or event-loop
stalls — the trigger of the Coatue 2026-08-18 incident. 60s tolerates
stalls while still detecting a genuinely dead link within ~90s worst
case. Set explicitly at both connect() call sites (with and without
auth headers) so the tuning is visible and pinned by test rather than
inherited from library defaults.
Between an unexpected close and the supervisor's successful re-dial,
self._ws still points at the DEAD socket, so the existing None-check
does not cover the backoff window: a send registered a future no reader
could resolve and blocked the caller for _outbound_timeout_s.
A live supervisor task is exactly the redial window (the loop returns
when a dial succeeds and its fresh reader takes over), so no new state
is needed: when self._supervisor is not done, return the error dict
immediately ({success: False, error: reconnecting}). Callers already
handle failed sends; a fast, honest failure beats a 30s wedge
(Coatue 2026-08-18).
The reader is the only thing that can resolve a pending outbound_result
future. When the socket dropped unexpectedly (vs a deliberate
disconnect(), which already failed pending), every in-flight
_request_response waiter blocked the full _outbound_timeout_s (~30s) on
a future no reader could resolve.
Coatue incident 2026-08-18: a 1011 keepalive ping timeout close left
final-send and error-notification calls wedged against the dead socket,
holding sessions active while the connector replayed inbound backlog.
Fail every not-done pending future from _read_loop's finally with the
dict shape callers expect ({success: False, error: connection lost}) —
never an exception on the outbound path — then clear the map. list()
snapshot avoids mutation-during-iteration from woken waiters' finally
pops.
Third CI hit today for tests/gateway/test_goal_verdict_send.py (twice on
salvage PRs, once on main's own push run), always the same shape:
adapter.sends == [] after the full drain.
Mechanism (reproduced, not log-read): the tests call GoalManager.set() on
the event-loop thread. _get_session_db() refuses to construct SessionDB on
a loop thread (loop-liveness guard from the 2026-08-14 crash-loop fix) and
waits only _DB_BOOTSTRAP_LOOP_WAIT_S=0.25s for the background bootstrap.
On a loaded CI runner the init overruns that window, set() degrades to a
silent no-op by design, the goal never exists, and the continuation path
correctly does nothing — so no amount of drain-waiting helps (the #88975
de-flake addressed a different, downstream race).
Fix: the hermes_home fixture pre-warms the SessionDB cache from its sync
context (direct construction path), so the loop-thread set() always finds
a cached DB. Production behavior untouched.
Proof: injecting a 0.4s-slow SessionDB.__init__ reproduces the exact CI
failure on the old fixture and passes 2/2 with the pre-warm.
Adds a reviewable in-app install path for Hermes plugins:
hermes://plugin/install?repo=owner/repo (and Settings -> Plugins ->
Install from Git) opens a confirmation modal showing the repo identity
and source links, shallow-clones to probe for agent and/or desktop
plugin artifacts, lets the user pick components, then installs — agent
side through the gateway's new plugins.manage `install` action (wrapping
the existing dashboard_install_plugin), desktop side through a new
Electron git-install module with subdir-escape guards, a 60s clone
timeout, non-interactive git env, and insecure-scheme warnings. Never
auto-installs; hybrid repos get one dialog. Legacy plugin-agent /
plugin-desktop deeplinks route into the same modal.
Salvaged from PR #82735 by @serefyarar (net diff applied onto current
main as a single authored commit; the branch carried merge commits).
The preview screenshot PNG from the original branch was intentionally
not carried over — images live in PR bodies, not the repo.
Follow-up to the salvaged CommandCode signature fix: accepting base_url
but ignoring it left custom endpoints (user-configured model.base_url /
COMMANDCODE_BASE_URL proxies) fetching the public catalog instead of the
configured one. Reviewer dansigma flagged this on PR #88851.
Class-wide fix, not a CommandCode patch:
- providers/base.py: a caller base_url that DIFFERS from the profile's
default now wins over models_url. Equality with the default means "not
customised" (callers pass base_url unconditionally, defaulting to the
profile's own URL) and keeps models_url as the endpoint, preserving the
OpenRouter-style split-catalog behavior.
- commandcode: _fetch_commandcode_models() takes the endpoint override;
both profile overrides forward base_url.
- Tests: base-class precedence (custom beats models_url, default does
not), CommandCode redirect via live local HTTP server incl. claude-*
filter, and default-echo hitting the canonical endpoint. All verified
to fail against the pre-fix implementation (sabotage run).
Plugin-only providers (commandcode, tencent-tokenhub, ...) are absent from
models.dev and HERMES_OVERLAYS, so resolve_provider_full returned None and
/model switches failed with "Unknown provider ..." even though the picker
lists them (CANONICAL_PROVIDERS auto-extends from the same registry).
Fall back to providers.get_provider_profile() before giving up, mapping the
profile api_mode to the ProviderDef transport.
The model picker's generic live-fetch path (hermes_cli/models.py
provider_model_ids) calls profile.fetch_models(api_key=..., base_url=...).
Both CommandCode overrides only accepted api_key/timeout, so every picker
open raised TypeError, which was silently swallowed, leaving the provider
with zero models.
Match the base ProviderProfile.fetch_models signature (base_url kwarg) and
add a regression test asserting both profiles accept it.
PR #67934 marked auto-discovered catalogs by writing two sentinel keys
INSIDE the user-facing ``models`` mapping of custom provider entries:
``__discovered_model_catalog__`` (written by
_save_discovered_models_to_config) and ``__explicit_model_allowlist__``
(injected by _normalize_custom_provider_entry). Every consumer of that
mapping — pickers, selectors, gateway/agent readers, and the user's own
config.yaml — had to know to filter those keys, and any site that
didn't listed them as phantom model IDs (``__discovered_model_catalog__``
showing up as a selectable "model"). The v11→v12 config migration and
the ACP session-state test caught exactly that leak on main.
Replace the in-mapping sentinels with a single entry-level flag:
- ``models_discovered: true`` now sits next to ``models``/``base_url``
on the provider entry; the models mapping stays a clean
``{model_id: metadata}`` dict with no reserved keys.
- _save_discovered_models_to_config writes the new shape and refreshes
catalogs it previously discovered (entry-level flag or legacy
sentinel) instead of treating them as user-curated metadata.
- _normalize_custom_provider_entry no longer injects
``__explicit_model_allowlist__``; a dict-shaped models mapping counts
as an explicit allowlist exactly when the entry is NOT marked
models_discovered.
- _models_config_is_allowlist takes the discovered flag as a parameter
(new helper _entry_models_discovered resolves it, including the
legacy in-mapping sentinel); all call sites updated
(model_switch.py, model_setup_flows.py, acp_adapter/server.py).
- Backward compat, no config version bump: configs written by a
pre-fix Hermes (sentinels inside models) still read correctly —
``__discovered_model_catalog__: true`` is treated as
models_discovered, both sentinel keys are stripped from model
listings, and the next discovery save migrates the entry to the
clean shape. Covered by a new regression test.
Also restore ``except Exception:`` on the pre-existing guards this PR
had narrowed to specific exception tuples (the resolve_runtime_provider
fallback in switch_model, the picker discovery/cache guards in
list_authenticated_providers, _get_model_config_dict, and
_credential_fingerprint). Those guards were intentionally broad on
main — a failed resolution or probe must degrade to the fallback path,
never crash the model switch. Guards the PR introduced for its own new
probe code keep their authored tuples.
The ACP new_session payload also goes back to
probe_current_custom_provider=False, matching the contract main's
test_new_session_returns_authenticated_cross_provider_model_state pins
(session opens must not block on live-probing the current custom
endpoint).
tools/skills_sync.py bound HERMES_HOME / SKILLS_DIR / MANIFEST_FILE at
import time — the third module in the same lineage as skills_tool
(f8723c478) and skill_manager_tool (c6a3d412d). In a long-lived
dashboard/TUI process, console skills commands (reset, diff,
list-modified, opt-in/out, repair-official) dispatched in-process under
_profile_scope's set_hermes_home_override(), but skills_sync's frozen
constants kept resolving against whichever profile was live at import.
Sharpest edge: reset_bundled_skill()'s #48200 rmtree strict-child guard
was computed against the WRONG skills root.
Fix: same call-time accessor pattern as the two prior fixes —
_hermes_home()/_skills_dir()/_manifest_file() honor an explicitly
patched module global (tests, retargeting) and otherwise re-resolve
from the live profile-scoped get_hermes_home() on every call. All 37
call sites migrated; module constants kept for compat.
Also documents in _profile_scope() that skills_sync needs no module
retargeting since the contextvar override now reaches it.
Regression tests (sabotage-verified: all 3 fail on the old binding):
- accessors follow set_hermes_home_override at call time
- explicit module patch still wins over the override
- rmtree guard anchors on the overridden profile's skills root
Fixes#65828
_write_manifest still used a hand-rolled mkstemp + atomic_replace,
so every sync reset .bundled_manifest to mkstemp's 0600, dropping a
group-readable or shared mode the operator had set. Replace the block
with utils.atomic_write_text(preserve_mode=True) — the same shared
writer and mode-preservation contract PR #86255 applied to the skill
manager's document writes.
This is the remaining half of PR #14410 by @sgaofen, who reported the
manifest mode reset first; the skill-manager half of that PR was
superseded by the atomic_write_text refactor and #86255.
Port the stale PR #61665 behavior to current main. The original two-file contribution is by yungchentang; this candidate preserves its scoped Codex OAuth fallback intent.
Co-authored-by: Taneli Mielikäinen <taneli.mielikainen@iki.fi>
The getAllTranscripts resourceData.id from the field report is a base64url
blob whose DECODED payload ends in "-TranscriptV2" while the encoded form
contains no readable marker, so the substring heuristic in
looks_like_transcript_id missed it. Add a best-effort base64 decode hint so
degraded notifications (no @odata.id) are still refused with the clear
guidance error instead of a cryptic Graph 400.
`hermes profile rename default <name>` (and the Desktop/dashboard rename
flows) now set a presentation-only `display_name` in profile.yaml instead
of erroring. The canonical id stays "default"; resolution, comparison,
and spawn paths are untouched. Named profiles keep real renames and their
display_name survives the move.
Surfaces: profile list/show/status, /profile (text only — data.profile
stays canonical), dashboard ProfilesPage, TUI-gateway profiles.list, and
Desktop (rail, switcher, Manage page, and the Bot Mode roster via a
displayName fallback so a renamed default shows its name, not "default").
Slimmer redo of the direction in PR #87760 by @yxssxn — thanks; see PR
body for what changed vs that approach.
test_real_aiagent_builds_section_once_and_keeps_it_out_of_static_prefix
builds the system prompt twice and asserts byte equality, but the prompt
embeds build_coding_workspace_block() — live git status/log output. A git
call failing between the two builds (xdist contention in CI) makes the
Branch/Recent-commits lines differ and fails the test on unrelated PRs
(first seen on #89027's run: diff showed only '- Branch: (detached HEAD)'
and recent-commit lines).
Mechanism reproduced locally: with coding posture on and cwd inside the
checkout, failing git calls on the second build only => first != rebuilt;
with the snapshot pinned, identical git failure => byte-equal. The real
block's byte-stability is coding_context's own contract; this test is
about plugin sections.
Per review: even with an active sandbox env, spilled tool results
belong in $HERMES_HOME/cache/spillover with the other Hermes-owned
caches — not the sandbox temp dir as primary storage.
- Host-side write happens first on every backend; local/no-env
sessions reference the host path directly (unchanged).
- cache/spillover joins the auto-mount/sync cache-dir list
(credential_files._CACHE_DIRS), so docker bind-mounts it and
modal/ssh/daytona file-sync it. Remote references use the
translated in-sandbox path after a readability probe.
- Probe failure (persistent containers created before spillover
joined the mount list, translation failures) falls back to the
previous in-sandbox temp-dir copy, so nothing regresses.
The steer-survives-budget tests pinned the inline 'Truncated:' fallback
shape, which only occurred because env=None persistence was broken.
Now that host-side spillover succeeds, budget enforcement produces a
<persisted-output> block instead. Assert the actual contract — the
oversized payload was replaced (persisted OR truncated) — via a shared
helper, not which replacement shape was used.
Sessions that never ran a terminal command (MCP-only, cron, gateway)
have no active sandbox environment, so maybe_persist_tool_result()
got env=None and fell through to the inline-truncate fallback --
a 467K MCP result was cut to a ~1.3K preview with no file written
('Full output could not be saved to sandbox').
Now the host-side cases (env=None or the local backend) write the
spill file directly to $HERMES_HOME/cache/spillover/<id>.txt,
alongside the other Hermes-owned caches instead of littering /tmp.
Remote backends (docker/ssh/modal/daytona) keep the in-sandbox
env.execute() write since read_file resolves in-sandbox there.
Cleanup: the gateway housekeeping loop prunes spillover hourly with
the other media caches, and a once-per-process best-effort prune on
first spill covers CLI-only installs.
test_session_chat_stream_treats_pre_existing_poisoned_row_as_no_model
asserted mock_run.call_args right after the 200 status, but the stream
handler runs _run_agent inside asyncio.create_task(_run_and_signal())
and response.prepare() returns the 200 before that task necessarily
starts — on loaded CI runners call_args was still None (TypeError:
cannot unpack non-iterable NoneType). Draining the SSE body (resp.text())
joins the stream end, which guarantees the runner task completed.
test_goal_verdict_send used fixed asyncio.sleep(0.05) waits before
asserting on sends/enqueues produced by spawned tasks; replaced with a
bounded _drain_until() poll (5s cap, returns as soon as the condition
holds) so the asserts stay exact without the fixed-delay race.
These three tests red-flagged unrelated main pushes and PR runs on
Aug 18 (runs 32099139396, 32101135100, 32106224479, 32101070064).
Control path: delegate_task(action=list/steer/stop) resolved ownership
purely through the _delegate_parent_ref weakref identity chain. The CLI
rebuilds its AIAgent mid-session (self.agent = None on route-signature
change, credential refresh, /model, MoA one-shots), so a running child's
chain pointed at a dead object and the child went invisible/unsteerable
while completion delivery (durable session-id routed) still worked.
Observed live 2026-08-17: deleg_88454b70 / sa-0-dc0100f4.
Fix: register each child with the owning conversation's durable session
id (owner_agent_session_id, the same spine delivery routes by) and add a
second ownership tier that matches it against the calling parent's
session_id with compression-lineage resolution on both sides. Foreign
sessions still fail closed.
Presentation path: background processes started BY a subagent (task_id ==
subagent_id) route their notify_on_complete notifications to the parent
conversation by design, but arrived as anonymous raw output walls. The
formatter now resolves the task_id against the live + recently-finished
subagent registry (bounded retention survives child completion) and adds
a provenance line (subagent id, delegation id, goal snippet), trimming
the output tail for subagent-owned processes. Parent-owned process
notifications are byte-identical to before.
test_update_hangup_protection pinned the exact stdout of
_print_update_completion; the new branch+HEAD suffix (parked-branch guard)
broke that pin. The two receipt tests assert the action-identity contract,
not the branch display, so they now neutralize _branch_head_suffix — the
suffix behavior itself is covered by test_update_parked_branch_guard.py.
Live incident 2026-08-17: the source checkout was parked on a stale feature
branch (claude-code-inspired/local-terminal-memory-limit, days behind main),
left there by earlier tooling. 'hermes update' autostashed, refreshed lazy
backends, synced skills, and printed '✓ Code updated!' / '✓ Update complete!'
while the checkout stayed on the stale branch with none of main's new code.
Two sessions burned time on 'the fix is missing' confusion.
- Parked-branch guard: auto-switch back to the update target ONLY when the
parked branch is clean and fully merged (git cherry origin/<target> shows
nothing unmerged); the checkout then STAYS on the target instead of being
re-parked. Otherwise: loud CODE UPDATE SKIPPED block naming the branch,
behind-count, and resolution commands; exit 1; branch untouched.
- The up-to-date (commit_count == 0) path no longer switches back to a
fully-merged parked branch either.
- Post-pull gate additionally refuses to print '✓ Code updated!' when HEAD
ends up attached to a non-target branch.
- Summary lines now carry the actual branch + HEAD short-sha:
'✓ Update complete! [main @ 30fcf9580]' — drift visible at a glance.
- New config toggle updates.auto_switch_parked_branch (default true).
- Real-git-fixture regression tests (init/clone/branch, no subprocess
mocks): clean+merged auto-switch, dirty skip, unmerged skip, cherry-picked
equivalence, config opt-out, unverifiable ref, on-main fast path,
up-to-date no-repark, summary branch/sha assertions.
Ben Barclay