Phase 2 review HIGH (empirically reproduced): a hard kill between
os.rename(dst, backup) and os.rename(staging, dst) leaves dst missing and
the backup as the ONLY copy of that entry. On retry, _stage_replacement
deleted that backup as a 'leftover' BEFORE staging the fresh copy — so a
staging failure (disk exhaustion is likeliest exactly after writing a full
staging copy) left a hole in the install with nothing to roll back to.
Restore the backup to dst first when dst is missing; it's a same-filesystem
rename. Mutation-verified: removing the restore makes the new test fail.
CI slice 8/8 red:
test_verify_core_dependencies.py::test_uses_virtual_env_from_environment
AssertionError: assert None == PosixPath('.../newvenv/Scripts/python.exe')
The Phase 2 reviewer flagged this exact risk (W4) and I under-weighted it as
"latent, not broken". It was neither — it was already failing.
The suite exercises Windows-only paths on Linux CI by patching predicates
(`hermes_cli.main._is_windows`, `is_windows`, `platform.system`). Routing
those call sites through a helper that reads `sys.platform` unconditionally
meant the patches no longer reached the path derivation: the test built
`Scripts/python.exe` while the code looked for `bin/python`.
venv_bin_dir/venv_python_path now take an optional `windows=` verdict,
defaulting to the host. Every converted site passes its own predicate, so
the patched-predicate coverage is restored — the dedup keeps the layout in
one place without hijacking the platform decision.
Verified by causation: dropping `windows=` reproduces the CI failure exactly;
restoring it goes green. Added two regression tests, including one asserting
a patched `_is_windows` still reaches the derivation.
Phase 2 review findings on the first commit.
C1 (critical) — the two-phase replace covered directories only, so the 20
first-party modules at the repo root (run_agent.py, cli.py,
hermes_constants.py, model_tools.py, toolsets.py, ...) were still copied
one-at-a-time with shutil.copy2 straight onto live paths. A failure in that
loop left all directories new and the root modules stale: precisely the
ImportError shape this PR exists to prevent. Worse, copy2 truncates in place,
so a crash mid-copy could leave a half-written cli.py — strictly worse than
stale on the flaky-AV path this code runs on.
Stage files the same way as directories and swap them in the same commit
phase. The docstring's "wholly new or wholly old" is now actually true.
C2 (critical) — a phase-1 failure (disk exhaustion being the likely one)
orphaned one staging copy per entry already processed, up to a second copy
of the tree. The user then follows our "re-run hermes update" advice with
LESS free space and the retry fails harder. Added _discard_staged() on the
staging path. Verified: staging failure now leaves zero litter.
W1 — _stage_replacement duplicated _atomic_replace_dir's first half verbatim.
_atomic_replace_dir is now a 1-line shim over the two-phase helpers; its
#49145 regression test still passes.
W2 — the failure message still said "some directories were replaced and
others were not", which the fix makes false. Now says the install was left
in place.
W3 — the free-space gate demanded 2x the tree when only the staging copy is
new (the live tree already occupies its space; swaps are renames). Relaxed
to need * 1.2, so we stop blocking updates that would have succeeded on the
space-constrained machines most likely to hit this.
W5/W6 — the lint-style guard used `"if" in line`, which matches "modify" and
"verify" and still missed os.path.join(venv, "Scripts"). Rewritten as an AST
check; it immediately found the real offender the substring version missed
(stdio.py, now explicitly exempted — it lists literal Windows-only PATH
candidates, not a cross-platform derivation). Softened venv_bin_dir's
"single source of truth" claim, since sites outside hermes_cli/ remain.
S1 — the rollback loop now logs instead of silently swallowing OSError.
Both C1 and C2 fixes are mutation-verified: reverting either makes the new
tests fail.
Closes#76104, closes#76105.
#76104 — `_atomic_replace_dir` (#49145) made each individual directory swap
safe, but `_update_via_zip` replaced ~70 top-level entries in a loop with no
atomicity across iterations. `agent/` lands at os.listdir index 13 and
`tools/` at 66, so an interruption between them left the new
`agent/context_compressor.py` (module-level `from tools.todo_tool import
TODO_INJECTION_HEADER`) beside a stale `tools/todo_tool.py` — every file
valid Python, the tree unbootable. That is the mechanism behind the
ImportError fixed in #76091, and the "partial update" field report in #63717.
Split into stage-all-then-swap-all:
- `_stage_replacement` copies each dir to a sibling staging path, touching
nothing live, so a failure during the long copy phase is a no-op.
- `_commit_staged_replacements` performs the renames and, if any fails,
restores every entry already swapped — the tree lands wholly new or
wholly old, never mixed.
This shrinks the failure window from a full tree copy to N renames and makes
what remains recoverable. Added an up-front free-space check, since staging
needs a second copy of the tree; a clear error beats running out mid-swap.
#76105 — venv interpreter resolution was open-coded in 7 places across 4
files using 3 different Windows predicates. #76091 added the seventh because
the correct behaviour lived 2400 lines away. Hoisted `venv_bin_dir()` /
`venv_python_path()` into hermes_constants (import-safe, no new imports) and
routed every site through them; `managed_uv._venv_python` now delegates so
its 6 callers are untouched.
`_atomic_replace_dir` is retained — it is re-exported from main.py and has
its own #49145 regression test; removing it is out of scope here.
Tests: 10 new (rollback-on-mid-swap-failure is mutation-verified — it fails
when the rollback loop is removed), plus a guard that fails if a new call
site hand-rolls Scripts/bin again. E2E-verified against the real staging +
commit helpers with a live tree.
/simplify-code reuse reviewer (HIGH): the probe and the user-facing hint
each carried their own hand-written list of first-party package roots,
and they had already diverged on day one —
module probe hint
cli False True <- rollback with no explanation
hermesx True False <- third-party blamed on our updater
Hoist a single FIRST_PARTY_MODULE_ROOTS + is_first_party_module() into
hermes_constants (import-safe, no new imports) and have both consume it;
the probe gets the set injected into its source rather than re-typing it.
Also completes the roster — cron, utils, run_agent, model_tools,
toolsets, tui_gateway, acp_adapter were missing from both copies.
Verified by executing the real probe source against 19 module roots:
0 disagreements. Added a test that fails if either side grows a private
copy again.
Phase 2 review (C2) and /simplify-code findings.
C2 — the git path ran the import guard before `_clear_bytecode_cache`,
wired into the syntax guard's `git reset --hard` rollback. But
`cannot import name 'X'` is ALSO the documented signature of the
stale-bytecode class (#6207, #60242, see
_sweep_stale_bytecode_if_checkout_changed), which the very next steps —
and the launch-time sweep — already self-heal. A false positive there
would destroy a good update over a state that fixes itself.
Remove the guard from the rollback path entirely and re-add it at the
end of the git path, after bytecode sweep + dependency reinstall + lazy
refresh, as a WARNING only. By then every benign source of a transient
ImportError has run, and we never reset the user's checkout.
W6 — the headline regression test was vacuous: it patched
`hermes_main._UPDATE_CRITICAL_FILES`, but the syntax guard reads
`update_cmd`'s global, so the stub files were never examined and the
(True, None, None) came from "no files found" rather than "parses
clean". Patch the right module; mutation-checked (the test now fails
when the guard is disabled).
S5 — `startswith(("tools","agent","hermes","gateway"))` also matched
third-party `agents`/`agentops`/`toolsets`. Compare the first dotted
segment against an exact set instead.
S6 — hoist the per-line ChatConsole() instantiation.
Phase 2 review caught a false-rollback I introduced: on the git path the
import guard runs at the post-pull syntax check, which is BEFORE the
dependency sync. A release that adds a new third-party requirement would
fail the probe and trigger `git reset --hard` on a perfectly good update.
Rather than reorder the git path (the guard belongs with the rollback it
feeds), make the probe ignore a missing module that isn't ours. A missing
third-party package means deps aren't installed yet; a missing first-party
module means the update dropped a file, which IS the skew we're hunting.
This also makes the ZIP path's ordering non-load-bearing.
Verified: third-party absent -> (True, None, None); first-party absent ->
flagged; and the original TODO_INJECTION_HEADER skew is still caught.
Self-review against the sibling probe `_venv_core_imports_healthy`
surfaced this: that helper deliberately resolves the project venv's
python rather than using `sys.executable`, because `hermes update` may
be driven by a different interpreter than the install's own.
The new import guard had the same requirement and missed it. Probing
`sys.executable` would validate a tree the user never actually runs —
and that divergence is most likely on Windows, the exact platform this
guard was added for.
Falls back to the running interpreter when there is no venv (normal in
a dev checkout). Regression test asserts the venv python is chosen; it
fails when the fix is reverted.
A Windows user reported every startup dying with `ImportError: cannot
import name 'TODO_INJECTION_HEADER' from 'tools.todo_tool'`. The symbol
exists on main; their tree had the new `agent/context_compressor.py`
(which imports it at module level) alongside a pre-update
`tools/todo_tool.py`.
The post-update guard missed it. `_validate_critical_files_syntax` only
py_compiles files, and every file in a skewed tree parses fine — it is
the combination that is broken. The guard reported success and the
update completed over an install that could not start.
The ZIP-update path (Windows-only, used when git file I/O is broken)
is where the skew comes from: its copy loop replaces top-level entries
one at a time in `os.listdir` order, so `agent/` lands at index 13 and
`tools/` at index 66. Any failure between them leaves exactly this
mismatch — and that path had no post-copy validation or rollback at all.
- Add `_validate_critical_modules_import`: imports the four startup
modules in a subprocess (~0.4s) so cross-module breakage is caught.
Non-import errors (config/env) are ignored; a probe that cannot spawn
is non-fatal so we never block an update on our own tooling.
- Run it after the syntax guard on the git path, reusing the existing
auto-rollback.
- Run it on the ZIP path after dependency install (so a genuinely-new
requirement is not misreported as a partial copy), and make the ZIP
failure message state the install may be half-updated.
- Add `partial_update_hint()` and print it under "Failed to initialize
agent", so users see "re-run hermes update" instead of a bare
ImportError. Stays silent for ModuleNotFoundError and third-party
imports, which need different remediation.
Verified by simulating the exact skew: the syntax guard returns ok=True
while the import guard returns the user's error verbatim.
Let terminal keys explicitly present in config.yaml override matching stale TERMINAL_* values while preserving environment values for omitted keys. Merged defaults remain backfill-only.
Exercise the real config.yaml to _get_env_config path for backend selection, partial terminal sections, matching-key overrides, environment fallback, one-shot bridging, and config read failures.
Closes#71137
Setting prompt_caching.cache_ttl to a falsy value (false, null, off,
disabled, no, none) now fully disables prompt caching instead of
being silently ignored.
The disable propagates through anthropic_prompt_cache_policy() (early
return when _cache_disabled flag is set) and restore_primary_runtime()
(override after snapshot restore), so it survives /model switches and
fallback re-derivation — the gap that caused #56105 to be reverted in
#56126.
Salvage of #33555 by @BB-light, with model-switch/fallback survival
gap fixed on top.
Co-authored-by: BB-light <BB-light@users.noreply.github.com>
Non-interactive sessions (hermes chat -q, hermes -z) snapshot the tool
registry at AIAgent construction time. If background MCP discovery hasn't
finished, MCP tools are invisible for the entire session — and unlike
interactive mode, there is no between-turns late-binding refresh to recover.
Root cause: wait_for_mcp_discovery() only joins an already-created discovery
thread, so it no-ops if a direct/single-query path reaches agent construction
before MCP startup created that thread. Oneshot._run_agent() didn't call it
at all.
Fix:
- Add ensure_mcp_discovery_before_agent_build() helper to mcp_startup.py:
idempotently starts discovery if needed + bounded wait. Fail-open on errors.
- Add single_query parameter to _resolve_discovery_timeout/wait_for_mcp_discovery:
uses mcp_single_query_discovery_timeout (default 15s) instead of the
interactive mcp_discovery_timeout (1.5s) because one-shot sessions have no
second turn to recover.
- Wire into CLI _init_agent (single_query from _single_query_mode flag set
in cli.py's single-query path) and oneshot._run_agent (single_query=True).
- Interactive sessions unchanged: keep 1.5s bound (between-turns refresh covers).
Closes#38448, #51316, #37013, #68137
Composite salvage of #60017 (chrishart0), #51322 (Bartok9), #38620 (buptwz),
#43544 (halonke), #36882 (vanhoof).
get_custom_provider_extra_headers() was returning the result of
normalize_extra_headers() on the first matching base_url, even when
that entry had no extra_headers configured. A later providers.<name>
entry sharing the same URL but with headers set was therefore ignored.
Fix: store the normalized headers and only return when non-empty,
otherwise continue searching the remaining entries.
Fixes#74465
set_config_value() and unset_config_value() silently replaced the
entire config with an empty dict when config.yaml could not be
parsed. A single YAML syntax error would cause 'hermes config set'
to wipe all settings and write only the new key. Now exits with
error and preserves the existing file.
When model is a bare scalar (e.g. 'model: gpt-4o'), running
'hermes config set model.provider openai' silently destroyed the
model id because _set_nested replaced the scalar with an empty dict
before writing the sub-key. Now the scalar is normalized to
{default: <id>} first, preserving the model id.
Follow-ups to the previous commit (#74414 by @webtecnica, re #74373):
- When distribution_owned is OMITTED, restore the legacy contract: every
staged entry outside USER_OWNED_EXCLUDE is copied. The cherry-picked
filter consulted owned_paths(), which silently narrowed omitted-list
distributions to DEFAULT_DIST_OWNED and dropped undeclared payload
(extra top-level files/dirs existing distributions legitimately ship).
- Make explicit allowlists path-aware so documented nested entries like
skills/research/ and cron/digest.json select exactly that subtree/file
instead of being dropped by the top-level name comparison. Traversal
segments (.., absolute) and USER_OWNED_EXCLUDE roots are still rejected.
- Regression tests: omitted-list legacy behavior + nested-path allowlist.
_copy_dist_payload() in profile_distribution.py iterated all staged
entries without consulting the manifest's distribution_owned allowlist,
so manifests that restricted distribution_owned only had cosmetic effect.
Fix: compute manifest.owned_paths() at the top of _copy_dist_payload()
and skip entries not in that set, after the USER_OWNED_EXCLUDE check.
The owned_paths() method already existed on DistributionManifest and
correctly falls back to DEFAULT_DIST_OWNED when no explicit
distribution_owned is set, so the new filter preserves backward
compatibility for existing manifests.
Closes#74373
resolve_display_context_length() runs two blocking chains: the route
comparison in should_clear_context_pin() and the provider probe ladder in
get_model_context_length() (blocking requests calls to Anthropic /v1/models,
Copilot, Nous, Codex, GMI, Ollama, models.dev and OpenRouter).
The gateway message path already offloads both via
get_model_context_length_async() and should_clear_context_pin_async(), but
the /model slash-command handlers (_handle_model_command, _finish_switch)
called the sync helper directly, freezing the whole event loop for the
duration of the probe ladder - no messages processed on any platform, and
the Discord heartbeat timeouts that get_model_context_length_async() was
introduced to prevent.
Add resolve_display_context_length_async(), a thin asyncio.to_thread wrapper
mirroring the two existing *_async helpers (no logic duplication), and await
it at both handlers.
setup_path() only wrote a 'hermes' launcher into the command-link dir,
even though pyproject.toml declares three [project.scripts]:
hermes, hermes-agent (run_agent:main), hermes-acp (acp_adapter.entry:main).
Fresh venv installs (the common case on macOS/Linux) leave
~/.local/bin/{hermes-agent,hermes-acp} empty, so external tools
expecting 'hermes-acp' as a standalone command (documented as
first-tier supported in website/docs/user-guide/features/acp.md)
fail to find it after a fully successful install.
Loop over the three console-script names, writing a shim per entry
that exec's the venv interpreter with the right checked-in
entrypoint. --no-venv keeps the old single-shim behaviour since
it does not manage the venv and only 'hermes' is guaranteed on PATH.
Fixes#74819
_is_env_config_key() already routes _API_KEY and _TOKEN suffixed
keys to .env for safe credential storage. Add _SECRET to the suffix
list so keys like CLIENT_SECRET and ENCRYPTION_SECRET are stored
in .env (excluded from git by default) rather than config.yaml.
7b5a18817 migrated the sibling slug sites to custom_provider_slug, which
keeps a keyed providers: entry's config key as its durable identity. It
covered find_custom_provider_identity_by_model; canonical_custom_identity's
third recovery source - the configured-provider fallback - still built
f"custom:{normalized}" out of whatever string the caller happened to hold.
_get_named_custom_provider matches on either spelling, so a display name
that differs from its config key matches the entry and then heals to
custom:<display-name>. That is a second identity for one endpoint: the
endpoint- and model-based sources of the same function return
custom:<config-key>, and so does everything that persists or restores a
session's provider override. canonical_custom_identity exists precisely to
make a bare "custom" routable again, and tui_gateway calls it on the
session-persist, resume and recovery paths - so the divergence lands in
stored session identity.
Re-resolve through the endpoint the matched entry owns, reusing the
function's own URL-based canonicaliser rather than duplicating the match
logic. Legacy unkeyed custom_providers: entries keep their name identity,
and an unconfigured candidate still returns None.
Ports the negative limit/offset fix onto the current router modules
(hermes_cli/web_routers/sessions.py, profiles.py) since the handlers
moved out of web_server.py in 011ec4513e after this PR was opened.
Per review feedback: only add Query(..., ge=0) — no le=500. The
messages route already clamps oversized requests with min(limit, 500)
and must keep that behavior (succeed + cap) rather than reject them;
the two session-list routes never had a public 500 cap and shouldn't
gain a new rejecting one as a side effect of this fix.
_resolve_explicit_runtime's generic-provider branch accepted model_cfg's
persisted api_mode unconditionally, letting a stale mode from a previous
provider (e.g. anthropic_messages) leak into a newly-switched provider
(e.g. gemini) and break the transport. Reuse the existing
_provider_supports_explicit_api_mode guard, already used by the copilot
and named-custom-provider resolution paths for exactly this case, so the
persisted mode is only honored when model_cfg's provider matches the one
being resolved.
Closes#74318
resolve_entry_api_key() and the duplicated _fallback_entry_api_key()
read key_env via a raw os.getenv(), bypassing per-profile secret
scoping in the multiplexed gateway. Under multiplexing this can hand
a fallback request another profile's credential. Both now resolve
through agent.secret_scope.get_secret(), which reads the active
profile scope when multiplexing is on and falls back to os.environ
unchanged when it's off, so single-profile behavior is preserved.
Closes#74311
_load_auth_store() treated every exception from reading auth.json as
corruption and returned an empty store. EMFILE under fd exhaustion,
EACCES, EIO and a stalled network mount all reached that branch. This
module does read-modify-write in roughly fifteen places, so the empty
store was one _save_auth_store() away from erasing every stored
credential.
Separate OSError from parse failure: a file that exists but cannot be
read now raises, naming the real cause and leaving the file on disk
untouched. Only a genuine parse failure takes the preserve-and-start-
empty branch, which is unchanged.
The backup was also unreliable in exactly the conditions that triggered
it: shutil.copy2 opens a file, so under EMFILE it failed too, its bare
except swallowed that, and the log still said "Corrupt file preserved
at ..." when nothing had been written. Track whether the copy landed
and say so accurately.
_venv_launcher_ancestors() ran after
_wait_for_windows_update_gateway_exit(), but the drain stops tracking a
PID exactly when it dies - for the common graceful-drain case the worker
is gone by the time the wait returns, and a dead pid's parent cannot be
recovered, so the launcher stop never fired on that path. Resolve
launcher ancestors before draining and stop the snapshot afterwards
alongside the survivors; a launcher that already exited with its worker
raises ProcessLookupError at the kill and is skipped.
The set-cover invariant test now marks drained workers uninspectable
(construction raises, like psutil.NoSuchProcess), so a post-drain
launcher lookup can never reappear unnoticed.
_is_pausable_gateway() hand-rolled a second gateway parser and regressed
a valid form: in `--profile gateway gateway run` the profile VALUE
shadowed the subcommand token, so the scan reported that gateway as a
fatal preflight holder. Delegate to
gateway.status.looks_like_gateway_command_line() - profile-selector
aware, shlex-tokenizing, run-only - so the preflight exemption, the
pause discovery, and the updater's guard fallback share one parser.
Non-run gateway subcommands, serve backends, and REPLs still block; the
bare-`gateway` form now classifies as a running gateway, mirroring the
canonical matcher's contract.
The pause stops every gateway its discovery maps, but the venv-holder
guard sees the process table as it is now: a gateway respawned by its
supervisor (Scheduled Task, login watchdog) inside the pause-to-guard
window, or one started through a spawn path discovery does not map,
still holds venv .pyds - and the guard dead-ended the update on exactly
the kind of process the pause machinery exists to stop.
When every remaining holder classifies as a pausable gateway - using the
same _is_pausable_gateway matcher the Desktop preflight uses, so the two
views cannot drift - stop them and re-scan once. Any non-gateway holder
(REPL, stray script, Desktop backend) keeps the hard refusal exactly as
before, and a survivor after the stop still aborts.
The Desktop update preflight (`scanVenvBlockers` -> `python -m
hermes_cli._scan_venv_blockers`) reports every venv-side python as a
blocker and aborts the handoff:
main.ts: scanVenvBlockers(...) <- aborts HERE
return { ok:false, error:'venv-blocked' }
spawnUpdaterProcess(hermes-setup ...) <- never reached
But a *gateway* is not a dead-end holder. `hermes-setup` invokes
`hermes update --yes --gateway`, and the CLI updater's
`_pause_windows_gateways_for_update()` gracefully drains and stops
running gateways before touching the venv — machinery added for exactly
these processes (#50090 and follow-ups). The preflight replicated the
CLI's *guard* without its *pause*, so a Windows service-mode gateway
(e.g. a Scheduled Task running `gateway run`) made every Desktop update
abort forever with
[updates] venv-blocked: N process(es) hold the install
PID ... python.exe ... -m hermes_cli.main gateway run --replace
while the component one layer down was never allowed to run and handle
it. The abort points at a process the updater knows how to stop.
Fix: `_is_pausable_gateway()` exempts `hermes_cli.main ... gateway run`
invocations (both halves of the venv-shim launcher/worker chain match,
since the uv-side worker re-runs the same argv). Everything else keeps
blocking — the Desktop `serve` backend, other `gateway` subcommands,
operator REPLs and stray scripts have no pause machinery downstream.
The CLI updater's own post-pause venv guard is untouched, so a pause
that genuinely fails still aborts before any .pyd mutation.
The JSON gains a diagnostic `pausable_gateways` count. The TS consumer
validates only `ok`/`blocked`/`processes` and ignores unknown keys, so
old and new Desktop builds both accept the new document; no Electron
rebuild is required for the fix to take effect (the scan runs from the
repo's Python).
On Windows a gateway started through the venv shim is a two-process chain:
venv\Scripts\python.exe (launcher — keeps venv .pyd files mapped)
└─ uv\python\...\python.exe (worker — writes the gateway PID file)
`_pause_windows_gateways_for_update()` builds its pause set from
`find_gateway_pids()`, which reads the PID file and therefore only ever
sees the *worker*. The venv-holder guard immediately downstream
(`_detect_venv_python_processes()`) matches on the venv path prefix, so it
only ever sees the *launcher*.
The two sets are disjoint. A gateway the updater had just gracefully
drained still left its launcher alive, the guard reported that launcher as
a venv holder, and the update aborted — every time. On the Desktop path
this surfaces as the dead-end dialog:
[updates] venv-blocked: 2 process(es) hold the install
PID ... python.exe ...\venv\Scripts\python.exe -m hermes_cli.main gateway run --replace
Note the reported holder is a gateway the updater believes it stopped.
The Desktop path is affected because `hermes-setup.exe` runs
`hermes update --yes --gateway --force`, and `--force` deliberately does
NOT bypass the venv guard (that needs `--force-venv`), so the abort is
correct behaviour reacting to an incomplete pause.
Fix: after the graceful drain, walk one hop up from each mapped gateway
PID and force-kill parents that live under the project venv.
Deliberately additive, not a substitution:
- The planned-stop marker and the graceful drain still target the worker
(the PID that wrote the PID file), so clean shutdown is unchanged and
updates don't get pushed onto the hard-kill path.
- `terminate_pid(force=True)` is `taskkill /T` (tree kill), so killing a
launcher that outlived its worker also reaps stragglers.
- `_resume_windows_gateways_after_update()` needs no change: the mapped
respawn argv is rebuilt from the profile name
(`_gateway_run_args_for_profile`), never from the killed PID, and the
restart watcher's `_pid_exists()` wait still terminates because the
tree kill takes the whole chain down.
- Only the venv-side parent is returned. Unrelated ancestors (a Scheduled
Task's `cmd.exe`, an operator's shell) are ignored, and the caller's own
process chain is excluded so a CLI `hermes update` never nominates
itself.
Tests assert the invariant the two PID-resolution paths must satisfy —
the pause's kill set must cover the guard's abort set — rather than
snapshotting PIDs. Verified to fail without the fix:
AssertionError: pause stopped [] but the venv guard aborts on [400]
— disjoint sets abort the update
The /indicator command was registered in COMMAND_REGISTRY, listed in
/help, offered by tab-completion, recommended by the tips system, and
even documented in config.py — but it had no actual handler. Running
/indicator in the CLI produced "Unknown command: indicator".
Add _handle_indicator_command to CLICommandsMixin that:
- Shows the current indicator style when called with no args or "status"
- Validates the requested style against the shared INDICATOR_STYLES
allowlist (ascii | emoji | kaomoji | unicode)
- Persists the choice to display.tui_status_indicator in config.yaml
via the existing save_config_value helper
- Falls back to session-only when config save fails
The indicator-style allowlist is defined once in hermes_constants as
INDICATOR_STYLES + DEFAULT_INDICATOR_STYLE and imported by all three
consumers (CLI handler, command registry, TUI gateway config handler),
preventing drift between the TUI and CLI validation.
Also adds tests/cli/test_indicator_command.py covering dispatch,
validation, persistence, and registry integration.
Signed-off-by: dongjiang <dongjiang1989@126.com>
--force was silently ignored for 'model' keys — the guard always
redirected to model.default even when the user explicitly asked to
replace the entire section. Now --force triggers a warning and
proceeds with the destructive overwrite for model too, matching
the non-model mapping --force behaviour.
Prevent 'hermes config set <section> <scalar>' from silently destroying
an existing mapping. The bare 'model' shorthand is preserved by
redirecting to 'model.default' — all other mapping sections are refused
with a helpful error unless --force is used.
Closes#74995
_save_xai_oauth_tokens had the identical self-sealing bug as
_sync_device_code_entry_to_auth_store: key-presence check before
_store_provider_state, which unconditionally creates the key.
Use _load_provider_state_with_source to decide write-through from
the actual grant source, not key presence.
Also update regression test per review: use the real
_write_through_provider_state_to_global_root helper and assert
rotated token pair values in the root store after each refresh
instead of just counting mock calls.
- Bound ALL reads of the on-disk JWT store through one _read_jwt_store()
helper (load, eviction, save-merge) — the 1 MiB cap previously only
covered the load path; eviction and save could still parse an
oversized/corrupt store and rewrite it back out (sweeper finding).
- Fix the class, not the site: the recovery gates checked the literal
provider == "copilot" while /model and profile configs can leave the
alias spelling in place (the reporter's own log shows provider=copilot
AND provider=github-copilot in one session — the aliased turns would
have silently skipped recovery). Single owner:
AIAgent._is_copilot_provider() (slug aliases + Copilot base-URL
fallback), used by both run_agent recovery methods and both
conversation_loop gates.
- Update the salvaged 401 test to current main's client-retirement
contract (release deferred to GC — no synchronous .close()).
- Add copilot_stale_cred_retry_attempted to the TurnRetryState field
contract test; add bounded-store and alias-gate regression tests.
Copilot degrades in two related ways that both abort a turn as non-retryable
and only clear on a gateway restart (a cold process re-runs the token exchange):
1. HTTP 400 model_not_available_for_integrator / model_not_supported — a
raw/degraded token routes to the restricted copilot-language-server
integrator whose allowlist omits enterprise-only models (e.g.
claude-opus-4.8). Because it is a 400 (not 401), the existing 401 refresh
path never fired. Prevented (retry-with-backoff exchange + on-disk JWT
persistence + header guard at the client chokepoint) and self-healed at
runtime (single-shot forced re-exchange + client rebuild + retry before
fallback).
2. HTTP 401 'IDE token expired: unauthorized: token expired' — the short-TTL
*exchanged* IDE token expires mid-turn. The clean-401 path DID fire and call
_try_refresh_copilot_client_credentials(), but that method only re-resolved
the stable raw ghu_ token and rebuilt the client — it never evicted the
cached exchanged JWT or forced a fresh exchange, so the retry put the SAME
expired token back on the wire, 401'd again, and the single-shot guard
aborted the turn. Fix: force a fresh IDE-token exchange (evict cached JWT via
evict_cached_exchanged_token + re-mint via get_copilot_api_token) before the
client rebuild, mirroring the merged auxiliary-path recovery (#59837) and the
400 recovery in this same PR. Graceful fallback to the resolved token if the
exchange endpoint is unreachable; picks up the enterprise base_url on
re-exchange.
Brings main-loop clean-401 recovery to parity with the merged auxiliary path
(#59837), using the newer on-disk-aware evict helper. Companion context: #58743
(this PR, expanded), #51313, #63204 (which assumed the 401 path already
recovered — it reached the method but the method was too weak).
Tests: exchange retry/persist round-trip, restart-blip disk reuse, stale-cred
400 classifier, 400 recovery, and 3 new 401 cases (fresh exchanged token on the
wire; network-blip fallback to resolved token). 58 copilot tests green on
current main.
The stale-staged-updater deadlock is not Windows-specific: hermes-setup
under ~/.hermes is only refreshed by a full installer run
(copy_self_to_hermes_home no-ops during --update), so every desktop whose
staged updater predates the HERMES_UPDATE_HANDOFF_PID export (8c76fe19)
runs an old parent that never sends the env var against a new child that
demands it — exit 2 ('Hermes is still running') forever, on macOS and
Linux just as on Windows.
Replace the wmic ancestry walk (deprecated, absent on current Win11,
GBK decode juggling) with psutil.Process().parents() — psutil is already
a hard dependency and is the project's canonical no-kill process probe.
Drop the os.name == 'nt' gate so all platforms heal. Add tests: a marker
owned by our parent process is recognized as our orchestrator; a live
non-ancestor holder is still refused.
On Windows the Tauri updater writes its PID to the update marker, then
spawns 'hermes update' via the venv shim. The child receives
HERMES_UPDATE_HANDOFF_PID to recognize the parent lock, but on some
Windows configs the env var is not inherited through the shim subprocess
chain, deadlocking every GUI update.
Fix: when the env-var handoff fails, fall back to walking the Windows
process tree via wmic /format:csv to collect ancestor PIDs. If the
marker PID is found in the ancestor chain, treat it as our own
orchestrating parent and allow the update.
Also fixes wmic output encoding on localized Windows (GBK/CP936) by
using /format:csv which produces clean UTF-8 output.
Independent review pass: utils.base_url_host_matches already owns the
exact-or-dot-suffix hostname contract (userinfo/port stripped, lowercased,
trailing dot removed), so the predicate delegates instead of hand-rolling
a second suffix match to keep in sync. Also locks in the normalization
behavior the review verified empirically: uppercase+port, trailing-dot,
userinfo-stripped, and IPv6-literal cases added to the contract tests.
Three catalog-side defects from the same report, all downstream of the
exact-host assumption and the config/env asymmetry:
- Discovery read only $OPENAI_BASE_URL, so a config-set
model.base_url (the supported way to select a data-residency host)
was ignored and /model listed the catalog of api.openai.com, not the
configured endpoint. New _openai_discovery_base_url() resolves
env override -> matching model.base_url -> canonical default, the same
precedence inference uses.
- _credential_fingerprint() hashed env vars only, so hermes config set
model.base_url kept serving the previous endpoint's cached catalog
until TTL expiry. The effective endpoint is now folded into the
fingerprint for openai/openai-api.
- is_default_openai matched two literal URLs, so regional hosts (which
serve the identical 120+ entry dump) bypassed the curated intersection
and flooded the picker with whisper/tts/embedding/dall-e rows. Now uses
the shared official-host predicate; custom OpenAI-compatible proxies
keep the verbatim live list.
- validate_requested_model's curated-catalog soft-accept (#46850) no
longer applies on official OpenAI hosts: their /v1/models listing is
access-scoped and authoritative, so accepting an absent model
manufactures a selection that 400s at first use. Custom proxies and
other providers keep the #46850 fallback. The #37404
empty-intersection -> curated picker fallback is deliberately left
unchanged.
The P1 from the enterprise data-residency report: with
model.base_url=https://us.api.openai.com/v1, every tool-calling turn 400'd
('Function tools with reasoning_effort are not supported ... use
/v1/responses') because the runtime resolvers hardcoded
api_mode=chat_completions and consulted URL detection only. openai-api
declares codex_responses in its overlay; the declaration was never
consulted, so any OpenAI host that wasn't literally api.openai.com landed
on the wrong wire protocol.
New _fallback_api_mode(provider, base_url, model): URL detection first
(host-mandated wire shapes keep priority), then
providers.determine_api_mode() (the provider's declared transport), then
chat_completions only for genuinely unknown providers. All three runtime
fallback sites route through it: the pool-entry path, the explicit-runtime
path, and the API-key-provider path, so the lanes cannot drift apart.
Blast radius beyond openai-api: minimax, minimax-cn, and copilot-acp were
the other overlays whose declared non-chat transport fell through to
chat_completions on the same paths (same latent bug class). openrouter is
unaffected (declares openai_chat). _detect_api_mode_for_url also now uses
the shared official-host predicate, so regional hosts detect as
codex_responses on the direct-URL lane too.
Pointing openai-api at OpenAI's documented regional hosts
(us.api.openai.com / eu.api.openai.com, mandatory for customers with
data-residency obligations) silently degraded Hermes because three
subsystems tested 'is this OpenAI' with exact-hostname equality against
api.openai.com.
Adds providers.is_official_openai_host(): canonical host plus dot-suffix
subdomains of api.openai.com, hostname-parsed only. Lookalike hosts
(api.openai.com.attacker.test) and path spoofs (proxy.test/api.openai.com/v1)
stay rejected, preserving the #32243 hardening: a genuine *.api.openai.com
subdomain requires control of openai.com DNS.
host_mandated_api_mode() now routes through the predicate, so regional
hosts mandate codex_responses exactly like the canonical host.
Port from nanocoai/nanoclaw#2748: Docker's built-in 64 MB /dev/shm silently
breaks shared-memory-hungry workloads inside the sandbox — Chromium/Playwright
renderers crash tabs, and PyTorch DataLoader workers die with 'bus error' /
'insufficient shared memory'. tmpfs is lazily allocated, so the higher ceiling
costs nothing until actually used, and usage still counts against the
container's --memory cgroup limit.
- tools/environments/docker.py: --shm-size 1g in resource args (not
cgroup-gated; tmpfs mount option). Skipped when docker_extra_args already
sets --shm-size, or when configured empty/'0' (Docker default).
- terminal.docker_shm_size config key (DEFAULT_CONFIG + all three
config->TERMINAL_DOCKER_SHM_SIZE env bridges: CLI, gateway, config.py map)
- tests: default emit, custom value, opt-out, extra_args precedence,
helper edge cases (sabotage-verified: default/custom tests fail without
the emit)
kshitij