A gateway active-session lease is acquired against the root HERMES_HOME,
but release_active_session()/transfer_active_session() re-resolved the
registry path from the *current* HERMES_HOME. Under native multiplex a
routed turn runs agent cleanup inside _profile_runtime_scope, so the
release looked under the named profile while the root entry stayed
alive — after max_concurrent_sessions routed turns every new session was
rejected with 'Hermes is at the active session limit' (#85431).
Pin state/lock paths on the lease at acquisition time and prefer them on
release and transfer. Fixes#85431.
Review follow-up (final round PASS with a repeated suggestion): the pets
CLI regression test only exercised real bools, leaving the exact bug this
fix shipped untested. Add a quoted-'false' test driving _has_active_pet
(now False) and toggle_pet_display (now takes the ENABLE branch,
distinguished by the 'no pets installed' error instead of err=None from
the old wrong-way disable). Verified RED on the pre-fix pets.py and GREEN
on the fix.
* feat(sessions): generic 'hidden' session flag (sidebar-hide, still resumable)
Adds a source-orthogonal, archive-orthogonal 'hidden' session flag meaning
'don't show in the global Sessions sidebar, but stay fully resumable by the
surface that owns it'. Mirrors the existing archived/pinned capability end to
end, so it's a generic widening (any plugin that owns its own session lifecycle
- kanban, Bot Mode, future plugins - can keep its sessions out of the shared
recents list) rather than a per-plugin special-case.
- Schema: hidden INTEGER NOT NULL DEFAULT 0 on sessions (additive; lands on
existing DBs via the declarative _reconcile_columns ADD COLUMN path, same as
archived/pinned - no version-gated migration).
- DB: SessionDB.set_session_hidden(session_id, hidden) (clones set_session_pinned
incl. the compression-lineage recursive CTE); list_sessions_rich gains
include_hidden=False, appending 's.hidden = 0' by default so hidden rows drop
from every listing path (and the REST sidebar endpoints inherit it with no
change).
- Gateway: session.set_hidden RPC (mirrors session.title); session.create accepts
hidden=true, deferred via pending_hidden and applied in _ensure_session_db_row
when the row is lazily created (mirrors pending_title).
- REST parity: PATCH /api/sessions/{id} accepts+bool-validates 'hidden' ->
set_session_hidden; _session_response exposes it.
Enables Hermes-Bot-Mode to hide canonical 'Bot Chat' sessions from the sidebar
(NousResearch/Hermes-Bot-Mode#46) WITHOUT retagging source (which would mis-set
the agent platform). Bot Chats keep source=desktop. Gateway RPC needs a
SERVE-backend restart to take effect live. 1 focused test (default-exclude /
include_hidden / unhide round-trip).
* fix: teach lost-and-found recovery about the 55-column sessions layout
Adding the 'hidden' column makes the current sessions table 55 columns. The
SQLite lost-and-found recovery classifier keys off the physical field count
(SESSIONS_LAYOUT_NFIELDS) to identify a salvaged sessions row, so a recovered
current-layout row (nfield=55) would otherwise be unrecognized and dropped.
Add 55 to the frozenset (54/52 stay as historical prefixes) and update the
column-count assertions + synthetic current-layout insert in the recovery test.
---------
Co-authored-by: Teknium <teknium1@users.noreply.github.com>
On Windows _get_service_pids() is empty (no systemd/launchd query), so a
Scheduled-Task-supervised gateway whose gateway.pid record is missing or
stale is invisible to both the service-PID and recorded-PID exclusions the
reaper already applies (#86658) — and gets SIGTERM'd on every desktop open
(#86098 class, pidfile-less path).
Add a Windows-only backstop: any reaper candidate whose parent chain
reaches services.exe (the Task Scheduler launches tasks under the services
tree) is spared even with no pidfile.
The backstop is deliberately inert on POSIX: every process there has PID 1
(launchd/init/systemd) in its ancestry — and a genuine orphan is reparented
directly to PID 1 — so supervisor-name ancestry carries zero supervision
signal and would disable the reaper entirely on macOS/WSL (#51325, #75936).
POSIX supervised gateways are already covered pidfile-independently by the
_get_service_pids() exclusion.
Known limitation (fail-open, documented): if the Task-launched bootstrap
parent has already exited, Windows does not reparent the gateway, the chain
breaks before services.exe, and the gateway is treated as an orphan.
Salvaged from #86702 by @EvanProgramming (authorship preserved); reduced to
the genuinely-new Windows backstop — the PR's other two hunks were already
merged on main via #86658 (one in a strictly stronger full-parent-chain
form) and its POSIX ancestry checks were dropped as unsound (verified
empirically: a true double-fork orphan's psutil parent IS launchd).
Follow-up to the salvaged #76252 addressing both review gaps:
- New TestMinorLineFallForward class with a direct test of the
explicit-patch fallback branch: bare '3.12' resolves to a VULNERABLE
build while an explicit 3.12.x patch is fixed, so recovery must go
through _list_available_patches on the next minor line. Asserts the
exact `uv python install` request sequence.
- New all-minors-exhausted test: everything vulnerable on 3.11-3.13
returns None with per-line attempts bounded by _MAX_PATCH_RETRIES and
no requests beyond 3.13 (requires-python is <3.14).
- test_retry_is_bounded_by_max_retries_constant now actually uses its
counting wrapper and asserts the collected install calls (the
previous version collected them into a dead variable).
Also dedupes the fallback loop the same way the same-minor loop does:
_attempt_install_generation can now record the probed candidate version
into a caller-supplied tried_versions set, so the explicit-patch pass
skips the version the bare-minor request already resolved to and
rejected -- previously that wasted a full download+install+probe+delete
cycle per minor line re-trying a known-vulnerable build.
When every patch on the current minor line (e.g. 3.11) still links a
vulnerable SQLite (e.g. 3.50.4 on Windows), the provisioner now tries
the next supported minor line (3.12, then 3.13) before giving up.
Previously, _install_safe_python_generation only tried patches within
the same minor line. On Windows, where python-build-standalone may not
publish a fixed build for the installed patch, users were stuck with a
repeated warning on every `hermes update` with no path forward.
The requires-python constraint (>=3.11,<3.14) and the downstream
import smoke test already gate compatibility, so the minor-line
upgrade is safe.
Adds allow_minor_upgrade parameter to _attempt_install_generation to
relax the same-minor-line version guard when called from the fallback
path.
Fixes#76106
`connect()` caches every path it has initialized in the process-local
`_INITIALIZED_PATHS` set and then skips all first-open work for it —
header validation, integrity probe, `SCHEMA_SQL`, additive migrations.
That cache is keyed on a path, but the schema it stands for lives in a
file, and the two can drift apart: delete or replace `kanban.db` under a
live gateway/dispatcher/dashboard process and the next `connect()` takes
the fast path, lets SQLite create a fresh empty database, and hands back
a connection with no tables in it.
Nothing notices. Every query then fails with `no such table: tasks`,
`plugin_api._conn()` logs its init warning and carries on, and the board
renders empty. Because the cache entry survives, the process re-creates
the same schema-less ~4 KB file on every restart of the desktop app in
front of it — only killing the backing process clears it.
Verify the sentinel table on the fast path and self-heal when it is gone:
drop the stale cache entry and fall through to the existing init path,
which re-runs the probes and the schema script under the cross-process
init lock. The check is one `sqlite_master` lookup on the already-resident
page 1, so the steady-state path stays lock-free (#36644) and does no
schema work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-up to HexLab98's salvaged commits:
- Apply the same bounded worker join before raising InterruptedError at
the two sibling interrupt sites that share the raise-without-join
shape: the non-streaming API poll loop and the Bedrock streaming poll
loop. Both workers run Relay-managed physical attempts, so raising
immediately allowed turn teardown to race a still-open physical scope
exactly as in the streaming path.
- Address the #81601 review finding (egilewski): the pinned nemo-relay
binding's get_scope_stack() returns a native ScopeStack object which
scope.pop rejects with TypeError, so the orphan drain never drained
under the real binding. current_top() now prefers the version-correct
scope.get_handle() accessor and falls back to the old list-unwrap for
fake/legacy shapes. Handle comparisons go through same_handle(),
comparing by uuid, because native ScopeHandle instances do not
implement value equality.
- Add a real-binding regression test that reproduces the orphaned-scope
session close against the pinned native wheel (skips where the native
binding is unavailable), alongside the existing fake-based coverage.
The gateway auto-restart phase used to swallow every exception at debug
level, so tests driving cmd_update end-to-end never noticed it touching
real gateway discovery. With #78574 surfacing an aborted restart as a
failed update, an unmocked find_gateway_pids on a box with a live
gateway hits the conftest live-system guard and turns into a spurious
sys.exit(1).
Add an autouse fixture in test_cmd_update.py (discovery returns nothing,
systemd unsupported) and the same seams in test_update_head_moved_gate's
helper so the phase is a clean no-op for tests that do not assert on
gateway restarts.
Reviewer egilewski found the original defer was circular (#83590 comment):
the self-lock preflight wrote .update-incomplete and exited, but the next
launch only ran the full recovery AFTER main.py's third-party imports —
so a healthy venv's probes made the early pass a no-op, main.py imported
cryptography eagerly, the .pyd got mapped again, and the deferred install
re-hit the exact self-lock it was meant to escape.
Close the loop by making the marker guarantee the install runs BEFORE any
native extension can be imported:
- hermes_cli/_install_repair.py (new, stdlib-only): single source of truth
for the core .[all] reinstall — ensurepip bootstrap, uv-pip/pip
resolution with VIRTUAL_ENV, Termux env stripping, Windows hermes*.exe
quarantine, per-extra fallback ladder, and fd1→fd2 routing for acp
safety. Deliberately free of managed_uv/hermes_constants imports so it
stays importable in the corrupted-venv state it exists to repair.
- hermes_cli/_early_recovery.py: recover_if_needed now completes a pending
.update-incomplete install BEFORE the import probes, on every launch
that sees the marker (unless argv is update). Success clears the
marker; failure bumps an attempts counter inside the marker body and
keeps it. A 3-attempt ceiling stops a persistently-failing install
from reinstall-hammering every launch (hermes acp included) — past the
ceiling the late post-import recovery takes over with its manual
recovery instructions. Single-flight lock shared with the late path.
- hermes_cli/main.py: _recover_core_update_marker_locked delegates the
install to the shared executor (no duplicated logic); ensure_uv stays
in the late path so a venv whose uv vanished mid-update still
bootstraps it.
- tests: 7 new regressions — the reviewer's exact case (marker + healthy
venv → install runs while sys.modules has no cryptography), failure
keeps marker + increments attempts, retry ceiling, lazy marker does
not trigger core install (#58004 invariant), argv-update skip, and
corrupt/missing marker bodies. The key test was sabotage-verified:
removing the pre-import branch makes it fail with zero install calls,
while a lone-lazy-marker test still passes; restoring the branch makes
it pass again.
Refs #83569
Two gaps left every Windows git-checkout install unable to recover from
the exact failure state #83569 reports:
1. Self-lock detection. _detect_venv_python_processes() always excludes
the calling process by design — a CLI hermes update IS the venv python.
An updater that had already imported a native venv extension (the
canonical one being cryptography.hazmat.bindings._rust, mapped while
hermes_cli.main resolved external secret sources) passed every
preflight and then died mid-sync with os error 5 when uv tried to
rewrite the mapped .pyd, stranding the venv half-updated. A new
preflight now refuses the sync before touching the checkout, writes
the update-incomplete marker so the next fresh launch completes the
install, and exits 2. Verified on a live Windows 11 host: after
importing hermes_cli.main, tasklist /m _rust.pyd shows the .pyd mapped
in the caller, and a peer process cannot open it read-write
(Permission denied) — while a rename succeeds, matching how uv/pip
actually fail (truncate+write, not rename).
2. Early-recovery install path. _early_recovery._run_repair_install used
sys.executable -m pip unconditionally. Windows git checkouts install
on a uv-managed base interpreter (python-build-standalone), whose
EXTERNALLY-MANAGED marker makes plain pip abort with
externally-managed-environment — the repair no-oped and the venv
stayed broken. The repair now detects the PEP 668 marker, prefers
uv pip install with VIRTUAL_ENV pointed at the project venv, and
falls back to pip --break-system-packages when no uv binary exists.
Both fixes ship with subprocess/unit regressions (sabotage-verified):
the new tests fail on pre-fix code and pass with it. Complements #77517,
which keeps the updater from importing cryptography in the first place;
this PR is the defence-in-depth when any future path loads it anyway.
Fixes#83569
_cold_start_windows_gateway_after_update() printed the success line off a
successful Popen return alone, which only proves CreateProcess succeeded,
not that the child survived. On Windows, a job object denying
CREATE_BREAKAWAY_FROM_JOB hard-kills the child during updater teardown
before it logs anything, yet the updater still printed "Starting Windows
gateway after update (PID ...)" — leaving Telegram/Discord/etc. offline
with no indication anything failed (#84185).
Route the success report through gateway_windows._report_gateway_start(),
the same post-spawn liveness poll every other _spawn_detached() caller
already uses, so a dead child is reported as a failure with a
manual-recovery hint instead of a false success.
Review follow-up (#78574): the aborted-restart handler only flagged the fleet
stale when the post-failure survivor probe was None or non-empty. A positive
empty probe was treated as proof-of-safety — but `[]` is only safe when
nothing was running before the phase. If a gateway was discovered, stopped
(SIGTERM/drain), and its replacement never came back, the probe is empty at
exactly that unsafe moment and the update reported success — the fail-open
contract this fix exists to close.
Snapshot the pre-restart gateway PIDs before any stop/drain and route the
handler decision through a pure _restart_phase_failure_is_incomplete() helper
that fails closed on an empty survivor set whenever a gateway existed
pre-restart (or the pre-state could not be read). Add decision-level regression
tests covering the stopped-without-replacement gap, unknown pre-state, and the
truly-no-gateway positive control.
The gateway auto-restart phase in `hermes update` was wrapped in a blanket
`except Exception` that only logged at debug level. When the phase raised
early — e.g. importing `hermes_cli.gateway` from the freshly pulled checkout
inside a process that already loaded pre-update modules — every drain and
restart line vanished from the update output, the update printed
"Update complete!" and exited 0, and the still-running gateway kept serving
pre-update modules against replaced source files. The next Telegram turn died
with `ImportError: cannot import name 'is_trivial_prompt'`.
The handler now probes for surviving gateway processes and, unless it can
positively prove none are running, prints the cause plus a manual recovery
command and marks the fleet restart incomplete — which exits nonzero and
writes the gateway-mode exit-code marker, matching the existing
failed-or-stale-unit path.
Fixes#78574
A detached/pinned checkout can report 'N new commit(s)' against origin,
run the ff-only merge successfully, and still sit on the old commit
afterward (the branch-switch step re-detaches to the raw SHA). Before
this guard 'hermes update' printed '✓ Code updated!' and reinstalled
deps + rebuilt the desktop app against the stale tree - no error, no
warning, 'hermes doctor' healthy.
Compare pre-pull and post-pull HEAD; if they match, fail loudly with a
reattach hint instead of claiming success.
The shared load-boundary flatten added for dict-valued model.default only
ran on the user/default merge; _load_config_impl then deep-merged the raw
managed overlay without normalizing, so a managed model.default:
{provider, model} still reached status/fallback/runtime readers as a dict.
Normalize the managed overlay (same _normalize_root_model_keys pass, plus
the bare model-string -> model.default promotion used by
managed_scope.apply_managed_overlay) before expanding and merging, so
every overlay is canonical before load_config returns.
Adds load_config() regressions for a nested managed default and a bare
managed model string.
Layer 2 of the #81163 / #78050 fix: _get_platform_tools computed
plugin_ts_keys = _get_plugin_toolset_keys() but only used
CONFIGURABLE_TOOLSETS in the explicit-config filter, so a user-listed
plugin key like `a2a` in `platform_toolsets.cli: [hermes-cli, a2a]` was
silently dropped. The filter now unions configurable and plugin toolset
keys when evaluating has_explicit_config and when admitting per-key
entries.
Cherry-picked from PR #81190 (Layer 2 hunks only; Layer 1 is covered by
the provides_tools mechanism from PR #78842).
Rebased onto current main. `hermes_cli/plugins.py` grew 103KB -> 265KB
across 49 commits since the original branch point, and the attribution
mechanism this change hooks into was replaced along the way: the
`_tools_before` / `_plugin_tool_names` snapshot diff is now a
registration ledger sliced from `registration_start`, and `_plugin_id`
is `plugin_key`.
Re-anchored accordingly:
- Discovery-time pre-registration, module reuse, and the `provides_tools`
opt-in are unchanged.
- Attribution credits `_predeclared_tools` ahead of the ledger slice,
since those tools registered before `registration_start` and the slice
cannot see them.
- A failed materialization no longer carries attribution across. The
failure path now sweeps the whole ownership ledger for the plugin key,
not just the `registration_start:` slice, so the pre-registered tools
are disposed along with the adapter. Attribution and the registry now
agree at zero instead of reporting tools the process is not serving.
tests/hermes_cli/test_deferred_platform_client_tools.py 13/13.
test_plugins.py, test_plugins_cmd_list.py, test_plugin_cli_registration.py
65/65.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pre-creates a 0o6755 target, imports a member over it, and asserts the
published file is 0o755 with both elevated bits gone — plus that the
staged temp file never carried them either, so there is no window where
archive content sits behind an elevated mode.
The existing coverage in this class cannot see the failure: every mode
assertion masks with ``& 0o777``, which discards exactly the bits at
issue, and the fixtures chmod their targets to ordinary modes that never
had them set. Without the mask on the preserved mode this test reports
the published file still holding S_ISUID.
Skipped where the platform or filesystem refuses setuid on a user-owned
file, so the assertion never depends on running as root.
Follow-up on the atomic-import restore, delegating both metadata concerns to
the shared helpers instead of half-handling them locally.
Owner preservation was missing entirely. `tempfile.mkstemp` + `atomic_replace`
publishes a temp file owned by the *writing* user, so `sudo hermes import`
re-owned every restored file to root — on the disaster-recovery path, and on
exactly the Docker/NAS volume installs `utils._restore_file_owner` was added
for. `_extract_member_atomically` now captures `_preserve_file_owner(target)`
before staging and calls `_restore_file_owner` after the replace, before the
mode restore (chown clears setuid/setgid, so the mode has to go back last).
Mode handling was also only half applied before the replace: the `os.fchmod`
branch applied it to the temp fd, but the platforms without `fchmod` fell
through to a best-effort post-replace chmod, leaving the published file at
mkstemp's 0600 until that chmod landed — permanently if the process died in
between — and making `atomic_replace`'s EXDEV/EBUSY `shutil.copystat` fallback
copy 0600 onto the target. The mode is now applied to the temp file on both
branches, with the post-replace `_restore_file_mode` kept as the belt-and-
braces path.
This is the same shape `atomic_write_text` and `atomic_yaml_write` already
carry after 3556728a5 and 43fc86562; capture and restore now reuse
`utils._preserve_file_mode` / `_preserve_file_owner` / `_restore_file_mode` /
`_restore_file_owner` rather than re-deriving them, which also drops the local
`import stat`.
Tests (tests/hermes_cli/test_backup.py, class TestImportAtomicWrites):
- test_restore_preserves_existing_file_owner — forces a uid/gid so it does not
need root; asserts chown fires once, with the captured owner, on the
pre-existing file only (a newly created member has no prior owner).
Mutation-checked: dropping only the `_restore_file_owner` call reds it.
- test_mode_is_applied_before_the_replace_without_fchmod — `monkeypatch.delattr`
on `os.fchmod`, spies the temp file's mode at replace time. Reads 0o600
without the fix, 0o644 with it. Mutation-checked the same way.
`hermes import` wrote every zip member with `open(target, "wb")` followed by
`dst.write(src.read())`, at both restore sites in `run_import`. Opening for
write truncates the user's existing file to zero *before* any replacement
bytes exist, so a Ctrl-C, an ENOSPC, a corrupt zip member, or a crash leaves
`config.yaml`, `.env`, or an external provider config (e.g.
`~/.honcho/config.json`) empty with nothing behind it — during the
disaster-recovery path the user is running precisely because they already
lost something. The `_external/` branch writes outside HERMES_HOME, into
third-party configs under the user's home, so the blast radius is not
confined to Hermes state.
Both sites now stage the member into the target's own directory, fsync it,
and publish with `utils.atomic_replace`, so the target only ever moves from
its old contents to the complete new contents.
`atomic_replace` rather than a bare `os.replace`: it resolves a symlinked
target first, so deployments that link `config.yaml` into a dotfiles repo
keep the link instead of having it silently swapped for a regular file
(#16743), and it falls back to copy/fsync/unlink on EXDEV/EBUSY for
cross-device and bind-mount installs. Members stream through
`shutil.copyfileobj` instead of being read whole into memory. The temp file
is removed on any failure so a partial import leaves no residue, and
permission bits are carried across the replace so mkstemp's 0600 does not
silently tighten restored files.
This extends the module's own established idiom — `backup.py` already
publishes atomically via `os.replace` in `_atomic_output_path` and in the
snapshot writer — into the one path that still overwrote user files in place.
Filter manual dashboard/serve respawn candidates after update: skip
ephemeral --port 0 backends (Desktop-owned), dedupe normalized cmdlines,
and cap one restart per profile/HERMES_HOME so orphan counts no longer
grow across successive updates.
Compose the service-PID exclusion (#85743, RelaxJonh) and the recorded-PID +
parent-chain exemption (#86100, arccat-114) into one cross-platform rule:
- _get_service_pids() exclusion now runs unconditionally, not only under
is_macos() — it is the authoritative "supervised" signal for launchd and
any systemd unit visible on a host that got past the systemd gate.
- The recorded-healthy-gateway (get_running_pid()) + parent-chain exemption
now runs on every platform, not only Windows. A recorded, liveness-verified
gateway is by definition not an orphan "the pidfile/runtime record can't
see", so the reaper must never target it — this covers Windows Scheduled
Task / Startup VBS supervision, standalone launcher-started gateways
(the case #85743 alone would miss), and macOS/WSL equivalents.
True orphans (no service registration, no valid runtime record) are still
found and reaped, preserving the #51325/#75936 duplicate-port protection.
Existing macOS regression tests updated to pin get_running_pid to None for
their scenario; Windows regression tests from #86100 carry over unchanged.
Bug class: #83683 (root), #86287, #86098, #85738, #85368, #85344, #85044,
#84855, #84824, #84200.
The orphan reaper kills a healthy gateway (and its Scheduled-Task bootstrap
parent chain) every time the Desktop backend starts on Windows, because
_get_service_pids() only implements systemd/launchd and returns an empty
set on Windows — a supervised gateway is therefore indistinguishable from
an unsupervised orphan.
Exempt the recorded healthy gateway PID and its parent chain from the
orphan scan on Windows, mirroring the macOS launchd exemption (#85913).
The Scheduled-Task bootstrap's argv matches the gateway scan, so without
exempting the parent chain killing the bootstrap takes the detached
gateway down with it.
Fixes#86098
After `hermes update`, an existing state.db on an old schema made every
GET /api/sessions poll fail with sqlite3.OperationalError "no such
column: s.last_read_at" (or s.last_activity_at) until something
unrelated forced a writable open — the desktop sidebar showed "No
sessions yet" while every row sat intact on disk (#79531, #80037).
Two remaining root causes (the stale hand-written read probe was
already replaced by the SCHEMA_SQL-derived probe on main, prototyped in
draft PR #80030 by @Tilly-YL):
1. Migrations ran lazily: _init_schema/_reconcile_columns only ran on a
writable open, typically the user's first NEW session. The dashboard
backend now schedules one writable open of its own state.db from the
lifespan (daemon thread, never blocks the ready-probe socket, never
raises), so the store is brought current before the first session-
list poll on every `hermes serve` / `hermes dashboard` / Desktop
headless entrypoint.
2. _reconcile_columns caught sqlite3.OperationalError around every
ALTER TABLE ADD COLUMN and logged at DEBUG. Lock contention from
orphaned sibling backends made the ALTER fail silently — startup
"succeeded" with a half-reconciled schema, and the open-time lock
patience (#74478) never saw the error because it was swallowed
inside first. Now: "duplicate column" races stay at DEBUG,
locked/busy re-raises so _connect_and_init_with_lock_patience
retries the whole idempotent init with jittered backoff, and any
other failure (e.g. un-ADDable NOT NULL) logs at WARNING.
Regression tests: a store missing sessions.last_read_at is healed by
the eager startup reconcile and serves list_sessions_rich; a locked
ALTER propagates and is retried to success by the open lock patience;
duplicate-column races stay quiet; other ALTER failures warn.
Fixes#79531Fixes#80037
Reported-by: @yenhunghuang (#79531) and @FLOW3R0111 (#80037)
Root-cause analysis: @wangyi0177-eng (stale read probe) and
@www654cc-pixel (_reconcile_columns DEBUG-swallow under lock
contention); draft PR #80030 by @Tilly-YL prototyped the probe fix.
The Hermes-managed Node tree at %HERMES_HOME%\node is destructively
rewritten while the desktop app's Node processes execute from it:
the Node-26 heal did shutil.rmtree + move, the EBADENGINE repair ran
npm install --global --prefix into the tree, and install.ps1's
Test-Node did Remove-Item + Move-Item. Windows rejects those writes
with PermissionError: [WinError 5] on npm.cmd.
- _heal_managed_node_windows: stage the fully-downloaded tree in a
sibling node.new-* dir, then rename-swap (live tree -> node.old-*,
staged -> node). The live tree is never deleted before its
replacement is ready, so an interrupted heal cannot gut it; a
refused rename is the OS-level in-use signal and defers (returns
None) instead of forcing the write.
- heal_hermes_managed_node: an in-use deferral does not record the
once-per-process attempt, so the heal retries once the tree is free.
- managed_node_tree_in_use: cheap psutil pre-check (Windows only) that
avoids pointless 30-50MB re-downloads in long-lived processes.
- upgrade_managed_npm: defer the in-place npm self-upgrade while the
tree is in use, with a notice.
- install.ps1: Test-ManagedNodeInUse guard around Update-ManagedNpm and
the Test-Node install branch, which now rename-swaps instead of
delete-then-move.
An in-use-but-outdated tree keeps serving the old runnable Node (old
Node beats no Node), and every npm resolution re-evaluates the heal, so
the upgrade applies automatically on the next update with the app
closed.
Dedupe key now includes tool_call_id/tool_calls/tool_name: compaction
copies carry those fields verbatim, so identical tool messages across
generations still collapse, while distinct tool calls sharing
role/content/timestamp are never merged. Add endpoint-level coverage
for the desktop's real read path (limit + order=latest +
include_compacted=true).
The s6 finish script for profile-gateway services restarted on ANY exit
except EX_CONFIG (78) — including clean exit 0. Restart-on-normal-exit
turns an intentional stop into a reconnect loop: the ashriel-discord
storm in #76435 made 1,000+ connections and got the bot token reset by
Discord.
The finish script now exits 125 (permanent failure, no restart) for both
clean exit 0 and EX_CONFIG; only non-zero, non-78 exits (genuine crashes)
restart normally.
Scope note: #76435 bundles a second, separable symptom (Windows desktop
updater showing the literal 'managed outside dashboard' sentinel). Its
root cause is undiagnosed and #22733 covers the dialog-explanation path;
this PR is the gateway half only.
Tests: behavioral — the rendered finish script is executed via sh for
exit codes 0/78/1/137, asserting no-restart for clean stop and fatal
config, restart for crashes. Pre-existing EX_CONFIG test still passes.
Port of the bug class from earendil-works/pi#7933 (DeepSeek base-URL
detection matched by raw substring, missing case variants and matching
lookalike URLs). Hermes had the same class at five sites:
- cli_agent_setup_mixin.py: keyless-custom-endpoint detection treated any
URL containing the OpenRouter host substring (path segment, lookalike
domain) as OpenRouter, and missed case variants of the real host.
- models.py validate_requested_model: same substring check for routing an
openrouter provider with a custom base_url to the custom catalog.
- runtime_provider.py: local-endpoint autodetect matched the string
localhost anywhere in the URL, including remote hostnames containing it.
- gateway/run.py: /status endpoint display, same local-host substring.
- agent_runtime_helpers.py: Nous Portal cache-layout detection matched
the nousresearch substring anywhere in the URL.
All sites now use the existing base_url_host_matches / base_url_hostname
helpers (exact host or subdomain, case-insensitive). Regression tests
proven to fail against the old predicates.
- The gateway api_server fire webhook acknowledges 202 only after a
durable claim + execution row exist (admission failure stays retryable
as 503; a live claim answers 200 duplicate), then dispatches the
claimed snapshot with the live runner adapters (delivery parity with
the built-in ticker, including relay-fronted and E2EE platforms).
- Legacy single-phase providers (a documented fire_due override without
split hooks) keep being driven through their own hook. Capability
detection now credits claim_fire AND fire_claimed overrides, so
Chronos is correctly classified split-aware (its re-arm lives in
fire_claimed; the redundant fire_due passthrough override is removed).
- Multi-profile dashboards fail closed for external providers: an
unscoped reconcile would disarm other profiles' armed one-shots in the
shared NAS registry.
- Manual runs (cronjob run) carry the owner-bearing claimed snapshot
through every entry point, composing with upstream's manual-run
heartbeat (#76502) and background dispatch.
Note: current main moved the dashboard NAS webhook to a pure
forward-to-gateway design (the gateway owns execution and live
adapters), so the dashboard-side claim/tracking machinery from earlier
revisions of this PR is dropped; the durable admission contract lives in
the gateway webhook path.
ActualProfile.fetch_models() overrides ProviderProfile's default
implementation with its own Actual-specific base_url resolution
(ACTUAL_BASE_URL env var, hosted-vs-local normalization), but called raw
urllib.request.urlopen(req, timeout=timeout) directly instead of the base
class's open_credentialed_url(). Every other provider either uses the
base class default or forwards to it via super() and gets
SafeCredentialRedirectHandler for free — Actual is the only provider that
attaches a Bearer token to its own Request object and opens it with the
stdlib's default redirect handling, which forwards every header,
including Authorization, across a cross-origin redirect.
Actual's own feature surface makes the trigger realistic: ACTUAL_BASE_URL
is a first-class, documented way to point this provider at a self-hosted
or local-offline endpoint (see the local-loopback no-auth path already
handled elsewhere in this provider), so a misconfigured or compromised
endpoint 302-ing to another host leaks ACTUAL_API_KEY to it.
Fix: import and call the same open_credentialed_url() the base class
uses, keeping Actual's own URL-resolution logic unchanged.
Adds an end-to-end regression test using two real local HTTP servers (no
mocking of the security module itself) — one redirects, the other
records the Authorization header it receives — mirroring
test_urllib_security.py's own redirect tests. Also repoints the existing
fetch_models test's mock from urllib.request.urlopen to
hermes_cli.urllib_security.open_credentialed_url, since fetch_models no
longer calls the former. Mutation-verified: the new redirect test fails
on pre-fix code with the Authorization header observed at the redirect
target.
Earlier releases accepted api_mode: openai on custom provider entries.
The canonical transport set is now {chat_completions, codex_responses,
anthropic_messages, bedrock_converse, codex_app_server}, and an
unrecognized value was silently ignored at both consumption sites
(_normalize_custom_provider_entry passes the raw string through and
agent_init's accepted-set check drops it; _parse_api_mode returns None),
falling through to hostname-based detection.
For hosts with a detection rule the provider silently switches
transports after an update. Observed live: a custom entry for
api.actual.inc with api_mode: openai (valid when written) flipped to
codex_responses via the hostname rule, and every reasoning-bearing
request to the relay's /v1/responses failed with a wrapped non-JSON
error while /v1/chat/completions worked throughout.
Fix: one shared alias map (_canonical_api_mode) consulted by both
sites. openai/openai_chat -> chat_completions, responses ->
codex_responses, anthropic/messages -> anthropic_messages, bedrock ->
bedrock_converse. Canonical names and unknown values pass through
unchanged, so invalid-config behavior is untouched.
Tests: alias map contract (every alias lands in _VALID_API_MODES),
normalizer canonicalization incl. the transport: key alias, and the
runtime gate accepting legacy spellings while still rejecting unknowns.
* feat(dashboard-auth): extend RFC 8252 native sign-in to password providers
The desktop app runs password sign-in for gated gateways in an embedded
Electron BrowserWindow, where OS password managers (macOS Passwords /
iCloud Keychain autofill) cannot reach the form — Chromium-in-Electron
has no bridge to them, so users retype credentials by hand even though
the /login form already carries the right autocomplete attributes.
The existing RFC 8252 native flow (system browser + loopback + PKCE)
solves exactly this for OAuth providers, but was explicitly disabled for
password providers on the grounds that they have "no IDP round trip to
broker". The brokering is still worth having: it moves the credential
form into the system browser, where password-manager autofill just works.
Gateway-only change; the desktop needs no changes (runNativeLogin is
already page-agnostic), and older desktop builds pick the capability up
automatically once the gateway advertises it:
* /auth/native/authorize now accepts a supports_password provider:
register the pending broker authorization as usual, then 302 the
system browser to the interactive /login form with the opaque
broker_state in the gateway's PKCE cookie (the same server-controlled
channel the OAuth branch uses) instead of an IDP redirect.
* /auth/password-login: when the server-set PKCE cookie carries a
broker handle, a successful credential check completes the pending
authorization exactly like the /auth/callback native branch — mint
the one-time loopback code, return the loopback redirect (validated
loopback-only at authorize time) as `next`, clear the PKCE cookie,
and set NO session cookies. A lapsed broker is a clean 400 telling
the user to restart sign-in; a failed credential attempt leaves the
pending entry intact so the user can retype.
* /api/status now advertises "native_pkce" whenever any interactive
session provider is registered (previously only for non-password
providers), so the desktop selects the system-browser strategy for
password-only gateways.
Security posture is unchanged from the existing flow: loopback-literal
redirect_uri enforcement, PKCE S256 binding, single-use short-TTL codes,
constant-time comparison, and the same rate limiter on password attempts.
Tests: full authorize → /login → password-login → loopback → token →
bearer round trip, wrong-password keeps the pending entry, lapsed broker
→ 400, no-broker browser login keeps minting cookies, and the /api/status
advertisement for password-only gateways.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(dashboard-auth): bind native password completion to the authorize-time provider
Review follow-ups for #75808:
* /auth/password-login now enforces that body.provider matches the
provider recorded in the server-set PKCE cookie by
/auth/native/authorize before completing a pending native
authorization. /login renders a form for every session provider, so
without this a native flow started for provider A could be completed
with provider B's credentials, binding B's session into A's pending
entry. The mismatch is rejected BEFORE credential verification (no
session minted, no oracle) and preserves both the pending entry and
the cookie, so the user can still submit the correct provider's form.
Covered by a two-password-provider E2E regression test.
* Update the two docs spots that still said password-only providers do
not advertise native_pkce (website desktop-native-signin guide and the
auth_flows type comment in web/src/lib/api.ts).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: map contributor email for #75808 (buffpesos)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Brooklyn Nicholson <brooklyn.bb.nicholson@gmail.com>
Everything Browser Use is now managed by Hermes: the canonical binary
is the one install_cli() provisions into HERMES_HOME/bin, and every
resolution and provisioning site prefers it.
- _find_cli(): probe order flipped to managed bin -> PATH ->
user-level tool dir (then uvx across the same order). A user's own
uv tool install can no longer shadow the Hermes-managed copy with a
drifted version; side installs only matter when we have nothing.
- install_cli(): a browser-use on PATH no longer short-circuits the
install — only the managed copy does, so selecting any backend
provisions the copy Hermes controls and updates.
- _ensure_browser_use_cli() (hermes tools): drops its own PATH check
and always delegates to install_cli(), the single owner of the
managed-copy policy.
- install.sh / install.ps1: same short-circuit fix — only
HERMES_HOME/bin/browser-use counts as installed.
Follow-up to #86240 and #86320: with every non-Camofox backend
selection installing the CLI, managed-first closes the remaining
version-drift/shadowing class instead of guarding single sites.
Tests updated to pin the new contract: managed beats PATH and
user-local; PATH install does not satisfy install_cli; helper always
delegates. E2E-verified precedence chain with real files and a real
degraded-PATH install attempt.
Fixes#84851
The SSH fast path in _check_via_local_git compared only the exact tip SHA
of local HEAD against upstream main. When a local carried commit makes the
SHAs differ, it returned 1 ('behind') without checking ancestry, so an
ahead-of-origin checkout was misreported as '1 commit behind' — nudging the
user to run 'hermes update' and wipe their carried work. Fall back to
git rev-list --count HEAD..origin/main (mirroring the full-clone path) when
the SHAs differ.
Ports Claude Code's /loop (and its /proactive alias) across every Hermes
surface. /loop [interval] <prompt> re-runs a prompt or slash command on a
recurring cadence inside the live session; omitting the interval enables
self-paced mode (starts at the floor, backs off exponentially while the
agent's replies stop changing, snaps back on change — local digest
comparison, zero extra LLM cost).
Stop conditions: agent-emitted LOOP_COMPLETE marker, --times N,
--until <condition> (judged by the existing goal_judge aux task,
fail-open), /loop stop, and a loops.max_ticks backstop budget.
Core: hermes_cli/loops.py (LoopState + LoopManager + shared
dispatch_loop_command), persisted per session in SessionDB state_meta
(loop:<sid>) so /resume picks it up; migrates across compression
boundaries like /goal. New SessionDB.list_meta_prefix() powers the
gateway's cross-session scan.
Surfaces:
- CLI: /loop handler + idle-fire and post-turn-complete hooks in
process_loop (mirrors the /goal hook shape; Ctrl+C pauses the loop)
- Gateway: /loop handler with route capture, mid-run control-verb guard,
post-turn tick completion, and a supervised loop_wakeup_watcher that
injects due wakeups into idle chats via the synthetic-message path
- TUI/dashboard/desktop: command.dispatch handler + per-session
notification-poller wakeup driver + post-turn completion in the turn
dispatcher; /loop added to the desktop slash palette
- /goal mixing: an active non-parked goal owns the idle boundary — loop
ticks defer until it finishes, pauses, or parks; real user input always
wins over both
Config: loops.{min_interval_seconds,max_ticks,self_paced_floor_seconds,
self_paced_ceiling_seconds}. Docs page + sidebar entry. 77 new tests.
Slack's 50-slash cap: /version moves to /hermes version to free the
native slot for /loop.
Teknium