`hermes profile delete` read the target profile's gateway.pid raw and
SIGTERMed it. When that pid file was poisoned by a sibling profile's gateway
(the #89315 shape), deleting profile A killed profile B's running gateway.
- gateway/status.py: `_pid_record_belongs_to_profile()` helper — a pid
record whose recorded home differs from the expected profile home is not
ours; legacy records without a home prove nothing and are left alone.
- hermes_cli/profiles.py: `_stop_gateway_process` refuses (and says so)
when the record belongs to another profile; still stops its own gateway.
The stop/restart paths in hermes_cli/gateway.py did not need a guard:
`get_running_pid()` already filters cross-profile records and unlinks the
poisoned pid file before any kill can happen — verified live; the test for
that path now pins the real contract (returns False, other process alive,
poisoned pid file gone).
Live repro (unpatched main): `_stop_gateway_process(tim_home)` -> "Gateway
stopped (PID ...)" and the OTHER profile's process exits -15. After: "Refusing
to stop PID ..." and the process stays alive. 8 tests; sabotage (guard
removed) fails 1.
Add _pid_record_belongs_to_current_profile() helper that verifies a
PID record's persisted hermes_home matches the current process. Use
it in get_running_pid() and get_runtime_status_running_pid() so the
default-profile gateway never mistakes another profile's gateway PID
as its own.
In _apply_profile_override(), clear HERMES_HOME instead of returning
early when it points to a profile directory but no --profile flag was
given, letting the sticky active_profile logic resolve the right one.
In _guard_existing_gateway_process_conflict(), detect stale PID files
from other profiles and emit a warning.
Salvage hardening on top of the three cherry-picked contributor commits
(#91297 gebilaowang404 + AlexMnrs, #96741 burak33bb, #98826 ayushnangia),
closing the remaining unverified-PID kill sites as one class (#98814, #89614):
- pid_is_hermes: token-boundary 'hermes' match (no more loose substring
false-positives), and an explicit start-time expectation is now honored
on POSIX too (a mismatched fingerprint is a recycled PID on any platform).
- kill_process_tree: drop the guard on our OWN retained Popen child — a
retained handle pins the PID, so the check could only false-refuse.
- gateway.status.terminate_pid: POSIX force-kills also refuse when a
caller-provided expected_start_time no longer matches.
- kill_gateway_processes: re-verify the LIVE cmdline at kill time (the
scan-time match is a TOCTOU window).
- _reap_unsupervised_gateway_orphans: fingerprint orphans at scan time and
require a still-matching identity before the delayed SIGKILL escalation.
- whatsapp _kill_port_process: never kill a bare netstat/lsof-scanned PID
unless the live process is actually a node bridge (was a stranger-kill).
- browser daemon reap/close paths: pass the start-time fingerprint into
ProcessRegistry._terminate_host_pid (previously unverified), and the
session-close path now runs the same daemon identity verification as
the orphan reaper.
- tests/hermes_cli/test_taskkill_identity_windows_live.py: live Windows
probes (real spawned processes, real psutil ancestry) wired into the
on-demand windows-latest wine2e lane.
Fixes#98814Fixes#89614
A long-lived process whose checkout was updated underneath it (hot git
pull, interrupted hermes update) serves mixed sys.modules. When such a
stale process races a fresh gateway for the cron tick lock and wins the
minute, every agent job it dispatches can die on ImportErrors whose real
cause is staleness — and the fresh gateway's ticker skips the same minute
as lock-loser, so the user's scheduled job fires broken or not at all.
tick() now checks, BEFORE acquiring the tick lock:
skew detected (boot fingerprint != disk revision)
AND this process does not own the gateway runtime lock
AND that lock is held (a fresh gateway is alive)
-> raise CronTickYielded, skipping the tick entirely
Each arm alone keeps the old behavior:
- skew + self-owned lock -> proceed (delivery-path stale-code hint stays
the surface for gateway-owned dispatches)
- skew + no lock holder -> proceed (desktop-standalone users must not
lose their only ticker to a silent yield)
- skew None (non-git install, no boot fingerprint, probe failure) ->
proceed; yielding is a certainty claim, never a guess
The yield RAISES instead of returning 0 so the provider loops record it
via record_ticker_error and mark the heartbeat success=False — a yielded
tick must not look like a healthy one (hermes cron status shows why),
mirroring the EMFILE propagation contract (#87644). Yield logging is
throttled to once per skew episode. Self-healing: when the fresh gateway
dies, its lock releases and the stale ticker's next tick proceeds.
Multiplex loop: a yield for one profile no longer cancels sibling
profiles' ticks in the same cycle; only the yielding profile records an
unsuccessful beat.
gateway/status.py gains owns_gateway_runtime_lock() —
is_gateway_runtime_lock_active() is True for the lock's own owner too, so
a caller deciding whether to yield to a FRESH gateway needs the
in-process handle as the discriminator.
On Windows installs where the gateway runs as an SCM service (WinSW,
NSSM, sc.exe create), the existing pause machinery kills the gateway
process directly — and the service wrapper's failure ladder resurrects
it within seconds, re-taking the venv file locks mid-update. The update
then dies partway through dependency sync with access-denied errors.
This extends _pause_windows_gateways_for_update() to detect when a
gateway's process tree is owned by a running SCM service, and to stop
the SERVICE through sc.exe instead of killing the child:
- gateway/status.py: expose service-ownership discovery for gateway
runtimes (find_windows_gateway_services maps validated gateway PIDs
through process ancestry to running SCM service PIDs, with
create-time identity checks against PID reuse).
- hermes_cli/update_cmd.py: stop verified services via sc.exe before
venv mutation and restart them afterward. Stops wait for a stable
SCM 'stopped' state AND for the original descendant processes to
exit (service 'Stopped' is not proof the child released its
handles). Failure to prove ownership, stop a service, or restart it
fails closed; rollback restores attempted services, and rollback
failures are surfaced rather than swallowed.
- Fail-closed throughout: unreadable identities, ambiguous ancestry,
or a service that will not reach a stable state abort the update
before any file mutation.
Complements #37039 (gateway-only concurrent instances no longer abort):
that fix lets the update proceed past the gate; this one makes the
pause actually stick when the gateway is service-supervised.
Note: tests/gateway/test_status.py::TestReadProcessCmdlinePsFallback::
test_ps_fallback_when_proc_unavailable fails on Windows on current main
before this change as well (POSIX ps fallback asserted on a platform
without it); all other touched suites pass (155 passed, 5 skipped).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 1 of the fleet-update reliability plan (#91277): the updater now
proves its outcome instead of assuming it.
- hermes_cli/build_info.py: get_code_identity() — process-cached code
identity (git sha for source installs, baked .hermes_build_sha for
Docker images, pyproject version).
- gateway/status.py: every runtime-status write stamps the writer's
code_sha/code_version into gateway_state.json, so a running gateway's
actual code generation is observable from disk.
- hermes_cli/update_receipt.py (new): machine-readable receipt of each
update run (steps, skips with reasons, gateway restart outcome, fleet
snapshot) under ~/.hermes/logs/update_receipts/ with a latest.json
pointer for the dashboard/desktop; plus collect_fleet_versions() /
print_fleet_version_matrix() comparing every live profile gateway
against the freshly updated checkout.
- hermes_cli/update_cmd.py: wires receipt begin/steps/finalize into the
git, ZIP, and hard-failure paths; after the restart phase, prints the
fleet version matrix and escalates provably-stale gateways into the
existing gateway_fleet_restart_incomplete exit-1 contract. Pre-stamp
gateways report 'unknown' and never fail the update (no false
positives during rollout).
Silent-failure classes made visible: #88848, #74973, #85753, #81193.
Mixed-version fleet classes made loud: #88654, #69754, #77553, #56717.
The freshness window (updated_at >= live process create_time - 2s) had a
P1 boundary hole: a stale failure written by the PREVIOUS process
immediately before a fast restart landed inside the slack and was
aggregated; if that platform was then removed, the new process never
replaces the entry and NAS stays degraded indefinitely.
Replace clock heuristics with persisted writer identity:
- write_runtime_status now stamps every platform entry with the writing
process's (writer_pid, writer_start_time) — the same PID-reuse
fingerprint the liveness checks use, so a recycled PID never
masquerades as the original writer.
- The aggregation ownership filter requires exact equality between an
entry's stamp and the profile's validated live gateway process
(get_runtime_status_running_pid + _get_process_start_time). No slack,
no timestamps. Legacy entries without a stamp fail closed.
- Writer stamps are process recon (same class as the auth-gated
gateway_pid) and are stripped from all /api/status projections, both
active-profile and merged cross-profile entries.
Near-boundary regression test: prior-process entry stamped 100ms before
restart is excluded; recycled-pid-different-fingerprint excluded;
legacy no-stamp excluded; current-process entry kept.
Scoped credential locks (Telegram bot token, Discord bot token, etc.) are
machine-global, but the conflict error only reported the holder's PID:
Telegram bot token already in use (PID 559). Stop the other gateway first.
On multi-profile hosts (e.g. hosted instances running 13 profiles), a bare
PID gives the operator no way to tell WHICH profile owns the credential —
the exact failure mode observed on zerocool-9781, where the 'default'
profile was misconfigured with the same bot token as 'lead-gen-outreach'
and logged an unattributable conflict every ~5 minutes (4,602 rows).
Fix:
- acquire_scoped_lock() now stamps a 'profile' label on lock records,
inferred from the process HERMES_HOME (<root>/profiles/<name> layouts,
'default' for the root home). Omitted when not inferable.
- New scoped_lock_owner_label() resolves the owning profile from a lock
record: prefers the explicit field, falls back to inferring from the
persisted hermes_home for locks written before the field existed.
Labels are validated against the profile-id grammar before use (lock
files are plain JSON on disk and the label flows into log lines and a
suggested CLI command).
- _acquire_platform_lock() conflict message now names the owning profile
and gives the correct remedy:
Telegram bot token already in use by the 'lead-gen-outreach' profile
gateway (PID 559). Stop that gateway first
(hermes --profile lead-gen-outreach gateway stop).
Records with no attribution signal keep the original PID-only wording.
Testing:
- New TestScopedLockOwnerLabel suite covering label inference (named,
Docker, root/default, unknown layouts), grammar validation, explicit-
field preference, hermes_home fallback, and legacy/malformed records.
- acquire_scoped_lock tests for profile stamping and omission.
- Adapter-level tests for profile-attributed, legacy-home-inferred, and
PID-only conflict messages.
- 76/76 targeted gateway tests pass; broad gateway suite failures are
baseline-identical (verified via git stash comparison). Ruff clean.
Fleet triage after the 2026-08-11 storm resolution found agents whose sole
platform had been silently 'retrying' for weeks: revoked Telegram tokens,
Discord privileged-intent rejections, and Photon sidecars that can never
start were all funnelled into the indefinite reconnect queue with no owner
signal (OOF-151/152/153, epic OOF-156).
Two-part fix:
1. Per-adapter classification — by exception TYPE only, never message text:
- telegram: InvalidToken/Forbidden -> telegram_auth_error, retryable=False
(new _looks_like_auth_error, mirrors _looks_like_network_error)
- discord: LoginFailure -> discord_auth_error, PrivilegedIntentsRequired
-> discord_intents_required (both retryable=False); every other path now
sets an explicit code (previously the generic branch set NO fatal info,
which the gateway read as 'probably transient')
- photon: new typed PhotonSidecarStartupError; deps-install failure ->
SIDECAR_DEPS_MISSING and missing node binary -> SIDECAR_NODE_MISSING
(retryable=False); ambiguous startup crashes stay retryable
- email: IMAP/SMTP failures now always set a fatal code;
SMTPAuthenticationError -> email_auth_error, retryable=False (IMAP4.error
is type-ambiguous between bad creds and transient NOs, so IMAP stays
retryable)
2. Gateway escalation — platforms continuously in the reconnect queue past
HERMES_RECONNECT_ATTENTION_AFTER_SECONDS (default 2h, 0 disables) get
needs_attention=true + retrying_since stamped into runtime status, once
per episode, cleared on successful reconnect.
Deliberately NOT a circuit breaker: retries never stop. The auto-pause
mechanism was removed for good reason (transient DNS outages left bots
silently dead); this preserves that and only adds visibility. No new
platform_state enum values — NAS's status schema is strict — only additive
fields.
Unknown exception types always stay retryable: a false terminal recreates
the silently-dead-bot problem, and the escalation path covers
misclassified permanent failures.
After Discord reconnect, on-disk start_time can be null while the live
record has a fingerprint. Requiring equality made the gateway treat its
own PID as a foreign token holder (#81468).
Salvaged from #57016 by @lEWFkRAD:
- cli.py: handle file:///C:/... drive-letter URIs on nt (strip the
leading slash urlparse leaves); join Termux example paths with literal
forward slashes so hints stay POSIX on Windows.
- gateway/status.py + hermes_cli/gateway.py: normalize backslashes to
forward slashes before the HERMES_HOME substring match so separator
style cannot defeat profile ownership detection.
- hermes_cli/banner.py: cprint degrades to plain print when
prompt_toolkit has no console (NoConsoleScreenBufferError on
redirected/absent Windows stdout).
- hermes_cli/browser_connect.py: posixpath.join for WSL /mnt/c/... bases
(os.path.join would emit backslashes on nt).
- Test hardening: symlink skip-guards, USERPROFILE alongside HOME for
ntpath.expanduser, SIGKILL absence skipif fixed via monkeypatch,
drive-letter URI / separator-normalization / banner-fallback coverage.
Dropped from the original PR: tests/cli/conftest.py fixture and the
AppSession _output monkeypatch — main's merged tests/cli/conftest.py
already handles that prompt_toolkit pollution.
The sidebar strip and the Channels page could contradict each other on
the same page load — "Gateway running" next to "The gateway is not
running." /api/status and /api/messaging/platforms each open-coded their
own liveness ladder: status probed GATEWAY_HEALTH_URL and scoped its
PID/state reads to the requested profile, messaging did neither and used
the uncached raw PID probe.
Three deployments hit the split: a cross-container gateway (no local PID,
only the health probe can see it), a profile-scoped dashboard (messaging
borrowed a DIFFERENT profile's runtime state, reporting a false
"connected" that hides a real outage — #71211), and a launch-service
managed gateway with no PID file.
Adds resolve_gateway_liveness() in gateway/status.py as the single ladder
(cached PID -> HTTP health probe -> runtime-status PID with
expected_home) and routes both endpoints, /api/messaging/platforms/{id}/test,
and the kanban dispatcher-presence probe through it. Probe callables are
injectable so the existing monkeypatch seams keep working, and
GatewayLiveness.probe_error distinguishes "down" from "couldn't tell" so
the kanban warning keeps failing OPEN instead of crying wolf.
Closes#71211.
Salvages the event-spine foundation from feat/telemetry-observability
(emitter, typed events, OTLP streaming, redaction — authorship preserved in
the preceding commits) and scopes it to the plane enterprise operators need
today: gateway Service Health Monitoring plus redacted Operational
Diagnostics, exported over OTLP.
Dropped from the salvaged branch, deliberately:
- run/model/tool trajectory capture (plugins/telemetry hooks, tel_spans)
- the local JSONL + state.db tel_* store (monitoring is egress, not storage)
- usage rollups/metrics, /insights integration, bulk export
- hermes telemetry CLI (replaced by hermes monitoring status)
Those planes — shared client usage metrics and enterprise trace telemetry —
are being designed on the NeMo Relay integration with distinct consent,
policy, and export boundaries; this keeps the monitoring plane content-free
and independently enableable.
Renames agent/telemetry -> agent/monitoring, config telemetry.* ->
monitoring.*, and pins the otlp extra at OpenTelemetry 1.39.1 (matching
uv.lock; 1.30.0 conflicts with mistralai>=2.4 on opentelemetry-api).
On Windows with Chinese locale (GBK), subprocess.run(text=True) without
explicit encoding causes UnicodeDecodeError crashes. This fix adds
encoding='utf-8', errors='replace' to all subprocess.run() and
subprocess.Popen() calls that use text=True across 76 non-test Python files.
Fixes#53428 (master tracker for Windows GBK locale crash).
Note: credential_pool.py and electron changes excluded per reviewer request —
those will be submitted as separate focused PRs.
The /api/status gateway_updated_at field and the gateway /health/detailed
updated_at field passed through whatever gateway_state.json contained,
untyped. All current writers emit RFC3339 via _utc_now_iso(), but legacy
gateways wrote unix epoch floats, and a corrupt or hand-edited state file
can inject numbers or arbitrary garbage — while the frontend types
(web/src/lib/api.ts) declare string | null.
Add normalize_updated_at() in gateway/status.py as the single funnel:
- str: accepted iff datetime.fromisoformat parses (trailing Z tolerated);
naive timestamps coerced to UTC; canonical isoformat returned
- int/float: treated as unix epoch seconds -> UTC ISO string, with a
plausibility guard (reject < 2000-01-01, > now+1day, non-finite)
- bool: rejected explicitly (int subclass, but never a timestamp)
- anything else: None
Apply it at both emit sites: the dashboard /api/status handler (covers
both the local read_runtime_status() branch and the remote
/health/detailed cross-container fallback branch) and the gateway API
server's /health/detailed response.
Contract tests: parametrized /api/status normalization (epoch float/int,
garbage string, None, bool, dict, absent key), remote-health numeric and
garbage bodies, dashboard shape test round-trip assertion, direct
normalize_updated_at units (range guards, Z suffix, naive coercion,
non-finite floats), and a write_runtime_status -> read_runtime_status
round-trip proving the writer side stays tz-aware parseable.
Builds on jbbottoms's #65178 takeover fix (cherry-picked as the previous
commit). Windows --replace already tree-kills via taskkill /T, but the
POSIX paths signalled only the recorded gateway PID — adapter
subprocesses that outlived their parent kept holding scoped token locks
and blocked the replacement gateway.
- gateway/status.py: _snapshot_gateway_children() captures the old
gateway's descendants (psutil, recursive) while it is still alive;
reap_gateway_children() SIGTERMs verified orphans after the main PID
is confirmed dead, waits bounded, SIGKILLs survivors. Identity-aware
(psutil is_running is PID+create-time), skips zombies and children
whose ppid still equals the old gateway (parent actually alive), and
never raises — best-effort with debug/info logging only.
- take_over_scoped_lock_holder() snapshots before terminating and reaps
only on a confirmed successful handoff.
- gateway/run.py: start_gateway --replace snapshots before SIGTERM and
reaps after the old PID is confirmed gone, mirroring taskkill /T.
- tests/gateway/test_replace_child_reap.py: reap/skip/never-raise unit
coverage plus end-to-end --replace ordering (snapshot → terminate →
reap) and the no---replace path never touching the old process.
When --replace misses a cross-HERMES_HOME Telegram token holder, platform
connect used to retry forever. Terminate a verified gateway holder once
(with the takeover marker) and re-acquire the scoped lock (#65176).
Co-authored-by: Cursor <cursoragent@cursor.com>
Verified: applies cleanly and the patched module compiles. Tests are
described in the PR body (not bundled in this commit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the unlink()+O_EXCL sequence in acquire_scoped_lock with an
atomic os.replace() of the stale lock to a <lock>.stale tombstone
followed by the existing O_EXCL create. With plain unlink(), two racing
starters could both judge the lock stale and the second unlink() would
silently delete the first racer's freshly-created lock — both would then
'win'. os.replace() guarantees exactly one racer claims the stale file;
the loser gets FileNotFoundError and falls through to O_EXCL, which
admits at most one winner. Tombstones are cleaned up immediately;
behavior is otherwise identical.
Widen the PermissionError handling from is_gateway_runtime_lock_active
(#42689) to the sibling open() in acquire_gateway_runtime_lock: a stale
root-owned gateway.lock left by a launchd Background session previously
crashed the acquiring process. Unlink the stale file and retry once; if
the unlink or retry fails, return False cleanly instead of raising.
When the macOS launchd service runs in a Background session, the gateway
process spawns as root and creates a root-owned gateway.lock. On restart
as the normal user, open() on that file raises PermissionError, crashing
the gateway immediately and entering a launchd crash loop.
Catch PermissionError in is_gateway_runtime_lock_active(), remove the
stale lock file, and return False so the new process can start cleanly.
Fixes#42685
On macOS, the lock record's start_time is None (no /proc at creation),
but psutil.Process(recycled_pid).create_time() returns a valid float
for the unrelated process that now owns the PID. The old condition
required both sides to be None before falling back to cmdline checking,
so the recycled PID was never detected as stale.
Change the fallback condition from AND to OR: when either side's
start_time is missing, fall back to cmdline-based gateway detection.
Fixes#53763
Re-applied against v0.18.2. Still unconditional KeepAlive <true/> with no
ThrottleInterval, and upstream's new restart_loop_guard only skips session
auto-resume (never throttles the boot), so the launchd respawn storm is
unguarded.
- launchd plist: add ThrottleInterval=30 + ExitTimeOut=25.
- status: record_start_and_check_storm — portable breaker (atomic gateway-starts.log,
backs off when too many starts land in a window); run_gateway sleeps it before
asyncio.run. Env HERMES_GATEWAY_MAX_STARTS (<=0 disables) / _START_WINDOW_S.
Separate file from upstream's restart_loop.json — no collision.
- tests: breaker (threshold/prune/atomic) + plist throttle keys.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(gateway): use process-level HERMES_HOME for identity files
Gateway identity files (PID, lock, runtime status, takeover/stop markers)
were written via get_hermes_home() which honours the _HERMES_HOME_OVERRIDE
contextvar used for per-session profile dispatch. When a profile-context
task happened to be active at write time, files landed in the wrong profile
directory.
Add _get_process_hermes_home() that skips the contextvar and uses only the
HERMES_HOME env var or platform default, and route all gateway identity file
paths through it.
Fixes#56986
* chore(release): map liuhao1024 author email for PR #56993 salvage
---------
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
Co-authored-by: Ben <ben@nousresearch.com>
The Windows desktop GUI runs its backend headless via pythonw.exe. Several
auxiliary subprocess sites that run inside that windowless backend spawned
console-subsystem children (git, gh, wmic, powershell, bash, rg, taskkill)
WITHOUT CREATE_NO_WINDOW, so Windows allocated a fresh conhost per call and
flashed a black window on screen — sometimes continuously (the dashboard
Projects-tree git probe alone fired ~118 spawns in 60s on startup).
The terminal tool, cron, browser, code_execution, and gateway-spawn paths
already carry windows_hide_flags(); these auxiliary probe/scan/launcher legs
were missed. Wire the existing helper into them:
- tui_gateway/git_probe.py: run_git (+ encoding=utf-8/errors=replace, fixes the
cp950 UnicodeDecodeError on CJK paths from the same site)
- agent/coding_context.py: _git (per-turn git status/log/diff)
- agent/context_references.py: _run_git + _rg_files (@file/@ref resolution)
- hermes_cli/copilot_auth.py: gh auth token probe (auxiliary provider:auto)
- hermes_cli/gateway.py: wmic + PowerShell Get-CimInstance PID scan
- hermes_cli/main.py: wmic stale-dashboard PID scan
- gateway/status.py: taskkill /T /F force-kill
windows_hide_flags() returns 0 on POSIX, so every changed call is a no-op on
Linux/macOS (verified: real git/rg probes still work; Windows-simulated calls
all pass creationflags=CREATE_NO_WINDOW).
Scoped to the windowless-backend paths that cause the reported flashing. The
Electron updater-handoff leg (main.cjs windowsHide:false) and the
interactive-CLI banner probes (cli.py) are intentionally NOT touched here —
the former needs a Windows-tested change of its own, the latter runs in a
visible console anyway.
Tracking: #54220
Refs: #53178#53631#53781#53957#49602#52982#53424#53053#53016
The zombie status probe referenced psutil.Process/NoSuchProcess/Error
unconditionally, which raised AttributeError when psutil is a partial
stub that only defines pid_exists (as in test_windows_native_support's
fallback tests). Guard the probe so any failure to read status degrades
to the authoritative pid_exists() instead of raising.
Under systemd Restart=always, the old gateway becomes a zombie (in the
process table, awaiting reap) when the replacement starts. _pid_exists()
reported the zombie as alive, so --replace waited on a PID that never
dies, then aborted with exit 1 — a silent crash loop. Standalone runs are
unaffected because nothing respawns the gateway into a zombie.
The live path is psutil.pid_exists(), which returns True for zombies, so
the check is added there (Process.status() == STATUS_ZOMBIE -> dead). The
psutil-less POSIX fallback also reads /proc/<pid>/stat (state Z) with a ps
state= fallback for macOS/BSD, before the os.kill(pid, 0) liveness probe.
Diagnosis and the /proc + ps POSIX fallback by MorAlekss (PR #44898);
extended to cover the psutil hot path so the fix applies on normal installs.
Co-authored-by: MorAlekss <mor.aleksandr@yahoo.com>
Two profile gateway services sharing the default ~/.hermes resolve the
takeover marker to the same path. A --replace from profile B could land
in profile A's marker, match on PID + start_time by coincidence of a
shared PID namespace, and make profile A exit 0 — only to be revived by
systemd Restart=always, which races the replacer again, flapping
indefinitely.
write_takeover_marker now stamps replacer_hermes_home; the shared
consume path rejects markers written under a different HERMES_HOME and
leaves them in place for the correct profile. Absent field (older
markers) is treated as same-home, so single-profile and mixed old/new
deployments are unaffected.
Salvaged from #31414 by @CryptoByz onto current main (branch was ~3962
commits behind; the consume function had since been refactored for
issue #34597). Co-authored-by: CryptoByz.
* fix(windows): stop terminal-window popups from background spawns
Native-Windows desktop/gateway users saw cmd/conhost windows flash on
gateway restart, image paste, the dashboard Projects tree, voice notes,
and ~5 min after closing the app (detached cron). Two root causes:
- Console-subsystem exes (taskkill, schtasks, wmic, netstat, tasklist,
agent-browser, git, ffmpeg, powershell, git-bash) spawned via raw
subprocess allocate a fresh console when the launching process has
none (pythonw desktop backend / detached gateway) - even with output
captured.
- uv venv pythonw shims re-exec console python.exe, so Python children
get a console regardless of how they're launched.
Fixes:
- Single hidden-spawn primitive (_subprocess_compat.run/.popen) that ORs
CREATE_NO_WINDOW on Windows, no-op on POSIX. Route every Hermes-owned
console-exe spawn through it.
- FreeConsole() catch-all in hermes_bootstrap: any Python child that
exclusively owns an auto-allocated console detaches it at startup
(GetConsoleProcessList()==1 gate leaves shared interactive consoles
untouched).
- Replace PowerShell/wmic gateway PID scans with in-process psutil.
- Skip schtasks queries on non-interactive desktop restarts.
- Prefer native agent-browser .exe over .cmd shims.
- Guard test bans raw subprocess spawns of the Windows-only console
tools repo-wide so the popup class can't regress.
* fix(windows): scope FreeConsole to background entry points; fix merge fallout
Console detach review (per #53810 feedback): GetConsoleProcessList()==1 can't
tell a uv pythonw->python phantom console apart from a user opening the
interactive CLI/TUI in its own fresh console (double-click, shortcut, ConPTY) —
both report a single attached process with a tty. Running FreeConsole() in the
import-time bootstrap therefore risked detaching a legitimately-interactive
terminal.
- Extract FreeConsole into explicit hermes_bootstrap.detach_orphan_console();
remove it from apply_windows_utf8_bootstrap() (import side effect).
- Call it only from known background mains: gateway run, dashboard backend
(start_server, what the desktop spawns), cron standalone, tui_gateway entry,
slash worker. Interactive CLI/TUI never calls it.
- Behavior-contract tests: frees only when solo owner, leaves shared console,
no-op without console / on POSIX, and asserts it's not an import side effect.
Merge fallout from origin/main (#53791):
- local.py: 3-way merge left a dangling **_popen_kwargs (NameError crashing
every terminal init). _subprocess_compat.popen already hides the window, so
drop it.
- discord adapter: merge stacked an undefined windows_hide_flags() onto the
primitive call; drop the redundant arg.
- test_gateway: scan now goes psutil-first (zero spawn); rewrite the
case-variant test to drive that production path.
* test(claw): mock _subprocess_compat.run seam for Windows process scan
claw.py's Windows tasklist/powershell scan routes through the hidden-spawn
primitive; the tests still patched claw_mod.subprocess, so on win32 the mock
was never hit and real spawns returned nothing. Patch the actual seam.
PR #52151 hardened the runtime-status liveness check to trust a readable
live process command line over stale gateway_state.json argv, so a recycled
PID now owned by an s6 supervisor no longer counts as a running gateway.
That fix is correct but incomplete for the reported symptom: the web
dashboard showed a named profile's gateway green while
`hermes -p <name> gateway status` showed it stopped. Two further issues:
1. Cross-profile PID reuse. In per-profile Docker supervision, one profile's
stale `gateway_state.json` can record a PID the OS later recycled onto a
DIFFERENT profile's live gateway. That PID's command line still
`looks_like_gateway`, so the dead profile was reported running. The
recorded argv has its `-p <name>` selector stripped in-process by
`_apply_profile_override`, so it cannot disambiguate; the live `/proc`
cmdline still carries it. `get_runtime_status_running_pid` now accepts an
`expected_home` and validates the live command line belongs to THAT
profile (mirroring `hermes_cli.gateway._matches_current_profile`, the
logic the CLI scan path already uses — which is why the CLI was correct).
`_check_gateway_running` passes the enumerated profile dir.
2. The existing regression test `test_gateway_running_check_falls_back_to_
runtime_state` used the live pytest PID with a gateway-shaped record; once
the live cmdline became authoritative it no longer looked like a gateway.
Updated to mock the live cmdline to the real separate-process scenario it
describes.
The active-profile path (`get_running_pid`) is intentionally left unscoped:
it is lock-verified and any live gateway cmdline is acceptable there. Multiplex
mode is unaffected — `running` state is only ever written to a gateway's own
home, never a secondary served profile's.
Adds coverage for: cross-profile PID reuse (named + default), matching
profile cmdline (`-p`, `--profile`, explicit HERMES_HOME=), the bare default
gateway, and the unreadable-cmdline cross-platform fallback. Each new
cross-profile assertion fails without the profile scope and passes with it.
Co-authored-by: helix4u <4317663+helix4u@users.noreply.github.com>
The dashboard Profiles view showed "Gateway stopped" for a gateway that
is in fact running — while the sidebar status strip and `hermes gateway
status` (CLI) both correctly showed it running. Reported on v0.17.0
running the gateway + dashboard in one Docker container.
Root cause: three liveness surfaces with three detection strengths, all
reading the same `gateway.pid`:
- `hermes gateway status` -> find_gateway_pids() (process-table scan)
- sidebar /api/status -> get_running_pid() + gateway_state.json PID
fallback + health-URL probe
- Profiles view -> _check_gateway_running() = get_running_pid()
ONLY, no fallback
`get_running_pid()` short-circuits to None the moment the runtime lock
(`gateway.lock`) doesn't register as held by the *calling* process —
which is always true when the reader is a separate process from the
gateway (the dashboard is its own s6 service in the container), and also
for any launch-service-managed gateway that left a fresh
`gateway_state.json` but no live PID file. So the Profiles view alone
reported the live gateway as stopped.
Fix: give _check_gateway_running the same fallback the sidebar already
has — after the pid-file/lock check misses, validate the PID recorded in
that profile's gateway_state.json against the live process table via the
existing get_runtime_status_running_pid(). read_runtime_status() gains an
optional path arg so a profile's state file can be read without mutating
the process-global HERMES_HOME (preserving the contextvar-based profile
isolation the dashboard relies on). Backward compatible: every existing
caller passes no argument.
Tests: a regression test that fails pre-fix (live gateway, lock check
returns None -> must still report running) and a guard test that a
'stopped' state file is never reported running even with a live PID.
The PID-reuse guard (#43846) reads /proc/<pid>/stat field 22, which only
exists on Linux — on macOS/Windows it returned None and the guard silently
degraded to a bare liveness check (a no-op, safety-wise). Add a
psutil.create_time() fallback (psutil is a hard dep, cross-platform),
quantized to centiseconds for stable equality, so the recycled-PID guard
actually protects macOS/Windows too. /proc always wins first on Linux and
always misses on macOS/Windows, so the two sources never mix on one host and
same-source equality is all the guard needs.
Second cleanup pass (simplify-code review of the first follow-up):
- write_runtime_status now clamps active_agents via parse_active_agents
instead of an inline max(0, int(...)). Removes the duplicated clamp the
helper's docstring acknowledged AND closes a write-side ValueError gap
(a non-numeric active_agents previously raised; now degrades to 0).
- hermes_cli/gateway.py draining-status line routes its active-agents count
through parse_active_agents too — the third coercion site of the same
persisted field, now consistent and non-raising with the two HTTP surfaces.
- web_server.py /api/status: the drain-timeout resolver fallback now catches
ImportError specifically and falls back to DEFAULT_GATEWAY_RESTART_DRAIN_TIMEOUT
(a real float) instead of a blanket 'except Exception -> None'. None would
have violated the surfaced field's int/float contract and stripped NAS's
poll-deadline hint silently.
- Dropped a redundant 'if runtime else 0' branch (parse_active_agents already
handles the empty/None case) and tightened the parse_active_agents docstring
to describe the actual single-contract role (write + both reads).
Follow-up cleanups on top of the busy/idle readout (PR #50103):
- web_server.py /api/status reused the single drain-timeout resolver
hermes_cli.gateway._get_restart_drain_timeout() (HERMES_RESTART_DRAIN_TIMEOUT
env -> agent.restart_drain_timeout config -> default) instead of inlining a
third hand-rolled copy of that precedence chain. Also fixes a subtle
divergence: the inline copy used os.environ.get() so a set-but-empty env var
was treated as a value rather than falling through to config; the shared
resolver .strip()s and falls through correctly.
- Added gateway.status.parse_active_agents() and routed BOTH HTTP surfaces
(/api/status and /health/detailed) through it, so the exposed active_agents
field is consistently clamped non-negative. Previously /api/status clamped
while /health/detailed exposed the raw file value, diverging on a corrupt
count.
- Added TestParseActiveAgents covering the shared coercion contract.
Give an external consumer (NAS) a trustworthy, always-reachable busy/idle
readout it can poll before a disruptive lifecycle action (restart,
migrate, stop, auto-update). The dashboard /api/status is the only HTTP
surface guaranteed up on a hosted agent regardless of which gateway
platforms are enabled, and it already reads gateway_state.json.
Add to /api/status (additive, non-breaking):
- active_agents — in-flight gateway-turn count (now refreshed
per-turn by the companion gateway-side commit)
- gateway_busy — running AND active_agents > 0
- gateway_drainable — running and live (a valid begin-drain target)
- restart_drain_timeout — resolved seconds, so the consumer can size its
poll deadline without out-of-band knowledge
(env HERMES_RESTART_DRAIN_TIMEOUT → config
agent.restart_drain_timeout → default)
The busy/drainable contract is defined once in gateway.status
(derive_gateway_busy / derive_gateway_drainable) and consumed by both
/api/status and /health/detailed so the two surfaces can never disagree.
Liveness keys off gateway_running (a live PID/health probe), NEVER
gateway_updated_at — a healthy idle gateway never advances that timestamp.
All derived fields degrade to safe falsy values when the gateway is down
or the status file is absent/corrupt (never a spurious "busy" that would
wedge the consumer). active_sessions (the 5-min DB recency heuristic the
SPA reads) is left exactly as-is — new signal, new fields.
Tests (behaviour contracts, not snapshots): the pure derivation contract
across every running/state/count/liveness combination; /api/status
integration for busy, idle-drainable, draining, down, stale-busy-file,
corrupt-count, and timeout surfacing; and /health/detailed parity.
- _guard_named_profile_under_multiplexer: when the default gateway is running
with gateway.multiplex_profiles=on, a named-profile 'hermes gateway run' hard
-errors (pointing at the multiplexer) instead of double-binding that
profile's platforms. Inert unless all hold: this invocation is a named
profile, a default-profile gateway is alive, and its config has multiplexing
on. --force overrides. Wired into run_gateway's guard chain.
- write_runtime_status gains served_profiles: the secondary-adapter startup
records [active] + multiplexed profiles into runtime_status.json so
'hermes status' can show per-profile coverage without a second probe. Absent
for single-profile gateways.
Tests: served_profiles round-trips and is absent by default; guard is inert for
the default profile / under --force / when no default gateway is running.
Address correctness gaps found in pre-PR review of the strict matcher:
- Profile selectors can appear on EITHER side of the `gateway` token
(`_apply_profile_override` strips `--profile`/`-p` from anywhere in argv
before argparse), so `hermes gateway --profile work run` and
`python -m hermes_cli.main gateway -p work run` are valid launches the
previous matcher wrongly rejected. Strip `--profile`/`-p`/`--profile=`/`-p=`
from anywhere before locating the subcommand.
- A profile literally named `gateway` (`hermes -p gateway gateway run`) made
the old token scan stop on the profile value; stripping the selector+value
first fixes it.
- Tokenize quote-aware with `shlex` so quoted Windows paths containing spaces
(`"C:\Program Files\Hermes\hermes-gateway.exe"`) are no longer split mid-path
and the dedicated-entrypoint match survives.
Without these, the matcher could MISS a real running gateway -> the opposite
failure (restart/status reporting "down" when up). Adds regression tests for
all three shapes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Teknium