From 4ba038b51d2569ec3b9563b2ccdf42c489f3ba08 Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Sat, 22 Aug 2026 00:18:58 -0700 Subject: [PATCH] docs(agents-md): update pipeline architecture, process-identity pitfall, gateway lifecycle contract, wine2e lane MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures the durable invariants from the fleet-update campaign (#91277) so contributors and the sweeper review against them: - Update Pipeline section: the transactional shape now on main (plan → snapshot → apply → restart-per-kind → verify → report), the per-stage invariants (no partial snapshot tiers, ZIP only on real git failure + dirty-tree refusal + release-dir graft, fleet-wide drain-first restarts, code-sha verify, exactly-once receipts), deployment kinds as first-class, and the #92091 socket direction. - Gateway lifecycle vs Desktop app: serve dies with the app by design, the detached gateway survives it; the Windows shim-unlock tree-kill is the known breach (#85265) and its replacement is pause-for-update — with the two anti-fix warnings. - Known Pitfall: process identity is never inferred from argv substrings (canonical matchers, parser-derived flag sets, ancestor carve-out, full-cmdline rule, socket-first for new heuristics). - Testing: the on-demand wine2e live Windows lane and its reproduce-first workflow. --- AGENTS.md | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 93705624e3..b951e238fa 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1201,6 +1201,79 @@ Full user-facing docs: `website/docs/user-guide/features/kanban.md`. --- +## Update Pipeline (`hermes update`) + +The updater is transactional in shape (fleet-update campaign, #91277 — +Aug 2026). Every stage exists because its absence was a real field +failure; PRs that weaken a stage need to answer for the failure class it +guards: + +``` +plan → snapshot → apply → restart-per-kind → verify → report +``` + +- **Plan** (`hermes_cli/update_inventory.py`, `hermes update --plan`): + read-only inventory — install kind, all profiles, every live gateway + with supervisor + running code version. Deployment kinds are + first-class: `git` updates in place; `docker`/`nix`/`apt` are NOT + in-place-updatable and the updater reports the correct external + command instead of fighting the deployment model. +- **Snapshot** (`hermes_cli/backup.py`): pre-update quick snapshot for + EVERY profile (the code swap + fleet restart touch all of them), each + into its own `state-snapshots/`, identical file set + 1 GiB per-file + cap + keep=1. **Never add a partial/tiered snapshot set** — mixed + coverage creates torn-restore states across schema generations. Quick + snapshots are FILE-LOSS RECOVERY (the per-profile cron-jobs safety + net restores from them), NOT code-rollback insurance; `--backup` full + mode owns rollback. +- **Apply**: git pull, or the Windows ZIP fallback — which fires ONLY + when git itself failed (`_should_zip_fallback_on_update_error`, + argv-classified; a dependency-install failure must never trigger a + tree-clobbering re-download), REFUSES a dirty working tree + (`-uall`, plus a pre-swap TOCTOU re-check), and grafts the live + `apps/desktop/release/` into the staged swap (the GitHub source ZIP + has no built desktop app; without the graft the swap deletes it). +- **Restart-per-kind**: systemd and launchd restarts are FLEET-WIDE + (every `hermes-gateway*` unit / `ai.hermes.gateway*` LaunchAgent), + drain-first (SIGUSR1) with per-unit/per-label failure isolation. + Restarting only the invoking profile's service leaves siblings on + stale `sys.modules` until they crash — the largest dupe-PR cluster in + the repo's history came from that bug. +- **Verify**: gateways stamp their running `code_sha`/`code_version` + into `gateway_state.json` on every runtime-status write + (`gateway/status.py`); after the restart phase the updater compares + each live gateway against the fresh checkout and prints a fleet + version matrix. A provably-stale gateway fails the update (exit 1) — + automation must never treat a mixed-version fleet as healthy. +- **Report**: every run writes a machine-readable receipt to + `~/.hermes/logs/update_receipts/` (`latest.json` pointer; steps, + skips WITH reasons, restart outcome, plan, fleet snapshot). + Finalization is owned by the `cmd_update` command boundary — early + `sys.exit` paths (preflight refusals, fetch failures) still persist + a receipt with the real exit code. A begun-but-unwritten receipt is + a bug: the refused/failed runs are the ones receipts exist for. + +Architecture direction: process-scan-based coordination between the +updater, serve/dashboard, and the gateway is being replaced by a +gateway-owned control socket (#92091). Do not add new scan heuristics +without checking that design; scans are the fallback layer. + +### Gateway lifecycle vs. the Desktop app + +`hermes serve` (control plane, desktop-spawned child) dies with the app +— by design. The messaging gateway (`gateway run`) SURVIVES the app: the +serve backend's `/api/gateway/*` endpoints spawn it detached +(`_spawn_hermes_action` — `start_new_session` / `DETACHED_PROCESS`), so +`before-quit`'s backend SIGTERM never reaches it. Bots keep running +when the user closes the app. The known breach of this contract is the +Windows shim-unlock teardown (`taskkill /T /F` on venv-shim holders, +#85265) — it exists to let updates proceed, and its replacement is +#92091's `pause-for-update`. Do not "fix" gateway-dies-with-app reports +by re-parenting the gateway under the backend, and do not "fix" update +locks by widening the tree-kill. + +--- + ## Important Policies ### Prompt Caching Must Not Break @@ -1314,6 +1387,27 @@ automatically scope to the active profile. ## Known Pitfalls +### DO NOT infer process identity from argv substrings +The bug class behind ~10 fleet-update issues (#90778, #87594, #78089, +#76129, #91964, ...): classifying a process by `"serve" in cmdline` or +similar. `kanban --preserve-cache` contains "serve"; a flag VALUE can +equal a subcommand (`-m dashboard serve`); truncated cmdlines hide the +real subcommand. Rules: +- Use the canonical matchers: `gateway.status.looks_like_gateway_command_line` + (gateway run), `hermes_cli.update_cmd._hermes_holder_subcommand` + (top-level subcommand of any Hermes argv). Never hand-roll token scans. +- Flag sets must be DERIVED from the parser + (`_holder_value_flags()` introspects `build_top_level_parser()`), never + hand-written lists — they drift. +- Never blanket-exclude ancestors from process scans: when `/update` runs + as the gateway's child, a gateway ancestor must stay visible to the + pause machinery (#87594). Exclude interactive ancestry, carve out + gateway-shaped ancestors. +- Match on FULL cmdlines; truncate only at display time (#78089). +- Before adding any new scan heuristic, read #92091 — the gateway control + socket replaces scans as the primary coordination mechanism; scans are + the fallback layer for old/crashed processes. + ### DO NOT hardcode `~/.hermes` paths Use `get_hermes_home()` from `hermes_constants` for code paths. Use `display_hermes_home()` for user-facing print/log messages. Hardcoding `~/.hermes` breaks profiles — each profile @@ -1491,6 +1585,22 @@ in order to pass, it belongs on that OS.** When one test body walks several platforms in sequence, split it. Keep the host-native arm on the Linux lane and move the other arm into its own marked test. +**Live Windows process-topology E2E: the `wine2e` lane.** For claims about +real Windows process behavior that mocks cannot reproduce (venv-holder +scans, process-tree parentage, launcher/worker chains, detach semantics), +there is an on-demand workflow `windows-venv-e2e.yml` that runs +`tests/hermes_cli/test_venv_holder_windows_live.py` on a real +`windows-latest` runner — spawning actual processes and driving the real +detection code, no mocked psutil. It fires ONLY on pushes to `wine2e/**` +branches (inert on PRs and main; costs nothing on normal work). The proven +workflow: write probes that pin CORRECT behavior, push to a `wine2e/` +branch to reproduce the bugs live on unfixed code, build the fix, iterate +until the lane is green, then open the PR — the live receipt on the exact +head is the Windows proof reviewers ask for. Extend the live suite when +touching that subsystem; assert against the gateway ANCESTOR found by +argv, not the direct parent (the venv shim makes every spawn a +launcher/worker chain). + **Use the marker, never a bare `skipif`.** `scripts/ci/list_os_marked_tests.py` decides which files the macOS/Windows lanes import by grepping for the marker *name*, and the lane then filters with `-m `. A test gated with