Reviewer egilewski found the original defer was circular (#83590 comment):
the self-lock preflight wrote .update-incomplete and exited, but the next
launch only ran the full recovery AFTER main.py's third-party imports —
so a healthy venv's probes made the early pass a no-op, main.py imported
cryptography eagerly, the .pyd got mapped again, and the deferred install
re-hit the exact self-lock it was meant to escape.
Close the loop by making the marker guarantee the install runs BEFORE any
native extension can be imported:
- hermes_cli/_install_repair.py (new, stdlib-only): single source of truth
for the core .[all] reinstall — ensurepip bootstrap, uv-pip/pip
resolution with VIRTUAL_ENV, Termux env stripping, Windows hermes*.exe
quarantine, per-extra fallback ladder, and fd1→fd2 routing for acp
safety. Deliberately free of managed_uv/hermes_constants imports so it
stays importable in the corrupted-venv state it exists to repair.
- hermes_cli/_early_recovery.py: recover_if_needed now completes a pending
.update-incomplete install BEFORE the import probes, on every launch
that sees the marker (unless argv is update). Success clears the
marker; failure bumps an attempts counter inside the marker body and
keeps it. A 3-attempt ceiling stops a persistently-failing install
from reinstall-hammering every launch (hermes acp included) — past the
ceiling the late post-import recovery takes over with its manual
recovery instructions. Single-flight lock shared with the late path.
- hermes_cli/main.py: _recover_core_update_marker_locked delegates the
install to the shared executor (no duplicated logic); ensure_uv stays
in the late path so a venv whose uv vanished mid-update still
bootstraps it.
- tests: 7 new regressions — the reviewer's exact case (marker + healthy
venv → install runs while sys.modules has no cryptography), failure
keeps marker + increments attempts, retry ceiling, lazy marker does
not trigger core install (#58004 invariant), argv-update skip, and
corrupt/missing marker bodies. The key test was sabotage-verified:
removing the pre-import branch makes it fail with zero install calls,
while a lone-lazy-marker test still passes; restoring the branch makes
it pass again.
Refs #83569
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
Keep .update-incomplete for full .[all] recovery only. Lazy refresh uses
.lazy-refresh-incomplete and clears only after confirmed import probes;
unavailable probes are indeterminate, not healthy (#58004 review).
Keep .update-incomplete across normal hermes.exe launches, heal via
package-only import probes first, and only clear the marker after repair
succeeds (#57828 / #58004 review).
Add two tests for the self-lock guard in _recover_from_interrupted_install:
one asserting it clears the marker and skips install when hermes.exe is a
process ancestor (breaking the #52378/#45542 loop), one asserting it falls
through to a normal recovery install when the shim is NOT an ancestor.
The guard's manual-recovery hint runs only inside the Windows branch, so
quote it for cmd.exe (cd /d, double-quoted paths) — the cross-platform
fallback hint at the end of the function is left POSIX-correct.
Map Icather in scripts/release.py AUTHOR_MAP for the salvage.
* fix(update): self-heal a venv left half-built by an interrupted install
An update killed mid dependency-install (Ctrl-C, terminal close, WSL OOM)
could leave the venv with pip wiped and core deps (e.g. Pillow) missing,
with no automatic recovery — the user had to manually run ensurepip +
reinstall.
Drop an install-scoped .update-incomplete breadcrumb right before the dep
install and clear it only after core-dependency verification passes. On the
next launch (any command except 'update' itself), if the marker is present,
unconditionally bootstrap pip via ensurepip then re-run the .[all] install +
verification, then clear the marker. Failure leaves the marker for retry and
prints the manual recovery command. Never raises — recovery cannot block
launch.
* fix(update): address review — stderr-only recovery output, single-flight lock, gitignore marker
- Route all recovery output (status lines + streamed pip/uv install via
fd-level dup2) to stderr so protocol-on-stdout launches (hermes acp)
never get install noise on the JSON-RPC stream.
- Single-flight O_EXCL lockfile (.update-incomplete.lock) so a gateway
start + CLI launch (or two profiles) can't run concurrent installs
into the shared venv; stale locks (>1h) are broken for the next launch.
- gitignore .update-incomplete + lock so source-tree installs keep a
clean git status and update's autostash skips them.
- Document why the loose 'update' argv substring match is intentional
(over-match defers one launch; under-match would race the real update).
- 4 new tests: lock held → skip, stale lock broken, lock released,
output lands on stderr only.
Halldrix