The #92434 mid-handshake pin (added on main after #95343 branched) latches
gatewayMocks.connect on a never-resolving mockImplementation; vi.clearAllMocks()
clears calls, not implementations, so the salvaged wedged-probe test inherited
a dial that never completes and timed out.
20s withTimeout() on the boot() and soft-switch paths (use-gateway-boot.ts),
since these are IPC round-trips into the main process with no timeout of
their own — a wedged main-process round-trip hangs the awaiting caller
forever instead of surfacing a failure.
Every other production call site of the same IPC pair was still unbounded:
- store/gateway.ts's openSecondary() and sharedPrimaryRoute() — the actual
connection-establishment underneath requestGatewayForProfile/Agent,
ensureGatewayForProfile/Agent, and every other exported routing entry
point that opens a non-primary profile's socket.
- use-gateway-request.ts's on-demand reconnect (the primary gateway's
"not connected" retry path hit by every RPC).
- voice-playback.ts's resolveSpeakStreamUrl().
- api/plugins.ts's activeConnection() (pluginSocket's connect()).
Extracted RECONNECT_ATTEMPT_TIMEOUT_MS into the shared lib/with-timeout.ts
(previously local to use-gateway-boot.ts) so every call site uses the same
budget instead of duplicating the constant.
Regression tests mirror the existing use-gateway-boot.test.tsx hang-repro
pattern: wedge getConnection()/getConnectionFor() with a never-resolving
promise, advance fake timers past the 20s bound, assert the caller settles
instead of hanging. Mutation-verified: reverted the production fix (kept
tests) and confirmed the 6 new tests fail — 4 by genuinely timing out at the
vitest level, 2 by TypeError on the not-yet-exported activeConnection —
restored the fix and confirmed all 70 tests across the gateway/voice/boot/
plugins suites pass, with tsc -p . --noEmit clean throughout.
Sibling site of the same class fixed in the previous commit: a failed
profile-store open during _init_session fell back to _get_db(), hydrating
and persisting a named-profile session's cwd row against the launch
state.db. Fail closed instead — skip the hydration (log a warning) so
nothing ever reads or writes the wrong profile's store. The other
profile-store open sites (_db_for_profile, _ensure_session_db_row,
_session_db) already degrade to None/skip and were left as-is.
A deferred agent build for a named-profile session swallowed a failed
profile-store open (except Exception -> session_db = None), so _make_agent
silently bound the launch _get_db() handle and every turn bled into the
wrong profile's state.db exactly when the profile store was briefly
unopenable. Opening the named profile then looked blank.
Route the open through _open_profile_session_db, which raises a clear
'profile session store unavailable' error instead; the deferred build's
existing except path turns that into agent_error + an error event, so the
user gets a clear failure and no agent turn against the wrong store.
Salvaged from #90219 (hardening half), adapted to main's current
deferred-build/_transfer_db_to_agent structure.
Related to #87723 and #89789. #88532 covered SessionStore only.
Sixth review - the first against the interval architecture - verdict:
the architecture holds (idempotence, order-independence, 4-process
concurrent-writer safety, rollback immunity, format consistency, and a
120-permutation order sweep all verified), with ONE high finding, which
I had independently reproduced while the review ran: the FORWARD clock
adversary was unhandled, and unlike every other failure mode in this
subsystem it failed OPEN.
The 'obs' mark is a MAX-upsert - monotonic in the leak direction. One
glitched-forward sample (NTP flap reading 2099) while consented dragged
last_confirmed_at to 2099; a later revoke stamped closed_at = 2099; the
closed window then CONTAINED every refused period that followed. Both
the reviewer and I reproduced refused packages becoming gate-eligible.
The rollback twin was mutation-tested since round 5; nobody had asked
whether the mirror image existed.
Two clamps, each covering what the other cannot:
- The obs mark advances at most MAX_OBS_ADVANCE_SECONDS (30 days) per
call. Honest heartbeats never bind it; a machine off for months
catches up in a few hook fires (fail-closed latency only); one insane
sample moves the horizon by a bounded step that real time overtakes.
- A close is MIN(last_confirmed_at, closing observation's raw stamp).
Confirmed-time keeps unobserved gaps out of windows (v1's leak); the
raw stamp lets an honest clock at revoke time pull a poisoned horizon
back to the true revoke moment. A rolled-back clock at close time
only closes earlier - fail-closed.
Also from the review:
- D2: the data-mark advance in the REAL package writer had no coverage
(the harness re-implemented the insert; deleting the production line
survived 314 tests). Now driven through create_and_export_package_if_due.
- D3: the "don't create ~/.hermes/telemetry for fully-disabled users"
skip was dead code - the store constructor creates the directory
before the exists() check ran. The probe now checks the default path
without constructing; verified empirically on a fresh HERMES_HOME.
- Upgrade note in A.4: pre-interval backlog is never transmitted after
upgrade (fail-closed; deliberate).
New harness scenarios: forward-poison-then-revoke (the leak), and
forward-poison-cannot-wedge (the cap). Mutation check: unclamping the
close, removing the cap, and removing the real writer's data-mark
advance each fail the suite.
273 tests pass; ruff and windows-footguns clean; staging E2E 202.
Proved on windows-latest that a locked profile blocks (no kill/hang), the
approved close terminates Chrome + releases the lock, and snapshot then copies a
valid DB — and autoclose-off blocks with quit guidance. Per policy proof
workflows never land on main. Product + portable unit tests remain.
Refines the Windows path per three requirements:
1. Only when the toggle is set — closing is offered only if
browser.real_profile_autoclose is on.
2. Blocked when locked — snapshot_real_profile NEVER kills; a locked profile
always returns the [profile-locked] signal and the copy is refused. A later
attempt that is still locked blocks again (no loop, no auto-kill).
3. Ask approval to close — closing is an explicit, user-approved step:
(new CLI subcommand) runs
close_browser_holding_profile only when the agent has the user's OK. The
locked error tells the agent to ask first, then run it, then retry.
- browser_connect: snapshot blocks with _PROFILE_LOCKED_PREFIX (autoclose-armed
message offers the close; off message says fully-quit); no in-snapshot kill.
- main.py: subcommand (identity+binding-verified
tree kill via close_browser_holding_profile); added to _BUILTIN_SUBCOMMANDS.
- browser_tool: surfaces the locked signal + the exact approved-close command.
- Docs/config: toggle arms + agent asks + blocked-if-still-locked.
Tests: snapshot blocks-not-kills with autoclose on AND off; process matcher
identity/binding. 73 real-profile tests pass. Windows live E2E (proof): locked
blocks fast without killing → approved close terminates Chrome → snapshot then
copies a valid DB; autoclose-off blocks with quit guidance.
Branch-only evidence — proved on windows-latest that consented auto-close
terminates a running Chrome, releases the lock, and produces a valid profile
copy (and that autoclose-off fails fast, not hangs). Per policy proof workflows
never land on main. Product fix + portable unit tests remain in
hermes_cli/browser_connect.py and tests/tools/test_browser_real_profile.py.
Auto-close live test asserted the legacy Default/Cookies path, but modern Chrome
writes Default/Network/Cookies. Accept either; on miss, print the copy's Default
listing so a real copy gap (vs a path-assertion bug) is visible.
Live Windows CI proved copy-while-running is impossible (Chrome opens the cookie
DB deny-all). So to make Windows actually WORK — not just fail cleanly — add
opt-in auto-close: browser.real_profile_autoclose (default false). When the
profile is locked and consent is on, snapshot_real_profile terminates the
browser process tree bound to THAT user-data-dir (psutil, identity+binding
verified like the daemon reaper — browser binary AND this exact --user-data-dir
in cmdline, fail-closed on ambiguity), waits for the lock to release, then
snapshots. Destructive (loses unsaved tabs) so it's off by default and the agent
asks first; the fail-fast message names the option. No effect on POSIX.
- close_browser_holding_profile: graceful terminate → kill → poll until the
cookie DB is openable again (bounded); reports relaunch/tray failure clearly.
- _processes_holding_profile: identity+binding matcher (never kills an
unrelated same-name process on a different dir).
- Config key + docs admonition.
Tests: autoclose closes-then-snapshots, autoclose-failure-reports, fail-fast
names the option, process-matcher identity/binding. 74 real-profile tests pass.
Windows live E2E (PROOF workflow, reverted before merge): autoclose-off fails
fast <30s; autoclose-on terminates real Chrome, lock releases, valid cookie DB
copied.
The windows-latest proof E2E and its live/diagnostic tests were branch-only
evidence (they proved the deny-all lock + fast-fail contract on a real runner).
Per policy proof workflows never land on main. The product fix (fast lock
probe + fail-fast message) and its portable unit tests remain in
tests/tools/test_browser_real_profile.py.
Live windows-latest proof settled it: a running Chrome opens its cookie DB
deny-all (even CreateFile with FILE_SHARE_READ|WRITE|DELETE fails; sqlite
mode=ro/immutable/nolock all 'unable to open'), so copy-while-running is
impossible on Windows without VSS/admin — and the prior code HUNG ~24min on the
locked file.
Fix: a fast up-front lock probe (_profile_is_locked: one open() of the active
profile's cookie DB; PermissionError = locked) runs BEFORE any copy in
snapshot_real_profile. If locked, bail immediately with 'fully quit the browser
(incl. background/tray) and retry, or turn browser.use_real_profile off'. Never
hangs, never a silent signed-out copy. POSIX has no mandatory locking so the
probe never trips there — copy-while-running still works on macOS/Linux.
Docs: admonition stating Windows needs the browser fully closed (background
apps included); the live-drive-while-running path is #95669.
Tests: lock-probe unit coverage (readable/no-db/PermissionError), snapshot
fails-fast-no-copytree when locked. Windows live E2E asserts the fast-fail
contract (returns <30s with the quit message) + the read-strategy diagnostic.
Prior run hung 24min in the product-path test (snapshot_real_profile against a
locked profile blocks on Windows — itself a finding). Diagnostic now runs FIRST
(each strategy internally bounded, reports fast), live test second under a
faulthandler 150s dump-and-die so a hang can't burn the job. Job timeout 12min.
Previous runs were auto-cancelling each other (ref-scoped group + cancel-in-progress). Per-sha group lets each proof run finish so the diagnostic actually reports.
Adds a Windows-live diagnostic that, against a cookie DB held by a running
Chrome, reports which read strategy succeeds: shutil, open-rb, sqlite mode=ro,
sqlite immutable=1, sqlite ro+nolock, raw win32 CreateFile with full share
flags. This tells us empirically whether any in-process read path exists
(immutable=1 / share-all open) before reaching for VSS/admin. Fails-closed test
marked xfail while the real behavior is derived from the diagnostic.
The first live Windows run DISPROVED the sqlite-online-backup claim: Chrome's
share lock on Windows is strong enough that even a read-only SQLite open is
refused by the OS (raw-copy precondition fired, _copy_auth_file still returned
False). Copy-while-Chrome-runs is impossible on Windows — the earlier fix was
theatre that only passed on Linux (no mandatory locking).
Corrected contract, now asserted live: with a running Chrome holding the cookie
DB, snapshot_real_profile FAILS CLOSED with 'could not read ... login data
(N locked). Close <browser> and retry' — never a silent signed-out/torn copy.
Second test proves the supported path (Chrome closed) copies cleanly. So
real-profile browsing on Windows requires the browser closed; Linux/macOS
unaffected; live-drive-the-real-profile is tracked in #95669.
PROOF branch evidence only — workflow + test reverted before merge.
One-shot windows-latest E2E: launches real Chrome on a user-data-dir so it holds
the cookie DB with a Windows share lock, asserts a RAW copy fails (WinError 32
precondition — else skip, no vacuous green), then asserts _copy_auth_file copies
it via SQLite online-backup and the result is a readable Cookies DB with the
cookies table.
This proves the Windows 'file in use' fix on a real runner — the coverage the
Linux lanes cannot provide. PROOF branch evidence only: this workflow + test are
reverted before merge and must never land on main.
On Windows a running Chrome holds Cookies / Login Data / Web Data with an
exclusive lock, so the raw file copy the snapshot used raised WinError 32
('being used by another process') and the best-effort skip left a signed-out
copy — the reported profile-cloning failure.
Fix: copy the SQLite auth DBs via SQLite's online-backup API (read-only
connection + Connection.backup()), which reads a consistent COMMITTED snapshot
while the writer holds the lock. Non-DB files (Preferences, Local State) stay a
plain copy. If even the online-backup can't read a DB, snapshot_real_profile now
FAILS CLOSED with an actionable 'close <browser> and retry' message instead of
launching a silently signed-out session.
- _copy_auth_file: sqlite-backup for Cookies/Login Data/Web Data, raw copy
otherwise, raw-copy fallback if backup fails.
- Drop -journal/-wal/-shm sidecars from the auth set + snapshot ignore: the
backed-up DB is self-contained; a stale sidecar next to it corrupts it.
- Fresh copytree excludes the auth DBs (raw copytree of a locked file raises on
Windows); they're always mirrored lock-aware afterward.
- _mirror_profile_auth returns the count of DBs it could not copy so the caller
can fail closed.
Tests: locked-DB copied-via-backup (open write txn = live-lock analog, 42
committed rows, uncommitted excluded, no journal sidecar), _copy_auth_file DB vs
plain, fail-closed when unreadable. 192 browser tests pass. Live: 68 real
cookies copied through the backup path and the session launches.
Addresses the round-3 findings from @Adolanium + @kshitijk4poor on #95620:
1. Overlay-before-reuse race (blocker): _real_profile_cdp ran snapshot_real_profile
BEFORE the session-reuse check, so a cold resolve that ends in reuse rewrote
Cookies/Login Data under a live Chromium holding the user-data-dir open (torn
DBs, locked txns, phantom logouts). Now: resolve copy dir as a PATH, probe
reuse first, return early on a hit; snapshot/overlay only on the relaunch
path when no live browser owns the dir.
2. Torn first copy poisoned freshness forever: freshness keyed on isdir(Default),
so a half-written copy (disk full / Ctrl+C) was treated as populated and only
ever got auth overlays. Now gated on a .hermes-snapshot-complete marker
written only after a full copy succeeds; a torn copy is rebuilt from scratch.
3. Consent revocation left copied credentials on disk: turning use_real_profile
off now deletes ~/.hermes/browser-profile/ on next browser use
(cleanup_real_profile_snapshots), so cookies/logins don't outlive consent.
4. Stale non-active profile copies: only the ACTIVE profile (last_used) is copied
into the copy's Default now — other Chrome profiles are never snapshotted
(smaller copy, no stale credential dirs lingering).
5. Docs/config/desktop wording aligned to actual behavior (active-profile only,
refresh on fresh session, consent-off cleanup).
Tests: overlay-skipped-on-reuse + overlay-runs-on-relaunch, done-marker gating +
torn-copy rebuild, active-only copy, consent-off cleanup (removes store +
idempotent + triggered from _real_profile_cdp). 206 browser + 222
backup/file_safety pass. Live: reuse skips re-snapshot; direct launch on the
active-only copy loads the real signed-in Gmail inbox.
Real-profile browsing routes all in-page browser work through the Browser Use
CLI (browser_exec), which obsoletes the cua-driver typed-browser surface baked
into computer_use. Remove it so computer_use is a pure DESKTOP-control tool
(screenshots / mouse / keyboard / window management) and every call's schema
drops ~24 browser-only params + 9 actions.
- schema.py: 9 cua_browser_* actions and the typed-browser param block removed;
14 desktop actions + shared params kept; description drops the browser rung.
- tool.py: cua_browser entries out of _SAFE/_DESTRUCTIVE_ACTIONS; the whole
cua_browser dispatch block deleted; {"type","cua_browser_type"} → "type"
(desktop typing untouched); _config_preauthorized (browser-prepare-only, a
no-op for every desktop action) and the browser-page escalation hint removed.
- browser_route.py deleted (no importers outside the package); cua_backend.py
drops the import + typed_browser_* methods; backend.py drops the non-abstract
defaults.
- tests: browser-route/contract suites removed; browser assertions trimmed.
Desktop control unchanged. 233 computer_use tests pass; the 1 remaining failure
(test_gateway_session_key_yolo_maps_to_unrestricted_mode) is a pre-existing
cross-test state leak — fails identically on origin/main, passes in isolation.
Addresses the five findings from @kshitijk4poor + @GottZ on #95620:
1. macOS 26 LSHandlers parser returned a version number ('7559.97') from the
nested LSHandlerPreferredVersions block instead of the bundle id — detection
returned None on a machine whose default IS Chrome. Strip the nested block
before the role regex.
2. Wrong profile launched (the LinkedIn/Gmail 'logged out' bug): Chrome opens
Default, but the session lives in Local State profile.last_used (e.g.
'Profile 6'). Resolve last_used and mirror its auth files into the copy's
Default on both fresh and refresh paths, so the launched browser is signed in.
3. Private-URL sidecar carried the real cookie jar to arbitrary LAN hosts:
_create_local_session gains allow_real_profile (default True); the
force_local sidecar passes False → always a throwaway profile, and a
real-profile resolve failure no longer breaks private-URL routing.
4. Snapshot permissions were set once (fresh only): now secure the snapshot dir
AND its browser-profile parent on every consented launch.
5. browser.engine=lightpanda + consent gave an unactionable error: guard with
_using_lightpanda_engine() before detection, naming the setting and the fix.
Tests: last_used mirroring (fresh+refresh+fallback), sidecar throwaway + error
isolation, macOS26 parser + detect, perms-on-refresh, lightpanda guard. 198
browser tests pass. Live: Profile-6 cookie DB lands in copy Default (file-level);
real Gmail (Default profile) still signed in.
Addresses two P1 review blockers (kshitij / @kxee) on the real-profile feature:
Credential-store lifecycle for ~/.hermes/browser-profile/ (copied Cookies/
Login Data):
- exclude the singular 'browser-profile' dir from backup AND import
(_EXCLUDED_DIRS drives both) — was silently archiving cookies/logins
- add a browser-profile/ directory-PREFIX read-deny to agent/file_safety.py,
same class as auth.json / mcp-tokens
- secure the snapshot dir through the canonical hermes_cli.config._secure_dir
(honors managed/NixOS group-share + HERMES_UID/GID), not a bespoke chmod
Channel identity (#95549 invariant — never normalize Beta/Dev/Canary to
stable, which would drive a different account's profile):
- detect recognized pre-release channels FIRST (Win ProgIds, macOS bundle ids,
Linux .desktop) and return UNSUPPORTED_CHANNEL
- macOS bundle match is now EXACT (was startswith); Linux/Win channel-before-
stable ordering; real_profile_data_dir/chromium_executable reject the sentinel
- _real_profile_cdp fails closed with a channel-specific message, never snapshots
Tests: channel-not-normalized (linux/darwin/windows), wrong-principal fail-closed,
backup exclusion, read-guard block/allow, snapshot dir secured. 187 browser +
222 backup/file_safety pass. Live re-verified: real Gmail inbox still loads.
Copy the user's default-Chromium profile (auth state only) into a managed
snapshot, launch Hermes' packaged Chromium on it via agent-browser, and hand
the CDP endpoint to the Browser Use CLI (and built-in tools) to drive. The
snapshot is a non-default dir, so it sidesteps Chrome 136+'s default-profile
remote-debugging block and never contends with the user's running browser;
launched without mock-keychain switches so keyring-encrypted cookies decrypt.
- consent-gated browser_exec 'local' arg (schema only appears with consent)
- fail-closed on non-Chromium default / snapshot failure
- stale-session guard: reuse only when the live session is on our copy dir
- snapshot excludes extensions/service-workers (renderer wedge) + caches
_navigation_session_key returned the ::local sidecar key for local_browser
before the cloud-provider and auto_local_for_private_urls checks. Two
consequences:
- Every private-URL gate in browser_navigate (credential-bearing query,
_is_safe_url pre-nav, post-redirect) is keyed off the sidecar key, so a
model-supplied browser_navigate(url, local_browser=True) opened LAN
addresses in a host-side Chromium — with the real profile's cookies — even
when the user had set browser.auto_local_for_private_urls: false. Consent to
the profile is not consent to override the LAN routing opt-out; a private
URL now follows auto_local_for_private_urls exactly as without the flag.
- Without a cloud provider the bare session already is the local Chromium
(and already carries the real profile when consented), so the flag created a
second session for the same task on the same user-data-dir, which Chromium's
process singleton refuses. The flag is now a no-op there.
The existing consent/CDP-precedence tests keep their assertions; the sidecar
test now states the cloud-provider precondition it silently relied on.
real_profile_data_dir hard-wired Linux to $XDG_CONFIG_HOME/<name>, and the
xdg fragment map only knew the native package names. Ubuntu's default snap
Chromium (xdg reports chromium_chromium.desktop, profile under
~/snap/chromium/common/chromium) and Flatpak builds (~/.var/app/<id>/config/…)
therefore ended in 'profile directory was not found' for a browser the user
runs every day, and Flatpak Chrome (com.google.Chrome.desktop) was reported as
'not a supported Chromium browser'.
Try the native, snap and Flatpak locations and return the first that exists;
fall back to the native path so the error message still names a concrete
directory. Map the Flatpak application ids in the xdg lookup.
Tests cover the xdg names for all four browsers in native and Flatpak form,
and the directory preference order with a temp HOME.
_detect_default_darwin matched a Chromium bundle id and the literal 'https'
anywhere in the whole LSHandlers dump, so a browser registered for ftp or a
content type was reported as the https default, and map order decided ties.
When nothing matched it fell back to the first installed Chromium app — with
Safari or Firefox as the actual default that drove a browser the user never
consented to, contradicting the docstring, the config comment and the desktop
copy ('a non-Chromium default fails with a clear message').
Parse the dump entry by entry, take the LSHandlerRoleAll/Viewer of the entry
whose LSHandlerURLScheme is https, and fail closed on anything else — an empty
handler list is what macOS stores while Safari is still the implicit default.
Tests feed real 'defaults read' output shapes instead of patching the detector
(reviewer fixture from the PR discussion: Safari on https, Chrome on ftp).
The two default-browser detectors call subprocess.run(text=True) without an
explicit encoding, which the Windows-footgun linter (and its full-repo test,
tests/scripts/test_footgun_subprocess_encoding.py) rejects. Pass
encoding='utf-8', errors='replace' like the rest of the tree.
Two existing tests replace browser_navigate / _navigation_session_key with
positional-only lambdas; both callables now receive local_browser= from the
registry handler and browser_navigate, so the spies raised TypeError. Accept
the keyword with its default.
Fixes the three CI failures on the PR head (Windows footguns lint,
test_browser_extension_router_wiring x2, test_browser_open_timeout).
* fix(terminal): stop claiming a Linux environment — point at the env section; near-neutral tokens
* feat(terminal): default GIT_PAGER/PAGER=cat in session env; drop schema lines the runtime already enforces; fix pty backend claim
* refactor(terminal): unify notify_on_complete+watch_patterns into notify (bool|list); trim pipe-masking prose (runtime hint owns it)
* fix(terminal): background param referenced the unadvertised legacy arg name
* refactor(terminal): background-only modifiers (pty, notify) fail loud on foreground calls with corrected shape
* fix(execute_code): block the new notify arg in the sandbox terminal stub (foreground-only)
Desktop builds legitimately produce no output for minutes on Windows
(the updater terminal is static until a step completes), so 300s risks
cancelling healthy long steps. 600s keeps the watchdog meaningful for
true stalls while clearing slow builds; HERMES_UPDATE_STEP_IDLE_SECONDS
still overrides. Per Teknium's review.
Review follow-ups on the salvage (#94914): the delete path derives its
scope from the removed row while saves derive from ownerRoute — a shape
drift orphaned the entry until LRU eviction. Deletes now sweep every
scope of the stored id (safe: deleted ids are never reused), while the
failed-resume path keeps the exact-scope drop (a same-id twin in another
profile must keep its tail). purgeLegacyV1 latches only after a
completed sweep so a mid-sweep throw retries next touch.
Stored session ids are only unique within one profile's state.db, and
localStorage survives profile switches in the same window. The durable
transcript-tail cache keyed entries by bare stored id, so a tail cached
while working in profile A was painted against profile B's backend after
a switch; that backend never held the session and retried it on every
wake ('session not found'), matching the desktop.log pattern of repeated
session.reclaimed (ws_orphan_reap) events for ids that live in another
profile's state.db (#94828).
Scope every entry by its owner — the same {connectionId, profile} shape
the REST layer already threads through getLatestSessionMessages and the
in-memory twin stores in TranscriptTailState:
- key entries under v2:[connectionId, profile, storedId]; loads only
return a tail saved under the SAME scope
- thread the session's owning scope through all call sites
(resumeSession's warm-path save + cold-paint load/rollback drop,
final save, removeSession's delete drop)
- sweep pre-scoping v1 entries once per window: a bare-id key cannot be
attributed to an owner, so it must never paint again
- dropTranscriptTail with a scope leaves other backends' same-id tails
intact
Main's pin (added after this branch was cut) froze the #93406 bug as the
contract: resume-token services demanded probe rows that SCM-paused
services can never produce, stalling every healthy Windows desktop update
(#95589). The pin now asserts the exclusion; restart-phase and
pre-restart-pid signals keep failing closed.
.gitattributes forces eol=crlf for *.ps1, so CI checkouts hand the test
CRLF content and the SelfTest-strip regex anchors never matched; it went
unnoticed while the stripped fixture blocks contained no offenders.
The cherry-pick landed the file with CRLF endings and a fixture
here-string whose col-0 brace prematurely terminated the handoff test's
SelfTest-block strip, tripping the drive-python-not-the-shim guard on
fixture code. LF restored (matching main), child-script loop inlined.
Fixes#93406 (residual). _fleet_probe_expected_runtimes counted the
_windows_gateway_resume pause/resume token (profiles/unmapped entries)
as an 'expected fleet rows' signal. The token is pause/resume
bookkeeping, not a runtime inventory, and its entries have no rows
collect_fleet_versions() can return: unmapped Scheduled-Task gateways
never publish gateway_state.json, and a resumed profile gateway
relaunches detached and may not republish within the probe window. So
every Windows update that paused a gateway set _fleet_rows_expected,
the verification loop silently waited out its polling window (~14 min
wall clock with the retry loop on user reports), printed 'Fleet version
check returned no rows', and exited 1 for an update that succeeded.
Expected-runtimes now keys only on row-capable signals: restart-phase
bookkeeping, the pre-restart PID snapshot, and the pre-update plan
inventory -- which already cover any genuinely live pre-update Windows
gateway.
Counterfactual proof: tests/hermes_cli/test_update_fleet_probe_resume_token.py
fails on the pre-fix predicate (token-only => True) and passes with the
fix; the row-capable signals are pinned unchanged.
The #95625 watchdog cancels a step after StepIdleTimeoutSeconds (300s)
with no stdout/stderr. But a real `hermes update` is stdout-silent for
40+ minutes by design: the Electron/vite build streams to
logs/update.log, not the child's pipes (hermes_cli/update_cmd.py's
update-log tee). An output-only ceiling would therefore kill every
healthy large update at 5 minutes and mark it exit 124.
The drain now fingerprints logs/update.log (size + mtime) and, when the
idle ceiling is otherwise reached, treats growth of that file as
progress: reset the clock instead of terminating the tree. The stat
runs only once the ceiling fires, so the hot drain path never touches
the filesystem. HERMES_UPDATE_STEP_IDLE_SECONDS remains the override;
HERMES_UPDATE_PROGRESS_LOG points the self-test at its own file.
TDD proof: -SelfTestPipeDrain gains a fourth arm, logstall -- a step
that is silent on its pipes but appends to the progress log every
second and must reach its natural exit 3, never 124. Linux CI pins the
same contract at source level (TestIdleWatchdogCountsUpdateLogGrowth);
sabotage-verified: making the log-growth consult inert fails
test_stall_branch_consults_log_growth_before_terminating.
Stdio MCP helper subprocesses (npx/binary servers) never import Hermes
code, so they could not self-register in the machine spawn ledger and an
unclean parent exit left them running invisibly forever.
- process_identity.register_child(pid, purpose): ledger mirror of
register_self for spawned children — records the CHILD (pid,
create_time) with this process as spawner. Refuses pid-only entries a
PID reuse could forge. Writes go through the single _append_entry
path under _LEDGER_LOCK (prune + atomic tmp/replace unchanged).
- 'mcp-helper' added to REAPABLE_PURPOSES so the updater's
_ledger_reapable_backend_pids rung flows helpers through its existing
spawner_is_dead gate (live spawner => never reaped).
- tools/mcp_tool.py: best-effort register_child(pid, 'mcp-helper') at
the post-spawn PID capture; never breaks MCP startup.
- reap_orphaned_mcp_helpers(): startup sweep mirroring
_reap_orphaned_desktop_local_serves but ledger-driven — kills only
helpers whose recorded spawner is PROVABLY dead, with a create_time
re-check at kill time. Wired next to the desktop serve reap in
web_server.py.
A failed update attempt can pull fresh code onto disk and then die before
the config-migration block (e.g. a PyPI timeout during the dependency
sync). The desktop hand-off retries; the retry takes the commit_count == 0
branch, repairs deps, prints 'Already up to date!' and returns early -
skipping _run_config_check_fresh / migrate_config entirely. The fresh
code (requiring a newer _config_version) then refuses to start against
the old config until 'hermes doctor --fix' is run.
Fix: _maybe_migrate_config_on_current() mirrors the version_bump_only
handling (silent, non-interactive) and is called on both repair-path
completion points before claiming success.
Also: scripts/desktop-update/posix.sh no longer retries when the update
was deliberately SKIPPED (checkout parked on a non-target branch) -, the
retry is deterministic and only wastes time. Uses a dedicated non-
colliding exit code (8) and an honest message instead of 'Update failed'.
New tests: tests/hermes_cli/test_update_config_migration_on_current.py
(5 cases: migrate-when-behind, noop-current, noop-ahead, warning re-
surface, silent check failure).
When an update was interrupted or failed mid-install (e.g. dependency install
timeout) after pulling new code, the subsequent update run takes the
'commit_count == 0' path and early-returned without checking or migrating
the configuration. Fresh code requiring a newer config version would fail to
boot on the next run.
Extract _check_and_apply_config_migration and invoke it across all update
completion paths (normal update, current checkout / node repair, and python
dependency repair).
Re-implements the #93042 main.ts wiring against current main (post-#94724
drift), making the extracted managed-ssh-update engine reachable:
- ManagedConnectionUpdateGate instance + owner-only recovery journal at
DESKTOP_MANAGED_SSH_RECOVERY_PATH (read/write/persist/mark/clear with
strict record validation).
- IPC: hermes:connections:update-managed (requestManagedSshUpdate with
correlation-id claim + in-flight dedupe); update-all's ssh rows now route
through the transactional drain/update/restore lifecycle instead of
POSTing the remote backend updater.
- Gate enforcement at every dial/mutate seam: bootstrapSshConnectionInner
(pre-dial + publication fence with exact-serve rollback via
rollbackSshBootstrapResult), resolveRemoteBackend, ensureRegistryBackend,
saveRegistryConnection dial-field edits, connections:remove, and
primary-routing mutations (set-primary, set-launch-mode,
connection-config save/apply, profile:set) via
assertCanMutateManagedPrimaryRouting.
- Scope capture/drain/restore drivers: captureManagedSshScopes (pool +
primary discovery, bootstrap fence join), drainManagedSshScope (exact
identity-re-proof termination, no-kill forward recovery),
ensureManagedSshBackend(AtKey)/restoreManagedPrimarySshBackend restores,
openManagedSshUpdateTransport for serve-less connections.
- Startup recovery (resumeManagedSshRecoveries before createWindow) and
before-quit join of in-flight update/recovery operations BEFORE the SSH
coordinator is sealed, so restore dials are not refused during quit.
- Extended sshConnections state (spawnNonce/creationTime(Ns)/startedAt/
hermesPath/hermesHome/pythonPath/remoteProfile/registryConnectionId/
primaryRegistryScope) so drain can prove the exact serve it owns;
bootstrap coordinator entries carry metadata for the update fence;
persistSshConnectionToken mirrors tokens per managedSshTokenPersistencePlan.
- preload/global.d.ts: connections.updateManaged +
DesktopManagedConnectionUpdateResult/Receipt types.
Renderer UI (fleet-updates store, about-settings, system.ts ProfileScope
plumbing, i18n) intentionally NOT wired — it belongs to the deferred fleet
rollout UI and follows separately.
Wiring re-implemented against current main; design from #93042 by @andrexibiza
tsc -p apps/desktop clean; electron project 1924/1924 passed (133 files,
incl. 131/131 across the three engine suites); eslint clean on touched files.
The Desktop can drive a full update of a REMOTE SSH instance: claim the
connection (ManagedConnectionUpdateGate pauses dials/mutations while the
update owns it), terminate the owned backend with an identity re-proof
at the signal boundary (terminateOwnedDashboardForUpdate — argv+creation
time re-read in the same remote shell that signals), run the updater
under an update-in-progress marker/mutex on the remote install root,
bootstrap the new backend, and fence its publication so a rollback
cannot leak a half-published serve (fenceManagedSshBootstrapPublication
+ waitForManagedSshBootstrapFence barrier).
Extraction notes (campaign #91277 Phase 4; rollout/canary engine stays
behind per sequencing):
- managed-ssh-update.ts + test: clean cherry-pick (new files).
- windows-remote-lifecycle.ts + test: applied as-is (zero main drift).
- remote-lifecycle.ts: PR hunks stitched AROUND current main's #95532
skew guards and #91668 SIGKILL-escalating cleanupStale, which are
PRESERVED verbatim — the PR's python identity-re-proof termination is
wired for the managed-update path only; connect's stale replacement
keeps main's proven kill. One PR test assertion re-pinned accordingly.
- One PR test fixed: floating coordinator.start() promise whose
rejection IS the contract under test now has an explicit handler
(vitest flagged it as an unhandled rejection).
tsc clean; 131/131 across the three touched electron suites. main.ts
wiring (IPC + deps bag) follows as a separate commit.
Structural fix after five review rounds put four blockers in the same
subsystem. The root cause was representational: consent history is a
sequence of on/off intervals, but it was stored as ONE moving day-stamp
plus a revoked flag. Every fix had to mutate that scalar at exactly the
right moment from exactly the right place, and each round the mutation
was missing from some reachable path (write-once stamp in R3; recorded
inside a loop that never runs when sending is off in R4; dead code
whenever collection was off in R5).
Consent is now recorded as explicit intervals (send_consent_windows) and
eligibility is a pure derivation: a package is sent only when its whole
period falls inside a recorded window. One writer -
reconcile_send_consent - derives window state from an observation of
(config, now). It is idempotent and order-independent, so the wizard,
the relay, and the mid-pass check all call the same function and cannot
disagree; there are no edges to detect and no ordering between writers
to get wrong. The relay reconciles once per process BEFORE the
collection gate, which fixes round-5 D1 (enabled:false made the only
idle-path observer unreachable). The claim reads the table and never
writes it, removing the read-path mutation (D2's rewrite vector).
Timestamp discipline, each rule load-bearing and mutation-tested:
- 'obs' high-water mark: monotonic, advanced only by observations;
confirms an open window forward (last_confirmed_at).
- 'data' high-water mark: advanced only by stored package period_end;
clamps window OPENS so a rolled-back clock cannot slide a window
under refused packages already on disk (round-5 D2).
- A close stamps last_confirmed_at, never "now": consent is asserted
only for observed time, so a hand-edited config with no process
running for 90 days fails closed (round-5 D1 strongest form).
- The gate requires period containment, not period_start >=, so an
intra-day revoke/re-enable holds back the day package (round-5 D3).
- Unlike the day-stamp, a revoke/re-enable cycle no longer destroys the
undelivered backlog from the earlier consented window (round-5 D4).
The redesign was validated BEFORE implementation against all 13
reproduced defect scenarios on a real store; the first two drafts each
failed scenarios in that harness (v1 leaked the unobserved-gap case by
closing at "now"; v2 leaked refused windows by letting data stamps
confirm consent). The harness ships as
tests/hermes_cli/test_shared_metrics_consent_windows.py.
Deleted: OPT_IN_PERIOD_KEY, SEND_REVOKED_KEY, LAST_SEEN_SEND_KEY,
opt_in_period(), record_revoked(), the relay edge detector body, and the
setup wizard's key bookkeeping (~170 lines of transition machinery).
Schema: two additive tables, version deliberately unchanged; verified
against a copy of the real production DB (13 rows intact, reopen no-op).
Also kills round-5's M8 survivor: the seen-exclusion mutation now fails
the suite. New mutation sweep: 8/8 killed, including one vacuous test of
my own this round (obs-mark monotonicity was covered only by
coincidence of the data mark; now pinned directly).
Documented cost: a fresh package waits at most one process start after
its period completes before release (fail-closed direction).
270 tests pass; ruff and windows-footguns clean. Staging E2E re-run
through the interval gate: both packages 202.
Teknium