Addresses the five findings from @kshitijk4poor + @GottZ on #95620:
1. macOS 26 LSHandlers parser returned a version number ('7559.97') from the
nested LSHandlerPreferredVersions block instead of the bundle id — detection
returned None on a machine whose default IS Chrome. Strip the nested block
before the role regex.
2. Wrong profile launched (the LinkedIn/Gmail 'logged out' bug): Chrome opens
Default, but the session lives in Local State profile.last_used (e.g.
'Profile 6'). Resolve last_used and mirror its auth files into the copy's
Default on both fresh and refresh paths, so the launched browser is signed in.
3. Private-URL sidecar carried the real cookie jar to arbitrary LAN hosts:
_create_local_session gains allow_real_profile (default True); the
force_local sidecar passes False → always a throwaway profile, and a
real-profile resolve failure no longer breaks private-URL routing.
4. Snapshot permissions were set once (fresh only): now secure the snapshot dir
AND its browser-profile parent on every consented launch.
5. browser.engine=lightpanda + consent gave an unactionable error: guard with
_using_lightpanda_engine() before detection, naming the setting and the fix.
Tests: last_used mirroring (fresh+refresh+fallback), sidecar throwaway + error
isolation, macOS26 parser + detect, perms-on-refresh, lightpanda guard. 198
browser tests pass. Live: Profile-6 cookie DB lands in copy Default (file-level);
real Gmail (Default profile) still signed in.
Addresses two P1 review blockers (kshitij / @kxee) on the real-profile feature:
Credential-store lifecycle for ~/.hermes/browser-profile/ (copied Cookies/
Login Data):
- exclude the singular 'browser-profile' dir from backup AND import
(_EXCLUDED_DIRS drives both) — was silently archiving cookies/logins
- add a browser-profile/ directory-PREFIX read-deny to agent/file_safety.py,
same class as auth.json / mcp-tokens
- secure the snapshot dir through the canonical hermes_cli.config._secure_dir
(honors managed/NixOS group-share + HERMES_UID/GID), not a bespoke chmod
Channel identity (#95549 invariant — never normalize Beta/Dev/Canary to
stable, which would drive a different account's profile):
- detect recognized pre-release channels FIRST (Win ProgIds, macOS bundle ids,
Linux .desktop) and return UNSUPPORTED_CHANNEL
- macOS bundle match is now EXACT (was startswith); Linux/Win channel-before-
stable ordering; real_profile_data_dir/chromium_executable reject the sentinel
- _real_profile_cdp fails closed with a channel-specific message, never snapshots
Tests: channel-not-normalized (linux/darwin/windows), wrong-principal fail-closed,
backup exclusion, read-guard block/allow, snapshot dir secured. 187 browser +
222 backup/file_safety pass. Live re-verified: real Gmail inbox still loads.
Copy the user's default-Chromium profile (auth state only) into a managed
snapshot, launch Hermes' packaged Chromium on it via agent-browser, and hand
the CDP endpoint to the Browser Use CLI (and built-in tools) to drive. The
snapshot is a non-default dir, so it sidesteps Chrome 136+'s default-profile
remote-debugging block and never contends with the user's running browser;
launched without mock-keychain switches so keyring-encrypted cookies decrypt.
- consent-gated browser_exec 'local' arg (schema only appears with consent)
- fail-closed on non-Chromium default / snapshot failure
- stale-session guard: reuse only when the live session is on our copy dir
- snapshot excludes extensions/service-workers (renderer wedge) + caches
_navigation_session_key returned the ::local sidecar key for local_browser
before the cloud-provider and auto_local_for_private_urls checks. Two
consequences:
- Every private-URL gate in browser_navigate (credential-bearing query,
_is_safe_url pre-nav, post-redirect) is keyed off the sidecar key, so a
model-supplied browser_navigate(url, local_browser=True) opened LAN
addresses in a host-side Chromium — with the real profile's cookies — even
when the user had set browser.auto_local_for_private_urls: false. Consent to
the profile is not consent to override the LAN routing opt-out; a private
URL now follows auto_local_for_private_urls exactly as without the flag.
- Without a cloud provider the bare session already is the local Chromium
(and already carries the real profile when consented), so the flag created a
second session for the same task on the same user-data-dir, which Chromium's
process singleton refuses. The flag is now a no-op there.
The existing consent/CDP-precedence tests keep their assertions; the sidecar
test now states the cloud-provider precondition it silently relied on.
real_profile_data_dir hard-wired Linux to $XDG_CONFIG_HOME/<name>, and the
xdg fragment map only knew the native package names. Ubuntu's default snap
Chromium (xdg reports chromium_chromium.desktop, profile under
~/snap/chromium/common/chromium) and Flatpak builds (~/.var/app/<id>/config/…)
therefore ended in 'profile directory was not found' for a browser the user
runs every day, and Flatpak Chrome (com.google.Chrome.desktop) was reported as
'not a supported Chromium browser'.
Try the native, snap and Flatpak locations and return the first that exists;
fall back to the native path so the error message still names a concrete
directory. Map the Flatpak application ids in the xdg lookup.
Tests cover the xdg names for all four browsers in native and Flatpak form,
and the directory preference order with a temp HOME.
_detect_default_darwin matched a Chromium bundle id and the literal 'https'
anywhere in the whole LSHandlers dump, so a browser registered for ftp or a
content type was reported as the https default, and map order decided ties.
When nothing matched it fell back to the first installed Chromium app — with
Safari or Firefox as the actual default that drove a browser the user never
consented to, contradicting the docstring, the config comment and the desktop
copy ('a non-Chromium default fails with a clear message').
Parse the dump entry by entry, take the LSHandlerRoleAll/Viewer of the entry
whose LSHandlerURLScheme is https, and fail closed on anything else — an empty
handler list is what macOS stores while Safari is still the implicit default.
Tests feed real 'defaults read' output shapes instead of patching the detector
(reviewer fixture from the PR discussion: Safari on https, Chrome on ftp).
The two default-browser detectors call subprocess.run(text=True) without an
explicit encoding, which the Windows-footgun linter (and its full-repo test,
tests/scripts/test_footgun_subprocess_encoding.py) rejects. Pass
encoding='utf-8', errors='replace' like the rest of the tree.
Two existing tests replace browser_navigate / _navigation_session_key with
positional-only lambdas; both callables now receive local_browser= from the
registry handler and browser_navigate, so the spies raised TypeError. Accept
the keyword with its default.
Fixes the three CI failures on the PR head (Windows footguns lint,
test_browser_extension_router_wiring x2, test_browser_open_timeout).
* fix(terminal): stop claiming a Linux environment — point at the env section; near-neutral tokens
* feat(terminal): default GIT_PAGER/PAGER=cat in session env; drop schema lines the runtime already enforces; fix pty backend claim
* refactor(terminal): unify notify_on_complete+watch_patterns into notify (bool|list); trim pipe-masking prose (runtime hint owns it)
* fix(terminal): background param referenced the unadvertised legacy arg name
* refactor(terminal): background-only modifiers (pty, notify) fail loud on foreground calls with corrected shape
* fix(execute_code): block the new notify arg in the sandbox terminal stub (foreground-only)
Desktop builds legitimately produce no output for minutes on Windows
(the updater terminal is static until a step completes), so 300s risks
cancelling healthy long steps. 600s keeps the watchdog meaningful for
true stalls while clearing slow builds; HERMES_UPDATE_STEP_IDLE_SECONDS
still overrides. Per Teknium's review.
Review follow-ups on the salvage (#94914): the delete path derives its
scope from the removed row while saves derive from ownerRoute — a shape
drift orphaned the entry until LRU eviction. Deletes now sweep every
scope of the stored id (safe: deleted ids are never reused), while the
failed-resume path keeps the exact-scope drop (a same-id twin in another
profile must keep its tail). purgeLegacyV1 latches only after a
completed sweep so a mid-sweep throw retries next touch.
Stored session ids are only unique within one profile's state.db, and
localStorage survives profile switches in the same window. The durable
transcript-tail cache keyed entries by bare stored id, so a tail cached
while working in profile A was painted against profile B's backend after
a switch; that backend never held the session and retried it on every
wake ('session not found'), matching the desktop.log pattern of repeated
session.reclaimed (ws_orphan_reap) events for ids that live in another
profile's state.db (#94828).
Scope every entry by its owner — the same {connectionId, profile} shape
the REST layer already threads through getLatestSessionMessages and the
in-memory twin stores in TranscriptTailState:
- key entries under v2:[connectionId, profile, storedId]; loads only
return a tail saved under the SAME scope
- thread the session's owning scope through all call sites
(resumeSession's warm-path save + cold-paint load/rollback drop,
final save, removeSession's delete drop)
- sweep pre-scoping v1 entries once per window: a bare-id key cannot be
attributed to an owner, so it must never paint again
- dropTranscriptTail with a scope leaves other backends' same-id tails
intact
Main's pin (added after this branch was cut) froze the #93406 bug as the
contract: resume-token services demanded probe rows that SCM-paused
services can never produce, stalling every healthy Windows desktop update
(#95589). The pin now asserts the exclusion; restart-phase and
pre-restart-pid signals keep failing closed.
.gitattributes forces eol=crlf for *.ps1, so CI checkouts hand the test
CRLF content and the SelfTest-strip regex anchors never matched; it went
unnoticed while the stripped fixture blocks contained no offenders.
The cherry-pick landed the file with CRLF endings and a fixture
here-string whose col-0 brace prematurely terminated the handoff test's
SelfTest-block strip, tripping the drive-python-not-the-shim guard on
fixture code. LF restored (matching main), child-script loop inlined.
Fixes#93406 (residual). _fleet_probe_expected_runtimes counted the
_windows_gateway_resume pause/resume token (profiles/unmapped entries)
as an 'expected fleet rows' signal. The token is pause/resume
bookkeeping, not a runtime inventory, and its entries have no rows
collect_fleet_versions() can return: unmapped Scheduled-Task gateways
never publish gateway_state.json, and a resumed profile gateway
relaunches detached and may not republish within the probe window. So
every Windows update that paused a gateway set _fleet_rows_expected,
the verification loop silently waited out its polling window (~14 min
wall clock with the retry loop on user reports), printed 'Fleet version
check returned no rows', and exited 1 for an update that succeeded.
Expected-runtimes now keys only on row-capable signals: restart-phase
bookkeeping, the pre-restart PID snapshot, and the pre-update plan
inventory -- which already cover any genuinely live pre-update Windows
gateway.
Counterfactual proof: tests/hermes_cli/test_update_fleet_probe_resume_token.py
fails on the pre-fix predicate (token-only => True) and passes with the
fix; the row-capable signals are pinned unchanged.
The #95625 watchdog cancels a step after StepIdleTimeoutSeconds (300s)
with no stdout/stderr. But a real `hermes update` is stdout-silent for
40+ minutes by design: the Electron/vite build streams to
logs/update.log, not the child's pipes (hermes_cli/update_cmd.py's
update-log tee). An output-only ceiling would therefore kill every
healthy large update at 5 minutes and mark it exit 124.
The drain now fingerprints logs/update.log (size + mtime) and, when the
idle ceiling is otherwise reached, treats growth of that file as
progress: reset the clock instead of terminating the tree. The stat
runs only once the ceiling fires, so the hot drain path never touches
the filesystem. HERMES_UPDATE_STEP_IDLE_SECONDS remains the override;
HERMES_UPDATE_PROGRESS_LOG points the self-test at its own file.
TDD proof: -SelfTestPipeDrain gains a fourth arm, logstall -- a step
that is silent on its pipes but appends to the progress log every
second and must reach its natural exit 3, never 124. Linux CI pins the
same contract at source level (TestIdleWatchdogCountsUpdateLogGrowth);
sabotage-verified: making the log-growth consult inert fails
test_stall_branch_consults_log_growth_before_terminating.
Stdio MCP helper subprocesses (npx/binary servers) never import Hermes
code, so they could not self-register in the machine spawn ledger and an
unclean parent exit left them running invisibly forever.
- process_identity.register_child(pid, purpose): ledger mirror of
register_self for spawned children — records the CHILD (pid,
create_time) with this process as spawner. Refuses pid-only entries a
PID reuse could forge. Writes go through the single _append_entry
path under _LEDGER_LOCK (prune + atomic tmp/replace unchanged).
- 'mcp-helper' added to REAPABLE_PURPOSES so the updater's
_ledger_reapable_backend_pids rung flows helpers through its existing
spawner_is_dead gate (live spawner => never reaped).
- tools/mcp_tool.py: best-effort register_child(pid, 'mcp-helper') at
the post-spawn PID capture; never breaks MCP startup.
- reap_orphaned_mcp_helpers(): startup sweep mirroring
_reap_orphaned_desktop_local_serves but ledger-driven — kills only
helpers whose recorded spawner is PROVABLY dead, with a create_time
re-check at kill time. Wired next to the desktop serve reap in
web_server.py.
A failed update attempt can pull fresh code onto disk and then die before
the config-migration block (e.g. a PyPI timeout during the dependency
sync). The desktop hand-off retries; the retry takes the commit_count == 0
branch, repairs deps, prints 'Already up to date!' and returns early -
skipping _run_config_check_fresh / migrate_config entirely. The fresh
code (requiring a newer _config_version) then refuses to start against
the old config until 'hermes doctor --fix' is run.
Fix: _maybe_migrate_config_on_current() mirrors the version_bump_only
handling (silent, non-interactive) and is called on both repair-path
completion points before claiming success.
Also: scripts/desktop-update/posix.sh no longer retries when the update
was deliberately SKIPPED (checkout parked on a non-target branch) -, the
retry is deterministic and only wastes time. Uses a dedicated non-
colliding exit code (8) and an honest message instead of 'Update failed'.
New tests: tests/hermes_cli/test_update_config_migration_on_current.py
(5 cases: migrate-when-behind, noop-current, noop-ahead, warning re-
surface, silent check failure).
When an update was interrupted or failed mid-install (e.g. dependency install
timeout) after pulling new code, the subsequent update run takes the
'commit_count == 0' path and early-returned without checking or migrating
the configuration. Fresh code requiring a newer config version would fail to
boot on the next run.
Extract _check_and_apply_config_migration and invoke it across all update
completion paths (normal update, current checkout / node repair, and python
dependency repair).
Re-implements the #93042 main.ts wiring against current main (post-#94724
drift), making the extracted managed-ssh-update engine reachable:
- ManagedConnectionUpdateGate instance + owner-only recovery journal at
DESKTOP_MANAGED_SSH_RECOVERY_PATH (read/write/persist/mark/clear with
strict record validation).
- IPC: hermes:connections:update-managed (requestManagedSshUpdate with
correlation-id claim + in-flight dedupe); update-all's ssh rows now route
through the transactional drain/update/restore lifecycle instead of
POSTing the remote backend updater.
- Gate enforcement at every dial/mutate seam: bootstrapSshConnectionInner
(pre-dial + publication fence with exact-serve rollback via
rollbackSshBootstrapResult), resolveRemoteBackend, ensureRegistryBackend,
saveRegistryConnection dial-field edits, connections:remove, and
primary-routing mutations (set-primary, set-launch-mode,
connection-config save/apply, profile:set) via
assertCanMutateManagedPrimaryRouting.
- Scope capture/drain/restore drivers: captureManagedSshScopes (pool +
primary discovery, bootstrap fence join), drainManagedSshScope (exact
identity-re-proof termination, no-kill forward recovery),
ensureManagedSshBackend(AtKey)/restoreManagedPrimarySshBackend restores,
openManagedSshUpdateTransport for serve-less connections.
- Startup recovery (resumeManagedSshRecoveries before createWindow) and
before-quit join of in-flight update/recovery operations BEFORE the SSH
coordinator is sealed, so restore dials are not refused during quit.
- Extended sshConnections state (spawnNonce/creationTime(Ns)/startedAt/
hermesPath/hermesHome/pythonPath/remoteProfile/registryConnectionId/
primaryRegistryScope) so drain can prove the exact serve it owns;
bootstrap coordinator entries carry metadata for the update fence;
persistSshConnectionToken mirrors tokens per managedSshTokenPersistencePlan.
- preload/global.d.ts: connections.updateManaged +
DesktopManagedConnectionUpdateResult/Receipt types.
Renderer UI (fleet-updates store, about-settings, system.ts ProfileScope
plumbing, i18n) intentionally NOT wired — it belongs to the deferred fleet
rollout UI and follows separately.
Wiring re-implemented against current main; design from #93042 by @andrexibiza
tsc -p apps/desktop clean; electron project 1924/1924 passed (133 files,
incl. 131/131 across the three engine suites); eslint clean on touched files.
The Desktop can drive a full update of a REMOTE SSH instance: claim the
connection (ManagedConnectionUpdateGate pauses dials/mutations while the
update owns it), terminate the owned backend with an identity re-proof
at the signal boundary (terminateOwnedDashboardForUpdate — argv+creation
time re-read in the same remote shell that signals), run the updater
under an update-in-progress marker/mutex on the remote install root,
bootstrap the new backend, and fence its publication so a rollback
cannot leak a half-published serve (fenceManagedSshBootstrapPublication
+ waitForManagedSshBootstrapFence barrier).
Extraction notes (campaign #91277 Phase 4; rollout/canary engine stays
behind per sequencing):
- managed-ssh-update.ts + test: clean cherry-pick (new files).
- windows-remote-lifecycle.ts + test: applied as-is (zero main drift).
- remote-lifecycle.ts: PR hunks stitched AROUND current main's #95532
skew guards and #91668 SIGKILL-escalating cleanupStale, which are
PRESERVED verbatim — the PR's python identity-re-proof termination is
wired for the managed-update path only; connect's stale replacement
keeps main's proven kill. One PR test assertion re-pinned accordingly.
- One PR test fixed: floating coordinator.start() promise whose
rejection IS the contract under test now has an explicit handler
(vitest flagged it as an unhandled rejection).
tsc clean; 131/131 across the three touched electron suites. main.ts
wiring (IPC + deps bag) follows as a separate commit.
Structural fix after five review rounds put four blockers in the same
subsystem. The root cause was representational: consent history is a
sequence of on/off intervals, but it was stored as ONE moving day-stamp
plus a revoked flag. Every fix had to mutate that scalar at exactly the
right moment from exactly the right place, and each round the mutation
was missing from some reachable path (write-once stamp in R3; recorded
inside a loop that never runs when sending is off in R4; dead code
whenever collection was off in R5).
Consent is now recorded as explicit intervals (send_consent_windows) and
eligibility is a pure derivation: a package is sent only when its whole
period falls inside a recorded window. One writer -
reconcile_send_consent - derives window state from an observation of
(config, now). It is idempotent and order-independent, so the wizard,
the relay, and the mid-pass check all call the same function and cannot
disagree; there are no edges to detect and no ordering between writers
to get wrong. The relay reconciles once per process BEFORE the
collection gate, which fixes round-5 D1 (enabled:false made the only
idle-path observer unreachable). The claim reads the table and never
writes it, removing the read-path mutation (D2's rewrite vector).
Timestamp discipline, each rule load-bearing and mutation-tested:
- 'obs' high-water mark: monotonic, advanced only by observations;
confirms an open window forward (last_confirmed_at).
- 'data' high-water mark: advanced only by stored package period_end;
clamps window OPENS so a rolled-back clock cannot slide a window
under refused packages already on disk (round-5 D2).
- A close stamps last_confirmed_at, never "now": consent is asserted
only for observed time, so a hand-edited config with no process
running for 90 days fails closed (round-5 D1 strongest form).
- The gate requires period containment, not period_start >=, so an
intra-day revoke/re-enable holds back the day package (round-5 D3).
- Unlike the day-stamp, a revoke/re-enable cycle no longer destroys the
undelivered backlog from the earlier consented window (round-5 D4).
The redesign was validated BEFORE implementation against all 13
reproduced defect scenarios on a real store; the first two drafts each
failed scenarios in that harness (v1 leaked the unobserved-gap case by
closing at "now"; v2 leaked refused windows by letting data stamps
confirm consent). The harness ships as
tests/hermes_cli/test_shared_metrics_consent_windows.py.
Deleted: OPT_IN_PERIOD_KEY, SEND_REVOKED_KEY, LAST_SEEN_SEND_KEY,
opt_in_period(), record_revoked(), the relay edge detector body, and the
setup wizard's key bookkeeping (~170 lines of transition machinery).
Schema: two additive tables, version deliberately unchanged; verified
against a copy of the real production DB (13 rows intact, reopen no-op).
Also kills round-5's M8 survivor: the seen-exclusion mutation now fails
the suite. New mutation sweep: 8/8 killed, including one vacuous test of
my own this round (obs-mark monotonicity was covered only by
coincidence of the data mark; now pinned directly).
Documented cost: a fresh package waits at most one process start after
its period completes before release (fail-closed direction).
270 tests pass; ruff and windows-footguns clean. Staging E2E re-run
through the interval gate: both packages 202.
Fourth independent review. Two more consent leaks, both reproduced through
the real relay entry point before and after the fix. Both are failures of
my own round-3 fix, which recorded revocation in the wrong place.
BLOCKER 1 - revoking while idle recorded nothing. _record_revocation lived
inside send_pending's loop, but _send_exported_packages returns early when
send is false, before a sender is ever constructed. The dominant case is a
user turning sending off while no pass is running, so the loop that was
meant to observe the revocation could never run. Reproduced: 6 periods
collected during a refused window were transmitted on re-enable.
The window now closes on the observed config EDGE, before the early return.
Last-seen send state is persisted because each hook fires in a fresh
process, so a true->false transition is only visible by comparison. The
rising edge also opens the window explicitly: the sender only runs when
there is something to send, so a user who opts in and out before any
package exists would otherwise have no window for record_revoked to close.
BLOCKER 2 - turning COLLECTION off never recorded revocation. The
not-enabled branch in setup.py force-set send=false and returned without
calling _record_send_consent_change, so `hermes tools` -> disable shared
metrics silently dropped consent while leaving the window open. Same
retroactive release on re-enable. Both consent surfaces now record, and
setup keeps the relay's edge detector in step.
Also, from the same review's mutation sweep:
- the scheme check is now pinned as an allowlist. Replacing the http test
with `if True` survived the entire suite, because every non-http case
targeted a REMOTE host where the loopback branch rejects anyway. Only a
non-http scheme on loopback distinguishes the two. Shipped behaviour was
already correct; nothing guarded it.
- A.3 no longer claims rotation bounds long-term linkability outright.
Measured against 11 real packages: resource is a stable low-entropy
tuple and periods are contiguous across a rotation, so for a RARE
configuration those can bridge windows. The honest claim is that
rotation raises the cost, not that it makes correlation impossible.
Two mutants are documented as unkillable rather than papered over with
tests that only appear to cover them: the _defer clamp is unreachable from
any current caller, and widening the falling-edge check to an
unconditional else is behaviourally equivalent because record_revoked is
idempotent and no-ops without an open window.
An earlier version of the anti-spurious-revocation test could not fail
either - it used a never-consented store, where record_revoked no-ops
regardless. Rewritten to opt in, revoke, re-enable, and then assert that a
steady enabled state does not re-close the reopened window.
259 tests pass. Staging E2E re-run: both packages 202.
On Windows installs where the gateway runs as an SCM service (WinSW,
NSSM, sc.exe create), the existing pause machinery kills the gateway
process directly — and the service wrapper's failure ladder resurrects
it within seconds, re-taking the venv file locks mid-update. The update
then dies partway through dependency sync with access-denied errors.
This extends _pause_windows_gateways_for_update() to detect when a
gateway's process tree is owned by a running SCM service, and to stop
the SERVICE through sc.exe instead of killing the child:
- gateway/status.py: expose service-ownership discovery for gateway
runtimes (find_windows_gateway_services maps validated gateway PIDs
through process ancestry to running SCM service PIDs, with
create-time identity checks against PID reuse).
- hermes_cli/update_cmd.py: stop verified services via sc.exe before
venv mutation and restart them afterward. Stops wait for a stable
SCM 'stopped' state AND for the original descendant processes to
exit (service 'Stopped' is not proof the child released its
handles). Failure to prove ownership, stop a service, or restart it
fails closed; rollback restores attempted services, and rollback
failures are surfaced rather than swallowed.
- Fail-closed throughout: unreadable identities, ambiguous ancestry,
or a service that will not reach a stable state abort the update
before any file mutation.
Complements #37039 (gateway-only concurrent instances no longer abort):
that fix lets the update proceed past the gate; this one makes the
pause actually stick when the gateway is service-supervised.
Note: tests/gateway/test_status.py::TestReadProcessCmdlinePsFallback::
test_ps_fallback_when_proc_unavailable fails on Windows on current main
before this change as well (POSIX ps fallback asserted on a platform
without it); all other touched suites pass (155 passed, 5 skipped).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Salvage adjustments to PR #94392 per review:
- Narrow the supervisor claim to the systemd-VERIFIED path only. The fresh
recovery child now probes 'systemctl --user is-active' after each relaunch;
only an observed-active systemd unit is reported 'verified'. A relaunch that
merely exited 0 is labelled 'relaunch_attempted', never counts as supervisor
coverage, and never clears gateway_fleet_restart_incomplete.
- Serve-owned runtimes (serve/dashboard entries from the spawn ledger, per the
update_inventory serve collector) are no longer silently skipped: the
recovery pass records them (and manual gateways) as skipped-with-reason in
the recovery result and the persisted update receipt.
- Receipt fresh_recovery persists the conservative vocabulary
(requested/verified/relaunch_attempted/failed/skipped); 'succeeded' is gone.
- Added an end-to-end test that drives the real recovery module in a genuinely
fresh interpreter (sitecustomize shim intercepts the grandchild
'gateway restart' and systemctl probes).
Every git fetch that dies mid-transfer (timeout, HTTP 429, dropped
line) strands a tmp_pack_* file in .git/objects/pack, and git never
cleans them. The banner's background update check is the main generator
on flaky lines — several aborted fetches a day — and the reporter's
install accumulated hundreds of files / 6.0 GB over 9 days until the
pack directory corrupted outright and every update check hung or
failed permanently.
clear_stale_tmp_packs() in gitlock.py sweeps tmp_pack_/tmp_idx_/
tmp_rev_/tmp_mtimes_ debris with the exact safety contract the lock
sweep already uses: only files past the 10-minute age floor, never
while any git process runs, never raises, real pack-*.pack/.idx files
untouchable by construction (prefix match). Wired into all three
fetch-adjacent sites: _cmd_update_check, the update apply path, and
the banner's passive check (generator = janitor).
Live E2E: 300 aged tmp_pack files (the reported scale-shape) swept
from a real repo; an in-flight fresh tmp and ancient real packs
survived; fsck clean and a real fetch round-trip succeeded after.
* refactor(clarify): halve the schema (880->436 tok/call) — same rules, half the words
* refactor(clarify): one question field — questions[] is the only advertised shape (single = one-entry array; legacy shape stays handler-accepted)
* refactor(clarify): unadvertise per-question id — response rows already carry question text + order
The repo's own override pinned nanoid@3.3.17 — the exact version
GHSA-2v37-7h3g-55p8 / CVE-2026-67213 flags (custom generators loop
indefinitely on size 0) — so every fresh install and every npm audit
shipped/reported the vulnerable pin no matter what transitives wanted.
3.3.18 is the patched release on the same major. Lockfile re-resolved;
npm audit now reports zero nanoid findings, and 3.3.18's zero-size
generator returns instead of hanging (verified live).
Addresses the upstream-pin quarter of #91931 (mechanism 1 of the
reporter's four); the updater-side skip/verify mechanisms and the
uv.lock staleness half (#91424) remain tracked there.
Addresses the two follow-up notes from review: document that
pre_restart_pids is a bare PID set (not (pid, start_time) pairs), so a
recycled PID from one gateway landing in another's stale record could
still mislabel it as down; and add a companion test asserting a
matching start_time still yields the live/current row.
collect_fleet_versions()'s gateway_state.json fallback path only checked
_pid_exists(pid) to decide whether a recorded gateway was still running.
On Windows, a paused gateway's PID can be recycled by an unrelated
process spawned during the update's own churn (npm/git/python
subprocesses) before the record is refreshed, so the dead gateway's
stale code_sha still gets compared against HEAD and reported STALE for a
PID that no longer belongs to it (#93258).
Switch to runtime_status_pid_is_live(), the existing (pid, start_time)
PID-reuse guard already used elsewhere in gateway/status.py, so a
recycled PID is treated the same as a dead one (DOWN row, or no row, per
the existing rollout-safety rules) instead of a false STALE.
#91277 Phase 2's plan-vs-execution reconciliation (match_runtime_outcomes)
cross-checks every runtime collect_runtime_inventory() saw against
restarted_services / relaunched_profiles / externally_supervised_profiles /
killed_pids — the systemd/launchd restart phase's bookkeeping. That
inventory is cross-platform (control-socket / PID-file based), so it
includes Windows gateways too, but Windows's own pause/resume mechanism
(_pause_windows_gateways_for_update / _resume_windows_gateways_after_update)
never wrote into any of that bookkeeping.
Result: a Windows gateway that was correctly stopped and relaunched by
_resume_windows_gateways_after_update was still classified "unaccounted" by
the reconciliation (the plan saw it and no bookkeeping mentions it) —
report_unaccounted_runtimes() escalates that into sys.exit(1), and in
gateway_mode also writes ".update_exit_code"="1". Every successful
`hermes update` on Windows with a running gateway reported itself as
failed, unconditionally (the sys.exit(1) is not gated to gateway_mode).
_resume_windows_gateways_after_update now records the profiles it
successfully relaunched onto the resume token; _cmd_update_impl merges
that into the shared relaunched_profiles list right before reconciliation
runs. A profile whose relaunch genuinely fails is deliberately left off
the list, so it still surfaces as unaccounted — Windows has no watcher to
recover a failed relaunch, so that escalation is the correct signal.
Regression tests exercise _resume_windows_gateways_after_update directly
(records successes, omits failures) and reproduce the reconciliation-level
bug end to end: the same plan row resolves "unaccounted" without the merge
and "restarted" with it. Mutation-verified: with the fix reverted, three of
the four new tests fail (KeyError on the token / wrong outcome).
Third independent review. Both blockers reproduced against a real store
before and after the fix.
BLOCKER 1 — head-of-line starvation. The claim query is LIMIT 1, and a
package already handled this pass was rejected AFTER the fetch, so
_claim_next returned None and send_pending read that as 'queue empty'.
Any row that sorts first and becomes eligible again mid-pass therefore
terminated the pass. This is reachable normally: a 429 with a short
Retry-After, or a pass outliving the 15-minute failure backoff (a legal
pass runs ~1900s). Measured: 10 of 19 healthy packages silently dropped.
The seen-set is now excluded IN SQL, so None genuinely means no eligible work.
Same scenario now delivers 19 of 19.
BLOCKER 2 — revoking consent leaked once it was re-granted. opt_in_period
was write-once, so packages collected while the user had send: false
still had period_start >= the ORIGINAL opt-in day; re-enabling released
the whole refused window. Reproduced: 5 packages from a 5-day opted-out
window transmitted on re-enable. Turning sending off now closes the
consent window, and the next enabled pass opens a new one from that day.
Recorded both in the setup wizard and in the sender itself, because
config.yaml can be hand-edited where the wizard never sees it.
Also: a send_attempts ceiling (a poisoned head row burned ~160 requests
over 30 days, unbounded), _defer clamps to >= 1s so it cannot write a
past deadline, and the dead skipped_not_due field is removed.
Test-quality fixes, since vacuous tests have been the recurring problem:
- the lease test asserted only 'in the future', passing for a 1s lease;
it now requires the lease to outlast one package's worst legal case
- test_shutdown_joins_the_send_thread grepped getsource for a method
name — a change-detector AGENTS.md rejects — and is now behavioural
- gzip determinism was unguarded: both retries in one pass compress in
the same second, so removing mtime=0 was caught by nothing. Now
compares output across a real second boundary.
All five new regressions are mutation-verified: reintroducing each bug
fails its test. The first attempt-ceiling test SURVIVED its mutation
(the seeded row was excluded by another predicate) and was rewritten to
drive the real loop.
251 tests pass. Staging E2E re-run: both packages 202.
The wrapper now getattr-defaults _processed_message_ts (object.__new__
adapters in sibling suites lack it), and the reaction-guard source pin
reads _handle_slack_message_impl where the production expression lives.
Follow-up for the #95417 salvage, addressing the review finding: the entry
claim closes the unfurl race but a handler that raises mid-enrichment would
hold the claim forever — neither a Slack retry nor a user edit could ever
re-drive the message. _handle_slack_message is now a thin guard around the
impl that releases only claims taken by the failed invocation itself, with a
warning log so swallowed turns are traceable. Pre-existing claims from a
successful turn are never released. Two failure-path tests pin both sides.
Slack emits `message_changed` for a link unfurl carrying a DIFFERENT event ts
than the original message. That ts legitimately misses the `_dedup` check, so
`_processed_message_ts` is the only guard against it becoming a second user
turn -- but it was only populated at the END of `_handle_slack_message`, after
thread context, permalink resolution and file downloads had all awaited.
An unfurl landing inside that window found the guard empty and was promoted to
a duplicate turn: a spurious "Interrupting current task" banner plus the same
answer posted twice.
Production capture (adminbot, 2026-08-22 02:23:30-31Z, channel C0BF1EYUA9H):
02:23:30.718 message ts=1787365409.908499 dedup_hit=False
02:23:30.737 app_mention ts=1787365409.908499 dedup_hit=True
02:23:31.675 message ts=1787365411.012100 dedup_hit=False <- leaked
subtype=message_changed
The original copy was still resolving two Slack permalinks when the unfurl
arrived 957ms later.
Claim the message ts once every filter has passed and the event is certain to
be delivered, before the slow enrichment awaits. Claiming any earlier (right
after the dedup check) also claims messages the handler then discards, which
breaks summoning the bot by editing "@bot" into a previously ignored message
(tests/gateway/test_slack.py::TestMessageRouting::
test_message_edit_with_new_mention_processed).
Eviction logic is extracted to `_remember_processed_message_ts` so both call
sites share one bounded implementation.
Two profiles can hold sessions with the SAME stored id (restored
backups, copied state.dbs, cross-profile imports). mergeSessionPage
keyed rows by bare id, so the twins collapsed into one sidebar row
whose title/activity carry stitched one profile's content onto the
other's route — clicking a row previewing profile A resumed profile B
and wedged on an eternal 'Waking up…' when the mismatched resume never
completed (live-reproduced on main).
- mergeSessionPage: identity + lineage + dedupe keys are now
(profile, id); a kept twin in another profile survives the incoming
page dedupe. Local profile normalize (importing @/store/profile would
be circular).
- Sidebar clicks carry the ROW as the identity: onResumeSession passes
the clicked SessionInfo, and wiring pins the row's own
(connection, profile) as the resume owner via requestSessionResume
before navigating. Untagged rows keep the id-only path.
A backend.lock.json that exists but doesn't match what this build writes
(unknown/future schemaVersion, truncated JSON, missing or foreign
ownershipId, malformed shape) was previously indistinguishable from 'no
lockfile': connect() would spawn a fresh backend on top of it and
overwrite the record, and cleanup paths could drop foreign state —
disarming the #78872 ownership guard exactly when another (e.g. forked)
desktop build shares the remote. readLockfile now returns a skew
sentinel for existing-but-foreign lockfiles; connect() refuses with a
'remote-lockfile-skew' error and a skew warning instead of
reaping/overwriting, disconnect() and cleanupStale() skip entirely.
Refs #95532
The Electron shell boots by fetching / and extracting
window.__HERMES_SESSION_TOKEN__ to authenticate /api/ws
(dashboard-token.ts adoptServedDashboardToken). Headless serve 404'd
every path, so when the renderer's spawn token drifted from the
backend's live token — e.g. hermes update replaced the backend and the
env pin no longer matched — the renderer had no way to adopt the served
token, the WebSocket handshake failed, and the primary window
white-screened (#95575).
Serve a minimal token-only HTML page at the exact root path in
mount_spa()'s headless branch, matching the renderer's extraction regex.
Gate it on app.state.auth_required read at request time: a gated
(non-loopback / remote public_url) serve keeps returning the 404 JSON so
the session token never leaks past the loopback boundary. Every other
path stays 404 JSON — the SPA remains unserved.
Regression tests: TestHeadlessServeTokenPage (3 cases) — verified to
fail against the pre-fix headless branch.
Teknium