Follow-up for salvaged PR #93451: main grew branchStoredSession tests (#93444)
that drive the same hoisted requestGatewayForProfile/ForAgent mocks, so the
not-called assertions in the warm-cache describe saw stale recorded calls.
When switching profiles, useRouteResume() could treat the target profile
gateway opening as a resume command for the old routed session before
React Router commits the /new pathname. The gatewayBecameOpen trigger
was not guarded by the existing freshDraftReady discriminator, unlike
the stuckOnRoutedSession guard on line 137 which already uses it.
Fix: add && !freshDraftReady to the gatewayBecameOpen condition in
shouldResume (line 143). This is consistent with the existing pattern
and preserves reconnect behavior (freshDraftReady=false during normal
reconnects).
Adds a regression test that simulates the exact race: profile switch
clears refs and sets freshDraftReady=true, gateway closes, then profile
B gateway opens while pathname is still /session-a. Asserts no resume
fires.
Clear the profile-door activation lease in a finally block whether the
secondary dial succeeds or rejects. Preserve reconnect scheduling and the
original error propagation.
Add a regression proving a rejected activation can be pruned immediately.
Rebase of #81165 onto post-#87600 main, slimmed to the error-surfacing UX
half as requested - the silent-misroute half already landed via #87600.
- ensureGatewayForProfile: keep scheduleReconnect() on a failed secondary
dial (transient failures still self-heal) but rethrow so the caller can
surface the failure instead of falling through to setActive() with a
closed socket (#81094). openSecondary logs the dial target and rethrows
the ORIGINAL error so reconnectSecondary's message-based fail-stop
classification ("No connection with id", "no longer exists") keeps working.
- profile.ensureGatewayProfile: propagate the rejection to await-callers
(session actions, slash commands surface it in their own flows); the three
fire-and-forget call sites (selectProfile, newSessionInProfile, voice
wiring) get explicit .catch(notifyError) so the failure is always visible.
- Tests: gateway.test.ts pins rethrow-without-activation plus backoff
self-heal; profile-switch-failure.test.ts pins the profile-door
propagation; gateway-shared-remote's pooled-reconnect test updated to
the new contract (reject first, publish once the backend returns).
Verified: tsc --noEmit clean; 65 tests across the gateway*/profile*
suites pass; eslint and prettier clean on touched files.
Treat the explicit local registry source like the legacy profile-only path so per-profile remote overrides still resolve. Keep non-local registry sources connection-scoped and cover both profile selection entry points.
Remember secondary scopes across renderer pool pruning and clear their process-local runtime bindings before a later socket can publish open. This prevents eager session RPCs from racing a respawned profile backend with an ID minted by the previous process.
Review follow-up. The two-phase switch closed the runtime-id leak for a
single switch, but two shapes still broke the wipe→publish ordering:
Overlapping: click A commits (wipes) and waits for its activation behind
the profile-store mutex; click B finishes its dial and wipes too; then A's
activation lands and publishes A AFTER B's destructive wipe, and A's
endGatewaySwitch() dropped the (boolean) barrier while B was mid-commit.
Now the wipe runs INSIDE the serialized section via a `beforeActivate`
commit hook on ensureGatewayAgent, synchronously right before the socket
is activated; the hook re-checks the click revision and declines when
superseded, so a queued-then-superseded switch neither wipes nor
activates. The barrier is token-owned (latest switch wins), and the mutex
wait is a loop so waiters that wake together can't run interleaved.
Stalled: a wedged spawn / ticket mint / IPC (the #93454 class) left the
spinner up, swallowed later clicks on the same source, and — inside
ensureGatewayAgent — latched the mutex and the barrier. Every await is now
bounded (dial, activation, descriptor lookups, setLastUsed) via a shared
withTimeout helper extracted from use-gateway-boot. A commit that timed
out after the new source was already published counts as committed; one
that stalls after the wipe lowers the barrier and repaints the source
that is still active.
Tests: queued-then-superseded switch, dial that never answers (and retry
not swallowed), activation stalled after/before publication, barrier
ownership, commit-hook ordering inside the mutex, bounded descriptor
lookup releasing the mutex. All fail against the previous revision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Sessions sidebar switcher (selectConnection) activated the target
gateway first and wiped session bindings afterwards — across an IPC
round-trip — so route/session effects saw the new source while
$activeSessionId still named the previous backend's runtime id, sent it
to a backend that never minted it, and got "session not found". The
Settings → Gateway apply (softSwitch) never had this problem: it raises
$gatewaySwitching, runs beforeConnectionSwitch and wipes before dialing.
Give both doors one commit point. store/gateway-switch gains
beginGatewaySwitch()/endGatewaySwitch(): barrier up, the registered
machine-context reset, session wipe — in one synchronous step.
useGatewayBoot registers its beforeConnectionSwitch/refreshSessions with
the store and softSwitch goes through the same entry point.
selectConnection becomes two-phase: dial the target WITHOUT activating
(openGatewayAgent → openGatewayForAgent with an activation lease, so a
live-work recompute can't prune it mid-spawn), then beginGatewaySwitch()
and activate the already-open socket with no await between the wipe and
the publication. A dead target fails with the current workspace intact;
a superseded click never activates its target; an activation that fails
after the wipe repaints the still-active source instead of leaving the
sidebar skeleton.
Tests: real-store end-to-end regression (real useGatewayBoot + gateway
registry + selectConnection over fake sockets) — red on the previous
code with the leaked runtime id observed at publication — plus store
ordering/failure-path, commit-point and prune-lease coverage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Step 6 of the shared-metrics exporter, plus a loopback E2E.
_export now triggers an opt-in send pass on a daemon thread. The hook
runs on finish_task — the user's interactive path — so a 30s network
timeout there would be felt directly; the thread keeps that latency off
the caller. A test asserts _export returns in under a second while a
send is deliberately blocked.
At most one pass is in flight per process: a queued second pass would
add nothing, because the next hook fire picks up whatever is still
pending. Shutdown joins the thread for at most two seconds, then lets
it go — the packages remain in SQLite and go out on the next run, so
blocking a user's exit on a slow network is the wrong trade.
Sending is resolved per pass from the profile's own config, so turning
it off takes effect at the next hook fire without a restart.
E2E (tests/hermes_cli/test_shared_metrics_sender_e2e.py): the real
sender against a real HTTPServer on loopback — actual urllib, gzip,
headers and sockets rather than an injected fake. Covers delivery and
sent-state, 400/429/5xx handling, a retry sending byte-identical
bytes, gzip shrinking a realistic 120-metric package and the server
parsing it back, install_id never crossing the wire, the outbox file
staying untouched, and a dead server deferring without raising.
Wiring tests: 12, all negative-space properties — no send without
opt-in, no blocking, no pile-up, no crash propagation.
Steps 4, 5 and 7 of the shared-metrics exporter: the send logic, the
consent gate, and backoff plus multi-process claiming. These arrive
together because the sender is not correct without all three.
Contract handling: 202 marks sent; 400 is permanent and never retried;
429 honours Retry-After (clamped to a day so a bogus value cannot park a
package); 5xx, timeouts and transport errors retry three times in-process
with 1s/5s/25s full-jitter backoff, then defer to a later pass.
Consent is gated on the package's PERIOD, not its creation time. A period
is split across packages created on different days, so a created-at gate
would send a period's tail while dropping its head and silently
undercount the opt-in day — data that looks complete and is wrong. The
opt-in day is recorded once and never moves, so toggling sending off and
on does not re-open the pre-consent backlog.
Rows are claimed in a write transaction, which is what stops two Hermes
processes sharing one database from sending the same package twice.
next_attempt_at persists backoff across restarts, so a hard-down service
is not retried on every task completion.
The body is recomputed from payload_json rather than stored a second
time: json.dumps is deterministic here (verified against the real outbox
— 11 of 11 files reproduce byte-for-byte), and the only mutable input,
the derived identity, is frozen on the row at first attempt. That keeps
retries byte-identical across a salt rotation for ~36 bytes instead of a
duplicate ~11 KB payload.
The outbox directory is never written to or deleted from. A 202 updates
SQLite only, because those files are the user's 30-day local history and
retention already owns their lifecycle.
Tests: 33. Two of them caught real defects in this commit — an
unreadable row aborted the claim transaction and blocked every package
behind it, and the compression assertions were passing through an
injected fake that bypassed the code under test.
Step 3 of the shared-metrics exporter.
The shared-metrics doc commits that a remote exporter 'must not reuse
the persistent local identifier by default'. install_id is therefore
never transmitted: each package carries
HMAC-SHA256(local-only rotation salt, install_id) instead.
Within a 30-day rotation window the value is stable, so distinct
installs remain countable — the first question the data has to answer.
Across windows it changes, bounding long-term linkability. The
derivation is one-way, so the service cannot recover install_id.
The salt lives in telemetry_state next to install_id, so removing the
shared-metrics directory resets both together and the documented reset
behaviour keeps working with no second cleanup path.
Rotation is deliberately not a bare 'age > interval' check: a clock
that jumps backwards must not read as an expired salt, and an
unparseable issued-at reissues instead of raising.
substitute_install_id replaces exactly one field and copies rather than
mutating, so payload schema evolution stays a sender-side concern.
Tests: 19, including that install_id never survives substitution, that
no other field changes, and — the property that keeps retries
contract-compliant — that a package rebuilt from a FROZEN derived id is
byte-stable across a salt rotation while a fresh derivation is not.
Step 1+2 of the shared-metrics exporter.
Config: telemetry.shared_metrics.send (default false) and .endpoint
(default production), resolved by a new shared_metrics_send_config
module. Precedence is HERMES_TELEMETRY_ENDPOINT > config > default; the
env var exists so the live staging E2E never has to mutate a user's
config. send requires enabled and never implies it — that combination
is a misconfiguration the user believes is working, so it logs an ERROR
once per process rather than silently doing nothing. Plaintext
endpoints are refused unless the host is loopback, so a typo cannot
send telemetry in clear text.
Per AGENTS.md, outbound telemetry needs a user-facing opt-in, so
setup_telemetry now prompts for sending as a second, separate question
and force-disables send when collection is turned off.
Storage: six additive nullable columns on package_outbox for send
bookkeeping. The store schema version deliberately does NOT move —
_ensure_schema_in_transaction raises on any version it does not
recognise and has no forward-compatibility branch, so bumping it would
hard-fail an older Hermes, a second profile on an older build, or a
rollback, against the same file. Old readers select named columns and
never SELECT *, so the additions are invisible to them.
Also corrects the two places that promised telemetry is never uploaded
(config_defaults comment and cli-config.yaml.example); leaving them
would make them false privacy statements once sending ships.
Tests: 26 covering config precedence, the enabled/send relationship,
transport safety, fresh-database creation, upgrade from a pre-send
database (rows preserved, version pinned, idempotent), and that the
shipped export query still runs. Mutation-checked: bumping the schema
version fails 5 of them.
The shared-metrics doc states that a future remote exporter 'must not
reuse the persistent local identifier by default' and 'requires a
separate product and privacy decision covering consent, identity scope,
rotation or keyed pseudonymization, reset behavior, retention, and
deletion'.
That exporter is now being built. Appendix A answers each of those six
items before any code lands, so the reasoning is reviewable on its own
and survives the implementation:
- consent is a separate opt-in from collection, gated on the PERIOD a
package covers rather than when it was created (a period is split
across packages made on different days, so a created_at gate would
send a period's tail while dropping its head and silently
undercount the first day)
- the transmitted identifier is HMAC-SHA256(local-only salt,
install_id), never install_id itself
- the salt rotates every 30 days
- reset gives a new remote identity but cannot unsend
- local retention is unchanged; send state does not extend it
- there is no self-service remote deletion, and the user -> derived-id
lookup that would enable one is deliberately not built
A.7 additionally records what the outbox directory IS (the user's local
history, not a send queue) because misreading it would have led to
deleting user data on acknowledgement.
Follow-ups on the #85416 salvage:
- _store_bin_names()/_sibling_names() derive the versioned interpreter name
from the RUNNING interpreter instead of a hardcoded 3.11-3.13 list, with a
sorted python3.* glob fallback for stores/aliases built by a different
Python minor (major bumps, fixtures) — no future version bump can silently
leave an alias resolving back into the versioned uv store.
- _repoint_alias_symlinks unions expected alias names with the versioned
symlinks actually on disk before repointing.
The sidebar row's Tip trigger test derives its fixture from the wall clock:
const startedAt = Math.floor(Date.now() / 1000) - 5 * 60
...
expect(age.getAttribute('aria-label')).toMatch(/^5m, Today at /)
"Five minutes ago" is only today when the run does not straddle local
midnight. Between 00:00 and 00:05 the timestamp falls into the previous
day, formatMessageTimestamp correctly returns the yesterday label, and
the test fails on a day boundary it was never written to exercise:
AssertionError: expected '5m, Yesterday at 11:56 PM'
to match /^5m, Today at /
This is a real CI failure, not a theoretical one - it took down a
check:test:ui shard on an unrelated desktop PR at 00:01 UTC, and it will
do so for any PR whose shard happens to land in that five-minute window.
Pin the clock to local noon before deriving the timestamp so the fixture
can never cross a day boundary. Only Date is faked (toFake: ['Date']),
so the component's own timers - the running arc and the tooltip open
delay - keep running for real; the neighbouring tooltip tests that
advance timers are unaffected. The describe block gains the
useRealTimers teardown its sibling already had.
The production formatter is not changed: rendering "Yesterday at 11:56
PM" for a session started five minutes before midnight is correct, and
the assertion is about the label's composition, not about which day it
names.
The #87196/#87720 conflict resolution kept the bounded-drain helper and
its constant; the windows_only kill-tree test still asserted the dropped
_CUA_INSTALLER_REAP_TIMEOUT name. Same 2-communicate contract, surviving
constant.
Re-enables the routine confirmed-upgrade path on Windows that #95008
deferred wholesale, now that every unattended-hostile surface is closed:
- stdin=DEVNULL (salvaged #79871): upstream's Read-Host consent prompt
can't block a hidden console.
- Bounded post-kill drain (salvaged #87720): a kill-surviving descendant
holding the stdout pipe can't strand the update past its ceiling.
- 120s background ceiling (salvaged #87196): safe now that a legitimate
600s lock wait can't occur on this path.
- NEW lock preflight: upstream's install lock held by a live process ->
skip in ~0s instead of eating its 600s stale-lock window (the actual
11-minute hang observed 2026-08-25; UAC was a red herring — base
install is no-admin by upstream design).
- NEW 5s network preflight: github.com unreachable -> skip immediately.
- Windows unattended runs pass -NoAutoStart, skipping the ONLY
install.ps1 branch that self-elevates (autostart task re-registration).
- Timeout diagnosability: partial installer output is logged on kill so
the next hang names its stage instead of dying silently.
Contract repairs and fresh installs stay interactive-only (SmartScreen /
first-time elevation legitimately need a human).
On Windows, `hermes update` can hang past its own 660s cua-driver timeout
until the user kills an orphaned PowerShell by hand. The timeout ceiling is
not the problem; the code that runs after it is.
`_run_cua_driver_installer` handles `TimeoutExpired` by killing the process
tree and then draining the pipes with a bare `proc.communicate()`. The kill
is best-effort by construction: every `psutil.Error` in `_kill_installer_tree`
is logged at debug level and stepped over, on the reasoning that a partly
killed tree beats none. That is the right call, but it means the drain has to
survive a partial kill, and an unbounded drain does not.
The concrete case is the one reported. `install.ps1` self-elevates through
`Start-Process -Verb RunAs`, so the descendant runs at High integrity and a
medium-integrity `child.kill()` raises `AccessDenied`. The per-child handler
logs it and continues. That survivor is still holding the `stdout=PIPE` write
handle it inherited, so the following `communicate()` waits for an EOF that
arrives only when somebody kills that process manually. A bounded 660s wait
becomes an unbounded one, after the warning has already printed.
Bound the drain instead. A kill that landed closes the pipe immediately, so
this costs nothing on the normal path; a kill that did not costs 15s rather
than forever. The original `TimeoutExpired` is re-raised either way, so the
existing manual re-run hint still prints and the update unwinds. Losing the
tail of a timed-out installer's log is the cheaper half of that trade, and it
is only lost in the case where the run already failed.
The drain deliberately does not close the pipe handles. `communicate()`'s
reader threads are still blocked on them and closing underneath them races;
they are daemon threads, so abandoning them does not hold the interpreter
open.
Both timeout handlers (streaming and captured) now go through one helper.
The streaming child inherits the console rather than a pipe, so it is much
harder to stall there, but the two branches should not drift on a rule this
small.
Tests: 5, in a new `TestInstallerTimeoutDrainIsBounded`. Two fail without the
fix, including the reported scenario end to end (a child kill refused with
`psutil.AccessDenied`, asserting the drain still carries a deadline). The
deadline is asserted as a kwarg rather than by timing, because a test that
proved the hang by hanging would be the same defect wearing a test's name.
Scope note: this does not touch the `stdin` inheritance that lets
`install.ps1`'s `Read-Host` block in the first place. That is #79684 and open
PR #79871 already carries the one-line `stdin=DEVNULL` fix; the two are
independent and neither subsumes the other, since `DEVNULL` cannot unblock a
UAC elevation dialog.
Fixes#87703
When `hermes update` runs the cua-driver installer non-interactively,
stdout is captured (PIPE) but stdin is inherited from the parent process.
The upstream install.ps1 prompts `[Y/n]` when it detects a stale
cua-driver daemon, but the prompt goes to captured stdout (invisible)
while stdin waits for input — causing an 11-minute hang until the
watchdog timeout.
Fix: redirect stdin to subprocess.DEVNULL on the non-verbose path so the
installer reads EOF immediately instead of blocking. The installer exits
quickly with a non-zero code, and the existing handler shows the manual
re-run hint with the installer's captured output.
Fixes#79684
profiles.list opened every profile state.db as a writable SessionDB,
which waits out write-lock patience while that profile's backend is
mid-turn. The desktop RPC timed out and Bot Mode's infinite React
Query retry kept the sidebar on a spinner.
Inspect those DBs read-only and bound roster retries so names still
paint.
Two follow-ups on top of the #86391 salvage:
- check_macos_tcc_grants: a certificate-anchored DR (hermes desktop
--setup-tcc-identity, or a notarized release) now reports as stable in its
own class instead of falling into the identifier-pinned message; the
identifier-pinned message points at --setup-tcc-identity for the strongest
anchor.
- collect_relay_plugin_cutover_findings: only merge process-level env vars
when env_map is None (run_doctor's live path). An explicit env_map is a
complete environment description — merging os.environ on top made
report_deprecated_config_and_env non-hermetic on boxes exporting legacy
relay vars (10 findings vs the expected 2 in
test_report_does_not_count_as_blocking_issue).
Review feedback (AI review on #86391):
- guard _macos_desktop_dr subprocess.run against TimeoutExpired/FileNotFoundError
so a hanging codesign degrades to the unreadable-DR warning, never crashing
the doctor run (matches the file's existing subprocess guard pattern)
- select the desktop bundle by newest-mtime across release/mac-*/Hermes.app,
matching _desktop_packaged_executable, instead of a fixed arch order
- note the cdhash-match proxy assumption at the classification site
- document why /Applications/Hermes.app (Hermes-Setup launcher,
com.nousresearch.hermes.setup, certificate-anchored) is deliberately not probed
- extend the repair hint to cover per-service resets
- regression tests: codesign timeout and missing-codesign paths
GPT-OSS review: an empty codesign output would fall through to the
'stable identity' branch and false-positive. Guard with and
cover the empty-string case. Flash review: the non-macOS silence test
mocked the bundle to None, so it never exercised the platform guard;
mock a real path instead.
TCC keys permission grants to the app's code-signing requirement. Grants
made to pre-#73681 builds carry a cdhash-pinned requirement that no
longer matches the rebuilt bundle, so macOS re-prompts on every capture
even though the System Settings toggle shows ON — and the modern prompt
has no Allow button, so users cannot complete the one-time re-grant.
- hermes doctor: new check_macos_tcc_grants() reports the desktop
bundle's DR class (cdhash-pinned → grants reset on every update;
identifier-pinned → stable) and prints the exact stale-grant repair
(tccutil reset, toggle ON, fully quit & relaunch).
- hermes update: after a successful update on macOS with a desktop app
installed, print the one-line stale-grant guidance.
- docs: desktop.md no longer claims grants persist 'out of the box';
documents the one-time re-grant for pre-fix grants.
Closes#86385
The heal decision is extracted into should_heal_self_marker_refusal()
so the contract is testable: heal ONLY on exit 2 + a marker naming this
process. Five tests pin it — self-owned heals, foreign owner (real live
sibling process) never heals, missing/garbage marker never heals,
non-exit-2 never heals, and the full acquire -> refuse -> drop-claim ->
retry-precondition lifecycle with a real UpdateMarkerGuard.
windows-rust-e2e.yml mirrors the wine2e pattern: fires only on
wine2e-rust/** pushes, runs the crate's cargo test --lib on
windows-latest (the shipping platform). The permanent Linux lane stays
authoritative for the unix-gated pipe-drain fixtures.
Main's live_marker_owner has adopted self-owned markers since
160586ff8/dbc2a9c8e (#74761), so the cherry-picked comment's claim that
it 'maps self-ownership to None' is stale. The raw read is still the
right tool — the heal needs the single fact 'does the marker name our
PID' without age/liveness policy folded in.
An updater binary spawns 'hermes update' while holding the update marker
with its own PID. A checkout that predates the HERMES_UPDATE_HANDOFF_PID
env fix (8c76fe19f) and the ancestor-pid fallback runs its pre-pull
update_lock.py, reads that marker as a live foreign update, and exits 2
— and the updater deliberately skips its retry for exit 2, so the
refusal loops forever: the update being refused is the one that ships
the fix, and the failure screen's Retry re-enters the same state.
Detect the case with a raw marker read (live_marker_owner deliberately
maps self-ownership to None, so it cannot answer this), drop our own
claim, and retry the child once with the marker absent. The guard
re-removes on Drop (idempotent) and the desktop is already gone at this
point, so nothing races the brief marker-free window.
Fixes#75788
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Teknium