Addresses review feedback on the regression test. The test previously parsed
the update_cmd.py AST to assert that each auto-restore call site cleared the
destination's sidecars before copying. That bound the fix to source text rather
than behaviour, and would break on unrelated refactors.
Extract _restore_state_db_from_snapshot(state_path, snap_state), which performs
the clear -> copy -> verify sequence as one unit and returns whether the
restored file passes its integrity check. Both auto-restore paths now call it,
so the ordering is guaranteed by construction instead of by inspection, and the
two byte-identical blocks collapse to a single call each.
The regression test now exercises that helper directly against a database that
still owns a hot WAL: removing the clear from inside the helper fails it with
201 rows where 400 were expected, so the guard remains bound to behaviour.
Also covers the two failure modes the callers already handle: a snapshot that
does not survive the copy returns False, and a missing snapshot raises OSError.
The post-update integrity guard (#68474) restores state.db from a pre-update
quick snapshot with a plain shutil.copy2, at both auto-restore sites: the
ZIP-update path in _update_via_zip and the git-pull path in _cmd_update_impl.
The snapshot image is produced by backup._safe_copy_db through sqlite3.backup(),
so it is already checkpointed and owns no WAL. That is precisely why
backup._EXCLUDED_SUFFIXES refuses to ship -wal/-shm/-journal inside a snapshot:
"shipping the live WAL / shared-memory / rollback-journal alongside would pair a
fresh snapshot with stale sidecar state and produce a torn restore on the next
open." The backup side excludes sidecars for that reason; the restore side never
cleared the destination's.
copy2 replaces only the main database file. A state.db-wal belonging to the old,
corrupt database survives the copy and is replayed over the fresh image on the
next open. The restored file then passes PRAGMA integrity_check while serving
the discarded database's contents, so _restored_ok reports valid and the CLI
prints "Auto-restored from snapshot" over data the user has lost. The first
subsequent checkpoint folds the stale WAL in permanently.
A hot -wal is reachable at exactly this moment: a second Hermes holder the
updater's drain did not stop, or the crash that corrupted state.db in the first
place, which is the very trigger for this code path.
Clearing the destination's sidecars is safe here specifically -- they belong to a
database the caller has already declared corrupt and is about to discard.
Contrast preflight_db_writability, which correctly refuses to delete a live WAL.
Reproduced against real SQLite: restoring a 400-row snapshot over a database
with a hot WAL yields 0 of the 400 rows, all 201 visible rows coming from the
old WAL, with integrity_check reporting ok.
`restore_quick_snapshot` swaps only the main `.db` file: copy to a temp name,
`unlink()` the destination, `move()` the temp into place. The destination's
`state.db-wal` / `state.db-shm` are left untouched.
The snapshot is a checkpointed `sqlite3.backup()` image (`_safe_copy_db`) that
owns no WAL, so a leftover `-wal` describes the database that was just
unlinked. An ungracefully killed gateway (SIGKILL / OOM / power loss /
container stop) strands exactly those sidecars -- and that is precisely when an
operator reaches for a snapshot restore. SQLite then replays that foreign WAL
over the restored file on the next open.
The module already documents this hazard for the backup archive:
`_EXCLUDED_SUFFIXES` says shipping sidecars next to a `backup()` copy "would
pair a fresh snapshot with stale sidecar state and produce a torn restore on
the next open." The same reasoning was never applied to the restore
destination.
Reproduced against the real `create_quick_snapshot` / `restore_quick_snapshot`:
origin/main restore_returned=True sidecars_left=['-wal','-shm']
integrity=*** in database main *** Tree 2 page 295:
btreeInitPage() returns error code 11 rows=DatabaseError
patched restore_returned=True sidecars_left=none
integrity=ok rows=2000
The restore reports success and returns True while leaving the database
malformed. `SessionDB` then fails to open on every subsequent start, and
`repair_state_db_schema` only rebuilds `sqlite_master`/FTS -- the damage is in
the `sessions`/`messages` b-trees, so it re-raises. The pre-restore `state.db`
is already unlinked, and because the stale `-wal` survives the restore,
re-running it corrupts the file again identically: the documented last-resort
recovery is wedged.
Every other DB-move site in the repo already handles sidecars --
`hermes_state.py:742` and `:1698`, `hermes_cli/kanban_db.py:1759`,
`hermes_cli/session_recovery.py:60`. `restore_quick_snapshot` was the outlier.
The two `hermes update` auto-restore sites (`hermes_cli/main.py`) have the same
shape and run unattended: they `copy2` the snapshot over `state.db` with the
sidecars still present, then `verify_sqlite_integrity` fails and prints
"Auto-restore FAILED -- restored copy also failed integrity". Same fix.
The last piece of the macOS permissions campaign (#52010 follow-up): macOS
prompts per-folder (Desktop, then Downloads, then Documents, ...) as the
agent touches each one — a drip-feed of dialogs on first use. ONE Full Disk
Access grant covers all of them permanently, and with the stable signing
identities merged this week it survives every update. Nothing in Hermes
taught users that.
- hermes doctor: check_macos_full_disk_access() — prompt-free probe (the
FDA-gated TCC db dir returns EPERM without a dialog; TCC only prompts on
protected-CATEGORY paths), reports granted state or prints the one-switch
setup with the Privacy_AllFiles deep link.
- hermes setup: same probe at the end of onboarding — the moment users are
primed to do system setup — silent when already granted, indeterminate,
or non-macOS.
- docs: desktop.md TCC section now leads with the one-switch guidance.
- 7 tests (granted / denied / indeterminate / non-macOS, both surfaces).
Follow-up on the cherry-picked #82644: the (st_dev, st_ino) snapshot of
site-packages does not survive contact with ext4 — a recreated directory
routinely REUSES the freed inode, so the exact reported repro
(rm -rf venv && uv venv) passed the intact check undetected. Proven live
during salvage: the E2E's replaced venv came back with the identical
inode and runtimeIntact stayed true.
Primary identity is now a marker file written into site-packages when
the SSH owner nonce activates: it deterministically dies with the old
tree on ANY replacement (same or different Python version) and survives
in-place pip/uv installs (no false stales). The stat snapshot remains as
the fallback for read-only site-packages, where it still catches
cross-device moves and version-bump path changes. Client classifier
semantics unchanged: only an explicit runtimeIntact:false rejects, so
older remotes stay compatible.
Three new tests: recreated-venv-with-reused-inode (the live-proven
case), in-place-install stays intact, read-only fallback arms the stat
tier.
Folds review findings: surface failed_deletes in CLI and gateway
/rollback output (new gateway.rollback.failed_deletes locale key, 17
locales), emit skipped_oversize on the nothing-to-restore early return
too, document all three report keys in the restore() docstring, and pin
the failed_deletes contract from both sides in tests.
mount_spa's WEB_DIST.exists() check ran ONCE at mount time: a long-lived
'hermes dashboard --skip-build' that survived a git pull (or launched
before the first build) installed a permanent no_frontend catch-all and
answered 404 'Frontend not built' on every route forever — even after
npm run build completed. Remote Desktop clients saw ERR_EMPTY_RESPONSE.
The missing-dist branch is now reserved for the headless-serve contract
only. The SPA routes mount unconditionally and already cope with a
missing dist per-request (_serve_index returns the same 404 JSON when
index.html is unreadable; the /assets mount gains check_dir=False so
StaticFiles 404s instead of raising at mount). The dashboard recovers
the moment a build lands on disk — no restart needed.
Direction from #82666 by @codexbt (his PR's rebase dropped the product
hunk, leaving only the test; the test is cherry-picked as-is and this
commit restores the behavior it pins, adapted to the current mount_spa
shape: headless guard preserved, per-request recovery instead of a
per-request exists() check).
huklaa's review: a failed fetch makes origin/main stale, so a stale ref
cannot prove *currentness* (rev-list 0 is inconclusive), but a stale
positive count is still sound evidence an update exists. On fetch
failure, compute the stale behind-count and return it when > 0;
otherwise return None (inconclusive) and still skip the cache write.
Regression tests:
- fetch failure + stale rev-list 0 -> None (not 'up to date')
- fetch failure + stale rev-list 5 -> 5 (update evidence preserved)
- fetch failure + rev-list error -> None
When _check_via_local_git's git fetch fails (timeout, offline, DNS),
the code silently fell through to compare HEAD against the stale
origin/main tracking ref, which can report 0 (up to date) even when
upstream has moved forward. Combined with the 6-hour cache in
check_for_updates, a single fetch failure could suppress update
notifications for days — the exact symptom in #82166 where the daily
cron reported 'up to date' for 4 days after v0.20.0 was released.
Two fixes:
1. _check_via_local_git now detects fetch failure (returncode != 0 or
exception) and returns None instead of falling through to stale
refs. The caller treats None as 'check could not run' rather than
'up to date'.
2. check_for_updates no longer caches None results. Previously, a
None from a failed check was cached for 6 hours, suppressing
retries until the cache expired. Now only conclusive results
(0 or >=1) are cached, so the next check attempt runs immediately
on the next call.
Added regression tests:
- test_check_via_local_git_fetch_failure_returns_none
- test_check_for_updates_does_not_cache_none
Follow-up to the salvaged #95207 fix, completing the misreport bug class:
- restore() now also drops delete_targets whose unlink failed (OSError
swallowed) from restored_files — the sibling of the kept-oversize
misreport the salvaged fix closed.
- /rollback output in the CLI (cli_commands_mixin) and gateway
(slash_commands + gateway.rollback.kept_oversize locale key in all 17
catalogs) now tells the user which files were kept because the size
cap excluded them from every checkpoint; previously the file was
correctly preserved but the user got no notice it was not reverted.
- Regression test for the failed-unlink misreport.
Integration fixups so #82529's generation signing and #95131's interpreter
anchor cover each other's gaps (without these, each fix leaves the other's
rotation path broken):
- macos_tcc_anchor store detection now recognizes .hermes-runtime/python/
generation-* stores: repair_vulnerable_runtime() rebuilds the venv against
a generation interpreter, replacing the anchored bin/python with a fresh
symlink — previously the anchor then read 'not uv-managed' and NEVER
re-anchored, so every SQLite CVE repair silently orphaned terminal TCC
grants (the exact #82427 scenario, path-keyed).
- _install_anchor signs the anchor copy with the same identifier-pinned DR
(via managed_uv._macos_sign_managed_python) before it goes live: copy2
carries the source build's cdhash-based signature, so an unsigned refresh
would still change the stored csreq on every patch bump/repair despite
the stable path. Best-effort, never blocks the anchor.
- Tests: generation-store recognition + repair-generation anchoring +
sign-on-install call (sabotage-verified: dropping the generation root
marker fails both new tests).
Follow-up on the cherry-picked #90030: candidate.resolve() breaks the fix
on the standard Linux venv layout, where venv/bin/python is a symlink to
the base interpreter. Resolving makes the venv python compare equal to
the dependency-less base (so the swap never happens), and returning the
resolved target would spawn the bare base interpreter, bypassing
pyvenv.cfg — the fix would silently not fix#90026 on the exact platform
it was reported from. Compare and return normalized UNRESOLVED paths:
the venv path IS the interpreter's identity. Adds the symlink-layout
regression test; live-E2E'd with a real dependency-less base runtime.
Under an SSH remote backend the web server is launched by running the uv
BASE interpreter with the venv's site-packages injected into sys.path at
startup, so sys.executable is a dependency-less python. Detached
dashboard actions spawned from it (Update now, restart, anything routed
through _spawn_hermes_action) inherited neither the injected path nor a
PYTHONPATH and died on the first third-party import — 'Update now'
always failed instantly with ModuleNotFoundError: No module named 'yaml'
while 'hermes update' from the venv worked (#90026).
_dashboard_spawn_executable now prefers the install's own venv
interpreter (venv/bin/python, venv/Scripts/python.exe) when it differs
from sys.executable, resolving the same dependency set the venv launcher
provides. Same-interpreter launches return sys.executable unchanged,
preserving the Windows console-ownership behavior verbatim, and layouts
without an install venv keep the old fallback.
Hardening on top of the TCC daemon-identity salvage:
- _validate_cua_driver_app_signature: codesign -dv gate requiring EXACT
Identifier=com.trycua.driver and the official team (4YEC26S9KF) before
/usr/bin/open hands the bundle to LaunchServices — the identity fix must
not double as a launcher for arbitrary/impostor bundles (suffixed
identifiers and wrong teams rejected; unsigned dev builds only via
computer_use.allow_unsigned_driver: true in config.yaml).
- _resolve_cua_driver_app_path: derive the bundle ONLY from the resolved
driver binary — the /Applications fallback could launch a DIFFERENT
install than the manifest resolved.
- open -n -g: don't activate/steal focus when launching the daemon.
- 7 new tests incl. sabotage-verified exact-match assertions.
Grafted from #76433's review direction (@Chadmc9889's original fail-closed
validation requirement).
Co-authored-by: Chadmc9889 <Chadmc9889@users.noreply.github.com>
Second independent review found the lease fix incomplete. Reproduced
each finding before fixing.
BLOCKER — the batch lease expired mid-pass. _claim took up to 20 rows
under ONE shared lease, but a single package can legally consume ~96s
(three 30s timeouts plus 1s+5s backoff), so a full batch runs ~1900s
against a 180s lease. Later rows' leases expired while this pass still
held them, and another process re-sent them. Reproduced: 192s elapsed,
pkg-2 POSTed twice.
Packages are now claimed ONE AT A TIME, immediately before being sent,
so a lease only has to cover the package actually in flight. Verified:
same scenario now sends each package exactly once.
HIGH — revoking consent did not stop a running pass. The runtime read
send consent once before starting the thread, so a pass could keep
transmitting for minutes after a user set send: false, contradicting
the documented promise that it 'stops transmission immediately'.
Consent is now re-read before every package and fails CLOSED if it
cannot be established.
MEDIUM — all non-429 4xx were treated as permanent, discarding data.
403 is the ingest service's own origin guard: a Transform Rule or edge
misconfiguration would have permanently dropped every package sent
during the incident. Only 400 (malformed envelope) and 413 (over the
1 MiB cap) are terminal now; everything else retries.
MEDIUM — valid JSON that is not an object blocked the whole queue.
json.loads('["a"]') succeeds, then .get() raised AttributeError inside
the claim transaction, rolling it back and starving every healthy
package behind it. Payload shape and install_id are now validated, and
an unusable row is rejected individually.
LOW — the clock-rollback comment and test name claimed the opposite of
the code. The behaviour is right (a future issued_at means the recorded
age is untrustworthy, so reissue); the wording is now honest about it.
LOW — removed the stale HERMES_TELEMETRY_ENDPOINT reference left in
config_defaults after the override was deleted.
247 tests pass (was 234). Staging E2E re-run: both packages 202.
Independent review found the claim mechanism did not work. Reproduced
against the real store: two senders POSTed the same package.
The claim wrote next_attempt_at = now, but selection requires
next_attempt_at <= now, so a concurrent pass matched the same row
immediately. It now writes a LEASE INTO THE FUTURE
(_CLAIM_LEASE_SECONDS), which is what actually excludes another pass,
and expires by itself if a process dies mid-send. _mark is additionally
guarded on send_state so a straggler whose lease lapsed cannot
overwrite a completed send back to pending.
The old concurrency test could not fail: it raised AssertionError from
inside a transport, and _send_one catches every exception as a
retryable transport error. It now records what the second pass saw.
Also from review:
- shutdown() never joined the send thread; the join was only wired into
deactivate(). A short-lived CLI therefore killed an in-flight send at
exit, on the only cadence this feature has.
- Removed HERMES_TELEMETRY_ENDPOINT. AGENTS.md reserves HERMES_* for
secrets, and a behavioural override here was a consent hazard: an
inherited variable could silently redirect telemetry a user agreed to
send to Nous. The staging E2E writes the endpoint into its throwaway
profile instead, which also exercises the real config path.
- Added the shared-metrics toggle that AGENTS.md requires
as the third opt-in surface, delegating to the setup prompt so the
consent rules stay in one place.
- Non-429 4xx (401/403/404/413/422) are now permanent. Only 400 was,
so a wrong path or oversized body retried every 15 minutes for 30
days until retention pruned it.
- The opt-in day is stamped when the user consents, not on the first
send pass, which silently dropped the opt-in day whenever the next
export crossed midnight UTC.
- gzip now uses mtime=0. The embedded timestamp made two sends of one
package differ on the wire, so the 'byte-identical retry' E2E was
comparing parsed bodies and could not have caught it. It now compares
raw request bytes.
- Reconciled the three stale claims in relay-shared-metrics.md that
said no remote-delivery path exists.
233 tests pass (was 213). Staging E2E re-run through the config path:
both packages 202, and the service logged both objects written to S3.
The legacy ad-hoc fallback signed and verified successfully but still
fell through to return False, contradicting the fixup's documented
contract. The success witness codified the contradiction. Return True
on the verified success path; the caller ignores the return value, so
no behavior change beyond the contract correction.
Addresses round-2 review feedback on #90961. The previous commits
scoped the keychain deletion to the legacy ad-hoc fallback, but the
reviewer correctly held the blocker: the fallback ran codesign with
check=False, ignored the result, and unconditionally deleted 'Hermes
Safe Storage' — permanently orphaning gateway and native OAuth
credentials even when signing failed or a configured identity had
failed and routed into the fallback.
This commit removes the deletion entirely:
- _desktop_macos_reset_keychain_safe_storage is gone; no code path
touches the keychain item anymore.
- The legacy fallback now checks the codesign result and runs
codesign --verify --deep --strict; any failure leaves the item
untouched and prints a warning.
- The keychain prompt after an ad-hoc re-sign is recoverable
(Always Allow updates the ACL partition list and preserves the
key); deletion is not. The durable proof-carrying migration
belongs in Electron (safeStorage can read the old key) and is
tracked as a follow-up.
Tests: 4 witnesses (stable path, default no-config success, fallback
failure, fallback success) all mutation-verified against both the
deletion regression and the ignored-codesign-result regression.
The previous commit deleted the 'Hermes Safe Storage' keychain item after
every successful re-sign, including the stable certificate-anchored
identity path. On that path the designated requirement is stable across
rebuilds, so after the first launch under the new identity the keychain
ACL already matches; deleting the item on every update permanently
orphaned gateway-token and native-OAuth credentials that were working
fine (both are safeStorage-backed: electron/main.ts connection config
and native-oauth-tokens.json).
Addresses review feedback on #90961:
- Rename _desktop_macos_update_keychain_acl -> _desktop_macos_reset_keychain_safe_storage (it deletes, it does not update an ACL).
- Only invoke it on the legacy ad-hoc fallback path, where every rebuild
produces a new cdhash so the ACL can never match and the alternative
is a recurring prompt. The trade-off (re-enter credentials once per
update) is documented; the durable fix is a stable signing identity.
- Add regression tests: stable path must NOT reset, ad-hoc fallback MUST.
The self-updater rebuilds the desktop app locally via electron-builder after
every update (4aa9f738ce). On macOS, the rebuilt app gets ad-hoc signed,
producing a different cdhash than the original CI-signed build. macOS ties
the 'Hermes Safe Storage' keychain item's ACL to the code signature, so
the new signature doesn't match → macOS re-prompts for keychain access on
every launch.
After re-signing, delete the existing keychain item so Electron recreates
it with the correct ACL for the newly-signed app on next launch. The
trade-off: previously encrypted tokens become unreadable (the user
re-enters the gateway token once), but the keychain prompt stops appearing
on every launch.
The delete-generic-password command doesn't require reading the secret
(no ACL check), so it runs without prompting.
Step 6 of the shared-metrics exporter, plus a loopback E2E.
_export now triggers an opt-in send pass on a daemon thread. The hook
runs on finish_task — the user's interactive path — so a 30s network
timeout there would be felt directly; the thread keeps that latency off
the caller. A test asserts _export returns in under a second while a
send is deliberately blocked.
At most one pass is in flight per process: a queued second pass would
add nothing, because the next hook fire picks up whatever is still
pending. Shutdown joins the thread for at most two seconds, then lets
it go — the packages remain in SQLite and go out on the next run, so
blocking a user's exit on a slow network is the wrong trade.
Sending is resolved per pass from the profile's own config, so turning
it off takes effect at the next hook fire without a restart.
E2E (tests/hermes_cli/test_shared_metrics_sender_e2e.py): the real
sender against a real HTTPServer on loopback — actual urllib, gzip,
headers and sockets rather than an injected fake. Covers delivery and
sent-state, 400/429/5xx handling, a retry sending byte-identical
bytes, gzip shrinking a realistic 120-metric package and the server
parsing it back, install_id never crossing the wire, the outbox file
staying untouched, and a dead server deferring without raising.
Wiring tests: 12, all negative-space properties — no send without
opt-in, no blocking, no pile-up, no crash propagation.
Steps 4, 5 and 7 of the shared-metrics exporter: the send logic, the
consent gate, and backoff plus multi-process claiming. These arrive
together because the sender is not correct without all three.
Contract handling: 202 marks sent; 400 is permanent and never retried;
429 honours Retry-After (clamped to a day so a bogus value cannot park a
package); 5xx, timeouts and transport errors retry three times in-process
with 1s/5s/25s full-jitter backoff, then defer to a later pass.
Consent is gated on the package's PERIOD, not its creation time. A period
is split across packages created on different days, so a created-at gate
would send a period's tail while dropping its head and silently
undercount the opt-in day — data that looks complete and is wrong. The
opt-in day is recorded once and never moves, so toggling sending off and
on does not re-open the pre-consent backlog.
Rows are claimed in a write transaction, which is what stops two Hermes
processes sharing one database from sending the same package twice.
next_attempt_at persists backoff across restarts, so a hard-down service
is not retried on every task completion.
The body is recomputed from payload_json rather than stored a second
time: json.dumps is deterministic here (verified against the real outbox
— 11 of 11 files reproduce byte-for-byte), and the only mutable input,
the derived identity, is frozen on the row at first attempt. That keeps
retries byte-identical across a salt rotation for ~36 bytes instead of a
duplicate ~11 KB payload.
The outbox directory is never written to or deleted from. A 202 updates
SQLite only, because those files are the user's 30-day local history and
retention already owns their lifecycle.
Tests: 33. Two of them caught real defects in this commit — an
unreadable row aborted the claim transaction and blocked every package
behind it, and the compression assertions were passing through an
injected fake that bypassed the code under test.
Step 3 of the shared-metrics exporter.
The shared-metrics doc commits that a remote exporter 'must not reuse
the persistent local identifier by default'. install_id is therefore
never transmitted: each package carries
HMAC-SHA256(local-only rotation salt, install_id) instead.
Within a 30-day rotation window the value is stable, so distinct
installs remain countable — the first question the data has to answer.
Across windows it changes, bounding long-term linkability. The
derivation is one-way, so the service cannot recover install_id.
The salt lives in telemetry_state next to install_id, so removing the
shared-metrics directory resets both together and the documented reset
behaviour keeps working with no second cleanup path.
Rotation is deliberately not a bare 'age > interval' check: a clock
that jumps backwards must not read as an expired salt, and an
unparseable issued-at reissues instead of raising.
substitute_install_id replaces exactly one field and copies rather than
mutating, so payload schema evolution stays a sender-side concern.
Tests: 19, including that install_id never survives substitution, that
no other field changes, and — the property that keeps retries
contract-compliant — that a package rebuilt from a FROZEN derived id is
byte-stable across a salt rotation while a fresh derivation is not.
Step 1+2 of the shared-metrics exporter.
Config: telemetry.shared_metrics.send (default false) and .endpoint
(default production), resolved by a new shared_metrics_send_config
module. Precedence is HERMES_TELEMETRY_ENDPOINT > config > default; the
env var exists so the live staging E2E never has to mutate a user's
config. send requires enabled and never implies it — that combination
is a misconfiguration the user believes is working, so it logs an ERROR
once per process rather than silently doing nothing. Plaintext
endpoints are refused unless the host is loopback, so a typo cannot
send telemetry in clear text.
Per AGENTS.md, outbound telemetry needs a user-facing opt-in, so
setup_telemetry now prompts for sending as a second, separate question
and force-disables send when collection is turned off.
Storage: six additive nullable columns on package_outbox for send
bookkeeping. The store schema version deliberately does NOT move —
_ensure_schema_in_transaction raises on any version it does not
recognise and has no forward-compatibility branch, so bumping it would
hard-fail an older Hermes, a second profile on an older build, or a
rollback, against the same file. Old readers select named columns and
never SELECT *, so the additions are invisible to them.
Also corrects the two places that promised telemetry is never uploaded
(config_defaults comment and cli-config.yaml.example); leaving them
would make them false privacy statements once sending ships.
Tests: 26 covering config precedence, the enabled/send relationship,
transport safety, fresh-database creation, upgrade from a pre-send
database (rows preserved, version pinned, idempotent), and that the
shipped export query still runs. Mutation-checked: bumping the schema
version fails 5 of them.
Follow-ups on the #85416 salvage:
- _store_bin_names()/_sibling_names() derive the versioned interpreter name
from the RUNNING interpreter instead of a hardcoded 3.11-3.13 list, with a
sorted python3.* glob fallback for stores/aliases built by a different
Python minor (major bumps, fixtures) — no future version bump can silently
leave an alias resolving back into the versioned uv store.
- _repoint_alias_symlinks unions expected alias names with the versioned
symlinks actually on disk before repointing.
Re-enables the routine confirmed-upgrade path on Windows that #95008
deferred wholesale, now that every unattended-hostile surface is closed:
- stdin=DEVNULL (salvaged #79871): upstream's Read-Host consent prompt
can't block a hidden console.
- Bounded post-kill drain (salvaged #87720): a kill-surviving descendant
holding the stdout pipe can't strand the update past its ceiling.
- 120s background ceiling (salvaged #87196): safe now that a legitimate
600s lock wait can't occur on this path.
- NEW lock preflight: upstream's install lock held by a live process ->
skip in ~0s instead of eating its 600s stale-lock window (the actual
11-minute hang observed 2026-08-25; UAC was a red herring — base
install is no-admin by upstream design).
- NEW 5s network preflight: github.com unreachable -> skip immediately.
- Windows unattended runs pass -NoAutoStart, skipping the ONLY
install.ps1 branch that self-elevates (autostart task re-registration).
- Timeout diagnosability: partial installer output is logged on kill so
the next hang names its stage instead of dying silently.
Contract repairs and fresh installs stay interactive-only (SmartScreen /
first-time elevation legitimately need a human).
On Windows, `hermes update` can hang past its own 660s cua-driver timeout
until the user kills an orphaned PowerShell by hand. The timeout ceiling is
not the problem; the code that runs after it is.
`_run_cua_driver_installer` handles `TimeoutExpired` by killing the process
tree and then draining the pipes with a bare `proc.communicate()`. The kill
is best-effort by construction: every `psutil.Error` in `_kill_installer_tree`
is logged at debug level and stepped over, on the reasoning that a partly
killed tree beats none. That is the right call, but it means the drain has to
survive a partial kill, and an unbounded drain does not.
The concrete case is the one reported. `install.ps1` self-elevates through
`Start-Process -Verb RunAs`, so the descendant runs at High integrity and a
medium-integrity `child.kill()` raises `AccessDenied`. The per-child handler
logs it and continues. That survivor is still holding the `stdout=PIPE` write
handle it inherited, so the following `communicate()` waits for an EOF that
arrives only when somebody kills that process manually. A bounded 660s wait
becomes an unbounded one, after the warning has already printed.
Bound the drain instead. A kill that landed closes the pipe immediately, so
this costs nothing on the normal path; a kill that did not costs 15s rather
than forever. The original `TimeoutExpired` is re-raised either way, so the
existing manual re-run hint still prints and the update unwinds. Losing the
tail of a timed-out installer's log is the cheaper half of that trade, and it
is only lost in the case where the run already failed.
The drain deliberately does not close the pipe handles. `communicate()`'s
reader threads are still blocked on them and closing underneath them races;
they are daemon threads, so abandoning them does not hold the interpreter
open.
Both timeout handlers (streaming and captured) now go through one helper.
The streaming child inherits the console rather than a pipe, so it is much
harder to stall there, but the two branches should not drift on a rule this
small.
Tests: 5, in a new `TestInstallerTimeoutDrainIsBounded`. Two fail without the
fix, including the reported scenario end to end (a child kill refused with
`psutil.AccessDenied`, asserting the drain still carries a deadline). The
deadline is asserted as a kwarg rather than by timing, because a test that
proved the hang by hanging would be the same defect wearing a test's name.
Scope note: this does not touch the `stdin` inheritance that lets
`install.ps1`'s `Read-Host` block in the first place. That is #79684 and open
PR #79871 already carries the one-line `stdin=DEVNULL` fix; the two are
independent and neither subsumes the other, since `DEVNULL` cannot unblock a
UAC elevation dialog.
Fixes#87703
When `hermes update` runs the cua-driver installer non-interactively,
stdout is captured (PIPE) but stdin is inherited from the parent process.
The upstream install.ps1 prompts `[Y/n]` when it detects a stale
cua-driver daemon, but the prompt goes to captured stdout (invisible)
while stdin waits for input — causing an 11-minute hang until the
watchdog timeout.
Fix: redirect stdin to subprocess.DEVNULL on the non-verbose path so the
installer reads EOF immediately instead of blocking. The installer exits
quickly with a non-zero code, and the existing handler shows the manual
re-run hint with the installer's captured output.
Fixes#79684
Two follow-ups on top of the #86391 salvage:
- check_macos_tcc_grants: a certificate-anchored DR (hermes desktop
--setup-tcc-identity, or a notarized release) now reports as stable in its
own class instead of falling into the identifier-pinned message; the
identifier-pinned message points at --setup-tcc-identity for the strongest
anchor.
- collect_relay_plugin_cutover_findings: only merge process-level env vars
when env_map is None (run_doctor's live path). An explicit env_map is a
complete environment description — merging os.environ on top made
report_deprecated_config_and_env non-hermetic on boxes exporting legacy
relay vars (10 findings vs the expected 2 in
test_report_does_not_count_as_blocking_issue).
Review feedback (AI review on #86391):
- guard _macos_desktop_dr subprocess.run against TimeoutExpired/FileNotFoundError
so a hanging codesign degrades to the unreadable-DR warning, never crashing
the doctor run (matches the file's existing subprocess guard pattern)
- select the desktop bundle by newest-mtime across release/mac-*/Hermes.app,
matching _desktop_packaged_executable, instead of a fixed arch order
- note the cdhash-match proxy assumption at the classification site
- document why /Applications/Hermes.app (Hermes-Setup launcher,
com.nousresearch.hermes.setup, certificate-anchored) is deliberately not probed
- extend the repair hint to cover per-service resets
- regression tests: codesign timeout and missing-codesign paths
GPT-OSS review: an empty codesign output would fall through to the
'stable identity' branch and false-positive. Guard with and
cover the empty-string case. Flash review: the non-macOS silence test
mocked the bundle to None, so it never exercised the platform guard;
mock a real path instead.
TCC keys permission grants to the app's code-signing requirement. Grants
made to pre-#73681 builds carry a cdhash-pinned requirement that no
longer matches the rebuilt bundle, so macOS re-prompts on every capture
even though the System Settings toggle shows ON — and the modern prompt
has no Allow button, so users cannot complete the one-time re-grant.
- hermes doctor: new check_macos_tcc_grants() reports the desktop
bundle's DR class (cdhash-pinned → grants reset on every update;
identifier-pinned → stable) and prints the exact stale-grant repair
(tccutil reset, toggle ON, fully quit & relaunch).
- hermes update: after a successful update on macOS with a desktop app
installed, print the one-line stale-grant guidance.
- docs: desktop.md no longer claims grants persist 'out of the box';
documents the one-time re-grant for pre-fix grants.
Closes#86385
Background processes started by subagents (task_id sa-*) route their
notify_on_complete / watch_pattern notifications to the parent
conversation (b95ec1cb5) because anything outliving the child needs a
durable consumer. In practice these 'npm ci finished' walls are noise
mid-conversation — the child's consolidated delegation result is the
deliverable.
- New config key delegation.surface_child_process_notifications
(default false = suppress). Flag true restores the previous behavior
exactly (delivery with subagent attribution line).
- drain_notifications drops (never requeues) completion/watch_match/
watch_disabled events whose task_id starts with 'sa-' when the flag
is false, logging at debug with session_id+task_id for diagnosis.
Requeueing would pin them forever — children never drain notifies.
- async_delegation events are NEVER suppressed (they ARE the result).
- watch_disabled emitters now carry task_id so sa- sessions' safety
events follow the same suppression as their other events.
- Config read errors fall back to the default (suppress) and never
crash the drain loop.
- Docs: delegation.md + configuration.md.
Follow-up to the salvaged #94296: the two guards covered the repair and
confirmed-update branches, but when cua-driver is enabled yet not
installed at all, control still reached _run_cua_driver_installer() and
an automatic 'hermes update' would launch the interactive install.ps1
anyway. Add the same defer before the installer run, keep POSIX
behavior unchanged, and give the confirmed-update message a natural
fallback when latest_version is unknown.
tool_search now takes queries: string[] (searched independently against
the same catalog, limit applies per query, default 5 / max 25) and
returns the split shape: per-query groups carry tool names only, one
shared tools map holds each matched tool's source, description (400-char
cap) and required parameter names once. When some queries miss, a single
top-level available_sources + hint block replaces the old per-response
fallback.
tool_describe now takes names: string[] and returns a map keyed by name;
unknown names collect in not_found (with the refresh hint) and
non-deferrable names keep their per-name spelling-check error in errors,
so one bad name no longer fails the whole call. Duplicates dedupe
silently.
The shared tokenizer now applies Snowball stemming (english, exact-pinned
snowballstemmer) at both index and query time, closing the measured
plural/singular miss where 'issues' failed to return create_issue. The
inline BM25 is unchanged. Stemmer instances are thread-local (they carry
mutable parse state and bridge dispatch can run on parallel tool-call
threads).
New config knobs under tools.tool_search: max_queries / max_describe_names
(default 10 each, floor 1, no upper clamp) bound the per-call array
inputs; over-cap calls error so the model repairs in one round-trip.
No backward compatibility with the single query/name shapes, by decision.
scripts/analyze_livetest.py renders both shapes since transcripts on disk
may predate this change.
Fixes the two live E2E blockers @ctaylor86 found on PR #77189 (macOS 26.3.1,
OpenSSL 3.6.3):
- retry the PKCS#12 export with -legacy when security import rejects the
OpenSSL 3 default format ('MAC verification failed during PKCS12 import')
- trust the self-signed root for the codeSign policy (security
add-trusted-cert -r trustRoot -p codeSign) — an imported-but-untrusted cert
is invisible to find-identity -v and unusable by codesign
- gate success on find-identity -v -p codesigning (postcondition), and use
the same -v probe for idempotency so an untrusted leftover cert is repaired
instead of reported as done
Tests rewritten as stateful fakes (valid only after import+trust), plus new
coverage for the -legacy retry, trust failure, postcondition gate, and the
untrusted-cert repair path; sabotage-verified (reverting to the name-in-output
probe fails 4 tests). Docs: manual fallback now includes the Trust step.
Adds a one-shot `hermes desktop --setup-tcc-identity` command that creates a
self-signed code-signing certificate in the login keychain (openssl +
security import), grants codesign access to it, writes
desktop.macos_signing_identity to config.yaml, and re-signs the packaged app
with a certificate-anchored Designated Requirement.
macOS persists permission grants (Full Disk Access, Accessibility, Files and
Folders, microphone) against the app's code-signing identity, not its path.
The default identifier-pinned ad-hoc signature is stable across rebuilds, but
a certificate-anchored identity is the strongest guarantee — the same
mechanism yabai/skhd rely on. Previously users had to create the certificate
manually in Keychain Access; this command automates the whole flow and is
idempotent (re-run after updates).
Docs: desktop.md TCC section now leads with the command, keeps manual steps.
Tests: 4 new — fresh cert creation path, idempotent reuse, non-macOS no-op,
cmd_gui early-exit before build.
Four fixes to the tool-search deferral layer, split from PR #92693 (the
availability-cache staleness fix ships separately):
1. The parallel batch planner now peels the tool_call bridge wrapper and
decides admission on the underlying tool — supports_parallel_tool_calls
works again when deferral is active. Unparseable wrappers stay
sequential barriers; bridged calls get exactly the admission the same
call gets direct. tool_search/tool_describe lookups batch concurrently.
2. _short_desc no longer truncates listing lines at 'e.g.', hostnames, or
version strings — a sentence terminator must be followed by whitespace.
3. BM25 indexes the source label (e.g. 'linear' for mcp-linear), so
service-name queries reach tools whose own name omits the service; the
dead 'mcp' prefix token is stripped.
4. Substring-fallback docstring corrected (token misses, not zero-IDF).
Salvaged from #92693 by @alt-glitch with authorship preserved.
On Windows, the pre-update concurrent-instance gate aborted with exit 2
whenever ANY other process held the venv hermes.exe shim — including the
gateway itself, which _pause_windows_gateways_for_update() stops moments
later and the post-update restart phase brings back. Users with a running
gateway were forced into a manual taskkill dance before every update.
The gate now filters gateway runtimes out of the abort list and proceeds
when nothing else is concurrent. Classification delegates to
_is_pausable_gateway -> gateway.status.looks_like_gateway_command_line
(the canonical shlex-tokenized, profile-selector-aware matcher shared by
the Desktop preflight exemption and the venv-holder guard fallback), so
the gate's exemption and the pause machinery cannot drift apart. Anything
not positively identified as a gateway — REPLs, dashboard, Desktop
backend children, gateway MANAGEMENT commands like 'gateway status',
unreadable cmdlines — still aborts exactly as before, and the abort
message now lists only the PIDs that are actually the user's problem.
Surgical reapply of PR #37039 by @damadorPL onto current main (the gate
moved from hermes_cli/main.py to hermes_cli/update_cmd.py in the main.py
decomposition); his substring classifier was replaced with the canonical
matcher, which also fixes the 'hermes gateway status' misclassification
flagged in review.
Co-authored-by: Hermes <hermes@nousresearch.com>
Port of @jeff-mettel's fix onto the post-#91378/#92902 fleet-restart
shape. The current-profile restart was gated on `launchctl list <label>`
exiting 0 - a booted-out job (plist present, definition deregistered:
crashed helper, manual bootout, failed prior update) fails that check,
so the branch silently skipped: no restart, no message, KeepAlive unable
to revive a definition launchd no longer knows, update printing
'Update complete!' with the gateway down. `launchctl list` is also
session-scoped and unreliable as a loaded/unloaded classifier.
- _restart_launchd_gateway_after_update() (his extraction, adapted):
plist-exists is the ONLY gate; launchd_restart() owns the
bootout/bootstrap/kickstart ladder for every plist-present state;
every failure path is loud and names the manual recovery command.
The gate-error 'except: pass' (the second silent variant) now counts
the label failed and tells the operator.
- Success still requires the #92902 supervision verify (fresh
supervised PID), composing his fix with the returned-is-not-supervised
guard.
- His regression suite adapted to the (restarted, failed) contract; the
old 'unregistered -> left alone' pinning test FLIPPED - it pinned the
bug.
A/B: his suite + the flipped test red on merge-base product code
(silent skip live), green at head. No macOS CI lane exists; field
evidence is #74973's reproductions plus the launchctl print output
shapes pinned in the suite.
* feat(cron): durable failure incidents with signature dedup and ack
Introduce a durable cron incident store (cron_incidents in the shared
cron/executions.db) that groups "same job + same error signature" across
runs, so a known recurring failure stops re-pinging the operator every run
once it has been acknowledged.
- cron/incidents.py: lazily-created incident table (detected -> alerted ->
reviewed -> closed lifecycle; closed is per-signature terminal), sha256
signature dedup over job_id + normalized error, redacted/truncated error
storage, failure-type classification, and ack/list/get/count helpers.
- cron/scheduler.py: record an incident on the failure delivery path and
suppress the per-run failure ping when the exact signature is acked (both
the normal failure path and the processing-raised retry path). Best-effort:
an incident-store error never breaks the cron run or delivery. Streak nudge,
alert-once markers, and delivery-error behavior are untouched.
- hermes_cli: add `hermes cron incidents [--state ...]` and
`hermes cron incidents ack <id>`.
- tests/cron/test_cron_incidents.py: dedup, lifecycle, redaction,
classification, lazy-schema, scheduler gating, and CLI coverage.
Non-goals deferred to later slices: Discord buttons/review view, HMAC action
tokens, owner-agent review launch, approval-gated fixes, incident playbooks.
* refactor(cron): tighten incident lifecycle, wire alerted state and suppressed_acked outcome
Follow-ups on top of the salvaged #94692:
- Drop the dead 'reviewed' state and the SQLite CHECK (state validity
lives in INCIDENT_STATES so future slices can add states without a
table rebuild); lifecycle is detected -> alerted -> closed.
- Actually mark incidents 'alerted' after a failure ping reaches
delivery, on both the normal and exception delivery paths.
- Record ack-suppressed runs with a distinct 'suppressed_acked'
delivery outcome (registered in cron_health monitoring) instead of
the ambiguous generic 'suppressed'.
- Drift-skip alerts explicitly bypass the ack gate (they carry the
remediation command and alert once via drift_alerted already).
- Docs: failure-incidents section in the cron guide.
- Tests for the alerted transition + never-resurrect-closed.
---------
Co-authored-by: Laura López Real <113060513+laulopezreal@users.noreply.github.com>
briandevans