main's hermes:connections:update-all handler grew (renderer-side exclusions +
the managed-SSH dispatch branch) after #95606 branched, pushing the claim-
guarded dial past the 2,000-char slice.
(connectionId, profile) scope, but it was only wired into hermes:connection,
hermes:connection:for, and the power-resume pool rebuild. Five other real
call sites still invoke ensureRegistryBackend()/ensureBackend() directly:
the media-protocol connection resolver, the terminal-pane backend resolver,
the ~5s roster-enumeration probe, the connections update-all dispatch, and
dispatchRegistryApiRequest (every registry-scoped hermes:api REST call).
ensureRegistryBackend() has a genuine await-then-check race
(reuseMatchingPrimarySshBackend before the pool entry check-and-set), so any
of these five racing a guarded dial for the same scope can each bootstrap
their own SSH tunnel / remote dashboard for the same connection — exactly
the symptom class #90812 was written to prevent, just reached through an
unguarded path instead of two renderer windows.
Left the ensureRegistryBackend() self-call inside its own dispatch-time
health-probe reconnect branch (registryDispatchRevalidation) unguarded —
that recursive path has its own coordination semantics and touching it
without modeling reentrancy against the newly-guarded entry points here is
a separate, riskier change better done on its own.
Re-implements the #93042 main.ts wiring against current main (post-#94724
drift), making the extracted managed-ssh-update engine reachable:
- ManagedConnectionUpdateGate instance + owner-only recovery journal at
DESKTOP_MANAGED_SSH_RECOVERY_PATH (read/write/persist/mark/clear with
strict record validation).
- IPC: hermes:connections:update-managed (requestManagedSshUpdate with
correlation-id claim + in-flight dedupe); update-all's ssh rows now route
through the transactional drain/update/restore lifecycle instead of
POSTing the remote backend updater.
- Gate enforcement at every dial/mutate seam: bootstrapSshConnectionInner
(pre-dial + publication fence with exact-serve rollback via
rollbackSshBootstrapResult), resolveRemoteBackend, ensureRegistryBackend,
saveRegistryConnection dial-field edits, connections:remove, and
primary-routing mutations (set-primary, set-launch-mode,
connection-config save/apply, profile:set) via
assertCanMutateManagedPrimaryRouting.
- Scope capture/drain/restore drivers: captureManagedSshScopes (pool +
primary discovery, bootstrap fence join), drainManagedSshScope (exact
identity-re-proof termination, no-kill forward recovery),
ensureManagedSshBackend(AtKey)/restoreManagedPrimarySshBackend restores,
openManagedSshUpdateTransport for serve-less connections.
- Startup recovery (resumeManagedSshRecoveries before createWindow) and
before-quit join of in-flight update/recovery operations BEFORE the SSH
coordinator is sealed, so restore dials are not refused during quit.
- Extended sshConnections state (spawnNonce/creationTime(Ns)/startedAt/
hermesPath/hermesHome/pythonPath/remoteProfile/registryConnectionId/
primaryRegistryScope) so drain can prove the exact serve it owns;
bootstrap coordinator entries carry metadata for the update fence;
persistSshConnectionToken mirrors tokens per managedSshTokenPersistencePlan.
- preload/global.d.ts: connections.updateManaged +
DesktopManagedConnectionUpdateResult/Receipt types.
Renderer UI (fleet-updates store, about-settings, system.ts ProfileScope
plumbing, i18n) intentionally NOT wired — it belongs to the deferred fleet
rollout UI and follows separately.
Wiring re-implemented against current main; design from #93042 by @andrexibiza
tsc -p apps/desktop clean; electron project 1924/1924 passed (133 files,
incl. 131/131 across the three engine suites); eslint clean on touched files.
The Desktop can drive a full update of a REMOTE SSH instance: claim the
connection (ManagedConnectionUpdateGate pauses dials/mutations while the
update owns it), terminate the owned backend with an identity re-proof
at the signal boundary (terminateOwnedDashboardForUpdate — argv+creation
time re-read in the same remote shell that signals), run the updater
under an update-in-progress marker/mutex on the remote install root,
bootstrap the new backend, and fence its publication so a rollback
cannot leak a half-published serve (fenceManagedSshBootstrapPublication
+ waitForManagedSshBootstrapFence barrier).
Extraction notes (campaign #91277 Phase 4; rollout/canary engine stays
behind per sequencing):
- managed-ssh-update.ts + test: clean cherry-pick (new files).
- windows-remote-lifecycle.ts + test: applied as-is (zero main drift).
- remote-lifecycle.ts: PR hunks stitched AROUND current main's #95532
skew guards and #91668 SIGKILL-escalating cleanupStale, which are
PRESERVED verbatim — the PR's python identity-re-proof termination is
wired for the managed-update path only; connect's stale replacement
keeps main's proven kill. One PR test assertion re-pinned accordingly.
- One PR test fixed: floating coordinator.start() promise whose
rejection IS the contract under test now has an explicit handler
(vitest flagged it as an unhandled rejection).
tsc clean; 131/131 across the three touched electron suites. main.ts
wiring (IPC + deps bag) follows as a separate commit.
A backend.lock.json that exists but doesn't match what this build writes
(unknown/future schemaVersion, truncated JSON, missing or foreign
ownershipId, malformed shape) was previously indistinguishable from 'no
lockfile': connect() would spawn a fresh backend on top of it and
overwrite the record, and cleanup paths could drop foreign state —
disarming the #78872 ownership guard exactly when another (e.g. forked)
desktop build shares the remote. readLockfile now returns a skew
sentinel for existing-but-foreign lockfiles; connect() refuses with a
'remote-lockfile-skew' error and a skew warning instead of
reaping/overwriting, disconnect() and cleanupStale() skip entirely.
Refs #95532
JSON.stringify does not escape '<' — a reloadUrl containing
'</script><script>…' would terminate the inline <script> element of the
data: error page and let an attacker-controlled URL inject markup/script.
Escape <, >, & (and U+2028/U+2029) as \uXXXX sequences after stringify;
add regression test.
A torn renderer bundle (update replaced the app while its files were
locked, e.g. antivirus or a still-running instance) loads index.html
fine and then dies on the first lazy import — a white screen with only
a desktop.log line. A main-frame load failure (missing index.html,
blocked file) was likewise log-only.
- resolveRendererIndex() already detects torn bundles; the primary
window now refuses to load one and shows a visible repair page
(error code, missing assets, 'hermes desktop --force-build', Reload)
instead of a blank window.
- did-fail-load on the main frame now gets bounded auto-reload through
the shared rolling reload budget (transient failures self-heal) and,
once the budget is exhausted, surfaces the visible error page.
ERR_ABORTED and sub-frame failures stay log-only, and helper windows
(OAuth/portal) keep their log-only policy (opt-in via
reloadOnFailedLoad).
Regression tests cover the policy decisions (reload / abort /
budget-exhausted surface), budget sharing with render-process-gone,
and the error page content + data: URL loading.
new BrowserWindow({ icon }) and app.dock.setIcon() decode the icon file
synchronously on the main process and throw on undecodable bytes. The
icon ladder was resolved with statSync().isFile(), which only proves a
file exists — a truncated or zero-byte PNG inside a packaged app.asar
(interrupted electron-builder run, partial copy) killed the main process
inside createWindow(): the window never appeared, running turns lost
their renderer, and the desktop log showed 'Uncaught exception: Error:
Failed to load image from path .../app.asar/public/apple-touch-icon.png
at createWindow'.
Resolution now runs through a decoding probe (nativeImage.createFromPath
must yield a non-empty image); a candidate that exists but does not
decode is skipped like a missing one, so the app falls through to the
next rung or starts with the platform default icon instead of dying.
The ladder and probe live in a pure module (electron/app-icon.ts) so
precedence is unit-testable without a running Electron app; window
factories re-resolve per call exactly as before.
Regression tests cover: skip-first-undecodable, all-fail -> undefined,
first-pass wins, missing/empty/directory rejection, and the unchanged
mac/Windows precedence ladder.
Co-authored-by: brooklyn! <brooklyn.bb.nicholson@gmail.com>
The #95085 quit teardown kills the owned serve --isolated before the
SSH tunnel closes, but a backend mid-turn (in-flight LLM call, live MCP
children) can ride out SIGTERM past cleanupStale's 5s graceful wait.
The old code then gave up (threw, kept the lockfile) and before-quit's
6s race closed SSH anyway — reparenting the still-running serve to
pid 1: the reported leak, now specific to quit-during-active-turn.
Escalate to kill -9 with a confirmed-exit wait; only an unkillable pid
(D-state, permissions) still throws and preserves the lock record so
the next connect's reap pass retries.
'Make primary' on a registered remote/cloud/ssh gateway only rewrites
connections.json — the v1 config.mode stays 'local', so startHermes()
resolved no remote route and spawned a loopback 'hermes serve' the
desktop never uses (full MCP set duplicated, port squat, respawn on
poll). resolveDesktopRemoteRoute gains a lowest-precedence registry-
primary rung (source: 'registry', existing v1/env/profile precedence
untouched), and globalRemoteActive() now recognizes a remote registry
primary so local-entry routes force pooled local children instead of
delegating into a primary that dials remote. A 'local' registry
primary still resolves null — genuinely-local desktops unchanged, and
local-profile secondaries keep their forced-local pooled backends.
- normalizeRegistry now preserves every malformed entry (unknown kind,
url-less remote/cloud, host-less ssh, mangled non-object items, and
any entry whose normalization throws) under a capped 'quarantined'
key that survives write cycles — healthy entries keep loading and
user data is never silently deleted.
- A whole-file parse failure preserves the original bytes in a
connections.json.corrupt-<ts> sidecar BEFORE the drift reconciler or
a save can overwrite the file with the degraded local-only registry.
- Loads log a quarantine notice and sanitizeConnectionsRegistry
surfaces reason+label summaries (never raw entries/token envelopes).
Two registered basic-auth gateways shared the single
persist:hermes-remote-oauth cookie jar, so signing in to gateway B
evicted gateway A's session cookies (Chromium jars ignore the port) and
A's cookie was silently presented to B on every request. Non-primary v2
registry remotes with cookie auth now ride a per-connection partition
(persist:hermes-remote-oauth:conn:<id>) resolved at the jar boundary;
the registry primary, v1 remote, cloud cascade, and portal flows keep
the legacy shared jar so upgrades do not sign anyone out. Fail closed:
a connection's requests can never see another connection's cookies.
The sidebar's home-dir repo crawl descends into ~/Pictures, ~/Music,
~/Movies and ~/Public — and into Photos/Music library packages — which
triggers Photos, Media Library and Files & Folders permission prompts
attributed to Hermes.app. Because the app is ad-hoc signed and re-signed
on every self-update (#49110), macOS drops all TCC grants after each
update, so these prompts re-fire every time.
Skip the media folders as direct children of a search root (nested dirs
like ~/dev/Music are ordinary and still scanned; an explicitly passed
root is still walked), and skip Apple library packages
(*.photoslibrary/*.musiclibrary/*.tvlibrary/*.aplibrary) at any depth.
Partial mitigation for #49110 / #52010: removes the Photos and Media
Library prompts entirely; the identity reset itself needs Developer ID
signed release artifacts (tracked in #49110).
reconnectGateway()'s in-flight lock lives at renderer module scope, so it
only dedupes reconnects inside ONE window. Two windows racing the same
wake both invoke the main-process backend ensure IPC, and for a pooled
SSH connection the loser of the pool-entry race could bootstrap a
duplicate remote backend (two tunnels, two remote serve processes).
Electron main is the single owner of backend lifecycles, so the claim
now lives there: BackendDialClaims keys in-flight dials by the pool
scope key from backendScopeKey(connectionId, profile) — the composite
identity seam wave-1 #93189 established for effective-identity reuse.
'hermes:connection' and 'hermes:connection:for' route through
backendDialClaims.run(), so concurrent renderer dials for one scope
coalesce onto one spawn and the second caller receives the first's
result. A claim exists only while its dial promise is unsettled: both
outcomes release it, a failed dial is never cached (fail closed, not
latched), and a synchronously-throwing dial rejects the claim instead
of escaping the seam.
The #93910 resume rebuild re-dials retired pool keys through the same
claim (redialPoolBackendAfterResume + new parseBackendScopeKey), so a
resume-driven rebuild and a concurrent renderer reconnect also coalesce
instead of racing.
After macOS sleep/resume, pooled remote SSH descriptors kept serving dead
tunnels: a remote entry has no child 'exit' to clear it, the renderer
keepalive spares it from the idle reaper, and the wake-path nudges from
b90289b04/febed060a only re-drive the PRIMARY renderer socket. The
background failure-streak policy needs several probe rounds before it
drops a descriptor, so the Bots pane showed 'Gateway offline' long after
the network was back.
New revalidateSuspectPooledRemoteBackends(): on resume every pooled
remote is suspect — probe each once (bounded by
REMOTE_LIVENESS_TIMEOUT_MS), retire the dead ones immediately (pool
entry + SSH bootstrap + tunnel/master teardown) and rebuild them through
the caller's dial path, while healthy descriptors are left untouched. A
failed retire skips the rebuild (never dial on top of an installed
descriptor); a failed rebuild is logged and left to the renderer's
normal reconnect — the sweep never throws.
attachPowerResumeRemoteRevalidation() wires the sweep to the Electron
powerMonitor 'resume'/'unlock-screen' seam with a 15s holdoff so the
near-simultaneous macOS wake signals coalesce into one sweep and can
never form a hot loop; overlapping kicks additionally join the one
in-flight sweep via the existing RemoteRevalidationCoordinator.
Live-confirmed on the Phase B build: hermesDesktop.connections.save()
succeeds and the registry on disk gains the row, but the switcher menu
(fed by the renderer $connectionsRegistry snapshot) keeps painting the
stale list until reload. remove() already broadcasts
hermes:connections:changed; save() only did so on the dial-material-edit
branch, so a brand-new connection or a label rename never reached the
switcher's onChanged re-pull (or any other window).
Fix at the publish seam only: saveRegistryConnection now broadcasts a new
'saved' reason for every successful save that isn't a dial-material edit.
'saved' is a pure registry-refresh signal — the use-gateway-boot listener
explicitly ignores it (nothing moved, so no dispose/redial/forget), while
the switcher's existing onChanged listener re-pulls the snapshot.
Tests:
- electron/hardening.test.ts pins both broadcast branches in
saveRegistryConnection (source-assertion pattern; main.ts has no exports).
- connection-switcher.test.tsx mirrors the live repro scenario
(/tmp/mg-ab/w2_95393.py): menu before save lacks the row, Electron's
'saved' push arrives, menu after — without reload — shows it.
Partial cherry-pick of PR #95007 (weismanfamily). Surviving scope:
- electron connection-registry: registrySourceOwnsPrimaryBackend() —
descriptor-level proof that a registry-scoped request names the
already-running primary backend, wired into ensureRegistryBackend as
the generic (non-SSH-fingerprint) primary-owns short-circuit so a
cloud/url registry primary cannot spawn a second isolated server.
- store/connections: waitForInitialConnection() before the boot-time
source restore, so the sidebar registry cannot dial the preferred
source a second time while the identical primary backend is still
publishing its connection identity.
Dropped scope (superseded on main): the renderer routing half —
primaryConnectionId plumbing, primaryOwnsAgent short-circuits in
requestGatewayForAgent/openGatewayForAgent/ensureGatewayForAgent
(main has isPrimaryRegistryRoute via 1ec32e738), the 3-arg
setPrimaryGateway boot wiring (main has setPrimaryGatewayConnection),
the use-session-list-actions stampConnectionOwner (row stamping lands
via #94656/#94901), and the wiring.tsx bare-profile promotion commit
65d106e41 (main's knownSessionOwner covers it).
Original-PR: #95007
Dropped-scope: renderer routing half of d1c0fb093; all of 65d106e41
teardownSshConnection closed the tunnel and SSH transport but never
killed the detached serve --isolated process. Spawn uses setsid/nohup,
so the backend reparents to pid 1, keeps state.db open, and accumulates
across Cmd+Q. Reuse cleanupStale via disconnect while SSH can still
exec, sequence remote kill before close, and seal the bootstrap
coordinator so reconnect during a prevented first quit cannot respawn.
The quit race is 6s to cover cleanupStale's 5s wait-for-exit loop.
Avoid opening a second SSH lifecycle when a migrated registry request targets the same primary/default backend already booted through the legacy route. Compare effective SSH configuration for representation-only drift, treat empty and default as the same root profile, and keep named profiles isolated.
Never-interacted remote bots painted as bare handles because roster rows
carried only profile names: display_name/title/ui_meta/has_avatar were
fetched lazily on first interaction (#91365). Thread credential-free
profile metadata from the enumeration-time /api/profiles body through
enumerateRegistryAgentSources (main.ts) and buildAgentRoster
(connection-registry.ts), keeping it attached to the connection-qualified
row across the same-install collapse. The plugin.js botRosterMeta half of
the original PR is dropped — superseded by landed #92731.
Fixes#91365
Salvaged (partial) from #92708.
The profile rail's live workspace switch never persisted the selection,
so the Desktop always booted back into the previous startup profile
(#79886). Route the successful primary-backend activation through a new
persistence-only hermes:profile:remember IPC (validated
writeActiveDesktopProfile) that records the choice WITHOUT tearing down
the backend or reloading the window like hermes:profile:set does.
Registry-source picks name another source's profiles and do not touch
the startup preference. Reapplied semantically over three weeks of
main.ts/preload.ts drift (selectProfile now routes through
activateOnCurrentSource, #91349/#91365 seams).
Fixes#79886
Salvaged from #79888.
Addresses AI-review feedback on #94653: note where the 'connect-on-demand'
sentinel is produced, and cover the interaction between
isLocalEnumerationFailure and localRouteFallbackProfiles directly (not just
the helper in isolation).
'connect-on-demand' means local roster enumeration was intentionally
skipped to avoid spawning a local backend on a remote-only workspace,
not that it failed. The plugin-profile-routes IPC handler passed
Boolean(error) straight through, so that deferral was treated as a
genuine failure and Bot Mode re-synthesized cached local profile rows
even though local was never dialed.
Fixes#94648
A pooled remote backend (Bot Mode, group chat) keeps its descriptor and SSH
forward cached in the backend pool. When the remote Desktop relaunches, the
remote process dies but the local forward stays LISTENing, so
ensureRegistryBackend() keeps returning the dead descriptor and every dispatch
to that machine fails until the app is restarted.
The background sweep cannot cover this: revalidatePooledRemoteBackends() only
runs from the renderer reconnect IPC, which never fires while the primary
connection stays healthy.
Validate the exact cached descriptor at dispatch time with a short /api/status
probe (2.5 s). On failure, retire the pool entry and its SSH forward, then
reconnect on demand. Concurrent dispatches share one retire/reconnect sequence
through a RemoteRevalidationCoordinator keyed on the cached promise, and
identity checks make a late failure from an old descriptor unable to tear
down a replacement another caller already installed.
Verified on a two-Mac setup (MacBook + Mac mini over SSH): after relaunching
the Mac mini's Desktop, a group-chat turn from the MacBook now reaches the
mini's backend and its reply lands, where it previously failed forever.
taskkill /T /F returns when termination is INITIATED, not completed, and
the pre-handoff unlock gate only probed the venv hermes.exe shim — which
the 'python.exe -m hermes_cli.main serve' backend need not hold at all.
The gate could therefore pass on its first iteration with zero dwell
while the killed pythons were still unmapping .pyd files; the
venv-blocker scan (no liveness filter) then reported those dying
processes as holders and aborted the hand-off. Every first update
attempt from the footbar failed; the manual retry succeeded because the
process table had settled by then.
The unlock gate now lives in backend-release-gate.ts (dependency-free,
backend-child.ts pattern) and requires BOTH the shim unlocked AND every
signalled PID to have actually left the process table; stragglers
collected per-pass are killed and join the watch set. On deadline the
old shim-only criterion survives as the escape hatch — lingering PIDs
past 15s are the venv-blocker re-scan's job. applyUpdates additionally
re-scans up to 2x with a 1.5s settle before aborting on 'blocked', so
untracked grandchildren an AV driver holds in teardown stop failing the
update while a REAL holder still aborts on the third scan.
Surgical reapply of PR #78037 fix 1 by @3x3xX3N0N onto the post-#87599
code shape (stopBackendTreesForUpdate extraction, stopSafeBlockers
re-scan path). The re-scan settle idea was first submitted by
@MaheshBhushan (#74831); the killed-PID tracking seam matches
@webtecnica's #74956.
Co-authored-by: MaheshBhushan <128616744+MaheshBhushan@users.noreply.github.com>
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Hermes <hermes@nousresearch.com>
Batch follow-ups on top of the salvaged privacy declarations:
- tests-js/desktop-mac-usage-descriptions.test.ts: EXPECTED_USAGE_DESCRIPTIONS
now pins the FINAL key set (camera + calendar x2 + reminders x2 + screen
capture + local network) so the drift-protection assertion locks the whole
batch as permanent regression coverage.
- entitlements.mac.plist: add com.apple.security.personal-information.reminders
alongside the calendars entitlement #65220 added — the reminders usage
descriptions need the matching entitlement under hardened runtime (sibling
site the original PR missed).
Teknium