On Windows the updater renames the live `hermes*.exe` shims aside
(`hermes.exe.old.<unix-ms>`) so uv can write replacements. When that quarantine
succeeds but the install then fails, the recovery path could leave the install
with no `hermes` on PATH at all — unrecoverable in place, because the command
that would repair it IS `hermes update` (#75584).
Restoring a quarantined shim happens at three sites: the updater, the
early-recovery installer, and the startup sweep's orphan rescue. Each was a
single un-retried rename whose OSError was swallowed in silence, while the
OUTBOUND quarantine rename already retried a lock. That is backwards — a failed
quarantine merely aborts an update, a failed restore removes `hermes` from
PATH — and the two sites that had messages had already drifted apart.
- `_early_recovery.restore_quarantined_shims()` is now the single
implementation: retry ladder, one recovery message, returns the pairs it
could not restore. It lives in the stdlib-only module that both `main` and
`_install_repair` already import, so the layers cannot drift again. A pair is
not a failure when the original reappeared or the quarantine file vanished —
two processes sweeping the same orphan must not produce a spurious error.
- `_cleanup_quarantined_exes` unlinked every `*.exe.old.*` on each invocation.
When the original shim was already missing, that .old file was the ONLY
surviving copy — deleting it converted a one-rename recovery into a full
reinstall. It now rescues the orphan through the shared helper instead, and
leaves anything inside a 15-minute grace window alone so it cannot destroy a
concurrent update's in-flight quarantine.
- Ordering is by the PARSED `.old.<unix-ms>` stamp, not the raw filename.
Lexicographic ordering only tracks recency while every stamp shares a digit
width; a stray `.old.999` sorts above a 13-digit epoch-ms stamp and would be
the copy rescued onto the live shim name.
- Names whose suffix does not parse as int-ms are not ours: never rescued,
never deleted. The sweep should not destroy files whose provenance it cannot
establish, and they are not produced by the quarantiner.
The stamp is read from the filename rather than st_mtime because `rename`
preserves the original shim's mtime, which records when uv wrote the shim —
days earlier, in general — not when it was quarantined. A regression test pins
that distinction.
Messages go to stderr: the sweep runs on EVERY hermes invocation and
`hermes acp` speaks JSON-RPC on stdout.
Scope note: `_quarantine_running_hermes_exe` is deliberately byte-identical to
main here. Why the outbound rename fails in the first place (the launcher
holding its own image without FILE_SHARE_DELETE) is #88121's subject; this is
the net underneath, covering the case where quarantine SUCCEEDS and the install
dies afterwards. The two touch disjoint functions and can merge in either order.
Reproduced and verified on Windows 11 (26200), Python 3.11.15: stranded the
shims, confirmed a normal `hermes` invocation now rescues the orphan instead of
deleting it, and confirmed an exhausted rescue prints the recovery command.
16 new tests; 35 pass across the four quarantine suites.
Runtime desktop plugins read their source through hermes:readFileText,
the preview IPC that silently truncates at TEXT_PREVIEW_MAX_BYTES
(512 KiB) — a larger plugin.js evaluated as a partial file (cryptic
syntax error, or worse, a half-module that parses).
- electron/main.ts: dedicated hermes:readPluginSource handler — full
read, 16 MiB cap enforced as a hard EFBIG via resolveReadableFileForIpc
(same path hardening + sensitive-file blocking), never truncation.
- preload.ts / global.d.ts: bridge + types (optional — older shells).
- runtime-loader.ts: loadDiskPlugin reads via readPluginSource; on older
shells falls back to readFileText but FAILS LOUDLY on truncated:true
(error toast + error inventory row) instead of evaluating a partial
file. Existence probe keeps the preview read (metadata is enough).
- tests: full-read path, loud old-shell truncation failure (sabotage-
verified), small-plugin fallback.
CPython interns identifier-like string literals, so 'is' cannot
distinguish an import alias from a copy-pasted literal (verified:
two exec'd namespaces each defining the literal share one object).
The equality assertions three lines above are the full honest guard.
Also reword a comment: raw == is marker-SENSITIVE, not asymmetric.
Reviewer point on #92539: nothing documented that an in-place content
mutation of a stamped dict must pop _DB_PERSISTED_MARKER (and invalidate
the bounded flush-scan prefix) or the DB silently goes stale. Both
existing mutators (turn_finalizer fill-empty-tail, context_compressor
micro-compaction defrag) already follow the contract; this states it at
the constant so the next one does too.
Review-pass follow-up on the load-time durability stamp:
- hermes_state.py: import the marker from agent.context_compressor instead
of a third synced literal (hermes_state already imports agent.* at module
level; only run_agent is circular). Old comment claimed otherwise.
- agent/turn_finalizer.py: replace the raw "_db_persisted" string at the
fill-empty-tail pop site with the shared constant (was outside the drift
guard).
- agent/conversation_compression.py: the no-op progress check now falls back
to a marker-insensitive comparison (_strip_marker_for_comparison). Loaded
rows are stamped at materialization time while compress() output is
marker-swept, so a semantically-identical no-op copy on a cold-resumed
session would previously compare unequal and take the progress branch.
Raw == still runs first so engine-returned list subclasses keep their
__eq__ semantics.
- test_marker_constant_in_sync extended to turn_finalizer + identity
assertions; new test_noop_progress_check_is_marker_insensitive
(mutation-checked: fails when the helper is neutered).
Resumed sessions loaded message dicts from state.db WITHOUT the
_DB_PERSISTED_MARKER, so any flush that lost the identity boundary
(compression durable-snapshot adoption, incremental tool-call persists,
rotation preflight on cold resume) re-appended the ENTIRE loaded
transcript as new rows. Compression cycles then doubled the copies:
the incident session grew 998 -> 1995 -> 3990 -> 7981 rows across
three aborted rotations (15,962 active rows, only 472 distinct).
Fix at the architectural chokepoint: SessionDB._rows_to_conversation
(shared by get_messages_as_conversation and get_resume_conversations)
now stamps the marker at row materialization time - a dict built FROM
a durable row is persisted by construction, regardless of which caller
loads it or how the list is later handed to a flush.
Safety:
- Wire-safe: every transport strips underscore-prefixed keys before
the API request (chat_completion_helpers, anthropic_adapter), same
contract as the existing _row_id stamp in the same function.
- Rotation handoffs still write: compression's assembly copies strip
the marker (_fresh_compaction_message_copy + the terminal
_strip_persistence_markers sweep), so compacted transcripts still
flush to the child session (#57491 invariant preserved).
- Branch/seed copies unaffected: /branch and _persist_branch_seed
build fresh field-projected dicts and write via append_messages_batch
directly, not through the marker-gated flush.
Tests: new regression suite (marker sync, load stamping, 3-cycle
amplification repro, new-tail write guard, compaction-copy handoff);
updated the #68454 control test that asserted the old double-write
behavior and the ACP restore shape test.
browser_vision's native fast path base64-encoded screenshots at full
resolution and baked them into the tool result uncapped — the exact
sibling of the vision_analyze path #92699 fixed. Apply the same
proactive 256KB/1568px resize before the embed enters reusable history.
Fail-open by design: without Pillow the resize helper falls back to raw
bytes and the compressor's keep-newest pass still retires stale embeds.
Sibling-gap follow-up for the #92725 salvage; the shared-cap approach
mirrors the policy-owner idea from #92748.
Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
Images locked in protect_last_n never shrank, so compression savings
stayed under 10% and anti-thrash disabled further compaction. Keep the
newest three tool-result screenshots live for follow-up QA and replace
older native embeds with placeholders.
vision_analyze baked up to 4 MB / 7900px screenshots into immutable
history, so every later turn re-sent ~400K chars. Cap embeds at 256 KB
and 1568px (the long edge models actually read) so screenshot QA no
longer blows the context.
Desktop writes backend.lock.json only after a remote profile backend announces readiness. Concurrent Bot Mode profile starts could therefore see their young siblings as unowned PPID-1 processes and mutually reap them, causing SSH reconnect storms and stale turn leases.\n\nProtect unregistered backends for a bounded startup grace period, fail closed when age cannot be read, and cover young, unknown-age, boundary, lock-owned, and old-orphan behavior.
Builds on @mrsucesso's durable-marker recovery (previous commit):
- GET /api/hermes/update/receipt — the full durable receipt (steps,
skips, gateway restart outcome, fleet matrix) + compact summary; the
authoritative update-outcome record (written by every run since
#91283, including refused/failed).
- /api/actions/hermes-update/status now attaches the receipt summary,
and when BOTH the in-memory registries and the update.log marker are
gone (dashboard restarted + log rotated — the #81193 state), a
finished receipt reports the outcome: success→0, partial→1. A
still-running receipt proves nothing (clients keep polling).
- Desktop (updates.ts): the apply poll reads the attached receipt — a
finished receipt whose run started at/after this apply is
authoritative, replacing timeout-based failure inference across the
update's restart gap ('Backend update failed' on successful updates,
#81193; 'boot failed' during update restarts, #87359).
Live-verified: real uvicorn server + real UpdateReceipt writer (the
exact code hermes update runs) over real HTTP — receipt endpoint 200
with summary; #81193 state (no registries, no marker) reports success
from the receipt alone; partial receipt with a DOWN fleet row maps to
exit 1 (no false success).
Persisted update completion markers survive the dashboard restart that clears in-memory action state. Recover the latest safe marker from update.log so remote Desktop clients do not report a successful backend update as failed.
The salvaged test drove the FULL post-update pipeline against the real
dev box: real fleet probes read live gateways as STALE (exit 1) and the
restart phase tripped the live-system guard on a real gateway PID. Pin
empty fleet/gateway discovery and make _reload_updated_runtime_modules
(the proof the post-update path ran — the bug returned before it) abort
the pipeline. Sabotage-verified: disabling the hoisted sync fails the
test.
On a fork, `hermes update` compares HEAD against origin/main, and only then
syncs the fork from upstream — inside the `commit_count == 0` branch, which
returns immediately afterwards. So an update that pulls hundreds of commits
from upstream prints "Already up to date!" and skips everything the
post-update path does, including the dependency sync and the gateway restart.
Observed on a fork-based deployment: 1654 commits pulled, "Already up to
date!", and the launchd gateway left running. It then held pre-update modules
in memory while lazily importing post-update ones, and failed later with an
AttributeError for a method that plainly exists on disk — a mixed runtime that
looks nothing like an update problem. Correlating every run in update.log, a
restart happened on exactly the runs that pulled upstream *without* also
claiming to be up to date, and never once they started co-occurring.
Decide before the branch: capture HEAD, sync, and if HEAD moved, set
commit_count from the range so the normal post-update path runs. The pull that
follows is a no-op (the sync updates origin too); reaching the restart is the
point. commit_count is floored at 1 — HEAD moving *is* the update, so a failed
or zero count query must not send us back down the early return.
steps still being skipped afterwards.
Refs #73108
Phase-1 verification gap (#91277, found auditing our own landed matrix
against the mapped issues): collect_fleet_versions only listed gateways
with a LIVE pid, so 'restart stopped it and nothing came back' produced
NO row at all — the exact silent-failure shape the matrix exists to
catch (#88848/#74973 class) passed with exit 0.
- collect_fleet_versions(pre_restart_pids=...): a dead pid becomes a
'down' row only when it was alive at update start AND its runtime
status still claims a running state. Rollout-safe: no snapshot (old
callers), clean stops, startup failures, and stale records from
long-dead gateways keep the historical no-row behavior.
- print_fleet_version_matrix escalates on down rows like stale ones
(exit 1) with the per-profile restart remediation.
- cmd_update passes its existing pre-restart PID snapshot.
Sabotage-verified (reverting the membership check fails the new test);
live-verified with a real spawned-then-killed process producing the
DOWN row and matrix escalation.
Address review feedback: the gate reused has_vertex_credentials(), which also
returns True for an ambient GOOGLE_APPLICATION_CREDENTIALS path. That var is
commonly set globally for unrelated GCP work, so a user who never configured
Hermes for Vertex would see it in the explicit-only picker and could spend
against those credentials — weakening the explicit-configuration guarantee the
gate documents (mirrors the existing _IMPLICIT_ENV_VARS carve-out).
Add has_explicit_vertex_config() in agent/vertex_adapter.py that checks only
Hermes-scoped signals — VERTEX_PROJECT_ID / vertex.project_id (project
override) or a resolvable VERTEX_CREDENTIALS_PATH — and NOT
GOOGLE_APPLICATION_CREDENTIALS. Route the auth gate through it.
Adds a regression test asserting an ambient GOOGLE_APPLICATION_CREDENTIALS
path alone does not mark Vertex explicit, and updates the existing test to
drive the real config signal instead of mocking has_vertex_credentials().
is_provider_explicitly_configured() only checked provider env vars for
auth_type="api_key" providers. Bedrock is registered with
auth_type="aws_sdk" and an empty api_key_env_vars tuple, so a user who
sets AWS_BEARER_TOKEN_BEDROCK (or an AWS_ACCESS_KEY_ID +
AWS_SECRET_ACCESS_KEY pair) in .env was never counted as having
explicitly configured the provider.
Symptom: the desktop model picker (and any consumer of
build_models_payload(explicit_only=True)) silently hid the Bedrock row
even though list_authenticated_providers had discovered credentials and
built a full model list for it. Reproduced on main:
build_models_payload(ctx, explicit_only=False)
-> ['moa', 'nous', 'bedrock', ...] # row exists, 132 models
build_models_payload(ctx, explicit_only=True)
-> ['nous', ...] # bedrock filtered out
Fix: aws_sdk-type providers now count as explicitly configured when
Bedrock-relevant env credentials are present. Deliberately env-var-only:
ambient sources (AWS_PROFILE / SSO, EC2 IMDS, container credentials)
still do NOT auto-surface, consistent with the gate's purpose (#56974)
and with a lone AWS_ACCESS_KEY_ID (no secret) not counting.
Tests: six behavior-contract cases in test_auth_provider_gate.py
covering bearer token, key pair, lone key id, ambient AWS_PROFILE,
no-credential baseline, and non-leakage into other providers.
Salvage of saralilyb's remote-bot routing work onto current main:
- kept: immutable (connectionId, profile) owner capture, requestForBot
routing, backendTargetProfile aliasing, group session owners,
connection-qualified deletion, focused-owner atoms, remote roster
merge, Electron profile-delete routing, sdk/store/transcript changes
- reconciled: canonical Bot Chat resolution stays NAME-identity (the
'Bot Chat' registry row) and FAIL-CLOSED on lookup errors — now
consulted on the bot's own source via the captured owner route, so
remote bots get the same no-fork guarantees
- dropped: pointer-pin plumbing (preferredSessionIds, saveBotMeta chat
writes, pin verification) — superseded by name-identity on main;
renderer-side remote DM delivery (deliverRemoteRosterMentions /
pollRemoteDmReply / ensureRemoteCanonicalChat) — superseded by the
message_agent tool architecture (#91802/#91915: middleware identifies,
never delivers); pointer-era test files deleted on main
- openStoredBotChat/createCanonicalChat: remote opens keep Desktop's
chrome home (keepAllProfilesScope: true on routed opens); local bots
keep the measured workspace re-home
- prepareBotSource: capability gate only — routed RPCs never require
activation authority
Make explicit registry qualification authoritative: only a current exact ID is accepted, while blank, malformed, unknown, or retired claims fail closed without endpoint inference. Restrict genuinely unqualified legacy descriptors to the shared full-envelope URL/Cloud/SSH matcher, reject zero or multiple matches, normalize SSH host/user identity, and prove remote-primary restoration keeps the exact (connectionId, profile) tuple.
Closes#90048.
Prior work by @teknium1 in #89719 and #88922, @andrexibiza in #90913, and @AndreasG78 in https://github.com/NousResearch/hermes-agent/issues/90048#issuecomment-5375227679 shaped this implementation. @saralilyb's #90006 remains downstream consumer context; the production stopgap is credited but excluded because registry primary does not prove route ownership.
The salvaged carve-out covered the PKCE file and Claude Code credentials
but missed the canonical wired-token location — auth.json
credential_pool.anthropic oauth entries. Discovery accepts those via
pool.has_credentials(), so the filter must too. Read-only dict access;
api_key pool entries intentionally stay excluded (E2E case 4).
The desktop explicit-only picker filter drops any provider row where
is_provider_explicitly_configured() is False. That gate only recognizes
active_provider, model.provider, and API-key env vars — so a user who
authenticated Anthropic via OAuth (Hermes device flow or Claude Code
~/.claude/.credentials.json) had the Anthropic row silently hidden from
the desktop model picker even though list_authenticated_providers()
accepted those same credentials when building it (model_switch.py has an
equivalent special-case in its discovery path).
Unlike ambient CLI tokens (gh -> copilot), an OAuth access token only
exists after an interactive login, so its presence is deliberate user
configuration. Add a narrow carve-out for the anthropic slug that keeps
the row when either OAuth reader finds an accessToken; the strict gate
still governs every other provider, and is_provider_explicitly_configured
itself stays untouched so PR #4210's consent guard for auxiliary tasks
keeps its current behavior.
The settings 'Applies to' store uses null = 'follow the active profile',
but the API helpers (profileScoped/capabilityScoped) use null = 'target
the primary/default backend'. Every page that passed the raw override to
a request silently read/wrote the primary profile whenever no override
was set — writes landed on the right profile via other paths while reads
repainted primary's values, so profile model changes appeared to revert
(#90549 class).
Close the class at the seam instead of per call site:
- store/settings-scope: new $settingsRequestProfile computed — the
request-shaped scope (string | undefined, never null). Documented as
THE value to hand to API helpers.
- config-settings, keys-settings, messaging: consume the request-shaped
computed; ModelSettings/MemoryConnect/ProviderConfigPanel/
useEnvCredentials props narrowed to string | undefined so a
primary-targeting null can no longer be plumbed through.
- keys-settings site was a live third instance: getEnvVars(null) read
primary's env store on non-default profiles.
Regression tests: store computed shape, ModelSettings unscoped+scoped
reads, KeysSettings unscoped fetch (all fail against the old behavior;
sabotage-verified).
ModelSettings passed scopeProfile (null when following the active
profile) directly to the Hermes API helpers. The helpers interpret
null as "target the primary/default backend", not as "follow the
active profile". This meant that when a non-default profile was
active (e.g. local with LM Studio), the model picker showed the
primary/default profile's providers instead — LM Studio was
invisible.
Fix: convert null → undefined before calling the API helpers, so
profileScoped() falls back to the app-wide active profile.
Added regression test verifying the helpers receive undefined (not
null) when no scope override is set.
Follow-ups on top of the salvaged #92665 commits, closing the two gaps
called out in #92647:
- SEARXNG_URL / CAMOFOX_URL now count as direct web/browser configuration
in _get_gateway_direct_credentials(), so env-only keyless local setups
are offered unchecked instead of pre-checked.
- Submitting the checklist with unconfigured tools left unchecked records
them in tool_gateway_declined_tools (new known root key) and stops
pre-checking them on subsequent Nous model swaps; opting in later clears
the decline. Cancel (Ctrl-C/ESC) records nothing.
- has_direct labels mention SearXNG/Camofox when that's what's detected.
- Docs: tool-gateway.md gains an enablement-checklist section.
The early "fail closed" returns (account fetch error, not entitled,
non-nous provider) still returned 3-element tuples after the function's
happy path and every caller moved to a 4-tuple, crashing
prompt_enable_tool_gateway with ValueError for any logged-in Nous
account that isn't paid or pool-entitled — the common case, hit
unconditionally from `hermes model`.
get_gateway_eligible_tools() classified a tool as "unconfigured" whenever
it found no direct API-key credential, ignoring an explicit non-nous
selection already stored in config (e.g. web.backend: searxng, a keyless
self-hosted backend). Every unconfigured tool is pre-checked in the
`hermes model` Tool Gateway checklist, so a single Enter during Nous model
setup silently rewrote web.backend (and similarly browser.cloud_provider,
tts/stt/image_gen.provider) to "nous" for users who had deliberately
configured a keyless local backend.
get_gateway_eligible_tools() now also resolves each tool's stored
selection via the existing _selected_provider() helper and routes an
explicit non-nous selection into a new explicit_configured bucket instead
of unconfigured. prompt_enable_tool_gateway() never offers those tools,
so they can no longer be pre-checked or accidentally overwritten.
Fixes#92647
On top of @686f6c61's premise-corrected #76745:
- _looks_like_desktop_control_plane now uses the parser-derived
_hermes_holder_subcommand instead of substring matching — the
#90778/#91869 class ('-m dashboard chat' and 'kanban --preserve-cache'
argv no longer read as control planes). Regression test added,
sabotage-verified (reverting to substrings fails it).
- Live E2E (this host, real processes + real spawn ledger): live
supervised serve owns lifecycle; killed spawner (orphan) does not;
dead serve entry excluded; empty ledger does not.
- Live Windows E2E for the wine2e lane: real self-registered ledger
entry suppresses the actual cold-start plan; dead serve restores it;
holder-scan fallback rung proves the token classifier live.
Co-authored-by: 686f6c61 <github@00b.tech>
Vestigial autostart is not proof the user wants a standalone gateway
run. When Desktop currently supervises this install's control plane,
the updater must not spawn a competing messaging daemon. Serve is not
treated as gateway-equivalent.
findExistingCanonicalChat() swallowed every lookup error and returned
null — indistinguishable from 'this bot has no Bot Chat yet' — so a
transient RPC failure against a just-restarted backend (the exact
post-desktop-update window) sent createCanonicalChat() straight to
session.create, minting a fresh 'Bot Chat' while the real one (data
intact, hidden) still held the canonical title. Users experienced this
as bots losing all context after every desktop update.
The lookup now fails CLOSED: a failed registry consultation throws,
both open paths surface their existing 'try again' toast, and
session.create can never fire off an unknown ownership state.
Tests: two new VM-executed regression tests (sabotage-verified — both
fail with the old fail-open catch); hide-bot-chats source-shape regex
updated for the new layout. 364/364 plugin tests green.
Both failed only in the full Linux suite, which the targeted local
battery never ran:
- test_update_zip_two_phase.py's AST guard (#76105) flags any code
literal "Scripts" in hermes_cli as an open-coded venv layout.
migrate_windows_bin_path's legacy PATH key now derives it via
venv_bin_dir(root / "venv", windows=True) — same value, canonical
helper. The literal `venv` component stays: the key must match what
the pre-#83797 installer wrote to the registry, not where the venv
lives now.
- The managed-bin marker tests built expected PATH entries from
tmp_path, so on a POSIX host they compared forward-slash strings
against the backslash markers and could never match. Markers match
Windows registry PATH entries, so the tests now feed Windows-shaped
literals — host-independent, same contract.
CI flake mechanism (PR #92617 red, reproduced standalone): tests plant
fake botocore modules via patch.dict; when the REAL botocore.exceptions
is first imported in an interpreter state where a fake parent is (or
was) installed, its 'from botocore.vendored import requests' resolves
against a module with no __path__ and every exception test in the worker
dies with "No module named 'botocore.vendored'" — ordering-dependent,
so green locally, red in CI workers.
Defenses (both, in depth):
- test_bedrock_adapter.py pre-imports the real botocore.exceptions at
module scope, before any test can stub sys.modules — later imports are
cache hits that can never re-execute the vendored import under a
poisoned parent. Proven standalone: fake-parent repro fails without
the pre-import, succeeds with it.
- autouse _boto_sys_modules_hygiene fixtures in all three files that
plant fake boto* modules (adapter, integration, model-picker):
snapshot every boto* sys.modules entry before each test, evict+restore
after — no stub window can leak state into a later test regardless of
worker ordering.
- importorskip targets botocore.exceptions (the module the tests
actually need) instead of bare botocore, so a torn install skips
instead of erroring.
148/148 across the four affected suites.