Hermes is an agent for one person. The credentials, the memory, the
sessions and the cron jobs all belong to that person. But the only
declarative path was a NixOS system service. Issue #9056 asks for the
user-level equivalent. 25 public Nix configurations already write one by
hand, and several of them copy nix/nixosModules.nix and edit the systemd
part.
This module is not a second copy of that file. The code that both modules
share moves into nix/moduleCommon.nix:
- the options
- the renderers for config.yaml, .env and the documents
- the activation body
- the command lines of the processes
nixosModules.nix keeps only the parts that need root. Those parts are the
service user, stateDir, addToSystemPackages, container mode and tmpfiles.
The file goes from 1008 lines to 666.
`services.hermes-agent` is now the same option set on both modules. A
NixOS example works on Home Manager without a change, and an option added
one time appears on both.
The Home Manager module is different only where it must be. It uses
systemd.user.services on Linux and launchd.agents on Darwin. It uses
home.activation and not system.activationScripts. It sets HERMES_HOME
directly, with the default ~/.hermes, so an existing directory continues
to work. It uses the modes 0600 and 0700, because the state has one user
and does not need the group-shared umask of the NixOS module. It does not
support container mode, which needs root and the Docker socket.
The change also makes four corrections that apply to both modules:
- backend.mode runs `hermes serve` or `hermes dashboard`. Both modules
had only the gateway. But Hermes Desktop and the web dashboard connect
to a different process, so six of the configurations in public repos
add a second unit by hand. serve and dashboard are one entry point with
one flag of difference, and you can run only one of them. Thus the
option is an enum. The NixOS module asserts against container mode with
a backend, and does not make a unit that cannot start.
- hermesHomeFiles installs files into HERMES_HOME. The `documents` option
installs into the working directory, which is correct for AGENTS.md but
wrong for SOUL.md and memories/. Hermes reads those files from
HERMES_HOME, in agent/prompt_builder.py:2095. A SOUL.md in `documents`
made a workspace file that Hermes never loaded as the identity. The
documentation said this in prose, but two directory diagrams showed the
opposite. This change corrects both. A key in either option can now
contain subdirectories.
- `documents` needs an explicit `workingDirectory`. The default of that
option is bad on both modules. It is the home directory of the user on
Home Manager, and ${stateDir}/workspace on NixOS. A user who declares
workspace files without a directory therefore gets a place that the
user did not select. The place is also different on each module. The
modules now refuse that combination.
The test is on the priority of the option and not on its value. An
option that nothing sets keeps the priority of its own default, and
each definition from a user is stronger. Thus a directory with the same
text as the default still counts as a selection, and so does a
mkDefault. A comparison of values detects neither case.
- Each activation writes .env again from a base in the Nix store, and
does not add to the file that exists. Thus a second activation cannot
put the same secret in the file two times, and a removed
environmentFile goes away. environmentFiles keeps the type `listOf
str` and not `path`, so Nix cannot copy a sops-nix or agenix path into
the Nix store, which all users can read.
- HERMES_MANAGED and the .managed marker now hold the name of the system
that manages the install. Thus a refusal says "managed by home-manager"
and not "managed by NixOS", and `hermes update` gives the Nix guidance
for both shapes. The CLI does not print a rebuild command for each
system. It names the owner, and the user knows their own tool. A bare
`true` and an empty marker still mean NixOS, so this does not change an
existing install.
Verification. Six new checks, all built:
nixos-module evaluates the module with evalModules and the
NixOS module list. It asserts both units, one
HERMES_HOME, and that the module refuses
container mode with a backend.
home-manager-module evaluates the module with the
homeManagerConfiguration function of
home-manager. The process assertions run against
systemd units on Linux and launchd agents on
Darwin.
module-option-parity asserts that each shared option is on both
modules, and that the two exclusion lists name
only options that exist.
env-file-assembly runs the real .env script and checks the
contents, the mode, that a second run gives the
same bytes, and that a removed file goes away.
workspace-files-need-a-directory
checks that the module refuses `documents`
without a directory, and accepts a directory
that has the same text as the default.
service-argv runs each command line that the modules build
through the real parser of the CLI, with one
sentinel flag added, and requires that argparse
refuses only the sentinel.
`nix flake check` passes, with 21 checks in total.
The CLI branches that treat an install as a Nix install move to one
helper, is_nix_install_method. Four call sites in main.py, web_server.py,
update_cmd.py and doctor.py tested the literal set {"nix", "nixos"}, and
each one missed home-manager. recommended_update_command asks the managed
state before the code-scoped stamp again, because a managed install can
carry a stale stamp that names an update path the managed guard refuses.
The metrics contract gets a home-manager bucket, so a Home Manager
install does not report as unknown.
Each check was mutation-probed. 22 faults were injected, and the checks
caught all 22:
- a lost --no-open
- a backend that runs the gateway
- an overwritten config.yaml
- documents in the wrong directory
- a different HERMES_HOME on the two processes
- a lost HERMES_HOME export
- a missing backend unit
- a removed assertion
- an .env file that grows at each activation
- an install that reports NixOS
- an empty .managed marker
- an option on the NixOS module only
- a stale entry in an exclusion list
- a renamed subcommand
- an unknown flag
- the workspace-files assertion always passes
- the assertion compares values instead of priorities
- an off-by-one that lets an untouched default through
- the assertion also fires for hermesHomeFiles
- a mkDefault no longer counts as a selection
- the Home Manager module stops wiring the assertion
- the NixOS module stops wiring the assertion
The 16 Python tests in tests/hermes_cli/test_managed_install_shapes.py
were probed the same way. 8 faults were injected and 8 were caught.
These tests fail on this tree. They fail in the same way on the stashed
HEAD, and they have no relation to Nix:
- test_git_probe_tree_kill.py (2 tests)
- test_update_import_guard.py (1 test)
- test_telegram_media_read_timeout.py (2 tests)
- test_teams.py (a collection error)
Closes#9056
# Conflicts:
# hermes_cli/main.py
# hermes_cli/update_cmd.py
# hermes_cli/web_server.py
wake instead of the full runtime boot (#89206 class)
zero trust's third bundle (on ae6578af, both prior fixes present) showed the
remaining failure: cold profile backends on slower Windows machines take
47-120s to fully boot, while the wake path's fixed budgets (20s hydration,
~15s resume retries) raced the whole boot and lost — "errors waking up BOTS"
while the backend came up healthy moments later.
Rather than raising timeouts, make the wake cheap:
- waitForFocusedSessionHydration: a history-bearing chat is hydrated when
the persisted transcript is PAINTED on the right session. The REST
prefetch delivers that seconds after the backend's HTTP is up; the full
runtime resume (agent build, MCP discovery, 114-skill load) keeps warming
in the background and binds the composer when it lands. Only an
expected-empty chat still waits for the runtime (nothing to paint).
- On hydration timeout, log a [bot-wake] phase breakdown (activation ms,
hydration ms, which conditions were unmet) to the renderer console so the
next support bundle pinpoints the slow phase directly.
- web_server: flush the headless "listening" line — block-buffered on the
Desktop's piped stdout, it surfaced minutes late and made boots look far
slower than they were in support bundles (the 120s "gap" in this bundle
was partly this artifact).
Sabotage-proven: restoring the runtime-gated wait fails the new paint-first
test by timing out — the exact field shape.
tools/skills_sync.py bound HERMES_HOME / SKILLS_DIR / MANIFEST_FILE at
import time — the third module in the same lineage as skills_tool
(f8723c478) and skill_manager_tool (c6a3d412d). In a long-lived
dashboard/TUI process, console skills commands (reset, diff,
list-modified, opt-in/out, repair-official) dispatched in-process under
_profile_scope's set_hermes_home_override(), but skills_sync's frozen
constants kept resolving against whichever profile was live at import.
Sharpest edge: reset_bundled_skill()'s #48200 rmtree strict-child guard
was computed against the WRONG skills root.
Fix: same call-time accessor pattern as the two prior fixes —
_hermes_home()/_skills_dir()/_manifest_file() honor an explicitly
patched module global (tests, retargeting) and otherwise re-resolve
from the live profile-scoped get_hermes_home() on every call. All 37
call sites migrated; module constants kept for compat.
Also documents in _profile_scope() that skills_sync needs no module
retargeting since the contextvar override now reaches it.
Regression tests (sabotage-verified: all 3 fail on the old binding):
- accessors follow set_hermes_home_override at call time
- explicit module patch still wins over the override
- rmtree guard anchors on the overridden profile's skills root
Fixes#65828
`hermes profile rename default <name>` (and the Desktop/dashboard rename
flows) now set a presentation-only `display_name` in profile.yaml instead
of erroring. The canonical id stays "default"; resolution, comparison,
and spawn paths are untouched. Named profiles keep real renames and their
display_name survives the move.
Surfaces: profile list/show/status, /profile (text only — data.profile
stays canonical), dashboard ProfilesPage, TUI-gateway profiles.list, and
Desktop (rail, switcher, Manage page, and the Bot Mode roster via a
displayName fallback so a renamed default shows its name, not "default").
Slimmer redo of the direction in PR #87760 by @yxssxn — thanks; see PR
body for what changed vs that approach.
Backend: /api/status now carries a stable random install_id persisted once
under the root HERMES_HOME, shared by every profile of the install.
Desktop: roster enumeration captures it per connection (TTL-cached probe),
buildAgentRoster collapses same-install rows with a deterministic canonical
pick (active > local > ssh > remote > cloud > earliest), the @name-device
handle rule runs after the collapse, and Settings → Gateways shows a
display-only 'Same backend as' hint. Backends without install_id bypass the
collapse (fully backward compatible).
Wire the continuity flag through every cron-creation surface, not just the
model tool:
- dashboard (web/): checkbox in the cron job editor; form state round-trips
the stored reserved 'self' entry into the toggle and strips it from the
context_from textarea; web_server dashboard validator skips 'self'
(create precedes the job's existence)
- Bot Mode Routines tab (hermes-bots plugin): Continuity checkbox in the
New Cronjob dialog, forwarded through cron.manage
- tui_gateway cron.manage RPC: optional continuity param on action=add
vitest cron-job suite 10/10 (4 new), tsc app project clean, py_compile clean.
Follow-up to #80451. /api/status cleared gateway_platforms whenever the
gateway process was down — correct for a clean stop (stale 'connected'
states are noise) but wrong for startup_failed, where the fatal entries
ARE the diagnosis: per-profile credential collisions and auth failures
(multiplex '<profile>:<platform>' keys) that the single exit_reason
string cannot express. #80451's writer-identity and freshness filters
already drop entries from other/older processes, so preserving
fatal-state entries here cannot leak another gateway's live state.
Live-validated shape: a real multiplex gateway (2 secondary profiles,
rejected tokens) persists telegram / alpha:telegram / beta:telegram
fatals in gateway_state.json; /api/status previously reported {} for
platforms while state was startup_failed.
The freshness window (updated_at >= live process create_time - 2s) had a
P1 boundary hole: a stale failure written by the PREVIOUS process
immediately before a fast restart landed inside the slack and was
aggregated; if that platform was then removed, the new process never
replaces the entry and NAS stays degraded indefinitely.
Replace clock heuristics with persisted writer identity:
- write_runtime_status now stamps every platform entry with the writing
process's (writer_pid, writer_start_time) — the same PID-reuse
fingerprint the liveness checks use, so a recycled PID never
masquerades as the original writer.
- The aggregation ownership filter requires exact equality between an
entry's stamp and the profile's validated live gateway process
(get_runtime_status_running_pid + _get_process_start_time). No slack,
no timestamps. Legacy entries without a stamp fail closed.
- Writer stamps are process recon (same class as the auth-gated
gateway_pid) and are stripped from all /api/status projections, both
active-profile and merged cross-profile entries.
Near-boundary regression test: prior-process entry stamped 100ms before
restart is excluded; recycled-pid-different-fingerprint excluded;
legacy no-stamp excluded; current-process entry kept.
Gateway startup deliberately preserves plain platform entries in
gateway_state.json across restarts, and the active-profile endpoint
compensates by filtering against current configuration. The cross-profile
aggregation copied raw maps, so a fatal entry for a platform the operator
had since disabled/removed could keep NAS reporting the instance degraded
indefinitely.
The aggregation has no cheap per-profile config context (platform sets
depend on tokens in each profile's .env behind its secret scope), so use
freshness instead: an entry is aggregatable only when its updated_at is
at/after the live gateway process's create time (validated PID via
get_runtime_status_running_pid + psutil create_time; the record's own
start_time field is a PID-reuse fingerprint in clock ticks, not a
timestamp). Config changes require a restart to take effect, so
restart-anchored freshness is exactly the config filter's semantics.
Fail closed: unparseable timestamps or no live process exclude the entry
— a false 'degraded forever' is the worse failure mode.
- /api/status now folds LIVE independent per-profile gateways' platform
failures (gateway_mode == 'multiple', the OOF-3 deployment mode) into
gateway_platforms under the validated <profile>:<platform> grammar, so
NAS fleet health sees them without a schema change. ?profile= requests
stay unmerged (single-profile view).
- Namespaced-key validation no longer fails open: colon-containing keys
are grammar-checked even when configured-platform loading throws.
- Platform key segment now accepts hyphens, matching plugin platform IDs
(plugins/platforms/<dir> names, e.g. foo-bar).
Since the managed-cron redesign (#84339, v2026.8.13) the dashboard fire
webhook forwards fires to the gateway process and returns 503 when it is
unreachable so NAS/QStash retries. Correct for transient windows — but an
operator-STOPPED gateway can never be fixed by retrying: every fire on
every job burns the full scheduler retry budget, NAS converts each 503 to
a retryable 502, and the resulting storms page on-call for a non-incident
(OOF-266 and its five duplicate tickets; +93% relay callback failures as
the fleet adopted v2026.8.13).
Split the unreachable path by durable operator intent:
- desired_state == "stopped" (written only by the s6 lifecycle commands;
the same intent signal container-boot reconciliation trusts) -> drop
the fire with 200 + a structured log line, mirroring NAS's own
instance_stopped drop. Jobs are not lost: the Chronos provider
reconciles and re-arms every job on the next gateway start.
- Anything else (crash loop, scale-to-zero wake, restart, legacy state
file without desired_state) -> keep the retryable 503, now stamped
with Retry-After: 60 so a scheduler that honors it spaces retries
past the wake/restart window instead of exhausting them inside it.
The gateway's own pass-through 503s (draining) get the same hint.
The intent check fails open (any parse/resolution error -> retryable
path) and is only consulted when the gateway is actually unreachable, so
a stale state file can never shadow a live gateway.
When the backend is spawned profile-scoped (`--profile <name>` sets
HERMES_HOME=<root>/profiles/<name>), _discover_dashboard_plugins()
scanned only get_process_hermes_home()/plugins — the profile directory,
which has no plugins/ content. Pooled per-profile backends therefore
discovered zero user plugins, mounted no plugin API routes, and every
plugin REST call fell through to the SPA catch-all 404.
Also scan get_default_hermes_root()/plugins (which unwraps
<root>/profiles/<name> to <root> and leaves a custom HERMES_HOME
untouched when it is itself the root), matching how hermes_cli.plugins
resolves install locations. The profile home is scanned first, so a
profile-local plugin of the same name stays authoritative via the
existing seen_names dedupe.
Adds regression tests for root-plugin discovery under a profile-scoped
process and for profile-over-root precedence.
Fixes#87197 (plugin discovery half — the misleading /api/* catch-all
half is addressed separately in #87270).
- cli.py breadcrumb call sites use getattr so object.__new__/SimpleNamespace
test fakes without the mixin method don't AttributeError (pitfall 17)
- web_server: fold one-field 'session' category into 'general' per the
single-field-category test contract
After `hermes update`, an existing state.db on an old schema made every
GET /api/sessions poll fail with sqlite3.OperationalError "no such
column: s.last_read_at" (or s.last_activity_at) until something
unrelated forced a writable open — the desktop sidebar showed "No
sessions yet" while every row sat intact on disk (#79531, #80037).
Two remaining root causes (the stale hand-written read probe was
already replaced by the SCHEMA_SQL-derived probe on main, prototyped in
draft PR #80030 by @Tilly-YL):
1. Migrations ran lazily: _init_schema/_reconcile_columns only ran on a
writable open, typically the user's first NEW session. The dashboard
backend now schedules one writable open of its own state.db from the
lifespan (daemon thread, never blocks the ready-probe socket, never
raises), so the store is brought current before the first session-
list poll on every `hermes serve` / `hermes dashboard` / Desktop
headless entrypoint.
2. _reconcile_columns caught sqlite3.OperationalError around every
ALTER TABLE ADD COLUMN and logged at DEBUG. Lock contention from
orphaned sibling backends made the ALTER fail silently — startup
"succeeded" with a half-reconciled schema, and the open-time lock
patience (#74478) never saw the error because it was swallowed
inside first. Now: "duplicate column" races stay at DEBUG,
locked/busy re-raises so _connect_and_init_with_lock_patience
retries the whole idempotent init with jittered backoff, and any
other failure (e.g. un-ADDable NOT NULL) logs at WARNING.
Regression tests: a store missing sessions.last_read_at is healed by
the eager startup reconcile and serves list_sessions_rich; a locked
ALTER propagates and is retried to success by the open lock patience;
duplicate-column races stay quiet; other ALTER failures warn.
Fixes#79531Fixes#80037
Reported-by: @yenhunghuang (#79531) and @FLOW3R0111 (#80037)
Root-cause analysis: @wangyi0177-eng (stale read probe) and
@www654cc-pixel (_reconcile_columns DEBUG-swallow under lock
contention); draft PR #80030 by @Tilly-YL prototyped the probe fix.
- The gateway api_server fire webhook acknowledges 202 only after a
durable claim + execution row exist (admission failure stays retryable
as 503; a live claim answers 200 duplicate), then dispatches the
claimed snapshot with the live runner adapters (delivery parity with
the built-in ticker, including relay-fronted and E2EE platforms).
- Legacy single-phase providers (a documented fire_due override without
split hooks) keep being driven through their own hook. Capability
detection now credits claim_fire AND fire_claimed overrides, so
Chronos is correctly classified split-aware (its re-arm lives in
fire_claimed; the redundant fire_due passthrough override is removed).
- Multi-profile dashboards fail closed for external providers: an
unscoped reconcile would disarm other profiles' armed one-shots in the
shared NAS registry.
- Manual runs (cronjob run) carry the owner-bearing claimed snapshot
through every entry point, composing with upstream's manual-run
heartbeat (#76502) and background dispatch.
Note: current main moved the dashboard NAS webhook to a pure
forward-to-gateway design (the gateway owns execution and live
adapters), so the dashboard-side claim/tracking machinery from earlier
revisions of this PR is dropped; the durable admission contract lives in
the gateway webhook path.
Persist backend ownership for reliable cleanup, park inactive panes, and
evict unreferenced transcripts so Desktop stays responsive over long sessions.
💘 Generated with Crush
Assisted-by: Crush:gpt-5.6
* feat(dashboard-auth): extend RFC 8252 native sign-in to password providers
The desktop app runs password sign-in for gated gateways in an embedded
Electron BrowserWindow, where OS password managers (macOS Passwords /
iCloud Keychain autofill) cannot reach the form — Chromium-in-Electron
has no bridge to them, so users retype credentials by hand even though
the /login form already carries the right autocomplete attributes.
The existing RFC 8252 native flow (system browser + loopback + PKCE)
solves exactly this for OAuth providers, but was explicitly disabled for
password providers on the grounds that they have "no IDP round trip to
broker". The brokering is still worth having: it moves the credential
form into the system browser, where password-manager autofill just works.
Gateway-only change; the desktop needs no changes (runNativeLogin is
already page-agnostic), and older desktop builds pick the capability up
automatically once the gateway advertises it:
* /auth/native/authorize now accepts a supports_password provider:
register the pending broker authorization as usual, then 302 the
system browser to the interactive /login form with the opaque
broker_state in the gateway's PKCE cookie (the same server-controlled
channel the OAuth branch uses) instead of an IDP redirect.
* /auth/password-login: when the server-set PKCE cookie carries a
broker handle, a successful credential check completes the pending
authorization exactly like the /auth/callback native branch — mint
the one-time loopback code, return the loopback redirect (validated
loopback-only at authorize time) as `next`, clear the PKCE cookie,
and set NO session cookies. A lapsed broker is a clean 400 telling
the user to restart sign-in; a failed credential attempt leaves the
pending entry intact so the user can retype.
* /api/status now advertises "native_pkce" whenever any interactive
session provider is registered (previously only for non-password
providers), so the desktop selects the system-browser strategy for
password-only gateways.
Security posture is unchanged from the existing flow: loopback-literal
redirect_uri enforcement, PKCE S256 binding, single-use short-TTL codes,
constant-time comparison, and the same rate limiter on password attempts.
Tests: full authorize → /login → password-login → loopback → token →
bearer round trip, wrong-password keeps the pending entry, lapsed broker
→ 400, no-broker browser login keeps minting cookies, and the /api/status
advertisement for password-only gateways.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(dashboard-auth): bind native password completion to the authorize-time provider
Review follow-ups for #75808:
* /auth/password-login now enforces that body.provider matches the
provider recorded in the server-set PKCE cookie by
/auth/native/authorize before completing a pending native
authorization. /login renders a form for every session provider, so
without this a native flow started for provider A could be completed
with provider B's credentials, binding B's session into A's pending
entry. The mismatch is rejected BEFORE credential verification (no
session minted, no oracle) and preserves both the pending entry and
the cookie, so the user can still submit the correct provider's form.
Covered by a two-password-provider E2E regression test.
* Update the two docs spots that still said password-only providers do
not advertise native_pkce (website desktop-native-signin guide and the
auth_flows type comment in web/src/lib/api.ts).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: map contributor email for #75808 (buffpesos)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Brooklyn Nicholson <brooklyn.bb.nicholson@gmail.com>
* fix(dashboard): suppress Ctrl+C shutdown traceback
* fix(dashboard): extend clean Ctrl+C exit to the Windows serve branch
The Windows loop-factory branch (and its pre-0.36 asyncio.run fallback)
runs under the same uvicorn capture_signals() re-raise as the POSIX
path, so console Ctrl+C leaked the identical KeyboardInterrupt
traceback there. Guard both serve calls with the same clean-exit
contract, keeping the import-resolution try/except comment accurate
(genuine serve-time errors still propagate).
Also ports the reworded POSIX-test docstring (the serve path is no
longer 'byte-for-byte unchanged'), wraps the POSIX KI test in
pytest.fail so a regression reports red instead of aborting the pytest
session, and adds the windows_only sibling test.
Extends #52970 to the whole bug class.
* chore: map contributor email for @wangs1203
* test(dashboard): actually exercise the pre-0.36 Windows fallback KI contract
Copilot review caught that patching uvicorn._compat.asyncio_run with
raising=False makes the import succeed, so _runner is non-None and the
extra asyncio.run patch never covered the fallback. Split it out: a
dedicated windows_only test halts the _compat import (None in
sys.modules) so the fallback branch is genuinely selected, then asserts
the same clean-KI contract on bare asyncio.run.
---------
Co-authored-by: Emiya·Leon <wangs.coder@gmail.com>
Adds hermes_cli/model_selection_guards.py: a single evaluation point that
runs every selection guard (cost + the new data-policy guard) and returns
the warnings that fired. All seven model-selection surfaces (CLI picker,
cli.py TUI modal, gateway typed /model, dashboard web_server, TUI gateway,
Telegram and Discord pickers) now call the registry instead of importing
model_cost_guard directly — so the data-training-tier warning from
PR #81416 fires everywhere at once, and future guards need zero surface
wiring.
Guard modules keep their public APIs; existing mock patch points
(hermes_cli.model_cost_guard.expensive_model_warning) remain valid.
Extends the NS-656 memory-pressure surface to cover disk exhaustion
(OOF-2 / OOF-107 lineage: agents fill their data volume — SQLite writes
fail, sessions stop persisting — while every dashboard looks healthy).
- gateway/disk_status.py (new): collect_disk_status() samples
shutil.disk_usage(HERMES_HOME) and classifies pressure
(critical: <256 MB free or >=95% used; elevated: <512 MB free).
Never raises — degrades to pressure="unknown" with null telemetry,
same contract as collect_memory_status().
- /api/status: sibling `disk` block next to `memory`, advisory only —
not folded into component/overall health.
- web: DiskPressureStatus type; MemoryPressureBanner generalized to a
resource banner with worst-first triggers (disk critical > memory
critical > OOM restart > disk elevated > memory elevated) and
cascading dismissals — hiding the top trigger surfaces the next one
instead of silencing everything. All dismissals stay boot_id-scoped.
- i18n: diskCriticalBanner / diskElevatedBanner (en, optional fields
with English fallback per existing pattern).
Tests: gateway/test_disk_status.py (14), web_server disk-block
presence/degradation, banner disk trigger/priority/dismissal-cascade
suite (21 total).
Hosted agents can be OOM-killed hourly while the dashboard and the NAS
agent card both look perfectly healthy — every memory signal the gateway
already produces (heartbeat mem samples, lifecycle-ledger unclean-exit
verdicts, cache-pressure evictions) dies in server-side log files. The
BlueAtlas incident (NS-608) ran for three days like this.
This is the read-side fix:
* New gateway/memory_status.py distills the existing 30s loop heartbeat
(gateway RSS + system MemAvailable/MemTotal + swap) and the lifecycle
sentinel into a compact `memory` block: pressure ok/elevated/critical/
unknown, coarse MB numbers, and last-boot unclean/suspected-OOM flags.
Pure file reads, no new sampling, no gateway IPC. Stale (>150s) or
future-dated heartbeats degrade pressure to "unknown" so a dead
gateway's final gasp can't render a live "critical" banner forever.
Critical thresholds mirror the ledger's OOM-suspicion heuristics: if a
level would make a later unclean death "suspected OOM", warn at that
level while the process is still alive.
* lifecycle_ledger.record_startup now carries prior_unclean_exit /
prior_suspected_oom onto the reclaimed sentinel — previously the
verdict survived only in append-only diag prose. Flags age out on the
next sentinel rewrite (scoped to the life after the crash).
* /api/status serves the block (profile-aware, executor-offloaded,
fail-safe to pressure=unknown). Deliberately NOT folded into
components/overall: memory pressure is advisory, and flipping overall
to "degraded" on it would page NAS's availability sweep for a
condition the eviction valve is already handling. Public-safety:
coarse numbers/enums/booleans only — same disclosure class as the
existing nous_session_valid field, added for the same NAS-sweep
audience.
* Dashboard: new MemoryPressureBanner (app-shell, next to
ProfileScopeBanner) with worst-first trigger precedence
(critical > suspected-OOM restart > elevated), per-trigger
session-scoped dismissal, and escalation re-opening past a dismissal.
i18n keys optional with English fallbacks, matching the
managingProfileBanner convention.
Tests: gateway/test_memory_status.py (classification bands, staleness,
clock skew, corrupt files, bool-is-not-int), lifecycle sentinel
carry-forward, /api/status contract (block always present, collector
crash degrades instead of 500), and 7 banner component tests.
NAS-side ingestion (agent-card notice + memory-tier upsell) ships
separately.
Refs NS-656; context: NS-608, NS-657, OOF-77.
- web_server CONFIG_SCHEMA: fold the one-field models_dev category
(models_dev.url) into the agent tab via _CATEGORY_MERGE, matching the
established pattern for single-field categories (slice 7,
test_no_single_field_categories).
- image_routing._lookup_supports_vision: pass allow_network=True to
get_model_capabilities. The vision-capability lookup runs when an
image actually needs routing (not per conversation turn), and the
#31179 text-only-main guard depends on catalog data — with the new
allow_network=False default a cold cache returned 'unknown', which
falls back to attempting the call and reintroduced the #31179
failure shape (slice 8, test_text_only_main_skipped_when_no_
aggregator). This preserves that path's historical
network-on-cold-cache behavior; the fetch stays 4h-TTL cached and
backoff-limited.
Renames the openai-codex provider's display label across the CLI
(hermes model picker, provider labels), the dashboard OAuth accounts
catalog, and the Desktop onboarding + settings provider pickers.
Slug, aliases, and auth flows are unchanged.
* fix(gateway): pass live adapters to cron fire webhook's fire_due
The Chronos fire webhook (/api/cron/fire) called
provider.fire_due(job_id, adapters=None, loop=loop), so every
externally-triggered fire delivered through the standalone path even
with a live gateway in-process. E2EE platforms and relay-fronted
logical platforms (whose ONLY send path is the live relay adapter — no
native credential exists on the box) failed every external fire with
"platform 'X' not configured/enabled", while the same job delivered
fine under the built-in ticker (gateway/run.py passes runner.adapters).
Resolve the runner (self.gateway_runner → app['gateway_runner'] →
_gateway_runner_ref(), the same chain the drain check uses) and forward
its adapters. No runner → adapters=None, preserving the historical
standalone path byte-identically.
Note: does not by itself fix Fly-hosted scale-to-zero deployments where
NAS's callback lands on the DASHBOARD process (internal_port 9119) —
_fire_cron_job_for_profile there has no gateway runner in-process. That
topology needs a separate fire handoff (design pending).
* fix(cron): dashboard forwards Chronos fires to the gateway (503 when unreachable)
The dashboard's /api/cron/fire executed cron jobs in the DASHBOARD
process via _fire_cron_job_for_profile with adapters=None. On hosted
deployments (Fly proxy exposes only the dashboard's port) that made
every managed-cron fire deliver through the standalone send path, which
cannot serve relay-fronted logical platforms (their only sender is the
live relay adapter in the gateway process — no native credential exists
on the box) or E2EE rooms. It also ran the whole agent turn inside the
dashboard: wrong process for memory/session ownership and fire-claim
attribution.
Restore the invariant that the GATEWAY owns cron execution:
- Dashboard route: after verifying the NAS JWT and resolving the job's
profile, FORWARD the fire to the gateway api_server's own
/api/cron/fire on loopback, NAS bearer preserved (the gateway
re-verifies the JWT — defense in depth, no new trust link), and pass
the gateway's response through. Gateway unreachable → 503 so NAS
retries per the Chronos contract (non-2xx = retryable; the store CAS
de-dupes the eventual double fire). Deliberately NO local-execution
fallback.
- Endpoint resolution mirrors gateway/config.py's api_server load order
per target profile (config.yaml extra.port → API_SERVER_PORT from
process env or the profile's .env → 8642), with /p/<profile>/ prefix
routing under multiplex.
- docker/stage2-hook.sh: generate a strong API_SERVER_KEY into .env on
first boot when absent (never overwrites an operator value), so the
loopback api_server passes its startup guard on hosted images. The
fire route itself is NAS-JWT-authed; the key gates the rest of the
api_server surface. The listener binds 127.0.0.1 by default and the
Fly service exposes only the dashboard port.
- _fire_cron_job_for_profile kept but deprecated (late-binding seam
compatibility); no route calls it.
- docs/chronos-managed-cron-contract.md: document the two-hop inbound
topology and the 503-retry semantics.
Depends on the previous commit (fire webhook passes live adapters to
fire_due) — together they make NAS→dashboard→gateway fires deliver over
relay end to end.
* fix(cron): read the profile api_server port via the canonical config loader
CI guard test_config_read_guard flagged the new _gateway_fire_endpoint
for a raw yaml.safe_load of the profile's config.yaml — the exact drift
class the guard exists to kill (raw reads miss the managed-scope
overlay, ${ENV_VAR} expansion, and root-model normalization).
Read through load_config() under a HERMES_HOME override scoped to the
target profile instead (the same pattern the deprecated
_fire_cron_job_for_profile uses for its store scope), and pull the port
with cfg_get. Test updated to stub load_config rather than write a raw
config.yaml.
* fix(gateway): only messaging platforms count for the scale-to-zero arm gate
The stage2 hook now generates API_SERVER_KEY for every Docker container,
and key presence force-enables the api_server platform. The scale-to-zero
arm gate counted every enabled platform, so the loopback api_server
listener made messaging_is_relay_only_or_absent False on every hosted
instance — silently disarming the feature (the not-armed log would show
enabled platforms=['relay','api_server']).
The arm gate and the not-armed logger now share one helper that filters
to enabled MESSAGING platforms, excluding LOCAL/API_SERVER/WEBHOOK —
the same non-messaging exclusion set _connect_platforms already uses.
A genuinely enabled direct-socket platform (Discord/Telegram) still
disarms. Two of the three new tests fail without this fix.
The custom-endpoint REST handlers ran bare load_config/save_config, so
every add/activate/delete landed in the process-level default profile
regardless of which profile the desktop settings UI was targeting. A
provider added under a non-default profile silently went to default:
visible only in default-bound sessions, absent everywhere else, and
un-addable to another profile without hand-editing its config.yaml.
Scope all four handlers (list/upsert/activate/delete) to the requested
profile via _config_profile_scope, matching /api/config, and spread the
active profile into the four hermes.ts wrappers alongside their existing
validateCustomEndpoint sibling.
An unreadable profile dir made (entry / '.env').exists() raise
PermissionError out of the sidebar fallback, 500ing /api/profiles.
Found by hostile fixture during live E2E of the scandir conversion.
Two-part fix for the dashboard fd exhaustion reported in #81547:
1. Raise RLIMIT_NOFILE soft limit on startup (before uvicorn binds).
macOS defaults to 256 for LaunchAgent processes — too tight for the
dashboard which opens 3 fds (db+wal+shm) per SessionDB per request
across all profiles. After days of polling the soft limit exhausts
and every os.listdir/open raises OSError [Errno 24]. The helper raises
to the hard limit (or minimum 4096), matching the reporter's ulimit
workaround. No-op on Windows (no resource module).
2. Replace bare Path.iterdir() with context-managed os.scandir() in four
dashboard hot paths: _fallback_profile_dicts, file manager list,
checkpoint listing, and plugin discovery. iterdir() returns a
generator that holds an open directory fd until fully consumed; if
an exception interrupts iteration the fd leaks. os.scandir() is an
explicit context manager that guarantees close on exit, following
the same idiom already used in /api/fs/list.
Tests: 6 passed, 3 skipped (resource-module tests skip on Windows).
_spawn_gateway_restart() now calls _reap_unsupervised_gateway_orphans()
before spawning a new `hermes gateway restart` child. On desktop-app
restart the old serve exits but its gateway child gets reparented to
launchd (PPID=1) and keeps its platform connection alive. The new
serve then spawns a fresh gateway, resulting in two live gateways
racing the same connection.
The reap was already implemented for the CLI restart path (#75936) but
the dashboard's _spawn_gateway_restart path was not covered.
Fixes#77276
On Desktop serve startup, reap orphan gateway processes (PPID=1) left
behind by a previous serve session that exited abnormally. This prevents
the old and new gateways from racing for the same QQ WebSocket
credential, which splits messages across parallel session trees (#77276).
Production incident: the orphan reap killed a legitimate SSH remote backend
started by another client machine. Its process sat at ppid 1 with the same
cmdline shape as a genuine orphan, and the exclusion list only covered THIS
app instance's children — ownership by OTHER clients was invisible.
The reap now treats every backend.lock.json under ~/.hermes/desktop-ssh/*/
as an ownership claim: lock payloads are schema-validated (mirroring
remote-lifecycle.ts) and their PIDs are excluded both before the scan and
re-checked after it (defense in depth against a lock written mid-scan).
Regression tests cover the exact incident shape: a lock-owned PID and a
genuine orphan with identical process shapes — only the orphan is reaped.
Also: fold the new single-field `runtime` config category into `agent`
(_CATEGORY_MERGE) and fix an env leak in the serve-startup test
(HERMES_SERVE_HEADLESS restored via monkeypatch) so the combined suites
run green in any order.
When Desktop exits uncleanly, leftover `hermes serve --host 127.0.0.1 --port 0`
processes can be reparented to pid 1 and keep full MCP trees alive. The next
boot then stacks another backend on top of the corpses until EMFILE kills
sidebar/session APIs and tabs disappear.
- Detect Desktop-local serve shape (loopback + ephemeral port 0)
- Only reap processes whose ppid is 0/1 (true orphans)
- Spare fixed-port remote serves (e.g. --port 9119) and HERMES_DESKTOP_CHILD_PID
- Run at Desktop backend start (HERMES_DESKTOP=1) before parent-death watchdog
Complements parent-death watchdog (prevents future orphans) and configurable
nofile soft limit (capacity floor). Together these stop the multi-backend
pile-up cascade observed on macOS Desktop SSH/local installs.
An unclean desktop exit (crash / SIGKILL / update handoff) stranded every
`hermes serve` profile backend as an orphan (ppid=1) still serving, each
holding its MCP child subtree — 31 orphans / ~1.3 GiB RSS on one install.
Root causes + fixes:
- serve had no parent-death watchdog: add _start_parent_death_watchdog() in
web_server.py (mirrors slash_worker.py), gated on HERMES_PARENT_PID; os._exit
cascades to MCP watchdogs. No-op for standalone `hermes serve`.
- desktop passes HERMES_PARENT_PID in both serve spawn env blocks (main.ts).
- POSIX teardown now group-kills (process.kill(-pid, ...)) so MCP grandchildren
die too (backend-child.ts + waitForBackendExit SIGKILL fallback).
Windows path unchanged (forceKillProcessTree). Tests updated + passing.
Two follow-ups to the off-loop move, from external review (both verified,
the second larger than reported):
- Config read-modify-write handlers moved to worker threads could now
interleave — _CONFIG_LOCK covers each load/save individually, never the
span between them; the event loop used to serialize these accidentally.
New _CONFIG_MUTATION_LOCK (worker-threads only, so it can never block
the loop) held across the whole load→mutate→save span in all seven RMW
handlers. update_config_raw skipped: it's a full-document replace with
no server-side read, so a lock cannot close its client-side window.
- The review flagged two skills routes still taking _SKILLS_PROFILE_LOCK
on the event loop; a systematic audit of hermes_cli/web_routers/ found
24 on-loop routes (skills 5, mcp 9, tools 10, cron 1). All moved to the
same inner-_run + asyncio.to_thread pattern, mutating ones under the
mutation lock, uniform lock order (_SKILLS_PROFILE_LOCK →
_CONFIG_MUTATION_LOCK). Await-safe _config_profile_scope routes, plain
def routes, and already-threaded routes unchanged.
Regression tests: concurrent theme+font updates both survive (fails with
the lock nulled: "theme write lost to a concurrent font write"); event
loop stays responsive while the profile lock is held during GET
/api/skills. 214 tests passing across the touched suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The diagnostics loop watchdog caught GET /api/config freezing the gateway
event loop for >1s, stack-sampled blocking on _SKILLS_PROFILE_LOCK inside
_profile_scope. Any async handler that entered _profile_scope (process-wide
threading lock) or called load_config()/save_config() on-loop could stall
every chat and WebSocket at once while a slow lock-holder ran.
Move 28 such handlers to the existing inner-_run + asyncio.to_thread
pattern (contextvar-safe: the whole scope enter/body/exit stays inside one
worker thread). Handlers using the await-safe _config_profile_scope, plain
def endpoints (FastAPI threadpool), and tui_gateway's contextvar-only
decorator are unaffected and unchanged.
Regression test holds _SKILLS_PROFILE_LOCK in a thread while calling
GET /api/config and asserts an event-loop heartbeat keeps ticking; it fails
against the pre-fix code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up to the salvaged registration contract:
- share one _raise_if_cron_registration_error() helper for the two
byte-identical dashboard 424 except-blocks (web_server + cron router,
via the existing late() seam)
- add endpoint-level 424 coverage for /api/cron/blueprints/instantiate
(previously only the sync worker was tested)
- give chat/CLI surfaces a human-facing user_message() (job name, no
exception class name) and add a recovery hint (pause/resume or update
re-registers via provider reconcile) to the model/REST message
- consolidate five inline provider test doubles into one ABC-subclassing
make_cron_provider conftest factory; the web_server test double now
subclasses CronScheduler so an ABC rename fails loudly
- narrow the wrapper facade to keyword-only (**kwargs) and route the
tool's partial-failure return through tool_error()
ethernet