hermes -p <name> gateway status, hermes gateway status and hermes status (under
Serves:) list the /p/<profile>/<path> URL per inbound-port platform the live
multiplexer serves, read from the <profile>:<platform> ingress_url in the default
home's gateway_state.json (hermes_cli/gateway_multiplex_served.py). The dashboard's
messaging payload carries the same ingress_url and the Channels page renders it.
The dashboard's 409 guard now covers only api_server/webhook (the mirrored pair):
enabling Twilio/LINE/Teams/... on a secondary is allowed because the gateway serves
it.
What `hermes update` does, blockers and fixes, URL change for
inbound-port profiles, the post-create restart reminder, rollback, and
the `gateway migrate` reference row.
GET /api/gateway/migrate/plan returns the CLI plan JSON; POST
/api/gateway/migrate spawns `hermes gateway migrate --multiplex --yes`
detached (action log gateway-migrate.log). The Gateway card shows the
button only for a multi-profile install that is not yet multiplexed, and
disables it while listing the blockers.
After the fleet restart is verified healthy, `hermes update` runs the
migration preflight on installs with >= 2 profiles and at least one
per-profile gateway. No blockers: migrate (same path as
`gateway migrate --multiplex --yes`, deterministic, never prompts).
Blockers: print them with their fixes and the one-liner, change nothing.
Skipped on the exit-1 (stale fleet) path and on single-profile installs.
Moves a per-profile-gateway install onto one multiplexed default gateway:
table-driven preflight (duplicate credential via the gateway's own
fingerprint; secondary port-binders without a /p/<profile>/ ingress),
--dry-run, apply (stop + uninstall each secondary's service, record it in
<default>/gateway_migration.json, flip gateway.multiplex_profiles through
the config API, restart/install the default on the same service manager,
verify served_profiles), and --standalone rollback from the manifest.
Idempotent; refuses cleanly when already multiplexed or blocked.
`hermes profile create` points at `hermes gateway restart` when a live
multiplexer is detected (the served set is snapshotted at startup).
The multiplexer skips a secondary profile that enables a port-binding
platform, unless the default listener already answers that platform under
/p/<profile>/. Which adapters do is now a class attribute on the adapter
(api_server and webhook today) instead of knowledge scattered in prose, so
the migration preflight can tell "URL changes" from "profile would be
skipped" and stays correct as new HTTP-inbound adapters gain the prefix.
DeepInfra catalog (fetched with the launch env's key via os.getenv), Copilot
context limits (api_key ignored on hit), Nous reasoning caps + once-per-process
guards, the curated OpenRouter list, the model-catalog in-process copy (mtime
without path), banner skills, the guest-mint back-off flag and the active skin
were single slots read under per-profile overrides by the gateway and the TUI
gateway; the SWR refresh thread ran without the caller's ContextVars.
Under an override each lives per home key (hermes_cli/models_profile_cache.py
holds the shared slot helper so models.py does not grow), credentials are read
through the scope-aware dotenv reader and keyed by fingerprint, and background
refreshes run under copy_context(). Unscoped behaviour is byte-identical.
The three helpers each restated why sudo moves the naming basis; keep it in _profile_suffix and leave the helpers their unique reasons. Drops a footgun marker the scanner has no pattern for and a raising=False on an attribute that exists.
`_bare_unit_pinned_home()` read the system unit for every caller, so an
unprivileged `hermes -p kimi gateway status` (user scope) resolved
`hermes-gateway` instead of `hermes-gateway-kimi` whenever the bare system
unit pinned that profile home — aliasing the profile onto the user's default
unit. Only an elevated process operates the system unit, so gate on root.
Also drops the unreachable `OSError` arm (non-strict resolve swallows it) and
routes the legacy-unit search through `_SYSTEM_UNIT_DIR`.
`_native_service_homes()` re-implemented the root+SUDO_USER -> `pw_dir/.hermes`
resolution that `hermes_cli.main._resolve_sudo_user_profile_env` already did.
Both now call `hermes_constants.sudo_invoker_default_home()`; main.py appends
`profiles/<name>` to it.
Also removes the local `from pathlib import Path as _Path` (Path is a module
import) and narrows the except to `KeyError`: `import pwd` cannot fail once
`os.geteuid` exists, and `getpwnam(str)` raises nothing else.
Review follow-ups to the unit-anchored service identity.
`_bare_unit_pinned_home()` now returns early off Linux. `_profile_suffix()` is
shared by the launchd label/plist helpers, the Windows scheduled-task name and
the s6/multiplex `_current_profile_name()` fallback, and a systemd unit is not
an identity authority for any of them. The gate is `is_linux()` (a plain
`sys.platform` test) rather than `supports_systemd_services()`, which can shell
out to `systemctl is-system-running` on WSL and containers -- unacceptable in a
helper that runs on every name resolution.
Document why the unit-pinned check must precede the profile branch, and pin it
with a test: `sudo hermes gateway install --system` resolves the BARE name from
root's default home, then writes the invoking user's remapped home into the
unit, so the bare unit legitimately carries a `<root>/profiles/<name>` home. If
the profile branch ran first it would answer `hermes-gateway-kimi` for a unit
installed as `hermes-gateway`, which is the original bug class.
Three more regressions: the named-profile-pinned bare unit above; a run that
drives the real `_sync_hermes_home_from_systemd_unit()` instead of simulating
the adoption with `setenv`; and an unreadable unit, which must fall through to
the suffix branches rather than hand its bare name to an unrelated home.
The class is now `linux_only`, because the gate makes the behaviour genuinely
host-dependent -- so the tests belong on the host that has it, not behind a
faked platform.
Verified on a real Linux kernel (WSL2, Python 3.12.13), not by simulation:
7/7 pass on this branch; with `hermes_cli/gateway.py` restored from origin/main
and the tests kept, 4 fail with the reported symptom
(`'hermes-gateway-kimi' == 'hermes-gateway'`, `'hermes-gateway-54de6eee' ==
'hermes-gateway'`) and the 3 guard tests still pass. Whole file on Linux:
origin/main 4 failed/106 passed/1 skipped, this branch 4 failed/113 passed/1
skipped -- same four pre-existing failures, exactly seven new passes.
Refs #108674
`sudo hermes gateway start|stop|restart|status|uninstall|install --system`
resolved the systemd unit as `hermes-gateway-<sha256[:8]>` while the installed
unit is `hermes-gateway.service`, failing with `Unit ... not found` (exit 5).
The service name was derived from the CURRENT PROCESS's HERMES_HOME, and under
sudo that value changes MID-COMMAND: sudo strips HERMES_HOME and sets
HOME=/root, so the `_require_service_installed()` pre-flight resolved the bare
name and passed; `_sync_hermes_home_from_systemd_unit()` then adopted the
unit's pinned `HERMES_HOME=/home/<user>/.hermes` into `os.environ` (deliberate,
for runtime-status/PID reads), and every later `get_service_name()` took the
hash branch. Regression from the #105525 fix, which correctly moved the
comparison basis to `_get_platform_default_hermes_home()` -- right for a
temp-dir/Docker home, but wrong for an elevated process whose `~/.hermes` is
not the home that owns the unit.
Read the naming basis from the unit instead of the process: the installed
`hermes-gateway.service` is the authority on which home owns the bare name.
`_bare_unit_pinned_home()` parses that unit's pinned HERMES_HOME, and
`_profile_suffix()` accepts it alongside the platform-native default. This is
stable for every elevated identity, including `sudo -i` and cron where
SUDO_USER is absent, and for a custom HERMES_HOME pinned in the unit.
The #105525 guard is untouched: with no installed bare unit, a temp-dir/Docker
/custom home still keeps its own hashed suffix and can never resolve to -- or
uninstall -- the operator's `hermes-gateway.service`. Only the single home that
unit pins is recognised; an unrelated home stays suffixed. Nothing is memoized,
because `hermes_cli/profiles.py::_cleanup_gateway_service` swaps HERMES_HOME
mid-process and depends on re-derivation. The native-default check stays first
so the common path short-circuits before any file I/O.
Refs #108674
_load_global_auth_store is memoised on (path, st_mtime_ns). A write-through
to the root (borrowed Codex cooldown clear, xAI/Anthropic root rotation)
followed by a fallback read in the same mtime tick — coarse-mtime
filesystems (exFAT, some network/overlay mounts) — kept serving the
pre-write store, so the resolve path could log "quota restored" and then
raise quota_exhausted from the stale memo. _save_auth_store(target_path=...)
now drops the memo.
clear_codex_pool_quota_cooldowns decided "borrow the root?" inside a nested
closure via a tri-state Optional[int] return (None = no rows), which forced
`cleared or 0` and duplicated the rule persist_pool_entries already owns.
Decide once with _profile_owns_pool_provider + _borrowed_single_use_pool_root,
then lock/load/clear/save exactly one store. Behaviour is unchanged for every
(mode x profile rows x root rows) cell; the pre-lock decision races only a
concurrent `hermes auth add` in the profile, whose fresh rows carry no cooldown.
Adds the missing negative invariant: a profile that OWNS Codex rows never has
the root store touched (0 cleared, root byte-identical).
`_read_codex_pool_entries` had become a pass-through whose only residual
behaviour was taking the active-store lock around a read every other
`read_credential_pool` caller does unlocked (and which never covered the
root file it fell back to). Both consumers now call the helper directly.
`clear_codex_pool_quota_cooldowns` decided "profile owns rows?" with an
unlocked pre-read, then re-read the same file under the lock — a TOCTOU
against a concurrent `hermes auth add` and a wasted parse. It now tries the
active store under its lock and falls back to the borrowed root only when
that store has no codex rows.
With reads now inheriting the global-root pool, a profile hitting a stale
root cooldown probes quota, sees it restored, and calls
clear_codex_pool_quota_cooldowns() — which only ever edited the (empty)
profile store, so the next resolve raised quota_exhausted again forever.
Pick the store the same way agent/credential_pool.py persists borrowed
rows (_profile_owns_pool_provider / _borrowed_single_use_pool_root) and
lock/save against that path. The fallback test now also binds
profile-wins precedence; one new test pins the root write.
Under gateway.multiplex_profiles a secondary profile X is ticked, dispatched
and notified from the default profile's process, where os.environ holds the
DEFAULT profile's .env and X's values live only in the per-turn secret scope /
HERMES_HOME override. Every remaining read that skipped that scope made X
behave differently from `hermes -p X gateway run`:
- cron: HERMES_CRON_TIMEOUT, HERMES_MODEL (job/preflight fallback),
HERMES_CRON_MAX_PARALLEL, inflight allowance, prefill file and the script
timeout were bare os.getenv → the default profile's values; a job without a
model silently ran on the default's HERMES_MODEL instead of refusing.
cron/env_settings.py::cron_env_setting reads the scope (fire) or the ticked
home's .env (tick thread), plain environ when multiplexing is off.
- child env: the restart-safe cron worker, the Bot Chat delivery child and the
kanban worker inherited the launch profile's non-credential .env settings
and bridged TERMINAL_* policy (TERMINAL_ENV=docker, default's image,
HERMES_MODEL) — X's worker ran in the default's docker image on the
default's model. tools/environments/local.py::strip_launch_profile_env drops
them when the child targets another served profile.
- kanban: the worker --toolsets pin was silently dropped for every served
assignee (toolset probes call get_secret without a scope → swallowed
UnscopedSecretError); notifier pings, artifact uploads and the wake text ran
under the default's media policy / display language (only wake() was scoped).
- /loop: _post_turn_loop_completion hopped to the executor without contextvars,
writing the completed tick into the DEFAULT profile's state.db and leaving
X's row awaiting_response forever; the --until judge ran with the default's
aux credentials.
- background processes: a secondary's processes.json (scope-relative since
adf23550f5) was never read at startup; its processes were not re-adopted
and notify_on_complete notices were lost. Startup recovers every served
home under its scope; recovery adopts each session once.
- completion delivery: background_process_notifications was evaluated once per
drain for the ambient profile (default's mode for everyone; X's `off`
dropped a sibling's `all` event), recovered watchers used the default's
mode, HERMES_BACKGROUND_NOTIFICATIONS was read raw from environ;
_deliver_platform_notice used the default's GatewayConfig so a secondary's
notice_delivery: private went public.
Not changed: gateway/run.py and tools/async_delegation.py (PR #106742
rewrites both). Known residue left for the env-bridge lane:
HERMES_SESSION_STALL_TIMEOUT is bridged once from the launch config.
scheduler bug, not a parity gap; unchanged here.
The multiplexing default gateway now serves default + every live named profile
under profiles/. profiles_to_serve(multiplex=True) is a pure directory read
(tombstoned profiles skipped, never mkdir); every reader — gateway served set,
/p/<profile>/ prefixes for api_server + webhook, the named-profile standalone
guard, the Desktop cron ticker (its #108428 standdown for a profile owned by a
running gateway is unchanged) — drops the allowlist parameter.
Config v43 migration deletes the key from user config.yaml; DEFAULT_CONFIG,
GatewayConfig and the top-level yaml bridge no longer carry it.
BREAKING: anyone who set an allowlist now has their excluded profiles served.
Archive or delete a profile you do not want served (Teknium approved).
hermes_cli/mcp_startup held one _mcp_discovery_started flag and one thread
for the whole process, so in a shared Desktop/dashboard backend the first
profile to build an agent won discovery and every later profile short-circuited
with zero MCP servers of its own. The slot, the thread table and the
wait/in-flight/join readers are now keyed by hermes_home_key() (follows the
context-local HERMES_HOME override). Single-profile processes have one key and
behave exactly as before.
Shape from #90027 (Eva / @100yenadmin) and #104534 (@Bergmann89), applied to the
one slot #108352 left open.
Co-authored-by: Eva <239388517+100yenadmin@users.noreply.github.com>
Co-authored-by: Bergmann89 <info@bergmann89.de>
_spawn_hermes_action copied the dashboard's os.environ verbatim into every
detached `hermes ...` action. The dashboard runs inside the gateway and has
loaded its own profile's .env into the process environment, so
`hermes -p <other> gateway restart` (and every other dashboard-driven
profile action) started with the DEFAULT profile's platform credentials
and ports already present. load_hermes_dotenv does not override keys that
are already set, so the named profile's own .env could not displace them:
an A2A-only profile ended up claiming the default Discord bot token and
binding the default API server / BlueBubbles ports.
For actions carrying a profile selector (`-p X`, `--profile X`,
`--profile=X`), build the child env from the standard scrubbed subprocess
environment, drop _PROFILE_MANAGED_ENV_KEYS plus every key defined by the
dashboard/default profile's .env and its hydrated secret sources, and pin
HERMES_HOME to the target profile so the child's normal startup loads that
profile's .env. Only the leading selector is inspected: argv after the
subcommand may legitimately contain -p for a nested process. Actions
without a selector keep the historical environment byte-for-byte.
Test: the new case asserts the leaked keys are gone, benign keys survive,
and (by running the real dotenv loader in a fresh interpreter with the
captured env) that the target profile's values load without reviving any
default-profile value.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A profile served by the default multiplexer owns no gateway.pid / gateway_state.json,
so every surface that reads per-profile identity files called it stopped while the CLI
status surfaces (hermes -p X status / gateway status / cron status) said "running via the
default-profile multiplexer":
- `/api/status?profile=X` and `/api/messaging/platforms?profile=X` reported
gateway_running=false / state=None / "gateway_stopped" in the same body that listed X
under gateways[].served_profiles. The shared ladder `resolve_gateway_liveness` gains a
fourth rung for a named profile_dir: the live default multiplexer that records X in
served_profiles IS X's gateway (pid = multiplexer pid, runtime = its record, X's
`<X>:<platform>` entries re-keyed to the standalone shape).
- `POST /api/gateway/stop?profile=X` spawned `hermes -p X gateway stop`, which printed
"No gateway running for this profile" (exit 0) into the action log while the UI flipped
to stopped and the multiplexer kept serving X; `/api/gateway/restart?profile=X` spawned
a `-p X gateway restart` that only exits 78. start/stop now answer 409 with the
multiplexer explanation (one helper shared with the existing start refusal) and restart
targets the multiplexer, the process that actually serves X. A `--force`-started
separate gateway for X (own pid file) keeps normal per-profile management.
- CLI `hermes -p X gateway stop` refuses with exit 78 like run/start/install/restart when
X has no gateway of its own, instead of a contradictory exit-0 "not running".
Docs: multi-profile-gateways.md §1 and §5 describe stop + the dashboard behaviour.
`voice.voice_chat_mode: gpt-live` swaps the desktop's chained STT → turn → TTS
loop for OpenAI's gpt-live-1: one voice model that listens while it speaks and
has no tools of its own. Every real request it hears becomes a normal Hermes
turn on the open chat — any model/provider the session selected, full toolset,
memory, approvals — and the voice paraphrases the reply aloud.
Backend
- tools/voice_live.py: mode/credential/persona resolution and the one server-side
step the API needs — POST /v1/live/sessions exchanging the renderer's SDP offer,
pinned to client delegation; the OpenAI key never reaches the renderer. Voice
persona follows the vendor prompting guide (role, style, labelled delegation
policy describing Hermes as the backend). VOICE_LIVE_TURN_NOTE is the per-turn
model-input note (transcript in, speakable prose out).
- REST: GET /api/audio/voice-live/status (mode + readiness, non-secret),
POST /api/audio/voice-live/session (SDP exchange). The offer is passed
byte-exact: a stripped trailing CRLF is a vendor 400 "unmarshal SDP: EOF".
- prompt.submit accepts surface=voice-live (+ voice_context) beside hud; the
note rides the model input via the existing _prepend_note seam, the persisted
user row stays the user's words, the system prompt stays byte-stable.
- config_defaults: voice.voice_chat_mode (chained|gpt-live), voice.gpt_live.*.
Desktop
- lib/voice-live.ts: RTCPeerConnection + oai-events data channel owner, transcript
accumulation, session.commentary/thinking/instructions appends (500-token
chunking), mute, graceful close waiting for session.closed.
- hooks/use-voice-live-conversation.ts: same public shape as useVoiceConversation;
delegation → prompt.submit(surface=voice-live); tool activity → quiet thinking
appends; reply streamed back per sentence; spoken stop phrase ends the chat;
a newer delegation interrupts an in-flight turn.
- use-composer-voice mounts both engines and latches one at conversation start
from the backend-resolved status; gpt-live without a key falls back to chained
with a notice. Settings → Voice gets the mode dropdown, voice picker, persona.
Live-verified on the worktree desktop build (headless Electron, CDP, synthetic
mic): "what is 17 times 23 and which model are you on" → delegation → Hermes
(Claude Sonnet 4.5 via OpenRouter) → spoken "391 … Claude Sonnet 4.5 through
OpenRouter"; follow-up "double that" resolved from the spoken context → 782;
"run uname -r" ran the terminal tool with "Hermes is working: terminal" fed as
quiet context → spoken kernel version; "stop" closed the session
(reason=close_requested). Chained mode creates no RTCPeerConnection.
A hosted room member runs on a hidden room_plumbing session with no client
transport, so the tool.start/complete, approval.request, message.delta and
reasoning.delta frames its turn already emits bottom out at stdio and vanish.
Between turn.started and turn.settled in the durable room log a client sees a
black box, and community clients (Hermes Crew) cannot render tool cards,
approvals or live member status without inferring them from text.
One seam in write_json (plus the connector bypass in tool_progress) re-routes
those frames, stamped with the session's _hosted_room_task coordinates
(room_id, thread_id, member_id, turn_id, task_id, execution_generation), to a
new observer hook through the bounded per-consumer queues on_stream_* already
use, so plugin code never runs on the token path. Nothing is written to the
room log: deltas would exhaust a room's byte budget in minutes and checkpoint
replay must stay a pure function of the durable events. The task stamp gains
member_id (the driver already knows it; the proof did not carry it).
Group Chat keeps execution, scheduling and persistence; plugins own
presentation.
Tests: fold the ambient GIT_CONFIG_KEY_n negative and the isolation-still-in-force
assertions into the ordering test, and drop the two positive-only cases it subsumes.
What remains pins the injected sequence to git's own `config -z --get-all` output and
asserts the real trust decision (named repo usable, unrelated cross-owner repo refused).
Cost: noninteractive_git_env() runs on every internal git call, including the banner
startup probe, and the two `git config` children added ~10 ms per call against ~0.2 ms
before. Memoise per process on the inputs that select the config files plus the
system/global candidates' mtimes, so an edit to ~/.gitconfig is picked up without a
restart (verified live: 128 -> 0 within one process after appending safe.directory).
Review of #107748 found the carry path serialized the user's trust policy
incorrectly, and the mistake could WIDEN trust rather than merely reformat it.
safe.directory is an ordered multi-valued protected setting: an empty value
resets every entry seen so far. That is the documented mechanism for revoking a
system-wide `safe.directory=*` and then naming only the repositories you
actually trust. The previous helper broke all three properties that make it work:
* read `--global` before `--system` (git's precedence is system, then global)
* dropped empty entries via `if entry:`, deleting the reset marker
* de-duplicated values, though it is a sequence and not a set
Reproduced with real git (2.55.0 here, 2.47.3 by the reviewer). Given
system `safe.directory=*`, global `safe.directory=` then `/trusted/only`:
real git, user's own policy -> rc=128 (dubious ownership)
previous head's sequence `/trusted/only`, `*` -> rc=0 (trust widened)
this head's sequence `*`, ``, `/trusted/only` -> rc=128 (matches real git)
So a user who had deliberately revoked a machine-wide wildcard silently got it
back inside Hermes's internal git calls. That contradicted the "read-only and
non-widening" claim the original change rested on.
Fix: read scopes lowest-precedence first (system, then global) and replay every
value verbatim -- no de-duplication, no dropping of empty reset markers. Read
with `git config -z` so a value containing whitespace or a newline stays the one
entry git reads it as, instead of being split into several bogus trust entries
by splitlines()/strip(). The trailing field of a -z stream is always empty and
is discarded; interior empty fields are real resets and survive.
Two invariant tests, both proven red on the previous implementation:
* the injected sequence equals `git config -z --get-all safe.directory` for
the same two config files, asserting git's documented reset shape
* an E2E trust decision under GIT_TEST_ASSUME_DIFFERENT_OWNER=1: the named
repo stays usable and an unrelated cross-owner repo is still refused,
proving the reset still revokes the wildcard
13 passed in tests/hermes_cli/test_noninteractive_git.py, 69 in
tests/security/test_gitspawn_config_injection.py and
tests/tools/test_checkpoint_manager.py.
`noninteractive_git_env()` blanks GIT_CONFIG_GLOBAL/SYSTEM to /dev/null so a
user's config cannot hang Hermes's internal git plumbing with pagers, hooks or
credential prompts. Sound intent, but it also discards `safe.directory` — and
git honours that key ONLY from global/system config (it is rejected from
repo-level config by design, so a hostile repo cannot self-authorise).
Result: every internal git call fails on a repo whose st_uid != geteuid():
$ hermes -w
✗ Failed to create worktree: fatal: detected dubious ownership in
repository at '/mnt/nas/py/repo'
That hits NFS/CIFS mounts without idmapping, shared checkouts, and containers
with a remapped uid. The user's own `git config --global --add safe.directory`
is correctly set and their interactive git works — Hermes throws the setting
away before git reads it, so the error's own suggested remedy can never fix it.
There is no config or env escape hatch: the blanking is unconditional.
Reproducer (any repo where the checkout uid differs from the caller's):
git rev-parse HEAD # works
GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null \
GIT_CONFIG_NOSYSTEM=1 git rev-parse HEAD # dubious ownership
Fix: read the user's real `safe.directory` values before the isolation is
applied, then re-inject them over the GIT_CONFIG_KEY_n channel, which survives
GIT_CONFIG_GLOBAL=/dev/null. Isolation is unchanged — global/system config stay
pointed at /dev/null and every hardening override still applies, since a later
key of the same name wins in git's config order.
Read-only and non-widening: only values already present in the user's own
config are carried, so this grants no trust they had not granted. Ambient
GIT_CONFIG_KEY_n injection is still stripped first, so a caller cannot launder
an attacker-controlled path in this way — covered by a regression test.
Tests: three cases in TestNoninteractiveGitEnv — entries carried past
isolation, no entries injected when the user configured none, and ambient
injection not trusted. All pin GIT_CONFIG_SYSTEM at an empty file, since a real
/etc/gitconfig on the test host can otherwise leak entries and mask the
assertions.
Verified on Arch Linux, git 2.x, repo on an NFSv4 mount (uid 1024 vs caller
1000): `git worktree add` under the patched env returns rc=0 where it
previously failed. 80 passed in tests/hermes_cli/test_noninteractive_git.py,
tests/security/test_gitspawn_config_injection.py and
tests/tools/test_checkpoint_manager.py, including the pre-existing assertions
that the isolation stays in force.
Internal moves get no re-export aliases: models_validate reads
hermes_constants.openrouter_variant_base directly and the private
_OPENROUTER_VARIANT_SUFFIXES/_openrouter_variant_base shims are dropped.
`:nitro`, `:floor`, `:exacto`, and `:online` are request-time routing
modifiers, not catalog models — OpenRouter's /models lists only the base
id, and a variant runs the same model with the same context window.
`get_model_context_length()` keyed every lookup on the full suffixed id,
so each one missed and the resolver fell through to a generic family
default or the 256K fallback:
openai/gpt-5.5:nitro -> 256K (real 1.05M)
x-ai/grok-4.6:nitro -> 131K (generic "grok" catch-all)
anthropic/claude-opus-4.6:nitro-> 200K (generic "claude" catch-all)
The window silently shrank, triggering early compression and a wrong
/usage readout. f14059fa fixed the sibling half of this bug class in
/model validation; this fixes the metadata half.
Strip a recognized variant suffix for LOOKUP only, keeping the suffixed
id on the wire so the routing opt-in survives. Applied after the explicit
config overrides (steps 0b/0c) so a user-pinned value still wins, and
before every cache/catalog lookup. Gated on the request actually routing
through OpenRouter, so a local Ollama `model:tag` is untouched.
`:free`/`:batch`/`:thinking` are deliberately excluded — those ARE
distinct catalog SKUs with their own windows, so stripping them would
report the wrong number.
The suffix set and base-id split move to hermes_constants (import-safe,
dependency-free) so the metadata layer shares one definition with
hermes_cli.models instead of duplicating it.
turn_in_flight read only the dashboard session table; an in-process cron run
never registers there, so the watchdog reported "no running turn" and exited
mid-job (tool calls then failed with "cannot schedule new futures after
interpreter shutdown", the execution was marked unknown, the slot lost). The
probe now also consults cron.scheduler.get_running_job_ids — the ledger the
gateway shutdown drain already uses.
Addresses #107485
The Desktop/serve backend ticked every installed profile (ignoring
gateway.multiplex_profile_allowlist) and gated only on the profile's OWN
gateway.pid. A satellite served by the default multiplexer has no pid file, so
both tickers raced its fires and the Desktop one won nondeterministically —
adapter-less standalone delivery, and the environment behind #107485.
Homes now come from profiles_to_serve with the default profile's allowlist
(the multiplexer's served set); the per-tick gate also consults
named_profile_served_by_running_multiplexer.
Addresses #107485, #94590
Co-authored-by: fangliquan <fangliquan@qq.com>
Under gateway.multiplex_profiles the default gateway serves every profile, yet
four startup/status paths still reasoned from the wrong source:
* A secondary profile's API_SERVER_KEY (which the docs REQUIRE for /p/<profile>/
auth) auto-enabled api_server in that profile's config, so
_load_secondary_profile_config raised SecondaryPortBindingConfigError and the
whole profile was skipped. gateway/config_env.py::_enable_from_env now leaves
`enabled` alone for port-binding platforms while a multiplexer loads a
NON-default profile (home override + multiplex flag, the same signal
gateway.config uses for scoped reads); the credential still lands in extra so
the shared listener can authenticate the prefix. Default profile unchanged.
* "Is this profile served?" was re-derived from the default config.yaml plus
GATEWAY_MULTIPLEX_PROFILES as seen by the CLI process. `hermes -p coder ...`
loads coder's .env, so an env-only opt-in on the default profile was invisible
(guard never fired, status said stopped) and an allowlist edit flipped the
answer before the restart. named_profile_served_by_running_multiplexer now
reads the pid-verified default gateway_state.json served_profiles (written by
_record_served_profiles) first and falls back to config derivation only when
the key is absent. The record helpers live in hermes_cli/gateway_multiplex_served.py.
* The served-profile guard ran only inside `gateway run`. `hermes -p X gateway
start|install|restart` reached the service manager, whose unit then exited 78
forever (systemd parks it while the CLI prints "started"; launchd KeepAlive
respawns every 30 s). The service verbs now run the same guard up front
(exit 78, same message) and accept --force; the Desktop /api/gateway/start
route returns 409 for a served profile instead of spawning a doomed child.
* Status surfaces disagreed: `hermes -p coder status` said stopped, `hermes -p
coder cron status` said "cron jobs will NOT fire" while `cron list` said fine,
and the default `hermes status` never listed served profiles. Both now route
through the probe / the recorded served set. The -p/--profile matcher in
_scan_gateway_pids and gateway.status._command_line_belongs_to_profile compares
the flag token for equality (`-p ops` no longer claims -- or lets `gateway
stop` SIGTERM -- an `-p ops-2` gateway).
Docs: multi-profile-gateways.md now describes the start/install refusal, the
--force flags, the API_SERVER_KEY behaviour and the single default-home
gateway_state.json (the per-profile runtime_status.json claim was wrong).
Fixes#100397
Addresses #89726#97360#71344
(cherry picked from commit d002c1864a7b6a22c53758b16b7b0cc79aea2edf)
Under gateway.multiplex_profiles, os.environ holds the DEFAULT profile's .env; a
secondary profile's values exist only in the per-turn secret scope. Every reader
below still read os.environ/os.getenv at call time, so a secondary profile's turn
silently used the default profile's value.
Credentials (F6): FIRECRAWL_API_KEY (read_file hosted OCR), OPENVIKING_API_KEY,
mem0-OSS OPENAI_API_KEY, MODAL_TOKEN_ID/SECRET and BROWSER_USE_API_KEY presence
gates, and the xAI video plugin's os.getenv("XAI_API_KEY") fallback AFTER the
scoped resolver had already missed — the exact fallback-after-miss shape
gateway/AGENTS.md forbids. Deleted, not re-scoped: the resolver is the scope.
Identity / tenant (F7): MEM0_USER_ID/AGENT_ID/HOST/MODE, SUPERMEMORY_CONTAINER_TAG,
RETAINDB_PROJECT, OPENVIKING_ACCOUNT/USER/AGENT (and the whole layered() env
read), HINDSIGHT_BANK_ID/MODE/retain shaping, HERMES_HONCHO_HOST. A raw read
put a secondary profile's memories into the default profile's account/bank/
project/tenant and recalled them back into the default's turns. Each now uses
get_secret with the provider's own per-profile default on a miss.
Endpoints (F8): OPENAI_BASE_URL (aux custom runtime + direct-alias expansion),
XAI_BASE_URL/HERMES_XAI_BASE_URL (aux OAuth), NOUS_INFERENCE_BASE_URL (#65941,
both the aux builder and hermes_cli.auth_nous._nous_inference_env_override),
GATEWAY_PROXY_URL (same UnscopedSecretError-only fallback shape as
GATEWAY_PROXY_KEY three lines below), FIRECRAWL_API_URL, BROWSERBASE_BASE_URL,
SUPERMEMORY/RETAINDB/HONCHO/HINDSIGHT URLs. The keys beside them were already
scoped, so a secondary's key was sent to the default profile's proxy or host.
Targets / display (F11): WEIXIN_HOME_CHANNEL (message posted into the default's
chat), HERMES_LANGUAGE, and agent/i18n's process-wide lru_cache of
display.language — now keyed by HERMES_HOME.
Outbound webhooks: hooks.outbound[].secret_env resolved from os.environ while
the gateway registers each profile's targets inside that profile's scope, so a
secondary's deliveries were signed with the default's secret or left unsigned.
Agent-cache eviction: _spawn_release_thread started a bare threading.Thread, so
commit_memory_session -> provider on_session_end ran with an EMPTY context. The
thread now runs copy_context() and, for the unscoped housekeeping sweep, enters
the owning profile's _profile_runtime_scope resolved from the session key
(agent:<profile>:...). The pressure batch does the same per key.
session_search (#82903): agent/inline_tool_executors.py::_session_search
forwarded every schema argument except `profile`, so a gateway agent could
never select a named profile's store. Forwarded; the ownership-scoping design
in #87779/#87847 is a separate design call and is not attempted here.
Live repro (/tmp/mux_audit/fix-tool-memory-reads/repro.py): 28 FAIL on
origin/main -> 0 FAIL with this change; 10 new invariant tests red on base.
Fixes#82903Fixes#65941Fixes#99121
Addresses #87779
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Michael Versluis (Berry) <michael@wve.nl>
hermes_cli/goals.py cached a bare SessionDB() per HERMES_HOME. The path was
right per profile, but it was a SECOND writer connection beside the
gateway's registry handle for the same file — its own token-writer thread
and close-time checkpoint the registry knows nothing about (the #90837
corruption shape), doubled per profile under multiplexing. This is the
same topology #100682 removed from the API server. Acquire through
hermes_state_registry and drop lost-race references with release_or_close.
Reverts the in-tree org skill-marketplace: hermes_wisdom package, three
model tools, CLI/gateway/desktop/dashboard/Telegram/Slack surfaces.
Later non-Wisdom work on shared files (guest onboarding i18n, dashboard
startup schema, Slack adapter, tui_gateway) is kept; Wisdom-only call
sites and config were stripped from those files.
Stop recommending a system-RAM spill when no curated model fits resident.
Preserve explicit model selection and the existing resident quality/speed
ranking, including the separate unified-memory policy.
Require a recommendation for automatic quickstart, expose Browse when
none exists, and rename Configure to Let me choose. Keep policy copy and
reason keys consistent across the four translated local-model sections.
Cover automatic refusal and explicit spilled setup against one budget,
plus the Browse, Download and Use interactions in the desktop pane.
The dashboard polls /api/analytics/usage and /api/analytics/models every
few seconds. When state.db is malformed the read raised straight through
the handler, so uvicorn logged a full traceback at ERROR on every poll —
one fleet host wrote ~520K identical journal entries in 24 h.
Wrap both analytics handlers in `corrupt_store_as_status`: a corrupt-image
sqlite3.DatabaseError (is_malformed_db_error) becomes a 503 with an
explicit `state_db_corrupt` payload pointing at `hermes doctor`, and the
warning is gated per store path via `{path: monotonic}` (>=300 s), then
debug. Busy/locked and every other error propagate unchanged, and the file
is never renamed or quarantined from the dashboard — repair stays with
`hermes doctor` / `hermes sessions repair`. `_session_db_path_for_profile`
is split out of `_open_session_db_for_profile` so the router can name the
store without opening it.
Refs #96591
Reported-by: #96591
`hermes sessions repair --check-only` printed the corruption reason and
exited 0, so scripts and the console wrapper gating on the status read a
broken state.db as healthy. Return 1 from the CLI handler (main.py already
sys.exits a truthy return) and from the console handler, where
`_capture_output` turns the status into a ConsoleCommandError carrying the
printed reason.
Salvaged from PR #103321 (the check-only reporting part only; the probe
rewrite and connection-tracking changes were not taken). Refs #63386.
`hermes doctor` reported every write-health-probe failure as "state.db FTS write
corruption" and `--fix` ran the FTS repair ladder — rebuild, REINDEX, sqlite_master
surgery + VACUUM — on the damaged file. When the damage is structural (canonical
tables/indexes), none of those rungs can fix it, each one writes to the torn file in
place, and the operator is then told to "restore from the backup copy beside
state.db": a `.malformed-backup` that is a snapshot of the same corrupt image.
`hermes sessions recover`, the tool that actually rebuilds canonical rows into a
fresh file, was never mentioned (#88587; the 1.7 GB field incident lost days to it).
Discriminate before mutating. hermes_state_repair.integrity_damage_is_structural
maps `PRAGMA integrity_check` output onto the file: a `Tree N` id resolved through
sqlite_master.rootpage, an index named in `row N missing from index X`, or a
`Freelist:` line is structural unless the object is a Hermes-owned messages_fts*
table/shadow (full-matched, so a user lookalike such as archive_fts_data is never
swept into the rebuildable set). state_db_has_structural_damage runs it read-only on
a fresh connection; an integrity_check that RAISES under the walk (torn root page)
is structural too — no FTS-only fixture does that while sessions/messages read
cleanly. doctor's state check consults it first: structural damage becomes a
manual issue naming `hermes [-p <profile>] sessions recover --source <this db>
--inspect-only` (profile pinned, #105887) and explicitly warning off the
.malformed-backup; nothing is mutated and no backup is written. FTS-only damage
keeps the existing in-place repair path.
Verified against real fixtures: a torn `sessions` root page (before: "FTS write
corruption", --fix wrote a 1:1 malformed-backup and failed; after: structural,
recover guidance, no writes) and the 16-byte DEADBEEF messages_fts_data stomp
(still repaired in place via rebuild_fts).
Salvaged from PR #88604 (liuhao1024) onto the split doctor_state.py; the
classifier lives beside the repair ladder in hermes_state_repair so the ladder
itself can consult it next.
Fixes#88587
A SIGKILL mid-write (OOM killer, #106667) can leave state.db with a garbage
page-1 header. SQLite refuses the file outright ("file is not a database",
SQLITE_NOTADB) and the sqlite3 shell's .recover opens the file like any other
client, so the lost_and_found lane failed with the same rc=26 twice although
every data page after the header survived. The #106587 quarantine now
preserves such a file as state.db.notadb-<ts>-<pid>.bak; this makes that
preserved file recoverable with `hermes sessions recover --source <bak>
--allow-partial`.
When both .recover attempts fail with "not a database", zero the 100-byte
header of the lane's private snapshot copy and rerun them. .recover trips
only on the magic check and infers page size and layout from the pages
themselves, so a zeroed header is enough; a spliced donor header (the PR's
original mechanism) instead advertises a database size / freelist that
contradicts the file and yields "database disk image is malformed" on a
direct open — verified live on a 139-page fixture, which also showed the
zeroed header recovers 60/60 sessions and 300/300 messages whether the
damage covers 100 bytes or the whole first page. The user's file is never
written; the report carries `sqlite3_cli.header_zeroed` and a warning about
the WAL boundary.
Live repro (sqlite3 shell 3.53.1 on PATH, header overwritten with random
bytes): BEFORE "page-level .recover salvage failed: ... file is not a
database (26)"; AFTER "Recovered 60 sessions and 300 messages", source md5
unchanged.
Salvaged from PR #102808 (intent; trimmed from 302 to ~40 source LOC by
dropping the donor-header/page-size sweep and the redundant preopen probe —
the shell's own refusal is the detector). Independent review on the PR by
@strzhao.
Refs #106667
Refs #106587
Reported-by: TaoMasterCoder
Cross-referenced-by: kshitijk4poor
When the leftmost (or rightmost) leaf of a table b-tree is damaged, the edge
probe `SELECT rowid ... ORDER BY rowid ASC LIMIT 1` walks the table tree and
raises, and _salvage_rowid_bounds fell back to INT64_MIN. The gallop from the
surviving edge cannot cap that side either (every probe crosses the damaged
leaf), so bisection burned the entire 10,000-query budget moving the bound
inward by a few thousand rowids out of 9.2e18 and the table was lost — a
4-row gateway_routing table in #98050, sessions + session_model_usage in
#100313.
`SELECT min(rowid), max(rowid)` is answered by the planner from any covering
index (every Hermes table has at least the PRIMARY KEY autoindex) without
touching the damaged leaf, which is exactly what the reporter verified by
hand. Ask it for the missing edge(s) first; only when it fails too does the
domain fallback + gallop run as before. Reported under `aggregate_edges` so
recovery.json still shows how the bound was obtained.
Live repro (real fixture: leftmost `sessions` leaf cell count overwritten,
400 rows): BEFORE bounds low=-9223372036854775808 copied=0 range_queries=10000
query_limit_reached=True status=failed; AFTER low=1 high=400 copied=391
range_queries=40 status=partial (only the damaged leaf's rows are lost).
Refs #98050
Refs #100313
Reported-by: Ace-Kelly
Corroborated-by: Proff506
`hermes sessions recover --allow-partial` walks damaged tables by rowid range
and falls back to an exact-rowid lookup for the last cell of a broken range.
A phantom row produced by page damage (a `sessions` row with a NULL
`started_at`, an `async_delegations` row with a NULL `state`) reads fine from
the source but violates the destination's NOT NULL constraint, and the
resulting sqlite3.IntegrityError escaped the exact-lookup boundary and aborted
the whole recovery — losing every healthy row behind it.
Catch IntegrityError at that boundary, count it under
`destination_rejected_rows` and record the rowid as a skipped singleton
("destination constraint rejected row: ...") so the run completes, verifies,
and the report shows exactly which rows were dropped.
Live repro (real fixture: source schema's NOT NULL relaxed via
writable_schema, phantom `sessions` row inserted): BEFORE
sqlite3.IntegrityError "NOT NULL constraint failed: sessions.started_at" at
session_recovery.py recover_exact_rowid; AFTER status=partial copied=3
destination_rejected_rows=1, verified=True. Reporter @i8ei confirmed the same
patch recovers the field database (22 sessions / 2,248 messages) in #102240.
Salvaged from PR #91413 (rebased onto the _RowidRangeSalvage refactor; test
reduced to one invariant).
Refs #102240
Reported-by: i8ei
SQLite dynamic typing lets a TEXT cell ('not-a-timestamp'), inf/nan or a
garbage double (8.4e252 salvaged from a damaged page) sit in a REAL
timestamp column. Every reader called datetime.fromtimestamp()/float
arithmetic on the raw cell, so ONE bad row raised TypeError/OverflowError
out of the row loop and took down the whole `hermes sessions list`/browse
table (#102399), all three exporters — JSONL/MD, QMD, HTML (#102352) —
and `hermes insights` (#99959).
Fix the class with ONE helper, hermes_cli.timefmt.coerce_epoch(): a
stored cell becomes float epoch seconds inside a sane 1970..2103 window
or None after a WARNING that names the session id. Every reader routes
through it — relative_time (list/browse/resume picker), format_epoch
(prune/candidates tables), the three exporters' timestamp formatters,
insights' _get_sessions/_day/period range — so a bad row renders as
'?'/'N/A'/raw text for that one cell and the command completes.
Write side: hermes_state_messages._coerce_timestamp (append_message,
append_messages_batch, import) and the import path's started_at now use
the same window, so a new out-of-range timestamp falls back to now()
instead of being persisted — new bad rows cannot be written by Hermes.
Reported-by: #102399, #102352, #99959 reporters; kokhlo's insights
analysis pointed at every reporting site, not just line 860.
delete_profile already force-closes holographic memory_store.db in this
process, but the shared SessionDB registry kept state.db open. Recreate
then failed with a replaced/locked database. Close every shared handle
under the doomed directory, same contract as MemoryStore.release_all_under.
Co-authored-by: Cursor <cursoragent@cursor.com>
The dashboard's `_eager_reconcile_own_session_db` did an unconditional
writable `acquire()` at every startup. When the gateway shares that
state.db the dashboard became a second long-lived writable SessionDB
owner: a close-time WAL checkpoint plus a possible FTS rebuild in
`_init_fts`, the two-writer vector behind the corruption reports in
#107688 and #100896 ("5 live SessionDB handles" precursor, gateway +
dashboard both holding the WAL).
Route the startup reconcile through `_open_session_db_at_path(...,
read_only=True)`, which already bootstraps a missing store and heals a
stale/malformed schema through exactly ONE writable open before
reopening read-only. A healthy store now gets zero writable opens from
the dashboard while the #79531/#80037 "bring schema current before the
first poll" contract is kept (existing heal test unchanged).
Live repro (healthy store, count writable SessionDB.__init__ calls from
the startup worker): before=1 after=0.
Reported-by: #107688, #100896 (@kokhlo diagnosis)
Refs #107688#100896
doctor --fix ran a raw sqlite3.connect + wal_checkpoint(PASSIVE) with no
holder guard. Under a running gateway that second connection joins the
live WAL and its close-time handling is the second-writer corruption
class #103339 tracks. Route the checkpoint through live_writer_holds_db
and skip with an actionable finding when the database is held.
[Salvage note: the original PR also flipped the repair probe's
DatabaseError branch to fail closed; that hunk is dropped here because
it refuses every header-destroyed DB (the exact class repair exists
for) — the probe's own connect raises before the EXCLUSIVE/BEGIN
statements ever run.]
Teknium