`hermes gateway migrate --multiplex` ran its one fallible step LAST (install +
start the default gateway) with nothing around it. On a fleet whose secondary
ran a system unit as root (#110850) that step raised, leaving the flag on, the
secondary's unit removed and no gateway anywhere, and the re-run hit the
"already multiplexing (flag on)" short-circuit over an empty fleet.
- apply_migration(): the default bring-up runs inside a rollback. On failure the
manifest written before the first destructive step restores the flag and
reinstalls every recorded per-profile gateway with its recorded User=.
- MigrationPlan.interrupted: flag on + manifest present + no live default
gateway is a half-applied migration, not "already multiplexed"; the re-run
resumes from the manifest (target manager and User= read from it, since the
units themselves are gone) instead of refusing. Flag off + leftover manifest
refuses to overwrite it and points at --standalone.
- ProfileGateway.services records EVERY installed unit (user and system) and the
manifest carries them; apply stops/uninstalls all of them and rollback
reinstalls all of them, so a second owner is never left live beside the
multiplexer. The unattended hook treats a two-unit profile as an ambiguous
topology and refuses (review finding on #110205).
- gateway_identity(): an unresolvable User= on a system unit stays None instead
of borrowing the profile directory's owner; the unattended hook treats the
unknown principal as a boundary (review finding on #110205).
- auto_migration_opted_out(): reads the effective config (load_config_readonly
under the default home), so a managed `false` wins over a user `true` and a
YAML string "false" is an opt-out, not a truthy value (review finding on
#110205).
Builds on KoNit-K's #110854 (run_as_user threaded through install, preserved
from the removed system unit).
The salvaged flag was spelled --profile, which collides with the global
-p/--profile that hermes_cli.main scans BEFORE argparse: `hermes webhook
subscribe x --profile compta` would switch this CLI process to compta's
HERMES_HOME and write the subscription into compta's webhook_subscriptions.json
— a file the default gateway's webhook adapter never reads — while the route
still lacked the profile key. #109020 special-cased the scanner for the webhook
subcommand; naming the flag --route-profile removes the ambiguity without
touching _scan_profile_flag: -p picks the gateway whose subscriptions file is
written, --route-profile picks which /p/<profile>/ prefix may hit the route.
Docs: cli-commands reference row, multi-profile-gateways webhook section, the
route `profile` field. Builds on #109020 (fangliquanflq). Fixes#109016.
Extends the multi-profile MCP paragraph with the identity rules landed in
this branch: OAuth tokens live per profile so each profile's calls run as
its own account; client_cert/client_key are part of the connection
identity; trust and supports_parallel_tool_calls are the consuming
profile's policy even when it shares another profile's connection.
Wording of the per-profile OAuth account guarantee follows the docs draft
in PR #109574 (its token-file fingerprint code path was not taken).
Co-authored-by: ly6751 <99090550+ly6751@users.noreply.github.com>
`main` is a valid profile name (only hermes/default/test/tmp/root/sudo are
reserved), but _session_key_namespace mapped it to `agent:main` — the default
profile's namespace. Both profiles then built byte-identical keys: one routing
entry, one cached agent, and, since 75ae2859b9 pinned default-namespace
keys to the launch store, profiles/main's scoped sessions were written into
the ROOT state.db instead of profiles/main/state.db.
Key the `main` profile as `agent:main~` (`~` is outside the profile-id
alphabet, so the marked form cannot be any other profile's id) and give the
namespace slot one inverse, profile_from_session_key_namespace, used by the
store's key parser, _parse_session_key, the update-marker profile reader and
the profile-delete eviction prefix. Default keys stay byte-identical.
Reshape the two salvaged commits onto current main (#109954):
- Move the boundary guard out of the gateway_migrate facade into a new sibling
hermes_cli/gateway_migrate_guards.py as a table of guard functions
(_AUTO_MIGRATION_GUARDS: service domain, UNIX user, HERMES_HOME tree) plus the
identity resolver. The facade grows by ~20 lines only (uid/runtime_home on
ProfileGateway, one seam, the hook wiring).
- Compare uids, not strings: live pid owner via /proc (ps fallback only on
macOS, where /proc does not exist), else the system unit's User= via
_read_systemd_user_from_unit (root when absent), else the home directory's
owner. None means unknown and never blocks.
- The home-tree guard reads the HERMES_HOME the installed unit pins, not the
directory the plan enumerated: that is where the gateway really runs and is
exactly the "stale copies under profiles/" shape from the report.
- When the default is detached, a service-managed secondary is a different
domain for the AUTO path (it must not elect the secondary's manager); the
explicit command keeps electing it as before.
- The explicit command surfaces the same findings as notices (dry run shows
them) and is never blocked by them; only the update hook refuses.
- Rename the opt-out key to gateway.auto_multiplex_migration (nested only, no
top-level alias) and read it before a plan is built, so false prints nothing
and touches nothing. The explicit command ignores it.
- Tests trimmed to the invariants: one parametrized boundary test that exercises
the real hook end to end (refuses, touches nothing, dry run shows the notice),
one "same user / same scope still migrates" control, one opt-out test.
- Docs: boundary table + renamed opt-out section in multi-profile-gateways.md;
one line in hermes_cli/AGENTS.md.
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Co-authored-by: Athena <athena@olympus.local>
`hermes update` folds an eligible multi-profile install onto one multiplexed
gateway on its own, and there is currently no way to say no. The only lever,
`gateway.multiplex_profiles: false`, is also the default: `_read_multiplex_flag`
returns `False` for "absent" and for an explicit `false` alike, so an operator
who has already decided to stay on per-profile gateways has no way to record
that decision. The migration runs again on the next update.
Add `gateway.auto_migrate` (bool, default `true`). Read from the default
profile's config, it gates the automatic path only:
- absent or `true` -> today's behaviour exactly, no change
- `false` -> `maybe_auto_migrate_after_update()` returns before
building a plan; no output, no changes
`hermes gateway migrate --multiplex` is an explicit request and still migrates
regardless of the flag, so it stays the supported way to opt back in.
One early return, one schema entry with the reasoning inline, one invariant
test (opt-out blocks the hook, absent/true do not, explicit command still
applies), one section in the multi-profile gateways guide.
The served-set paragraph documented create and delete as live operations; rename now
follows the same unroute-before-mutate protocol, so say so where operators look for it.
The multiplex guide now says client_cert/client_key are part of the
"same credentials" test and states the OAuth rule as its own sentence;
the MCP config reference names the per-profile token directory and the
never-shared-across-profiles rule next to the OAuth behaviour list.
Co-authored-by: ly6751 <99090550+ly6751@users.noreply.github.com>
`hermes config set gateway.multiplex_profiles true` warned "not a recognized
config key" although gateway/config.py reads it: the key (and profile_routes)
were never in DEFAULT_CONFIG["gateway"]. Both are registered with their doc
comment; the CLI loaders deep-merge new keys, so no _config_version bump.
`hermes gateway migrate --multiplex` with two or more profiles but no
secondary running its own gateway printed "nothing to migrate" and left the
flag OFF. The explicit command now applies the one remaining step — flag on,
default gateway (re)started, the same rollback manifest (empty secondaries)
for --standalone. `hermes update`'s automatic hook keeps treating that case
as a no-op: it never flips modes on an install where nothing was running.
Under gateway.multiplex_profiles a secondary's api_server and webhook are never built as
adapters (run_adapters skips SHARED_LISTENER_MIRROR_PLATFORMS: the default's listener answers
/p/<profile>/...). The multiplexer record therefore has no `<profile>:api_server` entry,
profile_platforms_from_multiplexer() returned {} for them and both /api/messaging/platforms
and /api/status?profile= fell through to `pending_restart`: the Desktop Messaging card and
Command Center said "Restart needed" forever for a platform that was answering.
- gateway.status.shared_listener_mirror_platforms projects the default's LIVE api_server /
webhook entry onto every served secondary with `ingress_url` = `<listener>/p/<profile>/v1`
(`.../webhooks/<route>`); a dead default listener is not mirrored. The api_server / webhook
adapters stamp the listener they actually bound (`listener_base`) on connect so the URL is
the real one, not a config guess. `hermes status` lists those URLs beside the other
shared-ingress platforms.
- /api/status?profile= reports `gateway_shared_with` (every profile the multiplexer carries)
when the served rung answered; null for a standalone gateway.
- Desktop: the messaging card shows the URL line; "Restart gateway" from a served profile
(statusbar menu, Cmd+K, messaging/webhooks banners, Command Center) confirms "Restart the
shared gateway? All bots on this device reconnect: default, alpha, beta" (Restart all /
Cancel) and toasts "Shared gateway restarted (3 bots)". Standalone keeps the silent path.
- Dashboard: same confirm + toast on the System page and the sidebar restart; the 409 from
start/stop on a served profile renders as an inline notice instead of a raw error toast.
A `gateway.multiplex_profiles` gateway enumerated `profiles/` once at boot, so a profile
created afterwards (CLI, dashboard, Desktop, TUI) was never served until `hermes gateway
restart`; Desktop and the dashboard gave no reminder, so a new profile's bot simply never
connected.
The served set is now reconciled at runtime (`gateway/run_profile_reconcile.py`):
- `hermes_cli/profiles.py` create/delete ping the multiplexer over its control socket
(new `rescan-profiles` verb); a supervised watcher rescans every 30s as the safety net.
- A new profile gets its adapters under its own runtime scope from its config/.env
(`_start_one_profile_adapters`, same duplicate-credential guard as boot, now seeded
with the LIVE secondaries' claims), `served_profiles` in gateway_state.json is
updated, MCP discovery + log routing run for it. Other profiles' adapters are never
touched.
- A served profile whose config.yaml/.env changed is re-scanned so a token added after
create builds the adapter; already-live/queued platforms are skipped (no second poller).
- A deleted profile (tombstone) has its reconnects cancelled, adapters torn down,
pairing/busy bookkeeping and cached agents dropped, and this process's SQLite /
memory-store handles released so the deleter's rmtree succeeds.
- The in-process cron ticker takes a live enumerator so new profiles' jobs fire.
- PUT /api/messaging/platforms/<id>?profile=X returns `hot_served` when a live
multiplexer rebuilt X's adapters; Desktop/dashboard skip the restart banner then.
- `hermes profile create` confirms hot-serve; the restart reminder stays for a gateway
that did not pick the profile up (older build / signal failed).
Replace the 'a secondary must not enable a port-binding platform' rule with the
shared-listener contract and a per-platform URL table (Twilio, LINE, Teams,
BlueBubbles, Microsoft Graph, WhatsApp Cloud, WeCom callback, Feishu webhook), plus
the status/dashboard surfaces that print the URL.
What `hermes update` does, blockers and fixes, URL change for
inbound-port profiles, the post-create restart reminder, rollback, and
the `gateway migrate` reference row.
Under gateway.multiplex_profiles a secondary profile X is ticked, dispatched
and notified from the default profile's process, where os.environ holds the
DEFAULT profile's .env and X's values live only in the per-turn secret scope /
HERMES_HOME override. Every remaining read that skipped that scope made X
behave differently from `hermes -p X gateway run`:
- cron: HERMES_CRON_TIMEOUT, HERMES_MODEL (job/preflight fallback),
HERMES_CRON_MAX_PARALLEL, inflight allowance, prefill file and the script
timeout were bare os.getenv → the default profile's values; a job without a
model silently ran on the default's HERMES_MODEL instead of refusing.
cron/env_settings.py::cron_env_setting reads the scope (fire) or the ticked
home's .env (tick thread), plain environ when multiplexing is off.
- child env: the restart-safe cron worker, the Bot Chat delivery child and the
kanban worker inherited the launch profile's non-credential .env settings
and bridged TERMINAL_* policy (TERMINAL_ENV=docker, default's image,
HERMES_MODEL) — X's worker ran in the default's docker image on the
default's model. tools/environments/local.py::strip_launch_profile_env drops
them when the child targets another served profile.
- kanban: the worker --toolsets pin was silently dropped for every served
assignee (toolset probes call get_secret without a scope → swallowed
UnscopedSecretError); notifier pings, artifact uploads and the wake text ran
under the default's media policy / display language (only wake() was scoped).
- /loop: _post_turn_loop_completion hopped to the executor without contextvars,
writing the completed tick into the DEFAULT profile's state.db and leaving
X's row awaiting_response forever; the --until judge ran with the default's
aux credentials.
- background processes: a secondary's processes.json (scope-relative since
adf23550f5) was never read at startup; its processes were not re-adopted
and notify_on_complete notices were lost. Startup recovers every served
home under its scope; recovery adopts each session once.
- completion delivery: background_process_notifications was evaluated once per
drain for the ambient profile (default's mode for everyone; X's `off`
dropped a sibling's `all` event), recovered watchers used the default's
mode, HERMES_BACKGROUND_NOTIFICATIONS was read raw from environ;
_deliver_platform_notice used the default's GatewayConfig so a secondary's
notice_delivery: private went public.
Not changed: gateway/run.py and tools/async_delegation.py (PR #106742
rewrites both). Known residue left for the env-bridge lane:
HERMES_SESSION_STALL_TIMEOUT is bridged once from the launch config.
scheduler bug, not a parity gap; unchanged here.
The multiplexing default gateway now serves default + every live named profile
under profiles/. profiles_to_serve(multiplex=True) is a pure directory read
(tombstoned profiles skipped, never mkdir); every reader — gateway served set,
/p/<profile>/ prefixes for api_server + webhook, the named-profile standalone
guard, the Desktop cron ticker (its #108428 standdown for a profile owned by a
running gateway is unchanged) — drops the allowlist parameter.
Config v43 migration deletes the key from user config.yaml; DEFAULT_CONFIG,
GatewayConfig and the top-level yaml bridge no longer carry it.
BREAKING: anyone who set an allowlist now has their excluded profiles served.
Archive or delete a profile you do not want served (Teknium approved).
Under gateway.multiplex_profiles a secondary profile's turn ran with the LAUNCH
profile's working directory, command allowlist, redact_secrets switch, credential
file mounts, browser engine/headed flags, LSP service, auxiliary-provider health
marks and MCP stderr log, and several TERMINAL_ENV consumers read the process env
instead of the routed profile's terminal scope. A standalone `hermes -p X gateway
run` never behaved that way.
- tools/terminal_scope.py: resolve the terminal.cwd placeholder inside the
profile scope with the same rule gateway/run.py applies at import (local ->
$HOME, sandbox default otherwise) so the system prompt, context files and the
terminal of a routed turn start where the profile's standalone gateway would.
- tools/image_source.py, credential_files.py, image_generation_tool.py,
skills_tool.py, delegate_tool_progress.py, agent/tool_executor.py: read
TERMINAL_ENV / TERMINAL_CWD through the terminal scope.
- tools/approval.py (+ approval_floors.py): one permanent allowlist per routed
profile home; the unscoped module set stays for single-profile processes.
- agent/redact.py: `_redact_enabled()` resolves security.redact_secrets for the
routed profile (scope .env, then config); launch snapshot kept when unscoped.
- tools/credential_files.py, agent/auxiliary_health.py, agent/lsp/__init__.py,
tools/browser_tool_cloud.py, tools/mcp_tool_config.py,
tools/tool_result_storage.py: key process caches by profile home (or bypass
the slot under an override).
Tests: tests/tools/test_multiplex_turn_parity.py (4, red on base).
Docs: multi-profile-gateways.md isolation table.
A profile served by the default multiplexer owns no gateway.pid / gateway_state.json,
so every surface that reads per-profile identity files called it stopped while the CLI
status surfaces (hermes -p X status / gateway status / cron status) said "running via the
default-profile multiplexer":
- `/api/status?profile=X` and `/api/messaging/platforms?profile=X` reported
gateway_running=false / state=None / "gateway_stopped" in the same body that listed X
under gateways[].served_profiles. The shared ladder `resolve_gateway_liveness` gains a
fourth rung for a named profile_dir: the live default multiplexer that records X in
served_profiles IS X's gateway (pid = multiplexer pid, runtime = its record, X's
`<X>:<platform>` entries re-keyed to the standalone shape).
- `POST /api/gateway/stop?profile=X` spawned `hermes -p X gateway stop`, which printed
"No gateway running for this profile" (exit 0) into the action log while the UI flipped
to stopped and the multiplexer kept serving X; `/api/gateway/restart?profile=X` spawned
a `-p X gateway restart` that only exits 78. start/stop now answer 409 with the
multiplexer explanation (one helper shared with the existing start refusal) and restart
targets the multiplexer, the process that actually serves X. A `--force`-started
separate gateway for X (own pid file) keeps normal per-profile management.
- CLI `hermes -p X gateway stop` refuses with exit 78 like run/start/install/restart when
X has no gateway of its own, instead of a contradictory exit-0 "not running".
Docs: multi-profile-gateways.md §1 and §5 describe stop + the dashboard behaviour.
The "one gateway for all profiles" section had drifted from the runtime. Each
claim was re-verified at its defining symbol on current main and rewritten to
the behaviour users will actually see:
- named-profile guard: only `gateway run` refuses (exit 78 / EX_CONFIG,
systemd RestartPreventExitStatus; launchd KeepAlive still retries);
`start`/`install` do not refuse themselves and the message lands in the
service log; `--force` is a `run`-only flag
(hermes_cli/gateway.py::_guard_named_profile_under_multiplexer, _cmd_start)
- same (platform, token) in two profiles: the duplicate adapter is parked as
fatal/duplicate_credential and the gateway keeps running — it was described
as a fail-fast startup error (gateway/run_adapters.py::_refuse_duplicate_claim)
- status surfaces: one gateway_state.json under the default home with
`<profile>:<platform>` entries + served_profiles; nothing is written under a
secondary home (the page claimed a per-profile runtime_status.json), and
`hermes status` does not list served profiles — `gateway list`,
`-p X gateway status` and /api/status do (gateway/status.py::write_runtime_status,
hermes_cli/status.py::_render_gateway, hermes_cli/gateway.py::_cmd_status)
- API_SERVER_KEY in a secondary .env auto-enables api_server and trips the
port-binding skip; document the `enabled: false` pin
(gateway/config_env.py::_api_server / _enable_from_env)
- allowlist: it is a start-time snapshot, and the Desktop backend's cron
ticker enumerates every local profile regardless of it
(hermes_cli/web_server.py::_start_desktop_cron_ticker)
- routed-profile cron via the shared bot: only when the profile has no live
adapter of its own, and a route carrying guild_id never matches a cron
target because delivery matches on chat_id/thread_id only
(cron/scheduler_provider.py::tick_adapters_for,
cron/scheduler_preflight.py::SharedRouteAdapters.get)
- add a "What is isolated per profile" table (credentials, authorization,
endpoints, media denylist, MCP child env, outbound egress, session
namespace, logs, terminal) describing behaviour, not PR numbers
- configuration.md: the ${VAR} scoping paragraph now says where it applies
and links to the table
One multiplexed gateway process serves every profile, but several per-turn
reads still went through state frozen from the LAUNCH profile:
- `_current_max_iterations` re-bridged `agent.max_turns`/`sessions.*` from the
module constant `_hermes_home` into one process-wide HERMES_MAX_ITERATIONS,
so every secondary ran with the default profile's turn budget. A routed turn
(HERMES_HOME override) now resolves `agent.max_turns` from its own config.
- `_refresh_fallback_model` read `_hermes_home/config.yaml` into one runner-wide
slot, so secondaries fell back through the default's provider/model with their
own keys. It now reads the active gateway home and keeps a last-known-good
chain per home.
- `_load_prefill_messages` resolved relative paths against the launch home.
- `agent/auxiliary_client._AUTH_JSON_PATH` was an import-time constant, so a
secondary's compression/title/vision calls authenticated to Nous with the
default profile's token when it had no pool entry. Resolved per call via
`hermes_cli.auth._auth_file_path()` (patched constant still wins in tests).
- `gateway/hooks.HOOKS_DIR` was frozen at import and one `HookRegistry` was
loaded outside any profile scope, so secondaries' `hooks/` never ran and the
default profile's handlers received every profile's messages, responses and
user ids. `HOOKS_DIR` now resolves per call (salvaged from #56508) and the
runner holds one registry per served home, picked from the active scope at
emit time and front-loaded under each secondary's startup scope.
- Shell-hook subprocesses inherited the launch `os.environ` (default HERMES_HOME
and the default profile's secrets). They now get the routed HERMES_HOME via
`build_subprocess_env`, scrubbed under multiplexing, and the stdin payload
carries `profile` so one script can tell which profile fired it.
- Media-delivery policy (`gateway.strict`, `media_delivery_allow_dirs`,
`trust_recent_files*`) was bridged once into env at startup and read from env
per delivery; under a HERMES_HOME override the validator now reads the routed
profile's config. Single-profile runs keep the env-bridge contract.
Audit: /tmp/mux_audit F3, F4, F6 (auth.json half), F7, F12 (media). Live repro
(temp HERMES_HOME A with profiles/B): before, B saw max_iterations 7,
fallback A/fallback, TOKEN_A, A's hooks, strict=A; after, all B's values.
Telegram DM chat_id == user_id for every bot, so a `chat_id` route meant to
pin a user's DM with the SHARED bot to profile `ops` also captured that
user's DM with team_b's dedicated bot: the turn, session key, state.db and
secrets were ops's while the reply left via team_b's transport.
`ProfileRoute` gains an optional `bot_profile` discriminator (None = the
default profile's bot) and `matches()` requires it to equal the receiving
adapter's owner profile. `build_source` passes `_owner_profile` and stamps it
as the fallback profile for secondary adapters; `_profile_name_for_source`
derives it from the transport ref for sources built elsewhere.
Secondary→secondary routes remain possible by naming the bot explicitly.
Salvages the discriminator idea from #105040 and #105049 in a smaller shape
(no bot-username sniffing across adapter internals; the owning profile is
already declared at `set_owner_profile`).
Co-authored-by: 0xAlyDev <agentai891@gmail.com>
Co-authored-by: Ahmett101 <Ahmett101@users.noreply.github.com>
Under gateway.multiplex_profiles the default gateway serves every profile, yet
four startup/status paths still reasoned from the wrong source:
* A secondary profile's API_SERVER_KEY (which the docs REQUIRE for /p/<profile>/
auth) auto-enabled api_server in that profile's config, so
_load_secondary_profile_config raised SecondaryPortBindingConfigError and the
whole profile was skipped. gateway/config_env.py::_enable_from_env now leaves
`enabled` alone for port-binding platforms while a multiplexer loads a
NON-default profile (home override + multiplex flag, the same signal
gateway.config uses for scoped reads); the credential still lands in extra so
the shared listener can authenticate the prefix. Default profile unchanged.
* "Is this profile served?" was re-derived from the default config.yaml plus
GATEWAY_MULTIPLEX_PROFILES as seen by the CLI process. `hermes -p coder ...`
loads coder's .env, so an env-only opt-in on the default profile was invisible
(guard never fired, status said stopped) and an allowlist edit flipped the
answer before the restart. named_profile_served_by_running_multiplexer now
reads the pid-verified default gateway_state.json served_profiles (written by
_record_served_profiles) first and falls back to config derivation only when
the key is absent. The record helpers live in hermes_cli/gateway_multiplex_served.py.
* The served-profile guard ran only inside `gateway run`. `hermes -p X gateway
start|install|restart` reached the service manager, whose unit then exited 78
forever (systemd parks it while the CLI prints "started"; launchd KeepAlive
respawns every 30 s). The service verbs now run the same guard up front
(exit 78, same message) and accept --force; the Desktop /api/gateway/start
route returns 409 for a served profile instead of spawning a doomed child.
* Status surfaces disagreed: `hermes -p coder status` said stopped, `hermes -p
coder cron status` said "cron jobs will NOT fire" while `cron list` said fine,
and the default `hermes status` never listed served profiles. Both now route
through the probe / the recorded served set. The -p/--profile matcher in
_scan_gateway_pids and gateway.status._command_line_belongs_to_profile compares
the flag token for equality (`-p ops` no longer claims -- or lets `gateway
stop` SIGTERM -- an `-p ops-2` gateway).
Docs: multi-profile-gateways.md now describes the start/install refusal, the
--force flags, the API_SERVER_KEY behaviour and the single default-home
gateway_state.json (the per-profile runtime_status.json claim was wrong).
Fixes#100397
Addresses #89726#97360#71344
(cherry picked from commit d002c1864a7b6a22c53758b16b7b0cc79aea2edf)
Under gateway.multiplex_profiles every connection ledger in tools/mcp_tool.py
(_servers, _server_scope_keys/_server_tool_scopes, connecting/error/cooldown
maps, the circuit breaker, lazy schema-cache configs, trust metadata) was keyed
by the bare server NAME. The common per-tenant layout — each profile names its
server `github`/`notion` with its own token — gave only the first profile a
connection: the second profile's register_mcp_servers saw the name as "already
connected", refused to adopt it (different credentials, 4ddbcbd35e), and left
the profile silently tool-less with a healthy-looking `configured` status
(#106005 Bug 1/2, #91654). Siblings of the same bug: profile A's failing `x`
put profile B's healthy `x` into A's 10-minute connect cooldown and A's open
circuit breaker short-circuited B's calls; toolsets._resolve_toolset_memo was
not scope-keyed, so B resolved A's `mcp-<server>` tool names.
Keys are now the connection key from the new tools/mcp_tool_scope.py: the bare
name outside a multiplexer (single-profile processes are unchanged) and
(owner_scope, name) under one. Call-time lookups (_resolve_server_key) prefer
the calling scope's own connection, then a shared connection it adopted, so
identical-route profiles still share one subprocess. _select_new_servers,
the cooldown/breaker/trust maps, lazy registration and get_mcp_status all read
and write through the composite key; teardown resolves a task's key by
identity (the MCP loop has no profile context). The toolset memo key includes
registry.current_scope_key().
An owner's scoped /reload-mcp tore down its connection and, with it, every
adopting profile's tool overlay; nothing re-ran the adopters' discovery until
they reloaded. shutdown_mcp_servers(scope=) now records the orphaned adopters
and register_mcp_servers re-registers them under their own home + secret scope
once the owner's rediscovery pass completes.
Docs: multi-profile-gateways.md states the per-profile connection rule.
Fixes#106005Fixes#91654
Co-authored-by: Bergmann89 <info@bergmann89.de>
Co-authored-by: Izzy-Gottz <srulynj@gmail.com>
Under gateway.multiplex_profiles, os.environ holds the DEFAULT profile's .env; a
secondary profile's values exist only in the per-turn secret scope. Every reader
below still read os.environ/os.getenv at call time, so a secondary profile's turn
silently used the default profile's value.
Credentials (F6): FIRECRAWL_API_KEY (read_file hosted OCR), OPENVIKING_API_KEY,
mem0-OSS OPENAI_API_KEY, MODAL_TOKEN_ID/SECRET and BROWSER_USE_API_KEY presence
gates, and the xAI video plugin's os.getenv("XAI_API_KEY") fallback AFTER the
scoped resolver had already missed — the exact fallback-after-miss shape
gateway/AGENTS.md forbids. Deleted, not re-scoped: the resolver is the scope.
Identity / tenant (F7): MEM0_USER_ID/AGENT_ID/HOST/MODE, SUPERMEMORY_CONTAINER_TAG,
RETAINDB_PROJECT, OPENVIKING_ACCOUNT/USER/AGENT (and the whole layered() env
read), HINDSIGHT_BANK_ID/MODE/retain shaping, HERMES_HONCHO_HOST. A raw read
put a secondary profile's memories into the default profile's account/bank/
project/tenant and recalled them back into the default's turns. Each now uses
get_secret with the provider's own per-profile default on a miss.
Endpoints (F8): OPENAI_BASE_URL (aux custom runtime + direct-alias expansion),
XAI_BASE_URL/HERMES_XAI_BASE_URL (aux OAuth), NOUS_INFERENCE_BASE_URL (#65941,
both the aux builder and hermes_cli.auth_nous._nous_inference_env_override),
GATEWAY_PROXY_URL (same UnscopedSecretError-only fallback shape as
GATEWAY_PROXY_KEY three lines below), FIRECRAWL_API_URL, BROWSERBASE_BASE_URL,
SUPERMEMORY/RETAINDB/HONCHO/HINDSIGHT URLs. The keys beside them were already
scoped, so a secondary's key was sent to the default profile's proxy or host.
Targets / display (F11): WEIXIN_HOME_CHANNEL (message posted into the default's
chat), HERMES_LANGUAGE, and agent/i18n's process-wide lru_cache of
display.language — now keyed by HERMES_HOME.
Outbound webhooks: hooks.outbound[].secret_env resolved from os.environ while
the gateway registers each profile's targets inside that profile's scope, so a
secondary's deliveries were signed with the default's secret or left unsigned.
Agent-cache eviction: _spawn_release_thread started a bare threading.Thread, so
commit_memory_session -> provider on_session_end ran with an EMPTY context. The
thread now runs copy_context() and, for the unscoped housekeeping sweep, enters
the owning profile's _profile_runtime_scope resolved from the session key
(agent:<profile>:...). The pressure batch does the same per key.
session_search (#82903): agent/inline_tool_executors.py::_session_search
forwarded every schema argument except `profile`, so a gateway agent could
never select a named profile's store. Forwarded; the ownership-scoping design
in #87779/#87847 is a separate design call and is not attempted here.
Live repro (/tmp/mux_audit/fix-tool-memory-reads/repro.py): 28 FAIL on
origin/main -> 0 FAIL with this change; 10 new invariant tests red on base.
Fixes#82903Fixes#65941Fixes#99121
Addresses #87779
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Michael Versluis (Berry) <michael@wve.nl>
Under gateway.multiplex_profiles, os.environ holds the DEFAULT profile's .env. Several
adapter-owned authorization gates still read GATEWAY_ALLOW_ALL_USERS, GATEWAY_ALLOWED_USERS
or their platform allowlist/allow-all raw from os.environ, so the default profile opting
into open access opened every secondary email/QQ/WhatsApp/Matrix/Teams/Slack/LINE/DingTalk
bot to any sender (email additionally skipped From: authentication), the default's Matrix
allowlist decided who may approve tool calls on a secondary bot, and a secondary that
opted in only in its own .env was silently deny-all.
Every such read now goes through the adapter's existing module-local scoped reader
(gateway.platforms._shared.get_scoped_secret / matrix _startup_env_secret): profile
scope first, scoped miss = default, never os.environ; the unscoped default-profile and
single-profile paths keep the environ read, where it IS the profile's own value.
Sites: email _allow_all_senders/_allowlist_in_effect; qqbot _open_dm_opted_in;
whatsapp_common _open_dm_opted_in/_live_dm_allow_from; teams _card_action_denied;
matrix _is_authorized_user, MATRIX_ALLOWED_USERS, MATRIX_IGNORE_USER_PATTERNS,
_extra_csv_set (allowed/free-response rooms); slack _slack_allow_bots/_slack_api_human_users;
line _truthy_env/allowlist (allow-all, user/group/room allowlists); dingtalk _extra_get
(allowed_users/chats, free-response chats, require_mention).
Live repro (temp HERMES_HOME, multiplex on, default env GATEWAY_ALLOW_ALL_USERS=true,
secondary scope without opt-in): EmailAdapter._allow_all_senders() True -> False,
QQAdapter._open_dm_opted_in() True -> False, Matrix _is_authorized_user('@stranger')
True -> False, Teams card action allowed -> denied.
Co-authored-by: Drexuxux <drexux0@gmail.com>
Co-authored-by: MoonsvnLyn <FirmamentalSpring@users.noreply.github.com>
Co-authored-by: svector-anu <anuoluwakolapo94@gmail.com>
Co-authored-by: babatorik <durgun.ismail@gmail.com>
Co-authored-by: salch-cred <salch-cred@users.noreply.github.com>
Under gateway.multiplex_profiles a turn running for a secondary profile P resolved
its live adapter by bare platform from runner.adapters — the DEFAULT profile's map —
so P's send_message tool calls (send/react/media on slack, matrix, wecom, buzz, ntfy,
every plugin platform), its "Gateway shutting down/restarted" and /update notices,
its /loop wakeups, and its Discord bot's unauthorized-slash operator alert all left
through the default bot (Telegram DMs landed in the user's chat with the other bot).
Every such door now resolves through the profile-aware, fail-closed resolver already
used by the inbound reply path (authz_mixin: _adapters_for_profile /
_authorization_adapter / _adapter_for_source): P's own adapter, or None → a clear
error, never the default bot.
- tools/send_message_senders.py::_live_adapter — resolve via
runner._authorization_adapter(platform, get_active_profile_name()); shared by
_send_via_adapter, _handle_react/unreact, media sends, matrix E2EE fast path and
the WeCom standalone sender.
- gateway/authz_mixin.py — extract _adapters_for_profile (the whole map, for relay-
aware resolve_delivery_transport callers); _authorization_adapter reuses it.
- gateway/run_shutdown.py — shutdown/restart notice for a running session uses the
session's source transport / agent:<profile>: key lane, never self.adapters.
- gateway/slash_commands.py + run_notifications.py — /restart and /update markers
persist `profile`; the restart notice, update result and update prompt resolve the
requester's own adapter (legacy markers fall back to the session_key lane).
/goal, /heartbeat, /approve, /deny confirmations use the source's own transport.
- gateway/slash_commands_goals.py + run_goals.py — /loop persists `profile` in its
route; the wakeup watcher scans every served profile's store under its own scope
(same shape as _handoff_watcher) and fires through that profile's adapter map.
- plugins/platforms/discord/adapter.py::_notify_unauthorized_slash — alert stays in
the owning profile (its adapters and its home channels).
- plugins/platforms/wecom/adapter.py::_standalone_send — via _live_adapter.
- docs: website/docs/user-guide/multi-profile-gateways.md (outbound identity).
Tests (red on base): tests/tools/test_send_message_multiplex_profile_adapter.py,
tests/gateway/test_multiplex_notice_egress_profile_adapter.py.
Co-authored-by: pierrenode <298902573+pierrenode@users.noreply.github.com>
Under gateway.multiplex_profiles a /p/<profile>/ webhook route (or one with
`profile: <name>`) executed under that profile but delivered its reply as the
FIRST profile owning the target platform: `_find_adapter` took
`runner.adapters` then iterated `_profile_adapters` in dict order, and the
home-channel fallback read `runner.config` (the default profile's). The gh leg
for `github_comment` inherited the process environ, i.e. the default profile's
GH_TOKEN. Both directions leaked: a secondary route posted through the default
bot, and a default route borrowed a platform parked only on a secondary.
The api_server `/p/<profile>/api/platforms/<platform>/events` callback had the
same shape — `_get_platform_callback_adapter` read `runner.adapters` regardless
of `_api_request_profile`, so a secondary's Google Chat / Teams events were
verified and dispatched by the default adapter.
Now:
- webhook: `_delivery_info` / the deliver_only dict carry the resolved
profile; `_find_adapter(platform, profile)` resolves through the runner's
shared fail-closed `_authorization_adapter`; the delivery leg runs inside
`_profile_scope(profile)` and takes the home channel from that profile's
`load_gateway_config()`; `gh` gets GH_TOKEN/GITHUB_TOKEN from the profile
secret scope (default's values dropped from the child env when the profile
has none).
- api_server: the callback adapter resolves via
`_authorization_adapter(platform, _api_request_profile.get())`; a named
profile without the adapter is a 503, never the primary's adapter.
A profile without the target platform fails closed ("not connected" → 502 /
503) instead of a silent cross-profile send.
Fixes#65939Fixes#84266
Co-authored-by: 604maestro <604maestro@protonmail.com>
Co-authored-by: mjshorty <mjshorty@users.noreply.github.com>
Co-authored-by: StellarisW <stellarisw@users.noreply.github.com>
Under a multiplexed gateway, `_build_safe_env` forwarded `os.environ[name]` for
every name tagged in the process-global `_SECRET_SOURCES` map. That map is
filled by EVERY served profile's secret-source hydration, while `os.environ`
only ever holds the LAUNCH (default) profile's values — so once any profile's
1Password/Bitwarden source supplied e.g. GITHUB_TOKEN, every profile's stdio
MCP server was started with the default profile's token.
Resolve those names through the active profile's secret scope (`get_secret`)
instead: the routed profile's value, or omitted when that profile has none.
Under multiplex `get_secret` never falls through to environ; single-profile
runs keep the .env overlay + environ behaviour, so the existing "vault vars
reach MCP subprocesses" contract still holds there. `secret_source_names()`
exposes the tagged NAMES only — values are never read from the shared map.
Docs: the multi-profile guide's "MCP subprocesses only see their own profile's
secrets" claim is now true for source-injected names too; say so explicitly.
Under `gateway.multiplex_profiles` one process serves every `<root>/profiles/*`,
but `_media_delivery_denied_paths` expanded `_ROOT_CREDENTIAL_PATHS` only under
the import-time `_HERMES_HOME` / `_HERMES_ROOT`. A `MEDIA:<root>/profiles/<B>/.env`
(or auth.json, state.db, config.yaml, sessions/, mcp-tokens/) emitted in ANY
profile's turn — including B's own, whose HERMES_HOME override was never
consulted — passed validation and was natively uploaded to the chat. The ALLOW
side (`_profile_cache_roots`) already enumerated profiles at check time; the
DENY side did not.
`_credential_home_roots()` now yields the active `get_hermes_home()`, the shared
root and every `<root>/profiles/*` at check time (shared `_profile_dirs()` with
the allow side), and the denylist is built from that. Profile cache artifacts
and plain agent-written files under a profile stay deliverable.
Live repro (/tmp/mux_audit/fix-media-denylist/repro.py): before, all five of
profiles/B/{.env,auth.json,state.db,config.yaml,sessions/s1.json} validated as
deliverable while <root>/.env was blocked; after, all None, cache/images/gen.png
and report.pdf still deliverable.
No prior report. Write-side analogue: #107327 / #107335 (memo keying, different
mechanism — left as is).
`ProfileRoute.matches()` compared `chat_id` as an exact string, so a
WhatsApp route written as a phone number never matched the JID
(`…@s.whatsapp.net`) or LID (`…@lid`) the bridge actually delivers, and
the inbound fell through to the default profile. Allowlists and session
keys already canonicalize these via `gateway.whatsapp_identity`.
Exact compare still wins first; only whatsapp / whatsapp_cloud *user*
chats get the alias intersection fallback (applied to both `chat_id` and
`parent_chat_id`). Groups, broadcasts and every other platform stay exact.
Salvaged from #85081 (Kong); parent_chat_id fallback added on top.
Follow-ups on the salvaged #20096 profile-routing feature:
- _profile_name_for_source now returns None unless gateway.multiplex_profiles
is on. Routing stamps source.profile, which namespaces session/batch keys,
but the profile-scoped agent run only activates under multiplexing — without
the gate, configured routes with multiplexing off split batch/session keys
into agent:<profile> while the agent still ran from agent:main.
- Widen the profile-aware _text_batch_key fix from Discord to every adapter
that builds batch keys via build_session_key (telegram, whatsapp, matrix,
feishu, wecom, weixin) — routing is platform-generic, so the batch-key
namespace fix must be too.
- Downgrade the no-route-matched log from INFO to DEBUG (fired on every
unrouted inbound message).
- GatewayConfig.to_dict(): serialize profile_routes as plain dicts
(ProfileRoute dataclasses are not JSON-safe).
- Docs: correct the 'independent of multiplexing' claim in
docs/profile-routing.md (routing requires multiplexing), fix the
platform-only specificity row (0, not 1), and document profile_routes in
website/docs/user-guide/multi-profile-gateways.md.
- Tests: pin the multiplex gate (routes ignored when off, active when on,
build_source end-to-end stays in agent:main when off).
Extend the 'Running Many Gateways at Once' user-guide page with a
'one gateway for all profiles (multiplexing)' section, kept to a single page:
- How to opt in (gateway.multiplex_profiles on the default profile) and when to
prefer it vs one-process-per-profile.
- Every contract change a user sees when the flag is on:
1. secondary-profile 'gateway start' is a hard error (--force escape hatch),
2. HTTP-inbound reached via /p/<profile>/ prefix; secondary profiles must NOT
enable a port-binding platform (webhook/api_server/msgraph_webhook/feishu/
wecom_callback/bluebubbles/sms) — config error at startup,
3. per-credential platforms still need their own token per profile,
4. session keys namespaced agent:<profile>: (default stays agent:main:),
5. single PID/lock + aggregated hermes status, per-profile runtime_status.json.
- What does NOT change: per-profile .env credential isolation (stricter, incl.
MCP/Kanban subprocess env), Kanban, profile-scoped skills/memory/SOUL, routing.
All inert when the flag is off.
Covers running multiple Hermes profiles as managed services on one host:
- A shell-loop wrapper pattern for start/stop/restart/status across every
profile (the per-profile CLI commands stay unchanged).
- Per-platform service file locations (LaunchAgent on macOS, systemd user
unit on Linux), plus the rules around clashes.
- Log paths per profile and how to tail every gateway at once.
- Config file layout per profile and the restart-after-edit workflow.
- Keeping the host awake: caffeinate flags on macOS,
systemd-inhibit + loginctl enable-linger on Linux.
- Token-conflict auditing across .env files.
- Troubleshooting for the common "Could not find service in domain for
user gui: 501" message and stale PIDs after a crash.
Tested locally with five profiles on macOS launchd.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
teknium1