Commit Graph

2798 Commits

Author SHA1 Message Date
Teknium 600d5166f0 test(gateway): prove delivered rows are never reclaimed by the reconnect sweep 2026-08-26 04:49:39 -07:00
milnerrad 8e1db41041 fix(gateway): redeliver transient failures after reconnect 2026-08-26 04:49:39 -07:00
Teknium 2664599644 test(gateway): prove the profile_route_rejected sentinel is observable
Review follow-up for the salvaged #94848: the ProfileRouteRejected marker
looked write-only inside the primary handler. Document that
_handle_message's ingress gate reads the same marker to drop the message
fail-closed, and add a regression test showing the rejected route is
stamped once, dispatch falls back to the default home, and routing is not
re-run on redelivery.
2026-08-25 23:15:15 -07:00
GarrettGlass 2afed50863 fix(gateway): authorize routed messages in transport scope 2026-08-25 23:15:15 -07:00
GarrettGlass 9ab748abb9 fix(gateway): scope routed history before session lookup 2026-08-25 23:15:15 -07:00
ruangraung dce4abe917 fix(gateway): log secondary startup-reconnect handoff failures instead of dropping them
Review follow-up to the AI code-review pass on PR #92074: the bridge task's
handoff into _schedule_secondary_profile_reconnect was unguarded at both call
sites inside the parked coroutine. The scheduler touches live registries
(_profile_failed_platforms slot creation, background-task registration), so an
unexpected raise there would kill the parked task as an unretrieved-task
exception — logged only at GC time via "Task exception was never retrieved",
where no operator ever looks. A fix whose entire purpose is to stop a platform
dying silently should not contain its own silent-death path; both handoff sites
now wrap the scheduler call with logger.exception so the failure lands in
gateway.log with profile and platform context.

The early-exit branch (gateway already _running when the bridge starts) had the
identical exposure and is guarded the same way — same bug class, fixed together.
Regression test drives a handoff raise end-to-end through the real bridge task:
the await completes cleanly, the error is captured in gateway.run's logger, and
no adapter or failed-platform slot leaks behind the failed handoff.
2026-08-25 22:55:07 -07:00
ruangraung 96489f3c1b fix(gateway): schedule secondary-profile reconnect when initial adapter connect fails
When gateway.multiplex_profiles is active, a secondary profile whose platform
adapter fails its initial connect at startup was silently given up on: the
failure branches in _start_one_profile_adapters() logged and disconnected, but
never scheduled recovery. One unlucky connect window during a Telegram API
outage left the profile permanently silent until manual restart (~80 min in
the observed incident), while the mid-run fatal path already recovers via
_handle_profile_adapter_fatal_error() -> _schedule_secondary_profile_reconnect().
The same gap hit both failure shapes: a clean False return from
_connect_initial_adapter_with_timeout() and an exception escaping it.

Fix: call _schedule_secondary_profile_startup_reconnect() from both startup
failure branches after _safe_adapter_disconnect(). Because secondary adapters
are started mid-start(), before self._running flips True, the regular
scheduler's not-self._running guard would silently drop the request — so the
new bridge parks a background task until startup completes (or shutdown
begins) and then hands off to _schedule_secondary_profile_reconnect()
verbatim: backoff, fresh-adapter rebuild under the profile runtime scope,
slot dedupe, and shutdown cancellation all come from the existing path.
Non-retryable failures are dropped at scheduling time exactly as the regular
scheduler drops them, keeping duplicate-credential/auth-failed startups dead
instead of looping.

Fixes #92064
2026-08-25 22:55:07 -07:00
Teknium 15f7b7293c fix(terminal): persistent Docker containers are profile-scoped, not per-session
Commit a270c4ade's session-key fallback in _resolve_container_task_id was
added to stop cross-profile SSH environment reuse, but it wasn't backend-
gated: persistent Docker silently fragmented into one container per gateway
session, breaking the product contract (one long-lived container per profile,
shared by CLI and every session of that profile). #93950's vanishing MEDIA
attachments were downstream damage.

- persistent Docker (container_persistent: true) now keys to the profile:
  literal 'default' for the default profile (same container as CLI),
  'profile:<name>' for named profiles
- SSH and non-persistent Docker keep session scoping (the original leak fix
  and the #82731 isolation contract are untouched)
- gateway MEDIA translation follows the profile layout and keeps the legacy
  bug-window per-session sandboxes as fallback candidates, trying each until
  the file resolves — old sessions self-heal, no migration
- /root/.hermes credential-surface refusal preserved across all layouts
2026-08-25 02:30:38 -07:00
kshitijk4poor d634b37047 refactor(gateway): retire private repair alias per replay_cleanup precedent
Phase 2c on the full final diff flagged the re-export shim as
contradicting the adjacent house pattern (agent.replay_cleanup import,
which documents retiring private aliases once tests migrate). Migrate
all six tests to the canonical gateway.media_repair seam, import the
canonical name in run.py, and drop the dead 'and result' guard at the
background-task call site.
2026-08-25 13:25:48 +05:30
kshitijk4poor 105999a0c9 refactor(gateway): unify computer-use repair call sites after review
- Make repair_explicit_computer_use_media_paths fail-open internally
  (cosmetic repair must never abort delivery); drop the cron-only
  try/except so all three call sites are identical one-liners.
- Drop cron's redundant 'MEDIA:' pre-check (helper early-returns).
- Document the intentional lazy BasePlatformAdapter import (verified:
  no cycle either way; keeps module import cheap for cron processes).
- Point the two new regression tests at the canonical
  gateway.media_repair seam; pre-existing tests keep pinning the
  gateway.run re-export shim.
- Docstring: matching is case-insensitive, say so.
2026-08-25 13:06:43 +05:30
kshitijk4poor bb0d5503c2 fix(gateway): widen computer-use media path repair to sibling surfaces
Follow-up to the salvaged fix from PR #94439:

- Extract the repair into gateway/media_repair.py (shared module) and
  re-export under the historical private name in gateway/run.py.
- Wire the repair into the two bypassed delivery surfaces: gateway
  background tasks (_run_background_task_inner) and cron job delivery
  (cron/scheduler.py) — both call agent.run_conversation directly and
  never pass the main turn chokepoint.
- Fail closed on malformed/truncated JSON tool results: parse JSON-looking
  content first instead of regex-scanning the raw string, which yielded a
  doubled-backslash path artifact and rewrote the response to a path the
  model never wrote.
- Deduplicate the tool_name_by_call_id builder (three verbatim copies in
  gateway/run.py) into the shared module; hoist the abs-path prefix regex.
- Add regression tests: malformed-JSON fail-closed (mutation-checked) and
  the compression-fallback last-user slice (incl. no-user fail-closed).
2026-08-25 13:06:43 +05:30
Michael Nguyen 34041faea8 test(gateway): accept session_key kwarg in media resend dedup stubs
Same follow-through as the other filter-static stub updates: the three
lambdas patching filter_local_delivery_paths rejected the new keyword
and failed CI (tests/gateway/test_73771_media_resend_dedup.py).
2026-08-24 23:50:23 -07:00
Michael Nguyen a15533b646 feat(gateway): warn when a Docker sandbox MEDIA path fails translation
De-silence the #93950 failure mode: when a container-absolute MEDIA path
under /workspace or /root cannot be resolved to a host sandbox file while
TERMINAL_ENV=docker, log the reason (no mounts / no prefix match / host
file missing) plus the delivering session key instead of only the generic
'Skipping unsafe MEDIA directive path' line upstream.
2026-08-24 23:50:23 -07:00
Michael Nguyen d4f31a8f36 fix(gateway): resolve session-scoped Docker sandboxes for MEDIA delivery (#93950)
Persistent Docker containers bind <sandboxes>/docker/<task>/{workspace,home}
where <task> is sanitize_task_id_for_path("session:<session_key>") — but the
gateway's synthetic mounts hardcoded the literal "default" sandbox
(_default_docker_workspace_host_root / _docker_persistent_home_host_root).
For any session-scoped deployment the longest-prefix match missed, the
container path fell through to a host-filesystem resolve that could not
exist, and every MEDIA attachment was silently dropped.

The post-handler delivery pipeline also runs after
_handle_message_with_agent cleared the turn's session contextvars, so even
a correct sandbox derivation consulting ambient state would collapse onto
"default". Thread the delivering session's key explicitly through
validate_media_delivery_path -> _translate_docker_container_media_path ->
the two host-root helpers (same pattern as the TTS fix for #57049/#36685).

Default-sandbox resolution and the /root/.hermes credential exclusion are
preserved; contexts without a key keep the historical behavior.
2026-08-24 23:50:23 -07:00
Gille 1fac440865 fix(gateway): recover explicit computer-use media paths 2026-08-24 23:39:05 -07:00
xxxigm fcda325cd4 test(telegram): cover send() waiting for reconnect after a network blip
Pin immediate replacement, mid-wait restore, timeout retryable=True,
and permanent-fatal fail-closed without waiting.
2026-08-25 12:07:36 +05:30
xxxigm 27640c5844 test(teams): cover connect when the namespace exists but App is unbound
Pin the NoneType crash: TEAMS_SDK_AVAILABLE true plus a failed bind must
return False without calling App(). Also pin plugin import when the
microsoft_teams parent namespace is missing.
2026-08-25 12:07:33 +05:30
lkz-de cbc8d1804d fix(signal): chunk long cron deliveries instead of truncating 2026-08-24 20:03:24 -07:00
kshitijk4poor 111d809562 refactor(telegram): migrate _await_with_thread_deadline onto agent.deadline.run_bounded_async (#85125 2f)
The adapter's private thread-deadline helper was the ancestor of the
unified deadline layer's run_bounded_async (#85147 was extracted from
it, plus the caller-cancellation leak fix the original still lacked).
Consolidate: the helper body becomes a thin wrapper mapping
BoundedResult.timed_out back to the asyncio.TimeoutError its 9 call
sites (the PTB retry ladder) expect. ~90 duplicated lines die, along
with the adapter-local copies of the abandon-cleanup runner and the
blocked-loop faulthandler diagnostics (both live in agent/deadline.py).

Everything the call sites rely on is preserved by the unified layer:
- thread-timer deadline that survives a blocked event loop (#63309)
- abandonment of cancellation-shielded tasks (PTB/httpcore anyio init)
- detached best-effort on_abandon cleanup (no httpx pool leak per retry)
- off-loop stack dump when the loop never processes the expiry
Plus one behavior IMPROVEMENT inherited from the shared copy: a caller
cancelling the wrapper no longer leaks the inner task unobserved (the
telegram original had that leak; the extraction fixed it).

test_telegram_init_deadline.py: the #63309 diagnostics probe now pins
the shared layer's dump hook (label "telegram-init") — same contract,
new seam. Wedge + cleanup-crash tests pass unchanged.
2026-08-24 17:04:55 +05:30
Teknium ec5e369fe6 Merge pull request #93784 from NousResearch/salv/81234-retry-carrier
fix: /retry and /undo no longer replay an older message after compaction (#81233, salvage #81234)
2026-08-24 03:31:29 -07:00
Shanthan Subramaniam e210fd8c1f fix(gateway): resolve PairingStore's default pairing dir lazily, not at import time
PairingStore(profile=None) resolved its storage directory from the
module-level PAIRING_DIR constant, which was computed exactly once, at
module import time. A long-lived process (the gateway, started once at
container/process boot) can import this module before HERMES_HOME or a
profile's context is fully established, freezing PAIRING_DIR to a wrong
value for the rest of that process's lifetime -- even though a freshly
started, short-lived process (e.g. the `hermes pairing` CLI) re-imports
the module later with the environment already correct.

That asymmetry is exactly what made pending pairing codes issued by the
gateway process unrecoverable (the pending-code write landed under the
stale, wrong directory) while CLI-invoked writes to the same nominal
directory kept working -- see #93449 for the full writeup and a live
reproduction. tests/hermes_cli/test_dashboard_admin_endpoints.py already
carried a comment acknowledging this exact staleness in passing ("the
module-level PAIRING_DIR is bound at import"), and
TestProfileScopedStorage::test_default_store_uses_global_dir's own
comment describes working around it rather than it being intentional
behavior -- this fixes the underlying cause both were compensating for.

The profile-scoped branch already resolved its directory lazily inside
__init__ (matching this docstring's claim that resolution is lazy); this
brings the non-profile branch in line with it.

Fix keeps PAIRING_DIR as the same test seam already used throughout the
test suite (`patch("gateway.pairing.PAIRING_DIR", tmp_path)`, ~30 call
sites) unchanged: it's now a None sentinel instead of an eagerly computed
path, and a new _default_pairing_dir() helper resolves it fresh on every
call, honoring a patched (non-None) value when one is set. No existing
test needed to change.

Added a regression test that does not patch PAIRING_DIR directly and
instead exercises the real lazy-resolution path across two different
HERMES_HOME values in the same process -- confirmed it fails on the
pre-fix code (gets stuck with whatever the first PairingStore() call in
the test session happened to see) and passes with the fix.

Verified: tests/gateway/test_pairing.py (39, incl. the new one),
tests/hermes_cli/test_pairing.py, and tests/tools/test_pr_6656_regressions.py
all pass unmodified.
2026-08-24 03:20:46 -07:00
aniruddhaadak80 7befc1d2dd fix(gateway): route platform authorization reads through the profile secret scope
Under gateway.multiplex_profiles, secondary profiles are constructed
inside _profile_runtime_scope and their .env lives in the profile's
secret scope - gateway/run.py explicitly does NOT mutate os.environ with
it. Four adapters still read their AUTHORIZATION config via raw
os.getenv, so every secondary profile either (a) silently missed its own
env-only allowlists/policies (fail-closed: all DMs dropped at intake) or
(b) inherited the default profile's GATEWAY_ALLOW_ALL_USERS=true /
allowlists from the shared process env (fail-open admissions):

- weixin.py: WEIXIN_DM_POLICY / WEIXIN_ALLOWED_USERS /
  WEIXIN_GROUP_ALLOWED_USERS / WEIXIN_ALLOW_ALL_USERS +
  GATEWAY_ALLOW_ALL_USERS in _open_dm_opted_in
- yuanbao.py: YUANBAO_DM_POLICY / DM_ALLOW_FROM / GROUP_POLICY /
  GROUP_ALLOW_FROM / ALLOW_ALL_USERS (new _yb_secret helper; AccessPolicy
  hard-gates intake)
- signal.py: SIGNAL_GROUP_ALLOWED_USERS / SIGNAL_ALLOWED_USERS (new
  _sig_secret helper; empty scoped group list previously meant "drop all
  groups" silently)
- wecom/adapter.py: WECOM_DM_POLICY / WECOM_ALLOWED_USERS /
  WECOM_GROUP_POLICY / WECOM_ALLOW_ALL_USERS + GATEWAY_ALLOW_ALL_USERS -
  while credentials one line above already used _get_scoped_secret
- gateway/run.py::_own_policy_open_startup_violation: the open-policy
  startup guard validated GATEWAY_ALLOW_ALL_USERS via raw os.getenv even
  though its sibling dm/group reads already used the scoped _getenv

All reads now go through the canonical fail-closed scoped shape QQ's
_resolve_qq_secret already used (scope hit wins; unscoped single-profile
callers keep legacy os.environ behavior). Regression suite drives the
real scope contextvar across all four helpers plus the admission gates
and the startup guard, asserting both directions: profile values are
visible under multiplex, default-profile values never leak.

Fixes #93522
2026-08-24 03:20:06 -07:00
chelsealong d7e4204e77 fix(gateway): scope multiplex-profile authorization reads (weixin/yuanbao/wecom)
WEIXIN_DM_POLICY/ALLOWED_USERS/GROUP_ALLOWED_USERS, YUANBAO's equivalents,
WECOM_DM_POLICY/ALLOWED_USERS/GROUP_POLICY, and the startup guard's
GATEWAY_ALLOW_ALL_USERS check still read raw os.getenv at adapter
construction time. Under gateway.multiplex_profiles that reads the process
env instead of the per-profile secret scope, so a secondary profile either
silently drops every DM (its own env-only allowlist is invisible) or
inherits the default profile's allow-all/allowlist config.

Route these reads through the existing scoped helpers (_wx_secret,
_get_scoped_secret, gateway.authz_mixin._platform_gate_env, and
gateway.config._getenv) already used for the adjacent credential reads in
the same adapters.

Fixes #93522.
2026-08-24 03:20:06 -07:00
Teknium 1a95d0d58e Merge branch 'pr-81234' into salv/81234-retry-carrier 2026-08-24 03:15:07 -07:00
BrunoBza 2a27e1ffb3 fix(gateway): prove --replace ownership from the bound pid record
Review v2 of #93084: the readable-cmdline path used substring matching
as destructive authority. That fails open on prefix collisions
(--profile timothy vs our profile tim) and same-name profiles under
different roots, so a poisoned record could still reach SIGTERM.

Ownership is now decided by the persisted identity record ALONE — exact
_same_hermes_home equality, bound to the live target by exact pid +
start-time. Missing/legacy/unbound/foreign records all refuse. A
readable argv feeds only a token-exact consistency check
(_looks_like_profile_conflict_from_cmdline via shlex tokens) that
refuses explicit contradictions like --profile timothy under tim; bare
or matching argv adds nothing.

Also fixes the review's source-of-record concern: the guard validates
the record that authorizes get_running_pid()'s answer rather than
assuming {HERMES_HOME}/gateway.pid is always the source.

Adds the requested signal-boundary regression: start_gateway(replace=
True) with unprovable ownership returns False without calling
terminate_pid or writing a takeover marker; the bound same-home
counterpart still reaches the replace flow. Legacy replace-flow tests
updated to stage a valid bound record for their legitimate-replace
fixtures.
2026-08-24 15:37:22 +05:30
Adolanium 2912c36aa4 fix(gateway): stop multiplex allowlist leak and bot-relay python -c injection
_auth_env fell through to os.environ on a scoped miss, so one profile
could inherit another profile's allowlists and allow-all flags.

bot_relay.waiter_command put connection_id into python -c source. A
quote in the id broke the waiter. A crafted id could run extra Python
in the sender gateway.
2026-08-23 20:00:30 -07:00
joaomarcos f5a9ba9ee6 perf(bluebubbles): move attachment reads off the event loop 2026-08-23 20:00:07 -07:00
Finn763 bf15b050b1 fix(telegram): watchdog silent long-poll death via last getUpdates progress (#92991) 2026-08-23 19:26:41 -07:00
fangliquanflq ee8a66233f test(gateway): preserve replacement handles across close races 2026-08-23 18:25:12 -07:00
fangliquanflq 80cec2785d fix(gateway): preserve routing state across recovery 2026-08-23 18:25:12 -07:00
fangliquanflq 4b659f0e33 fix(gateway): retry failed session database opens 2026-08-23 18:25:12 -07:00
Teknium 12395e57b4 feat: /review command — independent reviewer subagent on every surface
/review takes the last 10 chat messages plus optional instructions,
spawns a full-privilege background subagent (the async delegation
rail) that investigates the referenced work (PR, code, docs), and its
complete review re-enters the spawning session as a normal
async-delegation completion the primary agent can act on.

- agent/review_engine.py: shared engine (snapshot, briefing,
  auxiliary.review credential resolution, dispatch, note formatting)
- tools/delegate_tool.py: internal credentials_cfg per-call override
  (never model-facing) resolved through the same credential system as
  delegation.provider pins
- auxiliary.review config block (provider/model/base_url/api_key/
  api_mode); provider auto + empty model = inherit the main model
- Surfaces: CLI process_command, gateway run.py dispatch +
  slash_commands handler (binds the approval session key so the
  completion routes back), TUI/Desktop live dispatch in
  tui_gateway/server.py, CommandDef registry (+Slack /hermes-only cap)
- Docs: delegation.md section + slash-commands.md (both tables)
- Tests: 15 engine tests (sabotage-verified: credentials_cfg and
  dispatch tests fail without the fix), 4 gateway handler tests
  through the real async rail
2026-08-23 17:38:38 -07:00
loulanyue 4e8419dadb fix(gateway): honor approval scope capabilities 2026-08-23 17:45:47 -05:00
Teknium b766607b5b test: adapt misdelivery tests to the shared profile_matches_home seam 2026-08-23 03:57:47 -07:00
Teknium 6cb1085d3d fix(gateway): /p/<profile>/ on a non-multiplex gateway fails closed instead of serving the owner profile
A /p/<profile>/ URL prefix on a gateway with multiplex_profiles off was
silently ignored: the request was handled as the gateway-owning profile,
so /p/lokaj/v1/toolsets reported the OWNER's platform_toolsets (and every
other profile-owned config read — skills, capabilities, model options,
agent-run toolset resolution — resolved from the owner too). That is the
exact repro in #91583 defect 2: enabling computer_use with
'hermes -p lokaj tools enable computer_use --platform api_server' showed
enabled in lokaj's config while /p/lokaj/v1/toolsets stayed false, and
enabling it on the owner profile flipped it true.

Per-profile capability isolation is the intended design (ruling on
a different profile's config. Multiplexed gateways were already correct —
the profile-prefix middleware enters _profile_runtime_scope and every
canonical config loader honors the HERMES_HOME override contextvar
(verified empirically for load_config, get_config_path and
_load_gateway_config) — the leak was only the non-multiplex fallthrough.

Fix at the one seam both adapters share: _resolve_request_profile now
rejects (404) a prefix naming any profile other than the one the gateway
actually serves. A self-referential prefix (/p/default/ on the default
gateway, /p/lokaj/ on a gateway launched for lokaj) still falls through
so existing well-formed clients keep working. Same change in the webhook
adapter, which had the identical fallthrough. New shared helper
hermes_cli.profiles.profile_matches_home does the home comparison,
fail-closed.

Tests: tests/gateway/test_multiplex_toolsets_profile_isolation.py —
E2E-style with two real profile homes + config.yamls under a temp
HERMES_HOME, real aiohttp routing through the profile-prefix middleware:
per-profile /p/<x>/v1/toolsets isolation for both owner and secondary
(the #91583 repro asserts computer_use true under /p/lokaj only),
cross-profile key rejection, and the fail-closed non-multiplex prefix
for both adapters. Sabotage-verified: reverting the adapter change fails
the 3 fail-closed tests.

Fixes #91583 (defect 2). Repro and live validation by @kubaboski.
2026-08-23 03:57:47 -07:00
John Paul Soliva 265bdcac82 fix(api-server): a /p/<profile> prefix on a non-multiplexed gateway fails closed instead of misdelivering
The prefix is an address: the caller is naming WHICH agent the request is
for. With gateway.multiplex_profiles off, _resolve_request_profile ignored
the prefix entirely — "don't 404 a would-be valid route" — so a request
explicitly addressed to one agent was silently answered by a different one.
Observed live (Aug 2026): `hermes peer dm mini/researcher` was answered by
the mini's DEFAULT agent with no error on either side, because that host
runs one LaunchDaemon per profile and only the default daemon hosted an
api_server. A wrong-agent answer is strictly worse than an error: the
sender believes the addressee got the message.

With multiplexing off the process serves exactly one profile, so the prefix
is honored when it names that profile (peers address single-profile daemons
this way without knowing the host's topology — get_active_profile_name() is
the same identity the file already uses for model resolution) and rejected
otherwise through the existing _PROFILE_REJECTED path (404). A process that
cannot resolve its own identity rejects too: if it cannot prove who it is,
it must not answer as anyone.

Unprefixed requests are untouched, and multiplexed hosts are untouched —
the change is confined to the prefix-present, multiplexing-off branch that
previously discarded the caller's addressing.
2026-08-23 03:57:47 -07:00
Teknium 231e613d3d fix(peer): resolve hidden canonical Bot Chats in hermes peer dm
Bot Mode always hides canonical 'Bot Chat' sessions, but _find_bot_chat's
GET /api/sessions listing used the default include_hidden=False path, so
the existing hidden row was invisible, _ensure_bot_chat tried to create a
duplicate, and the peer DB's UNIQUE(title) guard rejected it — DM failed.

- api_server: GET /api/sessions now accepts an exact-title lookup
  (?title=...) and honors include_hidden=1 ONLY alongside a title filter,
  so canonical hidden rows resolve without exposing a blanket hidden
  listing on the client surface. The title needle is pushed into SQL
  (search_query) so old hidden rows outside the recency window are found.
- peer dm client: _find_bot_chat sends title + include_hidden=1; older
  peers ignore the unknown params and degrade to today's behavior.
- Clear diagnosable error on the older-peer duplicate-create rejection,
  naming the hidden canonical chat and the PATCH hidden:false workaround.
- Unit tests (hidden resolution, no duplicate create, older-peer error,
  older-peer visible fallback) + real-gateway E2E over a real state.db.

Root-cause analysis and regression recipe by @kubaboski in #91583.

Fixes #91583
2026-08-23 03:56:28 -07:00
Teknium 7a54ab22e6 fix(gateway): control-socket hardening from #92447 post-merge review
- bind under umask 0o177 so the socket is never world-connectable, even
  pre-chmod (review pt 3)
- verb handlers run in an executor: state-file reads stay off the
  adapter event loop (pt 2)
- inventory dedupes one multiplex gateway answering identify for several
  homes — one runtime record per pid, with regression test (pt 1)
- v1 wire contract (one request per connection) documented in the module
  docstring (pt 5); /tmp-unwritable skip in the short-home test

Live-verified: perms 600 at bind, identify 4.3ms via executor path, 20
rapid queries healthy.
2026-08-22 18:05:26 -07:00
Teknium 530028c213 test(gateway): pipe E2E compares the server's self-reported pid, tree-kills the uv trampoline
First windows-latest run proved the design premise in miniature: identify
answered pid 616 while Popen.pid said 8000 — uv's Windows python.exe is a
trampoline that spawns the real interpreter as a child, so the spawner's
PID view is wrong and the process's self-declaration is right. Assert
against the child's printed os.getpid(); use taskkill /T for teardown.
2026-08-22 16:45:00 -07:00
Teknium 67aac4d863 fix(gateway): control-socket fallback survives deep TMPDIR; tests bind-location-aware
CI runners put pytest tmp roots past sun_path, which routed every test
home through the fallback: two tests assumed in-home binding. Tests now
assert against the resolved bind location, and the fallback itself
prefers /tmp when tempfile.gettempdir() is too deep to fit sun_path.
2026-08-22 16:45:00 -07:00
Teknium d1c84fc5d3 test(gateway): live Windows named-pipe E2E for the control socket (wine2e lane)
Real child process binding the real pipe via the proactor loop with the
default handlers; real sync client; real collect_fleet_versions consumer;
kill-and-fallback proof. Skipped everywhere except a real Windows host.
2026-08-22 16:45:00 -07:00
Teknium 60b6269142 feat(gateway): gateway-owned control socket — identify/status verbs, fleet consumers prefer it over scans (#92091 step 1)
The gateway now creates a local control socket at startup (Unix domain
socket at $HERMES_HOME/gateway.sock with a pointer-file fallback for
long paths; named pipe on Windows) and answers versioned JSON verbs:

- identify: pid, profile, hermes_home, code_sha/code_version (#91283
  stamps, now queryable live), self-declared supervisor kind, start_time
- status: the live runtime-status payload, answered by the process itself

Bound immediately after the PID-file O_EXCL claim (the moment the
process becomes the authoritative gateway for its HERMES_HOME), removed
on clean shutdown; a successor clears any stale socket on bind. Strictly
non-fatal: bind failure only means consumers use the old path.

Consumers migrated (observability only, scan layer demoted to fallback,
never deleted):
- collect_fleet_versions() (post-update fleet matrix): prefers a live
  identify answer over gateway_state.json; entries carry source=socket
- collect_runtime_inventory() (hermes update --plan): prefers the
  socket, and takes the gateway's own supervisor declaration instead of
  inferring it from PID scans

Old gateways mid-upgrade, crashed processes, and bind failures behave
exactly as before. Never a TCP port; filesystem/pipe ACLs are the auth
boundary (0600 socket).

Part of #91277 (fleet-update reliability). Design: #92091.
2026-08-22 16:45:00 -07:00
poisdahl a36d6704b6 Merge remote-tracking branch 'origin/main' into agent/81234-merge-20260821 2026-08-22 17:31:04 +02:00
poisdahl fd41164861 fix(history): keep carrier rewinds race-safe after refresh 2026-08-22 17:30:35 +02:00
Kshitij Kapoor 8ee0103ea2 fix(gateway): finite-bounded watchdog knob validation + wire keys through load_gateway_config
Addresses both review findings from @egilewski on #89134:

- Non-finite values: _coerce_int now degrades int(inf) (OverflowError
  previously ABORTED gateway config loading); the clamp requires
  math.isfinite plus sane upper bounds (interval <=3600s, timeout
  <=600s, strikes <=1000), falling back to the shutdown_watchdog
  constants.
- Loader wiring: load_gateway_config builds gw_data FLAT and never
  forwarded the yaml gateway: section, so loop_watchdog* keys —
  including the PRE-EXISTING loop_watchdog bool documented in
  config_defaults — were silently ignored on the real startup path.
  Bridged with the established top-level-wins/nested-fallback pattern.

E2E: config.yaml with loop_watchdog:false + strikes:12 + interval:.inf
now yields False/12/30.0 through the real loader.
2026-08-22 20:40:23 +05:30
Kshitij Kapoor 3616145723 fix(gateway): keep loop-watchdog default at 3 strikes; dedupe constants; register knobs in config defaults
Downscope of the salvaged #89134 per review: the 3->8 default raise was
symptom tolerance for the false-positive class the off-loop heartbeat +
two-witness probe fixes at the root — fleet-wide it would only delay
genuine-wedge recovery ~2.7x. The three tuning knobs keep independent
operator value and stay:

- default max_strikes back to 3 everywhere (constant, dataclass,
  from_dict fallback, floor clamp, tests)
- gateway/config.py + gateway/run.py now reference the
  shutdown_watchdog DEFAULT_* constants instead of duplicating literals
  in three places (drift hazard)
- knobs registered in hermes_cli/config_defaults.py alongside the
  sibling gateway.loop_watchdog bool
2026-08-22 20:40:23 +05:30
devops aa08cb8ccb fix(gateway): make loop-liveness watchdog tolerant of transient reconnect stalls
The event-loop liveness watchdog (gateway.shutdown_watchdog) hard-exited with
code 75 after 3 consecutive missed probes (probe_interval=30s, timeout=10s,
max_strikes=3), i.e. ~90-120s of loop block. Telegram/Discord reconnect during
a network blip does synchronous socket I/O on the loop and can block it for
60-90s; these stalls self-recover (recurring fleet incidents on 2026-08-17
stalled cron dispatch ~21h via restart churn, kanban t_0f76430f).

Raise the default max_strikes 3->8 so a transient reconnect stall is tolerated
while a genuine multi-minute wedge still escalates, and expose the three
tolerance knobs via config.yaml (gateway.loop_watchdog_probe_interval_s /
_probe_timeout_s / _max_strikes) so operators can tune per deployment.

Refs: kanban t_70483f23
2026-08-22 20:40:23 +05:30
poisdahl a5b326a471 Merge remote-tracking branch 'origin/main' into agent/81234-merge-20260821
# Conflicts:
#	tests/agent/test_reference_handoff_active_turn.py
2026-08-22 16:47:39 +02:00
Kshitij Kapoor 684e95a001 fix(gateway): claim ledger rows and clear resume_pending inline before the abandonable boot-send task
Post-merge follow-up to #92173. The claim + resume-clear lived inside
the boot-send task AFTER the restart notification — itself a
flood-controllable send. If that notification outlived the restore-gate
timeout, the gate opened with zero rows claimed and the resume
scheduler replayed turns whose answers were already in the ledger,
while the background task later redelivered them too (duplicate
delivery + re-paid turn).

Split _redeliver_pending_obligations into _claim_pending_obligations
(pure DB: sweep + resume clear, awaited inline before the send task
exists) and _redeliver_claimed_obligations (network half, stays inside
the bounded task). The original name remains as a composition wrapper.
Mutation-checked: both updated gate tests fail on pre-split run.py.
2026-08-22 16:26:51 +05:30
fangliquan 91fb175188 fix(gateway): preserve systemd handoff recovery 2026-08-22 15:27:26 +05:30