Ports the test coverage from PR #29496 onto the salvaged implementation:
adapter-level serialization/workspace isolation/disconnect sealing, and
gateway-level ID-correlated concurrent duplicate tools plus the editable
text fallback when the native stream fails.
Slack's Agents & AI Apps feature ships a native streaming surface that
renders a live-typing message instead of the edit-based progressive
updates the adapter used until now.
The adapter now implements the existing draft-streaming interface:
- supports_draft_streaming() opts in whenever the app is connected and
native streaming hasn't been detected as unavailable.
- send_draft() starts a stream on the first frame (chat.startStream,
anchored to the resolved thread_ts, with recipient_team_id/user_id
for channel streams) and appends only the delta on subsequent frames
(chat.appendStream is append-only). The consumer's trailing cursor
glyph is stripped before delta computation.
- Unlike Telegram drafts (ephemeral, replaced by a real sendMessage),
a Slack stream IS the final message. send() therefore intercepts the
turn-final delivery for a chat with an active stream whose streamed
text is a prefix of the final content, and seals it via
chat.stopStream with the remaining delta instead of posting a
duplicate. Rich Block Kit (when enabled) is applied to the sealed
message via chat_update, mirroring the finalize path in edit_message.
- Feature-gate errors from chat.startStream (not_allowed,
missing_scope, unknown_method, ...) are cached on the adapter so
subsequent runs skip straight to edit-based streaming with a single
warning naming the fix (enable Agents & AI Apps for the app);
transient errors only disable drafts for the current run via the
consumer's existing send_draft failure handling.
- Segment breaks (new draft_id) and disconnect() seal any open stream
so chats are never left with a dangling live-typing indicator.
No consumer or config changes: streaming.transport auto/draft now
lights up native streaming on Slack through the same interface
Telegram drafts use, and the edit-based path remains the fallback.
_resolve_thread_id() falls back to _last_inbound_thread[chat_id] when no
explicit thread is present. That fallback exists for interactive DMs, where
Google Chat spawns a fresh thread per top-level user message and the adapter
drops thread_id to keep the session key stable. It also fired for cron
deliveries, which carry job_id in their metadata but no thread: the output
landed as a reply inside the last inbound thread instead of starting a new
top-level message.
Bypass the _last_inbound_thread fallback when metadata has a job_id (i.e. the
message is an automated cron delivery), so cron output posts at top level
unless an explicit thread is requested.
The NS-570 epoch stamp clears a drain marker that survives a machine
restart — but it assumes every drain-gated action ends in a restart. When
a maintenance action completes WITHOUT recreating the container and the
writer never cancels the drain, the orphaned marker still carries the
current epoch, so the 1s drain watcher honours it forever and the gateway
bounces every inbound message with the 'draining for a maintenance
action' text (observed in the field: a Hermes Cloud instance refused all
Telegram turns for ~3 days).
The marker already records requested_at; now the readers check it. A
marker older than DRAIN_REQUEST_MAX_AGE_SECONDS (1h) reads as stale in
drain_requested() and drain_notification_suppressed(), with a loud
warning log. Leniency mirrors the epoch check: a missing or unparseable
timestamp still reads as drain-active (fail-safe toward quiescing), and
a legitimately long drain keeps a sanctioned keep-alive — re-calling
write_drain_request() refreshes requested_at.
Fixes#85433
Salvage of #37865 by @verybigdog. Adds delivery_mode (notify / notify+wake / wake)
on kanban notify subscriptions, persists chat_type + user_id_alt so a woken turn
reconstructs the creator's real session key, inherits the return path to child
tasks, and keeps wake out of the model-exposed send_message schema.
Original commits were authored under a local placeholder identity
(hermes-agent@users.noreply.local); re-attributed to the contributor's
public email.
Third lane of the same contract (found in live staging validation):
/sethome run as a top-level relay-fronted Slack DM message captured the
adapter's session-keying thread stamp (the /sethome message's own id)
into the persisted HomeChannel.thread_id and its legacy env mirror.
Every bare-platform delivery (deliver="slack") then resolved home chat +
home thread and landed inside the ephemeral thread around the old
/sethome message. Extracted _home_thread_from_source with the same
synthetic-stamp recognition as cron origin capture; a /sethome run
inside a genuine thread keeps that thread as the home target. Users
repair an already-poisoned home target by rerunning /sethome.
Review on #74658 flagged that the response-shape suite exercised identity,
ephemeral and upload paths but left two changed call sites untested:
- create_handoff_thread's seed-message ts (adapter.py:2262), which anchors
every subsequent handoff send onto the thread;
- the standalone media branch's chat_postMessage reads (adapter.py:8721 text
post, :8749 caption fallback), where an SDK-shaped reply used to drop the
ts and report a caption-only delivery as 'nothing deliverable'.
Both new cases run against the hand-rolled stand-in and the real
AsyncSlackResponse. Verified they fail against the pre-fix adapter.
Co-authored-by: Junie <junie@jetbrains.com>
Slack Web API calls return `SlackResponse`/`AsyncSlackResponse`, which are
mapping-like but not `dict` subclasses, so every `isinstance(resp, dict)`
gate took its "unexpected shape" branch at runtime: user and channel names
collapsed to raw IDs, every user resolved as a non-bot (defeating the
allow_bots loop guard), ephemeral replies were reported as failures, and
uploads/caption fallbacks lost their message_id.
Normalize responses through a single `_slack_response_payload()` helper
(dict passes through, SDK response yields `.data`, anything else yields
`{}` so callers keep their fallbacks) and use it at every call site.
Existing Slack tests injected plain dicts, which is why the defect was
invisible; the new tests run each behavioral case against a real
`AsyncSlackResponse` as well.
The session chat stream registered its wrapper task in _active_run_tasks,
but that turn is already counted by active_agent_work_count() via
_inflight_agent_runs (_run_agent) — the drain saw 2 for one turn
(test_session_chat_sse_turn_is_interrupted). Keep only the agent-ref
registration; run-scoped steer control doesn't need the task entry.
Adds POST /v1/runs/{run_id}/steer and bridges Browser-Extension/WebUI
session chat streams into the active run registry so live runs on those
surfaces are steerable too.
- steer accepted only while run status is exactly 'running'; stop/stopping/
terminal states return 409 run_not_accepting_steer even while cooperative
shutdown retains the agent reference
- session SSE disconnect/cancellation interrupts and drains the executor-
backed run instead of cancelling only the async wrapper; control refs stay
registered until the turn actually exits
- undelivered steer text (accepted after the final response) is preserved as
pending_steer on the terminal run.completed event/status so clients can
replay it as the next user turn instead of losing it
- docs for the endpoint, next-tool-boundary delivery, acceptance-vs-delivery
semantics
Salvaged from PR #54466 by @abundantbeing.
Webhook agent runs default to the constrained hermes-webhook toolset
(web/vision/clarify) because payloads can carry untrusted third-party
content. That default is right for public webhooks but wrong for trusted
local pushes (e.g. an OOM monitor daemon that needs the agent to run
ps/free/py-spy): the only workaround was widening platform_toolsets.webhook,
which elevates EVERY webhook route at once.
This adds a 'toolsets' key on individual webhook route configs (static
routes in config.yaml and dynamic subscriptions in
webhook_subscriptions.json) that replaces the platform-level resolution
for that route only:
- BasePlatformAdapter.toolsets_for_source(): per-source override hook,
default None (no behavior change for any other platform).
- WebhookAdapter.toolsets_for_source(): maps the session chat_id
(webhook:{route}:{delivery_id}) back to its route config and returns
the route's toolsets list.
- GatewayRunner._resolve_enabled_toolsets_for_source(): shared resolver
used by both agent-run call sites; validates the override through the
SAME _get_platform_tools path as platform config, so unknown names and
platform-restricted toolsets (e.g. discord_admin) are dropped rather
than trusted.
Deliberately NOT exposed via 'hermes webhook subscribe': granting elevated
tools is a manual config edit only, so an agent-created subscription
cannot self-grant terminal at runtime.
Cache-safe: the toolset list is resolved before agent construction and is
constant for a route, so the per-session agent signature and frozen system
prompt are unaffected mid-conversation.
Follow-up to the salvaged #80489 substring-fallback removal: the
structured 401/403 branch still exited with a bare return, leaving
_running True — dead listener, healthy-looking is_connected(), gateway
never told (the zombie half of the bug, OOF-156 class). It now sets a
non-retryable mattermost_auth_error with token guidance and notifies
the gateway fatal handler.
Also: pytest.importorskip for aiohttp in the verifier probe file
(module-level import crashed collection in envs without the optional
dep), and probe fixtures updated for the escalation attributes.
Independent-verifier boundary probes for commit fdd1a11ac5, covering
cases the implementer's regression tests did not exercise:
- WSServerHandshakeError(status=403) also stops the loop (only 401 tested)
- WSServerHandshakeError(status=500) does NOT stop the loop (structured
check must not over-match on type alone)
- transient error containing the word 'unauthorized' (not digit substring)
now retries correctly
- 5 consecutive transient errors all retry, not just the first
Verified these 2nd/4th tests fail against the pre-fix baseline commit
(01a1037d1e) and pass against the fix (fdd1a11ac5), confirming they
have real signal.
The WS reconnect loop had a fallback check that looked for "401", "403",
or "unauthorized" as substrings anywhere in an exception's string form.
A transient error whose message happens to contain those digits (a proxy
body, a stack trace, anything) got treated as a permanent auth failure
and stopped reconnection for good.
I removed the substring fallback and kept only the structured check:
aiohttp.WSServerHandshakeError with status in {401, 403}. That's the only
signal that reliably means the server rejected our credentials.
Added two regression tests: one proving a transient error containing
"401" in its text still retries, and one confirming the existing
_closing early-return path is untouched by the removal.
Simplify-code finding: test hardcoded exit code 75 in string
assertion while already importing GATEWAY_SERVICE_RESTART_EXIT_CODE.
Use f-string interpolation so the assertion tracks the constant.
Follow-ups to the salvaged #80032 fatal-error escalation, closing the
gaps its review thread identified plus a sibling of the same class:
1. Partial-batch loss: _check_inbox now dispatches whatever the fetch
returned BEFORE escalating a failure — the early-return dropped
already-fetched messages whose UIDs were marked seen.
2. Seen-after-fetch: UIDs enter _seen_uids only after their fetch
returns a response, so a mid-batch connection failure leaves the
remaining UIDs eligible for the next poll. Per-message processing
moved to _parse_fetched_message behind a poison guard: a message
that fails parsing/auth-verification is marked seen, logged with
its UID, and skipped once — never an eternal crash loop.
3. Reconnect mail loss: connect(is_reconnect=True) restores the
account's seen-UID baseline from a class-level snapshot instead of
re-marking the entire mailbox seen — mail that arrived during an
outage is now processed after the reconnect the escalation triggers.
7 new regression tests.
_fetch_new_messages() wrapped the whole IMAP connect/login/select/search/
fetch sequence in a bare except that logged and returned an empty list —
indistinguishable from a genuinely empty inbox. The adapter never invoked
its fatal-error handler, so the gateway's reconnect/backoff/status
machinery never learned the mailbox was unreachable; outages lasted until
a manual restart.
Track fetch failure on the adapter and, when the poll loop observes it,
set a retryable fatal error (email_imap_fetch_failed) and notify the
gateway handler so the platform enters the reconnect queue just like a
startup connection failure.
The cherry-picked #79448 predated #85049's _classify_connect_exception,
so it added a parallel PrivilegedIntentsRequired branch ahead of the
classifier (plus its own _is_privileged_intents_required detector).
Fold the tailored guidance into the classifier's existing intents arm
instead: one classification path, one error code (discord_intents_required),
and the message now names exactly the intents Hermes requested (Message
Content always; Server Members only when username/role allowlists need it).
Wizard callout, docs corrections, and tests from #79448 kept as-is.
PrivilegedIntentsRequired is a Developer Portal config error; surface which
intents Hermes requested as a non-retryable fatal and teach setup/docs.
Co-authored-by: Cursor <cursoragent@cursor.com>
Fleet triage after the 2026-08-11 storm resolution found agents whose sole
platform had been silently 'retrying' for weeks: revoked Telegram tokens,
Discord privileged-intent rejections, and Photon sidecars that can never
start were all funnelled into the indefinite reconnect queue with no owner
signal (OOF-151/152/153, epic OOF-156).
Two-part fix:
1. Per-adapter classification — by exception TYPE only, never message text:
- telegram: InvalidToken/Forbidden -> telegram_auth_error, retryable=False
(new _looks_like_auth_error, mirrors _looks_like_network_error)
- discord: LoginFailure -> discord_auth_error, PrivilegedIntentsRequired
-> discord_intents_required (both retryable=False); every other path now
sets an explicit code (previously the generic branch set NO fatal info,
which the gateway read as 'probably transient')
- photon: new typed PhotonSidecarStartupError; deps-install failure ->
SIDECAR_DEPS_MISSING and missing node binary -> SIDECAR_NODE_MISSING
(retryable=False); ambiguous startup crashes stay retryable
- email: IMAP/SMTP failures now always set a fatal code;
SMTPAuthenticationError -> email_auth_error, retryable=False (IMAP4.error
is type-ambiguous between bad creds and transient NOs, so IMAP stays
retryable)
2. Gateway escalation — platforms continuously in the reconnect queue past
HERMES_RECONNECT_ATTENTION_AFTER_SECONDS (default 2h, 0 disables) get
needs_attention=true + retrying_since stamped into runtime status, once
per episode, cleared on successful reconnect.
Deliberately NOT a circuit breaker: retries never stop. The auto-pause
mechanism was removed for good reason (transient DNS outages left bots
silently dead); this preserves that and only adds visibility. No new
platform_state enum values — NAS's status schema is strict — only additive
fields.
Unknown exception types always stay retryable: a false terminal recreates
the silently-dead-bot problem, and the escalation path covers
misclassified permanent failures.
Extend the normalized-envelope pipeline shipped in #82063 with new event
types, each with its own versioned, event-local payload contract:
- Telegram: message_edited (edited_message updates; editor-identity auth
extraction, forum topic thread_id, bounded text/caption, ISO edited_at)
- Discord: message_edited, message_deleted, thread_created, thread_renamed
(on_message_edit/delete, on_thread_create/update fire-sites with has_hook
no-subscriber fast-paths, bot-authored events dropped, rename-only
filtering on thread updates)
All events flow through the same gateway-owned post-auth boundary; malformed
or unauthorized events drop, fail closed. Raw SDK payload access is
deliberately NOT shipped (round-2 correction: needs its own
gateway.raw_events capability and design).
The Discord fire-site machinery (no-subscriber fast-path, observer isolation,
connect-time wiring) adapts the observer-hook design from PR #62584
(@paoloantinori) onto the normalized-envelope contract; PR #36875's raw
telegram update hook is superseded by the same correction.
Docs: hooks.md gains per-event payload contract tables.
Co-authored-by: Paolo Antinori <pantinor@redhat.com>
The pre_transcription hook threads (path, language, source) through
gateway voice transcription; three sibling telegram-voice tests pinned
the old single-arg call shape.
Main now fail-closes profile routes targeting unserved profiles
(_profile_name_for_source checks _multiplex_profile_homes). The
provenance test routes to profile 'work', which no longer stamps in a
bare test env; mock the served-profile set so the route resolves.
Builds on Paolo Antinori's #68431 salvage for #64176. Move plugin dispatch behind the profile-scoped runner authorization boundary, fail closed on malformed reaction identities, preserve observer registration across Telegram app rebuilds, and document the deliberately observer-only contract.
Co-authored-by: Paolo Antinori <pantinor@redhat.com>
- _on_platform_update: log the normalize and auth errors with exc_info=True so
a regression that silently drops reactions leaves a traceback, not just a
one-line message (matches the intake auth fallback's exc_info usage).
- TestRegisterHandlers: also assert five core handlers land in the default
group and only the observer is in group 99.
DoD: hook + auth tests green (34 passed). For a log-line + assertion change
the substantive gate is the test run; /simplify and /code-review were applied
proportionately.
Response to teknium1's hermes-sweeper review (keep_open, salvageability=medium).
1. Post-auth gate. The group-99 catch-all fired gateway_platform_event before
the authorization boundary. Extract _is_source_authorized(source) from
_is_user_authorized_from_message and add _source_from_reaction_for_auth;
reactions whose actor the intake would reject no longer reach plugins.
Fails closed if source extraction raises, so a future non-reaction event
type cannot silently bypass auth before its own extraction is wired.
2. Shared registration. Extract _register_handlers(app) from connect() so the
gateway_platform_event observer (group 99) is re-registered alongside the
core handlers on any rebuild path.
3. Trim inert hook surface. Drop the three reserved gateway_* names from
VALID_HOOKS (keep only gateway_platform_event). The others land with their
real contracts and fire-sites when #64231 is finalized.
Tests: unauthorized/authorized/open reaction gating, fail-closed for a future
non-reaction event type, and _register_handlers re-registration.
Ran /simplify and /code-review (high) before pushing.
First slice of #64176's observer-hook half — a normalized-envelope inbound
event hook, replacing raw-SDK handler args with a stable contract (per #64176's
"normalized versioned envelopes only; raw SDK gated behind a capability" rule).
- VALID_HOOKS: register the four gateway_* names from #64176
(gateway_platform_event fires today; gateway_session_titled /
gateway_message_delivered / gateway_thread_created reserved pending #64176's
fire-sites).
- BasePlatformAdapter._fire_gateway_hook: reusable, has_hook-guarded,
per-call-isolated fire helper (the no-subscriber common case short-circuits).
- TelegramAdapter: a group-99 catch-all TypeHandler normalizes inbound updates
into gateway_platform_event envelopes. message_reaction -> {platform,
event_type:"reaction", payload{emojis, custom_emoji_ids, chat_id, message_id,
thread_id}} (custom-emoji reactions captured via custom_emoji_id; standard via
.emoji — no None in consumer-facing lists). Other update types return None
pending #64176's taxonomy (#64231). Normalization is wrapped so a malformed
update can't raise into PTB dispatch.
Observer-only — zero behavioral change to core dispatch. Supersedes the raw
inbound half of #62584 (telegram:update -> normalized gateway_platform_event).
Tests: VALID_HOOKS registration; _fire_gateway_hook routing/has_hook/isolation;
_normalize for standard, custom-emoji, and mixed reactions + non-reaction;
_on_platform_update firing + normalize-error isolation.
Ran code-review (high) + simplify before pushing.
* fix(gateway): pass live adapters to cron fire webhook's fire_due
The Chronos fire webhook (/api/cron/fire) called
provider.fire_due(job_id, adapters=None, loop=loop), so every
externally-triggered fire delivered through the standalone path even
with a live gateway in-process. E2EE platforms and relay-fronted
logical platforms (whose ONLY send path is the live relay adapter — no
native credential exists on the box) failed every external fire with
"platform 'X' not configured/enabled", while the same job delivered
fine under the built-in ticker (gateway/run.py passes runner.adapters).
Resolve the runner (self.gateway_runner → app['gateway_runner'] →
_gateway_runner_ref(), the same chain the drain check uses) and forward
its adapters. No runner → adapters=None, preserving the historical
standalone path byte-identically.
Note: does not by itself fix Fly-hosted scale-to-zero deployments where
NAS's callback lands on the DASHBOARD process (internal_port 9119) —
_fire_cron_job_for_profile there has no gateway runner in-process. That
topology needs a separate fire handoff (design pending).
* fix(cron): dashboard forwards Chronos fires to the gateway (503 when unreachable)
The dashboard's /api/cron/fire executed cron jobs in the DASHBOARD
process via _fire_cron_job_for_profile with adapters=None. On hosted
deployments (Fly proxy exposes only the dashboard's port) that made
every managed-cron fire deliver through the standalone send path, which
cannot serve relay-fronted logical platforms (their only sender is the
live relay adapter in the gateway process — no native credential exists
on the box) or E2EE rooms. It also ran the whole agent turn inside the
dashboard: wrong process for memory/session ownership and fire-claim
attribution.
Restore the invariant that the GATEWAY owns cron execution:
- Dashboard route: after verifying the NAS JWT and resolving the job's
profile, FORWARD the fire to the gateway api_server's own
/api/cron/fire on loopback, NAS bearer preserved (the gateway
re-verifies the JWT — defense in depth, no new trust link), and pass
the gateway's response through. Gateway unreachable → 503 so NAS
retries per the Chronos contract (non-2xx = retryable; the store CAS
de-dupes the eventual double fire). Deliberately NO local-execution
fallback.
- Endpoint resolution mirrors gateway/config.py's api_server load order
per target profile (config.yaml extra.port → API_SERVER_PORT from
process env or the profile's .env → 8642), with /p/<profile>/ prefix
routing under multiplex.
- docker/stage2-hook.sh: generate a strong API_SERVER_KEY into .env on
first boot when absent (never overwrites an operator value), so the
loopback api_server passes its startup guard on hosted images. The
fire route itself is NAS-JWT-authed; the key gates the rest of the
api_server surface. The listener binds 127.0.0.1 by default and the
Fly service exposes only the dashboard port.
- _fire_cron_job_for_profile kept but deprecated (late-binding seam
compatibility); no route calls it.
- docs/chronos-managed-cron-contract.md: document the two-hop inbound
topology and the 503-retry semantics.
Depends on the previous commit (fire webhook passes live adapters to
fire_due) — together they make NAS→dashboard→gateway fires deliver over
relay end to end.
* fix(cron): read the profile api_server port via the canonical config loader
CI guard test_config_read_guard flagged the new _gateway_fire_endpoint
for a raw yaml.safe_load of the profile's config.yaml — the exact drift
class the guard exists to kill (raw reads miss the managed-scope
overlay, ${ENV_VAR} expansion, and root-model normalization).
Read through load_config() under a HERMES_HOME override scoped to the
target profile instead (the same pattern the deprecated
_fire_cron_job_for_profile uses for its store scope), and pull the port
with cfg_get. Test updated to stub load_config rather than write a raw
config.yaml.
* fix(gateway): only messaging platforms count for the scale-to-zero arm gate
The stage2 hook now generates API_SERVER_KEY for every Docker container,
and key presence force-enables the api_server platform. The scale-to-zero
arm gate counted every enabled platform, so the loopback api_server
listener made messaging_is_relay_only_or_absent False on every hosted
instance — silently disarming the feature (the not-armed log would show
enabled platforms=['relay','api_server']).
The arm gate and the not-armed logger now share one helper that filters
to enabled MESSAGING platforms, excluding LOCAL/API_SERVER/WEBHOOK —
the same non-messaging exclusion set _connect_platforms already uses.
A genuinely enabled direct-socket platform (Discord/Telegram) still
disarms. Two of the three new tests fail without this fix.
The relay interactions passthrough lane (_discord_interaction_to_event)
built its SessionSource with platform=Platform.RELAY and no
delivered_via_upstream_relay marker — unlike the relay text lane
(ws_transport._event_from_wire), which maps the connector's platform to
the logical enum and stamps the authenticated-upstream flag.
Consequences of the mismatch:
- /sethome sent as a Discord slash command persisted the home channel
under platforms.relay.home_channel (invisible to cron delivery, which
looks up the logical platform) and mirrored it into the dead
RELAY_HOME_CHANNEL env var — so cron jobs with deliver='discord' kept
falling back to local-only even after the resolution/delivery fixes.
The absent trust marker also meant via_relay=False, so the handler's
'Relay does not authenticate this logical home target' guard —
designed to reject exactly this misfiled shape — never engaged.
- Session keys forked: the connector binds the interaction's follow-up
capability under buildSessionKey with platform 'discord' and
chat_type 'group' (interactionSessionSource), while the gateway keyed
the same interaction as relay/channel.
- _capture_scope skipped recording _platform_by_chat (it ignores the
generic 'relay'), losing the egress sender hint for the chat.
Stamp Platform.DISCORD (the lane statically parses Discord interaction
wire payloads), chat_type 'group' for guild channels (native-adapter and
connector parity), and delivered_via_upstream_relay=True (parity with
the text lane; set locally, never read off the wire).
With this, slash-command /sethome files under platforms.discord and
passes the via_relay guard legitimately, and cron delivery over relay
works end to end with the #84300 resolution fixes.
_scale_to_zero_has_live_background_work() counted every task in
_background_tasks — but _spawn_supervised parks all permanent watchers
there (session-expiry, kanban, reconnect, the scale-to-zero watcher
itself, ...). An armed gateway therefore considered itself busy forever
and never went dormant or suspended. Verified live on staging
(hermes-agent-stg-test-6698, 2026-08-12): armed at 05:25, fully idle for
25+ minutes, zero 'going dormant' lines. Fly's coarse proxy autostop used
to mask the bug; once the gateway took ownership of the suspend (#84295)
it became load-bearing.
_spawn_supervised now tags its tasks and the busy check skips them.
Transient tasks (startup-resume events, delegation, tracked processes)
still block suspend. New tests exercise the REAL _spawn_supervised path
rather than a stubbed _background_tasks set — the stubbing is exactly why
the earlier tests missed this (same call-site trap as the F25 arm bug);
the key test fails on main and passes with the fix.
Fly Proxy autostop judges idle exclusively on inbound proxied connections.
It cannot see an in-flight agent turn (outbound-only LLM traffic), and since
Fly's mid-2026 proxy change an open outbound socket (the relay WS) no longer
holds a machine awake. With autostop:"suspend", Fly suspended machines while
they were still processing long-running jobs, and could suspend before the
gateway flipped the relay destination (the buffered-event black hole).
The scale-to-zero watcher now owns the suspend: after the idle predicate
holds (no running agents, no live background work, inbound-quiet) and the
go_dormant() quiesce completes (relay drained + flipped), it POSTs
/v1/apps/{app}/machines/{id}/suspend on the local /.fly/api flaps socket.
Suspend is skipped when the quiesce fails or inbound lands mid-quiesce
(flip-before-freeze), and off-Fly the step is a no-op (fail-awake).
Pairs with the NAS change that provisions scale-to-zero machines with
autostop:"off" (gateway-owned suspend); wake is unchanged (Fly-proxied
wakeUrl poke + autostart).
* feat(relay): ambient token endpoint mode for gateway.idp.token_url
When gateway.idp.token_url is configured WITHOUT client_id/client_secret,
treat the URL as a metadata-server-style ambient credential endpoint:
plain GET, response body is the token (raw JWT or {"access_token": ...}
JSON envelope). Covers workload-identity proxies such as Domino's
$DOMINO_API_PROXY/access-token, which mint short-lived user-scoped OIDC
tokens with no client registration.
Previously this configuration was a hard error (client_id/client_secret
missing), so no working deployment changes behaviour: creds present keeps
the OAuth2 client_credentials POST, no token_url keeps Nous Portal. The
misconfig error now self-diagnoses (names the ambient fallback and how to
select the client_credentials grant instead).
* fix(relay): reject short plain-text bodies in ambient token shape gate
Review finding: the shape gate accepted any base64url-alphabet word, so an
IdP answering the ambient GET with a terse error body ('unauthorized',
'error', 'null') had that word returned as a bearer token instead of the
fail-closed misconfiguration error. Tighten the gate to JWT-like dotted
tokens (3+ segments) or long opaque tokens (>= 32 chars); short bare words
now raise the self-diagnosing ambient error.
* fix(relay): partial IdP client credentials keep the loud error, never select ambient GET
The ambient-endpoint dispatch used 'not client_id or not client_secret',
so configuring exactly one credential (a mistyped client_credentials
setup) silently issued a GET at the IdP token endpoint and then raised
'no client_id/client_secret configured' — factually wrong for that
operator, and a stray request the old hard error never made.
Ambient mode now requires NEITHER credential; a partial pair raises
immediately, names the missing key, and issues no HTTP request (tests
assert urlopen is never called). Docstring and relay.md now say
'neither' instead of 'without'.
* fix(relay): ambient JSON envelope requires a string access_token, no coercion
Review finding (P2): the JSON-envelope branch accepted any truthy
access_token via str() coercion — a number became '12345…', a boolean
became 'True', an object became its Python repr — bypassing the fail-
closed contract and deferring the failure to the connector, where it
hides the real endpoint problem.
The envelope value must now be a non-empty string, the same contract the
client_credentials path enforces on its token response. Deliberately NO
shape gate on envelope values: an envelope is an intentional token
response (mode-1 symmetry), and opaque tokens may use the standard-base64
alphabet the raw-body gate rejects. Mutation check: reverting the branch
to str() coercion sends the 3 coercion tests red (3 failed, 15 passed).
---------
Co-authored-by: Ben Barclay <ben@nousresearch.com>
* fix(relay): stop sibling gateways answering another instance's button press
A Discord button press arrives on the passthrough plane, and the connector
fans a passthrough forward out to EVERY live gateway session of the tenant
(relayServer.routeBusMessage delivers `passthrough` via sessionsByTenant),
unlike a message, which it narrows to the admitted instance set. The prompt
went out from exactly one instance and _pending_prompts is process-local, so
every sibling gateway saw an answer for a prompt it never minted, could not
tell that from its own prompt expiring, and fell through to chat dispatch --
where the option-shaped text ("/c1") is not a real command and run.py replied
"Unknown command `/c1`". One copy per sibling, under the single real ack.
Prompt ids are now minted as `<per-process nonce>.<8 hex>`, so an answer can
be attributed to the process that minted it. A prompt answer is always
consumed, never re-dispatched as chat: a sibling's prompt and a repeat answer
are both dropped silently, and an expired prompt of our own gets a short
"no longer waiting" notice from the owning gateway only.
Ids stay inside the connector codec's contract ([A-Za-z0-9_.-], <=32 chars,
64-byte callback budget -- verified against promptCodec.ts: 52 bytes worst
case with a full-length option id). An id with no nonce segment (a prompt in
flight across an in-place upgrade) is still treated as ours.
Tests: 4 added, each verified to fail without the fix. Full relay suite green
(160 tests).
* style(tests): ruff-format the added relay prompt tests
Three separate reds on main. Two are fixed here; the third needs no code.
1. tests/gateway/test_multiplex_busy_input_mode.py (blocks every merge)
Fails "Python tests / Run tests slice 5/12" and therefore "All required
checks pass". Semantic merge conflict between two PRs merged ~1h apart:
a31be480 fix(gateway): respect routed profile busy modes (added the test)
c8f235a1 feat(gateway): allow selective multiplex profile serving (added the gate)
c8f235a1 taught _profile_name_for_source to reject a route whose target
profile is not in the served set (profiles_to_serve). Each PR was green on
its own base; neither ran against the other's merge result.
The test asserts a route to profile "research" resolves to that profile's
busy mode, but never patches profiles_to_serve — so it reads the runner's
REAL on-disk profiles. "research" is not among them, the route is rejected
before the busy-mode snapshot is consulted, and the assertion gets the
gateway default:
WARNING gateway.run: Rejecting profile route 'research-chat':
target profile 'research' is not served
AssertionError: assert 'interrupt' == 'steer'
Patch profiles_to_serve for the assertion — the same seam every sibling
test in tests/gateway/test_profile_resolution.py already patches
(test_route_inside_allowlist_resolves, test_route_outside_allowlist_rejects).
This also removes an ambient-state dependency: the test previously passed
or failed based on which profiles happened to exist on the machine running
it. Verified passing under an empty HERMES_HOME.
Test-only. The serving gate from c8f235a1 is correct and left intact.
2. Skills-index workflows: local action used without actions/checkout
check-freshness has failed on all 12 of its last 12 scheduled runs:
##[error]Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under
'.../.github/actions/get-app-token'. Did you forget to run
actions/checkout before running your local action?
./.github/actions/get-app-token is a LOCAL composite action and cannot
resolve without the repo on disk. skills-index-freshness.yml had no
checkout step at all. The step is gated on `status != 'ok'`, so the
watchdog broke exactly when it was supposed to file its issue — the live
index is currently 521.4h stale (limit 26h) and nobody was told.
An audit of all workflows for this bug class found one more instance:
skills-index.yml's `trigger-deploy` job, which re-triggers the docs deploy
so a refreshed index reaches the live site. Its sibling `build-index` job
checks out; this one did not. That is plausibly why the index went stale
in the first place. Both are fixed; the audit now reports zero remaining
jobs that use a local action without a prior checkout.
Pinned to the same actions/checkout SHA used by the other 35 call sites.
3. "Publish inline E2E evidence" — no fix needed
Failed once at 13:33Z on a transient TLS error reaching api.github.com
("certificate is not valid for any names") while installing a gh
extension. The last 25 runs of that workflow are 25/25 success. Infra
blip, not a code defect.
* fix(gateway): stop frozen-preview finals and dropped idle-session delegation callbacks
Two relay-plane delivery losses from the 2026-08-09 staging incident:
1. stream_consumer: the skip-redundant-finalize branch recorded _accumulated
as the delivered turn-final payload even when the last ACKED edit was an
earlier throttled preview snapshot, so delivered_final_matches reconciled
True and the gateway suppressed the corrective final send — the user was
left with a cut-off message ending in the streaming cursor. Extracted
_mark_skip_redundant_finalize(): records the last acked wire payload
(cursor-stripped), so a preview/final mismatch now returns False and the
normal final send fires.
2. run.py: _classify_completion_target classified every ended parent session
terminal unless it ended by compression. Idle/timeout session ends are the
norm on scale-to-zero relay deployments and the chat route remains valid;
completed async delegation results were terminally dropped. Ended parents
now classify deliver unless the end was an explicit user boundary
(session_reset / user_exit / session_switch).
* fix(relay): drain in-flight outbound frames before transport teardown
disconnect() failed every pending outbound future immediately with
'relay transport closed', so a trailing finalize edit racing turn
teardown was lost even though the connector socket could still serve
it. Bounded drain grace (5s) lets in-flight requests resolve; silent
connectors still tear down promptly. asyncio.wait (not gather+wait_for)
so a timeout doesn't cancel futures owned by the fail-remaining loop.
* fix(gateway): route completion injection through the alias-aware transport resolver
Third relay-plane delivery loss from the 2026-08-09 staging incidents: a
delegation batch completed while the gateway was up, the watcher drained
the event, and delivery vanished with no log line. _inject_watch_notification
resolved its adapter with a literal p.value == platform_name scan of
self.adapters — a relay-fronted gateway registers ONE adapter under
Platform.RELAY fronting N logical platforms, so 'slack' never matched and
the injection returned None ('no gateway route'), silently dropping the
completion. The handoff path already documents this exact trap and uses
resolve_delivery_transport; the injection path now does the same (native
wins; relay eligible only when it fronts the logical platform), with the
literal scan kept as fallback for stub runners and exotic platforms.
* fix(relay): clamp disconnect drain grace to the runner's adapter-disconnect budget
Review finding (JoaoMarcos44, #82592): a fixed 5.0s drain in front of the
three 1.0s sequential teardown awaits gives an 8.0s worst case inside the
runner's 5.0s asyncio.wait_for(adapter.disconnect()) — tripping it cancels
teardown mid-drain, skips the fail-pending loop, and leaves outbound
callers blocked until _OUTBOUND_TIMEOUT_S (30s). The effective grace is
now budget - 3*TEARDOWN - margin (env-aware via the same
HERMES_GATEWAY_ADAPTER_DISCONNECT_TIMEOUT the runner reads), so the drain
can never push teardown past its caller's budget; a budget too small for
any drain disables it cleanly.
* test(gateway): pin the final-send suppression contract across a behaviour matrix
The gateway skips its own final send when the stream consumer claims the turn
final already reached the user. Every incident in that family — #71643 (stale
finalize snapshot), #78541 (payload-less multi-message split), #82656 (frozen
preview left with a visible cursor) — is the same failure: the consumer claimed
delivery for text the platform never rendered, so the corrective send was
suppressed and the answer was lost with no retry.
Each was fixed with a scenario test pinned to one branch of
GatewayStreamConsumer.run(). The got_done handler now has five sibling branches
that each set the suppression flags and record a turn-final payload, and nothing
checks them as a group: a new branch, or a new early `return True` in
_send_or_edit, can reintroduce the class without failing a test.
Pin the invariant instead of the branch — if the consumer offers the gateway any
signal it would trust, the complete final text must have reached the wire — and
assert it across {edit always / dies / never / lies} x {send always / never} x
{fresh-final on / off} x {clean / interrupted stream}.
The adapter records only frames that actually rendered, so an ACK the platform
drops does not count as delivery. 24 honest-transport scenarios hold the
invariant as a hard assertion. The 16 lying-transport scenarios are checked too;
the single combination that still violates it is reported as an expected
failure documenting the open exposure rather than asserting it away.
Refs #82656
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(gateway,relay): prime relay egress routing for synthetic injections + cap stale completion replay
Defect #4 from the 2026-08-09 staging incidents (upgrade-robustness):
after every gateway restart the durable async-delegation replay injected
completions correctly (post-741663cf1) but their replies bounced at the
connector — 'slack egress declined: target not routed to an onboarded
tenant'. The relay adapter re-attaches tenant discriminators
(metadata.scope_id / metadata.user_id) from per-chat caches warmed ONLY by
inbound traffic; synthetic turns race those cold caches on every deploy,
scale-to-zero wake, and crash recovery.
- relay adapter: prime_routing_cache() — feeds a synthetic event's
session-store origin through the same _capture_scope used for real
inbound (never raises).
- run.py injection path: prime the resolved adapter before handle_message
(duck-typed; native adapters unaffected).
- async_delegation: 48h staleness cap in restore_undelivered_completions —
a pending completion older than the cap is terminally dropped (payload
stays queryable) instead of re-run as a fresh full-context turn; the
post-restart replay of a July session burned a 102K-token context.
Also carried: JoaoMarcos44's suppression behaviour-matrix harness
(cherry-picked from #82676, authorship preserved) — 39 passed + 1 xfail
(the documented ACK-then-drop transport-honesty residue).
* test: use recent timestamps in restored-ownership fixtures
test_restore_stamps_restored_flag persisted its completion with epoch-era
toy timestamps (dispatched_at=1.0), which the new 48h replay staleness cap
correctly classifies as stale — the fixture then exercised the cap instead
of the restored-flag contract (CI slice 4 failure). Timestamps are now
now-relative; the staleness behavior itself is pinned separately in
test_relay_injection_egress_priming.py.
* fix(gateway,relay): close four review findings on the relay delivery fixes
Review follow-ups on this branch (NousResearch#82592):
1. HIGH — classifier/resolver mismatch (falsely-acknowledged loss).
_classify_completion_target now returns "deliver" for idle-ended
parents, but _resolve_async_delegation_session still dropped every
non-compression-ended pin: the durable row was acked at adapter
acceptance, then the injection died inside the pipeline with no
retry — strictly worse than the honest terminal drop on main, and
the delivery leg defect #2's fix depends on did not exist. The
resolver now retargets non-user-boundary ends (idle/timeout/
lifecycle) to the chat's current session — session_entry already IS
the routing key's current session for the same chat — while user
boundaries (session_reset / new_session / user_exit /
session_switch) stay fail-closed. Both sides share one module-level
_USER_BOUNDARY_END_REASONS so the verdict and the routing decision
cannot drift again; a coherence test asserts deliver-verdicts
resolve non-None across representative end reasons.
2. HIGH — drain clamp missed adapter-level spend. The effective drain
grace budgeted drain + 3x teardown, but RelayAdapter.disconnect
spends revocation-monitor teardown + go_idle time BEFORE the
transport drain inside the same runner wait_for; worst case still
blew the budget and cancelled teardown mid-drain (skipping the
fail-pending loop). The adapter now measures its own elapsed time
and threads the REMAINING budget into
transport.disconnect(budget_s=...); legacy/stub transports without
the keyword fall back to the no-arg signature.
3. P1 — _request_response racing disconnect() could register a future
after the fail-pending loop already ran, stranding the caller for
the full _OUTBOUND_TIMEOUT_S (30s). Fail fast with the same
"relay transport closed" error once _closing is set.
4. P1 — _build_process_event_source's last-resort reconstruction
dropped scope_id, so a scoped relay completion whose session-store
origin was unavailable primed no tenant discriminator and could
still bounce off the connector's fail-closed egress guard.
scope_id now threads through the reconstructed SessionSource, with
a warning when a scoped chat reconstructs without one.
All four: RED reproduced with the fix reverted, GREEN after; relay/
delegation delivery families pass (43 + 71 + 179 across the touched
suites); full tests/gateway run shows only failures already failing
identically on merge base 2446c8bb6 (env/dep issues).
* fix(gateway,relay): make pending-frame failure cancellation-safe; persist completion routing origin
Two remaining review findings on this branch (NousResearch#82592):
1. Cancellation could strand outbound waiters past the fail-pending
loop. transport.disconnect() failed pending futures only at the END
of the drain + three teardown awaits; a cancellation landing
mid-drain (the runner's wait_for budget, an outer cleanup deadline)
skipped the loop entirely and left registered futures unresolved —
their callers blocked until _OUTBOUND_TIMEOUT_S (30s). The budget
threading added earlier shrinks the window but is not a hard
guarantee. The fail-pending loop (and the going_idle ack failure)
now run in a `finally`, so no exit path — normal, error, or
cancelled — can leave a registered future unresolved. Idempotent:
done futures are skipped, a second disconnect() pass is a no-op.
2. Durable completions did not persist their routing origin, so the
scope_id threading in the fallback SessionSource reconstruction had
nothing to carry on the exact path it exists for (restart replay
with session store + source cache gone): the async-delegation event
producers never populated scope_id and the durable rows never
stored it. Dispatch now snapshots the originating turn's
scope_id/user_id/user_name from the session context
(_capture_routing_origin — a new HERMES_SESSION_SCOPE_ID contextvar
bound by the gateway at session-bind time alongside the existing
vars), stores them in the existing task_json payload (no schema
migration), and re-attaches them to all three completion-event
shapes (live single, live batch, crash-recovery rebuild). The
gateway's fallback reconstruction then primes both discriminators
after a restart.
Tests: cancellation mid-drain -> every pending future resolves with
"relay transport closed" (mutation: moving the loop out of the finally
goes RED); second-pass disconnect idempotence; end-to-end
dispatch -> owner-death recovery -> event carries scope_id -> fallback
SessionSource primes it (mutations: dropping the dispatch capture or
the task_json persistence both go RED); live completion event carries
the origin. 94 passed + 1 xfailed across the delivery/delegation
suites; tests/tools delegation family 73 passed (2 collection errors
pre-existing on merge base 2446c8bb6).
---------
Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Ben Barclay <ben@nousresearch.com>
Retryable primary errors can leave pooled sockets in CLOSE_WAIT while fallback retries continue. Replace and close failed primary generation before fallback selection.\n\nRefs #82920
many tests patched sys.platform or a module's _IS_WINDOWS flag, then
ran on linux ci. the patch selects the branch under test, but the host
does not have the behavior the branch exists for. the test proves the
patch, not the platform. some gated assertions never ran on any host.
this commit adds three markers: linux_only, macos_only, windows_only.
a conftest hook skips a marked test on the other hosts, with a clear
reason. no test fakes a host now. two documented fakes remain
(android/termux, freebsd) because no ci runner exists for them.
each fake site got one of four treatments:
- gate it: the real host supplies the platform; mocks cover real
dependencies only, never host identity
- patch the module's own probe when the subject is the probe's consumer
- assert against the real host when the fake stood in for any non-x host
- delete the patch when it set the value the host already has
bare skipif(sys.platform != ...) guards became markers too. the lane
model skips these on linux and never imports them on windows, so they
ran on no host. platform parametrize tables are now one marked test
per os.
running on real hosts found real errors: a chrome-sandbox failure in
test_gui_command that main hides, and two windows failures fixed here.
the agents.md testing section now documents the policy.
Simon van Laak