a189286aea8abbf33dc93e2871cf67d8d87b0d2b
50 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ben Barclay
|
a189286aea |
fix(relay): stop sibling gateways answering another instance's button press (#83677)
* fix(relay): stop sibling gateways answering another instance's button press
A Discord button press arrives on the passthrough plane, and the connector
fans a passthrough forward out to EVERY live gateway session of the tenant
(relayServer.routeBusMessage delivers `passthrough` via sessionsByTenant),
unlike a message, which it narrows to the admitted instance set. The prompt
went out from exactly one instance and _pending_prompts is process-local, so
every sibling gateway saw an answer for a prompt it never minted, could not
tell that from its own prompt expiring, and fell through to chat dispatch --
where the option-shaped text ("/c1") is not a real command and run.py replied
"Unknown command `/c1`". One copy per sibling, under the single real ack.
Prompt ids are now minted as `<per-process nonce>.<8 hex>`, so an answer can
be attributed to the process that minted it. A prompt answer is always
consumed, never re-dispatched as chat: a sibling's prompt and a repeat answer
are both dropped silently, and an expired prompt of our own gets a short
"no longer waiting" notice from the owning gateway only.
Ids stay inside the connector codec's contract ([A-Za-z0-9_.-], <=32 chars,
64-byte callback budget -- verified against promptCodec.ts: 52 bytes worst
case with a full-length option id). An id with no nonce segment (a prompt in
flight across an in-place upgrade) is still treated as ours.
Tests: 4 added, each verified to fail without the fix. Full relay suite green
(160 tests).
* style(tests): ruff-format the added relay prompt tests
|
||
|
victor-kyriazakos
|
6a7cf19302 |
fix(gateway,relay): stop frozen-preview finals and dropped idle-session delegation callbacks (#82592)
* fix(gateway): stop frozen-preview finals and dropped idle-session delegation callbacks
Two relay-plane delivery losses from the 2026-08-09 staging incident:
1. stream_consumer: the skip-redundant-finalize branch recorded _accumulated
as the delivered turn-final payload even when the last ACKED edit was an
earlier throttled preview snapshot, so delivered_final_matches reconciled
True and the gateway suppressed the corrective final send — the user was
left with a cut-off message ending in the streaming cursor. Extracted
_mark_skip_redundant_finalize(): records the last acked wire payload
(cursor-stripped), so a preview/final mismatch now returns False and the
normal final send fires.
2. run.py: _classify_completion_target classified every ended parent session
terminal unless it ended by compression. Idle/timeout session ends are the
norm on scale-to-zero relay deployments and the chat route remains valid;
completed async delegation results were terminally dropped. Ended parents
now classify deliver unless the end was an explicit user boundary
(session_reset / user_exit / session_switch).
* fix(relay): drain in-flight outbound frames before transport teardown
disconnect() failed every pending outbound future immediately with
'relay transport closed', so a trailing finalize edit racing turn
teardown was lost even though the connector socket could still serve
it. Bounded drain grace (5s) lets in-flight requests resolve; silent
connectors still tear down promptly. asyncio.wait (not gather+wait_for)
so a timeout doesn't cancel futures owned by the fail-remaining loop.
* fix(gateway): route completion injection through the alias-aware transport resolver
Third relay-plane delivery loss from the 2026-08-09 staging incidents: a
delegation batch completed while the gateway was up, the watcher drained
the event, and delivery vanished with no log line. _inject_watch_notification
resolved its adapter with a literal p.value == platform_name scan of
self.adapters — a relay-fronted gateway registers ONE adapter under
Platform.RELAY fronting N logical platforms, so 'slack' never matched and
the injection returned None ('no gateway route'), silently dropping the
completion. The handoff path already documents this exact trap and uses
resolve_delivery_transport; the injection path now does the same (native
wins; relay eligible only when it fronts the logical platform), with the
literal scan kept as fallback for stub runners and exotic platforms.
* fix(relay): clamp disconnect drain grace to the runner's adapter-disconnect budget
Review finding (JoaoMarcos44, #82592): a fixed 5.0s drain in front of the
three 1.0s sequential teardown awaits gives an 8.0s worst case inside the
runner's 5.0s asyncio.wait_for(adapter.disconnect()) — tripping it cancels
teardown mid-drain, skips the fail-pending loop, and leaves outbound
callers blocked until _OUTBOUND_TIMEOUT_S (30s). The effective grace is
now budget - 3*TEARDOWN - margin (env-aware via the same
HERMES_GATEWAY_ADAPTER_DISCONNECT_TIMEOUT the runner reads), so the drain
can never push teardown past its caller's budget; a budget too small for
any drain disables it cleanly.
* test(gateway): pin the final-send suppression contract across a behaviour matrix
The gateway skips its own final send when the stream consumer claims the turn
final already reached the user. Every incident in that family — #71643 (stale
finalize snapshot), #78541 (payload-less multi-message split), #82656 (frozen
preview left with a visible cursor) — is the same failure: the consumer claimed
delivery for text the platform never rendered, so the corrective send was
suppressed and the answer was lost with no retry.
Each was fixed with a scenario test pinned to one branch of
GatewayStreamConsumer.run(). The got_done handler now has five sibling branches
that each set the suppression flags and record a turn-final payload, and nothing
checks them as a group: a new branch, or a new early `return True` in
_send_or_edit, can reintroduce the class without failing a test.
Pin the invariant instead of the branch — if the consumer offers the gateway any
signal it would trust, the complete final text must have reached the wire — and
assert it across {edit always / dies / never / lies} x {send always / never} x
{fresh-final on / off} x {clean / interrupted stream}.
The adapter records only frames that actually rendered, so an ACK the platform
drops does not count as delivery. 24 honest-transport scenarios hold the
invariant as a hard assertion. The 16 lying-transport scenarios are checked too;
the single combination that still violates it is reported as an expected
failure documenting the open exposure rather than asserting it away.
Refs #82656
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(gateway,relay): prime relay egress routing for synthetic injections + cap stale completion replay
Defect #4 from the 2026-08-09 staging incidents (upgrade-robustness):
after every gateway restart the durable async-delegation replay injected
completions correctly (post-741663cf1) but their replies bounced at the
connector — 'slack egress declined: target not routed to an onboarded
tenant'. The relay adapter re-attaches tenant discriminators
(metadata.scope_id / metadata.user_id) from per-chat caches warmed ONLY by
inbound traffic; synthetic turns race those cold caches on every deploy,
scale-to-zero wake, and crash recovery.
- relay adapter: prime_routing_cache() — feeds a synthetic event's
session-store origin through the same _capture_scope used for real
inbound (never raises).
- run.py injection path: prime the resolved adapter before handle_message
(duck-typed; native adapters unaffected).
- async_delegation: 48h staleness cap in restore_undelivered_completions —
a pending completion older than the cap is terminally dropped (payload
stays queryable) instead of re-run as a fresh full-context turn; the
post-restart replay of a July session burned a 102K-token context.
Also carried: JoaoMarcos44's suppression behaviour-matrix harness
(cherry-picked from #82676, authorship preserved) — 39 passed + 1 xfail
(the documented ACK-then-drop transport-honesty residue).
* test: use recent timestamps in restored-ownership fixtures
test_restore_stamps_restored_flag persisted its completion with epoch-era
toy timestamps (dispatched_at=1.0), which the new 48h replay staleness cap
correctly classifies as stale — the fixture then exercised the cap instead
of the restored-flag contract (CI slice 4 failure). Timestamps are now
now-relative; the staleness behavior itself is pinned separately in
test_relay_injection_egress_priming.py.
* fix(gateway,relay): close four review findings on the relay delivery fixes
Review follow-ups on this branch (NousResearch#82592):
1. HIGH — classifier/resolver mismatch (falsely-acknowledged loss).
_classify_completion_target now returns "deliver" for idle-ended
parents, but _resolve_async_delegation_session still dropped every
non-compression-ended pin: the durable row was acked at adapter
acceptance, then the injection died inside the pipeline with no
retry — strictly worse than the honest terminal drop on main, and
the delivery leg defect #2's fix depends on did not exist. The
resolver now retargets non-user-boundary ends (idle/timeout/
lifecycle) to the chat's current session — session_entry already IS
the routing key's current session for the same chat — while user
boundaries (session_reset / new_session / user_exit /
session_switch) stay fail-closed. Both sides share one module-level
_USER_BOUNDARY_END_REASONS so the verdict and the routing decision
cannot drift again; a coherence test asserts deliver-verdicts
resolve non-None across representative end reasons.
2. HIGH — drain clamp missed adapter-level spend. The effective drain
grace budgeted drain + 3x teardown, but RelayAdapter.disconnect
spends revocation-monitor teardown + go_idle time BEFORE the
transport drain inside the same runner wait_for; worst case still
blew the budget and cancelled teardown mid-drain (skipping the
fail-pending loop). The adapter now measures its own elapsed time
and threads the REMAINING budget into
transport.disconnect(budget_s=...); legacy/stub transports without
the keyword fall back to the no-arg signature.
3. P1 — _request_response racing disconnect() could register a future
after the fail-pending loop already ran, stranding the caller for
the full _OUTBOUND_TIMEOUT_S (30s). Fail fast with the same
"relay transport closed" error once _closing is set.
4. P1 — _build_process_event_source's last-resort reconstruction
dropped scope_id, so a scoped relay completion whose session-store
origin was unavailable primed no tenant discriminator and could
still bounce off the connector's fail-closed egress guard.
scope_id now threads through the reconstructed SessionSource, with
a warning when a scoped chat reconstructs without one.
All four: RED reproduced with the fix reverted, GREEN after; relay/
delegation delivery families pass (43 + 71 + 179 across the touched
suites); full tests/gateway run shows only failures already failing
identically on merge base
|
||
|
Brooklyn Nicholson
|
34577fcb03 |
fix(gateway): rename a Discord thread once, after the reply lands
Titling is two-stage — a slice of the user's own words lands inline, the model's version replaces it a second later — and the platform rename lanes fired on both. That is two rate-limited calls to reach one name, and Discord allows two channel renames per ten minutes, so the throwaway could be the one that survived. The callback now carries which stage it is, and the lanes take the model's. The relay lane also asked where the reply landed at title time, which is before the model has answered: it polled the send-result cache for ten seconds and read the timeout as "never auto-threaded", so any turn with tool calls in it silently kept its raw thread name. Wait on the send itself instead — the adapter already owns that cache, so it can say when a reply arrives and, just as usefully, that one arrived carrying nothing. |
||
|
Dannyzen
|
3b0bb3b8bb |
fix(gateway): keep event loop alive during /compress and Relay drain
Offload manual /compress temporary-agent cleanup through the existing bounded off-loop helper so a slow agent.close() cannot freeze the gateway event loop, heartbeat, or platform polling. Guarantee Relay transport teardown even when the runner cancels adapter.disconnect() during go_idle: shielded finally, 2s drain-path idle ACK budget under the 5s outer disconnect budget, and bounded supervisor/reader/ws.close awaits. Original commits: - fix(gateway): offload manual /compress cleanup from the event loop - fix(gateway): tear down Relay transport even if go_idle is cancelled - fix(gateway): keep Relay disconnect budgets inside the runner window By @Dannyzen (PR #78027), salvaged onto current main. |
||
|
Ben Barclay
|
fed098bbf0 |
fix(gateway): use connector-owned no-clobber guard for relay thread rename + trace logs (#75912)
Live staging (2026-08-01): relay semantic thread rename still declined silently despite both #74482 and #75581 deployed — thread kept its initial-words name, session title generated fine. Root cause is the no-clobber guard string mismatch (see paired gateway-gateway PR): the gateway can't reproduce the thread's initial name byte-for-byte, so the connector's only_if_current_name check always failed. - relay rename lane now passes prefer_connector_created=True instead of the fragile initial-name string; the connector resolves the guard from its own created-name memory. Native-marker lane keeps the legacy only_if_current_name string (source carries the real initial name). - rename_thread: prefer_connector_created param -> only_if_connector_created on the wire, precedence over the legacy string. - INFO logs at rename dispatch (thread/lane/new_title) and result (applied=bool): the whole failure hunt needed telemetry the gateway never emitted — this makes the outcome visible in fly logs. Tests: connector-guard wire shape + precedence over legacy string; the title-turn race test updated to assert the connector-owned guard. Relay suite 149 passed; ruff + footguns clean. |
||
|
Ben Barclay
|
d26983e485 |
fix(gateway): relay TTS attachments + semantic auto-thread rename on the title turn (#74482)
Two relay-lane bugs from live Discord staging testing (2026-07-29): 1. TTS audio never attached over relay (any platform, any lane). _history_media_paths_for_session excluded only the trailing assistant entry from the persisted transcript when building the delivered-media dedup set. The agent persists rows as it produces them, so THIS turn's text_to_speech tool result (media_tag JSON) was already in the transcript at delivery time — the fresh TTS path deduped against itself and extract_media's attachment was silently stripped (response_delivery_dropped for a MEDIA-tag-only reply; fly logs show the exact signature). Fix: exclude everything from the last USER message onward (the current turn); prior-turn dedup unchanged. Affects every platform adapter (native + relay) on the non-streaming delivery path — the streaming path passes explicit history and was unaffected. 2. Connector-auto-created threads never got the LLM session-title rename. The title fires on the FIRST exchange, whose source is the PARENT channel event — the thread didn't exist at ingest, so the Phase 4 auto-thread markers can't be present and _is_discord_auto_thread_lane never matches on the relay title turn (initial titles worked; semantic renames never happened; staging telemetry shows zero thread_rename ops ever sent). Fix: consume the connector's new send-result feedback (paired gateway-gateway PR — contract §SendResult thread_id/auto_thread_name, additive): RelayAdapter.send() caches (thread_id, initial_name) per chat (bounded 256), run.py's title-callback registration + rename lane read it back and pass initial_name as only_if_current_name so the human-rename-wins guard holds on the relay lane too. Native marker path unchanged; connectors that don't stamp the fields degrade to exactly the old behavior. Tests: 3 new (send-result feedback capture, absence, bound) in test_relay_threads.py; 3 new in test_history_media_current_turn.py (current-turn TTS not deduped, prior-turn still deduped, no-user-row fallback). Relay suite 144 passed. |
||
|
Ben Barclay
|
0dc293f4f7 |
fix(relay): coerce Slack behavior flags exactly as the native adapter does
Both relay Slack knobs read their value through bool(), while the native
adapter they mirror uses str(raw).strip().lower() in {"1","true","yes","on"}.
A YAML-quoted string diverges:
dm_top_level_threads_as_sessions: "false" → relay True, native False
Non-empty strings are truthy, so the escape hatch is silently ignored in
exactly the shape an operator writes to switch it OFF. reply_in_thread has the
same defect and gates reply placement, session keying and run.py's progress
resolver, so one quoted "false" misfires three ways.
Route both through a shared _coerce_flag mirroring native's predicate. Real
booleans pass through untouched; None falls back to the default. Contract §8
documents the accepted spellings.
Tests: both knobs parametrized over the true/false spellings native accepts,
plus the absent-key default.
|
||
|
Ben Barclay
|
33833e232e |
fix(relay): apply the Slack thread anchor on the media lane too
The DM thread-anchor contract was resolved only in send(). _send_media() —
backing send_image, send_image_file, send_voice, send_video and send_document
— passed reply_to straight to the frame and never touched metadata, so
attachments egressing through the same connector-side Slack sender got both
failure shapes this branch set out to remove:
flat mode → reply_to survives, the image threads UNDER the user's DM
message (the original reported symptom)
thread mode → no metadata.thread_id, and threadTs() never reads reply_to,
so the image lands in the home channel instead of the
per-message thread
Both are reachable: gateway/run.py delivers agent artifacts through
send_voice/send_document.
Extract the three steps that must always happen together (mode gate, mirrored
reply_to_message_id strip, metadata promotion) into
_apply_slack_thread_anchor and route BOTH lanes through it, so text and media
cannot drift again. The media lane copies caller metadata rather than mutating
it — these helpers are called in loops with a shared mapping.
Also fold send_typing/stop_typing's duplicated status-anchor blocks into
_with_status_thread_anchor. They had already drifted (stop_typing omitted the
platform check) and the clear must target the thread the heartbeat set or the
status line sticks until Slack's own timeout.
Tests: media lane pinned in both modes plus the channel and
no-caller-mutation cases; verified as real by reverting the fix and watching
them fail.
|
||
|
Victor Kyriazakos
|
dd866eef34 |
Merge remote-tracking branch 'origin/main' into feat/relay-slack-blockkit-native-parity
# Conflicts: # gateway/relay/adapter.py |
||
|
Victor Kyriazakos
|
277fc97a0a |
feat(relay): dm_top_level_threads_as_sessions escape hatch — native session-keying parity
Review finding: native gates per-message DM sessions behind platforms.slack.extra.dm_top_level_threads_as_sessions; the relay lane coupled session keying to reply_in_thread alone, so 'threaded replies + one rolling session' was expressible on native but not here. Adds the same knob to the relay subset (platforms.relay.extra.slack. dm_top_level_threads_as_sessions, default true = per-message sessions, unchanged behavior). false keeps thread-per-message reply placement but skips the session stamp — one rolling DM session, legacy steer posture. TDD: opt-out + default-unchanged tests written first. |
||
|
Victor Kyriazakos
|
3e628edeb9 |
docs(relay): replace internal QA-N tracker markers with behavior descriptions
Review finding: QA-1/3/5/6/7 are internal campaign tracker ids meaning nothing to future readers of this file. Comments now describe the behavior (status thread targeting, metadata-only threading, session-keying parity) instead of citing the tracker. Comment-only change. |
||
|
Victor Kyriazakos
|
09c4a1d349 |
refactor(relay): remove dead _strip_synthetic_dm_thread; pin the run.py anchor-suppression boundary
Review finding (2026-07-28): every path through _strip_synthetic_dm_thread returned metadata unmodified — the actual strip was removed when prompts switched to trusting the run.py thread stamp, leaving a 50-line no-op and four tests that passed against it (verified by reviewer's negative control). - delete the function + its _send_prompt call site (verbatim pass-through with a pointer comment to the single mode authority) - rewrite the three pass-through tests as end-to-end placement contracts (forward run.py's stamp untouched) - NEW boundary tests pinning run.py._resolve_progress_thread_id itself: flat mode suppresses the synthetic self-anchor / preserves real threads; thread mode keeps the first-turn self-anchor. This is the cross-module coupling the review flagged as unpinned — if the upstream suppression regresses, these fail instead of prompts silently threading. |
||
|
Victor Kyriazakos
|
85a75f3155 |
refactor(relay): drop the flat_dm_status knob — liveliness is unconditional; add relay docs page
flat_dm_status was speculative config (rubric violation): no user wants 'make my agent look dead', and the only real consumer of status suppression was native's placement-contamination guard — which the relay lane handles structurally (QA-6/7 send-side anchor strip, leak-guard test), not via preference. Status now anchors whenever an inbound ts exists, in both modes. Docs: new website/docs/user-guide/messaging/relay.md — enterprise-only relay lane page documenting the platforms.relay.extra.<platform> subset shape (nested wins, flat fallback), the Slack reply_in_thread control, and always-on liveliness. Kept out of the native slack.md on purpose: relay controls are not Slack config. |
||
|
Victor Kyriazakos
|
9864e00fb4 |
feat(relay): flat-DM liveliness — status anchors to the triggering ts, replies stay flat (QA-8)
Victor's correction: flat DMs CAN have a live thinking status. setStatus on the triggering message's ts renders '… thinking'/per-tool phrases in that message's thread-footer space and clears without leaving a message artifact. Native suppresses this because ITS reply routing could inherit the activated thread; the relay lane's flat-mode sends strip their anchors explicitly (QA-6/7), so the status anchor cannot leak into reply placement — proven by the new leak-guard test. send_typing/stop_typing now anchor the status in flat mode too, gated by platforms.relay.extra.slack.flat_dm_status (default ON; false restores the fully anchorless posture). Thread mode unchanged. |
||
|
Victor Kyriazakos
|
71d5c47e21 |
fix(relay): per-message sessions for fronted Slack DMs — stamp the inbound ts as session thread (QA-3)
A 2nd top-level DM while a turn was in flight resolved to the SAME
session key and steered the running turn ('Redirected current run')
instead of starting its own. Native SlackAdapter stamps thread_ts =
event.thread_ts or ts on EVERY inbound, so build_session_key isolates
each top-level message; the connector normalizes top-level messages with
thread_id=null and the relay lane never reproduced the stamp.
_stamp_slack_session_thread applies native parity on the inbound bridge:
top-level Slack message + thread-per-message mode => source.thread_id =
its own ts (fresh session, parallel turns). Real thread replies and flat
mode untouched (flat keeps the shared rolling DM session on purpose).
Also introduces the enterprise config shape for relay-fronted Slack:
platforms.relay.extra.slack.<subset of native Slack fields> (nested
object wins; legacy flat extra.reply_in_thread still honoured). All
reply_in_thread reads (send/typing/stop_typing/run.py progress) now
route through one resolver.
|
||
|
Victor Kyriazakos
|
a51a17ebe3 |
fix(relay): promote the surviving reply_to anchor into metadata.thread_id on Slack sends (QA-7)
The connector's Slack sender threads on metadata ONLY: threadTs() reads metadata.thread_id/thread_ts and never the frame's reply_to. base.py's final-reply lane (and its stream-fallback 'first response' resend) builds metadata from source.thread_id — None for a top-level DM — so its sends carried reply_to as the sole threading signal and posted to the home channel (2026-07-27 post-approval report; the 15:17:03 frame showed meta_keys=['notify','user_id']). After the QA-6 mode gate keeps the anchor, copy it into metadata.thread_id so the wire carries the signal where the connector reads it. Flat mode unaffected (anchor already nulled); explicit thread metadata wins; non-Slack untouched. |
||
|
Victor Kyriazakos
|
467534b43e |
fix(relay): typing/status targets the per-message thread — synthesize the anchor from the inbound ts (QA-1)
Slack's thinking-status line (thread replies footer, plain chat:write — no assistant scopes needed) is thread-only: the connector's typing case no-ops without thread_ts. The typing lane's metadata has no anchor for a top-level DM (base.py builds from source.thread_id = None), so every status heartbeat was silently dropped — the trace showed typing frames with meta_keys=['user_id'] only. Cache the triggering message ts per chat on inbound (_capture_scope) and synthesize metadata.thread_id on send_typing/stop_typing in thread-per-message mode, mirroring native send_typing's _resolve_thread_ts(metadata.message_id). Flat mode unchanged (#18859); real-thread metadata wins over the cache; the clear frame targets the same synthesized thread so the status never sticks. |
||
|
Victor Kyriazakos
|
b493bf63c7 |
feat(relay): rich Slack status-line parity — advertise supports_status_text, carry live per-tool phrase on typing frames (QA-1)
Native Slack shows dynamic assistant-status text ('Finding answers…',
'is running pytest…') because SlackAdapter sets supports_status_text=True
and renders the set_status_text() phrase in send_typing. The relay lane
advertised nothing, so run.py's live-status lane never fed it phrases and
the connector fell back to the static default.
- supports_status_text: descriptor-gated property (Slack only; other
fronted platforms keep textless bubbles)
- send_typing: carry the stashed phrase as the typing op's content; omit
when unset (empty string is Slack's explicit clear, reserved for
stop_typing). Connector already renders content via
assistant.threads.setStatus (#154).
|
||
|
Victor Kyriazakos
|
be9de31967 |
fix(relay): final DM reply honors thread-per-message mode — gate the reply_to strip on reply_in_thread (QA-6)
_resolve_reply_to_for_send dropped the triggering-ts reply_to on every Slack DM with no metadata thread_id. But the final-reply lane (platforms/ base.py) builds metadata from source.thread_id only — None for a top-level DM — so in thread-per-message mode that reply_to is the final reply's ONLY threading signal, and stripping it exiled the final message to the DM root while progress bubbles stayed threaded (sibling of the QA-5 prompt bug). Mirror native _resolve_thread_ts: suppress the synthetic anchor only when platforms.slack.extra.reply_in_thread=false. Flat mode behavior unchanged; real threads and channels unchanged. |
||
|
Victor Kyriazakos
|
95103db645 |
fix(relay): prompts trust the run.py thread stamp — no self-anchor re-derivation (QA-5)
The threading mode (flat vs thread-per-message) is decided once, in run.py's _resolve_progress_thread_id (reply_in_thread knob), and encoded in the metadata stamp: flat => no thread_id, threaded => thread_id for the turn (first turn: == message_id, the synthetic root IS the thread). _strip_synthetic_dm_thread re-derived the mode with an unconditional thread_id == message_id strip, exiling approval/clarify cards (and their resolved-state swaps) to the DM root while progress bubbles honoured the thread (2026-07-27 mixed-placement report). Trust the stamp instead; flat mode is unaffected because flat metadata never carries an anchor. |
||
|
Ben Barclay
|
96996a55bf |
fix(relay): per-platform capability descriptors for multi-platform gateways (#70717)
One relay adapter fronts N platforms on one WS, but the capability surface (MAX_MESSAGE_LENGTH / message_len_fn) was a scalar from whichever descriptor resolved the handshake — and the transport's read loop OVERWROTE it on every descriptor frame (last-writer-wins). A Discord chat on a gateway whose applied descriptor was Telegram's inherited the 4,096-char cap and over-sent into Discord's 2,000-char API 400 (observed live: 2,543/2,641-char replies silently lost while inbound kept working). - ws_transport: accumulate one descriptor per platform in _descriptors_by_platform (exposed via descriptor_for_platform); the FIRST descriptor of a connection generation stays the session default instead of last-writer-wins; the map resets on re-dial. - BasePlatformAdapter: new max_message_length_for_chat / message_len_fn_for_chat hooks defaulting to the scalar surface (native single-platform adapters unchanged). - RelayAdapter: overrides resolve the chat's platform from _platform_by_chat (the same map per-frame egress uses) and look up that platform's negotiated descriptor; falls back to the scalar for unknown chats/transports. - stream_consumer (streaming budget, _raw_message_limit, fallback-continuation chunking) + run.py tool-progress limit now resolve per-chat. Tests: tests/gateway/relay/test_relay_per_platform_caps.py (7) — verified fail-without/pass-with (all 7 fail with the fix stashed). Relay + stream consumer suites green (213 + 223). |
||
|
Victor Kyriazakos
|
42e4f70eef |
fix(relay): native-parity Slack approval button styles + labels
Slack Block Kit buttons only support style primary (green) / danger (red) / default (white). The relay approval + slash-confirm prompts emitted an invalid style 'success' (Slack silently drops it → white/stroke button) and baked emoji into the labels (non-native). Native Slack Hermes uses plain labels with primary/danger. Map to valid styles (once→primary, deny/cancel→danger, session/always→default) and drop the emoji from labels. The connector already compensates success→primary, but emitting valid values at the source is correct and removes the fragile dependency on that compensation. |
||
|
Victor Kyriazakos
|
15d65da5a2 |
fix(relay): stream Slack DM replies flat at DM root (native _resolve_thread_ts parity)
On the relay lane a Slack DM's streamed reply was sent with reply_to=the triggering message ts; the connector maps a raw reply_to to a Slack thread_ts, so the DM reply posted threaded under the user's message and lost progressive edit-streaming (flat reply, no thinking status). Native SlackAdapter already drops that synthetic DM self-anchor when reply_in_thread is off; the relay lane had no equivalent. Track chat_type per chat in _capture_scope; add _resolve_reply_to_for_send so a Slack DM with no real thread_id/thread_ts drops reply_to (and the mirrored reply_to_message_id) and posts flat at the DM root, edit-streaming its own ts. Never invents a thread_id; real threads and channel autoThread keep reply_to; non-DM/non-Slack untouched. Adapted to main's phase-3 prompt architecture. |
||
|
Ben Barclay
|
40eebc7d70 |
feat(relay): Phase 4 thread lifecycle — handoff threads, semantic renames, reply_to context, hello command manifest (#71624)
- RelayAdapter.create_handoff_thread → one op-gated thread_create op
(Discord channel thread / Telegram forum topic / Slack named seed root);
None fallback contract preserved for the handoff watcher.
- RelayAdapter.rename_thread → thread_rename with only_if_current_name
no-clobber guard on the wire (connector enforces; Telegram guarded
renames fail safe). The native semantic-rename lane
(_is_discord_auto_thread_lane) lights over the relay via the
connector-stamped auto_thread_created/auto_thread_initial_name markers
parsed onto SessionSource in _event_from_wire.
- reply_to {text, author, is_own} wire parse onto the SAME MessageEvent
reply-context fields native adapters populate.
- gateway/relay/command_manifest.py: the gateway-declared slash-command
manifest (native Discord tree mirror) sent on the DISCORD hello; the
connector reconciles Discord's global registration (additive field,
older connectors ignore).
- Contract doc §OutboundAction ops + Phase 4 semantics sections.
- 12 new tests (tests/gateway/relay/test_relay_threads.py); relay suite
266 passed.
|
||
|
Ben Barclay
|
fffa661223 |
feat(relay): Phase 3 interactive — native prompt UX (approvals/confirms/clarify) + react ack lifecycle (#71404)
- RelayAdapter.send_exec_approval / send_slash_confirm / send_clarify:
override the base text fallbacks with ONE platform-abstract `prompt` op
(connector renders Discord components / Telegram inline keyboards /
Slack Block Kit / WhatsApp buttons+lists). Option sets mirror the native
adapters exactly (once/session/always/deny with the same
allow_session/allow_permanent/smart_denied gating; once/always/cancel;
choices + Other). Clarify option ids are positional (c0..cN/other) —
choice text is arbitrary UTF-8, callback budgets are 64 bytes.
- Pending-prompt registry: gateway-minted 8-hex prompt ids →
{kind, session_key, extras}; one answer wins, lazy expiry, unanswered
prompts swept opportunistically. Wire timeout_s stays advisory.
- _consume_prompt_response (wired into _on_inbound AND the Discord
passthrough lane): routes answers to the SAME primitives the native
button handlers call — tools.approval.resolve_gateway_approval,
tools.slash_confirm.resolve, tools.clarify_gateway
resolve/mark_awaiting_text — then acks in-channel. Unknown/expired ids
fall through as command-shaped text (typed-reply degradation, the
relay's analog of the native 'approval expired' edit).
- Discord type-3 stub replaced: an hp1:<prompt>:<option> custom_id decodes
to a structured prompt_response (codec mirrored from the connector's
promptCodec); foreign custom_ids keep the legacy best-effort text shape.
- MessageEvent.prompt_response field + ws_transport wire parsing (additive).
- react ack lifecycle: on_processing_start/complete → `react` ops
(👀 → ✅/❌, remove-then-add), op-gated on supported_ops, best-effort by
contract (a react failure never touches the turn).
- Op gating throughout: a connector not advertising `prompt` gets
success=False from send_exec_approval/send_slash_confirm (run.py's text
fallback takes over — same contract as a failed native button send) and
the base numbered-text clarify; `react` silently no-ops.
- docs/relay-connector-contract.md §4: prompt / prompt_response / react
semantics (callback token, budgets, authorization-parity, foreign-id
behavior, per-platform react mappings).
- tests: tests/gateway/relay/test_relay_interactive.py (19) — option-set
rendering + gating matrices, registry consume-once/expiry, resolver
routing for all three kinds (monkeypatched primitives), fall-through
cases, Discord hp1 decode + foreign-id shape, react lifecycle
(success/failure/cancelled), op-gated/best-effort react.
Cross-repo pair: gateway-gateway 'Phase 3 interactive' PR (prompt/react
senders on all four lanes + interaction ingest).
|
||
|
Ben Barclay
|
689b51bef6 |
feat(relay): Phase 2 media parity — send_media egress + inbound media localization (#71363)
- gateway/relay/media.py: RelayMediaClient for the connector's /relay/media plane (upload local files → re-host reference for send_media; download re-hosted inbound attachments → local temp paths). Same connector base URL the WS dials, same per-gateway signed bearer as the upgrade (auth.py) — no new configuration. stdlib urllib in a thread executor (no new deps); 25MB cap mirroring the connector's MEDIA_MAX_BYTES. - RelayAdapter: send_image / send_image_file / send_voice / send_video / send_document overrides route through ONE send_media op (media by reference: local paths upload first, public URLs pass through). Gated on supported_ops advertising send_media — legacy connectors keep today's text fallbacks; connector declines/failed uploads degrade the same way. Scope/user egress discriminators ride metadata exactly like send. - Inbound: _localize_inbound_media downloads each media_urls entry to a local temp path (native-adapter parity — vision/file tools consume paths); dead re-host refs are dropped, public URLs survive a missing client. Best-effort, never blocks handle_message. - docs/relay-connector-contract.md §4: send_media op row + media ingress/egress semantics (replaces the 'deferred to a later revision' note). Additive within contract_version 1. - tests: tests/gateway/relay/test_relay_media.py (15) — kind mapping, upload-first path handling, op gating (explicit + legacy-empty), decline/upload-failure fallbacks, scope metadata, inbound localization matrix, client URL derivation/credential gating. Stub connector grew a canned send_media result. Cross-repo pair: gateway-gateway 'Phase 2 media parity' PR (re-host plane + four ingress lanes + four platform send_media senders). |
||
|
Ben Barclay
|
ebab890ae5 |
feat(relay): Phase 1 parity — supported_ops discovery, wire identity fields, /handoff aliasing, provision displayName (#71300)
- CapabilityDescriptor.supported_ops + supports_op() with legacy-op-set fallback (additive, contract doc §2 updated) - get_chat_info gated on op discovery (skip round trip on legacy connectors) - _event_from_wire consumes user_display_name/user_handle (§3 fields previously dropped); native display-name parity, session-key stable - /handoff <fronted-platform> works on relay-fronted gateways: CLI pre-check reads the GATEWAY_RELAY_PLATFORMS fronted set; watcher resolves through resolve_delivery_transport and replies via send_for_platform - self-provision forwards displayName (env > skin branding, stock brand suppressed) — the primary name source for gg#171 attribution |
||
|
Ben Barclay
|
e0dfcf275a |
fix(relay): normalize forwarded Discord interactions to leading-slash commands (#71048)
A real APPLICATION_COMMAND interaction forwarded over the relay arrived
slash-less: _discord_interaction_to_event set text = data['name'] ("new",
not "/new"), MessageType.TEXT, and dropped options entirely — so a
registered /new dispatched as plain chat instead of a command
(MessageEvent.is_command() is text.startswith("/")).
Port the connector's Slack slash-command precedent (normalizeSlackCommand
builds `${command} ${args}`.trim() with a leading slash and explicit
command type): for type-2 interactions build "/" + name, append rendered
options space-separated (scalar options contribute their value, matching
the native adapter's f"/model {name}" shape; SUB_COMMAND/
SUB_COMMAND_GROUP contribute their name then recurse into nested
options), and set MessageType.COMMAND. Type-3 (custom_id) and other
interaction types are unchanged. This implements the interaction->command
sub-design previously flagged as deferred in the _on_passthrough
docstring.
Companion connector fix in gateway-gateway: fix(relay): strip own-mention
prefix so addressed slash commands dispatch.
|
||
|
Victor Kyriazakos
|
45a408f41a | fix(gateway): deliver relay-backed homes after restart | ||
|
Ben Barclay
|
305a3c7424 |
fix(relay): restore streaming delivery, Slack command parity, and status clearing (salvage of #69716) (#69747)
* fix(gateway): restore relay streaming delivery * fix(relay): route Slack parent commands before session gates * fix(relay): clear Slack typing status after turns --------- Co-authored-by: Victor Kyriazakos <victor@rocketfueldev.com> |
||
|
Ben Barclay
|
e5b83633ec |
feat(relay): egress typing indicators through the connector (op="typing") (#69721)
The base class spawns _keep_typing for every adapter — a 2s refresh loop
that runs for the whole turn, including while the stream consumer sends
the response — but RelayAdapter inherited BasePlatformAdapter's no-op
send_typing. The loop ran every turn and emitted nothing, so relay-fronted
chats (hosted Discord/Telegram/Signal/Slack) never showed 'is typing…'.
The rest of the pipe already existed on both sides: OutboundOp "typing"
is in the wire contract (contract_version 1), and every connector-side
sender implements it (Discord POST /channels/{id}/typing, Telegram
sendChatAction, Signal sendTyping, Slack assistant status).
This bridges the tick onto the existing outbound frame, mirroring send():
- _with_scope re-attaches the tenant discriminator (metadata.scope_id, or
user_id for DMs) — the connector's routedEgressGuard wraps ALL ops, so
an undiscriminated typing frame is declined like a bare send would be.
- the Phase 1.5 per-frame platform tag routes typing through the platform
the chat lives on for multi-platform gateways.
Best-effort by design: transport errors are swallowed (typing is cosmetic;
the next tick retries), no transport is a silent no-op, and stop_typing
stays the base no-op (platform indicators self-expire). Additive within
contract_version 1 — an older connector returns an unsupported-op result
we ignore. Contract doc §4 updated (typing now carries metadata?).
|
||
|
Ben Barclay
|
f4df260f26 |
fix(relay): attach metadata.user_id on guild replies for egress fallback (#68320)
The relay adapter re-attaches an egress discriminator on outbound replies so the connector can resolve the owning tenant. It captured scope_id for scoped (guild) messages and user_id for DMs, but as MUTUALLY EXCLUSIVE: a scoped inbound hit an early return, so the author's user_id was never recorded, and _with_scope only attached user_id when there was no scope_id. Guild replies therefore went out with scope_id only. That's fine while the guild has a provision-time route row. But a MANAGED Discord agent joins guilds dynamically (the shared bot is added to / removed from servers at runtime), and GATEWAY_RELAY_ROUTE_KEYS — the only thing that writes guild route rows — is a self-hosted, static field never stamped for managed agents. So their guild has no route row, the connector's guild-route lookup misses, and with no user_id on the frame there's nothing to fall back to → every guild reply is declined "discord egress declined: target not routed to an onboarded tenant" even though INBOUND resolved the same guild fine (via the author-first SharedSocketRouter.targets() fallback). Fix: capture the authentic author user_id for EVERY inbound (DM and scoped alike) and re-attach it on the outbound reply alongside scope_id. The connector consults it only on a route/scope miss, so carrying both never overrides routing-table resolution. This is the gateway half of the paired gateway-gateway change (makeDiscordTenantOf guild-route-miss author-binding fallback); together they make guild replies resolve the same observed-author way inbound already does. Tests (tests/gateway/relay/test_relay_adapter.py): a guild reply now carries both scope_id AND user_id; a scoped inbound with no author still yields scope_id only (never invents one). Verified fail-without / pass-with. |
||
|
Ben Barclay
|
729bbb7a30 |
refactor(relay): purge platform-specific scope terminology from the relay adapter (D-Q2.5c) (#56016)
The gateway HALF of the D-Q2.5c cleanup (connector half: gateway-gateway #92). Scope is STRICTLY the relay adapter (gateway/relay/) — session.py and every native platform adapter are untouched (SessionSource.guild_id remains for their use; it is NOT relay-only). Within gateway/relay/, drop the D-Q2.5 wire dual-write/dual-read alias AND genericize all platform-specific (Discord "guild") scope terminology: - ws_transport._event_from_wire: read scope_id only (drop the ?? guild_id fallback). - adapter._with_scope: emit scope_id only on outbound metadata (drop the guild_id dual-write); genericize the "GUILD reply" docstring to "SCOPED reply". - adapter._capture_scope: read source.scope_id only; rename the local `guild` var to `scope`; genericize the docstring + the _scope_by_chat/_dm_user_by_chat field comments ("guild_id (Discord)" -> "scope_id (server/workspace scope)"). - __init__.relay_route_keys docstring: "guild_ids" -> "scope_ids". - The ONE real Discord `guild_id` kept: the raw inbound interaction payload field (payload.get("guild_id")), which is Discord's own wire field, mapped straight into the generic scope_id slot — unchanged. Contract doc (docs/relay-connector-contract.md): reframe the `guild_id` row as a legacy alias the connector no longer reads (session.py's agent-wide to_dict() still emits it for non-relay persistence, so it stays documented + wire-present but ignored) — accurate, and keeps the to_dict()-vs-doc conformance test green. Tests (relay only): migrate the wire-key writes + assertions guild_id -> scope_id across test_relay_adapter / _ws_transport / _passthrough / _roundtrip / _roundtrip_telegram / _multiplatform; keep raw Discord `type:2` interaction payloads' guild_id (real Discord field) and the conformance test's guild_id parametrize (validates the kept legacy field stays wire-reachable). Gate: 156 relay tests pass, ruff clean. Cross-repo E2E — all 14 drivers pass BOTH ways: connector#92 (scope_id-only) x agent-main (still dual-reads) AND connector#92 x this worktree (scope_id-only). Deploy-order-safe either way. |
||
|
Ben Barclay
|
3a55f66602 |
refactor(relay): adopt scope_id wire key (guild_id → scope_id dual-read/write) (#55289)
Gateway half of relay-platform-parity Phase 2.5 (D-Q2.5). The relay wire's platform-neutral scope discriminator is renamed guild_id → scope_id; this is the hermes-agent side of the cross-repo wire-compatible migration. - SessionSource: scope_id is canonical; guild_id kept as @deprecated alias. __post_init__ mirrors the two so all existing SessionSource(guild_id=...) constructors across native adapters keep working unchanged. to_dict dual-WRITES scope_id+guild_id; from_dict dual-READS scope_id ?? guild_id. - relay/adapter.py: capture + outbound metadata dual-read/write scope_id. - relay/ws_transport.py: _frame_to_event dual-reads scope_id ?? guild_id. - docs/relay-connector-contract.md: document scope_id (canonical) + guild_id (deprecated alias) in the §3 SessionSource field table (conformance test). 250 relay+session+contract tests green. Solo lane (relay). |
||
|
Ben Barclay
|
6e4e5967f7 |
feat(relay): multi-platform-per-agent — list identity, provision-loop, N-hello, per-frame egress (Phase 1.5) (#52830)
Cut over the agent half of Shape A (D-Q1.5a/b.1/c) to front a SET of platforms on
one relay WS:
- relay_platform_identities() parses GATEWAY_RELAY_PLATFORMS (list) +
GATEWAY_RELAY_BOT_IDS (JSON keyed map {platform:{botId,username?}}). Cut over
from the scalar GATEWAY_RELAY_PLATFORM/_BOT_ID (no fallback, D-Q1.5c).
- self_provision_relay() loops one /relay/provision per platform under one
gatewayId+secret, partial-failure-tolerant.
- WebSocketRelayTransport takes the identity SET, sends one hello per identity
(connector accumulates the advertised set), and stamps the per-frame
OutboundFrame.platform + its matching advertised botId on outbound.
- RelayAdapter remembers each chat's underlying source.platform (mirroring the
existing guild/dm scope capture) and tags the reply's egress platform.
- send_relay_policy() declares one relevance policy per fronted platform (the
connector keys policy by (tenant,platform,instanceId)).
Single-platform deploys are byte-identical on the wire (1-element list, no per-frame
tag -> connector session-default fallback). typecheck/ruff clean; relay unit 221 pass
(+10 new); all 15 cross-repo E2E drivers green vs connector origin/main.
|
||
|
Ben Barclay
|
cb7d1f68f8 |
fix(relay): accept is_reconnect kwarg in RelayAdapter.connect (#52911)
The gateway reconnect watcher (gateway/run.py) recovers a platform after a
fatal adapter error by building a fresh adapter and calling
connect(is_reconnect=True). Every BasePlatformAdapter implements
connect(*, is_reconnect: bool = False) for this — except RelayAdapter, whose
connect() was bare. So the watcher's recovery path raised:
TypeError: connect() got an unexpected keyword argument 'is_reconnect'
Observed live on a hosted staging agent: after a fatal relay adapter error the
watcher could never re-establish relay, so the shared-bot inbound never reached
the gateway and Discord DMs stopped (dashboard surfaced the TypeError).
Relay deliberately ignores the flag: the #46621 server-side-queue-preservation
concern doesn't apply, because relay's outage buffer is the connector's durable
buffer (replayed on the transport's re-handshake), not a gateway-side queue the
adapter owns. Routine WS drops are already handled by the transport's own
reconnect supervisor (WebSocketRelayTransport, reconnect=True); the watcher path
is fatal-error recovery, and the fatal handler disconnect()s the old adapter
(cancelling its supervisor) before a fresh adapter+transport is built, so there
is no double-dial.
Adds two regression tests (both proven red without the fix): connect(is_reconnect=True)
reaches the same transport-less RuntimeError instead of TypeError, and the
signature matches BasePlatformAdapter.connect.
|
||
|
Ben
|
0c3f197cff |
fix(relay): re-attach DM author user_id on outbound for connector egress
A DM reply carries no guild_id, so the connector's egress guard cannot resolve the owning tenant from metadata.guild_id and declines the send with "discord egress declined: target not routed to an onboarded tenant" — the bug behind "the bot never replies in DMs". Guild replies are unaffected (they carry guild_id), which is why the guild path worked end-to-end while DMs looked broken. The connector now resolves a DM reply's tenant from the recipient's author binding (gateway-gateway #67, resolveByUser keyed on metadata.user_id) — the outbound counterpart to inbound Phase 7a author-first resolution. But it needs the recipient user_id ON the outbound action, and the adapter only re-attached guild_id (_capture_scope/_with_scope), no-op for DMs (the docstring even said so). This extends the adapter's inbound-scope capture: for a DM (no guild_id) remember chat_id -> the authentic author user_id we observed, and re-attach it as metadata.user_id on outbound. Guild capture is unchanged and wins when present; user_id is the DM-only fallback. The id is the one the connector observed inbound (never gateway-asserted), so the trust invariant holds. +4 unit tests (DM reply re-attaches user_id + no guild_id; unknown chat invents nothing; explicit user_id preserved; guild reply never carries user_id). Proved load-bearing (reverting the re-attach fails the DM test). 144 relay tests pass, ruff clean. Pairs with gateway-gateway #67 (the connector-side resolver). Together they close the DM-reply egress gap end-to-end. |
||
|
Ben
|
96af4bec30 |
feat(relay): add go_dormant() transport mode for scale-to-zero (0.E0)
Net-new WebSocketRelayTransport.go_dormant() + RelayAdapter.go_dormant() — the third transport mode the scale-to-zero behaviour layer needs, distinct from both disconnect() and an unexpected close (decisions.md D12/F14): - disconnect() sets _closing=True and CANCELS the reconnect supervisor (terminal "shutting down for good") -> a suspended machine never re-dials on wake, stranding its buffered backlog. - an unexpected close re-dials IMMEDIATELY -> the socket never stays down, so the platform proxy never suspends the machine. go_dormant(): going_idle->ack (reuse go_idle), then close the socket WITHOUT setting _closing, so the reader's fall-through still arms the reconnect supervisor (wake path stays live) but on the longer _dormant_redial_s cadence so it doesn't fight the platform suspend window. A successful re-dial clears _dormant. Honors the §3.4 wake->reconnect->drain contract. Tests: 6 new in test_relay_going_idle.py incl. the F14 regression guard (routing dormancy through disconnect() fails exactly the 4 wake-path tests). Full relay suite 140 passed. |
||
|
Ben
|
d335164833 |
fix(relay): authorize relay inbound via connector-enforced upstream authz
A hosted instance fronted by the Team Gateway connector dropped EVERY relay
message as "Unauthorized user" and the agent never replied — despite the
message routing correctly through the connector to the instance.
Root cause: gateway authorization (_is_user_authorized) had no notion of
upstream-enforced authz. Platform.RELAY matches no {PLATFORM}_ALLOWED_USERS
allowlist and isn't in the HA/WEBHOOK always-authorized set, so a relay user
with no env allowlist configured hit the default-deny ("No user allowlists
configured. All unauthorized users will be denied."). The message was received,
then silently denied before reaching the agent.
This is incorrect for relay: the connector authenticates the gateway's WS with
a per-instance secret and performs owner-only author-binding resolution BEFORE
delivering. A message only reaches this gateway because the connector resolved
it to THIS instance's bound user (user_instance_binding), keyed on the author id
the connector OBSERVED off the event — never a gateway claim. The authorization
decision is already made by a trusted, authenticated upstream; there is no local
RELAY_ALLOWED_USERS allowlist to consult, and default-denying for its absence is
the bug.
Fix: add a generic BasePlatformAdapter.authorization_is_upstream capability
(default False) that the relay adapter overrides to True, plus a dedicated
trusted branch in _is_user_authorized that honors it. This is delegation to a
trusted upstream, NOT a fail-open: it fires only for an adapter that explicitly
declares the flag; every direct network-exposed adapter leaves it False and the
env-allowlist default-deny (SECURITY.md §2.6) is unchanged. Distinct from
enforces_own_access_policy, which mirrors a LOCAL config-driven allowlist —
this delegates to an authenticated upstream's decision.
Tests: behavior contract that the base defaults False, the relay adapter
declares True, a relay user (group + DM) is authorized with no env allowlist,
and crucially a non-upstream adapter with no allowlist still default-denies
(guards against the fix becoming a blanket fail-open). 6 new tests; relay +
authz + config-policy suites green (134 + 90).
Found via live staging debug of the Discord self-serve onboarding flow.
|
||
|
Ben
|
c93b9f9057 |
feat(relay): terminal 4401 (opt-out) → clean "Relay disabled" state
Phase 7 Unit 7d-B. When an operator opts an instance OUT of the Team Gateway
relay (Unit 7b deprovision), the connector revokes the per-gateway secret and
closes the gateway's WS with 4401. The reconnect supervisor previously treated
EVERY close as retryable, so the live process spun "retrying 4401" forever and
the dashboard showed a red error — opt-out looked like a failure.
Now a 4401 close that arrives AFTER a successful handshake is recognized as a
terminal credential revocation:
- ws_transport.py: track `_handshake_succeeded` (set when a descriptor is
received); on a 4401 close after a prior success, latch `auth_revoked` and do
NOT spawn the reconnect supervisor. A 4401 BEFORE any successful handshake
stays retryable (cold-start / not-yet-provisioned race, not a revocation).
New `auth_revoked` property + a websockets-version-safe close-code reader
(prefers `.rcvd`/`.sent` Close frames; `.code` is deprecated in websockets 13+).
- adapter.py: a revocation monitor turns `transport.auth_revoked` into a clean,
NON-retryable `relay_disabled` fatal and notifies the gateway's fatal-error
handler (so the adapter is removed and NOT queued for reconnection — the
credential is dead until the instance is recreated). Monitor is cancelled on
disconnect; only started when the transport exposes `auth_revoked` (prod WS).
- run.py: `_handle_adapter_fatal_error` maps the `relay_disabled` code to a
`disabled` platform_state (not `fatal`/`retrying`).
- web: PlatformsCard renders the `disabled` state with a neutral outline badge,
a PowerOff icon, and muted (not destructive-red) text + message. New optional
`status.disabled` i18n string ("Disabled").
Also bundles the Phase 7 contract-doc update (this doc is authoritative in
hermes-agent): docs/relay-connector-contract.md gains an "Author-first
resolution + the account-link (DM) path" section documenting the
multi-tenant-guild rule (D-7.2 — route by authenticated author binding, never by
guild; unlinked → fail-closed), the `/link <code>` DM flow, and the
connector-authoritative opt-out + terminal-4401 behavior this PR implements.
Tests: +2 ws_transport (4401-after-handshake terminal / no-reconnect;
4401-before-handshake stays retryable) and +2 adapter (revocation → non-retryable
relay_disabled fatal + handler fired; no-revocation → no fatal). 138 relay tests
pass (incl. the contract-doc conformance test); ruff clean; web tsc clean.
Phase 7 Unit 7d-B (relay-adapter solo lane). Q17 → Option 2; Option 3 (live
de-register, no recreate) + the restart-re-provision hole deferred post-alpha.
|
||
|
Ben Barclay
|
40fddc9e4c |
feat(relay): Phase 5 §5.3 going-idle / buffered-flip primitive (gateway side) (#51572)
The gateway half of the going-idle/buffered-flip primitive (scale-to-zero PRIMITIVE, not the behaviour). Integrates with the EXISTING drain transition: - ws_transport: `go_idle()` sends `going_idle` + awaits the connector's `going_idle_ack` (connector-authoritative flip-then-ack, Q-5.3c — stays serving until the ack so nothing is lost in the flip window); acks a buffered inbound (bufferId present) via `inbound_ack` after the handler runs (drain-without-dup on the delivery leg); NET-NEW reconnect loop re-dials + re-handshakes after an unexpected close (off by default, on in production). - adapter: emits `going_idle` from its existing `disconnect()` drain seam before tearing down the socket; best-effort + guarded (never blocks shutdown). - transport Protocol + contract doc §3.2 document the 3 new frames. +6 relay tests (124 pass). NOT in scope: the autonomous idle timer / machine suspend / NAS health model (deferred behaviour). Ben's relay-adapter solo lane. |
||
|
Ben Barclay
|
64a507da44 |
feat(relay): handle passthrough_forward over the WS (Phase 5 §5.1, gateway half) (#50702)
The connector half (gateway-gateway) moves the passthrough plane's post-ACK forward off the HTTP gatewayEndpoint onto the gateway's outbound /relay WS via a new passthrough_forward frame. This is the gateway side: the relay adapter now RECEIVES and handles that frame, so a hosted gateway (no public IP) can process forwarded Class-2/3 traffic (Discord interactions, Twilio) over the socket it already holds — closing the "passthrough inbound doesn't work for hosted gateways" gap. - ws_transport.py: decode the passthrough_forward frame; PassthroughForward dataclass + _passthrough_from_wire (base64 body -> exact bytes, byte parity with the connector's toPassthroughForward); set_passthrough_handler mirrors set_interrupt_inbound_handler. - transport.py: PassthroughHandler type + set_passthrough_handler on the RelayTransport protocol. - adapter.py: connect() wires the passthrough handler; _on_passthrough decodes the (already-sanitized, token-free) forward and, for a Discord interaction, converts it to a MessageEvent routed through the normal agent path (handle_message) — the reply egresses over the outbound / token-less follow_up path, so the gateway never holds the interaction credential. Never raises (a bad forward can't kill the read loop). Non-discord forwards (Twilio) are logged + dropped for now. - docs/relay-connector-contract.md: document the passthrough_forward frame + PassthroughForward shape + §3.1. The interaction -> MessageEvent CONVERSION semantics (slash-command vs button UX, option rendering) are the open sub-design flagged in the spec; the TRANSPORT + receive mechanism (this) is settled per Ben's Gate-2 decision: "the relay adapter handles receiving these events over the WS." Tests (tests/gateway/relay/test_relay_passthrough.py): byte-preservation round-trip (+ malformed-body tolerance), connect() wiring, application-command and message-component interactions route through handle_message with correct session source + scope capture, malformed/non-discord forwards dropped cleanly. 100 relay tests green. Pairs with the connector PR (gateway-gateway). |
||
|
Ben Barclay
|
a64fc490fe |
fix(relay): make hosted gateways actually connect AND complete the inbound/outbound round-trip (#48828)
* fix(relay): enable RELAY platform + normalize dial URL so hosted gateways actually connect Three bugs blocked a self-provisioned hosted gateway from ever establishing its inbound relay WS (found while standing up the live staging end-to-end). Each masked the next; all three are needed for inbound to work. 1. RELAY platform never enabled in config.platforms (gateway/config.py). register_relay_adapter() puts the adapter in the platform_registry, but start_gateway()'s connect loop iterates self.config.platforms — which never contained Platform.RELAY. So the adapter was "registered" but never connected (logs showed "relay adapter registered" then "No messaging platforms enabled"). Fix: _apply_env_overrides now enables Platform.RELAY (mirroring relay_url into extra for the connected-checker) when GATEWAY_RELAY_URL (env) or gateway.relay_url (yaml) is set. Absent -> no RELAY entry (direct/ single-tenant gateways unaffected). 2. URL scheme not converted for the WS dial (gateway/relay/ws_transport.py). The relay URL is configured once as the http(s):// base (used as-is for the provision POST), but websockets.connect rejects http(s):// with "scheme isn't ws or wss". Fix: _ws_dial_url converts https->wss / http->ws. 3. /relay path not appended (same helper). The connector mounts its WebSocketServer at path "/relay" and returns HTTP 400 on an upgrade to any other path. GATEWAY_RELAY_URL is the base (no /relay), so the dial hit "/" -> 400. Fix: _ws_dial_url ensures the path ends in /relay. Idempotent — a URL already carrying ws(s):// and/or /relay is unchanged, so provision's _provision_url (which derives /relay/provision from either form) still works. Why the cross-repo E2E missed #2/#3: the stub connector binds ws://host:port and its websockets.serve accepts ANY path, so neither the scheme nor the /relay path was exercised. Real connector needs both. Verified live on staging hermes-agent-stg-automated-perception-5054: after the fixes the gateway logs "Connecting to relay..." -> "✓ relay connected" -> "Gateway running with 1 platform(s)" against wss://gateway-gateway.staging-nousresearch.com/relay, stable. Tests: added _ws_dial_url scheme+path+idempotency cases (test_ws_transport.py) and RELAY-platform-enablement cases for env + yaml + absent (test_config.py). Full gateway/relay + config suites green (191 passed). Relay-adapter lane. EXPERIMENTAL. * fix(relay): re-attach guild_id to outbound so connector egress resolves the tenant The final bug in the hosted-relay round-trip. Inbound worked end to end (Discord -> connector -> bus -> agent WS -> agent runs -> reply), but the reply's egress was declined by the connector: "discord egress declined: target not routed to an onboarded tenant". Cause: the connector's routedEgressGuard resolves the owning tenant from the OUTBOUND action's metadata.guild_id (Discord's routing discriminator). The gateway's generic delivery path builds outbound metadata via run.py _thread_metadata_for_source, which only carries thread_id (and returns None entirely for a non-threaded message) — so guild_id never reached the connector, tenant resolution failed, and the shared bot refused to post. Fix (relay-adapter-local, no perturbation of the generic delivery path or other platforms): RelayAdapter learns chat_id -> guild_id from each inbound event (_capture_scope) and re-attaches it to the outbound action's metadata in send() (_with_scope) when not already present. No-op for chats we never saw inbound (e.g. DMs) and never overwrites an explicit guild_id. Verified live on staging hermes-agent-stg-automated-perception-5054: an @mention in #general now produces a visible bot reply — full multi-tenant relay round-trip (real Discord -> shared connector bot -> tenant routing -> agent WS -> reply egress -> Discord). Tests: _capture_scope/_with_scope reattach, no-scope no-op, explicit-guild_id preserved (test_relay_adapter.py). Full relay + config suites green (160 passed). Relay-adapter lane. EXPERIMENTAL. |
||
|
Ben Barclay
|
d2c53ff558 |
feat(relay): WS-only inbound on the gateway adapter (Phase 3) (#48294)
The connector now delivers inbound (messages + interrupts) over the gateway's OUTBOUND /relay WebSocket, not a signed HTTP POST to an inbound endpoint. The gateway needs no inbound HTTP port — which is what makes hosted gateways (no public IP) able to receive inbound at all. - gateway/relay/adapter.py: connect() wires set_interrupt_inbound_handler( self.on_interrupt) so connector->gateway interrupt_inbound frames bridge into the existing per-session interrupt path (the inbound message handler was already wired). Removed _maybe_start_inbound_receiver() + the _inbound_runner lifecycle — there is no HTTP receiver anymore. - gateway/relay/inbound_receiver.py: deleted (the signed-HTTP InboundDelivery receiver). - gateway/relay/__init__.py: removed relay_inbound_config() (dead with the receiver gone). The delivery key is still set in-process by self-provision for forward-compat but is no longer consumed for inbound. - docs/relay-connector-contract.md: §3 rewritten — inbound is the WS back-channel routed cross-instance via the connector's relay bus; §5 interrupt + §6 auth table updated; the old signed-HTTP-POST + per-tenant-delivery-key-signing path is documented as superseded. gatewayEndpoint noted as passthrough-plane only. Tests: stub_connector grows set_interrupt_inbound_handler + push_interrupt; new test_relay_interrupt case proves connect() wires BOTH inbound handlers and an interrupt_inbound frame over the WS cancels the right session. Removed the HTTP-receiver test; updated the crypto-shedding scan + self-provision delivery-key assertion. 88 relay tests pass. EXPERIMENTAL. Pairs with gateway-gateway (relay bus + WsGatewayDelivery) and the NAS GATEWAY_RELAY_URL stamp. The cross-repo E2E (connector repo) proves the full multi-instance path against this production adapter code. |
||
|
Ben Barclay
|
c276b017ad |
feat(relay): connector⇄gateway channel auth + signed-HTTP inbound receiver + enroll CLI (#48147)
* feat(relay): authenticate the connector⇄gateway WS channel
The relay gateway may be customer-managed and internet-exposed, so the
connector⇄gateway channel is itself authenticated (distinct from the
platform crypto the relay path sheds). Add gateway/relay/auth.py — a
Python port of the connector's HMAC token + delivery-signature schemes
(relayAuthToken.ts / deliverySigning.ts), verified byte-for-byte against
the connector's compiled TypeScript via cross-language test vectors.
Present an Authorization bearer on the /relay WS upgrade keyed by the
per-gateway secret (resolved from GATEWAY_RELAY_ID / GATEWAY_RELAY_SECRET
in env or config). The connector rejects an unauthenticated/invalid/
revoked upgrade with close 4401.
* feat(relay): signed-HTTP inbound delivery receiver
The connector delivers normalized inbound events to a tenant's gateway
over a signed HTTP POST, not the outbound /relay WS: the connector
instance owning a platform socket is generally not the instance a given
gateway dialed out to, so inbound targets a tenant endpoint that may
load-balance across gateway instances.
Add gateway/relay/inbound_receiver.py — verifies x-relay-signature /
x-relay-timestamp over the EXACT raw request bytes (re-serializing would
break the HMAC: JS JSON.stringify is compact, Python json.dumps spaces)
against the per-tenant delivery key verify list within a 300s replay
window, then dispatches messages to handle_message and interrupts to the
interrupt handler. Wire it into the adapter lifecycle (start in connect()
when a delivery key + bind port are configured, tear down in disconnect();
a purely-outbound dev gateway runs without it).
Refine test_relay_sheds_crypto to distinguish PLATFORM crypto (Discord
ed25519, Twilio/WeCom HMAC — still shed) from the connector⇄gateway
CHANNEL auth (intended): auth.py / inbound_receiver.py are exempt from
the platform-symbol scan but still banned from importing platform-crypto
modules, plus a positive guard that auth.py uses only stdlib hmac/hashlib.
* feat(relay): hermes gateway enroll CLI
Add the gateway half of zero-touch enrollment. `hermes gateway enroll`
resolves a fresh Nous Portal access token (the tenant-proving identity),
POSTs {enrollmentToken, gatewayId} to the connector's /relay/enroll, and
persists GATEWAY_RELAY_ID / GATEWAY_RELAY_SECRET / GATEWAY_RELAY_DELIVERY_KEY
to ~/.hermes/.env. The per-gateway secret authenticates the WS upgrade;
the per-tenant delivery key verifies signed inbound deliveries.
Refuses under is_managed() (hosted installs get the secret stamped in by
the orchestrator). Added as an 'enroll' subcommand on the existing
gateway subparser — not a new top-level command.
* docs(relay): inbound is signed HTTP, not WS; document channel auth
Fix the stale contract: §3/§5 said inbound rode the WS socket (single-
instance only, predates the multi-instance socket-ownership + channel-auth
model). Inbound + connector→gateway interrupt are signed HTTP POSTs to the
tenant endpoint. Add §6.1 documenting the two channel-auth schemes (per-
gateway WS-upgrade secret, per-tenant inbound delivery key) and how they
differ from the platform crypto the relay path sheds.
* test(relay): update build_gateway_parser callers for cmd_gateway_enroll
The enroll subcommand added cmd_gateway_enroll as a required keyword-only
arg to build_gateway_parser, but two existing parser-extraction tests still
called it with only cmd_gateway/cmd_proxy — failing CI with TypeError.
Thread the new handler through both call sites and add a test asserting
`gateway enroll` dispatches to cmd_gateway_enroll with its flags parsed.
|
||
|
Ben
|
6b03874d07 |
feat(gateway): production WebSocketRelayTransport + descriptor negotiation
Adds the concrete transport behind the RelayTransport Protocol — the missing
'later-phase work' the relay scaffold deferred. The gateway dials OUT to the
connector over a WebSocket and speaks the newline-delimited JSON frame protocol
(docs/relay-connector-contract.md; connector src/relay/protocol.ts):
- connect(): opens the ws, sends hello{platform,botId}, starts a background
read loop, and resolves handshake() when the connector's descriptor frame
arrives.
- inbound frames -> the registered InboundHandler (rebuilt into a MessageEvent
via _event_from_wire, mapping the snake_case SessionSource wire form back
onto the gateway dataclasses).
- send_outbound / send_follow_up / get_chat_info: request/response correlated
by a uuid requestId against a per-request future, with a timeout so a caller
never hangs; send_interrupt is fire-and-forget.
- disconnect(): cancels the reader, closes the ws, and fails any in-flight
outbound waiters with a structured error.
RelayAdapter.connect() now negotiates the real CapabilityDescriptor from the
transport and adopts it (_apply_descriptor updates MAX_MESSAGE_LENGTH +
markdown surface), replacing the construction-time placeholder. Lazy
'import websockets' mirrors gateway/platforms/feishu.py; WEBSOCKETS_AVAILABLE
gates construction.
|
||
|
Ben
|
3db9b3e616 |
feat(gateway): token-less follow_up outbound op (A2 capability action)
The relay outbound surface had send/edit/typing but no way to act on a SHARED-identity capability (e.g. a Discord interaction follow-up token, ~15min) that the connector captured + stripped at the edge. Under A2 that credential never reaches the gateway, so the gateway can't just 'send with the token' — it needs a semantic op naming the session it's already in. Adds the follow_up op end to end on the gateway side: - RelayTransport.send_follow_up(action): protocol method. Action carries op='follow_up' + session_key + kind + content (+ metadata) and NO token. - RelayAdapter.send_follow_up(session_key, kind, content, metadata): builds that action and returns a SendResult. The connector resolves the real capability (its resolveOutboundCapability), enforces the tenant match so tenant B can't wield tenant A's capability, and egresses; success=False when the capability is absent/expired/mismatched (nothing to retry — a leaked gateway holds zero capability material). - StubConnector records follow_ups + a canned next_follow_up_result. Tests: round-trips without a token; the wire action carries only session refs (no credential value field — the 'kind' string is a type ref, not the secret); failure surfaces when the connector can't resolve; no-transport fails cleanly. 55 passed. §4 doc entry follows in the contract-rewrite commit. |
||
|
Ben
|
a3cdd8c39d |
feat(relay): route mid-turn /stop over relay interrupt channel
RelayAdapter.on_interrupt(session_key, chat_id) bridges a connector-delivered mid-turn /stop into the existing interrupt_session_activity path, setting the per-session _active_sessions Event and clearing typing — cancelling exactly the targeted session's turn without touching siblings (mirrors test_stop_thread_ sibling isolation). Transport.send_interrupt carries the gateway-side egress to the connector for socket-owner routing. Phase 1, Task 1.4 of the gateway-relay plan. |
||
|
Ben
|
259e78e175 |
feat(relay): transport protocol + test-only stub connector
Defines RelayTransport (lifecycle/handshake/inbound/outbound/interrupt) as the gateway<->connector wire contract; RelayAdapter.connect now registers an inbound handler that bridges connector-delivered MessageEvents into handle_message. Adds an in-memory StubConnector under tests/ and an E2E round-trip proving: connect registers the handler, inbound events reach the adapter, guild_id drives build_session_key isolation (two guilds -> two keys; same guild/channel/user -> one), outbound send round-trips, get_chat_info is proxied. Phase 1, Task 1.2 of the gateway-relay plan. |
||
|
Ben
|
b0999c82f3 |
feat(relay): generic RelayAdapter advertising negotiated capabilities
One BasePlatformAdapter subclass that reads its capability profile from a CapabilityDescriptor: MAX_MESSAGE_LENGTH attribute, message_len_fn (table-driven by len_unit: chars=len, utf16=Telegram-style code units), supports_draft_streaming. Implements the four abstract methods (connect/disconnect/send/get_chat_info) by delegating to an injected RelayTransport (full protocol lands in Task 1.2). Adds Platform.RELAY enum member. No per-platform gateway code. Phase 1, Task 1.1 of the gateway-relay plan. |