cf60ebbdfd264c3fc067b9e3230df8752c50d2ec
3662 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 abundantbeing
|
97e32d49ac |
fix(api): hide compaction scaffolding from clients
Project client-visible session messages through the canonical compaction classifier. Hide standalone handoffs, unwrap merged carriers to their authentic prior-tail content, strip inherited internal fields, and keep model-facing recovery history unchanged. |
||
|
IAvecilla
|
6a9903fb5f | Avoid unnecessary sleep checks for azure sandboxes | ||
|
poisdahl
|
abf87e7248 | Merge current main into composite-carrier fix | ||
|
Teknium
|
216c98aaed |
fix(gateway): scope draft-final finalize skip to fresh persistent sends
The salvaged skip condition keyed on _use_draft_streaming alone, which also suppressed the explicit REQUIRES_EDIT_FINALIZE pass when a draft-streaming run had degraded to edit-based delivery (draft failure fallback sets _message_id). Key the skip on the got_done update being a fresh persistent send through the native-draft transport (_message_id is None), which is the only case where the update already carried its own finalize. |
||
|
Gille
|
790c850144 | fix(telegram): preserve rich finals after DM drafts | ||
|
Teknium
|
1d74833d8d |
feat(update): structured update receipts + post-update fleet version verification
Phase 1 of the fleet-update reliability plan (#91277): the updater now proves its outcome instead of assuming it. - hermes_cli/build_info.py: get_code_identity() — process-cached code identity (git sha for source installs, baked .hermes_build_sha for Docker images, pyproject version). - gateway/status.py: every runtime-status write stamps the writer's code_sha/code_version into gateway_state.json, so a running gateway's actual code generation is observable from disk. - hermes_cli/update_receipt.py (new): machine-readable receipt of each update run (steps, skips with reasons, gateway restart outcome, fleet snapshot) under ~/.hermes/logs/update_receipts/ with a latest.json pointer for the dashboard/desktop; plus collect_fleet_versions() / print_fleet_version_matrix() comparing every live profile gateway against the freshly updated checkout. - hermes_cli/update_cmd.py: wires receipt begin/steps/finalize into the git, ZIP, and hard-failure paths; after the restart phase, prints the fleet version matrix and escalates provably-stale gateways into the existing gateway_fleet_restart_incomplete exit-1 contract. Pre-stamp gateways report 'unknown' and never fail the update (no false positives during rollout). Silent-failure classes made visible: #88848, #74973, #85753, #81193. Mixed-version fleet classes made loud: #88654, #69754, #77553, #56717. |
||
|
Ben Barclay
|
efb6b40f94 |
Merge pull request #91237 from NousResearch/fix/relay-env-exclusive-messaging
fix(gateway): GATEWAY_RELAY_URL env stamp disables direct messaging platforms |
||
|
Ben Barclay
|
952d1cef18 |
fix(gateway): prompt-send failures keep their diagnostic detail; clarify caller path gets contract tests
Two review nits on the clarify ambiguity fix: - _approval_send_outcome swallowed the failure detail the old inline callers logged (scheduling exception text / SendResult error). Both the approval and clarify lanes now share one warning with the detail, logged in the classifier itself. - The clarify tests pinned the disposition helper but nothing proved the ambiguous branch actually reaches the bounded wait. The send-then-wait sequence is extracted to _clarify_send_then_wait (the callback closure now just binds context onto it) and the suite gains caller-path tests: ambiguous/sent -> wait_for_response with the generated clarify_id and configured timeout; definitive failure -> sentinel without waiting; no-response timeout sentinel preserved; plus caplog assertions that failed sends log their detail. Relay + gateway sweep 289/289. |
||
|
Ben Barclay
|
a7cfe391f7 |
fix(gateway): clarify prompt sends get the same ambiguous-timeout handling as approvals
The clarify caller treated a send-scheduling timeout as a definitive failure: clear_session() + '[clarify prompt could not be delivered]'. Same physics as the approval card fixed earlier in this PR — the card may well have posted with a late connector ack — so the teardown ran out from under a rendered clarify card and the user's answer resolved nothing. New _clarify_send_disposition() routes the outcome through _approval_send_outcome: only a DEFINITIVE failure (error result, non-timeout exception, no future) clears the registration and aborts; ambiguous logs a warning and falls through to wait_for_response, whose existing bounded wait already handles the truly-lost-card case. This makes the boundary rule stated in the ambiguity test docstring hold for the clarify lane, not just approvals. Tests: 5 disposition tests mirroring the approval suite, including clear_session-not-called on timeout. Mutation-verified: folding ambiguous into the failed branch sends test_timeout_keeps_registration_armed_and_proceeds_to_wait red. Relay + gateway sweep 283/283. |
||
|
Victor Kyriazakos
|
424d07edac |
fix(relay): prompt-lifecycle acks are fire-and-forget — awaiting them ON the read loop self-deadlocked the transport
Round 2 of the approval-turn stuck-stream hunt. Round 1 (interim-marked
acks) fixed the draft-hijack-by-matching path — live logs confirm the
absorption fallback no longer fires — but the freeze persisted because
of a second, deeper defect on the same codepath:
_consume_prompt_response executes ON the transport read loop (inbound
frame -> _handle_frame -> _inbound handler). The handler awaited
self.send() for its '✅ Approved once' ack — but send() blocks on an
outbound_result future that ONLY the read loop can resolve, and the
read loop is blocked inside this very handler. Guaranteed self-deadlock
for the full outbound timeout (30s) on EVERY button tap. While wedged,
everything on the transport starved: draft appends (the frozen stream
right after approving), sibling approval-card sends (timed out into
'possibly-delivered' — the observed double-approval ambiguity), and the
turn's seal (timed out ambiguous -> plain-send fallback -> duplicate
final). Log signature was the tell: card-send timeout at tap time, no
absorption INFO, no seal-failed WARNING, no suppression line.
Fix: _send_lifecycle_ack() — acks ride a background task with strong
ref retention; the handler returns immediately and the read loop keeps
consuming, so the ack's own result frame resolves normally. Applied to
all six lifecycle sends (approval ack, slash-confirm ack + result text,
clarify acks, expiry notice). Acks are cosmetic by contract; failure
logs at debug and never breaks the reader.
Tests: new deadlock-shape test (gated transport send; handler must
return within 1s and the ack must still egress afterwards — RED on the
awaited version via TimeoutError at the exact deadlock), prior 3 tests
green with a yield for the background task. Targeted sweep 203/203.
|
||
|
Victor Kyriazakos
|
57fe01e367 |
fix(relay): prompt-lifecycle acks are interim sends — the approval ack was sealing the turn's own draft stream
Live finding (rc.4 staging, 100% reproducible on approval turns): after
resolving an exec-approval prompt_response, the adapter sends a short
ack ('Approved once'). _prompt_reply_metadata carried only placement
metadata (thread_id) — no per-turn identity, no interim marker — so
send()'s single-open-stream fallback (review B2) matched the approval
turn's OWN live draft and sealed it with the ack text. From there,
silently: every later append died on the post-seal tombstone (built for
millisecond stragglers, deliberately quiet), freezing the visible draft
mid-word; the turn-final found no open draft and fell through to a
plain send — the duplicate 'fallback' message. No suppression line, no
seal-failed warning: the log signature was pure absence.
Fix: _prompt_reply_metadata stamps _interim_send=True, which send()
already honors by bypassing draft matching. One source covers the whole
lifecycle class (approval ack, slash-confirm ack, prompt-expired
notice — all six call sites route through it).
Observability (the quiet parts, out loud):
- single-open-stream absorption now logs at INFO with the absorbed key;
- the FIRST post-seal tombstone swallow per draft key logs at WARNING
(bounded FIFO dedup) — one swallow is the normal straggler race, a
burst means a live stream was sealed mid-flight by someone else.
Tests (RED-first: both ack tests failed on the unfixed adapter at the
'draft still armed' assertion): approval ack leaves the open draft
armed and egresses as a plain send op; expiry notice same; regression
control pins the B2 contract — a real identity-less turn-final still
absorbs into its single open stream.
|
||
|
Victor Kyriazakos
|
5210dd48b8 |
fix(gateway+relay): approval prompts survive ambiguity without duplicates; streamed finals keep block formatting
Three live findings from rc.4 staging, all on the relay-fronted Slack path, all with the failure observed in live logs before the fix: 1. Approval-send timeout is AMBIGUOUS, not failed (no re-ask). send_exec_approval through the connector can time out with the card already rendered — the connector may ack after the deadline (slow platform API call, transient backpressure, event-loop stall) — and the timeout-as-failure path re-sent and produced duplicate cards. The outcome is now tri-state: sent / failed / ambiguous. Ambiguous = no re-send, no text fallback; the prompt registration stays armed so a late tap still resolves. Only a definite send error falls back to text. 2. pending_approval tool results forbid re-issuing the command. With one card correctly armed, the agent could still mint a SECOND card by re-running a rephrased variant of the gated command after reading the pending_approval tool result (observed live: same command re-issued in a different form, two cards). The tool message now instructs: do not re-run/rephrase; wait or report pending. Applied to both the terminal and execute_code arms. 3. Draft interim AND seal frames carry format_hints. format_hints are stamped on send, edit, and send_for_platform, but both draft-frame builders (send_draft interim + _seal_open_draft seal) shipped bare metadata. A streamed final therefore arrived at the connector hintless and sealed as a plain code block while non-streamed sends rendered native markdown blocks (observed live: language-tagged block on send/edit, downgrade on streamed seal). Both sites now stamp _with_format_hints_for_chat (destination-resolved, same pattern as the existing lanes). Verified live after the fix against the platform's stored message payload: rich_text_preformatted with language field on a streamed seal. Tests: tri-state outcome unit tests (5), draft/seal hint stamping + knobs- off regression control (2, RED-first), existing format-hints suite intact (14/14). Mutation-verified: reverting the adapter hunk sends test_draft_interim_and_seal_frames_carry_hints red; restore -> green. Boundary sweep (text egress lanes crossing the frame contract): send ✓ (pre-existing) edit ✓ (pre-existing) send_for_platform ✓ (pre-existing) draft-interim ✓ (this PR) draft-seal ✓ (this PR); task_card lane carries no text content — exempt. |
||
|
Ben Barclay
|
2ef095ce46 |
fix(gateway): source-neutral log wording for relay-exclusive sweep
sol-reviewer MINOR: the _enabled_explicit marker is set by config.yaml, gateway.json, and dashboard PUTs alike, so the WARNING no longer claims config.yaml specifically. Also names the opt-out env var in the message so an operator seeing the WARNING knows the escape hatch. |
||
|
Ben Barclay
|
477a0222f9 |
fix(gateway): read relay-exclusive env vars through profile secret scope
sol-reviewer IMPORTANT: the relay trigger and opt-out read os.getenv() directly, bypassing the profile secret scope that _apply_env_overrides uses everywhere else. Under a multiplexed gateway a profile-scoped GATEWAY_RELAY_URL was invisible, and a process-global one leaked into every profile, disabling direct platforms in profiles that are not relay-fronted. Both reads now go through the scope-aware getenv. Also folds the NIT: opt-out truthiness now uses the shared is_truthy_value helper instead of a local truthy tuple. |
||
|
Ben Barclay
|
2d3f5c1554 |
feat(gateway): GATEWAY_RELAY_ALLOW_DIRECT_PLATFORMS opt-out for relay-exclusive mode
Deployments that intentionally mix connector-fronted and direct ingress can set GATEWAY_RELAY_ALLOW_DIRECT_PLATFORMS=true to keep directly- connected messaging adapters enabled beside the relay. Unset, the GATEWAY_RELAY_URL env stamp keeps its exclusive behavior. Like the trigger, the opt-out is a deploy-stamp env var, not config.yaml. |
||
|
Ben Barclay
|
67292ec5b8 |
fix(gateway): GATEWAY_RELAY_URL env stamp disables direct messaging platforms
A GATEWAY_RELAY_URL set in the process environment marks a connector-fronted deployment where the connector owns every platform connection. A directly-connected messaging adapter in the same process is a second, unmanaged ingress path: it causes duplicate deliveries and split sessions, and its live socket disarms scale-to-zero. At the end of _apply_env_overrides, after all enablement passes, the env stamp now disables every other enabled messaging platform: - Explicitly-enabled platforms (config.yaml enabled: true) are disabled with a WARNING that names the platform. - Credential-auto-enabled platforms are disabled with an INFO line. - Non-messaging surfaces (local, api_server, webhook) are untouched -- the same exclusion set as the scale-to-zero arm gate. - gateway.relay_url in config.yaml alone (no env stamp) keeps the old additive behavior: relay runs beside direct adapters. |
||
|
Ben Barclay
|
48f15c1ece |
chore(gateway): drop the default scale-to-zero idle timeout to 2 minutes
With the gateway owning the suspend, the idle predicate covers every work source (agent turns, cron jobs, API-server runs, background work, fail-awake on unreadable sources) and the relay drains + flips before the freeze, so a long timeout no longer buys safety — real work always blocks the suspend and resume is sub-second. 5 idle minutes just bills idle RAM. Per-instance override stays config.yaml gateway.scale_to_zero.idle_timeout_minutes (D2). New behavior-contract test: invalid config values degrade to the module default (whatever it is), never zero/negative — asserts the RELATION, not the literal, per the no-change-detector-tests rule. |
||
|
Ben Barclay
|
ee000768ce |
Merge pull request #90761 from NousResearch/fix/scale-to-zero-cron-aware-idle
fix(gateway): count cron and API-server work in the scale-to-zero idle predicate |
||
|
Ben Barclay
|
0215930526 |
review: fail-awake work accounting + rename is_idle param to active_work_count
Address sol-reviewer findings: - The shared shutdown-drain counters swallow exceptions to 0 — fine for a drain, unsafe for a suspend predicate (a transient read failure made live work look idle, reopening the mid-job freeze). The suspend path now reads both sources itself and treats an unreadable source as work (sentinel 1, fail-awake) with a debug log. A MISSING api_server adapter remains a normal not-work state. - is_idle()'s parameter renamed running_agent_count -> active_work_count: it receives the broad aggregate, and the old name invited future callers to pass only agents again. - New failure-path tests: unreadable cron source and unreadable API source each hold the machine awake (both fail against the fail-open shape); missing adapter stays idle-capable. |
||
|
Chris Fontes
|
64505a2b83 |
fix(resolve_turn_limit): gateway bridge null handling, TUI resolver, docs
Addresses teknium1 sweeper review on PR #67696: 1. Gateway bridge: Skip str(None) bridging when YAML value is Python None (from or bare ). Previously str(None) → None → unlimited instead of default 90. Now clears stale env var so resolver applies default. 2. TUI: Route _cfg_max_turns through resolve_turn_limit instead of bare int(). Old code crashed on none/unlimited and swallowed 0 via . HERMES_TUI_MAX_TURNS env var also routed through resolver. 3. Docs: Document unlimited spellings (none/unlimited/infinite/0/-1) in configuration.md. 4. Tests: Add TestGatewayBridgeNullHandling (4 tests) and TestTUIResolver (8 tests) covering null handling, string spellings, env var override, and legacy root-level config. All 50 tests pass. |
||
|
Chris Fontes
|
5046282867 |
feat(config): resolve_turn_limit — first-class 'none'/'unlimited' for agent.max_turns
Previously agent.max_turns only accepted positive integers. Setting it to
'none', 'unlimited', or 0 — all natural ways to say 'no limit' — either
crashed int() or was silently skipped by `or` checks, falling back to 90.
This adds resolve_turn_limit() in hermes_cli/config.py as the single
normalization point. It accepts:
- int/float → int(raw) (floats truncated)
- numeric string ('120') → int(raw)
- 'none'/'unlimited'/'infinite'/'∞'/'-1'/'0' (case-insensitive,
whitespace-tolerant) → sys.maxsize sentinel
- YAML None/null → default (90)
- bool/list/dict/garbage → default (with debug log)
All config-reading sites (cli.py, gateway/run.py, cron/scheduler.py) now
call this instead of bare int(), so agent.max_turns: none in config.yaml
becomes a first-class supported spelling of 'unlimited'.
The sentinel (sys.maxsize) survives the str()→int() round-trip through
the HERMES_MAX_ITERATIONS env-var bridge in gateway/run.py and works in
every <, >=, remaining = max - used comparison without requiring call
sites to learn about a special value.
Includes 38 tests covering the full spelling table, the str→int env-var
round-trip, and sentinel properties.
|
||
|
Teknium
|
95fa814269 |
feat(process): positive process identity — spawn tags, machine spawn ledger, Windows job-object self-attach
Every long-lived Hermes process is now positively identifiable so reapers never have to guess lineage from PPID archaeology or cmdline shape: - hermes_cli/process_identity.py (new): HERMES_SPAWN tag build/parse, spawn-ledger.json self-registration keyed on (pid, create_time) — PID reuse cannot forge the pair — with #89298-style corrupt-file quarantine, and a kill-on-close job-object self-attach (BREAKAWAY_OK preserved for the existing CREATE_BREAKAWAY_FROM_JOB escape hatches). - serve/dashboard (web_server.py) and the gateway entry point register themselves at startup and attach to the job; Desktop legacy HERMES_PARENT_PID/winms marker reused as spawner identity so lineage works with every Desktop version. - Desktop stamps HERMES_SPAWN on backend spawns (parent-process-identity.ts). - hermes update gets a positive-identity rung ahead of the heuristic ones: _ledger_reapable_backend_pids reaps holders the ledger PROVES are orphaned backends (purpose reapable + recorded spawner provably dead) in ANY update context. Ledger-unknown holders fall through to the existing rungs. 22 new tests, sabotage-verified. |
||
|
Ben Barclay
|
a1a1bee5c2 |
Merge branch 'main' into feat/relay-slack-parity
One conflict, gateway/relay/adapter.py send_for_platform: main added the turn-final draft-seal interception (_sfp_metadata with the _interim_send marker stripped, seal-or-fall-through); this branch added format-hint stamping on the same frame. COMPOSED: the plain-send frame now stamps _with_format_hints_for_platform over _sfp_metadata (the stripped copy), so both the seal fall-through contract and the cron-lane block hints hold. Note: the seal frame itself (op:draft final) does not stamp hints — cron sends are never open drafts, so the flagship path is unaffected; noted as a connector-PR follow-up for streamed interactive finals. |
||
|
Ben Barclay
|
743dc935f5 |
fix(gateway): count cron and API-server work in the scale-to-zero idle predicate
_scale_to_zero_is_idle() consumed _running_agent_count(), but cron jobs run through a standalone AIAgent on the scheduler's own thread pool and API-server runs live on the adapter — both outside _running_agents (the same blind spot the #60432 shutdown-drain fix addressed with _active_work_count()). The idle predicate therefore read True DURING a running cron job; a suspend at that moment freezes the job mid-flight. Observed live on staging 2026-08-20: is_idle held True throughout the 10:45:04-22 cron run — only watcher-tick timing (next tick 9s after completion) avoided a mid-job freeze. Use _active_work_count() (agents + cron + API runs). New tests cover a running cron job and an active API run each blocking idle, plus the all-quiet True case; both blocking tests fail without the fix. |
||
|
pierrenode
|
a1ddb54840 |
fix(gateway): tag the loop-liveness and heartbeat-poll tasks as permanent supervised watchers (#84558)
#84327 excluded _spawn_supervised's permanent watchers (session-expiry, kanban, reconnect, the scale-to-zero watcher itself, ...) from _scale_to_zero_has_live_background_work() via a _hermes_supervised_watcher tag, because counting them made an armed gateway consider itself busy forever and never go dormant. Two more permanent, infinite-loop tasks are added to _background_tasks OUTSIDE _spawn_supervised and were untagged: - _loop_heartbeat_task (loop_heartbeat_forever, #66892): a `while True` loop started unconditionally in start() on every gateway boot. Extracted the inline spawn block into _start_loop_heartbeat_task() so it's independently testable, matching the existing _start_heartbeat_poller() pattern. - _heartbeat_poll_task (_poll_loop in _start_heartbeat_poller): also a `while True` loop, started the first time a session registers a heartbeat watch, and then permanent for the rest of the process. Because _loop_heartbeat_task starts on every boot, it alone made _scale_to_zero_has_live_background_work() return True forever on every armed instance, regardless of the #84327 fix -- confirmed empirically against the real method with the exact untagged-task shape this task has. Two new regression tests spawn each task through its real production entry point and assert the busy check returns False; both fail against the unfixed code (missing method / real assertion failure). Co-authored-by: pierrenode <298902573+pierrenode@users.noreply.github.com> Co-authored-by: Ben Barclay <ben@nousresearch.com> |
||
|
Ben Barclay
|
162b23c3e2 |
fix(relay): D6 in_channel capability gate resolves the destination platform's descriptor
RelayAdapter.supports_inchannel_continuable is a scalar adopted from the PRIMARY identity's handshake descriptor, but one RelayAdapter fronts N platforms and the connector advertises the bit per platform. Reading the scalar for every logical platform both leaked a Slack-primary True onto other fronted platforms (activating the flat surface their descriptor never advertised) and suppressed a non-primary platform's advertised True (forcing thread mode on capable Slack behind a Discord primary). Add supports_inchannel_continuable_for_platform(platform): resolves the platform's own negotiated descriptor via descriptor_for_platform (the same Phase 1.5 seam max_message_length uses), scalar fallback only when the per-platform descriptor is unavailable. The scheduler's D6 gate prefers the query when the adapter provides it; native adapters keep the class-attribute path byte-identically. Tests: two-platform descriptor matrix (primary-True no-leak, non-primary-True honored, unknown-platform scalar fallback). |
||
|
Ben Barclay
|
79c39025c0 |
fix(relay): format hints resolve the DESTINATION platform, and stamp on send_for_platform
Two gaps in the block-formatting hint stamping: 1. Wrong descriptor: _format_hints gated on self.descriptor — the PRIMARY identity's scalar — while one RelayAdapter fronts N platforms. A Slack-primary adapter stamped Slack hints onto known Discord chats; a Discord-primary adapter suppressed hints for Slack chats whose own negotiated descriptor advertised the bit. Resolve per destination: send/edit use _descriptor_for_chat (the same seam max_message_length already uses) plus the chat's logical platform for the config sub-block; the knob lookup is now per-logical-platform (platforms.relay.extra.<platform>.*) instead of hardwired to slack. 2. Missing lane: send_for_platform — the scheduled/persisted-home lane (gateway/delivery.py), i.e. the CRON delivery path, the flagship consumer of the in_channel brief — never stamped hints at all. Stamp there too, resolving descriptor_for_platform(logical) off the transport; the scalar descriptor is used only when it belongs to that exact platform (fail closed). Tests: Slack-primary/Discord-chat no-leak, Discord-primary/Slack-chat still-stamps, send_for_platform stamps for capable platform and stays clean for incapable — all against a two-platform negotiated-descriptor transport. Existing single-platform suite unchanged and green. |
||
|
Ben Barclay
|
5a17b1f41d |
Merge pull request #89584 from victor-kyriazakos/relay-ws-hardening
fix(relay): rc.4 relay transport + inbound fixes — dedupe replays, fail pending on drop, fail fast mid-redial, WAN keepalive |
||
|
Teknium
|
2d59cb4386 |
fix(api-server): 'max' and 'ultra' reasoning efforts are no longer silently ignored on API/browser requests
_request_reasoning_config() whitelisted none..xhigh, so a client sending max or ultra (valid /reasoning + config.yaml levels) fell through to the default effort with no error. The server now accepts the full internal ladder (hermes_constants.VALID_REASONING_EFFORTS); per-provider wire clamping happens downstream via agent.reasoning_effort, same as every other entry surface. Salvages the api_server hunk of #78216 (credit @snowzlmbot); the un-clamping half of that PR was rejected separately. |
||
|
Ben Barclay
|
141af4febf |
Merge branch 'main' into relay-ws-hardening
#85796 (live-cards gateway half) landed on main and touches the same two files. Resolutions: - adapter.py __init__: union — the dedupe seen-set and the live-cards draft/seal caches are independent sibling attributes. - ws_transport.py _request_response: keep main's ambiguous-timeout contract AND this branch's raising-write catch, composed: a raise from the WRITE itself means the frame never reached the wire (definite non-delivery, no flag), while a failure surfaced after the frame was sent carries ambiguous=True like the timeout — tracked via a frame_sent marker. |
||
|
victor-kyriazakos
|
3683e70043 |
feat(relay): live-card ops — native draft streaming + task cards over the relay (gateway half) (#85796)
* feat(relay): live-card ops — native draft streaming + task cards over the relay (gateway half)
NS-658. Three additive ops within contract v1, emitted only when the
connector's negotiated descriptor advertises them:
{op: draft, chat_id, draft_id, content, final, metadata}
{op: task_card, chat_id, card_id, chunks, metadata}
{op: task_card_stop, chat_id, card_id, metadata}
The gateway side is deliberately dumb: no platform API knowledge, no new
config keys. Slack mechanics (chat.startStream/appendStream/stopStream,
per-workspace feature-gate cache, send+edit fallback) live connector-side
where the platform adapter lives in the relay model.
Semantic bridge: base send_draft is Telegram-shaped (draft clears; final
is a separate send). Slack native streaming makes the stream THE message.
The adapter tracks the open draft per chat and converts the turn-final
send() into draft(final=true) so the connector seals the stream instead
of posting a duplicate; the stream ts returns as the message identity.
A failed frame disarms interception so the edit-based fallback's real
send goes through untouched.
BEHAVIOR CHANGE (deliberate): relay supports_draft_streaming() now
requires the descriptor flag AND the draft op. Flag-only was a latent
lie — send_draft inherited NotImplementedError, so a connector setting
the flag without the op would have crashed the stream consumer's draft
path. supported_ops stays fail-open for legacy (pre-contract) ops;
draft/task_card did not exist pre-contract and must not fail open.
Task cards ride #85476's adapter-agnostic TurnRunner seam (hasattr on
send_native_task_card_progress); supports_native_task_cards() is the
descriptor probe. Connector half + E2E harness pair follow in the gg
repo.
* fix(relay): expose native_task_cards_enabled() on the relay adapter
Live-canary finding (Alice, staging): the TurnRunner's task-card lane
probes adapter.native_task_cards_enabled() (the native Slack adapter's
opt-in contract). The relay adapter only offered
supports_native_task_cards(), so the hasattr gate failed silently and
tool progress stayed on the text path — draft streaming worked, cards
never rendered. Alias it to the descriptor probe.
* fix(relay): match task-card methods to the TurnRunner's native keyword contract
Live-canary finding #2 (Alice, staging): gateway/run.py's card lane calls
send/stop_native_task_card_progress with the NATIVE Slack adapter's
signature (tasks/title/reply_to/metadata/fallback_text, keyword-only) —
PR 85796's relay methods took a positional card_id, so every call raised
TypeError('unexpected keyword argument reply_to') in the progress task,
repeatedly killing the card publisher (and the retry loop resent the
final delivery 4-5x). Card id now derives per turn thread
(turn:<reply_to>), thread_ts anchored like draft; title/fallback_text
accepted for parity, not forwarded (plan-mode stream renders chunks).
* fix(relay): one draft stream per turn for stream-is-the-message adapters
Live-canary finding #4 (Alice, staging): the stream consumer bumps
draft_id at every tool boundary so Telegram-shaped drafts animate each
text segment as a fresh preview. On relay Slack NATIVE streaming a new
draft_id opens a brand-new chat.startStream — the user saw one frozen
message per segment (stuck streaming cursor ▉, never sealed: only the
LAST stream gets the final=true seal) plus the real final; 5-6 cumulative
snapshots per turn. Adapters that mark draft_stream_is_message keep ONE
stream per turn: tool progress lives in the native task card, and the
connector's suffix-delta falls back to whole-text append on prefix
mismatch, so segments append cleanly. Telegram-shaped drafts keep the
per-segment bump.
* fix(relay): don't seal the native stream at tool boundaries — only the turn-final does
Live-canary finding #5 (Alice; supersedes the incomplete #4 which was
necessary but not sufficient). Root cause CONFIRMED by integration trace
(test_live_cards_flow_trace.py, real consumer semantics + real adapter +
stub transport): at every tool boundary the consumer calls
_send_or_edit(finalize=True), which skips the draft path and issues a
real send(); the relay adapter's seal-interception converts THAT into
draft(final=true) — sealing the stream once per segment. Timeline showed
3 seals for a 3-segment turn: exactly the frozen cumulative ▉ snapshots
seen live (the replaced stream never gets stopStream, keeping its cursor).
Fix: for draft_stream_is_message adapters, a segment-break finalize
(finalize=True, is_turn_final=False) stays ON the draft path as another
cumulative frame; only got_done (is_turn_final=True) falls through to
send() and seals. Telegram-shaped platforms unchanged. Trace test now
pins the invariant: ONE user-visible message per turn.
* fix(relay): strip the text cursor from native draft frames
Live-canary finding #6 (Alice) — the ACTUAL duplicate-content mechanism,
confirmed by full-flow scan of both sides' code + logs. The consumer
appends its text cursor (▉) to every non-final display_text tick. The
connector's stream sender diffs CUMULATIVE frames via prefix check:
'abc▉'.startsWith → 'abc def▉' is NEVER a prefix match (the cursor sits
mid-string), so deltaFor falls back to whole-text append on EVERY tick —
chat.appendStream stacks each full cumulative snapshot (cursor included)
into the ONE stream message. Exactly the observed thread: repeated
blocks, each ending in a frozen ▉, growing per tick.
Fixes #4/#5 were real (one stream per turn now) but this was the last
mechanism standing. Native streams render their own typing indicator, so
the text cursor is pure noise on this path: strip it from draft frames.
Prefix check now holds; every tick appends only its true suffix delta.
* fix(relay): seal-interception covers EVERY egress door, not just send()
Live-canary finding #7 (Alice): one duplication remained after #6 — the
stream froze mid-word with the live indicator (never sealed) and the
final posted as a separate message. Log receipt: 'Queued follow-up:
final text delivery confirmed; delivering explicit media before
continuing' — the turn's final went out via the DELIVERY RESOLVER lane
(gateway/delivery.py), which calls send_for_platform() DIRECTLY,
bypassing send() and its seal-interception. The open stream never
absorbed the final; it arrived as a plain 'send' op → chat.postMessage.
Fix: hoist the open-draft check to the top of send() (ahead of the
explicit-platform branch) AND add it to send_for_platform() — an open
native stream absorbs the turn-final regardless of which egress door it
arrives through. The stream IS the message.
* fix(relay): failed seal falls back to plain send (PR 85796 AI-review point 1)
A turn-final seal that fails at the transport must never swallow the
final answer: the stream consumer has already disabled the draft
transport for the run, so a failed _seal_open_draft returning
success=False meant the user got NOTHING. Both seal-interception sites
(send + send_for_platform) now fall through to the regular plain-send
path on seal failure, with a warning receipt. Also mitigates AI-review
point 2 (sticky _open_draft_by_chat after an abandoned turn): a stale
entry's failed seal no longer blocks the next turn's delivery.
* fix(relay): arm seal-interception optimistically; never disarm on ambiguous failure (audit G-D1)
Deep-audit defect G-D1 (HIGH): the outbound leg is at-most-once on the
wire but its ack channel is lossy — send_outbound timeout (30s) and
WS-drop 'failures' frequently mean the frame WAS delivered and the
connector stream is open. send_draft popped _open_draft_by_chat on any
failure, disarming seal-interception while the connector stream lived:
the turn-final went out as a plain send → orphaned mid-word stream +
complete duplicate final (intermittent; needs a drop/timeout inside the
draft window).
Fix: arm the entry BEFORE the transport call and keep it armed on
failure/exception. Safe in every case: sealing a non-existent stream
opens+seals a single complete message connector-side, and a truly failed
seal already falls back to plain send at both interception sites.
Stale-entry damage is self-healing (one warning + plain send).
* fix(relay): gateway-side sealed-draft tombstone — G-D1 arming must not resurrect sealed streams
Regression fix on G-D1 (live: 'worse than before' — escalating frozen
prefixes). Optimistic arming had no seal-awareness: a straggler frame
arriving AFTER the seal re-armed _open_draft_by_chat for the already-
sealed draft_id; the next send was converted to draft(final=true) on the
tombstoned connector key, which CLEARED the connector tombstone (final
frame = new-turn signal), re-opened a stream with cumulative content,
and left it frozen — repeating per straggler: 4-5 escalating frozen
snapshots. Mirror the connector: _sealed_draft_by_chat records the
sealed draft_id per chat (tombstoned BEFORE the seal's transport call);
send_draft for a sealed draft_id is a success no-op (content already in
the sealed message) and never arms. A new turn's fresh draft_id arms
normally.
* fix(relay): key stream/card state per (chat, turn anchor) — parallel turns must not collide (finding #10)
Live finding #10 (Alice; three concurrent turns in one flat DM): all
coordination state was keyed per CHAT on a one-active-turn assumption.
Three parallel turns produced: turn B's task card merged into turn A's
(both were card 'turn:root' — reply_to is None in flat DMs), B left
cardless, and _open/_sealed_draft_by_chat clobbered across writers (3x
duplicate finals on the last turn). Per-turn machinery was correct;
the keys were not.
Fix: _draft_key(chat, metadata) = chat + the turn's thread anchor
(inbound stamps thread_ts = event.thread_ts or ts on every top-level
message, so each turn has one even in flat DMs). draft arming, seal
tombstones, both interception sites, and the task-card id all derive
from the same anchor. New trace test pins two interleaved turns:
distinct cards, own-stream seals, no leaked plain send, no cross-turn
tombstone drops (289 tests green).
* fix(gateway): preserve cumulative native stream across tools
* fix(gateway): consumer-declared final — the seal carries the true final
Three composed fixes for the Slack live-cards duplicate-final class:
1. finish(final_text): TurnRunner passes the completed final_response
(verifier footer, completion explainer included) as the authoritative
finalize payload. The native-stream seal delivers the TRUE final, so
post-stream mutation no longer forks a corrective plain send (#11).
2. Interim-send contract: commentary and segment-tail sends carry a
gateway-internal _interim_send marker; relay seal-interception skips
them at both egress doors. A mid-turn interim send can no longer seal
the live stream and orphan the real final into a duplicate.
3. Queued-follow-up lane reconciles an unconfirmed final by EDITING the
consumer's delivered message in place (sealed stream = regular
message, chat.update live-verified); plain send only as fallback.
This was the actual duplicate lane in the parallel canaries — every
duplicated turn logged 'final stream delivery not confirmed; sending
first response' (subagent-completion queued inbound), not parallelism.
Also: draft frames stay prefix-stable gateway-side (no fence-closing, no
segment state reset, no commentary reset for stream-is-the-message
adapters; MagicMock-safe 'is True' guards).
* test+docs: streaming-contract coverage completeness + maintenance guidelines
Coverage: two gaps closed on the consumer-declared-final contract —
(1) send_for_platform (the delivery-resolver egress door) honors the
_interim_send contract: no seal, marker stripped before the wire;
(2) finish(final_text) on a turn that never streamed does not adopt the
final (delivery ownership stays with the gateway's normal send path for
non-streaming models / tool-only turns).
Docs: AGENTS.md 'Known Pitfalls' gains the streaming delivery contract —
the four invariants of stream-is-the-message adapters (prefix-stable
frames, consumer-declared final, interim-send marker, reconcile-by-edit),
each traced to its live incident, plus the live-probed Slack streaming
API ground truth and the MagicMock 'is True' guard-style note.
* fix(relay): seal transport failure must never silently lose the final (review B1)
Two halves of one silent-loss path, live-probed on the review branch:
1. adapter: _seal_open_draft did not catch transport exceptions. A socket
drop at seal time raised out of send(), skipping the fail-open plain
send entirely. Now: retry the SAME idempotent final frame once (the
connector's sealed-key tombstone returns the original stream ts for a
repeated final — a retry can never open a second stream or duplicate),
then report failure so the caller's fail-open path runs.
2. consumer: the turn-final retry (elif not _already_sent) called
_send_or_edit with finalize=False, which re-entered the DRAFT-FRAME
branch. Its no-op dedupe compared the adopted final against the last
unsealed frame, matched, and returned True with ZERO transport calls —
final_response_sent went green, delivered_final_matches reconciled,
the gateway suppressed its fallback, and the user never received the
answer. finalize=True keeps this retry out of the draft branch.
Regression suite: tests/gateway/test_relay_seal_failure.py (3 tests).
Mutation evidence in follow-up verification: reverting either half sends
the suite red.
* fix(relay): draft ids unique across gateway incarnations (review B3)
The relay connector tombstones sealed streams by (channel, draft_id) and
keeps up to 512 of them; they outlive the gateway process. Relay gateways
are disposable BY DESIGN (scale-to-zero), and _draft_id_counter restarted
at zero every incarnation — so the first turns after every scale-from-zero
in a recently-active channel replayed already-sealed wire identities. The
connector answered those frames straight out of the old tombstone: zero
Slack API calls, the OLD message ts returned as the new turn's identity,
the new answer silently dropped while gateway-side flags recorded success.
Seed the counter from wall-clock milliseconds at process start. Ids stay
plain ints within the existing contract op; incarnations cannot overlap
for realistic turn counts and restart gaps.
Regression: tests/gateway/test_draft_id_restart_uniqueness.py — the seed
test fails on the old code (seed 0 is not epoch-scale).
* fix(relay): stream/card state keyed per TURN, not per thread anchor (review B2)
The thread anchor is the wrong coordination identity — simultaneously:
- too coarse: two parallel turns replying INSIDE ONE Slack thread share
thread_ts. Live-probed on the review branch: turn A's final sealed turn
B's stream with A's content while A's own stream stayed open, and B's
final degraded to a plain send.
- too fragile: a flat DM with no thread metadata degraded to the bare
chat id, re-creating the original finding-#10 collision the anchor was
meant to fix.
_draft_key now prefers the triggering inbound message id (message_id /
reply_to_message_id — per-turn by construction; the gateway's Slack
thread metadata and the consumer's send path both stamp it), falling back
to the thread anchor, then the bare chat. The consumer stamps the same
reply_to_message_id on draft frames so frames and the turn-final resolve
to one key. Task-card ids share the derivation via _card_key (one helper
for send AND stop, so the stop always hits the stream the send opened).
Legacy resolver-lane callers with placement-only metadata still seal via
_match_open_draft's fallback — but ONLY when exactly one stream is open.
With several open, an identity-less send stays a plain send: a duplicate
message is recoverable, sealing someone else's stream is not.
Regression: tests/gateway/relay/test_relay_turn_keying.py (7 tests).
* fix(relay): stream-is-the-message is a Slack semantic, gate it on the descriptor (review B4)
draft_stream_is_message was hardcoded True on the relay adapter class,
i.e. for EVERY relay platform. The base send_draft contract is
Telegram-shaped — the draft clears client-side and the final arrives as
a separate real send that becomes the history message. With the flag
forced on, any non-Slack connector advertising the draft op had its
turn-final intercepted into draft(final=true): probed on the review
branch with a telegram descriptor, the op stream was
[draft(final=false), draft(final=true)] and NO send — no history message
would ever be posted.
Gate the flag on the negotiated descriptor platform (slack), and skip
arming seal-interception entirely when it is off. A future platform with
genuine stream-is-the-message native streaming should advertise it via
the descriptor rather than widening the platform check by guesswork.
Regression: tests/gateway/relay/test_relay_stream_semantics_gating.py
(4 tests: gating both ways, telegram final is a real send, slack final
still seals).
* fix(gateway): mark every mid-turn status lane interim — heartbeats must not seal the stream (review B5)
Seal-interception treats the first unmarked send to an armed (chat, turn)
key as the turn-final. The consumer's own interim lanes (commentary, tail
flush) carry _interim_send, but four gateway-side lanes that fire DURING
a streaming turn did not:
- long-running heartbeat (default every 180s — probed live: at 3 minutes
it sealed the live stream with '⏳ Working — 3 min', the real final
posted as a duplicate, and later frames were silently swallowed by the
seal tombstone)
- inactivity warning
- plain-text approval fallback (button lane failed)
- background-review notice
Add _interim_metadata() beside _non_conversational_metadata and wrap all
four call sites. The marker is gateway-internal; the relay adapter strips
it before the wire (existing behavior, pinned by test).
Note for follow-up: the opt-out shape remains fragile — any FUTURE
unmarked mid-turn send lane re-creates this bug. Inverting the contract
(explicitly mark the one turn-final send) is the durable fix but touches
every adapter's final-delivery path; deliberately kept out of this
review-fix series.
Regression: tests/gateway/test_interim_send_lanes.py (4 tests).
* fix(gateway): interrupted/incomplete turns must not adopt the diagnostic as the stream final (review B6)
The finish(final_text) adoption gate checked only 'not failed', but the
interrupt/abort returns in agent/conversation_loop.py are
{completed: False, interrupted: True, final_response: 'Operation
interrupted during …'} with NO failed key. Adopting that diagnostic:
1. sealed the user's streamed partial answer over with the interrupt
text (stream-is-the-message: the seal rewrites the whole message), and
2. recorded the diagnostic as the turn-final payload, so
delivered_final_matches reconciled and the gateway suppressed its own
error-delivery path — the diagnostic became the ONLY thing delivered.
Enumerated all 27 final_response-bearing return shapes in
conversation_loop.py: every non-happy-path shape carries completed:
False (several with a diagnostic final_response and neither failed nor
interrupted — retry exhaustion, truncation, codex-incomplete); the happy
path routes through turn_finalizer.finalize_turn (completed=True). Gate
is therefore: not failed AND not interrupted AND completed is not False.
Results lacking the completed key entirely (older callers/test doubles)
keep the previous behavior.
Regression: tests/gateway/test_stream_final_adoption_gate.py (6 tests,
incl. a source-level pin on the run.py call site).
* fix(relay): task-card transport failures degrade to failed SendResults (review B7)
send_native_task_card_progress and stop_native_task_card_progress let
transport exceptions escape. The stop runs inside the progress loop's
finally block on the turn-cleanup path, and the post-cancel awaits in
gateway/run.py caught only CancelledError — a socket drop during a card
publish/stop therefore aborted cleanup BEFORE the final-delivery
bookkeeping ran.
Three layers, outermost defends any adapter:
- both adapter methods catch transport exceptions and return failed
SendResults (progress is advisory; the TurnRunner's text fallback
already handles failure results)
- the progress loop's finally wraps the stop (best-effort; the connector
seals orphaned card streams on its own via recycling/eviction)
- the cleanup awaits log-and-continue on non-cancellation errors so
final-delivery bookkeeping always runs
Regression: tests/gateway/relay/test_relay_task_card_failures.py.
* fix(relay): a dying turn seals its native stream instead of orphaning it (review B8)
Stale-generation exits (/new, /stop mid-stream) and cancellations
returned from the consumer's run() with the native stream still open:
- the Slack message kept its live streaming indicator forever (the
cancellation best-effort edit only runs when _message_id exists, and
the native draft path deliberately keeps it None);
- the adapter's armed interception state survived the turn, so the next
turn on the same key could inherit it and seal a dead draft_id.
New adapter op abandon_open_draft(chat, content): seals in place with
the text already on screen (the consumer passes its last delivered
frame) — the seal adds nothing and claims nothing; delivery flags are
never set, so the gateway's normal paths still own whatever happens
next. Best-effort by contract (failure reported, never raised); the
connector reaps truly orphaned streams via recycling/eviction.
The consumer calls it from both death paths: the stale-generation early
return and the CancelledError handler.
Regression: tests/gateway/test_stream_abandon_on_turn_death.py (4 tests,
incl. the next-turn-inheritance hazard).
* fix(relay): bound the draft/seal coordination dicts (review M1)
_sealed_draft_by_chat's key embeds a per-turn identity, so every
completed turn wrote a permanent entry — unbounded growth for the life
of a long-running gateway process (the docstring said 'one entry per
chat', which stopped being true when the key gained the turn anchor).
_open_draft_by_chat could grow the same way via abandoned entries.
FIFO-evict both at 512 entries — the same idiom as the sibling bounded
cache (_auto_thread_by_chat, capped at 256) and the same size as the
connector's own tombstone store. The straggler window the tombstone
exists for is seconds long; FIFO is more than enough.
Regression: tests/gateway/relay/test_relay_state_bounds.py.
* fix(relay): explicit connector rejection disarms interception; exceptions stay armed (review P3)
The G-D1 optimistic-arming change silently dropped disarm-on-failure
entirely: after an EXPLICIT connector rejection (success=False result —
not a transport ambiguity), interception stayed armed even though the
stream consumer disables the draft transport on that failure and falls
back to edit-based streaming. Its turn-final would then be converted
into a seal on a stream the connector just told us is unusable.
test_draft_failure_result_propagates claimed to cover this ('must NOT
leave seal-interception armed') but passed for an unrelated reason: the
stub's canned failure also failed the SEAL, whose fail-open path did the
plain send.
Split the two semantics and pin each honestly:
- explicit rejection (result success=False): disarm — turn-final is a
real send (test_draft_failure_result_propagates, now testing what its
comment says)
- transport exception: ambiguous, stay armed — turn-final still seals
(test_draft_transport_exception_keeps_interception_armed, the G-D1
contract)
Also corrects commit ba3a24a's claim ('a failed frame disarms
interception so the edit-based fallback's real send goes through
untouched') to hold again for the rejection case it described.
* fix(relay): lost acks are ambiguous, not rejections — on the RESULT channel too (review r2, finding 1)
The production ws transport does not raise on ack timeout — it returns
{"success": False, "error": "relay outbound timed out"}. The round-1
ambiguity handling keyed entirely on the exception channel, so the shape
production actually produces was misclassified as a definite connector
rejection. Probed on the head:
- lost SEAL ack: skipped the idempotent retry, fell straight to a plain
send — duplicate final whenever the seal had actually applied;
- lost FRAME ack: the round-1 disarm-on-rejection fired — interception
disarmed, frozen native stream beside a plain final. This re-created
the original G-D1 ambiguous-ack defect on the result channel.
Contract now spans both channels:
- transport: the ack-timeout branch tags ambiguous=True. The fail-fast
branches (closing / not connected) never sent anything and stay
unmarked — they are definite non-delivery.
- adapter frame path: ambiguous results keep interception armed (same
as exceptions); only definite rejections disarm.
- adapter seal path: one shared _attempt() classifier — exception and
ambiguous result both mean "unknown"; the SAME idempotent frame is
retried once (connector tombstone returns the original stream ts for
a repeated final). Only after both attempts stay ambiguous does the
caller's fail-open plain send run: a possible duplicate after double
ack loss beats a silent loss, and double ack loss on one socket
almost always means the transport is down for the plain send too.
Regression: tests/gateway/relay/test_relay_ack_ambiguity.py (6 tests,
incl. a source-of-truth check that the transport tags the timeout branch
and leaves fail-fast branches unmarked).
* fix(relay): stream semantics + draft capability resolve per CHAT, not per primary (review r2, finding 2)
One RelayAdapter fronts N platforms (Phase 1.5): descriptors accumulate
per platform on the transport and egress is tagged per chat — but the
round-1 gate keyed draft_stream_is_message and supports_draft_streaming()
off the PRIMARY scalar descriptor. Probed on the head:
- Slack primary + Telegram chat: the Telegram chat's turn-final was
intercepted into draft(final=true) — no real Telegram history message;
- Telegram primary + Slack chat: the Slack chat was denied native
streaming entirely.
Resolve both through _descriptor_for_chat — the same per-chat machinery
max_message_length already uses (added for the identical class of bug:
the primary's 39000-char cap over-sending into Discord 400s):
- new stream_is_message_for_chat(chat_id) on the adapter; arming and
NotImplementedError gating use it. The class attribute remains as the
single-platform value and legacy-probe fallback.
- supports_draft_streaming() gains an optional chat_id kwarg (base
signature updated; single-platform adapters ignore it). The consumer
passes chat_id with a TypeError fallback for out-of-tree adapters.
- the consumer's four draft_stream_is_message reads collapse into one
_stream_is_message() helper that prefers the per-chat probe
(class-resolved, MagicMock-safe) over the attribute.
Platform-name inference ("slack") stays deliberate: a descriptor-level
semantic field is the right eventual contract but is a cross-repo wire
change — noted for the gg follow-up so future platforms advertise the
semantic explicitly.
Regression: tests/gateway/relay/test_relay_multiplatform_semantics.py
(5 tests: both starvation directions, scalar fallback, per-chat
capability gate).
* fix(gateway): split delivery + authoritative footer reconciles by suffix, not full resend (review r2, finding 3)
The _FINAL_TEXT adoption guard refuses wholesale adoption on split turns
— correct (#78541: sealed heads would repeat inside the tail) but it was
absolute: a post-split verifier footer never entered the ledger,
delivered_final_matches() reported a mismatch, and the gateway resent
the ENTIRE body+footer after the split chunks (the #11 duplicate class,
one level up).
When the authoritative final strictly prefix-extends the split ledger,
the missing suffix is the only undelivered content: append it to the
live tail and the ledger, so the finalize carries it and the recorded
payload reconciles. Non-prefix rewrites keep the full-resend fallback —
a rewrite cannot be patched onto sealed heads.
Regression: tests/gateway/test_split_final_suffix_reconcile.py (3 tests:
suffix rides the tail + reconciles, rewrite still mismatches, unsplit
adoption unchanged).
* fix(relay): cancellation mid-seal restores open state so abandon can close the stream (review r2, finding 4)
_seal_open_draft pops the open entry and writes the local tombstone
BEFORE awaiting transport I/O — correct ordering for the straggler race,
but CancelledError is not an Exception: a cancel during the await
bypassed all failure handling, leaving the remote stream live (visible
streaming indicator until connector eviction) while the local state said
'nothing open'. The consumer's abandon pass — added for exactly this
turn-death case — found nothing to close and no-oped.
On CancelledError: restore the open entry, drop the premature tombstone
(only if it is still ours), re-raise. The abandon path then seals the
stream in place with the on-screen text.
Regression: tests/gateway/relay/test_relay_seal_cancellation.py (2
tests: state restoration, and end-to-end cancel→abandon→remote seal).
* fix(relay): thread anchors are placement, not turn identity — revive the placement-only fallback (review r2, finding 5)
_match_open_draft's single-open-stream fallback was dead for its primary
intended callers: metadata carrying thread_ts/thread_id (placement-only
resolver lanes) was classified as having 'turn identity', so those sends
never reached the fallback — probed: a plain final posted beside the
still-open turn-keyed stream.
Only per-turn MESSAGE ids are identity now. Thread-anchored and bare
callers share the fallback: absorb into the chat's open stream when
EXACTLY one is open; stay a plain send when several are (duplicate is
recoverable, wrong-stream seal is not). Callers WITH a message id whose
key misses never fall back — their identity is authoritative and a miss
means the stream belongs to a different turn.
Regression: 4 new tests in test_relay_turn_keying.py (thread-anchored
seal, both ambiguous-stay-plain shapes, id-mismatch never steals).
* fix(relay): random process nonce for draft-id seeding (review r2, follow-up 6)
The epoch-millisecond seed (round-1 B3 fix) mitigates the restart-replay
class but is not a uniqueness guarantee: two gateways starting in the
same millisecond, a forked process inheriting the class state, or a
clock step backwards can all mint colliding wire identities against the
connector's per-(channel, draft_id) tombstone store.
Seed from secrets.randbits(49) instead: collision probability negligible,
no clock dependence, and ids + realistic per-process turn counts stay
comfortably inside the connector's JS number range (draft_id?: number,
2^53). Regression test now spawns two real interpreters and asserts
their seeds differ — the exact scale-to-zero restart shape, and both
start within the same second so a clock-locked seed would fail it.
* fix(relay): stamp per-turn Slack egress identity — cache is fallback only (R3-5)
The connector (gateway-gateway#210) fills chat.startStream's
recipient_user_id / recipient_team_id — required by Slack when
streaming to a channel — from metadata.user_id / metadata.scope_id.
The gateway stamped only slack_team_id per-turn and left user_id (and
scope_id) to RelayAdapter._with_scope, whose per-chat caches are keyed
on chat_id alone and overwritten by every inbound message: with users
U1 and U2 running overlapping turns in one channel, U2's arrival
overwrote the cache before U1's stream opened, and U1's stream carried
U2 as recipient_user_id.
_thread_metadata_for_source now stamps scope_id and user_id from the
turn's OWN source (setdefault — explicit values win), so identity is
turn-scoped data on the wire. _with_scope is unchanged and fill-only:
the caches keep serving restart/synthetic sends that carry no per-turn
identity, which is all they were ever safe for.
Mutation evidence: reverting the run.py hunk sends
test_thread_metadata_stamps_per_turn_user_and_scope and
test_concurrent_turns_carry_their_own_identity red; restore returns
green. The _with_scope fill-only tests pass on both trees (existing
correct behavior, now pinned against regression).
---------
Co-authored-by: Ben Barclay <ben@nousresearch.com>
|
||
|
Ben Barclay
|
bb0a4193d1 |
fix(relay): a raising socket write returns the result dict, not an exception
The socket can die BETWEEN _request_response's 'is None' liveness guard
and the actual write: the reader's finally hasn't cleared _ws yet, so
_send raises ConnectionClosed straight into callers whose contract is a
result dict (RelayAdapter.send consumes it with no try — only the
cosmetic typing lanes wrap the call). No liveness check can close this
window; it has to be caught at the write.
Convert the raise to {'success': False, 'error': ...} like every other
failed send, log the traceback at debug (the returned string alone
can't distinguish an ordinary dead socket from a defect in the frame
building above), and rely on the existing finally to drop the pending
entry. CancelledError is a BaseException, so cancellation still
propagates.
Same disposition as the equivalent guard in PR #82238; regression test
drives a socket whose write raises while the reader is still parked,
and fails without the except clause.
|
||
|
Ben Barclay
|
c60a525380 |
fix(relay): dedupe key platform component is spelling-invariant
The key derived its platform component with getattr(platform, 'value', ''), which handles the Platform enum the wire decoder always produces but collapses a plain-string platform — or a missing one — to the same empty string. Two DIFFERENT string platforms would then share one key component (cross-platform id collisions conflate), and enum vs string spellings of the SAME platform would produce two keys (a replay decoded differently would not dedupe). Inert on today's wire path (the decoder canonicalizes unknowns to Platform.RELAY), but alternate event constructors carry strings, so normalize at the key: enum value when present, the string itself otherwise, empty only when there is genuinely no platform. Missing platform intentionally still yields a key — fail-open on identity is reserved for missing message/chat ids. Tests pin all three properties; the spelling-invariance pair fails against the previous expression. |
||
|
Ben Barclay
|
2f77ca1993 |
fix(relay): reader without a socket settles pending waiters instead of asserting
_read_loop opened with 'assert self._ws is not None' — an exit that escaped BEFORE the finally that fails pending futures, contradicting the 'fails pending on ANY exit path' invariant the hardening commit established. Production currently assigns _ws before scheduling the reader, so this was latent, but any future lifecycle change hitting it would strand every in-flight waiter for the full outbound timeout with only an AssertionError in the logs. Turn it into a guarded early-return INSIDE the try: the reader logs the lifecycle bug and unwinds through the same finally as every other exit, settling all waiters. Regression test drives _read_loop with _ws=None against a registered pending future. |
||
|
Ben Barclay
|
ca3438d395 |
fix(relay): gate sends on the socket handle, not supervisor state
The mid-redial fail-fast guard used 'supervisor task not done' as the definition of the redial window, and that signal is wrong from both directions: - Too narrow: the wedge it fixed also occurs on reader exits that arm NO supervisor (terminal 4401 revocation, reconnect=False) — those stayed wedged. - Too broad: _reconnect_loop -> _dial_and_start installs the fresh socket and starts its reader, THEN awaits one hello send per fronted identity before the supervisor unwinds. Through those awaits the transport is fully live, yet the guard rejected every send as 'reconnecting' — refusing real traffic on a healthy socket. With the previous commit the reader clears _ws on unexpected exit, so the existing 'is None' check now covers the entire outage window honestly: _ws is the single liveness signal. Drop the supervisor-state guard. The redial-window test now drives the real sequence (reader exit arms the supervisor and clears _ws) instead of hand-crafting a stale-_ws state the transport can no longer reach, and a new test pins the post-dial window: a send issued while the supervisor is still unwinding past a live socket must reach that socket (fails with the guard reinstated). |
||
|
Ben Barclay
|
52f9979a6c |
fix(relay): reader clears the dead socket handle on unexpected exit
Two reader-exit paths arm NO reconnect supervisor: a terminal 4401 revocation (deliberately never re-dials) and reconnect=False transports. On both, _ws kept pointing at the dead socket after the reader unwound, so the 'is None' liveness guard reported connected and a send registered a future nothing could ever resolve — the full _outbound_timeout_s (~30s) wedge this PR exists to eliminate. The revocation case is the sharpest: the fatal-error notification that path emits is itself an outbound send, so it ate the stall. Null the handle in the reader's finally, identity-guarded (only if _ws still points at the socket THIS reader served) so a supervisor re-dial that already installed a fresh socket is never clobbered, and gated on not _closing so disconnect() keeps sole ownership of teardown. This also makes _ws the single honest liveness signal for the redial window itself — groundwork for retiring the supervisor-state send guard, which misreads that window from both directions. Regression tests cover both uncovered paths, asserting _ws is cleared and a post-drop send fails fast; both wedge (fail) with the finally reverted. |
||
|
Ben Barclay
|
22dfdc1c56 |
fix(relay): dedupe key reads chat identity from event.source, keyed per platform
The dedupe key read event.chat_id — a field MessageEvent does not have (chat identity lives on event.source.chat_id; see how every other read in this adapter resolves it). getattr defaulted to None, so _inbound_dedupe_key returned None for EVERY production event: the fail-open branch always taken, the seen-set permanently empty, and a replayed inbound still re-ran the whole turn. The tests passed because their SimpleNamespace events carried a top-level chat_id no production code path produces. Read the chat id from event.source, and join the underlying platform into the key: one relay adapter fronts several platforms (Phase 1.5 multiplex), and two platforms' numeric chat/message ids must not collide into one replay identity. The test event factory now builds real MessageEvent/SessionSource objects in the wire decoder's shape, so the replay and bounded-set tests fail against the broken key instead of green-lighting it. |
||
|
Victor Kyriazakos
|
d2b9b73986 |
chore(gateway): loud mirror diagnostics — name the exact drop reason
The 19:53 canary run failed the in_channel seed EVEN WITH the deterministic session_id fix live, and the mirror's two failure paths (no-session bail, append exception) both logged at debug — invisible in production. WARNING both, with the explicit session id in the exception path, so the next run names the failing branch instead of another blind retest. |
||
|
Victor Kyriazakos
|
4673836f79 |
fix(cron): deterministic in_channel seed + companion thread-surface seed
Two live failures from the Alice canary (2026-08-19, jobs 28a24afebd81 / 83b93f8be379), both leaving a continuable in_channel cron with amnesia: 1. Seed mirrored via origin heuristics and silently dropped the brief. _seed_cron_channel_session created the flat session row, then mirror_to_session RE-DISCOVERED the target via find_session_by_origin — whose multi-candidate bail-out returns None on a populated chat (flat session + N per-message thread sessions sharing one chat_id, mixed user_ids). Receipt: 'in_channel seed did NOT land on slack:D0BJTDCSR7C'. mirror_to_session now accepts an explicit session_id and both cron seeds pass the exact row they just created; origin-scan remains the fallback for callers that genuinely don't know the target. 2. The brief's OWN THREAD was never seeded. in_channel delivers flat, but a flat Slack message still invites a thread reply (the natural mobile affordance — exactly what the user did). That reply keys to (chat, thread=<brief ts>), which no seed touched. The delivery's message_id now anchors a companion _seed_cron_thread_session so BOTH reply surfaces (plain channel message AND in-thread reply) continue the job. Also: thread-seed failures upgraded debug→WARNING (silent seed failure IS the user-facing bug), and the thread seed reports landed/not-landed. Regression tests drive both against the live failure shapes: exact-session mirror asserted via session_id kwarg; thread companion asserted via the SendResult message_id anchor. Clean-fixture blind spot noted: the E2E harness used a fresh store with one row, which is why heuristic rediscovery looked fine pre-production. |
||
|
Brooklyn Nicholson
|
3f6d4c6338 |
fix(gateway): run Windows /update as a module, not through the shim
The Windows branch spawned the updater as `hermes.exe update --gateway`, so the update held the very shim it had to replace and failed with os error 32. Invoke it as `python -m hermes_cli.main update --gateway` under the same interpreter the gateway already runs, which maps no shim. Salvaged from #89970 by @Akloenx123. |
||
|
Victor Kyriazakos
|
31a4b8503d |
feat(relay): block-formatting hints on relay text egress (rich/markdown blocks)
Field report (enterprise side-by-side, 2026-08-18, finding 2): identical agent output renders native rich_text lists, Block Kit tables, and highlighted code on native Slack, but literal '-' bullets and code-fence tables on the relay lane. Native reads platforms.slack.extra.rich_blocks / markdown_blocks and renders Block Kit locally; relay frames carried no formatting signal, so the connector had no way to know the operator wants block rendering. Contract (additive, v1): the connector advertises supports_block_formatting in its capability descriptor. When it does AND the operator enables platforms.relay.extra.slack.rich_blocks / markdown_blocks (same per-platform sub-block and same _coerce_flag semantics as the other relay Slack knobs), the gateway stamps format_hints into outbound metadata on BOTH text egress lanes — send and edit (a streamed reply's final edit carries the finished markdown, so it must signal too or streams seal as plain text). The connector renders blocks and keeps plain text as the fallback. Old connector: never advertises -> no dead metadata ever sent. Old gateway: never stamps -> connector renders plain text as today. Knobs default OFF, matching native's opt-in posture. 8 new tests: descriptor default/from_json, hint stamping (capable+enabled), capability-absent suppression, knobs-off suppression, YAML-quoted-false coercion, partial knobs, edit-lane parity. |
||
|
Victor Kyriazakos
|
85b89451f6 |
feat(relay): flat in_channel continuable cron surface on the relay lane
Field report (enterprise side-by-side, 2026-08-18, finding 1 — the relay-only blocker): on relay-fronted Slack, cron briefs always deliver into a dedicated thread; the flat continuable surface (cron_continuable_surface: in_channel) that native Slack supports is inert, so plain DM replies never continue the job and the main conversation never sees the brief. Three gaps closed: - CapabilityDescriptor gains supports_inchannel_continuable (default False, additive within contract_version 1; from_json ignores it from old connectors, old gateways filter it as unknown). The connector advertises it per platform at handshake. - RelayAdapter maps the bit onto the adapter capability surface in both the constructor and _apply_descriptor (renegotiation), so the scheduler's D6 fail-safe gate sees it exactly like native Slack's class attribute. - _resolve_cron_surface_mode replaces the scheduler's inline flat-key read: native keeps the shipped flat shape; the relay lane reads the same per-logical-platform sub-block as the documented relay Slack knobs (platforms.relay.extra.slack.cron_continuable_surface), sub-block wins, scoped so a slack block cannot leak onto other fronted platforms. The seed path needs no changes: RelayAdapter inherits set_session_store (wired by the generic adapter boot loop) and _seed_cron_channel_session keys the flat session off the logical platform_name. 12 new tests: descriptor default/from_json/legacy-absence, adapter mapping constructor + renegotiation, and the surface-knob matrix (native flat key, relay sub-block, per-platform scoping, precedence, defaults). |
||
|
Victor Kyriazakos
|
b5f95d0890 |
fix(relay): drop replayed inbounds — bounded dedupe on (chat_id, message_id)
Live-canary finding #3 (Alice, staging): the relay inbound leg is at-least-once. On WS re-handshake the connector replays its durable per-instance buffer; a long multi-tool turn (60-100s) straddling a quiet socket drop got its ORIGINAL inbound replayed after the turn finished, re-running the entire turn — the user saw the final answer posted 2-5x (each a separate execution, hence slightly different texts). Receipts: same msg text at history=0 in back-to-back sessions 121647/121840, no Slack-side retry on the connector (envelope dedupe never fired). Consumer-side idempotency: bounded FIFO seen-set (512) keyed by platform message identity; events without a message_id never dedupe (fail-open — dropping a real message is worse than rerunning one). No wire change; contract v1 untouched. Transplanted-from: victor-fork/feat/relay-slack-live-cards@73ce04ae75 (extracted for the rc.4 relay-fixes train; tests moved to a standalone file with no live-cards dependencies) |
||
|
69k4xmdfm2-blip
|
21260c3281 |
fix(gateway): carry the profile in adapter-derived session keys (#88404)
Adapter ingress derives a session key BEFORE the runner stamps source.profile in _make_profile_message_handler, so the namespace fell back to the active profile and every bot in a multiplexed gateway produced agent:main:<platform>:<chat>. A Telegram private chat reports the user's own id as chat.id, identical for every bot, so two profiles sharing one human collapsed onto a single lane: _pending_text_batches, _active_sessions, the busy-session guard and _post_delivery_callbacks are all keyed on that string. A day of production logs across two bots shows 60 flushes, none carrying the secondary profile's namespace. set_owner_profile records credential ownership on the adapter and _session_key_profile resolves the namespace as source.profile -> _owner_profile -> the session store's resolver, so a secondary adapter keys into its own namespace even before the source is stamped. Stamped sources keep priority, so relay/connector ingress, which routes per event rather than per credential, is unchanged. _configure_profile_adapter installs the owner alongside the other handlers, covering startup and reconnect. Every candidate is type-checked as a non-blank str, and every attribute read goes through getattr: adapters are routinely built without BasePlatformAdapter.__init__, and a duck-typed session store returns a truthy non-string that would otherwise be interpolated into the key as agent:<MagicMock ...>:. Also routes the four call sites that passed no profile at all (feishu media batches, raft, slack _session_key_for_source, telegram photo batches) through the same resolver. test_multiplex_busy_input_mode's secondary-adapter busy case seeded _active_sessions with the unstamped agent:main: key, asserting the pre-fix collapse. It now seeds the lane the profile-owned adapter actually derives. A primary adapter has no owner and an unstamped source, so it resolves exactly as before; with multiplex_profiles off the resolver returns None and every key is byte-identical to today's. |
||
|
Teknium
|
bdd0a79c6a |
fix(goals): /goal resume actually restarts work after budget exhaustion
After a standing goal auto-paused on turn-budget exhaustion, every
surface's /goal resume handler only flipped the persisted state back to
active (and reset turns_used) and rendered an acknowledgement — nothing
re-entered the conversation loop, so the goal sat idle until the user
sent another ordinary message.
Fix the whole class by scheduling the canonical
GoalManager.next_continuation_prompt() through each surface's existing
input path after a successful resume:
- Desktop/TUI (tui_gateway/methods_tools.py command.dispatch): return a
sendable {type: "send"} dispatch with the continuation as the message,
a "Continuing now" notice, and display "/goal resume" so the
transcript shows the concise invocation instead of the model-facing
scaffolding. No-goal keeps the exec response.
- Classic CLI (hermes_cli/cli_commands_mixin.py): put the continuation
on _pending_input, same as the /goal <text> kickoff.
- Messaging gateway (gateway/slash_commands.py): enqueue a continuation
MessageEvent through the adapter FIFO — the same path the post-turn
judge uses — so queued real user messages preempt naturally and the
pause/clear stale-continuation cleanup recognizes it.
Also correct the now-misleading gateway.goal.resumed copy ("Send any
message to continue…") across all 17 locale files.
Regression tests cover exact budget exhaustion → resume on the real CLI
handler, the real gateway handler (including the
_is_goal_continuation_event guard contract), and the TUI
command.dispatch boundary; verified each fails on the pre-fix code.
Fixes #75362
|
||
|
Victor Kyriazakos
|
534f8d4fb0 |
fix(relay): WAN-friendly keepalive tuning (ping_interval=30, ping_timeout=60)
Customer gateways cross WAN paths to the connector; the websockets library defaults (ping 20s, pong deadline 20s) close the socket with 1011 keepalive ping timeout under transient latency or event-loop stalls — the trigger of the Coatue 2026-08-18 incident. 60s tolerates stalls while still detecting a genuinely dead link within ~90s worst case. Set explicitly at both connect() call sites (with and without auth headers) so the tuning is visible and pinned by test rather than inherited from library defaults. |
||
|
Victor Kyriazakos
|
a7946f6502 |
fix(relay): fail sends fast while the reconnect supervisor is mid-redial
Between an unexpected close and the supervisor's successful re-dial,
self._ws still points at the DEAD socket, so the existing None-check
does not cover the backoff window: a send registered a future no reader
could resolve and blocked the caller for _outbound_timeout_s.
A live supervisor task is exactly the redial window (the loop returns
when a dial succeeds and its fresh reader takes over), so no new state
is needed: when self._supervisor is not done, return the error dict
immediately ({success: False, error: reconnecting}). Callers already
handle failed sends; a fast, honest failure beats a 30s wedge
(Coatue 2026-08-18).
|
||
|
Victor Kyriazakos
|
d2975f4219 |
fix(relay): _read_loop fails in-flight pending futures on any exit
The reader is the only thing that can resolve a pending outbound_result
future. When the socket dropped unexpectedly (vs a deliberate
disconnect(), which already failed pending), every in-flight
_request_response waiter blocked the full _outbound_timeout_s (~30s) on
a future no reader could resolve.
Coatue incident 2026-08-18: a 1011 keepalive ping timeout close left
final-send and error-notification calls wedged against the dead socket,
holding sessions active while the connector replayed inbound backlog.
Fail every not-done pending future from _read_loop's finally with the
dict shape callers expect ({success: False, error: connection lost}) —
never an exception on the outbound path — then clear the map. list()
snapshot avoids mutation-during-iteration from woken waiters' finally
pops.
|
||
|
Teknium
|
c69b6471e6 |
fix(gateway): real user text stays clean in the transcript on resume-pending turns (#86580)
The salvaged fix persisted the recovery note unconditionally — correct for the synthesized empty auto-resume turn, but a user who typed real text while resume was pending would get the [System note: ...] scaffold persisted as their own words, leaking scaffolding into the durable transcript (the same class as #81841 on the assistant side). _prepare_resume_pending_message now persists the note only when the original message is blank; real text persists verbatim while the model still receives the wrapped note. Whitespace-only counts as blank. Tests cover all three shapes. |
||
|
andyst-dev
|
0cc26777bb |
fix(gateway): persist resume recovery notes
Fixes #86580 |