cf60ebbdfd264c3fc067b9e3230df8752c50d2ec
102 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Teknium
|
a47a35f44b | refactor(gateway/stream): split GatewayStreamConsumer into transport/fallback/think mixins + fences leaf; drop dead _MEDIA_RE | ||
|
Teknium
|
1b8af7350b | refactor(gateway/stream): flatten approval-boundary handling, unify final-delivered flag marking, compact __init__ state notes | ||
|
Teknium
|
1d99e2a2d1 | refactor(gateway/stream): split _send_or_edit into per-transport collaborators; unify fallback send-with-flood-retry | ||
|
Teknium
|
8e3705c925 | refactor(gateway/stream): decompose run() into _Tick + per-phase collaborators; unify frame/seed/preview-delete/draft-metadata helpers | ||
|
Teknium
|
659c6b9fff | refactor(gateway): simplify session, status, stream_consumer, kanban_watchers, relay adapter, hosted-room driver/discussion/replicas, shutdown/lifecycle, pairing, channel_directory, control_socket and small modules | ||
|
Teknium
|
aed6720dab |
refactor(gateway/run, slash_commands): dispatch tables, helper unification and hand-reviewed comment compaction
run.py: - built-in adapter creation: 9-branch if/elif -> _BUILTIN_ADAPTERS table - idle slash-command routing: 35 `if canonical == ...` branches -> _gateway_idle_command_handlers() - shared helpers: _send_command_ack (4 sites), _command_origin_for_source (2), _session_entry_for_manager (goal/heartbeat), _toggle_adapter_auto_tts_set (2), _load_env_or_agent_cfg_timeout (2), _float_env reuse (2), _resolve_session_key_or_none (3), _running_agent_ids (4), _schedule_rename_from_title_thread (2), _write_runtime_status_quiet (5), _AUTO_RESET_CONTEXT_NOTES/_auto_reset_reason_text - ruff SIM102/SIM103/SIM105/SIM108/SIM118 + F401 across gateway/ (semantics re-reviewed; sqlite Row `.keys()` and side-effecting assignments kept) - two hand-reviewed comment/docstring compaction passes (AST-identical, rationale kept) slash_commands.py: - /model: typed path and picker callback shared one 200-line commit block -> _perform_model_switch + _commit_model_switch - comment/docstring compaction (AST-identical) |
||
|
Teknium
|
fab36436b2 |
fix(gateway): keep one Telegram bubble when the finalize edit fails under reply_to_mode=first (#71047)
Problem B of #71047: with streaming + reply_to_mode='first', the streamed
preview is a reply-quote of the user's message. When the turn-final edit
hits flood control, the empty-tail fresh-commit resend either (a) also got
flood-capped -> the consumer reported 'failed', the gateway's normal final
send fired, and the never-deleted preview + the fresh final left TWO
visible bubbles, or (b) succeeded but as a plain non-reply message that
didn't match the preview's anchor.
- preserve the turn's reply anchor (initial_reply_to_id) on the
empty-fallback fresh-commit resend so the replacement message quotes the
user's message exactly like the preview and the non-streaming path
- retry a flood-rejected preview deleteMessage once (delete_message
returns False rather than raising) so the stale preview doesn't linger
next to the fresh final; still best-effort, and the preview is only ever
deleted AFTER the replacement send succeeded
- regression tests for the anchor, the delete retry, and the
flood-capped-resend single-bubble suppression decision
Builds on @fangliquanflq's PR #96097 ('preview' verdict for flood-rejected
fresh commits), cherry-picked as the previous commit with the conflict
against
|
||
|
fangliquanflq
|
d7e6461b5f | fix(gateway): preserve streamed final after flood rejection | ||
|
teknium1
|
46e7ad8e12 |
fix(gateway): gate record-less visible-text match on _already_sent
Draft frames set _last_sent_text for dedupe without setting _already_sent (they are ephemeral); an ungated has_delivered_text match let a draft-only preview count as durable delivery and regressed test_relay_seal_failure's dead-transport guarantee on CI. |
||
|
teknium1
|
fd998120c1 |
fix(gateway): judge delivery success against final content, not flag trust (#95382, #98552)
A record-less delivery flag (final_response_sent / final_content_delivered set with no recorded turn-final payload) was trusted blindly by delivered_final_matches (None -> legacy trust), so a first-edit prefix or a truncated finalize suppressed the gateway's corrective send — silent partial delivery. - delivered_final_matches: record-less flags are now reconciled against the FINAL content via has_delivered_text; only the explicitly-marked ambiguous-timeout path (_delivery_ambiguous) keeps legacy trust. - _try_fresh_final and the native-streaming optimistic finalize now record their delivered payload (the last record-less flag setters); the optimistic record rolls back on definitive dispatch failure. - Discord adapter: dead-transport send failures (client gone, WS closed/reset) are classified as send_path_degraded (retryable) so the delivery-obligation ledger's reconnect sweep replays the stranded final response instead of losing it until a process restart. Fixes #95382; closes the #98552 false-positive class. |
||
|
Teknium
|
02ecc4be27 |
fix(gateway): gate stream commentary reply-to on platforms that need it
Discord interim commentary was replying to the user's trigger message on
every update (
|
||
|
NanakoIce
|
66fa6e41c4 |
fix(buzz): reply in-thread instead of flat channel posts
Buzz has no native thread_id; channel threading is entirely --reply-to on the triggering event. Interim commentary and progress bubbles only passed the anchor via metadata.reply_to_message_id (or not at all), so most Kathy posts landed as new top-level messages and cluttered channels. - Honor metadata.reply_to_message_id in BuzzAdapter.send - Pass reply_to on stream commentary sends - Treat buzz like slack/mattermost for progress thread resolution - Set _progress_reply_to to the trigger event for buzz - Add unit tests for adapter metadata and progress routing |
||
|
Ben Barclay
|
1f99a4b2f2 |
fix(relay): resolve fresh-final unfurl decision per chat, not per primary identity (#99206)
The stream consumer called prefers_fresh_final_streaming(text, metadata=...) only, and no metadata producer stamps a platform key — so RelayAdapter's hook always fell back to the PRIMARY descriptor's platform (the scalar-vs-per-chat capability seam, third occurrence). Two failure directions on multiplexed relays with platforms.relay.extra.slack.unfurl_links/media: true (#97957): - Slack primary fronting Telegram/Discord: every link-bearing streamed final on the non-Slack chats finalized as a fresh send with no delete op advertised -> the answer delivered TWICE (orphaned preview). - Non-Slack primary fronting Slack: the hook returned False, leaving the force-on unfurl feature dark on exactly the chats it shipped for. Pass chat_id=self.chat_id from the consumer; the relay hook already accepted it and resolves via _platform_by_chat + the per-platform negotiated descriptor. Graduated TypeError fallback keeps the single-platform hook signatures (Telegram, base class) and legacy test doubles working unchanged. Both regression tests verified RED against the unfixed consumer, GREEN with the fix; single-platform relays are unaffected (#97957's own 30 tests unchanged-green). |
||
|
wansui
|
9faa953cd1 |
fix(wecom): eliminate duplicate + split bubbles in native streaming
Fixes two related native-streaming bubble defects surfaced in production:
1. Duplicate bubble on long turns — when keep-alive already refreshed the
6-min reply window, the Layer-2 clock fallback still declined the finalize
frame and forced a proactive send(), duplicating the message. Skip the
clock fallback while keep-alive is active; intermediate-frame failures are
now fully fire-and-forget (only a failed FINAL frame falls back to send()).
2. Split / mini bubbles ('Cla' + 'ude ...') — two compounding root causes:
a) In native streaming a mid-turn commentary (e.g. a Hindsight recall
notice) called _reset_segment_state(), clearing the cumulative
_accumulated so the next delta + finalize frame carried only the few
chars accumulated after the reset. Native streaming now skips that
reset (commentary still posts as its own message via send()).
b) The adapter-side _BlockChunker.update() 'only grow' guard silently
dropped any cumulative snapshot shorter than its high-water mark, so
after a baseline reset the leading characters were stranded before
_emitted_len. Removed the _BlockChunker sentence-alignment + idle-flush
layer entirely; intermediate frames are pure identity-dedup, matching
the fire-and-forget model.
Also removes ~232 lines of now-dead code (_BlockChunker class, idle-flush
machinery, block-stream constants) and aligns the test suite with the
fire-and-forget frame model, including a regression test that locks the
native-commentary-no-reset behavior.
Tests: 177 passed, 3 skipped (wecom + stream_consumer suites).
|
||
|
wansui
|
2ecb544551 |
feat(wecom): native reply streaming (per-turn isolation, dedup-safe delivery, interaction boundaries)
Implement native reply streaming for the WeCom (企业微信) adapter over the long-connection "msgtype: stream" transport, so a reply renders as a single live-updating typing bubble instead of one final block. Aligns with the official wecom-openclaw-plugin streaming behavior. Includes the machinery intrinsic to native streaming on WeCom: - Transport: seed frame (<think></think>) opens the typing bubble, intermediate frames update it, a finalize frame closes it; native-streaming adapters are let past the edit-only gate. Fire-and-forget intermediate frames (WeCom long-connection mode has no documented edit-rate limit); an adapter-level frame cap is retained. (Early builds gated frames behind a char throttle; removed in favor of fire-and-forget + identity dedup.) - Per-turn isolation: each turn owns a unique turn_id; concurrent messages are isolated via (chat_id, turn_id)-keyed state. Dual-lane priority queue (control vs normal) plus a per-chat token bucket to stay under WeCom's rate limit (errcode 846607). - Dedup-safe delivery + ack-race handling: deliver-once contract (a frame is delivered the moment it is emitted; failures logged, not re-sent; delivery marked once per turn), per-req_id reply queue with ack tracking, and the timeout-inversion / orphan-queue race fixes. Robust fallback on 846608 / 846609 / errcode 6000 / passive-reply timeout via proactive send. - Interaction boundaries: finalize + reset before approval/clarify prompts so the prompt is the last thing on screen and never traps a lingering bubble; eager re-seed after a clarify answer so the typing bubble reappears instantly. - Stream-level keepalive: optional periodic finish=false frame + finalize-time stream-age guard to refresh WeCom's ~6-minute reply-stream window on long turns (mitigates 846604 / 846608). Off by default; tunable via config.yaml. - Tool-progress folded into the same native-stream bubble instead of separate messages; image+text double-callback merged into one turn. Tests cover the streaming lifecycle, per-turn isolation, duplicate-send / ack timing, approval + clarify boundaries, eager re-seed, and tool-progress. |
||
|
Teknium
|
216c98aaed |
fix(gateway): scope draft-final finalize skip to fresh persistent sends
The salvaged skip condition keyed on _use_draft_streaming alone, which also suppressed the explicit REQUIRES_EDIT_FINALIZE pass when a draft-streaming run had degraded to edit-based delivery (draft failure fallback sets _message_id). Key the skip on the got_done update being a fresh persistent send through the native-draft transport (_message_id is None), which is the only case where the update already carried its own finalize. |
||
|
Gille
|
790c850144 | fix(telegram): preserve rich finals after DM drafts | ||
|
victor-kyriazakos
|
3683e70043 |
feat(relay): live-card ops — native draft streaming + task cards over the relay (gateway half) (#85796)
* feat(relay): live-card ops — native draft streaming + task cards over the relay (gateway half)
NS-658. Three additive ops within contract v1, emitted only when the
connector's negotiated descriptor advertises them:
{op: draft, chat_id, draft_id, content, final, metadata}
{op: task_card, chat_id, card_id, chunks, metadata}
{op: task_card_stop, chat_id, card_id, metadata}
The gateway side is deliberately dumb: no platform API knowledge, no new
config keys. Slack mechanics (chat.startStream/appendStream/stopStream,
per-workspace feature-gate cache, send+edit fallback) live connector-side
where the platform adapter lives in the relay model.
Semantic bridge: base send_draft is Telegram-shaped (draft clears; final
is a separate send). Slack native streaming makes the stream THE message.
The adapter tracks the open draft per chat and converts the turn-final
send() into draft(final=true) so the connector seals the stream instead
of posting a duplicate; the stream ts returns as the message identity.
A failed frame disarms interception so the edit-based fallback's real
send goes through untouched.
BEHAVIOR CHANGE (deliberate): relay supports_draft_streaming() now
requires the descriptor flag AND the draft op. Flag-only was a latent
lie — send_draft inherited NotImplementedError, so a connector setting
the flag without the op would have crashed the stream consumer's draft
path. supported_ops stays fail-open for legacy (pre-contract) ops;
draft/task_card did not exist pre-contract and must not fail open.
Task cards ride #85476's adapter-agnostic TurnRunner seam (hasattr on
send_native_task_card_progress); supports_native_task_cards() is the
descriptor probe. Connector half + E2E harness pair follow in the gg
repo.
* fix(relay): expose native_task_cards_enabled() on the relay adapter
Live-canary finding (Alice, staging): the TurnRunner's task-card lane
probes adapter.native_task_cards_enabled() (the native Slack adapter's
opt-in contract). The relay adapter only offered
supports_native_task_cards(), so the hasattr gate failed silently and
tool progress stayed on the text path — draft streaming worked, cards
never rendered. Alias it to the descriptor probe.
* fix(relay): match task-card methods to the TurnRunner's native keyword contract
Live-canary finding #2 (Alice, staging): gateway/run.py's card lane calls
send/stop_native_task_card_progress with the NATIVE Slack adapter's
signature (tasks/title/reply_to/metadata/fallback_text, keyword-only) —
PR 85796's relay methods took a positional card_id, so every call raised
TypeError('unexpected keyword argument reply_to') in the progress task,
repeatedly killing the card publisher (and the retry loop resent the
final delivery 4-5x). Card id now derives per turn thread
(turn:<reply_to>), thread_ts anchored like draft; title/fallback_text
accepted for parity, not forwarded (plan-mode stream renders chunks).
* fix(relay): one draft stream per turn for stream-is-the-message adapters
Live-canary finding #4 (Alice, staging): the stream consumer bumps
draft_id at every tool boundary so Telegram-shaped drafts animate each
text segment as a fresh preview. On relay Slack NATIVE streaming a new
draft_id opens a brand-new chat.startStream — the user saw one frozen
message per segment (stuck streaming cursor ▉, never sealed: only the
LAST stream gets the final=true seal) plus the real final; 5-6 cumulative
snapshots per turn. Adapters that mark draft_stream_is_message keep ONE
stream per turn: tool progress lives in the native task card, and the
connector's suffix-delta falls back to whole-text append on prefix
mismatch, so segments append cleanly. Telegram-shaped drafts keep the
per-segment bump.
* fix(relay): don't seal the native stream at tool boundaries — only the turn-final does
Live-canary finding #5 (Alice; supersedes the incomplete #4 which was
necessary but not sufficient). Root cause CONFIRMED by integration trace
(test_live_cards_flow_trace.py, real consumer semantics + real adapter +
stub transport): at every tool boundary the consumer calls
_send_or_edit(finalize=True), which skips the draft path and issues a
real send(); the relay adapter's seal-interception converts THAT into
draft(final=true) — sealing the stream once per segment. Timeline showed
3 seals for a 3-segment turn: exactly the frozen cumulative ▉ snapshots
seen live (the replaced stream never gets stopStream, keeping its cursor).
Fix: for draft_stream_is_message adapters, a segment-break finalize
(finalize=True, is_turn_final=False) stays ON the draft path as another
cumulative frame; only got_done (is_turn_final=True) falls through to
send() and seals. Telegram-shaped platforms unchanged. Trace test now
pins the invariant: ONE user-visible message per turn.
* fix(relay): strip the text cursor from native draft frames
Live-canary finding #6 (Alice) — the ACTUAL duplicate-content mechanism,
confirmed by full-flow scan of both sides' code + logs. The consumer
appends its text cursor (▉) to every non-final display_text tick. The
connector's stream sender diffs CUMULATIVE frames via prefix check:
'abc▉'.startsWith → 'abc def▉' is NEVER a prefix match (the cursor sits
mid-string), so deltaFor falls back to whole-text append on EVERY tick —
chat.appendStream stacks each full cumulative snapshot (cursor included)
into the ONE stream message. Exactly the observed thread: repeated
blocks, each ending in a frozen ▉, growing per tick.
Fixes #4/#5 were real (one stream per turn now) but this was the last
mechanism standing. Native streams render their own typing indicator, so
the text cursor is pure noise on this path: strip it from draft frames.
Prefix check now holds; every tick appends only its true suffix delta.
* fix(relay): seal-interception covers EVERY egress door, not just send()
Live-canary finding #7 (Alice): one duplication remained after #6 — the
stream froze mid-word with the live indicator (never sealed) and the
final posted as a separate message. Log receipt: 'Queued follow-up:
final text delivery confirmed; delivering explicit media before
continuing' — the turn's final went out via the DELIVERY RESOLVER lane
(gateway/delivery.py), which calls send_for_platform() DIRECTLY,
bypassing send() and its seal-interception. The open stream never
absorbed the final; it arrived as a plain 'send' op → chat.postMessage.
Fix: hoist the open-draft check to the top of send() (ahead of the
explicit-platform branch) AND add it to send_for_platform() — an open
native stream absorbs the turn-final regardless of which egress door it
arrives through. The stream IS the message.
* fix(relay): failed seal falls back to plain send (PR 85796 AI-review point 1)
A turn-final seal that fails at the transport must never swallow the
final answer: the stream consumer has already disabled the draft
transport for the run, so a failed _seal_open_draft returning
success=False meant the user got NOTHING. Both seal-interception sites
(send + send_for_platform) now fall through to the regular plain-send
path on seal failure, with a warning receipt. Also mitigates AI-review
point 2 (sticky _open_draft_by_chat after an abandoned turn): a stale
entry's failed seal no longer blocks the next turn's delivery.
* fix(relay): arm seal-interception optimistically; never disarm on ambiguous failure (audit G-D1)
Deep-audit defect G-D1 (HIGH): the outbound leg is at-most-once on the
wire but its ack channel is lossy — send_outbound timeout (30s) and
WS-drop 'failures' frequently mean the frame WAS delivered and the
connector stream is open. send_draft popped _open_draft_by_chat on any
failure, disarming seal-interception while the connector stream lived:
the turn-final went out as a plain send → orphaned mid-word stream +
complete duplicate final (intermittent; needs a drop/timeout inside the
draft window).
Fix: arm the entry BEFORE the transport call and keep it armed on
failure/exception. Safe in every case: sealing a non-existent stream
opens+seals a single complete message connector-side, and a truly failed
seal already falls back to plain send at both interception sites.
Stale-entry damage is self-healing (one warning + plain send).
* fix(relay): gateway-side sealed-draft tombstone — G-D1 arming must not resurrect sealed streams
Regression fix on G-D1 (live: 'worse than before' — escalating frozen
prefixes). Optimistic arming had no seal-awareness: a straggler frame
arriving AFTER the seal re-armed _open_draft_by_chat for the already-
sealed draft_id; the next send was converted to draft(final=true) on the
tombstoned connector key, which CLEARED the connector tombstone (final
frame = new-turn signal), re-opened a stream with cumulative content,
and left it frozen — repeating per straggler: 4-5 escalating frozen
snapshots. Mirror the connector: _sealed_draft_by_chat records the
sealed draft_id per chat (tombstoned BEFORE the seal's transport call);
send_draft for a sealed draft_id is a success no-op (content already in
the sealed message) and never arms. A new turn's fresh draft_id arms
normally.
* fix(relay): key stream/card state per (chat, turn anchor) — parallel turns must not collide (finding #10)
Live finding #10 (Alice; three concurrent turns in one flat DM): all
coordination state was keyed per CHAT on a one-active-turn assumption.
Three parallel turns produced: turn B's task card merged into turn A's
(both were card 'turn:root' — reply_to is None in flat DMs), B left
cardless, and _open/_sealed_draft_by_chat clobbered across writers (3x
duplicate finals on the last turn). Per-turn machinery was correct;
the keys were not.
Fix: _draft_key(chat, metadata) = chat + the turn's thread anchor
(inbound stamps thread_ts = event.thread_ts or ts on every top-level
message, so each turn has one even in flat DMs). draft arming, seal
tombstones, both interception sites, and the task-card id all derive
from the same anchor. New trace test pins two interleaved turns:
distinct cards, own-stream seals, no leaked plain send, no cross-turn
tombstone drops (289 tests green).
* fix(gateway): preserve cumulative native stream across tools
* fix(gateway): consumer-declared final — the seal carries the true final
Three composed fixes for the Slack live-cards duplicate-final class:
1. finish(final_text): TurnRunner passes the completed final_response
(verifier footer, completion explainer included) as the authoritative
finalize payload. The native-stream seal delivers the TRUE final, so
post-stream mutation no longer forks a corrective plain send (#11).
2. Interim-send contract: commentary and segment-tail sends carry a
gateway-internal _interim_send marker; relay seal-interception skips
them at both egress doors. A mid-turn interim send can no longer seal
the live stream and orphan the real final into a duplicate.
3. Queued-follow-up lane reconciles an unconfirmed final by EDITING the
consumer's delivered message in place (sealed stream = regular
message, chat.update live-verified); plain send only as fallback.
This was the actual duplicate lane in the parallel canaries — every
duplicated turn logged 'final stream delivery not confirmed; sending
first response' (subagent-completion queued inbound), not parallelism.
Also: draft frames stay prefix-stable gateway-side (no fence-closing, no
segment state reset, no commentary reset for stream-is-the-message
adapters; MagicMock-safe 'is True' guards).
* test+docs: streaming-contract coverage completeness + maintenance guidelines
Coverage: two gaps closed on the consumer-declared-final contract —
(1) send_for_platform (the delivery-resolver egress door) honors the
_interim_send contract: no seal, marker stripped before the wire;
(2) finish(final_text) on a turn that never streamed does not adopt the
final (delivery ownership stays with the gateway's normal send path for
non-streaming models / tool-only turns).
Docs: AGENTS.md 'Known Pitfalls' gains the streaming delivery contract —
the four invariants of stream-is-the-message adapters (prefix-stable
frames, consumer-declared final, interim-send marker, reconcile-by-edit),
each traced to its live incident, plus the live-probed Slack streaming
API ground truth and the MagicMock 'is True' guard-style note.
* fix(relay): seal transport failure must never silently lose the final (review B1)
Two halves of one silent-loss path, live-probed on the review branch:
1. adapter: _seal_open_draft did not catch transport exceptions. A socket
drop at seal time raised out of send(), skipping the fail-open plain
send entirely. Now: retry the SAME idempotent final frame once (the
connector's sealed-key tombstone returns the original stream ts for a
repeated final — a retry can never open a second stream or duplicate),
then report failure so the caller's fail-open path runs.
2. consumer: the turn-final retry (elif not _already_sent) called
_send_or_edit with finalize=False, which re-entered the DRAFT-FRAME
branch. Its no-op dedupe compared the adopted final against the last
unsealed frame, matched, and returned True with ZERO transport calls —
final_response_sent went green, delivered_final_matches reconciled,
the gateway suppressed its fallback, and the user never received the
answer. finalize=True keeps this retry out of the draft branch.
Regression suite: tests/gateway/test_relay_seal_failure.py (3 tests).
Mutation evidence in follow-up verification: reverting either half sends
the suite red.
* fix(relay): draft ids unique across gateway incarnations (review B3)
The relay connector tombstones sealed streams by (channel, draft_id) and
keeps up to 512 of them; they outlive the gateway process. Relay gateways
are disposable BY DESIGN (scale-to-zero), and _draft_id_counter restarted
at zero every incarnation — so the first turns after every scale-from-zero
in a recently-active channel replayed already-sealed wire identities. The
connector answered those frames straight out of the old tombstone: zero
Slack API calls, the OLD message ts returned as the new turn's identity,
the new answer silently dropped while gateway-side flags recorded success.
Seed the counter from wall-clock milliseconds at process start. Ids stay
plain ints within the existing contract op; incarnations cannot overlap
for realistic turn counts and restart gaps.
Regression: tests/gateway/test_draft_id_restart_uniqueness.py — the seed
test fails on the old code (seed 0 is not epoch-scale).
* fix(relay): stream/card state keyed per TURN, not per thread anchor (review B2)
The thread anchor is the wrong coordination identity — simultaneously:
- too coarse: two parallel turns replying INSIDE ONE Slack thread share
thread_ts. Live-probed on the review branch: turn A's final sealed turn
B's stream with A's content while A's own stream stayed open, and B's
final degraded to a plain send.
- too fragile: a flat DM with no thread metadata degraded to the bare
chat id, re-creating the original finding-#10 collision the anchor was
meant to fix.
_draft_key now prefers the triggering inbound message id (message_id /
reply_to_message_id — per-turn by construction; the gateway's Slack
thread metadata and the consumer's send path both stamp it), falling back
to the thread anchor, then the bare chat. The consumer stamps the same
reply_to_message_id on draft frames so frames and the turn-final resolve
to one key. Task-card ids share the derivation via _card_key (one helper
for send AND stop, so the stop always hits the stream the send opened).
Legacy resolver-lane callers with placement-only metadata still seal via
_match_open_draft's fallback — but ONLY when exactly one stream is open.
With several open, an identity-less send stays a plain send: a duplicate
message is recoverable, sealing someone else's stream is not.
Regression: tests/gateway/relay/test_relay_turn_keying.py (7 tests).
* fix(relay): stream-is-the-message is a Slack semantic, gate it on the descriptor (review B4)
draft_stream_is_message was hardcoded True on the relay adapter class,
i.e. for EVERY relay platform. The base send_draft contract is
Telegram-shaped — the draft clears client-side and the final arrives as
a separate real send that becomes the history message. With the flag
forced on, any non-Slack connector advertising the draft op had its
turn-final intercepted into draft(final=true): probed on the review
branch with a telegram descriptor, the op stream was
[draft(final=false), draft(final=true)] and NO send — no history message
would ever be posted.
Gate the flag on the negotiated descriptor platform (slack), and skip
arming seal-interception entirely when it is off. A future platform with
genuine stream-is-the-message native streaming should advertise it via
the descriptor rather than widening the platform check by guesswork.
Regression: tests/gateway/relay/test_relay_stream_semantics_gating.py
(4 tests: gating both ways, telegram final is a real send, slack final
still seals).
* fix(gateway): mark every mid-turn status lane interim — heartbeats must not seal the stream (review B5)
Seal-interception treats the first unmarked send to an armed (chat, turn)
key as the turn-final. The consumer's own interim lanes (commentary, tail
flush) carry _interim_send, but four gateway-side lanes that fire DURING
a streaming turn did not:
- long-running heartbeat (default every 180s — probed live: at 3 minutes
it sealed the live stream with '⏳ Working — 3 min', the real final
posted as a duplicate, and later frames were silently swallowed by the
seal tombstone)
- inactivity warning
- plain-text approval fallback (button lane failed)
- background-review notice
Add _interim_metadata() beside _non_conversational_metadata and wrap all
four call sites. The marker is gateway-internal; the relay adapter strips
it before the wire (existing behavior, pinned by test).
Note for follow-up: the opt-out shape remains fragile — any FUTURE
unmarked mid-turn send lane re-creates this bug. Inverting the contract
(explicitly mark the one turn-final send) is the durable fix but touches
every adapter's final-delivery path; deliberately kept out of this
review-fix series.
Regression: tests/gateway/test_interim_send_lanes.py (4 tests).
* fix(gateway): interrupted/incomplete turns must not adopt the diagnostic as the stream final (review B6)
The finish(final_text) adoption gate checked only 'not failed', but the
interrupt/abort returns in agent/conversation_loop.py are
{completed: False, interrupted: True, final_response: 'Operation
interrupted during …'} with NO failed key. Adopting that diagnostic:
1. sealed the user's streamed partial answer over with the interrupt
text (stream-is-the-message: the seal rewrites the whole message), and
2. recorded the diagnostic as the turn-final payload, so
delivered_final_matches reconciled and the gateway suppressed its own
error-delivery path — the diagnostic became the ONLY thing delivered.
Enumerated all 27 final_response-bearing return shapes in
conversation_loop.py: every non-happy-path shape carries completed:
False (several with a diagnostic final_response and neither failed nor
interrupted — retry exhaustion, truncation, codex-incomplete); the happy
path routes through turn_finalizer.finalize_turn (completed=True). Gate
is therefore: not failed AND not interrupted AND completed is not False.
Results lacking the completed key entirely (older callers/test doubles)
keep the previous behavior.
Regression: tests/gateway/test_stream_final_adoption_gate.py (6 tests,
incl. a source-level pin on the run.py call site).
* fix(relay): task-card transport failures degrade to failed SendResults (review B7)
send_native_task_card_progress and stop_native_task_card_progress let
transport exceptions escape. The stop runs inside the progress loop's
finally block on the turn-cleanup path, and the post-cancel awaits in
gateway/run.py caught only CancelledError — a socket drop during a card
publish/stop therefore aborted cleanup BEFORE the final-delivery
bookkeeping ran.
Three layers, outermost defends any adapter:
- both adapter methods catch transport exceptions and return failed
SendResults (progress is advisory; the TurnRunner's text fallback
already handles failure results)
- the progress loop's finally wraps the stop (best-effort; the connector
seals orphaned card streams on its own via recycling/eviction)
- the cleanup awaits log-and-continue on non-cancellation errors so
final-delivery bookkeeping always runs
Regression: tests/gateway/relay/test_relay_task_card_failures.py.
* fix(relay): a dying turn seals its native stream instead of orphaning it (review B8)
Stale-generation exits (/new, /stop mid-stream) and cancellations
returned from the consumer's run() with the native stream still open:
- the Slack message kept its live streaming indicator forever (the
cancellation best-effort edit only runs when _message_id exists, and
the native draft path deliberately keeps it None);
- the adapter's armed interception state survived the turn, so the next
turn on the same key could inherit it and seal a dead draft_id.
New adapter op abandon_open_draft(chat, content): seals in place with
the text already on screen (the consumer passes its last delivered
frame) — the seal adds nothing and claims nothing; delivery flags are
never set, so the gateway's normal paths still own whatever happens
next. Best-effort by contract (failure reported, never raised); the
connector reaps truly orphaned streams via recycling/eviction.
The consumer calls it from both death paths: the stale-generation early
return and the CancelledError handler.
Regression: tests/gateway/test_stream_abandon_on_turn_death.py (4 tests,
incl. the next-turn-inheritance hazard).
* fix(relay): bound the draft/seal coordination dicts (review M1)
_sealed_draft_by_chat's key embeds a per-turn identity, so every
completed turn wrote a permanent entry — unbounded growth for the life
of a long-running gateway process (the docstring said 'one entry per
chat', which stopped being true when the key gained the turn anchor).
_open_draft_by_chat could grow the same way via abandoned entries.
FIFO-evict both at 512 entries — the same idiom as the sibling bounded
cache (_auto_thread_by_chat, capped at 256) and the same size as the
connector's own tombstone store. The straggler window the tombstone
exists for is seconds long; FIFO is more than enough.
Regression: tests/gateway/relay/test_relay_state_bounds.py.
* fix(relay): explicit connector rejection disarms interception; exceptions stay armed (review P3)
The G-D1 optimistic-arming change silently dropped disarm-on-failure
entirely: after an EXPLICIT connector rejection (success=False result —
not a transport ambiguity), interception stayed armed even though the
stream consumer disables the draft transport on that failure and falls
back to edit-based streaming. Its turn-final would then be converted
into a seal on a stream the connector just told us is unusable.
test_draft_failure_result_propagates claimed to cover this ('must NOT
leave seal-interception armed') but passed for an unrelated reason: the
stub's canned failure also failed the SEAL, whose fail-open path did the
plain send.
Split the two semantics and pin each honestly:
- explicit rejection (result success=False): disarm — turn-final is a
real send (test_draft_failure_result_propagates, now testing what its
comment says)
- transport exception: ambiguous, stay armed — turn-final still seals
(test_draft_transport_exception_keeps_interception_armed, the G-D1
contract)
Also corrects commit ba3a24a's claim ('a failed frame disarms
interception so the edit-based fallback's real send goes through
untouched') to hold again for the rejection case it described.
* fix(relay): lost acks are ambiguous, not rejections — on the RESULT channel too (review r2, finding 1)
The production ws transport does not raise on ack timeout — it returns
{"success": False, "error": "relay outbound timed out"}. The round-1
ambiguity handling keyed entirely on the exception channel, so the shape
production actually produces was misclassified as a definite connector
rejection. Probed on the head:
- lost SEAL ack: skipped the idempotent retry, fell straight to a plain
send — duplicate final whenever the seal had actually applied;
- lost FRAME ack: the round-1 disarm-on-rejection fired — interception
disarmed, frozen native stream beside a plain final. This re-created
the original G-D1 ambiguous-ack defect on the result channel.
Contract now spans both channels:
- transport: the ack-timeout branch tags ambiguous=True. The fail-fast
branches (closing / not connected) never sent anything and stay
unmarked — they are definite non-delivery.
- adapter frame path: ambiguous results keep interception armed (same
as exceptions); only definite rejections disarm.
- adapter seal path: one shared _attempt() classifier — exception and
ambiguous result both mean "unknown"; the SAME idempotent frame is
retried once (connector tombstone returns the original stream ts for
a repeated final). Only after both attempts stay ambiguous does the
caller's fail-open plain send run: a possible duplicate after double
ack loss beats a silent loss, and double ack loss on one socket
almost always means the transport is down for the plain send too.
Regression: tests/gateway/relay/test_relay_ack_ambiguity.py (6 tests,
incl. a source-of-truth check that the transport tags the timeout branch
and leaves fail-fast branches unmarked).
* fix(relay): stream semantics + draft capability resolve per CHAT, not per primary (review r2, finding 2)
One RelayAdapter fronts N platforms (Phase 1.5): descriptors accumulate
per platform on the transport and egress is tagged per chat — but the
round-1 gate keyed draft_stream_is_message and supports_draft_streaming()
off the PRIMARY scalar descriptor. Probed on the head:
- Slack primary + Telegram chat: the Telegram chat's turn-final was
intercepted into draft(final=true) — no real Telegram history message;
- Telegram primary + Slack chat: the Slack chat was denied native
streaming entirely.
Resolve both through _descriptor_for_chat — the same per-chat machinery
max_message_length already uses (added for the identical class of bug:
the primary's 39000-char cap over-sending into Discord 400s):
- new stream_is_message_for_chat(chat_id) on the adapter; arming and
NotImplementedError gating use it. The class attribute remains as the
single-platform value and legacy-probe fallback.
- supports_draft_streaming() gains an optional chat_id kwarg (base
signature updated; single-platform adapters ignore it). The consumer
passes chat_id with a TypeError fallback for out-of-tree adapters.
- the consumer's four draft_stream_is_message reads collapse into one
_stream_is_message() helper that prefers the per-chat probe
(class-resolved, MagicMock-safe) over the attribute.
Platform-name inference ("slack") stays deliberate: a descriptor-level
semantic field is the right eventual contract but is a cross-repo wire
change — noted for the gg follow-up so future platforms advertise the
semantic explicitly.
Regression: tests/gateway/relay/test_relay_multiplatform_semantics.py
(5 tests: both starvation directions, scalar fallback, per-chat
capability gate).
* fix(gateway): split delivery + authoritative footer reconciles by suffix, not full resend (review r2, finding 3)
The _FINAL_TEXT adoption guard refuses wholesale adoption on split turns
— correct (#78541: sealed heads would repeat inside the tail) but it was
absolute: a post-split verifier footer never entered the ledger,
delivered_final_matches() reported a mismatch, and the gateway resent
the ENTIRE body+footer after the split chunks (the #11 duplicate class,
one level up).
When the authoritative final strictly prefix-extends the split ledger,
the missing suffix is the only undelivered content: append it to the
live tail and the ledger, so the finalize carries it and the recorded
payload reconciles. Non-prefix rewrites keep the full-resend fallback —
a rewrite cannot be patched onto sealed heads.
Regression: tests/gateway/test_split_final_suffix_reconcile.py (3 tests:
suffix rides the tail + reconciles, rewrite still mismatches, unsplit
adoption unchanged).
* fix(relay): cancellation mid-seal restores open state so abandon can close the stream (review r2, finding 4)
_seal_open_draft pops the open entry and writes the local tombstone
BEFORE awaiting transport I/O — correct ordering for the straggler race,
but CancelledError is not an Exception: a cancel during the await
bypassed all failure handling, leaving the remote stream live (visible
streaming indicator until connector eviction) while the local state said
'nothing open'. The consumer's abandon pass — added for exactly this
turn-death case — found nothing to close and no-oped.
On CancelledError: restore the open entry, drop the premature tombstone
(only if it is still ours), re-raise. The abandon path then seals the
stream in place with the on-screen text.
Regression: tests/gateway/relay/test_relay_seal_cancellation.py (2
tests: state restoration, and end-to-end cancel→abandon→remote seal).
* fix(relay): thread anchors are placement, not turn identity — revive the placement-only fallback (review r2, finding 5)
_match_open_draft's single-open-stream fallback was dead for its primary
intended callers: metadata carrying thread_ts/thread_id (placement-only
resolver lanes) was classified as having 'turn identity', so those sends
never reached the fallback — probed: a plain final posted beside the
still-open turn-keyed stream.
Only per-turn MESSAGE ids are identity now. Thread-anchored and bare
callers share the fallback: absorb into the chat's open stream when
EXACTLY one is open; stay a plain send when several are (duplicate is
recoverable, wrong-stream seal is not). Callers WITH a message id whose
key misses never fall back — their identity is authoritative and a miss
means the stream belongs to a different turn.
Regression: 4 new tests in test_relay_turn_keying.py (thread-anchored
seal, both ambiguous-stay-plain shapes, id-mismatch never steals).
* fix(relay): random process nonce for draft-id seeding (review r2, follow-up 6)
The epoch-millisecond seed (round-1 B3 fix) mitigates the restart-replay
class but is not a uniqueness guarantee: two gateways starting in the
same millisecond, a forked process inheriting the class state, or a
clock step backwards can all mint colliding wire identities against the
connector's per-(channel, draft_id) tombstone store.
Seed from secrets.randbits(49) instead: collision probability negligible,
no clock dependence, and ids + realistic per-process turn counts stay
comfortably inside the connector's JS number range (draft_id?: number,
2^53). Regression test now spawns two real interpreters and asserts
their seeds differ — the exact scale-to-zero restart shape, and both
start within the same second so a clock-locked seed would fail it.
* fix(relay): stamp per-turn Slack egress identity — cache is fallback only (R3-5)
The connector (gateway-gateway#210) fills chat.startStream's
recipient_user_id / recipient_team_id — required by Slack when
streaming to a channel — from metadata.user_id / metadata.scope_id.
The gateway stamped only slack_team_id per-turn and left user_id (and
scope_id) to RelayAdapter._with_scope, whose per-chat caches are keyed
on chat_id alone and overwritten by every inbound message: with users
U1 and U2 running overlapping turns in one channel, U2's arrival
overwrote the cache before U1's stream opened, and U1's stream carried
U2 as recipient_user_id.
_thread_metadata_for_source now stamps scope_id and user_id from the
turn's OWN source (setdefault — explicit values win), so identity is
turn-scoped data on the wire. _with_scope is unchanged and fill-only:
the caches keep serving restart/synthetic sends that carry no per-turn
identity, which is all they were ever safe for.
Mutation evidence: reverting the run.py hunk sends
test_thread_metadata_stamps_per_turn_user_and_scope and
test_concurrent_turns_carry_their_own_identity red; restore returns
green. The _with_scope fill-only tests pass on both trees (existing
correct behavior, now pinned against regression).
---------
Co-authored-by: Ben Barclay <ben@nousresearch.com>
|
||
|
victor-kyriazakos
|
6a7cf19302 |
fix(gateway,relay): stop frozen-preview finals and dropped idle-session delegation callbacks (#82592)
* fix(gateway): stop frozen-preview finals and dropped idle-session delegation callbacks
Two relay-plane delivery losses from the 2026-08-09 staging incident:
1. stream_consumer: the skip-redundant-finalize branch recorded _accumulated
as the delivered turn-final payload even when the last ACKED edit was an
earlier throttled preview snapshot, so delivered_final_matches reconciled
True and the gateway suppressed the corrective final send — the user was
left with a cut-off message ending in the streaming cursor. Extracted
_mark_skip_redundant_finalize(): records the last acked wire payload
(cursor-stripped), so a preview/final mismatch now returns False and the
normal final send fires.
2. run.py: _classify_completion_target classified every ended parent session
terminal unless it ended by compression. Idle/timeout session ends are the
norm on scale-to-zero relay deployments and the chat route remains valid;
completed async delegation results were terminally dropped. Ended parents
now classify deliver unless the end was an explicit user boundary
(session_reset / user_exit / session_switch).
* fix(relay): drain in-flight outbound frames before transport teardown
disconnect() failed every pending outbound future immediately with
'relay transport closed', so a trailing finalize edit racing turn
teardown was lost even though the connector socket could still serve
it. Bounded drain grace (5s) lets in-flight requests resolve; silent
connectors still tear down promptly. asyncio.wait (not gather+wait_for)
so a timeout doesn't cancel futures owned by the fail-remaining loop.
* fix(gateway): route completion injection through the alias-aware transport resolver
Third relay-plane delivery loss from the 2026-08-09 staging incidents: a
delegation batch completed while the gateway was up, the watcher drained
the event, and delivery vanished with no log line. _inject_watch_notification
resolved its adapter with a literal p.value == platform_name scan of
self.adapters — a relay-fronted gateway registers ONE adapter under
Platform.RELAY fronting N logical platforms, so 'slack' never matched and
the injection returned None ('no gateway route'), silently dropping the
completion. The handoff path already documents this exact trap and uses
resolve_delivery_transport; the injection path now does the same (native
wins; relay eligible only when it fronts the logical platform), with the
literal scan kept as fallback for stub runners and exotic platforms.
* fix(relay): clamp disconnect drain grace to the runner's adapter-disconnect budget
Review finding (JoaoMarcos44, #82592): a fixed 5.0s drain in front of the
three 1.0s sequential teardown awaits gives an 8.0s worst case inside the
runner's 5.0s asyncio.wait_for(adapter.disconnect()) — tripping it cancels
teardown mid-drain, skips the fail-pending loop, and leaves outbound
callers blocked until _OUTBOUND_TIMEOUT_S (30s). The effective grace is
now budget - 3*TEARDOWN - margin (env-aware via the same
HERMES_GATEWAY_ADAPTER_DISCONNECT_TIMEOUT the runner reads), so the drain
can never push teardown past its caller's budget; a budget too small for
any drain disables it cleanly.
* test(gateway): pin the final-send suppression contract across a behaviour matrix
The gateway skips its own final send when the stream consumer claims the turn
final already reached the user. Every incident in that family — #71643 (stale
finalize snapshot), #78541 (payload-less multi-message split), #82656 (frozen
preview left with a visible cursor) — is the same failure: the consumer claimed
delivery for text the platform never rendered, so the corrective send was
suppressed and the answer was lost with no retry.
Each was fixed with a scenario test pinned to one branch of
GatewayStreamConsumer.run(). The got_done handler now has five sibling branches
that each set the suppression flags and record a turn-final payload, and nothing
checks them as a group: a new branch, or a new early `return True` in
_send_or_edit, can reintroduce the class without failing a test.
Pin the invariant instead of the branch — if the consumer offers the gateway any
signal it would trust, the complete final text must have reached the wire — and
assert it across {edit always / dies / never / lies} x {send always / never} x
{fresh-final on / off} x {clean / interrupted stream}.
The adapter records only frames that actually rendered, so an ACK the platform
drops does not count as delivery. 24 honest-transport scenarios hold the
invariant as a hard assertion. The 16 lying-transport scenarios are checked too;
the single combination that still violates it is reported as an expected
failure documenting the open exposure rather than asserting it away.
Refs #82656
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(gateway,relay): prime relay egress routing for synthetic injections + cap stale completion replay
Defect #4 from the 2026-08-09 staging incidents (upgrade-robustness):
after every gateway restart the durable async-delegation replay injected
completions correctly (post-741663cf1) but their replies bounced at the
connector — 'slack egress declined: target not routed to an onboarded
tenant'. The relay adapter re-attaches tenant discriminators
(metadata.scope_id / metadata.user_id) from per-chat caches warmed ONLY by
inbound traffic; synthetic turns race those cold caches on every deploy,
scale-to-zero wake, and crash recovery.
- relay adapter: prime_routing_cache() — feeds a synthetic event's
session-store origin through the same _capture_scope used for real
inbound (never raises).
- run.py injection path: prime the resolved adapter before handle_message
(duck-typed; native adapters unaffected).
- async_delegation: 48h staleness cap in restore_undelivered_completions —
a pending completion older than the cap is terminally dropped (payload
stays queryable) instead of re-run as a fresh full-context turn; the
post-restart replay of a July session burned a 102K-token context.
Also carried: JoaoMarcos44's suppression behaviour-matrix harness
(cherry-picked from #82676, authorship preserved) — 39 passed + 1 xfail
(the documented ACK-then-drop transport-honesty residue).
* test: use recent timestamps in restored-ownership fixtures
test_restore_stamps_restored_flag persisted its completion with epoch-era
toy timestamps (dispatched_at=1.0), which the new 48h replay staleness cap
correctly classifies as stale — the fixture then exercised the cap instead
of the restored-flag contract (CI slice 4 failure). Timestamps are now
now-relative; the staleness behavior itself is pinned separately in
test_relay_injection_egress_priming.py.
* fix(gateway,relay): close four review findings on the relay delivery fixes
Review follow-ups on this branch (NousResearch#82592):
1. HIGH — classifier/resolver mismatch (falsely-acknowledged loss).
_classify_completion_target now returns "deliver" for idle-ended
parents, but _resolve_async_delegation_session still dropped every
non-compression-ended pin: the durable row was acked at adapter
acceptance, then the injection died inside the pipeline with no
retry — strictly worse than the honest terminal drop on main, and
the delivery leg defect #2's fix depends on did not exist. The
resolver now retargets non-user-boundary ends (idle/timeout/
lifecycle) to the chat's current session — session_entry already IS
the routing key's current session for the same chat — while user
boundaries (session_reset / new_session / user_exit /
session_switch) stay fail-closed. Both sides share one module-level
_USER_BOUNDARY_END_REASONS so the verdict and the routing decision
cannot drift again; a coherence test asserts deliver-verdicts
resolve non-None across representative end reasons.
2. HIGH — drain clamp missed adapter-level spend. The effective drain
grace budgeted drain + 3x teardown, but RelayAdapter.disconnect
spends revocation-monitor teardown + go_idle time BEFORE the
transport drain inside the same runner wait_for; worst case still
blew the budget and cancelled teardown mid-drain (skipping the
fail-pending loop). The adapter now measures its own elapsed time
and threads the REMAINING budget into
transport.disconnect(budget_s=...); legacy/stub transports without
the keyword fall back to the no-arg signature.
3. P1 — _request_response racing disconnect() could register a future
after the fail-pending loop already ran, stranding the caller for
the full _OUTBOUND_TIMEOUT_S (30s). Fail fast with the same
"relay transport closed" error once _closing is set.
4. P1 — _build_process_event_source's last-resort reconstruction
dropped scope_id, so a scoped relay completion whose session-store
origin was unavailable primed no tenant discriminator and could
still bounce off the connector's fail-closed egress guard.
scope_id now threads through the reconstructed SessionSource, with
a warning when a scoped chat reconstructs without one.
All four: RED reproduced with the fix reverted, GREEN after; relay/
delegation delivery families pass (43 + 71 + 179 across the touched
suites); full tests/gateway run shows only failures already failing
identically on merge base
|
||
|
teknium1
|
70d165222c |
fix(gateway): cover all finalized stream sends
Apply the same metadata invariant to sealed split chunks, keep expect_edits on live previews, and exercise the real Telegram adapter path with rich messages enabled but rich drafts disabled.\n\nCredits PR #78525 by @Slobaka for the reproduced rich_messages/rich_drafts combination. |
||
|
liuhao1024
|
0f22727167 |
fix(gateway): omit expect_edits on finalized draft sends
A finalized native draft is the first persistent send and will not be edited again. Omitting expect_edits lets Telegram use sendRichMessage for the persistent final instead of degrading tables through MarkdownV2.\n\nAdapted from PR #46536. |
||
|
kshitij
|
68ebb198c7 |
fix(gateway): don't claim deleted head chunks as delivered in the empty-fallback recovery
Follow-up to #79669. That PR routed the three fallback recorder sites through _record_turn_final_payload so a split turn would record the unsplit ledger instead of a tail-only payload. For two of them that is right. For _send_empty_fallback_final it is wrong, and it reintroduces the #78541 swallow at the one site that was supposed to be fixed. _send_empty_fallback_final is a *replacement* recovery: it sends the completed text as a fresh message and deletes every tracked segment preview -- which on an overflow split includes the sealed head chunks. After it runs, the only thing on screen is the message it just sent. Recording the ledger there claims delivery for text the same function just removed, so delivered_final_matches() returns True, the gateway suppresses its own send, and the user is left with a fraction of the answer. Observed with a probe driving the real run() loop (543-char reply, 475-char head sealed then deleted, 67-char tail committed): before this fix recorded=543 matches=True -> suppressed, 67/543 on screen after this fix recorded=67 matches=False -> gateway sends the full answer Record final_text verbatim here instead. The sibling site in _send_fallback_final keeps the recorder: its delete is gated on `continuation == final_text` and targets only the single active partial, never the sealed heads, so the ledger correctly describes what survives. The distinction is whether a recovery ADDS to what is on screen or REPLACES it. Additive paths may record the ledger; replacing paths must record only what they leave behind. _try_fresh_final is the same shape and #79669 handled it by refusing the route on split turns. Test drives the real seal-then-delete sequence and asserts the mismatch, so the gateway is required to re-send. Mutation-checked: restoring the recorder call turns it red. |
||
|
kshitij
|
392e3a8c53 |
fix(gateway): finish the split-delivery bug class so the fix cannot duplicate or still swallow
The salvaged fix changed only the gateway's verdict: a payload-less multi-message split stopped inheriting legacy trust. But six code paths set _turn_split_delivery, and only one of them was taught to record a payload, so the remaining five swapped the swallow for the opposite defect. Fix the producers instead of only distrusting them at the boundary: - _send_or_edit failed-final-edit branch: record the visible payload on split turns too. It deliberately skipped recording, which now reads as a mismatch and re-sends an answer already on screen -- reintroducing the duplicate #45517 fixed (#36965 / #25349). - _send_fallback_final (x2) and _send_empty_fallback_final: route through _record_turn_final_payload instead of assigning _delivered_final_text directly. On a split turn their final_text is only the trailing chunk, so a fully delivered heads+tail reply recorded a tail-only payload and was re-sent in full. - _try_fresh_final: refuse the fresh-final route once a head chunk is sealed. It replaces every tracked preview with one message, which only holds the whole answer on a single-message turn. After a split it deleted the sealed heads while sending just the tail, so the complete reply was still lost -- on Telegram, the default finalize route and the shape #78541 reports. - Set _turn_split_delivery at seal time rather than after the tail send, so the tail's own finalize sees the split state. The sibling overflow path already did this; the divergence is what let fresh-final delete the heads. - run.py stale-finalize reconciliation: skip the in-place edit on a split delivery. message_id is only the LAST chunk there, so editing it with the complete response repeated every sealed head's text inside the tail message. Fall through to the normal final send. Also drop a dead `or "".join(chunks)` fallback (all growth funnels through _append_accumulated, so the ledger is never empty at that call site, and joined chunks carry injected fence markers that could never match final_response), and document that _record_turn_final_payload intentionally ignores its argument on split turns. Tests: four end-to-end cases driving the real overflow-split loop instead of hand-setting private flags -- complete split still suppresses (no duplicate), split missing a tail does not suppress, fresh-final keeps sealed heads, and a flood-controlled final edit after a split stays suppressed. Each was mutation-checked: reverting any individual fix turns its test red. The pre-existing gateway-boundary test asserted the recovery *route* (the reconcile edit) rather than the guarantee. Relaxed to the real contract: either _run_agent puts the complete text on the wire, or it declines to claim delivery so the caller's normal final send does. Co-authored-by: HexLab98 <liruixinch@outlook.com> |
||
|
HexLab98
|
c46027b049 |
fix(gateway): stop payload-less split delivery from swallowing finals
Record an unsplit stream ledger for multi-message deliveries and refuse legacy trust when split delivery left no payload, so Telegram group sessions no longer suppress a complete reply after an early/partial finalize (#78541). |
||
|
Teknium
|
30878411b8 |
fix(gateway): stop stale streamed finalize from suppressing the complete Telegram response
A successful finalize edit can carry only the last streamed preview snapshot: deltas generated between the last preview edit and stream completion never reach any Bot API call, yet final_response_sent / final_content_delivered were set from the call's success and suppressed the gateway's normal final send — losing the tail permanently. The stream consumer now records the exact cleaned payload of every turn-final delivery (delivered_final_matches tri-state), and gateway/run.py reconciles that record against the completed final_response before trusting either suppression flag. On a demonstrable mismatch it edits the streamed message up to the complete response, falling back to the normal final send if the edit fails. Multi-message split deliveries and legacy paths without a record keep the existing flag-trusting behavior, so overflow splits and the failed-finalize handling (#51828/#33793) are untouched. Fixes #71643 |
||
|
teknium1
|
bd93ccb890 | refactor(gateway): shared fence-aware markdown chunker core (yuanbao-derived) + canonical table-row splitter | ||
|
teknium1
|
5b751dc0ad |
chore: remove unused imports and dead locals (ruff F401/F841 sweep)
Cleans F401 unused imports and F841 dead local assignments across root *.py, agent/, hermes_cli/, tools/, gateway/, cron/, tui_gateway/ (tests/, plugins/, skills/ excluded). Intentionally KEPT (false positives / test-patch surfaces): - agent/transports/__init__.py package re-exports - cli.py browser_connect re-exports (DEFAULT_BROWSER_CDP_URL area, used by tests/cli/test_cli_browser_connect.py) - hermes_cli/main.py _prompt_auth_credentials_choice / _model_flow_bedrock_api_key (accessed via main_mod attr in tests) - gateway/run.py aliased replay_cleanup + whatsapp_identity re-exports and _PORT_BINDING_PLATFORM_VALUES (test-referenced) - hermes_cli/web_server.py get_running_pid (tests monkeypatch it) and _OAUTH_TOKEN_URL availability probe - hermes_cli/config.py get_process_hermes_home re-export (noqa'd F811 chain) and yaml availability-probe import - hermes_cli/nous_subscription.py managed_nous_tools_enabled (tests patch hermes_cli.nous_subscription.managed_nous_tools_enabled) - try/except ImportError availability probes (env_loader, tts_tool, mcp_tool, web_server anthropic OAuth block) - tools/web_tools.py noqa F401 re-exports - hermes_cli/setup_whatsapp_cloud.py:263 'proceed' skipped: possible missing-guard bug, flagged for separate review - unused function parameters (signature changes out of scope) Side-effect RHS calls preserved where only the binding was dead (e.g. web_server proc = _spawn_hermes_action -> bare call). |
||
|
Ben Barclay
|
96996a55bf |
fix(relay): per-platform capability descriptors for multi-platform gateways (#70717)
One relay adapter fronts N platforms on one WS, but the capability surface (MAX_MESSAGE_LENGTH / message_len_fn) was a scalar from whichever descriptor resolved the handshake — and the transport's read loop OVERWROTE it on every descriptor frame (last-writer-wins). A Discord chat on a gateway whose applied descriptor was Telegram's inherited the 4,096-char cap and over-sent into Discord's 2,000-char API 400 (observed live: 2,543/2,641-char replies silently lost while inbound kept working). - ws_transport: accumulate one descriptor per platform in _descriptors_by_platform (exposed via descriptor_for_platform); the FIRST descriptor of a connection generation stays the session default instead of last-writer-wins; the map resets on re-dial. - BasePlatformAdapter: new max_message_length_for_chat / message_len_fn_for_chat hooks defaulting to the scalar surface (native single-platform adapters unchanged). - RelayAdapter: overrides resolve the chat's platform from _platform_by_chat (the same map per-frame egress uses) and look up that platform's negotiated descriptor; falls back to the scalar for unknown chats/transports. - stream_consumer (streaming budget, _raw_message_limit, fallback-continuation chunking) + run.py tool-progress limit now resolve per-chat. Tests: tests/gateway/relay/test_relay_per_platform_caps.py (7) — verified fail-without/pass-with (all 7 fail with the fix stashed). Relay + stream consumer suites green (213 + 223). |
||
|
Teknium
|
eaa35ae68f |
fix(gateway): balance code fences on every remaining chunk-split path
Widen #48476's fence guarantees to the two splitters that still emitted fence-broken chunks: * GatewayStreamConsumer._split_text_chunks (fallback final send): close the orphaned ``` at each chunk boundary and reopen it — with the original language tag — on the next chunk, mirroring BasePlatformAdapter.truncate_message's contract. Headroom is reserved so balanced chunks stay within the platform limit. * Slack block_kit._split_text (3000-char section chunking): same close/reopen balancing for mrkdwn section text carrying fences. With these, every chunk boundary — non-streaming send (truncate_message), streaming overflow (_truncate_for_stream via adapter.truncate_message per #45938), fallback final (_split_text_chunks), final-send balance (ensure_closed_code_fences), and Block Kit section splits — delivers fence-balanced chunks. Regression tests probe each path with fenced fixtures, assert per-chunk balance, limit compliance, language-tag reopening, and prose passthrough. |
||
|
2001Y
|
72f714ca36 | fix(gateway): preserve adapter overflow splitting in streams | ||
|
2001y
|
b86ca7cae5 | fix(gateway): keep overflow stream chunks editable | ||
|
skywind
|
b932997f44 |
fix: also close orphaned single-backtick inline-code spans
ensure_closed_code_fences previously only handled triple-backtick (```) code-block fences. Single backtick (`) inline-code spans have the same problem: an orphaned opening backtick causes the remainder of the message to render as inline code on Discord and other platforms. After balancing triple-backtick fences, strip complete ```...``` regions and count remaining standalone backtick markers. If odd, append a closing backtick. Same trade-off as the triple-backtick fix: a stray closing backtick may create a brief empty inline-code span, which is far less harmful than the rest of the message being inline code. |
||
|
skywind
|
3e2fe3fcab |
fix: close orphaned code fences on all send paths
Adds `ensure_closed_code_fences()` helper to detect text with an odd count of triple-backtick markers (indicating an unclosed code block) and append a closing fence. Applies the fix to all four identified gap paths: G1: truncate_message early-break path for final chunk G2: _send_or_edit streaming edit path (most commonly hit) G3: overflow split first chunk (covered by G2's fix) G4: _send_fallback_final fallback send path Closes: #TBD |
||
|
skywind
|
5e44413b7c |
fix: escape triple-backtick in reasoning before wrapping in outer code block
B1: When reasoning content contains ``` e.g. model quoting code in its thinking, wrapping it in an outer ``` for display causes the inner fence to break the outer block. Adds escape_code_fences_for_display() in gateway/stream_consumer.py, called from gateway/run.py before wrapping reasoning in the outer ``` display block. |
||
|
brooklyn!
|
7d28e84e72 |
fix(gateway): deliver assistant prose before the clarify poll (#69775)
* fix(gateway): deliver assistant prose before clarify poll The clarify poll is sent on a separate, agent-thread-blocking path while buffered assistant prose (interim commentary / streamed deltas) sits in the GatewayStreamConsumer queue, drained asynchronously. The poll won the race, so the question rendered ABOVE its own explanation, and a redundant 'clarify: ...' tool-progress bubble wedged between them. - Add GatewayStreamConsumer.flush_pending_sync(): a synchronous flush barrier (_FLUSH sentinel + threading.Event) that blocks the agent thread until everything queued before it is finalized and delivered. - Call it in the gateway clarify callback before send_clarify, so prose always lands before the poll. Best-effort with a 3s timeout. - Suppress the redundant clarify tool-progress bubble (the poll already shows the question + options). Tests: 3 new ordering/timeout cases in test_stream_consumer.py. (cherry picked from commit 9a6e27badb7646c839902db1d2f4a8affcf38b1e) * chore(contributors): map matvey.sakhnenko03@icloud.com -> sakhnenkoff Attribution mapping for the salvaged #54328 commit (Cluster C). --------- Co-authored-by: Matvii Sakhnenko <matvey.sakhnenko03@icloud.com> |
||
|
ethernet
|
c363db81e0 |
fix(desktop, ink): don't wipe messages before final message (#65919)
* fix(desktop): preserve interim assistant text wiped at message.complete
When the agent emits interim text (commentary alongside tool calls, or the
attempted final answer before a verify-on-stop nudge), all UI surfaces
streamed it live but then wiped it at message.complete — keeping only the
final response. The user saw text appear during inference, then disappear.
This is the complete fix across all three layers: agent core, gateway
transport, and all UI surfaces (desktop + Ink TUI).
The verify-on-stop and pre_verify paths flagged the assistant's attempted
final answer as _verification_stop_synthetic, suppressing it from both
state.db and the UI. The user only saw the terse post-verification reply.
Now the assistant response is real content: it's persisted to state.db and
emitted as an interim message via _emit_interim_assistant_message(force_display=True)
before the verification loop runs. Only the synthetic nudge messages keep
the synthetic flags. The turn finalizer drops nudges from live history and
compares content (not just role) to avoid duplicating a published candidate.
Message sequence repair collapses verification candidates in the
consecutive-assistant merge.
Wire agent.interim_assistant_callback both at construction (_agent_cbs())
and per-turn (defense-in-depth), emitting a new message.interim event with
{text, already_streamed}. Gated on display.interim_assistant_messages
(default true). Cleared in the finally block so a stale closure can't
fire on a later turn.
Add message.interim to the GatewayEventName union (apps/shared) and a
typed payload to the TUI's GatewayEvent discriminated union.
The TUI already had the segment-anchoring machinery (flushStreamingSegment +
finalTail) but had no handler for message.interim. Added recordInterimMessage
+ interimBoundaryIndex to seal segments mid-turn, and updated
recordMessageComplete to only dedupe segments after the interim boundary.
Replaced the fragile sealed-set approach with a proper interimBoundaryPending
state flag on ClientSessionState. finalizeInterimAssistantMessage finalizes
the streaming bubble in place (or creates a standalone one), rotates the
stream ID so next deltas create a new bubble, and sets the flag. When the
final text equals an already-sealed interim, they stay as distinct messages.
Extracted mergeFinalAssistantText() as a pure function in chat-messages.ts,
used by both completeAssistantMessage and finalizeInterimAssistantMessage.
Split the bidirectional dedup predicate: reasoning is a restatement only when
the final FULLY covers it. A short final ("Done.") no longer swallows a
longer reasoning block that merely starts with it.
Honor display.interim_assistant_messages (default true) across all layers:
the tui_gateway gates the callback, the desktop wires it to a nanostores
atom via use-hermes-config. Updated hermes_cli/config.py and
cli-config.yaml.example comments to document the Desktop behavior.
_split_segment_tokens now accepts posix=False and _find_ad_hoc_match tries
both posix modes so ad-hoc verification scripts with Windows backslash
paths are matched correctly. (response_previewed forwarding from #53553
is not included — our emit-interim + persist approach makes it unnecessary
since the attempted answer is now surfaced before the verification loop.)
- tsc: clean (desktop + TUI + shared)
- vitest desktop: 73/73 pass (7 interim-sealing + 5 mergeFinalAssistantText + 4 config atom)
- vitest TUI: 83/83 pass (4 new message.interim tests)
- python: 390 tests pass (340 tui_gateway + 33 verification/finalizer + 6 config gating + 3 evidence + 8 continuation budget)
Co-authored-by: Liam Zhang <yingliang-zhang@users.noreply.github.com>
Co-authored-by: Lucas D'Alessandro <lucasfdale@users.noreply.github.com>
Co-authored-by: Eric Manganaro <superposition@users.noreply.github.com>
Co-authored-by: sweetcornna <sweetcornna@users.noreply.github.com>
Co-authored-by: DECK6 <DECK6@users.noreply.github.com>
Co-authored-by: matantsevs <matantsevs@users.noreply.github.com>
Co-authored-by: gitcommit90 <gitcommit90@users.noreply.github.com>
* fix: prefix-match interim streamed content to avoid benign duplicate bubbles
_interim_content_was_streamed used exact equality (streamed == visible_content),
so a final response that was the streamed text plus a trailing delta — or a
partial stream before the verify nudge fired — failed the match and left
_response_was_previewed false. The turn then showed two bubbles (interim +
identical final) instead of settling the interim in place.
Relax to a prefix check (visible_content.startswith(streamed)) in both the
core match and the desktop's settle-in-place gate. The TUI already used
prefix matching via finalTail. The reverse direction (streamed longer than
final) is intentionally not matched — that could suppress a needed resend
in the gateway path where already_streamed=True calls on_segment_break().
* test(desktop): add partial-stream-then-nudge dedup edge case
Third edge case for the interim-sealing dedup: model streams part of its
answer via message.delta, verify nudge fires, interim seals the streamed
prefix, then the final response is the same text plus a trailing delta.
Asserts one bubble (not two) containing the full final text.
Acceptance protocol #2 — covers all three dedup edges:
1. interim == final (existing)
2. interim = strict prefix of final (existing)
3. partial-stream-then-nudge (this commit)
---------
Co-authored-by: Liam Zhang <yingliang-zhang@users.noreply.github.com>
Co-authored-by: Lucas D'Alessandro <lucasfdale@users.noreply.github.com>
Co-authored-by: Eric Manganaro <superposition@users.noreply.github.com>
Co-authored-by: sweetcornna <sweetcornna@users.noreply.github.com>
Co-authored-by: DECK6 <DECK6@users.noreply.github.com>
Co-authored-by: matantsevs <matantsevs@users.noreply.github.com>
Co-authored-by: gitcommit90 <gitcommit90@users.noreply.github.com>
|
||
|
Teknium
|
9412f2dd84 |
fix(discord): persist streamed final delivery
Carry the original reply anchor through stream metadata so a successful final Discord edit marks the recovered source message complete. |
||
|
kshitijk4poor
|
4aa499ff9f |
fix(telegram): harden flood fallback recovery
Keep empty-tail recovery scoped to the current stream segment and bound fallback flood retries. Preserve Telegram's server retry hint without blocking final delivery through a long cooldown. |
||
|
Gille
|
04898631cb | fix(telegram): recover final delivery after stream flood | ||
|
YLChen-007
|
e23f723389 |
fix: make streaming reasoning-tag filter case-insensitive
The streaming think-tag suppressors in cli.py (_stream_delta) and gateway/stream_consumer.py (_filter_and_accumulate) matched tag names with case-sensitive str.find(), so only the exact-case literals in the tag tuples were caught. Mixed-case variants a model may emit — <Think>, <ThInK>, <REASONING>, <Thought> — slipped through and leaked raw reasoning into the user-visible stream. Match against a lowercased view of the buffer with lowercased tag names at all three sites (open-tag boundary search, partial-tag hold-back, close-tag search) in both paths. Only KNOWN tag names are matched — no substring matching — and the block-boundary gating that protects prose mentions of <think> is preserved. - 6 parametrized case-insensitive regression tests in each of tests/gateway/test_stream_consumer.py and tests/cli/test_stream_delta_think_tag.py. Salvaged from PR #27289 by @YLChen-007. |
||
|
testingbuddies24
|
e07768a53f |
fix(gateway): strip orphan think-tag close tags in progressive stream
When a model emits an inline <think>...</think> block but the opening tag is dropped upstream (thinking-mode toggle, truncated stream, or incomplete upstream filtering), the bare </think> close tag leaked through to the user in the live progressive edit. The agent-side final scrubber (agent/think_scrubber.py) already had _strip_orphan_close_tags; this ports the same logic into GatewayStreamConsumer so the streaming display stays clean too. - _filter_and_accumulate: strip orphan close tags before appending the 'no-opening-tag' branch text to _accumulated. - _flush_think_buffer: same on stream end for held-back partials. - 14 regression tests (TestStripOrphanCloseTags): all 6 close-tag variants, multi-tag, partial-tag-untouched, trailing whitespace, and end-to-end through _filter_and_accumulate / _flush_think_buffer. Only strips KNOWN close-tag names (case-insensitive) — never arbitrary tag-shaped substrings — so comparison operators and unrelated prose are preserved. Salvaged from PR #43192 by @testingbuddies24. |
||
|
Ben
|
5f7deeba84 |
fix(gateway): suppress NO_REPLY/[SILENT] markers on the streaming path
The agent emits a bare control marker (NO_REPLY / [SILENT] / …) when it
intentionally chooses not to reply. The gateway's whole-response filter
(is_intentional_silence_agent_result) suppresses this on the non-streaming
delivery path, but the streaming path (GatewayStreamConsumer) had no silence
awareness: it edited the raw marker onto the screen delta-by-delta and
finalized it BEFORE the whole-response filter could run. On any
streaming-capable adapter (Slack, Telegram, Discord, …) users saw a literal
'NO_REPLY' message leak into chat.
Fix (contained in the stream consumer + a shared predicate; no new config,
no platform-specific code):
- gateway/response_filters.py: add is_partial_silence_marker() — the
streaming counterpart to is_intentional_silence_response(), sharing the
same marker set and canonicalization so the two never drift.
- gateway/stream_consumer.py:
- Mid-stream hold-back: defer edits while the accumulated buffer is still a
prefix of a silence marker, so a partial marker never flashes on an
interval tick.
- On stream end (got_done): if the final buffer is exactly a marker, retract
any preview already shown (best-effort delete_message, reusing the
_try_fresh_final cleanup path) and leave the delivery flags False so the
gateway's own filter turns the marker into '' and no fallback send fires.
Substantive prose that merely mentions a marker is still delivered normally.
Tests: tests/gateway/test_stream_consumer_silence.py — predicate truth table
+ end-to-end run() suppression (single-shot + token-by-token), preview
retraction, no-delete-support best-effort, [SILENT] parity, and
prose-passthrough. Prove-fail verified by reverting only the consumer change
(the 4 behavioral tests fail: 'NO_REPLY'/'[SILENT]' leaks).
|
||
|
HexLab98
|
6da181062b |
fix(gateway): deliver MEDIA tags for extension-less files when path validates
Files like Caddyfile or Makefile have no extension, so MEDIA_TAG_CLEANUP_RE never matched them and Telegram showed the raw MEDIA: line as text. Extract and strip validated extension-less tags via a second pass. |
||
|
jasonQin6
|
6dd188d786 |
fix(gateway): add session staleness guard to stream consumer
GatewayStreamConsumer.run() processed queued deltas in an infinite loop
with no check on whether the session was still current. On /new or /stop
mid-stream, the consumer kept editing and delivering stale response
fragments alongside the 'Session reset!' ack.
PR #11016 (
|
||
|
sgaofen
|
194bff0687 |
fix(gateway): confirm final delivery before suppressing send
Fixes #14238. During a compression/session split at the response boundary, the interim callback delivered unrelated commentary, setting response_previewed=True. The suppression logic treated that as proof the final reply had been delivered and skipped the normal send — the response was persisted to the child session but never sent to chat. Only suppress the normal final send when the stream consumer confirms final delivery (final_response_sent / final_content_delivered) or the exact final response text was delivered as a preview. |
||
|
liuhao1024
|
b5b8a4cd56 |
fix(gateway): respect adapter decline of fresh-final to prevent double delivery
When a streamed Telegram reply finalizes, the stream consumer could take the fresh-final path (send a new sendRichMessage + best-effort delete the preview) purely because the time-based _should_send_fresh_final() threshold elapsed — even though Telegram's prefers_fresh_final_streaming returns False. The fresh Rich Message then overlapped the legacy MarkdownV2 preview already on screen, leaving both visible (the #47048 table + bullet double-render). Honor the adapter's decision: when prefers_fresh_final_streaming exists on the adapter (checked on the class + instance __dict__ so MagicMock auto-attrs don't false-positive) and declines, the time threshold no longer overrides it. Adapters without the hook keep the time-based fresh-final for backward compat. Fixes #47048 |
||
|
LeonSGP43
|
09a96ba0f6 |
fix(gateway): pause Telegram typing before stream finalize
In Telegram streaming, the typing indicator persisted through the slow final rich-text/MarkdownV2 finalize edit, so the '...typing' bubble lingered for seconds after the last streamed token. Add a one-shot on_before_finalize hook to GatewayStreamConsumer, fired once when the stream transitions into its finalization path, and wire it on both Telegram streaming call sites to call pause_typing_for_chat() before the final edit. Cover hook ordering and once-only behavior in tests. Fixes #49712 |
||
|
Wolfram Ravenwolf
|
16fc717091 |
fix(mattermost): harden delivery hygiene
PROBLEM: Mattermost threads can become invalid or enormous, exposing two failure modes: internal scratch/reasoning/commentary displays could leak into persistent Mattermost threads via global display toggles, while rejected threaded user-visible replies could disappear unless every failed send fell back flat. A broad flat fallback would pollute channels with tool/status/progress noise. SOLUTION: Require explicit Mattermost platform opt-in for scratch displays, keep using the existing notify=True metadata marker for user-visible final text/media/file replies, and allow the Mattermost plugin adapter to flat-fallback only notify-worthy sends whose threaded POST failure looks like a broken root/thread. Keep tool/status/progress and other non-notify sends thread-strict. Add regression tests for display opt-in, notify-only broken-thread fallback, generic API failure suppression, and stream notify metadata. Verification: tests/gateway/test_mattermost.py tests/gateway/test_stream_consumer.py tests/gateway/test_stream_consumer_thread_routing.py tests/gateway/test_stream_consumer_fresh_final.py tests/gateway/test_stream_consumer_draft.py; tests/gateway/test_session_api.py tests/gateway/test_status_command.py tests/gateway/test_resume_command.py tests/hermes_cli/test_commands.py; py_compile touched gateway files; git diff --check. Session: Mattermost thread 6qg8e9dd1pd9pkhi74xyaa1mry, 2026-06-01. |
||
|
Teknium
|
a1f51feb72 | fix(telegram): avoid rich final duplicate previews (#46206) | ||
|
ITheEqualizer
|
7c0605bf22 | fix(telegram): preserve rich formatting on stream final |