cronjob(action='run', prompt=...) context was silently dropped when the
manual run forwarded to the gateway (#96010 follow-up): POST
/api/jobs/{id}/run took no body. The forward now sends {prompt} in the
request body; the api_server validates it (length cap + strict injection
scan, same as stored prompts) and trigger_job stamps it as a transient
manual_run_prompt alongside manual_run_at. run_one_job consumes the stamp
for that single fire and mark_job_run clears it, so it never persists
into the job definition or later scheduled fires.
The off-loop heartbeat write broke the producer->consumer invariant #86860
depends on: file freshness no longer equals loop schedulability, yet the
probe still classified a stale file as WEDGED — and WEDGED is destructive
authority (SIGTERM -> SIGKILL, bypassing the #86684 cron drain floor). The
measured motivating stall (112.6s max) exceeds the 90s stale budget, so a
healthy loop blocked inside the watchdog's own write could be killed, and
executor saturation produces the same false positive. The inverse edge
also existed: an off-loop write landing after the loop froze refreshes the
file mtime, manufacturing a false-fresh liveness proof.
The gateway loop now also arms a loop-scheduling witness: a UNIX socket
(state/gateway.loop-tick.<pid>.sock) answered by the loop itself via
await asyncio.start_unix_server — socket-buffer writes, no fsync, no disk
I/O, so it keeps working on the filesystem that stalls the heartbeat
write. The heartbeat payload records whether the witness is armed
(loop_tick_socket).
The classifier is now two-witness:
- socket answers -> ALIVE (file age irrelevant: a stalled write
or saturated executor can no longer produce a wedge verdict)
- file fresh, socket silent -> UNKNOWN (a late off-loop write can no
longer manufacture a liveness proof)
- file stale, socket silent, producer armed -> WEDGED (both witnesses
agree the loop stopped scheduling)
- legacy payload (no flag) -> unchanged single-witness contract: the
legacy producer wrote on-loop, so staleness is still proof
- any conflict/ambiguity -> UNKNOWN, never escalate
Tests are a producer->consumer composition: a real heartbeat loop with a
stalled write probes ALIVE while the file is past the stale budget, and
launchd_restart fed by the real probe drains instead of escalating; a
silent socket with a fresh file denies ALIVE; WEDGED requires the armed
socket to agree; a bind-failed producer disables stale escalation; legacy
payloads keep the old contract; a source-inspection test pins that the
witness is awaited on the loop. Mutation-checked: reverting either source
file fails the new tests. 45 tests pass across the watchdog suites; ruff
clean.
`loop_heartbeat_forever` wrote the heartbeat inline on the gateway loop. That
write ends in `atomic_json_write` -> `os.fsync`, and on a stalling filesystem the
fsync blocks whichever thread runs it — which was the loop the liveness watchdog
exists to monitor. So the watchdog timed out its probe
(DEFAULT_LOOP_WATCHDOG_TIMEOUT_S = 10s, MAX_STRIKES = 3, a ~90-120s budget) and
took the hard exit, for a loop that was unresponsive because it was blocked
inside the watchdog's own liveness write.
Measured on the install that prompted this: a WSL2 VHDX under io pressure
(/proc/pressure/io full avg300 = 7.60) stalled a trivial stat-and-fsync probe at
p99 31s and max 112s — longer than the entire watchdog budget. Two of that day's
three gateway restarts carry byte-identical stack dumps parked in the heartbeat
writer.
The write now goes to a thread. Awaited, not fire-and-forget, and that distinction
is the whole design: the docstring promises that a frozen loop lets the file age,
because that staleness is how an external supervisor notices. The loop still
initiates and awaits the write, so a wedged loop still stops refreshing the file —
while a blocked fsync no longer stops the loop from answering the probe. One write
in flight at a time, so a 112s stall cannot queue a thread per interval behind it.
Scope: the timeout and strike defaults are untouched. Raising them would only
delay the same kill, and picking a budget above this box's p99 is a deployment
decision, not a fix.
2 tests. The behavioural one patches a slow write and asserts the loop still
completes ~15 ticks while it blocks; it is bounded by a fixed sleep rather than an
Event handshake so a regression fails on the tick count instead of hanging. The
second pins that the write stays awaited, since fire-and-forget would pass the
first test while destroying the staleness signal.
Verified by mutation: reverting only gateway/shutdown_watchdog.py fails both new
tests and leaves the 5 existing ones green. 15 passed across
test_loop_liveness_watchdog / test_shutdown_watchdog /
test_systemd_watchdog_lifecycle / test_watchdog_review_76354.
Platforms added to main after the original branch was cut; keeps the
source invariant (every connectable adapter calls _wire_plugin_handlers)
true, and adds qqbot to the invariant test's gateway list.
ctx.register_platform_handler(platform, factory) — the generic surface for
plugins to wire native handlers into any platform adapter at connect()
time. Factories receive (native, adapter): the platform's client/app
object (PTB Application, discord.py Bot, slack_bolt AsyncApp, Teams App,
DingTalkStreamClient, aiohttp web.Application) or None for adapters with
no separate native object.
- BasePlatformAdapter._wire_plugin_handlers(native): shared, isolated
invocation helper — a raising plugin cannot block a platform connect.
- All 27 connectable adapters call it: telegram/slack/teams/line/
api_server/msgraph_webhook wire before their dispatch tables freeze;
the rest hook at connect success.
- register_telegram_handler and get_telegram_handler_factories retained
as thin back-compat aliases over the telegram bucket.
- Source-invariant test guarantees every adapter with connect() keeps
calling the hook.
Mirrors the Slack precedent (register_slack_action_handler): plugins queue
a factory at register() time; the Telegram adapter invokes each factory
with (application, adapter) at connect() time, before the core handlers
register, so pattern-scoped plugin handlers take precedence for their own
updates while everything else falls through unchanged. Factories are
isolated — a raising plugin cannot prevent Telegram from connecting.
Unblocks standalone plugins that need PTB update types the core adapter
doesn't route (Telegram Business API secretary bots, custom callback
prefixes, chat-member events) without touching core files.
Review found _create_entry_from_recovered_row builds a minimal entry:
replacing the live object would silently drop model_override, token/cost
counters, resume_pending/queued-work markers, and metadata. Keep the
original entry (routing is unchanged, so no sessions.json rewrite either)
and log at INFO — this is a success path, not a corrective action.
Regression test now asserts state preservation, reopen_session call, and
no save.
- launchd_restart resolves _launchd_domain() once (live launchctl probe,
up to 2x5s per call; two calls could also disagree)
- wedged-integration tests mock _wait_for_launchd_service_pid so the
observation poll doesn't burn 15s of real sleep per test (39s -> 16s)
- PEP8 blank lines in test_platform_base.py
Rebuilt branch from upstream/main f751a8c546 and re-applied the PR
changes. Resolved one conflict in tests/gateway/test_platform_base.py:
main had added TestDockerProfileSandboxMediaTranslation in the same
region — kept both main's new tests and the PR's
TestPlatformLockTakeoverGovernance regression suite.
Local: tests/gateway/test_platform_base.py +
tests/hermes_cli/test_gateway_service.py — 185 passed, 2 skipped;
ruff clean.
Refs: #79096
Fixes two related native-streaming bubble defects surfaced in production:
1. Duplicate bubble on long turns — when keep-alive already refreshed the
6-min reply window, the Layer-2 clock fallback still declined the finalize
frame and forced a proactive send(), duplicating the message. Skip the
clock fallback while keep-alive is active; intermediate-frame failures are
now fully fire-and-forget (only a failed FINAL frame falls back to send()).
2. Split / mini bubbles ('Cla' + 'ude ...') — two compounding root causes:
a) In native streaming a mid-turn commentary (e.g. a Hindsight recall
notice) called _reset_segment_state(), clearing the cumulative
_accumulated so the next delta + finalize frame carried only the few
chars accumulated after the reset. Native streaming now skips that
reset (commentary still posts as its own message via send()).
b) The adapter-side _BlockChunker.update() 'only grow' guard silently
dropped any cumulative snapshot shorter than its high-water mark, so
after a baseline reset the leading characters were stranded before
_emitted_len. Removed the _BlockChunker sentence-alignment + idle-flush
layer entirely; intermediate frames are pure identity-dedup, matching
the fire-and-forget model.
Also removes ~232 lines of now-dead code (_BlockChunker class, idle-flush
machinery, block-stream constants) and aligns the test suite with the
fire-and-forget frame model, including a regression test that locks the
native-commentary-no-reset behavior.
Tests: 177 passed, 3 skipped (wecom + stream_consumer suites).
Implement native reply streaming for the WeCom (企业微信) adapter over the
long-connection "msgtype: stream" transport, so a reply renders as a single
live-updating typing bubble instead of one final block. Aligns with the
official wecom-openclaw-plugin streaming behavior.
Includes the machinery intrinsic to native streaming on WeCom:
- Transport: seed frame (<think></think>) opens the typing bubble, intermediate
frames update it, a finalize frame closes it; native-streaming adapters are
let past the edit-only gate. Fire-and-forget intermediate frames (WeCom
long-connection mode has no documented edit-rate limit); an adapter-level
frame cap is retained. (Early builds gated frames behind a char throttle;
removed in favor of fire-and-forget + identity dedup.)
- Per-turn isolation: each turn owns a unique turn_id; concurrent messages are
isolated via (chat_id, turn_id)-keyed state. Dual-lane priority queue
(control vs normal) plus a per-chat token bucket to stay under WeCom's rate
limit (errcode 846607).
- Dedup-safe delivery + ack-race handling: deliver-once contract (a frame is
delivered the moment it is emitted; failures logged, not re-sent; delivery
marked once per turn), per-req_id reply queue with ack tracking, and the
timeout-inversion / orphan-queue race fixes. Robust fallback on 846608 /
846609 / errcode 6000 / passive-reply timeout via proactive send.
- Interaction boundaries: finalize + reset before approval/clarify prompts so
the prompt is the last thing on screen and never traps a lingering bubble;
eager re-seed after a clarify answer so the typing bubble reappears instantly.
- Stream-level keepalive: optional periodic finish=false frame + finalize-time
stream-age guard to refresh WeCom's ~6-minute reply-stream window on long
turns (mitigates 846604 / 846608). Off by default; tunable via config.yaml.
- Tool-progress folded into the same native-stream bubble instead of separate
messages; image+text double-callback merged into one turn.
Tests cover the streaming lifecycle, per-turn isolation, duplicate-send / ack
timing, approval + clarify boundaries, eager re-seed, and tool-progress.
Add WeCom to the per-platform streaming defaults (DEFAULT_CONFIG display
plumbing) so native streaming is enabled by default for the WeCom adapter,
alongside the existing per-platform flags. Non-secret config lives in
config.yaml (no HERMES_* env vars).
httpx timeout exceptions (ReadTimeout, WriteTimeout) stringify to "",
which defeats _is_timeout_error's first-line guard (if not error: return
False). The base-layer plain-text fallback then re-sends an already-
delivered message — the user receives it twice.
Replace error=str(exc) with error=str(exc) or type(exc).__name__ at every
httpx-based adapter boundary so the existing matcher ("readtimeout",
"writetimeout") still fires. ConnectTimeout intentionally stays
unmatched: if the connection never opened the message was not delivered,
so retry/fallback remains correct.
Applies to BlueBubbles (send + _create_chat_for_handle), WhatsApp Cloud
(text + interactive + media), QQ Bot (send chunk + keyboard + media), and
Yuanbao media handler — the same latent bug exists in every adapter that
stores error=str(exc) from an httpx call.
The send_media lane (08b95c3) had no committed regression test: cover
explicit-bool stamping on media frames, the descriptor-platform
fallback when _platform_by_chat is empty (post-restart proactive
sends), and the omitted-key absence case.
The send and send_media lanes resolved the platform only from
_platform_by_chat, which is empty until an inbound frame arrives (e.g.
after a gateway restart). A proactive send to a Slack chat then missed
the unfurl stamp. Mirror the streaming gate and delivery resolver:
fall back to the negotiated descriptor's platform.
Relay-plane parity: hermes config set / Railway persist YAML booleans
as strings, and _slack_unfurl_kwargs silently dropped them — so
'unfurl_links: "false"' was a no-op on native while working on relay.
Coerce recognized string booleans exactly as _slack_unfurl_hints does;
unrecognized values still drop so junk config keeps Slack's default
instead of accidentally suppressing previews.
Replaces test_send_ignores_non_boolean_unfurl_options (which froze the
dropped-string behavior) with coercion + junk-drop tests.
Live staging (Coatue Slack):
- hermes config set / Railway knobs persist "true" as a string; bots that
omit unfurl_links do NOT inherit the human default, so dropping the
string looked like suppression.
- chat.startStream cannot carry unfurl_*. Native SlackAdapter already
falls back to chat.postMessage; the relay now matches.
Relay-fronted Slack reads platforms.relay.extra.slack.unfurl_links/unfurl_media
and stamps explicit booleans onto the frame metadata; the connector forwards
them to chat.postMessage with no config of its own (mirrors reply_in_thread).
Covers send, send_for_platform (cron/scheduled), and send_media lanes.
* fix(relay): map wire media[] → event.media_types; accept message_type voice
A relayed voice note arrived as MessageType.AUDIO with media_types=[] —
the STT gate (_event_media_is_stt_input) excludes AUDIO unconditionally
and its per-attachment MIME rescue was unreachable, so STT never fired
and the agent fell back to the "user sent an audio file attachment"
context note (live-verified on staging 2026-08-26, Discord + Telegram).
Two wire-boundary fixes, both additive within contract_version 1:
- "voice" parses to MessageType.VOICE: the enum already had it — pinned
by test so a future refactor can't collapse the two.
- media[] is now mapped into event.media_types (positional alignment
with media_urls; mime-less entries keep their slot as ""). This is
what run.py's per-attachment classifiers key off, so EVERY relayed
attachment — image vs document, audio vs voice — now routes like its
native-adapter equivalent, not just voice notes.
Behaviour pinned: new-connector voice → STT-eligible; legacy
audio-typed events unchanged (no STT); music uploads never STT-eligible
(direct _event_media_is_stt_input assertions on real wire-parsed
events, not mocks).
Pairs with the gateway-gateway PR that puts "voice" on the wire.
* review: pin the STT gate by test; fail safe on media/media_urls mismatch
Addresses independent review of #95274.
1. The PR's acceptance criterion is STT ROUTING, but no committed test
called _event_media_is_stt_input — it was only asserted ad-hoc. Adds
TestSttGate: voice→eligible, voice-without-media_types→eligible
(the new-connector/old-gateway shape), legacy audio-typed voice
note→not eligible, music→not eligible. Mutation-verified: removing
the VOICE branch from the gate turns these RED.
2. media_urls and media[] are INDEPENDENT wire fields that consumers
index by the same i. Mapping MIMEs positionally without checking
agreement means a disagreeing producer misassociates a MIME with the
wrong URL and mis-routes that attachment — strictly worse than no
MIME, which degrades safely to message-level classification.
_media_types_from_wire() now maps only when the lengths agree, warns
and returns [] otherwise.
Note for the record: MessageType.VOICE predates this PR and the gate's
VOICE branch ignores media_types, so a NEW connector against an OLD
gateway ALREADY fires STT. That is desirable, but it is not "unchanged"
— the PR body's rollout matrix said otherwise and is corrected.
* fix(relay): send a User-Agent on relay media requests (Discord CDN 403)
Discord's CDN rejects urllib's default "Python-urllib/x.y" User-Agent
with HTTP 403, and RelayMediaClient never set one. Every Discord CDN
pass-through download therefore failed; _localize_inbound_media then
kept the raw URL (its "a public URL still has value" branch), and the
consumer tried to open a URL as a FILE PATH:
WARNING gateway.relay.media: relay media download failed for
https://cdn.discordapp.com/...voice-message.ogg: HTTP Error 403
INFO gateway.run: Voice transcription failed for https://cdn.discord...
: Audio file not found: https://cdn.discordapp.com/...
This killed ALL Discord relay media inbound — voice notes, images and
documents alike — not just the voice lane. Telegram/WhatsApp were
unaffected because their media is connector-re-hosted (/relay/media/{id},
fetched from our own host) and localizes to real /tmp paths.
Reproduced from a clean shell against a live CDN URL:
curl (own UA) -> 200
urllib, no UA -> 403 Forbidden
urllib + descriptive UA -> 200, 14583 bytes, OggS magic
Fix: a module-level _MEDIA_USER_AGENT sent on both download() and
upload(). upload() only ever targets our own connector so it was not
broken, but a single client should identify itself consistently.
Validated on staging: hot-patched hermes-agent-stg-test-6698, restarted
the gateway service, and Ben's Discord voice note transcribed
successfully — zero new 403s and zero new transcription failures after
the patch (last 403 predates it).
Test is mutation-verified: removing the UA from download() turns it RED
while the other five media tests stay green.
* fix(relay): keep url↔mime pairing through media localization
Addresses a blocking review finding on my own change: mapping media[]
into media_types created a POSITIONAL contract that the rest of the
inbound path then broke.
1. _localize_inbound_media (adapter.py) filtered media_urls without
filtering media_types. Dropping a dead connector re-host is a NORMAL
best-effort path, so every surviving attachment inherited its
neighbour's mime. Reproduced through the real functions:
before urls [.../relay/media/dead, .../kept.png]
types [application/pdf, image/png]
after urls [.../kept.png]
types [application/pdf, image/png] <-- PNG reads as PDF
_event_media_is_image(ev, 0) -> False
The loop now carries (url, mime) as PAIRS, so a dropped URL drops its
mime with it.
2. _media_types_from_wire compared LENGTHS only, which is not alignment:
equal-length-but-reordered wire fields were accepted and paired
wrongly, and an absent media_urls skipped the check entirely while
still emitting types. Resolution is now BY URL (url -> mime lookup
over media_urls); an unmatched URL degrades to "" and falls back to
message-level classification.
Tests: 4 new cases driving the real chain (wire parse -> localization ->
run.py classifier), incl. the dropped-first-attachment case the existing
localization test could not catch (it builds events without
media_types). The obsolete length-mismatch test now asserts the stronger
by-url guarantee. Both fixes mutation-verified: reinstating the URL-only
filter fails 1 test, reverting to positional resolution fails 3.
Relay suite 258 passed; media/voice/stt selection 685 passed; ruff clean;
cross-repo integration payload re-verified.
* fix(relay): media_types is always one slot per media_url
Self-review after two review rounds flagged this bug class in adjacent
seams: I checked the function I edited, not every consumer of the
parallel arrays I created. Grepping ALL writers found a third instance.
merge_pending_message_event (gateway/platforms/base.py:2725-2735)
EXTENDS media_urls and media_types together when a second media message
merges into a pending one. My mapping could emit a POPULATED media_urls
with an EMPTY media_types (an older connector sends media_urls but no
media[]), so extend() concatenated lists of different lengths:
A urls [old1.png, old2.png] types []
B urls [new.pdf] types [application/pdf]
merged urls [old1.png, old2.png, new.pdf]
types [application/pdf]
-> old1.png reads as application/pdf; the real PDF gets ''
Fix: media_types is now ALWAYS len(media_urls), padded with '' — the
url-keyed lookup runs even when media[] is absent, and the localizer
rewrites the list unconditionally (no short-circuit that
could leave a stale/short list behind).
Tests: 4 new cases — padding with no media[], the merge shift above
driven through the real merge_pending_message_event, localization
preserving the invariant while dropping an entry, and normalization of
a short/empty media_types arriving from a non-wire source. All
mutation-verified: removing the padding fails 4; restoring the
guard fails 1.
Relay 262 passed; media/voice/stt selection 689 passed; ruff clean;
cross-repo integration payload re-verified.
On Windows installs where the gateway runs as an SCM service (WinSW,
NSSM, sc.exe create), the existing pause machinery kills the gateway
process directly — and the service wrapper's failure ladder resurrects
it within seconds, re-taking the venv file locks mid-update. The update
then dies partway through dependency sync with access-denied errors.
This extends _pause_windows_gateways_for_update() to detect when a
gateway's process tree is owned by a running SCM service, and to stop
the SERVICE through sc.exe instead of killing the child:
- gateway/status.py: expose service-ownership discovery for gateway
runtimes (find_windows_gateway_services maps validated gateway PIDs
through process ancestry to running SCM service PIDs, with
create-time identity checks against PID reuse).
- hermes_cli/update_cmd.py: stop verified services via sc.exe before
venv mutation and restart them afterward. Stops wait for a stable
SCM 'stopped' state AND for the original descendant processes to
exit (service 'Stopped' is not proof the child released its
handles). Failure to prove ownership, stop a service, or restart it
fails closed; rollback restores attempted services, and rollback
failures are surfaced rather than swallowed.
- Fail-closed throughout: unreadable identities, ambiguous ancestry,
or a service that will not reach a stable state abort the update
before any file mutation.
Complements #37039 (gateway-only concurrent instances no longer abort):
that fix lets the update proceed past the gate; this one makes the
pause actually stick when the gateway is service-supervised.
Note: tests/gateway/test_status.py::TestReadProcessCmdlinePsFallback::
test_ps_fallback_when_proc_unavailable fails on Windows on current main
before this change as well (POSIX ps fallback asserted on a platform
without it); all other touched suites pass (155 passed, 5 skipped).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The wrapper now getattr-defaults _processed_message_ts (object.__new__
adapters in sibling suites lack it), and the reaction-guard source pin
reads _handle_slack_message_impl where the production expression lives.
Follow-up for the #95417 salvage, addressing the review finding: the entry
claim closes the unfurl race but a handler that raises mid-enrichment would
hold the claim forever — neither a Slack retry nor a user edit could ever
re-drive the message. _handle_slack_message is now a thin guard around the
impl that releases only claims taken by the failed invocation itself, with a
warning log so swallowed turns are traceable. Pre-existing claims from a
successful turn are never released. Two failure-path tests pin both sides.
Slack emits `message_changed` for a link unfurl carrying a DIFFERENT event ts
than the original message. That ts legitimately misses the `_dedup` check, so
`_processed_message_ts` is the only guard against it becoming a second user
turn -- but it was only populated at the END of `_handle_slack_message`, after
thread context, permalink resolution and file downloads had all awaited.
An unfurl landing inside that window found the guard empty and was promoted to
a duplicate turn: a spurious "Interrupting current task" banner plus the same
answer posted twice.
Production capture (adminbot, 2026-08-22 02:23:30-31Z, channel C0BF1EYUA9H):
02:23:30.718 message ts=1787365409.908499 dedup_hit=False
02:23:30.737 app_mention ts=1787365409.908499 dedup_hit=True
02:23:31.675 message ts=1787365411.012100 dedup_hit=False <- leaked
subtype=message_changed
The original copy was still resolving two Slack permalinks when the unfurl
arrived 957ms later.
Claim the message ts once every filter has passed and the event is certain to
be delivered, before the slow enrichment awaits. Claiming any earlier (right
after the dedup check) also claims messages the handler then discards, which
breaks summoning the bot by editing "@bot" into a previously ignored message
(tests/gateway/test_slack.py::TestMessageRouting::
test_message_edit_with_new_mention_processed).
Eviction logic is extracted to `_remember_processed_message_ts` so both call
sites share one bounded implementation.
Deliver changes_requested review outcomes through kanban subscriptions and
wake the origin for notify+wake / wake modes. Review-specific only: no task
mutation. Reasons are redacted, path-scrubbed and truncated before delivery.
Salvaged from #88694; conflicts with the #87733 wake-kinds expansion resolved
keep-both.
slack_bolt builds a fresh AsyncWebClient for every inbound request and
copies proxy=app.client.proxy into its constructor, where slack_sdk reads
a None/blank proxy *argument* as "unspecified" and reloads HTTP(S)_PROXY
from the environment. aiohttp then treats that env value as an explicit
proxy and skips its own NO_PROXY check, so the adapter's resolved decision
to go direct - a NO_PROXY bypass, or a proxy scheme aiohttp cannot use -
holds on every client except the one authorization spends on auth.test.
The failure looks like a healthy bot: Socket Mode connects, outbound sends
keep working, and every inbound event is rejected with "Failed to authorize
with the given token" - forever, since a failed auth_test_result is not
cached and never retried differently.
Re-apply the resolved proxy through AsyncApp(before_authorize=...), which
bolt inserts before the authorization middleware: the request-scoped client
already exists there and has not been used yet. Assigning the attribute
post-construction is the only way to express "no proxy" to slack_sdk.
Co-authored-by: Junie <junie@jetbrains.com>
`completed` already puts the worker's summary inside the synthetic wake
turn, so the woken creator sees what was done. `review_requested` did
not: the summary rode the passive ping only, and the wake turn said just
"handed off for review", forcing the woken reviewer to re-read the board
(and losing the PR link the worker had already written).
Reuse the same first-line handoff the `completed` branch builds, so the
existing `gateway.kanban.wake.handoff` string renders it — no new locale
keys, no change to the passive message.
`review_requested` and `block_loop_detected` are terminal event kinds that
hand a decision back to the origin subscriber, but neither was listed in the
gateway notifier's `_WAKE_KINDS`. A `notify+wake` subscription therefore got
the passive ping only and the origin agent never took a turn — so an agent
that delegated implementation work slept through the "ready for review"
handoff and through a task being routed to triage, while the equivalent
`blocked` event woke it.
Add both kinds to the wake set, add their status strings to the synthetic
wake message in every locale, and document which events wake.
Slack sends an authored message twice: flat in `event.text` and structurally
in `event.blocks`. The blocks are rendered so quoted and forwarded content is
not lost, and whatever the render carries beyond the flat text is appended to
the message. That comparison had several ways to fail on the *same* sentence,
each of which showed the author their own words a second time:
1. HTML entities — the flat copy escapes `&`/`<`/`>` while `blocks[].link.url`
stays raw, so any link with query parameters (every "Copy link" on a
thread) mismatched.
2. Permalink unfurls — the live inbound path skips `is_msg_unfurl`
attachments, thread/parent hydration did not, so the linked message's body
was appended again.
3. The Block Kit dump — it serialized the authored `rich_text` alongside the
UI blocks it exists for, and its allowlist drops `url`, so the sentence
reappeared with every link removed.
4. Unknown inline elements — the renderer knew eight types and silently
dropped the rest. A pasted message permalink arrives as `message_mention`,
so the link vanished from the render and the sides stopped comparing equal.
5. `message_mention` without a url — `url` is optional on that element while
`channel_id` and `message_ts` are not, so the element rendered as nothing
and the sentence came back with a blank in the link's place.
6. `date` elements — `fallback` and `url` are both optional, and the flat
`<!date^…>` form was never read down to what the rich text renders.
7. Labelled mentions — Slack may attach a label (`<@U…|name>`,
`<#C…|general>`, `<!subteam^S…|@marketing>`, `<!here|@here>`) in the flat
text while the blocks carry the bare id. The bot's own mention is one of
these, and stripping only its bare form left it in the flat copy.
8. Autolink schemes — only `https` and `mailto` were matched, so a `tel:` link
kept its angle brackets and mismatched too.
Unknown inline types are now read by their `url`/`text`/`fallback` so a type
Slack adds later still renders, and `team`, `color` and a fallback-less `date`
render into the flat form Slack sends. Every field is read as a string or
not at all: Block Kit carries text as an object in many places, and a
non-string one reaches the renderer's `str.join` and raises there, which
costs the whole message. `channel_id` and `message_ts` are the
permalink's own components, so a url-less `message_mention` renders the
permalink's tail; the workspace host and the thread query cannot be rebuilt
from the element, so a permalink on either side is reduced to that same tail.
Canonicalization is used for matching only -- the authored text still reaches
the agent verbatim, so a mistake here can cost an unrendered element, never an
altered or missing message.
An element carrying neither a url nor a label still renders as nothing, and a
message containing one is still appended twice. Suppressing such a render was
tried and is worse: an app message whose body lives only in the blocks
disappears, and a forwarded quote is dropped. Genuinely additional content --
quotes, lists, code blocks, attachments, interactive bot blocks -- is
unaffected throughout.
Tests cover both merge sites (live inbound and thread hydration) and the
negative cases.
Windows updates forced a choice between 'gateway survives' and 'update
proceeds': the pause machinery's only tools were the planned-stop marker
poll and the force-kill ladder, so a mid-turn gateway was tree-killed and
its active turn lost. Step 2 of the socket migration adds the
pause-for-update verb: the updater ASKS the gateway to drain in-flight
turns and exit cleanly — releasing every venv file handle on the way out
— through the same request_restart(via_service=True) drain path SIGUSR1
and service restarts already use.
- gateway/run.py: pause-for-update verb handler registered on the
existing control server; marshals onto the loop thread, ACKs with
{pausing, already_stopping, pid, drain_timeout}.
- gateway/control_socket.py: pause_gateway_for_update() client — None on
no-answer (older gateway / no socket), so every caller keeps the
legacy path when the verb is missing.
- update_cmd.py (_pause_windows_gateways_for_update): socket-first ask
per mapped profile gateway before the drain wait; positive ACKs extend
the wait to the gateway's own declared drain budget (+ teardown grace)
so a mid-turn gateway isn't force-killed at the end of a too-short
local default. Marker write + force-kill ladder retained verbatim as
the fallback.
Live E2E: real gateway process (isolated HERMES_HOME), real socket:
identify -> pause ACK {pausing: true} -> gateway drained and exited on
its own (rc=75, zero signals) -> dead-gateway re-ask returns None.
A step-1 gateway without the verb answers ok:false -> client None ->
legacy path (pinned by test).
The three existing assertions are absence checks, so the test also
passed when the loop got no iteration inside the sleep window (with
interval=5.0 it passes without the gate ever executing). Checking
_scale_to_zero_no_suspend_logged proves the branch was taken.
Review follow-up for the salvaged #94848: the ProfileRouteRejected marker
looked write-only inside the primary handler. Document that
_handle_message's ingress gate reads the same marker to drop the message
fail-closed, and add a regression test showing the rejected route is
stamped once, dispatch falls back to the default home, and routing is not
re-run on redelivery.
Review follow-up to the AI code-review pass on PR #92074: the bridge task's
handoff into _schedule_secondary_profile_reconnect was unguarded at both call
sites inside the parked coroutine. The scheduler touches live registries
(_profile_failed_platforms slot creation, background-task registration), so an
unexpected raise there would kill the parked task as an unretrieved-task
exception — logged only at GC time via "Task exception was never retrieved",
where no operator ever looks. A fix whose entire purpose is to stop a platform
dying silently should not contain its own silent-death path; both handoff sites
now wrap the scheduler call with logger.exception so the failure lands in
gateway.log with profile and platform context.
The early-exit branch (gateway already _running when the bridge starts) had the
identical exposure and is guarded the same way — same bug class, fixed together.
Regression test drives a handoff raise end-to-end through the real bridge task:
the await completes cleanly, the error is captured in gateway.run's logger, and
no adapter or failed-platform slot leaks behind the failed handoff.
When gateway.multiplex_profiles is active, a secondary profile whose platform
adapter fails its initial connect at startup was silently given up on: the
failure branches in _start_one_profile_adapters() logged and disconnected, but
never scheduled recovery. One unlucky connect window during a Telegram API
outage left the profile permanently silent until manual restart (~80 min in
the observed incident), while the mid-run fatal path already recovers via
_handle_profile_adapter_fatal_error() -> _schedule_secondary_profile_reconnect().
The same gap hit both failure shapes: a clean False return from
_connect_initial_adapter_with_timeout() and an exception escaping it.
Fix: call _schedule_secondary_profile_startup_reconnect() from both startup
failure branches after _safe_adapter_disconnect(). Because secondary adapters
are started mid-start(), before self._running flips True, the regular
scheduler's not-self._running guard would silently drop the request — so the
new bridge parks a background task until startup completes (or shutdown
begins) and then hands off to _schedule_secondary_profile_reconnect()
verbatim: backoff, fresh-adapter rebuild under the profile runtime scope,
slot dedupe, and shutdown cancellation all come from the existing path.
Non-retryable failures are dropped at scheduling time exactly as the regular
scheduler drops them, keeping duplicate-credential/auth-failed startups dead
instead of looping.
Fixes#92064
Commit a270c4ade's session-key fallback in _resolve_container_task_id was
added to stop cross-profile SSH environment reuse, but it wasn't backend-
gated: persistent Docker silently fragmented into one container per gateway
session, breaking the product contract (one long-lived container per profile,
shared by CLI and every session of that profile). #93950's vanishing MEDIA
attachments were downstream damage.
- persistent Docker (container_persistent: true) now keys to the profile:
literal 'default' for the default profile (same container as CLI),
'profile:<name>' for named profiles
- SSH and non-persistent Docker keep session scoping (the original leak fix
and the #82731 isolation contract are untouched)
- gateway MEDIA translation follows the profile layout and keeps the legacy
bug-window per-session sandboxes as fallback candidates, trying each until
the file resolves — old sessions self-heal, no migration
- /root/.hermes credential-surface refusal preserved across all layouts
Phase 2c on the full final diff flagged the re-export shim as
contradicting the adjacent house pattern (agent.replay_cleanup import,
which documents retiring private aliases once tests migrate). Migrate
all six tests to the canonical gateway.media_repair seam, import the
canonical name in run.py, and drop the dead 'and result' guard at the
background-task call site.
- Make repair_explicit_computer_use_media_paths fail-open internally
(cosmetic repair must never abort delivery); drop the cron-only
try/except so all three call sites are identical one-liners.
- Drop cron's redundant 'MEDIA:' pre-check (helper early-returns).
- Document the intentional lazy BasePlatformAdapter import (verified:
no cycle either way; keeps module import cheap for cron processes).
- Point the two new regression tests at the canonical
gateway.media_repair seam; pre-existing tests keep pinning the
gateway.run re-export shim.
- Docstring: matching is case-insensitive, say so.
Teknium