Commit Graph

2624 Commits

Author SHA1 Message Date
Teknium f92accd05d fix(tui): lead slash_worker PATH with Hermes' managed bin dir
Follow-up on the #83854 salvage: prepend $HERMES_HOME/bin ahead of the
venv and user-local bin dirs, matching the managed-first Browser Use
CLI resolution policy — the worker resolves the same canonical binary
the agent process does.
2026-08-14 16:22:47 -07:00
webtecnica 73b4e41ab2 fix(tui): prepend Hermes venv and user-local bin to slash_worker PATH (#83845) 2026-08-14 16:22:47 -07:00
Seashore Shi 5bc13cf574 fix(gateway): tolerate list-shaped gateway.platforms from setup (#83185)
`hermes gateway setup` writes `gateway.platforms` as a LIST of
enabled platform names (e.g. `- telegram`), not a dict.  Treat any
non-dict shape as "no per-platform overrides" instead of crashing
on `.get()` for every incoming turn (#83185).

Co-authored-by: SeashoreShi <seashore.shi@gmail.com>
2026-08-15 03:05:23 +05:30
teknium1 f79440e0f4 feat: /loop — recurring in-session wakeups (Claude Code parity)
Ports Claude Code's /loop (and its /proactive alias) across every Hermes
surface. /loop [interval] <prompt> re-runs a prompt or slash command on a
recurring cadence inside the live session; omitting the interval enables
self-paced mode (starts at the floor, backs off exponentially while the
agent's replies stop changing, snaps back on change — local digest
comparison, zero extra LLM cost).

Stop conditions: agent-emitted LOOP_COMPLETE marker, --times N,
--until <condition> (judged by the existing goal_judge aux task,
fail-open), /loop stop, and a loops.max_ticks backstop budget.

Core: hermes_cli/loops.py (LoopState + LoopManager + shared
dispatch_loop_command), persisted per session in SessionDB state_meta
(loop:<sid>) so /resume picks it up; migrates across compression
boundaries like /goal. New SessionDB.list_meta_prefix() powers the
gateway's cross-session scan.

Surfaces:
- CLI: /loop handler + idle-fire and post-turn-complete hooks in
  process_loop (mirrors the /goal hook shape; Ctrl+C pauses the loop)
- Gateway: /loop handler with route capture, mid-run control-verb guard,
  post-turn tick completion, and a supervised loop_wakeup_watcher that
  injects due wakeups into idle chats via the synthetic-message path
- TUI/dashboard/desktop: command.dispatch handler + per-session
  notification-poller wakeup driver + post-turn completion in the turn
  dispatcher; /loop added to the desktop slash palette
- /goal mixing: an active non-parked goal owns the idle boundary — loop
  ticks defer until it finishes, pauses, or parks; real user input always
  wins over both

Config: loops.{min_interval_seconds,max_ticks,self_paced_floor_seconds,
self_paced_ceiling_seconds}. Docs page + sidebar entry. 77 new tests.
Slack's 50-slash cap: /version moves to /hermes version to free the
native slot for /loop.
2026-08-14 13:40:19 -07:00
Teknium 3885c1096a fix(gateway): stop internal bookkeeping writes from advancing the session activity clock
set_session_metadata() and advance_compression_session()'s repoint both
stamped entry.updated_at = now. updated_at is the user-activity clock that
drives idle/daily reset policy and the restart-resume freshness gate
(suspend_recently_active, #85709), so a background metadata write (e.g.
Slack thread watermark) or a background compression repoint on a long-idle
session could make it look freshly active and get it falsely resume_pending
after a gateway restart.

These are the last internal stamp sites after 784f733cf (recover) and
5462f689b (touch_activity gating): drop the stamps, keep the durable save.

Follow-up to #85895 (closed) — credit @GodsBoy for the report-side push and
@chelsealong for the analysis on #85709.
2026-08-14 13:08:39 -07:00
Teknium a90d5369f7 feat(gateway): dump wedged worker stacks when the turn reaper fires
When the inactivity reaper interrupts a timed-out turn, the interrupt
frees the blocked frame — destroying the only evidence of where the
turn was wedged. The Aug 2026 zombie-turn incident (WhatsApp session,
Relay-corrupted scope stack) wedged every turn for exactly the 1800s
timeout somewhere between 'Turn ended' and run_sync returning, and the
wedge point was unprovable post-mortem.

The reaper now logs the stack of every thread with turn-machinery
frames BEFORE interrupting, so the next occurrence names the exact
blocked line. Best-effort, bounded (8 threads, 25 frames), pure
in-process, never raises into the reaper.
2026-08-14 11:22:14 -07:00
Teknium afe09c7942 fix(gateway): exclude wedged turns from the restart after-turn wait
A turn idle past agent.gateway_timeout (the same threshold the turn
reaper uses) no longer defers an in-band restart. Restart is usually
the remedy for a wedged turn; waiting restart_after_turn_timeout on
one inverts the graceful path's purpose — a wedged WhatsApp turn
pinned 'hermes update' in draining until SIGTERM was sent manually
(Aug 2026). stop()'s bounded drain interrupts wedged turns instead.

gateway_timeout=0 (unbounded turns) disables wedge detection; cron
and API-server work has no per-turn activity clock and is never
counted as wedged; unreadable activity summaries fail open.
2026-08-14 11:22:05 -07:00
Soheil Fakour 0c6761c511 fix(gateway): restart_after_turn_timeout default 6h -> 30min (#79133)
The 21600s (6h) default shipped in #77184 makes an interactive
'hermes gateway restart' block for up to six hours when a turn wedges
(hung tool call, wedged event loop, stuck provider stream) — the exact
scenario the cap exists for. The intent (don't force-kill an agent
mid-turn) is sound, but the default must be a safety valve for hung
agents, not a target latency.

Lower to 1800s (30 min): still protects the overwhelming majority of
long autonomous turns (tool calls have their own timeouts well below
that), keeps worst-case interactive restart latency human-tolerable, and
users running very long unattended turns can raise it in config.yaml.

RED: new contract test fails on old 21600 default. GREEN: 4/4.
2026-08-14 11:22:05 -07:00
dsad 858a6008be fix(gateway): keep process notification routing off-loop 2026-08-14 01:21:52 -07:00
Teknium 7619564fbd fix(gateway): gate background-process completions on spawning-session boundary
Plain type=completion events built in _run_process_watcher carried only
session_key (chat/thread routing) with no spawning-session stamp, so after
/new (or a session switch) a completion notification from the OLD session
was injected into the chat's NEW session. Main already solved this exact
class for async delegations via the _classify_completion_target pre-flight
(_USER_BOUNDARY_END_REASONS drop on user-closed sessions, deliver on
idle-ends, follow the compression-tip chain), but the gate only ran for
type=async_delegation events.

Kernel salvage of #16455:

- Stamp the spawning conversation's session-db id (HERMES_SESSION_ID via
  session-scoped env) on the ProcessSession and the pending_watchers entry
  at spawn time in tools/terminal_tool.py; persist it through the process
  registry checkpoint/restore so recovered watchers keep the stamp.
- Thread the stamp into the completion_evt built by _run_process_watcher
  (watcher entry first, ProcessSession fallback for recovered watchers).
- In _deliver_completion_notification, run the SAME pre-flight classifier
  for stamped type=completion events: terminal -> drop with a log (output
  stays available via process(action='log')), retry -> False so the
  watcher re-polls, deliver -> proceed. The policy has exactly one owner
  (_classify_completion_target); nothing is forked. Unstamped legacy
  events keep today's deliver-always behavior, and the async-delegation
  path is untouched.

Based on the session-boundary approach from #16455 by @Tosko4 (original PR
was over-scoped across adapters/slash-commands/cron; this lands the kernel
only).

Tests: completion from a /new-closed session is dropped; completion after
an idle-end still delivers; unstamped legacy event delivers; retry verdict
returns retryable False without adapter injection; async_delegation gate
unchanged; stamp survives checkpoint recovery.
2026-08-14 01:09:42 -07:00
Teknium b9e7bead13 fix(gateway): coalesce same-tick async-delegation completions into one turn (#70300)
The async-delegation watcher drained the completion queue as a batch but
then delivered each event as its own synthetic turn, flooding the session
when a fan-out of background subagents finished together. Builds on the
per-process completion batching salvaged from PR #71898 (thanks
@yuzilongleif-collab) which coalesces concurrent _run_process_watcher
completions behind a short per-route fan-in window.

This commit adds the async-delegation half: group the drained batch by
full routing key (session_key + parent_session_id + platform/chat/thread/
user) and inject ONE consolidated turn per group. Durable-ack handling
stays honest: sibling rows are claimed up front via claim_event_delivery;
rows another consumer owns are excluded from the consolidated text (no
double-delivery); sibling claims are acknowledged only after adapter
acceptance and released (still pending) on failure. Events for different
sessions never coalesce, and a single-event group rides the existing
per-event path unchanged (latency and text identical).

Tests: 3 same-tick events -> exactly one adapter.handle_message carrying
all 3 results with all 3 durable rows delivered; 2 sessions -> 2 turns;
single-event path unchanged; failed batch releases claims and retries;
foreign-claimed sibling excluded and left pending.
2026-08-14 01:08:25 -07:00
yuzilongleif-collab a96cd10349 fix(gateway): force-redact coalesced completion output 2026-08-14 01:08:25 -07:00
yuzilongleif-collab 7536655b8f fix(gateway): own completion batch task lifecycle 2026-08-14 01:08:25 -07:00
yuzilongleif-collab cf09a30a9a fix(gateway): coalesce concurrent process completions 2026-08-14 01:08:25 -07:00
Teknium 8dc9401d7e fix(gateway): persist internal synthetic turns typed as internal_notification (#82888)
Async-delegation batch completions and background watch notifications
re-enter the gateway as synthetic MessageEvent(internal=True) turns via
_inject_watch_notification, but were persisted as bare role='user' rows —
indistinguishable from real user input in transcripts and the desktop UI.

Thread the event's internal flag through to persistence: when
event.internal is set, the turn's persisted user row is stamped
display_kind='internal_notification' (the existing DB-only presentation
sidecar used by auto_continue / model_switch rows). Wired through
_run_agent → _run_agent_inner → TurnContext → run_conversation's
persist_user_display_kind, and onto the three gateway-side fallback user
rows (transient failure, no-new-messages, pre-run crash), whose
append_to_transcript writer now forwards display_kind/display_metadata
to SessionDB.append_message.

Invariants preserved: role stays 'user' (alternation untouched), no new
injections, no past-context mutation, and display_kind is already popped
from every provider-bound copy in conversation_loop, so replayed sessions
never leak the marker to the API.

Regression tests: internal turn marked, real user turn unmarked, fallback
rows marked/unmarked per event, and a DB round-trip proving replay keeps
role/content intact while the provider copy drops the marker.
2026-08-14 01:08:18 -07:00
spfcraze d2d7766750 fix(tools,gateway): format watch_overflow events instead of dropping them
format_process_notification had no case for watch_overflow_tripped /
watch_overflow_released, so a watch-pattern notification flood surfaced
as '[IMPORTANT: Background process  exited (exit code ?)]' — a phantom
exit notification for a process that never existed — while the actual
'watch flood, N notifications suppressed' summary in the event's
message field was silently dropped. The gateway delivery path was
worse: _drain_gateway_watch_events retained only watch_match and
watch_disabled, discarding overflow events entirely before formatting.
Route both event types through the message field in the shared
formatter and the gateway formatter, and retain them in the gateway
drain.
2026-08-14 01:08:13 -07:00
Lidang-Jiang 9b554758bd fix(gateway): honor notification-off watch reinjection
Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>
2026-08-14 01:08:13 -07:00
Teknium 1c971769ec feat(gateway): concise background process notifications by default
Background process completions on messaging platforms now default to a
one-line status message (✅/❌ + command + duration; failures append a
short output tail) instead of dumping the raw output buffer into the
chat. New display.background_process_notifications mode 'concise' is
the default; 'all' keeps the old raw-dump behavior for anyone who wants
it. Config migration v35 moves users still on the old implicit default
'all' to 'concise' on their next update; explicit result/error/off
choices are preserved.
2026-08-14 00:26:11 -07:00
Shannon Sands 6977d21fa7 feat(gateway): disk-usage telemetry + dashboard disk-pressure banner (NS-656)
Extends the NS-656 memory-pressure surface to cover disk exhaustion
(OOF-2 / OOF-107 lineage: agents fill their data volume — SQLite writes
fail, sessions stop persisting — while every dashboard looks healthy).

- gateway/disk_status.py (new): collect_disk_status() samples
  shutil.disk_usage(HERMES_HOME) and classifies pressure
  (critical: <256 MB free or >=95% used; elevated: <512 MB free).
  Never raises — degrades to pressure="unknown" with null telemetry,
  same contract as collect_memory_status().
- /api/status: sibling `disk` block next to `memory`, advisory only —
  not folded into component/overall health.
- web: DiskPressureStatus type; MemoryPressureBanner generalized to a
  resource banner with worst-first triggers (disk critical > memory
  critical > OOM restart > disk elevated > memory elevated) and
  cascading dismissals — hiding the top trigger surfaces the next one
  instead of silencing everything. All dismissals stay boot_id-scoped.
- i18n: diskCriticalBanner / diskElevatedBanner (en, optional fields
  with English fallback per existing pattern).

Tests: gateway/test_disk_status.py (14), web_server disk-block
presence/degradation, banner disk trigger/priority/dismissal-cascade
suite (21 total).
2026-08-13 20:30:12 -07:00
Shannon Sands ba5dc00bb2 fix(memory-status): review follow-ups — incident-keyed dismissal, honest copy, single mobile offset (NS-656)
Addresses the human review findings on the memory-pressure feature:

* [P2] Dismissal hid later incidents of the same kind. The gateway now
  publishes `boot_id` (the lifecycle sentinel's started_at — changes on
  every gateway life) in the /api/status memory block, and the dashboard
  keys OOM-restart dismissal on it: acknowledging one restart no longer
  mutes the NEXT one (the OOM-loop case this banner exists for). Live
  pressure dismissals now also reset once pressure is demonstrably back
  to "ok" — "unknown" (stale heartbeat) is absence of evidence and
  clears nothing. Dismissal storage moved to a JSON list; old bare-string
  entries fail JSON.parse and degrade to a clean reset.

* [P2] suspected_oom is a heuristic (unclean exit + low-memory final
  heartbeat), not proof the OOM killer acted — banner copy now says
  "restarted unexpectedly, most likely because it ran out of memory"
  instead of stating OOM as fact.

* [P3] Mobile header clearance was applied per-banner (mt-14 on both
  MemoryPressureBanner and ProfileScopeBanner) AND on the content
  (pt-14), double/triple-stacking 56px gaps when banners were visible.
  Replaced with a single h-14 spacer above the banner stack.
2026-08-13 20:30:12 -07:00
Shannon Sands e11d1ddc7f feat(status): surface memory pressure and suspected-OOM restarts to users (NS-656)
Hosted agents can be OOM-killed hourly while the dashboard and the NAS
agent card both look perfectly healthy — every memory signal the gateway
already produces (heartbeat mem samples, lifecycle-ledger unclean-exit
verdicts, cache-pressure evictions) dies in server-side log files. The
BlueAtlas incident (NS-608) ran for three days like this.

This is the read-side fix:

* New gateway/memory_status.py distills the existing 30s loop heartbeat
  (gateway RSS + system MemAvailable/MemTotal + swap) and the lifecycle
  sentinel into a compact `memory` block: pressure ok/elevated/critical/
  unknown, coarse MB numbers, and last-boot unclean/suspected-OOM flags.
  Pure file reads, no new sampling, no gateway IPC. Stale (>150s) or
  future-dated heartbeats degrade pressure to "unknown" so a dead
  gateway's final gasp can't render a live "critical" banner forever.
  Critical thresholds mirror the ledger's OOM-suspicion heuristics: if a
  level would make a later unclean death "suspected OOM", warn at that
  level while the process is still alive.

* lifecycle_ledger.record_startup now carries prior_unclean_exit /
  prior_suspected_oom onto the reclaimed sentinel — previously the
  verdict survived only in append-only diag prose. Flags age out on the
  next sentinel rewrite (scoped to the life after the crash).

* /api/status serves the block (profile-aware, executor-offloaded,
  fail-safe to pressure=unknown). Deliberately NOT folded into
  components/overall: memory pressure is advisory, and flipping overall
  to "degraded" on it would page NAS's availability sweep for a
  condition the eviction valve is already handling. Public-safety:
  coarse numbers/enums/booleans only — same disclosure class as the
  existing nous_session_valid field, added for the same NAS-sweep
  audience.

* Dashboard: new MemoryPressureBanner (app-shell, next to
  ProfileScopeBanner) with worst-first trigger precedence
  (critical > suspected-OOM restart > elevated), per-trigger
  session-scoped dismissal, and escalation re-opening past a dismissal.
  i18n keys optional with English fallbacks, matching the
  managingProfileBanner convention.

Tests: gateway/test_memory_status.py (classification bands, staleness,
clock skew, corrupt files, bool-is-not-int), lifecycle sentinel
carry-forward, /api/status contract (block always present, collector
crash degrades instead of 500), and 7 banner component tests.

NAS-side ingestion (agent-card notice + memory-tier upsell) ships
separately.

Refs NS-656; context: NS-608, NS-657, OOF-77.
2026-08-13 20:30:12 -07:00
Dmytro Afanasiev 294272c18c fix(kanban): preserve notifier subscription after done 2026-08-13 12:21:04 -07:00
kyinhub e5fd1c7b43 fix(kanban): make creator wake turns graph-safe
Carry the worker's completion handoff into the synthetic creator wake
turn and label it as an automatic notification with inspect-the-board /
don't-recreate guidance, so a woken orchestrator doesn't re-decompose
work that already exists (#70752).

Salvaged from PR #71100 by @yinkev; ported onto the restructured wake
region (delivery_mode gating, scope_id, sub chat_id destinations). The
auto_subscribe_on_create config-default half of the original PR was
dropped as already superseded on main.
2026-08-13 12:10:34 -07:00
Teknium 056f10acf6 fix(gateway): goal/heartbeat manager lookups for internal events skip activity touch
Widen #62804's class fix: _get_goal_manager_for_event and
_get_heartbeat_manager_for_event also call get_or_create_session on
behalf of the triggering event; when that event is internal the lookup
must not advance the user-activity clock either.
2026-08-13 12:04:35 -07:00
embwl0x 77166a5c46 test(gateway): cover activity-touch call contract 2026-08-13 12:04:35 -07:00
embwl0x 5462f689ba fix(gateway): keep internal wakes from extending sessions 2026-08-13 12:04:35 -07:00
Teknium f57116b231 fix(gateway): run wake-only kanban delivery before cursor advance with rewind/retry
For a push-adapter subscription with delivery_mode='wake' the visible text
ping is intentionally skipped (the send_passive gate), so the wake injection
IS the sole delivery — yet the event cursor advanced BEFORE the wake, which
then ran best-effort with its failure swallowed. A single failed wake
permanently lost the event.

Apply the same ordering the non-push (api_server) self-post branch already
uses: attempt the wake BEFORE advancing the cursor; on failure rewind the
claim (_kanban_rewind) and bump the per-sub failure counter so the next tick
retries; on success reset the counter; drop the subscription after
MAX_SEND_FAILURES consecutive failures like text sends do. notify+wake mode
is unchanged: the text ping is the delivery and the wake stays best-effort
after the cursor advance.

Extracts the residual delivery-ordering insight from closed PR #84191.

Co-authored-by: MaximCrabbe <crabbemaxim@gmail.com>
2026-08-13 12:04:07 -07:00
xaviersudre 0b600c859a fix(kanban): wake API subscriptions in destination session 2026-08-13 11:56:43 -07:00
Teknium 56f7ccd7a6 test(kanban): opt wake-scope subscriptions into notify+wake delivery mode
The delivery_mode gate on push-adapter wake injection landed on main after
PR #78391 branched; the plain 'notify' default never reaches the wake path
these tests assert. Salvage adaptation for salv-78391.
2026-08-13 11:41:19 -07:00
Nikita Barkov 5d3f75110a fix(kanban): key terminal-event wakes to the creator's workspace scope
Slack session keys include the workspace id since #70190, but the kanban
notifier rebuilds the wake source from a subscription row that has no scope
column, so every terminal-event wake keyed without the workspace.

The legacy-key adoption shipped in the same change (`_legacy_slack_session_key`,
`_recovered_row_matches_source_scope`) resolves that unscoped key onto the same
session_id, so the wake passes the busy guards that are keyed by routing key
(`_active_sessions`, `_running_agents`) and only collides afterwards, on session
id, under the per-session turn lease (#64934) — which serializes it behind the
live turn's flush. On a live Slack gateway that shows up as a duplicate run on
one task plus 400+s of waiting before the woken turn starts.

Same failure mode as #56580 / #72191 (chat_type), one field over, and it needs
no schema change: `_thread_metadata_for_source()` already stamps
`slack_team_id`, the notify subscription persists that dict as
`delivery_metadata`, and the notifier already unpacks it. Rows written by
`kanban_tools._maybe_auto_subscribe` carry no workspace, so fall back to the
adapter's channel → workspace map via `scope_id_for_chat()`, read with getattr
so adapters opt in and unscoped platforms' keys stay byte-identical. Slack
answers it from `_remember_channel_team`, which drops channels claimed by two
workspaces, so an unknown or ambiguous channel degrades to today's behavior
instead of guessing wrong.

Also adds the contributor email mapping the attribution check requires.

Co-authored-by: Junie <junie@jetbrains.com>
2026-08-13 11:41:19 -07:00
embwl0x 5a10537b24 fix(sessions): stabilize legacy reset lineage on resume 2026-08-13 23:45:21 +05:30
embwl0x ce89afa59c fix(sessions): keep reset conversations listable 2026-08-13 23:45:21 +05:30
Simon van Laak 21245fefac test(slack): cover native task-card progress — ID correlation, workspace scoping, fallback
Ports the test coverage from PR #29496 onto the salvaged implementation:
adapter-level serialization/workspace isolation/disconnect sealing, and
gateway-level ID-correlated concurrent duplicate tools plus the editable
text fallback when the native stream fails.
2026-08-13 11:12:19 -07:00
Arman 47ed5e4964 feat(slack): native streaming via chat.startStream/appendStream/stopStream
Slack's Agents & AI Apps feature ships a native streaming surface that
renders a live-typing message instead of the edit-based progressive
updates the adapter used until now.

The adapter now implements the existing draft-streaming interface:

- supports_draft_streaming() opts in whenever the app is connected and
  native streaming hasn't been detected as unavailable.
- send_draft() starts a stream on the first frame (chat.startStream,
  anchored to the resolved thread_ts, with recipient_team_id/user_id
  for channel streams) and appends only the delta on subsequent frames
  (chat.appendStream is append-only). The consumer's trailing cursor
  glyph is stripped before delta computation.
- Unlike Telegram drafts (ephemeral, replaced by a real sendMessage),
  a Slack stream IS the final message. send() therefore intercepts the
  turn-final delivery for a chat with an active stream whose streamed
  text is a prefix of the final content, and seals it via
  chat.stopStream with the remaining delta instead of posting a
  duplicate. Rich Block Kit (when enabled) is applied to the sealed
  message via chat_update, mirroring the finalize path in edit_message.
- Feature-gate errors from chat.startStream (not_allowed,
  missing_scope, unknown_method, ...) are cached on the adapter so
  subsequent runs skip straight to edit-based streaming with a single
  warning naming the fix (enable Agents & AI Apps for the app);
  transient errors only disable drafts for the current run via the
  consumer's existing send_draft failure handling.
- Segment breaks (new draft_id) and disconnect() seal any open stream
  so chats are never left with a dangling live-typing indicator.

No consumer or config changes: streaming.transport auto/draft now
lights up native streaming on Slack through the same interface
Telegram drafts use, and the edit-based path remains the fallback.
2026-08-13 11:12:19 -07:00
andyst-dev e49a7fe568 fix(google-chat): post cron deliveries as new top-level threads, not replies
_resolve_thread_id() falls back to _last_inbound_thread[chat_id] when no
explicit thread is present. That fallback exists for interactive DMs, where
Google Chat spawns a fresh thread per top-level user message and the adapter
drops thread_id to keep the session key stable. It also fired for cron
deliveries, which carry job_id in their metadata but no thread: the output
landed as a reply inside the last inbound thread instead of starting a new
top-level message.

Bypass the _last_inbound_thread fallback when metadata has a job_id (i.e. the
message is an automated cron delivery), so cron output posts at top level
unless an explicit thread is requested.
2026-08-13 11:05:40 -07:00
kshitij 8855766716 fix(gateway): expire orphaned drain markers past a max-age so a leaked marker can't wedge the gateway (#85433)
The NS-570 epoch stamp clears a drain marker that survives a machine
restart — but it assumes every drain-gated action ends in a restart. When
a maintenance action completes WITHOUT recreating the container and the
writer never cancels the drain, the orphaned marker still carries the
current epoch, so the 1s drain watcher honours it forever and the gateway
bounces every inbound message with the 'draining for a maintenance
action' text (observed in the field: a Hermes Cloud instance refused all
Telegram turns for ~3 days).

The marker already records requested_at; now the readers check it. A
marker older than DRAIN_REQUEST_MAX_AGE_SECONDS (1h) reads as stale in
drain_requested() and drain_notification_suppressed(), with a loud
warning log. Leniency mirrors the epoch check: a missing or unparseable
timestamp still reads as drain-active (fail-safe toward quiescing), and
a legitimately long drain keeps a sanctioned keep-alive — re-calling
write_drain_request() refreshes requested_at.

Fixes #85433
2026-08-13 23:23:48 +05:30
verybigdog 6e81ce273c feat(kanban): explicit notify/wake delivery modes with faithful wake session routing
Salvage of #37865 by @verybigdog. Adds delivery_mode (notify / notify+wake / wake)
on kanban notify subscriptions, persists chat_type + user_id_alt so a woken turn
reconstructs the creator's real session key, inherits the return path to child
tasks, and keeps wake out of the model-exposed send_message schema.

Original commits were authored under a local placeholder identity
(hermes-agent@users.noreply.local); re-attributed to the contributor's
public email.
2026-08-13 10:47:40 -07:00
Victor Kyriazakos 9c5d08c0d5 fix(gateway): /sethome must not persist Slack's synthetic per-message session thread as the home target
Third lane of the same contract (found in live staging validation):
/sethome run as a top-level relay-fronted Slack DM message captured the
adapter's session-keying thread stamp (the /sethome message's own id)
into the persisted HomeChannel.thread_id and its legacy env mirror.
Every bare-platform delivery (deliver="slack") then resolved home chat +
home thread and landed inside the ephemeral thread around the old
/sethome message. Extracted _home_thread_from_source with the same
synthetic-stamp recognition as cron origin capture; a /sethome run
inside a genuine thread keeps that thread as the home target. Users
repair an already-poisoned home target by rerunning /sethome.
2026-08-13 10:46:13 -07:00
Nikita Barkov 24ba866275 test(slack): cover handoff-thread ts and standalone media send response reads
Review on #74658 flagged that the response-shape suite exercised identity,
ephemeral and upload paths but left two changed call sites untested:

- create_handoff_thread's seed-message ts (adapter.py:2262), which anchors
  every subsequent handoff send onto the thread;
- the standalone media branch's chat_postMessage reads (adapter.py:8721 text
  post, :8749 caption fallback), where an SDK-shaped reply used to drop the
  ts and report a caption-only delivery as 'nothing deliverable'.

Both new cases run against the hand-rolled stand-in and the real
AsyncSlackResponse. Verified they fail against the pre-fix adapter.

Co-authored-by: Junie <junie@jetbrains.com>
2026-08-13 10:32:07 -07:00
Nikita Barkov 9cf2cbd382 fix(slack): read real SDK responses instead of gating on isinstance dict
Slack Web API calls return `SlackResponse`/`AsyncSlackResponse`, which are
mapping-like but not `dict` subclasses, so every `isinstance(resp, dict)`
gate took its "unexpected shape" branch at runtime: user and channel names
collapsed to raw IDs, every user resolved as a non-bot (defeating the
allow_bots loop guard), ephemeral replies were reported as failures, and
uploads/caption fallbacks lost their message_id.

Normalize responses through a single `_slack_response_payload()` helper
(dict passes through, SDK response yields `.data`, anything else yields
`{}` so callers keep their fallbacks) and use it at every call site.

Existing Slack tests injected plain dicts, which is why the defect was
invisible; the new tests run each behavioral case against a real
`AsyncSlackResponse` as well.
2026-08-13 10:32:07 -07:00
uperLu 02df90fc0c fix(gateway): propagate compression exhaustion result 2026-08-13 22:07:32 +05:30
Teknium 75736cd3a4 fix: don't double-count session-stream turns in the shutdown drain
The session chat stream registered its wrapper task in _active_run_tasks,
but that turn is already counted by active_agent_work_count() via
_inflight_agent_runs (_run_agent) — the drain saw 2 for one turn
(test_session_chat_sse_turn_is_interrupted). Keep only the agent-ref
registration; run-scoped steer control doesn't need the task entry.
2026-08-13 09:34:08 -07:00
Jon Komet 001bcb908e feat(api): steer active runs
Adds POST /v1/runs/{run_id}/steer and bridges Browser-Extension/WebUI
session chat streams into the active run registry so live runs on those
surfaces are steerable too.

- steer accepted only while run status is exactly 'running'; stop/stopping/
  terminal states return 409 run_not_accepting_steer even while cooperative
  shutdown retains the agent reference
- session SSE disconnect/cancellation interrupts and drains the executor-
  backed run instead of cancelling only the async wrapper; control refs stay
  registered until the turn actually exits
- undelivered steer text (accepted after the final response) is preserved as
  pending_steer on the terminal run.completed event/status so clients can
  replay it as the next user turn instead of losing it
- docs for the endpoint, next-tool-boundary delivery, acceptance-vs-delivery
  semantics

Salvaged from PR #54466 by @abundantbeing.
2026-08-13 09:34:08 -07:00
Christopher 136a911065 fix(whatsapp): classify npm install failures as non-retryable fatal errors (#80095) 2026-08-13 02:37:12 -07:00
Teknium e4aeb65599 feat(webhook): per-route toolset overrides for webhook agent runs
Webhook agent runs default to the constrained hermes-webhook toolset
(web/vision/clarify) because payloads can carry untrusted third-party
content. That default is right for public webhooks but wrong for trusted
local pushes (e.g. an OOM monitor daemon that needs the agent to run
ps/free/py-spy): the only workaround was widening platform_toolsets.webhook,
which elevates EVERY webhook route at once.

This adds a 'toolsets' key on individual webhook route configs (static
routes in config.yaml and dynamic subscriptions in
webhook_subscriptions.json) that replaces the platform-level resolution
for that route only:

- BasePlatformAdapter.toolsets_for_source(): per-source override hook,
  default None (no behavior change for any other platform).
- WebhookAdapter.toolsets_for_source(): maps the session chat_id
  (webhook:{route}:{delivery_id}) back to its route config and returns
  the route's toolsets list.
- GatewayRunner._resolve_enabled_toolsets_for_source(): shared resolver
  used by both agent-run call sites; validates the override through the
  SAME _get_platform_tools path as platform config, so unknown names and
  platform-restricted toolsets (e.g. discord_admin) are dropped rather
  than trusted.

Deliberately NOT exposed via 'hermes webhook subscribe': granting elevated
tools is a manual config edit only, so an agent-created subscription
cannot self-grant terminal at runtime.

Cache-safe: the toolset list is resolved before agent construction and is
constant for a route, so the per-session agent signature and frozen system
prompt are unaffected mid-conversation.
2026-08-13 01:51:19 -07:00
Teknium f4749a77a5 fix(mattermost): escalate genuine WS auth failures through the fatal-error hook
Follow-up to the salvaged #80489 substring-fallback removal: the
structured 401/403 branch still exited with a bare return, leaving
_running True — dead listener, healthy-looking is_connected(), gateway
never told (the zombie half of the bug, OOF-156 class). It now sets a
non-retryable mattermost_auth_error with token guidance and notifies
the gateway fatal handler.

Also: pytest.importorskip for aiohttp in the verifier probe file
(module-level import crashed collection in envs without the optional
dep), and probe fixtures updated for the escalation attributes.
2026-08-13 01:51:13 -07:00
Stephen Chin 684c18b428 test(mattermost): add verifier adversarial coverage for 401/403 classify fix
Independent-verifier boundary probes for commit fdd1a11ac5, covering
cases the implementer's regression tests did not exercise:
- WSServerHandshakeError(status=403) also stops the loop (only 401 tested)
- WSServerHandshakeError(status=500) does NOT stop the loop (structured
  check must not over-match on type alone)
- transient error containing the word 'unauthorized' (not digit substring)
  now retries correctly
- 5 consecutive transient errors all retry, not just the first

Verified these 2nd/4th tests fail against the pre-fix baseline commit
(01a1037d1e) and pass against the fix (fdd1a11ac5), confirming they
have real signal.
2026-08-13 01:51:13 -07:00
Stephen Chin d184d68f37 fix(mattermost): stop misclassifying transient errors as auth failures
The WS reconnect loop had a fallback check that looked for "401", "403",
or "unauthorized" as substrings anywhere in an exception's string form.
A transient error whose message happens to contain those digits (a proxy
body, a stack trace, anything) got treated as a permanent auth failure
and stopped reconnection for good.

I removed the substring fallback and kept only the structured check:
aiohttp.WSServerHandshakeError with status in {401, 403}. That's the only
signal that reliably means the server rejected our credentials.

Added two regression tests: one proving a transient error containing
"401" in its text still retries, and one confirming the existing
_closing early-return path is untouched by the removal.
2026-08-13 01:51:13 -07:00
kshitij 04d8222115 test: use constant in log assertion instead of hardcoded 75
Simplify-code finding: test hardcoded exit code 75 in string
assertion while already importing GATEWAY_SERVICE_RESTART_EXIT_CODE.
Use f-string interpolation so the assertion tracks the constant.
2026-08-13 14:02:05 +05:30
fangliquan 64aaf56dbc fix(gateway): contain cron provider shutdown exits 2026-08-13 14:02:05 +05:30