Commit Graph

1646 Commits

Author SHA1 Message Date
doncazper 6a375a24a5 fix(gateway): timestamp shared stderr output 2026-08-15 01:04:19 -07:00
Teknium 059e346422 fix(gateway): abort-aware parallel startup connects (restart race parity) 2026-08-14 21:57:41 -07:00
Teknium c22815fcaa fix(gateway): cap Telegram cold-start connect so startup reaches running fast (#85993)
CI / Docs Site (push) Has been cancelled
CI / Detect affected areas (push) Has been cancelled
CI / Python tests (push) Has been cancelled
CI / OS-specific tests (push) Has been cancelled
CI / Python lints (push) Has been cancelled
CI / JS & TS checks (push) Has been cancelled
CI / Installer tests (push) Has been cancelled
CI / Desktop E2E (push) Has been cancelled
CI / Deny unrelated histories (push) Has been cancelled
CI / Check contributors (push) Has been cancelled
CI / Check uv.lock (push) Has been cancelled
CI / Check no committed infographics (push) Has been cancelled
CI / package-lock.json diff (push) Has been cancelled
CI / Lint Docker scripts (push) Has been cancelled
CI / Supply-chain scan (push) Has been cancelled
CI / Review label gate (push) Has been cancelled
CI / OSV scan (push) Has been cancelled
CI / All required checks pass (push) Has been cancelled
CI / CI timing report (push) Has been cancelled
Deploy Site / deploy-vercel (push) Has been cancelled
Deploy Site / deploy-docs (push) Has been cancelled
Docker Build, Test, and Publish / Detect affected areas (push) Has been cancelled
Docker Build, Test, and Publish / build (amd64, type=gha,scope=docker-amd64, type=gha,mode=max,scope=docker-amd64, linux/amd64, ubuntu-latest) (push) Has been cancelled
Docker Build, Test, and Publish / build (arm64, type=gha,scope=docker-arm64, type=gha,mode=max,scope=docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Docker Build, Test, and Publish / publish (amd64, type=gha,scope=docker-amd64, type=gha,mode=max,scope=docker-amd64, linux/amd64, ubuntu-latest) (push) Has been cancelled
Docker Build, Test, and Publish / publish (arm64, type=gha,scope=docker-arm64, type=gha,mode=max,scope=docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Docker Build, Test, and Publish / merge (push) Has been cancelled
auto-fix lint issues & formatting / Generate eslint --fix patch (push) Has been cancelled
auto-fix lint issues & formatting / Apply patch (push) Has been cancelled
The initial (pre-running) connect awaited during gateway startup now uses
a capped 45s budget for Telegram instead of the full 180s (#67498) budget.
On timeout the platform is queued for the reconnect watcher, which retries
with the full budget and is_reconnect=True (preserving the offline update
queue, #46621). Combined with the parallel startup connects, an unreachable
Telegram no longer holds the whole gateway out of the running state.
2026-08-14 21:57:41 -07:00
EvanProgramming d86c67dc7e fix(gateway): connect messaging platforms in parallel at startup (#83791)
GatewayRunner.start() previously awaited each platform's connect() (with its
own timeout) in a serial for-loop. A single slow/failing platform (e.g.
Telegram behind a dead proxy) delayed every later platform's connect by a full
timeout window, cascading one platform's failure onto WeChat/QQ/etc.

Now the slow connect() calls run concurrently via asyncio.gather while the
serial pre-filter (checks, adapter creation, handler wiring) and the
single-threaded result aggregation (shared-state mutation, error handling)
are unchanged. A failing platform no longer blocks the others.

Adds regression tests proving connect() calls overlap and that one failing
platform leaves the others connected.
2026-08-14 21:57:41 -07:00
kshitij 24573b396b refactor: bind gateway anti-growth guard to locals, trim overlong comment
Simplify-code pass: gateway/run.py called estimate_messages_tokens_rough
6x on the same data in the anti-growth guard (condition + warning f-string).
Bind to _hyg_in_toks/_hyg_out_toks locals like the conversation_compression.py
guard already does. Also trim the comment from 10 lines to 4 (keep the WHY,
drop the WHAT) and remove an extra blank line before TestCompactedTurnsStaySearchable.
2026-08-15 10:22:47 +05:30
dhruv kejriwal e7418f1621 fix(gateway): never persist a hygiene compression that grows the transcript
Session hygiene could persist a compressed transcript LARGER than the
original (observed: 427K -> 598K), when the generated summary was bigger
than the middle it replaced. Compare like-for-like (both rough estimates)
before persisting a rotated transcript; on growth, keep the original
unchanged so a failed compression is a strict no-op, never a net increase.
2026-08-15 10:22:47 +05:30
joaomarcos baf7034850 fix(gateway): deliver interrupted-cron notices before adapters disconnect
When the shutdown drain times out and kills an in-flight cron job, the
job's owner is never told. The cron worker does try: `_is_interrupted()`
forces the failure path with an honest "interrupted by gateway shutdown"
error, and failed jobs always deliver. But that worker is a thread, it
reaches `_deliver_result()` asynchronously, and by then
`_bounded_adapter_teardown()` has closed the transport. The reporter of

Worse, the loss is silent twice over: `_consume_interrupted_flag()`
returns True — the gateway already wrote `last_status` — so
`mark_job_run()` is skipped, and the `delivery_error` from the failed
send is discarded with it. The run's only trace is a generic line in
jobs.json.

The gateway already owns the right window. `_notify_active_sessions_of_
shutdown()` runs while adapters are up, precisely so shutdown messages
can be sent — but it iterates `_running_agents`, and cron work lives on
the scheduler's own thread pool. Same structural blindness already fixed
for counting (#60432) and draining (#63529), never fixed for notifying.

So notify from the post-interrupt phase, which is the last point where
the transport is still up: `_kill_tool_subprocesses()` now returns the
job IDs it marked, and `_notify_interrupted_cron_jobs()` sends each one's
owner a notice on the job's own resolved delivery targets. Adapter
teardown order is untouched — it is load-bearing for #53175 and #8202.

Jobs with `deliver: local`, and `deliver: origin` jobs with no resolvable
origin (#43014), resolve to zero targets and stay silent. Per-platform
`gateway_restart_notification: false` is honoured, matching the chat
path. Every failure is swallowed so a wedged adapter cannot extend
shutdown.

Second, when the interrupted flag short-circuits `mark_job_run()`, the
delivery failure is now persisted on its own via `update_job()`, so a
notice that still cannot be sent is at least recorded. `update_job()`
rather than a second `mark_job_run()`: the latter also advances
`next_run_at` and the repeat counter, and running that twice for one run
would skip a fire or auto-delete the job early.

Fixes #82232. Related: #82161, #82224.
2026-08-14 21:47:16 -07:00
joaomarcos 4b06d9e9b1 fix(gateway): keep the cron drain floor compatible with shutdown test doubles
CI slice 5/12 caught two ways the new cron budget broke `_stop_impl_body`
for callers that are not real GatewayRunner instances:

- `_FakeGateway` in test_shutdown_cache_cleanup.py borrows `_stop_impl`
  without subclassing, so it never picked up the class-level
  `_cron_drain_timeout` default and raised AttributeError. Read it through
  the getattr-guard convention the same function already uses for its
  liveness-guard machinery.
- The same double overrides `_drain_active_agents(self, timeout)`, so
  passing the cron budget raised "takes 2 positional arguments but 3 were
  given". The double now mirrors the real optional parameter. It is the
  only override in the tree; test_startup_restart_race.py uses AsyncMock,
  which accepts any signature.

Verified against a stashed clean tree: the 22 gateway test files that
still fail locally fail identically with and without this branch (80 = 80,
empty set difference both ways) — they are pre-existing Windows-only
failures (setsid, POSIX modes) unrelated to this change.
2026-08-14 21:47:16 -07:00
joaomarcos 45bb486b26 fix(gateway): give in-flight cron work its own drain floor
`agent.restart_drain_timeout` defaults to 0 and governed every class of
in-flight work at once. That default is deliberate for chat turns: the
gateway announces the restart to the user and pre-marks the session
resume_pending, so interrupting one is cheap and recoverable.

A cron run has neither property. Nobody is waiting on it, it is written
to jobs.json as a permanent failure, and a recurring job simply skips to
its next schedule. Sharing the chat budget meant `_drain_active_agents()`
short-circuited on `timeout <= 0` before entering the wait loop, so the
drain reported `drain took 0.00s, timed_out=True, cron_at_start=1,
cron_now=1` — it detected the job and killed it anyway.

Cron work now drains on its own deadline, `agent.cron_drain_timeout`
(default 30s, 0 opts out). The floor is clamped to the shutdown-watchdog
leash minus a teardown reserve, so the longer wait can never consume the
post-drain cleanup window: being SIGKILLed mid-cleanup would leave the
job wedged at `last_status=running`, strictly worse than the bug. Being
bounded also means a cron-triggered restart cannot deadlock on itself.

The `timeout <= 0` special case is gone — an expired deadline expresses
the legacy "interrupt immediately" behaviour, so `timed_out` is always
computed from real state instead of asserted up front. The drain-timeout
warning now reports the elapsed wait rather than the configured budget,
which is what made "timed out after 0.0s" so confusing in the report.

Chat-only shutdowns are unchanged: `restart_drain_timeout: 0` still
interrupts chat turns immediately.

Relates to #82161 (complements #82195, which removes the `hermes update`
self-deadlock that triggered the reported instance).
2026-08-14 21:47:16 -07:00
deacon-botdoctor 9bff109783 fix(gateway): cancel native clarify only on free prose
Teknium review on #75732: releasing the pending clarify whenever
resolve_text_response_for_session returned False also cancelled
retryable multi-select invalid selections (out-of-range numbers,
unrecognised comma-lists).

Classify rejected typed replies in clarify_gateway:
- rejected_prose → cancel clarify, fall through busy routing (deadlock break)
- rejected_selection → keep clarify armed so the user can retry

Add native multi-select gateway regressions for both paths.
2026-08-14 21:24:36 -07:00
loulanyue c1c3557723 fix(gateway): release rejected native clarifies before steering 2026-08-14 21:24:36 -07:00
Teknium bc5805c35f fix: compare base-URL hostnames, not substrings, in provider-identity checks
Port of the bug class from earendil-works/pi#7933 (DeepSeek base-URL
detection matched by raw substring, missing case variants and matching
lookalike URLs). Hermes had the same class at five sites:

- cli_agent_setup_mixin.py: keyless-custom-endpoint detection treated any
  URL containing the OpenRouter host substring (path segment, lookalike
  domain) as OpenRouter, and missed case variants of the real host.
- models.py validate_requested_model: same substring check for routing an
  openrouter provider with a custom base_url to the custom catalog.
- runtime_provider.py: local-endpoint autodetect matched the string
  localhost anywhere in the URL, including remote hostnames containing it.
- gateway/run.py: /status endpoint display, same local-host substring.
- agent_runtime_helpers.py: Nous Portal cache-layout detection matched
  the nousresearch substring anywhere in the URL.

All sites now use the existing base_url_host_matches / base_url_hostname
helpers (exact host or subdomain, case-insensitive). Regression tests
proven to fail against the old predicates.
2026-08-14 20:47:13 -07:00
Evgenii acaafcc6bb fix(cron): make immediate execution race-safe
- claim_job_for_fire returns the atomically claimed snapshot with a unique
  fire owner; heartbeat_fire_claim renews the lease; mark_job_run fences
  terminal writes by expected_fire_owner so a stale worker cannot record
  over a replacement claim.
- run_one_job heartbeats the fire claim and forwards a combined cancel
  event (ownership loss OR external cancel) into run_job; the agent path
  is interrupted cooperatively and script-based jobs (no_agent + pre-run
  scripts) are hard-stopped with a process-tree kill (POSIX killpg
  SIGTERM then SIGKILL for surviving group members; Windows
  taskkill /T /F), with a bounded pipe drain so a SIGTERM-ignoring
  descendant cannot wedge the worker on communicate() EOF.
- Shutdown interruption is scoped to the exact execution token instead of
  the bare job ID, so a replacement run of the same job never consumes a
  stale interrupted flag.
- fire_claim_fence serializes save/deliver side effects per profile+job
  with a cross-process flock; remove_job prunes the fence-lock entry.
- Preserves upstream BaseException terminal recording (#73973),
  completed one-shot retention (#80624), blocked_config preflight
  (T1-26), and the advance_next_runs batch on top of current main.
2026-08-14 20:46:50 -07:00
Seashore Shi 5bc13cf574 fix(gateway): tolerate list-shaped gateway.platforms from setup (#83185)
`hermes gateway setup` writes `gateway.platforms` as a LIST of
enabled platform names (e.g. `- telegram`), not a dict.  Treat any
non-dict shape as "no per-platform overrides" instead of crashing
on `.get()` for every incoming turn (#83185).

Co-authored-by: SeashoreShi <seashore.shi@gmail.com>
2026-08-15 03:05:23 +05:30
teknium1 f79440e0f4 feat: /loop — recurring in-session wakeups (Claude Code parity)
Ports Claude Code's /loop (and its /proactive alias) across every Hermes
surface. /loop [interval] <prompt> re-runs a prompt or slash command on a
recurring cadence inside the live session; omitting the interval enables
self-paced mode (starts at the floor, backs off exponentially while the
agent's replies stop changing, snaps back on change — local digest
comparison, zero extra LLM cost).

Stop conditions: agent-emitted LOOP_COMPLETE marker, --times N,
--until <condition> (judged by the existing goal_judge aux task,
fail-open), /loop stop, and a loops.max_ticks backstop budget.

Core: hermes_cli/loops.py (LoopState + LoopManager + shared
dispatch_loop_command), persisted per session in SessionDB state_meta
(loop:<sid>) so /resume picks it up; migrates across compression
boundaries like /goal. New SessionDB.list_meta_prefix() powers the
gateway's cross-session scan.

Surfaces:
- CLI: /loop handler + idle-fire and post-turn-complete hooks in
  process_loop (mirrors the /goal hook shape; Ctrl+C pauses the loop)
- Gateway: /loop handler with route capture, mid-run control-verb guard,
  post-turn tick completion, and a supervised loop_wakeup_watcher that
  injects due wakeups into idle chats via the synthetic-message path
- TUI/dashboard/desktop: command.dispatch handler + per-session
  notification-poller wakeup driver + post-turn completion in the turn
  dispatcher; /loop added to the desktop slash palette
- /goal mixing: an active non-parked goal owns the idle boundary — loop
  ticks defer until it finishes, pauses, or parks; real user input always
  wins over both

Config: loops.{min_interval_seconds,max_ticks,self_paced_floor_seconds,
self_paced_ceiling_seconds}. Docs page + sidebar entry. 77 new tests.
Slack's 50-slash cap: /version moves to /hermes version to free the
native slot for /loop.
2026-08-14 13:40:19 -07:00
Teknium a90d5369f7 feat(gateway): dump wedged worker stacks when the turn reaper fires
When the inactivity reaper interrupts a timed-out turn, the interrupt
frees the blocked frame — destroying the only evidence of where the
turn was wedged. The Aug 2026 zombie-turn incident (WhatsApp session,
Relay-corrupted scope stack) wedged every turn for exactly the 1800s
timeout somewhere between 'Turn ended' and run_sync returning, and the
wedge point was unprovable post-mortem.

The reaper now logs the stack of every thread with turn-machinery
frames BEFORE interrupting, so the next occurrence names the exact
blocked line. Best-effort, bounded (8 threads, 25 frames), pure
in-process, never raises into the reaper.
2026-08-14 11:22:14 -07:00
Teknium afe09c7942 fix(gateway): exclude wedged turns from the restart after-turn wait
A turn idle past agent.gateway_timeout (the same threshold the turn
reaper uses) no longer defers an in-band restart. Restart is usually
the remedy for a wedged turn; waiting restart_after_turn_timeout on
one inverts the graceful path's purpose — a wedged WhatsApp turn
pinned 'hermes update' in draining until SIGTERM was sent manually
(Aug 2026). stop()'s bounded drain interrupts wedged turns instead.

gateway_timeout=0 (unbounded turns) disables wedge detection; cron
and API-server work has no per-turn activity clock and is never
counted as wedged; unreadable activity summaries fail open.
2026-08-14 11:22:05 -07:00
xxxigm e37a0321eb fix(approval): show CLI Dangerous Command prompt when ask-mode has no notifier
HERMES_EXEC_ASK (and gateway platform markers without a notify callback)
were short-circuiting interactive CLI into silent pending_approval, so
the Approve/Deny panel never appeared. Prefer the registered CLI callback
when present, and set HERMES_EXEC_ASK only in start_gateway so importing
gateway.run from CLI tools cannot poison the process.
2026-08-14 17:48:28 +05:30
dsad 858a6008be fix(gateway): keep process notification routing off-loop 2026-08-14 01:21:52 -07:00
Teknium 7619564fbd fix(gateway): gate background-process completions on spawning-session boundary
Plain type=completion events built in _run_process_watcher carried only
session_key (chat/thread routing) with no spawning-session stamp, so after
/new (or a session switch) a completion notification from the OLD session
was injected into the chat's NEW session. Main already solved this exact
class for async delegations via the _classify_completion_target pre-flight
(_USER_BOUNDARY_END_REASONS drop on user-closed sessions, deliver on
idle-ends, follow the compression-tip chain), but the gate only ran for
type=async_delegation events.

Kernel salvage of #16455:

- Stamp the spawning conversation's session-db id (HERMES_SESSION_ID via
  session-scoped env) on the ProcessSession and the pending_watchers entry
  at spawn time in tools/terminal_tool.py; persist it through the process
  registry checkpoint/restore so recovered watchers keep the stamp.
- Thread the stamp into the completion_evt built by _run_process_watcher
  (watcher entry first, ProcessSession fallback for recovered watchers).
- In _deliver_completion_notification, run the SAME pre-flight classifier
  for stamped type=completion events: terminal -> drop with a log (output
  stays available via process(action='log')), retry -> False so the
  watcher re-polls, deliver -> proceed. The policy has exactly one owner
  (_classify_completion_target); nothing is forked. Unstamped legacy
  events keep today's deliver-always behavior, and the async-delegation
  path is untouched.

Based on the session-boundary approach from #16455 by @Tosko4 (original PR
was over-scoped across adapters/slash-commands/cron; this lands the kernel
only).

Tests: completion from a /new-closed session is dropped; completion after
an idle-end still delivers; unstamped legacy event delivers; retry verdict
returns retryable False without adapter injection; async_delegation gate
unchanged; stamp survives checkpoint recovery.
2026-08-14 01:09:42 -07:00
Teknium b9e7bead13 fix(gateway): coalesce same-tick async-delegation completions into one turn (#70300)
The async-delegation watcher drained the completion queue as a batch but
then delivered each event as its own synthetic turn, flooding the session
when a fan-out of background subagents finished together. Builds on the
per-process completion batching salvaged from PR #71898 (thanks
@yuzilongleif-collab) which coalesces concurrent _run_process_watcher
completions behind a short per-route fan-in window.

This commit adds the async-delegation half: group the drained batch by
full routing key (session_key + parent_session_id + platform/chat/thread/
user) and inject ONE consolidated turn per group. Durable-ack handling
stays honest: sibling rows are claimed up front via claim_event_delivery;
rows another consumer owns are excluded from the consolidated text (no
double-delivery); sibling claims are acknowledged only after adapter
acceptance and released (still pending) on failure. Events for different
sessions never coalesce, and a single-event group rides the existing
per-event path unchanged (latency and text identical).

Tests: 3 same-tick events -> exactly one adapter.handle_message carrying
all 3 results with all 3 durable rows delivered; 2 sessions -> 2 turns;
single-event path unchanged; failed batch releases claims and retries;
foreign-claimed sibling excluded and left pending.
2026-08-14 01:08:25 -07:00
yuzilongleif-collab a96cd10349 fix(gateway): force-redact coalesced completion output 2026-08-14 01:08:25 -07:00
yuzilongleif-collab 7536655b8f fix(gateway): own completion batch task lifecycle 2026-08-14 01:08:25 -07:00
yuzilongleif-collab cf09a30a9a fix(gateway): coalesce concurrent process completions 2026-08-14 01:08:25 -07:00
Teknium 8dc9401d7e fix(gateway): persist internal synthetic turns typed as internal_notification (#82888)
Async-delegation batch completions and background watch notifications
re-enter the gateway as synthetic MessageEvent(internal=True) turns via
_inject_watch_notification, but were persisted as bare role='user' rows —
indistinguishable from real user input in transcripts and the desktop UI.

Thread the event's internal flag through to persistence: when
event.internal is set, the turn's persisted user row is stamped
display_kind='internal_notification' (the existing DB-only presentation
sidecar used by auto_continue / model_switch rows). Wired through
_run_agent → _run_agent_inner → TurnContext → run_conversation's
persist_user_display_kind, and onto the three gateway-side fallback user
rows (transient failure, no-new-messages, pre-run crash), whose
append_to_transcript writer now forwards display_kind/display_metadata
to SessionDB.append_message.

Invariants preserved: role stays 'user' (alternation untouched), no new
injections, no past-context mutation, and display_kind is already popped
from every provider-bound copy in conversation_loop, so replayed sessions
never leak the marker to the API.

Regression tests: internal turn marked, real user turn unmarked, fallback
rows marked/unmarked per event, and a DB round-trip proving replay keeps
role/content intact while the provider copy drops the marker.
2026-08-14 01:08:18 -07:00
handnewb 84b4fb9eca fix(gateway): force user-facing redaction on the direct completion and running-output sends
The terminal redactor is called without force=True on all three
user-facing sends in _run_process_watcher, so process output reaches the
platform raw when security.redact_secrets is disabled. The agent-notify
path was already covered; this covers the two direct adapter.send()
paths the sweeper identified.

Review: teknium1 (#73547)
2026-08-14 01:08:13 -07:00
handnewb c0d204810d fix(gateway): redact secrets in background process completion output
_output passes through redact_terminal_output (force=False) but
not _redact_gateway_user_facing_secrets (force=True). When
security.redact_secrets is disabled or the command is not an
env-dump, the output can leak credentials (Authorization: Bearer,
PGPASSWORD=, etc.) into the session transcript and chat platforms.

Add the same redaction gate applied to _command so both paths
are protected by force=True + _GATEWAY_SECRET_PATTERNS.
2026-08-14 01:08:13 -07:00
spfcraze d2d7766750 fix(tools,gateway): format watch_overflow events instead of dropping them
format_process_notification had no case for watch_overflow_tripped /
watch_overflow_released, so a watch-pattern notification flood surfaced
as '[IMPORTANT: Background process  exited (exit code ?)]' — a phantom
exit notification for a process that never existed — while the actual
'watch flood, N notifications suppressed' summary in the event's
message field was silently dropped. The gateway delivery path was
worse: _drain_gateway_watch_events retained only watch_match and
watch_disabled, discarding overflow events entirely before formatting.
Route both event types through the message field in the shared
formatter and the gateway formatter, and retain them in the gateway
drain.
2026-08-14 01:08:13 -07:00
Lidang-Jiang 9b554758bd fix(gateway): honor notification-off watch reinjection
Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>
2026-08-14 01:08:13 -07:00
Teknium 1c971769ec feat(gateway): concise background process notifications by default
Background process completions on messaging platforms now default to a
one-line status message (✅/❌ + command + duration; failures append a
short output tail) instead of dumping the raw output buffer into the
chat. New display.background_process_notifications mode 'concise' is
the default; 'all' keeps the old raw-dump behavior for anyone who wants
it. Config migration v35 moves users still on the old implicit default
'all' to 'concise' on their next update; explicit result/error/off
choices are preserved.
2026-08-14 00:26:11 -07:00
Teknium 056f10acf6 fix(gateway): goal/heartbeat manager lookups for internal events skip activity touch
Widen #62804's class fix: _get_goal_manager_for_event and
_get_heartbeat_manager_for_event also call get_or_create_session on
behalf of the triggering event; when that event is internal the lookup
must not advance the user-activity clock either.
2026-08-13 12:04:35 -07:00
embwl0x 5462f689ba fix(gateway): keep internal wakes from extending sessions 2026-08-13 12:04:35 -07:00
Simon van Laak eb137762ff feat(slack): render tool progress as native plan/task cards (opt-in)
Adds platforms.slack.extra.native_task_cards: when enabled, live tool
calls render as Slack-native plan/task cards via chat.startStream /
chat.appendStream (task_display_mode: plan, task_update chunks) instead
of text/edit progress bubbles. ID-bearing tool_start/tool_complete
callbacks correlate concurrent same-name tool calls correctly; any
native API failure falls back to one continuously edited text update.
The stream is stopped exactly once when the turn finalizes.

Salvaged from PR #29496 onto current main (TurnRunner/TurnContext seam);
closes #29483.
2026-08-13 11:12:19 -07:00
verybigdog 6e81ce273c feat(kanban): explicit notify/wake delivery modes with faithful wake session routing
Salvage of #37865 by @verybigdog. Adds delivery_mode (notify / notify+wake / wake)
on kanban notify subscriptions, persists chat_type + user_id_alt so a woken turn
reconstructs the creator's real session key, inherits the return path to child
tasks, and keeps wake out of the model-exposed send_message schema.

Original commits were authored under a local placeholder identity
(hermes-agent@users.noreply.local); re-attributed to the contributor's
public email.
2026-08-13 10:47:40 -07:00
uperLu 02df90fc0c fix(gateway): propagate compression exhaustion result 2026-08-13 22:07:32 +05:30
Teknium e4aeb65599 feat(webhook): per-route toolset overrides for webhook agent runs
Webhook agent runs default to the constrained hermes-webhook toolset
(web/vision/clarify) because payloads can carry untrusted third-party
content. That default is right for public webhooks but wrong for trusted
local pushes (e.g. an OOM monitor daemon that needs the agent to run
ps/free/py-spy): the only workaround was widening platform_toolsets.webhook,
which elevates EVERY webhook route at once.

This adds a 'toolsets' key on individual webhook route configs (static
routes in config.yaml and dynamic subscriptions in
webhook_subscriptions.json) that replaces the platform-level resolution
for that route only:

- BasePlatformAdapter.toolsets_for_source(): per-source override hook,
  default None (no behavior change for any other platform).
- WebhookAdapter.toolsets_for_source(): maps the session chat_id
  (webhook:{route}:{delivery_id}) back to its route config and returns
  the route's toolsets list.
- GatewayRunner._resolve_enabled_toolsets_for_source(): shared resolver
  used by both agent-run call sites; validates the override through the
  SAME _get_platform_tools path as platform config, so unknown names and
  platform-restricted toolsets (e.g. discord_admin) are dropped rather
  than trusted.

Deliberately NOT exposed via 'hermes webhook subscribe': granting elevated
tools is a manual config edit only, so an agent-created subscription
cannot self-grant terminal at runtime.

Cache-safe: the toolset list is resolved before agent construction and is
constant for a route, so the per-session agent signature and frozen system
prompt are unaffected mid-conversation.
2026-08-13 01:51:19 -07:00
fangliquan 64aaf56dbc fix(gateway): contain cron provider shutdown exits 2026-08-13 14:02:05 +05:30
Teknium 6397776fe8 refactor: simplify discord classifier + move attention threshold to config.yaml
- Collapse the duplicated discord LoginFailure/PrivilegedIntentsRequired
  classification (name-match + isinstance blocks repeated the same code/
  message tuples) into a single _is() helper — one message per failure.
- Replace the user-facing HERMES_RECONNECT_ATTENTION_AFTER_SECONDS env var
  with agent.reconnect_attention_after in config.yaml (default 7200, 0
  disables), bridged internally like gateway_timeout. .env is for secrets.
- Use _float_env for robust parsing instead of bare int(os.getenv(...)).
- Document terminal classification + needs_attention escalation in
  website/docs/user-guide/configuration.md.
2026-08-12 22:16:12 -07:00
Shannon Sands 91bc822330 fix(gateway): classify terminal adapter connect failures + escalate long-lived retry loops (OOF-156)
Fleet triage after the 2026-08-11 storm resolution found agents whose sole
platform had been silently 'retrying' for weeks: revoked Telegram tokens,
Discord privileged-intent rejections, and Photon sidecars that can never
start were all funnelled into the indefinite reconnect queue with no owner
signal (OOF-151/152/153, epic OOF-156).

Two-part fix:

1. Per-adapter classification — by exception TYPE only, never message text:
   - telegram: InvalidToken/Forbidden -> telegram_auth_error, retryable=False
     (new _looks_like_auth_error, mirrors _looks_like_network_error)
   - discord: LoginFailure -> discord_auth_error, PrivilegedIntentsRequired
     -> discord_intents_required (both retryable=False); every other path now
     sets an explicit code (previously the generic branch set NO fatal info,
     which the gateway read as 'probably transient')
   - photon: new typed PhotonSidecarStartupError; deps-install failure ->
     SIDECAR_DEPS_MISSING and missing node binary -> SIDECAR_NODE_MISSING
     (retryable=False); ambiguous startup crashes stay retryable
   - email: IMAP/SMTP failures now always set a fatal code;
     SMTPAuthenticationError -> email_auth_error, retryable=False (IMAP4.error
     is type-ambiguous between bad creds and transient NOs, so IMAP stays
     retryable)

2. Gateway escalation — platforms continuously in the reconnect queue past
   HERMES_RECONNECT_ATTENTION_AFTER_SECONDS (default 2h, 0 disables) get
   needs_attention=true + retrying_since stamped into runtime status, once
   per episode, cleared on successful reconnect.

Deliberately NOT a circuit breaker: retries never stop. The auto-pause
mechanism was removed for good reason (transient DNS outages left bots
silently dead); this preserves that and only adds visibility. No new
platform_state enum values — NAS's status schema is strict — only additive
fields.

Unknown exception types always stay retryable: a false terminal recreates
the silently-dead-bot problem, and the escalation path covers
misclassified permanent failures.
2026-08-12 22:16:12 -07:00
Teknium 4ea2a0e546 Revert "Inspired by Perplexity Computer: Model Council mode for Mixture of Agents"
This reverts commit 8d9e18d40b.
2026-08-12 21:50:35 -07:00
Hermes Agent 8d9e18d40b Inspired by Perplexity Computer: Model Council mode for Mixture of Agents
Adds a 'council' synthesis style to MoA (per preset via synthesis_style,
one-shot via the new /council command on CLI + gateway). Reference models
answer independently; the aggregator chairs the deliberation and produces
a user-facing report of consensus, per-model disagreements (with the
differing assumptions behind them), unique contributions, and a
recommendation with an explicit confidence level.

Inspired by Perplexity's Model Council rollout to Perplexity Computer
(changelog 08/04/26): pick a board of 2-8 models, run them independently,
synthesize where they agree/disagree and what each uniquely surfaces.
2026-08-12 19:44:09 -07:00
Teknium 11310068c6 feat(plugins): pre_command observer hook + capability-gated ctx.call_mcp (#64204)
Part A — pre_command observer hook (observer-first per #64182 ground rule 3):
- New VALID_HOOKS event `pre_command`: fires when a recognized slash command
  is about to be dispatched, BEFORE the handler runs, on both surfaces:
  - CLI: cli.py process_command (right after alias resolution)
  - Gateway: gateway/run.py _handle_message cold-path canonical dispatch
- Payload: surface ('cli'|'gateway'), command (canonical), alias_used,
  args_raw, session_key, platform. Return values IGNORED in v1; a plugin
  returning a directive-shaped dict gets a debug log so future
  block/rewrite adopters are discoverable (#64231 taxonomy).
- Deliberately NOT fired on the gateway running-agent intercept path
  (/stop, /approve, busy_policy dispatch during an active run): those are
  control-plane escape hatches on an in-flight run and must stay outside
  plugin observation/veto reach.
- fire_pre_command_hook() helper never raises, so broken plugin infra can
  never break command dispatch.

Part B — ctx.call_mcp (capability-gated, default-off, ground rule 4):
- PluginContext.call_mcp(server, tool, arguments, timeout=30): synchronous,
  callable from plugin hooks/tools, routes through the EXISTING native MCP
  client machinery (tools.mcp_tool._make_tool_handler: background loop,
  trust-tier gates, circuit breaker, reconnect) — never a parallel client.
- Gate: plugins.entries.<id>.mcp_allowlist (list of server names).
  Absent key / unreadable config / non-list value => default-deny.
  Unlisted server raises PermissionError naming the exact config key.
  TODO seam left for the #64228 declared-capability model.
- Bounded: timeout clamped to 1-600s and forwarded to the MCP loop call;
  results capped at 64KB with truncation marker; stable
  {ok, result|error, structuredContent?, truncated?} envelope.

Tests (transport mocked, no live MCP servers):
- tests/hermes_cli/test_pre_command_hook.py: both surfaces fire, canonical
  alias reporting (/exit->quit, /q->queue), hook-before-handler ordering,
  control-plane exclusion, hook failure non-fatal, observer-only directive
  handling.
- tests/hermes_cli/test_plugin_call_mcp.py: default-deny (absent entry,
  unreadable config, non-list, '*'), allowlist enforced per-server,
  denied calls never touch transport, timeout forwarding/clamping,
  result truncation, error/structuredContent envelopes.

Docs: hooks.md shipped-catalog row for pre_command; plugins.md
"Calling MCP servers from plugins" section with the security note.

Closes #64204
2026-08-12 19:16:59 -07:00
doncazper 85020f2238 fix(plugins): isolate ownership by profile 2026-08-12 19:13:32 -07:00
Hans 52eb8eb533 feat(plugins): add pre_transcription hook and STT prompt threading
Adds a pre_transcription transform hook (prompt/language/model mutable,
file_path read-only, last-writer-wins per the transform_* convention)
fired before any STT backend, threads prompt to faster-whisper
(initial_prompt) and OpenAI/Groq/Mistral/DeepInfra (prompt), adds an
optional stt.prompt config key on the same plumbing, and keeps the
no-hook dispatch path byte-identical. Fixes #64168.

Documents the new surface for users: a "Transcription prompt
(vocabulary hints)" subsection in the configuration guide (composition
order, per-provider support matrix, length contract, privacy warning),
a pre_transcription entry in the hooks reference, and the mirrored row
in the plugins hook table.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AG6LyYMvHC2o6HbVUozmVR
2026-08-12 19:01:30 -07:00
GodsBoy e64fb2b614 fix(review): harden plugin gateway injection 2026-08-12 18:25:33 -07:00
GodsBoy f46c600a54 feat(gateway): allow plugins to inject session messages 2026-08-12 18:25:33 -07:00
Teknium 1e46e09bbd fix(gateway): scope reaction observers to routed profiles 2026-08-12 16:42:28 -07:00
Teknium 9994bc9ec9 fix(gateway): enforce post-auth normalized reaction observer
Builds on Paolo Antinori's #68431 salvage for #64176. Move plugin dispatch behind the profile-scoped runner authorization boundary, fail closed on malformed reaction identities, preserve observer registration across Telegram app rebuilds, and document the deliberately observer-only contract.

Co-authored-by: Paolo Antinori <pantinor@redhat.com>
2026-08-12 16:42:28 -07:00
Ben Barclay bb597e1c02 fix(cron): managed-cron fires execute in the gateway process (live adapters + dashboard forwarder) (#84339)
* fix(gateway): pass live adapters to cron fire webhook's fire_due

The Chronos fire webhook (/api/cron/fire) called
provider.fire_due(job_id, adapters=None, loop=loop), so every
externally-triggered fire delivered through the standalone path even
with a live gateway in-process. E2EE platforms and relay-fronted
logical platforms (whose ONLY send path is the live relay adapter — no
native credential exists on the box) failed every external fire with
"platform 'X' not configured/enabled", while the same job delivered
fine under the built-in ticker (gateway/run.py passes runner.adapters).

Resolve the runner (self.gateway_runner → app['gateway_runner'] →
_gateway_runner_ref(), the same chain the drain check uses) and forward
its adapters. No runner → adapters=None, preserving the historical
standalone path byte-identically.

Note: does not by itself fix Fly-hosted scale-to-zero deployments where
NAS's callback lands on the DASHBOARD process (internal_port 9119) —
_fire_cron_job_for_profile there has no gateway runner in-process. That
topology needs a separate fire handoff (design pending).

* fix(cron): dashboard forwards Chronos fires to the gateway (503 when unreachable)

The dashboard's /api/cron/fire executed cron jobs in the DASHBOARD
process via _fire_cron_job_for_profile with adapters=None. On hosted
deployments (Fly proxy exposes only the dashboard's port) that made
every managed-cron fire deliver through the standalone send path, which
cannot serve relay-fronted logical platforms (their only sender is the
live relay adapter in the gateway process — no native credential exists
on the box) or E2EE rooms. It also ran the whole agent turn inside the
dashboard: wrong process for memory/session ownership and fire-claim
attribution.

Restore the invariant that the GATEWAY owns cron execution:

- Dashboard route: after verifying the NAS JWT and resolving the job's
  profile, FORWARD the fire to the gateway api_server's own
  /api/cron/fire on loopback, NAS bearer preserved (the gateway
  re-verifies the JWT — defense in depth, no new trust link), and pass
  the gateway's response through. Gateway unreachable → 503 so NAS
  retries per the Chronos contract (non-2xx = retryable; the store CAS
  de-dupes the eventual double fire). Deliberately NO local-execution
  fallback.
- Endpoint resolution mirrors gateway/config.py's api_server load order
  per target profile (config.yaml extra.port → API_SERVER_PORT from
  process env or the profile's .env → 8642), with /p/<profile>/ prefix
  routing under multiplex.
- docker/stage2-hook.sh: generate a strong API_SERVER_KEY into .env on
  first boot when absent (never overwrites an operator value), so the
  loopback api_server passes its startup guard on hosted images. The
  fire route itself is NAS-JWT-authed; the key gates the rest of the
  api_server surface. The listener binds 127.0.0.1 by default and the
  Fly service exposes only the dashboard port.
- _fire_cron_job_for_profile kept but deprecated (late-binding seam
  compatibility); no route calls it.
- docs/chronos-managed-cron-contract.md: document the two-hop inbound
  topology and the 503-retry semantics.

Depends on the previous commit (fire webhook passes live adapters to
fire_due) — together they make NAS→dashboard→gateway fires deliver over
relay end to end.

* fix(cron): read the profile api_server port via the canonical config loader

CI guard test_config_read_guard flagged the new _gateway_fire_endpoint
for a raw yaml.safe_load of the profile's config.yaml — the exact drift
class the guard exists to kill (raw reads miss the managed-scope
overlay, ${ENV_VAR} expansion, and root-model normalization).

Read through load_config() under a HERMES_HOME override scoped to the
target profile instead (the same pattern the deprecated
_fire_cron_job_for_profile uses for its store scope), and pull the port
with cfg_get. Test updated to stub load_config rather than write a raw
config.yaml.

* fix(gateway): only messaging platforms count for the scale-to-zero arm gate

The stage2 hook now generates API_SERVER_KEY for every Docker container,
and key presence force-enables the api_server platform. The scale-to-zero
arm gate counted every enabled platform, so the loopback api_server
listener made messaging_is_relay_only_or_absent False on every hosted
instance — silently disarming the feature (the not-armed log would show
enabled platforms=['relay','api_server']).

The arm gate and the not-armed logger now share one helper that filters
to enabled MESSAGING platforms, excluding LOCAL/API_SERVER/WEBHOOK —
the same non-messaging exclusion set _connect_platforms already uses.
A genuinely enabled direct-socket platform (Discord/Telegram) still
disarms. Two of the three new tests fail without this fix.
2026-08-12 17:04:44 +10:00
Ben Barclay 5fffe56066 fix(gateway): exclude permanent supervised watchers from the scale-to-zero busy check (#84327)
_scale_to_zero_has_live_background_work() counted every task in
_background_tasks — but _spawn_supervised parks all permanent watchers
there (session-expiry, kanban, reconnect, the scale-to-zero watcher
itself, ...). An armed gateway therefore considered itself busy forever
and never went dormant or suspended. Verified live on staging
(hermes-agent-stg-test-6698, 2026-08-12): armed at 05:25, fully idle for
25+ minutes, zero 'going dormant' lines. Fly's coarse proxy autostop used
to mask the bug; once the gateway took ownership of the suspend (#84295)
it became load-bearing.

_spawn_supervised now tags its tasks and the busy check skips them.
Transient tasks (startup-resume events, delegation, tracked processes)
still block suspend. New tests exercise the REAL _spawn_supervised path
rather than a stubbed _background_tasks set — the stubbing is exactly why
the earlier tests missed this (same call-site trap as the F25 arm bug);
the key test fails on main and passes with the fix.
2026-08-12 16:39:01 +10:00