4bdd64b334
* fix(auth): close the free tier's gaps against the gateway's welcome-tier contract The inference gateway's welcome tier (NousResearch/api DOCS/anon-tier/plan.md) serves an anonymous account exactly one model on its own host, refuses everything else with a structured 429, cross-refuses a request on the wrong host with a 400 (403 while the tier is dark), and tells a signed-in account that still asks for `nous/welcome` what to switch to in an `x-nous-model-switch` header. Four client-side gaps against that contract: - Auxiliary calls were refused on every session. The auxiliary client asked the welcome host for the Portal's recommended compaction/vision model, a guaranteed 429 `model_not_free` before each fallback. On the welcome host it now uses `nous/welcome` (its backing model covers auxiliary work) and skips Nous for vision, which the welcome model does not take. - The structured 429 body was never read. The classifier now parses `reason` / `retry_after` / `alternates` / `upgrade_url`: `model_not_free` and `feature_not_free` are non-retryable gates that fall back; `at_capacity`, `admission_closed` and `rate_limited` are rate limits that honour `retry_after` and never rotate the free tier's only credential. The wrong-host 400 and the dark-tier 403 are deterministic, so they abort this route and fall back instead of retrying or re-exchanging. The terminal paths say what happened and name the sign-in (`/login` in a chat, `hermes auth upgrade` in a terminal). - The `x-nous-model-switch` header was ignored. The chat-completions transport records it beside the rate-limit and credits headers; the next call moves the session, and the config default when it still names `nous/welcome`, to the backing model the gateway named. - A guest fell back to the paid host. With `inference_base_url` absent from the exchange or outside the host allowlist, routing defaulted to inference-api, where every request is a 400. A guest now defaults to the welcome literal at the exchange, in the shared store's shape, and in effective routing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit fc758aad7efceff6223fc144a9b5c69f13e41bd8) * feat(auth): the free tier is set up on request; nous.guest_setup decides whether also on first use A caller that names nous/welcome on a Nous route with no Nous identity in reach — the guided setup's session (provider=nous, which skips the resolver's nothing-configured rung), the free-tier picker row, a bare --provider nous pointed at it — is asking for the free tier. The OAuth runtime rung now sets it up there instead of failing "not logged in", so the guided chat no longer races the root profile's first-run mint. nous.guest_setup is the policy seam: "auto" (default) keeps today's first-use setup wherever nothing else is configured; "on-request" mints only when the free tier is asked for by name (nous/welcome, /login, hermes auth upgrade, replacing a retired identity). Implicit callers — the resolver's last rung, the first-run check, free_tier.status, the CLI's background setup, the connector token path — still adopt what the shared store holds, so every profile follows the one identity the guided setup created, but never create one on their own. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit ae915ddc65ecdb81b81e29b604671d15cd49233c) (cherry picked from commit 62ad1ff3ab200ea064975a32c502041b25910165) * feat(auth): the guided setup provisions the free tier explicitly; nous.guest_setup is auto | explicit Two questions govern the free tier: may it exist (nous.guest) and who may CREATE the identity (nous.guest_setup). "auto" (default) keeps today's first-use setup wherever nothing else is configured. "explicit" means Hermes never creates one on its own: the only creator is the new provision_free_tier() primitive, exposed as the free_tier.provision RPC, which the guided setup on Hermes Desktop calls as its first step — on the root gateway, before the setup profile and before the guided chat exists — so the identity lands in the root store every profile reads through and is there before any session asks for nous/welcome. That closes the race against the backend's own setup, and makes "only when the setup-bot flow is used" literally true. The earlier "on-request" tier is replaced: it minted whenever any caller named nous/welcome (the hermes model row, --provider nous), which treated a model name as intent and was broader than the guided setup. Under "explicit" a nous/welcome request with no identity fails "not logged in" as before the free tier existed, and /login or hermes auth upgrade report nothing to sign in from. Implicit callers still adopt an identity the shared store holds, and a retired credential is replaced (a continuation, not a creation). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit c63d2c935c1e59016164fdfb90cf70b4094466a0) * fix(auth): remove the nous.guest_setup knob; the free tier is created on first use `nous.guest_setup: auto | explicit` decided who may CREATE the free-tier identity. Under its default every line it added was inert (`may_mint` always true), nothing in tree set `explicit`, unknown values read as `auto`, and under `explicit` a CLI-only install could never get an identity, which contradicts the first-run contract (first command mints, then chats). The mint race the knob accompanied is already benign: every caller takes the profile lock then the shared-store lock, and the loser adopts what the winner wrote. What makes the guided setup win deterministically is `provision_free_tier()` behind the `free_tier.provision` RPC, which stays. `nous.guest` remains the only free-tier policy. Removed: `guest_setup_policy()` and its constants, the `explicit=` / `may_mint=` threading through `ensure_portal_identity` and `_reconcile_and_provision`, the flag at the three replacement call sites (now no-ops), the config default, the docs section, and the four `guest_setup` test-config entries. The three policy tests that hold regardless of the knob are kept under `TestExplicitProvision`; the two that only tested the knob are deleted. (cherry picked from commit d8a50526d93c374c0067dd935b5a65055e0af261) * fix(gateway): a server-driven model switch off nous/welcome does not evict the cached agent When a signed-in account still asks the paid host for `nous/welcome`, the inference gateway serves the current backing model and names it in `x-nous-model-switch`. `apply_model_switch` moves the live session to that model and moves `config.yaml`'s default off the alias in the same step. The messaging gateway's fallback-eviction check compares the agent's model with the config default and evicts on any mismatch that is not a /model override, so when the config write did not land (unreadable config, lock) the cached agent was evicted once per turn, and prompt caching with it. `apply_model_switch` now stamps the alias it moved the session off on the agent, and `_is_intentional_model_switch` treats "agent moved off the alias the config still carries" as deliberate, beside the existing /model override case. The check takes the agent and the config model instead of a bare model string; its one caller in `_run_agent_evict_on_fallback` passes them. (cherry picked from commit 696d1ec86b69db28bf002c841e9389b85178a954) * fix(auth): the free tier outranks implicit host credentials in provider resolution On a fresh install with a leftover ~/.aws profile, resolve_provider("auto") reached the Bedrock rung before the free-tier rung, so the first turn ran on Bedrock and failed 403 while the free tier was still being minted in the background at agent setup (NS-829). Live on a Mac with ~/.aws present: 28 s, three retries, no answer; the next process then switched to nous/welcome. The free-tier rung now sits directly above the Bedrock chain: when nous.guest is on, an existing free-tier identity answers, else a blocking mint runs, and only then does the boto chain get a say. Everything above is unchanged and still wins: CLI creds, config.yaml model.provider, env keys, the OpenRouter pool, a logged-in active_provider. nous.guest: false skips the rung, and a failed mint still falls through to Bedrock and the no-provider guidance. Tests: six precedence cases (identity present, fresh mint, free tier off, env key still wins, sign-in still wins, failed mint falls through). The opt-out test now neutralizes the AWS chain like the precedence tests do; on a machine with ~/.aws it was failing for the same reason as the bug. Live after the fix, same Mac, AWS credentials visible, isolated shared store: identity minted 2 s in, turn on model=nous/welcome provider=nous, answer in 11 s. (cherry picked from commit a04b05260cd334dd7199ad9b6cd5b2538364c75a) * fix(auth): review follow-ups for the free-tier rung (NS-829) - tests/agent/test_bedrock_integration.py: the Bedrock auto-detect test switches the free tier off; its contract is the boto chain, and the free tier now sits above it. - gateway/run_notifications.py: the free-tier startup line reads auth.json before consulting the resolver, so a gateway boot on a machine with AWS credentials never mints or refreshes over the network. - hermes_cli/anon_auth.py: module docstring says where the free tier sits in the ladder instead of "the ladder is untouched". - tests/hermes_cli/test_provider_precedence.py: two invariant tests instead of six (parametrized ladder cases; a failed mint that returns None or raises falls through to Bedrock). scripts/run_tests.sh on the five affected files: 147 passed, 0 failed. (cherry picked from commit 10790d148c60ada11b9ecdde2cd2c836c6a82a11) * feat(auth): HERMES_GUEST_ONBOARDING=1 is the one launch gate for the free tier; HERMES_FORCE_GUEST is gone The free tier is pre-GA. Until GA it must not exist for anyone who did not ask for it: no identity minted, no portal traffic, no free-tier copy on any surface. One environment variable now decides that, and one function reads it. `guest_enabled()` returns False unless `HERMES_GUEST_ONBOARDING` is exactly "1"; only then does `nous.guest` (the user's off switch) get consulted. Every free-tier site already funnels through `guest_enabled()`, so the gate closes minting, routing, connector entitlement, status lines and the picker row in one place. With the variable unset, `resolve_provider("auto")` on a fresh install raises `no_provider_configured` exactly as upstream does. `HERMES_FORCE_GUEST` and `force_guest_mode()` are removed. They inverted the gate (forced the tier ON over `nous.guest: false`), their "new" value re-minted identities as a side effect of provider resolution, and `_has_any_provider_ configured` read them ahead of every other check, making the CLI a second reader of a flag that must have exactly one. `_forced_new_done` and the `force` parameter of `_reconcile_and_provision` go with them. Supersedes the dev lever introduced in fcf9d11679 (rung 1) and hardened in b5c162c3ec. Ruling: NS-845 Q1.1 (recorded on NS-847). Not a user preference: the variable is never written to config.yaml or .env and never shown in setup. It is deleted at GA together with its comment in anon_auth.py. This is a deliberate, temporary exception to the "no new HERMES_* env vars for non-secret config" rule. Tests: fixtures set the gate instead of deleting the old lever; one new invariant (`test_launch_gate_off_means_no_free_tier_at_all`) proves that "", "0", "true" and "new" all leave the tier off with zero portal calls, red on the previous commit. The `HERMES_FORCE_GUEST=new` re-mint test is deleted with the feature. * feat(auth): the free-tier identity is created in one place, at boot; every other site is a read Before this commit eight sites could create a Nous free-tier identity as a side effect of something else: resolving a provider, the CLI's first-run check, the CLI's session setup (in the background beside an own key), a connector bearer read, the desktop polling `free_tier.status`, the sign-in precondition, the desktop's `free_tier.provision`, and the dead-credential re-mint. A poll could mint. Provider resolution could hit the network. Two of them raced each other on a fresh install. Now `hermes_cli/free_tier_bootstrap.py::run_bootstrap` is the only creator. `hermes serve` runs it on a daemon thread from `_lifespan` beside the other background boots; `cmd_chat` runs it synchronously before the first-run guard. It inventories credentials first (`resolve_provider("auto", skip_free_tier=True)`: what would carry inference if the free tier did not exist), creates the identity only when `guest_enabled()`, resolves inference, records a `SetupRecord` in process memory and broadcasts ONE `setup.ready` event. It runs on every boot; only the mint is gated. `ensure_portal_identity` now requires `explicit=True` and raises otherwise. Its callers are the bootstrap, the desktop's `free_tier.provision` (the explicit retry when the boot could not create the identity) and the two dead-credential replacements (`auth_nous.resolve_nous_runtime_credentials`, `managed_tool_gateway._replace_dead_guest_token`). The background thread path and `provision_free_tier` are deleted with their last callers. Reads that used to mint and now only read: `auth.py::resolve_provider` rung 7 (an existing identity still outranks the Bedrock chain, NS-829 ordering kept), `main.py::_has_any_provider_configured`, `cli_agent_setup_mixin._ensure_runtime_credentials`, `managed_tool_gateway.read_nous_access_token` (no identity -> None), `anon_sign_in.run_sign_in` (no identity -> Unavailable), `methods_free_tier` `free_tier.status`. `setup.status` answers from the record for the launch profile, blocking up to 8 s while the bootstrap is in flight so a client's first poll lands after the identity exists rather than racing it; a named profile, or a process that never ran the bootstrap, keeps today's live probe. The record's fields ride along additively (`ready`, `free_tier`, `other_providers`, `inference_provider`). Identity and inference are decoupled (NS-845 Q1.3): the mint sets `active_provider="nous"` only when the inventory found nothing else usable (`_mint_locked(carries_inference=)`); an adopted account always does. A token refresh no longer re-elects the provider it refreshed (`_save_provider_state_to_source` writes credentials, not the user's choice) — that write was how an own-key install ended up on the free tier after the first connector call. Supersedes the mint sites in fcf9d11679, a42d0748fc (first-run check), bbbaa8935a (CLI background setup), 0179efc989 (`free_tier.status` mint), 62ad1ff3ab / c63d2c935c / d8a50526d9 (the `nous.guest_setup` knob and `provision_free_tier`), and a04b05260c (blocking mint in the resolver). Ruling: NS-845 Q1.2 + Q1.3, recorded on NS-847. Tests: `TestBootstrapIsTheOneCreator` (one mint per process; own key keeps inference; reads never reach the portal; a refused mint is memoised), `free_tier.status` fails loudly if it ever calls the creator, the resolver stub fails loudly if resolution ever mints, `setup.status` reads the record, `skip_free_tier` proves the inventory question. The three sign-in tests for the deleted pre-mint collapse into one (`no identity -> Unavailable, zero portal calls`). Live: real `_lifespan` boot with a fake portal, gate on and off (/tmp/ns847-recon/evidence/e2e-rung5-c2-serve-boot.txt), and the CLI matrix incl. an own-key cell (e2e-rung5-c2-bootstrap.txt), 20/20. * fix(credits): the welcome host is free-tier evidence, so a free-tier identity never sees "run /topup" A free-tier identity carries $0 by design, so the portal seed reports `paid_access=False` for it. `is_free_tier_model` did not know the welcome host, read that as a depleted account, and every free-tier turn ended with the credits-depleted notice telling the user to top up an account they do not have. Rule (4) in `is_free_tier_model`: a `base_url` on the Nous welcome host (`anon_auth.route_is_welcome_host`) is the free tier. The host is the evidence, not the model name: the paid inference host can serve `nous/welcome` to a named account and that account's depletion is real, so `("nous/welcome", <inference host>)` stays False. Local data only, like the three rules above it. Restores the two contracts dropped by hermes-magic 674e11d1eaa (the prototype line ran without unit tests): the welcome host is free without any pricing evidence; the model name alone is not. The first is red without this fix. * fix(copy): free-tier text stops promising a connector transfer and never names the config key Sign-in copy on every surface said "Sign in to keep your connectors" and ended with "Your connectors are kept." The transfer registry that would make that true is empty (NS-821): nothing carries over today. The copy now says what signing in does give ("unlock more models and tools") and the completion line names the account, not a transfer. The docs page loses the "connectors carry over" paragraph for the same reason. The picker's off-state line exposed `nous.guest: false` and the word "guest"; user copy names the free tier only (R-USR-1). The docs page gains the pre-rollout note: until GA nothing on it happens without `HERMES_GUEST_ONBOARDING=1`. Its "first command mints" and "replaced on next use" sentences now describe the boot bootstrap. zh is a strict locale: the `freeTier` block was English placeholder text copied from `en`; it is now Chinese. `connectorsKept` is renamed `completedBody` since it no longer talks about connectors. * feat(desktop): the free-tier launch flag is decided once in Electron and stamped onto every backend spawn The Python backend reads `HERMES_GUEST_ONBOARDING` and treats exactly "1" as on. Until now nothing in the desktop set it, so a packaged app could never turn the free tier on, and a backend spawned by the app could disagree with the app about whether the tier was live. `electron/guest-onboarding.ts` owns the decision: `guestOnboardingEnabled` is true when the launch env has `HERMES_GUEST_ONBOARDING=1` or argv has `--guest-onboarding` (the packaged-app spelling). It is read ONCE at launch into a module constant. `desktopBackendSpawnEnv` wraps every backend env as the outermost call and writes the flag LAST, as "1" or an explicit "0", so no earlier spread (`process.env`, `backend.env`) can resurrect a stray value from the parent shell. Stamped onto all three spawn sites: the primary `serve` spawn, the pooled per-profile spawn, and the remote SSH `exec env ...` command (which gains ` HERMES_GUEST_ONBOARDING=1` only when on). The embedded terminal PTY and the backend probes are not backend spawns and do not get it: a `hermes --tui` typed in the pane must not mint. The renderer learns the same fact read-only through the existing `hermes:launch-flags` sync IPC (`guestOnboarding`) and preload (`window.hermesDesktop.guestOnboardingEnabled`). Ruling: NS-845 Q1.1 / Q2 (env var is the contract, `--guest-onboarding` maps to it in main). Two invariant tests on the pure helpers: only "1" or the argv flag enables; the spawn env carries "1"/"0" as the last word and preserves every other key. * feat(desktop): the renderer learns free-tier readiness from one `setup.ready` push, not a 60 s poll The backend's boot bootstrap now announces `setup.ready` once, after it has created (or refused) the free-tier identity and resolved the inference route. The renderer used to discover both by polling `setup.status`, `setup.runtime_check` and `free_tier.status` every 60 s from `useStatusSnapshot`; a fresh install's chip, notice strip and onboarding overlay could sit stale for up to a minute after boot, and three RPCs a minute per window kept asking a question whose answer changes only at boundaries the backend already announces. `handleLifecycleEvent` routes `setup.ready` (active source only, like `skin.changed`) to `notifySetupReady()`, a one-shot tick atom in `live-sync.ts` beside the other change ticks. `useStatusSnapshot` listens to it and runs one readiness round at once (`setup.status` + `setup.runtime_check` + `free_tier.status`). The readiness legs also run once on open and on return from another app, as today. The 60 s tick keeps only `getStatus()`. `SetupStatusSnapshot` types the record's additive fields (`ready`, `free_tier`, `other_providers`, `inference_provider`); readiness semantics are unchanged and still key on `provider_configured` + `runtime_check`. Ruling: NS-845 Q1.2 (renderer half). Tests: the lifecycle branch fires one refresh from the active source and none from another; the snapshot hook's contract is three legs on open, one leg on the tick. * fix(cli): the banner names the free tier's model instead of "no model configured" The welcome banner prints before credentials resolve, so on a fresh install `model` is empty and the banner said, in red, "no model configured — run /model or hermes setup". Under the free tier that is false: the route is already known from local state (identity on disk, tier on), and the first message will run on `nous/welcome`. `_banner_left_lines` now asks the route the same question when `model` is empty (`guest_carries_inference()`, a local read) and shows `welcome · Nous Research`. When nothing resolves the red line stays. Ruling: NS-845 ("the banner's 'no model configured' line reads the resolved route"). Live: fresh HERMES_HOME + fake portal, gate on -> `welcome · Nous Research`; gate off -> the red line, zero portal calls. * fix(aux): vision on the free tier uses nous/welcome too The text-only modality on the gateway's `nous/welcome` row is DeepSeek V4 Flash's, the backing model until the repoint; `z-ai/glm-5.3-flash` is natively multimodal and the repoint declares the welcome row `text+image->text`. Skipping Nous for vision on the welcome host would have sent every image step past the free tier for no reason, so the auxiliary client pins the route's one model for every lane. A backing model that takes no images answers with the upstream's own error, which the ladder handles as it always has. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 7456e028faba55480db43015dc2c8df3e393a415) * fix(gateway): hermes gateway run is a boot owner of the free tier too Rung 5 made every demand-time free-tier site a read: resolve_provider, the connector token, the /login precondition. That is only correct if every process that can reach those sites ran the bootstrap first. The CLI (cmd_chat) and hermes serve (_lifespan) did; the standalone messaging gateway did not. A fresh HERMES_HOME with the gate on and `hermes gateway run` reached provider resolution with no identity to consume, and /login returned Unavailable. Reported by @andrexibiza on #107697 (P1). GatewayRunner.start now runs `free_tier_bootstrap.run_bootstrap` on an executor thread right after startup recovery and BEFORE any adapter connects, so a fast first DM cannot arrive with nothing to resolve. It is its own step, not part of the turn-machinery warm-up: the warm-up is an optimisation with an off switch (HERMES_STARTUP_WARMUP_TIMEOUT<=0); the bootstrap is correctness and must always run. With the gate unset it is a local inventory and no network. Live, real GatewayRunner.start against a fake portal in a fresh home: gate on -> 1 create, identity persisted, resolve_runtime_provider=nous, /login precondition sees the identity gate off -> 0 portal calls, no identity, no_provider_configured Before the fix the gate-on row was identical to the gate-off row. Test: the bootstrap seam runs before _start_prefilter_platforms and delegates to the one creator. Red on 5554eb6993 (no seam), green here. --------- Co-authored-by: Robin Fernandes <robin@soal.org> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
3893 lines
221 KiB
Python
3893 lines
221 KiB
Python
"""Agent-turn execution for GatewayRunner: _handle_message_with_agent, _run_agent*, proxy path,
|
||
background tasks, MCP reload. Bound onto ``GatewayRunner`` via the MRO; ``gateway.run`` internals
|
||
are imported lazily inside method bodies (import cycle) so ``patch("gateway.run.X")`` still works.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import logging
|
||
from typing import TYPE_CHECKING
|
||
import asyncio
|
||
import dataclasses
|
||
import inspect
|
||
import json
|
||
import os
|
||
import queue
|
||
import threading
|
||
import time
|
||
from agent.i18n import t
|
||
from agent.session_activity import format_iteration_progress
|
||
from contextlib import nullcontext, suppress
|
||
from contextvars import copy_context
|
||
from gateway.config import Platform
|
||
from gateway.media_repair import repair_explicit_computer_use_media_paths
|
||
from gateway.platforms.base import BasePlatformAdapter
|
||
from gateway.platforms.event import MessageEvent
|
||
from gateway.session import (
|
||
SessionSource, _session_key_namespace, build_channel_continuity_note,
|
||
build_session_context,
|
||
)
|
||
from gateway.session_transcript import TranscriptReadError
|
||
from gateway.turn_context import TurnContext
|
||
from gateway.turn_lease import DEFAULT_LEASE_WAIT, TurnLeaseTimeoutError
|
||
from hermes_constants import get_hermes_home_override
|
||
from pathlib import Path
|
||
from typing import Any, Callable, Dict, List, Optional, Tuple
|
||
from utils import base_url_hostname
|
||
|
||
if TYPE_CHECKING: # string annotations only; never imported at runtime (cycle)
|
||
from gateway.run import GatewayRunner # noqa: F401
|
||
from gateway.run_turn_runner import TurnRunner # noqa: F401
|
||
|
||
# Log-record parity with the origin module.
|
||
logger = logging.getLogger("gateway.run")
|
||
|
||
|
||
_CONTEXT_OVERFLOW_ERROR_PHRASES = (
|
||
"context length", "context size", "context window",
|
||
"maximum context", "token limit", "too many tokens",
|
||
"reduce the length", "exceeds the limit",
|
||
"request entity too large", "prompt is too long",
|
||
"payload too large", "input is too long",
|
||
)
|
||
|
||
|
||
def is_context_overflow_failure_result(agent_result: dict, history_len: int) -> bool:
|
||
"""One verdict for "this failed turn is a context overflow", shared by transcript persistence
|
||
(#1630 skip) and the user-facing reply so the two can never disagree.
|
||
|
||
Multi-word phrases (not bare "exceed"/"token") avoid matching "rate limit exceeded" or
|
||
"invalid authentication token"; a bare 400 only counts on a long session."""
|
||
if not agent_result.get("failed"):
|
||
return False
|
||
if agent_result.get("compression_exhausted"):
|
||
return True
|
||
err = str(agent_result.get("error") or "").lower()
|
||
return any(p in err for p in _CONTEXT_OVERFLOW_ERROR_PHRASES) or ("400" in err and history_len > 50)
|
||
|
||
|
||
class GatewayTurnMixin:
|
||
"""Agent-turn execution for GatewayRunner (see module docstring)."""
|
||
|
||
def _resolve_session_agent_runtime(
|
||
self, *, source: Optional[SessionSource] = None, session_key: Optional[str] = None,
|
||
user_config: Optional[dict] = None,
|
||
) -> tuple[str, dict]:
|
||
"""Resolve model/runtime for a session.
|
||
|
||
Priority (highest first): session ``/model`` → ``channel_overrides`` → global config/env
|
||
(``_resolve_gateway_model(user_config)`` and default provider resolution)."""
|
||
from gateway.run import (
|
||
_credential_pool_for_provider, _get_channel_override, _resolve_gateway_model,
|
||
_resolve_runtime_agent_kwargs, _resolve_runtime_agent_kwargs_for_provider,
|
||
)
|
||
skey = self._resolve_session_key_or_none(source, session_key)
|
||
|
||
model = _resolve_gateway_model(user_config)
|
||
if skey:
|
||
self._rehydrate_session_model_override(skey)
|
||
_override_state = self._peek_session_state(skey) if skey else None
|
||
override = _override_state.conversation.model_override if _override_state else None
|
||
if override:
|
||
override_model = override.get("model", model)
|
||
override_runtime = {
|
||
k: override.get(k) for k in (
|
||
"provider", "requested_provider", "api_key", "base_url", "api_mode",
|
||
"max_tokens", "credential_pool", "request_overrides", "capabilities",
|
||
)
|
||
}
|
||
override_runtime["capabilities"] = dict(override_runtime["capabilities"] or {})
|
||
if override_runtime.get("api_key"):
|
||
if override_runtime.get("credential_pool") is None:
|
||
override_runtime["credential_pool"] = _credential_pool_for_provider(override.get("provider"))
|
||
logger.debug(
|
||
"Session model override (fast): session=%s config_model=%s -> override_model=%s provider=%s",
|
||
skey or "", model, override_model, override_runtime.get("provider"),
|
||
)
|
||
return override_model, override_runtime
|
||
# No api_key on the override: env-based resolution below, override model/provider on top.
|
||
logger.debug(
|
||
"Session model override (no api_key, fallback): session=%s config_model=%s override_model=%s",
|
||
skey or "", model, override_model,
|
||
)
|
||
else:
|
||
logger.debug(
|
||
"No session model override: session=%s config_model=%s override_keys=%s",
|
||
skey or "", model,
|
||
[
|
||
_key for _key, _st in list(self._sessions_map().items())
|
||
if _st.conversation.model_override is not None
|
||
][:5] or "[]",
|
||
)
|
||
|
||
runtime_kwargs = _resolve_runtime_agent_kwargs()
|
||
runtime_model = runtime_kwargs.pop("model", None)
|
||
if runtime_model:
|
||
logger.info("Runtime provider supplied explicit model override: %s -> %s", model, runtime_model)
|
||
model = runtime_model
|
||
|
||
cfg = getattr(self, "config", None) # getattr: bare object.__new__ test runners
|
||
if cfg and source is not None:
|
||
ch = _get_channel_override(
|
||
cfg, source.platform, str(source.chat_id) if source.chat_id else "",
|
||
thread_id=str(source.thread_id) if getattr(source, "thread_id", None) else None,
|
||
parent_id=str(source.parent_chat_id) if getattr(source, "parent_chat_id", None) else None,
|
||
)
|
||
if ch:
|
||
if ch.model:
|
||
model = ch.model
|
||
if ch.provider:
|
||
runtime_kwargs = _resolve_runtime_agent_kwargs_for_provider(ch.provider)
|
||
ch_runtime_model = runtime_kwargs.pop("model", None)
|
||
# Adopt the provider's bundled model only when the override named none.
|
||
if ch_runtime_model and not ch.model:
|
||
model = ch_runtime_model
|
||
|
||
if override and skey:
|
||
model, runtime_kwargs = self._apply_session_model_override(skey, model, runtime_kwargs)
|
||
|
||
# Provider resolved but no model.default (`hermes auth add` without `hermes model`): use the
|
||
# provider's first catalog model.
|
||
if not model and runtime_kwargs.get("provider"):
|
||
with suppress(Exception):
|
||
from hermes_cli.models import get_default_model_for_provider
|
||
model = get_default_model_for_provider(runtime_kwargs["provider"])
|
||
if model:
|
||
logger.info(
|
||
"No model configured — defaulting to %s for provider %s", model, runtime_kwargs["provider"],
|
||
)
|
||
|
||
# Final safety net: an empty model (transient config-cache miss) makes every API call 400 and
|
||
# the session goes silent — reuse the last model resolved for this session, else process-wide.
|
||
if not model:
|
||
_lr_state = self._peek_session_state(skey) if skey else None
|
||
_lr_star = self._peek_session_state("*")
|
||
_recovered = (
|
||
(_lr_state.conversation.last_resolved_model if _lr_state else "")
|
||
or (_lr_star.conversation.last_resolved_model if _lr_star else "")
|
||
)
|
||
if _recovered:
|
||
logger.warning(
|
||
"Empty model resolved for session=%s — recovering "
|
||
"last-known-good model %s (config read likely returned "
|
||
"empty; see #35314)", skey or "", _recovered,
|
||
)
|
||
model = _recovered
|
||
else:
|
||
# Cache the good resolution for future recovery turns.
|
||
if skey:
|
||
self._session_state(skey).conversation.last_resolved_model = model
|
||
self._session_state("*").conversation.last_resolved_model = model
|
||
|
||
return model, runtime_kwargs
|
||
|
||
def _resolve_turn_agent_config(self, user_message: str, model: str, runtime_kwargs: dict) -> dict:
|
||
"""Effective model/runtime config for one turn. With `/fast` priority on, fast-mode
|
||
``request_overrides`` are deep-merged OVER the per-provider ones so both reach the model."""
|
||
from gateway.run import _deep_merge_request_overrides
|
||
from hermes_cli.models import resolve_fast_mode_overrides
|
||
# Tests bind this method onto bare namespaces, so no class-level tables here.
|
||
runtime = {
|
||
k: runtime_kwargs.get(k) for k in (
|
||
"api_key", "base_url", "provider", "requested_provider", "api_mode", "command", "args",
|
||
"credential_pool", "max_tokens", "capabilities",
|
||
)
|
||
}
|
||
runtime["args"] = list(runtime["args"] or [])
|
||
runtime["capabilities"] = dict(runtime["capabilities"] or {})
|
||
base_request_overrides = dict(runtime_kwargs.get("request_overrides") or {})
|
||
route = {
|
||
"model": model,
|
||
"runtime": runtime,
|
||
"signature": (
|
||
model, runtime["provider"], runtime["requested_provider"], runtime["base_url"],
|
||
runtime["api_mode"], runtime["command"], tuple(runtime["args"]),
|
||
),
|
||
}
|
||
if getattr(self, "_service_tier", None) != "priority":
|
||
# None / auto / cold: the bounded window is applied per request by agent.fast_mode.
|
||
route["request_overrides"] = base_request_overrides
|
||
return route
|
||
try:
|
||
overrides = resolve_fast_mode_overrides(
|
||
route["model"], provider=runtime["provider"], base_url=runtime["base_url"],
|
||
)
|
||
except Exception:
|
||
overrides = None
|
||
# Fast-mode keys (service_tier / speed) are top-level and don't collide with extra_body.
|
||
route["request_overrides"] = _deep_merge_request_overrides(base_request_overrides, overrides or {})
|
||
return route
|
||
|
||
def _sync_session_model_from_agent(self, session_id: str, agent: Any) -> None:
|
||
"""Persist the runtime model/provider a gateway turn actually used (provider fallback can
|
||
switch them after the row was created). Runs in the ``run_sync`` executor thread, so it
|
||
uses the sync ``SessionDB`` (``_db``), not the AsyncSessionDB forwarder."""
|
||
if not session_id or agent is None or self._session_db is None:
|
||
return
|
||
model = getattr(agent, "model", None)
|
||
if not model:
|
||
return
|
||
runtime = {k: getattr(agent, k, None) for k in ("provider", "base_url", "api_mode")}
|
||
runtime["fallback_active"] = bool(getattr(agent, "_fallback_activated", False))
|
||
runtime = {k: v for k, v in runtime.items() if v not in (None, "")}
|
||
try:
|
||
db = self._session_db._db
|
||
row = db.get_session(session_id)
|
||
if not row:
|
||
return
|
||
# Legacy backfill: canonical Bot Chats created BEFORE the follow_profile_config contract existed
|
||
# carry no marker, yet they are still the plugin-owned forever-DM. The plugin's own identity
|
||
# rule is "the profile's session titled exactly 'Bot Chat'" (UNIQUE(title) makes that an exact
|
||
# registry, and pre-policy rows may be visible OR hidden), so mirror that rule here. Without
|
||
# this, every Bot Chat that already exists in the field stays pinned to its stale stored
|
||
# provider until the user deletes it — the exact live-report shape (#89497 / #94818).
|
||
raw_config = row.get("model_config")
|
||
config = {}
|
||
with suppress(Exception):
|
||
config = json.loads(raw_config) if raw_config else {}
|
||
if not isinstance(config, dict):
|
||
config = {}
|
||
gateway_runtime = dict(config.get("gateway_runtime") or {})
|
||
if row.get("model") == model and all(gateway_runtime.get(k) == v for k, v in runtime.items()):
|
||
return
|
||
config["gateway_runtime"] = runtime
|
||
db.update_session_meta(session_id, json.dumps(config), model=model)
|
||
except Exception:
|
||
logger.debug("Failed to sync gateway session model metadata", exc_info=True)
|
||
|
||
def _event_thread_metadata(self, event, source):
|
||
"""Thread metadata for a send that replies to ``event`` on ``source``."""
|
||
return self._thread_metadata_for_source(source, self._reply_anchor_for_event(event))
|
||
|
||
@staticmethod
|
||
def _pop_post_delivery_callback(adapter, key, generation):
|
||
"""Pop the adapter's deferred post-delivery callback for ``key`` (legacy dict fallback)."""
|
||
if getattr(type(adapter), "pop_post_delivery_callback", None) is not None:
|
||
return adapter.pop_post_delivery_callback(key, generation=generation)
|
||
if adapter and hasattr(adapter, "_post_delivery_callbacks"):
|
||
return adapter._post_delivery_callbacks.pop(key, None)
|
||
return None
|
||
|
||
@staticmethod
|
||
def _is_intentional_silence(agent_result, response) -> bool:
|
||
try:
|
||
from gateway.response_filters import is_intentional_silence_agent_result
|
||
return is_intentional_silence_agent_result(agent_result, response)
|
||
except Exception:
|
||
return False
|
||
|
||
async def _hmwa_resolve_session(self, event, source):
|
||
"""Resolve ``source`` to its session entry (topic recovery, internal-route guards, Telegram
|
||
topic-binding heal). Returns ``(source, session_entry, session_key)`` or ``None`` to drop
|
||
the event."""
|
||
# Topic-mode DMs: rewrite a stale/foreign thread_id to the user's last-active topic so a
|
||
# cross-topic Reply doesn't fragment the conversation.
|
||
event_metadata = getattr(event, "metadata", None) or {}
|
||
expected_session_key = str(event_metadata.get("gateway_session_key") or "").strip()
|
||
recovered = (await asyncio.to_thread(self._recover_telegram_topic_thread_id, source)
|
||
if not expected_session_key else None)
|
||
if recovered is not None:
|
||
logger.info(
|
||
"telegram topic recovery: chat=%s user=%s %r -> %s",
|
||
source.chat_id, source.user_id, source.thread_id, recovered,
|
||
)
|
||
source = dataclasses.replace(source, thread_id=recovered)
|
||
with suppress(Exception):
|
||
event.source = source
|
||
|
||
if expected_session_key:
|
||
derived_session_key = self._session_key_for_source(source)
|
||
if derived_session_key != expected_session_key:
|
||
logger.warning(
|
||
"Dropping internally routed event after route recovery: expected session=%s derived=%s",
|
||
expected_session_key, derived_session_key,
|
||
)
|
||
return
|
||
|
||
strict_session = bool(event_metadata.get("gateway_session_strict"))
|
||
pinned_session_id = str(event_metadata.get("gateway_session_id") or "").strip()
|
||
if strict_session:
|
||
session_entry = await self.async_session_store.lookup_by_session_key(expected_session_key)
|
||
if session_entry is None or not pinned_session_id or session_entry.session_id != pinned_session_id:
|
||
logger.warning(
|
||
"Dropping internally routed event: expected session id=%s is no longer current for key=%s",
|
||
pinned_session_id or "missing", expected_session_key or "missing",
|
||
)
|
||
return
|
||
else:
|
||
# Internal wakes observe reset policy without counting as user activity, or periodic
|
||
# notifications keep the routing key alive across every daily/idle boundary.
|
||
session_entry = await self.async_session_store.get_or_create_session(
|
||
source, touch_activity=not bool(getattr(event, "internal", False)),
|
||
)
|
||
session_key = session_entry.session_key
|
||
if not strict_session and pinned_session_id:
|
||
resolved_entry = await self._resolve_async_delegation_session(session_entry, pinned_session_id)
|
||
if resolved_entry is None:
|
||
return
|
||
session_entry = resolved_entry
|
||
self._cache_session_source(session_key, source)
|
||
if await asyncio.to_thread(self._is_telegram_topic_lane, source):
|
||
session_entry = await self._hmwa_heal_telegram_topic_binding(source, session_entry, session_key)
|
||
from gateway.run_heartbeat_acceptance import resolve_heartbeat_owner
|
||
if not await resolve_heartbeat_owner(self, event, session_entry):
|
||
return
|
||
return source, session_entry, session_key
|
||
|
||
async def _hmwa_heal_telegram_topic_binding(self, source, session_entry, session_key):
|
||
"""Follow the (chat_id, thread_id) topic binding — healed to its compression tip — or record
|
||
a fresh one. Returns the (possibly switched) session entry."""
|
||
binding = None
|
||
try:
|
||
if self._session_db:
|
||
binding = await self._session_db.get_telegram_topic_binding(
|
||
chat_id=str(source.chat_id), thread_id=str(source.thread_id),
|
||
profile_name=self._telegram_topic_profile_name(source),
|
||
)
|
||
except Exception:
|
||
logger.debug("Failed to read Telegram topic binding", exc_info=True)
|
||
if not binding:
|
||
try:
|
||
await asyncio.to_thread(self._record_telegram_topic_binding, source, session_entry)
|
||
except Exception:
|
||
logger.debug("Failed to record Telegram topic binding", exc_info=True)
|
||
return session_entry
|
||
stored_session_id = str(binding.get("session_id") or "")
|
||
bound_session_id = stored_session_id
|
||
# A binding pointing at a pre-compression parent is walked forward to the tip so the next
|
||
# message resumes the compressed child instead of reloading the oversized parent.
|
||
# Returns the input unchanged when the session isn't a compression parent, so this is cheap and
|
||
# safe. See #20470, #29712, #33414.
|
||
if bound_session_id and self._session_db is not None:
|
||
try:
|
||
canonical_session_id = await self._session_db.get_compression_tip(bound_session_id)
|
||
except Exception:
|
||
logger.debug("compression-tip lookup failed for %s", bound_session_id, exc_info=True)
|
||
canonical_session_id = bound_session_id
|
||
if canonical_session_id and canonical_session_id != bound_session_id:
|
||
bound_session_id = canonical_session_id
|
||
if bound_session_id and bound_session_id != session_entry.session_id:
|
||
# Route through SessionStore so the key → id mapping persists and the previous lane
|
||
# session ends cleanly (in-place mutation split-brained the JSON index).
|
||
switched = await self.async_session_store.switch_session(session_key, bound_session_id)
|
||
if switched is not None:
|
||
session_entry = switched
|
||
if bound_session_id and bound_session_id != stored_session_id:
|
||
# The stored binding pointed at a parent: rewrite it to the canonical descendant.
|
||
await asyncio.to_thread(
|
||
self._sync_telegram_topic_binding, source, session_entry, reason="compression-tip-walk",
|
||
)
|
||
return session_entry
|
||
|
||
async def _hmwa_open_session(self, session_entry, session_key, source):
|
||
"""Consume auto-reset / fresh-reset flags and emit ``session:start`` for new sessions.
|
||
Returns ``(_was_auto_reset, _is_new_session)``."""
|
||
# Consume was_auto_reset immediately so it cannot re-fire and wipe overrides set between turns.
|
||
# Capture and immediately consume was_auto_reset so it does not re-fire on subsequent messages —
|
||
# preventing the cleanup from wiping model/reasoning overrides set between turns (Closes #48031).
|
||
_was_auto_reset = getattr(session_entry, "was_auto_reset", False)
|
||
if _was_auto_reset:
|
||
# Conversation boundary: the funnel clears every conversation-scoped dict; evict the cached
|
||
# agent so context_compressor._previous_summary cannot leak into new summaries.
|
||
# Treat auto-reset as a full conversation boundary — clear every conversation-scoped per-session
|
||
# dict in one funnel call so the fresh session does not inherit the previous conversation's
|
||
# model/reasoning overrides, a queued "/model switched" note, or a stale resolved-model cache
|
||
# (#48031, #58403). See _CONVERSATION_SCOPED_STATE.
|
||
self._clear_conversation_scope(session_key, reason="auto_reset")
|
||
self._evict_cached_agent(session_key)
|
||
session_entry.was_auto_reset = False
|
||
|
||
_is_fresh_reset = getattr(session_entry, "is_fresh_reset", False)
|
||
_is_new_session = session_entry.created_at == session_entry.updated_at or _was_auto_reset or _is_fresh_reset
|
||
# Consume is_fresh_reset so it doesn't leak onto later messages in the same session.
|
||
if _is_fresh_reset:
|
||
# See #6508.
|
||
session_entry.is_fresh_reset = False
|
||
if _is_new_session:
|
||
await self.hooks.emit("session:start", {
|
||
"platform": source.platform.value if source.platform else "",
|
||
"user_id": source.user_id,
|
||
"session_id": session_entry.session_id,
|
||
"session_key": session_key,
|
||
})
|
||
return _was_auto_reset, _is_new_session
|
||
|
||
async def _hmwa_deliver_auto_reset_notice(self, session_entry, source, turn_sidecar_notes):
|
||
"""Stage the auto-reset sidecar note for the agent and notify the user (policy-gated)."""
|
||
from gateway.run import _AUTO_RESET_CONTEXT_NOTES
|
||
reset_reason = getattr(session_entry, 'auto_reset_reason', None) or 'suspended'
|
||
context_note = _AUTO_RESET_CONTEXT_NOTES.get(reset_reason, _AUTO_RESET_CONTEXT_NOTES["suspended"])
|
||
# Long-lived channels: point the agent at the prior same-channel session for session_search.
|
||
try:
|
||
# Returns None (appends nothing) for other platforms or when there's no prior activity to
|
||
# recall. Deterministic — no extra API/DB calls (#36220).
|
||
continuity_note = build_channel_continuity_note(session_entry, source)
|
||
except Exception:
|
||
continuity_note = None
|
||
if continuity_note:
|
||
context_note = context_note + "\n\n" + continuity_note
|
||
turn_sidecar_notes.append(context_note)
|
||
|
||
try:
|
||
should_notify = reset_reason == "suspended"
|
||
adapter = self._adapter_for_source(source) if should_notify else None
|
||
if adapter:
|
||
notice = (
|
||
"◐ Session reset after being stopped. "
|
||
f"Conversation history cleared.\n"
|
||
f"Use /resume to browse and restore a previous session.\n"
|
||
)
|
||
with suppress(Exception):
|
||
session_info = await asyncio.to_thread(self._reset_notice_session_info, source)
|
||
if session_info:
|
||
notice = f"{notice}\n\n{session_info}"
|
||
await adapter.send(source.chat_id, notice, metadata=self._thread_metadata_for_source(source))
|
||
except Exception as e:
|
||
logger.debug("Auto-reset notification failed (non-fatal): %s", e)
|
||
|
||
# was_auto_reset was consumed in _hmwa_open_session; only the reason needs clearing.
|
||
session_entry.auto_reset_reason = None
|
||
|
||
def _hmwa_auto_load_skills(self, event, _auto, _quick_key, session_key):
|
||
"""Prepend topic/channel-bound skill payload(s) to ``event.text`` on a new session."""
|
||
_skill_names = [_auto] if isinstance(_auto, str) else list(_auto)
|
||
try:
|
||
from agent.skill_commands import _load_skill_payload, _build_skill_message
|
||
_combined_parts: list[str] = []
|
||
_loaded_names: list[str] = []
|
||
for _sname in _skill_names:
|
||
_loaded = _load_skill_payload(_sname, task_id=_quick_key)
|
||
if not _loaded:
|
||
logger.warning("[Gateway] Auto-skill '%s' not found", _sname)
|
||
continue
|
||
_loaded_skill, _skill_dir, _display_name = _loaded
|
||
_part = _build_skill_message(
|
||
_loaded_skill, _skill_dir,
|
||
f'[IMPORTANT: The "{_display_name}" skill is auto-loaded. '
|
||
f"Follow its instructions for this session.]",
|
||
)
|
||
if _part:
|
||
_combined_parts.append(_part)
|
||
_loaded_names.append(_sname)
|
||
if _combined_parts:
|
||
_combined_parts.append(event.text) # user's original text after the payloads
|
||
event.text = "\n\n".join(_combined_parts)
|
||
logger.info("[Gateway] Auto-loaded skill(s) %s for session %s", _loaded_names, session_key)
|
||
except Exception as e:
|
||
logger.warning("[Gateway] Failed to auto-load skill(s) %s: %s", _skill_names, e)
|
||
|
||
async def _hmwa_acquire_turn_lease(self, _quick_key, run_generation, session_entry, _session_env_tokens):
|
||
"""Serialize [load history → run → flush] per resolved SESSION_ID so another routing key on
|
||
the same session waits for the prior flush. Fail-closed on timeout (outer dispatch returns
|
||
a resend notice). Released in _handle_message's finally, granted per (routing key, run
|
||
generation) so a stale unwind can't release a newer turn's."""
|
||
from gateway.run import _float_env
|
||
_lease_registry = getattr(self, "_turn_leases", None)
|
||
if _lease_registry is None:
|
||
return
|
||
try:
|
||
_lease_token = await _lease_registry.acquire(
|
||
session_entry.session_id, owner_key=_quick_key, generation=run_generation,
|
||
timeout=_float_env("HERMES_TURN_LEASE_TIMEOUT", DEFAULT_LEASE_WAIT),
|
||
)
|
||
except TurnLeaseTimeoutError:
|
||
# The cleanup finally starts later; restore the tokens here or this exit leaks identity.
|
||
self._clear_session_env(_session_env_tokens)
|
||
raise
|
||
if _lease_token is not None:
|
||
_lease_state = self._session_state(_quick_key).turn
|
||
_lease_state.lease_token = _lease_token
|
||
_lease_state.lease_generation = run_generation
|
||
|
||
@dataclasses.dataclass
|
||
class _HygienePlan:
|
||
"""Hygiene pre-check outcome for one turn."""
|
||
|
||
needs_compress: bool
|
||
approx_tokens: int
|
||
msg_count: int
|
||
warn_token_threshold: int
|
||
|
||
@staticmethod
|
||
def _hmwa_hygiene_read_config(hs, data):
|
||
"""Apply model / compression knobs from the gateway config onto ``hs`` (invalid values keep the defaults)."""
|
||
# Resolve model name (same logic as run_sync)
|
||
_model_cfg = data.get("model", {})
|
||
if isinstance(_model_cfg, str):
|
||
hs.model = _model_cfg
|
||
elif isinstance(_model_cfg, dict):
|
||
hs.model = _model_cfg.get("default") or _model_cfg.get("model") or hs.model
|
||
_raw_ctx = _model_cfg.get("context_length")
|
||
if _raw_ctx is not None:
|
||
with suppress(TypeError, ValueError):
|
||
hs.config_context_length = int(_raw_ctx)
|
||
hs.provider = _model_cfg.get("provider") or None
|
||
hs.base_url = _model_cfg.get("base_url") or None
|
||
|
||
# Only the enabled flag is shared with the agent's compression config (hygiene runs higher).
|
||
_comp_cfg = data.get("compression", {})
|
||
if not isinstance(_comp_cfg, dict):
|
||
return
|
||
hs.compression_enabled = str(_comp_cfg.get("enabled", True)).lower() in {"true", "1", "yes"}
|
||
|
||
def _knob(key, current, cast, allow_zero=False):
|
||
raw = _comp_cfg.get(key)
|
||
if raw is None:
|
||
return current
|
||
try:
|
||
parsed = cast(raw)
|
||
except (TypeError, ValueError):
|
||
return current
|
||
return parsed if (parsed >= 0 if allow_zero else parsed > 0) else current
|
||
|
||
hs.hard_msg_limit = _knob("hygiene_hard_message_limit", hs.hard_msg_limit, int)
|
||
hs.timeout_seconds = _knob("hygiene_timeout_seconds", hs.timeout_seconds, float)
|
||
hs.total_ceiling_seconds = _knob("hygiene_total_ceiling_seconds", hs.total_ceiling_seconds, float)
|
||
# The ceiling can never be tighter than one idle window, or the extension loop would be dead code.
|
||
hs.total_ceiling_seconds = max(hs.total_ceiling_seconds, hs.timeout_seconds)
|
||
hs.max_turn_hold_seconds = _knob("hygiene_max_turn_hold_seconds", hs.max_turn_hold_seconds, float)
|
||
hs.failure_cooldown_seconds = _knob(
|
||
"hygiene_failure_cooldown_seconds", hs.failure_cooldown_seconds, float, allow_zero=True,
|
||
)
|
||
|
||
async def _hmwa_hygiene_settings(self, source, session_key):
|
||
"""Resolve model/provider/context-length + hygiene knobs (fail-soft: errors keep defaults).
|
||
|
||
The 0.85 threshold is deliberately HIGHER than the agent's compressor (0.50): a safety net
|
||
for sessions that grew between turns. ``max_turn_hold_seconds`` bounds the TURN wait
|
||
(compressor keeps running detached, commit fenced); kept below transport idle-timeouts."""
|
||
from gateway.run import _load_gateway_config
|
||
hs = self._HygieneSettings(
|
||
model="anthropic/claude-sonnet-4.6", threshold_pct=0.85, compression_enabled=True,
|
||
hard_msg_limit=5000, timeout_seconds=30.0, total_ceiling_seconds=600.0,
|
||
max_turn_hold_seconds=10.0, failure_cooldown_seconds=300.0, config_context_length=None,
|
||
provider=None, base_url=None, api_key=None, data={},
|
||
)
|
||
try:
|
||
hs.data = _load_gateway_config()
|
||
if hs.data:
|
||
self._hmwa_hygiene_read_config(hs, hs.data)
|
||
configured_model, configured_provider, configured_base_url = hs.model, hs.provider, hs.base_url
|
||
|
||
with suppress(Exception):
|
||
hs.model, _hyg_runtime = self._resolve_session_agent_runtime(
|
||
source=source, session_key=session_key,
|
||
user_config=hs.data if isinstance(hs.data, dict) else None,
|
||
)
|
||
hs.provider = _hyg_runtime.get("provider") or hs.provider
|
||
hs.base_url = _hyg_runtime.get("base_url") or hs.base_url
|
||
hs.api_key = _hyg_runtime.get("api_key") or hs.api_key
|
||
|
||
if hs.config_context_length is not None:
|
||
try:
|
||
from hermes_cli.route_identity import should_clear_context_pin_async
|
||
|
||
if await should_clear_context_pin_async(
|
||
configured_model, hs.model, configured_base_url, hs.base_url,
|
||
configured_provider, hs.provider,
|
||
):
|
||
hs.config_context_length = None
|
||
except Exception:
|
||
hs.config_context_length = None
|
||
|
||
# custom_providers per-model context_length fallback (as in run_agent.py); needs base_url.
|
||
if hs.config_context_length is None and hs.base_url:
|
||
with suppress(TypeError, ValueError):
|
||
try:
|
||
from hermes_cli.config import (
|
||
get_compatible_custom_providers as _gw_gcp,
|
||
get_custom_provider_context_length as _gw_gccl,
|
||
)
|
||
_hyg_custom_providers = _gw_gcp(hs.data)
|
||
except Exception:
|
||
_hyg_custom_providers = hs.data.get("custom_providers")
|
||
if not isinstance(_hyg_custom_providers, list):
|
||
_hyg_custom_providers = []
|
||
_hyg_custom_ctx = _gw_gccl(
|
||
model=hs.model, base_url=hs.base_url, custom_providers=_hyg_custom_providers,
|
||
)
|
||
if _hyg_custom_ctx:
|
||
hs.config_context_length = int(_hyg_custom_ctx)
|
||
except Exception:
|
||
pass
|
||
return hs
|
||
|
||
async def _hmwa_hygiene_plan(self, hs, history, session_entry, session_key):
|
||
"""Decide whether hygiene compression fires this turn (token/message thresholds, DB-backed
|
||
failure cooldown, in-flight compression)."""
|
||
from agent.model_metadata import estimate_messages_tokens_rough, get_model_context_length_async
|
||
_hyg_context_length = await get_model_context_length_async(
|
||
hs.model, base_url=hs.base_url or "", api_key=hs.api_key or "",
|
||
config_context_length=hs.config_context_length, provider=hs.provider or "",
|
||
)
|
||
_compress_token_threshold = int(_hyg_context_length * hs.threshold_pct)
|
||
_warn_token_threshold = int(_hyg_context_length * 0.95)
|
||
_msg_count = len(history)
|
||
|
||
# Real usage decides: the API-reported prompt count, else the anchor persisted on the session
|
||
# row (real count + delta of what was appended since, survives gateway restarts), else the
|
||
# rough estimate (runs 30-50% high, which only fires hygiene early — safe). Do NOT compensate
|
||
# with a threshold multiplier.
|
||
from agent.image_token_cost import image_cost_context, learned_image_token_cost
|
||
_anchored = None
|
||
# Images in any local delta/estimate are priced at the cost learned from this model's usage.
|
||
with image_cost_context(learned_image_token_cost(hs.model, hs.base_url)):
|
||
if session_entry.last_prompt_tokens <= 0:
|
||
from agent.usage_anchor import persisted_anchor_tokens
|
||
_session_db = getattr(self, "_session_db", None)
|
||
_anchored = persisted_anchor_tokens(
|
||
getattr(_session_db, "_db", _session_db), session_entry.session_id, history,
|
||
)
|
||
if session_entry.last_prompt_tokens > 0:
|
||
_approx_tokens, _token_source = session_entry.last_prompt_tokens, "actual"
|
||
elif _anchored is not None:
|
||
_approx_tokens, _token_source = _anchored, "anchored"
|
||
else:
|
||
_approx_tokens, _token_source = estimate_messages_tokens_rough(history), "estimated"
|
||
|
||
# Hard safety valve: force compression at an extreme message count regardless of tokens,
|
||
# breaking the disconnect → no token data → no compression spiral. 5000 clears 1M+ sessions.
|
||
_needs_compress = _approx_tokens >= _compress_token_threshold or _msg_count >= hs.hard_msg_limit
|
||
|
||
if _needs_compress:
|
||
# DB-backed cooldown (shared with context_compressor.py): survives gateway restarts, so a
|
||
# failing compression is not re-triggered on every restart.
|
||
# The in-memory dict was reset on every restart, re-triggering the same failing compression and
|
||
# wedging session storage (#74136).
|
||
_session_db = getattr(self, "_session_db", None)
|
||
_getter = getattr(getattr(_session_db, "_db", _session_db), "get_compression_failure_cooldown", None)
|
||
if _getter is not None:
|
||
_cooldown_state = None
|
||
with suppress(Exception):
|
||
_cooldown_state = _getter(session_entry.session_id)
|
||
if _cooldown_state and _cooldown_state.get("remaining_seconds", 0) > 0:
|
||
logger.info(
|
||
"Session hygiene: skipping compression for %s; "
|
||
"previous failure cooldown active for %.1fs",
|
||
session_entry.session_id, _cooldown_state["remaining_seconds"],
|
||
)
|
||
_needs_compress = False
|
||
|
||
if _needs_compress and await self._session_has_compression_in_flight(session_key):
|
||
# A prior compression still holds the durable lock (e.g. a shielded worker left by /stop):
|
||
# another attempt would wait up to 600s behind a commit the fence will refuse.
|
||
logger.info(
|
||
"Session hygiene: skipping compression for %s; "
|
||
"another compression is already in flight", session_entry.session_id,
|
||
)
|
||
_needs_compress = False
|
||
|
||
if _needs_compress:
|
||
logger.info(
|
||
"Session hygiene: %s messages, ~%s tokens (%s) — auto-compressing "
|
||
"(threshold: %s%% of %s = %s tokens)",
|
||
_msg_count, f"{_approx_tokens:,}", _token_source,
|
||
int(hs.threshold_pct * 100), f"{_hyg_context_length:,}", f"{_compress_token_threshold:,}",
|
||
)
|
||
return self._HygienePlan(_needs_compress, _approx_tokens, _msg_count, _warn_token_threshold)
|
||
|
||
async def _hmwa_hygiene_wait_for_summary(self, attempt, hs, session_entry):
|
||
"""Progress-aware inline wait for the detached hygiene compressor. Returns the compressed
|
||
transcript; raises ``HygieneTurnHoldExceeded`` (turn-hold budget) or
|
||
``asyncio.TimeoutError`` (idle/ceiling/fence cancel) for the caller's handlers.
|
||
|
||
Idle timeout (fence ticks per streamed token) + hard ceiling + turn-hold cap."""
|
||
from gateway.run import HygieneTurnHoldExceeded, hygiene_wait_should_extend
|
||
fence = attempt.commit_fence
|
||
while True:
|
||
if fence.is_cancelled:
|
||
raise asyncio.TimeoutError
|
||
# Charge the idle budget from the LAST PROGRESS event, else silence can approach 2x timeout.
|
||
_hyg_waited = time.monotonic() - attempt.wait_started
|
||
_slice = min(
|
||
max(hs.timeout_seconds - fence.seconds_since_progress(), 0.005),
|
||
max(hs.total_ceiling_seconds - _hyg_waited, 0.005),
|
||
)
|
||
# Cap the slice at the remaining turn-hold budget so a continuously-streaming worker can't
|
||
# hold the turn until the ceiling. Budget exhausted → immediate timeout → abandonment.
|
||
_turn_hold_remaining = hs.max_turn_hold_seconds - (time.monotonic() - attempt.wait_started)
|
||
_slice = 0.005 if _turn_hold_remaining <= 0 else min(_slice, max(_turn_hold_remaining, 0.005))
|
||
# Short poll so a /stop or /restart cancel is not stuck behind a full idle window.
|
||
_idle_left = max(hs.timeout_seconds - fence.seconds_since_progress(), 0.005)
|
||
_slice = min(_slice, 0.25)
|
||
try:
|
||
_compressed, _ = await asyncio.wait_for(asyncio.shield(attempt.future), timeout=_slice)
|
||
return _compressed
|
||
except asyncio.TimeoutError:
|
||
if fence.is_cancelled:
|
||
raise
|
||
_hyg_waited = time.monotonic() - attempt.wait_started
|
||
_idle = fence.seconds_since_progress()
|
||
# Never hold the TURN past the budget even while the summary streams: proceed on the
|
||
# uncompressed transcript so the wire never trips a transport idle-timeout.
|
||
if _hyg_waited >= hs.max_turn_hold_seconds:
|
||
logger.info(
|
||
"Session hygiene compression for session %s exceeded the turn-hold "
|
||
"budget (%.1fs >= %.1fs) — abandoning inline wait, proceeding "
|
||
"without compression this turn",
|
||
session_entry.session_id, _hyg_waited, hs.max_turn_hold_seconds,
|
||
)
|
||
raise HygieneTurnHoldExceeded(
|
||
f"turn-hold budget {hs.max_turn_hold_seconds:.1f}s "
|
||
f"elapsed after {_hyg_waited:.1f}s"
|
||
)
|
||
if hygiene_wait_should_extend(
|
||
idle=_idle, timeout=hs.timeout_seconds, waited=_hyg_waited,
|
||
ceiling=hs.total_ceiling_seconds, fence_cancelled=fence.is_cancelled,
|
||
):
|
||
if _slice >= _idle_left - 1e-9:
|
||
logger.info(
|
||
"Session hygiene compression for session %s still streaming after "
|
||
"%.0fs (last progress %.1fs ago) — extending wait (ceiling %.0fs)",
|
||
session_entry.session_id, _hyg_waited, _idle, hs.total_ceiling_seconds,
|
||
)
|
||
continue
|
||
raise
|
||
|
||
async def _hmwa_hygiene_cancel_or_adopt(self, attempt, context):
|
||
"""Cancel the worker at the commit fence; on success release its lease and defer agent
|
||
cleanup, returning ``None``. When the worker already crossed into its commit, consume and
|
||
return the compressed transcript instead (a successful compaction is never a timeout; the
|
||
turn may be held past the budget by up to the commit duration — by design). The lock-free
|
||
``commit_in_flight`` marker keeps the poll from spinning on a hung commit."""
|
||
fence = attempt.commit_fence
|
||
while not fence.commit_in_flight:
|
||
cancelled = fence.try_cancel_before_commit()
|
||
if cancelled is None:
|
||
await asyncio.sleep(0.025)
|
||
elif cancelled:
|
||
fence.release_cancelled_compression_lock()
|
||
self._hmwa_hygiene_defer_cleanup(attempt, context)
|
||
return None
|
||
else:
|
||
break
|
||
_compressed, _ = await attempt.future
|
||
return _compressed
|
||
|
||
def _hmwa_hygiene_defer_cleanup(self, attempt, context):
|
||
"""Hand the agent's cleanup to the still-running worker future and mark it deferred."""
|
||
self._defer_agent_cleanup_until_future_done(attempt.future, attempt.agent, context=context)
|
||
attempt.cleanup_deferred = True
|
||
|
||
@staticmethod
|
||
def _hmwa_hygiene_stamp(agent, desc, provenance_name, debug_label):
|
||
from agent.session_activity import ActivityProvenance
|
||
from gateway.run import _stamp_hygiene_compression_provenance
|
||
_stamp_hygiene_compression_provenance(agent, desc, getattr(ActivityProvenance, provenance_name), debug_label)
|
||
|
||
async def _hmwa_hygiene_notify(self, source, meta, message, what):
|
||
"""Best-effort user notice on the hygiene thread; failure is logged, never raised."""
|
||
try:
|
||
_adapter = self._adapter_for_source(source)
|
||
if _adapter and source.chat_id:
|
||
await _adapter.send(source.chat_id, message, metadata=meta)
|
||
except Exception as _werr:
|
||
logger.warning("Failed to deliver %s to user: %s", what, _werr)
|
||
|
||
async def _hmwa_hygiene_record_failure_cooldown(self, hs, session_key, session_id, reason):
|
||
"""Escalate the failure streak (off-loop) and persist the cooldown, when enabled."""
|
||
from gateway.run import _hygiene_cooldown_for_failure, _record_hygiene_cooldown
|
||
if hs.failure_cooldown_seconds < 0:
|
||
return
|
||
_hyg_cooldown = await asyncio.to_thread(
|
||
_hygiene_cooldown_for_failure, self, session_key, hs.failure_cooldown_seconds,
|
||
)
|
||
_record_hygiene_cooldown(self, session_id, _hyg_cooldown, reason)
|
||
|
||
async def _hmwa_hygiene_on_turn_hold(self, attempt, hs, session_entry, session_key, source):
|
||
"""``except HygieneTurnHoldExceeded`` body: keep or cancel the worker's commit admission,
|
||
notify the user, and re-raise; returns the compressed transcript only when the worker
|
||
was already committing.
|
||
|
||
Turn-hold expiry is an availability boundary, not a failure: the streak must NOT advance,
|
||
only flat retry spacing is recorded. A watermark-fenced commit (rows appended after
|
||
compression start survive as cloned tail) KEEPS admission: the turn proceeds uncompressed
|
||
now and the summary is adopted at the worker's fenced commit — always cancelling burned
|
||
every attempt for thinking summary models. Without the fence a late commit could clobber
|
||
newer turns, so cancel."""
|
||
from gateway.run import (
|
||
_HYGIENE_TURNHOLD_RETRY_SECONDS, _record_hygiene_cooldown, _reset_hygiene_failure_streak
|
||
)
|
||
fence = attempt.commit_fence
|
||
_hyg_keep_admission = bool(getattr(fence, "commit_watermark_fenced", False)) and not fence.is_cancelled
|
||
if _hyg_keep_admission:
|
||
self._hmwa_hygiene_defer_cleanup(attempt, "session hygiene turn-hold")
|
||
# NO retry-after here (it would also block the agent-side preflight compressor); spacing
|
||
# comes from the durable compression lock. The done-callback records the flat retry-after
|
||
# ONLY if the worker ends without committing anything.
|
||
_sid, _skey, _agent = session_entry.session_id, session_key, attempt.agent
|
||
|
||
def _hyg_adopt_or_space_retry(_fut, _gw=self, _sid=_sid, _skey=_skey, _agent=_agent):
|
||
try:
|
||
_exc = _fut.exception()
|
||
except (asyncio.CancelledError, Exception):
|
||
_committed = False
|
||
else:
|
||
_committed = _exc is None and (
|
||
bool(getattr(_agent, "_last_compaction_in_place", False))
|
||
or getattr(_agent, "session_id", _sid) != _sid
|
||
)
|
||
if _committed:
|
||
logger.info(
|
||
"Session hygiene compression for session %s finished after the "
|
||
"turn-hold was released — summary adopted at the watermark-fenced "
|
||
"commit boundary (#97963)", _sid,
|
||
)
|
||
try:
|
||
_reset_hygiene_failure_streak(_gw, _skey)
|
||
except Exception as _rs_err:
|
||
logger.debug("hygiene streak reset after deferred adoption failed: %s", _rs_err)
|
||
else:
|
||
# Nothing to adopt (summary failed / fence refused / superseded): flat spacing so
|
||
# sustained traffic doesn't spawn and abandon a compressor every turn.
|
||
_record_hygiene_cooldown(
|
||
_gw, _sid, _HYGIENE_TURNHOLD_RETRY_SECONDS,
|
||
"hygiene compression deferred: turn-hold budget expired and the "
|
||
"detached attempt did not commit",
|
||
)
|
||
|
||
attempt.future.add_done_callback(_hyg_adopt_or_space_retry)
|
||
_log_suffix = (
|
||
" — the watermark-fenced worker keeps its commit admission and the summary "
|
||
"will be adopted when it finishes"
|
||
)
|
||
else:
|
||
_adopted = await self._hmwa_hygiene_cancel_or_adopt(attempt, "session hygiene turn-hold")
|
||
if _adopted is not None:
|
||
return _adopted
|
||
# Short flat retry-after, else every turn re-spawns, holds and cancels a compressor.
|
||
_record_hygiene_cooldown(
|
||
self, session_entry.session_id, _HYGIENE_TURNHOLD_RETRY_SECONDS,
|
||
"hygiene compression deferred: turn-hold budget expired while the "
|
||
"summary was still streaming",
|
||
)
|
||
_log_suffix = ""
|
||
self._hmwa_hygiene_stamp(
|
||
attempt.agent, "session hygiene compression turn-hold",
|
||
"AGENT_COMPRESSION_TURNHOLD", "hygiene compression turn-hold activity stamp failed",
|
||
)
|
||
logger.info(
|
||
"Session hygiene compression for session %s exceeded turn-hold budget (%.1fs); "
|
||
"proceeding without compression this turn%s",
|
||
session_entry.session_id, time.monotonic() - attempt.wait_started, _log_suffix,
|
||
)
|
||
await self._hmwa_hygiene_notify(
|
||
source, attempt.meta, t("gateway.compress.turnhold_deferred"), "compression-turnhold notice",
|
||
)
|
||
raise
|
||
|
||
async def _hmwa_hygiene_on_timeout(self, attempt, hs, session_entry, session_key, source):
|
||
"""``except asyncio.TimeoutError`` body: cancel at the commit fence, record the failure
|
||
cooldown, warn the user, and re-raise; returns the compressed transcript only when the
|
||
worker crossed the commit boundary first."""
|
||
from gateway.run import _hygiene_compression_timeout_message
|
||
fence = attempt.commit_fence
|
||
_hyg_waited = time.monotonic() - attempt.wait_started
|
||
_hyg_total_exhausted = _hyg_waited >= hs.total_ceiling_seconds or fence.deadline_exceeded
|
||
if _hyg_total_exhausted:
|
||
# The worker checks this deadline between digest calls; keep its lease until it exits so
|
||
# an unchanged session cannot overlap a retry (the release below is then a no-op).
|
||
fence.retain_compression_lock_until_worker_done()
|
||
# Capture fence state BEFORE try_cancel (which itself sets is_cancelled).
|
||
_hyg_fence_cancelled = fence.is_cancelled
|
||
_adopted = await self._hmwa_hygiene_cancel_or_adopt(attempt, "session hygiene timeout")
|
||
if _adopted is not None:
|
||
return _adopted
|
||
await self._hmwa_hygiene_record_failure_cooldown(
|
||
hs, session_key, session_entry.session_id,
|
||
"session hygiene compression " + (
|
||
"cancelled at commit fence" if _hyg_fence_cancelled
|
||
else "total ceiling exhausted" if _hyg_total_exhausted
|
||
else "timed out with no output from the summary model"
|
||
),
|
||
)
|
||
self._hmwa_hygiene_stamp(
|
||
attempt.agent,
|
||
"session hygiene compression cancelled at commit fence" if _hyg_fence_cancelled
|
||
else "session hygiene compression timed out",
|
||
"AGENT_COMPRESSION_TIMEOUT", "hygiene compression timeout activity stamp failed",
|
||
)
|
||
if _hyg_fence_cancelled:
|
||
logger.warning(
|
||
"Session hygiene compression for session %s was cancelled at the "
|
||
"commit fence; continuing without compression", session_entry.session_id,
|
||
)
|
||
raise
|
||
_hyg_elapsed = time.monotonic() - attempt.wait_started
|
||
if _hyg_total_exhausted:
|
||
logger.warning(
|
||
"Session hygiene compression for session %s reached its total ceiling after "
|
||
"%.1fs (progress observed=%s); continuing without compression",
|
||
session_entry.session_id, _hyg_elapsed, fence.progress_observed,
|
||
)
|
||
else:
|
||
logger.warning(
|
||
"Session hygiene compression for session %s made no progress for %.1fs "
|
||
"(total wait %.1fs, ceiling %.1fs); continuing without compression",
|
||
session_entry.session_id, fence.seconds_since_progress(), _hyg_elapsed, hs.total_ceiling_seconds,
|
||
)
|
||
await self._hmwa_hygiene_notify(
|
||
source, attempt.meta,
|
||
_hygiene_compression_timeout_message(
|
||
total_exhausted=_hyg_total_exhausted, elapsed=_hyg_elapsed,
|
||
idle_timeout=hs.timeout_seconds, progress_observed=fence.progress_observed,
|
||
),
|
||
"compression-timeout warning",
|
||
)
|
||
raise
|
||
|
||
def _hmwa_hygiene_on_unwind(self, attempt, hs, session_entry, session_key):
|
||
"""``except BaseException`` body (caller re-raises): revoke commit admission BEFORE the host
|
||
unwinds so the detached worker can never commit later, and record a cooldown — otherwise
|
||
the next turn re-arms hygiene and waits up to 600s behind a fence that refuses again."""
|
||
from gateway.run import _hygiene_cooldown_for_failure, _record_hygiene_cooldown
|
||
attempt.commit_fence.revoke_commit_admission()
|
||
if not attempt.cleanup_deferred:
|
||
self._hmwa_hygiene_defer_cleanup(attempt, "session hygiene unwind")
|
||
if hs.failure_cooldown_seconds >= 0:
|
||
try:
|
||
_record_hygiene_cooldown(
|
||
self, session_entry.session_id,
|
||
_hygiene_cooldown_for_failure(self, session_key, hs.failure_cooldown_seconds),
|
||
"session hygiene compression cancelled at commit fence",
|
||
)
|
||
except Exception as _cd_err:
|
||
logger.debug("hygiene unwind cooldown record failed: %s", _cd_err)
|
||
|
||
async def _hmwa_hygiene_adopt_transcript(
|
||
self, attempt, _compressed, history, plan, *, session_entry, source, _quick_key, run_generation,
|
||
):
|
||
"""Adopt a finished compression (rotation / in-place / refused); publishes the transcript to
|
||
continue with on ``attempt.history``. Returns ``(rotated, in_place, new_count, new_tokens)``.
|
||
|
||
Rewrite only on rotation (NEW session id): in-place compaction already soft-archived the
|
||
old rows and rewrite_transcript() would DELETE them; neither rotation nor in-place signals
|
||
FAILURE and an unconditional rewrite would leave only the summary. Write-before-repoint:
|
||
a repoint-then-failed-rewrite would point the live entry at an empty session."""
|
||
from agent.model_metadata import estimate_messages_tokens_rough
|
||
_hyg_agent = attempt.agent
|
||
# _compress_context rotates to a NEW session_id so the old transcript stays intact/searchable.
|
||
_hyg_new_sid = _hyg_agent.session_id
|
||
_hyg_rotated = _hyg_new_sid != session_entry.session_id
|
||
_hyg_in_place = bool(getattr(_hyg_agent, "_last_compaction_in_place", False))
|
||
# Anti-growth guard: refuse a compression that did not shrink the transcript (seen 427K→598K).
|
||
_hyg_in_toks = estimate_messages_tokens_rough(history)
|
||
_hyg_out_toks = estimate_messages_tokens_rough(_compressed)
|
||
if _hyg_rotated and _hyg_out_toks > _hyg_in_toks:
|
||
logger.warning(
|
||
"Gateway hygiene compression for session %s would grow transcript (~%s -> ~%s "
|
||
"tokens); keeping the original transcript unchanged",
|
||
session_entry.session_id, f"{_hyg_in_toks:,}", f"{_hyg_out_toks:,}",
|
||
)
|
||
_hyg_rotated = False
|
||
_compressed = history
|
||
# Only rewrite the transcript when rotation produced a NEW session id. In-place compaction does NOT
|
||
# need a rewrite: archive_and_compact() has already soft-archived the previous active rows and
|
||
# inserted the compacted messages as the new active set inside _compress_context(). Calling
|
||
# rewrite_transcript() after in-place compaction would invoke replace_messages(active_only=False)
|
||
# which DELETEs ALL rows — including the archived turns that archive_and_compact() deliberately
|
||
# preserved (silent data loss, #61145). The danger this guards against (mirrors the /compress fix
|
||
# #44794/#39704): if _compress_context returns a summary but neither rotates nor completes
|
||
# archive_and_compact(), the session_id is unchanged for a FAILURE reason, and an unconditional
|
||
# rewrite_transcript() would DELETE the original messages and replace them with only the compressed
|
||
# summary (permanent data loss, #21301). Write-before-repoint (mirrors manual /compress): if we
|
||
# repointed session_entry onto the child SID and rewrite_transcript then failed (lock/ENOSPC), the
|
||
# live entry would already reference a brand-new empty session while the turn continues — the
|
||
# conversation silently vanishes. Persist the child transcript first; only then rebind the live
|
||
# entry.
|
||
if _hyg_rotated:
|
||
if not await self.async_session_store.rewrite_transcript(_hyg_new_sid, _compressed):
|
||
logger.error(
|
||
"Session hygiene: failed to persist compressed transcript for rotated session "
|
||
"%s → %s; keeping the live entry on the original session so the "
|
||
"conversation is not dropped", session_entry.session_id, _hyg_new_sid,
|
||
)
|
||
# Fail closed: treat like no rotation.
|
||
_hyg_rotated = False
|
||
_hyg_in_place = False
|
||
else:
|
||
session_entry.session_id = _hyg_new_sid
|
||
# The held turn lease follows the rotation (alias keys still serialize on this turn).
|
||
self._rebind_turn_lease(_quick_key, run_generation, _hyg_new_sid)
|
||
await self.async_session_store._save()
|
||
await asyncio.to_thread(
|
||
self._sync_telegram_topic_binding, source, session_entry, reason="hygiene-compression",
|
||
)
|
||
|
||
if _hyg_rotated or _hyg_in_place:
|
||
# Rewritten (rotation) or persisted by archive_and_compact() (in-place): reset token count.
|
||
session_entry.last_prompt_tokens = 0
|
||
attempt.history = _compressed
|
||
_new_count = len(_compressed)
|
||
_new_tokens = estimate_messages_tokens_rough(_compressed)
|
||
else:
|
||
# No rewrite happened — post-compression counts equal the pre-compression ones.
|
||
_new_count = plan.msg_count
|
||
_new_tokens = plan.approx_tokens
|
||
logger.warning(
|
||
"Gateway hygiene compression for session %s did not rotate or compact in place (no "
|
||
"session_db on the hygiene agent) — preserving the original transcript instead "
|
||
"of overwriting it with the summary (#21301).", session_entry.session_id,
|
||
)
|
||
|
||
logger.info(
|
||
"Session hygiene: compressed %s → %s msgs, ~%s → ~%s tokens",
|
||
plan.msg_count, _new_count, f"{plan.approx_tokens:,}", f"{_new_tokens:,}",
|
||
)
|
||
if _new_tokens >= plan.warn_token_threshold:
|
||
logger.warning("Session hygiene: still ~%s tokens after compression", f"{_new_tokens:,}")
|
||
return _hyg_rotated, _hyg_in_place, _new_count, _new_tokens
|
||
|
||
async def _hmwa_hygiene_apply_result(
|
||
self, attempt, hs, _compressed, history, plan, *,
|
||
session_entry, session_key, source, _quick_key, run_generation,
|
||
):
|
||
"""Adopt a finished hygiene compression, rebind the session + turn lease, record
|
||
streak/cooldown, and warn the user on abort."""
|
||
from gateway.run import _reset_hygiene_failure_streak, hygiene_compaction_recovered
|
||
_hyg_rotated, _hyg_in_place, _new_count, _new_tokens = await self._hmwa_hygiene_adopt_transcript(
|
||
attempt, _compressed, history, plan, session_entry=session_entry, source=source,
|
||
_quick_key=_quick_key, run_generation=run_generation,
|
||
)
|
||
# Summary failure aborts the compressor (nothing dropped). Warn the user visibly — agent.log
|
||
# is invisible on TG/Discord — so they know the chat is "frozen" and can /compress or /reset.
|
||
_comp = getattr(attempt.agent, "context_compressor", None)
|
||
_hyg_aborted = _comp is not None and getattr(_comp, "_last_compress_aborted", False)
|
||
# A fence-cancelled _compress_context returns the original transcript with
|
||
# _last_compress_aborted False: treat that no-op as an abort so hygiene records a cooldown
|
||
# instead of retrying into the 600s wait. A committed rotate/in-place is never an abort.
|
||
_hyg_fence_cancelled = bool(attempt.commit_fence.is_cancelled and not _hyg_rotated and not _hyg_in_place)
|
||
if _hyg_fence_cancelled:
|
||
_hyg_aborted = True
|
||
# Recovery decision lives in the unit-tested predicate: the "neither rotated nor in place"
|
||
# path reuses pre-compression counts, so a numbers-only check would read a no-op as success.
|
||
if not _hyg_aborted and hygiene_compaction_recovered(
|
||
aborted=_hyg_aborted, rotated=_hyg_rotated, in_place=_hyg_in_place,
|
||
msg_count=plan.msg_count, new_count=_new_count, approx_tokens=plan.approx_tokens,
|
||
new_tokens=_new_tokens,
|
||
):
|
||
await asyncio.to_thread(_reset_hygiene_failure_streak, self, session_key)
|
||
if _hyg_aborted:
|
||
await self._hmwa_hygiene_record_failure_cooldown(
|
||
hs, session_key, session_entry.session_id,
|
||
"session hygiene compression cancelled at commit fence" if _hyg_fence_cancelled
|
||
else getattr(_comp, "_last_summary_error", None),
|
||
)
|
||
self._hmwa_hygiene_stamp(
|
||
attempt.agent, "session hygiene compression aborted",
|
||
"AGENT_COMPRESSION_COOLDOWN", "hygiene compression abort activity stamp failed",
|
||
)
|
||
if not _hyg_fence_cancelled:
|
||
# Force-redact: provider exception text may contain credentials; this reaches users.
|
||
from agent.redact import redact_sensitive_text
|
||
_err = redact_sensitive_text(getattr(_comp, "_last_summary_error", None) or "unknown error", force=True)
|
||
await self._hmwa_hygiene_notify(
|
||
source, attempt.meta, "⚠️ Context compression aborted "
|
||
f"({_err}). No messages were dropped — "
|
||
"conversation is unchanged. Run /compress to retry, /reset for a clean "
|
||
"session, or check your auxiliary.compression model configuration.",
|
||
"compression-failure warning",
|
||
)
|
||
# Configured aux model failed, recovered on the main model: only the user can fix that config.
|
||
elif _comp is not None and getattr(_comp, "_last_aux_model_failure_model", None):
|
||
_aux_model = getattr(_comp, "_last_aux_model_failure_model", "")
|
||
_aux_err = getattr(_comp, "_last_aux_model_failure_error", None) or "unknown error"
|
||
await self._hmwa_hygiene_notify(
|
||
source, attempt.meta, f"ℹ️ Configured compression model `{_aux_model}` "
|
||
f"failed ({_aux_err}). Recovered using your main "
|
||
"model — context is intact — but you may want to "
|
||
"check `auxiliary.compression.model` in config.yaml.",
|
||
"aux-model-fallback notice",
|
||
)
|
||
|
||
async def _hmwa_hygiene_codex_compaction(self, hs, plan, history, session_entry, session_key, _hyg_runtime):
|
||
"""codex app-server runtime: the real context is the server-side thread, not the transcript
|
||
mirror. The detached-agent path would only rewrite the mirror and its finally-eviction
|
||
would destroy the live thread (next turn starts blank), so use the cached agent's
|
||
thread/compact/start and KEEP it cached."""
|
||
from gateway.run import run_codex_hygiene_compaction
|
||
# codex app-server runtime: the model's real context is the app-server's server-side thread, not the
|
||
# transcript mirror. See #73503.
|
||
_hyg_codex_auto = "native"
|
||
_hyg_comp_cfg = hs.data.get("compression") if isinstance(hs.data, dict) else None
|
||
if isinstance(_hyg_comp_cfg, dict):
|
||
_hyg_codex_auto = str(_hyg_comp_cfg.get("codex_app_server_auto", "native") or "native")
|
||
_hyg_codex_outcome = await run_codex_hygiene_compaction(
|
||
self, session_key, session_entry.session_id, auto_mode=_hyg_codex_auto, history=history,
|
||
approx_tokens=plan.approx_tokens, timeout_seconds=hs.total_ceiling_seconds,
|
||
failure_cooldown_seconds=hs.failure_cooldown_seconds,
|
||
)
|
||
logger.info(
|
||
"Session hygiene (codex app-server): %s (session=%s, mode=%s, ~%s tokens)",
|
||
_hyg_codex_outcome, session_entry.session_id, _hyg_codex_auto, f"{plan.approx_tokens:,}",
|
||
)
|
||
|
||
async def _hmwa_hygiene_build_agent(self, _hyg_model, _hyg_runtime, session_entry):
|
||
"""Build the detached hygiene ``AIAgent`` with the live session's system prompt. Returns
|
||
``(agent, sync_session_db)``."""
|
||
from gateway.run import _GATEWAY_HYGIENE_PLATFORM, _seed_hygiene_system_prompt
|
||
from run_agent import AIAgent
|
||
try:
|
||
_hyg_session_row = await self._session_db.get_session(session_entry.session_id)
|
||
except Exception as exc:
|
||
_hyg_session_row = None
|
||
logger.warning(
|
||
"Session hygiene could not restore the system prompt for session %s: %s. "
|
||
"Preserving an empty prompt so the live turn rebuilds it with its "
|
||
"configured providers.", session_entry.session_id, exc, exc_info=True,
|
||
)
|
||
_hyg_session_db = getattr(self._session_db, "_db", self._session_db)
|
||
# With compression.checkpoint_required on, load the memory provider so the checkpoint exists
|
||
# before any mutation; otherwise keep the fast path (no provider init).
|
||
from hermes_cli.config import load_config as _load_cfg
|
||
from utils import is_truthy_value as _is_truthy
|
||
|
||
_hyg_checkpoint_required = _is_truthy(
|
||
((_load_cfg() or {}).get("compression") or {}).get("checkpoint_required"), default=False,
|
||
)
|
||
_hyg_agent = AIAgent(
|
||
**_hyg_runtime, model=_hyg_model, max_iterations=4, quiet_mode=True,
|
||
skip_memory=not _hyg_checkpoint_required, enabled_toolsets=["memory"],
|
||
session_id=session_entry.session_id, session_db=_hyg_session_db,
|
||
)
|
||
_seed_hygiene_system_prompt(_hyg_agent, _hyg_session_row)
|
||
# A rebuilt (not retained) prompt is deliberately stale for every real gateway surface.
|
||
_hyg_agent.platform = _GATEWAY_HYGIENE_PLATFORM
|
||
return _hyg_agent, _hyg_session_db
|
||
|
||
async def _hmwa_hygiene_detached_attempt(
|
||
self, attempt, hs, plan, history, _hyg_msgs, _hyg_model, _hyg_runtime,
|
||
source, session_entry, session_key, _quick_key, run_generation,
|
||
):
|
||
"""Run one detached hygiene compression attempt end to end; publishes the transcript to
|
||
continue with (compressed or original) on ``attempt.history``."""
|
||
from gateway.run import HygieneTurnHoldExceeded
|
||
from agent.conversation_compression import CompressionCommitFence
|
||
_hyg_agent, _hyg_session_db = await self._hmwa_hygiene_build_agent(_hyg_model, _hyg_runtime, session_entry)
|
||
attempt.agent = _hyg_agent
|
||
try:
|
||
# Hygiene owns the session binding, so prefer in-place compaction over minting a
|
||
# continuation child. Without a SessionDB this stays False.
|
||
_hyg_agent.compression_in_place = True
|
||
_bind_hyg_state = getattr(getattr(_hyg_agent, "context_compressor", None), "bind_session_state", None)
|
||
if callable(_bind_hyg_state):
|
||
_bind_hyg_state(_hyg_session_db, session_entry.session_id)
|
||
# Never finalize on close() — that would end the live gateway session row.
|
||
_hyg_agent._end_session_on_close = False
|
||
_hyg_agent._print_fn = lambda *a, **kw: None
|
||
|
||
loop = asyncio.get_running_loop()
|
||
_hyg_commit_fence = CompressionCommitFence(total_ceiling_seconds=hs.total_ceiling_seconds)
|
||
# Default executor (NOT self._get_executor): a hung summary must never occupy an
|
||
# agent-work slot. MUST run in the caller's contextvars (multiplex secret scope).
|
||
attempt.commit_fence = _hyg_commit_fence
|
||
attempt.future = loop.run_in_executor(
|
||
None,
|
||
# But it MUST run inside the caller's contextvars: under multiplex_profiles the profile
|
||
# secret scope / HERMES_HOME override live in ContextVars, and a bare run_in_executor worker
|
||
# starts with an empty Context — the summary model's get_secret(<PROVIDER>_API_KEY) then
|
||
# fails closed (UnscopedSecretError) and every hygiene compaction silently degrades to a
|
||
# lossy truncation (#100849 bundle).
|
||
copy_context().run,
|
||
lambda: _hyg_agent._compress_context(
|
||
_hyg_msgs, "", approx_tokens=plan.approx_tokens, commit_fence=_hyg_commit_fence,
|
||
),
|
||
)
|
||
attempt.wait_started = time.monotonic()
|
||
try:
|
||
_compressed = await self._hmwa_hygiene_wait_for_summary(attempt, hs, session_entry)
|
||
except HygieneTurnHoldExceeded:
|
||
_compressed = await self._hmwa_hygiene_on_turn_hold(attempt, hs, session_entry, session_key, source)
|
||
except asyncio.TimeoutError:
|
||
_compressed = await self._hmwa_hygiene_on_timeout(attempt, hs, session_entry, session_key, source)
|
||
except BaseException:
|
||
self._hmwa_hygiene_on_unwind(attempt, hs, session_entry, session_key)
|
||
raise
|
||
|
||
await self._hmwa_hygiene_apply_result(
|
||
attempt, hs, _compressed, history, plan, session_entry=session_entry,
|
||
session_key=session_key, source=source, _quick_key=_quick_key,
|
||
run_generation=run_generation,
|
||
)
|
||
finally:
|
||
# Evict the cached agent so the next turn rebuilds its system prompt.
|
||
self._evict_cached_agent(session_key)
|
||
if not attempt.cleanup_deferred:
|
||
await self._cleanup_agent_resources_off_loop(_hyg_agent, context="session hygiene")
|
||
|
||
async def _hmwa_run_session_hygiene(
|
||
self, event, source, session_entry, session_key, history, _quick_key, run_generation,
|
||
):
|
||
"""Auto-compress pathologically large transcripts before the agent starts so oversized
|
||
histories don't cause repeated truncation/context failures. Token source: the API's
|
||
prompt_tokens from the last turn, else a char/4 estimate."""
|
||
from gateway.run import HygieneTurnHoldExceeded
|
||
if not history or len(history) < 4:
|
||
return history
|
||
|
||
hs = await self._hmwa_hygiene_settings(source, session_key)
|
||
if not hs.compression_enabled:
|
||
return history
|
||
plan = await self._hmwa_hygiene_plan(hs, history, session_entry, session_key)
|
||
if not plan.needs_compress:
|
||
return history
|
||
|
||
attempt = self._HygieneAttempt(agent=None, meta=self._event_thread_metadata(event, source), history=history)
|
||
try:
|
||
_hyg_model, _hyg_runtime = self._resolve_session_agent_runtime(
|
||
source=source, session_key=session_key,
|
||
user_config=hs.data if isinstance(hs.data, dict) else None,
|
||
)
|
||
if str(_hyg_runtime.get("api_mode") or "").lower() == "codex_app_server":
|
||
await self._hmwa_hygiene_codex_compaction(hs, plan, history, session_entry, session_key, _hyg_runtime)
|
||
elif _hyg_runtime.get("api_key"):
|
||
# Pass the FULL transcript (tool results included) as the agent loop does: filtering
|
||
# to user/assistant starved the compressor (tool results are the bulk of context).
|
||
_hyg_msgs = [m for m in history if m.get("role") in {"user", "assistant", "tool"}]
|
||
if len(_hyg_msgs) >= 4:
|
||
await self._hmwa_hygiene_detached_attempt(
|
||
attempt, hs, plan, history, _hyg_msgs, _hyg_model, _hyg_runtime,
|
||
source, session_entry, session_key, _quick_key, run_generation,
|
||
)
|
||
except HygieneTurnHoldExceeded:
|
||
# Availability boundary, not a failure — already logged at INFO by the turn-hold handler.
|
||
# Must not hit the generic "auto-compress failed" warning below: that log is how thinking-model
|
||
# deployments read as permanently broken (#97963; surfaced by @686f6c61 in PR #99657).
|
||
pass
|
||
except Exception as e:
|
||
logger.warning("Session hygiene auto-compress failed: %s", e)
|
||
return attempt.history
|
||
|
||
async def _hmwa_first_contact_notes(self, source, history, turn_sidecar_notes):
|
||
"""First-ever-message onboarding note + one-time 'no home channel' prompt (both only when
|
||
the session has no history). Delivered on the user message (sidecar), NOT the ephemeral
|
||
system prompt: present-on-turn-1/absent-on-turn-2 was a guaranteed prompt diff + rebuild."""
|
||
from gateway.run import _hermes_home, _home_target_env_var, _load_gateway_config
|
||
if history:
|
||
return
|
||
if not await self.async_session_store.has_any_sessions():
|
||
_intro_note = (
|
||
"[System note: This is the user's very first message ever. "
|
||
"Briefly introduce yourself and mention that /help shows available commands. "
|
||
"Keep the introduction concise -- one or two sentences max.]"
|
||
)
|
||
# onboarding.profile_build == "ask" (default) and not yet offered: swap the plain intro for
|
||
# a consent-gated profile-build directive. Fires at most once.
|
||
try:
|
||
from agent.onboarding import (
|
||
PROFILE_BUILD_FLAG, is_seen, mark_seen, profile_build_directive,
|
||
profile_build_mode,
|
||
)
|
||
_onb_cfg = _load_gateway_config()
|
||
if profile_build_mode(_onb_cfg) == "ask" and not is_seen(_onb_cfg, PROFILE_BUILD_FLAG):
|
||
turn_sidecar_notes.append(profile_build_directive().strip())
|
||
mark_seen(_hermes_home / "config.yaml", PROFILE_BUILD_FLAG)
|
||
else:
|
||
turn_sidecar_notes.append(_intro_note)
|
||
except Exception as _pb_err:
|
||
logger.debug("Profile-build onboarding directive failed, using plain intro: %s", _pb_err)
|
||
turn_sidecar_notes.append(_intro_note)
|
||
|
||
# One-time prompt if no home channel is set (webhooks deliver to configured targets instead).
|
||
if not source.platform or source.platform in (Platform.LOCAL, Platform.WEBHOOK):
|
||
return
|
||
platform_name = source.platform.value
|
||
env_key = _home_target_env_var(platform_name)
|
||
# Multiplex: the home channel may live only in the profile secret scope, not os.environ.
|
||
home_env = ""
|
||
if env_key:
|
||
with suppress(Exception):
|
||
from agent.secret_scope import get_secret
|
||
home_env = (get_secret(env_key) or "").strip()
|
||
home_env = home_env or (os.getenv(env_key) or "").strip()
|
||
# Also honor in-memory / yaml home_channel on this platform.
|
||
with suppress(Exception):
|
||
if not home_env and self.config.get_home_channel(source.platform):
|
||
home_env = "set"
|
||
# Secondary-profile platforms may only exist under that profile's config — re-read in scope.
|
||
if not home_env:
|
||
with suppress(Exception):
|
||
from gateway.config import load_gateway_config as _lgc
|
||
prof = (getattr(source, "profile", None) or "").strip()
|
||
if prof and prof != "default" and _lgc().get_home_channel(source.platform):
|
||
home_env = "set"
|
||
if not home_env:
|
||
# Slack routes every command through the parent `/hermes`; bare `/sethome` would fail.
|
||
sethome_cmd = "/hermes sethome" if source.platform == Platform.SLACK else "/sethome"
|
||
await self._deliver_platform_notice(
|
||
source, f"📬 No home channel is set for {platform_name.title()}. "
|
||
f"A home channel is where Hermes delivers cron job results and cross-platform "
|
||
f"messages.\n\nType {sethome_cmd} to make this chat your home channel, or ignore "
|
||
f"to skip.",
|
||
)
|
||
|
||
def _hmwa_apply_message_timestamp(self, event, message_text):
|
||
"""Capture the platform event time as message metadata and keep the persisted transcript
|
||
clean (strip any leading timestamp prefix) regardless of the toggle; only the in-context
|
||
RENDER is gated behind gateway.message_timestamps.enabled (default OFF)."""
|
||
from gateway.run import _load_gateway_config, _message_timestamps_enabled
|
||
persist_user_message = None
|
||
persist_user_timestamp = None
|
||
try:
|
||
from hermes_time import get_timezone as _get_evt_tz
|
||
from gateway.message_timestamps import (
|
||
coerce_message_timestamp as _coerce_msg_ts,
|
||
render_user_content_with_timestamp as _render_msg_ts,
|
||
strip_leading_message_timestamps as _strip_msg_ts,
|
||
)
|
||
_evt_tz = _get_evt_tz()
|
||
if message_text and isinstance(message_text, str):
|
||
_clean_message_text, _embedded_ts = _strip_msg_ts(message_text, tz=_evt_tz)
|
||
persist_user_message = _clean_message_text
|
||
_event_epoch = _coerce_msg_ts(getattr(event, "timestamp", None), tz=_evt_tz)
|
||
persist_user_timestamp = _event_epoch if _event_epoch is not None else _embedded_ts
|
||
if _message_timestamps_enabled(_load_gateway_config()):
|
||
message_text = _render_msg_ts(_clean_message_text, persist_user_timestamp, tz=_evt_tz)
|
||
else:
|
||
# Toggle off: the model sees the clean message; timestamp stored for later opt-in.
|
||
message_text = _clean_message_text
|
||
except Exception as _ts_err:
|
||
logger.debug("Message timestamp injection failed (non-fatal): %s", _ts_err)
|
||
return message_text, persist_user_message, persist_user_timestamp
|
||
|
||
async def _hmwa_stop_typing_for_turn(self, event, source):
|
||
"""Stop the typing indicator (never raises). Slack AI status is scoped to a thread/
|
||
workspace, so preserve the routing metadata used by the response delivery path."""
|
||
with suppress(Exception):
|
||
_typing_adapter = self._adapter_for_source(source)
|
||
_kind = type(_typing_adapter)
|
||
if _typing_adapter and callable(getattr(_kind, "_stop_typing_with_metadata", None)):
|
||
await _typing_adapter._stop_typing_with_metadata(source.chat_id, self._event_thread_metadata(event, source))
|
||
elif _typing_adapter and callable(getattr(_kind, "stop_typing", None)):
|
||
await _typing_adapter.stop_typing(source.chat_id)
|
||
|
||
async def _hmwa_shape_agent_response(
|
||
self, agent_result, source, history, session_entry, session_key,
|
||
_quick_key, run_generation, _run_start_session_id, _platform_name, _msg_start_time,
|
||
):
|
||
"""Turn the raw agent result into the outbound text: sentinel/silence handling, response
|
||
logging, resume-pending clear, empty-response normalization, and identity-guarded
|
||
post-compression session_id propagation. Returns
|
||
``(response, _intentional_silence, agent_messages)``."""
|
||
from gateway.run import (
|
||
_is_gateway_hidden_reasoning_incomplete_turn, _normalize_empty_agent_response,
|
||
_sanitize_gateway_final_response, _should_clear_resume_pending_after_turn,
|
||
)
|
||
response = agent_result.get("final_response") or ""
|
||
# Hidden-reasoning-only retry exhaustion: the loop's sentinel text doubles as final_response
|
||
# and would be delivered verbatim (peer agents would ingest it as a completed turn).
|
||
if _is_gateway_hidden_reasoning_incomplete_turn(agent_result):
|
||
response = ""
|
||
_intentional_silence = self._is_intentional_silence(agent_result, response)
|
||
|
||
# "(empty)" = the model produced no visible content after exhausting all retries.
|
||
if response == "(empty)" and not _intentional_silence:
|
||
response = (
|
||
"⚠️ The model returned no response after processing tool results. This can happen "
|
||
"with some models — try again or rephrase your question."
|
||
)
|
||
agent_messages = agent_result.get("messages", [])
|
||
logger.info(
|
||
"response ready: platform=%s chat=%s time=%.1fs api_calls=%d response=%d chars",
|
||
_platform_name, source.chat_id or "unknown",
|
||
time.time() - _msg_start_time, agent_result.get("api_calls", 0), len(response),
|
||
)
|
||
|
||
# Successful turn: clear the consecutive-restart stuck-loop counter and resume_pending (set
|
||
# by drain-timeout shutdown) so later messages don't get the restart-interruption note.
|
||
if session_key and _should_clear_resume_pending_after_turn(agent_result):
|
||
await self._clear_restart_failure_count(session_key)
|
||
try:
|
||
await self.async_session_store.clear_resume_pending(session_key)
|
||
except Exception as _e:
|
||
logger.debug("clear_resume_pending failed for %s: %s", session_key, _e)
|
||
|
||
# Normalize empty responses: surface errors, partial failures, and work-without-text.
|
||
# Fix for #18765.
|
||
if not _intentional_silence:
|
||
response = _normalize_empty_agent_response(agent_result, response, history_len=len(history))
|
||
response = _sanitize_gateway_final_response(source.platform, response)
|
||
|
||
# The agent thread already updated the contextvar; propagate to SessionEntry + _save() only
|
||
# if the binding still points at the session this run was launched against.
|
||
if agent_result.get("session_id") and agent_result["session_id"] != session_entry.session_id:
|
||
if session_entry.session_id == _run_start_session_id:
|
||
session_entry.session_id = agent_result["session_id"]
|
||
# The held turn lease follows the rotation (persistence writes to the NEW id).
|
||
self._rebind_turn_lease(_quick_key, run_generation, session_entry.session_id)
|
||
await self.async_session_store._save()
|
||
await self.async_session_store._record_gateway_session_peer(
|
||
session_entry.session_id, session_key, source,
|
||
)
|
||
await asyncio.to_thread(
|
||
self._sync_telegram_topic_binding, source, session_entry, reason="agent-result-compression",
|
||
)
|
||
else:
|
||
logger.info(
|
||
"Skipping agent-result session split sync for %s because the session binding "
|
||
"moved from %s to %s before compression finished",
|
||
session_key or "?", _run_start_session_id, session_entry.session_id,
|
||
)
|
||
return response, _intentional_silence, agent_messages
|
||
|
||
# reasoning_style → (header line, per-line quote prefix for blank / non-blank lines)
|
||
_REASONING_QUOTE_STYLES = {
|
||
"subtext": ("-# 💭 Reasoning", "-# ", "-#"), "blockquote": ("> 💭 **Reasoning:**", "> ", ">")
|
||
}
|
||
|
||
def _hmwa_prepend_reasoning(self, agent_result, response, source, _intentional_silence):
|
||
"""Prepend the last reasoning block when show_reasoning is on for this platform. Mattermost
|
||
requires an explicit per-platform opt-in (scratch text, not final-answer content)."""
|
||
from gateway.run import _load_gateway_config, _platform_config_key, _resolve_gateway_display_bool
|
||
try:
|
||
_show_reasoning_effective = _resolve_gateway_display_bool(
|
||
_load_gateway_config(), _platform_config_key(source.platform), "show_reasoning",
|
||
default=bool(getattr(self, "_show_reasoning", False)), platform=source.platform,
|
||
require_platform_override_for={Platform.MATTERMOST},
|
||
)
|
||
except Exception:
|
||
_show_reasoning_effective = (
|
||
False if source.platform == Platform.MATTERMOST else getattr(self, "_show_reasoning", False)
|
||
)
|
||
last_reasoning = agent_result.get("last_reasoning")
|
||
if not (_show_reasoning_effective and response and not _intentional_silence and last_reasoning):
|
||
return response
|
||
from gateway.stream_consumer_fences import escape_code_fences_for_display
|
||
# Collapse long reasoning to keep messages readable
|
||
lines = last_reasoning.strip().splitlines()
|
||
if len(lines) > 15:
|
||
display_reasoning = "\n".join(lines[:15]) + f"\n_... ({len(lines) - 15} more lines)_"
|
||
else:
|
||
display_reasoning = last_reasoning.strip()
|
||
# Per-platform render style: Discord defaults to "-# " subtext, others keep the code block.
|
||
try:
|
||
from gateway.display_config import resolve_display_setting
|
||
_reasoning_style = resolve_display_setting(
|
||
_load_gateway_config(), _platform_config_key(source.platform), "reasoning_style", "code",
|
||
)
|
||
except Exception:
|
||
_reasoning_style = "code"
|
||
_quote = self._REASONING_QUOTE_STYLES.get(_reasoning_style)
|
||
if _quote:
|
||
header, prefix, empty = _quote
|
||
_quoted = "\n".join(f"{prefix}{ln}" if ln else empty for ln in display_reasoning.splitlines())
|
||
return f"{header}\n{_quoted}\n\n{response}"
|
||
# Escape ``` inside reasoning so inner fences don't break the outer code block.
|
||
display_reasoning = escape_code_fences_for_display(display_reasoning)
|
||
return f"💭 **Reasoning:**\n```\n{display_reasoning}\n```\n\n{response}"
|
||
|
||
def _hmwa_runtime_footer_line(self, agent_result, source, _turn_seconds):
|
||
"""Runtime-metadata footer for the FINAL message of the turn; off by default
|
||
(display.runtime_footer.enabled=false)."""
|
||
from gateway.run import _load_gateway_config, _platform_config_key, _terminal_scope_cwd
|
||
try:
|
||
from gateway.runtime_footer import build_footer_line as _bfl
|
||
return _bfl(
|
||
user_config=_load_gateway_config(),
|
||
platform_key=_platform_config_key(source.platform), model=agent_result.get("model"),
|
||
context_tokens=agent_result.get("last_prompt_tokens", 0) or 0,
|
||
context_length=agent_result.get("context_length") or None,
|
||
cwd=_terminal_scope_cwd(""), turn_seconds=_turn_seconds,
|
||
)
|
||
except Exception as _footer_err:
|
||
logger.debug("runtime_footer build failed: %s", _footer_err)
|
||
return ""
|
||
|
||
async def _hmwa_post_turn_hooks(self, hook_ctx, agent_result, response):
|
||
"""agent:end hook, process-watcher scheduling, and watch-notification drain."""
|
||
await self.hooks.emit("agent:end", {
|
||
**hook_ctx, "response": (response or "")[:500], "model": agent_result.get("model", ""),
|
||
"provider": agent_result.get("provider", ""),
|
||
})
|
||
|
||
# Pending process watchers (check_interval on background processes)
|
||
try:
|
||
from tools.process_registry import process_registry
|
||
# Detach the batch atomically (reassign, not clear()) so concurrent appends aren't dropped.
|
||
watchers = process_registry.pending_watchers
|
||
process_registry.pending_watchers = []
|
||
for i, watcher in enumerate(watchers):
|
||
asyncio.create_task(self._run_process_watcher(watcher))
|
||
if i % 100 == 99:
|
||
await asyncio.sleep(0)
|
||
except Exception as e:
|
||
logger.error("Process watcher setup error: %s", e)
|
||
|
||
# Drain watch notifications that arrived during the run; the queue also carries process /
|
||
# async-delegation completions owned elsewhere — inject only watch-type events.
|
||
try:
|
||
from tools.process_registry import process_registry as _pr
|
||
await self._drain_watch_notifications(_pr.completion_queue)
|
||
except Exception as e:
|
||
logger.debug("Watch queue drain error: %s", e)
|
||
|
||
def _hmwa_classify_turn_failure(self, agent_result, history, session_entry):
|
||
"""Classify a finished turn for transcript persistence. Returns
|
||
``(agent_failed_early, hidden_reasoning_incomplete, is_context_overflow_failure)``.
|
||
|
||
Context-overflow failures must NOT persist the user message (session would grow and
|
||
reproduce the failure forever); transient failures (429/timeout/5xx) DO."""
|
||
from gateway.run import _is_gateway_hidden_reasoning_incomplete_turn
|
||
# Save the full conversation to the transcript, including tool calls. This preserves the complete
|
||
# agent loop (tool_calls, tool results, intermediate reasoning) so sessions can be resumed with full
|
||
# context and transcripts are useful for debugging and training data. IMPORTANT: For
|
||
# context-overflow failures (compression exhausted, generic 400 on large sessions) we must NOT
|
||
# persist the user's message — doing so would grow the session further and cause the same failure on
|
||
# the next attempt, an infinite loop. (#1630, #9893) Transient failures (429, timeout, connection
|
||
# error, provider 5xx) are different: the session is not oversized, and silently dropping the user
|
||
# message causes severe context loss on retry — the agent forgets what was just asked. Persist the
|
||
# user turn so the conversation is preserved. (#7100)
|
||
agent_failed_early = bool(agent_result.get("failed"))
|
||
hidden_reasoning_incomplete = _is_gateway_hidden_reasoning_incomplete_turn(agent_result)
|
||
is_context_overflow_failure = is_context_overflow_failure_result(agent_result, len(history))
|
||
if is_context_overflow_failure:
|
||
logger.info(
|
||
"Skipping transcript persistence for context-overflow "
|
||
"failure in session %s to prevent session growth loop.", session_entry.session_id,
|
||
)
|
||
elif agent_failed_early:
|
||
logger.info(
|
||
"Transient agent failure in session %s — persisting user "
|
||
"message so conversation context is preserved on retry.", session_entry.session_id,
|
||
)
|
||
elif hidden_reasoning_incomplete:
|
||
logger.warning(
|
||
"Suppressing hidden-reasoning-only incomplete gateway turn for session %s: %s",
|
||
session_entry.session_id, agent_result.get("error", "processing incomplete"),
|
||
)
|
||
return agent_failed_early, hidden_reasoning_incomplete, is_context_overflow_failure
|
||
|
||
async def _hmwa_compression_exhaustion_reset(
|
||
self, agent_result, response, session_entry, session_key, source,
|
||
):
|
||
"""Auto-reset a permanently oversized session so the next message starts fresh instead of
|
||
replaying the oversized context forever. Never on a lock-contended defer — that is the
|
||
OPPOSITE case (a concurrent path holds the lock and is shrinking it). Returns
|
||
``(response, session_entry)``."""
|
||
# When compression is exhausted, the session is permanently too large to process. (#9893) Never wipe
|
||
# the session for that — retry-next-message semantics apply (#69870 lock-skip consumer; salvaged
|
||
# from #49874).
|
||
if agent_result.get("compression_deferred"):
|
||
logger.info(
|
||
"Compression deferred for session %s — the compression "
|
||
"lock is held by a concurrent compressor. Keeping the "
|
||
"session intact; the next message retries normally.",
|
||
session_entry.session_id if session_entry else "?",
|
||
)
|
||
elif agent_result.get("compression_exhausted") and session_entry and session_key:
|
||
logger.info("Auto-resetting session %s after compression exhaustion.", session_entry.session_id)
|
||
new_entry = await self.async_session_store.reset_session(session_key)
|
||
self._evict_cached_agent(session_key)
|
||
# Conversation boundary: the funnel clears every conversation-scoped per-session dict.
|
||
self._clear_conversation_scope(session_key, reason="compression_exhausted_reset")
|
||
if new_entry is not None:
|
||
# Re-point the Telegram topic binding at the fresh session, or the binding-heal walk
|
||
# switches the next message back onto the bloated child and re-triggers exhaustion
|
||
# forever. No-op on non-topic lanes.
|
||
# Compression rotated session_entry.session_id to the oversized compressed child earlier
|
||
# this turn (the agent-result sync above), and that _sync also rewrote the (chat_id,
|
||
# thread_id) -> bloated-child binding. reset_session swaps in a clean, parentless session,
|
||
# but without re-syncing the binding the next inbound message in this topic gets
|
||
# switch_session'd back onto the bloated child by the binding-heal walk, reloads the
|
||
# oversized transcript, and re-triggers compression exhaustion forever (#35809 — regression
|
||
# of the #9893/#10063 auto-reset).
|
||
session_entry = new_entry
|
||
await asyncio.to_thread(
|
||
self._sync_telegram_topic_binding, source, session_entry, reason="compression-exhausted-reset",
|
||
)
|
||
response = (response or "") + (
|
||
"\n\n🔄 Session auto-reset — the conversation exceeded the maximum context size and "
|
||
"could not be compressed further. Your next message will start a fresh session."
|
||
)
|
||
return response, session_entry
|
||
|
||
@staticmethod
|
||
def _hmwa_user_transcript_entry(event, prepared, ts):
|
||
"""Transcript row for the inbound user turn (clean text + event time when captured)."""
|
||
# Transient failure (429/timeout/5xx): persist only the user message so the next message can load a
|
||
# transcript that reflects what was said. Skip the assistant error text since it's a
|
||
# gateway-generated hint, not model output. Hidden- reasoning-only incomplete turns follow the same
|
||
# persistence rule so peer-agent channels don't ingest them as completed assistant turns. (#7100,
|
||
# #51628)
|
||
_user_entry = {
|
||
"role": "user",
|
||
"content": (
|
||
prepared.persist_user_message if prepared.persist_user_message is not None
|
||
else prepared.message_text
|
||
),
|
||
"timestamp": prepared.persist_user_timestamp if prepared.persist_user_timestamp is not None else ts,
|
||
}
|
||
if prepared.persist_user_display_kind:
|
||
_user_entry["display_kind"] = prepared.persist_user_display_kind
|
||
if prepared.persistence_owner:
|
||
_user_entry["display_metadata"] = {"gateway_input_owner": prepared.persistence_owner}
|
||
if getattr(event, "message_id", None):
|
||
_user_entry["message_id"] = str(event.message_id)
|
||
return _user_entry
|
||
|
||
async def _hmwa_persist_turn_transcript(
|
||
self, *, event, source, session_entry, session_key, agent_result, agent_messages,
|
||
prepared, response, agent_failed_early, hidden_reasoning_incomplete, is_context_overflow_failure,
|
||
):
|
||
"""Persist this turn to the transcript (session_meta on first turn, user-only on transient
|
||
failure, nothing on context overflow), update last_prompt_tokens, and re-baseline the
|
||
cached agent's message count."""
|
||
from gateway.run import _resolve_gateway_model
|
||
ts = time.time() # Unix epoch float — consistent with DB storage
|
||
store = self.async_session_store
|
||
sid = session_entry.session_id
|
||
history = prepared.history
|
||
# The agent already persisted this turn's rows (codex app-server reports agent_persisted=True
|
||
# too); skip the DB write. Default = a session DB exists; non-persisting runtimes pass False.
|
||
# The agent already persisted these messages to SQLite via _flush_messages_to_session_db(), so skip
|
||
# the DB write here to prevent the duplicate-write bug (#860 / #42039). This holds for the codex
|
||
# app-server runtime too: although it early-returns and bypasses conversation_loop's per-step
|
||
# flushes, it flushes its own projected assistant/tool messages before returning and reports
|
||
# agent_persisted=True (see agent/codex_runtime.py). Reading the flag (default = self._session_db is
|
||
# not None) keeps the persistence contract explicit and lets any future non-persisting runtime opt
|
||
# into a gateway-side write by returning False.
|
||
agent_persisted = agent_result.get("agent_persisted", self._session_db is not None)
|
||
_user_row = self._hmwa_user_transcript_entry(event, prepared, ts)
|
||
|
||
if is_context_overflow_failure:
|
||
pass # Skip all transcript writes — don't grow a broken session
|
||
else:
|
||
if not history:
|
||
# Fresh session: the tool definitions (as sent in the API request) make the transcript
|
||
# self-describing.
|
||
await store.append_to_transcript(sid, {
|
||
"role": "session_meta",
|
||
"tools": agent_result.get("tools", []) or [],
|
||
"model": _resolve_gateway_model(),
|
||
"platform": source.platform.value if source.platform else "",
|
||
"timestamp": ts,
|
||
})
|
||
if agent_failed_early or hidden_reasoning_incomplete:
|
||
# Transient failure / hidden-reasoning incomplete: persist only the user message (the
|
||
# assistant error text is a gateway hint, not model output). Dedupe on platform
|
||
# message_id (Telegram retries after transient failures).
|
||
if event.message_id and await store.has_platform_message_id(sid, str(event.message_id)):
|
||
logger.info(
|
||
"Skipping duplicate user turn (message_id=%s) in session %s",
|
||
event.message_id, sid,
|
||
)
|
||
else:
|
||
await store.append_to_transcript(sid, _user_row, skip_db=agent_persisted)
|
||
else:
|
||
# Only the NEW messages: history_offset (what the agent saw), not len(history), which
|
||
# counts session_meta entries stripped before the agent saw them.
|
||
history_len = agent_result.get("history_offset", len(history))
|
||
new_messages = agent_messages[history_len:] if len(agent_messages) > history_len else []
|
||
if not new_messages:
|
||
# Edge case: fall back to simple user/assistant rows.
|
||
await store.append_to_transcript(sid, _user_row, skip_db=agent_persisted)
|
||
if response:
|
||
await store.append_to_transcript(
|
||
sid, {"role": "assistant", "content": response, "timestamp": ts},
|
||
skip_db=agent_persisted,
|
||
)
|
||
else:
|
||
# Attach the inbound platform message_id to the first user entry so platform-level
|
||
# quote-resolution (e.g. Yuanbao) can find earlier @bot messages by original id.
|
||
_user_msg_id_attached = False
|
||
for msg in new_messages:
|
||
if msg.get("role") == "system":
|
||
continue # rebuilt each run
|
||
entry = {**msg, "timestamp": ts}
|
||
if (
|
||
not _user_msg_id_attached
|
||
and msg.get("role") == "user"
|
||
and event.message_id
|
||
and "message_id" not in entry
|
||
):
|
||
entry["message_id"] = str(event.message_id)
|
||
_user_msg_id_attached = True
|
||
await store.append_to_transcript(sid, entry, skip_db=agent_persisted)
|
||
|
||
# The agent persists token counts/model itself; keep only last_prompt_tokens for hygiene.
|
||
await store.update_session(
|
||
session_entry.session_key, last_prompt_tokens=agent_result.get("last_prompt_tokens", 0),
|
||
touch_activity=not bool(getattr(event, "internal", False)),
|
||
)
|
||
|
||
# Re-baseline the cached agent's message_count now that ALL of this turn's writes are done:
|
||
# the coherence guard snapshots at agent-BUILD time, so our own writes would otherwise
|
||
# trigger a rebuild next turn (destroying prompt caching).
|
||
await self._refresh_agent_cache_message_count(session_key, sid)
|
||
|
||
async def _hmwa_deliver_turn_response(
|
||
self, event, source, session_entry, session_key, run_generation,
|
||
agent_result, agent_messages, response, _footer_line, _intentional_silence,
|
||
):
|
||
"""Final delivery decisions: intentional silence, voice reply, streamed-turn media/footer.
|
||
Returns the text for the adapter to send, or ``None`` when already delivered."""
|
||
# Intentional silence is a delivery decision: the [SILENT] turn stays persisted (alternation).
|
||
if _intentional_silence:
|
||
logger.info("Suppressing intentional silence marker for session %s", session_entry.session_id)
|
||
response = ""
|
||
|
||
adapter = self._adapter_for_source(source)
|
||
# Auto voice reply (TTS audio before the text) unless streaming TTS already delivered audio.
|
||
_streaming_tts_done = adapter is not None and bool(
|
||
getattr(adapter, "_streaming_tts_turn_completed", lambda *_a, **_k: False)(session_key, run_generation)
|
||
)
|
||
if not _streaming_tts_done and self._should_send_voice_reply(
|
||
event, response, agent_messages, already_sent=bool(agent_result.get("already_sent")),
|
||
):
|
||
await self._send_voice_reply(event, response)
|
||
|
||
# Streamed responses still need MEDIA: files delivered (chunks carry the tags verbatim). Never
|
||
# skip when the agent failed: the error text is new content streaming didn't show.
|
||
if agent_result.get("already_sent") and not agent_result.get("failed"):
|
||
if response and adapter:
|
||
await self._deliver_media_from_response(response, event, adapter)
|
||
# Streaming delivered the body, but the footer was held back (`not already_sent` gate).
|
||
if _footer_line and adapter:
|
||
try:
|
||
await adapter.send(source.chat_id, _footer_line, metadata=self._event_thread_metadata(event, source))
|
||
except Exception as _e:
|
||
logger.debug("trailing footer send failed: %s", _e)
|
||
# Return None so the body isn't sent twice; stash the delivered text on the event for the
|
||
# /loop and /goal hooks that read the return value.
|
||
with suppress(Exception):
|
||
event._streamed_final_response = str(response or "")
|
||
return None
|
||
|
||
return response
|
||
|
||
_STATUS_HINTS = {
|
||
401: " Check your API key or run `claude /login` to refresh OAuth credentials.",
|
||
402: " Your API balance or quota is exhausted. Check your provider dashboard.",
|
||
529: " The API is temporarily overloaded. Please try again shortly.",
|
||
}
|
||
|
||
async def _hmwa_agent_error_reply(self, e, event, source, session_entry, session_key, prepared):
|
||
"""``except Exception`` body of the agent turn: stop typing, log, persist the inbound user
|
||
turn once, and build the sanitized user-facing error reply."""
|
||
# Retain Slack thread/workspace routing so a failed turn cannot leave its status visible.
|
||
await self._hmwa_stop_typing_for_turn(event, source)
|
||
logger.exception("Agent error in session %s", session_key)
|
||
# Replay can coalesce inputs; only this input's durable marker establishes ownership.
|
||
try:
|
||
if prepared.message_text is not None and session_entry is not None:
|
||
_owned = await self.async_session_store.has_input_owner(
|
||
prepared.persistence_session_id, prepared.persistence_owner,
|
||
)
|
||
if not _owned:
|
||
await self.async_session_store.append_to_transcript(
|
||
session_entry.session_id, self._hmwa_user_transcript_entry(event, prepared, time.time()),
|
||
)
|
||
except Exception:
|
||
logger.debug("Failed to persist inbound user message after agent exception", exc_info=True)
|
||
# Never expose raw exception types/messages to end users (info-leakage risk).
|
||
status_code = getattr(e, "status_code", None)
|
||
status_hint = self._STATUS_HINTS.get(status_code, "")
|
||
if status_code == 429:
|
||
# Plan usage limit (resets on a schedule) vs a transient rate limit
|
||
_err_json = {}
|
||
with suppress(Exception):
|
||
_err_json = e.response.json().get("error", {})
|
||
if not isinstance(_err_json, dict):
|
||
_err_json = {}
|
||
_resets_in = _err_json.get("resets_in_seconds")
|
||
if _err_json.get("type") != "usage_limit_reached":
|
||
status_hint = " You are being rate-limited. Please wait a moment and try again."
|
||
elif _resets_in and _resets_in > 0:
|
||
import math
|
||
status_hint = f" Your plan's usage limit has been reached. It resets in ~{math.ceil(_resets_in / 3600)}h."
|
||
else:
|
||
status_hint = " Your plan's usage limit has been reached. Please wait until it resets."
|
||
elif status_code in {400, 500}:
|
||
# 400/500 on a large session: context overflow / payload too large.
|
||
if len(prepared.history) > 50:
|
||
return (
|
||
"⚠️ Session too large for the model's context window.\nUse /compact to "
|
||
"compress the conversation, or /reset to start fresh."
|
||
)
|
||
elif status_code == 400:
|
||
status_hint = " The request was rejected by the API."
|
||
return (
|
||
f"Sorry, I encountered an unexpected error.{status_hint}\n"
|
||
"Try again or use /reset to start a fresh session."
|
||
)
|
||
|
||
def _hmwa_discard_stale_result(self, source, _quick_key, run_generation):
|
||
"""A newer run generation superseded this turn: drop its deferred post-delivery callback."""
|
||
logger.info(
|
||
"Discarding stale agent result for %s — generation %d is no longer current",
|
||
_quick_key or "?", run_generation,
|
||
)
|
||
self._pop_post_delivery_callback(self._adapter_for_source(source), _quick_key, run_generation)
|
||
|
||
@dataclasses.dataclass
|
||
class _PreparedTurn:
|
||
"""Inputs to the agent run assembled by ``_hmwa_prepare_turn``."""
|
||
|
||
history: Any
|
||
context_prompt: str
|
||
message_text: Any
|
||
persist_user_message: Any
|
||
persist_user_timestamp: Any
|
||
persist_user_display_kind: Optional[str]
|
||
persistence_session_id: Optional[str] = None
|
||
persistence_owner: Optional[str] = None
|
||
|
||
async def _hmwa_prepare_turn(self, event, source, session_entry, session_key, _quick_key, run_generation):
|
||
"""Everything between session resolution and the agent run: session open, task-local env,
|
||
context prompt, sidecar notes, turn lease, transcript load + hygiene, inbound text. Returns
|
||
``(_PreparedTurn, env_tokens)``; a ``str`` first element is a reply to send instead of
|
||
running (history unreadable); ``None`` drops the turn (inbound text rejected)."""
|
||
from gateway.run import _load_gateway_config
|
||
_was_auto_reset, _is_new_session = await self._hmwa_open_session(session_entry, session_key, source)
|
||
context = build_session_context(source, self.config, session_entry)
|
||
# Session context variables for tools (task-local, concurrency-safe)
|
||
_session_env_tokens = self._set_session_env(context)
|
||
# Self-injected turns (MessageEvent(internal=True)) persist with a DB-only display_kind so
|
||
# UIs render timeline notices, not user bubbles; role/content untouched.
|
||
persist_user_display_kind = "internal_notification" if getattr(event, "internal", False) else None
|
||
_redact_pii = False # privacy.redact_pii, re-read per message
|
||
with suppress(Exception):
|
||
_redact_pii = bool((_load_gateway_config().get("privacy") or {}).get("redact_pii", False))
|
||
|
||
# The context prompt render is pinned per session, keyed by a hash of the renderer inputs, so
|
||
# the system prompt cannot drift turn-over-turn; a miss (thread rename, /sethome) re-renders.
|
||
context_prompt = self._pinned_session_context_prompt(context, _redact_pii, session_key)
|
||
|
||
# Per-turn notes ride the user message via the api_content sidecar, NOT context_prompt
|
||
# (appending to the ephemeral system prompt forced a full agent rebuild).
|
||
turn_sidecar_notes: List[str] = []
|
||
if _was_auto_reset:
|
||
await self._hmwa_deliver_auto_reset_notice(session_entry, source, turn_sidecar_notes)
|
||
|
||
# Auto-load bound skill(s) only on NEW sessions; ongoing ones carry the content in history.
|
||
_auto = getattr(event, "auto_skill", None)
|
||
if _is_new_session and _auto:
|
||
self._hmwa_auto_load_skills(event, _auto, _quick_key, session_key)
|
||
|
||
await self._hmwa_acquire_turn_lease(_quick_key, run_generation, session_entry, _session_env_tokens)
|
||
|
||
# A turn becomes durable recovery work only after it owns the per-session lease; marking
|
||
# earlier would falsely recover a message that never began processing.
|
||
await self._mark_durable_active_turn(event, session_entry.session_key)
|
||
|
||
# An unreadable store is not an empty conversation: stop before the agent invents continuity
|
||
# from []. Restore task-local context here (before the broad cleanup finally).
|
||
try:
|
||
history = await self.async_session_store.load_transcript(session_entry.session_id)
|
||
history = await self._hmwa_run_session_hygiene(
|
||
event, source, session_entry, session_key, history, _quick_key, run_generation,
|
||
)
|
||
except TranscriptReadError:
|
||
self._clear_session_env(_session_env_tokens)
|
||
return (
|
||
"⚠️ This session's history is temporarily unavailable, so this message was not "
|
||
"processed. Ask the operator to inspect state.db, then resend after it is healthy. "
|
||
"Use /reset only if you intentionally want to start a new conversation."
|
||
), _session_env_tokens
|
||
|
||
await self._hmwa_first_contact_notes(source, history, turn_sidecar_notes)
|
||
|
||
# Voice channel state rides the user message ONLY when changed (in the system prompt it
|
||
# forced a rebuild + prompt-cache re-key per message).
|
||
_vc_note = self._voice_channel_sidecar_note(event, source, session_key)
|
||
if _vc_note:
|
||
turn_sidecar_notes.append(_vc_note)
|
||
|
||
# Auto-analyze user images so the model gets a description plus the local path.
|
||
message_text = await self._prepare_profile_scoped_inbound_message_text(
|
||
event=event, source=source, history=history, session_key=session_key,
|
||
)
|
||
if message_text is None:
|
||
return None, _session_env_tokens
|
||
|
||
message_text, persist_user_message, persist_user_timestamp = (
|
||
self._hmwa_apply_message_timestamp(event, message_text)
|
||
)
|
||
|
||
# Stage the notes (one-shot; consumed in run_sync) AFTER the early-out so an aborted turn
|
||
# cannot leak them into the next turn.
|
||
if turn_sidecar_notes and session_key:
|
||
self._set_pending_turn_sidecar_notes(session_key, turn_sidecar_notes)
|
||
|
||
# Bind this run generation to the adapter so deferred post-delivery callbacks are released
|
||
# by the run that registered them.
|
||
self._bind_adapter_run_generation(self._adapter_for_source(source), session_key, run_generation)
|
||
# Delivery IDs are only unique in their transport namespace. Keyless turns
|
||
# need their own identity, even when another process writes to this session.
|
||
import uuid
|
||
namespace = [source.platform.value, source.profile, source.scope_id,
|
||
source.chat_id, source.thread_id, str(event.message_id)]
|
||
owner = (str(uuid.uuid5(uuid.NAMESPACE_URL, json.dumps(namespace)))
|
||
if event.message_id else str(uuid.uuid4()))
|
||
return self._PreparedTurn(
|
||
history, context_prompt, message_text, persist_user_message, persist_user_timestamp,
|
||
persist_user_display_kind, session_entry.session_id, owner,
|
||
), _session_env_tokens
|
||
|
||
async def _handle_message_with_agent(self, event, source, _quick_key: str, run_generation: int):
|
||
"""Inner handler that runs under the _running_agents sentinel guard."""
|
||
_msg_start_time = time.time()
|
||
_platform_name = source.platform.value if hasattr(source.platform, "value") else str(source.platform)
|
||
logger.info(
|
||
"inbound message: platform=%s user=%s chat=%s msg=%r reply_to_id=%s reply_to_text=%r",
|
||
_platform_name, source.user_name or source.user_id or "unknown",
|
||
source.chat_id or "unknown", (event.text or "")[:80].replace("\n", " "),
|
||
getattr(event, "reply_to_message_id", None),
|
||
(getattr(event, "reply_to_text", None) or "")[:80].replace("\n", " "),
|
||
)
|
||
|
||
resolved = await self._hmwa_resolve_session(event, source)
|
||
if resolved is None:
|
||
return
|
||
source, session_entry, session_key = resolved
|
||
prepared, _session_env_tokens = await self._hmwa_prepare_turn(
|
||
event, source, session_entry, session_key, _quick_key, run_generation,
|
||
)
|
||
if not isinstance(prepared, self._PreparedTurn):
|
||
return prepared
|
||
history, message_text = prepared.history, prepared.message_text
|
||
|
||
try:
|
||
hook_ctx = {
|
||
"platform": source.platform.value if source.platform else "",
|
||
"user_id": source.user_id,
|
||
"chat_id": source.chat_id or "",
|
||
"thread_id": str(source.thread_id) if getattr(source, "thread_id", None) else "",
|
||
"chat_type": getattr(source, "chat_type", "") or "",
|
||
"session_id": session_entry.session_id,
|
||
"message": message_text[:500],
|
||
}
|
||
await self.hooks.emit("agent:start", hook_ctx)
|
||
|
||
# Capture the launch session id so post-run compression publication is identity-guarded
|
||
# (a /new may move session_entry.session_id while the old run is still unwinding).
|
||
from gateway.run_heartbeat_acceptance import heartbeat_owner_is_current
|
||
if not heartbeat_owner_is_current(self, event, session_key):
|
||
return
|
||
_run_start_session_id = session_entry.session_id
|
||
_turn_started_monotonic = time.monotonic()
|
||
# Admission/typing is not execution. All routing, authorization and
|
||
# turn preparation gates have passed when the agent runner is entered.
|
||
event._heartbeat_execution_started = True
|
||
agent_result = await self._run_agent(
|
||
message=message_text, context_prompt=prepared.context_prompt, history=history, source=source,
|
||
session_id=_run_start_session_id, session_key=session_key,
|
||
run_generation=run_generation, event_message_id=self._reply_anchor_for_event(event),
|
||
inbound_message_id=str(event.message_id) if event.message_id else None,
|
||
channel_prompt=event.channel_prompt, moa_config=getattr(event, "_moa_config", None),
|
||
persist_user_message=prepared.persist_user_message,
|
||
persist_user_timestamp=prepared.persist_user_timestamp,
|
||
persist_user_display_kind=prepared.persist_user_display_kind,
|
||
persist_user_display_metadata={"gateway_input_owner": prepared.persistence_owner},
|
||
message_type=event.message_type,
|
||
)
|
||
_turn_seconds = time.monotonic() - _turn_started_monotonic
|
||
|
||
# A queued (/queue) chain answered the LAST message of the chain, so the outer final
|
||
# send (bracketed by the adapter against this event) must be ledgered under that
|
||
# message's id or it collides with an earlier turn's row carrying the same text. Reply
|
||
# routing is untouched: the anchor still comes from this event.
|
||
if isinstance(agent_result, dict):
|
||
_terminal_inbound = agent_result.get("queued_terminal_inbound_id")
|
||
if _terminal_inbound:
|
||
event.ledger_message_id = str(_terminal_inbound)
|
||
|
||
await self._hmwa_stop_typing_for_turn(event, source)
|
||
|
||
if not self._is_session_run_current(_quick_key, run_generation):
|
||
self._hmwa_discard_stale_result(source, _quick_key, run_generation)
|
||
return None
|
||
|
||
response, _intentional_silence, agent_messages = await self._hmwa_shape_agent_response(
|
||
agent_result, source, history, session_entry, session_key,
|
||
_quick_key, run_generation, _run_start_session_id, _platform_name, _msg_start_time,
|
||
)
|
||
response = self._hmwa_prepend_reasoning(agent_result, response, source, _intentional_silence)
|
||
_footer_line = self._hmwa_runtime_footer_line(agent_result, source, _turn_seconds)
|
||
# Streaming already delivered the body: the footer goes out as a trailing send instead.
|
||
if _footer_line and response and not agent_result.get("already_sent") and not _intentional_silence:
|
||
response = f"{response}\n\n{_footer_line}"
|
||
await self._hmwa_post_turn_hooks(hook_ctx, agent_result, response)
|
||
|
||
agent_failed_early, hidden_reasoning_incomplete, is_context_overflow_failure = (
|
||
self._hmwa_classify_turn_failure(agent_result, history, session_entry)
|
||
)
|
||
response, session_entry = await self._hmwa_compression_exhaustion_reset(
|
||
agent_result, response, session_entry, session_key, source,
|
||
)
|
||
await self._hmwa_persist_turn_transcript(
|
||
event=event, source=source, session_entry=session_entry, session_key=session_key,
|
||
agent_result=agent_result, agent_messages=agent_messages, prepared=prepared,
|
||
response=response, agent_failed_early=agent_failed_early,
|
||
hidden_reasoning_incomplete=hidden_reasoning_incomplete,
|
||
is_context_overflow_failure=is_context_overflow_failure,
|
||
)
|
||
return await self._hmwa_deliver_turn_response(
|
||
event, source, session_entry, session_key, run_generation,
|
||
agent_result, agent_messages, response, _footer_line, _intentional_silence,
|
||
)
|
||
|
||
except Exception as e:
|
||
return await self._hmwa_agent_error_reply(e, event, source, session_entry, session_key, prepared)
|
||
finally:
|
||
# Restore session context variables to their pre-handler state
|
||
self._clear_session_env(_session_env_tokens)
|
||
|
||
def _profile_scope_for_source(self, source: SessionSource):
|
||
"""``_profile_runtime_scope`` for ``source``'s profile when multiplexing, else a no-op context.
|
||
|
||
Under multiplexing config/skills/memory resolve to the source profile's home AND credentials
|
||
come from its secret scope (never process-global ``os.environ``)."""
|
||
from gateway.run import _profile_runtime_scope
|
||
if getattr(getattr(self, "config", None), "multiplex_profiles", False):
|
||
return _profile_runtime_scope(self._resolve_profile_home_for_source(source))
|
||
return nullcontext()
|
||
|
||
def _reset_notice_session_info(self, source: SessionSource) -> str:
|
||
"""Session-info block for the auto-reset notice, resolved inside the profile serving ``source``.
|
||
|
||
Call via ``asyncio.to_thread``: resolution can block (credential refresh, context-length
|
||
probes), and the scope is entered here so contextvars behave in the worker thread."""
|
||
with self._profile_scope_for_source(source):
|
||
return self._format_session_info()
|
||
|
||
def _format_session_info(self) -> str:
|
||
"""Model / provider / context-length / endpoint block so users can spot bad context detection."""
|
||
from gateway.run import _resolve_gateway_model_context
|
||
resolved = _resolve_gateway_model_context()
|
||
context_length = resolved.context_length
|
||
ctx_source = {
|
||
"config": "config",
|
||
"default": "default — set model.context_length in config to override",
|
||
}.get(resolved.context_source, "detected")
|
||
ctx_display = (
|
||
f"{context_length / 1_000_000:.1f}M" if context_length >= 1_000_000
|
||
else f"{context_length // 1_000}K" if context_length >= 1_000 else str(context_length)
|
||
)
|
||
lines = [
|
||
f"◆ Model: `{resolved.model}`",
|
||
f"◆ Provider: {resolved.provider or 'openrouter'}",
|
||
f"◆ Context: {ctx_display} tokens ({ctx_source})",
|
||
]
|
||
base_url = resolved.base_url
|
||
if base_url and base_url_hostname(base_url) in ("localhost", "127.0.0.1", "0.0.0.0"):
|
||
lines.append(f"◆ Endpoint: {base_url}")
|
||
return "\n".join(lines)
|
||
|
||
async def _run_background_task(
|
||
self, prompt: str, source: "SessionSource", task_id: str,
|
||
event_message_id: Optional[str] = None, media_urls: Optional[List[str]] = None,
|
||
media_types: Optional[List[str]] = None,
|
||
) -> None:
|
||
"""Profile-scoping wrapper around the background agent task (mirrors ``_run_agent``)."""
|
||
with self._profile_scope_for_source(source):
|
||
return await self._run_background_task_inner(
|
||
prompt, source, task_id, event_message_id, media_urls, media_types,
|
||
)
|
||
|
||
def _resolve_enabled_toolsets_for_source(
|
||
self, user_config: dict, source: "SessionSource", platform_key: str,
|
||
) -> list:
|
||
"""Enabled toolsets for an agent run, honoring an adapter ``toolsets_for_source()`` override
|
||
validated through the SAME ``_get_platform_tools`` path (unknown / platform-restricted
|
||
toolsets dropped, not trusted)."""
|
||
from hermes_cli.tools_config import _get_platform_tools
|
||
try:
|
||
adapter = self._adapter_for_source(source)
|
||
override = adapter.toolsets_for_source(source) if adapter is not None else None
|
||
except Exception:
|
||
override = None
|
||
if override and isinstance(override, list):
|
||
pts = dict(user_config.get("platform_toolsets") or {})
|
||
pts[platform_key] = [str(x) for x in override]
|
||
user_config = {**user_config, "platform_toolsets": pts}
|
||
return sorted(_get_platform_tools(user_config, platform_key))
|
||
|
||
def _resolve_turn_toolsets(self, user_config: dict, source: "SessionSource", platform_key: str):
|
||
"""``(enabled_toolsets, disabled_toolsets)`` for an agent run on ``source``."""
|
||
from agent.skill_utils import parse_config_string_list
|
||
enabled = self._resolve_enabled_toolsets_for_source(user_config, source, platform_key)
|
||
disabled = parse_config_string_list((user_config.get("agent") or {}).get("disabled_toolsets")) or None
|
||
return enabled, disabled
|
||
|
||
async def _run_background_task_inner(
|
||
self, prompt: str, source: "SessionSource", task_id: str,
|
||
event_message_id: Optional[str] = None, media_urls: Optional[List[str]] = None,
|
||
media_types: Optional[List[str]] = None,
|
||
) -> None:
|
||
"""Execute a background agent task and deliver the result to the chat."""
|
||
from gateway.run import (
|
||
_checkpoint_agent_kwargs, _current_max_iterations, _load_gateway_config,
|
||
_platform_config_key,
|
||
)
|
||
from run_agent import AIAgent
|
||
media_urls = media_urls or []
|
||
media_types = media_types or []
|
||
adapter = self._adapter_for_source(source)
|
||
if not adapter:
|
||
logger.warning("No adapter for platform %s in background task %s", source.platform, task_id)
|
||
return
|
||
_thread_metadata = self._thread_metadata_for_source(source, event_message_id)
|
||
|
||
try:
|
||
user_config = _load_gateway_config()
|
||
model, runtime_kwargs = self._resolve_session_agent_runtime(source=source, user_config=user_config)
|
||
if not runtime_kwargs.get("api_key"):
|
||
await adapter.send(
|
||
source.chat_id,
|
||
f"❌ Background task {task_id} failed: no provider credentials configured.",
|
||
metadata=_thread_metadata,
|
||
)
|
||
return
|
||
|
||
platform_key = _platform_config_key(source.platform)
|
||
enabled_toolsets, disabled_toolsets = self._resolve_turn_toolsets(user_config, source, platform_key)
|
||
pr = self._provider_routing
|
||
max_iterations = _current_max_iterations()
|
||
reasoning_config = self._resolve_session_reasoning_config(source=source, model=model)
|
||
self._reasoning_config = reasoning_config
|
||
self._service_tier = self._resolve_session_service_tier(source=source)
|
||
turn_route = self._resolve_turn_agent_config(prompt, model, runtime_kwargs)
|
||
|
||
# Enrich the prompt with image descriptions (same as the main flow).
|
||
enriched_prompt = prompt
|
||
image_paths = [
|
||
path for i, path in enumerate(media_urls)
|
||
if (media_types[i] if i < len(media_types) else "").startswith("image/")
|
||
]
|
||
if image_paths:
|
||
try:
|
||
enriched_prompt = await self._enrich_message_with_vision(prompt, image_paths)
|
||
except Exception as e:
|
||
logger.warning("Background task vision enrichment failed: %s", e)
|
||
|
||
def run_sync():
|
||
agent = AIAgent(
|
||
model=turn_route["model"],
|
||
**turn_route["runtime"],
|
||
**_checkpoint_agent_kwargs(user_config),
|
||
max_iterations=max_iterations,
|
||
quiet_mode=True,
|
||
verbose_logging=False,
|
||
enabled_toolsets=enabled_toolsets,
|
||
disabled_toolsets=disabled_toolsets,
|
||
reasoning_config=reasoning_config,
|
||
service_tier=self._service_tier,
|
||
request_overrides=turn_route.get("request_overrides"),
|
||
providers_allowed=pr.get("only"),
|
||
providers_ignored=pr.get("ignore"),
|
||
providers_order=pr.get("order"),
|
||
provider_sort=pr.get("sort"),
|
||
provider_require_parameters=pr.get("require_parameters", False),
|
||
provider_data_collection=pr.get("data_collection"),
|
||
session_id=task_id,
|
||
platform=platform_key,
|
||
**{k: getattr(source, k) for k in (
|
||
"user_id", "user_id_alt", "user_name", "chat_id", "chat_name", "chat_type", "thread_id",
|
||
)},
|
||
session_db=getattr(self._session_db, "_db", self._session_db),
|
||
# Reload from disk — do not reuse the startup snapshot.
|
||
# See #60955.
|
||
fallback_model=self._refresh_fallback_model(),
|
||
)
|
||
try:
|
||
return agent.run_conversation(user_message=enriched_prompt, task_id=task_id)
|
||
finally:
|
||
self._cleanup_agent_resources(agent)
|
||
|
||
result = await self._run_in_executor_with_context(run_sync)
|
||
|
||
response = result.get("final_response", "") if result else ""
|
||
if not response and result and result.get("error"):
|
||
response = f"Error: {result['error']}"
|
||
# Fresh conversation, so history_offset=0: every message in the run belongs to this turn.
|
||
if response:
|
||
response = repair_explicit_computer_use_media_paths(response, result.get("messages", []))
|
||
|
||
preview = prompt[:60] + ("..." if len(prompt) > 60 else "")
|
||
header = f'✅ Background task complete\nPrompt: "{preview}"\n\n'
|
||
images, media_files, text_content = [], [], ""
|
||
if response:
|
||
media_files, response = adapter.extract_media(response)
|
||
media_files = BasePlatformAdapter.filter_media_delivery_paths(media_files)
|
||
images, text_content = adapter.extract_images(response)
|
||
if text_content:
|
||
await adapter.send(chat_id=source.chat_id, content=header + text_content, metadata=_thread_metadata)
|
||
elif not images and not media_files:
|
||
await adapter.send(
|
||
chat_id=source.chat_id, content=header + "(No response generated)", metadata=_thread_metadata,
|
||
)
|
||
for image_url, alt_text in (images or []):
|
||
with suppress(Exception):
|
||
await adapter.send_image(
|
||
chat_id=source.chat_id, image_url=image_url, caption=alt_text, metadata=_thread_metadata,
|
||
)
|
||
# Route each media file by type (voice bubble / video / image / document), as the
|
||
# streaming + kanban paths do.
|
||
from gateway.platforms.base import should_send_media_as_audio as _should_send_media_as_audio
|
||
from gateway.run_notifications import _IMAGE_EXTS, _VIDEO_EXTS
|
||
for media_path, _is_voice in (media_files or []):
|
||
_ext = os.path.splitext(media_path)[1].lower()
|
||
with suppress(Exception):
|
||
if _should_send_media_as_audio(source.platform, _ext, _is_voice):
|
||
await adapter.send_voice(
|
||
chat_id=source.chat_id, audio_path=media_path, metadata=_thread_metadata,
|
||
is_voice=_is_voice,
|
||
)
|
||
else:
|
||
sender, key = (
|
||
(adapter.send_video, "video_path") if _ext in _VIDEO_EXTS
|
||
else (adapter.send_image_file, "image_path") if _ext in _IMAGE_EXTS
|
||
else (adapter.send_document, "file_path")
|
||
)
|
||
await sender(chat_id=source.chat_id, metadata=_thread_metadata, **{key: media_path})
|
||
|
||
except Exception as e:
|
||
logger.exception("Background task %s failed", task_id)
|
||
with suppress(Exception):
|
||
await adapter.send(
|
||
chat_id=source.chat_id, content=f"❌ Background task {task_id} failed: {e}",
|
||
metadata=_thread_metadata,
|
||
)
|
||
|
||
def _mcp_reload_refresh_cached_agents(self, multiplex: bool, profile) -> None:
|
||
"""Refresh cached agents so existing sessions see new MCP tools on their next turn without
|
||
a history-destroying ``/new``. Each agent keeps its build-time toolset selection EXACTLY: a
|
||
session built with restricted enabled_toolsets (e.g. ["safe"]) must NOT silently gain tools."""
|
||
try:
|
||
from tools.mcp_tool_agent import refresh_agent_mcp_tools
|
||
_cache = getattr(self, "_agent_cache", None)
|
||
_cache_lock = getattr(self, "_agent_cache_lock", None)
|
||
if _cache_lock is None or not _cache:
|
||
return
|
||
# Multiplex: only this profile's sessions (another profile's agent would get this registry).
|
||
_ns_prefix = _session_key_namespace(profile) + ":" if multiplex else None
|
||
with _cache_lock:
|
||
for _sess_key, _entry in list(_cache.items()):
|
||
if _ns_prefix and not str(_sess_key).startswith(_ns_prefix):
|
||
continue
|
||
_agent = _entry[0] if isinstance(_entry, tuple) else _entry
|
||
if _agent is not None:
|
||
refresh_agent_mcp_tools(_agent, quiet_mode=True)
|
||
except Exception as _exc:
|
||
logger.debug("Failed to update cached agent tools after MCP reload: %s", _exc)
|
||
|
||
async def _execute_mcp_reload(self, event: MessageEvent) -> str:
|
||
"""Disconnect, reconnect, and notify MCP tool changes (shared by button / text / no-confirm paths).
|
||
|
||
Under multiplex the reload runs inside the requesting profile's runtime scope (entered here
|
||
when the caller did not) and only that profile's servers are torn down and rediscovered.
|
||
|
||
See #95518.
|
||
"""
|
||
from gateway.run import _profile_runtime_scope
|
||
multiplex = bool(getattr(self.config, "multiplex_profiles", False))
|
||
if multiplex and not get_hermes_home_override():
|
||
profile_home = self._resolve_profile_home_for_source(event.source)
|
||
with _profile_runtime_scope(Path(profile_home)):
|
||
return await self._execute_mcp_reload(event)
|
||
try:
|
||
from tools.mcp_tool_lifecycle import shutdown_mcp_servers
|
||
from tools.mcp_tool_discovery import discover_mcp_tools
|
||
from tools.mcp_tool import _servers, _lock, _server_visible_in_scope
|
||
from tools.mcp_tool_agent import reprobe_tool_availability
|
||
from tools.registry import registry
|
||
|
||
reload_scope = registry.current_scope_key() if multiplex else None
|
||
|
||
def _scoped_server_names() -> set:
|
||
with _lock:
|
||
return {
|
||
name for name in _servers
|
||
if _server_visible_in_scope(name, reload_scope)
|
||
}
|
||
|
||
old_servers = _scoped_server_names()
|
||
await self._run_in_executor_with_context(lambda: shutdown_mcp_servers(scope=reload_scope))
|
||
# Explicit reload also re-probes tool availability (check_fn).
|
||
reprobe_tool_availability()
|
||
# Reconnect by discovering tools (reads config.yaml fresh).
|
||
new_tools = await self._run_in_executor_with_context(discover_mcp_tools)
|
||
|
||
connected_servers = _scoped_server_names()
|
||
if reload_scope is not None:
|
||
from tools.mcp_tool import _mcp_tool_server_names
|
||
with _lock:
|
||
new_tools = [n for n in new_tools if _mcp_tool_server_names.get(n) in connected_servers]
|
||
# (label, i18n key, names); i18n lines list reconnected first, the injected note added first.
|
||
changes = (
|
||
("Reconnected", "gateway.reload_mcp.reconnected", connected_servers & old_servers),
|
||
("Added", "gateway.reload_mcp.added", connected_servers - old_servers),
|
||
("Removed", "gateway.reload_mcp.removed", old_servers - connected_servers),
|
||
)
|
||
lines = [t("gateway.reload_mcp.header")] + [
|
||
t(key, names=", ".join(sorted(names))) for _label, key, names in changes if names
|
||
]
|
||
if not connected_servers:
|
||
lines.append(t("gateway.reload_mcp.none_connected"))
|
||
else:
|
||
lines.append(t("gateway.reload_mcp.tools_available", tools=len(new_tools), servers=len(connected_servers)))
|
||
|
||
self._mcp_reload_refresh_cached_agents(multiplex, event.source.profile)
|
||
|
||
# Append a note at the END of the history (preserves the prompt-cache prefix).
|
||
change_parts = [
|
||
f"{label} servers: {', '.join(sorted(names))}"
|
||
for label, _key, names in (changes[1], changes[2], changes[0]) if names
|
||
]
|
||
tool_summary = f"{len(new_tools)} MCP tool(s) now available" if new_tools else "No MCP tools available"
|
||
change_detail = ". ".join(change_parts) + ". " if change_parts else ""
|
||
reload_msg = {
|
||
"role": "user",
|
||
"content": f"[IMPORTANT: MCP servers have been reloaded. {change_detail}{tool_summary}. The tool list for this conversation has been updated accordingly.]",
|
||
}
|
||
with suppress(Exception): # Best-effort; don't fail the reload over a transcript write
|
||
session_entry = await self.async_session_store.get_or_create_session(event.source)
|
||
await self.async_session_store.append_to_transcript(session_entry.session_id, reload_msg)
|
||
|
||
return "\n".join(lines)
|
||
|
||
except Exception as e:
|
||
logger.warning("MCP reload failed: %s", e)
|
||
return t("gateway.reload_mcp.failed", error=e)
|
||
|
||
def _get_proxy_url(self) -> Optional[str]:
|
||
"""Proxy URL if proxy mode is configured (GATEWAY_PROXY_URL env wins over ``gateway.proxy_url``)."""
|
||
from gateway.run import _load_gateway_config
|
||
url = os.getenv("GATEWAY_PROXY_URL", "").strip()
|
||
if not url:
|
||
url = ((_load_gateway_config().get("gateway") or {}).get("proxy_url") or "").strip()
|
||
return url.rstrip("/") if url else None
|
||
|
||
def _build_stream_consumer_config(
|
||
self, source: "SessionSource", scfg: Any, adapter: Any, *, on_missing_cursor: str,
|
||
) -> "tuple[Any, Optional[Callable[[], None]]]":
|
||
"""Build the shared ``StreamConsumerConfig`` and optional Telegram pause-typing closure.
|
||
For non-editing adapters ``on_missing_cursor="fallback"`` streams with an empty cursor;
|
||
``"raise"`` raises ``RuntimeError`` so the caller skips streaming entirely."""
|
||
from gateway.stream_consumer import StreamConsumerConfig
|
||
_pause_typing_before_finalize = None
|
||
if source.platform == Platform.TELEGRAM and hasattr(adapter, "pause_typing_for_chat"):
|
||
def _pause_typing_before_finalize(_adapter=adapter, _chat_id=source.chat_id) -> None:
|
||
_adapter.pause_typing_for_chat(_chat_id)
|
||
# Non-editing platforms (QQ, WeChat) skip streaming — the partial first message could never
|
||
# be updated — unless they have a native-streaming transport (WeCom msgtype "stream").
|
||
_adapter_supports_edit = getattr(adapter, "SUPPORTS_MESSAGE_EDITING", True)
|
||
_adapter_supports_native_stream = bool(getattr(adapter, "SUPPORTS_NATIVE_STREAMING", False))
|
||
if not _adapter_supports_edit and not _adapter_supports_native_stream and on_missing_cursor == "raise":
|
||
raise RuntimeError("skip streaming for non-editable platform")
|
||
_effective_cursor = scfg.cursor if _adapter_supports_edit else ""
|
||
# Some Matrix clients render the cursor as tofu: stream text, no cursor.
|
||
_buffer_only = source.platform == Platform.MATRIX
|
||
if _buffer_only:
|
||
_effective_cursor = ""
|
||
# Fresh-final applies to Telegram only (others edit in place cheaply).
|
||
# Fresh-final applies to Telegram only — other platforms either edit in place cheaply (Discord,
|
||
# Slack) or don't have the timestamp-on-edit / edit-timestamp-stays-stale problem. (Ported from
|
||
# openclaw/openclaw#72038.)
|
||
_fresh_final_secs = (
|
||
float(getattr(scfg, "fresh_final_after_seconds", 0.0) or 0.0)
|
||
if source.platform == Platform.TELEGRAM else 0.0
|
||
)
|
||
_consumer_cfg = StreamConsumerConfig(
|
||
edit_interval=scfg.edit_interval, buffer_threshold=scfg.buffer_threshold,
|
||
cursor=_effective_cursor, buffer_only=_buffer_only,
|
||
fresh_final_after_seconds=_fresh_final_secs, transport=scfg.transport or "edit",
|
||
chat_type=getattr(source, "chat_type", "") or "",
|
||
)
|
||
return _consumer_cfg, _pause_typing_before_finalize
|
||
|
||
def _run_still_current_fn(self, session_key: Optional[str], run_generation: Optional[int]) -> Callable[[], bool]:
|
||
"""Predicate: does this run's generation still own ``session_key``? (always True when untracked)."""
|
||
def _run_still_current() -> bool:
|
||
if run_generation is None or not session_key:
|
||
return True
|
||
return self._is_session_run_current(session_key, run_generation)
|
||
return _run_still_current
|
||
|
||
@staticmethod
|
||
def _proxy_error_result(text: str) -> Dict[str, Any]:
|
||
return {"final_response": text, "messages": [], "api_calls": 0, "tools": []}
|
||
|
||
def _proxy_stream_consumer(self, source: "SessionSource", event_message_id, _thread_metadata, _run_still_current):
|
||
"""Platform stream consumer for the proxy path when streaming is enabled, else ``None``."""
|
||
from gateway.run import _load_gateway_config, _platform_config_key
|
||
_scfg = getattr(getattr(self, "config", None), "streaming", None)
|
||
# #60671 — streaming TTS consumer is created on the outer event-loop thread before run_sync
|
||
# launches. run_sync only reads it via ``streaming_tts_consumer_holder[0]`` for delta callback
|
||
# wiring.
|
||
if _scfg is None:
|
||
from gateway.config import StreamingConfig
|
||
_scfg = StreamingConfig()
|
||
from gateway.display_config import resolve_display_setting
|
||
_plat_streaming = resolve_display_setting(_load_gateway_config(), _platform_config_key(source.platform), "streaming")
|
||
_streaming_enabled = (
|
||
_scfg.enabled and _scfg.transport != "off" if _plat_streaming is None else bool(_plat_streaming)
|
||
)
|
||
if not _streaming_enabled:
|
||
return None
|
||
try:
|
||
from gateway.stream_consumer import GatewayStreamConsumer
|
||
_adapter = self._adapter_for_source(source)
|
||
if not _adapter:
|
||
return None
|
||
_consumer_cfg, _pause_typing_before_finalize = self._build_stream_consumer_config(
|
||
source, _scfg, _adapter, on_missing_cursor="fallback",
|
||
)
|
||
return GatewayStreamConsumer(
|
||
adapter=_adapter, chat_id=source.chat_id, config=_consumer_cfg,
|
||
metadata=_thread_metadata, on_before_finalize=_pause_typing_before_finalize,
|
||
initial_reply_to_id=event_message_id, run_still_current=_run_still_current,
|
||
)
|
||
except Exception as _sc_err:
|
||
logger.debug("Proxy: could not set up stream consumer: %s", _sc_err)
|
||
return None
|
||
|
||
async def _run_agent_via_proxy(
|
||
self, message: str, context_prompt: str, history: List[Dict[str, Any]],
|
||
source: "SessionSource", session_id: str, session_key: str = None,
|
||
run_generation: Optional[int] = None, event_message_id: Optional[str] = None,
|
||
) -> Dict[str, Any]:
|
||
"""Forward the message to a remote Hermes API server instead of running a local AIAgent.
|
||
|
||
Lets a Docker container handle Matrix E2EE while the agent runs on the host with full
|
||
access to local files, memory, skills, and a unified session store."""
|
||
from gateway.run import _GATEWAY_PROXY_SSE_BUFFER_MAX_CHARS
|
||
try:
|
||
from aiohttp import ClientSession as _AioClientSession, ClientTimeout
|
||
except ImportError:
|
||
return self._proxy_error_result("⚠️ Proxy mode requires aiohttp. Install with: pip install aiohttp")
|
||
|
||
proxy_url = self._get_proxy_url()
|
||
if not proxy_url:
|
||
return self._proxy_error_result("⚠️ Proxy URL not configured (GATEWAY_PROXY_URL or gateway.proxy_url)")
|
||
|
||
# The proxy key is a per-profile credential: honor the installed secret scope under multiplex.
|
||
# Only UnscopedSecretError / import failures fall back to the env; any other get_secret()
|
||
# error propagates (same as BASE) rather than silently degrading to the ambient key.
|
||
try:
|
||
from agent.secret_scope import UnscopedSecretError, get_secret
|
||
|
||
try:
|
||
proxy_key = (get_secret("GATEWAY_PROXY_KEY") or "").strip()
|
||
except UnscopedSecretError:
|
||
proxy_key = os.getenv("GATEWAY_PROXY_KEY", "").strip()
|
||
except Exception:
|
||
proxy_key = os.getenv("GATEWAY_PROXY_KEY", "").strip()
|
||
|
||
_run_still_current = self._run_still_current_fn(session_key, run_generation)
|
||
|
||
def _stale_result(what: str) -> Dict[str, Any]:
|
||
logger.info(
|
||
"Discarding stale proxy %s for %s — generation %d is no longer current",
|
||
what, session_key or "?", run_generation or 0,
|
||
)
|
||
return {
|
||
"final_response": "", "messages": [], "api_calls": 0, "tools": [],
|
||
"history_offset": len(history), "session_id": session_id, "response_previewed": False,
|
||
}
|
||
|
||
# OpenAI chat format. The remote keeps continuity via X-Hermes-Session-Id; send the current
|
||
# message plus a compact text-only history for a remote that has none yet.
|
||
api_messages: List[Dict[str, str]] = [{"role": "system", "content": context_prompt}] if context_prompt else []
|
||
api_messages += [
|
||
{"role": msg.get("role"), "content": msg.get("content")}
|
||
for msg in history if msg.get("role") in {"user", "assistant"} and msg.get("content")
|
||
]
|
||
api_messages.append({"role": "user", "content": message})
|
||
|
||
headers: Dict[str, str] = {"Content-Type": "application/json"}
|
||
if proxy_key:
|
||
headers["Authorization"] = f"Bearer {proxy_key}"
|
||
if session_id:
|
||
headers["X-Hermes-Session-Id"] = session_id
|
||
body = {"model": "hermes-agent", "messages": api_messages, "stream": True}
|
||
|
||
_thread_metadata: Optional[Dict[str, Any]] = self._thread_metadata_for_source(source, event_message_id)
|
||
_stream_consumer = self._proxy_stream_consumer(source, event_message_id, _thread_metadata, _run_still_current)
|
||
stream_task = asyncio.create_task(_stream_consumer.run()) if _stream_consumer else None
|
||
|
||
_adapter = self._adapter_for_source(source)
|
||
if _adapter:
|
||
with suppress(Exception):
|
||
await _adapter.send_typing(source.chat_id, metadata=_thread_metadata)
|
||
|
||
full_response = ""
|
||
_start = time.time()
|
||
try:
|
||
_timeout = ClientTimeout(total=0, sock_read=1800)
|
||
async with _AioClientSession(timeout=_timeout) as session:
|
||
async with session.post(f"{proxy_url}/v1/chat/completions", json=body, headers=headers) as resp:
|
||
if resp.status != 200:
|
||
error_text = await resp.text()
|
||
logger.warning("Proxy error (%d) from %s: %s", resp.status, proxy_url, error_text[:500])
|
||
return self._proxy_error_result(f"⚠️ Proxy error ({resp.status}): {error_text[:300]}")
|
||
|
||
buffer = ""
|
||
async for chunk in resp.content.iter_any():
|
||
if not _run_still_current():
|
||
return _stale_result("stream")
|
||
buffer += chunk.decode("utf-8", errors="replace")
|
||
while "\n" in buffer:
|
||
line, buffer = buffer.split("\n", 1)
|
||
line = line.strip()
|
||
if not line.startswith("data: "):
|
||
continue
|
||
data = line[6:]
|
||
if data.strip() == "[DONE]":
|
||
break
|
||
try:
|
||
choices = json.loads(data).get("choices", [])
|
||
except json.JSONDecodeError:
|
||
continue
|
||
content = choices[0].get("delta", {}).get("content", "") if choices else ""
|
||
if content:
|
||
full_response += content
|
||
if _stream_consumer:
|
||
_stream_consumer.on_delta(content)
|
||
if len(buffer) > _GATEWAY_PROXY_SSE_BUFFER_MAX_CHARS:
|
||
raise ValueError("Proxy SSE stream exceeded max buffer size without a line boundary")
|
||
except asyncio.CancelledError:
|
||
raise
|
||
except Exception as e:
|
||
logger.error("Proxy connection error to %s: %s", proxy_url, e)
|
||
if not full_response:
|
||
return self._proxy_error_result(f"⚠️ Proxy connection error: {e}")
|
||
# Partial response — return what we got
|
||
finally:
|
||
if _stream_consumer:
|
||
_stream_consumer.finish()
|
||
if stream_task:
|
||
try:
|
||
await asyncio.wait_for(stream_task, timeout=5.0)
|
||
except (asyncio.TimeoutError, asyncio.CancelledError):
|
||
stream_task.cancel()
|
||
|
||
_elapsed = time.time() - _start
|
||
if not _run_still_current():
|
||
return _stale_result("result")
|
||
logger.info(
|
||
"proxy response: url=%s session=%s time=%.1fs response=%d chars",
|
||
proxy_url, (session_id or "")[:20], _elapsed, len(full_response),
|
||
)
|
||
return {
|
||
"final_response": full_response or "(No response from remote agent)",
|
||
"messages": [
|
||
{"role": "user", "content": message},
|
||
{"role": "assistant", "content": full_response},
|
||
],
|
||
"api_calls": 1,
|
||
"tools": [],
|
||
"history_offset": len(history),
|
||
"session_id": session_id,
|
||
"response_previewed": _stream_consumer is not None and bool(full_response),
|
||
}
|
||
|
||
async def _run_agent(
|
||
self, message: str, context_prompt: str, history: List[Dict[str, Any]],
|
||
source: SessionSource, session_id: str, **turn_kwargs,
|
||
) -> Dict[str, Any]:
|
||
"""Profile-scoping wrapper around ``_run_agent_inner`` (same keyword parameters; pass-through
|
||
when multiplexing is off)."""
|
||
with self._profile_scope_for_source(source):
|
||
return await self._run_agent_inner(message, context_prompt, history, source, session_id, **turn_kwargs)
|
||
|
||
def _run_agent_display_settings(self, source: SessionSource) -> "GatewayRunner._RunAgentDisplay":
|
||
"""Resolve per-platform display, progress, status and streaming-surface settings for a turn."""
|
||
from gateway.run import (
|
||
_gateway_platform_value, _has_platform_display_override, _load_gateway_config,
|
||
_platform_config_key,
|
||
)
|
||
from gateway.display_config import resolve_display_setting
|
||
from gateway.status_phrases import choose_status_phrase, resolve_status_phrase_catalog
|
||
user_config = _load_gateway_config()
|
||
platform_key = _platform_config_key(source.platform)
|
||
enabled_toolsets, disabled_toolsets = self._resolve_turn_toolsets(user_config, source, platform_key)
|
||
adapter = self._adapter_for_source(source)
|
||
# display.platforms.<platform>.<key> → display.<key> → built-in platform defaults.
|
||
_display_cfg = user_config.get("display", {})
|
||
if not isinstance(_display_cfg, dict):
|
||
_display_cfg = {}
|
||
|
||
# Tool preview length (0 = no limit) and friendly tool labels (default on), per-platform.
|
||
for _setter, _setting, _default, _cast in (
|
||
("set_tool_preview_max_len", "tool_preview_length", 0, lambda v: int(v) if v else 0),
|
||
("set_friendly_tool_labels", "friendly_tool_labels", True, bool),
|
||
):
|
||
with suppress(Exception):
|
||
from agent import display as _agent_display
|
||
_val = resolve_display_setting(user_config, platform_key, _setting, _default)
|
||
getattr(_agent_display, _setter)(_cast(_val))
|
||
|
||
# Tool progress mode; HERMES_TOOL_PROGRESS_MODE wins only when the config never set it.
|
||
_resolved_tp = resolve_display_setting(user_config, platform_key, "tool_progress")
|
||
_env_tp = os.getenv("HERMES_TOOL_PROGRESS_MODE")
|
||
_platform_cfg = (_display_cfg.get("platforms") or {}).get(platform_key) or {}
|
||
_legacy_tp_overrides = _display_cfg.get("tool_progress_overrides") or {}
|
||
_tool_progress_configured = "tool_progress" in _display_cfg or any(
|
||
isinstance(cfg, dict) and key in cfg
|
||
for cfg, key in ((_platform_cfg, "tool_progress"), (_legacy_tp_overrides, platform_key))
|
||
)
|
||
progress_mode = _env_tp if _env_tp and not _tool_progress_configured else (_resolved_tp or _env_tp or "all")
|
||
# "accumulate" (edit one bubble) or "separate" (one msg per tool)
|
||
progress_grouping = resolve_display_setting(user_config, platform_key, "tool_progress_grouping") or "accumulate"
|
||
_generic_status_recent: List[str] = []
|
||
_generic_status_catalog = resolve_status_phrase_catalog(user_config, platform_key)
|
||
|
||
def _display_surface_mode(
|
||
setting: str, *, default: bool = False,
|
||
require_platform_override_for: set[Any] | None = None, allow_generic: bool = False,
|
||
) -> str:
|
||
"""Return off|raw|generic for a gateway visibility surface."""
|
||
if require_platform_override_for:
|
||
current_platform = _gateway_platform_value(source.platform)
|
||
platform_only = {_gateway_platform_value(item) for item in require_platform_override_for}
|
||
if (
|
||
current_platform in platform_only
|
||
and not _has_platform_display_override(user_config, platform_key, setting)
|
||
):
|
||
return "off"
|
||
value = resolve_display_setting(user_config, platform_key, setting, default)
|
||
if isinstance(value, str) and value.strip().lower() == "generic":
|
||
return "generic" if allow_generic else "off"
|
||
return "raw" if bool(value) else "off"
|
||
|
||
def _generic_status_phrase(kind: str, *, tool_name: str | None = None, preview: str | None = None, args: Any = None) -> str:
|
||
try:
|
||
return choose_status_phrase(
|
||
kind, tool_name=tool_name, preview=preview, args=args,
|
||
recent=_generic_status_recent, catalog=_generic_status_catalog,
|
||
)
|
||
except Exception as _phrase_err:
|
||
logger.debug("generic status phrase selection failed: %s", _phrase_err)
|
||
return "still on it" if kind in {"heartbeat", "waiting", "long_running", "status"} else "one sec"
|
||
|
||
# Webhooks can't edit messages, so tool progress / log mode are off there.
|
||
is_webhook = source.platform == Platform.WEBHOOK
|
||
tool_progress_enabled = progress_mode not in {"off", "log"} and not is_webhook
|
||
# Live status for text-rendering typing indicators (Slack); independent of tool_progress.
|
||
_live_status_mode = resolve_display_setting(user_config, platform_key, "live_status", "full")
|
||
_live_status_adapter = (
|
||
adapter if getattr(adapter, "supports_status_text", False) and _live_status_mode != "off" else None
|
||
)
|
||
# "log" mode: tool calls go to ~/.hermes/logs/tool_calls.log instead of the chat. Gateway-only.
|
||
log_mode_enabled = progress_mode == "log" and not is_webhook
|
||
# Interim assistant messages and thinking_progress are independent of tool progress (same
|
||
# queue). Mattermost requires a per-platform opt-in: scratch text leaks into public threads.
|
||
interim_assistant_messages_mode = _display_surface_mode(
|
||
"interim_assistant_messages", default=True, require_platform_override_for={Platform.MATTERMOST},
|
||
)
|
||
interim_assistant_messages_enabled = not is_webhook and interim_assistant_messages_mode != "off"
|
||
_thinking_enabled = _display_surface_mode(
|
||
"thinking_progress", default=False, require_platform_override_for={Platform.MATTERMOST},
|
||
) != "off"
|
||
# Slack-native task cards need the progress queue even with text tool_progress off.
|
||
# Slack-native task cards (#29483): when the Slack adapter's opt-in is set, tool progress renders as
|
||
# native plan/task cards via chat.startStream — the progress queue is needed even though Slack keeps
|
||
# ordinary text tool_progress off by default (requiring both flags would silently leave the native
|
||
# feature inactive).
|
||
_native_slack_task_cards = False
|
||
if source.platform == Platform.SLACK and hasattr(adapter, "native_task_cards_enabled"):
|
||
try:
|
||
_native_slack_task_cards = bool(adapter.native_task_cards_enabled())
|
||
except Exception:
|
||
logger.debug("Slack native task-card config check failed", exc_info=True)
|
||
return self._RunAgentDisplay(
|
||
user_config=user_config, platform_key=platform_key, enabled_toolsets=enabled_toolsets,
|
||
disabled_toolsets=disabled_toolsets, resolve_display_setting=resolve_display_setting,
|
||
progress_mode=progress_mode, progress_grouping=progress_grouping,
|
||
_display_surface_mode=_display_surface_mode,
|
||
tool_progress_enabled=tool_progress_enabled, _live_status_mode=_live_status_mode,
|
||
_live_status_adapter=_live_status_adapter, log_mode_enabled=log_mode_enabled,
|
||
log_queue=queue.Queue() if log_mode_enabled else None,
|
||
interim_assistant_messages_enabled=interim_assistant_messages_enabled,
|
||
_thinking_enabled=_thinking_enabled, _native_slack_task_cards=_native_slack_task_cards,
|
||
needs_progress_queue=tool_progress_enabled or _thinking_enabled or _native_slack_task_cards,
|
||
_generic_status_phrase=_generic_status_phrase,
|
||
)
|
||
|
||
# _RunAgentDisplay fields copied verbatim onto the TurnContext.
|
||
_DISPLAY_TO_TURN_CTX = (
|
||
"_live_status_adapter", "_live_status_mode", "_thinking_enabled", "progress_mode",
|
||
"progress_grouping", "tool_progress_enabled", "log_queue", "resolve_display_setting",
|
||
"user_config", "enabled_toolsets", "disabled_toolsets", "log_mode_enabled",
|
||
"interim_assistant_messages_enabled", "needs_progress_queue", "_native_slack_task_cards",
|
||
)
|
||
|
||
def _run_agent_build_turn_context(
|
||
self, disp: "GatewayRunner._RunAgentDisplay", AIAgent: Any, *, message: str, source: SessionSource,
|
||
session_key: Optional[str], run_generation: Optional[int], **turn_params,
|
||
) -> Tuple[TurnContext, TurnRunner, Any]:
|
||
"""Build the ``TurnContext`` and its ``TurnRunner``; ``turn_params`` (history, context_prompt,
|
||
session_id, persist_user_*, …) are stored verbatim. Returns ``(turn_ctx, turn_runner,
|
||
cleanup_adapter)``."""
|
||
from gateway.run_turn_runner import TurnRunner
|
||
# Discord voice "verbal ack" on the FIRST tool call (discord.voice_fx.enabled): resolve the
|
||
# guild whose voice connection is bound to this text channel (mirrors DiscordAdapter.play_tts).
|
||
_voice_ack_guild: List[Optional[int]] = [None]
|
||
if source.platform == Platform.DISCORD:
|
||
_va = self.adapters.get(Platform.DISCORD)
|
||
_vtc = getattr(_va, "_voice_text_channels", None)
|
||
if isinstance(_vtc, dict) and hasattr(_va, "voice_mixer_active"):
|
||
_voice_ack_guild[0] = next(
|
||
(_gid for _gid, _tc in _vtc.items() if str(_tc) == str(source.chat_id) and _va.voice_mixer_active(_gid)),
|
||
None,
|
||
)
|
||
|
||
# Auto-cleanup of temporary progress bubbles needs a real ``delete_message`` (getattr on the
|
||
# type: a fake adapter without it means "can't delete", not a crash).
|
||
_cleanup_progress = bool(
|
||
disp.resolve_display_setting(disp.user_config, disp.platform_key, "cleanup_progress")
|
||
)
|
||
_cleanup_adapter = self._adapter_for_source(source) if _cleanup_progress else None
|
||
if _cleanup_adapter is not None and getattr(type(_cleanup_adapter), "delete_message", None) in (
|
||
None, BasePlatformAdapter.delete_message,
|
||
):
|
||
_cleanup_progress = False
|
||
_cleanup_adapter = None
|
||
|
||
# The one-slot progress/holder containers shared with the callbacks are TurnContext defaults.
|
||
turn_ctx = TurnContext(
|
||
source=source, message=message, AIAgent=AIAgent, session_key=session_key,
|
||
run_generation=run_generation, _cleanup_progress=_cleanup_progress,
|
||
_run_still_current=self._run_still_current_fn(session_key, run_generation),
|
||
progress_queue=queue.Queue() if disp.needs_progress_queue else None,
|
||
_voice_ack_guild=_voice_ack_guild, _voice_ack_loop=asyncio.get_running_loop(),
|
||
**{name: getattr(disp, name) for name in self._DISPLAY_TO_TURN_CTX}, **turn_params,
|
||
)
|
||
turn_runner = TurnRunner(self, turn_ctx)
|
||
# Agent tool-lifecycle callbacks live on the runner (bound methods, same signatures).
|
||
turn_ctx.progress_callback = turn_runner.progress_callback
|
||
turn_ctx.voice_ack_callback = turn_runner.voice_ack_callback
|
||
turn_ctx.native_tool_start_callback = turn_runner.combined_tool_start_callback
|
||
turn_ctx.native_tool_complete_callback = turn_runner.native_tool_complete_callback
|
||
return turn_ctx, turn_runner, _cleanup_adapter
|
||
|
||
def _thread_metadata_for_progress(
|
||
self, source: SessionSource, event_message_id: Optional[str], _progress_thread_id: Any,
|
||
_relay_prospective_thread_id: Optional[str],
|
||
) -> Optional[Dict[str, Any]]:
|
||
"""Thread metadata for a progress-lane send; relay Discord auto-thread lane falls back to the reply anchor.
|
||
|
||
The connector will auto-thread on the reply anchor (thread is born on its FIRST send), so
|
||
carrying it routes progress / status bubbles into the same thread as the final reply."""
|
||
if not _progress_thread_id:
|
||
metadata = None
|
||
elif _progress_thread_id == source.thread_id:
|
||
metadata = self._thread_metadata_for_source(source, event_message_id)
|
||
else:
|
||
metadata = self._thread_metadata_for_target(
|
||
source.platform, source.chat_id, _progress_thread_id,
|
||
chat_type=getattr(source, "chat_type", None), reply_to_message_id=event_message_id,
|
||
)
|
||
if metadata is None and _relay_prospective_thread_id:
|
||
metadata = {"reply_to_message_id": event_message_id}
|
||
return metadata
|
||
|
||
def _run_agent_progress_threading(
|
||
self, source: SessionSource, event_message_id: Optional[str], _native_slack_task_cards: bool
|
||
) -> Tuple[Optional[dict], Optional[str], Optional[dict]]:
|
||
"""Resolve where progress bubbles are threaded (platform-specific).
|
||
|
||
Returns ``(progress_metadata, progress_reply_to, status_thread_metadata)``; the latter is
|
||
for status / approval / stream sends (Feishu topics need the triggering message id via the
|
||
reply API, so carry it as a fallback). Slack and Buzz honour the user's reply_in_thread
|
||
opt-out: never synthesise a thread for progress, or every later reply inherits it."""
|
||
from gateway.run import _non_conversational_metadata, _resolve_progress_thread_id
|
||
is_buzz = str(getattr(source.platform, "value", source.platform) or "").lower() == "buzz"
|
||
_progress_reply_in_thread = True
|
||
_adapter = self._adapter_for_source(source) if source.platform == Platform.SLACK or is_buzz else None
|
||
if _adapter is not None:
|
||
try:
|
||
if is_buzz:
|
||
_progress_reply_in_thread = getattr(_adapter, "_reply_to_mode", "first") != "off"
|
||
else:
|
||
# Relay lane: the adapter owns mode resolution; native lane: flat extra key.
|
||
_mode_fn = getattr(_adapter, "_effective_reply_in_thread", None)
|
||
_progress_reply_in_thread = bool(
|
||
_mode_fn() if callable(_mode_fn) else _adapter.config.extra.get("reply_in_thread", True)
|
||
)
|
||
except Exception:
|
||
_progress_reply_in_thread = True
|
||
_progress_thread_id = _resolve_progress_thread_id(
|
||
source.platform, source.thread_id, event_message_id, reply_in_thread=_progress_reply_in_thread,
|
||
)
|
||
# Relay Discord auto-thread lane: the connector stamps prospective_thread_id at ingest.
|
||
_relay_prospective_thread_id = (
|
||
str(getattr(source, "prospective_thread_id", None))
|
||
if source.platform == Platform.DISCORD
|
||
and getattr(source, "delivered_via_upstream_relay", False)
|
||
and getattr(source, "prospective_thread_id", None)
|
||
and not source.thread_id
|
||
else None
|
||
)
|
||
_progress_metadata = _non_conversational_metadata(
|
||
self._thread_metadata_for_progress(
|
||
source, event_message_id, _progress_thread_id, _relay_prospective_thread_id,
|
||
),
|
||
platform=source.platform,
|
||
)
|
||
if _native_slack_task_cards:
|
||
# chat.startStream in channels requires the recipient team/user pair; harmless elsewhere.
|
||
_progress_metadata = dict(_progress_metadata or {})
|
||
if source.scope_id:
|
||
_progress_metadata.setdefault("recipient_team_id", source.scope_id)
|
||
_progress_metadata.setdefault("slack_team_id", source.scope_id)
|
||
if source.user_id:
|
||
_progress_metadata.setdefault("recipient_user_id", source.user_id)
|
||
# Buzz has no native thread_id: thread via reply-to unless the user opted out.
|
||
_progress_reply_to = (
|
||
event_message_id
|
||
if (source.platform in (Platform.FEISHU, Platform.MATTERMOST) and source.thread_id and event_message_id)
|
||
or (is_buzz and event_message_id and _progress_reply_in_thread)
|
||
or _relay_prospective_thread_id
|
||
else None
|
||
)
|
||
if source.platform == Platform.FEISHU and source.thread_id and event_message_id:
|
||
_status_thread_metadata = {"thread_id": _progress_thread_id, "reply_to_message_id": event_message_id}
|
||
else:
|
||
_status_thread_metadata = self._thread_metadata_for_progress(
|
||
source, event_message_id, _progress_thread_id, _relay_prospective_thread_id,
|
||
)
|
||
return _progress_metadata, _progress_reply_to, _status_thread_metadata
|
||
|
||
async def _run_agent_write_tool_log(self, log_queue: Any) -> None:
|
||
"""Drain log_queue and append tool-call lines to tool_calls.log (tool_progress=log).
|
||
|
||
RotatingFileHandler (5MB × 3) bounds the log; RedactingFormatter keeps secrets off disk."""
|
||
from gateway.run import _hermes_home
|
||
if log_queue is None:
|
||
return
|
||
from logging.handlers import RotatingFileHandler
|
||
from agent.redact import RedactingFormatter
|
||
|
||
log_dir = _hermes_home / "logs"
|
||
log_dir.mkdir(parents=True, exist_ok=True)
|
||
file_handler = RotatingFileHandler(
|
||
log_dir / "tool_calls.log", maxBytes=5 * 1024 * 1024, backupCount=3, encoding="utf-8",
|
||
)
|
||
file_handler.setFormatter(RedactingFormatter("%(message)s"))
|
||
tool_logger = logging.getLogger(f"hermes.tool_calls.{id(log_queue)}")
|
||
tool_logger.setLevel(logging.INFO)
|
||
tool_logger.propagate = False
|
||
tool_logger.addHandler(file_handler)
|
||
try:
|
||
while True:
|
||
try:
|
||
tool_logger.info("%s", log_queue.get_nowait())
|
||
except queue.Empty:
|
||
await asyncio.sleep(0.3)
|
||
except Exception as e:
|
||
logger.error("write_tool_log error: %s", e)
|
||
await asyncio.sleep(1)
|
||
except asyncio.CancelledError:
|
||
pass
|
||
finally:
|
||
# Drain remaining entries so late tool calls from the final iteration aren't lost.
|
||
with suppress(Exception):
|
||
while True:
|
||
tool_logger.info("%s", log_queue.get_nowait())
|
||
tool_logger.removeHandler(file_handler)
|
||
with suppress(Exception):
|
||
file_handler.flush()
|
||
file_handler.close()
|
||
|
||
def _run_agent_start_streaming_tts(
|
||
self, source: SessionSource, message_type: Optional[str],
|
||
_status_thread_metadata: Optional[Dict[str, Any]], streaming_tts_consumer_holder: list,
|
||
) -> None:
|
||
"""Start the streaming-TTS consumer for a voice-input turn on an auto-TTS chat.
|
||
|
||
Created on the gateway loop thread (not run_sync's executor); an inactive consumer leaves
|
||
the holder None so the whole-file fallback path runs."""
|
||
# Skip when streaming TTS already delivered audio for this turn (#60671).
|
||
# This avoids a cross-scope NameError: the outer interrupt / finalisation paths reference the
|
||
# consumer via ``streaming_tts_consumer_holder[0]``. Gates: voice input, auto-TTS enabled for this
|
||
# chat, adapter supports streaming, and a usable streaming TTS provider configured. See #60671.
|
||
_stts_adapter = self._adapter_for_source(source)
|
||
_is_voice_input = (
|
||
message_type is not None
|
||
and str(getattr(message_type, "value", message_type)).lower() == "voice"
|
||
)
|
||
if _stts_adapter is None or not _is_voice_input or not _stts_adapter._should_auto_tts_for_chat(source.chat_id):
|
||
return
|
||
try:
|
||
from gateway.streaming_tts_consumer import StreamingTTSConsumer
|
||
from tools.tts_tool import _load_tts_config
|
||
_stts_consumer = StreamingTTSConsumer(
|
||
adapter=_stts_adapter, chat_id=source.chat_id, tts_config=_load_tts_config(),
|
||
loop=self._gateway_loop or asyncio.get_event_loop(),
|
||
metadata=_status_thread_metadata,
|
||
)
|
||
if _stts_consumer.active:
|
||
streaming_tts_consumer_holder[0] = _stts_consumer
|
||
_stts_consumer.start()
|
||
except Exception as _stts_err:
|
||
logger.debug("Could not set up streaming TTS consumer: %s", _stts_err)
|
||
|
||
async def _run_agent_stream_consumer_task(self, stream_consumer_holder: list) -> None:
|
||
"""Wait (up to 10s) for the stream consumer to be created inside run_sync, then run it."""
|
||
for _ in range(200):
|
||
if stream_consumer_holder[0] is not None:
|
||
await stream_consumer_holder[0].run()
|
||
return
|
||
await asyncio.sleep(0.05)
|
||
|
||
@staticmethod
|
||
async def _await_stream_task(stream_task) -> None:
|
||
"""Give the stream consumer task 5s to flush, then cancel it."""
|
||
try:
|
||
await asyncio.wait_for(stream_task, timeout=5.0)
|
||
except (asyncio.TimeoutError, asyncio.CancelledError):
|
||
stream_task.cancel()
|
||
with suppress(asyncio.CancelledError):
|
||
await stream_task
|
||
|
||
async def _run_agent_track_agent(self, turn_ctx: TurnContext) -> None:
|
||
"""Track this agent as running for the session (interrupt support) once it is created — only
|
||
if this run is still current, else leave the newer run's slot alone."""
|
||
session_key, run_generation, agent_holder = turn_ctx.session_key, turn_ctx.run_generation, turn_ctx.agent_holder
|
||
while agent_holder[0] is None:
|
||
await asyncio.sleep(0.05)
|
||
if not session_key:
|
||
return
|
||
if run_generation is not None and not self._is_session_run_current(session_key, run_generation):
|
||
logger.info(
|
||
"Skipping stale agent promotion for %s — generation %s is no longer current",
|
||
session_key or "", run_generation,
|
||
)
|
||
return
|
||
self._session_state(session_key).turn.agent = agent_holder[0]
|
||
if self._draining:
|
||
self._update_runtime_status("draining")
|
||
|
||
async def _run_agent_fire_pending_interrupt(
|
||
self, adapter: Any, agent: Any, source: SessionSource, session_key: str,
|
||
_interrupt_detected: "asyncio.Event", streaming_tts_consumer_holder: list, *,
|
||
log_context: str, log: Callable[[], None],
|
||
) -> None:
|
||
"""Peek the adapter's pending event, transcribe voice, then signal the agent + abort streaming TTS.
|
||
|
||
Peek WITHOUT consuming: the event must stay for the post-run ``_dequeue_pending_event()``
|
||
(popping races the agent finishing). Transcribe BEFORE signaling so voice interrupts carry
|
||
the real transcript."""
|
||
from gateway.run import _build_media_placeholder
|
||
_peek_event = adapter._pending_messages.get(session_key)
|
||
pending_text = None
|
||
if _peek_event is not None:
|
||
pending_text = _peek_event.text or ""
|
||
if self._pending_event_audio_paths(_peek_event):
|
||
pending_text, _ = await self._transcribe_and_echo_pending_voice(
|
||
_peek_event, adapter, source, pending_text, log_context=log_context,
|
||
metadata={"thread_id": source.thread_id} if source.thread_id else None,
|
||
)
|
||
elif not pending_text and (getattr(_peek_event, "media_urls", None) or []):
|
||
pending_text = _build_media_placeholder(_peek_event)
|
||
log()
|
||
agent.interrupt(pending_text)
|
||
_interrupt_detected.set()
|
||
# Abort streaming TTS on barge-in.
|
||
# See #60671.
|
||
# See #60671.
|
||
# See #60671.
|
||
# Finalize the streaming-TTS consumer (#60671). finish() is called from the outer event-loop thread
|
||
# (not the executor worker) so early returns from run_sync are also finalised. wait_complete()
|
||
# drains queued audio; on timeout the consumer is aborted unconditionally — if audio was audible,
|
||
# suppression is preserved so the gateway does not replay from the beginning; if no audio was
|
||
# audible, the whole-file fallback path is permitted.
|
||
_stts = streaming_tts_consumer_holder[0]
|
||
if _stts is not None:
|
||
_stts.abort("barge-in")
|
||
|
||
async def _run_agent_monitor_for_interrupt(self, turn_ctx: TurnContext, _interrupt_detected: "asyncio.Event") -> None:
|
||
"""Poll the adapter for interrupts (new messages) every 200ms and signal the agent.
|
||
|
||
Level 1 (base.py) catches regular text before _handle_message(); the inactivity poll loop
|
||
has a BACKUP check in case this task dies. Keyed by session_key, NOT source.chat_id."""
|
||
source, session_key, agent_holder = turn_ctx.source, turn_ctx.session_key, turn_ctx.agent_holder
|
||
streaming_tts_consumer_holder = turn_ctx.streaming_tts_consumer_holder
|
||
if not session_key:
|
||
return
|
||
while True:
|
||
await asyncio.sleep(0.2)
|
||
try:
|
||
# Re-resolve the adapter each iteration so reconnects don't leave a stale reference.
|
||
_adapter = self._adapter_for_source(source)
|
||
if not _adapter:
|
||
continue
|
||
if hasattr(_adapter, 'has_pending_interrupt') and _adapter.has_pending_interrupt(session_key):
|
||
agent = agent_holder[0]
|
||
if agent:
|
||
await self._run_agent_fire_pending_interrupt(
|
||
_adapter, agent, source, session_key, _interrupt_detected,
|
||
streaming_tts_consumer_holder, log_context="Voice-interrupt",
|
||
log=lambda: logger.debug("Interrupt detected from adapter, signaling agent..."),
|
||
)
|
||
break
|
||
except asyncio.CancelledError:
|
||
raise
|
||
except Exception as _mon_err:
|
||
logger.debug("monitor_for_interrupt error (will retry): %s", _mon_err)
|
||
|
||
async def _run_agent_backup_interrupt_check(
|
||
self, turn_ctx: TurnContext, _interrupt_detected: "asyncio.Event", interrupt_monitor: "asyncio.Task",
|
||
) -> None:
|
||
"""Backup interrupt check: if the monitor task died or missed the interrupt, catch it here."""
|
||
source, session_key = turn_ctx.source, turn_ctx.session_key
|
||
if _interrupt_detected.is_set() or not session_key:
|
||
return
|
||
_backup_adapter = self._adapter_for_source(source)
|
||
_backup_agent = turn_ctx.agent_holder[0]
|
||
if (_backup_adapter and _backup_agent
|
||
and hasattr(_backup_adapter, 'has_pending_interrupt')
|
||
and _backup_adapter.has_pending_interrupt(session_key)):
|
||
await self._run_agent_fire_pending_interrupt(
|
||
_backup_adapter, _backup_agent, source, session_key, _interrupt_detected,
|
||
turn_ctx.streaming_tts_consumer_holder,
|
||
log_context="Voice-backup-interrupt",
|
||
log=lambda: logger.info(
|
||
"Backup interrupt detected for session %s (monitor task state: %s)",
|
||
session_key, "done" if interrupt_monitor.done() else "running",
|
||
),
|
||
)
|
||
|
||
@staticmethod
|
||
def _run_agent_stream_confirmed_final_delivery(consumer, final_text: str, *, previewed: bool = False) -> bool:
|
||
"""True only when the actual final reply reached the user: a finalize call may carry only the
|
||
last preview snapshot, so reconcile against the recorded payload — a demonstrable mismatch
|
||
(False) overrides the flag; None keeps legacy trust."""
|
||
if consumer is None:
|
||
return False
|
||
if getattr(consumer, "final_response_sent", False):
|
||
matcher = getattr(consumer, "delivered_final_matches", None)
|
||
if callable(matcher):
|
||
with suppress(Exception):
|
||
if matcher(final_text) is False:
|
||
return False
|
||
return True
|
||
if previewed:
|
||
has_delivered_text = getattr(consumer, "has_delivered_text", None)
|
||
if callable(has_delivered_text):
|
||
try:
|
||
return bool(has_delivered_text(final_text))
|
||
except Exception:
|
||
return False
|
||
return False
|
||
|
||
def _run_agent_start_turn_worker(self, turn_ctx: TurnContext, run_sync: Callable[[], Any]) -> "GatewayRunner._RunAgentWorker":
|
||
"""Schedule ``run_sync`` on the executor plus the inactivity watchdog thread.
|
||
|
||
*Inactivity* timeout (agent.gateway_timeout / HERMES_AGENT_TIMEOUT, env wins; 0 = unlimited),
|
||
not wall-clock. The daemon watchdog is independent of asyncio: cgroup memory reclaim can
|
||
starve the loop that runs the normal timeout poll."""
|
||
from gateway.run import _float_env, _watch_gateway_turn_inactivity
|
||
from tools.process_registry import process_registry
|
||
agent_holder, session_key, run_generation = turn_ctx.agent_holder, turn_ctx.session_key, turn_ctx.run_generation
|
||
_agent_timeout, _agent_warning = (
|
||
v if v > 0 else None
|
||
for v in (_float_env("HERMES_AGENT_TIMEOUT", 1800), _float_env("HERMES_AGENT_TIMEOUT_WARNING", 900))
|
||
)
|
||
|
||
# background=true processes survive a turn: reap only children created by THIS turn on timeout.
|
||
_turn_task_id = turn_ctx.session_id or ""
|
||
# The daemon watchdog is independent of asyncio: cgroup memory reclaim may starve the event loop
|
||
# that runs the normal timeout poll, but it need not also postpone cleanup until the loop recovers
|
||
# (#76115).
|
||
_turn_process_baseline = process_registry.snapshot_running_ids(_turn_task_id)
|
||
turn_ctx.process_task_id = _turn_task_id
|
||
turn_ctx.process_baseline = _turn_process_baseline
|
||
# task_id is session-scoped: gate the reap on this claim still being current so a replacement
|
||
# turn's fresh process isn't killed by this turn's stale baseline.
|
||
worker = self._RunAgentWorker(
|
||
agent_timeout=_agent_timeout, agent_warning=_agent_warning, task_id=_turn_task_id,
|
||
process_baseline=_turn_process_baseline, worker_done=threading.Event(),
|
||
timeout_fired=threading.Event(), cleanup_lock=threading.Lock(),
|
||
is_current=(
|
||
(lambda: self._is_session_run_current(session_key, run_generation))
|
||
if run_generation is not None
|
||
else (lambda: True)
|
||
),
|
||
)
|
||
|
||
def _run_sync_with_timeout_lifecycle():
|
||
try:
|
||
return run_sync()
|
||
finally:
|
||
worker.worker_done.set()
|
||
# `.turn.agent` stays reachable until the *next* turn is claimed; clearing the
|
||
# ownership markers now means a /stop on the finished turn no longer reaps background
|
||
# work it left running.
|
||
# `.turn.agent` on the session state is only reset to _AGENT_PENDING_SENTINEL when the
|
||
# *next* turn is claimed (see _session_state(...).turn.agent = ... at claim time), so a
|
||
# stale reference to this exact agent instance stays reachable from
|
||
# _interrupt_and_clear_session() until then. See #76115.
|
||
_finished_agent = agent_holder[0] if agent_holder else None
|
||
if _finished_agent is not None:
|
||
_finished_agent._gateway_turn_process_task_id = ""
|
||
_finished_agent._gateway_turn_process_baseline = frozenset()
|
||
|
||
if _agent_timeout is not None:
|
||
threading.Thread(
|
||
target=_watch_gateway_turn_inactivity,
|
||
kwargs={
|
||
"agent_holder": agent_holder, "timeout": _agent_timeout, "poll_interval": 5.0,
|
||
**self._reaper_kwargs(worker),
|
||
},
|
||
name=f"gateway-turn-watchdog-{_turn_task_id[:12]}",
|
||
daemon=True,
|
||
).start()
|
||
worker.executor_task = asyncio.ensure_future(
|
||
self._run_in_executor_with_context(_run_sync_with_timeout_lifecycle)
|
||
)
|
||
return worker
|
||
|
||
@staticmethod
|
||
def _reaper_kwargs(worker: "GatewayRunner._RunAgentWorker") -> dict:
|
||
"""Shared kwargs of the watchdog + timeout-reaper threads."""
|
||
return {
|
||
**{k: getattr(worker, k) for k in ("task_id", "process_baseline", "worker_done", "timeout_fired", "cleanup_lock")},
|
||
"is_still_current": worker.is_current,
|
||
}
|
||
|
||
@staticmethod
|
||
def _agent_activity_summary(agent: Any) -> dict:
|
||
"""``agent.get_activity_summary()`` or ``{}`` when unavailable / failing."""
|
||
if agent and hasattr(agent, "get_activity_summary"):
|
||
with suppress(Exception):
|
||
return agent.get_activity_summary()
|
||
return {}
|
||
|
||
async def _run_agent_inactivity_warning(self, worker, source, _status_thread_metadata) -> None:
|
||
"""Staged one-shot warning before the inactivity timeout escalates."""
|
||
from gateway.run import _interim_metadata
|
||
_warn_adapter = self._adapter_for_source(source)
|
||
if not _warn_adapter:
|
||
return
|
||
try:
|
||
await _warn_adapter.send(
|
||
source.chat_id, f"⚠️ No activity for {int(worker.agent_warning // 60) or 1} min. "
|
||
"If the agent does not respond soon, it will be timed out in "
|
||
f"{int((worker.agent_timeout - worker.agent_warning) // 60) or 1} min. "
|
||
"You can continue waiting or use /reset.",
|
||
metadata=_interim_metadata(_status_thread_metadata),
|
||
)
|
||
except Exception as _warn_err:
|
||
logger.debug("Inactivity warning send error: %s", _warn_err)
|
||
|
||
def _run_agent_timeout_result(self, worker, turn_ctx: TurnContext) -> dict:
|
||
"""Synthetic failed run dict for an inactivity timeout, with the activity-tracker diagnostic;
|
||
interrupts the agent if it is still running so the thread pool worker is freed."""
|
||
from gateway.run import _INTERRUPT_REASON_TIMEOUT, request_hard_interrupt
|
||
session_key, result_holder, tools_holder = turn_ctx.session_key, turn_ctx.result_holder, turn_ctx.tools_holder
|
||
_timed_out_agent = turn_ctx.agent_holder[0]
|
||
_activity = self._agent_activity_summary(_timed_out_agent)
|
||
_last_desc = _activity.get("last_activity_desc", "unknown")
|
||
_secs_ago = _activity.get("seconds_since_activity", 0)
|
||
_cur_tool = _activity.get("current_tool")
|
||
_iter_n = _activity.get("api_call_count", 0)
|
||
_iter_max = _activity.get("max_iterations", 0)
|
||
# Operator-facing log keeps the raw resolved value; only the user-facing lines hide the sentinel.
|
||
logger.error(
|
||
"Agent idle for %.0fs (timeout %.0fs) in session %s "
|
||
"| last_activity=%s | iteration=%s/%s | tool=%s",
|
||
_secs_ago, worker.agent_timeout, session_key, _last_desc, _iter_n, _iter_max,
|
||
_cur_tool or "none",
|
||
)
|
||
if _timed_out_agent:
|
||
request_hard_interrupt(_timed_out_agent, _INTERRUPT_REASON_TIMEOUT)
|
||
_timeout_mins = int(worker.agent_timeout // 60) or 1
|
||
_iter_progress = format_iteration_progress(_iter_n, _iter_max)
|
||
_diag_lines = [
|
||
f"⏱️ Agent inactive for {_timeout_mins} min — no tool calls or API responses."
|
||
]
|
||
if _cur_tool:
|
||
_diag_lines.append(
|
||
f"The agent appears stuck on tool `{_cur_tool}` ({_secs_ago:.0f}s since last "
|
||
f"activity, {_iter_progress})."
|
||
)
|
||
else:
|
||
_diag_lines.append(
|
||
f"Last activity: {_last_desc} ({_secs_ago:.0f}s ago, "
|
||
f"{_iter_progress}). "
|
||
"The agent may have been waiting on an API response."
|
||
)
|
||
_diag_lines.append(
|
||
"To increase the limit, set agent.gateway_timeout in config.yaml (value in seconds, 0 "
|
||
"= no limit) and restart the gateway.\nTry again, or use /reset to start fresh."
|
||
)
|
||
return {
|
||
"final_response": "\n".join(_diag_lines),
|
||
"messages": result_holder[0].get("messages", []) if result_holder[0] else [],
|
||
"api_calls": _iter_n,
|
||
"tools": tools_holder[0] or [],
|
||
"history_offset": 0,
|
||
"failed": True,
|
||
}
|
||
|
||
async def _run_agent_await_turn_worker(
|
||
self, worker: "GatewayRunner._RunAgentWorker", turn_ctx: TurnContext,
|
||
_interrupt_detected: "asyncio.Event", interrupt_monitor: "asyncio.Task",
|
||
) -> Any:
|
||
"""Poll the executor future (inactivity timeout + backup interrupt checks); return its result,
|
||
or a synthetic failed run dict on inactivity timeout. Polls even with an unlimited timeout
|
||
so the backup interrupt check runs if monitor_for_interrupt() silently died."""
|
||
from gateway.run import _abandon_timed_out_gateway_turn
|
||
agent_holder = turn_ctx.agent_holder
|
||
_warning_fired = False
|
||
while True:
|
||
done, _ = await asyncio.wait({worker.executor_task}, timeout=5.0)
|
||
if done:
|
||
# Prefer the real result even if the watchdog fired in the same window (the run already
|
||
# persisted its reply).
|
||
return worker.executor_task.result()
|
||
if worker.agent_timeout is not None:
|
||
if worker.timeout_fired.is_set():
|
||
break
|
||
_idle_secs = self._agent_activity_summary(agent_holder[0]).get("seconds_since_activity", 0.0)
|
||
if not _warning_fired and worker.agent_warning is not None and _idle_secs >= worker.agent_warning:
|
||
_warning_fired = True
|
||
await self._run_agent_inactivity_warning(worker, turn_ctx.source, turn_ctx._status_thread_metadata)
|
||
if _idle_secs >= worker.agent_timeout:
|
||
threading.Thread(
|
||
target=_abandon_timed_out_gateway_turn,
|
||
kwargs={"agent_holder": agent_holder, **self._reaper_kwargs(worker)},
|
||
name=f"gateway-turn-reaper-{worker.task_id[:12]}", daemon=True,
|
||
).start()
|
||
break
|
||
await self._run_agent_backup_interrupt_check(turn_ctx, _interrupt_detected, interrupt_monitor)
|
||
return self._run_agent_timeout_result(worker, turn_ctx)
|
||
|
||
def _run_agent_evict_on_fallback(self, turn_ctx: TurnContext) -> None:
|
||
"""Evict the cached agent when a fallback model activated on a SUCCESSFUL run (so /model shows
|
||
the active model and the next message retries the primary). Skip failed runs: evicting
|
||
would loop bad model → fallback → evict → recreate."""
|
||
from gateway.run import _resolve_gateway_model
|
||
session_key = turn_ctx.session_key
|
||
_agent = turn_ctx.agent_holder[0]
|
||
_result_for_fb = turn_ctx.result_holder[0]
|
||
if _agent is None or not hasattr(_agent, 'model') or (_result_for_fb and _result_for_fb.get("failed")):
|
||
return
|
||
_cfg_model = _resolve_gateway_model()
|
||
# Normalize as AIAgent.__init__ does (vendor prefix stripped on native providers), else the
|
||
# cached agent is evicted every turn, destroying prompt caching.
|
||
with suppress(Exception):
|
||
from hermes_cli.model_normalize import _AGGREGATOR_PROVIDERS, normalize_model_for_provider
|
||
_agent_provider = getattr(_agent, 'provider', '') or ''
|
||
if _agent_provider and _agent_provider not in _AGGREGATOR_PROVIDERS:
|
||
_cfg_model = normalize_model_for_provider(_cfg_model, _agent_provider)
|
||
if _agent.model != _cfg_model and not self._is_intentional_model_switch(session_key, _agent, _cfg_model):
|
||
self._evict_cached_agent(session_key)
|
||
|
||
async def _run_agent_finalize_streaming_tts(self, turn_ctx: TurnContext, adapter: Any) -> None:
|
||
"""Finalize the streaming-TTS consumer on the outer event-loop thread (covers early returns
|
||
from run_sync). On drain timeout abort to free the task — audible streams keep whole-file
|
||
suppression, silent streams stay eligible for the whole-file fallback."""
|
||
_stts = turn_ctx.streaming_tts_consumer_holder[0]
|
||
if _stts is None:
|
||
return
|
||
_stts.finish()
|
||
try:
|
||
await _stts.wait_complete(timeout=10.0)
|
||
except Exception as _stts_done_err:
|
||
logger.debug("streaming TTS wait_complete error: %s", _stts_done_err)
|
||
if not _stts.done:
|
||
_stts.abort("streaming TTS finalisation timeout")
|
||
await _stts.wait_complete(timeout=2.0)
|
||
if _stts.suppress_whole_file and adapter is not None:
|
||
_mark_turn = getattr(adapter, "_mark_streaming_tts_completed_turn", None)
|
||
if callable(_mark_turn):
|
||
_mark_turn(turn_ctx.session_key, turn_ctx.run_generation)
|
||
|
||
async def _run_agent_drain_pending(
|
||
self, result: Any, adapter: Any, source: SessionSource, session_key: Optional[str]
|
||
) -> Tuple[Any, Optional[str]]:
|
||
"""Dequeue the adapter's pending / interrupt / leftover-steer follow-up as ``(pending_event, pending)``.
|
||
|
||
Keyed by session_key (not source.chat_id) to match the adapter's storage keys."""
|
||
from gateway.run import (
|
||
_build_media_placeholder, _dequeue_pending_event, _is_control_interrupt_message
|
||
)
|
||
pending_event = None
|
||
pending = None
|
||
if result and adapter and session_key:
|
||
pending_event = _dequeue_pending_event(adapter, session_key)
|
||
# /queue overflow: promote the next queued event into the consumed "next-up" slot so the
|
||
# recursive drain sees it (keeps FIFO order; a mid-chain /queue can't jump the queue).
|
||
pending_event = self._promote_queued_event(session_key, adapter, pending_event)
|
||
if result.get("interrupted") and not pending_event and result.get("interrupt_message"):
|
||
interrupt_message = result.get("interrupt_message")
|
||
if _is_control_interrupt_message(interrupt_message):
|
||
logger.info(
|
||
"Ignoring control interrupt message for session %s: %s",
|
||
session_key or "?", interrupt_message,
|
||
)
|
||
else:
|
||
pending = interrupt_message
|
||
elif pending_event:
|
||
# Transcribe audio BEFORE it becomes the next user turn (real transcript, not a path).
|
||
_pending_text = pending_event.text or ""
|
||
if self._pending_event_audio_paths(pending_event):
|
||
pending, _ = await self._transcribe_and_echo_pending_voice(
|
||
pending_event, adapter, source, _pending_text, log_context="Voice-drain",
|
||
metadata={"thread_id": source.thread_id} if source.thread_id else None,
|
||
)
|
||
pending = pending or _build_media_placeholder(pending_event)
|
||
else:
|
||
pending = _pending_text or _build_media_placeholder(pending_event)
|
||
if pending:
|
||
logger.debug("Processing queued message after agent completion: '%s...'", pending[:40])
|
||
|
||
# Leftover /steer (arrived after the last tool batch): deliver as the next user turn.
|
||
if result and not pending and not pending_event and result.get("pending_steer"):
|
||
pending = result.get("pending_steer")
|
||
logger.debug("Delivering leftover /steer as next turn: '%s...'", pending[:40])
|
||
|
||
# Safety net: a pending slash command is never passed to the agent as user input.
|
||
if pending and pending.strip().startswith("/"):
|
||
_pending_cmd_word = pending.strip().split(None, 1)[0][1:].lower()
|
||
if _pending_cmd_word:
|
||
with suppress(Exception):
|
||
from hermes_cli.commands import resolve_command as _rc_pending
|
||
if _rc_pending(_pending_cmd_word):
|
||
logger.info(
|
||
"Discarding command '/%s' from pending queue — "
|
||
"commands must not be passed as agent input", _pending_cmd_word,
|
||
)
|
||
pending_event = None
|
||
pending = None
|
||
|
||
if self._draining and (pending_event or pending):
|
||
logger.info(
|
||
"Discarding pending follow-up for session %s during gateway %s",
|
||
session_key or "?", self._status_action_label(),
|
||
)
|
||
pending_event = None
|
||
pending = None
|
||
return pending_event, pending
|
||
|
||
async def _run_agent_deliver_first_response(
|
||
self, turn_ctx: TurnContext, adapter: Any, response: Any, result: Any, stream_task: Any,
|
||
) -> None:
|
||
"""Deliver the first response before a queued follow-up runs, unless streaming already did."""
|
||
session_key = turn_ctx.session_key
|
||
_sc = turn_ctx.stream_consumer_holder[0]
|
||
if _sc and stream_task:
|
||
try:
|
||
await self._await_stream_task(stream_task)
|
||
except Exception as e:
|
||
logger.debug("Stream consumer wait before queued message failed: %s", e)
|
||
# Delivery uses the finalized task result (empty/failure normalization), not raw ``result``.
|
||
_delivery_result = response if isinstance(response, dict) else (result or {})
|
||
first_response = _delivery_result.get("final_response", "")
|
||
_already_streamed = self._run_agent_stream_confirmed_final_delivery(
|
||
_sc, first_response, previewed=bool(_delivery_result.get("response_previewed")),
|
||
)
|
||
# Same silence predicate as the normal path, else this branch leaks the literal marker.
|
||
if self._is_intentional_silence(_delivery_result, first_response):
|
||
logger.info(
|
||
"Queued follow-up for session %s: suppressing intentional silence marker before continuing.",
|
||
session_key or "?",
|
||
)
|
||
elif first_response:
|
||
logger.info(
|
||
"Queued follow-up for session %s: final text delivery confirmed; delivering explicit media before continuing."
|
||
if _already_streamed else
|
||
"Queued follow-up for session %s: final stream delivery not confirmed; sending first response before continuing.",
|
||
session_key or "?",
|
||
)
|
||
try:
|
||
await self._deliver_queued_first_response(
|
||
first_response, source=turn_ctx.source, adapter=adapter,
|
||
metadata=turn_ctx._status_thread_metadata, event_message_id=turn_ctx.event_message_id,
|
||
text_already_delivered=_already_streamed,
|
||
deliver_media=not _delivery_result.get("failed"), stream_consumer=_sc,
|
||
# The text send records a delivery-ledger obligation under this key, keyed on
|
||
# the raw inbound id (the anchor above is only the reply target).
|
||
session_key=session_key, inbound_message_id=turn_ctx.inbound_message_id,
|
||
)
|
||
except Exception as e:
|
||
logger.warning("Failed to send first response before queued message: %s", e)
|
||
# Release deferred bg-review notifications: pop (no double-fire in base.py's finally) and call.
|
||
_bg_cb = self._pop_post_delivery_callback(adapter, session_key, turn_ctx.run_generation)
|
||
if callable(_bg_cb):
|
||
with suppress(Exception):
|
||
_bg_result = _bg_cb()
|
||
if inspect.isawaitable(_bg_result):
|
||
await _bg_result
|
||
|
||
async def _run_agent_queued_followup(
|
||
self, turn_ctx: TurnContext, adapter: Any, pending: Optional[str], pending_event: Any,
|
||
response: Any, result: Any, stream_task: Any,
|
||
) -> Any:
|
||
"""Run the queued / interrupting follow-up as the next turn (recursive ``_run_agent``)."""
|
||
from gateway.platforms.base import merge_pending_message_event
|
||
from gateway.run import _preserve_queued_followup_history_offset
|
||
source, session_id, session_key, run_generation = (
|
||
turn_ctx.source, turn_ctx.session_id, turn_ctx.session_key, turn_ctx.run_generation,
|
||
)
|
||
_interrupt_depth, history, _status_thread_metadata = (
|
||
turn_ctx._interrupt_depth, turn_ctx.history, turn_ctx._status_thread_metadata,
|
||
)
|
||
logger.debug("Processing pending message: '%s...'", pending[:40])
|
||
|
||
# Clear the interrupt event so the recursive _run_agent isn't re-interrupted (infinite loop).
|
||
_active = getattr(adapter, "_active_sessions", None) if adapter else None
|
||
if _active and session_key and session_key in _active:
|
||
_active[session_key].clear()
|
||
|
||
# Cap recursion depth (user keeps sending while the agent keeps failing).
|
||
# (#816)
|
||
if _interrupt_depth >= self._MAX_INTERRUPT_DEPTH:
|
||
logger.warning(
|
||
"Interrupt recursion depth %d reached for session %s — "
|
||
"queueing message instead of recursing.", _interrupt_depth, session_key,
|
||
)
|
||
adapter = self._adapter_for_source(source)
|
||
if adapter and pending_event:
|
||
merge_pending_message_event(adapter._pending_messages, session_key, pending_event)
|
||
elif adapter and hasattr(adapter, 'queue_message'):
|
||
adapter.queue_message(session_key, pending)
|
||
return turn_ctx.result_holder[0] or {"final_response": response, "messages": history}
|
||
|
||
# Interrupted: discard the response ("Operation interrupted." is noise).
|
||
if not result.get("interrupted"):
|
||
await self._run_agent_deliver_first_response(turn_ctx, adapter, response, result, stream_task)
|
||
|
||
updated_history = result.get("messages", history)
|
||
next_source, next_message, next_session_key = source, pending, session_key
|
||
# message_type is carried into the recursive call so queued voice turns can stream TTS.
|
||
next_message_id = next_channel_prompt = next_message_type = None
|
||
# The raw inbound id keys the delivery-ledger obligation for the follow-up's own final send,
|
||
# distinct from the reply anchor above (None in forum topics). Carry it or two chained
|
||
# topic turns with the same text would collide on one obligation id (queued-final-ledger).
|
||
next_inbound_id = None
|
||
# See #60671.
|
||
if pending_event is not None:
|
||
next_source = getattr(pending_event, "source", None) or source
|
||
if self._is_goal_continuation_event(pending_event) and not self._goal_still_active_for_session(session_id):
|
||
logger.info(
|
||
"Discarding stale goal continuation for session %s — goal is no longer active",
|
||
session_key or "?",
|
||
)
|
||
return result
|
||
# Resolve the follow-up's session key BEFORE preparing the inbound text: native image
|
||
# paths are buffered under the key given and consumed under next_session_key.
|
||
try:
|
||
next_session_key = self._session_key_for_source(next_source)
|
||
except Exception:
|
||
logger.debug(
|
||
"Queued follow-up session-key resolution failed; reusing %s",
|
||
session_key or "?", exc_info=True,
|
||
)
|
||
next_message = await self._prepare_profile_scoped_inbound_message_text(
|
||
event=pending_event, source=next_source, history=updated_history, session_key=next_session_key,
|
||
)
|
||
if next_message is None:
|
||
return result
|
||
next_message_id = self._reply_anchor_for_event(pending_event)
|
||
next_inbound_id = str(pending_event.message_id) if getattr(pending_event, "message_id", None) else None
|
||
next_channel_prompt = getattr(pending_event, "channel_prompt", None)
|
||
next_message_type = getattr(pending_event, "message_type", None)
|
||
|
||
# Clear the prior turn's streaming-TTS completion marker so the recursive turn isn't suppressed.
|
||
# See #60671.
|
||
_clear_adapter = self._adapter_for_source(source)
|
||
_completed_turns = getattr(_clear_adapter, "_streaming_tts_completed_turns", None)
|
||
_prior_key = getattr(_clear_adapter, "_streaming_tts_turn_key", None)
|
||
if _completed_turns is not None and callable(_prior_key) and session_key and run_generation is not None:
|
||
_pk = _prior_key(session_key, run_generation)
|
||
if _pk:
|
||
_completed_turns.discard(_pk)
|
||
|
||
# Restart the typing indicator; the outer typing task may be stale.
|
||
if _clear_adapter:
|
||
with suppress(Exception):
|
||
await _clear_adapter.send_typing(source.chat_id, metadata=_status_thread_metadata)
|
||
|
||
# Re-baseline the cached agent's message_count before recursing, else the coherence guard
|
||
# rebuilds on OUR OWN flushed rows (the outer handler re-baselines only after the chain).
|
||
# Re-baseline the cached agent's message_count snapshot before recursing into the in-band queued
|
||
# (/queue) follow-up turn. The first turn has completed and flushed its own user + assistant rows to
|
||
# the SessionDB, so the cross-process coherence guard (#45966) — which this recursive _run_agent
|
||
# call re-enters — would otherwise see the grown on-disk count against the stale build-time snapshot
|
||
# and rebuild the agent on THIS process's OWN writes, destroying the prompt-cache prefix #46237 was
|
||
# merged to preserve. The existing re-baseline in _handle_message_with_agent only runs after the
|
||
# whole _run_agent chain unwinds — too late for the in-band follow-up. Use the same (session_key,
|
||
# session_id) the recursive call runs under so the snapshot matches exactly what the follow-up's
|
||
# guard will consult. Fail-safe in helper.
|
||
await self._refresh_agent_cache_message_count(session_key, session_id)
|
||
|
||
followup_result = await self._run_agent(
|
||
message=next_message, context_prompt=turn_ctx.context_prompt, history=updated_history,
|
||
source=next_source, session_id=session_id, session_key=next_session_key,
|
||
run_generation=run_generation, _interrupt_depth=_interrupt_depth + 1,
|
||
event_message_id=next_message_id, inbound_message_id=next_inbound_id,
|
||
channel_prompt=next_channel_prompt, message_type=next_message_type,
|
||
)
|
||
merged = _preserve_queued_followup_history_offset(result, followup_result)
|
||
# The TERMINAL turn of the chain owns the ledger identity for the outer final send, which
|
||
# the adapter brackets against the event that OPENED the chain. Without this the terminal
|
||
# reply is recorded under the first message's id, so a first reply that was refused (flood
|
||
# control) has its outstanding row replaced and marked delivered by an identical-text
|
||
# terminal reply, and is never redelivered. A deeper recursion has already set its own id,
|
||
# so only fill the key while it is still absent: the innermost turn wins.
|
||
if isinstance(merged, dict) and "queued_terminal_inbound_id" not in merged:
|
||
merged = {**merged, "queued_terminal_inbound_id": next_inbound_id}
|
||
return merged
|
||
|
||
async def _run_agent_cleanup_turn_tasks(
|
||
self, turn_ctx: TurnContext, *, progress_task: Any, log_task: Any, interrupt_monitor: "asyncio.Task",
|
||
_notify_task: "asyncio.Task", tracking_task: "asyncio.Task", stream_task: Any,
|
||
) -> None:
|
||
"""``finally`` half of a turn: cancel background tasks, flush stream, release the session slot."""
|
||
stream_consumer_holder, session_key = turn_ctx.stream_consumer_holder, turn_ctx.session_key
|
||
for task in (progress_task, log_task, interrupt_monitor, _notify_task):
|
||
if task:
|
||
task.cancel()
|
||
|
||
if stream_task:
|
||
# No stream consumer was created: nothing to flush, cancel instead of waiting out 5s.
|
||
if not (stream_consumer_holder and stream_consumer_holder[0] is not None):
|
||
stream_task.cancel()
|
||
with suppress(asyncio.CancelledError):
|
||
await stream_task
|
||
else:
|
||
await self._await_stream_task(stream_task)
|
||
|
||
# Abort + bounded wait for streaming TTS: covers paths where normal finalisation was skipped.
|
||
_stts_finally = turn_ctx.streaming_tts_consumer_holder[0]
|
||
# See #60671.
|
||
if _stts_finally is not None and not _stts_finally.done:
|
||
_stts_finally.abort("cleanup")
|
||
with suppress(Exception):
|
||
await _stts_finally.wait_complete(timeout=2.0)
|
||
|
||
tracking_task.cancel()
|
||
if session_key:
|
||
# Release the slot only if this run's generation still owns it (/stop or /new may have
|
||
# installed its own state).
|
||
self._release_running_agent_state(session_key, run_generation=turn_ctx.run_generation)
|
||
if self._draining:
|
||
self._update_runtime_status("draining")
|
||
|
||
for task in (progress_task, log_task, interrupt_monitor, tracking_task, _notify_task):
|
||
if task:
|
||
try:
|
||
await task
|
||
except asyncio.CancelledError:
|
||
pass
|
||
except Exception:
|
||
# A background task that died of a real error must not abort the cleanup path.
|
||
logger.debug("background turn task failed during cleanup", exc_info=True)
|
||
|
||
async def _run_agent_edit_streamed_message(
|
||
self, _sc, source, response, content, *, _sk, ok, fail_result, fail_exc,
|
||
) -> None:
|
||
"""Edit the stream consumer's message in place with ``content``; on success mark
|
||
``response["already_sent"]`` and log ``ok``. ``fail_result`` (None = trust the call) logs a
|
||
returned failure as ``(session, error)``; ``fail_exc`` logs an exception as ``(session, exc)``."""
|
||
try:
|
||
_res = await _sc.adapter.edit_message(
|
||
chat_id=source.chat_id, message_id=_sc.message_id, content=content, finalize=True,
|
||
)
|
||
except Exception as _edit_err:
|
||
logger.warning(fail_exc, _sk, _edit_err)
|
||
return
|
||
if fail_result is not None and not getattr(_res, "success", True):
|
||
logger.warning(fail_result, _sk, getattr(_res, "error", None))
|
||
return
|
||
response["already_sent"] = True
|
||
logger.info(*ok)
|
||
|
||
async def _run_agent_mark_streamed_delivery(self, response: Any, turn_ctx: TurnContext) -> None:
|
||
"""Set ``response["already_sent"]`` when streaming already delivered the final reply.
|
||
|
||
Never when the agent failed (the error is unseen content) or on "(empty)". Both suppression
|
||
flags reflect call success, not content, so reconcile against the recorded turn-final
|
||
payload: a mismatch (False, incl. payload-less split delivery) never suppresses; None (no
|
||
record) keeps legacy trust."""
|
||
_sc, source, session_key = turn_ctx.stream_consumer_holder[0], turn_ctx.source, turn_ctx.session_key
|
||
if not isinstance(response, dict) or response.get("failed"):
|
||
return
|
||
_final = response.get("final_response") or ""
|
||
_is_empty_sentinel = not _final or _final == "(empty)"
|
||
# response_previewed: only suppress if that EXACT text was delivered, not unrelated commentary.
|
||
# Unrelated commentary/progress must not be mistaken for the final response (#14238).
|
||
_previewed = bool(response.get("response_previewed"))
|
||
_content_delivered = bool(_sc and getattr(_sc, "final_content_delivered", False))
|
||
# #71643: a *successful* finalize edit can still carry only the last preview snapshot — deltas
|
||
# generated between that edit and stream completion never reach any API call, and both suppression
|
||
# flags are set from the call's success rather than its content. Reconcile the consumer's recorded
|
||
# turn-final payload against the completed response: on a demonstrable mismatch (False) neither
|
||
# final_response_sent nor final_content_delivered may suppress the normal final send. False also
|
||
# covers payload-less multi-message split delivery (#78541). None (no record on a non-split legacy
|
||
# path) keeps legacy trust; the failed-finalize family (#51828 / #33793) is unaffected because those
|
||
# paths leave the flags False or record the complete fallback payload.
|
||
_stale_finalized = False
|
||
if _content_delivered and not _is_empty_sentinel:
|
||
_matcher = getattr(_sc, "delivered_final_matches", None)
|
||
if callable(_matcher):
|
||
with suppress(Exception):
|
||
_stale_finalized = _matcher(_final) is False
|
||
if _stale_finalized:
|
||
_content_delivered = False
|
||
# Plugin hooks may append content after streaming finished — then send the final version.
|
||
_transformed = bool(response.get("response_transformed"))
|
||
# Suppress the normal send only when the actual final reply reached the user.
|
||
_streamed = self._run_agent_stream_confirmed_final_delivery(_sc, _final, previewed=_previewed)
|
||
if _is_empty_sentinel:
|
||
return
|
||
_sk = session_key or "?"
|
||
if not _transformed and (_streamed or _content_delivered):
|
||
logger.info(
|
||
"Suppressing normal final send for session %s: final delivery already confirmed (streamed=%s previewed=%s content_delivered=%s).",
|
||
_sk, _streamed, _previewed, _content_delivered,
|
||
)
|
||
response["already_sent"] = True
|
||
elif not _transformed and _stale_finalized and _sc is not None:
|
||
# Stale finalize: edit the streamed message up to the complete response (on failure the
|
||
# normal send delivers). Not for split delivery — message_id is only the LAST chunk.
|
||
_sc_msg_id = _sc.message_id
|
||
if getattr(_sc, "_turn_split_delivery", False):
|
||
logger.info(
|
||
"Stale streamed finalize detected for session %s on a multi-message split; skipping the in-place reconciliation edit and delivering the complete response via normal final send (#78541).",
|
||
_sk,
|
||
)
|
||
elif _sc_msg_id and _sc_msg_id != "__no_edit__" and getattr(_sc, "adapter", None) is not None:
|
||
await self._run_agent_edit_streamed_message(
|
||
_sc, source, response, _final, _sk=_sk,
|
||
ok=("Reconciled stale streamed finalize for session %s: edited message %s with the complete response (#71643).", _sk, _sc_msg_id),
|
||
fail_result="Stale-finalize reconciliation edit failed for session %s (%s); sending complete response via normal final send.",
|
||
fail_exc="Stale-finalize reconciliation edit failed for session %s: %s; sending complete response via normal final send.",
|
||
)
|
||
else:
|
||
logger.info(
|
||
"Stale streamed finalize detected for session %s with no editable message; delivering complete response via normal final send (#71643).",
|
||
_sk,
|
||
)
|
||
elif _transformed and _sc is not None:
|
||
# Transformed after streaming: edit the streamed message instead of sending a duplicate.
|
||
if _sc.message_id:
|
||
await self._run_agent_edit_streamed_message(
|
||
_sc, source, response, response["final_response"], _sk=_sk,
|
||
ok=("Edited streamed message %s for session %s to include plugin-transformed content.", _sc.message_id, _sk),
|
||
fail_result=None, fail_exc="Failed to edit streamed message for session %s: %s",
|
||
)
|
||
elif _sc is not None:
|
||
# DUPLICATE-RISK DIAGNOSTIC: a stream consumer existed but suppression did NOT fire; log the
|
||
# decision inputs ("signal never set" vs "ack-pending race").
|
||
logger.warning(
|
||
"Normal final-send NOT suppressed despite active stream consumer for session %s: "
|
||
"streamed=%s previewed=%s content_delivered=%s transformed=%s final_len=%d — "
|
||
"possible duplicate send (see wecom ack-timeout RCA).",
|
||
_sk, _streamed, _previewed, _content_delivered, _transformed, len(_final),
|
||
)
|
||
|
||
def _run_agent_schedule_bubble_cleanup(self, response: Any, _cleanup_adapter: Any, turn_ctx: TurnContext) -> None:
|
||
"""Schedule deletion of tracked temporary progress bubbles after the final response lands.
|
||
|
||
Failed runs keep them as breadcrumbs. Only on adapters with ``delete_message``; failures swallowed."""
|
||
from gateway.run import safe_schedule_threadsafe
|
||
_cleanup_msg_ids, session_key = turn_ctx._cleanup_msg_ids, turn_ctx.session_key
|
||
if not (
|
||
turn_ctx._cleanup_progress
|
||
and _cleanup_adapter is not None
|
||
and _cleanup_msg_ids
|
||
and session_key
|
||
and isinstance(response, dict)
|
||
and not response.get("failed")
|
||
and hasattr(_cleanup_adapter, "register_post_delivery_callback")
|
||
):
|
||
return
|
||
_ids_snapshot = list(_cleanup_msg_ids)
|
||
_chat_id_snapshot = turn_ctx.source.chat_id
|
||
_loop_snapshot = asyncio.get_running_loop()
|
||
|
||
def _cleanup_temp_bubbles() -> None:
|
||
async def _delete_all() -> None:
|
||
for _mid in _ids_snapshot:
|
||
with suppress(Exception):
|
||
await _cleanup_adapter.delete_message(_chat_id_snapshot, _mid)
|
||
with suppress(Exception):
|
||
safe_schedule_threadsafe(
|
||
_delete_all(), _loop_snapshot, logger=logger,
|
||
log_message="Temp bubble cleanup scheduling error",
|
||
)
|
||
|
||
try:
|
||
_cleanup_adapter.register_post_delivery_callback(
|
||
session_key, _cleanup_temp_bubbles, generation=turn_ctx.run_generation,
|
||
)
|
||
except Exception as _rpe:
|
||
logger.debug("Post-delivery cleanup registration failed: %s", _rpe)
|
||
|
||
def _run_agent_bind_turn_wiring(
|
||
self, turn_ctx: TurnContext, turn_runner: TurnRunner, source: SessionSource,
|
||
event_message_id: Optional[str], _native_slack_task_cards: bool,
|
||
) -> Optional[Dict[str, Any]]:
|
||
"""Resolve progress threading, then publish progress metadata and the sync→async bridges onto
|
||
``turn_ctx`` (the one-slot holders shared with run_sync's executor thread are TurnContext
|
||
defaults). Returns ``_status_thread_metadata``."""
|
||
turn_ctx._progress_metadata, turn_ctx._progress_reply_to, _status_thread_metadata = (
|
||
self._run_agent_progress_threading(source, event_message_id, _native_slack_task_cards)
|
||
)
|
||
# Bridges: sync step/event/status callbacks → async hooks.emit and adapter.send.
|
||
turn_ctx._loop_for_step = asyncio.get_running_loop()
|
||
turn_ctx._hooks_ref = self.hooks
|
||
turn_ctx._step_callback_sync = turn_runner._step_callback_sync
|
||
turn_ctx._event_callback_sync = turn_runner._event_callback_sync
|
||
turn_ctx._status_callback_sync = turn_runner._status_callback_sync
|
||
turn_ctx._status_adapter = self._adapter_for_source(source)
|
||
turn_ctx._status_chat_id = source.chat_id
|
||
turn_ctx._status_thread_metadata = _status_thread_metadata
|
||
return _status_thread_metadata
|
||
|
||
async def _run_agent_notify_long_running(
|
||
self, disp: "GatewayRunner._RunAgentDisplay", turn_ctx: TurnContext, _executor_task_holder: list,
|
||
) -> None:
|
||
"""Periodic "still working" heartbeat, edited in place where supported. Stops once this run
|
||
no longer owns the session slot or the executor finished. ``_executor_task_holder[0]`` is
|
||
bound just after this task is scheduled (reads as None until then).
|
||
|
||
Interval: agent.gateway_notify_interval / HERMES_AGENT_NOTIFY_INTERVAL (default 180s; 0 or
|
||
long_running_notifications=off disables)."""
|
||
from gateway.run import _float_env, _interim_metadata, _non_conversational_metadata
|
||
_notify_start = time.time()
|
||
_NOTIFY_INTERVAL = _float_env("HERMES_AGENT_NOTIFY_INTERVAL", 180)
|
||
_long_running_mode = disp._display_surface_mode("long_running_notifications", default=True, allow_generic=True)
|
||
if _NOTIFY_INTERVAL <= 0 or _long_running_mode == "off":
|
||
return
|
||
source, session_key, agent_holder = turn_ctx.source, turn_ctx.session_key, turn_ctx.agent_holder
|
||
_status_thread_metadata = turn_ctx._status_thread_metadata
|
||
_notify_adapter = self._adapter_for_source(source)
|
||
if not _notify_adapter:
|
||
return
|
||
_heartbeat_msg_id: Optional[str] = None
|
||
while True:
|
||
await asyncio.sleep(_NOTIFY_INTERVAL)
|
||
if not self._should_emit_long_running_notification(
|
||
session_key, agent_holder[0], _executor_task_holder[0]
|
||
):
|
||
break
|
||
_elapsed_mins = int((time.time() - _notify_start) // 60)
|
||
# Terse heartbeat by default; the iteration counter is gated on busy_ack_detail.
|
||
_status_detail = ""
|
||
_want_iteration_detail = bool(
|
||
disp.resolve_display_setting(disp.user_config, disp.platform_key, "busy_ack_detail", True)
|
||
)
|
||
_a = self._agent_activity_summary(agent_holder[0])
|
||
with suppress(Exception):
|
||
if _a:
|
||
_parts = []
|
||
if _want_iteration_detail:
|
||
_parts.append(format_iteration_progress(_a["api_call_count"], _a["max_iterations"]))
|
||
_action = _a.get("current_tool") or _a.get("last_activity_desc")
|
||
if _action:
|
||
_parts.append(str(_action))
|
||
if _parts:
|
||
_status_detail = " — " + ", ".join(_parts)
|
||
_heartbeat_text = (
|
||
disp._generic_status_phrase("status")
|
||
if _long_running_mode == "generic"
|
||
else f"⏳ Working — {_elapsed_mins} min{_status_detail}"
|
||
)
|
||
try:
|
||
_notify_res = None
|
||
if _heartbeat_msg_id:
|
||
try:
|
||
_notify_res = await _notify_adapter.edit_message(source.chat_id, _heartbeat_msg_id, _heartbeat_text)
|
||
except Exception as _ee:
|
||
logger.debug("Heartbeat edit failed: %s", _ee)
|
||
_notify_res = None
|
||
if not (_notify_res and getattr(_notify_res, "success", False)):
|
||
_notify_res = await _notify_adapter.send(
|
||
source.chat_id, _heartbeat_text,
|
||
metadata=_interim_metadata(_non_conversational_metadata(_status_thread_metadata, platform=source.platform)),
|
||
)
|
||
if getattr(_notify_res, "success", False) and getattr(_notify_res, "message_id", None):
|
||
_heartbeat_msg_id = str(_notify_res.message_id)
|
||
if turn_ctx._cleanup_progress:
|
||
turn_ctx._cleanup_msg_ids.append(_heartbeat_msg_id)
|
||
except Exception as _ne:
|
||
logger.debug("Long-running notification error: %s", _ne)
|
||
|
||
async def _run_agent_inner(
|
||
self, message: str, context_prompt: str, history: List[Dict[str, Any]],
|
||
source: SessionSource, session_id: str, session_key: str = None,
|
||
run_generation: Optional[int] = None, _interrupt_depth: int = 0,
|
||
event_message_id: Optional[str] = None, inbound_message_id: Optional[str] = None,
|
||
channel_prompt: Optional[str] = None, moa_config: Optional[dict] = None,
|
||
persist_user_message: Optional[Any] = None, persist_user_timestamp: Optional[float] = None,
|
||
persist_user_display_kind: Optional[str] = None, message_type: Optional[str] = None,
|
||
persist_user_display_metadata: Optional[dict] = None,
|
||
) -> Dict[str, Any]:
|
||
"""Run the agent; returns the full run_conversation result dict.
|
||
|
||
Keys: "final_response", "messages", "api_calls", "completed"."""
|
||
if self._get_proxy_url():
|
||
return await self._run_agent_via_proxy(
|
||
message=message, context_prompt=context_prompt, history=history, source=source,
|
||
session_id=session_id, session_key=session_key, run_generation=run_generation,
|
||
event_message_id=event_message_id,
|
||
)
|
||
|
||
from run_agent import AIAgent
|
||
|
||
disp = self._run_agent_display_settings(source)
|
||
turn_ctx, turn_runner, _cleanup_adapter = self._run_agent_build_turn_context(
|
||
disp, AIAgent, message=message, source=source, session_key=session_key,
|
||
run_generation=run_generation, context_prompt=context_prompt, history=history,
|
||
session_id=session_id, _interrupt_depth=_interrupt_depth,
|
||
event_message_id=event_message_id, inbound_message_id=inbound_message_id,
|
||
channel_prompt=channel_prompt, moa_config=moa_config,
|
||
persist_user_message=persist_user_message,
|
||
persist_user_timestamp=persist_user_timestamp,
|
||
persist_user_display_kind=persist_user_display_kind,
|
||
persist_user_display_metadata=persist_user_display_metadata,
|
||
)
|
||
_status_thread_metadata = self._run_agent_bind_turn_wiring(
|
||
turn_ctx, turn_runner, source, event_message_id, disp._native_slack_task_cards,
|
||
)
|
||
self._run_agent_start_streaming_tts(
|
||
source, message_type, _status_thread_metadata, turn_ctx.streaming_tts_consumer_holder,
|
||
)
|
||
|
||
# Progress sender drains BOTH tool-progress lines and thinking bubbles (needs_progress_queue).
|
||
spawn = asyncio.create_task
|
||
progress_task = spawn(turn_runner.send_progress_messages()) if disp.needs_progress_queue else None
|
||
log_task = spawn(self._run_agent_write_tool_log(disp.log_queue)) if disp.log_mode_enabled else None
|
||
# The stream consumer is created inside run_sync; this task polls for it.
|
||
stream_task = spawn(self._run_agent_stream_consumer_task(turn_ctx.stream_consumer_holder))
|
||
tracking_task = spawn(self._run_agent_track_agent(turn_ctx))
|
||
_interrupt_detected = asyncio.Event() # shared with backup check
|
||
interrupt_monitor = spawn(self._run_agent_monitor_for_interrupt(turn_ctx, _interrupt_detected))
|
||
# Periodic "still working" notifications so the user knows the agent hasn't died.
|
||
_executor_task_holder: list = [None] # bound once the executor future exists (see below)
|
||
_notify_task = spawn(self._run_agent_notify_long_running(disp, turn_ctx, _executor_task_holder))
|
||
|
||
try:
|
||
# run_sync is TurnRunner.run_sync (bound method; executor call unchanged).
|
||
worker = self._run_agent_start_turn_worker(turn_ctx, turn_runner.run_sync)
|
||
_executor_task_holder[0] = worker.executor_task # read late by _notify_long_running
|
||
response = await self._run_agent_await_turn_worker(worker, turn_ctx, _interrupt_detected, interrupt_monitor)
|
||
self._run_agent_evict_on_fallback(turn_ctx)
|
||
|
||
# Interrupted OR queued message (/queue)?
|
||
result = turn_ctx.result_holder[0]
|
||
adapter = self._adapter_for_source(source)
|
||
await self._run_agent_finalize_streaming_tts(turn_ctx, adapter)
|
||
pending_event, pending = await self._run_agent_drain_pending(result, adapter, source, session_key)
|
||
if pending_event or pending:
|
||
return await self._run_agent_queued_followup(
|
||
turn_ctx, adapter, pending, pending_event, response, result, stream_task,
|
||
)
|
||
finally:
|
||
await self._run_agent_cleanup_turn_tasks(
|
||
turn_ctx, progress_task=progress_task, log_task=log_task, interrupt_monitor=interrupt_monitor,
|
||
_notify_task=_notify_task, tracking_task=tracking_task, stream_task=stream_task,
|
||
)
|
||
|
||
await self._run_agent_mark_streamed_delivery(response, turn_ctx)
|
||
self._run_agent_schedule_bubble_cleanup(response, _cleanup_adapter, turn_ctx)
|
||
return response
|