ouyangbo/hermes-agent
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

refactor(gateway): dead-code removal, helper unification, defensive-layer collapse and rationale-preserving comment compaction across 40 modules

Browse Source 
authz_mixin, browser_control_broker, delivery, delivery_ledger, display_config, drain_control,
hosted_room_links/peer/policy_checkpoint, hosted_rooms, platform_registry, relay/__init__,
relay/ws_transport, run.py and slash_commands.py (comments), session_context, session_state,
streaming_tts_consumer, turn_lease and small modules.

- HostedRoomPolicyCheckpoint._apply_event -> per-kind handler table
- WebSocketRelayTransport._handle_frame -> frame-handler table
- GatewayAuthorizationMixin: unified adapter setting/flag/extra readers
- dead symbols removed (verified zero references): RoomLinkProbe/select_room_link,
  relay_bot_username, is_restart_loop_tripped, debug_rows, DeadTargetRegistry.all_dead,
  BrowserControlBroker.detach_owner/_prune_tickets, StreamingTTSConsumer.started/_enqueue_done/
  _iter_stream_chunks/_next_stream_chunk, RecoverableHandleCache.status_for, _auth_env,
  _copy_default_catalog, _parse_timestamp_prefix, _present_* helpers, _send_result_error_kind,
  _truthy_env, SessionFieldView/TurnLeaseTokenView dunder shims, and their orphaned tests.
- lost WHY/invariant text from the earlier compaction restored compactly (541 hunks audited)
This commit is contained in:
Teknium
2026-09-02 09:20:51 -07:00
parent a3d33fe22f
commit 3c7069bdcb
49 changed files with 7507 additions and 13727 deletions
Download Patch File Download Diff File Expand all files Collapse all files
gateway/authz_mixin.py
+307 -636
View File
File diff suppressed because it is too large Load Diff
gateway/browser_control_broker.py
+210 -451
View File
@@ -1,66 +1,33 @@
"""Transport-neutral browser-control broker core.
This module is the in-process heart of the browser-control feature: it binds
an *identity-scoped controller* (the party that physically drives a browser)
to *callers* (agents talking to that browser over any transport) without the
broker itself knowing anything about HTTP, WebSocket, or any wire format. The
transport layers built in later phases wrap this core; nothing here routes
traffic.
Binds an identity-scoped *controller* (the party driving a browser) to
*callers* (agents talking to it over any transport) without knowing HTTP,
WebSocket, or any wire format. The browser is a stateful single-owner
resource while the agent side is multi-tenant and multi-transport, so every
rule here makes a class of violation structurally impossible:
Why a broker at all: the browser is a stateful, single-owner resource and the
agent side is multi-tenant (many principals, profiles, sessions) and
multi-transport (local API, remote API, …). A controller must never be
addressable by a caller that merely resembles the right identity, and a
command must never be completable twice, cancellable by a stranger, or
observable after its owner has gone away. Every rule below exists to make one
of those violations structurally impossible rather than merely discouraged.
Contract (each rule is exercised by tests/gateway/test_browser_control_broker.py):
- **Registration tickets are short-lived, single-use, identity-bound, and
cryptographically random.** ``mint_ticket`` returns an opaque value
(``secrets``-derived, >= 32 chars) plus an expiry derived from the injected
clock; ``consume_ticket`` exchanges it exactly once for the
:class:`ControllerScope` it was minted for, raising
:class:`ControllerTicketInvalid` for unknown, already-consumed, or expired values.
The ticket is the only cross-transport credential minted here; transports
- Registration tickets are short-lived, single-use, identity-bound, and
``secrets``-random; ``consume_ticket`` exchanges one exactly once. The
ticket is the only cross-transport credential minted here; transports
decide how to carry it.
- ``select`` matches on *every* stable identity field (principal, profile,
session, controller id, browser profile id, transport family) plus the
controller's currently negotiated capability set; partial matches yield
``None``. A same-identity reconnect renegotiates capabilities in place.
- One pending command per command id; ``complete`` is single-shot (late
completion after cancel/detach returns ``False``).
- ``cancel`` aborts only the command matching scope + tool_call_id.
- ``detach`` fails closed: pending work of that scope raises
:class:`ControllerCancelled` instead of racing a late ``complete``.
- ``disconnect`` marks a transport offline without cancelling running work;
re-attaching the same identity refreshes the callback and may still
deliver the original result. Detach, cancel, identity replacement, and
timeout remain terminal.
- **Exact identity and capability selection.** ``attach`` registers a send
callback under a :class:`ControllerScope`; ``select`` returns a controller
only when the caller's scope matches on *every stable identity field* —
principal, profile, session, controller id, browser profile id, and
transport family — and the requested capability is present in the
controller's current negotiated capability set. Partial matches return
``None``. A same-identity reconnect may renegotiate capabilities without
creating an ambiguous second controller.
- **One pending command per command id; single-shot completion.** Each
``dispatch`` mints a fresh command id, emits one
``browser.controller.command`` frame, and parks a waiter keyed by that id.
``complete`` resolves a command exactly once and returns ``False`` for any
later attempt (late completion after cancellation or detach is ignored).
- **Scoped cancellation.** ``cancel`` aborts only the pending command whose
scope and tool_call_id match, emits a ``browser.controller.cancel`` frame
for it, and returns ``False`` when nothing matched.
- **Detach fails closed.** ``detach`` removes the controller and cancels every
pending command of that scope; waiting dispatchers observe
:class:`ControllerCancelled` rather than hanging or racing a detached
controller's late ``complete``.
- **Unexpected disconnects are recoverable.** ``disconnect`` marks an exact
transport owner offline without accepting new dispatches or cancelling
already-running work. Re-attaching the same stable identity refreshes the
transport callback and can deliver a terminal result for the original
command. Explicit detach, cancel, identity replacement, and timeout remain
terminal boundaries.
Thread-safety: all public state transitions happen under a single reentrant
lock; the send callback is invoked *outside* the lock so a controller may
synchronously ``complete`` from inside its own send (the no-op round trip),
and waiters are parked on per-command events, not on the broker lock.
Thread-safety: all state transitions happen under one reentrant lock; the
send callback is invoked *outside* it so a controller may synchronously
``complete`` from inside its own send, and waiters park on per-command
events, never on the broker lock.
"""
from __future__ import annotations
@@ -83,12 +50,11 @@ DEFAULT_COMMAND_TIMEOUT = 30.0
MAX_DEFERRED_CANCELS = 512
#: Current wire protocol version. Registration requires this exact integer;
#: booleans are rejected even though ``bool`` subclasses ``int`` in Python.
#: booleans are rejected even though ``bool`` subclasses ``int``.
BROWSER_CONTROL_PROTOCOL_VERSION = 1
#: Exact controller capability contract shared by every transport. The broker
#: never accepts arbitrary browser methods: raw CDP, script evaluation, console
#: access, uploads, and other privileged surfaces remain outside this allowlist.
#: Exact controller capability allowlist shared by every transport. Raw CDP,
#: script evaluation, console access, and other privileged surfaces stay out.
BROWSER_CONTROL_CAPABILITIES = frozenset(
{
"controller.noop",
@@ -105,39 +71,20 @@ BROWSER_CONTROL_CAPABILITIES = frozenset(
}
)
#: Privileged capabilities (Phase 8 Task 30) that are never negotiable through
#: the base allowlist. ``browser_evaluate`` executes JavaScript in the page
#: context; ``browser_cdp`` is raw CDP. Both are fail-closed unless the broker
#: runs in Developer Mode (``browser.extension_control.developer_mode``) AND
#: the controller explicitly negotiated the capability.
BROWSER_CONTROL_DEVELOPER_CAPABILITIES = frozenset(
{
"browser_cdp",
"browser_evaluate",
}
)
#: Privileged capabilities (JS eval, raw CDP): fail-closed unless Developer
#: Mode (``browser.extension_control.developer_mode``) is on AND the
#: controller explicitly negotiated them.
BROWSER_CONTROL_DEVELOPER_CAPABILITIES = frozenset({"browser_cdp", "browser_evaluate"})
#: Artifact-transport capabilities (Phase 8 Task 29). These are regular
#: (non-developer) capabilities because upload/download of bounded, validated
#: artifacts is a safe surface; the payloads are never carried in controller
#: frames. Artifact actions are dispatched only after the broker validates the
#: referenced artifact id against the attached store ("approved artifact id
#: only").
#: Artifact transport capabilities. Non-developer because payloads never ride
#: in controller frames: only a store-validated ``artifact_id`` is dispatched.
BROWSER_CONTROL_ARTIFACT_CAPABILITIES = frozenset(
{
"browser_artifact_download",
"browser_artifact_upload",
}
{"browser_artifact_download", "browser_artifact_upload"}
)
#: The complete set a controller may negotiate: base + artifact. Developer
#: capabilities are admitted by :func:`filter_browser_control_capabilities`
#: only when Developer Mode is enabled.
BROWSER_CONTROL_ALL_CAPABILITIES = frozenset(
BROWSER_CONTROL_CAPABILITIES
| BROWSER_CONTROL_ARTIFACT_CAPABILITIES
| BROWSER_CONTROL_DEVELOPER_CAPABILITIES
)
#: Wire method names for controller frames; transports carry them verbatim.
FRAME_COMMAND = "browser.controller.command"
FRAME_CANCEL = "browser.controller.cancel"
def browser_control_protocol_supported(value: Any) -> bool:
@@ -145,18 +92,12 @@ def browser_control_protocol_supported(value: Any) -> bool:
return type(value) is int and value == BROWSER_CONTROL_PROTOCOL_VERSION
def browser_control_developer_mode(config: Optional[dict] = None) -> bool:
"""Return the explicit Developer Mode flag (disabled by default).
Reads ``browser.extension_control.developer_mode`` from the global
config. Developer Mode is the *additional* gate for ``browser_evaluate``
and raw CDP; it never widens the base action allowlist on its own.
"""
def _extension_control_flag(config: Optional[dict], key: str) -> bool:
"""Read ``browser.extension_control.<key>`` as a literal ``True`` (default off)."""
if config is None:
try:
# Read-only flag probe on every browser tool call / check_fn
# evaluation: skip load_config()'s defensive deepcopy (~135us);
# this function only reads nested dicts and never mutates.
# Hot path (every browser tool call / check_fn): the read-only
# loader skips load_config()'s deepcopy; we never mutate.
from hermes_cli.config import load_config_readonly
config = load_config_readonly()
@@ -170,7 +111,17 @@ def browser_control_developer_mode(config: Optional[dict] = None) -> bool:
extension_control = browser.get("extension_control")
if not isinstance(extension_control, dict):
return False
return extension_control.get("developer_mode", False) is True
return extension_control.get(key, False) is True
def browser_control_developer_mode(config: Optional[dict] = None) -> bool:
"""Explicit Developer Mode flag; gates ``browser_evaluate``/raw CDP only."""
return _extension_control_flag(config, "developer_mode")
def browser_control_enabled(config: Optional[dict] = None) -> bool:
"""Return the explicit browser-control feature flag (disabled by default)."""
return _extension_control_flag(config, "enabled")
def filter_browser_control_capabilities(
@@ -180,30 +131,19 @@ def filter_browser_control_capabilities(
) -> frozenset:
"""Return the permitted subset of a JSON/RPC capability list.
A malformed non-list value has no capabilities. Unknown or non-string
entries are ignored; registration rejects an empty returned set.
Base and artifact capabilities always pass. Developer capabilities
(``browser_evaluate``, ``browser_cdp``) pass only when Developer Mode
is explicitly enabled — either passed in or read from the live config.
Non-list input has no capabilities; unknown/non-string entries are
dropped (registration rejects an empty returned set). Developer
capabilities pass only when Developer Mode is on (passed in, else read
from live config).
"""
if not isinstance(value, list):
return frozenset()
allowed = frozenset(BROWSER_CONTROL_CAPABILITIES | BROWSER_CONTROL_ARTIFACT_CAPABILITIES)
allowed = BROWSER_CONTROL_CAPABILITIES | BROWSER_CONTROL_ARTIFACT_CAPABILITIES
if developer_mode is None:
developer_mode = browser_control_developer_mode()
if developer_mode is True:
allowed = frozenset(allowed | BROWSER_CONTROL_DEVELOPER_CAPABILITIES)
return frozenset(
capability
for capability in value
if isinstance(capability, str) and capability in allowed
)
#: Wire method names for controller frames. Transport-neutral by contract:
#: transports carry these envelopes verbatim.
FRAME_COMMAND = "browser.controller.command"
FRAME_CANCEL = "browser.controller.cancel"
allowed = allowed | BROWSER_CONTROL_DEVELOPER_CAPABILITIES
return frozenset(c for c in value if isinstance(c, str) and c in allowed)
class BrowserControlError(Exception):
@@ -232,12 +172,7 @@ class ControllerRejected(BrowserControlError):
@dataclass(frozen=True)
class ControllerScope:
"""Exact identity of a browser controller plus its capability set.
Equality is structural over *all* fields, so two scopes differing in any
single field (including ``transport_family``) never match — this is the
"exact identity" contract.
"""
"""Exact controller identity plus capability set; equality is over all fields."""
principal_id: Optional[str] = None
profile_id: Optional[str] = None
@@ -249,7 +184,7 @@ class ControllerScope:
def _scope_identity(scope: ControllerScope) -> tuple:
"""Return stable controller identity, excluding negotiated capabilities."""
"""Stable identity fields, excluding negotiated capabilities."""
return (
scope.principal_id,
scope.profile_id,
@@ -286,9 +221,9 @@ class _Controller:
owner: Any = None
connected: bool = True
deferred_cancels: list[dict] = field(default_factory=list)
# Serialize command/cancel writes with detach or replacement. Broker state
# is never held while waiting for this lock, so a transport callback may
# synchronously call complete() without deadlocking the broker.
# Serializes command/cancel writes with detach or replacement. Broker
# state is never held while waiting on it, so a transport callback may
# synchronously complete() without deadlocking.
send_lock: threading.Lock = field(default_factory=threading.Lock)
@@ -307,16 +242,7 @@ class _PendingCommand:
class BrowserControlBroker:
"""Thread-safe broker core binding controllers to callers.
Parameters
----------
ticket_ttl:
Lifetime of minted tickets in clock seconds.
command_timeout:
Seconds a ``dispatch`` waits for completion before raising
:class:`ControllerTimeout`.
clock:
Injectable time source (defaults to ``time.monotonic``); tests pin it
to make expiry deterministic.
``clock`` is injectable (default ``time.monotonic``) so tests pin expiry.
"""
def __init__(
@@ -334,18 +260,15 @@ class BrowserControlBroker:
self._tickets: Dict[str, _TicketRecord] = {}
self._controllers: Dict[ControllerScope, _Controller] = {}
self._pending: Dict[str, _PendingCommand] = {}
# Developer Mode gates privileged capabilities (browser_evaluate,
# browser_cdp). None defers to the live config on every selection so
# a mid-process config change is honored without restart — including
# REVOKING raw CDP/eval from already-attached controllers; an
# explicit bool pins the gate for tests and multi-tenant hosts.
# None defers to live config on every selection so flipping
# developer_mode off REVOKES raw CDP/eval from attached controllers
# without restart; an explicit bool pins the gate (tests, multi-tenant).
self._developer_mode_pinned: Optional[bool] = (
None if developer_mode is None else developer_mode is True
)
# Artifact stores keyed by resolved profile id; ``None`` is the
# default/unscoped store (tests, single-profile hosts). A multiplex
# listener attaches one store per profile so profile A touching the
# artifact route first can never pin profile B to A's physical root.
# Artifact stores keyed by profile id; ``None`` is the default slot.
# Per-profile stores keep multiplex profile A from pinning profile B
# to A's physical root.
self._artifact_stores: Dict[Optional[str], Any] = {}
def _developer_mode_now(self) -> bool:
@@ -357,17 +280,20 @@ class BrowserControlBroker:
except Exception:
return False
@property
def developer_mode(self) -> bool:
"""Whether privileged capabilities may be selected/dispatched."""
return self._developer_mode_now()
def attach_artifact_store(
self, store: Any, *, profile_id: Optional[str] = None
) -> None:
"""Attach an artifact store for "approved artifact id only".
"""Attach a store exposing ``validate(artifact_id, *, scope) -> receipt``
(raising the artifacts module's :class:`ArtifactError` subclasses).
``store`` must expose ``validate(artifact_id, *, scope) -> receipt``
raising the artifacts module's :class:`ArtifactError` subclasses.
``profile_id`` scopes the store to one profile on multiplex hosts;
``None`` registers the default store. ``store=None`` clears that
slot; dispatching an artifact action without a resolvable store
fails closed.
``None`` registers the default store. ``store=None`` clears the slot.
Artifact actions without a resolvable store fail closed.
"""
if store is None:
self._artifact_stores.pop(profile_id, None)
@@ -375,22 +301,11 @@ class BrowserControlBroker:
self._artifact_stores[profile_id] = store
def _artifact_store_for_scope(self, scope: "ControllerScope") -> Any:
"""Select the artifact store for one controller scope.
Prefers the exact profile-scoped store, falling back to the default
(``None``) slot so single-profile hosts and existing tests keep the
historical one-store behaviour.
"""
"""Exact profile-scoped store, else the default (``None``) slot so
single-profile hosts keep the historical one-store behaviour."""
profile = getattr(scope, "profile_id", None) or None
store = self._artifact_stores.get(profile)
if store is not None:
return store
return self._artifact_stores.get(None)
@property
def developer_mode(self) -> bool:
"""Whether privileged capabilities may be selected/dispatched."""
return self._developer_mode_now()
return store if store is not None else self._artifact_stores.get(None)
# ------------------------------------------------------------------
# Registration tickets
@@ -400,18 +315,19 @@ class BrowserControlBroker:
"""Mint a short-lived, single-use ticket bound to ``scope``."""
now = self._clock()
with self._lock:
self._prune_tickets(now)
for stale in [v for v, rec in self._tickets.items() if rec.expires_at <= now]:
del self._tickets[stale]
value = secrets.token_urlsafe(32)
record = _TicketRecord(scope=scope, expires_at=now + self._ticket_ttl)
self._tickets[value] = record
return Ticket(value=value, expires_at=record.expires_at)
def consume_ticket(self, value: str) -> ControllerScope:
"""Exchange a ticket for its scope, exactly once.
"""Exchange a ticket for its scope exactly once.
Raises :class:`ControllerTicketInvalid` for unknown, already-consumed, or
expired tickets. The expiry check happens against the live clock at
consume time, so a ticket that outlived its TTL can never be used.
Raises :class:`ControllerTicketInvalid` for unknown, already-consumed,
or expired tickets; expiry is checked against the live clock at
consume time.
"""
now = self._clock()
with self._lock:
@@ -425,16 +341,27 @@ class BrowserControlBroker:
record.consumed = True
return record.scope
def _prune_tickets(self, now: float) -> None:
"""Drop expired tickets (caller must hold the lock)."""
expired = [value for value, rec in self._tickets.items() if rec.expires_at <= now]
for value in expired:
del self._tickets[value]
# ------------------------------------------------------------------
# Controller registration / selection
# ------------------------------------------------------------------
def _controller_for_identity_locked(self, scope: ControllerScope) -> Optional[_Controller]:
"""Attached controller sharing ``scope``'s stable identity (any capabilities)."""
return next(
(c for c in self._controllers.values() if _same_scope_identity(c.scope, scope)),
None,
)
def _lane_scopes_locked(self, session_id: str, principal_id: str, family: str) -> list[ControllerScope]:
"""Attached scopes bound to one session lane (session + principal + transport)."""
return [
scope
for scope in self._controllers
if scope.session_id == session_id
and scope.principal_id == principal_id
and scope.transport_family == family
]
def attach(
self,
scope: ControllerScope,
@@ -444,55 +371,39 @@ class BrowserControlBroker:
) -> None:
"""Attach or refresh the controller for one stable identity.
A reconnect with the same principal/profile/session/controller/browser
profile/transport identity refreshes the send callback and negotiated
capabilities without cancelling pending work. Capabilities are not an
identity field. A different controller or browser profile in the same
authenticated session lane hard-replaces the previous identity.
A same-identity reconnect refreshes the send callback and negotiated
capabilities without cancelling pending work. A different controller
or browser profile in the same authenticated session lane
hard-replaces the previous identity.
"""
while True:
with self._lock:
existing_entry = next(
(
(candidate_scope, controller)
for candidate_scope, controller in self._controllers.items()
if _same_scope_identity(candidate_scope, scope)
),
None,
)
existing = self._controller_for_identity_locked(scope)
lane_scopes = [
candidate_scope
for candidate_scope in self._controllers
if candidate_scope.principal_id == scope.principal_id
and candidate_scope.profile_id == scope.profile_id
and candidate_scope.session_id == scope.session_id
and candidate_scope.transport_family == scope.transport_family
and not _same_scope_identity(candidate_scope, scope)
candidate
for candidate in self._controllers
if candidate.principal_id == scope.principal_id
and candidate.profile_id == scope.profile_id
and candidate.session_id == scope.session_id
and candidate.transport_family == scope.transport_family
and not _same_scope_identity(candidate, scope)
]
if existing_entry is None and not lane_scopes:
self._controllers[scope] = _Controller(
scope=scope,
send=send,
owner=owner,
)
if existing is None and not lane_scopes:
self._controllers[scope] = _Controller(scope=scope, send=send, owner=owner)
return
# A different identity in the same authenticated session lane is a
# hard replacement, not a recoverable reconnect. Terminalize it
# before inserting the successor so session lookup stays unique.
# Hard replacement, not a recoverable reconnect: terminalize the
# lane siblings before inserting so session lookup stays unique.
if lane_scopes:
for lane_scope in lane_scopes:
self.detach(lane_scope, notify_controller=False)
continue
if existing_entry is not None:
existing_scope, existing = existing_entry
with existing.send_lock:
with self._lock:
if self._controllers.get(existing_scope) is not existing:
if self._controllers.get(existing.scope) is not existing:
continue
self._controllers.pop(existing_scope, None)
self._controllers.pop(existing.scope, None)
existing.scope = scope
existing.send = send
existing.owner = owner
@@ -509,17 +420,13 @@ class BrowserControlBroker:
try:
send(frame)
except Exception:
logger.exception(
"failed to flush deferred browser-controller cancel"
)
logger.exception("failed to flush deferred browser-controller cancel")
unsent = deferred[index:]
break
if unsent:
with self._lock:
if self._controllers.get(scope) is existing:
existing.deferred_cancels = unsent[
-MAX_DEFERRED_CANCELS:
]
existing.deferred_cancels = unsent[-MAX_DEFERRED_CANCELS:]
raise ConnectionError(
"browser controller reconnect could not flush deferred cancels"
)
@@ -531,49 +438,28 @@ class BrowserControlBroker:
def select(self, scope: ControllerScope, capability: str) -> Optional[_Controller]:
"""Return the connected controller matching identity and capability.
The caller's capabilities are not authoritative on reconnect. Selection
matches the stable identity fields, then checks the attached
controller's current negotiated capability set. Offline controllers
preserve old pending work but never accept new dispatches.
Privileged capabilities (``browser_evaluate``, ``browser_cdp``) are
additionally gated on Developer Mode: with the gate off they are
never selectable, even when a controller somehow negotiated them.
The gate consults the LIVE flag on every selection (unless pinned at
construction), so flipping ``developer_mode`` off in config revokes
raw CDP/eval from already-attached controllers without a restart.
The caller's capabilities are not authoritative: identity is matched,
then the controller's *current* negotiated set is checked. Offline
controllers never accept new dispatches. Developer capabilities are
additionally gated on the LIVE Developer Mode flag (unless pinned).
"""
if (
capability in BROWSER_CONTROL_DEVELOPER_CAPABILITIES
and not self._developer_mode_now()
):
if capability in BROWSER_CONTROL_DEVELOPER_CAPABILITIES and not self._developer_mode_now():
return None
with self._lock:
matches = [
controller
for controller in self._controllers.values()
if _same_scope_identity(controller.scope, scope)
and controller.connected
and capability in controller.scope.capabilities
]
return matches[0] if len(matches) == 1 else None
controller = self._controller_for_identity_locked(scope)
if controller is None or not controller.connected:
return None
return controller if capability in controller.scope.capabilities else None
def is_owner(self, scope: ControllerScope, owner: Any) -> bool:
"""Return whether ``owner`` is the exact live transport for ``scope``.
"""Whether ``owner`` is the exact live transport for ``scope``.
Ownership is independent of capabilities. Transport handlers use this
for heartbeat and result admission so a least-privilege controller does
not need to request ``controller.noop`` merely to complete a real action.
Independent of capabilities, so a least-privilege controller need not
request ``controller.noop`` merely to heartbeat or complete an action.
"""
with self._lock:
matches = [
controller
for controller in self._controllers.values()
if _same_scope_identity(controller.scope, scope)
and controller.connected
and controller.owner is owner
]
return len(matches) == 1
controller = self._controller_for_identity_locked(scope)
return controller is not None and controller.connected and controller.owner is owner
def disconnect(
self,
@@ -583,20 +469,12 @@ class BrowserControlBroker:
) -> bool:
"""Mark one exact controller transport offline without cancelling work."""
with self._lock:
entry = next(
(
(candidate_scope, controller)
for candidate_scope, controller in self._controllers.items()
if _same_scope_identity(candidate_scope, scope)
),
None,
)
if entry is None:
controller = self._controller_for_identity_locked(scope)
if controller is None:
return False
candidate_scope, controller = entry
with controller.send_lock:
with self._lock:
if self._controllers.get(candidate_scope) is not controller:
if self._controllers.get(controller.scope) is not controller:
return False
if owner is not _OWNER_UNSET and controller.owner is not owner:
return False
@@ -613,10 +491,8 @@ class BrowserControlBroker:
) -> None:
"""Remove the controller for ``scope`` and fail its pending work closed.
Every pending command of the scope is marked cancelled and resolved,
so waiting dispatchers raise :class:`ControllerCancelled`; a late
``complete`` for any of them returns ``False`` (the command id is no
longer pending).
Pending commands resolve cancelled (dispatchers raise
:class:`ControllerCancelled`); a late ``complete`` returns ``False``.
"""
with self._lock:
controller = self._controllers.get(scope)
@@ -634,7 +510,7 @@ class BrowserControlBroker:
pendings = self._pending_for_scope_locked(scope)
for pending in pendings:
self._resolve_pending(pending, cancelled=True)
# Keep the old generation's send lock through cancellation so a
# Hold the old generation's send lock through cancellation so a
# command frame can never overtake its terminal cancel frame.
if notify_controller:
self._emit_cancel_frames(controller, pendings)
@@ -653,25 +529,11 @@ class BrowserControlBroker:
) -> Any:
"""Send one controller command and block for its completion.
Emits a ``browser.controller.command`` frame carrying a fresh command
id, then waits up to ``command_timeout`` seconds. Returns the
controller's completion result, or raises:
- :class:`ControllerUnavailable` — no exact scope/capability match;
- :class:`ControllerCancelled` — cancelled via ``cancel``/``detach``;
- :class:`ControllerTimeout` — no completion within the timeout;
- :class:`ControllerRejected` — completed with ``ok=False``.
Exactly one pending command exists per command id; ``complete`` is
single-shot, so a command can never resolve twice.
Artifact actions (``browser_artifact_upload`` /
``browser_artifact_download``) additionally require an attached
artifact store and a live, scope-bound artifact reference: the
``arguments`` mapping must carry an ``artifact_id`` whose validation
passes against the store ("approved artifact id only"). The payload
is never carried in the frame — only the id travels to the
controller.
Raises :class:`ControllerUnavailable` (no exact match),
:class:`ControllerCancelled`, :class:`ControllerTimeout`, or
:class:`ControllerRejected` (``ok=False``). Artifact actions also
require an attached store and an approved ``artifact_id`` argument;
only the id travels in the frame, never the payload.
"""
controller = self.select(scope, action)
if controller is None:
@@ -702,17 +564,9 @@ class BrowserControlBroker:
)
with controller.send_lock:
with self._lock:
# select() intentionally runs outside the send lock. Revalidate
# the exact live controller after acquiring it so disconnect or
# identity replacement cannot leave a stale command waiting.
attached = next(
(
candidate
for candidate in self._controllers.values()
if _same_scope_identity(candidate.scope, scope)
),
None,
)
# select() ran outside the send lock; revalidate the live
# controller so disconnect/replacement can't strand a command.
attached = self._controller_for_identity_locked(scope)
if attached is not controller or not controller.connected:
raise ControllerUnavailable(
f"controller for scope {scope!r} detached before dispatch"
@@ -723,8 +577,7 @@ class BrowserControlBroker:
try:
controller.send(frame)
except Exception:
# The command never left the building; unreserve the id and
# surface the transport failure to the caller.
# Never left the building: unreserve the id, surface the error.
with self._lock:
self._pending.pop(command_id, None)
raise
@@ -741,16 +594,7 @@ class BrowserControlBroker:
if timed_out:
with controller.send_lock:
with self._lock:
active = next(
(
candidate
for candidate in self._controllers.values()
if _same_scope_identity(candidate.scope, scope)
),
None,
)
if active is None:
active = controller
active = self._controller_for_identity_locked(scope) or controller
if not active.connected:
self._defer_cancel_locked(active, pending)
active = None
@@ -779,10 +623,8 @@ class BrowserControlBroker:
) -> bool:
"""Resolve a pending command by id; ``False`` when none is pending.
Safe to call from inside the controller's own ``send`` callback (the
broker never holds its lock across a send). Late completions — after
``cancel`` or ``detach`` already resolved the command — are ignored
and report ``False``.
Safe from inside the controller's own ``send`` callback. Late
completions after ``cancel``/``detach`` are ignored (``False``).
"""
with self._lock:
pending = self._pending.get(command_id)
@@ -800,43 +642,28 @@ class BrowserControlBroker:
def cancel(self, scope: ControllerScope, *, tool_call_id: Optional[str]) -> bool:
"""Cancel exactly the pending command matching ``scope`` + tool_call_id.
Emits one ``browser.controller.cancel`` frame naming the cancelled
command's id. Returns ``True`` when a command was cancelled and
``False`` when nothing matched (so transports can answer idempotently
without inventing state).
Emits one cancel frame; returns ``False`` when nothing matched so
transports can answer idempotently.
"""
with self._lock:
controller = next(
(
candidate
for candidate in self._controllers.values()
if _same_scope_identity(candidate.scope, scope)
),
None,
)
controller = self._controller_for_identity_locked(scope)
if controller is None or not controller.connected:
return False
with controller.send_lock:
with self._lock:
attached = next(
attached = self._controller_for_identity_locked(scope)
if attached is not controller or not controller.connected:
return False
target = next(
(
candidate
for candidate in self._controllers.values()
if _same_scope_identity(candidate.scope, scope)
p
for p in self._pending.values()
if _same_scope_identity(p.scope, scope)
and p.tool_call_id == tool_call_id
and not p.done
),
None,
)
if attached is not controller or not controller.connected:
return False
target = None
for pending in self._pending.values():
if (
_same_scope_identity(pending.scope, scope)
and pending.tool_call_id == tool_call_id
and not pending.done
):
target = pending
break
if target is None:
return False
self._resolve_pending(target, cancelled=True)
@@ -844,11 +671,10 @@ class BrowserControlBroker:
return True
# ------------------------------------------------------------------
# Internals (all callers must hold the lock)
# Internals (callers hold the lock unless noted)
# ------------------------------------------------------------------
def _resolve_pending(self, pending: _PendingCommand, *, cancelled: bool) -> None:
"""Mark ``pending`` resolved and drop it from the registry."""
pending.cancelled = cancelled
pending.done = True
del self._pending[pending.command_id]
@@ -860,19 +686,14 @@ class BrowserControlBroker:
action: str,
arguments: dict,
) -> None:
"""Fail closed unless ``arguments`` carries an approved artifact id.
"""Fail closed unless ``arguments`` carries a store-approved artifact id.
The store is consulted through the duck-typed ``validate`` contract
(raises :class:`ArtifactError` subclasses on any problem), so the
broker never guesses at artifact validity: missing store, missing id,
traversal, expiry, checksum, or scope mismatch all surface as
:class:`ControllerRejected` before any frame is emitted.
Missing store/id, traversal, expiry, checksum, or scope mismatch all
surface as :class:`ControllerRejected` before any frame is emitted.
"""
store = self._artifact_store_for_scope(scope)
if store is None:
raise ControllerRejected(
f"{action} requires an attached artifact store"
)
raise ControllerRejected(f"{action} requires an attached artifact store")
artifact_id = arguments.get("artifact_id")
if not isinstance(artifact_id, str) or not artifact_id.strip():
raise ControllerRejected(f"{action} requires a non-empty artifact_id")
@@ -895,34 +716,37 @@ class BrowserControlBroker:
},
}
def _defer_cancel_locked(
self,
controller: _Controller,
pending: _PendingCommand,
) -> None:
def _defer_cancel_locked(self, controller: _Controller, pending: _PendingCommand) -> None:
controller.deferred_cancels.append(self._cancel_frame(pending))
if len(controller.deferred_cancels) > MAX_DEFERRED_CANCELS:
del controller.deferred_cancels[:-MAX_DEFERRED_CANCELS]
def _pending_for_scope_locked(self, scope: ControllerScope) -> list[_PendingCommand]:
return [
pending
for pending in list(self._pending.values())
if _same_scope_identity(pending.scope, scope)
]
return [p for p in list(self._pending.values()) if _same_scope_identity(p.scope, scope)]
def _emit_cancel_frames(
self, controller: _Controller, pendings: list[_PendingCommand]
) -> None:
def _emit_cancel_frames(self, controller: _Controller, pendings: list[_PendingCommand]) -> None:
"""Send cancel frames (caller holds ``send_lock``, never the broker lock)."""
for pending in pendings:
frame = self._cancel_frame(pending)
try:
controller.send(frame)
controller.send(self._cancel_frame(pending))
except Exception:
logger.exception(
"failed to emit cancel frame for command %r", pending.command_id
)
# ------------------------------------------------------------------
# Session-lane lookups / bulk teardown
# ------------------------------------------------------------------
@staticmethod
def _lane_key(session_id, task_id, principal_id, transport_family) -> Optional[tuple[str, str, str]]:
target = str(session_id or task_id or "").strip()
principal = str(principal_id or "").strip()
family = str(transport_family or "").strip()
if not target or not principal or not family:
return None
return target, principal, family
def scope_for_session(
self,
*,
@@ -933,24 +757,15 @@ class BrowserControlBroker:
) -> Optional[ControllerScope]:
"""Return one unambiguous attached scope for a server-owned session.
A public session id is only a lookup hint. The caller must also supply
its server-derived principal and transport family; missing identity,
no match, or multiple matches fail closed rather than selecting by
insertion order.
The public session id is only a hint; the caller must also supply its
server-derived principal and transport family. Missing identity, no
match, or multiple matches fail closed rather than picking by order.
"""
target = str(session_id or task_id or "").strip()
principal = str(principal_id or "").strip()
family = str(transport_family or "").strip()
if not target or not principal or not family:
key = self._lane_key(session_id, task_id, principal_id, transport_family)
if key is None:
return None
with self._lock:
matches = [
scope
for scope in self._controllers
if scope.session_id == target
and scope.principal_id == principal
and scope.transport_family == family
]
matches = self._lane_scopes_locked(*key)
return matches[0] if len(matches) == 1 else None
def lane_registered(
@@ -961,55 +776,23 @@ class BrowserControlBroker:
principal_id: Optional[str] = None,
transport_family: Optional[str] = None,
) -> bool:
"""Return whether ANY controller (even offline) registered for this lane.
"""Whether ANY controller (even offline) registered for this lane.
Distinguishes "a controller bound this session lane and is currently
unavailable" (fail closed — the extension lane stays authoritative)
from "no controller ever registered here" (the caller keeps the
legacy browser backend). Ambiguous lanes report True so the caller
still fails closed rather than silently switching browsers.
Distinguishes "lane bound but currently unavailable" (fail closed;
the extension lane stays authoritative) from "never registered"
(caller keeps the legacy backend). Ambiguous lanes report True.
"""
target = str(session_id or task_id or "").strip()
principal = str(principal_id or "").strip()
family = str(transport_family or "").strip()
if not target or not principal or not family:
key = self._lane_key(session_id, task_id, principal_id, transport_family)
if key is None:
return False
with self._lock:
return any(
scope.session_id == target
and scope.principal_id == principal
and scope.transport_family == family
for scope in self._controllers
)
return bool(self._lane_scopes_locked(*key))
def disconnect_owner(self, owner: Any) -> int:
"""Mark every controller owned by one lost transport offline."""
with self._lock:
scopes = [
scope
for scope, controller in self._controllers.items()
if controller.owner is owner
]
disconnected = 0
for scope in scopes:
disconnected += int(self.disconnect(scope, owner=owner))
return disconnected
def detach_owner(self, owner: Any, *, notify_controller: bool = True) -> int:
"""Hard-detach every controller owned by one transport connection."""
with self._lock:
scopes = [
scope
for scope, controller in self._controllers.items()
if controller.owner is owner
]
for scope in scopes:
self.detach(
scope,
owner=owner,
notify_controller=notify_controller,
)
return len(scopes)
scopes = [s for s, c in self._controllers.items() if c.owner is owner]
return sum(int(self.disconnect(scope, owner=owner)) for scope in scopes)
def reset(self) -> None:
"""Fail all live work closed and clear tickets (tests/shutdown)."""
@@ -1019,8 +802,7 @@ class BrowserControlBroker:
self.detach(scope)
with self._lock:
self._tickets.clear()
# Defensive cleanup for any pending entry whose controller was
# concurrently removed by a transport teardown.
# Pending entries whose controller a concurrent teardown removed.
for pending in list(self._pending.values()):
self._resolve_pending(pending, cancelled=True)
@@ -1042,26 +824,3 @@ _GLOBAL_BROKER = BrowserControlBroker()
def get_browser_control_broker() -> BrowserControlBroker:
"""Process-local broker shared by API and dashboard Gateway transports."""
return _GLOBAL_BROKER
def browser_control_enabled(config: Optional[dict] = None) -> bool:
"""Return the explicit browser-control feature flag (disabled by default)."""
if config is None:
try:
# Read-only flag probe on every browser tool call / check_fn
# evaluation: skip load_config()'s defensive deepcopy (~135us);
# this function only reads nested dicts and never mutates.
from hermes_cli.config import load_config_readonly
config = load_config_readonly()
except Exception:
return False
if not isinstance(config, dict):
return False
browser = config.get("browser")
if not isinstance(browser, dict):
return False
extension_control = browser.get("extension_control")
if not isinstance(extension_control, dict):
return False
return extension_control.get("enabled", False) is True
gateway/code_skew.py
+11 -18
View File
@@ -1,18 +1,15 @@
"""Detect when the gateway is running stale code after a hot ``git pull``.
The gateway is a single long-lived process; its ``sys.modules`` is frozen at
boot. If the checkout is updated underneath it (a manual ``git pull``, or the
window before ``hermes update``'s graceful restart fires), a first-time lazy
import on a new code path can resolve a freshly-pulled consumer module against a
stale cached dependency -> ImportError (see
``tests/test_stale_utils_module_import.py`` for the exact failure).
The gateway's ``sys.modules`` is frozen at boot. If the checkout is updated
underneath it (manual ``git pull``, or the window before ``hermes update``'s
graceful restart), a first-time lazy import can resolve a freshly-pulled module
against a stale cached dependency -> ImportError
(``tests/test_stale_utils_module_import.py``). We snapshot the revision at
startup so risky callers (e.g. ``/model`` switching) can refuse with a clear
"restart the gateway" message instead.
We snapshot the checkout revision at gateway startup and compare on demand, so
risky callers (e.g. ``/model`` switching) can refuse with a clear "restart the
gateway" message instead of crashing on a cryptic import error.
If the revision can't be read (non-git install, IO error), the boot snapshot
stays ``None`` and skew detection no-ops — it never produces a false positive.
If the revision can't be read (non-git install, IO error) the boot snapshot
stays ``None`` and detection no-ops — never a false positive.
"""
from __future__ import annotations
@@ -24,12 +21,8 @@ _boot_fingerprint: str | None = None
def _fingerprint() -> str | None:
"""Current checkout fingerprint, reusing the CLI's git-rev reader.
``hermes_cli.main`` is always already imported in a gateway process (it's
the entry point), so this import is free and avoids duplicating the
worktree-aware ref resolution.
"""
"""Current checkout fingerprint via the CLI's worktree-aware git-rev reader
(``hermes_cli.main`` is always already imported in a gateway process)."""
try:
from hermes_cli.main import _read_git_revision_fingerprint
gateway/cwd_placeholder.py
+3 -11
View File
@@ -12,10 +12,6 @@ from __future__ import annotations
CWD_PLACEHOLDERS = frozenset({".", "auto", "cwd"})
def _truthy_env(value: str | None) -> bool:
return (value or "").strip().lower() in {"true", "1", "yes"}
def resolve_placeholder_terminal_cwd(
*,
configured_cwd: str,
@@ -37,13 +33,9 @@ def resolve_placeholder_terminal_cwd(
return configured_cwd
backend = (terminal_backend or "local").strip().lower()
messaging = (messaging_cwd or "").strip()
if backend == "local":
messaging = (messaging_cwd or "").strip()
return messaging or home_fallback
if backend == "docker" and docker_mount_cwd_to_workspace:
messaging = (messaging_cwd or "").strip()
if messaging and messaging not in CWD_PLACEHOLDERS:
return messaging
if backend == "docker" and docker_mount_cwd_to_workspace and messaging and messaging not in CWD_PLACEHOLDERS:
return messaging
return None
gateway/dead_targets.py
+22 -45
View File
@@ -1,25 +1,18 @@
"""Persistent registry of delivery targets that are confirmed unreachable.
When a messaging platform reports that a target chat is permanently gone — a
deleted group (``Forbidden: the group chat was deleted``), a bot kicked/blocked,
or a deactivated user — re-sending to it on every cron tick or every fan-out
delivery wastes a send attempt against the platform's flood-control envelope and
spams the logs. This registry lets the delivery layer short-circuit a target it
has already proven dead, while staying self-healing: any successful send to that
target clears the flag, so a user who re-adds the bot (or restores the chat)
recovers automatically with no manual cleanup.
When a platform reports a target chat is permanently gone (deleted group,
bot kicked/blocked, deactivated user), re-sending on every cron tick wastes a
send against flood control and spams logs. The delivery layer short-circuits
targets proven dead; any later successful send clears the flag (self-healing).
Scope is deliberately narrow. Only *whole-chat* deaths are recorded — the
``forbidden`` and chat-level ``not_found`` (``chat not found``) error kinds.
Thread/topic-level ``not_found`` is NOT recorded here: the adapters already
self-heal that by retrying without ``reply_to`` (see the Telegram adapter's
reply-target-deleted path), and a deleted topic does not mean the parent chat is
dead.
Scope is deliberately narrow: only *whole-chat* deaths (``forbidden`` and
chat-level ``not_found``) are recorded. Thread/topic-level ``not_found`` is
NOT — adapters self-heal that by retrying without ``reply_to``, and a deleted
topic does not mean the parent chat is dead.
The store is a small JSON file under the active profile's HERMES_HOME so each
profile keeps its own dead set. Reads/writes are best-effort: a corrupt or
unwritable file degrades to an in-memory-only registry rather than raising on
the delivery path.
Storage is a JSON file under the active profile's HERMES_HOME (each profile
keeps its own dead set). Reads/writes are best-effort: a corrupt or unwritable
file degrades to in-memory-only rather than raising on the delivery path.
"""
from __future__ import annotations
@@ -35,8 +28,8 @@ from hermes_cli.config import get_hermes_home
logger = logging.getLogger(__name__)
# Error kinds (from gateway.platforms.base.classify_send_error) that mean the
# *whole chat* is unreachable, not a transient or thread-level problem.
# Error kinds (gateway.platforms.base.classify_send_error) meaning the whole
# chat is unreachable, not a transient or thread-level problem.
_DEAD_ERROR_KINDS = frozenset({"forbidden", "not_found"})
@@ -46,33 +39,24 @@ def _normalize(platform: str, chat_id: str) -> str:
class DeadTargetRegistry:
"""Thread-safe, persistent set of confirmed-dead delivery targets.
"""Thread-safe, persistent set of confirmed-dead targets keyed ``platform:chat_id``.
Keyed on ``platform:chat_id``. Stores the reason and a timestamp for
observability. Self-healing: :meth:`clear` (called on a successful send)
removes the flag.
Each entry stores reason + timestamp for observability; :meth:`clear`
(called on a successful send) removes the flag.
"""
def __init__(self, path: Optional[Path] = None) -> None:
self._lock = threading.RLock()
self._dead: Dict[str, Dict[str, object]] = {}
if path is not None:
self._path = path
else:
self._path = get_hermes_home() / "gateway" / "dead_targets.json"
self._path = path if path is not None else get_hermes_home() / "gateway" / "dead_targets.json"
self._load()
# -- persistence -------------------------------------------------------
def _load(self) -> None:
try:
if self._path.exists():
raw = json.loads(self._path.read_text(encoding="utf-8"))
if isinstance(raw, dict):
# Only keep well-shaped entries.
self._dead = {
k: v for k, v in raw.items() if isinstance(v, dict)
}
self._dead = {k: v for k, v in raw.items() if isinstance(v, dict)}
except (OSError, ValueError) as exc:
logger.debug("dead_targets: could not load %s (%s) — starting empty",
self._path, exc)
@@ -85,14 +69,12 @@ class DeadTargetRegistry:
tmp.write_text(json.dumps(self._dead, indent=2), encoding="utf-8")
tmp.replace(self._path)
except OSError as exc:
# Best-effort: keep the in-memory state, don't break delivery.
# Best-effort: keep in-memory state, never break delivery.
logger.debug("dead_targets: could not persist %s (%s)", self._path, exc)
# -- public API --------------------------------------------------------
@staticmethod
def is_dead_error_kind(error_kind: Optional[str]) -> bool:
"""Return True when ``error_kind`` denotes a permanent whole-chat death."""
"""True when ``error_kind`` denotes a permanent whole-chat death."""
return bool(error_kind) and error_kind in _DEAD_ERROR_KINDS
def is_dead(self, platform: str, chat_id: Optional[str]) -> bool:
@@ -103,7 +85,7 @@ class DeadTargetRegistry:
def mark_dead(self, platform: str, chat_id: Optional[str],
reason: str = "") -> bool:
"""Record a target as confirmed-dead. Returns True if newly added."""
"""Record a target as confirmed-dead. Returns True if newly added."""
if not chat_id:
return False
key = _normalize(platform, chat_id)
@@ -125,7 +107,7 @@ class DeadTargetRegistry:
return not existed
def clear(self, platform: str, chat_id: Optional[str]) -> bool:
"""Remove a target's dead flag (self-healing). Returns True if it was set."""
"""Remove a target's dead flag (self-healing). Returns True if it was set."""
if not chat_id:
return False
key = _normalize(platform, chat_id)
@@ -136,8 +118,3 @@ class DeadTargetRegistry:
logger.info("dead_targets: cleared %s (delivery succeeded again)", key)
return True
return False
def all_dead(self) -> Dict[str, Dict[str, object]]:
"""Snapshot of the current dead set (for diagnostics / `hermes` CLI)."""
with self._lock:
return {k: dict(v) for k, v in self._dead.items()}
gateway/delivery.py
+193 -343
View File
@@ -1,11 +1,9 @@
"""
Delivery routing for cron job outputs and agent responses.
Routes messages to the appropriate destination based on:
- Explicit targets (e.g., "telegram:123456789")
- Platform home channels (e.g., "telegram" → home channel)
- Origin (back to where the job was created)
- Local (always saved to files)
Routes messages by target: explicit ("telegram:123456789"), platform home
channel ("telegram"), origin (back to where the job was created), or local
(saved to files).
"""
import logging
@@ -21,18 +19,15 @@ from hermes_cli.config import get_hermes_home
logger = logging.getLogger(__name__)
# Cap before gateway-level truncation of cron output for non-chunking platform
# delivery. Telegram's hard API limit is 4096; the headroom covers the "full
# output saved to …" footer appended on truncation. Adapters that split long
# messages natively (BasePlatformAdapter.splits_long_messages) bypass this
# entirely — the adapter chunks in its own send() and the full output is
# preserved.
# delivery. Telegram's hard API limit is 4096; the headroom covers the "full
# output saved to …" footer. Adapters that split long messages natively
# (BasePlatformAdapter.splits_long_messages) bypass this entirely.
MAX_PLATFORM_OUTPUT = 4000
# Matches strings that are *only* a "silence" narration with optional markdown
# wrappers. Covers: *(silent)*, _silent_, `silent`, ~silent~, (silent), silent,
# 🔇, a bare ".", "…", and the whitespace/marker-padded variants seen in the
# wild. Anchored to start/end so substantive messages that merely *contain* the
# word "silent" are never matched.
# wrappers: *(silent)*, _silent_, `silent`, (silent), silent, 🔇, a bare ".",
# "…", and marker-padded variants. Anchored so substantive messages that merely
# *contain* the word "silent" never match.
_SILENCE_NARRATION = re.compile(
r'^[\s*_~`]*\(?\s*(silent|silence|no\s+response|no\s+reply)\s*\.?\)?[\s*_~`]*$'
r'|^[\s*_~`]*[\U0001F507\.\u2026]+[\s*_~`]*$',
@@ -41,16 +36,11 @@ _SILENCE_NARRATION = re.compile(
def _is_silence_narration(content: Optional[str]) -> bool:
"""Return True when ``content`` is *only* a silence-narration token.
Length-guarded (real messages are longer) and anchored to the whole string
so legitimate prose like "The deployment ran silently" or "Silence is
golden — here is the plan..." is never flagged.
"""
"""True when ``content`` is *only* a silence-narration token (length-guarded)."""
if not content:
return False
stripped = content.strip()
if not stripped or len(stripped) > 64: # length guard
if not stripped or len(stripped) > 64:
return False
return bool(_SILENCE_NARRATION.match(stripped))
@@ -81,10 +71,7 @@ class DeliveryTransport:
"""Send through this transport while preserving the logical platform."""
if self.is_relay:
return await self.adapter.send_for_platform(
logical_platform,
chat_id,
content,
metadata=metadata,
logical_platform, chat_id, content, metadata=metadata,
)
return await self.adapter.send(chat_id, content, metadata=metadata)
@@ -98,21 +85,16 @@ def resolve_delivery_transport(
A concrete native adapter always wins. Relay is eligible only when its
authenticated transport explicitly advertises that it fronts the logical
platform, which keeps restart-time delivery independent of per-chat caches
without letting Relay hijack unrelated platform targets.
platform, so restart-time delivery is independent of per-chat caches without
letting Relay hijack unrelated platform targets.
"""
live_adapters = adapters or {}
native = live_adapters.get(platform)
native_config = config.platforms.get(platform)
# Preserve DeliveryRouter's historical support for explicitly supplied live
# adapters with no config block, but never let an explicitly disabled native
# adapter shadow an enabled Relay transport.
# Explicitly supplied live adapters with no config block are honored, but an
# explicitly disabled native adapter never shadows an enabled Relay transport.
if native is not None and (native_config is None or native_config.enabled):
return DeliveryTransport(
adapter=native,
config=native_config,
transport_platform=platform,
)
return DeliveryTransport(native, native_config, platform)
relay = live_adapters.get(Platform.RELAY)
relay_config = config.platforms.get(Platform.RELAY)
@@ -123,51 +105,47 @@ def resolve_delivery_transport(
and callable(fronts_platform)
and fronts_platform(platform)
):
return DeliveryTransport(
adapter=relay,
config=relay_config,
transport_platform=Platform.RELAY,
)
return DeliveryTransport(relay, relay_config, Platform.RELAY)
return None
def _as_int(value: Any) -> Optional[int]:
try:
return int(value)
except (TypeError, ValueError):
return None
def looks_like_telegram_private_chat_id(chat_id: Optional[str]) -> bool:
"""True when ``chat_id`` is a positive int — Telegram's private-chat shape.
Telegram private chats use positive chat IDs; groups/channels/supergroups
use negative IDs. This is the single source of truth for that heuristic,
reused by the handoff seed path in ``gateway/run.py`` so handoff-created
DM topics key the same way as inbound DM-topic messages.
Groups/channels/supergroups use negative IDs. Single source of truth for the
heuristic, reused by the handoff seed path in ``gateway/run.py`` so
handoff-created DM topics key the same way as inbound DM-topic messages.
"""
if chat_id is None:
return False
try:
return int(chat_id) > 0
except (TypeError, ValueError):
return False
parsed = _as_int(chat_id)
return parsed is not None and parsed > 0
def _looks_like_int(value: Optional[str]) -> bool:
if value is None:
return False
try:
int(value)
return True
except (TypeError, ValueError):
return False
return value is not None and _as_int(value) is not None
def _result_field(result: Any, name: str, default: Any = None) -> Any:
"""Read ``name`` from a SendResult object or a plain result dict."""
if isinstance(result, dict):
return result.get(name, default)
return getattr(result, name, default)
def _send_result_failed(result: Any) -> bool:
if isinstance(result, dict):
return result.get("success") is False
return getattr(result, "success", True) is False
return _result_field(result, "success", True) is False
def _send_result_error(result: Any) -> Optional[str]:
if isinstance(result, dict):
error = result.get("error")
else:
error = getattr(result, "error", None)
error = _result_field(result, "error")
return str(error) if error else None
@@ -176,21 +154,12 @@ def _is_thread_not_found_delivery_error(result: Any) -> bool:
return bool(error and "thread not found" in error.lower())
def _send_result_error_kind(result: Any) -> Optional[str]:
"""Return the machine-readable error_kind from a SendResult/dict, if any."""
if isinstance(result, dict):
kind = result.get("error_kind")
else:
kind = getattr(result, "error_kind", None)
return str(kind) if kind else None
def _classify_dead_from_error_text(error_text: Optional[str]) -> Optional[str]:
"""Best-effort dead-target classification from a raised error's text.
``_deliver_to_platform`` raises (it does not return a SendResult) on a hard
failure, so the ``deliver()`` loop only has the exception string. Reuse the
platform-neutral classifier to recover the error_kind from that text.
``_deliver_to_platform`` raises on hard failure (no SendResult), so the
``deliver()`` loop only has the exception string; reuse the platform-neutral
classifier to recover the error_kind from it.
"""
if not error_text:
return None
@@ -202,9 +171,9 @@ def _classify_dead_from_error_text(error_text: Optional[str]) -> Optional[str]:
if not DeadTargetRegistry.is_dead_error_kind(kind):
return None
# ``not_found`` collapses chat-level and thread/topic/message-level failures.
# Only a whole-chat not_found means the target is dead — a deleted forum topic
# or an edited-away message must not mark the entire chat (and all of its future
# deliveries) dead. See gateway.dead_targets' documented scope.
# Only a whole-chat not_found means the target is dead — a deleted forum
# topic or an edited-away message must not mark the entire chat (and all its
# future deliveries) dead (see gateway.dead_targets).
if kind == "not_found" and not is_chat_level_not_found(error_text=error_text):
return None
return kind
@@ -212,35 +181,20 @@ def _classify_dead_from_error_text(error_text: Optional[str]) -> Optional[str]:
@dataclass
class DeliveryTarget:
"""
A single delivery target.
Represents where a message should be sent:
- "origin" → back to source
- "local" → save to local files
- "telegram" → Telegram home channel
- "telegram:123456" → specific Telegram chat
"""
"""A single delivery target: "origin", "local", "telegram" (home channel),
or "telegram:123456[:thread]" (specific chat)."""
platform: Platform
chat_id: Optional[str] = None # None means use home channel
thread_id: Optional[str] = None
is_origin: bool = False
is_explicit: bool = False # True if chat_id was explicitly specified
@classmethod
def parse(cls, target: str, origin: Optional[SessionSource] = None) -> "DeliveryTarget":
"""
Parse a delivery target string.
Formats:
- "origin" → back to source
- "local" → local files only
- "telegram" → Telegram home channel
- "telegram:123456" → specific Telegram chat
"""
"""Parse "origin" | "local" | "<platform>" | "<platform>:<chat_id>[:<thread_id>]"."""
target_stripped = target.strip()
target_lower = target_stripped.lower()
if target_lower == "origin":
if origin:
return cls(
@@ -249,35 +203,28 @@ class DeliveryTarget:
thread_id=origin.thread_id,
is_origin=True,
)
else:
# Fallback to local if no origin
return cls(platform=Platform.LOCAL, is_origin=True)
return cls(platform=Platform.LOCAL, is_origin=True)
if target_lower == "local":
return cls(platform=Platform.LOCAL)
# Check for platform:chat_id or platform:chat_id:thread_id format
# Use the original case for chat_id/thread_id to preserve case-sensitive IDs
# Platform names are case-insensitive; chat/thread ids keep original case.
# Unknown platforms are treated as local.
if ":" in target_stripped:
parts = target_stripped.split(":", 2)
platform_str = parts[0].lower() # Platform names are case-insensitive
chat_id = parts[1] if len(parts) > 1 else None
thread_id = parts[2] if len(parts) > 2 else None
try:
platform = Platform(platform_str)
return cls(platform=platform, chat_id=chat_id, thread_id=thread_id, is_explicit=True)
platform = Platform(parts[0].lower())
except ValueError:
# Unknown platform, treat as local
return cls(platform=Platform.LOCAL)
# Just a platform name (use home channel)
return cls(platform=platform, chat_id=chat_id, thread_id=thread_id, is_explicit=True)
try:
platform = Platform(target_lower)
return cls(platform=platform)
return cls(platform=Platform(target_lower))
except ValueError:
# Unknown platform, treat as local
return cls(platform=Platform.LOCAL)
def to_string(self) -> str:
"""Convert back to string format."""
if self.is_origin:
@@ -291,30 +238,33 @@ class DeliveryTarget:
return self.platform.value
async def _ensure_named_dm_topic(adapter: Any, chat_id: str, name: str, *, refresh: bool) -> str:
"""Create (or force-recreate) a named Telegram private DM topic; return its thread id."""
verb = "refresh" if refresh else "create"
ensure_dm_topic = getattr(adapter, "ensure_dm_topic", None)
if ensure_dm_topic is None:
raise RuntimeError(f"Telegram adapter cannot {verb} named private DM topics")
if refresh:
thread_id = await ensure_dm_topic(chat_id, name, force_create=True)
else:
thread_id = await ensure_dm_topic(chat_id, name)
if not thread_id:
raise RuntimeError(f"Failed to {verb} Telegram private DM topic '{name}'")
return str(thread_id)
class DeliveryRouter:
"""
Routes messages to appropriate destinations.
Handles the logic of resolving delivery targets and dispatching
messages to the right platform adapters.
"""
"""Resolves delivery targets and dispatches messages to platform adapters."""
def __init__(self, config: GatewayConfig, adapters: Dict[Platform, Any] = None,
dead_targets: Optional[DeadTargetRegistry] = None):
"""
Initialize the delivery router.
Args:
config: Gateway configuration
adapters: Dict mapping platforms to their adapter instances
dead_targets: Optional shared registry of confirmed-unreachable
targets. When omitted, a profile-local registry is created.
"""
"""``dead_targets``: shared registry of confirmed-unreachable targets;
a profile-local registry is created when omitted."""
self.config = config
self.adapters = adapters or {}
self.output_dir = get_hermes_home() / "cron" / "output"
self.dead_targets = dead_targets or DeadTargetRegistry()
async def deliver(
self,
content: str,
@@ -323,32 +273,16 @@ class DeliveryRouter:
job_name: Optional[str] = None,
metadata: Optional[Dict[str, Any]] = None
) -> Dict[str, Any]:
"""
Deliver content to all specified targets.
Args:
content: The message/output to deliver
targets: List of delivery targets
job_id: Optional job ID (for cron jobs)
job_name: Optional job name
metadata: Additional metadata to include
Returns:
Dict with delivery results per target
"""
"""Deliver content to all targets; returns per-target results keyed by target string."""
results = {}
for target in targets:
# Skip targets we've already proven permanently unreachable
# (deleted group, blocked/kicked bot, deactivated user). Re-sending
# to them on every tick wastes a send against flood control and
# spams logs. Self-healing: a later successful send clears the flag.
# Skip targets proven permanently unreachable (deleted group, blocked
# bot, deactivated user) — re-sending each tick wastes flood-control
# budget. Self-healing: a later successful send clears the flag.
# LOCAL/origin-without-chat targets are never dead-tracked.
if (
target.platform != Platform.LOCAL
and target.chat_id
and self.dead_targets.is_dead(target.platform.value, target.chat_id)
):
tracked = target.platform != Platform.LOCAL and target.chat_id
if tracked and self.dead_targets.is_dead(target.platform.value, target.chat_id):
logger.info(
"Skipping delivery to known-dead target %s:%s "
"(send to it again to clear)",
@@ -365,31 +299,23 @@ class DeliveryRouter:
result = self._deliver_local(content, job_id, job_name, metadata)
else:
result = await self._deliver_to_platform(target, content, metadata)
# Successful platform delivery — clear any stale dead flag.
if target.chat_id and not _send_result_failed(result):
self.dead_targets.clear(target.platform.value, target.chat_id)
results[target.to_string()] = {
"success": True,
"result": result
}
results[target.to_string()] = {"success": True, "result": result}
except Exception as e:
# A hard failure raises here. If the platform reported a
# whole-chat death, record it so future deliveries short-circuit.
if target.platform != Platform.LOCAL and target.chat_id:
# Hard failures raise. Record a whole-chat death so future
# deliveries short-circuit.
if tracked:
dead_kind = _classify_dead_from_error_text(str(e))
if dead_kind:
self.dead_targets.mark_dead(
target.platform.value, target.chat_id,
reason=f"{dead_kind}: {str(e)[:120]}",
)
results[target.to_string()] = {
"success": False,
"error": str(e)
}
results[target.to_string()] = {"success": False, "error": str(e)}
return results
def _deliver_local(
self,
content: str,
@@ -399,43 +325,23 @@ class DeliveryRouter:
) -> Dict[str, Any]:
"""Save content to local files."""
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
if job_id:
output_path = self.output_dir / job_id / f"{timestamp}.md"
else:
output_path = self.output_dir / "misc" / f"{timestamp}.md"
output_path = self.output_dir / (job_id or "misc") / f"{timestamp}.md"
output_path.parent.mkdir(parents=True, exist_ok=True)
# Build the output document
lines = []
if job_name:
lines.append(f"# {job_name}")
else:
lines.append("# Delivery Output")
lines.append("")
lines.append(f"**Timestamp:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
lines = [
f"# {job_name}" if job_name else "# Delivery Output",
"",
f"**Timestamp:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
]
if job_id:
lines.append(f"**Job ID:** {job_id}")
if metadata:
for key, value in metadata.items():
lines.append(f"**{key}:** {value}")
lines.append("")
lines.append("---")
lines.append("")
lines.append(content)
for key, value in (metadata or {}).items():
lines.append(f"**{key}:** {value}")
lines += ["", "---", "", content]
output_path.write_text("\n".join(lines), encoding="utf-8")
return {
"path": str(output_path),
"timestamp": timestamp
}
return {"path": str(output_path), "timestamp": timestamp}
def _save_full_output(self, content: str, job_id: str) -> Path:
"""Save full cron output to disk and return the file path."""
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
@@ -446,17 +352,55 @@ class DeliveryRouter:
return path
def _filter_silence_narration_enabled(self) -> bool:
"""Whether the outbound silence-narration filter is active.
``HERMES_FILTER_SILENCE_NARRATION`` env var overrides config when set;
otherwise the ``gateway.filter_silence_narration`` config flag wins
(default True).
"""
"""``HERMES_FILTER_SILENCE_NARRATION`` env overrides the
``gateway.filter_silence_narration`` config flag (default True)."""
env = os.getenv("HERMES_FILTER_SILENCE_NARRATION")
if env is not None:
return env.strip().lower() in ("1", "true", "yes", "on")
return bool(getattr(self.config, "filter_silence_narration", True))
def _cap_oversized_output(self, adapter: Any, content: str, job_id: str) -> str:
"""Audit-save oversized cron output; truncate it for non-chunking adapters.
Two independent decisions: (1) above MAX_PLATFORM_OUTPUT the full output
is always written to disk as an audit trail, best-effort — a failed save
(full disk, permissions) never blocks delivery; (2) non-chunking adapters
get the content truncated with a footer pointing to the saved file, while
``splits_long_messages`` adapters receive the full payload.
"""
if len(content) <= MAX_PLATFORM_OUTPUT:
return content
saved_path: Optional[Path] = None
try:
saved_path = self._save_full_output(content, job_id)
except OSError as exc:
logger.warning(
"Audit save failed for cron output (%d chars, job=%s): %s — "
"delivery proceeds without audit copy",
len(content), job_id, exc,
)
if getattr(adapter, "splits_long_messages", False):
if saved_path:
logger.info(
"Cron output preserved for chunking adapter (%d chars) — "
"full output saved to %s",
len(content), saved_path,
)
return content
# The footer needs a valid path: if the best-effort save failed, retry
# (a failure now is a real delivery problem and propagates).
if saved_path is None:
saved_path = self._save_full_output(content, job_id)
footer = f"\n\n... [truncated, full output saved to {saved_path}]"
visible = max(0, MAX_PLATFORM_OUTPUT - len(footer))
logger.info(
"Cron output truncated (%d chars) — full output: %s",
len(content), saved_path,
)
return content[:visible] + footer
async def _deliver_to_platform(
self,
target: DeliveryTarget,
@@ -471,71 +415,21 @@ class DeliveryRouter:
if not target.chat_id:
raise ValueError(f"No chat ID for {target.platform.value} delivery")
# Guard: handle oversized cron output.
#
# Two independent decisions:
# 1. AUDIT SAVE — when content exceeds MAX_PLATFORM_OUTPUT, the full
# output is always written to disk as a recoverable audit trail.
# This fires regardless of adapter capability (best-effort).
# 2. TRUNCATION — for non-chunking adapters, content above the cap is
# truncated with a footer pointing to the saved file. Chunking-
# capable adapters (splits_long_messages=True) receive the full
# payload and split natively in their send().
job_id = (metadata or {}).get("job_id", "unknown")
saved_path: Optional[Path] = None
if len(content) > MAX_PLATFORM_OUTPUT:
# Step 1 — audit save (best-effort). The save is a side-effect
# audit trail, not essential to delivery. If it fails (full disk,
# permissions), delivery proceeds — the content reaches the adapter
# regardless.
try:
saved_path = self._save_full_output(content, job_id)
except OSError as exc:
logger.warning(
"Audit save failed for cron output (%d chars, job=%s): %s — "
"delivery proceeds without audit copy",
len(content), job_id, exc,
)
content = self._cap_oversized_output(
adapter, content, (metadata or {}).get("job_id", "unknown")
)
# Step 2 — truncation (only for non-chunking adapters).
if getattr(adapter, "splits_long_messages", False):
# Adapter chunks natively — deliver full payload.
if saved_path:
logger.info(
"Cron output preserved for chunking adapter (%d chars) — "
"full output saved to %s",
len(content), saved_path,
)
else:
# Non-chunking adapter — truncate with footer. The footer
# needs a valid path, so if the best-effort save above failed,
# retry it here (a failure now is a real delivery problem).
if saved_path is None:
saved_path = self._save_full_output(content, job_id)
footer = f"\n\n... [truncated, full output saved to {saved_path}]"
visible = max(0, MAX_PLATFORM_OUTPUT - len(footer))
logger.info(
"Cron output truncated (%d chars) — full output: %s",
len(content), saved_path,
)
content = content[:visible] + footer
# Substrate-level anti-loop guard: drop hallucinated "silence narration"
# (*(silent)*, 🔇, a bare ".", etc.) before it ever reaches the adapter.
# In bot-to-bot channels these tokens mirror back and forth until a
# model crashes with "no content after all retries". Behavioral prompt
# rules drift across providers; this single chokepoint covers every
# platform adapter regardless of which persona's prompt failed.
# Local/file delivery (_deliver_local) is a separate path and is never
# filtered — saved silence has no loop risk.
# Cron output is an ARTIFACT, not model chatter: a job whose brief is
# legitimately terse ("...", a single 🔇 from a script) has no bot-to-bot
# mirror loop to guard against, and dropping it here while returning
# {"success": True} is exactly how a cron was logged as delivered with
# nothing on the wire (#77763). Cron sends carry job_id in metadata;
# every other caller keeps the filter unchanged.
# (*(silent)*, 🔇, a bare ".") before it reaches any adapter — in
# bot-to-bot channels these mirror back and forth until a model crashes
# with "no content after all retries"; prompt rules drift across
# providers, so this single chokepoint covers every platform. Local/file
# delivery is never filtered (saved silence has no loop risk).
# Cron output is an ARTIFACT, not model chatter: a legitimately terse job
# ("...", a single 🔇) has no mirror loop, and dropping it while returning
# success is how a cron gets logged as delivered with nothing on the
# wire. Cron sends carry job_id in metadata; everything else is filtered.
is_cron_artifact = "job_id" in (metadata or {})
if (
self._filter_silence_narration_enabled()
@@ -562,96 +456,52 @@ class DeliveryRouter:
send_metadata["user_id"] = home.user_id
if home.scope_id:
send_metadata["scope_id"] = home.scope_id
is_named_telegram_private_topic = False
# Caller-supplied thread routing always wins over target.thread_id.
named_telegram_private_topic_name: Optional[str] = None
if target.thread_id:
has_explicit_direct_topic = (
"direct_messages_topic_id" in send_metadata
or "telegram_direct_messages_topic_id" in send_metadata
)
target_thread_id = target.thread_id
is_named_telegram_private_topic = (
thread_id = target.thread_id
thread_unrouted = thread_id and not any(
key in send_metadata
for key in ("thread_id", "message_thread_id",
"direct_messages_topic_id", "telegram_direct_messages_topic_id")
)
if thread_unrouted:
is_telegram_private = (
target.platform == Platform.TELEGRAM
and looks_like_telegram_private_chat_id(target.chat_id)
and not _looks_like_int(target_thread_id)
and "thread_id" not in send_metadata
and "message_thread_id" not in send_metadata
and not has_explicit_direct_topic
)
if is_named_telegram_private_topic:
named_telegram_private_topic_name = target_thread_id
ensure_dm_topic = getattr(adapter, "ensure_dm_topic", None)
if ensure_dm_topic is None:
raise RuntimeError(
"Telegram adapter cannot create named private DM topics"
)
created_thread_id = await ensure_dm_topic(target.chat_id, target_thread_id)
if not created_thread_id:
raise RuntimeError(
f"Failed to create Telegram private DM topic '{target_thread_id}'"
)
target_thread_id = str(created_thread_id)
send_metadata["thread_id"] = target_thread_id
if is_telegram_private and not _looks_like_int(thread_id):
# Named topic: create via createForumTopic, use message_thread_id directly.
named_telegram_private_topic_name = thread_id
send_metadata["thread_id"] = await _ensure_named_dm_topic(
adapter, target.chat_id, thread_id, refresh=False
)
send_metadata["telegram_dm_topic_created_for_send"] = True
elif (
target.platform == Platform.TELEGRAM
and looks_like_telegram_private_chat_id(target.chat_id)
and "thread_id" not in send_metadata
and "message_thread_id" not in send_metadata
and not has_explicit_direct_topic
):
# Legacy private topic/thread ids that were not created by this
# send path may still need a reply anchor to stay visible in the
# requested lane. Named targets are created above via
# createForumTopic and can use message_thread_id directly.
reply_anchor = send_metadata.get("telegram_reply_to_message_id")
if reply_anchor is None:
elif is_telegram_private:
# Legacy numeric private topic ids not created by this send path
# need a reply anchor to stay visible in the requested lane.
if send_metadata.get("telegram_reply_to_message_id") is None:
raise RuntimeError(
"Telegram private DM topic delivery requires telegram_reply_to_message_id; "
"send to the bare chat or provide a reply anchor"
)
send_metadata["thread_id"] = target_thread_id
send_metadata["thread_id"] = thread_id
send_metadata["telegram_dm_topic_reply_fallback"] = True
elif "thread_id" not in send_metadata and "message_thread_id" not in send_metadata and not has_explicit_direct_topic:
send_metadata["thread_id"] = target_thread_id
else:
send_metadata["thread_id"] = thread_id
result = await transport.send(
target.platform,
target.chat_id,
content,
metadata=send_metadata or None,
target.platform, target.chat_id, content, metadata=send_metadata or None,
)
if _send_result_failed(result):
if (
is_named_telegram_private_topic
and named_telegram_private_topic_name
and _is_thread_not_found_delivery_error(result)
):
ensure_dm_topic = getattr(adapter, "ensure_dm_topic", None)
if ensure_dm_topic is None:
raise RuntimeError(
"Telegram adapter cannot refresh named private DM topics"
)
refreshed_thread_id = await ensure_dm_topic(
target.chat_id,
named_telegram_private_topic_name,
force_create=True,
if named_telegram_private_topic_name and _is_thread_not_found_delivery_error(result):
send_metadata["thread_id"] = await _ensure_named_dm_topic(
adapter, target.chat_id, named_telegram_private_topic_name, refresh=True
)
if not refreshed_thread_id:
raise RuntimeError(
f"Failed to refresh Telegram private DM topic '{named_telegram_private_topic_name}'"
)
send_metadata["thread_id"] = str(refreshed_thread_id)
send_metadata["telegram_dm_topic_created_for_send"] = True
result = await transport.send(
target.platform,
target.chat_id,
content,
metadata=send_metadata or None,
target.platform, target.chat_id, content, metadata=send_metadata or None,
)
if _send_result_failed(result):
raise RuntimeError(_send_result_error(result) or f"{target.platform.value} delivery failed")
return result
gateway/delivery_ledger.py
+88 -163
View File
@@ -1,48 +1,42 @@
"""Durable delivery-obligation ledger for gateway final responses.
A final agent response that was generated but not yet confirmed-delivered
to the messaging platform is the one artifact the gateway can lose without
a trace: the turn already burned its tokens, the text exists only in a
Python local, and a crash / planned restart between finalize and platform
ACK drops it silently (#58818, #41696, #63695).
A final agent response generated but not yet confirmed-delivered is the one
artifact the gateway can lose without a trace: the turn already burned its
tokens, the text exists only in a Python local, and a crash / restart between
finalize and platform ACK drops it silently.
This module records a small durable row per outbound final response in the
shared ``state.db`` (same file and conventions as
``tools.async_delegation`` — WAL, owner pid + process-start-time liveness,
bounded retention). The gateway writes three checkpoints around the send:
Each outbound final response gets a small durable row in the shared ``state.db``
(same conventions as ``tools.async_delegation`` — WAL, owner pid + process-start
liveness, bounded retention). The gateway writes three checkpoints:
record_obligation() state='pending' before any send attempt
mark_attempting() state='attempting' immediately before the await
mark_delivered() / state='delivered' only on SendResult.success
mark_failed() state='failed' on a definitive rejection
On startup, ``sweep_recoverable()`` claims rows whose owning process is
dead and hands them to the gateway for redelivery. After a platform adapter
reconnects without a process restart, ``sweep_failed_for_runtime()`` may claim
only the same live process's explicitly allowlisted transient failures. Crash
semantics are explicit about ambiguity (the contract review of the earlier
delivery-outbox attempt, #61790, closed it for silently resending ambiguous
sends):
On startup ``sweep_recoverable()`` claims rows whose owner is dead and hands
them back for redelivery. After an adapter reconnects without a restart,
``sweep_failed_for_runtime()`` may claim only the same live process's
allowlisted transient failures. Crash semantics are explicit about ambiguity
(never silently resend an ambiguous send):
- ``pending`` — the send never started: redeliver plainly, no dup risk.
- ``attempting`` — crashed mid-await: the platform MAY already have the
message. Redelivered WITH a visible recovered-reply marker so the
contract is honest at-least-once, never a silent duplicate.
- ``failed`` — definitively rejected once; the restart is a natural
retry boundary. Also carries the marker.
- ``pending`` — send never started: redeliver plainly, no dup risk.
- ``attempting`` — crashed mid-await: the platform MAY already have it.
Redelivered WITH a visible recovered-reply marker (honest at-least-once).
- ``failed`` — definitively rejected once; restart is a natural retry
boundary. Also carries the marker.
- ``delivered`` — nothing to do; retention prunes.
Poison rows cannot spin: attempts are capped, stale rows expire, and both
transition to ``abandoned`` (kept briefly for inspection, then pruned).
Poison rows cannot spin: attempts are capped and stale rows expire, both
transitioning to ``abandoned`` (kept briefly, then pruned).
Everything here is best-effort by design: ledger failures must never block
or delay an actual send. Callers wrap every call in try/except.
Everything here is best-effort: ledger failures must never block or delay an
actual send. Callers wrap every call in try/except.
"""
from __future__ import annotations
import hashlib
import json
import logging
import os
import sqlite3
@@ -57,9 +51,8 @@ logger = logging.getLogger(__name__)
_DB_LOCK = threading.Lock()
# Redelivery policy knobs (module constants; deliberately not config — the
# ledger itself is gated by ``gateway.delivery_ledger`` and these bounds
# only matter in the rare recovery path).
# Redelivery policy knobs (deliberately not config — the ledger is gated by
# ``gateway.delivery_ledger`` and these only matter in the rare recovery path).
MAX_ATTEMPTS = 3
STALE_AFTER_SECONDS = 24 * 60 * 60
_RETENTION_SECONDS = 7 * 24 * 60 * 60
@@ -72,18 +65,16 @@ RECOVERED_MARKER = (
"so this may be a duplicate:\n\n"
)
# Runtime recovery uses a distinct marker because no gateway restart occurred.
# Keep the ambiguity explicit: a network rejection normally means the platform
# did not accept the message, but an acknowledgement can be lost independently.
# Runtime recovery uses a distinct marker: no restart occurred, but a network
# rejection's acknowledgement can still have been lost independently.
RECONNECTED_MARKER = (
"♻️ Recovered reply — the messaging platform reconnected after the original "
"delivery failed, so this may be a duplicate:\n\n"
)
# Runtime replay is deliberately fail-closed. Only errors whose send contract
# proves they are transient reconnect failures belong here; permanent rejects
# (blocked bot, bad auth, missing chat) must not be retried merely because an
# adapter reconnected.
# Runtime replay is fail-closed: only errors whose send contract proves they are
# transient reconnect failures. Permanent rejects (blocked bot, bad auth, missing
# chat) must not be retried merely because an adapter reconnected.
_RUNTIME_RETRYABLE_ERRORS = frozenset({"send_path_degraded"})
@@ -98,8 +89,7 @@ def _connect() -> sqlite3.Connection:
try:
_initialize_schema(conn)
except Exception:
# A PRAGMA/DDL failure after a successful connect() must not leak the
# just-opened connection back to the caller.
# A PRAGMA/DDL failure after connect() must not leak the connection.
conn.close()
raise
return conn
@@ -127,14 +117,10 @@ def _initialize_schema(conn: sqlite3.Connection) -> None:
adapter_profile TEXT
)"""
)
columns = {
row[1] for row in conn.execute("PRAGMA table_info(delivery_obligations)")
}
columns = {row[1] for row in conn.execute("PRAGMA table_info(delivery_obligations)")}
if "adapter_profile" not in columns:
try:
conn.execute(
"ALTER TABLE delivery_obligations ADD COLUMN adapter_profile TEXT"
)
conn.execute("ALTER TABLE delivery_obligations ADD COLUMN adapter_profile TEXT")
except sqlite3.OperationalError as exc:
# Concurrent first-use connections can both observe the old schema.
if "duplicate column" not in str(exc).lower():
@@ -145,13 +131,10 @@ def _initialize_schema(conn: sqlite3.Connection) -> None:
def _transaction() -> Iterator[sqlite3.Connection]:
"""Open a connection, commit/rollback on exit, and ALWAYS close it.
``sqlite3.Connection.__enter__``/``__exit__`` only commit or roll back the
transaction; they do not close the connection. Using ``with _connect()``
alone therefore leaks a connection — and its WAL/SHM file descriptors — on
every call, deferring the close to the garbage collector. On a long-running
gateway that exhausts ``RLIMIT_NOFILE`` (the cron-ledger sibling of this
bug was #69567 / PR #69594). ``record_obligation`` runs on every outbound
final response, so this ledger is the highest-frequency leaker.
``sqlite3.Connection`` as a context manager only commits/rolls back; it does
not close. ``with _connect()`` alone would leak a connection (and its WAL/SHM
fds) per call until GC — ``record_obligation`` runs on every outbound final
response, so this ledger would exhaust ``RLIMIT_NOFILE`` on a long-running gateway.
"""
conn = _connect()
try:
@@ -186,20 +169,16 @@ def _owner_alive(pid: Any, started_at: Any) -> bool:
except Exception:
current_start = None
if current_start is None:
# No such process (or unreadable) — treat unreadable-but-extant
# processes as alive only if the pid exists. Route through the
# cross-platform probe: ``os.kill(pid, 0)`` on Windows is NOT a
# no-op (bpo-14484 — CPython maps sig=0 to
# ``GenerateConsoleCtrlEvent(0, pid)``), so a raw probe here could
# Ctrl+C the gateway's own console group whenever psutil failed to
# read the start time of a live pid. ``_pid_exists`` keeps the
# EPERM-means-alive semantics (exists but owned by another user).
# Start time unreadable: alive iff the pid exists. Route through the
# cross-platform probe — ``os.kill(pid, 0)`` on Windows is NOT a no-op
# (bpo-14484: it maps to ``GenerateConsoleCtrlEvent(0, pid)`` and could
# Ctrl+C the gateway's own console group). ``_pid_exists`` keeps the
# EPERM-means-alive semantics (pid exists but is owned by another user).
try:
from gateway.status import _pid_exists
except Exception:
if os.name == "nt":
# Never fall back to a raw sig-0 probe on Windows.
return False
return False # never fall back to a raw sig-0 probe on Windows
try:
os.kill(pid, 0) # windows-footgun: ok — POSIX-only fallback branch
except ProcessLookupError:
@@ -222,10 +201,9 @@ def _owner_alive(pid: Any, started_at: Any) -> bool:
def compute_obligation_id(session_key: str, message_ref: str, content: str) -> str:
"""Stable id: same turn + same content re-records idempotently, while
distinct threads/topics on the same chat can never collide (the
session_key carries platform, chat and thread; ``message_ref`` is the
triggering inbound message id, distinguishing turns in one session)."""
"""Stable id: same turn + same content re-records idempotently, while distinct
threads/topics on one chat never collide (session_key carries platform, chat and
thread; ``message_ref`` is the triggering inbound message id)."""
payload = f"{session_key}|{message_ref}|{content}"
return hashlib.sha256(payload.encode("utf-8", "replace")).hexdigest()[:24]
@@ -273,11 +251,10 @@ def mark_failed(obligation_id: str, error: str = "") -> None:
def release_runtime_claim(obligation_id: str, error: str = "") -> bool:
"""Return an unsent runtime claim to ``failed`` without spending an attempt.
Runtime recovery claims before clearing ``resume_pending`` so that two
reconnect paths cannot send the same row. If the session flag cannot be
cleared, no platform send was attempted and the claim must not consume the
bounded redelivery budget. Release is fail-closed to the exact current
process instance and the ``attempting`` state.
Runtime recovery claims before clearing ``resume_pending`` so two reconnect
paths cannot send the same row. If the session flag cannot be cleared, no
send was attempted and the claim must not consume the redelivery budget.
Fail-closed to the exact current process instance and ``attempting`` state.
"""
pid, started = _owner_stamp()
if started is None:
@@ -306,31 +283,32 @@ def _update_state(obligation_id: str, state: str, error: str = "") -> None:
)
def _exhausted(attempts: int, created_at: float, now: float) -> bool:
return attempts >= MAX_ATTEMPTS or (now - created_at) > STALE_AFTER_SECONDS
def sweep_recoverable(
now: Optional[float] = None,
*,
deliverable_platforms: Optional[set] = None,
deliverable_targets: Optional[set] = None,
) -> List[Dict[str, Any]]:
"""Claim undelivered rows owned by dead processes; return them for
redelivery.
"""Claim undelivered rows owned by dead processes; return them for redelivery.
Claiming atomically re-stamps the owner to THIS process and increments
``attempts``, so a second gateway racing the same sweep cannot
double-claim (the UPDATE is guarded on the previous owner stamp).
Rows over the attempts cap or older than the stale cutoff transition to
'abandoned' instead of being returned.
``attempts`` (the UPDATE is guarded on the previous owner stamp, so a second
gateway racing the same sweep cannot double-claim). Rows over the attempts
cap or stale cutoff transition to 'abandoned' instead of being returned.
``deliverable_platforms`` (platform value strings) restricts claiming to
platforms the caller can actually send on this boot. ``attempts`` is the
redelivery budget, so it must only be spent on a real send: a platform
that failed to connect would otherwise burn one attempt per boot and hit
the cap having never been sent once. Rows for absent platforms are left
untouched for a later boot; the stale cutoff still bounds them.
``deliverable_platforms`` restricts claiming to platforms the caller can
actually send on this boot: ``attempts`` is the redelivery budget and must
only be spent on a real send, else a platform that failed to connect burns
one attempt per boot and hits the cap having never been sent once. Rows for
absent platforms are left untouched (the stale cutoff still bounds them).
``deliverable_targets`` further scopes multiplexed gateways by exact
``(platform, adapter_profile)`` identity, preventing one connected bot from
spending another disconnected bot's retry budget.
``(platform, adapter_profile)`` so one connected bot cannot spend another
disconnected bot's retry budget.
"""
now = now if now is not None else time.time()
pid, started = _owner_stamp()
@@ -348,24 +326,16 @@ def sweep_recoverable(
adapter_profile) in rows:
if _owner_alive(owner_pid, owner_started_at):
continue # a live gateway still owns this row
if attempts >= MAX_ATTEMPTS or (now - created_at) > STALE_AFTER_SECONDS:
if _exhausted(attempts, created_at, now):
conn.execute(
"""UPDATE delivery_obligations
SET state='abandoned', updated_at=? WHERE obligation_id=?""",
(now, oid),
)
continue
if (
deliverable_platforms is not None
and platform not in deliverable_platforms
):
# No adapter for this platform this boot — the caller cannot
# send, so claiming would spend an attempt on a no-op.
continue
if (
deliverable_targets is not None
and (platform, adapter_profile) not in deliverable_targets
):
if deliverable_platforms is not None and platform not in deliverable_platforms:
continue # no adapter this boot — claiming would spend an attempt on a no-op
if deliverable_targets is not None and (platform, adapter_profile) not in deliverable_targets:
continue
cursor = conn.execute(
"""UPDATE delivery_obligations
@@ -382,8 +352,7 @@ def sweep_recoverable(
"chat_id": chat_id,
"thread_id": thread_id,
"content": content,
# pending = send never started, redeliver plainly;
# attempting/failed = ambiguous or rejected, carry marker.
# pending = never started, redeliver plainly; else carry marker.
"needs_marker": state != "pending",
"profile": adapter_profile,
"attempts": attempts + 1,
@@ -399,32 +368,27 @@ def sweep_failed_for_runtime(
) -> List[Dict[str, Any]]:
"""Claim this process's reconnect-retryable failed rows for one adapter.
``profile`` scopes multiplexed gateways to the bot identity that actually
owned the failed send; ``None`` means the primary/default adapter. The
persisted adapter owner is independent of the routed session namespace.
``profile`` scopes multiplexed gateways to the bot identity that owned the
failed send; ``None`` means the primary/default adapter. The persisted
adapter owner is independent of the routed session namespace. Unowned rows
and rows owned by another process are left for the startup/dead-owner sweep.
Startup recovery intentionally ignores rows owned by a live gateway. That
protects concurrent processes, but it also means a final response rejected
with ``send_path_degraded`` remains stranded when only the platform adapter
reconnects. This runtime sweep closes that gap without weakening ownership:
- only rows stamped to this exact process instance are eligible;
- only explicitly allowlisted transient errors are eligible;
- attempts/staleness bounds match startup recovery;
- every update is guarded by the prior owner stamp and ``failed`` state.
Unowned rows and rows owned by another process are left untouched for the
normal startup/dead-owner sweep. Claimed rows always carry the reconnect
Startup recovery ignores rows owned by a live gateway, so a final response
rejected with ``send_path_degraded`` would stay stranded when only the
adapter reconnects. This sweep closes that gap without weakening ownership:
only rows stamped to this exact process instance, only allowlisted transient
errors, same attempts/staleness bounds, every update guarded by the prior
owner stamp and ``failed`` state. Claimed rows always carry the reconnect
marker because the failed send's acknowledgement is not safe to infer.
"""
now = now if now is not None else time.time()
pid, started = _owner_stamp()
if started is None:
# PID equality alone cannot distinguish this process from a stale row
# left by an earlier process incarnation after PID reuse. Runtime replay
# is optional recovery, so fail closed when the process fingerprint is
# unavailable; startup recovery remains the durable fallback.
# PID alone cannot distinguish this process from a stale row left after
# PID reuse. Runtime replay is optional, so fail closed; startup
# recovery remains the durable fallback.
return []
expected_profile = "default" if not profile or profile == "default" else str(profile)
claimed: List[Dict[str, Any]] = []
with _DB_LOCK, _transaction() as conn:
rows = conn.execute(
@@ -435,33 +399,18 @@ def sweep_failed_for_runtime(
WHERE state='failed' AND platform=?""",
(platform,),
).fetchall()
for (
oid,
session_key,
row_platform,
chat_id,
thread_id,
content,
attempts,
created_at,
owner_pid,
owner_started_at,
last_error,
adapter_profile,
) in rows:
expected_profile = (
"default" if not profile or profile == "default" else str(profile)
)
for (oid, session_key, row_platform, chat_id, thread_id, content,
attempts, created_at, owner_pid, owner_started_at, last_error,
adapter_profile) in rows:
if adapter_profile != expected_profile:
continue
# Runtime reconnect recovery may act only on its own rows. Exact
# process-start matching prevents PID reuse from stealing work.
# Exact process-start matching prevents PID reuse from stealing work.
if owner_pid != pid or owner_started_at != started:
continue
if str(last_error or "").strip().lower() not in _RUNTIME_RETRYABLE_ERRORS:
continue
owner_guard = (oid, owner_pid, owner_started_at)
if attempts >= MAX_ATTEMPTS or (now - created_at) > STALE_AFTER_SECONDS:
if _exhausted(attempts, created_at, now):
conn.execute(
"""UPDATE delivery_obligations
SET state='abandoned', updated_at=?
@@ -504,9 +453,7 @@ def _prune(now: Optional[float] = None) -> None:
WHERE state IN ('delivered', 'abandoned') AND updated_at < ?""",
(cutoff,),
)
total = conn.execute(
"SELECT COUNT(*) FROM delivery_obligations"
).fetchone()[0]
total = conn.execute("SELECT COUNT(*) FROM delivery_obligations").fetchone()[0]
excess = max(0, total - _MAX_ROWS)
if excess:
conn.execute(
@@ -538,25 +485,3 @@ def ledger_enabled(config: Optional[Dict[str, Any]] = None) -> bool:
return bool(value)
except Exception:
return True
def debug_rows(limit: int = 20) -> str:
"""Human-readable dump for ad-hoc inspection (sqlite3-free path)."""
with _DB_LOCK, _transaction() as conn:
rows = conn.execute(
"""SELECT obligation_id, session_key, state, attempts,
created_at, updated_at, last_error
FROM delivery_obligations
ORDER BY updated_at DESC LIMIT ?""",
(limit,),
).fetchall()
return json.dumps(
[
{
"id": r[0], "session": r[1], "state": r[2], "attempts": r[3],
"created_at": r[4], "updated_at": r[5], "last_error": r[6],
}
for r in rows
],
indent=2,
)
gateway/disk_status.py
+27 -44
View File
@@ -1,30 +1,21 @@
"""Disk-usage rollup for ``/api/status`` (NS-656).
"""Disk-usage rollup for ``/api/status``.
Companion to :mod:`gateway.memory_status`, closing the same class of gap
for storage: a hosted agent can fill its data volume completely — SQLite
writes failing, session persistence dead, config saves lost — while its
dashboard and the NAS agent card both look perfectly healthy. Fleet
incidents OOF-2 (unrecoverable disk-full) and OOF-107 (fleet-wide disk
exhaustion, remediated by hand) are exactly this failure mode.
Companion to :mod:`gateway.memory_status`: a hosted agent can fill its data
volume (SQLite writes failing, sessions and config saves lost) while its
dashboard and the NAS agent card look healthy. ``gateway/readiness.py``
already probes disk, but readiness is a component verdict, not user-facing
telemetry — this module produces the public block the dashboard SPA and the
NAS availability sweep consume.
The readiness endpoint already probes disk (``gateway/readiness.py::
_probe_disk``), but readiness is a component verdict, not user-facing
telemetry — nothing renders it. This module produces the public block
the dashboard SPA and the NAS availability sweep actually consume.
Disk is sampled live via :func:`shutil.disk_usage` (one ``statvfs`` call), so
there is no staleness dimension and no ``sampled_at``.
Unlike the memory block (which distills already-persisted heartbeat
files), disk is sampled live via :func:`shutil.disk_usage` — a single
``statvfs`` call, the same thing the readiness probe does per request.
There is no meaningful "staleness" dimension, so no ``sampled_at``.
Public-safety: ``/api/status`` is unauthenticated (``PUBLIC_API_PATHS``); this
block carries only coarse numbers (MB, one-decimal percent) and an enum — the same
disclosure class as the ``memory`` block.
Public-safety note: ``/api/status`` is an unauthenticated liveness probe
(``PUBLIC_API_PATHS``). This block carries only coarse numbers (MB
granularity, whole-percent usage) and an enum — the same disclosure
class as the ``memory`` block.
Everything is best-effort and read-only: an unreadable filesystem
degrades to ``pressure="unknown"`` rather than raising into the status
endpoint.
Best-effort and read-only: an unreadable filesystem degrades to
``pressure="unknown"`` rather than raising into the status endpoint.
"""
from __future__ import annotations
@@ -36,12 +27,10 @@ from typing import Any, Dict, Optional
logger = logging.getLogger(__name__)
# Disk-pressure thresholds. Percent alone misleads in both directions:
# 90% used on a 100 GB volume leaves a comfortable 10 GB, while 50% used
# on a tiny volume can be one image download from write failures. So the
# percent triggers are gated on absolute headroom also being low, and a
# hard absolute floor applies regardless of size — below it, SQLite
# journaling and config writes are at genuine risk on any volume.
# Percent alone misleads both ways: 90% used on 100 GB leaves 10 GB, while 50%
# on a tiny volume is one download from write failures. So percent triggers are
# gated on absolute headroom also being low, and a hard absolute floor applies
# regardless of size (below it SQLite journaling / config writes are at risk).
_CRITICAL_FREE_MB = 256 # < 256 MB free: critical on any volume
_CRITICAL_PERCENT = 95.0 # >= 95% used AND < 1 GB free: critical
_CRITICAL_HEADROOM_MB = 1024
@@ -69,28 +58,22 @@ def classify_disk_pressure(free_mb: Any, total_mb: Any) -> str:
if free is None or total is None or total <= 0:
return "unknown"
used_percent = (1 - free / total) * 100.0
if free < _CRITICAL_FREE_MB or (
used_percent >= _CRITICAL_PERCENT and free < _CRITICAL_HEADROOM_MB
for level, free_floor, percent_floor, headroom in (
("critical", _CRITICAL_FREE_MB, _CRITICAL_PERCENT, _CRITICAL_HEADROOM_MB),
("elevated", _ELEVATED_FREE_MB, _ELEVATED_PERCENT, _ELEVATED_HEADROOM_MB),
):
return "critical"
if free < _ELEVATED_FREE_MB or (
used_percent >= _ELEVATED_PERCENT and free < _ELEVATED_HEADROOM_MB
):
return "elevated"
if free < free_floor or (used_percent >= percent_floor and free < headroom):
return level
return "ok"
def collect_disk_status(home: Optional[Path] = None) -> Dict[str, Any]:
"""Build the ``disk`` block for ``/api/status``.
``home`` scopes the sample to a profile's HERMES_HOME (the status
endpoint's ``?profile=`` handling passes it through); on hosted
images every profile shares the ``/opt/data`` volume, so the answer
is the same — but scoping keeps the contract identical to the
``memory`` block's.
Always returns a dict — an unreadable/unmounted filesystem yields
``{"pressure": "unknown", ...}``. Never raises.
``home`` scopes the sample to a profile's HERMES_HOME (same contract as the
``memory`` block; on hosted images every profile shares one volume).
Always returns a dict and never raises — an unreadable/unmounted filesystem
yields ``{"pressure": "unknown", ...}``.
"""
status: Dict[str, Any] = {
"pressure": "unknown",
gateway/display_config.py
+172 -236
View File
@@ -1,12 +1,10 @@
"""Per-platform display/verbosity configuration resolver.
Provides ``resolve_display_setting()`` — the single entry-point for reading
display settings with platform-specific overrides and sensible defaults.
Resolution order (first non-None wins):
1. ``display.platforms.<platform>.<key>`` — explicit per-platform user override
``resolve_display_setting()`` is the single entry-point. Resolution order
(first non-None wins):
1. ``display.platforms.<platform>.<key>`` — explicit per-platform override
2. ``display.<key>`` — global user setting
3. ``_PLATFORM_DEFAULTS[<platform>][<key>]`` — built-in sensible default
3. ``_PLATFORM_DEFAULTS[<platform>][<key>]`` — built-in platform default
4. ``_GLOBAL_DEFAULTS[<key>]`` — built-in global default
Exception: ``display.streaming`` is CLI-only. Gateway streaming follows the
@@ -14,70 +12,52 @@ top-level ``streaming`` config unless ``display.platforms.<platform>.streaming``
sets an explicit per-platform override.
Backward compatibility: ``display.tool_progress_overrides`` is still read as a
fallback for ``tool_progress`` when no ``display.platforms`` entry exists. A
config migration (version bump) automatically moves the old format into the new
``display.platforms`` structure.
fallback for ``tool_progress`` when no ``display.platforms`` entry exists (a
config migration moves the old format into ``display.platforms``).
"""
from __future__ import annotations
from typing import Any
# ---------------------------------------------------------------------------
# Overrideable display settings and their global defaults
# ---------------------------------------------------------------------------
# These are the settings that can be configured per-platform.
# Other display settings (compact, personality, skin, etc.) are CLI-only
# and don't participate in per-platform resolution.
# Settings configurable per-platform, with global defaults. Other display
# settings (compact, personality, skin, ...) are CLI-only.
_GLOBAL_DEFAULTS: dict[str, Any] = {
"tool_progress": "all",
"tool_progress_grouping": "accumulate", # "accumulate" = edit one bubble; "separate" = one msg per tool
"show_reasoning": False,
# How a reasoning/thinking summary is rendered when show_reasoning is on.
# "code" -> 💭 **Reasoning:** + fenced code block (legacy default)
# "blockquote"-> each line prefixed with "> "
# "subtext" -> each line prefixed with "-# " (Discord small grey subtext)
# Discord defaults to "subtext"; everywhere else defaults to "code".
# Reasoning summary rendering: "code" (💭 **Reasoning:** + fenced block),
# "blockquote" ("> " lines), "subtext" ("-# " Discord small grey text).
"reasoning_style": "code",
"tool_preview_length": 0,
"streaming": None, # None = follow top-level streaming config
# Gateway-only assistant/status chatter controls. These default on for
# back-compat, but mobile platforms can opt down to final-answer-first.
# Gateway-only assistant/status chatter; default on for back-compat, mobile
# platforms opt down to final-answer-first.
"interim_assistant_messages": True,
"long_running_notifications": True,
"busy_ack_detail": True,
# Whether busy_input_mode=steer sends a visible "Steered into current run"
# acknowledgment after successfully injecting the user's mid-turn message.
# Disable when the platform should steer silently (the text still lands in
# the active run; only the confirmation echo is suppressed).
# busy_input_mode=steer confirmation echo ("Steered into current run").
# Disabling only suppresses the echo; the text still lands in the run.
"busy_steer_ack_enabled": True,
# When true, delete tool-progress / "⏳ Working — N min" / status bubbles
# after the final response lands on platforms that support message
# deletion (e.g. Telegram). Off by default — progress is still shown
# live, just cleaned up after success so the chat doesn't fill up with
# stale breadcrumbs. Failed runs leave bubbles in place as breadcrumbs.
# Delete tool-progress / "⏳ Working" / status bubbles after the final
# response on platforms that support deletion (e.g. Telegram). Off by
# default; progress is still shown live, only cleaned up after success so
# the chat doesn't fill with stale breadcrumbs. Failed runs leave bubbles
# in place as breadcrumbs.
"cleanup_progress": False,
# Live working-state status on platforms whose typing indicator renders
# text (Slack's assistant status line). Values:
# "full" / true -> verb + argument preview ("is running pytest…")
# "verb" -> verb only ("is running…") — keeps file paths and
# commands out of shared channels
# "off" / false -> static text (typing_status_text or "is thinking...")
# Independent of tool_progress: works even when progress bubbles are off
# (Slack's default), and costs no extra API calls — the existing typing
# refresh cadence just renders different text.
# Live working-state text on platforms whose typing indicator renders text
# (Slack assistant status): "full"/true = verb + argument preview,
# "verb" = verb only (keeps paths/commands out of shared channels),
# "off"/false = static text. Independent of tool_progress and free: the
# existing typing refresh cadence just renders different text.
"live_status": "full",
}
# ---------------------------------------------------------------------------
# Sensible per-platform defaults — tiered by platform capability
# ---------------------------------------------------------------------------
# Tier 1 (high): Supports message editing, typically personal/team use
# Tier 2 (medium): Supports editing but often workspace/customer-facing
# Tier 3 (low): No edit support — each progress msg is permanent
# Tier 4 (minimal): Batch/non-interactive delivery
# Per-platform defaults tiered by capability:
# HIGH — message editing, personal/team use
# MEDIUM — editing, but often workspace/customer-facing
# LOW — no edit support: each progress message is permanent
# MINIMAL — batch / non-interactive delivery
_TIER_HIGH = {
"tool_progress": "all",
"show_reasoning": False,
@@ -87,108 +67,63 @@ _TIER_HIGH = {
"long_running_notifications": True,
"busy_ack_detail": True,
}
_TIER_MEDIUM = {
"tool_progress": "new",
"show_reasoning": False,
"tool_preview_length": 40,
"streaming": None,
"interim_assistant_messages": True,
"long_running_notifications": True,
"busy_ack_detail": True,
}
_TIER_MEDIUM = {**_TIER_HIGH, "tool_progress": "new"}
_TIER_LOW = {
**_TIER_HIGH,
"tool_progress": "off",
"show_reasoning": False,
"tool_preview_length": 40,
"streaming": False,
"interim_assistant_messages": False,
"long_running_notifications": False,
"busy_ack_detail": False,
}
_TIER_MINIMAL = {
"tool_progress": "off",
"show_reasoning": False,
"tool_preview_length": 0,
"streaming": False,
"interim_assistant_messages": False,
"long_running_notifications": False,
"busy_ack_detail": False,
}
_TIER_MINIMAL = {**_TIER_LOW, "tool_preview_length": 0}
_PLATFORM_DEFAULTS: dict[str, dict[str, Any]] = {
# Tier 1 — full edit support, personal/team use
# Telegram is usually a mobile inbox: keep tool_progress quiet and skip
# the verbose busy-ack iteration counter, but DO surface real mid-turn
# assistant commentary (interim_assistant_messages) and DO send periodic
# heartbeats (long_running_notifications) so the user has signal between
# turn start and final answer. Otherwise it looks like "typing..." for
# 30 minutes with nothing happening. Opt in to verbose iteration detail
# via display.platforms.telegram.busy_ack_detail / tool_progress.
"telegram": {
**_TIER_HIGH,
"tool_progress": "off",
"busy_ack_detail": False,
},
# Discord has a native "subtext" primitive (-# small grey text) that reads
# as metadata rather than content, so reasoning summaries default to it
# here instead of the fenced code block used elsewhere.
"discord": {**_TIER_HIGH, "reasoning_style": "subtext"},
# Telegram is usually a mobile inbox: quiet tool_progress and no busy-ack
# iteration counter, but DO surface interim assistant commentary and
# heartbeats so it doesn't look like "typing..." for 30 minutes.
"telegram": {**_TIER_HIGH, "tool_progress": "off", "busy_ack_detail": False},
# Discord's "-# " subtext reads as metadata, so reasoning defaults to it.
"discord": {**_TIER_HIGH, "reasoning_style": "subtext"},
# Tier 2 — edit support, often customer/workspace channels
# Slack: tool_progress off by default — Bolt posts cannot be edited like CLI;
# "new"/"all" spam permanent lines in channels (hermes-agent#14663).
"slack": {
# Slack: Bolt posts cannot be edited like CLI; "new"/"all" spam permanent lines.
"slack": {
**_TIER_MEDIUM,
"tool_progress": "off",
"long_running_notifications": False,
"busy_ack_detail": False,
},
"mattermost": _TIER_MEDIUM,
"matrix": _TIER_MEDIUM,
"feishu": _TIER_MEDIUM,
# Buzz (Nostr relay via buzz-cli): messages can be edited in place
# (`buzz messages edit`), so grouped/accumulating progress works, but
# channels are shared community spaces — keep the medium tier. Without
# this entry Buzz inherited the verbose _GLOBAL_DEFAULTS and every
# interim update became a separate permanent channel post (#95841).
"buzz": _TIER_MEDIUM,
"mattermost": _TIER_MEDIUM,
"matrix": _TIER_MEDIUM,
"feishu": _TIER_MEDIUM,
# Buzz (Nostr via buzz-cli) can edit in place but channels are shared
# community spaces; without an entry it inherited the verbose globals.
"buzz": _TIER_MEDIUM,
# Tier 3 — no edit support, progress messages are permanent
"signal": _TIER_LOW,
"whatsapp": _TIER_MEDIUM, # Baileys bridge supports /edit
# WhatsApp Cloud API: Meta added message editing in 2023 but the
# Hermes Cloud adapter doesn't implement edit_message yet, so we
# stay on TIER_LOW (tool_progress off) to avoid spamming each
# status update as a separate message. Promote to TIER_MEDIUM once
# Cloud's edit_message lands.
"whatsapp_cloud": _TIER_LOW,
# Photon (managed iMessage over the gRPC sidecar) and BlueBubbles are both
# permanent-message iMessage inboxes with no message-edit support, so both
# stay TIER_LOW. This keeps tool progress, interim scratch commentary,
# "still working" heartbeats, and busy-ack iteration detail out of the
# user's iMessage thread. Without this entry Photon inherited the noisy
# global ("all") defaults and compacted/narrated on nearly every turn.
"photon": _TIER_LOW,
"bluebubbles": _TIER_LOW,
"weixin": _TIER_LOW,
# WeCom is technically non-editable but exposes a native streaming
# transport (msgtype: "stream" via aibot_respond_msg) that the gateway
# consumer routes mid-stream content through. Enable streaming by default
# so the WeCom client renders the typing animation and cumulative content
# updates instead of a single one-shot markdown drop.
"wecom": {**_TIER_LOW, "streaming": True},
"wecom_callback": _TIER_LOW,
"dingtalk": _TIER_LOW,
"signal": _TIER_LOW,
"whatsapp": _TIER_MEDIUM, # Baileys bridge supports /edit
# Cloud API supports editing but our adapter lacks edit_message; promote
# to MEDIUM once it lands.
"whatsapp_cloud": _TIER_LOW,
# Photon and BlueBubbles are permanent-message iMessage inboxes (no edit);
# without an entry Photon inherited the noisy "all" globals and narrated
# on nearly every turn.
"photon": _TIER_LOW,
"bluebubbles": _TIER_LOW,
"weixin": _TIER_LOW,
# WeCom is non-editable but has a native streaming transport (msgtype
# "stream") the consumer routes mid-stream content through; streaming on
# gives the client a typing animation + cumulative updates instead of a
# single one-shot markdown drop.
"wecom": {**_TIER_LOW, "streaming": True},
"wecom_callback": _TIER_LOW,
"dingtalk": _TIER_LOW,
# Tier 4 — batch or non-interactive delivery
"email": _TIER_MINIMAL,
"sms": _TIER_MINIMAL,
"webhook": _TIER_MINIMAL,
"homeassistant": _TIER_MINIMAL,
"api_server": {**_TIER_HIGH, "tool_preview_length": 0},
"email": _TIER_MINIMAL,
"sms": _TIER_MINIMAL,
"webhook": _TIER_MINIMAL,
"homeassistant": _TIER_MINIMAL,
"api_server": {**_TIER_HIGH, "tool_preview_length": 0},
}
# Canonical set of per-platform overrideable keys (for validation).
@@ -203,120 +138,121 @@ def resolve_display_setting(
) -> Any:
"""Resolve a display setting with per-platform override support.
Parameters
----------
user_config : dict
The full parsed config.yaml dict.
platform_key : str
Platform config key (e.g. ``"telegram"``, ``"slack"``). Use
``_platform_config_key(source.platform)`` from gateway/run.py.
setting : str
Display setting name (e.g. ``"tool_progress"``, ``"show_reasoning"``).
fallback : Any
Fallback value when the setting isn't found anywhere.
Returns
-------
The resolved value, or *fallback* if nothing is configured.
``platform_key`` is the platform config key (``"telegram"``, ``"slack"``;
see ``_platform_config_key`` in gateway/run.py). Returns *fallback* when
nothing is configured.
"""
display_cfg = user_config.get("display") or {}
# 1. Explicit per-platform override (display.platforms.<platform>.<key>)
platforms = display_cfg.get("platforms") or {}
plat_overrides = platforms.get(platform_key)
if isinstance(plat_overrides, dict):
val = plat_overrides.get(setting)
if val is not None:
return _normalise(setting, val)
# 1. Explicit per-platform override
plat_overrides = (display_cfg.get("platforms") or {}).get(platform_key)
if isinstance(plat_overrides, dict) and plat_overrides.get(setting) is not None:
return _normalise(setting, plat_overrides[setting])
# 1b. Backward compat: display.tool_progress_overrides.<platform>
if setting == "tool_progress":
legacy = display_cfg.get("tool_progress_overrides")
if isinstance(legacy, dict):
val = legacy.get(platform_key)
if val is not None:
return _normalise(setting, val)
if isinstance(legacy, dict) and legacy.get(platform_key) is not None:
return _normalise(setting, legacy[platform_key])
# 2. Global user setting (display.<key>). Skip display.streaming because
# that key controls only CLI terminal streaming; gateway token streaming is
# governed by the top-level streaming config plus per-platform overrides.
if setting != "streaming":
val = display_cfg.get(setting)
if val is not None:
return _normalise(setting, val)
# 2. Global user setting. display.streaming controls only CLI terminal
# streaming; gateway streaming follows the top-level config + platform overrides.
if setting != "streaming" and display_cfg.get(setting) is not None:
return _normalise(setting, display_cfg[setting])
# 3. Built-in platform default
plat_defaults = _PLATFORM_DEFAULTS.get(platform_key)
if plat_defaults:
val = plat_defaults.get(setting)
if val is not None:
return val
# 4. Built-in global default
val = _GLOBAL_DEFAULTS.get(setting)
if val is not None:
return val
return fallback
# 3. Built-in platform default, 4. built-in global default
val = _PLATFORM_DEFAULTS.get(platform_key, {}).get(setting)
if val is None:
val = _GLOBAL_DEFAULTS.get(setting)
return fallback if val is None else val
# ---------------------------------------------------------------------------
# Helpers
# Normalisation of YAML quirks (bare ``off`` → False in YAML 1.1, etc.)
# ---------------------------------------------------------------------------
_TRUTHY = {"true", "1", "yes", "on"}
_FALSY = {"false", "0", "no"}
def _norm_tool_progress(value: Any) -> str:
if value is False:
return "off"
if value is True:
return "all"
val = str(value).strip().lower()
if val in _FALSY:
return "off"
if val in _TRUTHY:
return "all"
return val if val in {"off", "new", "all", "verbose", "log"} else "all"
def _norm_bool(value: Any) -> bool:
if isinstance(value, str):
return value.strip().lower() in _TRUTHY | {"raw", "verbose"}
return bool(value)
def _norm_long_running(value: Any) -> Any:
if isinstance(value, str) and value.strip().lower() == "generic":
return "generic"
return _norm_bool(value)
def _norm_cleanup_progress(value: Any) -> bool:
if isinstance(value, str):
return value.lower() in _TRUTHY
return bool(value)
def _norm_live_status(value: Any) -> str:
"""Tri-state: "full" (verb + preview), "verb" (verb only), "off"."""
if value is True:
return "full"
if value is False:
return "off"
val = str(value).strip().lower()
if val in _TRUTHY | {"all"}:
return "full"
if val in _FALSY:
return "off"
return val if val in {"full", "verb", "off"} else "full"
def _norm_choice(choices: tuple[str, ...]) -> Any:
def norm(value: Any) -> str:
val = str(value).lower()
return val if val in choices else choices[0]
return norm
def _norm_int(value: Any) -> int:
try:
return int(value)
except (TypeError, ValueError):
return 0
_NORMALISERS: dict[str, Any] = {
"tool_progress": _norm_tool_progress,
"show_reasoning": _norm_bool,
"streaming": _norm_bool,
"interim_assistant_messages": _norm_bool,
"long_running_notifications": _norm_long_running,
"busy_ack_detail": _norm_bool,
"busy_steer_ack_enabled": _norm_bool,
"thinking_progress": _norm_bool,
"cleanup_progress": _norm_cleanup_progress,
"live_status": _norm_live_status,
"tool_progress_grouping": _norm_choice(("accumulate", "separate")),
"reasoning_style": _norm_choice(("code", "blockquote", "subtext")),
"tool_preview_length": _norm_int,
}
def _normalise(setting: str, value: Any) -> Any:
"""Normalise YAML quirks (bare ``off`` → False in YAML 1.1)."""
if setting == "tool_progress":
if value is False:
return "off"
if value is True:
return "all"
val = str(value).strip().lower()
if val in {"false", "0", "no"}:
return "off"
if val in {"true", "1", "yes", "on"}:
return "all"
return val if val in {"off", "new", "all", "verbose", "log"} else "all"
if setting in {
"show_reasoning",
"streaming",
"interim_assistant_messages",
"long_running_notifications",
"busy_ack_detail",
"busy_steer_ack_enabled",
"thinking_progress",
}:
if isinstance(value, str):
val = value.strip().lower()
if val == "generic" and setting == "long_running_notifications":
return "generic"
return val in {"true", "1", "yes", "on", "raw", "verbose"}
return bool(value)
if setting == "cleanup_progress":
if isinstance(value, str):
return value.lower() in {"true", "1", "yes", "on"}
return bool(value)
if setting == "live_status":
# Tri-state: "full" (verb + preview), "verb" (verb only), "off".
if value is True:
return "full"
if value is False:
return "off"
val = str(value).strip().lower()
if val in {"true", "1", "yes", "on", "all"}:
return "full"
if val in {"false", "0", "no"}:
return "off"
return val if val in {"full", "verb", "off"} else "full"
if setting == "tool_progress_grouping":
val = str(value).lower()
return val if val in ("accumulate", "separate") else "accumulate"
if setting == "reasoning_style":
val = str(value).lower()
return val if val in ("code", "blockquote", "subtext") else "code"
if setting == "tool_preview_length":
try:
return int(value)
except (TypeError, ValueError):
return 0
return value
"""Normalise a user-supplied value for *setting*; unknown settings pass through."""
norm = _NORMALISERS.get(setting)
return norm(value) if norm else value
gateway/drain_control.py
+76 -219
View File
@@ -1,65 +1,35 @@
"""External drain-control marker contract (dashboard → gateway).
Task 2.2 of the safe-shutdown plan (decisions.md Q-B, option A): the dashboard
has no way to call into a running gateway — there is no HTTP control channel
into the gateway process (guardrails: "there is NO external control channel
into a running gateway"). Restart/drain is driven only by the gateway reacting
to its own inputs: slash commands, process signals, and file markers it writes
itself (``.restart_notify.json``).
So the begin/cancel-drain dashboard endpoint communicates with the running
gateway the same way: it writes (or removes) a marker file, and a gateway
background watcher reacts to it. This module owns that marker contract so both
sides — the dashboard endpoint (writer) and the gateway watcher (reader) —
share one definition and can never disagree.
Contract (presence-based, mirroring ``.restart_notify.json``):
There is no control channel into a running gateway, so the dashboard's
begin/cancel-drain endpoint talks to it the same way ``.restart_notify.json``
does: it writes (or removes) a marker file and a gateway watcher reacts. This
module owns the contract so writer and reader can never disagree.
Contract (presence-based):
* begin-drain → write ``{HERMES_HOME}/.drain_request.json`` with
``{"action": "drain", "requested_at": <iso>, "principal": <str>,
"epoch": <instantiation-epoch>, "suppress_notification": <bool>}``.
* cancel-drain → remove the marker.
* The gateway watcher treats **presence of a marker stamped with the current
instantiation epoch** as "external drain active": flip
``gateway_state -> "draining"`` and stop accepting new turns. Absence (or a
marker from a *prior* instantiation) means "not draining" (revert to
``running`` if we had flipped it).
* The watcher treats presence of an ACTIVE marker as "external drain": flip
``gateway_state -> "draining"`` and stop accepting new turns. Absence or a
stale marker means "not draining" (revert to ``running``).
Why the epoch (NS-570). ``HERMES_HOME`` is a **durable** store — on Hermes
Cloud it is a persistent Fly volume (``/opt/data``). A begin-drain marker
written there *survives a machine restart*. But the disruptive lifecycle
actions a drain protects (auto-update / image migrate / env edit / profile
change) all **restart the machine**, which is exactly the signal that the drain
is over. Without the epoch, a freshly-restarted gateway re-reads the orphaned
marker on boot and parks itself right back in ``draining`` forever (NS-570: an
auto-updated instance refused every turn for ~52 min). Stamping the marker with
an identity of *this* container/VM instantiation, and ignoring a marker whose
epoch doesn't match, makes "a deliberate restart clears the drain" true by
construction — while a marker written during the *current* instantiation (the
live drain) still matches, and an s6 respawn of just the gateway (PID 1 / init
unchanged) still honours an in-flight drain.
Staleness — two independent, individually-lenient signals (either suffices):
* epoch mismatch: HERMES_HOME is a durable volume on Hermes Cloud, so a
marker survives the machine restart that the drain-gated action (update,
migrate, env edit) ends in; honouring it would park the fresh gateway in
``draining`` forever. A marker is stamped with this instantiation's epoch
and ignored only on a *definite* mismatch.
* expiry: a same-epoch orphan (action completed without a restart, writer
never cancelled) is ignored once ``requested_at`` is older than
:data:`DRAIN_REQUEST_MAX_AGE_SECONDS`. Re-calling :func:`write_drain_request`
refreshes the timestamp — the sanctioned keep-alive for a long drain.
Reading the marker never raises: a malformed/half-written file reads as
"present but contentless", which the watcher still treats as drain-active
(fail-safe toward quiescing — a corrupt begin marker must not be ignored). The
epoch check is deliberately **lenient**: it ignores a marker only on a
*definite* epoch mismatch. A marker with no epoch (legacy/corrupt/contentless),
or an environment where the epoch cannot be computed (non-Linux, no ``/proc``),
both degrade to the original presence-only behaviour — never fail-closed.
Why the max-age (#85433). The epoch handles the restart case, but it bakes in
the assumption that the action a drain protects always ends in a machine
restart. When a drain-gated action completes *without* recreating the container
and the writer never cancels the drain (writer crash, forgotten cleanup), the
orphaned marker still carries the *current* epoch — so the epoch check honours
it and the gateway bounces every inbound message forever (observed in the
field: a cloud instance refused all Telegram turns for ~3 days). The marker's
``requested_at`` timestamp is therefore also checked: a marker older than
:data:`DRAIN_REQUEST_MAX_AGE_SECONDS` reads as stale. Same leniency contract as
the epoch — only a *definite* expiry (timestamp present, parseable, and too
old) is ignored; a missing/corrupt timestamp still reads as drain-active. A
deliberately long drain has a sanctioned keep-alive: re-calling
:func:`write_drain_request` refreshes ``requested_at``.
Reading never raises: a malformed/half-written file reads as "present but
contentless" (``{}``), which still counts as drain-active — fail-safe toward
quiescing. Both staleness checks only reject on a *definite* verdict; a marker
with no epoch/timestamp, or a host without ``/proc``, degrades to the
presence-only behaviour and never fails closed.
"""
from __future__ import annotations
@@ -77,76 +47,41 @@ _log = logging.getLogger(__name__)
_DRAIN_REQUEST_FILENAME = ".drain_request.json"
# Max-age fallback for a same-epoch orphaned marker (#85433). Drain-gated
# lifecycle actions complete in minutes; an hour is comfortably past any
# legitimate drain while still bounding the wedge a leaked marker can cause
# (vs. the unbounded outage observed in the field). Long-running drains
# refresh the marker via write_drain_request() (idempotent re-write bumps
# ``requested_at``) rather than raising this bound.
# Drain-gated lifecycle actions complete in minutes; an hour bounds the wedge a
# leaked marker can cause. Long drains refresh the marker instead of raising this.
DRAIN_REQUEST_MAX_AGE_SECONDS = 3600.0
# Dedup guard for the expired-marker warning: the drain watcher re-reads the
# marker every second, and an expired orphan sits on disk until removed, so
# an unconditional warning would fire ~86k times/day. Keyed by the marker's
# ``requested_at`` string — a keep-alive re-write (new timestamp) that later
# expires again logs again, which is the desired behaviour.
# Dedup for the expired-marker warning: the watcher re-reads every second and an
# expired orphan sits on disk until removed. Keyed by ``requested_at`` so a
# keep-alive re-write that later expires again logs again.
_expiry_logged_for: Optional[str] = None
@functools.lru_cache(maxsize=1)
def current_instantiation_epoch() -> str:
"""Identity of THIS container / VM instantiation.
"""Identity of THIS container / VM instantiation ("<boot_id>:<pid1_start>").
Stable for the life of the PID-1 init process — so an s6 respawn of just
the gateway keeps the same epoch and an in-flight drain is honoured — but
changes when the machine/container is recreated (a fresh PID 1 → a fresh
epoch). Composed from two ``/proc`` facts:
Stable for the life of PID 1 (so an s6 respawn of just the gateway, or a
host ``hermes gateway restart`` under systemd/launchd, keeps honouring an
in-flight drain) but changes whenever the machine is recreated: boot_id
changes on a VM/microVM reboot, PID 1's start time on a plain
``docker restart`` (same host kernel, brand-new ``/init``).
* the kernel **boot id** (``/proc/sys/kernel/random/boot_id``) — changes
on a VM / microVM reboot (e.g. a Fly Firecracker machine restart);
* **PID 1's start time** (field 22 of ``/proc/1/stat``) — changes on a
plain ``docker restart`` (the host kernel, hence boot_id, is unchanged,
but ``/init`` is a brand-new process).
Together they discriminate every restart mode that matters:
| event | boot_id | pid1 start | epoch | marker |
|--------------------------------|---------|------------|--------|--------|
| Fly microVM reboot (auto-upd.) | changes | changes | NEW | reject |
| plain ``docker restart`` | same | changes | NEW | reject |
| s6 respawn of the gateway only | same | same | SAME | honour |
| host ``hermes gateway restart``| same | same(init) | SAME | honour |
The last row is intentional: a host install has no durable-volume drain
bug, and honouring a drain across a deliberate process restart is the
intended reversible behaviour (D4a) — PID 1 there is the long-lived init
(systemd/launchd), so the epoch is stable.
Returns ``""`` when neither identity source is readable (non-Linux, no
``/proc``). An empty epoch disables the staleness check downstream,
degrading to the released presence-only behaviour — never fail-closed.
Memoised: the epoch is constant for the life of the process.
Returns ``""`` when neither source is readable (non-Linux, no ``/proc``),
which disables the epoch check downstream — never fail-closed.
"""
boot_id = ""
try:
boot_id = (
Path("/proc/sys/kernel/random/boot_id")
.read_text(encoding="utf-8")
.strip()
)
boot_id = Path("/proc/sys/kernel/random/boot_id").read_text(encoding="utf-8").strip()
except OSError:
pass
pid1_start = ""
try:
# /proc/1/stat: "<pid> (<comm>) <state> ... <starttime@field22> ...".
# comm can contain spaces and parens, so split on the LAST ')' and
# index into the whitespace-delimited tail. starttime is field 22
# (1-indexed); after the comm the tail starts at field 3, so it is the
# tail's index 19.
# "<pid> (<comm>) <state> ...": comm may contain spaces/parens, so split
# on the LAST ')'. starttime is field 22 (1-indexed) = tail index 19.
stat = Path("/proc/1/stat").read_text(encoding="utf-8")
tail = stat.rsplit(")", 1)[1].split()
pid1_start = tail[19]
pid1_start = stat.rsplit(")", 1)[1].split()[19]
except (OSError, IndexError):
pass
@@ -167,27 +102,16 @@ def write_drain_request(
suppress_notification: bool = False,
home: Optional[Path] = None,
) -> dict[str, Any]:
"""Write the begin-drain marker. Returns the payload written.
"""Write the begin-drain marker atomically. Returns the payload written.
Atomic write so the gateway watcher never reads a half-written file.
Idempotent: re-writing while a drain is already in progress refreshes
``requested_at`` — the sanctioned keep-alive for a drain that legitimately
needs longer than :data:`DRAIN_REQUEST_MAX_AGE_SECONDS`.
Stamps the marker with :func:`current_instantiation_epoch` so a marker that
later survives a machine restart on the durable HERMES_HOME volume can be
recognised as stale and ignored (NS-570).
``suppress_notification`` is a generic "be quiet on the shutdown that ends
this drain" flag. When the drain culminates in a process exit (e.g. NAS
recreates the machine for an auto-update image migration), the gateway's
shutdown path reads it via :func:`drain_notification_suppressed` and skips
the *home-channel* "gateway shutting down" broadcast — the operator-flavoured
ping that would otherwise fire on every routine auto-update, potentially
dozens of times a day. It NEVER suppresses the per-active-session interrupt
ping. The gateway stays agnostic about *why* the drain is quiet; the policy
of which drain causes set the flag lives entirely in the caller (NAS). The
field defaults False so legacy/operator drains behave exactly as before.
Idempotent: re-writing refreshes ``requested_at`` (keep-alive past the
max-age). ``suppress_notification`` asks the shutdown that ends this drain
to skip ONLY the home-channel "gateway shutting down" broadcast (the
per-active-session interrupt ping is never suppressed); the policy of
which drains are quiet lives entirely in the caller. Defaults False so
legacy/operator drains behave exactly as before. The marker is stamped
with :func:`current_instantiation_epoch` so a copy surviving a machine
restart on the durable volume can be recognised as stale.
"""
payload = {
"action": "drain",
@@ -201,10 +125,7 @@ def write_drain_request(
def clear_drain_request(*, home: Optional[Path] = None) -> bool:
"""Remove the drain marker (cancel-drain). Returns True if one existed.
Best-effort: a missing file is not an error (cancel is idempotent).
"""
"""Remove the drain marker (cancel-drain, idempotent). Returns True if one existed."""
path = drain_request_path(home)
try:
path.unlink()
@@ -217,48 +138,19 @@ def clear_drain_request(*, home: Optional[Path] = None) -> bool:
def _marker_epoch_is_stale(body: dict[str, Any]) -> bool:
"""True iff ``body``'s epoch is a *definite* mismatch with this process.
Lenient by design — returns False (i.e. "not stale, honour it") whenever it
can't be sure:
* the current epoch can't be computed ("" fallback, no /proc), OR
* the marker carries no epoch (legacy marker, or a corrupt/contentless
``{}`` body).
Only a marker whose epoch is present AND differs from the current
instantiation epoch is considered stale. This preserves the
fail-safe-toward-quiescing contract for malformed markers.
"""
"""True iff ``body``'s epoch is a *definite* mismatch (both epochs known and differ)."""
current = current_instantiation_epoch()
if not current:
return False
marker_epoch = body.get("epoch")
if not marker_epoch:
return False
return marker_epoch != current
return bool(current and marker_epoch and marker_epoch != current)
def _marker_is_expired(body: dict[str, Any]) -> bool:
"""True iff ``body``'s ``requested_at`` is *definitely* too old (#85433).
"""True iff ``requested_at`` parses AND is older than the max-age.
The max-age fallback for a same-epoch orphan: a drain-gated action that
completes WITHOUT a machine restart leaves a marker the epoch check cannot
reject, and if the writer never cancels the drain the gateway is wedged in
``draining`` for the life of the container. Bounding the marker's lifetime
by :data:`DRAIN_REQUEST_MAX_AGE_SECONDS` converts that unbounded outage
into a self-healing one.
Same leniency contract as :func:`_marker_epoch_is_stale` — returns False
("not expired, honour it") whenever it can't be sure:
* the marker carries no ``requested_at`` (legacy/corrupt/contentless
body), OR
* the timestamp isn't a parseable ISO-8601 string.
Only a timestamp that parses AND lies more than the max-age in the past is
considered expired. A future-dated timestamp (clock skew) is honoured. The
expiry is logged loudly — but once per marker, not per poll: the gateway's
drain watcher re-reads the marker every second, and an expired orphan stays
on disk until an operator or writer removes it, so an unconditional warning
would repeat ~86k times/day. This path only fires when a writer leaked a
marker, and the log is the operator's breadcrumb.
Missing/unparseable timestamps and future-dated ones (clock skew) are
honoured. The expiry is logged once per marker, not per poll: this path
only fires when a writer leaked a marker, and the warning is the
operator's breadcrumb.
"""
global _expiry_logged_for
raw = body.get("requested_at")
@@ -289,72 +181,37 @@ def _marker_is_expired(body: dict[str, Any]) -> bool:
def _marker_is_stale(body: dict[str, Any]) -> bool:
"""True iff the marker is definitely from a drain that is already over.
Two independent, individually-lenient signals (either suffices):
* epoch mismatch — the marker survived a machine restart (NS-570);
* expiry — a same-epoch orphan outlived any legitimate drain (#85433).
"""
"""True iff the marker is definitely from a drain that is already over."""
return _marker_epoch_is_stale(body) or _marker_is_expired(body)
def drain_requested(*, home: Optional[Path] = None) -> bool:
"""True iff a begin-drain marker for THIS instantiation is present.
A marker whose ``epoch`` does not match the current instantiation epoch is
treated as absent: it survived a container/VM restart (HERMES_HOME is a
durable Fly volume on Hermes Cloud) and the lifecycle action that triggered
the drain has already completed — honouring it would wedge the
freshly-restarted gateway in ``draining`` (NS-570). A marker whose
``requested_at`` is older than :data:`DRAIN_REQUEST_MAX_AGE_SECONDS` is
likewise treated as absent: it is a same-epoch orphan whose drain-gated
action completed without a restart and was never cancelled (#85433). Both
staleness checks are lenient (see :func:`_marker_epoch_is_stale` /
:func:`_marker_is_expired`): a legacy/corrupt marker with no epoch and no
timestamp, or an environment without ``/proc``, still reads as
drain-active.
"""
def _active_drain_body(home: Optional[Path]) -> Optional[dict[str, Any]]:
"""Marker body if present AND not stale, else None."""
body = read_drain_request(home=home)
if body is None:
return False
if _marker_is_stale(body):
return False
return True
if body is None or _marker_is_stale(body):
return None
return body
def drain_requested(*, home: Optional[Path] = None) -> bool:
"""True iff an active (present, same-epoch, unexpired) begin-drain marker exists."""
return _active_drain_body(home) is not None
def drain_notification_suppressed(*, home: Optional[Path] = None) -> bool:
"""True iff an ACTIVE drain marker asks to suppress the shutdown broadcast.
"""True iff an ACTIVE drain marker explicitly asks to suppress the shutdown broadcast.
"Active" means exactly what :func:`drain_requested` means — a marker present
AND stamped with the current instantiation epoch AND not past its max-age.
A stale (other-epoch) marker that survived a machine restart on the durable
HERMES_HOME volume, or an expired same-epoch orphan (#85433), is
ignored here just as it is for drain state (NS-570): we must never let an
orphaned marker's flag silence a *fresh* gateway's legitimate shutdown
broadcast.
Only honours the flag when it is explicitly truthy in the marker body. A
legacy marker without the field, a corrupt/contentless ``{}`` body, or an
absent marker all read as "not suppressed" (False) — fail toward the louder,
more-visible behaviour, consistent with :func:`read_drain_request`'s
never-raise contract. The gateway's shutdown path uses this to skip ONLY the
home-channel broadcast; the per-active-session interrupt ping is unaffected.
Uses the same activeness rule as :func:`drain_requested`, so an orphaned
marker can never silence a fresh gateway's legitimate broadcast. A legacy
marker without the field or a contentless ``{}`` body reads as False
(fail toward the louder behaviour).
"""
body = read_drain_request(home=home)
if body is None:
return False
if _marker_is_stale(body):
return False
return bool(body.get("suppress_notification"))
body = _active_drain_body(home)
return bool(body and body.get("suppress_notification"))
def read_drain_request(*, home: Optional[Path] = None) -> Optional[dict[str, Any]]:
"""Return the marker payload, or ``None`` if absent.
A present-but-unparseable marker returns ``{}`` (truthy-presence preserved
via :func:`drain_requested`; callers that need the body get an empty dict
rather than an exception). Never raises.
"""
"""Return the marker payload, ``{}`` if present but unparseable, ``None`` if absent. Never raises."""
path = drain_request_path(home)
try:
raw = path.read_text(encoding="utf-8")
gateway/hosted_room_links.py
+33 -68
View File
@@ -36,6 +36,21 @@ _LEGACY_FIELDS = {
"trace_id",
"updated_at",
}
_OPTIONAL_FIELDS = {"transport_security", "status"}
# SQLite record columns that map 1:1 onto mapping fields, in record order
# (``catalog_json`` is the serialized ``catalog``).
_RECORD_FIELDS = (
"room_id",
"member_id",
"target_url",
"target_profile",
"grant",
"cancellation_scope_id",
"trace_id",
"transport_security",
"status",
"updated_at",
)
_STATUSES = {"ready", "unavailable", "needs_reauthorization"}
@@ -55,8 +70,7 @@ class StoredRoomLink:
@classmethod
def from_mapping(cls, value: Mapping[str, Any]) -> "StoredRoomLink":
allowed = _LEGACY_FIELDS | {"transport_security", "status"}
if set(value) - allowed or not _LEGACY_FIELDS.issubset(value):
if set(value) - _LEGACY_FIELDS - _OPTIONAL_FIELDS or not _LEGACY_FIELDS.issubset(value):
raise HostedRoomPeerError("stored room link fields are invalid")
room_id = _short_string(value["room_id"], "room_id")
member_id = _short_string(value["member_id"], "member_id")
@@ -96,54 +110,19 @@ class StoredRoomLink:
catalog = json.loads(str(value["catalog_json"]))
except Exception as exc:
raise HostedRoomPeerError("stored room link catalog is unreadable") from exc
return cls.from_mapping({
"room_id": value["room_id"],
"member_id": value["member_id"],
"target_url": value["target_url"],
"target_profile": value["target_profile"],
"grant": value["grant"],
"catalog": catalog,
"cancellation_scope_id": value["cancellation_scope_id"],
"trace_id": value["trace_id"],
"transport_security": value["transport_security"],
"status": value["status"],
"updated_at": value["updated_at"],
})
return cls.from_mapping(
{**{name: value[name] for name in _RECORD_FIELDS}, "catalog": catalog}
)
def catalog_mapping(self) -> dict[str, Any]:
value = {
"installation_id": self.catalog.installation_id,
"protocol_versions": list(self.catalog.protocol_versions),
"link_modes": list(self.catalog.link_modes),
"persistent_process": self.catalog.persistent_process,
"text": self.catalog.text,
"attachments": self.catalog.attachments,
"execution_policy": self.catalog.execution_policy.as_mapping(),
"catalog_digest": self.catalog.catalog_digest,
}
if (
self.catalog.endpoint_url is not None
or self.catalog.endpoint_reason is not None
):
value["endpoint"] = self.catalog.endpoint_mapping()
return value
return self.catalog.as_mapping()
def as_record(self) -> dict[str, Any]:
return {
"room_id": self.room_id,
"member_id": self.member_id,
"target_url": self.target_url,
"target_profile": self.target_profile,
"grant": self.grant,
"catalog_json": json.dumps(
self.catalog_mapping(), sort_keys=True, separators=(",", ":")
),
"cancellation_scope_id": self.cancellation_scope_id,
"trace_id": self.trace_id,
"transport_security": self.transport_security,
"status": self.status,
"updated_at": self.updated_at,
}
record = {name: getattr(self, name) for name in _RECORD_FIELDS}
record["catalog_json"] = json.dumps(
self.catalog_mapping(), sort_keys=True, separators=(",", ":")
)
return record
def _short_string(value: Any, field: str) -> str:
@@ -153,23 +132,24 @@ def _short_string(value: Any, field: str) -> str:
return normalized
def load_room_links(db_path: Path | str) -> tuple[StoredRoomLink, ...]:
def _link_rows(db_path: Path | str) -> list[dict[str, Any]]:
rows = hosted_rooms.list_room_link_records(db_path)
if len(rows) > MAX_LINKS:
raise HostedRoomPeerError("stored room link list is invalid")
return tuple(StoredRoomLink.from_record(row) for row in rows)
return rows
def load_room_links(db_path: Path | str) -> tuple[StoredRoomLink, ...]:
return tuple(StoredRoomLink.from_record(row) for row in _link_rows(db_path))
def load_room_links_tolerant(
db_path: Path | str,
) -> tuple[tuple[StoredRoomLink, ...], tuple[str, ...]]:
"""Load healthy routes while quarantining malformed rows by identity."""
rows = hosted_rooms.list_room_link_records(db_path)
if len(rows) > MAX_LINKS:
raise HostedRoomPeerError("stored room link list is invalid")
links = []
errors = []
for row in rows:
for row in _link_rows(db_path):
try:
links.append(StoredRoomLink.from_record(row))
except Exception:
@@ -225,22 +205,7 @@ def make_stored_link(
"target_url": target_url,
"target_profile": target_profile,
"grant": grant,
"catalog": {
"installation_id": catalog.installation_id,
"protocol_versions": list(catalog.protocol_versions),
"link_modes": list(catalog.link_modes),
"persistent_process": catalog.persistent_process,
"text": catalog.text,
"attachments": catalog.attachments,
"execution_policy": catalog.execution_policy.as_mapping(),
"catalog_digest": catalog.catalog_digest,
**(
{"endpoint": catalog.endpoint_mapping()}
if catalog.endpoint_url is not None
or catalog.endpoint_reason is not None
else {}
),
},
"catalog": catalog.as_mapping(),
"cancellation_scope_id": cancellation_scope_id,
"trace_id": trace_id,
"transport_security": transport_security,
gateway/hosted_room_peer.py
+78 -179
View File
@@ -1,8 +1,8 @@
"""Typed contracts for autonomous cross-gateway hosted-room members.
The Desktop may bootstrap an invitation, but it is never the issuer or the
runtime courier. The target gateway verifies a scoped grant and the full task
coordinates before it admits any model or tool work.
The Desktop may bootstrap an invitation but is never the issuer or runtime
courier: the target gateway verifies a scoped grant and the full task
coordinates before admitting any model or tool work.
"""
from __future__ import annotations
@@ -18,10 +18,10 @@ import re
import stat
import time
import urllib.parse
from dataclasses import dataclass
from dataclasses import asdict, dataclass
from functools import lru_cache
from pathlib import Path
from typing import Any, Iterable, Literal, Mapping
from typing import Any, Callable, Iterable, Literal, Mapping
from gateway.hosted_room_execution_policy import (
RoomExecutionPolicy,
@@ -29,21 +29,15 @@ from gateway.hosted_room_execution_policy import (
)
# Version 2 adds authority/member lineage to scoped grants. It is intentionally
# not wire-compatible with the unpublished v1 draft; mixed gateways must fall
# back to Desktop-driven rooms instead of accepting a weaker token shape.
# v2 adds authority/member lineage to scoped grants and is deliberately not
# wire-compatible with the unpublished v1 draft; mixed gateways must fall back
# to Desktop-driven rooms instead of accepting a weaker token shape.
PROTOCOL_VERSION = 2
MAX_TOKEN_BYTES = 16 * 1024
MAX_PROMPT_BYTES = 256 * 1024
_IDENTIFIER_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:@/-]{0,255}$")
_DIGEST_RE = re.compile(r"^[0-9a-f]{64}$")
_LINK_PRIORITY = {
"direct": 0,
"overlay": 1,
"relay": 2,
"pull": 3,
"desktop": 4,
}
_LINK_MODES = frozenset({"direct", "overlay", "relay", "pull", "desktop"})
LinkMode = Literal["direct", "overlay", "relay", "pull", "desktop"]
TransportSecurity = Literal["tls", "loopback"]
@@ -255,28 +249,16 @@ class GatewayRoomCatalog:
raise HostedRoomPeerError("link_modes must be a non-empty list")
links: list[LinkMode] = []
for item in links_raw:
if item not in _LINK_PRIORITY:
if item not in _LINK_MODES:
raise HostedRoomPeerError("link_modes contains an unsupported mode")
if item not in links:
links.append(item)
for field in ("persistent_process", "text", "attachments"):
if not isinstance(value[field], bool):
raise HostedRoomPeerError(f"{field} must be a boolean")
policy = RoomExecutionPolicy.from_mapping(value["execution_policy"])
unsigned = {
"installation_id": installation_id,
"protocol_versions": list(versions),
"link_modes": links,
"persistent_process": value["persistent_process"],
"text": value["text"],
"attachments": value["attachments"],
"execution_policy": RoomExecutionPolicy.from_mapping(
value["execution_policy"]
).as_mapping(),
}
endpoint_url = None
endpoint_reason = None
transport_security = None
endpoint_url = endpoint_reason = transport_security = None
if "endpoint" in value:
endpoint = value["endpoint"]
if not isinstance(endpoint, Mapping) or not isinstance(
@@ -296,11 +278,6 @@ class GatewayRoomCatalog:
raise HostedRoomPeerError(
"endpoint transport_security does not match its URL"
)
normalized_endpoint = {
"available": True,
"url": endpoint_url,
"transport_security": transport_security,
}
else:
_exact_fields(
endpoint,
@@ -310,30 +287,41 @@ class GatewayRoomCatalog:
endpoint_reason = _identifier(
endpoint["reason"], field="endpoint.reason"
)
normalized_endpoint = {
"available": False,
"reason": endpoint_reason,
}
unsigned["endpoint"] = normalized_endpoint
expected = hashlib.sha256(_canonical_json(unsigned)).hexdigest()
supplied = _digest(value["catalog_digest"], field="catalog_digest")
if not hmac.compare_digest(expected, supplied):
raise HostedRoomPeerError("catalog_digest does not match the catalog")
return cls(
catalog = cls(
installation_id=installation_id,
protocol_versions=versions,
link_modes=tuple(links),
persistent_process=value["persistent_process"],
text=value["text"],
attachments=value["attachments"],
execution_policy=RoomExecutionPolicy.from_mapping(
value["execution_policy"]
),
catalog_digest=supplied,
execution_policy=policy,
catalog_digest=_digest(value["catalog_digest"], field="catalog_digest"),
endpoint_url=endpoint_url,
endpoint_reason=endpoint_reason,
transport_security=transport_security,
)
unsigned = catalog.as_mapping()
del unsigned["catalog_digest"]
expected = hashlib.sha256(_canonical_json(unsigned)).hexdigest()
if not hmac.compare_digest(expected, catalog.catalog_digest):
raise HostedRoomPeerError("catalog_digest does not match the catalog")
return catalog
def as_mapping(self) -> dict[str, Any]:
"""Canonical catalog mapping; ``endpoint`` appears only when advertised."""
value = {
"installation_id": self.installation_id,
"protocol_versions": list(self.protocol_versions),
"link_modes": list(self.link_modes),
"persistent_process": self.persistent_process,
"text": self.text,
"attachments": self.attachments,
"execution_policy": self.execution_policy.as_mapping(),
"catalog_digest": self.catalog_digest,
}
if self.endpoint_url is not None or self.endpoint_reason is not None:
value["endpoint"] = self.endpoint_mapping()
return value
def endpoint_mapping(self) -> dict[str, Any]:
"""Return the normalized self-advertised endpoint capability."""
@@ -362,9 +350,8 @@ def catalog_mapping(
execution_policy: Mapping[str, Any] | None = None,
) -> dict[str, Any]:
"""Build a canonical catalog mapping with its digest."""
# A Desktop-managed gateway exits with the app. Treat the caller's flag
# as an upper bound so every local catalog construction site stays honest,
# including older call sites that still pass ``True`` explicitly.
# A Desktop-managed gateway exits with the app: the caller's flag is only an
# upper bound, so call sites that still pass ``True`` stay honest.
persistent_process = bool(persistent_process and os.getenv("HERMES_DESKTOP") != "1")
checked_policy = RoomExecutionPolicy.from_mapping(
execution_policy
@@ -377,9 +364,8 @@ def catalog_mapping(
)
)
# A RoomLink run is initiated by another installation. Process-wide YOLO
# mode bypasses the scoped approval ContextVar, so it cannot be made safe by
# rewriting the advertised policy. Refuse to advertise or accept remote
# room execution until the target enables manual or smart approvals.
# mode bypasses the scoped approval ContextVar, so rewriting the advertised
# policy cannot make it safe: refuse until manual or smart approvals are on.
if checked_policy.approval_mode == "off":
raise HostedRoomPeerError(
"remote room execution requires manual or smart approvals"
@@ -392,7 +378,7 @@ def catalog_mapping(
# Direct HTTPS/loopback is the only RoomLink transport implemented by
# this backend slice. Do not advertise pull/relay placeholders.
"link_modes": [mode for mode in dict.fromkeys(link_modes) if mode == "direct"],
"persistent_process": bool(persistent_process),
"persistent_process": persistent_process,
"text": bool(text),
"attachments": bool(attachments),
"execution_policy": checked_policy.as_mapping(),
@@ -423,7 +409,6 @@ def local_catalog_mapping(
persistent_process=True,
text=text,
attachments=attachments,
endpoint=local_room_link_endpoint(),
target_profile=target_profile,
execution_policy=execution_policy,
)
@@ -467,7 +452,7 @@ def _room_link_url_from_config(home: str) -> str | None:
def _configured_room_link_url() -> str | None:
"""Resolve the explicit endpoint with environment override precedence."""
"""Resolve the explicit endpoint: env override > profile config > root config."""
override = os.getenv("HERMES_ROOM_LINK_URL")
if override is not None:
return override
@@ -479,9 +464,8 @@ def _configured_room_link_url() -> str | None:
return configured
# RoomLink is a gateway reachability property, not a Bot personality
# setting. Named profiles may override it, but otherwise inherit the
# process gateway's root endpoint so adding a Bot does not require
# repeating network configuration in every profile.
# setting: named profiles may override it but otherwise inherit the process
# gateway's root endpoint, so adding a Bot needs no repeated network config.
root = get_default_hermes_root()
if root != home:
return _room_link_url_from_config(str(root))
@@ -491,9 +475,8 @@ def _configured_room_link_url() -> str | None:
def validate_room_link_url(value: Any) -> tuple[str, TransportSecurity]:
"""Validate a RoomLink endpoint and classify its transport protection.
Scoped grants and room prompts may use plaintext HTTP only when the peer
is reached through the local loopback interface. Every non-loopback
endpoint must use HTTPS.
Plaintext HTTP is allowed only toward the local loopback interface; every
non-loopback endpoint must use HTTPS.
"""
raw = str(value or "").strip().rstrip("/")
try:
@@ -523,6 +506,27 @@ def validate_room_link_url(value: Any) -> tuple[str, TransportSecurity]:
return raw, "loopback"
# Validator per dispatch field, in validation order. ``prompt`` and
# ``prompt_digest`` are checked together against each other in from_mapping.
_DISPATCH_FIELDS: dict[str, Callable[..., Any]] = {
"protocol_version": _positive_int,
"room_id": _identifier,
"home_install_id": _identifier,
"authority_gateway_id": _identifier,
"authority_epoch": _positive_int,
"member_id": _identifier,
"target_install_id": _identifier,
"target_profile": _identifier,
"task_id": _identifier,
"execution_generation": _positive_int,
"source_event_seq": _positive_int,
"cancellation_scope_id": _identifier,
"capability_digest": _digest,
"execution_policy_digest": _digest,
"trace_id": _identifier,
}
@dataclass(frozen=True)
class HostedMemberDispatch:
"""Recipient-validated identity for one remote room member attempt."""
@@ -547,48 +551,15 @@ class HostedMemberDispatch:
def as_mapping(self) -> dict[str, Any]:
"""Return the canonical wire mapping used for fingerprinting."""
return {
"protocol_version": self.protocol_version,
"room_id": self.room_id,
"home_install_id": self.home_install_id,
"authority_gateway_id": self.authority_gateway_id,
"authority_epoch": self.authority_epoch,
"member_id": self.member_id,
"target_install_id": self.target_install_id,
"target_profile": self.target_profile,
"task_id": self.task_id,
"execution_generation": self.execution_generation,
"source_event_seq": self.source_event_seq,
"cancellation_scope_id": self.cancellation_scope_id,
"prompt": self.prompt,
"prompt_digest": self.prompt_digest,
"capability_digest": self.capability_digest,
"execution_policy_digest": self.execution_policy_digest,
"trace_id": self.trace_id,
}
return asdict(self)
@classmethod
def from_mapping(cls, value: Mapping[str, Any]) -> "HostedMemberDispatch":
required = {
"protocol_version",
"room_id",
"home_install_id",
"authority_gateway_id",
"authority_epoch",
"member_id",
"target_install_id",
"target_profile",
"task_id",
"execution_generation",
"source_event_seq",
"cancellation_scope_id",
"prompt",
"prompt_digest",
"capability_digest",
"execution_policy_digest",
"trace_id",
}
_exact_fields(value, required=required, label="dispatch")
_exact_fields(
value,
required=set(_DISPATCH_FIELDS) | {"prompt", "prompt_digest"},
label="dispatch",
)
prompt = value["prompt"]
if not isinstance(prompt, str) or not prompt.strip():
raise HostedRoomPeerError("prompt must be a non-empty string")
@@ -599,87 +570,15 @@ class HostedMemberDispatch:
if not hmac.compare_digest(expected_prompt_digest, prompt_digest):
raise HostedRoomPeerError("prompt_digest does not match prompt")
return cls(
protocol_version=_positive_int(
value["protocol_version"], field="protocol_version"
),
room_id=_identifier(value["room_id"], field="room_id"),
home_install_id=_identifier(
value["home_install_id"], field="home_install_id"
),
authority_gateway_id=_identifier(
value["authority_gateway_id"], field="authority_gateway_id"
),
authority_epoch=_positive_int(
value["authority_epoch"], field="authority_epoch"
),
member_id=_identifier(value["member_id"], field="member_id"),
target_install_id=_identifier(
value["target_install_id"], field="target_install_id"
),
target_profile=_identifier(value["target_profile"], field="target_profile"),
task_id=_identifier(value["task_id"], field="task_id"),
execution_generation=_positive_int(
value["execution_generation"], field="execution_generation"
),
source_event_seq=_positive_int(
value["source_event_seq"], field="source_event_seq"
),
cancellation_scope_id=_identifier(
value["cancellation_scope_id"], field="cancellation_scope_id"
),
prompt=prompt,
prompt_digest=prompt_digest,
capability_digest=_digest(
value["capability_digest"], field="capability_digest"
),
execution_policy_digest=_digest(
value["execution_policy_digest"],
field="execution_policy_digest",
),
trace_id=_identifier(value["trace_id"], field="trace_id"),
**{
name: check(value[name], field=name)
for name, check in _DISPATCH_FIELDS.items()
},
)
@dataclass(frozen=True)
class RoomLinkProbe:
"""One gateway-verified route candidate."""
mode: LinkMode
verified: bool
encrypted: bool
latency_ms: float
def select_room_link(
probes: Iterable[RoomLinkProbe],
*,
desktop_available: bool,
) -> RoomLinkProbe | None:
"""Choose the fastest safe route without weakening encryption."""
candidates = [
probe
for probe in probes
if probe.verified
and probe.encrypted
and probe.mode != "desktop"
and math.isfinite(probe.latency_ms)
and probe.latency_ms >= 0
]
if candidates:
return min(
candidates,
key=lambda item: (_LINK_PRIORITY[item.mode], item.latency_ms),
)
if desktop_available:
return RoomLinkProbe(
mode="desktop",
verified=True,
encrypted=True,
latency_ms=0,
)
return None
_GRANT_FIELDS = {
"version",
"grant_id",
gateway/hosted_room_policy_checkpoint.py
+272 -280
View File
@@ -11,7 +11,7 @@ import json
import sqlite3
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Mapping
from typing import Any, Callable, Mapping
from gateway import hosted_rooms
@@ -26,6 +26,68 @@ _TERMINAL_KINDS = frozenset({
"turn.deferred",
})
_SCHEMA_DDL = (
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_cursors (
room_id TEXT PRIMARY KEY,
through_seq INTEGER NOT NULL DEFAULT 0,
stopped_through_seq INTEGER NOT NULL DEFAULT 0,
updated_at REAL NOT NULL DEFAULT 0
)""",
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_threads (
room_id TEXT NOT NULL,
thread_id TEXT NOT NULL,
discussion_event_id TEXT NOT NULL,
latest_user_seq INTEGER NOT NULL,
completed INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY(room_id, thread_id)
)""",
"""CREATE INDEX IF NOT EXISTS idx_hosted_room_policy_pending
ON hosted_room_policy_threads(
room_id, completed, latest_user_seq, thread_id
)""",
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_events (
room_id TEXT NOT NULL,
thread_id TEXT NOT NULL,
discussion_event_id TEXT NOT NULL,
seq INTEGER NOT NULL,
event_json TEXT NOT NULL,
PRIMARY KEY(room_id, seq)
)""",
"""CREATE INDEX IF NOT EXISTS idx_hosted_room_policy_events_active
ON hosted_room_policy_events(
room_id, discussion_event_id, seq
)""",
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_watermarks (
room_id TEXT NOT NULL,
thread_id TEXT NOT NULL,
member_id TEXT NOT NULL,
seen_through_seq INTEGER NOT NULL,
PRIMARY KEY(room_id, thread_id, member_id)
)""",
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_publications (
room_id TEXT NOT NULL,
task_id TEXT NOT NULL,
kind TEXT NOT NULL,
execution_generation INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL,
PRIMARY KEY(room_id, task_id, kind, execution_generation)
)""",
# Transcript stores only references into the already bounded room log, so
# prompt payloads are never duplicated outside room byte limits.
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_transcript (
room_id TEXT NOT NULL,
thread_id TEXT NOT NULL,
seq INTEGER NOT NULL,
kind TEXT NOT NULL,
settled_seq INTEGER,
PRIMARY KEY(room_id, thread_id, seq)
)""",
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_transcript_state (
room_id TEXT PRIMARY KEY,
schema_version INTEGER NOT NULL
)""",
)
@dataclass(frozen=True)
class PolicySnapshot:
@@ -37,12 +99,36 @@ class PolicySnapshot:
watermarks: Mapping[tuple[str, str], int]
def _event_from_room_row(row: sqlite3.Row) -> dict[str, Any]:
return {
"room_id": str(row["room_id"]),
"seq": int(row["seq"]),
"event_id": str(row["event_id"]),
"kind": str(row["kind"]),
"actor": json.loads(row["actor_json"]),
"authority_epoch": row["authority_epoch"],
"payload": json.loads(row["payload_json"]),
"created_at": float(row["created_at"]),
"idempotent": False,
}
def _require_room(conn: sqlite3.Connection, room_id: str) -> None:
if conn.execute(
"SELECT 1 FROM hosted_rooms WHERE room_id=?",
(room_id,),
).fetchone() is None:
raise hosted_rooms.RoomNotFoundError("hosted room not found")
class HostedRoomPolicyCheckpoint:
"""Incrementally index room policy without compacting visible history."""
def __init__(self, db_path: Path | str) -> None:
self.db_path = Path(db_path)
self._initialize()
with self._connect() as conn:
for ddl in _SCHEMA_DDL:
conn.execute(ddl)
def _connect(self) -> sqlite3.Connection:
from hermes_state import apply_wal_with_fallback
@@ -53,97 +139,8 @@ class HostedRoomPolicyCheckpoint:
apply_wal_with_fallback(conn, db_label="state.db (room policy checkpoint)")
return conn
def _initialize(self) -> None:
with self._connect() as conn:
conn.execute(
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_cursors (
room_id TEXT PRIMARY KEY,
through_seq INTEGER NOT NULL DEFAULT 0,
stopped_through_seq INTEGER NOT NULL DEFAULT 0,
updated_at REAL NOT NULL DEFAULT 0
)"""
)
conn.execute(
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_threads (
room_id TEXT NOT NULL,
thread_id TEXT NOT NULL,
discussion_event_id TEXT NOT NULL,
latest_user_seq INTEGER NOT NULL,
completed INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY(room_id, thread_id)
)"""
)
conn.execute(
"""CREATE INDEX IF NOT EXISTS idx_hosted_room_policy_pending
ON hosted_room_policy_threads(
room_id, completed, latest_user_seq, thread_id
)"""
)
conn.execute(
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_events (
room_id TEXT NOT NULL,
thread_id TEXT NOT NULL,
discussion_event_id TEXT NOT NULL,
seq INTEGER NOT NULL,
event_json TEXT NOT NULL,
PRIMARY KEY(room_id, seq)
)"""
)
conn.execute(
"""CREATE INDEX IF NOT EXISTS idx_hosted_room_policy_events_active
ON hosted_room_policy_events(
room_id, discussion_event_id, seq
)"""
)
conn.execute(
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_watermarks (
room_id TEXT NOT NULL,
thread_id TEXT NOT NULL,
member_id TEXT NOT NULL,
seen_through_seq INTEGER NOT NULL,
PRIMARY KEY(room_id, thread_id, member_id)
)"""
)
conn.execute(
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_publications (
room_id TEXT NOT NULL,
task_id TEXT NOT NULL,
kind TEXT NOT NULL,
execution_generation INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL,
PRIMARY KEY(room_id, task_id, kind, execution_generation)
)"""
)
conn.execute(
# Store only references into the already bounded room log. This
# avoids duplicating prompt payloads outside room byte limits.
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_transcript (
room_id TEXT NOT NULL,
thread_id TEXT NOT NULL,
seq INTEGER NOT NULL,
kind TEXT NOT NULL,
settled_seq INTEGER,
PRIMARY KEY(room_id, thread_id, seq)
)"""
)
conn.execute(
"""CREATE TABLE IF NOT EXISTS hosted_room_policy_transcript_state (
room_id TEXT PRIMARY KEY,
schema_version INTEGER NOT NULL
)"""
)
@staticmethod
def _event_json(event: Mapping[str, Any]) -> str:
return json.dumps(
dict(event),
ensure_ascii=True,
sort_keys=True,
separators=(",", ":"),
)
def _store_active_event(
self,
conn: sqlite3.Connection,
*,
event: Mapping[str, Any],
@@ -159,12 +156,17 @@ class HostedRoomPolicyCheckpoint:
thread_id,
discussion_event_id,
int(event["seq"]),
self._event_json(event),
json.dumps(
dict(event),
ensure_ascii=True,
sort_keys=True,
separators=(",", ":"),
),
),
)
@staticmethod
def _store_transcript_event(
self,
conn: sqlite3.Connection,
*,
event: Mapping[str, Any],
@@ -207,20 +209,6 @@ class HostedRoomPolicyCheckpoint:
(event["room_id"], thread_id, int(cutoff["seq"])),
)
@staticmethod
def _event_from_room_row(row: sqlite3.Row) -> dict[str, Any]:
return {
"room_id": str(row["room_id"]),
"seq": int(row["seq"]),
"event_id": str(row["event_id"]),
"kind": str(row["kind"]),
"actor": json.loads(row["actor_json"]),
"authority_epoch": row["authority_epoch"],
"payload": json.loads(row["payload_json"]),
"created_at": float(row["created_at"]),
"idempotent": False,
}
def _backfill_transcript(
self,
conn: sqlite3.Connection,
@@ -259,7 +247,7 @@ class HostedRoomPolicyCheckpoint:
and row["event_id"] not in settled_seq_by_message
):
continue
event = self._event_from_room_row(row)
event = _event_from_room_row(row)
thread_id = str(event["payload"].get("thread_id") or "")
if thread_id:
self._store_transcript_event(
@@ -269,13 +257,29 @@ class HostedRoomPolicyCheckpoint:
settled_seq=settled_seq_by_message.get(str(row["event_id"])),
)
def _transcript_events(
def _discussion_events(
self,
conn: sqlite3.Connection,
*,
room_id: str,
thread_id: str,
discussion_event_id: str,
bound_error: str,
) -> list[dict[str, Any]]:
"""Merge the thread transcript with the active projection, ordered by seq."""
active_rows = conn.execute(
"""SELECT event_json FROM hosted_room_policy_events
WHERE room_id=? AND discussion_event_id=?
ORDER BY seq LIMIT ?""",
(
room_id,
discussion_event_id,
MAX_ACTIVE_POLICY_EVENTS + 1,
),
).fetchall()
if len(active_rows) > MAX_ACTIVE_POLICY_EVENTS:
raise RuntimeError(bound_error)
rows = conn.execute(
"""WITH transcript_events(seq) AS (
SELECT seq FROM hosted_room_policy_transcript
@@ -294,22 +298,30 @@ class HostedRoomPolicyCheckpoint:
ORDER BY events.seq""",
(room_id, thread_id, room_id, thread_id, room_id),
).fetchall()
return [self._event_from_room_row(row) for row in rows]
events_by_seq = {
int(event["seq"]): event
for event in (
*(_event_from_room_row(row) for row in rows),
*(json.loads(row["event_json"]) for row in active_rows),
)
}
return [events_by_seq[seq] for seq in sorted(events_by_seq)]
def _apply_event(self, conn: sqlite3.Connection, event: Mapping[str, Any]) -> None:
# -- per-kind projection handlers (dispatched by _apply_event) -----------
def _apply_user_message(
self,
conn: sqlite3.Connection,
event: Mapping[str, Any],
payload: Mapping[str, Any],
) -> None:
room_id = str(event["room_id"])
seq = int(event["seq"])
kind = str(event.get("kind") or "")
payload = event.get("payload")
payload = payload if isinstance(payload, Mapping) else {}
if kind == "message.user":
thread_id = str(payload.get("thread_id") or "")
event_id = str(event.get("event_id") or "")
if not thread_id or not event_id:
return
conn.execute(
"""INSERT INTO hosted_room_policy_threads(
thread_id = str(payload.get("thread_id") or "")
event_id = str(event.get("event_id") or "")
if not thread_id or not event_id:
return
conn.execute(
"""INSERT INTO hosted_room_policy_threads(
room_id, thread_id, discussion_event_id,
latest_user_seq, completed
) VALUES (?, ?, ?, ?, 0)
@@ -317,92 +329,92 @@ class HostedRoomPolicyCheckpoint:
discussion_event_id=excluded.discussion_event_id,
latest_user_seq=excluded.latest_user_seq,
completed=0""",
(room_id, thread_id, event_id, seq),
)
self._store_active_event(
conn,
event=event,
thread_id=thread_id,
discussion_event_id=event_id,
)
self._store_transcript_event(
conn,
event=event,
thread_id=thread_id,
)
return
(room_id, thread_id, event_id, int(event["seq"])),
)
self._store_active_event(
conn,
event=event,
thread_id=thread_id,
discussion_event_id=event_id,
)
self._store_transcript_event(conn, event=event, thread_id=thread_id)
if kind in {"message.member", *_TERMINAL_KINDS}:
thread_id = str(payload.get("thread_id") or "")
discussion_event_id = str(payload.get("discussion_event_id") or "")
source = conn.execute(
"""SELECT 1 FROM hosted_room_policy_events
def _apply_discussion_event(
self,
conn: sqlite3.Connection,
event: Mapping[str, Any],
payload: Mapping[str, Any],
) -> None:
"""Index member messages and terminal turn outcomes of a known discussion."""
room_id = str(event["room_id"])
seq = int(event["seq"])
kind = str(event.get("kind") or "")
thread_id = str(payload.get("thread_id") or "")
discussion_event_id = str(payload.get("discussion_event_id") or "")
source = conn.execute(
"""SELECT 1 FROM hosted_room_policy_events
WHERE room_id=? AND discussion_event_id=? LIMIT 1""",
(room_id, discussion_event_id),
).fetchone()
if source is None:
return
self._store_active_event(
conn,
event=event,
thread_id=thread_id,
discussion_event_id=discussion_event_id,
)
if kind in _TERMINAL_KINDS:
task_id = str(payload.get("task_id") or "")
execution_generation = (
int(payload.get("execution_generation") or 0)
if kind == "turn.deferred"
else 0
)
if task_id:
conn.execute(
"""INSERT OR IGNORE INTO hosted_room_policy_publications(
(room_id, discussion_event_id),
).fetchone()
if source is None:
return
self._store_active_event(
conn,
event=event,
thread_id=thread_id,
discussion_event_id=discussion_event_id,
)
if kind not in _TERMINAL_KINDS:
return
task_id = str(payload.get("task_id") or "")
execution_generation = (
int(payload.get("execution_generation") or 0)
if kind == "turn.deferred"
else 0
)
if task_id:
conn.execute(
"""INSERT OR IGNORE INTO hosted_room_policy_publications(
room_id, task_id, kind, execution_generation, seq
) VALUES (?, ?, ?, ?, ?)""",
(
room_id,
task_id,
kind,
execution_generation,
seq,
),
)
member_id = str(payload.get("member_id") or "")
seen_through_seq = int(payload.get("seen_through_seq") or 0)
if kind == "turn.settled" and payload.get("message_event_id"):
messages = conn.execute(
"""SELECT seq, event_json FROM hosted_room_policy_events
(
room_id,
task_id,
kind,
execution_generation,
seq,
),
)
member_id = str(payload.get("member_id") or "")
seen_through_seq = int(payload.get("seen_through_seq") or 0)
if kind == "turn.settled" and payload.get("message_event_id"):
messages = conn.execute(
"""SELECT seq, event_json FROM hosted_room_policy_events
WHERE room_id=? AND discussion_event_id=?""",
(room_id, discussion_event_id),
).fetchall()
decoded_messages = [
json.loads(message["event_json"])
for message in messages
]
committed = next(
(
message
for message in decoded_messages
if message.get("event_id")
== payload["message_event_id"]
),
None,
(room_id, discussion_event_id),
).fetchall()
committed = next(
(
message
for message in (
json.loads(row["event_json"]) for row in messages
)
if committed is not None:
seen_through_seq = max(
seen_through_seq,
int(committed["seq"]),
)
self._store_transcript_event(
conn,
event=committed,
thread_id=thread_id,
settled_seq=int(event["seq"]),
)
if member_id and seen_through_seq > 0:
conn.execute(
"""INSERT INTO hosted_room_policy_watermarks(
if message.get("event_id") == payload["message_event_id"]
),
None,
)
if committed is not None:
seen_through_seq = max(seen_through_seq, int(committed["seq"]))
self._store_transcript_event(
conn,
event=committed,
thread_id=thread_id,
settled_seq=int(event["seq"]),
)
if member_id and seen_through_seq > 0:
conn.execute(
"""INSERT INTO hosted_room_policy_watermarks(
room_id, thread_id, member_id, seen_through_seq
) VALUES (?, ?, ?, ?)
ON CONFLICT(room_id, thread_id, member_id) DO UPDATE SET
@@ -410,43 +422,63 @@ class HostedRoomPolicyCheckpoint:
hosted_room_policy_watermarks.seen_through_seq,
excluded.seen_through_seq
)""",
(room_id, thread_id, member_id, seen_through_seq),
)
return
(room_id, thread_id, member_id, seen_through_seq),
)
if kind == "room.activity":
thread_id = str(payload.get("thread_id") or "")
discussion_event_id = str(payload.get("discussion_event_id") or "")
conn.execute(
"""DELETE FROM hosted_room_policy_events
def _apply_room_activity(
self,
conn: sqlite3.Connection,
event: Mapping[str, Any],
payload: Mapping[str, Any],
) -> None:
room_id = str(event["room_id"])
thread_id = str(payload.get("thread_id") or "")
discussion_event_id = str(payload.get("discussion_event_id") or "")
conn.execute(
"""DELETE FROM hosted_room_policy_events
WHERE room_id=? AND discussion_event_id=?""",
(room_id, discussion_event_id),
)
conn.execute(
"""DELETE FROM hosted_room_policy_threads
(room_id, discussion_event_id),
)
conn.execute(
"""DELETE FROM hosted_room_policy_threads
WHERE room_id=? AND thread_id=?""",
(room_id, thread_id),
)
return
(room_id, thread_id),
)
if kind == "room.stop_requested":
conn.execute(
"""UPDATE hosted_room_policy_cursors
def _apply_stop_requested(
self,
conn: sqlite3.Connection,
event: Mapping[str, Any],
payload: Mapping[str, Any],
) -> None:
conn.execute(
"""UPDATE hosted_room_policy_cursors
SET stopped_through_seq=MAX(stopped_through_seq, ?)
WHERE room_id=?""",
(seq, room_id),
)
(int(event["seq"]), str(event["room_id"])),
)
_APPLY_BY_KIND: dict[str, Callable[..., None]] = {
"message.user": _apply_user_message,
"message.member": _apply_discussion_event,
**dict.fromkeys(_TERMINAL_KINDS, _apply_discussion_event),
"room.activity": _apply_room_activity,
"room.stop_requested": _apply_stop_requested,
}
def _apply_event(self, conn: sqlite3.Connection, event: Mapping[str, Any]) -> None:
handler = self._APPLY_BY_KIND.get(str(event.get("kind") or ""))
if handler is None:
return
payload = event.get("payload")
handler(self, conn, event, payload if isinstance(payload, Mapping) else {})
def sync(self, *, room_id: str, latest_seq: int) -> int:
"""Materialize each unseen event exactly once by durable cursor."""
with self._connect() as conn:
conn.execute("BEGIN IMMEDIATE")
if conn.execute(
"SELECT 1 FROM hosted_rooms WHERE room_id=?",
(room_id,),
).fetchone() is None:
raise hosted_rooms.RoomNotFoundError("hosted room not found")
_require_room(conn, room_id)
conn.execute(
"""INSERT OR IGNORE INTO hosted_room_policy_cursors(
room_id, through_seq, stopped_through_seq, updated_at
@@ -499,11 +531,7 @@ class HostedRoomPolicyCheckpoint:
raise RuntimeError("hosted room policy cursor did not advance")
with self._connect() as conn:
conn.execute("BEGIN IMMEDIATE")
if conn.execute(
"SELECT 1 FROM hosted_rooms WHERE room_id=?",
(room_id,),
).fetchone() is None:
raise hosted_rooms.RoomNotFoundError("hosted room not found")
_require_room(conn, room_id)
for event in rows:
self._apply_event(conn, event)
updated = conn.execute(
@@ -545,44 +573,26 @@ class HostedRoomPolicyCheckpoint:
events=(),
watermarks={},
)
active_rows = conn.execute(
"""SELECT event_json FROM hosted_room_policy_events
WHERE room_id=? AND discussion_event_id=?
ORDER BY seq LIMIT ?""",
(
room_id,
str(thread["discussion_event_id"]),
MAX_ACTIVE_POLICY_EVENTS + 1,
),
).fetchall()
if len(active_rows) > MAX_ACTIVE_POLICY_EVENTS:
raise RuntimeError("active room policy projection exceeded its bound")
transcript_events = self._transcript_events(
thread_id = str(thread["thread_id"])
events = self._discussion_events(
conn,
room_id=room_id,
thread_id=str(thread["thread_id"]),
thread_id=thread_id,
discussion_event_id=str(thread["discussion_event_id"]),
bound_error="active room policy projection exceeded its bound",
)
watermark_rows = conn.execute(
"""SELECT member_id, seen_through_seq
FROM hosted_room_policy_watermarks
WHERE room_id=? AND thread_id=?""",
(room_id, str(thread["thread_id"])),
(room_id, thread_id),
).fetchall()
events_by_seq = {
int(event["seq"]): event
for event in (
*transcript_events,
*(json.loads(row["event_json"]) for row in active_rows),
)
}
return PolicySnapshot(
through_seq=through_seq,
stopped_through_seq=stopped_through_seq,
events=tuple(events_by_seq[seq] for seq in sorted(events_by_seq)),
events=tuple(events),
watermarks={
(str(thread["thread_id"]), str(row["member_id"])): int(
row["seen_through_seq"]
)
(thread_id, str(row["member_id"])): int(row["seen_through_seq"])
for row in watermark_rows
},
)
@@ -634,31 +644,13 @@ class HostedRoomPolicyCheckpoint:
).fetchone()
if source is None:
return []
active_rows = conn.execute(
"""SELECT event_json FROM hosted_room_policy_events
WHERE room_id=? AND discussion_event_id=?
ORDER BY seq LIMIT ?""",
(
room_id,
str(source["discussion_event_id"]),
MAX_ACTIVE_POLICY_EVENTS + 1,
),
).fetchall()
transcript_events = self._transcript_events(
return self._discussion_events(
conn,
room_id=room_id,
thread_id=str(source["thread_id"]),
discussion_event_id=str(source["discussion_event_id"]),
bound_error="task policy projection exceeded its bound",
)
if len(active_rows) > MAX_ACTIVE_POLICY_EVENTS:
raise RuntimeError("task policy projection exceeded its bound")
events_by_seq = {
int(event["seq"]): event
for event in (
*transcript_events,
*(json.loads(row["event_json"]) for row in active_rows),
)
}
return [events_by_seq[seq] for seq in sorted(events_by_seq)]
def compact_completed(self, *, room_id: str) -> None:
"""Drop any completed projections left by an interrupted sync."""
gateway/hosted_rooms.py
+103 -279
View File
@@ -282,11 +282,7 @@ def local_authority_gateway_id() -> str:
install_id = get_install_id()
if not install_id:
raise HostedRoomError("stable gateway install identity is unavailable")
return _validate_identifier(
f"install:{install_id}",
label="authority_gateway_id",
max_chars=MAX_ACTOR_ID_CHARS,
)
return _actor_id(f"install:{install_id}", "authority_gateway_id")
def _canonical_json(value: Any, *, label: str, max_bytes: int) -> str:
@@ -313,13 +309,46 @@ def _validate_identifier(value: Any, *, label: str, max_chars: int) -> str:
return value
def _room_id(value: Any) -> str:
return _validate_identifier(value, label="room_id", max_chars=MAX_ROOM_ID_CHARS)
def _event_id(value: Any, *, label: str = "event_id") -> str:
return _validate_identifier(value, label=label, max_chars=MAX_EVENT_ID_CHARS)
def _actor_id(value: Any, label: str) -> str:
return _validate_identifier(value, label=label, max_chars=MAX_ACTOR_ID_CHARS)
def _require_positive_int(value: Any, label: str) -> None:
if isinstance(value, bool) or not isinstance(value, int) or value < 1:
raise HostedRoomError(f"{label} must be a positive integer")
def _now(now: float | None) -> float:
return time.time() if now is None else float(now)
def _system_actor_json(actor_id: str) -> str:
return _canonical_json({"kind": "system", "id": actor_id}, label="actor", max_bytes=4 * 1024)
def _claim_payload_json(previous_gateway_id: str, new_gateway_id: str, epoch: int) -> str:
return _canonical_json(
{
"previous_gateway_id": previous_gateway_id,
"authority_gateway_id": new_gateway_id,
"authority_epoch": epoch,
},
label="payload",
max_bytes=MAX_EVENT_JSON_BYTES,
)
def user_event_id(client_event_id: Any) -> str:
"""Map a client retry key into the server-owned user-event namespace."""
normalized = _validate_identifier(
client_event_id,
label="event_id",
max_chars=MAX_EVENT_ID_CHARS,
)
normalized = _event_id(client_event_id)
digest = hashlib.sha256(normalized.encode("utf-8")).hexdigest()
return f"user:{digest}"
@@ -415,11 +444,7 @@ def _validate_actor(value: Any, *, kind: str) -> tuple[dict[str, str], str]:
if kind not in _EVENT_KINDS_BY_ACTOR[actor_kind]:
raise HostedRoomError(f"actor kind '{actor_kind}' cannot append '{kind}'")
actor_id = _validate_identifier(
value.get("id"),
label="actor.id",
max_chars=MAX_ACTOR_ID_CHARS,
)
actor_id = _actor_id(value.get("id"), "actor.id")
actor = {"kind": actor_kind, "id": actor_id}
for field, max_chars in (
("display_name", MAX_ACTOR_LABEL_CHARS),
@@ -557,11 +582,7 @@ def _initialize_schema(conn: sqlite3.Connection) -> None:
# Draft builds before the actor contract carried no identity. Preserve
# their inert replay rows explicitly as legacy system events rather
# than guessing a user or Bot author.
legacy_actor = _canonical_json(
{"kind": "system", "id": "legacy"},
label="actor",
max_bytes=4 * 1024,
)
legacy_actor = _system_actor_json("legacy")
escaped_actor = legacy_actor.replace("'", "''")
conn.execute(
"ALTER TABLE hosted_room_events "
@@ -731,7 +752,7 @@ def update_room_link_status(
WHERE room_id=? AND member_id=?""",
(
status,
float(now if now is not None else time.time()),
_now(now),
room_id,
member_id,
),
@@ -781,7 +802,7 @@ def revoke_room_grant_scope(
) -> None:
"""Revoke every grant issued at or before now for one exact room scope."""
scope_key = _room_grant_scope_key(claims)
timestamp = float(now if now is not None else time.time())
timestamp = _now(now)
expiry = float(expires_at)
if expiry <= timestamp:
return
@@ -827,27 +848,15 @@ def reserve_peer_room(
) -> None:
"""Fence direct Desktop prompts before the first peer run is admitted."""
timestamp = float(now if now is not None else time.time())
timestamp = _now(now)
expiry = float(expires_at)
if expiry <= timestamp:
raise HostedRoomError("peer room reservation must expire in the future")
values = (
_validate_identifier(
claims.get("room_id"), label="room_id", max_chars=MAX_ROOM_ID_CHARS
),
_validate_identifier(
claims.get("member_id"), label="member_id", max_chars=MAX_ACTOR_ID_CHARS
),
_validate_identifier(
claims.get("target_profile"),
label="target_profile",
max_chars=MAX_ACTOR_ID_CHARS,
),
_validate_identifier(
claims.get("authority_gateway_id"),
label="authority_gateway_id",
max_chars=MAX_ACTOR_ID_CHARS,
),
_room_id(claims.get("room_id")),
_actor_id(claims.get("member_id"), "member_id"),
_actor_id(claims.get("target_profile"), "target_profile"),
_actor_id(claims.get("authority_gateway_id"), "authority_gateway_id"),
int(claims.get("authority_epoch") or 0),
)
if values[4] < 1:
@@ -919,7 +928,7 @@ def peer_room_is_reserved(
) -> bool:
"""Return whether a live target-side RoomLink reservation fences Desktop."""
timestamp = float(now if now is not None else time.time())
timestamp = _now(now)
with _transaction(db_path) as conn:
row = conn.execute(
"""SELECT 1 FROM hosted_room_peer_reservations
@@ -927,14 +936,8 @@ def peer_room_is_reserved(
AND expires_at>? AND revoked_at IS NULL
LIMIT 1""",
(
_validate_identifier(
room_id, label="room_id", max_chars=MAX_ROOM_ID_CHARS
),
_validate_identifier(
target_profile,
label="target_profile",
max_chars=MAX_ACTOR_ID_CHARS,
),
_room_id(room_id),
_actor_id(target_profile, "target_profile"),
timestamp,
),
).fetchone()
@@ -949,23 +952,11 @@ def peer_room_grant_is_current(
) -> bool:
"""Require a grant to match the target's current live reservation."""
timestamp = float(now if now is not None else time.time())
room_id = _validate_identifier(
claims.get("room_id"), label="room_id", max_chars=MAX_ROOM_ID_CHARS
)
member_id = _validate_identifier(
claims.get("member_id"), label="member_id", max_chars=MAX_ACTOR_ID_CHARS
)
target_profile = _validate_identifier(
claims.get("target_profile"),
label="target_profile",
max_chars=MAX_ACTOR_ID_CHARS,
)
authority_gateway_id = _validate_identifier(
claims.get("authority_gateway_id"),
label="authority_gateway_id",
max_chars=MAX_ACTOR_ID_CHARS,
)
timestamp = _now(now)
room_id = _room_id(claims.get("room_id"))
member_id = _actor_id(claims.get("member_id"), "member_id")
target_profile = _actor_id(claims.get("target_profile"), "target_profile")
authority_gateway_id = _actor_id(claims.get("authority_gateway_id"), "authority_gateway_id")
authority_epoch = int(claims.get("authority_epoch") or 0)
if authority_epoch < 1:
raise HostedRoomError("authority_epoch must be positive")
@@ -995,7 +986,7 @@ def room_grant_is_revoked(
now: float | None = None,
) -> bool:
"""Return whether a grant predates its exact scope's revocation fence."""
timestamp = float(now if now is not None else time.time())
timestamp = _now(now)
scope_key = _room_grant_scope_key(claims)
issued_at = float(claims.get("issued_at") or 0)
with _transaction(db_path) as conn:
@@ -1018,7 +1009,7 @@ def upsert_remote_run_receipt(
now: float | None = None,
) -> None:
"""Durably bind one logical peer task attempt to its remote run handle."""
timestamp = float(now if now is not None else time.time())
timestamp = _now(now)
identity = _remote_run_identity(record)
with _transaction(db_path, immediate=True) as conn:
existing = conn.execute(
@@ -1236,9 +1227,7 @@ def _room_from_row(row: sqlite3.Row, *, idempotent: bool = False) -> dict[str, A
return room
def _event_storage_bytes(
*, event_id: str, kind: str, actor_json: str, payload_json: str
) -> int:
def _event_storage_bytes(event_id: str, kind: str, actor_json: str, payload_json: str) -> int:
return len((event_id + kind + actor_json + payload_json).encode("utf-8"))
@@ -1390,7 +1379,7 @@ def prune_disbanded_rooms(
) -> int:
"""Purge deleted Group Chat payloads while reserving their identities."""
timestamp = time.time() if now is None else float(now)
timestamp = _now(now)
with _transaction(db_path, immediate=True) as conn:
return _prune_disbanded_rooms_locked(conn, now=timestamp)
@@ -1421,19 +1410,11 @@ def create_room(
now: float | None = None,
) -> dict[str, Any]:
"""Create a room, or return the identical existing room idempotently."""
room_id = _validate_identifier(
room_id,
label="room_id",
max_chars=MAX_ROOM_ID_CHARS,
)
room_id = _room_id(room_id)
name = _validate_room_name(name)
normalized_members, members_json = _validate_members(members)
authority_gateway_id = _validate_identifier(
authority_gateway_id,
label="authority_gateway_id",
max_chars=MAX_ACTOR_ID_CHARS,
)
now = time.time() if now is None else float(now)
authority_gateway_id = _actor_id(authority_gateway_id, "authority_gateway_id")
now = _now(now)
with _transaction(db_path, immediate=True) as conn:
if conn.execute(
@@ -1464,32 +1445,10 @@ def create_room(
if legacy_adoption:
target_epoch = int(existing["authority_epoch"]) + 1
seq = int(existing["next_seq"])
claim_actor_json = _canonical_json(
{"kind": "system", "id": "authority-control"},
label="actor",
max_bytes=4 * 1024,
)
claim_payload_json = _canonical_json(
{
"previous_gateway_id": "legacy",
"authority_gateway_id": authority_gateway_id,
"authority_epoch": target_epoch,
},
label="payload",
max_bytes=MAX_EVENT_JSON_BYTES,
)
claim_bytes = _event_storage_bytes(
event_id="system:authority-adopted",
kind="authority.claimed",
actor_json=claim_actor_json,
payload_json=claim_payload_json,
)
_assert_event_capacity(
conn,
room=existing,
additional_bytes=claim_bytes,
allow_control=True,
)
claim_actor_json = _system_actor_json("authority-control")
claim_payload_json = _claim_payload_json("legacy", authority_gateway_id, target_epoch)
claim_bytes = _event_storage_bytes("system:authority-adopted", "authority.claimed", claim_actor_json, claim_payload_json)
_assert_event_capacity(conn, room=existing, additional_bytes=claim_bytes, allow_control=True)
conn.execute(
"""INSERT INTO hosted_room_events
(room_id, seq, event_id, kind, actor_json,
@@ -1627,19 +1586,11 @@ def rename_room(
now: float | None = None,
) -> dict[str, Any]:
"""Rename a live room and append its replay event atomically."""
room_id = _validate_identifier(
room_id, label="room_id", max_chars=MAX_ROOM_ID_CHARS
)
event_id = _validate_identifier(
event_id, label="event_id", max_chars=MAX_EVENT_ID_CHARS
)
room_id = _room_id(room_id)
event_id = _event_id(event_id)
name = _validate_room_name(name)
now = time.time() if now is None else float(now)
actor_json = _canonical_json(
{"kind": "system", "id": "room-control"},
label="actor",
max_bytes=4 * 1024,
)
now = _now(now)
actor_json = _system_actor_json("room-control")
payload_json = _canonical_json(
{"name": name}, label="payload", max_bytes=MAX_EVENT_JSON_BYTES
)
@@ -1671,12 +1622,7 @@ def rename_room(
return result
seq = int(room["next_seq"])
epoch = int(room["authority_epoch"])
event_bytes = _event_storage_bytes(
event_id=event_id,
kind="room.renamed",
actor_json=actor_json,
payload_json=payload_json,
)
event_bytes = _event_storage_bytes(event_id, "room.renamed", actor_json, payload_json)
_assert_event_capacity(conn, room=room, additional_bytes=event_bytes)
conn.execute(
"""UPDATE hosted_rooms
@@ -1727,16 +1673,8 @@ def append_event(
Repeating the same ``event_id`` and immutable content returns the original
event. Reusing the id for different content fails closed.
"""
room_id = _validate_identifier(
room_id,
label="room_id",
max_chars=MAX_ROOM_ID_CHARS,
)
event_id = _validate_identifier(
event_id,
label="event_id",
max_chars=MAX_EVENT_ID_CHARS,
)
room_id = _room_id(room_id)
event_id = _event_id(event_id)
kind = _validate_event_kind(kind)
normalized_actor, actor_json = _validate_actor(actor, kind=kind)
authority_scoped = normalized_actor["kind"] in {
@@ -1748,22 +1686,13 @@ def append_event(
normalized_authority_gateway_id: str | None = None
normalized_authority_epoch: int | None = None
if authority_scoped:
normalized_authority_gateway_id = _validate_identifier(
authority_gateway_id,
label="authority_gateway_id",
max_chars=MAX_ACTOR_ID_CHARS,
)
normalized_authority_gateway_id = _actor_id(authority_gateway_id, "authority_gateway_id")
if (
normalized_actor["kind"] == "gateway"
and normalized_actor["id"] != normalized_authority_gateway_id
):
raise HostedRoomError("gateway actor.id must match authority_gateway_id")
if (
isinstance(authority_epoch, bool)
or not isinstance(authority_epoch, int)
or authority_epoch < 1
):
raise HostedRoomError("authority_epoch must be a positive integer")
_require_positive_int(authority_epoch, "authority_epoch")
normalized_authority_epoch = authority_epoch
elif authority_gateway_id is not None or authority_epoch is not None:
raise HostedRoomError(
@@ -1771,12 +1700,8 @@ def append_event(
)
if not isinstance(payload, dict):
raise HostedRoomError("payload must be an object")
payload_json = _canonical_json(
payload,
label="payload",
max_bytes=MAX_EVENT_JSON_BYTES,
)
now = time.time() if now is None else float(now)
payload_json = _canonical_json(payload, label="payload", max_bytes=MAX_EVENT_JSON_BYTES)
now = _now(now)
with _transaction(db_path, immediate=True) as conn:
existing = conn.execute(
@@ -1811,12 +1736,7 @@ def append_event(
):
raise AuthorityConflictError("stale hosted room authority")
seq = int(room["next_seq"])
event_bytes = _event_storage_bytes(
event_id=event_id,
kind=kind,
actor_json=actor_json,
payload_json=payload_json,
)
event_bytes = _event_storage_bytes(event_id, kind, actor_json, payload_json)
_assert_event_capacity(
conn,
room=room,
@@ -1874,11 +1794,7 @@ def probe_hosted_room(db_path: Path | str, *, room_id: Any) -> bool:
WebSocket reader for SQLite's normal ten-second timeout.
"""
checked_room_id = _validate_identifier(
room_id,
label="room_id",
max_chars=MAX_ROOM_ID_CHARS,
)
checked_room_id = _room_id(room_id)
path = Path(db_path)
if not path.is_file():
return False
@@ -1916,20 +1832,12 @@ def probe_peer_room_reservation(
) -> bool:
"""Check a peer reservation without creating or migrating shared state."""
checked_room_id = _validate_identifier(
room_id,
label="room_id",
max_chars=MAX_ROOM_ID_CHARS,
)
checked_profile = _validate_identifier(
target_profile,
label="target_profile",
max_chars=MAX_ACTOR_ID_CHARS,
)
checked_room_id = _room_id(room_id)
checked_profile = _actor_id(target_profile, "target_profile")
path = Path(db_path)
if not path.is_file():
return False
checked_now = float(now if now is not None else time.time())
checked_now = _now(now)
try:
conn = sqlite3.connect(path, timeout=0.05)
try:
@@ -1964,11 +1872,7 @@ def room_state(
include_disbanded: bool = False,
) -> dict[str, Any]:
"""Return durable replay and authority state for one room."""
room_id = _validate_identifier(
room_id,
label="room_id",
max_chars=MAX_ROOM_ID_CHARS,
)
room_id = _room_id(room_id)
with _transaction(db_path) as conn:
row = conn.execute(
"""SELECT room_id, name, members_json, authority_gateway_id,
@@ -2006,11 +1910,7 @@ def request_room_stop(
) -> dict[str, Any]:
"""Append an idempotent fence that supersedes earlier user turns."""
cancel_id = _validate_identifier(
cancel_id,
label="cancel_id",
max_chars=MAX_EVENT_ID_CHARS,
)
cancel_id = _event_id(cancel_id, label="cancel_id")
digest = hashlib.sha256(cancel_id.encode()).hexdigest()[:32]
return append_event(
db_path,
@@ -2040,50 +1940,15 @@ def claim_authority(
replicated driver must call it only after its lease/quorum policy has
established that the previous owner can no longer commit.
"""
room_id = _validate_identifier(
room_id,
label="room_id",
max_chars=MAX_ROOM_ID_CHARS,
)
expected_gateway_id = _validate_identifier(
expected_gateway_id,
label="expected_gateway_id",
max_chars=MAX_ACTOR_ID_CHARS,
)
new_gateway_id = _validate_identifier(
new_gateway_id,
label="new_gateway_id",
max_chars=MAX_ACTOR_ID_CHARS,
)
event_id = _validate_identifier(
event_id,
label="event_id",
max_chars=MAX_EVENT_ID_CHARS,
)
if (
isinstance(expected_epoch, bool)
or not isinstance(expected_epoch, int)
or expected_epoch < 1
):
raise HostedRoomError("expected_epoch must be a positive integer")
now = time.time() if now is None else float(now)
room_id = _room_id(room_id)
expected_gateway_id = _actor_id(expected_gateway_id, "expected_gateway_id")
new_gateway_id = _actor_id(new_gateway_id, "new_gateway_id")
event_id = _event_id(event_id)
_require_positive_int(expected_epoch, "expected_epoch")
now = _now(now)
target_epoch = expected_epoch + 1
claim_actor = {"kind": "system", "id": "authority-control"}
claim_actor_json = _canonical_json(
claim_actor,
label="actor",
max_bytes=4 * 1024,
)
claim_payload = {
"previous_gateway_id": expected_gateway_id,
"authority_gateway_id": new_gateway_id,
"authority_epoch": target_epoch,
}
claim_payload_json = _canonical_json(
claim_payload,
label="payload",
max_bytes=MAX_EVENT_JSON_BYTES,
)
claim_actor_json = _system_actor_json("authority-control")
claim_payload_json = _claim_payload_json(expected_gateway_id, new_gateway_id, target_epoch)
with _transaction(db_path, immediate=True) as conn:
row = conn.execute(
@@ -2121,18 +1986,8 @@ def claim_authority(
raise AuthorityConflictError("hosted room authority changed")
else:
seq = int(row["next_seq"])
claim_bytes = _event_storage_bytes(
event_id=event_id,
kind="authority.claimed",
actor_json=claim_actor_json,
payload_json=claim_payload_json,
)
_assert_event_capacity(
conn,
room=row,
additional_bytes=claim_bytes,
allow_control=True,
)
claim_bytes = _event_storage_bytes(event_id, "authority.claimed", claim_actor_json, claim_payload_json)
_assert_event_capacity(conn, room=row, additional_bytes=claim_bytes, allow_control=True)
conn.execute(
"""INSERT INTO hosted_room_events
(room_id, seq, event_id, kind, actor_json, authority_epoch,
@@ -2201,23 +2056,10 @@ def disband_room(
now: float | None = None,
) -> dict[str, Any]:
"""Tombstone a room id permanently and idempotently."""
room_id = _validate_identifier(
room_id,
label="room_id",
max_chars=MAX_ROOM_ID_CHARS,
)
expected_gateway_id = _validate_identifier(
expected_gateway_id,
label="expected_gateway_id",
max_chars=MAX_ACTOR_ID_CHARS,
)
if (
isinstance(expected_epoch, bool)
or not isinstance(expected_epoch, int)
or expected_epoch < 1
):
raise HostedRoomError("expected_epoch must be a positive integer")
now = time.time() if now is None else float(now)
room_id = _room_id(room_id)
expected_gateway_id = _actor_id(expected_gateway_id, "expected_gateway_id")
_require_positive_int(expected_epoch, "expected_epoch")
now = _now(now)
with _transaction(db_path, immediate=True) as conn:
room = conn.execute(
@@ -2268,28 +2110,14 @@ def disband_room(
):
raise AuthorityConflictError("stale hosted room authority")
seq = int(room["next_seq"])
actor_json = _canonical_json(
{"kind": "system", "id": "room-control"},
label="actor",
max_bytes=4 * 1024,
)
actor_json = _system_actor_json("room-control")
payload_json = _canonical_json(
{"room_id": room_id},
label="payload",
max_bytes=MAX_EVENT_JSON_BYTES,
)
disband_bytes = _event_storage_bytes(
event_id="system:room-disbanded",
kind="room.disbanded",
actor_json=actor_json,
payload_json=payload_json,
)
_assert_event_capacity(
conn,
room=room,
additional_bytes=disband_bytes,
allow_control=True,
)
disband_bytes = _event_storage_bytes("system:room-disbanded", "room.disbanded", actor_json, payload_json)
_assert_event_capacity(conn, room=room, additional_bytes=disband_bytes, allow_control=True)
conn.execute(
"""INSERT INTO hosted_room_events
(room_id, seq, event_id, kind, actor_json, authority_epoch,
@@ -2357,11 +2185,7 @@ def read_events(
include_disbanded: bool = False,
) -> dict[str, Any]:
"""Read a monotonic room-log delta after ``since_seq``."""
room_id = _validate_identifier(
room_id,
label="room_id",
max_chars=MAX_ROOM_ID_CHARS,
)
room_id = _room_id(room_id)
if isinstance(since_seq, bool) or not isinstance(since_seq, int) or since_seq < 0:
raise HostedRoomError("since_seq must be a non-negative integer")
if (
gateway/media_policy.py
+31 -40
View File
@@ -7,24 +7,15 @@ from environment variables:
- ``HERMES_MEDIA_ALLOW_DIRS`` <- gateway.media_delivery_allow_dirs
- ``HERMES_MEDIA_TRUST_RECENT_FILES`` <- gateway.trust_recent_files
Historically the config.yaml -> env translation ran ONLY in gateway startup
(gateway/run.py), so any process that delivers media without booting the
gateway — a manual ``hermes cron run`` in the CLI, ``hermes send``, a
standalone cron tick — filtered MEDIA paths under DIFFERENT policy than the
gateway's scheduled deliveries. In strict/allowlisted enterprise deployments
that divergence silently dropped attachments from manual cron runs while
scheduled runs delivered them (text is unaffected — only media goes through
path validation).
The translation used to run only in gateway startup, so standalone delivery
paths (``hermes cron run``, ``hermes send``, a standalone cron tick) filtered
MEDIA paths under a different policy and silently dropped attachments in
strict/allowlisted deployments (text is unaffected -- only media goes through
path validation). ``apply_media_policy_env()`` is the shared,
idempotent helper every delivery entrypoint calls before filtering media paths.
``apply_media_policy_env()`` is that same translation as a shared, idempotent
helper. Gateway startup calls it, and every standalone delivery entrypoint
calls it immediately before filtering media paths.
Precedence: an explicitly-set environment variable WINS over config.yaml.
This preserves both the operator contract (env overrides are how deployments
pin behavior) and gateway/run.py's historical shape (it only wrote the env
var when the config key was present; we additionally refuse to overwrite a
pre-existing env value so a shell-exported override survives).
Precedence: an explicitly-set environment variable WINS over config.yaml, so a
shell-exported override (and gateway startup's own earlier run) survives.
"""
from __future__ import annotations
@@ -52,37 +43,37 @@ def _load_gateway_cfg(config: Optional[Dict[str, Any]] = None) -> Dict[str, Any]
return gateway_cfg if isinstance(gateway_cfg, dict) else {}
def _set_env_default(env: str, value: str) -> None:
"""Set ``env`` only when unset/empty and ``value`` is non-empty (env wins)."""
if value and not os.environ.get(env):
os.environ[env] = value
def _allow_dirs_str(allow_dirs: Any) -> str:
if isinstance(allow_dirs, str):
return allow_dirs
if isinstance(allow_dirs, (list, tuple)):
return os.pathsep.join(str(p) for p in allow_dirs if p)
return ""
def apply_media_policy_env(config: Optional[Dict[str, Any]] = None) -> None:
"""Bridge gateway media-policy settings from config.yaml into the env.
Idempotent and env-wins: a variable already present in the environment is
never overwritten, so gateway startup (which runs this same helper) and
operator shell exports keep precedence. Never raises — a policy-bridge
failure must not break delivery; the validator falls back to its
defaults exactly as before.
Idempotent and env-wins: a variable already present is never overwritten.
Never raises — a policy-bridge failure must not break delivery; the
validator falls back to its defaults exactly as before.
"""
try:
gateway_cfg = _load_gateway_cfg(config)
if not gateway_cfg:
return
strict = gateway_cfg.get("strict")
if strict is not None and not os.environ.get(_STRICT_ENV):
os.environ[_STRICT_ENV] = "1" if strict else "0"
for key, env in (("strict", _STRICT_ENV), ("trust_recent_files", _TRUST_RECENT_ENV)):
flag = gateway_cfg.get(key)
if flag is not None:
_set_env_default(env, "1" if flag else "0")
allow_dirs = gateway_cfg.get("media_delivery_allow_dirs")
if allow_dirs and not os.environ.get(_ALLOW_DIRS_ENV):
if isinstance(allow_dirs, str):
allow_dirs_str = allow_dirs
elif isinstance(allow_dirs, (list, tuple)):
allow_dirs_str = os.pathsep.join(str(p) for p in allow_dirs if p)
else:
allow_dirs_str = ""
if allow_dirs_str:
os.environ[_ALLOW_DIRS_ENV] = allow_dirs_str
trust_recent = gateway_cfg.get("trust_recent_files")
if trust_recent is not None and not os.environ.get(_TRUST_RECENT_ENV):
os.environ[_TRUST_RECENT_ENV] = "1" if trust_recent else "0"
if allow_dirs:
_set_env_default(_ALLOW_DIRS_ENV, _allow_dirs_str(allow_dirs))
except Exception: # noqa: BLE001 - policy bridge must never break delivery
logger.debug("apply_media_policy_env failed", exc_info=True)
gateway/media_repair.py
+48 -72
View File
@@ -1,21 +1,19 @@
"""Repair model-mangled ``computer_use`` screenshot paths in final responses.
``computer_use`` persists a bounded screenshot into the Hermes image cache and
tells the model its absolute path. Some models rewrite a Windows path into a
POSIX-looking one (``C:\\Users\\Alice\\...`` -> ``/Users/Alice/...``) when
emitting an explicit ``MEDIA:`` directive, so delivery-path validation rejects
the nonexistent path and the attachment is dropped.
``computer_use`` persists a screenshot into the Hermes image cache and tells the
model its absolute path. Some models rewrite a Windows path into a POSIX-looking
one (``C:\\Users\\Alice\\...`` -> ``/Users/Alice/...``) inside an explicit
``MEDIA:`` directive, so delivery-path validation rejects it and drops the
attachment.
The repair is deliberately narrow: it only rewrites paths inside a response
that *already* carries an explicit ``MEDIA:`` directive, and only when the
directive's generated ``computer_use_<uuid>`` basename exactly matches a
canonical screenshot path returned by ``computer_use`` in the current turn.
It never auto-attaches captures, and normal media path validation still runs
after the repair.
The repair is deliberately narrow: it only rewrites paths inside a response that
*already* carries an explicit ``MEDIA:`` directive, and only when the directive's
generated ``computer_use_<uuid>`` basename exactly matches a canonical screenshot
path returned by ``computer_use`` in the current turn. It never auto-attaches
captures; normal media path validation still runs afterwards.
This lives in its own module (mirroring ``gateway/media_policy.py``) so every
delivery surface — the messaging gateway's main turn path, gateway background
tasks, and cron job delivery — shares one implementation.
Own module (like ``gateway/media_policy.py``) so the gateway turn path, gateway
background tasks and cron delivery share one implementation.
"""
from __future__ import annotations
@@ -27,9 +25,8 @@ from typing import Any, Dict, Iterator, List
logger = logging.getLogger(__name__)
# Absolute-path prefix accepted for canonical capture paths: Windows drive
# letter, POSIX root, or UNC share. Kept as a pattern string so the summary
# regex below and the compiled prefix check stay in sync.
# Absolute-path prefix for canonical capture paths: Windows drive letter, POSIX
# root, or UNC share. Shared string so the summary regex stays in sync.
_ABS_PATH_PREFIX_PATTERN = r"(?:[A-Za-z]:[/\\]|/|\\\\)"
_ABS_PATH_PREFIX_RE = re.compile(r"^" + _ABS_PATH_PREFIX_PATTERN)
@@ -61,30 +58,25 @@ def tool_name_by_call_id(messages: List[Dict[str, Any]]) -> Dict[str, str]:
def _computer_use_capture_basename(path: Any) -> str:
"""Return a canonical capture basename for either path separator style."""
"""Canonical (lowercased) capture basename for either separator style, or ''."""
value = str(path or "").strip().strip("`\"'")
basename = re.split(r"[/\\]", value)[-1]
if _COMPUTER_USE_CAPTURE_BASENAME_RE.fullmatch(basename):
return basename.lower()
return ""
return basename.lower() if _COMPUTER_USE_CAPTURE_BASENAME_RE.fullmatch(basename) else ""
def _iter_computer_use_capture_paths(content: Any) -> Iterator[str]:
"""Yield persisted screenshot paths from computer_use result content.
The tool can return JSON, a multimodal content list, or a text fallback.
The latter two retain the canonical path in the human-readable summary
even though the multimodal envelope's ``meta`` dictionary is not stored in
the tool message.
The tool can return JSON, a multimodal content list, or a text fallback; the
latter two keep the canonical path in the human-readable summary even though
the multimodal envelope's ``meta`` is not stored in the tool message.
"""
if isinstance(content, str):
stripped = content.strip()
if stripped.startswith(("{", "[")):
# JSON-looking content: parse first, never regex-scan the raw
# text. JSON escaping doubles backslashes, so a summary-regex hit
# on the raw string would yield ``C:\\\\Users\\\\...`` — a path
# that exists nowhere. Fail closed on unparseable JSON (tool
# output truncation) rather than repair to a corrupted path.
# Parse JSON first, never regex-scan it: JSON escaping doubles
# backslashes, so a raw-text hit would yield a path that exists
# nowhere. Fail closed on unparseable (truncated) JSON.
try:
payload = json.loads(stripped)
except Exception:
@@ -111,15 +103,29 @@ def _iter_computer_use_capture_paths(content: Any) -> Iterator[str]:
if isinstance(meta, dict) and isinstance(meta.get("screenshot_path"), str):
yield meta["screenshot_path"]
# Producer shapes (tools/computer_use/tool.py::_capture_response):
# "content"/"text" — multimodal envelope parts; "text_summary"/"summary" —
# the human-readable summary carrying the "(shareable screenshot saved
# to ...)" line.
# content/text = multimodal parts; text_summary/summary = the line carrying
# "(shareable screenshot saved to ...)".
for field in ("content", "text", "text_summary", "summary"):
nested = content.get(field)
if isinstance(nested, (str, dict, list)):
yield from _iter_computer_use_capture_paths(nested)
def _current_turn_messages(messages: List[Dict[str, Any]], history_offset: int) -> List[Dict[str, Any]]:
if not history_offset:
return messages
if len(messages) >= history_offset:
return messages[history_offset:]
# Compression can invalidate the slice boundary: recover the turn from its
# last user message, fail closed if none remains. Deliberately narrower than
# gateway/run.py::_collect_auto_append_media_tags' scan-everything fallback —
# that decides whether to ATTACH; this only rewrites paths already emitted.
for index in range(len(messages) - 1, -1, -1):
if messages[index].get("role") == "user":
return messages[index:]
return []
def repair_explicit_computer_use_media_paths(
response: str,
messages: List[Dict[str, Any]],
@@ -127,13 +133,10 @@ def repair_explicit_computer_use_media_paths(
) -> str:
"""Recover model-mangled paths for explicitly requested screenshots.
Repair only an already-explicit ``MEDIA:`` directive whose unique
generated basename case-insensitively matches a canonical screenshot
path from this turn. This does not auto-attach ordinary computer-use
captures, and normal media path validation still runs after the repair.
Fail-open: the repair is cosmetic, so an unexpected error returns the
response unchanged rather than aborting delivery.
Repairs only an already-explicit ``MEDIA:`` directive whose generated
basename case-insensitively matches a canonical screenshot path from this
turn. Fail-open: the repair is cosmetic, so any unexpected error returns
the response unchanged rather than aborting delivery.
"""
try:
return _repair_explicit_computer_use_media_paths_inner(
@@ -152,27 +155,7 @@ def _repair_explicit_computer_use_media_paths_inner(
if "MEDIA:" not in response:
return response
if history_offset and len(messages) >= history_offset:
turn_messages = messages[history_offset:]
elif history_offset:
# Compression can invalidate the original slice boundary. Recover the
# current turn from its last user message; fail closed if none
# remains. (Deliberately narrower than the scan-everything fallback
# in gateway/run.py::_collect_auto_append_media_tags — that helper
# decides whether to ATTACH, this one only rewrites paths the model
# already explicitly emitted.)
last_user = next(
(
index
for index in range(len(messages) - 1, -1, -1)
if messages[index].get("role") == "user"
),
None,
)
turn_messages = messages[last_user:] if last_user is not None else []
else:
turn_messages = messages
turn_messages = _current_turn_messages(messages, history_offset)
call_id_names = tool_name_by_call_id(turn_messages)
canonical_by_basename: Dict[str, str] = {}
@@ -181,10 +164,7 @@ def _repair_explicit_computer_use_media_paths_inner(
continue
call_id = str(msg.get("tool_call_id") or msg.get("call_id") or "")
tool_name = str(
msg.get("name")
or msg.get("tool_name")
or call_id_names.get(call_id)
or ""
msg.get("name") or msg.get("tool_name") or call_id_names.get(call_id) or ""
)
if tool_name != "computer_use":
continue
@@ -196,18 +176,14 @@ def _repair_explicit_computer_use_media_paths_inner(
if not canonical_by_basename:
return response
# Lazy on purpose: keeps `import gateway.media_repair` cheap for
# standalone cron processes that may never hit a MEDIA: response.
# No import cycle either way (base.py imports neither this module
# nor gateway.run at module level).
# Lazy: keeps `import gateway.media_repair` cheap for standalone cron
# processes that may never hit a MEDIA: response. No import cycle either way.
from gateway.platforms.base import BasePlatformAdapter
media_files, _ = BasePlatformAdapter.extract_media(response)
repaired = response
for emitted_path, _is_voice in media_files:
canonical = canonical_by_basename.get(
_computer_use_capture_basename(emitted_path)
)
canonical = canonical_by_basename.get(_computer_use_capture_basename(emitted_path))
if canonical and emitted_path != canonical:
repaired = repaired.replace(emitted_path, canonical)
return repaired
gateway/memory_status.py
+52 -98
View File
@@ -1,36 +1,18 @@
"""Memory status rollup for ``/api/status`` (NS-656).
"""Memory status rollup for ``/api/status``.
The gateway already *produces* every memory-pressure signal a user would
want to know about, but all of it dies in log files:
Read side for memory-pressure signals the gateway already persists but only
logs: the 30s ``state/gateway.heartbeat`` (RSS + MemAvailable/MemTotal + swap,
from :func:`gateway.shutdown_watchdog.write_loop_heartbeat`) and the lifecycle
sentinel's ``suspected_oom`` flag (:func:`gateway.lifecycle_ledger.record_startup`).
Distills them into a compact block the dashboard SPA and the NAS availability
sweep consume — no new sampling, no IPC, two small file reads.
* :func:`gateway.shutdown_watchdog.write_loop_heartbeat` embeds a
:func:`gateway.lifecycle_ledger.sample_memory` snapshot (gateway RSS +
system MemAvailable/MemTotal + swap) in ``state/gateway.heartbeat``
every 30 seconds.
* :func:`gateway.lifecycle_ledger.record_startup` detects an unclean
previous death and flags ``suspected_oom`` — but only into
``gateway-exit-diag.log`` and a WARNING line.
* ``gateway/agent_cache_pressure.py`` evicts transcripts under pressure,
again log-only.
Public-safety: ``/api/status`` is unauthenticated (``PUBLIC_API_PATHS``), so this
block carries only coarse numbers (MB granularity), enums, and booleans — the
same disclosure class as ``active_agents`` / ``nous_session_valid``.
So a hosted agent can be OOM-killed hourly (the BlueAtlas incident,
NS-608) while its dashboard and the NAS agent card both look perfectly
healthy. This module is the read side that closes the gap: it distills
the *already-persisted* heartbeat + lifecycle sentinel into a compact,
public-safe block that ``/api/status`` can serve to the dashboard SPA
and the NAS availability sweep — no new sampling, no IPC with the
gateway process, just two small file reads.
Public-safety note: ``/api/status`` is an unauthenticated liveness probe
(``PUBLIC_API_PATHS``), which is exactly why NAS can consume it. This
block therefore carries only coarse numbers (MB granularity), enums, and
booleans — the same disclosure class as the existing ``active_agents``
count and ``nous_session_valid`` field (which was added for the same
NAS-sweep audience).
Everything here is best-effort and read-only: a missing/corrupt file
degrades to ``pressure="unknown"`` rather than raising into the status
endpoint.
Best-effort and read-only: a missing/corrupt file degrades to
``pressure="unknown"`` rather than raising into the status endpoint.
"""
from __future__ import annotations
@@ -42,30 +24,32 @@ from typing import Any, Dict, Optional
logger = logging.getLogger(__name__)
# Pressure thresholds on system MemAvailable. ``critical`` deliberately
# mirrors the lifecycle ledger's OOM-suspicion heuristics
# (:data:`gateway.lifecycle_ledger._LOW_MEM_AVAILABLE_KIB` /
# ``_LOW_MEM_AVAILABLE_FRACTION``): if a memory level would make a
# subsequent unclean death "suspected OOM", the user should already have
# been warned at that level while the process was still alive.
# Thresholds on system MemAvailable. ``critical`` mirrors the lifecycle ledger's
# OOM-suspicion heuristics (_LOW_MEM_AVAILABLE_KIB / _LOW_MEM_AVAILABLE_FRACTION):
# a level that would make a later unclean death "suspected OOM" should already
# warn while the process is alive.
_CRITICAL_AVAILABLE_KIB = 64 * 1024 # < 64 MiB available
_CRITICAL_AVAILABLE_FRACTION = 0.05 # < 5% of MemTotal
_ELEVATED_AVAILABLE_KIB = 128 * 1024 # < 128 MiB available
_ELEVATED_AVAILABLE_FRACTION = 0.15 # < 15% of MemTotal
# A heartbeat older than this no longer describes the present. The writer
# cadence is 30s (DEFAULT_HEARTBEAT_INTERVAL_S); 150s of slack tolerates a
# briefly stalled loop without letting a long-dead gateway's last sample
# masquerade as current pressure.
# Writer cadence is 30s (DEFAULT_HEARTBEAT_INTERVAL_S); 150s tolerates a briefly
# stalled loop without letting a long-dead gateway's last sample pose as current.
_HEARTBEAT_FRESH_TTL_S = 150.0
_KIB_PER_MB = 1024
def _mb(kib: Any) -> Optional[int]:
if isinstance(kib, bool) or not isinstance(kib, int) or kib < 0:
def _nonneg_int(value: Any) -> Optional[int]:
"""Return *value* if it is a non-negative int (bools rejected), else None."""
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
return None
return kib // _KIB_PER_MB
return value
def _mb(kib: Any) -> Optional[int]:
kib = _nonneg_int(kib)
return None if kib is None else kib // _KIB_PER_MB
def _parse_iso(value: Any) -> Optional[datetime]:
@@ -75,40 +59,26 @@ def _parse_iso(value: Any) -> Optional[datetime]:
parsed = datetime.fromisoformat(value)
except ValueError:
return None
if parsed.tzinfo is None:
parsed = parsed.replace(tzinfo=timezone.utc)
return parsed
return parsed if parsed.tzinfo is not None else parsed.replace(tzinfo=timezone.utc)
def classify_pressure(
available_kib: Any, total_kib: Any
) -> str:
def classify_pressure(available_kib: Any, total_kib: Any) -> str:
"""Map a MemAvailable/MemTotal pair to ``ok``/``elevated``/``critical``.
``unknown`` when the sample is missing or malformed — the caller must
not treat "we could not read it" as "memory is fine".
"""
if (
isinstance(available_kib, bool)
or not isinstance(available_kib, int)
or available_kib < 0
):
available = _nonneg_int(available_kib)
if available is None:
return "unknown"
fraction: Optional[float] = None
if (
not isinstance(total_kib, bool)
and isinstance(total_kib, int)
and total_kib > 0
total = _nonneg_int(total_kib)
fraction = available / total if total else None
for level, kib_floor, frac_floor in (
("critical", _CRITICAL_AVAILABLE_KIB, _CRITICAL_AVAILABLE_FRACTION),
("elevated", _ELEVATED_AVAILABLE_KIB, _ELEVATED_AVAILABLE_FRACTION),
):
fraction = available_kib / total_kib
if available_kib < _CRITICAL_AVAILABLE_KIB or (
fraction is not None and fraction < _CRITICAL_AVAILABLE_FRACTION
):
return "critical"
if available_kib < _ELEVATED_AVAILABLE_KIB or (
fraction is not None and fraction < _ELEVATED_AVAILABLE_FRACTION
):
return "elevated"
if available < kib_floor or (fraction is not None and fraction < frac_floor):
return level
return "ok"
@@ -124,10 +94,7 @@ def _read_heartbeat(home: Optional[Path]) -> Optional[Dict[str, Any]]:
def _read_sentinel(home: Optional[Path]) -> Optional[Dict[str, Any]]:
try:
from gateway.lifecycle_ledger import (
_read_json,
get_lifecycle_sentinel_path,
)
from gateway.lifecycle_ledger import _read_json, get_lifecycle_sentinel_path
return _read_json(get_lifecycle_sentinel_path(home))
except Exception:
@@ -141,14 +108,10 @@ def collect_memory_status(
) -> Dict[str, Any]:
"""Build the ``memory`` block for ``/api/status``.
``home`` scopes the read to a profile's HERMES_HOME (the status
endpoint's ``?profile=`` handling passes it through); ``None`` means
the active profile. ``now`` is injectable for tests.
Always returns a dict — a gateway that is down, has never written a
heartbeat, or whose files are corrupt yields
``{"pressure": "unknown", ...}`` with whatever fields could still be
recovered. Never raises.
``home`` scopes the read to a profile's HERMES_HOME (``None`` = active
profile); ``now`` is injectable for tests. Always returns a dict and never
raises — a down/never-started gateway or corrupt files yield
``{"pressure": "unknown", ...}`` plus whatever fields could be recovered.
"""
moment = now or datetime.now(timezone.utc)
status: Dict[str, Any] = {
@@ -160,11 +123,9 @@ def collect_memory_status(
"sampled_at": None,
"last_boot_unclean": False,
"last_boot_suspected_oom": False,
# Identity of the CURRENT gateway life (the sentinel's started_at).
# A suspected-OOM restart writes a fresh sentinel, so this changes on
# every boot — the dashboard keys banner dismissal on it so that
# acknowledging one OOM restart does not mute reports of the NEXT
# one (the hourly-restart-loop case is exactly the one that matters).
# Identity of the CURRENT gateway life (sentinel started_at) — changes on
# every boot, so the dashboard can key banner dismissal on it and
# acknowledging one OOM restart does not mute the NEXT one.
"boot_id": None,
}
@@ -179,23 +140,16 @@ def collect_memory_status(
status["swap_used_mb"] = _mb(mem.get("swap_used_kib"))
if sampled_at is not None:
status["sampled_at"] = sampled_at.isoformat()
age_s = (moment - sampled_at).total_seconds()
if 0 <= age_s <= _HEARTBEAT_FRESH_TTL_S:
status["pressure"] = classify_pressure(
mem.get("mem_available_kib"),
mem.get("mem_total_kib"),
)
# else: stale sample — numbers are still reported (they are
# honest about *when* via sampled_at) but pressure stays
# "unknown" so a dead gateway's final gasp cannot render a
# live "critical" banner forever.
# Stale sample: numbers still reported (sampled_at says when) but
# pressure stays "unknown" so a dead gateway's final gasp cannot
# render a live "critical" banner forever.
if 0 <= (moment - sampled_at).total_seconds() <= _HEARTBEAT_FRESH_TTL_S:
status["pressure"] = classify_pressure(mem.get("mem_available_kib"), mem.get("mem_total_kib"))
sentinel = _read_sentinel(home)
if sentinel:
status["last_boot_unclean"] = bool(sentinel.get("prior_unclean_exit"))
status["last_boot_suspected_oom"] = bool(
sentinel.get("prior_suspected_oom")
)
status["last_boot_suspected_oom"] = bool(sentinel.get("prior_suspected_oom"))
started_at = sentinel.get("started_at")
if isinstance(started_at, str) and started_at:
status["boot_id"] = started_at
gateway/message_timestamps.py
+50 -85
View File
@@ -21,55 +21,71 @@ _HUMAN_TIMESTAMP_RE = re.compile(
)
# Older gateway format: [2026-04-13T17:02:06+0200] or [+02:00]
_ISO_TIMESTAMP_RE = re.compile(
r"^\[(?P<iso>\d{4}-\d{2}-\d{2}T[^\]]+)\]\s*"
)
_ISO_TIMESTAMP_RE = re.compile(r"^\[(?P<iso>\d{4}-\d{2}-\d{2}T[^\]]+)\]\s*")
def _localize(dt: datetime, tz) -> float:
"""Epoch for ``dt``; naive values take ``tz`` if given, else the local zone."""
if dt.tzinfo is None:
dt = dt.replace(tzinfo=tz) if tz is not None else dt.astimezone()
return float(dt.timestamp())
def _parse_iso(text: str, tz=None) -> Optional[float]:
"""Parse an ISO-8601 string (incl. ``+0200`` offsets fromisoformat rejects)."""
try:
dt = datetime.fromisoformat(text)
except (TypeError, ValueError):
try:
dt = datetime.strptime(text, "%Y-%m-%dT%H:%M:%S%z")
except (TypeError, ValueError):
return None
return _localize(dt, tz)
def _parse_timestamp_match(match: re.Match, tz=None) -> Optional[float]:
if match.groupdict().get("iso"):
return _parse_iso(match.group("iso"), tz)
try:
dt = datetime.strptime(f"{match.group('date')} {match.group('time')}", "%Y-%m-%d %H:%M:%S")
except ValueError:
return None
return _localize(dt, tz)
def _match_timestamp_prefix(text: str) -> Optional[re.Match]:
return _HUMAN_TIMESTAMP_RE.match(text) or _ISO_TIMESTAMP_RE.match(text)
def coerce_message_timestamp(ts_value: Any, tz=None) -> Optional[float]:
"""Coerce a timestamp-like value to Unix epoch seconds.
Accepts Unix epoch numbers, datetime objects, ISO strings, and the gateway's
bracketed human-readable timestamp format. Returns ``None`` when the value
cannot be interpreted.
Accepts epoch numbers, datetime objects, ISO strings, and the gateway's
bracketed human-readable format. Returns ``None`` when uninterpretable.
"""
if ts_value is None:
return None
if isinstance(ts_value, (int, float)):
return float(ts_value)
if hasattr(ts_value, "timestamp"):
try:
return float(ts_value.timestamp())
except Exception:
return None
if isinstance(ts_value, str):
text = ts_value.strip()
if not text:
return None
parsed = _parse_timestamp_prefix(text, tz=tz)
if parsed is not None:
return parsed
match = _match_timestamp_prefix(text)
if match is not None:
parsed = _parse_timestamp_match(match, tz=tz)
if parsed is not None:
return parsed
try:
return float(text)
except (TypeError, ValueError):
pass
try:
dt = datetime.fromisoformat(text)
except (TypeError, ValueError):
try:
dt = datetime.strptime(text, "%Y-%m-%dT%H:%M:%S%z")
except (TypeError, ValueError):
return None
if dt.tzinfo is None:
if tz is not None:
dt = dt.replace(tzinfo=tz)
else:
dt = dt.astimezone()
return float(dt.timestamp())
return _parse_iso(text, tz)
return None
@@ -78,89 +94,38 @@ def format_message_timestamp(ts_value: Any, tz=None) -> str:
epoch = coerce_message_timestamp(ts_value, tz=tz)
if epoch is None:
return ""
if tz is not None:
dt = datetime.fromtimestamp(epoch, tz=tz)
else:
dt = datetime.fromtimestamp(epoch).astimezone()
dt = datetime.fromtimestamp(epoch, tz=tz) if tz is not None else datetime.fromtimestamp(epoch).astimezone()
return "[" + dt.strftime("%a %Y-%m-%d %H:%M:%S %Z") + "]"
def strip_leading_message_timestamps(content: str, tz=None) -> Tuple[str, Optional[float]]:
"""Strip one or more leading gateway timestamp prefixes from ``content``.
Returns ``(clean_content, embedded_epoch)``. If multiple timestamp prefixes
are present, the timestamp closest to the actual message text wins. That
preserves the original platform-send time for legacy contaminated rows like
``[processing time] [platform time] [sender] message``.
Returns ``(clean_content, embedded_epoch)``. With multiple prefixes the one
closest to the message text wins, preserving the original platform-send
time for legacy contaminated rows like ``[processing time] [platform time] [sender] message``.
"""
if not isinstance(content, str) or not content:
return content, None
text = content
embedded_epoch: Optional[float] = None
while True:
match = _HUMAN_TIMESTAMP_RE.match(text) or _ISO_TIMESTAMP_RE.match(text)
if not match:
break
while (match := _match_timestamp_prefix(text)) is not None:
parsed = _parse_timestamp_match(match, tz=tz)
if parsed is not None:
embedded_epoch = parsed
text = text[match.end():]
return text, embedded_epoch
def render_user_content_with_timestamp(content: str, ts_value: Any = None, tz=None) -> str:
"""Render a user message for LLM context with exactly one timestamp prefix.
Existing leading timestamp prefixes are removed first. If such a prefix was
present, its parsed time wins over ``ts_value``; otherwise ``ts_value`` is
formatted and prepended. If no timestamp is available, the cleaned content is
returned unchanged.
An existing leading prefix is stripped and its parsed time wins over
``ts_value``. If no timestamp is available the cleaned content is returned.
"""
clean_content, embedded_epoch = strip_leading_message_timestamps(content, tz=tz)
effective_ts = embedded_epoch if embedded_epoch is not None else ts_value
prefix = format_message_timestamp(effective_ts, tz=tz)
if not prefix:
return clean_content
if clean_content:
return f"{prefix} {clean_content}"
return prefix
def _parse_timestamp_prefix(text: str, tz=None) -> Optional[float]:
match = _HUMAN_TIMESTAMP_RE.match(text) or _ISO_TIMESTAMP_RE.match(text)
if not match:
return None
return _parse_timestamp_match(match, tz=tz)
def _parse_timestamp_match(match: re.Match, tz=None) -> Optional[float]:
if "iso" in match.groupdict() and match.group("iso"):
iso_text = match.group("iso")
try:
dt = datetime.fromisoformat(iso_text)
except ValueError:
try:
dt = datetime.strptime(iso_text, "%Y-%m-%dT%H:%M:%S%z")
except ValueError:
return None
if dt.tzinfo is None:
if tz is not None:
dt = dt.replace(tzinfo=tz)
else:
dt = dt.astimezone()
return float(dt.timestamp())
date_part = match.group("date")
time_part = match.group("time")
try:
dt = datetime.strptime(f"{date_part} {time_part}", "%Y-%m-%d %H:%M:%S")
except ValueError:
return None
if tz is not None:
dt = dt.replace(tzinfo=tz)
else:
dt = dt.astimezone()
return float(dt.timestamp())
return f"{prefix} {clean_content}" if clean_content else prefix
gateway/platform_registry.py
+193 -367
View File
@@ -1,30 +1,23 @@
"""
Platform Adapter Registry
Allows platform adapters (built-in and plugin) to self-register so the gateway
can discover and instantiate them without hardcoded if/elif chains.
Platform adapters (built-in and plugin) self-register here so the gateway can
discover and instantiate them without hardcoded if/elif chains. Plugins
register via ``PluginContext.register_platform()``; ``GatewayRunner
._create_adapter()`` consults the registry first and falls back to the legacy
built-in path.
Built-in adapters continue to use the existing if/elif in _instantiate_adapter()
for now. Plugin adapters register here via PluginContext.register_platform()
and are looked up first -- if nothing is found the gateway falls through to
the legacy code path. GatewayRunner._create_adapter() wraps both paths and
binds every successful adapter to its runner.
Usage (plugin side):
from gateway.platform_registry import platform_registry, PlatformEntry
Usage (plugin side)::
platform_registry.register(PlatformEntry(
name="irc",
label="IRC",
name="irc", label="IRC",
adapter_factory=lambda cfg: IRCAdapter(cfg),
check_fn=check_requirements,
validate_config=lambda cfg: bool(cfg.extra.get("server")),
required_env=["IRC_SERVER"],
install_hint="pip install irc",
required_env=["IRC_SERVER"], install_hint="pip install irc",
))
Usage (gateway side):
Usage (gateway side)::
adapter = platform_registry.create_adapter("irc", platform_config)
"""
@@ -39,6 +32,9 @@ from hermes_constants import hermes_home_key
logger = logging.getLogger(__name__)
_LoadKey = tuple[Optional[str], str]
_Loader = Callable[[], None]
def _plugin_scope_from_callable(callback: Callable) -> Optional[str]:
"""Infer a plugin profile from code registered outside PluginContext."""
@@ -64,169 +60,113 @@ def _caller_plugin_scope() -> Optional[str]:
class PlatformEntry:
"""Metadata and factory for a single platform adapter."""
# Identifier used in config.yaml (e.g. "irc", "viber").
# Identifier used in config.yaml (e.g. "irc") and human-readable label.
name: str
# Human-readable label (e.g. "IRC", "Viber").
label: str
# Factory callable: receives a PlatformConfig, returns an adapter instance.
# Using a factory instead of a bare class lets plugins do custom init
# (e.g. passing extra kwargs, wrapping in try/except).
# Receives a PlatformConfig, returns an adapter instance. A factory (not a
# bare class) lets plugins do custom init / wrapping.
adapter_factory: Callable[[Any], Any]
# PASSIVE dependency probe: returns True when the platform's dependencies
# are available RIGHT NOW. Must be side-effect free — it is called from
# status displays (``hermes setup``, ``hermes status``, the dashboard
# readiness probe) and the config enablement pass, none of which may
# trigger a pip install. Put install logic in ``ensure_deps_fn`` instead.
# PASSIVE dependency probe: True when deps are importable RIGHT NOW. Must
# be side-effect free -- it runs from status displays (`hermes setup`,
# `hermes status`, dashboard readiness) and the config enablement pass,
# none of which may pip-install. Install logic belongs in ensure_deps_fn.
check_fn: Callable[[], bool]
# Optional: given a PlatformConfig, is it properly configured?
# If None, the registry skips config validation and lets the adapter
# fail at connect() time with a descriptive error.
# Optional config check. None = skip and let connect() fail descriptively.
validate_config: Optional[Callable[[Any], bool]] = None
# ACTIVE dependency installer: make the platform's dependencies available,
# installing them (pip / lazy_deps) if needed. Returns True once deps are
# importable, False if they could not be installed. Called by
# ``create_adapter()`` when ``check_fn`` returns False — i.e. exactly at
# the moment the gateway is about to bring the platform up and the user
# has it enabled/configured. None = no auto-install; a False ``check_fn``
# is then a hard block (correct for platforms with no optional deps).
#
# Why two fields (#79812): when the ACTIVE installer was registered as
# ``check_fn``, every status display pip-installed SDKs as a side effect
# (desktop boot-loop at 94%, see gateway/config.py enablement comments);
# when the PASSIVE probe was registered instead, ``create_adapter()``
# returned None before ``connect()`` could lazy-install, so the deps
# never installed at all (Teams deadlock). Splitting the two roles makes
# both call sites correct by construction.
# ACTIVE dependency installer: make deps importable (pip / lazy_deps),
# returning True on success. Called by ``create_adapter()`` only when
# ``check_fn`` is False -- the one moment the user has the platform
# enabled+configured and the gateway is about to connect it. None = a
# False ``check_fn`` is a hard block (correct for platforms without
# optional deps). The passive/active split exists because a single field
# either pip-installed from every status display or never installed at all.
ensure_deps_fn: Optional[Callable[[], bool]] = None
# Optional: given a PlatformConfig, is the platform connected/enabled?
# Used by ``GatewayConfig.get_connected_platforms()`` and setup UI status.
# If None, falls back to ``validate_config`` or ``check_fn``.
# Is the platform connected/enabled for this PlatformConfig? Used by
# ``GatewayConfig.get_connected_platforms()`` and setup UI status; None
# falls back to ``validate_config`` or ``check_fn``.
is_connected: Optional[Callable[[Any], bool]] = None
# Env vars this platform needs (for ``hermes setup`` display).
# Env vars this platform needs (``hermes setup`` display) and the hint
# shown when check_fn returns False.
required_env: list = field(default_factory=list)
# Hint shown when check_fn returns False.
install_hint: str = ""
# Optional setup function for interactive configuration.
# Signature: () -> None (prompts user, saves env vars).
# If None, falls back to _setup_standard_platform (needs token_var + vars)
# or a generic "set these env vars" display.
# Interactive setup ``() -> None``. None falls back to
# _setup_standard_platform (needs token_var + vars) or a generic env display.
setup_fn: Optional[Callable[[], None]] = None
# "builtin" or "plugin"
# "builtin" or "plugin"; plugin_name is the owning manifest (empty for
# built-ins) so ``hermes gateway setup`` can auto-enable the plugin.
source: str = "plugin"
# Name of the plugin manifest that registered this entry (empty for
# built-ins). Used by ``hermes gateway setup`` to auto-enable the
# owning plugin when the user configures its platform.
plugin_name: str = ""
# ── Auth env var names (for _is_user_authorized integration) ──
# E.g. "IRC_ALLOWED_USERS" — checked for comma-separated user IDs.
# Auth env var names for _is_user_authorized: comma-separated allowed user
# IDs, and a truthy "allow everyone" switch.
allowed_users_env: str = ""
# E.g. "IRC_ALLOW_ALL_USERS" — if truthy, all users authorized.
allow_all_env: str = ""
# ── Message limits ──
# Max message length for smart-chunking. 0 = no limit.
# Max message length for smart-chunking; 0 = no limit.
max_message_length: int = 0
# ── Privacy ──
# If True, session descriptions redact PII (phone numbers, etc.)
# If True, session descriptions redact PII (phone numbers, etc.).
pii_safe: bool = False
# ── Display ──
# Emoji for CLI/gateway display (e.g. "💬")
# Emoji for CLI/gateway display.
emoji: str = "🔌"
# Whether this platform should appear in _UPDATE_ALLOWED_PLATFORMS
# (allows /update command from this platform).
# Whether /update may be issued from this platform (_UPDATE_ALLOWED_PLATFORMS).
allow_update_command: bool = True
# ── LLM guidance ──
# Platform hint injected into the system prompt (e.g. "You are on IRC.
# Do not use markdown."). Empty string = no hint.
# Platform hint injected into the system prompt; empty = none.
platform_hint: str = ""
# ── Env-driven auto-configuration ──
# Optional: read env vars, return a dict of ``PlatformConfig.extra`` fields
# to seed when the platform is auto-enabled. Called during
# ``_apply_env_overrides`` BEFORE the adapter is constructed, so
# ``gateway status`` etc. can reflect env-only configuration without
# instantiating the adapter. Return ``None`` (or an empty dict) to skip.
# Signature: () -> Optional[dict[str, Any]]
# Env-driven auto-configuration ``() -> Optional[dict]``: returns
# ``PlatformConfig.extra`` fields to seed when the platform is auto-enabled.
# Runs during ``_apply_env_overrides`` BEFORE the adapter is constructed so
# ``gateway status`` reflects env-only config. None/empty dict = skip.
env_enablement_fn: Optional[Callable[[], Optional[dict]]] = None
# ── YAML→env config bridge ──
# Optional: translate this platform's ``config.yaml`` keys into env vars
# and/or seed ``PlatformConfig.extra`` directly. Lets a plugin own its
# YAML config translation instead of forcing core ``gateway/config.py``
# to know every platform's schema.
#
# Signature: (yaml_cfg: dict, platform_cfg: dict) -> Optional[dict]
# Called from ``load_gateway_config()`` after the generic shared-key loop
# and before ``_apply_env_overrides``. Mutating ``os.environ`` is allowed
# (use ``not os.getenv(...)`` guards to preserve env > YAML precedence);
# any returned dict is merged into ``PlatformConfig.extra``. Exceptions
# are caught and logged at debug level.
# See website/docs/developer-guide/adding-platform-adapters.md for the
# full contract and a worked example.
# YAML->env bridge ``(yaml_cfg, platform_cfg) -> Optional[dict]``: lets a
# plugin own its config.yaml translation instead of core gateway/config.py.
# Called from ``load_gateway_config()`` after the generic shared-key loop and
# before ``_apply_env_overrides``. May mutate ``os.environ`` (guard with
# ``not os.getenv(...)`` to keep env > YAML precedence); the returned dict is
# merged into ``PlatformConfig.extra``. Exceptions are logged at debug.
# Full contract: website/docs/developer-guide/adding-platform-adapters.md.
apply_yaml_config_fn: Optional[Callable[[dict, dict], Optional[dict]]] = None
# Optional: home-channel env var name for cron/notification delivery
# (e.g. ``"IRC_HOME_CHANNEL"``). When set, ``cron.scheduler`` treats this
# platform as a valid ``deliver=<name>`` target and reads the env var to
# resolve the default chat/room ID. Empty = no cron home-channel support.
# Home-channel env var (e.g. "IRC_HOME_CHANNEL"). When set, cron.scheduler
# accepts ``deliver=<name>`` and reads it for the default chat/room ID.
cron_deliver_env_var: str = ""
# ── Target parsing ──
# Optional: callable that parses a raw target string for this platform into
# a (chat_id, thread_id) tuple, or None if the string is not a recognized
# explicit target. Invoked by ``tools/send_message_tool._parse_target_ref``
# before channel-directory fallback so plugin platforms can declare their
# own native target syntax (e.g. ``fmsg:@alice@example.com``) without
# hard-casing in Hermes core.
#
# Signature:
# (target_ref: str) -> Optional[tuple[str, Optional[str]]]
#
# If the callable returns None the target proceeds to channel-directory
# resolution. No opaque fallback is applied.
# Target parsing ``(target_ref) -> Optional[(chat_id, thread_id)]``, run by
# ``tools/send_message_tool._parse_target_ref`` before channel-directory
# fallback so plugins can declare native target syntax
# (e.g. ``fmsg:@alice@example.com``). None result = continue to directory
# resolution; no opaque fallback is applied.
parse_target_ref_fn: Optional[Callable[[str], Optional[tuple[str, Optional[str]]]]] = None
# Optional validation applied after parsing/normalization or
# channel-directory resolution. Return True to accept, False to reject, or
# a non-empty string to reject with that diagnostic.
# Validation after parsing/normalization or directory resolution: True to
# accept, False to reject, or a non-empty string to reject with diagnostic.
validate_target_ref_fn: Optional[Callable[[str], bool | str]] = None
# Optional whole-request handler for custom platform delivery. Receives
# (args, normalized_chat_id, platform_name, pconfig) and may be sync/async.
# Prefer standalone_sender_fn when the standard send contract is enough.
# Whole-request delivery handler ``(args, normalized_chat_id, platform_name,
# pconfig)``, sync or async. Prefer standalone_sender_fn when the standard
# send contract suffices.
send_message_handler: Optional[Callable[[dict, str, str, Any], Any]] = None
# ── Standalone (out-of-process) sending ──
# Optional: async coroutine that delivers a message without a live
# gateway adapter. Called by ``tools/send_message_tool._send_via_adapter``
# when ``cron`` runs in a separate process from the gateway and the
# in-process adapter weakref is therefore ``None``.
#
# Signature:
# Out-of-process sender used by ``_send_via_adapter`` when cron runs apart
# from the gateway and the in-process adapter weakref is None::
# async (pconfig, chat_id, message, *, thread_id=None,
# media_files=None, force_document=False) -> dict
#
# Returns ``{"success": True, "message_id": ...}`` on success or
# ``{"error": str}`` on failure. Plugin authors typically open an
# ephemeral connection / acquire a fresh OAuth token, send, and close.
# Without this hook, plugin platforms cannot serve as cron ``deliver=``
# targets when the gateway is not co-resident with the cron process.
# Returns ``{"success": True, "message_id": ...}`` or ``{"error": str}``.
# Without it, plugin platforms cannot be cron ``deliver=`` targets when the
# gateway is not co-resident.
standalone_sender_fn: Optional[Callable[..., Awaitable[dict]]] = None
@@ -239,139 +179,107 @@ class PlatformRegistry:
def __init__(self) -> None:
self._lock = threading.RLock()
# Process-global registrations (for example the built-in relay).
# Process-global registrations (e.g. the built-in relay).
self._entries: dict[str, PlatformEntry] = {}
# Plugin adapters are isolated per resolved HERMES_HOME and overlay the
# process-global entries for lookups in that profile's runtime scope.
self._scoped_entries: dict[str, dict[str, PlatformEntry]] = {}
# Deferred platform loaders: name -> zero-arg callable that imports the
# owning plugin module (which calls register() and populates _entries).
#
# Why this exists: platform adapter modules import heavy, platform-
# specific SDKs at module level (lark_oapi, microsoft_teams, discord.py,
# slack_bolt, ...). Eagerly loading all ~20 bundled platform plugins at
# plugin-discovery time added several seconds to *every* `hermes`
# invocation -- including plain `hermes chat`, which never touches any
# gateway platform. Discovery now registers a cheap deferred loader per
# platform; the real module is imported only when a registry lookup
# actually asks for that platform (gateway start, cron delivery,
# `hermes setup`/`gateway status`, send_message).
self._deferred: dict[str, Callable[[], None]] = {}
self._scoped_deferred: dict[str, dict[str, Callable[[], None]]] = {}
self._inflight: dict[tuple[Optional[str], str], threading.Event] = {}
self._inflight_loaders: dict[
tuple[Optional[str], str], Callable[[], None]
] = {}
self._inflight_owners: dict[tuple[Optional[str], str], int] = {}
self._cancelled_inflight: set[tuple[Optional[str], str]] = set()
# Deferred loaders: name -> zero-arg callable that imports the owning
# plugin module (which calls register()). Adapter modules import heavy
# SDKs at module level; eagerly loading ~20 bundled platforms added
# seconds to every `hermes` invocation, so the real import happens only
# when a lookup actually asks for that platform.
self._deferred: dict[str, _Loader] = {}
self._scoped_deferred: dict[str, dict[str, _Loader]] = {}
self._inflight: dict[_LoadKey, threading.Event] = {}
self._inflight_loaders: dict[_LoadKey, _Loader] = {}
self._inflight_owners: dict[_LoadKey, int] = {}
self._cancelled_inflight: set[_LoadKey] = set()
# A failed loader is no longer discoverable, but its identity remains
# until ownership teardown can CAS-restore the displaced predecessor.
self._consumed_loaders: dict[
tuple[Optional[str], str], Callable[[], None]
] = {}
self._consumed_loaders: dict[_LoadKey, _Loader] = {}
@staticmethod
def current_scope_key() -> str:
return hermes_home_key()
def _scope_maps(
self,
scope: Optional[str],
*,
create: bool = False,
) -> tuple[dict[str, PlatformEntry], dict[str, Callable[[], None]]]:
self, scope: Optional[str], *, create: bool = False
) -> tuple[dict[str, PlatformEntry], dict[str, _Loader]]:
if scope is None:
return self._entries, self._deferred
if create:
return (
self._scoped_entries.setdefault(scope, {}),
self._scoped_deferred.setdefault(scope, {}),
)
return (
self._scoped_entries.get(scope, {}),
self._scoped_deferred.get(scope, {}),
)
return self._scoped_entries.setdefault(scope, {}), self._scoped_deferred.setdefault(scope, {})
return self._scoped_entries.get(scope, {}), self._scoped_deferred.get(scope, {})
def _registration_state(
self, scope: Optional[str], name: str, *, create: bool = False
) -> tuple[Optional[PlatformEntry], Optional[_Loader]]:
"""(entry, loader) for *name*; the loader falls back to in-flight, then consumed."""
entries, deferred = self._scope_maps(scope, create=create)
entry = entries.get(name)
loader = deferred.get(name)
if entry is None and loader is None:
loader = self._inflight_loaders.get((scope, name))
if entry is None and loader is None:
loader = self._consumed_loaders.get((scope, name))
return entry, loader
def _prune_scope(self, scope: Optional[str]) -> None:
if scope is None:
return
if not self._scoped_entries.get(scope):
self._scoped_entries.pop(scope, None)
if not self._scoped_deferred.get(scope):
self._scoped_deferred.pop(scope, None)
# -- deferred loading ----------------------------------------------------
def register_deferred(
self,
name: str,
loader: Callable[[], None],
*,
scope: Optional[str] = None,
) -> None:
def register_deferred(self, name: str, loader: _Loader, *, scope: Optional[str] = None) -> None:
"""Register a lazy loader for a platform that hasn't been imported yet.
*loader* is a zero-arg callable that imports the owning plugin module,
which is expected to call :meth:`register` with the real entry for
*name*. The loader runs at most once, the first time *name* is looked
up (or when the full entry list is materialized). A real entry that is
registered directly (e.g. a built-in) takes precedence -- the deferred
loader is then dropped.
*loader* imports the owning plugin module, which must call
:meth:`register` for *name*. It runs at most once, on first lookup
(or full materialization). A concrete registration takes precedence
and drops the loader.
"""
with self._lock:
entries, deferred = self._scope_maps(scope, create=True)
self._consumed_loaders.pop((scope, name), None)
if name in entries:
# Already concretely registered; no need to defer.
return
deferred[name] = loader
if name not in entries:
deferred[name] = loader
def snapshot_registration(
self,
name: str,
*,
scope: Optional[str] = None,
) -> tuple[Optional[PlatformEntry], Optional[Callable[[], None]]]:
"""Return the concrete and deferred state for *name* without resolving it.
self, name: str, *, scope: Optional[str] = None
) -> tuple[Optional[PlatformEntry], Optional[_Loader]]:
"""Concrete and deferred state for *name* without resolving it.
This host-facing snapshot lets the plugin ledger restore a deferred
platform loader that a concrete registration displaced, without
importing the displaced adapter as a side effect of taking the
snapshot.
Lets the plugin ledger restore a deferred loader displaced by a concrete
registration without importing the displaced adapter as a side effect.
"""
with self._lock:
entries, deferred = self._scope_maps(scope)
loader = deferred.get(name)
if entries.get(name) is None and loader is None:
loader = self._inflight_loaders.get((scope, name))
if entries.get(name) is None and loader is None:
loader = self._consumed_loaders.get((scope, name))
return entries.get(name), loader
return self._registration_state(scope, name)
def restore_registration(
self,
name: str,
current: tuple[Optional[PlatformEntry], Optional[Callable[[], None]]],
previous: tuple[Optional[PlatformEntry], Optional[Callable[[], None]]],
current: tuple[Optional[PlatformEntry], Optional[_Loader]],
previous: tuple[Optional[PlatformEntry], Optional[_Loader]],
*,
scope: Optional[str] = None,
) -> bool:
"""Restore a platform registration if its full state is still current.
"""Restore a registration if its full state is still *current* (CAS).
The identity checks protect a later registration from being removed
while still allowing an unloaded override to reveal the registration
it displaced. Both concrete entries and deferred loaders are part of
the state because bundled platform plugins load lazily.
Identity checks protect a later registration from removal while letting
an unloaded override reveal what it displaced. Deferred loaders are
part of the state because bundled platform plugins load lazily.
"""
with self._lock:
entries, deferred = self._scope_maps(scope, create=True)
entry = entries.get(name)
loader = deferred.get(name)
load_key = (scope, name)
if entry is None and loader is None:
loader = self._inflight_loaders.get(load_key)
if entry is None and loader is None:
loader = self._consumed_loaders.get(load_key)
current_state = (entry, loader)
is_current = not (
current_state[0] is not current[0]
or current_state[1] is not current[1]
)
if not is_current:
entry, loader = self._registration_state(scope, name, create=True)
if entry is not current[0] or loader is not current[1]:
return False
entries, deferred = self._scope_maps(scope)
load_key = (scope, name)
previous_entry, previous_loader = previous
if previous_entry is None:
entries.pop(name, None)
@@ -384,18 +292,14 @@ class PlatformRegistry:
if load_key in self._inflight:
self._cancelled_inflight.add(load_key)
self._consumed_loaders.pop(load_key, None)
if scope is not None:
if not entries:
self._scoped_entries.pop(scope, None)
if not deferred:
self._scoped_deferred.pop(scope, None)
self._prune_scope(scope)
return True
def _resolve(self, name: str, scope: Optional[str] = None) -> None:
"""Run the deferred loader for *name* if one is pending."""
loader: Optional[Callable[[], None]] = None
loader: Optional[_Loader] = None
event: Optional[threading.Event] = None
load_key: tuple[Optional[str], str]
load_key: _LoadKey
is_loader = False
with self._lock:
active_scope = scope or self.current_scope_key()
@@ -420,43 +324,27 @@ class PlatformRegistry:
is_loader = True
if event is None:
return
if (
not is_loader
and self._inflight_owners.get(load_key) == threading.get_ident()
):
logger.warning(
"Deferred platform '%s' recursively requested while loading",
name,
)
if not is_loader and self._inflight_owners.get(load_key) == threading.get_ident():
logger.warning("Deferred platform '%s' recursively requested while loading", name)
return
if not is_loader:
event.wait()
# Teardown may have restored an older deferred generation while
# cancelling the one we waited for. Resolve that predecessor in
# the same lookup instead of returning a one-shot false negative.
# cancelling the one we waited for; resolve that predecessor now
# instead of returning a one-shot false negative.
self._resolve(name, active_scope)
return
try:
loader()
except Exception as e:
logger.warning(
"Deferred load of platform '%s' failed: %s",
name,
e,
exc_info=True,
)
logger.warning("Deferred load of platform '%s' failed: %s", name, e, exc_info=True)
finally:
with self._lock:
was_cancelled = load_key in self._cancelled_inflight
load_scope, _load_name = load_key
entries, deferred = self._scope_maps(load_scope)
if (
not was_cancelled
and name not in entries
and name not in deferred
):
entries, deferred = self._scope_maps(load_key[0])
if not was_cancelled and name not in entries and name not in deferred:
self._consumed_loaders[load_key] = loader
self._inflight.pop(load_key, None)
self._inflight_loaders.pop(load_key, None)
@@ -466,12 +354,7 @@ class PlatformRegistry:
if was_cancelled:
self._resolve(name, active_scope)
def is_deferred_load_cancelled(
self,
name: str,
*,
scope: Optional[str] = None,
) -> bool:
def is_deferred_load_cancelled(self, name: str, *, scope: Optional[str] = None) -> bool:
"""Return whether ownership teardown cancelled an in-flight loader."""
with self._lock:
return (scope, name) in self._cancelled_inflight
@@ -479,10 +362,10 @@ class PlatformRegistry:
def _resolve_all(self) -> None:
"""Run every pending deferred loader.
Used by the iterate-all accessors (``all_entries``/``plugin_entries``),
which are only called by paths that genuinely need every adapter:
gateway startup, ``hermes setup``/``gateway status``, channel
directory. CLI chat never iterates the full set.
Only the iterate-all accessors (``all_entries``/``plugin_entries``) call
this, from paths that genuinely need every adapter (gateway startup,
``hermes setup``/``gateway status``, channel directory). CLI chat never
iterates the full set.
"""
active_scope = self.current_scope_key()
with self._lock:
@@ -496,22 +379,11 @@ class PlatformRegistry:
global_names.add(name)
# Load outside the registry lock; each name has an in-flight event so
# concurrent readers wait for the same materialization.
for name in sorted(scoped_names):
self._resolve(name, active_scope)
for name in sorted(global_names):
for name in (*sorted(scoped_names), *sorted(global_names)):
self._resolve(name, active_scope)
def register(
self,
entry: PlatformEntry,
*,
scope: Optional[str] = None,
) -> None:
"""Register a platform adapter entry.
If an entry with the same name exists, it is replaced (last writer
wins -- this lets plugins override built-in adapters if desired).
"""
def register(self, entry: PlatformEntry, *, scope: Optional[str] = None) -> None:
"""Register a platform adapter entry (last writer wins on name clash)."""
with self._lock:
if scope is None and entry.source == "plugin":
scope = _caller_plugin_scope()
@@ -523,13 +395,10 @@ class PlatformRegistry:
entries, deferred = self._scope_maps(scope, create=True)
self._consumed_loaders.pop((scope, entry.name), None)
deferred.pop(entry.name, None)
if entry.name in entries:
prev = entries[entry.name]
prev = entries.get(entry.name)
if prev is not None:
logger.info(
"Platform '%s' re-registered (was %s, now %s)",
entry.name,
prev.source,
entry.source,
"Platform '%s' re-registered (was %s, now %s)", entry.name, prev.source, entry.source
)
entries[entry.name] = entry
logger.debug("Registered platform adapter: %s (%s)", entry.name, entry.source)
@@ -543,25 +412,27 @@ class PlatformRegistry:
if inferred_scope is not None or name in entries or name in deferred:
deferred.pop(name, None)
removed = entries.pop(name, None) is not None
if not entries:
self._scoped_entries.pop(active_scope, None)
if not deferred:
self._scoped_deferred.pop(active_scope, None)
self._prune_scope(active_scope)
return removed
self._deferred.pop(name, None)
return self._entries.pop(name, None) is not None
def _load_pending(self, scope: str, name: str) -> bool:
"""True when a lookup of *name* must run/await a deferred loader (lock held)."""
_entries, deferred = self._scope_maps(scope)
return (
name in deferred
or (name not in self._entries and name in self._deferred)
or (scope, name) in self._inflight
or (None, name) in self._inflight
)
def get(self, name: str) -> Optional[PlatformEntry]:
"""Look up a platform entry by name."""
scope = self.current_scope_key()
with self._lock:
entries, deferred = self._scope_maps(scope)
needs_resolve = name not in entries and (
name in deferred
or (name not in self._entries and name in self._deferred)
or (scope, name) in self._inflight
or (None, name) in self._inflight
)
entries, _deferred = self._scope_maps(scope)
needs_resolve = name not in entries and self._load_pending(scope, name)
if needs_resolve:
self._resolve(name, scope)
with self._lock:
@@ -578,53 +449,33 @@ class PlatformRegistry:
def plugin_entries(self) -> list[PlatformEntry]:
"""Return only plugin-registered platform entries."""
self._resolve_all()
return [e for e in self.all_entries() if e.source == "plugin"]
def registered_names(self) -> set[str]:
"""Return concrete and deferred platform names without loading adapters.
"""Concrete and deferred platform names without loading adapters.
Mirrors ``is_registered()``'s scope semantics: names registered under
the current profile scope AND process-global names both count. Plugin
platforms register deferred loaders under a profile scope, so reading
only the global maps would miss every plugin platform.
Same scope semantics as ``is_registered()``: current profile scope AND
process-global names. Plugin platforms register deferred loaders under
a profile scope, so the global maps alone would miss every plugin.
"""
with self._lock:
scope = self.current_scope_key()
entries, deferred = self._scope_maps(scope)
return (
entries.keys()
| deferred.keys()
| self._entries.keys()
| self._deferred.keys()
)
entries, deferred = self._scope_maps(self.current_scope_key())
return entries.keys() | deferred.keys() | self._entries.keys() | self._deferred.keys()
def is_registered(self, name: str) -> bool:
# A deferred (not-yet-imported) platform still counts as registered --
# the loader will materialize it on first real use. This keeps cheap
# membership checks (toolset resolution, webhook deliver-target checks)
# from triggering a heavy import.
# A deferred (not-yet-imported) platform still counts as registered so
# cheap membership checks (toolset resolution, webhook deliver-target
# checks) never trigger a heavy import.
with self._lock:
scope = self.current_scope_key()
entries, deferred = self._scope_maps(scope)
return (
name in entries
or name in deferred
or name in self._entries
or name in self._deferred
or (scope, name) in self._inflight
or (None, name) in self._inflight
)
entries, _deferred = self._scope_maps(scope)
return name in entries or name in self._entries or self._load_pending(scope, name)
def create_adapter(self, name: str, config: Any) -> Optional[Any]:
"""Create an adapter instance for the given platform name.
"""Create an adapter instance for *name*.
Returns None if:
- No entry registered for *name*
- check_fn() returns False and deps can't be installed
(no ensure_deps_fn, or ensure_deps_fn() returned False)
- validate_config() returns False (misconfigured)
- The factory raises an exception
Returns None when no entry exists, deps are missing and cannot be
installed, ``validate_config`` fails, or the factory raises.
"""
entry = self.get(name)
if entry is None:
@@ -634,63 +485,38 @@ class PlatformRegistry:
try:
deps_ok = bool(entry.check_fn())
except Exception as e:
logger.warning(
"Platform '%s' check_fn raised: %s", entry.label, e
)
logger.warning("Platform '%s' check_fn raised: %s", entry.label, e)
if not deps_ok and entry.ensure_deps_fn is not None:
# Deps missing but the platform can install them on demand.
# This is the ONE place the active installer runs in the adapter
# path: the platform is enabled+configured and the gateway is
# about to connect it, so an install is what the user wants
# (#79812 — Teams' installer previously lived behind this very
# gate inside connect(), which could never be reached).
logger.info(
"Platform '%s' dependencies missing — attempting install...",
entry.label,
)
# The ONE place the active installer runs in the adapter path: the
# platform is enabled+configured and about to connect, so an install
# is what the user wants. (An installer inside connect() is never
# reached when check_fn is False -- create_adapter() would have
# already returned None.)
logger.info("Platform '%s' dependencies missing — attempting install...", entry.label)
try:
deps_ok = bool(entry.ensure_deps_fn())
except Exception as e:
logger.warning(
"Platform '%s' dependency install raised: %s",
entry.label,
e,
)
logger.warning("Platform '%s' dependency install raised: %s", entry.label, e)
deps_ok = False
if not deps_ok:
hint = f" ({entry.install_hint})" if entry.install_hint else ""
logger.warning(
"Platform '%s' requirements not met%s",
entry.label,
hint,
)
logger.warning("Platform '%s' requirements not met%s", entry.label, hint)
return None
if entry.validate_config is not None:
try:
if not entry.validate_config(config):
logger.warning(
"Platform '%s' config validation failed",
entry.label,
)
logger.warning("Platform '%s' config validation failed", entry.label)
return None
except Exception as e:
logger.warning(
"Platform '%s' config validation error: %s",
entry.label,
e,
)
logger.warning("Platform '%s' config validation error: %s", entry.label, e)
return None
try:
adapter = entry.adapter_factory(config)
return adapter
return entry.adapter_factory(config)
except Exception as e:
logger.error(
"Failed to create adapter for platform '%s': %s",
entry.label,
e,
exc_info=True,
"Failed to create adapter for platform '%s': %s", entry.label, e, exc_info=True
)
return None
gateway/profile_routing.py
+38 -82
View File
@@ -1,6 +1,6 @@
"""Profile-based routing for the gateway with hierarchical matching.
Allows a single Hermes instance to route specific Discord guilds/channels/threads
Lets a single Hermes instance route specific Discord guilds/channels/threads
to different profiles — each with their own model, tools, memory, and persona.
Matching priority (most specific first):
@@ -9,13 +9,11 @@ Matching priority (most specific first):
3. platform + guild_id (guild/server route) — specificity 2
4. No match → default profile
Parent-chain matching:
For Discord threads and forum posts, ``parent_chat_id`` carries the
direct parent (the channel for a thread, the forum channel for a post).
Routes keyed on a channel match both direct messages and messages in
any thread/post whose parent is that channel.
Parent-chain matching: for Discord threads and forum posts, ``parent_chat_id``
carries the direct parent, so a route keyed on a channel also matches messages
in any thread/post under it.
Configuration (config.yaml):
Configuration (config.yaml)::
gateway:
profile_routes:
@@ -23,19 +21,11 @@ Configuration (config.yaml):
platform: discord
guild_id: "YOUR_GUILD_ID"
profile: server-profile
- name: special-channel
platform: discord
guild_id: "YOUR_GUILD_ID"
chat_id: "YOUR_CHANNEL_ID"
profile: channel-profile
- name: thread-route
platform: discord
chat_id: "YOUR_CHANNEL_ID"
thread_id: "YOUR_THREAD_ID"
profile: thread-profile
- name: owner-whatsapp
platform: whatsapp
chat_id: "15551234567" # phone, JID, or LID — all equivalent
@@ -51,9 +41,8 @@ import logging
logger = logging.getLogger(__name__)
# Baileys and Cloud share phone/JID/LID identity rules. Other platforms keep
# exact string compare so Telegram numeric ids and Discord snowflakes stay
# unchanged.
# Baileys and Cloud share phone/JID/LID identity rules. Other platforms keep
# exact string compare so Telegram numeric ids and Discord snowflakes stay unchanged.
_WHATSAPP_IDENTITY_PLATFORMS = {"whatsapp", "whatsapp_cloud"}
_WHATSAPP_NON_USER_SUFFIXES = ("@g.us", "@broadcast", "@newsletter")
@@ -69,11 +58,10 @@ def _is_whatsapp_non_user_chat(chat_id: Optional[str]) -> bool:
def _whatsapp_user_chat_ids_match(platform: str, left: Optional[str], right: Optional[str]) -> bool:
"""True when two WhatsApp *user* chat_ids refer to the same person.
Reuses :func:`gateway.whatsapp_identity.expand_whatsapp_aliases` so a
bare phone number, a ``@s.whatsapp.net`` JID, and a ``@lid`` LID collapse
to one identity — the same helper session keys and adapter allowlists
already use. Group/broadcast JIDs are excluded: those are chats, not
senders. Returns False for non-WhatsApp platforms (exact match only).
Uses ``expand_whatsapp_aliases`` (same helper as session keys and adapter
allowlists) so a bare number, ``@s.whatsapp.net`` JID and ``@lid`` LID
collapse to one identity. Group/broadcast JIDs are chats, not senders,
and non-WhatsApp platforms always return False (exact match only).
"""
if (platform or "").strip().lower() not in _WHATSAPP_IDENTITY_PLATFORMS:
return False
@@ -84,9 +72,7 @@ def _whatsapp_user_chat_ids_match(platform: str, left: Optional[str], right: Opt
from gateway.whatsapp_identity import expand_whatsapp_aliases
left_aliases = expand_whatsapp_aliases(str(left))
if not left_aliases:
return False
return bool(left_aliases & expand_whatsapp_aliases(str(right)))
return bool(left_aliases and left_aliases & expand_whatsapp_aliases(str(right)))
class ProfileRouteRejected(RuntimeError):
@@ -108,14 +94,7 @@ class ProfileRoute:
@property
def specificity(self) -> int:
"""Higher value = more specific match."""
s = 0
if self.guild_id:
s += 2
if self.chat_id:
s += 4
if self.thread_id:
s += 8
return s
return 2 * bool(self.guild_id) + 4 * bool(self.chat_id) + 8 * bool(self.thread_id)
def matches(
self,
@@ -125,50 +104,36 @@ class ProfileRoute:
thread_id: Optional[str] = None,
parent_chat_id: Optional[str] = None,
) -> bool:
"""Return True if this route matches the given source fields.
"""Return True if every discriminator the route declares holds (AND).
All configured discriminators are matched conjunctively (AND): every
discriminator that the route declares must hold. ``chat_id`` supports
hierarchical matching for Discord forums/threads:
- Direct channel match: chat_id == route.chat_id
- Thread in channel: parent_chat_id == route.chat_id
A route declaring both ``guild_id`` and ``chat_id`` requires both to
match (a chat match alone does not satisfy a guild constraint).
WhatsApp / WhatsApp Cloud ``chat_id`` also matches across user-identity
forms (bare number, JID, LID) after the exact-string check. Exact
matches always win first, so existing configs keep working. Groups
(``@g.us``) and broadcasts stay exact-only.
``chat_id`` matches the channel directly or as the parent of a
thread/forum post; a route with both ``guild_id`` and ``chat_id``
requires both. WhatsApp ``chat_id`` also matches across user-identity
forms (number/JID/LID) after the exact check; groups/broadcasts stay
exact-only.
"""
if not self.enabled:
return False
if self.platform != platform:
if not self.enabled or self.platform != platform:
return False
if self.thread_id and self.thread_id != thread_id:
return False
if self.chat_id and self.chat_id != chat_id and self.chat_id != parent_chat_id:
if not (
_whatsapp_user_chat_ids_match(platform, self.chat_id, chat_id)
or _whatsapp_user_chat_ids_match(platform, self.chat_id, parent_chat_id)
):
return False
if self.guild_id and self.guild_id != guild_id:
if (
self.chat_id
and self.chat_id not in (chat_id, parent_chat_id)
and not _whatsapp_user_chat_ids_match(platform, self.chat_id, chat_id)
and not _whatsapp_user_chat_ids_match(platform, self.chat_id, parent_chat_id)
):
return False
return True
return not (self.guild_id and self.guild_id != guild_id)
def _coerce_route_id(value: Any) -> Optional[str]:
"""Normalize a route discriminator to str for strict equality matching.
PyYAML loads unquoted numeric IDs (Discord snowflakes, Telegram negative
chat ids) as ``int``. Inbound ``SessionSource`` fields are always ``str``
via ``build_source``, so leaving ints here makes ``matches()`` fail silently.
Only ``int`` is coerced (the legitimate YAML-numeric case). ``bool`` is an
``int`` subclass but never a valid id; floats and other types stringify to
something (``"123.0"``) that can never equal an inbound id — recreating the
silent no-match this exists to fix — so they are passed through with a
load-time warning instead of being silently "fixed" (#86470).
PyYAML loads unquoted numeric IDs as ``int`` while inbound ``SessionSource``
fields are always ``str``, so ints must be coerced or ``matches()`` fails
silently. Only ``int`` (not ``bool``) is coerced; floats and other types
stringify to something (``"123.0"``) that can never match, so they pass
through with a load-time warning instead of being silently "fixed".
"""
if value is None or isinstance(value, str):
return value
@@ -183,10 +148,7 @@ def _coerce_route_id(value: Any) -> Optional[str]:
def parse_profile_routes(raw: Optional[List[Dict[str, Any]]]) -> List[ProfileRoute]:
"""Parse profile_routes from config.yaml into ProfileRoute objects.
Returns routes sorted by specificity (most specific first).
"""
"""Parse profile_routes from config.yaml, sorted most-specific-first."""
if not raw:
return []
routes: List[ProfileRoute] = []
@@ -197,18 +159,13 @@ def parse_profile_routes(raw: Optional[List[Dict[str, Any]]]) -> List[ProfileRou
platform = entry.get("platform", "")
profile = entry.get("profile", "")
if not platform or not profile:
logger.warning(
"Skipping profile route %s: missing platform or profile",
name,
)
logger.warning("Skipping profile route %s: missing platform or profile", name)
continue
# Validate profile name to prevent path traversal. Lazy import avoids a
# Validate profile name to prevent path traversal. Lazy import avoids a
# circular dependency at module load time.
try:
from hermes_cli.profiles import (
normalize_profile_name,
validate_profile_name,
)
from hermes_cli.profiles import normalize_profile_name, validate_profile_name
profile = normalize_profile_name(profile)
validate_profile_name(profile)
except (ValueError, ImportError):
@@ -225,7 +182,6 @@ def parse_profile_routes(raw: Optional[List[Dict[str, Any]]]) -> List[ProfileRou
enabled=entry.get("enabled", True),
)
)
# Sort: most specific first so the first match wins.
routes.sort(key=lambda r: r.specificity, reverse=True)
logger.debug("Loaded %d profile routes (most-specific-first)", len(routes))
return routes
@@ -239,7 +195,7 @@ def match_profile_route(
thread_id: Optional[str] = None,
parent_chat_id: Optional[str] = None,
) -> Optional[ProfileRoute]:
"""Return the best-matching route, or None for no match."""
"""Return the first (most specific) matching route, or None."""
for route in routes:
if route.matches(platform, guild_id=guild_id, chat_id=chat_id, thread_id=thread_id, parent_chat_id=parent_chat_id):
return route
gateway/relay/__init__.py
+250 -501
View File
File diff suppressed because it is too large Load Diff
gateway/relay/transport.py
+36 -65
View File
@@ -2,19 +2,11 @@
The ``RelayAdapter`` (gateway side) delegates all wire I/O to a ``RelayTransport``.
The gateway dials OUT to the connector, so a production transport is a WebSocket
client; in tests it is an in-memory stub (``tests/gateway/relay/stub_connector.py``).
This module defines the protocol surface only — no concrete transport. The
contract has four concerns:
1. Lifecycle: ``connect`` / ``disconnect``.
2. Handshake: ``handshake`` returns the ``CapabilityDescriptor`` the connector
advertises for the platform this adapter fronts.
3. Inbound: ``set_inbound_handler`` registers a callback the transport invokes
with each normalized ``MessageEvent`` the connector delivers.
4. Outbound: ``send_outbound`` carries send/edit/typing actions back to the
connector; ``get_chat_info`` proxies a chat-info lookup; ``send_interrupt``
routes a mid-turn /stop down the socket that owns the session_key.
client (``ws_transport.py``); in tests it is an in-memory stub
(``tests/gateway/relay/stub_connector.py``). This module defines the protocol
surface only: lifecycle (connect/disconnect), handshake (the advertised
``CapabilityDescriptor``), inbound (``set_inbound_handler``) and outbound
(``send_outbound`` / ``get_chat_info`` / ``send_interrupt``).
EXPERIMENTAL: may change without a deprecation cycle until >=2 Class-1 platforms
validate it. See docs/relay-connector-contract.md.
@@ -30,11 +22,10 @@ from gateway.relay.descriptor import CapabilityDescriptor
# Callback the transport invokes for each inbound normalized event.
InboundHandler = Callable[[MessageEvent], Awaitable[None]]
# Callback the transport invokes for each forwarded passthrough request (§5.1).
# The first arg is a PassthroughForward (gateway/relay/ws_transport.py) — typed
# as Any here to keep this protocol module free of a concrete-transport import
# (ws_transport imports FROM this module). The second is an optional bufferId
# (Phase 5 §5.3 buffered flip) the handler acks after durable handoff.
# Callback for each forwarded passthrough request (§5.1): ``(forward, buffer_id)``.
# ``forward`` is a ws_transport.PassthroughForward, typed Any here because
# ws_transport imports FROM this module; ``buffer_id`` (§5.3 buffered flip) is
# acked by the handler after durable handoff.
PassthroughHandler = Callable[[Any, Optional[str]], Awaitable[None]]
@@ -59,14 +50,11 @@ class RelayTransport(Protocol):
...
def set_passthrough_handler(self, handler: "PassthroughHandler") -> None:
"""Register the callback invoked with each forwarded passthrough request.
"""Register the callback for each forwarded passthrough request (§5.1).
Phase 5 §5.1: the passthrough plane (Discord interactions, Twilio, …)
answers the provider's edge ACK at the connector, then forwards the real
request to the gateway over this same outbound socket (a hosted gateway
has no public inbound port). The transport invokes ``handler(forward,
buffer_id)`` for each ``passthrough_forward`` frame. Optional on a
transport (an in-memory stub may not implement it).
The connector answers the provider's edge ACK itself, then forwards the
real request over this same outbound socket (a hosted gateway has no
public inbound port). Optional on a transport (a stub may not implement it).
"""
...
@@ -75,14 +63,10 @@ class RelayTransport(Protocol):
) -> Dict[str, Any]:
"""Carry an outbound action (send/edit/typing) to the connector.
Returns a result dict; for ``op == "send"`` it carries
``success`` and optionally ``message_id`` / ``error``.
``platform`` (Phase 1.5) tags WHICH fronted platform this reply targets,
carried on the OutboundFrame envelope so a gateway fronting N platforms
egresses each reply through the right sender (the transport resolves the
matching advertised botId). Omitted ⇒ the connector falls back to the
session's default platform (single-platform deploys unchanged).
Returns a result dict; for ``op == "send"`` it carries ``success`` and
optionally ``message_id`` / ``error``. ``platform`` tags WHICH fronted
platform this reply targets (the transport resolves the matching botId);
omitted ⇒ the connector uses the session's default platform.
"""
...
@@ -93,24 +77,20 @@ class RelayTransport(Protocol):
async def send_interrupt(self, session_key: str, reason: Optional[str] = None) -> None:
"""Route a mid-turn /stop to the connector for ``session_key``.
The connector forwards it down the socket owned by the gateway
instance running that session (the /stop routing invariant). On the
gateway side this is the OUTBOUND direction; the actual task
cancellation happens when the connector echoes an interrupt inbound
(handled in Task 1.4).
The connector forwards it down the socket owned by the gateway instance
running that session. This is the OUTBOUND direction; the actual task
cancellation happens when the connector echoes an interrupt inbound.
"""
...
async def go_idle(self, timeout_s: float = 10.0) -> bool:
"""Ask the connector to flip this instance to buffered-only (Phase 5 §5.3).
"""Ask the connector to flip this instance to buffered-only (§5.3).
Sends ``going_idle`` and awaits the connector's ``going_idle_ack`` — the
connector-authoritative confirmation that live delivery stopped and inbound
now buffers durably for replay on reconnect (Q-5.3c). Returns True on ack,
False on timeout / not-connected (the caller proceeds to close regardless;
without §5.3 wiring there is simply no buffering). Optional on a transport
(an in-memory stub may not implement it). Emitted as part of the gateway's
EXISTING drain transition — not a new idle path.
Sends ``going_idle`` and awaits the connector-authoritative
``going_idle_ack`` (live delivery stopped; inbound now buffers for replay
on reconnect). Returns True on ack, False on timeout / not-connected —
the caller closes regardless. Optional on a transport. Emitted as part of
the gateway's EXISTING drain transition -- not a new idle path.
"""
...
@@ -119,25 +99,16 @@ class RelayTransport(Protocol):
) -> Dict[str, Any]:
"""Act on a shared-identity capability bound to a session (A2 outbound).
Some platforms hand the connector a credential that acts on the SHARED
bot identity (e.g. a Discord interaction follow-up token, valid ~15min).
Under A2 that credential NEVER reaches the gateway — the connector
stripped it at the edge and bound it in its capability vault keyed by
the session. To use it, the gateway issues a SEMANTIC action against the
session it is already in; it never names or holds a token.
Some platforms hand the connector a credential acting on the SHARED bot
identity (e.g. a Discord interaction follow-up token). Under A2 it NEVER
reaches the gateway: the connector binds it in its capability vault keyed
by session, and the gateway issues a SEMANTIC action against that session.
The action dict carries:
``op`` == ``"follow_up"``
``session_key`` the session whose bound capability to wield
``kind`` the capability kind (e.g. ``"discord.interaction_token"``)
``content`` the message content to send via that capability
``metadata?`` optional extras
The connector resolves the real capability (``resolveOutboundCapability``
on its side), enforces the tenant match (tenant B can never wield tenant
A's capability), and egresses. Returns ``{success, message_id?, error?}``;
``success`` is False when the capability is absent/expired or the tenant
doesn't match — the gateway then has nothing to retry with (by design: a
leaked gateway holds zero capability material).
Action dict: ``op == "follow_up"``, ``session_key``, ``kind`` (e.g.
``"discord.interaction_token"``), ``content``, optional ``metadata``.
The connector resolves the capability, enforces the tenant match, and
egresses. Returns ``{success, message_id?, error?}``; ``success`` is False
when the capability is absent/expired or the tenant mismatches — nothing to
retry with (by design: a leaked gateway holds zero capability material).
"""
...
gateway/relay/ws_transport.py
+331 -643
View File
File diff suppressed because it is too large Load Diff
gateway/response_filters.py
+55 -78
View File
@@ -1,8 +1,7 @@
"""Gateway response filtering helpers.
These helpers operate at the gateway boundary: they decide whether a completed
agent turn should be delivered to the chat, not what should be persisted in the
conversation history.
These decide whether a completed agent turn should be delivered to the chat,
not what should be persisted in conversation history.
"""
from __future__ import annotations
@@ -10,11 +9,8 @@ from __future__ import annotations
import unicodedata
from typing import Any
# Canonical model-emitted control token for intentional silence.
SILENT_REPLY_TOKEN = "NO_REPLY"
# Exact whole-response markers that mean "the agent intentionally chose not to
# reply". Keep this list small and explicit; arbitrary empty output remains an
# Exact whole-response markers meaning "the agent intentionally chose not to
# reply". Keep small and explicit; arbitrary empty output remains an
# error/empty-response path, not silence.
LIVE_GATEWAY_SILENT_MARKERS = frozenset({
"[SILENT]",
@@ -23,23 +19,25 @@ LIVE_GATEWAY_SILENT_MARKERS = frozenset({
"NO REPLY",
})
# Longer than any marker could plausibly be, even with stray punctuation.
_MARKER_LENGTH_CAP = 64
def _canonical_silence_candidate(text: str) -> str:
return " ".join(text.strip().upper().split())
def _strip_edge_silence_punctuation(text: str) -> str:
"""Strip stray edge punctuation without erasing marker structure.
def _is_edge_punctuation(ch: str) -> bool:
# Square brackets stay structural so malformed ``[SILENT`` cannot become ``SILENT``.
return ch not in "[]" and unicodedata.category(ch).startswith("P")
Models sometimes emit ``.NO_REPLY`` or ``*NO_REPLY*`` instead of the exact
marker. Keep square brackets structural so malformed ``[SILENT`` does not
become ``SILENT``.
"""
start = 0
end = len(text)
while start < end and text[start] not in "[]" and unicodedata.category(text[start]).startswith("P"):
def _strip_edge_silence_punctuation(text: str) -> str:
"""Strip stray edge punctuation (``.NO_REPLY``, ``*NO_REPLY*``) without erasing marker structure."""
start, end = 0, len(text)
while start < end and _is_edge_punctuation(text[start]):
start += 1
while end > start and text[end - 1] not in "[]" and unicodedata.category(text[end - 1]).startswith("P"):
while end > start and _is_edge_punctuation(text[end - 1]):
end -= 1
return text[start:end].strip()
@@ -49,42 +47,39 @@ def _canonical_silence_candidates(text: str) -> tuple[str, ...]:
stripped = _strip_edge_silence_punctuation(text.strip())
if stripped == text.strip():
return (exact,)
fallback = _canonical_silence_candidate(stripped)
return (exact, fallback)
return (exact, _canonical_silence_candidate(stripped))
def _short_stripped(text: Any) -> str:
"""Stripped text if it is a non-empty string within the marker cap, else ''."""
if not isinstance(text, str):
return ""
stripped = text.strip()
return stripped if 0 < len(stripped) <= _MARKER_LENGTH_CAP else ""
def is_intentional_silence_response(response: Any) -> bool:
"""Return True only when ``response`` is exactly a silence marker.
"""True only when ``response`` is exactly a silence marker.
Substantive prose that merely mentions ``NO_REPLY`` or ``[SILENT]`` must be
delivered normally. A blank response is also not silence; blank output is
handled by the empty-response failure path.
Prose that merely mentions ``NO_REPLY`` must be delivered normally. A blank
response is not silence either — that is the empty-response failure path.
"""
if not isinstance(response, str):
return False
stripped = response.strip()
if not stripped:
return False
if len(stripped) > 64:
return False
return any(candidate in LIVE_GATEWAY_SILENT_MARKERS for candidate in _canonical_silence_candidates(stripped))
stripped = _short_stripped(response)
return bool(stripped) and any(
c in LIVE_GATEWAY_SILENT_MARKERS for c in _canonical_silence_candidates(stripped)
)
def is_autonomous_silence_response(response: Any) -> bool:
"""Loose silence matcher for autonomous lanes (cron, webhook).
Autonomous lanes instruct the agent to emit ``[SILENT]`` when a tick
produced nothing worth a human's attention, and models reliably bracket
the marker with a short note explaining why they stayed quiet. Unlike
:func:`is_intentional_silence_response` (the interactive-chat rule, which
demands the response be EXACTLY a marker), this suppresses when a marker
is the whole response, sits on its own first or last line, or the
bracketed sentinel opens the response (the documented
``[SILENT] No changes detected`` pattern). A token buried mid-sentence
in a genuine report is still delivered.
Shares :data:`LIVE_GATEWAY_SILENT_MARKERS` so the interactive and
autonomous marker sets can never drift apart.
Autonomous lanes ask for ``[SILENT]`` when a tick produced nothing worth
attention, and models reliably bracket the marker with a short note. Unlike
:func:`is_intentional_silence_response` (interactive rule: EXACTLY a marker),
this suppresses when a marker is the whole response, sits on its own first or
last line, or the bracketed sentinel opens the response (``[SILENT] No
changes detected``). A token buried mid-sentence is still delivered.
Shares :data:`LIVE_GATEWAY_SILENT_MARKERS` so the two sets cannot drift.
"""
if not isinstance(response, str):
return False
@@ -95,53 +90,35 @@ def is_autonomous_silence_response(response: Any) -> bool:
def _is_token(line: str) -> bool:
return _canonical_silence_candidate(line) in LIVE_GATEWAY_SILENT_MARKERS
# Whole response is exactly a token.
if _is_token(stripped):
return True
# Marker on its own first or last line (leading/trailing note on a
# separate line — e.g. "2 deals filtered\n\n[SILENT]").
lines = [ln for ln in stripped.splitlines() if ln.strip()]
if lines and (_is_token(lines[0]) or _is_token(lines[-1])):
return True
# Bracketed sentinel used as a same-line prefix — the documented pattern
# "[SILENT] No changes detected". Restricted to the bracketed form so a
# bare word like "Silent retry succeeded" is NOT swallowed.
if stripped.upper().startswith("[SILENT]"):
return True
return False
# Bracketed form only, so a bare "Silent retry succeeded" is NOT swallowed.
return stripped.upper().startswith("[SILENT]")
def is_intentional_silence_agent_result(agent_result: dict | None, response: Any) -> bool:
"""Silence markers suppress delivery only for successful agent turns."""
if not isinstance(agent_result, dict):
return False
if agent_result.get("failed"):
if not isinstance(agent_result, dict) or agent_result.get("failed"):
return False
return is_intentional_silence_response(response)
def is_partial_silence_marker(text: Any) -> bool:
"""Return True while ``text`` could still resolve to a silence marker.
"""True while streamed ``text`` could still resolve to a silence marker.
The streaming path accumulates the reply delta-by-delta and must decide,
before the whole response is known, whether to show what it has so far.
A buffer whose canonical form is a non-empty *prefix* of a silence marker
(e.g. ``"NO"`` on the way to ``"NO_REPLY"``, or an exact marker that has
not yet been terminated by stream-end) is held back so a raw marker is
never edited onto the screen and then belatedly retracted.
Anything that has already diverged from every marker (ordinary prose) —
and anything longer than the marker cap — returns False so normal
streaming resumes immediately. This is the streaming counterpart to
:func:`is_intentional_silence_response`, sharing the same marker set and
canonicalization so the two never drift.
The streaming path must decide, before the whole response is known, whether
to show its buffer. A buffer whose canonical form is a non-empty *prefix* of
a marker (``"NO"`` on the way to ``"NO_REPLY"``, or an exact marker not yet
terminated by stream-end) is held back so a raw marker is never shown and
then retracted. Anything that has diverged from every marker, or exceeds the
marker cap, returns False so normal streaming resumes. Shares the marker set
and canonicalization with :func:`is_intentional_silence_response`.
"""
if not isinstance(text, str):
return False
stripped = text.strip()
if not stripped or len(stripped) > 64:
return False
for candidate in _canonical_silence_candidates(stripped):
if candidate and any(marker.startswith(candidate) for marker in LIVE_GATEWAY_SILENT_MARKERS):
return True
return False
stripped = _short_stripped(text)
return bool(stripped) and any(
c and any(marker.startswith(c) for marker in LIVE_GATEWAY_SILENT_MARKERS)
for c in _canonical_silence_candidates(stripped)
)
gateway/restart.py
+77 -135
View File
@@ -6,38 +6,32 @@ from collections.abc import Mapping
from hermes_cli.config import DEFAULT_CONFIG
# EX_TEMPFAIL from sysexits.h — used to ask the service manager to restart
# the gateway after a graceful drain/reload path completes.
# EX_TEMPFAIL (sysexits.h): ask the service manager to restart the gateway
# after a graceful drain/reload path completes.
GATEWAY_SERVICE_RESTART_EXIT_CODE = 75
# EX_CONFIG from sysexits.h — fatal configuration error (e.g. token
# collision, no messaging platforms). The s6 finish script translates
# this into exit 125 (permanent failure) so the supervisor stops
# restarting the gateway. See #51228.
# EX_CONFIG (sysexits.h): fatal configuration error (token collision, no
# platforms). The s6 finish script translates this into exit 125 (permanent
# failure) so the supervisor stops restarting the gateway.
GATEWAY_FATAL_CONFIG_EXIT_CODE = 78
def is_global_startup_conflict(error_code: str | None) -> bool:
"""Return True when an adapter's fatal error is a single-writer ownership conflict.
"""True when an adapter's fatal error is a single-writer ownership conflict.
``BasePlatformAdapter._acquire_platform_lock`` emits ``{scope}_lock``
with ``retryable=True`` on purpose: a *mid-run* reconnect must be able to
recover once the live holder exits or a stale record is cleared (#54167).
At startup, though, a live foreign holder is a configuration conflict —
two gateways cannot poll one bot token — so the startup router must not
treat that flag as "transient blip, retry-queue forever". This matches by
error CODE only (the ``{scope}_lock`` / ``lock_conflict`` families every
adapter emits for scoped-lock and identity conflicts), never by message
text.
Adapters emit ``{scope}_lock`` with ``retryable=True`` so a *mid-run*
reconnect can recover once the holder exits or a stale record is cleared.
At startup a live foreign
holder is a configuration conflict (two gateways cannot poll one token), so
the startup router must not treat it as a transient blip. Matches by error
CODE only (``lock_conflict`` / ``*_lock``), never by message text.
"""
code = (error_code or "").strip().lower()
if not code:
return False
return code == "lock_conflict" or code.endswith("_lock")
return bool(code) and (code == "lock_conflict" or code.endswith("_lock"))
# Set by ``hermes gateway run --external-supervisor``. Unlike systemd's
# INVOCATION_ID and launchd's XPC_SERVICE_NAME, this survives wrappers that
# intentionally replace the child environment (for example ``sudo env -i``).
# replace the child environment (e.g. ``sudo env -i``).
EXTERNAL_GATEWAY_SUPERVISOR_ENV = "HERMES_GATEWAY_EXTERNAL_SUPERVISOR"
DEFAULT_GATEWAY_RESTART_DRAIN_TIMEOUT = float(
@@ -48,36 +42,33 @@ DEFAULT_GATEWAY_SIGNAL_INTERRUPT_GRACE_TIMEOUT = float(
)
DEFAULT_GATEWAY_POST_INTERRUPT_GRACE_TIMEOUT = 5.0
# In-band restart (``/restart``, SIGUSR1, self-restart from a child CLI)
# waits for active turns to finish *before* ``stop()`` begins. Distinct
# from ``restart_drain_timeout``, which is the force-interrupt budget
# once ``stop()`` is running (and must stay short under systemd
# TimeoutStopSec). See #77184.
# In-band restart (``/restart``, SIGUSR1, self-restart) waits for active turns
# to finish *before* ``stop()`` begins. Distinct from ``restart_drain_timeout``,
# the force-interrupt budget once ``stop()`` is running (must stay short under
# systemd TimeoutStopSec).
DEFAULT_GATEWAY_RESTART_AFTER_TURN_TIMEOUT = float(
DEFAULT_CONFIG["agent"]["restart_after_turn_timeout"]
)
# Cron-only floor under the ``stop()`` drain. ``restart_drain_timeout``
# defaults to 0 because interrupting a *chat* turn is cheap and recoverable:
# the user is told the gateway is restarting and the session is pre-marked
# resume_pending. An interrupted *cron* run has neither property — nobody is
# waiting on it, it lands in jobs.json as a permanent failure, and a recurring
# job just waits for its next schedule — so a zero-second drain silently
# destroys work. See #82161.
# Cron-only floor under the ``stop()`` drain. ``restart_drain_timeout`` defaults
# to 0 because interrupting a *chat* turn is cheap and recoverable (user is
# told, session pre-marked resume_pending). An interrupted *cron* run has
# neither property — nobody is waiting on it, it lands in jobs.json as a
# permanent failure and a recurring job just waits for its next schedule — so a
# zero-second drain silently destroys work.
DEFAULT_GATEWAY_CRON_DRAIN_TIMEOUT = float(
DEFAULT_CONFIG["agent"]["cron_drain_timeout"]
)
# Seconds of the shutdown watchdog leash held back for the work that still has
# to happen after the drain returns: interrupt agents, kill tool subprocesses,
# mark in-flight jobs interrupted, disconnect adapters. Waiting for cron past
# that point trades a job that is killed *and recorded* for one that is
# SIGKILLed mid-write and stays wedged at ``last_status=running`` forever.
# Seconds of the shutdown watchdog leash held back for post-drain work
# (interrupt agents, kill tool subprocesses, mark jobs interrupted, disconnect
# adapters). Waiting for cron past that trades a job killed *and recorded* for
# one SIGKILLed mid-write and wedged at ``last_status=running`` forever.
CRON_DRAIN_CLEANUP_RESERVE_S = 10.0
# systemd TimeoutStopSec headroom after the stop-path drain budget, and the
# floor used when that budget is still the default immediate (0s) chat drain.
# Keep these in lockstep with generate_systemd_unit() / #94759.
# floor when that budget is still the default immediate (0s) chat drain.
# Keep in lockstep with generate_systemd_unit().
SYSTEMD_STOP_HEADROOM_S = 30.0
SYSTEMD_TIMEOUT_STOP_SEC_FLOOR = 60.0
@@ -87,9 +78,7 @@ def is_gateway_supervisor_process(
) -> bool:
"""Return whether this gateway process is owned by a supervisor."""
env = os.environ if environ is None else environ
if env.get("INVOCATION_ID"):
return True
if env.get("HERMES_S6_SUPERVISED_CHILD"):
if env.get("INVOCATION_ID") or env.get("HERMES_S6_SUPERVISED_CHILD"):
return True
xpc_service = env.get("XPC_SERVICE_NAME", "")
if xpc_service and xpc_service != "0":
@@ -103,17 +92,33 @@ def is_gateway_supervisor_process(
def is_container_restart_context() -> bool:
"""Return whether the gateway is running inside a container for restart
routing purposes (Docker/Podman ⇒ the detached setsid path dies with the
cgroup; exit-75 service restart is the only viable path).
Extracted from the inline probe in the /restart handler so tests can mock
container detection hermetically — a real ``/.dockerenv`` on a
containerized CI runner otherwise flips the routing under the test.
"""
"""Whether the gateway runs in a container (Docker/Podman): the detached
setsid restart path dies with the cgroup, so exit-75 service restart is the
only viable path. Separate function so tests can mock container detection
(a real ``/.dockerenv`` on CI otherwise flips the routing)."""
return os.path.exists("/.dockerenv") or os.path.exists("/run/.containerenv")
def _seconds(value: object, fallback: float = 0.0) -> float:
"""Non-negative float, or ``fallback`` on non-numeric input."""
try:
return max(float(value), 0.0) # type: ignore[arg-type]
except (TypeError, ValueError):
return fallback
def _parse_timeout_keeping_zero(raw: object, default: float) -> float:
"""Parse a timeout where ``0`` is a deliberate disable (must NOT fall
through to ``default``), unlike None / blank / non-numeric input."""
if raw is None or (isinstance(raw, str) and not raw.strip()):
return default
try:
value = float(raw)
except (TypeError, ValueError):
return default
return max(0.0, value)
def parse_restart_drain_timeout(raw: object) -> float:
"""Parse a configured drain timeout, falling back to the shared default."""
try:
@@ -124,38 +129,13 @@ def parse_restart_drain_timeout(raw: object) -> float:
def parse_restart_after_turn_timeout(raw: object) -> float:
"""Parse the after-turn wait cap for in-band restart, falling back to default.
``0`` is a deliberate disable (legacy immediate drain) and must not fall
through to the default — unlike empty/missing input.
"""
if raw is None:
return DEFAULT_GATEWAY_RESTART_AFTER_TURN_TIMEOUT
if isinstance(raw, str) and not raw.strip():
return DEFAULT_GATEWAY_RESTART_AFTER_TURN_TIMEOUT
try:
value = float(raw)
except (TypeError, ValueError):
return DEFAULT_GATEWAY_RESTART_AFTER_TURN_TIMEOUT
return max(0.0, value)
"""Parse the after-turn wait cap for in-band restart (``0`` = legacy immediate drain)."""
return _parse_timeout_keeping_zero(raw, DEFAULT_GATEWAY_RESTART_AFTER_TURN_TIMEOUT)
def parse_cron_drain_timeout(raw: object) -> float:
"""Parse the cron-only drain floor, falling back to the shared default.
``0`` is a deliberate opt-out — cron work is then interrupted on the same
budget as chat work, the pre-#82161 behaviour — and must not fall through
to the default, unlike empty/missing input.
"""
if raw is None:
return DEFAULT_GATEWAY_CRON_DRAIN_TIMEOUT
if isinstance(raw, str) and not raw.strip():
return DEFAULT_GATEWAY_CRON_DRAIN_TIMEOUT
try:
value = float(raw)
except (TypeError, ValueError):
return DEFAULT_GATEWAY_CRON_DRAIN_TIMEOUT
return max(0.0, value)
"""Parse the cron-only drain floor (``0`` = opt out; cron interrupted on the chat budget)."""
return _parse_timeout_keeping_zero(raw, DEFAULT_GATEWAY_CRON_DRAIN_TIMEOUT)
def resolve_cron_drain_budget(
@@ -168,26 +148,14 @@ def resolve_cron_drain_budget(
) -> float:
"""Seconds the shutdown drain may spend waiting on in-flight cron work.
The configured floor is clamped to what this process can actually honour.
The shutdown watchdog hard-exits at ``watchdog_delay`` and the service
manager's ``TimeoutStopSec`` is sized from the full stop budget (drain
vs cron floor + cleanup reserve, plus headroom — see
``resolve_systemd_timeout_stop_sec``), so waiting past that leash
(minus ``cleanup_reserve_s`` for the teardown that follows the drain)
would swap a cleanly-interrupted job for a SIGKILL that leaves it
wedged mid-run — strictly worse than the bug being fixed.
Never returns less than ``drain_timeout``: the cron floor only ever
extends the wait, so an operator who deliberately configured a long
``restart_drain_timeout`` keeps it.
The configured floor is clamped to what this process can honour: the
shutdown watchdog hard-exits at ``watchdog_delay`` (and TimeoutStopSec is
sized from the same budget), so waiting past that leash minus
``cleanup_reserve_s`` swaps a cleanly-interrupted job for a SIGKILL that
leaves it wedged. Never returns less than ``drain_timeout``: the cron floor
only ever extends the wait, so an operator who deliberately configured a
long ``restart_drain_timeout`` keeps it.
"""
def _seconds(value: object, fallback: float = 0.0) -> float:
try:
return max(float(value), 0.0) # type: ignore[arg-type]
except (TypeError, ValueError):
return fallback
drain = _seconds(drain_timeout)
floor = _seconds(cron_drain_timeout)
if floor <= 0.0:
@@ -210,30 +178,17 @@ def resolve_systemd_timeout_stop_sec(
) -> int:
"""Seconds systemd ``TimeoutStopSec`` must cover the full stop budget.
``restart_drain_timeout`` is only the chat-turn interrupt budget (default
0). The stop path may wait longer for in-flight cron work —
``cron_drain_timeout`` plus ``cleanup_reserve_s`` — before it even starts
interrupting. Sizing the unit from drain alone lets systemd SIGKILL an
in-budget drain (#94759).
A zero ``cron_drain_timeout`` is a deliberate opt-out and does not extend
the budget. Non-numeric inputs degrade to 0 rather than raising.
``restart_drain_timeout`` is only the chat-turn interrupt budget (default 0);
the stop path may first wait ``cron_drain_timeout`` + ``cleanup_reserve_s``
for cron work, so sizing from drain alone lets systemd SIGKILL an in-budget
drain. A zero cron timeout is an opt-out and does not extend the budget.
Non-numeric inputs degrade to 0.
"""
def _seconds(value: object) -> float:
try:
return max(float(value), 0.0) # type: ignore[arg-type]
except (TypeError, ValueError):
return 0.0
drain = _seconds(drain_timeout)
cron = _seconds(cron_drain_timeout)
reserve = _seconds(cleanup_reserve_s)
headroom = _seconds(headroom_s)
floor = _seconds(floor_s)
cron_budget = (cron + reserve) if cron > 0.0 else 0.0
cron_budget = (cron + _seconds(cleanup_reserve_s)) if cron > 0.0 else 0.0
stop_budget = max(drain, cron_budget)
return int(max(floor, stop_budget + headroom))
return int(max(_seconds(floor_s), stop_budget + _seconds(headroom_s)))
def resolve_restart_exit_wait_budget(
@@ -244,24 +199,11 @@ def resolve_restart_exit_wait_budget(
) -> float:
"""Seconds a CLI should wait for the gateway PID to exit after SIGUSR1.
In-band restart may defer ``stop()`` until active turns finish
(``after_turn_timeout``) and then spend up to ``drain_timeout`` inside
``stop()``. Callers that fall back to a hard kill on wait expiry must
cover both phases or they reintroduce #77184.
In-band restart may defer ``stop()`` until turns finish (``after_turn_timeout``)
and then spend ``drain_timeout`` inside ``stop()``; callers that hard-kill on
expiry must cover both phases.
"""
try:
drain = max(float(drain_timeout), 0.0)
except (TypeError, ValueError):
drain = 0.0
try:
after_turn = max(float(after_turn_timeout), 0.0)
except (TypeError, ValueError):
after_turn = 0.0
try:
margin = max(float(headroom), 0.0)
except (TypeError, ValueError):
margin = 0.0
return drain + after_turn + margin
return _seconds(drain_timeout) + _seconds(after_turn_timeout) + _seconds(headroom)
def parse_signal_interrupt_grace_timeout(raw: object) -> float:
gateway/restart_loop_guard.py
+41 -98
View File
@@ -1,33 +1,23 @@
"""Auto-resume restart-loop breaker (#30719, defense-3).
"""Auto-resume restart-loop breaker (defense-3).
Defenses 1 and 2 (the ``_HERMES_GATEWAY`` guard on ``hermes gateway
stop|restart`` + ``terminal_tool``, and the cron-creation lifecycle
filter) stop the agent from scheduling its own restart via the cron and
CLI paths. They do NOT cover every SIGTERM source: an agent running a
raw ``terminal("launchctl kickstart -k gui/<uid>/ai.hermes.gateway")``,
an external monitor with a bad trigger, or any other repeated crash can
still drive the supervisor (launchd ``KeepAlive`` / systemd ``Restart=``)
into a tight respawn loop. On each boot the gateway auto-resumes the
restart-interrupted session, whose next turn re-runs the offending
logic — SIGTERM every ~10 seconds until manually broken.
stop|restart`` + ``terminal_tool``, and the cron-creation lifecycle filter)
stop the agent scheduling its own restart via cron/CLI. They do NOT cover
every SIGTERM source (raw ``launchctl kickstart``, a bad external monitor,
any repeated crash): the supervisor respawns, the gateway auto-resumes the
restart-interrupted session, whose next turn re-runs the offending logic.
This module is the last-resort circuit breaker: it records a timestamp
each time the gateway boots with restart-interrupted sessions pending,
keeps the current chain of such boots persisted across processes (each
boot is a fresh process, so in-memory state is useless), and reports the
loop as "tripped" once too many of them chain together. Boots chain
while consecutive gaps stay within ``max_gap_seconds``, so the breaker
sees slow crash cycles (a wedged event loop killed by the liveness
watchdog every ~150s, #81642) exactly as well as the fast ~10s respawn
loop it was originally written for.
When tripped, the caller SKIPS auto-resume for that boot — the gateway
still starts and serves real inbound messages, it just stops replaying
the session that keeps killing it, which breaks the cycle and puts a
human back in the loop.
This module is the last-resort circuit breaker. Each boot with
restart-interrupted sessions pending is timestamped and persisted (each boot
is a fresh process, so in-memory state is useless). Boots CHAIN while
consecutive gaps stay within ``max_gap_seconds``, so slow crash cycles (a
wedged loop killed by the liveness watchdog every ~150s) trip exactly like the
fast ~10s respawn loop. When tripped, the caller SKIPS auto-resume for that
boot — the gateway still serves real inbound messages, it just stops replaying
the session that keeps killing it.
State lives in ``<HERMES_HOME>/gateway/restart_loop.json`` so it is
profile-scoped and survives process death. It is intentionally tiny and
best-effort: any read/write failure fails OPEN (no false trip) because a
State lives in ``<HERMES_HOME>/gateway/restart_loop.json`` (profile-scoped).
Best-effort: any read/write failure fails OPEN (no false trip) because a
broken breaker must never wedge a healthy gateway.
"""
@@ -42,26 +32,20 @@ from hermes_constants import get_hermes_home
logger = logging.getLogger("gateway.run")
# Defaults chosen so a legitimate operator restart (or two) never trips the
# breaker, but the documented ~10s respawn loop does within a few cycles.
# A legitimate operator restart (or two) never trips; a ~10s respawn loop does
# within a few cycles.
DEFAULT_MAX_RESTARTS = 3
DEFAULT_WINDOW_SECONDS = 60
# Longest gap between two consecutive restart-interrupted boots that still
# counts them as the SAME loop (#81642). A fixed ``window_seconds`` prune can
# only see crash cycles faster than the window: a loop whose period exceeds it
# drops its own history on every boot, so the counter never leaves 1 and the
# breaker never trips no matter how long the loop runs. The reported cycle was
# ~150s (wedged event loop -> ~90s liveness watchdog hard-exit -> respawn ->
# auto-resume replays the same session), i.e. structurally invisible to the 60s
# window. Chaining on the inter-boot GAP instead makes the breaker period-
# agnostic: any repeating cycle trips once ``max_restarts`` links accumulate,
# and a single boot followed by real quiet resets the chain.
# Longest gap between consecutive restart-interrupted boots that still counts
# them as the SAME loop. A fixed-window prune only sees cycles faster than the
# window (a slower loop drops its own history every boot and never trips);
# chaining on the inter-boot gap makes the breaker period-agnostic, and a
# single boot followed by real quiet resets the chain.
DEFAULT_MAX_GAP_SECONDS = 300
# Cap the persisted chain so a long-running loop cannot grow the state file
# without bound. Only the newest ``max_restarts`` entries can change a
# verdict; the rest are kept for forensics.
# Cap the persisted chain; only the newest ``max_restarts`` entries can change
# a verdict, the rest are forensics.
_MAX_STORED_BOOTS = 50
@@ -71,10 +55,8 @@ def _state_path():
def _load_boots() -> List[float]:
try:
raw = _state_path().read_text(encoding="utf-8")
data = json.loads(raw)
boots = data.get("boots", [])
return [float(t) for t in boots if isinstance(t, (int, float))]
data = json.loads(_state_path().read_text(encoding="utf-8"))
return [float(t) for t in data.get("boots", []) if isinstance(t, (int, float))]
except (OSError, ValueError, TypeError):
return []
@@ -89,28 +71,24 @@ def _save_boots(boots: List[float]) -> None:
def _chain_gap(window_seconds: int, max_gap_seconds: int) -> float:
"""Effective inter-boot gap that still links two boots into one loop.
Floored by ``window_seconds`` so an operator who widens the window never
ends up with a breaker that is *less* sensitive than they asked for.
"""
"""Inter-boot gap that still links two boots. Floored by ``window_seconds`` so
widening the window never makes the breaker *less* sensitive."""
return float(max(1, window_seconds, max_gap_seconds))
def _chain_ending_at(boots: List[float], ts: float, gap: float) -> List[float]:
"""Return the unbroken chain of boots leading up to ``ts``.
"""Unbroken chain of boots leading up to ``ts`` (oldest first).
Walks backwards from ``ts`` and keeps boots while each successive gap stays
within ``gap``. The first gap that exceeds it ends the chain: everything
older belongs to a previous, already-resolved episode. A chain broken at
the head (nothing recent enough) yields an empty list, which is how a
healthy gateway forgets an old loop.
Walks backwards keeping boots while each successive gap stays within
``gap``; the first wider gap ends the chain (older boots belong to an
already-resolved episode). Nothing recent enough -> empty list, which is how
a healthy gateway forgets an old loop.
"""
chain: List[float] = []
prev = ts
for t in sorted(boots, reverse=True):
if t > ts:
# Clock moved backwards (NTP step, restored state file). Treat the
# Clock moved backwards (NTP step, restored state file): treat the
# future entry as adjacent rather than dropping the whole chain.
chain.append(t)
continue
@@ -128,49 +106,17 @@ def record_restart_interrupted_boot(
now: Optional[float] = None,
max_gap_seconds: int = DEFAULT_MAX_GAP_SECONDS,
) -> List[float]:
"""Record that the gateway just booted with restart-interrupted sessions.
"""Record a restart-interrupted boot; return the pruned chain + now (most recent last).
Drops boots that belong to an earlier, already-broken chain (any gap wider
than ``max_gap_seconds``) and appends the current time. Returns the
pruned+appended list (most recent last). Best-effort — a persistence
failure returns the in-memory list without raising.
Best-effort — a persistence failure returns the in-memory list without raising.
"""
ts = time.time() if now is None else now
gap = _chain_gap(window_seconds, max_gap_seconds)
boots = _chain_ending_at(_load_boots(), ts, gap)
boots = _chain_ending_at(_load_boots(), ts, _chain_gap(window_seconds, max_gap_seconds))
boots.append(ts)
_save_boots(boots[-_MAX_STORED_BOOTS:])
return boots
def is_restart_loop_tripped(
max_restarts: int = DEFAULT_MAX_RESTARTS,
window_seconds: int = DEFAULT_WINDOW_SECONDS,
*,
now: Optional[float] = None,
max_gap_seconds: int = DEFAULT_MAX_GAP_SECONDS,
) -> bool:
"""Return True if the gateway has restarted ``>= max_restarts`` times with
restart-interrupted sessions in one unbroken chain ending at ``now``.
Reads the persisted boot log written by
``record_restart_interrupted_boot`` and counts the boots that still chain
together (consecutive gaps within ``max_gap_seconds``), so the verdict does
not depend on how fast the crash cycle happens to be.
Fails OPEN (returns False) on any error — a broken breaker must never
wedge a healthy gateway.
"""
if max_restarts <= 0:
return False
ts = time.time() if now is None else now
gap = _chain_gap(window_seconds, max_gap_seconds)
try:
recent = _chain_ending_at(_load_boots(), ts, gap)
except Exception: # pragma: no cover — _load_boots already guards
return False
return len(recent) >= max_restarts
def clear() -> None:
"""Remove the persisted boot log (used on clean shutdown / by tests)."""
try:
@@ -186,13 +132,10 @@ def check_and_record(
now: Optional[float] = None,
max_gap_seconds: int = DEFAULT_MAX_GAP_SECONDS,
) -> bool:
"""Record this restart-interrupted boot and report whether the loop is now
tripped.
"""Record this boot and return True when auto-resume should be SKIPPED.
This is the single entry point the gateway calls: it appends the current
boot, then checks whether the (now-updated) chain has reached the
threshold. Returns True when auto-resume should be SKIPPED to break the
loop.
The single entry point the gateway calls: appends the current boot, then
checks whether the updated chain has reached ``max_restarts``.
"""
boots = record_restart_interrupted_boot(
window_seconds, now=now, max_gap_seconds=max_gap_seconds
gateway/rich_sent_store.py
+22 -35
View File
@@ -1,13 +1,10 @@
"""Local index of text we've sent via ``sendRichMessage`` (Bot API 10.1).
Telegram does NOT echo a rich message's content back in ``reply_to_message``
when a user replies to it (verified: ``.text``/``.caption`` empty,
``.api_kwargs`` None). So replies to the launchd briefings / any rich send
arrive with no quotable text and the agent is blind to what was referenced.
Fix: remember ``message_id -> text`` at send time, look it up by
``reply_to_id`` on inbound. This module is the single source of truth for that
index.
(``.text``/``.caption`` empty, ``.api_kwargs`` None), so replies to rich sends
arrive with no quotable text. We remember ``message_id -> text`` at send time
and look it up by ``reply_to_id`` on inbound. This module is the single source
of truth for that index.
Best-effort and dependency-free: every operation swallows errors and degrades
to a no-op / ``None`` so it can never break a send or an inbound message.
@@ -25,17 +22,25 @@ _MAX_TEXT_CHARS = 2000
def _store_path() -> str:
# Resolve via get_hermes_home() so the active profile override is honored.
# get_hermes_home() honors the active profile override.
from hermes_constants import get_hermes_home
home = get_hermes_home()
return os.path.join(str(home), "state", "rich_sent_index.json")
return os.path.join(str(get_hermes_home()), "state", "rich_sent_index.json")
def _key(chat_id, message_id) -> str:
return f"{chat_id}:{message_id}"
def _load(path: str) -> dict:
try:
with open(path, "r", encoding="utf-8") as fh:
data = json.load(fh)
except (FileNotFoundError, ValueError):
return {}
return data if isinstance(data, dict) else {}
def record(chat_id, message_id, text: Optional[str]) -> None:
"""Persist ``text`` for ``(chat_id, message_id)``. No-op on any failure."""
if not text or message_id is None or chat_id is None:
@@ -43,22 +48,11 @@ def record(chat_id, message_id, text: Optional[str]) -> None:
path = _store_path()
try:
os.makedirs(os.path.dirname(path), exist_ok=True)
try:
with open(path, "r", encoding="utf-8") as fh:
data = json.load(fh)
if not isinstance(data, dict):
data = {}
except (FileNotFoundError, ValueError):
data = {}
data[_key(chat_id, message_id)] = {
"t": text[:_MAX_TEXT_CHARS],
"ts": int(time.time()),
}
# Trim oldest by timestamp when over cap.
if len(data) > _MAX_ENTRIES:
for k, _ in sorted(
data.items(), key=lambda kv: kv[1].get("ts", 0)
)[: len(data) - _MAX_ENTRIES]:
data = _load(path)
data[_key(chat_id, message_id)] = {"t": text[:_MAX_TEXT_CHARS], "ts": int(time.time())}
if len(data) > _MAX_ENTRIES: # trim oldest by timestamp
by_age = sorted(data.items(), key=lambda kv: kv[1].get("ts", 0))
for k, _ in by_age[: len(data) - _MAX_ENTRIES]:
data.pop(k, None)
tmp = f"{path}.tmp.{os.getpid()}"
with open(tmp, "w", encoding="utf-8") as fh:
@@ -72,12 +66,5 @@ def lookup(chat_id, message_id) -> Optional[str]:
"""Return stored text for ``(chat_id, message_id)`` or ``None``."""
if message_id is None or chat_id is None:
return None
try:
with open(_store_path(), "r", encoding="utf-8") as fh:
data = json.load(fh)
entry = data.get(_key(chat_id, message_id))
if isinstance(entry, dict):
return entry.get("t") or None
except (FileNotFoundError, ValueError, AttributeError):
return None
return None
entry = _load(_store_path()).get(_key(chat_id, message_id))
return (entry.get("t") or None) if isinstance(entry, dict) else None
gateway/run.py
+3372 -5916
View File
File diff suppressed because it is too large Load Diff
gateway/runtime_footer.py
+56 -92
View File
@@ -1,35 +1,21 @@
"""Gateway runtime-metadata footer.
Renders a compact footer showing runtime state (model, context %, cwd) and
appends it to the FINAL message of an agent turn when enabled. Off by default
"""Gateway runtime-metadata footer (model · context % · cwd), off by default
to keep replies minimal.
Config (``~/.hermes/config.yaml``)::
Config (``display.runtime_footer``, per-platform override under
``display.platforms.<platform>.runtime_footer``; toggled by ``/footer on|off``)::
display:
runtime_footer:
enabled: true # off by default
enabled: true
fields: [model, context_pct, cwd] # order shown; drop any to hide
Available fields:
model — bare model id, vendor prefix dropped (``gpt-5.4``)
context_pct — last-call context occupancy as a percent (``5%``)
latency — wall-clock duration of the turn (``22s``, ``1m05s``)
cwd — home-relative working dir (``~``)
Fields: ``model`` (vendor prefix dropped), ``context_pct`` (last-call
occupancy), ``latency`` (turn wall-clock, opt-in — NOT in the default set so an
unset ``fields`` renders exactly as before), ``cwd`` (home-relative).
``latency`` is opt-in: it is NOT in the default field set, so a footer whose
``fields`` are unset renders exactly as before.
Per-platform overrides live under ``display.platforms.<platform>.runtime_footer``.
Users can toggle the global setting with ``/footer on|off`` from both the CLI
and any gateway platform.
The footer is appended to the final response text in ``gateway/run.py`` right
before returning the response to the adapter send path — so it only lands on
the final message a user sees, not on tool-progress updates or streaming
partials. When streaming is on and the final text has already been delivered
piecemeal, the footer is sent as a separate trailing message via
``send_trailing_footer()``.
``gateway/run.py`` appends the footer to the final response only (never to
tool-progress or streaming partials); when streaming already delivered the text,
it goes out as a trailing message via ``send_trailing_footer()``.
"""
from __future__ import annotations
@@ -56,44 +42,40 @@ def _home_relative_cwd(cwd: str) -> str:
def _model_short(model: Optional[str]) -> str:
"""Drop ``vendor/`` prefix for readability (``openai/gpt-5.4`` → ``gpt-5.4``)."""
if not model:
return ""
return model.rsplit("/", 1)[-1]
"""Drop ``vendor/`` prefix (``openai/gpt-5.4`` → ``gpt-5.4``)."""
return model.rsplit("/", 1)[-1] if model else ""
def _env_cwd() -> str:
try:
from tools.terminal_scope import terminal_env
except ImportError:
return os.environ.get("TERMINAL_CWD", "")
return terminal_env("TERMINAL_CWD", "")
def _apply_footer_section(resolved: dict[str, Any], section: Any) -> None:
if not isinstance(section, dict):
return
if "enabled" in section:
resolved["enabled"] = bool(section.get("enabled"))
if isinstance(section.get("fields"), list) and section["fields"]:
resolved["fields"] = [str(f) for f in section["fields"]]
def resolve_footer_config(
user_config: dict[str, Any] | None,
platform_key: str | None = None,
) -> dict[str, Any]:
"""Resolve effective runtime-footer config for *platform_key*.
Merge order (later wins):
1. Built-in defaults (enabled=False)
2. ``display.runtime_footer``
3. ``display.platforms.<platform_key>.runtime_footer``
"""
"""Resolve effective footer config: defaults (enabled=False) <
``display.runtime_footer`` < ``display.platforms.<platform_key>.runtime_footer``."""
resolved = {"enabled": False, "fields": list(_DEFAULT_FIELDS)}
cfg = (user_config or {}).get("display") or {}
global_cfg = cfg.get("runtime_footer")
if isinstance(global_cfg, dict):
if "enabled" in global_cfg:
resolved["enabled"] = bool(global_cfg.get("enabled"))
if isinstance(global_cfg.get("fields"), list) and global_cfg["fields"]:
resolved["fields"] = [str(f) for f in global_cfg["fields"]]
_apply_footer_section(resolved, cfg.get("runtime_footer"))
if platform_key:
platforms = cfg.get("platforms") or {}
plat_cfg = platforms.get(platform_key)
plat_cfg = (cfg.get("platforms") or {}).get(platform_key)
if isinstance(plat_cfg, dict):
plat_footer = plat_cfg.get("runtime_footer")
if isinstance(plat_footer, dict):
if "enabled" in plat_footer:
resolved["enabled"] = bool(plat_footer.get("enabled"))
if isinstance(plat_footer.get("fields"), list) and plat_footer["fields"]:
resolved["fields"] = [str(f) for f in plat_footer["fields"]]
_apply_footer_section(resolved, plat_cfg.get("runtime_footer"))
return resolved
@@ -108,6 +90,12 @@ def _format_latency(seconds: float) -> str:
return f"{m}m{sec:02d}s"
def _context_pct(context_tokens: int, context_length: Optional[int]) -> str:
if context_length and context_length > 0 and context_tokens >= 0:
return f"{max(0, min(100, round((context_tokens / context_length) * 100)))}%"
return ""
def format_runtime_footer(
*,
model: Optional[str],
@@ -117,40 +105,19 @@ def format_runtime_footer(
turn_seconds: Optional[float] = None,
fields: Iterable[str] = _DEFAULT_FIELDS,
) -> str:
"""Render the footer line, or return "" if no fields have data.
"""Render the footer line, or "" if no fields have data.
Fields are skipped silently when their underlying data is missing — a
partially-populated footer is better than a line with ``?%`` or empty slots.
Fields whose data is missing (and unknown field names) are skipped silently —
a partial footer beats ``?%`` or empty slots.
"""
parts: list[str] = []
for field in fields:
if field == "model":
m = _model_short(model)
if m:
parts.append(m)
elif field == "context_pct":
if context_length and context_length > 0 and context_tokens >= 0:
pct = max(0, min(100, round((context_tokens / context_length) * 100)))
parts.append(f"{pct}%")
elif field == "latency":
# Wall-clock turn duration. Skipped when the caller supplied no
# timing (call sites that don't measure) or the value is negative.
if turn_seconds is not None and turn_seconds >= 0:
parts.append(_format_latency(turn_seconds))
elif field == "cwd":
try:
from tools.terminal_scope import terminal_env as _tenv
except ImportError:
env_cwd = os.environ.get("TERMINAL_CWD", "")
else:
env_cwd = _tenv("TERMINAL_CWD", "")
rel = _home_relative_cwd(cwd or env_cwd)
if rel:
parts.append(rel)
# Unknown field names are silently ignored.
if not parts:
return ""
renderers = {
"model": lambda: _model_short(model),
"context_pct": lambda: _context_pct(context_tokens, context_length),
# Skipped when the caller did not measure (None) or the value is negative.
"latency": lambda: _format_latency(turn_seconds) if turn_seconds is not None and turn_seconds >= 0 else "",
"cwd": lambda: _home_relative_cwd(cwd or _env_cwd()),
}
parts = [value for field in fields if (render := renderers.get(field)) and (value := render())]
return _SEP.join(parts)
@@ -164,15 +131,12 @@ def build_footer_line(
cwd: Optional[str] = None,
turn_seconds: Optional[float] = None,
) -> str:
"""Top-level entry point used by gateway/run.py.
"""Entry point for gateway/run.py: footer text, or "" when disabled / no data.
Callers append it to the final response themselves, preserving a single
blank line of separation.
Returns the footer text (empty string when disabled or no data). Callers
append this to the final response themselves, preserving a single blank
line of separation.
``turn_seconds`` is the wall-clock duration of the agent run, measured by
the caller with ``time.monotonic()``. Callers that don't measure it leave
it ``None`` and the ``latency`` field is skipped.
``turn_seconds`` is the caller-measured (``time.monotonic()``) run duration;
``None`` skips the ``latency`` field.
"""
cfg = resolve_footer_config(user_config, platform_key)
if not cfg.get("enabled"):
gateway/scale_to_zero.py
+86 -142
View File
@@ -1,39 +1,29 @@
"""Scale-to-zero idle detection + dormant-quiesce for the gateway (Phase 0).
"""Scale-to-zero idle detection + dormant-quiesce for the gateway.
This is the gateway-side BEHAVIOUR layer that consumes the relay scale-to-zero
PRIMITIVES (gateway-gateway Phase 5: the buffered-flip, the durable per-instance
buffer, the wakeUrl poke, the reconnect supervisor). It owns the *decision* to go
idle, drives the relay transport's ``go_dormant()`` (D12), and then SUSPENDS the
machine itself through the local Fly Machines API socket. Wake stays platform-side:
autostart-on-wakeUrl (decisions.md Q3=C′).
Gateway-side behaviour layer over the relay scale-to-zero primitives: it owns
the *decision* to go idle, drives the relay transport's ``go_dormant()``, then
SUSPENDS the machine through the local Fly Machines API socket. Wake stays
platform-side (autostart-on-wakeUrl).
Why the gateway self-suspends instead of relying on ``autostop:"suspend"``: Fly
Proxy judges idle exclusively on INBOUND proxied connections — it cannot see an
in-flight agent turn (outbound-only LLM traffic) and there is no way for the app
to signal "not ready to suspend". Mid-2026 the proxy also stopped treating open
OUTBOUND sockets as activity, so the relay WebSocket no longer masks the race:
Fly would suspend a machine mid-job, and could suspend BEFORE ``go_dormant()``
flipped the relay destination (the buffered-event black hole). Owning the suspend
call closes both: it only ever fires after the idle predicate (no running agents,
no live background work, inbound-quiet) holds AND the dormant quiesce completed.
Why the gateway self-suspends instead of relying on Fly ``autostop:"suspend"``:
Fly Proxy judges idle only on INBOUND proxied connections — it cannot see an
in-flight agent turn (outbound-only LLM traffic) and no longer counts open
outbound sockets, so it would suspend mid-job or BEFORE ``go_dormant()`` flipped
the relay destination (buffered-event black hole). Owning the call means it only
fires after the idle predicate holds AND the dormant quiesce completed.
Design constraints (decisions.md):
- Per-instance enable is gated SOLELY by the NAS "Labs" toggle, carried to the
gateway as the ``HERMES_SCALE_TO_ZERO`` env stamp (D11/Q8=A). NOT a user
config key; ``scale_to_zero.idle_timeout_minutes`` IS config.yaml (D2).
- Arm only when messaging is relay-only or absent (D1/F6) AND a wakeUrl is
registered (§3.4(1)) AND the flag is set.
- Idle = no in-flight agent turn AND no inbound for N min AND no live
background work (D2/D3/F7).
- The quiesce uses ``go_dormant()`` (socket closed + supervisor preserved),
NEVER the stop/restart drain or ``disconnect()`` (F12/F14). The process stays
alive; Fly freezes+resumes it.
- ``mark_resume_pending`` is deliberately NOT called here (D13 — suspend
preserves RAM; revive only if we move to autostop:"stop" or see kills).
Design constraints:
- Per-instance enable is gated SOLELY by the NAS "Labs" toggle, carried as
the ``HERMES_SCALE_TO_ZERO`` env stamp — not a config key;
``scale_to_zero.idle_timeout_minutes`` IS config.yaml.
- Arm only when messaging is relay-only or absent AND a wakeUrl is registered
AND the flag is set.
- Idle = no in-flight work AND no inbound for N min AND no live background work.
- Quiesce uses ``go_dormant()`` (socket closed, supervisor preserved), NEVER
the stop/restart drain or ``disconnect()``. The process stays alive.
- ``mark_resume_pending`` is deliberately NOT called: suspend preserves RAM.
The pure helpers (``parse_idle_timeout_seconds``, ``scale_to_zero_enabled``,
``messaging_is_relay_only_or_absent``, ``is_idle``, ``should_arm``) take plain
inputs so they unit-test without a live gateway.
The pure helpers take plain inputs so they unit-test without a live gateway.
"""
from __future__ import annotations
@@ -42,56 +32,59 @@ import json
import logging
import os
import socket
import time
from pathlib import Path
from typing import Any, Iterable, Optional
logger = logging.getLogger(__name__)
# Env flag stamped by NAS when the scaleToZero Labs toggle is on (D11/Q8=A),
# mirroring how the `relay` feature stamps GATEWAY_RELAY_URL. Truthy values only.
# Env flag stamped by NAS when the scaleToZero Labs toggle is on. Truthy values only.
SCALE_TO_ZERO_ENV = "HERMES_SCALE_TO_ZERO"
# Fly-injected machine identity (present on every Fly machine). Used by the
# self-suspend call; both must be present for self_suspend_available().
# Fly-injected machine identity; both must be present for self_suspend_available().
FLY_APP_NAME_ENV = "FLY_APP_NAME"
FLY_MACHINE_ID_ENV = "FLY_MACHINE_ID"
# The local flaps (Fly Machines API) unix socket, available inside every Fly
# machine. A POST to /v1/apps/{app}/machines/{id}/suspend snapshots RAM and
# suspends THIS machine — the Fly-endorsed replacement for proxy autostop when
# the app must own the idle decision (https://fly.io/docs/reference/suspend-resume/).
# Local flaps (Fly Machines API) unix socket. POST .../suspend snapshots RAM and
# suspends THIS machine (https://fly.io/docs/reference/suspend-resume/).
FLY_API_SOCKET = "/.fly/api"
# config.yaml default (D2). Behavioural setting -> config, not env.
# 2 minutes: with the gateway owning the suspend (idle predicate covers agent
# turns, cron, API runs, and background work; the relay drains + flips before
# the freeze), a short window is safe — real work always blocks the suspend and
# resume-from-suspend is sub-second, so the only cost of waking "too eagerly"
# after a quiet spell is a Fly-proxied poke away. Longer windows just bill idle
# RAM. Raise per-instance via gateway.scale_to_zero.idle_timeout_minutes.
# config.yaml default (behavioural setting -> config, not env). Short is safe
# because real work always blocks the suspend and resume is sub-second; longer
# windows just bill idle RAM. Raise per-instance via
# gateway.scale_to_zero.idle_timeout_minutes.
DEFAULT_IDLE_TIMEOUT_MINUTES = 2
_TRUTHY = {"1", "true", "yes", "on"}
# Dashboard-client liveness marker. The dashboard process (tui_gateway/ws.py, a
# DIFFERENT process on hosted instances) touches this on every /api/ws connect
# and inbound frame (clients ping every 15s). The gateway folds the mtime into
# its inbound clock so an open client holds the box awake like a chat message
# does (and gets the same idle_timeout grace after it disconnects) — otherwise
# the box suspends under the client, whose reconnect re-pokes
# the wake URL and the instance flaps every ~60s. Deliberately NO staleness
# cutoff: the mtime is a real inbound timestamp and is_idle already decides
# whether it is recent enough.
DASHBOARD_CLIENT_HEARTBEAT_REL = os.path.join("state", "dashboard_clients.heartbeat")
def _env_str(env: Optional[dict], key: str) -> str:
return str((os.environ if env is None else env).get(key, "")).strip()
def scale_to_zero_enabled(environ: Optional[dict] = None) -> bool:
"""Whether the per-instance Labs toggle is on (the HERMES_SCALE_TO_ZERO stamp).
D11/Q8=A: this env flag is the SOLE per-instance enable signal reaching the
gateway. Absent/blank/falsey -> disabled (fail-safe default off).
"""
env = environ if environ is not None else os.environ
return str(env.get(SCALE_TO_ZERO_ENV, "")).strip().lower() in _TRUTHY
"""Whether the Labs toggle stamp is set. Absent/blank/falsey -> disabled."""
return _env_str(environ, SCALE_TO_ZERO_ENV).lower() in _TRUTHY
def parse_idle_timeout_seconds(
cfg_value: Any, default_minutes: int = DEFAULT_IDLE_TIMEOUT_MINUTES
) -> float:
"""Coerce ``scale_to_zero.idle_timeout_minutes`` (config.yaml, D2) to seconds.
"""Coerce ``scale_to_zero.idle_timeout_minutes`` to seconds.
Degrades to the default on any non-numeric / non-positive value (never raises,
never returns <= 0 — a zero/negative timeout would make the gateway go dormant
instantly, which is never the intent).
Any non-numeric / non-positive value degrades to the default (never <= 0:
that would make the gateway go dormant instantly).
"""
try:
minutes = float(cfg_value)
@@ -103,22 +96,18 @@ def parse_idle_timeout_seconds(
def messaging_is_relay_only_or_absent(platforms: Iterable[Any]) -> bool:
"""True iff the only connected messaging platform is RELAY, or there is none
(a Chronos-only / no-platform agent) — the F6/D1 structural precondition.
"""True iff the only connected platform is RELAY, or there is none.
A directly-connected platform (Discord/Telegram/Slack/...) holds a live
socket and cannot scale to zero, so its presence disarms the feature. We
compare by the platform's ``.value``/name to avoid importing the enum here
(keeps this module import-light and unit-testable).
A directly-connected platform holds a live socket and cannot scale to zero.
Compared by ``.value``/name so this module stays enum-import-free.
"""
names = {_platform_name(p) for p in platforms}
names.discard("relay")
return len(names) == 0
return not names
def _platform_name(platform: Any) -> str:
value = getattr(platform, "value", platform)
return str(value).strip().lower()
return str(getattr(platform, "value", platform)).strip().lower()
def should_arm(
@@ -127,13 +116,10 @@ def should_arm(
relay_only_or_absent: bool,
wake_url: Optional[str],
) -> bool:
"""Whether to start the idle watcher at all (D1/D11/§3.4(1)).
ALL must hold: the Labs flag is on, messaging is relay-only/absent, and a
wakeUrl is registered (a suspended instance with no reachable wake target is
a black hole — §3.4(1)). Any unmet -> the watcher never starts (no idle
timer, no dormancy), so a non-opted instance behaves exactly as today.
"""
"""Start the idle watcher only if ALL hold: flag on, relay-only/absent messaging,
wakeUrl registered (a suspended instance with no wake target is a black hole).
Any unmet -> the watcher never starts (no idle timer, no dormancy), so a
non-opted instance behaves exactly as before."""
return bool(enabled) and bool(relay_only_or_absent) and bool(wake_url)
@@ -144,48 +130,22 @@ def is_idle(
idle_timeout_seconds: float,
has_live_background_work: bool,
) -> bool:
"""The idle predicate (D2/D3/F7). Pure — composes the three conjuncts.
"""The pure idle predicate: no counted active work, no inbound within the
timeout window, and no live background work (backgrounded delegate_task /
kanban / bg terminal) — suspending mid-flight would lose it.
Idle iff: no counted active work (in-flight agent turns + cron jobs +
API-server runs — the caller aggregates every foreground work source),
no inbound within the timeout window, and no live background work
(backgrounded delegate_task / kanban / bg terminal). Any active work
keeps the gateway awake — suspending mid-flight would lose it.
``active_work_count`` deliberately names the BROAD aggregate, not just
agents: a caller passing only ``len(_running_agents)`` reopens the
mid-cron-job suspend hole. Callers that cannot read a work source must
fail AWAKE (pass a positive sentinel), never fail to 0.
``active_work_count`` is the BROAD aggregate (agent turns + cron + API runs),
not just agents — passing only ``len(_running_agents)`` reopens the
mid-cron-job suspend hole. Callers that cannot read a work source must fail
AWAKE (pass a positive sentinel), never fail to 0.
"""
if active_work_count > 0:
return False
if has_live_background_work:
if active_work_count > 0 or has_live_background_work:
return False
return seconds_since_last_inbound >= idle_timeout_seconds
# Dashboard-client liveness marker. The dashboard process (tui_gateway/ws.py,
# a DIFFERENT process from the gateway on hosted instances) touches this file
# on every /api/ws connect and inbound frame — the desktop app, web dashboard
# and TUI all send `gateway.ping` every 15s (apps/shared/src/json-rpc-gateway.ts,
# ui-tui/src/gatewayClient.ts). The gateway folds the mtime into its inbound
# clock, so an open client holds the box awake exactly like a chat message does
# and gets the same idle_timeout grace after it disconnects. Without this the
# box suspends under the open client, the client's reconnect loop re-pokes the
# Fly-proxied hostname, autostart resumes it, and the instance flaps every ~60s
# (13 of 72 active opted-in prod instances, 2026-09-02).
#
# There is deliberately NO staleness cutoff here: the mtime is a timestamp of
# real inbound, and is_idle already decides whether it is recent enough. A
# lingering marker cannot pin the box — once it is older than idle_timeout it
# no longer counts, exactly like an old _last_inbound_at.
DASHBOARD_CLIENT_HEARTBEAT_REL = os.path.join("state", "dashboard_clients.heartbeat")
def dashboard_client_heartbeat_path(hermes_home: Optional[os.PathLike | str] = None):
"""Path of the dashboard-client liveness marker under HERMES_HOME."""
from pathlib import Path
if hermes_home is None:
from hermes_constants import get_hermes_home
@@ -214,18 +174,15 @@ def dashboard_client_last_seen(
) -> Optional[float]:
"""Epoch seconds a dashboard client last sent a WS frame, or None if never.
Missing marker -> None (the steady state on a box nobody has the dashboard
open on — NOT fail-awake, or every instance would never sleep). An
unreadable marker -> ``now`` (fail-awake: an unreadable source counts as
activity, same rule as the work counters in ``is_idle``).
Missing marker -> None (steady state when nobody has the dashboard open —
NOT fail-awake, or no instance would ever sleep). Unreadable marker ->
``now`` (fail-awake, same rule as the work counters in ``is_idle``).
"""
import time
current = time.time() if now is None else now
p = dashboard_client_heartbeat_path() if path is None else path
try:
# Clamp to now: a wall-clock step-back (NTP) can leave the mtime in the
# future, which would push idle out by the step size for no reason.
# Clamp to now: an NTP step-back can leave the mtime in the future,
# which would push idle out by the step size for no reason.
return min(os.stat(p).st_mtime, current)
except FileNotFoundError:
return None
@@ -234,17 +191,14 @@ def dashboard_client_last_seen(
def self_suspend_available(environ: Optional[dict] = None) -> bool:
"""Whether this process can suspend its own machine via the flaps socket.
"""True iff Fly machine identity is present AND the local Machines API socket exists.
True iff the Fly-injected machine identity is present AND the local Machines
API socket exists. Off-Fly (local dev, Azure ACA, tests) this is False and
the watcher skips the quiesce entirely: the platform owns the freeze, so
the gateway stays connected until it lands.
Off-Fly (local dev, other clouds, tests) the watcher skips the quiesce:
the platform owns the freeze, so the gateway stays connected until it lands.
"""
env = environ if environ is not None else os.environ
return bool(
str(env.get(FLY_APP_NAME_ENV, "")).strip()
and str(env.get(FLY_MACHINE_ID_ENV, "")).strip()
_env_str(environ, FLY_APP_NAME_ENV)
and _env_str(environ, FLY_MACHINE_ID_ENV)
and os.path.exists(FLY_API_SOCKET)
)
@@ -257,24 +211,14 @@ def suspend_self(
) -> bool:
"""POST /v1/apps/{app}/machines/{id}/suspend on the local flaps socket.
Fly's in-machine Machines API needs no token — the socket itself is the
credential. Equivalent to:
curl --unix-socket /.fly/api -X POST \\
http://flaps/v1/apps/$FLY_APP_NAME/machines/$FLY_MACHINE_ID/suspend
Returns True when flaps accepted the request (2xx). The caller should treat
this as fire-and-forget: on success the kernel freezes this process shortly
after, so there may be nothing meaningful to run afterwards. Never raises —
a failed suspend just leaves the machine running (fail-awake, never
fail-frozen), which costs money but loses no work.
stdlib-only on purpose: a plain unix-socket HTTP/1.1 request, no httpx/
requests dependency in the hot path and no async plumbing to freeze
mid-await.
No token needed — the socket is the credential. Returns True when flaps
accepted (2xx); on success the kernel freezes this process shortly after,
so treat as fire-and-forget. Never raises: a failed suspend just leaves the
machine running (fail-awake, never fail-frozen). stdlib-only on purpose —
a plain unix-socket HTTP/1.1 request, no async plumbing to freeze mid-await.
"""
env = environ if environ is not None else os.environ
app = str(env.get(FLY_APP_NAME_ENV, "")).strip()
machine_id = str(env.get(FLY_MACHINE_ID_ENV, "")).strip()
app = _env_str(environ, FLY_APP_NAME_ENV)
machine_id = _env_str(environ, FLY_MACHINE_ID_ENV)
if not app or not machine_id:
logger.warning("scale-to-zero: suspend_self called without Fly machine identity")
return False
gateway/session_context.py
+194 -356
View File
@@ -1,198 +1,152 @@
"""
Session-scoped context variables for the Hermes gateway.
"""Session-scoped context variables for the Hermes gateway.
Replaces the previous ``os.environ``-based session state
(``HERMES_SESSION_PLATFORM``, ``HERMES_SESSION_CHAT_ID``, etc.) with
Python's ``contextvars.ContextVar``.
Replaces the old ``os.environ``-based session state (``HERMES_SESSION_*``)
with ``contextvars.ContextVar``. The gateway processes messages concurrently
via asyncio; ``os.environ`` is process-global, so message B silently
overwrote message A's thread id before A's agent finished and notifications
routed to the wrong thread. ContextVar values are task-local (inherited by
``run_in_executor`` threads), so concurrent messages never interfere.
**Why this matters**
The gateway processes messages concurrently via ``asyncio``. When two
messages arrive at the same time the old code did:
os.environ["HERMES_SESSION_THREAD_ID"] = str(context.source.thread_id)
Because ``os.environ`` is *process-global*, Message A's value was
silently overwritten by Message B before Message A's agent finished
running. Background-task notifications and tool calls therefore routed
to the wrong thread.
``contextvars.ContextVar`` values are *task-local*: each ``asyncio``
task (and any ``run_in_executor`` thread it spawns) gets its own copy,
so concurrent messages never interfere.
**Backward compatibility**
The public helper ``get_session_env(name, default="")`` mirrors the old
``os.getenv("HERMES_SESSION_*", ...)`` calls. Existing tool code only
needs to replace the import + call site:
# before
import os
platform = os.getenv("HERMES_SESSION_PLATFORM", "")
# after
from gateway.session_context import get_session_env
platform = get_session_env("HERMES_SESSION_PLATFORM", "")
``get_session_env(name, default="")`` is a drop-in for
``os.getenv("HERMES_SESSION_*", default)`` at existing tool call sites.
"""
from contextlib import contextmanager
from contextvars import ContextVar
from typing import Any, Iterator
# Sentinel to distinguish "never set in this context" from "explicitly set to empty".
# When a contextvar holds _UNSET, we fall back to os.environ (CLI/cron compat).
# When it holds "" (after clear_session_vars resets it), we return "" — no fallback.
# Distinguishes "never set in this context" (fall back to os.environ for
# CLI/cron compat) from "explicitly set to empty" by clear_session_vars (no fallback).
_UNSET: Any = object()
# Process-level flag: has any code in this process bound a session via
# set_session_vars()? Concurrent multi-session hosts (the messaging gateway, the
# ACP adapter, the API server, the TUI, cron) all do; a pure single-process
# CLI/one-shot that never engages the session-context system does not.
#
# The subprocess-env bridge (tools/environments/local.py) reads this to choose
# its leak policy: when engaged, the ContextVars are authoritative and an _UNSET
# var means "no session bound in THIS task" — so a process-global os.environ
# mirror (written last-writer-wins by whatever concurrent session ran most
# recently) must NOT be inherited into a child process. When never engaged, the
# os.environ fallback is preserved (no concurrency to leak across). Monotonic
# latch — once any host binds a session, the process stays engaged for life.
# Process-level monotonic latch: has any code in this process bound a session via
# set_session_vars()? Concurrent multi-session hosts (gateway, ACP, API server,
# TUI, cron) do; a pure single-process CLI/one-shot does not. The subprocess-env
# bridge (tools/environments/local.py) reads this to pick its leak policy: when
# engaged, the ContextVars are authoritative and an _UNSET var means "no session
# bound in THIS task", so the last-writer-wins os.environ mirror must NOT be
# inherited by a child process. When never engaged, the os.environ fallback is
# preserved (no concurrency to leak across).
_session_context_engaged: bool = False
def session_context_engaged() -> bool:
"""True if any session has been bound via set_session_vars in this process.
See the ``_session_context_engaged`` comment for the leak-policy rationale.
"""
"""True if any session has been bound via set_session_vars in this process."""
return _session_context_engaged
# ---------------------------------------------------------------------------
# Per-task session variables
# ---------------------------------------------------------------------------
_SESSION_PLATFORM: ContextVar = ContextVar("HERMES_SESSION_PLATFORM", default=_UNSET)
_SESSION_SOURCE: ContextVar = ContextVar("HERMES_SESSION_SOURCE", default=_UNSET)
_SESSION_CHAT_ID: ContextVar = ContextVar("HERMES_SESSION_CHAT_ID", default=_UNSET)
_SESSION_CHAT_TYPE: ContextVar = ContextVar("HERMES_SESSION_CHAT_TYPE", default=_UNSET)
_SESSION_CHAT_NAME: ContextVar = ContextVar("HERMES_SESSION_CHAT_NAME", default=_UNSET)
_SESSION_THREAD_ID: ContextVar = ContextVar("HERMES_SESSION_THREAD_ID", default=_UNSET)
_SESSION_USER_ID: ContextVar = ContextVar("HERMES_SESSION_USER_ID", default=_UNSET)
_SESSION_USER_ID_ALT: ContextVar = ContextVar("HERMES_SESSION_USER_ID_ALT", default=_UNSET)
_SESSION_USER_NAME: ContextVar = ContextVar("HERMES_SESSION_USER_NAME", default=_UNSET)
# Platform-neutral scope discriminator (Discord guild / Slack workspace /
# Matrix server) of the originating chat. Captured at session-bind time so
# async producers (delegate_task background=True, terminal watchers) can
# persist a completion's full routing origin — on a relay-fronted deployment
# the connector's fail-closed egress guard needs scope_id (or a user binding)
# to resolve the tenant for a scoped reply after a restart.
_SESSION_SCOPE_ID: ContextVar = ContextVar("HERMES_SESSION_SCOPE_ID", default=_UNSET)
_SESSION_KEY: ContextVar = ContextVar("HERMES_SESSION_KEY", default=_UNSET)
_SESSION_ID: ContextVar = ContextVar("HERMES_SESSION_ID", default=_UNSET)
# In-process UI session/window id for multi-session desktop/TUI hosts. This is
# intentionally separate from HERMES_SESSION_ID: the latter is the durable
# conversation/session-db id, while the UI id is the live frontend tab/window
# that commissioned a detached completion. Background completions use it as a
# precise return address so a stale/rotated durable session key cannot be
# consumed by whichever desktop poller wakes first.
_SESSION_UI_SESSION_ID: ContextVar = ContextVar("HERMES_UI_SESSION_ID", default=_UNSET)
# ID of the message that triggered the current turn. Used as a reply anchor
# so background-process notifications stay inside the originating Telegram
# private-chat topic (those lanes route only with thread id + reply anchor).
_SESSION_MESSAGE_ID: ContextVar = ContextVar("HERMES_SESSION_MESSAGE_ID", default=_UNSET)
def _var(name: str) -> ContextVar:
return ContextVar(name, default=_UNSET)
_SESSION_PROFILE: ContextVar = ContextVar("HERMES_SESSION_PROFILE", default=_UNSET)
_BROWSER_CONTROL_PRINCIPAL: ContextVar = ContextVar(
"HERMES_BROWSER_CONTROL_PRINCIPAL", default=_UNSET
)
_BROWSER_CONTROL_TRANSPORT_FAMILY: ContextVar = ContextVar(
"HERMES_BROWSER_CONTROL_TRANSPORT_FAMILY", default=_UNSET
# --- Per-task session variables --------------------------------------------
_SESSION_PLATFORM = _var("HERMES_SESSION_PLATFORM")
_SESSION_SOURCE = _var("HERMES_SESSION_SOURCE")
_SESSION_CHAT_ID = _var("HERMES_SESSION_CHAT_ID")
_SESSION_CHAT_TYPE = _var("HERMES_SESSION_CHAT_TYPE")
_SESSION_CHAT_NAME = _var("HERMES_SESSION_CHAT_NAME")
_SESSION_THREAD_ID = _var("HERMES_SESSION_THREAD_ID")
_SESSION_USER_ID = _var("HERMES_SESSION_USER_ID")
_SESSION_USER_ID_ALT = _var("HERMES_SESSION_USER_ID_ALT")
_SESSION_USER_NAME = _var("HERMES_SESSION_USER_NAME")
# Platform-neutral scope discriminator (Discord guild / Slack workspace / Matrix
# server). Captured at bind time so async producers (delegate_task
# background=True, terminal watchers) can persist a completion's full routing
# origin: a relay connector's fail-closed egress guard needs scope_id (or a user
# binding) to resolve the tenant for a scoped reply after a restart.
_SESSION_SCOPE_ID = _var("HERMES_SESSION_SCOPE_ID")
_SESSION_KEY = _var("HERMES_SESSION_KEY")
_SESSION_ID = _var("HERMES_SESSION_ID")
# In-process UI tab/window id for multi-session desktop/TUI hosts — deliberately
# separate from the durable HERMES_SESSION_ID. Background completions use it as
# a precise return address so a stale/rotated durable key cannot be consumed by
# whichever desktop poller wakes first.
_SESSION_UI_SESSION_ID = _var("HERMES_UI_SESSION_ID")
# Triggering message id: reply anchor so background notifications stay inside
# the originating Telegram private-chat topic (routes only with thread id + anchor).
_SESSION_MESSAGE_ID = _var("HERMES_SESSION_MESSAGE_ID")
_SESSION_PROFILE = _var("HERMES_SESSION_PROFILE")
_BROWSER_CONTROL_PRINCIPAL = _var("HERMES_BROWSER_CONTROL_PRINCIPAL")
_BROWSER_CONTROL_TRANSPORT_FAMILY = _var("HERMES_BROWSER_CONTROL_TRANSPORT_FAMILY")
# Per-session cron marker, tri-state: _UNSET keeps the legacy env fallback for
# CLI/tests; "1" marks cron; "" explicitly marks non-cron and masks leaked env.
_CRON_SESSION = _var("HERMES_CRON_SESSION")
# Whether this session's channel can route an ASYNC completion back to the agent
# AFTER the turn ends (wake a fresh turn). True for long-lived CLI sessions and
# real gateway platforms (persistent outbound channel + watcher/drain loops);
# False for finite runtimes that may exit before a detached completion returns
# (stateless API-server requests, dispatcher-spawned Kanban workers). Tools that
# promise async delivery (terminal notify_on_complete / watch_patterns,
# delegate_task background=True) read ``async_delivery_supported()`` and refuse a
# promise the channel can't keep. Default _UNSET => supported, so the CLI (never
# sets a platform) and contextvar-unaware paths keep working; stateless adapters
# opt OUT via ``supports_async_delivery = False`` on the adapter class, which the
# gateway propagates here at session-bind time.
_SESSION_ASYNC_DELIVERY = _var("HERMES_SESSION_ASYNC_DELIVERY")
# Cron auto-delivery vars — set per-job in run_job() so concurrent jobs don't
# clobber each other's delivery targets.
_CRON_AUTO_DELIVER_PLATFORM = _var("HERMES_CRON_AUTO_DELIVER_PLATFORM")
_CRON_AUTO_DELIVER_CHAT_ID = _var("HERMES_CRON_AUTO_DELIVER_CHAT_ID")
_CRON_AUTO_DELIVER_THREAD_ID = _var("HERMES_CRON_AUTO_DELIVER_THREAD_ID")
# Vars bound by set_session_vars / cleared to "" by clear_session_vars, in order.
_SESSION_VARS = (
_SESSION_PLATFORM,
_SESSION_SOURCE,
_SESSION_CHAT_ID,
_SESSION_CHAT_TYPE,
_SESSION_CHAT_NAME,
_SESSION_THREAD_ID,
_SESSION_USER_ID,
_SESSION_USER_ID_ALT,
_SESSION_USER_NAME,
_SESSION_SCOPE_ID,
_SESSION_KEY,
_SESSION_ID,
_SESSION_UI_SESSION_ID,
_SESSION_MESSAGE_ID,
_SESSION_PROFILE,
_BROWSER_CONTROL_PRINCIPAL,
_BROWSER_CONTROL_TRANSPORT_FAMILY,
_CRON_SESSION,
)
# Per-session cron marker. Unlike the process-global legacy env var, this is
# scoped to one cron job / inbound session. _UNSET preserves the legacy env
# fallback for CLI/tests; "1" marks cron; "" explicitly marks non-cron and
# masks any leaked process env value.
_CRON_SESSION: ContextVar = ContextVar("HERMES_CRON_SESSION", default=_UNSET)
# Whether the current session's delivery channel can route an ASYNC completion
# back to the agent AFTER the current turn ends (i.e. wake a fresh turn).
#
# True — long-lived CLI sessions (in-process completion_queue drain) and the
# real gateway platforms (Telegram/Discord/Slack/...), which hold a
# persistent outbound channel and run the watcher/drain loops.
# False — finite runtimes that can end before a detached completion returns:
# stateless API-server requests and dispatcher-spawned Kanban workers.
#
# Tools that promise async delivery (terminal notify_on_complete /
# watch_patterns, delegate_task background=True) read this via
# ``async_delivery_supported()`` and refuse to hand out a promise the channel
# can't keep — turning a silent no-op into an explicit contract.
#
# Default _UNSET => treated as supported, so CLI (which never sets a platform)
# and any contextvar-unaware path keep working. Stateless adapters opt OUT by
# setting ``supports_async_delivery = False`` on the adapter class; the gateway
# propagates that into this contextvar at session-bind time.
_SESSION_ASYNC_DELIVERY: ContextVar = ContextVar("HERMES_SESSION_ASYNC_DELIVERY", default=_UNSET)
# Cron auto-delivery vars — set per-job in run_job() so concurrent jobs
# don't clobber each other's delivery targets.
_CRON_AUTO_DELIVER_PLATFORM: ContextVar = ContextVar("HERMES_CRON_AUTO_DELIVER_PLATFORM", default=_UNSET)
_CRON_AUTO_DELIVER_CHAT_ID: ContextVar = ContextVar("HERMES_CRON_AUTO_DELIVER_CHAT_ID", default=_UNSET)
_CRON_AUTO_DELIVER_THREAD_ID: ContextVar = ContextVar("HERMES_CRON_AUTO_DELIVER_THREAD_ID", default=_UNSET)
# Legacy env-var name -> ContextVar, for get_session_env. _SESSION_ASYNC_DELIVERY
# is deliberately absent: it is a bool capability read via async_delivery_supported.
_VAR_MAP = {
"HERMES_SESSION_PLATFORM": _SESSION_PLATFORM,
"HERMES_SESSION_SOURCE": _SESSION_SOURCE,
"HERMES_SESSION_CHAT_ID": _SESSION_CHAT_ID,
"HERMES_SESSION_CHAT_TYPE": _SESSION_CHAT_TYPE,
"HERMES_SESSION_CHAT_NAME": _SESSION_CHAT_NAME,
"HERMES_SESSION_THREAD_ID": _SESSION_THREAD_ID,
"HERMES_SESSION_USER_ID": _SESSION_USER_ID,
"HERMES_SESSION_USER_ID_ALT": _SESSION_USER_ID_ALT,
"HERMES_SESSION_USER_NAME": _SESSION_USER_NAME,
"HERMES_SESSION_SCOPE_ID": _SESSION_SCOPE_ID,
"HERMES_SESSION_KEY": _SESSION_KEY,
"HERMES_SESSION_ID": _SESSION_ID,
"HERMES_UI_SESSION_ID": _SESSION_UI_SESSION_ID,
"HERMES_SESSION_MESSAGE_ID": _SESSION_MESSAGE_ID,
"HERMES_SESSION_PROFILE": _SESSION_PROFILE,
"HERMES_BROWSER_CONTROL_PRINCIPAL": _BROWSER_CONTROL_PRINCIPAL,
"HERMES_BROWSER_CONTROL_TRANSPORT_FAMILY": _BROWSER_CONTROL_TRANSPORT_FAMILY,
"HERMES_CRON_SESSION": _CRON_SESSION,
"HERMES_CRON_AUTO_DELIVER_PLATFORM": _CRON_AUTO_DELIVER_PLATFORM,
"HERMES_CRON_AUTO_DELIVER_CHAT_ID": _CRON_AUTO_DELIVER_CHAT_ID,
"HERMES_CRON_AUTO_DELIVER_THREAD_ID": _CRON_AUTO_DELIVER_THREAD_ID,
var.name: var
for var in (*_SESSION_VARS, _CRON_AUTO_DELIVER_PLATFORM, _CRON_AUTO_DELIVER_CHAT_ID, _CRON_AUTO_DELIVER_THREAD_ID)
}
def _clear_session_cwd() -> None:
try:
from agent.runtime_cwd import clear_session_cwd
clear_session_cwd()
except Exception:
pass
def set_current_session_id(session_id: str) -> None:
"""Synchronize ``HERMES_SESSION_ID`` across ContextVar and ``os.environ``.
Long-lived single-process entrypoints like the CLI can rotate sessions via
``/new``, ``/resume``, ``/branch``, or compression splits without
reconstructing the entire agent. Tools still consult
``get_session_env("HERMES_SESSION_ID")`` with an ``os.environ`` fallback,
so both storage paths must move together when the active session changes.
Long-lived single-process entrypoints (CLI) rotate sessions via /new,
/resume, /branch, or compression splits without rebuilding the agent; tools
read ``get_session_env("HERMES_SESSION_ID")`` with an os.environ fallback,
so both stores must move together.
Delegated subagent children are the exception: they are constructed inside
the parent process within ``delegated_child_context()``, and their
``AIAgent.__init__`` calls this same helper. Writing a child's internal
session id to ``os.environ`` (process-global) would clobber the parent's
``HERMES_SESSION_ID`` for the rest of the process — leaking the child id
into parent tools and subprocesses spawned after the child was built. The
ContextVar write below is task-local and safe for concurrent children; only
the process-global ``os.environ`` mirror is suppressed for delegated
children. Root agents (CLI, gateway, cron) keep both paths.
Delegated subagent children are the exception: they are built in the parent
process inside ``delegated_child_context()`` and their ``AIAgent.__init__``
calls this helper. Writing the child's id to process-global ``os.environ``
would clobber the parent's id for the rest of the process, so only the
task-local ContextVar write happens for them. Root agents keep both paths.
"""
import os
_SESSION_ID.set(session_id)
# Skip the process-global os.environ write for delegated children. The
# child's own tools and subprocesses still resolve their id through the
# ContextVar (task-local), while the parent's process-wide env keeps the
# parent's session identity. See HermesPRDelegationSessionContext task.
try:
from agent.delegation_context import is_delegated_child_context
@@ -200,7 +154,6 @@ def set_current_session_id(session_id: str) -> None:
return
except Exception:
pass
os.environ["HERMES_SESSION_ID"] = session_id
@@ -208,9 +161,9 @@ def set_current_session_id(session_id: str) -> None:
def scoped_current_session_id(session_id: str | None = None) -> Iterator[None]:
"""Bind a task-local session id and restore the prior value on exit.
With ``session_id=None`` this acts as a save/restore boundary around code
that may call :func:`set_current_session_id` itself (notably delegated
``AIAgent`` construction). It intentionally never mutates ``os.environ``.
``session_id=None`` makes this a pure save/restore boundary around code that
may call :func:`set_current_session_id` itself (delegated ``AIAgent``
construction). Never mutates ``os.environ``.
"""
previous = _SESSION_ID.get()
if session_id is not None:
@@ -245,49 +198,26 @@ def set_session_vars(
) -> list:
"""Set all session context variables and return reset tokens.
Call ``clear_session_vars(tokens)`` in a ``finally`` block when the handler
exits. Note ``clear_session_vars`` resets every var to ``""`` (to suppress
the ``os.environ`` fallback) rather than restoring prior values — these
helpers are not nestable/stack-safe, and the returned tokens are accepted
only for API compatibility.
Call ``clear_session_vars(tokens)`` in a ``finally`` when the handler exits.
These helpers are not nestable: clearing resets every var to ``""`` rather
than restoring prior values, and the tokens are accepted only for API compat.
``cwd`` pins the logical working directory for this context.
``async_delivery`` declares whether this session's channel can route a
background completion back to the agent after the turn ends (see
``_SESSION_ASYNC_DELIVERY`` / ``async_delivery_supported``). Stateless
request/response adapters (the API server) pass ``False``.
``cron_session`` is tri-state: ``_UNSET`` preserves legacy
``os.environ["HERMES_CRON_SESSION"]`` fallback, ``"1"`` marks a cron job,
and ``""`` explicitly marks a non-cron session while masking leaked env.
``cwd`` pins the logical working directory. ``async_delivery`` declares
whether the channel can route a background completion back after the turn
(stateless adapters such as the API server pass ``False``). ``cron_session``
is tri-state; see ``_CRON_SESSION``.
"""
# Mark the session-context machinery engaged for this process. The
# subprocess-env bridge uses this to switch from "os.environ fallback" to
# "ContextVar-authoritative, strip on _UNSET" — see session_context_engaged.
# Latch the process as engaged — see _session_context_engaged.
global _session_context_engaged
_session_context_engaged = True
tokens = [
_SESSION_PLATFORM.set(platform),
_SESSION_SOURCE.set(source),
_SESSION_CHAT_ID.set(chat_id),
_SESSION_CHAT_TYPE.set(chat_type),
_SESSION_CHAT_NAME.set(chat_name),
_SESSION_THREAD_ID.set(thread_id),
_SESSION_USER_ID.set(user_id),
_SESSION_USER_ID_ALT.set(user_id_alt),
_SESSION_USER_NAME.set(user_name),
_SESSION_SCOPE_ID.set(scope_id),
_SESSION_KEY.set(session_key),
_SESSION_ID.set(session_id),
_SESSION_UI_SESSION_ID.set(ui_session_id),
_SESSION_MESSAGE_ID.set(message_id),
_SESSION_PROFILE.set(profile),
_BROWSER_CONTROL_PRINCIPAL.set(browser_control_principal),
_BROWSER_CONTROL_TRANSPORT_FAMILY.set(browser_control_transport_family),
_CRON_SESSION.set(cron_session),
_SESSION_ASYNC_DELIVERY.set(bool(async_delivery)),
]
values = (
platform, source, chat_id, chat_type, chat_name, thread_id, user_id,
user_id_alt, user_name, scope_id, session_key, session_id, ui_session_id,
message_id, profile, browser_control_principal,
browser_control_transport_family, cron_session,
)
tokens = [var.set(value) for var, value in zip(_SESSION_VARS, values)]
tokens.append(_SESSION_ASYNC_DELIVERY.set(bool(async_delivery)))
try:
from agent.runtime_cwd import set_session_cwd
@@ -300,133 +230,68 @@ def set_session_vars(
def clear_session_vars(tokens: list) -> None:
"""Mark session context variables as explicitly cleared.
Sets all variables to ``""`` so that ``get_session_env`` returns an empty
string instead of falling back to (potentially stale) ``os.environ``
values. The *tokens* argument is accepted for API compatibility with
callers that saved the return value of ``set_session_vars``, but the
actual clearing uses ``var.set("")`` rather than ``var.reset(token)``
to ensure the "explicitly cleared" state is distinguishable from
"never set" (which holds the ``_UNSET`` sentinel).
Sets every var to ``""`` (not ``var.reset(token)``) so ``get_session_env``
returns empty instead of falling back to stale ``os.environ`` values while
staying distinguishable from "never set" (``_UNSET``). Async-delivery is
reset to ``_UNSET`` rather than a falsy value: a cleared context must fall
back to default-supported, not look like an opted-out stateless adapter.
"""
for var in (
_SESSION_PLATFORM,
_SESSION_SOURCE,
_SESSION_CHAT_ID,
_SESSION_CHAT_TYPE,
_SESSION_CHAT_NAME,
_SESSION_THREAD_ID,
_SESSION_USER_ID,
_SESSION_USER_ID_ALT,
_SESSION_USER_NAME,
_SESSION_SCOPE_ID,
_SESSION_KEY,
_SESSION_ID,
_SESSION_UI_SESSION_ID,
_SESSION_MESSAGE_ID,
_SESSION_PROFILE,
_BROWSER_CONTROL_PRINCIPAL,
_BROWSER_CONTROL_TRANSPORT_FAMILY,
_CRON_SESSION,
):
for var in _SESSION_VARS:
var.set("")
# Reset async-delivery capability to the "never set" sentinel rather than a
# falsy value: a cleared context should fall back to the default-supported
# behavior (CLI / unaware paths), not be mistaken for an opted-out
# stateless adapter.
_SESSION_ASYNC_DELIVERY.set(_UNSET)
try:
from agent.runtime_cwd import clear_session_cwd
clear_session_cwd()
except Exception:
pass
_clear_session_cwd()
def reset_session_vars() -> None:
"""Reset every session context variable to ``_UNSET`` for THIS context.
Distinct from :func:`clear_session_vars`, which sets the vars to ``""``
("explicitly cleared" — suppresses the os.environ fallback and is used when
a handler *finishes*). This helper restores the ``_UNSET`` sentinel
("never bound in this context"), which is what a freshly-spawned task should
look like *before* it binds its own session.
Unlike :func:`clear_session_vars` (``""`` = "explicitly cleared", used when a
handler *finishes*), this restores "never bound here" — what a freshly
spawned task should look like *before* binding its own session.
🔴 Why this exists — the cross-session ContextVar inheritance leak.
Each gateway message is processed in its own ``asyncio`` task, created via
``create_task`` (which snapshots the *current* context with
``copy_context``). When message B's task is spawned from a context where a
concurrent message A had already called :func:`set_session_vars`, B inherits
A's **set** ContextVars. Until B calls its own ``set_session_vars`` there is
a window where any subprocess B spawns (e.g. a tool shelling out) reads
*A's* ``HERMES_SESSION_*`` identity via the subprocess-env bridge. The
bridge's ``_UNSET``-strip guard cannot help: the vars are not ``_UNSET``,
they are set-to-A. Calling ``reset_session_vars`` at the top of the
per-message handler drops the inherited identity so the window strips safe
(no session) instead of leaking the foreign one; the handler then binds its
own via ``set_session_vars`` a few steps later. See
Why: ``create_task`` snapshots the current context, so message B's task can
inherit message A's already-**set** vars. Until B binds its own session,
any subprocess it spawns reads A's identity through the subprocess-env
bridge — whose _UNSET-strip guard cannot help because the vars are set-to-A.
Calling this at the top of the per-message handler makes that window strip
safe (no session) instead of leaking the foreign one. See
tests/tools/test_local_env_session_leak.py and
tests/gateway/test_session_context_inheritance.py.
Note ``_SESSION_ASYNC_DELIVERY`` lives outside ``_VAR_MAP`` (it is a bool
capability flag read via :func:`async_delivery_supported`, not a string
``HERMES_SESSION_*`` env var read via :func:`get_session_env`), so it is
reset explicitly below. Without it, a task spawned from a context where a
sibling adapter bound ``async_delivery=False`` (the stateless API server)
inherits that ``False`` through the pre-bind window, and
``async_delivery_supported`` wrongly reports the new turn's channel as
unable to route a background completion until ``set_session_vars`` runs.
``_SESSION_ASYNC_DELIVERY`` is reset explicitly (it lives outside
``_VAR_MAP``): otherwise a task spawned from a context where a sibling
adapter bound ``async_delivery=False`` inherits that ``False`` through the
pre-bind window and misreports the new channel as unable to deliver.
"""
for var in _VAR_MAP.values():
var.set(_UNSET)
# Reset the async-delivery capability to "never bound here" (_UNSET) for the
# same inheritance-leak reason as the mapped vars above — see clear_session_vars,
# which resets this var on the handler-exit path for the symmetric concern.
_SESSION_ASYNC_DELIVERY.set(_UNSET)
try:
from agent.runtime_cwd import clear_session_cwd
clear_session_cwd()
except Exception:
pass
_clear_session_cwd()
def get_session_env(name: str, default: str = "") -> str:
"""Read a session context variable by its legacy ``HERMES_SESSION_*`` name.
Drop-in replacement for ``os.getenv("HERMES_SESSION_*", default)``.
Resolution order:
1. Context variable (set by the gateway for concurrency-safe access).
If the variable was explicitly set (even to ``""``) via
``set_session_vars`` or ``clear_session_vars``, that value is
returned — **no fallback to os.environ**.
2. ``os.environ`` (only when the context variable was never set in
this context — i.e. CLI, cron scheduler, and test processes that
don't use ``set_session_vars`` at all).
3. *default*
Drop-in for ``os.getenv(name, default)``. Resolution: the ContextVar if it
was ever set in this context (even to ``""`` — no fallback); else
``os.environ`` (CLI, cron scheduler, tests that never bind); else *default*.
"""
import os
var = _VAR_MAP.get(name)
if var is not None:
value = var.get()
if value is not _UNSET:
return value
# Fall back to os.environ for CLI, cron, and test compatibility
if var is not None and (value := var.get()) is not _UNSET:
return value
return os.getenv(name, default)
# Surfaces that are not a human chat channel. The gateway binds a platform
# value (``telegram``) to HERMES_SESSION_PLATFORM, while the CLI, TUI, and
# desktop bind HERMES_SESSION_SOURCE (``cli``, ``tui``, ``desktop``) and leave
# the platform empty — so both have to be consulted. ``local``, ``api_server``,
# ``webhook``, and ``msgraph_webhook`` are real Platform values that reach
# HERMES_SESSION_PLATFORM but have no attachment channel behind them.
# Default-deny: an unrecognized identity counts as messaging so a newly added
# chat platform is never treated as a private surface before this set is
# updated. Mirrors LOCAL_SESSION_SOURCE_IDS in
# apps/desktop/src/lib/session-source.ts; keep roughly in sync when adding a
# local or programmatic surface.
# Surfaces that are not a human chat channel. The gateway binds a platform value
# (``telegram``) to HERMES_SESSION_PLATFORM while the CLI/TUI/desktop bind
# HERMES_SESSION_SOURCE and leave platform empty, so both are consulted.
# ``local``, ``api_server``, ``webhook``, ``msgraph_webhook`` are real Platform
# values with no attachment channel behind them. Default-deny: an unrecognized
# identity counts as messaging so a new chat platform is never treated as a
# private surface before this set is updated. Mirrors LOCAL_SESSION_SOURCE_IDS
# in apps/desktop/src/lib/session-source.ts; keep roughly in sync.
NON_MESSAGING_SESSION_SURFACES = frozenset(
{
"",
@@ -448,47 +313,33 @@ NON_MESSAGING_SESSION_SURFACES = frozenset(
def session_is_messaging_surface() -> bool:
"""Whether this turn is delivered over a human messaging channel.
Callers use this to decide anything that differs between "the user is
reading a chat message" and "the user is at a machine they own": whether
to emit a delivery tag, whether a file has to land somewhere the gateway
is allowed to send from, whether narration would read as chat noise.
Resolves ``HERMES_PLATFORM``, then the session platform, then the session
source, and reports messaging when any of them names a surface outside
Decides "user is reading a chat message" vs "user is at a machine they own":
delivery tags, whether a file must land somewhere the gateway can send from,
whether narration reads as chat noise. Checks ``HERMES_PLATFORM``, then the
session platform, then the session source against
:data:`NON_MESSAGING_SESSION_SURFACES`.
"""
import os
platform = os.getenv("HERMES_PLATFORM") or get_session_env("HERMES_SESSION_PLATFORM", "")
source = get_session_env("HERMES_SESSION_SOURCE", "")
for identity in (platform, source):
identity = str(identity or "").strip().lower()
if identity and identity not in NON_MESSAGING_SESSION_SURFACES:
return True
return False
return any(
(ident := str(identity or "").strip().lower()) and ident not in NON_MESSAGING_SESSION_SURFACES
for identity in (platform, source)
)
def declare_stateless_channel() -> None:
"""Declare that this session cannot receive an async background completion.
Binds only the delivery capability, leaving every other session var unset.
Use this instead of ``set_session_vars(async_delivery=False)`` on a pure
single-process runner: ``set_session_vars`` also latches
``_session_context_engaged`` (see above), which switches the subprocess
env bridge from "os.environ fallback" to "ContextVar-authoritative, strip on
_UNSET" in ``tools/environments/local.py``. A one-shot CLI that never engages
the session-context system must not flip that latch as a side effect of
declaring a capability.
Callers that already build a full session context (cron's ``run_job``) get
the same state by passing ``async_delivery=False`` to ``set_session_vars``.
A session that cannot take a late completion makes ``delegate_task`` fall
through to its existing inline/synchronous path, so subagent results are
returned within the turn instead of being dispatched to a channel that will
never deliver them.
See NousResearch/hermes-agent#53027 and #63142.
Binds only the delivery capability. Use this instead of
``set_session_vars(async_delivery=False)`` on a pure single-process runner:
``set_session_vars`` also latches ``_session_context_engaged``, which flips
the subprocess env bridge to ContextVar-authoritative — a one-shot CLI must
not flip that latch as a side effect of declaring a capability. Callers that
build a full context (cron's ``run_job``) pass ``async_delivery=False``.
``delegate_task`` then falls through to its inline path so results return
within the turn instead of going to a channel that never delivers.
"""
_SESSION_ASYNC_DELIVERY.set(False)
@@ -496,30 +347,17 @@ def declare_stateless_channel() -> None:
def async_delivery_supported() -> bool:
"""Whether the current session can deliver a background completion later.
Returns ``False`` for finite runtimes that can end before a detached result
is delivered: sessions explicitly bound by a stateless channel — an adapter
that cannot route a notification back after the turn ends (the API server),
or a one-shot runner that exits after its final response (``hermes -z``,
cron — see :func:`declare_stateless_channel`) — and dispatcher-spawned
Kanban workers (identified by ``HERMES_KANBAN_TASK``), which are one-shot
``chat -q`` subprocesses. The real gateway platforms, the interactive CLI,
and any other path that never bound the contextvar return ``True``.
Tools that promise async delivery (``terminal`` notify_on_complete /
watch_patterns, ``delegate_task`` background=True) consult this before
registering a watcher / dispatching a detached child, so they can refuse a
promise the channel can't keep instead of silently no-op'ing.
False for finite runtimes: sessions bound by a stateless channel (API
server, ``hermes -z``, cron — see :func:`declare_stateless_channel`) and
dispatcher-spawned Kanban workers (``HERMES_KANBAN_TASK``), which are
one-shot ``chat -q`` subprocesses whose parent disappears after the quiet
turn, so a later completion has no durable consumer. Gateway platforms,
the interactive CLI, and any path that never bound the var return True.
"""
import os
# A Kanban worker is a one-shot subprocess. Its parent session and process
# disappear after the quiet turn returns, so a completion queued later has
# no durable consumer even though an ordinary CLI session can drain that
# queue. Force tools onto their existing synchronous/polling fallbacks.
# Kanban worker: force tools onto their synchronous/polling fallbacks.
if os.environ.get("HERMES_KANBAN_TASK"):
return False
value = _SESSION_ASYNC_DELIVERY.get()
if value is _UNSET:
return True
return bool(value)
return True if value is _UNSET else bool(value)
gateway/session_db_recovery.py
+36 -57
View File
@@ -22,7 +22,7 @@ class _Unavailable:
class _HealthSource:
pass
"""Weak-keyable identity for one cache's health entries."""
_health_lock = threading.Lock()
@@ -32,29 +32,26 @@ _health_states: weakref.WeakKeyDictionary[_HealthSource, dict[Path, str]] = (
def _publish_health(source: _HealthSource, path: Path, state: str) -> None:
"""Publish one privacy-safe aggregate for every live gateway DB cache."""
"""Publish one privacy-safe aggregate (no paths, no errors) across all live caches."""
with _health_lock:
states = _health_states.setdefault(source, {})
states[path] = state
all_states = [value for item in _health_states.values() for value in item.values()]
if "retrying" in all_states:
aggregate = "retrying"
elif "unavailable" in all_states:
aggregate = "unavailable"
else:
aggregate = "ok"
_health_states.setdefault(source, {})[path] = state
all_states = {value for item in _health_states.values() for value in item.values()}
aggregate = next((s for s in ("retrying", "unavailable") if s in all_states), "ok")
try:
from gateway.status import write_runtime_status
write_runtime_status(session_store={"status": aggregate})
except Exception:
# Runtime health is diagnostic only; persistence must not depend on it.
pass
pass # Runtime health is diagnostic only; persistence must not depend on it.
class RecoverableHandleCache:
"""Cache handles by path while allowing failed opens to heal in-process."""
"""Cache handles by path while allowing failed opens to heal in-process.
Opens run OUTSIDE ``lock`` (single-flight per path via ``in_flight``); a
``close_all`` bumps ``_generation`` so any open that completes afterwards is
treated as stale and rejected rather than resurrecting a drained cache.
"""
def __init__(
self,
@@ -69,14 +66,16 @@ class RecoverableHandleCache:
self.lock = lock if lock is not None else threading.Lock()
self._clock = clock
self._initial_retry_delay = max(0.0, float(initial_retry_delay))
self._max_retry_delay = max(
self._initial_retry_delay, float(max_retry_delay)
)
self._max_retry_delay = max(self._initial_retry_delay, float(max_retry_delay))
self._unavailable: dict[Path, _Unavailable] = {}
self._health_source = _HealthSource()
self._generation = 0
self._close_rejected: Callable[[Any], None] | None = None
def _is_stale(self, path: Path, unavailable: _Unavailable, generation: int) -> bool:
"""Caller holds ``lock``: True when close_all ran or the slot was replaced mid-open."""
return generation != self._generation or self._unavailable.get(path) is not unavailable
def get(
self,
path: Path,
@@ -86,15 +85,18 @@ class RecoverableHandleCache:
on_recovered: Callable[[], None] | None = None,
non_cacheable: Callable[[Exception], bool] | None = None,
) -> Any:
"""Return a cached handle or make one bounded, single-flight open attempt."""
"""Return a cached handle or make one bounded, single-flight open attempt.
Returns None while a retry is in flight or backing off (callers fall back).
``non_cacheable`` exceptions (e.g. a live-system guard) are re-raised
without recording a failure so the next call retries immediately.
"""
path = Path(path)
with self.lock:
if path in self.handles:
return self.handles[path]
unavailable = self._unavailable.setdefault(path, _Unavailable())
now = self._clock()
if unavailable.in_flight or now < unavailable.next_retry_at:
if unavailable.in_flight or self._clock() < unavailable.next_retry_at:
return None
unavailable.in_flight = True
was_unavailable = unavailable.failures > 0
@@ -106,46 +108,33 @@ class RecoverableHandleCache:
try:
handle = opener()
except Exception as exc:
if non_cacheable is not None and non_cacheable(exc):
with self.lock:
if (
generation == self._generation
and self._unavailable.get(path) is unavailable
):
self._unavailable.pop(path, None)
raise
uncacheable = non_cacheable is not None and non_cacheable(exc)
with self.lock:
current = self._unavailable.get(path)
stale = generation != self._generation or current is not unavailable
stale = self._is_stale(path, unavailable, generation)
if uncacheable:
if not stale:
self._unavailable.pop(path, None)
raise
if not stale:
unavailable.failures += 1
delay = min(
self._initial_retry_delay
* (2 ** min(unavailable.failures - 1, 30)),
self._initial_retry_delay * (2 ** min(unavailable.failures - 1, 30)),
self._max_retry_delay,
)
unavailable.next_retry_at = self._clock() + delay
unavailable.in_flight = False
if stale:
if raise_on_error:
raise
return None
_publish_health(self._health_source, path, "unavailable")
if not stale:
_publish_health(self._health_source, path, "unavailable")
if raise_on_error:
raise
return None
with self.lock:
stale = (
generation != self._generation
or self._unavailable.get(path) is not unavailable
)
if stale:
close_rejected = self._close_rejected
else:
stale = self._is_stale(path, unavailable, generation)
if not stale:
self.handles[path] = handle
self._unavailable.pop(path, None)
close_rejected = None
close_rejected = self._close_rejected if stale else None
if stale:
if close_rejected is not None:
try:
@@ -177,13 +166,3 @@ class RecoverableHandleCache:
if states is not None:
for path in paths:
states.pop(path, None)
def status_for(self, path: Path) -> str:
"""Return a sanitized state for tests and internal diagnostics."""
with self.lock:
if Path(path) in self.handles:
return "ok"
unavailable = self._unavailable.get(Path(path))
if unavailable is None:
return "unknown"
return "retrying" if unavailable.in_flight else "unavailable"
gateway/session_stall.py
+44 -74
View File
@@ -1,21 +1,15 @@
"""Gateway session stall notification policy (#72016 item 2).
"""Gateway session stall notification policy.
Consumes the shared activity observation contract from
``agent.session_activity`` / ``AIAgent.get_activity_summary()``
(#72039) as the **single progress source**. This module owns only the
notify-once policy for "pending inbound + stale progress"; it does not
invent a parallel progress clock from turn-start or inbound event
timestamps.
Consumes the shared activity observation contract from ``agent.session_activity``
/ ``AIAgent.get_activity_summary()`` as the **single progress source**. This
module owns only the notify-once policy for "pending inbound + stale progress";
it never derives a parallel progress clock from turn-start or inbound timestamps.
Boundaries (keep separate):
- ``gateway/shutdown_watchdog.py`` — process / event-loop liveness
- ``gateway/delivery_ledger.py`` — outbound delivery obligations
- Pending inbound here is a stall *policy gate* (queued follow-up exists),
not an outbound obligation and not a progress timestamp.
Notification / timeout / kill / retry policy stay in their own components;
the shared contract remains observation-only (timestamp + bounded
description + provenance).
Boundaries (keep separate): ``gateway/shutdown_watchdog.py`` is process /
event-loop liveness; ``gateway/delivery_ledger.py`` is outbound delivery
obligations. Pending inbound here is a stall *policy gate* (a queued follow-up
exists), not an obligation and not a progress timestamp. Timeout / kill / retry
policy stay in their own components.
"""
from __future__ import annotations
@@ -32,15 +26,13 @@ def should_emit_session_stall_notification(
already_notified: bool,
) -> bool:
"""Return True when a stall warning should be sent for this session."""
if timeout_seconds <= 0:
return False
if not has_pending_inbound:
return False
if already_notified:
return False
if idle_seconds is None:
return False
return idle_seconds >= timeout_seconds
return (
timeout_seconds > 0
and has_pending_inbound
and not already_notified
and idle_seconds is not None
and idle_seconds >= timeout_seconds
)
def should_clear_session_stall_notification(
@@ -50,23 +42,27 @@ def should_clear_session_stall_notification(
has_pending_inbound: bool,
) -> bool:
"""Return True when a prior stall notice may be cleared (episode ended)."""
if not has_pending_inbound:
return True
if timeout_seconds <= 0:
if not has_pending_inbound or timeout_seconds <= 0:
return True
# Unknown progress: hold the latch. Do not treat observation gaps as recovery.
if idle_seconds is None:
return False
return idle_seconds < timeout_seconds
return idle_seconds is not None and idle_seconds < timeout_seconds
def format_session_stall_notification(idle_seconds: float) -> str:
"""User-facing stall warning (ASCII minutes; matches issue #72016 copy)."""
"""User-facing stall warning (ASCII minutes)."""
mins = max(1, int(idle_seconds // 60))
return (
f"⚠️ Agent session appears stalled (last activity {mins} min ago). "
f"Try /new to reset."
)
return f"⚠️ Agent session appears stalled (last activity {mins} min ago). Try /new to reset."
def _finite_float(value: Any) -> Optional[float]:
"""``value`` as a finite float, or None (bools are rejected as non-numeric)."""
if value is None or isinstance(value, bool):
return None
try:
result = float(value)
except (TypeError, ValueError):
return None
return result if math.isfinite(result) else None
def resolve_session_idle_seconds_from_activity(
@@ -74,52 +70,26 @@ def resolve_session_idle_seconds_from_activity(
*,
now: Optional[float] = None,
) -> Optional[float]:
"""Idle seconds from a shared activity snapshot only (#72039 contract).
"""Idle seconds from a shared activity snapshot only.
Prefers ``seconds_since_activity`` when present and finite; otherwise
derives from ``last_activity_at`` / ``last_activity_ts``. Returns
``None`` when there is no usable progress timestamp — callers must
not fall back to turn-start or pending-inbound clocks.
Prefers ``seconds_since_activity`` when present and finite; otherwise derives
from ``last_activity_at`` / ``last_activity_ts``. Returns None when there is
no usable progress timestamp — callers must not fall back to turn-start or
pending-inbound clocks.
"""
if not activity:
return None
elapsed = activity.get("seconds_since_activity")
if isinstance(elapsed, bool):
elapsed = None
if elapsed is not None:
try:
idle = float(elapsed)
except (TypeError, ValueError):
idle = None
else:
if math.isfinite(idle):
if idle < 0:
return 0.0
return idle
# Non-finite: fall through to last_activity_at / last_activity_ts
idle = _finite_float(activity.get("seconds_since_activity"))
if idle is not None:
return max(0.0, idle)
ts = activity.get("last_activity_at")
if ts is None:
ts = activity.get("last_activity_ts")
if ts is None:
when = _finite_float(activity.get("last_activity_ts") if ts is None else ts)
if when is None:
return None
if isinstance(ts, bool):
return None
try:
when = float(ts)
except (TypeError, ValueError):
return None
if not math.isfinite(when):
return None
if now is None:
import time as _time
clock = float(_time.time())
else:
clock = float(now)
idle = clock - when
if idle < 0:
return 0.0
return idle
now = _time.time()
return max(0.0, float(now) - when)
gateway/session_state.py
+180 -330
View File
@@ -1,85 +1,66 @@
"""Per-session gateway state consolidated into one container.
GatewayRunner historically carried ~19 separate ``Dict[str, ...]`` attributes
keyed by session_key, each with its own ad-hoc lifecycle. Three failure
classes grew out of that shape:
keyed by session_key, each with an ad-hoc lifecycle. That shape bred three
bug classes, all structurally closed here:
1. Boundary drift — every conversation boundary carried a hand-copied
pop-list that went stale when a new dict was added (#48031, #58403,
#10702, #35809). Mitigated by the ``_CONVERSATION_SCOPED_STATE`` registry,
now structurally fixed: the fields live in one ``ConversationState``
dataclass with a single ``clear()``.
2. Turn-release drift — ad-hoc ``del self._running_agents[key]`` sites that
popped different subsets of the turn dicts. Mitigated by
``_release_running_agent_state``, now ``TurnState.clear()``.
3. Wholesale-reset races — lazy-init paths like
``self._session_reasoning_overrides = {}`` replaced the ENTIRE dict,
discarding concurrent sessions' entries when raced. Structurally
impossible now: state is per-session, resets touch one field of one
``SessionState``.
1. Boundary drift — hand-copied pop-lists at conversation boundaries went
stale when a dict was added. Now one ``ConversationState.clear()``.
2. Turn-release drift — ad-hoc ``del self._running_agents[key]`` sites popped
different subsets of the turn dicts. Now ``TurnState.clear()``.
3. Wholesale-reset races — lazy-init ``self._x = {}`` replaced the ENTIRE dict,
discarding concurrent sessions' entries. Resets now touch one field of
one ``SessionState``.
Scopes (placement follows where each dict is CLEARED today):
- ``SessionState.turn`` — reset at end of every running turn.
- ``SessionState.conversation`` — reset at conversation boundaries
(/new, /resume, auto-reset, expiry, compression-exhausted reset).
- ``SessionState.persistent`` — own lifecycles (approval resolution, update
prompt answer, native-image consumption); ``run_generation`` is monotonic
and NEVER reset (#28686).
Scopes follow where each dict was CLEARED: ``turn`` resets at the end of every
running turn; ``conversation`` at conversation boundaries (/new, /resume,
auto-reset, expiry, compression-exhausted reset); ``persistent`` fields have
their own lifecycles and ``run_generation`` is monotonic and NEVER reset.
Entries in ``GatewayRunner._sessions`` are never evicted (matching the old
dicts, most of which also leaked empty/stale entries for dead sessions —
see the migration table in the consolidating commit). Follow-up work may
add eviction of fully-default SessionStates.
dicts, which also leaked empty/stale entries for dead sessions). Eviction of
fully-default SessionStates is possible follow-up work.
"""
from __future__ import annotations
from collections.abc import MutableMapping
from dataclasses import dataclass, field
from typing import Any, Callable, Dict, Iterator, List, Optional, Tuple
from typing import Any, Callable, Dict, Iterator, List, NamedTuple, Optional, Tuple
# Presence-sensitive sentinel: /fast stores "priority" or None (explicit
# normal), so key PRESENCE — not value truthiness — decides whether the
# override applies. ``_UNSET_TIER`` means "no override recorded".
_UNSET_TIER = object()
# Public alias for callers outside this module.
SERVICE_TIER_UNSET = _UNSET_TIER
SERVICE_TIER_UNSET = _UNSET_TIER # public alias
@dataclass
class TurnState:
"""State scoped to one running gateway turn.
Cleared by ``GatewayRunner._release_running_agent_state`` (via
``clear()``) at every site that ends a running turn. ``lease_token`` /
``lease_generation`` are deliberately NOT cleared here — they are owned
by ``_release_turn_lease`` (#64934), which must release the registry
lease exactly once per acquiring turn.
``clear()`` runs at every site that ends a running turn. ``lease_token`` /
``lease_generation`` are deliberately NOT cleared by it — they are owned by
``_release_turn_lease``, which must release the registry lease exactly once
per acquiring turn.
"""
# Running AIAgent instance (or _AGENT_PENDING_SENTINEL); None = idle.
agent: Any = None
# Turn start timestamp (0.0 = not running).
started_ts: float = 0.0
# Cross-process active-session slot lease (None = none held).
lease: Any = None
# Last busy-ack timestamp (debounce; 0.0 = never acked).
busy_ack_ts: float = 0.0
# Held turn-lease token + the run generation that acquired it. The old
# ``_turn_lease_tokens`` dict was keyed by (session_key, generation) so a
# stale unwind could never free a newer turn's lease; the pair encoding
# preserves that: release/rebind only match when generation is current.
started_ts: float = 0.0 # 0.0 = not running
lease: Any = None # cross-process active-session slot lease
busy_ack_ts: float = 0.0 # debounce; 0.0 = never acked
# Held turn-lease token + the run generation that acquired it. The pair
# replaces the old (session_key, generation)-keyed dict so a stale unwind
# can never free a newer turn's lease: release/rebind only match when the
# generation is current.
lease_token: Any = None
lease_generation: Optional[int] = None
def clear(self) -> None:
"""Reset the per-turn slot (agent / start ts / lease / busy-ack).
Mirrors the exact clear set of the old _release_running_agent_state:
_running_agents, _running_agents_ts, _active_session_leases (popped
by the caller so it can call ``lease.release()``), _busy_ack_ts.
The caller pops ``lease`` first so it can call ``lease.release()``.
"""
self.agent = None
self.started_ts = 0.0
@@ -93,78 +74,45 @@ class ConversationState:
# /model per-session override (model/provider/api_key/base_url/api_mode).
model_override: Optional[Dict[str, Any]] = None
# /model --once restore snapshot.
one_turn_restore: Optional[Dict[str, Any]] = None
# /reasoning per-session override.
reasoning_override: Optional[Dict[str, Any]] = None
one_turn_restore: Optional[Dict[str, Any]] = None # /model --once snapshot
reasoning_override: Optional[Dict[str, Any]] = None # /reasoning override
# /fast per-session override: "priority" or None; _UNSET_TIER = absent.
service_tier_override: Any = _UNSET_TIER
# Last successfully-resolved non-empty model (#35314 recovery).
last_resolved_model: str = ""
# /queue overflow FIFO (adapter slot holds the head).
queued_events: List[Any] = field(default_factory=list)
# Per-turn must-deliver sidecar notes (one-shot).
sidecar_notes: List[str] = field(default_factory=list)
# Pinned session-context bytes: (change_key, text).
ephemeral_pin: Optional[Tuple[Any, ...]] = None
# Last voice-channel context delivered (None = never delivered).
vc_last: Optional[str] = None
last_resolved_model: str = "" # last successfully-resolved non-empty model
queued_events: List[Any] = field(default_factory=list) # /queue overflow FIFO (adapter slot holds the head)
sidecar_notes: List[str] = field(default_factory=list) # one-shot must-deliver notes
ephemeral_pin: Optional[Tuple[Any, ...]] = None # pinned session-context bytes: (change_key, text)
vc_last: Optional[str] = None # last voice-channel context delivered
def clear(self) -> None:
"""Reset every conversation-scoped field to its default.
The structural successor of the ``_CONVERSATION_SCOPED_STATE``
pop-loop: adding a field here means every boundary clears it
automatically.
"""
self.model_override = None
self.one_turn_restore = None
self.reasoning_override = None
self.service_tier_override = _UNSET_TIER
self.last_resolved_model = ""
self.queued_events = []
self.sidecar_notes = []
self.ephemeral_pin = None
self.vc_last = None
"""Reset every field to its default — adding a field here means every
conversation boundary clears it automatically."""
self.__dict__.update(ConversationState().__dict__)
@dataclass
class PersistentState:
"""State with its own lifecycle — NOT cleared by turn or boundary resets
wholesale (approvals/update prompts ARE cleared by the boundary
*security* funnel, but individually, matching the old behavior)."""
"""State with its own lifecycle — NOT cleared wholesale by turn or boundary
resets (approvals/update prompts ARE cleared by the boundary *security*
funnel, but individually)."""
# Pending exec approval ({"command": ..., "pattern_key": ...}).
approvals: Optional[Dict[str, Any]] = None
# /update prompt awaiting a user response.
update_prompt_pending: bool = False
# Image paths staged for native (inline) attachment; consumed one-shot.
native_image_paths: List[str] = field(default_factory=list)
# Legacy runner-level pending message text (write-mostly; flushed to
# disk on shutdown — see #72680). NOTE: distinct from the adapter-level
# ``_pending_messages`` (Dict[str, MessageEvent]) in gateway/base.py,
# which is a different store that happens to share the old name.
approvals: Optional[Dict[str, Any]] = None # {"command": ..., "pattern_key": ...}
update_prompt_pending: bool = False # /update prompt awaiting a reply
native_image_paths: List[str] = field(default_factory=list) # consumed one-shot
# Legacy runner-level pending message text (write-mostly; flushed to disk on
# shutdown). Distinct from the adapter-level ``_pending_messages``
# (Dict[str, MessageEvent]) in gateway/base.py, which shares the old name.
pending_command_text: Optional[str] = None
# Monotonic run-generation counter (#28686). NEVER reset: clearing it
# would break stale-run detection.
# Monotonic run-generation counter. NEVER reset: stale-run detection depends on it.
run_generation: int = 0
# Consecutive session-hygiene compression failures for this session
# (#79624). The in-agent compressor escalates repeat timeouts via
# ContextCompressor._consecutive_timeout_failures, but hygiene builds a
# FRESH AIAgent per run and bind_session_state() zeroes that counter, so
# the in-agent ladder is structurally unreachable from the gateway.
# Tracking the streak here — outside the per-run agent — lets hygiene
# escalate its cooldown instead of retrying on a flat interval forever.
# Reset on a successful compression, not by turn/boundary resets.
#
# PROCESS-LOCAL, deliberately: `PersistentState` means "survives turn and
# boundary resets", NOT "survives a restart" — this field has no disk flush
# (unlike `pending_command_text` above, #72680), so a gateway restart drops
# escalation back to rung 1 while the DB-backed deadline itself survives
# (#74136). Keying on `session_key` rather than `session_id` is what buys
# correctness across compaction ROTATION (the sid changes, the chat does
# not). gateway.run mirrors this value to the DB keyed by session_key so
# the same semantics also survive gateway restarts.
# Consecutive session-hygiene compression failures. The in-agent compressor's
# own timeout ladder is unreachable from the gateway (hygiene builds a FRESH
# AIAgent per run and bind_session_state() zeroes that counter), so the streak
# lives here and lets hygiene escalate its cooldown. Reset on a successful
# compression only. PROCESS-LOCAL by design: no disk flush, so a restart drops
# escalation to rung 1 while the DB-backed deadline survives; gateway.run
# mirrors it to the DB keyed by session_key (not session_id) so it also holds
# across compaction ROTATION, where the sid changes but the chat does not.
hygiene_failure_streak: int = 0
@@ -181,106 +129,44 @@ class SessionState:
# Legacy dict-view adapters.
#
# Dozens of tests construct bare runners (object.__new__) and read/write the
# old dict attributes directly (``runner._running_agents = {}``,
# ``assert key in runner._pending_approvals``...). These views keep that
# surface working: each is a LIVE MutableMapping over one SessionState field
# across all sessions. Production code in gateway/run.py accesses state via
# ``self._session_state(key).<scope>.<field>`` — the views exist for tests
# and for the thin remaining mixin/adapter call sites documented in the
# consolidating commit.
# old dict attributes directly (``runner._running_agents = {}``, ``assert key
# in runner._pending_approvals``). Each view is a LIVE MutableMapping over one
# SessionState field across all sessions. Production code in gateway/run.py
# uses ``self._session_state(key).<scope>.<field>``.
# ---------------------------------------------------------------------------
class _FieldSpec:
"""Describes one legacy dict: scope attr, field name, absence test."""
class _FieldSpec(NamedTuple):
"""One legacy dict: scope attr, field name, default factory, presence test."""
__slots__ = ("scope", "name", "default", "is_present")
def __init__(
self,
scope: str,
name: str,
default: Callable[[], Any],
is_present: Callable[[Any], bool],
) -> None:
self.scope = scope
self.name = name
self.default = default
self.is_present = is_present
scope: str
name: str
default: Callable[[], Any]
is_present: Callable[[Any], bool]
def _present_not_none(v: Any) -> bool:
return v is not None
def _spec(scope: str, name: str, default: Any) -> _FieldSpec:
"""``default`` is either a type (presence = truthiness) or a sentinel value
such as ``None`` / ``_UNSET_TIER`` (presence = ``is not`` sentinel)."""
if isinstance(default, type):
return _FieldSpec(scope, name, default, bool)
return _FieldSpec(scope, name, lambda: default, lambda v: v is not default)
def _present_nonzero(v: Any) -> bool:
return bool(v)
class _RunnerView(MutableMapping):
"""Shared plumbing: live view over ``runner._sessions``, dict-comparable."""
__slots__ = ("_runner",)
def _present_not_unset(v: Any) -> bool:
return v is not _UNSET_TIER
class SessionFieldView(MutableMapping):
"""Live dict-like view of one SessionState field across sessions."""
__slots__ = ("_runner", "_spec")
def __init__(self, runner: Any, spec: _FieldSpec) -> None:
def __init__(self, runner: Any) -> None:
self._runner = runner
self._spec = spec
# -- internals --------------------------------------------------------
def _sessions(self) -> Dict[str, SessionState]:
return self._runner.__dict__.get("_sessions") or {}
def _value(self, state: SessionState) -> Any:
return getattr(getattr(state, self._spec.scope), self._spec.name)
# -- mapping protocol ---------------------------------------------------
def __getitem__(self, key: str) -> Any:
state = self._sessions().get(key)
if state is None:
raise KeyError(key)
value = self._value(state)
if not self._spec.is_present(value):
raise KeyError(key)
return value
def __setitem__(self, key: str, value: Any) -> None:
state = self._runner._session_state(key)
setattr(getattr(state, self._spec.scope), self._spec.name, value)
def __delitem__(self, key: str) -> None:
state = self._sessions().get(key)
if state is None or not self._spec.is_present(self._value(state)):
raise KeyError(key)
setattr(
getattr(state, self._spec.scope),
self._spec.name,
self._spec.default(),
)
def __iter__(self) -> Iterator[str]:
for key, state in list(self._sessions().items()):
if self._spec.is_present(self._value(state)):
yield key
def __len__(self) -> int:
return sum(1 for _ in self)
def __contains__(self, key: object) -> bool:
state = self._sessions().get(key) # type: ignore[arg-type]
return state is not None and self._spec.is_present(self._value(state))
def clear(self) -> None: # avoid MutableMapping's popitem loop
for state in list(self._sessions().values()):
setattr(
getattr(state, self._spec.scope),
self._spec.name,
self._spec.default(),
)
# Mapping doesn't provide __eq__; tests compare against plain dicts.
def __eq__(self, other: object) -> bool:
if isinstance(other, (dict, MutableMapping)):
@@ -291,185 +177,149 @@ class SessionFieldView(MutableMapping):
result = self.__eq__(other)
return NotImplemented if result is NotImplemented else not result
class SessionFieldView(_RunnerView):
"""Live dict-like view of one SessionState field across sessions."""
__slots__ = ("_spec",)
def __init__(self, runner: Any, spec: _FieldSpec) -> None:
super().__init__(runner)
self._spec = spec
def _value(self, state: SessionState) -> Any:
return getattr(getattr(state, self._spec.scope), self._spec.name)
def _set(self, state: SessionState, value: Any) -> None:
setattr(getattr(state, self._spec.scope), self._spec.name, value)
def __getitem__(self, key: str) -> Any:
state = self._sessions().get(key)
if state is None or not self._spec.is_present(value := self._value(state)):
raise KeyError(key)
return value
def __setitem__(self, key: str, value: Any) -> None:
self._set(self._runner._session_state(key), value)
def __delitem__(self, key: str) -> None:
state = self._sessions().get(key)
if state is None or not self._spec.is_present(self._value(state)):
raise KeyError(key)
self._set(state, self._spec.default())
def __iter__(self) -> Iterator[str]:
for key, state in list(self._sessions().items()):
if self._spec.is_present(self._value(state)):
yield key
def __contains__(self, key: object) -> bool:
state = self._sessions().get(key) # type: ignore[arg-type]
return state is not None and self._spec.is_present(self._value(state))
def clear(self) -> None: # avoid MutableMapping's popitem loop
for state in list(self._sessions().values()):
self._set(state, self._spec.default())
def __repr__(self) -> str: # pragma: no cover - debug aid
return f"SessionFieldView({self._spec.scope}.{self._spec.name}, {dict(self.items())!r})"
class TurnLeaseTokenView(MutableMapping):
class TurnLeaseTokenView(_RunnerView):
"""Legacy view of ``_turn_lease_tokens``: keyed by (session_key, generation).
The pair now lives on ``TurnState.lease_token`` / ``lease_generation``;
at most one held token exists per session key (the lease registry
serializes acquisition per session), so the single slot is equivalent to
the old tuple-keyed dict.
The pair lives on ``TurnState.lease_token`` / ``lease_generation``; the lease
registry serializes acquisition per session, so at most one held token
exists per session key and the single slot equals the old tuple-keyed dict.
"""
__slots__ = ("_runner",)
__slots__ = ()
def __init__(self, runner: Any) -> None:
self._runner = runner
def _sessions(self) -> Dict[str, SessionState]:
return self._runner.__dict__.get("_sessions") or {}
@staticmethod
def _split(key: Any) -> Tuple[str, int]:
def _held(self, key: Any) -> Tuple[Any, SessionState]:
"""Return (session_key, state) for a currently-held (key, gen) or raise KeyError."""
if not isinstance(key, tuple) or len(key) != 2:
raise KeyError(key)
return key[0], key[1]
state = self._sessions().get(key[0])
if state is None or state.turn.lease_token is None or state.turn.lease_generation != key[1]:
raise KeyError(key)
return key[0], state
def __getitem__(self, key: Any) -> Any:
session_key, generation = self._split(key)
state = self._sessions().get(session_key)
if (
state is None
or state.turn.lease_token is None
or state.turn.lease_generation != generation
):
raise KeyError(key)
return state.turn.lease_token
return self._held(key)[1].turn.lease_token
def __setitem__(self, key: Any, value: Any) -> None:
session_key, generation = self._split(key)
state = self._runner._session_state(session_key)
state.turn.lease_token = value
state.turn.lease_generation = generation
if not isinstance(key, tuple) or len(key) != 2:
raise KeyError(key)
turn = self._runner._session_state(key[0]).turn
turn.lease_token, turn.lease_generation = value, key[1]
def __delitem__(self, key: Any) -> None:
session_key, generation = self._split(key)
state = self._sessions().get(session_key)
if (
state is None
or state.turn.lease_token is None
or state.turn.lease_generation != generation
):
raise KeyError(key)
state.turn.lease_token = None
state.turn.lease_generation = None
turn = self._held(key)[1].turn
turn.lease_token = turn.lease_generation = None
def __iter__(self) -> Iterator[Tuple[str, Any]]:
for key, state in list(self._sessions().items()):
if state.turn.lease_token is not None:
yield (key, state.turn.lease_generation)
def __len__(self) -> int:
return sum(1 for _ in self)
def __eq__(self, other: object) -> bool:
if isinstance(other, (dict, MutableMapping)):
return dict(self.items()) == dict(other)
return NotImplemented
def __ne__(self, other: object) -> bool:
result = self.__eq__(other)
return NotImplemented if result is NotImplemented else not result
def clear(self) -> None: # avoid MutableMapping's popitem loop
for key in list(self):
del self[key]
# One spec per legacy dict attribute (see the consolidating commit's
# migration table for declared-line / clear-path provenance).
# One spec per legacy dict attribute.
LEGACY_FIELD_SPECS: Dict[str, _FieldSpec] = {
"_running_agents": _FieldSpec("turn", "agent", lambda: None, _present_not_none),
"_running_agents_ts": _FieldSpec("turn", "started_ts", float, _present_nonzero),
"_active_session_leases": _FieldSpec("turn", "lease", lambda: None, _present_not_none),
"_busy_ack_ts": _FieldSpec("turn", "busy_ack_ts", float, _present_nonzero),
"_session_model_overrides": _FieldSpec(
"conversation", "model_override", lambda: None, _present_not_none
),
"_pending_one_turn_model_restores": _FieldSpec(
"conversation", "one_turn_restore", lambda: None, _present_not_none
),
"_session_reasoning_overrides": _FieldSpec(
"conversation", "reasoning_override", lambda: None, _present_not_none
),
"_session_service_tier_overrides": _FieldSpec(
"conversation", "service_tier_override", lambda: _UNSET_TIER, _present_not_unset
),
"_last_resolved_model": _FieldSpec(
"conversation", "last_resolved_model", str, _present_nonzero
),
"_queued_events": _FieldSpec("conversation", "queued_events", list, _present_nonzero),
"_pending_turn_sidecar_notes": _FieldSpec(
"conversation", "sidecar_notes", list, _present_nonzero
),
"_session_ephemeral_pin": _FieldSpec(
"conversation", "ephemeral_pin", lambda: None, _present_not_none
),
"_session_vc_last": _FieldSpec(
"conversation", "vc_last", lambda: None, _present_not_none
),
"_pending_approvals": _FieldSpec(
"persistent", "approvals", lambda: None, _present_not_none
),
"_update_prompt_pending": _FieldSpec(
"persistent", "update_prompt_pending", bool, _present_nonzero
),
"_pending_native_image_paths_by_session": _FieldSpec(
"persistent", "native_image_paths", list, _present_nonzero
),
"_pending_messages": _FieldSpec(
"persistent", "pending_command_text", lambda: None, _present_not_none
),
"_session_run_generation": _FieldSpec(
"persistent", "run_generation", int, _present_nonzero
),
"_running_agents": _spec("turn", "agent", None),
"_running_agents_ts": _spec("turn", "started_ts", float),
"_active_session_leases": _spec("turn", "lease", None),
"_busy_ack_ts": _spec("turn", "busy_ack_ts", float),
"_session_model_overrides": _spec("conversation", "model_override", None),
"_pending_one_turn_model_restores": _spec("conversation", "one_turn_restore", None),
"_session_reasoning_overrides": _spec("conversation", "reasoning_override", None),
"_session_service_tier_overrides": _spec("conversation", "service_tier_override", _UNSET_TIER),
"_last_resolved_model": _spec("conversation", "last_resolved_model", str),
"_queued_events": _spec("conversation", "queued_events", list),
"_pending_turn_sidecar_notes": _spec("conversation", "sidecar_notes", list),
"_session_ephemeral_pin": _spec("conversation", "ephemeral_pin", None),
"_session_vc_last": _spec("conversation", "vc_last", None),
"_pending_approvals": _spec("persistent", "approvals", None),
"_update_prompt_pending": _spec("persistent", "update_prompt_pending", bool),
"_pending_native_image_paths_by_session": _spec("persistent", "native_image_paths", list),
"_pending_messages": _spec("persistent", "pending_command_text", None),
"_session_run_generation": _spec("persistent", "run_generation", int),
}
def legacy_dict_property(attr_name: str) -> property:
"""Build a legacy dict-shaped @property for one migrated attribute.
def _legacy_property(make_view: Callable[[Any], MutableMapping], doc: str) -> property:
"""Dict-shaped @property over a live view.
Getter returns a live :class:`SessionFieldView`; setter accepts a plain
dict (the ubiquitous test pattern ``runner._X = {...}``), resetting the
field on every known session and then applying the given entries.
Getter returns the view; setter accepts a plain dict (the ubiquitous test
pattern ``runner._X = {...}``), resetting the field on every known session
and then applying the given entries; ``del runner._X`` (older tests
simulating a runner without the attribute) means "no entries".
"""
spec = LEGACY_FIELD_SPECS[attr_name]
def fget(self: Any) -> SessionFieldView:
return SessionFieldView(self, spec)
def fset(self: Any, mapping: Optional[Dict[str, Any]]) -> None:
view = SessionFieldView(self, spec)
def fset(self: Any, mapping: Optional[Dict[Any, Any]]) -> None:
view = make_view(self)
view.clear()
for key, value in (mapping or {}).items():
view[key] = value
def fdel(self: Any) -> None:
# ``del runner._busy_ack_ts`` in older tests simulated a runner
# without the attribute; the closest equivalent is "no entries".
SessionFieldView(self, spec).clear()
return property(make_view, fset, lambda self: make_view(self).clear(), doc=doc)
return property(
fget,
fset,
fdel,
doc=(
f"Legacy dict view over SessionState.{spec.scope}.{spec.name} "
"(kept for tests that access the pre-SessionState attribute)."
),
def legacy_dict_property(attr_name: str) -> property:
"""Legacy dict-shaped @property for one migrated attribute."""
spec = LEGACY_FIELD_SPECS[attr_name]
return _legacy_property(
lambda self: SessionFieldView(self, spec),
f"Legacy dict view over SessionState.{spec.scope}.{spec.name} "
"(kept for tests that access the pre-SessionState attribute).",
)
def legacy_lease_token_property() -> property:
"""Legacy (session_key, generation)-keyed view of held turn-lease tokens."""
def fget(self: Any) -> TurnLeaseTokenView:
return TurnLeaseTokenView(self)
def fset(self: Any, mapping: Optional[Dict[Tuple[str, int], Any]]) -> None:
view = TurnLeaseTokenView(self)
for key in list(view):
del view[key]
for key, value in (mapping or {}).items():
view[key] = value
def fdel(self: Any) -> None:
view = TurnLeaseTokenView(self)
for key in list(view):
del view[key]
return property(
fget,
fset,
fdel,
doc="Legacy (session_key, generation)-keyed turn-lease token view.",
return _legacy_property(
TurnLeaseTokenView, "Legacy (session_key, generation)-keyed turn-lease token view."
)
gateway/slash_commands.py
+432 -812
View File
File diff suppressed because it is too large Load Diff
gateway/status_phrases.py
+42 -73
View File
@@ -1,17 +1,12 @@
"""Human-friendly generic gateway status phrases.
These helpers deliberately avoid relaying raw model scratch text. They turn
Hermes' long-running gateway status surface into short status lines suitable
for chat surfaces.
Turns Hermes' long-running gateway status surface into short chat-safe lines
without relaying raw model scratch text: only configured phrase strings are
used — tool args, commands, previews, and reasoning are never interpolated.
Built-in defaults live in ``gateway/assets/status_phrases.yaml``. Users can add
portable, profile-relative phrase catalogs under ``HERMES_HOME`` either by using
conventional paths::
~/.hermes/status_phrases.yaml
~/.hermes/status_phrases/*.yaml
or by pointing config at a relative file/directory::
profile-relative catalogs under ``HERMES_HOME`` via the conventional paths
``status_phrases.yaml`` / ``status_phrases/*.yaml`` or via config::
display:
status_phrases:
@@ -19,10 +14,7 @@ or by pointing config at a relative file/directory::
mode: append # append (default) or replace
Absolute paths and ``..`` escapes are ignored on purpose so config stays
profile-portable and does not accidentally read arbitrary files.
Only configured phrase strings are used; raw tool args, commands, previews, and
reasoning text are never interpolated into the returned phrase.
profile-portable and cannot read arbitrary files.
"""
from __future__ import annotations
@@ -36,13 +28,14 @@ import yaml
from hermes_constants import get_hermes_home
# These are Hermes UI surfaces, not app/vendor/domain buckets. Keep this
# long-running-only: regular tool/thinking/interim chatter is intentionally not
# rewritten into generic placeholders because that gets noisy fast in chat.
# Hermes UI surfaces, not app/vendor buckets. Long-running-only: regular
# tool/thinking/interim chatter is deliberately not rewritten (too noisy in chat).
_STATUS_SURFACES = ("status", "generic")
_MAX_CUSTOM_PHRASES_PER_SURFACE = 80
_MAX_PHRASE_CHARS = 160
_CONVENTIONAL_RELATIVE_PATHS = ("status_phrases.yaml", "status_phrases")
_YAML_SUFFIXES = {".yaml", ".yml"}
_CONFIG_KEYS = ("generic_status_phrases", "status_phrases") # legacy alias first
_FALLBACK_PHRASES: dict[str, list[str]] = {
"status": ["still on it", "still working through it", "waiting for the result"],
@@ -54,25 +47,20 @@ def _clean_phrase_list(value: Any) -> list[str]:
if not isinstance(value, list):
return []
cleaned: list[str] = []
seen: set[str] = set()
for item in value[:_MAX_CUSTOM_PHRASES_PER_SURFACE]:
phrase = str(item or "").strip()
if not phrase or len(phrase) > _MAX_PHRASE_CHARS or phrase in seen:
continue
cleaned.append(phrase)
seen.add(phrase)
if phrase and len(phrase) <= _MAX_PHRASE_CHARS and phrase not in cleaned:
cleaned.append(phrase)
return cleaned
def _merge_phrase_mapping(catalog: dict[str, list[str]], section: Mapping[str, Any], *, inherited_mode: str | None = None) -> None:
mode = str(section.get("mode") or inherited_mode or "append").strip().lower()
replace = mode == "replace"
replace = str(section.get("mode") or inherited_mode or "append").strip().lower() == "replace"
phrase_map = section.get("phrases") if isinstance(section.get("phrases"), Mapping) else section
for surface in _STATUS_SURFACES:
phrases = _clean_phrase_list(phrase_map.get(surface) if isinstance(phrase_map, Mapping) else None)
if not phrases:
continue
catalog[surface] = phrases if replace else [*catalog.get(surface, []), *phrases]
if phrases:
catalog[surface] = phrases if replace else [*catalog.get(surface, []), *phrases]
def _merge_phrase_file(catalog: dict[str, list[str]], path: Path, *, inherited_mode: str | None = None) -> None:
@@ -101,56 +89,43 @@ def _relative_path_under(base_dir: Path, raw_path: Any) -> Path | None:
def _iter_phrase_files(path: Path) -> list[Path]:
if path.is_file() and path.suffix.lower() in {".yaml", ".yml"}:
if path.is_file() and path.suffix.lower() in _YAML_SUFFIXES:
return [path]
if path.is_dir():
return sorted(
child for child in path.iterdir()
if child.is_file() and child.suffix.lower() in {".yaml", ".yml"}
)
return sorted(c for c in path.iterdir() if c.is_file() and c.suffix.lower() in _YAML_SUFFIXES)
return []
def _merge_phrase_paths(
catalog: dict[str, list[str]],
paths: Any,
*,
base_dir: Path,
inherited_mode: str | None = None,
) -> None:
def _merge_phrase_paths(catalog: dict[str, list[str]], paths: Any, *, base_dir: Path, inherited_mode: str | None = None) -> None:
if paths is None:
return
raw_paths = paths if isinstance(paths, list) else [paths]
for raw_path in raw_paths:
for raw_path in paths if isinstance(paths, list) else [paths]:
resolved = _relative_path_under(base_dir, raw_path)
if resolved is None:
continue
for phrase_file in _iter_phrase_files(resolved):
_merge_phrase_file(catalog, phrase_file, inherited_mode=inherited_mode)
if resolved is not None:
for phrase_file in _iter_phrase_files(resolved):
_merge_phrase_file(catalog, phrase_file, inherited_mode=inherited_mode)
def _copy_catalog(catalog: Mapping[str, list[str]]) -> dict[str, list[str]]:
return {surface: list(phrases) for surface, phrases in catalog.items()}
def _load_builtin_catalog() -> dict[str, list[str]]:
catalog = {surface: list(phrases) for surface, phrases in _FALLBACK_PHRASES.items()}
catalog_path = Path(__file__).resolve().parent / "assets" / "status_phrases.yaml"
_merge_phrase_file(catalog, catalog_path, inherited_mode="replace")
catalog = _copy_catalog(_FALLBACK_PHRASES)
_merge_phrase_file(catalog, Path(__file__).resolve().parent / "assets" / "status_phrases.yaml", inherited_mode="replace")
return catalog
_DEFAULT_PHRASES: dict[str, list[str]] = _load_builtin_catalog()
def _copy_default_catalog() -> dict[str, list[str]]:
return {surface: list(phrases) for surface, phrases in _DEFAULT_PHRASES.items()}
def _merge_phrase_config(catalog: dict[str, list[str]], section: Any, *, base_dir: Path | None = None) -> None:
"""Merge one display.status_phrases-style section into ``catalog``."""
def _merge_phrase_config(catalog: dict[str, list[str]], section: Any, *, base_dir: Path) -> None:
"""Merge one display.status_phrases-style section (files first, then inline phrases)."""
if not isinstance(section, Mapping):
return
mode = str(section.get("mode") or "append").strip().lower()
if base_dir is not None:
_merge_phrase_paths(catalog, section.get("path"), base_dir=base_dir, inherited_mode=mode)
_merge_phrase_paths(catalog, section.get("paths"), base_dir=base_dir, inherited_mode=mode)
for key in ("path", "paths"):
_merge_phrase_paths(catalog, section.get(key), base_dir=base_dir, inherited_mode=mode)
_merge_phrase_mapping(catalog, section)
@@ -162,7 +137,7 @@ def resolve_status_phrase_catalog(user_config: Mapping[str, Any] | None, platfor
alias ``generic_status_phrases``), then
``display.platforms.<platform>.status_phrases``.
"""
catalog = _copy_default_catalog()
catalog = _copy_catalog(_DEFAULT_PHRASES)
hermes_home = get_hermes_home()
_merge_phrase_paths(catalog, list(_CONVENTIONAL_RELATIVE_PATHS), base_dir=hermes_home)
@@ -170,15 +145,13 @@ def resolve_status_phrase_catalog(user_config: Mapping[str, Any] | None, platfor
if not isinstance(display, Mapping):
return catalog
_merge_phrase_config(catalog, display.get("generic_status_phrases"), base_dir=hermes_home)
_merge_phrase_config(catalog, display.get("status_phrases"), base_dir=hermes_home)
sections = [display]
platforms = display.get("platforms")
if platform_key and isinstance(platforms, Mapping):
platform_display = platforms.get(platform_key)
if isinstance(platform_display, Mapping):
_merge_phrase_config(catalog, platform_display.get("generic_status_phrases"), base_dir=hermes_home)
_merge_phrase_config(catalog, platform_display.get("status_phrases"), base_dir=hermes_home)
if platform_key and isinstance(platforms, Mapping) and isinstance(platforms.get(platform_key), Mapping):
sections.append(platforms[platform_key])
for section in sections:
for key in _CONFIG_KEYS:
_merge_phrase_config(catalog, section.get(key), base_dir=hermes_home)
return catalog
@@ -190,8 +163,7 @@ def classify_status_context(
args: Any = None,
) -> str:
"""Classify an internal gateway event into a Hermes UI-surface bucket."""
normalized = str(kind or "").strip().lower()
if normalized in {"heartbeat", "waiting", "long_running", "status"}:
if str(kind or "").strip().lower() in {"heartbeat", "waiting", "long_running", "status"}:
return "status"
return "generic"
@@ -216,11 +188,8 @@ def choose_status_phrase(
candidates = list(phrase_catalog.get(category) or phrase_catalog.get("generic") or _DEFAULT_PHRASES["generic"])
if recent:
recent_set = set(recent)
fresh = [phrase for phrase in candidates if phrase not in recent_set]
if fresh:
candidates = fresh
picker = rng or _random
phrase = picker.choice(candidates)
candidates = [p for p in candidates if p not in recent_set] or candidates
phrase = (rng or _random).choice(candidates)
if recent is not None:
recent.append(phrase)
del recent[:-6]
gateway/sticker_cache.py
+17 -57
View File
@@ -1,11 +1,7 @@
"""
Sticker description cache for Telegram.
"""Sticker description cache for Telegram.
When users send stickers, we describe them via the vision tool and cache
the descriptions keyed by file_unique_id so we don't re-analyze the same
sticker image on every send. Descriptions are concise (1-2 sentences).
Cache location: ~/.hermes/sticker_cache.json
Stickers are described via the vision tool once and cached by file_unique_id
(``~/.hermes/sticker_cache.json``) so the same image is never re-analyzed.
"""
import json
@@ -19,7 +15,7 @@ from hermes_cli.config import get_hermes_home
CACHE_PATH = get_hermes_home() / "sticker_cache.json"
# Vision prompt for describing stickers -- kept concise to save tokens
# Kept concise to save tokens.
STICKER_VISION_PROMPT = (
"Describe this sticker in 1-2 sentences. Focus on what it depicts -- "
"character, action, emotion. Be concise and objective."
@@ -27,7 +23,6 @@ STICKER_VISION_PROMPT = (
def _load_cache() -> dict:
"""Load the sticker cache from disk."""
if CACHE_PATH.exists():
try:
return json.loads(CACHE_PATH.read_text(encoding="utf-8"))
@@ -37,11 +32,9 @@ def _load_cache() -> dict:
def _save_cache(cache: dict) -> None:
"""Save the sticker cache to disk atomically."""
"""Write the cache atomically (temp file + fsync + replace)."""
CACHE_PATH.parent.mkdir(parents=True, exist_ok=True)
fd, tmp_path = tempfile.mkstemp(
dir=str(CACHE_PATH.parent), suffix=".tmp"
)
fd, tmp_path = tempfile.mkstemp(dir=str(CACHE_PATH.parent), suffix=".tmp")
try:
with os.fdopen(fd, "w", encoding="utf-8") as f:
json.dump(cache, f, indent=2, ensure_ascii=False)
@@ -57,65 +50,32 @@ def _save_cache(cache: dict) -> None:
def get_cached_description(file_unique_id: str) -> Optional[dict]:
"""
Look up a cached sticker description.
"""Return ``{description, emoji, set_name, cached_at}`` or None."""
return _load_cache().get(file_unique_id)
Returns:
dict with keys {description, emoji, set_name, cached_at} or None.
"""
def cache_sticker_description(file_unique_id: str, description: str, emoji: str = "", set_name: str = "") -> None:
"""Store a vision-generated description under Telegram's stable sticker id."""
cache = _load_cache()
return cache.get(file_unique_id)
def cache_sticker_description(
file_unique_id: str,
description: str,
emoji: str = "",
set_name: str = "",
) -> None:
"""
Store a sticker description in the cache.
Args:
file_unique_id: Telegram's stable sticker identifier.
description: Vision-generated description text.
emoji: Associated emoji (e.g. "😀").
set_name: Sticker set name if available.
"""
cache = _load_cache()
cache[file_unique_id] = {
"description": description,
"emoji": emoji,
"set_name": set_name,
"cached_at": time.time(),
}
cache[file_unique_id] = {"description": description, "emoji": emoji, "set_name": set_name, "cached_at": time.time()}
_save_cache(cache)
def build_sticker_injection(
description: str,
emoji: str = "",
set_name: str = "",
) -> str:
"""
Build the warm-style injection text for a sticker description.
Returns a string like:
[The user sent a sticker 😀 from "MyPack"~ It shows: "A cat waving" (=^.w.^=)]
def build_sticker_injection(description: str, emoji: str = "", set_name: str = "") -> str:
"""Warm-style injection text, e.g.
``[The user sent a sticker 😀 from "MyPack"~ It shows: "A cat waving" (=^.w.^=)]``.
``set_name`` is only shown together with an emoji.
"""
context = ""
if set_name and emoji:
context = f" {emoji} from \"{set_name}\""
elif emoji:
context = f" {emoji}"
return f"[The user sent a sticker{context}~ It shows: \"{description}\" (=^.w.^=)]"
def build_animated_sticker_injection(emoji: str = "") -> str:
"""
Build injection text for animated/video stickers we can't analyze.
"""
"""Injection text for animated/video stickers we can't analyze."""
if emoji:
return (
f"[The user sent an animated sticker {emoji}~ "
gateway/stream_dispatch.py
+37 -75
View File
@@ -1,21 +1,14 @@
"""Adapter-driven dispatch of structured stream events to a delivery sink.
``GatewayEventDispatcher`` is the seam Tobi asked for: the agent emits typed
events (gateway/stream_events.py), and the *adapter* decides how each one is
delivered. The dispatcher holds an adapter + the stream consumer (sink) + the
resolved per-channel presentation settings (tool-progress mode, preview length)
and routes each event through the adapter's render hooks.
``GatewayEventDispatcher`` holds an adapter, the stream consumer (sink) and the
resolved per-channel presentation settings, and routes each typed event
(gateway/stream_events.py) through the adapter's render hooks. Message events
flow into the consumer; tool events are formatted by the adapter — which may
return None to *eat* them on platforms without tool chrome — and enqueued onto
the same tool-progress queue the gateway drains, so the two paths never race.
Message/commentary/segment events flow into the consumer (native draft on
Telegram DMs, edit-in-place elsewhere). Tool events are formatted by the
adapter — which may return None to *eat* the event on platforms that can't
render tool chrome — and the rendered line is enqueued onto the same tool
progress queue the gateway already drains, so the two no longer race through
independent code paths.
This module deliberately has no platform knowledge and no asyncio: it is a thin
synchronous router callable from the agent's worker thread, exactly like the
callbacks it replaces.
No platform knowledge and no asyncio: a thin synchronous router callable from
the agent's worker thread, exactly like the callbacks it replaced.
"""
from __future__ import annotations
@@ -24,14 +17,7 @@ import logging
from typing import Any, Callable, Optional
from gateway.stream_events import (
Commentary,
GatewayNotice,
LongToolHint,
MessageChunk,
MessageStop,
StreamEvent,
ToolCallChunk,
ToolCallFinished,
Commentary, GatewayNotice, LongToolHint, MessageChunk, MessageStop, StreamEvent, ToolCallChunk,
)
logger = logging.getLogger("gateway.stream_events")
@@ -40,28 +26,16 @@ logger = logging.getLogger("gateway.stream_events")
class GatewayEventDispatcher:
"""Route typed stream events through an adapter onto a delivery sink.
Parameters
----------
adapter:
The platform adapter. Provides ``render_message_event`` and
``format_tool_event`` (BasePlatformAdapter defaults reproduce today's
behavior; adapters may override for native rendering).
sink:
The GatewayStreamConsumer for assistant-text delivery. May be None
when streaming is disabled, in which case message events are dropped
(the final response still goes out via the normal send path).
enqueue_tool_line:
Callback that places a rendered tool-progress line onto the gateway's
progress queue (the same queue ``send_progress_messages`` drains). May
be None when tool progress is disabled for this channel.
tool_mode:
Resolved tool-progress mode for this channel ("all" / "new" / "verbose"
/ "off").
preview_max_len:
Resolved ``tool_preview_length`` (0 = no cap in verbose mode).
on_long_tool / on_notice:
Optional hooks for LongToolHint / GatewayNotice events, letting the
gateway own the "should I surface this here?" decision.
adapter: provides ``render_message_event`` / ``format_tool_event``
(BasePlatformAdapter defaults reproduce legacy behavior).
sink: the GatewayStreamConsumer; None when streaming is disabled (message
events are dropped — the final response still goes out normally).
enqueue_tool_line: puts a rendered tool-progress line on the gateway's
progress queue; None when tool progress is disabled for the channel.
tool_mode: "all" / "new" / "verbose" / "off".
preview_max_len: resolved ``tool_preview_length`` (0 = no cap in verbose).
on_long_tool / on_notice: optional hooks so the gateway owns the
"should I surface this here?" decision.
"""
def __init__(
@@ -82,8 +56,7 @@ class GatewayEventDispatcher:
self.preview_max_len = preview_max_len
self._on_long_tool = on_long_tool
self._on_notice = on_notice
# "new" mode dedup — only report when the tool changes.
self._last_tool: Optional[str] = None
self._last_tool: Optional[str] = None # "new"-mode dedup
def dispatch(self, event: StreamEvent) -> None:
"""Route a single event. Never raises into the agent's worker thread."""
@@ -96,37 +69,26 @@ class GatewayEventDispatcher:
if isinstance(event, (MessageChunk, MessageStop, Commentary)):
if self.sink is not None:
self.adapter.render_message_event(event, self.sink)
return
elif isinstance(event, ToolCallChunk):
self._dispatch_tool_call(event)
elif isinstance(event, LongToolHint) and self._on_long_tool is not None:
self._on_long_tool(event)
elif isinstance(event, GatewayNotice) and self._on_notice is not None:
self._on_notice(event)
# ToolCallFinished: no chrome on completion (only "started" is rendered);
# completion only drives onboarding hints (LongToolHint).
if isinstance(event, ToolCallChunk):
if self.tool_mode == "off" or self._enqueue_tool_line is None:
return
# "new" mode: only emit when the tool changes.
if self.tool_mode == "new" and event.tool_name == self._last_tool:
return
self._last_tool = event.tool_name
line = self.adapter.format_tool_event(
event, mode=self.tool_mode, preview_max_len=self.preview_max_len,
)
# None == adapter chose to eat this event (can't render tool chrome).
if line:
self._enqueue_tool_line(line)
def _dispatch_tool_call(self, event: ToolCallChunk) -> None:
if self.tool_mode == "off" or self._enqueue_tool_line is None:
return
if isinstance(event, ToolCallFinished):
# Default: no chrome on completion (matches today — the gateway only
# rendered "started" events). Completion drives onboarding hints.
return
if isinstance(event, LongToolHint):
if self._on_long_tool is not None:
self._on_long_tool(event)
return
if isinstance(event, GatewayNotice):
if self._on_notice is not None:
self._on_notice(event)
if self.tool_mode == "new" and event.tool_name == self._last_tool:
return
self._last_tool = event.tool_name
line = self.adapter.format_tool_event(
event, mode=self.tool_mode, preview_max_len=self.preview_max_len,
)
if line: # None/"" == adapter chose to eat this event
self._enqueue_tool_line(line)
__all__ = ["GatewayEventDispatcher"]
gateway/stream_events.py
+44 -103
View File
@@ -1,35 +1,23 @@
"""Structured streaming events — the agent→gateway delivery contract.
Historically the agent drove gateway delivery through a fan of loosely-typed
callbacks (``stream_delta_callback(text)``, ``tool_progress_callback(event_type,
tool_name, preview, args)``, ``interim_assistant_callback(text)`` …) and each
gateway callback decided *both* what to render and how to send it. That
coupling is why tool-progress bubbles and the streaming draft raced each other
on Telegram, and why tool-call formatting lived agent-side even though only the
gateway knows what a given platform can render.
A small typed vocabulary naming *what happened* without prescribing *how it is
delivered*. The agent emits these from its worker thread; the gateway's stream
consumer (``GatewayStreamConsumer``) is the single sink and the platform adapter
decides rendering (Telegram may stream a native draft; iMessage may drop tool
chrome). This replaced a fan of loosely-typed callbacks whose gateway side
decided both rendering and sending — the cause of tool-progress bubbles racing
the streaming draft.
This module defines a small, typed event vocabulary that names *what happened*
without prescribing *how it is delivered*. The gateway's stream consumer
(``GatewayStreamConsumer``) is the single sink; the platform adapter decides how
to render each event (Telegram can stream a MarkdownV2 ```bash``` block as a
native draft; iMessage has no rich formatting and may collapse or drop tool
chrome). Separation of concerns: smart agent emits structured data, smart
gateway decides delivery.
Plain frozen dataclasses: no behavior, no platform knowledge, no I/O, safe to
hand across the thread/async boundary.
These are intentionally plain frozen dataclasses — no behavior, no platform
knowledge, no I/O. They are cheap to construct on the agent's worker thread and
safe to hand across the thread/async boundary into the consumer queue.
Design constraints (see hermes-agent-dev skill — message-flow + cache
invariants):
* Events describe *transport*, never *context*. Nothing here is persisted to
conversation history; what the gateway chooses to "eat" (e.g. tool chrome on
a platform that can't render it) must never diverge from the bytes stored in
the agent's message history. History is owned by the agent; these events are
a presentation-layer stream only.
* Backward compatible by construction. The gateway adapts its existing
callbacks into these events at the boundary; adapters that don't opt into
event-native rendering get identical behavior via the base-class default.
Invariants:
* Events describe *transport*, never *context*. Nothing here is persisted;
whatever the gateway chooses to "eat" must never diverge from the bytes in
the agent's message history, which the agent alone owns.
* Backward compatible by construction: the gateway adapts existing callbacks
into events at the boundary; adapters that don't opt in get identical
behavior via the base-class default.
"""
from __future__ import annotations
@@ -38,94 +26,62 @@ from dataclasses import dataclass, field
from typing import Any, Dict, Optional, Union
# ── Message (assistant text) events ──────────────────────────────────────────
@dataclass(frozen=True)
class MessageChunk:
"""A delta of streamed assistant text.
``text`` is the incremental content as it arrives from the model. The
consumer accumulates chunks and progressively renders them (native draft on
Telegram DMs, edit-in-place elsewhere). Reasoning/think-block content is
filtered upstream and never arrives as a MessageChunk.
"""
"""A delta of streamed assistant text (reasoning/think-block content is
filtered upstream and never arrives here); the consumer accumulates chunks
and renders progressively (native draft on Telegram DMs, edit-in-place
elsewhere)."""
text: str
@dataclass(frozen=True)
class MessageStop:
"""The current assistant message segment is complete.
"""The current assistant text segment is complete.
Emitted when a contiguous run of assistant text ends — either the whole
response finished, or a tool boundary interrupts the text so the next
segment should render as a fresh message *below* any tool chrome.
``final`` is True only for the terminal stop of the whole turn; an
intermediate stop (text → tool call → more text) carries ``final=False`` so
the consumer finalizes the current bubble and prepares a new segment without
treating the turn as done.
``final`` is True only for the terminal stop of the turn; an intermediate
stop (text → tool call → more text) carries ``final=False`` so the consumer
finalizes the current bubble and starts a fresh segment below tool chrome.
"""
final: bool = False
@dataclass(frozen=True)
class Commentary:
"""A complete interim assistant message emitted between tool iterations.
Example: the model says "I'll inspect the repo first." before issuing a tool
call. Unlike a MessageChunk this is already-complete text (not a delta); the
consumer renders it as its own message so it reads as a distinct beat.
"""
"""A complete interim assistant message between tool iterations (not a delta)."""
text: str
# ── Tool-call events ─────────────────────────────────────────────────────────
@dataclass(frozen=True)
class ToolCallChunk:
"""A tool invocation has started (or its in-progress state changed).
Carries the raw facts about the call — name, a short argument ``preview``,
and the full ``args`` dict — and lets the *gateway* decide presentation
(emoji, truncation, verbose vs compact, or eat it entirely on platforms that
don't show tool chrome). Previously the agent's gateway callback baked the
emoji + preview formatting in; that decision now belongs to the adapter.
"""
"""A tool invocation started. Raw facts only; the adapter decides presentation."""
tool_name: str
preview: Optional[str] = None
args: Optional[Dict[str, Any]] = None
# Monotonic per-turn index, so the consumer can correlate a finish with its
# start and so "new"-mode dedup (only report when the tool changes) works
# without the consumer tracking call order itself.
# Monotonic per-turn index: correlates a finish with its start and lets
# "new"-mode dedup work without the consumer tracking call order.
index: int = 0
@dataclass(frozen=True)
class ToolCallFinished:
"""A tool invocation completed.
"""A tool invocation completed. Tool *output* never travels here (it is history).
``duration`` is wall-clock seconds. ``ok`` reflects whether the tool
returned without raising. The gateway uses this to clear/settle a progress
bubble and to drive one-time onboarding hints (e.g. suggest /verbose after a
long tool run). No tool *output* travels here — output is the agent's
concern and is persisted to history, not streamed as presentation.
The gateway uses it to clear/settle a progress bubble and to drive one-time
onboarding hints (e.g. suggest /verbose after a long tool run).
"""
tool_name: str
duration: float = 0.0
ok: bool = True
duration: float = 0.0 # wall-clock seconds
ok: bool = True # returned without raising
index: int = 0
# ── Gateway control / lifecycle events ───────────────────────────────────────
@dataclass(frozen=True)
class LongToolHint:
"""One-shot onboarding nudge when a tool runs longer than the threshold.
The gateway gates this on platform capability (the /verbose command must be
usable) and on the user not having seen the hint before. Modeled as an
event so the *gateway* owns the "should I surface this here?" decision rather
than the agent.
The gateway (not the agent) gates it on platform capability (the /verbose
command must be usable) and on the user not having seen it before.
"""
tool_name: str = ""
duration: float = 0.0
@@ -133,39 +89,24 @@ class LongToolHint:
@dataclass(frozen=True)
class GatewayNotice:
"""A gateway-originated control message (restart, online, long-run notice).
"""A gateway-originated control message.
``kind`` is a stable string the adapter can switch on
(``"restart"`` / ``"online"`` / ``"long_run"`` / …). ``text`` is the
human-readable default the base class renders when an adapter has no
platform-specific treatment.
``kind`` is a stable string adapters switch on (``"restart"`` / ``"online"``
/ ``"long_run"`` / …); ``text`` is the default rendering.
"""
kind: str
text: str = ""
extra: Dict[str, Any] = field(default_factory=dict)
# Union of every event the consumer's dispatcher accepts. Kept explicit (rather
# than a marker base class) so a missing ``case`` in an exhaustive match is a
# visible type error rather than a silent fall-through.
# Explicit union (not a marker base class) so a missing ``case`` in an
# exhaustive match is a visible type error rather than a silent fall-through.
StreamEvent = Union[
MessageChunk,
MessageStop,
Commentary,
ToolCallChunk,
ToolCallFinished,
LongToolHint,
GatewayNotice,
MessageChunk, MessageStop, Commentary,
ToolCallChunk, ToolCallFinished, LongToolHint, GatewayNotice,
]
__all__ = [
"MessageChunk",
"MessageStop",
"Commentary",
"ToolCallChunk",
"ToolCallFinished",
"LongToolHint",
"GatewayNotice",
"StreamEvent",
"MessageChunk", "MessageStop", "Commentary", "ToolCallChunk",
"ToolCallFinished", "LongToolHint", "GatewayNotice", "StreamEvent",
]
gateway/streaming_tts_consumer.py
+67 -175
View File
@@ -1,39 +1,27 @@
"""Gateway streaming-TTS consumer — LLM deltas to adapter PCM audio sink.
Bridges the synchronous agent ``stream_delta_callback`` (fired from the
worker thread) to a voice-capable platform adapter's streaming-audio
contract, so playback begins while the LLM is still generating.
Bridges the synchronous agent ``stream_delta_callback`` (worker thread) to a
voice-capable adapter's streaming-audio contract so playback begins while the
LLM is still generating.
Lifecycle::
consumer = StreamingTTSConsumer(adapter, chat_id, tts_config, loop, metadata)
agent.stream_delta_callback = consumer.on_delta # sync, non-blocking
... agent runs in executor ...
consumer.finish() # signal end-of-text
consumer.finish() # signal end-of-text
success = await consumer.wait_complete(timeout=10)
if consumer.suppress_whole_file:
# suppress whole-file auto-TTS for this turn
consumer.abort("cancelled") # idempotent cancellation
if consumer.suppress_whole_file: ... # skip whole-file auto-TTS
consumer.abort("cancelled") # idempotent cancellation
Design:
- ``on_delta`` is synchronous and never blocks the agent thread. It feeds
deltas into a ``SentenceChunker`` and queues completed clauses onto a
thread-safe ``queue.Queue``.
- An asyncio task (``run``) runs on the gateway event loop, draining the
queue, synthesising each clause via a ``StreamingTTSProvider``, and
writing PCM chunks to the adapter.
- Per-turn state is isolated: each consumer instance owns its own chunker,
queue, handle, and flags. Concurrent chats cannot cross-contaminate.
- On successful completion (all clauses synthesised and written), the
consumer reports ``completed=True`` so the gateway can suppress the
duplicate whole-file auto-TTS.
- On failure before any audible output, the consumer reports
``completed=False`` and clears ``suppress_whole_file`` so the gateway can
fall back to whole-file TTS.
- On failure after partial audible output, the consumer reports
``completed=False`` but keeps ``suppress_whole_file=True`` so the gateway
does NOT replay the whole response from the beginning.
- Cancellation/abort is idempotent: late chunks are silently dropped.
``on_delta`` never blocks: it feeds a ``SentenceChunker`` and queues clauses on a
thread-safe ``queue.Queue``; the ``_run`` task on the gateway loop drains it,
synthesises via a ``StreamingTTSProvider`` and writes PCM to the adapter. State
is per instance (concurrent chats cannot cross-contaminate); abort is idempotent
and late chunks are dropped. Outcome contract: full success -> ``completed``;
failure before any audible output -> ``suppress_whole_file=False`` (gateway falls
back to whole-file TTS); failure after partial audio -> ``partial`` and
``suppress_whole_file=True`` (never replay the response from the beginning).
"""
from __future__ import annotations
@@ -69,12 +57,10 @@ class StreamingTTSConsumer:
self._adapter = adapter
self._chat_id = chat_id
self._tts_config = tts_config
self._loop = loop
self._metadata = metadata
# Resolve the streaming provider once. If unavailable, the consumer is
# inactive and the gateway falls back to whole-file TTS.
# Resolved once; None => inactive, gateway falls back to whole-file TTS.
self._streamer = resolve_streaming_provider(tts_config)
self._chunker = SentenceChunker()
@@ -87,12 +73,10 @@ class StreamingTTSConsumer:
else:
self._audio_format = audio_format or AudioFormat()
# Thread-safe queue: completed clauses and the occasional abort sentinel.
# Thread-safe queue of completed clauses plus the _DONE/_ABORT sentinels.
self._queue: "queue.Queue[Any]" = queue.Queue(maxsize=256)
# Per-turn state.
self._handle: Optional[StreamingTTSHandle] = None
self._started = False
self._completed = False
self._partial = False
self._aborted = False
@@ -101,58 +85,36 @@ class StreamingTTSConsumer:
self._suppress_whole_file = False
self._task: Optional[asyncio.Task] = None
self._lock = threading.Lock()
# Pre-allocate the strip-markdown helper lazily to avoid import cycles.
self._strip_markdown = None
# ------------------------------------------------------------------
# Public properties
# ------------------------------------------------------------------
self._strip_markdown = None # lazily imported to avoid import cycles
@property
def active(self) -> bool:
"""True when this consumer has a usable streaming provider."""
def active(self) -> bool: # usable streaming provider resolved
return self._streamer is not None
@property
def completed(self) -> bool:
"""True when streaming audio was fully delivered."""
def completed(self) -> bool: # streaming audio fully delivered
return self._completed
@property
def partial(self) -> bool:
"""True when some audio was audible before a failure or drop."""
def partial(self) -> bool: # some audio was audible before a failure/drop
return self._partial
@property
def started(self) -> bool:
"""True when the adapter accepted the streaming session."""
return self._started
@property
def audible(self) -> bool:
"""True once the first PCM chunk has been written."""
def audible(self) -> bool: # first PCM chunk has been written
return bool(self._handle and self._handle.audible)
@property
def dropped(self) -> bool:
"""True when queue saturation dropped at least one clause."""
def dropped(self) -> bool: # queue saturation dropped at least one clause
return self._dropped
@property
def suppress_whole_file(self) -> bool:
"""True when the gateway should skip the legacy whole-file TTS fallback."""
def suppress_whole_file(self) -> bool: # gateway should skip whole-file TTS fallback
return self._suppress_whole_file
@property
def done(self) -> bool:
"""True once the async drain task has terminated."""
def done(self) -> bool: # async drain task has terminated
return self._task is not None and self._task.done()
# ------------------------------------------------------------------
# Sync callback (agent worker thread)
# ------------------------------------------------------------------
def on_delta(self, text: str) -> None:
"""Receive a text delta from the agent. Non-blocking."""
if self._aborted or not self.active or self._finished:
@@ -167,11 +129,10 @@ class StreamingTTSConsumer:
logger.debug("streaming TTS on_delta error", exc_info=True)
def finish(self) -> None:
"""Signal end-of-text and flush the chunker tail.
"""Signal end-of-text, flush the chunker tail, then enqueue ``_DONE``.
Enqueues a ``_DONE`` sentinel after all flushed clauses so the
drain loop has a deterministic termination signal that cannot
race with a late ``on_delta`` or be lost when the queue is full.
The sentinel follows all flushed clauses so the drain loop has a
deterministic termination that cannot race a late ``on_delta``.
"""
if self._finished:
return
@@ -186,13 +147,7 @@ class StreamingTTSConsumer:
logger.debug("streaming TTS queue full while flushing tail")
except Exception:
pass
# Guarantee the _DONE sentinel reaches the queue. If the bounded
# queue is full, drain one item to make room — the sentinel is
# load-bearing and must not be lost (#60671 hardening).
self._enqueue_done()
def _enqueue_done(self) -> None:
"""Enqueue the _DONE sentinel, evicting a queued clause if necessary."""
# The load-bearing _DONE sentinel must never be lost: evict a clause if full.
while True:
try:
self._queue.put_nowait(_DONE)
@@ -204,106 +159,72 @@ class StreamingTTSConsumer:
except queue.Empty:
continue
# ------------------------------------------------------------------
# Async lifecycle (gateway event loop)
# ------------------------------------------------------------------
def start(self) -> asyncio.Task:
"""Create and return the async drain task on the gateway loop."""
if self._task is not None:
return self._task
self._task = self._loop.create_task(self._run())
"""Create (once) and return the async drain task on the gateway loop."""
if self._task is None:
self._task = self._loop.create_task(self._run())
return self._task
def _settle(self, *, failed: bool) -> None:
"""Set the outcome flags from what was audible.
Never report completion after a failure or a dropped clause; keep
suppression whenever audio was audible so the gateway does not replay
the response from the beginning.
"""
audible = self._handle.audible
degraded = failed or self._dropped
self._completed = audible and not degraded
if audible and degraded:
self._partial = True
self._suppress_whole_file = audible
async def _run(self) -> None:
"""Drain clauses from the queue, synthesise, and write to the adapter."""
if not self.active:
return
if not self._adapter.supports_streaming_tts(self._chat_id, self._audio_format):
logger.debug("adapter %s does not support streaming TTS", getattr(self._adapter, "name", "?"))
return
try:
self._handle = await self._adapter.begin_streaming_tts(
self._chat_id,
self._audio_format,
metadata=self._metadata,
self._chat_id, self._audio_format, metadata=self._metadata,
)
except Exception as exc:
logger.debug("begin_streaming_tts failed: %s", exc)
self._handle = None
return
if self._handle is None:
return
self._started = True
self._suppress_whole_file = False
try:
while True:
if self._aborted:
break
while not self._aborted:
try:
item = await asyncio.to_thread(self._queue.get, True, 0.1)
except queue.Empty:
continue
if item is _ABORT:
break
if item is _DONE:
if item is _ABORT or item is _DONE or self._aborted:
break
if not isinstance(item, str):
continue
if self._aborted:
break
try:
await self._synthesise_and_write(item)
except Exception as exc:
logger.warning("streaming TTS clause failed: %s", exc)
if self._handle and self._handle.audible:
self._partial = True
self._suppress_whole_file = True
else:
self._suppress_whole_file = False
self._completed = False
self._settle(failed=True)
await self._safe_abort(str(exc))
return
if not self._aborted and self._handle is not None:
_finish_failed = False
try:
await self._adapter.finish_streaming_tts(self._handle, interrupted=self._aborted)
except Exception as exc:
logger.debug("finish_streaming_tts error: %s", exc)
_finish_failed = True
if _finish_failed:
# finish_streaming_tts() raised — never report full
# completion. If audio was already audible, report
# partial and preserve suppression so the gateway
# does not replay from the beginning. If no audio
# was audible, permit whole-file fallback.
if self._handle.audible:
self._partial = True
self._completed = False
self._suppress_whole_file = True
else:
self._completed = False
self._suppress_whole_file = False
self._settle(failed=True)
await self._safe_abort("finish_streaming_tts failed")
elif self._handle.audible and not self._dropped:
self._completed = True
self._suppress_whole_file = True
elif self._handle.audible and self._dropped:
self._partial = True
self._completed = False
self._suppress_whole_file = True
else:
self._completed = False
self._suppress_whole_file = False
self._settle(failed=False)
except Exception as exc:
logger.warning("streaming TTS consumer error: %s", exc)
await self._safe_abort(str(exc))
@@ -316,17 +237,17 @@ class StreamingTTSConsumer:
async def _synthesise_and_write(self, clause: str) -> None:
"""Synthesise one clause via the streamer and write PCM chunks."""
if self._handle is None or self._handle.aborted:
if self._handle is None or self._handle.aborted or self._streamer is None:
return
cleaned = self._strip_markdown_for_tts(clause)
if not cleaned or not cleaned.strip():
if not cleaned.strip():
return
if self._streamer is None:
return
async for chunk in self._iter_stream_chunks(cleaned):
iterator = iter(self._streamer.stream(cleaned))
while True:
# next() runs in a thread so a blocking provider never stalls the loop.
chunk = await asyncio.to_thread(next, iterator, _DONE)
if chunk is _DONE:
return
if self._aborted or self._handle.aborted:
return
if not chunk:
@@ -337,24 +258,6 @@ class StreamingTTSConsumer:
self._handle.audible = True
self._suppress_whole_file = True
async def _iter_stream_chunks(self, text: str):
"""Yield provider PCM chunks one at a time without blocking the loop."""
if self._streamer is None:
return
iterator = iter(self._streamer.stream(text))
while True:
has_chunk, chunk = await asyncio.to_thread(self._next_stream_chunk, iterator)
if not has_chunk:
break
yield chunk
@staticmethod
def _next_stream_chunk(iterator: Any) -> tuple[bool, Optional[bytes]]:
try:
return True, next(iterator)
except StopIteration:
return False, None
def _strip_markdown_for_tts(self, text: str) -> str:
"""Lazy-import and apply the TTS markdown stripper."""
if self._strip_markdown is None:
@@ -377,19 +280,14 @@ class StreamingTTSConsumer:
if self._handle:
self._handle.aborted = True
# ------------------------------------------------------------------
# Cancellation and completion
# ------------------------------------------------------------------
def abort(self, reason: str = "cancelled") -> None:
"""Idempotent cancellation from any thread."""
with self._lock:
if self._aborted:
return
self._aborted = True
# Guarantee the _ABORT sentinel reaches the queue. If the bounded
# queue is full, drain one item to make room — the sentinel must
# not be lost (#60671 hardening).
# The _ABORT sentinel is load-bearing and must reach the queue even when
# the bounded queue is full: evict an item to make room.
for _attempt in range(3):
try:
self._queue.put_nowait(_ABORT)
@@ -403,21 +301,15 @@ class StreamingTTSConsumer:
logger.debug("streaming TTS _ABORT sentinel could not be enqueued")
if self._handle is not None and not self._handle.aborted:
try:
self._loop.call_soon_threadsafe(
asyncio.create_task,
self._safe_abort(reason),
)
self._loop.call_soon_threadsafe(asyncio.create_task, self._safe_abort(reason))
except Exception:
pass
async def wait_complete(self, timeout: float = 10.0) -> bool:
"""Wait for the drain task to finish. Returns True only on full success."""
if self._task is None:
return self._completed
try:
await asyncio.wait_for(asyncio.shield(self._task), timeout=timeout)
except (asyncio.TimeoutError, asyncio.CancelledError):
pass
except Exception:
pass
if self._task is not None:
try:
await asyncio.wait_for(asyncio.shield(self._task), timeout=timeout)
except (asyncio.CancelledError, Exception):
pass
return self._completed
gateway/turn_lease.py
+79 -136
View File
@@ -1,51 +1,36 @@
"""Per-session turn lease — serializes the [load history → run → flush] region.
Why this exists (#64934): the gateway's busy guards are keyed by ROUTING KEY
(``_active_sessions`` in the adapter, ``_running_agents`` in the runner), but
the durable transcript is owned by SESSION_ID — and ``switch_session()`` makes
the key→id mapping many-to-one (``/resume`` of a named session from a second
chat/topic, CLI-continuity rebinding, async-delegation completion pinning,
Telegram topic-binding tip-walks). Two routing keys mapped to one session_id
run concurrent turns on two different agent objects, so no per-key guard ever
sees the collision. The two turns then interleave their flushes on one
transcript: rows persist in completion order instead of arrival order, the
identity-marker dedup over shared history dicts can swallow a row outright,
and the second turn runs on a history base that never saw the first turn's
exchange — leaving a permanent ``user;user`` alternation wedge that
``repair_message_sequence`` re-repairs on every request forever.
Why: the gateway's busy guards are keyed by ROUTING KEY (adapter
``_active_sessions``, runner ``_running_agents``), but the durable transcript is
owned by SESSION_ID, and ``switch_session()`` makes key→id many-to-one (/resume
of a named session from a second chat, CLI-continuity rebinding, async-delegation
completion pinning, Telegram topic-binding tip-walks). Two routing keys on one
session_id ran concurrent turns on two agent objects that no per-key guard could
see; their flushes interleaved on one transcript (rows in completion order,
identity-marker dedup swallowing rows, a permanent ``user;user`` alternation
wedge that ``repair_message_sequence`` re-repaired forever).
The lease closes that route by serializing per RESOLVED session_id: it is
acquired after session resolution is final (post ``switch_session``/tip-walk),
immediately before the transcript load, and released in the dispatch layer's
``finally`` on every exit path. Same-key messages never reach the acquisition
point while a turn runs (both routing-key guards hold them), so the lock is
uncontended everywhere except the alias-key route — where the second turn now
waits for the first turn's flush and logs one WARNING naming the session and
both routing keys (pairing with the cross-agent tripwire in
``agent/agent_runtime_helpers.note_turn_start``).
The lease serializes per RESOLVED session_id: acquired after resolution is final
(post switch_session/tip-walk), immediately before the transcript load, and
released in the dispatch layer's ``finally`` on every exit path. Same-key
messages never reach acquisition while a turn runs (routing-key guards hold
them), so the lock is uncontended except on the alias-key route, where the
second turn waits for the first's flush and logs one WARNING naming the session
and both keys (pairing with ``agent_runtime_helpers.note_turn_start``).
Safety properties:
- Generation-scoped, identity-checked, idempotent release: a token records its
owner (routing key, run generation) and only the exact current holder frees
the lease — a stale unwind can never release a newer turn's lease.
- Fail-closed on timeout: a timed-out waiter raises :class:`TurnLeaseTimeoutError`
and must be rejected with a visible resend notice; it never runs against the
still-held lease.
- Bounded registry: eviction only removes idle (unheld, uncontended) entries.
- **Generation-scoped, identity-checked release.** A token records its owner
(routing key, run generation) and release only frees the lease when that
exact token is the current holder — a stale unwind can never release a
newer turn's lease (the #28686 ownership lesson applied). Release is
idempotent.
- **Fail-closed on timeout.** A timed-out waiter raises
:class:`TurnLeaseTimeoutError` and must be rejected by the dispatch layer
with a visible resend notice. It never runs concurrently against the
still-held lease and therefore cannot defeat the serialization invariant.
- **Bounded registry.** The per-session lease map is size-capped; eviction
only ever removes idle (unheld, uncontended) entries, never a live lease.
Known limits (deliberate, flagged on #64934):
- A CLI process sharing the session via CLI-continuity is outside any
in-process lock — that pair needs a DB-level lease (separate design).
- Mid-turn compression rotation leaves a small alias window: the tip-walk can
resolve a fresh child id while the parent-holding turn is still in flight.
The mid-turn binding-sync sites are the right place to alias the lease in a
follow-up.
Known limits: a CLI process sharing the session via CLI-continuity is outside
any in-process lock (needs a DB-level lease); mid-turn compression rotation
leaves a small alias window, closed by :meth:`SessionTurnLeaseRegistry.rebind`
at the mid-turn binding-sync sites.
"""
import asyncio
@@ -55,29 +40,26 @@ from typing import Dict, Optional
logger = logging.getLogger(__name__)
# Upper bound on tracked per-session leases. Idle entries (no holder or
# pending acquire) are evicted oldest-first once the cap is reached; live
# leases are never evicted, so a burst of distinct sessions can transiently
# exceed the cap rather than break serialization.
# Cap on tracked per-session leases. Idle entries are evicted oldest-first; live
# leases never are, so a burst of distinct sessions may transiently exceed the cap
# rather than break serialization.
DEFAULT_MAX_LEASES = 512
# Fallback wait (seconds) when the caller passes no positive timeout. The
# gateway carries this independently through its internal
# HERMES_TURN_LEASE_TIMEOUT bridge because lease contention is not agent
# inactivity. A caller that reaches this bound must reject the turn rather than
# run it concurrently with the holder.
# Keep contention fail-closed, but never pin a sequential platform updater for
# minutes. A waiter that cannot acquire promptly is rejected with a resend
# notice; it is never authorized to run without serialization.
# Fallback wait (seconds) when the caller passes no positive timeout. The gateway
# carries this through its internal HERMES_TURN_LEASE_TIMEOUT bridge because lease
# contention is not agent inactivity. Fail-closed but short: never pin a
# sequential platform updater for minutes — a waiter that cannot acquire promptly
# is rejected with a resend notice, never authorized to run unserialized.
DEFAULT_LEASE_WAIT = 5.0
class TurnLeaseTimeoutError(TimeoutError):
"""The session lease stayed held for the caller's full wait budget.
def _holder_desc(holder: Optional["TurnLeaseToken"]) -> tuple:
return (holder.owner_key, holder.generation) if holder else ("?", "?")
This is a fail-closed signal: the caller did not acquire the lease and
must not enter the transcript load/run/flush region for this turn.
"""
class TurnLeaseTimeoutError(TimeoutError):
"""The lease stayed held for the caller's full wait budget (fail-closed: the
caller must not enter the transcript load/run/flush region)."""
def __init__(
self,
@@ -100,19 +82,13 @@ class TurnLeaseTimeoutError(TimeoutError):
class TurnLeaseToken:
"""Handle returned by :meth:`SessionTurnLeaseRegistry.acquire`.
A timeout raises :class:`TurnLeaseTimeoutError` instead of returning a
token, so every token handed out is a held lease. ``released`` makes
release idempotent.
Every token handed out is a held lease (timeouts raise instead); ``released``
makes release idempotent.
"""
__slots__ = ("session_id", "owner_key", "generation", "released")
def __init__(
self,
session_id: str,
owner_key: str,
generation: int,
) -> None:
def __init__(self, session_id: str, owner_key: str, generation: int) -> None:
self.session_id = session_id
self.owner_key = owner_key
self.generation = generation
@@ -127,13 +103,7 @@ class TurnLeaseToken:
class _SessionLease:
__slots__ = (
"lock",
"holder",
"acquired_at",
"last_used",
"pending_acquires",
)
__slots__ = ("lock", "holder", "acquired_at", "last_used", "pending_acquires")
def __init__(self) -> None:
self.lock = asyncio.Lock()
@@ -144,20 +114,15 @@ class _SessionLease:
@property
def idle(self) -> bool:
"""True when this lease can be evicted: nobody holds or awaits it."""
return (
self.holder is None
and not self.lock.locked()
and self.pending_acquires == 0
)
"""True when evictable: nobody holds or awaits it."""
return self.holder is None and not self.lock.locked() and self.pending_acquires == 0
class SessionTurnLeaseRegistry:
"""Asyncio lease per resolved session_id serializing transcript turns.
Process-local and single-event-loop by design — the same visibility scope
as the routing-key guards it extends. All methods must be called from the
gateway's event loop.
Process-local and single-event-loop by design — the same visibility scope as
the routing-key guards it extends. Call only from the gateway's event loop.
"""
def __init__(self, max_entries: int = DEFAULT_MAX_LEASES) -> None:
@@ -171,16 +136,13 @@ class SessionTurnLeaseRegistry:
lease = self._leases.get(session_id)
if lease is None:
self._evict_idle()
lease = _SessionLease()
self._leases[session_id] = lease
lease = self._leases[session_id] = _SessionLease()
lease.last_used = time.time()
return lease
def _evict_idle(self) -> None:
"""Drop oldest idle entries so a new lease fits under the cap.
Never evicts a held or contended lease — correctness beats the cap.
"""
"""Drop oldest idle entries so a new lease fits under the cap. Never
evicts a held or contended lease — correctness beats the cap."""
overflow = len(self._leases) - self._max_entries + 1
if overflow <= 0:
return
@@ -201,10 +163,8 @@ class SessionTurnLeaseRegistry:
) -> Optional[TurnLeaseToken]:
"""Acquire the turn lease for ``session_id``, waiting if held.
Returns a held :class:`TurnLeaseToken`. Raises
:class:`TurnLeaseTimeoutError` when the wait budget expires; the caller
must reject rather than enter the serialized region. Returns ``None``
for a falsy ``session_id``.
Raises :class:`TurnLeaseTimeoutError` when the wait budget expires (the
caller must reject the turn). Returns None for a falsy ``session_id``.
"""
if not session_id:
return None
@@ -213,7 +173,6 @@ class SessionTurnLeaseRegistry:
lease = self._get_or_create(session_id)
if lease.lock.locked():
holder = lease.holder
logger.warning(
"turn lease contention on session %s: routing key %s (gen %s) "
"waiting behind in-flight turn held by routing key %s (gen %s, "
@@ -223,21 +182,19 @@ class SessionTurnLeaseRegistry:
session_id,
owner_key,
generation,
holder.owner_key if holder else "?",
holder.generation if holder else "?",
*_holder_desc(lease.holder),
time.time() - lease.acquired_at if lease.acquired_at else -1.0,
)
# Lock.release() wakes a waiter while leaving the lock momentarily
# unlocked. Track every in-progress acquire across that handoff so
# unlocked. Count every in-progress acquire across that handoff so
# eviction cannot orphan the old lock and create a second lock for the
# same session. Count even apparently-uncontended acquires: wait_for()
# may schedule them before the underlying lock coroutine runs.
# same session — even apparently-uncontended ones, since wait_for() may
# schedule them before the underlying lock coroutine runs.
lease.pending_acquires += 1
try:
await asyncio.wait_for(lease.lock.acquire(), timeout=wait)
except asyncio.TimeoutError:
holder = lease.holder
logger.error(
"turn lease wait timed out after %.0fs on session %s "
"(waiter: routing key %s gen %s; holder: routing key %s "
@@ -247,47 +204,37 @@ class SessionTurnLeaseRegistry:
session_id,
owner_key,
generation,
holder.owner_key if holder else "?",
holder.generation if holder else "?",
*_holder_desc(lease.holder),
)
raise TurnLeaseTimeoutError(
session_id,
owner_key=owner_key,
generation=generation,
wait_seconds=wait,
session_id, owner_key=owner_key, generation=generation, wait_seconds=wait
) from None
finally:
lease.pending_acquires -= 1
# The lock is held and there is no await before holder publication, so
# the lease cannot become evictable after the pending count is cleared.
# Lock held and no await before holder publication, so the lease cannot
# become evictable after the pending count is cleared.
lease.holder = token
lease.acquired_at = time.time()
lease.last_used = lease.acquired_at
lease.acquired_at = lease.last_used = time.time()
return token
def rebind(self, token: Optional[TurnLeaseToken], new_session_id: str) -> bool:
"""Alias a HELD lease onto ``new_session_id`` after mid-turn rotation.
Compression can rotate the durable session_id while a turn is in
flight (session-hygiene pre-compression, in-agent compression). The
turn's flush then targets the NEW id — so the serialization boundary
must follow it, or an alias routing key resolving the new id (e.g. a
topic tip-walk landing on the fresh child) could start a concurrent
turn the lease never sees. This closes the rotation-alias window
flagged on #64934.
Compression can rotate the durable session_id mid-turn (session-hygiene
pre-compression, in-agent compression); the flush then targets the NEW
id, so the serialization boundary must follow it or an alias key
resolving the new id could start a turn the lease never sees.
Mechanism: the SAME ``_SessionLease`` object is registered under the
new id (the old mapping stays until it goes idle and is evicted), so
acquirers on either id serialize against one lock — no lock state is
moved, no asyncio internals are touched. Only the current holder can
rebind (identity-checked like release), and the token follows to the
new id so release frees the shared object.
Mechanism: the SAME ``_SessionLease`` is registered under the new id (the
old mapping stays until idle and evicted), so acquirers on either id
serialize on one lock — no lock state moves. Only the current holder can
rebind (identity-checked like release), and the token follows so release
frees the shared object.
Edge: if the new id already has a live lease of its own (another
turn is running on the target session), the two serialization
domains cannot be merged mid-wait — log loudly and keep the token on
the old id. Fail-open, never deadlock: a holder cannot wait mid-turn.
Edge: if the new id already has a live lease of its own, the two domains
cannot be merged mid-wait — log loudly and keep the token on the old id.
Fail-open, never deadlock: a holder cannot wait mid-turn.
"""
if (
token is None
@@ -302,7 +249,6 @@ class SessionTurnLeaseRegistry:
existing = self._leases.get(new_session_id)
if existing is not None and existing is not lease and not existing.idle:
holder = existing.holder
logger.warning(
"turn lease rebind blocked: session %s rotated to %s mid-turn "
"(holder: routing key %s gen %s) but the target session's "
@@ -313,8 +259,7 @@ class SessionTurnLeaseRegistry:
new_session_id,
token.owner_key,
token.generation,
holder.owner_key if holder else "?",
holder.generation if holder else "?",
*_holder_desc(existing.holder),
new_session_id,
)
return False
@@ -325,12 +270,10 @@ class SessionTurnLeaseRegistry:
return True
def release(self, token: Optional[TurnLeaseToken]) -> bool:
"""Release ``token``'s lease. Idempotent; ownership-checked.
"""Release ``token``'s lease. Idempotent; ownership-checked.
Returns True only when this exact token was the current holder and
the lock was freed. A re-release or a stale token whose slot has
since been granted to a newer turn are both safe no-ops — a stale
unwind can never release a newer turn's lease.
True only when this exact token was the current holder. A re-release or
a stale token whose slot went to a newer turn are safe no-ops.
"""
if token is None or token.released:
return False
tests/gateway/relay/test_relay_multiplatform.py
-4
View File
@@ -3,7 +3,6 @@
Covers the agent half of Shape A (gateway-gateway D-Q1.5b.1 / D-Q1.5c):
- relay_platform_identities() parsing the GATEWAY_RELAY_PLATFORMS list +
GATEWAY_RELAY_BOT_IDS keyed map (the cut-over shape — no scalar fallback),
- relay_bot_username() reading the per-platform username,
- self_provision_relay() looping one /relay/provision POST per platform under
one gatewayId + one secret, partial-failure-tolerant,
- the RelayAdapter stamping the per-frame egress platform on outbound from the
@@ -55,9 +54,6 @@ def test_identities_multi_platform_keyed_map(monkeypatch):
assert relay.relay_platform_identities() == [("discord", "app-1"), ("telegram", "bot-9")]
# The PRIMARY is the first listed platform.
assert relay.relay_platform_identity() == ("discord", "app-1")
# Username folded into the per-platform entry; the leading @ is stripped.
assert relay.relay_bot_username("telegram") == "my_bot"
assert relay.relay_bot_username("discord") is None
def test_bot_ids_malformed_json_degrades_to_empty(monkeypatch):
tests/gateway/test_browser_control_broker_hardening.py
+2 -3
View File
@@ -402,7 +402,7 @@ def test_session_lane_replacement_and_owner_detach_are_scoped():
transport_family="cloud-ticket-ws",
) == other
assert broker.detach_owner("owner-shared") == 1
broker.detach(second, owner="owner-shared")
assert broker.scope_for_session(
session_id="session-fixture",
principal_id="principal-fixture",
@@ -413,7 +413,6 @@ def test_session_lane_replacement_and_owner_detach_are_scoped():
principal_id="principal-fixture",
transport_family="cloud-ticket-ws",
) == other
assert broker.detach_owner("missing-owner") == 0
broker.reset()
assert broker.scope_for_session(
@@ -476,7 +475,7 @@ def test_transport_teardown_can_cancel_waiters_without_writing_to_closing_peer()
scope = _scope()
thread, outcome, frames = _start_pending(broker, scope)
assert broker.detach_owner("owner-fixture", notify_controller=False) == 1
broker.detach(scope, owner="owner-fixture", notify_controller=False)
thread.join(timeout=1.0)
assert isinstance(outcome.get("error"), ControllerCancelled)
tests/gateway/test_delivery_ledger.py
-2
View File
@@ -610,8 +610,6 @@ class TestAttemptsOnlySpentOnRealSends:
_orphan("ob-1")
assert dl.sweep_recoverable(deliverable_platforms={"discord"}) == []
row = dl.debug_rows()
assert "abandoned" not in row
with dl._connect() as conn:
state, attempts = conn.execute(
"SELECT state, attempts FROM delivery_obligations "
tests/gateway/test_delivery_ledger_fd_leak.py
-1
View File
@@ -76,7 +76,6 @@ def test_ledger_operations_close_every_connection(monkeypatch, tmp_path):
dl.mark_attempting(oid)
dl.mark_delivered(oid)
dl.sweep_recoverable()
dl.debug_rows()
assert opened, "expected at least one connection to be opened"
assert len(opened) == len(closed)
tests/gateway/test_hosted_room_peer.py
-29
View File
@@ -17,13 +17,11 @@ from gateway.hosted_room_peer import (
HostedRoomGrantError,
HostedRoomPeerError,
PROTOCOL_VERSION,
RoomLinkProbe,
catalog_mapping,
derive_room_grant_secret,
gateway_room_grant_secret,
issue_room_grant,
local_room_link_endpoint,
select_room_link,
verify_room_grant,
)
from hermes_constants import reset_hermes_home_override, set_hermes_home_override
@@ -292,33 +290,6 @@ def test_room_grant_fails_closed_for_tamper_expiry_and_permission():
verify_room_grant(SECRET, token[:-1] + "A", dispatch, now=105)
def test_link_selection_prefers_safe_direct_then_overlay_then_relay_then_pull():
selected = select_room_link(
[
RoomLinkProbe("relay", True, True, 10),
RoomLinkProbe("direct", True, True, 50),
RoomLinkProbe("overlay", True, True, 5),
RoomLinkProbe("pull", True, True, 1),
],
desktop_available=False,
)
assert selected is not None
assert selected.mode == "direct"
def test_link_selection_never_falls_back_to_unencrypted_route():
assert (
select_room_link(
[RoomLinkProbe("direct", True, False, 1)],
desktop_available=False,
)
is None
)
fallback = select_room_link([], desktop_available=True)
assert fallback is not None
assert fallback.mode == "desktop"
def test_local_catalog_is_honest_for_app_managed_process(monkeypatch):
from gateway.hosted_room_peer import local_catalog_mapping
tests/gateway/test_session_db_recovery.py
-4
View File
@@ -33,7 +33,6 @@ def test_failed_open_obeys_backoff_then_recovers() -> None:
return handle
assert cache.get(path, opener) is None
assert cache.status_for(path) == "unavailable"
clock.now = 1.99
assert cache.get(path, opener) is None
assert calls == 1
@@ -42,7 +41,6 @@ def test_failed_open_obeys_backoff_then_recovers() -> None:
assert cache.get(path, opener) is handle
assert cache.get(path, opener) is handle
assert calls == 2
assert cache.status_for(path) == "ok"
def test_retry_is_single_flight_for_concurrent_callers() -> None:
@@ -74,7 +72,6 @@ def test_retry_is_single_flight_for_concurrent_callers() -> None:
# and keep using the fallback rather than opening or blocking behind it.
assert cache.get(path, opener) is None
assert calls == 2
assert cache.status_for(path) == "retrying"
release.set()
thread.join(timeout=5)
@@ -196,7 +193,6 @@ def test_non_cacheable_guard_is_retried_immediately() -> None:
except RuntimeError:
pass
assert calls == 2
assert cache.status_for(path) == "unknown"
def test_close_all_rejects_and_closes_inflight_success() -> None:
tests/gateway/test_streaming_tts_consumer.py
-2
View File
@@ -171,7 +171,6 @@ def _make_consumer(adapter, chat_id, loop, streamer):
consumer = StreamingTTSConsumer.__new__(StreamingTTSConsumer)
consumer._adapter = adapter
consumer._chat_id = chat_id
consumer._tts_config = {}
consumer._loop = loop
consumer._metadata = None
consumer._audio_format = AudioFormat(
@@ -183,7 +182,6 @@ def _make_consumer(adapter, chat_id, loop, streamer):
consumer._chunker = SentenceChunker()
consumer._queue = queue.Queue(maxsize=256)
consumer._handle = None
consumer._started = False
consumer._completed = False
consumer._partial = False
consumer._aborted = False
tests/hermes_cli/test_gateway_restart_loop.py
-13
View File
@@ -1750,17 +1750,6 @@ class TestRestartLoopGuard:
import gateway.restart_loop_guard as rlg
rlg.clear()
def test_is_tripped_reads_without_recording(self):
import gateway.restart_loop_guard as rlg
rlg.record_restart_interrupted_boot(60, now=1000.0)
rlg.record_restart_interrupted_boot(60, now=1001.0)
assert rlg.is_restart_loop_tripped(3, 60, now=1002.0) is False
rlg.record_restart_interrupted_boot(60, now=1002.0)
assert rlg.is_restart_loop_tripped(3, 60, now=1003.0) is True
def test_clear_resets(self):
import gateway.restart_loop_guard as rlg
rlg.check_and_record(3, 60, now=1000.0)
@@ -1794,7 +1783,6 @@ class TestRestartLoopGuard:
rlg.check_and_record(3, 60, now=1150.0)
# 1h later: unrelated restart, chain reset to a single boot.
assert rlg.check_and_record(3, 60, now=4800.0) is False
assert rlg.is_restart_loop_tripped(3, 60, now=4801.0) is False
def test_fast_respawn_loop_still_trips(self):
"""#30719 regression: the original ~10s loop must keep tripping."""
@@ -1823,7 +1811,6 @@ class TestRestartLoopGuard:
import gateway.restart_loop_guard as rlg
for ts in (1000.0, 1150.0, 1300.0, 1450.0):
assert rlg.check_and_record(0, 60, now=ts) is False
assert rlg.is_restart_loop_tripped(0, 60, now=1451.0) is False
class TestTerminalToolGatewayLifecycleGuardRemote:
"""Remote-backend and two-session cwd regression coverage."""