feat(gateway): session activity watchdog, stall notify, compress timeout (#72424)

Three mechanisms to detect and notify when gateway sessions stall silently:

1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
   and hermes status show progress during long turns without new message rows.

2. Stall watchdog: when a busy session has pending inbound and the shared
   activity clock is idle past agent.session_stall_timeout (default 300),
   log a WARNING and notify the user once to try /new. Notify-only; does
   not kill the turn.

3. Compaction timeout: fenceless compress_context callers get a progress-aware
   host budget (compression.context_timeout_seconds default 120 idle,
   compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
   cancel via commit fence, skip compaction without dropping messages, and
   continue the turn.

Closes #72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).

Cherry-picked from PR #72424 by @fangliquanflq.
This commit is contained in:
fangliquanflq
2026-07-27 23:16:24 +05:00
committed by Teknium
parent f01c193be4
commit c2088efe9e
26 changed files with 2801 additions and 112 deletions
+3 -8
View File
@@ -18,6 +18,7 @@ from hermes_state_common import (
SCHEMA_SQL,
_PREVIEW_RAW_SELECT,
_shape_preview,
_sql_session_last_active,
)
# Moved methods logged under the "hermes_state" logger before the split;
@@ -108,10 +109,7 @@ class SessionPortabilityMixin:
ORDER BY m.timestamp, m.id LIMIT 1),
''
) AS _preview_raw,
COALESCE(
(SELECT MAX(m2.timestamp) FROM messages m2 WHERE m2.session_id = s.id),
s.started_at
) AS last_active
{_sql_session_last_active("s")} AS last_active
FROM sessions s
WHERE s.source = 'cron' AND s.id >= ? AND s.id < ?
ORDER BY s.started_at DESC, s.id DESC
@@ -148,10 +146,7 @@ class SessionPortabilityMixin:
ORDER BY m.timestamp, m.id LIMIT 1),
''
) AS _preview_raw,
COALESCE(
(SELECT MAX(m2.timestamp) FROM messages m2 WHERE m2.session_id = s.id),
s.started_at
) AS last_active
{_sql_session_last_active("s")} AS last_active
FROM sessions s
WHERE s.id = ?
"""