Files
hermes-agent/hermes_cli
joaomarcos bcdfdd51e5 fix(gateway): make the restart-loop breaker see slow crash cycles (#81642)
The auto-resume restart-loop breaker (#30719, defense-3) pruned its boot
log against an absolute `window_seconds` (default 60s). That prune is
period-sensitive: a crash cycle slower than the window drops its own
history on every boot, so the counter never leaves 1 and the breaker can
never trip, no matter how long the loop runs.

The cycle reported in #81642 is ~150s — a wedged event loop, the liveness
watchdog hard-exiting at ~90s, a supervisor respawn, and auto-resume
replaying the same session that wedges it again. Structurally invisible to
a 60s window: `gateway/restart_loop.json` kept a single timestamp across 15
kills in one morning. Because every cycle leaves a gateway that cannot
process SIGTERM, `hermes update` has no drainable gateway to stop, which is
the reported hang.

Chain boots on the inter-boot GAP instead of an absolute window: two boots
belong to the same loop when they are no more than `max_gap_seconds` apart
(default 300s, floored by `window_seconds` so widening the window never
makes the breaker less sensitive). The verdict becomes period-agnostic —
the original ~10s respawn loop still trips in 3 boots, and so does a 150s
one — while a boot after real quiet resets the chain, so occasional
operator restarts still never accumulate. The persisted chain is capped at
50 entries.

- gateway/restart_loop_guard.py: gap-chained pruning (`_chain_ending_at`),
  `DEFAULT_MAX_GAP_SECONDS`, `max_gap_seconds` kwarg on the three entry
  points, clock-step tolerance, bounded state file
- gateway/run.py: `_restart_loop_guard_config` reads and returns
  `max_gap_seconds`; the auto-resume call site passes it through
- hermes_cli/config_defaults.py: `gateway.restart_loop_guard.max_gap_seconds`

Tests: 7 new cases in TestRestartLoopGuard covering the slow cycle, chain
persistence, quiet-period reset, the #30719 fast loop, the config knob, the
window floor, and the disabled breaker. Verified RED before the fix (the
slow-cycle case asserted `[1300.0] == [1000.0, 1150.0, 1300.0]`, exactly
the single-timestamp state file from the report) and GREEN after.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 21:40:23 +05:30
..
2026-08-03 09:57:23 -07:00
…