fix(kanban): text dispatch output and both "dispatcher stuck" warnings name the hold reason
`hermes kanban dispatch` (plain output), the standalone daemon's stuck warning and the gateway's embedded dispatcher stuck warning all reported a bare `Spawned: 0` / "0 workers spawned" while the respawn guard held every ready card — the reason existed only as a `respawn_guarded` task event visible via `hermes kanban tail`. Operators watching the gateway health warning for 73+ ticks (#111910) had nothing to act on. - `kanban_db_dispatch.describe_suppression()` renders the guard reasons per task plus rate_limited / skipped_locked / memory_pressure for one or more DispatchResults, so the CLI daemon and gateway warnings share one wording: `Last tick held back: active_pr=1, memory_pressure=elevated.` - plain `dispatch` output prints `Guarded (<reason>): <task id>` and the tick-level holds, mirroring the JSON fields. - kanban docs: how to see why a ready card is not spawning. Co-authored-by: Steven Saehrig <trac3r726@users.noreply.github.com> Part of #111910
This commit is contained in:
@@ -272,6 +272,7 @@ class GatewayKanbanWatchersMixin:
|
||||
# broken PATH, missing venv, or credential loss.
|
||||
bad_ticks = 0
|
||||
last_warn_at = 0
|
||||
results: Optional[list] = None
|
||||
dispatcher = _KanbanDispatcher(_kb, settings)
|
||||
|
||||
logger.info("kanban dispatcher: embedded in gateway (interval=%.1fs)", interval)
|
||||
@@ -304,12 +305,13 @@ class GatewayKanbanWatchersMixin:
|
||||
bad_ticks = bad_ticks + 1 if ready_pending and not any_spawned else 0
|
||||
now = int(time.time())
|
||||
if bad_ticks >= _HEALTH_WINDOW and now - last_warn_at >= 300:
|
||||
held = _kbd.describe_suppression(res for _slug, res in (results or []))
|
||||
logger.warning(
|
||||
"kanban dispatcher stuck: ready queue non-empty for "
|
||||
"%d consecutive ticks but 0 workers spawned. Check "
|
||||
"%d consecutive ticks but 0 workers spawned.%s Check "
|
||||
"profile health (venv, PATH, credentials) and "
|
||||
"`hermes kanban list --status ready`.",
|
||||
bad_ticks,
|
||||
bad_ticks, f" Last tick held back: {held}." if held else "",
|
||||
)
|
||||
last_warn_at = now
|
||||
except asyncio.CancelledError:
|
||||
|
||||
Reference in New Issue
Block a user