fix(update): break the cron-update three-way restart deadlock (#100179)
When hermes-auto-update runs \hermes update\ from cron, the update
process lives INSIDE the gateway's own process tree. Waiting for that
gateway to exit is a circular wait:
gateway waits on all in-flight work units (#77184 don't-amputate)
-> cron agent session waits on the \hermes update\ process to exit
-> \hermes update\ waits on the gateway to exit [back to A]
The wedged-loop probe (#81642) cannot break it: the cron session posts
activity every ~180s (process-tool poll return), so it is 'actively
waiting forever' and never marked wedged. The gateway logs
'Restart deferred: waiting on 1 active work unit(s)' every 30s until the
1800s force-drain cap amputates its own updater's session — reported as
a 5+ minute hang with gateway_state.json stuck at draining +
restart_requested (v0.21.0, main @ 530aa7b10f).
Fix (the issue's recommended option 1): at both drain sites in
update_cmd.py — systemd (line ~9862) and the bare-process/launchd path
(line ~10203) — check \_is_pid_ancestor_of_current_process(pid)\ before
drain-waiting. When the target gateway IS an ancestor, use
\_request_gateway_self_restart\ (SIGUSR1, no exit-wait) and return: the
gateway's own restart flow completes normally once this process, and
therefore the cron work unit holding it, exits.
Both helpers already exist in hermes_cli/gateway.py (277-304) and
\_request_gateway_self_restart\ already refuses non-ancestor PIDs, so a
normal out-of-tree \hermes update\ keeps its full drain semantics
(including the #86684 cron floor) untouched.
Tests (tests/hermes_cli/test_update_cron_deadlock_guard.py, 6):
- own PID / parent PID are ancestors; 0 and negative are not
- self-restart refuses a non-ancestor PID [linux]
- ancestor path sends SIGUSR1 and NEVER calls _wait_for_pid_exit
(the deadlock witness — a wait there is the bug) [linux]
- non-ancestor path still drain-waits with the given budget [linux]
Existing graceful/sigusr1/restart tests pass unchanged (9 passed).
Fixes #100179
This commit is contained in:
salch-cred
@@ -9921,10 +9921,47 @@ def _cmd_update_impl(args, gateway_mode: bool):
|
||||
from hermes_cli.gateway import (
|
||||
GATEWAY_LOOP_WEDGED,
|
||||
_escalate_wedged_gateway,
|
||||
_is_pid_ancestor_of_current_process,
|
||||
_request_gateway_self_restart,
|
||||
probe_gateway_loop_liveness,
|
||||
)
|
||||
|
||||
if (
|
||||
if _is_pid_ancestor_of_current_process(_main_pid):
|
||||
# THREE-WAY DEADLOCK BREAK (#100179).
|
||||
#
|
||||
# When `hermes update` runs INSIDE the gateway's
|
||||
# own process tree — the hermes-auto-update cron
|
||||
# job is the canonical case — waiting for that
|
||||
# gateway to exit is a circular wait:
|
||||
#
|
||||
# gateway waits on all in-flight work units
|
||||
# (#77184 don't-amputate-turns)
|
||||
# └─ cron agent session waits on the
|
||||
# `hermes update` process to exit
|
||||
# └─ `hermes update` waits on the
|
||||
# gateway to exit ← back to A
|
||||
#
|
||||
# The wedged-loop probe cannot break it: the cron
|
||||
# session posts activity every ~180s (process-tool
|
||||
# poll return), so it is "actively waiting
|
||||
# forever" and never marked wedged. The gateway
|
||||
# then burns the full force-drain cap (1800s)
|
||||
# before killing its own updater's session.
|
||||
#
|
||||
# Fire-and-forget instead: signal the restart and
|
||||
# return immediately. The gateway's own restart
|
||||
# flow completes normally once THIS process (and
|
||||
# therefore the cron work unit holding it) exits.
|
||||
print(
|
||||
f" → {svc_name}: update is running inside "
|
||||
"this gateway's process tree — signalling "
|
||||
"restart and letting the gateway drain "
|
||||
"itself (avoids the cron-update deadlock)"
|
||||
)
|
||||
_graceful_ok = _request_gateway_self_restart(
|
||||
_main_pid
|
||||
)
|
||||
elif (
|
||||
probe_gateway_loop_liveness(_main_pid)
|
||||
== GATEWAY_LOOP_WEDGED
|
||||
):
|
||||
@@ -10234,10 +10271,25 @@ def _cmd_update_impl(args, gateway_mode: bool):
|
||||
from hermes_cli.gateway import (
|
||||
GATEWAY_LOOP_WEDGED,
|
||||
_escalate_wedged_gateway,
|
||||
_is_pid_ancestor_of_current_process,
|
||||
_request_gateway_self_restart,
|
||||
probe_gateway_loop_liveness,
|
||||
)
|
||||
|
||||
if probe_gateway_loop_liveness(pid) == GATEWAY_LOOP_WEDGED:
|
||||
if _is_pid_ancestor_of_current_process(pid):
|
||||
# Same three-way deadlock break as the systemd path
|
||||
# (#100179): this update is running inside that gateway's
|
||||
# process tree (hermes-auto-update cron), so waiting for it
|
||||
# to exit is a circular wait — the gateway waits on the
|
||||
# cron work unit, the cron session waits on this process,
|
||||
# this process waits on the gateway. Signal and return.
|
||||
print(
|
||||
f" → {proc.profile}: update runs inside this gateway's "
|
||||
"process tree — signalling restart without waiting "
|
||||
"(avoids the cron-update deadlock)"
|
||||
)
|
||||
drained = _request_gateway_self_restart(pid)
|
||||
elif probe_gateway_loop_liveness(pid) == GATEWAY_LOOP_WEDGED:
|
||||
# Loop-liveness probe: this gateway's event loop is
|
||||
# provably dead (#81642) — SIGUSR1/SIGTERM shutdown can
|
||||
# never run, so the drain wait would burn the full budget
|
||||
|
||||
Reference in New Issue
Block a user