fix(cli): one-shot runs linger for notify_on_complete background processes so Bot Mode replies survive parent exit
A Bot Mode agent invoked by a handoff runs as a short-lived `hermes -p <bot> chat -Q --query-file ...` process. When it dispatches its reply via message_agent / bot_relay — spawned as terminal(background=true, notify_on_complete=true) per the Bot Chat protocol — the one-shot parent exits as soon as the turn ends. The reply child writes to a stdout pipe owned by the dying parent and is destroyed a few seconds later, so the handoff reply is silently lost while the sender waits for a notification that can never come (#90879). Fix (class-wide, not DM-specific): before the one-shot exit paths tear down, the parent now lingers — bounded by the new terminal.oneshot_completion_wait_seconds config (default 600s, 0 disables) — for every tracked background process spawned with notify_on_complete=true. Plain background processes (servers, daemons, watch-pattern monitors) carry no completion contract and are never waited on. - tools/process_registry.py: ProcessRegistry.wait_for_pending_completions() — bounded, interrupt-safe wait over pending notify_on_complete sessions; reconciles orphaned-pipe exits (#17327) each pass so a wedged reader cannot burn the full bound; KeyboardInterrupt aborts the linger without skipping the caller's durable teardown. - cli.py: _finalize_single_query() lingers first, before the durable session flush / cleanup (covers -q and -Q, i.e. the DM recipient shape and bot_relay waiter spawns from one-shot agents). - hermes_cli/oneshot.py: same linger before agent.close() (which kill_all()s the task's processes) on the -z path. - hermes_cli/config_defaults.py: terminal.oneshot_completion_wait_seconds. Tests: tests/tools/test_oneshot_completion_linger.py — unit coverage of the wait semantics (no-op, completion, timeout, task filter, disable, config fallback, reconcile path), exit-path ordering contracts, and a real-process E2E: a short-lived python parent spawns a delivery child through the real ProcessRegistry, lingers, exits, and the delivery completes; sabotaging the linger makes the same E2E reproduce the destroyed-delivery symptom. Fixes #90879
This commit is contained in:
@@ -1436,9 +1436,50 @@ def _flush_one_shot_session_store(cli) -> None:
|
||||
logger.debug("one-shot end_session failed", exc_info=True)
|
||||
|
||||
|
||||
def _wait_for_oneshot_background_completions(cli) -> None:
|
||||
"""Bounded linger for notify_on_complete background processes (#90879).
|
||||
|
||||
A one-shot run (``-q`` / ``-Q``) that spawned bounded background work —
|
||||
most importantly a Bot Mode handoff reply via ``message_agent`` /
|
||||
``bot_relay``, spawned as ``terminal(background=true,
|
||||
notify_on_complete=true)`` — must not exit while that work is still
|
||||
running: the children write to pipes owned by this process and are
|
||||
destroyed shortly after it dies. Delegates the actual wait (and its
|
||||
``terminal.oneshot_completion_wait_seconds`` bound) to the process
|
||||
registry. Cheap no-op when nothing is pending.
|
||||
"""
|
||||
from tools.process_registry import process_registry
|
||||
|
||||
agent = getattr(cli, "agent", None)
|
||||
task_id = getattr(agent, "session_id", None) or getattr(cli, "session_id", None)
|
||||
# Wait on the whole registry, not just this task's processes: a one-shot
|
||||
# CLI process hosts exactly one agent, so every tracked process in this
|
||||
# interpreter was spawned by this run (task_id filtering would silently
|
||||
# skip processes registered before the session id settled).
|
||||
result = process_registry.wait_for_pending_completions(None)
|
||||
if result.get("waited"):
|
||||
logger.info(
|
||||
"One-shot exit linger for session %s: completed=%s timed_out=%s",
|
||||
task_id or "<unknown>",
|
||||
result.get("completed"),
|
||||
result.get("timed_out"),
|
||||
)
|
||||
|
||||
|
||||
def _finalize_single_query(cli) -> None:
|
||||
"""Close one-shot CLI resources before releasing the active session lease."""
|
||||
try:
|
||||
# Linger (bounded) for background processes the turn spawned with
|
||||
# notify_on_complete=true BEFORE any teardown. The one-shot parent
|
||||
# owns those children's stdout pipes; exiting now kills the delivery
|
||||
# a few seconds later. Bot Mode handoff replies dispatched from a
|
||||
# short-lived `hermes -p <bot> chat -Q` recipient (message_agent /
|
||||
# bot_relay spawns) are exactly this shape and were silently
|
||||
# destroyed on parent exit (#90879).
|
||||
try:
|
||||
_wait_for_oneshot_background_completions(cli)
|
||||
except Exception:
|
||||
logger.debug("one-shot background completion wait failed", exc_info=True)
|
||||
# Durable flush FIRST: memory-provider shutdown inside _run_cleanup
|
||||
# can issue aux-LLM calls, and nothing after it may fail in a way
|
||||
# that loses the turn (#88583).
|
||||
|
||||
Reference in New Issue
Block a user