fix(cli): one-shot runs linger for notify_on_complete background processes so Bot Mode replies survive parent exit

A Bot Mode agent invoked by a handoff runs as a short-lived
`hermes -p <bot> chat -Q --query-file ...` process. When it dispatches
its reply via message_agent / bot_relay — spawned as
terminal(background=true, notify_on_complete=true) per the Bot Chat
protocol — the one-shot parent exits as soon as the turn ends. The
reply child writes to a stdout pipe owned by the dying parent and is
destroyed a few seconds later, so the handoff reply is silently lost
while the sender waits for a notification that can never come (#90879).

Fix (class-wide, not DM-specific): before the one-shot exit paths tear
down, the parent now lingers — bounded by the new
terminal.oneshot_completion_wait_seconds config (default 600s, 0
disables) — for every tracked background process spawned with
notify_on_complete=true. Plain background processes (servers, daemons,
watch-pattern monitors) carry no completion contract and are never
waited on.

- tools/process_registry.py: ProcessRegistry.wait_for_pending_completions()
  — bounded, interrupt-safe wait over pending notify_on_complete
  sessions; reconciles orphaned-pipe exits (#17327) each pass so a
  wedged reader cannot burn the full bound; KeyboardInterrupt aborts
  the linger without skipping the caller's durable teardown.
- cli.py: _finalize_single_query() lingers first, before the durable
  session flush / cleanup (covers -q and -Q, i.e. the DM recipient
  shape and bot_relay waiter spawns from one-shot agents).
- hermes_cli/oneshot.py: same linger before agent.close() (which
  kill_all()s the task's processes) on the -z path.
- hermes_cli/config_defaults.py: terminal.oneshot_completion_wait_seconds.

Tests: tests/tools/test_oneshot_completion_linger.py — unit coverage of
the wait semantics (no-op, completion, timeout, task filter, disable,
config fallback, reconcile path), exit-path ordering contracts, and a
real-process E2E: a short-lived python parent spawns a delivery child
through the real ProcessRegistry, lingers, exits, and the delivery
completes; sabotaging the linger makes the same E2E reproduce the
destroyed-delivery symptom.

Fixes #90879
This commit is contained in:
Teknium
2026-08-23 03:01:29 -07:00
parent 3638961da8
commit 9e18197745
5 changed files with 581 additions and 0 deletions
+41
View File
@@ -1436,9 +1436,50 @@ def _flush_one_shot_session_store(cli) -> None:
logger.debug("one-shot end_session failed", exc_info=True)
def _wait_for_oneshot_background_completions(cli) -> None:
"""Bounded linger for notify_on_complete background processes (#90879).
A one-shot run (``-q`` / ``-Q``) that spawned bounded background work —
most importantly a Bot Mode handoff reply via ``message_agent`` /
``bot_relay``, spawned as ``terminal(background=true,
notify_on_complete=true)`` — must not exit while that work is still
running: the children write to pipes owned by this process and are
destroyed shortly after it dies. Delegates the actual wait (and its
``terminal.oneshot_completion_wait_seconds`` bound) to the process
registry. Cheap no-op when nothing is pending.
"""
from tools.process_registry import process_registry
agent = getattr(cli, "agent", None)
task_id = getattr(agent, "session_id", None) or getattr(cli, "session_id", None)
# Wait on the whole registry, not just this task's processes: a one-shot
# CLI process hosts exactly one agent, so every tracked process in this
# interpreter was spawned by this run (task_id filtering would silently
# skip processes registered before the session id settled).
result = process_registry.wait_for_pending_completions(None)
if result.get("waited"):
logger.info(
"One-shot exit linger for session %s: completed=%s timed_out=%s",
task_id or "<unknown>",
result.get("completed"),
result.get("timed_out"),
)
def _finalize_single_query(cli) -> None:
"""Close one-shot CLI resources before releasing the active session lease."""
try:
# Linger (bounded) for background processes the turn spawned with
# notify_on_complete=true BEFORE any teardown. The one-shot parent
# owns those children's stdout pipes; exiting now kills the delivery
# a few seconds later. Bot Mode handoff replies dispatched from a
# short-lived `hermes -p <bot> chat -Q` recipient (message_agent /
# bot_relay spawns) are exactly this shape and were silently
# destroyed on parent exit (#90879).
try:
_wait_for_oneshot_background_completions(cli)
except Exception:
logger.debug("one-shot background completion wait failed", exc_info=True)
# Durable flush FIRST: memory-provider shutdown inside _run_cleanup
# can issue aux-LLM calls, and nothing after it may fail in a way
# that loses the turn (#88583).