fix(cron): stop interrupted jobs from delivering their pre-kill output
Follow-up to the previous commit on #60432. The status-write guard (_consume_interrupted_flag, checked right before mark_job_run) closes the false-success bookkeeping gap, but run_one_job delivers its result BEFORE that check: delivery happens right after run_job() returns, mark_job_run happens at the very end. A job whose tool subprocess was killed mid-flight can still produce a plausible-looking final_response from the truncated output, and that response would reach the user via _deliver_result before the interrupted flag was ever consulted -- correct status in jobs.json, wrong message already sent. Adds _is_interrupted(), a non-destructive peek at the same _interrupted_job_ids set (_consume_interrupted_flag stays as the consuming, authoritative check right before the status write -- this needed a peek instead since the flag has to still be visible there). Checked right after save_job_output, before the deliver_content decision: if the run looked successful but was flagged interrupted, force success=False with an explicit interruption message. This routes delivery through the existing _summarize_cron_failure_for_delivery path (the same one a real failure already uses) instead of the raw final_response, so the user gets an honest "this run was interrupted" instead of a truncated/misleading result. Testing: 4 new tests in tests/cron/test_shutdown_interrupt.py -- _is_interrupted peek semantics (false/true/does-not-clear, as opposed to the consuming _consume_interrupted_flag), and the delivery-gate test itself, which mocks run_job to return a normal-looking success with a "plausible final response" while the job is pre-marked interrupted, and asserts _deliver_result receives the failure summary ("This run was interrupted.") instead, with the summarizer's error argument confirmed to mention the interruption. Fail-then-pass: reverted cron/scheduler.py only, the 4 new tests fail (3 on the missing _is_interrupted attribute, 1 -- the delivery-gate test -- on _summarize_cron_failure_for_delivery never being called, i.e. the raw response would have gone out); restored, all 16 tests in the file pass. Regression: tests/cron/ (683 tests) + test_cron_active_work_drain.py + test_gateway_shutdown.py + test_shutdown_cache_cleanup.py -- 11 pre-existing failures (Unix file-permission-bit and path-tilde assertions that don't apply on this Windows dev box), matching the same set already established as pre-existing in the prior commit's regression check. Zero new failures. Continues #60432
This commit is contained in:
@@ -371,6 +371,21 @@ def mark_running_jobs_interrupted(reason: str) -> list:
|
||||
return marked
|
||||
|
||||
|
||||
def _is_interrupted(job_id: str) -> bool:
|
||||
"""Non-destructive peek at whether the shutdown path has marked
|
||||
``job_id`` interrupted (see ``mark_running_jobs_interrupted``).
|
||||
|
||||
Called by ``run_one_job`` BEFORE it decides what to deliver — a job
|
||||
whose tool subprocess was killed mid-flight may still produce a
|
||||
plausible-looking ``final_response`` from the truncated output, and
|
||||
that must not go out to the user as if it were a normal result.
|
||||
Unlike ``_consume_interrupted_flag`` below, this does not clear the
|
||||
flag: the later, authoritative check (right before ``last_status`` is
|
||||
written) still needs to see it."""
|
||||
with _running_lock:
|
||||
return job_id in _interrupted_job_ids
|
||||
|
||||
|
||||
def _consume_interrupted_flag(job_id: str) -> bool:
|
||||
"""Return True and clear the flag if the shutdown path already marked
|
||||
``job_id`` interrupted (see ``mark_running_jobs_interrupted``).
|
||||
@@ -3420,6 +3435,20 @@ def run_one_job(job: dict, *, adapters=None, loop=None, verbose: bool = False) -
|
||||
if verbose:
|
||||
logger.info("Output saved to: %s", output_file)
|
||||
|
||||
# If the gateway shutdown killed this job's tool subprocess
|
||||
# mid-flight (#60432), the agent may still have produced a
|
||||
# plausible-looking final_response from the truncated output --
|
||||
# force the failure path so the delivered message is an honest
|
||||
# "this run was interrupted" summary instead of that response.
|
||||
# Peek-only: the flag stays set for the authoritative check
|
||||
# right before mark_job_run below.
|
||||
if success and _is_interrupted(job["id"]):
|
||||
success = False
|
||||
error = (
|
||||
"Interrupted by gateway shutdown before the run finished "
|
||||
"(tool subprocess was killed mid-flight)."
|
||||
)
|
||||
|
||||
# Deliver the final response to the origin/target chat.
|
||||
# If the agent responded with [SILENT], skip delivery (but
|
||||
# output is already saved above). Failed jobs always deliver.
|
||||
|
||||
Reference in New Issue
Block a user