Teknium
6767c06d34
Batches join on the slowest sibling before ONE consolidated block re-enters (the design: one results block per fan-out). Failure is the case that should not wait. In the 1,393-agent refactor run every wave-1 child died in the 08:29 401 storm; the parent learned of it at 09:36, when the batch's "unknown outcome" block finally arrived: 66 minutes of a dead wave with nothing running, the single largest idle gap of the run. _run_children_parallel, for DETACHED batches only (the sync path prints a completion line the parent is already watching), pushes ONE type="async_delegation" event with task_failure_notice=True and a single-entry results list when a child ends in a failure status while siblings are still pending. Same event shape and routing fields as the batch result, so every drain/ownership/format path treats it identically; the batch record is not finalized and its consolidated result still arrives unchanged. The formatter renders "[ASYNC DELEGATION TASK FAILED — <id>, task i/n]" with the task, status, error and live transcript path, and says the batch result is still coming. The TUI dedup key distinguishes a notice from the final result and from a sibling's notice. Live through the real detached dispatch (3 children, one fails at 0.1 s, two succeed at 8 s): notice drained at t+0.3 s, batch final at t+8.4 s. On main the parent hears nothing until t+8.4 s. Tests (2): the notice reaches the queue with the record's routing fields while the record stays running, and formats as an early warning; no notice for a finished batch; dedup key differs from the final result. Delegation/registry/notification suites (129 files) 1,344 passed.
49 lines
2.8 KiB
Python
49 lines
2.8 KiB
Python
"""A failed child of a still-running detached fan-out is surfaced to the parent immediately.
|
|
|
|
Batches join on the slowest sibling before ONE consolidated block re-enters. In a 1,393-agent run every wave-1
|
|
child died in a 401 storm at 08:29 and the parent learned of it at 09:36 from the batch's "unknown outcome"
|
|
block: 66 minutes of a dead wave with nothing running.
|
|
"""
|
|
import queue
|
|
from unittest.mock import patch
|
|
|
|
from tools import async_delegation as ad
|
|
from tools.process_registry_notifications import format_process_notification
|
|
from tui_gateway.session_notifications import _notification_event_dedup_key
|
|
|
|
|
|
def _record(status="running"):
|
|
return {"delegation_id": "deleg_x", "status": status, "is_batch": True, "goals": ["a", "b", "c"], "goal": "a",
|
|
"session_key": "sk", "origin_ui_session_id": "ui", "origin_session_id": "", "parent_session_id": "root",
|
|
"dispatched_at": 1.0, "role": "leaf", "model": "m", "context": None, "toolsets": None}
|
|
|
|
|
|
def test_failure_notice_reaches_the_queue_while_the_batch_keeps_running_and_formats_as_early_warning():
|
|
q = queue.Queue()
|
|
entry = {"task_index": 1, "status": "error", "error": "401 authentication_error: key invalid", "duration_seconds": 12.5,
|
|
"live_transcript": "/tmp/live/task-1.log"}
|
|
with patch.object(ad, "_records", {"deleg_x": _record()}), \
|
|
patch("tools.process_registry.process_registry") as reg:
|
|
reg.completion_queue = q
|
|
ad.push_task_failure_notice("deleg_x", entry, n_tasks=3)
|
|
assert ad._records["deleg_x"]["status"] == "running" # not finalized
|
|
evt = q.get_nowait()
|
|
assert evt["type"] == "async_delegation" and evt["task_failure_notice"] is True
|
|
assert (evt["session_key"], evt["origin_ui_session_id"], evt["parent_session_id"]) == ("sk", "ui", "root")
|
|
text = format_process_notification(evt)
|
|
assert text.startswith("[ASYNC DELEGATION TASK FAILED — deleg_x, task 2/3]")
|
|
assert "Task: b" in text and "401 authentication_error" in text and "/tmp/live/task-1.log" in text
|
|
assert "consolidated results will still arrive" in text
|
|
|
|
|
|
def test_notice_is_not_sent_for_a_finished_batch_and_does_not_dedup_against_the_final_result():
|
|
q = queue.Queue()
|
|
with patch.object(ad, "_records", {"deleg_x": _record(status="completed")}), \
|
|
patch("tools.process_registry.process_registry") as reg:
|
|
reg.completion_queue = q
|
|
ad.push_task_failure_notice("deleg_x", {"task_index": 0, "status": "error"}, n_tasks=3)
|
|
assert q.empty()
|
|
notice = {"type": "async_delegation", "delegation_id": "deleg_x", "task_failure_notice": True, "results": [{"task_index": 2}]}
|
|
final = {"type": "async_delegation", "delegation_id": "deleg_x", "is_batch": True, "results": []}
|
|
assert _notification_event_dedup_key(notice) != _notification_event_dedup_key(final)
|