The async-delegation watcher drained the completion queue as a batch but
then delivered each event as its own synthetic turn, flooding the session
when a fan-out of background subagents finished together. Builds on the
per-process completion batching salvaged from PR #71898 (thanks
@yuzilongleif-collab) which coalesces concurrent _run_process_watcher
completions behind a short per-route fan-in window.
This commit adds the async-delegation half: group the drained batch by
full routing key (session_key + parent_session_id + platform/chat/thread/
user) and inject ONE consolidated turn per group. Durable-ack handling
stays honest: sibling rows are claimed up front via claim_event_delivery;
rows another consumer owns are excluded from the consolidated text (no
double-delivery); sibling claims are acknowledged only after adapter
acceptance and released (still pending) on failure. Events for different
sessions never coalesce, and a single-event group rides the existing
per-event path unchanged (latency and text identical).
Tests: 3 same-tick events -> exactly one adapter.handle_message carrying
all 3 results with all 3 durable rows delivered; 2 sessions -> 2 turns;
single-event path unchanged; failed batch releases claims and retries;
foreign-claimed sibling excluded and left pending.
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
Adapter acceptance is not proof of delivery: the inner #55578 resolver can
still fail closed inside the message pipeline after the adapter accepted the
synthetic event, which falsely acknowledged the durable row as delivered and
silently discarded the delegation result.
Pre-flight the delivery target in _deliver_completion_notification before
adapter acceptance (adapted from #65838 by @henrynguyeninfo1):
- live parent or verified live compression tip -> deliver (inner resolver
still owns the actual route retarget)
- explicit-reset / unknown parent -> terminal 'dropped' disposition via new
drop_completion_delivery() (not falsely 'delivered', not eternally
'pending')
- transient uncertainty (DB error, mid-flight rotation without a visible
continuation) -> release the claim for retry
release_completion_delivery() now converges to a terminal 'dropped' state
once _MAX_DELIVERY_ATTEMPTS is exhausted, so an undeliverable completion
cannot replay on every gateway restart forever.
Teknium