fix(codex): retired stream requests must not synthesize a completed response

When a watchdog (TTFB / stream-idle / stale-call) force-closes a Codex
Responses request, the worker thread can still be draining SSE frames.
`_consume_codex_event_stream` returns `status=terminal_status`, which defaults
to `"completed"`, and its only truncation guard is
`if not saw_terminal and not output`. A mid-stream kill leaves
`saw_terminal=False` but `output`/text non-empty, so the partial text came back
as a `finish_reason=stop` response and got persisted as a finished assistant
turn — a long reply just stops mid-sentence with no error surfaced.

Observed as a long generation dying at `1. Create (6/6)` and never emitting its
end marker, with the truncated text already stored in state.db.

Fix: publish a per-request retirement token so the worker can tell it has been
retired.

- `agent/chat_completion_helpers.py`: `interruptible_api_call` installs
  `agent._active_codex_stream_request_token` before handing off to the worker
  (codex_responses only) and clears it at all four kill sites plus the worker's
  own `finally`. Retirement is cleared BEFORE `_close_request_client_once`,
  which can raise — every other call site wraps it in try/except, and a leaked
  token would let a later worker mistake itself for the owning attempt. The
  request-local `_codex_request_retired` mirror also swallows the transport
  error our own force-close causes, so the worker's local error cannot replace
  the watchdog's retryable TimeoutError (same split as `_request_cancelled`).
- `agent/codex_runtime.py`: `run_codex_stream` captures the token and raises
  `TimeoutError` from `interrupt_check` when it no longer owns the request —
  raising rather than breaking, because a break returns the partial `final`.
  The four stream callbacks also drop post-retirement frames so an abandoned
  attempt cannot stream tokens into the live turn's bubble (the gateway caches
  AIAgent instances per session).

`TimeoutError` is not an httpx / ConnectionError / RuntimeError subclass, so it
passes through the four `except` clauses around the consume call untouched.
No token installed (auxiliary callers such as `handle_max_iterations` drive
`_run_codex_stream` directly) means every check passes — behavior unchanged.

Tests: 5 new cases. Retirement raises instead of returning partial output;
post-retirement deltas stop reaching callbacks; the no-token path keeps its
existing terminal-frame tolerance; the watchdog installs and clears the token;
non-codex api_modes install nothing. A `_LazyCreateStream` helper is needed
because `_FakeCreateStream` materializes events in __init__, which would run
the retirement side effect before consumption starts.
This commit is contained in:
AgentLinker
2026-08-31 00:40:35 +08:00
committed by Teknium
parent c905c2b4b5
commit cac9db7caf
4 changed files with 297 additions and 7 deletions
+32 -1
View File
@@ -1101,7 +1101,9 @@ def _consume_codex_event_stream(
* ``on_first_delta()`` — one-shot, fires on the first text delta only.
* ``on_event(event)`` — fires for every event before any other processing.
Used for watchdog activity, debug logging, anything wire-shape-agnostic.
* ``interrupt_check()`` — returns True to break the loop early.
* ``interrupt_check()`` — returns True to break the loop early, or raises
``TimeoutError`` / ``InterruptedError`` for request-retirement control
flow that must not be converted into a partial final response.
"""
collected_output_items: List[Any] = []
# output_index of each collected_output_items entry, appended in lockstep
@@ -1606,18 +1608,39 @@ def run_codex_stream(agent, api_kwargs: dict, client: Any = None, on_first_delta
max_stream_retries = 1
# Accumulate streamed text so callers / compat shims can read it.
agent._codex_streamed_text_parts: list = []
# Retirement token for THIS request, installed by
# ``interruptible_api_call`` before it hands off to the worker thread. When
# a watchdog (TTFB / stream-idle / stale-call) kills the connection it
# clears the agent-level token, so a worker that is still draining frames
# can tell it has been retired. ``None`` means no watchdog owns this call
# (auxiliary callers drive this function directly) — then every check
# passes and behavior is unchanged.
request_token = getattr(agent, "_active_codex_stream_request_token", None)
def _request_is_current() -> bool:
if request_token is None:
return True
return getattr(agent, "_active_codex_stream_request_token", None) is request_token
def _on_text_delta(text: str) -> None:
if not _request_is_current():
return
agent._codex_streamed_text_parts.append(text)
agent._fire_stream_delta(text)
def _on_reasoning_delta(text: str) -> None:
if not _request_is_current():
return
agent._fire_reasoning_delta(text)
def _on_commentary_message(text: str) -> None:
if not _request_is_current():
return
agent._fire_streamed_codex_commentary(text)
def _on_event(event: Any) -> None:
if not _request_is_current():
return
# TTFB watchdog and activity touch — runs once per SSE event.
agent._codex_stream_last_event_ts = time.time()
agent._touch_activity("receiving stream response")
@@ -1721,6 +1744,14 @@ def run_codex_stream(agent, api_kwargs: dict, client: Any = None, on_first_delta
raise
def _interrupt_or_superseded() -> bool:
# A retired request must NOT break out of the consume loop: breaking
# returns the partial `final` (status defaults to "completed"), which
# the caller persists as a finished assistant turn. Raise so the
# watchdog's own TimeoutError is what the retry path sees.
if not _request_is_current():
raise TimeoutError(
"Codex Responses stream request retired before terminal response"
)
return bool(agent._interrupt_requested)
try: