Skip checkpoint evaluation when a turn is interrupted so cancellation rows
remain durable without urging continued execution. The existing minimal-agent
interrupt regression also avoids dereferencing an absent iteration budget.
Consolidate the warning coverage into two invariants, including real SQLite
readback and dispatcher/child scope controls. Cold-start tool availability
between construction cases to model independent worker processes. Place ratio
normalization beside the existing iteration budget instead of growing init.
Real cancelled-tool A/B against current main, draft, and fix: three cancelled
rows and zero writes on all arms; persisted checkpoint notices 0 / 1 / 0.
Repeated scripted HTTP/SQLite loop A/B preserves completion opportunity,
ordinary default-off behavior, and blocked/two-failure exhaustion behavior.
Local targeted run initially passed 15 cases with one fixture cache-isolation
failure; corrected target and inherited affected suites remain queued behind
the campaign lock. This commit is not a CI-green or merge-ready claim.
Give dispatcher-owned workers a tool-capable reporting opportunity before the
hard iteration cap, without accepting arbitrary diffs or weakening failure
counting. Add opt-in per-turn iteration checkpoints for ordinary agents.
Persist checkpoint text with the fresh tool result, never rewrite cached rows.
Salvages the opt-in ratio and per-turn reset implementation from #104683;
credits the earlier default-off signpost proposal in #92438.
Local fixture wire A/B: Kanban ready/1 failure -> done/0; deliberately stuck
workers still reach blocked/2 after two runs. Default-off control unchanged.
Targeted and affected-directory suites queued behind campaign test lock.
Co-authored-by: fangliquanflq <fangliquan@qq.com>
Co-authored-by: C. Michael Gibbs <252231331+MikeGibbsOnyx@users.noreply.github.com>
fix(nous): adopt a same-account fresh key before expiry and start the keepalive in every process (620 hourly 401s → 0; review-found account takeover closed)
Portal will serve anthropic/* from more than one upstream (OpenRouter
passthrough today; GMI/Vertex once it is back online). The native Messages
wire is the better transport but is only safe where the upstream keeps
prompt-cache routing sticky: measured false on the OpenRouter path (14-20% of
consecutive calls re-write the previous turn; #104284 moved the default to
chat), untested on GMI. Hermes cannot see the upstream in the request, only
in the response: OpenRouter stamps `provider` (chat wire) and mints
`gen-<unix>-<rand>` ids; GMI/Vertex returns Anthropic-native `msg_...` ids
and no provider.
`auto` therefore starts every session on chat (correct on both upstreams),
classifies the first response, and switches that session to native only
when the upstream is GMI AND `agent/nous_wire.py::GMI_NATIVE_WIRE_CLEARED`
is True. The switch is scheduled at response time and applied at the start
of the next iteration (turn_iteration_prep), so nothing is rebuilt while a
response is being consumed; it goes through switch_model so the client,
cache policy and _primary_runtime stay consistent. One decision per session,
call 1 only; unknown upstream never switches; a failed switch logs and stays.
GMI_NATIVE_WIRE_CLEARED is False: until the 20x6 concurrency probe
(evals/postmortem/live_ab) is clean on a GMI-served anthropic/* id on the
native wire, `auto` behaves exactly like `chat`. Flipping it is the whole
rollout once GMI is measured. Default stays `chat`.
Tests (17): classifier on real Portal response shapes from both wires and
both upstreams; chat for openrouter/unknown, GMI gated on the flag; one
decision per session, call 1 only, explicit chat/native never auto-switch,
other providers/models untouched, switch failure swallowed and final;
record_response_usage on a real AIAgent invokes the hook once.
Live (auto, real Portal, Fable 5.1): arm A, real classification
(OpenRouter today) - stays on chat through a tool loop and a second turn,
cache 97-99%. Arm B, classifier forced to gmi with the flag on - call 1 on
chat, switch applied before call 2, calls 2-3 on the native wire in the
same session, tool result and both turns correct, cache 97-99%. An earlier
shape that switched inside the response path broke call 1 (SimpleNamespace
has no .content); the scheduled apply is why.
The Nous agent key lives 3,599 s. In the 1,393-agent refactor run every
in-process agent learned about the hourly expiry from its own 401: 620
authentication_error 401s in the logged window (177 in one hour), each a
failed attempt the model never saw, and the credential pool benched the
sole credential for all of them at once. At 08:30 the storm took the
parent process down.
Two gaps. The proactive refresher (hermes_cli/nous_auth_keepalive.py) is
started only by the gateway and the web server; the CLI process, and every
subagent built inside it, never started it. And even with a fresh key in
the store, nothing adopted it before a request: a request went out with
whatever key the agent was constructed with until it 401'd.
Now _finalize_routing starts the keepalive (idempotent, process-wide,
daemon) whenever an agent resolves onto provider "nous", and
prepare_iteration calls _adopt_nous_key_before_expiry(): the agent key is
a JWT, its exp is read locally, and inside a 180 s skew the store is
re-read under the auth-store lock with force_refresh=False, so the
keepalive's (or a peer's) fresh key is adopted without a POST; when none
exists, ONE refresh runs there instead of N reactive ones after N 401s.
_try_refresh_nous_client_credentials no longer rebuilds the client when
the store returns the key already in hand.
Live A/B (local server: 401s any bearer but FRESH; store patched to hold
FRESH; 40 agents holding a JWT that expires in 30 s fire concurrently):
main 40 x 401 then recover, branch 0 x 401.
Tests (4): far from expiry the store is not touched; inside the skew the
store's fresh key is adopted with force_refresh=False; the same key back
from the store is not re-adopted; a real AIAgent routed to nous starts
the keepalive and one routed to openrouter does not.
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
- turn_context_compaction: shared home for _clear_overflow_warn,
_reset_retry_state_after_compaction, _blocked_compress_reason,
_apply_grown_window, _refund_api_call (used by turn_preflight and the
turn-start preflight passes).
- turn_context: _with_persist_lock + _clear_staged_cli_input_if_persisted +
two try/except/finally wrappers -> one _persist_under_lock; MoA sanitize
model extracted to _sanitize_model_for; predicates flattened.
- turn_preflight: PreflightVerdict merged into PreflightGateVerdict (the
gate's carrier) and mutated in place; duplicate guard chain + refund blocks
collapsed onto the shared helpers.
- turn_preflight_gate: builds the carrier once, no re-unpack of the inner
verdict.
- turn_iteration_prep: step-callback tool-round scan and steer injection lifted
into helpers; dead _verdict closures on always-fallthrough paths removed.
Teknium