The corrupt-cause recovery guidance hardcoded `~/.hermes/backups/` while
every other path in the same message follows the active HERMES_HOME
(`{db_path}` is already interpolated). A custom-home or named-profile
deployment was told to restore from a directory that may not exist at all,
mid data-loss incident. Both sites (turn-completion explainer and gateway
startup broadcast) now interpolate `<hermes_root>/backups` via
get_default_hermes_root(), matching hermes_cli/backup.py's real backup
location.
Fixes#104250
The memory_tool schema advertises new_text as an alias for content, and
memory_tool resolves it when content is None. But the table-driven inline
executor's arg_specs (agent/inline_tool_executors.py) did not list new_text,
so _call_tool's allowlist silently dropped it: a replace call using the
documented alias reached memory_tool with both fields None and failed with
"content is required for 'replace' action." — even though the caller
supplied the value. Forward new_text alongside content/old_text so the
documented alias fires and content still wins when both are set, matching
what the batch path (op.get("content") or op.get("new_text")) already
accepts.
Slim salvage of #104688: place project context before workspace state and
keep cwd outside the stable prefix. Put runtime hints behind a final
renderer-owned boundary so quoted operator, memory, plugin and embedder
examples cannot override the persisted runtime cwd or identity fields.
Retain legacy unmarked prompt validation, add two invariant tests and a
credential-free real-AIAgent/git-worktree replay harness. No provider
cache-hit or billing measurements are claimed.
Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
Co-authored-by: HexLab98 <liruixinch@outlook.com>
Track raw task identities across an agent's turns and match them against
process owner_task_id during close. Session IDs and shared terminal keys
are not process ownership, so the old bulk cleanup missed delegated work.
Preserve parent/sibling processes and consume teardown notifications.
Move task-resource cleanup into the lifecycle mixin, add real-process
isolation regressions, and document background process lifetime.
Use request-local stream silence for the waiting notice, preserving quiet
activity heartbeats and all existing watchdog policies. Distinguish a stream
that stopped from a request with no response, and clear this request's notice
on the next poll when events resume. Existing fresh first-event retry phases
also reset the display; recovery deadlines explicitly use total call elapsed.
Add two invariant tests (eight cases), proven red on main, plus EN/ZH docs.
Local SDK SSE through classic CLI callbacks in a PTY verifies active reasoning,
true silence, and an already-visible warning clearing on resumed reasoning.
Related: #92657 addresses repeated waiting notices; its phase deduplication
still labels active streams as no response and is not incorporated here.
The image-part membership test in _payload_chars() ran on every dict in
the request, so a tool JSON Schema with a parameter named "type" (its
value is the sub-schema dict) or a multi-type value like
["string", "null"] raised TypeError: unhashable type before the
request reached the provider. Guard the membership test with
isinstance(str): structured "type" values are payload data, not
content parts, and keep the legacy walk for them.
Codex OAuth caps gpt-6-astra at the same 272K window as gpt-5.4/5.5/5.6,
so the global 50% trigger compacted at ~136K. Extend the existing
codex_gpt55_autoraise gate to any slug containing "astra" (minus the
opt-in -900k picker variants, which already unlock the wider window).
Other routes (OpenAI direct, OpenRouter) keep the user threshold.
estimate_request_context_tokens (the stale-stream / non-stream watchdog and Codex TTFB floor
estimator) did len(str(payload)) // 4 over the wire payload, so one native screenshot read as
~100K tokens and selected the 1200s giant-conversation floor while the provider's real prompt
was ~2K (#63871, #76411). Image content parts on both wire shapes (Chat image_url / Responses
input_image) now cost the per-image price learned from provider usage (agent/image_token_cost,
bound per turn; worker threads inherit it via copy_context); base64-looking STRINGS stay text
and tool schemas mentioning image are not images.
A/B, one 300KB screenshot: main estimate 100,545 -> codex floor 1200s / stale 300s; now 2,025
-> floor 0s / stale 180s.
Design and boundary (structured parts only, tools/instructions opaque, plain data-URL strings
stay text) from #76471 by @crdesign8; re-authored to reuse the learned cost instead of a second
fixed constant, and trimmed to 2 invariant tests.
The Chat Completions schema has no `name` field on `role: tool` (only on
the long-removed `role: function`), but Hermes carries the tool name over
onto the result message. Permissive providers ignore it; strict ones
(aki.io) reject the whole payload with `contains item with unknown key
name`, which breaks every tool call in the session.
Follow-up to the review on #51365:
- The strip now goes through the copy-on-write `mutable_msg()` path in
`convert_messages()`, preserving the identity/copy-on-write contract
instead of mutating `msg` in place.
- `handle_max_iterations()` hand-builds its summary payload and calls
`chat.completions.create()` directly, bypassing the transport, so it
leaked `name` even with the transport fixed. It now mirrors the same
role-qualified removal, next to the existing tool_name/codex_*/timestamp
strips.
The removal is role-qualified: `name` stays on user/assistant messages,
where it is schema-valid.
Regression coverage on both paths; both tests fail without the fix.
A flat per-image constant (1500 in the trigger estimator, 1600 in the tail-budget walk) is wrong in
both directions: a screenshot costs ~1,100 tokens on one provider and 4,000+ on a local mmproj
model. In a GUI loop on a 64K window the estimate sat at ~20K while the real prompt passed 80K,
so compaction never fired and the provider rejected every request (#70328).
The provider prices every image exactly on the request that carries it, so the cost is
observable from usage alone, with no vendor formula: with a fresh usage anchor, the residual
between the next real prompt_tokens and anchor + text-only delta is the price of the N images
that delta introduced.
- agent/image_token_cost.py: calibrate_from_usage() runs in record_response_usage before the new
anchor is captured; the learned value (EMA, plausibility-banded) is kept per model@host in
~/.hermes/cache/image_token_costs.json and bound per turn through a ContextVar.
- estimate_messages_tokens_rough, _content_length_for_budget (tail walk) and gateway hygiene all
read the same bound value, so trigger and walk agree; the per-message memo now caches text
tokens and image COUNT so a recalibration re-prices cached rows.
- One flat default (1500) remains only until the first vision turn; the duplicate 1600 is gone.
evals/token_accounting/ab_image_cost_calibration.py (real AIAgent, fake provider pricing images
at 4,000, one screenshot per turn, 64K window): main learns nothing (1500) and the tail walk
under-prices its own protected tail by 56.5%; this branch learns 4,374 after one vision turn
and the walk's error is +8.5%.
Reporter and first-fix credit: @JonthanaHanh (#70328, #70463).
Diagnosing the 1,393-agent run's cache misses took a state.db join and a live
probe because none of the three were on the one line we log per call:
write=<n> cache_creation tokens; a write costs 50x a read, so this is the
money, and "read stuck, write large" on consecutive lines is a
routing miss visible without a probe
id=<id> the provider's response id (Anthropic msg_..., OpenRouter/Nous
gen-...): what a provider needs to look a request up
upstream=<n> who actually served, when the route reports it (OpenRouter's
`provider`); how we learned GMI was not serving
Fields are appended to the existing line and omitted when absent, so every
existing parser (evals/postmortem, the two cache_prefix probes) keeps
matching; the forensics parser reads them when present.
The streamed chat path fabricated `id="stream-<uuid>"` and dropped the
chunks' id/provider on the floor; it now keeps the first chunk's id and
provider, falling back to the fabricated id only when the stream never sent
one. Nothing depended on the prefix.
Live (Nous, Fable 5.1, both wires): chat -> `write=3759
id=gen-1788727882-... upstream=Anthropic`; native -> `id=gen-1788727891-...`
(the Anthropic Message object's id was already real). Tests: fields present
and omitted, prefix unchanged; forensics parser reads new and old lines.
The prefetch spill is the fix; the is_builtin registration flag and duplicate-
instance rejection defended against a provider naming itself "builtin", which no
live path can do (only a test fixture does). Restore the name-based check and the
four test files that only changed for the new kwarg; keep two spill invariants.
Codex Responses reasoning and compaction items carry ciphertext the provider prices by its own
token count, never by bytes; a single native compaction checkpoint is ~5M chars, which the
bytes/4 estimator turned into ~1.29M "tokens" against a 204K threshold (#100611). #104192 deferred
that decision for one request; this removes the mis-pricing at the source so the preflight
estimator and the tail-budget walk agree (a mismatched size class protects blob-heavy rows as
"small" and compaction re-fires). Only real usage prices these items, and the usage anchor carries
that price forward.
evals/native_compaction/ab_checkpoint_preflight.py: preflight estimate after checkpoint
1,292,413 -> 58 rough tokens; the over-threshold negative arm still compresses.
Two parallel "real usage" mechanisms fought each other: the usage anchor (real + delta) and the
compressor's rough/real projection (should_defer_preflight_to_real_usage with
last_rough_tokens_when_real_prompt_fit / _pending_request_rough_tokens / note_request_rough_estimate
baselines). The projection stored an anchored, real-scale figure as its "rough" baseline, so a
rewind that invalidated the anchor produced phantom growth and a spurious compaction (#103391).
Now there is one authority:
- Post-tool gate (turn_preflight.compress_after_tool_results): anchored figure first (the raw
last_prompt_tokens ignored the tool results just appended), then real, then rough.
- Gateway hygiene (run_turn._hmwa_hygiene_plan): real session count, else the anchor persisted on
the session row, else rough.
- Preflight / pre-API gates: an anchored figure is never deferred. A whole-context rough estimate
over threshold waits ONE request for the provider's real count instead of compressing on a guess
(first request, rewind/edit-resend, reloaded history without a persisted anchor).
- The wait is one request, never a disable: a provider that omits usage
(note_usage_less_response, #2153 class), a real reading already over threshold, a rough figure
past the whole window, and provider-proven overflow all compress immediately; the post-compaction
latch (#36718 / #104192) is unchanged.
- Projection baselines and their bookkeeping deleted (-101 LOC in context_compressor); the fixtures
that scripted whole-history estimates now state the fact they relied on (provider omits usage).
Fixes#103391 (closes#103397 by construction — the baseline it repaired no longer exists).
The usage anchor (real usage.prompt_tokens + delta estimate of what was appended since)
identified the priced transcript by id() of the last message, so it was None on EVERY
gateway turn (history is re-read from the DB each turn) and in every fresh process
(--resume, desktop per-turn serve). Those are exactly the surfaces where the bytes/4
estimate then fired local compression against payloads the provider priced far under
threshold (#99421, #104462).
- agent/usage_anchor.py owns the anchor: content fingerprint instead of id(), persisted on
the session row (model_config._usage_anchor) via set_usage_anchor(), restored on the first
resumed turn while the durable transcript still matches, cleared with the row on
compaction / codex-native rewrite / session reset.
- Callers repointed from model_metadata (the compat table follows).
Design and persistence slot from #99585 by @686f6c61; re-authored against the Sep 2026
layout (the branch predates the model_metadata / agent_init split).
The codex_app_server runtime bypasses the conversation loop, so the
usage-anchored context accounting captured there never ran: agent._usage_anchor
stayed None forever. Every preflight estimate therefore fell back to the rough
mirror-transcript heuristic, which is deliberately never compacted on this
runtime (_record_codex_app_server_compaction preserves the mirror), so it grows
monotonically while the real thread may be tiny or freshly compacted. With
compression.codex_app_server_auto=hermes that estimate alone tripped the
threshold and fired thread/compact/start on nearly every turn of a long-lived
session (#100381).
Mirror the main loop's post-response capture: _record_codex_app_server_usage
now snapshots the reported thread usage into agent._usage_anchor (base prompt
+ completion exactly as the provider counted them, with estimation confined to
messages appended since). A usage-less turn keeps the previous anchor, and the
post-compaction invalidation site is unchanged.
Older desktop builds appended a frozen "## Messaging other agents" section (roster
included) to SOUL.md. Since the server started injecting the live section into Bot Chat
sessions, that copy did two wrong things: every CLI/TUI/messenger session paid ~600 tok
for a bot-only protocol, and in Bot Chat itself the probe went silent when SOUL carried
the heading, so bots saw the stale roster instead of the live one.
- load_soul_md strips the legacy section at read time (covers un-migrated profiles and
the ambient-home edge cases the same way the SOUL isolation fix does)
- bot_mode_probe drops the SOUL-carries-heading suppression; a SOUL-era stored Bot Chat
prompt now counts as legacy and is upgraded once (stamped, so it cannot loop)
- config migration v41 rewrites SOUL.md across the default + every profile once
With a request-local watchdog state on every codex request, agent._codex_stream_last_event_ts / _last_progress_ts had no remaining reader; the state-None snapshot only runs for non-codex requests where no watchdog consults it.
- Explicit-vs-implicit idle timeout detection uses env_float with a sentinel
instead of a hand-rolled parse (unset and unparseable both mean implicit).
- _on_event writes the agent-level timestamps only when there is no
request-local watchdog state; with state present they were dual bookkeeping
nobody read (the snapshot prefers the state).
- _abort_request keeps main's close-then-retire order: the reorder had no
test teeth and is a separate concern from #90449.
fix(nous): adopt a same-account fresh key before expiry and start the keepalive in every process (620 hourly 401s → 0; review-found account takeover closed)
fix(delegate): nested orchestrators get their workers' results back — delegate_task exempt from the 420 s tool deadline; summary budget uses current prompt, not the session sum
- acquire(): the discard-and-recurse path becomes one more iteration of the existing wait loop;
the two inline 'with lifecycle_lock: _teardown(db)' copies reuse _teardown_generation, and the
type-narrowing asserts go away with the recursion. release() reads generation.path directly.
- Restore the real os.replace inode swaps in test_state_db_file_identity.py and the registry tests:
those files carry no windows_only marker so they never run on Windows, and the monkeypatched
predicate stopped exercising the stat->identity->halt path anywhere.
- Drop the auto-archive change and its 4 tests: on main the sweep gets a bare SessionDB and
db.close() already releases a registry-shared handle, so the described NameError leak only
existed on this branch's earlier head. trace_upload: acquire(None) already defaults.
- Trim the barrier tests to the invariant pair (retired drain must not lift a pending current
teardown; replacement not published before the last close settles) plus the raising-close
settlement; comments say the WHY once.
The #102827 corruption is pure zero holes -- frames lost across a WAL
generation. SessionDB.close() produces exactly that when it runs against a
file another live handle is still writing: PRAGMA wal_checkpoint(PASSIVE),
then the connection close that lets SQLite unlink -wal/-shm. The dangerous
event is a physical close overlapping any other live physical lifetime for
the same path, so both sides of it are closed here.
Late write vs. close: a cron watchdog timeout only stops waiting, and
ThreadPoolExecutor.shutdown(wait=False) cannot interrupt a worker already
inside run_conversation. The agent and its registry reference are now held
until that worker's Future completes, so its last frames land before any
checkpoint.
Close vs. open: the per-path barrier now COUNTS admitted teardowns. A path
can own several closes at once -- the current generation's final release and
a retired generation's drain are admitted independently under the registry
lock, and the per-path mutex only serializes teardowns that already entered
it. With one bare event per path, a releasing thread descheduled between
generation removal and the mutex let the next teardown to settle remove and
signal the shared event: close_all() returned over a pending close and
acquire() published a replacement writer on top of a handle still inside
checkpoint/unlink. _TeardownBarrier tracks event + pending count,
_admit_teardown_locked registers each close in the same lock section that
removes the generation, and only the last settled teardown lifts the
barrier. Physical I/O stays outside the registry lock and unrelated paths
still progress independently.
The auto-archive sweep called release_or_close in its finally while the
import was local to a different function, so every eligible sweep raised
NameError, the outer except Exception swallowed it at debug level, and the
borrowed registry reference was never returned -- a holder leak that pins a
retired generation open. The helper is now bound in the calling scope.
Remaining in-process writable SessionDB() call sites (trace upload, the
API-server profile cache, the web-server writable paths, startup schema
reconcile) go through the canonical registry acquire/release_or_close, and
gateway maintenance borrows pinned handles instead of iterating an unpinned
snapshot.
Regressions: overlapping final releases of the current and retired
generations in both orderings with the first paused before the lifecycle
mutex, teardown-error settlement, an unrelated-path control, and refcount
assertions for the auto-archive sweep on success, on failure, across
repeated sweeps and with auto-archive disabled.
Fixes#102827
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzxCWw6SuHXhXMdkiEwMa2
The pin from the previous commit lives in _SESSION_STATE but nothing cleared it, so a CLI
/new, /resume or /branch (same AIAgent, reset_session_state + _invalidate_system_prompt)
replayed the previous session's git snapshot into the new session's prompt. Clear it in
reset_session_state next to the other session anchors; one invariant test (red without it).
The TUI/Desktop session.context_breakdown RPC ran the prompt builder on the RPC thread
with no session cwd bound, so it re-probed against the backend's cwd and overwrote the
session's pin — one /context between compactions restored the divergence this fix removes.
Bind the session context around the build like the live rebuild in server.py does.
Also: trim _coding_parts' docstring to the WHY, drop the isinstance/len guard on a value only
this function writes, and remove the tests' assertions on the private pin shape.
- pin session-start workspace snapshot (_frozen_workspace_snapshot) on first build so dynamic git probes don't churn Tier 2 during compaction rebuilds in active coding sessions
- replay pinned snapshot across rebuilds when cwd matches; re-probe only on cwd switch
- honor coding_context invariant that workspace is a session-start snapshot, preventing prefix-cache divergence at offset ~4,662
- add invariant tests covering workspace snapshot pinning across git mutations and cwd transitions
- addresses upstream prompt divergence identified in #103326
Portal will serve anthropic/* from more than one upstream (OpenRouter
passthrough today; GMI/Vertex once it is back online). The native Messages
wire is the better transport but is only safe where the upstream keeps
prompt-cache routing sticky: measured false on the OpenRouter path (14-20% of
consecutive calls re-write the previous turn; #104284 moved the default to
chat), untested on GMI. Hermes cannot see the upstream in the request, only
in the response: OpenRouter stamps `provider` (chat wire) and mints
`gen-<unix>-<rand>` ids; GMI/Vertex returns Anthropic-native `msg_...` ids
and no provider.
`auto` therefore starts every session on chat (correct on both upstreams),
classifies the first response, and switches that session to native only
when the upstream is GMI AND `agent/nous_wire.py::GMI_NATIVE_WIRE_CLEARED`
is True. The switch is scheduled at response time and applied at the start
of the next iteration (turn_iteration_prep), so nothing is rebuilt while a
response is being consumed; it goes through switch_model so the client,
cache policy and _primary_runtime stay consistent. One decision per session,
call 1 only; unknown upstream never switches; a failed switch logs and stays.
GMI_NATIVE_WIRE_CLEARED is False: until the 20x6 concurrency probe
(evals/postmortem/live_ab) is clean on a GMI-served anthropic/* id on the
native wire, `auto` behaves exactly like `chat`. Flipping it is the whole
rollout once GMI is measured. Default stays `chat`.
Tests (17): classifier on real Portal response shapes from both wires and
both upstreams; chat for openrouter/unknown, GMI gated on the flag; one
decision per session, call 1 only, explicit chat/native never auto-switch,
other providers/models untouched, switch failure swallowed and final;
record_response_usage on a real AIAgent invokes the hook once.
Live (auto, real Portal, Fable 5.1): arm A, real classification
(OpenRouter today) - stays on chat through a tool loop and a second turn,
cache 97-99%. Arm B, classifier forced to gmi with the flag on - call 1 on
chat, switch applied before call 2, calls 2-3 on the native wire in the
same session, tool result and both turns correct, cache 97-99%. An earlier
shape that switched inside the response path broke call 1 (SimpleNamespace
has no .content); the scheduled apply is why.
A native Responses compaction checkpoint is opaque ciphertext; the rough
preflight estimator counts it as text (5.17M chars -> ~1.29M tokens against
a 204K trigger) and fires local compression on a request whose real prompt is
~116K. Arm the existing one-response real-usage latch when a replayable
checkpoint is captured (build_assistant_message) or restored into a fresh
agent (_hydrate_from_history), honor it in the post-tool gate and idle
compaction, and require non-empty encrypted_content for a checkpoint.
Squash of the author's source commits from #100642 (0e3c234ea0, 771e1b3365,
bb1505a119) plus the fdf140c81d test refresh, re-based onto current main by
patch application. Source delta is byte-identical to the PR head d6ce3e236d.
Fixes#100611
/simplify-code pass on the Retry-After salvage stack:
- compute_error_backoff now decides "no usable cooldown" exactly once:
a parsed 0.0 (retry-after: 0, or an HTTP-date in the past, which the
shared parser clamps to 0) is treated as absent instead of falling
through an accidental falsy check — prevents a hot-loop retry and
keeps the sentinel semantics uniform (is None / is not None at all
four sites).
- Comment corrected: the sibling unwrap lives in
extract_api_error_context, not _extract_rate_limit_context.
- Test driver deduped: one _retryable_error factory + one _drive_once
shared by the 429 and 524 tests; added the over-cap (3600 → 600) and
no-cooldown fallback rows requested in the #103722 review.
Mutation checks: nested-unwrap neutralized → nested row red; cap
removed → over-cap row red; restored → all 298 green.
Salvage follow-ups on #88236 (krunkosaurus):
- Some providers nest the cooldown as body["error"]["retry_after"] (the
same unwrap _extract_rate_limit_context already uses); only the top-level
shape was read, so those errors silently fell back to jittered backoff.
- A 5xx Retry-After can reach the 600s cap; that wait was buffered
(replayed only on terminal failure), leaving the user silent for
minutes. Long provider cooldowns now emit immediately, mirroring the
zai_coding_overload_long path. Jittered waits keep the old buffering.
Test widened with the nested-body parametrize row; proven red when the
unwrap is neutralized.
Retryable server errors can carry provider cooldowns just like 429 responses. Parse Retry-After from response headers or structured error bodies before falling back to jittered backoff, and cover HTTP 524 behavior with runtime regression tests.
The helper returned container names OR a synthetic pseudo-name so the caller
would refuse; on a hung daemon that rendered as "already has running
container(s) (<docker compose ps timed out ...>)". It now returns the refusal
reason or None, and the caller prints the reason as given.
Review follow-up on the salvaged #103577 guard. Fail-open on a hung or failing
`docker compose ps` reopened the #103567 window: containers may be live and
unobservable, and `docker compose build` would fail against the same daemon
anyway, so refusing loses nothing. Only a missing docker binary falls through.
The guard now also runs only when a mutating phase (build, or start without
skip_start) is selected. Tests collapsed to three contracts: refusal spawns
nothing mutating (running / timed out / failed probe), fall-through (none
running / docker absent), guard skipped for non-mutating phase selections.
Fixes#103567.
`hermes verify`'s compose recipe (`agent/verify/recipes.py::_detect_compose_recipe`)
unconditionally runs `docker compose build` + `docker compose up` against any
directory containing a `docker-compose.yml`, with no check for whether that
directory is already a live deployment. On an image-hash change, `docker
compose up` replaces the running containers -- destroying any state stored
on a container-local path or anonymous volume (not a bind mount).
This caused a real incident: several kanban workers each ran `hermes verify`
as routine self-verification against a workspace that was simultaneously a
live compose deployment; each invocation auto-built and auto-upped the
project; the container carrying a live state DB was replaced (state lost,
recovered from a periodic guard backup after a ~98 minute outage). Nothing
in verify's output indicated it was about to touch live containers.
Implemented the issue's "at minimum" suggested fix: before running any
phase for a `kind == "compose"` recipe, check for running containers via
`docker compose ps --status running` (read-only, never mutates anything).
If any are found, refuse outright with a clear message naming the running
containers, without ever invoking the real `build`/`start` commands as
subprocesses. The refusal surfaces through the exact same PhaseResult/
VerifyResult contract every other phase failure uses (a synthetic failed
"build" phase), so `hermes verify`'s existing JSON/text reporting requires
no special-casing.
The check is best-effort: when `docker compose ps` itself can't run (docker
not installed, or some other environment issue), the function returns None
(not an empty list) and verify proceeds normally rather than blocking on an
unrelated environment gap -- matching how every other recipe kind already
behaves when its own tooling is unavailable. The guard is scoped strictly
to `kind == "compose"`; every other recipe kind (Node, Python, Go, Rust,
Java, Makefile) is completely untouched and never even probes docker.
Verified directly: constructed the exact reported scenario (a compose
Recipe, `docker compose ps` returning running container names via a mocked
subprocess.run) and confirmed run_verify() returns a failed result with a
single synthetic phase, no build/start subprocess ever spawned. Also
verified: no running containers proceeds normally (the real build phase is
attempted); the docker-unavailable case proceeds normally rather than
blocking; and a non-compose recipe never even calls the guard function.
Added 5 regression tests to the existing
tests/verify/test_environment_and_runner.py, following its established
Recipe/run_verify/tmp_path test pattern. Verified as a genuine regression:
reverted just the guard block and confirmed the primary refusal test fails
(the guard's protective behavior is gone) -- a second test that would have
gone on to actually invoke `docker compose build` as a real subprocess
without the guard was interrupted rather than left to run against a
sandbox with no docker daemon, since the first test's failure already
conclusively demonstrates the regression.
24/24 pass in the modified test file; 21/21 pass across two additional
related verify test files (no regression).
The compression/aux fallback ladder decides "payment error" via
_is_payment_error(), which only knew the spaced "resource exhausted". NVIDIA NIM
and gRPC-style wrappers serialize the same quota signal as ResourceExhausted /
RESOURCE_EXHAUSTED / resource-exhausted, so a 403/429/status-less body carrying
it re-raised instead of walking the configured fallback_chain, and compression
fell to the lossy emergency path. Add the three separator variants to
_PAYMENT_KEYWORDS (same status gate, no broader "exhausted" matching).
evals/auxiliary_resource_exhausted.py drives the real call_llm/async_call_llm
against two local OpenAI-compatible listeners (nvidia profile primary, named
custom fallback) for the before/after check. #85649
liuhao1024