The simplify reviewers flagged that every cooldown parametrize row was
> 60s, so the emit branch was never asserted against its buffered
complement. _drive_once now records which surface the status went
through, and a 30s row asserts short cooldowns keep the buffered line.
Mutation-checked: flipping the threshold fails the four emit rows.
/simplify-code pass on the Retry-After salvage stack:
- compute_error_backoff now decides "no usable cooldown" exactly once:
a parsed 0.0 (retry-after: 0, or an HTTP-date in the past, which the
shared parser clamps to 0) is treated as absent instead of falling
through an accidental falsy check — prevents a hot-loop retry and
keeps the sentinel semantics uniform (is None / is not None at all
four sites).
- Comment corrected: the sibling unwrap lives in
extract_api_error_context, not _extract_rate_limit_context.
- Test driver deduped: one _retryable_error factory + one _drive_once
shared by the 429 and 524 tests; added the over-cap (3600 → 600) and
no-cooldown fallback rows requested in the #103722 review.
Mutation checks: nested-unwrap neutralized → nested row red; cap
removed → over-cap row red; restored → all 298 green.
Salvage follow-ups on #88236 (krunkosaurus):
- Some providers nest the cooldown as body["error"]["retry_after"] (the
same unwrap _extract_rate_limit_context already uses); only the top-level
shape was read, so those errors silently fell back to jittered backoff.
- A 5xx Retry-After can reach the 600s cap; that wait was buffered
(replayed only on terminal failure), leaving the user silent for
minutes. Long provider cooldowns now emit immediately, mirroring the
zai_coding_overload_long path. Jittered waits keep the old buffering.
Test widened with the nested-body parametrize row; proven red when the
unwrap is neutralized.
Retryable server errors can carry provider cooldowns just like 429 responses. Parse Retry-After from response headers or structured error bodies before falling back to jittered backoff, and cover HTTP 524 behavior with runtime regression tests.
Independent review showed the retry version was a scheduling lottery: under a
forced consumer-first schedule (provider generator held at the terminal chunk
until the consumer took it) both tests failed after 12 attempts with correct
production code. The reviewer also demonstrated the deterministic alternative;
this adopts it. At the chosen chunk the hook releases the finalizer and pumps
the stream's OWN event loop (stream._loop.run_until_complete over a
to_thread wait) until the finalizer has completed, then hands the chunk to the
consumer: a real happens-before on every schedule. A schedule probe's gate on
the provider generator is released first so the pump can reach EOF. The retry
helper and the race-forced plumbing are gone; each test runs once.
Adverse consumer-first schedule (reviewer's probe plugin): 3/3 pass (was 0/3).
Normal schedule: 6/6. Production fix 74de0fd4fe reverted: tool-call test still
fails, so the guard is intact.
Relay's producer is pumped by the consumer thread's own event loop (ManagedLlmStream.__next__ -> run_until_complete), so the finalizer can only START inside a consumer next(). The harness blocked the consumer in _count_chunk waiting for the finalizer to start; when the loop had not reached it yet by the final chunk, that wait could never be satisfied and expired at 5 s. Reproduced ~1 run in 6 locally with a thread dump (consumer parked in Event.wait, no other thread anywhere near Relay); it took two unrelated PRs red in CI the same day (#103476, #103486).
The hook now forces the ordering only when the finalizer has already started (that is the race under test), releases it otherwise, reports whether the race was forced, and the two tests repeat the stream until it was, asserting the invariant on every run. 15/15 green; with 74de0fd4fe's agent/ change reverted the tool-call test still fails 3/3, so it keeps guarding what it pins.
Exercise startup and polling against a local IMAP peer using real imaplib. Cover missing ID, accepted ID, and tagged ID rejection without external accounts or network dependencies. Supports NousResearch/hermes-agent#92979 and #39856.
_send_imap_id() sent RFC 2971 ID unconditionally after login (a
163/NetEase requirement). Servers without the extension can react badly:
Purelymail answers with an untagged '* BYE Unknown command.' and closes
the connection, which imaplib cannot surface at the call site — the
failure appears one command later as 'IMAP connection failed: command:
SELECT => Unknown command.' and the adapter retries forever.
Gate the ID send on the server's advertised capabilities
(imap.capabilities, populated by imaplib at connect), keeping the
existing exception handler for servers that advertise ID but reject it.
Ports PR #39861 to the current plugin layout, as requested by the
hermes-sweeper review there.
Fixes#39856
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
The helper returned container names OR a synthetic pseudo-name so the caller
would refuse; on a hung daemon that rendered as "already has running
container(s) (<docker compose ps timed out ...>)". It now returns the refusal
reason or None, and the caller prints the reason as given.
Review follow-up on the salvaged #103577 guard. Fail-open on a hung or failing
`docker compose ps` reopened the #103567 window: containers may be live and
unobservable, and `docker compose build` would fail against the same daemon
anyway, so refusing loses nothing. Only a missing docker binary falls through.
The guard now also runs only when a mutating phase (build, or start without
skip_start) is selected. Tests collapsed to three contracts: refusal spawns
nothing mutating (running / timed out / failed probe), fall-through (none
running / docker absent), guard skipped for non-mutating phase selections.
Fixes#103567.
`hermes verify`'s compose recipe (`agent/verify/recipes.py::_detect_compose_recipe`)
unconditionally runs `docker compose build` + `docker compose up` against any
directory containing a `docker-compose.yml`, with no check for whether that
directory is already a live deployment. On an image-hash change, `docker
compose up` replaces the running containers -- destroying any state stored
on a container-local path or anonymous volume (not a bind mount).
This caused a real incident: several kanban workers each ran `hermes verify`
as routine self-verification against a workspace that was simultaneously a
live compose deployment; each invocation auto-built and auto-upped the
project; the container carrying a live state DB was replaced (state lost,
recovered from a periodic guard backup after a ~98 minute outage). Nothing
in verify's output indicated it was about to touch live containers.
Implemented the issue's "at minimum" suggested fix: before running any
phase for a `kind == "compose"` recipe, check for running containers via
`docker compose ps --status running` (read-only, never mutates anything).
If any are found, refuse outright with a clear message naming the running
containers, without ever invoking the real `build`/`start` commands as
subprocesses. The refusal surfaces through the exact same PhaseResult/
VerifyResult contract every other phase failure uses (a synthetic failed
"build" phase), so `hermes verify`'s existing JSON/text reporting requires
no special-casing.
The check is best-effort: when `docker compose ps` itself can't run (docker
not installed, or some other environment issue), the function returns None
(not an empty list) and verify proceeds normally rather than blocking on an
unrelated environment gap -- matching how every other recipe kind already
behaves when its own tooling is unavailable. The guard is scoped strictly
to `kind == "compose"`; every other recipe kind (Node, Python, Go, Rust,
Java, Makefile) is completely untouched and never even probes docker.
Verified directly: constructed the exact reported scenario (a compose
Recipe, `docker compose ps` returning running container names via a mocked
subprocess.run) and confirmed run_verify() returns a failed result with a
single synthetic phase, no build/start subprocess ever spawned. Also
verified: no running containers proceeds normally (the real build phase is
attempted); the docker-unavailable case proceeds normally rather than
blocking; and a non-compose recipe never even calls the guard function.
Added 5 regression tests to the existing
tests/verify/test_environment_and_runner.py, following its established
Recipe/run_verify/tmp_path test pattern. Verified as a genuine regression:
reverted just the guard block and confirmed the primary refusal test fails
(the guard's protective behavior is gone) -- a second test that would have
gone on to actually invoke `docker compose build` as a real subprocess
without the guard was interrupted rather than left to run against a
sandbox with no docker daemon, since the first test's failure already
conclusively demonstrates the regression.
24/24 pass in the modified test file; 21/21 pass across two additional
related verify test files (no regression).
A rate-limited error classifies as network above and the retry loop only breaks
on a non-transient, non-rate-limited error, so the fallback-path guard could
never fire. Tests: the classifier delegator's vocabulary tests were a
change-detector; two rate-limit loop tests were subsumed by the exhaustion
contract (3 sends, no notice, no fallback).
Review follow-up on the salvaged #100072 flood handling. Honouring retry_after
is right, but sleeping it verbatim in _send_with_retry would pin the send
coroutine for a long penalty (a 97-minute FloodWait once froze inbound on
every platform, #91969 -- the Telegram adapter already fails closed at 5s for
the same reason and hands the wait to this loop). Past 60s the typed failure
goes back to the delivery ledger, which owns redelivery after the cooldown.
The six classifier tests collapse into one parametrized contract.
Flood-capped / rate-limited sends (Telegram FloodWait, Weixin bare RuntimeError)
were not honored as retryable when retry_after was absent, so _send_with_retry
fell through to the truncating plain-text fallback, which re-entered the server
ban and dropped the message tail — silently burning the turn with no delivery.
Three changes to _send_with_retry:
1. Route rate-limit checks through _is_rate_limited_error (single wrapper over
classify_send_error) so rate limits are treated as transient even when the
platform omits retry_after.
2. Reclassify rate-limit per retry attempt, not just the initial send, so the
in-loop break/continue reflects the CURRENT attempt (covers transient->flood
and rate-limited->formatting transitions).
3. Never take the truncating plain-text fallback for a rate-limited send — return
the typed failure so the delivery ledger owns redelivery after the cooldown.
And when retries exhaust on a rate-limited / retry_after-carrying failure,
return BEFORE sending the delivery-failure notice: the notice send would land
inside the same flood penalty and re-enter the ban ([0, 189, 378, 378] -> 3
sends, no 4th). Ordinary exhausted network errors keep the existing notice.
Honors server retry_after (when present) as the backoff delay instead of the
default exponential schedule.
Rebased onto current main (was 5163 commits behind, CONFLICTING) and folded in
reviewer feedback. Regression coverage in tests/gateway/test_send_retry.py:
rate-limit classifier, rate-limited-without-retry_after retries & succeeds,
rate-limited exhaustion returns a typed failure with no fallback and no notice
inside the active flood penalty, and failure-kind transitions between attempts.
Serve/dashboard runtimes return from match_runtime_outcomes before
_gateway_named_in is reached, and the systemd flavour of this contract is
already pinned by test_serve_reconciles_against_its_own_unit_vocabulary.
The restart phase records macOS LaunchAgent labels (ai.hermes.gateway).
match_runtime_outcomes used a substring check for "hermes-gateway", so a
successful Desktop update on the default profile always tripped
"Planned runtimes the restart phase never touched" and exited 1.
Use the exact systemd/launchd/s6 matcher for both plan reconciliation
and abort-recovery so the two cannot drift.
Folds the /heartbeat driver into the existing /loop poll slot (one cadence, one
try/except table) and closes the consumed-tick gap: a dispatch that raises OR is
refused by _admit_prompt_turn (returns False) releases the session claim and rewinds
the persisted fire via HeartbeatManager.abandon_fire(), so the tick stays due for
the next poll instead of advancing fire_count with no turn. abandon_fire refuses to
overwrite a pause/resume/clear that landed between claim and dispatch (mirrors
LoopManager.abandon_tick). Tests use the real SessionDB under a temp HERMES_HOME and
drive the poller loop itself; docs note the TUI/Desktop surface.
Rollback-on-failure idea credited to jerrygooch (#104011, 51df1ed4d04 / 283fa972169);
the driver placement is Halldrix's (#102118). Reported in #102056 / #103044 (Vksh07).
/heartbeat is parsed by the slash-worker process, which starts its own
heartbeat watchdog thread there — but that process never runs turns, so
the queued prompt is armed-but-dead (countdown froze at ~0s, no turn ever
injected). Mirror the existing /loop driver: poll for due heartbeats in
the per-session notification poller and re-enter the live session through
_run_prompt_submit as a plain user turn, preserving alternation and prompt
caching.
(cherry picked from commit 23ae971075f2d8156642f4c268dd010510a7af41)
Publish-UiEvent only enters the delivery-wait loop under $script:UiServer,
and Show-ProgressWindow returns before the WinForms card is built whenever
the browser shim is up, so $script:Ui is always null there. The DoEvents
call could never run; the 50ms sleep loop is the whole wait.
A delayed browser could miss the 900ms terminal event and spin forever after the updater exited. Retain terminal delivery until the page acknowledges it, bound unavailable-client teardown and failed requests, and preserve a truthful final display.
Fixes#103747. Builds on OutThisLife and Teknium detached handoff work in #83634 and the #75895 quiet-window design. Continues Axl Ibiza Windows update investigation (#60233, #94107, #100763), including source/review contributions carried by merged #93353 and #85170. Existing #102373, #103140, #95719, #97299 and #103632 retain their separate scopes.
When a sash drag folds a tool zone to its rail, the store commit re-renders
only the wrappers whose style prop changed; the flex sibling the gesture
had pinned to `flex: 0 1 <px>` kept that inline preview and could not grow
into the freed space. Restore every captured style attribute before the
commit, on every release path.
A sash drag used to stop the moment its seam partner hit its min size, so a
row of panes behaved like unrelated boxes: growing the Browser tile could not
take space from Chat once the pane between them was at its floor. The drag
now plans the whole run from the pointerdown sizes — the partner donates
first, then its next visible sibling, and so on — and commits once on
release (fixed zones get px overrides, the flex run gets weights).
A reverse drag after a cascade stays local to the seam. Release only folds a
tool zone that THIS gesture took to its floor, so an unrelated rail already
resting there (a minimized Terminal) no longer cancels the Files/Chat commit.
Salvaged from #103166 (Jerry Gooch), trimmed to the sash-drag change; the
width/height lock feature, zone-body context menu and the restored-tool CSS
floor are held as separate decisions.
A background delegate_task call used to be ONE async unit: the runner joined on
every child and a single consolidated message re-entered the conversation when
the SLOWEST finished. Fifteen independent PR reviews therefore waited on the
fifteenth before the parent could act on the first.
Each task now carries an optional `group`. `_units_of` partitions the call's
children into units — one per distinct group, one per ungrouped task — and each
unit is dispatched to the async registry on its own, so its results re-enter the
conversation as soon as THAT unit is done. Tasks that must be compared or merged
share a group and still return together.
Capacity is unchanged: every unit of one call joins the first unit's pool slot
(`slot_key` in `async_delegation._dispatch`), so splitting never consumes more of
`delegation.max_concurrent_children` than the call did. Unit ids suffix the
call's id (`deleg_xxxx-1`, `-2`, …) so live transcripts stay under one dir; the
completion block names the group and notes that sibling units report separately;
`active_task_count` counts a unit's own tasks.
test_nous_anthropic_fallback_uses_the_messages_wire and
test_nous_child_rederives_api_mode_from_model describe the native wire, which
is now opt-in; select it in the test the same way the wire-contract suite
does, so both keep guarding the flip-back.
The compression/aux fallback ladder decides "payment error" via
_is_payment_error(), which only knew the spaced "resource exhausted". NVIDIA NIM
and gRPC-style wrappers serialize the same quota signal as ResourceExhausted /
RESOURCE_EXHAUSTED / resource-exhausted, so a 403/429/status-less body carrying
it re-raised instead of walking the configured fallback_chain, and compression
fell to the lossy emergency path. Add the three separator variants to
_PAYMENT_KEYWORDS (same status gate, no broader "exhausted" matching).
evals/auxiliary_resource_exhausted.py drives the real call_llm/async_call_llm
against two local OpenAI-compatible listeners (nvidia profile primary, named
custom fallback) for the before/after check. #85649
Portal serves anthropic/* on two routes. The native /v1/messages wire, which
Hermes has used since 02d5e23085, re-writes the previous turn's prompt cache
on 14-20% of consecutive calls in concurrent tool loops; the chat/completions
route does not. Measured 2026-09-06, 20 concurrent sessions x 6 tool calls on
Fable 5.1, same account, same hour, same first-party pin:
Nous /v1/messages 180 pairs, 25 stuck (13.9%); 3 earlier 40-session
runs 15.0 / 17.3 / 20.2%; unchanged by the pin
Nous /v1/chat/completions 320 pairs (2 runs), 0 stuck
OpenRouter direct, pinned 161 pairs, 0 stuck
"stuck" = cache_read on call k+1 equals cache_read on call k instead of
call k's prompt size: the last write was not visible, the turn (~28K) was
written twice. Every stuck pair had byte-identical system, tools and
per-message shas, so it is not client-side mutation. At $10-20/M for
cache writes that is 15-20% of a fan-out's write bill.
The cause is inside the portal's native route (every element of its
outgoing request reproduces clean from outside; NousResearch/api#227
carries the diagnostics). Until it is fixed, anthropic/* rides
chat/completions. nous.anthropic_wire: native opts back in. Cost of
chat: prior-turn thinking travels as OpenAI-style reasoning fields, and
cache_control scopes are translated by the portal's adapter.
Tests: new test_nous_anthropic_wire_default.py reads the knob through the
real config loader in a temp HERMES_HOME and through
resolve_runtime_provider (both settings); the existing native-wire
contract suite selects native via an autouse fixture so the wire keeps
working for the flip-back. Live: a real AIAgent on this branch,
provider=nous + anthropic/claude-fable-5.1, dispatches to
chat/completions with the session_id intact.
Follow-up to the salvaged contributor commit: the Gemini and Vertex guide model
tables now list both Flash generations the pickers offer, and an invariant test
ties the OpenRouter/Nous curated google/gemini-*-flash entries to (a) a Google
official-docs pricing row on the direct Gemini and Vertex routes and (b)
membership in the direct gemini/vertex picker lists. Red on main
(gemini-3.7-flash -> unknown), green with the contributor commit.
Campaign: https://github.com/NousResearch/hermes-agent/issues/104154
- Generalize Gemini 3 thinking config model prefix match to gemini-3*
- Add pricing snapshot entries for gemini-3.7-flash and gemini-3.8-flash
- Add model identifiers to Google, OpenRouter, and Vertex CLI lists
- Add test coverage for gemini-3.8-flash thinkingLevel configuration
Trim the salvaged suite to the behaviour contracts: a picker exception or Ctrl+C leaves
config.yaml's model exactly as snapshotted (parametrized), and an absent active_provider
stays absent through snapshot+restore. The mock-dispatch tests (asserting calls into our own
helpers) are dropped. evals/cli_fallback_add_picker_error.py drives the real
`hermes fallback add` under a Linux PTY into an ordinary picker OSError and checks the
persisted model: stranded on base, restored after.
Two corrections on top of the mention-preservation pick:
- Stripping our own trigger handle from every group message was replaced by never stripping
it in groups, which broke the pre-existing contract that `@bot 2` answers a pending clarify
prompt and `@bot ok` approves (the intercepts match the exact stripped text). Strip only when
no other participant is named; when other bots/users are mentioned the text is kept verbatim
so `@research_bot , @ops_bot …` no longer arrives as `, @ops_bot …`.
- The addressing block carried a per-message fact ("explicitly mentions you: yes/no") inside
`channel_prompt`, which is part of the cached-agent signature: a mention turn followed by a
reply turn rebuilt the AIAgent (prompt-cache miss) every time the shape flipped. Keep only
the username line, byte-identical for the life of the session.
Tests rewritten to pin both contracts (sole-addressee stripping; identical channel_prompt and
agent signature across a mention turn and a reply turn).
Follow-up to the truncation removal: with the full reply text in the prepared message, a
`@file:`/`@url:` reference inside the *quoted* message reached the context-reference expander
and read a local file on the replier's behalf (the old 500-char slice hid this for long
quotes; short quotes were already exposed on main). Expansion now runs on the new message
text only; the reply pointer is prepended afterwards.
The ChatGPT Codex backend answers a rejected encrypted-reasoning replay with the
same bare 400 {code: invalid_prompt, message: "Request blocked."} it uses for
genuine blocks. classify_api_error bucketed it as format_error, so the one-shot
repair that already exists for invalid_encrypted_content (disable replay, strip
cached codex_reasoning_items, retry once) never ran and the turn aborted.
Classify exactly that envelope from provider openai-codex — SDK 400 body, SSE
``error`` frame, or the ``response.failed`` text — as invalid_encrypted_content
while keeping format_error's abort-and-fallback hints, so the only behavioural
delta is turn_recovery's replay strip, which still requires cached reasoning
items. A block with nothing to strip aborts exactly as before; other providers,
other messages/codes and the #18028 safety refusals are untouched.
evals/codex_masked_replay_review.py drives a real AIAgent (codex_responses)
against a synthetic local Responses server for the before/after check.
Refs #92353, #92357, #94077.
kshitijk4poor