N concurrent real AIAgent sessions on a growing tool loop; per-call cache_read /
cache_creation / response id / upstream / prefix shas; consecutive pairs classified
ideal / stuck / collapse. argparse (--provider nous|openrouter|anthropic, --wire,
--pin, --settle, --ttl, --model); wire defaults to what Hermes would pick. Summary
JSON carries bad pairs with both response ids. Smoke-run against Nous from the repo
path (picked chat via nous_api_mode; 6/6 ideal).
Since #104299 a background delegate_task call is split into completion units
(one per `group`, one per ungrouped task). A multi-child unit still joined on
all its children before anything was written durably, so an owner crash
between the first and last child lost the finished work and replayed the whole
unit as "outcome unknown" — the restart-granularity gap that #104233 (Xipong's
#76228/#76229 direction) solved with a second row per child.
Each finished child of a detached unit is now recorded on the unit's OWN row
(`record_unit_child` → result_json {results, partial}) as its future lands;
the real result overwrites it at finalize. `recover_abandoned_delegations`
replays recorded children with their real summaries and marks only the
unfinished ones unknown, naming the count. No new rows, no new consumer shape.
`task_indexes` is persisted so recovery knows a split unit's members.
- session-control-goal.tsx: a 'send' dispatch against a busy session now
parks the kickoff on the composer queue (the backend already resumed the
goal) instead of reporting continuationFailed. Shared helper
queueKickoffIfSessionBusy() extracted from slash.ts so both paths agree.
- gateway-switch.ts: wipeSessionListsForGatewaySwitch clears
$sessionControlBySession (runtime-id keyed; new backend re-mints ids).
Dead resetSessionControlAfterGatewayRebind removed; clearSessionControl
now called from the session delete path beside clearQueuedPrompts.
- session-control.tsx: read/hydration failures use controlUnavailable copy,
not actionFailed.
- i18n: heartbeatDueWaitingForIdle added to ja/ru/zh-hant; new
continuationQueued/continuationBusy/controlUnavailable in all locales.
- /goal and /loop are command.dispatch built-ins (never reach the slash worker), so the
worker-branch publish never fired for the most common typed commands and the structured
card stayed stale until the next turn. The publish helper now lives beside _snapshot_control
in methods_session_control.py and is called from both slash paths plus the end of the turn
tail — after the goal judge / loop tick evaluation, which mutate state AFTER message.complete.
- session.control skips the duplicate emit for dispatcher-backed actions (still exactly one
update per mutation).
- Removed tui_gateway/desktop_heartbeat_driver.py, the entry/ws hooks and the hermes_cli lock
rework: #104224 drives /heartbeat from the existing per-session poller for every TUI client.
- test_session_control.py: dropped presentation-shaped cases, added the publication invariants
(both slash paths, unknown dispatch publishes nothing, real _run_prompt_submit turn publishes
post-judge). 3/3 new tests fail on the contributor backend.
Portal will serve anthropic/* from more than one upstream (OpenRouter
passthrough today; GMI/Vertex once it is back online). The native Messages
wire is the better transport but is only safe where the upstream keeps
prompt-cache routing sticky: measured false on the OpenRouter path (14-20% of
consecutive calls re-write the previous turn; #104284 moved the default to
chat), untested on GMI. Hermes cannot see the upstream in the request, only
in the response: OpenRouter stamps `provider` (chat wire) and mints
`gen-<unix>-<rand>` ids; GMI/Vertex returns Anthropic-native `msg_...` ids
and no provider.
`auto` therefore starts every session on chat (correct on both upstreams),
classifies the first response, and switches that session to native only
when the upstream is GMI AND `agent/nous_wire.py::GMI_NATIVE_WIRE_CLEARED`
is True. The switch is scheduled at response time and applied at the start
of the next iteration (turn_iteration_prep), so nothing is rebuilt while a
response is being consumed; it goes through switch_model so the client,
cache policy and _primary_runtime stay consistent. One decision per session,
call 1 only; unknown upstream never switches; a failed switch logs and stays.
GMI_NATIVE_WIRE_CLEARED is False: until the 20x6 concurrency probe
(evals/postmortem/live_ab) is clean on a GMI-served anthropic/* id on the
native wire, `auto` behaves exactly like `chat`. Flipping it is the whole
rollout once GMI is measured. Default stays `chat`.
Tests (17): classifier on real Portal response shapes from both wires and
both upstreams; chat for openrouter/unknown, GMI gated on the flag; one
decision per session, call 1 only, explicit chat/native never auto-switch,
other providers/models untouched, switch failure swallowed and final;
record_response_usage on a real AIAgent invokes the hook once.
Live (auto, real Portal, Fable 5.1): arm A, real classification
(OpenRouter today) - stays on chat through a tool loop and a second turn,
cache 97-99%. Arm B, classifier forced to gmi with the flag on - call 1 on
chat, switch applied before call 2, calls 2-3 on the native wire in the
same session, tool result and both turns correct, cache 97-99%. An earlier
shape that switched inside the response path broke call 1 (SimpleNamespace
has no .content); the scheduled apply is why.
hermes_cli.foreign_sessions._walk owns discovery (glob per source, per-entry
OSError tolerance, symlink-escape and regular-file checks, newest first); the
CLI listing and the desktop browser's candidate rows both consume it instead of
carrying two scans of the same directories.
_first_user_line indexed splitlines()[0] on a whitespace-only user message
(image-only / tool-only turn) and raised IndexError, which aborted the whole
CLI picker and the desktop session.foreign.list RPC. partition("\n") yields
"" for blank text and the loop moves on to the first real user line.
Reported first in #92290.
- Sidebar: the entry joins SIDEBAR_NAV (same chrome, active state, data-tour
handle as the other rows) instead of a one-off Button below the rail;
i18n moves to sidebar.nav['session-import'].
- Reuse common.retry / common.refresh / common.back instead of duplicating them
under sessionImport in six locales.
- hermes_cli/foreign_session_browser.py -> foreign_sessions_browser.py so it
sorts as a sibling of the foreign_sessions module it extends.
One unreadable or rotated log must not hide the rest of the list; discovery
tolerates OSError from resolve()/stat() per entry and reuses the stat result.
Browse the backend host's foreign CLI session logs, preview a bounded read-only
transcript, and continue a copy in Hermes under the selected profile. Reuses the
hermes_cli.foreign_sessions parsers and the portability validator/writer;
imports are transactional and deduplicated on the recorded origin.
Fake local Responses SSE server plays the Codex backend; scenarios cover live
capture (CLI same-object history and gateway reloaded history), SessionDB
close/reopen restore into a fresh agent, and the over-threshold negative where
local compression must still fire after real usage arrives.
A native Responses compaction checkpoint is opaque ciphertext; the rough
preflight estimator counts it as text (5.17M chars -> ~1.29M tokens against
a 204K trigger) and fires local compression on a request whose real prompt is
~116K. Arm the existing one-response real-usage latch when a replayable
checkpoint is captured (build_assistant_message) or restored into a fresh
agent (_hydrate_from_history), honor it in the post-tool gate and idle
compaction, and require non-empty encrypted_content for a checkpoint.
Squash of the author's source commits from #100642 (0e3c234ea0, 771e1b3365,
bb1505a119) plus the fdf140c81d test refresh, re-based onto current main by
patch application. Source delta is byte-identical to the PR head d6ce3e236d.
Fixes#100611
Two shape fixes from the review of #92979, no behavior change:
- Read imap.capabilities directly instead of the getattr(...) or ()
fallback — a real imaplib.IMAP4 always sets the attribute in
_connect() (raising if the server sends no CAPABILITY), so the
default branch only existed for a MagicMock(spec=["xatom"]) that no
production path produces. Drop the test that pinned it.
- Trim tests to the invariant bar: the four mock tests duplicated what
test_email_imap_id_protocol.py proves on the wire (real imaplib, real
socket, command order), and the phase parametrize (startup vs poll)
exercised the same single _send_imap_id call site twice. Keep the
three id_mode cases through the poll path.
42 tests pass; mutation check: guard removed -> the absent-ID case
fails with the #39856 SELECT error, restored -> green.
The simplify reviewers flagged that every cooldown parametrize row was
> 60s, so the emit branch was never asserted against its buffered
complement. _drive_once now records which surface the status went
through, and a 30s row asserts short cooldowns keep the buffered line.
Mutation-checked: flipping the threshold fails the four emit rows.
/simplify-code pass on the Retry-After salvage stack:
- compute_error_backoff now decides "no usable cooldown" exactly once:
a parsed 0.0 (retry-after: 0, or an HTTP-date in the past, which the
shared parser clamps to 0) is treated as absent instead of falling
through an accidental falsy check — prevents a hot-loop retry and
keeps the sentinel semantics uniform (is None / is not None at all
four sites).
- Comment corrected: the sibling unwrap lives in
extract_api_error_context, not _extract_rate_limit_context.
- Test driver deduped: one _retryable_error factory + one _drive_once
shared by the 429 and 524 tests; added the over-cap (3600 → 600) and
no-cooldown fallback rows requested in the #103722 review.
Mutation checks: nested-unwrap neutralized → nested row red; cap
removed → over-cap row red; restored → all 298 green.
Salvage follow-ups on #88236 (krunkosaurus):
- Some providers nest the cooldown as body["error"]["retry_after"] (the
same unwrap _extract_rate_limit_context already uses); only the top-level
shape was read, so those errors silently fell back to jittered backoff.
- A 5xx Retry-After can reach the 600s cap; that wait was buffered
(replayed only on terminal failure), leaving the user silent for
minutes. Long provider cooldowns now emit immediately, mirroring the
zai_coding_overload_long path. Jittered waits keep the old buffering.
Test widened with the nested-body parametrize row; proven red when the
unwrap is neutralized.
Retryable server errors can carry provider cooldowns just like 429 responses. Parse Retry-After from response headers or structured error bodies before falling back to jittered backoff, and cover HTTP 524 behavior with runtime regression tests.
Independent review showed the retry version was a scheduling lottery: under a
forced consumer-first schedule (provider generator held at the terminal chunk
until the consumer took it) both tests failed after 12 attempts with correct
production code. The reviewer also demonstrated the deterministic alternative;
this adopts it. At the chosen chunk the hook releases the finalizer and pumps
the stream's OWN event loop (stream._loop.run_until_complete over a
to_thread wait) until the finalizer has completed, then hands the chunk to the
consumer: a real happens-before on every schedule. A schedule probe's gate on
the provider generator is released first so the pump can reach EOF. The retry
helper and the race-forced plumbing are gone; each test runs once.
Adverse consumer-first schedule (reviewer's probe plugin): 3/3 pass (was 0/3).
Normal schedule: 6/6. Production fix 74de0fd4fe reverted: tool-call test still
fails, so the guard is intact.
Relay's producer is pumped by the consumer thread's own event loop (ManagedLlmStream.__next__ -> run_until_complete), so the finalizer can only START inside a consumer next(). The harness blocked the consumer in _count_chunk waiting for the finalizer to start; when the loop had not reached it yet by the final chunk, that wait could never be satisfied and expired at 5 s. Reproduced ~1 run in 6 locally with a thread dump (consumer parked in Event.wait, no other thread anywhere near Relay); it took two unrelated PRs red in CI the same day (#103476, #103486).
The hook now forces the ordering only when the finalizer has already started (that is the race under test), releases it otherwise, reports whether the race was forced, and the two tests repeat the stream until it was, asserting the invariant on every run. 15/15 green; with 74de0fd4fe's agent/ change reverted the tool-call test still fails 3/3, so it keeps guarding what it pins.
Exercise startup and polling against a local IMAP peer using real imaplib. Cover missing ID, accepted ID, and tagged ID rejection without external accounts or network dependencies. Supports NousResearch/hermes-agent#92979 and #39856.
_send_imap_id() sent RFC 2971 ID unconditionally after login (a
163/NetEase requirement). Servers without the extension can react badly:
Purelymail answers with an untagged '* BYE Unknown command.' and closes
the connection, which imaplib cannot surface at the call site — the
failure appears one command later as 'IMAP connection failed: command:
SELECT => Unknown command.' and the adapter retries forever.
Gate the ID send on the server's advertised capabilities
(imap.capabilities, populated by imaplib at connect), keeping the
existing exception handler for servers that advertise ID but reject it.
Ports PR #39861 to the current plugin layout, as requested by the
hermes-sweeper review there.
Fixes#39856
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
The helper returned container names OR a synthetic pseudo-name so the caller
would refuse; on a hung daemon that rendered as "already has running
container(s) (<docker compose ps timed out ...>)". It now returns the refusal
reason or None, and the caller prints the reason as given.
Review follow-up on the salvaged #103577 guard. Fail-open on a hung or failing
`docker compose ps` reopened the #103567 window: containers may be live and
unobservable, and `docker compose build` would fail against the same daemon
anyway, so refusing loses nothing. Only a missing docker binary falls through.
The guard now also runs only when a mutating phase (build, or start without
skip_start) is selected. Tests collapsed to three contracts: refusal spawns
nothing mutating (running / timed out / failed probe), fall-through (none
running / docker absent), guard skipped for non-mutating phase selections.
Fixes#103567.
`hermes verify`'s compose recipe (`agent/verify/recipes.py::_detect_compose_recipe`)
unconditionally runs `docker compose build` + `docker compose up` against any
directory containing a `docker-compose.yml`, with no check for whether that
directory is already a live deployment. On an image-hash change, `docker
compose up` replaces the running containers -- destroying any state stored
on a container-local path or anonymous volume (not a bind mount).
This caused a real incident: several kanban workers each ran `hermes verify`
as routine self-verification against a workspace that was simultaneously a
live compose deployment; each invocation auto-built and auto-upped the
project; the container carrying a live state DB was replaced (state lost,
recovered from a periodic guard backup after a ~98 minute outage). Nothing
in verify's output indicated it was about to touch live containers.
Implemented the issue's "at minimum" suggested fix: before running any
phase for a `kind == "compose"` recipe, check for running containers via
`docker compose ps --status running` (read-only, never mutates anything).
If any are found, refuse outright with a clear message naming the running
containers, without ever invoking the real `build`/`start` commands as
subprocesses. The refusal surfaces through the exact same PhaseResult/
VerifyResult contract every other phase failure uses (a synthetic failed
"build" phase), so `hermes verify`'s existing JSON/text reporting requires
no special-casing.
The check is best-effort: when `docker compose ps` itself can't run (docker
not installed, or some other environment issue), the function returns None
(not an empty list) and verify proceeds normally rather than blocking on an
unrelated environment gap -- matching how every other recipe kind already
behaves when its own tooling is unavailable. The guard is scoped strictly
to `kind == "compose"`; every other recipe kind (Node, Python, Go, Rust,
Java, Makefile) is completely untouched and never even probes docker.
Verified directly: constructed the exact reported scenario (a compose
Recipe, `docker compose ps` returning running container names via a mocked
subprocess.run) and confirmed run_verify() returns a failed result with a
single synthetic phase, no build/start subprocess ever spawned. Also
verified: no running containers proceeds normally (the real build phase is
attempted); the docker-unavailable case proceeds normally rather than
blocking; and a non-compose recipe never even calls the guard function.
Added 5 regression tests to the existing
tests/verify/test_environment_and_runner.py, following its established
Recipe/run_verify/tmp_path test pattern. Verified as a genuine regression:
reverted just the guard block and confirmed the primary refusal test fails
(the guard's protective behavior is gone) -- a second test that would have
gone on to actually invoke `docker compose build` as a real subprocess
without the guard was interrupted rather than left to run against a
sandbox with no docker daemon, since the first test's failure already
conclusively demonstrates the regression.
24/24 pass in the modified test file; 21/21 pass across two additional
related verify test files (no regression).
A rate-limited error classifies as network above and the retry loop only breaks
on a non-transient, non-rate-limited error, so the fallback-path guard could
never fire. Tests: the classifier delegator's vocabulary tests were a
change-detector; two rate-limit loop tests were subsumed by the exhaustion
contract (3 sends, no notice, no fallback).
Review follow-up on the salvaged #100072 flood handling. Honouring retry_after
is right, but sleeping it verbatim in _send_with_retry would pin the send
coroutine for a long penalty (a 97-minute FloodWait once froze inbound on
every platform, #91969 -- the Telegram adapter already fails closed at 5s for
the same reason and hands the wait to this loop). Past 60s the typed failure
goes back to the delivery ledger, which owns redelivery after the cooldown.
The six classifier tests collapse into one parametrized contract.
Flood-capped / rate-limited sends (Telegram FloodWait, Weixin bare RuntimeError)
were not honored as retryable when retry_after was absent, so _send_with_retry
fell through to the truncating plain-text fallback, which re-entered the server
ban and dropped the message tail — silently burning the turn with no delivery.
Three changes to _send_with_retry:
1. Route rate-limit checks through _is_rate_limited_error (single wrapper over
classify_send_error) so rate limits are treated as transient even when the
platform omits retry_after.
2. Reclassify rate-limit per retry attempt, not just the initial send, so the
in-loop break/continue reflects the CURRENT attempt (covers transient->flood
and rate-limited->formatting transitions).
3. Never take the truncating plain-text fallback for a rate-limited send — return
the typed failure so the delivery ledger owns redelivery after the cooldown.
And when retries exhaust on a rate-limited / retry_after-carrying failure,
return BEFORE sending the delivery-failure notice: the notice send would land
inside the same flood penalty and re-enter the ban ([0, 189, 378, 378] -> 3
sends, no 4th). Ordinary exhausted network errors keep the existing notice.
Honors server retry_after (when present) as the backoff delay instead of the
default exponential schedule.
Rebased onto current main (was 5163 commits behind, CONFLICTING) and folded in
reviewer feedback. Regression coverage in tests/gateway/test_send_retry.py:
rate-limit classifier, rate-limited-without-retry_after retries & succeeds,
rate-limited exhaustion returns a typed failure with no fallback and no notice
inside the active flood penalty, and failure-kind transitions between attempts.
Serve/dashboard runtimes return from match_runtime_outcomes before
_gateway_named_in is reached, and the systemd flavour of this contract is
already pinned by test_serve_reconciles_against_its_own_unit_vocabulary.
The restart phase records macOS LaunchAgent labels (ai.hermes.gateway).
match_runtime_outcomes used a substring check for "hermes-gateway", so a
successful Desktop update on the default profile always tripped
"Planned runtimes the restart phase never touched" and exited 1.
Use the exact systemd/launchd/s6 matcher for both plan reconciliation
and abort-recovery so the two cannot drift.
Folds the /heartbeat driver into the existing /loop poll slot (one cadence, one
try/except table) and closes the consumed-tick gap: a dispatch that raises OR is
refused by _admit_prompt_turn (returns False) releases the session claim and rewinds
the persisted fire via HeartbeatManager.abandon_fire(), so the tick stays due for
the next poll instead of advancing fire_count with no turn. abandon_fire refuses to
overwrite a pause/resume/clear that landed between claim and dispatch (mirrors
LoopManager.abandon_tick). Tests use the real SessionDB under a temp HERMES_HOME and
drive the poller loop itself; docs note the TUI/Desktop surface.
Rollback-on-failure idea credited to jerrygooch (#104011, 51df1ed4d04 / 283fa972169);
the driver placement is Halldrix's (#102118). Reported in #102056 / #103044 (Vksh07).