/heartbeat is parsed by the slash-worker process, which starts its own
heartbeat watchdog thread there — but that process never runs turns, so
the queued prompt is armed-but-dead (countdown froze at ~0s, no turn ever
injected). Mirror the existing /loop driver: poll for due heartbeats in
the per-session notification poller and re-enter the live session through
_run_prompt_submit as a plain user turn, preserving alternation and prompt
caching.
(cherry picked from commit 23ae971075f2d8156642f4c268dd010510a7af41)
A delayed browser could miss the 900ms terminal event and spin forever after the updater exited. Retain terminal delivery until the page acknowledges it, bound unavailable-client teardown and failed requests, and preserve a truthful final display.
Fixes#103747. Builds on OutThisLife and Teknium detached handoff work in #83634 and the #75895 quiet-window design. Continues Axl Ibiza Windows update investigation (#60233, #94107, #100763), including source/review contributions carried by merged #93353 and #85170. Existing #102373, #103140, #95719, #97299 and #103632 retain their separate scopes.
A background delegate_task call used to be ONE async unit: the runner joined on
every child and a single consolidated message re-entered the conversation when
the SLOWEST finished. Fifteen independent PR reviews therefore waited on the
fifteenth before the parent could act on the first.
Each task now carries an optional `group`. `_units_of` partitions the call's
children into units — one per distinct group, one per ungrouped task — and each
unit is dispatched to the async registry on its own, so its results re-enter the
conversation as soon as THAT unit is done. Tasks that must be compared or merged
share a group and still return together.
Capacity is unchanged: every unit of one call joins the first unit's pool slot
(`slot_key` in `async_delegation._dispatch`), so splitting never consumes more of
`delegation.max_concurrent_children` than the call did. Unit ids suffix the
call's id (`deleg_xxxx-1`, `-2`, …) so live transcripts stay under one dir; the
completion block names the group and notes that sibling units report separately;
`active_task_count` counts a unit's own tasks.
test_nous_anthropic_fallback_uses_the_messages_wire and
test_nous_child_rederives_api_mode_from_model describe the native wire, which
is now opt-in; select it in the test the same way the wire-contract suite
does, so both keep guarding the flip-back.
The compression/aux fallback ladder decides "payment error" via
_is_payment_error(), which only knew the spaced "resource exhausted". NVIDIA NIM
and gRPC-style wrappers serialize the same quota signal as ResourceExhausted /
RESOURCE_EXHAUSTED / resource-exhausted, so a 403/429/status-less body carrying
it re-raised instead of walking the configured fallback_chain, and compression
fell to the lossy emergency path. Add the three separator variants to
_PAYMENT_KEYWORDS (same status gate, no broader "exhausted" matching).
evals/auxiliary_resource_exhausted.py drives the real call_llm/async_call_llm
against two local OpenAI-compatible listeners (nvidia profile primary, named
custom fallback) for the before/after check. #85649
Portal serves anthropic/* on two routes. The native /v1/messages wire, which
Hermes has used since 02d5e23085, re-writes the previous turn's prompt cache
on 14-20% of consecutive calls in concurrent tool loops; the chat/completions
route does not. Measured 2026-09-06, 20 concurrent sessions x 6 tool calls on
Fable 5.1, same account, same hour, same first-party pin:
Nous /v1/messages 180 pairs, 25 stuck (13.9%); 3 earlier 40-session
runs 15.0 / 17.3 / 20.2%; unchanged by the pin
Nous /v1/chat/completions 320 pairs (2 runs), 0 stuck
OpenRouter direct, pinned 161 pairs, 0 stuck
"stuck" = cache_read on call k+1 equals cache_read on call k instead of
call k's prompt size: the last write was not visible, the turn (~28K) was
written twice. Every stuck pair had byte-identical system, tools and
per-message shas, so it is not client-side mutation. At $10-20/M for
cache writes that is 15-20% of a fan-out's write bill.
The cause is inside the portal's native route (every element of its
outgoing request reproduces clean from outside; NousResearch/api#227
carries the diagnostics). Until it is fixed, anthropic/* rides
chat/completions. nous.anthropic_wire: native opts back in. Cost of
chat: prior-turn thinking travels as OpenAI-style reasoning fields, and
cache_control scopes are translated by the portal's adapter.
Tests: new test_nous_anthropic_wire_default.py reads the knob through the
real config loader in a temp HERMES_HOME and through
resolve_runtime_provider (both settings); the existing native-wire
contract suite selects native via an autouse fixture so the wire keeps
working for the flip-back. Live: a real AIAgent on this branch,
provider=nous + anthropic/claude-fable-5.1, dispatches to
chat/completions with the session_id intact.
Follow-up to the salvaged contributor commit: the Gemini and Vertex guide model
tables now list both Flash generations the pickers offer, and an invariant test
ties the OpenRouter/Nous curated google/gemini-*-flash entries to (a) a Google
official-docs pricing row on the direct Gemini and Vertex routes and (b)
membership in the direct gemini/vertex picker lists. Red on main
(gemini-3.7-flash -> unknown), green with the contributor commit.
Campaign: https://github.com/NousResearch/hermes-agent/issues/104154
- Generalize Gemini 3 thinking config model prefix match to gemini-3*
- Add pricing snapshot entries for gemini-3.7-flash and gemini-3.8-flash
- Add model identifiers to Google, OpenRouter, and Vertex CLI lists
- Add test coverage for gemini-3.8-flash thinkingLevel configuration
Trim the salvaged suite to the behaviour contracts: a picker exception or Ctrl+C leaves
config.yaml's model exactly as snapshotted (parametrized), and an absent active_provider
stays absent through snapshot+restore. The mock-dispatch tests (asserting calls into our own
helpers) are dropped. evals/cli_fallback_add_picker_error.py drives the real
`hermes fallback add` under a Linux PTY into an ordinary picker OSError and checks the
persisted model: stranded on base, restored after.
Two corrections on top of the mention-preservation pick:
- Stripping our own trigger handle from every group message was replaced by never stripping
it in groups, which broke the pre-existing contract that `@bot 2` answers a pending clarify
prompt and `@bot ok` approves (the intercepts match the exact stripped text). Strip only when
no other participant is named; when other bots/users are mentioned the text is kept verbatim
so `@research_bot , @ops_bot …` no longer arrives as `, @ops_bot …`.
- The addressing block carried a per-message fact ("explicitly mentions you: yes/no") inside
`channel_prompt`, which is part of the cached-agent signature: a mention turn followed by a
reply turn rebuilt the AIAgent (prompt-cache miss) every time the shape flipped. Keep only
the username line, byte-identical for the life of the session.
Tests rewritten to pin both contracts (sole-addressee stripping; identical channel_prompt and
agent signature across a mention turn and a reply turn).
Follow-up to the truncation removal: with the full reply text in the prepared message, a
`@file:`/`@url:` reference inside the *quoted* message reached the context-reference expander
and read a local file on the replier's behalf (the old 500-char slice hid this for long
quotes; short quotes were already exposed on main). Expansion now runs on the new message
text only; the reply pointer is prepended afterwards.
The ChatGPT Codex backend answers a rejected encrypted-reasoning replay with the
same bare 400 {code: invalid_prompt, message: "Request blocked."} it uses for
genuine blocks. classify_api_error bucketed it as format_error, so the one-shot
repair that already exists for invalid_encrypted_content (disable replay, strip
cached codex_reasoning_items, retry once) never ran and the turn aborted.
Classify exactly that envelope from provider openai-codex — SDK 400 body, SSE
``error`` frame, or the ``response.failed`` text — as invalid_encrypted_content
while keeping format_error's abort-and-fallback hints, so the only behavioural
delta is turn_recovery's replay strip, which still requires cached reasoning
items. A block with nothing to strip aborts exactly as before; other providers,
other messages/codes and the #18028 safety refusals are untouched.
evals/codex_masked_replay_review.py drives a real AIAgent (codex_responses)
against a synthetic local Responses server for the before/after check.
Refs #92353, #92357, #94077.
Fold the adapter-level unsigned-preservation case and the name-detector matrices
into one parametrized invariant (thinking family keeps unsigned only; non-thinking
DeepSeek and unrelated models still take the generic third-party strip) plus the
native-endpoint immutability guard. Add evals/anthropic_proxy_thinking_replay.py,
a real-AIAgent probe against a synthetic local Anthropic-compatible relay used for
the before/after wire check.
Third-party Anthropic-compatible relays serving a DeepSeek thinking model
(deepseek-r/v4/pro/flash) follow the same replay contract as DeepSeek's
native /anthropic endpoint: strip signed thinking blocks, preserve
unsigned ones. Previously these sessions hit the generic third-party path
and lost ALL thinking blocks, breaking cross-turn reasoning coherence.
Detection is model-name based (vendor prefixes stripped), gated on the
existing _is_third_party check so direct Anthropic traffic is untouched.
Follow-up on the cherry-picked fix: _defer_update_notice no longer takes the Rich console it
can never safely use (the notice always lands after patch_stdout owns stdout), the unit test
asserts the contract at prompt_toolkit's boundary (an ANSI fragment reaches
print_formatted_text, with no raw ESC/markup in the visible text) instead of patching our own
cprint, and evals/cli_deferred_notice.py drives the real CLI under a Linux PTY with a FIFO-gated
update cache to A/B the late-notice cases (garbled on base, clean after).
The deferred update notice ("N commits behind") was rendered via
console.print which writes ANSI escapes directly to stdout. Under
patch_stdout (active in the interactive CLI), the StdoutProxy strips
ESC bytes, leaving visible [1;33m...[0m artifacts.
Fix: render Rich markup to an ANSI string via a captured Console, then
route through cprint (which uses prompt_toolkit ANSI parser) to
bypass the StdoutProxy mangling.
This is the same class of bug as #2262 (fixed by #2448 for agent._print_fn
and display.py), but the _defer_update_notice callsite in banner.py was
not covered by that fix.
Fixes#83969
GitLab (lib/gitlab/web_hooks.rb, app/services/web_hook_service.rb) sends
webhook-id and webhook-timestamp on EVERY delivery and adds webhook-signature
only when a signing token is configured. Selecting the HMAC path on any
webhook-* header would 401 every legacy X-Gitlab-Token install the moment it
upgrades to GitLab 19, so only the signature header selects the path; id and
timestamp then travel with it and the validator still fails closed when either
is missing. svix-* keeps its existing any-header selection.
Tests trimmed to the invariant bar: one parameterized contract (whsec_ and raw
secrets accept; wrong secret, tampered body and stale timestamp reject) plus the
GitLab legacy-token coexistence contract. evals/webhook_auth/standard_webhooks_ab.py
drives a real aiohttp WebhookAdapter on a dedicated loopback port with real
signed requests for before/after evidence.
Related: #47849 (HwangJohn, cherry-picked here), #92024 (earlier salvage of
#47849), #102080 and #103167 (same alias fix, same target).
The salvaged helper was appended to the gateway/run.py facade; new behaviour
belongs in a topical sibling. agent/session_activity.py already owns the
activity-snapshot contract the three render sites read from, so the formatter
moves there as format_iteration_progress and the three call sites import it at
module level instead of late-importing the facade.
Tests trimmed to the salvage bar (<= 2 invariant tests per fix): one
parameterized contract on the helper (unbounded/None hide the ceiling, a real
budget keeps N/M) plus the contributor's end-to-end busy-ack test through the
real render path. The per-site heartbeat/timeout tests exercised the same
helper through mocks and are dropped; evals/gateway_status_render/
iteration_ceiling_ab.py drives all three real render sites with a real AIAgent
for before/after evidence (3 sentinel leaks on main -> 0).
Related: #103109 (same fix, same target module; credited in the PR),
#102845 (third render site, closed by its author in favour of #102817).
Rebase of the existing fix onto current main: the god-file split that
was in flight when this PR opened has landed, moving both original
call sites out of gateway/run.py into gateway/run_busy.py and
gateway/run_turn.py, which is why this PR showed a merge conflict.
Also folds in a third call site (see below) that a competing PR
(#102845, closed by its author in favor of this one) identified after
this PR first opened.
Three user-facing gateway status lines render "iteration N/M" from
AIAgent.get_activity_summary()'s api_call_count / max_iterations pair:
the long-running heartbeat (run_turn.GatewayTurnMixin.
_run_agent_notify_long_running), the busy-session acknowledgment
(run_busy.GatewayBusySessionMixin._compose_busy_ack_message), and the
gateway-timeout diagnostic message shown to the user when a run is
force-timed-out for inactivity (run_turn.GatewayTurnMixin.
_run_agent_timeout_result). AIAgent.max_iterations defaults to
sys.maxsize (unlimited tool-calling iterations for a top-level session
-- see run_agent.py), so all three printed the literal
9223372036854775807 as the denominator, e.g.:
⏳ Working — 3 min — iteration 2/9223372036854775807, receiving stream response
That reads as a bug rather than "unbounded" and is meaningless to a
user.
Add _format_iteration_progress() to gateway/run.py, a small shared
formatting helper: once the configured max_iterations is at or above
sys.maxsize, it renders "iteration N" alone and omits the denominator;
a genuinely finite budget (e.g. a subagent's delegation.
max_iterations: 250) still renders "iteration N/M" as before. All
three call sites now go through it. The gateway-timeout diagnostic
message's own operator-facing logger.error() call keeps the raw
resolved value for diagnostics -- only the two user-facing diag_lines
built from it are reformatted.
Added tests/gateway/test_format_iteration_progress.py (the helper's
own unit tests: unbounded default, above-sentinel values, finite
budgets, and graceful handling of a missing/malformed max_iterations),
tests/gateway/test_gateway_timeout_iteration_progress.py (both
diagnostic-message branches, unbounded and finite, plus the heartbeat
call site exercised end to end through its async polling loop), and a
new regression test in tests/gateway/test_busy_session_ack.py::
TestBusySessionAck::test_status_detail_omits_denominator_for_unbounded_max_iterations
that exercises the busy-ack call site end to end with the real
sys.maxsize default and asserts the sentinel never reaches the
rendered text.
Defect 1 in the original report (a direct_result tool's raw output
occasionally becoming the user-visible reply) is left for a separate
change -- the reporter frames it as an open core-level design question
("a per-turn suppression option... would fix the class for all
plugins"), not a drop-in fix, and it touches reply composition rather
than status-line formatting.
Part of #102806 (defect 2: the iteration-ceiling display; defect 1 is a separate change).
chat.appendStream rejects a request carrying both markdown_text and
chunks with `cannot_provide_both_markdown_text_and_chunks`, so every
native task-card update failed and each turn silently downgraded to the
plain-text fallback message (#87743). The chunks are the native
rendering; the gateway caller keeps its own editable-text fallback rail
for when the native call itself fails, so the markdown_text field was
both rejected and redundant.
Fixes#87743
Parametrized refusal test (bare/fenced/uppercase-fenced truncated objects) asserts
profile.yaml is byte-identical afterwards; one prose test keeps the lenient
fallback contract. Replaces five near-duplicate tests from #104075.
Campaign tracker: https://github.com/NousResearch/hermes-agent/issues/104154
hermes_cli/profile_describer.py's aux-LLM response parser is documented as
"lenient, never raises": when the reply doesn't parse as JSON via
_extract_json_blob, it falls back to treating the WHOLE raw reply as plain
prose and persists it (truncated to 280 chars) as profile.yaml's
description.
That fallback exists for models that ignore the JSON-output instruction and
just answer in prose -- reasonable. But it doesn't distinguish that case
from a reply that DID start out as the requested JSON object and was cut
off mid-object by the aux model or transport. _extract_json_blob requires a
matching closing brace, so a truncated object (no `}` at all, or one cut
off mid-string) returns None just like plain prose does, and the fallback
then persists the raw JSON fragment verbatim: literal leading `{`, `\n`
escapes, a sentence chopped mid-word (#104067).
Fix: when parsing fails, check whether the reply (after the existing
code-fence strip) still looks JSON-shaped (starts with `{`). If so, it's a
malformed/truncated structured reply, not prose -- refuse it and return
ok=False instead of persisting the fragment. Only fall through to the
raw-text-as-prose fallback when the reply never looked like JSON to begin
with, so genuinely prose-only aux replies keep working exactly as before.
Also logs one INFO line on the refusal path, per the issue's note that the
silent fallback made this undiagnosable.
Update (review feedback from @jerrygooch): profile_describer._FENCE_RE was
case-sensitive, so an uppercase ```JSON fence wasn't stripped -- stripping
left "JSON\n{..." which doesn't start with "{", so a truncated response
under that fence variant still fell through to the prose fallback and got
persisted, defeating the fix for that case. Added re.IGNORECASE to
_FENCE_RE, aligning it with kanban_specify._FENCE_RE (which already used
re.IGNORECASE), and added a regression test for the uppercase-fence case.
Added tests/hermes_cli/test_profile_describer.py coverage: a truncated
object cut off mid-sentence (the real-world shape from the issue), one cut
off right after the opening brace, the same truncation under a ```json
fence, the same truncation under an uppercase ```JSON fence, and a
regression guard that a genuine plain-prose reply still hits the existing
lenient fallback unchanged.
Fixes#104067
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LUtGTop5iGKi5vfWnKnwET
The plugin loader execs sibling modules before the package __init__, so a
module-level `from . import _read_mem0_json` in _setup.py failed against the
empty parent shell and the whole module silently dropped out — every later
`hermes memory setup mem0` died with "cannot import name 'post_setup'".
The test loads mem0 through the real discovery path from a cold sys.modules
and asserts the cached _setup module exposes post_setup (red on base).
Also maps the contributor email for #103078 credit.
Campaign tracker: https://github.com/NousResearch/hermes-agent/issues/104154
#76996 raised DEFAULT_READ_LIMIT, the read_file_tool signature and the schema
default from 500 to 2000, but the registry dispatch handler `_handle_read_file`
kept its own literal `args.get("limit", 500)`. Since models omit `limit` on most
reads, every dispatched read still stopped at 500 lines with `truncated: true`,
so the default flip never reached production traffic.
All three sites now read the single DEFAULT_READ_LIMIT constant, so the schema,
the Python default and the dispatch fallback cannot drift again.
Live probe (registry.dispatch on a 1500-line file, no limit): before 501 lines /
truncated=true; after 1501 lines / truncated=false.
`format_batch_tag` handed out `set N` ordinals from one process-wide table,
so every conversation on a shared backend and every child's nested fan-out
advanced the same counter. A user's second wave of 15 lanes rendered as
`[set 20 · 13/15]`, which reads like 20 batches were spawned.
Scope the ordinal table by the parent conversation (`parent_agent.session_id`)
and thread the parent through the three render sites (batch header,
completion lines, child tree-line prefix via the shared session_ref). The
first fan-out in a conversation is `set 1`, the next `set 2`; sibling
conversations and nested child batches no longer inflate it.
Invariant test proven red on origin/main, green with the fix.
The spawn_local systemd test asserted the property was present; it now
asserts the invariant the fix establishes (no OOMPolicy= on a transient
scope), so a reintroduction fails here instead of on older systemd hosts.
The tail clause was correct only by ordering (exit_code != 0 there meant
exit_code is None). Name what it encodes: a stop_reason counts only when
nothing vouched for success. Same truth table. The three literal-dict tests on
the predicate collapse into one parametrized contract; the handoff-exit test
binds to COMMAND_BOUNDARY_STOP_REASON instead of re-spelling it.
update_contract writes {"outcome": "refused", "stop_reason": <code>} with no
exit_code; that is the one production receipt where the stop_reason clause in
_receipt_looks_unfinished is load-bearing. The previous negative control used
exit_code=1, which the exit_code branch already catches. Docstring reworded:
a KeyboardInterrupt never lands on a success receipt (the boundary finalize is
a no-op once the inner path finalized).
A delegated child copies the parent's prompt_caching.cache_ttl. The 1h tier
is priced for a person who steps away between turns (2x write vs 1.25x for
5m); a subagent calls every few seconds for minutes and is gone, so it paid
2x on every tool result and never collected the retention. Live measurement
(Sep 5, 40 concurrent Fable 5.1 children via OpenRouter, 1h markers on the
wire): cache writes billed at $20/M against $12.51/M for the same run at
5m, ~60% of a write-dominated bill.
_apply_child_cache_ttl runs right after child construction: 1h -> 5m,
5m stays, disabled stays disabled, parent untouched. Tests: unit (markers
on the wire drop ttl; parent's 1h layout still differs) and the real spawn
path with cache_ttl: 1h in a temp HERMES_HOME.
`provider_routing.models.<model-id>` now takes the same only/ignore/order/sort/
require_parameters/data_collection keys and overlays the flat provider_routing
values whenever the agent is on that model. Resolution lives in the one
chokepoint every request path already uses (_provider_preferences_for_agent),
so CLI, gateway, TUI/Desktop, cron, /model switches, fallback activation and
delegated children on another model all honour it with no per-surface plumbing.
Matching is spelling-tolerant, sharing _canonical_model_variants with
agent.reasoning_overrides.
The OpenRouter profile's speed-tier pin no longer overwrites an explicit user
`only` on the BASE gpt-6-astra slug: the pin exists to keep default routing off
flex/fast, and a user pin is the stronger intent (only: [openai] stays [openai]
instead of becoming [openai, azure, azure/us]). Tier slugs (-fast/-flex) keep
owning `only`.
Live A/B (config only: {gpt-6-astra: [openai], claude-fable-5.1: [anthropic]}):
main sent {"sort":"price"} for fable and OpenRouter served it from Azure; with
this change it sends {"only":["anthropic"],"sort":"price"} and Anthropic serves it.
Schema proposed in #24495 (samplesabotage) and #100711 (Artemonim); this is a
slim chokepoint implementation of that design.
Co-authored-by: samplesabotage <samplesabotage@users.noreply.github.com>
_dedupe_display_generations chose the right representative row per logical
message but sorted the survivors by that representative's id. A protected-tail
copy written into a newer compaction generation has a higher id than messages
emitted after the original, so include_compacted reads came back as C, A, B.
Anchor the sort on the logical message's first-ever row id instead.
Salvaged from #93869 (the tui_gateway half of that PR is superseded by #100504
and #104137); the code moved from hermes_state.py to hermes_state_messages.py
since, so the change is re-applied to its new home with the PR's regression
test verbatim.
Re-scope #98106 onto the activity-based orphan policy from #100504. Keep timer ownership across callbacks and continuations, serialize reconnect paths against interrupt claims, and avoid recursive eager-resume locking. Leave cleanup polling armed when concurrent cold reuse is rejected.
- probe_only is consumed inside __init__ only; no instance attribute.
- _sync_manager is assigned only on the probe branch, after the connection is
up, so a normal SSHEnvironment whose constructor fails still behaves exactly
as before (its __del__ cleanup does not reach the shared socket teardown).
- _remote_home has no reader on the probe path; not assigned.
- prompt_builder passes probe_only=True unconditionally: which backends honor
it is the builder table's decision, not a caller-side env_type check.
- Tests trimmed to the probe_only contract: the cleanup-raises case was green
on main (pre-existing try/except), the socket-name length and double-cleanup
assertions covered pre-existing behaviour.
The prompt-time backend probe built a normal SSHEnvironment just to run a
one-line `uname`. That constructor detects the remote home, creates the
~/.hermes tree, force-uploads every sync file and snapshots a login session;
when the throwaway object was later garbage-collected, __del__ -> cleanup()
ran sync_back() and `ssh -O exit` against the ControlMaster socket the
agent's real environment shares (keyed by user@host:port).
Add an internal probe_only construction path for SSH: an isolated,
same-length ControlMaster socket (keyed by the instance's session id), no
remote dir setup, no FileSyncManager, no session snapshot. The probe now
tears its own connection down explicitly on success, non-zero exit and
exception, without replacing the probe result when cleanup fails. Normal
SSH callers and non-SSH backends are unchanged.
Salvaged from #77933 onto the facade/sibling layout (the probe body moved to
_run_backend_probe, _create_environment to tools/terminal_tool_backends.py).
Halldrix