* fix(relay): accept a provision response that issues no secret
Lockstep with gateway-gateway #223 (finding F-004). The connector used to return
the stored per-gateway secret from POST /relay/provision on EVERY replay, to
anyone reaching the endpoint with the right gatewayId. It now returns credential
material only to a caller that proves recorded ownership or possession, and
answers secretIssued:false otherwise with secret/deliveryKey ABSENT. This
gateway treats that as a hard failure, so #223 cannot merge until this ships.
1. A secret-less response is a VALUE, not an error.
_post_provision() raised on any response without a secret. Two legitimate
responses hit that raise once #223 lands:
- every platform after the first in a multi-platform boot. All platforms share
ONE gatewayId, so the first POST mints the secret and the rest are replays of
an unchanged binding. self_provision_relay() catches RuntimeError per
platform and continues, so a Telegram+Discord gateway would front Telegram
and leave Discord unfronted with a warning.
- a re-provision by a caller that legitimately no longer holds the secret.
Refusing to hand it back IS F-004; /relay/rotate is the way back.
Only a malformed body raises now. No secret AND no secretIssued key is still the
old unexpected-response case (a pre-F-004 connector), so that keeps failing
loudly -- which is what makes this deployable in either order.
2. Only a response that CARRIES credentials may write them.
A separate bug that change 1 exposes rather than causes. The guard tested the
same env var it wrote: once a withheld response reaches it, it writes an empty
secret -- which then reads as "still unset" on the next pass, masking itself,
while GATEWAY_RELAY_ID and _DELIVERY_KEY have ALREADY been stamped from a
response carrying no credentials. Guard on the secret being present.
Tests: 5 invariants in tests/gateway/relay/test_provision_secret_optional.py.
Proven red on base -- reverting gateway/relay/__init__.py to origin/main turns 2
of the 5 red, one per fix, then green with the fix restored. My first draft of
the multi-platform test passed on base because it had the FIRST platform issue
the secret, which satisfies the old guard so it never re-enters and the bug
never fires; it proved nothing until the ordering was fixed and a direct
write-rule test added.
tests/gateway/relay: 303 passed, 0 failed. The 6 failures in the wider
tests/gateway run (systemd_notify, wecom_callback, shutdown_forensics,
scale_to_zero) reproduce identically on clean origin/main -- pre-existing.
* fix(relay): fail closed on the F-004 discriminator; a fully-withheld boot is not a provision
Two review findings on the parent commit, each with a red-on-parent test.
1. `_post_provision()` accepted ANY present `secretIssued` value when `secret`
was absent (`true`, `"false"`, `0`, …) and settled them as a successful
provision. The only credential-less shape the connector emits is literal
JSON `false`, so match the discriminator by value: `is not False` raises.
2. `self_provision_relay()` returned True and logged the INFO "self-provisioned"
line when every platform answered `secretIssued:false` — no credential in
hand, WS upgrade about to be refused, nothing in the boot log to say why.
Now: WARNING naming `/relay/rotate` / pin `GATEWAY_RELAY_SECRET`, return False.
tests/gateway/relay/: 308 passed, 0 failed.
scripts/run_tests.sh tests/gateway/ on a clean macOS checkout fails 7 tests
that pass on the Linux CI lane. The macOS CI lane runs only -m macos_only,
so these had never executed on a Mac. Each failure was diagnosed
individually, not assumed to share a cause:
* test_systemd_notify::test_notify_supports_systemd_abstract_socket
binds a NUL-prefixed abstract AF_UNIX name, a Linux kernel feature.
-> @pytest.mark.linux_only (the abstract socket IS the subject).
* test_shutdown_forensics::test_spawns_subprocess_and_writes_output
spawn_async_diagnostic() wraps a Linux-only script (ps --sort,
/proc/loadavg, dmesg, pstree) in GNU `timeout`, which macOS lacks.
Popen raises, the producer returns None by design (fail-soft).
-> @pytest.mark.linux_only, replacing a bare skipif(win32).
* test_scale_to_zero::test_suspend_self_{posts_suspend,non_2xx}
bound the fake flaps socket under pytest tmp_path, which exceeds
sun_path (104 B on macOS/BSD, 108 on Linux). Production connects to the
fixed /.fly/api so the code is not length-exposed; only the fake needed
a short home. -> short_sock_dir fixture (gettempdir -> /tmp fallback,
skip if neither fits). Verified with a 130-char TMPDIR on Linux.
* test_buzz_adapter::test_live_media_redacts_long_path_before_bounding
built 160-char path components; a single component is capped at 255 B
on both hosts and the full path hit PATH_MAX (1024) on macOS. The
invariant (path redacted BEFORE the 900-char bound) needs the path to
exceed 900 chars, not 1000+. -> grow 80-char components until the path
passes _MAX_CLI_MESSAGE_CHARS, assert < 1024. Mutation-checked: moving
the redaction after the bound fails the test.
* test_wecom_callback::{test_build_event_extracts_text_message,
test_poll_loop_dispatches_handle_message}
NOT environmental: `_build_event` parses with defusedxml, which ships
only via the `wecom` extra (or transitively via `youtube` -> [all] in
CI). A dev venv without it gets ET=None. -> dependency skipif on
DEFUSEDXML_AVAILABLE (not an OS marker; the 4 other tests in the file
still run).
Verified: scripts/run_tests.sh on macOS: 226 passed, 4 skipped (2
linux_only, 2 defusedxml). python:3.11-slim container with [dev,wecom]:
-m "linux_only and not integration" selects and passes both marked tests;
all 5 files 229 passed.
Desktop opens /auth/native/authorize without naming a provider. The empty-provider
auto-select filtered password providers out of the candidate set, so a deployment
with one OAuth provider plus username/password went straight to OAuth and never
offered the password option — even though native sign-in brokers password
providers through /login since 56f1afc834. The filter's rationale ("a password
provider can never be the target of the native flow", ed5e17f4b8) predates that
change.
Render a provider chooser when more than one interactive session provider is
registered. Each link re-enters the same validated authorize route with an
explicit provider, so the choice never leaves the native PKCE flow. A single
provider still auto-selects, and the chooser is emitted before any broker state
is allocated or any cookie set.
The desktop only shell-opens the authorize URL and never parses its response
(apps/desktop/electron/native-oauth-login.ts), so the 200 chooser page is safe on
existing desktop builds.
Supersedes #101713, #76941.
Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>
Co-authored-by: Phuong Lambert <vmphuongit@gmail.com>
A rejected OAuth token (HTTP 401 'User not found' from Nous Portal, Codex,
xAI…) reached the desktop error card as 'Provider error' with Retry as the
first action, which just replays the same dead credential.
Backend: nonretryable_client_error_result dropped failure_reason /
failure_retryable, so error_surface classified every non-retryable 4xx as a
retryable provider failure. It now stamps the classifier verdict like the
max-retries path, and auth-layer descriptors carry auth_kind
(oauth|api_key, derived from the provider catalog tab) + provider_label.
Desktop: an auth/oauth surface renders 'Authentication error', explains that
the <provider> sign-in expired/was revoked, and offers 'Sign in to <provider>
again' which launches that provider's existing onboarding OAuth flow scoped
to the failed session's gateway profile. Retry stays as the follow-up click.
Re-login to the provider already in use keeps the current model instead of
swapping in the recommended default.
Follow-up to #106866 from the post-merge audits (@ehz0ah, @victor-kyriazakos):
- _recoverable_pool_provider: the "endpoint mismatch, not a dead key" shield now (a) compares the full
origin via base_url_origin() (scheme+host+port — a different port or an HTTPS->HTTP downgrade is a
different trust boundary) and (b) applies only when the rejected client carried the SESSION's key, so
an independently owned auxiliary pool keeps rotating at its own configured origin.
- _resolve_named_custom_branch: explicit/per-task base_url and api_key compose OVER the providers.<name>
entry field-by-field instead of the entry silently replacing them (URL-only, key-only, both, neither).
- _acreate_with_progress: plain-create fallback only for a rejected stream NEGOTIATION (mirrors the sync
wrapper); a mid-stream failure propagates to the classified recovery ladder instead of re-sending the
whole prompt non-streaming.
Tests (all red on main): foreign origin x {host, port, scheme} shields the session key; same origin and
independent aux pool still rotate; named-provider override matrix through resolve_provider_client on a
real temp HERMES_HOME; async content-then-error is not retried non-streaming.
Subagent completion notices ("✓ [set 7 · 2/2] … (2175.14s)"), background
process notices and spinner print_above text are emitted from worker threads
through agent._print_fn. In the interactive CLI that was bound straight to
_cprint, so a notice landing while a reply streamed was printed between two
paragraphs inside the response box.
_print_fn is now HermesCLI._agent_status_print: while a response or reasoning
box is open the line is parked in _held_status_lines and released right after
the box footer; outside a box it prints immediately as before. Gateway, ACP
and TUI printers are untouched (they never bound _cprint).
Follow-up on the cherry-picked #106372 fix (@gaoanze888), which reused the
existing 10s _WS_WRITE_TIMEOUT_S as the deadline and only latched _closed.
Why a separate, longer constant: _WS_WRITE_TIMEOUT_S is the worker-side wait
for the LOOP to run the scheduled send; it deliberately does not mark the
transport dead because a GIL-heavy turn stalls the loop for >10s routinely
(#48445/#55545). The new clock is different in kind — asyncio.wait_for starts
it only once the coroutine actually runs on the loop, and a healthy socket
returns from send_text without waiting, so only real kernel backpressure can
consume it (verified: a loop stall 3x longer than the deadline does not trip
wait_for on 3.11/3.12/3.13 because the wakeup and the timer land in the same
iteration and the result wins). Still, tying it to the 10s worker constant
would let one test-time monkeypatch flip both semantics, and the issue asks
explicitly not to declare the peer dead at the 10s worker wait. 30s is 3x the
worker wait and under the client's 45s heartbeat deadline.
Why close the socket: latching _closed alone leaves handle_ws parked in
receive_text on a half-open peer, so the disconnect teardown (session
detach/reap, live-transport unregister, client reconnect) never runs — the
frontend keeps waiting on a socket that will never carry another frame.
ws.close(1011) unblocks the read loop; websockets bounds the close itself
(close_timeout -> abort) when the same full buffer stalls the close frame.
Diagnostic now says which side stalled: "socket stalled, loop responsive"
versus write()'s "loop stalled".
Tests trimmed to the two invariants (≤2 per fix): a stalled send terminates
within the deadline, fails both queued writes, latches closed and closes the
socket; progress → complete → RPC reply ordering is preserved on a healthy
socket. The stalled test uses the issue's own shape (two queued write_async
calls) and monkeypatches the deadline with raising=False so on a base without
the fix it fails on the symptom (sends never terminate), not on a missing
attribute.
Refs #106369
`auxiliary.<task>.provider: openai` was rewritten to custom + api.openai.com/v1 unconditionally, and the
api-key discovery rung used the registry default endpoint, so proxy/gateway users (OPENAI_BASE_URL or
providers.openai) had compression hop to the public endpoint with a proxy-issued key -> 401 -> the
key/pool was quarantined and the session sat over the compression threshold.
- _expand_direct_api_alias: a providers.openai entry keeps its name (named-custom branch applies its
base_url/key); otherwise OPENAI_BASE_URL wins over the public default.
- _resolve_api_key_provider: when the bound session runtime is that provider, use its endpoint + key.
- _recoverable_pool_provider: a rejection at a host other than the session's configured endpoint for
the same provider is an endpoint mismatch, not a dead key -> no rotation / unhealthy mark on the pool.
Complete the seam default from the salvaged commit: _relay_async_completion needs an async twin of
_create_with_progress, and the async primary attempt previously streamed only for stream-only
providers, so a hooked async compression call ticked the watchdog zero times.
Adds one invariant test: with a hook installed, BOTH relay defaults stream and tick per chunk;
without a hook they are byte-identical plain creates. Red on origin/main.
* feat: add session-scoped connector access for onboarding
* fix(connectors): availability is the config flag AND the portal entitlement — no free-tier leg
The port carried a third availability leg from hermes-magic: a stored guest
(free-tier) identity short-circuits the managed-tool entitlement check. That
leg reads hermes_cli.anon_auth, which does not exist on hermes-agent main, so
connectors_available() raised ImportError inside its fail-closed try and the
whole connector surface was silently dark on a plain upstream checkout.
On this tree availability is the two-leg AND the design started with:
tools.connectors.enabled AND managed_nous_tools_enabled(). The free-tier leg
is a hermes-magic concern and belongs in hermes-magic's own delta over this
branch, next to the identity it depends on. Its integration test goes with it.
* docs(tool-search): connectors section — remote tools through the bridge
The squashed port carried the code but not the user-facing docs. Restores the
Connectors section of the Tool Search page and the connector-gateway host /
CONNECTOR_GATEWAY_URL override on the Tool Gateway page, updated for the
manage_connections tool and the pure-connector batch rule.
* fix(tool-search): connector tools rank with local tools in one pass instead of taking leftover slots
dispatch_tool_search ran BM25 over the local catalog, filled `limit` slots,
then appended connector hits only into slots left empty. On a 300-tool
catalog no slot was ever empty, so with Gmail and Google Calendar connected
"send gmail email" returned five betterstack tools and zero connector tools.
The gateway's hits for a query now become catalog entries (connector name,
slug words, description as the search text) and join the local catalog for
that query's BM25 pass. One ranking, one rarest-token admission rule for both
sources, `limit` as the total per query. The merge loop and the separate
record builder for connector hits are gone; `_shared_tool_record` serves both
sources.
The gateway search timeout rises from 8 s to 30 s. One request with six
use_cases measured 7 s, so 8 s sat on the edge and cut real answers off; the
failure path is unchanged (local-only results, no error to the model).
Live, 311 local tools + gateway, before -> after:
"send gmail email": 5 betterstack tools -> gmail SEND_EMAIL, CREATE_EMAIL_DRAFT
"read google calendar events": 5 betterstack tools -> googlecalendar EVENTS_LIST_ALL_CALENDARS
"linear create issue", "betterstack incident": unchanged
Benchmark (25 labelled queries): connector recall 0.09 -> 0.82, precision@5
0.18 -> 0.59, false positives on absent intents 17 -> 2.
* refactor(tool-search): connector leg into tools/connector_search.py
tools/tool_search.py is a facade. The connector leg (gateway hits as catalog
entries for tool_search, remote schemas for tool_describe, the
connections_in_scope gate) was appended to it by the port. It now lives in
its own sibling, tools/connector_search.py, and the facade imports the three
entry points: connections_in_scope, connector_entries_by_group,
remote_schemas_for.
No behaviour change. The tool_describe remote block became
remote_schemas_for(names, current_tool_defs, connector_describe) with the
same inputs, the same silent-degradation contract and the same injection
seam the tests already use.
* fix(tool-search): at most 7 queries per call, the gateway's search limit
One tool_search call sends all its queries to the connector gateway as one
search request. The gateway answers 7 use_cases per request and returns
HTTP 502 for 8 or more (measured 2026-09-09, re-measured with one-word
use_cases: it is a count limit, not a size limit). With the client cap at
10, a model sending 8 to 10 queries lost every connector hit for that call
and saw local-only results with no error.
The shared constant splits: _MAX_QUERIES_PER_CALL = 7 for search,
_MAX_DESCRIBE_NAMES_PER_CALL = 10 for describe, which has no remote count
limit. Eight or more queries now get the existing "too many queries" retry
hint before any request is made. No chunking: one call, one request.
* fix(tool-search): the model is told that connectors__ names are manage_connections accounts
tool_search results carry names like connectors__gmail__CREATE_EMAIL_DRAFT and
manage_connections is the tool that checks and connects those accounts, but
nothing told the model the two are the same thing. A model that hit
CONNECTION_REQUIRED had to infer the fix on its own.
The tool_search description gains one sentence making the link, added at
assembly only when manage_connections is in the session's tools. Signed out
or with connectors off the tool is absent and the description is unchanged,
so it never names a tool the model cannot call. This follows the existing
rule for cross-tool references (tools/AGENTS.md): they are added dynamically
from the session's actual tool set, never hardcoded in a schema.
Tool defs are fixed for the life of a conversation, so the description is
byte-stable per conversation; this is a one-time prefix change.
Live, real get_tool_definitions() against a signed-in home: sentence present.
Same home with auth.json removed: manage_connections absent, sentence absent.
* fix(connectors): /stop halts a connector batch before the next remote call
dispatch_connector_batch runs every remote entry of a tool_call batch in
sequence. The executor only checks the interrupt flag between tools, and
the whole batch is one tool to it, so a /stop landing during entry 1 of
20 still sent the other 19 to the gateway.
The loop now reads tools.interrupt.is_interrupted before each dispatch.
Once set, it stops calling handle_function_call and fills every unstarted
slot with the loop's existing error-slot shape, code INTERRUPTED and the
message "Stopped by the user before this call was made.", so the result
envelope stays valid and the counts stay honest. Entries already
dispatched keep their real results.
Test: three connector calls where the fake client sets the interrupt on
the first execute. The client sees exactly one call and slots 2 and 3
carry INTERRUPTED. Red on the base branch, green with the fix.
* test(connections): schema assertions become dispatch contracts
test_schema_documents_wait_and_its_timeout froze description fragments
("REQUIRED", "can NOT disconnect", "Nous Portal"). A wording edit fails
it while a real regression (a disconnect that reaches the gateway) does
not. That is a snapshot of prose, not a behaviour contract.
Delete it. The requirement that wait needs connectors is already covered
by test_wait_requires_connectors. The user-only disconnect boundary is
now asserted as behaviour: action disconnect with a connector returns an
error and the fake client records no call. That replaces the earlier
de-authenticate test, which only checked that the word "dashboard"
appeared in the error text.
Test count in the file goes from 26 to 25.
* docs(tool-search): connector batches are one gateway request per entry
The user guide said a connector batch travels as one gateway request. It
does not: model_tools_connectors.dispatch_connector_batch re-enters core
dispatch per entry, and each entry becomes its own execute request in
bridge._run_remote (plus at most one literal-slug retry when the gateway
reports TOOL_NOT_FOUND under the conventional slug). The docstrings in
tools/tool_gateway/bridge.py and tools/tool_gateway/__init__.py still
described the abandoned V1 plan and claimed nothing outside the package
imports it.
Rewrite those sentences to match the code: one request per entry, in
input order, dispatched from model_tools_connectors.py, with the per-entry
approval and interrupt behaviour that motivated the split. The guide also
still showed the single-call shape tool_call(name, arguments); both
places now show the `calls: [{name, arguments}]` array the schema
advertises and note that a single local call is an array of one.
Docs only, no test.
* fix(tools): the between-turns refresh never rewrites the bridge tools
The per-turn MCP refresh folds a fresh tool snapshot into the live array
with preserve_prefix: order and membership stay, but a name present in both
takes the fresh schema. That is right for ordinary tools, whose schema is a
constant. tool_search is the one tool whose description is derived from the
session: the deferred-tool count, the embedded listing, and, on this branch,
whether manage_connections was present. A late MCP server or one failed
portal lookup (manage_connections' check_fn fails closed) changed those bytes
on the next turn, and every byte after tool_search in the cached prefix was
re-prefilled. The array also contradicted itself in that case: the flapping
manage_connections was carried forward while the description lost its hint.
The bridge entries now keep the bytes they were built with for the life of
the conversation. Nothing is lost: tool_search reads the live catalog at
dispatch, so tools that arrived late are still found; connector availability
is checked at dispatch too. The compaction-boundary rebuild (content_aware,
the one sanctioned cache break) still refreshes the description.
Consequence: connector exposure in the prompt is decided once, at agent
build, by whether the user was signed in then. That is the intended
contract.
* refactor(tool-search): normalize_tool_call_entries lives with the other argument validation
The port appended the tool_call argument parser to the tool_search facade.
The family already has tools/tool_search_validation.py for exactly this
work (schema validation of deferred call arguments), so the parser moves
there and the facade imports it. No behaviour change; the one test that
imported it now imports from the defining module.
* refactor(connectors): delete the unused batch dispatcher; _run_remote becomes run_remote
bridge.dispatch_calls and its helpers (_dispatch_calls_inner, _run_pre_dispatch,
_run_local, _error_slot, _maybe_parse_json) and the LocalDispatch / PreDispatch
seams had no production caller. Connector dispatch runs through
model_tools_connectors: dispatch_connector_batch re-enters handle_function_call
once per entry, so scope, hook, approval and middleware policy fire against each
composed name inside core dispatch, and dispatch_connector_call hands the single
planned entry to the bridge's transport function. Only tests called the batch
dispatcher, and they exercised policy seams that production never wires.
The transport function is the module's real entry point, so it drops the
underscore: _run_remote becomes run_remote, body unchanged. The module
docstring now describes the two legs that exist (availability with D32 silent
degradation, and run_remote) instead of the injected seams. Imports that only
the deleted code used are gone; merge.py is untouched because every export
still has a caller.
Tests that drove dispatch_calls are deleted where they covered the removed
seams (pre_dispatch blocks and rewrites, local_dispatch classification, mixed
batches). The literal-slug fallback, the per-entry transport failure, and the
hook rewrite reaching the gateway request body are re-targeted at
handle_function_call('tool_call', ...) with the fake client swapped in at
bridge._default_client_factory, the same seam test_connector_dispatch_policy
uses. Each re-targeted test fails when the retry is disabled in run_remote.
* fix(connectors): search keeps the twin a colliding name reaches, and says so
format_connector_name strips the toolkit prefix, so GMAIL_FETCH_PROFILE and a
literal FETCH_PROFILE on gmail both compose to connectors__gmail__FETCH_PROFILE.
describe and execute decode that name to the prefixed slug first, so the
literal twin is unreachable under it. If a vendor ever shipped both, search
could describe the literal under a name that runs the prefixed tool.
Search is the one place that sees both twins in one response. It now keeps
the twin the name reaches and drops the other with a WARNING that names both
slugs, whichever the gateway listed first. Short names stay; no marker, no
per-process map, no change to describe or execute. No such pair exists in the
live catalog today; the guard turns a silent alias into a logged one.
search_catalog admitted every document with BM25 score > 0 and then padded
to `limit`. BM25 sums over the tokens a document shares with the query, so
on a 300-tool catalog "send gmail email" returned five incident tools that
shared only "email", and the discriminating word ("gmail", in no document)
had no say. The model read those as the answer.
Admission is now the query's rarest token: a document is a result only if it
contains the query token with the highest IDF, the one that names the intent.
Common verbs ("send", "read", "create") sit in dozens of documents and never
gate; vendor and object words ("gmail", "github", "incident") do. A token no
document carries admits nothing, and the existing empty-group hint tells the
model to retry without it. The name-substring fallback is deleted: it admitted
tools that matched no query token at all.
Result descriptions are clipped at 500 characters instead of 400. Over 353
vendor tool descriptions, 500 keeps 91% whole and every first sentence
(first-sentence max 329); 400 kept 82%.
Measured on the live 311-tool catalog with 25 hand-labelled queries:
precision@5 0.18 -> 0.43, wrong names returned 102 -> 66, false positives on
absent intents 17 -> 13. Live before/after: "send gmail email" went from five
betterstack tools to an empty group with the retry hint; "linear create issue"
and "betterstack incident" are unchanged.
claim_job_for_fire refused any claim younger than FIRE_CLAIM_TTL_SECONDS (300s)
regardless of whether its owner was still alive, so a `hermes cron run` killed
mid-flight (timeout, Ctrl-C, OOM) blocked the next manual run with "already being
fired" for up to five minutes. The executions table already reaps dead owners on
sight; the job-record fire_claim now does the same via gateway.status._pid_exists
when the claim's `by` names a pid on this host. Foreign hosts, explicit
HERMES_MACHINE_ID values, and probe failures keep the TTL (fail safe).
#106623 (ca16cafee4) blocks any test subprocess whose argv resolves to
`hermes gateway run|start|restart`, so the harness cannot spawn a runtime that
outlives the worker and restarts the developer's gateway. The matcher strips the
`-p <profile>` selector and reads through `sh -c`, so it also fired on
`docker exec -u hermes <ctr> sh -c 'hermes -p x gateway start'` and broke
tests/docker/test_profile_gateway.py on every Docker build since. A gateway
launched inside a container cannot reach the host unit or webhook port; the
guard now skips commands whose argv[0] is a container runtime (docker, podman,
nerdctl). Host-side spawns stay blocked.
_is_azure_foundry_responses matches the azure-foundry provider or the
project-scoped services.ai.azure.com host. #101243 proposes narrowing it
to that host alone. The multi-item rejection this branch handles happens
on resource-level *.openai.azure.com hosts too, so the pruning gate now
uses _is_azure_responses: the provider, or either Azure host.
Refs #105369
The first request of every turn after the first replays the encrypted
reasoning item of every earlier response. Azure Foundry accepts one such
item and rejects two or more with HTTP 400 "Conflicting authenticated
continuation identities". In-turn requests never hit this because
_is_post_tool_replay already suppresses replay on Azure when the request
ends on a tool result.
On Azure Foundry, build_kwargs now keeps codex_reasoning_items only on the
newest assistant row that has any. include still requests
reasoning.encrypted_content, compaction checkpoints stay on every row, and
the canonical messages are not mutated. Other Responses endpoints are
unchanged.
Rebuilding a failed session from state.db through the transport and
sending it live: unpatched, 5 reasoning items, 400; patched, 1 item, 200.
Fixes#105369
Azure Foundry (gpt-6-astra) answers a request that replays encrypted
reasoning from more than one prior response with HTTP 400 "Conflicting
authenticated continuation identities", code invalid_value. The classifier
mapped that to a non-retryable client error, so the turn aborted and every
later turn in the session failed the same way.
Map the message to invalid_encrypted_content. The existing recovery in
turn_recovery then strips the replay state and retries once.
Refs #105369
The Desktop settings UI persists a registry provider's API key as a
credential pointer -- model.key_env in config.yaml pointing at an env var
in $HERMES_HOME/.env (e.g. HERMES_CUSTOM_LMSTUDIO_API_KEY) -- while
keeping model.provider on the registry id. _resolve_api_key_provider_secret
resolved registry providers exclusively from
PROVIDER_REGISTRY[...].api_key_env_vars, so the UI-saved key was silently
ignored; for lmstudio the runtime then substituted the no-auth placeholder
(dummy-lm-api-key) and auth-enabled LM Studio servers rejected every
request with an opaque, retryable-flagged 401.
Fix: consult model.key_env (only when config.yaml's main model targets the
provider being resolved, so the pointer never leaks across providers)
before the registry env vars, running the value through the same
_usable_declared_secret prefix validation. Precedence for the canonical
env vars and the credential pool is unchanged; a valid canonical key still
wins when no pointer is set.
Complements #75373, which covers providers.<id>.key_env at the provider
level; the Desktop save path writes the pointer at the model level, which
that PR does not consult.
Fixes#106336
Salvage bar is at most two invariant tests per fix. The monotonic-watermark
case (small, oversized, small -> is_truncated(s, 1)) is the same contract the
oversized-frame test already pins, so it lives there as a final block instead
of a third test function.
Move the "explicit scratch means no project" decision into the one resolver
every surface funnels through, `kanban_db.create_task`: board-project
inheritance now runs only when the caller left `workspace_kind` open
(`None`), and `workspace_kind` defaults to scratch after that check. The
tool handler keeps the #106347 fix for `project=""` (no `or` collapse) and
`board=` scoping but drops its handler-local sentinel logic, since the
resolver now owns the rule; the `self_task` project inheritance for
dispatcher-owned workers is unchanged.
Sibling surfaces had the same bug through the same line and are fixed by
the same change:
- CLI `hermes kanban create --workspace scratch` on a project-scoped board
produced a project worktree; `--workspace` no longer defaults in
argparse so the resolver can tell "omitted" from "scratch".
- Dashboard `POST /tasks` with `workspace_kind: "scratch"` did the same;
`CreateTaskBody.workspace_kind` defaults to `None` for the same reason.
- `kanban_swarm.create_swarm` threads `None` through for consistency.
Tests: one resolver invariant in test_kanban_board_project.py (explicit
scratch stays scratch, omitted still inherits) and the salvaged tool test
folded into a single parametrized matrix over scoped/unscoped target boards.
Pass the caller's board into create_task and treat explicit workspace_kind=scratch or empty project as no-project so ambient board project_id cannot override MCP args.
Co-authored-by: Cursor <cursoragent@cursor.com>
The salvaged fix only ran the CLI in its own session on POSIX and kept
plain subprocess.run on Windows — the one platform where the wedge in
#106244 is actually reproducible: CPython's run() retries an unbounded
communicate() after kill() there, so a grandchild holding the capture
pipes blocks the worker forever. On POSIX run() wait()s the PID and
returns promptly; the grandchild merely leaks (live-reproduced on Linux).
One code path for both platforms:
- _group_popen_kwargs: start_new_session=True on POSIX,
CREATE_NEW_PROCESS_GROUP + hide flags on Windows (replaces the
hide-only _windows_popen_kwargs).
- _kill_cli_process_group: os.killpg SIGKILL on POSIX, taskkill /T /F
on Windows (same kwarg set as the sibling taskkill sites).
- The Popen decodes with encoding="utf-8", errors="replace" like every
other subprocess call in this file (windows footgun rule).
- Drain test patches the kill helper instead of os.killpg so it runs on
every host.
Windows behaviour is not live-verifiable on this Linux host.
Drop test_gone_group_still_surfaces_timeout: the killpg-race branch is a
contextlib.suppress and the test only pinned the exact communicate()
timeout sequence (a change-detector). The real-grandchild test and the
bounded-drain test remain — the two behaviour contracts of the fix.
subprocess.run only kills the direct CLI child on TimeoutExpired; a
browser_harness daemon / Chrome helper grandchild inherits the stdout/
stderr pipes and keeps them open, so the internal communicate() blocks
on pipe EOF forever. The wedged tool call never returns, its activity
heartbeat keeps stamping last_activity_at every 30s, and the session is
pinned at "now" in the desktop sidebar indefinitely (#106244).
On POSIX, run the CLI in its own session (start_new_session=True) and
SIGKILL the whole process group on timeout, then drain the pipes under
a bounded deadline. Windows keeps subprocess.run.
Replace the six cap-mapping unit tests (exact 5536 px value, floor, Anthropic
path, two None cases, worst-case tile math) with one behavioral test: the cap
derived from the real Codex rejection makes the production shrink pass rewrite
a real 5444x6200 PNG Responses ``input_image`` part to within the 30000-patch
budget. That is the user-visible contract; the exact pixel value is an
implementation detail. Keeps the classifier test as the second invariant.
The classifier fix routes Codex patch-budget 400s into the shrink
recovery, but _image_error_max_dimension still returned None for the
Codex wording, so the recovery fell back to the 8000 px default cap
and skipped images between ~5542 and 8000 px that already exceed the
30000-tile budget — burning the single shrink retry without shrinking
anything. Parse the reported patch limit and convert it to a per-side
pixel cap of isqrt(limit)*32 (5536 px for 30000 patches), which keeps
a square image under the budget.
OpenAI Codex Responses rejects an image whose tile-patch budget exceeds
its 30000-patch ceiling with wording ("requires N patches after
processing, exceeding the limit") that contains none of the existing
image-size vocabulary, so the 400 fell through to format_error
(non-retryable). The reactive image-shrink recovery in turn_recovery was
therefore bypassed and the session kept failing — or failover re-sent
the identical oversized image to another model.
Route the patch-budget wording to image_too_large (retryable) so the
existing shrink pass re-encodes the image under the ceiling and retries.
Fixes#106337
Drop the floor-pin test from #106290 (a snapshot of the static table); the
end-to-end merge test through provider_model_ids with the real curated floor
already fails if the slug is re-added. Note beside
_OPENCODE_FREE_KEYED_SUFFIX_MODELS why ox-alpha-free stays excluded from the
keyless catalog even though it left the curated floor.
The Go relay (GET /zen/go/v1/models) delisted ox-alpha-free 2026-09-09, but
_PROVIDER_MODELS["opencode-go"] still carried it. _profile_live_catalog merges
the curated floor into the live list (live-first for opencode-go), so the
model picker kept offering a model that now 401s — the same failure class as
#95914 (opencode-free / x-preview-f-free).
Remove it from the floor and add two regression tests: an end-to-end merge
test through provider_model_ids with the real floor (fails if a stale floor
resurrects it) and a floor-pin test asserting the known-delisted model stays
out of the offline fallback.
(cherry picked from commit 091fd85865caa09e828928f93ee985dae7e9580d)
Keep exactly the issue's Repro A (heartbeat must not become the cached answer)
and Repro B (stale READY mapping / expired button must not swallow the next
answer); both fail on origin/main and pass with the fix. The other six tests
from #106470 (config override, PENDING-tap guard, per-layer variants) were
either change-detectors or subsumed by these two.
Keep the two tests that fail on main without the fix:
- test_fal_common: a keyed managed submit makes exactly one POST (plus the
negative arm: an unkeyed submit still goes through the SDK retry ladder)
- test_image_generation: the 409 BILLING_ERROR body surfaces
`unsupported_pricing_meter` instead of the generic "not yet enabled" text
Dropped from #106484: the duplicate video-plugin billing test (same helper,
same assertion), the `_fal_client = fake` / `import_fal_client` stub churn
and the `tools.lazy_deps` stub — fal-client is installed in CI (`--extra fal`)
so those fixtures were not needed; the `_load_fal_client` no-op fixture on
TestManagedGatewayErrorTranslation for the same reason.
Also drop the redundant `retry_request is None` re-check in
`_ManagedFalSyncClient.submit` — `__init__` already raises when the helper
is missing.
Avoid retrying idempotent managed FAL submissions because the retry can mask the initial billing failure. Surface structured Nous billing diagnostics consistently for image and video paths, with hermetic regression coverage.
(cherry picked from commit 289ce039e9a522dc8016ae4a512214c05d0a8bc0)
finish_reason='length' has two causes: the answer was long (max_tokens reached),
or the prompt itself left no room to generate. _continue_text treated both the
same: append the fragment + a continuation nudge and retry, up to 4 times. In the
second case every retry sends a strictly longer prompt, so each attempt is worse
(Ollama n_ctx=32768: 32,638 -> 32,685 -> 32,732 prompt tokens, all truncated), the
user is told "model hit max output tokens", and max_tokens is not the lever.
The response's usage already carries prompt_tokens and the compressor already
resolves the model's context window; compare them once per truncation. Under
_MIN_CONTINUATION_HEADROOM (512) free tokens the turn ends on the first
truncation, keeps the partial text, names the context window as the cause and
points at /compress or a larger window. Unknown usage or window keeps today's
behaviour. max_tokens semantics untouched.
Co-authored-by: gaoanze888 <214786078+gaoanze888@users.noreply.github.com>
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Drop test_reap_idle_closes_doomed_and_keeps_attached from #106415: it
pins reap_idle's pre-existing selection behaviour (dead/idle reaped,
attached kept), which the fix does not change, and is not red on base.
Fold the duplicated two-idle-sessions + gated-close setup into one
helper shared by the two interleaving tests that do fail without the
fix (reap vs reap, close_all vs reap).
reap_idle() is reached concurrently from attach_or_spawn() and the
background run_reaper loop, and each pop is followed by an awaited
close(); while one reap is parked inside close(), a second reap reaching
the same doomed key hit dict.pop(key) without a default and raised
KeyError straight into the websocket handler. close_all() has the same
overlap window against an in-flight reap.
Pop with a default and skip keys a concurrent reap already removed, so
overlapping reaps are idempotent. Which sessions get reaped is
unchanged.
Fixes#106400
(cherry picked from commit 8ea63a845424263551487886a7526ca2f4bbd1ee)
The two facades post-#102117 (chat_completion_helpers, agent_runtime_helpers) must not grow
behaviour; the marker/predicate belong with the sibling that already owns primary-cooldown
state shared by the fallback walk and restore_primary_runtime. Also drops the two
try/except-return-False wrappers around pool.entries() and normalize_model_for_provider
(neither raises for a live pool / the never-raising normalizer).
Tests trimmed to the two invariants (#106475): the marker fires only on the single-credential
Codex entitlement 400 and the walk skips the rejected slug in either form; restore is gated on
a rejected primary and still restores an unrelated one.
A Codex ChatGPT-account 400 ('The X model is not supported when using Codex
with a ChatGPT account.') names the model, so with a single credential the
slug is dead for that account. The fallback walk still re-selected it and
restore_primary_runtime switched back to the primary at the start of every
turn, announcing an unverified 'Primary model restored' — the two warnings
alternated forever with zero delivered answers (#106475).
Record the rejected (provider, model) pair on the non-retryable client-error
path (only when no multi-credential pool exists — rotation covers that case,
#71970), skip rejected entries during the fallback walk, and gate
restore_primary_runtime on the primary's slug so the session fails closed
with the terminal entitlement error instead of oscillating. Fixes#106475.
(cherry picked from commit 471435b10288f15387b2549a8b02551d82c0f670)
The parametrized lifecycle test drove both transports through a
ClosedResourceError on the first call_tool and asserted a replay. For the
HTTP arm that is still the session-expired contract; for the stdio arm a
pipe closing after dispatch is now the ambiguous mid-call death and must
surface outcome_uncertain with exactly one invocation. The transport is
still rebuilt on both arms.
The child watcher polls every 250 ms, so in practice the MCP SDK sees the
closed pipe first and `call_tool` raises ClosedResourceError / "Connection
closed" before the watcher fires. That exception is not a _StdioChildExited,
so it fell past the stdio recoverer into _handle_session_expired_and_retry,
which reconnects and replays the call -- the exact duplicated-side-effect
path the previous two commits close. Live repro against a real stdio child
that applies an effect then exits without replying: 2 effects with the
contributor's commits alone, 1 effect + outcome_uncertain after this.
On a stdio server, a session-expired-class error raised by an RPC that was
already dispatched is re-raised as _StdioChildExited(in_flight=True) so the
stdio recoverer owns it. HTTP servers are untouched: their session-expired
retry is still the right recovery.
Tests trimmed to the salvage bar: the contributor's
test_precall_respawn_retry_dying_midcall_is_uncertain_without_replay
(a variant of the watcher-race case) is replaced by the SDK-first regression
the review on #106440 asked for; the existing pre-call retry test still pins
that a never-sent call is retried once.
The gateway's startup warm-up (_warm_turn_machinery_sync) primed the
run_agent import graph, tool schemas and context files, but left the
process-local Python toolchain probe to the first AIAgent. That agent
starts the probe worker in its constructor and its first system-prompt
build then blocks in get_environment_probe_line() on the python/pip
subprocesses (~250 ms measured) before the first model request — with
local models too, so provider metadata warm-up (#105999) does not cover it.
Call the same resolver from the warm-up: it owns the agent.environment_probe
opt-out check here, and the resolver itself owns remote-backend omission,
the single worker, its cache and the bounded fail-open wait, so the first
turn simply finds the line cached. No agent is constructed and no session
prompt is frozen at startup; a disabled probe or a remote terminal backend
leaves the host untouched.
Slim redo of #106075 by @francip: same mechanism, folded into the existing
sync warm-up instead of a new module + asyncio.gather + second suppress
block; the two behavior tests are kept, the concurrency tests dropped.
Co-authored-by: Franci Penov <49422+francip@users.noreply.github.com>
The curated fallback is the only list subscription tokens and lagged
/v1/models responses see. Assert Opus 5 and Fable 5.1 stay on that list,
newest-first, including when live discovery omits them.
The salvaged detector recorded a second MONOSPACE range from a hand-rolled
position walker and then had to rebase earlier ranges by the padding delta
(the contributor's follow-up commit). Two range bookkeepers over one text is
where the bugs were: a pipe table inside a fenced code block was realigned
AND double-styled, because the walker ran after the fences were stripped and
could no longer tell code from table.
Run table realignment on the raw markdown instead, outside fences only, and
wrap each aligned table in a fence. The existing code-block pass then emits
exactly one MONOSPACE range per table and the existing removal/adjust
machinery shifts every later range — no second offset model, no rebase loop.
The contributor's fixup commit becomes unnecessary and is not carried.
Tests trimmed to two invariants: a CJK/emoji table aligns by display width
with correct UTF-16 offsets for the range and the style after it, and a
fenced pipe table stays code while a real table before it keeps a later code
range pointed at its own text (also covers the lone-pipe negative).
Refs #106181
markdown_to_signal() passed GFM tables through untouched, so Signal's
proportional font left columns unaligned. Detect table blocks after
code-block extraction, re-align them with realign_markdown_tables(),
and cover each rendered block with a single MONOSPACE style range.
Fixes#106181
Replace the 16+4 parametrized delivery cases with the two behaviors the fix
exists for: the None tool-boundary delta releases already-streamed speech
before the tool result arrives, and completed commentary reaches TTS exactly
once. Both fail on main with the SentenceChunker TypeError. The other
permutations (transcript on/off, interim opt-out, terminal guards) exercise
paths the fix did not change.
Ben Barclay