* feat: add session-scoped connector access for onboarding
* fix(connectors): availability is the config flag AND the portal entitlement — no free-tier leg
The port carried a third availability leg from hermes-magic: a stored guest
(free-tier) identity short-circuits the managed-tool entitlement check. That
leg reads hermes_cli.anon_auth, which does not exist on hermes-agent main, so
connectors_available() raised ImportError inside its fail-closed try and the
whole connector surface was silently dark on a plain upstream checkout.
On this tree availability is the two-leg AND the design started with:
tools.connectors.enabled AND managed_nous_tools_enabled(). The free-tier leg
is a hermes-magic concern and belongs in hermes-magic's own delta over this
branch, next to the identity it depends on. Its integration test goes with it.
* docs(tool-search): connectors section — remote tools through the bridge
The squashed port carried the code but not the user-facing docs. Restores the
Connectors section of the Tool Search page and the connector-gateway host /
CONNECTOR_GATEWAY_URL override on the Tool Gateway page, updated for the
manage_connections tool and the pure-connector batch rule.
* fix(tool-search): connector tools rank with local tools in one pass instead of taking leftover slots
dispatch_tool_search ran BM25 over the local catalog, filled `limit` slots,
then appended connector hits only into slots left empty. On a 300-tool
catalog no slot was ever empty, so with Gmail and Google Calendar connected
"send gmail email" returned five betterstack tools and zero connector tools.
The gateway's hits for a query now become catalog entries (connector name,
slug words, description as the search text) and join the local catalog for
that query's BM25 pass. One ranking, one rarest-token admission rule for both
sources, `limit` as the total per query. The merge loop and the separate
record builder for connector hits are gone; `_shared_tool_record` serves both
sources.
The gateway search timeout rises from 8 s to 30 s. One request with six
use_cases measured 7 s, so 8 s sat on the edge and cut real answers off; the
failure path is unchanged (local-only results, no error to the model).
Live, 311 local tools + gateway, before -> after:
"send gmail email": 5 betterstack tools -> gmail SEND_EMAIL, CREATE_EMAIL_DRAFT
"read google calendar events": 5 betterstack tools -> googlecalendar EVENTS_LIST_ALL_CALENDARS
"linear create issue", "betterstack incident": unchanged
Benchmark (25 labelled queries): connector recall 0.09 -> 0.82, precision@5
0.18 -> 0.59, false positives on absent intents 17 -> 2.
* refactor(tool-search): connector leg into tools/connector_search.py
tools/tool_search.py is a facade. The connector leg (gateway hits as catalog
entries for tool_search, remote schemas for tool_describe, the
connections_in_scope gate) was appended to it by the port. It now lives in
its own sibling, tools/connector_search.py, and the facade imports the three
entry points: connections_in_scope, connector_entries_by_group,
remote_schemas_for.
No behaviour change. The tool_describe remote block became
remote_schemas_for(names, current_tool_defs, connector_describe) with the
same inputs, the same silent-degradation contract and the same injection
seam the tests already use.
* fix(tool-search): at most 7 queries per call, the gateway's search limit
One tool_search call sends all its queries to the connector gateway as one
search request. The gateway answers 7 use_cases per request and returns
HTTP 502 for 8 or more (measured 2026-09-09, re-measured with one-word
use_cases: it is a count limit, not a size limit). With the client cap at
10, a model sending 8 to 10 queries lost every connector hit for that call
and saw local-only results with no error.
The shared constant splits: _MAX_QUERIES_PER_CALL = 7 for search,
_MAX_DESCRIBE_NAMES_PER_CALL = 10 for describe, which has no remote count
limit. Eight or more queries now get the existing "too many queries" retry
hint before any request is made. No chunking: one call, one request.
* fix(tool-search): the model is told that connectors__ names are manage_connections accounts
tool_search results carry names like connectors__gmail__CREATE_EMAIL_DRAFT and
manage_connections is the tool that checks and connects those accounts, but
nothing told the model the two are the same thing. A model that hit
CONNECTION_REQUIRED had to infer the fix on its own.
The tool_search description gains one sentence making the link, added at
assembly only when manage_connections is in the session's tools. Signed out
or with connectors off the tool is absent and the description is unchanged,
so it never names a tool the model cannot call. This follows the existing
rule for cross-tool references (tools/AGENTS.md): they are added dynamically
from the session's actual tool set, never hardcoded in a schema.
Tool defs are fixed for the life of a conversation, so the description is
byte-stable per conversation; this is a one-time prefix change.
Live, real get_tool_definitions() against a signed-in home: sentence present.
Same home with auth.json removed: manage_connections absent, sentence absent.
* fix(connectors): /stop halts a connector batch before the next remote call
dispatch_connector_batch runs every remote entry of a tool_call batch in
sequence. The executor only checks the interrupt flag between tools, and
the whole batch is one tool to it, so a /stop landing during entry 1 of
20 still sent the other 19 to the gateway.
The loop now reads tools.interrupt.is_interrupted before each dispatch.
Once set, it stops calling handle_function_call and fills every unstarted
slot with the loop's existing error-slot shape, code INTERRUPTED and the
message "Stopped by the user before this call was made.", so the result
envelope stays valid and the counts stay honest. Entries already
dispatched keep their real results.
Test: three connector calls where the fake client sets the interrupt on
the first execute. The client sees exactly one call and slots 2 and 3
carry INTERRUPTED. Red on the base branch, green with the fix.
* test(connections): schema assertions become dispatch contracts
test_schema_documents_wait_and_its_timeout froze description fragments
("REQUIRED", "can NOT disconnect", "Nous Portal"). A wording edit fails
it while a real regression (a disconnect that reaches the gateway) does
not. That is a snapshot of prose, not a behaviour contract.
Delete it. The requirement that wait needs connectors is already covered
by test_wait_requires_connectors. The user-only disconnect boundary is
now asserted as behaviour: action disconnect with a connector returns an
error and the fake client records no call. That replaces the earlier
de-authenticate test, which only checked that the word "dashboard"
appeared in the error text.
Test count in the file goes from 26 to 25.
* docs(tool-search): connector batches are one gateway request per entry
The user guide said a connector batch travels as one gateway request. It
does not: model_tools_connectors.dispatch_connector_batch re-enters core
dispatch per entry, and each entry becomes its own execute request in
bridge._run_remote (plus at most one literal-slug retry when the gateway
reports TOOL_NOT_FOUND under the conventional slug). The docstrings in
tools/tool_gateway/bridge.py and tools/tool_gateway/__init__.py still
described the abandoned V1 plan and claimed nothing outside the package
imports it.
Rewrite those sentences to match the code: one request per entry, in
input order, dispatched from model_tools_connectors.py, with the per-entry
approval and interrupt behaviour that motivated the split. The guide also
still showed the single-call shape tool_call(name, arguments); both
places now show the `calls: [{name, arguments}]` array the schema
advertises and note that a single local call is an array of one.
Docs only, no test.
* fix(tools): the between-turns refresh never rewrites the bridge tools
The per-turn MCP refresh folds a fresh tool snapshot into the live array
with preserve_prefix: order and membership stay, but a name present in both
takes the fresh schema. That is right for ordinary tools, whose schema is a
constant. tool_search is the one tool whose description is derived from the
session: the deferred-tool count, the embedded listing, and, on this branch,
whether manage_connections was present. A late MCP server or one failed
portal lookup (manage_connections' check_fn fails closed) changed those bytes
on the next turn, and every byte after tool_search in the cached prefix was
re-prefilled. The array also contradicted itself in that case: the flapping
manage_connections was carried forward while the description lost its hint.
The bridge entries now keep the bytes they were built with for the life of
the conversation. Nothing is lost: tool_search reads the live catalog at
dispatch, so tools that arrived late are still found; connector availability
is checked at dispatch too. The compaction-boundary rebuild (content_aware,
the one sanctioned cache break) still refreshes the description.
Consequence: connector exposure in the prompt is decided once, at agent
build, by whether the user was signed in then. That is the intended
contract.
* refactor(tool-search): normalize_tool_call_entries lives with the other argument validation
The port appended the tool_call argument parser to the tool_search facade.
The family already has tools/tool_search_validation.py for exactly this
work (schema validation of deferred call arguments), so the parser moves
there and the facade imports it. No behaviour change; the one test that
imported it now imports from the defining module.
* refactor(connectors): delete the unused batch dispatcher; _run_remote becomes run_remote
bridge.dispatch_calls and its helpers (_dispatch_calls_inner, _run_pre_dispatch,
_run_local, _error_slot, _maybe_parse_json) and the LocalDispatch / PreDispatch
seams had no production caller. Connector dispatch runs through
model_tools_connectors: dispatch_connector_batch re-enters handle_function_call
once per entry, so scope, hook, approval and middleware policy fire against each
composed name inside core dispatch, and dispatch_connector_call hands the single
planned entry to the bridge's transport function. Only tests called the batch
dispatcher, and they exercised policy seams that production never wires.
The transport function is the module's real entry point, so it drops the
underscore: _run_remote becomes run_remote, body unchanged. The module
docstring now describes the two legs that exist (availability with D32 silent
degradation, and run_remote) instead of the injected seams. Imports that only
the deleted code used are gone; merge.py is untouched because every export
still has a caller.
Tests that drove dispatch_calls are deleted where they covered the removed
seams (pre_dispatch blocks and rewrites, local_dispatch classification, mixed
batches). The literal-slug fallback, the per-entry transport failure, and the
hook rewrite reaching the gateway request body are re-targeted at
handle_function_call('tool_call', ...) with the fake client swapped in at
bridge._default_client_factory, the same seam test_connector_dispatch_policy
uses. Each re-targeted test fails when the retry is disabled in run_remote.
* fix(connectors): search keeps the twin a colliding name reaches, and says so
format_connector_name strips the toolkit prefix, so GMAIL_FETCH_PROFILE and a
literal FETCH_PROFILE on gmail both compose to connectors__gmail__FETCH_PROFILE.
describe and execute decode that name to the prefixed slug first, so the
literal twin is unreachable under it. If a vendor ever shipped both, search
could describe the literal under a name that runs the prefixed tool.
Search is the one place that sees both twins in one response. It now keeps
the twin the name reaches and drops the other with a WARNING that names both
slugs, whichever the gateway listed first. Short names stay; no marker, no
per-process map, no change to describe or execute. No such pair exists in the
live catalog today; the guard turns a silent alias into a logged one.
search_catalog admitted every document with BM25 score > 0 and then padded
to `limit`. BM25 sums over the tokens a document shares with the query, so
on a 300-tool catalog "send gmail email" returned five incident tools that
shared only "email", and the discriminating word ("gmail", in no document)
had no say. The model read those as the answer.
Admission is now the query's rarest token: a document is a result only if it
contains the query token with the highest IDF, the one that names the intent.
Common verbs ("send", "read", "create") sit in dozens of documents and never
gate; vendor and object words ("gmail", "github", "incident") do. A token no
document carries admits nothing, and the existing empty-group hint tells the
model to retry without it. The name-substring fallback is deleted: it admitted
tools that matched no query token at all.
Result descriptions are clipped at 500 characters instead of 400. Over 353
vendor tool descriptions, 500 keeps 91% whole and every first sentence
(first-sentence max 329); 400 kept 82%.
Measured on the live 311-tool catalog with 25 hand-labelled queries:
precision@5 0.18 -> 0.43, wrong names returned 102 -> 66, false positives on
absent intents 17 -> 13. Live before/after: "send gmail email" went from five
betterstack tools to an empty group with the retry hint; "linear create issue"
and "betterstack incident" are unchanged.
claim_job_for_fire refused any claim younger than FIRE_CLAIM_TTL_SECONDS (300s)
regardless of whether its owner was still alive, so a `hermes cron run` killed
mid-flight (timeout, Ctrl-C, OOM) blocked the next manual run with "already being
fired" for up to five minutes. The executions table already reaps dead owners on
sight; the job-record fire_claim now does the same via gateway.status._pid_exists
when the claim's `by` names a pid on this host. Foreign hosts, explicit
HERMES_MACHINE_ID values, and probe failures keep the TTL (fail safe).
#106623 (ca16cafee4) blocks any test subprocess whose argv resolves to
`hermes gateway run|start|restart`, so the harness cannot spawn a runtime that
outlives the worker and restarts the developer's gateway. The matcher strips the
`-p <profile>` selector and reads through `sh -c`, so it also fired on
`docker exec -u hermes <ctr> sh -c 'hermes -p x gateway start'` and broke
tests/docker/test_profile_gateway.py on every Docker build since. A gateway
launched inside a container cannot reach the host unit or webhook port; the
guard now skips commands whose argv[0] is a container runtime (docker, podman,
nerdctl). Host-side spawns stay blocked.
_is_azure_foundry_responses matches the azure-foundry provider or the
project-scoped services.ai.azure.com host. #101243 proposes narrowing it
to that host alone. The multi-item rejection this branch handles happens
on resource-level *.openai.azure.com hosts too, so the pruning gate now
uses _is_azure_responses: the provider, or either Azure host.
Refs #105369
The first request of every turn after the first replays the encrypted
reasoning item of every earlier response. Azure Foundry accepts one such
item and rejects two or more with HTTP 400 "Conflicting authenticated
continuation identities". In-turn requests never hit this because
_is_post_tool_replay already suppresses replay on Azure when the request
ends on a tool result.
On Azure Foundry, build_kwargs now keeps codex_reasoning_items only on the
newest assistant row that has any. include still requests
reasoning.encrypted_content, compaction checkpoints stay on every row, and
the canonical messages are not mutated. Other Responses endpoints are
unchanged.
Rebuilding a failed session from state.db through the transport and
sending it live: unpatched, 5 reasoning items, 400; patched, 1 item, 200.
Fixes#105369
Azure Foundry (gpt-6-astra) answers a request that replays encrypted
reasoning from more than one prior response with HTTP 400 "Conflicting
authenticated continuation identities", code invalid_value. The classifier
mapped that to a non-retryable client error, so the turn aborted and every
later turn in the session failed the same way.
Map the message to invalid_encrypted_content. The existing recovery in
turn_recovery then strips the replay state and retries once.
Refs #105369
The Desktop settings UI persists a registry provider's API key as a
credential pointer -- model.key_env in config.yaml pointing at an env var
in $HERMES_HOME/.env (e.g. HERMES_CUSTOM_LMSTUDIO_API_KEY) -- while
keeping model.provider on the registry id. _resolve_api_key_provider_secret
resolved registry providers exclusively from
PROVIDER_REGISTRY[...].api_key_env_vars, so the UI-saved key was silently
ignored; for lmstudio the runtime then substituted the no-auth placeholder
(dummy-lm-api-key) and auth-enabled LM Studio servers rejected every
request with an opaque, retryable-flagged 401.
Fix: consult model.key_env (only when config.yaml's main model targets the
provider being resolved, so the pointer never leaks across providers)
before the registry env vars, running the value through the same
_usable_declared_secret prefix validation. Precedence for the canonical
env vars and the credential pool is unchanged; a valid canonical key still
wins when no pointer is set.
Complements #75373, which covers providers.<id>.key_env at the provider
level; the Desktop save path writes the pointer at the model level, which
that PR does not consult.
Fixes#106336
Salvage bar is at most two invariant tests per fix. The monotonic-watermark
case (small, oversized, small -> is_truncated(s, 1)) is the same contract the
oversized-frame test already pins, so it lives there as a final block instead
of a third test function.
Move the "explicit scratch means no project" decision into the one resolver
every surface funnels through, `kanban_db.create_task`: board-project
inheritance now runs only when the caller left `workspace_kind` open
(`None`), and `workspace_kind` defaults to scratch after that check. The
tool handler keeps the #106347 fix for `project=""` (no `or` collapse) and
`board=` scoping but drops its handler-local sentinel logic, since the
resolver now owns the rule; the `self_task` project inheritance for
dispatcher-owned workers is unchanged.
Sibling surfaces had the same bug through the same line and are fixed by
the same change:
- CLI `hermes kanban create --workspace scratch` on a project-scoped board
produced a project worktree; `--workspace` no longer defaults in
argparse so the resolver can tell "omitted" from "scratch".
- Dashboard `POST /tasks` with `workspace_kind: "scratch"` did the same;
`CreateTaskBody.workspace_kind` defaults to `None` for the same reason.
- `kanban_swarm.create_swarm` threads `None` through for consistency.
Tests: one resolver invariant in test_kanban_board_project.py (explicit
scratch stays scratch, omitted still inherits) and the salvaged tool test
folded into a single parametrized matrix over scoped/unscoped target boards.
Pass the caller's board into create_task and treat explicit workspace_kind=scratch or empty project as no-project so ambient board project_id cannot override MCP args.
Co-authored-by: Cursor <cursoragent@cursor.com>
The salvaged fix only ran the CLI in its own session on POSIX and kept
plain subprocess.run on Windows — the one platform where the wedge in
#106244 is actually reproducible: CPython's run() retries an unbounded
communicate() after kill() there, so a grandchild holding the capture
pipes blocks the worker forever. On POSIX run() wait()s the PID and
returns promptly; the grandchild merely leaks (live-reproduced on Linux).
One code path for both platforms:
- _group_popen_kwargs: start_new_session=True on POSIX,
CREATE_NEW_PROCESS_GROUP + hide flags on Windows (replaces the
hide-only _windows_popen_kwargs).
- _kill_cli_process_group: os.killpg SIGKILL on POSIX, taskkill /T /F
on Windows (same kwarg set as the sibling taskkill sites).
- The Popen decodes with encoding="utf-8", errors="replace" like every
other subprocess call in this file (windows footgun rule).
- Drain test patches the kill helper instead of os.killpg so it runs on
every host.
Windows behaviour is not live-verifiable on this Linux host.
Drop test_gone_group_still_surfaces_timeout: the killpg-race branch is a
contextlib.suppress and the test only pinned the exact communicate()
timeout sequence (a change-detector). The real-grandchild test and the
bounded-drain test remain — the two behaviour contracts of the fix.
subprocess.run only kills the direct CLI child on TimeoutExpired; a
browser_harness daemon / Chrome helper grandchild inherits the stdout/
stderr pipes and keeps them open, so the internal communicate() blocks
on pipe EOF forever. The wedged tool call never returns, its activity
heartbeat keeps stamping last_activity_at every 30s, and the session is
pinned at "now" in the desktop sidebar indefinitely (#106244).
On POSIX, run the CLI in its own session (start_new_session=True) and
SIGKILL the whole process group on timeout, then drain the pipes under
a bounded deadline. Windows keeps subprocess.run.
Replace the six cap-mapping unit tests (exact 5536 px value, floor, Anthropic
path, two None cases, worst-case tile math) with one behavioral test: the cap
derived from the real Codex rejection makes the production shrink pass rewrite
a real 5444x6200 PNG Responses ``input_image`` part to within the 30000-patch
budget. That is the user-visible contract; the exact pixel value is an
implementation detail. Keeps the classifier test as the second invariant.
The classifier fix routes Codex patch-budget 400s into the shrink
recovery, but _image_error_max_dimension still returned None for the
Codex wording, so the recovery fell back to the 8000 px default cap
and skipped images between ~5542 and 8000 px that already exceed the
30000-tile budget — burning the single shrink retry without shrinking
anything. Parse the reported patch limit and convert it to a per-side
pixel cap of isqrt(limit)*32 (5536 px for 30000 patches), which keeps
a square image under the budget.
OpenAI Codex Responses rejects an image whose tile-patch budget exceeds
its 30000-patch ceiling with wording ("requires N patches after
processing, exceeding the limit") that contains none of the existing
image-size vocabulary, so the 400 fell through to format_error
(non-retryable). The reactive image-shrink recovery in turn_recovery was
therefore bypassed and the session kept failing — or failover re-sent
the identical oversized image to another model.
Route the patch-budget wording to image_too_large (retryable) so the
existing shrink pass re-encodes the image under the ceiling and retries.
Fixes#106337
Drop the floor-pin test from #106290 (a snapshot of the static table); the
end-to-end merge test through provider_model_ids with the real curated floor
already fails if the slug is re-added. Note beside
_OPENCODE_FREE_KEYED_SUFFIX_MODELS why ox-alpha-free stays excluded from the
keyless catalog even though it left the curated floor.
The Go relay (GET /zen/go/v1/models) delisted ox-alpha-free 2026-09-09, but
_PROVIDER_MODELS["opencode-go"] still carried it. _profile_live_catalog merges
the curated floor into the live list (live-first for opencode-go), so the
model picker kept offering a model that now 401s — the same failure class as
#95914 (opencode-free / x-preview-f-free).
Remove it from the floor and add two regression tests: an end-to-end merge
test through provider_model_ids with the real floor (fails if a stale floor
resurrects it) and a floor-pin test asserting the known-delisted model stays
out of the offline fallback.
(cherry picked from commit 091fd85865caa09e828928f93ee985dae7e9580d)
Keep exactly the issue's Repro A (heartbeat must not become the cached answer)
and Repro B (stale READY mapping / expired button must not swallow the next
answer); both fail on origin/main and pass with the fix. The other six tests
from #106470 (config override, PENDING-tap guard, per-layer variants) were
either change-detectors or subsumed by these two.
Keep the two tests that fail on main without the fix:
- test_fal_common: a keyed managed submit makes exactly one POST (plus the
negative arm: an unkeyed submit still goes through the SDK retry ladder)
- test_image_generation: the 409 BILLING_ERROR body surfaces
`unsupported_pricing_meter` instead of the generic "not yet enabled" text
Dropped from #106484: the duplicate video-plugin billing test (same helper,
same assertion), the `_fal_client = fake` / `import_fal_client` stub churn
and the `tools.lazy_deps` stub — fal-client is installed in CI (`--extra fal`)
so those fixtures were not needed; the `_load_fal_client` no-op fixture on
TestManagedGatewayErrorTranslation for the same reason.
Also drop the redundant `retry_request is None` re-check in
`_ManagedFalSyncClient.submit` — `__init__` already raises when the helper
is missing.
Avoid retrying idempotent managed FAL submissions because the retry can mask the initial billing failure. Surface structured Nous billing diagnostics consistently for image and video paths, with hermetic regression coverage.
(cherry picked from commit 289ce039e9a522dc8016ae4a512214c05d0a8bc0)
finish_reason='length' has two causes: the answer was long (max_tokens reached),
or the prompt itself left no room to generate. _continue_text treated both the
same: append the fragment + a continuation nudge and retry, up to 4 times. In the
second case every retry sends a strictly longer prompt, so each attempt is worse
(Ollama n_ctx=32768: 32,638 -> 32,685 -> 32,732 prompt tokens, all truncated), the
user is told "model hit max output tokens", and max_tokens is not the lever.
The response's usage already carries prompt_tokens and the compressor already
resolves the model's context window; compare them once per truncation. Under
_MIN_CONTINUATION_HEADROOM (512) free tokens the turn ends on the first
truncation, keeps the partial text, names the context window as the cause and
points at /compress or a larger window. Unknown usage or window keeps today's
behaviour. max_tokens semantics untouched.
Co-authored-by: gaoanze888 <214786078+gaoanze888@users.noreply.github.com>
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Drop test_reap_idle_closes_doomed_and_keeps_attached from #106415: it
pins reap_idle's pre-existing selection behaviour (dead/idle reaped,
attached kept), which the fix does not change, and is not red on base.
Fold the duplicated two-idle-sessions + gated-close setup into one
helper shared by the two interleaving tests that do fail without the
fix (reap vs reap, close_all vs reap).
reap_idle() is reached concurrently from attach_or_spawn() and the
background run_reaper loop, and each pop is followed by an awaited
close(); while one reap is parked inside close(), a second reap reaching
the same doomed key hit dict.pop(key) without a default and raised
KeyError straight into the websocket handler. close_all() has the same
overlap window against an in-flight reap.
Pop with a default and skip keys a concurrent reap already removed, so
overlapping reaps are idempotent. Which sessions get reaped is
unchanged.
Fixes#106400
(cherry picked from commit 8ea63a845424263551487886a7526ca2f4bbd1ee)
The two facades post-#102117 (chat_completion_helpers, agent_runtime_helpers) must not grow
behaviour; the marker/predicate belong with the sibling that already owns primary-cooldown
state shared by the fallback walk and restore_primary_runtime. Also drops the two
try/except-return-False wrappers around pool.entries() and normalize_model_for_provider
(neither raises for a live pool / the never-raising normalizer).
Tests trimmed to the two invariants (#106475): the marker fires only on the single-credential
Codex entitlement 400 and the walk skips the rejected slug in either form; restore is gated on
a rejected primary and still restores an unrelated one.
A Codex ChatGPT-account 400 ('The X model is not supported when using Codex
with a ChatGPT account.') names the model, so with a single credential the
slug is dead for that account. The fallback walk still re-selected it and
restore_primary_runtime switched back to the primary at the start of every
turn, announcing an unverified 'Primary model restored' — the two warnings
alternated forever with zero delivered answers (#106475).
Record the rejected (provider, model) pair on the non-retryable client-error
path (only when no multi-credential pool exists — rotation covers that case,
#71970), skip rejected entries during the fallback walk, and gate
restore_primary_runtime on the primary's slug so the session fails closed
with the terminal entitlement error instead of oscillating. Fixes#106475.
(cherry picked from commit 471435b10288f15387b2549a8b02551d82c0f670)
The parametrized lifecycle test drove both transports through a
ClosedResourceError on the first call_tool and asserted a replay. For the
HTTP arm that is still the session-expired contract; for the stdio arm a
pipe closing after dispatch is now the ambiguous mid-call death and must
surface outcome_uncertain with exactly one invocation. The transport is
still rebuilt on both arms.
The child watcher polls every 250 ms, so in practice the MCP SDK sees the
closed pipe first and `call_tool` raises ClosedResourceError / "Connection
closed" before the watcher fires. That exception is not a _StdioChildExited,
so it fell past the stdio recoverer into _handle_session_expired_and_retry,
which reconnects and replays the call -- the exact duplicated-side-effect
path the previous two commits close. Live repro against a real stdio child
that applies an effect then exits without replying: 2 effects with the
contributor's commits alone, 1 effect + outcome_uncertain after this.
On a stdio server, a session-expired-class error raised by an RPC that was
already dispatched is re-raised as _StdioChildExited(in_flight=True) so the
stdio recoverer owns it. HTTP servers are untouched: their session-expired
retry is still the right recovery.
Tests trimmed to the salvage bar: the contributor's
test_precall_respawn_retry_dying_midcall_is_uncertain_without_replay
(a variant of the watcher-race case) is replaced by the SDK-first regression
the review on #106440 asked for; the existing pre-call retry test still pins
that a never-sent call is retried once.
The gateway's startup warm-up (_warm_turn_machinery_sync) primed the
run_agent import graph, tool schemas and context files, but left the
process-local Python toolchain probe to the first AIAgent. That agent
starts the probe worker in its constructor and its first system-prompt
build then blocks in get_environment_probe_line() on the python/pip
subprocesses (~250 ms measured) before the first model request — with
local models too, so provider metadata warm-up (#105999) does not cover it.
Call the same resolver from the warm-up: it owns the agent.environment_probe
opt-out check here, and the resolver itself owns remote-backend omission,
the single worker, its cache and the bounded fail-open wait, so the first
turn simply finds the line cached. No agent is constructed and no session
prompt is frozen at startup; a disabled probe or a remote terminal backend
leaves the host untouched.
Slim redo of #106075 by @francip: same mechanism, folded into the existing
sync warm-up instead of a new module + asyncio.gather + second suppress
block; the two behavior tests are kept, the concurrency tests dropped.
Co-authored-by: Franci Penov <49422+francip@users.noreply.github.com>
The curated fallback is the only list subscription tokens and lagged
/v1/models responses see. Assert Opus 5 and Fable 5.1 stay on that list,
newest-first, including when live discovery omits them.
The salvaged detector recorded a second MONOSPACE range from a hand-rolled
position walker and then had to rebase earlier ranges by the padding delta
(the contributor's follow-up commit). Two range bookkeepers over one text is
where the bugs were: a pipe table inside a fenced code block was realigned
AND double-styled, because the walker ran after the fences were stripped and
could no longer tell code from table.
Run table realignment on the raw markdown instead, outside fences only, and
wrap each aligned table in a fence. The existing code-block pass then emits
exactly one MONOSPACE range per table and the existing removal/adjust
machinery shifts every later range — no second offset model, no rebase loop.
The contributor's fixup commit becomes unnecessary and is not carried.
Tests trimmed to two invariants: a CJK/emoji table aligns by display width
with correct UTF-16 offsets for the range and the style after it, and a
fenced pipe table stays code while a real table before it keeps a later code
range pointed at its own text (also covers the lone-pipe negative).
Refs #106181
markdown_to_signal() passed GFM tables through untouched, so Signal's
proportional font left columns unaligned. Detect table blocks after
code-block extraction, re-align them with realign_markdown_tables(),
and cover each rendered block with a single MONOSPACE style range.
Fixes#106181
Replace the 16+4 parametrized delivery cases with the two behaviors the fix
exists for: the None tool-boundary delta releases already-streamed speech
before the tool result arrives, and completed commentary reaches TTS exactly
once. Both fail on main with the SentenceChunker TypeError. The other
permutations (transcript on/off, interim opt-out, terminal guards) exercise
paths the fix did not change.
Hermes Console ran each command on a ThreadPoolExecutor and cancelled only
the asyncio waiter. A command that forks an AIAgent (`curator run
--consolidate`) kept its worker thread and its in-flight provider request
alive after the prompt said "cancelled" — a llama.cpp generation kept
decoding for 30+ minutes and held the inference slot (#106179).
Root cause: the host owning the thread never knew about the agent created
deep inside the synchronous command, so it could not call the existing
cooperative `interrupt()` path that closes the request sockets.
Fix: `agent/interrupt_scope.py` gives the host an `InterruptScope`; the
console binds it around the worker (ContextVar), and every
`AIAgent.run_conversation()` registers itself with the bound scope for the
turn. On cancel, timeout and disconnect the console calls `scope.cancel()`
(hard-interrupts every registered agent; an agent registering after the
cancel is interrupted on entry so the cancel cannot lose the race with a
turn that has not started) and awaits the worker Future with a 10s bound
before reporting cancelled/timeout. Queued-but-unstarted work is dropped via
`Future.cancel()` alone.
Live repro (fake OpenAI-compatible provider blocking like llama.cpp, real
/api/console, real curator dispatch, real AIAgent + direct request path):
origin/main at the "cancelled" frame -> request_exited=false,
worker_exited=false; with this change -> both true, provider saw the peer
close, prompt reported cancelled 0.27s after the frame.
Salvage of #106197 by @kyssta-exe (executor-future handle, cancel/timeout/
disconnect propagation) and #106320 by @Xixiartemis (deterministic
lifecycle regression: terminal(cancelled) => no owned request or worker
remains live; interrupt-on-late-registration). Both rebuilt slimmer: #106197 keyed its fallback on Future.cancel()
returning False, but the handle it held was run_in_executor's asyncio
wrapper, whose cancel() returns True while the thread keeps running, so its
thread-name abort registry was never consulted; #106320's command-scoped
ownership model is folded into one small module hooked at the turn facade
instead of a per-caller `bind_agent`.
Co-authored-by: kyssta-exe <218078013+kyssta-exe@users.noreply.github.com>
Co-authored-by: Xixiartemis <182932319+Xixiartemis@users.noreply.github.com>
An explicit X-Hermes-Session-Id client re-sending a pre-rotation id read
the closed parent's stale history slice, bound the turn and the wake
target to the closed parent, and — once the detached delivery writer
adopted the live tip — could no longer consume a late delegation delivery
row persisted on the tip, while still being granted wake_capable="1".
Resolve the client-addressed id through the same canonical
resolve_resume_session_id resolution the delivery writer (gateway/wake.py)
and /v1/runs use, before loading history: the read, the turn, and the wake
binding all select the live continuation tip. Response headers (SSE and
JSON) keep echoing the client-provided id, so the external identity stays
stable across rotations regardless of when a turn rotates relative to
prepared response headers.
Compression-tip read/write routing and the stable-echo identity contract
originate from #13437 (manuelschipper), ported onto the post-refactor
route owner with real-SessionDB regressions (JSON and streaming).
Reported-and-investigated-by: shojikumaru (Sho + Alpha) via #98619
Co-authored-by: manuelschipper <11737902+manuelschipper@users.noreply.github.com>
#98619 review follow-up: an explicit session_id plus non-empty
caller-supplied conversation_history skipped the SessionDB load yet was
still marked wake-capable ('not previous_response_id' alone). Such a
turn is authoritative on the caller snapshot and never consumes the
SessionDB delivery row a detached completion persists, so wake is now
fixed before the history load can overwrite the variable and denied on
the same default-deny contract as previous_response_id continuations.
Adds an HTTP regression proving the binder receives wake_capable=""
and _conversation_history_for_session stays uncalled on that branch.
Address the compression-rotation blocker from the follow-up review of #98619:
The parent run can compress/rotate before a detached delegate completes.
The captured origin session id is then a closed parent: the delivery
append is rejected with CompressionSessionClosedError forever (the
watcher retries the same stale id), and no read surface exposes the row.
- persist_delegation_delivery now resolves the live continuation tip via
the canonical resolve_resume_session_id (the same resolution
/api/sessions/{id}/messages reads use) before appending, failing open
to the original id.
- /v1/runs adopts the resolved tip for a client-addressed session id
(body id or declared key): history loads from it, the turn and wake
binding target it, so the run no longer writes into the closed parent.
End-to-end regression: parent compresses, detached delivery persists to
the stale id, and the next same-id run loads the delivery row from the
live tip and binds it.
Address the two /v1/runs blockers from re-review of #98619:
- P1: resolve the session identity (body id > response chain > declared
X-Hermes-Session-Key > run_id fallback) BEFORE loading history, so a
header-only declared-key run now loads its selected session's SessionDB
history. A persisted async-delegation delivery row therefore reaches the
next same-key run's context, which is what the wake opt-in promises.
- P1: a previous_response_id continuation consumes its ResponseStore
snapshot and can never see a SessionDB delivery row (async completion
persists to SessionDB only), so it is no longer granted wake_capable;
delegate_task keeps its synchronous fallback on that branch until a
merge contract exists for the chain.
Regressions: declared-key history load (delivery row consumed), no-row
declared key stays fresh, response-chain continuation denied wake with
snapshot history preserved, plain run still granted wake.
Rebuilt against the post-refactor owners: the chat-completions route now
lives in gateway/platforms/api_server_openai_routes.py, the wake gate in
tools/delegate_tool_dispatch.py, and session_context.py was reshaped —
the original patch aimed at code main no longer has.
Header-less OpenAI-compatible clients get a fingerprint-derived session
id bound as the api_server chat_id. delegate_task's background gate
treated ANY bound session id as wake-capable and dispatched detached
subagents, but for derived ids the wake self-post lands in a session
whose history the client never reloads — the result is undeliverable by
construction.
Bind a wake_capable provenance flag at session-bind time, DEFAULT-DENY
at the central boundary (set_session_vars and _bind_api_server_session
both treat an omitted declaration as denied; a binder that says nothing
grants no wake authority): "1" only from audited producers whose client
can address the id again (explicit X-Hermes-Session-Id — 403-gated on
API_SERVER_KEY — native /api/sessions/{id} routes, /v1/runs); "" for
fingerprint-derived ids. The delegate gate requires the flag (fail-closed,
captured pre-child-construction alongside origin_wake_sid) and keeps the
forced-sync fallback with its honest note.
Reported-and-investigated-by: shojikumaru (Sho + Alpha) via #98619
Siddharth Balyan