_to_async_client rebuilds default_headers from scratch and dropped the blank
Authorization set by _create_openai_client, so every async aux call (the path
aux tasks actually take) for a free-tier OpenCode model still shipped the
keyless placeholder as a bearer and 401'd. Re-apply the same keyless policy on
the async twin.
Review finding: fourth client builder (_to_async_client) missed the keyless header policy.
A free slug (deepseek-v4-flash-free) re-resolved under the paid `opencode` profile — via
`/model --provider opencode`, a fallback switch, or a config default — resolves to the
keyless placeholder, but the primary client builder blanked the Authorization header only
when `agent.provider == "opencode-free"`. The placeholder then went on the wire as a
bearer, the Zen relay 401'd every request, and the credential pool (size 0) had nothing to
rotate: the loop reported in #110831.
Key the header policy on the placeholder as well as the provider, matching agent_init and
auxiliary_client which already do so.
Stored tool call ids are minted per turn (terminal:0, terminal:1…), so the
same id recurs on later turns of one session. The Responses adapter replayed
each pair verbatim, putting the same call_id on the wire N times; strict
validators (opencode Console, OpenAI) reject the request with 400
"Duplicate function_call_output for call_id" and every retry fails the same
way (#102629, #111231 bug 1).
Every occurrence past the first now gets a `_dup<n>` wire id and the matching
tool output pops the id its function_call was given, in call order, so pairs
stay intact and stored history is untouched.
Re-applied onto the facade split (helpers moved into _replay_tool_call_items /
_tool_output_items) from PR #102634.
With no fallback_chain (or an exhausted one) the ladder tail returned the
_RERAISE_ORIGINAL sentinel and call_llm's bare `raise` re-raised the ORIGINAL
error. After a 401 that the refresh rung already healed, the retry's actionable
"404 requires available credits" was hidden behind "401 Unauthorized", steering
the user to re-login instead of top-up. Raise the narrowed `first_err` from the
ladder tail instead and drop the sentinel; the pool-rotation rung shares the
same tail so it is covered by the same change.
Review finding: exhausted no-chain path re-raised the healed 401 instead of the retry's 404 credits error.
Review feedback on the fall-through fix: the accept boundary and the
retry-fails-with-a-connection-error path through the credential rung were
reachable but untested, and two comments described behavior the code does
not have. Tests and comments only, no behavior change.
Both auth-refresh rungs in the auxiliary recovery ladder performed their
post-refresh retry with a bare `yield` inside a `return` statement, so an
exception from that retry escaped `_aux_recovery_ladder` instead of being
converted into `(None, err)` by `_rung()`. The ladder therefore never reached
`_ladder_provider_fallback` and the configured
`auxiliary.<task>.fallback_chain` was silently skipped.
Observable case: a stale Nous runtime token makes the auxiliary request return
401; the ladder refreshes the credential and retries; when the retried model is
out of credits the resulting 404 escapes the ladder and the task fails (context
compression degrades to truncation) even though a healthy fallback_chain is
configured. With a fresh token the same 404 takes the payment path, which
already falls through correctly.
Guard both rungs with `_rung()` - the helper the sibling payment rung and the
credential-pool rung below already use - so a failed retry resumes the ladder.
_is_structured_output_rejection matched several phrasings for a provider refusing the
structured-output field, but not gateways that validate the body with a strict pydantic
model and reject the OBJECT-form json_schema by shape:
HTTP 422: {"detail":[{"loc":["body","response_format","json_schema"],
"msg":"str type expected","type":"type_error.str"}]}
422 already passes the status check; only the phrasing was missing, so the
one-retry-without-response_format rung never engaged and the auxiliary task failed
hard (title generation left 'HTTP 422' in the session title). Treat the shape error as
a rejection: the field is what the provider refuses and the remedy is identical.
Fixes#110631
_is_unsupported_parameter_error matched "does not support" but not the contraction Bedrock
Converse returns for xAI Grok ("This model doesn't support the temperature field"), nor the
inference-profile Claude wording ("`temperature` is deprecated for this model"), so the aux
ladder's retry-without-temperature rung never fired and the ValidationException surfaced on
title/vision/compression calls (#111043, second half).
Bedrock-hosted xAI Grok (us./global. inference-profile prefixes) rejects
temperature/topP in Converse with a hard 400 — the same restriction as
Claude Opus 4.6+, but _forbids_sampling_params is Claude-only, so Grok
needs its own denylist gate in build_converse_kwargs.
Fixes#111043
The bridge wrote GATEWAY_ALLOW_ALL_USERS into os.environ only when unset and
never cleared it. In-process restart paths (gateway restart watcher, dashboard
profile actions) copy os.environ into the child, so a config.yaml grant became
a sticky env var: flipping allow_all_users to false and restarting left the
gateway OPEN. The bridge now tracks its own write (module flag), overwrites or
clears it on reload, exports only a truthy grant (presence-based readers such
as the Telegram intake prefilter treated "false" as configured auth), and the
two restart env builders drop the bridge-owned value so the child re-derives
the posture from its own config.yaml. Under multiplex_profiles the DEFAULT
profile's events are authorized inside its secret scope, where gate readers
never fall to os.environ; the bridged grant is now seeded into that profile's
scope mapping only (secondaries never inherit it).
Review finding: bridged GATEWAY_ALLOW_ALL_USERS survives restart and overrides a flipped config.yaml; inert for the default profile under multiplex; "false" exported as configured auth.
carry_unadmitted_user_message appends the interrupted turn's user row to the
early result's in-memory history only; it is never persisted because that turn
never owned the lease. When the follow-up turn also has to wait for the lease
(the common case: the other process that made the first turn wait is usually
still busy), admit_durable_turn_lease reloads conversation_history from the DB
after admission and replaced the caller's list wholesale, so the tagged row was
dropped from both the model input and state.db. Re-append the tagged rows that
have no _row_id after the reload so the follow-up turn sees and flushes them.
Review finding: waited-reload branch of admit_durable_turn_lease discarded the
_persist_after_admission_interrupt row carried from the aborted turn.
Move the pre-admission carry-forward out of run_conversation (facade) into
agent/turn_facade_lease.py::carry_unadmitted_user_message next to the early
result it repairs, and drop the extra turn-start flush in build_turn_context:
the follow-up turn's normal turn-start persist already writes the marked row
because _db_flush_collect no longer stamps it as durable (live probe: exactly
one A row in state.db after two flushes). Trim to two invariant tests
(carry-forward with metadata; flushed exactly once); the hard-stop negative is
covered by the E2E probe in the PR body.
Three paths still resolved the raw model/remote-supplied string before the
guard could refuse it, so on Windows the NTLM-leak trigger (resolving the
path) ran anyway: the file-checkpoint helper stats write_file/patch targets
before the tool executes; the ACP file bridge resolves fs/read_text_file and
fs/write_text_file paths before its read/write denylists; and @file:/@folder:
references resolve their target before the reference allow-check. Each now
checks the raw string first and refuses. The GLOBALROOT form now requires
its path separator so a GLOBALROOT-prefixed local name is not misclassified.
The rationale comment names the vector instead of another product's
changelog, and the security docs say the row is enforced on reads as well
as writes, since it sits under the write-guard table.
Claude Code v2.1.234 (Aug 17, 2026) hardened its pre-approval file
accesses to reject Windows NT-namespace (\??\) paths against the NTLM
credential-leak vector. Port the same guard into Hermes file safety:
- agent/file_safety.py: is_nt_namespace_path() / get_nt_namespace_error()
raw-string check (never resolves — resolving IS the leak trigger).
Wired as the first check in get_read_block_error() and the write
denial classifier.
- tools/file_tools.py: raw-string guard at read_file_tool entry and in
_check_sensitive_path (covers write_file_tool + patch_tool), before
the task-base join can anchor the prefix under a POSIX base dir.
- Blocks \??\, \\.\, \\?\UNC\, \\?\GLOBALROOT. Extended-length
local drive paths (\\?\C:\...) and plain UNC shares stay allowed.
- tests/agent/test_nt_namespace_guard.py: 10 blocked forms, 11 allowed
forms, no-resolve proof, tool-layer chokepoint coverage.
- docs: protected-paths table in user-guide/security.md
Why: the rebase conflict resolution in tests/tools/test_model_tools.py
deleted the unrelated TestBridgeDispatch class (3 tests from 73163e3);
it is restored verbatim from main with TestBrowserRetrievalHints after it.
The fix had five tests for one invariant: the OPENAI_MODEL_EXECUTION_GUIDANCE
check duplicated test_phantom_tool_references, and the two static-schema
"toolset-neutral" checks are now folded into test_silent_without_web_tools,
which runs _apply_dynamic_schemas over the real browser_navigate/browser_cdp
schemas so the rendered descriptions are what is asserted.
execution_guidance_text() no longer takes valid_tool_names: the guidance is
toolset-neutral, so the parameter was ignored; the single caller in
agent/system_prompt.py and its test are updated.
The rebased guidance text no longer names web_search anywhere, so
execution_guidance_text()'s replace() calls (3733e4aff5) matched
nothing and were dead; the function now returns the neutral text for
every toolset and its phantom-tool test asserts "no web tool named"
instead of the removed sentence. model_tools ports the PR's hint layer
into main's _DYNAMIC_SCHEMA_REWRITERS table (browser_navigate +
browser_cdp) rather than a second pass after it.
Tests: the two browser_cdp registry tests were re-added by the PR but
main pruned them in 39975613b13b4; replaced with one schema-neutrality
invariant. Exact-wording assertions ("lightweight retrieval tool",
"appropriate permitted retrieval/search tool") were change detectors and
are dropped. tools-reference.md row updated to the new schema text.
Squashed integration of the user-facing message audit for this surface set.
Full per-finding receipts: /tmp/ux-audit/lanes/*-receipt.md (campaign artifacts).
INLINE_TOOL_EXECUTORS["session_search"] is the production dispatch on every surface
(tool_executor + agent_runtime_helpers) and maps schema args to kwargs explicitly, so the
schema advertised after/before/exclude_session_ids while the executor silently dropped them
and every time-bounded call ran unbounded. Add the three mappings.
Also drop the stranded `as_exclusive_end` parameter on _parse_iso_bound (accepted, never
read; exclusivity is the SQL `<` predicate) and the Python re-check of the time window over
FTS rows in _discover — _search_filter_clauses already bounds sessions.started_at on every
route, so the only place the window still needs a Python check is the title-match branch,
which bypasses that query.
The guide said a proxy root like http://localhost:4000/gemini "works the same"
as spelling out /v1beta, but the chat/aux clients only take the native Gemini
adapter when is_native_gemini_base_url() matches the
generativelanguage.googleapis.com host; normalize_gemini_base_url() applies to
the Google host, TTS and the tier probe. Reword the docs to those cases and
tell proxy users to configure an OpenAI-compatible URL. Also note in the
normalize_gemini_base_url docstring that only the last path segment is
inspected and that it does not decide routing.
A GEMINI_BASE_URL (or tts.gemini.base_url / providers.gemini base_url) set
to a host root — https://generativelanguage.googleapis.com or a proxy root
like http://localhost:4000/gemini — produced native requests to
{base}/models/{model}:generateContent with no API version segment, a
guaranteed 404. Google's own google-genai client treats the base URL as a
host root and appends the version itself, so users reasonably configure it
that way.
normalize_gemini_base_url() appends /v1beta unless the URL already ends
with a version segment (v1, v1beta, v1alpha, ...). Applied at every native
request builder: GeminiNativeClient, probe_gemini_tier, Gemini TTS
(tts_tool.py), and streaming TTS (tts_streaming.py). /openai-suffixed
URLs are untouched (OpenAI-compat path).
Port of cline/cline#13329, which fixed the same bug class after their
ai-sdk migration.
close() escalated to kill_process_tree() but never imported it; the NameError
was swallowed by contextlib.suppress, so on the TimeoutExpired path neither the
kill nor the post-kill wait ran and a root codex ignoring SIGTERM leaked (a
regression vs the previous self._proc.kill()). Import it from agent.deadline
and add a test forcing the timeout path that asserts the tree kill and the
follow-up wait both run. Also drop the upstream product reference from the
test docstring (credit stays in the PR body) and pass encoding= to the PID
file reads flagged by the Windows footgun scanner.
Port from openclaw/openclaw#126285: snapshot Codex app-server descendants before root retirement and sweep the proven process identities after close so independently grouped stdio MCP children cannot survive client shutdown.
The profile scope (HERMES_HOME override, secret scope, terminal policy) is a
contextvar bundle bound per turn. A bare threading.Thread / Timer / gRPC
callback starts with an empty context and resolves the LAUNCH profile:
- agent/title_generator.py: the auto-title thread read
auxiliary.title_generation (model, language, provider key) from the default
profile's config and billed the default's key for a secondary's session.
Spawn via agent.memory_provider.spawn_context_thread (copy_context).
- tui_gateway/session_lifecycle.py: every teardown caller is a bare Timer
(ws-orphan reap), the idle-reaper thread, atexit _shutdown_sessions, the
session.close pool RPC, superseded_by_resume or compute_host flush - none
carries a scope, yet on_session_end / commit_memory_session / agent.close ->
shutdown_memory_provider read the provider's config + credentials at call
time. Under multiplex they failed closed (tail never committed, #110622
class); on the Desktop backend a secondary's transcript went to the launch
profile's memory tenant. _finalize_session and _teardown_session now bind
_session_profile_runtime_scope(session) around those blocks, which covers
every spawn site through the single chokepoint.
- plugins/platforms/google_chat/adapter.py: Pub/Sub callbacks run on the gRPC
SubscriberClient's threads and run_coroutine_threadsafe copies THAT empty
context onto the loop task, so _dispatch_message and everything under it
(attachment cache, per-user OAuth token store via _acquire_user_chat_api ->
_load_per_user_chat_api, TTS keys, delivery ledger, bot-id cache) resolved
the launch profile. connect() captures its scope; _on_pubsub_message and
_submit_on_loop run under a per-callback copy of it.
spawn_context_thread gains a kwargs passthrough for the title thread's
callbacks.
Under gateway.multiplex_profiles (and the Desktop/dashboard backend serving named
profiles) os.environ holds the LAUNCH profile's .env. Five spawn sites built a
child's env from it while acting for another profile, so the child saw the
launch profile's HERMES_HOME (bot_relay, key_cmd), its credentials, HERMES_MODEL
and TERMINAL_* policy, and none of the served profile's own .env:
- tui_gateway/server.py _SlashWorker: pinned HERMES_HOME but kept the launch
base with tier-2 credentials + settings.
- tools/bot_relay.py delivery_env (relay RPC + --run-delivery): dict(os.environ).
- tools/browser_tool.py _build_browser_env: re-added BROWSERBASE/FIRECRAWL/
BROWSER_USE keys from os.environ after the scrub.
- plugins/platforms/a2a/adapter.py _forward_to_profile: {**os.environ}.
- agent/command_token_source.py _mint: key_cmd helper inherited os.environ.
tools.environments.local.served_profile_child_env is the one builder: pin the
target home, drop the launch profile's .env residue and bridged TERMINAL_*
(strip_launch_profile_env), and for children that legitimately run with the
profile's credentials (agent worker, token helper) overlay the target profile's
own secrets - what a standalone `hermes -p X` loads itself, never a sibling's.
The browser keeps the provider scrub and re-adds only its passthrough keys via
get_secret. Outside multiplex the env is unchanged.
Live proof from inside the child (launch A, served B, multiplex on): all five
children print HERMES_HOME == B, see B_MARKER=b from B's .env and do not see
A_MARKER; the browser child gets B's FIRECRAWL_API_KEY. On base every one leaked
A_MARKER and lacked B_MARKER; bot_relay and key_cmd also had A's HERMES_HOME.
HERMES_DELEGATED_CHILD_CONTEXT=1 is deliberately carried into every shell/
execute_code subprocess a delegate_task child spawns (the fence must survive
exec so a grandchild `hermes kanban complete` cannot promote itself). But the
readers treated the bare flag as "fence every Kanban DB": kanban_db_connect
opened ANY board ?mode=ro and write_txn refused ANY mutation. A subagent
running a Kanban reproduction against a scratch HERMES_HOME therefore got a
silently read-only board with a misleading "descendants require an
initialized board" error; only one lane in the retrospective ever discovered
why (deleg_15dac332), every earlier kanban repro ran degraded.
The marker's value is now the fenced board ROOT (kanban_home() at spawn) and
readers deny only paths under that root or the dispatcher-pinned
HERMES_KANBAN_DB (kanban_path_is_fenced). In-process children and a legacy
"1" marker still fence everything; an inherited path marker is never
re-derived, so a grandchild that moved HERMES_HOME cannot unfence the real
board. Owner-gate tests (test_kanban_descendant_scope, cron env isolation,
kanban CLI exit status) are unchanged and green.
3fad83df31 (Aug 11) moved Relay exporter config to a plugins.toml
selected by HERMES_NEMO_RELAY_PLUGINS_TOML. A .env still carrying the legacy
exporter vars and no TOML logs ONE warning and initialises no exporters, so
users who followed the earlier docs lost every trace silently (the
maintainer's stopped Aug 20, noticed Sep 14; five multiplexed profiles on
the same box carry the same eight vars today).
- `hermes_cli/relay_plugin_migrate.py`: build the document from the
`nemo_relay.observability` dataclasses (`ComponentSpec(...).to_dict()`,
so the `type = "file"` sink discriminator is emitted), validate it by
activating it through `nemo_relay.plugin.initialize` + `clear_async`,
write `<home>/relay-plugins.toml` (tomli_w when installed, minimal emitter
otherwise), set HERMES_NEMO_RELAY_PLUGINS_TOML in that .env, and comment
the legacy lines out (never delete). Defaults mirror the removed plugin so
files land where they used to.
- `hermes update` runs it for the default home AND every live named profile
(each writes its own TOML) as a best-effort post-update step, with a loud
notice; `hermes migrate relay [--all-profiles] [--no-validate]` runs it on
demand.
- The runtime WARNING and the `hermes doctor` finding now say "NO traces
are being exported" and name the exact command and file path.
- Docs: environment-variables.md + built-in-plugins.md carry the migration
note and a complete plugins.toml example including `type = "file"`.
- codex_runtime._CODEX_PROGRESS_DELTA_TYPES gains response.refusal.delta so the
stream watchdog sees progress on a refusal-only stream instead of timing it
out as idle.
- auxiliary_client._parse_codex_final_response reads type=refusal content
parts; without it an aux refusal-only turn parsed to content=None and hit the
empty-response path the main loop was just taught to avoid.
- tests: parametrize test_streamed_refusal_accumulated (refusal-only /
alongside-content) so there is one test per surface; drop upstream product
references from docstrings (credit stays in the PR body); pass encoding= to
the read_text calls flagged by the Windows footgun scanner.
- docs: fallback-providers notes that a streamed refusal is a terminal
content_filter result, not an empty response to retry.
A model that declines mid-stream delivers the explanation on the
structured refusal channel (chat_completions delta.refusal; Responses
response.refusal.delta / refusal content parts) and leaves content
empty. The streaming accumulators dropped that channel entirely, so a
streamed refusal assembled into an empty message and fell into the
empty/invalid-response retry loops - burning paid retries reproducing a
deterministic refusal - while the non-streaming path had already fixed
this class in #46013.
- chat_completions streaming: accumulate delta.refusal (incl.
model_extra), expose message.refusal on the assembled mock response so
ChatCompletionsTransport.normalize_response applies the existing
sole-payload -> content_filter promotion; count refusal deltas in the
zero-chunk guard; carry refusal in the Relay final-response dict.
- Codex Responses stream consumer: collect response.refusal.delta as
answer text so a refusal-only stream no longer raises 'did not emit a
terminal response' with zero usable content.
- Responses normalizer: read type=refusal content parts in
_extract_responses_message_text (attr and dict shapes).
Sabotage-verified: each new test fails with its wiring line disabled.
E2E: refusal-only stream -> terminal content_filter with explanation;
refusal-alongside-content stays a normal usable turn; plain-text
streams unchanged.
Why: the tui_gateway live formatter (`_format_live_context_output`, used when
the session runs on a compute host) renders its own summary and never got the
"Context files" block, and `session.context_breakdown` had no structured rows,
so Desktop's popover could not show them. The formatter now appends
render_context_file_lines() with the session cwd bound (the RPC thread has no
session context, so the discovery walk would key on the backend's cwd), and
the RPC payload gains a `context_files` list (contract + generated TS/OpenRPC
+ Desktop type). The docs sentence is scoped to the surfaces that render it.
A file whose content _scan_context_content replaces with a BLOCKED marker was
reported "loaded"; the manifest now runs the same scan and reports `blocked`.
The module docstring names the frontmatter-strip / chain-cap approximations
and drops the product-name attribution (credit stays in the PR body).
Review follow-up (Enough1122) on the salvaged #91272: the original
list_context_file_sources() hand-mirrored the priority ladder inside
build_context_files_prompt, so the two would drift the moment the builder
gained a context type or changed precedence — misreporting what the prompt
holds is worse than not showing it.
Now prompt_builder exposes one candidate finder per context type
(_CONTEXT_FILE_CANDIDATES → discover_context_files) and BOTH the loaders and
the manifest walk it. The manifest lives in the new sibling
agent/context_file_sources.py (not appended to the facade) and:
- reports empty / unreadable files truthfully instead of "✓ 0 tokens",
- mirrors the install-tree guard ("suppressed") so a Desktop session that
fell back into the Hermes tree sees why nothing loaded,
- lists every .cursor/rules/*.mdc as loaded, matching the builder which
concatenates all of them,
- measures truncation on the rendered "## label" section like the builder.
The block now renders on every surface that shows the /context category
table: CLI/TUI (hermes_cli/cli_info_mixin.py) and the messaging gateway
(gateway/slash_commands_status.py). The Desktop popover consumes the raw
session.context_breakdown payload (no text table) and is left as-is.
Tests trimmed to the two invariants: manifest/prompt parity across every
context type at once, and truncated/suppressed follow the builder.
Why: build_auto_load_prompt read config via ambient load_config_readonly()
and looked skills up under the ambient SKILLS_DIR. Gateway bot threads lose
the HERMES_HOME ContextVar, so a bot profile's pinned skills came from the
launch profile — the docs promise profile scoping. _auto_load_parts now
passes home_override=_agent_home(agent) and build_auto_load_prompt binds it
for config, disabled-list and <home>/skills lookup, the same seam
_skills_prompt uses via skills_dir_override.
_auto_load_parts was unconditional and injected pinned SKILL.md bytes into
delegate children, curator/background_review forks and gateway hygiene
agents; it now mirrors _skills_prompt's gate (nothing without the skills
toolset) and returns [] when skip_context_files is set.
cli.py's HERMES_IGNORE_RULES check used == "1" while system_prompt used
is_truthy_value; both use is_truthy_value now.
Tests stay at 4: the build test asserts the home-scoped resolution, the
ignore-rules test also covers the subagent / no-skills-toolset gates.
The system prompt must stay byte-stable for the life of a conversation:
`_auto_load_skills_result` is seeded in `_SESSION_STATE` and filled on
the FIRST prompt build only (HERMES_IGNORE_RULES captured then too), so
model switches, compression and static-prefix restoration reuse the
exact rendered bytes rather than re-reading config or skill files.
CLI: auto_load renders in the existing background `--skills` preload
thread (real session id for ${HERMES_SESSION_ID}), `-s` names dedupe
against the auto-loaded canonical names via
`build_preloaded_skills_prompt(excluded_loaded_names=)`, the activated
skills line shows auto_load first, and the lazily built agent is seeded
with the pre-resolved bytes. `--ignore-rules` skips auto-load with the
rest of the auto-injected context.
Re-implementation of #74060 by @ctaylor86 against current main.
`skills.auto_load: [name, ...]` in config.yaml renders the listed skills
as fully loaded skill blocks in the system prompt of every new agent —
CLI, TUI, gateway, cron and API — the persistent counterpart of `-s`.
Missing or operator-disabled names are warned about and skipped; a
config typo never blocks session start.
Re-implementation of #26840 by @ArcherQAQ (via #74060) against current
main — the original patch targeted the pre-decomposition cli.py /
system_prompt.py god files; design and diagnosis preserved. The loader
reuses `_load_skill_blocks` (same disabled gate and Curator usage bump
as `-s`) instead of a parallel loop.
Slot arguments are always complete json.dumps output (Gemini re-sends full args), so the mid-stream JSON check could never fire; the id key needs no tool name; _new_call_id and the slot lookup now share _provider_call_id.
Two different calls to the same tool arriving in separate stream events
collided in one accumulator slot: Gemini 2.5 sends no call id and
part_index restarts at 0 per event, so the second call's arguments were
emitted as a delta on the first call's index and concatenated downstream
into unparseable JSON, dropping a call. Gemini 3 ids are now the slot
identity (part_index and thought signature drift across events of one
call); without an id, a call whose arguments are not a continuation or
resend of the slot's accumulated JSON opens its own slot, kept reachable
as key#N so a later resend lands on it.
Re-applied by hand onto the collapsed translate_stream_event on main from
#75528 (9371874010 + f4c8863cdc). #24676 by cdbartholomew (May 13) was
the first fix for this collision (value-based slot matching without the
id key) and is credited as co-author.
Co-authored-by: Chris Bartholomew <chris.bartholomew@vectorize.io>
The stale-attempt socket shutdown re-implemented two blocks that already
live in agent_runtime_helpers: the settimeout(0)+shutdown(SHUT_RDWR)
body of force_close_tcp_sockets (now _shutdown_socket) and the
candidate->socket lookup of _iter_pool_sockets (now _socket_from_candidate).
The hand-unrolled _httpcore_stream unwrapping is dead since
_connection_candidates walks _stream/_httpcore_stream itself, so the helper
starts from the network_stream extension and the response stream only.
Also add ReadError to _TRANSIENT_TRANSPORT_ERRORS (the third classifier of
the same abort-induced read; the other two were already updated), drop the
incidental gettimeout() assertion from the shape test, and cut the E2E
from ~3 s to ~1.5 s (stale budget 1 s, serve_forever poll 50 ms).
Keep the real httpx 0.28 wrapper-shape test (proves the shutdown reaches
the socket through BoundSyncStream/ResponseStream/PoolByteStream) and the
loopback E2E (a parked reader unwinds within its stale budget and the
retry lands). The other four were narrower restatements of the same paths.
Also treat httpx.ReadError as a transport error in codex_runtime: it is the
same abort-induced-read class the streaming retry loop now recovers from.
Capping ``read`` to ``stale`` silently overrode an explicit
HERMES_STREAM_READ_TIMEOUT and the local-endpoint ``read = base`` branch,
and let the stale-kill E2E pass via ReadTimeout alone rather than through
the socket-shutdown unwedge this change is about. The shutdown path is
sufficient on its own: the E2E still passes with the cap removed.
The stale-stream monitor aborted a wedged provider stream only via
force_close_tcp_sockets() -> shutdown(SHUT_RDWR). That is best-effort: a
parked body read is not unblocked on every platform (Windows keeps the
pending recv parked) and the sweep can miss the socket. The worker then
stayed blocked in the provider read, so the retry loop never retried; the
monitor re-killed every stale interval and the call only ended at the
byte-read timeout, far past the stale budget - the reported
"No response from provider for 180-240s ... Reconnecting" loop ending in
"The model server is not responding".
- _kill_stale_stream now also closes the killed attempt's own provider
response (identity-guarded self._attempt_stream_response), which is what
actually unblocks a parked reader; a racing retry's fresh response is
never touched.
- an abort-induced httpx.ReadError counts as a transient connection error,
so the aborted attempt reconnects instead of ending the turn.
Reproduced with a local SSE server: before, the worker stayed parked and no
second request was issued (recovery only at the byte-read timeout); after,
the kill unblocks the reader at the stale budget and the retry lands.
Answers the P1 review on #111187: build_profile_secret_scope() held only
<profile>/.env plus that profile's external-source snapshot, never the
administrator-managed .env. The launch process applies that file LAST with
override (_apply_managed_env), so a managed key beats the user's own value in
os.environ. Under multiplex semantics get_secret() stops falling back to
os.environ on a scope miss, so inside a routed cron fire (and equally inside a
real multiplex gateway turn, which builds its scope through the same function
via gateway/run.py::_load_profile_secret_scope) a managed-only credential
resolved as absent and a managed-vs-user collision resolved to the USER value:
reversed precedence.
Fix at the source: build_profile_secret_scope() overlays load_managed_env()
last, after the profile .env and external sources, skipping process-global
names exactly as it does for the other two layers. Every multiplex-authoritative
scope (gateway turn, routed desktop fire, external worker env build) is built
here, so managed authority is composed once instead of restored per consumer.
No generic ambient-env fallback is reintroduced: only the managed file's own
keys enter the scope, and only with the managed file's values.
Regression (parametrized, two invariants): inside a routed fire a managed-only
key resolves through get_secret(); a managed-vs-user collision yields the
managed value.
_hydrate_profile_secret_sources and _apply_external_secret_sources each
rebuilt the same "applied + skipped_existing" set and pushed it into
_SOURCE_SUPPLIED_NAMES; the routed-child scrub depends on both sites
agreeing, so give them one helper.
refresh_installed_secret_scope cleared the installed scope before
refilling it, which left a window where a concurrent reader of the same
fire saw no credentials at all. Update first, then pop the names the
rebuild no longer supplies; stale values still disappear.
Two credential-isolation gaps found in review of the previous head.
1. strip_launch_profile_env() only knows dotenv- and terminal-config-owned names,
but external secret sources (vault, 1Password, ...) also write their names into
the shared os.environ and are tracked in secret_source_names(). A name the LAUNCH
profile's source supplied therefore still reached a routed no_agent child. Drop
every non-global source-owned name from the base; the routed scope overlay that
follows puts back exactly the ones that profile's OWN sources supply, since
build_profile_secret_scope folds get_secret_source_values(home) in.
2. refresh_installed_secret_scope() merged the rebuild with dict.update(), so a
name a source had stopped supplying -- rotated, revoked, source removed -- kept
its old value for the rest of the fire. Replace the mapping contents instead: the
rebuild is the profile's current truth.
Regressions: a routed child sees <unset> for a launch-source name while its own
source value comes through, and a refresh whose rebuild omits a name drops it.
Both fail if the corresponding change is reverted.
(cherry picked from commit ecd51517c4828a75acb5f458ed99aea0bc3e5e9f)
The desktop backend ticks EVERY local profile's cron store from one process — its own docstring
says "like a multiplex gateway" (hermes_cli/web_server.py) — but never sets the process-global
multiplex flag, and cannot: its own chat turns are unscoped and would fail closed. Every
isolation in the tree keys on that flag — the guard that keeps a routed `.env` out of the shared
`os.environ`, `get_secret`'s fail-closed miss, passthrough resolution, the MCP and kanban
subprocess scrubs — so all of it was inert for a sibling profile's fire. Verified: a secondary
profile's API keys replaced the launch profile's in `os.environ` with `override=True` and stayed
there after the tick, and a scope miss read the launch profile's tokens (#107692).
Give multiplex mode a context-local counterpart. `set_multiplex_context` (agent/secret_scope.py)
is OR'd into `is_multiplex_active()`. `_profile_cron_scope` only MARKS a fire whose home is not
the process's own (`routed_profile_fire`, decided against `get_process_hermes_home()`, the
override-immune resolver); `_install_fire_secret_scope` in cron/scheduler.py installs the
profile's hydrated secret scope and, for a marked fire, the multiplex context — for exactly that
span, dropped again before the scope by `_reset_fire_secret_scope`. Multiplex semantics are
therefore never active in cron without a scope to read: `run_one_job`'s restart-safe handoff runs
before the body's scope and keeps today's semantics (its own scope is #107413 / #106050's seam,
left untouched so this composes with whichever lands). Every existing multiplex-keyed isolation
applies inside the routed fire with no per-site patching; the launch profile's own fires and the
backend's turns keep single-profile semantics; marker and override both reach the pool worker via
`copy_context()`. `get_secret` read the raw global in its miss branch; it now goes through
`is_multiplex_active()`. The dotenv guard keeps its pinned flag-only form (#77970).
Two consequences of suppressing the write are handled rather than left as regressions:
- a `no_agent` script's env is `os.environ.copy()`, which no longer carries the routed `.env`;
the runner overlays the installed scope onto the base BEFORE sanitizing, so the same scrub /
passthrough rules apply to those values and the parent process is never mutated;
- plugin secret sources are discovered on the fire's first agent build, after the scope froze,
and the post-discovery reload is hydrate-only under multiplex semantics; the refresh now folds
the values into the installed scope in place (`refresh_installed_secret_scope`, the pattern
`_publish_env_value` already uses for `.env` writes under multiplex).
And the profile's external secret sources are hydrated before the scope is frozen, the order
gateway/run.py and the external cron worker already use.
Tests pin each direction: the marker without the semantics before the scope, the semantics on and
off exactly with it, the marker reaching a copy_context worker; the process's own profile staying
single-profile; the restart-safe handoff's child env building without raising under a routed tick
with a passthrough key registered; a real child process receiving the routed values while
`os.environ` keeps the launch value; a source registered after the freeze reaching the fire
through the real PluginManager refresh. Reverting any one direction fails a distinct test.
(cherry picked from commit 2f87677425d2cca19286ac83bc45cab23e546669)
`_CodexCompletionsAdapter._build_responses_kwargs` already asks
`classify_responses_route` for the GitHub flag but hand-rolled its own
x.ai host match for `is_xai`. Two classifiers for one route drift; use
`route.is_xai_responses` so the aux path agrees with the main transport.
`is_copilot` (used for `is_github_responses` replay stripping) is left
as-is.
`_classify_responses_issuer` reimplemented endpoint canonicalisation with
its own urlsplit/urlunsplit pass. The repo already owns that logic in
`hermes_cli/route_identity.py::normalize_route_base_url` (stdlib-only,
used by agent/backend_identity.py), so delegate to it.
Reasoning items persisted before canonicalisation were stamped with the
raw `other:<agent.base_url>` (trailing slash, host case). Comparing them
verbatim against the now-canonical `current_issuer_kind` marked them
foreign and dropped them on the very endpoint that minted them. Run the
persisted stamp through the same canonicaliser (`_canonical_issuer_kind`,
non-`other:` kinds untouched) before comparing.
Also fixes the `_chat_messages_to_responses_input` docstring, which still
stated the pre-stack rule that legacy endpoint-stamped items drop when the
current model is known; the stack replays them on a matching issuer.
The openai SDK appends a trailing slash to `client.base_url`, so the aux
adapter stamped `other:https://h/v1/` while the main transport stamped
`other:https://h/v1`. On custom Responses endpoints every aux call
(compression, flush_memories) therefore dropped all main-minted reasoning
items as "foreign".
`_classify_responses_issuer` now strips whitespace and trailing slashes and
lowercases scheme+netloc before stamping. The aux adapter also derives its
route flags from `classify_responses_route` — the single owner of the
codex/xai/github predicates — instead of an inline chatgpt.com host check,
and reuses the same flags for the effort clamp.