Extends the picker fix to the read that actually uses the key: switch_model's
user-provider credential resolution (the ${VAR} api_key expansion and the
key_env fallback at the resolve-credentials step) still read os.environ raw
and passed the result to resolve_runtime_provider as explicit_api_key — so
under multiplex_profiles the actual switch, not just the picker listing,
could adopt another profile's key. Same _scoped_key_env helper, same
fail-closed semantics; identical behavior when multiplexing is off.
Adds end-to-end switch_model tests pinning that an installed scope wins over
the process environment for both read shapes.
854007d1c routed the remaining main-agent fallback key reads through
agent.secret_scope so the multiplexed gateway's per-profile scope applies.
list_authenticated_providers - which gateway/slash_commands.py calls
directly for /model - still resolved custom-endpoint and fallback-entry
credentials with raw os.environ.get(key_env), so under multiplex_profiles
one profile's picker reads whatever key the process environment happens to
hold, i.e. another profile's.
no multiplexing : profileA-key (unchanged)
scope installed : profileB-key (was profileA-key)
Route both reads through a _scoped_key_env() helper over
secret_scope.get_secret(). get_secret is identical to os.getenv when
multiplexing is off, so single-profile deployments are byte-for-byte
unchanged; a fail-closed UnscopedSecretError is treated as "no credential
visible for this profile", which is how the picker already handles a
missing key.
Scope: only the two key_env credential reads. The other environment reads
in that function are provider-presence probes (AWS creds, LM_BASE_URL),
a separate concern.
Per-turn .env adoption could rewrite agent.api_key while leaving
_credential_pool_entry_id on a previously rotated fallback. The next 429
then marked the healthy fallback exhausted via credential_id precedence
(#79156).
- Sync pool entry id after a successful env credential refresh
- First look does not stomp a pool-rotated key with the env primary
- mark_exhausted_and_rotate prefers api_key_hint when it disagrees with
credential_id
Fixes#79156
_detect_venv_python_processes() returned cmdline_raw[:120]. Gateways
autostarted via the managed-runtime interpreter carry a >120-char exe path
(.hermes-runtime\python\generation-...\cpython-3.11-...), so the truncated
cmdline ended inside the exe path, before '-m hermes_cli.main gateway run'.
The Desktop preflight's pausable-gateway exemption
(_scan_venv_blockers._is_pausable_gateway) therefore never matched, the
gateway was reported as a blocker, and every Desktop update aborted with
'Update didn't finish' even with all windows closed — the updater's own
gateway pause never got a chance to run.
Fix: return the full cmdline from the detector and truncate only at
display time (_format_venv_python_holders_message and the scan's JSON
cmdline field, after redaction).
Reproduced live on Windows 11: scan reported blocked=true for
'...cpython-3.1' (truncated); after the fix the same gateway pair scans
clear with pausable_gateways=2.
An alias that family-matches multiple catalog models (/model opus) used to
silently pick one via _model_sort_key heuristics. The heuristics have
guessed wrong repeatedly — dated snapshots like claude-opus-4-20250514
parsed as version 20,250,514 and outranked claude-opus-4-8; suffix
tiebreaks landed on the cheapest tier — and every wrong guess silently
switches the user to a model they did not ask for.
resolve_alias now raises AmbiguousAliasError whenever more than one model
matches the alias family; switch_model catches it at all three call sites
(explicit-provider path, current-provider path, authenticated-provider
fallback) and returns a failure result listing the candidates
(best-guess-first ordering, capped at 10) with instructions to pick an
exact name. A single match still resolves automatically, and DIRECT_ALIASES
exact mappings are unaffected.
The date-stamp split from #67571 is kept, demoted from selection logic to
display ordering of the candidate list.
Supersedes the auto-pick approach of #67571; credit to @Sahaun and @GottZ
for the date-stamp parser analysis that this builds on.
_model_sort_key treated YYYYMMDD snapshot stamps (e.g.
claude-opus-4-20250514) as version components, so 20250514 > 8
and resolve_alias("opus", "anthropic") returned the wrong model.
Fix: split components ≥ 19_000_101 (smallest plausible date stamp)
out of the version tuple, keeping them as a trailing tiebreaker so
bare IDs sort before their dated snapshots and newer snapshots
before older ones. Shorter numeric components (mistral-large-2411,
gpt-4-0613) keep their current behavior. No models.dev dependency
in the sort path.
_resource_attributes() in otlp_exporter.py built its own hardcoded
resource dict (service.name/instance.id/telemetry.scope only) instead
of reusing gateway_health_export.py's _runtime_resource_attributes(),
which already applies the resource_attributes allowlist from config.
Result: operator-configured attributes like deployment.environment.name
reached metrics and diagnostic logs but never spans.
Span resource building now delegates to the same
_runtime_resource_attributes() helper metrics/logs already use,
removing the duplicate implementation instead of patching it in place.
Generic thinking fields (reasoning / reasoning_content + the
reasoning_details text charge) are replayed for at most the NEWEST
assistant turn on every transport: Anthropic strips all-but-newest at
convert time, Bedrock Converse never replays thinking, and strict
chat-completions providers reject or one-space-pad the field. The tail
budget walks charged them on every message anyway, spending 19-24% of
the budget (per the issue's 1,025-message measurement) on bytes that
provably never reach the wire — so the tail cut landed early and each
compaction discarded more real transcript than configured.
_estimate_msg_budget_tokens now partitions the replay keys:
* _ALWAYS_REPLAYED_BUDGET_KEYS (codex_reasoning_items,
codex_message_items) — charged unconditionally. These ride the wire
on every retained turn (#55572), and codex_reasoning_items now also
carries native server-side compaction checkpoints (#81747).
* _NEWEST_TURN_ONLY_BUDGET_KEYS (reasoning, reasoning_content) + the
reasoning_details text charge — charged only for the newest assistant
turn via charge_stale_thinking, resolved by the three budget walks
(tail cut, raw-budget re-walk, proactive-prune boundary).
Default stays the conservative full charge for callers without
turn-position context. A partition invariant test pins that any future
_REPLAY_BUDGET_KEYS entry must be classified into exactly one class.
Direction credit: #73669 (@x7peeps) and #73730 (@webtecnica) both
attacked this; the keep_open reviews asked for provider/API-mode-aware
accounting that keeps Codex carriers charged — this implements that
shape.
get_messages() only deserializes content and tool_calls; the structured
reasoning columns (reasoning_details, codex_reasoning_items,
codex_message_items) come back as the raw TEXT they were stored as.
Feeding those rows straight back into a write, which is exactly what
the POST /api/sessions/{id}/fork handler does by piping get_messages()
into replace_messages(), hit an unguarded json.dumps() and stored the
already-serialized string encoded a second time. On replay of the fork,
json.loads() then yields the inner string instead of a list, and every
consumer's isinstance(..., list) gate silently drops it: preserved
Anthropic thinking blocks, Codex encrypted-reasoning/message-item
replay, and OpenRouter multi-turn reasoning context are all lost after
a fork, with one more encoding layer added per fork.
The /branch copy loop had the same defect from the other side: it
forwarded reasoning but none of the structured columns, and both TUI
branch writers persisted role/content alone, dropping reasoning and
reasoning_content along with them.
Route the six dumps sites in append_message and _insert_message_rows
through a shared guard that keeps already-serialized strings as-is;
structured values from the live runtime are dumped exactly as before.
Forward the reasoning fields in all three branch writers, matching the
set gateway/slash_commands.py already forwards on its own /branch path.
Apply the same metadata invariant to sealed split chunks, keep expect_edits on live previews, and exercise the real Telegram adapter path with rich messages enabled but rich drafts disabled.\n\nCredits PR #78525 by @Slobaka for the reproduced rich_messages/rich_drafts combination.
A finalized native draft is the first persistent send and will not be edited again. Omitting expect_edits lets Telegram use sendRichMessage for the persistent final instead of degrading tables through MarkdownV2.\n\nAdapted from PR #46536.
New tests/skills/test_authoring_standards.py parametrizes every bundled +
optional SKILL.md (1148 checks) against the mechanically-verifiable subset
of the hardline standards:
- required frontmatter fields (name/description/version/author/license/
platforms) + tags
- frontmatter name == directory name
- description <= 60 chars, ends with period, no marketing words
- related_skills resolve in-repo
- no machine-local paths
- <= 100k chars
Grandfather dict for legacy debt ships EMPTY — all pre-existing violations
fixed in this PR:
- 13 frontmatter names canonicalized to their directory names (the install
identifier); all related_skills references updated (comfyui -> stable-
diffusion). Fixes the class behind PR #42788's report; also fixes
here.now's invalid dot-name.
- optional-skills/devops/cli -> inference-sh-cli (dir was the generic
'cli'; fm name was right) incl. docs pages (en + zh-Hans), catalog row,
sidebar entry.
- pytorch-fsdp: 157k generated 'Quick Reference' dump moved to
references/common-patterns.md; SKILL.md 159k -> 2.5k with a pointer.
- research-paper-writing: 31.7k Phase 5 drafting section moved to
references/phase5-paper-drafting.md; SKILL.md 103k -> 71k.
Docs regenerated with scope discipline.
Titling fired on the first response, so a session sat unnamed for the whole
opening turn - p50 151s, p90 1212s across real sessions, because a turn is
tool calls, not one round-trip. A turn that failed or was interrupted never
got a title at all. Four surfaces each carried their own copy of the call.
Move it into the shared turn prologue and split it in two: a deterministic
title derived from the user's opening message, written inline before the
model runs, then one small-model call that upgrades it. The response is
constrained to a JSON object so there is no preamble to strip, and control
wrappers are stripped rather than refused, so a slash command titles as
what the user asked for instead of the command itself.
Titling ran on the user's main chat model, so a five-word title was billed
to a frontier reasoning model and inherited its latency. Pinning a cheap
model id instead just moves the problem: the hardcoded default was already
dead upstream and every call paid a 404 before the retry net caught it.
Match model FAMILIES against the provider's live /v1/models catalog,
preferring rolling '-latest' aliases where a provider publishes them, and
order the families by measured latency. Nothing to bump when a provider
ships a new mini/flash/haiku. Opt-in per task, so compression, vision, and
search keep 'auto means my chat model'.
A session title had no notion of who set it, so two bugs followed. An
auto-generated title could clobber a name the user typed, and every
compression rotation renumbered the conversation it forked - one piece of
work reaching 'Smallville Map Architecture Plan #10' in the sidebar.
Titles now carry a source (derived < llm < user) enforced by one
compare-and-swap, so an automatic write can only ever replace a title of
strictly lower authority. Compression carries the name across unchanged.
Legacy NULL rows rank as user, so auto-titling only fills genuinely
empty titles on existing data.
Per review: upscaling should be the default behavior (like the original
flux-2-pro chain), not agent opt-in. Policy: every image model whose
native output is below ~2MP now sets upscale=True in its catalog —
users never silently get low-res images. Native hi-res models
(Seedream 5 Pro/Lite, Krea 2 Large) stay off to avoid paying to
upscale already-large output.
- FAL catalog: 16 models flipped to upscale=True (klein, z-image,
nano-banana pro/2/2-lite, gpt-image 1.5/2, ideogram v3/v4, recraft
v4/v4.1, qwen image/3, krea-2 medium on FAL, MAI 2.5 pro).
- Krea plugin: per-model upscale defaults (medium + medium-turbo ON at
1.5K native; large OFF at 2K native), precedence explicit kwarg >
image_gen.krea.upscale config > catalog default.
- The 'upscale' tool param remains as a per-call override in both
directions (false = fast draft, true = force on hi-res/edits).
- Video unchanged: opt-in only (default-on would double every video's
cost and latency).
- Sibling tests updated: routing/payload tests pass upscale=False where
the assertion targets the generation submit; catalog test now pins
the native-resolution policy instead of the flux-2-pro snapshot.
The generated-media surface previously had almost no upscaler coverage:
only fal-ai/flux-2-pro chained Clarity Upscaler (hardcoded catalog
default), every other image model returned ~1MP output with no high-res
path, and video had no upscaler at all. Krea's API treats the enhancer
as a standard second pass; this brings the same shape to Hermes.
- image_generate: new optional 'upscale' boolean in the tool schema.
Explicit true chains the backend upscaler on ANY model (including
edits); explicit false disables flux-2-pro's automatic default;
omitted keeps per-model catalog behavior. Response now reports
'upscaled' so the agent knows which resolution it got.
- FAL image path: explicit flag overrides the catalog 'upscale' default
(Clarity Upscaler, 2x). Failure falls back to the native image.
- Krea plugin: upscale=true chains Krea Enhance
(/generate/enhance/krea/enhance, 2x, prompt-guided) through the same
BYO/managed base URL + auth as generation, with a best-effort poll
loop that never fails a successful generation.
- video_generate: new optional 'upscale' boolean; FAL video plugin
chains ByteDance SeedVR2 (fal-ai/seedvr/upscale/video, 2x factor
mode). Providers without upscalers ignore the kwarg per the ABC
contract (documented in both ABCs).
Validation: targeted suites green (123 tests across 6 files, including
new coverage for override-wins/default-kept/failure-fallback on all
three paths); live E2E on direct FAL verified both chains end-to-end
(klein 9b + Clarity upscaled image; pixverse-v6 1s 360p + SeedVR2
upscaled video).
Two corrections on top of the #71077 base (the whole bug class):
1. Turn boundary = last USER message, not last assistant message. A Codex
turn spans several assistant messages (assistant+tool_calls -> tool ->
... -> final assistant) whose reasoning items must replay together; the
last-assistant boundary would strip reasoning mid-chain from the active
turn (the gap flagged in PR #71077 review).
2. type="compaction" checkpoints (native server-side compaction, PR #81747)
are exempt: they carry already-pruned history, not per-turn reasoning.
Pruning filters items instead of popping the sidecar key.
Sibling site fixed in the same class: the Codex incomplete-continuation
dedup path blind-overwrote codex_reasoning_items on visually-duplicate
interim messages, which would drop the only copy of a checkpoint captured
on the earlier response. Extracted merge_interim_reasoning_items() into
agent/native_compaction.py; newer reasoning wins, prior checkpoints are
preserved unless the newer payload carries its own.
In HUD mode Hermes is a strip over the app the user is actually working
in, so "what's under you?" or "look up the weather" is almost always
about that app — but the agent had no way to know it was floating, and
answered from its own browser and panes instead.
The desktop tags a HUD submit with `surface: 'hud'` and the gateway turns
that into a per-turn note pointing at read_window_below, and at carrying
the work out in the app underneath. It rides the model-bound message
beside the reaction and speech-interrupted notes rather than the system
prompt: one session can be driven from the app window on one turn and the
HUD on the next, and the system prompt has to stay byte-stable.
Every tool the note names is checked against the agent's own schema
first, so a session without computer_use or read_window_below is never
pointed at a tool it cannot call.
* fix(model-picker): serve cached custom-provider catalog on no-probe opens
#58183 stopped GUI picker opens from live-probing saved custom
OpenAI-compatible endpoints so a stopped local server could not stall the
picker. It gated the whole discovery block, not just the network call, so
`cached_fetch_api_models()` was skipped too — and with it the catalog an
earlier probe had already written to `provider_models_cache.json`.
A custom endpoint that is not the current provider therefore renders only
the models named in its config entry. A local server with 8 models loaded
shows the 1 model that was saved when the provider was first added, on
every picker open, while an explicit Refresh shows all 8.
Add `cache_only` to `cached_fetch_api_models()`: answer from disk within
the existing stale-serve window, never fetch, never revalidate off-thread,
return None on a miss. Split the three call sites in
`list_authenticated_providers()` into what the user's config permits
(`discover_models`, an explicit `models:` allowlist) and how we may obtain
it, so suppressing the probe now downgrades to a cached read instead of
skipping discovery outright. `discover_models: false` still pins, and a
cache hit no longer writes back to config since the probe that populated
it already did.
The latency win stands: a cold cache is a miss, so picker opens against
offline endpoints still make zero network calls.
* test(model-picker): pin the cached-catalog contract for no-probe opens
Cover both halves of the invariant, since fixing either one alone
reintroduces a bug the other guards against.
`cache_only` on `cached_fetch_api_models()`: a fresh entry and an entry
past its TTL but inside the stale-serve window both serve; an entry beyond
that window, an empty cache, rotated credentials, `force_refresh`, and a
missing base_url are all misses — and none of them fetch or spawn a
background revalidation.
`list_authenticated_providers()` on the GUI path: a non-current endpoint
with a warm cache reports its full catalog across all three provider
shapes (`custom_providers`, `providers:`, bare `provider: custom`) with no
live fetch attempted. A cold cache keeps the configured list and still
makes no network call, which is the #58183 guarantee. `discover_models:
false` keeps pinning, and a cache hit does not write back to config.
* fix: persist discovered custom-provider models in the hermes model flow
The `hermes model` named-custom-provider flow (_model_flow_named_custom)
probes the endpoint and shows the full catalog, but never persists it to the
entry's `models:` list. No-probe surfaces (dashboard, desktop, ACP) call
build_models_payload(..., probe_custom_providers=False) and only render the
configured `models:` list, so a provider added via `hermes model` collapses
to the single `model:` default everywhere except the CLI. OpenAI-compatible
providers added via a probing picker already benefit from
_save_discovered_models_to_config; the CLI flow did not.
Persist the live catalog after a successful probe, mirroring the picker path
in model_switch.py. A failed save is non-fatal.
* fix(model-picker): stop an auto-saved catalog pinning a keyless endpoint
The cached-catalog read added for no-probe picker opens still sat behind
the no-key discovery gate, so it never reached the shape that motivated
it: a keyless local model server.
`bool(api_key) or not has_explicit_models` is a network-cost gate. It
exists so Hermes does not probe an endpoint it cannot authenticate to
when that endpoint already declares its catalog (5f00f36ba, 1039e90b5).
Reading a catalog an earlier probe already paid for costs nothing, so
the gate belongs on the probe, not on discovery as a whole.
Left on the discovery side it re-pins the endpoint it was meant to
spare. A successful probe calls `_save_discovered_models_to_config()`,
which writes a plain list into `models:` — exactly the shape
`_models_config_is_allowlist()` reads back as an explicit user
allowlist. A keyless server therefore froze on the catalog of its first
probe and could never widen again, which is the "lineup changes after
config was written" case. f66319097 already carved the dict shape out of
this trap for the same reason; the list shape is the other door into it.
Move the clause to `_probe_live` at both custom-endpoint sites. Probe
suppression is unchanged — verified byte-identical to main across the
keyed/keyless x declared/undeclared matrix — and `discover_models: false`
remains the documented way to pin a catalog.
* test(model-picker): cover the keyless auto-save pinning trap
Three tests around the gate move, each failing on the code before it:
- a keyless endpoint carrying an auto-saved `models:` list still reads
its full cached catalog
- the same row, cold cache and probing enabled, still makes zero live
fetches — the network-cost gate the clause exists for
- an end-to-end round trip: persist a probe result via
`_save_discovered_models_to_config()`, reload it, and assert the shape
we wrote does not read back as a user pin
The round-trip test guards the whole chain rather than one branch, so a
future change that makes the saved shape look like an intentional
allowlist fails here even if the gate logic is refactored.
* fix(model-picker): key the custom-endpoint model cache by api_mode
`cached_fetch_api_models()` fingerprints entries with `api_mode`, but no
call site in `list_authenticated_providers()` passed it, so every custom
row resolved to the `api_mode=None` fingerprint. Two rows sharing a
base_url and credential but differing by `api_mode` are deliberately
distinct picker rows — it is part of `group_key` at both sites — yet they
collapsed onto one cache entry.
That was latent while probing was the only way to fill a row: a mismatched
entry was overwritten by the row's own live fetch. Serving that entry
without a probe makes it visible, so an `anthropic_messages` row could
render the catalog an OpenAI-mode row cached against the same URL. The
wire protocols differ (`x-api-key` + `anthropic-version` vs
`Authorization: Bearer`), so those catalogs are not interchangeable.
Persist `api_mode` on the group at both grouping sites — it is already
part of `group_key`, so it is constant across the group — and pass it
into the cache read. Section 3b (bare `provider: custom`) has no
`api_mode` in scope and already reads with the empty-credential
fingerprint, so it is unchanged.
Reported by Copilot review on #81973.
---------
Co-authored-by: xxxigm <tuancanhnguyen706@gmail.com>
Co-authored-by: Navlem <114683850+Navlem@users.noreply.github.com>
A gate runs whatever command the operator configured, so its output is
arbitrary bytes. run_gate captured it with text=True and no encoding, which
decodes with locale.getpreferredencoding() under errors="strict".
One byte the decoder rejects — a test runner's checkmarks or CJK on a
non-UTF-8 Windows console, a stray binary byte anywhere in the stream — kills
subprocess's reader thread. proc.stdout comes back None, the `or ""` fallback
turns that into an empty tail, and an unhandled traceback is dumped to stderr.
The gate's pass/fail verdict still lands on the exit code, but the output tail
is exactly what the retry prompt feeds back so the agent can fix the failure.
With it empty the agent is told a gate failed and given nothing to act on, so
it burns every retry and the goal auto-pauses.
workspace_fingerprint has the same two calls; there a non-ASCII path in
`git status --porcelain` empties the fingerprint, silently disabling the
unchanged-gate skip that exists to stop a stalled agent re-running the same
red suite.
Decode as UTF-8 with errors="replace" — what git and modern toolchains emit,
and what 262 of the repo's 299 text-mode subprocess calls already do.
_run() used text=True without an encoding, so Windows decoded git's
UTF-8 output with the locale code page (cp932) and raised
UnicodeDecodeError on non-ASCII filenames or diff content, breaking
the "Never raises on git failure" contract in its docstring. Match
the utf-8 + errors="replace" policy checkpoint_manager's _run_git
already uses. Legacy cp932-encoded blob content degrades to
replacement characters instead of crashing; a test pins that
trade-off so it stays a documented choice.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes the read_file half of #80308 and the class behind #80261, #80250,
The binary sniff sampled files via 'head -c 1000' through the terminal
transport, which decodes stdout with errors="replace". A multibyte
character cut at byte 1000 therefore arrived as U+FFFD, and
_is_likely_binary treated any U+FFFD as binary — flagging valid CJK and
emoji text as unreadable. At the text layer a stored replacement char
and a transport-manufactured one are indistinguishable, which is why
per-callsite adjustments kept leaving siblings open.
Sample as 'head -c 1000 | base64' so raw bytes survive the transport
(fail-open to the legacy heuristic when the transport cannot produce
clean base64), then classify bytes: NUL => binary; valid UTF-8 allowing
one incomplete multibyte sequence at the sample end => text; mid-stream
invalid UTF-8 (latin-1, true binaries) => read-only, preserving the
anti-mojibake guarantee the old check existed for. Files legitimately
containing U+FFFD become readable.
test_cli_entrypoint_end_to_end copies add_contributor.py with
read_text()/write_text() and no encoding argument, so both fall back to
the system locale. add_contributor.py contains UTF-8 multi-byte
characters (an em dash), which makes the read raise UnicodeDecodeError
on any non-UTF-8 Windows locale (observed on cp950 / Traditional
Chinese). The trailing mapping-file read gets the same treatment for
symmetry.
Same footgun class as the subprocess text=True sweep in #60741, just on
the pathlib read_text/write_text side.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Two Windows bugs in scripts/run_tests_parallel.py:
- --files/--paths/HERMES_TEST_PATHS were split on ':', which shreds
absolute Windows paths at the drive letter ('C:\repo\tests' ->
['C', '\repo\tests']): the drive letter became a phantom discovery
root and the rooted remainder only resolved by WindowsPath
re-anchoring it onto repo_root's drive. New _split_pathspec() keeps
drive-letter colons glued to their path and accepts ';' (os.pathsep)
on Windows, while ':'-joined lists (CI generate job) keep working.
- With piped stdout (CI, subprocess capture) Windows encodes the
runner's output as the ANSI code page, so printing the per-file
progress glyphs raised UnicodeEncodeError inside the executor
done-callback and every progress line was silently lost -- which is
also why test_bare_value_flag_keeps_its_value failed on win32 (no
'1[check]' line, and the summary says '1 tests passed', which does not
contain '1 passed'). The runner now reconfigures its own
stdout/stderr to UTF-8 on Windows, and the tests decode the captured
output as UTF-8.
Adds regression tests: os.pathsep-joined absolute roots (all
platforms) and no-phantom-drive-root (win32).
Fixes#57149
`tests/hermes_cli/test_plugins_cmd.py::TestNoAutoActivation::test_compressor_default_ignores_plugin`
fails on every Windows machine:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in
position 47744: character maps to <undefined>
The test reads `run_agent.py` back as text to assert a removed comment is
gone, but called `open()` with no `encoding=`. Python then falls back to
the locale preferred encoding, which is cp1252 on a default Windows
install rather than UTF-8. `run_agent.py` contains nine bytes cp1252
leaves undefined, so the read raises before the assertion is reached. On
Linux and macOS the preferred encoding is UTF-8 and the same line is
fine, which is why CI never caught it.
That one line is the only active failure. The rest of this change closes
the same gap in the files it touches, which `scripts/check-windows-footguns.py`
flags and which the #71014 read_text campaign has been working through
elsewhere in the tree:
- `tests/hermes_cli/test_plugins_cmd.py`: nine bare `write_text`/`read_text`
calls writing YAML manifests, config and plugin sources
- `tests/tools/test_web_tools_truncate.py`: reads stored extracted web text,
which is arbitrary content from the internet
- `tests/stress/test_atypical_scenarios.py`: writes and reads worker task
ids and a barrier file
All three files are now clean under `check-windows-footguns.py`.
Reads go through `Path.read_text(encoding="utf-8")` rather than
`open(...).read()`, which also closes the handle instead of leaving it to
the garbage collector. On Windows a live handle blocks tmpdir cleanup, so
that part is not cosmetic either.
No new test. The repaired test is the regression coverage: it fails
before this change and passes after, on Windows.
Complements the cherry-picked contributor fixes and closes out the
remaining sites of the 'missing explicit encoding' bug class, which is
now permanently gated by ruff PLW1514 (enabled repo-wide in
pyproject.toml and enforced by the blocking `ruff check .` step in
.github/workflows/lint.yml):
- tools/memory_tool.py: read MEMORY.md/USER.md via utf-8-sig so a
Notepad BOM never glues U+FEFF onto the first entry (issue #10878,
PR #10888 by @easyvibecoding — strict-decode contract of
_read_raw_checked preserved rather than errors="replace", so
undecodable files still refuse read-modify-write instead of being
lossily rewritten). Regression tests included.
- tools/skills_tool.py: SKILL.md and skill file reads pinned to
utf-8-sig + errors="replace" — deterministic across platforms instead
of the locale fallback proposed in PR #51701 (superseded: falling back
to cp1252/GBK makes the same skill render differently per host); .env
reader aligned with the canonical utf-8-sig dialect in hermes_cli/config.py.
- agent/shell_hooks.py, hermes_cli/main.py, gateway/slash_commands.py:
explicit utf-8 on the remaining fdopen/open text-mode sites flagged by
the AlexFucuson9 sweep series (#56033#56940#65565#66782#66791).
Co-authored-by: easyvibecoding <easyvibecoding@users.noreply.github.com>
Co-authored-by: AlexFucuson9 <AlexFucuson9@users.noreply.github.com>
Co-authored-by: flyingdoubleg <wangzhe00zju@gmail.com>
Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com>
_remove_env_source() decides whether a credential var lives in ~/.hermes/.env
or the shell by scanning the .env with env_path.read_text(errors="replace") —
no encoding. read_text() with no encoding falls back to the system locale
(cp1252/GBK on Windows) and never strips a BOM.
The canonical .env readers in hermes_cli/config.py all use
encoding="utf-8-sig" precisely because 'users may edit .env in Notepad which
adds one' (a BOM), and doctor.py documents that .env is written as UTF-8
everywhere. This sibling reader diverged: on a Notepad-edited .env the BOM
prefixes the first line, so line.strip().startswith(f"{env_var}=") is False
for the first variable — the detector reports a .env-backed key as a phantom
shell export and prints a misleading 'still set in your shell environment'
hint on .
Match the canonical reader (utf-8-sig + errors=replace). Adds a regression
test with a BOM'd .env.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review asked for a BOM-prefixed JSON case alongside the existing
control-character coverage. The sandbox script now also feeds
json_parse a \ufeff-prefixed document and asserts the parsed value
round-trips (fails against the pre-fix helper, passes with the
BOM strip).
tools/managed_tool_gateway._read_nous_provider_state read auth.json with a
bare read_text(), which on Windows decodes as cp1252 and raises on any
non-ASCII byte (e.g. an accented Nous provider label). The broad except
swallowed it and returned None, so the gateway treated Nous as
unconfigured — the same Windows UTF-8 hazard the other auth.json readers
in this PR already fix.
Add encoding="utf-8-sig" (consistent with the sibling readers) plus a
non-ASCII regression test reusing the windows_default_encoding fixture.
This covers the one auth.json reader in #66782 not already handled here
(tools/managed_tool_gateway.py:40); the other two readers #66782 touches
(agent/auxiliary_client.py, tools/xai_http.py) are already fixed in this
PR. RED-verified.
Address review feedback on #58158: the regression suite covered four of
the changed readers but not _read_shared_nous_state (auth.py) or
_has_any_provider_configured (main.py), which also read UTF-8 stores the
Windows cp1252 default can corrupt.
Add a non-ASCII UTF-8 regression case for each, reusing the existing
windows_default_encoding fixture and _write_utf8 helper:
- _read_shared_nous_state: a nous_auth.json with an accented display_name
and valid tokens must round-trip intact (not return None). Pins
HERMES_SHARED_AUTH_DIR to tmp to satisfy the shared-store seat belt.
- _has_any_provider_configured: an auth.json whose active provider carries
a CJK label must still report a configured provider (the read must not
raise into the swallowing except). get_auth_status is faked so the
result is driven by the read, and provider env vars are cleared to reach
the auth.json branch.
Both tests are RED-verified — they fail when the respective
read_text(encoding=...) is reverted.
Follow-up to the auth.json UTF-8 read fix in this PR. A repo-wide scan for
the same bug class found three more callers that read ~/.hermes/auth.json
via Path.read_text() with no encoding — same Windows cp1252 hazard:
- agent/auxiliary_client.py _read_nous_auth: a non-ASCII byte raised
UnicodeDecodeError, the broad except swallowed it, and Nous silently
stopped being available as the auxiliary (vision/summarization) provider.
- tools/xai_http.py has_xai_credentials: same failure mode — xAI OAuth
silently looked absent on Windows.
- hermes_cli/main.py is_setup_complete: same; has a config.yaml fallback so
the impact is milder, but the read is still wrong.
All three now use read_text(encoding="utf-8-sig"), matching _save_auth_store's
write encoding. A repo-wide grep confirms there are no remaining
json.loads(...read_text()) reads of auth.json without an explicit encoding.
Tests: rewrote the Windows-encoding regression tests to actually exercise the
bug on POSIX too — a new windows_default_encoding fixture forces a no-encoding
read_text() to decode as cp1252 (the Windows default), and _write_utf8 now
emits real non-ASCII UTF-8 bytes (ensure_ascii=False) so the bytes actually
trip cp1252. Verified each test fails when its fix is reverted (including
the two new sibling-reader tests).
The auth store readers (_load_auth_store, _import_codex_cli_tokens, and the
shared Nous store reader) called Path.read_text() with no encoding, so bytes
were decoded with locale.getpreferredencoding() — cp1252 on Windows. The
stores are *written* as UTF-8 (os.fdopen(..., encoding="utf-8")), so any
non-ASCII byte (a CJK or emoji credential label, an accented display name in
OAuth state) raised UnicodeDecodeError on read.
Worst case: _load_auth_store's broad except then copied the file to .corrupt
and returned an empty store, silently wiping every provider credential on the
next launch. The sibling reader at line 2161 already used
read_text(encoding="utf-8"), confirming the omission was unintentional.
Use utf-8-sig (matching the .env handling in config.py) so a BOM from a
Notepad-edited file is tolerated too.
Adds regression tests covering the UTF-8 round-trip with a non-ASCII label,
BOM tolerance, no-corrupt-on-valid-load, and that the readers pass an explicit
encoding (guard against future regressions). Verified the tests fail when the
fix is reverted.
Closes no issue — found via cross-platform code audit (the bug is not in the
issue tracker).
send_cmd._load_hermes_env intentionally reimplements a minimal dotenv
load (no secret-source pulls, no sanitize rewrite, get_hermes_home path
resolution incl. Windows/profile override), so the shared-loader BOM fix
is mirrored in place: utf-8-sig primary read, BOM strip before the
latin-1 stream fallback.
Claude-Session: https://claude.ai/code/session_01JPmJz5u1Bvtw4cCRvRWnYr
utf-8-sig only covers the primary decode. BOM + invalid UTF-8 (e.g.
PowerShell BOM + cp1252 body) forced latin-1, which kept EF BB BF as
part of the first key name and dropped the canonical name. Strip the
BOM before latin-1 decode and load via stream so override= is preserved.
PowerShell 5.1 Set-Content -Encoding UTF8 and Windows Notepad write a
UTF-8 BOM. load_dotenv(encoding="utf-8") kept U+FEFF on the first key
name, so the canonical name was absent from os.environ and Hermes looked
unconfigured with no error. utf-8-sig strips the BOM and is a no-op for
BOM-less UTF-8; latin-1 fallback unchanged.
Own the surrogate-crash class at three chokepoints instead of leaf sites:
- finalize_turn scrubs final_response once where model text leaves the
conversation loop — covers oneshot stdout (#80366), NIM/any-provider
responses (#19819), and every delivery consumer of the turn result.
- _sanitize_gateway_final_response scrubs at the gateway chat-surface
boundary — Telegram utf16_len (#55309) and Signal formatting (#55143)
can no longer see a lone surrogate; raw-text surfaces keep passthrough.
- run_conversation walks the fully-built api_kwargs with
_sanitize_structure_surrogates so tool descriptions (session_search,
#50959) and every other request-body leaf are JSON-encodable before
any provider sees them.
Regression tests pin all three chokepoints plus helper semantics.
Cherry-picked alongside #79240 (TheophilusChinomona) and #80374
(rainbowgore) whose commits precede this one with authorship preserved.
Prevent UnicodeEncodeError when model text contains U+D800-range
surrogates by sanitizing to U+FFFD before writing to UTF-8 stdout.
Co-authored-by: Cursor <cursoragent@cursor.com>
Teknium