state.db corrupted twice in two days with the torn-b-tree signature —
repeated "2nd reference to page", "Rowid out of order", and long runs of
"never used" pages in messages (rootpage 5) and idx_messages_session.
macOS fsync() guarantees neither data-on-platter nor write ordering, which
_enforce_macos_synchronous_full already documents: a rewrite interrupted by
process or OS termination leaves half-written b-tree pages. The mitigation
is per-connection (synchronous=FULL + checkpoint_fullfsync=1) and was
applied only through apply_wal_with_fallback(). The repair path opened
state.db with a bare sqlite3.connect() six times and then ran REINDEX,
VACUUM and writable_schema surgery through it — the operations that rewrite
nearly every page of the file — with no barrier at all.
- _connect_repair_durable() routes every repair/probe connection through the
barriers. Applying them is best-effort by necessity: SQLite loads the
schema before any statement, so on a malformed schema even
PRAGMA synchronous=FULL raises DatabaseError, and a malformed database is
precisely this helper's input. _reapply_durability_barriers() retakes them
before REINDEX and VACUUM, once the schema parses and they can stick.
- verify_state_db_integrity() adds the proactive check that was missing.
Repair only ever ran reactively, after a caller already hit a malformed
error, so a database torn in pages no query happened to touch stayed live
and kept accepting writes. On 2026-08-19 that gap was 11 hours across two
restarts that both reported a clean start. Size-aware: degrades to an O(1)
probe above 2 GiB rather than pegging a CPU at startup.
Also restores two fixes lost when `hermes update` reset the tree to
origin/main before they were committed:
- _db_fingerprint keys the repair ledger on dev+inode+size instead of
size+mtime_ns. The old form was justified as "stable for a file nothing
can successfully write to"; that premise is false, because on FTS
corruption this module deliberately keeps canonical writes enabled with
FTS detached. mtime churned on every write, so each pass re-keyed the
ledger and reset the counter to 1 — the cap could never be reached and the
damaging surgery could retry forever.
- _live_writer_holds_db() refuses surgery while another connection holds the
database. The cross-process lock only serialises repairers against each
other; it says nothing about the gateway, Desktop or a CLI. Rewriting
b-tree pages under a concurrent writer is what spread the 2026-08-18/19
damage out of the FTS shadow tables and into the canonical ones. Fails
open, so it cannot strand the self-heal path it protects.
The guard's own tests built a two-table toy schema, so every repair aborted
on "no such table: sessions" before reaching the guards under test — the
assertions were passing over a code path that never ran. They now build
through a real SessionDB.
Targeted state/repair suites: 330 passed, 1 pre-existing unrelated failure.
Broader sweep: 50 failed/1221 passed -> 46 failed/1225 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Dhanesh Purohit <dhanesh@users.noreply.github.com>
text.splitlines() returns [] for empty strings. Accessing msg_lines[0]
then raises IndexError, making session resume crash when the session
contains a message with empty or whitespace-only text (e.g. reasoning-only
turns, tool-only assistant messages).
Guard with `or [""]` in all three branches (user, assistant_last,
regular assistant) so an empty message renders as a blank line.
Fixes#59265
Co-authored-by: AlexFucuson9 <AlexFucuson9@users.noreply.github.com>
Address review feedback from @jackulau on PR #90871:
1. psutil.open_files() silently drops '(deleted)' WAL sidecar entries
on Linux because isfile_strict() stats the literal path including
the suffix and fails. Switch to direct /proc/<pid>/fd readlinks
which preserve the '(deleted)' suffix so _canonical can match.
2. psutil.process_iter() converts AccessDenied to None, which
or-() skips silently — the fail-closed branch never runs. For the
root-gateway vs user-desktop topology in the issue, the fd table is
unreadable but /proc/<pid>/cmdline is world-readable. Add a cmdline
fallback that flags uninspectable processes.
Also keep the psutil path for macOS/BSD (no '(deleted)' convention).
Addresses @helix4u's review on #91493:
- conversation_loop now stamps failure_retryable (the real ClassifiedError
verdict) next to failure_reason; error_surface prefers it and only falls
back to the reason set for older results. Fallback set corrected to match
classify_api_error (auth, format_error, billing_unverified now
non-retryable).
- The descriptor carries the failing session's provider/model captured at
classification time; Copy error details prefers them over the foreground
composer atoms.
- Open logs is labeled 'Open Desktop logs' on remote/cloud connections —
the local folder holds transport logs, not the remote runtime's.
- API-exception module allowlist widened to botocore/boto3/google/grpc/
requests/aiohttp so other adapter SDKs don't misclassify as gateway.
Turn errors now carry a structured {layer, code, retryable} descriptor
(agent/error_surface.py) built from the same classifier the retry loop
uses. The tui_gateway stamps it on terminal error frames, retained
failed-turn snapshots, and resume replay; the Desktop error card renders
the layer title (provider / endpoint / streaming / auth / billing /
gateway / runtime / disk) plus matched actions: Retry, Switch provider,
Open logs, Copy diagnostics.
Older backends that omit the descriptor keep today's behavior (generic
title, string-sniff fallbacks) — the field is advisory on both sides.
Bot Mode agents now DM teammates through a real tool instead of
hand-assembled shell commands. message_agent(target, message) validates
the target against the live roster, applies the sender's attribution
prefix server-side, and delivers over the existing proven transports
(hermes -p ... --query-file for local teammates, hermes peer dm for
peer gateways) as a tracked background process with notify-on-complete
— fire-and-forget, the reply wakes the sender on a later turn.
Containment: the schema is injected per-turn ONLY into a bot's
canonical 'Bot Chat' session on Bot-Mode-managed installs (same gate as
the protocol section); it is never registered in the tool registry or
any toolset, and dispatch re-gates on the session title so a forged
call from any other session refuses. The gate is session-stable, so the
tool list stays byte-identical across turns (prompt-cache safe).
The protocol section is rewritten to teach the tool and now carries the
teammate roster WITH ROLES (Bot Mode title + profile description), so
bots know who does what before picking a recipient. Roles and a
protocol version salt join the capability fingerprint: existing eternal
Bot Chats adopt the v2 protocol + tool with one epoch refresh, and a
rename/description edit refreshes the roster on the next message.
Review feedback on #89507: in-place merging suits a branch that tracks the
target with a small patch set, but a long-lived feature branch (a PR branch
hundreds of commits deep) does not want an update-driven merge commit
written into its history. Reported against a checkout carrying 819 unmerged
commits.
--switch-branch routes the unmerged case to the switch path instead: the
checkout moves to the update target and updates there, and the branch is
left byte-identical — no merge, no commit, nothing written to it. The tree
is known clean on that path (the guard checks dirty before cherry), so a
dirty tree still gets the loud skip, unchanged.
Opt-in: without the flag the default remains the in-place update, which is
what keeps a small-patch-set branch's running code current.
Tests: the flag switches and leaves the branch tip byte-identical; the
default without it still updates in place. The first fails if the flag's
branch is severed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The parked-branch guard (8ce8ffd429) distinguishes checkouts by what the
branch carries, then treats both non-clean cases the same: a stale
fully-merged leftover is switched back to the target (correct), but a
branch with unmerged commits — a branch someone is actually working on —
gets CODE UPDATE SKIPPED and exit 1. For anyone running a maintained
custom branch on top of main, every update now refuses, and the guidance
('checkout main') abandons their branch.
The guard's own reason codes already separate the cases, so use them:
- fully merged -> switch back to the target (unchanged)
- unmerged:N -> update the branch IN PLACE: fetch, then bring
origin/<target> into the checkout. Fast-forward
when possible; on divergence, a true merge behind
a pre-update safety tag, stopping cleanly on
conflict. The checkout never moves; local commits
survive; the running code advances.
- dirty/unverifiable/opted out -> skip loudly (unchanged)
The post-pull success gate learns that an in-place update legitimately
ends on a non-target branch: origin/<target> was merged INTO the checkout,
so refusing to claim success there would fail every update that did
exactly the right thing.
Guard tests updated: the unmerged case now asserts the in-place outcome —
target code arrives (b.txt from c3), the branch's own commit survives, and
HEAD never moves. 18/18 guard tests, 20/20 with the diverged-update suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A clean checkout parked on a feature branch now always switches to the
update target. Unmerged commits are safe on the branch (git checkout
never discards committed work) and get a loud 'kept' notice naming the
branch, count, and the checkout command to resume the work. Previously
the update hard-skipped with exit 1 — a dead end for the desktop update
button, gateway /update, and cron, which have no way to resolve a skip.
Dirty trees (uncommitted changes) still skip loudly, and the
updates.auto_switch_parked_branch: false opt-out still pins the branch.
Drop the bulk test additions from the original PR; keep only mandatory
picker-assertion adaptations (Mantle IDs join the discovery lists), one
allowlist routing test covering all four Mantle model IDs, the 272K
context check, and the two review-mandated auxiliary regressions
(config-region-beats-env for the Mantle path, aux Responses client).
Address review feedback on #65076:
- Add resolve_bedrock_runtime_region() to agent/bedrock_adapter.py: the
config-first region resolution (bedrock.region in config.yaml, then
AWS_REGION/AWS_DEFAULT_REGION/botocore profile/us-east-1) that the main
runtime resolver uses, exposed as a shared helper.
- Switch auxiliary client resolution (agent/auxiliary_client.py aws_sdk
branch) to the new helper. Previously it derived its region with bare
resolve_bedrock_region() (env-first), so when config.yaml pinned
bedrock.region to a different region than the ambient AWS env, auxiliary
calls (compression, memory, vision) left the primary runtime's region.
Both the AnthropicBedrock/Converse path and the new Mantle OpenAI
Responses path now resolve identically to the main runtime.
- Add regression tests covering the bedrock.region-vs-AWS_REGION mismatch
for both the Claude auxiliary path and the Mantle auxiliary path.
- Update website/docs/guides/aws-bedrock.md: the guide claimed Hermes never
uses the OpenAI-compatible endpoint, which the Mantle route made stale.
Document the triple routing (AnthropicBedrock / Mantle OpenAI Responses /
Converse), the Mantle auth model (bearer token or SigV4), and add the
GPT-5.5/5.6 model IDs to the models table.
GPT-5.6 Sol, Terra, and Luna went GA on Amazon Bedrock on 2026-07-13.
Like GPT-5.5, they are served exclusively from the Bedrock Mantle
OpenAI-compatible Responses endpoint (the model cards list
bedrock-runtime/Converse as unsupported), so they ride the allowlist
routing introduced for GPT-5.5:
- Add openai.gpt-5.6-{sol,terra,luna} to BEDROCK_OPENAI_RESPONSES_MODEL_IDS
so runtime resolution, auxiliary calls, and MoA slots all take the
SigV4/bearer Mantle Responses path.
- Surface the family in the curated Bedrock picker list.
- Record the 272K context window from the AWS model cards for all four
Mantle OpenAI models (previously fell back to the 128K default).
- Generalize picker tests from the hardcoded single-model checks to the
BEDROCK_OPENAI_RESPONSES_MODEL_IDS allowlist so future Mantle model
additions do not require test surgery; add routing, picker, and
context-length coverage for the 5.6 family.
Docs: https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards-openai.html
Preserve the Bedrock provider identity for MoA reference and aggregator slots so Bedrock OpenAI Responses models use the aws_sdk/SigV4 runtime instead of being downgraded to a generic custom endpoint. Add regression coverage for Bedrock GPT-5.5 MoA slots.
Route Bedrock-hosted OpenAI GPT-5.5 through the Bedrock Mantle OpenAI Responses endpoint with SigV4 request signing. Keep native Bedrock Converse and Claude Bedrock routing unchanged, and add picker/runtime regression coverage.
prefers_fresh_final_streaming read only the raw direct_messages_topic_id
key; the adapter's canonical accessor _metadata_direct_messages_topic_id
also accepts the documented telegram_direct_messages_topic_id alias
(treated as equivalent in gateway/delivery.py), so an alias-only lane
would still flatten tables. Route the gate through the accessor and pin
the alias with a regression (mutation-checked: raw-key gate fails it).
Also reshape the happy-path endpoint assertion into the actual invariant
(sendRichMessage present, no rich draft frames) instead of a frozen call
list. Surfaced during review of PR #91436.
Pins integer topic routing on send_draft, a successful topic stream
that finalizes through sendRichMessage, and the reporter path where
sendMessageDraft and in-place rich edits both fail — the persistent
payload must still be the raw pipe table, not convert_table_to_bullets.
GLM-5.3 accepts a graded low/medium/high/max reasoning_effort scale
(verified live in #91789: monotonic reasoning-token scaling, no 400s),
but the effort mapper reused GLM-5.2's two-level vocabulary, silently
rewriting low/medium to high. Adds GLM53_EFFORTS/GLM53_OVERRIDES and a
per-model vocabulary pick in the zai plugin; 5.2 keeps its high/max
clamp. Closes#91789. Also covers the gap noted when closing #86947
(credit @santhanakrishnan-d and @terje1965 for the graded-scale finding).
Use the existing wall-clock deadline helper for updater.stop() during network recovery. If PTB cleanup remains cancellation-shielded past the deadline, escalate to retryable fatal recovery so the runner builds a fresh adapter instead of calling start_polling() while the old Updater may still hold its lifecycle lock.
Add regression coverage with stop() swallowing cancellation while holding the same lock start_polling() needs, and verify the old Updater is never reused.
A recurring job that fails at the scheduler layer - an exception escaping
run_one_job's body before the agent is ever constructed - has delivered a
failure alert since 4668750fa. It has never carried the repeated-failure
review nudge the normal agent-failure delivery carries: the nudge (#80752,
2026-08-06) predates that second delivery site by eight days and only ever
composed the first one.
The streak itself is layer-agnostic. mark_job_run increments failure_streak
for an escaped failure exactly as it does for an agent failure, and the
escape handler calls it. So the counter climbs correctly and shows up in
`hermes cron list`, but the chat message that spends it is unreachable for a
job whose failures ALL escape - a half-applied update leaving a bad import,
a provider client that cannot construct. Those are precisely the failures
that repeat identically on every tick, so the operator gets the same one-line
error every 10 minutes indefinitely and is never told the automation itself
is worth reviewing or pausing.
Compose the nudge at the escape handler's delivery exactly as the normal
path does. It stays config-gated and threshold-gated by the same helper, so
a first-time escaped failure reads exactly as it did before.
Docs said the streak counts "runs where the agent failed", which is what the
reporter read and reasonably concluded their failures were out of scope. The
counter never worked that way; correct the sentence to match the code.
Tests: two cases on the escaped-failure delivery path - streak at threshold
appends the nudge (fails on the unfixed handler with the bare summary), and
streak below threshold delivers the unchanged one-liner, so the guard also
proves the nudge is not unconditional. The existing nudge tests only ever
exercised the helper in isolation, which is why the second delivery site
could be added without it.
Fixes#88655
`_guard_named_profile_under_multiplexer` correctly refuses a named-profile
gateway while the default gateway is multiplexing — starting a second one would
double-bind that profile's platforms. The refusal is right; its exit code was
not.
The refusal is decided entirely by configuration (`multiplex_profiles` plus the
allowlist), so it is permanent: no number of retries can change the answer.
Exiting 1 made it look transient to a service manager.
That matters because this module generates the systemd unit, and the template
pairs `Restart=always` / `RestartSec=5` with `StartLimitIntervalSec=0` — it
deliberately trades systemd's generic start-rate limiter for the specific
`RestartPreventExitStatus=GATEWAY_FATAL_CONFIG_EXIT_CODE` backstop declared
three lines below it. Returning 1 left that backstop unarmed with the limiter
already disabled, so a correct, permanent refusal became an unbounded restart
loop. Observed on a host running `multiplex_profiles: true` with a leftover
per-profile unit: 136 refusals in ~13 minutes, stopped only by hand.
`GATEWAY_FATAL_CONFIG_EXIT_CODE` (78, EX_CONFIG) is this codebase's existing
answer for exactly this case — `gateway/restart.py` documents it as the fatal
configuration error that the s6 finish script translates into 125 "permanent
failure" (#51228). This adopts that contract rather than inventing one, so the
fix also works on s6 hosts, not just systemd.
After: one refusal, `status=78/CONFIG`, `NRestarts=0`, unit settles in `failed`.
Also strengthens the two guard tests. They asserted
`pytest.raises(SystemExit, match="1")`, but `match=` is a regex search over
`str(exc)`, so it passed for 1, 21, 100 and 111 alike — it read like an exit-code
assertion while pinning nothing. They now assert
`excinfo.value.code == GATEWAY_FATAL_CONFIG_EXIT_CODE`. The exit code is the
contract here: it is the only thing that tells a supervisor the failure is
permanent.
hermes backup freezes mid-archive when a .db file under HERMES_HOME is
locked by another process — e.g. a live Chromium profile database held
with an exclusive lock by a running browser. sqlite3.Connection.backup()
retries SQLITE_BUSY indefinitely and never honors the connection's busy
timeout, while a plain statement on the same source fails cleanly after
~5s with "database is locked".
Fixes:
- Probe the source with a cheap read before snapshotting, so a locked
database fails fast instead of hanging the whole backup.
- Add a watchdog that interrupts the source connection after 15 minutes
as a last resort for pathological cases.
- Exclude browser-profiles/ from full backups: the CDP browser profile is
live, regenerable (cache + re-login), and unsafe to snapshot while
running. On a real install this cut the backup from 28,396 files /
1.1 GB to ~4,000 files / 548 MB, completing in ~33s instead of hanging.
The pre-update automatic backup shares this code path and was equally
at risk.
Adds a regression test that holds an EXCLUSIVE transaction in a separate
process and asserts _safe_copy_db returns False in bounded time.
`hermes backup` already skips `backups/` so a full zip never re-ships
earlier pre-update zips. `state-snapshots/` (written by `hermes backup
--quick`, `/snapshot create`, and the pre-update safety net) has the same
shape — every retained snapshot holds its own copy of state.db — but was
not in `_EXCLUDED_DIRS`, so a full backup shipped the DB once per
retained snapshot on top of the live one.
Two places hit this in practice:
- `hermes update` in `full` mode takes the quick snapshot *before* the
full zip, so the pre-update zip always nests the snapshot it just made
(state.db twice in every pre-update-*.zip).
- Any recurring `hermes backup --quick` (default keep=20) makes a daily
`hermes backup` grow by roughly one compressed state.db per retained
snapshot; a 750 MB state.db with two snapshots on disk pushed a daily
zip from 1.8 GB to 2.3 GB.
Add `_QUICK_SNAPSHOTS_DIR` to `_EXCLUDED_DIRS` (moving the constant up
next to the exclusion rules so there is one source of truth). Both walk
sites and `_should_exclude` share the set, so `hermes backup`, the
pre-update zip and the auto-backup path all pick it up. Restoring
snapshots after a machine move was never the point of the full backup —
`profiles.py` already excludes `state-snapshots/` from `--clone-all` for
the same reason.
Tests: unit case next to the `backups/` one, plus two end-to-end cases
that use the real `create_quick_snapshot` producer and assert the zip
carries exactly one state.db (full backup and pre-update-order).
Free ($0/$0) Nous Portal models sat with a blank discount column and no
sale star (stealth/ox-alpha, upstage/solar-pro4:free), reading as missing
data next to the -20% sale rows. compute_sale_discount now returns a flat
100% for free models; was_* raws pass through only when the gateway served
a pricing.original, so natively-free models render bare '-100%' with no
fabricated 'was ?/?'. CLI picker star follows on_sale automatically;
inventory feed carries discount_percent=100 to Desktop, whose FREE badge
row now renders the amber -100% pill beside it.
The OpenCode Zen wire slug for the Ox Alpha stealth model is opaque
(x-preview-f-free); users searching the picker for 'ox' or 'ox-alpha'
found nothing. Adds the search alias across all four synced alias
tables (CLI, desktop, web, TUI) plus tests. Wire id is unchanged and
still what renders and gets sent to the provider, matching the k3 →
kimi-k3 precedent. No canonical-dedup collision with opencode-go's
keyed ox-alpha-free slug.
resolve_exec_command wrote the repo hermes script (env-python shebang)
straight into Exec=; spawned by the DE that shebang escapes the venv and
dies on the first import, invisibly (Terminal=false, entry rewritten
every launch). A python-script launcher whose shebang points outside the
running interpreter's env now gets Exec={sys.executable} {script} desktop;
native binaries, bash wrappers, and venv-shebang scripts are untouched.
The code swap and gateway fleet restart touch all profiles, but the
pre-update quick snapshot photographed only the invoking profile's home
— siblings had no snapshot for the post-update safety nets or manual
restore to draw on.
- backup.py: create_pre_update_snapshots_all_profiles() — the SAME
snapshot set, per-file 1GiB cap, and keep policy as the invoking
profile (no partial tier, no new restore-coherence class), each into
the sibling's own state-snapshots/; restore_cron_jobs_all_profiles()
runs the #34600 cron-loss safety net per profile against its OWN
snapshot (same-generation by construction).
- update_cmd.py: sibling snapshots taken right after the invoking
profile's (best-effort, receipt-recorded); post-update cron restore
extended to every sibling.
- Docs: updating.md pre-update snapshot step now states the per-profile
behavior and the file-loss-recovery vs rollback contract.
- 9 unit tests + E2E (real files: sibling snapshot on disk, clobbered
jobs.json restored 7/7 from the sibling's own snapshot, keep=1 prune).
cron_delivery_targets() now also lists machine-local bot-chat:<profile>
entries; the sibling test's exact set-equality assertion predates them.
Scope the platform assertions to gateway entries and pin that bot-chat
entries are always home_target_set.
deliver='bot-chat[:<profile>]' is a machine-local pseudo-platform: the
scheduler delivers job output as a real inbound turn in the target
profile's canonical Bot Chat via the chat CLI lane (--in ~ -c "Bot Chat"
--create-if-missing -Q --query-file), the same lane Bot Mode
agent-to-agent messages use. The bot reads the output, acts on it, and
responds in its chat — instead of the output only landing in Run history.
- cron/scheduler.py: token parsing, target resolution (own profile /
named local profile / unknown -> skipped with warning), subprocess
delivery lane with cron.bot_chat_delivery_timeout_seconds (default
600s), preflight exemption, and bot-chat entries in
cron_delivery_targets() for UI pickers. Excluded from 'all' by design.
- tools/cronjob_tools.py: create/update-time validation — named profiles
must exist on this machine (fail at create, not at 3am); deliver schema
documents the new token.
- tui_gateway/methods_tools.py: cron.manage add forwards deliver.
- hermes_cli/profiles.py: list_profile_names() cheap name-only scan.
- hermes-bots plugin: Create Cronjob dialog gains a 'Send results to'
picker (Run history only / <bot>'s chat); bot-chat jobs send the BARE
token on the profile-scoped create so Desktop-side aliases can never
name a profile the backend doesn't have.
- Docs: user cron guide, automate-with-cron, cron-internals.
Machine-local by construction: names resolve only against the executing
machine's ~/.hermes/profiles/, so overlapping profile names across
multiple connected gateways are unambiguous.
The global broker snapshotted browser.extension_control.developer_mode once
at construction, so flipping it OFF in config did not revoke raw CDP/eval
from already-attached controllers until process restart — a revocation
failure at the highest-privilege browser surface (blocker 3 of
andrexibiza's #91535 review). select() now consults the live config on
every privileged selection (explicit bool still pins for tests); off->on
also unlocks without restart. Regression test drives both directions
against an attached controller. Also drops the dead back-compat
_artifact_store property (zero readers).
Addresses both merge blockers from @andrexibiza's review of #85351:
1. HTTP-uploaded artifacts could never be consumed by broker dispatch:
artifact_scope_key hashed (principal, session, family), the HTTP routes
store with an EMPTY session (API-key auth has no server session) while
broker validation carries a session-bearing ControllerScope — every
real upload->dispatch journey died with ArtifactScopeMismatch
(reproduced before fixing). Canonical ownership is now
principal/transport-family (documented in the scope-key docstring);
ids stay unguessable server-minted 32-hex and downloads one-shot.
New composition regression: HTTP-shape upload -> registered controller
scope -> broker artifact dispatch, mutation-checked (re-adding session
to the key makes it fail).
2. The 'profile-scoped' artifact store was first-profile-wins process
state: one adapter-level singleton pinned profile B to profile A's
physical root on multiplex listeners (same frozen-handle class as
#88734). Stores are now cached by resolved profile, and the broker
selects the store from the controller scope's profile_id (default-slot
fallback preserves single-profile/test behaviour). New A/B multiplex
regression proves distinct physical roots regardless of touch order.
Also documents the advertised ticket_expires_at as best-effort wall clock
(broker enforces expiry monotonically) per review feedback.
- Rename the broker's TicketInvalid to ControllerTicketInvalid: the same
exception name already exists in hermes_cli/dashboard_auth/ws_tickets.py
and BOTH are caught in the same WS auth flow this feature touches — two
unrelated same-named exception types in one blast radius invited a wrong
except clause.
- Import the 'server-internal' sentinel identity from its canonical
definition (ws_tickets.INTERNAL_USER_ID/INTERNAL_PROVIDER) instead of
re-declaring the strings; drift would have silently broken the
internal-peer exclusion in _is_authenticated_identity.
Surfaced during review of PR #85351.
The router treated any server-stamped principal as a bound lane, so with the
flag ON every authenticated dashboard/API session lost the legacy browser
backend even when no extension controller ever registered (scope_for_session
returns None -> ControllerUnavailable, no fallback) — while check_fns still
advertised the tools via the legacy OR-gate.
New broker.lane_registered() distinguishes the two cases:
- lane never registered -> generic callers keep the legacy backend
- lane registered (controller offline/ambiguous) -> fail closed, unchanged —
a control-this-tab session never silently jumps to another browser
Also makes the four non-allowlisted wrapped tools (cdp/console/vision/
get_images) behave correctly for never-registered lanes (legacy backend)
while staying fail-closed for registered lanes.
Surfaced during review of PR #85351.
Generic Hermes callers still use the existing browser backend when extension control is disabled or no server-bound controller identity exists.
Once the gateway binds a controller identity, missing scope, disconnect, or capability loss now fail closed instead of silently switching a control-this-tab request to another local or cloud browser. Covers the schema-build to dispatch disconnect race.
Treat unexpected controller transport loss as recoverable until each command's original deadline. Same-identity reconnects refresh transport and capability state, flush deferred cancels before new dispatch, and can complete already-started work.
Keep explicit detach and different controller/browser identity replacement terminal, owner-gate every inbound lifecycle frame, distinguish slow in-flight WebSocket writes from real send failures, and exclude browser-control session identity from shared shell snapshots.
Keep extension control opt-in and preserve existing browser backends unless an exact server-bound controller is available. Centralize protocol and capability admission across API and dashboard transports, make selected-controller results authoritative, bypass stale availability caches only inside bound requests, and serialize structured results for the existing tool contract.
Add a real browser_snapshot route-table/WebSocket E2E, strict admission and ownership regressions, public configuration and protocol documentation, and tests proving feature-off/no-controller compatibility.
Project client-visible session messages through the canonical compaction classifier. Hide standalone handoffs, unwrap merged carriers to their authentic prior-tail content, strip inherited internal fields, and keep model-facing recovery history unchanged.
Identify a completed merged assistant handoff from the carrier's own stop state instead of an unrelated adjacent history row. Keep carriers with pending tool calls actionable so compaction cannot abort a live tool chain.
Treat a merged assistant-role summary carrier as the driving reference handoff when it immediately follows a completed assistant stop. Its preserved prose and stale tool_calls are assistant continuity, not a fresh live user request.
Keep legitimate in-flight behavior unchanged when there is no completed stop, a real user turn follows, or a distinct later assistant tool-call row continues the loop.
Extends the #80622 active-turn guard for the merged-carrier shape reported under #42768.
The Codex OAuth backend (chatgpt.com/backend-api/codex) intermittently
injects prompt_cache_retention into its own upstream call and then rejects
it, returning HTTP 400 invalid_parameter. Hermes never sends that field on
this route (see agent/transports/codex.py::_default_prompt_cache_retention_
for_request, which only sets it for api.meta.ai and bedrock-mantle hosts).
Reproduced live: a minimal 1-message request carrying no cache parameters
at all failed 4/20 (20%) with this error, so the rejection is not
deterministic and retrying the identical request is the correct recovery.
Previously the catch-all in _classify_400 returned format_error/
retryable=False, which tripped the is_client_error abort gate in
conversation_loop and killed the turn on the first attempt - burning an
entire large-context request (~550k tokens) per failure.
Classify these as retryable server_error (should_compress=False - the
request shape was never the problem). The same guard is applied to the
sibling 5xx request-validation branch, where a fronting proxy can surface
the identical rejection.
Deliberately narrow: keyed on parameters we only send on specific routes,
and skipped when the current provider is one that legitimately sends them,
so a genuine client-side bad parameter (max_tokens on GPT-5) still fails
fast as a format_error.
prune_pre_checkpoint_items() had a hardcoded role=='user' filter that
discarded all non-user messages before a checkpoint — including Hermes'
own compression summaries (role='assistant'), causing total context amnesia
about past conversation summaries.
The fix:
- _is_summary_item delegates to the canonical
agent.context_compressor.is_compaction_summary_message provenance check
(not an ad-hoc heuristic)
- Summaries are retained whole (never byte-sliced) within a 32k token budget
- Idempotent across repeated checkpoints (dedup by identical text)
- _chat_messages_to_responses_input threads item_sources (raw chat messages)
through to the pruner, so it can read summary content directly from the
source when the Responses conversion shape is lossy (tool-result carrier
becomes function_call_output, or stale codex_message_items replay shadows
merged content)
Fixes#90975.
Salvage of #90976 by @JoaoMarcos44.
First actioned report from the overhauled model-catalog-scout cron
(2026-08-21 validation run), every item re-verified live before edit:
Delisted (gone from live catalogs):
- opencode-zen curated: claude-opus-4-1, qwen3.7-max, qwen3.7-plus
(absent from live zen /v1/models; qwen3.7 family remains on Go)
- OPENROUTER_MODELS free section: poolside/laguna-m.1:free (rotated to
s-2.1/xs-2.1), tencent/hy3:free, inclusionai/ring-2.6-1t:free
Added (present + verified in live catalogs):
- OpenRouter free: z-ai/glm-5.2:free (256K), poolside/laguna-s-2.1:free
+ laguna-xs-2.1:free (262K), nvidia/nemotron-3.5-lightning:free (1M)
- opencode-go curated: ox-alpha-free (Go-subscription twin of the Zen
keyless Ox Alpha; keyed — Go relay 401s anonymous requests)
Metadata:
- DEFAULT_CONTEXT_LENGTHS: laguna-s-2.1/xs-2.1 262144;
nemotron-3.5-lightning 1M (overrides the generic 131K nemotron entry);
glm-5.2:free 256K (the free variant is capped below the 1M paid entry)
Keyless-heal hardening (the real find):
- opencode_zen_free_runtime now gates the zen/go→keyless heal on
MEMBERSHIP in the verified opencode-free catalog, not the -free
suffix — ox-alpha-free is a KEYED Go model despite its suffix, and
suffix-based healing would have routed it to a Zen relay that
doesn't serve it (verified: zen 401s 'not supported', go 401s
'Missing API key'). New regression test pins this.
Fixture sweep: tencent/hy3:free catalog assertion updated (delisted
slug); nous-route fixtures using hy3:free as incidental model names
left alone (self-consistent mocks). model-catalog.json regenerated.
A keyless provider has no credential to lack, but every auth-gated
surface treated 'no key' as 'not authenticated', so opencode-free was
invisible in /model, provider:model listing, and the desktop model
pickers unless a user had unrelated OpenCode env vars set.
One policy, three gates, all derived from the HermesOverlay keyless
flag (#91358):
- auth.py get_api_key_provider_status: keyless providers report
configured/logged_in=True with key_source 'keyless' — flows through
get_auth_status to every status consumer (hermes status, dashboards,
list_available_providers).
- model_switch.py list_authenticated_providers: keyless overlay rows
get has_creds=True before any env/pool/auth-store checks — this is
the source for /model, the TUI picker, and the desktop
/api/model/options payload.
- inventory.py explicit-only filter (desktop chat pickers): keyless
providers are kept — there is nothing to 'explicitly configure', and
hiding a zero-setup provider defeats its purpose.
E2E (temp HERMES_HOME, all keys stripped): get_auth_status logged_in,
list_available_providers authenticated, picker row with 6 models,
desktop payload default AND explicit_only both include the provider,
and the full switch pipeline (parse free:x-preview-f-free →
switch_model) resolves to the keyless runtime. 4 new tests.
Phase 2 core slice of #91277: the updater now knows WHAT it is operating
on before it mutates anything.
- hermes_cli/update_inventory.py (new): side-effect-free runtime
inventory — install kind via detect_install_method (git / docker / nix
/ apt, updatable-in-place or not, with the correct external update
command for image/package-managed installs), all profiles, every live
gateway with its supervisor (systemd / launchd / manual via the
fleet-wide _get_service_pids), running code_sha/code_version from the
#91283 gateway_state.json stamps, and the restart mechanism each
runtime will get.
- hermes update --plan: prints the plan and exits; runs BEFORE the
docker/nix refusal gates so image-managed installs get a useful
'not updatable in place + right command' report instead of a bare
refusal. Read-only, safe on a live fleet.
- Every real update run now records the pre-update plan in its receipt
('plan' key) and prints a one-line fleet summary, so post-mortems can
compare what the update SAW against what it did.
- Docs: updating.md (--plan section + receipts/fleet-check section),
cli-commands.md (flag row + receipts behavior bullet).
- 11 tests: two-profile fleet classification, docker not-in-place,
dead-PID exclusion, PID-file fallback dedupe, all-probes-fail
never-raises, JSON round-trip for the receipt, print output shapes,
receipt integration.
Dhanesh Purohit