Desktop opens /auth/native/authorize without naming a provider. The empty-provider
auto-select filtered password providers out of the candidate set, so a deployment
with one OAuth provider plus username/password went straight to OAuth and never
offered the password option — even though native sign-in brokers password
providers through /login since 56f1afc834. The filter's rationale ("a password
provider can never be the target of the native flow", ed5e17f4b8) predates that
change.
Render a provider chooser when more than one interactive session provider is
registered. Each link re-enters the same validated authorize route with an
explicit provider, so the choice never leaves the native PKCE flow. A single
provider still auto-selects, and the chooser is emitted before any broker state
is allocated or any cookie set.
The desktop only shell-opens the authorize URL and never parses its response
(apps/desktop/electron/native-oauth-login.ts), so the 200 chooser page is safe on
existing desktop builds.
Supersedes #101713, #76941.
Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>
Co-authored-by: Phuong Lambert <vmphuongit@gmail.com>
Subagent completion notices ("✓ [set 7 · 2/2] … (2175.14s)"), background
process notices and spinner print_above text are emitted from worker threads
through agent._print_fn. In the interactive CLI that was bound straight to
_cprint, so a notice landing while a reply streamed was printed between two
paragraphs inside the response box.
_print_fn is now HermesCLI._agent_status_print: while a response or reasoning
box is open the line is parked in _held_status_lines and released right after
the box footer; outside a box it prints immediately as before. Gateway, ACP
and TUI printers are untouched (they never bound _cprint).
* feat: add session-scoped connector access for onboarding
* fix(connectors): availability is the config flag AND the portal entitlement — no free-tier leg
The port carried a third availability leg from hermes-magic: a stored guest
(free-tier) identity short-circuits the managed-tool entitlement check. That
leg reads hermes_cli.anon_auth, which does not exist on hermes-agent main, so
connectors_available() raised ImportError inside its fail-closed try and the
whole connector surface was silently dark on a plain upstream checkout.
On this tree availability is the two-leg AND the design started with:
tools.connectors.enabled AND managed_nous_tools_enabled(). The free-tier leg
is a hermes-magic concern and belongs in hermes-magic's own delta over this
branch, next to the identity it depends on. Its integration test goes with it.
* docs(tool-search): connectors section — remote tools through the bridge
The squashed port carried the code but not the user-facing docs. Restores the
Connectors section of the Tool Search page and the connector-gateway host /
CONNECTOR_GATEWAY_URL override on the Tool Gateway page, updated for the
manage_connections tool and the pure-connector batch rule.
* fix(tool-search): connector tools rank with local tools in one pass instead of taking leftover slots
dispatch_tool_search ran BM25 over the local catalog, filled `limit` slots,
then appended connector hits only into slots left empty. On a 300-tool
catalog no slot was ever empty, so with Gmail and Google Calendar connected
"send gmail email" returned five betterstack tools and zero connector tools.
The gateway's hits for a query now become catalog entries (connector name,
slug words, description as the search text) and join the local catalog for
that query's BM25 pass. One ranking, one rarest-token admission rule for both
sources, `limit` as the total per query. The merge loop and the separate
record builder for connector hits are gone; `_shared_tool_record` serves both
sources.
The gateway search timeout rises from 8 s to 30 s. One request with six
use_cases measured 7 s, so 8 s sat on the edge and cut real answers off; the
failure path is unchanged (local-only results, no error to the model).
Live, 311 local tools + gateway, before -> after:
"send gmail email": 5 betterstack tools -> gmail SEND_EMAIL, CREATE_EMAIL_DRAFT
"read google calendar events": 5 betterstack tools -> googlecalendar EVENTS_LIST_ALL_CALENDARS
"linear create issue", "betterstack incident": unchanged
Benchmark (25 labelled queries): connector recall 0.09 -> 0.82, precision@5
0.18 -> 0.59, false positives on absent intents 17 -> 2.
* refactor(tool-search): connector leg into tools/connector_search.py
tools/tool_search.py is a facade. The connector leg (gateway hits as catalog
entries for tool_search, remote schemas for tool_describe, the
connections_in_scope gate) was appended to it by the port. It now lives in
its own sibling, tools/connector_search.py, and the facade imports the three
entry points: connections_in_scope, connector_entries_by_group,
remote_schemas_for.
No behaviour change. The tool_describe remote block became
remote_schemas_for(names, current_tool_defs, connector_describe) with the
same inputs, the same silent-degradation contract and the same injection
seam the tests already use.
* fix(tool-search): at most 7 queries per call, the gateway's search limit
One tool_search call sends all its queries to the connector gateway as one
search request. The gateway answers 7 use_cases per request and returns
HTTP 502 for 8 or more (measured 2026-09-09, re-measured with one-word
use_cases: it is a count limit, not a size limit). With the client cap at
10, a model sending 8 to 10 queries lost every connector hit for that call
and saw local-only results with no error.
The shared constant splits: _MAX_QUERIES_PER_CALL = 7 for search,
_MAX_DESCRIBE_NAMES_PER_CALL = 10 for describe, which has no remote count
limit. Eight or more queries now get the existing "too many queries" retry
hint before any request is made. No chunking: one call, one request.
* fix(tool-search): the model is told that connectors__ names are manage_connections accounts
tool_search results carry names like connectors__gmail__CREATE_EMAIL_DRAFT and
manage_connections is the tool that checks and connects those accounts, but
nothing told the model the two are the same thing. A model that hit
CONNECTION_REQUIRED had to infer the fix on its own.
The tool_search description gains one sentence making the link, added at
assembly only when manage_connections is in the session's tools. Signed out
or with connectors off the tool is absent and the description is unchanged,
so it never names a tool the model cannot call. This follows the existing
rule for cross-tool references (tools/AGENTS.md): they are added dynamically
from the session's actual tool set, never hardcoded in a schema.
Tool defs are fixed for the life of a conversation, so the description is
byte-stable per conversation; this is a one-time prefix change.
Live, real get_tool_definitions() against a signed-in home: sentence present.
Same home with auth.json removed: manage_connections absent, sentence absent.
* fix(connectors): /stop halts a connector batch before the next remote call
dispatch_connector_batch runs every remote entry of a tool_call batch in
sequence. The executor only checks the interrupt flag between tools, and
the whole batch is one tool to it, so a /stop landing during entry 1 of
20 still sent the other 19 to the gateway.
The loop now reads tools.interrupt.is_interrupted before each dispatch.
Once set, it stops calling handle_function_call and fills every unstarted
slot with the loop's existing error-slot shape, code INTERRUPTED and the
message "Stopped by the user before this call was made.", so the result
envelope stays valid and the counts stay honest. Entries already
dispatched keep their real results.
Test: three connector calls where the fake client sets the interrupt on
the first execute. The client sees exactly one call and slots 2 and 3
carry INTERRUPTED. Red on the base branch, green with the fix.
* test(connections): schema assertions become dispatch contracts
test_schema_documents_wait_and_its_timeout froze description fragments
("REQUIRED", "can NOT disconnect", "Nous Portal"). A wording edit fails
it while a real regression (a disconnect that reaches the gateway) does
not. That is a snapshot of prose, not a behaviour contract.
Delete it. The requirement that wait needs connectors is already covered
by test_wait_requires_connectors. The user-only disconnect boundary is
now asserted as behaviour: action disconnect with a connector returns an
error and the fake client records no call. That replaces the earlier
de-authenticate test, which only checked that the word "dashboard"
appeared in the error text.
Test count in the file goes from 26 to 25.
* docs(tool-search): connector batches are one gateway request per entry
The user guide said a connector batch travels as one gateway request. It
does not: model_tools_connectors.dispatch_connector_batch re-enters core
dispatch per entry, and each entry becomes its own execute request in
bridge._run_remote (plus at most one literal-slug retry when the gateway
reports TOOL_NOT_FOUND under the conventional slug). The docstrings in
tools/tool_gateway/bridge.py and tools/tool_gateway/__init__.py still
described the abandoned V1 plan and claimed nothing outside the package
imports it.
Rewrite those sentences to match the code: one request per entry, in
input order, dispatched from model_tools_connectors.py, with the per-entry
approval and interrupt behaviour that motivated the split. The guide also
still showed the single-call shape tool_call(name, arguments); both
places now show the `calls: [{name, arguments}]` array the schema
advertises and note that a single local call is an array of one.
Docs only, no test.
* fix(tools): the between-turns refresh never rewrites the bridge tools
The per-turn MCP refresh folds a fresh tool snapshot into the live array
with preserve_prefix: order and membership stay, but a name present in both
takes the fresh schema. That is right for ordinary tools, whose schema is a
constant. tool_search is the one tool whose description is derived from the
session: the deferred-tool count, the embedded listing, and, on this branch,
whether manage_connections was present. A late MCP server or one failed
portal lookup (manage_connections' check_fn fails closed) changed those bytes
on the next turn, and every byte after tool_search in the cached prefix was
re-prefilled. The array also contradicted itself in that case: the flapping
manage_connections was carried forward while the description lost its hint.
The bridge entries now keep the bytes they were built with for the life of
the conversation. Nothing is lost: tool_search reads the live catalog at
dispatch, so tools that arrived late are still found; connector availability
is checked at dispatch too. The compaction-boundary rebuild (content_aware,
the one sanctioned cache break) still refreshes the description.
Consequence: connector exposure in the prompt is decided once, at agent
build, by whether the user was signed in then. That is the intended
contract.
* refactor(tool-search): normalize_tool_call_entries lives with the other argument validation
The port appended the tool_call argument parser to the tool_search facade.
The family already has tools/tool_search_validation.py for exactly this
work (schema validation of deferred call arguments), so the parser moves
there and the facade imports it. No behaviour change; the one test that
imported it now imports from the defining module.
* refactor(connectors): delete the unused batch dispatcher; _run_remote becomes run_remote
bridge.dispatch_calls and its helpers (_dispatch_calls_inner, _run_pre_dispatch,
_run_local, _error_slot, _maybe_parse_json) and the LocalDispatch / PreDispatch
seams had no production caller. Connector dispatch runs through
model_tools_connectors: dispatch_connector_batch re-enters handle_function_call
once per entry, so scope, hook, approval and middleware policy fire against each
composed name inside core dispatch, and dispatch_connector_call hands the single
planned entry to the bridge's transport function. Only tests called the batch
dispatcher, and they exercised policy seams that production never wires.
The transport function is the module's real entry point, so it drops the
underscore: _run_remote becomes run_remote, body unchanged. The module
docstring now describes the two legs that exist (availability with D32 silent
degradation, and run_remote) instead of the injected seams. Imports that only
the deleted code used are gone; merge.py is untouched because every export
still has a caller.
Tests that drove dispatch_calls are deleted where they covered the removed
seams (pre_dispatch blocks and rewrites, local_dispatch classification, mixed
batches). The literal-slug fallback, the per-entry transport failure, and the
hook rewrite reaching the gateway request body are re-targeted at
handle_function_call('tool_call', ...) with the fake client swapped in at
bridge._default_client_factory, the same seam test_connector_dispatch_policy
uses. Each re-targeted test fails when the retry is disabled in run_remote.
* fix(connectors): search keeps the twin a colliding name reaches, and says so
format_connector_name strips the toolkit prefix, so GMAIL_FETCH_PROFILE and a
literal FETCH_PROFILE on gmail both compose to connectors__gmail__FETCH_PROFILE.
describe and execute decode that name to the prefixed slug first, so the
literal twin is unreachable under it. If a vendor ever shipped both, search
could describe the literal under a name that runs the prefixed tool.
Search is the one place that sees both twins in one response. It now keeps
the twin the name reaches and drops the other with a WARNING that names both
slugs, whichever the gateway listed first. Short names stay; no marker, no
per-process map, no change to describe or execute. No such pair exists in the
live catalog today; the guard turns a silent alias into a logged one.
#106582 (1c6683e8e0) added display_identity and display_order to the messages
table in SCHEMA_SQL without appending the matching events to
SCHEMA_HISTORY["messages"], so test_replayed_history_ends_at_current_schema
has been red on main since 17:06Z. Append the two '+' events; no runtime change.
The Desktop settings UI persists a registry provider's API key as a
credential pointer -- model.key_env in config.yaml pointing at an env var
in $HERMES_HOME/.env (e.g. HERMES_CUSTOM_LMSTUDIO_API_KEY) -- while
keeping model.provider on the registry id. _resolve_api_key_provider_secret
resolved registry providers exclusively from
PROVIDER_REGISTRY[...].api_key_env_vars, so the UI-saved key was silently
ignored; for lmstudio the runtime then substituted the no-auth placeholder
(dummy-lm-api-key) and auth-enabled LM Studio servers rejected every
request with an opaque, retryable-flagged 401.
Fix: consult model.key_env (only when config.yaml's main model targets the
provider being resolved, so the pointer never leaks across providers)
before the registry env vars, running the value through the same
_usable_declared_secret prefix validation. Precedence for the canonical
env vars and the credential pool is unchanged; a valid canonical key still
wins when no pointer is set.
Complements #75373, which covers providers.<id>.key_env at the provider
level; the Desktop save path writes the pointer at the model level, which
that PR does not consult.
Fixes#106336
Move the "explicit scratch means no project" decision into the one resolver
every surface funnels through, `kanban_db.create_task`: board-project
inheritance now runs only when the caller left `workspace_kind` open
(`None`), and `workspace_kind` defaults to scratch after that check. The
tool handler keeps the #106347 fix for `project=""` (no `or` collapse) and
`board=` scoping but drops its handler-local sentinel logic, since the
resolver now owns the rule; the `self_task` project inheritance for
dispatcher-owned workers is unchanged.
Sibling surfaces had the same bug through the same line and are fixed by
the same change:
- CLI `hermes kanban create --workspace scratch` on a project-scoped board
produced a project worktree; `--workspace` no longer defaults in
argparse so the resolver can tell "omitted" from "scratch".
- Dashboard `POST /tasks` with `workspace_kind: "scratch"` did the same;
`CreateTaskBody.workspace_kind` defaults to `None` for the same reason.
- `kanban_swarm.create_swarm` threads `None` through for consistency.
Tests: one resolver invariant in test_kanban_board_project.py (explicit
scratch stays scratch, omitted still inherits) and the salvaged tool test
folded into a single parametrized matrix over scoped/unscoped target boards.
Drop the floor-pin test from #106290 (a snapshot of the static table); the
end-to-end merge test through provider_model_ids with the real curated floor
already fails if the slug is re-added. Note beside
_OPENCODE_FREE_KEYED_SUFFIX_MODELS why ox-alpha-free stays excluded from the
keyless catalog even though it left the curated floor.
The Go relay (GET /zen/go/v1/models) delisted ox-alpha-free 2026-09-09, but
_PROVIDER_MODELS["opencode-go"] still carried it. _profile_live_catalog merges
the curated floor into the live list (live-first for opencode-go), so the
model picker kept offering a model that now 401s — the same failure class as
#95914 (opencode-free / x-preview-f-free).
Remove it from the floor and add two regression tests: an end-to-end merge
test through provider_model_ids with the real floor (fails if a stale floor
resurrects it) and a floor-pin test asserting the known-delisted model stays
out of the offline fallback.
(cherry picked from commit 091fd85865caa09e828928f93ee985dae7e9580d)
reap_idle() is reached concurrently from attach_or_spawn() and the
background run_reaper loop, and each pop is followed by an awaited
close(); while one reap is parked inside close(), a second reap reaching
the same doomed key hit dict.pop(key) without a default and raised
KeyError straight into the websocket handler. close_all() has the same
overlap window against an in-flight reap.
Pop with a default and skip keys a concurrent reap already removed, so
overlapping reaps are idempotent. Which sessions get reaped is
unchanged.
Fixes#106400
(cherry picked from commit 8ea63a845424263551487886a7526ca2f4bbd1ee)
OpenRouter and Nous already ship these ids, but the native Anthropic
curated list still stopped at Fable 5 / Sonnet 5. Live /v1/models often
lags or 401s on subscription tokens, so the picker fell back to that
stale list and hid models that already work when addressed directly.
Hermes Console ran each command on a ThreadPoolExecutor and cancelled only
the asyncio waiter. A command that forks an AIAgent (`curator run
--consolidate`) kept its worker thread and its in-flight provider request
alive after the prompt said "cancelled" — a llama.cpp generation kept
decoding for 30+ minutes and held the inference slot (#106179).
Root cause: the host owning the thread never knew about the agent created
deep inside the synchronous command, so it could not call the existing
cooperative `interrupt()` path that closes the request sockets.
Fix: `agent/interrupt_scope.py` gives the host an `InterruptScope`; the
console binds it around the worker (ContextVar), and every
`AIAgent.run_conversation()` registers itself with the bound scope for the
turn. On cancel, timeout and disconnect the console calls `scope.cancel()`
(hard-interrupts every registered agent; an agent registering after the
cancel is interrupted on entry so the cancel cannot lose the race with a
turn that has not started) and awaits the worker Future with a 10s bound
before reporting cancelled/timeout. Queued-but-unstarted work is dropped via
`Future.cancel()` alone.
Live repro (fake OpenAI-compatible provider blocking like llama.cpp, real
/api/console, real curator dispatch, real AIAgent + direct request path):
origin/main at the "cancelled" frame -> request_exited=false,
worker_exited=false; with this change -> both true, provider saw the peer
close, prompt reported cancelled 0.27s after the frame.
Salvage of #106197 by @kyssta-exe (executor-future handle, cancel/timeout/
disconnect propagation) and #106320 by @Xixiartemis (deterministic
lifecycle regression: terminal(cancelled) => no owned request or worker
remains live; interrupt-on-late-registration). Both rebuilt slimmer: #106197 keyed its fallback on Future.cancel()
returning False, but the handle it held was run_in_executor's asyncio
wrapper, whose cancel() returns True while the thread keeps running, so its
thread-name abort registry was never consulted; #106320's command-scoped
ownership model is folded into one small module hooked at the turn facade
instead of a per-caller `bind_agent`.
Co-authored-by: kyssta-exe <218078013+kyssta-exe@users.noreply.github.com>
Co-authored-by: Xixiartemis <182932319+Xixiartemis@users.noreply.github.com>
The dpaste.com fallback path kept pastes for 7 days (vs the promised
6-hour auto-delete), could not be deleted by `hermes debug delete`
(anonymous pastes have no owner token), and the pending-paste sweep
skipped dpaste URLs entirely — while the share output still claimed
pastes auto-delete in 6 hours.
- Default the dpaste.com fallback retention to 1 day (dpaste's minimum)
across --expire, upload_to_pastebin, and build_debug_share so the
dashboard endpoint benefits too
- Print an accurate warning when uploads fall back to dpaste.com
(retention window + no delete path) instead of the 6-hour promise
- Mention the fallback retention in the CLI and gateway privacy notices
before the user consents to uploading
- Explain in `hermes debug delete` errors why dpaste.com pastes cannot
be deleted and that they expire on their own
Refs #106164
[salvage: trimmed to 2 invariant tests (dpaste delete error, dpaste fallback notice);
5 duplicative default-value tests dropped]
Widens #106272 to the one remaining sibling: `launchd_stop()` boots out with check=True and
already handles exit 3/113/125 (job unloaded) and 5/125 (domain unmanageable) by falling through
to the PID kill, yet inherited stderr — so `hermes gateway stop` against an unloaded job printed
"Boot-out failed: 3: No such process" next to "✓ Service stopped". Same `_CAPTURE_TEXT` kwargs as
the sibling calls; an unexpected exit still raises with `e.stderr` populated.
Replaces the contributor's two kwarg-assertion tests (`capture_output is True` on a mocked
`subprocess.run`) with two invariant tests that run a real fake `launchctl` on PATH and read fd 2
through `capfd`: restart-on-unloaded prints only its ↻/✓ lines and drives
kickstart→bootout→bootstrap→kickstart; stop-on-unloaded is silent, while a real bootout failure
(exit 1) still raises with the captured stderr. Both red on origin/main, green here.
Best-effort bootout calls (unloaded-job recovery, stale-EIO retry,
plist refresh, uninstall) and the handled kickstart -k in
launchd_restart inherited the terminal's stderr, so an expected
unloaded job printed raw launchctl errors around the CLI's own lines:
Could not find service "ai.hermes.gateway" in domain for user gui: 501
↻ launchd job was unloaded; reloading
Boot-out failed: 3: No such process
Capture them with _CAPTURE_TEXT instead. The kickstart error stays
available as e.stderr for the update_cmd failure diagnostic, and the
post-bootstrap kickstart intentionally stays loud (its failure feeds
the domain-unsupported fallback). Same precedent as the reload
helper, which already runs bootout with 2>/dev/null.
[salvage: picked hermes_cli/gateway.py only; the two capture_output kwarg-assertion tests are
replaced by fd-level invariant tests with a fake launchctl in the follow-up commit]
An aux task pinned to a private endpoint via `base_url` (a home Ollama
box at `byron.local`, a LAN IP, localhost) is the intended per-task
endpoint feature and can never bill a provider. The Settings → Model
banner still counted it as "still run on openai" forever and offered
"Reset all to main", which would wipe the working local setup; the
post-switch `stale_aux` report had the same blind spot; and the aux row
never showed the `base_url` the backend already sends, so the pin was
indistinguishable from a paid-provider pin.
- `GET /api/model/auxiliary` now stamps each task with `local_endpoint`,
the verdict of the one canonical classifier
(`agent/model_metadata.py::is_local_endpoint`) — no TS mirror of the
private-range rules, so frontend and runtime cannot drift.
- Desktop: the persistent banner filter is the pure
`staleAuxAssignments()` and skips `local_endpoint` pins; the pinned row
appends ` · <base_url>` when one is set.
- `_stale_aux_pins` (post-switch report) skips local pins the same way.
- `is_local_endpoint`: `*.local` (RFC 6762 mDNS) now counts as local, and
IPv6 literals no longer ride the "no dots ⇒ unqualified host" rule, so
a global-scope address (`2607:f8b0::1`) is not local while `::1`,
ULA and link-local still are via the `ipaddress` scope checks.
Slim redo of #106236 (@webtecnica) and #106234 (@huklaa), which fixed the
same symptom with a client-side classifier copy; the bug class, row
display and mDNS/IPv6 classifier corrections are theirs.
Refs #106228
Co-authored-by: Hukla <129692708+huklaa@users.noreply.github.com>
Co-authored-by: webtecnica <webtecnica@gmail.com>
Move the WSL / /dev/dxg / d3d12_dri.so probes into _prefer_wsl_d3d12 with
the probed paths as module constants, so the launcher call site is a single
line and a test can lay out a fake WSLg host without touching real
/dev or /usr/lib. The two tests now run the real _desktop_launch_env end to
end (selected under WSL+dxg+driver; untouched with an explicit Mesa
override, off WSL, without /dev/dxg, or without the driver file) instead of
unit-testing the helper with a precomputed boolean.
Docs: one paragraph in the Desktop guide on the automatic selection and
the env vars that keep an explicit choice authoritative.
Follow-up to Xipong's fix for #106117 (salvaged from #106118).
Fold the helper into _npm_lifecycle_env itself: the whole fix is one
is_file() check plus a setdefault, so a separate function, the
try/except around get_hermes_home() (it never raises) and the
dict-returning indirection were shape-gate violations. Build the path
with os.fspath so it is correct on Windows too.
Tests: keep the two invariants (file present -> NPM_CONFIG_USERCONFIG
points at it; explicit process/caller value wins and a missing file
sets nothing), fold the other two into the negative test.
Docs: one paragraph in the desktop troubleshooting page next to
ELECTRON_MIRROR describing $HERMES_HOME/npmrc.
Refs #106373
- _ssl_interop_hint: also match ssl.SSLError instances (and one level of
__cause__/__context__) plus the bare UNEXPECTED_EOF marker, so an
SSLEOFError whose text httpx did not repeat still gets the hint. The
hint now names the TLS 1.2 diagnostic and links the providers docs
note instead of an issue number.
- tests: 3 -> 2 invariants (parametrized login_post/poll SSL case keeps
the raw text + hint + cause; a plain httpx timeout gets no hint).
- docs: providers.md Codex note carries the reporter's exact openssl.cnf
classic-groups snippet (EN + existing zh-Hans copy).
Refs #106384. The TLS max-version cap itself stays PR #44392's scope.
Device-login requests on networks whose middlebox rejects the larger
TLS 1.3 ClientHello sent by OpenSSL 3.5+ (post-quantum hybrid groups)
fail with SSLEOFError / handshake timeouts while curl still works, so
they masquerade as a Codex outage (#106384). The polling loop let the
raw httpx error escape unshaped, and _codex_login_post dropped the
exception chain and gave no actionable hint.
- add _ssl_interop_hint() applied to both device-login transport paths
- re-raise _codex_login_post failures with 'from exc' to preserve cause
- wrap the poll POST so transport failures become a shaped AuthError
(device_code_poll_error) carrying the SSL detail and OPENSSL_CONF
workaround hint; KeyboardInterrupt handling is unchanged
(cherry picked from commit 8cd94c36ce8437db5b00290b9edbedcd2116c02c)
`hermes kanban promote --force <id>` printed `Promoted <id> -> ready` and
then the very next claim (a human `claim`, or the dispatcher tick seconds
later) demoted the task back to `todo` with `claim_rejected
{parents_not_done}` and returned None (#106195). The non-force refusal
even pointed operators at `--force` as the escape hatch.
The claim gate is deliberate: `claim_task` is the single enforcement point
("never ready -> running with an undone parent, whichever writer set
'ready'", cda20eec0c), and `complete_task`/`request_review` re-check the
same predicate, so a child let through by a forced claim could still never
finish. A promotion override therefore has no honest outcome; the
dependency edge is the real knob.
- drop `--force` from `promote` (parser, CLI handler, `promote_task`
kwarg, the `forced` event field nothing read)
- the refusal message now states why the gate cannot be bypassed and names
the working remedies: complete the parents or `hermes kanban unlink`
- two invariant tests: refusal on an undone parent leaves `todo` with no
fake `ready`; the flag no longer parses
Salvage direction from #75354 by @vyacheslavk (diagnosis of the promote ->
claim gap); the consume-at-claim authorization there is not taken because
the same parent gate also blocks completion of the forced child.
`--in DIR` only chdir'd. Every cwd consumer (resolve_agent_cwd -> Codex
app-server thread cwd, the terminal tool, context-file discovery) prefers
TERMINAL_CWD over the process cwd, so a value inherited from a parent
Hermes surface, the shell or .env survived the chdir and the session kept
running in the old directory. The local backend was rescued by cli.py's
force-export at import time; docker/ssh backends and the TUI launch path,
which never imports cli.py, were not.
Refresh TERMINAL_CWD to the --in target when it is already set. An unset
variable stays unset so the backends keep deriving from the new process
cwd and no host path is pre-seeded into ssh/container backends.
Fixes#106220
Slim redo of the mechanism from #106410 on top of its pick (no wrappers, no
persisted "kind" enum, no process-local flag that dies with the process):
- Futility = the SAME holder PID set has blocked >= _FTS_HOLDER_FUTILE_ATTEMPTS
(10) deferrals over >= _FTS_HOLDER_FUTILE_SECONDS (30 min); tracked as
holders_since/holders_attempts in the persisted fts_rebuild_deferral record
and reset whenever the holder set changes. The 3-deferral/60 s escalate
window is the orphan-reap gate and stays as is.
- ONE escalated ERROR line names each holder pid + cmdline and the remedy that
can actually be followed from inside a gateway session: stop ONLY the other
holder; this process's own retry admits the rebuild within 60 s. The old
"with the gateway stopped" advice was unrunnable from a gateway-hosted
session (the gateway is the session) and is gone from both log and doctor.
- hermes doctor renders the futile record distinctly.
- retry_deferred_fts_recovery: a capped backoff earned by holder set X no
longer applies once the live holder set differs from X, so stopping the
other service is followed by a retry on the next tick, not up to an hour
later (the issue's 16-min wait).
- Tests trimmed from 5 to 2 invariants (futile line + doctor entry after N
same-holder deferrals; backoff reset when the holder set changes); the
contributor's control tests for changing PIDs / orphan reap are covered by
the existing test_repeated_deferrals_reap_inactive_orphan_then_rebuild.
The "canonical writes and LIKE search remain available" WARNING is kept
because it is true on origin/main: a stale open drops every FTS trigger, so
the messages INSERT succeeds (probed live with a real state.db + a second
process holding it). Writes fail only when a peer re-publishes triggers over
the corrupt index — a separate class, not this diagnostic.
Refs #106393
A supervised peer never satisfies the orphan reap, so stale-FTS repair retried forever with a misleading "canonical writes remain available" warning.
(cherry picked from commit e57f3a975d311aa44da1e92c5e727eba7c8cff70)
Defence at the exact boundary the incident crossed: systemd_uninstall() and
uninstall._remove_systemd_gateway() unlinked whatever get_systemd_unit_path()
returned. Before stop/disable/unlink, read the unit's own
Environment="HERMES_HOME=..." line (the parser status/refresh already use)
and, when it names a different home than this process, warn with both paths
and leave the unit alone. A unit without the line (hand-written) is still
removed as before.
With the previous commit a Docker/custom root (HERMES_HOME=/opt/data) gets a
hashed host-service suffix. Three callers used `_profile_suffix() or
"default"` as the PROFILE id, which is a different question: the s6
supervisor's slot for the root home is `gateway-default` regardless of where
the root lives, and the multiplexer's "am I a named profile" probe must not
treat a hash as a profile name. Route them through hermes_constants.
profile_name_for_home() (root -> "default", <root>/profiles/<name> -> name)
with the service suffix as the fallback for unknown layouts.
_profile_suffix() compared HERMES_HOME against get_default_hermes_root(),
which treats ANY home outside ~/.hermes (Docker /opt/data, a mktemp dir) as
"the root itself". Every such home therefore collapsed to the bare
`hermes-gateway` service name and the default profile's unit path
(~/.config/systemd/user/hermes-gateway.service); the documented
"else a short hash of the path" branch was unreachable.
A parity harness run with HERMES_HOME=$(mktemp -d) called
uninstall_gateway_service(), resolved to the production unit, ran
`systemctl --user stop/disable`, unlinked it and daemon-reloaded. With the
unit gone Restart= could not revive it: all cron jobs and every messaging
platform were down for 6.5 days.
Compare against the platform-native default home (~/.hermes) for the bare
name; keep the profile name for <root>/profiles/<name>; everything else
(temp dirs, Docker /opt/data) gets its sha256[:8] suffix as the docstring
always promised. The Docker image supervises with s6 (`gateway-<profile>`
slots), not systemd/launchd, so the bare host-service name was never load-
bearing there.
_size duplicated _mtime_ns with a different attribute and the fingerprint
stat'ed each of the four files twice, so mtime and size could come from two
different versions of the file. _stat_sig returns both from one stat. The
persisted compare goes through a JSON round-trip so nested tuples match the
lists they read back as.
`_heal_forked_single_use_oauth_grants()` runs on every `load_pool()`, and its
clean mark lived only in memory. Every fresh `hermes` invocation and every new
worker therefore re-took the heal's two nested EXCLUSIVE auth-store locks just
to rediscover a store it had already cleared — 4 acquisitions per process on a
two-provider profile, every one of them finding nothing to consolidate. Behind
a sibling process holding those locks that costs a full
`AUTH_LOCK_TIMEOUT_SECONDS` per provider before the process can do anything at
all: measured 30.1s for two providers.
Persist the mark next to the store it describes (`<profile>/cache/
oauth_heal_clean.json`, 0600, paths and stat data only — no credential
material) and consult it BEFORE taking any lock.
Outliving the process means the mark needs a stronger key than the in-memory
one did:
- The ROOT store joins the fingerprint. This heal consolidates root → profile,
so root acquiring a counterpart turns a row the heal deliberately KEPT into a
fork it must strip. The in-memory mark could ignore root because it died with
the process; a persisted mark would keep skipping a heal that has become
necessary.
- File sizes join it too, so a metadata-preserving rewrite (`rsync -t`,
`tar -p`, a restore) cannot leave a stale mark looking current indefinitely
rather than for one process.
Measured on an isolated HERMES_HOME with two OAuth providers:
lock acquisitions per fresh process 4 -> 0
contended load_pool() x2 30.1s -> 0.00s
mark-file reads per 100 load_pool() - -> 2 (one per provider)
The mark stays a cache: absent, unreadable, corrupt or wrong-shaped content all
mean "unknown" and fall through to the locked heal, and a failed write only
means the next process re-runs it — the behaviour before this cache existed.
An HTTPError means the host answered — a 401 from a wrong API key must not
be remembered as "unreachable" for the next 60s, or a user who fixes the key
gets a cached empty catalog on the immediate re-probe. Connection-level
failures (timeouts, refused, DNS) are the only thing the cache records.
_probe_neg_key hand-rolled scheme/port defaulting that utils.base_url_origin
already provides; use it.
refresh_interval_seconds() honours model_catalog.ttl_minutes / legacy ttl_hours;
reading DEFAULT_TTL_MINUTES would let the snapshot and the manifest it is
filtered from go stale on different clocks for anyone who changed the TTL.
Re-derived from PR #96099 (f127ec4e) on current main; the :nitro/:floor validate hunk is omitted because main already handles routing suffixes in hermes_cli/models_validate.py.
_nous_picker_model_ids only uses the ids the Portal unions append — both
unions discard the pricing map (`model_ids, _ = union_with_portal_*`) — yet
it called get_pricing_for_provider("nous") without cached_only, so a cold
pricing cache paid a full /v1/models round-trip (network timeout on a slow
Portal) on the picker-open path for nothing. Pass cached_only=True; the
background pricing prewarm (#101685) fills the same cache for later opens.
Re-derived from #102099 by @finn763: the original patched
hermes_cli/model_switch.py, which 3b1ecfc0a1 decomposed; the live call site is
hermes_cli/model_switch_providers.py.
Based on #102099 by @finn763.
The all-profiles sidebar polls GET /api/profiles/projects/tree, the one
heavy sidebar endpoint that was not wrapped in @_sidebar_singleflight_cache.
Every poll fanned out list_profiles() + _build_project_tree() over every
profile (51 on this box, ~17k SKILL.md files walked), and each profile
resolution re-parsed its config.yaml because read_user_config_raw() ran
uncached. On a 2-vCore VPS running 'hermes serve' for the Desktop remote
backend this pinned both cores (py-spy: 110-116% sustained, HostHighCPU).
- Wrap get_profiles_projects_tree in the existing single-flight cache,
matching get_profiles_sessions_sidebar (5s TTL, errors[] not cached).
- Memoize read_user_config_raw() on a (st_dev, st_ino, st_size,
st_mtime_ns) fingerprint under _CONFIG_LOCK, same strategy as
read_raw_config(). Hits return a deepcopy so write-back round-trips keep
their fresh-dict semantics; parse errors are never cached; the docstring
no longer claims 'no caching'.
- Tests: memo semantics (deepcopy isolation, inode-replace reparse, error
non-caching, parse-count), a wrapper-presence pin for both heavy sidebar
endpoints, and a cold-cache autouse fixture in the scope tests (the 5s
TTL otherwise leaks one test's payload into the next).
Measured on the affected box: list_profiles 1.9s -> 0.16s warm, serve CPU
112% -> 22-34%, host CPU 50-70% -> ~26%.
- Pass 1 no longer materialises every classified record (full
`messages.content` included) until pass 2; `LayoutEvidence` keeps only the
capped per-position value sets (+ sessions rows for the one cross-column
invariant) and pass 2 re-streams the lost_and_found tables. A 276 MB
corrupted store no longer has to fit in memory.
- `_sentinel_holds` / `_text_shape_holds` if-ladders become rule tables.
- Tests trimmed to the three that bind behaviour (upgraded store maps by
name; verifier refuses when rows matched no layout; replayed history ends
at the current schema — the drift guard). No behaviour change; reverting
inference to "no layout" still fails the name-mapping test.
A store's physical column order depends on which schema it was created at and
which ALTER TABLE ADD COLUMNs it lived through; one hardcoded "upgraded"
layout cannot cover them. `session_schema_history` records the declared
schema of the salvaged tables over time; `reachable_physical_layouts`
replays it to enumerate every physical order a store can have. The mapper
infers the layout once per kind from the whole recovered population (one
store wrote all of them), maps cells by name, and counts records whose width
matched no layout; the recovery verifier refuses to report such a salvage as
healthy.
Rebased onto the simplified session modules (c88d60551e, b9b4600cb2,
7e5a1a11d9, 1915a0d27e); no behaviour change from the pre-rebase branch.
Upgraded state.db files gain columns via ALTER TABLE ADD COLUMN, so
physical order diverges from SCHEMA_SQL. Prefix-mapping onto the fresh
template put started_at at 0 and shifted titles/models. Insert by name
using known physical layouts; keep the plausibility gate.
Fixes#101409
- hermes_cli/plugins_cmd_catalog.py: new sibling owning resolution, the
.hermes-catalog.json provenance sidecar, search/info/validate, re-pin on
update, and the dashboard/TUI payload builders. plugins_cmd.py only
gains the hooks (cmd_install catalog branch, cmd_update / dashboard
update re-pin, dashboard_install_plugin catalog_name + kill list,
dispatch entries); the community index (plugin_index.py) is gone.
- hermes_cli/plugin_catalog.py: catalog_dir parameter replaces the
test-only HERMES_PLUGIN_CATALOG_DIR env var; live refresh reads ONE
published document (/docs/api/plugin-catalog.json, 6h cache, in-tree
fallback) instead of the unauthenticated GitHub contents API (60 req/h,
1 request per entry); in-tree and live removals are unioned so a stale
cache can never un-block.
- Catalog route lives in web_routers/dashboard_ui.py (the facade is off
limits); _plugin_runtime_status shared from web_server_dashboard.py;
hub rows carry removed_reason. TUI plugins.manage gains catalog_name
install, catalog row fields and an update action.
- plugin_validate: the probe context honours ctx.get_config defaults
(real plugins do int(ctx.get_config("timeout", 180)) in register()).
- Installed-state merge matches through the sidecar's catalog_name
first — catalog names rarely equal manifest names.
- extract-plugins.py emits plugin-catalog.json; deploy-site triggers on
plugin-catalog/** so entry merges republish it.
Re-port of the PR's version gate onto the decomposed layout: the field and
parser live in plugins_manifest.py (with running_hermes_version /
version_satisfies helpers), the load-time skip in plugins_loader.py
before any import. Unsatisfied plugins record an error and never run
register(); one invariant test proves both halves.
A global model/provider change must never stop a cron job. The #44585 guard
raised [drift_skip] for every unpinned job whose provider_snapshot /
model_snapshot no longer matched the live global default, so one `hermes model`
switch silently killed whole fleets (reported by fastfinge, nitinthewiz,
Dr-ilies; 13 of 60 jobs on the project lead's box after
claude-fable-5 -> claude-fable-5.1).
The snapshot is now the job's effective pin: _load_cron_job_config prefers
job['model_snapshot'] over the global default and _resolve_job_runtime passes
job['provider_snapshot'] as `requested` when neither a per-job pin nor a
cron.model / cron.model_provider fleet default covers the axis. One INFO line
per differing axis tells the operator what the job is running on and how to
move it. Jobs without a snapshot (legacy records) still follow the global
default; the existing fallback chain still handles a snapshot provider that
fails to resolve.
Both goals of #44585 hold: no silent inherit of a paid default (the job runs on
what it was created under) and no outage. Owner decision (Teknium): "main agent
model changing should not stop crons from executing, ever".
Removed as unreachable: _check_model_drift, DRIFT_SKIP markers, the
drift_alerted alert-once bit (mark_drift_alerted + the _record_run_outcome pop),
the drift special-cases in _compose_run_delivery and
_summarize_cron_failure_for_delivery, cron_model_drift_guard_enabled and the
cron.model_drift_guard config key (v42 migration drops it from existing
configs). The PLUGIN-COMPAT clear_drift_alerted block is untouched (scheduled
revert).
The `hermes config set model.default` notice and the Desktop model-change toast
are reworded from "will fail closed / will be skipped" to "keep running on the
model they were created under"; the impact payload drops guard_enabled (all six
desktop locales updated).
Cron jobs are scheduled work bound to the source profile and its origin
channel. A clone that inherited cron/jobs.json fired every job twice: two
gateways with identical job ids running the same weekly jobs in parallel
(double spend, duplicate deliveries) until one gateway died.
Root cause: `cron` was not in _CLONE_ALL_HISTORY_EXCLUDE_ROOT, so the
copytree in _clone_all_into carried jobs.json along. Add it to the
per-profile history exclude set (applies to any source, CLI, dashboard
and TUI/desktop RPC all funnel through create_profile), recreate the
_PROFILE_DIRS skeleton after the copy so the clone still has an empty
cron/ (and sessions/), and say so in the CLI summary line and docs.
--clone (config-only) never copied cron; export/backup keep cron as
before (an archive is a portable snapshot, not a second live profile).
Python plugin CLI/loader/web/tui files taken from main wholesale; the
catalog layer is re-ported onto main's decomposed shapes in the
following commits. plugin_index.py removed (catalog is the sole
discovery system).
run_backup() previously wrote "hermes-backup-<timestamp>.zip" on every
invocation without deleting old ones. Hourly callers accumulated 157 zips
(14 GiB). Add _prune_run_backups() to keep the newest N (default 3,
configurable via backup.run_backup_keep or --keep CLI flag).
Austin Pickett