Follow-up to the off-loop move: once the credential-pool handlers run on
worker threads, the dashboard's periodic /api/credentials/pool polls can
overlap, and during a DNS outage each poll would have started its own
exchange and abandoned its own hung resolver thread.
- Per-fingerprint threading.Lock around the exchange: concurrent callers
wait on the one in-flight attempt, then hit the positive or negative
cache (bounded worker count, no duplicate network calls).
- _urlopen_bounded: when the hard cap fires and the abandoned worker later
succeeds, close the HTTPResponse instead of leaking the socket.
- Tests (none shipped with the original PR): hard cap + late-close,
single-flight success and failure paths, and the pool endpoint running
off-loop / keeping the loop responsive under a 200 ms blocking read.
Network off (unplugged) froze the backend 17 minutes: the async
/api/credentials/pool endpoints (GET/POST/DELETE) called load_pool()
synchronously on the event-loop thread -> Copilot token exchange ->
blocking urlopen -> getaddrinfo stuck in C for 1016s, immune to
urlopen(timeout=10). WS dropped (1006), sessions detached, even log
writes stalled.
Fix 1 (web_server.py): move the three endpoint bodies into
asyncio.to_thread, matching the file's existing _run pattern.
Fix 2 (copilot_auth.py): _urlopen_bounded() runs the request in a
daemon thread with a wall-clock hard cap (timeout+5s) so DNS hangs
can no longer block any caller indefinitely.
Measured: simulated DNS hang now raises TimeoutError after 6.0s
instead of freezing the loop.
The module already binds run_in_threadpool (used by list_profiles_endpoint)
and every sibling router uses the same starlette helper; the nine new
loop.run_in_executor(None, _run) sites now go through that alias so the
file has one offload idiom. Behaviour-identical (both hand the callable to
a worker thread).
Also sweeps the one endpoint the PR left synchronous:
update_profile_model_endpoint's _write_profile_model reads and rewrites
the profile's config.yaml on the event loop.
Two assertions per offloaded site:
- a loop probe, where the stubbed callee records whether an event loop is
running in its own thread — the idiom already used by
tests/hermes_cli/test_cron_dashboard_off_loop.py; and
- a concurrency proof, where the stubbed callee blocks on a threading.Event
while an unrelated request is timed. On the unfixed handlers that request
waits out the whole block; served off the loop it returns in
milliseconds.
The concurrency proof needs a single event loop across requests, so the
client fixture enters the TestClient context manager: that pins one
blocking portal for the whole fixture, where a bare TestClient(app) would
spin up a fresh loop per request and pass even unfixed.
Also covers the status-code mapping through the executor hop (404 on a
missing profile, 400 on a rename collision, 404 from the resolve that stays
on the loop ahead of describe-auto) and the _MISSING sentinel cases: a
desktop.json holding `null` still reports exists=true, an absent one
reports exists=false, and an empty SOUL.md is still distinguishable from a
missing one.
The client fixtures read web_server._SESSION_TOKEN from the module rather
than pinning a literal. web_server resolves that token once at import, so
whichever test file imports it first fixes the value for the session and a
later monkeypatch.setenv is silently ignored — two files hardcoding
different tokens would 401 depending on collection order.
The remaining in-scope handlers in this router read and write profile
documents inline on the ASGI event loop:
- GET /api/profiles/{name}/soul reads SOUL.md
- PUT /api/profiles/{name}/soul atomic_write_text(SOUL.md)
- PUT /api/profiles/{name}/description write_profile_meta(profile.yaml)
- GET /api/profiles/{name}/desktop-overlay reads desktop.json
The persona save is the sharpest of the four: atomic_write_text() writes a
temp file, fsyncs it and replaces the original, so the loop is parked for
however long the filesystem takes to durably commit — unbounded on a slow
or contended disk, and paid on every Save in the editor.
Each handler keeps its existing status-code mapping. The reads probe and
load in a single executor hop rather than two, which also avoids widening
the gap between the existence check and the read.
Both readers return a _MISSING sentinel rather than None for an absent
file. desktop.json may legitimately contain the document `null`; collapsing
that onto None would newly report an existing-but-empty overlay as absent.
The same distinction is what the SOUL.md durability tests rely on, where
"file missing" and "file empty" must not both read as never-set.
_resolve_profile_dir() stays on the loop in all four, as it does in the
rest of this sweep: it is a name check plus one stat, and it owns the
400/404 responses.
Three more handlers in this router did filesystem work inline on the ASGI
event loop:
- PATCH /api/profiles/{name} calls rename_profile(), which stops a running
gateway through the same 10-second _stop_gateway_process() poll that
delete uses, then renames the profile directory, rewrites the Honcho
host blocks and regenerates the wrapper script.
- GET /api/profiles/active reads the active_profile state file and
resolves HERMES_HOME against the profiles root. The sidebar polls it.
- POST /api/profiles/active stats the target profile, creates the state
directory and writes active_profile via a temp file plus replace.
Rename carries the same worst case as delete and belongs off the loop for
the same reason. The two active-profile handlers are individually cheap,
but they are the routes the dashboard polls, so they are the ones most
likely to be queued behind something slower — and leaving them inline is
what made the router inconsistent with list_profiles_endpoint, which
already offloads a plain directory listing eight lines above.
The two reads in GET share one executor hop rather than taking one each.
POST /api/profiles/{name}/describe-auto called
profile_describer.describe_profile() inline. That function is a plain def;
it reaches agent.auxiliary_client.call_llm(), also a plain def, which makes
a synchronous provider request with a 60-second ceiling.
Held on the ASGI event loop that is six times the 10-second WebSocket
ready-probe threshold web_server.py records as the point where the desktop
app gives up (GH-73083). A single describe-auto on a slow or unreachable
auxiliary provider therefore takes the whole dashboard offline for up to a
minute, including the /api/ws and /api/pty sockets the desktop app and the
Chat tab run on.
Move the import and call into the default executor. _resolve_profile_dir()
deliberately stays on the loop ahead of the hop: it is a name validation
plus a single stat, and it owns the 400/404 responses that the handler's
`except Exception` would otherwise turn into a 500.
DELETE /api/profiles/{name} called profiles.delete_profile() inline on the
ASGI event loop. When the target profile has a gateway running, that call
stops it via _stop_gateway_process(), which polls the PID every 500 ms for
up to 10 s before escalating to a force kill, and then removes the profile
tree.
For the whole of that window the dashboard process serves nothing else.
web_server.py's own notes record what that costs: a stall of this length
"caus[ed] the Desktop's 10-second WebSocket ready-probe to time out
(GH-73083)", and both the desktop app and the dashboard's Chat tab drive the
agent over those WebSockets. Deleting a profile whose gateway is up is a
routine action that reliably reaches the full ten seconds — the handler's
own output announces "Gateway is running - it will be stopped".
Move the call into the default executor via run_in_executor, matching
list_profiles_endpoint, export_profile_endpoint and import_profile_endpoint
in this same module. The exception-to-status mapping is unchanged:
FileNotFoundError/ValueError are raised inside the worker and re-raised by
the await, so they still map to 404/400.
Extends the off-loop suite to the third and last blocking method on the
`UpstreamAdapter` contract, the 401/429 rotation.
As with the two existing pairs, the primary assertion is **thread identity**,
not latency: a latency assertion measured by an HTTP client on the blocked
loop is vacuous, because the client's own timer cannot advance until the
block ends and it therefore reports a fast response on provably frozen code.
* `test_get_retry_credential_runs_off_the_event_loop` records
`threading.get_ident()` inside the fake adapter and compares it to the
loop thread, and checks the rotation still works end to end (rejected
bearer forwarded first, rotated bearer second).
* `test_event_loop_keeps_running_while_the_retry_credential_resolves`
samples a loop-side heartbeat counter from inside the stalled adapter. On
the unfixed handler it records exactly 0 loop iterations across a 0.5s
rotation.
* `test_retry_credential_failure_still_returns_the_upstream_rejection`
guards the error contract the change must leave alone: a raising rotation
is still swallowed and the upstream's own 401 is streamed back, with no
second forward.
A new `_build_rejecting_upstream` harness drives the `status in {401, 429}`
branch by rejecting every bearer except the rotated one.
`handle_proxy` already offloads the two credential reads on the happy path,
but the rotation inside the `upstream_resp.status in {401, 429}` branch still
called `adapter.get_retry_credential` inline on the event loop.
That is the most expensive of the three blocking methods on the
`UpstreamAdapter` contract, not the cheapest:
* `NousPortalAdapter.get_retry_credential` routes into
`_get_credential(force_refresh=True)`, so the token-refresh POST that
`get_credential` performs only near expiry is unconditional here — and it
runs under the same `_auth_store_lock()`, a cross-process advisory lock
with a 15s timeout.
* `XAIGrokAdapter.get_retry_credential` loads the key pool off disk and
calls `try_refresh_current` / `mark_exhausted_and_rotate` under its lock.
So every upstream 401 or 429 froze the proxy's single event loop — and with
it every other in-flight streaming completion — for the whole rotation. A 429
is exactly when the proxy is busiest, which is the worst moment to stall.
Wrap it in `asyncio.to_thread`, matching the two sites above. The error
contract is unchanged: `to_thread` re-raises the worker's exception in the
awaiting frame, so the existing `except Exception -> retry_cred = None` still
swallows a failed rotation and streams the upstream's own rejection back.
Extends `test_proxy_off_loop.py` with the `/health` half, using the same
two-assertion shape as the credential tests:
- `test_is_authenticated_runs_off_the_event_loop` compares the thread the
adapter's `is_authenticated` ran on against the loop thread. Before the
fix they are the same ident.
- `test_event_loop_keeps_running_while_health_resolves_auth_state` reads a
loop-side heartbeat counter sampled by the adapter across its own stall.
Before the fix exactly 0 iterations run across 0.5s.
Both also assert the response is unchanged (`200`, `authenticated: true`),
so the offload cannot quietly alter what `/health` reports.
`handle_health` called `adapter.is_authenticated()` inline from an
`async def`. `UpstreamAdapter.is_authenticated` is documented as
"Should be cheap — no network calls. Used by `proxy start` for a clear
up-front error before binding a port." (`adapters/base.py`), and that is
true of the `proxy start` preflight, which runs in a plain synchronous
CLI function. It is not true on the event loop:
`NousPortalAdapter.is_authenticated` goes through `_read_state()`, which
takes `_auth_store_lock()` — the same cross-process lock with a 15s
timeout as credential resolution — and `XAIGrokAdapter` reads its key
pool off disk.
`/health` is precisely what a supervisor, systemd unit, container
healthcheck or load balancer polls, on a fixed interval, so it is the
endpoint least able to afford a lock wait; and a wait here freezes every
concurrent proxied stream, not just the healthcheck.
Offload it with `asyncio.to_thread`. The response body is byte-identical;
only the scheduling changes.
Adds `tests/hermes_cli/test_proxy_off_loop.py`, mirroring the harness in
`test_proxy.py`: the proxy and a fake upstream run as real aiohttp
servers on ephemeral ports under a single `asyncio.run`, guarded by
`pytest.importorskip("aiohttp")` — no pytest-aiohttp dependency.
The primary assertion is thread identity, not latency. A latency
assertion measured with an HTTP client on the blocked loop is vacuous:
the client's own timer cannot advance until the block ends, so it reports
a fast response on code that was provably frozen.
- `test_get_credential_runs_off_the_event_loop` records
`threading.get_ident()` inside the adapter and compares it to the loop
thread. Before the fix both are the same ident.
- `test_event_loop_keeps_running_while_credentials_resolve` runs a
heartbeat task on the loop and has the adapter sample its counter on
entry and exit, so the reading is taken from the loop rather than
through a client that shares it. Before the fix exactly 0 iterations
run across a 0.5s stall; after it, ~50.
- `test_credential_failure_still_maps_to_401` pins the error contract
across the change of call form. It is deliberately not in the
red-before set — it guards behaviour the fix must leave alone.
`create_app` registers `handle_proxy` as an `async def`, and it called
`adapter.get_credential()` directly on the aiohttp event loop.
`UpstreamAdapter` is a synchronous contract (`adapters/base.py` — every
method is a plain `def`), and the shipped adapters implement it with
blocking I/O. `NousPortalAdapter.get_credential` takes
`_auth_store_lock()` — a cross-process advisory lock with
`AUTH_LOCK_TIMEOUT_SECONDS = 15.0` (`hermes_cli/auth.py:110`) — reads
`auth.json` off disk, and may issue a token-refresh POST; on a terminal
`AuthError` it takes that lock a second time to persist the quarantined
state. `XAIGrokAdapter.get_credential` reads its key pool off disk.
The proxy is one process with one loop, and `handle_proxy` streams with
`sock_read=300`, so long-lived completions are the normal case. Blocking
inside credential resolution therefore freezes *every* concurrent
in-flight stream mid-token for the duration — a concurrent `hermes auth`
command holding the auth-store lock is enough to do it. Every proxied
request goes through this path.
Dispatch through `asyncio.to_thread` instead. This is a pure scheduling
change: `to_thread` re-raises the worker's exception in the awaiting
frame, so the existing `except Exception` -> 401 `upstream_auth_failed`
mapping is unchanged, and the adapters' own `self._lock` still serialises
concurrent resolutions exactly as before. Fixing it at the handler also
leaves the synchronous `UpstreamAdapter` ABC untouched, so it covers
every adapter without conflicting with in-flight work that subclasses it.
Submodule and worktree checkouts store .git as a file (gitdir: pointer);
the carry-over only looked at directory names, so that form was still lost
on rollback. Handle files with the same guard. Docstring now states the
deliberate limit: an excluded entry whose skill dir the target snapshot
lacks is dropped with staging (no orphan .git) and is not undoable via the
safety snapshot, which excludes these paths as well.
Excluding nested .git from snapshots has a side effect on rollback: the
staging move takes the whole live skill dir (including its .git) into
.rollback-staging-*, the extract restores the snapshot without it, and the
staging dir is then deleted — so a skill that is itself a git checkout lost
its .git on any rollback. Reproduced: main preserves it, the exclusion-only
branch did not.
After a successful extract, move excluded subtrees from the staged copy back
under their restored skill dir (mirroring how a top-level .git survives by
never being staged). Regression test included.
Folds the incident explanation from #91458 (@liuhao1024) into the comment on
_EXCLUDE_TOP_LEVEL so the reason .git is excluded — compounding snapshot
growth, not just rollback safety — survives next to the set.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
_safe_skills_path() is the sibling of iter_skills_files(): when a symlink in
skills/ forces a sanitized copy for mount-based backends (Docker/Singularity),
it rglob-copied the whole tree — .hub, .curator_backups, node_modules and all.
Prune EXCLUDED_SKILL_DIRS before descending, same rule as the sync generator,
so the mounted copy never carries (or walks) the bookkeeping trees either.
Replaces the three hand-copied rglob loops + post-hoc parts check with one
os.walk generator that drops EXCLUDED_SKILL_DIRS from dirnames before
recursing. Same file set as the cherry-picked fix (the test binds it), but
the walk no longer stats every file under .hub/.curator_backups/node_modules
on each 5s FileSyncManager tick.
Bench (synthetic skills tree: 20 skills + 400 .hub files + 5x8MB curator
tarballs + 50 archived files): iter_skills_files() 35ms -> 2.4ms.
iter_skills_files() walked the skills tree with a bare rglob("*"), so the
.hub download cache, .archive, curator backups, and any node_modules/.git
under a skill package were uploaded to the sandbox on every sync. The
sandbox never reads them: skill content is resolved host-side.
EXCLUDED_SKILL_DIRS is already the canonical exclusion set, honoured by
discovery and backup. Apply it to the sync path too, across all three
roots iter_skills_files() walks (local, external, project-local), and add
.curator_backups to the set.
Measured on a local install: 900 files / 67.3 MB -> 771 files / 8.4 MB.
This is not just wasted bandwidth on the SSH backend, where the oversized
payload can exceed the 120s _ssh_bulk_upload deadline and surface as the
agent hanging on every tool call.
The filter intentionally does not reuse is_excluded_skill_path(), which
also prunes references/, templates/, assets/ and scripts/ -- those hold
support files and bundled scripts the sandbox does read and execute.
Teknium