The API server intentionally lets concurrent runs share a client-provided
session_id (= process task_id), so the SSE-disconnect reap could kill a
process a still-live concurrent run spawned after the disconnecting run's
baseline — the same stale-reaper bug class the gateway path gates via
run_generation.
- Per-task-id run epochs (monotonic counter): each run claims the epoch at
publish; a reaper holding a superseded epoch declines to kill. A missing
entry (the run's own clear pruned it) still reaps, so the leak fix isn't
silently disabled.
- _publish_turn_process_ownership / _clear_turn_process_ownership helpers
replace the copy-pasted marker set/clear blocks, so attribute names and
epoch bookkeeping can't drift between surfaces.
- /v1/runs — the third own-lifecycle surface — now records ownership and
reaps on POST /v1/runs/{id}/stop and on server-side SSE cancellation,
closing the remaining sibling paths of #76115.
Two follow-ups from review of the salvaged fix:
- _reap_gateway_turn_processes now returns 0 for a blank task_id.
ProcessSession.task_id defaults to empty for sessionless callers, so a
blank turn id would match (and kill) every unrelated empty-task process
instead of the turn's own.
- The asyncio poll loop checks executor completion BEFORE the watchdog's
timeout flag. When both race in the same window, the completed run has
already persisted its real reply to session history; surfacing the
'agent inactive' diagnostic would contradict the stored transcript.
This matches _abandon_timed_out_gateway_turn's own worker-done-wins
tiebreak.
dbbb10d39 shipped without direct test coverage for its own new logic
— the same gap teknium's review flagged on the competing PR. Close it:
- _reap_gateway_turn_processes: skips when is_still_current() is
False, proceeds when True, fails open (reaps) if the check itself
raises rather than silently disabling the leak fix.
- _abandon_timed_out_gateway_turn: still marks the turn abandoned
(interrupt fires) even when the reap itself is skipped.
- api_server._reap_disconnected_agent_processes: reaps the
baseline-diff for an owned turn, no-ops when the agent never
recorded ownership markers.
- APIServerAdapter._run_agent: markers are populated with the right
task_id/baseline during the turn and cleared once it completes,
closing the same race window fixed in gateway/run.py for this
separate agent-lifecycle surface.
Addresses the hermes-sweeper review on #76188:
1. task_id is session-scoped (task_id == session_id), not turn-scoped,
and the reap runs on a detached thread. A replacement turn could
claim the same session and spawn a legitimate process before the
previous turn's reaper thread actually enumerates its targets,
killing that new process by mistake.
Fixed by gating the reap on the existing run_generation mechanism
(_is_session_run_current) instead of inventing a new ownership
token: the timeout path captures its own run_generation at turn
start, the interrupt path captures the generation immediately after
invalidating it. If a newer turn has since claimed the session, the
reap is skipped — that newer turn owns its own baseline, so nothing
is left permanently unreaped.
2. gateway/platforms/api_server.py's SSE handlers for chat-completions
and the /api/sessions responses endpoint run their own agent
lifecycle via _run_agent() and never passed through TurnRunner, so
client-disconnect abandonment there had no baseline and no reap —
contradicting the PR's stated disconnect coverage. Both disconnect
handlers now snapshot/reap through the same
tools.process_registry primitives, via a small
_reap_disconnected_agent_processes() helper shared by both call
sites.
An agent turn can spawn a long-running background subprocess (e.g.
`next build`) and later be abandoned via inactivity timeout, /stop,
/new, or a client disconnect. Before this fix the gateway interrupted
the agent loop but never touched the subprocess: it kept running
inside the gateway's cgroup, unbounded, until memory pressure starved
the event loop and made every platform/cron look hung (#76115).
The process registry already knew how to kill a process tree — the
missing piece was per-turn ownership: nothing distinguished a process
that predates the turn (must survive), a process the turn started and
finished successfully (must survive), and a process an abandoned turn
left running (must be reaped).
- tools/process_registry.py: snapshot_running_ids() captures a turn's
starting baseline; kill_started_since() reaps only IDs created after
it, scoped to one task_id.
- gateway/turn_context.py: TurnContext carries process_task_id +
process_baseline so the timeout/interrupt paths can reach them.
- gateway/run.py: baseline is snapshotted right before the turn's
executor task starts; the inactivity-timeout path and the explicit
/stop|/new|disconnect interrupt path both reap via the same helper.
A daemon-thread watchdog backs up the asyncio-based timeout poll,
since a starved event loop is exactly the failure mode this bug
causes. The turn's own worker clears its ownership markers the
instant it finishes, closing a race where a /stop landing right
after normal completion could reap a background process the turn
deliberately left running.
Related but insufficient on their own: #37454 (cgroup ExecStopPost
reaper only fires on service restart) and #68915 (orphaned-pipe
grandchild detection, a registry bug not a turn-lifecycle gap).
Neither ties process cleanup to turn abandonment.
- logger.warning when the 6h ceiling stops the heartbeat (matches the
delegate_task stale-stop precedent) so the eventual watchdog reap is
explainable from logs instead of silent
- new mutation-checked test: past the ceiling the heartbeat stops while
the job still completes
- clearer assertion messages (surface res on failure)
- heartbeat loop continues past a raising activity callback instead of
silently stopping (matches delegate_task / touch_activity_if_due
swallow-and-continue semantics) — one transient error must not drop
watchdog protection for the rest of a long job
- hard 6h elapsed ceiling so a wedged job under HERMES_CRON_TIMEOUT=0
(unlimited child watchdog) cannot mask the gateway watchdog forever
- public get_activity_callback() accessor in tools/environments/base.py
instead of importing the private _get_activity_callback cross-module
- tests: deterministic heartbeat test (event-gated, no timing sleep),
no-callback test now asserts the thread is truly never created, new
exception-survival guard; dead started event removed
- fix comment: delegate_task heartbeat cadence is 30s, not 10s
/simplify-code review found _poll_for_token has a second caller:
web_server._nous_poller (dashboard/desktop device login), which surfaces
str(e) as the UI error_message — so wrapping only in
_nous_device_code_login left the dashboard showing the bare timeout.
Move the enrichment into _poll_for_token's deadline raise so every
caller inherits the guidance, and drop the now-redundant try/except
wrap in the CLI login. Add a source-level regression test driving the
real poll loop (authorization_pending stub client) to the deadline.
A bare 'Timed out waiting for device authorization' gives the user
nothing to act on. The most common cause is Portal sign-in failing in
the opened browser tab (including the server-side CAPTCHA loop from
issue #20605), so point at the Portal login page and the hermes portal
retry command.
Salvaged from PR #75290 by @HexLab98 (timeout-guidance kernel only).
The URL-rewrite portion of that PR was dropped: the live Portal has no
/device route (verified 404 with a real user_code), so rewriting the
manage-subscription verification URL would break login entirely.
Guidance text reworded to reference only real URLs.
/simplify-code findings on the salvage stack:
- extract _category_skill_dirs() as the single category detector; the
install guard and hermes_cli._existing_categories() now share it
(third copy of the heuristic eliminated)
- filter rglob hits through is_excluded_skill_path so vendored /
support-dir SKILL.md files (node_modules, references/pkg) no longer
misclassify a plain directory as a category and block install
- fix inaccurate WHY comment (lock-file check only guards hub-installed
skills, not hand-authored dirs), drop underscore prefixes on locals,
fold the file-collision guard under the single exists() check
Follow-up to the salvaged #76000 guard:
- refuse installing a skill INTO an existing skill directory (hybrid
skill-plus-category dirs whose later update/uninstall rmtree would
destroy the nested skill — sibling case of #75983)
- refuse a stray regular file at the install path with the caller's
ValueError contract instead of an uncaught NotADirectoryError
- regression tests: nested-only category (skills at depth >= 2),
category-inside-skill, file collision
The salvaged hydrate_profile_secret_sources (#74549) seeded its
profile-local env from <home>/.env only, but the documented 1Password
bootstrap flow puts OP_SERVICE_ACCOUNT_TOKEN in the gitignored
<home>/.op.env (mirrored from load_hermes_dotenv). A cold profile using
that flow still failed 1Password hydration — the one unaddressed item
from the sweeper review on #74549. Seed .op.env via setdefault so .env
values win; never touches os.environ. Two regression tests.
Follow-ups on the #75382 salvage (review findings):
- _wenv/_get_wsecret now catch UnscopedSecretError and fall back to
os.getenv for the DEFAULT profile's adapter, which constructs and sends
outside any _profile_runtime_scope under multiplexing — a bare
get_secret would crash its WhatsApp path (fixing one profile by
breaking another). Same pattern as Slack SLACK_APP_TOKEN (#59739) and
the Matrix recovery key. Scoped misses still return the default — no
cross-profile borrow.
- bridge_env overlay extended to the full WHATSAPP_* set bridge.js
consumes (DEBUG, FORWARD_OWNER_MESSAGES, REPLY_PREFIX,
MAX_MESSAGE_LENGTH, CHUNK_DELAY_MS, SEND_TIMEOUT_MS).
- Removed the always-true conditional on WHATSAPP_MODE injection.
Fix#75349
Root cause:
Under multiplex_profiles, secondary profiles run inside
_profile_runtime_scope which installs a per-profile secret scope via
set_secret_scope. The WhatsApp adapter (and the shared
WhatsAppBehaviorMixin + Cloud API adapter) read WHATSAPP_MODE,
WHATSAPP_DM_POLICY, etc. via raw os.getenv(), bypassing the secret
scope. Since os.environ doesn't contain secondary profile .env values,
the bridge silently falls back to 'self-chat' and rejects all inbound
messages with self_chat_mode_rejects_non_self.
Fix:
- Add _wenv() helper in adapter.py that reads WHATSAPP_* vars through
get_secret() (agent.secret_scope), which honors the active scope.
- Replace all os.getenv('WHATSAPP_*') calls in adapter.py,
whatsapp_common.py, and whatsapp_cloud.py with get_secret()-based
equivalents.
- Inject resolved WHATSAPP_* values into the bridge subprocess
environment so the Node.js bridge (which reads process.env) sees the
profile's own configuration.
Changes:
- plugins/platforms/whatsapp/adapter.py: 37 lines (+ helper, bridge_env
injection, 2 os.getenv→_wenv)
- gateway/platforms/whatsapp_common.py: 13 lines (6 os.getenv→_get_wsecret)
- gateway/platforms/whatsapp_cloud.py: 21 lines (9 os.getenv→_get_wsecret)
- New regression test: 6 test cases covering scope isolation, fallback,
and cross-profile non-leakage.
The Matrix adapter read MATRIX_RECOVERY_KEY via os.getenv, so under
gateway.multiplex_profiles every profile resolved the default profile's
key. That produced "recovery key verification failed: Key MAC does not
match" and broke E2EE for secondary profiles (#69090).
Route the read through agent.secret_scope.get_secret, which honors the
active profile's scope, with an os.getenv fallback for an unscoped read
under multiplex (default-profile startup loop) — mirroring the Slack
app-token pattern (#59739). Applied to both the startup verification
site and the status diagnostic.
Fixes#69090
Multiplexed gateways resolve credentials through the fail-closed
per-profile secret scope (agent/secret_scope.py, Workstream A): any
get_secret() read outside a set_secret_scope(...) block raises
UnscopedSecretError. The agent turn installs the scope via _run_agent's
profile-scoping wrapper, but slash-command dispatch does not — so manual
/compress reached provider resolution unscoped and every invocation on a
gateway.multiplex_profiles: true deployment failed with:
Manual compress failed: get_secret('OPENROUTER_BASE_URL') called with
no profile secret scope active while multiplexing is on.
Same bug class as the cron scheduler (#57692) and the /v1/runs agent
path — an un-migrated call site the fail-closed design is meant to catch.
Two changes, both required:
- _handle_compress_command becomes a profile-scoping wrapper around the
existing handler (renamed _handle_compress_command_inner), mirroring
_run_agent: gated on multiplex_profiles, resolves the source profile's
home and runs the whole handler inside _profile_runtime_scope. Covers
the coroutine-side read (_resolve_session_agent_runtime).
- The compressor call switches from a bare loop.run_in_executor(None, …)
to the existing _run_in_executor_with_context helper, so the scope
contextvar survives the thread hop into _compress_context, where the
aux-client provider resolution reads credentials.
Single-profile gateways take the pass-through branch — zero behavior
change (pinned by test).
Tests: 2 added (scoped read inside the executor under fail-closed
multiplexing reproduces the field failure pre-fix; single-profile
pass-through). 162 gateway compress/multiplex-scope tests green.
The Playwright suite fails identically on every PR regardless of diff
(verified on a Python-only PR and a docs-only PR): the mock-backend
Electron window never gets a title, so boot/chat/setup/interim specs all
fail; only the dead-backend boot-failure path still passes. Breakage
window matches the Aug 1 night engines/npm churn (#76499/#76562/#76575).
Gated with 'false &&' in the job condition — delete that to re-enable.
Root-fix + re-enable tracked in #76627 (Ari).
- Merge _aux_free_only() + _aux_openrouter_model() into single
_aux_openrouter_settings() that reads config once via
load_config_readonly (avoids double deepcopy).
- Remove 15-line block comment and 5-line inline comment that
restated what the code already says.
- Trim module docstring from 10 lines to 3.
- Update test patches to target load_config_readonly.
Desktop edit / regenerate / restore-checkpoint send truncate_before_user_ordinal
on prompt.submit. The handler rewrote session['history'] first, then called
replace_messages, and on failure only printed to stderr and still started the
turn. When the durable write fails, in-memory history is already truncated
and history_version is bumped while state.db still holds the pre-edit tail.
The agent flush is append-only for history-dict identities, so the new
exchange is appended on top of the 'undone' turns — durable zombie history.
Fix: persist first; only then mutate memory. On failure return 5008 and
leave memory/DB unchanged.
Based on #72876 by @necoweb3. Adapted: the handler moved from
tui_gateway/server.py to tui_gateway/methods_prompt.py since the PR's base.
Start from main's 13 tests (renamed test_openai_available_reflects_key
to test_openai_available_reflects_audio_key_resolution, added 4 new
tests for xai oauth, elevenlabs secret resolver, openai configured
key, stream cap). Append 12 new regression tests from PR #71084 for
the prefetch pipeline, PCM misalignment, and PortAudio resilience.
Patch platform.system in stream-path tests for main's macOS guard.
Replaces the synchronous per-sentence streamer.stream() loop in
stream_tts_to_speaker with a per-sentence prefetch pipeline. Each
sentence gets its own background thread that fires the HTTP request
immediately, buffering PCM chunks into a per-segment queue (capped at
3 concurrent via semaphore). A single playback worker drains segments
in FIFO order with PortAudio error recovery (reinit up to 3 attempts,
then temp-file fallback). Also fixes PCM chunk alignment for odd-byte
HTTP chunks and increases the worker join timeout from 30s to 300s.
Salvage of PR #71084 onto current main.
Closes#71084
Follow-ups from review of #76113:
- Extract cache_ttl_means_disabled() as the single disable-synonym
predicate; agent_init and prompt_caching_disabled_from_config both use
it so the two detection sites can no longer drift (drift would recreate
the #76085 bug class).
- Mirror _run_reference's not-None injection guard in
aggregate_moa_context (stamping None was a harmless no-op copy).
- Replace a vacuous trailing test assertion with the intended
input-non-mutation check; drop a stray blank line.
- Add a predicate-parity regression test (unknown TTL values keep
caching enabled, matching historical agent_init semantics).
Absorb the useful deltas from the parallel #76121 approach: a single
blank_cache_policy_stub factory so _cache_disabled cannot be left off
hand-rolled SimpleNamespaces, and pin the live agent disable onto MoA
advisor fan-out and one-shot aggregate_moa_context decoration so those
paths track conversation state rather than a fresh config re-read.
Keeps the earlier tri-state prepared-aggregator no-agent fix. Adds
factory and synthesis/advisor regressions.
Coordinates with #76121 / #76085.
Co-authored-by: JoaoMarcos44 <87440198+JoaoMarcos44@users.noreply.github.com>
Prepared-aggregator facades built via __new__ lack _agent. Accessing
self._agent raised inside the planner try and bool-coercion of a missing
snapshot forced False, suppressing config fallback for cache_ttl=off.
Pass a tri-state value and add a no-agent/config-off regression.
Blank SimpleNamespace stubs used by MoA decoration and
plan_cache_sections_for_destination never set _cache_disabled, so
anthropic_prompt_cache_policy re-injected cache_control markers after
operators turned caching off. Stamp the disable onto those stubs from
an explicit flag or the live config, and pass the agent flag from the
MoA aggregator path.
Fixes#76085
Extend phone/JID alias matching to whatsapp_cloud and treat a removed
allowlist env key as empty so sole-entry revoke cannot revive a stale
adapter snapshot.
Track which source seeded the DM allowlist so live intake does not let a
stale env carrier override explicit config, while env-seeded adapters still
reread pairing mutations.
After cleaning up an expired cloud browser session, re-check under
_cleanup_lock whether another thread has already created a replacement.
If so, return the live replacement instead of falling through to create
yet another session (which would be orphaned, or worse, a second
concurrent cleanup could destroy the replacement).
Follow-up to helix4u's expired-session renewal fix in PR #76356.
`hermes desktop` still failed with EBADENGINE demanding Node >=26 after
#76562, on a machine whose `apps/desktop/package.json` already said
`^20.19.0 || >=22.12.0`. #76562 fixed the manifest but not its mirror in
`package-lock.json`, and `npm ci` reads engines from the lockfile:
package.json apps/desktop -> {'node': '^20.19.0 || >=22.12.0'}
package-lock apps/desktop -> {'node': '>=26.0.0'} <- what gated
Chasing that exposed a second, pre-existing problem: the floor #76562
declared was too generous. Running the real `npm ci` against the whole
workspace on Node 22.21.1 fails on a transitive dependency —
npm error notsup Not compatible with your version of node/npm:
react-router@8.3.0
npm error notsup Required: {"node":">=22.22.0"}
react-router 8.3.0 (a direct dependency of both `apps/desktop` and `web`)
declares `>=22.22.0`, which is tighter than Vite's `^20.19 || >=22.12` and
excludes all of Node 20. So `>=20.0.0` promised support the tree cannot
deliver: an install on Node 20 or early 22 passed the installer's gate and
then died inside `npm ci` on someone else's package.
All four engine declarations now state the floor the dependency tree
actually has, `>=22.22.0`: root `package.json`, `apps/desktop/package.json`,
and both of their `package-lock.json` mirrors. The installer gates move with
them (`node_satisfies_build` in install.sh, `Test-NodeVersionOk` in
install.ps1) so a too-old system Node is replaced with the managed one
*before* npm runs, and the failure a user does see names hermes-agent rather
than a transitive package. NODE_VERSION stays 22 — latest-v22.x is 22.23.2,
comfortably above the floor.
The invariant test gains the case that would have caught the mirror drift on
its own: the desktop assertion now pins the tightest floor a dependency
actually declares, and the managed-runtime check compares majors, since
install.sh fetches latest-v{major}.x rather than {major}.0.0.
Verified with real `npm ci --dry-run` over the full workspace:
- node 22.23.2 (what install.sh provisions) -> 1258 packages
- node 26.5.1 -> 1189 packages
- node 22.21.1 (below the floor) -> EBADENGINE naming
hermes-agent, i.e. our own manifest, not react-router
kshitijk4poor