Three orchestrator failures traced through the Sep 7 gpt-6-astra campaign sessions:
1. delegation.independent_completions (new, default false). #104299 made every
ungrouped task its own completion message, so a 15-task call woke the
orchestrator up to 15 times; one chain received 132 notices and answered
130 of them with "already incorporated". A multi-task call now returns as
ONE consolidated message unless the flag is on; `group` is inert until then.
2. Queued units were killed before they started. Units of one call share a
pool slot but the executor was still sized by slots, so with 15 units live
a new unit queued behind a full pool; the stale monitor's clock ran from
dispatch, interrupted it at 450 s, and the child exited `interrupted 0.02s`
when its thread finally came up (13 such lanes in one session). The
executor now grows to the number of live units and the stall clock arms
when the runner actually starts.
3. The tool text said "do not wait or poll — just continue" without saying
that completions are delivered only BETWEEN turns. A model that never ends
its turn (one 203-minute turn, 717 API calls) never received 40 finished
results. Tool description, dispatch note and completion header now say to
finish independent work, give a one-line status, and end the turn.
Capture the exact UTC scheduled instant before either due scanning or the
external fire claim advances jobs.json. Bind it to the durable attempt
before worker handoff; manual and unclassified direct attempts stay null.
Consult any retained completed matching row, independently of stale stamps,
claim-time windows, and newer failed attempts. Preserve unknown and legacy
attempt eligibility rather than guessing that a side effect completed.
Real isolated restart probes reproduce duplicate script writes on base and
suppress them on the fix for builtin tick and provider fire. Distinct and
manual occurrences still execute. The campaign-serialized cron suite is
queued; this progressive commit preserves the verified integration step.
Credit holny's issue #104790 and guard proposal #104323; exact identity
replaces the approximation rather than importing its legacy heuristic.
Co-authored-by: holny <holny@foxmail.com>
Salvage #99095 (e7ea53074ab2b64a1530641659399d9f1bb4b435), completing one-time migration for both boolean values and using the existing storage helpers. Hydration and local toggles never edit backend configuration. Fixes#99076.
Desktop-only backends now poll curator and personal/org skill sync without another long-lived loop. Respect active turns, the actual idle threshold, and messaging gateway ownership. Credit Jackal991 for the report and candidate #95453.
Wire the existing consent-aware, profile-keyed hook registrar at agent
construction, where the correct session home is already bound. This covers
serve and TUI agent construction without a startup-only registration or a
new helper that swallows registration failures.
Live isolated serve/WebSocket probes reproduce the missing registration on
base for both write_file and terminal, then verify each consented profile
blocks its configured tool while the unapproved profile still runs normally.
Repeated alpha construction does not duplicate hook callbacks.
Slim implementation of the agent-build placement proposed in #57020;
thanks also to the profile-scoped analysis in #102691.
Co-authored-by: grimmjoww578 <willies578@gmail.com>
Keep two invariants covering empty create/cwd/save/fork, genuine content and existing-row metadata. Preserve original authorship and avoid source-only legacy pruning: an empty ACP row does not prove its owner is dead. Native ACP wire plus a local streaming model fixture verifies the first turn and nonempty fork remain durable.
Track raw task identities across an agent's turns and match them against
process owner_task_id during close. Session IDs and shared terminal keys
are not process ownership, so the old bulk cleanup missed delegated work.
Preserve parent/sibling processes and consume teardown notifications.
Move task-resource cleanup into the lifecycle mixin, add real-process
isolation regressions, and document background process lifetime.
Keep parsing, contracts, gates and persisted goal mutations in one dispatcher. Adapters retain authorization, rendering and scheduling; TUI drafting resolves the target session profile off the RPC reader. Document ACP as unsupported rather than implying a goal loop exists.
Slimmed after review: the 200K default is dropped. Children compact at the same
0.50 x window ratio trigger as their parent (500K on a 1M model). Reasons:
- the run this came from happened at 0.85 (850K); main was already at 0.50, so
the real delta against main was 500K -> 200K, not 850K -> 200K;
- a replay of the run's 22,489 logged calls (evals/postmortem, cap sweep) put
200K-400K caps within 5% of each other in cost once cache prefixes are intact,
because the write price dominates and the cap only trims read volume;
- every compaction is a chance to lose detail, and the accuracy side was never
measured; at 500K a 1M child compacts roughly never.
What stays: the reviewer's finding that the value was coerced, not validated
(YAML true -> int 1 -> a one-token trigger; "200k" -> silently off). Values are
validated: int >= 16000 enables the cap, 0/false/null/unset = off, anything else
is warned and ignored. Docs and config comment restated accordingly.
Folds the /heartbeat driver into the existing /loop poll slot (one cadence, one
try/except table) and closes the consumed-tick gap: a dispatch that raises OR is
refused by _admit_prompt_turn (returns False) releases the session claim and rewinds
the persisted fire via HeartbeatManager.abandon_fire(), so the tick stays due for
the next poll instead of advancing fire_count with no turn. abandon_fire refuses to
overwrite a pause/resume/clear that landed between claim and dispatch (mirrors
LoopManager.abandon_tick). Tests use the real SessionDB under a temp HERMES_HOME and
drive the poller loop itself; docs note the TUI/Desktop surface.
Rollback-on-failure idea credited to jerrygooch (#104011, 51df1ed4d04 / 283fa972169);
the driver placement is Halldrix's (#102118). Reported in #102056 / #103044 (Vksh07).
A background delegate_task call used to be ONE async unit: the runner joined on
every child and a single consolidated message re-entered the conversation when
the SLOWEST finished. Fifteen independent PR reviews therefore waited on the
fifteenth before the parent could act on the first.
Each task now carries an optional `group`. `_units_of` partitions the call's
children into units — one per distinct group, one per ungrouped task — and each
unit is dispatched to the async registry on its own, so its results re-enter the
conversation as soon as THAT unit is done. Tasks that must be compared or merged
share a group and still return together.
Capacity is unchanged: every unit of one call joins the first unit's pool slot
(`slot_key` in `async_delegation._dispatch`), so splitting never consumes more of
`delegation.max_concurrent_children` than the call did. Unit ids suffix the
call's id (`deleg_xxxx-1`, `-2`, …) so live transcripts stay under one dir; the
completion block names the group and notes that sibling units report separately;
`active_task_count` counts a unit's own tasks.
Portal decides routing centrally per model and returns HTTP 400 on any
caller-supplied `provider` object (live-confirmed). Drop the five "OpenRouter or
Nous Portal" claims from the provider-routing page and say plainly that the
setting is ignored on Portal now that the profile no longer forwards it.
`provider_routing.models.<model-id>` now takes the same only/ignore/order/sort/
require_parameters/data_collection keys and overlays the flat provider_routing
values whenever the agent is on that model. Resolution lives in the one
chokepoint every request path already uses (_provider_preferences_for_agent),
so CLI, gateway, TUI/Desktop, cron, /model switches, fallback activation and
delegated children on another model all honour it with no per-surface plumbing.
Matching is spelling-tolerant, sharing _canonical_model_variants with
agent.reasoning_overrides.
The OpenRouter profile's speed-tier pin no longer overwrites an explicit user
`only` on the BASE gpt-6-astra slug: the pin exists to keep default routing off
flex/fast, and a user pin is the stronger intent (only: [openai] stays [openai]
instead of becoming [openai, azure, azure/us]). Tier slugs (-fast/-flex) keep
owning `only`.
Live A/B (config only: {gpt-6-astra: [openai], claude-fable-5.1: [anthropic]}):
main sent {"sort":"price"} for fable and OpenRouter served it from Azure; with
this change it sends {"only":["anthropic"],"sort":"price"} and Anthropic serves it.
Schema proposed in #24495 (samplesabotage) and #100711 (Artemonim); this is a
slim chokepoint implementation of that design.
Co-authored-by: samplesabotage <samplesabotage@users.noreply.github.com>
Independent review: a YAML `true` coerced to int 1 and gave every child a
one-token compression trigger; "200k" silently disabled the default cap.
Values are now validated: an int >= 16000 is used, 0/false/null disable on
purpose, anything else warns and falls back to the 200K default so a typo
never costs money. Config comment and docs say it caps the compaction
trigger, not a hard request-size limit.
Test: true / "200k" / 5 -> default; 0 / false / None -> disabled; valid ints
pass.
A delegate_task child inherits the compression threshold as a RATIO of the
model window. On a 1M-window model at the run's configured 0.85 that is an
850K-token trigger: in the 1,393-agent refactor run 1,373 of 1,375 children
never compressed once, 62% of all API calls carried >150K of context, and the
calls above 200K carried ~$10.9k of the $19.3k bill (58% of it cache WRITES,
i.e. re-sending a 300-800K prefix on every call). A sawtooth replay of the
logged calls with a 200K cap / 65K floor cuts context spend by ~49% (~$7k).
Children are brief-driven and disposable; they re-read their brief and the
files they touch, so a large window buys them little. New
delegation.compression_threshold_tokens (default 200000) is applied to the
child's ContextCompressor right after construction as the lower of it and
any global compression.threshold_tokens; the parent's own trigger is
untouched. 0 disables the subagent-specific cap. The compressor applies
threshold_tokens_cap on first window resolution, so this is byte-equivalent
to the user having set compression.threshold_tokens for the child.
Live through the real spawn path (_build_child_agent, real imports, temp
HERMES_HOME, 1M-window model): main child trigger 500,000 / branch 200,000;
parent 500,000 on both.
Tests (3): default caps a 1M child at 200K; the cap is the lower of the
delegation and global values and never raises a small-window child's
trigger; 0 disables and an already-resolved trigger is re-clamped.
Docs: delegation.md, configuration.md.
Symptom: with Browser Use mode (the default) and no cloud provider / CDP
override, browser_exec left BU_CDP_* unset, so the browser-use harness ran
its own local discovery — hunting for the user's INSTALLED Chrome on its
default profile. That path needs the chrome://inspect remote-debugging
toggle plus an "Allow remote debugging?" popup per run, is blocked outright
on Chrome >=136, and on a headless host (or one without Chrome) fails with
"chrome-not-running: no supported Chromium-family browser is running". The
built-in browser_* tools never had this problem: they drive the Chromium
Hermes installs, launched through agent-browser.
Change: `_resolve_backend_cdp` step 4 becomes the local ENGINE —
lightpanda when configured, otherwise `_resolve_managed_chromium_cdp`,
which runs `agent-browser --session <key> get cdp-url` through the legacy
`_run_browser_command` (Chromium preflight/auto-install, per-key session
cache, inactivity reaper, atexit) and exports the returned ws:// endpoint
as BU_CDP_WS. Each cache key (task, or bu-named-<session>) gets its own
Chromium, so named sessions are private and skip the own-tab preamble.
Running `get cdp-url` on every call also refreshes agent-browser's idle
timer (it never sees the harness's direct CDP traffic) and follows a
relaunch after the reaper closed the browser.
Live: before, browser_exec on origin/main → exit 1 "chrome-not-running";
after → new_tab/page_info succeed on a cold start, warm reuse, post-reap
relaunch, and a named session; a missing Chromium surfaces the install
hint instead of the harness's Chrome hunt.
The skill review fork, the combined memory+skill review, and the curator's
consolidation pass all described references/ as the place for
"session-specific detail", and the curator's demote step said to move a
sibling's file under the umbrella. Followed literally over months that
produced one dev skill with a 100k SKILL.md dense in PR numbers and 443
one-per-session reference files, plus five sibling skills restating the
same rules and the repo's AGENTS.md.
The three prompts now share one shape contract: an entry is an imperative
rule plus one clause of why, stated once; no PR/issue numbers, dates, or
quoted chat as content; references/ is a small topical set extended in
place, never a per-session file; skills do not restate always-loaded
context. Consolidation is defined as distilling, and copying a sibling
verbatim under references/ is named as the failure. skill_manage's schema
carries the one-sentence version.
Two advisory linter rules make the shape visible in the tool result the
moment it starts to drift: incident-log-shape (PR/issue-number density in
prose) and references-sprawl (>60 reference files), the latter also run
on references/ writes. On the real before/after: the old skill trips both,
the consolidated one trips neither.
Adds plugins/web/perplexity — a keyed-only WebSearchProvider over httpx:
- search: POST https://api.perplexity.ai/search (documented Search API),
search_context_size=low so `snippet` stays description-sized;
results[].snippet -> description, max_results capped at the API's 20.
- extract: POST /sdk/content/snippets — the query-relevant page-excerpt
route behind `pplx content snippets` (the CLI's `content fetch` is
deprecated upstream). web_extract has no query, so the URLs' path words
serve as the relevance query; per-URL `error` entries survive a 200.
- Wired into the same touchpoints as the other keyed vendors: legacy
backend set + credential ladder + availability probe (web_tools),
registry preference walk, OPTIONAL_ENV_VARS, `hermes config`/status/
dump key lists, nous_subscription direct-credential detection, setup
summary, test conftests, docs.
Not a keyless-ring member (Perplexity has no anonymous tier). Related
closed PRs #9192 / #23981 / #45225 predate the plugin ABC.
Multi-root servers (pyright) are keyed by server_id; a file whose resolved
root is new for a running client is attached with
workspace/didChangeWorkspaceFolders instead of spawning another server.
Single-root servers keep the (server_id, workspace_root) key and behavior.
A profiled fan-out across ~30 worktrees ran 30-60 pyright processes
(~8.7 GB); the same fan-out now runs one.
Salvage follow-up for PR #100269. The flag landed upstream in 0.3.x;
binaries before it (e.g. 0.2.8, verified locally) fatally reject the
flag with 'unknown argument', breaking every Browser Use launch.
Probe 'lightpanda help' once per process and omit the flag when the
binary predates it. Also soften the unverified concurrency claim in
the _http_cache_dir docstring and tell docs readers to stop sessions
before deleting the live sqlite cache.
Lightpanda's HTTP cache is opt-in (`--http-cache-dir`, off by default), and
the launcher never passed it, so every Browser Use navigation re-fetched
every asset.
Point all Hermes-spawned instances at one shared cache under
$HERMES_HOME/cache/browser-use/lightpanda/http-cache. Sharing it across
sessions keeps assets warm through session churn; Lightpanda stores it in
sqlite (WAL), so a write that loses a race degrades to a cache miss rather
than a failed load, and --http-cache-entry-limit (default 1000) bounds the
directory without Hermes managing eviction.
Measured over 25 navigations across 5 sites, median warm navigation drops
from 0.40s to 0.18s on news.ycombinator.com and 0.14s to 0.10s on
wikipedia; total navigation time 7.0s -> 5.9s.
The flag has existed since Lightpanda 0.3.x (April 2026), so this needs no
minimum-version bump.
Expose the read deadline as a top-level config.yaml key beside
context_file_max_chars (same load_config_readonly resolution shape), default
5s, documented in context-files.md. Narrow the reader thread's catch from
BaseException to Exception: control-flow exceptions can't originate inside
read_text on a worker thread, and re-raising one would bypass the sites'
except Exception / except (OSError, UnicodeDecodeError) handlers.
N processes sharing one Nous OAuth pool entry hit the hourly expiry
together; each force-refreshed, each rotation invalidated the token a
sibling had just adopted, and processes that lost the auth-store flock
race had their only entry benched ("matched no nous entry ... pool size
0") — ~120 sessions surfaced 401 'out of funds' on Sep 2 2026.
- resolve_nous_runtime_credentials(stale_access_token=): under the store
lock, skip the refresh POST when the on-disk token differs from the one
that failed and is usable (a peer already rotated) — adopt instead.
- credential_pool nous path: adopt a peer-rotated key after the pre-sync,
pass the failed bearer through, and treat a lock TimeoutError as
'retry later', never as an exhausted credential.
- Live 120-process stampede harness: 41 refreshes/9 unrecovered -> 1
refresh/0 unrecovered.
Policy: availability-gated tools (check_fn probes — Docker, HASS_TOKEN,
OAuth…) are frozen for the life of a session. tools[] only changes on
/new, /reload-mcp, or compaction. Two doors remained after #100638:
* Gateway agent-cache eviction (LRU/idle sweep/cross-process invalidation)
rebuilds a fresh AIAgent for the SAME session and agent_init re-derives
agent.tools from live probes with no predecessor to preserve. Persist
the session's resolved tool-name order in a new `sessions.tool_names`
JSON column (declarative reconciliation, SCHEMA_VERSION 28), written
alongside the system prompt and re-pinned on every published refresh
(so /reload-mcp and compaction naturally reset it; /new mints a new
row). On restore-for-existing-session the fresh definitions are folded
onto the saved order via the SAME `_merge_preserving_prefix` helper —
a probe-flipped tool is carried forward from the registry schema, a
deregistered one dropped, new tools appended at the tail.
* /reload-mcp (CLI, gateway, TUI RPC) now also calls
`reprobe_tool_availability()` — drops the check_fn verdict cache and the
get_tool_definitions memo — so a user can consciously pick up a
credential/daemon that appeared mid-session. Docs updated.
Under a multiplexed gateway every profile's outbound webhooks share one
delivery worker, so receivers could not tell which profile fired an
event. Add a top-level `profile` field to the payload, resolved at fire
time from the bound Hermes home via get_active_profile_name() ("default"
outside profiles). Documents the field in the wire-format section.
Reported by @vszgdcn8cj-ctrl.
Fixes#92674
Port the run-ownership invariants from PR #93747 onto main's `_run_owners`
model in gateway/platforms/api_server_runs.py:
- `_request_owns_run` no longer admits run state that exists without an
owner stamp. Under gateway.multiplex_profiles every served profile holds
a valid key, so the "backward compatibility" branch made the boundary
allow-all whenever provenance was missing. Unstamped state now fails
closed; only an in-memory owner match or a durable idempotency record
under the caller's own scope admits a run.
- POST /api/sessions/{id}/chat/stream claims `_run_owners` at the run mint,
inside the request's profile scope, so its run is confined to the
creating profile like /v1/runs.
- Owner release is tied to "no run-keyed state survives"
(`_release_run_owner_if_forgotten`) and runs at every retirement point
(task finally, SSE stream close, both sweep loops, chat-stream finally),
not only the terminal-status sweep — no stranded entries, no stateful id
ever left unowned.
Docs: note that runs are per-profile scoped (replaces the now-false
visibility admonition proposed in PR #92822).
Fixes#93689Fixes#90415
Supersedes #93747, #93704, #92822
Co-authored-by: RickyYii <237135932+RickyYii@users.noreply.github.com>
Co-authored-by: liuhao1024 <11816344+liuhao1024@users.noreply.github.com>
Extends the TTS lease from #100912 (4e3feb8bbb) beyond built-in local
engines: when the configured tts.provider is user-declared, acquiring
the first lease and releasing the last one now reach it, so a
self-hosted TTS server can preload its model when read-aloud / voice
conversation turns on and unload when it turns off (Discord request).
- agent/tts_provider.py: TTSProvider gains concrete no-op warm() /
release() (not abstract — existing plugins are unaffected).
- tools/tts_tool.py: _signal_user_tts_provider() forwards the lease
hook; plugin providers get warm()/release(), command providers run
optional `warm_command` / `release_command` (config.yaml, under
tts.providers.<name>) through the existing _run_command_tts helper
on a daemon thread — best-effort, output discarded, failures at
debug. warm_tts_provider() and release_tts_provider() call it.
- tests/tools/test_tts_lifecycle_leases.py: fake plugin provider and
fake command provider observe warm/release through acquire/release
lease (both fail on main with action == "noop").
- docs: features/tts.md — lease section, command-provider optional
keys table, plugin optional hooks.
Teknium