Salvage follow-up for PR #100269. The flag landed upstream in 0.3.x;
binaries before it (e.g. 0.2.8, verified locally) fatally reject the
flag with 'unknown argument', breaking every Browser Use launch.
Probe 'lightpanda help' once per process and omit the flag when the
binary predates it. Also soften the unverified concurrency claim in
the _http_cache_dir docstring and tell docs readers to stop sessions
before deleting the live sqlite cache.
Lightpanda's HTTP cache is opt-in (`--http-cache-dir`, off by default), and
the launcher never passed it, so every Browser Use navigation re-fetched
every asset.
Point all Hermes-spawned instances at one shared cache under
$HERMES_HOME/cache/browser-use/lightpanda/http-cache. Sharing it across
sessions keeps assets warm through session churn; Lightpanda stores it in
sqlite (WAL), so a write that loses a race degrades to a cache miss rather
than a failed load, and --http-cache-entry-limit (default 1000) bounds the
directory without Hermes managing eviction.
Measured over 25 navigations across 5 sites, median warm navigation drops
from 0.40s to 0.18s on news.ycombinator.com and 0.14s to 0.10s on
wikipedia; total navigation time 7.0s -> 5.9s.
The flag has existed since Lightpanda 0.3.x (April 2026), so this needs no
minimum-version bump.
With servers configured but none selected by -t/--toolsets, discovery still
paid the ~260ms mcp SDK import before discovering it had nothing to spawn.
Filter first; the test now asserts the SDK probe is not reached.
The cherry-picked commit added the allowed_mcp_names filter to
discover_mcp_tools(). Since then CLI startup grew a second discovery path —
start_background_mcp_discovery / the deferred desktop start in
hermes_cli/mcp_startup.py — so wiring the filter only into the inline call
would leave `hermes chat -t terminal` (the default backgrounded path) still
spawning every server.
Store the filter once in mcp_startup (set_mcp_server_filter, called from
_prepare_agent_startup from args.toolsets; `all`/`*`/empty clears it) and
have both the inline and the background discovery honor it. The unfiltered
call shape is unchanged so zero-arg test stubs keep working.
Dropped from the original PR: the atexit/SIGINT/SIGTERM oneshot MCP reap —
main already does this in _cleanup_oneshot_runtime() -> shutdown_mcp_servers().
E2E (3 configured stdio servers, real subprocesses, 5 runs median):
no filter 3 spawned / 2.0 s; `-t terminal` 0 spawned / 1 ms.
Guards the off-loop assertion against becoming vacuous if run.py ever hoists
the loops imports out of the watcher body (the class/module patches would then
be bypassed and on_loop_calls would be trivially empty).
The writer-lock test exits at the no-adapter branch, so it only covered the
first of the three off-loop calls. Add a full-path test (adapter present,
slash-command loop) asserting each SessionDB call runs off the loop thread;
with main's run.py it fails naming all three.
Follow-up to the two cherry-picked commits: use _run_in_executor_with_context
(the file's established off-loop helper — keeps the multiplex HERMES_HOME
override alive in the worker thread, same as the warm-up above) instead of a
bare run_in_executor(None, ...), drop the now-stale rationale that blamed
list_meta_prefix's writer lock (main reads via _read_ctx), and add a
regression test that holds the SessionDB writer lock from another thread
during one watcher scan and asserts the event loop keeps ticking. On main
the same test measures a 0.53 s loop stall for a 0.6 s hold.
- Route list_meta_prefix() through _read_ctx() instead of self._lock,
fixing contention at the source for every caller (CLI, TUI, gateway),
not just this one call site
- Wrap fire_tick() and complete_tick() in run_in_executor() to avoid
blocking the event loop on writes (BEGIN IMMEDIATE) that acquire
self._lock
- Use asyncio.get_running_loop() instead of get_event_loop() to match
established convention in the file
These changes fix the actual freeze case (when a loop IS due and writes
happen), not just the common case (when no loop is due).
list_active_loops() calls list_meta_prefix() which acquires self._lock,
and that lock is also held by writers doing BEGIN IMMEDIATE and periodic
FTS5-merge/WAL-checkpoint work. A slow write holding the lock while the
watcher blocks the entire event loop on the same lock froze it for 90+
seconds until the liveness watchdog force-exited.
Moving list_active_loops() to run_in_executor() prevents the event loop
from blocking on the SQLite lock acquisition.
Fixes dynamic-hermes-agent#130
Expose the read deadline as a top-level config.yaml key beside
context_file_max_chars (same load_config_readonly resolution shape), default
5s, documented in context-files.md. Narrow the reader thread's catch from
BaseException to Exception: control-flow exceptions can't originate inside
read_text on a worker thread, and re-raising one would bypass the sites'
except Exception / except (OSError, UnicodeDecodeError) handlers.
Context files (SOUL.md, .hermes.md/HERMES.md, AGENTS.md, CLAUDE.md,
.cursorrules, .cursor/rules/*.mdc) and subdirectory hint files are read on
the startup / per-turn path. On network-backed filesystems (iCloud Drive,
OneDrive, NFS) a cold read of an evicted file can block indefinitely, which
stalls system-prompt assembly before the first turn.
Read them on a daemon thread with a 5s deadline via _read_text_with_timeout;
a timed-out file is logged at WARNING and skipped so the next context source
still loads. Read errors propagate exactly as before, so each site's
existing try/except handling is unchanged.
Re-cut against current main from PR #10110 (the original diff predates the
context_length plumbing and the AGENTS.md directory-chain loader); helper,
sites and regression tests follow the original.
Follow-ups from review of the two salvaged #87490 commits:
- _pressure_with_real_floor now applies only on the rough fallback branch.
A valid usage anchor is provider-exact and wins as-is: on MoA turns the
anchor deliberately uses the pre-fold aggregator usage while
last_real_prompt_tokens holds the folded figure, so flooring the anchored
value would re-add fan-out tokens the anchor exists to exclude. Docstring
rewritten to describe the real path split (anchor since d3a1c46510).
- estimate_tokens_rough: encode with errors="replace". main's estimator
never raised; text.encode() on a lone surrogate (routine in tool output,
see message_sanitization) raised UnicodeEncodeError and would abort a
turn where main produced a slightly-off number.
- Record the cl100k/o200k/Qwen2.5 calibration for the bytes/4 rule.
- tests: accented Latin within +10% of the ASCII rule; mixed Cyrillic/ASCII
counts ASCII at one byte; lone surrogates don't raise; anchored pressure
is never floored (wiring shape).
The ~4 chars/token rule is calibrated for ASCII; Cyrillic, Greek, Arabic and
similar 2-byte scripts tokenize at ~2-3 chars/token, so chars/4 under-counts
them ~2x and the pre-flight pressure figure trails real usage by tens of
percent on non-English sessions. Counting UTF-8 BYTES at ~4/token uses the
encoding width itself as the corrective: ASCII is unchanged (1 byte/char),
2-byte scripts count at chars/2, and the CJK dense path keeps its explicit
~1 token/char rule with the sparse remainder byte-counted. The ASCII
isascii() O(1) fast path is preserved; the non-ASCII paths add a single
C-level encode over text that was already being regex-scanned.
Complements the last-real-prompt floor: the floor catches sessions that are
already at the ceiling, this keeps the estimate from lagging in the first
place.
The chars/4 rough estimate under-counts Cyrillic and other non-ASCII scripts
by up to ~2x, so a session can ride the provider's real context ceiling while
the rough pressure stays under the compaction threshold. On providers that
silently clip over-window prompts (ollama /v1) the reactive overflow handler
never fires either, and the length-continuation retry path re-enters the API
call without passing the post-response gate — reproducing the truncation
death spiral this branch already addresses (observed live after the first
commit: real prompts 64,842 -> 64,995 against a 55,705 threshold, output
room shrinking 694 -> 541 tokens).
Floor the pre-API pressure figure at the provider's last reported
prompt_tokens — authoritative, script-independent — except for the one turn
after a compaction when that value is known-stale (#36718's
awaiting_real_usage_after_compression window).
Idle compaction (`compression.idle_compact_after_seconds`) skipped work only
when the context was at or below `threshold_tokens × summary_target_ratio`.
That is a *theoretical* post-compaction target: a real pass lands well above
it because the system prompt, the tool schemas and the protected head/tail
are an incompressible floor.
So a session that compacted to well above the target stayed above it forever,
and every later idle resume re-ran a full summarisation over a transcript that
had not grown. In the reported session the 17:01 pass reduced 64,105 -> 44,579
tokens; the 17:36 resume re-fired because 44,579 was still above the 25,502
theoretical floor, blocking the prompt for another 256 s on a ~55 tok/s local
route and reclaiming nothing. Neither pass was followed by a single API call.
`_should_idle_compact` now also honours `last_compression_rough_tokens` — what
the previous pass on this session actually produced, recorded by
`compress_context` with the same `estimate_request_tokens_rough` shape the idle
estimate uses. When it is known, the transcript must accumulate at least one
`floor_tokens` worth of new content on top of it before another pass is worth
its wall clock. A raised floor is a deferral, not an off switch.
`0` — nothing compacted yet, or the counter cleared by a rebind/recalibration
— keeps the original semantics exactly, so the first idle compaction of any
session is unaffected. The call site type-pins the read so compressor doubles
that expose a Mock there fall back to the original floor.
Fixes a root cause behind #97239. Deliberately does not touch the TUI status
surface (#97253), the digest chain / total ceiling (#93241), or turn-settle
scheduling (#96891).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HV3k1v5nx9ag5d5wXSFZ7o
Two /simplify-code findings on the salvaged #95160:
- The native read walked every line of the whole file with per-line
Python bookkeeping even after the requested window had passed, because
total_lines and the trailing byte are still needed. Once lineno is past
end_line, switch to chunk.count(b'\\n') for the remainder -- same result,
C speed. Page 1 of a 3M-line (123 MB) file: 448 ms -> 49 ms. Parity
harness native vs shell: 20 shapes (1 MiB boundary, no trailing newline,
past-EOF offsets, long lines) identical.
- _native_read_enabled re-implemented the LocalEnvironment check that
_lsp_local_only() already does in this class (same env-None / import
policy) and memoised it on self although self.env is bound once in
__init__ and never rebound. Call the existing helper, drop the cache.
Follow-up to the salvaged #95160. The test wrote the NFC spelling of
café.txt and read the NFD spelling. APFS resolves NFD lookups to the NFC
file directly, so on macOS the compound probe simply found the file, the
retry path never ran and r.hint was None -> TypeError (Linux CI passed
because ext4 does not alias normalisation forms).
Use a curly-vs-ASCII apostrophe instead: still visually identical, still
mapped by _unicode_variant_match, and never aliased by any filesystem, so
the retry path is exercised everywhere.
Phase-2c finding on #101598: the empty-set release closed the pipe and
dropped the Popen without reaping it, so an idle gateway that once
connected a stdio server held one zombie until the next Popen in the
process. wait(timeout=5) after close -- the supervisor exits on EOF
immediately with nothing registered (E2E: ps shows no entry at all).
Also: the replay test asserted against the last line only, which could
not catch the regression it names; assert against the full stream and a
post-unregister replay. Module docstring now states the stdlib-only /
no tools/ import constraint and why _reap's sweep is a deliberate copy.
Contract requested in the #93517 review, now that main has profile-scoped
MCP ownership: two owners each hold a real stdio process group; a scoped
_kill_orphaned_mcp_children(include_active=True, server_name=A) reaps
only A's group, sends 'unregister' only for A, and the per-process
supervisor still holds B.
Addresses @andrexibiza's Blocker 1 on #93517 (reproduced): after a
broken-pipe write dropped the supervisor while groups were still
registered, the no-spawn fast path was keyed on the incoming verb
(`unregister` + no proc => return), so a clean teardown of one server
left the survivors recorded in _supervised_pgids but unsupervised until
the next register. Key the fast path on the supervised set being empty
instead; any later call respawns and replays the survivors.
Regression test: two live groups, write fails, unregister one ->
replacement receives `register` for the survivors. Mutation-checked
against the old verb-keyed guard.
Follow-up to the salvaged #93517. Two gaps found in review:
- After the last unregister the supervisor stayed resident for the life of
the process (~15 MB + a pipe) in any gateway/cron that ever connected a
stdio server; main's per-server watchdog exited with its server. Close
our write end when the supervised set empties: EOF with nothing
registered makes the supervisor exit without reaping, and the next
register already respawns and replays.
- A child that raced and exited before os.getpgid dropped its group from
coverage entirely. The SDK spawns stdio servers as session leaders
(pgid == pid), so fall back to the pid instead; the prune forgets the
group once nothing in it is alive.
Tests: fake-supervisor release/re-spawn sequence, and a real-process EOF
release (supervisor exits 0, unregistered child untouched). Mutation
checked: disabling the release branch fails both.
Review raised a real gap: we reap by pgid, so a registration is only as
meaningful as the group's identity. A group we deliberately keep registered --
an orphan teardown failed to kill, such as the `node` mcp-remote leaves behind
-- can later exit on its own, after which the kernel may hand that pgid to an
unrelated process owned by the same user. An ungraceful Hermes death while the
registration is stale would then signal a stranger. `_is_safe_target` cannot
catch it, because the value is stale rather than invalid.
Prune registrations whose group has no members left on every registration
change, and tell the supervisor to forget them. Signal 0 is a pure existence
question -- it cannot terminate anything -- so this is cheap and safe to run on
the hot path. An ambiguous answer (EPERM: exists but not ours) keeps the
registration, since dropping real coverage is the more expensive mistake.
This narrows the window rather than closing it: a group can still die and its
pgid be recycled between two probes. Closing it completely means proving group
identity at reap time, e.g. stamping children with a boot-unique env marker and
checking a member still carries it. That was judged not worth putting a `ps`
parse into the one process whose job is to stay simple enough to always work,
so the residual is now documented in the module docstring instead, along with
the note that Hermes's existing killpg-based orphan cleanup already carries the
same exposure (upstream #88350).
Also records why supervisor recovery is deliberately two-step: a failed write
drops the handle, and the next call rebuilds coverage from `_supervised_pgids`,
which -- not the pipe -- is the record of what needs reaping.
The protocol tests register synthetic pgids that were never real process
groups, so they now state that precondition through an `all_groups_alive`
fixture instead of depending on pid-space luck.
29 tests pass. Verified the change adds no failures: same selection, my changes
stashed vs applied, 42 failed either way (pre-existing in a locally rebuilt
venv) with passed going 617 -> 619 for the two new tests. Footgun linter clean.
(cherry picked from commit 3a7d219620a8cb57a7fe2c2346ce3f055320e743)
Every stdio MCP server was wrapped in its own CPython watchdog that polled
getppid() every 2s to notice an ungraceful Hermes exit (kill -9, OOM, crash,
force-quit), since macOS has no PR_SET_PDEATHSIG. That is a whole interpreter
per server for a job that does nothing until the moment Hermes dies: 10.1 MB
physical footprint each, measured on macOS/arm64.
Replace the fleet of pollers with a single supervisor per Hermes process
holding the read end of a pipe only Hermes writes to. Death detection becomes
EOF on that pipe -- exact and instant, rather than up to a poll interval late.
Hermes sends `register <pgid>` / `unregister <pgid>` as servers come and go;
on EOF the supervisor killpg's whatever is still registered, which is exactly
the set whose teardown never ran. A clean shutdown unregisters as it goes, so
EOF then finds nothing to kill.
Servers are now spawned unwrapped. The MCP SDK already starts each stdio child
in its own session, so the pgid recorded for killpg is the server's own group
and the existing cleanup paths reach it unchanged. That also deletes the
signal-forwarding layer the wrapper needed: wrapping had put the real server in
a different session from the pgid being tracked, so a graceful killpg would
have hit only the wrapper.
Measured on a 5-gateway host: 10 watchdogs (~98 MB) -> 5 supervisors (~49 MB).
One supervisor costs about what one watchdog did (9.9 vs 10.1 MB), so the win
is (servers_per_process - 1) x ~10 MB, and a process with no stdio servers now
spawns nothing at all.
The supervisor reads length-capped lines rather than iterating the stream: a
writer that never sends a newline would otherwise grow it without bound, which
it must not be vulnerable to when it is the last defense against leaked
servers. Found by feeding it /dev/zero, where it reached 15 GB.
Verified beyond unit coverage: a real stdio MCP server connects and its tools
are discovered on the unwrapped path; with a live Hermes holding a real
connection, kill -9 reaped the server, its grandchild (in the server's group,
the mcp-remote `node` case), and the supervisor exited on its own. The reap
tests were sabotage-checked in both directions -- a no-op reaper fails all
three, while the test pinning that a cleanly unregistered server survives keeps
passing -- and each wiring half fails independently when removed.
(cherry picked from commit a252d4ce7ff1722f687635fdbf0cff79f538c3f1)
Follow-up to #92164: the delta window now ends on a character boundary
(up to 3 trailing continuation bytes are held for the next poll), so
multibyte output no longer decodes to U+FFFD at the seam. Verified on
bash, dash and busybox sh; exhaustive-prefix regression test added.
tests/gateway/test_feishu_bot_admission.py builds FeishuAdapter via
object.__new__ and installs dedup state by hand (install_dedup_state), so
the asyncio.Lock added in __init__ was missing -> AttributeError in CI.
Resolve the lock through a getattr-with-create helper and add it to the
shared test helper.
Offloading the fsync to a worker thread (previous commit) lets two inbound
messages have two persists in flight. Without ordering, a slow first flush
lands after the second and the disk file loses the newer token/id/seen-id
(in-memory state stayed correct, so the loss only surfaced after restart).
The loop-side dict could also be mutated while the worker iterated it.
- weixin ContextTokenStore / discord non-conversational tracker: snapshot
the payload on the loop, hand it to the worker, serialize with a per-store
asyncio.Lock.
- feishu dedup: same asyncio.Lock around the offloaded persist (its snapshot
was already taken under _dedup_lock; the write was not).
- Regression tests: two concurrent persists with a slow first write must
leave the union on disk. All three fail on the previous commit.
atomic_json_write() calls os.fsync(), which blocks until the write
reaches stable storage. #83906 offloaded this in channel_directory.py,
gateway/run.py, and gateway/slash_commands.py, but the same blocking
pattern was still present on three platform-adapter message hot paths
that were never touched by that pass:
- plugins/platforms/feishu/adapter.py: _is_duplicate() persists the
dedup cache on every new inbound message (shared by both websocket
and webhook transports). The in-memory bookkeeping stays under
_dedup_lock; the fsync now runs off it via asyncio.to_thread so
concurrent callers on the ws-thread loop aren't blocked on disk I/O.
- gateway/platforms/weixin.py: ContextTokenStore.set() persists the
context-token cache on every inbound message carrying a
context_token, called from the async poll-loop handler
_process_message.
- plugins/platforms/discord/adapter.py: mark_many() persists the
non-conversational message-id tracker, called from both send() and
send_update_prompt() when a message is tagged nonconversational.
Each fix mirrors the existing asyncio.to_thread(atomic_json_write, ...)
pattern from #83906. Regression tests assert the persist call runs on
a different thread than the event loop, mirroring
test_directory_write_runs_off_event_loop_thread in
test_channel_directory.py.
Extract the signature-checking cleanup dispatch that cleanup_vm already had
into tools.terminal_tool._cleanup_env and call it from the prompt-time
backend probe instead of carrying a second copy. Comment on the probe
trimmed to the non-obvious part (why ssh is skipped).
Follow-up to the cherry-picked fix:
- decide force_remove by inspect.signature (the idiom cleanup_vm already
uses) instead of catching TypeError, which would misread a TypeError
raised inside cleanup() as a signature mismatch and call cleanup twice
- skip teardown for ssh: SSHEnvironment has no task-scoped sandbox and its
cleanup() runs sync_back + 'ssh -O exit' on a ControlMaster socket keyed
by user@host:port, shared with the agent's real environment
`_probe_remote_backend` spins up an environment with
`task_id="prompt-backend-probe"` purely to run one `uname` for the system
prompt, then drops the reference without cleaning up. Container backends
default to `container_persistent` / `docker_persist_across_processes`, so
that throwaway sandbox stays up for the entire process lifetime, sitting
idle next to the agent's own `default` sandbox.
Observed on a three-profile Docker-backend host: six running containers
where three were doing any work. Every profile permanently paid for one
extra container whose only job had been a single `uname` at startup.
Fix: wrap the probe in `try/finally` and call
`env.cleanup(force_remove=True)` — the explicit-teardown path that
`DockerEnvironment.cleanup` already documents ("No current caller passes
`force_remove=True`; the parameter is here so the explicit-teardown
semantics can be wired up later"). Backends that inherit the kwarg-less
base `cleanup()` fall back to the bare call.
Verified on a live aarch64 Docker backend: the probe container is now
created, used and reclaimed within the same call. Three regression tests
cover the success path, the exception path, and the kwarg-less fallback;
all three fail without the source change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PLbMzWqHVTPNBFisM8UiNB
Findings from the efficiency review pass on the two salvages, applied as one
small follow-up:
- copilot_auth: check the negative cache BEFORE taking the per-fingerprint
exchange lock. During the 60 s post-failure window, dashboard polls now
raise immediately instead of parking an executor thread behind the
in-flight holder (up to ~50 s) to learn the same answer. Test hangs
without the check (timeout 124), passes with it.
- buzz _localize_inbound_media: download_path.read_bytes() was still
evaluated on the loop as the argument to the offloaded cache call — up to
the 128 MiB inbound cap. Read off the loop too.
- test_list_credential_pool_keeps_loop_responsive: 0.5 s block / 0.25 s
threshold (2x margin) so runner descheduling cannot false-fail it while a
real regression still trips it.
`hermes_home_key` called `Path.resolve()` on every call, which is a
filesystem call. `ToolRegistry.current_scope_key()` calls it, and
`_merged_tools()` calls that, so every single registry lookup went to
the disk: `get_entry`, `get_toolset_for_tool`, `get_emoji`, and the rest.
Anything that looks up many tools in a row paid for it. Building the
deferred-tool catalog for `tool_search` calls `get_entry` once per tool,
so with 400 deferrable tools one search did 800 path resolutions. On
Windows that is `nt._getfinalpathname`, and it took 132ms of the 148ms
the catalog build spent. The tokenizing everyone would suspect first was
under 4ms of it.
The resolved value is now remembered per input path. A process only sees
a handful of home paths, so the dict stays tiny. Measured on the same
400-tool catalog build, the lookups drop from 132ms to 3.1ms with one
entry cached.
A path that does not exist yet is resolved but not remembered, because
the answer can change once it is created, for instance when part of the
path turns out to be a link. A different home path is a different key,
so switching profiles or `HERMES_HOME` gets its own entry rather than
reusing a stale one.
Same retry semantics (only a non-FileNotFoundError OSError leaves the latch
unset), one latch assignment instead of five; docstring now says the call
runs per get/put and does work once.
kshitijk4poor