mcp 2.0.0 implements MCP revision 2026-07-28 and makes three breaking
changes Hermes sits on top of: `mcp.server.fastmcp` is gone, every model
field is renamed to snake_case (camelCase survives only as a
serialization alias, which pydantic does not expose to attribute
access), and the SDK's own HTTP stack moved from `httpx` to `httpx2`.
Bump the pin across the dev/mcp/computer-use extras and port the tree:
- `mcp_serve.py` and `agent/transports/hermes_tools_mcp_server.py` move
from `FastMCP` to `mcp.server.MCPServer`, which has the same
decorator/add_tool surface. The hermes-tools server already
synthesised `__signature__` from Hermes' JSON Schema, which is exactly
what 2.0's `add_tool` reads.
- SDK model reads go through `mcp_field(obj, snake, camel)`, which reads
both spellings. A single-spelling read fails *silently* on the other
generation — empty tool schemas, dropped structured content, tool
results vanishing from sampling conversations — and `mcp` is an
optional extra users install at their own version.
- `sdk_httpx()` resolves the httpx flavour from the SDK's own transport
module, so objects handed to `streamable_http_client`, the `sse_client`
factory, and the OAuth metadata helpers come from the module the
installed SDK actually imports.
- HTTP support is gated on either streamable-HTTP entry point, not just
the deprecated alias 2.0 removed.
- OAuth: `OAuthClientProvider` lost its `timeout` argument (the
configured `oauth.timeout` now bounds the callback waiter's own poll
loop, where the browser round-trip was always awaited), and
`callback_handler` must return `AuthorizationCodeResult` rather than a
tuple. 2.0 also validates the RFC 9207 `iss` parameter, so the
callback handler and paste fallback capture it.
`mcp`/`mcp-types` 2.0.0 are inside the 14-day `exclude-newer` window, so
two narrow `exclude-newer-package` entries unblock `uv lock`, annotated
for removal on or after 2026-08-11. `httpx2` needs no exemption: 2.7.0 is
already outside the window and satisfies mcp's floor.
Refs #69931
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A same-day version-floor bump (0.20 runtime contract) left every install
with an older cua-driver hard-failing on all computer_use calls: the
start() gate fails closed, while the `hermes update` refresh defers to the
driver's own check-update verb — whose ~20h cache routinely answers "no
update available" right after we raise the floor. Hermes knew it required
0.20+ but never acted on that knowledge.
Two changes:
- tools_config.install_cua_driver(): a contract-failed installed driver is
repaired on the upgrade=True path too (previously only upgrade=False).
The contract failure itself is the confirmation, so the
require_confirmed_update gate and the check-update short-circuit are
bypassed for repairs — an indeterminate or stale-cached check can no
longer pin users on an unusable driver.
- cua_backend.CuaDriverBackend.start(): when the contract gate fails on an
installed binary, attempt one automatic repair per process via the
standard install path, then re-probe. HERMES_CUA_DRIVER_CMD overrides
are never repaired (explicit override is authoritative even when broken)
and a missing binary still just reports the install hint. A failing
installer can't loop: the second start() surfaces the original error.
Tests: contract-repair coverage in test_computer_use.py (auto-repair
success, failed repair surfaces the original error, once-per-process
guard, override never repaired, missing binary never repaired) and
test_install_cua_driver.py (incompatible driver repairs despite an
indeterminate check-update, check-update not consulted). All new tests
verified to fail against the unfixed source (sabotage run).
`--yolo` / `-z` route the session onto a private embedded daemon in
`unrestricted` mode. That daemon was constructed without the configured
capability manifest, and the serve command only attached
`--capability-manifest` when the mode was exactly `bounded`. So the moment a
run was bypassed, the user's declared ceiling was dropped:
without -z: --permission-mode bounded --capability-manifest ...
with -z: --permission-mode unrestricted --dangerously-bypass-approvals
No manifest, no warning. The most carefully configured run - a reviewed
ceiling, written by hand - became the least constrained one, silently, and
it failed open.
That was never a driver limitation. cua-driver documents the manifest as a
ceiling across modes ("A manifest can narrow a profile but never widen it";
its own authorization table calls it `optional_capability_manifest_ceiling`),
and accepts it alongside `--permission-mode unrestricted`.
The forwarding is version-aware, because the two manifest schemas differ
(cua-driver session_manifest.rs):
* v1/v2 are legacy and must declare `mode: bounded`. Handing one to an
unrestricted runtime aborts startup with "legacy capability manifest mode
must be bounded", so a naive forward would turn a working session into a
hard failure. These are forwarded for bounded only, and a warning names
the migration when one cannot apply.
* v3 must not declare a mode. It is the mode-independent ceiling, and it now
rides along with unrestricted.
Unreadable or unparseable manifests are not forwarded outside bounded, on
the same fail-safe reasoning; bounded still forwards unconditionally and
lets the driver be the authority there.
Verified against cua-driver 0.20.0 on Windows. Launch args now carry
`--permission-mode unrestricted --dangerously-bypass-approvals
--capability-manifest <v3> --approve-capability-manifest`, and the ceiling
is enforced in the bypassed run - a tool outside the manifest is refused
("outside the capability manifest for this session ... blocked as a
protected resource") where the same config previously ran unbounded. A
legacy manifest was confirmed to abort driver startup when forwarded, which
is what the version gate prevents.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Live-testing the Cua Driver 0.20 convergence on Windows 11 (session 2,
cua-driver 0.20.0) surfaced three defects in the existing-profile browser
path and in install status.
1. The config grant was silently nullified by an approval bypass.
`--yolo` / `-z` map onto a private unrestricted daemon, which answers every
browser_prepare. Because the host delegated the entire existing-profile
decision to the driver, that bypass also nullified
`computer_use.grant_existing_profile: false`: a plain `hermes -z` attached
to the user's real Chrome profile and read live page content over CDP, with
the driver reporting it as "the approved existing Chromium profile". It was
never approved.
An approval bypass is consent to skip prompts, not consent to read an
existing profile's pages, cookies, and storage. CuaTypedBrowserRoute.prepare
now enforces the key itself, regardless of permission mode. bounded stays
exempt - its reviewed capability manifest is the authorization boundary.
The authorization inputs are resolved in the backend from config and the
backend's immutable mode, never from model-supplied kwargs.
2. The grant, once set, still could not be used.
With `grant_existing_profile: true` the runtime is launched
`--grant existing-profile` correctly, but cua_browser_prepare then hit a
runtime approval prompt anyway - re-asking the user to authorize what the
config already authorized, and making the documented opt-in unusable on any
non-interactive run, where the prompt has nobody to answer it and the call
dies on approval timeout. The durable, file-backed grant now stands in for
that prompt. Scope is narrow: only the existing-profile prepare, only when
the grant is present; isolated launches still prompt and any resolution
failure falls closed to prompting.
3. `computer-use status` hid a custom override and spliced its output.
With HERMES_CUA_DRIVER_CMD pointed at cmd.exe, status printed the child's
multi-line banner and prompt inside the one-line version field, never
mentioned the override, and advised `hermes computer-use install` - which
install itself (correctly) refuses to run against an overridden path. It now
names the override and mirrors install's update-or-unset guidance, and
version output is reduced to one bounded line.
Verified on the reported host: `-z` existing-profile attach now refuses and
names the key; `grant: true` no longer prompts (33s vs a 300s approval
timeout); status names the override and prints one line. No change to the
reconciliation path - driver SHA256 unchanged end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A headless Mac or asleep built-in panel leaves ScreenCaptureKit with 0
shareable displays while TCC grants pass — health_report stays ok and
every capture silently returns 0x0 (#67165). Guard at the report seam
(_apply_display_count_guard, both real and fallback paths): flips the
screen_capture_capability check to fail with recovery actions (wake
display / HDMI dummy / virtual display) and downgrades ok -> degraded.
The empty-discovery reason ladder gains the matching darwin rung.
Composed from #52949 (sujeet111) and #67259 (webtecnica); both PRs
predate the doctor rewrite and the envelope normalization on main, so
this reimplements their shared intent at the current seams.
Co-authored-by: Sujeet <64351924+sujeet111@users.noreply.github.com>
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
capture() read any non-None pid/window_id as a request for exact-window
targeting. Several providers emit every declared schema property on every
tool call, zero-filling unused optional integers, so those calls arrive as
pid=0, window_id=0. The exact-target branch was then entered, the caller's
app= was discarded, _positive_int(0) returned None for both ids, and the
capture failed with a message pointing at pid/window_id. For that class of
model capture(app=...) and frontmost capture never worked at all.
Normalize non-positive ids to None before the branch decision so dispatch
falls through to app/frontmost discovery. Malformed non-numeric ids are
deliberately not treated as placeholders: they still reach the existing
validation error instead of being silently ignored.
Fixes#81333
Two live-QA findings from a locked KDE desktop (real cua-driver 0.20.0):
1. capture() with zero discovered windows returned a bare
'capture mode=ax 0x0' — no hint that the desktop session was LOCKED,
which freezes renderers and hides windows. New
_empty_discovery_reason() names the dominant causes in order: locked
session (loginctl LockedHint probe, fail-safe), missing DISPLAY,
else a pointer at hermes computer-use doctor. Surfaced through the
existing window_title -> summary path, so the model and the user see
it inline.
2. _call_tool_via_cli retried 'daemon is not running' 4x with ~3.5s of
backoff sleeps — a permanent condition for that invocation (the CLI
transport needs the machine-wide daemon; Hermes' MCP runtime does
not). Now fails fast on the first attempt with a message naming the
split. Transient empty output (EAGAIN congestion) keeps the retry
loop — pinned by test.
Live-verified on the locked desktop: capture now reports the lock and
the unlock action; CLI fallback errors immediately with the transport
explanation. 7 new tests; 191 sibling tests green.
Live-tested against the real cua-driver 0.19.3 binary (Linux x86_64):
- bounded serve flags corrected: the daemon accepts
--session-policy/--approve-session-policy, not the docs'
--capability-manifest names (which it rejects). Verified end-to-end:
a bounded daemon with a real policy file starts and reports running.
- browser-approve verified real but interactive-only (refuses without a
TTY) and its token is a legacy compatibility path disabled by default
on current drivers (per the live browser_prepare schema). Kept as a
passthrough; no longer presented as the primary route.
- NEW primary standard-mode route, verified live: launch the runtime
with cua-driver's trusted-launcher grant. config opt-in
computer_use.grant_existing_profile: true appends
--grant existing-profile to the standard-mode MCP spawn (MCP
initialize verified accepting the flag). Default false = attachment
keeps failing closed. Never applied to bounded/unrestricted daemons.
- Skill, system prompt, tool schema, and docs updated to the verified
ladder: config grant > bounded manifest > YOLO; token = legacy.
Completes the typed cua_browser_* route (PR #74166 lineage) with the
authorization surface that makes existing-profile attachment and
repeatable bounded automation reachable by real users:
- hermes computer-use browser-approve: CLI passthrough that mints
cua-driver's five-minute single-use attachment token for one exact
(pid, window_id). The user, never the model, is the token source.
- approval_token passthrough on cua_browser_prepare (schema + dispatch +
browser_route), forwarded only for existing_profile and only as a
non-empty string.
- computer_use.permission_mode: bounded + capability_manifest config:
private per-session embedded daemon launched with
--capability-manifest/--approve-capability-manifest; missing manifest
fails loudly. 'unrestricted' is deliberately NOT a config value —
it stays bound to the explicit per-session YOLO toggle.
- Skill + system-prompt + docs guidance for the three authorization
rungs and the isolated-profile-first default.
E2E-verified against a temp HERMES_HOME: real config resolution to
bounded, loud failure without a manifest, real argparse path driving a
fake cua-driver binary, standard default preserved.
Follow-up to the #62821 salvage: the _cua_driver_supports_no_overlay
--help probe is another Windows-reachable spawn; give it the same
windows_hide_flags() treatment. Also adjust the status test to stub
_resolve_driver_cmd (permissions.py resolves via that helper, not
shutil.which).
A Windows-installed cua-driver can return an absolute
``C:\Users\...\cua-driver.exe`` mcp_invocation.command to a Hermes
process running inside WSL. POSIX spawning can't use the raw Windows
string even though the binary is reachable through DrvFS. Translate
``<drive>:\...`` to ``/mnt/<drive>/...`` in _resolve_mcp_invocation
(before the path-separator check, since backslash is not a separator
on POSIX), only when actually running under WSL.
Salvaged from #63532 by @motoblurr (original commit carried a
placeholder 'Hermes Agent <hermes@local>' identity; re-authored).
Fixes#63938 premise.
Apply windows_hide_flags() (CREATE_NO_WINDOW; 0 on POSIX) at the
Windows-reachable cua-driver subprocess boundaries: manifest probe,
update checker, CLI fallback transport, doctor health-report spawn,
and the permissions/status runner. Prevents OpenConsole/Windows
Terminal windows flashing into the foreground when spawned from
GUI-backed Gateway/Desktop processes.
The env-probe half of the original PR was already implemented on main
and is not re-applied here.
Salvaged from #62821 by @ZundamonnoVRChatkaisetu (original commits
carried a placeholder 'Claude Code Enterprise' identity; re-authored
to the contributor's GitHub identity).
cua-driver 0.7.x can return list_windows/list_apps payloads under
structuredContent.windows, data.windows, data._legacy_windows, or
top-level windows/_legacy_windows (direct CLI responses). The wrapper
only read structuredContent.windows, so discovery came back empty
(capture 0x0, list_apps []) while raw cua-driver calls worked.
- add _windows_from_tool_result(): walks the known envelope shapes in
priority order, skipping empty higher-priority envelopes
- route _load_windows() (MCP + CLI re-fetch paths) through the helper,
covering capture() and focus_app()
- harden _ingest_windows(): skip non-dict members, normalize untrusted
app_name/title/z_index fields
- list_apps(): prefer structuredContent.apps, fall through populated
data/top-level envelopes, derive unique apps from window-shaped
payloads via _apps_from_windows(), keep the text-line fallback last
- tests for every envelope shape, precedence, malformed records, and
app derivation
Salvaged from #63037 (Reaper-Legion), which itself preserved the
original implementation from #57961 (kohoj); #73007 (umi008)
independently proposed the same normalization later.
Fixes#57905
Co-authored-by: Reaper <248977840+Reaper-Forge@users.noreply.github.com>
Co-authored-by: Ulises Millan Guerrero <ulises.millanguerrero@gmail.com>
'Refreshing cua-driver (Computer Use)...' could hang for minutes on
Windows: when the driver's native check-update verb returned an
indeterminate result (old driver without the verb, offline, GitHub
rate-limited, or the probe timing out), install_cua_driver(upgrade=True)
fell through to the full upstream installer — a silent, output-captured
run with a 660s ceiling, plus install.ps1's 600s concurrency-lock wait
on Windows on top. Every 'hermes update' paid that cost.
Two changes:
- install_cua_driver() grows require_confirmed_update: with it set, an
indeterminate check keeps the installed version and returns fast,
printing the force path (hermes computer-use install --upgrade).
'hermes update' passes it; the explicit --upgrade CLI keeps the old
fall-through so a force refresh still works when the check can't
answer.
- cua_driver_update_check() default timeout is now 25s on Windows
(8s unchanged on POSIX): first-spawn of the exe under Defender /
SmartScreen routinely exceeds 8s, and a false timeout is exactly the
indeterminate result that used to trigger the multi-minute reinstall.
AST-driven pass over every subprocess.run/Popen/check_output/check_call/call
with text=True (or universal_newlines=True) and no explicit encoding=:
append encoding='utf-8', errors='replace' at the kwarg site. 136 call
sites across 28 files (cli.py, hermes_cli/main.py, tools_config.py,
environments, computer_use, gateway, scripts, skills helpers, agent/*).
Together with the salvaged #55339/#60741 commits this closes out issue
#53428's bug class; the salvaged #60751 linter rule in
check-windows-footguns.py now enforces it repo-wide (verified: 807 files
scanned, zero findings).
Treat cua-driver's Linux `is_on_screen: null` as unknown instead of
off-screen, and skip GNOME Shell desktop/backdrop helper windows
(ding "Desktop Icons", @!x,y;BDHF) when selecting the default capture
target — they are targetable X11 windows but capture as empty.
Reconciled with the _NET_ACTIVE_WINDOW fallback from #58030: helper
windows are filtered out of the candidate pool first, then the tied
z-order active-window probe runs on the remaining real app windows.
Also falls back to the requested app name for _last_app when Linux
windows carry no app_name.
Salvaged from #54173 by @dnth.
Guard the post-start set_agent_cursor_enabled on _session._started so
call_tool cannot re-enter session.start() (matches the start_session
lifecycle guard).
Cut steady-state Computer Use latency without changing default behavior
or waiting on cua-driver:
- Cap screenshots via set_config(max_image_dimension) on session start
(config: computer_use.max_image_dimension, default 1456)
- Cache aux-vision routing per (provider, model) so captures skip
repeated load_config()
- Add computer_use.capture_after_mode (default som) so users can opt
follow-ups down to ax (elements only) for speed
Auto-detect now disables the cursor overlay on darwin as well as
headless/WSL2 Linux. After start_session, also call
set_agent_cursor_enabled(false) when the policy is on so older drivers
without --no-overlay still tear the overlay down.
Co-authored-by: David Metcalfe <80915+DavidMetcalfe@users.noreply.github.com>
The hermes-sweeper review #4701565902 (2026-07-15) flagged two
consistency issues in `_cua_driver_supports_no_overlay` and one
additive-config concern:
1. `cua-backend.py:260` — the `cua-driver --help` support probe
inherited the full parent environment. cua-driver is a third-party
binary; every other spawn site in this file (manifest probe at
`:214`, MCP spawn at `:697`, install probe at `:997`) uses
`_sanitize_subprocess_env(cua_driver_child_env())`. The `--help`
probe should match. This was a low-impact leak (only help output
exits), but inconsistency is the wrong default for a third-party
subprocess.
2. `cua_backend.py:238` — when the manifest returned a `command`
different from the input `driver_cmd` parameter (e.g. a relocated
executable at `/opt/relocated/cua-driver` while the system binary
is at `/usr/bin/cua-driver`), the support probe ran against
`_CUA_DRIVER_CMD` (the default) instead of the manifest-discovered
`command`. Two failure modes:
- The wrapper binary supports `--no-overlay` but the system binary
doesn't → probe returns False → overlay kept despite capability.
- The system binary supports `--no-overlay` but the wrapper doesn't
→ probe returns True → MCP spawn crashes on the unknown flag.
3. The original commit bumped `_config_version` 31→32 for an additive
default (`computer_use.no_overlay: None`). AGENTS.md specifies that
additive defaults in existing sections are handled by deep merge
and should NOT trigger a version bump. After cherry-picking onto
current `origin/main` (which is already at 33), the bump is
effectively dropped — resolved to main's 33.
Changes:
- Add `env=_sanitize_subprocess_env(cua_driver_child_env())` to the
`--help` subprocess (with the same import + rationale comment as
the manifest probe).
- Pass `driver_cmd=command` (or `driver_cmd=driver_cmd` for the
fallback path) into `_mcp_args_with_overlay_flag`, so the support
probe runs against the binary that will actually be launched.
Tests (3 new):
- `test_help_probe_passes_sanitized_env` — verifies `subprocess.run`
is called with an `env=` kwarg.
- `test_manifest_command_drives_support_probe` — verifies the probe
runs against the manifest command when it differs from the input
driver_cmd.
- `test_fallback_uses_input_driver_cmd_for_support_probe` — verifies
the fallback path (no command in manifest) uses the input
driver_cmd.
- `test_probe_distinguishes_support_between_binaries` — sanity check
that the lru_cache key on `driver_cmd` prevents cross-binary
cache leakage.
File-revert negative test confirmed all three of the new
"manifest/probe" tests are load-bearing: with the pre-fix code, they
fail (probe runs against the default binary instead of the resolved
one); with the fix, they pass. 20/20 tests in
`tests/computer_use/test_cua_no_overlay.py` green.
`TestMcpInvocationResolution` (8/8) still green.
Refs: sweeper review #4701565902
Address review feedback from cross-vendor review (Flash + GPT-OSS):
1. Auto-detect now checks for headless Linux (no DISPLAY), WSL2
(/proc/version contains 'microsoft'), instead of all Linux.
Desktop Linux with a compositor keeps the overlay.
2. Add _cua_driver_supports_no_overlay() that probes cua-driver --help
to check if the flag is supported. Older drivers (< 0.6.x) reject
unknown flags, so passing --no-overlay would crash the MCP spawn.
3. Update tests to cover headless vs desktop Linux, WSL2 detection,
version probe, and the unsupported-driver fallback path.
cua-driver's cursor overlay rendering loop can consume CPU indefinitely
when idle (#28152, #47032). On Linux/WSL2, the overlay serves no visual
purpose and the rendering path is the primary source of idle CPU usage.
Add computer_use.no_overlay config option (default: auto-detect) that
passes --no-overlay to cua-driver when enabled. Auto-detection disables
the overlay on Linux (covers WSL2, headless, containers) where it has no
benefit, and keeps it enabled on macOS/Windows where it is visually
useful.
Refs: #28152, #47032
Limit the _NET_ACTIVE_WINDOW xprop fallback to unqualified default
captures so exact pid/window_id targeting does not pay up to a 2s
subprocess probe on Linux/X11.
When Linux/X11 reports the same z_index for every on-screen window,
prefer _NET_ACTIVE_WINDOW via xprop instead of list order. Keep the
higher-z-index-is-frontmost contract when ordering is informative.
Bug 1 of #55048: when the MCP connection dropped (driver crash / restart),
_lifecycle_coro exited but left _started=True, so the next list_apps/capture
passed _require_started() and then operated on a None session — hanging
forever instead of reconnecting.
- _lifecycle_coro's finally now resets _started=False on ANY exit, so a dead
session is re-enterable (idempotent no-op on the normal stop() path; atomic
bool write, safe from the bridge-loop thread without the lock stop() holds).
- call_tool() re-enters start() when the session isn't active, rebuilding it
before the call. The start_session/end_session handshake (driven by start()/
stop() themselves) is exempted so bootstrap doesn't recurse.
Tests: two cases in test_computer_use_delivery_ladder.py — finally resets
_started, and call_tool restarts a dead session exactly once. Full
computer_use suite green (233).
Refs #55048 (Bug 1). Bug 2 (expose foreground dispatch) is covered by the
delivery_mode work in #67123.
Hermes' computer_use wrapper dropped cua-driver's structured action verdicts,
exposed no delivery_mode, and injected background-only guidance — so the agent
reported unverified no-ops as success and concluded cua-driver 'cannot drive'
Electron/Chromium surfaces (observed live on tldraw offline). Fixes#67052.
Phase A — preserve the result contract:
- ActionResult carries verified/effect/escalation/path/degraded/code/delivery_mode
- CuaDriverBackend._action() reads structuredContent (was data-only); a helper
normalizes it, additive and None-safe on old drivers
- _text_response surfaces the fields additively (ok stays transport-only)
Phase B — bounded, model-reachable foreground:
- delivery_mode (background|foreground) + bring_to_front on the schema, dispatcher,
ABC, and all input methods
- foreground is capability-gated (input.delivery_mode); old drivers get a
structured foreground_unsupported refusal, never a silent background downgrade
- no automatic/hidden foreground retry — the model selects it from the signal
Phase C — guidance + isolation:
- system prompt (prompt_builder) and bundled skills/computer-use/SKILL.md go from
background-ONLY to background-FIRST, teaching the AX→PX→foreground ladder driven
by returned effect/escalation, not predicted from the app being Electron
- foreground approval scoped by (action, delivery_mode): a background approval
never silently authorizes foreground
- approval state keyed per session_id so concurrent gateway runs don't leak unlocks
Tests: tests/tools/test_computer_use_delivery_ladder.py (15) cover confirmed/
unverifiable/suspected_noop/degraded/old-driver verdicts, delivery_mode gating +
foreground_unsupported, and session-scoped foreground approval. Existing 265
computer_use tests still green.
Live E2E (real cua-driver 0.8.3 + tldraw offline on Linux/X11): a background click
returned effect='unverifiable'/path='ax' (no fabricated success), and a foreground
request returned code='foreground_unsupported' — correct on a driver that predates
the input.delivery_mode capability.
* fix(computer-use): target Linux app windows reliably
Resolve app filters through the canonical cua-driver MCP app metadata and join running app PIDs back to windows. Preserve an exact selected window across capture_after, support direct capture by pid/window_id, and send the active window ID for coordinate pointer actions on Linux.
Co-authored-by: annguyenNous <annguyenNous@users.noreply.github.com>
Co-authored-by: grimmjoww578 <willies578@gmail.com>
Co-authored-by: ai-ag2026 <261867348+ai-ag2026@users.noreply.github.com>
* fix(computer-use): address review on PR #63725
Address three review comments from @f-trycua:
1. type_text, press_key, and hotkey now carry _active_window_id and
fail closed when it is missing. Previously they sent only the PID,
so CUA Driver fell back to the first window for that PID — input
could reach the wrong window in multi-window apps.
2. Coordinate scroll x/y are now capability-gated behind
input.scroll.coordinates. CUA Driver 0.7.1 Linux schema rejects
x/y on scroll; omitting them when the driver doesn't advertise
support avoids the schema rejection while still routing via
window_id.
3. Windows are sorted by z_index descending (higher = front, per CUA
Driver semantics) instead of ascending. Null z_index (Wayland) is
coerced to 0 in _ingest_windows so it doesn't crash the sort and
sorts to the back instead of being selected as the capture target.
---------
Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Co-authored-by: annguyenNous <annguyenNous@users.noreply.github.com>
Co-authored-by: grimmjoww578 <willies578@gmail.com>
Co-authored-by: ai-ag2026 <261867348+ai-ag2026@users.noreply.github.com>
PR #58889 fixed the CLI-fallback transport; review of that fix found the
same leak class at four sibling spawn sites of the third-party cua-driver
binary:
- _resolve_mcp_invocation (cua-driver manifest): no env= at all — full
parent environment inherited
- cua_driver_update_check (check-update --json): telemetry env but no
secret sanitization
- doctor._drive_health_report (<binary> mcp Popen): telemetry env only
- permissions._run (every macOS/Linux permission probe): telemetry env only
All now route through _sanitize_subprocess_env(cua_driver_child_env()),
matching the sanctioned MCP spawn and the #53503/#55709/#58889 strip-by-
default policy for non-terminal spawns. Sanitization degrades gracefully
(falls back to the telemetry env) so doctor/permission probes never break
on an import error.
4 regression tests covering each site.
_CuaDriverSession._call_tool_via_cli() (the EAGAIN/silent-empty MCP
fallback transport) invokes `cua-driver call <tool> <json>` via
subprocess.run() with no env= argument, so the third-party cua-driver
binary inherits the full, unsanitized parent environment. The primary
MCP spawn site (_lifecycle_coro) already applies
_sanitize_subprocess_env(cua_driver_child_env()) before opening the
stdio client, per the same policy #53503/#55709 established for other
subprocess spawn points — this fallback path, added alongside the
EAGAIN/silent-empty-capture hardening, missed it.
Fix: apply the same env=_sanitize_subprocess_env(cua_driver_child_env())
to the subprocess.run() call in _call_tool_via_cli(), mirroring the
sanctioned spawn site exactly (telemetry policy applied first, then
Hermes-managed secrets filtered).
The SOM/AX element list dropped labels for two extremely common cua-driver
render forms, leaving the model unable to target elements by name:
- [79] AXButton (Dark) -> parenthesised label
- [4] AXStaticText = "Wi-Fi" -> = "value" form
- [92] AXPopUpButton = "Automatic" -> = "value" form
The old regex only matched quoted "label" and id=Label, so System Settings
buttons/text/popups all surfaced with empty labels. That's why selecting the
macOS Appearance 'Dark' button by element index required guessing — the
labels weren't available to aim with.
Fix: extend _ELEMENT_LINE_RE to capture all four label forms (= "value",
"quoted", (parenthesised), id=Label), skipping a pure-digit (N) order number
in favour of the id= label. Verified live against System Settings: the
Appearance buttons now surface as Auto/Light/Dark.
Adds a regression test covering all label forms. Full suite: 84 passed.
The first fix handled the EAGAIN McpError path. But the persistent MCP
session (long-running gateway/desktop worker) has a second failure mode:
list_windows or get_window_state 'succeed' over MCP yet return a
degenerate/empty payload (no windows, or no screenshot + blank tree)
WITHOUT raising — typically when the bridge reconnected mid-call and
dropped the heavy response. That surfaced to the model as a silent 0x0
capture with no error and no fallback firing (0.00s empty return).
Fix: detect empty results in capture() and re-fetch over the CLI
transport before giving up:
- empty list_windows -> CLI re-fetch the window list
- empty get_window_state (som/ax) -> CLI re-fetch the AX tree + screenshot
- empty screenshot (vision) -> CLI re-fetch get_window_state for the PNG
Adds 2 regression tests. Full suite: 83 passed.
The cua-driver MCP stdio bridge intermittently (and on some machines
persistently) fails to forward heavier calls like get_window_state to
the daemon with POSIX EAGAIN — 'daemon transport error forwarding
get_window_state: Resource temporarily unavailable (os error 35)'.
The wrapper surfaced this as an empty 0x0 capture, so computer_use
returned blank screenshots even though the display, permissions, and
the daemon were all healthy (the direct 'cua-driver call' CLI path
worked fine throughout).
Fix: when the MCP path raises the transient/transport error, fall back
to the 'cua-driver call' subprocess transport, which talks to the
daemon over a different socket. The CLI fallback routes get_window_state
screenshots to a temp file via screenshot_out_file (tiny JSON response
instead of a multi-MB base64 blob that congests the socket), reads the
PNG back, retries with backoff, and remaps the JSON into the same
{data, images, structuredContent, isError} shape the MCP path produces
so capture()/_action() are transport-agnostic.
Adds _is_transient_daemon_error() classifier and 3 regression tests.
Verified live: captures that returned 0x0 now return full
1567x905 screenshots with the AX element tree.
The 'never reached ready' error (issue #57025) was undiagnosable — doctor
and MCP test pass while the wrapper times out, with no hint where startup
stalled. Track a phase marker through _lifecycle_coro (binary-check →
manifest-discovery → mcp-initialize → capability-discovery → ready) and
include it in the timeout RuntimeError plus a pointer to doctor and the
agent.log phase timings.
Complements the 15s→30s bump + success-path phase timing log from #58760.
On Windows, the cua-driver MCP session initialization can exceed the 15s
timeout due to manifest subprocess discovery + MCP transport setup.
This makes the computer_use tool permanently unavailable even though
hermes computer-use doctor and hermes mcp test both pass.
- Increase _ready_event.wait timeout from 15s to 30s
- Add startup timing instrumentation (manifest + mcp_init durations)
- Log timing at INFO level for diagnosability
Fixes#57025
On X11 a window's PID comes from the optional _NET_WM_PID property, so
cua-driver's list_windows legitimately returns pid: null for windows that
don't set it (desktop root, panels, override-redirect popups). capture()
and focus_app() coerced every entry via int(w["pid"]) inside a list
comprehension, so a single null-pid window raised TypeError and aborted
the whole enumeration before any screenshot — capture was impossible on
any X11 desktop with even one such window.
Route both ingestion sites through a new _ingest_windows() helper that
skips entries lacking a usable pid/window_id (uncapturable anyway) and
coerces the rest.
Adds tests/tools/test_computer_use_null_pid_windows.py covering the
helper's filtering/coercion and an end-to-end capture() regression that
reproduced the crash.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Vision mode called a `screenshot` MCP tool that cua-driver dropped in
0.5.x (full-window PNG capture was folded into `get_window_state`). The
driver replied "Unknown tool: screenshot", so `images` came back empty,
`png_b64` stayed None, and capture returned a 0x0 result with no image
on every call. `som`/`ax` were unaffected because they already use
`get_window_state`, which masked the regression.
Route vision by capability:
- driver advertises `screenshot` (older builds) -> use it (no AX walk)
- otherwise -> call `get_window_state` but discard the AX tree/elements,
returning only the PNG so vision stays free of element noise
- capabilities not yet discovered -> try `screenshot`, fall back to
`get_window_state` on an empty image, so the path self-heals
Add `_image_from_tool_result` to pull the PNG from either an MCP image
content-part or `structuredContent.screenshot_png_b64`, and use it on
the som path too so the image won't silently drop on driver builds that
deliver it via structuredContent instead of a content part.
Verified live (vision: 1568x954, 0 elements; som: image + 527 elements)
and with unit coverage of all four routing cases.
capture(app='screen'|'desktop') now resolves to the OS shell/desktop
window (Windows Progman/WorkerW desktop or Shell_TrayWnd taskbar, macOS
Finder/Dock) so 'show me my screen' and 'click the taskbar' work.
Previously capture() only matched application windows, and the schema
advertised 'or the whole screen' without any code path delivering it.
cua-driver is window-oriented (no virtual-desktop or per-monitor MCP
tool), so a single image still cannot span multiple monitors — the
schema now states this and the no-desktop-window path returns a clear
message instead of silently grabbing the frontmost app.
elphamale