diff --git a/skills/autonomous-ai-agents/computer-use/SKILL.md b/skills/autonomous-ai-agents/computer-use/SKILL.md index ed280672cb..bd94b6c305 100644 --- a/skills/autonomous-ai-agents/computer-use/SKILL.md +++ b/skills/autonomous-ai-agents/computer-use/SKILL.md @@ -28,8 +28,9 @@ Hermes drives [cua-driver](https://github.com/trycua/cua) under the hood. This wrapper skill teaches the Hermes `computer_use` workflow and action vocabulary. Call the actions documented below instead of raw cua-driver MCP tools. For driver internals and platform-specific behavior, follow the Cua -skill installed by `cua-driver skills install`; that command detects Hermes -and links the skill pack automatically. +skill installed by `cua-driver skills install`. Hermes autodetection is a +planned cua-driver follow-up, so currently point Hermes at the resulting +`~/.cua-driver/skills/cua-driver` directory or symlink it into your skill space. ## The canonical workflow @@ -196,25 +197,26 @@ Prefer `isolated_new` unless the task genuinely needs the user's signed-in session — attaching to an existing profile exposes its live pages, cookies, and storage over the browser protocol. -Authorization paths for `existing_profile`, in preference order: +Authorization paths for `existing_profile`: -1. **Config grant (standard mode).** When +1. **Config grant (standard and unrestricted modes).** When `computer_use.grant_existing_profile: true` is set, the runtime is - launched pre-authorized (`--grant existing-profile`) and the prepare - succeeds against the exact proven `(pid, window_id)`. If it's not set, - the prepare fails closed — tell the user to flip that config key (and - restart the session) if they want this; do not retry or work around it. + launched pre-authorized in standard mode (`--grant existing-profile`) and + Hermes applies the same host-side floor in unrestricted mode. If it is not + set, both modes fail closed. Tell the user to flip that config key and + restart the session if they want this; do not retry or work around it. 2. **Bounded manifest.** When `computer_use.permission_mode: bounded` is configured with a reviewed `capability_manifest`, prepares inside the manifest's scope succeed without prompts and everything else fails closed. -3. **Explicit Hermes YOLO** (`--yolo`, `/yolo`, or `approvals.mode: off`) - launches a private cua-driver runtime in `unrestricted` after that risk - acceptance, so there are no runtime Cua approval prompts. + +Explicit Hermes YOLO (`--yolo`, `/yolo`, or `approvals.mode: off`) launches an +unrestricted runtime with no runtime Cua approval prompts, but it does not +substitute for `grant_existing_profile: true`. These settings belong to runtime launch. The agent cannot add or change them -after the runtime starts. Without one of these paths, `existing_profile` fails -closed. Report the refusal and name the config key; do not retry, downgrade -trust, or work around it. +after the runtime starts. Without the applicable grant or bounded manifest, +`existing_profile` fails closed. Report the refusal and name the config key; +do not retry, downgrade trust, or work around it. Every MCP transport owns a private lifecycle session inside the runtime. The public session name only labels cursor identity and session-scoped state. It @@ -381,7 +383,6 @@ These are platform deep dives, not duplicates — when the user reports `WINDOWS.md` for the UIA / UWP context that explains why and what to do differently. -When `cua-driver skills install` autodetects Hermes (planned follow-up -in trycua/cua), this happens automatically on install. Until then, ask -the user to run the command and the pack lands in their agent skill -space alongside this skill. +Hermes autodetection is a planned follow-up in trycua/cua. For now, the command +installs the pack under `~/.cua-driver/skills/cua-driver`; point Hermes at that +directory or symlink it into the user's skill space. diff --git a/tests/tools/test_computer_use_browser_contract_020.py b/tests/tools/test_computer_use_browser_contract_020.py index 56e07b693f..9562e5cedb 100644 --- a/tests/tools/test_computer_use_browser_contract_020.py +++ b/tests/tools/test_computer_use_browser_contract_020.py @@ -74,6 +74,30 @@ def test_browser_state_forwards_screenshot_request_and_preserves_mcp_image(): assert result["_mcp_images"] == [ {"data": "/9j/browser-shot", "mime_type": "image/jpeg"} ] + assert "screenshot_deferred" not in result + + +def test_browser_bind_reports_screenshot_deferred_only_when_no_image_returned(): + driver = _Driver([ + { + "structuredContent": { + "status": "ok", + "target_id": "target-a", + "binding_quality": "exact", + "mutation_allowed": True, + "tabs": [{"tab_id": "tab-a"}], + }, + } + ]) + + result = _route(driver).observe( + pid=101, + window_id=202, + include_screenshot=True, + ) + + assert result["screenshot_deferred"] is True + assert "_mcp_images" not in result def test_browser_state_dispatch_returns_mcp_image_as_multimodal_content(): @@ -81,9 +105,7 @@ def test_browser_state_dispatch_returns_mcp_image_as_multimodal_content(): backend.typed_browser_state.return_value = { "status": "ok", "url": "https://example.test/", - "_mcp_images": [ - {"data": "iVBORbrowser-shot", "mime_type": "image/png"} - ], + "_mcp_images": [{"data": "iVBORbrowser-shot", "mime_type": "image/png"}], } result = _dispatch( diff --git a/tools/computer_use/browser_route.py b/tools/computer_use/browser_route.py index 3e54ef6e8a..ef45cec297 100644 --- a/tools/computer_use/browser_route.py +++ b/tools/computer_use/browser_route.py @@ -309,9 +309,10 @@ class CuaTypedBrowserRoute: "snapshot_format, include_screenshot) to take the snapshot " "this binding requires before any mutation." ) - if include_screenshot: - # A bind carries no page content, so the flag had nothing to - # attach to. Surface that instead of dropping it silently. + if include_screenshot and not payload.get("_mcp_images"): + # A bind normally carries no page content. Report deferral + # only when the driver did not attach the requested image; + # some driver versions do return a native screenshot here. payload["screenshot_deferred"] = True payload["exact_binding"] = quality == "exact" if quality != "exact" or not mutation_allowed: diff --git a/website/docs/user-guide/features/computer-use.md b/website/docs/user-guide/features/computer-use.md index da0e2e1912..f84c6b5ba4 100644 --- a/website/docs/user-guide/features/computer-use.md +++ b/website/docs/user-guide/features/computer-use.md @@ -69,11 +69,13 @@ the upstream installer. A binary selected with `HERMES_CUA_DRIVER_CMD` stays under your control, so Hermes reports the incompatibility and leaves it unchanged. -If you install Cua Driver first, `cua-driver skills install` detects Hermes -and links Cua's skill pack into the Hermes skills directory automatically. -You can also register raw Cua MCP tools as a custom MCP server, but that is an -alternative for users who need the low-level interface. The built-in toolset -provides Hermes actions, configuration, approvals, and diagnostics. +If you install Cua Driver first, `cua-driver skills install` installs Cua's +skill pack under `~/.cua-driver/skills/cua-driver`. Hermes autodetection is a +planned cua-driver follow-up, so currently point Hermes at that directory or +symlink it into your skill space. You can also register raw Cua MCP tools as a +custom MCP server, but that is an alternative for users who need the low-level +interface. The built-in toolset provides Hermes actions, configuration, +approvals, and diagnostics. After installing, regardless of which path you took, grant the platform-appropriate prereqs: @@ -102,7 +104,7 @@ grant are launch settings. They cannot change after the runtime starts: |---|---|---|---| | Manual or smart approvals (default) | `standard` | Normal Hermes approvals; Cua stops at its protected boundary | Refuses unless `computer_use.grant_existing_profile: true` (one-time config opt-in) | | `computer_use.permission_mode: bounded` + reviewed manifest | private `bounded` daemon | You review and approve the capability manifest once, at launch | Allowed only within the manifest's declared profiles/origins/tools; everything else fails closed | -| `--yolo`, `/yolo`, or `approvals.mode: off` | private `unrestricted` daemon | One explicit Hermes risk acceptance; no runtime Cua prompts | Allowed within Cua's built-in, managed, and user policy ceilings | +| `--yolo`, `/yolo`, or `approvals.mode: off` | private `unrestricted` daemon | One explicit Hermes risk acceptance; no runtime Cua prompts | Refuses unless `computer_use.grant_existing_profile: true`; YOLO does not substitute for this grant | ### Attaching to your signed-in browser @@ -232,9 +234,11 @@ maintains directly: cua-driver skills install ``` -This command detects Hermes and links the pack into its skill directory -automatically. The wrapper remains the workflow layer and points to Cua's -installed skill for driver behavior. After running it, an agent gets access to: +The command installs the pack under `~/.cua-driver/skills/cua-driver`. Hermes +autodetection is a planned cua-driver follow-up, so currently point Hermes at +that directory or symlink it into your skill space. The wrapper remains the +workflow layer and points to Cua's installed skill for driver behavior. The +pack contains: | File | Topic | |---|---| @@ -388,7 +392,7 @@ Permission mode and manifest (see computer_use: permission_mode: standard # standard (default) | bounded capability_manifest: "" # capability manifest path, required for bounded - grant_existing_profile: false # opt-in: attach to signed-in browser in standard mode + grant_existing_profile: false # opt-in: attach in standard or unrestricted mode ``` Override the driver binary path (tests / CI / local builds): @@ -555,9 +559,9 @@ autostart pattern — see (macOS no-foreground contract, Windows UIA + Session 0, Linux AT-SPI + X11/Wayland, recording, browser pages), run `cua-driver skills install` and read `MACOS.md` / `WINDOWS.md` / - `LINUX.md` / `RECORDING.md` / `WEB_APPS.md`. Once `cua-driver skills - install` autodetects Hermes (planned follow-up), this happens - automatically on install. + `LINUX.md` / `RECORDING.md` / `WEB_APPS.md`. Hermes autodetection is a + planned follow-up; currently point Hermes at the installed pack directory + or symlink it into your skill space. - **cua.ai/docs** — the cua-driver project's documentation: - [What is computer use?](https://cua.ai/docs/explanation/what-is-computer-use) — concept intro - [The no-foreground contract](https://cua.ai/docs/explanation/the-no-foreground-contract) — *why* background mode matters diff --git a/website/docs/user-guide/skills/bundled/autonomous-ai-agents/autonomous-ai-agents-computer-use.md b/website/docs/user-guide/skills/bundled/autonomous-ai-agents/autonomous-ai-agents-computer-use.md index 5ae22756ce..40cfe3ff6f 100644 --- a/website/docs/user-guide/skills/bundled/autonomous-ai-agents/autonomous-ai-agents-computer-use.md +++ b/website/docs/user-guide/skills/bundled/autonomous-ai-agents/autonomous-ai-agents-computer-use.md @@ -44,8 +44,9 @@ Hermes drives [cua-driver](https://github.com/trycua/cua) under the hood. This wrapper skill teaches the Hermes `computer_use` workflow and action vocabulary. Call the actions documented below instead of raw cua-driver MCP tools. For driver internals and platform-specific behavior, follow the Cua -skill installed by `cua-driver skills install`; that command detects Hermes -and links the skill pack automatically. +skill installed by `cua-driver skills install`. Hermes autodetection is a +planned cua-driver follow-up, so currently point Hermes at the resulting +`~/.cua-driver/skills/cua-driver` directory or symlink it into your skill space. ## The canonical workflow @@ -201,25 +202,26 @@ Prefer `isolated_new` unless the task genuinely needs the user's signed-in session. Attaching to an existing profile exposes its live pages, cookies, and storage over the browser protocol. -Authorization paths for `existing_profile`, in preference order: +Authorization paths for `existing_profile`: -1. **Config grant (standard mode).** When +1. **Config grant (standard and unrestricted modes).** When `computer_use.grant_existing_profile: true` is set, the runtime is - launched pre-authorized (`--grant existing-profile`) and the prepare - succeeds against the exact proven `(pid, window_id)`. If it is unset, - the prepare fails closed. Tell the user to set that config key and restart - the session if they want this. Do not retry or work around it. + launched pre-authorized in standard mode (`--grant existing-profile`) and + Hermes applies the same host-side floor in unrestricted mode. If it is not + set, both modes fail closed. Tell the user to set that config key and + restart the session if they want this. Do not retry or work around it. 2. **Bounded manifest.** When `computer_use.permission_mode: bounded` is configured with a reviewed `capability_manifest`, prepares inside the manifest's scope succeed without prompts and everything else fails closed. -3. **Explicit Hermes YOLO** (`--yolo`, `/yolo`, or `approvals.mode: off`) - launches a private cua-driver runtime in `unrestricted` after that risk - acceptance, so there are no runtime Cua approval prompts. + +Explicit Hermes YOLO (`--yolo`, `/yolo`, or `approvals.mode: off`) launches an +unrestricted runtime with no runtime Cua approval prompts, but it does not +substitute for `grant_existing_profile: true`. These settings belong to runtime launch. The agent cannot add or change them -after the runtime starts. Without one of these paths, `existing_profile` fails -closed. Report the refusal and name the config key; do not retry, downgrade -trust, or work around it. +after the runtime starts. Without the applicable grant or bounded manifest, +`existing_profile` fails closed. Report the refusal and name the config key; +do not retry, downgrade trust, or work around it. Every MCP transport owns a private lifecycle session inside the runtime. The public session name only labels cursor identity and session-scoped state. It @@ -386,7 +388,6 @@ These are platform deep dives, not duplicates — when the user reports `WINDOWS.md` for the UIA / UWP context that explains why and what to do differently. -When `cua-driver skills install` autodetects Hermes (planned follow-up -in trycua/cua), this happens automatically on install. Until then, ask -the user to run the command and the pack lands in their agent skill -space alongside this skill. +Hermes autodetection is a planned follow-up in trycua/cua. For now, the command +installs the pack under `~/.cua-driver/skills/cua-driver`; point Hermes at that +directory or symlink it into the user's skill space.