ouyangbo/hermes-agent
1
0
Fork 0
Code Issues Pull Requests Actions 9 Packages Projects Releases Wiki Activity

fix(computer-use): align browser guidance and screenshots

Browse Source
This commit is contained in:
f-trycua
2026-08-16 12:52:23 -05:00
committed by Teknium
parent 5b010f448f
commit 12b1f0f83d
5 changed files with 84 additions and 55 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/autonomous-ai-agents/computer-use/SKILL.md
+19 -18
View File
@@ -28,8 +28,9 @@ Hermes drives [cua-driver](https://github.com/trycua/cua) under the hood.
This wrapper skill teaches the Hermes `computer_use` workflow and action
vocabulary. Call the actions documented below instead of raw cua-driver MCP
tools. For driver internals and platform-specific behavior, follow the Cua
skill installed by `cua-driver skills install`; that command detects Hermes
and links the skill pack automatically.
skill installed by `cua-driver skills install`. Hermes autodetection is a
planned cua-driver follow-up, so currently point Hermes at the resulting
`~/.cua-driver/skills/cua-driver` directory or symlink it into your skill space.
## The canonical workflow
@@ -196,25 +197,26 @@ Prefer `isolated_new` unless the task genuinely needs the user's signed-in
session — attaching to an existing profile exposes its live pages, cookies,
and storage over the browser protocol.
Authorization paths for `existing_profile`, in preference order:
Authorization paths for `existing_profile`:
1. **Config grant (standard mode).** When
1. **Config grant (standard and unrestricted modes).** When
`computer_use.grant_existing_profile: true` is set, the runtime is
launched pre-authorized (`--grant existing-profile`) and the prepare
succeeds against the exact proven `(pid, window_id)`. If it's not set,
the prepare fails closed — tell the user to flip that config key (and
restart the session) if they want this; do not retry or work around it.
launched pre-authorized in standard mode (`--grant existing-profile`) and
Hermes applies the same host-side floor in unrestricted mode. If it is not
set, both modes fail closed. Tell the user to flip that config key and
restart the session if they want this; do not retry or work around it.
2. **Bounded manifest.** When `computer_use.permission_mode: bounded` is
configured with a reviewed `capability_manifest`, prepares inside the
manifest's scope succeed without prompts and everything else fails closed.
3. **Explicit Hermes YOLO** (`--yolo`, `/yolo`, or `approvals.mode: off`)
launches a private cua-driver runtime in `unrestricted` after that risk
acceptance, so there are no runtime Cua approval prompts.
Explicit Hermes YOLO (`--yolo`, `/yolo`, or `approvals.mode: off`) launches an
unrestricted runtime with no runtime Cua approval prompts, but it does not
substitute for `grant_existing_profile: true`.
These settings belong to runtime launch. The agent cannot add or change them
after the runtime starts. Without one of these paths, `existing_profile` fails
closed. Report the refusal and name the config key; do not retry, downgrade
trust, or work around it.
after the runtime starts. Without the applicable grant or bounded manifest,
`existing_profile` fails closed. Report the refusal and name the config key;
do not retry, downgrade trust, or work around it.
Every MCP transport owns a private lifecycle session inside the runtime. The
public session name only labels cursor identity and session-scoped state. It
@@ -381,7 +383,6 @@ These are platform deep dives, not duplicates — when the user reports
`WINDOWS.md` for the UIA / UWP context that explains why and what to
do differently.
When `cua-driver skills install` autodetects Hermes (planned follow-up
in trycua/cua), this happens automatically on install. Until then, ask
the user to run the command and the pack lands in their agent skill
space alongside this skill.
Hermes autodetection is a planned follow-up in trycua/cua. For now, the command
installs the pack under `~/.cua-driver/skills/cua-driver`; point Hermes at that
directory or symlink it into the user's skill space.
tests/tools/test_computer_use_browser_contract_020.py
+25 -3
View File
@@ -74,6 +74,30 @@ def test_browser_state_forwards_screenshot_request_and_preserves_mcp_image():
assert result["_mcp_images"] == [
{"data": "/9j/browser-shot", "mime_type": "image/jpeg"}
]
assert "screenshot_deferred" not in result
def test_browser_bind_reports_screenshot_deferred_only_when_no_image_returned():
driver = _Driver([
{
"structuredContent": {
"status": "ok",
"target_id": "target-a",
"binding_quality": "exact",
"mutation_allowed": True,
"tabs": [{"tab_id": "tab-a"}],
},
}
])
result = _route(driver).observe(
pid=101,
window_id=202,
include_screenshot=True,
)
assert result["screenshot_deferred"] is True
assert "_mcp_images" not in result
def test_browser_state_dispatch_returns_mcp_image_as_multimodal_content():
@@ -81,9 +105,7 @@ def test_browser_state_dispatch_returns_mcp_image_as_multimodal_content():
backend.typed_browser_state.return_value = {
"status": "ok",
"url": "https://example.test/",
"_mcp_images": [
{"data": "iVBORbrowser-shot", "mime_type": "image/png"}
],
"_mcp_images": [{"data": "iVBORbrowser-shot", "mime_type": "image/png"}],
}
result = _dispatch(
tools/computer_use/browser_route.py
+4 -3
View File
@@ -309,9 +309,10 @@ class CuaTypedBrowserRoute:
"snapshot_format, include_screenshot) to take the snapshot "
"this binding requires before any mutation."
)
if include_screenshot:
# A bind carries no page content, so the flag had nothing to
# attach to. Surface that instead of dropping it silently.
if include_screenshot and not payload.get("_mcp_images"):
# A bind normally carries no page content. Report deferral
# only when the driver did not attach the requested image;
# some driver versions do return a native screenshot here.
payload["screenshot_deferred"] = True
payload["exact_binding"] = quality == "exact"
if quality != "exact" or not mutation_allowed:
website/docs/user-guide/features/computer-use.md
+17 -13
View File
@@ -69,11 +69,13 @@ the upstream installer. A binary selected with `HERMES_CUA_DRIVER_CMD` stays
under your control, so Hermes reports the incompatibility and leaves it
unchanged.
If you install Cua Driver first, `cua-driver skills install` detects Hermes
and links Cua's skill pack into the Hermes skills directory automatically.
You can also register raw Cua MCP tools as a custom MCP server, but that is an
alternative for users who need the low-level interface. The built-in toolset
provides Hermes actions, configuration, approvals, and diagnostics.
If you install Cua Driver first, `cua-driver skills install` installs Cua's
skill pack under `~/.cua-driver/skills/cua-driver`. Hermes autodetection is a
planned cua-driver follow-up, so currently point Hermes at that directory or
symlink it into your skill space. You can also register raw Cua MCP tools as a
custom MCP server, but that is an alternative for users who need the low-level
interface. The built-in toolset provides Hermes actions, configuration,
approvals, and diagnostics.
After installing, regardless of which path you took, grant the
platform-appropriate prereqs:
@@ -102,7 +104,7 @@ grant are launch settings. They cannot change after the runtime starts:
|---|---|---|---|
| Manual or smart approvals (default) | `standard` | Normal Hermes approvals; Cua stops at its protected boundary | Refuses unless `computer_use.grant_existing_profile: true` (one-time config opt-in) |
| `computer_use.permission_mode: bounded` + reviewed manifest | private `bounded` daemon | You review and approve the capability manifest once, at launch | Allowed only within the manifest's declared profiles/origins/tools; everything else fails closed |
| `--yolo`, `/yolo`, or `approvals.mode: off` | private `unrestricted` daemon | One explicit Hermes risk acceptance; no runtime Cua prompts | Allowed within Cua's built-in, managed, and user policy ceilings |
| `--yolo`, `/yolo`, or `approvals.mode: off` | private `unrestricted` daemon | One explicit Hermes risk acceptance; no runtime Cua prompts | Refuses unless `computer_use.grant_existing_profile: true`; YOLO does not substitute for this grant |
### Attaching to your signed-in browser
@@ -232,9 +234,11 @@ maintains directly:
cua-driver skills install
```
This command detects Hermes and links the pack into its skill directory
automatically. The wrapper remains the workflow layer and points to Cua's
installed skill for driver behavior. After running it, an agent gets access to:
The command installs the pack under `~/.cua-driver/skills/cua-driver`. Hermes
autodetection is a planned cua-driver follow-up, so currently point Hermes at
that directory or symlink it into your skill space. The wrapper remains the
workflow layer and points to Cua's installed skill for driver behavior. The
pack contains:
| File | Topic |
|---|---|
@@ -388,7 +392,7 @@ Permission mode and manifest (see
computer_use:
permission_mode: standard # standard (default) | bounded
capability_manifest: "" # capability manifest path, required for bounded
grant_existing_profile: false # opt-in: attach to signed-in browser in standard mode
grant_existing_profile: false # opt-in: attach in standard or unrestricted mode
```
Override the driver binary path (tests / CI / local builds):
@@ -555,9 +559,9 @@ autostart pattern — see
(macOS no-foreground contract, Windows UIA + Session 0, Linux AT-SPI
+ X11/Wayland, recording, browser pages), run
`cua-driver skills install` and read `MACOS.md` / `WINDOWS.md` /
`LINUX.md` / `RECORDING.md` / `WEB_APPS.md`. Once `cua-driver skills
install` autodetects Hermes (planned follow-up), this happens
automatically on install.
`LINUX.md` / `RECORDING.md` / `WEB_APPS.md`. Hermes autodetection is a
planned follow-up; currently point Hermes at the installed pack directory
or symlink it into your skill space.
- **cua.ai/docs** — the cua-driver project's documentation:
- [What is computer use?](https://cua.ai/docs/explanation/what-is-computer-use) — concept intro
- [The no-foreground contract](https://cua.ai/docs/explanation/the-no-foreground-contract) — *why* background mode matters
website/docs/user-guide/skills/bundled/autonomous-ai-agents/autonomous-ai-agents-computer-use.md
+19 -18
View File
@@ -44,8 +44,9 @@ Hermes drives [cua-driver](https://github.com/trycua/cua) under the hood.
This wrapper skill teaches the Hermes `computer_use` workflow and action
vocabulary. Call the actions documented below instead of raw cua-driver MCP
tools. For driver internals and platform-specific behavior, follow the Cua
skill installed by `cua-driver skills install`; that command detects Hermes
and links the skill pack automatically.
skill installed by `cua-driver skills install`. Hermes autodetection is a
planned cua-driver follow-up, so currently point Hermes at the resulting
`~/.cua-driver/skills/cua-driver` directory or symlink it into your skill space.
## The canonical workflow
@@ -201,25 +202,26 @@ Prefer `isolated_new` unless the task genuinely needs the user's signed-in
session. Attaching to an existing profile exposes its live pages, cookies,
and storage over the browser protocol.
Authorization paths for `existing_profile`, in preference order:
Authorization paths for `existing_profile`:
1. **Config grant (standard mode).** When
1. **Config grant (standard and unrestricted modes).** When
`computer_use.grant_existing_profile: true` is set, the runtime is
launched pre-authorized (`--grant existing-profile`) and the prepare
succeeds against the exact proven `(pid, window_id)`. If it is unset,
the prepare fails closed. Tell the user to set that config key and restart
the session if they want this. Do not retry or work around it.
launched pre-authorized in standard mode (`--grant existing-profile`) and
Hermes applies the same host-side floor in unrestricted mode. If it is not
set, both modes fail closed. Tell the user to set that config key and
restart the session if they want this. Do not retry or work around it.
2. **Bounded manifest.** When `computer_use.permission_mode: bounded` is
configured with a reviewed `capability_manifest`, prepares inside the
manifest's scope succeed without prompts and everything else fails closed.
3. **Explicit Hermes YOLO** (`--yolo`, `/yolo`, or `approvals.mode: off`)
launches a private cua-driver runtime in `unrestricted` after that risk
acceptance, so there are no runtime Cua approval prompts.
Explicit Hermes YOLO (`--yolo`, `/yolo`, or `approvals.mode: off`) launches an
unrestricted runtime with no runtime Cua approval prompts, but it does not
substitute for `grant_existing_profile: true`.
These settings belong to runtime launch. The agent cannot add or change them
after the runtime starts. Without one of these paths, `existing_profile` fails
closed. Report the refusal and name the config key; do not retry, downgrade
trust, or work around it.
after the runtime starts. Without the applicable grant or bounded manifest,
`existing_profile` fails closed. Report the refusal and name the config key;
do not retry, downgrade trust, or work around it.
Every MCP transport owns a private lifecycle session inside the runtime. The
public session name only labels cursor identity and session-scoped state. It
@@ -386,7 +388,6 @@ These are platform deep dives, not duplicates — when the user reports
`WINDOWS.md` for the UIA / UWP context that explains why and what to
do differently.
When `cua-driver skills install` autodetects Hermes (planned follow-up
in trycua/cua), this happens automatically on install. Until then, ask
the user to run the command and the pack lands in their agent skill
space alongside this skill.
Hermes autodetection is a planned follow-up in trycua/cua. For now, the command
installs the pack under `~/.cua-driver/skills/cua-driver`; point Hermes at that
directory or symlink it into the user's skill space.