ouyangbo/hermes-agent
1
0
Fork 0
Code Issues Pull Requests Actions 9 Packages Projects Releases Wiki Activity

feat(computer_use): cross-platform cua-driver (macOS/Windows/Linux)

Browse Source 
Make the computer_use toolset platform-agnostic by driving cua-driver on
macOS, Windows, and Linux. Consumes the 8 cua-driver decoupling surfaces
(capability discovery, structuredContent AX tree, opaque element_token,
click button enum, explicit mimeType, machine-readable manifest,
structured list_windows, structured health_report), each degrading
gracefully on older drivers.

Adds `hermes computer-use doctor` (drives cua-driver health_report with a
per-OS check matrix and an exit 0/1/2 ok/degraded/blocked contract), full
typed wrappers for the previously-uncovered cua-driver tools plus a generic
call_tool escape hatch, per-session agent-cursor lifecycle, platform-aware
system-prompt guidance (host-deterministic, cache-safe), and honors
HERMES_CUA_DRIVER_CMD end-to-end.

Replaces the macOS-only skills/apple/macos-computer-use skill with a
cross-platform skills/computer-use skill, and refreshes the EN + zh-Hans
docs.

Supersedes #44221 (Windows-enablement salvage of #30660).

Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
This commit is contained in:
Francesco Bonacci
2026-06-21 20:04:05 -07:00
committed by Teknium
parent 17dfc6bec4
commit f2e37549c6
22 changed files with 4130 additions and 657 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/computer_use/schema.py
+13 -9
View File
@@ -16,14 +16,15 @@ from typing import Any, Dict
COMPUTER_USE_SCHEMA: Dict[str, Any] = {
"name": "computer_use",
"description": (
"Drive the macOS desktop in the background — screenshots, mouse, "
"keyboard, scroll, drag — without stealing the user's cursor, "
"keyboard focus, or Space. Preferred workflow: call with "
"Drive the desktop in the background via cua-driver — screenshots, "
"mouse, keyboard, scroll, drag — without stealing the user's cursor "
"or keyboard focus. Supported on macOS, Windows, and Linux. "
"Preferred workflow: call with "
"action='capture' (mode='som' gives numbered element overlays), "
"then click by `element` index for reliability. Pixel coordinates "
"are supported for models trained on them. Works on any window — "
"hidden, minimized, on another Space, or behind another app. "
"macOS only; requires cua-driver to be installed."
"hidden, minimized, or behind another app. Requires cua-driver to "
"be installed."
),
"parameters": {
"type": "object",
@@ -70,9 +71,9 @@ COMPUTER_USE_SCHEMA: Dict[str, Any] = {
"type": "string",
"description": (
"Optional. Limit capture/action to a specific app "
"(by name, e.g. 'Safari', or bundle ID, "
"'com.apple.Safari'). If omitted, operates on the "
"frontmost app's window or the whole screen."
"(by name, e.g. 'Safari' or 'Notepad', or bundle ID "
"where the platform supports it). If omitted, operates "
"on the frontmost app's window or the whole screen."
),
},
"max_elements": {
@@ -126,7 +127,10 @@ COMPUTER_USE_SCHEMA: Dict[str, Any] = {
"type": "array",
"items": {
"type": "string",
"enum": ["cmd", "shift", "option", "alt", "ctrl", "fn"],
"enum": [
"cmd", "shift", "option", "alt", "ctrl", "fn",
"win", "windows", "super", "meta",
],
},
"description": "Modifier keys held during the action.",
},