feat(computer_use): cross-platform cua-driver (macOS/Windows/Linux)

Make the computer_use toolset platform-agnostic by driving cua-driver on
macOS, Windows, and Linux. Consumes the 8 cua-driver decoupling surfaces
(capability discovery, structuredContent AX tree, opaque element_token,
click button enum, explicit mimeType, machine-readable manifest,
structured list_windows, structured health_report), each degrading
gracefully on older drivers.

Adds `hermes computer-use doctor` (drives cua-driver health_report with a
per-OS check matrix and an exit 0/1/2 ok/degraded/blocked contract), full
typed wrappers for the previously-uncovered cua-driver tools plus a generic
call_tool escape hatch, per-session agent-cursor lifecycle, platform-aware
system-prompt guidance (host-deterministic, cache-safe), and honors
HERMES_CUA_DRIVER_CMD end-to-end.

Replaces the macOS-only skills/apple/macos-computer-use skill with a
cross-platform skills/computer-use skill, and refreshes the EN + zh-Hans
docs.

Supersedes #44221 (Windows-enablement salvage of #30660).

Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
This commit is contained in:
Francesco Bonacci
2026-06-21 20:04:05 -07:00
committed by Teknium
parent 17dfc6bec4
commit f2e37549c6
22 changed files with 4130 additions and 657 deletions
+13
View File
@@ -24,6 +24,13 @@ class UIElement:
pid: int = 0 # owning process PID
window_id: int = 0 # SkyLight / CG window ID
attributes: Dict[str, Any] = field(default_factory=dict)
# Opaque per-snapshot element handle from cua-driver
# (trycua/cua#1961 — Surface 6 of NousResearch/hermes-agent#47072).
# When set, downstream calls can pass it alongside `index` for
# explicit stale-detection: a stale token returns an error from
# cua-driver rather than silently re-resolving to a different
# element. None for pre-#1961 drivers that didn't carry the field.
element_token: Optional[str] = None
def center(self) -> Tuple[int, int]:
x, y, w, h = self.bounds
@@ -52,6 +59,12 @@ class CaptureResult:
window_title: str = ""
# Raw bytes we sent to Anthropic, for token estimation.
png_bytes_len: int = 0
# Explicit MIME type for `png_b64` when the backend supplied it
# (cua-driver-rs emits `mimeType` on every image part as of
# trycua/cua#1961 — Surface 7 of NousResearch/hermes-agent#47072).
# When None, downstream consumers fall back to base64-prefix
# sniffing for back-compat with older drivers.
image_mime_type: Optional[str] = None
@dataclass