feat(computer_use): cross-platform cua-driver (macOS/Windows/Linux)
Make the computer_use toolset platform-agnostic by driving cua-driver on macOS, Windows, and Linux. Consumes the 8 cua-driver decoupling surfaces (capability discovery, structuredContent AX tree, opaque element_token, click button enum, explicit mimeType, machine-readable manifest, structured list_windows, structured health_report), each degrading gracefully on older drivers. Adds `hermes computer-use doctor` (drives cua-driver health_report with a per-OS check matrix and an exit 0/1/2 ok/degraded/blocked contract), full typed wrappers for the previously-uncovered cua-driver tools plus a generic call_tool escape hatch, per-session agent-cursor lifecycle, platform-aware system-prompt guidance (host-deterministic, cache-safe), and honors HERMES_CUA_DRIVER_CMD end-to-end. Replaces the macOS-only skills/apple/macos-computer-use skill with a cross-platform skills/computer-use skill, and refreshes the EN + zh-Hans docs. Supersedes #44221 (Windows-enablement salvage of #30660). Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
This commit is contained in:
committed by
Teknium
parent
17dfc6bec4
commit
f2e37549c6
@@ -24,6 +24,13 @@ class UIElement:
|
||||
pid: int = 0 # owning process PID
|
||||
window_id: int = 0 # SkyLight / CG window ID
|
||||
attributes: Dict[str, Any] = field(default_factory=dict)
|
||||
# Opaque per-snapshot element handle from cua-driver
|
||||
# (trycua/cua#1961 — Surface 6 of NousResearch/hermes-agent#47072).
|
||||
# When set, downstream calls can pass it alongside `index` for
|
||||
# explicit stale-detection: a stale token returns an error from
|
||||
# cua-driver rather than silently re-resolving to a different
|
||||
# element. None for pre-#1961 drivers that didn't carry the field.
|
||||
element_token: Optional[str] = None
|
||||
|
||||
def center(self) -> Tuple[int, int]:
|
||||
x, y, w, h = self.bounds
|
||||
@@ -52,6 +59,12 @@ class CaptureResult:
|
||||
window_title: str = ""
|
||||
# Raw bytes we sent to Anthropic, for token estimation.
|
||||
png_bytes_len: int = 0
|
||||
# Explicit MIME type for `png_b64` when the backend supplied it
|
||||
# (cua-driver-rs emits `mimeType` on every image part as of
|
||||
# trycua/cua#1961 — Surface 7 of NousResearch/hermes-agent#47072).
|
||||
# When None, downstream consumers fall back to base64-prefix
|
||||
# sniffing for back-compat with older drivers.
|
||||
image_mime_type: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
|
||||
Reference in New Issue
Block a user