Commit Graph

12 Commits

Author SHA1 Message Date
nftpoetrist d6ace971d4 docs(computer-use): remove the deleted typed browser-page route
#95620 removed computer_use's cua_browser_* actions entirely
(browser_route.py, the action enum, and their dispatch/escalation hint) —
computer_use is desktop-only now, and page content goes through the
separate browser_navigate/browser_click/... toolset (or browser_exec
under the Browser Use CLI backend).

The skill doc never caught up: it still taught the model to call
cua_browser_state/cua_browser_prepare/etc. and to escalate to a "page"
rung that _enrich_escalation can no longer recommend. Following that
guidance fails schema validation on the first call.

- SKILL.md: replace the "Typed browser page rung" section (including the
  existing_profile authorization walkthrough, which described a
  model-facing action parameter that no longer exists anywhere in
  schema.py/tool.py) with a short pointer to the current browser toolset;
  drop "page" from the escalation.recommended union and the ladder step
  that referenced it.
- website/docs/user-guide/features/computer-use.md: grant_existing_profile
  and the rest of the permission-mode config are still real and current
  (cua_backend.py still reads them for the runtime launch grant) — only
  reworded the one sentence naming cua_browser_prepare as the mechanism
  that consumes the grant, since driving a signed-in browser window now
  goes through the standard capture/click/type actions instead.
- Regenerated the mirrored skill doc via
  website/scripts/generate-skill-docs.py rather than hand-editing it, per
  its own header. Kept the diff scoped to computer-use only.
2026-08-29 19:13:33 -07:00
Teknium 65605d4a7a fix(computer_use): pitch background-FIRST (not background-only) in schema, prompt block, and skill 2026-08-26 04:50:13 -07:00
f-trycua 12b1f0f83d fix(computer-use): align browser guidance and screenshots 2026-08-16 11:34:40 -07:00
Francesco Bonacci a403fe6f92 feat(computer-use): support Cua Driver 0.20 runtime contracts 2026-08-16 11:34:40 -07:00
Teknium b7f6280259 fix(computer-use): refuse wrong-window input on app= mismatch; hint near-miss actions
Live complex-action QA on a real KDE desktop (kcalc + kate multi-app
flows) found two dispatch gaps:

1. Wrong-window input reported as success. Input actions deliver to the
   backend's sticky target (last capture/focus_app); the app= argument
   models routinely pass on the input call itself was silently dropped.
   Proven live: with kcalc sticky, type(text='777', app='kate') returned
   ok:true and typed 777 INTO KCALC. New guard: provable mismatch
   (both names known, neither substring of the other — list_windows
   names are localized/variant) refuses with input_target_mismatch and
   a one-call fix instruction. Unknown current target fails open so
   legacy no-app flows are untouched.

2. Near-miss unknown actions were dead ends. A model emitting 'hotkey'
   got a bare unknown-action error. Suggestion map now names the real
   action ('did you mean key?') without aliasing — we never repair bad
   model output, we just point at the schema.

Also documents the verified-lost-keystroke rung in the computer-use
skill: KTextEditor (Kate/KWrite) discards synthetic X keystrokes at the
toolkit level — foreground type reports ok but AX shows nothing arrived,
and a raw XTest control fails identically outside our stack. Guidance:
after one verified-lost round trip, switch to file/DBus I/O instead of
looping the ladder.

Live proof on the fixed build: mismatch refused, kcalc display clean,
same call after capture(app=kate) succeeds, 'hotkey' suggests 'key'.
11 new tests; 158 sibling tests green.
2026-08-15 22:18:09 -07:00
Teknium 20cf326bd1 fix(computer-use): align browser authorization with live-verified cua-driver 0.19.3 contract
Live-tested against the real cua-driver 0.19.3 binary (Linux x86_64):

- bounded serve flags corrected: the daemon accepts
  --session-policy/--approve-session-policy, not the docs'
  --capability-manifest names (which it rejects). Verified end-to-end:
  a bounded daemon with a real policy file starts and reports running.
- browser-approve verified real but interactive-only (refuses without a
  TTY) and its token is a legacy compatibility path disabled by default
  on current drivers (per the live browser_prepare schema). Kept as a
  passthrough; no longer presented as the primary route.
- NEW primary standard-mode route, verified live: launch the runtime
  with cua-driver's trusted-launcher grant. config opt-in
  computer_use.grant_existing_profile: true appends
  --grant existing-profile to the standard-mode MCP spawn (MCP
  initialize verified accepting the flag). Default false = attachment
  keeps failing closed. Never applied to bounded/unrestricted daemons.
- Skill, system prompt, tool schema, and docs updated to the verified
  ladder: config grant > bounded manifest > YOLO; token = legacy.
2026-08-15 15:04:32 -07:00
Teknium 48dd9c87cf feat(computer-use): user-facing authorization for cua-driver browser attachment
Completes the typed cua_browser_* route (PR #74166 lineage) with the
authorization surface that makes existing-profile attachment and
repeatable bounded automation reachable by real users:

- hermes computer-use browser-approve: CLI passthrough that mints
  cua-driver's five-minute single-use attachment token for one exact
  (pid, window_id). The user, never the model, is the token source.
- approval_token passthrough on cua_browser_prepare (schema + dispatch +
  browser_route), forwarded only for existing_profile and only as a
  non-empty string.
- computer_use.permission_mode: bounded + capability_manifest config:
  private per-session embedded daemon launched with
  --capability-manifest/--approve-capability-manifest; missing manifest
  fails loudly. 'unrestricted' is deliberately NOT a config value —
  it stays bound to the explicit per-session YOLO toggle.
- Skill + system-prompt + docs guidance for the three authorization
  rungs and the isolated-profile-first default.

E2E-verified against a temp HERMES_HOME: real config resolution to
bounded, loud failure without a manifest, real argparse path driving a
fake cua-driver binary, standard default preserved.
2026-08-15 15:04:32 -07:00
teknium1 1c9433897c chore(skills): standards sweep — bring 42 bundled/optional skills up to hardline
Audited all 191 in-repo skills (77 bundled, 114 optional) against the
authoring standards (AGENTS.md hardline + PR #80800). Fixed 42:

- 8 overlong descriptions rewritten to <= 60 chars, one sentence, period
  (one-three-one-rule 525ch, drug-discovery 405ch, web-pentest 358ch,
  fitness-nutrition 352ch, neuroskill-bci 332ch, oss-forensics 320ch,
  computer-use 307ch, memento-flashcards 252ch)
- 24 skills with missing frontmatter fields: author (credited from git
  history: f-trycua, SHL0MS, teyrebaz33, haileymarshall, FurkanL0,
  teknium1), license, version, platforms, tags
- 7 machine-local paths scrubbed (/home/bb, /home/user, /home/ubuntu ->
  portable placeholders)
- 11 marketing-word intros reworded (Comprehensive/state-of-the-art)
- docs catalogs + per-skill pages regenerated, scope-disciplined

Deferred (not in this PR):
- 14 frontmatter/dir name mismatches — open PR #42788 already proposes
  the dir-rename approach for 4 of them; resolve there as one class
- 2 skills over 100k chars (pytorch-fsdp 159k, research-paper-writing
  103k) — need content splits into references/, separate PRs
- comfyui dangling related_skill resolves after the name-mismatch class
2026-08-08 15:27:29 -07:00
Francesco Bonacci c268397752 feat(computer_use): align cua-driver 0.10 permission modes 2026-07-29 12:19:37 -07:00
Francesco Bonacci 847e401b74 feat(computer_use): align cua-driver 0.9 contracts
Salvaged from PR #67807 by @f-trycua onto current main.

- Foreground gate: discover delivery_mode support from the live tools/list
  inputSchema.properties (fail closed), not the never-shipped
  input.delivery_mode capability token
- bring_to_front: standalone strict-schema MCP tool (inject_session=False),
  separate approval scope, requires foreground
- Verdict precedence: confirmed > unverifiable (verify before retry) >
  suspected_noop/refusal (escalate); surfaced as explicit verdict field
- Typed cua_browser_* route inside computer_use (browser_route.py) with
  exact-binding, adapter-injected session, snapshot-scoped refs
- Per-Hermes-session backend isolation + release_computer_use_session seam
  wired into AIAgent.close()
- Recorded 0.9 tools/list fixture replaces fabricated capability tokens
2026-07-29 12:19:37 -07:00
teknium1 73b01fb7b6 docs(skills): fix remaining 13 broken related_skills refs repo-wide
Widening pass on top of the #38820 salvage: a full-graph audit of every
SKILL.md (bundled + optional) found 13 more references to skills that
no longer exist. Classes:

- deleted in the 38d3c49aaf bundled-skill cleanup: generative-widgets,
  spotify, cloudflared-quick-tunnel, webhook-subscriptions,
  debugging-hermes-tui-commands -> dropped
- native-mcp absorbed into the hermes-agent hub skill -> re-pointed
- toolset names that were never skills: browser, image_gen -> dropped

Audit now reports zero broken related_skills references.
2026-07-24 07:54:05 -07:00
Teknium b718725121 fix(skills): move computer-use skill into the autonomous-ai-agents category
skills/computer-use/SKILL.md sat at the root of the bundled skills
tree as an uncategorized single-skill directory. Move it to
skills/autonomous-ai-agents/computer-use/ alongside claude-code,
codex, opencode, and hermes-agent.

- git mv skills/computer-use -> skills/autonomous-ai-agents/computer-use
  (history preserved)
- root-level-skill docstring examples (commands.py,
  generate-skill-docs.py) now use a generic placeholder instead of
  naming a specific skill, since categorizing root-level skills is
  ongoing
- docs: move the bundled page to autonomous-ai-agents-computer-use,
  update sidebars.ts, skills-catalog.md, and the two skill-path
  references in features/computer-use.md

E2E validated: fresh sync_skills() copies to the new nested path;
existing installs with the old flat copy keep it untouched (manifest
name-keyed, hash unchanged -> skipped, no duplicate);
_get_category_from_path resolves 'autonomous-ai-agents'; docusaurus
build green; 396 targeted tests pass.
2026-07-23 21:06:39 -07:00