Teknium
5033aedaa3
Reconstructs the 204-run benchmark battery behind #81958 (Browser Use CLI 3.0 mode) as a rerunnable eval under evals/browser-use/, following the toolperf_abeval / evals-compaction pattern. - tasks/easy.json + tasks/hard.json: the oracle-checked toscrape task batteries (5 easy, 6 hard) exactly as run for the PR - single_run.py: one cell = task x arm (base | pr | prns) x model x rep; throwaway HERMES_HOME, web-fetch creds stripped, arms pinned to separate trees via BUBENCH_BASE_TREE / BUBENCH_PR_TREE - orchestrate.py: resume-safe local-CDP battery driver - orchestrate_cloud.py: backend matrix (nous-cloud via the browser_use provider plugin, browserbase via REST) with per-cell session lifecycle - report.py: scorecard aggregation with vs-base token deltas - README.md: design, run instructions, and the recovered Aug 8-10 2026 baseline scorecards (hard battery, backend matrix, easy round 1, digest ablation) The original /tmp/bu-bench workspace was lost to a tmpfs reboot; harness and readouts were recovered verbatim from the benchmark session's tool-call history in state.db, with hardcoded paths parameterized. Smoke-verified live: report.py aggregation, and single-cell runs (pr + base arms) against a real headless Chrome CDP with sonnet-5 driving browser_exec, oracle pass.
23 lines
1.3 KiB
JSON
23 lines
1.3 KiB
JSON
{
|
|
"simple_price": {
|
|
"prompt": "Using your browser tools, open https://books.toscrape.com , find the book 'Sharp Objects' and tell me its listed price. Give the exact price in your final answer.",
|
|
"oracle_any": ["47.82"]
|
|
},
|
|
"multi_extract": {
|
|
"prompt": "Using your browser tools, go to https://books.toscrape.com , open the 'Travel' category, and list EVERY book in that category with its price. In your final answer state the total count of books and the title and price of the cheapest one.",
|
|
"oracle_all": ["11", "23.21"]
|
|
},
|
|
"count_aggregate": {
|
|
"prompt": "Using your browser tools, open https://quotes.toscrape.com and determine which author appears most often on page 1 and how many quotes they have there. Final answer must name the author and the count.",
|
|
"oracle_all": ["Einstein", "3"]
|
|
},
|
|
"form_login": {
|
|
"prompt": "Using your browser tools, go to https://quotes.toscrape.com/login , log in with username 'admin' and password 'admin', and tell me the text of the link that replaced 'Login' in the top navigation after logging in.",
|
|
"oracle_any": ["Logout", "logout"]
|
|
},
|
|
"pagination_nav": {
|
|
"prompt": "Using your browser tools, open https://quotes.toscrape.com , navigate to page 2, and tell me the author of the FIRST quote on page 2.",
|
|
"oracle_any": ["Marilyn Monroe"]
|
|
}
|
|
}
|