Teknium
5033aedaa3
Reconstructs the 204-run benchmark battery behind #81958 (Browser Use CLI 3.0 mode) as a rerunnable eval under evals/browser-use/, following the toolperf_abeval / evals-compaction pattern. - tasks/easy.json + tasks/hard.json: the oracle-checked toscrape task batteries (5 easy, 6 hard) exactly as run for the PR - single_run.py: one cell = task x arm (base | pr | prns) x model x rep; throwaway HERMES_HOME, web-fetch creds stripped, arms pinned to separate trees via BUBENCH_BASE_TREE / BUBENCH_PR_TREE - orchestrate.py: resume-safe local-CDP battery driver - orchestrate_cloud.py: backend matrix (nous-cloud via the browser_use provider plugin, browserbase via REST) with per-cell session lifecycle - report.py: scorecard aggregation with vs-base token deltas - README.md: design, run instructions, and the recovered Aug 8-10 2026 baseline scorecards (hard battery, backend matrix, easy round 1, digest ablation) The original /tmp/bu-bench workspace was lost to a tmpfs reboot; harness and readouts were recovered verbatim from the benchmark session's tool-call history in state.db, with hardcoded paths parameterized. Smoke-verified live: report.py aggregation, and single-cell runs (pr + base arms) against a real headless Chrome CDP with sonnet-5 driving browser_exec, oracle pass.
27 lines
2.2 KiB
JSON
27 lines
2.2 KiB
JSON
{
|
|
"multi_page_filter": {
|
|
"prompt": "Using your browser tools, open https://books.toscrape.com , go to the 'Mystery' category, and go through ALL of its pages. Count how many Mystery books cost less than \u00a330.00, and identify the single cheapest Mystery book. Final answer must state the count and the cheapest book's exact title and price.",
|
|
"oracle_all": ["19", "Tastes Like Fear", "10.69"]
|
|
},
|
|
"rating_aggregate": {
|
|
"prompt": "Using your browser tools, open https://books.toscrape.com , browse the ENTIRE 'Mystery' category (all pages). Find every book with a five-star rating, then compute the average price of those five-star books. Final answer must state how many five-star Mystery books exist and the average price rounded to 2 decimals.",
|
|
"oracle_all": ["5", "32.27"]
|
|
},
|
|
"js_rendered": {
|
|
"prompt": "Using your browser tools, open https://quotes.toscrape.com/js/ (the content on this page is rendered by JavaScript). Tell me the author of the LAST quote on page 1. Final answer must name the author.",
|
|
"oracle_any": ["Steve Martin"]
|
|
},
|
|
"delayed_render": {
|
|
"prompt": "Using your browser tools, open https://quotes.toscrape.com/js-delayed/ (content appears only after a delayed JavaScript render). Tell me the author of the FIRST quote on the page. Final answer must name the author.",
|
|
"oracle_any": ["Albert Einstein"]
|
|
},
|
|
"login_then_extract": {
|
|
"prompt": "Using your browser tools: (1) log in at https://quotes.toscrape.com/login with username 'admin' and password 'admin' and confirm the login worked, (2) then navigate to the tag page for the tag 'books' and report how many quotes are shown there and which authors appear more than once on that page. Final answer must state the quote count and name every author who appears twice.",
|
|
"oracle_all": ["10", "Mark Twain", "Jane Austen"]
|
|
},
|
|
"cross_category_compare": {
|
|
"prompt": "Using your browser tools, open https://books.toscrape.com and compare the 'Mystery' and 'Historical Fiction' categories across ALL their pages: which category contains the single most expensive book overall? Final answer must name that category, the book's title, and its exact price.",
|
|
"oracle_all": ["Historical Fiction", "Sara de Vos", "55.55"]
|
|
}
|
|
}
|