 Teknium
|
037ce5cf75
|
eval(tool-search): check in the core-tool-deferral live A/B harness
The harness behind this PR's 288-run maintainer battery, ported from /tmp
into evals/ alongside the readtool / session_search_schema harnesses.
Arm trees parametrized via ABDEFER_BASE_TREE / ABDEFER_PR_TREE (pinned
plain checkouts), results/python roots via ABDEFER_RESULTS / ABDEFER_PYTHON.
- tasks.py: 14 tasks — per-deferred-tool coverage, multistep, long-range,
clarify ambiguity trap, eager-only control, false-discovery distractor;
programmatic graders with partial credit
- worker.py: isolated per-cell subprocess, temp HERMES_HOME, hermetic env,
seeded session DB with decoys, deterministic desktop/computer_use/
image_generate stubs, interactivity-fairness continuation, exit-3
infra-abort (misconfig never scores)
- orchestrator.py: resume-safe battery runner, wall timeouts, retry of
errored records, infra-abort fuse
- report.py: per-task A/B tables + mean-of-task-means
- results/SUMMARY.md: the shipped verdict; rep JSONs gitignored
Ported worker re-verified live post-port (real terra cell, score 1.0).
|
2026-08-29 18:42:59 -07:00 |
|